1.6 Million Hams in Your Pocket: Compacting the FCC Callsign Database

The problem

HTCommander can look up any US amateur callsign offline — name, license class, status, city/state/ZIP, expiration — with no server, no SQLite, no network round-trip. The data comes from the FCC's Universal Licensing System (ULS): a weekly "complete" amateur dump of pipe-delimited .dat files that, once joined, describes roughly 1.59 million licenses.

That's a lot of rows to ship to a phone. The naive approaches all disappoint:

• Bundle a SQLite database — pulls in a dependency, and the file is large.
• Ship JSON/CSV — enormous, slow to parse, and you'd load it all into RAM.
• Query an online API — defeats the entire point of offline lookup.

So HTCommander uses a purpose-built binary format, the .cdb ("Callsign DataBase"). The design has exactly two jobs:

1. Be small — both the download and the on-disk file.
2. Be searchable in place — find one callsign out of 1.6 million with a handful of reads, without loading the whole thing into memory.

Everything below follows from those two goals. The reader/writer lives in callsign_database.dart; the offline builder that produces the shipped file is build_fcc_db.py. The two must agree on every byte, and a round-trip test plus a Python→Dart cross-check keep them honest.

The shape of a searchable file

The core idea is old and reliable: a sorted index you binary-search, next to a records section you seek into.

┌───────────────┐
│ Header (64 B) │  magic, counts, section offsets, epoch
├───────────────┤
│ Dictionaries  │  state · class/status · city  (shared lookup tables)
├───────────────┤
│ Keys block    │  one packed key per record, sorted ascending
├───────────────┤
│ Lengths block │  one u16 record length per record
├───────────────┤
│ Records block │  the variable-length record bodies, in key order
└───────────────┘

To look up K7VZT, we:

1. Pack the callsign into an integer key (more on that below).
2. Binary-search the keys block — ~21 comparisons for 1.6 M records.
3. Use the record's position to find its byte offset, then read and decode one record body.

Only the keys and a couple of small tables live in memory; the record bodies stay on disk (or in the downloaded buffer) and are read one at a time. All multi-byte integers are little-endian unless noted.

Trick 1: Don't store what you can reconstruct — the callsign

The obvious thing to put in each record is the callsign. But the index already contains it — that's what we binary-search on. Storing it again in the body is pure duplication.

US amateur callsigns are uppercase [A-Z0-9], six characters or fewer. So the key is the callsign, and the record body omits it entirely. On read we reconstruct the callsign from the key. That's ~7 bytes/record gone for free.

Trick 2: Pack the key into base-37

Naively, a key is 8 ASCII bytes (max callsign length, zero-padded). But each position only ever holds one of 37 things: nothing (padding), a digit 0–9, or a letter A–Z. That's a base-37 digit.

Eight base-37 digits fit in $37^8 \approx 3.5\times10^{12}$, which needs only 42 bits — so the whole key packs into a 6-byte big-endian integer instead of 8 bytes.

The clever part is preserving sort order. Binary search needs the packed integers to sort exactly like the original padded callsigns. We map:

and place the real characters in the high digits, so shorter callsigns (padding in the low digits) sort before longer ones sharing a prefix — the same result as comparing zero-padded ASCII, because padding (0) is the smallest code. Store the integer big-endian and a plain byte compare on the keys block still matches integer order. Lookups even get slightly faster: we compare 48-bit ints instead of walking 8-byte arrays.

// 'K7VZT' -> value*37 + code, real chars in the high digits, then
// multiply by 37 for each missing character to push padding low.
value = value * 37 + code;   // per character
for (; n < keyChars; n++) value *= 37;   // pad the tail

Trick 3: An index with no offsets

Classic layouts store, per entry, (key, offset) — the key to search and the absolute byte offset of the record. That offset is a u32: four bytes × 1.6 M = ~6 MB of pointers.

But the records are written in the same order as the keys, back to back. So we don't need absolute offsets at all — we store each record's length as a u16 in a separate "lengths block," and reconstruct the offsets by prefix-summing them once when the file opens. Ten bytes per index entry (6 key + 2 length… plus the record itself) instead of twelve, and the length column is far more compressible than a column of ever-growing absolute offsets.

Splitting keys and lengths into separate contiguous blocks (rather than interleaving them) also helps the compressor: each block is internally self-similar.

Trick 4: Dates as 16-bit day counts — with the epoch in the header

License expiration is a date. Stored as YYYYMMDD it's a u32 (4 bytes). But we don't need absolute precision across all of history — just a few decades around now. So we store days since a base date in a u16 (2 bytes), where 0 means "unknown."

The nice touch: the base date lives in the file header (epochDate), not hard-coded in the reader. A u16 day count covers ~179 years, so with a base of 2000-01-01 the window runs to ~2179. If we ever approach that, a future build just slides the epoch forward — the reader always honors whatever the header says. It works "forever" without a format change.

Trick 5–7: Dictionaries for repetitive fields

Three fields repeat heavily across 1.6 M records. Instead of writing the value in every record, we write each distinct value once in a small dictionary and let records reference it by a compact index. The dictionaries are built from the data itself (sorted for reproducible output) and stored in the file, so the reader never assumes a fixed set — any value the FCC emits is handled.

• State — ~60 distinct 2-letter codes. Dictionary of 2-byte entries; each record stores a 1-byte index. (4 bytes → 1.)
• Class + status — a handful of single letters each. We pack the pair (operatorClass, status) into one 2-byte dictionary entry and reference it with a single 1-byte index. (2 bytes → 1, and both fields covered by one lookup.)
• City — tens of thousands of distinct names, but wildly repetitive ("SPRINGFIELD" appears everywhere). A dictionary of u8 len + UTF-8 names, referenced by a 24-bit index (3 bytes, room for ~16 M cities). This turns a ~10-byte inline string into a 3-byte pointer and deduplicates the rest — the single biggest raw win.

name is deliberately not dictionary-encoded: person names are mostly unique, so a dictionary would buy nothing. It's the one variable-length field kept inline, and it rides on the whole-file compressor instead.

Trick 8: ZIP as a number

A ZIP code looks like text but is really an integer: 06111 or a 9-digit ZIP+4 like 061111234. We pack it into a fixed u32 (0xFFFFFFFF = none), and reconstruct on read by zero-padding to 5 or 9 digits — the magnitude tells us which (a 9-digit ZIP is always ≥ 100000). Leading zeros survive the round-trip. FCC amateur ZIPs are numeric (foreign addresses leave the field blank, which maps cleanly to the sentinel), so this is lossless in practice, and a fixed-width number compresses better than a length-prefixed string.

Trick 9: xz instead of zip for the download

Everything above shrinks the uncompressed file. The download gets one more lever: the transport compressor.

The database is overwhelmingly text (names and city dictionary), which compresses well — but DEFLATE (plain .zip) leaves a lot on the table. Switching the download to xz/LZMA typically buys another 25–40% over DEFLATE on this kind of data. The app already depends on the archive package, which ships an XZDecoder, so decoding is a one-liner; the builder writes the stream with Python's stdlib lzma. xz embeds no filenames or timestamps, so identical FCC input still produces a byte-identical archive.

The format, precisely

For anyone implementing a reader, here is the whole v2 layout.

Header (64 bytes):

Sections, in file order:

• State dictionary — stateCount × 2 bytes (ASCII, zero-padded).
• Class/status dictionary — classStatusCount × 2 bytes: [classByte, statusByte] (0 = empty).
• City dictionary — cityCount entries of u8 len + UTF-8.
• Keys block — recordCount × 6-byte big-endian packed base-37 keys, sorted ascending.
• Lengths block — recordCount × u16 record byte-lengths.
• Records block — the bodies, in key order.

Record body:

The callsign is not in the body — it's reconstructed from the key.

Where the bytes went

Field by field, a typical record shrinks dramatically versus a first-cut "just write everything" layout:

That's roughly half the uncompressed size of the naive record — before xz, which then compresses the text-heavy remainder far better than the old zip. The three dictionaries cost only a few hundred kilobytes total, shared across all 1.6 million records.

The numbers, measured

Theory is nice; here is the real deployed database. The same weekly FCC dump (1,588,146 licenses), built the old way and the new way:

That's a 45.7% smaller download — a 1.84× reduction, ~17.9 MiB shaved off — for the exact same records. The win is the format compaction (packed keys, dictionaries, numeric ZIPs, epoch dates, offset-free index, dropped callsign) and the xz-over-zip switch working together. On a phone on cellular data, that's the difference an operator actually feels.

How a lookup actually runs

Putting it together, lookup("K7VZT-5"):

1. Pack the callsign to a base-37 integer, stopping at the - (SSID is ignored): K7VZT.
2. Binary-search the in-memory keys block for that integer → record index.
3. Offset = prefix-summed lengths → the record's byte range.
4. Read + decode one record body: name inline; city/state/class/status via their dictionary indices; ZIP unpacked from its u32; expiration expanded from epochDate + days.
5. The callsign itself is unpacked from the key.

One binary search, one small read, a few array indexes. No parsing of 1.6 million rows, no external database engine.

Reproducibility and versioning

Two properties make this safe to ship on a schedule:

• Byte-identical builds. Given the same FCC data date, the builder emits an identical file every time — dictionaries are sorted, the source date is fixed from the FCC Last-Modified header, and xz carries no timestamps. The CI job only republishes when the FCC data actually changes.
• A version gate. The header carries formatVersion, and the app validates a freshly downloaded file before installing it. An older app simply refuses a newer format rather than misreading it — so the format can keep evolving.

What's next: incremental updates via an overlay database

The full download is ~21 MiB, and the CI republishes it whenever the FCC posts a new weekly dump. But week to week the FCC changes only a sliver of the 1.6 million records — new grants, renewals (a fresh expiry or status), the odd address change, cancellations. Re-shipping the entire file to move a few tens of thousands of records is wasteful on a phone's cellular data. The next iteration adds incremental updates — and does it without any of the usual patch-format machinery.

The key idea: the diff is itself a .cdb. Instead of a bespoke patch format that the app has to parse, merge, and re-encode, we ship a second, tiny database — an overlay — in the exact same format as the baseline. The app then does two lookups instead of one: it searches the overlay first, and only falls through to the baseline on a miss. A hit in the overlay wins. That's the entire mechanism:

lookup("K7VZT")
  ├─ binary-search the OVERLAY  (tens of thousands of records)  → hit? done.
  └─ binary-search the BASELINE (1.6 million records)           → fall-through.

Two binary searches instead of one — about 15 comparisons in the overlay plus ~21 in the baseline, still microseconds. No decode-merge-re-encode step, no multi-hundred-megabyte rebuild on the device, and zero new format: the overlay reuses the reader, the writer, the xz transport, the round-trip test, and the download/validate path already in callsign_database.dart and callsign_lookup_service.dart.

A few consequences fall out of this design:

• The overlay is cumulative since the baseline. Each week it's rebuilt to contain every record that changed since the last full baseline, not just the last week's. So a device only ever needs two files — the baseline plus the latest overlay — no matter how many weeks it skipped. No patch chains, no "apply these six diffs in order," no fallback gymnastics.
• Updates are free; deletes are ignored. A renewal, status flip, vanity reassignment, or address change is just a newer record that the overlay supersedes — always fresh, because the overlay wins. The one thing an overlay can't express is a deletion: a licence fully removed from the FCC set keeps showing its stale baseline record. For an offline lookup tool that's cosmetic, and it buys enormous simplicity, so deletes are deliberately not tracked. (A tombstone list could close the gap later if it ever matters.)
• The baseline and overlay cadences decouple. The full baseline is republished only occasionally; the small overlay is refreshed weekly and grows from nothing up to a few MiB as changes accumulate.

When does the overlay stop being worth it? As the weeks pass, the cumulative overlay grows. At some point re-downloading a bloated overlay is no better than just taking a fresh baseline. So the CI job measures the ratio

$$r = \frac{\text{size}(\text{overlay.xz})}{\text{size}(\text{baseline.xz})}$$

after each build. While $r$ stays under a threshold (initially 20%), it publishes only the refreshed overlay and leaves the baseline untouched. Once $r$ crosses the threshold — or on the very first run, when there's no baseline yet — the job promotes: the freshly built full database becomes the new baseline and the overlay resets to empty. The diff never grows unbounded; it self-heals into a new baseline exactly when carrying it stops paying off.

The 20% figure isn't arbitrary. For a weekly updater the long-run average download works out to $C(\theta) = \tfrac{\theta B}{2} + \tfrac{g}{\theta}$, where $B$ is the baseline size and $g$ the overlay's weekly growth — the first term is overlay bloat, the second is amortizing the occasional full baseline. Minimizing gives an optimal threshold $\theta^* = \sqrt{2g/B}$, i.e. roughly $\sqrt{2f}$ where $f$ is the weekly growth as a fraction of the baseline. With the FCC's ~2% weekly churn that lands near 20%, promoting a fresh baseline about every ten weeks and cutting a weekly updater's traffic from ~21 MiB to under 5 MiB. The cost curve is flat between ~15% and ~30%, so the exact number is forgiving; it's kept as a workflow input so it can be retuned once real overlay growth is measured.

Generating the overlay stays disk- and memory-thrifty, in the same spirit as the streaming full build. The job downloads the current baseline .cdb.xz from the rolling release (~21 MiB), decompresses it, and builds the new full database to a temporary file — two ~50 MiB files, trivial on a CI runner. It then walks the two sorted key blocks in lockstep — a streaming merge-join — decoding each side through the same reader and emitting an overlay record wherever a key is new or its decoded fields differ. One record from each side at a time, so peak memory stays flat regardless of the 1.6-million-record total.

The manifest grows a nested section, backward-compatibly. Its top-level fields keep describing the baseline, so an older app that never heard of overlays simply keeps downloading the full baseline as it does today. A new overlay object alongside them describes the small file that newer apps also fetch:

{
  "schemaVersion": 2,
  "sourceDate": 20260712, "url": ".../fcc_amateur.cdb.xz", "recordCount": 1588146,
  "md5": "…", "sizeBytes": 22400780,               // ← the baseline (legacy fields)
  "overlay": {
    "sourceDate": 20260719, "url": ".../fcc_amateur_overlay.cdb.xz",
    "recordCount": 20345, "md5": "…", "sizeBytes": 512345
  }
}

On the device, the update logic reads simply: if the baseline's sourceDate changed (a promotion), pull the new baseline and a fresh overlay; otherwise, if only the overlay's sourceDate moved, pull just the overlay. A weekly update becomes a sub-megabyte download instead of 21 MiB — the same win the xz switch delivered, again, for operators who update often.

The honest ledger

A few deliberate trade-offs worth naming:

• The callsign, ZIP, and dictionary tricks assume US amateur data shape (alphanumeric callsigns ≤ 8 chars, numeric ZIPs, small state/class/status alphabets). Values outside those assumptions degrade gracefully — a non-numeric ZIP becomes "none," an unmapped index reads as empty — but this format is not a general-purpose ULS mirror. The same format does stretch to other countries with a few targeted encoding changes — see Compacting the Canadian Callsign Database, which reuses this exact .cdb behind a single header flag to handle alphanumeric postal codes, qualification bitmasks, and non-expiring licences.
• We keep all licenses, including expired and cancelled ones. Filtering to active-only would be the single largest size cut available, but it's a product decision, not an encoding one — so it stays opt-in for now.
• name is left uncompressed inline. A name dictionary was measured and discarded: near-zero payoff for real added complexity.

The result is a file that a phone can download quickly, store compactly, and search instantly — 1.6 million hams, offline, in your pocket.

Related: Compacting the Canadian Callsign Database · callsign database reader/writer · offline builder