Uber’s Translation Cache Spent 800 MB Storing Strings Nobody Read

Eight hundred megabytes of translation strings became forty. The team did not compress them and did not bolt another layer in front. Uber’s Go services load translation tables per service, and memory profiles showed why the bill kept climbing: in some services garbage collection was burning more than 20 percent of CPU, with translations accounting for roughly 70 percent of allocated objects. The InfoQ summary covers a redesign that cut the initial footprint by over 90 percent while holding p99 cache miss latency below one millisecond.

Two details in that summary matter more than the headline percentage.

The map was the wrong shape

Uber started by replacing sync.Map with a plain map[string]string under an RWMutex. That change alone improved CPU utilization by more than 30 percent in the affected service, and it is the least surprising result in the post once you look at what sync.Map is for. It targets keys written once and read from many goroutines, or key sets that different goroutines never touch in common. Translation tables are neither. They are read-mostly, refreshed wholesale now and then, and never deleted key by key. Under that workload sync.Map allocates an entry struct and an interface box per key, and at two to three million keys the collector spends its day walking pointers nobody needs walked.

I watch people reach for sync.Map to avoid reasoning about locking. It reads as the concurrency-aware choice, when in practice it is a specialized structure that costs memory in exactly the case most services have.

That fixed the CPU graph and did nothing for the memory graph. Those were two separate problems wearing one name, which is where a lot of tuning work stalls: you fix what the profiler pointed at, the numbers improve, and the failure stays in place.

Peak memory is a swap-time property

The out-of-memory kills had nothing to do with steady state. The cache refreshed by building a complete replacement in the background and atomically swapping it in, so for the length of a rebuild both copies were resident. At an 800 MB plateau that means 1.6 GB of live heap for some seconds, on every refresh, forever. Services that looked comfortable at rest died on a schedule nobody had charted, because the average looked fine and the average was never the constraint.

Uber considered an external cache and considered generating per-service translation subsets at build time. Both work, both push the problem into another team’s pager. If the translation table already ships inside the service binary, adding a network hop and a second operational surface to hold strings that mostly go unread is a bad trade. They rejected both.

Keys stay resident, values leave

The design they landed on keeps every key in memory and moves values to disk-backed files. The table is split into 256 shards, each with its own read-write lock, its own in-memory entries, and its own file. Keys are hashed down to six bytes to shrink metadata, with a fallback to the original key string on collision. A time-based LRU policy governs which values stay resident, and a refresh preloads whatever translations were touched in the previous 15 minutes so the new cache does not start cold.

The numbers are the argument. A dataset that used about 800 MB for entries now needs around 40 MB of metadata. With a 50,000-entry cap, resident memory typically lands between 80 and 120 MB. On one workload, 30,000 of more than four million entries were in memory and the hit rate still held above 99.9 percent.

Keeping the keys resident is what makes the disk tier cheap. A lookup either hits a value in RAM or reads one known offset from one known shard file. Page the keys out as well and every miss becomes two I/Os behind a lock instead of one.

Sub-millisecond p99 on a disk miss is only credible because of that split. The resident set of values is tiny, the index is complete, and each miss touches a single file with no directory walk.

The design assumes a skewed access distribution, and the writeup says it quietly. Thirty thousand entries covering four million keys is a Zipf curve, not a uniform one. If traffic spread evenly across every key, the resident set would grow until the SSD became the bottleneck and this architecture would collapse into a slow in-memory map with extra files. Uber could publish these numbers because their usage was as lopsided as most real translation traffic turns out to be.

The test I would run before copying any of it: take a day of production traffic, count distinct keys touched inside a 15-minute window, and stare at that number. It sets your memory floor, because the warmup policy will load exactly that much on every refresh. Tens of thousands and the hybrid pays for itself. If the window needs millions of distinct keys, the strings were never going to fit a map that small, and the design that helped Uber just moves your pager threshold somewhere else.

Press ⌘K to search & switch themes