How we saved 100 terabytes of memory by optimizing 1.1.1.1’s DNS cache
- Scale and Impact: Cloudflare's Big Pineapple platform manages over 250 billion DNS cache entries. At this volume, wasting a single byte per entry wastes over 250 GB of RAM fleet-wide.
- Total Gains: Implementing five key memory layout optimizations reduced per-entry memory consumption by over 50%, saving approximately 100 terabytes of RAM (equivalent to ~130 Gen 13 servers), while boosting insert throughput by 43% and lowering lookup latency by 19%.
- Key Optimizations:
- Replacing
Vec<T>andStringwithBox<[T]>andBox<str>: Eliminated unneeded 8-byte capacity fields and unused reserved heap allocations for immutable cached entries, saving over 15 TB alone. - Single Array with Offsets: Merged separate record lists (answer, authority, and additional sections) into a single contiguous array using
u16offsets instead of multiple 8-byte pointers and lengths. - Bitflag Packing and Alignment Optimization: Consolidated boolean fields into bitflags and eliminated struct padding overhead.
- Deduplicating Record Owners: Avoided storing duplicate domain names when record owners matched the queried domain name.
- Replacing
- Benchmarking & Validation: Used a custom memory allocator wrapping Rust's system allocator to simulate production distributions (56% A, 25% AAAA, 19% TXT records) and verify reduced heap churn and improved memory locality.
Hacker News Discussion
- Timing of Optimization (Make It Work vs. Make It Fast):
- Debate on whether post-scale optimization is the ideal product development approach vs. whether basic memory-efficient designs should have been whiteboarded from day one.
- Commenters noted that while individual data layout tricks (like replacing vectors with boxed slices) appear simple in isolation, changing live data structures in hot production cache paths requires months of cautious, staggered rollouts.
- Premature Optimization vs. Architectural Incompetence:
- Multiple performance engineers cited Donald Knuth and industry experiences (e.g., Rico Mariani at Microsoft), arguing that designing within hardware reality from the start is good engineering, not "premature optimization."
- Neglecting memory efficiency early on can lead to brittle architectures and immense hardware sprawl before teams are forced to refactor.
- Critique of "Enterprise" Software Practices:
- The discussion touched on how modern enterprise software often masks poor resource efficiency with hardware scaling rather than optimizing low-level data structures.