The Hidden Engine of Performance: It’s All About Where the Data Lives

1. The Real Reason Systems Are Fast: It’s All About Cache
2. Forget CPU Speed — Data Locality Wins Every Time
3. Why Cache Is the Hidden Engine of All High-Performance Systems
4. Performance Isn’t About GHz. It’s About Distance.
5. Your CPU Is Waiting on Memory: The Untold Story of Cache
6. Data Locality: The Most Important Thing in Computing Nobody Talks About
7. Where Your Data Lives Determines Everything
8. Cache Rules Everything Around Me: A Guide to Real Performance
9. The Memory Hierarchy: The Silent Killer (or Savior) of Performance
10. Why Modern Performance Is a Battle Against Latency, Not Compute
11. The Real Secret to Performance: Where Data is Stored Determines Everything
12. System Speed ​​Isn’t Determined by CPU, But by Data Distance
13. Caching is the Core of Modern Computing Performance
14. Ignoring Data Locality Makes All Performance Optimizations Futile
15. Performance Bottlenecks Aren’t in Computing Power, But in Memory
16. Data Locality: An Underestimated Performance Analyst
17. The CPU Is Waiting for Your Memory: The Real Cost of Caching

We love talking about CPU clock speeds, but in real systems the key question is: where does your data live?

Modern CPUs rely on a hierarchy (registers → L1 → L2 → L3 → DRAM). L1 might cost ~4 cycles; DRAM can take 200+. That’s 50× slower. If your working set fits in cache, everything flies. If not, the CPU stalls.

Why Cache Dominates Everything

Packet processing is a great example. Each packet triggers table lookups. If the tables stay hot in cache, you can push millions of packets per second. If they spill into DRAM, throughput collapses.

So the real design question: Will it fit in cache?

CPU Structure: Register, Cache

Instructions Matter Too

Caches aren’t just for data. Instruction-cache misses can ruin tail latency. Some HFT systems deliberately keep their hot loop firing constantly so the I-cache stays warm, only enabling the NIC when needed. A single I-cache stall in a trading loop can dominate the entire latency budget.

Where Abstraction Fails

High-level strategies like “cloud for everything” ignore low-level realities. Virtualized network functions depend on things like:

• exclusive core pinning (so cache stays warm)
• interrupt coalescing trade-offs
• NUMA locality
• physical vs virtual NIC behavior

Sales decks say it “works,” but the fine print is usually: needs 3× hardware and still won’t match bare metal. Once you depend on cache behavior, pinning, and locality, the platform is no longer interchangeable.

AI Hits the Same Wall

Bigger models don’t change physics. Data movement still dominates compute. Locality wins.

• Arrays beat pointer-heavy structures because they’re contiguous
• Prefetchers work only with predictable patterns
• Cache lines get used efficiently when memory layout is sane

Even Robotics Shows This

In multi-axis motion control, the first axis warms the cache and pays the miss cost; later axes compute in half the time. Same principle: locality = speed.

IBM Telum: Caches on a Different Scale

The IBM Telum processor takes this idea to an extreme:

• Ten 36 MB L2 caches
• 360 MB virtual L3
• 2.8 GB virtual L4

IBM Telum Processor

The architecture can even convert L2 into L3 on demand. IBM hasn’t published access latencies, but it would be fascinating to see how they balance size vs distance vs hit time.

Conclusion

Performance is ultimately about how close your data and instructions stay to the core.

Design for locality and your systems will sing. Ignore it and no GHz number or cloud abstraction will save you.

We often talk about CPU speed, but rarely about where the data lives.

Performance is dominated by the proximity of the data. Registers, L1, L2, L3, main memory – each step adds latency and drops throughput. A main memory access can take 200 cycles, 50x slower than L1 cache.

When your working set fits in cache, your code flies. When it doesn’t, the CPU just waits.

In packet processing, this difference decides everything. Each packet triggers table lookups. If those tables stay in cache, you can push millions of packets per second. If not, throughput collapses.

So the next time you design a data structure, ask:

Will it fit in cache?

Because in performance-critical systems, cache isn’t just an optimization – it defines the system.

And not only data but also instructions! I’ve seen HFT engineers talk about their strategies where they programmed the hot paths to be firing all the time, and only switching the network card on when a packet needs to leave the system. This keeps their instruction cache hot as well.

Keeping the instruction cache hot is just as critical as keeping the data hot, especially in workloads where predictability matters. Shaping your hot path so the CPU never falls out of the I-cache is important, because even a small stall can dominate tail latency. It is an excellent reminder that architecture design is really about keeping both instructions and data as close to the core as possible.

So many technical decision makers are set on blanket strategy: eg. cloud for everything – that they think any virtualized workload can work in any virtualization environment and the underlying hardware and the virtualization is just a commodity. This doesn’t apply for virtualized network functions where vendors knew exclusive thread core pinning gives the execution threads exclusive use of CPU cache. The vendors knew interrupt coalescing in virtualized environments decreases “CPU usage” at the expense of latency. They knew about NUMA locality and even put it all in their docs. Of course when the sales guys come around, they want to be aligned with high level strategy and use the best optimized benchmarks and have another separate discussion about cloud or hypervisor support without nuance. Yeah that will work*fine print: you’ll need 3x the licenses/hardware and still won’t get optimal performance>. There is such a lack of interest in low level performance and such a skills gap where it seems to be addressed by adding layers of abstraction and vendors to obscure accountability. If Everest was the test of tech leadership or vendor accountability, it would be nice to know which would die on the hill or sell parkas at the bottom. Totally. Once you rely on cache behavior, core pinning, and NUMA locality, the platform stops being interchangeable. The low level details matter far more than most high level strategies

most of the heavy AI workloads still run straight into the same memory hierarchy limits. The models keep getting bigger, but the physics of moving data around the chip haven’t changed much. Understanding locality is still a big part of getting good performance.

Arrays give the CPU exactly what it wants: contiguous memory and predictable access patterns. That means the prefetcher can actually do its job, the cache lines get used efficiently, and you avoid the pointer-chasing penalties you get with scattered structures. It is one of the simplest ways to stay cache-friendly.

The same in multi axes motion control for robotics. first axis warm up cache and takes the impact of cache miss, calculation of the next axis takes half time.

The IBM Telum processor can confirm that, with L2 converted to L3 on demand and L4 cache, accessible from any other cpu. Plus the clock speed is always at 5.5 GHz. The chip includes ten 36 MB Level-2 caches1 and expanded virtual Level-3 (360 MB) and Level-4 (2.8 GB) caches

That’s a fascinating chip. The cache sizes are enormous compared to most architectures, and it makes me wonder how that affects access latency at each level. I couldn’t find any published latencies for Telum’s caches, which is a pity, because it would be interesting to see how IBM balances size, fabric distance, and hit latency in practice.

See: Cache is the King!

–EOF (The Ultimate Computing & Technology Blog) —