Q4Computer Architecture

Cache memory is used because there is a very large and continually widening speed gap between the CPU's processing speed and the access speed of main memory (DRAM) — if every single memory access made by the CPU had to wait for the full main memory access latency, the processor would spend the vast majority of its time idle, waiting for data, drastically reducing overall system performance regardless of how fast the CPU's internal execution units themselves might be.

Cache memory addresses this problem by exploiting the principle of locality of reference — the well-established empirical observation that programs tend to access a relatively small subset of their total memory addresses repeatedly within short time intervals (temporal locality, e.g., a loop repeatedly accessing the same variables) and tend to access memory addresses that are close to recently-accessed addresses (spatial locality, e.g., sequentially accessing consecutive array elements or executing sequential instruction addresses). By placing a small amount of very fast memory (cache), holding copies of the most recently and frequently accessed main-memory data, physically close to (or directly within) the CPU, the vast majority of actual memory accesses can be satisfied directly from this fast cache (a 'cache hit') rather than requiring a slow main-memory access (a 'cache miss'), dramatically reducing the average memory access time experienced by the CPU across the overall running program.

Modern processors typically implement multiple cache levels (L1, L2, and often L3), with L1 cache being the smallest, fastest, and closest to the CPU core (often split into separate instruction and data caches), and each successive level (L2, L3) being progressively larger but somewhat slower, forming a graduated hierarchy that balances the competing goals of very high speed (favoring small cache size) and high hit rate (favoring larger cache size) — this cache hierarchy is itself a smaller-scale instance of the broader memory hierarchy concept discussed elsewhere in this paper, applied specifically to bridge the CPU-to-main-memory speed gap.