A modern CPU can execute an instruction in 1 clock cycle.
Fetching data from physical RAM takes about 200 clock cycles because the signal has to leave the CPU chip, travel across the motherboard via the System Bus, and find the data in the RAM sticks.
If the CPU always read from RAM, it would spend 99% of its time frozen.
2. The Cache Hierarchy
To fix this, we glue tiny amounts of ultra-fast memory directly onto the CPU chip.
L1 Cache: Extremely tiny (e.g., 32KB), but blazing fast (1 cycle cost). Private to each core.
L3 Cache: Large (e.g., 32MB), slower (40 cycles). Shared among all cores.
3. Hits, Misses, and AMAT
The OS constantly tries to guess what data you need next and moves it into L1.
Cache Hit: The data is found! (Fast).
Cache Miss: The data isn't there. The CPU stalls and searches the next level down.
AMAT (Average Memory Access Time): The ultimate metric. By having a 90% Hit Rate in L1, the *average* cost to fetch data drops from 200 cycles down to roughly 2 or 3 cycles!