09 · Hardware & Infrastructure
HBM
High Bandwidth Memory
High-bandwidth memory packaged close to an accelerator. It supplies model weights and activations quickly enough to keep compute units busy; capacity and bandwidth can be important constraints depending on the workload.
Concrete example
Two chips with identical compute can perform very differently if one is starved of memory bandwidth.
Why it matters
Memory bandwidth, not raw compute, is frequently the real constraint.