04 · How Models Run
Latency / Throughput
Two competing speed metrics
Latency is the delay for a single request (matters to one waiting user). Throughput is total requests handled per unit time (matters to a company serving millions). Optimizing one often hurts the other — a classic tradeoff.
Concrete example
Batching many users together raises throughput but can make each one wait a little longer.
Why it matters
The core tension in serving AI cheaply versus serving it snappily.