Ask why a model feels slow and you'll get vibes. Ask what its tokens-per-second should be on given hardware and the room goes quiet. Yet LLM inference speed is one of the most predictable numbers in this entire field — a back-of-napkin division problem you can do before buying a GPU or signing a contract. Speed isn't a mystery; it's memory bandwidth divided by model size, with an asterisk or two. Once you can run that math, marketing benchmarks stop working on you.
This guide covers the two phases of inference, the bandwidth arithmetic, what actually moves the number (quantization, batching, speculative decoding), and the UX thresholds that decide whether your app feels instant or broken.
Every request has two acts. Prefill reads your whole prompt and computes attention over it in parallel — it's compute-bound, hammering the GPU's math units, and you experience it as time-to-first-token (TTFT). Decode then generates one token at a time — and each token requires reading essentially every weight of the model from memory. That makes decode memory-bandwidth-bound: the GPU's math units mostly sit idle, waiting on the memory bus.
This split explains behavior that otherwise looks random. Long prompt, short answer? Your cost is prefill; TTFT dominates. Short prompt, long answer? Decode dominates; tokens-per-second is everything. Optimizations target one phase or the other — prefix reuse and caching attack prefill (we've measured ~5× there — see prompt caching explained), while quantization and bandwidth attack decode.
Decode speed ≈ memory bandwidth ÷ bytes-per-token. Bytes-per-token is roughly the size of the model in memory (every weight gets read once per token, ignoring cache effects and overlap).
Worked example: a 7B-parameter model quantized to ~5 bits per weight occupies about 4.5 GB. On a consumer GPU with 1,000 GB/s of bandwidth: 1000 ÷ 4.5 ≈ 220 t/s theoretical ceiling; real-world lands at 50–70% of ceiling, so expect 110–160 t/s. Same model on a CPU with 60 GB/s of system RAM bandwidth: 60 ÷ 4.5 ≈ 13 t/s ceiling, maybe 7–9 real. A 70B model at 4-bit (~40 GB) on that same GPU — if it even fits — comes in around 1000 ÷ 40 = 25 t/s ceiling. Notice what happened: we predicted an order-of-magnitude UX difference with grade-school division. Run your own configuration through the free inference speed calculator, and check whether the model fits in memory at all with the VRAM calculator — fit comes before speed, because spillover to system RAM murders bandwidth.
Quantization is the honest speedup: fewer bits per weight = fewer bytes read per token = proportionally faster decode (plus it fits bigger models in the same VRAM). Going FP16 → 4-bit cuts bytes ~4× and typically buys 2–3× real throughput with modest quality cost at sane levels. Batching raises aggregate throughput — the same weight-read serves many concurrent sequences — which is why servers report huge t/s numbers; it does nothing for a single user's stream. Speculative decoding lets a small draft model propose tokens the big model verifies in bulk — real 1.5–3× wins on the single stream, workload-dependent. KV-cache growth is the silent killer: at long context, reading the attention cache competes with reading weights, which is why t/s degrades as conversations get long, and why context discipline (see the context window planner) is a speed decision, not just a cost one.
Human reading speed is ~5–7 words/second — call it 8–10 t/s. Below that, users watch text crawl and assume something is broken. At 20–30 t/s streaming feels fluid; past ~50 t/s extra speed is imperceptible for reading and only matters for agent chains, code generation, or bulk work where the consumer is a machine. TTFT has its own budget: under ~500 ms feels instant, past 2–3 s users start re-clicking. Engineer to these thresholds, not to leaderboard numbers — a model that's "slower" but streams immediately usually feels faster than one with better t/s and lazy TTFT.
API latency adds network round-trips and multi-tenant queueing you can't control or predict — the same endpoint is snappy at 3 a.m. and sluggish at peak. Local inference trades peak silicon for determinism: your t/s is your t/s, every hour of every day. That predictability is worth more than most benchmarks admit, and it's a core reason QADIR OS is local-first: a sovereign agentic OS that routes routine work to the fastest model your own hardware serves — with autocomplete-grade paths tuned to sub-30 ms medians on our own rig — and escalates to cloud only when a task genuinely needs the bigger brain. Speed you own doesn't rent-spike.
Predict before you buy. Run the division in the free inference speed calculator, check fit with the VRAM calculator, or join QADIR OS early access and get speed that's yours. No card required.