LLM Inference Speed Calculator

Pick a model, a quantization, and a context length. See how many tokens per second each GPU can actually generate — from a single RTX 4090 to the dual RTX 5090 ABUZ8 box and an H100 — using the memory-bandwidth roofline that governs single-stream decode.

🔒 100% in-browser — nothing you select is uploaded

1. The model you want to run

Real parameter counts and attention shapes (layers, KV dim, and active params for MoE) for accurate decode math.
Billions of weights (total).
= total for dense models; smaller for MoE.
Transformer blocks.
kv_heads × head_dim (often 1024).

2. Quantization, context & efficiency

Bits per weight. Q4_K_M is the popular default.
The cache is re-read for every generated token.
Tokens already in the window while generating.
How much of the GPU's rated bandwidth the runtime actually reaches on single-stream decode. Real-world llama.cpp / vLLM land around 55–85%. Lower it for a conservative estimate.
Decode speed (primary GPU)
0 tok/s
on the selected GPU
Reads per token
0 GB
weights + KV cache
Words per second
0
≈ tok/s × 0.75
GPU Bandwidth Decode tok/s Feel vs reading

Theoretical roofline, not a guarantee. Single-stream decode is memory-bandwidth bound: each new token reads the active weights (params × bytes-per-weight, set by the quant) plus the KV cache (2 × layers × kv-dim × context × KV-precision) once. So tok/s ≈ MBU × GPU bandwidth ÷ bytes-per-token. Real throughput varies with the backend (llama.cpp / vLLM / ExLlama), flash-attention, batching, and CPU offload — batched serving can far exceed these single-stream numbers. MoE models read only their active params per token. The dual-GPU ABUZ8 row assumes tensor-parallel scaling (~1.7× one card) and is approximate. Prefill (prompt processing) is compute-bound and is not modeled here — it sets time-to-first-token, not the steady typing speed shown above. A human reads ~5–8 tok/s.

Own the floor. Rent only the ceiling.

Speed is step two; the VRAM has to fit first. QADIR OS is the sovereign layer on top of your own hardware: it loads the largest model your GPU can serve at the speed you just sized, and routes only the calls it can't clear to a cloud API — Opus-grade output, Haiku-grade bill, your data on your machine.

✓ You're on the list. We'll be in touch.
→ First check it fits: Local LLM VRAM Calculator

How fast a local LLM actually generates

What sets the tokens-per-second I'll see?

For a single request, generation speed is set by memory bandwidth, not raw compute. To produce each token the GPU streams the model's weights (and the KV cache) out of VRAM once, so tokens per second is roughly your card's bandwidth divided by the bytes read per token. Halve the model size with a lower quant, or move to a card with more bandwidth, and the number goes up almost proportionally.

Why is the first token slower than the rest?

Generating tokens (decode) is memory-bound and one-at-a-time. Processing your prompt (prefill) runs all those tokens in parallel and is compute-bound — it's the part that sets time-to-first-token. A long prompt can make you wait for the first token even on a fast card, then the steady typing speed shown above takes over.

Why does long context slow it down?

Each new token must read the whole KV cache, and that cache grows linearly with context. At 128k it can rival the weights themselves, so the bytes-per-token climb and tokens-per-second fall. Storing the cache in 8-bit (the KV-precision option above) halves that part. This is the same KV-cache growth that drives the VRAM calculator.

Will batching beat these numbers?

Yes. These are single-stream figures — one conversation at a time, which is what you feel locally. A server batching many requests amortizes each weight read across all of them and reaches far higher aggregate tokens per second. If you're sizing a one-user desktop, single-stream is the honest number; if you're serving traffic, treat these as a per-request floor.

Is my data sent anywhere?

No. This calculator runs entirely in your browser. Nothing you select or type is uploaded, logged, or stored.