Every major LLM provider sells the same quiet deal: send your work as a batch instead of one request at a time, accept that results arrive within hours instead of seconds, and pay roughly half price. That's the whole pitch of the LLM batch API — a flat ~50% discount on both input and output tokens for anything asynchronous. The strange part isn't the discount. The strange part is how many teams run enormous offline jobs — classification backlogs, embedding pipelines, nightly report generation — through real-time endpoints at full price, paying a premium for a latency guarantee nobody asked for.
This guide covers how batch endpoints work, which workloads qualify, the real math including failure handling, and when batch is the wrong call.
Instead of one HTTPS call per prompt, you upload a file of requests — typically JSONL, one request per line, each with a custom ID — and the provider processes them on spare capacity. You poll (or get a webhook) until the job completes, then download a results file keyed by your IDs. The standard completion window is 24 hours; in practice most jobs land far sooner, but you design for the guarantee, not the average.
Why does slack in the deadline cut the price in half? Because peak GPU capacity is the provider's scarcest resource. Real-time traffic forces them to keep hardware idle for demand spikes. Batch jobs fill the valleys. You're not getting charity — you're selling back flexibility they can schedule around. It's off-peak electricity pricing, for tokens.
The test is one question: does a human wait on this response? If nobody is staring at a spinner, the work is a batch candidate. That covers more than most teams think: bulk classification and tagging of historical data, content moderation backlogs, embedding generation for search indexes, evaluation suites and regression tests on prompts, synthetic training data, enrichment of CRM records, summarizing yesterday's tickets, and scheduled reports. If your prompts also share a long common prefix, stack the discounts — prompt caching and batch pricing compound on some platforms, and a shared-prefix batch is exactly the shape caches love.
Take a nightly job: 200,000 support tickets to classify, averaging 1,500 input tokens and 50 output tokens each. At $3/M input and $15/M output, real-time cost is roughly $900 input + $150 output = $1,050 per run. Batched at 50% off: about $525. Run it nightly and that's ~$190k a year saved for changing an endpoint and a file format — no model change, no quality change, identical outputs.
Two costs the brochure skips. First, engineering time: batch jobs need assembly, submission, polling, retry logic for the handful of requests that fail inside an otherwise-successful job, and re-merge logic on your side. That's real work the first week and near-zero after. Second, iteration speed: a 24-hour worst case means you don't debug prompts through the batch lane. Prototype in real time on a small sample, then batch the volume. Before committing, put your own numbers into the free batch API savings calculator — and sanity-check provider rates side by side with the LLM price comparison tool.
Don't batch anything conversational, anything gating a user action, or anything whose value decays in minutes — fraud checks, live routing, chat. Don't batch tiny volumes either: if the job costs less than the coffee you drink while writing the JSONL, the discount doesn't pay for the plumbing. And watch queue-position risk around deadlines — if results absolutely must exist by 9 a.m., submit with margin against the full completion window, not against the average you observed last week.
Batch pricing is one instance of a general law: urgency is the most expensive ingredient in AI. Every relaxation you can tolerate — looser latency, smaller models on easy rows, cached prefixes — converts directly into margin. Cost-aware routing applies the same law per-request: send routine work to the cheapest model that's still correct, escalate only when needed. Do the arithmetic on your own pipeline with the embedding cost calculator if vectors are your volume driver.
Follow the logic one step further. If a workload tolerates hours of latency, it also tolerates running overnight on your own GPU — where the marginal price per token is zero, forever, and no completion window is someone else's promise. That's the thesis behind QADIR OS: a sovereign, local-first agentic OS where the batch lane is just... your machine, running while you sleep. Its router treats cloud APIs as the escalation path, not the default. The cheapest batch discount is the meter that doesn't exist.
Half price is good. No meter is better. Run your numbers in the free batch savings calculator, then join QADIR OS early access — agents that do the overnight work on your own hardware. No card required.