If you're calling the Claude or GPT API one request at a time for tasks that don't need real-time responses — data labeling, content generation, document summarization, lead scoring — you're paying double what you should. Both Anthropic and OpenAI offer batch processing APIs that let you submit thousands of requests as a single job, get results within 24 hours, and pay 50% less per token. Here's how to use them.
Batch is the right choice whenever you don't need the response immediately. That covers more use cases than most people realize:
Data processing: Categorize 10,000 support tickets. Extract entities from 5,000 contracts. Score 50,000 leads based on their company descriptions. These are offline analytics tasks — you don't need the result in 200ms, you need it done accurately and cheaply by morning.
Content generation: Generate product descriptions for 2,000 SKUs. Write meta descriptions for 500 blog posts. Create personalized email variations for 20 segments. The content doesn't go live until you review it anyway, so real-time generation is wasted speed.
Document analysis: Summarize 1,000 PDFs. Extract key terms from 3,000 research papers. Compare 200 contracts against a compliance checklist. Heavy reading work that an LLM handles well but that accumulates massive costs at standard API pricing.
The rule of thumb: if you'd be happy getting results in 1–24 hours instead of 1–5 seconds, batch saves you 50%.
Anthropic's Message Batches API accepts a JSONL file where each line is a standard Messages API request with a custom_id for tracking. You submit the batch, Anthropic processes it within 24 hours (typically faster), and you retrieve the results as another JSONL file with the same custom_ids mapped to responses.
Key details: batches can contain up to 100,000 requests. Each request uses the same model and API parameters you'd use for a real-time call — same system prompts, same temperature, same max tokens. The only difference is latency and price. You get the same Claude output quality at 50% of the token cost.
Error handling is per-request: if request #4,372 fails (malformed input, context too long), the other 99,999 still process. You get a results file with successes and failures clearly labeled. No all-or-nothing failure mode.
Same concept, slightly different implementation. You upload a JSONL file of chat completion requests, create a batch job referencing the file, and poll for completion. OpenAI's batch window is also 24 hours, pricing is 50% off standard rates, and the maximum batch size is 50,000 requests or 200MB, whichever comes first.
One difference: OpenAI batches support the /v1/embeddings endpoint too, so you can batch-generate embeddings for large document sets at half the cost. If you're building a RAG system and need to embed 100,000 document chunks, batch embeddings alone can save hundreds of dollars.
If you're running models locally on your own GPU, batch processing means feeding multiple prompts in parallel to maximize GPU utilization. vLLM's continuous batching keeps the GPU saturated — as one request finishes, the next one starts processing without waiting for the entire batch to complete. This gives you 3–5x higher throughput than sequential processing.
Ollama doesn't have native batch support, but you can achieve similar results with a simple queue: spin up N parallel requests (where N depends on your VRAM and model size), feed them from a task queue, and collect results. For a self-hosted setup, this is the fastest path to high-throughput processing without API costs.
Regardless of provider, the pattern is the same: (1) Prepare your input as a JSONL file — one JSON object per line, each containing the prompt and any task-specific parameters. (2) Submit the batch to the API. (3) Poll for completion or set up a webhook. (4) Download results and match them back to your original data using custom_id. (5) Handle failures — retry individual failed requests, don't resubmit the entire batch.
The hardest part isn't the API call — it's structuring your prompts for consistency at scale. When you're processing 10,000 items, a prompt that works 95% of the time fails 500 times. Test your prompt on 100 representative items before running the full batch. Invest in structured output (JSON mode) so parsing is deterministic.
Processing 10,000 documents with Claude Sonnet (1,000 input tokens + 500 output tokens each): real-time API costs roughly $45. Batch API costs roughly $22.50. At 100,000 documents: $450 vs. $225. The savings scale linearly. For businesses doing regular bulk processing — monthly report generation, weekly data labeling, daily content updates — batch processing cuts your AI API bill in half permanently.
Compare that to local processing: zero API cost, but you need the hardware. A single RTX 5080 running a 13B model processes roughly 2,000–5,000 documents per hour depending on document length. For 10,000 documents, that's 2–5 hours of GPU time. The 5080 costs $1,000 — it pays for itself in 4–5 batch runs versus cloud API pricing.
QADIR OS uses batch processing for its content and analytics engines. Weekly blog posts, social media content, SEO analysis, lead scoring — all batch-processed overnight. The agent workflow system queues batch-eligible tasks automatically: if a task doesn't need real-time output, it routes to the batch queue. You wake up to results instead of watching a progress bar.
QADIR OS routes tasks to batch or real-time automatically — half the API cost, same results. Try 164+ free AI tools, or join early access — no card required.