The AI proof of concept works. Everyone is impressed. Six months later it is still a proof of concept, and nobody can quite explain why. Moving an AI proof of concept to production fails so predictably because a POC is designed to answer one question — is this possible? — and production asks six entirely different ones that the POC was never built to touch.
That the model can do the task on inputs someone chose, once, with a human present to interpret the output. That is a real and useful finding. It is not evidence about behaviour on unchosen inputs, at volume, unattended, when a provider is down, when the API changes, or when the person who built it goes on holiday. Confusing the first for the second is the entire failure mode, and it happens because the demo is emotionally convincing in a way that a p95 latency chart is not.
1. The unhappy path. POC inputs are curated by definition — the builder picked documents that parse, tickets that are legible, questions that are answerable. Production sends the scanned fax, the two-language email, the ticket that is three unrelated complaints in one paragraph. Estimate the share of real traffic that is messy, then check whether the POC ever saw any of it. Usually the answer is none.
2. Cost at real volume. A POC runs a few hundred calls and costs less than lunch, so nobody models it. Agentic workloads are the ones where this bites hardest, because a loop makes many model calls per human action — the multiplier between "a user did something" and "we made an API call" is often ten or more. Do the arithmetic at target volume before committing, using the method in how much an AI agent costs.
3. Failure behaviour. During a POC, when something breaks, a human notices and reruns it. In production nobody is watching, so the question is what the system does on its own — retry, fail over, escalate, or silently produce nothing. The seven common collapse modes are catalogued in why AI agents fail in production.
4. Measurement. POC success is a room agreeing the output looked good. Production success is a defined metric, computed automatically, trending over time. If you cannot state what percentage of tasks succeeded last week without asking someone, you do not have a production system — you have a demo running continuously. The metric set is in agent reliability metrics.
5. Change. Models get deprecated, prices change, tool APIs shift shape, prompts that worked degrade after a provider updates a checkpoint. A POC is a photograph; production is a film. Someone has to own the ongoing work, and if no name is attached to that in the plan, the system decays quietly from its first week.
6. Blast radius. A POC reads. Production often writes — sends the email, updates the record, moves the money. The moment an agent takes actions with consequences, the reliability bar rises sharply and the approval architecture becomes part of the product rather than a wrapper around it.
Write the exit criteria before the POC starts. "We proceed if it achieves X% on Y task class over Z real inputs, at a modelled cost under $N per completed task." Written afterwards, the criteria will be quietly shaped by the result — not through dishonesty, just through the ordinary human tendency to grade against what happened.
Run the POC on real inputs from the beginning, including the ugly ones. It will look worse. That is the point — a POC that only handles clean data has deferred the hard finding to the expensive phase.
Then instrument before you scale, not after. The columns you did not log during the pilot are questions you cannot answer when someone asks why the bill tripled; the minimum viable schema is in the routing telemetry guide. Then ship narrow: one task class, real users, human approval on anything with side effects, and remove the approval only after the measured success rate earns it.
Ship narrow rather than broad. One task class fully in production teaches you more than five in pilot, because only the production one encounters the whole distribution of reality.
Where the system runs turns out to shape almost everything above. On metered APIs, every unhappy-path retry and every extra evaluation run has a price, which quietly discourages exactly the testing that makes the system trustworthy. On hardware you own, the marginal call costs electricity — so you can afford to run the full ugly dataset nightly, replay a month of traffic against a new prompt, and grade a sample every week without a budget conversation.
That is the design premise behind QADIR OS: local-first inference across lanes you control, with commercial models available when a task genuinely needs one, and every call written to a local database so the cost and reliability questions are queries rather than estimates. Our own routing log holds 1,469 calls across 89.9 days, with 1.4% of traffic reaching premium lanes at a total spend of $0.14 — and it also holds a 19.9% success rate defect we can name precisely because the telemetry exists. Both halves came from instrumenting first. The trade-offs of owning the compute are in local AI vs. cloud AI.
How long should an AI POC run? Long enough to hit real variance — typically four to six weeks on genuine inputs. Shorter than that and you have measured a good week; much longer without a decision usually means the exit criteria were never written down.
What is the single most common reason a pilot stalls? No owner for the production phase. The pilot is exciting and staffed; the hardening, instrumentation and change management that follow are neither, so the work sits unassigned while everyone waits for a decision nobody is responsible for making.
Should we build or buy at this stage? Decide it on the exit-criteria data rather than in advance. Once you know your real success rate, real cost per completed task and real failure distribution, the build-or-buy question usually answers itself — and you can hold any vendor to the same numbers using the vendor diligence checklist.
Write your exit criteria this week — one sentence naming the task class, the success threshold, the input set and the cost ceiling. Then instrument before you scale. If you want a runtime that logs it all locally by default, join the QADIR OS early-access list.