Most AI procurement runs on a demo and a reference call. That was defensible when the software was a feature; it is not defensible when the software reads your customer data, makes decisions on your behalf, and depends on four subprocessors you have never heard of. This AI vendor due diligence checklist is the set of questions worth asking before signature — organised so you can send them as an email and judge the vendor as much by how they answer as by what they answer.
1. Which model providers see our data, by name? Many AI products are a thin layer over two or three foundation-model APIs plus a vector database and an aggregator gateway. You are not onboarding one vendor, you are onboarding a chain. Ask for the full subprocessor list, and ask whether it can change without notice.
2. Is our data used for training, by the vendor or anyone downstream? The vendor's own answer is the easy half. The harder half is whether their upstream providers' enterprise terms carry the same guarantee, and whether that survives a plan downgrade or a provider swap.
3. What is the retention period for prompts, outputs and logs — and can we set it to zero? Prompt logs are the most under-governed data store in the industry. They contain whatever your staff pasted in, which in practice is everything.
4. Where is data processed geographically? If you have regional obligations, "our servers are in the EU" is insufficient when the model call itself leaves the region. Ask about the inference endpoint, not the application.
5. What is the measured task success rate on a workload like ours, and how was success defined? A definition written after the measurement is not a definition. The metrics worth insisting on are laid out in agent reliability metrics.
6. Show us the failure breakdown by class. Auth, rate-limit, timeout, provider 5xx, malformed output, refusal. A vendor who cannot produce this either does not instrument it or does not want to. Both are findings.
7. Can we reproduce your headline cost or performance claim ourselves, in one command, against real data? This is the single highest-yield question in the list. The method for pressing it is in how to verify AI cost savings claims.
8. What happens on a provider outage? Failover to another lane, degraded mode, or a full stop? Ask for the last incident and what the blast radius was.
Grade the answering behaviour, not just the answers. A vendor who says "we do not measure that yet, here is when we will" is more trustworthy than one who produces a confident number with no methodology attached. The first has a process. The second has a slide.
9. What exactly can we export, in what format, and how long does it take? Conversation history, agent configurations, evaluation results, embeddings, fine-tuned artifacts. "You can export your data" means nothing until you have seen the file. Ask for a sample export during the trial — not a description of one.
10. If you are acquired or shut down, what happens to our deployment? The honest answers range from a source-escrow arrangement to "you would have thirty days." Both are workable if you know which one you are buying. AI tooling is in a consolidation phase; a two-year contract routinely outlives the company that signed it.
11. Can this run on infrastructure we control? Self-hosted, VPC-deployed, or local. This is the exit question that answers itself — if the software runs on your hardware, vendor failure is an inconvenience rather than an outage. It is the entire premise behind local-first architecture, discussed in local AI vs. cloud AI.
12. What is the total cost at 3x our current volume? Per-seat pricing and per-call pricing diverge violently as usage grows, and agentic workloads grow faster than human ones because a loop makes many calls per human action. Model the bill at the volume you hope to have, not the one you have today — the arithmetic is in how much an AI agent costs.
First, run a real workload during the trial, not the vendor's demo dataset. Send your ugliest documents, your ambiguous tickets, the edge cases your team argues about. Demos are built from the happy path; your business is not.
Second, ask the vendor to name their worst number and what they are doing about it. This is a character test with a technical payload. We publish ours for exactly this reason: our routing telemetry shows a 19.9% overall call success rate across 1,469 logged calls, driven by 345 HTTP 403 responses from one provider and 756 failures that were never labelled with an error class — sitting in the same document as the finding that only 1.4% of traffic went to premium lanes and the window cost $0.14 against a $65.28 all-flagship counterfactual. A vendor willing to hand you the ugly half is a vendor whose good half was probably not curated either.
Is a SOC 2 report enough for AI vendor diligence? It is necessary and not sufficient. SOC 2 speaks to security controls and says nothing about model provenance, training-data use, prompt retention, failure behaviour, or what happens to your agents if the company disappears. Treat it as one input among twelve.
What if the vendor is a small startup with no formal answers? Small is not disqualifying — vagueness is. A two-person company that answers precisely about subprocessors, retention and exit is a lower risk than a well-funded one that deflects. Where the startup is genuinely early, ask for contractual commitments in place of documentation.
How much of this applies to self-hosted tools? Roughly half, and the half that matters most gets easier. Data residency, retention and exit largely resolve themselves when the software runs on your infrastructure. The performance, failure-behaviour and cost questions still apply in full.
Take the twelve to your next vendor call. If question 11 keeps coming back as a no, look at the local-first alternative — QADIR OS runs on hardware you own, so exit is a non-event. Join the early-access list.