Right now, whether you decided anything or not, your site has an AI training policy. If your robots.txt doesn't mention GPTBot, ClaudeBot, or CCBot, your policy is "take everything." Maybe that's what you want — AI answer engines are a real traffic source now. Maybe it isn't. Either way it should be a choice, and a robots.txt generator is the fastest way to make that choice explicit: pick which crawlers get in, which stay out, and ship a file that doesn't accidentally torch your Google rankings in the process.
This guide covers what robots.txt actually enforces, the AI crawler roster in 2026, the block-or-allow tradeoff nobody gives you a straight answer on, and the syntax traps that break real sites.
Robots.txt is a plain text file at your domain root that tells crawlers which paths they may fetch. It's etiquette, not enforcement: compliant bots (Googlebot, Bingbot, GPTBot, ClaudeBot) honor it, and scrapers that don't identify themselves ignore it entirely. Putting /admin in robots.txt doesn't protect it — it advertises it. Access control is authentication's job; robots.txt just manages the polite crowd, which happens to include every crawler that matters for search and AI visibility.
One more distinction that trips people: robots.txt controls crawling, not indexing. A blocked URL can still appear in Google if other sites link to it — as a bare link with no snippet. If you need a page out of the index, that's a noindex meta tag, and the page must be crawlable for Google to see the tag. Blocking and noindexing the same page cancels itself out. (For the head-tag side of this, see the meta tag guide.)
The user-agents worth knowing, grouped by what they feed:
Training crawlers — fetch content to train models: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google's AI training switch — separate from Googlebot, blocking it does not affect Search), CCBot (Common Crawl, the dataset half the industry trains on), Bytespider (ByteDance), meta-externalagent (Meta), Applebot-Extended (Apple AI training).
Answer-time fetchers — retrieve pages live to answer a user's question and often cite the source: OAI-SearchBot and ChatGPT-User (OpenAI), Claude-SearchBot and Claude-User (Anthropic), PerplexityBot and Perplexity-User. Blocking these removes you from AI answers and their citation links.
The roster shifts every few months — new agents appear, names change. That's why we keep the list maintained inside the free AI robots.txt generator: 25+ AI crawlers, each with a toggle, grouped by training versus answer-time, on top of the classic search bot rules.
The honest version, without the outrage or the hype:
The case for blocking training bots: your content is your asset. Training crawlers take it wholesale, compensation is zero, and attribution inside a trained model is nonexistent. Publishers with paywalled or differentiated content increasingly block GPTBot, CCBot, and Bytespider on principle and lose nothing measurable.
The case for allowing answer-time bots: AI assistants are becoming a discovery channel. When Perplexity or ChatGPT cites your page, that's a referral path that didn't exist three years ago. Blocking answer-time fetchers is opting out of that channel — defensible, but know that's what you're doing.
The pragmatic middle, and the most common configuration we see: block bulk training crawlers, allow answer-time fetchers and citation bots, always allow classic search. You keep Google, keep AI citations, and stop feeding training pipelines for free. Any of the three positions is legitimate — the failure mode is not choosing.
Robots.txt looks trivial and misfires constantly. The classics: the stray blanket disallow — a Disallow: / left under User-agent: * from staging is still the most efficient way to deindex a business. Group scoping — rules apply only to the user-agent block above them; a crawler matches its most specific block and ignores the rest, so your * rules do NOT cascade into the GPTBot block. Case sensitivity — paths are case-sensitive; /Admin and /admin are different rules. Crawl-delay — ignored by Google entirely. And put a Sitemap: line in the file; it's the one place every crawler reliably looks for it.
Validate after deploying: fetch yourdomain.com/robots.txt, confirm a 200 and the content you expect — a misconfigured redirect can serve your www and apex domains different files. (A status code reference helps when the response isn't what you expected.)
Robots.txt is a small lever, but it's the same principle we build on: you decide what leaves your infrastructure. QADIR OS applies that to the other side of the equation — AI agents that run on your machine, with your data staying local, instead of your workflows living on someone else's metered cloud. Deciding who reads your content and deciding where your agents run are the same decision at different scales.
Set your AI crawler policy in 60 seconds. The free AI robots.txt generator covers GPTBot, ClaudeBot, and 25+ crawlers with per-bot toggles. Then fix your meta tags — or join early access to run sovereign agents of your own. No card required.