Decide exactly which AI crawlers can touch your site. Block or allow GPTBot, ClaudeBot, PerplexityBot and 25+ known AI bots, add sitemaps and path rules, and generate an llms.txt — all 100% in your browser.
llms.txt (llmstxt.org) is a Markdown map of your site for AI assistants — the inverse of blocking. Curate what AI should read so it cites you accurately.
QADIR OS agents honor robots.txt out of the box — and can audit, monitor, and update your site's crawler policy automatically.
Join the WaitlistEvery AI company runs crawlers that read the public web. Some collect training data for future models, some fetch pages live to answer user questions, and some index for AI search. robots.txt is the standard file — placed at https://yoursite.com/robots.txt — that tells each of them what they may access.
Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, meta-externalagent) collect content to train future models. Blocking them opts your content out of training datasets without affecting anything else.
Assistant fetchers (ChatGPT-User, Claude-User, Perplexity-User) retrieve a page live when a human asks about it — closer to a person clicking your link. Many sites keep these allowed for citation traffic.
AI search indexers (OAI-SearchBot, PerplexityBot, DuckAssistBot) build AI search results that can send visitors your way. Blocking them removes you from those results.
Where robots.txt says keep out, llms.txt says here's the good stuff. It's an emerging standard — a Markdown file at your site root listing your most important pages with short notes, so AI assistants working within a limited context window read the right things and cite you accurately. This tool generates both files from one place.
No. Googlebot and Bingbot are separate user-agents from the AI training crawlers — Google explicitly separated Google-Extended (Gemini training) from Googlebot (search) so you can block one without the other. This generator keeps search crawlers allowed unless you choose Fortress mode.
robots.txt is a voluntary protocol. Major AI companies publicly commit to honoring it, and the bot list here reflects publicly documented user-agents as of mid-2026 — but no file can physically stop a bad-faith scraper. For hard enforcement, pair this with WAF or bot-management rules at your CDN.