A new species of visitor is reading your website, and your robots.txt — the fifty-year-old-feeling text file most businesses haven’t opened since launch — is where you decide how to receive it. GPTBot wants your pages for OpenAI’s training corpus. ChatGPT-User fetches them live when a user’s question needs your content. PerplexityBot indexes for an answer engine that cites its sources. Google-Extended governs whether your content feeds Gemini’s training — separately from the Googlebot crawl your rankings depend on. ClaudeBot, Bingbot’s AI appetites, Meta’s crawlers, a lengthening tail of research scrapers — each announcing (or not) its user agent, each respecting (or not) your directives, and each representing a genuinely different trade: some send you cited visibility, some take training value and send nothing back, and some are the same company wearing different hats for different purposes.
The public conversation about this is loud and mostly unhelpful for a small business — publishers with licensing lawsuits and paywalled archives face a real content-economics question; a Dallas service company whose content exists to make the phone ring faces a much simpler one: which crawlers feed the answer engines your prospects use, and which merely consume? Get that distinction wrong in the blocking direction and you’ve quietly removed yourself from the recommendation surfaces you’re elsewhere paying to win — the “we blocked all the bots” version of the staging-noindex disaster. Get it wrong in the permissive direction and… honestly, for most service businesses, not much happens — which is itself a load-bearing fact this guide will be honest about.
What follows is the practical decision: how AI crawlers actually differ (training vs. retrieval vs. hybrid — the taxonomy that makes the decision tractable), the current major user agents and what each one’s access actually buys or costs you, the robots.txt mechanics and their honest limits (voluntary compliance, the well-behaved-bots-only problem), the recommended default policy for a lead-generation business and the situations that justify deviating from it, the implementation with verification, and the maintenance reality — this landscape changes quarterly, and a policy set in 2024 is already an artifact.
AI crawlers split into three types, and the type — not the company — drives the decision: retrieval/search crawlers (fetch pages to ground live, usually cited answers — PerplexityBot, ChatGPT’s user-triggered fetching, Bing’s AI surfaces via Bingbot) are the visibility channel: blocking them removes you from answers your prospects see; training crawlers (harvest for model corpora — GPTBot, ClaudeBot, Meta’s, and Google-Extended as the Gemini-training control) offer diffuse, unattributed value in exchange for your content — a values call with modest practical stakes at service-business scale; infrastructure crawlers you must not touch — Googlebot and Bingbot proper, where blocking kills your search presence (and note Google’s AI Overviews ride the normal Googlebot crawl: there is no separate opt-out that keeps rankings). Recommended default for a lead-gen business: allow retrieval, allow the search infrastructure, decide training on principle — blocking training bots costs you little visibility; blocking retrieval bots costs you answers. Mechanics: per-agent User-agent: blocks in robots.txt (most AI companies publish their agents and honor the file), knowing the honest limits — compliance is voluntary, bad actors ignore it, and robots.txt is a request, not a wall. Verify with log checks and the mention audit; revisit the agent list quarterly — it changes.
The Taxonomy That Makes the Decision Tractable
- Retrieval (search/grounding) crawlers fetch your pages to answer live questions — the results are typically cited, linked, and current, and the visitor they represent is a prospect mid-question. This is the channel the whole citation strategy feeds: block it and your carefully built resources can’t be retrieved, your business can’t be grounded in answers, and the recommendation surfaces synthesize your category from competitors’ pages instead.
- Training crawlers harvest content for model corpora — value to you is diffuse (the model “knows about” your content class, unattributed, frozen at training time), and the trade is your content for essentially nothing direct. Reasonable businesses land on both sides of this one; the point is that it’s a separate decision from retrieval, and the platforms increasingly provide separate controls precisely so you can split it.
- Hybrid and infrastructure agents complicate the clean split: Googlebot’s single crawl feeds classic search and AI Overviews (no separating them without leaving search entirely; Google-Extended controls only Gemini training, not Overviews); Bingbot similarly underwrites both Bing search and Copilot’s grounding; and some companies’ agents serve multiple documented purposes. For these, the rule is simple: anything your search presence rides on is untouchable, and the rest gets decided by its documented purpose.
The Current Major Agents — and What Each Decision Buys
| User agent | Type / purpose | The practical trade for a service business |
|---|---|---|
Googlebot | Infrastructure — search index, which also feeds AI Overviews | Never block. Your rankings, your Overview citations, your existence in Google — one crawl |
Google-Extended | Training control — opting content out of Gemini model training; does not affect search or Overviews | The clean values-call token: block it and lose nothing visible; allow it and contribute to training — genuinely your choice |
Bingbot | Infrastructure — Bing search + Copilot grounding | Don’t block — Bing’s index underwrites more AI surfaces than its search share suggests |
GPTBot | Training — OpenAI’s corpus crawler | The values call, OpenAI edition — blocking it does not remove you from ChatGPT’s live search answers |
ChatGPT-User / OpenAI’s search agents | Retrieval — user-triggered fetching and search grounding for ChatGPT’s cited answers | Allow — this is ChatGPT sending your content to a prospect with attribution; blocking it is blocking the storefront |
PerplexityBot | Retrieval — the citation-forward answer engine’s index | Allow — Perplexity’s entire model is cited sources; exclusion is self-removal from its answers |
ClaudeBot / Anthropic’s agents | Primarily training; retrieval agents documented separately | Split per the published docs — same logic: training by principle, retrieval for visibility |
| Meta’s crawlers, Apple’s, ByteDance’s, the research bots (CCBot etc.) | Mostly training/corpus | The values call again — with CCBot the notable multiplier (Common Crawl feeds many downstream models and datasets) |
The list above decays. Agents get added, renamed, and re-purposed — every major AI company documents its current crawlers and their purposes; the quarterly maintenance check (below) is reading those pages, not memorizing this table.
Before any Disallow ships, ask: does my search presence or my AI-answer visibility ride on this agent? The two catastrophic mistakes in this domain are both blocking-direction: the business that blocked Googlebot-adjacent infrastructure chasing an AI opt-out (and fell out of search), and the business that blocked the retrieval agents in a blanket ‘no AI bots’ sweep (and vanished from the answer engines while competitors’ content grounded every response). Training-crawler blocks are reversible shrugs; retrieval and infrastructure blocks are self-inflicted invisibility. When unsure which kind an agent is: allow it while you check the company’s crawler documentation — the asymmetry of the two error types makes permissive-while-verifying the safe posture for a business that wants to be found.
Robots.txt Mechanics — and Their Honest Limits
- Per-agent blocks are simple: a
User-agent: GPTBotgroup withDisallow: /excludes that agent site-wide; path-scoped disallows work per agent exactly as they do for Googlebot; and an agent with no matching group falls through to yourUser-agent: *rules — which is why a legacy blanketDisallowunder*can be silently excluding every AI crawler you never decided about. Audit the file you actually have before writing policy. - Compliance is voluntary: robots.txt is a convention, not an enforcement layer — the major AI companies publicly commit to honoring it (reputationally load-bearing for them), while scrapers, unannounced agents, and bad actors ignore it freely. The file governs the well-behaved; rate-limiting, WAF rules, and bot management at the CDN layer govern the rest — different tools for a different problem, and conflating them (“robots.txt doesn’t stop scrapers so why bother”) misses that the well-behaved cohort includes precisely the answer engines whose behavior you want to shape.
- Adjacent controls exist and evolve: meta-level and header-level signals for AI usage, emerging standards for expressing training preferences, and platform-specific opt-out registries — the ecosystem is actively building finer instruments than the blunt robots.txt block. Policy note in your file (a comment line with the date and intent) plus a calendar entry beats chasing every proposal in real time.
- Don’t confuse crawl control with index control: the same distinction as ever — robots.txt governs fetching; what appears where is governed downstream. And a blocked page can still be mentioned from other sources — blocking retrieval doesn’t erase you from answers; it removes your voice from how you’re described in them.
The Recommended Default — and When to Deviate
The default for a lead-generation service business: allow all search infrastructure (non-negotiable), allow all documented retrieval/answer-engine agents (this is the AEO channel — the same strategy as being in Google), and make the training decision consciously — where the honest note is that for most service businesses either answer is defensible: allowing training costs you content whose competitive value is mostly local and relational anyway; blocking it (GPTBot, Google-Extended, ClaudeBot, CCBot et al.) costs you approximately nothing visible and expresses a legitimate position on the trade. Deviate toward more blocking when: your content is the product (proprietary research, paid courses, licensable data — where training-value leakage is a real economics question, and where the conversation may eventually be licensing, not robots.txt); you have specific sections worth excluding (the methodology and data assets stay open — they’re your citation bait — but a client portal or paid-content path gets path-scoped disallows under every agent). Deviate toward blanket-open when: you’re in a citation land-grab (a category where nobody local is being cited yet) and maximal retrievability is the whole strategy — just do it as a decision, not a default-by-neglect.
Ship it like the production change it is. Write the file with explicit per-agent groups and a dated comment block stating the policy (‘# AI crawler policy, 2026-06: retrieval allowed, training blocked — see /ai-policy’); validate syntax (one malformed line can change how every group parses, and the difference between Disallow: / and Disallow: is the difference between everything and nothing); confirm the User-agent: * group still says what you think it says; and test the critical agents’ access with a tester tool before and after. Then verify in reality: server logs (or your CDN’s bot analytics) showing the allowed agents fetching and the blocked ones stopping — noting that compliant bots may take days to re-read the file; your search presence unchanged (Search Console’s crawl stats as the Googlebot canary); and the mention audit on its normal cadence confirming the answer engines still cite you. The horror story this paragraph prevents: the blanket Disallow: / under User-agent: * shipped in an ‘AI blocking’ edit — which is not an AI policy but a de-listing from the entire searchable web.
Maintenance: A Policy, Reviewed Quarterly
This landscape moves faster than any other file on your server: new agents ship, purposes split (companies separating training from retrieval agents — usually in your favor, letting you refine a blunt block), standards emerge, and the answer engines’ market shares shift under your prospects’ habits. The quarterly fifteen minutes: re-read the major platforms’ crawler documentation against your file (new agents get sorted into the taxonomy and the policy applied); scan logs for significant unrecognized crawlers (sort into “documented — decide,” “undocumented — CDN problem, not robots.txt problem”); and reconcile with the visibility data — if the measurement stack shows an engine driving real leads whose crawler you’re blocking, or the audit shows you absent from a platform you thought you’d allowed, the file is the first suspect. The policy itself — retrieval open, infrastructure sacred, training by principle — tends to be stable; the agent list implementing it is what the calendar entry maintains.
5 Common AI-Crawler Policy Mistakes
- The blanket “block all AI bots” sweep. Training and retrieval blocked together — a values statement paid for in vanished answer-engine visibility.
- Touching the infrastructure. Googlebot/Bingbot blocks (or a
*-group accident) chasing an AI opt-out — the search-presence extinction event. - Policy by legacy file. The 2019 robots.txt’s
*rules silently deciding your 2026 AI posture — audit what’s actually live before writing what’s intended. - Expecting robots.txt to stop scrapers. It governs the compliant; the non-compliant are a CDN/WAF problem — and abandoning the file because it isn’t a wall abandons your leverage over the engines that do listen.
- Set-and-forget. A quarterly-changing agent landscape managed by a file last edited at launch — the fifteen-minute review is the whole maintenance cost.
Frequently Asked Questions
What's the actual robots.txt syntax to block training bots but allow AI search engines?
The structure is per-agent groups, each explicitly decided — conceptually: for each training agent you’re blocking (GPTBot, Google-Extended, ClaudeBot, CCBot, Meta’s external agent, and whichever others your policy covers), a two-line group — User-agent: [AgentName] followed by Disallow: /; for the retrieval agents you’re allowing (ChatGPT-User and OpenAI’s search agent, PerplexityBot, and peers), either no group at all (they fall through to your general rules) or an explicit User-agent: [AgentName] + Allow: / group if your User-agent: * section contains disallows you don’t want inherited — the fall-through behavior is the subtle part, so check what your star group actually says; and your existing infrastructure and general rules (Googlebot untouched, your normal path disallows, the Sitemap line) stay as they are. Three implementation notes: exact agent-name spelling matters (copy from each company’s current crawler documentation rather than from blog posts — including this one — because names change); Disallow: / versus an empty Disallow: are opposites (everything versus nothing — the classic typo); and date-stamp the policy in a comment so future-you knows when it was last a decision rather than an inheritance. Validate with a robots tester before shipping, then confirm in logs that the blocked agents actually stop — compliant crawlers re-read the file on their own schedule, typically within days.
If I block GPTBot, does my business disappear from ChatGPT's answers?
No — and the distinction is the single most decision-relevant fact in this topic. GPTBot is OpenAI’s training crawler: blocking it keeps your content out of future model-training corpora, which affects what the model ‘absorbed’ but not what its live search retrieves. ChatGPT’s answers about current, specific things — including ‘who does [service] in [city]’ and ‘tell me about [your company]’ — increasingly run through its search/browsing layer, which fetches pages via separate retrieval agents (ChatGPT-User and OpenAI’s documented search crawler) and cites what it used; allow those, and your pages remain fully groundable and citable regardless of your GPTBot stance. What you do give up by blocking training: whatever diffuse benefit comes from the model having internalized your content between training runs — for a local service business, modest, since the queries that matter to you get answered from live retrieval and from the broader web’s corroborating sources anyway. Two caveats keep this honest: model-memory answers (when the engine responds without searching) draw on training-era data, so a heavily-blocked footprint contributes less to those — but those answers about small businesses are exactly the stale, ungrounded ones you least want anyway; and the agent architecture evolves — the training/retrieval split is a per-company documentation fact to re-verify quarterly, not a law of nature. The strategic summary: block GPTBot freely if the training trade offends you; guard the retrieval agents like you guard your Google presence.
Should I block AI crawlers to protect my content from being copied by competitors?
This concern misroutes — the threat model and the tool don’t match. What AI training does with your content: dissolves it into a statistical corpus alongside billions of pages — the model doesn’t store your articles for competitors to check out; the ‘copying’ risk from training is diffuse style-and-knowledge absorption, not retrieval of your pages. What actually enables competitor copying: your content being publicly visible at all — a competitor (or their content agency) copying your site does it with a browser, or a scraper that never read robots.txt; no crawler policy touches that vector, and the real protections are the ones that already exist: copyright (verbatim theft is actionable regardless of AI anything), and the competitive moat this guide’s cluster keeps pointing at — content whose value can’t be copied because it’s made of your data, your cases, your local specificity, and your attributed expertise (a competitor can imitate your cost page’s format; they cannot cite your 214 jobs). Meanwhile, the cost side of blocking-for-protection is concrete: sweep the retrieval agents into the block and you’ve removed your content from the answer engines — where its visibility was the point — to prevent a copying vector the block doesn’t actually close. If specific assets genuinely need protection from all automated access (client portals, paid content, proprietary datasets you license), the tools are authentication and path-scoped exclusion for those paths — surgical, effective, and compatible with keeping your public citation assets maximally retrievable, which is where their business value lives.
How do I know which AI bots are actually crawling my site right now?
Read the traffic you’re already receiving — three lookouts of increasing convenience. Server access logs: filter by user-agent string for the known names (GPTBot, PerplexityBot, ClaudeBot, Google-Extended, ChatGPT-User, CCBot, Bingbot, and the current list from each platform’s docs) — a simple log grep or your hosting panel’s log viewer shows who’s fetching what and how often; expect the training crawlers to sweep broadly and the retrieval agents to fetch specific pages in bursts (each burst is, encouragingly, usually a real user’s question being grounded). CDN/security dashboards: if you’re behind a CDN, its bot analytics typically categorize AI crawlers by name with request volumes — the friendliest view, and also where you’d see the non-compliant traffic that never announces itself honestly (user-agent strings are self-reported; sophisticated scrapers lie, which is why serious verification cross-checks the declared name against the platform’s published IP ranges — most major AI companies publish theirs precisely for this). And the absence check after policy changes: post-block, the disallowed agents’ requests should taper within days as they re-read robots.txt — persistent fetching by an agent you’ve blocked means either a spoofed user-agent (CDN problem) or a genuinely non-compliant crawler (also a CDN problem); escalate those to rate-limiting or WAF rules rather than shouting into the text file. Cadence: a quarterly log scan alongside the policy review — and any unrecognized high-volume crawler gets sorted by one question: documented by a real company (decide per the taxonomy) or not (block at the edge, not in robots.txt).
Does allowing AI crawlers slow down my website or cost me bandwidth?
Marginally, and for most service-business sites the honest answer is ‘not enough to notice’ — with real exceptions worth knowing. The scale: a typical local-business site (dozens to a few hundred pages) sees AI-crawler traffic as a small fraction of total bot traffic, itself a fraction of total requests — training crawlers sweep periodically rather than continuously, and retrieval agents fetch on-demand per user question; against a normally-provisioned host or any CDN, the load is rounding error, and the bandwidth cost of a bot reading HTML is trivial next to your human visitors loading images. The exceptions where crawler load becomes a real conversation: very large sites (tens of thousands of URLs, where aggressive sweeps consume measurable crawl-adjacent resources), sites on minimal shared hosting with strict resource caps, dynamically-expensive pages (each fetch triggering heavy computation — a design issue crawlers merely expose), and the occasional misbehaving crawler hammering at abusive rates — which the compliant ones don’t (major AI crawlers respect reasonable pacing and many honor crawl-delay conventions), and the non-compliant ones are, again, a rate-limiting/WAF matter rather than a policy one. The practical posture: don’t make performance the deciding factor in your AI-crawler policy at service-business scale — the visibility economics dominate by orders of magnitude; do put the site behind a CDN if it isn’t already (which mostly absorbs the question), and if logs ever show a specific agent at genuinely abusive volume, handle that agent at the edge as the operational incident it is, separate from the strategic policy this guide is about.
Is your robots.txt a policy or an accident?
We’ll audit what your file actually says to every agent, set the policy that protects your search presence and your answer-engine visibility, ship it with verification — and hand you the quarterly checklist that keeps it current.
Get an AI Access Audit Explore SEO Services