Should You Block AI Crawlers From Your Website?

Not all AI crawlers do the same thing

Training crawlers collect pages that may be used to train AI models. OpenAI's GPTBot, Anthropic's ClaudeBot and Common Crawl's CCBot are examples.
Search and answer crawlers fetch pages so an assistant can find and cite them for a user's question, such as OpenAI's OAI-SearchBot.
Control tokens such as Google-Extended are not separate crawlers; they tell Google whether content it already crawls may be used for certain Gemini uses, without affecting Google Search.

Each company documents its own bots and they change, so check the current pages: OpenAI's crawlers, Anthropic's crawler, Google's crawlers and tokens and Common Crawl's CCBot.

Robots.txt is the control, not a lock

Your /robots.txt file tells well-behaved crawlers which paths they may visit; Google's robots.txt introduction explains the format. Examples:

Block selected training crawlers: User-agent: GPTBot / Disallow: /, then the same for ClaudeBot and CCBot.
Allow ChatGPT search but not OpenAI training: User-agent: OAI-SearchBot / Allow: / and User-agent: GPTBot / Disallow: /.
Limit Google's AI use without touching Google Search: User-agent: Google-Extended / Disallow: /.

Robots.txt is voluntary and public. It only works for crawlers that identify themselves and obey it, and it does not remove content already copied or protect private pages. Anything truly private belongs behind a login.

The visibility trade-off

Blocking can matter when your content is the product: original research, courses, recipes, documentation or a niche database. But AI assistants increasingly shape discovery. If they cannot read your pages, they may leave you out or answer from less accurate sources, a big cost for local services and online stores. Our guide Is ChatGPT recommending your competitor? covers the discovery side.

A common middle ground: block training crawlers, allow search and answer crawlers plus Googlebot and Bingbot.

A simple decision guide

Local service business: allow search and answer crawlers; block training only if your content is unusually original.
Online store: allow product pages; block carts, accounts, internal search and filter pages for every crawler.
Consultant or agency: usually allow public expertise pages, which earn leads when cited; protect downloads and client areas.
Publisher, course or paid research site: consider blocking training crawlers and keep premium content behind login.
Software documentation: allow public docs for answer crawlers; keep admin, security and customer-only docs private.

Check you are not blocking by accident

Many sites block AI crawlers without the owner knowing, through a security plugin, firewall rule, CDN bot setting or an old robots.txt line. Look at your robots.txt, your firewall or CDN bot settings and your server logs: which bots visit, how often, what status codes they get and whether they hit slow pages. Some crawlers do not run JavaScript, so make sure important content is in the page HTML.

What llms.txt is

llms.txt is a proposed file, placed at /llms.txt, that gives language models a short map of your most useful pages. It can help documentation and developer sites, but it is a proposal rather than an official standard, and it does not replace robots.txt, sitemaps, structured data or good internal links. This site publishes one at /llms.txt as an example.

A sensible default

Decide whether your content itself is what you sell.
If not, allow search and answer crawlers; decide separately about training crawlers.
Block private and low-value paths for everyone.
Check firewall and CDN settings match robots.txt.
Review logs every quarter and re-read each company's crawler documentation.

Where ASL fits

SiteFit, our website audit tool, gives a scored, read-only review of your public site with clear fix priorities, a good starting point before changing crawler rules. Pair it with our website audit checklist, or contact us for help with robots.txt and server settings.

Frequently asked questions

Should I block AI crawlers?

Only if protecting your content matters more than being found and cited by AI tools. Most small businesses allow search and answer crawlers and decide separately about training crawlers.

What is GPTBot?

OpenAI's crawler for content that may be used to train its models. It is separate from OAI-SearchBot, which helps surface sites in ChatGPT search.

Does blocking Google-Extended hurt my Google ranking?

Google says Google-Extended controls certain Gemini uses of content and does not affect inclusion or ranking in Google Search.

Does robots.txt stop all AI scraping?

No. It only works for crawlers that identify themselves and obey it. Private content should be behind a login.

Do I need an llms.txt file?

It is optional. It is a proposal, not a standard, and can help documentation sites point AI tools at key pages, but it does not replace robots.txt or sitemaps.

Related guides

Explore the complete product

Read features, current access and limitations