How to block AI crawlers: robots.txt, server rules and IP checks

02How to block AI crawlers in robots.txt

List the tokens to refuse in one group and give it Disallow: /. Several User-agent lines may share a group. RFC 9309 makes crawlers match their token case-insensitively and obey the group that names them, falling back to the * group only when none does, so search crawlers that are not listed keep following the * group.

Two details catch people out. robots.txt applies per host, so every subdomain needs its own file, as Anthropic's documentation points out. And changes are not instant: RFC 9309 lets crawlers cache the file for up to 24 hours, and OpenAI, Perplexity and Meta each mention a delay of about a day.

# Opt out of the AI crawlers and fetchers in the table above (checked October 2026)
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
User-agent: meta-externalfetcher
User-agent: meta-webindexer
Disallow: /

# Everyone else, search engines included
User-agent: *
Disallow: /account/

03What robots.txt cannot do

robots.txt states a preference that well-run crawlers honour. RFC 9309 says its rules 'are not a form of access authorization', and the protocol has several further limits.

• It does not stop clients that ignore it, including user-triggered fetchers whose operators say robots.txt may not apply
• It works only from now on: Anthropic, for example, describes a ClaudeBot block as excluding the site's future materials from training
• It cannot reach copies held elsewhere, such as archives other crawlers already made or pages that users paste into an assistant
• It does not stop impersonators or unnamed scrapers, which is why web scraping needs its own defences
• A robots.txt that answers with a 5xx error tells every crawler, search engines included, to assume complete disallow, while a 4xx answer tells them they may crawl everything

04How to block AI crawlers in nginx

Server rules enforce what robots.txt only requests. A map in the http context sets a variable from the User-Agent header, and the location refuses matching requests with 403. The patterns are case-insensitive regular expressions, so the documented tokens match however an operator capitalises them.

Keep robots.txt itself outside the block. Under RFC 9309 a 4xx answer for robots.txt means the file is unavailable and crawlers may access everything, and Anthropic warns that blocking its crawlers' addresses stops them reading the opt-out. A refused robots.txt is therefore worse than none.

This works only for crawlers that announce themselves. Google-Extended and Applebot-Extended never appear in a User-Agent header, and refusing Googlebot or Applebot would remove the site from their search engines, so those two can be handled only in robots.txt.

# /etc/nginx/conf.d/ai-crawlers.conf (http context)
map $http_user_agent $ai_crawler {
    default               0;
    ~*GPTBot              1;
    ~*ClaudeBot           1;
    ~*CCBot               1;
    ~*Bytespider          1;
    ~*meta-externalagent  1;
    # add OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User,
    # Claude-User, Perplexity-User, meta-externalfetcher or meta-webindexer to refuse them too
}

# in the server block
location = /robots.txt {
    try_files $uri =404;    # served to everyone, AI crawlers included
}

location / {
    if ($ai_crawler) {
        return 403;
    }
    # ... the site's normal configuration
}

05How to tell a real AI crawler from an impersonator

A User-Agent header proves nothing, so when one AI crawler is allowed and others are refused, check the address too. OpenAI, Anthropic, Perplexity and Common Crawl publish their ranges as JSON files with a creationTime field showing when each list was generated, and Common Crawl also supports reverse DNS under crawl.commoncrawl.org. The general method, including reverse DNS for search engines, is in the guide to verifying web crawlers, and an IP lookup shows which network an address belongs to.

The snippet below turns OpenAI's search crawler list into an nginx geo file and refuses requests that claim to be OAI-SearchBot from anywhere else. Regenerate the file on a schedule and reload nginx afterwards, because the lists change. Behind a CDN or load balancer, configure the realip module first, or every request appears to come from the proxy and the real crawler is refused as well.

# 1. Build the geo file from OpenAI's published list (via a temporary file, so a failed download keeps the old list)
curl -fsS https://openai.com/searchbot.json | python3 -c 'import json,sys; [print(p.get("ipv4Prefix") or p.get("ipv6Prefix"), "1;") for p in json.load(sys.stdin)["prefixes"]]' > /etc/nginx/openai-searchbot.geo.new && mv /etc/nginx/openai-searchbot.geo.new /etc/nginx/openai-searchbot.geo

# 2. In the http context
geo $from_openai_search {
    default 0;
    include /etc/nginx/openai-searchbot.geo;
}
map $http_user_agent $claims_oai_search {
    default          0;
    ~*OAI-SearchBot  1;
}
map "$claims_oai_search:$from_openai_search" $fake_oai_search {
    default  0;
    "1:0"    1;
}

# 3. In location /
if ($fake_oai_search) {
    return 403;
}

06How to see which AI crawlers visit a site

Count them in the access log before and after a change: it shows whether the opt-out took effect and which agents ignore it. In nginx's default combined log format the User-Agent is the sixth field when a line is split on double quotes. OpenAI says it may add a robots.txt marker to its user-agent when it fetches that file, which makes those checks easy to spot.

# requests per AI agent (one count per request)
awk -F'"' '{ua = tolower($6)}
  match(ua, /gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|claude-user|perplexitybot|perplexity-user|ccbot|bytespider|meta-externalagent|meta-externalfetcher|meta-webindexer/) {
    print substr(ua, RSTART, RLENGTH)
  }' /var/log/nginx/access.log | sort | uniq -c | sort -rn

# what the training crawlers requested and the status they got
awk -F'"' 'tolower($6) ~ /gptbot|claudebot|ccbot|bytespider|meta-externalagent/ {split($3, s, " "); print $2, s[1]}' \
  /var/log/nginx/access.log | sort | uniq -c | sort -rn

07Why is an AI crawler still visiting?

08What a CDN or WAF can and cannot do about AI crawlers

An edge in front of the site can apply the same rules before requests reach the origin, check claimed crawlers against published ranges at scale, rate-limit the agents that are tolerated and serve them cached pages, so a heavy crawl costs the origin little. Bot management goes further, challenging clients that claim to be browsers but behave like scripts.

No edge does everything. It cannot recognise every undeclared scraper with certainty or recall content that was collected or copied earlier, and challenging all automation also catches assistants, accessibility tools and monitors that may be wanted. Blocking has a price too: OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, and Anthropic says blocking Claude-SearchBot or Claude-User may reduce visibility in its search results. Decide agent by agent, not all at once.