What are web crawlers and bots?

02The main kinds of crawler

Crawlers are one family of internet bots, and most of the named categories are worth having. Search crawlers bring visitors, preview fetchers make your links look right when someone shares them, and AI search crawlers increasingly decide whether your product is mentioned when someone asks an assistant a question.

The split between training, search and user-triggered fetching matters because operators treat them differently. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page. Anthropic says its bots, Claude-User included, honour robots.txt.

03Why a user-agent proves nothing

Any client can claim to be anything. Sending Googlebot/2.1 as a user-agent takes one line of code, and impersonating Googlebot is a common way to get past a filter, precisely because so many sites allowlist it on sight.

The reliable check is the network the request came from. Google, Microsoft, Apple, OpenAI, Anthropic, Perplexity and DuckDuckGo all publish the IP ranges their crawlers use as machine-readable JSON files, and Google, Bing and Apple also support verification by reverse DNS. A request claiming to be Googlebot from an address outside Google's published ranges is not Googlebot.

For a quick manual look, an IP and ASN lookup shows which network announces an address. Google, Bing and Apple crawl from their own networks, identified by their autonomous system numbers, so a 'Googlebot' arriving from a budget hosting provider is an impersonator. Some AI operators crawl from cloud platforms, which is why their published lists, not the network owner, are the real test.

04How to verify a crawler yourself

Both methods take a minute with standard tools. The example uses a Googlebot address; the same pattern works for every operator in the table.

Treat a failed check as 'not that crawler' rather than 'attacker': the request may come from a harmless tool that borrowed a famous name. What it must not get is the trust you extend to the real crawler.

# 1. Reverse lookup: the name must end in googlebot.com, google.com or googleusercontent.com
host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

# 2. Forward lookup: the name must resolve back to the same address
host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

# 3. Or match the address against Google's published list (prints True or False)
curl -s -o google.json https://developers.google.com/static/crawling/ipranges/common-crawlers.json
python3 -c 'import ipaddress,json,sys; ip=ipaddress.ip_address(sys.argv[1]); print(any(ip in ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in json.load(open("google.json"))["prefixes"]))' 66.249.66.1

05Managing crawler traffic

Crawlers are not attacks, so blocking them outright is usually the wrong response, but letting them run unlimited is not right either. A large crawl of a database-backed site can look a lot like an application-layer attack from the server's point of view.

Remember too that robots.txt is public. RFC 9309 warns that listing paths in it 'makes the paths discoverable', so never use it to hide a sensitive URL. For traffic that ignores your rules, bot management takes over where robots.txt stops.

• robots.txt states your rules: respected by legitimate crawlers, ignored by everyone else. RFC 9309 says plainly that its rules are not a form of access authorisation
• Verify by network before granting any privilege, so impersonators gain nothing from the user-agent string
• Rate limit rather than block, so a legitimate crawler slows down instead of losing your pages from the index; Google reads a 429 response as a signal to crawl more slowly
• Serve crawlable pages from a CDN cache: a crawler hitting a cached page costs your origin nothing
• Keep robots.txt itself fast and static. Under RFC 9309, a robots.txt that answers with a server error tells crawlers to assume they may crawl nothing
• Watch for crawlers that ignore robots.txt entirely; that alone tells you what kind of operator you are dealing with

06AI crawlers are a new decision

Crawlers that collect data for language models raise a question search crawlers never did: a search engine sends you visitors in exchange for indexing you, while an AI system may answer the question your page would have answered, without the visit. Because the major operators separate training, search and user-triggered agents, and Google-Extended and Applebot-Extended are robots.txt tokens rather than crawlers, you can stay visible in AI search while opting out of model training. The current tokens, a tested robots.txt and what to do about bots that ignore it are covered in how to block AI crawlers.

07When crawler management goes wrong

Crawler problems tend to surface weeks after the change that caused them, as lost search traffic or broken previews.