Quick answer
A web crawler — also called a spider or bot — is an automated program that requests pages the way a browser would, but without a person driving it. Crawlers power search engines, AI training datasets, link previews in chat apps, uptime monitors and price comparison sites. They are not inherently good or bad: the same technique that lets Google index your shop lets a competitor copy your entire catalogue. What separates them is who operates them, whether they identify themselves honestly, and whether they respect the limits you set.
How a crawler works
A crawler starts with a list of URLs, fetches each one, extracts the links it finds, and adds those to the queue. Repeat at scale and you eventually map a large part of the public web. That loop has barely changed since the 1990s; what has changed is the scale and the purpose.
Well-behaved crawlers announce themselves in the User-Agent header, check `/robots.txt` before fetching, and pace themselves so they do not overwhelm a small site. Badly behaved ones do none of this — and the worst deliberately impersonate browsers or other people's crawlers to avoid being filtered.
The main kinds of crawler
The first four categories are usually worth having. Search crawlers bring visitors; preview fetchers make your links look right when someone shares them; AI crawlers increasingly decide whether your product is mentioned when someone asks an assistant a question.
| Kind | Examples | Wants |
|---|---|---|
| Search engines | Googlebot, Bingbot, YandexBot | To index your pages so people can find them |
| AI / LLM crawlers | GPTBot, ClaudeBot, PerplexityBot | Content for training or answering questions |
| Social previews | Discordbot, Twitterbot, WhatsApp | The title, description and image for a shared link |
| Monitoring | Uptime and SEO tools | To check availability or audit your pages |
| Scrapers | Mostly unnamed | Your prices, listings, content or contact data |
| Attack tooling | Disguised as anything | Vulnerabilities, credentials, or simply to exhaust your server |
Why a user-agent proves nothing
Any client can claim to be anything. Sending `Googlebot/2.1` as a user-agent takes one line of code, and impersonating Googlebot is one of the most common tricks for getting past a filter, precisely because so many sites allowlist it on sight.
The only reliable check is the network the request came from. Google, Bing, Apple, OpenAI, DuckDuckGo and Perplexity all publish the IP ranges their crawlers use, in machine-readable files intended exactly for this. A request claiming to be Googlebot from an address outside Google's published ranges is not Googlebot.
Managing crawler traffic
Crawlers are not attacks, so blocking them outright is usually the wrong response — but letting them run unlimited is not right either. A large crawl of a database-backed site can look a lot like an application-layer attack from the server's point of view.
- robots.txt states your rules — respected by legitimate crawlers, ignored by everyone else, so it is a policy document rather than an enforcement mechanism
- Verify by network before granting any privilege, so impersonators gain nothing from the user-agent string
- Rate limit rather than block, so a legitimate crawler slows down instead of losing your pages from the index
- Cache aggressively — a crawler hitting a cached page costs you nothing at all
- Watch for crawlers that ignore robots.txt entirely; that alone tells you what kind of operator you are dealing with
AI crawlers are a new decision
Crawlers that collect data for language models raise a question search crawlers never did. A search engine sends you visitors in exchange for indexing you. An AI crawler may answer the question your page would have answered, without the visit.
Most major AI crawlers now respect robots.txt and publish their ranges, so the choice is genuinely yours: allow them and be represented in AI answers, or disallow them and stay out of the training data. There is no universally correct answer — a documentation site and a subscription news site will reasonably decide differently.
User-agent: Googlebot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /Frequently asked questions
Last updated