What is web scraping?

02AI crawlers and robots.txt

The newest large source of automated reads is AI. Some of it is declared: operators such as OpenAI document separate training and search crawlers that honour robots.txt and publish their IP ranges, while user-triggered fetchers such as ChatGPT-User act on a person's request, so robots.txt rules may not apply to them. The tokens, a tested robots.txt and server-side blocks for the bots that ignore it are covered in how to block AI crawlers.

robots.txt is a request, not a lock. RFC 9309, which standardised the Robots Exclusion Protocol in 2022, states that it is not a substitute for valid content security measures, and listing a path in it advertises that the path exists. Use it to tell well-behaved web crawlers what you want, and use rate limits and challenges for everything that ignores it.

03When it is actually a problem

It is worth being honest about which of these applies to you. Some scraping is genuinely harmless, and defences aimed at all automation tend to catch the crawlers that bring you visitors. Good bot management is about telling the two apart, not about blocking everything that is not a person.

• Competitors tracking your prices in real time and undercutting automatically
• Your content republished elsewhere, competing with you in search results
• Listings and inventory copied wholesale onto another marketplace
• Contact data harvested for spam
• Server load and bandwidth spent on traffic that never converts

04What reduces scraping

Scraping cannot be eliminated: anything a browser can display can be extracted. What is achievable is making it expensive and slow enough to stop being worthwhile.

• Rate limit per client on the endpoints that expose your valuable data, not globally
• Require a challenge before serving bulk listing or search endpoints
• Watch for breadth rather than volume: real users view a handful of products, scrapers view all of them
• Treat datacentre networks differently from residential ones for anonymous bulk access; the autonomous system behind an address tells you which kind it is
• Put the most valuable data behind authentication so abuse is attributable to an account
• Monitor for one fingerprint requesting a wide spread of distinct URLs

05Matching the response to the scraper

Different scrapers call for different answers, and the cheapest effective one is usually the right one. Blocking everything automated is rarely proportionate and almost always catches something you wanted.

API design matters more than people expect. A search or listing endpoint that returns thousands of items per call hands a scraper your catalogue in a handful of requests, so page-size limits and per-key quotas often do more than any detection rule.

06Distinguishing a scraper from a customer

Request volume alone is a poor signal: a real person browsing quickly can outpace a polite scraper. The reliable difference is in the shape of the session.

A customer arrives from somewhere, looks at a few items, loads images and scripts, and leaves. A scraper often enters directly on a listing page, requests items in order, fetches no sub-resources, and works through far more distinct URLs than any person would view.

Your access log already holds the evidence. The first query below ranks clients by how many distinct URLs they requested; the second lists clients that loaded many pages but never a stylesheet, script or image. Anything near the top deserves a look: an IP lookup shows which network an address belongs to, and verified crawlers should be excluded before you act.

# distinct URLs per client address
awk '{print $1, $7}' access.log | sort -u \
  | awk '{n[$1]++} END {for (ip in n) print n[ip], ip}' | sort -rn | head

# clients with more than 50 page views and no static assets at all
awk '$7 !~ /\.(css|js|png|jpg|svg|woff2?)$/ {p[$1]++}
     $7 ~  /\.(css|js|png|jpg|svg|woff2?)$/ {a[$1]++}
     END {for (ip in p) if (!a[ip] && p[ip] > 50) print p[ip], ip}' access.log | sort -rn