Quick answer
Web scraping is the automated extraction of data from websites — fetching pages and parsing out prices, listings, articles or contact details. The technique itself is neutral and powers search engines, price comparison and research. It becomes a business problem when a competitor mirrors your catalogue in real time, when your content is republished elsewhere, or when the traffic costs you more to serve than the visitors it displaces.
How scrapers work
The simplest scraper is a script that fetches a URL and extracts data with a parser or regular expression. It is cheap, fast, and easy to detect because it does not behave like a browser at all: no sub-resources, no JavaScript, unusual headers.
More capable scrapers drive a real browser, which renders JavaScript and behaves far more convincingly. The most determined route requests through residential proxy networks so the traffic appears to come from ordinary home connections, which removes network reputation as a signal.
When it is actually a problem
It is worth being honest about which of these applies to you. Some scraping is genuinely harmless, and defences aimed at all automation tend to catch the crawlers that bring you visitors.
- Competitors tracking your prices in real time and undercutting automatically
- Your content republished elsewhere, competing with you in search results
- Listings and inventory copied wholesale onto another marketplace
- Contact data harvested for spam
- Server load and bandwidth spent on traffic that never converts
What reduces scraping
Scraping cannot be eliminated — anything a browser can display can be extracted. What is achievable is making it expensive and slow enough to stop being worthwhile.
- Rate limit per client on the endpoints that expose your valuable data, not globally
- Require a challenge before serving bulk listing or search endpoints
- Watch for breadth rather than volume: real users view a handful of products, scrapers view all of them
- Treat datacentre networks differently from residential ones for anonymous bulk access
- Put the most valuable data behind authentication so abuse is attributable to an account
- Monitor for one fingerprint requesting a wide spread of distinct URLs
Distinguishing a scraper from a customer
Request volume alone is a poor signal — a real person browsing quickly can outpace a polite scraper. The reliable difference is in the shape of the session.
A customer arrives from somewhere, looks at a few items, loads images and scripts, and leaves. A scraper often enters directly on a listing page, requests items in order, fetches no sub-resources, and works through far more distinct URLs than any person would view.
Frequently asked questions
Last updated