Quick answer
Bot management is the practice of telling automated traffic apart from human traffic and applying a different policy to each kind — allowing the bots you want, limiting the ones you tolerate, and stopping the ones that cost you money. It is distinct from DDoS protection: bots may arrive at completely ordinary request rates and still cause harm by scraping prices, testing stolen credentials or buying limited stock before your customers can.
Why bots need their own policy
Roughly half of all web traffic is automated, and most of it is neither an attack nor a customer. Treating everything as human means scrapers copy your catalogue; treating everything as suspicious means you break the integrations your business depends on.
- Scraping — competitors copying prices, listings or content wholesale
- Credential stuffing — testing leaked username and password pairs against your login
- Card testing — validating stolen card numbers with small purchases
- Inventory hoarding — bots reserving limited stock so real buyers cannot
- Ad fraud — fake impressions and clicks on the traffic you pay for
- Spam — automated posting to forms, comments and signups
How bots are detected
No single signal is reliable, so detection stacks many weak ones. Individually each has false positives; together they are hard to fake consistently.
| Signal | What it reveals |
|---|---|
| TLS fingerprint (JA3/JA4) | Which library performed the handshake, regardless of the user-agent claimed |
| Header order and completeness | Real browsers send a specific set in a specific order; tools rarely reproduce it |
| HTTP/2 settings | Frame and priority patterns differ between browser engines and libraries |
| Behaviour | Whether sub-resources are fetched, how navigation flows, how fast pages are requested |
| Network | Datacentre ranges are expected for API clients and suspicious for a claimed home browser |
| Challenge outcome | Whether the client can execute JavaScript at all |
Fingerprinting, and its limits
A TLS fingerprint such as JA3 or JA4 is a hash of how a client negotiates the connection — cipher suites offered, extensions and their order. It identifies the underlying library, which is why a Python script claiming to be Chrome is trivially caught: it negotiates like Python, not like Chrome.
The limit is that fingerprints are shared, not unique. Every Chrome user on the same version and platform produces the same value, so a fingerprint identifies a class of client, never an individual. That is a feature for privacy and a constraint for detection: a fingerprint alone can support a decision, but it must not make one.
Responses, from lightest to heaviest
Bot management is not a block button. The response should be proportionate to the confidence and to the cost of being wrong.
- Allow — verified crawlers you want, passed without friction
- Rate limit — tolerated automation kept to a sane pace rather than shut out
- Automatic challenge — a proof-of-work the browser solves invisibly; costs a person nothing and a script everything
- Interactive challenge — a visible check, reserved for clients that already failed the automatic one
- Block — for unambiguous cases only: exploit probes, known-bad tooling, verified impersonators
The failure mode to design against
Every bot management system has a dial between catching bots and blocking customers, and the two are not equally expensive. Missing a scraper costs you some data. Blocking a customer costs you the customer, and they rarely tell you — they just leave.
Practical safeguards: require several independent signals before acting, prefer challenges over blocks for anything ambiguous, clear a client's penalties once it proves itself, and never apply browser-shaped expectations to routes you know are APIs.
Frequently asked questions
Last updated