Anti-Bot Protection: What Scrapers Must Respect

Many websites now use anti-bot protection to manage automated traffic. Challenge pages, rate limits and bot scores protect sites from credential stuffing, content theft, inventory hoarding and server overload. For anyone collecting web data, these systems raise an important question: what should a responsible scraper do when it meets them? The short answer is to treat anti-bot protection as a clear statement of the site owner's wishes, not as a puzzle to solve. This guide explains how these systems work at a high level, what signals they send, how a responsible data collector should respond and which legitimate alternatives exist when a site does not want automated access.

What anti-bot protection does

Anti-bot protection refers to tools and techniques that sites use to distinguish automated traffic from human visitors and to decide how to treat it. Common goals include:

• Blocking credential stuffing and account takeover attempts
• Preventing large-scale content or price copying
• Stopping automated purchases of limited stock
• Protecting servers from excessive load
• Allowing helpful automation, such as search engine crawlers, while restricting the rest

These tools are part of how site owners protect their users, infrastructure and business.

How these systems assess traffic

At a high level, bot management services combine many signals to assess whether a request is likely automated. Cloudflare, for example, describes a bot score from 1 to 99, where a score of 1 means the request is quite certainly automated and 99 means it likely came from a human (Cloudflare bot score). Site owners then set rules, such as allowing, challenging or blocking traffic in certain score ranges.

Many providers also maintain lists of verified bots, such as major search engine crawlers, that site owners may choose to allow (Cloudflare bot solutions). Being recognised as a verified, well-behaved crawler is very different from trying to look like a human visitor.

Signals a scraper should respect

When collecting data, several signals tell you what the site owner wants:

• Terms of use that restrict automated access or data extraction
• Robots.txt rules disallowing certain paths
• HTTP 429 Too Many Requests responses, sometimes with a Retry-After header
• HTTP 403 Forbidden responses
• Challenge pages, such as interactive checks or waiting pages
• Login walls protecting content

The robots.txt standard, RFC 9309, notes that its rules "are not a form of access authorization" (RFC 9309), so following it is necessary good behaviour but not sufficient permission on its own.

How to respond to rate limits

A 429 response means you are sending too many requests. MDN describes it as indicating that the client has sent too many requests in a given amount of time (MDN 429). The right response is to slow down:

• Honour any Retry-After header, which indicates how long to wait (MDN Retry-After)
• Reduce concurrency and request rates permanently, not just briefly
• Add backoff so repeated limits lead to longer waits
• Cache results to avoid refetching unchanged pages

Rate limits are an invitation to collect more gently, not to spread requests across more addresses.

How to respond to blocks and challenges

When a site returns 403 responses or challenge pages to your crawler, it is telling you it does not want that traffic. A responsible collector should:

• Stop collection from that site
• Review the site's terms and robots.txt again
• Consider whether an API, data licence or partnership exists
• Contact the site owner to request permission, explaining what you need and how you will collect it

Techniques designed to defeat these protections, such as disguising automated traffic as human, solving challenges automatically or rotating identities to avoid detection, can breach terms of use, may break laws in some jurisdictions and damage trust. We do not recommend or build them. This article is general information, not legal advice.

Worked example: responding to anti-bot protection signals

The table below shows how a responsible collection process might respond to common signals. It describes policy, not configuration for any specific tool.

SignalWhat it meansResponsible responseEscalationRobots.txt disallows target pathOwner prefers no crawling thereDo not crawl the pathSeek permission or alternative source429 with Retry-AfterRequests too frequentWait the stated time, then reduce rate permanentlyReview schedule if repeatedOccasional 5xx errorsServer strain or faultsBack off and retry later with limitsPause if persistent403 on most requestsAccess refusedStop collection from the siteContact owner or use licensed dataChallenge or interstitial pageTraffic assessed as automatedStop; do not attempt to solve or evadeRequest permission or use an APIContent behind loginAccess subject to accepted termsDo not scrape unless terms expressly allowUse official API or agreement

Each row ends with stopping or seeking permission when the site has clearly said no. That is the core of responsible collection.

Be transparent about your crawler

Responsible crawlers identify themselves:

• Use a descriptive user agent naming your organisation or crawler
• Provide a contact page or email in the user agent or on a linked page
• Publish what you collect and how site owners can opt out
• Honour opt-out requests promptly

Transparency allows site owners to recognise and, if they choose, allow your traffic.

Design collection to minimise load

Good engineering reduces the chance of triggering protection at all:

• Collect only the pages and fields you need
• Schedule crawls at modest rates and during quieter hours
• Use conditional requests and caching where supported
• Avoid rendering full pages in a browser when a simple request works

Our guide to headless browser scraping explains when browser automation is genuinely needed and why it adds load and cost.

Understand what proxies are for

Proxies have legitimate uses, such as viewing region-specific content or separating traffic for reliability. They are not a way to gain permission. Using them to evade blocks or rate limits that a site has deliberately applied conflicts with responsible collection. Our guide to rotating proxies for web scraping discusses these trade-offs in more detail.

Look for legitimate alternatives

When a site protects its data, there are usually better routes:

• Official APIs, often with clear terms and rate limits
• Data licences from the site owner
• Specialist data providers with rights to the data
• Partnerships or data-sharing agreements
• Public datasets published by governments or industry bodies

These options are usually more reliable and less risky than fighting protection systems.

Frequently Asked Questions

What is anti-bot protection?

It is the set of tools sites use to identify automated traffic and decide whether to allow, challenge, limit or block it, protecting users and infrastructure.

What should a scraper do when it gets rate limited?

Slow down, honour any Retry-After header, add backoff and reduce request rates permanently. Do not spread requests across more addresses to avoid the limit.

Is it acceptable to bypass bot challenges?

We advise against it. Challenges signal that the site does not want automated access, and bypassing them can breach terms and, in some places, laws.

How can I get data from a site with strong bot protection?

Ask for permission, use an official API, license the data from the owner or a specialist provider, or use public datasets.

Collect data the right way

We design data collection that respects site owners: polite crawling, transparent identification and APIs or licensed data where sites restrict access. See our B2B lead mining and prospect verification service, or talk to us about responsible data collection.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.