Web Crawler Design: Crawl Politely and Reliably

A web crawler is a program that discovers and fetches web pages automatically, usually by starting from a set of addresses, following links or sitemaps and recording what it finds. Businesses use crawlers to monitor their own websites for broken links, track public information such as product listings or job postings, build search indexes for internal content and feed data pipelines. A crawler that is poorly designed can overload websites, fetch the same pages endlessly, ignore site owners' wishes and produce messy, duplicated data. Good web crawler design is about discovering the right pages efficiently while behaving politely and reliably. This guide covers discovery, robots rules, rate limits, duplicate handling, failures and logging.

What a web crawler does

At its simplest, a web crawler repeats a loop:

1. Take the next address from a queue, often called the frontier
2. Check whether it is allowed and due to be fetched
3. Fetch the page and record the response
4. Extract links and data from the page
5. Add new, relevant addresses to the queue

Everything else, from politeness to deduplication, is about making that loop safe, efficient and useful.

Define the scope first

Decide exactly what the crawler should cover before writing code:

• Which domains and sections are in scope
• Which page types matter, such as product pages or articles
• How deep to follow links
• How often pages should be revisited
• What data to extract, and what to ignore

Clear scope keeps the crawler from wandering into calendars, search results, filters and other areas that generate endless URLs.

Start from sitemaps where available

Many sites publish XML sitemaps listing the pages they want discovered. Sitemaps can be a more efficient starting point than following every link. The sitemaps protocol limits each sitemap file to 50,000 URLs and 50MB uncompressed, and larger sites use sitemap index files that point to several sitemaps (sitemaps.org protocol). Sitemaps often include last-modified dates, which can help decide which pages to revisit, though they are not always accurate.

Follow robots.txt rules

The Robots Exclusion Protocol, standardised as RFC 9309, lets site owners tell crawlers which paths they may access (RFC 9309: Robots Exclusion Protocol). Key points from the standard for crawler builders:

• Rules are grouped by user agent, so identify your crawler with a clear name
• When several rules match a path, the most specific match, the one with the most octets, must be used
• If robots.txt is unavailable, for example returning a status in the 400 range, the crawler may access any resources
• If robots.txt is unreachable because of server or network errors, the crawler must assume complete disallow
• Crawlers should not use a cached copy for more than 24 hours, unless the file is unreachable

The standard also states that these rules are not a form of access authorisation. Respecting them is a matter of good conduct and reduces the risk of disputes; it does not on its own establish that crawling a site is permitted. Check terms of use and applicable law as well. Our guide to web scraping compliance covers these wider considerations.

Limit request rates

Politeness means not overwhelming the sites you crawl:

• Limit concurrent requests to each host
• Add delays between requests to the same host
• Slow down when responses become slower or errors increase
• Respect any rate guidance the site provides
• Crawl during quieter hours where appropriate

Rate limits should apply per host, not just globally, so one site does not receive most of the traffic.

Identify your crawler

Use a descriptive user agent that names your organisation and provides a contact address or page. This lets site owners understand who is crawling and get in touch if there is a problem, rather than simply blocking you.

Avoid duplicate and endless URLs

Many sites expose the same content at several addresses, through tracking parameters, sorting options, session IDs or trailing slashes. Normalise URLs before adding them to the queue: lowercase hostnames, remove known tracking parameters, sort query parameters and apply consistent trailing slash rules. Track which URLs have been fetched. Set limits on queue size, depth and pages per host to avoid crawler traps that generate infinite URLs.

Worked example: crawler policy settings

The table below is a hypothetical policy for a crawler that monitors public product pages on a small set of retailer sites. The values are illustrative, not recommendations for any particular site.

SettingValueReasonScopeProduct pages on listed domains onlyAvoid unrelated sectionsDiscoverySitemaps first, then in-scope linksEfficient and respects site structureRobots rulesFetched per host, cached up to 24 hoursFollows RFC 9309 guidanceConcurrency per hostOne request at a timeLight load on each siteDelay between requestsSeveral seconds, longer if errors risePolitenessURL normalisationStrip tracking parameters, sort query stringsReduce duplicatesRevisit frequencyDaily for listed productsMatches business needMaximum pages per host per runCappedPrevents traps and overload

Handle failures gracefully

Expect timeouts, server errors, redirects and blocked responses. Retry temporary failures a limited number of times with increasing delays. Follow redirects but cap the number of hops. Record permanent failures and stop retrying them. If a host returns many errors, pause crawling it rather than pushing harder.

Choose fetching tools to fit the pages

Simple HTML pages can be fetched with an HTTP client, which is fast and light. Pages that build their content with JavaScript may need a headless browser, which is slower and heavier. Use browser rendering only where necessary. Our guide to Playwright web scraping covers when and how to use a headless browser for JavaScript-heavy pages.

Log every fetch

Keep a record of each request: URL, time, status code, response size, duration and any error. Logs help you debug extraction problems, prove that rate limits and robots rules were respected and spot sites that have changed structure.

Store and process data carefully

Separate fetching from extraction where possible, storing raw responses for a limited time so extraction can be rerun after fixes. Validate extracted data, deduplicate records and avoid collecting personal data you do not need.

Monitor and maintain

Monitor pages fetched, error rates, queue size and extraction success per site. Alert when a site's results drop sharply, which often means its structure has changed. Review scope and policies regularly.

Frequently Asked Questions

What is a web crawler?

It is a program that discovers and fetches web pages automatically by following links or sitemaps from a starting set of addresses and recording what it finds.

Must a web crawler follow robots.txt?

Well-behaved crawlers do. RFC 9309 sets out how to read and apply the rules, though it states they are not a form of access authorisation.

How do you stop a crawler from overloading websites?

Limit concurrent requests and delays per host, slow down when errors rise, identify your crawler clearly and crawl only what you need.

How do crawlers avoid duplicate pages?

By normalising URLs, removing tracking parameters, tracking fetched addresses and capping depth, queue size and pages per host.

Build crawlers that behave well

We design and run web crawlers with clear scope, polite rate limits and reliable data handling. See our B2B lead mining and prospect verification service, or talk to us about web crawler design.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.