Rotating Proxies for Web Scraping, Done Responsibly

Use rotating proxies for web scraping without getting blocked: pool design, backoff, health checks and robots.txt rules. See a worked Python proxy pool.

Once a scraping project grows beyond a few hundred pages a day, the network layer starts to matter. Requests from a single server can be slowed or blocked, region-specific prices cannot be seen from one location, and a few unhealthy connections can drag down an entire crawl. Rotating proxies for web scraping spread requests across a pool of IP addresses and locations, so a distributed crawler stays reliable. This guide explains when proxies are genuinely needed, the main proxy types, how to design rotation across many scraper nodes, and a worked example of a health-aware proxy pool in Python, all within the bounds of responsible scraping.

First, decide whether you need proxies at all

Many projects do not. If you collect a modest number of pages from a few sites, a polite crawler running from one server, with sensible delays and a clear User-Agent, is usually enough. Proxies add cost and complexity, so we add them only when there is a concrete reason:

You need data as it appears in specific countries or cities, such as local prices or availability
Your crawl volume across many sites is large enough that one egress IP becomes a bottleneck
You run many scraper nodes and want consistent, managed egress rather than each node's own IP
You need to separate traffic for different clients or projects

Proxies are not a way to ignore a site's rules. If a site's robots.txt or terms forbid automated access, or the site is clearly asking you to slow down, the right response is to respect that, not to rotate around it. The Robots Exclusion Protocol is defined in RFC 9309.

The main proxy types

TypeWhat it isTypical useTrade-offsDatacenterIPs hosted in cloud or hosting providersHigh-volume crawling of tolerant sitesFast and inexpensive; easily identified as non-residentialISP (static residential)IPs registered to consumer ISPs but hosted in data centresStable sessions that need a residential-looking IPCosts more than datacenter; smaller poolsResidentialIPs from real household connections, via a provider networkGeo-specific data in many locationsHigher cost; check how the provider sources consent from IP ownersMobileIPs from mobile carrier networksMobile-specific contentUsually the most expensive; use only when required

When choosing a residential or mobile provider, ask how its network is sourced and whether device owners have knowingly opted in. Ethical sourcing is part of responsible scraping.

Rotation strategies

Rotation is not just "a new IP for every request". The right strategy depends on what the site and your task need.

Per-request rotation

Each request uses a different proxy. This suits independent page fetches, such as product pages, where no session state is needed.

Sticky sessions

The same proxy is kept for a short series of requests, such as a listing page followed by its detail pages. This looks like normal browsing and avoids breaking sites that tie sessions to an IP.

Per-domain pools

Different target domains get separate pools and separate rate limits. A problem on one site then never affects crawling elsewhere.

Geo routing

Requests for country-specific data are routed through proxies in that country, with the location recorded alongside the data so analysts know where each price was seen.

Running rotating proxies for web scraping across distributed nodes

When several scraper nodes share one proxy pool, coordination matters more than the number of IPs. We design around four components:

A shared proxy registry, usually in Redis, holding each proxy's location, health score and cool-down time.
Global per-domain rate limits, enforced across all nodes, so adding nodes never increases the load on a target site.
Health scoring, where successes raise a proxy's score and errors, timeouts and block pages lower it.
Cool-down rather than retry storms: a proxy that receives a 429 or a block page is rested for a period, and the request is retried later with backoff.

The second point is the one teams most often miss. Scaling out scraper nodes should increase reliability, not the request rate seen by any single website.

Worked example: a health-aware proxy pool in Python

The illustrative code below keeps a pool of proxies with health scores and cool-downs, picks a healthy proxy for each request, and treats HTTP 403 and 429 responses as a signal to rest that proxy and slow down. It uses the proxies argument described in the Requests documentation. In production, the pool state would live in Redis so every node shares it.

import random, time
from dataclasses import dataclass, field
import requests

@dataclass
class Proxy:
    url: str                     # e.g. "http://user:[email protected]:8000"
    country: str
    score: float = 1.0
    cool_until: float = 0.0

@dataclass
class ProxyPool:
    proxies: list[Proxy]
    min_score: float = 0.3
    domain_delay: dict = field(default_factory=dict)   # domain -> seconds between requests
    last_hit: dict = field(default_factory=dict)

    def pick(self, country=None):
        now = time.time()
        healthy = [p for p in self.proxies
                   if p.cool_until <= now and p.score >= self.min_score
                   and (country is None or p.country == country)]
        if not healthy:
            return None
        weights = [p.score for p in healthy]
        return random.choices(healthy, weights=weights, k=1)[0]

    def wait_for_domain(self, domain):
        delay = self.domain_delay.get(domain, 5.0)
        elapsed = time.time() - self.last_hit.get(domain, 0)
        if elapsed < delay:
            time.sleep(delay - elapsed)
        self.last_hit[domain] = time.time()

    def report(self, proxy, ok, blocked=False):
        if ok:
            proxy.score = min(1.0, proxy.score + 0.05)
        else:
            proxy.score = max(0.0, proxy.score - 0.2)
            if blocked:
                proxy.cool_until = time.time() + 15 * 60   # rest for 15 minutes

def fetch(pool, url, domain, country=None):
    for attempt in range(3):
        proxy = pool.pick(country)
        if proxy is None:
            time.sleep(60)
            continue
        pool.wait_for_domain(domain)
        try:
            r = requests.get(url, proxies={"http": proxy.url, "https": proxy.url},
                             headers={"User-Agent": "AcmeResearchBot/1.0 ([email protected])"},
                             timeout=30)
        except requests.RequestException:
            pool.report(proxy, ok=False)
            continue
        if r.status_code in (403, 429):
            pool.report(proxy, ok=False, blocked=True)
            pool.domain_delay[domain] = pool.domain_delay.get(domain, 5.0) * 2   # slow down
            continue
        pool.report(proxy, ok=r.ok)
        return r
    return None

Notice that a block response does two things: it rests the proxy and doubles the delay for that domain. The crawler becomes gentler when a site pushes back, which is both more responsible and, over time, more reliable.

What to monitor

A proxy layer needs its own dashboard. We track:

Success rate per proxy, per provider and per target domain
Block responses (403, 429 and known block pages) per domain
Median and slowest response times
Cost per successful page, which is the number that matters for budgeting
Records extracted per run, so a silent failure upstream is spotted quickly

When the block rate for a domain rises, the first response should be to reduce the request rate and review your access, not to add more IPs.

Common mistakes

Buying a large residential pool for a job a single polite server could handle
Rotating IPs while keeping the same cookies and headers, which defeats the purpose of session handling
Letting each node rate-limit itself, so total load grows with every node added
Retrying immediately after a 429, which makes the problem worse
Not recording which country a data point was collected from

Proxies are one layer of a wider scraping system. For the overall pipeline, from data dictionary to delivery, see our guide to custom web scraping services for market research. For JavaScript-heavy sites at volume, read about architecting headless browser clusters for B2B lead scraping.

Frequently Asked Questions

Do we need rotating proxies for web scraping?

Not always. Small, polite crawls often work fine from one server. Rotating proxies help when you need geo-specific data, large distributed crawls, or managed egress across many scraper nodes.

Are residential proxies legal to use?

Using them is generally lawful, but sourcing practices vary between providers. Choose a provider that explains how device owners consent, and make sure your scraping itself respects site rules and data protection law.

How often should proxies rotate?

It depends on the task. Independent page fetches can rotate per request; multi-page journeys work better with sticky sessions that keep one IP for a short sequence.

What should we do when a site starts blocking requests?

Slow down, review the site's robots.txt and terms, and check whether an official API or data feed exists. Adding more IPs to push past a site's limits is not a responsible fix.

Plan your scraping infrastructure with us

We design scraping systems with responsible rate limits, managed proxy layers and monitoring built in. Explore our B2B lead mining and prospect verification service, or talk to us about your data collection project.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.