Headless Browser Scraping for B2B Data

Many sites that hold useful B2B information, such as directories, marketplaces and company listings, build their pages with JavaScript. A plain HTTP request returns an almost empty page, and the data only appears once a real browser runs the scripts. Headless browser scraping solves this by driving a browser without a visible window, letting it render the page, and extracting the data from the result. It is also slower and far more resource-hungry than simple requests, so running it at volume needs careful design. This guide explains when headless browsers are worth it, how we design clusters that run them reliably and politely, and a worked example in Python with Playwright.

When you need headless browser scraping, and when you do not

Use the lightest tool that works. Before reaching for a browser, check whether:

• The data is available through an official API or a downloadable dataset
• The page's data is loaded from a JSON endpoint you are permitted to call directly
• The HTML returned by a simple request already contains what you need

A headless browser is the right choice when content only exists after scripts run, when the site requires interactions such as clicking "load more", or when you must see the page exactly as a user would. Every page rendered in a browser costs many times more CPU, memory and time than a simple request, so reserve it for the pages that need it.

Responsible scraping comes first

Scraping at scale affects the sites you visit, and it carries legal and ethical obligations:

• Respect robots.txt rules, as defined in RFC 9309, and the site's terms of use
• Keep request rates low and spread out, so you never degrade a site's service
• Collect only business information you have a legitimate reason to use, and handle any personal data in line with the privacy laws that apply to you
• Identify a contact point for site owners and stop promptly if asked

These rules shape the architecture as much as any performance concern.

How a headless browser scraping cluster is organised

ComponentRoleDesign notesURL queueHolds pages to visit, with priority and retry countsA database table or message queue; de-duplicate URLsSchedulerReleases URLs per domain at a controlled ratePer-domain limits matter more than global throughputBrowser workersRun headless browsers and render pagesReuse browsers, create a fresh context per jobExtractorsTurn rendered pages into structured recordsVersioned per site; validate every fieldStorageRaw snapshots and cleaned recordsKeep raw HTML briefly for debuggingMonitoringSuccess rates, timings, blocks, data qualityAlert on sudden changes per site

Reuse browsers, isolate jobs

Starting a browser is expensive. Each worker keeps a browser process running and opens a new, isolated context for each job, then closes it. That keeps cookies and storage separate while avoiding constant start-up costs.

Block what you do not need

Images, fonts and media rarely matter for data extraction. Blocking them reduces bandwidth for you and load for the site.

Set strict timeouts

Pages that never finish loading will tie up workers. Every navigation and wait needs a timeout, and failed jobs go back to the queue with a retry limit.

Control rate per domain

A cluster can easily overwhelm a single site. The scheduler should enforce a maximum number of concurrent pages and a minimum delay per domain, adjusting downwards when errors rise.

Worked example: a polite Playwright worker

The illustrative worker below checks robots.txt, limits concurrency, blocks heavy resources and extracts company cards from a rendered listing page. It uses Playwright for Python. Selectors and URLs are placeholders.

import asyncio, re
from urllib import robotparser
from urllib.parse import urlparse
from playwright.async_api import async_playwright

USER_AGENT = "ExampleResearchBot/1.0 (contact: [email protected])"
MAX_CONCURRENT = 2          # per domain; keep this low
DELAY_SECONDS = 5
BLOCKED = {"image", "font", "media"}
_robots = {}

def allowed(url):
    host = urlparse(url).scheme + "://" + urlparse(url).netloc
    if host not in _robots:
        rp = robotparser.RobotFileParser(host + "/robots.txt"); rp.read(); _robots[host] = rp
    return _robots[host].can_fetch(USER_AGENT, url)

async def block_heavy(route):
    if route.request.resource_type in BLOCKED:
        await route.abort()
    else:
        await route.continue_()

async def scrape(browser, url, sem):
    if not allowed(url):
        return {"url": url, "skipped": "robots.txt"}
    async with sem:
        context = await browser.new_context(user_agent=USER_AGENT)
        await context.route(re.compile(r".*"), block_heavy)
        page = await context.new_page()
        try:
            await page.goto(url, wait_until="domcontentloaded", timeout=30000)
            await page.wait_for_selector(".company-card", timeout=15000)
            cards = await page.eval_on_selector_all(".company-card", """els => els.map(e => ({
                name: e.querySelector('.name')?.innerText?.trim(),
                industry: e.querySelector('.industry')?.innerText?.trim(),
                website: e.querySelector('a.website')?.href }))""")
            return {"url": url, "records": cards}
        finally:
            await context.close()
            await asyncio.sleep(DELAY_SECONDS)

async def main(urls):
    sem = asyncio.Semaphore(MAX_CONCURRENT)
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        results = await asyncio.gather(*(scrape(browser, u, sem) for u in urls))
        await browser.close()
    return results

In production, the scheduler feeds URLs from a queue, results go to storage with validation, and failures are retried with backoff. The contact address in the user agent should be real.

Validating the extracted data

Rendered pages change without notice. Validate every record before it enters your CRM or database:

• Required fields present, such as company name and website
• Websites normalised to a domain and checked for format
• Duplicates merged on domain, not on name alone
• Each record tagged with source URL and collection date

A sudden drop in valid records from one site usually means its layout changed, and the extractor needs updating.

Scaling the cluster

Scale by adding workers, not by raising per-domain limits. Run workers in containers with fixed memory limits so a runaway page cannot take down a host, and restart browsers periodically to avoid memory growth. Spread jobs across many domains so that each individual site sees only light traffic. Proxies are sometimes used to distribute requests, but they do not change your obligations to the sites you visit; our guide to rotating proxies for web scraping covers when and how to use them responsibly.

Handling logins, pagination and infinite scroll

Some listings load more results as you scroll or click. Script these interactions explicitly: scroll or click, wait for new items to appear, and stop at a sensible page limit. Never collect content that sits behind a login unless you have the right to access it and the site's terms allow automated use. For paginated listings, prefer following page links over simulated scrolling, because links are easier to resume after a failure and easier to schedule politely.

Keeping extractors maintainable

Write one extractor per site, keep selectors in configuration where practical, and store a few sample pages with expected outputs as tests. When a site changes, update the extractor, add the new sample, and rerun the tests before deploying.

What to monitor

• Pages attempted, succeeded, blocked and timed out, per site
• Median and slowest render times
• Records extracted per page and validation failure rates
• Worker memory and CPU use
• Robots.txt changes for the sites you collect from

Keeping costs under control

Headless rendering is the most expensive part of most scraping systems. Cache pages that change rarely, revisit listings on a sensible schedule rather than constantly, and only render detail pages when the listing data shows something new. Measuring cost per valid record, rather than cost per page, keeps attention on what matters.

For the wider picture of turning collected data into usable research, see our guide to custom web scraping for market research.

Frequently Asked Questions

What is headless browser scraping?

It is collecting data by driving a real browser without a visible window, so pages that depend on JavaScript are fully rendered before data is extracted.

Is headless browser scraping slower than normal scraping?

Yes. Rendering pages uses far more CPU, memory and time than simple HTTP requests, so use it only for pages that need it.

Which tool should we use for headless browsers?

Playwright and Puppeteer are common choices. Playwright supports several browsers and languages, including Python.

How do we scrape without harming websites?

Respect robots.txt and terms of use, keep per-site rates low, block unnecessary resources, identify yourself and stop when asked.

Plan your data collection with us

We build responsible scraping systems, including headless browser clusters with rate control, validation and monitoring. See our B2B lead mining and prospect verification service, or talk to us about your data collection project.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.