B2B Lead Scraping From Public Sources

Lead lists bought from data vendors are often stale and shared with every competitor. Collecting your own B2B leads from public sources, such as association member directories, trade show exhibitor lists and company websites, can produce fresher, better-targeted data. B2B lead scraping is the automated collection of company and contact information from those public pages, and it only works well when it is narrow in scope, respectful of the sites involved, careful with personal data and followed by verification. This guide explains which sources are worth collecting from, how to stay within legal and ethical limits, how to turn raw pages into usable CRM records and how to check quality before anyone sends an email.

Choosing sources for B2B lead scraping

The best sources describe exactly the companies you sell to. Examples include:

• Industry association member directories, which list companies in a specific sector
• Trade show exhibitor lists, which show companies investing in a market
• Public registries and licensing bodies for regulated industries
• Supplier and partner directories published by software vendors or manufacturers
• Company websites, for details such as services offered, locations and team pages

Before collecting anything, write down what you need and why. A firm selling to logistics companies in two countries might need company name, website, city, size indicators and the relevant department head. Collecting less is faster, cheaper and easier to defend.

Staying within the rules

Collecting public data does not mean anything goes. Several layers apply, and this is general information rather than legal advice.

Site terms and robots.txt: read each site's terms of use, and respect its robots.txt rules, which are described in RFC 9309. Many directories restrict automated access or commercial reuse.

Load: collect slowly, during quiet hours where possible, and identify your crawler. A directory run by a small association should never notice your crawler.

Personal data: names, job titles and business email addresses of individuals are personal data under laws such as the UK and EU GDPR. You need a lawful basis, a clear purpose, and to tell people where you got their details when you contact them.

Marketing rules: email marketing rules vary by country and by whether the recipient is a company or an individual. The UK ICO's guidance on electronic mail marketing, for example, says you can send marketing emails to companies, but it is good practice to keep a "do not email" list of companies that object.

When a source's terms forbid collection, look for an official export, a paid membership list or a data partnership instead.

From pages to records

A reliable pipeline separates collection from processing, so problems in one stage do not corrupt the others:

1. Fetch pages politely and store the raw HTML with the URL and date
2. Parse company fields from the stored pages, using selectors you can update when a site changes
3. Normalise names, websites, countries and phone numbers into consistent formats
4. Deduplicate against your CRM by website domain first, then by name and location
5. Enrich only where needed, such as finding the website for a company listed without one
6. Load into the CRM with the source and date on every record

For JavaScript-heavy directories, a headless browser may be needed; our guide to Playwright web scraping covers that approach. For a broader view of scraping projects, see our guide to custom web scraping services.

Verification before outreach

Raw scraped data is a starting point, not a list ready to email. Verify before use:

• Company still trading, checked against its website or a registry
• Website reachable and matching the company name
• Contact still in the role, checked against the company's own site where possible
• Email addresses verified by a validation service, never guessed and sent blind
• Existing customers, open deals and suppressed domains removed

Record the verification date. Data that was accurate six months ago may not be today.

Worked example: collecting exhibitors for a regional trade show

Suppose a packaging supplier wants to reach food manufacturers exhibiting at a regional trade show. The organiser's site lists exhibitors publicly, its robots.txt allows the exhibitor pages, and its terms do not prohibit collection. The supplier plans this pipeline:

StageWhat happensOutputCheckCollectFetch the exhibitor list and each profile page, one request every few seconds, at nightStored HTML for each exhibitorPage count matches the site's exhibitor totalParseExtract company name, website, country and stand categoryStructured rowsRows missing a name or website set asideFilterKeep food and beverage manufacturers in target countriesShortlistCategory mapping reviewed by salesDeduplicateMatch against CRM accounts by website domainNew companies onlyExisting accounts flagged to owners insteadResearchSales assistant finds the operations or purchasing lead on each company's websiteNamed contactsSource page recordedVerifyValidate emails and confirm rolesReady recordsUnverifiable contacts excludedLoadCreate accounts and contacts with source and dateCRM recordsSample of 20 reviewed before outreach

The request rate, categories and sample size are the supplier's own choices. Its first message references the trade show, which explains how the company found the prospect.

Keeping collection jobs healthy

Directories change their layouts, add pagination or move behind logins, and a parser that worked last quarter can quietly return empty or wrong fields. Build simple checks into every run: compare the number of records with the previous run, flag fields that are suddenly blank, and keep a few known companies as reference rows whose values you check each time. When a check fails, stop the load rather than pushing partial data into the CRM. Schedule collection only as often as the source actually changes; an annual exhibitor list needs one careful run, not a weekly crawl. Keep the code and selectors under version control so changes are reviewed.

Common mistakes

• Collecting everything a site lists instead of what you need
• Ignoring terms of use and robots.txt
• Loading unverified data straight into sequences
• Guessing email address patterns and sending without validation
• Losing the source and date, so nobody can answer "where did you get my details?"
• Re-collecting the same site too often

Frequently Asked Questions

What is B2B lead scraping?

It is the automated collection of company and contact details from public business sources, such as directories and exhibitor lists, to build prospect lists.

Is B2B lead scraping legal?

It depends on the site's terms, how you collect, and how you use personal data. Respect robots.txt and terms, have a lawful basis, and take legal advice for your situation.

Should scraped leads be emailed straight away?

No. Verify companies, roles and email addresses, remove existing customers and suppressed domains, and record the source before any outreach.

How do you keep scraped data accurate?

Store the source and collection date on every record, re-verify before each campaign and expire records that have not been verified recently.

Build lead lists you can trust

We design responsible lead collection pipelines with verification and source tracking built in. See our B2B lead mining and prospect verification service, or talk to us about your target market.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.