Scraped Data Validation: Trust What You Collect

Set up scraped data validation: define a schema, check types and ranges, compare batches to past runs, quarantine bad records and alert on sudden changes.

A scraper that runs without errors is not necessarily collecting good data. Sites change their layouts, rename labels, show different content to different regions or return a polite error page with a normal status code. When that happens, a scraper can keep saving records with empty prices, shifted columns or text from the wrong element, and nobody notices until a sales rep calls a wrong number or a report shows an impossible trend. Scraped data validation is the set of checks that catches these problems between extraction and use. This guide explains which checks to run, where to put them in the pipeline and what to do with records that fail.

Why scraped data goes wrong quietly

Most scraping failures are not crashes. Common quiet failures include:

A selector still matches, but now matches a different element, such as a promotional price instead of the list price
A field disappears from some pages, so values come back empty
The site serves a consent page, a login prompt or a block page instead of the content
Number and date formats change, for example when the site switches locale
Duplicate records appear because the same item is listed in several categories

Each of these produces records that look valid at a glance. Only checks that know what good data looks like will catch them.

Define what good data looks like

Start by writing a schema for each record type: the fields, their types, which ones are required and the allowed values or ranges. For a company record, that might be a name that is never empty, a website that parses as a URL, a country from a known list and an employee range from a fixed set of bands. For a product record, it might be a price that is a positive number, a currency code from an allowed list and a product URL on the expected domain.

Write the schema in code so it can be enforced automatically. In Python, libraries such as Pydantic let you describe records with type annotations and validate data against them; its documentation explains that schema validation and serialization are controlled by type annotations (Pydantic documentation). JSON Schema is another option when records are exchanged between systems written in different languages.

Scraped data validation in layers

Good scraped data validation works at three levels:

Field checks: is the value present, of the right type and within the allowed range or format?
Record checks: do the fields make sense together, such as a sale price that is not higher than the list price, or a city that belongs to the stated country?
Batch checks: does this run look like previous runs, in record count, share of empty fields and distribution of key values?

Field and record checks catch individual bad rows. Batch checks catch the problems that affect everything at once, such as a layout change that empties one field on every page. A run where the share of records with no price jumps from almost none to most of them should stop the pipeline, even if each record passes its own checks because price was optional.

Detect block pages and wrong content

Check that each page is the one you expected before extracting from it. Simple signals include the presence of a known element that appears only on real content pages, the page title, the final URL after redirects and the absence of phrases that appear on consent or block pages. Record these signals with each fetch so you can see when a site starts serving something different. If a site begins blocking your requests, slow down or stop and review whether you should be collecting from it at all; our guide to the web scraping pipeline covers polite fetching and the stages around extraction.

Quarantine instead of deleting

When a record fails validation, do not silently drop it and do not let it through. Move it to a quarantine table or file with the reason it failed, the source URL and the time of the fetch. This gives you:

Evidence for fixing the scraper, because you can see exactly what came back
A way to reprocess records once the extraction is fixed
A measure of data quality over time

Set a threshold for each batch. If quarantined records exceed it, stop loading that batch into downstream systems and alert the owner.

Worked example: a hypothetical validation plan

The table below is a hypothetical plan for scraped company listings that will feed a CRM. The fields and thresholds are illustrative, not recommendations for a specific source.

CheckLevelRuleOn failureCompany name presentFieldNot empty after trimmingQuarantine recordWebsite formatFieldParses as a URL with a domainQuarantine recordCountry valueFieldIn the allowed country listQuarantine recordPhone matches countryRecordCountry code consistent with country fieldFlag for reviewDuplicate listingRecordSame domain already seen this runMerge, keep first sourceRecord countBatchWithin the range of recent runsHold batch, alert ownerShare of empty websitesBatchSimilar to recent runsHold batch, alert owner

Validate again before loading into the CRM

The CRM or database that receives scraped data should have its own rules, such as required fields, picklists and duplicate checks. These act as a second line of defence. Our guide to CRM data validation explains how to set those rules at the point of entry. Keep the source and fetch time on every record, so anyone looking at a CRM entry can see where the data came from and how old it is.

Monitor quality over time

Track a few numbers for each source and run: records fetched, records passed, records quarantined, the share of empty values for key fields and the number of block or error pages. Plot them over time. Gradual drift, such as a slowly rising share of missing phone numbers, often signals a partial layout change or a source that is going stale. Review the quarantine regularly and fix the causes rather than letting it grow.

Respect the source and the people in the data

Validation is also a good place to enforce your collection rules. Drop fields you do not need, especially personal data, and check that every source is one you are allowed to use under its terms and the applicable law. Keeping less data makes validation simpler and reduces risk.

Frequently Asked Questions

What is scraped data validation?

It is the set of checks that confirms scraped records are complete, correctly typed and consistent before they are loaded into other systems, plus batch-level checks that compare each run to previous runs.

Why not just let the CRM reject bad data?

CRM rules help, but they cannot spot problems such as a selector picking up the wrong price or a whole batch missing a field. Checks closer to the scraper catch these earlier and with more context.

What should happen to records that fail validation?

Move them to a quarantine area with the reason and source URL, so the scraper can be fixed and the records reprocessed.

How do I detect a site layout change?

Batch checks are the most reliable signal: sudden changes in record counts or in the share of empty fields usually mean the extraction no longer matches the page.

Collect data you can rely on

We design scraping pipelines with schema checks, quarantine and monitoring so bad data stops before it reaches your CRM. See our B2B lead mining and prospect verification service, or talk to us about scraped data validation.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.