Custom Web Scraping Services for Market Research

A lot of market research is still done by hand: someone opens competitor websites, copies prices into a spreadsheet, checks a few marketplaces, and repeats the job next month. It is slow, it covers only a sample, and by the time the spreadsheet is finished, some of it is out of date. Custom web scraping services replace that routine with Python scrapers that collect the data on a schedule, clean it and enrich it so it is ready for analysis. This guide covers what a custom build involves, how to keep it legal and polite, a worked example in Python, and how to decide whether to build in-house or use a service.

What market research data can be collected automatically

The most common sources we see in B2B and e-commerce research are:

• Competitor product catalogues, prices and stock status
• Marketplace listings, sellers and ratings
• Customer reviews and ratings over time
• Public business directories and association member lists
• Job postings, as a signal of which companies are growing or adopting a technology
• Public tenders and procurement notices

The common thread is publicly available information that changes often enough to make manual collection painful.

Start with a data dictionary, not a scraper

Before writing any code, define exactly what you need. A short data dictionary keeps the project focused and makes validation possible later.

Illustrative data dictionary for a pricing study

FieldTypeExampleRulesourcetextcompetitor-a.exampleRequiredproduct_nametextStainless Steel Valve 25mmRequired, trimmedskutextSSV-25Optionalpricedecimal1499.00Required, greater than 0currencytext (ISO code)INRRequiredin_stockbooleantrueRequiredurltexthttps://competitor-a.example/p/ssv-25Required, unique per runscraped_attimestamp (UTC)2026-10-08T06:30:00ZSet by the scraper

If a field cannot be validated, it will eventually be wrong without anyone noticing.

Keep it legal and polite

Responsible scraping protects both you and the sites you collect from. Our baseline rules on every project are:

• Check and respect robots.txt. The Robots Exclusion Protocol is standardised in RFC 9309, and Python's standard library includes a robots.txt parser.
• Read the site's terms of use, and prefer an official API or data feed when one exists.
• Do not bypass logins, paywalls or technical access controls.
• Limit request rates so your scraper never affects the site's performance.
• Be careful with personal data. Collecting information about individuals brings obligations under laws such as the EU's GDPR and India's Digital Personal Data Protection Act. If personal data is involved, get legal advice first.

This is general guidance, not legal advice. Each project's sources and use case should be reviewed on their own merits.

How a custom scraping pipeline is structured

We design every pipeline as five stages, so each one can be tested and fixed independently:

1. Fetch: download pages politely, with retries and rate limits
2. Parse: extract fields from the HTML or from the JSON a page loads
3. Validate: check every record against the data dictionary
4. Enrich: normalise, deduplicate and add context
5. Deliver: send clean data to a spreadsheet, database, dashboard or CRM

For simple, server-rendered pages, requests and BeautifulSoup are enough. For pages that build their content with JavaScript, a headless browser such as Playwright is needed. For larger crawls across many pages, Scrapy provides scheduling, retries and throttling out of the box.

Worked example: a polite Python scraper for product prices

The illustrative script below collects product names, prices and stock status from a listing page on a placeholder site, checks robots.txt first, waits between requests, validates each record, and writes a CSV. The CSS selectors are placeholders; real sites need their own.

import csv, time
from datetime import datetime, timezone
from decimal import Decimal, InvalidOperation
from urllib import robotparser
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

BASE = "https://competitor-a.example"
USER_AGENT = "AcmeResearchBot/1.0 (contact: [email protected])"
DELAY_SECONDS = 5

robots = robotparser.RobotFileParser(urljoin(BASE, "/robots.txt"))
robots.read()

def fetch(url):
    if not robots.can_fetch(USER_AGENT, url):
        raise PermissionError(f"Disallowed by robots.txt: {url}")
    r = requests.get(url, headers={"User-Agent": USER_AGENT}, timeout=30)
    r.raise_for_status()
    time.sleep(DELAY_SECONDS)
    return r.text

def parse_price(text):
    try:
        return Decimal(text.replace("₹", "").replace(",", "").strip())
    except InvalidOperation:
        return None

def scrape_listing(path):
    soup = BeautifulSoup(fetch(urljoin(BASE, path)), "html.parser")
    for card in soup.select("div.product-card"):
        name = card.select_one(".product-title")
        price = card.select_one(".price")
        link = card.select_one("a")
        record = {
            "source": "competitor-a.example",
            "product_name": name.get_text(strip=True) if name else None,
            "price": parse_price(price.get_text()) if price else None,
            "currency": "INR",
            "in_stock": card.select_one(".out-of-stock") is None,
            "url": urljoin(BASE, link["href"]) if link else None,
            "scraped_at": datetime.now(timezone.utc).isoformat(),
        }
        if record["product_name"] and record["price"] and record["price"] > 0 and record["url"]:
            yield record
        else:
            print("Rejected record:", record)   # send to a review log in production

with open("prices.csv", "w", newline="", encoding="utf-8") as f:
    writer = None
    for row in scrape_listing("/category/valves"):
        if writer is None:
            writer = csv.DictWriter(f, fieldnames=row.keys())
            writer.writeheader()
        writer.writerow(row)

The User-Agent identifies the bot and gives a contact address, which is good practice. Rejected records are logged rather than silently dropped, so a layout change shows up as a spike in rejections.

Scaling up with Scrapy

When a project grows to many pages or sources, we move to Scrapy. These settings keep a crawl polite by default, using Scrapy's AutoThrottle extension to adapt the request rate to the site's response times.

# settings.py (excerpt)
ROBOTSTXT_OBEY = True
USER_AGENT = "AcmeResearchBot/1.0 (contact: [email protected])"
CONCURRENT_REQUESTS_PER_DOMAIN = 2
DOWNLOAD_DELAY = 3
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 5
AUTOTHROTTLE_MAX_DELAY = 60
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
RETRY_TIMES = 3
HTTPCACHE_ENABLED = True   # avoid re-downloading unchanged pages during development

Enrichment: turning raw data into research

Raw scraped data is rarely ready to analyse. The enrichment stage is where most of the value is added:

• Normalising currencies, units and product names
• Matching competitor products to your own catalogue, by SKU where possible and by fuzzy name matching with human review where not
• Removing duplicates across sources and runs
• Adding categories, regions or company size from other datasets
• Linking companies to accounts already in your CRM

For a deeper look at the enrichment side, see our article on automated CRM enrichment scripts that replace manual data hygiene.

Delivering the data where your team works

Research data is only useful if people actually look at it. We agree the delivery format at the start, based on who uses the data and how often:

• A shared spreadsheet or Google Sheet for small studies and one-off projects
• A database table that analysts can query, with one row per product per run
• A dashboard in your BI tool showing price movements, stock changes and new entrants
• Records in your CRM, so sales teams see competitor or market signals on the accounts they own

Whatever the format, keep the history. A single snapshot tells you today's prices; a time series tells you how competitors behave.

Keeping scrapers running

Websites change, and scrapers break when they do. Plan for it:

• Alert when a run returns zero records, or far fewer than usual
• Validate every record and track the rejection rate per source
• Keep raw HTML for a short period so failures can be diagnosed
• Review selectors on a schedule for business-critical sources

At larger scale, network design also matters. Our guide on automating proxy rotation across distributed scraper nodes covers when and how to distribute requests responsibly.

Build in-house or use custom web scraping services?

Build in-house if you have Python developers with time to maintain scrapers, a small number of stable sources, and no urgent deadline. Use a service when you need many sources, JavaScript-heavy sites, scheduled delivery with monitoring, or enrichment and CRM integration that your team does not have time to build. A good provider should explain how they handle robots.txt, rate limits and personal data before quoting.

Frequently Asked Questions

Is web scraping legal?

It depends on the data, the source and how you use it. Respecting robots.txt and terms of use, avoiding access controls, limiting request rates and taking care with personal data reduce risk. Get legal advice for anything involving personal data.

How often should market data be refreshed?

Match the schedule to how fast the data changes and how you use it. Prices on fast-moving marketplaces may justify daily runs; directories and catalogues often need only weekly or monthly updates.

What happens when a website changes its layout?

The scraper stops finding fields, which is why validation and alerts matter. A well-monitored pipeline flags the drop on the first failed run so selectors can be updated quickly.

Can scraped data go straight into our CRM?

Yes, after validation and deduplication. We usually match records to existing accounts first, so new data enriches what you have rather than creating duplicates.

See what a custom build covers

We build Python scrapers, enrichment pipelines and CRM integrations for market research and lead data. Explore our B2B lead mining and prospect verification service, or tell us what data you need and we will scope it.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.