Back to blog

Guide

Web Data Collection Checklist: Run Every Job Accountably

Use this web data collection checklist to define scope, select proxies, control request rates, validate quality, and keep every collection run accountable.

7 min read

Structured checklists prevent the category of errors that come from moving too fast: wrong proxy type for the target, missing quality validation, undefined retry behavior, no compliance review before a new target is added. This checklist covers the steps that matter before, during, and after a web data collection run, organized so that each section can be reviewed independently by the person responsible for it.

Before the Run: Scope Definition

Target inventory

  • List every URL pattern or domain being collected in this run
  • Confirm the data type for each target: structured page data, search results, pricing, listings, or other
  • Note the expected response format for each target: HTML, JSON API, rendered JavaScript

Compliance review

  • Review robots.txt for each domain and confirm the collection paths are not disallowed
  • Confirm the data being collected is publicly accessible without authentication
  • Confirm no personal data (user profiles, reviews with identifying information, contact details) is in scope
  • If collecting from a jurisdiction with specific data regulations (EU, California), confirm the data type is covered by your legal review
  • If this is a new target not previously collected, confirm legal has reviewed it or it falls within pre-approved categories

Data scope definition

  • Define exactly which fields are being extracted per target
  • Define the geographic markets being collected (countries, cities, or regions)
  • Define the collection frequency: one-time, hourly, daily, or other
  • Define the time window for this run: when it starts, when it ends, what triggers a stop

Before the Run: Proxy Configuration

Proxy type selection

  • For each target, select the appropriate proxy type based on anti-bot sophistication:
    • Permissive targets (no IP filtering): datacenter proxies acceptable
    • Moderate targets (IP-type filtering): rotating residential required
    • Strict targets (behavioral analysis): rotating residential with browser automation required
  • Confirm you have sufficient bandwidth provisioned for the estimated run volume (record count × average page size)

Geographic targeting

  • Configure country and city targeting parameters for each market in scope
  • Verify geo-accuracy for each target market using a test request: confirm the IP geolocates to the correct city before the run

Session configuration

  • Determine whether sticky sessions are required (multi-step workflows, login-required pages) or whether per-request rotation is appropriate
  • If using sticky sessions, set session duration appropriate to workflow length (see workflow duration guide)
  • Confirm session ID naming convention is consistent with logging format

Credentials and access

  • Confirm proxy credentials are current and have not expired
  • Confirm bandwidth allocation is sufficient and won't hit account limits during the run
  • Confirm gateway endpoint and port are correct for the proxy type being used

Before the Run: Request Configuration

Rate limiting

  • Set request rate per target based on target tier (permissive: higher; strict: conservative)
  • Configure per-thread request intervals — do not leave this at maximum throughput without a deliberate rate limit
  • If the same domain is being collected across multiple geographic sessions, confirm the combined request rate across all sessions is within safe limits

Retry behavior

  • Define retry count and backoff strategy for network errors
  • Define IP rotation behavior on block signals (4xx from target): rotate session, do not retry on the same IP
  • Define the threshold for pausing or stopping a job if block rate exceeds a limit (e.g., >15% failure rate on a target)
  • Confirm that captcha responses are handled explicitly — not silently stored as valid data

Headers and browser configuration

  • Set realistic User-Agent strings; avoid default library user agents
  • For browser automation jobs, confirm browser fingerprint configuration is realistic (window size, language, timezone matching proxy geography)
  • Confirm JavaScript execution is enabled for targets that require it (dynamic pricing, rendered ads)

During the Run: Monitoring

Success rate tracking

  • Monitor per-target success rate (2xx response with expected content) in real time
  • Set alarm threshold: if success rate drops below target (e.g., 90%), investigate immediately
  • Monitor for unexpected response types: a sudden increase in 403s, redirects, or captcha pages indicates a proxy or behavior issue

Latency tracking

  • Monitor p95 response time per target
  • Investigate sudden latency increases — they can indicate proxy pool congestion or target-side throttling

Bandwidth consumption

  • Track bandwidth consumed against estimate
  • Investigate significant over-consumption: pages larger than expected may indicate a configuration issue (JavaScript rendering enabled on a target that does not require it)

Parser health

  • Monitor for records where required fields are null or in unexpected format — this indicates a parser failure or target page change
  • Set a threshold for parser failure rate that triggers a job pause for investigation

After the Run: Quality Validation

Record completeness

  • Confirm the total record count matches the expected count for the target catalog and geographic scope
  • Identify missing records: which SKUs, URLs, or markets are absent, and why

Field validation

  • Validate that each required field is populated in each record
  • Validate field formats: prices are numeric, dates are parseable, URLs are valid
  • Flag records outside expected value ranges for manual review (prices that are zero, negative, or implausibly high)

Spot checks

  • Select a random sample of records (minimum 50 per target, or 1% of volume, whichever is larger)
  • Manually verify each sampled record against the live page it was collected from
  • Compare parser output to live page values field by field — discrepancies indicate parser drift

Anomaly review

  • Identify records where values changed significantly from the previous collection (>30% price change, availability flip)
  • Confirm each anomaly is a genuine data change, not a collection or parsing error

After the Run: Documentation

Run record

  • Log the run parameters: targets, proxy type, geographic scope, start and end time, total records collected
  • Log success rate and parser accuracy results
  • Log any incidents: blocks, unexpected target changes, parser failures, proxy issues

Issue tracking

  • Create tickets for any parser failures or target changes that need investigation
  • Note any targets where block rate was elevated — investigate proxy configuration or request behavior before the next run
  • Update the target tier classification if a target's anti-bot sophistication changed

A checklist works only if it is used. Integrating these steps into your team's run process — as a pre-run review, a monitoring dashboard, and a post-run report — turns them from a reference document into an operational standard. FlameProxies provides the proxy infrastructure component: residential and datacenter pools with geographic targeting, session management, and API access for the programmatic configuration that production collection pipelines require. The rest of the checklist is on your team.