Large-Scale Data Collection Example: E-Commerce Price Tracking

Walking through a concrete large-scale data collection example is more useful than abstract guidance for teams designing or evaluating their own pipeline. This post describes a realistic e-commerce price tracking operation: what the targets are, how proxy routing is configured, how the collection and quality layers work, what compliance practices are applied, and what the output looks like.

The architecture is representative of how production price monitoring operations are actually built — not a proof of concept, but a design that would run reliably at millions of records per day.

The Use Case: Retail Price Monitoring at Scale

The goal is to track product prices across five major e-commerce platforms in three markets (US, UK, Germany) for a catalog of approximately 50,000 SKUs per market, refreshed twice daily. That is 300,000 product page loads per refresh cycle, or 600,000 per day. At an average page weight of 1.5 MB, this consumes roughly 900 GB of bandwidth per day.

The output is a structured dataset: SKU ID, platform, market, price, currency, availability status, promotional flag, and timestamp. The dataset feeds a pricing analytics dashboard used by a retail strategy team.

Target Classification

The five target platforms vary in their anti-bot sophistication. Before building the collection pipeline, each target is classified into one of three tiers:

Tier 1 (permissive): Two of the five platforms — smaller marketplaces — return product page HTML without JavaScript rendering and apply light rate limiting. Datacenter proxies work cleanly with a moderate request rate.

Tier 2 (moderate): Two platforms apply IP type filtering but are accessible via residential proxies at reasonable request rates. JavaScript rendering is required for accurate price extraction on one of the two.

Tier 3 (strict): One platform has invested significantly in anti-bot infrastructure. It evaluates IP reputation, browser fingerprint, JavaScript execution behavior, and navigation patterns simultaneously. Residential proxies with careful behavioral configuration are required. The request rate must be conservative to avoid elevated challenge rates.

This classification drives proxy type selection and request rate configuration per target. Using residential bandwidth for Tier 1 targets wastes money. Using datacenter proxies on Tier 3 produces a failure rate that makes collection impractical.

Proxy Routing Configuration

The proxy layer uses a gateway endpoint from a residential proxy provider with separate datacenter bandwidth available.

Tier 1 targets: Datacenter proxies. Rotating per-request, concurrency up to 200 threads per target, 200 ms average interval between requests per thread. Cost: ~$0.80/GB.

Tier 2 targets: Rotating residential proxies. Country-targeted to US, UK, or Germany respectively. Per-request rotation for static page targets. Sticky sessions (10-minute duration) for the JavaScript-rendered target that benefits from session continuity. Concurrency 50 threads per target per market. Cost: ~$5/GB.

Tier 3 target: Rotating residential proxies with behavioral scraping configuration — full headless browser, stealth mode, randomized request intervals (3–8 seconds), sticky sessions for complete page navigation sequences. Concurrency limited to 20 threads for this target. Cost: ~$8/GB including the headless browser overhead.

Total estimated proxy cost: roughly $1,500–2,000/day at full scale — the largest single operational cost in the pipeline.

Collection Architecture

The collection pipeline has four stages.

Job scheduling: A scheduler generates a prioritized queue of (SKU, platform, market) collection jobs, weighting by last-collected time and prioritizing products where price volatility has historically been high. This ensures the most commercially important prices are refreshed most frequently within the twice-daily cycle.

Request execution: Worker processes pull jobs from the queue, configure the appropriate proxy type and session parameters, and execute page loads. Each worker handles one request at a time, manages retries, and logs response metadata (status code, response time, content length, proxy session ID).

Parsing and validation: Successful responses pass through platform-specific parsers that extract price, currency, availability, and promotional flag from the page HTML or rendered DOM. Each parsed record passes validation rules: price must be a positive number, currency must match the market, availability must be a recognized value. Records failing validation are flagged for review rather than discarded, in case the parser needs updating.

Storage: Validated records are written to a time-series data store. Each record includes the collection timestamp, allowing price history to be reconstructed and changes detected. Failed and flagged records are written to a separate table for investigation.

Quality Control

Several quality signals are monitored continuously:

Per-platform success rate: Percentage of collection jobs that produce a valid, parsed record. Alarm threshold: below 92% on any platform over a 30-minute window. A sudden drop signals either an anti-bot response, a site change that breaks the parser, or a proxy pool issue.

Parser accuracy spot checks: A daily random sample of 200 records per platform is manually verified against the live page. This detects parser drift caused by gradual site changes that do not trigger immediate failure.

Price anomaly detection: Records where the price change from the previous collection exceeds 30% are flagged for review. Most are genuine promotions or restocking events; some reveal parsing errors that slipped past validation.

Response latency monitoring: p95 response time per platform, per proxy type. Sustained latency increases can indicate gateway congestion or proxy pool degradation.

Compliance Practices

This operation collects publicly displayed pricing data — the same information any visitor to the site would see. The collection design reflects several compliance principles:

No authentication bypass: The pipeline accesses only pages that are publicly accessible without login. Pricing behind login walls, account-specific pricing, or cart-based pricing is out of scope.

Robots.txt compliance for applicable paths: The scheduler checks robots.txt directives for the URL paths being collected and excludes disallowed paths. Product page pricing is typically not disallowed, but the check is automated.

Rate limiting: The request rates above are set well below the point at which the collection would affect site performance for genuine users. The Tier 3 target's 20 concurrent threads at 3–8 second intervals represents a minimal fraction of its traffic.

No personal data collection: The dataset contains only pricing, availability, and product identifiers — no user data, reviews, or other personal information.

Output and Delivery

The final dataset is delivered as a daily export in Parquet format, partitioned by platform and market. The analytics team queries it through a SQL-compatible interface that allows aggregation across markets, time-series analysis, and anomaly surfacing.

A daily data quality report accompanies each export: collection success rate per platform, number of anomaly-flagged records, parser accuracy from spot checks, and any known issues from the previous 24 hours.

This architecture — target classification, tiered proxy routing, quality monitoring, and compliance-first design — is transferable to a wide range of large-scale collection use cases beyond price tracking. The same pattern applies to product availability monitoring, review aggregation, job posting tracking, and any other high-volume structured data collection that requires reliable operation over time. FlameProxies provides the residential and datacenter proxy infrastructure this type of pipeline depends on, with the geographic targeting, concurrency support, and API access that production-scale collection requires.