Back to blog

Guide

How to Rotate IPs for Crawling Across Many Domains

Learn how to rotate IPs for crawling with the right sessions, cadence, targeting, and monitoring to reduce blocks and collect reliable web data at scale.

Max Krause · Head of Growth at FlameProxies8 min read

Crawling and scraping have different rotation problems. Scraping hits a handful of known targets whose tolerance you can measure individually. Crawling follows links across a long tail of domains you have never seen, each with its own unknown limits, discovered as you go. You cannot hand-tune rotation per domain when the domain set is discovered at runtime and numbers in the thousands.

Rotation for crawling therefore has to be per-domain, adaptive, and stateful — which is a meaningfully different architecture from a rotating proxy pointed at a scraper.

Rotation State Belongs to the Domain, Not the Job

The core design decision: maintain rotation state keyed by domain, not globally.

A global rotation policy applied across a crawl treats a permissive blog and a heavily defended marketplace identically. One gets throttled unnecessarily; the other gets blocked. Per-domain state lets each converge on its own correct behavior.

from dataclasses import dataclass, field
import time, uuid
 
@dataclass
class DomainState:
    domain: str
    budget: int = 20          # requests before rotating session
    used: int = 0
    session: str = field(default_factory=lambda: uuid.uuid4().hex[:12])
    delay: float = 2.0        # seconds between requests
    consecutive_ok: int = 0
    blocked_until: float = 0.0
 
    def rotate(self):
        self.session = uuid.uuid4().hex[:12]
        self.used = 0
 
    def on_success(self):
        self.used += 1
        self.consecutive_ok += 1
        if self.used >= self.budget:
            self.rotate()
        if self.consecutive_ok >= 50:
            self.budget = min(200, int(self.budget * 1.25))
            self.delay = max(0.5, self.delay * 0.9)
            self.consecutive_ok = 0
 
    def on_block(self):
        self.budget = max(3, self.budget // 2)
        self.delay = min(30.0, self.delay * 2)
        self.consecutive_ok = 0
        self.rotate()
        self.blocked_until = time.time() + 60

Start every unknown domain conservatively, expand on sustained success, contract sharply on blocks. Over a long crawl each domain converges toward its actual tolerance without anyone configuring it.

Rotate on a Budget, Not a Timer

Interval-based rotation — a new IP every five minutes — is wrong for crawling because request rate per domain varies enormously. Five minutes against a domain you are hitting twice is wasteful; against one you are hitting two hundred times it is far too slow.

Rotate on a request budget per domain: retire the session after N requests against that domain, where N adapts as above. This keeps per-IP history bounded regardless of how fast you are moving through a given host.

Note that one proxy session can legitimately serve several domains concurrently. What matters is requests-per-IP-per-domain, since that is what each target counts. Tracking the budget per (session, domain) pair rather than per session alone is more accurate, though per-domain state with fresh sessions is usually close enough and much simpler.

The Frontier Needs to Be Domain-Aware

A naive URL frontier produces bursts: a crawl discovers 500 links on one domain and queues them consecutively, so your workers hammer that host while other domains sit idle. This creates blocks on a target that would have been fine at a reasonable pace.

Interleave by domain instead:

from collections import defaultdict, deque
import heapq, time
 
class Frontier:
    def __init__(self):
        self.queues = defaultdict(deque)
        self.ready = []            # heap of (next_allowed_time, domain)
        self.seen = set()
 
    def add(self, url, domain):
        if url in self.seen:
            return
        self.seen.add(url)
        if not self.queues[domain]:
            heapq.heappush(self.ready, (time.time(), domain))
        self.queues[domain].append(url)
 
    def next(self, states):
        while self.ready:
            when, domain = heapq.heappop(self.ready)
            st = states[domain]
            earliest = max(when, st.blocked_until)
            if earliest > time.time():
                heapq.heappush(self.ready, (earliest, domain))
                return None                      # nothing ready yet
            url = self.queues[domain].popleft()
            if self.queues[domain]:
                heapq.heappush(
                    self.ready, (time.time() + st.delay, domain)
                )
            return url, domain
        return None

This gives you politeness per host and parallelism across hosts simultaneously — which is exactly the shape you want. Many domains in flight, each progressing gently.

Geographic Targeting in a Crawl

Most crawling does not need geographic precision, and requesting it narrows your usable pool for no benefit. Default to no country targeting, or to a broad region group, and reserve specific country targeting for the subset of domains where location changes the content.

Where it does matter — regional ecommerce, localized news, market-specific catalogs — pin the country per domain and keep it stable. A domain crawled from Germany on one pass and Brazil on the next produces inconsistent content attributed to the same source.

Region groups are the right abstraction here. Expressing coverage as EU or LATAM rather than maintaining a country list avoids pushing narrow-pool constraints onto work that does not need the precision.

Politeness Is Not Optional

Crawling generates load on sites that did not ask for it, many of them small. Rotation that lets you exceed what a site can comfortably serve is not a win.

Honor robots.txt. Fetch and cache it per domain, respect Disallow, and honor Crawl-delay where present — treating it as a floor on your per-domain delay, not a suggestion.

def min_delay_for(domain, robots, default=2.0):
    cd = robots.crawl_delay("*")
    return max(default, cd) if cd else default

Honor Retry-After. When a server tells you when to come back, comply.

Keep concurrent connections per host low. One or two. Parallelism belongs across domains, not within one.

Identify yourself consistently if your crawl is the kind that should be identifiable, and provide a way to be contacted. Many site operators will whitelist a well-behaved, identifiable crawler rather than block it.

Cut off domains that block you. A circuit breaker per domain, retiring hosts that consistently refuse, protects both the target and your bandwidth. Chasing a domain that does not want to be crawled is expensive and pointless.

Distribute Across Networks, Not Just Addresses

Many sites aggregate reputation at subnet or ASN level. A crawl rotating through thousands of addresses concentrated in a few autonomous systems presents as a small number of identities, and blocks arrive sooner than the address count suggests.

Track success rate per ASN alongside per domain. When failures concentrate in specific networks — and they usually do — steer away from them. ASN targeting makes this actionable rather than something you can only observe.

Monitoring a Crawl

Aggregate metrics hide everything in crawling, because the domain set is heterogeneous by nature. Instrument per domain:

  • Valid response rate per domain, with a circuit breaker threshold
  • Current adapted budget and delay per domain — a domain whose budget has collapsed to the floor is telling you something
  • Blocks per domain over time — rising on a previously fine domain means it tightened
  • Per-ASN success rate across the crawl
  • Domains retired by circuit breaker — review these periodically; some can be re-enabled later
  • Bytes per retained document, since crawls discard a lot of what they fetch

Keep a sampled body prefix on failures. With thousands of domains nobody is watching individually, that sample is what tells you whether a domain started returning challenge pages or simply restructured its HTML.

Reducing What You Fetch

Crawls fetch a great deal they ultimately discard, and everything discarded was paid for.

Normalize and deduplicate URLs before queuing — strip tracking parameters, resolve relative paths, unify trailing slashes. Paginated and print variants of the same document are a large source of waste.

Content-hash early to catch near-duplicates that survived URL normalization.

Skip assets. Most crawls want text. Block images, media, and fonts.

Use conditional requests on recrawls. If-None-Match and If-Modified-Since turn most recrawl traffic into cheap 304s.

Cap spend per domain. A ceiling prevents one pathological host — infinite pagination, generated calendar URLs — from consuming a disproportionate share before anyone notices.

The Concurrency Requirement

Everything above describes the same shape: many domains in flight simultaneously, each crawled gently. That is both the polite pattern and the fast one, because throughput comes from breadth rather than from pressure on any single host.

It also depends entirely on being able to run many concurrent sessions. A crawl touching 500 domains with a two-second per-domain delay needs hundreds of sessions in flight to make progress. Capped at a few dozen, you either crawl far fewer domains at once or raise per-domain rates into blocking territory. The cap converts a polite wide crawl into an impolite narrow one.

FlameProxies applies no concurrency cap and no request-rate limits, with 80M+ residential IPs across 180+ countries plus ASN and region targeting — so crawl breadth is set by your frontier and your own infrastructure rather than by a thread allowance, and per-domain politeness does not cost you total throughput.