Back to blog

Guide

Optimize Concurrent Proxy Requests for Throughput

Learn how to optimize concurrent proxy requests with practical limits, rotation, retries, and monitoring for faster, more reliable data workflows daily.

Max Krause · Head of Growth at FlameProxies8 min read

Concurrency tuning is where most collection pipelines leave throughput on the table. Teams pick a worker count that seemed reasonable, never revisit it, and end up either saturating a bottleneck they cannot see or running far below what their infrastructure could sustain. Both are measurable, and the measurement is straightforward.

This is a performance-engineering view of concurrency, separate from block avoidance.

The Relationship You Are Tuning

Throughput, concurrency, and latency are related by Little's Law:

throughput = concurrency / average_latency

At 800ms average latency, one worker produces 1.25 requests per second. Fifty workers produce 62.5 — if nothing else constrains you.

Two implications worth internalizing:

Latency reduction and concurrency increase are interchangeable for throughput purposes. Cutting latency in half doubles throughput exactly as doubling workers does. Blocking assets, avoiding unnecessary rendering, and using conditional requests all reduce latency and are usually cheaper than adding parallelism.

Throughput stops scaling when something saturates. Finding which thing saturates first is the whole exercise.

Find Your Saturation Point

Ramp and measure. Do not guess.

import asyncio, time, httpx
 
async def measure(proxy, url, concurrency, total=400):
    sem = asyncio.Semaphore(concurrency)
    lat, errs = [], 0
    start = time.perf_counter()
 
    async def one(client):
        nonlocal errs
        async with sem:
            t = time.perf_counter()
            try:
                r = await client.get(url)
                if r.status_code == 200:
                    lat.append(time.perf_counter() - t)
                else:
                    errs += 1
            except Exception:
                errs += 1
 
    limits = httpx.Limits(max_connections=concurrency + 20,
                          max_keepalive_connections=concurrency + 20)
    async with httpx.AsyncClient(proxy=proxy, limits=limits,
                                 timeout=45) as c:
        await asyncio.gather(*(one(c) for _ in range(total)))
 
    wall = time.perf_counter() - start
    lat.sort()
    return {
        "concurrency": concurrency,
        "throughput": len(lat) / wall,
        "p50": lat[len(lat)//2] if lat else None,
        "p95": lat[int(len(lat)*0.95)] if lat else None,
        "error_rate": errs / total,
    }
 
async def ramp(proxy, url):
    for c in (5, 10, 25, 50, 100, 200, 400):
        print(await measure(proxy, url, c))

Read the output for the inflection point:

Throughput rises, latency flat → not saturated. Keep increasing.

Throughput rises, latency rising proportionally → approaching saturation. You are queuing somewhere.

Throughput flat, latency rising linearly → saturated. Additional workers are adding queue time, not output. This is the point past which concurrency is pure waste.

Throughput falling, errors rising → past saturation and degrading.

Run at your optimum for thirty minutes afterward. Short bursts hide problems that only appear sustained.

Identify Which Bottleneck You Hit

The saturation point tells you that something constrained you; these tests tell you what.

Your own CPU or memory. Check host metrics during the ramp. Browser-based collection saturates memory fast — each Chrome instance is hundreds of megabytes, so a machine running 30 browsers is doing well. Plain HTTP clients saturate CPU on TLS and parsing much later.

Your HTTP client's connection pool. The most common invisible bottleneck. Default pool sizes are small — often 10. Fifty workers against a 10-connection pool queue inside your process, producing latency that looks like network latency and is not.

# requests
adapter = requests.adapters.HTTPAdapter(
    pool_connections=200, pool_maxsize=200
)
session.mount("https://", adapter)
 
# httpx
limits = httpx.Limits(max_connections=200,
                      max_keepalive_connections=100)

Diagnose by raising the pool and re-measuring. If throughput jumps, that was it.

The event loop or thread scheduler. With async clients, CPU-bound parsing inside the loop blocks everything. Move parsing to a thread or process pool. With threads, context switching costs rise past a few hundred.

The target. If latency rises with your rate specifically against one target while a neutral endpoint stays flat, the target is throttling you. This is a pacing problem, not a concurrency problem, and adding workers makes it worse.

The proxy layer. If a neutral endpoint also degrades, the constraint is upstream. Check whether your provider caps concurrent sessions or applies rate limits — if so, that ceiling is your ceiling regardless of what your infrastructure could handle.

Concurrency Is Per-Target, Not Global

A single global worker count applied across targets is almost always wrong, because targets differ by orders of magnitude in what they tolerate.

Structure it per target, with a global cap protecting your own infrastructure:

import asyncio
 
GLOBAL = asyncio.Semaphore(400)         # protects your host
PER_TARGET = {
    "permissive-api.example.com": asyncio.Semaphore(150),
    "moderate-retail.example.com": asyncio.Semaphore(50),
    "strict-search.example.com":  asyncio.Semaphore(12),
}
 
async def fetch(client, url, host):
    async with GLOBAL:
        async with PER_TARGET[host]:
            return await client.get(url)

The global semaphore prevents one permissive target from consuming all your capacity. The per-target ones keep each within its measured tolerance.

Make It Adaptive

Static limits drift out of correctness as targets change. A feedback loop converges without supervision:

class AdaptiveLimit:
    def __init__(self, start=20, floor=3, ceiling=300):
        self.limit = start
        self.floor, self.ceiling = floor, ceiling
        self.ok = 0
 
    def on_success(self):
        self.ok += 1
        if self.ok >= 100:
            self.limit = min(self.ceiling, int(self.limit * 1.15))
            self.ok = 0
 
    def on_failure(self, kind):
        if kind in ("blocked", "throttled", "timeout"):
            self.limit = max(self.floor, int(self.limit * 0.7))
            self.ok = 0

Additive-increase, multiplicative-decrease — the same principle TCP congestion control uses, for the same reason. Rise slowly while things are healthy, back off fast when they are not.

Backpressure Over Unbounded Queues

A common failure: the scheduler enqueues work faster than workers consume it, memory grows, and a crash loses everything in flight.

Bound the queue and let it push back:

queue = asyncio.Queue(maxsize=1000)
 
async def producer(urls):
    for u in urls:
        await queue.put(u)        # blocks when full — this is the point

A bounded queue means your producer runs at consumer speed automatically. It also makes queue depth a usable health metric: persistently full means workers are the constraint, persistently empty means the producer is.

Retries Consume Concurrency

Retries occupy workers that could be doing new work. A pipeline with a 20% failure rate and three retries each is spending a large fraction of its capacity re-fetching.

Cap attempts. Three is usually right.

Fail fast on hopeless cases. A 403 retried on the same IP will fail again. Rotate first or abandon.

Use a separate, smaller pool for retries so they cannot crowd out first attempts:

RETRY_POOL = asyncio.Semaphore(30)

Circuit-break per target. When failure rate crosses a threshold, stop sending entirely for a cooldown. Continuing to push a blocking target burns concurrency and bandwidth for zero output.

Track attempts per successful record. It is the cleanest measure of how much capacity retries are consuming.

Concurrency and IP Count Are Linked

The connection most teams miss: your concurrency ceiling and your IP requirement are the same number.

Every concurrent request needs its own IP at that instant to keep per-IP load low. Running 300 concurrent requests through 50 IPs means each address carries six simultaneous connections — which is itself a detection signal and pushes per-IP volume past tolerance.

IPs needed ≈ concurrency × (1 + retry_rate)

This is why a provider's concurrency cap matters more than its pool size. A cap of 100 means at most 100 IPs in flight no matter how many million exist behind the gateway. Your achievable throughput is then 100 / average_latency, full stop — and the only way to go faster is to push each IP harder, which is exactly what triggers the blocks you were avoiding.

Monitor Continuously

  • Throughput (valid records per second), per target
  • p50/p95/p99 latency, per target
  • Error rate by class, per target
  • Queue depth and time-in-queue
  • Attempts per valid record
  • Current adaptive limit per target
  • Host CPU, memory, open file descriptors

The signature worth alerting on: rising p95 at flat throughput and flat error rate. That is queuing — either inside your process or upstream — and it means your current concurrency is above the useful point.

FlameProxies applies no cap on concurrent sessions and no request-rate limits, so the saturation point you find in the ramp above is a property of your own infrastructure and your targets rather than an allowance being enforced. With 80M+ residential IPs, the IP-count requirement that scales with concurrency is available rather than rationed.