Scrapy Proxy Setup: Middleware, Auth and Per-Request Rotation

Scrapy handles proxies better than most frameworks, including authenticated ones, and it needs almost no setup: set request.meta['proxy'] and the built-in HttpProxyMiddleware does the rest. It is enabled by default, so there is nothing to install and nothing to register for a basic configuration.

import scrapy
 
class ExampleSpider(scrapy.Spider):
    name = "example"
 
    def start_requests(self):
        yield scrapy.Request(
            "https://api.ipify.org",
            meta={"proxy": "http://gateway.example.com:8000"},
            callback=self.parse,
        )
 
    def parse(self, response):
        self.logger.info("exit IP: %s", response.text.strip())

Authenticated proxies just work

This is where Scrapy is genuinely better than Selenium or Puppeteer. Put the credentials in the URL:

yield scrapy.Request(
    url,
    meta={"proxy": "http://user:pass@gateway.example.com:8000"},
)

HttpProxyMiddleware strips the credentials out, base64-encodes them, and sets a Proxy-Authorization header. No extension generation, no separate authenticate call, no interception layer.

Keep them out of source:

import os
 
PROXY = (
    f"http://{os.environ['PROXY_USER']}:{os.environ['PROXY_PASS']}"
    f"@gateway.example.com:8000"
)

If your password contains URL-special characters, encode it — an @ or : will split the URL in the wrong place:

from urllib.parse import quote
pw = quote(os.environ["PROXY_PASS"], safe="")

Applying a proxy to every request

Setting meta on every yield gets repetitive. A tiny downloader middleware handles it globally:

# myproject/middlewares.py
import os
 
class ProxyMiddleware:
    def __init__(self):
        self.proxy = (
            f"http://{os.environ['PROXY_USER']}:{os.environ['PROXY_PASS']}"
            f"@gateway.example.com:8000"
        )
 
    def process_request(self, request, spider):
        request.meta.setdefault("proxy", self.proxy)

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ProxyMiddleware": 350,
    # scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware runs at 750
}

Order matters. HttpProxyMiddleware sits at 750, so anything that sets meta['proxy'] must run before it — a lower number. Register yours at 750 or later and it will appear to do nothing at all. This is the single most common Scrapy proxy mistake.

Using setdefault rather than assignment means a request that already specifies its own proxy keeps it, which is what you want for per-request overrides.

Rotation from a list

If you are managing individual proxies rather than using a gateway:

import random, logging
 
class RotatingProxyMiddleware:
    def __init__(self, proxies):
        self.proxies = list(proxies)
        self.logger = logging.getLogger(__name__)
 
    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.getlist("PROXY_LIST"))
 
    def process_request(self, request, spider):
        if "proxy" not in request.meta:
            request.meta["proxy"] = random.choice(self.proxies)
 
    def process_exception(self, request, exception, spider):
        failed = request.meta.get("proxy")
        if failed in self.proxies and len(self.proxies) > 1:
            self.proxies.remove(failed)
            self.logger.warning(
                "Dropped failing proxy %s, %d remaining", failed, len(self.proxies)
            )
        new = request.copy()
        new.meta.pop("proxy", None)
        new.dont_filter = True
        return new

process_exception is the part worth copying: when a proxy fails at the connection level, it drops that proxy from the pool and retries the request with a different one. Without it, a dead proxy keeps getting selected.

A rotating gateway avoids all of this, because the provider handles pool health. Set one URL and each connection gets a fresh exit:

# settings.py — no rotation code needed
HTTPPROXY_ENABLED = True

request.meta["proxy"] = "http://user:pass@gateway.example.com:8000"

See proxy rotation strategies for the trade-off.

Retries

Tell Scrapy which responses mean "the proxy failed, try again":

# settings.py
RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 407, 408]
 
DOWNLOAD_TIMEOUT = 20

407 is the one to remember — proxy authentication failure. Include it and a transient auth blip gets retried; omit it and the request is dropped.

Also set DOWNLOAD_TIMEOUT. The default is 180 seconds, and a dead proxy will hold a slot for the full three minutes, which throttles the whole crawl.

Concurrency and politeness

Scrapy is built for volume, which makes it easy to hammer a target even through a large proxy pool. Rotation distributes load across addresses; it does not reduce the load the target sees.

# settings.py
CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 8
DOWNLOAD_DELAY = 0.25
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 4.0

AUTOTHROTTLE_ENABLED is worth turning on for proxied crawls specifically. It adapts the request rate to observed latency, which naturally backs off when the proxy or target is under strain instead of piling on retries.

Sticky sessions

For flows with server-side state, hold one IP. Most providers encode the session in the username:

import uuid
 
def start_requests(self):
    session = uuid.uuid4().hex[:8]
    proxy = f"http://user-session-{session}:pass@gateway.example.com:8000"
    yield scrapy.Request(login_url, meta={"proxy": proxy}, callback=self.after_login)

Pass the same proxy string through subsequent requests in the chain via meta, or the session breaks. How to maintain sticky sessions covers why rotating mid-flow is itself a detection signal.

Environment variables

HttpProxyMiddleware falls back to http_proxy, https_proxy and no_proxy when meta['proxy'] is unset. Convenient in development, and a source of surprise in production when an ambient variable routes a crawl somewhere unexpected. Set meta explicitly for anything that matters.

To exclude a request entirely:

request.meta["proxy"] = None

Common errors

Proxy ignored. Your middleware runs at or after 750. Move it earlier.

407 on every request. Bad credentials, or unencoded special characters in the password.

TunnelError. The proxy refused CONNECT for the target — often the proxy blocking the destination.

Crawl stalls. Dead proxies holding slots for the full DOWNLOAD_TIMEOUT. Lower it and add process_exception handling.

High bandwidth bills. Scrapy fetches what you tell it to, but a careless CrawlSpider rule set can pull images and assets. Restrict allow patterns and consider deny_extensions.

Blocked despite good residential IPs. Scrapy sends raw HTTP requests, so its TLS fingerprint identifies it as a library rather than a browser — often the failing signal rather than the IP. For hardened targets you need a real browser engine; see Playwright and anti-bot detection in 2026.

Which proxy type

Scrapy is a volume tool, so the economics matter more here than in browser automation. Requests are small and numerous, which suits per-gigabyte residential billing well — a crawl pulling HTML only moves far less data than a headless browser fetching full pages with assets.

Datacenter proxies for unprotected targets at high volume; residential for anything that filters. FlameProxies supports HTTP(S) and SOCKS5 across 80M-plus residential IPs in 180-plus countries at $0.50/GB, dropping to $0.45/GB above 1TB, with unlimited concurrent sessions — which matters directly for Scrapy, since a concurrency cap becomes your crawl's ceiling. Rates are on the pricing page.