Proxies for AI Agents: Browser Use, Agent Browsers and Live Web Access

An AI agent that browses the live web is a web client, and the web treats it accordingly. The single most common surprise in agent development is that everything works on a laptop and breaks in production — because your laptop has a residential IP from a consumer ISP, and your cloud host has a datacenter IP that protected sites filter by default. Nothing in the code changed. The address did.

This guide covers what actually fixes that: which agent tasks need a proxy, how to configure one in browser-driven agents, and how agent traffic differs from scraping traffic in ways that change the setup.

Which agent tasks actually need a proxy

Start by not proxying everything. Agents do a lot of work that needs no proxy at all, and every proxy hop costs latency that a waiting user feels.

The rule of thumb: if the target is yours, skip the proxy. If the target is someone else's and has a commercial reason to block automation, proxy it.

Why agent traffic gets blocked more than you expect

Four properties make agent traffic conspicuous, and none of them are about the model.

Datacenter egress. Agent runtimes live on cloud hosts, containers or serverless platforms. Those IPs sit in hosting ASNs, and sites filter them wholesale because blocking a hosting range costs them almost no real visitors.

Fan-out bursts. One agent turn can trigger a dozen parallel fetches as the model explores. From the target's perspective that is a request spike from a single address.

Shared egress across users. If you run an agent as a product, every user's browsing exits from the same IPs. Rate limits are per-address, so your traffic scales into the limit even though no individual user is heavy.

Interaction timing that is not human. A browser-driven agent can fill a form in 80 milliseconds. Real people cannot. Behavioural detection notices, and IP quality alone will not save you.

That last point is the honest caveat about proxies generally: the proxy fixes the IP layer, not the other layers. A pristine residential IP paired with an inconsistent fingerprint, a timezone that contradicts the exit country, or superhuman typing speed still gets challenged. Anyone selling you proxies as a complete answer to bot detection is overselling.

Configuring proxies in browser-driven agents

Most capable web agents drive a real browser rather than issuing raw HTTP requests, because a real browser produces a realistic fingerprint and handles JavaScript. That changes where the proxy goes.

The key distinction

For an HTTP-based agent, HTTPS_PROXY in the environment is usually enough — any standard client library reads it.

For a browser-based agent, it usually is not. The browser process manages its own networking, so the proxy has to be passed at browser launch. This catches people out constantly: the environment variable is set, the agent still leaks the host IP, and nothing errors.

browser-use and Playwright-based agents

browser-use drives a real browser through Playwright, so the proxy is configured on the browser rather than in the agent loop. Playwright takes it at launch:

from playwright.sync_api import sync_playwright
 
with sync_playwright() as p:
    browser = p.chromium.launch(
        proxy={
            "server": "http://gateway.example.com:8000",
            "username": "your-username",
            "password": "your-password",
        }
    )
    page = browser.new_page()
    page.goto("https://api.ipify.org")
    print(page.content())   # should show the proxy's address

When the agent framework constructs the browser for you, look for the browser configuration object it exposes and set the proxy there. The detail worth knowing: Playwright takes the credentials as separate username and password fields rather than embedding them in the server URL, and embedding them in the URL is a common cause of silent auth failure.

Full mechanics in the Playwright proxy guide. For Puppeteer-based agents, the Puppeteer guide covers the different syntax and the separate authentication step it requires.

Verifying the exit — do this in production

The only check that matters:

page.goto("https://api.ipify.org")
# Compare against the host's own address

Run it in the deployed environment, not locally. Locally it will pass whether the proxy works or not, because your laptop's IP is already residential. That false pass is exactly how the laptop-to-production surprise happens.

MCP-based tools

If your agent reaches the web through MCP servers, the proxy belongs in the MCP server's environment rather than the agent's. That has its own mechanics — see MCP proxy.

Sessions: the part agents get wrong

Scrapers mostly want fresh IPs. Agents frequently want the opposite, and this is where agent setups fail in ways scraper setups do not.

Rotate per request for independent exploration — unrelated pages, multiple sites, search results. A rotating gateway handles this with no code.

Hold a sticky session for anything stateful. An agent working through a login, a multi-step form, a cart or a paginated flow must keep one address. Rotating mid-flow looks like session hijacking and gets treated as such. How to maintain sticky sessions covers the setup.

One identity per concurrent session. Two agent runs sharing one sticky IP will interleave requests into one apparent session, which is incoherent from the target's side — two different pages being navigated at once by one visitor.

One identity per tenant. Multi-user products need per-user separation, or one user's rate limit becomes everyone's.

This is why agents need far fewer IPs than scrapers but much better session control. A scraper wants a large pool. An agent wants the right identity held for the right duration.

Latency, and failing fast

A scraper does not care about 200ms. An agent often has a user waiting, and a single turn may chain several tool calls.

Three things that help:

1. Put the exit near the target. A US host fetching a German site through a German exit is frequently faster than through a US exit — the long hop happens once on a fast backbone instead of on the last mile.
2. Use datacenter proxies where trust is not needed. They are consistently faster; residential routes traverse consumer connections by definition.
3. Set timeouts low and fail loudly. Agents handle explicit failure well — they try another approach. They handle a hanging tool badly, because the entire turn stalls. Three seconds to connect, ten to read, one retry, then return the error to the model.

That third point is the one most worth internalising. Returning "this fetch failed" to the model is useful information it can act on. Returning nothing for thirty seconds is a dead turn.

Which proxy type

Most production agents need datacenter plus residential, split by tool, rather than one type for everything. Routing all agent traffic through residential IPs is the most common way to overspend; routing all of it through datacenter IPs is the most common way to get blocked.

FlameProxies residential proxies run on 80M-plus IPs across 180-plus countries at $0.50/GB, falling to $0.45/GB above 1TB, with HTTP(S) and SOCKS5 and unlimited concurrent sessions. That last point matters specifically for agents, because a fan-out turn opens many connections simultaneously and a concurrency cap turns into queued tool calls.

Failure modes

Works locally, blocked in production. Residential laptop, datacenter host. The defining agent proxy problem.

Environment variable set, still leaking host IP. The browser is managing its own networking. Pass the proxy at browser launch.

Agent summarises a CAPTCHA page as if it were content. The fetch returned 200, so nothing errored. Check response bodies for challenge markers, not just status codes — this is the most dangerous failure because it produces confident wrong answers.

Agent reports the wrong country's data. Wrong exit, or no proxy. Verify exit country before trusting any geo-specific claim.

Stateful flow breaks halfway. Rotation mid-session. Switch to sticky.

Concurrent runs interfere. Shared sticky identity. One per session.

Everything slow after adding a proxy. Exit too far from target, or residential where datacenter would do.

A setup that holds up

1. Split tools by target. Internal calls bypass the proxy entirely.
2. Default external browsing to residential, since the host is datacenter-resident.
3. Configure the proxy at browser launch, not just in the environment.
4. Verify the exit in production with an IP-reporting fetch.
5. Rotate per request; hold sticky for stateful flows.
6. One identity per concurrent session and per tenant.
7. Timeouts low, one retry, then surface the failure to the model.
8. Detect challenge pages in response content.
9. Keep the browser consistent with the exit — timezone, locale and language should agree with the country you are exiting from. A German IP with a US locale is a worse signal than a clean datacenter IP with everything aligned.

Current rates for both proxy types are on the pricing page. For the detection side of this — what anti-bot systems actually look at beyond IP — see anti-bot detection in 2026.