Rotating Proxies in Playwright

A scraper that sends every request from one egress address gets rate limited, then challenged, then blocked — and once the address is burned, restarting the process does not help. Playwright exposes a proxy option at three levels (browserType.launch(), browser.newContext(), and apiRequest.newContext()), but the option is read once when the object is created and cannot be changed afterwards. Rotation is therefore a lifecycle problem, not a configuration problem: you rotate by disposing the object that holds the proxy and creating a new one. This page shows how to build that lease cycle around BrowserContext, how to attach credentials correctly, and how to tell a dead proxy apart from a hostile site.

Root cause: the proxy is frozen at object-creation time

BrowserContext inherits its network stack settings — proxy, user agent, locale, cookie jar — at construction and holds them for its whole life. There is no context.setProxy(), and there is no per-navigation override, because the browser process wires the proxy into the network service before the first byte moves. A second complication is Chromium-specific: unless the browser process was launched with a proxy configured, Chromium starts without proxy plumbing enabled and silently ignores a per-context proxy object, so every context egresses from the host's real IP. The fix is the documented sentinel proxy: { server: 'per-context' } passed to launch(), which turns the plumbing on without pinning a real upstream. Understanding the isolation model behind that lifetime is covered in Browser Contexts & Isolation, part of Scraper Performance & Scaling under Web Scraping & Data Extraction.

Minimal reproducible example

The test below leases each pool entry in turn, proves the exit IP changed, and disposes the context to release it. Every non-obvious line is annotated, because the mistakes in this snippet — inline credentials, a missing launch sentinel, a reused context — are exactly the ones that make rotation look like it works while every request leaves from the same address.

import { test, expect, chromium, type Browser, type BrowserContext } from '@playwright/test';

// One entry per upstream exit. `session` is the sticky-session token that most
// residential vendors read out of the username field rather than a separate header.
type Lease = { host: string; port: number; user: string; pass: string; session: string };

const POOL: Lease[] = [
  { host: 'gw.provider.net', port: 7000, user: 'acct1', pass: process.env.PROXY_PASS!, session: 'a1' },
  { host: 'gw.provider.net', port: 7000, user: 'acct1', pass: process.env.PROXY_PASS!, session: 'b2' },
  { host: 'gw.provider.net', port: 7000, user: 'acct1', pass: process.env.PROXY_PASS!, session: 'c3' },
];

test('each context egresses from a different exit IP', async () => {
  // Chromium ignores per-context proxies unless the process itself started with one.
  // 'per-context' is the documented sentinel: it enables the plumbing, pins nothing.
  const browser: Browser = await chromium.launch({ proxy: { server: 'per-context' } });
  const seenIps = new Set<string>();

  for (const lease of POOL) {
    const context: BrowserContext = await browser.newContext({
      proxy: {
        server: `http://${lease.host}:${lease.port}`,       // scheme + host + port ONLY
        username: `${lease.user}-session-${lease.session}`,  // sticky id rides in the username
        password: lease.pass,                                // never inline creds into `server`
        bypass: 'localhost, 127.0.0.1, .internal.test',       // keep local fixtures off the proxy
      },
    });

    // context.request uses the same network settings, so this echo is honest.
    const echo = await context.request.get('https://api.ipify.org?format=json');
    expect(echo.ok()).toBeTruthy();
    seenIps.add(((await echo.json()) as { ip: string }).ip);

    const page = await context.newPage();
    await page.goto('https://example.com/catalog', { waitUntil: 'domcontentloaded' });
    await expect(page.getByRole('heading', { name: 'Catalog' })).toBeVisible();

    await context.close(); // disposing the context is the rotation — the next loop leases another
  }

  expect(seenIps.size).toBe(POOL.length); // three contexts, three distinct exits
  await browser.close();
});

Two details in that snippet are worth stating plainly. The bypass list keeps local fixtures, health endpoints and internal hosts off the paid gateway, which matters because a vendor bills by transferred byte and a mocked asset routed through an upstream exit costs real money for no benefit. And the assertion on seenIps.size is deliberately placed at the end rather than inside the loop: a per-iteration check would pass on the first lease and hide a pool that collapses to a single exit from the second lease onwards, which is the exact shape of the bug most rotation code ships with.

The three creation points each rotate at a different granularity, and picking the wrong one is the usual reason a scraper spends more time restarting Chromium than fetching pages. Launch-level proxies are correct when an entire run must egress from one region — a compliance requirement, or a site whose pricing varies by country. Context-level proxies are the working default. Request-level contexts, created through apiRequest.newContext(), carry no browser at all and suit backfill jobs that read JSON endpoints directly once the HTML crawl has discovered their URLs.

Step-by-step fix

1. Launch once with the per-context sentinel. Call chromium.launch({ proxy: { server: 'per-context' } }) a single time per worker process. Without it Chromium discards the per-context proxy and leaks your real address; with a real upstream in launch() instead, every context inherits that one exit and rotation quietly stops working.
2. Model the pool as data, not configuration. Keep an array of entries carrying host, port, credentials, sticky-session token, a state field (available, leased, cooling, retired), and a cooldownUntil timestamp. A pool you can query is a pool you can drain fairly across the concurrent tasks described in Running Parallel Scrapers with Worker Pools.
3. Create one context per lease and close it to rotate. Acquire an entry, pass it to browser.newContext({ proxy }), do the work, then await context.close(). Closing releases the exit and discards the cookie jar with it, so the next lease starts with a clean identity instead of carrying the previous session's tracking cookies to a new IP — an inconsistency that fingerprinting checks look for directly.
4. Put credentials in username and password, never in the URL. Chromium strips userinfo from http://user:pass@host:port and then answers 407 Proxy Authentication Required. Keep server as scheme, host and port; encode a vendor's sticky-session token inside username (acct1-session-b2) so the same physical gateway hands you a different exit per token.
5. Classify proxy failures separately from site failures. A net::ERR_TUNNEL_CONNECTION_FAILED or net::ERR_PROXY_CONNECTION_FAILED thrown by page.goto() means the proxy is broken and the lease must be retired; a 429 or a challenge page means the exit is burned and belongs in cooldown; a 500 from the origin is the site's problem and should be retried on the same lease, following the backoff model in Handling Rate Limits and Retries When Scraping.
6. Verify the exit before the first real navigation. Issue one cheap context.request.get() against an IP echo endpoint. context.request shares the context's proxy and cookie jar, so the answer reflects what the page will actually use, and a lease that fails the echo can be retired before it wastes a page load.

Troubleshooting variants

page.goto: net::ERR_TUNNEL_CONNECTION_FAILED on every navigation

Chromium reports this when the CONNECT request to the proxy is refused or dropped, and it fires before any site code runs, so it is always a transport problem rather than a block. Check three things in order: the scheme in server (a SOCKS gateway addressed as http:// fails immediately, and Chromium cannot authenticate SOCKS5 proxies at all — it raises net::ERR_NO_SUPPORTED_PROXIES), the port (many vendors publish separate ports for rotating and sticky pools), and whether the account has been suspended for overage. A quick isolation test is to run the same request through apiRequest.newContext({ proxy }) in Node; if that also fails, the browser is not involved.

Rotation runs but every context reports the same IP

Two causes dominate. The first is a real upstream passed to launch(), which takes precedence and makes the per-context object cosmetic — replace it with the per-context sentinel. The second is a sticky-session token you forgot to vary: residential gateways return the same exit for the same username, by design, so acct1-session-a1 will keep resolving to one address for the session's whole lifetime. Vary the token per lease, or use the vendor's rotating endpoint, which assigns a new exit per TCP connection. Confirm which case you are in by logging the echo response for each lease rather than trusting the pool bookkeeping.

Credentials are ignored and the proxy answers 407

This is almost always userinfo embedded in server. Chromium removes it during URL canonicalisation, Firefox handles it inconsistently, and the resulting request arrives with no Proxy-Authorization header. Move the values into the username and password fields of the proxy object and read them from environment variables so they never reach the repository. If the header is present and still rejected, the vendor is likely rejecting a malformed sticky-session suffix — send the bare username once to confirm the base account works before layering the token back on.

Verification

A rotating pool fails quietly. Nothing throws when every context egresses from the same address, and the symptom — a slowly rising block rate — looks identical to a site tightening its defences, so teams often spend a week tuning delays before discovering the proxy object was never applied. Prove rotation three ways rather than assuming it. First, assert on distinct exits: collect the echo response per lease into a Set and assert its size equals the number of leases, as the example test does — a pool that silently collapses to one exit fails immediately instead of a week later. Second, read the network panel of a saved trace, where the request headers and timings for each context are recorded separately; the technique is covered in Reading Network and Console Tabs in Traces. Third, watch the block rate itself: log the response status of the first navigation per lease and alert when the share of 403 and 429 answers rises, because that curve, not the proxy count, tells you whether the pool is healthy.

Rotation is one control among several. Pair it with realistic client characteristics from Emulating Devices, Locales and Timezones so the fingerprint matches the geography of the exit, trim request volume using Blocking Images and Fonts to Speed Up Scraping so each lease buys more pages, and keep the crawl inside the limits set out in Respecting robots.txt and Crawl Delays. Where a route handler already rewrites traffic, the patterns in Intercepting and Modifying Network Requests let you fulfil selected requests from an APIRequestContext on a different exit without disturbing the page's own identity.

Frequently Asked Questions