Running Parallel Scrapers with Worker Pools

A scraper that walks a list of URLs one at a time is limited by network latency, not by your machine: the CPU idles for hundreds of milliseconds per page while the response travels. The obvious fix — wrap the list in Promise.all() and let every URL start at once — trades one bottleneck for a much worse one, because each concurrent job opens its own browser context and its own renderer process. The correct shape is a fixed-size worker pool: a small number of long-lived loops that pull jobs from a shared queue, each owning exactly one BrowserContext at a time. This page shows how to build one, how to size it, and how to keep it alive across a run of hundreds of thousands of pages.

Root cause: concurrency is not the same as scheduling

Promise.all() does not schedule anything — the promises it receives have already started, because urls.map(scrapeOne) invoked every function before Promise.all ever ran. Each browser.newContext() allocates an isolated cookie jar, cache and storage partition, and each context.newPage() costs a renderer process that reserves 60–120 MB before it has parsed a byte of HTML, so a few hundred simultaneous jobs will exhaust RAM, file descriptors or /dev/shm long before they exhaust the URL list. The fix is to make concurrency a property of the runner rather than of the input size, which is exactly what Browser Contexts & Isolation exists to support.

Minimal reproducible example

The script below is the version almost every scraper starts as. It works for twenty URLs and collapses at five hundred, usually with browserContext.newPage: Target closed or a bare Protocol error (Target.createTarget): Target closed once the kernel refuses another process.

import { chromium, type Browser } from 'playwright';

const urls: string[] = Array.from({ length: 500 }, (_, i) => `https://example.com/p/${i}`);

async function scrapeOne(browser: Browser, url: string): Promise<string> {
  const context = await browser.newContext();  // isolated cookie jar + cache
  const page = await context.newPage();        // a whole renderer process
  await page.goto(url, { waitUntil: 'domcontentloaded' });
  const title = await page.getByRole('heading', { level: 1 }).innerText();
  await context.close();
  return title;
}

async function main(): Promise<void> {
  const browser = await chromium.launch();
  // map() calls scrapeOne 500 times synchronously, so 500 contexts are
  // requested in the same tick. Promise.all only waits — it never throttles.
  const titles = await Promise.all(urls.map((url) => scrapeOne(browser, url)));
  console.log(titles.length);
  await browser.close();
}

void main();

Two failures hide in there beyond the memory blowup. The array of 500 results is held in memory until the last promise settles, and a single rejected job — one 503, one navigation timeout — rejects the whole Promise.all, discarding 499 successful extractions. Switching to Promise.allSettled rescues the results but not the resource usage, and adding a semaphore around scrapeOne fixes the resource usage but still pays a context construction on every single URL. Both problems disappear once jobs are consumed by long-lived workers instead of launched by the input list, because the unit of concurrency becomes the worker rather than the job.

Step-by-step fix

The pool is a handful of asynchronous loops sharing one mutable array. Because Node runs your JavaScript on a single thread, queue.pop() between two await points is atomic with respect to the other workers: no two loops can ever receive the same job, and no lock is required.

1. Size the pool from the host, never from the input. Start at Math.min(os.cpus().length, 8) and treat that as a ceiling, not a target. A Chromium context with an active page costs roughly 150–350 MB depending on how heavy the site is, so a 4 GB CI container realistically supports four to six concurrent contexts. Measure before raising it.
2. Put the jobs in one shared queue and start exactly N loops. Every worker calls the same queue.pop(); when it returns undefined the loop exits and that worker resolves. The pool's width is fixed by how many loops you started, so no input size can widen it.
3. Give each worker one context and open a page per job. The context carries the session, the cache and any rotating proxy assignment; the page is disposable. Closing the page in a finally block guarantees the renderer is released even when extraction throws.
4. Recycle the context on a page budget. Long-lived contexts accumulate service workers, IndexedDB rows and HTTP cache entries. Closing and reopening the context every 20–50 pages resets that growth at a cost of a few hundred milliseconds, which is far cheaper than a mid-run out-of-memory kill.
5. Make failure per-job, not per-run. Wrap each job in try/catch, increment an attempt counter, and push it back on the queue while attempts remain. A bounded budget stops one permanently broken URL from cycling forever; pair it with the backoff rules in Handling Rate Limits and Retries When Scraping.
6. Cap the cost of every job. Pass an explicit timeout to page.goto() and prefer waitUntil: 'domcontentloaded' over 'networkidle', which rarely fires on pages with polling or analytics beacons. Combining this with request blocking for images and fonts often doubles pool throughput without adding a single worker.
7. Stream results and shut down deterministically. Append each extracted row to NDJSON or a database instead of collecting an array, then close contexts in finally blocks and the browser last, so an exception anywhere still leaves no orphaned processes behind.

The full implementation is under sixty lines and has no dependency beyond Playwright and Node's standard library.

import { chromium, type Browser, type BrowserContext } from 'playwright';
import { cpus } from 'node:os';
import { appendFile, readFile } from 'node:fs/promises';

type Job = { url: string; attempts: number };

const MAX_ATTEMPTS = 3;        // per-job retry budget, not a global one
const PAGES_PER_CONTEXT = 25;  // recycle before cache and storage bloat
const CONCURRENCY = Math.max(1, Math.min(cpus().length, 8));

async function scrapeJob(context: BrowserContext, job: Job): Promise<void> {
  const page = await context.newPage();
  try {
    // networkidle would hang on any page that polls; domcontentloaded is enough.
    await page.goto(job.url, { waitUntil: 'domcontentloaded', timeout: 20_000 });
    const title = await page.getByRole('heading', { level: 1 }).innerText();
    const price = await page.getByTestId('price').innerText();
    // Stream each row out so heap size is independent of the job count.
    await appendFile('out.ndjson', `${JSON.stringify({ url: job.url, title, price })}\n`);
  } finally {
    await page.close();  // release the renderer even when extraction threw
  }
}

async function worker(browser: Browser, queue: Job[], id: number): Promise<void> {
  let context = await browser.newContext({ viewport: { width: 1280, height: 800 } });
  let pagesUsed = 0;
  try {
    // pop() between awaits is atomic on Node's single thread: no job is shared.
    for (let job = queue.pop(); job !== undefined; job = queue.pop()) {
      try {
        await scrapeJob(context, job);
      } catch (error) {
        job.attempts += 1;
        if (job.attempts < MAX_ATTEMPTS) queue.unshift(job);  // retry later, not next
        else console.error(`worker ${id} abandoned ${job.url}:`, error);
      }
      if (++pagesUsed >= PAGES_PER_CONTEXT) {
        await context.close();                 // drop accumulated cache + storage
        context = await browser.newContext();
        pagesUsed = 0;
      }
    }
  } finally {
    await context.close();
  }
}

async function main(): Promise<void> {
  const urls: string[] = JSON.parse(await readFile('urls.json', 'utf8'));
  const queue: Job[] = urls.map((url) => ({ url, attempts: 0 }));
  const browser = await chromium.launch();
  try {
    // Exactly CONCURRENCY loops exist, whatever the length of the queue.
    await Promise.all(
      Array.from({ length: CONCURRENCY }, (_, i) => worker(browser, queue, i)),
    );
  } finally {
    await browser.close();  // closing the browser reaps every remaining context
  }
}

void main();

Three properties of that loop are worth naming, because they are what make the pool survive a long run. Peak resource usage is a constant: CONCURRENCY contexts and at most CONCURRENCY open pages exist at any instant, regardless of whether the queue holds five hundred URLs or five million. Heap usage is also a constant, because nothing accumulates in memory between iterations — each extracted row leaves the process as soon as it is produced. And progress is monotonic: a job either lands in the output file, gets requeued, or is logged as abandoned, so the sum of those three counts always equals the input size and any discrepancy points at a specific bug rather than a vague suspicion.

If your extraction already lives in the Playwright test runner, you get most of this for free: set workers and fullyParallel: true in playwright.config.ts, shard the URL list across test.describe.configure({ mode: 'parallel' }) blocks, and hoist the expensive per-worker setup into a fixture as described in Worker-Scoped Fixtures for Expensive Setup. The runner then owns process management and you own only the extraction logic.

Troubleshooting variants

The run hangs with workers alive but no progress

A worker that never returns from page.goto() blocks its slot forever, and with four workers stuck the pool is dead while the process still looks healthy. Playwright's default navigation timeout is 30 seconds but only applies to the navigation itself — a locator awaiting an element that never appears uses the separate expect or action timeout. Set both explicitly (context.setDefaultNavigationTimeout() and context.setDefaultTimeout()), and log a heartbeat per job so a stalled worker is visible in the run output. If the site sits behind a captcha wall, the worker is waiting on a page that will never resolve, which the techniques in Anti-Bot Defenses & Rate Limiting address directly.

Target page, context or browser has been closed mid-run

This error means the object you awaited was destroyed underneath you. The two common causes in a pool are a recycle that closed the context while a page from the previous job was still settling, and a browser-level crash that takes every context with it. Never recycle while a job is in flight — do it between jobs, as in the loop above — and never share one context across two concurrently running jobs. If the message appears together with a Chromium crash dump, you are out of memory: reduce CONCURRENCY, and in Docker add --disable-dev-shm-usage or mount a larger /dev/shm, since the default 64 MB shared-memory segment is a classic cause covered in Dockerizing Playwright for Headless CI.

Throughput falls as concurrency rises

Past a point that depends on the host, adding workers makes the run slower. Every renderer competes for the same cores, page load times inflate, the fixed timeouts you set start firing on pages that would have loaded fine at lower concurrency, and each of those timeouts turns into a retry — so the pool ends up doing strictly more work than it did with half the workers. A slower degradation looks the same from the outside: throughput that is fine for the first thousand pages and halves by the ten-thousandth is usually a leak rather than contention, and shortening PAGES_PER_CONTEXT will confirm it in one experiment. To find the right width, chart pages-per-minute against worker count on a sample of a few hundred representative URLs and pick the value just before the curve flattens, not the one that maximises CPU usage. Remember that the target server has its own knee, and a pool tuned past it will find rate limiting long before it finds your hardware limit.

Verification

Prove the bound holds rather than assuming it. Increment a counter when a worker opens a context and decrement it when it closes one, then assert the peak never exceeds CONCURRENCY; if it does, a recycle path is leaking. Watch the browser's own process tree during a run — on Linux, pgrep -c -f "chromium.*--type=renderer" should hover at the pool width plus one or two in transition, and a number that climbs monotonically is a page that is never closed. Sample process.memoryUsage().rss every thirty seconds and confirm it plateaus instead of trending upward, which is the signal that context recycling is doing its job. Finally, verify correctness as well as capacity: the output NDJSON line count must equal the input URL count minus the abandoned jobs your error log records, and the extracted fields should match a single-worker run over the same sample. If jobs disappear silently, the retry branch is dropping them — trace one job end to end with the Playwright Trace Viewer to see where it stopped.

One more check catches the subtlest class of pool bug. Run the same input twice at different widths — once with CONCURRENCY = 1 and once at your production setting — and diff the sorted output. Identical files mean no job is leaking state across workers; a diff means something is shared that should not be, usually a module-level variable mutated inside scrapeJob or a login session reused after the context that owned it was recycled. Doing this on a fixed sample after every change to the extraction code costs a few minutes and is the only cheap way to distinguish a genuine site change from a concurrency defect once the scraper is running in production.

Frequently Asked Questions