Scraper Performance & Scaling

A scraper that reads one page in four seconds reads a hundred thousand pages in four and a half days. Throughput, not correctness, is what stops most extraction projects, and the usual reflex — launch more browsers — hits a wall within a few hundred concurrent pages because every browser costs a few hundred megabytes of RAM and a full CPU core under load. The gains that actually matter come from a different direction: fetching less, waiting less, and reusing more. This guide breaks the per-page cost budget apart, shows which parts you can delete outright, and then builds up a concurrency model that saturates a machine without crashing it. It sits under Web Scraping & Data Extraction and assumes you already extract records reliably, as covered in Structured Data Extraction.

Why a naive scraper is slow

The instinct is to blame the network, but for a real-world commercial page the network is a small slice. A default page.goto() with waitUntil: 'load' drives a full browser through the same work a human visitor's browser does: it fetches the document, parses it, discovers dozens of subresources, downloads hero images at retina resolution, loads three or four webfont files, executes analytics and tag-manager bundles, runs framework hydration, and fires whatever the page does on idle. Your extraction — a few textContent() calls — accounts for a rounding error at the end of that sequence.

There are three independent cost axes, and confusing them is why tuning efforts stall. Bytes is what the page downloads; it is dominated by images, media, and fonts, and it is the axis that resource blocking attacks. Latency is dead time where the process waits on a signal that may not mean anything useful — the classic case is networkidle, which waits for a 500 ms quiet period that a page with a polling heartbeat or an open WebSocket never reaches. Overhead is the per-unit fixed cost of the automation itself: launching a browser process, creating a context, spinning up a renderer, and tearing it all down. Deleting bytes does nothing for a scraper whose real problem is that it launches a fresh Chromium for every URL.

Prerequisites

Before tuning, get four things in place. Run a recent Playwright (1.40 or newer) so that route.fetch(), route.fallback(), and the blob-friendly APIRequestContext behave as documented. Have a stable extraction path already — a scraper that is fast and wrong is worse than one that is slow and right, so lock the locators down first using the patterns in Handling Dynamic Content. Understand the Browser → BrowserContext → Page hierarchy, because every reuse strategy below trades isolation for speed at one of those levels; Browser Contexts & Isolation covers the semantics. Finally, decide your politeness envelope up front: concurrency is a weapon, and the pacing rules in Anti-Bot Defenses & Rate Limiting constrain everything that follows.

Measure first: build a per-page cost breakdown

Optimising without numbers produces confident nonsense. The cheapest instrumentation attaches to the context's request and response events, tallies bytes and counts by resourceType(), and brackets the navigation with a monotonic timer. Ten minutes of this on twenty representative URLs tells you exactly which of the three cost axes you are fighting.

import { chromium } from 'playwright';

// Profile one page: how many requests, of what type, and how long each phase took.
async function profile(url: string): Promise<void> {
  const browser = await chromium.launch();
  const context = await browser.newContext();
  const byType = new Map<string, number>();          // resourceType -> request count
  const page = await context.newPage();

  page.on('request', (req) => {
    const type = req.resourceType();                  // 'document' | 'image' | 'font' | 'script' | 'xhr' | ...
    byType.set(type, (byType.get(type) ?? 0) + 1);
  });

  const t0 = performance.now();
  // 'domcontentloaded' returns as soon as the HTML is parsed — the earliest useful marker.
  await page.goto(url, { waitUntil: 'domcontentloaded' });
  const parsed = performance.now();

  // Wait for the one element that proves the data you want exists.
  await page.getByRole('heading', { level: 1 }).waitFor();
  const ready = performance.now();

  console.log('html parsed  ms', Math.round(parsed - t0));
  console.log('data ready   ms', Math.round(ready - t0));
  console.log('requests by type', Object.fromEntries(byType));

  await browser.close();
}

profile('https://example.com/product/1');

Run it across a sample rather than a single URL, and record the median and the 95th percentile separately. Averages hide the shape of the problem: a scrape whose median page takes 900 ms but whose p95 takes 14 seconds is not slow, it is stalling on a specific class of page — a redirect chain, a challenge interstitial, or a template that loads a heavy embed. Fixing the tail is usually a bigger win than shaving the median, and it is invisible if you only look at the total run time.

Read the output as a diagnosis. Sixty image requests and a two-second gap means you are byte-bound — block resources. Six requests but a four-second gap means you are latency-bound — your wait condition is wrong. A small gap per page but a slow overall run means you are overhead-bound — you are paying browser startup too often, and no amount of request filtering will help.

Choose the cheapest extraction path

The fastest browser work is the browser work you never do. Before optimising a render, ask whether the render is required at all. Many pages that look client-rendered are fed by a JSON endpoint you can call directly, and many that look dynamic actually ship the data in the initial HTML as a __NEXT_DATA__ script or a JSON-LD block. Playwright's APIRequestContext gives you an HTTP client that shares the browser's cookie jar and proxy settings, so you can authenticate once in a real browser and then pull thousands of records without a renderer.

The hybrid pattern below logs in once with a browser, hands the resulting cookies to an APIRequestContext, and then pages through an internal API at HTTP speed. Session reuse itself is covered in Scraping Data Behind Login Sessions.

import { chromium, request as playwrightRequest } from 'playwright';

type Product = { id: string; name: string; price: number };

async function harvestViaApi(): Promise<Product[]> {
  const browser = await chromium.launch();
  const context = await browser.newContext();
  const page = await context.newPage();

  // Do the expensive, JS-heavy part exactly once: authenticate in a real browser.
  await page.goto('https://example.com/login');
  await page.getByLabel('Email').fill(process.env.SCRAPE_USER!);
  await page.getByLabel('Password').fill(process.env.SCRAPE_PASS!);
  await page.getByRole('button', { name: 'Sign in' }).click();
  await page.waitForURL('**/dashboard');

  // Export the authenticated cookies and localStorage as a storage state object.
  const state = await context.storageState();
  await browser.close();

  // The API context speaks HTTP only — no renderer, no layout, no paint.
  const api = await playwrightRequest.newContext({
    baseURL: 'https://example.com',
    storageState: state,
  });

  const all: Product[] = [];
  for (let cursor = 0; ; cursor += 100) {
    const res = await api.get(`/api/products?offset=${cursor}&limit=100`);
    if (!res.ok()) throw new Error(`API returned ${res.status()} ${res.statusText()}`);
    const batch = (await res.json()) as { items: Product[] };
    if (batch.items.length === 0) break;             // authoritative stop signal
    all.push(...batch.items);
  }

  await api.dispose();                                // releases sockets deterministically
  return all;
}

A run like this is typically thirty to a hundred times faster per record than driving the UI, and it is far gentler on the target. Reserve the browser for pages that genuinely need one.

Cut the payload before it is fetched

When a render is unavoidable, delete the bytes. context.route() intercepts requests before they leave, and aborting by resourceType() removes images, fonts, media, and stylesheets that no extraction path reads. On an image-heavy catalogue page this routinely halves the wall clock and cuts transferred bytes by 80–90 percent.

import { chromium } from 'playwright';

const BLOCKED = new Set(['image', 'font', 'media']);
// Third-party hosts that only cost time: analytics, tag managers, ad exchanges.
const BLOCKED_HOSTS = [/googletagmanager\.com/, /doubleclick\.net/, /hotjar\.com/];

async function scrapeLean(url: string): Promise<string[]> {
  const browser = await chromium.launch();
  // Route on the context so every page in it inherits the filter.
  const context = await browser.newContext();

  await context.route('**/*', (route) => {
    const request = route.request();
    if (BLOCKED.has(request.resourceType())) return route.abort();
    if (BLOCKED_HOSTS.some((re) => re.test(request.url()))) return route.abort();
    return route.continue();                          // everything else proceeds untouched
  });

  const page = await context.newPage();
  await page.goto(url, { waitUntil: 'domcontentloaded' });
  const titles = await page.getByTestId('product-title').allTextContents();
  await browser.close();
  return titles;
}

Two cautions. Aborted requests surface in the page as net::ERR_FAILED, and an application that treats a failed font or image as a fatal error can leave its own loading state stuck — so verify your extraction still succeeds after each addition to the block list. And routing is not free: a '**/*' pattern proxies every request through the Node process, adding roughly a millisecond of round trip each. On a page with 400 requests, narrowing the pattern to the handful of URL globs you actually intend to block is measurably cheaper than intercepting everything. The full decision procedure, including how to block CSS without breaking layout-dependent extraction, lives in Blocking Images and Fonts to Speed Up Scraping. The interception mechanics themselves are covered in Network Interception Basics.

Stop paying for navigation waits you do not need

page.goto() accepts a waitUntil option, and the default 'load' is almost never what a scraper wants. Choosing the earliest state that guarantees your data exists is the single highest-leverage line change in most scrapers.

The correct pattern is to navigate with 'domcontentloaded' and then wait on a specific assertion about the data you want, which is both faster and stricter than any global page state. A page that hydrates in 300 ms is read in 300 ms; you never wait 500 ms for a quiet network that carries no information about whether your rows exist.

import { chromium } from 'playwright';

async function readRows(url: string): Promise<number> {
  const browser = await chromium.launch();
  const page = await (await browser.newContext()).newPage();

  // Cap the navigation so a hung target cannot stall a worker indefinitely.
  page.setDefaultNavigationTimeout(15_000);

  // Return as soon as the HTML is parsed, not when the last tracking pixel lands.
  await page.goto(url, { waitUntil: 'domcontentloaded' });

  // The real readiness signal: the first row exists in the DOM.
  const rows = page.getByRole('row');
  await rows.first().waitFor({ state: 'attached', timeout: 10_000 });

  const count = await rows.count();
  await browser.close();
  return count;
}

Single-page applications need one more distinction. After the first navigation, moving between routes fires no navigation event at all — the framework swaps the view client-side, so page.waitForLoadState() returns instantly against the state of the document you are already on and tells you nothing. Wait on the content instead: assert that a heading matching the new route is attached, or register page.waitForResponse() for the data call before clicking. The same trap catches scrapers that click a client-side "next page" control and immediately read the DOM, harvesting the previous page's rows twice.

Note state: 'attached' rather than the default 'visible'. Extraction reads the DOM, not the pixels, so requiring visibility makes you wait for layout and paint you do not consume — and it breaks outright when you have blocked the CSS that would have made the element visible.

Reuse the browser: contexts, pages, and pooling

Launching Chromium costs 250–400 ms and a fresh renderer process; creating a BrowserContext costs single-digit milliseconds. A scraper that calls chromium.launch() per URL pays browser startup as a tax on every record. The rule is: one browser per process, one context per unit of isolation, and a page recycled within the context.

How much isolation you need is a judgement call. A fresh context per URL guarantees no cookie, cache, or localStorage bleed between targets — essential when a site personalises based on prior visits, or when you rotate identities. Reusing a single context across many pages of the same site is faster still because the HTTP cache and connection pool stay warm, which can cut repeat navigations by several hundred milliseconds. The compromise most production scrapers land on is a recycled context: keep one per worker, and close and recreate it every N pages to release the memory that accumulates in a long-lived renderer.

import { chromium, type Browser, type BrowserContext } from 'playwright';

// A worker that recycles its context every `recycleAfter` pages to bound memory.
class ScrapeWorker {
  private context: BrowserContext | null = null;
  private handled = 0;

  constructor(private browser: Browser, private recycleAfter = 50) {}

  private async ensureContext(): Promise<BrowserContext> {
    if (this.context && this.handled < this.recycleAfter) return this.context;
    if (this.context) await this.context.close();      // frees the renderer's heap
    this.handled = 0;
    this.context = await this.browser.newContext({
      // A viewport this small still lays out the DOM but paints far less.
      viewport: { width: 1280, height: 800 },
      javaScriptEnabled: true,
    });
    return this.context;
  }

  async scrape(url: string): Promise<string> {
    const context = await this.ensureContext();
    const page = await context.newPage();
    try {
      await page.goto(url, { waitUntil: 'domcontentloaded' });
      await page.getByRole('heading', { level: 1 }).waitFor({ state: 'attached' });
      return (await page.getByRole('heading', { level: 1 }).textContent()) ?? '';
    } finally {
      await page.close();                              // always close, even on throw
      this.handled += 1;
    }
  }
}

The finally block matters more than it looks. A page that is never closed keeps its renderer alive, and a few hundred leaked pages will exhaust a container long before the queue drains. The same discipline applied to test suites is described in Playwright Config & Fixtures, where worker-scoped resources are created once and torn down deterministically.

Scale out with bounded concurrency

Once per-page cost is minimised, throughput is a function of how many pages run at once — up to the point where the machine saturates and every page slows down together. Chromium needs roughly one CPU core and 150–300 MB of resident memory per active renderer under load, so a 4-core, 8 GB box tops out somewhere between six and ten concurrent pages. Beyond that, contention makes total throughput fall while error rates climb.

Pull-based dispatch beats splitting the URL list into equal chunks. With fixed slices, one worker that draws a batch of slow pages finishes minutes after the rest; with a shared queue, every worker keeps pulling until the queue is empty and the tail flattens out. Node's single-threaded event loop makes Array.prototype.shift() safe here — no lock is needed because no other task can run between the check and the removal.

import { chromium, type Browser } from 'playwright';

type Row = { url: string; title: string };

async function runPool(urls: string[], size = 6): Promise<Row[]> {
  const browser: Browser = await chromium.launch();
  const queue = [...urls];                             // shared, mutated by all workers
  const rows: Row[] = [];
  const failures: string[] = [];

  // Each worker owns exactly one context for its whole lifetime.
  const worker = async (): Promise<void> => {
    const context = await browser.newContext();
    await context.route('**/*', (route) =>
      ['image', 'font', 'media'].includes(route.request().resourceType())
        ? route.abort()
        : route.continue(),
    );
    for (let url = queue.shift(); url !== undefined; url = queue.shift()) {
      const page = await context.newPage();
      try {
        await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 15_000 });
        const heading = page.getByRole('heading', { level: 1 });
        await heading.waitFor({ state: 'attached', timeout: 8_000 });
        rows.push({ url, title: (await heading.textContent())?.trim() ?? '' });
      } catch (error) {
        // One bad URL must never kill the worker or the run.
        failures.push(url);
      } finally {
        await page.close();
      }
    }
    await context.close();
  };

  await Promise.all(Array.from({ length: size }, () => worker()));
  await browser.close();
  console.log(`ok=${rows.length} failed=${failures.length}`);
  return rows;
}

Size the pool empirically: run at 2, 4, 8, and 16 and plot pages per minute against error rate. The correct value is the last one where throughput still rises and timeouts stay flat. Queue design, retries with re-enqueueing, and multi-process sharding are worked through in Running Parallel Scrapers with Worker Pools.

Spread egress across proxies

Concurrency raises a second wall that has nothing to do with your hardware: the target's per-IP limits. Ten workers behind one address look like one very strange visitor, and the response is a 429, a challenge page, or a silent block. Distributing requests across an egress pool keeps per-IP rates inside acceptable bounds while total throughput scales. Playwright takes a proxy option at both launch() and newContext(), and the context-level form is what makes rotation practical — you assign an address per worker, or per context recycle, without restarting the browser.

import { chromium } from 'playwright';

const POOL = [
  { server: 'http://gw1.proxy.example:8000', username: 'u1', password: 'p1' },
  { server: 'http://gw2.proxy.example:8000', username: 'u2', password: 'p2' },
];

// One egress identity per context, chosen round-robin as contexts are recycled.
async function contextWithProxy(index: number) {
  const browser = await chromium.launch();
  return browser.newContext({ proxy: POOL[index % POOL.length] });
}

Treat the pool as a health-checked resource, not a static list: a dead gateway produces net::ERR_TUNNEL_CONNECTION_FAILED or net::ERR_PROXY_CONNECTION_FAILED on every navigation, and without eviction one bad address poisons a fixed share of your run. Rotation policy, sticky sessions for multi-step flows, and failure handling are detailed in Rotating Proxies in Playwright.

Do less work overall: dedupe, cache, and crawl incrementally

Every optimisation above makes a page cheaper. The larger win is not fetching the page at all. Most production scrapers re-fetch the same URLs on every scheduled run because the crawl frontier is rebuilt from scratch each time, so a job that nominally covers 200,000 products spends the bulk of its budget re-reading records that have not changed since yesterday.

Three habits remove that waste. First, deduplicate the frontier before it reaches the queue: canonicalise URLs by stripping tracking parameters, session identifiers, and fragment hashes, sort the remaining query keys, and hold the normalised form in a Set. Listing pages routinely link the same product through half a dozen decorated URLs, and a catalogue crawl that skips them does a fraction of the work for identical coverage.

Second, ask the server what changed. If the target sends ETag or Last-Modified headers, store them alongside each record and replay them as If-None-Match or If-Modified-Since on the next run. A 304 Not Modified costs one round trip and no rendering at all, which is cheaper than the fastest possible page load. Where the site exposes a sitemap with <lastmod> timestamps, or a feed ordered by update time, use it as the frontier and stop as soon as you reach a timestamp older than your last successful run.

Third, separate discovery from extraction. Run a cheap, frequent pass that only reads listing pages to find new or changed identifiers, and a slower pass that renders detail pages for that delta. The discovery pass is usually API-shaped or HTML-only, so it costs almost nothing, and the expensive renderer work shrinks to the records that actually moved. Combined with a persisted cursor, this turns a multi-day full crawl into a nightly job measured in minutes, and it reduces load on the target at the same time — which is the cheapest form of politeness there is.

Failure modes and debugging

Performance work introduces its own failure class, and the error strings are specific enough to diagnose from a log line.

page.goto: Timeout 30000ms exceeded. after switching to networkidle means the page never goes quiet — a heartbeat poll, an open WebSocket, or a video player is enough. Switch to 'domcontentloaded' plus an element wait.

locator.click: Timeout 30000ms exceeded. appearing right after you add a block rule means the application depends on the blocked resource. A lazy-loading gallery that waits for an image onload before revealing its container is the common culprit; unblock image for that host or wait on the underlying data instead of the visual state.

Target page, context or browser has been closed and Target crashed almost always mean memory. In Docker, the default 64 MB /dev/shm is far too small for Chromium; run the container with --shm-size=1gb, or pass --ipc=host, or launch with --disable-dev-shm-usage. The container recipe is in Dockerizing Playwright for Headless CI.

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory is your own process, not the browser: you are accumulating every extracted record in one array. Flush to disk or a database per batch and keep only a cursor in memory.

Error: spawn EAGAIN or EMFILE: too many open files means you are launching browsers rather than contexts, or leaking pages. Count live pages with context.pages().length and confirm it stays flat over a long run.

Silent slowdowns are harder. If page times creep upward over a run, you are leaking — check for page.on() handlers registered inside a loop, which accumulate on a reused page and are re-invoked for every subsequent request. When a specific page is inexplicably slow, capture a trace for just that URL and read the network panel; Trace Viewer & Debugging shows how to step through the recorded timeline and see which request stalled.

CI/CD and production considerations

Scheduled scrapers live in containers, and containers lie about their resources. os.cpus().length inside a Kubernetes pod reports the host's core count, not the pod's CPU limit, so a pool sized from it will oversubscribe by an order of magnitude and produce cascading timeouts. Read the cgroup quota, or simply pin the pool size in configuration per deployment.

Build the image from mcr.microsoft.com/playwright at a tag matching your Playwright version exactly — a mismatch between the npm package and the bundled browsers produces browserType.launch: Executable doesn't exist at /ms-playwright/.... Cache the browser download between builds rather than reinstalling it on every run; the CI/CD Integration guide covers the caching keys.

Instrument the run itself with four numbers: pages per minute, p95 page duration, error rate by class, and bytes transferred per page. A regression in bytes per page is usually a block rule that stopped matching after a site redesign, and it will show up days before anyone notices the job getting slower. Emit these as structured log lines so they can be graphed without parsing prose.

Make the job resumable and interruptible. Persist a cursor — the last completed URL or page index — after every batch, so a pod eviction costs one batch rather than the whole run. Handle SIGTERM by draining in-flight pages, flushing the buffer, and calling browser.close(); an orphaned Chromium in a terminating pod keeps its memory until the node reaps it. Wrap every browser lifecycle in try/finally so a thrown error cannot leak a process.

Finally, keep back-pressure in the design. If the sink is a database and it slows down, an unbounded queue of extracted rows in memory will kill the process before anything alerts. Cap the buffer, block workers when it is full, and let throughput degrade gracefully instead of failing catastrophically. Pacing under load ties back into Anti-Bot Defenses & Rate Limiting, because a scraper that backs off when the target struggles is also a scraper that does not get blocked.

Deep dives beneath this guide

• Blocking Images and Fonts to Speed Up Scraping — the exact route() filters that strip images, fonts, and media without breaking the extraction you depend on.
• Running Parallel Scrapers with Worker Pools — building a queue-driven pool with retries, per-worker contexts, and a pool size derived from real machine limits.
• Rotating Proxies in Playwright — assigning egress addresses per context, evicting dead gateways, and holding sticky sessions across multi-step flows.

Together with Pagination & Infinite Scroll, which governs how far each traversal runs, these three cover the full production surface of a high-volume extraction system.

Frequently Asked Questions