Anti-Bot Defenses & Rate Limiting

Servers protect themselves from automated traffic for good reasons: a careless scraper can degrade a site for real users, distort analytics, and run up someone else's infrastructure bill. The professional response is not to fight those defenses but to be the kind of client they are happy to serve — one that reads robots.txt, stays under documented rate limits, slows down when asked, and never hammers an endpoint in a tight loop. This guide treats scraping as reliability engineering: how to pace requests, cap concurrency, back off exponentially when a server signals overload, and handle 429 and 503 responses correctly. The goal throughout is respectful, robust automation. It sits under Web Scraping & Data Extraction and is detailed step by step in Handling Rate Limits and Retries When Scraping.

How servers meter traffic

Before you can pace politely, it helps to understand what you are pacing against. Almost every rate limiter reduces to one of three shapes, and knowing which one a host uses changes how you should time your requests. A fixed window counts requests per calendar interval — say 100 per minute — and resets the counter at the top of each minute. It is simple but bursty: a client that spends its whole budget in the first two seconds and then goes quiet still trips the limit, and two clients straddling a window boundary can briefly double the intended rate. A sliding window smooths that by counting requests over a rolling interval, so there is no cliff edge to exploit or accidentally hit. A token bucket is the most common on production APIs: the bucket holds a fixed number of tokens, each request spends one, and tokens refill at a steady rate. Bursts are allowed up to the bucket's capacity, but the sustained rate you can hold is exactly the refill rate.

The practical upshot is that steady, evenly spaced traffic is the safest posture against every one of these algorithms. A burst that would be fine against a token bucket with spare capacity is exactly what trips a fixed window, so pacing to a constant interval — rather than firing a batch and waiting — keeps you under all three at once. The diagram below contrasts an unpaced burst that exhausts a window in the first second with a paced stream that spreads the same work across the interval and never crosses the line.

Prerequisites

This guide assumes a working Playwright install and a project that already runs tests or scripts against a live target. The pacing and backoff patterns here are plain TypeScript that runs inside any @playwright/test file or a standalone script, so no extra dependency is required. When you parallelize, you will lean on Browser Contexts & Isolation to keep worker sessions separate, and when you inspect responses you will use the same request and response objects covered in Network Interception Basics. Have the target's robots.txt and any published API documentation open before you start — the limits they state are the numbers your pacing should be built around.

Read robots.txt and honor its rules

The first request a respectful scraper makes is for robots.txt. It tells you which paths the site owner permits automated agents to fetch and, increasingly, a Crawl-delay that sets a minimum interval between requests. Treat both as binding: skip disallowed paths entirely, and if a crawl delay is present, make it the floor of your pacing. Many sites also publish terms of service and a documented public API; where an API exists, prefer it over scraping the rendered page, because it is the channel the owner built for programmatic access and it usually carries explicit rate-limit headers. Reading and obeying robots.txt costs one request and is the clearest signal that your automation is acting in good faith. Parsing the file correctly — matching the right user-agent group, resolving the most specific rule, and reading a Crawl-delay into your pacer — has enough detail to warrant its own walkthrough in Respecting robots.txt and Crawl Delays.

A subtlety worth internalizing: a missing or unreachable robots.txt is not a green light. A 404 conventionally means no restrictions are published, but a 5xx or a timeout means you could not determine the rules, and the conservative reading is to hold off rather than assume permission. Cache the parsed result for the duration of a run so you are not re-fetching the policy on every request, but do not cache it forever across days — site owners update these files, and yesterday's permission may be today's disallow.

Pace requests deliberately

The single most important habit is to insert a deliberate, bounded delay between requests rather than firing them as fast as the event loop allows. A fixed minimum interval — say one request per second, or whatever the site's crawl delay specifies — keeps your traffic indistinguishable from steady, low-volume use and protects the server's headroom. Pacing also makes runs reproducible: timing-dependent failures vanish when you are not racing the server. Add small randomization to the interval so requests are not robotically periodic, which both smooths bursts and avoids synchronizing with the server's own measurement windows.

import { test } from '@playwright/test';

// A simple token-paced fetch: never send faster than one request per interval.
class Pacer {
  private last = 0;
  constructor(private minIntervalMs: number) {}
  async wait(): Promise<void> {
    const elapsed = Date.now() - this.last;
    const remaining = this.minIntervalMs - elapsed;
    // Sleep only if we are ahead of schedule.
    if (remaining > 0) await new Promise((r) => setTimeout(r, remaining));
    this.last = Date.now();
  }
}

test('paces requests to one per second', async ({ page }) => {
  const pacer = new Pacer(1000); // honor at least a 1s gap
  for (const path of ['/p/1', '/p/2', '/p/3']) {
    await pacer.wait(); // block until the interval has elapsed
    await page.goto(path);
  }
});

The randomization worth adding is a small positive jitter on top of the floor, not a symmetric spread that could dip below the crawl delay. Compute the base interval from the site's stated limit, then add a random 0–300 ms so successive requests land at, say, 1.0–1.3 seconds apart. That keeps you strictly above the minimum while breaking up any periodic signature. Resist the temptation to shrink the interval when a run feels slow: pacing is a budget you spend against the server's patience, and the cost of a block is far higher than the cost of a slower run.

Cap concurrency

Parallelism is where good intentions turn into accidental denial of service. Twenty workers each firing without coordination can hit a server with twenty simultaneous requests, far above any reasonable limit. Cap the number of in-flight requests with a small concurrency pool — often two to four for a single host — and let workers wait for a slot before proceeding. When you do parallelize, isolate each worker with its own Browser Contexts & Isolation so sessions and cookies do not bleed across requests, and keep the aggregate rate, not just the per-worker rate, under the documented limit.

import { test, expect } from '@playwright/test';

// Run an array of async tasks with at most `limit` running at once.
async function withConcurrency<T>(
  items: T[],
  limit: number,
  worker: (item: T) => Promise<void>,
): Promise<void> {
  const queue = [...items];
  // Spawn exactly `limit` runners that each pull from the shared queue.
  const runners = Array.from({ length: limit }, async () => {
    let next: T | undefined;
    while ((next = queue.shift()) !== undefined) {
      await worker(next);
    }
  });
  await Promise.all(runners);
}

test('caps in-flight requests at three', async ({ browser }) => {
  const urls = ['/a', '/b', '/c', '/d', '/e'];
  await withConcurrency(urls, 3, async (url) => {
    const ctx = await browser.newContext(); // isolate each worker
    const page = await ctx.newPage();
    await page.goto(url);
    await ctx.close();
  });
  expect(urls.length).toBe(5);
});

The distinction between per-worker and aggregate rate is the one most scrapers get wrong. Four workers each pacing to one request per second is four requests per second at the host, which may be triple the crawl delay. The fix is to share a single pacer across the pool so the interval is enforced globally rather than per worker, or to divide the per-worker interval by the pool size. When you scale beyond a handful of workers, the coordination problem becomes its own topic, covered in Running Parallel Scrapers with Worker Pools under Scraper Performance & Scaling.

Back off when the server signals overload

A 429 (Too Many Requests) or 503 (Service Unavailable) is the server telling you to slow down. The correct response is to wait and retry, not to repeat immediately. If the response includes a Retry-After header, that value is authoritative — wait exactly that long. Otherwise, use exponential backoff: double the wait after each failed attempt, and add jitter so a fleet of clients does not retry in lockstep and create a thundering herd. Cap the number of retries and the maximum delay so a persistently failing endpoint surfaces as an error instead of looping forever. The full implementation — detecting 429, reading Retry-After, exponential backoff with jitter, and throttling — is in Handling Rate Limits and Retries When Scraping.

Modeling a request as a small state machine keeps the logic honest. A request is dequeued, waits for a slot, goes in flight, and then resolves one of two ways: a success terminates the cycle, or a throttle response sends it into a backoff state that waits and re-enters the queue — until it either succeeds on a later attempt or exhausts its retry budget and fails for good. Drawing the states explicitly makes the two things that trip people up obvious: backoff is not a dead end but a loop back to the queue, and there must be a terminal failure edge so the loop cannot run forever.

Read rate-limit headers to pace adaptively

Fixed pacing is a safe floor, but many APIs tell you exactly how much budget remains, and reading those signals lets you run as fast as the server permits without ever crossing the line. The common headers are X-RateLimit-Limit (the ceiling per window), X-RateLimit-Remaining (how many requests you have left), and X-RateLimit-Reset (when the window refills, either as a Unix timestamp or as seconds from now). An adaptive pacer reads these on every response and stretches its interval as the remaining budget shrinks, so you glide down to a near-stop just before the window resets rather than slamming into a 429. Because Playwright's request context exposes the full response, you can read these headers directly without intercepting network traffic.

import { test, expect, request } from '@playwright/test';

// Adaptive pacer: read rate-limit headers and stretch the gap as budget runs low.
test('adapts pacing to rate-limit headers', async () => {
  const api = await request.newContext({ baseURL: 'https://api.example.com' });
  let delayMs = 250; // start optimistic while budget is plentiful
  for (const id of ['1', '2', '3', '4', '5']) {
    const res = await api.get(`/items/${id}`);
    expect(res.status()).toBeLessThan(500); // surface server errors loudly

    const headers = res.headers();
    const remaining = Number(headers['x-ratelimit-remaining'] ?? '999');
    const resetSec = Number(headers['x-ratelimit-reset'] ?? '0');

    // When few requests remain, spread them evenly across the time left in the window.
    if (remaining <= 5 && resetSec > 0) {
      delayMs = Math.ceil((resetSec * 1000) / Math.max(remaining, 1));
    }
    await new Promise((r) => setTimeout(r, delayMs)); // hold the computed gap
  }
  await api.dispose(); // release the context and its sockets
});

Header names are case-insensitive and Playwright lowercases them, so always read them in lowercase. Not every API uses the X-RateLimit-* convention — GitHub does, but others use RateLimit-Remaining (the draft IETF standard) or bespoke names — so make the header keys configurable rather than hard-coded. When the headers are absent entirely, the pacer should fall back to your fixed floor, never to zero delay. Treat the adaptive path as an optimization layered on top of the conservative default, not a replacement for it.

Choose a backoff strategy deliberately

Not every retry policy is equally kind to a struggling server. A fixed delay retries at the same interval every time, which recovers slowly and, worse, tends to synchronize a fleet of clients into repeated simultaneous waves. Linear backoff grows the wait by a constant each attempt, which is gentler but still predictable. Exponential backoff doubles the wait each time, so a healthy endpoint recovers fast while a failing one is quickly given room. The decisive addition is jitter: randomizing each wait within its computed range breaks the lockstep that otherwise turns a fleet's retries into a thundering herd against a server that is already down. The combination — exponential growth with jitter — is the default worth reaching for, and the matrix below summarizes why.

Treat retries as a reliability discipline

Backoff and retry logic is the same machinery that stabilizes a flaky end-to-end suite, and the mindset transfers directly from Flaky Test Management: distinguish transient failures (429, 503, network blips) that deserve a retry from deterministic failures (404, 403, a parse error) that do not. Retrying a 404 just wastes the server's time and yours. Log every retry with its reason and delay so a run is auditable, and make retries idempotent so a repeated request never double-submits a side effect. A scraper that retries thoughtfully is both kinder to the server and more reliable for you.

The classification is not always clean, and the edge cases are where judgment matters. A 403 usually means forbidden and should not be retried — but if it appears only after a burst, it can be a soft rate-limit dressed up as an authorization error, in which case backing off once and retrying is reasonable while retrying in a tight loop is not. A 500 is ambiguous: it may be a transient hiccup worth one retry, or a deterministic failure on a specific record that will never succeed, so cap retries tightly and give up early. Timeouts and connection resets are transient by nature and safe to retry, but only if the request is idempotent; retrying a non-idempotent POST after a timeout risks a double write when the first request actually succeeded but the response was lost. When in doubt, retry conservatively and log loudly, so the pattern is visible in your run history rather than hidden inside a silent loop.

Failure modes and debugging

The failure that hurts most is the one you cannot see: a scraper that appears to succeed but is quietly being served decoy content. A soft block often returns a 200 with an empty results page, a CAPTCHA interstitial, or a trimmed dataset, so a status-code check alone passes while your extracted data is garbage. Guard against it by asserting on the shape of the data, not just the response code — if a listing page should contain twenty rows and you extract zero, treat that as a failure worth investigating rather than an empty-but-valid result. Watching your success rate and extracted-row counts over time surfaces a creeping block long before a hard 403 does.

When requests do fail visibly, the Playwright trace is the fastest way to see what the server actually sent. Capturing the response body and headers on a throttled request tells you whether a Retry-After was present, whether the body was a real error or a challenge page, and whether a redirect quietly bounced you to a login wall. Reviewing that evidence in the Playwright Trace Viewer turns a vague "it stopped working" into a specific diagnosis. Two recurring culprits are worth naming: pacing that is measured per worker instead of in aggregate, which multiplies your true rate by the pool size, and a backoff that grows without a cap, which turns a temporary outage into a scraper that sleeps for an hour before failing. Both are visible the moment you log every request's timestamp, status, and computed delay.

CI/CD considerations

A scraper that runs on a schedule needs the same pacing discipline as one you run by hand, but the failure surface is different because no one is watching. Keep the pacing floor and concurrency cap in configuration, not hard-coded, so you can dial them down from a single place when a target tightens its limits. In continuous integration, a run that hits sustained 429s should fail the job with a clear signal rather than silently returning partial data — a scheduled job that reports green while collecting nothing is worse than one that fails visibly. Emit the retry count, the final status distribution, and the extracted-row total as job output so a drop is caught by monitoring. When you containerize the scraper for scheduled runs, the same base image and browser-caching patterns from CI/CD Integration apply, and routing egress through rotating addresses is covered in Rotating Proxies in Playwright. Above all, resist the pressure to shorten a run by shrinking the interval; a CI job that finishes faster by tripling its request rate is trading a durable data source for one deadline.

Identify your client honestly

Send a descriptive User-Agent that names your automation and a contact address, so a site operator who notices your traffic can reach you rather than block you blind. Combined with honoring robots.txt, conservative pacing, and capped concurrency, an honest identity marks your scraper as a cooperative client. This is the opposite of evasion: the aim is to be legible and easy to work with, which is what keeps long-running data collection sustainable.

Deep dives beneath this guide

This guide sets the policy; the deep dive beneath it works through the mechanics in full. Respecting robots.txt and Crawl Delays shows how to fetch and parse the policy file, match the correct user-agent group, resolve the most specific allow or disallow rule, and feed a discovered Crawl-delay straight into your pacer. Pair it with Handling Rate Limits and Retries When Scraping for the numbered walkthrough of detecting a 429, reading Retry-After, and implementing exponential backoff with jitter.

Frequently Asked Questions