Handling Rate Limits and Retries When Scraping

A scraper that ignores rate limits fails twice: it gets blocked, and it burdens the server it depends on. When a site returns HTTP 429 (Too Many Requests) or 503 (Service Unavailable), it is asking you to slow down, and the correct response is to wait the right amount of time and retry — not to repeat the request immediately or abandon the run. This page shows how to detect a rate-limit response, read and honor the Retry-After header, fall back to exponential backoff with jitter when that header is absent, and throttle concurrency so you stay under the limit in the first place. It is the detailed walkthrough beneath Anti-Bot Defenses & Rate Limiting, part of Web Scraping & Data Extraction.

Root cause: the server is rate limiting you

A 429 means you crossed a request budget the server enforces over some window — per second, per minute, or per token. A 503 often means the server is shedding load. Both are transient and both are explicit instructions to back off. Two mistakes turn a recoverable situation into a failed run. The first is retrying immediately, which keeps you over the limit and can extend a temporary throttle into a long block. The second is retrying in a tight, perfectly periodic loop across many workers, which produces a thundering herd that all hammer the server the instant the window resets. The fix is to wait the amount the server asks for, and when it does not specify, to grow the wait exponentially with randomized jitter. This is the same transient-versus-deterministic reasoning that drives Flaky Test Management: retry what is transient, fail fast on what is not.

Not every failure earns a retry. Before any backoff logic runs, a resilient scraper classifies the response it received: transient statuses such as 429 and 503 mean wait and try again, while deterministic statuses such as 403 and 404 mean stop — the identical request will fail the same way no matter how long you wait. Retrying a deterministic failure wastes the server's capacity and your own runtime, and hammering a 403 can deepen a block by piling on load the server has already refused. In code this classification is a single comparison at the top of the retry loop, but getting it wrong is the most common cause of a scraper that either retries forever or gives up too soon. The matrix below is the decision table the helper encodes before it ever computes a delay.

Minimal reproducible example

The helper below uses Playwright's request API to fetch a URL, honors Retry-After when present, and otherwise backs off exponentially with jitter, capping the attempts.

import { test, expect, type APIRequestContext } from '@playwright/test';

async function fetchWithRetry(
  request: APIRequestContext,
  url: string,
  maxAttempts = 5,
): Promise<string> {
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    const res = await request.get(url);
    // Success: return the body immediately.
    if (res.status() === 200) return res.text();

    // Only 429 and 503 are retryable; everything else fails fast.
    if (res.status() !== 429 && res.status() !== 503) {
      throw new Error(`Non-retryable status ${res.status()} for ${url}`);
    }

    // Prefer the server's instruction when it gives one.
    const retryAfter = res.headers()['retry-after'];
    const waitMs = retryAfter
      ? Number(retryAfter) * 1000 // Retry-After is in seconds
      : Math.min(2 ** attempt * 500, 30_000) + Math.random() * 500; // backoff + jitter

    await new Promise((r) => setTimeout(r, waitMs));
  }
  throw new Error(`Exhausted ${maxAttempts} attempts for ${url}`);
}

test('retries a rate-limited endpoint', async ({ request }) => {
  const body = await fetchWithRetry(request, '/api/items');
  expect(body.length).toBeGreaterThan(0);
});

Step-by-step fix

1. Detect the rate-limit status. After each request, inspect res.status(). Treat 429 and 503 as retryable signals to back off. Return immediately on 200, and fail fast on non-retryable codes like 403 or 404 so you do not waste attempts on errors a retry cannot fix.
2. Read Retry-After and honor it exactly. Check res.headers()['retry-after']. When present it is the server's explicit instruction — a number of seconds (multiply by 1000 for milliseconds) or an HTTP date (compute the delay to that time). Waiting exactly this long is both polite and the fastest path back to success.
3. Fall back to exponential backoff. When no Retry-After is given, compute the wait as a base interval times two to the power of the attempt number, so successive waits grow 0.5s, 1s, 2s, 4s, and so on. Cap it at a maximum (for example 30 seconds) so a stubborn endpoint does not produce absurd delays.
4. Add jitter. Add a random component to every computed delay so multiple workers do not retry in lockstep. Jitter spreads retries across the window and prevents a thundering herd from re-overloading the server the instant its limit resets.
5. Cap attempts and surface failure. Bound the loop with a maximum attempt count and throw a clear error when it is exhausted. A scrape that fails loudly after reasonable retries is better than one that loops forever or silently drops records.
6. Throttle concurrency to avoid 429s in the first place. Limit in-flight requests with a small pool and pace each worker, isolating them with Browser Contexts & Isolation. Prevention beats recovery: staying under the limit means fewer backoffs and a faster overall run.

How the backoff schedule grows

Exponential backoff turns one fixed base interval into a rapidly widening gap between attempts, giving a genuinely overloaded endpoint progressively more room to recover while a healthy one is retried almost immediately. Doubling from a 500ms base produces waits of roughly 0.5s, 1s, 2s, 4s, and 8s, and the cap stops the fifth or sixth attempt from ballooning into a minute-long stall. Jitter then widens each step into a small random band, so a fleet of workers that all struck the limit at the same instant scatter their retries across the window instead of colliding a second time. The chart below plots that schedule, one bar per attempt, each topped by the jitter band that keeps the fleet desynchronized.

Troubleshooting variants

Retry-After is an HTTP date, not a number

The header can hold either a delta-seconds integer or an absolute HTTP-date. Detect which: if Number(value) is NaN, parse it with Date.parse(value) and wait parsed - Date.now() milliseconds (clamped to at least zero). Honoring the date form is just as important as the numeric form.

Retries keep failing and the block gets longer

You are likely still over the limit because concurrency is too high or jitter is missing, so workers retry together. Reduce the concurrency pool, increase the base backoff, and confirm jitter is applied to every delay. If a host repeatedly throttles you, lower the steady-state pacing for the whole run rather than relying on retries to absorb it.

The same request runs twice and double-submits

A retry must be idempotent. For GET requests this is automatic, but for any request with a side effect, include an idempotency key or guard so a retried request does not create a duplicate record. When you need to inspect or adjust the outgoing request, combine this with Intercepting and Modifying Network Requests.

Verification

Confirm the retry logic three ways. First, point it at a test endpoint that returns 429 with a Retry-After for the first two calls and 200 after — the helper should succeed on the third attempt having waited the header's interval. Second, log each attempt's status and computed delay and confirm the waits grow exponentially with jitter when no header is present, and match Retry-After when it is. Third, capture a trace with --trace on and review the timing of the retried requests in the Playwright Trace Viewer to verify the spacing between attempts is what you intended.

Frequently Asked Questions