Respecting robots.txt and Crawl Delays

Most scrapers that get blocked were never malicious — they simply never asked permission. A site publishes its rules for automated agents at /robots.txt, and RFC 9309 defines exactly how a client is meant to fetch, cache, and interpret that file. Playwright gives you everything needed to do this properly: an APIRequestContext that can fetch the file without launching a page, fixtures that let you enforce a delay before every navigation, and page.route() as a last line of defence against a URL that slips through. This page shows how to fetch and cache robots.txt per origin, choose the correct user-agent group, resolve a path with the longest-match rule, and turn a Crawl-delay directive into a per-host scheduler that your whole run obeys. It is the detailed procedure beneath Anti-Bot Defenses & Rate Limiting, part of Web Scraping & Data Extraction.

Root cause: robots.txt is a protocol, not a courtesy note

robots.txt is a machine-readable access-control document scoped to one origin — scheme, host, and port together — and RFC 9309 specifies its grammar, its matching semantics, and how each HTTP status class must be interpreted. Scrapers get this wrong in three predictable ways: they never fetch the file at all, they fetch it and apply the first Disallow line they see rather than the longest matching rule, or they read Crawl-delay and then ignore it because nothing in their code enforces pacing. The result looks identical from the server's side to a deliberate abuse pattern, which is how a run that would have finished cleanly ends in a 429 storm or an IP-level block. Getting it right is mechanical work, and every piece of it fits inside a Playwright fixture.

Minimal reproducible example

The test below fetches robots.txt with Playwright's request API, parses it into user-agent groups, and resolves a single path with the longest-match rule. Save the two exported helpers as robots.ts — the later snippets import them.

import { test, expect, request as apiRequest } from '@playwright/test';

type Rule = { allow: boolean; pattern: string };
type Group = { agents: string[]; rules: Rule[]; crawlDelaySec?: number };

// Escape everything a RegExp treats specially, so only * and $ stay meaningful.
const escapeRe = (s: string) => s.replace(/[.+?^${}()|[\]\\]/g, '\\$&');

export function parseRobots(body: string): Group[] {
  const groups: Group[] = [];
  let current: Group | null = null;
  let acceptingAgents = false; // consecutive User-agent lines form ONE group
  for (const raw of body.split(/\r?\n/)) {
    const line = raw.split('#')[0].trim();      // strip comments, then blanks
    if (!line) continue;
    const idx = line.indexOf(':');
    if (idx < 0) continue;                       // malformed lines are skipped
    const field = line.slice(0, idx).trim().toLowerCase();
    const value = line.slice(idx + 1).trim();
    if (field === 'user-agent') {
      if (!acceptingAgents) {                    // a rule closed the last header
        current = { agents: [], rules: [] };
        groups.push(current);
        acceptingAgents = true;
      }
      current!.agents.push(value.toLowerCase());
      continue;
    }
    if (!current) continue;                      // directives before any group
    acceptingAgents = false;
    if (field === 'allow') current.rules.push({ allow: true, pattern: value });
    if (field === 'disallow') current.rules.push({ allow: false, pattern: value });
    if (field === 'crawl-delay') current.crawlDelaySec = Number(value);
  }
  return groups;
}

// Returns the pattern's length when it matches, or -1 when it does not.
// Length is the specificity score RFC 9309 uses to rank competing rules.
function matchLength(pattern: string, path: string): number {
  if (pattern === '') return -1;                 // "Disallow:" with no value = no rule
  const anchored = pattern.endsWith('$');        // $ pins the end of the path
  const body = anchored ? pattern.slice(0, -1) : pattern;
  const re = new RegExp(
    '^' + body.split('*').map(escapeRe).join('.*') + (anchored ? '$' : ''),
  );
  return re.test(path) ? body.length : -1;
}

export function isAllowed(groups: Group[], token: string, path: string): boolean {
  const lower = token.toLowerCase();
  // An exact product-token group wins outright; otherwise fall back to "*".
  const group =
    groups.find((g) => g.agents.includes(lower)) ??
    groups.find((g) => g.agents.includes('*'));
  if (!group) return true;                       // no applicable group = allowed
  let best = { len: -1, allow: true };
  for (const rule of group.rules) {
    const len = matchLength(rule.pattern, path);
    if (len < 0) continue;
    // Longer pattern wins; on an exact tie, Allow beats Disallow.
    if (len > best.len || (len === best.len && rule.allow)) {
      best = { len, allow: rule.allow };
    }
  }
  return best.allow;
}

test('resolves a path against the site rules', async () => {
  const ctx = await apiRequest.newContext();
  const res = await ctx.get('https://example.com/robots.txt');
  expect(res.status()).toBe(200);
  const groups = parseRobots(await res.text());
  expect(isAllowed(groups, 'acme-research-bot', '/catalog/items')).toBe(true);
  expect(isAllowed(groups, 'acme-research-bot', '/admin/users')).toBe(false);
  await ctx.dispose();
});

Step-by-step fix

1. Fetch robots.txt once per origin and cache the result. Build the URL by taking the target's scheme, host, and port and appending /robots.txt. Fetch it with an APIRequestContext created from request.newContext() rather than a browser page — you need bytes, not a rendered document, and skipping the browser saves a full navigation. Store the parsed result in a Map keyed on the origin string, honour any max-age in the response Cache-Control header, and re-fetch after 24 hours at most. Persisting the cache in a worker-scoped fixture, as described in Worker-Scoped Fixtures for Expensive Setup, keeps one fetch per worker instead of one per test.
2. Decide policy from the HTTP status before you parse anything. A 200 means parse the body. A redirect means follow it — RFC 9309 asks clients to follow at least five hops before giving up. A 4xx other than 429 means the file is unavailable and every path is permitted. A 429 or any 5xx means the file is unreachable, and the correct interpretation is a complete disallow until you can fetch it again; reuse the last known-good copy if you have one rather than assuming free rein. Handling those transient statuses is the same discipline covered in Handling Rate Limits and Retries When Scraping.
3. Parse into groups and select the one matching your product token. A group is a run of consecutive User-agent lines followed by its rules; the next User-agent line after a rule starts a new group. Groups are never merged. Compare case-insensitively against the product token in the User-Agent header you actually send, take the exact match if one exists, and fall back to the * group only when no specific group names you. A scraper that evaluates the * rules while sending a token the site has written a stricter group for is reading the wrong contract.
4. Resolve each path with the longest-match rule. Score every Allow and Disallow pattern in the selected group against the URL's path plus query string. * matches any run of characters and a trailing $ anchors the end of the path; matching is byte-wise and case-sensitive, so percent-encode the path the same way the file does. The rule with the longest pattern wins regardless of where it sits in the file, and when two patterns are the same length Allow beats Disallow. An empty Disallow: value is not a rule at all — it is the idiom for permitting everything.
5. Turn Crawl-delay into a per-host minimum interval. If the selected group carries a Crawl-delay, multiply it by 1000 and treat the result as the floor between two requests to that host. Some sites publish Request-rate: 1/10s instead, which expresses the same budget; convert it to milliseconds and take whichever value is larger. When neither directive is present, pick your own conservative default — one request per second per host is a defensible starting point — and lower your concurrency ceiling to match, using the patterns in Scraper Performance & Scaling.
6. Gate every navigation through a per-host scheduler. Reading the delay changes nothing unless a queue enforces it. Keep a map of host to next-free timestamp, reserve the slot synchronously before awaiting, then sleep for the remaining interval. Reserving before the await is what makes the scheduler correct under concurrency: two callers that arrive in the same tick get consecutive slots instead of the same one. Expose it as a fixture so tests call politeGoto(url) and cannot forget the wait.
7. Enforce the verdict at the route layer and fail loudly. Add a page.route() handler that aborts any document request whose path the rules disallow, so a stray link or a redirect chain cannot escape the check. Aborting with route.abort('blockedbyclient') produces page.goto: net::ERR_BLOCKED_BY_CLIENT, an unambiguous signal in logs and traces. Send a User-Agent naming your automation and a contact URL, and make sure the token in it is the token you matched in step 3.

The scheduler is small enough to read in one pass, and the reservation-before-await detail is the only subtle part:

import { test as base, expect } from '@playwright/test';
import { parseRobots, isAllowed } from './robots';

class HostScheduler {
  private nextFreeAt = new Map<string, number>();
  constructor(private readonly delayMs: number) {}

  async acquire(url: string): Promise<void> {
    const { host } = new URL(url);
    const now = Date.now();
    // Earliest moment this host may be hit again; default to "right now".
    const earliest = Math.max(now, this.nextFreeAt.get(host) ?? now);
    // Reserve the NEXT slot before awaiting, so concurrent callers queue up
    // instead of all reading the same timestamp and firing together.
    this.nextFreeAt.set(host, earliest + this.delayMs);
    const waitMs = earliest - now;
    if (waitMs > 0) await new Promise((r) => setTimeout(r, waitMs));
  }
}

const test = base.extend<{}, { politeGoto: (url: string) => Promise<void> }>({
  politeGoto: [
    async ({ browser }, use) => {
      const ctx = await browser.newContext({
        userAgent: 'acme-research-bot/1.0 (+https://acme.example/bot)',
      });
      const page = await ctx.newPage();
      const res = await page.request.get('https://example.com/robots.txt');
      // 4xx (except 429) means "unavailable" — every path is permitted.
      const groups = res.status() === 200 ? parseRobots(await res.text()) : [];
      const delayMs = 1000; // replace with Crawl-delay * 1000 when present
      const scheduler = new HostScheduler(delayMs);

      await use(async (url: string) => {
        const { pathname, search } = new URL(url);
        if (!isAllowed(groups, 'acme-research-bot', pathname + search)) {
          throw new Error(`robots.txt disallows ${pathname}`);
        }
        await scheduler.acquire(url); // blocks until this host's slot opens
        await page.goto(url);
      });
      await ctx.close();
    },
    { scope: 'worker' },
  ],
});

test('paces two fetches to the same host', async ({ politeGoto }) => {
  const started = Date.now();
  await politeGoto('https://example.com/catalog/items?page=1');
  await politeGoto('https://example.com/catalog/items?page=2');
  expect(Date.now() - started).toBeGreaterThanOrEqual(1000);
});

Troubleshooting variants

The crawl delay pushes the test past its timeout

A 10-second delay across 20 URLs is 200 seconds of deliberate waiting, and Playwright will cut it short with Test timeout of 30000ms exceeded. — the delay is working exactly as intended and the budget is wrong. Raise it for the affected spec with test.setTimeout(300_000), or set a higher timeout on that project in playwright.config.ts. Do not shrink the delay to fit the timeout; that inverts the priority. If the wall-clock cost is genuinely unacceptable, spread the work across hosts rather than compressing it against one — a scheduler keyed on host lets ten different origins proceed in parallel while each stays paced, which is the shape described in Running Parallel Scrapers with Worker Pools.

robots.txt returns 200 but the body is an HTML error page

Single-page-app hosting frequently serves the index document for any unmatched route, so /robots.txt answers 200 with <!doctype html>. A naive parser finds no :-delimited directives, produces zero groups, and your code concludes that everything is permitted — a silent failure with real consequences. Guard against it: check that res.headers()['content-type'] starts with text/plain, and treat a body whose first non-comment line does not parse as a directive as an unavailable file rather than an empty rule set. Log the anomaly loudly, because an origin whose rules you cannot read deserves your most conservative pacing, not your least.

Rules from the wrong origin get applied

The cache key must be the full origin. https://example.com and https://www.example.com are different origins with potentially different files, and so are http:// and https:// variants and any non-default port. Keying the cache on the registrable domain — or worse, caching a single global rule set — makes a scraper apply a permissive apex-domain file to a locked-down subdomain. The same trap appears when traffic is routed through changing egress points, since a geo-targeted site can serve different files to different regions; if you use the techniques in Rotating Proxies in Playwright, key the cache on origin plus proxy identity and re-fetch when the route changes.

Verification

Confirm the implementation on three axes. For correctness of matching, write unit assertions against a fixture file that exercises the awkward cases: an Allow that is longer than a competing Disallow, a pattern ending in $, an empty Disallow:, and a URL whose query string participates in the match. These are pure functions, so they run in milliseconds and catch the ranking bugs that are otherwise invisible until a site complains. For status handling, stub the robots.txt response with Intercepting and Modifying Network Requests and assert that a 503 yields a total disallow while a 404 yields a total allow — a pair of tests most scrapers never write and most scrapers get backwards.

For pacing, measure it rather than trusting it. Record Date.now() around each politeGoto call and assert the gap meets the configured floor, then run the same suite with --workers=4 and confirm the floor still holds per host; a scheduler that reserves its slot after awaiting will pass at one worker and fail at four. Finally, run the scrape once with --trace on and open the result in the Playwright Trace Viewer: the network panel shows every request the browser actually issued with its timing, which is the only evidence that no disallowed URL slipped past the route guard and that the spacing you configured is the spacing the server saw.

Frequently Asked Questions