Scraping Cursor-Based Pagination APIs

Modern listing endpoints have largely abandoned ?page= and ?offset= in favour of an opaque cursor: the server hands back a token such as eyJpZCI6MTAyfQ== alongside each batch, and you send that exact token back to ask for the next one. Stripe, Slack, Shopify, GitHub and every Relay-style GraphQL schema work this way. The token is deliberately meaningless to you — you cannot compute the next one, skip ahead, or jump to "page 7". A scraper that treats a cursor like a page number produces silent duplicates, endless loops, or a crawl that stops at batch two with 400 invalid_cursor. This page shows how to read a cursor contract off the wire with Playwright, drive the loop through page.request so it inherits the browser's session, and terminate on a signal the server actually gives you.

Root cause: the cursor is state you do not own

An offset is arithmetic you can perform locally, so a scraper can guess the next request. A cursor is server state — usually a base64-encoded sort key plus a tiebreaker id, sometimes an encrypted blob with an expiry — and the only valid next value is the one the previous response contained. Every failure mode on this page traces back to a loop that manufactures, mangles, or fails to advance that token, because unlike an offset there is no way for the client to detect that the value is wrong before the server rejects it. This page sits under Pagination & Infinite Scroll, within Web Scraping & Data Extraction.

Reading the cursor contract off the wire

Before writing a loop, find out which of the three common conventions the endpoint uses. Open the listing page in headed mode, let it load one batch, and inspect the response body — the Playwright Trace Viewer network tab shows the full JSON without any extra tooling. What you are looking for is two things: the field carrying the next token, and the field that tells you to stop. They are frequently not the same field, and trusting the wrong one is the single most common source of truncated datasets.

Minimal reproducible example

This test reproduces the two failures that account for most broken cursor crawls: a token corrupted by naive string interpolation, and a loop that never advances because the response echoes the cursor it was given. Run it against any endpoint returning base64 cursors and it will collect the same batch fifty times.

import { test, expect } from '@playwright/test';

test('naive cursor loop silently collects the same batch', async ({ page }) => {
  await page.goto('https://example.com/catalog');

  // Sniff the listing XHR the page fires on load to learn the response shape.
  const first = await page.waitForResponse(
    (r) => r.url().includes('/api/v1/items') && r.status() === 200,
  );
  const body = await first.json();
  // { data: [...], next_cursor: 'eyJpZCI6MTAyfQ==', has_more: true }

  let cursor: string | null = body.next_cursor;
  const rows: unknown[] = [...body.data];

  for (let i = 0; i < 50 && cursor; i++) {
    // FAULT 1: the token is base64 and contains '=' and '+'. Interpolating it
    // raw into the query string corrupts it, and the API answers
    // 400 {"error":"invalid_cursor"} instead of the next batch.
    const res = await page.request.get(`/api/v1/items?limit=100&after=${cursor}`);
    const batch = await res.json();

    // FAULT 2: no status check. On a 400 many APIs echo the request cursor back,
    // so `cursor` never changes and every iteration re-fetches the same rows.
    rows.push(...batch.data);
    cursor = batch.next_cursor;
  }

  // Passes with 5000 rows, of which 100 are distinct.
  expect(rows.length).toBeGreaterThan(0);
});

Step-by-step fix

1. Capture the contract from a real page load. Navigate with the browser, then await page.waitForResponse() on the listing endpoint and read the JSON. This tells you the token field, the stop field, the maximum accepted limit, and whether the endpoint requires headers the browser adds for you — knowledge you cannot get by guessing at the URL.
2. Issue the loop through page.request, not a bare fetch client. page.request is an APIRequestContext bound to the page's browser context, so it sends the same cookies, Origin, and proxy configuration the UI does. Requests made this way authenticate exactly as the logged-in session does, which is the mechanism behind scraping data behind login sessions.
3. Pass the cursor through the params option instead of interpolating it. page.request.get(url, { params: { after: cursor } }) percent-encodes the value, so =, + and / inside a base64 token survive the trip. This one change eliminates the 400 invalid_cursor class of failure outright.
4. Advance strictly on the token the server returned. Read next_cursor (or pageInfo.endCursor) from the parsed body of the response you just received, never from a variable you mutated elsewhere. If the field is absent, null, or an empty string, the crawl is finished — treat all three as terminal.
5. Guard the loop with a seen-cursor set and a hard cap. Record every cursor you have already sent in a Set. If a response hands back a token already in that set, the server is looping and you should throw rather than spin. Pair it with a maximum iteration count so a pathological endpoint cannot run your job forever.
6. Check response.ok() before parsing and handle 429 explicitly. A non-2xx body is not a batch. On 429 Too Many Requests, honour the Retry-After header and retry the same cursor; on 401 refresh the session and retry once; on anything else, throw with the status and the first 200 characters of the body attached. The backoff mechanics are covered in handling rate limits and retries when scraping.
7. Flush each batch to disk and keep only the cursor in memory. Append rows to an NDJSON file per iteration and persist the last successful cursor beside them. A crawl of 400,000 records then costs constant memory, and a crash resumes from the saved token instead of restarting.

The corrected loop puts all seven rules in one place:

import { test, expect, type APIRequestContext } from '@playwright/test';
import { appendFileSync, writeFileSync } from 'node:fs';

type Batch = { data: Record<string, unknown>[]; next_cursor: string | null; has_more: boolean };

async function fetchBatch(api: APIRequestContext, cursor: string | null): Promise<Batch> {
  // params encodes the opaque token; limit is the server's documented maximum.
  const res = await api.get('/api/v1/items', {
    params: cursor ? { limit: 100, after: cursor } : { limit: 100 },
  });

  if (res.status() === 429) {
    // Retry-After is seconds per RFC 6585; fall back to 5s when absent.
    const wait = Number(res.headers()['retry-after'] ?? 5) * 1000;
    await new Promise((r) => setTimeout(r, wait));
    return fetchBatch(api, cursor); // same cursor, not the next one
  }
  if (!res.ok()) {
    throw new Error(`items ${res.status()}: ${(await res.text()).slice(0, 200)}`);
  }
  return res.json() as Promise<Batch>;
}

test('walks every cursor page exactly once', async ({ page }) => {
  await page.goto('https://example.com/catalog'); // establishes the session cookies

  const seen = new Set<string>();
  let cursor: string | null = null;
  let total = 0;

  for (let i = 0; i < 5_000; i++) {
    const batch = await fetchBatch(page.request, cursor);

    appendFileSync('items.ndjson', batch.data.map((r) => JSON.stringify(r)).join('\n') + '\n');
    total += batch.data.length;

    // Terminate on the server's own signal; a missing token corroborates it.
    if (!batch.has_more || !batch.next_cursor) break;

    // Loop guard: a repeated token means the endpoint is not advancing.
    if (seen.has(batch.next_cursor)) {
      throw new Error(`cursor repeated after ${total} rows: ${batch.next_cursor}`);
    }
    seen.add(batch.next_cursor);

    cursor = batch.next_cursor;
    writeFileSync('resume-cursor.txt', cursor); // crash-resume checkpoint
  }

  expect(total).toBeGreaterThan(0);
});

Troubleshooting variants

The crawl stops after two batches with 400 invalid_cursor

Almost always an encoding problem. Base64url is safe in a query string, but plain base64 is not: + decodes to a space and = can be read as a parameter delimiter, so after=eyJpZCI6MTAyfQ== arrives at the server truncated. Use the params option so Playwright encodes the value, and check the exact bytes you sent by opening the trace and comparing the request URL against the token in the previous response body. If the API instead expects the cursor in a header or a POST body, page.request.post() with a data object avoids the query string entirely. A second, rarer cause is cursor expiry: some services sign tokens with a short TTL and reject them minutes later with the same status, which shows up as a crawl that dies at a consistent elapsed time rather than a consistent batch number.

The loop runs forever even though the loop guard is in place

Check what you are adding to the Set. If the endpoint returns a fresh token each time but the underlying rows repeat, the guard never trips because no cursor is literally reused — this happens when a filter parameter is dropped between requests and the API silently restarts from the beginning. Add a second guard on record identity: hash the first record's primary key per batch and abort when the same key reappears. Also verify that you are not resetting cursor to null inside the loop body, which restarts the traversal from the first batch on every iteration and looks identical from the outside.

Requests succeed in the UI but return 401 from page.request

page.request shares cookies with the page, but not everything the front end sends. Single-page applications commonly attach a bearer token from memory or a CSRF token from a meta tag, neither of which is a cookie. Capture the header from the observed request — first.request().headers()['authorization'] — and pass it through extraHTTPHeaders when you build the context, or re-read it from the DOM. A 401 that appears only after several thousand rows is a different problem: the session expired mid-crawl, so re-navigate to refresh it and resume from the checkpointed cursor rather than starting over.

Verification

Three checks prove the crawl was complete and not merely long. First, assert uniqueness on the primary key: load the NDJSON output, build a Set of ids, and confirm its size equals the line count — any shortfall means the loop re-fetched a batch. Second, compare the total against an authoritative count. Many cursor endpoints return total_count on the first response, and the listing UI usually renders "12,481 results"; extracting that figure with the same techniques used for extracting tables and lists to JSON gives you an independent number to reconcile against.

Third, inspect the request sequence rather than trusting the result count. Run once with --trace on and open the trace; every page.request call appears in the network panel with its full URL, so you can confirm each after parameter differs from the last and that the final response carried has_more: false instead of an error the loop swallowed. For a regression test that does not touch the network at all, replay the sequence against stubbed responses using the patterns in mocking API responses with Playwright — three fixture batches, the last one terminal, is enough to pin the loop's exit behaviour permanently. That test belongs in CI, where a change to the cursor contract will otherwise surface as a quietly shrinking dataset.

Frequently Asked Questions