Pagination & Infinite Scroll

A listing page rarely shows all of its records at once. It splits them across numbered pages, hides them behind a "load more" button, or streams them in as you scroll. Extracting the full dataset means driving that mechanism to its end and knowing, with certainty, when there is nothing left to fetch. The hard part is not reading rows — it is the loop control: advancing deterministically, waiting for the new batch to actually arrive, and stopping on a stable signal rather than a guessed timeout. This guide covers the traversal patterns you will meet — numbered pages, load-more buttons, infinite scroll, and cursor-based APIs — the stop conditions that make each one reliable, and how to keep memory bounded when a list runs to thousands of items. It sits under Web Scraping & Data Extraction and feeds the detailed walkthrough in Scraping Infinite Scroll Pages with Playwright.

Loop control is the hard part, not row reading

Every traversal splits into two concerns, and only one of them is difficult. Reading the current batch — turning visible rows into structured records — is a solved problem covered in Structured Data Extraction: you point a locator at the rows and map them to objects. The problem that actually breaks scrapers is the loop that surrounds that read: getting the next batch onto the page, waiting until it has genuinely arrived, and deciding whether to go around again or stop. All of the flakiness, all of the missed rows, and all of the runaway infinite loops live in that surrounding machinery, not in the extraction itself.

It helps to picture the loop as a small state machine with exactly three moving states — fetch the next batch, wait for it to land, and evaluate whether a stop signal is present — plus two terminal outcomes: continue or complete. Each traversal pattern is the same machine wearing different clothes. Numbered pages fetch by navigating a URL; load-more fetches by clicking; infinite scroll fetches by scrolling; a cursor API fetches by sending a token. What differs between them is the arrival signal you wait on and the stop signal you test. Once you see that shared shape, the design questions become concrete: what observable event proves the batch arrived, and what positive condition proves the list is exhausted?

Three properties make a traversal trustworthy. It must be deterministic — the same site produces the same result set, never a partial one because a settle window was a few milliseconds short. It must be idempotent per row — re-reading a page must not double-count records, which matters the moment a list is sorted by recency and items shift position between fetches. And it should be resumable — a crash at page 400 of 900 restarts near page 400, not at page one. Hold those three properties in mind and every decision below follows from them.

Prerequisites

You need Playwright installed and a working grasp of locators and auto-waiting — a locator resolves lazily and retries until the element is actionable, which removes most of the need for manual sleeps. Two supporting techniques recur throughout: synchronizing a read with rows that arrive asynchronously, covered in Handling Dynamic Content, and pairing an action with the response it triggers via waitForResponse(), covered in Network Interception Basics. If a listing exposes a JSON endpoint, intercepting that response is almost always more reliable than watching the DOM.

Pattern one: numbered pages

Numbered pagination is the most extraction-friendly because the page set is addressable. A listing at /products?page=1 continues at ?page=2 and so on, and the total is usually discoverable from a "last" link, a result count, or a 404/empty body past the end. The reliable loop is to navigate to a page, wait for the row container, read it, then either follow the "next" link or increment the query parameter — and stop the moment the next signal is absent.

Two stop conditions are robust. The first is the presence of a "next" control: when getByRole('link', { name: 'Next' }) is no longer present or is disabled, you are on the last page. The second is content identity: if the first row of the new page equals the first row of the previous page, the server clamped you to the last valid page and you should halt to avoid an infinite loop.

import { test, expect, type Page } from '@playwright/test';

// Walk numbered pages until the "Next" control disappears.
async function scrapeAllPages(page: Page): Promise<string[]> {
  const titles: string[] = [];
  for (let pageNum = 1; ; pageNum++) {
    await page.goto(`/products?page=${pageNum}`); // addressable URL per page
    // Wait for the row container, not a timeout, before reading.
    await page.getByRole('row').first().waitFor();
    const rows = await page.getByRole('row').allInnerTexts();
    titles.push(...rows);
    // Stop signal: the Next link is gone on the last page.
    const next = page.getByRole('link', { name: 'Next' });
    if (await next.count() === 0) break;
  }
  return titles;
}

test('collects every product across pages', async ({ page }) => {
  const all = await scrapeAllPages(page);
  expect(all.length).toBeGreaterThan(0);
});

Prefer driving the URL directly over clicking "next" when the parameter is stable: it survives a crashed run (you can resume from the last completed page), it parallelizes across Browser Contexts & Isolation, and it sidesteps client-side state that a click would mutate.

Two edge cases catch people out. The first is result drift: if the listing is sorted by "most recent" and new records are inserted while you page through it, the same item can appear on both page two and page three, or slip from page three back to page two and be missed. Defend against it by deduplicating on a stable record id rather than on row position, and where the site allows it, sort by an immutable key (an id or creation timestamp) instead of a mutable one. The second is a guessed total: never trust a "1,240 results" banner as your loop bound, because the count can be stale or approximate. Drive the loop off the observable next control and treat the banner as a sanity check, not a terminator.

Pattern two: load-more buttons

A "load more" button appends the next batch to the existing list instead of replacing it. The loop is: locate the button, click it, wait for the row count to grow, and repeat until the button is hidden, disabled, or removed from the DOM. The mistake that produces flaky scrapes is clicking again before the previous batch lands — so the wait must key off an observable change, either the network response that delivers the batch or the increased item count.

import { test, expect, type Page } from '@playwright/test';

async function clickThroughLoadMore(page: Page): Promise<number> {
  const items = page.getByTestId('list-item');
  await items.first().waitFor();
  while (true) {
    const before = await items.count();
    const loadMore = page.getByRole('button', { name: 'Load more' });
    // Stop when the control is gone or disabled — the list is exhausted.
    if (await loadMore.count() === 0 || await loadMore.isDisabled()) break;
    await loadMore.click();
    // Wait for the count to grow rather than guessing with a sleep.
    await expect.poll(() => items.count()).toBeGreaterThan(before);
  }
  return items.count();
}

test('exhausts a load-more list', async ({ page }) => {
  await page.goto('/feed');
  const total = await clickThroughLoadMore(page);
  expect(total).toBeGreaterThan(0);
});

When the button triggers an XHR you control, pairing the click with waitForResponse() is even tighter than counting, because it confirms the data arrived before you re-read the DOM. That technique — registering the waiter before the action — is covered under Network Interception Basics and is the same discipline used throughout reliable automation.

Guard two failure modes. Some designs keep a permanently visible button that simply stops loading new rows at the end rather than hiding itself; your count-growth poll will time out on that final click, so treat "clicked but count did not grow" as a legitimate stop condition, not an error. And because Playwright auto-scrolls a target into view before clicking, a button rendered inside its own scrolling panel usually just works — but if the button is re-created on each batch, cache the locator, not a resolved handle, so the next iteration re-resolves the fresh node instead of acting on a detached one.

Pattern three: infinite scroll

Infinite scroll fires new requests when a sentinel element near the bottom enters the viewport, usually via an IntersectionObserver. There is no button and often no page parameter — you advance by scrolling and you stop when the item count stops changing across consecutive scrolls. Because the list may be virtualized (the DOM only holds the visible window while off-screen rows are recycled), you generally cannot read everything from the final DOM state; you read each batch as it appears, or you intercept the data responses directly.

The robust stop condition is a stable count: scroll, wait for either a new data response or a count increase, and break after a fixed number of scrolls that yield no growth. The full numbered walkthrough — scroll loop, waitForResponse(), and a debounced stop — lives in Scraping Infinite Scroll Pages with Playwright.

import { test, expect, type Page } from '@playwright/test';

async function scrollToEnd(page: Page, maxIdleRounds = 3): Promise<number> {
  const items = page.getByTestId('card');
  await items.first().waitFor();
  let idle = 0;
  while (idle < maxIdleRounds) {
    const before = await items.count();
    // Scroll the document to the bottom to trip the IntersectionObserver.
    await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
    // Give the observer a chance; count growth is the real signal.
    await page.waitForTimeout(500);
    const after = await items.count();
    idle = after > before ? 0 : idle + 1; // reset idle streak on growth
  }
  return items.count();
}

test('reaches the end of an infinite feed', async ({ page }) => {
  await page.goto('/infinite');
  const total = await scrollToEnd(page);
  expect(total).toBeGreaterThan(0);
});

The waitForTimeout here is a deliberate, bounded settle window between scrolls, not a load guess — and it is replaced by waitForResponse() in the detailed guide when the feed exposes a paged API. Synchronizing reads with asynchronously arriving rows is the same problem covered in Handling Dynamic Content, which underpins every pattern on this page.

Two details decide whether this loop is correct. First, the scroll may need to target a specific overflow container rather than window — many feeds live inside a scrolling <div>, so scrolling the document does nothing and the sentinel never intersects. Scroll the container itself with element.scrollTop = element.scrollHeight inside evaluate, or scroll the last card into view. Second, if the list is virtualized, counting rendered nodes undercounts the true total because recycled rows leave the DOM; in that case the count is only a growth signal, and the authoritative record set comes from intercepting the data responses as they arrive.

Cursor and token-based pagination

Not every listing encodes its position in a page number. Timeline-style feeds, activity logs, and most modern REST and GraphQL APIs use a cursor — an opaque token that each response hands you to fetch the next slice. There is no page 47 to jump to; you can only follow the chain forward, sending the token you last received and stopping when the response returns a null or empty cursor. This is the most authoritative pattern of all because the server itself tells you, explicitly, when the data is exhausted, so you never infer completion from the DOM.

When the tokens are visible in the page's network traffic, you can bypass rendering entirely and drive the API directly with Playwright's APIRequestContext. Doing so is faster and far more deterministic than scrolling, because each request-response pair is a discrete, waitable unit with an unambiguous stop token.

import { test, expect } from '@playwright/test';

test('walks a cursor-based API to exhaustion', async ({ playwright }) => {
  // A request context reuses connections and needs no browser page.
  const api = await playwright.request.newContext({ baseURL: 'https://api.example.com' });
  const all: unknown[] = [];
  let cursor: string | null = null;               // first request carries no cursor
  do {
    const query = cursor ? `?limit=100&cursor=${cursor}` : '?limit=100';
    const res = await api.get(`/v1/items${query}`); // one discrete, waitable fetch
    expect(res.ok()).toBeTruthy();
    const body = await res.json() as { data: unknown[]; nextCursor: string | null };
    all.push(...body.data);                        // accumulate, or flush to disk here
    cursor = body.nextCursor;                       // the authoritative stop token
  } while (cursor !== null);                         // null cursor means no more pages
  expect(all.length).toBeGreaterThan(0);
  await api.dispose();
});

The diagram below traces the token chain: the first request sends no cursor, and each response returns the token to feed the next request, until the server answers with a null cursor and the loop exits. Deriving the endpoint, decoding opaque or signed cursors, and handling servers that reuse a token when clamped is the subject of Scraping Cursor-Based Pagination APIs, which walks the whole chain end to end.

Knowing when extraction is complete

Every pattern needs a positive completion signal, never the absence of evidence after a timeout. Numbered pages are done when the "next" control vanishes or content repeats. A load-more list is done when the button leaves the DOM, becomes disabled, or a click yields no new rows. An infinite feed is done when the item count holds steady across several scroll rounds, or when the backing API returns an empty page or hasMore: false. A cursor walk is done when the response hands back a null or empty token. Where the listing is backed by a JSON endpoint, that API field is the most authoritative stop condition of all — intercept it rather than inferring completion from the DOM.

The decision tree below maps each advance mechanism to the one stop signal you should trust for it. Identify how the list moves forward first, and the correct terminator follows directly.

Keeping memory and state bounded

Long lists punish naive loops. Three habits keep a run healthy. Flush each batch to disk or a database as you read it instead of accumulating everything in one array, so a 50,000-row scrape does not grow unbounded in memory. For virtualized lists, read rows per batch as they render rather than expecting the final DOM to hold them all. And make the loop resumable: persist the last completed page number or scroll cursor so a crash restarts from there instead of the top. For large scrapes you should also pace requests politely — that is the subject of Anti-Bot Defenses & Rate Limiting — and split the work across processes, which is where Scraper Performance & Scaling takes over.

Failure modes and debugging

Most traversal bugs fall into a handful of recognizable shapes. Duplicate rows almost always mean the list re-sorted between fetches or your loop re-read a page it already visited on resume; deduplicate on a stable id and treat position as unreliable. A loop that never ends is the signature of clamped pagination — the server keeps serving the last valid page instead of a 404 — so add the content-identity check that halts when the first row repeats the previous page's first row. Missed tail rows on infinite scroll come from a settle window that is too short or a stop that fires after one idle round instead of several; raise maxIdleRounds or, better, switch to a response-based wait. Clicks that silently do nothing usually mean the control scrolled out of the actionable area or was replaced by a fresh node mid-iteration; re-resolve the locator each pass rather than caching a handle.

When a run misbehaves, the fastest way to see exactly what the loop did is the Playwright Trace Viewer, which records every navigation, click, and network response in order, so you can watch the batch counts climb and pinpoint the iteration where growth stalled. Pair that with a waitForResponse() on the data endpoint and most timing questions answer themselves — you can see whether the request even fired.

CI/CD considerations

A scraper that passes on your laptop and fails in CI is almost always a synchronization problem that a slower, contended machine exposes. Prefer response-based waits (waitForResponse(), count-growth polling) over fixed waitForTimeout windows, because a settle window tuned to a fast local run is the first thing to break under CI load. Where you must keep a bounded settle, make it configurable so CI can widen it without a code change.

Partition large numbered runs by page range and hand each range to a separate worker, the same sharding discipline described in CI/CD Integration; because numbered URLs are addressable, a shard is simply a start and end page. Persist the last completed page or cursor as a job artifact so a re-run resumes instead of restarting, and treat the run as complete only when the positive stop signal fired — a job that ended because it hit a wall-clock limit has a partial dataset and must not be trusted as final. For cursor and API-driven runs, log each token you follow so a failed job can be replayed from the exact point it stopped.

Deeper walkthroughs

Two deep dives beneath this guide take single patterns to their full, runnable conclusion. Scraping Infinite Scroll Pages with Playwright builds the complete scroll loop with waitForResponse() synchronization and a debounced stop for virtualized feeds. Scraping Cursor-Based Pagination APIs follows an opaque token chain end to end, including decoding cursors and handling servers that clamp or reuse a token at the boundary.

Frequently Asked Questions