Extracting Tables and Lists to JSON with Playwright

HTML tables and lists are the most common shape of structured data on the web — pricing grids, sortable reports, leaderboards, search results. The markup is regular, which makes it ideal for extraction, but the naive approach (grab every <td> into a flat array) throws away the column meaning and leaves you with positional data nobody can use. The goal is a JSON array where each row is an object keyed by its column header: { "rank": 1, "team": "Acme", "points": 42 }. This page walks through deriving those keys from the header row, mapping each body row into a typed record with locator.evaluateAll(), and writing the result to a file. It is a focused application of the techniques in Structured Data Extraction, within the broader Web Scraping & Data Extraction workflow.

Why positional extraction fails

Reading cells by index — cells[0], cells[1] — works until the day a column is added, removed, or reordered, at which point every record is silently mislabeled and your downstream consumers ingest corrupt data without an error. That failure is quiet by construction: the shapes still match, the row count is unchanged, and only the meaning of each field has drifted, so nothing throws and the bad data flows straight into whatever aggregates or dashboards read it. Binding values to header names instead makes the extraction self-describing and resilient: a reordered column still lands under the right key, and a renamed header surfaces as a visible change in the output rather than a quiet data swap. The same principle applies to definition lists and card grids — derive the field name from the markup, never from position alone. For pages that also embed a machine-readable copy of the same data, Extracting JSON-LD and Microdata is often a cleaner source than the rendered table.

The contrast is easiest to see side by side: the same four changes that quietly corrupt an index-based reader are all either prevented or made loud by header-keyed records.

Step-by-step extraction

The extraction is a short pipeline: render the page, lock onto a real row, turn the header into keys, fold each body row into that key set, coerce the raw strings into typed values, and serialize. Each stage feeds the next, and skipping the wait stage is the single most common cause of an empty result set on a JavaScript-rendered table.

1. Navigate and wait for the table to render. Call page.goto(url) and then wait for the table body to have rows with page.locator('table tbody tr').first().waitFor(). On a single-page app the table is populated by JavaScript after load, so waiting for a concrete row — not a fixed sleep — is what the synchronization guidance in Handling Dynamic Content calls for.
2. Derive the keys from the header row. Read every <th> with allTextContents(), then trim and normalize each into a clean key (lowercase, spaces to underscores). These become the object property names for every record, so the dataset is self-describing.
3. Map each body row to an object. Use locator.evaluateAll() over tbody tr so the whole mapping runs in the browser in a single bridge call. Inside the callback, read each row's cells, trim the text, and zip them against the keys you derived to build one object per row.
4. Normalize the cell values. Parse numeric columns into numbers, strip currency symbols and thousands separators, and convert relative dates to ISO strings. Do this in the map step so the JSON holds clean typed values rather than display strings.
5. Handle lists the same way. For a <ul>/<ol> or a grid of cards, replace the row/cell logic with a per-item map: read the item's text and any getAttribute() values you need, and produce the same shape of object. The keying-by-meaning principle is identical.
6. Serialize and write the file. Pass the array to JSON.stringify(rows, null, 2) and write it with the Node fs module. For very large tables, write newline-delimited JSON so the file streams and a mid-run crash does not lose collected rows.

import { chromium } from 'playwright';
import { writeFile } from 'node:fs/promises';

interface Row { [key: string]: string | number; }

const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('https://example.com/leaderboard');

// Step 1: wait for a real row so we know the table has rendered.
await page.locator('table tbody tr').first().waitFor();

// Step 2: derive keys from the header cells.
const rawHeaders = await page.locator('table thead th').allTextContents();
const keys = rawHeaders.map((h) => h.trim().toLowerCase().replace(/\s+/g, '_'));

// Step 3 + 4: map each body row to a keyed object inside the browser.
const rows: Row[] = await page.locator('table tbody tr').evaluateAll((trs, headerKeys) =>
  trs.map((tr) => {
    const cells = Array.from(tr.querySelectorAll('td'));
    const record: Record<string, string | number> = {};
    cells.forEach((cell, i) => {
      const key = headerKeys[i] ?? `col_${i}`;          // fall back if header is short
      const text = cell.textContent?.trim() ?? '';
      // Numeric columns become numbers; everything else stays a string.
      const asNumber = Number(text.replace(/[^0-9.-]/g, ''));
      record[key] = text !== '' && !Number.isNaN(asNumber) && /\d/.test(text) ? asNumber : text;
    });
    return record;
  }, keys,
);

// Step 6: serialize the keyed records to a file.
await writeFile('out/leaderboard.json', JSON.stringify(rows, null, 2), 'utf-8');
console.log(`Wrote ${rows.length} rows`);
await browser.close();

Extracting a plain list

A list needs no header step — each item is already a record. Map every <li> (or card) directly, reading its text and any attributes such as a link target. Resolve relative URLs against the page URL so the output is portable, because a bare /product/12 is useless to any consumer that does not already know the origin.

Card grids are the common real-world version of this: each card carries a title, a price, an image, and a link, spread across several child elements rather than table cells. Read each field with a scoped querySelector inside the map so a field that is missing on one card becomes null rather than shifting every other value, and pull machine-readable data from attributes — data-id, datetime, a srcset — in preference to the rendered label whenever both exist. The output shape stays a flat array of uniform objects, which is exactly what makes the file easy to validate and to diff against the previous run.

import { chromium } from 'playwright';
import { writeFile } from 'node:fs/promises';

interface Link { label: string; href: string; }

const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('https://example.com/sitemap');

const base = page.url();
// Map each list item to a record, resolving relative hrefs to absolute.
const links: Link[] = await page.locator('ul.links li a').evaluateAll((anchors, baseUrl) =>
  anchors.map((a) => ({
    label: a.textContent?.trim() ?? '',
    href: new URL(a.getAttribute('href') ?? '', baseUrl).toString(),
  })), base,
);

await writeFile('out/links.json', JSON.stringify(links, null, 2), 'utf-8');
await browser.close();

Troubleshooting variants

Cells merged with colspan or rowspan

When a header or body cell spans multiple columns, the one-to-one zip between cells and keys breaks: a colspan="2" header occupies one <th> but two visual columns, so every key after it lands one slot early. Detect the span by reading the colSpan and rowSpan properties inside the evaluateAll() callback and expand the row into a normalized grid before zipping — push the cell's value once per column it covers. For a rowspan, carry the value downward into the rows below until the span is exhausted. If a table only ever merges a known header cell, a simpler fix is to hard-code the corrected key list rather than deriving it, but grid normalization is the resilient choice for tables you do not control.

Cells hold nested markup, icons, or hidden text

textContent returns every descendant text node, including visually hidden labels a screen reader uses and stray whitespace from pretty-printed markup. That turns a clean 42 into 42 ▲ up once an icon and an accessibility caption are added inside the cell. Target the specific child that holds the value with cell.querySelector('.value')?.textContent, or read innerText, which respects rendering and skips hidden nodes, and always collapse whitespace with a .replace(/\s+/g, ' ').trim(). For attribute-backed data such as a sortable timestamp stored in data-sort, read the attribute directly rather than the display text; it is usually already normalized to a machine format.

The header row is missing or split across two rows

Some tables carry no <thead>, or stack a grouping row above the real labels. When allTextContents() returns nothing usable, fall back to positional col_0, col_1 keys and rename them in a post-processing pass, or read the second header row explicitly with nth(1). Tables built for accessibility often expose the true column name in a scope="col" attribute or an aria-label, which survives visual redesigns better than the rendered text and makes a reliable key source when the visible header is only an icon.

Verification

Confirm the extraction three ways. First, assert the row count matches the visible table: expect(rows.length).toBe(await page.locator('table tbody tr').count()). Second, spot-check the first and last records against the page so off-by-one errors in the header/body split surface immediately. Third, validate the JSON shape — every record should have the same keys, and numeric columns should be numbers, not strings; a quick zod schema or a manual key check catches a malformed header row. When the table only appears after signing in, pair this with Scraping Data Behind Login Sessions so the page is authenticated before extraction runs. When a table spans multiple pages, combine this with the loop in Pagination & Infinite Scroll before writing the final file.

Frequently Asked Questions