Structured Data Extraction

Scraping is only half the job. The other half — the half that decides whether your output is usable — is turning a messy, presentational DOM into clean records with stable field names and correct types. A page renders a price as "$1,299.00" inside a nested span; your dataset needs the number 1299. A date shows as "3 days ago"; you want an ISO string. A rating lives in a title attribute while the visible stars are decorative SVG glyphs. This guide, part of the broader Web Scraping & Data Extraction discipline, covers the mechanics of mapping the rendered DOM to typed objects: choosing between locator.evaluateAll() and Node-side locators, reading textContent() and getAttribute(), harvesting embedded structured data, normalizing values, and serializing the result to JSON you can trust downstream.

Why extraction is a mapping problem, not a selection problem

It is tempting to think of extraction as "find the elements and read them", but the selecting is the easy part. The hard part is that the DOM is optimized for a human eye and a layout engine, not for a schema. Presentation and data are interleaved: the same visible string can be assembled from three inline spans, a currency symbol can be a ::before pseudo-element that no text API returns, and an out-of-stock badge might be conveyed purely by a greyed-out CSS class with no textual marker at all. Every field you want has a rendering that reads cleanly to a person and a structure that fights any naive .textContent read.

Framing the task as a mapping — a pure function from a repeated DOM subtree to one typed record — keeps that complexity contained. Each field becomes a small, independently testable rule: locate the source node, read a value, normalize it to the target type, and fall back deterministically when it is absent. When you separate the locating from the normalizing you can change either without breaking the other, and you gain a single place to enforce the invariant that matters most in a dataset: every record has the same shape, and a missing value is represented honestly rather than smuggled in as an empty string or a zero. The rest of this guide builds that mapping in layers, starting with the two mechanisms Playwright gives you to reach the nodes.

Prerequisites

You need a page whose target records are already rendered — extraction reads the DOM at the moment you query it, so if the content arrives after an XHR or hydrates late, settle it first with the synchronization patterns in Handling Dynamic Content. You should be comfortable with locators and prefer accessibility-first queries from getByRole & Accessibility Selectors over brittle class chains. The examples use TypeScript with the standalone playwright package for scripts and @playwright/test where an assertion clarifies intent; both run under Node with no extra dependencies.

Two extraction strategies

There are two ways to pull data out of a Playwright page, and the right choice depends on volume and complexity.

The first is locator.evaluateAll(), which serializes a function, runs it inside the browser against every matching node, and returns plain data across the bridge in a single round trip. For a list of a thousand cards with five fields each, this is dramatically faster than five thousand individual locator calls because nothing crosses the Node-to-browser boundary per field. The trade-off is that the callback runs in the page, so you write DOM APIs (querySelector, textContent) rather than Playwright locators, and you cannot use the auto-waiting that locators provide. Because the function is serialized and shipped into the browser, it also cannot close over Node variables or imports — everything it needs must be passed as the second argument to evaluateAll().

The second is Node-side locators: you iterate elements and call locator.textContent() or locator.getAttribute() from your test code. This is more readable, lets you reuse accessibility-first selectors such as those in getByRole & Accessibility Selectors, and inherits auto-waiting, but it costs one cross-process call per field. Use it for small or irregular result sets and where selector stability matters more than raw throughput.

import { chromium } from 'playwright';

interface Article { title: string; author: string; href: string; }

const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('https://example.com/blog');

// Strategy A: evaluateAll — one bridge call maps every <article> to a record.
const articles: Article[] = await page.locator('article').evaluateAll((nodes) =>
  nodes.map((n) => ({
    title: n.querySelector('h2')?.textContent?.trim() ?? '',
    author: n.querySelector('.byline')?.textContent?.trim() ?? '',
    href: n.querySelector('a')?.getAttribute('href') ?? '',
  })),
);

console.log(articles.length, 'articles extracted');
await browser.close();

Reading text and attributes cleanly

The two workhorses are textContent() for the visible text of a node and getAttribute() for attribute values like href, src, or data-*. Two pitfalls recur. First, textContent returns all descendant text, including hidden helper spans and whitespace from the markup's indentation, so always .trim() and collapse internal whitespace with a regex when a field spans multiple inline elements. Second, a missing element returns null, so guard every read with a nullish fallback to avoid a record full of undefined.

For text that a screen reader would announce — which often excludes decorative markup — locator.innerText() respects CSS visibility and is closer to what a user perceives. Use textContent() when you want the raw DOM text regardless of styling, and innerText() when you want only what is visually rendered. There is a third read worth knowing: locator.getAttribute('aria-label') and getAttribute('title') frequently hold the clean, canonical value a designer hid behind a decorative rendering — a star rating exposed as aria-label="4.5 out of 5" is far easier to parse than counting filled glyph paths.

Watch three edge cases that silently corrupt fields. A value split across nested inline elements (<span>$</span><span>1,299</span><span>.00</span>) concatenates without separators, which is usually what you want, but the same structure with whitespace between tags yields "$ 1,299 .00"; collapse runs of whitespace before you parse. An attribute that is present but empty (href="") passes a != null check yet resolves to a useless value, so validate content, not mere presence. And a boolean-style attribute like disabled or aria-hidden returns the empty string when set and null when absent, so test with !== null rather than truthiness, because "" is falsy and would read as "not disabled".

import { chromium } from 'playwright';

const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('https://example.com/product/42');

// textContent for raw text; getAttribute for structured values.
const name = (await page.locator('h1').textContent())?.trim() ?? '';
const rawPrice = (await page.locator('.price').textContent()) ?? '';
const sku = await page.locator('[data-sku]').getAttribute('data-sku');

// Collapse whitespace, then strip everything but digits and the decimal point.
const price = Number(rawPrice.replace(/\s+/g, ' ').replace(/[^0-9.]/g, ''));

// A present-but-empty attribute is not a valid value — check content, not presence.
const validSku = sku && sku.trim().length > 0 ? sku.trim() : null;

console.log({ name, price, sku: validSku });
await browser.close();

Mapping to typed objects

A record is only useful if its shape is predictable. Define a TypeScript interface for the record up front, then make the mapping function return that type. This forces you to handle every field, makes missing data visible as a compile error rather than a silent undefined, and documents the dataset's schema in one place. Normalize at the boundary: parse numbers, convert relative dates to ISO strings, and resolve relative URLs to absolute ones with new URL(href, page.url()) so the output is portable.

When extraction spans repeated structures with optional fields, model the optionality explicitly (price?: number) rather than coercing missing values to zero, which would silently corrupt aggregates downstream. A record that honestly says a field is absent is more valuable than one that lies with a default. The normalization step is where a raw string becomes a trustworthy value, and it helps to picture each field moving through a small pipeline of states: read, cleaned, coerced to its target type, and finally accepted or rejected against the schema. Fields that fail coercion should not silently become 0 or ""; they should be dropped to null and, ideally, counted so a run that suddenly rejects a third of its prices trips an alarm instead of shipping garbage.

Harvesting embedded structured data

Before you reverse-engineer a presentational layout, check whether the page already publishes its data in a machine-readable form. Many sites embed a <script type="application/ld+json"> block — a JSON-LD graph describing the product, article, or event — precisely so search engines can read it, and microdata itemprop attributes serve the same purpose inline. When present, this is the highest-quality source on the page: the values are already typed, canonical, and stable against restyles, because they exist to be parsed rather than displayed. Extracting from it turns a fragile DOM-scraping problem into a JSON-parsing problem.

The mechanics are simple. Read the raw text of every JSON-LD script, JSON.parse each one, and pluck the fields you need from the resulting object graph — remembering that a page may carry several blocks and that a single block may hold an @graph array of many entities. The deep dive Extracting JSON-LD and Microdata walks through the messy realities — multiple graphs, @type filtering, and microdata fallbacks — but the core read is short enough to show here.

import { chromium } from 'playwright';

interface Product { name: string; price: number | null; sku: string | null; }

const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('https://example.com/product/42');

// Read every JSON-LD block's raw text in one bridge call.
const blocks = await page
  .locator('script[type="application/ld+json"]')
  .evaluateAll((nodes) => nodes.map((n) => n.textContent ?? ''));

// Parse each block defensively — one malformed block must not sink the rest.
const products: Product[] = blocks.flatMap((raw) => {
  let parsed: unknown;
  try {
    parsed = JSON.parse(raw);
  } catch {
    return []; // skip invalid JSON rather than throwing the whole run
  }
  // A block may be a single node or an @graph array of many entities.
  const graph = Array.isArray(parsed) ? parsed : [parsed];
  return graph
    .filter((e: any) => e['@type'] === 'Product')
    .map((e: any) => ({
      name: String(e.name ?? ''),
      price: e.offers?.price != null ? Number(e.offers.price) : null,
      sku: e.sku != null ? String(e.sku) : null,
    }));
});

console.log(products);
await browser.close();

Treat embedded data as authoritative but not infallible: sites sometimes ship a JSON-LD price that lags the rendered one during a sale, so when both exist it is worth reconciling them and preferring whichever your use case trusts. Still, starting from a published graph rather than a <div> soup removes an entire category of normalization bugs.

Choosing an extraction path

With three mechanisms on the table — in-page evaluateAll(), Node-side locators, and embedded structured data — the decision is quick once you frame it by what the page offers and how much you are pulling. If a JSON-LD or microdata graph carries the fields you need, parse it and stop; nothing hand-rolled will be cleaner. If the data is tabular, drive off the header row. Otherwise the split is volume: large repeated lists favor a single in-page map, small or irregular sets favor readable Node-side reads.

Serializing to JSON

Once you hold an array of typed records, serialization is a single step. JSON.stringify(records, null, 2) produces readable output; write it with the Node fs module. For large runs, write newline-delimited JSON (one object per line) so the file can be streamed and appended incrementally rather than held entirely in memory, and so a crash mid-run does not lose everything collected so far. NDJSON also composes with pagination: as you walk each page of results — the technique covered in Pagination & Infinite Scroll — you append that page's records and flush, keeping memory flat regardless of how many pages the run spans.

Two serialization details save downstream pain. Dates should already be ISO strings by the time they reach stringify, because JSON.stringify turns a Date object into an ISO string but a parsed-then-reformatted string round-trips predictably across languages. And decide early whether null or omission represents a missing field — pretty-printed JSON with explicit nulls is easier to diff and validate than objects with varying key sets, so prefer emitting every key with a null value over dropping the key entirely.

import { chromium } from 'playwright';
import { writeFile } from 'node:fs/promises';

interface Row { name: string; price: number | null; scrapedAt: string; }

const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('https://example.com/catalog');

const rows: Row[] = await page.locator('.item').evaluateAll((items) =>
  items.map((i) => {
    const raw = i.querySelector('.price')?.textContent?.replace(/[^0-9.]/g, '') ?? '';
    const price = raw.length > 0 ? Number(raw) : null; // absent stays null, never 0
    return { name: i.querySelector('.name')?.textContent?.trim() ?? '', price };
  }),
).then((records) =>
  // Stamp each record with an ISO capture time in Node, outside the browser.
  records.map((r) => ({ ...r, scrapedAt: new Date().toISOString() })),
);

// Pretty-print to a file the rest of the pipeline can consume.
await writeFile('out/catalog.json', JSON.stringify(rows, null, 2), 'utf-8');
await browser.close();

Failure modes and debugging

Three failure patterns account for most broken extractions. The first is the silent empty field: a selector that matched yesterday now returns "" because the site renamed a class or wrapped the value in a new span. Empty strings pass most guards, so the run "succeeds" while quietly producing hollow records. Defend against it by asserting a floor — if more than a small fraction of a field comes back empty, fail the run loudly rather than writing the file. The second is the count mismatch: evaluateAll returned 40 records but the page rendered 60, because 20 were still lazy-loading below the fold. Extraction reads a snapshot, so always confirm the expected count is present before you map, using the waiting patterns from Handling Dynamic Content. The third is the encoding surprise: non-breaking spaces ( ), zero-width characters, and smart quotes ride along inside textContent and defeat a naive replace(/ /g, ''); normalize with \s classes and Unicode-aware regexes.

When a field is wrong rather than absent, the fastest tool is npx playwright codegen against the live page: click the element you want and read the selector Playwright generates, which often exposes a cleaner attribute source than the one you guessed. For batch runs, log a per-field rejection tally so a schema drift shows up as "SKU rejection jumped from 0% to 34%" instead of as a mysteriously smaller dataset a week later.

Running extraction in CI

Extraction jobs belong in the same disciplined pipeline as tests. Pin the browser build and run headless in a container so the DOM you parse in CI matches the one you developed against — the setup in CI/CD Integration applies unchanged to scraper jobs. Treat the schema as a contract: add a validation step after extraction that checks record count against a floor and field types against the interface, and fail the job when either drifts, so a site redesign surfaces as a red build rather than as corrupted downstream data. Persist the output JSON as a build artifact and keep the previous run to diff against; a sudden change in record count or a spike in null fields is the earliest signal that the target page changed. When runs grow large, the throughput and parallelism techniques in Scraper Performance & Scaling keep extraction inside its time budget without abandoning the mapping discipline described here.

Putting it into practice

Three focused walkthroughs apply these mechanics to the cases you will meet most often. Extracting Tables and Lists to JSON with Playwright handles tabular markup — reading a header row to derive keys, then mapping each body row to an object — and writes a clean file. Extracting JSON-LD and Microdata shows how to harvest the machine-readable graph a page already publishes, sidestepping presentational DOM entirely when it exists. Scraping Data Behind Login Sessions covers reaching records that require authentication, using a saved session so you extract without logging in on every run. All three build directly on the mapping and normalization patterns above, and all fit inside the larger pipeline described in Web Scraping & Data Extraction.

Frequently Asked Questions