Extracting JSON-LD and Microdata

Most commercial pages already publish the record you are trying to scrape. Product pages, recipes, job listings, events and articles carry a schema.org payload aimed at search engines — either a <script type="application/ld+json"> block holding a JSON object, or itemscope/itemprop attributes woven into the rendered markup. That payload names its own fields, types its own values and is maintained by the site owner because their search ranking depends on it, which makes it a far more stable extraction target than the presentational DOM around it. The catch is that reading it well is not a one-liner: a single page routinely ships four or five blocks, half of them wrapped in @graph, some injected by a tag manager seconds after load, and occasionally one that is not valid JSON at all. This page shows how to collect every block with Playwright, normalize the shapes into one flat node list, and fall back to a microdata tree walk when no JSON-LD exists.

Root cause: the machine-readable copy sits outside the text you can see

A JSON-LD block lives in a <script> element, and <script> is a raw-text element in the HTML parser — the browser never executes application/ld+json, never decodes character references inside it, and never exposes it through any rendered-text API. Microdata is the mirror image: its values hide in attributes (content, datetime, href) on elements whose visible text is formatted for a human reader, so textContent silently returns the wrong thing. Both carriers therefore need an explicit read path rather than the locator-plus-text pattern used for Extracting Tables and Lists to JSON with Playwright and the rest of Structured Data Extraction, inside the wider Web Scraping & Data Extraction workflow.

Minimal reproducible example

The test below collects every JSON-LD block on a product page, tolerates the three shapes a payload can take, and survives a malformed block without losing the rest of the page.

import { test, expect } from '@playwright/test';

type Node = Record<string, unknown>;

// A payload is one of: a single node, an array of nodes, or a { "@graph": [...] }
// wrapper. Flatten all three into one list before you look for anything.
function flatten(value: unknown): Node[] {
  if (Array.isArray(value)) return value.flatMap(flatten);
  if (value && typeof value === 'object') {
    const node = value as Node;
    const graph = node['@graph'];
    return graph ? flatten(graph) : [node];   // a @graph replaces its wrapper
  }
  return [];                                   // strings and numbers are not nodes
}

// "@type" is a string on most nodes and an array on multi-typed ones.
function hasType(node: Node, type: string): boolean {
  const t = node['@type'];
  return Array.isArray(t) ? t.includes(type) : t === type;
}

test('reads the Product node from a page with several JSON-LD blocks', async ({ page }) => {
  // Server-rendered structured data is in the markup before subresources load.
  await page.goto('https://example.com/products/widget-9000', {
    waitUntil: 'domcontentloaded',
  });

  const blocks = page.locator('script[type="application/ld+json"]');

  // A multi-match locator is fine as long as you never call a single-element
  // method on it. blocks.textContent() would throw:
  //   Error: strict mode violation: locator('script[type="application/ld+json"]')
  //   resolved to 4 elements
  await expect(blocks).not.toHaveCount(0);   // retries while a tag manager injects

  // textContent on a raw-text element returns the exact bytes the site shipped.
  const payloads = await blocks.allTextContents();

  const nodes = payloads.flatMap((raw) => {
    try {
      return flatten(JSON.parse(raw));
    } catch {
      return [];                             // one bad block must not fail the page
    }
  });

  const product = nodes.find((n) => hasType(n, 'Product'));
  expect(product, 'no schema.org Product node on the page').toBeTruthy();
  expect(typeof product!.name).toBe('string');
});

Step-by-step fix

Work the payload in two phases: collect first, interpret second. Every step below up to and including the flattening is mechanical and identical on every site, so it belongs in a shared helper you import into each scraper; only the type matching and the field validation are page-specific. Keeping that seam clean means a new source site costs you a schema definition rather than a new parser, and it keeps the failure surface small enough that a broken run points at one identifiable cause.

1. Navigate at domcontentloaded and select every block at once. Server-rendered JSON-LD is in the initial HTML, so page.goto(url, { waitUntil: 'domcontentloaded' }) is enough and saves the seconds load spends on images and fonts — the same budget argument made in Blocking Images and Fonts to Speed Up Scraping. Select with page.locator('script[type="application/ld+json"]') and never with .first(); the block you want is rarely the first one.
2. Wait for late-injected blocks with a retrying assertion. Google Tag Manager, Shopify apps and review widgets append their own blocks after hydration. await expect(blocks).not.toHaveCount(0) retries under the expect timeout, and page.waitForFunction() handles the harder case where you need a specific @type to appear. A bare read straight after goto fails with locator.textContent: Timeout 30000ms exceeded. on those sites.
3. Read the raw text, never the rendered text. Use locator.allTextContents(), which returns one string per matched node. innerText() is defined in terms of rendered layout and a <script> renders nothing, so it returns an empty string — a failure that looks like missing data rather than a wrong API.
4. Parse each block inside its own try/catch. One publisher-side templating bug produces SyntaxError: Expected double-quoted property name in JSON at position 412 and, without isolation, kills the extraction for a page whose other four blocks were perfectly valid. Catch per block, count the failures, and keep going.
5. Flatten arrays and @graph wrappers into one node list. Reaching straight for parsed.name yields undefined on the majority of WordPress and Yoast pages, where every entity hangs off a single @graph. Recurse as in the example above so downstream code only ever sees a flat list of nodes.
6. Resolve @id cross-references before you read fields. Inside a @graph, an Article points at its author as { "@id": "https://site/#/schema/person/1" } rather than embedding it. Build a Map from @id to node, then replace any object whose only key is @id with the node it names, so author.name resolves instead of returning undefined.
7. Match @type and @context tolerantly. Accept @type as a string or an array, treat http://schema.org and https://schema.org as identical, and strip any schema: prefix a JSON-LD context may have introduced. Compare on the final path segment of the type IRI rather than on the whole string.
8. Fall back to microdata, then validate before writing. If no node carries the type you need, walk the itemscope tree described below. Either way, assert the required fields exist and have the right primitive types before serializing, so a template change on the source site surfaces as a failed run rather than a file full of undefined.

Walking the microdata itemscope tree

When a site predates JSON-LD — classifieds, older marketplaces, government registries — the record is expressed with microdata instead. The spec is precise about where a property's value lives: on <meta> it is the content attribute, on <time> the datetime, on <data> the value, on <a>/<link> the resolved href, on <img> the resolved src, and on everything else the element's text. Nesting matters too: an element carrying both itemprop and itemscope is a property of its parent scope and the root of a child object, so a flat querySelectorAll('[itemprop]') mixes two levels of the record together. The walk below runs entirely in the page, which keeps it to one round trip regardless of how many properties the item has.

import { test, expect } from '@playwright/test';

test('extracts a nested microdata Product item', async ({ page }) => {
  await page.goto('https://example.com/listings/4821');

  // itemtype ends with the type name, so a suffix match covers both the
  // http://schema.org and https://schema.org spellings sites still use.
  const record = await page
    .locator('[itemscope][itemtype$="/Product"]')
    .first()
    .evaluate((root: Element) => {
      // Per the microdata spec the value depends on the element, not the text.
      const valueOf = (el: Element): string => {
        if (el instanceof HTMLMetaElement) return el.content;
        if (el instanceof HTMLTimeElement) return el.dateTime || el.textContent!.trim();
        if (el instanceof HTMLDataElement) return el.value;
        if (el instanceof HTMLImageElement) return el.src;    // already absolute
        if (el instanceof HTMLAnchorElement) return el.href;  // already absolute
        if (el instanceof HTMLLinkElement) return el.href;
        return (el.textContent ?? '').trim();
      };

      const collect = (scope: Element): Record<string, unknown> => {
        const out: Record<string, unknown> = {};
        for (const el of Array.from(scope.querySelectorAll('[itemprop]'))) {
          // The nearest itemscope ancestor owns the property. Start from the
          // parent so an element that is itself a scope is not its own owner.
          if (el.parentElement?.closest('[itemscope]') !== scope) continue;
          const name = el.getAttribute('itemprop')!;
          // A nested scope becomes a nested object; anything else is a value.
          out[name] = el.hasAttribute('itemscope') ? collect(el) : valueOf(el);
        }
        return out;
      };

      return collect(root);
    });

  expect(record.name).toBeTruthy();
  expect(record.offers).toHaveProperty('price');
});

Troubleshooting variants

SyntaxError: Expected double-quoted property name in JSON at position 412

The block is genuinely invalid JSON, and the cause is almost always publisher-side. Trailing commas from a hand-edited template, a raw newline inside a string, and an unescaped " in a product description are the common three. A fourth is subtler: because <script> is a raw-text element, character references are not decoded, so a CMS that HTML-escapes its output ships literal " sequences where quotes should be. Log the offending substring with raw.slice(390, 440) to see which case you have, decode entities only when you have confirmed that is the problem, and otherwise treat the block as unusable and move to the next carrier rather than trying to repair arbitrary broken JSON with regular expressions.

The block exists in DevTools but the locator finds nothing

You are reading before the injection happens. Sites that render structured data client-side add it during hydration or from a tag manager, so the block is absent at domcontentloaded and present a second later. Replace the immediate read with a condition on the data itself — await page.waitForFunction(() => [...document.querySelectorAll('script[type="application/ld+json"]')].some((s) => s.textContent!.includes('"Product"'))) — which waits for the specific node instead of a generic settle. The tradeoffs between that and a blanket network wait are covered in Waiting for Network Idle vs Element State and, more broadly, in Handling Dynamic Content. If the data still never appears, check whether the widget rendering it lives in an iframe, in which case you need a frame locator from Iframes & Embedded Content.

Two blocks describe the same entity with different values

Platforms stack plugins, and each plugin emits its own Product node — one with the list price, one with the sale price, one with a stale rating. Do not take the first match. Score the candidates instead: prefer the node carrying an @id that matches the canonical URL, then the one with the most populated fields, then the last in document order, since late-injected blocks usually reflect the current state. Record which block won alongside the extracted record so a later dispute is answerable, and if the values conflict on a field you care about, reconcile against the rendered price rather than guessing. The same duplication shows up between carriers: a page can ship a Product in JSON-LD and a second, older one in microdata that nobody has updated since the template was written. Decide the precedence order once, encode it as data rather than as branching code, and log the carrier that supplied each field so the provenance travels with the record into whatever store consumes it.

Verification

Verify the extraction on three axes rather than trusting a green test. First, assert on shape: check that the parsed node list is non-empty, that the node you selected carries every required property, and that numeric fields survived Number() without becoming NaN — a JSON-LD price is a string like "1299.00" far more often than a number. Second, verify against the page: compare the extracted name and price with the rendered values from a locator, so a stale block that no longer matches the visible page fails loudly instead of poisoning the dataset. Third, verify at scale: run the extractor across a sample of URLs and track the ratio of pages that produced a valid record, because structured-data coverage varies by template and a sudden drop marks a site redesign. When a page in that sample fails, the network and console panels described in Reading Network and Console Tabs in Traces show whether the injecting script ever ran, and if the failures all begin after a burst of requests you are being throttled — see Handling Rate Limits and Retries When Scraping.

Frequently Asked Questions