Tuning Screenshot Comparison Thresholds

A visual test that fails on four pixels is as useless as one that passes over a collapsed layout, and most teams arrive at the second by over-correcting the first. The usual sequence is a red build, a guess at threshold: 0.5, another red build, a bigger guess, and eventually a suite that would happily approve a page rendered entirely in the wrong font. The options exposed by toHaveScreenshot() are not interchangeable dials for "how fussy should this be" — they act at two different levels of the comparison, and only one of them is safe to raise without losing coverage. This page shows how to measure what your own suite's pixel noise actually is, and how to turn that measurement into threshold, maxDiffPixels, and maxDiffPixelRatio values you can defend in review.

Root cause: two independent gates and a default budget of zero

Playwright decodes both PNGs and runs pixelmatch over them. Pixelmatch applies the first gate per pixel: it computes a squared YIQ colour distance and counts the pixel as different only when that distance exceeds 35215 × threshold², where threshold defaults to 0.2. The resulting count then meets the second gate, which is an area budget assembled from maxDiffPixels and maxDiffPixelRatio — and when neither option is set, that budget is 0, so a single differing pixel fails the assertion. Almost every "screenshot tests are too flaky" complaint is a page whose real noise is thirty pixels of text edge being measured against a budget of zero, and almost every "screenshot tests never catch anything" complaint is that same page fixed by raising threshold — the per-pixel gate — until whole components can change colour without tripping it. The mechanics of the capture itself, along with the settle loop that runs before any of this, are covered in Visual Regression Testing.

Minimal reproducible example

The test below is correct in every respect except its tolerance policy. The page is stable, the data is fixed, the capture is element-scoped — and it still fails intermittently on a Linux runner because sub-pixel text rasterisation moves a handful of glyph edge pixels between runs.

import { test, expect } from '@playwright/test';

test('pricing card matches its baseline', async ({ page }) => {
  await page.goto('/pricing');

  // Functional gate first, so the capture cannot start on a skeleton.
  await expect(page.getByRole('heading', { name: 'Plans' })).toBeVisible();

  // No options object at all. threshold silently defaults to 0.2, but both
  // area budgets stay undefined, which resolves to a budget of ZERO pixels.
  // Any single pixel that trips gate one fails the whole assertion.
  await expect(page.getByTestId('plan-card')).toHaveScreenshot('plan-card.png');
});

The failure reads like a real regression but is not one:

// Error: Screenshot comparison failed:
//
//   312 pixels (ratio 0.01 of all image pixels) are different.
//
//   Expected: /work/tests/__screenshots__/chromium-linux/pricing.spec.ts/plan-card.png
//   Received: /work/test-results/pricing-pricing-card-chromium/plan-card-actual.png
//        Diff: /work/test-results/pricing-pricing-card-chromium/plan-card-diff.png

Three hundred pixels spread thinly along text edges is noise. The same three hundred pixels concentrated in one rectangle is a component that changed. The count alone cannot tell you which, which is why the diff image is the first thing to open and the calibration below is the second.

Measuring the noise floor before you touch a number

A tolerance chosen without a measurement is a wish. The number you need is the ceiling of the differing-pixel count across repeated captures of unchanged code, in the environment that owns your baselines — normally the pinned container described in Dockerizing Playwright for Headless CI. A mean is useless here: the budget has to survive the worst run, not the typical one.

The spec below produces that number directly. It captures the same element from a fresh browser context a dozen times and runs the identical pixelmatch comparison Playwright would run, so the counts it prints are on exactly the same scale as the ones in a failure message.

import { test, expect } from '@playwright/test';
import pixelmatch from 'pixelmatch';
import { PNG } from 'pngjs';

// Both packages already ship inside Playwright; add them as devDependencies
// so this calibration spec can import them directly.
const RUNS = 12;

test('calibrate the noise floor for the plan card', async ({ browser }) => {
  const counts: number[] = [];
  let reference: PNG | undefined;

  for (let run = 0; run < RUNS; run++) {
    // A brand-new context each iteration reproduces a cold run, which is where
    // font loading and first-paint timing differences actually show up.
    const context = await browser.newContext({ viewport: { width: 1280, height: 720 } });
    const page = await context.newPage();
    await page.goto('/pricing');
    await expect(page.getByRole('heading', { name: 'Plans' })).toBeVisible();

    // Same capture options the real assertion uses, or the numbers do not transfer.
    const buffer = await page.getByTestId('plan-card').screenshot({
      animations: 'disabled',
      caret: 'hide',
      scale: 'css',
    });
    await context.close();

    const shot = PNG.sync.read(buffer);
    if (!reference) {
      reference = shot;   // first capture becomes the stand-in baseline
      continue;
    }

    // pixelmatch throws on mismatched dimensions, which is itself a finding.
    const diff = new PNG({ width: reference.width, height: reference.height });
    counts.push(
      pixelmatch(reference.data, shot.data, diff.data, reference.width, reference.height, {
        threshold: 0.2,   // keep the default so the result maps to the assertion
      }),
    );
  }

  // The ceiling is the number the budget has to clear, not the average.
  console.log(`noise ceiling=${Math.max(...counts)} samples=${counts.join(',')}`);
});

Step-by-step fix

1. Reproduce with the budget pinned at zero. Before changing anything, set maxDiffPixels: 0 explicitly on the failing assertion. This makes the current policy visible instead of implicit and confirms that the failure is a pixel-count failure rather than a settle timeout or a dimension mismatch, which no budget can fix.
2. Read the pixel count, never the printed ratio. The ratio in the error message is computed as Math.ceil(count / area × 100) / 100, so it is rounded up to two decimal places. One differing pixel in a two-megapixel image still prints as ratio 0.01. Deriving maxDiffPixelRatio from that displayed number can inflate your budget by four orders of magnitude; always work from the raw count.
3. Remove genuine nondeterminism first. A budget is not a fix for a live timestamp, an unseeded avatar, or an unfrozen animation. Freeze the clock, pin the payloads with Mocking API Responses with Playwright, and cover what remains with Masking Dynamic Regions in Snapshots. Every pixel of avoidable churn you leave in place is coverage you are about to spend budget on.
4. Measure the ceiling over at least ten cold runs. Run the calibration spec above in the environment that owns the baselines and record the maximum, not the mean. If the samples are all zero, your page is byte-stable and the budget should stay at zero — that is the strongest possible configuration and it is achievable more often than people expect.
5. Set the area budget to roughly two to three times the ceiling. With a ceiling of 61, maxDiffPixels: 150 absorbs a worse-than-usual run while still failing on any change larger than a few characters of text. Use maxDiffPixels for element-scoped snapshots of known size and maxDiffPixelRatio when one policy has to cover images ranging from a small card to a tall full-page capture. Setting both is allowed, and Playwright then applies the stricter of the two — Math.min of the absolute count and the ratio converted to pixels — which is rarely what people assume when they set them together.
6. Leave threshold at 0.2 unless the diff is a uniform colour shift. Sensitivity scales with the square of the value: the tolerated squared YIQ distance is 35215 × threshold², so moving from 0.2 to 0.4 does not double the tolerance, it quadruples it. Raise it only when the diff image shows large flat regions differing by an imperceptible amount — a compositing or colour-profile difference — and never above 0.3 on a suite whose job is catching design regressions.
7. Scope every override to the narrowest place that needs it, then re-measure. Put the policy in playwright.config.ts, override per project only where a rendering engine genuinely differs, and override per assertion only for the one snapshot that earns it with a comment explaining why. After each change, rerun the calibration and confirm the ceiling still sits well under the new budget.

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  expect: {
    // Capture, encode and compare happen on every retry, so a screenshot
    // assertion needs far more than the default 5s expect budget.
    timeout: 15_000,
    toHaveScreenshot: {
      threshold: 0.2,        // per-pixel gate left at the default, deliberately
      maxDiffPixels: 150,    // ~2.5x the measured ceiling of 61 on this runner
      animations: 'disabled',
      scale: 'css',          // normalise Retina captures to CSS pixels
    },
  },
  projects: [
    { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
    {
      name: 'webkit',
      use: { ...devices['Desktop Safari'] },
      // WebKit rasterises text differently; its measured ceiling was 240 px,
      // so it gets its own budget rather than inflating the shared one.
      expect: { toHaveScreenshot: { maxDiffPixels: 600 } },
    },
  ],
});

Project-level expect blocks are the reason a browser matrix does not force one lax global policy. Chromium keeps a tight budget; WebKit and Firefox get budgets derived from their own measurements, which is the same per-engine calibration discipline described in Cross-Browser Execution. Config layering itself follows the ordinary runner rules covered in Playwright Config & Fixtures.

Troubleshooting variants

Raising threshold changes nothing, but a small budget fixes it

This is the signature of anti-aliasing on high-contrast edges, and it confuses people because the intuition points the wrong way. Pixelmatch already runs an anti-alias detector — a pixel that exceeds the distance gate is re-examined against its eight neighbours and discarded if it looks like an anti-aliased edge in one image but not the other. Pixels that survive that check are genuinely differently coloured, so nudging threshold up rarely removes them until the value is high enough to blind the comparison entirely. The differences that remain along glyph edges are usually different rasterisation, not partial coverage, and the correct response is a modest area budget rather than a looser per-pixel gate.

The error says the images are different sizes

Expected an image 1280px by 3204px, received 1280px by 3196px. means dimensions diverged, and this always fails no matter how large the budget is. Playwright pads both images out to the larger bounding box and compares anyway, so the padded strip counts as a solid block of differing pixels and the reported count explodes into the tens of thousands — ignore that number, it is an artefact. Look for a content-height change on a fullPage capture, a viewport that no longer matches the one the baseline was recorded with, or a deviceScaleFactor difference that scale: 'css' would normalise. The calibration spec surfaces the same problem earlier as Error: Image sizes do not match. thrown from pixelmatch. Cross-platform size drift is handled properly in Managing Screenshot Baselines Across Platforms.

The count varies wildly from run to run instead of hovering near a ceiling

A tight spread of counts is noise; a spread from 12 to 40,000 is a page that is not settled, and no budget can be derived from it. Something is still rendering at capture time — a lazily loaded image, a chart animating on requestAnimationFrame, a font swap landing after first paint. Treat it as an instability problem rather than a tolerance problem and apply the diagnosis loop from Detecting and Fixing Flaky Playwright Tests, then recalibrate once the counts settle into a narrow band. Raising the budget to cover the worst sample here would set it above the size of most real regressions.

Verification

Prove the new numbers three ways, in this order. First, rerun the suite with --repeat-each=20 and no retries; twenty green runs against a budget you derived from twelve samples is reasonable evidence that the ceiling was real. Second — and this is the step teams skip — inject a deliberate regression and confirm the suite still fails: change one component's padding by two pixels or shift a border colour by a single step in your design tokens, run the assertion, and check that it reports a count comfortably above the budget. A tolerance that survives that test is calibrated; one that does not is just a disabled test with extra configuration. Third, publish the HTML report so the recorded counts are visible over time, as set up in Reporters & Test Artifacts — if the counts on passing runs start creeping toward the budget, the environment has drifted and it is time to re-measure rather than to raise the number again. Pair that with the retry and timeout policy from Configuring Retries and Timeouts for Stable CI so a slow runner does not turn a calibrated comparison into a timeout.

Frequently Asked Questions