Flaky Test Management

A flaky test passes and fails against the same commit with no code change between runs. It is the most expensive failure mode in an automated suite because it trains engineers to ignore red builds, and once a real regression hides behind that noise it ships. Playwright removes most flakiness by design through auto-waiting and web-first assertions, but the remaining cases trace back to a small set of root causes: race conditions, animation timing, shared state between tests, and network non-determinism. This guide, part of Debugging & Test Observability, explains how to detect flakiness early, quarantine it so it stops blocking releases, and eliminate the underlying cause rather than papering over it with retries.

What flakiness actually is

A test is flaky when its outcome is not a pure function of the code under test. Some hidden input — wall-clock timing, scheduler order, leftover data from a previous run, a slow network hop, a random identifier, the machine's CPU contention — changes between executions and flips the result. That framing matters because it tells you what a fix has to accomplish. A fix either removes the hidden input from the equation or makes the test wait on the precise condition it actually cares about. Nothing else counts. Increasing a timeout does neither; it only widens the window in which the hidden input usually settles, which converts a 2% failure rate into a 0.2% failure rate and moves the problem out of view rather than out of the suite.

It is worth doing the arithmetic on that residual rate. A suite of 600 tests where every test independently fails 0.2% of the time produces a red pipeline roughly seven runs in ten. Per-test flakiness that looks negligible becomes a suite that is almost never green, and engineers respond rationally by re-running the build instead of reading it. The number to drive down is not the failure rate of your worst test but the probability that a clean commit produces a clean pipeline, and that quantity is brutally sensitive to suite size.

Playwright's design removes whole categories of hidden input before you write a line. Every action performs actionability checks — the element must be attached, visible, stable, enabled, and receiving events — before the click or fill is dispatched. Every expect() on a locator polls until the condition holds or the timeout expires. page.goto() waits for the load event. Because so much is handled, a flaky Playwright test is nearly always one of four things: an explicit waitForTimeout(), a value read into a JavaScript variable and then asserted with a non-retrying matcher, state bleeding in from another test or a previous run, or a dependency on a live system you do not control.

The distinction between retrying and non-retrying assertions is the single most useful thing to internalise. await expect(locator).toHaveText('1') re-queries the DOM on an interval until it matches. expect(await locator.textContent()).toBe('1') resolves the text exactly once, at whatever instant the runner happened to reach that line, and compares a dead string. The two lines look almost identical in a code review and behave completely differently under load. The rules for choosing between them are set out in Web-First Assertions, and reviewing for that one pattern will catch more flakiness than any other single check.

Prerequisites

This guide assumes a Playwright project with @playwright/test installed, a playwright.config.ts you can edit, and traces enabled on at least the first retry (trace: 'on-first-retry'). You also want a CI pipeline that publishes the HTML report as an artifact, because most of the diagnosis described below happens by reading a failed run's trace rather than by reproducing the failure locally. If your suite still shares a login by pointing every test at one seeded account, read Browser Contexts & Isolation first — no amount of assertion tuning will stabilise tests that mutate each other's data.

Root cause 1: race conditions

A race condition is an assertion that runs before the application has reached the state it asserts. The classic shape is reading a value immediately after triggering an async action:

import { test, expect } from '@playwright/test';

test('flaky: reads count before it updates', async ({ page }) => {
  await page.goto('/cart');
  await page.getByRole('button', { name: 'Add to cart' }).click();
  // BAD: the badge text is read synchronously, before the request resolves.
  const text = await page.getByTestId('cart-count').textContent();
  expect(text).toBe('1'); // fails whenever the network is a few ms slower
});

The remedy is a web-first assertion, which polls until the expectation holds or the timeout expires:

import { test, expect } from '@playwright/test';

test('stable: waits for the rendered count', async ({ page }) => {
  await page.goto('/cart');
  await page.getByRole('button', { name: 'Add to cart' }).click();
  // GOOD: expect() retries until the badge shows 1 or the timeout is hit.
  await expect(page.getByTestId('cart-count')).toHaveText('1');
});

Races hide in more places than a raw textContent() call. Any value you extract into a variable — count(), inputValue(), getAttribute(), evaluate() — freezes a snapshot, and every assertion downstream of that variable is a one-shot comparison. The same applies to isVisible() and isEnabled(), which return a boolean immediately rather than waiting; expect(await locator.isVisible()).toBe(true) is a coin flip where await expect(locator).toBeVisible() is not. Navigation introduces a second family: asserting on the old page's DOM after a client-side route change succeeds until the router happens to unmount faster, at which point the locator resolves against a detached node. Whenever you assert against content that loads asynchronously, lean on the patterns in Handling Dynamic Content instead of guessing a delay, and prefer waiting on a rendered element to waiting on networkidle, a distinction covered in Waiting for Network Idle vs Element State.

There is one legitimate reason to capture a value: when you need to compute with it. Even then, wrap the computation in expect.poll() so the whole expression retries rather than the assertion alone. That keeps the retry semantics without forcing you to express everything as a single matcher.

Root cause 2: animation and transition timing

Elements that slide, fade, or expand are mid-flight for a few hundred milliseconds. A click dispatched during a CSS transition can land on the element's old position, and a screenshot taken mid-animation differs pixel-for-pixel between runs. Playwright's actionability checks already wait for an element to be stable — it must report the same bounding box across two consecutive animation frames — before clicking, which covers most cases. For visual comparisons, disable animations so the rendered frame is identical every time:

import { test, expect } from '@playwright/test';

test('stable screenshot with animations frozen', async ({ page }) => {
  await page.goto('/modal');
  await page.getByRole('button', { name: 'Open' }).click();
  // 'disabled' fast-forwards CSS animations/transitions to their final state.
  await expect(page.getByRole('dialog')).toHaveScreenshot({ animations: 'disabled' });
});

The stability check has a blind spot worth knowing: it only observes the bounding box. An element animating opacity, colour, or a child's contents holds a constant box and passes the check while still visibly changing. Infinite animations are worse — a spinner, a marquee, a pulsing skeleton — because they never settle, so a screenshot assertion against a region containing one will differ on every run no matter how long you wait. animations: 'disabled' handles CSS-driven cases by fast-forwarding them to their end state, but animations driven by requestAnimationFrame in JavaScript are untouched and must be stubbed or masked. The techniques for excluding a volatile region from a comparison are in Masking Dynamic Regions in Snapshots, and the broader baseline strategy sits under Visual Regression Testing.

A second animation-adjacent trap is the element that moves after the click. Toast notifications, sticky headers that collapse on scroll, and virtualised lists that reflow when a row loads all shift the layout between the moment Playwright resolves a locator and the moment it dispatches the event. Playwright re-checks actionability and retries, so this usually self-corrects, but if the layout oscillates the retry loop can time out with "element is not stable". The fix is to wait on the settling condition explicitly — assert the toast is visible, then assert the list has the expected row count — before interacting with anything positioned relative to it.

Root cause 3: shared state between tests

If two tests touch the same user account, database row, or browser storage, the order they run in changes the result — and Playwright runs files in parallel by default across multiple worker processes. The cure is isolation. Each test should create the data it needs and never assume a clean global. Use per-test fixtures and a fresh browser context, both covered in Playwright Config & Fixtures, so no two tests can collide:

import { test, expect } from '@playwright/test';

// A fixture that provisions a unique account per test removes cross-test coupling.
const it = test.extend<{ account: string }>({
  account: async ({}, use) => {
    const id = `user-${Date.now()}-${Math.random().toString(16).slice(2)}`;
    await use(id); // each test gets its own id; nothing is shared
  },
});

it('signs in with an isolated account', async ({ page, account }) => {
  await page.goto(`/login?seed=${account}`);
  await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible();
});

Shared state is the root cause with the widest blast radius because it is not confined to your test process. Four layers can leak. Inside the browser, cookies and localStorage persist across tests in the same context — Playwright gives each test a fresh context by default, so this only bites when you deliberately reuse one. On disk, a storageState file written by an auth setup project is shared by every test that loads it, which is fine for reading and disastrous if a test mutates the underlying account; see Reusing Login State with storageState for the pattern that keeps roles separated. In the backend, a seeded fixture row that one test deletes and another expects is a classic order dependency. And in the runner itself, module-level let variables in a spec file are shared by every test in that file within a worker.

The diagnostic signature is unmistakable: the test passes alone and fails in the suite, or passes in the suite and fails when you shard. Confirm it by running the single file with --workers=1 --repeat-each=3; if the second and third repetitions fail while the first passes, the test is leaving residue behind. Resist the temptation to fix it with test.describe.serial(). Serial mode makes the order deterministic, which hides the coupling rather than removing it, and it means one failure skips every test after it. Reach for it only when the scenario is genuinely a multi-step flow that cannot be decomposed. If the expensive part is setup rather than data, a worker-scoped fixture gives you one setup per worker with per-test data on top — the approach described in Worker-Scoped Fixtures for Expensive Setup.

Root cause 4: network timing

Tests that hit a live backend inherit its latency, rate limits, and shifting seed data. The same spec can pass when the API is warm and fail when it is cold. Mock the unstable dependency so the response is fixed, and reserve live calls for a dedicated contract suite. The interception patterns under Network Interception Basics make a request deterministic with page.route() and route.fulfill().

The decision of what to stub is a judgement call, not a blanket rule. Stub anything you do not own and cannot make deterministic: third-party analytics, payment providers, feature-flag services, geocoding APIs. Stub anything whose content changes independently of your code — a "trending items" endpoint whose ordering shifts hourly will fail an assertion on the first row for reasons that have nothing to do with a regression. Do not stub the endpoint that is the subject of the test, or you are asserting against your own fixture. For suites that need realistic payloads without a live dependency, recording once and replaying is a better trade than hand-writing fixtures; see Recording and Replaying HAR Files.

Two network patterns cause flakiness even when the response is stubbed. The first is a request fired without being awaited — a fire-and-forget telemetry beacon or a prefetch that races the page's teardown, producing intermittent "Target closed" errors. Abort those routes so they never open. The second is a response that arrives too fast: a stub that resolves instantly can beat a loading state the test asserts on, so a test written against a 300ms live API fails against a 0ms mock. If the test genuinely needs to observe the intermediate state, delay the fulfilment deliberately rather than removing the assertion.

Removing time and randomness as hidden inputs

Two hidden inputs deserve their own treatment because they are invisible in a trace and produce failures that look like anything else. The first is the clock. A test that renders "3 minutes ago", expires a session after 15 minutes, or renders a date-picker defaulting to today will pass for weeks and then fail at a month boundary, across a daylight-saving transition, or when CI happens to run at 23:59:58. The second is randomness — Math.random() used for ids, shuffles, A/B bucketing, or sampled telemetry — which makes a different code path execute on a fraction of runs.

Playwright's page.clock API installs a controllable clock before any application code runs, and page.addInitScript() lets you replace Math.random with a deterministic sequence in the same window. Doing both turns two hidden inputs into fixed constants:

import { test, expect } from '@playwright/test';

test('renders relative timestamps deterministically', async ({ page }) => {
  // Freeze the page clock before navigation so the app never sees the real time.
  await page.clock.install({ time: new Date('2026-03-01T09:00:00Z') });

  // Replace Math.random with a seeded generator so ids and shuffles repeat.
  await page.addInitScript(() => {
    let seed = 42;
    Math.random = () => {
      // xorshift: deterministic, uniform enough for test fixtures
      seed ^= seed << 13;
      seed ^= seed >>> 17;
      seed ^= seed << 5;
      return (seed >>> 0) / 4294967296;
    };
  });

  await page.goto('/activity');
  // The feed item was created at 08:55Z in the fixture data.
  await expect(page.getByTestId('feed-age')).toHaveText('5 minutes ago');

  // Advance the virtual clock; timers and Date both move, no real waiting.
  await page.clock.fastForward('01:00');
  await expect(page.getByTestId('feed-age')).toHaveText('1 hour ago');

  // A session banner driven by a 15-minute setTimeout now fires predictably.
  await page.clock.fastForward('00:15');
  await expect(page.getByRole('alert', { name: 'Session expiring' })).toBeVisible();
});

Freezing the clock replaces a whole category of waitForTimeout() calls. A test that needed to sit idle for 15 real seconds to observe a timeout banner now advances a virtual clock instantly, which is both faster and deterministic. Set the timezone and locale explicitly in the same spirit — a machine configured for America/Los_Angeles renders different dates than a CI runner on UTC — using the emulation options described in Emulating Devices, Locales and Timezones. One caution: installing a clock before page.goto() is required for the application to pick it up, and any code that captured Date.now at module load in a previously loaded page will not be patched.

Detecting flakiness before it reaches main

Flakiness that only appears once in fifty runs is invisible to a single CI pass. Force it into the open by running a test many times in a row with --repeat-each, and let CI surface intermittent failures through the retry count. Amplify the effect by running the repetitions under contention — raise --workers past the core count, or throttle CPU — because most races only lose when the machine is busy. The full workflow of running under load, reading the flaky markers in the report, and converting hard waits to web-first assertions is covered in Detecting and Fixing Flaky Playwright Tests, which walks through a numbered diagnosis-to-fix procedure.

Measuring flake rate instead of guessing

Opinions about which tests are flaky are usually wrong; the tests engineers complain about are the ones that failed most recently, not the ones that fail most often. Replace the anecdote with a number. Every test result Playwright emits carries a retry index, so a test whose final status is passed with retry > 0 failed at least once — that is the definition of a flaky outcome. A small custom reporter accumulates those outcomes and writes a ranked table you can diff between runs:

import type { Reporter, TestCase, TestResult } from '@playwright/test/reporter';
import { writeFileSync } from 'node:fs';

// Tracks, per test, how many attempts ran and how many of them failed.
type Tally = { title: string; attempts: number; failures: number };

class FlakeRateReporter implements Reporter {
  private tallies = new Map<string, Tally>();

  onTestEnd(test: TestCase, result: TestResult) {
    // titlePath() gives project > file > describe > test, a stable identity.
    const key = test.titlePath().join(' > ');
    const tally = this.tallies.get(key) ?? { title: key, attempts: 0, failures: 0 };
    tally.attempts += 1;
    // A retried attempt that failed is exactly the signal we want to count.
    if (result.status === 'failed' || result.status === 'timedOut') tally.failures += 1;
    this.tallies.set(key, tally);
  }

  onEnd() {
    const rows = [...this.tallies.values()]
      .map((t) => ({ ...t, rate: t.failures / t.attempts }))
      .filter((t) => t.rate > 0)          // only tests that failed at least once
      .sort((a, b) => b.rate - a.rate);   // worst offenders first
    // Write a machine-readable artifact CI can publish and trend over time.
    writeFileSync('flake-rate.json', JSON.stringify(rows, null, 2));
  }
}

export default FlakeRateReporter;

Register it alongside the HTML reporter in playwright.config.ts and publish flake-rate.json as a build artifact. Over a few weeks the file becomes the input to every decision on this page: which test to quarantine, which quarantined test has actually been fixed, and whether the suite's overall flake rate is trending down. The reporter API surface — including onTestBegin, onStepEnd, and how to keep output readable in a sharded run — is documented in Writing a Custom Playwright Reporter, and the artifact plumbing sits under Reporters & Test Artifacts.

Set a policy on the number rather than debating each case. A reasonable starting threshold is that any test above a 1% failure rate over the last 200 attempts is automatically a bug, and any test above 5% is quarantined the same day.

Quarantine: protect the build without hiding the problem

When a flaky test blocks an urgent release, do not delete it and do not silently retry forever. Tag it so it runs but cannot fail the build, and track every quarantined test as a bug with an owner and a deadline. A common pattern is a @flaky annotation filtered into a non-blocking job:

import { test, expect } from '@playwright/test';

test('checkout total updates', { tag: '@flaky' }, async ({ page }) => {
  await page.goto('/checkout');
  await page.getByRole('button', { name: 'Apply coupon' }).click();
  await expect(page.getByTestId('total')).toHaveText('$90.00');
});

Run the blocking suite with --grep-invert @flaky and the quarantine suite separately. Quarantine is a holding pen, not a graveyard — a test that sits there for a month should be fixed or deleted.

The governance around the tag matters more than the tag itself. Quarantine must be visible: publish the count of quarantined tests on the same dashboard as the pass rate, and fail the build if the count grows past a ceiling. It must be attributed: the annotation should carry a ticket reference so nobody has to run git blame to find an owner. And it must expire: a scheduled job that lists tests tagged @flaky for longer than a sprint, and escalates them, is the difference between a holding pen and a landfill. The mechanics — the config projects, the CI job split, and the reporting that keeps quarantined results visible without blocking a merge — are laid out in Quarantining Flaky Tests Without Blocking CI, which shows how to keep the signal while removing the block.

Tuning retries and timeouts

Retries hide flakiness from the build but also mask it from you, so a retried test must always be reported as flaky, never silently green. Playwright already does this: a test that fails and then passes is marked flaky in the report rather than passed, and treating that status as a build warning rather than a success is a configuration decision you should make deliberately. The right retry count, the global timeout, and the per-assertion expect.timeout all belong in playwright.config.ts, with per-test overrides for the rare slow path. The complete configuration reference lives in Configuring Retries and Timeouts for Stable CI.

Two retries is the usual ceiling. One retry distinguishes a genuine failure from a transient one; two catches the rare double-fault; three or more mostly buys you a longer pipeline and a stronger illusion of stability. Note also that a retry restarts the whole test with a fresh context, so anything a test does to global state before failing is not undone — a test that half-created a record and then failed may fail its retry for a completely different reason, which is one more argument for per-test data.

Failure modes and debugging

Most flaky failures announce their family through the shape of the failure rather than its message. Matching the symptom to a likely cause before you open a trace saves a great deal of time.

Two failure modes deserve a note because they are commonly misdiagnosed. "Target closed" or "Browser has been closed" almost never means a browser crash; it means the test finished — or the context was disposed — while an operation was still in flight, usually an unawaited promise. Search the test for a missing await. And a strict-mode violation that appears intermittently is a resolution race, not a selector bug: the locator matched one element early in the render and two after a list hydrated. Tighten the scope rather than adding .first(), following Resolving Strict Mode Violations.

CI/CD considerations

CI is where flakiness is discovered and where it is most expensive, so configure the pipeline to produce evidence rather than just a verdict. Enable trace: 'on-first-retry' so every flaky outcome ships with a recording of the failed attempt at no cost to green runs. Publish the HTML report and the trace zips as artifacts, and keep them long enough to compare a failure against the same test's last success. Pin the runner image and the browser versions so a "new" flake is never just a silently upgraded engine; caching those browsers, as described in Caching Playwright Browsers in CI, keeps that pinning cheap.

Sharding interacts with flakiness in both directions. It reduces per-machine contention, which removes some timing races, but it also redistributes tests across shards on every run, which exposes order dependencies that a fixed single-machine order was hiding. Treat a failure that only appears after enabling sharding as a shared-state bug rather than a sharding bug; the layout is covered in Running Playwright Tests in GitHub Actions with Sharding and the general pipeline shape in CI/CD Integration.

Finally, resist the urge to give CI a longer global timeout than local development. A longer timeout does not make the suite more stable; it makes each genuine failure take longer to report and lets real regressions masquerade as slow paths. Prefer per-assertion timeouts on the handful of operations that really are slow.

Confirming a fix with the trace

Once you believe a test is fixed, prove it. Reproduce the original failure, open the recorded trace, and step through the exact action that flipped to find whether the application or the assertion was at fault. The Trace Viewer & Debugging guide shows how to read the action timeline, network panel, and DOM snapshots; the network and console tabs in particular usually reveal the late request or the swallowed error behind a race, as covered in Reading Network and Console Tabs in Traces. Then verify statistically rather than anecdotally: run the fixed test with --repeat-each=50 under a loaded machine and require fifty consecutive passes before you remove the quarantine tag. One green run proves nothing about a failure that happens two percent of the time.

Deep dives beneath this guide

Three focused walkthroughs sit under this page. Detecting and Fixing Flaky Playwright Tests is the hands-on procedure for reproducing an intermittent failure on demand and converting the hard waits behind it into web-first assertions. Configuring Retries and Timeouts for Stable CI covers every timeout knob in playwright.config.ts and which one to reach for when a specific operation is genuinely slow. Quarantining Flaky Tests Without Blocking CI shows the project and job split that lets a suspect test keep reporting signal without ever turning a merge red.

Frequently Asked Questions