Quarantining Flaky Tests Without Blocking CI

One test that fails on roughly a third of runs will block every unrelated pull request in the repository until somebody fixes it, and the usual reaction — commenting it out, wrapping it in test.skip(), or bumping retries to five for the whole suite — either deletes the coverage or hides real regressions across hundreds of other tests. Quarantine is the middle path: the suspect test keeps running on every commit, its failures keep producing traces and screenshots, but its verdict is deliberately excluded from the check that gates the merge. This page shows how to build that split with Playwright's own tag and project filtering, wire it into a pipeline as two jobs, and attach the governance that stops quarantine from becoming a permanent parking lot.

Root cause: one shared verdict for tests with different trust levels

A CI pipeline normally collapses every test result into a single process exit code, so a test you already know is unreliable carries exactly the same veto power as a test you trust. Playwright cannot infer that difference — retries applies suite-wide unless you scope it, and a retried test that eventually passes is still reported with the flaky status but does not fail the run, which is a blunt instrument that also masks genuinely broken tests. The fix is to make trust level an explicit, filterable property of each test and then run two filtered passes whose exit codes are consumed differently. This is the operational counterpart to the diagnosis work in Detecting and Fixing Flaky Playwright Tests, part of the Flaky Test Management guide under Debugging & Test Observability.

Why the obvious alternatives fail

Engineers reach for four built-in mechanisms before they reach for quarantine, and each loses something. test.skip() stops the test from executing at all, so you get no trace, no screenshot, and no evidence about whether the underlying bug is still present — the coverage silently rots. test.fixme() is honest about intent but behaves the same way at runtime: the body never runs. test.fail() is worse for a flaky test than for a broken one, because it inverts the verdict: Playwright marks the test as failing when the body unexpectedly passes, reporting Expected to fail, but passed., so a test that succeeds two runs in three turns your build red on those two runs. Deleting the spec is the only option that is at least honest, but it removes the only automated evidence you had for that user journey.

Minimal reproducible example

The test below is genuinely unreliable: the coupon total is rendered by an async price recalculation whose timing depends on a third-party tax service. Tagging it declares the trust level at the point where the test is defined, and the annotation records who owns the repair and when the exemption runs out.

import { test, expect } from '@playwright/test';

// The `tag` option is read by --grep / --grep-invert and by project-level
// grep filters, so it is the single switch that routes this test to a lane.
// The `annotation` option is surfaced in the HTML report and in trace metadata,
// which is what makes the quarantine auditable rather than invisible.
test(
  'checkout total updates after applying a coupon',
  {
    tag: '@quarantine',
    annotation: {
      type: 'quarantine',
      description: 'FLAKE-4821 owner:@dana expires:2026-08-21 tax service latency',
    },
  },
  async ({ page }) => {
    await page.goto('/checkout');
    await page.getByRole('button', { name: 'Apply coupon' }).click();
    // Fails intermittently: the total repaints twice, once optimistically
    // and once after the tax call resolves, so the first paint can win.
    await expect(page.getByTestId('order-total')).toHaveText('$90.00');
  },
);

Run that spec with npx playwright test --grep-invert @quarantine and it is not collected at all; run it with npx playwright test --grep @quarantine and it is the only test collected. Those two commands are the entire mechanism — everything that follows is about making them run automatically and making the quarantine list decay.

Step-by-step fix

1. Tag the test and record an owner at the declaration site. Add { tag: '@quarantine' } as the second argument to test() and attach an annotation carrying the issue key, the owning engineer, and an expiry date. Keeping this metadata in the spec rather than in a wiki page means the reviewer of the quarantine pull request sees exactly what is being exempted and for how long.
2. Define two projects in playwright.config.ts that partition the suite. Give the blocking project grepInvert: /@quarantine/ and the quarantine project grep: /@quarantine/. Because the two regular expressions are exact complements, every test lands in precisely one lane and no test can quietly escape both. Project-level configuration is covered more broadly in Playwright Config & Fixtures.
3. Give the quarantine project its own retry budget and artifact settings. Set retries: 3 and trace: 'retain-on-failure' on the quarantine project only, so you accumulate diagnostic material on the tests you are actively investigating without slowing the blocking lane or inflating artifact storage for the whole suite. The trade-offs of each retry count are worked through in Configuring Retries and Timeouts for Stable CI.
4. Run the two projects as separate CI jobs and gate only on the first. Invoke --project=blocking in the job that is a required status check and --project=quarantine in a job marked continue-on-error: true. Add --pass-with-no-tests to the quarantine invocation so the job stays green on the day the quarantine list is finally empty instead of exiting with Error: No tests found.
5. Publish the quarantine results instead of discarding them. Upload the quarantine job's HTML report and traces as build artifacts, and post the pass rate as a pull-request comment or a dashboard metric. A quarantined test whose pass rate silently drops from 70% to 0% has become a real regression, and you only see that if the numbers are visible — see Reporters & Test Artifacts for wiring the output up.
6. Enforce the expiry date from inside the blocking lane. Keep a quarantine.json manifest and add one ordinary test, untagged so it runs in the blocking project, that fails when any entry is past its expiry date. This is the piece that gives quarantine a half-life: an exemption nobody renews eventually blocks the merge exactly the way the flaky test used to.
7. Promote tests back out once they are green. Require a fixed number of consecutive clean quarantine runs — ten is a reasonable bar for a test that failed one run in three — then delete the tag, the annotation, and the manifest entry in a single pull request that references the trace proving the root cause was fixed.

Here is the configuration that implements steps 2 through 4, together with the expiry guard from step 6.

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  // Retries stay low globally; the quarantine project raises its own below.
  retries: process.env.CI ? 1 : 0,
  reporter: process.env.CI ? [['blob'], ['list']] : 'list',
  projects: [
    {
      name: 'blocking',
      use: { ...devices['Desktop Chrome'], trace: 'on-first-retry' },
      // Complement of the quarantine filter: everything without the tag.
      grepInvert: /@quarantine/,
    },
    {
      name: 'quarantine',
      use: { ...devices['Desktop Chrome'], trace: 'retain-on-failure' },
      // Only tagged tests; higher retries because we want failure statistics,
      // not a verdict, from this lane.
      grep: /@quarantine/,
      retries: 3,
    },
  ],
});

import { test, expect } from '@playwright/test';
import quarantine from '../quarantine.json';

type Entry = { owner: string; issue: string; expires: string };

// Untagged on purpose: this test runs in the blocking project, so a stale
// exemption fails the required check and forces a decision.
test('every quarantine entry is still within its expiry window', async () => {
  const today = new Date().toISOString().slice(0, 10);
  const expired = Object.entries(quarantine as Record<string, Entry>)
    .filter(([, entry]) => entry.expires < today)
    .map(([id, entry]) => `${id} (${entry.owner}, ${entry.issue})`);

  // The message is what the on-call engineer reads in the CI log.
  expect(expired, `Quarantine entries past their expiry date: ${expired.join('; ')}`).toEqual([]);
});

The pipeline side is two jobs over the same checkout, with the second one explicitly declawed:

jobs:
  blocking:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npx playwright install --with-deps
      - run: npx playwright test --project=blocking
  quarantine:
    runs-on: ubuntu-latest
    continue-on-error: true
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npx playwright install --with-deps
      - run: npx playwright test --project=quarantine --pass-with-no-tests
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: quarantine-report
          path: playwright-report/

Troubleshooting variants

The quarantine job exits 1 with "Error: No tests found"

Playwright treats a filter that matches nothing as a configuration mistake and exits non-zero, which is correct for a typo in --grep but wrong for a quarantine lane that has legitimately been emptied. Add --pass-with-no-tests to that invocation only. If the job fails even with tests present, check that the tag string in the spec starts with @ — Playwright rejects a bare tag at collection time with a message of the form Tag must start with "@" symbol, and a rejected tag means the filter matches nothing while the test itself still runs in the blocking lane.

A tag on test.describe swallows the whole file

Tags declared on a test.describe() block are inherited by every test inside it, and they compose with tags on the individual tests rather than replacing them. Quarantining one case by tagging its enclosing block therefore removes an entire feature's coverage from the gate, usually without anyone noticing until a regression ships. Tag the individual test() call, and if you genuinely need to quarantine a group, list the affected tests in the manifest so the count is visible. Verifying which tests landed where takes one command: npx playwright test --project=quarantine --list.

Quarantined failures still turn the pipeline red

continue-on-error set on an individual step still marks the job as failed in some CI systems while allowing later steps to run; it belongs at the job level, as in the workflow above. Two more causes are common. First, the quarantine job may still be listed as a required status check in the branch protection rules — remove it there, since the workflow flag cannot override repository settings. Second, if you run both projects in a single Playwright invocation, the process exits with the worst status across all projects, so quarantined failures inevitably fail the run; keep them as separate commands. When you shard the blocking lane, apply the same separation described in Running Playwright Tests in GitHub Actions with Sharding.

Verification

Prove the split three ways before you trust it. First, run npx playwright test --project=blocking --list and npx playwright test --project=quarantine --list and confirm that the two counts add up to the output of npx playwright test --list — if they do not, some test is matched by both regular expressions or by neither, and a test matched by neither is coverage you have silently lost. Second, force the failure: temporarily change the quarantined test's expected text to a value that can never match, push, and confirm that the merge check goes green while the quarantine job goes red and still uploads its report. Third, set an expiry date in the past in quarantine.json and confirm the guard test fails the blocking job with the entry listed in its message.

Then confirm the artifacts are usable. Download the quarantine job's report, open a failing run's trace, and check that the action timeline and network panel actually contain the evidence you need — a quarantine lane that produces no diagnostics is just a slower version of test.skip(). Reading those panels is covered in Analyzing Test Failures with the Playwright Trace Viewer, and the capture settings that control what ends up in them are in Capturing Screenshots and Video on Test Failure. If you want the quarantine pass rate emitted as a machine-readable metric rather than read by eye, a small reporter that counts results by tag is the cleanest place to compute it — see Writing a Custom Playwright Reporter.

Frequently Asked Questions