Best Flaky Test Detection Tools for Playwright (2026)

You turned on retries because a few tests kept failing in CI for reasons nobody could pin down, and the pipeline went green. Months later the suite is slower and some tests only pass on their second attempt. Nobody can say which ones, or how often. When a real regression finally turns a build red, the first reaction is to re-run it.

Playwright does record the signal. A test that fails on its first run and passes when retried is labelled "flaky" (Playwright's retry docs). But that label lives in one run's report. One report can't tell you that the same test has flaked on most of this month's runs, and that number tells you which test to fix first.

So judge a detection tool by what it does with that label. Does it read the label straight from Playwright, or rebuild it from a JUnit file? Does it keep it across runs, quarantine the test and tell someone? Our short answer, tool by tool: Currents' reporter is a native Playwright reporter, and its Test Explorer shows the tests with the highest flakiness rate. TestDino takes one line in the Playwright config and builds a flaky rate for every test across multiple runs. Trunk Flaky Tests takes Playwright results as JUnit XML reports, and Trunk says it works with any language, any test runner and any CI provider. Mergify Test Engine catches flaky tests on the pull request before they merge.

How we picked

We judged every tool first on how Playwright results reach it, because the route decides what the tool can see.

Reporter or JUnit XML

Playwright Test supports custom reporters, written as a class that implements some of the reporter methods. In the reporter API, a test that passes on a second retry has the outcome 'flaky'. A tool that ships a Playwright reporter gets Playwright's own verdict instead of rebuilding it from the attempts behind each outcome.

The other route is JUnit XML. Playwright writes the file, and a CI step uploads it. CircleCI, for example, reads the JUnit XML files that your test runner writes. The catch: the file sets the limit. One user whose separate flake detector reads JUnit reports asked for every retry attempt to appear in Playwright's JUnit output, because the detector could not see attempts that failed before a retry passed (#29446). The issue was closed as completed in March 2026 with no release label. Before you commit to a JUnit-based tool, open the XML your CI already produces and check what a fail-then-pass test looks like in it.

Trying a reporter

You can try a reporter-based tool without changing your config. The --add-reporter option adds a reporter on top of the ones configured in the config file, so one CI job can feed a vendor while every other run stays as it is. Here it adds the Allure reporter to a single run:

npm install --save-dev allure-playwright
npx playwright test --add-reporter=allure-playwright

History, quarantine and setup

After the route, three more criteria decided the ranking. History comes first: a tool that only repeats the single-run label adds little to what Playwright already prints, while one that tracks a test across runs and commits turns scattered flakes into a ranked list. Quarantine comes second, and only the kind that keeps a flaky test running and tracked counts. A skipped test produces no data, so nobody learns whether a fix worked. Last is what the tool asks of your CI: a reporter line, a config wrapper, an environment variable or an upload step.

The tools fall into four groups: Playwright-native reporters, JUnit-based detectors, flake detection built into a CI provider, and observability or test management platforms with flake tracking on top. We left out self-healing test authoring platforms. They aim to prevent flakes in new tests, not to find them in a suite you already run. We also left out tools whose own documentation gave us no route for Playwright results.

Playwright's built-in flake detection

Retries come first. They are off by default, so until you turn them on, a flaky test looks exactly like a broken one: both fail.

// Anti-pattern: retries absorb flaky tests, and the run still passes
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
});

import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  // Retry at the end, one test at a time, away from the rest of the suite
  retryStrategy: 'isolated',
  // Any flaky test fails the CI run
  failOnFlakyTests: !!process.env.CI,
  use: {
    // Keep a trace for every failed run and every retry
    trace: 'retain-on-failure-and-retries',
  },
});

Order and repetition

Some flakes only show up when tests run in a different order, or many times over. The --shuffle option runs tests in random order: it shuffles test files, and individual tests in parallel mode, while tests that must run together, such as a serial suite, keep their order. The repeatEach option sets how many times each test is repeated, and the docs call it useful for debugging flaky tests.

This config repeats each test only when a flake hunt is on:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  // Repeat every test five times, but only during a flake hunt
  ...(process.env.FLAKE_HUNT ? { repeatEach: 5 } : {}),
});

Then start the hunt with a shuffled order:

FLAKE_HUNT=1 npx playwright test --shuffle

Run the hunt nightly or locally, not on the pull request gate: a random order that fails once is a lead to chase, not a reason to block someone's merge.

Playwright Test labels each test passed, flaky or failed based on its first run and its retries. History across runs, quarantine and alerts are what the tools below add.

Currents

Currents is our product, and the Currents team writes this blog. It marks a Playwright test flaky when the test has retries enabled and doesn't pass on its first attempt.

The Currents Actions Engine runs custom workflows for testing. For example, it can conditionally and temporarily skip or quarantine a test.

Currents fits Playwright teams that want flake history, quarantine and alerts in one place. Its flaky label depends on retries, so a team that runs with retries off gets no flaky label to work from.

TestDino

TestDino describes itself as a Playwright cloud companion: it records CI runs, detects flaky tests, and keeps traces and fix workflows available to teams and to AI agents over MCP. Setup is one line in the Playwright config, and after every run TestDino receives test results, retry data and timing information.

Detection starts from the retry signal. TestDino flags a test as flaky when it fails and then passes on retry, and across runs it computes a flaky rate per test: the percentage of runs in which the test behaved flakily. You get a single-run label and a cross-run number for the same test. You can also set pass-rate or flaky thresholds on pull requests, and TestDino posts a status check that keeps code below the bar from merging (TestDino).

TestDino fits Playwright teams on GitHub Actions or GitLab CI that want the flaky rate to gate merges. TestDino's homepage names GitHub Actions and GitLab CI as places to add its reporter.

Mergify Test Insights

Mergify's product page names it Test Engine, and Test Engine is the test layer of Mergify's CI platform. It catches flaky tests on the pull request before they merge, and quarantines the flaky tests already in your suite so they stop blocking merges.

For Playwright, you wrap your playwright.config.ts export with Mergify's withMergify wrapper. The wrapper adds the Mergify reporter and keeps any reporters, globalSetup and globalTeardown you already have configured, so your existing list or HTML reports and your setup scripts stay as they are.

The PR check sets it apart. When a pull request adds or changes tests, Test Engine reruns them on the same commit, flags any that switch between pass and fail, and labels each new test healthy, flaky or broken before anyone merges. A quarantined test keeps running and reporting, but its failures stop blocking merges or failing the pipeline, on every branch (Mergify).

Mergify Test Insights fits teams whose main pain is new flakes arriving through pull requests. If you don't want your flake tooling tied to Mergify's CI platform, pick a standalone tool instead.

Trunk Flaky Tests

Trunk Flaky Tests finds, quarantines and removes flaky tests, and Trunk says it works with any language, test runner or CI provider. Playwright connects through JUnit XML: you configure Playwright to write JUnit reports, and a CI step uploads them to Trunk for analysis (Trunk's Playwright guide). Its quarantine is built around a habit worth breaking.

Who it fits

Trunk Flaky Tests fits organisations where Playwright is one runner among several and JUnit XML is the format they all share. A Playwright-only team gets less from it: Trunk sees whatever the JUnit file holds, while a reporter-based tool reads the retry outcome directly, so check that file first.

BuildPulse

BuildPulse finds flaky tests across every branch and pull request. It compares results across builds and commits to spot tests that pass and fail without a code change, then ranks them by the engineering time and disruption each one causes. You can quarantine the worst ones in one click.

Setup is a step in your CI workflow that reads the JUnit XML your runner already writes, and Playwright is one of the runners BuildPulse lists (BuildPulse). GitHub Actions and Bitbucket Pipelines have built-in integrations, and other CI systems upload results through the BuildPulse API.

BuildPulse fits teams with a flake backlog too long to fix at once, who need to know which flake costs the most. On a CI system other than GitHub Actions or Bitbucket Pipelines, budget time for the API integration.

Datadog Test Optimization

Datadog Test Optimization shows test metrics and results from CI, so teams can investigate failures and performance problems in the code they own rather than in the pipelines that run it. To connect Playwright, you set NODE_OPTIONS to load the Datadog tracer and run the tests as usual, optionally naming the test session with DD_TEST_SESSION_NAME (Datadog's JavaScript setup):

npm install --save-dev dd-trace
# Agentless mode for a cloud CI provider; add DD_SITE if your Datadog site isn't US1
NODE_OPTIONS="-r dd-trace/ci/init" DD_CIVISIBILITY_AGENTLESS_ENABLED=true DD_API_KEY=$DD_API_KEY DD_TEST_SESSION_NAME=playwright-e2e npx playwright test

From its Flaky Tests Management UI, you can quarantine or disable flaky tests so known flakes stop breaking builds. The same UI creates work items and Jira issues to track the fixes.

Datadog Test Optimization fits teams already on Datadog that want flaky tests next to the rest of their telemetry. If you're not on Datadog, adopting a full observability platform to get flake tracking is a lot of platform for one job.

Buildkite Test Engine

Buildkite Test Engine handles flaky tests in real time, splits tests across jobs, and adds performance analytics on top of your existing CI compute. By default it marks a test flaky when the test runs more than once on the same commit SHA with different results, whether those runs happen in one build or across several. You can also set up workflows that combine several detection heuristics, then categorise and quarantine problem tests automatically so merge queues keep moving.

Test Engine is not limited to Buildkite Pipelines: it also takes results from GitHub Actions, Jenkins, CircleCI and other CI tools (Buildkite Test Engine).

Buildkite Test Engine fits teams that want test splitting and flake management from one vendor. Besides CI tools, Buildkite lists test collectors for RSpec, Jest, Cypress, pytest and Swift, and accepts JUnit XML-formatted test results, so confirm how your Playwright results reach it before you commit.

CircleCI Test Insights

CircleCI Test Insights puts flake detection inside the CI provider itself. It analyses how your tests performed in recent runs, under the Tests tab on the Workflow Insights page. A test counts as flaky when it both failed and passed on the same commit within a 14-day window.

Results come from the JUnit XML Playwright writes, uploaded with the store_test_results step (CircleCI's test data docs). CircleCI says to use Playwright 1.34.2 or later, because earlier versions may write JUnit XML that some of its testing features can't read.

CircleCI Test Insights fits teams already on CircleCI as a first step that adds no new vendor. Test Insights uses the results your CircleCI jobs upload with store_test_results. Test Insights flags a test as flaky when it failed and passed on the same commit in a 14-day window. If you also run Playwright on a second CI system, or want flakes judged over a longer window than 14 days, move to a CI-independent tool.

Allure TestOps

Allure TestOps is a platform for managing manual and automated testing: it imports results from any source, integrates with any CI platform, tracks tests and reports failures. Qameta's Playwright guide installs allure-playwright and runs it alongside the line reporter (Qameta's Playwright guide):

npx playwright test --reporter=line,allure-playwright

Its flaky rule looks at status transitions over a test's most recent executions. Allure TestOps treats a test as unstable when its most recent 10 executions contain at least 3 status transitions.

Allure TestOps runs in Qameta's cloud or on-premise, so it fits teams that must self-host or that manage manual test cases alongside automated ones. If flake detection is all you need, a test management platform is a heavy way to get it.

Flakiness.io

Flakiness.io's Playwright package is a custom Playwright Test reporter. Once you add it to the config and run your tests, it writes a Flakiness Report to the ./flakiness-report folder. The reporter converts Playwright results into the Flakiness JSON format, recording test outcomes, attachments, system utilization and environment information.

That environment data is the reason to pick it. The reporter captures system utilization and environment information alongside test outcomes and attachments (Flakiness.io's README).

Flakiness.io fits teams that want flake data with environment context from a Playwright-native reporter. The reporter needs Playwright 1.57.0 or later, and 1.62.0 or later for its balanced sharding feature. If your suite is pinned to an older Playwright, start with another tool from this list.

At a glance

The tools split on two things: how they decide a test is flaky, and what they do once they have decided.

We left out prices and plans, which change faster than this list does.

How to choose

Start by asking whether you need a tool at all. With a small suite, one repo and a few flakes a month, turn on retries and failOnFlakyTests, and set the trace to 'retain-on-failure-and-retries', before you buy anything. With retries on, failOnFlakyTests makes the run fail if any test is marked flaky. A tool earns its place once you need history across runs or quarantine, and even then you can compute a flake rate from every run's report before you pay for one.

By situation

• If your suite is Playwright-only, default to a Playwright-native reporter, which reads the retry outcome directly, and pick TestDino if pass-rate or flaky thresholds on pull requests are the priority.
• If you run many languages and runners, use Trunk Flaky Tests or BuildPulse through JUnit XML, after you check whether your JUnit output records failed retry attempts.
• If new flakes keep entering through pull requests, use Mergify Test Insights.
• If you're already on CircleCI, Buildkite or Datadog, start with what your platform has; Buildkite Test Engine also takes test results from other CI tools such as GitHub Actions, Jenkins and CircleCI.
• If you self-host, or have manual tests to manage too, use Allure TestOps.

Whichever you pick, prefer quarantine that keeps the test running and leaves Broken tests out, the way Trunk documents it. A quarantine that only skips tells you a flake left the build, not whether it left the code.

Frequently asked questions