Build Log: A Flaky-Test Detector for CI

Why Flaky Tests Break CI Trust

Flaky tests break CI trust because merges require every test to pass, so intermittent red builds push developers to distrust results and overlook genuine failures, according to the pytest flaky-tests documentation. A detector’s first job is protecting that trust in CI results.

Once a suite trains people to re-run builds until they go green, a real regression can ship under the same red signal everyone has learned to ignore. This guide walks through a blueprint: an attempt record model, a rerun configuration, a quarantine rule, and a report. Every code block is illustrative and untested.

The Numbers That Justify a Detector

Google’s engineering blog reports, in a 2016 post, that about 1.5% of all test runs return a flaky result, almost 16% of tests have some flakiness, and about 84% of observed post-submit pass-to-fail transitions involve a flaky test (Google Testing Blog). This guide treats them as historical context, not current benchmarks.

These are one company’s observations. No team-level rate is available in the sources, so calibrate the detector on your own run records before setting any policy.

The Data Model: Defining Flaky as a Record

Under the definition in the Google Testing Blog, a test is flaky when it both passes and fails on the same code. The detector should therefore store each attempt as its own record, keyed by test, commit, and run, and aggregate attempts into stable-pass, stable-fail, or flaky (Google Testing Blog). Attempts are the unit of evidence, not the final status.

Illustrative and untested: assembled from the cited documentation, not executed.

-- Artifact A: attempt-level record (illustrative, untested)
CREATE TABLE test_attempt (
  test_id       TEXT NOT NULL,     -- suite + node ID
  commit_sha    TEXT NOT NULL,
  run_id        TEXT NOT NULL,
  attempt       INTEGER NOT NULL,  -- 1 = first run, 2+ = reruns
  status        TEXT NOT NULL CHECK (status IN ('pass','fail','error','skip')),
  failure_class TEXT CHECK (failure_class IN ('assertion','timeout','teardown')),
  duration_s    REAL,
  worker        TEXT,
  platform      TEXT,
  PRIMARY KEY (run_id, test_id, attempt)
);

How this can hurt you: if the schema keeps only the final status, the retry evidence that reveals flakiness is gone before detection runs.

The Attempt Log: Never Collapse Retries

The pytest-rerunfailures marker docs expose reruns, reruns_delay, condition, only_rerun, and rerun_except as per-test settings. Every attempt the plugin makes is a separate outcome, so this design persists each one as its own row. Collapsing them to a single pass would discard the evidence the detector exists to read.

Illustrative and untested: assembled from the cited documentation, not executed.

# Artifact B: marker usage (illustrative, untested)
# Register the marker in pytest.ini or pyproject.toml, as the plugin's
# documentation describes, before using it:
#   [tool.pytest.ini_options]
#   markers = ["flaky: rerun on failure (pytest-rerunfailures)"]
import pytest

@pytest.mark.flaky(reruns=3, reruns_delay=2)
def test_checkout_total():
    ...

@pytest.mark.flaky(reruns=3, reruns_delay=2, only_rerun=["AssertionError"])
def test_price_rounding():
    ...

@pytest.mark.flaky(reruns=2, rerun_except="ValueError")
def test_session_refresh():
    ...  # condition=... also exists; see the docs for its expression form

How this can hurt you: a rerun that passes is easy to read as success. The attempt log must keep that attempt visible.

Retry Policy: Rerun Only What Failed

Google’s post lists rerunning only failing tests, auto-rerun on failure, and marking a test flaky so it fails only after 3 consecutive failures among its documented mitigations, though the author calls the last approach “hardly a perfect solution” (Google Testing Blog). Retries are a containment tool, not a fix.

Several open-source plugins implement reruns, including pytest-rerunfailures, flaky, and pytest-flakefinder, as the pytest flaky page lists.

How this can hurt you: a retry that passes can turn a real race into a green build. Record the failed attempt even when the final status is pass.

Quarantine Policy: Threshold, Bug Filing, Expiry

Google’s post describes a tool that automatically quarantines tests whose flakiness is too high and files a bug. It does not give a numeric threshold, so this guide uses placeholders for the threshold, window, and review deadline. Those values are design choices for each team.

Illustrative and untested: assembled from the cited documentation, not executed. Placeholders are not measured values.

# Artifact C: quarantine rule (illustrative, untested)
quarantine_rule:
  match: "tests/**"                  # test ID glob
  metric: flaky_runs_in_window       # runs classified flaky
  threshold: <THRESHOLD>             # placeholder: set per team
  window: <WINDOW>                   # placeholder: rolling window
  actions:
    - file_bug
    - notify_owners
    - mark_non_blocking
  review_deadline: <REVIEW_DEADLINE> # placeholder: a date, not a duration

How this can hurt you: a threshold set too low files noise, and one set too high lets flakes accumulate unreviewed.

Failure Mode: Quarantine Can Hide a Real Bug

The Google Testing Blog author warns that quarantine “could easily mask a real race condition,” which is why this design adds a review deadline, keeps quarantined tests running, and treats expiry as a required field rather than an option (Google Testing Blog). The mitigation is a design choice, not a measured fix.

Quarantine hides failures from the gate, not from the log. Luo et al. report Concurrency and Async Wait among the top three root-cause categories (Luo et al.). Those categories are where a quarantined test is most worth a race review, but the paper does not rank them by likelihood of being a real bug. Each quarantined test should carry an owner who must answer whether it is a race before the review deadline.

How this can hurt you: quarantine can convert a real bug into a permanent non-blocking test that nobody re-examines.

Running Quarantined Tests Non-Blocking

The pytest docs say xfail with strict=False can act as a manual quarantine but is “rather dangerous to use permanently,” so this design keeps quarantined tests executing, reports them separately, and excludes them from the gating exit code (pytest flaky docs). The exit-code split is proposed design, not a documented pattern.

Illustrative and untested: the report and gating logic are design proposals, not executed code.

How this can hurt you: Luo et al. report that about 34% of Async Wait flaky tests use a simple call with a time delay to enforce ordering (Luo et al.). A quarantined test that sleeps can keep passing or failing for the wrong reason while it runs unnoticed.

Classifying Attempts per Test and Commit

Artifact D applies the flake definition from the Google post to the attempt records. A test is flaky only if one commit shows both a pass and a fail across its attempts and runs; a test that only skips is not counted as a failure.

Illustrative and untested: aggregation for the classification, not executed.

-- Artifact D: classify attempts per test and commit (illustrative, untested)
SELECT test_id, commit_sha,
  CASE
    WHEN SUM(status = 'pass') > 0
     AND SUM(status IN ('fail','error')) > 0 THEN 'flaky'
    WHEN SUM(status = 'pass') > 0           THEN 'stable-pass'
    WHEN SUM(status IN ('fail','error')) > 0 THEN 'stable-fail'
    ELSE 'skipped'
  END AS classification
FROM test_attempt
GROUP BY test_id, commit_sha;

Reporting: From Attempt Log to a Flake View

The pytest docs note that flaky tests appearing under pytest-xdist parallel runs can indicate test-order dependence, so the report this design proposes shows per-test pass/fail timelines against commit SHAs with worker and platform metadata (pytest docs). It also lists retry counts, a quarantined section, and a flake leaderboard.

Illustrative and untested: synthetic example, not from a real run.

Artifact E: report output (synthetic example, not from a real run)
run_id: <RUN_ID>   commit: <COMMIT_SHA>   platform: <PLATFORM>
Gating result: PASS (quarantined tests excluded from exit code)

Quarantined (non-blocking):
  tests/test_checkout.py::test_price_rounding  retries=<N>  status=flaky
    owner=<OWNER>  review_deadline=<REVIEW_DEADLINE>

Flake leaderboard (classified flaky in window <WINDOW>):
  1. tests/test_session.py::test_refresh  attempts=<N>  worker=<WORKER>

How this can hurt you: a report that shows only the latest status hides order dependence that appears under parallel runs.

Failure Modes and What to Fix Instead

Luo et al. studied 201 commits that likely fix flaky tests across 51 open-source projects and found Async Wait, Concurrency, and Test Order Dependency as the top three root-cause categories. The paper describes rerun-based approaches as “workarounds” rather than solutions (Luo et al.).

The same study reports that 78% of flaky tests were flaky the first time they were written, which suggests checking tests when they are added. It also reports that 96% of flaky tests are platform-independent. Among Async Wait flakes, 54% were fixed with waitFor, which often removes the flakiness rather than reducing its chance. Among Test Order Dependency flakes, 74% were fixed by cleaning shared state between runs (Luo et al.).

How this can hurt you: a detector that reports “passes on retry” can encourage adding sleeps. Treat sleep-based ordering in a fix as a review flag.

FAQ

Each answer names its source. Where a claim is proposed design rather than documented fact, the answer says so, since this guide was not executed and no benchmark of the detector exists.

Why rerun only the failing tests instead of the whole suite?

Rerunning only failing tests is one of the core mitigations Google describes, and it keeps the detector’s signal focused on the tests that actually flipped (Google Testing Blog). The rationale for isolating the flaky signal is design reasoning here, not a measured result from this build.

Isn’t quarantine just hiding the problem?

It can be. Google’s author warns quarantine “could easily mask a real race condition” (Google Testing Blog), so this design expires quarantine through a review deadline and keeps the test running non-blocking with its own report. The expiry values are placeholders, not recommendations from a source.

How do I tell order dependence from a timeout flake?

Flakes that appear only under pytest-xdist parallel runs can indicate test-order dependence, according to the pytest docs. Luo et al. report that cleaning shared state between runs fixed 74% of order-dependency flakes. The sources do not define a separate timeout class, so this design’s failure_class field is its own labelling choice, not a documented category.

The Bottom Line

Build the detector as a record-keeping system first and a quarantine policy second: persist every attempt, rerun only failing tests, and treat quarantine as a time-limited, non-blocking state, because sources document both its value and its masking risk (Google Testing Blog). Treat root-cause fixes as the real goal, since Luo et al. describe rerun-based approaches as workarounds.

Fill the placeholders with values your team chooses, and validate the design against your own run history before trusting it.

How This Guide Was Built

This guide is assembled from official documentation and community reports — we did not run this build ourselves. Sources are the Google Testing Blog, the Luo et al. FSE 2014 paper, the pytest flaky-tests docs, and the pytest-rerunfailures marker docs, each linked where cited.

All code and configuration is illustrative and untested. The sources do not report a public benchmark of this detector, no hands-on run was performed, and no team-specific flake rate is available. The Google post dates from 2016 and the Luo et al. study from 2014, so treat their figures as historical evidence for design choices, not current measurements.

📖 Related Reads

• ToolBrain — tool reviews, LLM comparisons, and AI workflow guides

Cross-links automatically generated from CodeIntel Log.