Flakiness analytics

Upliftr is AI debugging agents: they drive a real browser, read your servers and logs (read-only, they never mutate your code, data, or infrastructure), root-cause failures to the exact backend line, and re-run the checks they save for you in CI. Flakiness analytics is how the agents keep those checks honest. It computes flake and run analytics over your persisted run history, pass/fail trends, per-case flake rates, and the slowest cases, so you can find the unreliable checks and the slow ones instead of guessing. Maintenance is the single most-cited debugging pain; analytics is how the agents keep a suite trustworthy as it grows.

Get the analytics

curl http://127.0.0.1:8400/api/v1/analytics

It's computed on demand from the full run history, no separate job to schedule.

What it returns

{
  "total_runs": 42,
  "overall_pass_rate": 0.93,
  "flaky_cases": [ ... ],
  "cases": [ ... ],
  "trend": [ ... ],
  "slowest_cases": [ ... ]
}

Per-case history & flake rate

For each (suite_id, case_id), the agents track the outcome history (most recent first), the pass rate, and a flake rate:

{
  "suite_id": "checkout",
  "case_id": "pay-de",
  "title": "A German customer can pay",
  "runs": 20,
  "passed": 16,
  "failed": 4,
  "pass_rate": 0.8,
  "flaky": true,
  "flake_rate": 0.3,
  "history": ["passed", "failed", "passed", "passed", "failed", "..."],
  "avg_duration_ms": 12900
}

A case is flaky when its recent history (last 10 outcomes) is mixed, it both passed and didn't pass, or when a run explicitly marked it flaky. The flake_rate is the share of non-passing outcomes in that recent window. Cases are sorted flakiest-first, so the checks eroding your trust float to the top.

Pass-rate trend

trend is one point per run (oldest → newest): the run id, timestamp, suite, case count, and pass rate, ready to plot as a line so you can see whether the suite is getting healthier or sicker over time.

Slowest cases

slowest_cases ranks cases by average duration (top 10). Use it to find the steps worth tightening or splitting.

In the dashboard

The dashboard surfaces these as the analytics view: overall pass rate, a trend chart, a flaky-cases list, and the slowest cases, the at-a-glance health of your suite.

How to act on it

• Quarantine the flakiest cases at the top of the list; fix the underlying nondeterminism (timing, shared state, a genuinely flaky backend) rather than muting the symptom.
• Watch the trend after a refactor, a dipping pass rate is an early regression signal.
• Trim the slowest cases to keep CI fast.

Related: Runs & assertions · CI