[szibis]

Summary

Follow-up to the daily regression alerts (#796/#798), from a fresh review. Without it, the first daily run could turn red on a Loki timeout: Loki times out at 180 s on two unwrap-sum-without rows today, and that showed up as a proxy regression and a ratchet loss.

Changes (bench/ab/history.py, bench/ab/daily.py, .github/workflows/perf-daily.yaml, README, CHANGELOG)

1. Loki failures:
   • a result transition, or a ratchet parity loss, counts only when Loki answered 200 on the lines compared;
   • otherwise the row goes to an informational Loki failed list, which never fails the run;
   • rows where the proxy errors exactly as Loki does are not listed.
2. Full measurements only:
   • only full daily measurements are compared (all ranges up to 24h, 7 runs), using the same test as the skip gate;
   • each row is compared with the newest earlier full line that has it;
   • a same-day manual line is replaced by the scheduled full one, and never the reverse.
3. "Loki fails the same way" exemption: now tested for both the status rule and the ratchet.
4. PR body: capped at 50 entries, with the full list in the job summary and the artifact. Fail on regressions runs unless the run is cancelled, so it still reports if the PR step fails.
5. ratchet.json: a malformed file disables the ratchet for the run instead of crashing, and invalid entries are skipped. The alerts step can't stop the day's history and evidence from being delivered.
6. Carry step: only carries *.jsonl, so rows you remove by hand from ratchet.json on main stay removed.
7. Commit ranges: a repeated ratchet loss names the range where the row was last good.
8. Repeat alerts: a row already slower than the release no longer re-alerts on day-to-day jitter.

Smaller fixes:

• a 400 → 200 change while Loki still returns 400 is not an improvement;
• a 200 → missing status is a regression;
• the count of ratchet rows not measured this run is shown;
• the CLI only considers sets in shapes.json.

Proofs

• bench/ab/tests: 142 pass (was 124); 19 new alert tests. Mutations that now fail tests:
  • removing the status exemption;
  • fails_like_loki always false;
  • removing the ratchet exemption;
  • removing the Loki-200 check on parity transitions;
  • removing the Loki-200 check on ratchet parity losses;
  • removing the verdict guard.
• Committed history: alerts gives 0 regressions, 0 improvements, 0 ratchet losses, and 2 informational Loki-failed rows.
• trend.md, performance.md, gaps.md: regenerate with no diff.
• scripts/ci/tests: 77 pass.
• Linting and gates: actionlint (pinned and local) and zizmor 1.30.1 (pinned) report no findings; conformance gate ok.

Not applicable: no proxy code changes, so e2e vs Loki, the Grafana range matrix, visual proof, A/B, benchstat and the load test do not apply.

[github-actions]

⚪ Performance A/B: no change beyond noise

8 shape×range — 8 ⚪ same · results vs Loki: 7 same, 0 new difference(s), 1 pre-existing

Every shape is within noise of the base build (±30% and 100 ms).

8 unchanged shape×range (within noise)

	shape	range	base p50	PR p50	change	Loki p50	PR ÷ Loki	result vs Loki
⚪	drilldown service landing	1h	21 ms	18 ms	⚪ -14%	19 ms	0.95×	✅ same
⚪	instant ungrouped rate	instant	5 ms	6 ms	⚪ +20%	49 ms	0.12×	✅ same
⚪	json rate by level (no filter, no drop)	1h	400	400	⚪ +0%	400	—	✅ same
⚪	json volume no filters	1h	117 ms	118 ms	⚪ +1%	36 ms	3.28×	✅ same
⚪	logs plain	1h	51 ms	42 ms	⚪ -18%	334 ms	0.13×	✅ same
⚪	logs | json	1h	66 ms	62 ms	⚪ -6%	47 ms	1.32×	✅ same
⚪	stream-label count by app	1h	17 ms	16 ms	⚪ -6%	27 ms	0.59×	✅ same
⚪	topk rate by app	1h	27 ms	30 ms	⚪ +11%	70 ms	0.43×	➖ differs (as base)

Cold (first-run) timings — what a user opening a panel sees

shape	range	base cold	PR cold	Loki cold	PR ÷ Loki cold
drilldown service landing	1h	25 ms	24 ms	445 ms	0.05×
instant ungrouped rate	instant	5 ms	5 ms	50 ms	0.10×
json rate by level (no filter, no drop)	1h	400	400	400	—
json volume no filters	1h	110 ms	108 ms	609 ms	0.18×
logs plain	1h	103 ms	41 ms	311 ms	0.13×
logs | json	1h	104 ms	99 ms	423 ms	0.23×
stream-label count by app	1h	14 ms	15 ms	446 ms	0.03×
topk rate by app	1h	32 ms	26 ms	492 ms	0.05×

Why these shapes

• control smoke → 8: control/drilldown service landing, control/instant ungrouped rate, control/json rate by level (no filter, no drop), control/json volume no filters, control/logs plain, control/logs | json, control/stream-label count by app, control/topk rate by app

_{base e0c5a68219 → PR f8f741b01e · 4 runs (first is cold), warm p50 · ranges 1h · noise ±30% and 100 ms · VictoriaLogs restarts 0→0 · 1.8 min. Same runner, one fresh stack, targets interleaved per request. How to read and reproduce: bench/ab/README.md.}

[github-actions]

✅ Visual smoke: passed

Base e0c5a68219 vs PR 26009dadfd, Grafana through the proxy builds and Loki on one fresh stack; run.
Core set: 5 capture(s) at 1h. Detailed set: 0 entries (nothing visual touched). 5 capture(s), 15m/1h/6h, 24h and 7d stay out of PR runs.

page	range	set	base = PR (data)	vs Loki (PR)	pixel diff	result
dd-fields	1h	core	✅ identical	identical (4 explained, 1 history-dependent)	0.11%	✅
dd-landing	1h	core	✅ identical	identical (3 explained)	0.14%	✅
dd-service-logs	1h	core	✅ identical	identical (3 explained, 1 history-dependent)	0.11%	✅
explore-logs-plain	1h	core	✅ identical	1 difference(s), 1 on base too	0.07%	✅
explore-metric-plain	1h	core	✅ identical	1 difference(s), 1 on base too	0.04%	✅

Gate: a base-vs-PR data difference passes when the PR is closer to Loki, and fails when the PR diverges from Loki or Loki cannot decide it (at a range Loki does not hold, a difference passes when the same page proved it closer to Loki at a shorter range; otherwise label the pull request visual-change-expected to accept an intended change). Also failing: a panel empty on the PR but not on the base, an error banner, panel error, error answer or new "No data" panel on the PR, a PR side that never settled, a difference that did not reproduce on the recapture. Pixel differences only warn.

Core set montages (5): base | PR | Loki

dd-fields-1h

dd-landing-1h

dd-service-logs-1h

explore-logs-plain-1h

explore-metric-plain-1h

All montages of this run: pr-visuals/pr-800.

[github-actions]

PR Quality Report

Compared against base branch main.

Coverage and tests

Signal	Base	PR	Delta
Test count	11826	11826	0
Coverage	88.3%	88.3%	0.0% (stable)

Compatibility

Track	Base	PR	Delta
Loki API	100.0%	11/11 (100.0%)	0.0% (stable)
Logs Drilldown	100.0%	17/17 (100.0%)	0.0% (stable)
VictoriaLogs	100.0%	11/11 (100.0%)	0.0% (stable)

Performance smoke

Performance smoke was skipped for this PR because no perf-sensitive paths changed.

State

• Coverage, compatibility, and sampled performance are reported here from the same PR workflow.
• This is a delta report, not a release gate by itself. Required checks still decide merge safety.
• Performance smoke was intentionally skipped because no perf-sensitive paths changed in this PR.
• Delta states use the same noise guards as the quality gate (percent + absolute + low-baseline checks), so report labels match merge-gate behavior.