[szibis]

What

The A/B harness now compares a proxy build with Loki like for like. No product code, no CHANGELOG entry, no release (check_changelog_pr.py: "skipped, no releasable changes").

The stack's Loki (test/e2e-compat/loki-local-config.yaml) has align_queries_with_step: true and max_cache_freshness_per_query: 0s. Loki's defaults are off and 10m (v3.7.9 queryrangebase/roundtrip.go:49, validation/limits.go:407-408). Next to that Loki a build that does not align queries answered against exact results-cache hits (3-5 ms), and a confirm pass that re-ran windows Loki had already cached printed its cache-served cold as a plain number and put it in a ratio.

Every pass now runs against one of two profiles on the one seeded stack, and every comment row, gate finding, history line and evidence record names its profile:

profile	Loki	proxy builds
aligned	the stack as it is	-align-queries-with-step=true
loki-defaults	bench/ab/docker-compose.ab-loki-defaults.yml: -querier.align-querier-with-step=false -frontend.max-cache-freshness=10m (flags win over the config file, as in the uncached overlay). The shared e2e config is not touched	their defaults

• Flag support is probed per build (stack.probe_align_support: proxy -align-queries-with-step=true -help stops at the flag for a build without it, at -help for one with it; anything else counts as unsupported). A build without the flag runs without it in aligned; meta.json and the comment say so and its rows are marked ‡: shown, never compared (no verdict, PR ÷ Loki, gate, ratchet, above-2× row, evidence). A build whose own stack flags carry the flag (once the flag branch merges every compose proxy does) has it removed in loki-defaults.
• One seed, one Loki recreate. After the first profile stack.set_profile recreates only Loki on the same volume, checks /config shows the profile's two values, and waits until Loki and VictoriaLogs count the same lines again. Builds restart per profile with their own work dir and log; each profile has its own block of windows.
• --profiles auto (CI): loki-defaults for the whole selection, plus aligned for its range-metric shapes (the only queries the flag changes) when a build supports the flag. both, aligned, loki-defaults choose otherwise.
• Above 2× in either profile. The comment, ::warning:: annotations and verdict.json give both PR ÷ Loki ratios (not run for a profile that did not run). The warn-only switch (--over-2x-loki) is unchanged. over-2x-loki.json keys on the profile; its 165 rows are migrated as aligned, proxy_side: unaligned; loki-defaults starts empty (seed it from its first daily run, with your approval).
• Locked rows (the registry perf: bounds of ci: lock the performance acceptance tiers in the registry and the regression gates #893) are judged against their bound in each profile, as are the band distribution, its ratchet, history/acceptance.json (now days[].profiles.<profile>), the daily results (daily-<set>.json aligned, daily-<set>.loki-defaults.json), the history files (<set>.jsonl aligned and every older line, <set>.loki-defaults.jsonl), perf_evidence.py (prefers loki-defaults of the same date) and the baseline-run candidate files (ci: a CI-only pull request runs only the jobs that verify what it changes; a big change runs a full baseline #895).
• Cold. A Loki cold enters a ratio only when Loki computed it. A cache-served cold is † and in no ratio (comment, trend, evidence). The confirm pass keeps the first pass's cold columns, with their marks, whenever its own Loki cold was cache-served (cold_from_first_pass, ¹), for every re-measured row, not only the above-2× ones.
• Refresh pass on top of its block (perf_matrix.plan_windows): the pass that models a dashboard refresh at now is the nearest to now. Before, it sat below the first pass, inside the extents the first pass had put in Loki's cache, so Loki's "refresh" was mostly a cache hit by construction (refresh is off in CI and has no row in the baseline list). Raw rows carry the request time (at); meta.json records per profile the youngest and oldest age of a requested window (window_age_s).
• Second profile = a recreated Loki (empty caches, chunks read from the store, not the ingesters): its colds are slower. So the profiles run loki-defaults first (the Loki the seed wrote to, as the history always had), and aligned second; its lines carry loki_restarted and take no part in the cold ratchet. Every range-metric shape aligned runs also runs, cold included, in loki-defaults.

Validation (isolated stack, DOCKER_CONTEXT=colima-lvp, project ablike, ports 24101/24428/24190)

pr_smoke.py --base origin/main --head origin/main --ranges 1h,6h --refresh-s 60 --profiles both, selection: the 8 control smoke shapes + unwrap-conversion-error. 483 s. meta.json: align_flag_supported {base: false, pr: false}, so the whole aligned profile is ‡ / "not like-for-like" (32 rows n/a), the 32 loki-defaults rows are same, 0 new differences, 24 pre-existing (as on the base), 0 new above-2× rows. /config read back: aligned true/0s, loki-defaults false/10m.

Loki p50 (ms), refresh pass, control set 1h (the first block, windows 2-3 minutes old when requested, loki-defaults window ages 165-1719 s):

shape	Loki refresh aligned	Loki refresh loki-defaults	proxy refresh loki-defaults
json volume no filters	265	126	28
stream-label count by app	231	139	7
topk rate by app	285	253	27
drilldown service landing	237	165	6
json rate by level	817	114	2

The exact-hit 3-5 ms Loki refresh figures are gone in both profiles (they came from the refresh windows sitting inside the first pass's cached extents). Lower blocks of a selection are traversed after their windows are older than 10 minutes, where Loki's freshness horizon no longer applies (unwrap set: 17-138 ms in loki-defaults); the README says so under "Like for like".

Cache-served cold, a real run (before the profile order was flipped, same confirm code): the confirm pass of E2 json duration_ms p95 measured Loki's cold at 3 ms, counters cold_cache_served: [loki]. Before: printed as "Loki 3 ms", PR ÷ Loki cold 32× / 150×. Now the row keeps the first pass's computed cold (128 ms, 208 ms) with ¹, PR ÷ Loki 0.77× / 2.17×. † itself (a first-pass cache-served cold) is covered by unit tests only; no real run produced one.

A build that supports the flag, same stack (--head perf/align-queries-with-step, 12322ed3): proxy_align: {base: unsupported, pr: on}, not_like_for_like: [base], PR vs Loki compared, the base marked ‡.

Daily, real run with writes (daily.py --sets control --max-range 1h --runs 3 --profiles both, then the generated files reverted): daily-control.json, daily-control.loki-defaults.json, control.jsonl + control.loki-defaults.jsonl lines with profile, like_for_like; trend sections per profile; alerts ran; 2.0 min.

CI minutes (local: a laptop with a warm Go build cache; control smoke + unwrap-conversion-error, 1h, 4 runs, no refresh)

run	total	Loki recreate	matrix
before (one profile, the stack's Loki)	88 s	none	26 s
after, no build with the flag (today): loki-defaults only	95 s	none	35 s
after, a build with the flag: loki-defaults + aligned subset	190 s	44 s	35 + 23 s

Today: no extra cost (the single pass is loki-defaults; the probe is a 0.1 s -help). Once a build supports the flag: about +70 s here (recreate 44 s with the count check, 3 s of proxy starts, the range-metric subset's matrix); the 190 s also holds the build of the older branch used as the flag-supporting build. On a 17-minute hosted-runner run: about +10-20% (not measured on a runner). The seed does not grow. timeout-minutes: 30 has room. Daily: the second pass roughly adds the range-metric part of the matrix and one recreate with a 24h count check; its current duration was not available, compare the first two-profile run with timeout-minutes: 240 (until a build supports the flag, auto skips aligned and it costs nothing extra).

Tests

Python unittest in bench/ab/tests (wired in ci.yaml by discover), 340 tests, all green; check_conformance.py ok; selection.py --check ok; actionlint, shellcheck on validate_compose.sh. They cover: the overlay against the base command (drift), /config parsing by section, set_profile, the flag probe against fake binaries, align_args for every spelling of the flag, start_proxies states, profile planning and window blocks, measure and daily.main with both profiles (order, restarts, unlike lists, files), the confirm-cold rule, the profile-keyed gate, baseline migration, either-profile merge with both ratios, locked rows per profile, history files/ratchet/over-2x/distribution/acceptance per profile, loki_restarted, like_for_like, report and comment marks, the baseline-candidate files. Deviation to decide: the owner rule says new tests are Go; these test Python harness code, I found no Go test that runs the Python harness, so they follow the existing pattern (#876, #889, #893).

Known limits

• Freshness bites on windows near now only; later blocks and the daily run's single end are older than the horizon when requested. Documented, with the recorded window ages.
• Until a build supports -align-queries-with-step the whole aligned profile is not run in auto (and aligned history lines stop; last_commit takes the newest line of either profile).
• The leftover ab-tree-* directories in $TMPDIR dated Oct 25/26 are not from this change.

[github-actions]

⚪ Performance A/B: no change beyond noise

8 shape×range — 8 ⚪ same · results vs Loki: 7 same, 0 new difference(s), 1 pre-existing

Loki profiles compared (like for like):

• aligned: not run (no build supports -align-queries-with-step, so every aligned row would be not like-for-like)
• loki-defaults: 8 shape×range; proxy flag base off, pr off

Every shape is within noise of the base build (±30% and 100 ms).

8 unchanged shape×range (within noise)

	shape	range	Loki profile	base p50	PR p50	change	Loki p50	PR ÷ Loki	result vs Loki
⚪	drilldown service landing	1h	defaults	14 ms	19 ms	⚪ +36%	188 ms	0.10×	✅ same
⚪	instant ungrouped rate	instant	defaults	7 ms	7 ms	⚪ +0%	90 ms	0.08×	✅ same
⚪	json rate by level (no filter, no drop)	1h	defaults	400	400	⚪ +0%	400	—	✅ same
⚪	json volume no filters	1h	defaults	89 ms	96 ms	⚪ +8%	393 ms	0.24×	✅ same
⚪	logs plain	1h	defaults	118 ms	134 ms	⚪ +14%	1.14 s	0.12×	✅ same
⚪	logs | json	1h	defaults	104 ms	110 ms	⚪ +6%	1.90 s	0.06×	✅ same
⚪	stream-label count by app	1h	defaults	16 ms	10 ms	⚪ -38%	132 ms	0.08×	✅ same
⚪	topk rate by app	1h	defaults	50 ms	45 ms	⚪ -10%	204 ms	0.22×	➖ differs (as base)

Cold (first-run) timings — what a user opening a panel sees

shape	range	Loki profile	base cold	PR cold	Loki cold	PR ÷ Loki cold
drilldown service landing	1h	defaults	44 ms	39 ms	620 ms	0.06×
instant ungrouped rate	instant	defaults	7 ms	12 ms	80 ms	0.15×
json rate by level (no filter, no drop)	1h	defaults	400	400	400	—
json volume no filters	1h	defaults	268 ms	276 ms	1.66 s	0.17×
logs plain	1h	defaults	166 ms	123 ms	1.24 s	0.10×
logs | json	1h	defaults	212 ms	207 ms	1.67 s	0.12×
stream-label count by app	1h	defaults	33 ms	33 ms	1.22 s	0.03×
topk rate by app	1h	defaults	48 ms	46 ms	655 ms	0.07×

Why these shapes

• control smoke → 8: control/drilldown service landing, control/instant ungrouped rate, control/json rate by level (no filter, no drop), control/json volume no filters, control/logs plain, control/logs | json, control/stream-label count by app, control/topk rate by app

_{base 70931e0748 → PR d22e942077 · 4 runs (first is cold), warm p50 · ranges 1h · noise ±30% and 100 ms · VictoriaLogs restarts 0→0 · 1.8 min · phases stack_up 37s, seed_and_build 27s, loki-defaults:loki_profile 0s, loki-defaults:proxies 2s, loki-defaults:matrix 27s. Same runner, one fresh stack, targets interleaved per request. How to read and reproduce: bench/ab/README.md.}

[github-actions]

✅ Visual smoke: passed with warnings

Base 70931e0748 vs PR 39d888dec5, Grafana through the proxy builds and Loki on one fresh stack; run.
Core set: 5 capture(s) at 1h. Detailed set: 0 entries (nothing visual touched). 5 capture(s), 15m/1h/6h, 24h and 7d stay out of PR runs.

page	range	set	base = PR (data)	vs Loki (PR)	pixel diff	result
dd-fields	1h	core	✅ identical	identical (4 explained, 1 history-dependent)	0.12%	⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-landing	1h	core	✅ identical	identical (3 explained)	2.15%	✅
dd-service-logs	1h	core	✅ identical	identical (3 explained, 1 history-dependent)	0.12%	⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
explore-logs-plain	1h	core	✅ identical	identical (1 explained)	0.07%	✅
explore-metric-plain	1h	core	✅ identical	identical (1 explained)	0.04%	✅

Gate: a base-vs-PR data difference passes when the PR is closer to Loki, and fails when the PR diverges from Loki or Loki cannot decide it (at a range Loki does not hold, a difference passes when the same page proved it closer to Loki at a shorter range; otherwise label the pull request visual-change-expected to accept an intended change). Also failing: a panel empty on the PR but not on the base, an error banner, panel error, error answer or new "No data" panel on the PR, a PR side that never settled, a difference that did not reproduce on the recapture. Pixel differences only warn.

dd-fields-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-28T09%3A37%3

• history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-28T09%3A37%3A00.000Z&query=%7Bservic: body: content differs (rows 6 vs 2)
• vs Loki, explained: query ('sum by (service_name_extracted) (count_over_time({service_name="otel-collector"} | servi: request only on one side (right) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision)
• vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-28T09%3A37%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
• vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-28T09%3A37%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
• vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-service-logs-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-28T09%3A37%3

• history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-28T09%3A37%3A00.000Z&query=%7Bservic: body: content differs (rows 6 vs 2)
• vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-28T09%3A37%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
• vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-28T09%3A37%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
• vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

Core set montages (3): base | PR | Loki

dd-landing-1h

explore-logs-plain-1h

explore-metric-plain-1h

All montages of this run: pr-visuals/pr-896.

[github-actions]

PR Quality Report

Compared against base branch main.

Coverage and tests

Signal	Base	PR	Delta
Test count	15086	15086	0
Coverage	88.6%	88.6%	0.0% (stable)

Compatibility

Track	Base	PR	Delta
Loki API	100.0%	11/11 (100.0%)	0.0% (stable)
Logs Drilldown	100.0%	17/17 (100.0%)	0.0% (stable)
VictoriaLogs	100.0%	11/11 (100.0%)	0.0% (stable)

Performance smoke

Performance smoke was skipped for this PR because no perf-sensitive paths changed.

State

• Coverage, compatibility, and sampled performance are reported here from the same PR workflow.
• This is a delta report, not a release gate by itself. Required checks still decide merge safety.
• Performance smoke was intentionally skipped because no perf-sensitive paths changed in this PR.
• Delta states use the same noise guards as the quality gate (percent + absolute + low-baseline checks), so report labels match merge-gate behavior.