[szibis]

ci: a CI-only pull request runs only the jobs that verify what it changes, and a very large change re-measures the baseline once

Four recent CI-only pull requests ran the whole product CI (about 200 runner-minutes each) because change scope knew two answers, product and infra, and a path with no rule counted as product:

PR	what it changed	before	now
#888	test/e2e-compat/loki-*-config.yaml	product	stack lane
#889	bench/ab/*.py, its tests, fixtures, README	product	perf lane
#891	test/e2e-ui/README.md (plus docs, workflow)	product	infra
#892	scripts/ci/collect_quality_metrics.sh	product (P3)	product (P3, unchanged by design)
#890	workflow, scripts/ci	infra	infra

What changes

scripts/ci/change_scope.py sorts every path into one of four nested lanes. A pull request runs the jobs of the highest lane any path needs. lint, helm, Security / static and CodeQL run in every lane.

Lane	Paths	Adds
infra	.github/, scripts/, docs/, website/, examples/, Markdown at the root and under bench/ and test/, the two benchmark Loki configs, the compose files no job starts	nothing
perf	bench/ab/** except stack.py, selection.py, shapes.json and the compose overlays (read by bench/visual, bench/parity and the conformance gate: rule P5, product)	perf-smoke
stack	the compose files CI starts (e2e-compat, e2e-fleet, e2e-pipeline) and the configs the e2e-compat stack mounts (loki-local-config.yaml, grafana-*.yaml, vm*.yml, nginx-tail.conf)	every job that starts a stack: image, tuple-smoke, e2e-compat, e2e-fleet, e2e-ui, e2e-pipeline, loki/vl/drilldown pinned, Security / runtime, visual-smoke, perf-smoke
product	everything else, unknown paths, an empty list, any classifier error, the ci-full label, non-PR events, P1 to P4	everything

change-scope.yaml reports three cumulative outputs, product, stack and perf (product implies stack implies perf). A job is skipped only on an explicit false from a classifier that ran; anything else keeps every lane on. Skipped jobs report as skipped, which the required checks accept, exactly as the infra lane already did.

Wiring (ci.yaml): image gates on stack. tuple-smoke, e2e-compat-group, e2e-fleet and e2e-ui keep following test, but also run when test was skipped because only a stack path changed (needs.test.result == 'skipped' with product == 'false' and stack == 'true', compared with == so an empty output cannot take that route). The e2e-compat aggregator accepts skipped groups only when both lanes say false. The other workflows gate their stack jobs on stack and perf-smoke on perf; report and race-stress stay on product.

Two checks moved so that the lanes that skip the Go unit tests still run them, with no change for product pull requests: the bench module's vet and tests (7 s; they hold the drift guard between loki-local-config.yaml and the two benchmark configs) and the bench/ab unit tests (2 s) now run in lint instead of test and conformance.

Markdown is infra only where nothing reads it as data: not under conformance/ (the generated reports are compared by the conformance gate), charts/ (the chart README ships and the flag documentation gate reads it), cmd/, internal/, pkg/, dashboard/ or alerting/.

Compose files that a job starts moved from infra to the stack lane, so a pull request that edits only test/e2e-compat/docker-compose.yml, test/e2e-fleet/docker-compose.yml or test/e2e-pipeline/docker-compose.yml now runs the stack jobs it used to skip (about 120 runner-minutes more for such a pull request). That is deliberate: nothing in pull request CI checked a compose edit against the stack before, and e2e-compat core includes TestPinnedCompatibilityMatrixMatchesCompose, which compares the compose file with the pinned compatibility matrix. The three compose files no job starts (docker-compose.bench.yml, docker-compose.review.yml, docker-compose.http-discovery.yml) stay infra.

Proof that no product pull request loses a job

• LaneRepositoryTests: every tracked path with ships() true, every tracked path under cmd/, internal/, pkg/, charts/, dashboard/, alerting/ and go.mod, go.sum, Dockerfile, .dockerignore, every tracked path under conformance/ except the three generated reports already infra, and every file of bench/visual, bench/parity, bench/cmd and bench/internal classify as the full product lane.
• Replay of the last 40 merged pull requests (file lists from the API, paginated): 18 touch Go, chart, image or module paths; all 18 are product before and after, and none loses a job. Three pull requests change lane (ci: raise the gRPC message limit of the CI Loki stacks to 64 MiB #888, ci: the A/B matrix stops sleeping after every request and asks Loki once about a shape it times out on #889, ci: build the proxy image once for the e2e jobs and merge the e2e-ui shards from 9 to 5 #891); the other 37 keep their lane.
• Old against new classifier over all 1905 tracked paths: 39 change lane, 36 of them to a smaller lane (19 bench/ab files to perf, 8 stack configs to stack, 7 Markdown files and 2 benchmark configs to infra) and 3 to a larger one (the three CI compose files, infra to stack, deliberate).
• test_scope_wiring.py checks every gate of every workflow against its lane and has mutation cases for a stack job that runs after a failed test, for the skipped-test route on an empty output, a missing scope in needs, an aggregator that ignores stack, and an image gated on the wrong lane.

Baseline run for very large changes (second commit)

A very large change to the proxy now re-measures everything once, so later pull requests are compared with a fresh baseline. It is never run on every pull request: an ordinary product pull request keeps today's selection-based perf-smoke and visual-smoke.

A baseline run replaces the selection with:

• every A/B set of bench/ab/shapes.json (240 shapes in 25 sets) at the smoke's ranges (selection.py --all);
• every visual-smoke page (99 pages, 201 captures; plan.py --all);
• the whole VictoriaLogs version matrix (vl-matrix, the five supported versions), run on the pull request instead of only the weekly schedule.

Trigger (scripts/ci/change_scope.py, constants BASELINE_*, only for a pull request whose base is main):

• the baseline label, which also forces the full product CI; or
• the change is in the product lane on its own paths and its diff in shipped paths (ships(): tests and testdata do not count) has more than 1,500 changed lines, or more than 40 files, or at least 150 changed lines in the files every query passes through: internal/proxy/{proxy.go, query_translation.go, request_scope.go, cache_keys.go, middleware.go} and internal/translator/translator.go (route table, per-route wrapper and translation entry, tenant and routing scope, canonical cache keys, tenant validation and security headers, LogQL to LogsQL translation). The entry-point rule has a size because counting any edit would select 44% of the pull requests that change shipped code.

Both the baseline and the ci-full labels force it (with the full product CI). A label never forces it on a stacked pull request: only a pull request into main can get it. The baseline label exists on the repository.

Stacked pull requests never run it. ci.yaml, compat-vl.yaml, perf-ab.yaml and visual-smoke.yaml all trigger on pull_request: branches: [main], so a pull request based on another branch starts none of them; the classifier also refuses any base other than main, label or not (tests: test_heavy_workflows_only_run_for_pull_requests_into_main, test_a_stacked_pull_request_never_gets_it). Each layer is judged on its own diff once it targets main (retargeting fires edited, which these workflows do not listen for, so the run starts with the next push or a re-run). A big change split into layers that are each under the thresholds does not trigger; several big layers trigger once each. Put the baseline label on the last layer of a split refactor. The baseline output fails the other way from the lanes: an error, an unreadable label list, an empty diff or a non-PR event all report false.

Results are never committed:

• perf: pr_smoke.py --baseline-candidate DIR writes over-2x-loki.json (the committed list with the set x range pairs this run measured replaced by this run's rows, the 24h and 7d rows stay), acceptance-day.json (one day in the shape of an entry of history/acceptance.json: share below Loki, close, 1-2x, above 2x per mode), candidate.md and meta.json, uploaded as the perf-baseline-candidate artifact. The sticky A/B comment ends with a "New baseline candidate" section: the distribution next to the committed day's, and the above-2x rows added and no longer above.
• visual: the montages and comparison are uploaded as visual-baseline-candidate; the visual comment ends with a fixed "New baseline candidate" section pointing at it (rendered by the base branch's trusted job, no pull request data).
• VL matrix: pass or fail per version in the checks.
• What accepting means, file by file. over-2x-loki.json is a proposed replacement: a person copies it over bench/ab/history/over-2x-loki.json in a pull request (check_conformance.py checks that list against the registry locks; the daily run keeps it from there). acceptance-day.json is a report and is not copied: history/acceptance.json is rebuilt from history/*.jsonl by every daily run, so the real new distribution is the next daily run's, which runs whenever a measured path changed on main; the run on the pull request is its preview. The visual captures are a report to read. The jobs hold no write token for any of it, and a test checks that no job that reads the baseline output can commit, push or open a pull request.
• The profile differs from the daily run's: the pull request run uses the smoke's (4 runs, 1h, the smoke's line rate), the daily run 7 runs up to 24h at a third of the line rate. The numbers show the direction, not row for row the committed ones; aligning the two would cost about the daily's 115 to 155 minutes and is left as a decision.

Trigger rate (replay of the merged pull requests #678 to #894, file lists and line counts from the API; "changed shipped code" leaves out the release-metadata, dependency and daily-report bots):

window	pull requests	product lane	changed shipped code	baseline
last 40 (#852 to #894)	40	28	6	1 (#862) = 17% of 6, 3.6% of the product lane
last 200 (#678 to #894)	200	148	45	6 = 13% of 45, 4.1% of the product lane

Read against the pull requests that changed shipped code. Against the whole product lane it is 4.1%: product-lane pull requests that ship nothing (gates, registry, bench, conformance) can never trigger by size, so that denominator cannot reach 10 to 20% without firing on most real code pull requests.

The six: #862 (1,739 lines, 222 in the entry points), #846 (299 entry-point lines), #760 (1,710 lines), #747 (333 entry-point lines), #739 (282 entry-point lines), #704 (1,565 lines). Near misses: #758 (1,090 lines), #746 (1,364 lines, 10 files). Lines and files alone (the owner's proposal) select 3 of the 45 (7%); the entry-point rule brings it to 13%. None of the 200 would have run it on a non-main base or outside the product lane.

Cost of a baseline run, from the last full runs (perf-smoke 171 shape x range rows in 14.5 min plus 3 min of setup; visual-smoke 37 captures in 13.7 min including about 5 min of setup; vl-matrix 5 x 6.3 min):

job	smoke today	baseline run	extra runner-minutes
perf-smoke	about 17	about 35 (240 shapes, about 5 s per row)	+18
visual-smoke	about 14	about 52 (201 captures at about 14 s)	+38
vl-matrix	0 (weekly only)	about 31 (5 versions in parallel)	+31
total			about +87, wall-clock about 55 min (the visual job)

So a baseline pull request costs about 300 runner-minutes instead of about 210. These are estimates from the logs above; the first baseline run gives the measured number. Job timeouts are 90 minutes for perf and visual.

Setup before use: create the baseline label on GitHub (this change does not create it). Labels are read when the workflows run, so add it and then push or re-run the checks.

Estimated runner-minutes saved per pull request

From the job durations of one green full run (about 209 min with perf-smoke, 193 min when perf-smoke selects nothing; always-on jobs about 8 min):

Pull request	before	now	saved
Markdown or README only under bench/ or test/	193	8	about 185
benchmark Loki configs only	209	8	about 200
bench/ab Python change	209	25	about 184
bench/ab tests, fixtures or README only	193	9	about 184
stack config (loki-local-config.yaml, grafana-*.yaml, vm*.yml, nginx-tail.conf)	209	128	about 81
CI compose file only (was infra)	8	128	about 120 more
Go, chart, image, gate, classifier	unchanged		0

The e2e-ui time comes from the nine-shard layout that preceded #891, so the stack figure is slightly high on both sides.

Not changed

P1 to P4 keep their meaning; an unknown path is still product; editing the classifier or its workflow still runs everything. #892-type changes (P3) still run the full CI.

[github-actions]

⚪ Performance A/B: no change beyond noise

8 shape×range — 8 ⚪ same · results vs Loki: 7 same, 0 new difference(s), 1 pre-existing

Every shape is within noise of the base build (±30% and 100 ms).

8 unchanged shape×range (within noise)

	shape	range	base p50	PR p50	change	Loki p50	PR ÷ Loki	result vs Loki
⚪	drilldown service landing	1h	13 ms	14 ms	⚪ +8%	26 ms	0.54×	✅ same
⚪	instant ungrouped rate	instant	5 ms	5 ms	⚪ +0%	76 ms	0.07×	✅ same
⚪	json rate by level (no filter, no drop)	1h	400	400	⚪ -33%	400	—	✅ same
⚪	json volume no filters	1h	91 ms	89 ms	⚪ -2%	142 ms	0.63×	✅ same
⚪	logs plain	1h	154 ms	156 ms	⚪ +1%	699 ms	0.22×	✅ same
⚪	logs | json	1h	142 ms	148 ms	⚪ +4%	956 ms	0.15×	✅ same
⚪	stream-label count by app	1h	16 ms	14 ms	⚪ -12%	135 ms	0.10×	✅ same
⚪	topk rate by app	1h	50 ms	46 ms	⚪ -8%	228 ms	0.20×	➖ differs (as base)

Cold (first-run) timings — what a user opening a panel sees

shape	range	base cold	PR cold	Loki cold	PR ÷ Loki cold
drilldown service landing	1h	40 ms	40 ms	1.22 s	0.03×
instant ungrouped rate	instant	6 ms	7 ms	78 ms	0.09×
json rate by level (no filter, no drop)	1h	400	400	400	—
json volume no filters	1h	229 ms	218 ms	1.79 s	0.12×
logs plain	1h	165 ms	167 ms	681 ms	0.25×
logs | json	1h	412 ms	317 ms	969 ms	0.33×
stream-label count by app	1h	22 ms	23 ms	1.27 s	0.02×
topk rate by app	1h	50 ms	48 ms	1.61 s	0.03×

Why these shapes

• control smoke → 8: control/drilldown service landing, control/instant ungrouped rate, control/json rate by level (no filter, no drop), control/json volume no filters, control/logs plain, control/logs | json, control/stream-label count by app, control/topk rate by app

_{base 4764b1d211 → PR 1ffef6c425 · 4 runs (first is cold), warm p50 · ranges 1h · noise ±30% and 100 ms · VictoriaLogs restarts 0→0 · 1.7 min. Same runner, one fresh stack, targets interleaved per request. How to read and reproduce: bench/ab/README.md.}

[github-actions]

✅ Visual smoke: passed with warnings

Base 4764b1d211 vs PR 80d706852e, Grafana through the proxy builds and Loki on one fresh stack; run.
Loaded again, once and more patiently, after a first-pass difference or an unsettled page: dd-landing 1h.
Core set: 5 capture(s) at 1h. Detailed set: 0 entries (nothing visual touched). 5 capture(s), 15m/1h/6h, 24h and 7d stay out of PR runs.

page	range	set	base = PR (data)	vs Loki (PR)	pixel diff	result
dd-fields	1h	core	✅ identical	identical (4 explained, 1 history-dependent)	0.12%	⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-landing	1h	core	differs (1): resource /api/datasources/uid/*/resources/detected_labels?en	identical (3 explained)	0.14%	✅ improved (closer to Loki)
dd-service-logs	1h	core	✅ identical	identical (3 explained, 1 history-dependent)	0.12%	⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
explore-logs-plain	1h	core	✅ identical	identical (1 explained)	0.07%	✅
explore-metric-plain	1h	core	✅ identical	identical (1 explained)	0.04%	✅

Gate: a base-vs-PR data difference passes when the PR is closer to Loki, and fails when the PR diverges from Loki or Loki cannot decide it (at a range Loki does not hold, a difference passes when the same page proved it closer to Loki at a shorter range; otherwise label the pull request visual-change-expected to accept an intended change). Also failing: a panel empty on the PR but not on the base, an error banner, panel error, error answer or new "No data" panel on the PR, a PR side that never settled, a difference that did not reproduce on the recapture. Pixel differences only warn.

dd-fields-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-28T09%3A21%3

• history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-28T09%3A21%3A00.000Z&query=%7Bservic: body: content differs (rows 6 vs 2)
• vs Loki, explained: query ('sum by (service_name_extracted) (count_over_time({service_name="otel-collector"} | servi: request only on one side (right) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision)
• vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-28T09%3A21%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
• vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-28T09%3A21%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
• vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-service-logs-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-28T09%3A21%3

• history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-28T09%3A21%3A00.000Z&query=%7Bservic: body: content differs (rows 6 vs 2)
• vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-28T09%3A21%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
• vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-28T09%3A21%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
• vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

Core set montages (3): base | PR | Loki

dd-landing-1h

explore-logs-plain-1h

explore-metric-plain-1h

All montages of this run: pr-visuals/pr-895.

[github-actions]

PR Quality Report

Compared against base branch main.

Coverage and tests

Signal	Base	PR	Delta
Test count	15086	15086	0
Coverage	88.6%	88.6%	0.0% (stable)

Compatibility

Track	Base	PR	Delta
Loki API	100.0%	11/11 (100.0%)	0.0% (stable)
Logs Drilldown	100.0%	17/17 (100.0%)	0.0% (stable)
VictoriaLogs	100.0%	11/11 (100.0%)	0.0% (stable)

Performance smoke

Performance smoke was skipped for this PR because no perf-sensitive paths changed.

State

• Coverage, compatibility, and sampled performance are reported here from the same PR workflow.
• This is a delta report, not a release gate by itself. Required checks still decide merge safety.
• Performance smoke was intentionally skipped because no perf-sensitive paths changed in this PR.
• Delta states use the same noise guards as the quality gate (percent + absolute + low-baseline checks), so report labels match merge-gate behavior.