Skip to content

DO NOT MERGE: product-path check for the CI scope gating (#822, #823) - #824

Closed
szibis wants to merge 4 commits into
mainfrom
ci/scope-product-probe
Closed

szibis wants to merge 4 commits into
mainfrom
ci/scope-product-probe

Conversation

@szibis

@szibis szibis commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

Throwaway draft. It carries the commits of #822 and #823 plus a no-op comment under internal/, so the change scope is product and every product job must run with the new gating. A second push adds a deliberately failing unit test to show that test and e2e-compat go red and block the merge. This pull request is closed without merging.

Add a scope job to ci.yaml that calls the reusable change-scope workflow, and
gate test, bench and conformance on its output. The jobs that depend on test
keep the implicit success() and skip together with it. The gate is
`!cancelled() && needs.scope.outputs.product != 'false'`, so a scope job that
fails or returns nothing runs every product job; only a scope job that
succeeds with false skips them. Pushes and manual runs always
report true.

The two aggregator jobs, e2e-compat and race-stress, run under always(). They
pass when every group succeeded, or when the groups were skipped and the scope
output is false. A skipped group on a product pull request (for example when
test failed) or an empty scope output now fails them, so a skip never reads as
a pass by accident.

lint and helm have no condition and run on every pull request. lint now also
carries the infra checks that used to sit in test: the observability asset
sync check, the scripts/ci unit tests (plus the classifier and scope wiring
tests) and the configuration reference check. Two new steps join them: the
README support-policy test and scripts/ci/validate_compose.sh, which runs
docker compose config -q on every tracked compose file (standalone files and
each overlay on its base) and fails when a compose file is missing from its
list.

test_scope_wiring.py reads the workflows and asserts that lint and helm are
never skipped, that every job reading the scope output lists scope in needs
and uses the canonical gate, that jobs downstream of test cannot bypass
success(), and that both aggregators test the scope output.
…quests

Add the reusable change-scope job to the security, quality report, compatibility,
pipeline e2e, perf A/B and visual workflows, and gate their product jobs on its
output. A pull request that changes only infra paths now skips Security /
runtime, report, loki-pinned, vl-pinned, drilldown-pinned-runtime,
drilldown-grafana-pr-matrix, e2e-pipeline, perf-smoke and visual-smoke. A skipped
job reports success, so the required checks still pass.

The gate is "!cancelled() && needs.scope.outputs.product != 'false'", combined
with each job's existing condition. If the scope job fails its output is empty,
so the product jobs run. Pushes to main, scheduled runs and manual dispatches
are unchanged: the classifier reports product for every event that is not a
pull request. Security / static, the matrix-source jobs, the scheduled matrices
and the visual publish and comment jobs keep their current conditions; the last
two already skip when visual-smoke is skipped.

visual-smoke combines the gate with its label condition, and the scope job
carries the same condition so a label that does not re-run the check does not
start a scope run either. PR Quality Report moves pull-requests: write from the
workflow to the report job, which is the only job that needs it.
@github-actions github-actions Bot added size/L Large change scope/proxy Proxy core scope/ci CI/CD labels Oct 22, 2026
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

⚪ Performance A/B: no change beyond noise

8 shape×range — 8 ⚪ same · results vs Loki: 7 same, 0 new difference(s), 1 pre-existing

Every shape is within noise of the base build (±30% and 100 ms).

8 unchanged shape×range (within noise)
shape range base p50 PR p50 change Loki p50 PR ÷ Loki result vs Loki
⚪ drilldown service landing 1h 32 ms 37 ms ⚪ +16% 29 ms 1.28× ✅ same
⚪ instant ungrouped rate instant 8 ms 7 ms ⚪ -12% 76 ms 0.09× ✅ same
⚪ json rate by level (no filter, no drop) 1h 400 400 ⚪ +0% 400 — ✅ same
⚪ json volume no filters 1h 219 ms 222 ms ⚪ +1% 62 ms 3.58× ✅ same
⚪ logs plain 1h 106 ms 110 ms ⚪ +4% 221 ms 0.50× ✅ same
⚪ logs | json 1h 100 ms 94 ms ⚪ -6% 354 ms 0.27× ✅ same
⚪ stream-label count by app 1h 27 ms 30 ms ⚪ +11% 43 ms 0.70× ✅ same
⚪ topk rate by app 1h 45 ms 47 ms ⚪ +4% 95 ms 0.49× ➖ differs (as base)
Cold (first-run) timings — what a user opening a panel sees
shape range base cold PR cold Loki cold PR ÷ Loki cold
drilldown service landing 1h 37 ms 34 ms 833 ms 0.04×
instant ungrouped rate instant 8 ms 8 ms 80 ms 0.10×
json rate by level (no filter, no drop) 1h 400 400 400 —
json volume no filters 1h 213 ms 211 ms 2.04 s 0.10×
logs plain 1h 161 ms 121 ms 262 ms 0.46×
logs | json 1h 231 ms 233 ms 668 ms 0.35×
stream-label count by app 1h 28 ms 25 ms 888 ms 0.03×
topk rate by app 1h 47 ms 49 ms 782 ms 0.06×
Why these shapes
  • control smoke → 8: control/drilldown service landing, control/instant ungrouped rate, control/json rate by level (no filter, no drop), control/json volume no filters, control/logs plain, control/logs | json, control/stream-label count by app, control/topk rate by app
  • internal/proxy/proxy.go (endpoint handler) → 8: control/drilldown service landing, control/instant ungrouped rate, control/json rate by level (no filter, no drop), control/json volume no filters, control/logs plain, control/logs | json, control/stream-label count by app, control/topk rate by app

base 4a000ba59d → PR 62d2e9fd4a · 4 runs (first is cold), warm p50 · ranges 1h · noise ±30% and 100 ms · VictoriaLogs restarts 0→0 · 1.9 min. Same runner, one fresh stack, targets interleaved per request. How to read and reproduce: bench/ab/README.md.

@github-actions github-actions Bot added size/XS Extra small change and removed size/L Large change labels Oct 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

✅ Visual smoke: passed

Base 4a000ba59d vs PR 60406da94c, Grafana through the proxy builds and Loki on one fresh stack; run.
Core set: 5 capture(s) at 1h. Detailed set: 0 entries (nothing visual touched). 5 capture(s), 15m/1h/6h, 24h and 7d stay out of PR runs.

page range set base = PR (data) vs Loki (PR) pixel diff result
dd-fields 1h core ✅ identical identical (4 explained, 1 history-dependent) 0.11% ✅
dd-landing 1h core ✅ identical identical (3 explained) 0.14% ✅
dd-service-logs 1h core ✅ identical identical (3 explained, 1 history-dependent) 0.11% ✅
explore-logs-plain 1h core ✅ identical identical (1 explained) 0.07% ✅
explore-metric-plain 1h core ✅ identical identical (1 explained) 0.04% ✅

Gate: a base-vs-PR data difference passes when the PR is closer to Loki, and fails when the PR diverges from Loki or Loki cannot decide it (at a range Loki does not hold, a difference passes when the same page proved it closer to Loki at a shorter range; otherwise label the pull request visual-change-expected to accept an intended change). Also failing: a panel empty on the PR but not on the base, an error banner, panel error, error answer or new "No data" panel on the PR, a PR side that never settled, a difference that did not reproduce on the recapture. Pixel differences only warn.

Core set montages (5): base | PR | Loki

dd-fields-1h

dd-fields-1h

dd-landing-1h

dd-landing-1h

dd-service-logs-1h

dd-service-logs-1h

explore-logs-plain-1h

explore-logs-plain-1h

explore-metric-plain-1h

explore-metric-plain-1h

All montages of this run: pr-visuals/pr-824.

@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

PR Quality Report

Compared against base branch main.

Coverage and tests

Signal Base PR Delta
Test count 11827 11827 0
Coverage 88.3% 88.3% 0.0% (stable)

Compatibility

Track Base PR Delta
Loki API 100.0% 11/11 (100.0%) 0.0% (stable)
Logs Drilldown 100.0% 17/17 (100.0%) 0.0% (stable)
VictoriaLogs 100.0% 11/11 (100.0%) 0.0% (stable)

Performance smoke

Lower CPU cost (ns/op) is better. Lower benchmark memory cost (B/op, allocs/op) is better. Higher throughput is better. Lower load-test memory growth is better. Benchmark rows are medians from repeated samples.

Signal Base PR Delta
QueryRange cache-hit CPU cost 3527.0 ns/op 3528.0 ns/op +0.0% (stable)
QueryRange cache-hit memory 1240.0 B/op 1240.0 B/op 0.0% (stable)
QueryRange cache-hit allocations 27.0 allocs/op 27.0 allocs/op 0.0% (stable)
QueryRange cache-bypass CPU cost 4229.0 ns/op 4405.0 ns/op +4.2% (stable)
QueryRange cache-bypass memory 1306.0 B/op 1308.0 B/op +0.2% (stable)
QueryRange cache-bypass allocations 27.0 allocs/op 27.0 allocs/op 0.0% (stable)
Labels cache-hit CPU cost 928.2 ns/op 936.4 ns/op +0.9% (stable)
Labels cache-hit memory 48.0 B/op 48.0 B/op 0.0% (stable)
Labels cache-hit allocations 3.0 allocs/op 3.0 allocs/op 0.0% (stable)
Labels cache-bypass CPU cost 1537.0 ns/op 1564.0 ns/op +1.8% (stable)
Labels cache-bypass memory 400.0 B/op 400.0 B/op 0.0% (stable)
Labels cache-bypass allocations 14.0 allocs/op 14.0 allocs/op 0.0% (stable)

State

  • Coverage, compatibility, and sampled performance are reported here from the same PR workflow.
  • This is a delta report, not a release gate by itself. Required checks still decide merge safety.
  • Performance is a smoke comparison, not a full benchmark lab run.
  • Delta states use the same noise guards as the quality gate (percent + absolute + low-baseline checks), so report labels match merge-gate behavior.

@szibis szibis closed this Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

scope/ci CI/CD scope/proxy Proxy core size/XS Extra small change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant