Skip to content

ci: skip the product jobs in CI for infra-only pull requests - #822

Merged
szibis merged 1 commit into
mainfrom
ci/ci-yaml-scope
Oct 22, 2026
Merged

szibis merged 1 commit into
mainfrom
ci/ci-yaml-scope

Conversation

@szibis

@szibis szibis commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

Pull requests that change only tooling (workflows, scripts, Dependabot, security tooling, test-stack compose files, docs) run about 160 job-minutes of product jobs today, although they cannot change the proxy. This PR makes ci.yaml skip the product jobs when the change-scope classifier says no product path changed, and moves the checks that verify the infra itself into the always-on lint job, so an infra pull request is still checked.

Pushes to main, scheduled runs, manual dispatches and pull requests labelled ci-full keep running everything.

Builds on #820 (merged), which added the classifier and the reusable change-scope.yaml.

Changes

.github/workflows/ci.yaml

  • New scope job: calls ./.github/workflows/change-scope.yaml with contents: read and pull-requests: read. The workflow's top-level permissions are unchanged. The uses line carries a zizmor: ignore[self-repository] comment: zizmor suggests the $/ form for a same-repository call, and the pinned actionlint rejects it.
  • test, bench and conformance (the product jobs with no needs) gain needs: [scope] and if: ${{ !cancelled() && needs.scope.outputs.product != 'false' }}. A scope job that fails leaves the output empty, so the gate runs the job. Only a scope job that succeeds with false skips anything.
  • The jobs that already depend on test (tuple-contract, fuzz-smoke, race-stress shards, tuple-smoke, docker, e2e-compat groups, e2e-fleet, e2e-ui, stress-tests, memory-leak-tests) get no if:. They keep the implicit success(): they skip when test skips, and they do not run when test fails.
  • The two if: always() aggregators, e2e-compat (required) and race-stress, now also need scope. They pass when every group succeeded, or when the groups were skipped and the scope output is false. They fail otherwise, including when test failed on a product pull request and when the scope output is empty.
  • lint and helm have no if: and run on every pull request.
  • lint gains the infra checks, moved from test with identical commands: sync_observability_assets.sh --check, the scripts.ci.tests unittest list (plus test_change_scope and test_scope_wiring) and go run ./cmd/configdoc -check. It also gains two new steps: go test ./internal/logsql -run '^TestVictoriaLogsSupportPolicy$' -count=1 (so README edits stay checked) and scripts/ci/validate_compose.sh. They run after gofmt and golangci-lint.

scripts/ci/validate_compose.sh (new): runs docker compose config -q on every tracked compose file, 9 standalone files and 6 overlays stacked on the files they are applied after (15 stacks), and fails if a tracked docker-compose*.yml or compose*.yml is not in its list or a listed file is missing. It starts no container and needs no daemon.

scripts/ci/tests/test_scope_wiring.py (new): reads the workflows with a small stdlib parser (no PyYAML dependency) and asserts the wiring described below. It also mutates a copy of ci.yaml six ways (ungated test, missing scope in needs, weaker gate, a dependent with if: always(), a condition on lint, an aggregator that ignores the scope output) and checks each is caught.

Per-job behaviour

Job Class On an infra pull request How its check reports
scope (new) infra runs, about 10 s not required; on error the gates run the product jobs
lint always-on (now also the infra job) runs real result (required)
helm always-on runs real result (required)
test product skipped by the gate skipped counts as passing (required); a product pull request runs it and a failure blocks
bench, conformance product skipped by the gate skipped; not required
tuple-contract, fuzz-smoke, race-stress shards, tuple-smoke, docker, e2e-compat groups, e2e-ui, stress-tests, memory-leak-tests product skipped, because test is skipped skipped; not required (except through the aggregator below)
e2e-fleet product skipped, because test is skipped skipped counts as passing (required); on a product pull request a red test blocks as today
e2e-compat (aggregator) product runs, groups skipped and scope false, passes real result (required): fails if a group failed or was skipped on a product pull request, or the scope output is empty
race-stress (aggregator) product runs and passes, same rule real result; not required

Case walk-through

Case scope output test dependents bench, conformance e2e-compat, race-stress lint, helm
(a) infra pull request false skipped skipped skipped success (groups skipped, scope false) run
(b) product pull request, all green true success run, success run success (groups succeeded) run
(c) product pull request, test fails true failure skipped run failure (groups skipped, scope true) run
(d) scope job failed (runner-level) or output empty empty runs (!= 'false') skipped, because scope failed upstream and the implicit success() covers the whole chain run failure (groups skipped, scope output empty) run
(e) push to main (also schedule, dispatch, ci-full) true success run run success run

Case (d) fails closed: test, bench and conformance run, and e2e-compat goes red, so the merge is blocked; the dependents of test are not given an if: on purpose. The scope job rarely fails outright, because every step in it is continue-on-error and its result step defaults to true; a failure there means a runner-level problem.

Case (c) is the one that matters: e2e-compat and race-stress use always(), so without the explicit scope test a skipped group would read as a pass.

Proofs

  • actionlint (v1.7.7 as in the Security workflow, and the local 1.7.12): no findings across all workflows.
  • zizmor 1.30.1 --offline, same flags as the Security workflow: no findings; the one new self-repository hint on the scope call is suppressed inline as above.
  • python3 -m unittest discover -s scripts/ci/tests -p 'test_*.py': 127 tests pass, including test_workflow_pins, test_tool_pins, test_change_scope and the 7 in test_scope_wiring.
  • scripts/ci/validate_compose.sh with Compose v5.4.0: 15 of 15 stacks pass. It reports a tracked compose file that is missing from its list.
  • Moved steps verified locally: sync_observability_assets.sh --check, go run ./cmd/configdoc -check and TestVictoriaLogsSupportPolicy pass.
  • shellcheck on the new script: clean.

Verification on GitHub

  1. This pull request is infra-only. Its own run shows scope reporting product=false, every product job skipped, e2e-compat and race-stress green, and lint and helm running. The step summary of scope lists every path as infra.
  2. A throwaway draft pull request on top of this branch with a no-op comment change in internal/ shows every product job running with this workflow. Pushing one deliberately failing unit test to it should turn test red and e2e-compat red (its groups skip), and block the merge. The throwaway pull request is closed without merging.
  3. The first push to main after merge runs every job (scope reports true for a push).
  4. lint runs the compose validation with the runner's Compose version: confirm it passes there on this run.

Not applicable

Infra-only pull request; product checks (e2e vs Loki, Grafana range matrix, visual, A/B, benchstat, load) do not apply.

Add a scope job to ci.yaml that calls the reusable change-scope workflow, and
gate test, bench and conformance on its output. The jobs that depend on test
keep the implicit success() and skip together with it. The gate is
`!cancelled() && needs.scope.outputs.product != 'false'`, so a scope job that
fails or returns nothing runs every product job; only a scope job that
succeeds with false skips them. Pushes and manual runs always
report true.

The two aggregator jobs, e2e-compat and race-stress, run under always(). They
pass when every group succeeded, or when the groups were skipped and the scope
output is false. A skipped group on a product pull request (for example when
test failed) or an empty scope output now fails them, so a skip never reads as
a pass by accident.

lint and helm have no condition and run on every pull request. lint now also
carries the infra checks that used to sit in test: the observability asset
sync check, the scripts/ci unit tests (plus the classifier and scope wiring
tests) and the configuration reference check. Two new steps join them: the
README support-policy test and scripts/ci/validate_compose.sh, which runs
docker compose config -q on every tracked compose file (standalone files and
each overlay on its base) and fails when a compose file is missing from its
list.

test_scope_wiring.py reads the workflows and asserts that lint and helm are
never skipped, that every job reading the scope output lists scope in needs
and uses the canonical gate, that jobs downstream of test cannot bypass
success(), and that both aggregators test the scope output.
@github-actions github-actions Bot added size/L Large change scope/ci CI/CD maintenance Maintenance labels Oct 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR Quality Report

Compared against base branch main.

Coverage and tests

Signal Base PR Delta
Test count 11827 11827 0
Coverage 88.3% 88.4% +0.1% (improved)

Compatibility

Track Base PR Delta
Loki API 100.0% 11/11 (100.0%) 0.0% (stable)
Logs Drilldown 100.0% 17/17 (100.0%) 0.0% (stable)
VictoriaLogs 100.0% 11/11 (100.0%) 0.0% (stable)

Performance smoke

Performance smoke was skipped for this PR because no perf-sensitive paths changed.

State

  • Coverage, compatibility, and sampled performance are reported here from the same PR workflow.
  • This is a delta report, not a release gate by itself. Required checks still decide merge safety.
  • Performance smoke was intentionally skipped because no perf-sensitive paths changed in this PR.
  • Delta states use the same noise guards as the quality gate (percent + absolute + low-baseline checks), so report labels match merge-gate behavior.

@szibis
szibis merged commit bfc5184 into main Oct 22, 2026
49 of 51 checks passed
@szibis
szibis deleted the ci/ci-yaml-scope branch October 22, 2026 13:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintenance Maintenance scope/ci CI/CD size/L Large change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant