Skip to content

ci: classify pull request changes as infra or product - #820

Merged
szibis merged 1 commit into
mainfrom
ci/change-scope-classifier
Oct 22, 2026
Merged

szibis merged 1 commit into
mainfrom
ci/change-scope-classifier

Conversation

@szibis

@szibis szibis commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a path classifier that decides whether a pull request changes the proxy (product) or only the tooling around it (infra), and a reusable change-scope workflow that exposes the answer as an output. Nothing calls the workflow yet, so this PR changes no CI behaviour. It lands the classifier, its tests and the workflow so later PRs can gate jobs on needs.scope.outputs.product.

The classifier is fail-safe by construction: a path that matches no rule is product, and so is an empty or unreadable file list, a crash, an unknown argument, or any event other than pull_request.

Changes

  • scripts/ci/change_scope.py (new, stdlib only). classify(path) -> (is_product, rule_id) and scope(paths) -> bool. Two ordered glob tables, PRODUCT_OVERRIDES (P1 to P4) then NON_PRODUCT (I1 to I12); the first match wins and everything else is P-default. Paths that are absolute, contain .., or are blank are product. CLI: --files-from FILE|-, --force-product, --github-output FILE (appends product=true|false; prints it to stdout when absent), --summary FILE (markdown table of path, class, rule, capped at 500 rows). On empty input or any error it reports product=true and exits 0.
  • scripts/ci/tests/test_change_scope.py (new, 35 tests). Per-rule table, overrides beating scripts/** and security/**, unknown, empty and mixed lists, the file lists of 13 real pull requests as literal fixtures, every tracked file classifies, a dead-rule guard (every glob matches at least one tracked file), and the CLI (output format, append behaviour, --force-product, empty and missing input, unknown argument, unwritable summary).
  • .github/workflows/change-scope.yaml (new, on: workflow_call, output product). Reports true for any event other than pull_request and when the PR has the ci-full label, read through the API at run time so a re-run after adding the label picks it up. Otherwise it checks out the merge ref (fetch-depth: 2, persist-credentials: false), runs git diff --name-only --no-renames HEAD^1 HEAD, and pipes that into the classifier. --no-renames lists both sides of a rename, so moving a file out of a product directory into an infra one is still seen. The gate, checkout and classify steps are continue-on-error, and a final step reports false only when every step succeeded and the classifier said false; every other outcome is true. Values reach run: through env: only. Permissions: workflow {}, job contents: read and pull-requests: read. Callers must grant the same.
  • .github/workflows/changelog-pr.yaml: adds scripts.ci.tests.test_change_scope to the existing unittest step.

Classification

Rule Globs Class Why
P1 scripts/ci/run_security_regressions.sh, scripts/ci/run_zap_scan.sh, security/zap/** product Only the runtime security job runs them.
P2 scripts/ci/wait_e2e_stack.sh, scripts/ci/wait_loki_track_stack.sh, scripts/ci/wait_pipeline_stack.sh, scripts/smoke-test.sh, scripts/compat/** product Only e2e, compat, tuple-smoke and runtime jobs run them.
P3 scripts/ci/collect_quality_metrics.sh, scripts/ci/check_quality_gate.py, scripts/ci/render_pr_report.sh, scripts/ci/check_conformance.py product They are the report and conformance gates.
P4 scripts/ci/change_scope.py, .github/workflows/change-scope.yaml product A PR that edits the gating machinery runs every check. Distinct rule id so release tooling can treat it as non-release.
I1 .github/** infra actionlint, zizmor, CodeQL actions, pin tests.
I2 scripts/** infra scripts/ci unit tests; the helm job covers the helm values files.
I3 security/** infra Semgrep and nuclei run on a schedule; the tool-pin test covers the manifests.
I4 bench/ab/history/**, bench/ab/results/** infra Daily performance report data. The rest of bench/ is product.
I5 test/*/docker-compose*.yml infra Test stack compose files.
I6 test/e2e-ui/package.json, test/e2e-ui/package-lock.json infra Tooling dependency manifests.
I7 examples/**, docker-compose.yml infra Not shipped.
I8 website/** infra Docs site build.
I9 docs/**, results/** infra Documentation and stored results.
I10 README.md, CHANGELOG.md, LICENSE, SECURITY.md, CONTRIBUTING.md infra Root documents.
I11 conformance/registry/generated/perf-evidence.json, conformance/reports/performance.md, conformance/reports/gaps.md infra Outputs of the daily performance report.
I12 .gitignore, .gitleaks.toml, .trivyignore.yaml, .hadolint.yaml, .golangci.yml infra Read by static security and lint, which always run.
P-default everything else product cmd/, internal/, charts/, go.mod, Dockerfile, the rest of conformance/, test/ and bench/.

Matching is segment-aware: * and ? never cross /, ** does. So .github/** matches every nested path, while test/*/docker-compose*.yml matches only a compose file one directory below test/ (a deeper one stays product).

Proofs

$ python3 -m unittest scripts.ci.tests.test_change_scope -v
Ran 35 tests ... OK

$ python3 -m unittest discover -s scripts/ci/tests -p 'test_*.py'
Ran 120 tests ... OK        (85 existing + 35 new)

$ uv run --no-project --python 3.12 python -m unittest scripts.ci.tests.test_change_scope scripts.ci.tests.test_check_changelog_pr
Ran 67 tests ... OK         (the Python the changelog workflow uses)

$ python3 -m unittest scripts.ci.tests.test_workflow_pins scripts.ci.tests.test_tool_pins
Ran 10 tests ... OK

$ actionlint .github/workflows/change-scope.yaml .github/workflows/changelog-pr.yaml    # 1.7.12, shellcheck on PATH
exit 0, no output

$ uvx zizmor@1.30.1 --offline --no-progress --format plain .github/workflows
No findings to report. (23 ignored, 106 suppressed; same as main)

Real pull request file lists (gh api repos/.../pulls/N/files) through the CLI:

PR Paths by rule Result
#807 I1 x2, I2 x2, I10 x1 product=false
#810 I5 x1 product=false
#811 I6 x2 product=false
#814 I3 x2 product=false
#815 I8 x2 product=false
#816 I3 x1 product=false
#817 I1 x3 product=false
#805 I4 x42, I11 x3 product=false
#808 P1 x1, P-default x2 (chart test, bench/visual), plus infra product=true
#812 P-default x2 (go.mod, go.sum) product=true
#813 P-default x2 (bench/visual manifests) product=true
#804 P-default x2 (Go source), plus infra product=true
#582 P-default x12, plus infra product=true

Over git ls-files (1807 files): 510 infra, 1297 product. Infra by rule: I1 29, I2 25, I3 6, I4 177, I5 6, I6 2, I7 30, I8 34, I9 188, I10 5, I11 3, I12 5. Product by rule: P1 3, P2 5, P3 4, P4 2, P-default 1283. No glob is dead and the six compose files under test/ are the only I5 matches.

Merge-ref check in a scratch repository: a PR that renames internal/a.go to docs/a.go. With --no-renames the list contains both paths and the result is product=true; with rename detection only docs/a.go would be listed.

Not applicable: infra-only PR; product checks (e2e vs Loki, Grafana range matrix, visual, A/B, benchstat, load) do not apply.

Add a path classifier that decides whether a pull request touches the
proxy or only tooling around it. Two ordered tables of fnmatch globs
(product overrides first, then the infra rules) give every path a class
and a rule id; anything that matches no rule is product, and so is an
empty or unreadable file list. The CLI reads changed paths from a file or
stdin, appends product=true|false to the GitHub output file, optionally
writes a markdown table of path, class and rule to the step summary, and
reports product=true on any error instead of failing.

Add a reusable change-scope workflow that wraps it. It reports product
for every event other than a pull request and when the pull request has
the ci-full label (read through the API at run time so a re-run picks up
a label added later). Otherwise it checks out the merge ref, lists the
net change against the base with renames split into old and new paths,
and classifies it. Any failure in that path reports product. Nothing
calls the workflow yet.

The classifier tests run in the changelog PR workflow. They cover each
rule, the overrides, the fail-safe cases, the file lists of real pull
requests, every tracked file, a dead-rule guard and the CLI.
@github-actions github-actions Bot added size/XL Extra large change scope/ci CI/CD maintenance Maintenance labels Oct 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR Quality Report

Compared against base branch main.

Coverage and tests

Signal Base PR Delta
Test count 11827 11827 0
Coverage 88.3% 88.3% 0.0% (stable)

Compatibility

Track Base PR Delta
Loki API 100.0% 11/11 (100.0%) 0.0% (stable)
Logs Drilldown 100.0% 17/17 (100.0%) 0.0% (stable)
VictoriaLogs 100.0% 11/11 (100.0%) 0.0% (stable)

Performance smoke

Performance smoke was skipped for this PR because no perf-sensitive paths changed.

State

  • Coverage, compatibility, and sampled performance are reported here from the same PR workflow.
  • This is a delta report, not a release gate by itself. Required checks still decide merge safety.
  • Performance smoke was intentionally skipped because no perf-sensitive paths changed in this PR.
  • Delta states use the same noise guards as the quality gate (percent + absolute + low-baseline checks), so report labels match merge-gate behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintenance Maintenance no-release Skip auto-release for this PR scope/ci CI/CD size/XL Extra large change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant