Skip to content

ci: cut a release and require a changelog entry only when something that ships changes - #827

Merged
szibis merged 1 commit into
mainfrom
ci/release-only-when-shipped
Oct 22, 2026
Merged

szibis merged 1 commit into
mainfrom
ci/release-only-when-shipped

Conversation

@szibis

@szibis szibis commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

A release, and the changelog entry that goes with it, are for changes users receive. This PR makes both depend on one question, "does anything that ships change?", instead of on the infra or product classification and on commit subjects.

Bench, test suites (Go e2e, Playwright), the conformance registry and evidence, scripts, workflows, compose files, examples, docs and the website never cut a release and never need a changelog entry.

CI scoping is untouched: classify(), scope(), the rule tables and their test expectations are unchanged.

Shipped paths

A path ships when it matches one of these (SHIPPED in scripts/ci/change_scope.py):

Glob Why
cmd/**, internal/**, pkg/** compiled into the binary and image. *_test.go files and any testdata/ directory are excluded: they are compiled into nothing (no go:embed exists, and a test guards that)
go.mod, go.sum dependency set of the binary
Dockerfile, .dockerignore the image build and its context
charts/** the packaged Helm chart, including templates/tests/**
dashboard/**, alerting/** the sources scripts/ci/sync_observability_assets.sh copies into the chart, and the files the operations guide points users at

pkg/** has no files today; it is kept so a package added there ships without anyone editing the table (the dead-glob guard allows it explicitly and fails once the directory exists, so the exemption gets removed).

A path that cannot be placed (empty, absolute, ..) does not ship. This is the opposite of the CI classifier, which treats it as product, because releasing on a guess is the mistake this prevents. change_scope.py --ships prints ships=true|false; an empty list or an internal error prints ships=true, as the release bot always behaved, and its remaining checks (changelog content, labels) stay the deciding gate then.

Behaviour

Case Before After
fix: touching only internal/x/y_test.go entry required no entry
fix: / feat: touching only bench/, test/, conformance/, scripts/ or .github/ entry required for bench, test and registry paths and for the scripts the classifier keeps as product (rules P1 to P3: security, e2e and compat scripts, smoke-test.sh, the wait_* scripts, the quality and conformance gates); no entry for other scripts, .github/ and the classifier itself (P4) no entry
Fix that edits internal/x/y.go and internal/x/y_test.go entry required entry required
Any change under charts/ (including templates/tests/**), dashboard/, alerting/, Dockerfile, .dockerignore, go.mod, go.sum, cmd/, internal/ entry required entry required
Dependency-only PR that ships (go.mod, go.sum, Dockerfile) exempt exempt
Dependency-only PR for bench or test manifests exempt exempt
Release metadata sync, edit of released history unchanged unchanged
Push to main where nothing since the tag ships (after the Chart.yaml version-only filter) classifier decided; bench, test and registry changes released "Nothing that ships changed since ; skipping release"
Push to main where something ships release release
Go toolchain bumps nothing tied the workflow pins to the shipped files a workflow-only go-version bump fails CI; a real bump must change go.mod and the Dockerfile, which ship, so it releases

Go toolchain pin test (scripts/ci/tests/test_go_toolchain_pins.py, run in the lint job's unit-test step): the go.mod go directive (and toolchain directive when present), the Dockerfile golang: builder image, and every actions/setup-go go-version: in all workflows must be the same version. All workflows are included, not only auto-release.yaml and release.yaml (which build the release binaries and are asserted to be present), because every workflow already runs the same version and one rule is simpler than two. Each setup-go step must carry a literal go-version.

Changes

  • scripts/ci/change_scope.py: SHIPPED, NOT_SHIPPED_GO, ships(), any_ships(), --ships.
  • scripts/ci/check_changelog_pr.py: should_require_changelog is "any changed file ships" (with no files, a release-type subject is the only evidence). Obsolete path tables and helpers are removed. The dependency-only exemption, the released-history guard and the metadata-sync exemption are unchanged.
  • .github/workflows/auto-release.yaml: the since-last-tag decision calls change_scope.py --ships after the existing Chart.yaml version-only filter. No new run: interpolation.
  • .github/workflows/ci.yaml: one line, adds the toolchain pin test to the unit-test step.
  • Tests: test_change_scope.py (ships table, real PR fixtures, dead-glob guards, CLI), test_check_changelog_pr.py, new test_go_toolchain_pins.py.

Changed existing assertions in test_check_changelog_pr.py:

  • fix(proxy) with README.md and bench/ab/daily.py: now no entry (a second case with internal/proxy/tail.go keeps the entry);
  • test: with test/e2e-compat/features_test.go: now no entry;
  • the three "benchmark harness / registry changes still require changelog" assertions: now no entry (test renamed; the third case's subject changed from bench: to feat: to show the subject no longer matters);
  • fix: with internal/x/y_test.go: now no entry (test renamed);
  • is_dependency_only_pr with bench/visual/package.json and bench/visual/run.py: now exempt (the script ships nothing); the rejecting case now pairs the manifest with internal/proxy/proxy.go;
  • the "product paths are not non-release" list drops bench/ab/daily.py, bench/visual/package.json and conformance/registry/baseline.json, which move to a new non-release test; it gains dashboard/loki-vl-proxy.json, alerting/loki-vl-proxy-prometheusrule.yaml and the chart test template.

Proofs

Range Result
v2.24.4..0638246c (#807) skip
v2.25.1..origin/main (workflows, scripts, tests, Chart.yaml version only) skip
v2.24.5..6c20fb03 (#808) release (chart test template)
v2.25.0..17aa57c1 (#819) release (Dockerfile, go.mod, go.sum)
v2.24.3..v2.24.4 release (internal/)
  • actionlint: no findings. zizmor 1.30.1 --offline: identical output to main (no findings).
  • Negative check of the pin test: changing go-version in release.yaml alone fails it.

Not applicable

This PR ships nothing; product checks (e2e vs Loki, Grafana range matrix, visual, A/B, benchstat, load) do not apply. No changelog entry and no release.

It edits scripts/ci/change_scope.py, which is rule P4, so its own CI run is the full product run.

…hat ships changes

A release, and the changelog entry that goes with it, are for changes users
receive. Bench, test suites, the conformance registry, scripts, workflows,
compose files, examples, docs and the website change nothing that ships, so
they neither cut a release nor need an entry.

change_scope.py gains a ships() question next to the existing infra or
product classification, which is left as it is. A path ships when it is under
cmd/, internal/ or pkg/ (not a _test.go file and not testdata), is go.mod,
go.sum, Dockerfile or .dockerignore, or is under charts/, dashboard/ or
alerting/. A path that cannot be placed does not ship. The new --ships flag
prints ships=true|false; an empty list or an error still reports true, as the
release bot always did.

The changelog gate now asks only whether any changed file ships. A fix: that
only edits unit tests, the bench harness or the registry no longer needs an
entry, whatever the commit subject says. Dependency-only pull requests stay
exempt, and the released-history guard and the metadata-sync exemption are
unchanged.

auto-release.yaml decides on ships after its existing Chart.yaml version-only
filter and skips the release when nothing since the tag ships.

A Go bump changes the shipped binary, but a workflow-only edit of go-version
would not release it. A new test requires go.mod (go and toolchain), the
Dockerfile builder image and every setup-go go-version in the workflows to be
the same version, so a real bump has to touch go.mod and the Dockerfile.
@github-actions github-actions Bot added size/XL Extra large change scope/ci CI/CD maintenance Maintenance labels Oct 22, 2026
@szibis
szibis merged commit 304b554 into main Oct 22, 2026
43 checks passed
@szibis
szibis deleted the ci/release-only-when-shipped branch October 22, 2026 13:31
@github-actions

Copy link
Copy Markdown
Contributor

PR Quality Report

Compared against base branch main.

Coverage and tests

Signal Base PR Delta
Test count 11827 11827 0
Coverage 88.3% 88.3% 0.0% (stable)

Compatibility

Track Base PR Delta
Loki API 100.0% 11/11 (100.0%) 0.0% (stable)
Logs Drilldown 100.0% 17/17 (100.0%) 0.0% (stable)
VictoriaLogs 100.0% 11/11 (100.0%) 0.0% (stable)

Performance smoke

Performance smoke was skipped for this PR because no perf-sensitive paths changed.

State

  • Coverage, compatibility, and sampled performance are reported here from the same PR workflow.
  • This is a delta report, not a release gate by itself. Required checks still decide merge safety.
  • Performance smoke was intentionally skipped because no perf-sensitive paths changed in this PR.
  • Delta states use the same noise guards as the quality gate (percent + absolute + low-baseline checks), so report labels match merge-gate behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintenance Maintenance scope/ci CI/CD size/XL Extra large change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant