ci: harden workflows: bot token handling, expressions through env, zizmor, CodeQL for actions, approval-gated bot PRs by szibis · Pull Request #790 · ReliablyObserve/loki-vl-proxy

szibis · 2026-10-21T13:38:33Z

Summary

This change hardens the GitHub Actions workflows in three commits, keeping the existing release flow and its bot:

Release and daily workflows keep the bot token out of checkouts and scripts, and validate tag names.
- Checkouts no longer persist credentials.
- Pushes authenticate with an http.extraheader passed only to the steps that push, never a token in a URL or in .git/config.
- Secrets are passed to steps as environment variables, never written into script text, and registry logins use --password-stdin.
- Release tag and version strings are validated (^v[0-9]+\.[0-9]+\.[0-9]+$) and passed through the environment.
- release.yaml checks out refs/tags/<tag> and checks it equals HEAD.
- prepare-release is read-only.
Workflow hygiene across every workflow.
- No ${{ }} expression remains inside any run: script; they all go through step env:.
- zizmor 1.30.1 (pinned by digest) runs next to actionlint (now pinned by digest) in Security / static and fails on any new finding; exceptions are reviewed inline # zizmor: ignore[...] comments only.
- CodeQL also scans the actions language in a separate analyze (actions) job, so the Go analyze job keeps its identity and the required check name.
- persist-credentials: false is set on every checkout that does not push.
- Release and bot-token jobs use setup-go cache: false.
- A new repo test, scripts/ci/tests/test_workflow_pins.py, fails on an unpinned uses: or a missing version comment.
Release metadata and daily performance pull requests arm auto-merge and pass a scope check.
- auto-release (and a manual release.yaml republish) opens the metadata pull request as the release bot and arms auto-merge.
- Republish metadata commits are made through the API, so they are signed like the automatic ones.
- perf-daily arms auto-merge on the daily pull request.
- main requires one approval, so these merge right after the maintainer approves and the required checks pass. No workflow approves, and no bypass is used.
- The existing release-pr-validation.yaml becomes the scope check (release-pr-scope). It runs on pull_request_target from main's definition with no checkout, no secrets and no PR code executed, and reads the PR's files as data. It limits bot branches to their files:
  - metadata PRs: CHANGELOG, README, docs/observability.md, Chart.yaml, with matching versions, the bot or owner as author, and verified commits;
  - the daily PR: bench results, history and generated reports.
  Other pull requests pass immediately.
- scripts/ci/tests/test_release_pr_validation.py covers the scope rules and the workflow's shape, so adding a checkout to it fails CI.

Proofs

check	result
actionlint 1.7.7 (pinned image) and local	exit 0
zizmor 1.30.1 (pinned image) over `.github/workflows`	no findings (23 inline-ignored, each with its reason)
zizmor on copies with an injected `github.event.pull_request.title` in a `run:`, or a second cached `setup-go` job	exit 14 (caught)
`python3 -m unittest discover -s scripts/ci/tests`	73 tests pass (46 before)
scope-check simulation (the real `run:` text against a fake `gh` with recorded API answers)	19 cases: good metadata/daily PRs pass; files outside the allow-list, renames out of `internal/`, unsigned metadata commits, wrong versions, wrong authors and non-numeric PR numbers fail; unrelated, fork and non-main-base PRs pass immediately
header push (local logging HTTP server)	the `AUTHORIZATION: basic` header is sent, and the token is absent from the URL and `.git/config`
tag regex	accepts `v2.22.1`; rejects `v1.2.3$(id)`, `v1.2`, `main`, `v1.2.3-rc1`, trailing newline
`${{` inside any `run:` block (YAML parse)	none
conformance and changelog gates	ok

Required checks keep their names (analyze, test, Security / static, ...).

Shown by the first runs after merge:

the CodeQL actions results;
the zizmor SARIF in code scanning;
the first bot-opened metadata PR, which will show the release-pr-scope check attaching to its head. After that, it can be made a required check.

Not applicable: no proxy code changes, so e2e vs Loki, the Grafana range matrix, visual proof, A/B, benchstat and the load test do not apply.

…nd scripts, validate tag names The release workflows and the daily performance workflow checked the repository out with the release bot's token stored in the git configuration, so every later action in the job could read it, and they pushed by embedding the token in the push URL. Several steps also pasted secrets and tag-derived strings straight into script text. - Every checkout sets persist-credentials: false. The publish checkouts no longer receive the bot token at all (they only read). - The steps that push (release tag, release metadata branch in the manual republish, daily report branch) get the token through step-level env and send it as a masked one-command Authorization header, never in a URL or in .git/config. The tag push says that creating v* tags needs the bot token under the tag ruleset; the other fallbacks are unchanged. - No ${{ secrets.* }} remains inside run: scripts. Helm registry logins read the password from env and use --password-stdin. Docker Hub credentials are given only to the steps that use them; a job-wide HAS_DOCKERHUB flag replaces the job-level token variables in conditions. - Version, tag, revision and build time reach scripts through env. The manual republish accepts only vMAJOR.MINOR.PATCH, checks out refs/tags/<tag> and verifies it is the checked-out commit. The automatic release rejects a latest tag or computed version that is not plain semver. - prepare-release only reads, so its permissions drop to contents: read and pull-requests: read.

…QL for actions, pinned-uses test

…pass a scope check

github-advanced-security · 2026-10-21T13:39:17Z

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

github-actions · 2026-10-21T13:41:05Z

⚪ Performance A/B: no change beyond noise

8 shape×range — 8 ⚪ same · results vs Loki: 7 same, 0 new difference(s), 1 pre-existing

Every shape is within noise of the base build (±30% and 100 ms).

8 unchanged shape×range (within noise)

	shape	range	base p50	PR p50	change	Loki p50	PR ÷ Loki	result vs Loki
⚪	drilldown service landing	1h	41 ms	35 ms	⚪ -15%	25 ms	1.40×	✅ same
⚪	instant ungrouped rate	instant	10 ms	10 ms	⚪ +0%	72 ms	0.14×	✅ same
⚪	json rate by level (no filter, no drop)	1h	400	400	⚪ +0%	400	—	✅ same
⚪	json volume no filters	1h	222 ms	237 ms	⚪ +7%	45 ms	5.27×	✅ same
⚪	logs plain	1h	81 ms	78 ms	⚪ -4%	704 ms	0.11×	✅ same
⚪	logs \| json	1h	116 ms	113 ms	⚪ -3%	1.72 s	0.07×	✅ same
⚪	stream-label count by app	1h	28 ms	30 ms	⚪ +7%	43 ms	0.70×	✅ same
⚪	topk rate by app	1h	45 ms	46 ms	⚪ +2%	87 ms	0.53×	➖ differs (as base)

Cold (first-run) timings — what a user opening a panel sees

shape	range	base cold	PR cold	Loki cold	PR ÷ Loki cold
drilldown service landing	1h	37 ms	43 ms	567 ms	0.08×
instant ungrouped rate	instant	11 ms	9 ms	73 ms	0.12×
json rate by level (no filter, no drop)	1h	400	400	400	—
json volume no filters	1h	210 ms	206 ms	771 ms	0.27×
logs plain	1h	188 ms	142 ms	1.22 s	0.12×
logs \| json	1h	200 ms	192 ms	1.97 s	0.10×
stream-label count by app	1h	30 ms	25 ms	632 ms	0.04×
topk rate by app	1h	49 ms	46 ms	948 ms	0.05×

Why these shapes

control smoke → 8: control/drilldown service landing, control/instant ungrouped rate, control/json rate by level (no filter, no drop), control/json volume no filters, control/logs plain, control/logs | json, control/stream-label count by app, control/topk rate by app

_{base 91c73924f0 → PR 96dcec600e · 4 runs (first is cold), warm p50 · ranges 1h · noise ±30% and 100 ms · VictoriaLogs restarts 0→0 · 2.1 min. Same runner, one fresh stack, targets interleaved per request. How to read and reproduce: bench/ab/README.md.}

github-actions · 2026-10-21T13:43:14Z

✅ Visual smoke: passed with warnings

Base 91c73924f0 vs PR 5b4ef22f19, Grafana through the proxy builds and Loki on one fresh stack; run.
Core set: 5 capture(s) at 1h. Detailed set: 0 entries (nothing visual touched). 5 capture(s), 15m/1h/6h, 24h and 7d stay out of PR runs.

page	range	set	base = PR (data)	vs Loki (PR)	pixel diff	result
dd-fields	1h	core	✅ identical	identical (4 explained, 1 history-dependent)	0.12%	⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-landing	1h	core	✅ identical	identical (3 explained)	0.14%	✅
dd-service-logs	1h	core	✅ identical	identical (3 explained, 1 history-dependent)	0.12%	⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
explore-logs-plain	1h	core	✅ identical	identical (1 explained)	0.07%	✅
explore-metric-plain	1h	core	✅ identical	identical (1 explained)	0.04%	✅

Gate: a base-vs-PR data difference passes when the PR is closer to Loki, and fails when the PR diverges from Loki or Loki cannot decide it (at a range Loki does not hold, a difference passes when the same page proved it closer to Loki at a shorter range; otherwise label the pull request visual-change-expected to accept an intended change). Also failing: a panel empty on the PR but not on the base, an error banner, panel error, error answer or new "No data" panel on the PR, a PR side that never settled, a difference that did not reproduce on the recapture. Pixel differences only warn.

dd-fields-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-21T13%3A39%3

history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-21T13%3A39%3A00.000Z&query=%7Bservic: body: content differs (rows 6 vs 2)
vs Loki, explained: query ('sum by (service_name_extracted) (count_over_time({service_name="otel-collector"} | servi: request only on one side (right) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision)
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-21T13%3A39%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-21T13%3A39%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-service-logs-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-21T13%3A39%3

history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-21T13%3A39%3A00.000Z&query=%7Bservic: body: content differs (rows 6 vs 2)
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-21T13%3A39%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-21T13%3A39%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

Core set montages (3): base | PR | Loki

dd-landing-1h

explore-logs-plain-1h

explore-metric-plain-1h

All montages of this run: pr-visuals/pr-790.

github-actions · 2026-10-21T13:49:49Z

PR Quality Report

Compared against base branch main.

Coverage and tests

Signal	Base	PR	Delta
Test count	11763	11763	0
Coverage	88.3%	88.4%	+0.1% (improved)

Compatibility

Track	Base	PR	Delta
Loki API	100.0%	11/11 (100.0%)	0.0% (stable)
Logs Drilldown	100.0%	17/17 (100.0%)	0.0% (stable)
VictoriaLogs	100.0%	11/11 (100.0%)	0.0% (stable)

Performance smoke

Lower CPU cost (ns/op) is better. Lower benchmark memory cost (B/op, allocs/op) is better. Higher throughput is better. Lower load-test memory growth is better. Benchmark rows are medians from repeated samples.

Signal	Base	PR	Delta
QueryRange cache-hit CPU cost	3487.0 ns/op	3521.0 ns/op	+1.0% (stable)
QueryRange cache-hit memory	1240.0 B/op	1240.0 B/op	0.0% (stable)
QueryRange cache-hit allocations	27.0 allocs/op	27.0 allocs/op	0.0% (stable)
QueryRange cache-bypass CPU cost	4397.0 ns/op	4147.0 ns/op	-5.7% (stable)
QueryRange cache-bypass memory	1306.0 B/op	1305.0 B/op	-0.1% (stable)
QueryRange cache-bypass allocations	27.0 allocs/op	27.0 allocs/op	0.0% (stable)
Labels cache-hit CPU cost	928.2 ns/op	881.7 ns/op	-5.0% (stable)
Labels cache-hit memory	48.0 B/op	48.0 B/op	0.0% (stable)
Labels cache-hit allocations	3.0 allocs/op	3.0 allocs/op	0.0% (stable)
Labels cache-bypass CPU cost	1525.0 ns/op	1535.0 ns/op	+0.7% (stable)
Labels cache-bypass memory	399.0 B/op	400.0 B/op	+0.3% (stable)
Labels cache-bypass allocations	14.0 allocs/op	14.0 allocs/op	0.0% (stable)

State

Coverage, compatibility, and sampled performance are reported here from the same PR workflow.
This is a delta report, not a release gate by itself. Required checks still decide merge safety.
Performance is a smoke comparison, not a full benchmark lab run.
Delta states use the same noise guards as the quality gate (percent + absolute + low-baseline checks), so report labels match merge-gate behavior.

szibis added 3 commits October 21, 2026 15:37

ci: workflow expressions through env, zizmor next to actionlint, Code…

9a0a605

…QL for actions, pinned-uses test

ci: release metadata and daily perf pull requests arm auto-merge and …

5b4ef22

…pass a scope check

github-actions Bot added size/XL Extra large change scope/ci CI/CD scope/docs Documentation maintenance Maintenance labels Oct 21, 2026

szibis merged commit 784a634 into main Oct 21, 2026
36 checks passed

szibis deleted the ci/workflow-hygiene branch October 21, 2026 13:40

szibis mentioned this pull request Oct 21, 2026

perf: a detected_level filter before json reads the stored level fields, matching Loki #795

Merged

Navigation Menu

Repository navigation

ci: harden workflows: bot token handling, expressions through env, zizmor, CodeQL for actions, approval-gated bot PRs - #790

ci: harden workflows: bot token handling, expressions through env, zizmor, CodeQL for actions, approval-gated bot PRs#790
szibis merged 3 commits into
mainfrom
ci/workflow-hygiene

szibis commented Oct 21, 2026

Uh oh!

github-advanced-security AI commented Oct 21, 2026

Uh oh!

Uh oh!

github-actions Bot commented Oct 21, 2026

Uh oh!

github-actions Bot commented Oct 21, 2026

Uh oh!

github-actions Bot commented Oct 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

szibis commented Oct 21, 2026

Summary

Proofs

Uh oh!

github-advanced-security AI commented Oct 21, 2026

What Enabling Code Scanning Means:

Uh oh!

Uh oh!

github-actions Bot commented Oct 21, 2026

⚪ Performance A/B: no change beyond noise

Uh oh!

github-actions Bot commented Oct 21, 2026

✅ Visual smoke: passed with warnings

Uh oh!

github-actions Bot commented Oct 21, 2026

PR Quality Report

Coverage and tests

Compatibility

Performance smoke

State

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants