Repository navigation
ci: test against Loki 3.7.9 - #879
Merged
Merged
Conversation
Move the pinned Loki reference from 3.7.7 to 3.7.9, the latest stable release (2026-10-20), in the e2e-compat compose default, the review override, the root compose file and the loki-pinned, drilldown-pinned and vl-pinned workflow jobs. compatibility-matrix.json keeps every patch release of the supported families (its support_window: "Keep all patch releases in the supported families"), so the weekly Loki matrix gains 3.6.17, 3.6.18, 3.6.19, 3.7.8 and 3.7.9. matrix-link.json and the compatibility-matrix report are regenerated from it. Loki 3.7.7 to 3.7.9 changes one file the proxy mirrors: pkg/logql/accumulator.go (grafana/loki#24703). The stream accumulator of a sharded log query no longer drops entries while it still has room, so a query that has more than `limit` matching entries now returns `limit` entries instead of sometimes fewer. pkg/loghttp, pkg/querier (including queryrange), pkg/util/server, pkg/util/marshal, pkg/distributor, pkg/pattern, pkg/validation and pkg/logql/syntax are unchanged; the other changes are dependency and Go 1.26.10 security updates, a 5 MiB body limit on the ruler CreateRuleGroup write route (413) and on the LogQL analyzer, and an ingester WAL ordering fix. The extracted Loki route surface is identical apart from its version string.
Update the places that describe what the stack pins today: the Loki version table (3.7.9 pinned, 3.7.7 and 3.7.8 in the scheduled matrix, 3.6.x up to 3.6.19), the workflow table and pinned-stack list in the testing guide, the e2e guide's service table and LOKI_IMAGE default, the real-window note on the compose default and the review-override note. Measured results recorded against Loki 3.7.7 stay as they were.
github-actions
Bot
added
size/M
Contributor
⚪ Performance A/B: no change beyond noise8 shape×range — 8 ⚪ same · results vs Loki: 7 same, 0 new difference(s), 1 pre-existing Every shape is within noise of the base build (±30% and 100 ms). 8 unchanged shape×range (within noise)
Cold (first-run) timings — what a user opening a panel sees
Why these shapes
base |
Contributor
✅ Visual smoke: passed with warningsBase
Gate: a base-vs-PR data difference passes when the PR is closer to Loki, and fails when the PR diverges from Loki or Loki cannot decide it (at a range Loki does not hold, a difference passes when the same page proved it closer to Loki at a shorter range; otherwise label the pull request dd-fields-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-27T16%3A53%3
dd-service-logs-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-27T16%3A53%3
Core set montages (3): base | PR | Lokidd-landing-1h
explore-logs-plain-1h
explore-metric-plain-1h
All montages of this run: |
Contributor
PR Quality ReportCompared against base branch Coverage and tests
Compatibility
Performance smokeLower CPU cost (
State
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Based on
mainat7a6cb76b(v2.36.1 metadata).Move the Loki reference the proxy is tested against from 3.7.7 to 3.7.9, the latest stable release (2026-10-20). This PR changes only infrastructure: compose files, the compatibility workflows, the compatibility matrix and docs. It ships no code and needs no release.
Commits:
ci: test against Loki 3.7.9${LOKI_IMAGE:-grafana/loki:3.7.9}), the review override and the rootdocker-compose.yml.loki-pinned,drilldown-pinned-runtimeandvl-pinnedjobs (6LOKI_IMAGElines).compatibility-matrix.json:pinned_versionis now 3.7.9.matrix_versionsgains 3.6.17, 3.6.18, 3.6.19, 3.7.8 and 3.7.9, because the matrix's support window says "Keep all patch releases in the supported families". 3.6.17 and 3.6.18 had been left out.matrix-link.jsonandconformance/reports/compatibility-matrix.mdare regenerated from the matrix.docs: name Loki 3.7.9 as the tested referencedocs/compatibility-loki.md, the workflow table and pinned-stack list indocs/testing.md, the service table andLOKI_IMAGEdefault indocs/testing-e2e-guide.md, the compose note indocs/real-window-compatibility-gaps.mdand the review-override note.No CHANGELOG entry: no
cmd/,internal/,pkg/,charts/,go.modorDockerfilechange.Loki 3.7.7 → 3.7.9
I diffed the tags directly, along with the 3.7.8 and 3.7.9 release notes. Every package the proxy mirrors is unchanged except one file:
pkg/loghttp,pkg/querier(includingqueryrangelimits, splitting and the results cache),pkg/util/server/error.go,pkg/util/marshal,pkg/distributor/field_detection.go,pkg/pattern,pkg/validation/limits.go,pkg/logql/syntax;pkg/logql/accumulator.go.pkg/logql/accumulator.go:342,345(grafana/loki#24703): a sharded log query's stream accumulator no longer drops entries while it still has roomdownstream.go:733,759; the e2e stack runsparallelise_shardable_queries: trueon TSDB). Loki 3.7.7 could return fewer thanlimitentries when more matched; 3.7.9 returnslimitCreateRuleGroupand the LogQL analyzer: 413 above a 5 MiB / 1 MiB bodyThe Loki route surface extracted for v3.7.9 (
conformance/scripts/sync_loki.py) is identical to v3.7.7 apart from the version string. The registry keeps its v3.7.7 surface; moving it is a follow-up.e2e-compat groups: Loki 3.7.7 vs 3.7.9
How the runs were set up:
test/e2e-compat/docker-compose.yml(its own compose project, host ports +10000).fc506876. The Go code was unchanged frommainat that point; the rebase onto7a6cb76bbrings in onlymain's own changes.go test -v -tags=e2e -timeout=600s -count=1 -run '<ci.yaml group pattern>' ./test/e2e-compat/.LOKI_IMAGEdiffers between paired runs.The per-test status comparison covers every
--- PASS/FAIL/SKIPline with probe ids normalised. The score lines (Score: n/m) are identical between the releases in every group.Failures and differences, classified:
TestFeature_Tail_PipelineParity(v1.53.0 tail-multitenancy, both Loki releases): flaky, and not caused by the Loki version. The failure appeared on 3.7.7 and on 3.7.9 alike, and a rerun of the whole group passed 45 / 0 on both releases. A tail-only run on 3.7.9 also passed. Two separate symptoms:http://127.0.0.1:23101) closed:tail_pipeline_test.go:178: … tail closed. This caused all six failures on 3.7.7 and four of the six on 3.7.9.synthetic/categorize-labelsfailed differently: the proxy's synthetic tail delivered 3 entries where Loki delivered 7 (proxy tail lacks …).TestDrilldown_LongRangeVolume_SplitQueriesStayDense(v1.53.0 drilldown, 3.7.7 run): skipped by its own timing guard.TestComplex_*comparisons pass as "line count close" (for example Loki 73, proxy 76). The numbers are the same on 3.7.7 and 3.7.9, so they predate this change and are unrelated to the accumulator fix.A/B harness smoke (Loki 3.7.9)
Ran
python3 bench/ab/pr_smoke.py --base origin/mainonce on a fresh stack, with Loki 3.7.9 as the compose default. This was the same command CI runs, with the control smoke subset thatbench/ab/selection.pypicks for the compose change.a3d0cb56. PR: this branch before the rebase.passed, all 8 shape×range rows the same. Against Loki: 7 rows the same, 0 new differences, and 1 pre-existing difference (topk rate by appdiffers from Loki on the base as well).json rate by levelanswers 400 on Loki, the base and the PR alike, as before.instant ungrouped rateis within the harness noise floor (±30% and 100 ms). Two things explain the gap: the basea3d0cb56already had #877, which this branch picked up only with the rebase, and an instant query's p50 at this size is a few tens of ms.Follow-ups
generated/loki/v3.7.7/, 35loki_version:fields) to v3.7.9. The extraction output is identical apart from the version.TestFeature_Tail_PipelineParityis flaky on VictoriaLogs v1.53.0 in the full tail-multitenancy group: Loki's reference tail can close mid-test, and the proxy's synthetic tail once delivered fewer entries than Loki (3 vs 7). Both passed on rerun. Capture Loki and proxy logs on failure, and investigate the synthetic tail shortfall separately.