[szibis]

Summary

Follow-up to #562, from a post-merge review of the paths it touched. The headline problem: a grouped range metric whose grouping had more values than -max-stats-query-series was answered with a silent cut — the busiest 200 values on the short-range path and on the two-phase path used from two hours on — so a chart or an alert read a subset as if it were the whole answer, and the number 200 had nothing to do with the configured limit.

Loki's behaviour is the reference (pkg/logql/engine.go, logqlmodel.NewSeriesLimitError): a query over max_query_series fails with

maximum number of series (N) reached for a single query; consider reducing query cardinality by adding
more specific stream selectors, reducing the time range, or aggregating results with functions like
sum(), count() or topk()

as HTTP 400, and only a Logs Drilldown request (X-Query-Tags: Source=grafana-lokiexplore-app) gets a partial result with the warning ... reached for a single query; returning partial results in the body.

The proxy now does the same, on range and instant queries. The message is Loki's word for word, since Grafana renders it; the flag that set the limit is named in the proxy's log line for the rejected request and in the docs. The limit is applied while the VictoriaLogs response is being read, so an oversized answer is no longer buffered in full first, and it is never converted into a raw log scan.

The Drilldown paths lose their hardcoded 500: they read the same flag, ask VictoriaLogs for one series over it so a truncated answer is distinguishable from an exact fit, and report the cut as the partial-result warning instead of dropping series quietly. Where Loki keeps the first series it encounters, the proxy keeps the busiest by total count — the ones a Drilldown panel renders — which is stated in the registry case and in KNOWN_ISSUES.

Also in this change

Problem	Behaviour now
A failed or pending backend version probe made the proxy assume the stats_query_range offset arg is unavailable, so any range whose start was not aligned to the window fell back to reading raw log rows — data read then scaled with the logs stored	The arg is sent while the version is unknown (a backend that predates it ignores it and keeps the epoch-aligned buckets it would have had anyway) and the probe keeps retrying in the background. A version known to be older than v1.45 still uses epoch buckets only where they match the query's windows
count_over_time({app="x"}[5m] offset 5m) reported its sample at T-5m instead of Loki's T, shifting the series left by the offset, and cached it that way	Samples are stamped at the evaluation time on range and instant queries; the shifted body is not cached. Only sample responses are rewritten, so a log query still streams
A Drilldown panel with a range shorter than the step ([1m] at step 300) counted the whole step bucket: 30 where Loki answers 6	Those panels use the same window evaluator as every other client
A Grafana dashboard or Explore panel grouping by a high-cardinality field (sum by (trace_id) (…)) was rewritten to the window-sampled /hits path, which returns a bounded sample rather than Loki's series	The rewrite is limited to Drilldown, whose panels are built for it
The top-value in() filter was cut at a fixed count: long values could still exceed VictoriaLogs' default 16 KiB query length and be rejected, and filtering could leave an empty in() that matches nothing	The list is budgeted against the actual query text and always keeps at least the busiest value
Single-field groupings over two hours ran a ranking query and then a full bucket query over the same range — an ordinary sum by (namespace) (…) scanned the range twice	The bucket query runs alone; values are ranked only once the result exceeds the series limit
The ordered JSON metric path rejected at len(series) >= limit	Loki rejects only above the limit; a query with exactly the limit now passes

Start/end step alignment, the sixth review item, is not here: #574 covers it behind -align-queries-with-step, matching Loki's own knob.

Review round 2

Every item from the review is addressed:

1. Proxy-internal text out of the client body — the message ends at topk(); writeError adds limit_flag=-max-stats-query-series to the log line instead, and the three tests that pinned the suffix now assert its absence.
2. Instant queries guarded — handleQuery attaches the scope and proxyStatsQuery applies the limit; a unit subtest covers both branches (plain 400, Drilldown 200 + warning + capped vector).
3. Drilldown-only silent cuts closed — the batcher, the histogram, hybrid and two-phase paths all report through capSeriesToLimit, and the VictoriaLogs-side | limit is limit+1 so an exact-fit answer is distinguishable from a truncated one. The batched path cuts per request, so each caller's response carries its own warning.
4. No hardcoded cap — maxDrilldownSeries is gone; every Drilldown path reads -max-stats-query-series, so raising it to 2000 yields 2000 and a warning that says 2000.
5. Conflict resolved as advised (four return values), with the off-by-one fixed in the same edit.
6. CI exercises the new behaviour — nine compose proxies carry their Loki's limit; loki-vl-proxy-patterns-autodetect keeps the default 500 and the e2e limit subtests point at it (PROXY_SERIES_LIMIT_URL). The Drilldown partial subtest runs rather than skips.
7. Stale docs and Helm lines updated — configuration, operations, api-reference, real-window-compatibility-gaps, KNOWN_ISSUES, compatibility-drilldown and charts/.../values.yaml describe the 400 and the Drilldown warning; the version-gate paragraphs describe the new unknown-version behaviour.

Minors: the rewriter is now scoped to metric expressions (log queries stream); shiftLokiResultTimestamps writes status, warnings, data in Loki's order and keeps unknown keys under data; addSeriesLimitWarning parses for a top-level warnings key instead of scanning bytes; the busiest-vs-first difference is documented. seriesCountingReader still counts VictoriaLogs result entries (two per series when byte presence rides along, which it accounts for) — it is an early abort ahead of the exact proxy-side count, so a query within the limit is never rejected by it.

Related finding (byte cap): with the series limit raised, sum by (pod) (rate({service_name="api-gateway"}[5m])) over 24h and 7d answers 200 on this stack (1.7 MB at 7d), so the 64 MB manual cap does not bite here; making that cap an operator flag belongs to #585.

Conformance registry

Item	Kind	State	Served by	Tests declaring it
loki_api_v1_query	endpoint	proven	hybrid	2
loki_api_v1_query_range	endpoint	proven	hybrid	3
explore-logs-volume-json	translation	partial	—	none

behaviours/series-limits-and-partial-results and behaviours/window-bounds-and-step-alignment move from gap to partial, with three new case files — limits/series-limit-error, limits/drilldown-partial-with-warning, semantics/offset-sample-timestamps — declared by the unit and e2e tests through // conformance: markers. explore-logs-volume-json is touched only because this change edits files it lists; it is not part of this PR's contract. python3 scripts/ci/check_conformance.py passes.

Verification

• go test -race ./internal/..., go vet ./... (+ -tags=e2e), gofmt, golangci-lint (0 issues), changelog gate, conformance gate.
• e2e against live Loki on the compose stack, with two proxies from this branch — one carrying the stack Loki's limit, one at the default 500 — mirroring the compose arrangement: TestRangeMetricCompatibility(TumblingAndShortWindows|Sliding*|GroupedTumblingLongRange|BareParserWindows|EmptyWindowAggregations|OffsetAndSeriesLimit) all pass, including the Drilldown partial subtest (500 series, each equal to Loki's, plus the warning) and offset stamping on range and instant queries.
• Grafana-shaped range matrix 5m–7d (logs, grouped count/rate, bytes, sliding count, topk, volume, instant), sequential: no blockers — every panel 200 where Loki answers 200 (json metric grouped is 400 on both), and at 7d the proxy answers in 0.3–1.2s where Loki takes 16–107s. /labels 200 throughout. No load test.

Rebased onto main after #590, #591, #592, #593 and #595. The CHANGELOG was resolved by hand: the released sections are byte-identical to main and only the [Unreleased] / ### Fixed section carries this PR's entries (a plain rebase had dropped them into the 1.83.0 section). Two tests needed follow-up for main's newer topk route, which buckets inside the query (stats by (_time:… offset …)) instead of using the request offset, and for the service_name derivation landed by #579; both are updated and the whole unit suite, both vet builds, gofmt, golangci-lint, the changelog gate and scripts/ci/check_conformance.py pass on the rebased commit. The e2e semantics cases and the range matrix were re-run live against this commit with the same two-proxy arrangement.

behaviours/series-limits-and-partial-results now carries both #590's cases and this PR's in one state file, and python3 scripts/ci/check_conformance.py passes with every declared id backed by a case file.

One matrix pass hit a 502 on count by (level) at 7d while the shared VictoriaLogs container was restarted from another session mid-run (exit code 0, not OOM); the query returns 200 in 0.3–0.4s on retry and the following full pass was clean, with RestartCount unchanged (18 → 18) and the container up throughout.

[github-actions]

PR Quality Report

Compared against base branch main.

Coverage and tests

Signal	Base	PR	Delta
Test count	7387	7398	11
Coverage	90.1%	89.9%	-0.2% (regressed)

Compatibility

Track	Base	PR	Delta
Loki API	100.0%	11/11 (100.0%)	0.0% (stable)
Logs Drilldown	100.0%	17/17 (100.0%)	0.0% (stable)
VictoriaLogs	100.0%	11/11 (100.0%)	0.0% (stable)

Performance smoke

Lower CPU cost (ns/op) is better. Lower benchmark memory cost (B/op, allocs/op) is better. Higher throughput is better. Lower load-test memory growth is better. Benchmark rows are medians from repeated samples.

Signal	Base	PR	Delta
QueryRange cache-hit CPU cost	2029.0 ns/op	2009.0 ns/op	-1.0% (stable)
QueryRange cache-hit memory	1240.0 B/op	1240.0 B/op	0.0% (stable)
QueryRange cache-hit allocations	27.0 allocs/op	27.0 allocs/op	0.0% (stable)
QueryRange cache-bypass CPU cost	2252.0 ns/op	2354.0 ns/op	+4.5% (stable)
QueryRange cache-bypass memory	1280.0 B/op	1282.0 B/op	+0.2% (stable)
QueryRange cache-bypass allocations	27.0 allocs/op	27.0 allocs/op	0.0% (stable)
Labels cache-hit CPU cost	568.0 ns/op	574.4 ns/op	+1.1% (stable)
Labels cache-hit memory	48.0 B/op	48.0 B/op	0.0% (stable)
Labels cache-hit allocations	3.0 allocs/op	3.0 allocs/op	0.0% (stable)
Labels cache-bypass CPU cost	926.3 ns/op	941.4 ns/op	+1.6% (stable)
Labels cache-bypass memory	391.0 B/op	391.0 B/op	0.0% (stable)
Labels cache-bypass allocations	14.0 allocs/op	14.0 allocs/op	0.0% (stable)

State

• Coverage, compatibility, and sampled performance are reported here from the same PR workflow.
• This is a delta report, not a release gate by itself. Required checks still decide merge safety.
• Performance is a smoke comparison, not a full benchmark lab run.
• Delta states use the same noise guards as the quality gate (percent + absolute + low-baseline checks), so report labels match merge-gate behavior.