fix: answer metric queries over the series limit like Loki instead of cutting them silently

CHANGELOG.md

@@ -36,6 +36,66 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- Fail a metric query over the series limit the way Loki does, instead of
+  returning part of the data as if it were all of it. A grouped range metric
+  whose grouping has more values than `-max-stats-query-series` (500 by
+  default) was answered with a silent cut — the busiest 200 values on the
+  short-range path, the busiest 200 on the two-phase path used from two hours
+  on — so a chart or an alert read a subset as the whole answer. Loki answers
+  `maximum number of series (N) reached for a single query` with HTTP 400 for
+  such a query and only gives Grafana Logs Drilldown a partial result. The
+  proxy now does the same: the error names the limit and the
+  `-max-stats-query-series` flag that sets it, and Drilldown requests
+  (`X-Query-Tags: Source=grafana-lokiexplore-app`) get the busiest series plus
+  Loki's `returning partial results` warning in the response. The limit is
+  applied while the VictoriaLogs response is read, so an oversized answer is
+  no longer buffered in full, and it is never turned into a raw log scan. The
+  message stays Loki's word for word, since Grafana renders it; the proxy's log
+  line for the rejected request names the flag instead. Instant queries
+  (`/loki/api/v1/query`) follow the same rule, as they do in Loki. The Grafana
+  Logs Drilldown paths no longer carry a hardcoded 500: they read the flag, ask
+  VictoriaLogs for one series over it so a truncated answer is distinguishable
+  from an exact fit, and report the cut as the warning instead of dropping
+  series silently. Where Loki keeps the first series it encounters, the proxy
+  keeps the busiest by total count — the ones a Drilldown panel renders. The
+  e2e compose proxies now run with the series limit their Loki is configured
+  with, so a parity test compares two answers rather than two limits; one
+  variant keeps the default so the limit itself stays under test.
+- Stamp `offset` samples at the evaluation time. `count_over_time({app="x"}[5m]
+  offset 5m)` evaluates the window `(T-10m, T-5m]` but Loki reports it at `T`;
+  the proxy reported it at `T-5m`, shifting the whole series left by the
+  offset on range and instant queries alike (and caching it that way).
+- Keep evaluating range metrics on VictoriaLogs buckets while the backend
+  version is unknown. A failed or pending startup version probe made the proxy
+  assume the `stats_query_range` `offset` arg (VictoriaLogs v1.45+) is
+  unavailable, so any range whose start was not aligned to the window fell back
+  to fetching raw log rows — the amount of data read then scaled with the logs
+  stored, not with the query. The arg is now sent while the version is unknown
+  (a backend that does not know it ignores it, leaving the epoch-aligned
+  buckets it would have had anyway) and the version probe keeps retrying in the
+  background. A release known to be older than v1.45 still uses epoch-aligned
+  buckets only where they match the query's windows.
+- Count Logs Drilldown's short ranges over Loki's window. A Drilldown panel
+  with a range shorter than the step (`[1m]` at step 300) was answered from the
+  whole step bucket, so a panel showed 30 where Loki answers 6. Drilldown now
+  uses the same window evaluator as every other client for those panels.
+- Return exact series for high-cardinality groupings outside Drilldown. A
+  Grafana dashboard or Explore panel grouping by a high-cardinality field
+  (`sum by (trace_id) (...)`) was rewritten to the window-sampled `/hits` path,
+  which returns a bounded sample of the values rather than the series Loki
+  returns. That rewrite is now limited to Logs Drilldown, whose panels are
+  built for it.
+- Fit the top-value `in()` filter to VictoriaLogs' query length. The bounded
+  top-N paths cut the value list at a fixed count, which could still exceed
+  VictoriaLogs' default 16 KiB query limit (long pod or trace values) and be
+  rejected, or — after filtering — leave an empty `in()` that matches nothing.
+  The list is now budgeted against the actual query text and keeps at least the
+  busiest value.
+- Read a low-cardinality grouping once. Single-field groupings over two hours
+  ran a ranking query and then a full bucket query over the same range, so an
+  ordinary `sum by (namespace) (count_over_time(...))` scanned the range twice.
+  The bucket query now runs alone, and values are ranked only when the result
+  exceeds the series limit.
 - **Release metadata pull requests pass the changelog gate again.** The gate
   listed a pull request's files with a two-dot diff against the current base,
   so a branch cut before the base moved appeared to change every file merged

charts/loki-vl-proxy/values.yaml

@@ -110,7 +110,7 @@ extraArgs:
   # query-range-background-warm: "true"                # continue warming failed windows in background
   # query-range-background-warm-max-windows: "24"      # cap background warm fanout
   # default-max-query-length: "0"                      # max query time range for all tenants (0 = unlimited; per-tenant limits override)
-  # max-stats-query-series: "500"                       # max series returned by stats metric queries (count_over_time, rate, bytes_rate); 0 = built-in default of 500; proxy-evaluated metric paths return HTTP 502/503 instead of truncating
+  # max-stats-query-series: "500"                       # max series a metric query may return (count_over_time, rate, bytes_rate); 0 = built-in default of 500, Loki's own max_query_series default; above it the query fails with Loki's HTTP 400 "maximum number of series (N) reached for a single query" (Logs Drilldown gets a partial result with a warning)
   # stats-query-range-concurrency: "4"                  # max concurrent stats_query_range calls to VL; higher = more VL CPU pressure; 0 = built-in default of 4
   # drilldown-burst-window-ms: "50"
   # drilldown-burst-max-fields: "30"

conformance/registry/cases/limits/drilldown-partial-with-warning.yaml

@@ -0,0 +1,39 @@
+id: limits/drilldown-partial-with-warning
+title: Logs Drilldown gets a partial result and Loki's warning, not an error
+proves:
+  - series-limits-and-partial-results
+  - loki_api_v1_query_range
+  - loki_api_v1_query
+consumer: drilldown
+why: |
+  Loki keeps Logs Drilldown rendering when a query exceeds the series limit: it
+  returns the series it has with the warning "maximum number of series (N)
+  reached for a single query; returning partial results" in the response body,
+  where the plugin surfaces it. A partial result with no warning is the failure
+  this case guards against — the panel then looks complete.
+request:
+  method: GET
+  path: /loki/api/v1/query_range
+  params:
+    query: 'sum by (pod) (count_over_time({app="wide"}[1m]))'
+    start: "<start>"
+    end: "<start+30m>"
+    step: "300"
+  headers:
+    X-Scope-OrgID: "0"
+    X-Query-Tags: Source=grafana-lokiexplore-app
+compare:
+  mode: structural
+  fields: [status, warnings, data.resultType, data.result[].metric]
+  values:
+    mode: exact
+    field: data.result[].values[][1]
+    reason: |
+      Every returned series must equal Loki's for the same labels: the result is
+      partial in how many series it holds, never in what each one says. The proxy
+      keeps the busiest series by total count where Loki keeps the first series it
+      encounters, so the set may differ; each kept series is compared against
+      Loki's answer for the same label set.
+applies_to:
+  loki: ">=3.0.0"
+  victorialogs: ">=v1.44.0"

conformance/registry/cases/limits/series-limit-error.yaml

@@ -0,0 +1,40 @@
+id: limits/series-limit-error
+title: A metric query above the series limit fails with Loki's message, not a silent cut
+proves:
+  - series-limits-and-partial-results
+  - loki_api_v1_query_range
+  - loki_api_v1_query
+consumer: api
+why: |
+  Above `max_query_series` Loki rejects the query rather than answering with part
+  of the data: a chart or an alert that reads a truncated answer as the whole one
+  is wrong in a way nothing on screen reveals. The proxy's limit is
+  `-max-stats-query-series` (default 500, Loki's own default), and the message
+  stays Loki's word for word because Grafana renders it; the flag that set the
+  limit is named in the proxy's log line instead.
+request:
+  method: GET
+  path: /loki/api/v1/query_range
+  params:
+    query: 'sum by (pod) (count_over_time({app="wide"}[1m]))'
+    start: "<start>"
+    end: "<start+30m>"
+    step: "300"
+  headers:
+    X-Scope-OrgID: "0"
+compare:
+  mode: structural
+  fields: [status, errorType, error]
+  values:
+    mode: exact
+    reason: |
+      Loki's text is the contract: "maximum number of series (N) reached for a
+      single query; consider reducing query cardinality by adding more specific
+      stream selectors, reducing the time range, or aggregating results with
+      functions like sum(), count() or topk()", HTTP 400, errorType bad_data. The
+      comparison runs with both sides configured to the same limit; the e2e case
+      reads the proxy's limit out of its own answer and compares against a Loki
+      configured above the fixture's cardinality.
+applies_to:
+  loki: ">=3.0.0"
+  victorialogs: ">=v1.44.0"

conformance/registry/cases/semantics/offset-sample-timestamps.yaml

@@ -0,0 +1,35 @@
+id: semantics/offset-sample-timestamps
+title: An offset modifier moves the window, not the sample timestamps
+proves:
+  - window-bounds-and-step-alignment
+  - loki_api_v1_query_range
+  - loki_api_v1_query
+consumer: explore
+why: |
+  `count_over_time({app="x"}[5m] offset 5m)` evaluates the window (T-10m, T-5m]
+  but Loki reports the sample at T. Stamping it at T-5m shifts the whole series
+  left by the offset, which on a dashboard reads as data that moved in time
+  rather than a query that looked further back.
+request:
+  method: GET
+  path: /loki/api/v1/query_range
+  params:
+    query: 'sum by (pod) (count_over_time({app="tick"}[5m] offset 5m))'
+    start: "<start>"
+    end: "<start+30m>"
+    step: "300"
+  headers:
+    X-Scope-OrgID: "0"
+compare:
+  mode: structural
+  fields: [status, data.resultType, data.result[].metric, data.result[].values[][0]]
+  values:
+    mode: exact
+    field: data.result[].values[][1]
+    reason: |
+      Timestamps and values must both match Loki exactly: the sample at T holds
+      the count of the window one offset earlier. The instant form of the same
+      query is compared the same way.
+applies_to:
+  loki: ">=3.0.0"
+  victorialogs: ">=v1.44.0"

conformance/registry/generated/wiring.json

@@ -16,6 +16,10 @@
       "internal/proxy/sliding_window_stats_test.go",
       "test/e2e-compat/heavy_query_protection_test.go"
     ],
+    "limits/drilldown-partial-with-warning": [
+      "internal/proxy/tumbling_axis_parity_test.go",
+      "test/e2e-compat/range_metric_offset_series_limit_test.go"
+    ],
     "limits/drilldown-stats-rejection-keeps-429": [
       "internal/proxy/backend_heavy_test.go",
       "test/e2e-compat/heavy_query_protection_test.go"
@@ -50,6 +54,10 @@
     "limits/raw-metric-row-precheck": [
       "internal/proxy/backend_heavy_test.go"
     ],
+    "limits/series-limit-error": [
+      "internal/proxy/tumbling_axis_parity_test.go",
+      "test/e2e-compat/range_metric_offset_series_limit_test.go"
+    ],
     "limits/stats-row-limit-per-grid": [
       "internal/proxy/sliding_window_stats_test.go"
     ],
@@ -67,11 +75,17 @@
     "loki_api_v1_labels": [
       "internal/proxy/backend_heavy_test.go"
     ],
+    "loki_api_v1_query": [
+      "internal/proxy/tumbling_axis_parity_test.go",
+      "test/e2e-compat/range_metric_offset_series_limit_test.go"
+    ],
     "loki_api_v1_query_range": [
       "internal/proxy/backend_heavy_test.go",
       "internal/proxy/metadata_query_validation_test.go",
+      "internal/proxy/tumbling_axis_parity_test.go",
       "internal/translator/regexp_escaping_test.go",
-      "test/e2e-compat/heavy_query_protection_test.go"
+      "test/e2e-compat/heavy_query_protection_test.go",
+      "test/e2e-compat/range_metric_offset_series_limit_test.go"
     ],
     "operator-configurable-limits": [
       "cmd/proxy/main_test.go",
@@ -85,6 +99,10 @@
     "semantics/client-cancellation-propagates": [
       "internal/proxy/backend_heavy_test.go"
     ],
+    "semantics/offset-sample-timestamps": [
+      "internal/proxy/tumbling_axis_parity_test.go",
+      "test/e2e-compat/range_metric_offset_series_limit_test.go"
+    ],
     "semantics/sliding-window-recurrence-exact": [
       "internal/proxy/sliding_window_stats_test.go",
       "test/e2e-compat/heavy_query_protection_test.go"
@@ -95,7 +113,9 @@
     "series-limits-and-partial-results": [
       "internal/proxy/backend_heavy_test.go",
       "internal/proxy/sliding_window_stats_test.go",
-      "test/e2e-compat/heavy_query_protection_test.go"
+      "internal/proxy/tumbling_axis_parity_test.go",
+      "test/e2e-compat/heavy_query_protection_test.go",
+      "test/e2e-compat/range_metric_offset_series_limit_test.go"
     ],
     "severity-detected-level-derivation": [
       "internal/proxy/detected_level_test.go",
@@ -115,10 +135,12 @@
       "internal/proxy/volume_test.go"
     ],
     "window-bounds-and-step-alignment": [
-      "internal/proxy/sliding_window_stats_test.go"
+      "internal/proxy/sliding_window_stats_test.go",
+      "internal/proxy/tumbling_axis_parity_test.go",
+      "test/e2e-compat/range_metric_offset_series_limit_test.go"
     ]
   },
-  "registry_items": 159,
+  "registry_items": 162,
   "routes_without_registry_item": [],
   "test_to_items": {
     "cmd/proxy/main_test.go": [
@@ -171,6 +193,15 @@
       "series-limits-and-partial-results",
       "window-bounds-and-step-alignment"
     ],
+    "internal/proxy/tumbling_axis_parity_test.go": [
+      "limits/drilldown-partial-with-warning",
+      "limits/series-limit-error",
+      "loki_api_v1_query",
+      "loki_api_v1_query_range",
+      "semantics/offset-sample-timestamps",
+      "series-limits-and-partial-results",
+      "window-bounds-and-step-alignment"
+    ],
     "internal/proxy/volume_test.go": [
       "drilldown/index_volume/single-sample-returns-vector",
       "loki_api_v1_index_volume",
@@ -204,9 +235,18 @@
       "loki_api_v1_query_range",
       "semantics/sliding-window-recurrence-exact",
       "series-limits-and-partial-results"
+    ],
+    "test/e2e-compat/range_metric_offset_series_limit_test.go": [
+      "limits/drilldown-partial-with-warning",
+      "limits/series-limit-error",
+      "loki_api_v1_query",
+      "loki_api_v1_query_range",
+      "semantics/offset-sample-timestamps",
+      "series-limits-and-partial-results",
+      "window-bounds-and-step-alignment"
     ]
   },
-  "tests_with_markers": 15,
+  "tests_with_markers": 17,
   "unknown_ids": {},
   "unwired_items": [
     "api_prom_label",
@@ -234,7 +274,6 @@
     "loki_api_v1_label",
     "loki_api_v1_patterns",
     "loki_api_v1_push",
-    "loki_api_v1_query",
     "loki_api_v1_rules",
     "loki_api_v1_rules_namespace",
     "loki_api_v1_rules_namespace_groupname",
@@ -247,5 +286,5 @@
     "prometheus_api_v1_rules",
     "service-name-derivation"
   ],
-  "wired_items": 31
+  "wired_items": 35
 }

conformance/registry/state/behaviours/series-limits-and-partial-results.yaml

@@ -1,14 +1,21 @@
 id: series-limits-and-partial-results
 state: partial
 reason: |
+  limits/series-limit-error and limits/drilldown-partial-with-warning are written
+  and wired: a metric query above -max-stats-query-series fails with Loki's
+  message and status on every metric path, range and instant, and a Logs
+  Drilldown request gets a partial result with Loki's partial-result warning,
+  compared against live Loki by unit tests and an e2e case.
   limits/top-n-ranking-not-silent is written and wired: topk and bottomk over
   grouped counts now rank every series per step with bounded per-step heaps
   instead of the -max-stats-query-series busiest, proven by unit tests against a
   VictoriaLogs stats-pipe emulator and by a Loki comparison in
   test/e2e-compat/heavy_query_protection_test.go.
   limits/drilldown-stats-rejection-keeps-429 pins the boundary of the Drilldown
   partial-result carve-out: it covers the series limit, not backpressure.
-  limits/series-limit-error, limits/drilldown-partial-with-warning and
-  limits/resolution-limit-precedence are still named but not written.
+  Still open: limits/resolution-limit-precedence (the 11,000-point rejection
+  ahead of parsing) is named but not written, and the Drilldown partial keeps the
+  busiest series where Loki keeps the first it encounters, so the kept set can
+  differ from Loki's for equal-volume series.
 owner_decision: null
 waiver: null

conformance/registry/state/behaviours/window-bounds-and-step-alignment.yaml

@@ -1,6 +1,12 @@
 id: window-bounds-and-step-alignment
-state: gap
+state: partial
 reason: |
-  Seeded from behaviour we have hit in production use; cases are named but not yet written or wired.
+  semantics/offset-sample-timestamps is written and wired: an offset modifier
+  moves the window and leaves the sample timestamps on Loki's grid, on range and
+  instant queries, proven by unit tests and an e2e case against live Loki. The
+  remaining cases named by this behaviour — lower-bound-exclusive, step-alignment,
+  tumbling-relabel, range-smaller-than-step, grafana-24h-split and
+  resolution-limit-11000 — have no case files yet; step alignment itself is
+  carried by the -align-queries-with-step work, tracked separately.
 owner_decision: null
 waiver: null

conformance/reports/gaps.md

@@ -9,7 +9,6 @@ compares the endpoint against Loki, plus 2 when the proxy computes the result it
 |---:|---|---|---|---:|---:|---|
 | 12 | `/loki/api/v1/label/{name}/values` | explore, drilldown, datasource, api | native_vl | 0 | 0 | no test compares it against Loki |
 | 9 | `/loki/api/v1/detected_fields` | explore, drilldown, api | proxy_side | 29 | 9 | no test wired to the registry; proxy-side: justify in the registry or push down to VictoriaLogs |
-| 7 | `/loki/api/v1/query` | explore, drilldown, api | hybrid | 131 | 34 | no test wired to the registry |
 | 7 | `/loki/api/v1/detected_field/{name}/values` | drilldown, api | native_vl | 0 | 0 | no test compares it against Loki; no test wired to the registry |
 | 6 | `/loki/api/v1/series` | explore, datasource, api | hybrid | 29 | 8 | no test wired to the registry |
 | 6 | `/loki/api/v1/index/volume_range` | drilldown, api | proxy_side | 22 | 4 | no test wired to the registry; proxy-side: justify in the registry or push down to VictoriaLogs |
@@ -72,11 +71,11 @@ Semantics, severity, identity and data-quality behaviour the proxy must reproduc
 | semantics | `numeric-and-response-formatting` — Timestamps, number formatting and empty shapes | 5 | 0 | gap |
 | limits | `operator-configurable-limits` — Every bound on work is an operator flag, documented from one source | 11 | 4 | proven |
 | semantics | `parser-error-and-label-collision` — Parser errors, __error__ and _extracted collisions | 7 | 0 | gap |
-| limits | `series-limits-and-partial-results` — Series limits: error, or partial result with a warning | 7 | 3 | partial |
+| limits | `series-limits-and-partial-results` — Series limits: error, or partial result with a warning | 7 | 5 | partial |
 | identity | `service-name-derivation` — service_name follows Loki's discovery order | 8 | 0 | partial |
 | severity | `severity-detected-level-derivation` — detected_level is derived on the read path | 8 | 4 | partial |
 | severity | `severity-exposure-surfaces` — Where detected_level and level must appear | 6 | 3 | partial |
-| semantics | `window-bounds-and-step-alignment` — Range windows, bucket edges and step alignment | 7 | 1 | gap |
+| semantics | `window-bounds-and-step-alignment` — Range windows, bucket edges and step alignment | 7 | 3 | partial |
 
 ## LogQL surface (v3.7.7)