Skip to content

fix: count, min, max and avg read one value per Loki series - #582

Merged
szibis merged 1 commit into
mainfrom
fix/series-count-aggregations
Sep 23, 2026
Merged

szibis merged 1 commit into
mainfrom
fix/series-count-aggregations

Conversation

@szibis

@szibis szibis commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

count, min, max and avg of a range aggregation read one value per Loki series. They were translated to the same single stats pipe as sum, so they returned line counts:

  • count by (pod) (count_over_time({app="worker-service"}[1m])): 15 per pod, Loki answers 1.
  • count by (service_name) (count_over_time({namespace="data"}[15m])): 9,768, Loki answers 900.
  • max by (service_name) (count_over_time({namespace="data"}[15m])): the same sums (6,660 / 3,108) where Loki answers 15 / 7.

The inner stats pipe now keeps the series identity (the requested labels plus the stream, or the parsed line when the query has a parser stage) and an outer stats pipe aggregates those per-series values, for count_over_time, bytes_over_time, rate, bytes_rate and the unwrap functions. sum keeps its single pipe: summing the values of a group's series is the same as aggregating their rows. Both pipes run in VictoriaLogs, so no rows are buffered.

Verification

  • go test ./internal/... ./cmd/..., go vet and golangci-lint (v2.13.2, Go 1.27.1) are clean. New translator tests pin the emitted LogsQL for every shape, including sum, which must keep the single pipe.

  • New e2e-compat test TestCompat_SeriesCountAggregations (in the core shard) pushes three streams with 5/3/1 lines to Loki and VictoriaLogs and compares count by (pod), count, min, max, avg, max by (pod) and both sum forms against Loki's own answers. It passes on this build and fails on main.

  • Live, against Loki direct (instant, same evaluation time):

    query Loki main this build
    count by (pod) (count_over_time({app="worker-service"}[1m])) 30 series, total 30 total 450 total 30
    count by (service_name) (count_over_time({namespace="data"}[15m])) 450 / 450 6,660 / 3,108 444 / 444
    max by (service_name) (count_over_time({namespace="data"}[15m])) 15 / 7 6,660 / 3,108 15 / 7
    count(count_over_time({app="worker-service"}[1m])) 30 450 30
    min by (pod) (count_over_time({app="worker-service"}[1m])) 450 450 450

    (444 vs 450 is the stream visibility lag between the two backends at that instant, not a translation difference.)

Grouping sweep (15 query shapes, instant and range, 15m/1h/6h/24h)

Against a build of this branch's base (main at 7fb87b2). VictoriaLogs RestartCount 3 before and 3 after.

range queries exact match with Loki (main) exact (this build) grouping labels match (main) labels (this build)
15m 46 41 43 43 43
1h 46 40 43 42 43
6h 46 33 35 35 35
24h 46 33 34 35 34

Every difference is count by (service_name) (count_over_time(…)) becoming exact, plus topk(3, …) tie flips in both directions (places 3-5 are tied, so membership is arbitrary on both sides).

without (...)

count, min, max and avg with without (...) were summed by the response-side merge whatever the operator, so max without (pod) (count_over_time({app="worker-service"}[1m])) returned 450 (the line count) where Loki answers 15, and count without (pod) 30 where Loki answers 1. The outer stats pipe now keeps the identity and the merge folds each remaining label set with the query's operator. Live, both match Loki exactly for that app (count 30, max 15, avg 15, sum unchanged), and the e2e test covers all five operators.

Known limits (now stated correctly in the CHANGELOG and the tests)

The proxy's series identity is the stream and its level. Loki also splits a stream per distinct structured-metadata set and per set of labels a | json or | logfmt stage extracted, so it counts more series: count(count_over_time({app="worker-service"} | logfmt [1m])) is 450 on Loki and 30 here, and count without (pod) over a namespace whose lines carry a per-line trace_id counts streams rather than lines (858 series on Loki, 30 here). The earlier claim that "a parser stage makes the line the identity" was wrong and is gone from both the CHANGELOG and series_aggregation_test.go. #587 adds the | regexp / | pattern half of that identity, which is the part VictoriaLogs can group by.

Conformance registry

Item Kind State Served by Tests declaring it
metric-series-identity behaviour partial — 2

metric-series-identity is new: it records the per-series contract, the structured-metadata and dynamic-key-parser deviations as an owner decision with the waiver metric-identity-metadata-and-dynamic-keys (expiry 2027-03-31), and the measured numbers. python3 scripts/ci/check_conformance.py passes.

Other review points

  • count by (x) (rate(...)) no longer emits the | math … as __lvp_rate pipe nothing read.
  • addByClauseLast documents that it must run before any marker is appended, which is the invariant its LastIndex search depends on.
  • The e2e test now waits for both sides to hold the whole fixture before comparing, so a VictoriaLogs ingestion lag cannot be read as a mismatch.

@github-actions github-actions Bot added size/L Large change scope/translator LogQL translator scope/docs Documentation scope/tests Tests bugfix Bug fix labels Sep 22, 2026
@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

PR Quality Report

Compared against base branch main.

Coverage and tests

Signal Base PR Delta
Test count 7398 7400 2
Coverage 89.9% 89.9% 0.0% (stable)

Compatibility

Track Base PR Delta
Loki API 100.0% 11/11 (100.0%) 0.0% (stable)
Logs Drilldown 100.0% 17/17 (100.0%) 0.0% (stable)
VictoriaLogs 100.0% 11/11 (100.0%) 0.0% (stable)

Performance smoke

Lower CPU cost (ns/op) is better. Lower benchmark memory cost (B/op, allocs/op) is better. Higher throughput is better. Lower load-test memory growth is better. Benchmark rows are medians from repeated samples.

Signal Base PR Delta
QueryRange cache-hit CPU cost 2183.0 ns/op 2123.0 ns/op -2.7% (stable)
QueryRange cache-hit memory 1240.0 B/op 1240.0 B/op 0.0% (stable)
QueryRange cache-hit allocations 27.0 allocs/op 27.0 allocs/op 0.0% (stable)
QueryRange cache-bypass CPU cost 2803.0 ns/op 2869.0 ns/op +2.4% (stable)
QueryRange cache-bypass memory 1283.0 B/op 1284.0 B/op +0.1% (stable)
QueryRange cache-bypass allocations 27.0 allocs/op 27.0 allocs/op 0.0% (stable)
Labels cache-hit CPU cost 622.3 ns/op 625.3 ns/op +0.5% (stable)
Labels cache-hit memory 48.0 B/op 48.0 B/op 0.0% (stable)
Labels cache-hit allocations 3.0 allocs/op 3.0 allocs/op 0.0% (stable)
Labels cache-bypass CPU cost 1048.0 ns/op 1075.0 ns/op +2.6% (stable)
Labels cache-bypass memory 391.0 B/op 391.0 B/op 0.0% (stable)
Labels cache-bypass allocations 14.0 allocs/op 14.0 allocs/op 0.0% (stable)

State

  • Coverage, compatibility, and sampled performance are reported here from the same PR workflow.
  • This is a delta report, not a release gate by itself. Required checks still decide merge safety.
  • Performance is a smoke comparison, not a full benchmark lab run.
  • Delta states use the same noise guards as the quality gate (percent + absolute + low-baseline checks), so report labels match merge-gate behavior.

@szibis
szibis force-pushed the fix/series-count-aggregations branch from dfc3d9b to 809d8f4 Compare September 22, 2026 09:09
@github-actions github-actions Bot added size/L Large change and removed size/L Large change labels Sep 22, 2026
@szibis
szibis force-pushed the fix/series-count-aggregations branch from 809d8f4 to 0100302 Compare September 22, 2026 09:39
@github-actions github-actions Bot removed the size/L Large change label Sep 22, 2026
@szibis
szibis force-pushed the fix/series-count-aggregations branch from 0100302 to 300b508 Compare September 22, 2026 09:39
@github-actions github-actions Bot added size/L Large change and removed size/L Large change labels Sep 22, 2026
@szibis
szibis force-pushed the fix/series-count-aggregations branch from 300b508 to 0ba4328 Compare September 22, 2026 10:01
@github-actions github-actions Bot added size/XL Extra large change scope/proxy Proxy core and removed size/L Large change labels Sep 22, 2026
@szibis
szibis force-pushed the fix/series-count-aggregations branch from 0ba4328 to 1046731 Compare September 22, 2026 11:28
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Sep 22, 2026
@szibis
szibis force-pushed the fix/series-count-aggregations branch from 1046731 to b7acd53 Compare September 22, 2026 11:41
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Sep 22, 2026
@szibis
szibis force-pushed the fix/series-count-aggregations branch from b7acd53 to 86aa37c Compare September 22, 2026 13:25
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Sep 22, 2026
@szibis
szibis force-pushed the fix/series-count-aggregations branch from 86aa37c to 52162af Compare September 22, 2026 14:03
@github-actions github-actions Bot removed the size/XL Extra large change label Sep 22, 2026
@github-actions github-actions Bot added the size/XL Extra large change label Sep 22, 2026
@szibis
szibis force-pushed the fix/series-count-aggregations branch from 52162af to e260af5 Compare September 23, 2026 08:52
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Sep 23, 2026
@szibis
szibis force-pushed the fix/series-count-aggregations branch from e260af5 to a5dbfb2 Compare September 23, 2026 09:14
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Sep 23, 2026
count/min/max/avg of a range aggregation were translated to the same single
stats pipe as sum, so count by (pod) (count_over_time({app="x"}[1m]))
returned each pod's line count (30 where Loki answers 1) and max/avg
returned those sums.

The inner stats pipe now keeps the series identity (the requested labels
plus the stream, or the parsed line when the query has a parser stage) and
an outer stats pipe aggregates the per-series values, for count_over_time,
bytes_over_time, rate, bytes_rate and the unwrap functions. sum keeps its
single pipe: summing the values of a group's series is the same as
aggregating their rows. Both pipes run in VictoriaLogs.
@szibis
szibis force-pushed the fix/series-count-aggregations branch from a5dbfb2 to 8921d0f Compare September 23, 2026 09:24
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Sep 23, 2026
@szibis
szibis merged commit 4ce1412 into main Sep 23, 2026
51 of 54 checks passed
@szibis
szibis deleted the fix/series-count-aggregations branch September 23, 2026 09:41
szibis added a commit that referenced this pull request Sep 23, 2026
#608)

#582 merged after v1.89.0 was cut, from a head that predated the cut, and
the merge placed its two entries inside the released 1.89.0 section.
Restore 1.89.0 to what the release published and list the entries under
Unreleased, where the next release will pick them up.

Co-authored-by: szibis <szibis@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Bug fix scope/docs Documentation scope/proxy Proxy core scope/tests Tests scope/translator LogQL translator size/XL Extra large change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant