Repository navigation
fix: count, min, max and avg read one value per Loki series - #582
Merged
Merged
Conversation
Contributor
PR Quality ReportCompared against base branch Coverage and tests
Compatibility
Performance smokeLower CPU cost (
State
|
szibis
force-pushed
the
fix/series-count-aggregations
branch
from
September 22, 2026 09:09
dfc3d9b to
809d8f4
Compare
szibis
force-pushed
the
fix/series-count-aggregations
branch
from
September 22, 2026 09:39
809d8f4 to
0100302
Compare
szibis
force-pushed
the
fix/series-count-aggregations
branch
from
September 22, 2026 09:39
0100302 to
300b508
Compare
szibis
force-pushed
the
fix/series-count-aggregations
branch
from
September 22, 2026 10:01
300b508 to
0ba4328
Compare
szibis
force-pushed
the
fix/series-count-aggregations
branch
from
September 22, 2026 11:28
0ba4328 to
1046731
Compare
szibis
force-pushed
the
fix/series-count-aggregations
branch
from
September 22, 2026 11:41
1046731 to
b7acd53
Compare
szibis
force-pushed
the
fix/series-count-aggregations
branch
from
September 22, 2026 13:25
b7acd53 to
86aa37c
Compare
szibis
force-pushed
the
fix/series-count-aggregations
branch
from
September 22, 2026 14:03
86aa37c to
52162af
Compare
szibis
force-pushed
the
fix/series-count-aggregations
branch
from
September 23, 2026 08:52
52162af to
e260af5
Compare
szibis
force-pushed
the
fix/series-count-aggregations
branch
from
September 23, 2026 09:14
e260af5 to
a5dbfb2
Compare
count/min/max/avg of a range aggregation were translated to the same single
stats pipe as sum, so count by (pod) (count_over_time({app="x"}[1m]))
returned each pod's line count (30 where Loki answers 1) and max/avg
returned those sums.
The inner stats pipe now keeps the series identity (the requested labels
plus the stream, or the parsed line when the query has a parser stage) and
an outer stats pipe aggregates the per-series values, for count_over_time,
bytes_over_time, rate, bytes_rate and the unwrap functions. sum keeps its
single pipe: summing the values of a group's series is the same as
aggregating their rows. Both pipes run in VictoriaLogs.
szibis
force-pushed
the
fix/series-count-aggregations
branch
from
September 23, 2026 09:24
a5dbfb2 to
8921d0f
Compare
szibis
added a commit
that referenced
this pull request
Sep 23, 2026
#608) #582 merged after v1.89.0 was cut, from a head that predated the cut, and the merge placed its two entries inside the released 1.89.0 section. Restore 1.89.0 to what the release published and list the entries under Unreleased, where the next release will pick them up. Co-authored-by: szibis <szibis@users.noreply.github.com>
This was referenced Oct 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
count,min,maxandavgof a range aggregation read one value per Loki series. They were translated to the same singlestatspipe assum, so they returned line counts:count by (pod) (count_over_time({app="worker-service"}[1m])): 15 per pod, Loki answers 1.count by (service_name) (count_over_time({namespace="data"}[15m])): 9,768, Loki answers 900.max by (service_name) (count_over_time({namespace="data"}[15m])): the same sums (6,660 / 3,108) where Loki answers 15 / 7.The inner stats pipe now keeps the series identity (the requested labels plus the stream, or the parsed line when the query has a parser stage) and an outer stats pipe aggregates those per-series values, for
count_over_time,bytes_over_time,rate,bytes_rateand theunwrapfunctions.sumkeeps its single pipe: summing the values of a group's series is the same as aggregating their rows. Both pipes run in VictoriaLogs, so no rows are buffered.Verification
go test ./internal/... ./cmd/...,go vetandgolangci-lint(v2.13.2, Go 1.27.1) are clean. New translator tests pin the emitted LogsQL for every shape, includingsum, which must keep the single pipe.New e2e-compat test
TestCompat_SeriesCountAggregations(in thecoreshard) pushes three streams with 5/3/1 lines to Loki and VictoriaLogs and comparescount by (pod),count,min,max,avg,max by (pod)and bothsumforms against Loki's own answers. It passes on this build and fails onmain.Live, against Loki direct (instant, same evaluation time):
count by (pod) (count_over_time({app="worker-service"}[1m]))count by (service_name) (count_over_time({namespace="data"}[15m]))max by (service_name) (count_over_time({namespace="data"}[15m]))count(count_over_time({app="worker-service"}[1m]))min by (pod) (count_over_time({app="worker-service"}[1m]))(444 vs 450 is the stream visibility lag between the two backends at that instant, not a translation difference.)
Grouping sweep (15 query shapes, instant and range, 15m/1h/6h/24h)
Against a build of this branch's base (
mainat 7fb87b2). VictoriaLogsRestartCount3 before and 3 after.Every difference is
count by (service_name) (count_over_time(…))becoming exact, plustopk(3, …)tie flips in both directions (places 3-5 are tied, so membership is arbitrary on both sides).without (...)count,min,maxandavgwithwithout (...)were summed by the response-side merge whatever the operator, somax without (pod) (count_over_time({app="worker-service"}[1m]))returned 450 (the line count) where Loki answers 15, andcount without (pod)30 where Loki answers 1. The outer stats pipe now keeps the identity and the merge folds each remaining label set with the query's operator. Live, both match Loki exactly for that app (count30,max15,avg15,sumunchanged), and the e2e test covers all five operators.Known limits (now stated correctly in the CHANGELOG and the tests)
The proxy's series identity is the stream and its level. Loki also splits a stream per distinct structured-metadata set and per set of labels a
| jsonor| logfmtstage extracted, so it counts more series:count(count_over_time({app="worker-service"} | logfmt [1m]))is 450 on Loki and 30 here, andcount without (pod)over a namespace whose lines carry a per-linetrace_idcounts streams rather than lines (858 series on Loki, 30 here). The earlier claim that "a parser stage makes the line the identity" was wrong and is gone from both the CHANGELOG andseries_aggregation_test.go. #587 adds the| regexp/| patternhalf of that identity, which is the part VictoriaLogs can group by.Conformance registry
metric-series-identitymetric-series-identityis new: it records the per-series contract, the structured-metadata and dynamic-key-parser deviations as an owner decision with the waivermetric-identity-metadata-and-dynamic-keys(expiry 2027-03-31), and the measured numbers.python3 scripts/ci/check_conformance.pypasses.Other review points
count by (x) (rate(...))no longer emits the| math … as __lvp_ratepipe nothing read.addByClauseLastdocuments that it must run before any marker is appended, which is the invariant itsLastIndexsearch depends on.