Repository navigation
fix: unwrap sum, avg, max and min over time answer Loki's window for every range and step - #885
Merged
Merged
Conversation
github-actions
Bot
added
size/XL
Contributor
PR Quality ReportCompared against base branch Coverage and tests
Compatibility
Performance smokeLower CPU cost (
State
|
Contributor
✅ Visual smoke: passed with warningsBase Fix proofCaptures of the exact cases this pull request adds or fixes (registry case, base | PR | Loki). The verdict compares the PR with Loki where the base did not match. The gate fails a case unless one capture is fixed or improved, or, beyond the Loki data window, the base failed where Loki and the PR both answered with data (the label does not excuse it; only
explore-unwrap-sum-over-time-sliding-window-15m: improved
explore-unwrap-sum-over-time-sliding-window-1h: improved
explore-unwrap-sum-over-time-sliding-window-6h: undecided
Regression check
Gate: a base-vs-PR data difference passes when the PR is closer to Loki, and fails when the PR diverges from Loki or Loki cannot decide it (at a range Loki does not hold, a difference passes when the same page proved that query to answer like Loki at a shorter range; otherwise label the pull request dd-fields-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-28T09%3A54%3
dd-service-logs-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-28T09%3A54%3
explore-unwrap-sum-over-time-sliding-window-6h: no Loki data at this range; the same differences match Loki at a shorter range (1): query ('sum by (service_name) (sum_o
lvl-dd-field-pipeline-pill-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-28T09%3A54%3
Core set montages (3): base | PR | Lokidd-landing-1h
explore-logs-plain-1h
explore-metric-plain-1h
All montages of this run: |
Contributor
⚪ Performance A/B: no change beyond noise171 shape×range — 171 ⚪ same · results vs Loki: 161 same, 0 new difference(s), 10 pre-existing Loki profiles compared (like for like):
Every shape is within noise of the base build (±30% and 100 ms). 171 unchanged shape×range (within noise)
Cold (first-run) timings — what a user opening a panel sees
† Loki's results cache answered this cold run (its own counters, or identical answers for windows a minute apart): not a computed time, so it is in no ratio. ↺ Loki cold taken from the first set that ran the same query; Loki's cache answered this one. 20 shape×range above 2× Loki already (listed, or as slow on the base build)
Why these shapes
base |
szibis
force-pushed
the
perf/unwrap-conversion-check
branch
from
October 28, 2026 09:29
93af511 to
4efb03c
Compare
github-actions
Bot
added
size/XL
szibis
force-pushed
the
perf/unwrap-conversion-check
branch
from
October 28, 2026 09:51
4efb03c to
da1b25e
Compare
…ndow
Loki reads a step's sample from the window (t-range, t] only
(pkg/logql/range_vector.go:102-189, batchRangeVectorIterator.Next and load):
a line between two windows belongs to no sample. The window evaluator already
kept only those lines with a filter on the stored _time, but it appended the
filter after the whole log pipeline, so VictoriaLogs derived service_name,
unpacked and ran the unwrap gate on every line of the range first.
The filter now follows the query's filters, ahead of every pipe. For
sum by (service_name) (rate({env="production"} | logfmt | unwrap latency_ms [5m]))
over 7 days (step 10m, the daily E3 shape at 13 s cold and warm) half the
lines are parsed; a [1m] range skips nine lines in ten. The pipes the filter
moves ahead of are row-local and never write _time, so the stats see the
same rows and the answer is unchanged. A query holding the derived
detected_level chain keeps the filter after the chain, whose base keys its
remembered presence probe. The unwrap conversion-error lookup carries the
filter once, ahead of the parser, instead of appending a second one.
The conversion-error counter itself is not the cost: the 2026-10-08 A/B ran
E3 at 7d in 13.37 s on main without it and 13.44 s with it, and the metric
sends one stats_query_range and no lookup on clean data.
…than the step answer Loki's window
Loki reads each step's sample from (t-range, t] only
(pkg/logql/range_vector.go:102-189, batchRangeVectorIterator.Next and load).
With a range shorter than the step the windows are disjoint: most lines belong
to no sample and a step whose window is empty has none. The proxy answered
these functions from VictoriaLogs' anchored step buckets (t-step, t]
(statsRateRangeEqualsStepShift), so
sum by (service_name) (sum_over_time({service_name="api-gateway"} | json | unwrap duration_ms [1m]))
over 7 days at step 10m (the daily E1 shape; E4, E8 and E9 at 7d likewise)
summed ten minutes of lines per point where Loki sums one, and a step with
lines only outside its window had a sample Loki does not.
The stats query now carries the window filter the other range metrics use,
right after the query's filters, so each bucket holds exactly Loki's window and
VictoriaLogs parses only those lines. The unwrap conversion-error check rides
on the same query and counts only the lines inside a window, the only ones Loki
fails on.
TestUnwrapOverTimeShortRangeMatchesLoki compares the four functions with a
reference Loki evaluator over generated rows (edges, empty windows) on three
VictoriaLogs lines, with the error filter, with the conversion check, with a
scalar operand and between two vector operands: all 96 subtests fail before
this change and pass after. No visual fix proof: the visual stack holds Loki
for 1.5h, where Explore's step is seconds and every panel range is longer than
the step. A range longer than the step stays open
(semantics/unwrap-over-time-range-windows).
…for every function
Loki's batchRangeVectorIterator.load skips `sample.Timestamp <= start` (the
comment in that loop reads "the lower bound of the range is not inclusive")
and keeps samples up to and including the evaluation time, and unwrapped
ranges share that iterator (pkg/logql/range_vector.go:148-189, v3.7.7).
aggregateManualWindow excluded the lower bound only for rate,
count_over_time, bytes_over_time and bytes_rate, so every unwrapped range
function read the sample on windowStart that Loki drops, and answered one
sample high.
Correcting only the lower bound would leave the window open at both ends:
the 1ns extension of the VictoriaLogs fetch, which is exclusive at its end
where Loki is inclusive, was applied to quantile and the log-range functions
alone, so a sample exactly on the evaluation timestamp was never fetched. The
extension is now unconditional, on query_range and on instant queries.
TestLowerBoundIsExclusiveForEveryRangeFunction fails without the change and
passes with it (its quantile subcase uses q=0, which tells {1,2,3} from
{2,3}); TestCompatHelpers_AggregateManualWindow encoded the old bound and is
updated to Loki's.
…han the step answer Loki's window
Loki reads each step's sample from (t-range, t]
(pkg/logql/range_vector.go:102-189 batchRangeVectorIterator.Next and load;
the functions at range_vector.go:371-420), and the windows overlap when the
range is longer than the step. statsRateRangeEqualsStepShift read every range
of these functions as the step and shouldUseManualRangeMetricCompat kept them
off the window evaluator, so each point was one VictoriaLogs step bucket
(t-step, t]: sum(sum_over_time({app="payment-service"} | logfmt | unwrap
amount [5m])) at step 60 was 2.00257e+06 where Loki has 9.84307e+06 on the
stack, sum_over_time({app="uw"} | unwrap amount [150s]) at step 2m 725.875
where Loki has 942.125 on the emulated backend, and a line every 4.5 minutes
gave 13 samples in an hour where Loki has 61.
A range longer than the step now goes to the window evaluator that already
answers rate with unwrap (unwrapWindowBuckets). VictoriaLogs returns, per
series, buckets of gcd(range, step) anchored at the request with the partial
aggregate of the converted value (sum, max or min) and the line count
(unwrapBucketAgg; avg keeps the counts to divide by), and the proxy merges the
buckets of each window (mergeUnwrapBuckets) with a sample only where the
window holds a line. VictoriaLogs has no sliding-window stats and one step
bucket cannot hold an overlapping window, so the fold is done in the proxy,
over aggregated buckets only; a grid over 11,000 buckets goes to the raw-row
evaluator bounded by -manual-range-metric-row-limit, which reads the same
window since the previous commit. The route takes only the aggregations the
evaluator applies after the window (unwrapWindowForm): sums by labels or none,
topk and bottomk, label_replace, binary operands, count/min/max/avg by labels
and an outermost labelled without; an outer fold reads under the raw-row
budget (recordFoldBudget, shared with withWithoutFold) and Loki's series
limit applies to its answer. JSON parser queries grouped per stream and line
under an outer aggregation, nested aggregations and an empty without () keep
the step buckets (registered open:
semantics/unwrap-over-time-range-windows-residual-forms). Per-stream series of the route carry detected_level="unknown" for a
stream without a level, as Loki's and rate's do. Ranges up to the step and
instant queries are unchanged; the 7-day E1/E4/E8/E9 shapes ([1m] and [5m] at
step 10m) keep their single phase-filtered step bucket query.
The window phase filter now names the window start modulo the step instead
of the request's start: the same lines, but the query text, and so the stats
extent cache key, stays the same when a window moves by whole steps (one miss
and five partial answers where main had six misses).
Tests: TestUnwrapOverTimeWindowsDifferential (generated rows, edges at +-1ns,
offsets, instant queries, ten query forms, VictoriaLogs v1.49.0, v1.53.0 and
unknown) against a reference evaluator of Loki's windows,
TestUnwrapOverTimeRangeWindowEdgeCases (answers spelled out),
TestUnwrapOverTimeRawWindowBounds, TestUnwrapOverTimeSparseSamplePresence,
TestUnwrapOverTimeUngroupedSumIgnoresSeriesLimit,
TestUnwrapOverTimeResidualShapesKeepStatsBuckets,
TestUnwrapOverTimeShortRangeStaysOnStepBuckets,
TestWindowPhaseFilterKeepsExtentCacheAcrossSteps and the e2e
TestCompat_UnwrapOverTimeRangeWindows (shared fixtures plus lines on every
window edge, range and instant). The differential, edge and presence tests
fail on main for range > step. Pinned answers that read the step bucket move
to the window: five sliding extraction-list goldens and two max without
answers. Brought over from #740 (its tests, adapted to the supported
VictoriaLogs lines), which this supersedes.
… parser collectRangeMetricHitsAux parsed every stats_query_range answer with fj.ParseBytes, a new parser per call whose value cache is allocated again for every [timestamp, value] pair. A range longer than the step now sends sum, avg, max and min over time with unwrap through this read with a value and a count series per group, so the allocation grew with the answer: a canned 10-series answer of a 24h query at step 60 with a [5m] range allocated 50 MiB per request (main's step-bucket answer 10 MiB). It now takes a parser from statsQRFJPool, the pool the step-bucket trims use: 18 MiB per request, 1h at step 5s 28 MiB to 9 MiB. Nothing parsed outlives the call (labels are string copies, values floats).
… that folds The window route of sum, avg, max and min over time with unwrap recorded the raw-row budget in the request's series limit scope for every query it took, also a grouped sum that folds nothing; a folded read later in the same request (the other operand of a binary expression) would then read under that budget and its error text. It is now recorded only for the outer folds the route applies afterwards: a series-level aggregation chain (foldVectorAggregations) and an ungrouped sum of a function VictoriaLogs does not sum (foldUngroupedSum). A labelled without records it in withWithoutFold. Two handler tests cover what the differential's emulator cannot: TestUnwrapOverTimeRangeWindowsNaN sends the counting gate's nan group through the merged buckets (sum and avg are NaN in the five windows that hold the nan line, max and min keep the clean value; on main one step is NaN), and TestRateCounterReadsLokiWindow checks rate_counter's raw window on query_range and at an instant (0.666667 over (t-1m, t]; main answers 0.333333 with the old bounds).
…eads its buckets in place The corrected range-longer-than-step route cost the proxy 2 to 5 times the memory of main's (wrong) step-bucket answer. Measured with a canned 10-series VictoriaLogs answer served from memory (the proxy's own cost): E8 24h 6.8 MiB on main against 11.4 MiB, E4 6h 7.4 against 44.9 MiB. - max by (L) (max_over_time(...)) and min by (L) (min_over_time(...)), L possibly empty: Loki takes the extreme per series over the window and then per group (pkg/logql/range_vector.go:405-420; evaluator.go VectorAggEvaluator.Next, a NaN giving way in both), which is the extreme of the group's lines in the window. unwrapGroupPushdown has VictoriaLogs group the buckets by L instead of by stream (or stream and line for a JSON query), and the evaluator answers without the JSON fold. Loki's series limit counts the groups (pkg/logql/engine.go:530-590 JoinSampleVector), as the read does. The translated grouping must be L plus the series identity (_stream, _msg, level); anything else keeps the fold. JSON max by and min by of the same function leave the residual forms. - sum, max and min over time read no sample count: VictoriaLogs answers a bucket only when it holds a row, so the aggregate's point is the presence (the counting gate's nan group still poisons a sum's point); avg keeps it. - collectRangeMetricHitsAux reads the [ts, value] pairs in place with jsonparser and parses only each series' labels (pooled fastjson parser); buildHitsRangeMetricMatrix writes the points into a raw JSON array instead of an interface pair per point (byte for byte what encoding/json writes); the unwrap counter's kept flag leaves the answer in one in-place pass instead of three copies (dropKeptFlag, also on the step-bucket route). Same canned answer, after: E8 24h 4.5 MiB in 1.6k allocations (main 6.8 MiB, 4.2k), E8 1h 2.3 (3.0), E4 6h 4.5 (7.4), avg 24h 7.2 (6.3), the 7-day [1m] and [5m] shapes 2.9 (3.5, 4.7). TestUnwrapGroupPushdownMatchesPerStreamFold compares the pushed-down answer with the per-stream fold's (unwrapGroupPushdownOff) over generated rows, three VictoriaLogs versions, grouped, ungrouped and under topk, series limits 0 to 3: the same series byte for byte and the same 400s. The differential gains max by, min by, max and topk of max by over JSON; TestUnwrapOverTimeRangeWindowsNaN the max by and min by forms; TestDropKeptFlagMatchesReplacements holds the one-pass removal to the three replacements. Five sliding extraction-list goldens send the new bucket query; max(max_over_time(... | logfmt amount, route ...)) now answers Loki's window.
…t pairs directly buildHitsRangeMetricMatrix searched the window bounds of every step (two binary searches over the samples, one over the presence, two over the counts of avg). The windows only move forward with t, so the bounds are now cursors that only move forward; the prefix sums are built only for the additive functions that read them. eachBucketPoint reads VictoriaLogs' pairs `[number,"number"]` with a direct scan instead of two nested jsonparser walks per pair. Interleaved benchstat (n=10, canned 10-series answer served from memory, GOMAXPROCS=2, load average 7 to 9), main / before this branch's perf commits / now: E8 24h [5m] step 60 3.09 / 5.90 / 2.83 ms, 6.9 / 11.7 / 4.2 MiB; E4 6h max by 3.14 / 18.66 / 2.85 ms, 7.0 / 42.3 / 4.2 MiB; E8 1h step 5 1.59 / 3.43 / 2.15 ms; avg 24h 3.02 / 7.00 / 5.34 ms, 6.3 / 12.6 / 6.7 MiB; the 7-day range-shorter-than-step shapes 2.25 / 2.29 / 2.01 ms.
…nd keeps the fold for detected_level unwrapGroupPushdown checked that the translation's grouping held the outer labels as a set and then paired them by position with the user's by-list, while the collector names each grouped field after the label at its index (collectRangeMetricHitsAux vlToLoki): a by-list the translation writes in another order would put a value under the wrong name. Each grouped field is now named after the label whose name or VictoriaLogs name it is. A detected_level (or level) grouping keeps the per-stream fold: a stream without a level is labelled detected_level="unknown" by the per-stream read (buildMetricSeriesEntry), which a grouping by the stored field does not give. service_name is pushed down: the translation derives it ahead of the stats (the coalesce over the stored names), as for sum by (service_name). TestUnwrapGroupPushdownMatchesPerStreamFold gains max by (app, pod), min by (pod, app) and max by (detected_level, pod); TestUnwrapGroupPushdownDerivesServiceName checks the derivation ahead of the grouping. The e2e TestCompat_UnwrapOverTimeRangeWindows asks max by (app), min by (route) and max over max_over_time over json and logfmt, on the shared and the edge fixtures, range and instant.
…ot from the range
buildHitsRangeMetricMatrix reserved 32 bytes for every step of the range for
every series that held a point. A query that builds a series per stream before
an outer fold reserved that for thousands of sparse series: the 6h
`sum without (pod) (count_over_time({env="production", ...}[1m]))` shape (W8 of
the unwrap-sum-without set, step 10s, 2161 steps) held 1.9 GB live and
allocated 22 GB per six queries on the stack, a quarter more than main, whose
point slices reserved 24 bytes a step. The process ran at its memory limit and
spent the difference in the collector.
A point needs a bucket in its window, and a bucket lies in the window of at
most window/step+2 evaluation times, so a series cannot hold more points than
its buckets (its presence buckets for count and rate) times that, and never
more than the steps of the range. The reservation is now that bound; a series
that fills every step reserves what it did.
Measured on the A/B stack (6h, step 10s, the same windows, six queries each):
main 2.6 s and 3.3 GB peak RSS, before this commit 5.3 s and 4.1 GB, with it
1.4 s and 0.38 GB. The answers are identical (the same bytes per window).
szibis
force-pushed
the
perf/unwrap-conversion-check
branch
from
October 28, 2026 09:51
da1b25e to
44ca788
Compare
github-actions
Bot
added
size/XL
This was referenced Oct 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes
semantics/unwrap-over-time-range-windows:sum_over_time,avg_over_time,max_over_timeandmin_over_timewith unwrap answer Loki's window (t-range, t] on query_range for every relation of range and step: shorter, equal, longer and not a multiple, sliding, json and logfmt, grouped and ungrouped. Instant queries were already exact and stay so.This supersedes #740 (
fix/unwrap-over-time-range-windows). Its first four commits duplicate #727, which is already on main, and its VictoriaLogs v1.40 parts are below the v1.49.0 floor. Both are dropped here; its tests are brought over and adapted.It also includes #570 by @clain23, kept as a cherry-picked commit with the contributor as author: the raw-row evaluator reads (t-range, t] for every function.
What was wrong (range longer than the step on main)
Loki reads each step's sample from (t-range, t]:
pkg/logql/range_vector.go:102-189(batchRangeVectorIterator.Nextandload) defines the window: the lower bound is exclusive and the evaluation time is inclusive.range_vector.go:371-420definessumOverTime,avgOverTime,maxOverTimeandminOverTime.With a range longer than the step, the windows overlap.
Root cause on main:
statsRateRangeEqualsStepShift(internal/proxy/metric_binary.go:796-820) treats every range of these functions as the step (spec.Window = step).shouldUseManualRangeMetricCompat(range_metric_compat.go) kept them off the window evaluator.So every point was one VictoriaLogs step bucket (t-step, t]:
sum(sum_over_time({app="payment-service"} | logfmt | unwrap amount [5m]))at step 60: proxy 2.00257e+06, Loki 9.84307e+06 (stack measurement, Loki 3.7.7 and VictoriaLogs v1.52.0). The proxy sums one step of the five in the window.sum_over_time({app="uw"} | unwrap amount [150s])at step 2m: proxy 725.875, Loki 942.125 (emulated VictoriaLogs).[5m]at step 60 gave 13 samples in an hour; Loki has 61.Fix
2c9beeddperf51a8ee34fixsemantics/unwrap-over-time-short-range-windows)944deceafixsemantics/lower-bound-exclusive)5e2fca17fix0fa20eedperf18e48088fix67382665perfmax by/min byover the same function grouped upstream (unwrapGroupPushdown); sum, max and min buckets without a count; buckets read in place; points written as raw JSON; one-pass flag stripbd821cafperf5ac625a5fix93af511dperfA memory regression found by the stack A/B, fixed in the last commit.
sum without (pod) (count_over_time({env="production", instrumentation!="otel", app!="otel-collector"}[1m]))over 6h at step 10s (W8 ofunwrap-sum-without, not an unwrap query) was 1.7 to 2× slower than main in the first A/B of this branch (main 9.5 s, branch 17.6 s warm p50). The proxy time was in the collector, not in a computation: a heap profile showedbuildHitsRangeMetricMatrixholding 1.9 GB live for a 1 MB answer, because it reserves room for every step of the range for every series that holds a point, and this query builds a series per stream before the outer fold. The raw-JSON points of67382665reserved 32 bytes a step where main's point slices reserved 24, so the branch held a quarter more and ran at the process memory limit. The last commit reserves the bound a series can reach (its buckets times window/step+2, never more than the steps of the range). Same six windows, one stack, one proxy per build:The answers are byte for byte the same; the A/B below, taken with the fix, has W8 6h at 3.05 s on main and 1.60 s on the branch.
Design of range longer than the step: VictoriaLogs first, a bounded fold in the proxy.
Routing.
handleStatsCompatRangesends these functions to the existing window evaluator (proxyManualRangeMetricRange), the route rate with unwrap already takes. The route is used only whenunwrapWindowBucketsholds: a range longer than the step, a gated unwrap, and series that are a stats grouping.What VictoriaLogs computes. VictoriaLogs answers one
stats_query_rangeon buckets of gcd(range, step), anchored at the request with theoffsetargument (slidingStatsBucket,setSlidingStatsRangeParams). Each bucket holds the partial aggregate of the converted value (sum,maxormin;unwrapBucketAgg). VictoriaLogs answers only a bucket that holds a row, so the aggregate's point is the presence; avg addscount() as __sample_countto divide by.Group push-down.
max by (L) (max_over_time(...))andmin by (L) (min_over_time(...)), L possibly empty, are the extreme of the group's lines in the window whatever the series are (range_vector.go:405-420,VectorAggEvaluator.Next, a NaN giving way in both).unwrapGroupPushdownhas VictoriaLogs group the buckets by L instead of per stream (or per stream and line for JSON) and skips the JSON fold. Loki's series limit counts the groups (pkg/logql/engine.go:530-590JoinSampleVector), as the read does.avg byoveravg_over_timeis not decomposable and keeps the per-series fold.detected_levelgrouping keeps the fold: the per-stream read labels a stream with no levelunknown.service_nameis pushed down: the translation derives it ahead of the stats (the coalesce over the stored names), as forsum by (service_name).What the proxy computes.
mergeUnwrapBucketsfolds the buckets of every window:A step gets a sample only where the window holds a line.
Why VictoriaLogs cannot do it alone. A
stats_query_rangebucket is one tumbling interval, and VictoriaLogs has no sliding-window stats. Overlapping windows cannot be one bucket each. The proxy reads only aggregated buckets (series × buckets, bounded by the 64 MiB stats response limit), never raw rows.-manual-range-metric-row-limit. That happens for a range the step does not divide, for example[5m]at step 86.4s, which needs 2.4s buckets.Aggregations around the range.
unwrapWindowFormtakes only the ones the evaluator applies after the window:foldVectorAggregations);withoutas the outermost aggregation (withWithoutFold).A query that folds afterwards (a series-level chain, an ungrouped sum of a function VictoriaLogs does not sum) reads under the raw-row budget (
recordFoldBudget, shared withwithWithoutFold), and Loki's series limit applies to the folded answer. A grouped sum over the limit still answers Loki's 400.Per-stream series of this route carry
detected_level="unknown"for a stream without a level, as Loki's do (semantics/without-instant-detected-level-unknown) and as rate's already did.Unchanged. Ranges up to the step keep the native step bucket path. The 7-day E1, E4, E8 and E9 shapes (
[1m]and[5m]at step 10m) still send one phase-filteredstats_query_range(TestUnwrapOverTimeShortRangeStaysOnStepBuckets). Instant queries are unchanged.Extent cache. The window phase filter now names the window start modulo the step. A window moved by whole steps sends the same query text, so the stats extent cache reuses its buckets: one miss and five partial answers, where main had six misses (
TestWindowPhaseFilterKeepsExtentCacheAcrossSteps).unwrapgets an unwrap check:startUnwrapConversionCheck(unwrap_conversion_error.go:1342-1360) attaches a nan-counting check even with| __error__="".statsExtentPost(stats_extent_cache.go:116-119) sends every call of a checked request to VictoriaLogs.VictoriaLogs support: the evaluator needs the
stats_query_rangeoffsetargument andcount(),sum,maxandmin, all present from v1.49.0 to v1.53.0. No new capability gate.Tests
All of these run against the real handler over the emulated VictoriaLogs (
vlEmulator) and compare with a reference evaluator of Loki's windows written apart from the proxy (lokiOverTimeGroups).dfa8d041)TestUnwrapOverTimeWindowsDifferential: generated rows with edges at ±1ns, offsets, 1 in 6 cases instant, 11 query forms, json/logfmt/none, VictoriaLogs v1.49.0, v1.53.0 and unknownTestUnwrapOverTimeRangeWindowEdgeCases: answers spelled out for lines on every window edgeTestUnwrapOverTimeRawWindowBounds: the more than 11,000-bucket raw route, edges ±1nsTestUnwrapOverTimeSparseSamplePresenceTestUnwrapOverTimeUngroupedSumIgnoresSeriesLimit(plus the grouped 400)TestUnwrapOverTimeShortRangeStaysOnStepBuckets(7d routing)TestWindowPhaseFilterKeepsExtentCacheAcrossStepsTestUnwrapOverTimeResidualShapesKeepStatsBuckets(pins the open forms on step buckets, no raw rows)TestLowerBoundIsExclusiveForEveryRangeFunction(#570)TestUnwrapGroupPushdownMatchesPerStreamFold: push-down vs the per-stream fold (unwrapGroupPushdownOff), every series' values byte for byte under the same labels and the same 400s, series limits 0 to 3, three versions, two labels in either order, detected_levelTestUnwrapGroupPushdownDerivesServiceNameTestDropKeptFlagMatchesReplacements(one-pass strip = the three replacements, 5000 generated answers)TestRateCounterReadsLokiWindow: rate_counter through the handler, range and instantTestUnwrapOverTimeRangeWindowsNaN: the counting gate's nan group through the merged buckets (sum and avg NaN in the 5 windows that hold the line; max and min keep the clean value), with and without| __error__=""Conversion errors on the new route are covered by the existing
TestUnwrapConversionErrorAnswersLokiText. Itssum by (service_name) (sum_over_time(... | logfmt | unwrap v [5m]))andmax_over_time ... duration(t)cases at step 60 now take the window route and still answer Loki's 400 text. The intercepted form sends only the nan counter.Pinned answers that read the step bucket move to the window:
TestExtractionListOutsideWithoutKeepsItsGrouping;max without (pod)answers ofTestWithoutFoldIsLimitedToTheOperatorsItProves, now Loki's 36,36,36 and 40,40,40 by hand from the fixture, withdetected_level.TestQueryRange_AvgOverTimeByEmptyReturnsSingleSeriesanswers buckets of the sum and the count. The test harnesses now shut down every proxy and close every cache they build (runOverTimeCase,unwrapRangeResult).e2e:
TestCompat_UnwrapOverTimeRangeWindows(group core, compiled with-tags e2e). It now also asksmax by (app),min by (route)andmax(max_over_time)over json and logfmt:[1m],[2m],[150s]and[30s]at step 1m,[5m]at step 2m, and an instant query on an edge. The JSON instant goes to the raw evaluator, which covers fix(range): the range-vector lower bound is exclusive for every function #570 end to end.Proofs (local, no Docker)
Unit tests capped (
GOMEMLIMIT=2GiB GOMAXPROCS=4 go test -short -count=1 -p 1): pass.-raceon the touched tests (the window, push-down, ungrouped-sum, phase-filter, NaN, rate_counter and avg tests the bucket reader reaches, rerun on the last commit): pass.Full capped run,
GOMEMLIMIT=3GiB GOMAXPROCS=4 /usr/bin/time -l go test -short -count=1 -p 1 ./cmd/... ./internal/...: every package ok.250c5d3cc538f410(the stack-proof head, before the rebase onto507915a5)93af511d(after the rebase, idle host)All three: every package ok, run on an otherwise idle host. The peak RSS of this work's six earlier runs ranged from 1.735 to 1.906 GB on the same code family, so the peak depends on the order the tests are scheduled in; the branch is +5% over main here. Full runs taken while the host was loaded failed on
dial tcp ...: connect: can't assign requested address(macOS ephemeral ports in TIME_WAIT) and on the ten-minutego testtimeout, never on an assertion; they passed when repeated on an idle host.go vet ./...andgo vet -tags e2e ./test/e2e-compat/: clean.gofmt -l: clean.golangci-lint run ./...: 0 issues.python3 scripts/ci/check_conformance.py: conformance gate ok.python3 scripts/ci/check_changelog_pr.py --base origin/main --head HEAD: ok. Released sections are byte-identical to main.Proxy-side cost (benchuw: a canned 10-series VictoriaLogs answer served from memory, so only the proxy is measured;
GOMAXPROCS=2; 10 interleaved rounds of main, the branch before its last two perf commits (18e48088) and now (bd821caf); benchstat; load average 9.2 at the start, 9.0 at the end). VictoriaLogs time is not included; the stack A/B measures it.[1m]step 600 (range shorter than the step)[5m]step 600 (range shorter than the step)[5m]step 60[5m]step 5max by[1m]step 15min bymax(max_over_time)24h, logfmt[5m]step 60max(max_over_time(... | logfmt ...))answered the step bucket before; it is now exact through the push-down.scratchpad/benchuw/zz_bench_uw_test.go. Its fake keeps every answer in memory, and thelimitargument varies per iteration to miss the proxy's response cache. Results are inbenchuw/benchstat.txt.Registry
Changed:
New (pending owner review):
New with the prepared commit (pending owner review):
Visual fix-proof entry:
explore-unwrap-sum-over-time-sliding-window(bench/visual/spec.json),sum by (service_name) (sum_over_time({service_name="api-gateway"} | json | unwrap duration_ms | __error__="" [5m]))at Explore's step.Not covered (follow-ups)
semantics/unwrap-over-time-range-windows-residual-formsstays open. These keep the step bucket:without);without ().All need Loki's parsed series identity as a stats grouping first.
rate_counter's value has no Loki extrapolation (
range_vector.go:275-346); only its window membership is fixed.foldVectorAggregationsre-encodes JSON for the outer folds that are not decomposable (count by,avg by,max byover another function). Their proxy cost stays at the old level; the decomposablemax by/min byforms are pushed down.A range the step does not divide can pass 11,000 gcd buckets and fall back to raw rows (bounded). A phase-split alternative stays on step-wide buckets:
The outer
avg(...)over a logfmt query and outer count/min/max/avg over json differ from Loki for any range relation (series identity, also on main).detected_level="unknown"now appears on per-stream series when the range is longer than the step, as Loki puts it, but not on the native step path of a range up to the step. Zooming a panel can change the label set. The native path's per-stream series should carry it too (semantics/without-instant-detected-level-unknownis the instant side).An instant
topk(5, max by (pod) (max_over_time(... | json ...)))misses samples, on main too: the instant path does not apply a series-level chain under a ranking. The differential keeps that form to query_range (rangeOnly).E2 at 7d (
quantile_over_timeat[1m], step 10m) answering 502 was not reproduced here, because it needs the stack. Unverified hypothesis:fetchQuantileCountsreads value counts per bucket for 1008 buckets and passes-backend-max-buffered-response-bytes;badRequestStatusOrmaps that error to 502.The
withoutfolds over an unwrap function (W1, W2, W3, W4, W7 ofunwrap-sum-without, 6h, step 10s) stay at 7 to 40x Loki's warm and refresh time (113 to 180 ms against 3 to 18 ms), 2x main's because main answered the wrong window; W8 (sum without (pod) (count_over_time(...)), not an unwrap query) is 1.3 to 1.6 s against 10 to 87 ms on main and here alike. Both go through the window route, which builds a per-stream matrix and folds it; reading the buckets through the stats extent cache and folding from stored buckets is the next item (the stacked branch does not cover this route).E2 (
quantile_over_timeover unwrap, 6h) and U9 (rate_counterfolded per series) are above 2x Loki warm and refresh on main and here; they use the quantile and per-series fold routes.Open questions for the owner
TestUnwrapOverTime*alone peaks at 92 MB.Proofs on a stack
Everything below ran on one Colima VM (8 CPU, 14 GB), Loki 3.7.9, VictoriaLogs v1.49.0 (floor) and v1.53.0. Stack-proof heads are
c538f410(this branch) andbfaedadd(the stacked extent-cache branch), cut from250c5d3c. The branches were then rebased onto507915a5(heads93af511dandde6fadb7); the only change on main in between is #883 (a transport retry), the gates, the unit run and the e2e regex groups were repeated on the rebased heads and pass, and the A/B, 24h/7d, load and visual numbers carry over. VictoriaLogs and Loki restart counts were read before and after the e2e, A/B, 24h, 7d (VictoriaLogs) and load runs and were 0 every time; the visual runs did not record them.Status against the 2x Loki bar. It is correct (e2e, differentials, visual: its answers equal Loki's where main's differ) and it removes the memory regression it introduced. The shapes on the window route stay above 2x Loki: W1, W2, W3, W4, W7 (
withoutfolds over an unwrap function, 6h) at 7 to 40x Loki warm and refresh, and U9 (rate_counter, 1h and 6h) at 7 to 60x; main is above 2x on the same rows, answering a wrong window faster. These rows are kept on the above-2x baseline and tracked as follow-ups (follow-up 9 below, the stats extent cache for the window route, and step-aligned refresh), and this change lands now so the answers are Loki's.e2e against Loki (group core and the unwrap regex)
go test -tags e2e -count=1 -run '^(TestSetup_IngestLogs|TestCompat_UnwrapOverTimeRangeWindows|TestCompat_UnwrapOverTimeShortRange|TestCompat_UnwrapRangeEndSample|TestCompat_UnwrapUngroupedSum.*|TestCompat_UnwrapConversionErrorParity|TestCompat_UnwrapSumWithout.*|TestCompat_UnwrapFirstLast.*)$' ./test/e2e-compat/on an isolated compose project (every port +10000), plus the whole core group (TestSetup_IngestLogs,TestCompat_*,TestExtended_*,TestChaining_*, ...). Subtests ofTestCompat_UnwrapConversionErrorParity/openandTestCompat_UnwrapValueFormsParityskip by design (forms whose Loki error the proxy does not detect yet, registered open).TestCompat_*and friends)21:48load 6.77 6.31 7.4021:44load 5.10 5.72 7.71TestCompat_*and friends)21:55load 13.20 8.55 7.8021:46load 9.47 6.40 7.63Named tests:
TestChaining_MetricQueryUnwrapChainWorksPASS,TestCompat_UnwrapConversionErrorParityPASS,TestCompat_UnwrapDurationSecondsPASS,TestCompat_UnwrapFirstLastRangeLevelByPASS,TestCompat_UnwrapFirstLastTopkBottomkPASS,TestCompat_UnwrapFirstLastUnderAggregationPASS,TestCompat_UnwrapMissingAndNonNumericPASS,TestCompat_UnwrapOverTimeRangeWindowsPASS,TestCompat_UnwrapOverTimeShortRangePASS,TestCompat_UnwrapOverflowParityPASS,TestCompat_UnwrapQuantilePASS,TestCompat_UnwrapRangeEndSamplePASS,TestCompat_UnwrapRangeSamplesAreNumbersPASS,TestCompat_UnwrapSumWithoutExtractionListPASS,TestCompat_UnwrapSumWithoutKeepsSeriesPASS,TestCompat_UnwrapSumWithoutUnderAnotherAggregationPASS,TestCompat_UnwrapUngroupedSumPASS,TestCompat_UnwrapUngroupedSumPerSeriesFunctionsPASS,TestCompat_UnwrapValueFormsParityPASS,TestSetup_IngestLogsPASSThe rebased heads (
93af511d) repeat the regex group: 858 passed, 0 failed, 3 skipped on v1.49.0 and on v1.53.0.A/B: main, this branch and Loki (
bench/ab/pr_smoke.py)pr_smoke.py --base origin/main --head perf/unwrap-conversion-check --ranges 1h,6h --refresh-s 60 --refresh-runs 5 --loki-mem 6g --vl-mem 4gwith the setsunwrap-conversion-error,unwrap-gate,unwrap-validity,unwrap-ungrouped-sum,unwrap-sum-withoutandcontrol: 132 shape x range rows, 4 runs (the first cold), warm p50, refresh p50 over windows moving 60 s, targets interleaved per request, 26.9 minutes, load average 5.8 at the start. Base250c5d3c, headc538f410.Verdict: faster, no regression. 1 faster, 131 within noise; results against Loki: 120 same, 0 new differences, 12 pre-existing (all
differs (as base)). Above 2x Loki: 0 new, 0 worsened by the tool's rule, 67 on the committed list, 16 cleared.Proxy RSS is in the load test below; proxy CPU was not sampled in these runs. VictoriaLogs CPU, sampled during every interleaved row (one number per row, shared by the three targets): median row maximum 58%, highest 528% of one core (8 cores). VictoriaLogs restarts 0 to 0 in every set.
The one verdict-level mover is W8 6h (3.05 s to 1.60 s, the memory fix above). The tool's verdict misses smaller moves, so every cold, warm and refresh cell was compared with main's by hand: 396 cells, 6 faster than main by more than 50 ms and 25%, and 13 slower by more than 50 ms and 25%. Those 13 are the 6h cold, warm and refresh cells of W1, W2, W4 and W7 and the 1h refresh of W4 (all in
unwrap-sum-without;sum without (pod)ormax without (pod)over an unwrap function with a 5m range at a 10s step):sum without (pod) (max_over_time(... unwrap amount [5m]))6h cold / warm / refreshsum without (pod) (sum_over_time(...))6hmax without (pod) (max_over_time(...))6htopk(5, sum without (pod) (...))6hIn each of those rows main answers the step bucket and its answer differs from Loki (
baseline_parity_vs_reference: differs), and this branch's answer is the same as Loki's. The extra 80 to 90 ms is the cost of the correct answer (merging the sliding windows of the buckets instead of reading one), not a regression of a correct path. These rows are still 7 to 40x Loki's warm and refresh time, so they are listed as above 2x below and are not fixed here (the stacked branch does not cache this route either; see its description).Rows above 2x Loki (PR ÷ Loki above 2 and more than 50 ms over, cold, warm or refresh)
71 cells. Main is above 2x on almost all of them too; the base column shows it. The first-pass
pr_smokelist at the start of this work (before the memory fix) had 4 new and 14 worsened rows, all W1, W2, W3, W4, W7, W8 and U3, U12; with the fix none is new or worse than the committed list.71 mode-rows above 2x of 132 shape x range rows.
The full 132-row table (cold, warm, refresh for main, this branch and Loki, with every ratio) is generated from the run's summaries; it is posted with the A/B comment.
Long ranges: 24h against Loki, 7d without Loki
sum,rate,maxover unwrap, E1 E3 E4 E8 E9 ofunwrap-conversion-error, one stack, main250c5d3c, this branch (c538f410), the stacked branch (bfaedadd) and Loki side by side. Cold is the first request of the window, warm the median of three repeats, refresh the median of five windows moving 60 s forward. 24h: seed of 26h of generator data (Loki and VictoriaLogs), step 60 s, window ending an hour before the seed end. Answers against Loki: E1, E4, E8, E9 all 1441 (E8 7205) points equal on this branch and the stacked one; main has 0 of 7205 (E8) and 0 of 1441 (E9) points equal (the sliding-window bug); E3 has one series more than Loki in all three builds (the pre-existing series-identity difference of thediffers (as base)rows).7d: a seed of 170 hours into VictoriaLogs (retention raised to 9d with a scratch compose overlay). Against Loki it could not be run on this stack: with the same overlay for Loki (
-store.retentionand-validation.reject-old-samples.max-ageat 216h) its push endpoint closed the connection about a day and a half into the backfill (the log generator died withRemoteDisconnected; Loki used 2.2 GiB of its 6 GiB limit at the start; the container went down with the stack, so the cause was not captured), so no 7d Loki reference exists here. The VictoriaLogs-only run compares the three builds with each other (step 600 s, refresh moving 60 s per run, which is inside one step):Cold: E1 3.9 s to 1.7 s, E3 14.3 s to 8.2 s, E4 2.2 s to 0.85 s, E8 7.9 s to 4.6 s, E9 10.3 s to 5.8 s (all five have a range shorter than the 600 s step, so the gain is the window phase filter of the first commits). The 7d warm answers come from the response cache.
Load: the backend-safety test
load_safety.pyshape: 32 concurrent clients loop over 80 range metric queries at 6h for 180 s on randomly shifted windows, builds in the order main, stacked branch, stacked branch, main on one stack (Loki and VictoriaLogs) seeded with 7h; a sampler records proxy RSS every 2 s and a/labelsprobe every 2 s. "branch 2" is the stacked extent-cache branch (it contains this one). Load average 5.7 at the start.400, 429 and 502 answers occur in the same numbers on main and on the branch, so they are not caused by it: the 400s are Loki-style pipeline errors (
SampleExtractionErr,JSONParserErr) and parse errors of generated shapes, the 429s are the heavy-query guard (-backend-max-concurrent-heavy-queries=2, wait-backend-heavy-query-queue-wait=20s), and the 502s aremanual range metric row limit exceeded (1000000)on thesum by (detected_level) (count_over_time(... | json ...))shapes (a status Loki does not give; it is on main as well and untouched here)./labelsanswered 200 on every probe in every slot, and neither backend restarted.Visual
bench/visual/ci_run.py --base origin/main(isolated stack, main and this branch as host proxies, Loki direct, Grafana with the pinned plugins), 29 captures: the newexplore-unwrap-sum-over-time-sliding-windowpage (fix proof),explore-unwrap-max-over-time-end-sample,explore-unwrap-sum,explore-unwrap-quantile-over-time,explore-unwrap-rate-sum,explore-unwrap-conversion-error, the core Explore pages and the Drilldown landing, service logs and fields pages, at 15m, 1h and 6h (Loki holds 1.5h, so 6h compares main with the branch only), plus Live tail. Base507915a5, headc538f410.explore-unwrap-sum-over-time-sliding-window: main and the branch differ, as the fix intends: at 15m the series has 90 points on main and 901 on the branch, at 1h 360 against 721, at 6h the values differ (55,640 against 987,603 at one step). The branch's panel data is identical to Loki's at 15m and 1h; only theindex/statsbytes/chunks differ, which the tool lists as explained (VictoriaLogs has no chunk accounting).main = PR) on every request, including the Drilldown pages and the Live tail (1905 entries on the branch, Loki 434 of them in common over its 1.5h window)._extractedsuffix, sampled detected_labels cardinality,index/statsandindex/volumebytes, the drilldown-limits probe) and the history-dependent patterns. One capture needs a note:explore-unwrap-conversion-errorat 15m and 6h shows a request on one side only; Grafana sent the failing query (Loki's 400) twice on the branch's panel and once on main's, both answers are the same 400. It is a Grafana retry in that capture, not a different answer.Uptime and restarts