fix: a nested vector aggregation keeps every level's grouping on instant queries and tumbling windows, like Loki by szibis · Pull Request #746 · ReliablyObserve/loki-vl-proxy

szibis · 2026-10-12T12:01:17Z

Summary

Registry: semantics/nested-aggregation-outer-by-native-routes (fixed). Also fixes two rows that were open on main: semantics/without-over-inner-aggregation-collapses and semantics/without-nested-under-aggregation.

The problem. A nested vector aggregation lost the grouping of its outer levels. min by (zone) (avg by (pod, zone) (count_over_time(...))) answered one unlabelled series on an instant query and on a tumbling window (range == step). The same happened to sum by (zone) (sum by (pod, zone) (...)) and to the middle level of a three-level nest. A without below the outermost level was also mishandled: only the first list was kept, and it was folded once with the outermost operator.

Cause. The translator's recursive branch called applyOuterAggregation, which takes no grouping argument.

What Loki does. It folds innermost first, and each level keeps exactly its own labels (pkg/logql/evaluator.go:435-630, VectorAggEvaluator.Next).

Fix: VictoriaLogs first.

tryTranslateVectorChain (internal/translator/nested_vector_chain.go) handles 2 or more levels of sum/avg/min/max/count over a range aggregation.
- It emits one stats pipe per level, each with its own by, for example stats by (pod,zone,_stream,level) count() as a | stats by (pod,zone) avg(a) as b | stats by (zone) min(b).
- This is exact for all 25 operator pairs: every pipe aggregates the rows of the previous level, one row per Loki series of that level.
- An outer by label that the inner level lacks is cut to the intersection (Loki ends up with the same single series). A without over a named label set becomes a static by.
- Level 1 translates byte-identically to before.
An innermost without names the series' own labels, which VictoriaLogs cannot list. That query keeps the series rows (the without marker with the outermost level's operator, as main's fold does for a single without) and the proxy folds every level innermost first (applyWithoutChain, internal/proxy/nested_vector_fold.go), reusing applyWithoutGrouping.
- On a tumbling window (range equal to the step), when the excluded labels are plain names (letters and digits, none the proxy derives another label from: level, detected_level, service_name and its source labels; no custom field mapping renames them), VictoriaLogs folds it too (PushDownWithout, internal/translator/without_pushdown.go): the labels of a series are the pairs of its _stream text, so the pairs of the excluded labels are cut out of that text between the stats pipe that counts a series and the one that combines the series that now share a stream, stats by (_stream, level) count() as a | replace_regexp(...) at _stream | stats by (_stream, level) avg(a). The levels above read their labels as columns of the pipes. Sliding windows, instant queries, parsers and every other name keep the proxy's fold.
- The proxy's fold decodes the answer once (a scanner into the series, not a document tree), folds every level over the series and encodes it once (internal/proxy/typed_fold.go), where it parsed and wrote the JSON again at every level.
- A query with one aggregation skips the parse that looks for a nest (HasNestedVectorKeywords).
Sliding windows (range != step) are evaluated by the proxy from the first stats pipe, the series of the innermost level. fix: max, min, count and avg over a range function are one series per group on query_range #734's fold now also takes a nest of sums and a static without level (seriesAggregationChain, foldByGrouping), the evaluator names the series by the innermost level's labels (innermostByLabels; they took the outermost level's by position), and a nest spelled sum(sum(rate(...)) by (pod, zone)) by (zone) reads its range function from the text as written (nestedVectorRange; it was read as a count, not divided by the window).

Rebase onto main

Main moved a lot (#734, #739, #860, #855, #857, #862); the branch was CONFLICTING.

What conflicted and how it was resolved:

CHANGELOG.md: main's taken, this PR's entry re-inserted once under ## [Unreleased] / ### Fixed; released sections are byte-identical to main (check_changelog_pr.py ok).
internal/translator/translator.go: tryTranslateMetricQueryWithout gained foldWithout (series rows only for the <op> without the proxy folds), and the without marker gained the operator (__without__:<op>:labels). The chain is tried first in translateLogQLFull; for an innermost without it asks for series rows (hasWithout and foldWithout true) and returns the marker with the outermost level's operator, so main's withoutFoldOp treats the request as a fold (withWithoutFold: raw-row budget, the series limit on the folded answer, parsed group labels). Static without levels leave no marker.
internal/proxy/metric_agg.go, proxy.go: the four post-processing sites call applyWithoutChain(body, labels, query, withoutOp), which folds the chain when the query is one (also under topk, sort, label_replace: translator.PeelWithoutWrappers) and otherwise is main's applyWithoutGrouping(..., withoutOp) unchanged.
bench/visual/spec.json: main's taken, this PR's entry re-added.
Generated files (conformance/registry/generated/*, conformance/reports/*): main's taken, regenerated with the five generators.

How it composes with the rest of main:

fix: max, min, count and avg over a range function are one series per group on query_range #734 (foldVectorAggregations): same call sites on the sliding path, combined as described above. Sums-only nests and static without levels go through the fold now; without at the innermost level stays on the marker path.
perf: metric query_range reuses the VictoriaLogs buckets it read before #862 (stats extent cache, reflection-free encoder): no change to either. The chain's pipes are stats pipes, which VictoriaLogs groups by the time bucket itself, so each bucket depends on its own rows, the assumption statsExtentQueryCacheable relies on. The differential runs with the cache at its default.
fix: sum without over an unwrap range function keeps a series per remaining label set, like Loki #739/fix: sum, avg, max and min over time with unwrap answer the sample at end on query_range, like Loki #727 (without fold, over-time unwrap): the fold scope table TestWithoutFoldIsLimitedToTheOperatorsItProves and the extraction-list goldens pinned the shapes of two open rows. They are updated for the shapes this PR changes (below); every other golden is main's.

Goldens and pinned answers changed (only entries this PR's behaviour changes):

TestWithoutFoldIsLimitedToTheOperatorsItProves: max(sum without (pod) (...)) over max_over_time and over rate, sum by (ns) (sum without ...) (both), topk(2, max(sum without ...)) and sort(max(sum without ...)). The answers are now a series per ns / the largest per-ns sum (and limit 1 answers 400 only where the folded answer has more than one series). The rate range values are rates (they were counts: 8 -> 8/300).
TestExtractionListOutsideWithoutKeepsItsGrouping: the 9 goldens of min(sum without ...), max(sum without ...), sum by (ns) (sum without ...) (instant, tumbling, sliding). Checked against a hand-computed reference of the fixture (instant: 15, 183, {x: 525, y: 425}).
TestWithoutShapesStillDifferFromLoki: the cases of the two flipped rows moved to TestWithoutNestedAggregationMatchesLoki, which asserts Loki's answer on an instant query and a range.
e2e: TestCompat_UnwrapSumWithoutUnderMaxStillDiffers (pinned the proxy's old, wrong answer) is removed; TestCompat_UnwrapSumWithoutUnderAnotherAggregation now also holds max, min, avg and count over sum without (route) to Loki.

Tests

TestNestedVectorAggregationEmitsOneStatsPipePerLevel: 17 emitted chains. TestNestedVectorAggregationLeavesOtherShapesAlone, TestSplitVectorLevel, TestVectorLabelSetAlgebra.
TestNestedVectorAggregationDifferential and TestNestedVectorAggregationOverSeriesWithout, against a separately written Loki-semantics reference (1223 subtests pass, 202 of them sliding). They cover: all 25 operator pairs, absent labels, by (), no grouping, without over a by level, 3-level chains, innermost without; instant, tumbling and sliding windows; parser "" and logfmt; count_over_time and rate; VictoriaLogs ""/v1.52.0/v1.40.0 (sliding: v1.52.0). Every case asserts that no raw rows are read on the instant and tumbling routes. The sliding mode asserts count_over_time and rate; the window values of an over-time unwrap function on a sliding window are the open row semantics/unwrap-over-time-range-windows.
TestNestedVectorRangeSpecMatchesTheSingleLevelSpec: the original range metric spec of a nest equals the single-level one field by field, over 16 pipelines (line filters, regex matchers, parsers, drop, unwrap conversions, offset).
TestWithoutNestedAggregationMatchesLoki, TestWithoutOverInnerAggregationRateRangeMatchesLoki, TestNestedVectorAggregationOverSeriesWithout, TestNestedVectorAggregationUnderBinaryScalar, TestApplyWithoutChain*.
Tests that build a proxy shut it down and close the cache they create.
e2e vs Loki: TestCompat_NestedVectorAggregation, 33 forms x count_over_time/rate x tumbling, instant and two sliding shapes (range 5m / step 1m, range 1m / step 2m); the without over a by level and the innermost without forms are in the list.
Pushdown: TestWithoutPushdownMatchesFoldAndLoki (320 random label sets: a label on some streams only, excluded names the cut takes and does not take, values with quotes, commas, backslashes and pair-like text, 0 to 2 levels above, both functions; each held to a reference of Loki's evaluator and to the proxy's fold), TestWithoutPushdownDirectedStreams (streams that differ only by the excluded label, first, in the middle and last of the sorted pairs, absent from some streams; every operator, both functions, alone and under one or two by levels), TestWithoutPushdownOnlyForTumblingWindows, TestWithoutPushdownLeavesMappedLabelsToTheProxyFold, TestPushDownWithoutEmitsTheFoldAsStatsPipes, TestPushDownWithoutLeavesTheFoldToTheProxyWhenNotExact. Removing the first-pair cut or its comma makes the directed test fail.
Typed fold: TestTypedFoldMatchesGenericFold (600 random matrices and vectors, 1 to 3 levels of by and without, held to the generic fold), TestTypedScannerReadsWhatTheDocumentTreeReads (800 generated bodies in other layouts, with extra keys, escapes, null and short samples, held to encoding/json's tree), TestJSONNumberAndTextMatchEncodingJSON, BenchmarkFoldWithoutMatrix.
TestHasNestedVectorKeywords: every spelling of a nest is recognised, and tryTranslateVectorChain allocates nothing for a query that is not one.
e2e: TestCompat_NestedVectorAggregation gains seven single-level and without (baz) forms.
TestWithoutFoldRowBudgetIsNotASeriesLimit: a tumbling sum without of count_over_time is folded by VictoriaLogs and its two series fit, so it answers 200 as Loki does (it expected the row-limit error of the proxy's fold).

Proofs on this branch (rebased)

go build ./..., go vet ./..., go vet -tags e2e ./test/e2e-compat/, gofmt -l: clean.
Capped full run (GOMEMLIMIT=3GiB GOMAXPROCS=4 -short -p 1 ./cmd/... ./internal/...): all packages ok, peak RSS of the run 2.46 GB.
internal/proxy test binary, peak RSS (-test.short, GOMEMLIMIT=3GiB GOMAXPROCS=4): origin/main 2.07 and 2.08 GB (two runs); this branch 2.20 and 2.01 GB. At GOMEMLIMIT=1GiB: main 1.33 GB, branch 1.39 GB. (An earlier iteration with the sliding mode on three VictoriaLogs versions measured 2.3-2.75 GB: the extra proxies of the matrix; the sliding mode now runs on one version.)
-race on TestWithoutPushdown*, TestTypedFold*, TestNestedVector*, TestApplyWithoutChain*, TestWithoutNested*, TestWithoutFold* and the translator nested and pushdown tests: pass.
golangci-lint 2.14.0 (CI's version): 0 issues. check_conformance.py: gate ok. check_changelog_pr.py --base origin/main --head HEAD: ok. selection.py --check, bench/ab and bench/visual tests: ok.

Proofs

e2e against Loki

TestCompat_NestedVectorAggregation plus the controls (NonSumVectorAggregationRange, UnwrapSumWithoutKeepsSeries, UnwrapSumWithoutExtractionList, UnwrapSumWithoutUnderAnotherAggregation), final head: 979 of 979 subtests on VictoriaLogs v1.49.0 (the floor) and v1.53.0; TestCompat_NestedVectorAggregation alone 391 of 391 on v1.50.0, v1.51.1 and v1.52.0.

Targeted A/B: `nested-aggregation` set (16 shapes, final head)

Base dfa8d041 against this branch against Loki, 1h and 6h, 4 runs, refresh pass on, one fresh stack: 27 shape x range, 25 within noise, 2 faster, 0 slower; results against Loki 27 same. On main 22 of the 27 differ from Loki (one series without labels, or the wrong series count); on this branch none does. The shapes are tumbling ([$__auto]), instant and sliding nests of sum, max, min, avg and count, a three-level chain, | json and | logfmt inner levels, a without over a named level (N8), an instant innermost without (N13) and the innermost without on a tumbling window (N16). The selector leaves out the OTel-instrumented streams and the otel-collector: Loki carries their structured metadata in the series, so a level grouped by pod counts per line there, a difference main already had.

shape	range	base p50	PR p50	Loki p50	PR / Loki	base vs Loki	PR vs Loki
N1 sum by app of max by app,pod, count	1h	8 ms	34 ms	66 ms	0.52x	differs	same
N1 sum by app of max by app,pod, count	6h	25 ms	81 ms	46 ms	1.76x	differs	same
N2 max by namespace of sum by namespace,app, rate (7 runs)	1h	1 ms	1 ms	3 ms	0.33x	differs	same
N2 max by namespace of sum by namespace,app, rate (7 runs)	6h	2 ms	4 ms	5 ms	0.80x	differs	same
N3 count by namespace of sum by namespace,app	1h	60 ms	23 ms	63 ms	0.37x	differs	same
N3 count by namespace of sum by namespace,app	6h	97 ms	59 ms	97 ms	0.61x	differs	same
N4 avg by app of min by app,pod	1h	26 ms	34 ms	66 ms	0.52x	differs	same
N4 avg by app of min by app,pod	6h	60 ms	108 ms	43 ms	2.51x	differs	same
N5 three levels, max min sum	1h	21 ms	11 ms	55 ms	0.20x	differs	same
N5 three levels, max min sum	6h	28 ms	46 ms	75 ms	0.61x	differs	same
N6 json inner, sum of max	1h	66 ms	164 ms	67 ms	2.45x	differs	same
N6 json inner, sum of max	6h	42 ms	28 ms	46 ms	0.61x	differs	same
N7 logfmt inner, max of sum	1h	11 ms	7 ms	11 ms	0.64x	same	same
N7 logfmt inner, max of sum	6h	7 ms	16 ms	12 ms	1.33x	same	same
N8 without over a named by level, tumbling	1h	10 ms	18 ms	95 ms	0.19x	differs	same
N8 without over a named by level, tumbling	6h	20 ms	30 ms	51 ms	0.59x	differs	same
N9 instant sum of max	instant	6 ms	13 ms	100 ms	0.13x	differs	same
N10 instant min of avg, rate	instant	12 ms	24 ms	113 ms	0.21x	differs	same
N11 instant three levels	instant	28 ms	16 ms	83 ms	0.19x	differs	same
N12 instant json inner, count of sum	instant	9 ms	31 ms	134 ms	0.23x	differs	same
N13 instant without under by	instant	25 ms	44 ms	79 ms	0.56x	differs	same
N14 sliding sum of max (7 runs)	1h	814 ms	277 ms	5 ms	55.40x	same	same
N14 sliding sum of max (7 runs)	6h	6.88 s	4.98 s	19 ms	261.89x	same	same
N15 sliding sums only	1h	8 ms	9 ms	48 ms	0.19x	differs	same
N15 sliding sums only	6h	13 ms	17 ms	52 ms	0.33x	differs	same
N16 innermost without under by, tumbling	1h	8 ms	8 ms	35 ms	0.23x	differs	same
N16 innermost without under by, tumbling	6h	9 ms	14 ms	45 ms	0.31x	differs	same

N16 (innermost without, tumbling) was 1.44 s at 6h before the VictoriaLogs-side fold; it is 14 ms now (cold 178 ms; Loki 45 ms, cold 1.26 s). The single-level avg without (pod) (count_over_time({env="production"}[10s])) at 6h and a 10 s step: main 0.8 to 1.2 s, this branch 25 to 70 ms warm and 175 ms cold. N14 (sliding) is slow on both builds, the evaluator's own cost and not this change's; the typed fold makes it 28% (6h) and 66% (1h) faster. Several tiny shapes read 1.3x to 2.5x Loki at a few tens of ms, inside the 100 ms noise floor, on a host whose load average was 8 to 22.

Broad A/B

Earlier head, all 248 shape x range: 248 within noise, results against Loki 188 same, 0 new differences, 60 that main already had. Final head, in two parts on a fresh stack each (refresh pass on, 1h and 6h, 4 runs):

14 sets (control, json-filter-pushdown, metric-latency-gap, drilldown-breakdown-exact, loki-profile-otel-metadata, extracted-suffix, parenthesized-range-or-filters, plain-name-after-parser, unwrap-validity, extracted-pushdown, unwrap-conversion-error, unwrap-ungrouped-sum, pattern-line-filter, pattern-line-filter-anchoring): 201 shape x range, 201 same, 0 slower, 0 faster.
The other 5 (unwrap-sum-without, json-error-split, unwrap-gate, level-filter, level-filter-stored): all 95 shape×range rows are within noise of main, with no slower or faster row. Against Loki, 63 are the same and 32 differ exactly as they do on main; there is no new difference. VictoriaLogs restarts 0 before and after. Run on 2dd7a487; the later commit changes only how the typed fold reads its input.

W8 sum without pod, count_over_time (production) at 6h (a sliding window, so the proxy's fold, not the VictoriaLogs-side one) had read slower on the branch in an interrupted run (9.6 and 8.8 s against 5.8 s) that Loki's own timeouts and a stalling host had disturbed. In this run it is 4.63/4.43, 5.40/5.96, 6.94/9.55, 11.25/12.67 s (main/branch, the later runs slowed by the abandoned Loki queries) and 8.62 s against 8.28 s after the re-measure. On a quiet stack with Loki idle, 14 interleaved pairs on the final head: main median 6.43 s (5.31 to 7.62), branch median 5.92 s (4.97 to 7.38), the branch slower in 3 of 14 pairs. No regression.

Throughput under load, and what was wrong

The three first load runs (single runs, main first) showed the branch behind main. Causes found: (1) real: the nest detection cost 5 allocations per translation and 17 to 34 per request on queries that are not nests; a keyword gate now skips it (HasNestedVectorKeywords). Handler benchmarks of eight control queries and translations of eight, benchstat -count=10 -cpu 1, interleaved: allocs/op and B/op identical to main (before the gate +1.2% allocs). The translations of all 192 non-nested queries of every A/B set are byte-identical on main and the branch, so VictoriaLogs sees the same queries and the caches the same keys. (2) harness: $__auto was sent uninterpolated and instant shapes as 6h range queries (18 of 80 queries answered 400). (3) host: main alone ranged from 544 to 855 requests per 150 s in one stack as the load average went from 6 to 42. CPU profiles taken mid-run show the nest functions under 0.05% of CPU; mutex, block and goroutine profiles and VictoriaLogs calls per request match.

Harness fixed, 32 clients, 150 s per build, final head, ABBA order, load average recorded before each run (14 to 23, other work on the host), so only the relative numbers matter:

set	order	build	requests	200	400	429	502	p50	p95	peak RSS	`/labels` probe
nested set excluded	1	main	579	542	28	3	6	7.6 s	20.2 s	2.1 GB	82 probes, 0 not 200, max 2.0 s
nested set excluded	2	branch	607	568	30	3	6	6.5 s	20.2 s	2.4 GB	77, 0, 1.9 s
nested set excluded	3	branch	626	584	32	3	7	7.0 s	19.9 s	2.6 GB	75, 0, 1.7 s
nested set excluded	4	main	652	610	32	3	7	6.3 s	17.6 s	2.4 GB	76, 0, 2.3 s
full set (nested added)	1	branch	594	561	27	4	2	7.5 s	21.1 s	3.6 GB	76, 0, 1.5 s
full set	2	main	633	596	30	4	3	6.7 s	18.5 s	3.7 GB	75, 0, 2.2 s
full set	3	main	617	580	30	4	3	6.3 s	19.9 s	3.7 GB	76, 0, 1.2 s
full set	4	branch	693	651	32	6	4	6.1 s	18.7 s	4.3 GB	77, 0, 1.2 s

Totals: nested set excluded, main 1231 and branch 1233 requests; full set, branch 1287 and main 1250. VictoriaLogs and Loki restarts 0 in every run. One at a time on an idle stack the 80 queries take 59.6 s on main and 60.6 s on the branch.

Peak memory under load (the 4.3 GB peak)

In the first runs the branch's peak RSS (4.3 GB, against 3.7 to 3.8 GB for main) looked like variance. It was not. Method: the full set (nested shapes added), 32 clients, 150 s per slot, ABBA, RSS sampled every second, /debug/pprof/heap taken when RSS first crossed 3.5 GB (and again every 250 MB higher), plus mid-run and late.

head	build	peak RSS per slot
`2dd7a487`	branch	3.53, 3.78, 3.93, 3.73 GB
`2dd7a487`	main	3.17, 2.88, 3.16, 3.20 GB

The ranges do not overlap: +0.64 GB on average. The heap profile at each branch peak held 1.5 GB in fastjson.(*cache).getValue, reached from decodeTypedBody: the typed fold parsed the sliding nest's answer (millions of samples) into a document tree, one allocated value per number, string and array, and kept it alive while it folded. Main's profile at its peaks holds the evaluator's own matrices (buildHitsRangeMetricMatrix, 2.2 GB in one profile) and no tree. Fix (0f0ece61): decodeTypedBody scans the bytes into the series directly, 16 bytes a sample, and falls back to the generic fold for anything it does not read. 200 series of 2160 points: 36 ms, 28 MB and 7.3k allocations, against 47 ms, 67 MB and 89k for the tree and 143 ms, 99 MB and 3.1M for the generic fold.

Re-run on the fixed head, same method, load average 8 to 24 before each slot:

build	peak RSS per slot	requests in 4 slots	p50 per slot
branch	2.87, 3.06, 2.93, 2.81 GB	2672	6.8, 6.4, 6.4, 6.0 s
main	2.93, 2.88, 3.32, 2.91 GB	2501	6.8, 7.0, 6.8, 7.2 s

No branch heap crossed 3.5 GB, so no peak profile was taken; the branch is no longer above main. /labels probes: 0 not 200 in all eight slots (max 0.2 to 1.7 s), VictoriaLogs and Loki restarts 0.

Visual fix proof (Grafana, main and branch next to Loki)

Static window, 15m and 1h, three pages from bench/visual/spec.json, captured on 2dd7a487 (stack: the compose VictoriaLogs and Loki, both proxies, Grafana):

page	main (before)	branch (after)	Loki
`explore-nested-aggregation-outer-by`	1 series, no labels	15 series, one per `service_name`	15 series
`explore-without-over-inner-aggregation`	1 series, no labels	4 series, one per `level`	4 series
`explore-without-nested-under-aggregation`	1 series	1 series	1 series

Data to data, branch against Loki: explore-without-over-inner-aggregation 4 of 4 series, every point identical (360 per series at 15m, 1440 at 1h); explore-without-nested-under-aggregation 15m identical (the 1h Loki capture did not settle); explore-nested-aggregation-outer-by 10 of 15 series identical point for point and the same timestamps in all 15. The other 5 are checkout-svc, inventory-svc, notification-svc, pricing-svc and otel-collector, the OTel streams described above (the same difference exists for the single-level count_over_time on main). Grafana showed no panel error or "No data" on any of the 18 loads. The pages now also run in CI through the spec; screenshots are main.png, pr.png and loki.png per page and range.

Still open

Benchstat of a repeated-window nested tumbling query against Loki (extent cache path).
N14: the sliding evaluator spends its time in the proxy after VictoriaLogs answers (buildHitsRangeMetricMatrix loops every step for every series); the typed fold removed the re-parse between levels, the evaluator is the follow-up.
Register as gap drafts: sum without (l) (rate(...)) alone answers one series; a group/stddev/topk level inside a nest; the over-time unwrap window values on a sliding window inside a nest; the VictoriaLogs-side fold for underscore-named and parser-extracted excluded labels.

CI fixes after review

Security / static (gosec G103): the typed fold's sample-timestamp read used unsafe.String. It now parses whole-number timestamps of up to 15 digits straight from the bytes (exact in a float64) and everything else with strconv.ParseFloat(string(...)), which does not escape. TestScannerNumberMatchesParseFloat holds it bit for bit to strconv.ParseFloat. BenchmarkFoldWithoutMatrix/typed: 30–33 ms against 35–37 ms with unsafe, allocations unchanged.
Visual fix-proof pages: explore-nested-aggregation-outer-by now selects {env="production", instrumentation!="otel", app!="otel-collector"}. On the old selector the five OTel-instrumented services differed from Loki on main and on this branch alike, because Loki adds trace_id and span_id to the sample labels of those lines (pkg/logql/log/metrics_extraction.go:104,109), so the page measured that unrelated difference instead of the nest. explore-without-nested-under-aggregation now uses max(sum without (pod) (count_over_time({app="payment-service"}[5m]))): the old query merged no series on the generator's data, so main already matched Loki there. The new shape is a row of TestWithoutNestedAggregationMatchesLoki.
perf-smoke: the job was cancelled at its 30-minute limit. W8 (sum without (pod) (count_over_time({env="production"}[1m]))) took 1,019 s because Loki timed out at 90 s on every run, including a seven-run confirm pass. This pull request edits W8: it now uses the same OTel-excluding selector, so Loki answers it, as W1–W4 were changed in fix: sum without over an unwrap range function keeps a series per remaining label set, like Loki #739. The next daily Δ for W8 is a measurement change, not a code change.

…r-by-native-routes

github-actions · 2026-10-12T12:10:41Z

🟢 Performance A/B: faster, no regression

173 shape×range — 2 🟢 faster · 171 ⚪ same · results vs Loki: 141 same, 0 new difference(s), 32 pre-existing

	shape	range	base p50	PR p50	change	Loki p50	PR ÷ Loki	result vs Loki
🟢	N14 sliding sum of max	1h	1.86 s	504 ms ¹	🟢 -73%	141 ms	3.57×	✅ same
🟢	W8 sum without pod, count_over_time (production)	1h	438 ms	324 ms ¹	🟢 -26%	77 ms	4.21×	✅ same

¹ re-measured with 7 runs, on windows the first pass did not use, after it moved beyond noise.

171 unchanged shape×range (within noise)

	shape	range	base p50	PR p50	change	Loki p50	PR ÷ Loki	result vs Loki
⚪	bytes_over_time by app	1h	17 ms	14 ms	⚪ -18%	112 ms	0.12×	✅ same
⚪	drilldown service landing	1h	7 ms	10 ms	⚪ +43%	48 ms	0.21×	✅ same
⚪	instant json filtered	instant	10 ms	10 ms	⚪ +0%	96 ms	0.10×	✅ same
⚪	instant json volume	instant	32 ms	32 ms	⚪ +0%	121 ms	0.26×	✅ same
⚪	instant ungrouped rate	instant	7 ms	7 ms	⚪ +0%	95 ms	0.07×	✅ same
⚪	json rate by level (no filter, no drop)	1h	400	400	⚪ +0%	400	—	✅ same
⚪	json volume no filters	1h	41 ms	40 ms	⚪ -2%	95 ms	0.42×	✅ same
⚪	logfmt count by level, filter, drop	1h	14 ms	14 ms	⚪ +0%	80 ms	0.17×	✅ same
⚪	logfmt volume	1h	96 ms	92 ms	⚪ -4%	84 ms	1.10×	✅ same
⚪	logs plain	1h	70 ms	68 ms	⚪ -3%	61 ms	1.11×	✅ same
⚪	logs \| json	1h	108 ms	100 ms	⚪ -7%	88 ms	1.14×	✅ same
⚪	logs \| logfmt	1h	183 ms	147 ms	⚪ -20%	71 ms	2.07×	✅ same
⚪	stream-label count by app	1h	9 ms	12 ms	⚪ +33%	60 ms	0.20×	✅ same
⚪	topk rate by app	1h	39 ms	57 ms	⚪ +46%	110 ms	0.52×	➖ differs (as base)
⚪	ungrouped rate	1h	4 ms	5 ms	⚪ +25%	107 ms	0.05×	✅ same
⚪	DB1 label breakdown pod	1h	77 ms	80 ms	⚪ +4%	293 ms	0.27×	✅ same
⚪	DB2 label breakdown namespace	1h	7 ms	6 ms	⚪ -14%	91 ms	0.07×	✅ same
⚪	DB3 json field breakdown user_id	1h	7 ms	7 ms	⚪ +0%	47 ms	0.15×	✅ same
⚪	DB4 json field breakdown batch_id	1h	55 ms	39 ms	⚪ -29%	76 ms	0.51×	✅ same
⚪	X1 json filter level_extracted, by level	1h	112 ms	117 ms	⚪ +4%	62 ms	1.89×	✅ same
⚪	X2 json group by level_extracted	1h	11 ms	11 ms	⚪ +0%	64 ms	0.17×	✅ same
⚪	X3 json rate, level_extracted equality	1h	10 ms	12 ms	⚪ +20%	60 ms	0.20×	✅ same
⚪	X4 json bytes, level_extracted regexp	1h	106 ms	107 ms	⚪ +1%	115 ms	0.93×	✅ same
⚪	X5 logfmt filter level_extracted, by level	1h	18 ms	18 ms	⚪ +0%	73 ms	0.25×	✅ same
⚪	E1 logs, json parser over stream labels	1h	108 ms	111 ms	⚪ +3%	68 ms	1.63×	✅ same
⚪	E2 logs, logfmt parser over stream labels	1h	81 ms	59 ms	⚪ -27%	46 ms	1.28×	✅ same
⚪	E3 logs, json, existence filter on level_extracted	1h	137 ms	144 ms	⚪ +5%	81 ms	1.78×	✅ same
⚪	E4 logs, logfmt, existence filter on level_extracted	1h	90 ms	87 ms	⚪ -3%	52 ms	1.67×	✅ same
⚪	E5 logs, json, drop level_extracted	1h	108 ms	112 ms	⚪ +4%	63 ms	1.78×	✅ same
⚪	E6 logfmt count by level_extracted	1h	6 ms	6 ms	⚪ +0%	8 ms	0.75×	✅ same
⚪	E7 json count by level_extracted	1h	78 ms	79 ms	⚪ +1%	20 ms	3.95×	✅ same
⚪	E8 instant logfmt by level_extracted	instant	7 ms	7 ms	⚪ +0%	11 ms	0.64×	✅ same
⚪	J1 json no-error count by service_name, one service	1h	22 ms	23 ms	⚪ +5%	39 ms	0.59×	✅ same
⚪	J2 json no-error count by app, namespace	1h	50 ms	44 ms	⚪ -12%	111 ms	0.40×	✅ same
⚪	J3 json no-error count by parsed method	1h	43 ms	42 ms	⚪ -2%	32 ms	1.31×	✅ same
⚪	J4 json no-error count by service_name, all services	1h	28 ms	26 ms	⚪ -7%	150 ms	0.17×	✅ same
⚪	J5 json no-error bytes_rate by app	1h	16 ms	16 ms	⚪ +0%	90 ms	0.18×	✅ same
⚪	A explore volume, 2 filters, drop	1h	115 ms	126 ms	⚪ +10%	117 ms	1.08×	✅ same
⚪	B grouped sum, filter, no drop	1h	84 ms	82 ms	⚪ -2%	91 ms	0.90×	✅ same
⚪	C ungrouped sum, filter, no drop	1h	80 ms	82 ms	⚪ +3%	89 ms	0.92×	✅ same
⚪	D volume, no filters	1h	87 ms	88 ms	⚪ +1%	127 ms	0.69×	✅ same
⚪	E field breakdown pipeline (explore)	1h	106 ms	85 ms	⚪ -20%	123 ms	0.69×	✅ same
⚪	F field breakdown service_version (explore)	1h	104 ms	93 ms	⚪ -11%	70 ms	1.33×	✅ same
⚪	G field breakdown pipeline (drilldown)	1h	10 ms	10 ms	⚪ +0%	48 ms	0.21×	✅ same
⚪	H field breakdown service_version (drilldown)	1h	50 ms	50 ms	⚪ +0%	51 ms	0.98×	✅ same
⚪	I grafana volume of B	1h	79 ms	79 ms	⚪ +0%	60 ms	1.32×	✅ same
⚪	LF13 explore volume, level filter before json	1h	40 ms	40 ms	⚪ +0%	46 ms	0.87×	✅ same
⚪	LF15 drilldown field breakdown, level filter	1h	7 ms	7 ms	⚪ +0%	13 ms	0.54×	✅ same
⚪	LF16 drilldown label breakdown, level filter	1h	45 ms	46 ms	⚪ +2%	12 ms	3.83×	✅ same
⚪	LF20 logfmt count by detected_level	1h	11 ms	12 ms	⚪ +9%	12 ms	1.00×	✅ same
⚪	LF21 logfmt count by detected_level, level filter	1h	7 ms	8 ms	⚪ +14%	10 ms	0.80×	✅ same
⚪	LF23 json count by level, level filter	1h	40 ms	39 ms	⚪ -3%	16 ms	2.44×	✅ same
⚪	LF24 rate of errors by service	1h	16 ms	15 ms	⚪ -6%	90 ms	0.17×	➖ differs (as base)
⚪	LF25 count of warnings, error label dropped	1h	16 ms	16 ms	⚪ +0%	54 ms	0.30×	➖ differs (as base)
⚪	LF26 topk services by errors	1h	780 ms	793 ms	⚪ +2%	88 ms	9.01×	➖ differs (as base)
⚪	LF28 logs, keep detected_level	1h	49 ms	49 ms	⚪ +0%	18 ms	2.72×	✅ same
⚪	LF29 logs, drop detected_level	1h	77 ms	75 ms	⚪ -3%	18 ms	4.17×	✅ same
⚪	LF30 logs, drop level	1h	67 ms	73 ms	⚪ +9%	19 ms	3.84×	✅ same
⚪	LF31 count by detected_level, level dropped	1h	11 ms	11 ms	⚪ +0%	16 ms	0.69×	✅ same
⚪	LF32 count by detected_level, detected_level kept	1h	7 ms	7 ms	⚪ +0%	16 ms	0.44×	✅ same
⚪	LF7 logs, json then level equals	1h	141 ms	146 ms	⚪ +4%	209 ms	0.70×	✅ same
⚪	LF8 logs, logfmt then level regexp	1h	62 ms	62 ms	⚪ +0%	76 ms	0.82×	✅ same
⚪	LS1 logs, level stream label	1h	53 ms	54 ms	⚪ +2%	85 ms	0.64×	✅ same
⚪	LS2 logs, level stream label regexp, all services	1h	84 ms	86 ms	⚪ +2%	74 ms	1.16×	✅ same
⚪	LS3 logs, level in the selector	1h	48 ms	46 ms	⚪ -4%	43 ms	1.07×	✅ same
⚪	LS4 logs, json then level_extracted	1h	170 ms	171 ms	⚪ +1%	181 ms	0.94×	✅ same
⚪	LS5 logs, logfmt then level_extracted regexp	1h	76 ms	75 ms	⚪ -1%	74 ms	1.01×	✅ same
⚪	LS6 level stream label count	1h	4 ms	4 ms	⚪ +0%	54 ms	0.07×	✅ same
⚪	LS7 rate of level stream label by service	1h	8 ms	8 ms	⚪ +0%	89 ms	0.09×	✅ same
⚪	LS8 json count by level_extracted, collision	1h	75 ms	77 ms	⚪ +3%	17 ms	4.53×	✅ same
⚪	O2 explore log volume (Grafana form), underscore structured-metadata filter	1h	43 ms	41 ms	⚪ -5%	61 ms	0.67×	✅ same
⚪	DL1 drilldown field breakdown json-expr, level filter	1h	15 ms	19 ms	⚪ +27%	46 ms	0.41×	✅ same
⚪	DL2 drilldown field breakdown, level + stream-label filter before json	1h	41 ms	41 ms	⚪ +0%	11 ms	3.73×	✅ same
⚪	DL3 drilldown logfmt field breakdown, level filter	1h	101 ms	122 ms	⚪ +21%	109 ms	1.12×	➖ differs (as base)
⚪	DL4 instant count, level filter, json+logfmt, 1h	instant	475 ms	483 ms	⚪ +2%	1.27 s	0.38×	✅ sum ±0.26%
⚪	DL6 explore volume, level filter before json	1h	42 ms	41 ms	⚪ -2%	63 ms	0.65×	✅ same
⚪	N1 sum by app of max by app,pod, count	1h	5 ms	9 ms	⚪ +80%	72 ms	0.12×	✅ same
⚪	N10 instant min of avg, rate	instant	7 ms	8 ms	⚪ +14%	105 ms	0.08×	✅ same
⚪	N11 instant three levels	instant	6 ms	5 ms	⚪ -17%	68 ms	0.07×	✅ same
⚪	N12 instant json inner, count of sum	instant	16 ms	16 ms	⚪ +0%	81 ms	0.20×	✅ same
⚪	N13 instant without under by	instant	6 ms	16 ms	⚪ +167%	109 ms	0.15×	✅ same
⚪	N15 sliding sums only	1h	11 ms	11 ms	⚪ +0%	76 ms	0.14×	✅ same
⚪	N16 innermost without under by, tumbling	1h	6 ms	10 ms	⚪ +67%	60 ms	0.17×	✅ same
⚪	N2 max by namespace of sum by namespace,app, rate	1h	5 ms	7 ms	⚪ +40%	39 ms	0.18×	✅ same
⚪	N3 count by namespace of sum by namespace,app	1h	5 ms	6 ms	⚪ +20%	40 ms	0.15×	✅ same
⚪	N4 avg by app of min by app,pod	1h	6 ms	9 ms	⚪ +50%	44 ms	0.20×	✅ same
⚪	N5 three levels, max min sum	1h	5 ms	5 ms	⚪ +0%	40 ms	0.12×	✅ same
⚪	N6 json inner, sum of max	1h	8 ms	10 ms	⚪ +25%	45 ms	0.22×	✅ same
⚪	N7 logfmt inner, max of sum	1h	6 ms	5 ms	⚪ -17%	14 ms	0.36×	✅ same
⚪	N8 without over a named by level, tumbling	1h	7 ms	8 ms	⚪ +14%	52 ms	0.15×	✅ same
⚪	N9 instant sum of max	instant	8 ms	8 ms	⚪ +0%	84 ms	0.10×	✅ same
⚪	OR1 logs, or alternatives	1h	71 ms	77 ms	⚪ +8%	85 ms	0.91×	✅ same
⚪	OR2 logs, negated or alternatives	1h	78 ms	75 ms	⚪ -4%	65 ms	1.15×	✅ same
⚪	OR3 logs, regexp or alternatives	1h	56 ms	50 ms	⚪ -11%	85 ms	0.59×	✅ same
⚪	OR4 count by level, or alternatives with parser	1h	40 ms	40 ms	⚪ +0%	54 ms	0.74×	✅ same
⚪	OR5 rate, negated regexp or alternatives	1h	5 ms	5 ms	⚪ +0%	109 ms	0.05×	✅ same
⚪	OR6 instant, or alternatives	instant	8 ms	8 ms	⚪ +0%	80 ms	0.10×	✅ same
⚪	P1 api-gateway empty pattern	1h	23 ms	23 ms	⚪ +0%	18 ms	1.28×	✅ same
⚪	P2 api-gateway not empty pattern, line filter	1h	73 ms	64 ms	⚪ -12%	32 ms	2.00×	✅ same
⚪	P3 nginx count_over_time, pattern with spaces	1h	4 ms	4 ms	⚪ +0%	9 ms	0.44×	✅ same
⚪	P4 nginx count_over_time, GET request pattern	1h	6 ms	5 ms	⚪ -17%	7 ms	0.71×	✅ same
⚪	A1 nginx include GET, log query	1h	19 ms	21 ms	⚪ +11%	12 ms	1.75×	✅ same
⚪	A2 nginx exclude GET, log query	1h	36 ms	37 ms	⚪ +3%	15 ms	2.47×	✅ same
⚪	A3 nginx include GET, count_over_time	1h	7 ms	7 ms	⚪ +0%	7 ms	1.00×	✅ same
⚪	A4 nginx exclude GET, count_over_time	1h	7 ms	7 ms	⚪ +0%	8 ms	0.88×	✅ same
⚪	A5 drilldown exclude, volume by detected_level	1h	14 ms	15 ms	⚪ +7%	8 ms	1.88×	✅ same
⚪	A6 nginx include ends on a literal, log query	1h	14 ms	14 ms	⚪ +0%	6 ms	2.33×	✅ same
⚪	A7 nginx exclude ends on a literal, log query	1h	21 ms	22 ms	⚪ +5%	13 ms	1.69×	✅ same
⚪	A8 nginx include two literals, count_over_time	1h	7 ms	7 ms	⚪ +0%	7 ms	1.00×	✅ same
⚪	A9 nginx include two literals, ends on a literal, count_over_time	1h	6 ms	6 ms	⚪ +0%	7 ms	0.86×	✅ same
⚪	P1 logs, json, level equals	1h	119 ms	114 ms	⚪ -4%	88 ms	1.30×	✅ same
⚪	P2 logs, logfmt, level not equal	1h	30 ms	29 ms	⚪ -3%	9 ms	3.22×	✅ same
⚪	P3 logs, json, level existence	1h	133 ms	130 ms	⚪ -2%	68 ms	1.91×	✅ same
⚪	P4 logfmt count by level	1h	7 ms	5 ms	⚪ -29%	8 ms	0.62×	✅ same
⚪	P5 json count by level	1h	41 ms	41 ms	⚪ +0%	20 ms	2.05×	✅ same
⚪	P6 json count, level filter	1h	5 ms	5 ms	⚪ +0%	18 ms	0.28×	✅ same
⚪	P7 json rate by level, level regexp	1h	5 ms	5 ms	⚪ +0%	19 ms	0.26×	✅ same
⚪	P8 json count by a name that is no stream label	1h	40 ms	40 ms	⚪ +0%	17 ms	2.35×	✅ same
⚪	P9 instant logfmt by level	instant	6 ms	6 ms	⚪ +0%	10 ms	0.60×	✅ same
⚪	E1 json duration_ms sum by service	1h	78 ms	78 ms	⚪ +0%	24 ms	3.25×	➖ differs (as base)
⚪	E2 json duration_ms p95 by service	1h	247 ms	249 ms	⚪ +1%	18 ms	14×	✅ same
⚪	E3 logfmt latency_ms rate, production	1h	466 ms	470 ms	⚪ +1%	148 ms	3.18×	➖ differs (as base)
⚪	E4 logfmt size_bytes max by service	1h	43 ms	45 ms	⚪ +5%	17 ms	2.65×	➖ differs (as base)
⚪	E5 json duration_ms sum, instant	instant	27 ms	28 ms	⚪ +4%	31 ms	0.90×	✅ same
⚪	E6 logfmt ttl unit string (Loki 400)	1h	400	400	⚪ +2%	400	—	✅ same
⚪	E7 json hexadecimal trace_id (Loki 400)	1h	400	400	⚪ -5%	400	—	✅ same
⚪	E8 json duration_ms sum by service, namespace prod	1h	180 ms	180 ms	⚪ +0%	113 ms	1.59×	➖ differs (as base)
⚪	E9 logfmt latency_ms sum by service, namespace prod	1h	276 ms	284 ms	⚪ +3%	105 ms	2.70×	➖ differs (as base)
⚪	G1 logfmt amount sum_over_time, no filter	1h	36 ms	35 ms	⚪ -3%	19 ms	1.84×	➖ differs (as base)
⚪	G10 logfmt latency_ms sum_over_time, production	1h	424 ms	430 ms	⚪ +1%	149 ms	2.89×	➖ differs (as base)
⚪	G11 logfmt amount sum_over_time, instant	instant	8 ms	6 ms	⚪ -25%	12 ms	0.50×	✅ same
⚪	G12 json duration_ms sum_over_time, instant	instant	25 ms	26 ms	⚪ +4%	28 ms	0.93×	✅ same
⚪	G2 json duration_ms sum_over_time, no filter	1h	81 ms	77 ms	⚪ -5%	38 ms	2.03×	➖ differs (as base)
⚪	G3 logfmt amount sum_over_time, error filter	1h	37 ms	36 ms	⚪ -3%	18 ms	2.00×	➖ differs (as base)
⚪	G4 json duration_ms sum_over_time, error filter	1h	78 ms	79 ms	⚪ +1%	34 ms	2.32×	➖ differs (as base)
⚪	G5 logfmt amount sum by rate	1h	39 ms	38 ms	⚪ -3%	14 ms	2.71×	✅ same
⚪	G6 json duration_ms sum by rate	1h	86 ms	84 ms	⚪ -2%	32 ms	2.62×	✅ same
⚪	G7 duration() of a unit string	1h	62 ms	58 ms	⚪ -6%	14 ms	4.14×	➖ differs (as base)
⚪	G8 bytes() of a number	1h	56 ms	56 ms	⚪ +0%	10 ms	5.60×	➖ differs (as base)
⚪	G9 json duration_ms max_over_time	1h	86 ms	90 ms	⚪ +5%	139 ms	0.65×	➖ differs (as base)
⚪	W1 sum without pod, max_over_time	1h	45 ms	43 ms	⚪ -4%	23 ms	1.87×	➖ differs (as base)
⚪	W2 sum without pod, sum_over_time	1h	45 ms	41 ms	⚪ -9%	17 ms	2.41×	➖ differs (as base)
⚪	W3 sum without pod, rate	1h	124 ms	85 ms	⚪ -31%	17 ms	5.00×	✅ same
⚪	W4 max without pod, max_over_time	1h	44 ms	42 ms	⚪ -5%	18 ms	2.33×	➖ differs (as base)
⚪	W5 sum without pod, max_over_time, instant	instant	11 ms	7 ms	⚪ -36%	15 ms	0.47×	✅ same
⚪	W6 sum without pod, rate, instant	instant	8 ms	8 ms	⚪ +0%	16 ms	0.50×	✅ same
⚪	W7 topk 5 of sum without pod	1h	45 ms	46 ms	⚪ +2%	18 ms	2.56×	➖ differs (as base)
⚪	W9 label_replace control, instant	instant	5 ms	4 ms	⚪ -20%	90 ms	0.04×	✅ same
⚪	U1 logfmt amount rate	1h	35 ms	36 ms	⚪ +3%	20 ms	1.80×	✅ same
⚪	U2 json duration_ms rate	1h	79 ms	80 ms	⚪ +1%	32 ms	2.50×	✅ same
⚪	U3 json duration_ms rate, error filter	1h	84 ms	80 ms	⚪ -5%	41 ms	1.95×	✅ same
⚪	U4 logfmt latency_ms rate, production	1h	328 ms	331 ms	⚪ +1%	177 ms	1.87×	➖ differs (as base)
⚪	U5 logfmt amount rate, instant	instant	7 ms	8 ms	⚪ +14%	13 ms	0.62×	✅ same
⚪	U6 json duration_ms rate, instant	instant	11 ms	11 ms	⚪ +0%	25 ms	0.44×	✅ same
⚪	U7 json duration_ms sum_over_time, instant	instant	13 ms	14 ms	⚪ +8%	25 ms	0.56×	✅ same
⚪	U8 logfmt amount rate by service (grouped)	1h	38 ms	54 ms	⚪ +42%	14 ms	3.86×	✅ same
⚪	U9 logfmt amount rate_counter (folded per series)	1h	220 ms	196 ms	⚪ -11%	41 ms	4.78×	➖ differs (as base)
⚪	U1 json duration_ms sum by service	1h	79 ms	81 ms	⚪ +3%	25 ms	3.24×	➖ differs (as base)
⚪	U10 instant sum_over_time unwrap	instant	27 ms	29 ms	⚪ +7%	32 ms	0.91×	✅ same
⚪	U11 instant max_over_time missing label	instant	7 ms	7 ms	⚪ +0%	20 ms	0.35×	✅ same
⚪	U12 json duration_ms over every service	1h	240 ms	224 ms	⚪ -7%	63 ms	3.56×	➖ differs (as base)
⚪	U13 logfmt latency_ms over every service	1h	492 ms	486 ms	⚪ -1%	86 ms	5.65×	➖ differs (as base)
⚪	U14 duration() of a unit string	1h	55 ms	58 ms	⚪ +5%	8 ms	7.25×	➖ differs (as base)
⚪	U2 json duration_ms max by service	1h	88 ms	90 ms	⚪ +2%	49 ms	1.84×	➖ differs (as base)
⚪	U3 json duration_ms sliding sum	1h	86 ms	84 ms	⚪ -2%	39 ms	2.15×	➖ differs (as base)
⚪	U4 logfmt size_bytes sum by service	1h	41 ms	41 ms	⚪ +0%	10 ms	4.10×	➖ differs (as base)
⚪	U5 logfmt ttl unit string (no sample)	1h	39 ms	44 ms	⚪ +13%	7 ms	6.29×	✅ same
⚪	U6 missing label, no parser	1h	21 ms	20 ms	⚪ -5%	14 ms	1.43×	✅ same
⚪	U7 missing label, ungrouped sum	1h	19 ms	20 ms	⚪ +5%	15 ms	1.33×	✅ same
⚪	U8 rate of an unwrapped label	1h	86 ms	81 ms	⚪ -6%	18 ms	4.50×	✅ same
⚪	U9 bytes() conversion	1h	55 ms	53 ms	⚪ -4%	13 ms	4.08×	➖ differs (as base)

Cold (first-run) timings — what a user opening a panel sees

shape	range	base cold	PR cold	Loki cold	PR ÷ Loki cold
N14 sliding sum of max	1h	1.82 s	504 ms	138 ms †	3.65×
W8 sum without pod, count_over_time (production)	1h	456 ms	424 ms	71 ms †	5.97×
bytes_over_time by app	1h	68 ms	79 ms	846 ms	0.09×
drilldown service landing	1h	30 ms	34 ms	762 ms	0.04×
instant json filtered	instant	16 ms	12 ms	94 ms	0.13×
instant json volume	instant	32 ms	31 ms	118 ms	0.26×
instant ungrouped rate	instant	7 ms	7 ms	95 ms	0.07×
json rate by level (no filter, no drop)	1h	400	400	400	—
json volume no filters	1h	55 ms	88 ms	795 ms ↺	0.11×
logfmt count by level, filter, drop	1h	461 ms	412 ms	1.67 s	0.25×
logfmt volume	1h	215 ms	208 ms	1.68 s	0.12×
logs plain	1h	92 ms	88 ms	93 ms	0.95×
logs \| json	1h	105 ms	127 ms	87 ms	1.46×
logs \| logfmt	1h	232 ms	212 ms	94 ms	2.26×
stream-label count by app	1h	23 ms	22 ms	779 ms	0.03×
topk rate by app	1h	39 ms	47 ms	845 ms	0.06×
ungrouped rate	1h	11 ms	21 ms	1.00 s	0.02×
DB1 label breakdown pod	1h	127 ms	99 ms	1.12 s	0.09×
DB2 label breakdown namespace	1h	16 ms	16 ms	558 ms	0.03×
DB3 json field breakdown user_id	1h	58 ms	61 ms	513 ms	0.12×
DB4 json field breakdown batch_id	1h	114 ms	93 ms	815 ms	0.11×
X1 json filter level_extracted, by level	1h	2.88 s	2.93 s	968 ms	3.03×
X2 json group by level_extracted	1h	298 ms	307 ms	704 ms	0.44×
X3 json rate, level_extracted equality	1h	264 ms	269 ms	440 ms	0.61×
X4 json bytes, level_extracted regexp	1h	507 ms	377 ms	520 ms	0.72×
X5 logfmt filter level_extracted, by level	1h	636 ms	627 ms	1.01 s	0.62×
E1 logs, json parser over stream labels	1h	148 ms	110 ms	73 ms	1.51×
E2 logs, logfmt parser over stream labels	1h	58 ms	78 ms	56 ms	1.39×
E3 logs, json, existence filter on level_extracted	1h	156 ms	135 ms	70 ms	1.93×
E4 logs, logfmt, existence filter on level_extracted	1h	111 ms	99 ms	55 ms	1.80×
E5 logs, json, drop level_extracted	1h	113 ms	117 ms	63 ms	1.86×
E6 logfmt count by level_extracted	1h	42 ms	42 ms	52 ms	0.81×
E7 json count by level_extracted	1h	1.16 s	1.18 s	210 ms	5.61×
E8 instant logfmt by level_extracted	instant	10 ms	9 ms	13 ms	0.69×
J1 json no-error count by service_name, one service	1h	164 ms	154 ms	231 ms	0.67×
J2 json no-error count by app, namespace	1h	394 ms	394 ms	916 ms	0.43×
J3 json no-error count by parsed method	1h	138 ms	140 ms	215 ms	0.65×
J4 json no-error count by service_name, all services	1h	464 ms	449 ms	965 ms	0.47×
J5 json no-error bytes_rate by app	1h	263 ms	250 ms	642 ms	0.39×
A explore volume, 2 filters, drop	1h	1.96 s	2.21 s	485 ms	4.56×
B grouped sum, filter, no drop	1h	165 ms	121 ms	440 ms	0.27×
C ungrouped sum, filter, no drop	1h	117 ms	115 ms	436 ms	0.26×
D volume, no filters	1h	195 ms	194 ms	795 ms	0.24×
E field breakdown pipeline (explore)	1h	184 ms	172 ms	1.21 s	0.14×
F field breakdown service_version (explore)	1h	1.23 s	1.19 s	854 ms	1.40×
G field breakdown pipeline (drilldown)	1h	44 ms	45 ms	748 ms	0.06×
H field breakdown service_version (drilldown)	1h	83 ms	77 ms	1.07 s	0.07×
I grafana volume of B	1h	122 ms	105 ms	753 ms	0.14×
LF13 explore volume, level filter before json	1h	93 ms	96 ms	446 ms	0.22×
LF15 drilldown field breakdown, level filter	1h	92 ms	87 ms	90 ms	0.97×
LF16 drilldown label breakdown, level filter	1h	53 ms	53 ms	45 ms	1.18×
LF20 logfmt count by detected_level	1h	49 ms	49 ms	51 ms	0.96×
LF21 logfmt count by detected_level, level filter	1h	41 ms	43 ms	44 ms	0.98×
LF23 json count by level, level filter	1h	54 ms	53 ms	135 ms	0.39×
LF24 rate of errors by service	1h	409 ms	413 ms	494 ms	0.84×
LF25 count of warnings, error label dropped	1h	397 ms	396 ms	482 ms	0.82×
LF26 topk services by errors	1h	789 ms	790 ms	500 ms	1.58×
LF28 logs, keep detected_level	1h	48 ms	47 ms	18 ms	2.61×
LF29 logs, drop detected_level	1h	69 ms	73 ms	18 ms	4.06×
LF30 logs, drop level	1h	75 ms	80 ms	19 ms	4.21×
LF31 count by detected_level, level dropped	1h	41 ms	42 ms	113 ms	0.37×
LF32 count by detected_level, detected_level kept	1h	29 ms	25 ms	122 ms	0.20×
LF7 logs, json then level equals	1h	149 ms	144 ms	199 ms	0.72×
LF8 logs, logfmt then level regexp	1h	58 ms	62 ms	71 ms	0.87×
LS1 logs, level stream label	1h	59 ms	56 ms	85 ms	0.66×
LS2 logs, level stream label regexp, all services	1h	102 ms	98 ms	79 ms	1.24×
LS3 logs, level in the selector	1h	42 ms	45 ms	48 ms	0.94×
LS4 logs, json then level_extracted	1h	182 ms	190 ms	183 ms	1.04×
LS5 logs, logfmt then level_extracted regexp	1h	93 ms	90 ms	83 ms	1.08×
LS6 level stream label count	1h	11 ms	14 ms	396 ms	0.04×
LS7 rate of level stream label by service	1h	18 ms	18 ms	700 ms	0.03×
LS8 json count by level_extracted, collision	1h	1.11 s	1.12 s	115 ms	9.72×
O2 explore log volume (Grafana form), underscore structured-metadata filter	1h	192 ms	156 ms	599 ms	0.26×
DL1 drilldown field breakdown json-expr, level filter	1h	482 ms	503 ms	541 ms	0.93×
DL2 drilldown field breakdown, level + stream-label filter before json	1h	62 ms	52 ms	33 ms	1.58×
DL3 drilldown logfmt field breakdown, level filter	1h	632 ms	620 ms	1.27 s	0.49×
DL4 instant count, level filter, json+logfmt, 1h	instant	482 ms	467 ms	1.27 s	0.37×
DL6 explore volume, level filter before json	1h	207 ms	176 ms	695 ms	0.25×
N1 sum by app of max by app,pod, count	1h	48 ms	47 ms	627 ms	0.07×
N10 instant min of avg, rate	instant	8 ms	6 ms	104 ms	0.06×
N11 instant three levels	instant	5 ms	5 ms	71 ms	0.07×
N12 instant json inner, count of sum	instant	19 ms	17 ms	81 ms	0.21×
N13 instant without under by	instant	7 ms	18 ms	105 ms	0.17×
N15 sliding sums only	1h	28 ms	23 ms	449 ms	0.05×
N16 innermost without under by, tumbling	1h	35 ms	54 ms	841 ms	0.06×
N2 max by namespace of sum by namespace,app, rate	1h	16 ms	19 ms	397 ms	0.05×
N3 count by namespace of sum by namespace,app	1h	15 ms	17 ms	404 ms	0.04×
N4 avg by app of min by app,pod	1h	41 ms	47 ms	892 ms	0.05×
N5 three levels, max min sum	1h	21 ms	20 ms	411 ms	0.05×
N6 json inner, sum of max	1h	129 ms	133 ms	388 ms	0.34×
N7 logfmt inner, max of sum	1h	22 ms	22 ms	64 ms	0.34×
N8 without over a named by level, tumbling	1h	45 ms	50 ms	1.09 s	0.05×
N9 instant sum of max	instant	7 ms	10 ms	87 ms	0.11×
OR1 logs, or alternatives	1h	85 ms	72 ms	87 ms	0.83×
OR2 logs, negated or alternatives	1h	75 ms	95 ms	66 ms	1.44×
OR3 logs, regexp or alternatives	1h	62 ms	61 ms	86 ms	0.71×
OR4 count by level, or alternatives with parser	1h	239 ms	120 ms	855 ms	0.14×
OR5 rate, negated regexp or alternatives	1h	58 ms	56 ms	828 ms	0.07×
OR6 instant, or alternatives	instant	8 ms	7 ms	79 ms	0.09×
P1 api-gateway empty pattern	1h	31 ms	22 ms	94 ms	0.23×
P2 api-gateway not empty pattern, line filter	1h	73 ms	62 ms	33 ms	1.88×
P3 nginx count_over_time, pattern with spaces	1h	14 ms	13 ms	30 ms	0.43×
P4 nginx count_over_time, GET request pattern	1h	14 ms	14 ms	28 ms	0.50×
A1 nginx include GET, log query	1h	26 ms	20 ms	12 ms	1.67×
A2 nginx exclude GET, log query	1h	33 ms	33 ms	13 ms	2.54×
A3 nginx include GET, count_over_time	1h	8 ms	9 ms	28 ms ↺	0.32×
A4 nginx exclude GET, count_over_time	1h	21 ms	19 ms	24 ms	0.79×
A5 drilldown exclude, volume by detected_level	1h	48 ms	51 ms	30 ms	1.70×
A6 nginx include ends on a literal, log query	1h	14 ms	15 ms	19 ms	0.79×
A7 nginx exclude ends on a literal, log query	1h	20 ms	23 ms	13 ms	1.77×
A8 nginx include two literals, count_over_time	1h	23 ms	22 ms	22 ms	1.00×
A9 nginx include two literals, ends on a literal, count_over_time	1h	20 ms	21 ms	20 ms	1.05×
P1 logs, json, level equals	1h	126 ms	119 ms	78 ms	1.53×
P2 logs, logfmt, level not equal	1h	30 ms	29 ms	44 ms	0.66×
P3 logs, json, level existence	1h	120 ms	117 ms	179 ms	0.65×
P4 logfmt count by level	1h	30 ms	29 ms	44 ms	0.66×
P5 json count by level	1h	110 ms	109 ms	201 ms	0.54×
P6 json count, level filter	1h	14 ms	15 ms	161 ms	0.09×
P7 json rate by level, level regexp	1h	16 ms	19 ms	177 ms	0.11×
P8 json count by a name that is no stream label	1h	85 ms	89 ms	147 ms	0.61×
P9 instant logfmt by level	instant	6 ms	6 ms	10 ms	0.60×
E1 json duration_ms sum by service	1h	90 ms	79 ms	163 ms	0.48×
E2 json duration_ms p95 by service	1h	254 ms	241 ms	225 ms	1.07×
E3 logfmt latency_ms rate, production	1h	468 ms	460 ms	838 ms	0.55×
E4 logfmt size_bytes max by service	1h	54 ms	43 ms	245 ms	0.18×
E5 json duration_ms sum, instant	instant	28 ms	31 ms	31 ms	1.00×
E6 logfmt ttl unit string (Loki 400)	1h	400	400	400	—
E7 json hexadecimal trace_id (Loki 400)	1h	400	400	400	—
E8 json duration_ms sum by service, namespace prod	1h	182 ms	180 ms	618 ms	0.29×
E9 logfmt latency_ms sum by service, namespace prod	1h	281 ms	276 ms	618 ms	0.45×
G1 logfmt amount sum_over_time, no filter	1h	42 ms	36 ms	73 ms	0.49×
G10 logfmt latency_ms sum_over_time, production	1h	465 ms	464 ms	1.55 s	0.30×
G11 logfmt amount sum_over_time, instant	instant	9 ms	8 ms	14 ms	0.57×
G12 json duration_ms sum_over_time, instant	instant	28 ms	29 ms	29 ms	1.00×
G2 json duration_ms sum_over_time, no filter	1h	81 ms	77 ms	238 ms	0.32×
G3 logfmt amount sum_over_time, error filter	1h	38 ms	36 ms	65 ms	0.55×
G4 json duration_ms sum_over_time, error filter	1h	94 ms	80 ms	240 ms	0.33×
G5 logfmt amount sum by rate	1h	42 ms	43 ms	64 ms ↺	0.67×
G6 json duration_ms sum by rate	1h	107 ms	117 ms	250 ms	0.47×
G7 duration() of a unit string	1h	58 ms	58 ms	55 ms	1.05×
G8 bytes() of a number	1h	55 ms	55 ms	50 ms	1.10×
G9 json duration_ms max_over_time	1h	89 ms	88 ms	2.11 s	0.04×
W1 sum without pod, max_over_time	1h	64 ms	50 ms	101 ms	0.50×
W2 sum without pod, sum_over_time	1h	49 ms	48 ms	81 ms	0.59×
W3 sum without pod, rate	1h	118 ms	89 ms	78 ms	1.14×
W4 max without pod, max_over_time	1h	45 ms	50 ms	95 ms	0.53×
W5 sum without pod, max_over_time, instant	instant	7 ms	7 ms	15 ms	0.47×
W6 sum without pod, rate, instant	instant	7 ms	8 ms	15 ms	0.53×
W7 topk 5 of sum without pod	1h	45 ms	40 ms	93 ms	0.43×
W9 label_replace control, instant	instant	6 ms	5 ms	92 ms	0.05×
U1 logfmt amount rate	1h	45 ms	34 ms	60 ms	0.57×
U2 json duration_ms rate	1h	103 ms	104 ms	169 ms	0.62×
U3 json duration_ms rate, error filter	1h	82 ms	80 ms	181 ms	0.44×
U4 logfmt latency_ms rate, production	1h	331 ms	330 ms	1.33 s	0.25×
U5 logfmt amount rate, instant	instant	8 ms	7 ms	13 ms	0.54×
U6 json duration_ms rate, instant	instant	12 ms	13 ms	25 ms	0.52×
U7 json duration_ms sum_over_time, instant	instant	13 ms	15 ms	24 ms	0.62×
U8 logfmt amount rate by service (grouped)	1h	37 ms	63 ms	64 ms	0.98×
U9 logfmt amount rate_counter (folded per series)	1h	219 ms	186 ms	435 ms	0.43×
U1 json duration_ms sum by service	1h	88 ms	78 ms	159 ms	0.49×
U10 instant sum_over_time unwrap	instant	28 ms	28 ms	31 ms	0.90×
U11 instant max_over_time missing label	instant	7 ms	7 ms	19 ms	0.37×
U12 json duration_ms over every service	1h	232 ms	223 ms	591 ms	0.38×
U13 logfmt latency_ms over every service	1h	486 ms	487 ms	904 ms	0.54×
U14 duration() of a unit string	1h	56 ms	56 ms	42 ms	1.33×
U2 json duration_ms max by service	1h	88 ms	86 ms	756 ms	0.11×
U3 json duration_ms sliding sum	1h	79 ms	80 ms	172 ms	0.47×
U4 logfmt size_bytes sum by service	1h	44 ms	42 ms	40 ms	1.05×
U5 logfmt ttl unit string (no sample)	1h	39 ms	42 ms	34 ms	1.24×
U6 missing label, no parser	1h	21 ms	21 ms	69 ms	0.30×
U7 missing label, ungrouped sum	1h	22 ms	18 ms	69 ms	0.26×
U8 rate of an unwrapped label	1h	108 ms	116 ms	178 ms	0.65×
U9 bytes() conversion	1h	53 ms	53 ms	42 ms	1.26×

† Loki's own counters say its results cache answered this cold run: not a computed time.

↺ Loki cold taken from the first set that ran the same query; Loki's cache answered this one.

Why these shapes

internal/translator/translator.go → 168: control/bytes_over_time by app, control/instant json filtered, control/instant json volume, control/instant ungrouped rate, control/json rate by level (no filter, no drop), control/json volume no filters, control/logfmt count by level, filter, drop, control/logfmt volume …
internal/proxy/vector_matching.go → 47: control/stream-label count by app, nested-aggregation/N1 sum by app of max by app,pod, count, nested-aggregation/N10 instant min of avg, rate, nested-aggregation/N11 instant three levels, nested-aggregation/N12 instant json inner, count of sum, nested-aggregation/N13 instant without under by, nested-aggregation/N14 sliding sum of max, nested-aggregation/N15 sliding sums only …
internal/proxy/metric_agg.go → 27: control/bytes_over_time by app, control/instant json volume, control/instant ungrouped rate, control/json volume no filters, control/stream-label count by app, control/topk rate by app, control/ungrouped rate, drilldown-breakdown-exact/DB1 label breakdown pod …
internal/proxy/range_metric_compat.go → 25: control/bytes_over_time by app, control/instant json volume, control/instant ungrouped rate, control/json volume no filters, control/stream-label count by app, control/topk rate by app, control/ungrouped rate, json-filter-pushdown/A explore volume, 2 filters, drop …
shape added or edited → 17: nested-aggregation/N1 sum by app of max by app,pod, count, nested-aggregation/N10 instant min of avg, rate, nested-aggregation/N11 instant three levels, nested-aggregation/N12 instant json inner, count of sum, nested-aggregation/N13 instant without under by, nested-aggregation/N14 sliding sum of max, nested-aggregation/N15 sliding sums only, nested-aggregation/N16 innermost without under by, tumbling …
internal/proxy/nested_vector_fold.go → 16: nested-aggregation/N1 sum by app of max by app,pod, count, nested-aggregation/N10 instant min of avg, rate, nested-aggregation/N11 instant three levels, nested-aggregation/N12 instant json inner, count of sum, nested-aggregation/N13 instant without under by, nested-aggregation/N14 sliding sum of max, nested-aggregation/N15 sliding sums only, nested-aggregation/N16 innermost without under by, tumbling …
internal/proxy/typed_fold.go → 16: nested-aggregation/N1 sum by app of max by app,pod, count, nested-aggregation/N10 instant min of avg, rate, nested-aggregation/N11 instant three levels, nested-aggregation/N12 instant json inner, count of sum, nested-aggregation/N13 instant without under by, nested-aggregation/N14 sliding sum of max, nested-aggregation/N15 sliding sums only, nested-aggregation/N16 innermost without under by, tumbling …
internal/proxy/vector_agg_fold.go → 16: nested-aggregation/N1 sum by app of max by app,pod, count, nested-aggregation/N10 instant min of avg, rate, nested-aggregation/N11 instant three levels, nested-aggregation/N12 instant json inner, count of sum, nested-aggregation/N13 instant without under by, nested-aggregation/N14 sliding sum of max, nested-aggregation/N15 sliding sums only, nested-aggregation/N16 innermost without under by, tumbling …
internal/translator/nested_vector_chain.go → 16: nested-aggregation/N1 sum by app of max by app,pod, count, nested-aggregation/N10 instant min of avg, rate, nested-aggregation/N11 instant three levels, nested-aggregation/N12 instant json inner, count of sum, nested-aggregation/N13 instant without under by, nested-aggregation/N14 sliding sum of max, nested-aggregation/N15 sliding sums only, nested-aggregation/N16 innermost without under by, tumbling …
internal/translator/without_pushdown.go → 16: nested-aggregation/N1 sum by app of max by app,pod, count, nested-aggregation/N10 instant min of avg, rate, nested-aggregation/N11 instant three levels, nested-aggregation/N12 instant json inner, count of sum, nested-aggregation/N13 instant without under by, nested-aggregation/N14 sliding sum of max, nested-aggregation/N15 sliding sums only, nested-aggregation/N16 innermost without under by, tumbling …
control smoke → 8: control/drilldown service landing, control/instant ungrouped rate, control/json rate by level (no filter, no drop), control/json volume no filters, control/logs plain, control/logs | json, control/stream-label count by app, control/topk rate by app
internal/proxy/proxy.go (endpoint handler) → 8: control/drilldown service landing, control/instant ungrouped rate, control/json rate by level (no filter, no drop), control/json volume no filters, control/logs plain, control/logs | json, control/stream-label count by app, control/topk rate by app

_{base dfa8d0417d → PR 8402cc0192 · 4 runs (first is cold), warm p50 · ranges 1h · noise ±30% and 100 ms · VictoriaLogs restarts 0→0 · 17.5 min. Same runner, one fresh stack, targets interleaved per request. How to read and reproduce: bench/ab/README.md.}

github-actions · 2026-10-12T12:13:17Z

PR Quality Report

Compared against base branch main.

Coverage and tests

Signal	Base	PR	Delta
Test count	13796	15047	1251
Coverage	88.6%	88.6%	0.0% (stable)

Compatibility

Track	Base	PR	Delta
Loki API	100.0%	11/11 (100.0%)	0.0% (stable)
Logs Drilldown	100.0%	17/17 (100.0%)	0.0% (stable)
VictoriaLogs	100.0%	11/11 (100.0%)	0.0% (stable)

Performance smoke

Lower CPU cost (ns/op) is better. Lower benchmark memory cost (B/op, allocs/op) is better. Higher throughput is better. Lower load-test memory growth is better. Benchmark rows are medians from repeated samples.

Signal	Base	PR	Delta
QueryRange cache-hit CPU cost	2133.0 ns/op	2090.0 ns/op	-2.0% (stable)
QueryRange cache-hit memory	1240.0 B/op	1240.0 B/op	0.0% (stable)
QueryRange cache-hit allocations	27.0 allocs/op	27.0 allocs/op	0.0% (stable)
QueryRange cache-bypass CPU cost	2427.0 ns/op	2366.0 ns/op	-2.5% (stable)
QueryRange cache-bypass memory	1283.0 B/op	1281.0 B/op	-0.2% (stable)
QueryRange cache-bypass allocations	27.0 allocs/op	27.0 allocs/op	0.0% (stable)
Labels cache-hit CPU cost	634.7 ns/op	617.2 ns/op	-2.8% (stable)
Labels cache-hit memory	48.0 B/op	48.0 B/op	0.0% (stable)
Labels cache-hit allocations	3.0 allocs/op	3.0 allocs/op	0.0% (stable)
Labels cache-bypass CPU cost	1008.0 ns/op	969.2 ns/op	-3.8% (stable)
Labels cache-bypass memory	396.0 B/op	396.0 B/op	0.0% (stable)
Labels cache-bypass allocations	14.0 allocs/op	14.0 allocs/op	0.0% (stable)

State

Coverage, compatibility, and sampled performance are reported here from the same PR workflow.
This is a delta report, not a release gate by itself. Required checks still decide merge safety.
Performance is a smoke comparison, not a full benchmark lab run.
Delta states use the same noise guards as the quality gate (percent + absolute + low-baseline checks), so report labels match merge-gate behavior.

github-actions · 2026-10-12T12:14:15Z

✅ Visual smoke: passed with warnings

Base dfa8d0417d vs PR 34c972c885, Grafana through the proxy builds and Loki on one fresh stack; run.
Loaded again, once and more patiently, after a first-pass difference or an unsettled page: explore-nested-aggregation-outer-by 15m, explore-nested-aggregation-outer-by 1h, explore-nested-aggregation-outer-by 6h, explore-without-nested-under-aggregation 15m, explore-without-nested-under-aggregation 1h, explore-without-nested-under-aggregation 6h, explore-without-over-inner-aggregation 15m, explore-without-over-inner-aggregation 1h, explore-without-over-inner-aggregation 6h.
Core set: 5 capture(s) at 1h. Detailed set: 25 entries (explore-nested-aggregation-outer-by, explore-without-over-inner-aggregation, explore-without-nested-under-aggregation, explore-metric-plain, lvl-dd-fields-pill, lvl-dd-field-method-pill, lvl-dd-field-pipeline-pill, lvl-dd-label-pod-pill, ...). 37 capture(s), 15m/1h/6h, 24h and 7d stay out of PR runs.
Trimmed to the 1h range for the capture budget: explore-A, explore-B, explore-C, explore-E, explore-F, explore-logs-json, explore-logfmt-metric, explore-logs-extracted, explore-extracted-metric, explore-extracted-filter-metric, explore-extracted-group-metric, explore-logs-plain-level, explore-plain-level-metric, explore-logs-line-filter-or, explore-metric-line-filter-or, explore-unwrap-sum, explore-unwrap-unit-string, explore-unwrap-missing-label, explore-unwrap-conversion-error, explore-unwrap-rate-sum, explore-empty-pattern-filter, explore-pattern-filter-anchored, explore-unwrap-quantile-over-time, explore-json-error-filter-metric, explore-unwrap-max-over-time-end-sample, explore-unwrap-first-last-under-sum, explore-max-by-count-over-time-sliding, dd-labels-namespace, dd-field-service_version, dd-field-pipeline, explore-sum-without-rate, explore-topk-sum-without, explore-sum-without-count.
Not run (capture budget): explore-sum-without-count, explore-topk-sum-without, explore-sum-without-rate, dd-field-pipeline, dd-field-service_version, dd-labels-namespace, explore-max-by-count-over-time-sliding, explore-unwrap-first-last-under-sum, explore-unwrap-max-over-time-end-sample, explore-json-error-filter-metric, explore-unwrap-quantile-over-time, explore-pattern-filter-anchored, explore-empty-pattern-filter, explore-unwrap-rate-sum, explore-unwrap-conversion-error, explore-unwrap-missing-label, explore-unwrap-unit-string, explore-unwrap-sum, explore-metric-line-filter-or, explore-logs-line-filter-or, explore-plain-level-metric, explore-logs-plain-level, explore-extracted-group-metric, explore-extracted-filter-metric, explore-extracted-metric, explore-logs-extracted, explore-logfmt-metric, explore-logs-json, explore-F, explore-E, explore-C, explore-B, explore-A, lvl-explore-volume-level-extracted-stored, lvl-explore-volume-level-label-stored, lvl-explore-volume-drop-level, lvl-explore-volume-keep-dl.

Fix proof

Captures of the exact cases this pull request adds or fixes (registry case, base | PR | Loki). The verdict compares the PR with Loki where the base did not match. The gate fails a case unless one capture is fixed or improved, or, beyond the Loki data window, the base failed where Loki and the PR both answered with data (the label does not excuse it; only visual: none on the case does).

semantics/nested-aggregation-outer-by-native-routes: 🟡 improved

page	range	base = PR (data)	vs Loki (PR)	verdict
explore-nested-aggregation-outer-by	15m	differs (1)	1 difference(s), 1 on base too	🟡 improved
explore-nested-aggregation-outer-by	1h	differs (1)	1 difference(s), 1 on base too	🟡 improved
explore-nested-aggregation-outer-by	6h	differs (1)	n/a (beyond the Loki window)	⚪ undecided

explore-nested-aggregation-outer-by-15m: improved

explore-nested-aggregation-outer-by-1h: improved

explore-nested-aggregation-outer-by-6h: undecided

semantics/without-nested-under-aggregation: 🟡 improved

page	range	base = PR (data)	vs Loki (PR)	verdict
explore-without-nested-under-aggregation	15m	differs (1)	1 difference(s), 1 on base too	🟡 improved
explore-without-nested-under-aggregation	1h	differs (1)	1 difference(s), 1 on base too	🟡 improved
explore-without-nested-under-aggregation	6h	differs (1)	n/a (beyond the Loki window)	⚪ undecided

explore-without-nested-under-aggregation-15m: improved

explore-without-nested-under-aggregation-1h: improved

explore-without-nested-under-aggregation-6h: undecided

semantics/without-over-inner-aggregation-collapses: 🟡 improved

page	range	base = PR (data)	vs Loki (PR)	verdict
explore-without-over-inner-aggregation	15m	differs (1)	1 difference(s), 1 on base too	🟡 improved
explore-without-over-inner-aggregation	1h	differs (1)	1 difference(s), 1 on base too	🟡 improved
explore-without-over-inner-aggregation	6h	differs (1)	n/a (beyond the Loki window)	⚪ undecided

explore-without-over-inner-aggregation-15m: improved

explore-without-over-inner-aggregation-1h: improved

explore-without-over-inner-aggregation-6h: undecided

Regression check

page	range	set	base = PR (data)	vs Loki (PR)	pixel diff	result
dd-fields	1h	core	✅ identical	identical (4 explained, 1 history-dependent)	0.11%	✅
dd-landing	1h	core	✅ identical	identical (3 explained)	0.14%	✅
dd-service-logs	1h	core	✅ identical	identical (3 explained, 1 history-dependent)	0.11%	✅
explore-logs-plain	1h	core	✅ identical	identical (1 explained)	0.07%	✅
explore-metric-plain	1h	core	✅ identical	identical (1 explained)	0.04%	✅
explore-metric-plain	15m	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-metric-plain	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-nested-aggregation-outer-by	15m	detailed	differs (1): query ('min by (service_name) (avg by (service_name, level)	1 difference(s), 1 on base too	1.33%	✅ improved (closer to Loki)
explore-nested-aggregation-outer-by	1h	detailed	differs (1): query ('min by (service_name) (avg by (service_name, level)	1 difference(s), 1 on base too	3.08%	✅ improved (closer to Loki)
explore-nested-aggregation-outer-by	6h	detailed	differs (1): query ('min by (service_name) (avg by (service_name, level)	n/a (beyond the Loki window)	7.14%	✅ improved (closer to Loki)
explore-without-nested-under-aggregation	15m	detailed	differs (1): query ('max(sum without (pod) (count_over_time({app="payment	1 difference(s), 1 on base too	1.19%	✅ improved (closer to Loki)
explore-without-nested-under-aggregation	1h	detailed	differs (1): query ('max(sum without (pod) (count_over_time({app="payment	1 difference(s), 1 on base too	1.50%	✅ improved (closer to Loki)
explore-without-nested-under-aggregation	6h	detailed	differs (1): query ('max(sum without (pod) (count_over_time({app="payment	n/a (beyond the Loki window)	2.16%	✅ improved (closer to Loki)
explore-without-over-inner-aggregation	15m	detailed	differs (1): query ('sum without (service_name) (sum by (service_name, le	1 difference(s), 1 on base too	0.82%	✅ improved (closer to Loki)
explore-without-over-inner-aggregation	1h	detailed	differs (1): query ('sum without (service_name) (sum by (service_name, le	1 difference(s), 1 on base too	2.44%	✅ improved (closer to Loki)
explore-without-over-inner-aggregation	6h	detailed	differs (1): query ('sum without (service_name) (sum by (service_name, le	n/a (beyond the Loki window)	6.16%	✅ improved (closer to Loki)
lvl-dd-field-method-pill	1h	detailed	✅ identical	1 difference(s), 1 on base too; 2 explained	0.11%	✅
lvl-dd-field-pipeline-pill	1h	detailed	✅ identical	1 difference(s), 1 on base too; 2 explained, 1 history-dependent	0.11%	✅
lvl-dd-fields-pill	1h	detailed	✅ identical	25 difference(s), 25 on base too; 4 explained	0.11%	✅
lvl-dd-label-pod-pill	1h	detailed	✅ identical	1 difference(s), 1 on base too; 2 explained, 1 history-dependent	0.11%	✅
lvl-explore-count-warn-by-service	1h	detailed	✅ identical	1 difference(s), 1 on base too; 1 explained	0.04%	✅
lvl-explore-logs-drop-dl	1h	detailed	✅ identical	2 difference(s), 2 on base too; 1 explained	0.07%	✅
lvl-explore-logs-drop-level	1h	detailed	✅ identical	1 difference(s), 1 on base too; 1 explained	0.07%	✅
lvl-explore-logs-json-level	1h	detailed	✅ identical	identical (1 explained)	0.07%	✅
lvl-explore-logs-json-level-extracted-stored	1h	detailed	✅ identical	identical (1 explained)	0.07%	✅
lvl-explore-logs-keep-dl	1h	detailed	✅ identical	1 difference(s), 1 on base too; 1 explained	0.07%	✅
lvl-explore-logs-level-label-stored	1h	detailed	✅ identical	identical (1 explained)	0.07%	✅
lvl-explore-logs-level-selector-stored	1h	detailed	✅ identical	identical (1 explained)	0.07%	✅
lvl-explore-logs-logfmt-level-extracted-stored	1h	detailed	✅ identical	identical (1 explained)	0.07%	✅
lvl-explore-logs-logfmt-level-regexp	1h	detailed	✅ identical	identical (1 explained)	0.07%	✅
lvl-explore-rate-errors-by-service	1h	detailed	✅ identical	1 difference(s), 1 on base too; 1 explained	0.04%	✅
lvl-explore-topk-errors-by-service	1h	detailed	✅ identical	1 difference(s), 1 on base too; 1 explained	0.04%	✅
lvl-explore-volume-dl-prefilter-json	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
lvl-explore-volume-json-level-filter	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
lvl-explore-volume-logfmt-dl	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
lvl-explore-volume-logfmt-dl-filter	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
tail-json-filter	live	detailed	✅ identical	entries in the window: PR 1974, Loki 620, in both 620	5.83%	✅

Gate: a base-vs-PR data difference passes when the PR is closer to Loki, and fails when the PR diverges from Loki or Loki cannot decide it (at a range Loki does not hold, a difference passes when the same page proved it closer to Loki at a shorter range; otherwise label the pull request visual-change-expected to accept an intended change). Also failing: a panel empty on the PR but not on the base, an error banner, panel error, error answer or new "No data" panel on the PR, a PR side that never settled, a difference that did not reproduce on the recapture. Pixel differences only warn.

explore-nested-aggregation-outer-by-1h: pixel diff 3.1%

pixel diff 3.1%
vs Loki: resource /api/datasources/uid/*/resources/index/stats?query=%7Benv%3D%22production%22%2C%20instrume: body: content differs (rows 0 vs 0)

explore-nested-aggregation-outer-by-6h: no Loki data at this range; the same differences match Loki at a shorter range (1): query ('min by (service_name) (avg b

no Loki data at this range; the same differences match Loki at a shorter range (1): query ('min by (service_name) (avg by (service_name, level) (count_over_time({env="production", : series sets differ: 1 only left ["('A', '', '{}')"], 10 only r
pixel diff 7.1%

explore-without-nested-under-aggregation-6h: no Loki data at this range; the same differences match Loki at a shorter range (1): query ('max(sum without (pod) (count

no Loki data at this range; the same differences match Loki at a shorter range (1): query ('max(sum without (pod) (count_over_time({app="payment-service"}[5m])))', 'range', '179308: series sets differ: 3 only left ['(\'A\', \'\', \'{"app": "pay

explore-without-over-inner-aggregation-6h: no Loki data at this range; the same differences match Loki at a shorter range (1): query ('sum without (service_name) (

no Loki data at this range; the same differences match Loki at a shorter range (1): query ('sum without (service_name) (sum by (service_name, level) (count_over_time({env="producti: series sets differ: 1 only left ["('A', '', '{}')"], 4 only ri
pixel diff 6.2%

Core set montages (5): base | PR | Loki

dd-fields-1h

dd-landing-1h

dd-service-logs-1h

explore-logs-plain-1h

explore-metric-plain-1h

All montages of this run: pr-visuals/pr-746.

Loki folds the innermost level first and each level keeps exactly its own grouping labels (pkg/logql/evaluator.go VectorAggEvaluator.Next). The native routes (an instant query, a tumbling window) dropped the outer by of a nested aggregation and answered one series without labels. Each level is now a stats pipe with its own by; a level above the first keeps only labels the previous one kept, and an innermost without is folded over the series rows level by level in the proxy, under the without marker main's fold uses (the outermost level's operator, series rows, the limit on the folded answer). On a sliding window the proxy evaluates the first stats pipe, the series of the innermost level, and folds every level over them: the fold of the vector aggregations also takes a nest of sums and a static without level, the evaluator names the series by the innermost level's labels (they took the outermost level's by position), and a nest spelled sum(...) by (l) reads its range function from the text as written. The shapes the open without rows pinned (max(sum without ...), sum by (ns) (sum without ...), sum without (sum by ...)) now answer Loki's series and values; their pinned answers are updated, and the two rows are fixed.

… a tumbling window

…n chain was checked on

…r-by-native-routes

Adds the nested-aggregation set to bench/ab/shapes.json: 15 shapes of a vector aggregation over another one, on the generator's data. N1-N8 are range queries over tumbling windows (range == step via $__auto): two-level nests of sum, max, min, avg and count, a three-level chain, a `| json` inner level with an explicit extraction, a `| logfmt` inner level, and a `without` over a level that names its labels. N9-N13 are the same forms as instant queries, N13 with an innermost `without`. N14-N15 are sliding windows (range 5m over the step), one of them a nest of sums. An innermost `without` over a tumbling window has no range shape: its series rows are folded in the proxy exactly as the single-level `avg without (pod)` already is on main (about 1 s at 6h and a 10 s step on both builds), so a shape there would only repeat that cost against a base that answered the wrong thing. The selector leaves out the OTel-instrumented streams and the otel-collector: Loki carries their structured metadata (trace_id, span_id) in the series, so an inner level grouped by pod counts per line there, a series identity difference that predates the nest fix. Checked on a 90 minute seed against Loki first (all 200, non-empty): origin/main differs from Loki on N1-N13 and N15 (one series, or the wrong series count); this branch is the same as Loki on all 15. The registry item's `area` now names the source files with their paths (internal/translator/nested_vector_chain.go, internal/proxy/nested_vector_fold.go, internal/proxy/vector_agg_fold.go), so selection.py ties a change to them to the set; generated performance evidence regenerated.

…'s fold decodes once A nest of vector aggregations with an innermost `without` kept a row per stream and bucket and folded every level in the proxy: `avg without (pod) (count_over_time({env="production"}[10s]))` over 6h at a 10 s step took 0.8 to 1.2 s on main, and the same `without` under `min by (app)` 1.44 s on this branch. - A `<op> without (...)` over a tumbling window (range equal to the step), alone or as the innermost level of a nest, is cut in VictoriaLogs when the excluded labels are plain names (letters and digits; not level, detected_level, service_name or a label the service name is derived from; no custom field mapping renames them): the labels of a series are the pairs of its `_stream` text, so the pairs are cut out of it between the stats pipe that counts a series and the one that combines the series now sharing a stream, and the levels above read their labels as columns of the pipes (translator.PushDownWithout). A parser, a sliding window, an instant query and every other name keep the proxy's fold. Measured on a seeded stack, 6h, 10 s step: the single-level form 0.8 to 1.2 s on main, 25 to 70 ms warm and 175 ms cold here; the nest 14 ms against Loki's 45 ms (178 ms cold, Loki 1.26 s). - The proxy's fold of series rows (a sliding window, an instant query, a parser, the names above) decodes the answer once, folds every level over the series and encodes it once (typed_fold.go), where it parsed and wrote the JSON again at every level: 47 ms and 89k allocations against 164 ms and 3.1M on 200 series of 2160 points. The sliding `sum by (app) (max by (app, pod) (count_over_time(...[5m])))` is 28% (6h) and 66% (1h) faster. - A query with one aggregation no longer parses for a nest: the benchmarks of the control queries show the allocations of main (translate: 38 -> 43 per query before, 38 now; handler: +17 to +34 before, identical now). Differential: TestWithoutPushdownMatchesFoldAndLoki (320 random label sets, a label on some streams only, excluded names the cut takes and does not take, values with quotes, commas, backslashes and pair-like text) holds both routes to a reference of Loki's evaluator and to each other; TestWithoutPushdownDirectedStreams covers streams that differ only by the excluded label, first, in the middle and last of the pairs, for every operator and both functions; TestTypedFoldMatchesGenericFold holds the typed fold to the generic one. TestCompat_NestedVectorAggregation gains single-level `without` forms; it and the controls pass on VictoriaLogs v1.49.0, v1.50.0, v1.51.1, v1.52.0 and v1.53.0 (979 subtests on the floor and the newest, 391 on the others). TestWithoutFoldRowBudgetIsNotASeriesLimit no longer expects the row-limit error for a tumbling `sum without` of count_over_time: VictoriaLogs folds it and the two series it answers fit, as Loki answers it.

…tree Under 32 concurrent clients the proxy's peak RSS with the typed fold was 3.53, 3.78, 3.93 and 3.73 GB in four runs against 3.17, 2.88, 3.16 and 3.20 GB for main (same stack, same queries, ABBA). The heap profile taken at each of those peaks held 1.5 GB in fastjson.(*cache).getValue under decodeTypedBody: a document tree of a sliding nest's answer (millions of samples, one allocated value per number, string and array), alive while the fold ran, on top of the evaluator's own matrices that main also holds. decodeTypedBody now scans the bytes into the series directly (16 bytes a sample) and falls back to the generic fold for anything it does not read. On 200 series of 2160 points: 36 ms, 28 MB and 7.3k allocations against 47 ms, 67 MB and 89k for the tree and 143 ms, 99 MB and 3.1M for the generic fold. Re-run, same method, eight slots ABBA: peak RSS 2.87, 3.06, 2.93 and 2.81 GB for the branch against 2.93, 2.88, 3.32 and 2.91 GB for main, and 2672 requests against 2501 in 150 s per slot. TestTypedScannerReadsWhatTheDocumentTreeReads holds the scanner to encoding/json's tree over 800 generated bodies (indented or compact, extra keys and nested stats, escapes in labels, string and number values, null and short samples, result before resultType).

gosec v2.29.0 (Security / static) flagged G103 at typed_fold.go:220, where the scanner handed strconv.ParseFloat an unsafe.String over the body bytes to avoid a string conversion per number. main has no use of unsafe and gosec is clean on it. The scanner now reads a whole-second timestamp (up to 15 digits, which a float64 holds exactly) straight from the bytes, and passes anything else to strconv.ParseFloat through a plain string conversion that does not escape. BenchmarkFoldWithoutMatrix/typed, GOMAXPROCS=1, 20 iterations x3: 35.4-36.9 ms with unsafe.String, 42.6-43.6 ms with only the string conversion, 29.6-33.5 ms with the digit fast path; 28.0 MB and 7310 allocations in all three. TestScannerNumberMatchesParseFloat holds the scanner to strconv.ParseFloat, bit for bit, on integers at and past the 15-digit edge, fractions, signs, exponents and malformed numbers.

…ies identity explore-nested-aggregation-outer-by. On visual-smoke for 0f0ece6 the PR answered the 15 service_name series Loki answers (1365 and 5415 non-zero points on both at 15m and 1h; main answered one series {}), and the first differing series was checkout-svc: 2 against Loki's 1 at 15m, 5 against 1 at 1h. checkout-svc is one of the generator's OTel-instrumented services, whose lines carry trace_id and span_id as structured metadata. Loki puts structured metadata into a sample's labels (pkg/logql/log/metrics_extraction.go:104,109), so count_over_time there is one series per line, each counting 1, and the avg by (service_name, level) over them is 1; VictoriaLogs counts the stream. That is a series identity difference the single-level query has on main too, and the bench/ab nested-aggregation set leaves those streams out for the same reason. The page now uses the set's selector: {env="production", instrumentation!="otel", app!="otel-collector"}. main still collapses the nest on it (its translation ends in an ungrouped `stats min(...)` with either selector) and the PR keeps `stats by (service_name)`. explore-without-nested-under-aggregation. `sum without (service_name)` merges no series on the generator's data (every stream's app equals its service_name), so main's answer, the max over the unmerged streams, was already Loki's: base and PR were identical. The page now asks max(sum without (pod) (count_over_time({app="payment-service"}[5m]))): pods change every generator batch (10 s apart), so a 5m window sums several pods per level, and main reads the max over single pods. Through the handlers on emulated VictoriaLogs, main answers 5 where Loki's evaluation of the same rows is 20 (instant) and the PR 20; on query_range main answers two series {ns=x},{ns=y} where Loki answers one {} = 8,16,20. A tumbling $__auto window would not show it at 15m and 1h: their steps hold one batch each. The payment-service streams carry no structured metadata, which also leaves out the per-line OTel series that kept Loki from settling the old query at 1h.

… the timeout perf-smoke on #746 was cancelled at its 30-minute limit. Of the 1658 s the shapes ran, W8 (sum without (pod) (count_over_time({env="production"}[1m]))) took 1019 s: Loki timed out at 90 s on all 4 runs of the first pass and on all 7 runs of the confirm pass the PR's speedup on W8 (0.58 s against 0.80 s) started. Loki timed out on W8 in the perf-smoke runs of #739 and #734 as well (they had no confirm pass on W8). Under a `without`, Loki keeps the structured metadata of the OTel-instrumented streams (trace_id, span_id) in the series, one series per line; in the daily run its 1h answer filled 80.9 MB of results cache against the 41 series the proxy answers. W8 now uses the selector the nested-aggregation set uses ({env="production", instrumentation!="otel", app!="otel-collector"}), as W1-W4 moved to `| logfmt amount` for the same reason. It still measures the stream-label `sum without` fold. The other 172 selected shapes ran at about 5 s each, so the selection of 20 sets should finish in about 18 to 20 minutes with the stack and seed.

The visual fix proof page of semantics/without-nested-under-aggregation now asks max(sum without (pod) (count_over_time({app="payment-service"}[5m]))). TestWithoutNestedAggregationMatchesLoki held max over a without only for the unwrap rate; it now also holds this shape, range and instant, to Loki's evaluation of the emulator's rows (one series {} = 8,16,20; instant 20). On origin/main the same request answers {ns=x}=2,3,5 {ns=y}=2,4,5 and, at an instant, 5: the max over single pods.

szibis added a commit that referenced this pull request Oct 12, 2026

chore(conformance): record #746 as the fix of nested-aggregation-oute…

3132c63

…r-by-native-routes

github-actions Bot added size/XL Extra large change scope/translator LogQL translator scope/proxy Proxy core scope/docs Documentation bugfix Bug fix labels Oct 12, 2026

szibis added 7 commits October 26, 2026 19:43

test: nested vector aggregations against Loki on an instant query and…

526ff5a

… a tumbling window

docs(registry): state the VictoriaLogs versions the nested aggregatio…

9aa62f1

…n chain was checked on

chore(conformance): visual fix proof page for nested-aggregation-oute…

2448109

…r-by-native-routes

chore(conformance): record #746 as the fix of nested-aggregation-oute…

84b3b7b

…r-by-native-routes

szibis force-pushed the fix/nested-aggregation-outer-by branch from 3132c63 to 2dd7a48 Compare October 27, 2026 03:56

szibis marked this pull request as ready for review October 27, 2026 03:56

github-actions Bot added size/XL Extra large change scope/tests Tests and removed size/XL Extra large change labels Oct 27, 2026

github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 27, 2026

github-advanced-security Bot found potential problems Oct 27, 2026

View reviewed changes

Comment thread internal/proxy/typed_fold.go Fixed

szibis added 4 commits October 27, 2026 12:42

github-actions Bot removed the size/XL Extra large change label Oct 27, 2026

github-actions Bot added the size/XL Extra large change label Oct 27, 2026

szibis merged commit 3ee88f1 into main Oct 27, 2026
74 of 75 checks passed

szibis deleted the fix/nested-aggregation-outer-by branch October 27, 2026 12:32

Navigation Menu

Repository navigation

fix: a nested vector aggregation keeps every level's grouping on instant queries and tumbling windows, like Loki - #746

fix: a nested vector aggregation keeps every level's grouping on instant queries and tumbling windows, like Loki#746
szibis merged 12 commits into
mainfrom
fix/nested-aggregation-outer-by

szibis commented Oct 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

szibis commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Rebase onto main

Tests

Proofs on this branch (rebased)

Proofs

e2e against Loki

Targeted A/B: nested-aggregation set (16 shapes, final head)

Broad A/B

Throughput under load, and what was wrong

Peak memory under load (the 4.3 GB peak)

Visual fix proof (Grafana, main and branch next to Loki)

Still open

CI fixes after review

Uh oh!

github-actions Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Performance A/B: faster, no regression

Uh oh!

github-actions Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Quality Report

Coverage and tests

Compatibility

Performance smoke

State

Uh oh!

github-actions Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Visual smoke: passed with warnings

Fix proof

Regression check

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

szibis commented Oct 12, 2026 •

edited

Loading

Targeted A/B: `nested-aggregation` set (16 shapes, final head)

github-actions Bot commented Oct 12, 2026 •

edited

Loading

github-actions Bot commented Oct 12, 2026 •

edited

Loading

github-actions Bot commented Oct 12, 2026 •

edited

Loading