Repository navigation
perf: unwrap metrics read their buckets from the stats extent cache - #886
Merged
Merged
Conversation
github-actions
Bot
added
size/L
szibis
force-pushed
the
perf/unwrap-conversion-check
branch
3 times, most recently
from
October 28, 2026 09:51
da1b25e to
44ca788
Compare
szibis
force-pushed
the
perf/unwrap-extent-cache
branch
from
October 28, 2026 10:12
0f75dce to
ea111ef
Compare
…ache Every query holding unwrap carries the conversion-error check: it counts the values Loki's conversion rejects (Loki's 400, pkg/logql/evaluator.go:726-733) and the nan values, even with `| __error__=""`. statsExtentPost refused every stats_query_range call of such a request (stats_extent_cache.go, "checked unwraps are not cached"), so a refresh read the whole window from VictoriaLogs: on the A/B refresh pass all 77 unwrap rows were slower than Loki, up to 544 ms against 3 ms (G10 6h). Dropping the refusal alone would have been wrong: vlPostHTTP called the cache before the counting rewrite, so the cache would have sent and stored the plain query, no hit would have been recorded, and Loki's 400 would have become a 200. vlPostHTTP now rewrites the query to its counting form (countingStatsQuery) before statsExtentPost, which keys and stores that query's buckets: the rejected (flag 1) and nan (flag 2) values are groups of their own, local to their bucket like every series of the answer, which the cache rebuilds exactly. The strip hook runs on the answer the cache assembles, stored buckets and fresh ones alike, records the hits and applies the nan groups; the lookup of a hit reads its bucket's rows as before. A read the cache sends itself (statsExtentFetchKey) carries the rewritten query and is not rewritten again. TestUnwrapExtentCacheMatchesUncached runs five unwrap metrics (with and without the error filter, max by, avg over [1m], a [30s] range at step 2m) over a refresh moving a step at a time, a re-run, a window widened back and one moved, cache on against cache off over the same emulated VictoriaLogs: every answer byte for byte the same, Loki's 400 for a rejected value read only in the fresh part (6 answers) and stored in a bucket (18, two answered from stored buckets alone), NaN windows for a nan, and 570 buckets read against 2760. The emulator now models the counting gate's flags, the lookup of the flagged rows and the stored-row read. The e2e TestCompat_UnwrapExtentCacheRefresh asks Loki and the proxy the same moved, repeated and widened windows.
szibis
force-pushed
the
perf/unwrap-extent-cache
branch
from
October 28, 2026 10:19
ea111ef to
8ca2b16
Compare
Contributor
⚪ Performance A/B: no change beyond noise19 shape×range — 19 ⚪ same · results vs Loki: 15 same, 0 new difference(s), 4 pre-existing Loki profiles compared (like for like):
Every shape is within noise of the base build (±30% and 100 ms). 19 unchanged shape×range (within noise)
Cold (first-run) timings — what a user opening a panel sees
† Loki's results cache answered this cold run (its own counters, or identical answers for windows a minute apart): not a computed time, so it is in no ratio. Why these shapes
base |
Contributor
✅ Visual smoke: passed with warningsBase Fix proofCaptures of the exact cases this pull request adds or fixes (registry case, base | PR | Loki). The verdict compares the PR with Loki where the base did not match. The gate fails a case unless one capture is fixed or improved, or, beyond the Loki data window, the base failed where Loki and the PR both answered with data (the label does not excuse it; only
Regression check
Gate: a base-vs-PR data difference passes when the PR is closer to Loki, and fails when the PR diverges from Loki or Loki cannot decide it (at a range Loki does not hold, a difference passes when the same page proved that query to answer like Loki at a shorter range; otherwise label the pull request dd-fields-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-28T10%3A20%3
dd-service-logs-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-28T10%3A20%3
Core set montages (3): base | PR | Lokidd-landing-1h
explore-logs-plain-1h
explore-metric-plain-1h
All montages of this run: |
Contributor
PR Quality ReportCompared against base branch Coverage and tests
Compatibility
Performance smokeLower CPU cost (
State
|
szibis
added a commit
that referenced
this pull request
Oct 28, 2026
…ding sums in VictoriaLogs On the #886 stack A/B the unwrap warm and refresh cells above 2x Loki made no VictoriaLogs call: the extent cache answered them, and the proxy spent 110-170 ms (W1-W4, W7 at 6h; E2; U9 rate_counter) and 1.1-1.4 s (W8 at 6h) evaluating the window again, where Loki answers from its results cache in 3-17 ms. Settled steps (query_range_steps.go): the window evaluator's answers are kept per evaluation time. A step's sample is a function of its window (t-range, t] alone (pkg/logql/range_vector.go:102-189) and the folds and topk work per step (pkg/logql/evaluator.go VectorAggEvaluator.Next), so a request whose start is on the grid of an earlier one takes those steps and evaluates its missing ranges through the handler, one after another, and merges them by series labels, as Loki's results cache merges extents (pkg/storage/chunk/cache/resultscache/cache.go Do, handleHit; queryrangebase/query_range.go:310-350 matrixMerge). Steps are kept with the stats extent cache's rules and storage: older than -query-range-freshness, per tenant, query, step and grid phase, in the response cache tiers (memory, disk, peers), expiring with the oldest bucket they were evaluated from. The merged answer is written in the evaluator's envelope, headers and series order; Loki's series limit and the size limit apply to it; answers of different routes are never merged; anything else evaluates the whole window. Inner evaluations are not counted as requests. Two defects of the first version were found by the fleet proof and are fixed: a missing range was evaluated with bare headers (no nosniff where the stored steps had it), so the merge refused the pieces and every refresh evaluated the whole window; and the headers a writer below the handler adds when the answer goes out (Content-Encoding, Vary and Content-Length of a gzip client) made the answer unkeepable, so a client that accepts gzip, as Grafana does, left no steps. VictoriaLogs folds `sum without (l...)` over count_over_time or bytes_over_time of a selector and line filters on a sliding window bucket by bucket (withSlidingWithoutCut, translator.WithoutStreamCut): the excluded pairs are cut out of _stream and the buckets summed per remaining stream and level, instead of a series per stream folded by the proxy. rate and bytes_rate stay per stream: Loki sums each stream's rate, and the sum of the counts divided once differs in the last bits. This supersedes the unmerged perf/sliding-without branch; its tests, e2e, A/B set and visual page are brought over. The bucket matrix skips to the steps whose windows can hold a series' next bucket (nextHitsStep); quantile_over_time takes each window from a sliding Fenwick count of its buckets' values (same values bit for bit); the per-series evaluator checks cancellation every 256 series; the extent cache's partial store runs off the request path, or at once when no store slot is free. Measured (isolated stacks, Loki 3.7.9, VictoriaLogs v1.52.0, against origin/main, like for like with Loki's defaults): 328 shape x range cells, 23 faster, 141 same, 0 slower, 0 new differences; 92% of 777 cells below Loki; the committed worst rows at 24h warm E2 119x, G10 56x, U4 39.5x, W3 54.7x are now 0.57x, 0.15x, 0.08x, 0.43x. Backend-safety load ABBA: proxy CPU 28 s against 395-419 s at equal offered load, no VictoriaLogs or Loki restart, /labels 200 throughout. Hardening: edge-case tables and Loki-reference differentials over random windows and steps (TestQueryRangeStepsLokiDifferential, TestQueryRangeStepsMatchUncached, TestQueryStepsMergeMatchesWhole, TestSlidingSumWithoutCut*, TestQuantileSliderMatchesLoki, TestQueryRangeStepsBehindMiddleware), fuzz targets FuzzQueryStepsRoundTrip, FuzzSlidingWithoutCut and FuzzQuantileSlider in CI's fuzz-smoke; e2e TestCompat_QueryRangeSettledSteps and TestCompat_SlidingSumWithoutStreamFold; the fleet job runs the settled steps layer of TestFleetPeerCacheProof (peer read, disk loss, owner down, slow peer, all lost equal to Loki) and TestFleet_QueryRangeSettledStepsReachPeers. Registry: quality/query-range-settled-steps, semantics/without-sliding-sum-folded-buckets (locked); effort exceptions on six unwrap items for the rows that stay above Loki.
szibis
added a commit
that referenced
this pull request
Oct 28, 2026
…ding sums in VictoriaLogs On the #886 stack A/B the unwrap warm and refresh cells above 2x Loki made no VictoriaLogs call: the extent cache answered them, and the proxy spent 110-170 ms (W1-W4, W7 at 6h; E2; U9 rate_counter) and 1.1-1.4 s (W8 at 6h) evaluating the window again, where Loki answers from its results cache in 3-17 ms. Settled steps (query_range_steps.go): the window evaluator's answers are kept per evaluation time. A step's sample is a function of its window (t-range, t] alone (pkg/logql/range_vector.go:102-189) and the folds and topk work per step (pkg/logql/evaluator.go VectorAggEvaluator.Next), so a request whose start is on the grid of an earlier one takes those steps and evaluates its missing ranges through the handler, one after another, and merges them by series labels, as Loki's results cache merges extents (pkg/storage/chunk/cache/resultscache/cache.go Do, handleHit; queryrangebase/query_range.go:310-350 matrixMerge). Steps are kept with the stats extent cache's rules and storage: older than -query-range-freshness, per tenant, query, step and grid phase, in the response cache tiers (memory, disk, peers), expiring with the oldest bucket they were evaluated from. The merged answer is written in the evaluator's envelope, headers and series order; Loki's series limit and the size limit apply to it; answers of different routes are never merged; anything else evaluates the whole window. Inner evaluations are not counted as requests. Two defects of the first version were found by the fleet proof and are fixed: a missing range was evaluated with bare headers (no nosniff where the stored steps had it), so the merge refused the pieces and every refresh evaluated the whole window; and the headers a writer below the handler adds when the answer goes out (Content-Encoding, Vary and Content-Length of a gzip client) made the answer unkeepable, so a client that accepts gzip, as Grafana does, left no steps. VictoriaLogs folds `sum without (l...)` over count_over_time or bytes_over_time of a selector and line filters on a sliding window bucket by bucket (withSlidingWithoutCut, translator.WithoutStreamCut): the excluded pairs are cut out of _stream and the buckets summed per remaining stream and level, instead of a series per stream folded by the proxy. rate and bytes_rate stay per stream: Loki sums each stream's rate, and the sum of the counts divided once differs in the last bits. This supersedes the unmerged perf/sliding-without branch; its tests, e2e, A/B set and visual page are brought over. The bucket matrix skips to the steps whose windows can hold a series' next bucket (nextHitsStep); quantile_over_time takes each window from a sliding Fenwick count of its buckets' values (same values bit for bit); the per-series evaluator checks cancellation every 256 series; the extent cache's partial store runs off the request path, or at once when no store slot is free. Measured (isolated stacks, Loki 3.7.9, VictoriaLogs v1.52.0, against origin/main, like for like with Loki's defaults): 328 shape x range cells, 23 faster, 141 same, 0 slower, 0 new differences; 92% of 777 cells below Loki; the committed worst rows at 24h warm E2 119x, G10 56x, U4 39.5x, W3 54.7x are now 0.57x, 0.15x, 0.08x, 0.43x. Backend-safety load ABBA: proxy CPU 28 s against 395-419 s at equal offered load, no VictoriaLogs or Loki restart, /labels 200 throughout. Hardening: edge-case tables and Loki-reference differentials over random windows and steps (TestQueryRangeStepsLokiDifferential, TestQueryRangeStepsMatchUncached, TestQueryStepsMergeMatchesWhole, TestSlidingSumWithoutCut*, TestQuantileSliderMatchesLoki, TestQueryRangeStepsBehindMiddleware), fuzz targets FuzzQueryStepsRoundTrip, FuzzSlidingWithoutCut and FuzzQuantileSlider in CI's fuzz-smoke; e2e TestCompat_QueryRangeSettledSteps and TestCompat_SlidingSumWithoutStreamFold; the fleet job runs the settled steps layer of TestFleetPeerCacheProof (peer read, disk loss, owner down, slow peer, all lost equal to Loki) and TestFleet_QueryRangeSettledStepsReachPeers. Registry: quality/query-range-settled-steps, semantics/without-sliding-sum-folded-buckets (locked); effort exceptions on six unwrap items for the rows that stay above Loki.
szibis
added a commit
that referenced
this pull request
Oct 29, 2026
…ding sums in VictoriaLogs On the #886 stack A/B the unwrap warm and refresh cells above 2x Loki made no VictoriaLogs call: the extent cache answered them, and the proxy spent 110-170 ms (W1-W4, W7 at 6h; E2; U9 rate_counter) and 1.1-1.4 s (W8 at 6h) evaluating the window again, where Loki answers from its results cache in 3-17 ms. Settled steps (query_range_steps.go): the window evaluator's answers are kept per evaluation time. A step's sample is a function of its window (t-range, t] alone (pkg/logql/range_vector.go:102-189) and the folds and topk work per step (pkg/logql/evaluator.go VectorAggEvaluator.Next), so a request whose start is on the grid of an earlier one takes those steps and evaluates its missing ranges through the handler, one after another, and merges them by series labels, as Loki's results cache merges extents (pkg/storage/chunk/cache/resultscache/cache.go Do, handleHit; queryrangebase/query_range.go:310-350 matrixMerge). Steps are kept with the stats extent cache's rules and storage: older than -query-range-freshness, per tenant, query, step and grid phase, in the response cache tiers (memory, disk, peers), expiring with the oldest bucket they were evaluated from. The merged answer is written in the evaluator's envelope, headers and series order; Loki's series limit and the size limit apply to it; answers of different routes are never merged; anything else evaluates the whole window. Inner evaluations are not counted as requests. Two defects of the first version were found by the fleet proof and are fixed: a missing range was evaluated with bare headers (no nosniff where the stored steps had it), so the merge refused the pieces and every refresh evaluated the whole window; and the headers a writer below the handler adds when the answer goes out (Content-Encoding, Vary and Content-Length of a gzip client) made the answer unkeepable, so a client that accepts gzip, as Grafana does, left no steps. VictoriaLogs folds `sum without (l...)` over count_over_time or bytes_over_time of a selector and line filters on a sliding window bucket by bucket (withSlidingWithoutCut, translator.WithoutStreamCut): the excluded pairs are cut out of _stream and the buckets summed per remaining stream and level, instead of a series per stream folded by the proxy. rate and bytes_rate stay per stream: Loki sums each stream's rate, and the sum of the counts divided once differs in the last bits. This supersedes the unmerged perf/sliding-without branch; its tests, e2e, A/B set and visual page are brought over. The bucket matrix skips to the steps whose windows can hold a series' next bucket (nextHitsStep); quantile_over_time takes each window from a sliding Fenwick count of its buckets' values (same values bit for bit); the per-series evaluator checks cancellation every 256 series; the extent cache's partial store runs off the request path, or at once when no store slot is free. Measured (isolated stacks, Loki 3.7.9, VictoriaLogs v1.52.0, against origin/main, like for like with Loki's defaults): 328 shape x range cells, 23 faster, 141 same, 0 slower, 0 new differences; 92% of 777 cells below Loki; the committed worst rows at 24h warm E2 119x, G10 56x, U4 39.5x, W3 54.7x are now 0.57x, 0.15x, 0.08x, 0.43x. Backend-safety load ABBA: proxy CPU 28 s against 395-419 s at equal offered load, no VictoriaLogs or Loki restart, /labels 200 throughout. Hardening: edge-case tables and Loki-reference differentials over random windows and steps (TestQueryRangeStepsLokiDifferential, TestQueryRangeStepsMatchUncached, TestQueryStepsMergeMatchesWhole, TestSlidingSumWithoutCut*, TestQuantileSliderMatchesLoki, TestQueryRangeStepsBehindMiddleware), fuzz targets FuzzQueryStepsRoundTrip, FuzzSlidingWithoutCut and FuzzQuantileSlider in CI's fuzz-smoke; e2e TestCompat_QueryRangeSettledSteps and TestCompat_SlidingSumWithoutStreamFold; the fleet job runs the settled steps layer of TestFleetPeerCacheProof (peer read, disk loss, owner down, slow peer, all lost equal to Loki) and TestFleet_QueryRangeSettledStepsReachPeers. Registry: quality/query-range-settled-steps, semantics/without-sliding-sum-folded-buckets (locked); effort exceptions on six unwrap items for the rows that stay above Loki.
szibis
added a commit
that referenced
this pull request
Oct 29, 2026
… Loki's value (#915) * perf: window-evaluated metrics reuse their settled steps and fold sliding sums in VictoriaLogs On the #886 stack A/B the unwrap warm and refresh cells above 2x Loki made no VictoriaLogs call: the extent cache answered them, and the proxy spent 110-170 ms (W1-W4, W7 at 6h; E2; U9 rate_counter) and 1.1-1.4 s (W8 at 6h) evaluating the window again, where Loki answers from its results cache in 3-17 ms. Settled steps (query_range_steps.go): the window evaluator's answers are kept per evaluation time. A step's sample is a function of its window (t-range, t] alone (pkg/logql/range_vector.go:102-189) and the folds and topk work per step (pkg/logql/evaluator.go VectorAggEvaluator.Next), so a request whose start is on the grid of an earlier one takes those steps and evaluates its missing ranges through the handler, one after another, and merges them by series labels, as Loki's results cache merges extents (pkg/storage/chunk/cache/resultscache/cache.go Do, handleHit; queryrangebase/query_range.go:310-350 matrixMerge). Steps are kept with the stats extent cache's rules and storage: older than -query-range-freshness, per tenant, query, step and grid phase, in the response cache tiers (memory, disk, peers), expiring with the oldest bucket they were evaluated from. The merged answer is written in the evaluator's envelope, headers and series order; Loki's series limit and the size limit apply to it; answers of different routes are never merged; anything else evaluates the whole window. Inner evaluations are not counted as requests. Two defects of the first version were found by the fleet proof and are fixed: a missing range was evaluated with bare headers (no nosniff where the stored steps had it), so the merge refused the pieces and every refresh evaluated the whole window; and the headers a writer below the handler adds when the answer goes out (Content-Encoding, Vary and Content-Length of a gzip client) made the answer unkeepable, so a client that accepts gzip, as Grafana does, left no steps. VictoriaLogs folds `sum without (l...)` over count_over_time or bytes_over_time of a selector and line filters on a sliding window bucket by bucket (withSlidingWithoutCut, translator.WithoutStreamCut): the excluded pairs are cut out of _stream and the buckets summed per remaining stream and level, instead of a series per stream folded by the proxy. rate and bytes_rate stay per stream: Loki sums each stream's rate, and the sum of the counts divided once differs in the last bits. This supersedes the unmerged perf/sliding-without branch; its tests, e2e, A/B set and visual page are brought over. The bucket matrix skips to the steps whose windows can hold a series' next bucket (nextHitsStep); quantile_over_time takes each window from a sliding Fenwick count of its buckets' values (same values bit for bit); the per-series evaluator checks cancellation every 256 series; the extent cache's partial store runs off the request path, or at once when no store slot is free. Measured (isolated stacks, Loki 3.7.9, VictoriaLogs v1.52.0, against origin/main, like for like with Loki's defaults): 328 shape x range cells, 23 faster, 141 same, 0 slower, 0 new differences; 92% of 777 cells below Loki; the committed worst rows at 24h warm E2 119x, G10 56x, U4 39.5x, W3 54.7x are now 0.57x, 0.15x, 0.08x, 0.43x. Backend-safety load ABBA: proxy CPU 28 s against 395-419 s at equal offered load, no VictoriaLogs or Loki restart, /labels 200 throughout. Hardening: edge-case tables and Loki-reference differentials over random windows and steps (TestQueryRangeStepsLokiDifferential, TestQueryRangeStepsMatchUncached, TestQueryStepsMergeMatchesWhole, TestSlidingSumWithoutCut*, TestQuantileSliderMatchesLoki, TestQueryRangeStepsBehindMiddleware), fuzz targets FuzzQueryStepsRoundTrip, FuzzSlidingWithoutCut and FuzzQuantileSlider in CI's fuzz-smoke; e2e TestCompat_QueryRangeSettledSteps and TestCompat_SlidingSumWithoutStreamFold; the fleet job runs the settled steps layer of TestFleetPeerCacheProof (peer read, disk loss, owner down, slow peer, all lost equal to Loki) and TestFleet_QueryRangeSettledStepsReachPeers. Registry: quality/query-range-settled-steps, semantics/without-sliding-sum-folded-buckets (locked); effort exceptions on six unwrap items for the rows that stay above Loki. * fix: sum of rate_counter over one-sample parsed series answers Loki's zeros from stats `sum(rate_counter({service_name="payment-service"} | logfmt | unwrap amount [5m]))` answered 16,471/s at 1h where Loki answers 0, and 502 at 6h and 24h ("manual metric response exceeds 67108864 bytes"): the raw evaluator read every line (`app:="..." | unpack_logfmt | limit 1000001`, over 64 MiB at 6h) and rated all of them as one series. Loki rates each series of the sample extractor, the stream plus every parsed label less the unwrapped one (pkg/logql/log/metrics_extraction.go:160-166, labels.go:664-688); a sum does not hand rate_counter its grouping (pkg/logql/syntax/ast.go:1630-1640), and a series with fewer than two samples in a window rates 0 (pkg/logql/range_vector.go:286-291). With a per-line key (a transaction id) every window is 0. The proxy now proves that case in VictoriaLogs before reading any row: the lines carrying the unwrapped field are counted next to count_uniq_hash of (_stream, the parsed keys of a 1,000-line field_names sample). The tuple is never finer than Loki's series, so equal counts mean every series holds one sample; a collision or a missing key only fails the proof. Then the unwrap rate route's bucket presence answers, every value 0, with its conversion error detection. Otherwise the per-row evaluator answers as before. * perf: a heavy unwrap read reaches Loki's conversion error from its first window E6 and E7 (Loki 400 on a unit string or a hex value) took 81 ms against Loki's 19 at 6h cold and 106-116 ms against 45-58 at 24h: the answer waited for the metric's counting stats read over the whole range before the lookup that builds Loki's text. Loki stops at the first rejected value (pkg/logql/evaluator.go:726-733). When the metric's first counting stats query reads -backend-heavy-query-min-range or more, the conversion check now looks up the first evaluation window (start-range, start] beside it (at most three flagged rows). A row Loki fails on there answers first, in confirm() too, so the answer is the same whether the lookup or the stats read finishes first; the metric's queries are cancelled once it is found. A clean first window leaves the earlier rule (the earliest flagged bucket of the stats read) unchanged. * perf: settled steps evaluate a moved window's light gaps at the same time A moved window misses its old edge and its fresh steps; the two were evaluated one after another, so G7 and D2 at 24h warm made two sequential VictoriaLogs calls (32-37 ms against Loki's 9-15 ms). Gaps that each read less than -backend-heavy-query-min-range are now evaluated concurrently; a heavy gap keeps the sequential order, so a request never holds more than the one heavy-query slot its whole window would. * perf: quantile_over_time reads VictoriaLogs' value counts in place E2 (`quantile_over_time(0.95, ... | json | unwrap duration_ms [1m]) by (service_name)`) at 24h cold spent 260-270 ms in the proxy on top of VictoriaLogs' 330-780 ms (603 ms against Loki's 299). The 9 MB count answer was parsed into a DOM, the 430k samples sorted with sort.Slice and every window's values ranked by a sort of all samples. The counts are now read in place (jsonparser, eachBucketPoint, a pooled parser for each series' labels, as collectRangeMetricHits reads buckets), placed in time order by a counting sort on the bucket grid, and ranked by the sorted distinct values. Fake-VictoriaLogs benchmark of the 24h shape: 204 ms to 83-106 ms per request. * fix: the rate_counter one-sample proof parses the line itself The first version built the distinct tuple from field_names of the base query, which mixes stored VictoriaLogs fields with the base's own unpack: a stored per-line field Loki's `| logfmt` never extracts (a line ingested with its keys as fields) made the tuple finer than Loki's series and could give the stats-only 0 for one series holding many samples. The tuple is now (_stream, the line's logfmt keys parsed again under a prefix of their own), less the unwrapped label and every key whose Loki label name (sanitizeLabelKey, pkg/logql/log/util.go:22-39) another key shares. The proof covers one parameterless `| logfmt` only: `| json` (arrays rendered as text by VictoriaLogs, skipped by Loki), line_format and other parsers keep the per-row evaluator. Edge tests: a unique stored field, keys Loki names alike, a digit-leading key and its twin, a shared stored field, no line. * test: lock the rate_counter one-sample answer and the first-window 400 against Loki - registry: semantics/unwrap-rate-counter-one-sample-series (fixed, perf block), with the U9 shape covering it and a visual page reproducing it; - e2e (group core): TestCompat_UnwrapRateCounterOneSampleSeries (own logfmt fixture with a transaction id per line: sum and sum by, sliding, tumbling, shorter than the step, against Loki's zeros) and TestCompat_UnwrapConversionErrorFirstWindowHeavyRead (a 7h read whose only rejected value is in the first window: Loki's 400 naming that line); - fuzz-smoke: FuzzSortQuantileSamples and FuzzRateCounterOneSample; - generated registry outputs and the README quality section regenerated. * perf: read every point of an instant value-count answer in place An instant quantile over value counts reads a stats_query answer: a vector's value, and a matrix's values when the backend writes one, each at the evaluation time, as the DOM read did (TestSeriesLimit_EveryMetricRouteFollowsLoki). Adds BenchmarkQuantileCounts24h and BenchmarkRateCounterUnderSum6h. * refactor: move the rate_counter proof hooks out of the window evaluator golangci-lint gocyclo: proxyManualRangeMetricRange went over 30 with the proof's branches; startRateCounterProof, rateCounterBucketRoute and rateCounterRawSpec hold them. No behaviour change. * perf: reuse a query's rate_counter proof keys for its moved windows A moved window proved its missing steps with two sequential VictoriaLogs reads (the key sample, then the count); the keys of the query are now kept for 10 minutes (1,024 entries, per proxy, backend and tenant), so a refresh proves with one read beside the presence buckets. Keys an entry lacks only merge series. On a 22h window moved by a step: 58 ms to 8 ms. * docs: changelog and registry for the rate_counter, first-window and quantile changes CHANGELOG entries under Unreleased (Fixed: the rate_counter one-sample answer; Performance: the first-window lookup, concurrent settled-step gaps, the in-place quantile read). Registry: the U9 effort exception is removed (the row answers Loki's value below Loki), the conversion-error exception keeps only E2 cold (1-2x, VictoriaLogs' value-count read), and the gate, duration-seconds and sample-validity exceptions cover their 1-2x rows only, with the measured reason (a VictoriaLogs stream-filter search costs about 15 ms however small its range). Regenerated performance report. * test: the early first-window answer may cancel a stats call before the backend sees it The early-route assertion required the emulator to observe the cancellation; under -race the 400 can come before the stats call reaches the emulator. The test now requires that no stats call answered. * test: drop an unused slice in the rate_counter key reuse test (staticcheck SA4010) * perf: look the first window up beside a heavy read only for a query that erred lately The first-window lookup started beside every heavy counting read, so a clean query paid an extra VictoriaLogs search (on the A/B stack E1 6h cold went from one call in 132 ms to two in 587 ms). The lookup now follows a heavy read's hits (before them, so the reported line is the same either way), and starts beside the read only when the same tenant's query answered Loki's conversion error within 10 minutes (1,024 entries). A query that never errs pays nothing; TestUnwrapOverTimeShortRangeStaysOnStepBuckets is back to its original assertion. * docs: final-head measurements in the changelog and registry The first-window lookup is gated on a recent error, so the E6/E7 entries cite the final-head A/B (first heavy read: 1.2x and 1.6x Loki at 6h cold, was 2.0x and 3.1x) next to the early answer of a repeated read; the quantile entry cites benchstat; the U9 entry gives Loki's time next to the moved window and the key-sample collision it does not cover; the sample-validity exception no longer claims U14 6h (2.2x cold, 2.0x warm, open). * docs: the CHANGELOG entries under Unreleased and the regenerated reports after the rebase * refactor: the sliding stats route writes through one helper after the ranked route * fix: bound settled-step allocation and secure JSON responses * docs(changelog): keep PR entries under Unreleased after rebase * fix: bound peer ring allocation and seed replay scope * Keep query-range security headers and cache parity * Preserve Loki plain-text errors through steps cache tee * Allow race-covered proxy suite to finish in CI * Keep settled-steps race proof within CI budget * Regenerate conformance wiring after main rebase * Restore released changelog after main rebase * Skip bounded visual publishing for full baseline captures * Distinguish delayed visual requests from value flips * Replay interrupted test recorder queries once * Refresh conformance gaps after main rebase * Refresh quality summary after main rebase * Refresh conformance evidence after main rebase * Allow proxy race shard to complete on hosted runners --------- Co-authored-by: szibis <szibis@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Unwrap metrics now use the stats extent cache. A refresh or moved window of a step or more, and a re-run, reads only the VictoriaLogs buckets it has not read before; a window that moves by less than a step is not helped (measured below). Loki's answers stay byte for byte the same, its 400 for a rejected value included.
This builds on the unwrap window change (#885, merged), rebased onto
mainas8ca2b16e.What was wrong
Every query holding
unwrapcarries the conversion-error check. The check counts:pkg/logql/evaluator.go:726-733);It is attached even with
| __error__="", for the nan count.statsExtentPostrefused everystats_query_rangecall of a checked request (stats_extent_cache.go, "checked unwraps are not cached").So a dashboard refresh read the whole window from VictoriaLogs again. On the A/B refresh pass, all 77 unwrap rows were slower than Loki: median 6× at 1h and 20× at 6h. The worst was
sum by (service_name) (sum_over_time({env="production"} | logfmt | unwrap latency_ms [5m]))at 6h: 544 ms against Loki's 3 ms.Simply dropping the refusal would have been wrong.
vlPostHTTPcalled the cache beforecountingStatsQueryrewrote the query, so:Fix
vlPostHTTPnow rewrites the query to its counting form beforestatsExtentPost. The cache keys and stores the counting query's buckets.statsExtentFetchKey) already carries the rewritten query and is not rewritten again.-query-range-extent-cache-ttl). The newest-query-range-freshnessis always read. Loki's results cache has the same window.Tests
TestUnwrapExtentCacheMatchesUncached(unit): cache on against cache off, over the same emulated VictoriaLogs.| __error__="";max by;[1m];[30s]range at step 2m.TestCompat_UnwrapExtentCacheRefresh(e2e, group core, compiled with-tags e2e): an own fixture with one rejected value and one nan, asked of Loki and the proxy over windows that are moved, repeated, moved back and widened. It compares status, Loki's pipeline-error text and every value.Proofs (local, no Docker)
Unit tests capped: pass.
-raceon the touched tests (741 after the rebase): pass.Full capped run,
GOMEMLIMIT=3GiB GOMAXPROCS=4 /usr/bin/time -l go test -short -count=1 -p 1 ./cmd/... ./internal/...: every package ok.250c5d3cc538f410bfaedadd93af511d0f75dceeEvery package ok on an otherwise idle host. The peak depends on how the tests are scheduled (this work's earlier runs ranged from 1.735 to 2.7 GB); this branch peaks below its base and below main in these runs. Full runs taken on a loaded host failed on macOS ephemeral-port exhaustion (
can't assign requested address) or the ten-minutego testtimeout, never on an assertion, and passed when repeated idle.go vet ./...andgo vet -tags e2e ./test/e2e-compat/: clean.gofmt -l: clean.golangci-lint run ./...: 0 issues.Conformance gate ok. Changelog gate ok.
Registry
New (pending owner review):
Proofs on a stack
Colima VM (8 CPU, 14 GB), Loki 3.7.9, VictoriaLogs v1.49.0 and v1.53.0, one stack at a time. Stack-proof heads:
c538f410(the branch this one is stacked on) andbfaedadd(this branch), cut from250c5d3c; rebased onto507915a5they are93af511dandde6fadb7(main gained only #883, a transport retry; gates, unit run and the e2e regex group repeated on the rebased heads, all green; the other numbers carry over). The head0f75dceeisde6fadb7plus a text-only amend that adds the sub-step refresh limitation to the CHANGELOG entry and the registry row (found by the last run below; conformance, changelog, build, vet and lint gates rerun). VictoriaLogs and Loki restart counts were read before and after the e2e, A/B, 24h, 7d (VictoriaLogs) and load runs and were 0 every time; the visual runs did not record them.Status against the bar (warm and refresh of every unwrap row at most 2x Loki, target Loki + 5 ms): not met. 20 of 190 cells fail (E2, U9, W1 to W4, W7, and W8 which is not an unwrap query), and the sub-step refresh below means a dashboard that auto-refreshes faster than its step gets no benefit from this branch at any range: the 7d E1, E3, E4, E8, E9 refresh gap stays open for that pattern, and a 15 s refresh at 24h/60 s is 75 to 600x Loki (measured). What the branch does deliver: a refresh that moves a whole step or more is served from stored buckets (24h: 9 to 39 ms against Loki's 34 to 310 ms; 7d: 5 to 15 ms), with the same answers and the same 400s.
e2e against Loki
go test -tags e2e -count=1 -run '^(TestSetup_IngestLogs|TestCompat_UnwrapExtentCacheRefresh|TestCompat_UnwrapConversionErrorParity|TestCompat_UnwrapOverTimeRangeWindows|TestCompat_UnwrapValueFormsParity|TestCompat_UnwrapOverflowParity)$' ./test/e2e-compat/on an isolated compose project:21:45load 4.46 5.44 7.4821:47load 7.22 6.26 7.49Named tests:
TestCompat_UnwrapConversionErrorParityPASS,TestCompat_UnwrapExtentCacheRefreshPASS,TestCompat_UnwrapOverTimeRangeWindowsPASS,TestCompat_UnwrapOverflowParityPASS,TestCompat_UnwrapValueFormsParityPASS,TestSetup_IngestLogsPASSRebased head
de6fadb7: 571 passed, 0 failed, 87 skipped on v1.49.0 and v1.53.0. The skips are the open forms ofTestCompat_UnwrapValueFormsParityandTestCompat_UnwrapConversionErrorParity/open(registered, by design).A/B: the stacked base, this branch and Loki
pr_smoke.py --base perf/unwrap-conversion-check --head perf/unwrap-extent-cache --ranges 1h,6h --refresh-s 60 --refresh-runs 5overunwrap-conversion-error,unwrap-gate,unwrap-validity,unwrap-ungrouped-sum,unwrap-sum-withoutandcontrol(132 rows, load average 6.3 at the start, 27.2 minutes). Basec538f410, headbfaedadd.Verdict: faster, no regression. 8 faster, 124 within noise; results against Loki: 120 same, 0 new differences, 12 pre-existing. Above 2x Loki: 0 new, 0 worsened, 22 on the committed list, 28 cleared. No cell (cold, warm or refresh) is slower than the base by more than 50 ms and 25%; 66 are faster by that much.
The verdict-level movers (6h): E2 403 to 161 ms, E3 443 to 3 ms, E8 190 to 5 ms, E9 274 to 1 ms, G10 402 to 2 ms, U4 284 to 2 ms, U12 230 to 3 ms, U13 415 to 2 ms.
The bar: unwrap warm and refresh at most 2x Loki (target Loki + 5 ms). Over the 190 warm and refresh cells of the five unwrap sets: 170 meet 2x (or sit within the 50 ms floor) and 160 meet Loki + 5 ms. The 20 that do not, by form:
quantile_over_time(0.95, ... unwrap ...)6hfetchQuantileCounts), not the stats extent cacherate_counterfolded per series, 1h and 6hsum/max without (pod)over an unwrap function, 6hproxyManualRangeMetricRange), which reads its buckets from VictoriaLogs on every requestsum without (pod) (count_over_time(...)), 1h and 6h (not an unwrap query)Every other unwrap row (E1, E3 to E9, G1 to G12, U1 to U8, U10 to U14) meets the 2x bar on warm and refresh; 10 of those cells are between Loki + 5 ms and 2x. The forms above are not served by the stats extent cache at all, so they are outside the claim of this change; they stay above 2x and are the next item (route the window and quantile reads through the extent cache, and fold from stored buckets).
Rows above 2x Loki (PR ÷ Loki above 2 and more than 50 ms over)
The tool's "Above 2x Loki" section for this run lists 0 new and 0 worsened rows (22 are on the committed list, 28 cleared). All 25 cells above 2x are below; the base column shows that none was above 2x by less on the base:
25 mode-rows above 2x of 132 shape x range rows.
Long ranges
Same stack and table as the base branch's description (E1 E3 E4 E8 E9, main, the stacked base, this branch, Loki). 24h (step 60 s, refresh = one step): cold equals the base, warm 1 ms (response cache), refresh drops from 0.29 to 2.94 s on the base to 9 to 39 ms here, at or below Loki's 34 to 310 ms:
Answers equal to Loki's on all of them (E3 has the pre-existing second series).
7d (VictoriaLogs only, Loki infeasible at this length on the stack, see the base branch): a refresh that moves a whole step (600 s) is served from stored buckets, 5 to 15 ms against 0.78 to 8.4 s on the base:
A refresh that moves less than a step is not cached. At 7d, windows moving 60 s on a 600 s step (inside one step bucket): this branch equals its base within noise (E1 1.00 s against 1.32 s, E3 9.27 s against 9.22 s, E4 0.70 s against 0.74 s, E8 3.75 s against 3.95 s, E9 5.90 s against 5.73 s). The same at 24h, measured against Loki (step 60 s, windows moving 15 s, five runs, same stack as the 24h table):
Loki answers a refresh inside a step in 3 to 5 ms (its results cache, with
align_queries_with_step: truein the stack's Loki config); the three proxy builds read VictoriaLogs again (0.23 to 2.3 s), this branch included, so a 5 s, 10 s or 30 s auto-refresh gets nothing from the extent cache at 24h and at 7d. The cause is not isolated (the bucket edges probably follow the request's start). The CHANGELOG entry and the registry row now say so; it is the first follow-up.Load: the backend-safety test
The ABBA run of the base branch's description is this branch's (main, this branch, this branch, main; this branch contains the other):
Same status counts, p50 and p95 as main within noise,
/labels200 on every probe, peak RSS of this branch 2.4 to 2.6 GB against main's 3.1 GB, no restarts.Visual
ci_run.py --base perf/unwrap-conversion-check(the stacked base against this branch against Loki), 23 captures (the unwrap pages, the core Explore pages, Drilldown landing, service logs and fields; 15m, 1h, 6h): base and branch are identical on every request of every page (main = PRon 23 of 23). Against Loki the panel data is identical; theindex/statsdifference is listed unexplained on this stack only because its Loki counted every line twice (entries 99,772 against 49,886, chunks 3,922 against 1,961, the same streams); the metric queries are identical.Uptime and restarts
Not covered
rate_counterper series, thewithoutsliding-window route and the non-unwrapwithoutfold of W8) do not use the stats extent cache./select/logsql/querycalls are not cached by the extent cache: the sliding-window recurrence grids of topk, and the unwrap lookups. Separate item.stats_querycalls (instant queries) are not cached by the extent cache, as before.