Skip to content

fix: label_replace adds its label to instant answers and to post-aggregated series, like Loki - #744

Merged
szibis merged 17 commits into
mainfrom
fix/label-replace-new-label
Oct 28, 2026
Merged

fix: label_replace adds its label to instant answers and to post-aggregated series, like Loki#744
szibis merged 17 commits into
mainfrom
fix/label-replace-new-label

Conversation

@szibis

@szibis szibis commented Oct 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Gap semantics/label-replace-new-label (cluster a9d4f85c, instant queries). On Loki 3.7.7, label_replace works per step on both a vector and a matrix (pkg/logql/evaluator.go:1275-1309). The proxy's applyLabelReplace/applyLabelJoin decoded only a matrix, so an instant vector came back without the new label.

Fixed, matching Loki:

  • Transform on both shapes. transformSeriesLabels rewrites labels on a vector or a matrix and keeps the rest of the answer, e.g. data.stats.
  • Loki's label_replace rules. The regexp is anchored and the replacement is expanded with ExpandString. A series with no match is unchanged, and an empty expansion removes the destination label.
  • Duplicate label sets. These follow Loki's JoinSampleVector/vectorsToSeries (engine.go:440-470, 530-600): an instant query keeps both samples, and a range query files the points under one series.
  • Argument strings. These are unquoted like Loki's lexer (syntax/lex.go:191-198). "\\." is now the regexp \., and escaped quotes work.
  • No internal markers sent to VictoriaLogs. topk/bottomk/sort/stddev over a label_replace sent an internal marker to VictoriaLogs and got a 400. The transform now runs on the inner expression before the ranking, as Loki evaluates it.
  • query_range routing. label_replace around a binary expression now routes by the inner expression, as fix: an instant query keeps its failure status and always answers a resultType #735 does for instant.
  • Without-marker parsing. label_replace(sum without (level)(...)) no longer loses its transform.
  • Malformed calls. Wrong arity, an unquoted argument or an invalid escape is now a 400 parse error instead of a log query.

VictoriaLogs still does all aggregation. The proxy only rewrites labels on the aggregated series.

Tests

  • TestLabelReplaceEdgeTable: 29 rules × matrix and vector.
  • Duplicate label sets; label_join; instant and range; the registry row's query for curl, Explore and Drilldown clients; bare range aggregation; post-aggregations; sum without; malformed arguments; invalid regexp.
  • TestLabelReplaceDifferential: random nested transforms on both endpoints. Its reference is independent for the plumbing (translator markers, both endpoints, nested chains) and for Loki's duplicate merge; the per-series rule reuses Loki's ExpandString sequence.
  • Translator tests for unquoting and marker parsing.
  • e2e vs Loki: TestCompat_LabelReplaceNewLabel, about 38 shapes × query_range and query.

Local proofs (e2e vs Loki 3.7.7 on VictoriaLogs v1.52.0 and v1.47.0, A/B base e9b5ef1 vs branch)

Changes since the first push

The first e2e run in CI found two product bugs and two test problems; all are fixed here.

  • An instant label_replace(count_over_time({env="production"}[5m]), ...) (any bare range function) was one unlabeled {} total instead of one series per stream. The grouping by stream was added only for a query that itself starts with the function; it now follows the expression under the transform (preserveMetricStreamIdentity, and the same check in the two handlers).
  • A label_replace around a metric over | json skipped the evaluator that owns JSON parser errors: a line that is not JSON no longer failed the query, and | json | __error__="" kept every stream (14 apps) instead of the ones that parse (8, as Loki). It is now answered by that evaluator and the transform is applied to its answer.
  • Test: Loki's bare range-function series carry the entry's structured metadata and the proxy's keep the stream (metric-identity/structured-metadata-splits-series, owner-kept), so the two bare shapes leave those label names out of both sides; app_short, buzz, the stream labels and detected_level are still compared.
  • Test: topk(2, ...) over tied values (several apps at 6) lets Loki pick different ties per step; the shape is now topk(1, ...), where api-gateway is the unique maximum.

Unit tests: TestLabelReplaceOverBareRangeFunctionKeepsOneSeriesPerStream and TestLabelReplaceOverJSONMetricUsesTheOrderedJSONEvaluator; both fail without the fixes. go test -short of internal/translator, internal/proxy, internal/logql, go vet (also -tags e2e) and golangci-lint v2.14.0 are clean.

e2e vs Loki 3.7.7, fresh isolated stack per VictoriaLogs version

VictoriaLogs RestartCount 0 before and after each run (no OOM).

test v1.52.0 v1.47.0
TestCompat_LabelReplaceNewLabel pass, 70 subtests pass, 70
TestCompat_LabelReplaceDuplicatesBeforeRanking pass pass
TestMissingOps_LabelReplace pass, 1 pass + 1 skip (label_replace_sum, skipped by the test itself: the aggregation-over-label_replace gap, now registered) same
TestCompat_InstantResultTypeAlwaysSet pass, 32 pass, 32
TestLogQL_Exhaustive_ErrorParity / _QueryParity / _KnownGaps pass, 73 / 284 / 3 same
TestHardeningLive_ParserError* (4 tests, JSON parser-error parity, regression set for fix 2) pass, 64 / 8 / 16 / 64 same

Notes:

  • TestCompat_LabelReplaceDuplicatesBeforeRanking passes on this data although semantics/label-replace-duplicates-before-ranking is open: the shape it ranks does not reproduce a point-count difference here. The row stays open.
  • The two shapes that compare the points per step (two series with one label set, all labels removed, query_range) pass on a fresh stack and fail (153 points on Loki, 140 on the proxy) on every re-run against the same stack: a re-run ingests the fixture again, and VictoriaLogs counts the duplicates where Loki deduplicates them. They are not loosened; run the group once per fresh stack, as CI does.

A/B (base = e9b5ef1, branch = 4e21fc8, which holds the fixes; 1h with Loki, 5 runs, warm p50, one stack, targets interleaved)

Affected shapes (set label-replace):

shape range Loki p50 base p50 branch p50 base vs Loki branch vs Loki
L1 label_replace over sum by rate instant 109 ms 5 ms 5 ms same same
L2 label_replace over sum by rate 1h 166 ms 29 ms 35 ms same same
L3 topk over label_replace over sum by count 1h 70 ms 22 ms 48 ms differs differs
L4 label_replace over bare count_over_time instant 13 ms 3 ms 7 ms differs (one {} total) same
L5 label_replace over | json | __error__="" sum by instant 14 ms 18 ms 50 ms same same
L6 label_replace over | json | __error__="" sum by 1h 15 ms 21 ms 554 ms same same
  • The A/B compares values and series, not label names; the labels are proven by the e2e above.
  • L3 differs from Loki on base and branch alike (topk ties; the e2e shape uses topk(1)).
  • L5 and L6 move from the cheap VictoriaLogs stats query to the evaluator that reads rows, which is what makes the answer match Loki on streams with invalid JSON. Their cost is the metric's own: the same query without the label_replace takes 531 ms on the base and 509 ms on the branch for a 1h range, and 40 ms and 41 ms instant. The A/B flags L6 as the one slower shape; I did not hide it. On this generator data every api-gateway line is valid JSON, so base and branch give the same series here; on a stream with invalid lines the base answered wrongly (14 apps against Loki's 8 in the e2e fixtures).

Control set (20 shapes), 1h with Loki: 20 of 20 same, VictoriaLogs restarts 0 to 0, new differences from Loki 0, pre-existing 1 (topk rate by app, also on base). Largest absolute increases: instant json filtered 150 to 163 ms, drilldown service landing 10 to 14 ms, logs | logfmt 42 to 45 ms.

Saved results: bench/ab/results/2026-10-12-labelreplace-1h.json and 2026-10-12-labelreplace-control-1h.json; shape set label-replace.

Registry

  • Fixed row semantics/label-replace-new-label (text and planned tests updated).
  • New open rows: semantics/label-replace-aggregation-over-transform (an aggregation over a label_replace is a 400 on the proxy and answers on Loki) and semantics/label-replace-around-ranking-and-vector (a label_replace around topk, sort or vector()). Both answer the same on the base, so neither is new. The PR body's draft about a binary operand (label_replace(...) * 2) no longer reproduces and is not registered.
  • label_join is accepted by the proxy but not part of Loki 3.7.7's grammar; it is handled by fix: reject group(), label_join() and other names Loki does not have, and accept keywords in any case, like Loki #748 and only noted in the row text here.
  • Visual: the row keeps visual: none (the visible difference is on instant queries, which the visual harness does not capture; the json difference needs a stream with invalid lines).

Update: rebased onto main (2026-10-28)

Rebased onto main; this is the reconciled branch, with the conflicts resolved and the generated reports regenerated. One new commit adds the perf: blocks that main's conformance gate now requires for the four label_replace rows the A/B shapes cover. The CHANGELOG entries are under [Unreleased] only.

Checks:

  • GOMEMLIMIT=2GiB GOMAXPROCS=4 go test -short -count=1 -p 1 ./internal/proxy/ ./internal/translator/ passes.
  • go vet ./... and go vet -tags e2e ./test/e2e-compat/ pass.
  • The conformance and changelog gates pass.

Open before ready: L6 label_replace over json sum by at 1h measured 0.54 s cold against Loki's 0.11 s (about 5×) and 0.55 s warm against 0.01 s. That is above the 2× Loki bound, so it is fixed or documented as high effort before this is ready.

#741 (aggregation over a binary) is stacked on this PR.

szibis added a commit that referenced this pull request Oct 12, 2026
@szibis
chore(conformance): record #744 as the fix of label-replace-new-label
3f174b0
@github-actions github-actions Bot added size/XL Extra large change scope/translator LogQL translator scope/proxy Proxy core scope/docs Documentation scope/tests Tests bugfix Bug fix and removed size/XL Extra large change labels Oct 12, 2026
@github-actions

github-actions Bot commented Oct 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🟢 Performance A/B: faster, no regression

181 shape×range — 1 🟢 faster · 180 ⚪ same · results vs Loki: 172 same, 0 new difference(s), 9 pre-existing

Loki profiles compared (like for like):

  • aligned: not run (no build supports -align-queries-with-step, so every aligned row would be not like-for-like)
  • loki-defaults: 181 shape×range; proxy flag base off, pr off
shape range Loki profile base p50 PR p50 change Loki p50 PR ÷ Loki result vs Loki
🟢 LF26 topk services by errors 1h defaults 661 ms 15 ms ¹ 🟢 -98% 71 ms 0.21× ➖ differs (as base)

¹ re-measured with 7 runs, on windows the first pass did not use, after it moved beyond noise.

180 unchanged shape×range (within noise)
shape range Loki profile base p50 PR p50 change Loki p50 PR ÷ Loki result vs Loki
⚪ bytes_over_time by app 1h defaults 11 ms 10 ms ⚪ -9% 90 ms 0.11× ✅ same
⚪ drilldown service landing 1h defaults 8 ms 7 ms ⚪ -12% 41 ms 0.17× ✅ same
⚪ instant json filtered instant defaults 8 ms 8 ms ⚪ +0% 73 ms 0.11× ✅ same
⚪ instant json volume instant defaults 23 ms 23 ms ⚪ +0% 92 ms 0.25× ✅ same
⚪ instant ungrouped rate instant defaults 7 ms 6 ms ⚪ -14% 74 ms 0.08× ✅ same
⚪ json rate by level (no filter, no drop) 1h defaults 400 400 ⚪ +0% 400 — ✅ same
⚪ json volume no filters 1h defaults 31 ms 33 ms ⚪ +6% 78 ms 0.42× ✅ same
⚪ logfmt count by level, filter, drop 1h defaults 10 ms 11 ms ⚪ +10% 62 ms 0.18× ✅ same
⚪ logfmt volume 1h defaults 69 ms 69 ms ⚪ +0% 64 ms 1.08× ✅ same
⚪ logs plain 1h defaults 86 ms 84 ms ⚪ -2% 49 ms 1.71× ✅ same
⚪ logs | json 1h defaults 121 ms 110 ms ⚪ -9% 68 ms 1.62× ✅ same
⚪ logs | logfmt 1h defaults 145 ms 166 ms ⚪ +14% 56 ms 2.96× ✅ same
⚪ stream-label count by app 1h defaults 7 ms 8 ms ⚪ +14% 52 ms 0.15× ✅ same
⚪ topk rate by app 1h defaults 36 ms 12 ms ⚪ -67% 87 ms 0.14× ✅ sum ±0.04%
⚪ ungrouped rate 1h defaults 5 ms 4 ms ⚪ -20% 107 ms 0.04× ✅ same
⚪ DB1 label breakdown pod 1h defaults 57 ms 50 ms ⚪ -12% 234 ms 0.21× ✅ same
⚪ DB2 label breakdown namespace 1h defaults 7 ms 7 ms ⚪ +0% 68 ms 0.10× ✅ same
⚪ DB3 json field breakdown user_id 1h defaults 7 ms 7 ms ⚪ +0% 62 ms 0.11× ✅ same
⚪ DB4 json field breakdown batch_id 1h defaults 45 ms 45 ms ⚪ +0% 64 ms 0.70× ✅ same
⚪ X1 json filter level_extracted, by level 1h defaults 92 ms 109 ms ⚪ +18% 76 ms 1.43× ✅ same
⚪ X2 json group by level_extracted 1h defaults 9 ms 9 ms ⚪ +0% 54 ms 0.17× ✅ same
⚪ X3 json rate, level_extracted equality 1h defaults 8 ms 8 ms ⚪ +0% 47 ms 0.17× ✅ same
⚪ X4 json bytes, level_extracted regexp 1h defaults 88 ms 90 ms ⚪ +2% 109 ms 0.83× ✅ same
⚪ X5 logfmt filter level_extracted, by level 1h defaults 16 ms 15 ms ⚪ -6% 60 ms 0.25× ✅ same
⚪ E1 logs, json parser over stream labels 1h defaults 97 ms 103 ms ⚪ +6% 52 ms 1.98× ✅ same
⚪ E2 logs, logfmt parser over stream labels 1h defaults 61 ms 49 ms ⚪ -20% 43 ms 1.14× ✅ same
⚪ E3 logs, json, existence filter on level_extracted 1h defaults 126 ms 111 ms ⚪ -12% 66 ms 1.68× ✅ same
⚪ E4 logs, logfmt, existence filter on level_extracted 1h defaults 79 ms 71 ms ⚪ -10% 46 ms 1.54× ✅ same
⚪ E5 logs, json, drop level_extracted 1h defaults 93 ms 88 ms ⚪ -5% 50 ms 1.76× ✅ same
⚪ E6 logfmt count by level_extracted 1h defaults 9 ms 8 ms ⚪ -11% 10 ms 0.80× ✅ same
⚪ E7 json count by level_extracted 1h defaults 61 ms 60 ms ⚪ -2% 17 ms 3.53× ✅ same
⚪ E8 instant logfmt by level_extracted instant defaults 6 ms 5 ms ⚪ -17% 7 ms 0.71× ✅ same
⚪ J1 json no-error count by service_name, one service 1h defaults 21 ms 21 ms ⚪ +0% 30 ms 0.70× ✅ same
⚪ J2 json no-error count by app, namespace 1h defaults 46 ms 43 ms ⚪ -7% 88 ms 0.49× ✅ same
⚪ J3 json no-error count by parsed method 1h defaults 35 ms 33 ms ⚪ -6% 25 ms 1.32× ✅ same
⚪ J4 json no-error count by service_name, all services 1h defaults 23 ms 22 ms ⚪ -4% 95 ms 0.23× ✅ same
⚪ J5 json no-error bytes_rate by app 1h defaults 11 ms 12 ms ⚪ +9% 72 ms 0.17× ✅ same
⚪ A explore volume, 2 filters, drop 1h defaults 137 ms 127 ms ⚪ -7% 389 ms 0.33× ✅ same
⚪ B grouped sum, filter, no drop 1h defaults 65 ms 64 ms ⚪ -2% 118 ms 0.54× ✅ same
⚪ C ungrouped sum, filter, no drop 1h defaults 63 ms 61 ms ⚪ -3% 114 ms 0.54× ✅ same
⚪ D volume, no filters 1h defaults 68 ms 80 ms ⚪ +18% 149 ms 0.54× ✅ same
⚪ E field breakdown pipeline (explore) 1h defaults 59 ms 58 ms ⚪ -2% 111 ms 0.52× ✅ same
⚪ F field breakdown service_version (explore) 1h defaults 86 ms 86 ms ⚪ +0% 129 ms 0.67× ✅ same
⚪ G field breakdown pipeline (drilldown) 1h defaults 9 ms 9 ms ⚪ +0% 105 ms 0.09× ✅ same
⚪ H field breakdown service_version (drilldown) 1h defaults 41 ms 38 ms ⚪ -7% 115 ms 0.33× ✅ same
⚪ I grafana volume of B 1h defaults 62 ms 60 ms ⚪ -3% 113 ms 0.53× ✅ same
⚪ L1 label_replace over sum by rate, instant instant defaults 14 ms 12 ms ⚪ -14% 123 ms 0.10× ✅ same
⚪ L2 label_replace over sum by rate 1h defaults 17 ms 18 ms ⚪ +6% 83 ms 0.22× ✅ same
⚪ L3 topk over label_replace over sum by count 1h defaults 11 ms 7 ms ⚪ -36% 78 ms 0.09× ✅ sum ±0.03%
⚪ L4 label_replace over bare count_over_time, instant instant defaults 7 ms 13 ms ⚪ +86% 22 ms 0.59× ✅ same
⚪ L5 label_replace over json sum by, instant instant defaults 25 ms 46 ms ⚪ +84% 23 ms 2.00× ✅ same
⚪ L6 label_replace over json sum by 1h defaults 8 ms 21 ms ⚪ +162% 30 ms 0.70× ✅ same
⚪ L7 label_replace around a binary of two sums by level 1h defaults 7 ms 20 ms ⚪ +186% 14 ms 1.43× ✅ same
⚪ L8 label_replace over sum by level count, instant instant defaults 7 ms 7 ms ⚪ +0% 8 ms 0.88× ✅ same
⚪ LF13 explore volume, level filter before json 1h defaults 32 ms 34 ms ⚪ +6% 38 ms 0.89× ✅ same
⚪ LF15 drilldown field breakdown, level filter 1h defaults 11 ms 9 ms ⚪ -18% 11 ms 0.82× ✅ same
⚪ LF16 drilldown label breakdown, level filter 1h defaults 36 ms 36 ms ⚪ +0% 8 ms 4.50× ✅ same
⚪ LF20 logfmt count by detected_level 1h defaults 15 ms 14 ms ⚪ -7% 8 ms 1.75× ✅ same
⚪ LF21 logfmt count by detected_level, level filter 1h defaults 8 ms 8 ms ⚪ +0% 7 ms 1.14× ✅ same
⚪ LF23 json count by level, level filter 1h defaults 32 ms 32 ms ⚪ +0% 13 ms 2.46× ✅ same
⚪ LF24 rate of errors by service 1h defaults 12 ms 12 ms ⚪ +0% 69 ms 0.17× ➖ differs (as base)
⚪ LF25 count of warnings, error label dropped 1h defaults 13 ms 13 ms ⚪ +0% 43 ms 0.30× ➖ differs (as base)
⚪ LF28 logs, keep detected_level 1h defaults 51 ms 49 ms ⚪ -4% 15 ms 3.27× ✅ same
⚪ LF29 logs, drop detected_level 1h defaults 77 ms 72 ms ⚪ -6% 15 ms 4.80× ✅ same
⚪ LF30 logs, drop level 1h defaults 73 ms 69 ms ⚪ -5% 16 ms 4.31× ✅ same
⚪ LF31 count by detected_level, level dropped 1h defaults 15 ms 14 ms ⚪ -7% 13 ms 1.08× ✅ same
⚪ LF32 count by detected_level, detected_level kept 1h defaults 9 ms 8 ms ⚪ -11% 13 ms 0.62× ✅ same
⚪ LF7 logs, json then level equals 1h defaults 112 ms 119 ms ⚪ +6% 152 ms 0.78× ✅ same
⚪ LF8 logs, logfmt then level regexp 1h defaults 53 ms 54 ms ⚪ +2% 56 ms 0.96× ✅ same
⚪ LS1 logs, level stream label 1h defaults 39 ms 40 ms ⚪ +3% 58 ms 0.69× ✅ same
⚪ LS2 logs, level stream label regexp, all services 1h defaults 74 ms 66 ms ⚪ -11% 58 ms 1.14× ✅ same
⚪ LS3 logs, level in the selector 1h defaults 33 ms 32 ms ⚪ -3% 22 ms 1.45× ✅ same
⚪ LS4 logs, json then level_extracted 1h defaults 157 ms 131 ms ⚪ -17% 141 ms 0.93× ✅ same
⚪ LS5 logs, logfmt then level_extracted regexp 1h defaults 69 ms 71 ms ⚪ +3% 49 ms 1.45× ✅ same
⚪ LS6 level stream label count 1h defaults 7 ms 7 ms ⚪ +0% 42 ms 0.17× ✅ same
⚪ LS7 rate of level stream label by service 1h defaults 8 ms 8 ms ⚪ +0% 73 ms 0.11× ✅ same
⚪ LS8 json count by level_extracted, collision 1h defaults 61 ms 64 ms ⚪ +5% 14 ms 4.57× ✅ same
⚪ O2 explore log volume (Grafana form), underscore structured-metadata filter 1h defaults 31 ms 31 ms ⚪ +0% 50 ms 0.62× ✅ same
⚪ DL1 drilldown field breakdown json-expr, level filter 1h defaults 13 ms 16 ms ⚪ +23% 40 ms 0.40× ✅ same
⚪ DL2 drilldown field breakdown, level + stream-label filter before json 1h defaults 32 ms 31 ms ⚪ -3% 8 ms 3.88× ✅ same
⚪ DL3 drilldown logfmt field breakdown, level filter 1h defaults 111 ms 122 ms ⚪ +10% 102 ms 1.20× ➖ differs (as base)
⚪ DL4 instant count, level filter, json+logfmt, 1h instant defaults 382 ms 377 ms ⚪ -1% 988 ms 0.38× ✅ sum ±0.25%
⚪ DL6 explore volume, level filter before json 1h defaults 31 ms 32 ms ⚪ +3% 106 ms 0.30× ✅ same
⚪ N1 sum by app of max by app,pod, count 1h defaults 10 ms 9 ms ⚪ -10% 51 ms 0.18× ✅ same
⚪ N10 instant min of avg, rate instant defaults 8 ms 8 ms ⚪ +0% 71 ms 0.11× ✅ same
⚪ N11 instant three levels instant defaults 7 ms 7 ms ⚪ +0% 55 ms 0.13× ✅ same
⚪ N12 instant json inner, count of sum instant defaults 14 ms 13 ms ⚪ -7% 66 ms 0.20× ✅ same
⚪ N13 instant without under by instant defaults 16 ms 16 ms ⚪ +0% 73 ms 0.22× ✅ same
⚪ N14 sliding sum of max 1h defaults 380 ms 374 ms ⚪ -2% 108 ms 3.46× ✅ same
⚪ N15 sliding sums only 1h defaults 12 ms 11 ms ⚪ -8% 64 ms 0.17× ✅ same
⚪ N16 innermost without under by, tumbling 1h defaults 11 ms 10 ms ⚪ -9% 38 ms 0.26× ✅ same
⚪ N2 max by namespace of sum by namespace,app, rate 1h defaults 8 ms 8 ms ⚪ +0% 34 ms 0.24× ✅ same
⚪ N3 count by namespace of sum by namespace,app 1h defaults 8 ms 7 ms ⚪ -12% 33 ms 0.21× ✅ same
⚪ N4 avg by app of min by app,pod 1h defaults 9 ms 9 ms ⚪ +0% 36 ms 0.25× ✅ same
⚪ N5 three levels, max min sum 1h defaults 9 ms 8 ms ⚪ -11% 35 ms 0.23× ✅ same
⚪ N6 json inner, sum of max 1h defaults 10 ms 9 ms ⚪ -10% 37 ms 0.24× ✅ same
⚪ N7 logfmt inner, max of sum 1h defaults 7 ms 7 ms ⚪ +0% 9 ms 0.78× ✅ same
⚪ N8 without over a named by level, tumbling 1h defaults 16 ms 13 ms ⚪ -19% 63 ms 0.21× ✅ same
⚪ N9 instant sum of max instant defaults 9 ms 8 ms ⚪ -11% 66 ms 0.12× ✅ same
⚪ OR1 logs, or alternatives 1h defaults 62 ms 71 ms ⚪ +15% 64 ms 1.11× ✅ same
⚪ OR2 logs, negated or alternatives 1h defaults 101 ms 101 ms ⚪ +0% 53 ms 1.91× ✅ same
⚪ OR3 logs, regexp or alternatives 1h defaults 48 ms 55 ms ⚪ +15% 68 ms 0.81× ✅ same
⚪ OR4 count by level, or alternatives with parser 1h defaults 31 ms 30 ms ⚪ -3% 46 ms 0.65× ✅ same
⚪ OR5 rate, negated regexp or alternatives 1h defaults 5 ms 5 ms ⚪ +0% 90 ms 0.06× ✅ same
⚪ OR6 instant, or alternatives instant defaults 6 ms 6 ms ⚪ +0% 63 ms 0.10× ✅ same
⚪ P1 api-gateway empty pattern 1h defaults 19 ms 19 ms ⚪ +0% 14 ms 1.36× ✅ same
⚪ P2 api-gateway not empty pattern, line filter 1h defaults 68 ms 68 ms ⚪ +0% 21 ms 3.24× ✅ same
⚪ P3 nginx count_over_time, pattern with spaces 1h defaults 7 ms 6 ms ⚪ -14% 5 ms 1.20× ✅ same
⚪ P4 nginx count_over_time, GET request pattern 1h defaults 7 ms 7 ms ⚪ +0% 5 ms 1.40× ✅ same
⚪ A1 nginx include GET, log query 1h defaults 15 ms 16 ms ⚪ +7% 9 ms 1.78× ✅ same
⚪ A2 nginx exclude GET, log query 1h defaults 29 ms 29 ms ⚪ +0% 9 ms 3.22× ✅ same
⚪ A3 nginx include GET, count_over_time 1h defaults 7 ms 7 ms ⚪ +0% 4 ms 1.75× ✅ same
⚪ A4 nginx exclude GET, count_over_time 1h defaults 7 ms 7 ms ⚪ +0% 5 ms 1.40× ✅ same
⚪ A5 drilldown exclude, volume by detected_level 1h defaults 14 ms 14 ms ⚪ +0% 4 ms 3.50× ✅ same
⚪ A6 nginx include ends on a literal, log query 1h defaults 13 ms 12 ms ⚪ -8% 4 ms 3.00× ✅ same
⚪ A7 nginx exclude ends on a literal, log query 1h defaults 17 ms 16 ms ⚪ -6% 8 ms 2.00× ✅ same
⚪ A8 nginx include two literals, count_over_time 1h defaults 7 ms 7 ms ⚪ +0% 5 ms 1.40× ✅ same
⚪ A9 nginx include two literals, ends on a literal, count_over_time 1h defaults 7 ms 7 ms ⚪ +0% 4 ms 1.75× ✅ same
⚪ P1 logs, json, level equals 1h defaults 110 ms 109 ms ⚪ -1% 70 ms 1.56× ✅ same
⚪ P2 logs, logfmt, level not equal 1h defaults 25 ms 25 ms ⚪ +0% 6 ms 4.17× ✅ same
⚪ P3 logs, json, level existence 1h defaults 96 ms 94 ms ⚪ -2% 67 ms 1.40× ✅ same
⚪ P4 logfmt count by level 1h defaults 5 ms 5 ms ⚪ +0% 5 ms 1.00× ✅ same
⚪ P5 json count by level 1h defaults 31 ms 31 ms ⚪ +0% 16 ms 1.94× ✅ same
⚪ P6 json count, level filter 1h defaults 5 ms 4 ms ⚪ -20% 15 ms 0.27× ✅ same
⚪ P7 json rate by level, level regexp 1h defaults 5 ms 5 ms ⚪ +0% 14 ms 0.36× ✅ same
⚪ P8 json count by a name that is no stream label 1h defaults 31 ms 31 ms ⚪ +0% 16 ms 1.94× ✅ same
⚪ P9 instant logfmt by level instant defaults 5 ms 5 ms ⚪ +0% 8 ms 0.62× ✅ same
⚪ E1 json duration_ms sum by service 1h defaults 6 ms 6 ms ⚪ +0% 19 ms 0.32× ✅ same
⚪ E2 json duration_ms p95 by service 1h defaults 83 ms 90 ms ⚪ +8% 16 ms 5.62× ✅ same
⚪ E3 logfmt latency_ms rate, production 1h defaults 12 ms 12 ms ⚪ +0% 136 ms 0.09× ➖ differs (as base)
⚪ E4 logfmt size_bytes max by service 1h defaults 6 ms 6 ms ⚪ +0% 12 ms 0.50× ✅ same
⚪ E5 json duration_ms sum, instant instant defaults 26 ms 23 ms ⚪ -12% 25 ms 0.92× ✅ same
⚪ E6 logfmt ttl unit string (Loki 400) 1h defaults 400 400 ⚪ +0% 400 — ✅ same
⚪ E7 json hexadecimal trace_id (Loki 400) 1h defaults 400 400 ⚪ -33% 400 — ✅ same
⚪ E8 json duration_ms sum by service, namespace prod 1h defaults 9 ms 8 ms ⚪ -11% 92 ms 0.09× ✅ same
⚪ E9 logfmt latency_ms sum by service, namespace prod 1h defaults 9 ms 10 ms ⚪ +11% 85 ms 0.12× ✅ same
⚪ G1 logfmt amount sum_over_time, no filter 1h defaults 10 ms 8 ms ⚪ -20% 15 ms 0.53× ✅ same
⚪ G10 logfmt latency_ms sum_over_time, production 1h defaults 12 ms 11 ms ⚪ -8% 111 ms 0.10× ➖ differs (as base)
⚪ G11 logfmt amount sum_over_time, instant instant defaults 6 ms 6 ms ⚪ +0% 10 ms 0.60× ✅ same
⚪ G12 json duration_ms sum_over_time, instant instant defaults 26 ms 26 ms ⚪ +0% 24 ms 1.08× ✅ same
⚪ G2 json duration_ms sum_over_time, no filter 1h defaults 44 ms 41 ms ⚪ -7% 33 ms 1.24× ✅ same
⚪ G3 logfmt amount sum_over_time, error filter 1h defaults 10 ms 10 ms ⚪ +0% 12 ms 0.83× ✅ same
⚪ G4 json duration_ms sum_over_time, error filter 1h defaults 9 ms 10 ms ⚪ +11% 29 ms 0.34× ✅ same
⚪ G5 logfmt amount sum by rate 1h defaults 8 ms 8 ms ⚪ +0% 12 ms 0.67× ✅ same
⚪ G6 json duration_ms sum by rate 1h defaults 8 ms 8 ms ⚪ +0% 26 ms 0.31× ✅ same
⚪ G7 duration() of a unit string 1h defaults 17 ms 16 ms ⚪ -6% 10 ms 1.60× ✅ same
⚪ G8 bytes() of a number 1h defaults 13 ms 13 ms ⚪ +0% 7 ms 1.86× ✅ same
⚪ G9 json duration_ms max_over_time 1h defaults 9 ms 8 ms ⚪ -11% 131 ms 0.06× ✅ same
⚪ W1 sum without pod, max_over_time 1h defaults 33 ms 34 ms ⚪ +3% 18 ms 1.89× ✅ same
⚪ W2 sum without pod, sum_over_time 1h defaults 36 ms 36 ms ⚪ +0% 14 ms 2.57× ✅ same
⚪ W3 sum without pod, rate 1h defaults 45 ms 42 ms ⚪ -7% 15 ms 2.80× ✅ same
⚪ W4 max without pod, max_over_time 1h defaults 27 ms 27 ms ⚪ +0% 15 ms 1.80× ✅ same
⚪ W5 sum without pod, max_over_time, instant instant defaults 6 ms 6 ms ⚪ +0% 13 ms 0.46× ✅ same
⚪ W6 sum without pod, rate, instant instant defaults 6 ms 6 ms ⚪ +0% 12 ms 0.50× ✅ same
⚪ W7 topk 5 of sum without pod 1h defaults 28 ms 29 ms ⚪ +4% 16 ms 1.81× ✅ same
⚪ W8 sum without pod, count_over_time (production) 1h defaults 238 ms 227 ms ⚪ -5% 58 ms 3.91× ✅ same
⚪ W9 label_replace control, instant instant defaults 5 ms 4 ms ⚪ -20% 71 ms 0.06× ✅ same
⚪ U1 logfmt amount rate 1h defaults 6 ms 6 ms ⚪ +0% 12 ms 0.50× ✅ same
⚪ U2 json duration_ms rate 1h defaults 6 ms 6 ms ⚪ +0% 26 ms 0.23× ✅ same
⚪ U3 json duration_ms rate, error filter 1h defaults 6 ms 6 ms ⚪ +0% 25 ms 0.24× ✅ same
⚪ U4 logfmt latency_ms rate, production 1h defaults 10 ms 9 ms ⚪ -10% 133 ms 0.07× ➖ differs (as base)
⚪ U5 logfmt amount rate, instant instant defaults 6 ms 6 ms ⚪ +0% 12 ms 0.50× ✅ same
⚪ U6 json duration_ms rate, instant instant defaults 9 ms 9 ms ⚪ +0% 21 ms 0.43× ✅ same
⚪ U7 json duration_ms sum_over_time, instant instant defaults 10 ms 15 ms ⚪ +50% 21 ms 0.71× ✅ same
⚪ U8 logfmt amount rate by service (grouped) 1h defaults 6 ms 5 ms ⚪ -17% 11 ms 0.45× ✅ same
⚪ U9 logfmt amount rate_counter (folded per series) 1h defaults 181 ms 180 ms ⚪ -1% 35 ms 5.14× ➖ differs (as base)
⚪ U1 json duration_ms sum by service 1h defaults 7 ms 6 ms ⚪ -14% 25 ms 0.24× ✅ same
⚪ U10 instant sum_over_time unwrap instant defaults 23 ms 24 ms ⚪ +4% 25 ms 0.96× ✅ same
⚪ U11 instant max_over_time missing label instant defaults 5 ms 5 ms ⚪ +0% 13 ms 0.38× ✅ same
⚪ U12 json duration_ms over every service 1h defaults 9 ms 8 ms ⚪ -11% 55 ms 0.15× ✅ same
⚪ U13 logfmt latency_ms over every service 1h defaults 11 ms 11 ms ⚪ +0% 64 ms 0.17× ➖ differs (as base)
⚪ U14 duration() of a unit string 1h defaults 14 ms 14 ms ⚪ +0% 6 ms 2.33× ✅ same
⚪ U2 json duration_ms max by service 1h defaults 8 ms 6 ms ⚪ -25% 36 ms 0.17× ✅ same
⚪ U3 json duration_ms sliding sum 1h defaults 6 ms 6 ms ⚪ +0% 32 ms 0.19× ✅ same
⚪ U4 logfmt size_bytes sum by service 1h defaults 5 ms 5 ms ⚪ +0% 6 ms 0.83× ✅ same
⚪ U5 logfmt ttl unit string (no sample) 1h defaults 5 ms 4 ms ⚪ -20% 5 ms 0.80× ✅ same
⚪ U6 missing label, no parser 1h defaults 5 ms 4 ms ⚪ -20% 9 ms 0.44× ✅ same
⚪ U7 missing label, ungrouped sum 1h defaults 5 ms 4 ms ⚪ -20% 9 ms 0.44× ✅ same
⚪ U8 rate of an unwrapped label 1h defaults 6 ms 6 ms ⚪ +0% 15 ms 0.40× ✅ same
⚪ U9 bytes() conversion 1h defaults 11 ms 11 ms ⚪ +0% 10 ms 1.10× ✅ same
Cold (first-run) timings — what a user opening a panel sees
shape range Loki profile base cold PR cold Loki cold PR ÷ Loki cold
LF26 topk services by errors 1h defaults 648 ms 5 ms 372 ms ¹ 0.01×
bytes_over_time by app 1h defaults 48 ms 49 ms 716 ms 0.07×
drilldown service landing 1h defaults 29 ms 26 ms 424 ms 0.06×
instant json filtered instant defaults 8 ms 8 ms 73 ms 0.11×
instant json volume instant defaults 24 ms 24 ms 94 ms 0.26×
instant ungrouped rate instant defaults 7 ms 6 ms 77 ms 0.08×
json rate by level (no filter, no drop) 1h defaults 400 400 400 —
json volume no filters 1h defaults 43 ms 44 ms 634 ms ↺ 0.07×
logfmt count by level, filter, drop 1h defaults 448 ms 340 ms 734 ms 0.46×
logfmt volume 1h defaults 167 ms 167 ms 1.01 s 0.16×
logs plain 1h defaults 89 ms 77 ms 69 ms 1.12×
logs | json 1h defaults 115 ms 113 ms 64 ms 1.77×
logs | logfmt 1h defaults 164 ms 160 ms 58 ms 2.76×
stream-label count by app 1h defaults 16 ms 18 ms 433 ms 0.04×
topk rate by app 1h defaults 49 ms 11 ms 466 ms 0.02×
ungrouped rate 1h defaults 9 ms 20 ms 457 ms 0.04×
DB1 label breakdown pod 1h defaults 101 ms 59 ms 1.09 s 0.05×
DB2 label breakdown namespace 1h defaults 13 ms 12 ms 627 ms 0.02×
DB3 json field breakdown user_id 1h defaults 40 ms 36 ms 594 ms 0.06×
DB4 json field breakdown batch_id 1h defaults 86 ms 72 ms 730 ms 0.10×
X1 json filter level_extracted, by level 1h defaults 2.60 s 2.95 s 677 ms 4.36×
X2 json group by level_extracted 1h defaults 225 ms 222 ms 686 ms 0.32×
X3 json rate, level_extracted equality 1h defaults 208 ms 204 ms 492 ms 0.41×
X4 json bytes, level_extracted regexp 1h defaults 326 ms 298 ms 581 ms 0.51×
X5 logfmt filter level_extracted, by level 1h defaults 515 ms 508 ms 973 ms 0.52×
E1 logs, json parser over stream labels 1h defaults 97 ms 119 ms 58 ms † —
E2 logs, logfmt parser over stream labels 1h defaults 48 ms 69 ms 46 ms † —
E3 logs, json, existence filter on level_extracted 1h defaults 109 ms 109 ms 61 ms † —
E4 logs, logfmt, existence filter on level_extracted 1h defaults 65 ms 80 ms 49 ms † —
E5 logs, json, drop level_extracted 1h defaults 87 ms 90 ms 53 ms † —
E6 logfmt count by level_extracted 1h defaults 33 ms 32 ms 51 ms 0.63×
E7 json count by level_extracted 1h defaults 1.15 s 983 ms 130 ms 7.56×
E8 instant logfmt by level_extracted instant defaults 6 ms 6 ms 8 ms 0.75×
J1 json no-error count by service_name, one service 1h defaults 147 ms 133 ms 156 ms 0.85×
J2 json no-error count by app, namespace 1h defaults 320 ms 325 ms 440 ms 0.74×
J3 json no-error count by parsed method 1h defaults 124 ms 134 ms 137 ms 0.98×
J4 json no-error count by service_name, all services 1h defaults 379 ms 380 ms 601 ms 0.63×
J5 json no-error bytes_rate by app 1h defaults 203 ms 209 ms 644 ms 0.32×
A explore volume, 2 filters, drop 1h defaults 1.53 s 1.67 s 1.21 s 1.38×
B grouped sum, filter, no drop 1h defaults 146 ms 185 ms 924 ms 0.20×
C ungrouped sum, filter, no drop 1h defaults 93 ms 91 ms 442 ms 0.21×
D volume, no filters 1h defaults 152 ms 151 ms 634 ms 0.24×
E field breakdown pipeline (explore) 1h defaults 159 ms 172 ms 479 ms 0.36×
F field breakdown service_version (explore) 1h defaults 1.05 s 1.02 s 525 ms 1.94×
G field breakdown pipeline (drilldown) 1h defaults 36 ms 36 ms 432 ms 0.08×
H field breakdown service_version (drilldown) 1h defaults 67 ms 66 ms 711 ms 0.09×
I grafana volume of B 1h defaults 88 ms 83 ms 436 ms 0.19×
L1 label_replace over sum by rate, instant instant defaults 18 ms 12 ms 75 ms 0.16×
L2 label_replace over sum by rate 1h defaults 120 ms 99 ms 533 ms 0.19×
L3 topk over label_replace over sum by count 1h defaults 94 ms 7 ms 510 ms 0.01×
L4 label_replace over bare count_over_time, instant instant defaults 8 ms 13 ms 21 ms 0.62×
L5 label_replace over json sum by, instant instant defaults 28 ms 49 ms 24 ms 2.04×
L6 label_replace over json sum by 1h defaults 70 ms 104 ms 165 ms 0.63×
L7 label_replace around a binary of two sums by level 1h defaults 7 ms 27 ms 53 ms † —
L8 label_replace over sum by level count, instant instant defaults 8 ms 7 ms 8 ms 0.88×
LF13 explore volume, level filter before json 1h defaults 79 ms 88 ms 363 ms 0.24×
LF15 drilldown field breakdown, level filter 1h defaults 123 ms 137 ms 116 ms 1.18×
LF16 drilldown label breakdown, level filter 1h defaults 41 ms 42 ms 37 ms 1.14×
LF20 logfmt count by detected_level 1h defaults 42 ms 42 ms 41 ms 1.02×
LF21 logfmt count by detected_level, level filter 1h defaults 27 ms 27 ms 32 ms 0.84×
LF23 json count by level, level filter 1h defaults 42 ms 41 ms 114 ms 0.36×
LF24 rate of errors by service 1h defaults 332 ms 324 ms 397 ms 0.82×
LF25 count of warnings, error label dropped 1h defaults 328 ms 316 ms 359 ms 0.88×
LF28 logs, keep detected_level 1h defaults 48 ms 49 ms 15 ms † —
LF29 logs, drop detected_level 1h defaults 73 ms 84 ms 15 ms 5.60×
LF30 logs, drop level 1h defaults 74 ms 73 ms 15 ms 4.87×
LF31 count by detected_level, level dropped 1h defaults 35 ms 35 ms 90 ms 0.39×
LF32 count by detected_level, detected_level kept 1h defaults 19 ms 19 ms 98 ms 0.19×
LF7 logs, json then level equals 1h defaults 123 ms 133 ms 155 ms † —
LF8 logs, logfmt then level regexp 1h defaults 55 ms 53 ms 59 ms † —
LS1 logs, level stream label 1h defaults 54 ms 43 ms 60 ms 0.72×
LS2 logs, level stream label regexp, all services 1h defaults 74 ms 65 ms 55 ms 1.18×
LS3 logs, level in the selector 1h defaults 43 ms 32 ms 26 ms 1.23×
LS4 logs, json then level_extracted 1h defaults 137 ms 138 ms 140 ms † —
LS5 logs, logfmt then level_extracted regexp 1h defaults 75 ms 74 ms 49 ms † —
LS6 level stream label count 1h defaults 9 ms 9 ms 405 ms 0.02×
LS7 rate of level stream label by service 1h defaults 14 ms 14 ms 456 ms 0.03×
LS8 json count by level_extracted, collision 1h defaults 1.14 s 1.01 s 139 ms 7.27×
O2 explore log volume (Grafana form), underscore structured-metadata filter 1h defaults 116 ms 110 ms 505 ms 0.22×
DL1 drilldown field breakdown json-expr, level filter 1h defaults 375 ms 381 ms 433 ms 0.88×
DL2 drilldown field breakdown, level + stream-label filter before json 1h defaults 41 ms 51 ms 25 ms 2.04×
DL3 drilldown logfmt field breakdown, level filter 1h defaults 507 ms 504 ms 903 ms 0.56×
DL4 instant count, level filter, json+logfmt, 1h instant defaults 371 ms 384 ms 953 ms 0.40×
DL6 explore volume, level filter before json 1h defaults 146 ms 161 ms 609 ms 0.26×
N1 sum by app of max by app,pod, count 1h defaults 60 ms 44 ms 635 ms 0.07×
N10 instant min of avg, rate instant defaults 8 ms 14 ms 70 ms 0.20×
N11 instant three levels instant defaults 8 ms 7 ms 55 ms 0.13×
N12 instant json inner, count of sum instant defaults 14 ms 27 ms 65 ms † —
N13 instant without under by instant defaults 16 ms 16 ms 67 ms 0.24×
N14 sliding sum of max 1h defaults 433 ms 443 ms 1.20 s 0.37×
N15 sliding sums only 1h defaults 19 ms 20 ms 310 ms 0.06×
N16 innermost without under by, tumbling 1h defaults 48 ms 40 ms 736 ms 0.05×
N2 max by namespace of sum by namespace,app, rate 1h defaults 16 ms 23 ms 302 ms 0.08×
N3 count by namespace of sum by namespace,app 1h defaults 30 ms 15 ms 277 ms † —
N4 avg by app of min by app,pod 1h defaults 43 ms 37 ms 390 ms 0.09×
N5 three levels, max min sum 1h defaults 15 ms 30 ms 459 ms 0.07×
N6 json inner, sum of max 1h defaults 135 ms 100 ms 279 ms 0.36×
N7 logfmt inner, max of sum 1h defaults 19 ms 18 ms 46 ms 0.39×
N8 without over a named by level, tumbling 1h defaults 49 ms 40 ms 562 ms 0.07×
N9 instant sum of max instant defaults 12 ms 8 ms 71 ms 0.11×
OR1 logs, or alternatives 1h defaults 77 ms 62 ms 64 ms 0.97×
OR2 logs, negated or alternatives 1h defaults 84 ms 80 ms 58 ms 1.38×
OR3 logs, regexp or alternatives 1h defaults 48 ms 47 ms 65 ms 0.72×
OR4 count by level, or alternatives with parser 1h defaults 95 ms 94 ms 332 ms 0.28×
OR5 rate, negated regexp or alternatives 1h defaults 43 ms 45 ms 646 ms 0.07×
OR6 instant, or alternatives instant defaults 11 ms 6 ms 64 ms 0.09×
P1 api-gateway empty pattern 1h defaults 27 ms 19 ms 70 ms † —
P2 api-gateway not empty pattern, line filter 1h defaults 68 ms 59 ms 21 ms 2.81×
P3 nginx count_over_time, pattern with spaces 1h defaults 12 ms 11 ms 20 ms † —
P4 nginx count_over_time, GET request pattern 1h defaults 12 ms 12 ms 23 ms 0.52×
A1 nginx include GET, log query 1h defaults 22 ms 15 ms 6 ms 2.50×
A2 nginx exclude GET, log query 1h defaults 36 ms 36 ms 6 ms 6.00×
A3 nginx include GET, count_over_time 1h defaults 8 ms 8 ms 23 ms ↺ 0.35×
A4 nginx exclude GET, count_over_time 1h defaults 14 ms 15 ms 28 ms 0.54×
A5 drilldown exclude, volume by detected_level 1h defaults 31 ms 31 ms 27 ms 1.15×
A6 nginx include ends on a literal, log query 1h defaults 12 ms 12 ms 20 ms † —
A7 nginx exclude ends on a literal, log query 1h defaults 19 ms 17 ms 7 ms 2.43×
A8 nginx include two literals, count_over_time 1h defaults 18 ms 17 ms 24 ms 0.71×
A9 nginx include two literals, ends on a literal, count_over_time 1h defaults 17 ms 17 ms 21 ms † —
P1 logs, json, level equals 1h defaults 112 ms 111 ms 63 ms † —
P2 logs, logfmt, level not equal 1h defaults 25 ms 25 ms 28 ms † —
P3 logs, json, level existence 1h defaults 115 ms 112 ms 54 ms † —
P4 logfmt count by level 1h defaults 24 ms 23 ms 34 ms 0.68×
P5 json count by level 1h defaults 68 ms 69 ms 145 ms 0.48×
P6 json count, level filter 1h defaults 12 ms 12 ms 140 ms 0.09×
P7 json rate by level, level regexp 1h defaults 13 ms 12 ms 146 ms 0.08×
P8 json count by a name that is no stream label 1h defaults 71 ms 73 ms 128 ms 0.57×
P9 instant logfmt by level instant defaults 5 ms 5 ms 9 ms 0.56×
E1 json duration_ms sum by service 1h defaults 80 ms 79 ms 145 ms 0.54×
E2 json duration_ms p95 by service 1h defaults 196 ms 193 ms 139 ms 1.39×
E3 logfmt latency_ms rate, production 1h defaults 420 ms 385 ms 641 ms 0.60×
E4 logfmt size_bytes max by service 1h defaults 37 ms 35 ms 121 ms 0.29×
E5 json duration_ms sum, instant instant defaults 24 ms 26 ms 26 ms 1.00×
E6 logfmt ttl unit string (Loki 400) 1h defaults 400 400 400 —
E7 json hexadecimal trace_id (Loki 400) 1h defaults 400 400 400 —
E8 json duration_ms sum by service, namespace prod 1h defaults 160 ms 148 ms 410 ms 0.36×
E9 logfmt latency_ms sum by service, namespace prod 1h defaults 260 ms 260 ms 457 ms 0.57×
G1 logfmt amount sum_over_time, no filter 1h defaults 50 ms 42 ms 57 ms 0.74×
G10 logfmt latency_ms sum_over_time, production 1h defaults 387 ms 377 ms 650 ms 0.58×
G11 logfmt amount sum_over_time, instant instant defaults 6 ms 6 ms 10 ms 0.60×
G12 json duration_ms sum_over_time, instant instant defaults 24 ms 32 ms 24 ms 1.33×
G2 json duration_ms sum_over_time, no filter 1h defaults 98 ms 96 ms 179 ms 0.54×
G3 logfmt amount sum_over_time, error filter 1h defaults 42 ms 43 ms 48 ms 0.90×
G4 json duration_ms sum_over_time, error filter 1h defaults 89 ms 90 ms 211 ms ↺ 0.43×
G5 logfmt amount sum by rate 1h defaults 41 ms 42 ms 45 ms ↺ 0.93×
G6 json duration_ms sum by rate 1h defaults 88 ms 87 ms 133 ms 0.65×
G7 duration() of a unit string 1h defaults 49 ms 49 ms 31 ms 1.58×
G8 bytes() of a number 1h defaults 48 ms 51 ms 33 ms 1.55×
G9 json duration_ms max_over_time 1h defaults 69 ms 71 ms 1.10 s 0.06×
W1 sum without pod, max_over_time 1h defaults 78 ms 55 ms 90 ms 0.61×
W2 sum without pod, sum_over_time 1h defaults 55 ms 55 ms 73 ms 0.75×
W3 sum without pod, rate 1h defaults 65 ms 69 ms 72 ms 0.96×
W4 max without pod, max_over_time 1h defaults 27 ms 27 ms 92 ms 0.29×
W5 sum without pod, max_over_time, instant instant defaults 6 ms 6 ms 13 ms 0.46×
W6 sum without pod, rate, instant instant defaults 6 ms 6 ms 12 ms 0.50×
W7 topk 5 of sum without pod 1h defaults 27 ms 28 ms 85 ms 0.33×
W8 sum without pod, count_over_time (production) 1h defaults 285 ms 279 ms 650 ms 0.43×
W9 label_replace control, instant instant defaults 8 ms 5 ms 70 ms 0.07×
U1 logfmt amount rate 1h defaults 34 ms 27 ms 41 ms 0.66×
U2 json duration_ms rate 1h defaults 61 ms 61 ms 105 ms 0.58×
U3 json duration_ms rate, error filter 1h defaults 62 ms 61 ms 121 ms 0.50×
U4 logfmt latency_ms rate, production 1h defaults 270 ms 264 ms 881 ms 0.30×
U5 logfmt amount rate, instant instant defaults 12 ms 6 ms 14 ms 0.43×
U6 json duration_ms rate, instant instant defaults 21 ms 8 ms 21 ms 0.38×
U7 json duration_ms sum_over_time, instant instant defaults 10 ms 10 ms 21 ms 0.48×
U8 logfmt amount rate by service (grouped) 1h defaults 31 ms 31 ms 45 ms 0.69×
U9 logfmt amount rate_counter (folded per series) 1h defaults 180 ms 174 ms 263 ms † —
U1 json duration_ms sum by service 1h defaults 76 ms 80 ms 378 ms 0.21×
U10 instant sum_over_time unwrap instant defaults 22 ms 21 ms 25 ms 0.84×
U11 instant max_over_time missing label instant defaults 5 ms 5 ms 14 ms † —
U12 json duration_ms over every service 1h defaults 213 ms 188 ms 859 ms 0.22×
U13 logfmt latency_ms over every service 1h defaults 374 ms 370 ms 969 ms 0.38×
U14 duration() of a unit string 1h defaults 45 ms 44 ms 46 ms 0.96×
U2 json duration_ms max by service 1h defaults 74 ms 66 ms 584 ms 0.11×
U3 json duration_ms sliding sum 1h defaults 6 ms 6 ms 211 ms 0.03×
U4 logfmt size_bytes sum by service 1h defaults 35 ms 48 ms 51 ms 0.94×
U5 logfmt ttl unit string (no sample) 1h defaults 32 ms 31 ms 39 ms † —
U6 missing label, no parser 1h defaults 13 ms 13 ms 78 ms † —
U7 missing label, ungrouped sum 1h defaults 12 ms 12 ms 76 ms † —
U8 rate of an unwrapped label 1h defaults 65 ms 63 ms 194 ms 0.32×
U9 bytes() conversion 1h defaults 47 ms 46 ms 47 ms 0.98×

† Loki's results cache answered this cold run (its own counters, or identical answers for windows a minute apart): not a computed time, so it is in no ratio.

¹ (Loki cold) the confirm pass's cold run was cache-served, so every cold column of the row is the first pass's.

↺ Loki cold taken from the first set that ran the same query; Loki's cache answered this one.

13 shape×range above 2× Loki already (listed, or as slow on the base build)
profile mode shape range PR Loki PR ÷ Loki base ÷ Loki listed ÷ Loki why
known defaults cold logs | logfmt 1h 160 ms 58 ms 2.76× 2.83× — as slow on base
known defaults warm logs | logfmt 1h 166 ms 56 ms 2.96× 2.59× — as slow on base
known defaults cold E7 json count by level_extracted 1h 983 ms 130 ms 7.56× 8.86× — as slow on base
known defaults cold X1 json filter level_extracted, by level 1h 2.95 s 677 ms 4.36× 3.84× — as slow on base
known defaults warm E2 json duration_ms p95 by service 1h 90 ms 16 ms 5.62× 5.19× — as slow on base
known defaults warm U9 logfmt amount rate_counter (folded per series) 1h 180 ms 35 ms 5.14× 5.17× — as slow on base
known defaults warm W8 sum without pod, count_over_time (production) 1h 227 ms 58 ms 3.91× 4.10× — as slow on base
known defaults cold LF29 logs, drop detected_level 1h 84 ms 15 ms 5.60× 4.87× — as slow on base
known defaults warm LF29 logs, drop detected_level 1h 72 ms 15 ms 4.80× 5.13× — as slow on base
known defaults cold LF30 logs, drop level 1h 73 ms 15 ms 4.87× 4.93× — as slow on base
known defaults warm LF30 logs, drop level 1h 69 ms 16 ms 4.31× 4.56× — as slow on base
known defaults cold LS8 json count by level_extracted, collision 1h 1.01 s 139 ms 7.27× 8.19× — as slow on base
known defaults warm N14 sliding sum of max 1h 374 ms 108 ms 3.46× 3.52× — as slow on base
Why these shapes
  • internal/translator/translator.go → 172: control/bytes_over_time by app, control/instant json filtered, control/instant json volume, control/instant ungrouped rate, control/json rate by level (no filter, no drop), control/json volume no filters, control/logfmt count by level, filter, drop, control/logfmt volume …
  • internal/proxy/ordered_json_metric.go → 104: control/bytes_over_time by app, control/instant json filtered, control/instant json volume, control/json rate by level (no filter, no drop), control/json volume no filters, control/logfmt count by level, filter, drop, control/logfmt volume, control/logs | json …
  • internal/proxy/vector_matching.go → 49: control/stream-label count by app, label-replace/L7 label_replace around a binary of two sums by level, label-replace/L8 label_replace over sum by level count, instant, nested-aggregation/N1 sum by app of max by app,pod, count, nested-aggregation/N10 instant min of avg, rate, nested-aggregation/N11 instant three levels, nested-aggregation/N12 instant json inner, count of sum, nested-aggregation/N13 instant without under by …
  • internal/proxy/metric_agg.go → 27: control/bytes_over_time by app, control/instant json volume, control/instant ungrouped rate, control/json volume no filters, control/stream-label count by app, control/topk rate by app, control/ungrouped rate, drilldown-breakdown-exact/DB1 label breakdown pod …
  • internal/proxy/range_metric_compat.go → 25: control/bytes_over_time by app, control/instant json volume, control/instant ungrouped rate, control/json volume no filters, control/stream-label count by app, control/topk rate by app, control/ungrouped rate, json-filter-pushdown/A explore volume, 2 filters, drop …
  • control smoke → 8: control/drilldown service landing, control/instant ungrouped rate, control/json rate by level (no filter, no drop), control/json volume no filters, control/logs plain, control/logs | json, control/stream-label count by app, control/topk rate by app
  • internal/proxy/proxy.go (endpoint handler) → 8: control/drilldown service landing, control/instant ungrouped rate, control/json rate by level (no filter, no drop), control/json volume no filters, control/logs plain, control/logs | json, control/stream-label count by app, control/topk rate by app
  • shape added or edited → 8: label-replace/L1 label_replace over sum by rate, instant, label-replace/L2 label_replace over sum by rate, label-replace/L3 topk over label_replace over sum by count, label-replace/L4 label_replace over bare count_over_time, instant, label-replace/L5 label_replace over json sum by, instant, label-replace/L6 label_replace over json sum by, label-replace/L7 label_replace around a binary of two sums by level, label-replace/L8 label_replace over sum by level count, instant
  • internal/proxy/sliding_window_stats.go → 4: drilldown-breakdown-exact/DB1 label breakdown pod, drilldown-breakdown-exact/DB2 label breakdown namespace, drilldown-breakdown-exact/DB3 json field breakdown user_id, drilldown-breakdown-exact/DB4 json field breakdown batch_id

base 47c5727529 → PR 086d369af9 · 4 runs (first is cold), warm p50 · ranges 1h · noise ±30% and 100 ms · VictoriaLogs restarts 0→0 · 6.3 min · phases stack_up 39s, seed_and_build 85s, loki-defaults:loki_profile 0s, loki-defaults:proxies 2s, loki-defaults:matrix 232s. Same runner, one fresh stack, targets interleaved per request. How to read and reproduce: bench/ab/README.md.

@github-actions

github-actions Bot commented Oct 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

PR Quality Report

Compared against base branch main.

Coverage and tests

Signal Base PR Delta
Test count 18090 18213 123
Coverage 88.7% 88.7% 0.0% (stable)

Compatibility

Track Base PR Delta
Loki API 100.0% 11/11 (100.0%) 0.0% (stable)
Logs Drilldown 100.0% 17/17 (100.0%) 0.0% (stable)
VictoriaLogs 100.0% 11/11 (100.0%) 0.0% (stable)

Performance smoke

Lower CPU cost (ns/op) is better. Lower benchmark memory cost (B/op, allocs/op) is better. Higher throughput is better. Lower load-test memory growth is better. Benchmark rows are medians from repeated samples.

Signal Base PR Delta
QueryRange cache-hit CPU cost 3370.0 ns/op 3423.0 ns/op +1.6% (stable)
QueryRange cache-hit memory 1240.0 B/op 1240.0 B/op 0.0% (stable)
QueryRange cache-hit allocations 27.0 allocs/op 27.0 allocs/op 0.0% (stable)
QueryRange cache-bypass CPU cost 3980.0 ns/op 3971.0 ns/op -0.2% (stable)
QueryRange cache-bypass memory 1303.0 B/op 1301.0 B/op -0.2% (stable)
QueryRange cache-bypass allocations 27.0 allocs/op 27.0 allocs/op 0.0% (stable)
Labels cache-hit CPU cost 839.4 ns/op 918.4 ns/op +9.4% (stable)
Labels cache-hit memory 48.0 B/op 48.0 B/op 0.0% (stable)
Labels cache-hit allocations 3.0 allocs/op 3.0 allocs/op 0.0% (stable)
Labels cache-bypass CPU cost 1463.0 ns/op 1492.0 ns/op +2.0% (stable)
Labels cache-bypass memory 400.0 B/op 400.0 B/op 0.0% (stable)
Labels cache-bypass allocations 14.0 allocs/op 14.0 allocs/op 0.0% (stable)

State

  • Coverage, compatibility, and sampled performance are reported here from the same PR workflow.
  • This is a delta report, not a release gate by itself. Required checks still decide merge safety.
  • Performance is a smoke comparison, not a full benchmark lab run.
  • Delta states use the same noise guards as the quality gate (percent + absolute + low-baseline checks), so report labels match merge-gate behavior.

@github-actions

github-actions Bot commented Oct 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

✅ Visual smoke: passed with warnings

Base 47c5727529 vs PR 1e313f3ca9, Grafana through the proxy builds and Loki on one fresh stack; run.
Loaded again, once and more patiently, after a first-pass difference or an unsettled page: explore-label-replace-over-binary 15m, explore-label-replace-over-binary 1h, explore-label-replace-over-binary 6h.
Core set: 5 capture(s) at 1h. Detailed set: 29 entries (explore-label-replace-over-binary, explore-metric-plain, lvl-dd-fields-pill, lvl-dd-field-method-pill, lvl-dd-field-pipeline-pill, lvl-dd-label-pod-pill, lvl-explore-volume-dl-prefilter-json, lvl-explore-logs-json-level, ...). 37 capture(s), 15m/1h/6h, 24h and 7d stay out of PR runs.
Trimmed to the 1h range for the capture budget: explore-A, explore-B, explore-C, explore-E, explore-F, explore-logs-json, explore-logfmt-metric, explore-logs-extracted, explore-extracted-metric, explore-extracted-filter-metric, explore-extracted-group-metric, explore-logs-plain-level, explore-plain-level-metric, explore-logs-line-filter-or, explore-metric-line-filter-or, explore-unwrap-sum, explore-unwrap-unit-string, explore-unwrap-missing-label, explore-unwrap-conversion-error, explore-unwrap-rate-sum, explore-empty-pattern-filter, explore-pattern-filter-anchored, explore-unwrap-quantile-over-time, explore-json-error-filter-metric, explore-unwrap-max-over-time-end-sample, explore-unwrap-sum-over-time-sliding-window, explore-unwrap-first-last-under-sum, explore-max-by-count-over-time-sliding, explore-nested-aggregation-outer-by, explore-without-over-inner-aggregation, explore-without-nested-under-aggregation, dd-labels-namespace, dd-field-service_version, dd-field-pipeline, explore-sum-without-rate, explore-topk-sum-without, explore-sum-without-count, explore-topk-by-per-step, explore-topk-without-keeps-labels.
Not run (capture budget): explore-topk-without-keeps-labels, explore-topk-by-per-step, explore-sum-without-count, explore-topk-sum-without, explore-sum-without-rate, dd-field-pipeline, dd-field-service_version, dd-labels-namespace, explore-without-nested-under-aggregation, explore-without-over-inner-aggregation, explore-nested-aggregation-outer-by, explore-max-by-count-over-time-sliding, explore-unwrap-first-last-under-sum, explore-unwrap-sum-over-time-sliding-window, explore-unwrap-max-over-time-end-sample, explore-json-error-filter-metric, explore-unwrap-quantile-over-time, explore-pattern-filter-anchored, explore-empty-pattern-filter, explore-unwrap-rate-sum, explore-unwrap-conversion-error, explore-unwrap-missing-label, explore-unwrap-unit-string, explore-unwrap-sum, explore-metric-line-filter-or, explore-logs-line-filter-or, explore-plain-level-metric, explore-logs-plain-level, explore-extracted-group-metric, explore-extracted-filter-metric, explore-extracted-metric, explore-logs-extracted, explore-logfmt-metric, explore-logs-json, explore-F, explore-E, explore-C.

Fix proof

Captures of the exact cases this pull request adds or fixes (registry case, base | PR | Loki). The verdict compares the PR with Loki where the base did not match. The gate fails a case unless one capture is fixed or improved, or, beyond the Loki data window, the base failed where Loki and the PR both answered with data (the label does not excuse it; only visual: none on the case does).

  • semantics/label-replace-instant-vector: no visual proof, exempt: the difference is on instant queries; the visual harness captures Explore range panels (queryType range), where label_replace already added the label, so the fix is proven by the instant tests and the
  • semantics/label-replace-new-label: no visual proof, exempt: the visible difference is on instant queries; the visual harness captures Explore range panels (queryType range), where label_replace already matched, so the fix is proven by the instant tests and the
    semantics/label-replace-range-binary-marker: 🟡 improved
page range base = PR (data) vs Loki (PR) verdict
explore-label-replace-over-binary 15m differs (1) 2 difference(s), 2 on base too 🟡 improved
explore-label-replace-over-binary 1h differs (1) 2 difference(s), 2 on base too 🟡 improved
explore-label-replace-over-binary 6h differs (1) n/a (beyond the Loki window) ⚪ undecided
explore-label-replace-over-binary-15m: improved

explore-label-replace-over-binary-15m

explore-label-replace-over-binary-1h: improved

explore-label-replace-over-binary-1h

explore-label-replace-over-binary-6h: undecided

explore-label-replace-over-binary-6h

Regression check

page range set base = PR (data) vs Loki (PR) pixel diff result
dd-fields 1h core ✅ identical identical (4 explained, 1 history-dependent) 0.12% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-landing 1h core ✅ identical identical (3 explained) 0.14% ✅
dd-service-logs 1h core ✅ identical identical (3 explained, 1 history-dependent) 0.12% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
explore-logs-plain 1h core ✅ identical identical (1 explained) 0.07% ✅
explore-metric-plain 1h core ✅ identical identical (1 explained) 0.04% ✅
explore-A 1h detailed ✅ identical identical (1 explained) 0.04% ✅
explore-B 1h detailed ✅ identical identical (1 explained) 0.04% ✅
explore-label-replace-over-binary 15m detailed differs (1): query ('label_replace(sum by (level) (rate({service_name="pa 2 difference(s), 2 on base too 1.12% ✅ improved (closer to Loki)
explore-label-replace-over-binary 1h detailed differs (1): query ('label_replace(sum by (level) (rate({service_name="pa 2 difference(s), 2 on base too 1.44% ✅ improved (closer to Loki)
explore-label-replace-over-binary 6h detailed differs (1): query ('label_replace(sum by (level) (rate({service_name="pa n/a (beyond the Loki window) 1.22% ✅ improved (closer to Loki)
explore-metric-plain 15m detailed ✅ identical identical (1 explained) 0.04% ✅
explore-metric-plain 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
lvl-dd-field-method-pill 1h detailed ✅ identical 1 difference(s), 1 on base too; 2 explained 0.11% ✅
lvl-dd-field-pipeline-pill 1h detailed ✅ identical 1 difference(s), 1 on base too; 2 explained, 1 history-dependent 0.12% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
lvl-dd-fields-pill 1h detailed ✅ identical 25 difference(s), 25 on base too; 4 explained 0.11% ✅
lvl-dd-label-pod-pill 1h detailed ✅ identical 1 difference(s), 1 on base too; 2 explained, 1 history-dependent 0.12% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
lvl-explore-count-warn-by-service 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.04% ✅
lvl-explore-logs-drop-dl 1h detailed ✅ identical 2 difference(s), 2 on base too; 1 explained 0.07% ✅
lvl-explore-logs-drop-level 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.07% ✅
lvl-explore-logs-json-level 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-logs-json-level-extracted-stored 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-logs-keep-dl 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.07% ✅
lvl-explore-logs-level-label-stored 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-logs-level-selector-stored 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-logs-logfmt-level-extracted-stored 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-logs-logfmt-level-regexp 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-rate-errors-by-service 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.04% ✅
lvl-explore-topk-errors-by-service 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.04% ✅
lvl-explore-volume-dl-prefilter-json 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-drop-level 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-json-level-filter 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-keep-dl 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-level-extracted-stored 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-level-label-stored 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-logfmt-dl 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-logfmt-dl-filter 1h detailed ✅ identical identical (1 explained) 0.04% ✅
tail-json-filter live detailed ✅ identical entries in the window: PR 1898, Loki 581, in both 581 6.90% ✅

Gate: a base-vs-PR data difference passes when the PR is closer to Loki, and fails when the PR diverges from Loki or Loki cannot decide it (at a range Loki does not hold, a difference passes when the same page proved that query to answer like Loki at a shorter range; otherwise label the pull request visual-change-expected to accept an intended change). Also failing: a panel empty on the PR but not on the base, an error banner, panel error, error answer or new "No data" panel on the PR, a PR side that never settled, a difference that did not reproduce on the recapture. Pixel differences only warn.

dd-fields-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-28T19%3A38%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-28T19%3A38%3A00.000Z&query=%7Bservic: body: content differs (rows 6 vs 2)
  • vs Loki, explained: query ('sum by (service_name_extracted) (count_over_time({service_name="otel-collector"} | servi: request only on one side (right) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision)
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-28T19%3A38%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-28T19%3A38%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-fields-1h

dd-service-logs-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-28T19%3A38%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-28T19%3A38%3A00.000Z&query=%7Bservic: body: content differs (rows 6 vs 2)
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-28T19%3A38%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-28T19%3A38%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-service-logs-1h

explore-label-replace-over-binary-6h: no Loki data at this range; the same differences match Loki at a shorter range (1): query ('label_replace(sum by (level)
  • no Loki data at this range; the same differences match Loki at a shorter range (1): query ('label_replace(sum by (level) (rate({service_name="payment-service"}[$__auto])) / sum by : series sets differ: 0 only left [], 3 only right ['(\'A\', \'\

explore-label-replace-over-binary-6h

lvl-dd-field-pipeline-pill-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-28T19%3A38%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-28T19%3A38%3A00.000Z&query=%7Bservic: body: content differs (rows 6 vs 2)
  • vs Loki: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-28T19%3A38%3A00.000Z&query=%7: body: content differs (rows 36 vs 15)
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-28T19%3A38%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

lvl-dd-field-pipeline-pill-1h

lvl-dd-label-pod-pill-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-28T19%3A38%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-28T19%3A38%3A00.000Z&query=%7Bservic: body: content differs (rows 6 vs 2)
  • vs Loki: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-28T19%3A38%3A00.000Z&query=%7: body: content differs (rows 36 vs 15)
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-28T19%3A38%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

lvl-dd-label-pod-pill-1h

Core set montages (3): base | PR | Loki

dd-landing-1h

dd-landing-1h

explore-logs-plain-1h

explore-logs-plain-1h

explore-metric-plain-1h

explore-metric-plain-1h

All montages of this run: pr-visuals/pr-744.

@szibis szibis mentioned this pull request Oct 12, 2026
Merged
szibis added a commit that referenced this pull request Oct 12, 2026
@szibis
chore(conformance): record #744 as the fix of label-replace-new-label
7d31aac
@szibis
szibis force-pushed the fix/label-replace-new-label branch from 3f174b0 to a52da44 Compare October 12, 2026 11:50
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 12, 2026
@szibis szibis mentioned this pull request Oct 12, 2026
Merged
szibis added a commit that referenced this pull request Oct 12, 2026
@szibis
chore(conformance): record #744 as the fix of label-replace-new-label
7b007e2
@szibis
szibis force-pushed the fix/label-replace-new-label branch from 17c6e6b to 45187a3 Compare October 12, 2026 13:52
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 12, 2026
szibis added a commit that referenced this pull request Oct 12, 2026
@szibis
chore(conformance): record #744 as the fix of label-replace-new-label
c033671
@szibis
szibis force-pushed the fix/label-replace-new-label branch from 45187a3 to c9c6d52 Compare October 12, 2026 15:03
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 12, 2026
github-advanced-security[bot]
github-advanced-security Bot found potential problems Oct 12, 2026
View reviewed changes
Comment thread internal/proxy/ordered_json_metric.go Fixed
szibis added a commit that referenced this pull request Oct 12, 2026
@szibis
chore(conformance): record #744 as the fix of label-replace-new-label
4828cc7
szibis added 13 commits October 28, 2026 17:44
@szibis
fix: label_replace adds its destination label to instant answers and …
4b752ce 
…reads escaped regexps

The transform ran on range matrices only, so an instant vector came back
without the new label. It now rewrites the labels of a vector and of a matrix
alike, keeps the rest of the answer, expands the replacement as Loki does and
files the points of series that end with one label set under one series on a
range query, as JoinSampleVector does.

The string arguments are unquoted like the LogQL lexer does, a call without
the four string arguments is a 400 parse error, topk/sort/stddev over a
label_replace no longer send the translator's marker to VictoriaLogs, a
label_replace around a binary expression on query_range routes by the
expression under it, and the without marker no longer swallows the markers
that follow it.
@szibis
chore(conformance): record #744 as the fix of label-replace-new-label
b5c1803
@szibis
fix: label_replace sorts its result like Loki, routes every branch by…
22877f5 
… the expression under it, and reports Loki's argument errors

The series of a label_replace answer are ordered by label set after the
transform (a range result, and an instant vector unless the query sorts by
value). The request is cloned once, above the routing chain, when a label
transform wraps the expression. A malformed call reports the token names and
the string start column of Loki's parser. label_join is no longer claimed as
a feature of the transform. Ranking over duplicate label sets that a
label_replace produced is registered as an open row.
@szibis
fix: a label transform around a binary clones the request from its ow…
710f1c0 
…n query, so each operand keeps its parser
@szibis
fix: label_replace around a bare range function keeps one series per …
87849f9 
…stream, and around a json metric keeps the parser-error semantics

An instant label_replace(count_over_time(...)) lost the grouping by stream
(the check looked for a query that starts with the function) and answered one
unlabeled total. A label_replace around a metric over | json skipped the
evaluator that owns JSON parser errors and answered 200 for lines Loki fails
or drops. Both now follow the expression under the transform; the e2e leaves
the structured metadata Loki adds to a bare range function's series (an
owner-kept deviation) out of the comparison and ranks one series under topk
so ties do not decide the shape.
@szibis
test(ab): add the label-replace A/B shape set
08db09b
@szibis
docs: changelog for the bare range function and json metric fixes, wi…
5035978 
…th the label_replace A/B results
@szibis
chore(conformance): register the label_replace gaps found by the e2e,…
7d9e720 
… regenerate the reports
@szibis
fix: a label transform over the ordered JSON evaluator answers with a…
94ae5cd 
… JSON content type
@szibis
chore(conformance): record #744 as the fix of the instant-vector and …
014af71 
…range-binary label_replace rows

The rows semantics/label-replace-instant-vector and
semantics/label-replace-range-binary-marker were opened on main while this
change was open, with tests that pinned the old answers. The rebased change
answers both as Loki does (pkg/logql/evaluator.go:385-388, LabelReplaceEvaluator.Next
at :1279-1309): an instant label_replace around a vector carries the destination
label, and query_range routes a label_replace around a binary by the binary and
sends no marker to VictoriaLogs.

TestInstantLabelReplaceOnAVectorAddsNoLabel and
TestRangeLabelReplaceOfABinaryExpressionSendsTheMarker were converted in the
rebase to assert Loki's answer (both fail on main); the label_replace wrapper of
TestWithoutWrappedByAnOperator drops its instant exception. Both rows are marked
fixed by #744, wired to TestCompat_LabelReplaceNewLabel (its "row" shape on
query and its "over a binary of two aggregations" shape on query_range), and
the range-binary row gets a visual fix-proof page. Reports regenerated.
@szibis
test(ab): add the label_replace shapes of the instant-vector and rang…
e08f845 
…e-binary rows

L7 is a label_replace around a binary of two sums by level on query_range (semantics/label-replace-range-binary-marker); L8 the instant label_replace over a sum by level of the registry request (semantics/label-replace-instant-vector). Generated performance evidence regenerated.
@szibis
chore(conformance): perf blocks for the label_replace rows the A/B sh…
f6d2b8b 
…apes cover

Main now requires a perf: block on every registry item an A/B shape covers
(conformance gate, perf acceptance). The label_replace shape sets cover
semantics/label-replace-new-label, label-replace-instant-vector,
label-replace-range-binary-marker and pipeline-error-text-parity. The blocks
were written by `bench/ab/over2x.py locks seed`: the default bound (2.0x Loki
max, 1.0x target, cold/warm/refresh), not locked, because the daily results
of 2026-10-28 have no rows for these shapes yet.
@szibis
docs: regenerate the quality and performance section
ddaf56e
@szibis
szibis force-pushed the fix/label-replace-new-label branch from a24484b to ddaf56e Compare October 28, 2026 16:47
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 28, 2026
@szibis
szibis marked this pull request as ready for review October 28, 2026 16:55
szibis added 2 commits October 28, 2026 18:36
@szibis
fix: a ranked topk reads the stats buckets, so a repeat and a moved w…
0c1ae25 
…indow hit the extent cache; make the label_replace e2e hermetic

topk and bottomk over a grouped count or rate ranked every series per step
with the three-grid window recurrence over /select/logsql/query. That
endpoint has no extent cache, so the L3 shape (topk over label_replace over
sum by count, 15s step, 1h) asked VictoriaLogs three full-range questions
on every request: 235 ms cold against 129 ms before the label_replace
handling moved it onto the ranked route, 227 ms on a repeat against 9 ms,
1.83x Loki. Before that handling the query went through the stats route,
which answered quickly but sent the translator's marker to VictoriaLogs as
the stats alias and ranked left-edge buckets.

Ranking now reads the stats_query_range buckets of the unranked metric (one
call cold; the stats extent cache answers an identical repeat with 0 calls
and a window moved back a minute with the 4 buckets it lacks) when they hold
every series, and falls back to the recurrence for more series than
-max-stats-query-series, a Drilldown request that keeps the busiest series,
a range equal to the step, or a metric with no exact bucket grid. The
caller's ranking runs on the transformed series, as Loki ranks after
label_replace.

Go benchmark of the L3 request on a fake VictoriaLogs (benchstat, n=6,
GOMAXPROCS=2), base / branch head / this change:
  cold    2.47 ms / 9.53 ms (3 calls) / 2.54 ms (1 call)
  repeat  0.62 ms / 8.93 ms (3 calls) / 1.02 ms (0 calls)
  moved   0.65 ms / 9.18 ms (3 calls) / 1.01 ms (0 calls)

The label_replace e2e now ingests its own streams under a unique probe
label and selects only them, asserts the fixture answers exactly three
series on Loki and on the proxy before comparing, and checks the series
count of the bare and aggregated shapes. An instant bare range function
lacks detected_level=unknown on a stream with no level (also on the base
commit, with no label_replace): registered open as
semantics/instant-bare-range-detected-level-unknown and pinned by a unit
test; the e2e leaves that label out of the bare instant shapes until it is
fixed.
@szibis
fix: a failing backend and an over-limit grouping do not cost the ran…
f86be5a 
…ked topk route more; rank before the label_replace around it

Follow-up to the ranked topk bucket route:
- a VictoriaLogs failure on the bucket read is the client's answer, as it is
  for the window recurrence, instead of three more full-range grid queries
  against a backend that is already failing; only the series limit, a body
  over the bound and a VictoriaLogs without stats_query_range fall back;
- a grouping over -max-stats-query-series is remembered for five minutes
  (the key of the Drilldown over-limit memo: tenant, limit, query), so its
  heavy bucket read is not repeated on every request;
- topk and bottomk without a grouping rank before the label_replace around
  them renames the series, as Loki ranks the samples a label_replace leaves
  apart (two series that end with one label set stay two samples) and files
  the survivors afterwards; ranking after the merge gave two points per step
  under one series;
- the differential covers count_over_time, rate, bytes_over_time and
  bytes_rate, and the label_replace e2e duplicates test selects its own
  fixture streams.
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 28, 2026
@szibis
fix: a ranked topk falls back to the window recurrence when the stats…
7526c65 
… buckets are unreadable

The bucket route treated a 2xx stats_query_range answer that is not a
Prometheus success body (empty object, not JSON, no status) as a backend
failure and answered 502 with its text. TestSlidingWindowStats_TopKRanksExactWindowsWithoutRawFetch
(168h, a 604.8 s step, whose emulator serves /select/logsql/query only) hit
that in CI. Such an answer now falls back to the recurrence, as every case
the old route handled does; a backend failure is still the client's answer
and a readable answer keeps the cached route.

Covered by a table over four unreadable answers and a whole-second and a
604.8 s step, and the differential now draws the 604.8 s step too.
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 28, 2026
@szibis
test: the label_replace e2e fixture has no ties, so Loki and the prox…
1e313f3 
…y have one ranking to agree on

under_topk/query_range failed in CI: Loki answered {app=api-gateway} and
{app=db-main}, the proxy only api-gateway. The fixture logged all three
streams at the same instants every 10 seconds, so every 5 minute window held
the same number of lines per stream and topk(1, ...) was a tie at every step
(16 of 16 steps in the unit reproduction); Loki's heap keeps the first tied
series it meets, the proxy another, and both are right. The ranked route
itself is exact: with distinct counts it equals the per-step reference
strictly.

The streams now log every 5, 10 and 15 seconds (60, 30 and 20 lines per
window, give or take one), starting 22 minutes back so no window of a 15
minute query is partial. The comparison stays strict. On a local isolated
stack the old fixture fails under_topk/query_range 3 of 3 runs and the new
one passes TestCompat_LabelReplace(NewLabel|DuplicatesBeforeRanking) 5 of 5
(72 subtests, none skipped). TestLabelReplaceE2EFixtureTopK pins both facts
without a stack.
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 28, 2026
@szibis
szibis merged commit e4350cb into main Oct 28, 2026
73 checks passed
@szibis
szibis deleted the fix/label-replace-new-label branch October 28, 2026 20:03
@szibis szibis mentioned this pull request Oct 28, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@github-advanced-security github-advanced-security[bot] github-advanced-security[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

bugfix Bug fix scope/docs Documentation scope/proxy Proxy core scope/tests Tests scope/translator LogQL translator size/XL Extra large change

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@szibis @github-advanced-security