Skip to content

perf: unwrap range metrics answer below Loki and rate_counter answers Loki's value - #915

Merged
szibis merged 33 commits into
mainfrom
perf/unwrap-under-loki
Oct 29, 2026
Merged

szibis merged 33 commits into
mainfrom
perf/unwrap-under-loki

Conversation

@szibis

@szibis szibis commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

Unwrap range metrics answer below Loki on most shapes, and rate_counter under sum answers Loki's value at every range. Builds on #885 and #886 (merged).

  • Settled steps: a per-evaluation-time cache of window-evaluated answers, like Loki's results cache. A moved window evaluates only its missing steps, and light gaps run concurrently. Behind -query-range-extent-cache.
  • VictoriaLogs folds a sliding sum without over count_over_time and bytes_over_time bucket by bucket. Supersedes the unmerged perf/sliding-without.
  • Quantile: a sliding Fenwick count. Value-count answers are read in place: 260 → 101 ms, 786 → 138 MiB per op, -97% allocations.
  • U9: sum(rate_counter(... | logfmt | unwrap ...)) answers Loki's zeros. Loki rates each one-sample parsed series as 0, and VictoriaLogs proves from stats that every series holds one sample, with no rows read. It used to answer a different value at 1h and 502 at 6h and above.
  • E6/E7 (Loki's 400): reached early, from the first window, for a query that erred lately. Clean queries pay nothing.

Results (loki-defaults profile, 1h/6h, refresh on; PR÷Loki, base÷Loki)

row cold warm refresh
E2 p95 6h 1.43 (2.73) 0.66 (0.78) 0.33 (0.35)
E7 6h 1.56 (3.07) 0.30 (0.62) 0.34 (0.59)
U9 rate_counter 6h 0.09 (base 502) 0.19 (6.71) 0.14 (5.05)
U14 6h 2.20 (1.89) 2.00 (2.00) 1.50

Tiers (cold / warm / refresh):

below Loki 1–2× above 2×
cold 101 7 1
warm 104 9 0
refresh 106 7 0

0 regressions against base, 0 new differences from Loki, 0 restarts.

At 24h, cold: E2 1.0 s vs Loki 3.0 s, U9 354 ms vs 2.3 s.

Open before ready

  • Rebased onto main after fix: label_replace adds its label to instant answers and to post-aggregated series, like Loki #744. The ranked-topk route from fix: label_replace adds its label to instant answers and to post-aggregated series, like Loki #744 and settled steps meet in proxyManualRangeMetricRange. Two items are still open after the rebase:

    • golangci gocyclo on proxyManualRangeMetricRange (31 > 30);
    • TestRankedTopKRangeFallsBackToTheWindowRecurrence fails.
  • Still above 2× Loki:

    • U14 6h cold 2.20× (base 1.89×) and warm 2.00×;
    • at 24h, warm, with the window moved one step: G7 20.5 vs 3 ms, D2 28 vs 10 ms, E2 28.5 vs 12.5 ms.

    An investigation is checking whether the ~14 ms extra per stream-filtered VictoriaLogs search is real or a host↔VM measurement artifact.

  • Refreshes that move both bounds off the step grid need -align-queries-with-step, default on (separate PR).

  • Settled steps and the folded-step cache (N14) would keep the same answer twice. One has to own those queries before N14 lands.

  • e2e not yet run on a stack: TestCompat_UnwrapRateCounterOneSampleSeries and TestCompat_UnwrapConversionErrorFirstWindowHeavyRead. The fleet proof for settled steps passed (35 tests) on the earlier head.

  • fixed_by: TBD-PR in the new registry rows.

@github-actions github-actions Bot added size/XL Extra large change scope/translator LogQL translator scope/proxy Proxy core scope/ci CI/CD scope/docs Documentation scope/cli CLI flags scope/tests Tests performance Performance labels Oct 28, 2026
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Performance A/B: faster, no regression

260 shape×range — 9 🟢 faster · 251 ⚪ same · results vs Loki: 239 same, 0 new difference(s), 21 pre-existing

Loki profiles compared (like for like):

  • aligned: not run (no build supports -align-queries-with-step, so every aligned row would be not like-for-like)
  • loki-defaults: 260 shape×range; proxy flag base off, pr off
shape range Loki profile base p50 PR p50 change Loki p50 PR ÷ Loki result vs Loki
🟢 N14 sliding sum of max 1h defaults 493 ms 41 ms ¹ 🟢 -92% 141 ms 0.29× ✅ same
🟢 SW1 sum without pod, count_over_time 1h defaults 304 ms 17 ms ¹ 🟢 -94% 83 ms 0.20× ✅ same
🟢 SW2 sum without pod, bytes_over_time 1h defaults 423 ms 20 ms ¹ 🟢 -95% 65 ms 0.31× ✅ same
🟢 SW3 sum without pod and version, line filter 1h defaults 325 ms 26 ms ¹ 🟢 -92% 62 ms 0.42× ✅ same
🟢 SW4 trailing sum without pod 1h defaults 292 ms 5 ms ¹ 🟢 -98% 58 ms 0.09× ✅ same
🟢 SW5 sum without pod, rate (control) 1h defaults 301 ms 24 ms ¹ 🟢 -92% 57 ms 0.42× ✅ sum ±0.00%
🟢 SW6 max without pod (control) 1h defaults 290 ms 9 ms ¹ 🟢 -97% 58 ms 0.16× ✅ same
🟢 W8 sum without pod, count_over_time (production) 1h defaults 317 ms 18 ms ¹ 🟢 -94% 82 ms 0.22× ✅ same
🟢 U9 logfmt amount rate_counter (folded per series) 1h defaults 274 ms 21 ms ¹ 🟢 -92% 50 ms 0.42× ✅ same

¹ re-measured with 7 runs, on windows the first pass did not use, after it moved beyond noise.

28504 characters omitted to fit the comment limit; every row is in the `summary-.json` files of the `perf-ab-smoke` artifact
Cold (first-run) timings — what a user opening a panel sees
shape range Loki profile base cold PR cold Loki cold PR ÷ Loki cold
N14 sliding sum of max 1h defaults 571 ms 464 ms 1.13 s ¹ 0.41×
SW1 sum without pod, count_over_time 1h defaults 380 ms 81 ms 506 ms ↺ 0.16×
SW2 sum without pod, bytes_over_time 1h defaults 612 ms 131 ms 650 ms ¹ 0.20×
SW3 sum without pod and version, line filter 1h defaults 408 ms 113 ms 658 ms ¹ 0.17×
SW4 trailing sum without pod 1h defaults 306 ms 32 ms 864 ms ¹ 0.04×
SW5 sum without pod, rate (control) 1h defaults 333 ms 299 ms 637 ms ¹ 0.47×
SW6 max without pod (control) 1h defaults 306 ms 204 ms 740 ms ¹ 0.28×
W8 sum without pod, count_over_time (production) 1h defaults 393 ms 92 ms 506 ms ¹ 0.18×
U9 logfmt amount rate_counter (folded per series) 1h defaults 263 ms 56 ms 368 ms † —
by service_name 1h defaults 38 ms 30 ms 1.09 s 0.03×
bytes_over_time by app 1h defaults 62 ms 62 ms 948 ms 0.07×
detected_level volume 1h defaults 115 ms 112 ms 894 ms 0.13×
drilldown detected_level 1h defaults 94 ms 94 ms 1.17 s 0.08×
drilldown level volume 1h defaults 14 ms 15 ms 885 ms 0.02×
drilldown service landing 1h defaults 42 ms 54 ms 948 ms 0.06×
instant json filtered instant defaults 9 ms 9 ms 99 ms 0.09×
instant json volume instant defaults 34 ms 32 ms 125 ms 0.26×
instant ungrouped rate instant defaults 7 ms 7 ms 98 ms 0.07×
json rate by level (no filter, no drop) 1h defaults 400 400 400 —
json volume no filters 1h defaults 60 ms 73 ms 897 ms ↺ 0.08×
logfmt count by level, filter, drop 1h defaults 448 ms 465 ms 1.37 s 0.34×
logfmt volume 1h defaults 223 ms 221 ms 1.52 s 0.15×
logs plain 1h defaults 103 ms 74 ms 128 ms 0.58×
logs | json 1h defaults 112 ms 112 ms 144 ms 0.78×
logs | logfmt 1h defaults 317 ms 227 ms 124 ms 1.83×
plain volume 1h defaults 36 ms 135 ms 1.30 s 0.10×
stream-label count by app 1h defaults 21 ms 20 ms 960 ms 0.02×
topk rate by app 1h defaults 19 ms 14 ms 1.21 s 0.01×
ungrouped rate 1h defaults 10 ms 11 ms 1.27 s 0.01×
DB1 label breakdown pod 1h defaults 130 ms 145 ms 1.32 s 0.11×
DB2 label breakdown namespace 1h defaults 26 ms 26 ms 689 ms 0.04×
DB3 json field breakdown user_id 1h defaults 56 ms 58 ms 824 ms 0.07×
DB4 json field breakdown batch_id 1h defaults 106 ms 104 ms 623 ms 0.17×
X1 json filter level_extracted, by level 1h defaults 3.41 s 4.15 s 890 ms 4.67×
X2 json group by level_extracted 1h defaults 349 ms 335 ms 913 ms 0.37×
X3 json rate, level_extracted equality 1h defaults 305 ms 301 ms 744 ms 0.40×
X4 json bytes, level_extracted regexp 1h defaults 489 ms 463 ms 883 ms 0.52×
X5 logfmt filter level_extracted, by level 1h defaults 702 ms 702 ms 1.25 s 0.56×
E1 logs, json parser over stream labels 1h defaults 199 ms 170 ms 99 ms † —
E2 logs, logfmt parser over stream labels 1h defaults 119 ms 91 ms 82 ms † —
E3 logs, json, existence filter on level_extracted 1h defaults 200 ms 183 ms 97 ms † —
E4 logs, logfmt, existence filter on level_extracted 1h defaults 129 ms 110 ms 91 ms † —
E5 logs, json, drop level_extracted 1h defaults 147 ms 125 ms 84 ms † —
E6 logfmt count by level_extracted 1h defaults 52 ms 53 ms 47 ms 1.13×
E7 json count by level_extracted 1h defaults 1.37 s 1.34 s 177 ms 7.56×
E8 instant logfmt by level_extracted instant defaults 17 ms 22 ms 11 ms 2.00×
DLV1 drilldown logs volume, json 1h defaults 1.31 s 1.30 s 198 ms 6.57×
DLV2 json error filter, detected_level 1h defaults 5.55 s 5.47 s 892 ms 6.13×
DLV3 plain json, detected_level 1h defaults 1.36 s 1.45 s 204 ms 7.11×
DLV4 explore json volume, level and detected_level 1h defaults 91 ms 93 ms 183 ms 0.51×
DLV5 json error dropped, detected_level 1h defaults 22 ms 22 ms 816 ms 0.03×
DLV6 drilldown detected_level breakdown, json 1h defaults 2 ms 1 ms 1.02 s 0.00×
J1 json no-error count by service_name, one service 1h defaults 192 ms 185 ms 179 ms 1.03×
J2 json no-error count by app, namespace 1h defaults 446 ms 451 ms 500 ms 0.90×
J3 json no-error count by parsed method 1h defaults 293 ms 216 ms 145 ms 1.49×
J4 json no-error count by service_name, all services 1h defaults 535 ms 516 ms 723 ms 0.71×
J5 json no-error bytes_rate by app 1h defaults 336 ms 336 ms 636 ms 0.53×
A explore volume, 2 filters, drop 1h defaults 1.48 s 1.47 s 454 ms 3.24×
B grouped sum, filter, no drop 1h defaults 125 ms 132 ms 401 ms 0.33×
C ungrouped sum, filter, no drop 1h defaults 126 ms 129 ms 373 ms 0.35×
D volume, no filters 1h defaults 213 ms 216 ms 897 ms 0.24×
E field breakdown pipeline (explore) 1h defaults 128 ms 110 ms 403 ms 0.27×
F field breakdown service_version (explore) 1h defaults 1.77 s 2.37 s 1.58 s 1.50×
G field breakdown pipeline (drilldown) 1h defaults 47 ms 46 ms 869 ms 0.05×
H field breakdown service_version (drilldown) 1h defaults 71 ms 74 ms 667 ms 0.11×
I grafana volume of B 1h defaults 114 ms 113 ms 592 ms 0.19×
M1 multi-value variable, log query 1h defaults 110 ms 98 ms 28 ms 3.50×
M2 multi-value variable, count by app 1h defaults 17 ms 17 ms 153 ms 0.11×
M3 single-value variable, log query 1h defaults 49 ms 51 ms 23 ms 2.22×
M4 alternation over many streams, count 1h defaults 17 ms 17 ms 464 ms 0.04×
M5 json label filter, literal alternation 1h defaults 160 ms 125 ms 168 ms 0.74×
M6 json label filter, prefix (substring in Loki) 1h defaults 111 ms 110 ms 36 ms † —
L1 label_replace over sum by rate, instant instant defaults 34 ms 22 ms 88 ms 0.25×
L2 label_replace over sum by rate 1h defaults 134 ms 133 ms 905 ms 0.15×
L3 topk over label_replace over sum by count 1h defaults 13 ms 11 ms 587 ms 0.02×
L4 label_replace over bare count_over_time, instant instant defaults 21 ms 23 ms 26 ms 0.88×
L5 label_replace over json sum by, instant instant defaults 58 ms 60 ms 28 ms 2.14×
L6 label_replace over json sum by 1h defaults 146 ms 135 ms 175 ms 0.77×
L7 label_replace around a binary of two sums by level 1h defaults 40 ms 40 ms 63 ms † —
L8 label_replace over sum by level count, instant instant defaults 15 ms 15 ms 12 ms 1.25×
LF1 logs, detected_level equals 1h defaults 222 ms 204 ms 81 ms 2.52×
LF10 drilldown logs, level pill multi-select, json service 1h defaults 158 ms 162 ms 39 ms 4.15×
LF11 drilldown logs volume by level, existing level 1h defaults 603 ms 603 ms 660 ms 0.91×
LF12 drilldown logs volume by level, level pill 1h defaults 168 ms 168 ms 213 ms 0.79×
LF13 explore volume, level filter before json 1h defaults 129 ms 113 ms 529 ms 0.21×
LF14 explore volume, plain-text service 1h defaults 65 ms 63 ms 25 ms 2.52×
LF15 drilldown field breakdown, level filter 1h defaults 99 ms 104 ms 102 ms 1.02×
LF16 drilldown label breakdown, level filter 1h defaults 60 ms 56 ms 53 ms 1.06×
LF17 detected_level count, level regexp 1h defaults 537 ms 502 ms 557 ms 0.90×
LF18 detected_level count, level not equal 1h defaults 505 ms 501 ms 592 ms 0.85×
LF19 detected_level count, regexp with unknown 1h defaults 496 ms 488 ms 507 ms 0.96×
LF2 logs, detected_level regexp 1h defaults 213 ms 211 ms 32 ms 6.59×
LF20 logfmt count by detected_level 1h defaults 65 ms 63 ms 62 ms 1.02×
LF21 logfmt count by detected_level, level filter 1h defaults 40 ms 39 ms 51 ms 0.76×
LF22 json count by detected_level, level filter 1h defaults 23 ms 21 ms 166 ms 0.13×
LF23 json count by level, level filter 1h defaults 62 ms 63 ms 174 ms 0.36×
LF24 rate of errors by service 1h defaults 446 ms 447 ms 833 ms 0.54×
LF25 count of warnings, error label dropped 1h defaults 441 ms 439 ms 556 ms 0.79×
LF26 topk services by errors 1h defaults 7 ms 6 ms 563 ms 0.01×
LF27 detected_level count, instant instant defaults 65 ms 51 ms 85 ms 0.60×
LF28 logs, keep detected_level 1h defaults 65 ms 64 ms 20 ms † —
LF29 logs, drop detected_level 1h defaults 92 ms 94 ms 20 ms 4.70×
LF3 logs, detected_level not equal, all services 1h defaults 231 ms 229 ms 72 ms 3.18×
LF30 logs, drop level 1h defaults 93 ms 96 ms 19 ms 5.05×
LF31 count by detected_level, level dropped 1h defaults 31 ms 32 ms 143 ms 0.22×
LF32 count by detected_level, detected_level kept 1h defaults 31 ms 30 ms 151 ms 0.20×
LF33 drilldown logs count, level pill, instant instant defaults 17 ms 21 ms 18 ms 1.17×
LF4 logs, detected_level regexp with unknown 1h defaults 459 ms 435 ms 115 ms 3.78×
LF5 logs, plain-text service, detected_level regexp 1h defaults 82 ms 79 ms 15 ms 5.27×
LF6 logs, json then detected_level 1h defaults 160 ms 167 ms 204 ms † —
LF7 logs, json then level equals 1h defaults 159 ms 161 ms 202 ms † —
LF8 logs, logfmt then level regexp 1h defaults 117 ms 113 ms 90 ms † —
LF9 drilldown logs, level pill, logfmt service 1h defaults 88 ms 96 ms 29 ms 3.31×
LS1 logs, level stream label 1h defaults 66 ms 59 ms 74 ms 0.80×
LS2 logs, level stream label regexp, all services 1h defaults 111 ms 113 ms 82 ms 1.38×
LS3 logs, level in the selector 1h defaults 51 ms 50 ms 27 ms 1.85×
LS4 logs, json then level_extracted 1h defaults 213 ms 196 ms 182 ms † —
LS5 logs, logfmt then level_extracted regexp 1h defaults 93 ms 86 ms 65 ms † —
LS6 level stream label count 1h defaults 18 ms 19 ms 491 ms 0.04×
LS7 rate of level stream label by service 1h defaults 23 ms 23 ms 541 ms 0.04×
LS8 json count by level_extracted, collision 1h defaults 1.33 s 1.32 s 169 ms 7.79×
O1 explore logs, underscore structured-metadata filter 1h defaults 418 ms 358 ms 497 ms † —
O2 explore log volume (Grafana form), underscore structured-metadata filter 1h defaults 173 ms 179 ms 477 ms 0.38×
O3 explore logs, dotted structured-metadata filter 1h defaults 400 400 400 —
O4 explore log volume (Grafana form), dotted structured-metadata filter 1h defaults 400 400 400 —
label values env ending now 1h defaults 195 ms 200 ms 43 ms † —
label values env ending two days ago (control) 1h defaults 7 ms 6 ms 2 ms † —
labels ending now 1h defaults 148 ms 152 ms 4 ms † —
labels ending two days ago (control) 1h defaults 3 ms 3 ms 2 ms † —
drilldown detected fields 1h defaults 40 ms 40 ms 29 ms † —
drilldown detected labels 1h defaults 22 ms 21 ms 3 ms † —
drilldown landing volume 1h defaults 61 ms 88 ms 85 ms 1.04×
explore label values env 1h defaults 173 ms 178 ms 45 ms † —
explore label values service_name 1h defaults 89 ms 68 ms 53 ms † —
explore labels (label browser) 1h defaults 133 ms 98 ms 3 ms † —
explore labels with selector 1h defaults 111 ms 124 ms 58 ms † —
refresh label values env 1h defaults 6 ms 6 ms 39 ms † —
refresh label values service_name 1h defaults 3 ms 3 ms 54 ms † —
refresh labels (label browser) 1h defaults 3 ms 3 ms 2 ms † —
refresh labels with selector 1h defaults 3 ms 3 ms 59 ms † —
DL1 drilldown field breakdown json-expr, level filter 1h defaults 517 ms 498 ms 587 ms 0.85×
DL2 drilldown field breakdown, level + stream-label filter before json 1h defaults 69 ms 49 ms 37 ms 1.32×
DL3 drilldown logfmt field breakdown, level filter 1h defaults 680 ms 663 ms 1.45 s 0.46×
DL4 instant count, level filter, json+logfmt, 1h instant defaults 522 ms 511 ms 1.27 s 0.40×
DL6 explore volume, level filter before json 1h defaults 209 ms 200 ms 825 ms 0.24×
N1 sum by app of max by app,pod, count 1h defaults 67 ms 59 ms 568 ms 0.10×
N10 instant min of avg, rate instant defaults 17 ms 14 ms 90 ms 0.16×
N11 instant three levels instant defaults 23 ms 25 ms 154 ms 0.16×
N12 instant json inner, count of sum instant defaults 23 ms 24 ms 86 ms † —
N13 instant without under by instant defaults 29 ms 25 ms 97 ms 0.26×
N15 sliding sums only 1h defaults 29 ms 29 ms 385 ms 0.08×
N16 innermost without under by, tumbling 1h defaults 65 ms 61 ms 650 ms 0.09×
N2 max by namespace of sum by namespace,app, rate 1h defaults 26 ms 28 ms 357 ms 0.08×
N3 count by namespace of sum by namespace,app 1h defaults 22 ms 25 ms 352 ms † —
N4 avg by app of min by app,pod 1h defaults 59 ms 52 ms 457 ms 0.11×
N5 three levels, max min sum 1h defaults 28 ms 32 ms 380 ms 0.08×
N6 json inner, sum of max 1h defaults 211 ms 250 ms 875 ms 0.29×
N7 logfmt inner, max of sum 1h defaults 35 ms 34 ms 76 ms 0.45×
N8 without over a named by level, tumbling 1h defaults 84 ms 78 ms 849 ms 0.09×
N9 instant sum of max instant defaults 15 ms 18 ms 91 ms 0.20×
OR1 logs, or alternatives 1h defaults 129 ms 123 ms 87 ms 1.41×
OR2 logs, negated or alternatives 1h defaults 117 ms 122 ms 67 ms 1.82×
OR3 logs, regexp or alternatives 1h defaults 104 ms 86 ms 116 ms 0.74×
OR4 count by level, or alternatives with parser 1h defaults 135 ms 138 ms 431 ms 0.32×
OR5 rate, negated regexp or alternatives 1h defaults 61 ms 70 ms 577 ms 0.12×
OR6 instant, or alternatives instant defaults 15 ms 15 ms 85 ms 0.18×
PR1 count by level, parenthesised json 1h defaults 252 ms 208 ms 712 ms 0.29×
PR2 rate, parenthesised line filter 1h defaults 53 ms 55 ms 456 ms 0.12×
PR3 bytes by app, parenthesised regexp 1h defaults 58 ms 56 ms 499 ms 0.11×
PR4 count by level, parenthesised parser, offset 1h defaults 55 ms 54 ms 1.02 s 0.05×
PR5 rate by app, pipeline after the range 1h defaults 238 ms 244 ms 747 ms 0.33×
PR6 doubly parenthesised range 1h defaults 1 ms 1 ms 452 ms 0.00×
PR7 instant, parenthesised line filter instant defaults 15 ms 15 ms 85 ms 0.18×
P1 api-gateway empty pattern 1h defaults 42 ms 32 ms 78 ms † —
P2 api-gateway not empty pattern, line filter 1h defaults 106 ms 109 ms 30 ms 3.63×
P3 nginx count_over_time, pattern with spaces 1h defaults 20 ms 19 ms 23 ms † —
P4 nginx count_over_time, GET request pattern 1h defaults 20 ms 20 ms 22 ms 0.91×
A1 nginx include GET, log query 1h defaults 52 ms 43 ms 7 ms 6.14×
A2 nginx exclude GET, log query 1h defaults 137 ms 124 ms 8 ms † —
A3 nginx include GET, count_over_time 1h defaults 18 ms 18 ms 22 ms ↺ 0.82×
A4 nginx exclude GET, count_over_time 1h defaults 25 ms 23 ms 36 ms 0.64×
A5 drilldown exclude, volume by detected_level 1h defaults 51 ms 51 ms 33 ms 1.55×
A6 nginx include ends on a literal, log query 1h defaults 22 ms 23 ms 25 ms † —
A7 nginx exclude ends on a literal, log query 1h defaults 99 ms 99 ms 9 ms 11.00×
A8 nginx include two literals, count_over_time 1h defaults 26 ms 26 ms 32 ms 0.81×
A9 nginx include two literals, ends on a literal, count_over_time 1h defaults 25 ms 24 ms 27 ms † —
P1 logs, json, level equals 1h defaults 174 ms 174 ms 71 ms † —
P2 logs, logfmt, level not equal 1h defaults 40 ms 40 ms 30 ms † —
P3 logs, json, level existence 1h defaults 159 ms 151 ms 74 ms † —
P4 logfmt count by level 1h defaults 36 ms 34 ms 35 ms 0.97×
P5 json count by level 1h defaults 101 ms 103 ms 243 ms 0.42×
P6 json count, level filter 1h defaults 21 ms 21 ms 124 ms 0.17×
P7 json rate by level, level regexp 1h defaults 20 ms 21 ms 133 ms 0.16×
P8 json count by a name that is no stream label 1h defaults 104 ms 110 ms 125 ms 0.88×
P9 instant logfmt by level instant defaults 14 ms 16 ms 9 ms 1.78×
SF1 logs, json extraction list 1h defaults 173 ms 168 ms 67 ms 2.51×
SF10 volume, structured-metadata filter 1h defaults 34 ms 35 ms 411 ms 0.09×
SF2 logs, logfmt extraction list 1h defaults 202 ms 193 ms 63 ms 3.06×
SF3 logs, json then line_format 1h defaults 138 ms 165 ms 86 ms 1.92×
SF4 logs, line_format without a parser 1h defaults 95 ms 92 ms 57 ms 1.61×
SF5 logs, regexp capture 1h defaults 404 ms 244 ms 72 ms 3.39×
SF6 logs, structured-metadata filter 1h defaults 81 ms 81 ms 146 ms † —
SF7 logs, filter on a line key without a parser 1h defaults 138 ms 130 ms 517 ms † —
SF8 volume, json extraction list 1h defaults 26 ms 25 ms 814 ms 0.03×
SF9 volume, logfmt label_format rename 1h defaults 551 ms 544 ms 1.55 s 0.35×
E1 json duration_ms sum by service 1h defaults 105 ms 96 ms 218 ms 0.44×
E2 json duration_ms p95 by service 1h defaults 267 ms 193 ms 254 ms ¹ 0.76×
E3 logfmt latency_ms rate, production 1h defaults 524 ms 536 ms 1.43 s 0.37×
E4 logfmt size_bytes max by service 1h defaults 53 ms 56 ms 268 ms 0.21×
E5 json duration_ms sum, instant instant defaults 38 ms 37 ms 30 ms 1.23×
E6 logfmt ttl unit string (Loki 400) 1h defaults 400 400 400 —
E7 json hexadecimal trace_id (Loki 400) 1h defaults 400 400 400 —
E8 json duration_ms sum by service, namespace prod 1h defaults 219 ms 222 ms 688 ms 0.32×
E9 logfmt latency_ms sum by service, namespace prod 1h defaults 339 ms 358 ms 846 ms 0.42×
D1 logfmt ttl duration_seconds rate by service 1h defaults 80 ms 71 ms 51 ms 1.39×
D2 logfmt ttl duration rate by service 1h defaults 2 ms 2 ms 58 ms 0.03×
D3 logfmt ttl duration_seconds rate, instant instant defaults 30 ms 38 ms 10 ms 3.80×
D4 logfmt ttl duration rate, instant instant defaults 29 ms 31 ms 10 ms 3.10×
G1 logfmt amount sum_over_time, no filter 1h defaults 63 ms 45 ms 70 ms 0.64×
G10 logfmt latency_ms sum_over_time, production 1h defaults 514 ms 508 ms 1.08 s 0.47×
G11 logfmt amount sum_over_time, instant instant defaults 15 ms 16 ms 13 ms 1.23×
G12 json duration_ms sum_over_time, instant instant defaults 44 ms 40 ms 31 ms 1.29×
G2 json duration_ms sum_over_time, no filter 1h defaults 102 ms 101 ms 208 ms 0.49×
G3 logfmt amount sum_over_time, error filter 1h defaults 49 ms 48 ms 69 ms 0.70×
G4 json duration_ms sum_over_time, error filter 1h defaults 101 ms 101 ms 198 ms 0.51×
G5 logfmt amount sum by rate 1h defaults 47 ms 51 ms 60 ms ↺ 0.85×
G6 json duration_ms sum by rate 1h defaults 108 ms 102 ms 200 ms 0.51×
G7 duration() of a unit string 1h defaults 73 ms 71 ms 52 ms 1.37×
G8 bytes() of a number 1h defaults 71 ms 69 ms 45 ms 1.53×
G9 json duration_ms max_over_time 1h defaults 101 ms 106 ms 2.16 s 0.05×
W1 sum without pod, max_over_time 1h defaults 95 ms 70 ms 102 ms 0.69×
W2 sum without pod, sum_over_time 1h defaults 83 ms 77 ms 79 ms 0.97×
W3 sum without pod, rate 1h defaults 94 ms 87 ms 80 ms 1.09×
W4 max without pod, max_over_time 1h defaults 37 ms 29 ms 91 ms 0.32×
W5 sum without pod, max_over_time, instant instant defaults 16 ms 16 ms 16 ms 1.00×
W6 sum without pod, rate, instant instant defaults 17 ms 16 ms 15 ms 1.07×
W7 topk 5 of sum without pod 1h defaults 39 ms 29 ms 114 ms 0.25×
W9 label_replace control, instant instant defaults 14 ms 14 ms 90 ms 0.16×
U1 logfmt amount rate 1h defaults 60 ms 55 ms 56 ms 0.98×
U2 json duration_ms rate 1h defaults 95 ms 96 ms 138 ms 0.70×
U3 json duration_ms rate, error filter 1h defaults 93 ms 98 ms 260 ms 0.38×
U4 logfmt latency_ms rate, production 1h defaults 380 ms 371 ms 992 ms 0.37×
U5 logfmt amount rate, instant instant defaults 16 ms 17 ms 20 ms 0.85×
U6 json duration_ms rate, instant instant defaults 26 ms 22 ms 27 ms 0.81×
U7 json duration_ms sum_over_time, instant instant defaults 22 ms 23 ms 27 ms 0.85×
U8 logfmt amount rate by service (grouped) 1h defaults 47 ms 47 ms 60 ms 0.78×
U1 json duration_ms sum by service 1h defaults 117 ms 98 ms 204 ms 0.48×
U10 instant sum_over_time unwrap instant defaults 42 ms 42 ms 29 ms 1.45×
U11 instant max_over_time missing label instant defaults 14 ms 16 ms 19 ms † —
U12 json duration_ms over every service 1h defaults 291 ms 256 ms 752 ms 0.34×
U13 logfmt latency_ms over every service 1h defaults 500 ms 501 ms 1.65 s 0.30×
U14 duration() of a unit string 1h defaults 65 ms 66 ms 51 ms 1.29×
U2 json duration_ms max by service 1h defaults 97 ms 93 ms 987 ms 0.09×
U3 json duration_ms sliding sum 1h defaults 16 ms 14 ms 207 ms 0.07×
U4 logfmt size_bytes sum by service 1h defaults 53 ms 52 ms 44 ms 1.18×
U5 logfmt ttl unit string (no sample) 1h defaults 49 ms 48 ms 50 ms † —
U6 missing label, no parser 1h defaults 21 ms 21 ms 88 ms † —
U7 missing label, ungrouped sum 1h defaults 56 ms 62 ms 133 ms † —
U8 rate of an unwrapped label 1h defaults 211 ms 114 ms 193 ms 0.59×
U9 bytes() conversion 1h defaults 66 ms 67 ms 50 ms 1.34×

† Loki's results cache answered this cold run (its own counters, or identical answers for windows a minute apart): not a computed time, so it is in no ratio.

¹ (Loki cold) the confirm pass's cold run was cache-served, so every cold column of the row is the first pass's.

↺ Loki cold taken from the first set that ran the same query; Loki's cache answered this one.

45 shape×range above 2× Loki already (listed, or as slow on the base build)
profile mode shape range PR Loki PR ÷ Loki base ÷ Loki listed ÷ Loki why
known defaults cold A explore volume, 2 filters, drop 1h 1.47 s 454 ms 3.24× 3.25× — as slow on base
known defaults cold SF1 logs, json extraction list 1h 168 ms 67 ms 2.51× 2.58× — as slow on base
known defaults warm SF1 logs, json extraction list 1h 141 ms 66 ms 2.14× 2.48× — as slow on base
known defaults cold SF2 logs, logfmt extraction list 1h 193 ms 63 ms 3.06× 3.21× — as slow on base
known defaults warm SF2 logs, logfmt extraction list 1h 206 ms 69 ms 2.99× 2.86× — as slow on base
known defaults cold SF5 logs, regexp capture 1h 244 ms 72 ms 3.39× 5.61× — as slow on base
known defaults warm SF5 logs, regexp capture 1h 198 ms 70 ms 2.83× 2.76× — as slow on base
known defaults warm SF7 logs, filter on a line key without a parser 1h 141 ms 52 ms 2.71× 2.50× — as slow on base
known defaults warm E3 logs, json, existence filter on level_extracted 1h 207 ms 96 ms 2.16× 2.16× — as slow on base
known defaults cold E7 json count by level_extracted 1h 1.34 s 177 ms 7.56× 7.76× — as slow on base
known defaults warm E7 json count by level_extracted 1h 95 ms 20 ms 4.75× 4.85× — as slow on base
known defaults warm OR2 logs, negated or alternatives 1h 195 ms 68 ms 2.87× 2.79× — as slow on base
known defaults warm P3 logs, json, level existence 1h 150 ms 70 ms 2.14× 2.14× — as slow on base
known defaults cold X1 json filter level_extracted, by level 1h 4.15 s 890 ms 4.67× 3.83× — as slow on base
known defaults cold P2 api-gateway not empty pattern, line filter 1h 109 ms 30 ms 3.63× 3.53× — as slow on base
known defaults warm P2 api-gateway not empty pattern, line filter 1h 108 ms 20 ms 5.40× 5.35× — as slow on base
known defaults warm A2 nginx exclude GET, log query 1h 111 ms 7 ms 16× 17× — as slow on base
known defaults cold A7 nginx exclude ends on a literal, log query 1h 99 ms 9 ms 11× 11× — as slow on base
known defaults cold DLV1 drilldown logs volume, json 1h 1.30 s 198 ms 6.57× 6.62× — as slow on base
known defaults warm DLV1 drilldown logs volume, json 1h 1.28 s 24 ms 54× 54× — as slow on base
known defaults cold DLV2 json error filter, detected_level 1h 5.47 s 892 ms 6.13× 6.22× — as slow on base
known defaults warm DLV2 json error filter, detected_level 1h 5.49 s 106 ms 52× 53× — as slow on base
known defaults cold DLV3 plain json, detected_level 1h 1.45 s 204 ms 7.11× 6.66× — as slow on base
known defaults warm DLV3 plain json, detected_level 1h 1.31 s 20 ms 65× 66× — as slow on base
known defaults cold LF1 logs, detected_level equals 1h 204 ms 81 ms 2.52× 2.74× — as slow on base
known defaults warm LF1 logs, detected_level equals 1h 200 ms 80 ms 2.50× 2.56× — as slow on base
known defaults cold LF2 logs, detected_level regexp 1h 211 ms 32 ms 6.59× 6.66× — as slow on base
known defaults warm LF2 logs, detected_level regexp 1h 187 ms 31 ms 6.03× 6.35× — as slow on base
known defaults cold LF3 logs, detected_level not equal, all services 1h 229 ms 72 ms 3.18× 3.21× — as slow on base
known defaults cold LF4 logs, detected_level regexp with unknown 1h 435 ms 115 ms 3.78× 3.99× — as slow on base
known defaults warm LF4 logs, detected_level regexp with unknown 1h 409 ms 113 ms 3.62× 3.60× — as slow on base
known defaults cold LF5 logs, plain-text service, detected_level regexp 1h 79 ms 15 ms 5.27× 5.47× — as slow on base
known defaults warm LF5 logs, plain-text service, detected_level regexp 1h 81 ms 14 ms 5.79× 5.79× — as slow on base
known defaults cold LF9 drilldown logs, level pill, logfmt service 1h 96 ms 29 ms 3.31× 3.03× — as slow on base
known defaults warm LF9 drilldown logs, level pill, logfmt service 1h 86 ms 28 ms 3.07× 3.11× — as slow on base
known defaults cold LF10 drilldown logs, level pill multi-select, json service 1h 162 ms 39 ms 4.15× 4.05× — as slow on base
known defaults warm LF10 drilldown logs, level pill multi-select, json service 1h 152 ms 39 ms 3.90× 3.95× — as slow on base
known defaults cold LF29 logs, drop detected_level 1h 94 ms 20 ms 4.70× 4.60× — as slow on base
known defaults warm LF29 logs, drop detected_level 1h 90 ms 21 ms 4.29× 3.90× — as slow on base
known defaults cold LF30 logs, drop level 1h 96 ms 19 ms 5.05× 4.89× — as slow on base
… and 5 more (verdict.json counts them; the job's summary-<set>.json files have every row).
Why these shapes
  • baseline run: every shape of every set → 260: control/by service_name, control/bytes_over_time by app, control/detected_level volume, control/drilldown detected_level, control/drilldown level volume, control/drilldown service landing, control/instant json filtered, control/instant json volume …

base a9469549f9 → PR 5ee42ab923 · 4 runs (first is cold), warm p50 · ranges 1h · noise ±30% and 100 ms · VictoriaLogs restarts 0→0 · 12.5 min · phases stack_up 36s, seed_and_build 153s, loki-defaults:loki_profile 0s, loki-defaults:proxies 2s, loki-defaults:matrix 543s. Same runner, one fresh stack, targets interleaved per request. How to read and reproduce: bench/ab/README.md.

🧭 New baseline candidate

This pull request got the baseline run. Every shape of every set was measured at the smoke's ranges (1h, 4 runs). Nothing was committed. The perf-baseline-candidate artifact of this run holds over-2x-loki.json, which a person can copy over bench/ab/history/over-2x-loki.json to accept it, and acceptance-day.json, a report: the committed acceptance.json is rebuilt from the daily history, so the next daily run after the merge writes the real new distribution.

profile mode rows compared below Loki close 1–2× above 2× share below committed share below
loki-defaults cold 210 115 68 5 22 54.8% 50.5% (2026-10-29)
loki-defaults warm 255 64 161 7 23 25.1% 23.2% (2026-10-29)

The committed column is the newest daily run (7 runs, ranges up to 24h): the direction is comparable, the rows are not the same ones.

Above-2× list: 210 rows after this run, 45 added, 0 not above 2× in this run's profile (the daily run's two-run rule decides for the committed list). A profile that did not run leaves its committed rows as they are.

Added:

  • loki-defaults / extracted-pushdown / X1 json filter level_extracted, by level / 1h / cold
  • loki-defaults / extracted-suffix / E3 logs, json, existence filter on level_extracted / 1h / warm
  • loki-defaults / extracted-suffix / E7 json count by level_extracted / 1h / cold
  • loki-defaults / extracted-suffix / E7 json count by level_extracted / 1h / warm
  • loki-defaults / json-detected-level / DLV1 drilldown logs volume, json / 1h / cold
  • loki-defaults / json-detected-level / DLV1 drilldown logs volume, json / 1h / warm
  • loki-defaults / json-detected-level / DLV2 json error filter, detected_level / 1h / cold
  • loki-defaults / json-detected-level / DLV2 json error filter, detected_level / 1h / warm
  • loki-defaults / json-detected-level / DLV3 plain json, detected_level / 1h / cold
  • loki-defaults / json-detected-level / DLV3 plain json, detected_level / 1h / warm
  • … and 35 more (in over-2x-loki.json)

13 min.

@szibis
szibis marked this pull request as ready for review October 28, 2026 20:26
@szibis
szibis force-pushed the perf/unwrap-under-loki branch from e72c31b to b37c3ef Compare October 28, 2026 20:30
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 28, 2026
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

PR Quality Report

Compared against base branch main.

Coverage and tests

Signal Base PR Delta
Test count 18517 18657 140
Coverage 88.9% 89.1% +0.2% (improved)

Compatibility

Track Base PR Delta
Loki API 100.0% 11/11 (100.0%) 0.0% (stable)
Logs Drilldown 100.0% 17/17 (100.0%) 0.0% (stable)
VictoriaLogs 100.0% 11/11 (100.0%) 0.0% (stable)

Performance smoke

Lower CPU cost (ns/op) is better. Lower benchmark memory cost (B/op, allocs/op) is better. Higher throughput is better. Lower load-test memory growth is better. Benchmark rows are medians from repeated samples.

Signal Base PR Delta
QueryRange cache-hit CPU cost 3503.0 ns/op 3618.0 ns/op +3.3% (stable)
QueryRange cache-hit memory 1240.0 B/op 1256.0 B/op +1.3% (stable)
QueryRange cache-hit allocations 27.0 allocs/op 28.0 allocs/op +3.7% (stable)
QueryRange cache-bypass CPU cost 4273.0 ns/op 4543.0 ns/op +6.3% (stable)
QueryRange cache-bypass memory 1303.0 B/op 1324.0 B/op +1.6% (stable)
QueryRange cache-bypass allocations 27.0 allocs/op 28.0 allocs/op +3.7% (stable)
Labels cache-hit CPU cost 906.8 ns/op 924.3 ns/op +1.9% (stable)
Labels cache-hit memory 48.0 B/op 48.0 B/op 0.0% (stable)
Labels cache-hit allocations 3.0 allocs/op 3.0 allocs/op 0.0% (stable)
Labels cache-bypass CPU cost 1497.0 ns/op 1543.0 ns/op +3.1% (stable)
Labels cache-bypass memory 400.0 B/op 400.0 B/op 0.0% (stable)
Labels cache-bypass allocations 14.0 allocs/op 14.0 allocs/op 0.0% (stable)

State

  • Coverage, compatibility, and sampled performance are reported here from the same PR workflow.
  • This is a delta report, not a release gate by itself. Required checks still decide merge safety.
  • Performance is a smoke comparison, not a full benchmark lab run.
  • Delta states use the same noise guards as the quality gate (percent + absolute + low-baseline checks), so report labels match merge-gate behavior.

@github-actions github-actions Bot added size/XL Extra large change scope/cache Cache layer and removed size/XL Extra large change labels Oct 28, 2026
@szibis
szibis force-pushed the perf/unwrap-under-loki branch from ce9749b to 6c1e855 Compare October 28, 2026 22:02
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 28, 2026
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

✅ Visual smoke: passed with warnings

Base a9469549f9 vs PR 6a4a828c14, Grafana through the proxy builds and Loki on one fresh stack; run.
Loaded again, once and more patiently, after a first-pass difference or an unsettled page: explore-unwrap-rate-counter-one-sample 15m, explore-unwrap-rate-counter-one-sample 1h, explore-unwrap-rate-counter-one-sample 6h.
Core set: 5 capture(s) at 1h. Detailed set: 108 entries (explore-logs-plain, explore-metric-plain, lvl-dd-logs-pill-json, lvl-dd-logs-pill-otel, lvl-dd-fields-pill, lvl-dd-field-method-pill, lvl-dd-field-pipeline-pill, lvl-dd-logs-pill-logfmt, ...). 228 capture(s), 15m/1h/6h, 24h and 7d stay out of PR runs.

page range set base = PR (data) vs Loki (PR) pixel diff result
dd-fields 1h core ✅ identical identical (4 explained, 1 history-dependent) 0.11% ✅
dd-landing 1h core ✅ identical identical (3 explained) 0.14% ✅
dd-service-logs 1h core ✅ identical identical (3 explained, 1 history-dependent) 0.11% ✅
explore-logs-plain 1h core ✅ identical identical (1 explained) 0.07% ✅
explore-metric-plain 1h core ✅ identical identical (1 explained) 0.04% ✅
dd-field-pipeline 15m detailed ✅ identical identical (3 explained, 1 history-dependent) 0.11% ✅
dd-field-pipeline 1h detailed ✅ identical identical (3 explained, 1 history-dependent) 0.11% ✅
dd-field-pipeline 6h detailed ✅ identical n/a (beyond the Loki window) 0.11% ✅
dd-field-service_version 15m detailed ✅ identical identical (3 explained, 1 history-dependent) 0.11% ✅
dd-field-service_version 1h detailed ✅ identical identical (3 explained, 1 history-dependent) 0.11% ✅
dd-field-service_version 6h detailed ✅ identical n/a (beyond the Loki window) 0.11% ✅
dd-fields 15m detailed ✅ identical identical (4 explained, 1 history-dependent) 0.11% ✅
dd-fields 6h detailed ✅ identical n/a (beyond the Loki window) 0.11% ✅
dd-labels 15m detailed ✅ identical identical (3 explained, 1 history-dependent) 0.11% ✅
dd-labels 1h detailed ✅ identical identical (3 explained, 1 history-dependent) 0.11% ✅
dd-labels 6h detailed ✅ identical n/a (beyond the Loki window) 0.11% ✅
dd-labels-namespace 15m detailed ✅ identical identical (3 explained, 1 history-dependent) 0.11% ✅
dd-labels-namespace 1h detailed ✅ identical identical (3 explained, 1 history-dependent) 0.11% ✅
dd-labels-namespace 6h detailed ✅ identical n/a (beyond the Loki window) 0.11% ✅
dd-landing 15m detailed ✅ identical identical (3 explained) 2.15% ✅
dd-landing 6h detailed ✅ identical n/a (beyond the Loki window) 0.14% ✅
dd-patterns 15m detailed ✅ identical identical (3 explained, 1 history-dependent) 0.11% ✅
dd-patterns 1h detailed ✅ identical identical (3 explained, 1 history-dependent) 0.11% ✅
dd-patterns 6h detailed ✅ identical n/a (beyond the Loki window) 0.11% ✅
dd-service-logs 15m detailed ✅ identical identical (3 explained, 1 history-dependent) 0.11% ✅
dd-service-logs 6h detailed ✅ identical n/a (beyond the Loki window) 0.11% ✅
explore-A 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-A 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-A 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-B 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-B 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-B 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-C 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-C 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-C 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-D 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-D 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-D 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-DLV5 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-DLV5 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-DLV5 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-E 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-E 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-E 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-F 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-F 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-F 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-empty-pattern-filter 15m detailed ✅ identical 1 difference(s), 1 on base too 0.07% ✅
explore-empty-pattern-filter 1h detailed ✅ identical 1 difference(s), 1 on base too 0.07% ✅
explore-empty-pattern-filter 6h detailed ✅ identical n/a (beyond the Loki window) 0.07% ✅
explore-extracted-filter-metric 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-extracted-filter-metric 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-extracted-filter-metric 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-extracted-group-metric 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-extracted-group-metric 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-extracted-group-metric 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-extracted-metric 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-extracted-metric 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-extracted-metric 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-json-error-filter-metric 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-json-error-filter-metric 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-json-error-filter-metric 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-label-browser 15m detailed ✅ identical identical 0.02% ✅
explore-label-browser 1h detailed ✅ identical identical 0.02% ✅
explore-label-browser 6h detailed ✅ identical n/a (beyond the Loki window) 0.02% ✅
explore-label-replace-over-binary 15m detailed ✅ identical 2 difference(s), 2 on base too 0.04% ✅
explore-label-replace-over-binary 1h detailed ✅ identical 2 difference(s), 2 on base too 0.04% ✅
explore-label-replace-over-binary 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-log-context 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-log-context 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-log-context 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-logfmt-metric 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-logfmt-metric 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-logfmt-metric 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-logs-extracted 15m detailed ✅ identical 1 difference(s), 1 on base too 0.07% ✅
explore-logs-extracted 1h detailed ✅ identical 1 difference(s), 1 on base too 0.07% ✅
explore-logs-extracted 6h detailed ✅ identical n/a (beyond the Loki window) 0.07% ✅
explore-logs-json 15m detailed ✅ identical 1 difference(s), 1 on base too 0.07% ✅
explore-logs-json 1h detailed ✅ identical 1 difference(s), 1 on base too 0.07% ✅
explore-logs-json 6h detailed ✅ identical n/a (beyond the Loki window) 0.07% ✅
explore-logs-json-extraction 15m detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.07% ✅
explore-logs-json-extraction 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.07% ✅
explore-logs-json-extraction 6h detailed ✅ identical n/a (beyond the Loki window) 0.07% ✅
explore-logs-line-filter-or 15m detailed ✅ identical 1 difference(s), 1 on base too 0.07% ✅
explore-logs-line-filter-or 1h detailed ✅ identical 1 difference(s), 1 on base too 0.07% ✅
explore-logs-line-filter-or 6h detailed ✅ identical n/a (beyond the Loki window) 0.07% ✅
explore-logs-line-format 15m detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.07% ✅
explore-logs-line-format 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.07% ✅
explore-logs-line-format 6h detailed ✅ identical n/a (beyond the Loki window) 0.07% ✅
explore-logs-multivalue-regexp 15m detailed ✅ identical identical (1 explained) 0.07% ✅
explore-logs-multivalue-regexp 1h detailed ✅ identical identical (1 explained) 0.07% ✅
explore-logs-multivalue-regexp 6h detailed ✅ identical n/a (beyond the Loki window) 0.07% ✅
explore-logs-plain 15m detailed ✅ identical identical (1 explained) 0.07% ✅
explore-logs-plain 6h detailed ✅ identical n/a (beyond the Loki window) 0.07% ✅
explore-logs-plain-level 15m detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.07% ✅
explore-logs-plain-level 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.07% ✅
explore-logs-plain-level 6h detailed ✅ identical n/a (beyond the Loki window) 0.07% ✅
explore-max-by-count-over-time-sliding 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-max-by-count-over-time-sliding 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-max-by-count-over-time-sliding 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-metric-line-filter-or 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-metric-line-filter-or 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-metric-line-filter-or 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-metric-parenthesized 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-metric-parenthesized 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-metric-parenthesized 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-metric-plain 15m detailed ✅ identical identical (1 explained) 0.04% ✅
explore-metric-plain 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-nested-aggregation-outer-by 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-nested-aggregation-outer-by 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-nested-aggregation-outer-by 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-pattern-filter-anchored 15m detailed ✅ identical 1 difference(s), 1 on base too 0.07% ✅
explore-pattern-filter-anchored 1h detailed ✅ identical 1 difference(s), 1 on base too 0.07% ✅
explore-pattern-filter-anchored 6h detailed ✅ identical n/a (beyond the Loki window) 0.07% ✅
explore-plain-level-metric 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-plain-level-metric 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-plain-level-metric 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-sliding-sum-without-stream-fold 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-sliding-sum-without-stream-fold 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-sliding-sum-without-stream-fold 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-sum-without-count 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-sum-without-count 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-sum-without-count 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-sum-without-rate 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-sum-without-rate 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-sum-without-rate 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-topk-by-per-step 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-topk-by-per-step 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-topk-by-per-step 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-topk-sum-without 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-topk-sum-without 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-topk-sum-without 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-topk-without-keeps-labels 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-topk-without-keeps-labels 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-topk-without-keeps-labels 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-unwrap-conversion-error 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-conversion-error 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-conversion-error 6h detailed ✅ identical n/a (beyond the Loki window) 1.36% ✅
explore-unwrap-duration-seconds 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-duration-seconds 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-duration-seconds 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-unwrap-first-last-under-sum 15m detailed ✅ identical 2 difference(s), 2 on base too 0.04% ✅
explore-unwrap-first-last-under-sum 1h detailed ✅ identical 2 difference(s), 2 on base too 0.04% ✅
explore-unwrap-first-last-under-sum 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-unwrap-max-over-time-end-sample 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-max-over-time-end-sample 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-max-over-time-end-sample 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-unwrap-missing-label 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-missing-label 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-missing-label 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-unwrap-quantile-over-time 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-quantile-over-time 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-quantile-over-time 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-unwrap-rate-counter-one-sample 15m detailed differs (1): query ('sum(rate_counter({service_name="payment-service"} | 1 difference(s), 1 on base too 0.37% ✅ improved (closer to Loki)
explore-unwrap-rate-counter-one-sample 1h detailed differs (1): query ('sum(rate_counter({service_name="payment-service"} | 1 difference(s), 1 on base too 0.77% ✅ improved (closer to Loki)
explore-unwrap-rate-counter-one-sample 6h detailed differs (1): query ('sum(rate_counter({service_name="payment-service"} | n/a (beyond the Loki window) 2.17% ✅ improved (closer to Loki)
explore-unwrap-rate-sum 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-rate-sum 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-rate-sum 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-unwrap-sum 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-sum 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-sum 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-unwrap-sum-over-time-sliding-window 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-sum-over-time-sliding-window 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-sum-over-time-sliding-window 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-unwrap-unit-string 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-unit-string 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-unit-string 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-volume-label-filter-regexp 15m detailed ✅ identical identical (1 explained) 0.04% ✅
explore-volume-label-filter-regexp 1h detailed ✅ identical identical (1 explained) 0.04% ✅
explore-volume-label-filter-regexp 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-volume-multivalue-regexp 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-volume-multivalue-regexp 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-volume-multivalue-regexp 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-without-nested-under-aggregation 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-without-nested-under-aggregation 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-without-nested-under-aggregation 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-without-over-inner-aggregation 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-without-over-inner-aggregation 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-without-over-inner-aggregation 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
lvl-dd-field-method-pill 1h detailed ✅ identical 1 difference(s), 1 on base too; 2 explained 0.11% ✅
lvl-dd-field-pipeline-pill 1h detailed ✅ identical 1 difference(s), 1 on base too; 2 explained, 1 history-dependent 0.11% ✅
lvl-dd-fields-pill 1h detailed ✅ identical 25 difference(s), 25 on base too; 4 explained 0.11% ✅
lvl-dd-label-pod-pill 1h detailed ✅ identical 1 difference(s), 1 on base too; 2 explained, 1 history-dependent 0.11% ✅
lvl-dd-labels-pill 1h detailed ✅ identical 2 difference(s), 2 on base too; 2 explained 0.11% ✅
lvl-dd-logs-pill-json 1h detailed ✅ identical 1 difference(s), 1 on base too; 2 explained 0.11% ✅
lvl-dd-logs-pill-logfmt 1h detailed ✅ identical identical (4 explained, 1 history-dependent) 0.11% ✅
lvl-dd-logs-pill-multi-json 1h detailed ✅ identical 2 difference(s), 2 on base too; 2 explained, 1 history-dependent 0.11% ✅
lvl-dd-logs-pill-otel 1h detailed ✅ identical 1 difference(s), 1 on base too; 2 explained, 1 history-dependent 0.11% ✅
lvl-dd-logs-pill-plaintext-bodyonly 1h detailed ✅ identical 2 difference(s), 2 on base too; 2 explained, 1 history-dependent 0.11% ✅
lvl-explore-count-warn-by-service 1h detailed ✅ identical 2 difference(s), 2 on base too 0.04% ✅
lvl-explore-logs-dl-eq 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-logs-dl-error-plaintext-bodyonly 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
lvl-explore-logs-dl-neq-all 1h detailed ✅ identical 2 difference(s), 2 on base too; 1 explained 0.07% ✅
lvl-explore-logs-dl-plaintext 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.07% ✅
lvl-explore-logs-dl-regexp 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-logs-dl-unknown-bodyonly 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
lvl-explore-logs-dl-unknown-or-error 1h detailed ✅ identical 2 difference(s), 2 on base too; 1 explained 0.07% ✅
lvl-explore-logs-drop-dl 1h detailed ✅ identical 2 difference(s), 2 on base too; 1 explained 0.07% ✅
lvl-explore-logs-drop-level 1h detailed ✅ identical 2 difference(s), 2 on base too 0.07% ✅
lvl-explore-logs-json-dl 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-logs-json-level 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-logs-json-level-extracted-stored 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-logs-keep-dl 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.07% ✅
lvl-explore-logs-level-label-stored 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-logs-level-selector-stored 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-logs-logfmt-level-extracted-stored 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-logs-logfmt-level-regexp 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-rate-errors-by-service 1h detailed ✅ identical 2 difference(s), 2 on base too 0.04% ✅
lvl-explore-topk-errors-by-service 1h detailed ✅ identical 2 difference(s), 2 on base too 0.04% ✅
lvl-explore-volume-dl-neq 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.04% ✅
lvl-explore-volume-dl-prefilter-json 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-dl-regexp 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.04% ✅
lvl-explore-volume-dl-unknown-bodyonly 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
lvl-explore-volume-dl-unknown-or-error 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.04% ✅
lvl-explore-volume-drop-level 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
lvl-explore-volume-json-by-dl 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-json-dl-filter 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-json-error-by-dl 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-json-level-filter 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
lvl-explore-volume-keep-dl 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
lvl-explore-volume-level-extracted-stored 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
lvl-explore-volume-level-label-stored 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
lvl-explore-volume-logfmt-dl 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
lvl-explore-volume-logfmt-dl-filter 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-plaintext 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
tail-env live detailed ✅ identical entries in the window: PR 10104, Loki 7008, in both 7008 6.87% ✅
tail-json-filter live detailed ✅ identical entries in the window: PR 1614, Loki 446, in both 446 7.15% ✅

Gate: a base-vs-PR data difference passes when the PR is closer to Loki, and fails when the PR diverges from Loki or Loki cannot decide it (at a range Loki does not hold, a difference passes when the same page proved that query to answer like Loki at a shorter range; otherwise label the pull request visual-change-expected to accept an intended change). Also failing: a panel empty on the PR but not on the base, an error banner, panel error, error answer or new "No data" panel on the PR, a PR side that never settled, a difference that did not reproduce on the recapture. Pixel differences only warn.

Montages (base | PR | Loki): artifact. They are not on the repository for this run.

🧭 New baseline candidate

This pull request got the baseline run: every page of bench/visual/spec.json was captured at its own ranges. The captures are the visual-baseline-candidate artifact of https://github.com/ReliablyObserve/loki-vl-proxy/actions/runs/40848024585; nothing was committed or published as a baseline. Accept them by copying what you want to keep into a pull request.

@szibis
szibis force-pushed the perf/unwrap-under-loki branch from 0a33660 to 0920d0d Compare October 29, 2026 07:46
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 29, 2026
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 29, 2026
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 29, 2026
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 29, 2026
@szibis

szibis commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator Author

Verification on 6a4a828c14da6fe4fa9dbdeb19cf7febf99dbb8d

The PR is conflict-free (MERGEABLE). CI has no failed or pending checks on this head. Merge still requires a reviewer approval.

Proof Result
Unit tests Passed
Conformance gate Passed; regenerated performance evidence and gap reports are current
VictoriaLogs v1.49.0–v1.53.0 matrix All five versions passed, including the previously failing v1.52.0 recorder case
Visual smoke Passed; 44-minute full baseline capture, montage and comparison artifact
Race-stress gate Passed; all four shards passed. Shard 3 completed in 11m 2s with its 12m timeout
Performance quality report Passed

Local checks also passed: python3 -m unittest discover -s bench/visual/tests (148 tests), python3 -m unittest discover -s bench/ab/tests (362 tests), python3 scripts/ci/check_conformance.py, actionlint .github/workflows/ci.yaml, and the exact 674-test race shard 3 under -race (313s). The earlier hosted throughput failure was not reproducible: two unchanged-code reruns passed at +13.8% and -3.8% versus base.

@szibis
szibis merged commit 302ad15 into main Oct 29, 2026
77 checks passed
@szibis
szibis deleted the perf/unwrap-under-loki branch October 29, 2026 10:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance Performance scope/cache Cache layer scope/ci CI/CD scope/cli CLI flags scope/docs Documentation scope/proxy Proxy core scope/tests Tests scope/translator LogQL translator size/XL Extra large change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants