Skip to content

fix: reject group(), label_join() and other names Loki does not have, and accept keywords in any case, like Loki - #748

Merged
szibis merged 12 commits into
mainfrom
fix/group-aggregation-not-in-loki
Oct 29, 2026
Merged

fix: reject group(), label_join() and other names Loki does not have, and accept keywords in any case, like Loki#748
szibis merged 12 commits into
mainfrom
fix/group-aggregation-not-in-loki

Conversation

@szibis

@szibis szibis commented Oct 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Result

Loki rejects group(), label_join(), and other unknown LogQL function names with a 400 parse error. The proxy previously accepted some of them. Conversely, Loki accepts keywords in any case, while the proxy rejected or misrouted uppercase forms such as SUM BY, RATE, and | JSON. This PR returns Loki's error text and source column for unknown names, normalizes recognized keywords across GET, form POST, and multi-tenant requests, and removes the now-unreachable group() and label_join() translation and response rewriting. Selector labels such as Rate remain labels.

The branch is rebased onto main at bb8f348beeecc1a2e076210c2dc3e98e774d5e36 (including #737's visual publisher). The conformance registry marks semantics/proxy-accepts-operators-not-in-loki fixed. This PR also repairs quality-report failure evidence. Visual CI now runs nine general regression scenes on every PR, including docs and CI changes, and adds exact fix scenes for behavior changes. The baseline capture gate stays intact; one existing Compose stack serves main, PR, and Loki.

Verification

  • Parser and proxy regressions check unsupported names, error columns, uppercase keywords, no backend call on rejection, and multi-tenant forwarding.
  • Real Loki differential E2E passed locally on the rebased code: TestCompat_OperatorsNotInLokiRejected and TestCompat_UppercaseKeywordsMatchLoki. Direct API proof covered 20 GET/form POST cases across /query and /query_range: 8 exact Loki 400 errors and 12 matching successful uppercase responses. The TOPK comparison follows the project's ranked-points contract because 20 repeated identical Loki requests produced six tied label sets but one ranked-value result; other cases compare full series and values.
  • Fresh local Grafana capture against bb8f348b main and the final product Go tree: 17/17 scenes passed in 3.6 minutes. All nine core scenes returned the same data on main and PR; all eight fix: reject group(), label_join() and other names Loki does not have, and accept keywords in any case, like Loki #748 scenes showed a difference on main and PR answers identical to Loki. The local visual gate passed. The final signed head is fa7924bffcbbca3c9998a51a7583e852093df731; current-head visual CI also passed all capture, publisher, and comment jobs over 245 baseline captures.
  • Local validation passed: full go test -count=1 ./internal/logql ./internal/translator ./internal/proxy, 163 visual tests, 311 CI configuration tests, conformance wiring and parity checks, actionlint, changelog gate, E2E package compile, and git diff --check. Current-head CI has 72 successful checks, zero failures, one neutral, and four expected skips. Quality reports 19,044 PR tests and 89.2% coverage versus 18,874 and 89.2% on main; performance A/B has 264/264 comparable shape/range pairs within noise and zero new Loki answer differences.

Visual proof coverage

Grafana scene Exact LogQL query Before → PR / Loki
explore-748-reject-group-1h group(count_over_time({app="api-gateway"}[$__auto])) Main accepted; PR and Loki show the same parse error.
explore-748-reject-label-join-1h label_join(rate({app="api-gateway"}[$__auto]), "d", "-", "app", "level") Main accepted; PR and Loki show the same parse error.
explore-748-uppercase-sum-1h SUM BY (app) (COUNT_OVER_TIME({env="production"}[$__auto])) Main 400; PR and Loki show the same graph.
explore-748-uppercase-topk-rate-1h TOPK(3, SUM BY (app) (RATE({env="production"}[$__auto]))) Main 400; PR and Loki show the same ranked graph.
explore-748-uppercase-json-unwrap-1h SUM BY (app) (SUM_OVER_TIME({env="production"} | JSON | UNWRAP duration_ms [$__auto])) Main 400; PR and Loki show the same unwrap graph.
explore-748-uppercase-on-1h SUM BY (app) (COUNT_OVER_TIME({env="production"}[$__auto])) / ON (app) SUM BY (app) (COUNT_OVER_TIME({env="production"}[$__auto])) Main 400; PR and Loki show the same joined graph.
explore-748-uppercase-bool-1h SUM BY (app) (COUNT_OVER_TIME({env="production"}[$__auto])) > BOOL 0 Main 400; PR and Loki show the same Boolean graph.
explore-748-uppercase-offset-1h SUM BY (app) (COUNT_OVER_TIME({env="production"}[$__auto] OFFSET 1m)) Main 400; PR and Loki show the same shifted graph.

Current-head CI screenshots (main | PR | Loki)

Captured automatically in visual run 40910139389, base bb8f348b, head fa7924bf.

Reject group(): main accepted, PR and Loki reject

Reject group(): main accepted, PR and Loki reject

Reject label_join(): main accepted, PR and Loki reject

Reject label_join(): main accepted, PR and Loki reject

Uppercase SUM BY / COUNT_OVER_TIME: main 400, PR and Loki graph

Uppercase SUM BY / COUNT_OVER_TIME: main 400, PR and Loki graph

Uppercase TOPK / SUM / RATE: main 400, PR and Loki ranked graph

Uppercase TOPK / SUM / RATE: main 400, PR and Loki ranked graph

Uppercase JSON / UNWRAP: main 400, PR and Loki unwrap graph

Uppercase JSON / UNWRAP: main 400, PR and Loki unwrap graph

Uppercase ON: main 400, PR and Loki joined graph

Uppercase ON: main 400, PR and Loki joined graph

Uppercase BOOL: main 400, PR and Loki Boolean graph

Uppercase BOOL: main 400, PR and Loki Boolean graph

Uppercase OFFSET: main 400, PR and Loki shifted graph

Uppercase OFFSET: main 400, PR and Loki shifted graph

The automatic ✅ Visual smoke comment displays all eight exact fix montages and nine general core montages inline and open. It records base bb8f348b and head fa7924bf; all 17 raw image URLs returned HTTP 200 image/png. Transport, tenant, and quality-report collector behavior are API/CI properties proven by tests and report artifacts.

szibis added a commit that referenced this pull request Oct 12, 2026
@szibis
chore(conformance): record #748 as the fix of proxy-accepts-operators…
ccd6969 
…-not-in-loki
@github-actions github-actions Bot added size/XL Extra large change scope/translator LogQL translator scope/proxy Proxy core scope/docs Documentation scope/tests Tests bugfix Bug fix and removed size/XL Extra large change labels Oct 12, 2026
@github-actions

github-actions Bot commented Oct 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

⚪ Performance A/B: no change beyond noise

264 shape×range — 264 ⚪ same · results vs Loki: 242 same, 0 new difference(s), 22 pre-existing

Loki profiles compared (like for like):

  • aligned: not run (no build supports -align-queries-with-step, so every aligned row would be not like-for-like)
  • loki-defaults: 264 shape×range; proxy flag base off, pr off

Every shape is within noise of the base build (±30% and 100 ms).

29951 characters omitted to fit the comment limit; every row is in the `summary-.json` files of the `perf-ab-smoke` artifact
Cold (first-run) timings — what a user opening a panel sees
shape range Loki profile base cold PR cold Loki cold PR ÷ Loki cold
B1 sum by over rate divided by a literal 1h defaults 34 ms 25 ms 434 ms 0.06×
B2 sum by over a ratio of two per-stream metrics 1h defaults 598 ms 586 ms 12.57 s 0.05×
B3 sum over a literal times count, instant instant defaults 36 ms 36 ms 30 ms 1.20×
B4 outer sum over ignoring(machine) 1h defaults 56 ms 49 ms 700 ms 0.07×
by service_name 1h defaults 31 ms 32 ms 801 ms 0.04×
bytes_over_time by app 1h defaults 61 ms 70 ms 1.14 s 0.06×
detected_level volume 1h defaults 117 ms 117 ms 1.03 s 0.11×
drilldown detected_level 1h defaults 169 ms 98 ms 767 ms 0.13×
drilldown level volume 1h defaults 15 ms 14 ms 764 ms 0.02×
drilldown service landing 1h defaults 32 ms 34 ms 784 ms 0.04×
instant json filtered instant defaults 10 ms 9 ms 101 ms 0.09×
instant json volume instant defaults 33 ms 34 ms 123 ms 0.28×
instant ungrouped rate instant defaults 7 ms 6 ms 99 ms 0.06×
json rate by level (no filter, no drop) 1h defaults 400 400 400 —
json volume no filters 1h defaults 59 ms 60 ms 671 ms ↺ 0.09×
logfmt count by level, filter, drop 1h defaults 457 ms 447 ms 1.53 s 0.29×
logfmt volume 1h defaults 233 ms 239 ms 1.46 s 0.16×
logs plain 1h defaults 80 ms 81 ms 94 ms 0.86×
logs | json 1h defaults 114 ms 110 ms 83 ms 1.33×
logs | logfmt 1h defaults 169 ms 160 ms 171 ms 0.94×
plain volume 1h defaults 16 ms 16 ms 989 ms 0.02×
stream-label count by app 1h defaults 20 ms 21 ms 799 ms 0.03×
topk rate by app 1h defaults 18 ms 14 ms 849 ms 0.02×
ungrouped rate 1h defaults 12 ms 11 ms 836 ms 0.01×
DB1 label breakdown pod 1h defaults 130 ms 135 ms 1.60 s 0.08×
DB2 label breakdown namespace 1h defaults 26 ms 27 ms 724 ms 0.04×
DB3 json field breakdown user_id 1h defaults 115 ms 98 ms 733 ms 0.13×
DB4 json field breakdown batch_id 1h defaults 115 ms 115 ms 669 ms 0.17×
X1 json filter level_extracted, by level 1h defaults 3.69 s 3.67 s 1.18 s 3.10×
X2 json group by level_extracted 1h defaults 333 ms 335 ms 896 ms 0.37×
X3 json rate, level_extracted equality 1h defaults 297 ms 306 ms 702 ms 0.44×
X4 json bytes, level_extracted regexp 1h defaults 581 ms 464 ms 892 ms 0.52×
X5 logfmt filter level_extracted, by level 1h defaults 669 ms 696 ms 1.47 s 0.47×
E1 logs, json parser over stream labels 1h defaults 155 ms 154 ms 89 ms † —
E2 logs, logfmt parser over stream labels 1h defaults 84 ms 123 ms 76 ms † —
E3 logs, json, existence filter on level_extracted 1h defaults 174 ms 161 ms 91 ms † —
E4 logs, logfmt, existence filter on level_extracted 1h defaults 103 ms 103 ms 79 ms † —
E5 logs, json, drop level_extracted 1h defaults 146 ms 154 ms 83 ms † —
E6 logfmt count by level_extracted 1h defaults 49 ms 49 ms 51 ms 0.96×
E7 json count by level_extracted 1h defaults 1.49 s 1.60 s 200 ms 7.99×
E8 instant logfmt by level_extracted instant defaults 17 ms 17 ms 10 ms 1.70×
DLV1 drilldown logs volume, json 1h defaults 1.19 s 1.21 s 217 ms 5.57×
DLV2 json error filter, detected_level 1h defaults 5.10 s 5.08 s 989 ms 5.14×
DLV3 plain json, detected_level 1h defaults 1.27 s 1.27 s 217 ms 5.84×
DLV4 explore json volume, level and detected_level 1h defaults 94 ms 95 ms 202 ms 0.47×
DLV5 json error dropped, detected_level 1h defaults 22 ms 22 ms 1.19 s 0.02×
DLV6 drilldown detected_level breakdown, json 1h defaults 2 ms 2 ms 879 ms 0.00×
J1 json no-error count by service_name, one service 1h defaults 197 ms 193 ms 157 ms 1.23×
J2 json no-error count by app, namespace 1h defaults 570 ms 463 ms 529 ms 0.88×
J3 json no-error count by parsed method 1h defaults 167 ms 169 ms 154 ms 1.10×
J4 json no-error count by service_name, all services 1h defaults 538 ms 518 ms 678 ms 0.76×
J5 json no-error bytes_rate by app 1h defaults 327 ms 307 ms 487 ms 0.63×
A explore volume, 2 filters, drop 1h defaults 1.67 s 1.61 s 511 ms 3.14×
B grouped sum, filter, no drop 1h defaults 125 ms 128 ms 401 ms 0.32×
C ungrouped sum, filter, no drop 1h defaults 122 ms 123 ms 412 ms 0.30×
D volume, no filters 1h defaults 275 ms 222 ms 671 ms 0.33×
E field breakdown pipeline (explore) 1h defaults 109 ms 108 ms 545 ms 0.20×
F field breakdown service_version (explore) 1h defaults 1.65 s 2.05 s 1.81 s 1.13×
G field breakdown pipeline (drilldown) 1h defaults 46 ms 46 ms 1.03 s 0.04×
H field breakdown service_version (drilldown) 1h defaults 58 ms 59 ms 1.12 s 0.05×
I grafana volume of B 1h defaults 117 ms 112 ms 1.03 s 0.11×
M1 multi-value variable, log query 1h defaults 107 ms 94 ms 30 ms 3.13×
M2 multi-value variable, count by app 1h defaults 18 ms 17 ms 186 ms 0.09×
M3 single-value variable, log query 1h defaults 59 ms 58 ms 23 ms 2.52×
M4 alternation over many streams, count 1h defaults 18 ms 17 ms 552 ms 0.03×
M5 json label filter, literal alternation 1h defaults 127 ms 124 ms 200 ms 0.62×
M6 json label filter, prefix (substring in Loki) 1h defaults 56 ms 58 ms 30 ms † —
L1 label_replace over sum by rate, instant instant defaults 31 ms 19 ms 97 ms 0.20×
L2 label_replace over sum by rate 1h defaults 146 ms 150 ms 684 ms 0.22×
L3 topk over label_replace over sum by count 1h defaults 10 ms 10 ms 653 ms 0.02×
L4 label_replace over bare count_over_time, instant instant defaults 23 ms 23 ms 28 ms 0.82×
L5 label_replace over json sum by, instant instant defaults 55 ms 56 ms 30 ms 1.87×
L6 label_replace over json sum by 1h defaults 141 ms 144 ms 193 ms 0.75×
L7 label_replace around a binary of two sums by level 1h defaults 39 ms 43 ms 64 ms † —
L8 label_replace over sum by level count, instant instant defaults 15 ms 13 ms 12 ms 1.08×
LF1 logs, detected_level equals 1h defaults 201 ms 202 ms 82 ms 2.46×
LF10 drilldown logs, level pill multi-select, json service 1h defaults 161 ms 159 ms 45 ms 3.53×
LF11 drilldown logs volume by level, existing level 1h defaults 594 ms 575 ms 737 ms 0.78×
LF12 drilldown logs volume by level, level pill 1h defaults 166 ms 164 ms 115 ms 1.43×
LF13 explore volume, level filter before json 1h defaults 104 ms 111 ms 544 ms 0.20×
LF14 explore volume, plain-text service 1h defaults 61 ms 63 ms 24 ms 2.62×
LF15 drilldown field breakdown, level filter 1h defaults 111 ms 100 ms 112 ms 0.89×
LF16 drilldown label breakdown, level filter 1h defaults 56 ms 59 ms 54 ms 1.09×
LF17 detected_level count, level regexp 1h defaults 502 ms 503 ms 587 ms 0.86×
LF18 detected_level count, level not equal 1h defaults 506 ms 501 ms 587 ms 0.85×
LF19 detected_level count, regexp with unknown 1h defaults 775 ms 504 ms 556 ms 0.91×
LF2 logs, detected_level regexp 1h defaults 196 ms 203 ms 31 ms 6.55×
LF20 logfmt count by detected_level 1h defaults 65 ms 64 ms 64 ms 1.00×
LF21 logfmt count by detected_level, level filter 1h defaults 40 ms 49 ms 51 ms 0.96×
LF22 json count by detected_level, level filter 1h defaults 22 ms 25 ms 169 ms 0.15×
LF23 json count by level, level filter 1h defaults 64 ms 63 ms 166 ms 0.38×
LF24 rate of errors by service 1h defaults 452 ms 435 ms 593 ms 0.73×
LF25 count of warnings, error label dropped 1h defaults 428 ms 428 ms 586 ms 0.73×
LF26 topk services by errors 1h defaults 6 ms 6 ms 602 ms 0.01×
LF27 detected_level count, instant instant defaults 46 ms 51 ms 91 ms 0.56×
LF28 logs, keep detected_level 1h defaults 72 ms 80 ms 20 ms † —
LF29 logs, drop detected_level 1h defaults 103 ms 94 ms 19 ms 4.95×
LF3 logs, detected_level not equal, all services 1h defaults 319 ms 307 ms 76 ms 4.04×
LF30 logs, drop level 1h defaults 100 ms 102 ms 19 ms 5.37×
LF31 count by detected_level, level dropped 1h defaults 69 ms 55 ms 281 ms 0.20×
LF32 count by detected_level, detected_level kept 1h defaults 30 ms 33 ms 162 ms 0.20×
LF33 drilldown logs count, level pill, instant instant defaults 17 ms 19 ms 19 ms 1.00×
LF4 logs, detected_level regexp with unknown 1h defaults 434 ms 420 ms 131 ms 3.21×
LF5 logs, plain-text service, detected_level regexp 1h defaults 91 ms 91 ms 15 ms 6.07×
LF6 logs, json then detected_level 1h defaults 166 ms 152 ms 187 ms † —
LF7 logs, json then level equals 1h defaults 158 ms 152 ms 189 ms † —
LF8 logs, logfmt then level regexp 1h defaults 78 ms 78 ms 75 ms † —
LF9 drilldown logs, level pill, logfmt service 1h defaults 74 ms 72 ms 30 ms 2.40×
LS1 logs, level stream label 1h defaults 62 ms 75 ms 83 ms 0.90×
LS2 logs, level stream label regexp, all services 1h defaults 95 ms 96 ms 88 ms 1.09×
LS3 logs, level in the selector 1h defaults 47 ms 47 ms 27 ms 1.74×
LS4 logs, json then level_extracted 1h defaults 193 ms 189 ms 183 ms † —
LS5 logs, logfmt then level_extracted regexp 1h defaults 80 ms 80 ms 68 ms † —
LS6 level stream label count 1h defaults 17 ms 17 ms 546 ms 0.03×
LS7 rate of level stream label by service 1h defaults 22 ms 23 ms 593 ms 0.04×
LS8 json count by level_extracted, collision 1h defaults 1.49 s 1.57 s 178 ms 8.80×
O1 explore logs, underscore structured-metadata filter 1h defaults 403 ms 347 ms 491 ms † —
O2 explore log volume (Grafana form), underscore structured-metadata filter 1h defaults 166 ms 162 ms 506 ms 0.32×
O3 explore logs, dotted structured-metadata filter 1h defaults 400 400 400 —
O4 explore log volume (Grafana form), dotted structured-metadata filter 1h defaults 400 400 400 —
label values env ending now 1h defaults 262 ms 277 ms 46 ms † —
label values env ending two days ago (control) 1h defaults 5 ms 5 ms 2 ms † —
labels ending now 1h defaults 147 ms 167 ms 5 ms † —
labels ending two days ago (control) 1h defaults 3 ms 3 ms 2 ms † —
drilldown detected fields 1h defaults 31 ms 48 ms 30 ms † —
drilldown detected labels 1h defaults 29 ms 32 ms 5 ms † —
drilldown landing volume 1h defaults 62 ms 62 ms 97 ms 0.64×
explore label values env 1h defaults 116 ms 128 ms 43 ms † —
explore label values service_name 1h defaults 58 ms 52 ms 65 ms † —
explore labels (label browser) 1h defaults 74 ms 79 ms 3 ms † —
explore labels with selector 1h defaults 96 ms 75 ms 64 ms † —
refresh label values env 1h defaults 5 ms 5 ms 45 ms † —
refresh label values service_name 1h defaults 3 ms 3 ms 62 ms † —
refresh labels (label browser) 1h defaults 3 ms 3 ms 2 ms † —
refresh labels with selector 1h defaults 3 ms 3 ms 64 ms † —
DL1 drilldown field breakdown json-expr, level filter 1h defaults 519 ms 504 ms 654 ms 0.77×
DL2 drilldown field breakdown, level + stream-label filter before json 1h defaults 56 ms 70 ms 41 ms 1.71×
DL3 drilldown logfmt field breakdown, level filter 1h defaults 646 ms 683 ms 1.57 s 0.44×
DL4 instant count, level filter, json+logfmt, 1h instant defaults 490 ms 492 ms 1.59 s 0.31×
DL6 explore volume, level filter before json 1h defaults 274 ms 307 ms 925 ms 0.33×
N1 sum by app of max by app,pod, count 1h defaults 59 ms 52 ms 630 ms 0.08×
N10 instant min of avg, rate instant defaults 20 ms 16 ms 114 ms 0.14×
N11 instant three levels instant defaults 15 ms 15 ms 78 ms 0.19×
N12 instant json inner, count of sum instant defaults 27 ms 22 ms 91 ms † —
N13 instant without under by instant defaults 24 ms 24 ms 103 ms 0.23×
N14 sliding sum of max 1h defaults 466 ms 441 ms 1.57 s 0.28×
N15 sliding sums only 1h defaults 29 ms 30 ms 457 ms 0.07×
N16 innermost without under by, tumbling 1h defaults 58 ms 59 ms 1.32 s 0.04×
N2 max by namespace of sum by namespace,app, rate 1h defaults 27 ms 26 ms 417 ms 0.06×
N3 count by namespace of sum by namespace,app 1h defaults 26 ms 24 ms 440 ms † —
N4 avg by app of min by app,pod 1h defaults 53 ms 57 ms 605 ms 0.09×
N5 three levels, max min sum 1h defaults 28 ms 25 ms 699 ms 0.04×
N6 json inner, sum of max 1h defaults 156 ms 169 ms 400 ms 0.42×
N7 logfmt inner, max of sum 1h defaults 30 ms 28 ms 68 ms 0.41×
N8 without over a named by level, tumbling 1h defaults 54 ms 57 ms 945 ms 0.06×
N9 instant sum of max instant defaults 18 ms 17 ms 96 ms 0.18×
OR1 logs, or alternatives 1h defaults 150 ms 137 ms 88 ms 1.56×
OR2 logs, negated or alternatives 1h defaults 178 ms 178 ms 71 ms 2.51×
OR3 logs, regexp or alternatives 1h defaults 80 ms 82 ms 97 ms 0.85×
OR4 count by level, or alternatives with parser 1h defaults 139 ms 135 ms 532 ms 0.25×
OR5 rate, negated regexp or alternatives 1h defaults 62 ms 63 ms 935 ms 0.07×
OR6 instant, or alternatives instant defaults 18 ms 20 ms 88 ms 0.23×
PR1 count by level, parenthesised json 1h defaults 225 ms 218 ms 1.05 s 0.21×
PR2 rate, parenthesised line filter 1h defaults 54 ms 53 ms 577 ms 0.09×
PR3 bytes by app, parenthesised regexp 1h defaults 61 ms 59 ms 548 ms 0.11×
PR4 count by level, parenthesised parser, offset 1h defaults 55 ms 59 ms 755 ms 0.08×
PR5 rate by app, pipeline after the range 1h defaults 312 ms 333 ms 1.01 s 0.33×
PR6 doubly parenthesised range 1h defaults 1 ms 1 ms 526 ms 0.00×
PR7 instant, parenthesised line filter instant defaults 15 ms 14 ms 85 ms 0.16×
P1 api-gateway empty pattern 1h defaults 42 ms 32 ms 73 ms † —
P2 api-gateway not empty pattern, line filter 1h defaults 107 ms 110 ms 21 ms 5.24×
P3 nginx count_over_time, pattern with spaces 1h defaults 20 ms 18 ms 21 ms † —
P4 nginx count_over_time, GET request pattern 1h defaults 20 ms 20 ms 26 ms 0.77×
A1 nginx include GET, log query 1h defaults 60 ms 44 ms 9 ms 4.89×
A2 nginx exclude GET, log query 1h defaults 111 ms 111 ms 10 ms † —
A3 nginx include GET, count_over_time 1h defaults 19 ms 18 ms 26 ms ↺ 0.69×
A4 nginx exclude GET, count_over_time 1h defaults 24 ms 24 ms 38 ms 0.63×
A5 drilldown exclude, volume by detected_level 1h defaults 59 ms 55 ms 33 ms 1.67×
A6 nginx include ends on a literal, log query 1h defaults 24 ms 23 ms 26 ms † —
A7 nginx exclude ends on a literal, log query 1h defaults 56 ms 59 ms 11 ms 5.36×
A8 nginx include two literals, count_over_time 1h defaults 28 ms 27 ms 32 ms 0.84×
A9 nginx include two literals, ends on a literal, count_over_time 1h defaults 26 ms 26 ms 28 ms † —
P1 logs, json, level equals 1h defaults 167 ms 173 ms 81 ms † —
P2 logs, logfmt, level not equal 1h defaults 41 ms 40 ms 30 ms † —
P3 logs, json, level existence 1h defaults 155 ms 147 ms 75 ms † —
P4 logfmt count by level 1h defaults 37 ms 36 ms 28 ms 1.29×
P5 json count by level 1h defaults 119 ms 102 ms 122 ms 0.84×
P6 json count, level filter 1h defaults 22 ms 22 ms 116 ms 0.19×
P7 json rate by level, level regexp 1h defaults 23 ms 23 ms 123 ms 0.19×
P8 json count by a name that is no stream label 1h defaults 105 ms 111 ms 103 ms 1.08×
P9 instant logfmt by level instant defaults 14 ms 16 ms 9 ms 1.78×
SW1 sum without pod, count_over_time 1h defaults 97 ms 92 ms 691 ms 0.13×
SW2 sum without pod, bytes_over_time 1h defaults 122 ms 116 ms 950 ms 0.12×
SW3 sum without pod and version, line filter 1h defaults 108 ms 110 ms 659 ms 0.17×
SW4 trailing sum without pod 1h defaults 17 ms 19 ms 646 ms 0.03×
SW5 sum without pod, rate (control) 1h defaults 271 ms 262 ms 749 ms 0.35×
SW6 max without pod (control) 1h defaults 187 ms 220 ms 788 ms 0.28×
SF1 logs, json extraction list 1h defaults 161 ms 129 ms 64 ms 2.02×
SF10 volume, structured-metadata filter 1h defaults 34 ms 34 ms 465 ms 0.07×
SF2 logs, logfmt extraction list 1h defaults 197 ms 200 ms 65 ms 3.08×
SF3 logs, json then line_format 1h defaults 162 ms 137 ms 83 ms 1.65×
SF4 logs, line_format without a parser 1h defaults 88 ms 119 ms 64 ms 1.86×
SF5 logs, regexp capture 1h defaults 223 ms 220 ms 73 ms 3.01×
SF6 logs, structured-metadata filter 1h defaults 99 ms 88 ms 165 ms † —
SF7 logs, filter on a line key without a parser 1h defaults 132 ms 129 ms 524 ms † —
SF8 volume, json extraction list 1h defaults 25 ms 25 ms 838 ms 0.03×
SF9 volume, logfmt label_format rename 1h defaults 525 ms 526 ms 1.26 s 0.42×
E1 json duration_ms sum by service 1h defaults 99 ms 95 ms 224 ms 0.42×
E2 json duration_ms p95 by service 1h defaults 176 ms 192 ms 273 ms 0.70×
E3 logfmt latency_ms rate, production 1h defaults 503 ms 495 ms 1.24 s 0.40×
E4 logfmt size_bytes max by service 1h defaults 54 ms 51 ms 266 ms 0.19×
E5 json duration_ms sum, instant instant defaults 33 ms 32 ms 29 ms 1.10×
E6 logfmt ttl unit string (Loki 400) 1h defaults 400 400 400 —
E7 json hexadecimal trace_id (Loki 400) 1h defaults 400 400 400 —
E8 json duration_ms sum by service, namespace prod 1h defaults 220 ms 215 ms 971 ms 0.22×
E9 logfmt latency_ms sum by service, namespace prod 1h defaults 317 ms 329 ms 865 ms 0.38×
D1 logfmt ttl duration_seconds rate by service 1h defaults 81 ms 70 ms 45 ms 1.56×
D2 logfmt ttl duration rate by service 1h defaults 2 ms 2 ms 45 ms 0.04×
D3 logfmt ttl duration_seconds rate, instant instant defaults 30 ms 27 ms 9 ms 3.00×
D4 logfmt ttl duration rate, instant instant defaults 28 ms 27 ms 9 ms 3.00×
G1 logfmt amount sum_over_time, no filter 1h defaults 56 ms 47 ms 65 ms 0.72×
G10 logfmt latency_ms sum_over_time, production 1h defaults 499 ms 494 ms 1.33 s 0.37×
G11 logfmt amount sum_over_time, instant instant defaults 14 ms 17 ms 13 ms 1.31×
G12 json duration_ms sum_over_time, instant instant defaults 39 ms 35 ms 30 ms 1.17×
G2 json duration_ms sum_over_time, no filter 1h defaults 137 ms 129 ms 257 ms 0.50×
G3 logfmt amount sum_over_time, error filter 1h defaults 60 ms 65 ms 64 ms 1.02×
G4 json duration_ms sum_over_time, error filter 1h defaults 135 ms 132 ms 431 ms 0.31×
G5 logfmt amount sum by rate 1h defaults 66 ms 72 ms 66 ms ↺ 1.09×
G6 json duration_ms sum by rate 1h defaults 139 ms 132 ms 251 ms 0.53×
G7 duration() of a unit string 1h defaults 70 ms 73 ms 45 ms 1.62×
G8 bytes() of a number 1h defaults 70 ms 69 ms 48 ms 1.44×
G9 json duration_ms max_over_time 1h defaults 103 ms 102 ms 1.85 s 0.06×
W1 sum without pod, max_over_time 1h defaults 82 ms 70 ms 103 ms 0.68×
W2 sum without pod, sum_over_time 1h defaults 68 ms 70 ms 94 ms 0.74×
W3 sum without pod, rate 1h defaults 79 ms 98 ms 83 ms 1.18×
W4 max without pod, max_over_time 1h defaults 27 ms 26 ms 108 ms 0.24×
W5 sum without pod, max_over_time, instant instant defaults 19 ms 18 ms 16 ms 1.12×
W6 sum without pod, rate, instant instant defaults 15 ms 18 ms 16 ms 1.12×
W7 topk 5 of sum without pod 1h defaults 27 ms 27 ms 102 ms 0.26×
W8 sum without pod, count_over_time (production) 1h defaults 83 ms 84 ms 562 ms 0.15×
W9 label_replace control, instant instant defaults 14 ms 15 ms 99 ms 0.15×
U1 logfmt amount rate 1h defaults 50 ms 42 ms 56 ms 0.75×
U2 json duration_ms rate 1h defaults 92 ms 92 ms 137 ms 0.67×
U3 json duration_ms rate, error filter 1h defaults 124 ms 122 ms 207 ms 0.59×
U4 logfmt latency_ms rate, production 1h defaults 503 ms 640 ms 1.55 s 0.41×
U5 logfmt amount rate, instant instant defaults 23 ms 17 ms 18 ms 0.94×
U6 json duration_ms rate, instant instant defaults 32 ms 49 ms 44 ms 1.11×
U7 json duration_ms sum_over_time, instant instant defaults 23 ms 29 ms 25 ms 1.16×
U8 logfmt amount rate by service (grouped) 1h defaults 47 ms 48 ms 66 ms 0.73×
U9 logfmt amount rate_counter (folded per series) 1h defaults 53 ms 54 ms 368 ms † —
U1 json duration_ms sum by service 1h defaults 107 ms 93 ms 184 ms 0.51×
U10 instant sum_over_time unwrap instant defaults 33 ms 36 ms 30 ms 1.20×
U11 instant max_over_time missing label instant defaults 16 ms 16 ms 20 ms † —
U12 json duration_ms over every service 1h defaults 269 ms 269 ms 738 ms 0.36×
U13 logfmt latency_ms over every service 1h defaults 475 ms 483 ms 1.15 s 0.42×
U14 duration() of a unit string 1h defaults 64 ms 63 ms 48 ms 1.31×
U2 json duration_ms max by service 1h defaults 102 ms 96 ms 1.17 s 0.08×
U3 json duration_ms sliding sum 1h defaults 15 ms 17 ms 209 ms 0.08×
U4 logfmt size_bytes sum by service 1h defaults 51 ms 57 ms 39 ms 1.46×
U5 logfmt ttl unit string (no sample) 1h defaults 48 ms 49 ms 41 ms † —
U6 missing label, no parser 1h defaults 21 ms 23 ms 80 ms † —
U7 missing label, ungrouped sum 1h defaults 22 ms 22 ms 81 ms † —
U8 rate of an unwrapped label 1h defaults 104 ms 94 ms 175 ms 0.54×
U9 bytes() conversion 1h defaults 69 ms 65 ms 43 ms 1.51×

† Loki's results cache answered this cold run (its own counters, or identical answers for windows a minute apart): not a computed time, so it is in no ratio.

↺ Loki cold taken from the first set that ran the same query; Loki's cache answered this one.

48 shape×range above 2× Loki already (listed, or as slow on the base build)
profile mode shape range PR Loki PR ÷ Loki base ÷ Loki listed ÷ Loki why
known defaults cold A explore volume, 2 filters, drop 1h 1.61 s 511 ms 3.14× 3.27× — as slow on base
known defaults warm logs | logfmt 1h 192 ms 69 ms 2.78× 2.80× — as slow on base
known defaults cold SF1 logs, json extraction list 1h 129 ms 64 ms 2.02× 2.52× — as slow on base
known defaults warm SF1 logs, json extraction list 1h 170 ms 62 ms 2.74× 2.95× — as slow on base
known defaults cold SF2 logs, logfmt extraction list 1h 200 ms 65 ms 3.08× 3.03× — as slow on base
known defaults warm SF2 logs, logfmt extraction list 1h 240 ms 62 ms 3.87× 4.29× — as slow on base
known defaults warm SF3 logs, json then line_format 1h 175 ms 83 ms 2.11× 2.12× — as slow on base
known defaults warm SF4 logs, line_format without a parser 1h 166 ms 61 ms 2.72× 2.15× — as slow on base
known defaults cold SF5 logs, regexp capture 1h 220 ms 73 ms 3.01× 3.05× — as slow on base
known defaults warm SF5 logs, regexp capture 1h 251 ms 71 ms 3.54× 3.69× — as slow on base
known defaults warm SF7 logs, filter on a line key without a parser 1h 124 ms 51 ms 2.43× 2.31× — as slow on base
known defaults warm E1 logs, json parser over stream labels 1h 161 ms 80 ms 2.01× 1.98× — within noise of base
known defaults cold E7 json count by level_extracted 1h 1.60 s 200 ms 7.99× 7.46× — as slow on base
known defaults warm E7 json count by level_extracted 1h 99 ms 21 ms 4.71× 5.24× — as slow on base
known defaults cold OR2 logs, negated or alternatives 1h 178 ms 71 ms 2.51× 2.51× — as slow on base
known defaults warm OR2 logs, negated or alternatives 1h 186 ms 75 ms 2.48× 2.33× — as slow on base
known defaults warm P1 logs, json, level equals 1h 159 ms 75 ms 2.12× 2.05× — as slow on base
known defaults cold X1 json filter level_extracted, by level 1h 3.67 s 1.18 s 3.10× 3.11× — as slow on base
known defaults cold P2 api-gateway not empty pattern, line filter 1h 110 ms 21 ms 5.24× 5.10× — as slow on base
known defaults warm P2 api-gateway not empty pattern, line filter 1h 103 ms 24 ms 4.29× 4.58× — as slow on base
known defaults warm A2 nginx exclude GET, log query 1h 142 ms 9 ms 16× 15× — as slow on base
known defaults cold DLV1 drilldown logs volume, json 1h 1.21 s 217 ms 5.57× 5.49× — as slow on base
known defaults warm DLV1 drilldown logs volume, json 1h 1.18 s 28 ms 42× 42× — as slow on base
known defaults cold DLV2 json error filter, detected_level 1h 5.08 s 989 ms 5.14× 5.16× — as slow on base
known defaults warm DLV2 json error filter, detected_level 1h 5.06 s 74 ms 68× 69× — as slow on base
known defaults cold DLV3 plain json, detected_level 1h 1.27 s 217 ms 5.84× 5.86× — as slow on base
known defaults warm DLV3 plain json, detected_level 1h 1.20 s 21 ms 57× 57× — as slow on base
known defaults cold LF1 logs, detected_level equals 1h 202 ms 82 ms 2.46× 2.45× — as slow on base
known defaults warm LF1 logs, detected_level equals 1h 194 ms 78 ms 2.49× 2.53× — as slow on base
known defaults cold LF2 logs, detected_level regexp 1h 203 ms 31 ms 6.55× 6.32× — as slow on base
known defaults warm LF2 logs, detected_level regexp 1h 193 ms 33 ms 5.85× 5.91× — as slow on base
known defaults cold LF3 logs, detected_level not equal, all services 1h 307 ms 76 ms 4.04× 4.20× — as slow on base
known defaults warm LF3 logs, detected_level not equal, all services 1h 214 ms 75 ms 2.85× 2.92× — as slow on base
known defaults cold LF4 logs, detected_level regexp with unknown 1h 420 ms 131 ms 3.21× 3.31× — as slow on base
known defaults warm LF4 logs, detected_level regexp with unknown 1h 403 ms 126 ms 3.20× 3.29× — as slow on base
known defaults cold LF5 logs, plain-text service, detected_level regexp 1h 91 ms 15 ms 6.07× 6.07× — as slow on base
known defaults warm LF5 logs, plain-text service, detected_level regexp 1h 87 ms 15 ms 5.80× 5.47× — as slow on base
known defaults warm LF9 drilldown logs, level pill, logfmt service 1h 91 ms 29 ms 3.14× 2.97× — as slow on base
known defaults cold LF10 drilldown logs, level pill multi-select, json service 1h 159 ms 45 ms 3.53× 3.58× — as slow on base
known defaults warm LF10 drilldown logs, level pill multi-select, json service 1h 150 ms 44 ms 3.41× 3.41× — as slow on base
… and 8 more (verdict.json counts them; the job's summary-<set>.json files have every row).
Why these shapes
  • baseline run: every shape of every set → 264: aggregation-over-binary/B1 sum by over rate divided by a literal, aggregation-over-binary/B2 sum by over a ratio of two per-stream metrics, aggregation-over-binary/B3 sum over a literal times count, instant, aggregation-over-binary/B4 outer sum over ignoring(machine), control/by service_name, control/bytes_over_time by app, control/detected_level volume, control/drilldown detected_level …

base bb8f348bee → PR 550b533229 · 4 runs (first is cold), warm p50 · ranges 1h · noise ±30% and 100 ms · VictoriaLogs restarts 0→0 · 11.8 min · phases stack_up 36s, seed_and_build 154s, loki-defaults:loki_profile 0s, loki-defaults:proxies 2s, loki-defaults:matrix 501s. Same runner, one fresh stack, targets interleaved per request. How to read and reproduce: bench/ab/README.md.

🧭 New baseline candidate

This pull request got the baseline run. Every shape of every set was measured at the smoke's ranges (1h, 4 runs). Nothing was committed. The perf-baseline-candidate artifact of this run holds over-2x-loki.json, which a person can copy over bench/ab/history/over-2x-loki.json to accept it, and acceptance-day.json, a report: the committed acceptance.json is rebuilt from the daily history, so the next daily run after the merge writes the real new distribution.

profile mode rows compared below Loki close 1–2× above 2× share below committed share below
loki-defaults cold 214 118 73 2 21 55.1% 50.5% (2026-10-29)
loki-defaults warm 259 77 152 3 27 29.7% 23.2% (2026-10-29)

The committed column is the newest daily run (7 runs, ranges up to 24h): the direction is comparable, the rows are not the same ones.

Above-2× list: 213 rows after this run, 48 added, 0 not above 2× in this run's profile (the daily run's two-run rule decides for the committed list). A profile that did not run leaves its committed rows as they are.

Added:

  • loki-defaults / control / logs | logfmt / 1h / warm
  • loki-defaults / extracted-pushdown / X1 json filter level_extracted, by level / 1h / cold
  • loki-defaults / extracted-suffix / E1 logs, json parser over stream labels / 1h / warm
  • loki-defaults / extracted-suffix / E7 json count by level_extracted / 1h / cold
  • loki-defaults / extracted-suffix / E7 json count by level_extracted / 1h / warm
  • loki-defaults / json-detected-level / DLV1 drilldown logs volume, json / 1h / cold
  • loki-defaults / json-detected-level / DLV1 drilldown logs volume, json / 1h / warm
  • loki-defaults / json-detected-level / DLV2 json error filter, detected_level / 1h / cold
  • loki-defaults / json-detected-level / DLV2 json error filter, detected_level / 1h / warm
  • loki-defaults / json-detected-level / DLV3 plain json, detected_level / 1h / cold
  • … and 38 more (in over-2x-loki.json)

12 min.

@github-actions

github-actions Bot commented Oct 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

PR Quality Report

Compared against base branch main.

Coverage and tests

Signal Base PR Delta
Test count 18874 19044 170
Coverage 89.2% 89.2% 0.0% (stable)

Compatibility

Track Base PR Delta
Loki API 100.0% 11/11 (100.0%) 0.0% (stable)
Logs Drilldown 100.0% 17/17 (100.0%) 0.0% (stable)
VictoriaLogs 100.0% 11/11 (100.0%) 0.0% (stable)

Performance smoke

Lower CPU cost (ns/op) is better. Lower benchmark memory cost (B/op, allocs/op) is better. Higher throughput is better. Lower load-test memory growth is better. Benchmark rows are medians from repeated samples.

Signal Base PR Delta
QueryRange cache-hit CPU cost 3412.0 ns/op 3413.0 ns/op +0.0% (stable)
QueryRange cache-hit memory 1256.0 B/op 1256.0 B/op 0.0% (stable)
QueryRange cache-hit allocations 28.0 allocs/op 28.0 allocs/op 0.0% (stable)
QueryRange cache-bypass CPU cost 3985.0 ns/op 3991.0 ns/op +0.2% (stable)
QueryRange cache-bypass memory 1320.0 B/op 1320.0 B/op 0.0% (stable)
QueryRange cache-bypass allocations 28.0 allocs/op 28.0 allocs/op 0.0% (stable)
Labels cache-hit CPU cost 872.5 ns/op 885.2 ns/op +1.5% (stable)
Labels cache-hit memory 48.0 B/op 48.0 B/op 0.0% (stable)
Labels cache-hit allocations 3.0 allocs/op 3.0 allocs/op 0.0% (stable)
Labels cache-bypass CPU cost 1470.0 ns/op 1486.0 ns/op +1.1% (stable)
Labels cache-bypass memory 399.0 B/op 399.0 B/op 0.0% (stable)
Labels cache-bypass allocations 14.0 allocs/op 14.0 allocs/op 0.0% (stable)

State

  • Test count and coverage were collected for this PR.
  • Compatibility and sampled performance are reported here from the same PR workflow.
  • This is a delta report, not a release gate by itself. Required checks still decide merge safety.
  • Performance is a smoke comparison, not a full benchmark lab run.
  • Delta states use the same noise guards as the quality gate (percent + absolute + low-baseline checks), so report labels match merge-gate behavior.

@szibis szibis mentioned this pull request Oct 12, 2026
Merged
@szibis szibis mentioned this pull request Oct 23, 2026
Merged
@szibis
szibis marked this pull request as ready for review October 29, 2026 10:35
szibis added a commit that referenced this pull request Oct 29, 2026
@szibis
chore(conformance): record #748 as the fix of proxy-accepts-operators…
9b91d67 
…-not-in-loki
@szibis
szibis force-pushed the fix/group-aggregation-not-in-loki branch from ccd6969 to c0d9f6d Compare October 29, 2026 11:34
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 29, 2026
@szibis

szibis commented Oct 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Verification for PR #748, signed HEAD fa7924bffcbbca3c9998a51a7583e852093df731, rebased onto main bb8f348beeecc1a2e076210c2dc3e98e774d5e36 (#737). The branch is mergeable. Current-head CI completed with 72 successes, zero failures, one neutral and four expected skips; review is still required.

  • Fresh local Grafana main | PR | Loki replay on one Compose stack: 17/17 Playwright captures passed in 3.6 minutes. Nine general core captures at 1h had identical main and PR data; each of the eight fix: reject group(), label_join() and other names Loki does not have, and accept keywords in any case, like Loki #748 exact LogQL captures differed on main and matched Loki on the PR (2/2 or 3/3 backend requests). The local visual gate exited 0 and marked semantics/proxy-accepts-operators-not-in-loki fixed. The eight cases are group(), label_join(), uppercase SUM/COUNT, TOPK/SUM/RATE, JSON/UNWRAP, ON, BOOL, and OFFSET. The PR description lists every exact query and before/after/reference caption.
  • Go parser, translator, and proxy full suite passed (go test -count=1 ./internal/logql ./internal/translator ./internal/proxy). Visual tests 163/163, CI configuration tests 311/311, actionlint, conformance wiring and parity checks, plan.py --check, changelog gate, and git diff --check passed. Regression tests assert all eight fix scenes survive bounded and baseline planning, appear in rendered proof and image publication selection, and the nine core scenes run even on docs/CI-only PRs.
  • Real Loki differential E2E passed locally: TestCompat_OperatorsNotInLokiRejected and TestCompat_UppercaseKeywordsMatchLoki. Direct GET/form POST API proof across /query and /query_range covered 20 requests: 8 exact Loki 400 error matches and 12 matching successful uppercase answers. Other cases compare full series/values; TOPK compares ranked point values because 20 identical Loki requests yielded six tied label sets but one ranked-value result.
  • Current-head quality report: 19,044 PR tests versus 18,874 base, both 89.2% coverage; Loki API 11/11, Drilldown 17/17, VL 11/11. Current-head performance A/B: 264/264 shape×range pairs within noise and zero new Loki answer differences. The preceding core E2E /labels fixture failed once and passed on an identical-SHA job rerun.

The final-head Visual smoke run passed capture, bounded publication, and comment jobs over 245 baseline captures. Its automatic ✅ Visual smoke comment shows base bb8f348bee and PR fa7924bffc, marks the #748 case ✅ fixed, and displays all eight exact fix screenshots plus all nine core regression screenshots inline and open. Each of those 17 raw image URLs was checked and returned HTTP 200 image/png. The PR body embeds the eight fix images beside exact queries and before/after/reference captions.

@github-actions

github-actions Bot commented Oct 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

✅ Visual smoke: passed with warnings

Base bb8f348bee vs PR fa7924bffc, Grafana through the proxy builds and Loki on one fresh stack; run.
Loaded again, once and more patiently, after a first-pass difference or an unsettled page: explore-748-reject-group 1h, explore-748-reject-label-join 1h, explore-748-uppercase-bool 1h, explore-748-uppercase-json-unwrap 1h, explore-748-uppercase-offset 1h, explore-748-uppercase-on 1h, explore-748-uppercase-sum 1h, explore-748-uppercase-topk-rate 1h, lvl-dd-logs-pill-multi-json 1h.
Core set: 9 capture(s) at 1h. Detailed set: 119 entries (explore-logs-plain, explore-metric-plain, lvl-dd-logs-pill-json, lvl-dd-logs-pill-otel, lvl-dd-fields-pill, lvl-dd-field-method-pill, lvl-dd-field-pipeline-pill, lvl-dd-logs-pill-logfmt, ...). 245 capture(s), 15m/1h/6h, 24h and 7d stay out of PR runs.

Fix proof

Captures of the exact cases this pull request adds or fixes (registry case, base | PR | Loki). The verdict compares the PR with Loki where the base did not match. The gate fails a case unless one capture is fixed or improved, or, beyond the Loki data window, the base failed where Loki and the PR both answered with data (the label does not excuse it; only visual: none on the case does).

semantics/proxy-accepts-operators-not-in-loki: ✅ fixed

page range base = PR (data) vs Loki (PR) verdict
explore-748-reject-group 1h differs (1) identical ✅ fixed
explore-748-reject-label-join 1h differs (1) identical ✅ fixed
explore-748-uppercase-bool 1h differs (1) identical ✅ fixed
explore-748-uppercase-json-unwrap 1h differs (1) identical ✅ fixed
explore-748-uppercase-offset 1h differs (1) identical ✅ fixed
explore-748-uppercase-on 1h differs (1) identical ✅ fixed
explore-748-uppercase-sum 1h differs (1) identical ✅ fixed
explore-748-uppercase-topk-rate 1h differs (1) identical ✅ fixed

explore-748-reject-group-1h — base | PR | Loki

explore-748-reject-group-1h

explore-748-reject-label-join-1h: fixed

explore-748-reject-label-join-1h

explore-748-uppercase-bool-1h: fixed

explore-748-uppercase-bool-1h

explore-748-uppercase-json-unwrap-1h: fixed

explore-748-uppercase-json-unwrap-1h

explore-748-uppercase-offset-1h: fixed

explore-748-uppercase-offset-1h

explore-748-uppercase-on-1h: fixed

explore-748-uppercase-on-1h

explore-748-uppercase-sum-1h: fixed

explore-748-uppercase-sum-1h

explore-748-uppercase-topk-rate-1h: fixed

explore-748-uppercase-topk-rate-1h

Regression check

page range set base = PR (data) vs Loki (PR) pixel diff result
dd-fields 1h core ✅ identical identical (4 explained, 1 history-dependent) 0.12% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-landing 1h core ✅ identical identical (3 explained) 0.14% ✅
dd-service-logs 1h core ✅ identical identical (3 explained, 1 history-dependent) 0.12% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
explore-logs-plain 1h core ✅ identical identical (1 explained) 0.07% ✅
explore-metric-plain 1h core ✅ identical identical (1 explained) 0.04% ✅
explore-sum-by-over-binary 1h core ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-topk-by-per-step 1h core ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-sum 1h core ✅ identical 1 difference(s), 1 on base too 0.04% ✅
lvl-explore-rate-errors-by-service 1h core ✅ identical 1 difference(s), 1 on base too; 1 explained 0.04% ✅
dd-field-pipeline 15m detailed ✅ identical identical (3 explained, 1 history-dependent) 0.12% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-field-pipeline 1h detailed ✅ identical identical (3 explained, 1 history-dependent) 0.12% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-field-pipeline 6h detailed ✅ identical n/a (beyond the Loki window) 0.11% ✅
dd-field-service_version 15m detailed ✅ identical identical (3 explained, 1 history-dependent) 0.12% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-field-service_version 1h detailed ✅ identical identical (3 explained, 1 history-dependent) 0.12% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-field-service_version 6h detailed ✅ identical n/a (beyond the Loki window) 0.11% ✅
dd-fields 15m detailed ✅ identical identical (4 explained, 1 history-dependent) 0.12% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-fields 6h detailed ✅ identical n/a (beyond the Loki window) 0.11% ✅
dd-labels 15m detailed ✅ identical identical (3 explained, 1 history-dependent) 0.12% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-labels 1h detailed ✅ identical identical (3 explained, 1 history-dependent) 0.12% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-labels 6h detailed ✅ identical n/a (beyond the Loki window) 0.11% ✅
dd-labels-namespace 15m detailed ✅ identical identical (3 explained, 1 history-dependent) 0.12% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-labels-namespace 1h detailed ✅ identical identical (3 explained, 1 history-dependent) 0.12% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-labels-namespace 6h detailed ✅ identical n/a (beyond the Loki window) 0.11% ✅
dd-landing 15m detailed ✅ identical identical (3 explained) 0.14% ✅
dd-landing 6h detailed ✅ identical n/a (beyond the Loki window) 0.14% ✅
dd-patterns 15m detailed ✅ identical identical (3 explained, 1 history-dependent) 3.41% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-patterns 1h detailed ✅ identical identical (3 explained, 1 history-dependent) 6.09% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-patterns 6h detailed ✅ identical n/a (beyond the Loki window) 0.11% ✅
dd-service-logs 15m detailed ✅ identical identical (3 explained, 1 history-dependent) 0.12% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-service-logs 6h detailed ✅ identical n/a (beyond the Loki window) 0.11% ✅
explore-748-reject-group 1h detailed differs (1): query ('group(count_over_time({app="api-gateway"}[$__auto])) identical 2.46% ✅ improved (closer to Loki)
explore-748-reject-label-join 1h detailed differs (1): query ('label_join(rate({app="api-gateway"}[$__auto]), "d", identical 3.15% ✅ improved (closer to Loki)
explore-748-uppercase-bool 1h detailed differs (1): query ('SUM BY (app) (COUNT_OVER_TIME({env="production"}[$__ identical 2.92% ✅ improved (closer to Loki)
explore-748-uppercase-json-unwrap 1h detailed differs (1): query ('SUM BY (app) (SUM_OVER_TIME({env="production"} | JSO identical 3.48% ✅ improved (closer to Loki)
explore-748-uppercase-offset 1h detailed differs (1): query ('SUM BY (app) (COUNT_OVER_TIME({env="production"}[$__ identical 3.98% ✅ improved (closer to Loki)
explore-748-uppercase-on 1h detailed differs (1): query ('SUM BY (app) (COUNT_OVER_TIME({env="production"}[$__ identical 2.74% ✅ improved (closer to Loki)
explore-748-uppercase-sum 1h detailed differs (1): query ('SUM BY (app) (COUNT_OVER_TIME({env="production"}[$__ identical 3.97% ✅ improved (closer to Loki)
explore-748-uppercase-topk-rate 1h detailed differs (1): query ('TOPK(3, SUM BY (app) (RATE({env="production"}[$__aut identical 3.33% ✅ improved (closer to Loki)
explore-A 15m detailed ✅ identical identical (1 explained) 0.04% ✅
explore-A 1h detailed ✅ identical identical (1 explained) 0.04% ✅
explore-A 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-B 15m detailed ✅ identical identical (1 explained) 0.04% ✅
explore-B 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-B 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-C 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-C 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-C 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-D 15m detailed ✅ identical identical (1 explained) 0.04% ✅
explore-D 1h detailed ✅ identical identical (1 explained) 0.04% ✅
explore-D 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-DLV5 15m detailed ✅ identical identical (1 explained) 0.04% ✅
explore-DLV5 1h detailed ✅ identical identical (1 explained) 0.04% ✅
explore-DLV5 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-E 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-E 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-E 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-F 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-F 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-F 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-absent-over-time 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-absent-over-time 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-absent-over-time 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-empty-pattern-filter 15m detailed ✅ identical 1 difference(s), 1 on base too 0.07% ✅
explore-empty-pattern-filter 1h detailed ✅ identical 1 difference(s), 1 on base too 0.07% ✅
explore-empty-pattern-filter 6h detailed ✅ identical n/a (beyond the Loki window) 0.07% ✅
explore-extracted-filter-metric 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-extracted-filter-metric 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-extracted-filter-metric 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-extracted-group-metric 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-extracted-group-metric 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-extracted-group-metric 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-extracted-metric 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-extracted-metric 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-extracted-metric 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-json-error-filter-metric 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-json-error-filter-metric 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-json-error-filter-metric 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-label-browser 15m detailed ✅ identical identical 0.02% ✅
explore-label-browser 1h detailed ✅ identical identical 0.02% ✅
explore-label-browser 6h detailed ✅ identical n/a (beyond the Loki window) 0.02% ✅
explore-label-replace-over-binary 15m detailed ✅ identical 2 difference(s), 2 on base too 0.04% ✅
explore-label-replace-over-binary 1h detailed ✅ identical 2 difference(s), 2 on base too 0.04% ✅
explore-label-replace-over-binary 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-log-context 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-log-context 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-log-context 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-logfmt-metric 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-logfmt-metric 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-logfmt-metric 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-logs-extracted 15m detailed ✅ identical 1 difference(s), 1 on base too 0.07% ✅
explore-logs-extracted 1h detailed ✅ identical 1 difference(s), 1 on base too 0.07% ✅
explore-logs-extracted 6h detailed ✅ identical n/a (beyond the Loki window) 0.07% ✅
explore-logs-json 15m detailed ✅ identical 1 difference(s), 1 on base too 0.07% ✅
explore-logs-json 1h detailed ✅ identical 1 difference(s), 1 on base too 0.07% ✅
explore-logs-json 6h detailed ✅ identical n/a (beyond the Loki window) 0.07% ✅
explore-logs-json-extraction 15m detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.07% ✅
explore-logs-json-extraction 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.07% ✅
explore-logs-json-extraction 6h detailed ✅ identical n/a (beyond the Loki window) 0.07% ✅
explore-logs-line-filter-or 15m detailed ✅ identical 1 difference(s), 1 on base too 0.07% ✅
explore-logs-line-filter-or 1h detailed ✅ identical 1 difference(s), 1 on base too 0.07% ✅
explore-logs-line-filter-or 6h detailed ✅ identical n/a (beyond the Loki window) 0.07% ✅
explore-logs-line-format 15m detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.07% ✅
explore-logs-line-format 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.07% ✅
explore-logs-line-format 6h detailed ✅ identical n/a (beyond the Loki window) 0.07% ✅
explore-logs-multivalue-regexp 15m detailed ✅ identical identical (1 explained) 0.07% ✅
explore-logs-multivalue-regexp 1h detailed ✅ identical identical (1 explained) 0.07% ✅
explore-logs-multivalue-regexp 6h detailed ✅ identical n/a (beyond the Loki window) 0.07% ✅
explore-logs-plain 15m detailed ✅ identical identical (1 explained) 0.07% ✅
explore-logs-plain 6h detailed ✅ identical n/a (beyond the Loki window) 0.07% ✅
explore-logs-plain-level 15m detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.07% ✅
explore-logs-plain-level 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.07% ✅
explore-logs-plain-level 6h detailed ✅ identical n/a (beyond the Loki window) 0.07% ✅
explore-max-by-count-over-time-sliding 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-max-by-count-over-time-sliding 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-max-by-count-over-time-sliding 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-metric-line-filter-or 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-metric-line-filter-or 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-metric-line-filter-or 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-metric-parenthesized 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-metric-parenthesized 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-metric-parenthesized 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-metric-plain 15m detailed ✅ identical identical (1 explained) 0.04% ✅
explore-metric-plain 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-nested-aggregation-outer-by 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-nested-aggregation-outer-by 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-nested-aggregation-outer-by 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-pattern-filter-anchored 15m detailed ✅ identical 1 difference(s), 1 on base too 0.07% ✅
explore-pattern-filter-anchored 1h detailed ✅ identical 1 difference(s), 1 on base too 0.07% ✅
explore-pattern-filter-anchored 6h detailed ✅ identical n/a (beyond the Loki window) 0.07% ✅
explore-plain-level-metric 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-plain-level-metric 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-plain-level-metric 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-sliding-sum-without-stream-fold 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-sliding-sum-without-stream-fold 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-sliding-sum-without-stream-fold 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-sum-by-over-binary 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-sum-by-over-binary 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-sum-without-comparison-inside 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-sum-without-comparison-inside 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-sum-without-comparison-inside 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-sum-without-count 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-sum-without-count 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-sum-without-count 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-sum-without-rate 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-sum-without-rate 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-sum-without-rate 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-topk-by-per-step 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-topk-by-per-step 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-topk-sum-without 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-topk-sum-without 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-topk-sum-without 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-topk-without-keeps-labels 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-topk-without-keeps-labels 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-topk-without-keeps-labels 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-unwrap-conversion-error 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-conversion-error 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-conversion-error 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-unwrap-duration-seconds 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-duration-seconds 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-duration-seconds 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-unwrap-first-last-under-sum 15m detailed ✅ identical 2 difference(s), 2 on base too 0.04% ✅
explore-unwrap-first-last-under-sum 1h detailed ✅ identical 2 difference(s), 2 on base too 0.04% ✅
explore-unwrap-first-last-under-sum 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-unwrap-max-over-time-end-sample 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-max-over-time-end-sample 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-max-over-time-end-sample 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-unwrap-missing-label 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-missing-label 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-missing-label 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-unwrap-quantile-over-time 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-quantile-over-time 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-quantile-over-time 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-unwrap-rate-counter-one-sample 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-rate-counter-one-sample 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-rate-counter-one-sample 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-unwrap-rate-sum 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-rate-sum 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-rate-sum 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-unwrap-sum 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-sum 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-unwrap-sum-over-time-sliding-window 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-sum-over-time-sliding-window 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-sum-over-time-sliding-window 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-unwrap-unit-string 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-unit-string 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-unwrap-unit-string 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-volume-label-filter-regexp 15m detailed ✅ identical identical (1 explained) 0.04% ✅
explore-volume-label-filter-regexp 1h detailed ✅ identical identical (1 explained) 0.04% ✅
explore-volume-label-filter-regexp 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-volume-multivalue-regexp 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-volume-multivalue-regexp 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-volume-multivalue-regexp 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-without-nested-under-aggregation 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-without-nested-under-aggregation 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-without-nested-under-aggregation 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
explore-without-over-inner-aggregation 15m detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-without-over-inner-aggregation 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
explore-without-over-inner-aggregation 6h detailed ✅ identical n/a (beyond the Loki window) 0.04% ✅
lvl-dd-field-method-pill 1h detailed ✅ identical 1 difference(s), 1 on base too; 2 explained 0.11% ✅
lvl-dd-field-pipeline-pill 1h detailed ✅ identical 1 difference(s), 1 on base too; 2 explained, 1 history-dependent 0.12% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
lvl-dd-fields-pill 1h detailed ✅ identical 25 difference(s), 25 on base too; 4 explained 0.11% ✅
lvl-dd-label-pod-pill 1h detailed ✅ identical 1 difference(s), 1 on base too; 2 explained, 1 history-dependent 0.12% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
lvl-dd-labels-pill 1h detailed ✅ identical 2 difference(s), 2 on base too; 2 explained 0.11% ✅
lvl-dd-logs-pill-json 1h detailed ✅ identical 1 difference(s), 1 on base too; 2 explained 0.11% ✅
lvl-dd-logs-pill-logfmt 1h detailed ✅ identical identical (4 explained, 1 history-dependent) 0.11% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
lvl-dd-logs-pill-multi-json 1h detailed ✅ identical 2 difference(s), 2 on base too; 2 explained, 1 history-dependent 0.11% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
lvl-dd-logs-pill-otel 1h detailed ✅ identical 1 difference(s), 1 on base too; 2 explained, 1 history-dependent 0.12% ⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
lvl-dd-logs-pill-plaintext-bodyonly 1h detailed ✅ identical 2 difference(s), 2 on base too; 2 explained, 1 history-dependent 0.11% ✅
lvl-explore-count-warn-by-service 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.04% ✅
lvl-explore-logs-dl-eq 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-logs-dl-error-plaintext-bodyonly 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
lvl-explore-logs-dl-neq-all 1h detailed ✅ identical 2 difference(s), 2 on base too; 1 explained 0.07% ✅
lvl-explore-logs-dl-plaintext 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.07% ✅
lvl-explore-logs-dl-regexp 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-logs-dl-unknown-bodyonly 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
lvl-explore-logs-dl-unknown-or-error 1h detailed ✅ identical 2 difference(s), 2 on base too; 1 explained 0.07% ✅
lvl-explore-logs-drop-dl 1h detailed ✅ identical 2 difference(s), 2 on base too; 1 explained 0.07% ✅
lvl-explore-logs-drop-level 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.07% ✅
lvl-explore-logs-json-dl 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-logs-json-level 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-logs-json-level-extracted-stored 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-logs-keep-dl 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.07% ✅
lvl-explore-logs-level-label-stored 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-logs-level-selector-stored 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-logs-logfmt-level-extracted-stored 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-logs-logfmt-level-regexp 1h detailed ✅ identical identical (1 explained) 0.07% ✅
lvl-explore-topk-errors-by-service 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.04% ✅
lvl-explore-volume-dl-neq 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.04% ✅
lvl-explore-volume-dl-prefilter-json 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-dl-regexp 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.04% ✅
lvl-explore-volume-dl-unknown-bodyonly 1h detailed ✅ identical 1 difference(s), 1 on base too 0.04% ✅
lvl-explore-volume-dl-unknown-or-error 1h detailed ✅ identical 1 difference(s), 1 on base too; 1 explained 0.04% ✅
lvl-explore-volume-drop-level 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-json-by-dl 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-json-dl-filter 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-json-error-by-dl 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-json-level-filter 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-keep-dl 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-level-extracted-stored 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-level-label-stored 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-logfmt-dl 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-logfmt-dl-filter 1h detailed ✅ identical identical (1 explained) 0.04% ✅
lvl-explore-volume-plaintext 1h detailed ✅ identical identical (1 explained) 0.04% ✅
tail-env live detailed ✅ identical entries in the window: PR 9703, Loki 7725, in both 7725 6.73% ✅
tail-json-filter live detailed ✅ identical entries in the window: PR 1975, Loki 314, in both 314 7.35% ✅

Gate: a base-vs-PR data difference passes when the PR is closer to Loki, and fails when the PR diverges from Loki or Loki cannot decide it (at a range Loki does not hold, a difference passes when the same page proved that query to answer like Loki at a shorter range; otherwise label the pull request visual-change-expected to accept an intended change). Also failing: a panel empty on the PR but not on the base, an error banner, panel error, error answer or new "No data" panel on the PR, a PR side that never settled, a value difference that did not reproduce on the recapture. A delayed first-pass request and pixel differences only warn.

dd-fields-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3A00.000Z&query=%7Bservic: body: content differs (rows 5 vs 2)
  • vs Loki, explained: query ('sum by (service_name_extracted) (count_over_time({service_name="otel-collector"} | servi: request only on one side (right) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision)
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-fields-1h

dd-service-logs-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3A00.000Z&query=%7Bservic: body: content differs (rows 5 vs 2)
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-service-logs-1h

dd-field-pipeline-15m: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3A00.000Z&query=%7Bservic: body: content differs (rows 6 vs 5)
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-field-pipeline-15m

dd-field-pipeline-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3A00.000Z&query=%7Bservic: body: content differs (rows 5 vs 2)
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-field-pipeline-1h

dd-field-service_version-15m: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3A00.000Z&query=%7Bservic: body: content differs (rows 6 vs 5)
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-field-service_version-15m

dd-field-service_version-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3A00.000Z&query=%7Bservic: body: content differs (rows 5 vs 2)
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-field-service_version-1h

dd-fields-15m: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3A00.000Z&query=%7Bservic: body: content differs (rows 6 vs 5)
  • vs Loki, explained: query ('sum by (service_name_extracted) (count_over_time({service_name="otel-collector"} | servi: request only on one side (right) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision)
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-fields-15m

dd-labels-15m: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3A00.000Z&query=%7Bservic: body: content differs (rows 6 vs 5)
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-labels-15m

dd-labels-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3A00.000Z&query=%7Bservic: body: content differs (rows 5 vs 2)
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-labels-1h

dd-labels-namespace-15m: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3A00.000Z&query=%7Bservic: body: content differs (rows 6 vs 5)
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-labels-namespace-15m

dd-labels-namespace-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3A00.000Z&query=%7Bservic: body: content differs (rows 5 vs 2)
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-labels-namespace-1h

dd-patterns-15m: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3A00.000Z&query=%7Bservic: body: content differs (rows 6 vs 5)
  • pixel diff 3.4%
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-patterns-15m

dd-patterns-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3A00.000Z&query=%7Bservic: body: content differs (rows 5 vs 2)
  • pixel diff 6.1%
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-patterns-1h

dd-service-logs-15m: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3A00.000Z&query=%7Bservic: body: content differs (rows 6 vs 5)
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-service-logs-15m

explore-748-reject-label-join-1h: pixel diff 3.2%
  • pixel diff 3.2%

explore-748-reject-label-join-1h

explore-748-uppercase-json-unwrap-1h: pixel diff 3.5%
  • pixel diff 3.5%

explore-748-uppercase-json-unwrap-1h

explore-748-uppercase-offset-1h: pixel diff 4.0%
  • pixel diff 4.0%

explore-748-uppercase-offset-1h

explore-748-uppercase-sum-1h: pixel diff 4.0%
  • pixel diff 4.0%

explore-748-uppercase-sum-1h

explore-748-uppercase-topk-rate-1h: pixel diff 3.3%
  • pixel diff 3.3%

explore-748-uppercase-topk-rate-1h

lvl-dd-field-pipeline-pill-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3A00.000Z&query=%7Bservic: body: content differs (rows 5 vs 2)
  • vs Loki: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 36 vs 15)
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

lvl-dd-field-pipeline-pill-1h

lvl-dd-label-pod-pill-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3A00.000Z&query=%7Bservic: body: content differs (rows 5 vs 2)
  • vs Loki: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 36 vs 15)
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

lvl-dd-label-pod-pill-1h

lvl-dd-logs-pill-logfmt-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3A00.000Z&query=%7Bservic: body: content differs (rows 10 vs 4)
  • vs Loki, explained: query ('{service_name="payment-service"} | detected_level="warn" | json | logfmt | drop __er: ('logsPanelQuery', '', '{}'): content differs (rows 1000 vs 1000) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision)
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 8 vs 9) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 8 vs 8) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

lvl-dd-logs-pill-logfmt-1h

lvl-dd-logs-pill-multi-json-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3A00.000Z&query=%7Bservic: body: content differs (rows 50 vs 5)
  • first capture missed a delayed request; base and PR matched on the patient recapture
  • vs Loki: query ('sum(count_over_time({service_name="frontend-ssr"} [$__auto])) by (detected_level)',: request only on one side (left)
  • vs Loki: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 10 vs 0)
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 8 vs 8) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

lvl-dd-logs-pill-multi-json-1h

lvl-dd-logs-pill-otel-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3
  • history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T16%3A19%3A00.000Z&query=%7Bservic: body: content differs (rows 5 vs 2)
  • vs Loki: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 36 vs 15)
  • vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T16%3A19%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
  • vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

lvl-dd-logs-pill-otel-1h

Core set montages (7): base | PR | Loki

dd-landing-1h

dd-landing-1h

explore-logs-plain-1h

explore-logs-plain-1h

explore-metric-plain-1h

explore-metric-plain-1h

explore-sum-by-over-binary-1h

explore-sum-by-over-binary-1h

explore-topk-by-per-step-1h

explore-topk-by-per-step-1h

explore-unwrap-sum-1h

explore-unwrap-sum-1h

lvl-explore-rate-errors-by-service-1h

lvl-explore-rate-errors-by-service-1h

Published inline montages: pr-visuals/pr-748. Full montage set: artifact.

🧭 New baseline candidate

This pull request got the baseline run: every page of bench/visual/spec.json was captured at its own ranges. The full candidate is the visual-baseline-candidate artifact of https://github.com/ReliablyObserve/loki-vl-proxy/actions/runs/40910139389; only bounded inline review images were published, and the candidate was not accepted as a baseline.

@github-actions github-actions Bot added size/XL Extra large change scope/ci CI/CD and removed size/XL Extra large change labels Oct 29, 2026
szibis added a commit that referenced this pull request Oct 29, 2026
@szibis
chore(conformance): record #748 as the fix of proxy-accepts-operators…
61849d1 
…-not-in-loki
@szibis
szibis force-pushed the fix/group-aggregation-not-in-loki branch from 14ee8bd to 8bc0fbc Compare October 29, 2026 14:42
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 29, 2026
szibis added a commit that referenced this pull request Oct 29, 2026
@szibis
chore(conformance): record #748 as the fix of proxy-accepts-operators…
0003238 
…-not-in-loki
@szibis
szibis force-pushed the fix/group-aggregation-not-in-loki branch from 8bc0fbc to b56a093 Compare October 29, 2026 15:09
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 29, 2026
szibis and others added 12 commits October 29, 2026 17:02
@szibis
fix: reject group(), label_join() and other names Loki does not have,…
c6710a9 
… accept keywords in any case

Loki 3.7.7 has no group aggregation and no label_join: its lexer makes a function token only of the names in functionTokens, so any other name is an IDENTIFIER the grammar cannot place and the query is a 400 parse error at the column of the name. The proxy kept every unknown name(...) as an opaque call and answered it. The parser now keeps opaque only label_replace, vector and approx_topk and reports every other name the way Loki does, also inside a label_replace argument.

Loki lower-cases a word before it looks it up, so TOPK(3, X), SUM BY (a) (X), RATE(...) and OFFSET are the keywords they spell, while a label called Rate or Count stays a label. The scanner reads keywords in any case and the query parameter is written with lower-case keywords before a handler reads it.
@szibis
refactor: remove the group() and label_join() translation and respons…
9360e25 
…e rewriting

Loki has neither, and the validator now rejects both before the translator runs, so the marker suffixes, their parsers and the proxy-side rewrites of a matrix response were unreachable.
@szibis
test: compare group(), label_join() and uppercase keywords with Loki,…
a101e5f 
… and document the change

The e2e suite runs the queries Loki rejects against Loki and every Loki-compatible proxy on both query endpoints and compares status and text, and runs uppercase keyword queries and compares the series. group() and label_join() move from the parity and known-gap lists to the error parity list.
@szibis
fix: carry the lower-cased query to the multi-tenant fan-out, and com…
f8ff72c 
…pare the proxy with itself in the e2e

The multi-tenant fan-out re-reads the query from the URL, so the lower-case rewrite covers the URL query as well as the parsed form. The proxy tests now send each uppercase query as a GET, a form-encoded POST and a multi-tenant GET. The e2e compares the upper-case answer of the proxy with its own lower-case answer; Loki only has to answer 200 for both spellings.
@szibis
chore(conformance): record #748 as the fix of proxy-accepts-operators…
f6c8a66 
…-not-in-loki
@szibis
test: align label transform coverage with Loki parser
2377265
@szibis
refactor: remove unused group response type
f081222
@szibis
Preserve quality report evidence when coverage collection fails
67c52e5
@szibis
Align binary aggregation coverage with unsupported label_join
d3dc7a4
@szibis
test: prove LogQL parity with CI visual captures
2961b07
@szibis
docs(changelog): keep released entries unchanged
4836661
@szibis
ci: run visual regression core and fix proofs on every PR
fa7924b
@szibis
szibis force-pushed the fix/group-aggregation-not-in-loki branch from b56a093 to fa7924b Compare October 29, 2026 16:17
@github-actions github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 29, 2026
@szibis
szibis merged commit 91496eb into main Oct 29, 2026
80 checks passed
@szibis
szibis deleted the fix/group-aggregation-not-in-loki branch October 29, 2026 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

bugfix Bug fix scope/ci CI/CD scope/docs Documentation scope/proxy Proxy core scope/tests Tests scope/translator LogQL translator size/XL Extra large change

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@szibis