perf: a detected_level filter before json reads the stored level fields, matching Loki by szibis · Pull Request #795 · ReliablyObserve/loki-vl-proxy

szibis · 2026-10-21T17:45:55Z

Summary

Grafana Explore's logs volume with a level selected, sum by (level, detected_level) (count_over_time({env="production"} | detected_level="info" | json | drop __error__ [$__auto])), was slow and miscounted.

Cause: the translator rendered the detected_level filter before the parser as | unpack_logfmt | filter level:="info". That unpack overwrote the stored level field before the stats query read it, which caused two errors pulling in opposite directions:

Dropped: lines in an info stream whose body says level=error or level=warn were dropped, and so were lines whose level comes only from a JSON body or from a stored spelling such as Information;
Added: lines in another stream whose body says level=info were counted as info.

Fix: the filter now stays out of the pushdown query. The stored-level filter and the stored level fields decide the level, and rows without a stored level go to the level chain. The answer matches Loki, and the shape is several times faster. At 7d it answers where main timed out.

Changes

internal/proxy/ordered_json_metric.go: the ordered-JSON route (range and instant, one or several detected_level filters before | json) keeps the filter out of the pushdown query and reads the level from the stored fields.
- The parse-risk probes run on the selector plus the stored-level filter.
- The series limit and the raw-row fallback are the route's existing ones.
- Instant queries are answered from stats buckets.
Unchanged, still using the logfmt rendering (follow-ups):
- | logfmt plans;
- the withoutJSON native route (for example sum(count_over_time({...} | detected_level="info" | json | drop __error__ [1m])), or sum by (app) with app in the selector);
- the native Drilldown histogram.
With -detected-level-body-scan=false:
- a plan that reads the level only in this filter keeps the translator's rendering on the stats path;
- a plan that also groups by level or detected_level, and any window the raw-row evaluator answers, uses the stored-level filter and a level derived from stored fields only.
Tests (ordered_json_filter_pushdown_test.go):
- detectedLevelFilterFixture now includes a stored-debug stream whose body says level=info;
- TestOrderedJSONDetectedLevelFilterBeforeParserIgnoresMessageLevelToken asserts info=3 and debug=2 with no raw rows. It fails on main, which gives info=2 and error=1;
- TestOrderedJSONDetectedLevelFilterBeforeParserTwoFiltersAndInstant covers two filters, with and without stored levels, on range and instant.
Registry: a new case, semantics/detected-level-filter-before-parser-pushdown; label-filter-before-parser-pushdown and parser-error-and-label-collision are qualified; translations and reports are regenerated.
CHANGELOG: Performance and Fixed entries.

Proofs

Base ee900031 (main before #790; there is no proxy code change between it and the current main). Branch rebased onto 4a872d66 with only the CHANGELOG in conflict.

e2e vs Loki 3.7.7. Fresh isolated stack per run, compared with main by test name; VictoriaLogs restarts 0 → 0 in every run:

VictoriaLogs	group	branch	main
v1.52.0	semantics	903 passed / 0 failed / 3 skipped	903 / 0 / 3
v1.52.0	core	2116 / 2 / 3	2116 / 2 / 3, the same two failures
v1.49.0	semantics	903 / 0 / 3	—

The two core failures are one vmauth backend-version subtest on the local stack, identical on main.

A/B with Loki at 1h (bench/ab/pr_smoke.py, VictoriaLogs v1.52.0, refresh mode on). 77 shapes; none slower beyond noise; 0 new differences from Loki (62 same, 15 pre-existing on main too):

shape	main p50	branch p50	Loki p50
DL6, Explore volume with a level filter before json	94 ms	25 ms	37 ms

A/B at long ranges. VictoriaLogs only, 7 days seeded (607,860 s, 8-day retention), VictoriaLogs at 2 CPUs; 58 shapes, none slower:

shape	range	main	branch
DL6	6h	1.72 s	432 ms
DL6	24h	6.72 s	1.68 s
DL6	7d	HTTP 504	12.5 s
DL6, data without a stored level label	1h	293 ms	62 ms
DL6, data without a stored level label	24h	6.86 s	1.62 s

DL1–DL4, the json-filter-pushdown set and the control set all stay within noise. DL3 at 7d returns 502 on both builds.

Locally, Loki is OOM-killed by a 24h backfill. For a Loki reference at 24h, the daily run measures this shape at 52 ms warm and 4.19 s cold.

Grafana range matrix. Grafana 13.2.3, 47 pages (46 plus the fix-proof page) at 5m, 15m, 1h, 6h, 24h and 7d, plus Live tail. VictoriaLogs holds 7 days (48,253,273 lines); Loki holds the last 1.5 h. Main and branch are identical everywhere except four page groups:

The fix, explore-DL6: at 5m, 15m and 1h main differs from Loki on the info/info series (for example 442 against Loki's 444), while the branch matches Loki exactly. At 6h–7d the branch's values differ from main's in the same way.
dd-fields 7d and explore-extracted-filter-metric 7d: one-sided requests and an unsettled 7d capture (the known Grafana breakdown and timeout behaviour).
explore-logs-plain 24h/7d: the label list differs on the Grafana side, depending on earlier captures. Queried directly with identical requests, /labels returns the same 15 labels on both builds.

Against Loki: 0 new differences across all pages.

Fix proof, the Explore volume with detected_level="info" before | json (main, branch, Loki):

range
1h
24h
7d

benchstat. 19 benchmarks covering ordered JSON, detected_level and stats, 10 interleaved rounds, GOMAXPROCS=4: no significant change (time geomean −0.09%, B/op −0.05%, allocations identical).

Load test. 8 clients, 240 s, the control, DL6 and json-filter sets, 7 days of data, a fresh proxy per build; two runs:

	run 1 main	run 1 branch	run 2 main	run 2 branch
requests	49	58	52	54
200 / 429 / 504 / 502	24 / 12 / 9 / 4	29 / 22 / 6 / 1	29 / 16 / 5 / 2	23 / 22 / 7 / 2
`/labels`	61/61 OK	64/64 OK	75/75 OK	57/57 OK
proxy RSS peak / after	237 / 283 MB	416 / 266 MB	581 / 455 MB	455 / 463 MB
VictoriaLogs RSS peak	5,371 MB	5,315 MB	5,350 MB	7,508 MB
VictoriaLogs restarts	0 → 0	0 → 0	0 → 0	0 → 0

The heavy 7d shapes saturate this stack on both builds, so the 504s and the 429s appear on both, and the RSS peaks swing in both directions between runs. There is no consistent difference and no steady-state growth. The stats path buffers nothing new: its group-by fields are the same as main's, the level filter now runs inside VictoriaLogs, and responses are capped at 64 MiB.

Unit and gates:

capped full run GOMEMLIMIT=3GiB GOMAXPROCS=4 go test -short -p 1 ./...: 13/13 packages, peak RSS 1.54 GB;
-race on the touched tests (117);
gofmt, go vet (including the e2e tag), golangci-lint 0 issues;
the conformance gate (generated files regenerated with no diff) and the changelog gate.

Follow-ups, not in this change:

the routes still using the logfmt rendering, listed above;
the raw fallback now fetches every row without a stored level (a deliberate superset), so on tenants with no level label a probe-triggered fallback reaches -manual-range-metric-row-limit sooner;
the local vmauth version subtest.

… level fields The Explore logs volume with a level selected (`sum by (level, detected_level) (count_over_time({...} | detected_level="info" | json | drop __error__ [1m]))`) rendered the filter before the parser as a log query does, `| unpack_logfmt | filter level:="info"`, so VictoriaLogs parsed every row of the window as logfmt in the stats query and in each parse-risk probe, and a `level=` token of a message overwrote the stored level. Lines were counted under the wrong level in both directions: a line in an info stream whose body said level=error was dropped from the info count, and a line in a debug stream whose body said level=info was counted as info. detected_level is assigned at ingest and `| json` cannot change it, so the filter keeps the same lines before the parser as after it. The ordered-JSON metric route, range and instant, now applies a filter on detected_level before the first parser like the one after it: from the stored level fields, or the level chain when they do not decide, with the stats pushdown kept. The stats query starts from the selector and the line filters, the parse-risk probes run on the selector plus the stored-level filter, and the raw-row evaluator, when a probe sends a window to it, fetches the same narrowed rows and applies the filter to the level it derives for each. Plans with `| logfmt`, the route that skips `| json` when it cannot affect the aggregation and the native Drilldown histogram still render the filter as the logfmt unpack. With -detected-level-body-scan=false a plan that reads the level only in this filter keeps the translator's rendering on the stats path; a plan that also groups by level or detected_level, and any window the raw-row evaluator answers, use the stored-level filter and a level derived from the stored fields only. Other label filters before the parser and the translator's log-query rendering are unchanged.

github-actions · 2026-10-21T17:55:03Z

🟢 Performance A/B: faster, no regression

77 shape×range — 1 🟢 faster · 76 ⚪ same · results vs Loki: 62 same, 0 new difference(s), 15 pre-existing

	shape	range	base p50	PR p50	change	Loki p50	PR ÷ Loki	result vs Loki
🟢	DL6 explore volume, level filter before json	1h	550 ms	145 ms ¹	🟢 -74%	49 ms	2.96×	✅ same

¹ re-measured with 7 runs, on windows the first pass did not use, after it moved beyond noise.

76 unchanged shape×range (within noise)

	shape	range	base p50	PR p50	change	Loki p50	PR ÷ Loki	result vs Loki
⚪	bytes_over_time by app	1h	70 ms	78 ms	⚪ +11%	98 ms	0.80×	✅ same
⚪	drilldown service landing	1h	37 ms	32 ms	⚪ -14%	34 ms	0.94×	✅ same
⚪	instant json filtered	instant	11 ms	13 ms	⚪ +18%	76 ms	0.17×	✅ same
⚪	instant json volume	instant	34 ms	34 ms	⚪ +0%	101 ms	0.34×	✅ same
⚪	instant ungrouped rate	instant	6 ms	6 ms	⚪ +0%	77 ms	0.08×	✅ same
⚪	json rate by level (no filter, no drop)	1h	400	400	⚪ +0%	400	—	✅ same
⚪	json volume no filters	1h	206 ms	205 ms	⚪ -0%	63 ms	3.25×	✅ same
⚪	logfmt count by level, filter, drop	1h	485 ms	450 ms	⚪ -7%	65 ms	6.92×	✅ same
⚪	logfmt volume	1h	230 ms	225 ms	⚪ -2%	74 ms	3.04×	✅ same
⚪	logs plain	1h	72 ms	68 ms	⚪ -6%	47 ms	1.45×	✅ same
⚪	logs \| json	1h	98 ms	105 ms	⚪ +7%	67 ms	1.57×	✅ same
⚪	logs \| logfmt	1h	168 ms	184 ms	⚪ +10%	57 ms	3.23×	✅ same
⚪	stream-label count by app	1h	31 ms	26 ms	⚪ -16%	48 ms	0.54×	✅ same
⚪	topk rate by app	1h	45 ms	43 ms	⚪ -4%	97 ms	0.44×	➖ differs (as base)
⚪	ungrouped rate	1h	12 ms	13 ms	⚪ +8%	91 ms	0.14×	✅ same
⚪	DB3 json field breakdown user_id	1h	47 ms	49 ms	⚪ +4%	38 ms	1.29×	✅ same
⚪	DB4 json field breakdown batch_id	1h	101 ms	94 ms	⚪ -7%	69 ms	1.36×	✅ same
⚪	X1 json filter level_extracted, by level	1h	393 ms	391 ms	⚪ -1%	49 ms	7.98×	✅ same
⚪	X2 json group by level_extracted	1h	328 ms	332 ms	⚪ +1%	53 ms	6.26×	✅ same
⚪	X3 json rate, level_extracted equality	1h	385 ms	304 ms	⚪ -21%	47 ms	6.47×	✅ same
⚪	X4 json bytes, level_extracted regexp	1h	402 ms	401 ms	⚪ -0%	99 ms	4.05×	✅ same
⚪	X5 logfmt filter level_extracted, by level	1h	693 ms	677 ms	⚪ -2%	59 ms	11×	✅ same
⚪	E1 logs, json parser over stream labels	1h	121 ms	123 ms	⚪ +2%	67 ms	1.84×	✅ same
⚪	E2 logs, logfmt parser over stream labels	1h	78 ms	84 ms	⚪ +8%	54 ms	1.56×	✅ same
⚪	E3 logs, json, existence filter on level_extracted	1h	161 ms	169 ms	⚪ +5%	76 ms	2.22×	✅ same
⚪	E4 logs, logfmt, existence filter on level_extracted	1h	102 ms	104 ms	⚪ +2%	64 ms	1.62×	✅ same
⚪	E5 logs, json, drop level_extracted	1h	130 ms	121 ms	⚪ -7%	63 ms	1.92×	✅ same
⚪	E6 logfmt count by level_extracted	1h	49 ms	48 ms	⚪ -2%	9 ms	5.33×	✅ same
⚪	E7 json count by level_extracted	1h	148 ms	161 ms	⚪ +9%	22 ms	7.32×	✅ same
⚪	E8 instant logfmt by level_extracted	instant	8 ms	8 ms	⚪ +0%	9 ms	0.89×	✅ same
⚪	A explore volume, 2 filters, drop	1h	137 ms	148 ms	⚪ +8%	221 ms	0.67×	✅ same
⚪	B grouped sum, filter, no drop	1h	103 ms	103 ms	⚪ +0%	42 ms	2.45×	✅ same
⚪	C ungrouped sum, filter, no drop	1h	101 ms	102 ms	⚪ +1%	43 ms	2.37×	✅ same
⚪	D volume, no filters	1h	220 ms	226 ms	⚪ +3%	57 ms	3.96×	✅ same
⚪	E field breakdown pipeline (explore)	1h	103 ms	103 ms	⚪ +0%	47 ms	2.19×	✅ same
⚪	F field breakdown service_version (explore)	1h	138 ms	148 ms	⚪ +7%	52 ms	2.85×	✅ same
⚪	G field breakdown pipeline (drilldown)	1h	45 ms	52 ms	⚪ +16%	69 ms	0.75×	✅ same
⚪	H field breakdown service_version (drilldown)	1h	108 ms	112 ms	⚪ +4%	40 ms	2.80×	✅ same
⚪	I grafana volume of B	1h	102 ms	101 ms	⚪ -1%	43 ms	2.35×	✅ same
⚪	O2 explore log volume (Grafana form), underscore structured-metadata filter	1h	70 ms	69 ms	⚪ -1%	45 ms	1.53×	✅ same
⚪	DL1 drilldown field breakdown json-expr, level filter	1h	500 ms	516 ms	⚪ +3%	35 ms	15×	✅ same
⚪	DL2 drilldown field breakdown, level + stream-label filter before json	1h	47 ms	42 ms	⚪ -11%	8 ms	5.25×	✅ same
⚪	DL3 drilldown logfmt field breakdown, level filter	1h	628 ms	670 ms	⚪ +7%	109 ms	6.15×	➖ differs (as base)
⚪	DL4 instant count, level filter, json+logfmt, 1h	instant	506 ms	508 ms	⚪ +0%	1.26 s	0.40×	✅ sum ±0.26%
⚪	P1 logs, json, level equals	1h	160 ms	188 ms	⚪ +18%	109 ms	1.72×	✅ same
⚪	P2 logs, logfmt, level not equal	1h	31 ms	31 ms	⚪ +0%	7 ms	4.43×	✅ same
⚪	P3 logs, json, level existence	1h	148 ms	139 ms	⚪ -6%	75 ms	1.85×	✅ same
⚪	P4 logfmt count by level	1h	31 ms	35 ms	⚪ +13%	7 ms	5.00×	✅ same
⚪	P5 json count by level	1h	81 ms	88 ms	⚪ +9%	16 ms	5.50×	✅ same
⚪	P6 json count, level filter	1h	15 ms	15 ms	⚪ +0%	15 ms	1.00×	✅ same
⚪	P7 json rate by level, level regexp	1h	17 ms	19 ms	⚪ +12%	16 ms	1.19×	✅ same
⚪	P8 json count by a name that is no stream label	1h	87 ms	86 ms	⚪ -1%	15 ms	5.73×	✅ same
⚪	P9 instant logfmt by level	instant	6 ms	6 ms	⚪ +0%	9 ms	0.67×	✅ same
⚪	E1 json duration_ms sum by service	1h	84 ms	89 ms	⚪ +6%	20 ms	4.45×	➖ differs (as base)
⚪	E2 json duration_ms p95 by service	1h	252 ms	249 ms	⚪ -1%	16 ms	16×	✅ same
⚪	E3 logfmt latency_ms rate, production	1h	496 ms	502 ms	⚪ +1%	135 ms	3.72×	➖ differs (as base)
⚪	E4 logfmt size_bytes max by service	1h	47 ms	48 ms	⚪ +2%	16 ms	3.00×	➖ differs (as base)
⚪	E5 json duration_ms sum, instant	instant	36 ms	41 ms	⚪ +14%	29 ms	1.41×	✅ same
⚪	E6 logfmt ttl unit string (Loki 400)	1h	400	400	⚪ +2%	400	—	✅ same
⚪	E7 json hexadecimal trace_id (Loki 400)	1h	400	400	⚪ +2%	400	—	✅ same
⚪	E8 json duration_ms sum by service, namespace prod	1h	187 ms	190 ms	⚪ +2%	93 ms	2.04×	➖ differs (as base)
⚪	E9 logfmt latency_ms sum by service, namespace prod	1h	308 ms	308 ms	⚪ +0%	96 ms	3.21×	➖ differs (as base)
⚪	U1 json duration_ms sum by service	1h	84 ms	82 ms	⚪ -2%	20 ms	4.10×	➖ differs (as base)
⚪	U10 instant sum_over_time unwrap	instant	32 ms	32 ms	⚪ +0%	29 ms	1.10×	✅ same
⚪	U11 instant max_over_time missing label	instant	5 ms	5 ms	⚪ +0%	15 ms	0.33×	✅ same
⚪	U12 json duration_ms over every service	1h	235 ms	238 ms	⚪ +1%	53 ms	4.49×	➖ differs (as base)
⚪	U13 logfmt latency_ms over every service	1h	498 ms	498 ms	⚪ +0%	75 ms	6.64×	➖ differs (as base)
⚪	U14 duration() of a unit string	1h	45 ms	48 ms	⚪ +7%	7 ms	6.86×	➖ differs (as base)
⚪	U2 json duration_ms max by service	1h	91 ms	93 ms	⚪ +2%	39 ms	2.38×	➖ differs (as base)
⚪	U3 json duration_ms sliding sum	1h	86 ms	81 ms	⚪ -6%	37 ms	2.19×	➖ differs (as base)
⚪	U4 logfmt size_bytes sum by service	1h	39 ms	40 ms	⚪ +3%	8 ms	5.00×	➖ differs (as base)
⚪	U5 logfmt ttl unit string (no sample)	1h	39 ms	40 ms	⚪ +3%	7 ms	5.71×	✅ same
⚪	U6 missing label, no parser	1h	25 ms	25 ms	⚪ +0%	11 ms	2.27×	✅ same
⚪	U7 missing label, ungrouped sum	1h	18 ms	18 ms	⚪ +0%	10 ms	1.80×	✅ same
⚪	U8 rate of an unwrapped label	1h	86 ms	85 ms	⚪ -1%	16 ms	5.31×	✅ same
⚪	U9 bytes() conversion	1h	53 ms	53 ms	⚪ +0%	12 ms	4.42×	➖ differs (as base)

Cold (first-run) timings — what a user opening a panel sees

shape	range	base cold	PR cold	Loki cold	PR ÷ Loki cold
DL6 explore volume, level filter before json	1h	563 ms	162 ms	47 ms †	3.45×
bytes_over_time by app	1h	70 ms	76 ms	711 ms	0.11×
drilldown service landing	1h	38 ms	35 ms	651 ms	0.05×
instant json filtered	instant	10 ms	10 ms	77 ms	0.13×
instant json volume	instant	34 ms	41 ms	104 ms	0.39×
instant ungrouped rate	instant	9 ms	6 ms	79 ms	0.08×
json rate by level (no filter, no drop)	1h	400	400	400	—
json volume no filters	1h	207 ms	206 ms	1.06 s ↺	0.19×
logfmt count by level, filter, drop	1h	450 ms	460 ms	1.38 s	0.33×
logfmt volume	1h	241 ms	231 ms	1.22 s	0.19×
logs plain	1h	85 ms	84 ms	60 ms	1.40×
logs \| json	1h	122 ms	125 ms	63 ms	1.98×
logs \| logfmt	1h	190 ms	193 ms	55 ms	3.51×
stream-label count by app	1h	26 ms	26 ms	646 ms	0.04×
topk rate by app	1h	55 ms	51 ms	731 ms	0.07×
ungrouped rate	1h	12 ms	11 ms	665 ms	0.02×
DB3 json field breakdown user_id	1h	56 ms	58 ms	571 ms	0.10×
DB4 json field breakdown batch_id	1h	96 ms	96 ms	609 ms	0.16×
X1 json filter level_extracted, by level	1h	3.54 s	3.52 s	951 ms	3.70×
X2 json group by level_extracted	1h	308 ms	308 ms	760 ms	0.41×
X3 json rate, level_extracted equality	1h	311 ms	286 ms	565 ms	0.51×
X4 json bytes, level_extracted regexp	1h	413 ms	432 ms	649 ms	0.67×
X5 logfmt filter level_extracted, by level	1h	670 ms	668 ms	1.15 s	0.58×
E1 logs, json parser over stream labels	1h	146 ms	141 ms	74 ms	1.91×
E2 logs, logfmt parser over stream labels	1h	76 ms	77 ms	56 ms	1.38×
E3 logs, json, existence filter on level_extracted	1h	174 ms	164 ms	75 ms	2.19×
E4 logs, logfmt, existence filter on level_extracted	1h	89 ms	96 ms	67 ms	1.43×
E5 logs, json, drop level_extracted	1h	124 ms	115 ms	64 ms	1.80×
E6 logfmt count by level_extracted	1h	46 ms	61 ms	56 ms	1.09×
E7 json count by level_extracted	1h	1.41 s	1.41 s	227 ms	6.23×
E8 instant logfmt by level_extracted	instant	8 ms	10 ms	9 ms	1.11×
A explore volume, 2 filters, drop	1h	1.38 s	1.40 s	1.52 s	0.92×
B grouped sum, filter, no drop	1h	125 ms	128 ms	1.69 s	0.08×
C ungrouped sum, filter, no drop	1h	123 ms	118 ms	708 ms	0.17×
D volume, no filters	1h	213 ms	212 ms	1.06 s	0.20×
E field breakdown pipeline (explore)	1h	111 ms	111 ms	724 ms	0.15×
F field breakdown service_version (explore)	1h	1.38 s	1.38 s	833 ms	1.65×
G field breakdown pipeline (drilldown)	1h	43 ms	45 ms	719 ms	0.06×
H field breakdown service_version (drilldown)	1h	119 ms	122 ms	802 ms	0.15×
I grafana volume of B	1h	111 ms	112 ms	715 ms	0.16×
O2 explore log volume (Grafana form), underscore structured-metadata filter	1h	159 ms	146 ms	866 ms	0.17×
DL1 drilldown field breakdown json-expr, level filter	1h	547 ms	491 ms	694 ms	0.71×
DL2 drilldown field breakdown, level + stream-label filter before json	1h	86 ms	71 ms	29 ms	2.45×
DL3 drilldown logfmt field breakdown, level filter	1h	712 ms	678 ms	775 ms	0.87×
DL4 instant count, level filter, json+logfmt, 1h	instant	509 ms	511 ms	1.47 s	0.35×
P1 logs, json, level equals	1h	161 ms	163 ms	86 ms	1.90×
P2 logs, logfmt, level not equal	1h	32 ms	31 ms	36 ms	0.86×
P3 logs, json, level existence	1h	143 ms	129 ms	78 ms	1.65×
P4 logfmt count by level	1h	31 ms	32 ms	43 ms	0.74×
P5 json count by level	1h	91 ms	91 ms	174 ms	0.52×
P6 json count, level filter	1h	15 ms	15 ms	159 ms	0.09×
P7 json rate by level, level regexp	1h	17 ms	23 ms	179 ms	0.13×
P8 json count by a name that is no stream label	1h	99 ms	95 ms	143 ms	0.66×
P9 instant logfmt by level	instant	7 ms	6 ms	9 ms	0.67×
E1 json duration_ms sum by service	1h	94 ms	88 ms	178 ms	0.49×
E2 json duration_ms p95 by service	1h	245 ms	240 ms	174 ms	1.38×
E3 logfmt latency_ms rate, production	1h	503 ms	499 ms	867 ms	0.58×
E4 logfmt size_bytes max by service	1h	44 ms	47 ms	236 ms	0.20×
E5 json duration_ms sum, instant	instant	39 ms	37 ms	29 ms	1.28×
E6 logfmt ttl unit string (Loki 400)	1h	400	400	400	—
E7 json hexadecimal trace_id (Loki 400)	1h	400	400	400	—
E8 json duration_ms sum by service, namespace prod	1h	192 ms	185 ms	556 ms	0.33×
E9 logfmt latency_ms sum by service, namespace prod	1h	304 ms	306 ms	631 ms	0.48×
U1 json duration_ms sum by service	1h	93 ms	82 ms	152 ms	0.54×
U10 instant sum_over_time unwrap	instant	52 ms	35 ms	29 ms	1.21×
U11 instant max_over_time missing label	instant	5 ms	5 ms	15 ms	0.33×
U12 json duration_ms over every service	1h	239 ms	233 ms	567 ms	0.41×
U13 logfmt latency_ms over every service	1h	557 ms	607 ms	878 ms	0.69×
U14 duration() of a unit string	1h	47 ms	44 ms	50 ms	0.88×
U2 json duration_ms max by service	1h	92 ms	92 ms	975 ms	0.09×
U3 json duration_ms sliding sum	1h	82 ms	85 ms	192 ms	0.44×
U4 logfmt size_bytes sum by service	1h	41 ms	40 ms	37 ms	1.08×
U5 logfmt ttl unit string (no sample)	1h	40 ms	40 ms	38 ms	1.05×
U6 missing label, no parser	1h	19 ms	24 ms	73 ms	0.33×
U7 missing label, ungrouped sum	1h	17 ms	17 ms	60 ms	0.28×
U8 rate of an unwrapped label	1h	128 ms	85 ms	159 ms	0.53×
U9 bytes() conversion	1h	50 ms	55 ms	41 ms	1.34×

† Loki's own counters say its results cache answered this cold run: not a computed time.

↺ Loki cold taken from the first set that ran the same query; Loki's cache answered this one.

Why these shapes

internal/proxy/ordered_json_metric.go → 73: control/bytes_over_time by app, control/instant json filtered, control/instant json volume, control/json rate by level (no filter, no drop), control/json volume no filters, control/logfmt count by level, filter, drop, control/logfmt volume, control/logs | json …
control smoke → 8: control/drilldown service landing, control/instant ungrouped rate, control/json rate by level (no filter, no drop), control/json volume no filters, control/logs plain, control/logs | json, control/stream-label count by app, control/topk rate by app

_{base 047b0ff597 → PR c922a6f90c · 4 runs (first is cold), warm p50 · ranges 1h · noise ±30% and 100 ms · VictoriaLogs restarts 0→0 · 9.3 min. Same runner, one fresh stack, targets interleaved per request. How to read and reproduce: bench/ab/README.md.}

github-actions · 2026-10-21T17:55:58Z

PR Quality Report

Compared against base branch main.

Coverage and tests

Signal	Base	PR	Delta
Test count	11763	11826	63
Coverage	88.3%	88.4%	+0.1% (improved)

Compatibility

Track	Base	PR	Delta
Loki API	100.0%	11/11 (100.0%)	0.0% (stable)
Logs Drilldown	100.0%	17/17 (100.0%)	0.0% (stable)
VictoriaLogs	100.0%	11/11 (100.0%)	0.0% (stable)

Performance smoke

Lower CPU cost (ns/op) is better. Lower benchmark memory cost (B/op, allocs/op) is better. Higher throughput is better. Lower load-test memory growth is better. Benchmark rows are medians from repeated samples.

Signal	Base	PR	Delta
QueryRange cache-hit CPU cost	3457.0 ns/op	3566.0 ns/op	+3.2% (stable)
QueryRange cache-hit memory	1240.0 B/op	1240.0 B/op	0.0% (stable)
QueryRange cache-hit allocations	27.0 allocs/op	27.0 allocs/op	0.0% (stable)
QueryRange cache-bypass CPU cost	4131.0 ns/op	4137.0 ns/op	+0.1% (stable)
QueryRange cache-bypass memory	1303.0 B/op	1306.0 B/op	+0.2% (stable)
QueryRange cache-bypass allocations	27.0 allocs/op	27.0 allocs/op	0.0% (stable)
Labels cache-hit CPU cost	929.3 ns/op	928.0 ns/op	-0.1% (stable)
Labels cache-hit memory	48.0 B/op	48.0 B/op	0.0% (stable)
Labels cache-hit allocations	3.0 allocs/op	3.0 allocs/op	0.0% (stable)
Labels cache-bypass CPU cost	1520.0 ns/op	1520.0 ns/op	0.0% (stable)
Labels cache-bypass memory	400.0 B/op	399.0 B/op	-0.2% (stable)
Labels cache-bypass allocations	14.0 allocs/op	14.0 allocs/op	0.0% (stable)

State

Coverage, compatibility, and sampled performance are reported here from the same PR workflow.
This is a delta report, not a release gate by itself. Required checks still decide merge safety.
Performance is a smoke comparison, not a full benchmark lab run.
Delta states use the same noise guards as the quality gate (percent + absolute + low-baseline checks), so report labels match merge-gate behavior.

github-actions · 2026-10-21T18:04:14Z

✅ Visual smoke: passed

Base 047b0ff597 vs PR 25988a033f, Grafana through the proxy builds and Loki on one fresh stack; run.
Core set: 5 capture(s) at 1h. Detailed set: 21 entries (explore-metric-plain, explore-A, explore-B, explore-C, explore-E, explore-F, explore-logs-json, explore-logfmt-metric, ...). 27 capture(s), 15m/1h/6h, 24h and 7d stay out of PR runs.
Trimmed to the 1h range for the capture budget: explore-A, explore-B, explore-C, explore-E, explore-F, explore-logs-json, explore-logfmt-metric, explore-logs-extracted, explore-extracted-metric, explore-extracted-filter-metric, explore-extracted-group-metric, explore-logs-plain-level, explore-plain-level-metric, explore-unwrap-sum, explore-unwrap-unit-string, explore-unwrap-missing-label, explore-unwrap-conversion-error, dd-field-service_version, dd-field-pipeline.

page	range	set	base = PR (data)	vs Loki (PR)	pixel diff	result
dd-fields	1h	core	✅ identical	identical (4 explained, 1 history-dependent)	0.12%	✅
dd-landing	1h	core	✅ identical	identical (3 explained)	0.14%	✅
dd-service-logs	1h	core	✅ identical	identical (3 explained, 1 history-dependent)	0.12%	✅
explore-logs-plain	1h	core	✅ identical	identical (1 explained)	0.07%	✅
explore-metric-plain	1h	core	✅ identical	identical (1 explained)	0.04%	✅
dd-field-pipeline	1h	detailed	✅ identical	identical (3 explained, 1 history-dependent)	0.12%	✅
dd-field-service_version	1h	detailed	✅ identical	identical (3 explained, 1 history-dependent)	0.12%	✅
explore-A	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-B	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-C	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-E	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-F	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-extracted-filter-metric	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-extracted-group-metric	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-extracted-metric	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-logfmt-metric	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-logs-extracted	1h	detailed	✅ identical	identical (1 explained)	0.07%	✅
explore-logs-json	1h	detailed	✅ identical	identical (1 explained)	0.07%	✅
explore-logs-plain-level	1h	detailed	✅ identical	1 difference(s), 1 on base too; 1 explained	0.07%	✅
explore-metric-plain	15m	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-metric-plain	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-plain-level-metric	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-conversion-error	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-missing-label	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-sum	1h	detailed	✅ identical	2 difference(s), 2 on base too	0.04%	✅
explore-unwrap-unit-string	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
tail-json-filter	live	detailed	✅ identical	entries in the window: PR 1898, Loki 590, in both 590	6.96%	✅

Gate: a base-vs-PR data difference passes when the PR is closer to Loki, and fails when the PR diverges from Loki or Loki cannot decide it (at a range Loki does not hold, a difference passes when the same page proved it closer to Loki at a shorter range; otherwise label the pull request visual-change-expected to accept an intended change). Also failing: a panel empty on the PR but not on the base, an error banner, panel error, error answer or new "No data" panel on the PR, a PR side that never settled, a difference that did not reproduce on the recapture. Pixel differences only warn.

Core set montages (5): base | PR | Loki

dd-fields-1h

dd-landing-1h

dd-service-logs-1h

explore-logs-plain-1h

explore-metric-plain-1h

All montages of this run: pr-visuals/pr-795.

The release workflow could not create the metadata branch for v2.24.1 (it was cut from #794 while main had already moved on). Materialize the published 2.24.1 notes as the [2.24.1] section (#792's CI entry and #794's Security entry, exactly as released) and set the chart to 2.24.1; [Unreleased] keeps the entries merged since (#795, #796, #797). Co-authored-by: szibis <szibis@users.noreply.github.com>

github-actions Bot added size/XL Extra large change scope/proxy Proxy core scope/docs Documentation scope/tests Tests performance Performance labels Oct 21, 2026

szibis force-pushed the perf/detected-level-filter-before-json branch from da0b9c8 to 25988a0 Compare October 21, 2026 17:53

github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 21, 2026

szibis merged commit e069c37 into main Oct 21, 2026
36 checks passed

szibis deleted the perf/detected-level-filter-before-json branch October 21, 2026 17:59

szibis mentioned this pull request Oct 21, 2026

docs: sync release metadata for v2.24.1 #799

Merged

Navigation Menu

Repository navigation

perf: a detected_level filter before json reads the stored level fields, matching Loki - #795

perf: a detected_level filter before json reads the stored level fields, matching Loki#795
szibis merged 1 commit into
mainfrom
perf/detected-level-filter-before-json

szibis commented Oct 21, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 21, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 21, 2026 •

edited

Loading

Uh oh!

Uh oh!

github-actions Bot commented Oct 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

szibis commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Proofs

Uh oh!

github-actions Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Performance A/B: faster, no regression

Uh oh!

github-actions Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Quality Report

Coverage and tests

Compatibility

Performance smoke

State

Uh oh!

Uh oh!

github-actions Bot commented Oct 21, 2026

✅ Visual smoke: passed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

szibis commented Oct 21, 2026 •

edited

Loading

github-actions Bot commented Oct 21, 2026 •

edited

Loading

github-actions Bot commented Oct 21, 2026 •

edited

Loading