fix: absent_over_time is the series valued 1 at the steps without data, after a parser, under an aggregation and on Loki's step grid by szibis · Pull Request #737 · ReliablyObserve/loki-vl-proxy

szibis · 2026-10-12T11:06:44Z

Summary

Gap semantics/absent-over-time-semantics. On Loki 3.7.7, absent_over_time returns one series valued 1 at each step whose window has no sample. Its labels come from the selector's equality matchers, at most one per name. It returns nothing at steps with data. Sources: pkg/logql/evaluator.go:647-664, :754-778, :1320-1345.

Causes in the proxy:

With a parser stage, absent_over_time was taken by the raw-row bare-parser evaluator. It answered thousands of series where Loki answers none.
The absent handler read VictoriaLogs buckets [T, T+step) as Loki windows (T-range, T], so a tumbling answer was one step early.
- It ignored a range different from the step.
- It stepped on the epoch grid; Loki steps from start.
- Its instant form had no time bounds.
sum(absent_over_time(...)) and the other vector aggregations returned the inner count.

Fix (new internal/proxy/absent_over_time.go):

VictoriaLogs answers whether each step has data. The query goes through the normal metric path as sum(count_over_time(...)), or sum(rate(...)) with unwrap, so parser stages, label filters and Loki's windows are applied upstream.
The proxy fills the steps without data with 1, on start + k*step, with the matcher-derived labels.
Vector aggregations fold the single series as Loki does.
A degraded inner answer (an error status, a warning, a Drilldown partial answer) is passed on, never inverted.
An unwrapped value Loki rejects now fails with Loki's 400 pipeline error; it used to answer 200.

Tests

TestAbsentOverTimeDifferential, against a Loki reference evaluator. It covers:
- selector, line filter, logfmt, json, label filter and unwrap;
- range and instant;
- offset and millisecond starts;
- VictoriaLogs v1.40.0 and v1.52.0.
TestAbsentOverTimeUnderVectorAggregation, with 14 aggregation forms.
Edge tables for labels, spec parsing, series folding, the registry row's queries and degraded answers.
New conversion-error rows.

All of these failed before the fix.

e2e vs Loki: test/e2e-compat/absent_over_time_test.go, with fixtures 21h back in two bursts separated by a gap.

absent_over_time is removed from the lists of conversion-error shapes that aren't detected: the registry row, KNOWN_ISSUES and the translation reference.

Verification on rebased head

Rebased onto main at 14e13dc6e5cea0222aa51f8bb06f67d6348b2889 after the v2.46.0 release; all twelve PR commits are signed. Current head: effafa8e2e4ec297c9500eb4a5adac5df5b9dcda.

Current-head local conformance gate passed (359 registry items, 228 tests declaring coverage, 190 wired). Focused TestAbsentOverTime* Go tests, all 155 visual harness tests, 307 CI configuration tests, and actionlint passed. Full proxy and translator tests passed on the preceding HEAD before the release-only rebase (319.894 s and 8.096 s). On the prior 231-capture fix: absent_over_time is the series valued 1 at the steps without data, after a parser, under an aggregation and on Loki's step grid #737 artifact, the trusted v2.46.0 base renderer selected exactly 15 inline montages; the new workflow's bounded selector staged those 15 and the trusted v2.46.0 publisher accepted all. Tests cover foreign URLs, path traversal, symlinks and oversized files.
Current-head CI completed with 72 successful checks, no failures, one neutral advisory scan, and four expected skips. Full Go tests, race, fuzz, e2e, Loki/VictoriaLogs/Grafana compatibility, performance A/B, quality, and security all passed. The runtime security job passed on rerun after an initial ZAP daemon startup timeout; the source SHA did not change.
Current-head visual run passed all capture, publisher, and comment jobs: 237 main | PR | Loki captures, with three open inline fix montages in the CI bot comment. The 15m and 1h cases improve toward Loki; one unrelated difference shared with the base remains in each, so the visual verdict is correctly “improved,” not “fixed.” A 1h PNG returned HTTP 200 image/png (30,552 bytes) from the published branch. The full baseline candidate remains in the run artifact.
Before this rebase, local full proxy/translator tests and a fresh Loki/VictoriaLogs TestCompat_AbsentOverTime* e2e stack passed. The prior HEAD also passed visual smoke, performance A/B, and quality report. These are historical controls; the current-head jobs above are the acceptance proof.

Main | PR | Loki visual comparison

Current capture from main 14e13dc6e5 vs PR effafa8e2e. At 1h, the base showed “No data”; the PR and Loki show the absence series at 1. The image URL uses immutable pr-visuals commit e610898290ddf9a89379e92a7b9b6357dde1c800.

The current 15m and 6h montages are also published; the full CI visual comment embeds 18 unique comparison images.

github-actions · 2026-10-12T11:08:20Z

⚪ Performance A/B: no change beyond noise

264 shape×range — 264 ⚪ same · results vs Loki: 242 same, 0 new difference(s), 22 pre-existing

Loki profiles compared (like for like):

aligned: not run (no build supports -align-queries-with-step, so every aligned row would be not like-for-like)
loki-defaults: 264 shape×range; proxy flag base off, pr off

Every shape is within noise of the base build (±30% and 100 ms).

264 unchanged shape×range (within noise)

	shape	range	Loki profile	base p50	PR p50	change	Loki p50	PR ÷ Loki	result vs Loki
⚪	B1 sum by over rate divided by a literal	1h	defaults	5 ms	5 ms	⚪ +0%	31 ms	0.16×	✅ same
⚪	B2 sum by over a ratio of two per-stream metrics	1h	defaults	43 ms	35 ms	⚪ -19%	0	—	➖ differs (as base)
⚪	B3 sum over a literal times count, instant	instant	defaults	7 ms	7 ms	⚪ +0%	19 ms	0.37×	✅ same
⚪	B4 outer sum over ignoring(machine)	1h	defaults	17 ms	13 ms	⚪ -24%	95 ms	0.14×	✅ same
⚪	by service_name	1h	defaults	5 ms	4 ms	⚪ -20%	57 ms	0.07×	✅ same
⚪	bytes_over_time by app	1h	defaults	6 ms	5 ms	⚪ -17%	113 ms	0.04×	✅ same
⚪	detected_level volume	1h	defaults	8 ms	7 ms	⚪ -12%	84 ms	0.08×	✅ same
⚪	drilldown detected_level	1h	defaults	5 ms	5 ms	⚪ +0%	47 ms	0.11×	✅ same
⚪	drilldown level volume	1h	defaults	4 ms	4 ms	⚪ +0%	49 ms	0.08×	✅ same
⚪	drilldown service landing	1h	defaults	6 ms	8 ms	⚪ +33%	49 ms	0.16×	✅ same
⚪	instant json filtered	instant	defaults	6 ms	5 ms	⚪ -17%	69 ms	0.07×	✅ same
⚪	instant json volume	instant	defaults	16 ms	15 ms	⚪ -6%	85 ms	0.18×	✅ same
⚪	instant ungrouped rate	instant	defaults	5 ms	4 ms	⚪ -20%	79 ms	0.05×	✅ same
⚪	json rate by level (no filter, no drop)	1h	defaults	400	400	⚪ +0%	400	—	✅ same
⚪	json volume no filters	1h	defaults	23 ms	23 ms	⚪ +0%	98 ms	0.23×	✅ same
⚪	logfmt count by level, filter, drop	1h	defaults	9 ms	8 ms	⚪ -11%	69 ms	0.12×	✅ same
⚪	logfmt volume	1h	defaults	50 ms	50 ms	⚪ +0%	76 ms	0.66×	✅ same
⚪	logs plain	1h	defaults	47 ms	36 ms	⚪ -23%	60 ms	0.60×	✅ same
⚪	logs \| json	1h	defaults	79 ms	54 ms	⚪ -32%	88 ms	0.61×	✅ same
⚪	logs \| logfmt	1h	defaults	90 ms	90 ms	⚪ +0%	63 ms	1.43×	✅ same
⚪	plain volume	1h	defaults	4 ms	4 ms	⚪ +0%	59 ms	0.07×	✅ same
⚪	stream-label count by app	1h	defaults	5 ms	5 ms	⚪ +0%	58 ms	0.09×	✅ same
⚪	topk rate by app	1h	defaults	9 ms	10 ms	⚪ +11%	92 ms	0.11×	✅ sum ±0.02%
⚪	ungrouped rate	1h	defaults	4 ms	4 ms	⚪ +0%	150 ms	0.03×	✅ same
⚪	DB1 label breakdown pod	1h	defaults	34 ms	34 ms	⚪ +0%	265 ms	0.13×	✅ same
⚪	DB2 label breakdown namespace	1h	defaults	4 ms	4 ms	⚪ +0%	80 ms	0.05×	✅ same
⚪	DB3 json field breakdown user_id	1h	defaults	6 ms	5 ms	⚪ -17%	73 ms	0.07×	✅ same
⚪	DB4 json field breakdown batch_id	1h	defaults	23 ms	28 ms	⚪ +22%	62 ms	0.45×	✅ same
⚪	X1 json filter level_extracted, by level	1h	defaults	74 ms	73 ms	⚪ -1%	57 ms	1.28×	✅ same
⚪	X2 json group by level_extracted	1h	defaults	7 ms	7 ms	⚪ +0%	57 ms	0.12×	✅ same
⚪	X3 json rate, level_extracted equality	1h	defaults	6 ms	6 ms	⚪ +0%	52 ms	0.12×	✅ same
⚪	X4 json bytes, level_extracted regexp	1h	defaults	67 ms	63 ms	⚪ -6%	87 ms	0.72×	✅ same
⚪	X5 logfmt filter level_extracted, by level	1h	defaults	27 ms	16 ms	⚪ -41%	111 ms	0.14×	✅ same
⚪	E1 logs, json parser over stream labels	1h	defaults	77 ms	79 ms	⚪ +3%	53 ms	1.49×	✅ same
⚪	E2 logs, logfmt parser over stream labels	1h	defaults	58 ms	40 ms	⚪ -31%	44 ms	0.91×	✅ same
⚪	E3 logs, json, existence filter on level_extracted	1h	defaults	111 ms	109 ms	⚪ -2%	61 ms	1.79×	✅ same
⚪	E4 logs, logfmt, existence filter on level_extracted	1h	defaults	52 ms	70 ms	⚪ +35%	52 ms	1.35×	✅ same
⚪	E5 logs, json, drop level_extracted	1h	defaults	89 ms	91 ms	⚪ +2%	51 ms	1.78×	✅ same
⚪	E6 logfmt count by level_extracted	1h	defaults	4 ms	4 ms	⚪ +0%	5 ms	0.80×	✅ same
⚪	E7 json count by level_extracted	1h	defaults	48 ms	47 ms	⚪ -2%	16 ms	2.94×	✅ same
⚪	E8 instant logfmt by level_extracted	instant	defaults	4 ms	4 ms	⚪ +0%	7 ms	0.57×	✅ same
⚪	DLV1 drilldown logs volume, json	1h	defaults	694 ms	688 ms	⚪ -1%	21 ms	33×	✅ same
⚪	DLV2 json error filter, detected_level	1h	defaults	3.02 s	3.01 s	⚪ -0%	57 ms	53×	✅ same
⚪	DLV3 plain json, detected_level	1h	defaults	691 ms	690 ms	⚪ -0%	16 ms	43×	✅ same
⚪	DLV4 explore json volume, level and detected_level	1h	defaults	22 ms	21 ms	⚪ -5%	12 ms	1.75×	✅ same
⚪	DLV5 json error dropped, detected_level	1h	defaults	4 ms	4 ms	⚪ +0%	53 ms	0.08×	✅ same
⚪	DLV6 drilldown detected_level breakdown, json	1h	defaults	1 ms	1 ms	⚪ +0%	43 ms	0.02×	✅ same
⚪	J1 json no-error count by service_name, one service	1h	defaults	14 ms	13 ms	⚪ -7%	28 ms	0.46×	✅ same
⚪	J2 json no-error count by app, namespace	1h	defaults	25 ms	25 ms	⚪ +0%	90 ms	0.28×	✅ same
⚪	J3 json no-error count by parsed method	1h	defaults	40 ms	45 ms	⚪ +12%	40 ms	1.12×	✅ same
⚪	J4 json no-error count by service_name, all services	1h	defaults	17 ms	17 ms	⚪ +0%	92 ms	0.18×	✅ same
⚪	J5 json no-error bytes_rate by app	1h	defaults	8 ms	7 ms	⚪ -12%	66 ms	0.11×	✅ same
⚪	A explore volume, 2 filters, drop	1h	defaults	51 ms	53 ms	⚪ +4%	136 ms	0.39×	✅ same
⚪	B grouped sum, filter, no drop	1h	defaults	47 ms	44 ms	⚪ -6%	120 ms	0.37×	✅ same
⚪	C ungrouped sum, filter, no drop	1h	defaults	44 ms	46 ms	⚪ +5%	121 ms	0.38×	✅ same
⚪	D volume, no filters	1h	defaults	50 ms	49 ms	⚪ -2%	156 ms	0.31×	✅ same
⚪	E field breakdown pipeline (explore)	1h	defaults	46 ms	43 ms	⚪ -7%	129 ms	0.33×	✅ same
⚪	F field breakdown service_version (explore)	1h	defaults	57 ms	68 ms	⚪ +19%	142 ms	0.48×	✅ same
⚪	G field breakdown pipeline (drilldown)	1h	defaults	6 ms	6 ms	⚪ +0%	121 ms	0.05×	✅ same
⚪	H field breakdown service_version (drilldown)	1h	defaults	22 ms	23 ms	⚪ +5%	137 ms	0.17×	✅ same
⚪	I grafana volume of B	1h	defaults	45 ms	44 ms	⚪ -2%	134 ms	0.33×	✅ same
⚪	M1 multi-value variable, log query	1h	defaults	25 ms	21 ms	⚪ -16%	18 ms	1.17×	✅ same
⚪	M2 multi-value variable, count by app	1h	defaults	4 ms	3 ms	⚪ -25%	29 ms	0.10×	✅ same
⚪	M3 single-value variable, log query	1h	defaults	14 ms	13 ms	⚪ -7%	14 ms	0.93×	✅ same
⚪	M4 alternation over many streams, count	1h	defaults	4 ms	4 ms	⚪ +0%	79 ms	0.05×	✅ same
⚪	M5 json label filter, literal alternation	1h	defaults	23 ms	22 ms	⚪ -4%	21 ms	1.05×	✅ same
⚪	M6 json label filter, prefix (substring in Loki)	1h	defaults	34 ms	30 ms	⚪ -12%	22 ms	1.36×	✅ same
⚪	L1 label_replace over sum by rate, instant	instant	defaults	8 ms	7 ms	⚪ -12%	71 ms	0.10×	✅ same
⚪	L2 label_replace over sum by rate	1h	defaults	12 ms	12 ms	⚪ +0%	126 ms	0.10×	✅ same
⚪	L3 topk over label_replace over sum by count	1h	defaults	6 ms	7 ms	⚪ +17%	84 ms	0.08×	✅ sum ±0.03%
⚪	L4 label_replace over bare count_over_time, instant	instant	defaults	9 ms	8 ms	⚪ -11%	21 ms	0.38×	✅ same
⚪	L5 label_replace over json sum by, instant	instant	defaults	32 ms	31 ms	⚪ -3%	22 ms	1.41×	✅ same
⚪	L6 label_replace over json sum by	1h	defaults	14 ms	13 ms	⚪ -7%	31 ms	0.42×	✅ same
⚪	L7 label_replace around a binary of two sums by level	1h	defaults	13 ms	13 ms	⚪ +0%	16 ms	0.81×	✅ same
⚪	L8 label_replace over sum by level count, instant	instant	defaults	4 ms	4 ms	⚪ +0%	7 ms	0.57×	✅ same
⚪	LF1 logs, detected_level equals	1h	defaults	103 ms	106 ms	⚪ +3%	49 ms	2.16×	✅ same
⚪	LF10 drilldown logs, level pill multi-select, json service	1h	defaults	80 ms	82 ms	⚪ +3%	29 ms	2.83×	✅ same
⚪	LF11 drilldown logs volume by level, existing level	1h	defaults	13 ms	12 ms	⚪ -8%	45 ms	0.27×	✅ same
⚪	LF12 drilldown logs volume by level, level pill	1h	defaults	10 ms	11 ms	⚪ +10%	15 ms	0.73×	✅ same
⚪	LF13 explore volume, level filter before json	1h	defaults	23 ms	23 ms	⚪ +0%	80 ms	0.29×	✅ same
⚪	LF14 explore volume, plain-text service	1h	defaults	8 ms	7 ms	⚪ -12%	6 ms	1.17×	✅ same
⚪	LF15 drilldown field breakdown, level filter	1h	defaults	5 ms	5 ms	⚪ +0%	13 ms	0.38×	✅ same
⚪	LF16 drilldown label breakdown, level filter	1h	defaults	23 ms	22 ms	⚪ -4%	11 ms	2.00×	✅ same
⚪	LF17 detected_level count, level regexp	1h	defaults	10 ms	9 ms	⚪ -10%	55 ms	0.16×	➖ differs (as base)
⚪	LF18 detected_level count, level not equal	1h	defaults	10 ms	8 ms	⚪ -20%	50 ms	0.16×	➖ differs (as base)
⚪	LF19 detected_level count, regexp with unknown	1h	defaults	9 ms	8 ms	⚪ -11%	52 ms	0.15×	➖ differs (as base)
⚪	LF2 logs, detected_level regexp	1h	defaults	111 ms	111 ms	⚪ +0%	25 ms	4.44×	✅ same
⚪	LF20 logfmt count by detected_level	1h	defaults	8 ms	8 ms	⚪ +0%	12 ms	0.67×	✅ same
⚪	LF21 logfmt count by detected_level, level filter	1h	defaults	5 ms	5 ms	⚪ +0%	10 ms	0.50×	✅ same
⚪	LF22 json count by detected_level, level filter	1h	defaults	4 ms	4 ms	⚪ +0%	23 ms	0.17×	✅ same
⚪	LF23 json count by level, level filter	1h	defaults	21 ms	22 ms	⚪ +5%	15 ms	1.47×	✅ same
⚪	LF24 rate of errors by service	1h	defaults	9 ms	7 ms	⚪ -22%	73 ms	0.10×	➖ differs (as base)
⚪	LF25 count of warnings, error label dropped	1h	defaults	9 ms	8 ms	⚪ -11%	53 ms	0.15×	➖ differs (as base)
⚪	LF26 topk services by errors	1h	defaults	4 ms	4 ms	⚪ +0%	68 ms	0.06×	➖ differs (as base)
⚪	LF27 detected_level count, instant	instant	defaults	26 ms	24 ms	⚪ -8%	68 ms	0.35×	➖ differs (as base)
⚪	LF28 logs, keep detected_level	1h	defaults	29 ms	28 ms	⚪ -3%	16 ms	1.75×	✅ same
⚪	LF29 logs, drop detected_level	1h	defaults	49 ms	49 ms	⚪ +0%	17 ms	2.88×	✅ same
⚪	LF3 logs, detected_level not equal, all services	1h	defaults	114 ms	120 ms	⚪ +5%	63 ms	1.90×	➖ differs (as base)
⚪	LF30 logs, drop level	1h	defaults	46 ms	44 ms	⚪ -4%	17 ms	2.59×	✅ same
⚪	LF31 count by detected_level, level dropped	1h	defaults	5 ms	4 ms	⚪ -20%	15 ms	0.27×	✅ same
⚪	LF32 count by detected_level, detected_level kept	1h	defaults	5 ms	4 ms	⚪ -20%	15 ms	0.27×	✅ same
⚪	LF33 drilldown logs count, level pill, instant	instant	defaults	7 ms	7 ms	⚪ +0%	17 ms	0.41×	✅ same
⚪	LF4 logs, detected_level regexp with unknown	1h	defaults	236 ms	223 ms	⚪ -6%	85 ms	2.62×	➖ differs (as base)
⚪	LF5 logs, plain-text service, detected_level regexp	1h	defaults	41 ms	40 ms	⚪ -2%	9 ms	4.44×	➖ differs (as base)
⚪	LF6 logs, json then detected_level	1h	defaults	82 ms	81 ms	⚪ -1%	120 ms	0.68×	✅ same
⚪	LF7 logs, json then level equals	1h	defaults	105 ms	97 ms	⚪ -8%	124 ms	0.78×	✅ same
⚪	LF8 logs, logfmt then level regexp	1h	defaults	52 ms	44 ms	⚪ -15%	59 ms	0.75×	✅ same
⚪	LF9 drilldown logs, level pill, logfmt service	1h	defaults	40 ms	35 ms	⚪ -12%	22 ms	1.59×	➖ differs (as base)
⚪	LS1 logs, level stream label	1h	defaults	32 ms	32 ms	⚪ +0%	53 ms	0.60×	✅ same
⚪	LS2 logs, level stream label regexp, all services	1h	defaults	55 ms	52 ms	⚪ -5%	64 ms	0.81×	✅ same
⚪	LS3 logs, level in the selector	1h	defaults	30 ms	26 ms	⚪ -13%	17 ms	1.53×	✅ same
⚪	LS4 logs, json then level_extracted	1h	defaults	102 ms	105 ms	⚪ +3%	122 ms	0.86×	✅ same
⚪	LS5 logs, logfmt then level_extracted regexp	1h	defaults	43 ms	44 ms	⚪ +2%	44 ms	1.00×	✅ same
⚪	LS6 level stream label count	1h	defaults	4 ms	3 ms	⚪ -25%	46 ms	0.07×	✅ same
⚪	LS7 rate of level stream label by service	1h	defaults	4 ms	4 ms	⚪ +0%	71 ms	0.06×	✅ same
⚪	LS8 json count by level_extracted, collision	1h	defaults	45 ms	46 ms	⚪ +2%	17 ms	2.71×	✅ same
⚪	O1 explore logs, underscore structured-metadata filter	1h	defaults	209 ms	235 ms	⚪ +12%	296 ms	0.79×	✅ same
⚪	O2 explore log volume (Grafana form), underscore structured-metadata filter	1h	defaults	33 ms	31 ms	⚪ -6%	53 ms	0.58×	✅ same
⚪	O3 explore logs, dotted structured-metadata filter	1h	defaults	400	400	⚪ n/a	400	—	✅ same
⚪	O4 explore log volume (Grafana form), dotted structured-metadata filter	1h	defaults	400	400	⚪ n/a	400	—	✅ same
⚪	label values env ending now	1h	defaults	8 ms	8 ms	⚪ +0%	47 ms	0.17×	✅ same
⚪	label values env ending two days ago (control)	1h	defaults	1 ms	1 ms	⚪ +0%	2 ms	0.50×	➖ differs (as base)
⚪	labels ending now	1h	defaults	5 ms	4 ms	⚪ -20%	2 ms	2.00×	✅ same
⚪	labels ending two days ago (control)	1h	defaults	1 ms	1 ms	⚪ +0%	2 ms	0.50×	➖ differs (as base)
⚪	drilldown detected fields	1h	defaults	1 ms	0 ms	⚪ -100%	12 ms	—	➖ differs (as base)
⚪	drilldown detected labels	1h	defaults	0 ms	0 ms	⚪ n/a	1 ms	—	➖ differs (as base)
⚪	drilldown landing volume	1h	defaults	31 ms	29 ms	⚪ -6%	84 ms	0.35×	➖ differs (as base)
⚪	explore label values env	1h	defaults	1 ms	0 ms	⚪ -100%	35 ms	—	✅ same
⚪	explore label values service_name	1h	defaults	1 ms	0 ms	⚪ -100%	46 ms	—	✅ same
⚪	explore labels (label browser)	1h	defaults	0 ms	0 ms	⚪ n/a	1 ms	—	✅ same
⚪	explore labels with selector	1h	defaults	1 ms	1 ms	⚪ +0%	47 ms	0.02×	✅ same
⚪	refresh label values env	1h	defaults	19 ms	18 ms	⚪ -5%	43 ms	0.42×	✅ same
⚪	refresh label values service_name	1h	defaults	9 ms	9 ms	⚪ +0%	50 ms	0.18×	✅ same
⚪	refresh labels (label browser)	1h	defaults	12 ms	11 ms	⚪ -8%	2 ms	5.50×	✅ same
⚪	refresh labels with selector	1h	defaults	12 ms	11 ms	⚪ -8%	50 ms	0.22×	✅ same
⚪	DL1 drilldown field breakdown json-expr, level filter	1h	defaults	11 ms	9 ms	⚪ -18%	46 ms	0.20×	✅ same
⚪	DL2 drilldown field breakdown, level + stream-label filter before json	1h	defaults	23 ms	22 ms	⚪ -4%	8 ms	2.75×	✅ same
⚪	DL3 drilldown logfmt field breakdown, level filter	1h	defaults	51 ms	84 ms	⚪ +65%	84 ms	1.00×	➖ differs (as base)
⚪	DL4 instant count, level filter, json+logfmt, 1h	instant	defaults	257 ms	258 ms	⚪ +0%	741 ms	0.35×	✅ sum ±0.31%
⚪	DL6 explore volume, level filter before json	1h	defaults	23 ms	23 ms	⚪ +0%	83 ms	0.28×	✅ same
⚪	N1 sum by app of max by app,pod, count	1h	defaults	6 ms	5 ms	⚪ -17%	65 ms	0.08×	✅ same
⚪	N10 instant min of avg, rate	instant	defaults	5 ms	4 ms	⚪ -20%	71 ms	0.06×	✅ same
⚪	N11 instant three levels	instant	defaults	4 ms	3 ms	⚪ -25%	55 ms	0.05×	✅ same
⚪	N12 instant json inner, count of sum	instant	defaults	10 ms	12 ms	⚪ +20%	64 ms	0.19×	✅ same
⚪	N13 instant without under by	instant	defaults	11 ms	11 ms	⚪ +0%	71 ms	0.15×	✅ same
⚪	N14 sliding sum of max	1h	defaults	19 ms	19 ms	⚪ +0%	106 ms	0.18×	✅ same
⚪	N15 sliding sums only	1h	defaults	5 ms	4 ms	⚪ -20%	63 ms	0.06×	✅ same
⚪	N16 innermost without under by, tumbling	1h	defaults	6 ms	6 ms	⚪ +0%	41 ms	0.15×	✅ same
⚪	N2 max by namespace of sum by namespace,app, rate	1h	defaults	4 ms	4 ms	⚪ +0%	37 ms	0.11×	✅ same
⚪	N3 count by namespace of sum by namespace,app	1h	defaults	4 ms	4 ms	⚪ +0%	36 ms	0.11×	✅ same
⚪	N4 avg by app of min by app,pod	1h	defaults	5 ms	5 ms	⚪ +0%	40 ms	0.12×	✅ same
⚪	N5 three levels, max min sum	1h	defaults	4 ms	4 ms	⚪ +0%	37 ms	0.11×	✅ same
⚪	N6 json inner, sum of max	1h	defaults	6 ms	6 ms	⚪ +0%	41 ms	0.15×	✅ same
⚪	N7 logfmt inner, max of sum	1h	defaults	3 ms	3 ms	⚪ +0%	10 ms	0.30×	✅ same
⚪	N8 without over a named by level, tumbling	1h	defaults	9 ms	10 ms	⚪ +11%	93 ms	0.11×	✅ same
⚪	N9 instant sum of max	instant	defaults	5 ms	4 ms	⚪ -20%	64 ms	0.06×	✅ same
⚪	OR1 logs, or alternatives	1h	defaults	46 ms	46 ms	⚪ +0%	60 ms	0.77×	✅ same
⚪	OR2 logs, negated or alternatives	1h	defaults	41 ms	53 ms	⚪ +29%	53 ms	1.00×	✅ same
⚪	OR3 logs, regexp or alternatives	1h	defaults	38 ms	38 ms	⚪ +0%	61 ms	0.62×	✅ same
⚪	OR4 count by level, or alternatives with parser	1h	defaults	24 ms	22 ms	⚪ -8%	48 ms	0.46×	✅ same
⚪	OR5 rate, negated regexp or alternatives	1h	defaults	4 ms	4 ms	⚪ +0%	75 ms	0.05×	✅ same
⚪	OR6 instant, or alternatives	instant	defaults	7 ms	6 ms	⚪ -14%	80 ms	0.07×	✅ same
⚪	PR1 count by level, parenthesised json	1h	defaults	24 ms	24 ms	⚪ +0%	53 ms	0.45×	✅ same
⚪	PR2 rate, parenthesised line filter	1h	defaults	4 ms	4 ms	⚪ +0%	72 ms	0.06×	✅ same
⚪	PR3 bytes by app, parenthesised regexp	1h	defaults	4 ms	4 ms	⚪ +0%	68 ms	0.06×	✅ same
⚪	PR4 count by level, parenthesised parser, offset	1h	defaults	23 ms	24 ms	⚪ +4%	55 ms	0.44×	✅ same
⚪	PR5 rate by app, pipeline after the range	1h	defaults	24 ms	25 ms	⚪ +4%	90 ms	0.28×	✅ same
⚪	PR6 doubly parenthesised range	1h	defaults	1 ms	1 ms	⚪ +0%	68 ms	0.01×	✅ same
⚪	PR7 instant, parenthesised line filter	instant	defaults	9 ms	10 ms	⚪ +11%	112 ms	0.09×	✅ same
⚪	P1 api-gateway empty pattern	1h	defaults	15 ms	14 ms	⚪ -7%	17 ms	0.82×	✅ same
⚪	P2 api-gateway not empty pattern, line filter	1h	defaults	84 ms	73 ms	⚪ -13%	28 ms	2.61×	✅ same
⚪	P3 nginx count_over_time, pattern with spaces	1h	defaults	3 ms	3 ms	⚪ +0%	4 ms	0.75×	✅ same
⚪	P4 nginx count_over_time, GET request pattern	1h	defaults	3 ms	3 ms	⚪ +0%	4 ms	0.75×	✅ same
⚪	A1 nginx include GET, log query	1h	defaults	15 ms	14 ms	⚪ -7%	9 ms	1.56×	✅ same
⚪	A2 nginx exclude GET, log query	1h	defaults	15 ms	14 ms	⚪ -7%	8 ms	1.75×	✅ same
⚪	A3 nginx include GET, count_over_time	1h	defaults	3 ms	3 ms	⚪ +0%	5 ms	0.60×	✅ same
⚪	A4 nginx exclude GET, count_over_time	1h	defaults	3 ms	3 ms	⚪ +0%	4 ms	0.75×	✅ same
⚪	A5 drilldown exclude, volume by detected_level	1h	defaults	7 ms	7 ms	⚪ +0%	5 ms	1.40×	✅ same
⚪	A6 nginx include ends on a literal, log query	1h	defaults	10 ms	10 ms	⚪ +0%	4 ms	2.50×	✅ same
⚪	A7 nginx exclude ends on a literal, log query	1h	defaults	15 ms	15 ms	⚪ +0%	9 ms	1.67×	✅ same
⚪	A8 nginx include two literals, count_over_time	1h	defaults	4 ms	3 ms	⚪ -25%	4 ms	0.75×	✅ same
⚪	A9 nginx include two literals, ends on a literal, count_over_time	1h	defaults	4 ms	4 ms	⚪ +0%	4 ms	1.00×	✅ same
⚪	P1 logs, json, level equals	1h	defaults	83 ms	83 ms	⚪ +0%	55 ms	1.51×	✅ same
⚪	P2 logs, logfmt, level not equal	1h	defaults	17 ms	17 ms	⚪ +0%	5 ms	3.40×	✅ same
⚪	P3 logs, json, level existence	1h	defaults	82 ms	80 ms	⚪ -2%	43 ms	1.86×	✅ same
⚪	P4 logfmt count by level	1h	defaults	4 ms	4 ms	⚪ +0%	5 ms	0.80×	✅ same
⚪	P5 json count by level	1h	defaults	23 ms	25 ms	⚪ +9%	17 ms	1.47×	✅ same
⚪	P6 json count, level filter	1h	defaults	8 ms	8 ms	⚪ +0%	28 ms	0.29×	✅ same
⚪	P7 json rate by level, level regexp	1h	defaults	4 ms	4 ms	⚪ +0%	17 ms	0.24×	✅ same
⚪	P8 json count by a name that is no stream label	1h	defaults	23 ms	24 ms	⚪ +4%	17 ms	1.41×	✅ same
⚪	P9 instant logfmt by level	instant	defaults	4 ms	4 ms	⚪ +0%	6 ms	0.67×	✅ same
⚪	SW1 sum without pod, count_over_time	1h	defaults	8 ms	6 ms	⚪ -25%	74 ms	0.08×	✅ same
⚪	SW2 sum without pod, bytes_over_time	1h	defaults	8 ms	7 ms	⚪ -12%	48 ms	0.15×	✅ same
⚪	SW3 sum without pod and version, line filter	1h	defaults	8 ms	7 ms	⚪ -12%	52 ms	0.13×	✅ same
⚪	SW4 trailing sum without pod	1h	defaults	3 ms	3 ms	⚪ +0%	47 ms	0.06×	✅ same
⚪	SW5 sum without pod, rate (control)	1h	defaults	10 ms	10 ms	⚪ +0%	48 ms	0.21×	✅ same
⚪	SW6 max without pod (control)	1h	defaults	9 ms	8 ms	⚪ -11%	56 ms	0.14×	✅ same
⚪	SF1 logs, json extraction list	1h	defaults	75 ms	73 ms	⚪ -3%	55 ms	1.33×	✅ same
⚪	SF10 volume, structured-metadata filter	1h	defaults	7 ms	7 ms	⚪ +0%	47 ms	0.15×	✅ same
⚪	SF2 logs, logfmt extraction list	1h	defaults	106 ms	111 ms	⚪ +5%	55 ms	2.02×	✅ same
⚪	SF3 logs, json then line_format	1h	defaults	73 ms	72 ms	⚪ -1%	69 ms	1.04×	✅ same
⚪	SF4 logs, line_format without a parser	1h	defaults	51 ms	59 ms	⚪ +16%	51 ms	1.16×	✅ same
⚪	SF5 logs, regexp capture	1h	defaults	107 ms	100 ms	⚪ -7%	58 ms	1.72×	✅ same
⚪	SF6 logs, structured-metadata filter	1h	defaults	55 ms	52 ms	⚪ -5%	109 ms	0.48×	✅ same
⚪	SF7 logs, filter on a line key without a parser	1h	defaults	55 ms	49 ms	⚪ -11%	42 ms	1.17×	✅ same
⚪	SF8 volume, json extraction list	1h	defaults	4 ms	4 ms	⚪ +0%	83 ms	0.05×	✅ same
⚪	SF9 volume, logfmt label_format rename	1h	defaults	9 ms	9 ms	⚪ +0%	66 ms	0.14×	✅ same
⚪	E1 json duration_ms sum by service	1h	defaults	7 ms	6 ms	⚪ -14%	23 ms	0.26×	✅ same
⚪	E2 json duration_ms p95 by service	1h	defaults	23 ms	18 ms	⚪ -22%	20 ms	0.90×	✅ same
⚪	E3 logfmt latency_ms rate, production	1h	defaults	9 ms	10 ms	⚪ +11%	106 ms	0.09×	➖ differs (as base)
⚪	E4 logfmt size_bytes max by service	1h	defaults	6 ms	6 ms	⚪ +0%	12 ms	0.50×	✅ same
⚪	E5 json duration_ms sum, instant	instant	defaults	19 ms	18 ms	⚪ -5%	23 ms	0.78×	✅ same
⚪	E6 logfmt ttl unit string (Loki 400)	1h	defaults	400	400	⚪ +0%	400	—	✅ same
⚪	E7 json hexadecimal trace_id (Loki 400)	1h	defaults	400	400	⚪ -8%	400	—	✅ same
⚪	E8 json duration_ms sum by service, namespace prod	1h	defaults	8 ms	8 ms	⚪ +0%	100 ms	0.08×	✅ same
⚪	E9 logfmt latency_ms sum by service, namespace prod	1h	defaults	9 ms	7 ms	⚪ -22%	79 ms	0.09×	✅ same
⚪	D1 logfmt ttl duration_seconds rate by service	1h	defaults	11 ms	11 ms	⚪ +0%	10 ms	1.10×	✅ same
⚪	D2 logfmt ttl duration rate by service	1h	defaults	1 ms	1 ms	⚪ +0%	8 ms	0.12×	✅ same
⚪	D3 logfmt ttl duration_seconds rate, instant	instant	defaults	14 ms	13 ms	⚪ -7%	7 ms	1.86×	✅ same
⚪	D4 logfmt ttl duration rate, instant	instant	defaults	18 ms	12 ms	⚪ -33%	8 ms	1.50×	✅ same
⚪	G1 logfmt amount sum_over_time, no filter	1h	defaults	4 ms	4 ms	⚪ +0%	13 ms	0.31×	✅ same
⚪	G10 logfmt latency_ms sum_over_time, production	1h	defaults	8 ms	8 ms	⚪ +0%	102 ms	0.08×	➖ differs (as base)
⚪	G11 logfmt amount sum_over_time, instant	instant	defaults	4 ms	4 ms	⚪ +0%	9 ms	0.44×	✅ same
⚪	G12 json duration_ms sum_over_time, instant	instant	defaults	17 ms	19 ms	⚪ +12%	22 ms	0.86×	✅ same
⚪	G2 json duration_ms sum_over_time, no filter	1h	defaults	5 ms	5 ms	⚪ +0%	30 ms	0.17×	✅ same
⚪	G3 logfmt amount sum_over_time, error filter	1h	defaults	4 ms	4 ms	⚪ +0%	9 ms	0.44×	✅ same
⚪	G4 json duration_ms sum_over_time, error filter	1h	defaults	4 ms	4 ms	⚪ +0%	23 ms	0.17×	✅ same
⚪	G5 logfmt amount sum by rate	1h	defaults	6 ms	7 ms	⚪ +17%	10 ms	0.70×	✅ same
⚪	G6 json duration_ms sum by rate	1h	defaults	4 ms	5 ms	⚪ +25%	23 ms	0.22×	✅ same
⚪	G7 duration() of a unit string	1h	defaults	10 ms	9 ms	⚪ -10%	7 ms	1.29×	✅ same
⚪	G8 bytes() of a number	1h	defaults	7 ms	8 ms	⚪ +14%	6 ms	1.33×	✅ same
⚪	G9 json duration_ms max_over_time	1h	defaults	5 ms	4 ms	⚪ -20%	84 ms	0.05×	✅ same
⚪	W1 sum without pod, max_over_time	1h	defaults	6 ms	6 ms	⚪ +0%	18 ms	0.33×	✅ same
⚪	W2 sum without pod, sum_over_time	1h	defaults	5 ms	5 ms	⚪ +0%	12 ms	0.42×	✅ same
⚪	W3 sum without pod, rate	1h	defaults	6 ms	6 ms	⚪ +0%	14 ms	0.43×	✅ same
⚪	W4 max without pod, max_over_time	1h	defaults	2 ms	2 ms	⚪ +0%	15 ms	0.13×	✅ same
⚪	W5 sum without pod, max_over_time, instant	instant	defaults	5 ms	5 ms	⚪ +0%	13 ms	0.38×	✅ same
⚪	W6 sum without pod, rate, instant	instant	defaults	5 ms	5 ms	⚪ +0%	12 ms	0.42×	✅ same
⚪	W7 topk 5 of sum without pod	1h	defaults	2 ms	2 ms	⚪ +0%	15 ms	0.13×	✅ same
⚪	W8 sum without pod, count_over_time (production)	1h	defaults	6 ms	7 ms	⚪ +17%	69 ms	0.10×	✅ same
⚪	W9 label_replace control, instant	instant	defaults	4 ms	4 ms	⚪ +0%	70 ms	0.06×	✅ same
⚪	U1 logfmt amount rate	1h	defaults	4 ms	4 ms	⚪ +0%	14 ms	0.29×	✅ same
⚪	U2 json duration_ms rate	1h	defaults	5 ms	5 ms	⚪ +0%	26 ms	0.19×	✅ same
⚪	U3 json duration_ms rate, error filter	1h	defaults	5 ms	5 ms	⚪ +0%	22 ms	0.23×	✅ same
⚪	U4 logfmt latency_ms rate, production	1h	defaults	7 ms	7 ms	⚪ +0%	100 ms	0.07×	➖ differs (as base)
⚪	U5 logfmt amount rate, instant	instant	defaults	5 ms	4 ms	⚪ -20%	11 ms	0.36×	✅ same
⚪	U6 json duration_ms rate, instant	instant	defaults	7 ms	7 ms	⚪ +0%	18 ms	0.39×	✅ same
⚪	U7 json duration_ms sum_over_time, instant	instant	defaults	9 ms	8 ms	⚪ -11%	21 ms	0.38×	✅ same
⚪	U8 logfmt amount rate by service (grouped)	1h	defaults	4 ms	4 ms	⚪ +0%	11 ms	0.36×	✅ same
⚪	U9 logfmt amount rate_counter (folded per series)	1h	defaults	5 ms	5 ms	⚪ +0%	28 ms	0.18×	✅ same
⚪	U1 json duration_ms sum by service	1h	defaults	6 ms	6 ms	⚪ +0%	23 ms	0.26×	✅ same
⚪	U10 instant sum_over_time unwrap	instant	defaults	19 ms	19 ms	⚪ +0%	22 ms	0.86×	✅ same
⚪	U11 instant max_over_time missing label	instant	defaults	4 ms	4 ms	⚪ +0%	12 ms	0.33×	✅ same
⚪	U12 json duration_ms over every service	1h	defaults	7 ms	6 ms	⚪ -14%	56 ms	0.11×	✅ same
⚪	U13 logfmt latency_ms over every service	1h	defaults	10 ms	9 ms	⚪ -10%	72 ms	0.12×	➖ differs (as base)
⚪	U14 duration() of a unit string	1h	defaults	11 ms	10 ms	⚪ -9%	4 ms	2.50×	✅ same
⚪	U2 json duration_ms max by service	1h	defaults	5 ms	5 ms	⚪ +0%	35 ms	0.14×	✅ same
⚪	U3 json duration_ms sliding sum	1h	defaults	6 ms	5 ms	⚪ -17%	34 ms	0.15×	✅ same
⚪	U4 logfmt size_bytes sum by service	1h	defaults	4 ms	4 ms	⚪ +0%	5 ms	0.80×	✅ same
⚪	U5 logfmt ttl unit string (no sample)	1h	defaults	4 ms	4 ms	⚪ +0%	3 ms	1.33×	✅ same
⚪	U6 missing label, no parser	1h	defaults	4 ms	4 ms	⚪ +0%	9 ms	0.44×	✅ same
⚪	U7 missing label, ungrouped sum	1h	defaults	4 ms	4 ms	⚪ +0%	10 ms	0.40×	✅ same
⚪	U8 rate of an unwrapped label	1h	defaults	5 ms	5 ms	⚪ +0%	15 ms	0.33×	✅ same
⚪	U9 bytes() conversion	1h	defaults	8 ms	8 ms	⚪ +0%	7 ms	1.14×	✅ same

Cold (first-run) timings — what a user opening a panel sees

shape	range	Loki profile	base cold	PR cold	Loki cold	PR ÷ Loki cold
B1 sum by over rate divided by a literal	1h	defaults	18 ms	12 ms	140 ms	0.09×
B2 sum by over a ratio of two per-stream metrics	1h	defaults	385 ms	374 ms	0	—
B3 sum over a literal times count, instant	instant	defaults	8 ms	6 ms	20 ms	0.30×
B4 outer sum over ignoring(machine)	1h	defaults	25 ms	34 ms	563 ms	0.06×
by service_name	1h	defaults	20 ms	17 ms	470 ms	0.04×
bytes_over_time by app	1h	defaults	57 ms	47 ms	533 ms	0.09×
detected_level volume	1h	defaults	61 ms	65 ms	659 ms	0.10×
drilldown detected_level	1h	defaults	39 ms	36 ms	438 ms	0.08×
drilldown level volume	1h	defaults	23 ms	9 ms	448 ms	0.02×
drilldown service landing	1h	defaults	18 ms	18 ms	618 ms	0.03×
instant json filtered	instant	defaults	9 ms	8 ms	68 ms	0.12×
instant json volume	instant	defaults	16 ms	16 ms	87 ms	0.18×
instant ungrouped rate	instant	defaults	5 ms	4 ms	73 ms	0.05×
json rate by level (no filter, no drop)	1h	defaults	400	400	400	—
json volume no filters	1h	defaults	32 ms	31 ms	807 ms ↺	0.04×
logfmt count by level, filter, drop	1h	defaults	261 ms	234 ms	958 ms	0.24×
logfmt volume	1h	defaults	116 ms	120 ms	884 ms	0.14×
logs plain	1h	defaults	53 ms	37 ms	104 ms	0.36×
logs \| json	1h	defaults	85 ms	52 ms	76 ms	0.68×
logs \| logfmt	1h	defaults	108 ms	88 ms	62 ms	1.42×
plain volume	1h	defaults	10 ms	12 ms	703 ms	0.02×
stream-label count by app	1h	defaults	12 ms	21 ms	464 ms	0.05×
topk rate by app	1h	defaults	15 ms	9 ms	529 ms	0.02×
ungrouped rate	1h	defaults	7 ms	7 ms	470 ms	0.01×
DB1 label breakdown pod	1h	defaults	80 ms	86 ms	802 ms	0.11×
DB2 label breakdown namespace	1h	defaults	9 ms	10 ms	366 ms	0.03×
DB3 json field breakdown user_id	1h	defaults	28 ms	28 ms	304 ms	0.09×
DB4 json field breakdown batch_id	1h	defaults	52 ms	64 ms	538 ms	0.12×
X1 json filter level_extracted, by level	1h	defaults	1.75 s	1.72 s	532 ms	3.23×
X2 json group by level_extracted	1h	defaults	156 ms	181 ms	548 ms	0.33×
X3 json rate, level_extracted equality	1h	defaults	142 ms	141 ms	449 ms	0.31×
X4 json bytes, level_extracted regexp	1h	defaults	204 ms	220 ms	518 ms	0.42×
X5 logfmt filter level_extracted, by level	1h	defaults	356 ms	356 ms	708 ms	0.50×
E1 logs, json parser over stream labels	1h	defaults	82 ms	64 ms	59 ms †	—
E2 logs, logfmt parser over stream labels	1h	defaults	35 ms	38 ms	47 ms †	—
E3 logs, json, existence filter on level_extracted	1h	defaults	91 ms	108 ms	62 ms †	—
E4 logs, logfmt, existence filter on level_extracted	1h	defaults	99 ms	115 ms	56 ms †	—
E5 logs, json, drop level_extracted	1h	defaults	82 ms	62 ms	49 ms †	—
E6 logfmt count by level_extracted	1h	defaults	26 ms	25 ms	27 ms	0.93×
E7 json count by level_extracted	1h	defaults	697 ms	700 ms	98 ms	7.14×
E8 instant logfmt by level_extracted	instant	defaults	5 ms	4 ms	7 ms	0.57×
DLV1 drilldown logs volume, json	1h	defaults	695 ms	684 ms	108 ms	6.33×
DLV2 json error filter, detected_level	1h	defaults	3.20 s	3.02 s	470 ms	6.42×
DLV3 plain json, detected_level	1h	defaults	707 ms	700 ms	104 ms	6.73×
DLV4 explore json volume, level and detected_level	1h	defaults	44 ms	41 ms	103 ms	0.40×
DLV5 json error dropped, detected_level	1h	defaults	10 ms	11 ms	422 ms	0.03×
DLV6 drilldown detected_level breakdown, json	1h	defaults	1 ms	1 ms	632 ms	0.00×
J1 json no-error count by service_name, one service	1h	defaults	93 ms	86 ms	113 ms	0.76×
J2 json no-error count by app, namespace	1h	defaults	214 ms	219 ms	341 ms	0.64×
J3 json no-error count by parsed method	1h	defaults	76 ms	76 ms	109 ms	0.70×
J4 json no-error count by service_name, all services	1h	defaults	373 ms	369 ms	522 ms	0.71×
J5 json no-error bytes_rate by app	1h	defaults	200 ms	131 ms	322 ms	0.41×
A explore volume, 2 filters, drop	1h	defaults	1.05 s	908 ms	497 ms	1.83×
B grouped sum, filter, no drop	1h	defaults	69 ms	69 ms	416 ms	0.17×
C ungrouped sum, filter, no drop	1h	defaults	67 ms	68 ms	433 ms	0.16×
D volume, no filters	1h	defaults	114 ms	113 ms	807 ms	0.14×
E field breakdown pipeline (explore)	1h	defaults	63 ms	63 ms	446 ms	0.14×
F field breakdown service_version (explore)	1h	defaults	788 ms	755 ms	520 ms	1.45×
G field breakdown pipeline (drilldown)	1h	defaults	28 ms	27 ms	673 ms	0.04×
H field breakdown service_version (drilldown)	1h	defaults	41 ms	41 ms	510 ms	0.08×
I grafana volume of B	1h	defaults	66 ms	66 ms	443 ms	0.15×
M1 multi-value variable, log query	1h	defaults	35 ms	29 ms	29 ms	1.00×
M2 multi-value variable, count by app	1h	defaults	6 ms	6 ms	118 ms	0.05×
M3 single-value variable, log query	1h	defaults	17 ms	14 ms	22 ms	0.64×
M4 alternation over many streams, count	1h	defaults	6 ms	9 ms	501 ms	0.02×
M5 json label filter, literal alternation	1h	defaults	59 ms	56 ms	132 ms	0.42×
M6 json label filter, prefix (substring in Loki)	1h	defaults	24 ms	28 ms	23 ms †	—
L1 label_replace over sum by rate, instant	instant	defaults	13 ms	8 ms	71 ms	0.11×
L2 label_replace over sum by rate	1h	defaults	73 ms	72 ms	333 ms	0.22×
L3 topk over label_replace over sum by count	1h	defaults	13 ms	9 ms	349 ms	0.03×
L4 label_replace over bare count_over_time, instant	instant	defaults	8 ms	8 ms	22 ms	0.36×
L5 label_replace over json sum by, instant	instant	defaults	31 ms	32 ms	27 ms	1.19×
L6 label_replace over json sum by	1h	defaults	66 ms	66 ms	102 ms	0.65×
L7 label_replace around a binary of two sums by level	1h	defaults	21 ms	21 ms	41 ms †	—
L8 label_replace over sum by level count, instant	instant	defaults	4 ms	4 ms	7 ms	0.57×
LF1 logs, detected_level equals	1h	defaults	111 ms	107 ms	57 ms	1.88×
LF10 drilldown logs, level pill multi-select, json service	1h	defaults	77 ms	77 ms	30 ms	2.57×
LF11 drilldown logs volume by level, existing level	1h	defaults	308 ms	315 ms	416 ms	0.76×
LF12 drilldown logs volume by level, level pill	1h	defaults	89 ms	99 ms	76 ms	1.30×
LF13 explore volume, level filter before json	1h	defaults	57 ms	58 ms	341 ms	0.17×
LF14 explore volume, plain-text service	1h	defaults	29 ms	32 ms	17 ms	1.88×
LF15 drilldown field breakdown, level filter	1h	defaults	59 ms	54 ms	71 ms	0.76×
LF16 drilldown label breakdown, level filter	1h	defaults	31 ms	34 ms	36 ms	0.94×
LF17 detected_level count, level regexp	1h	defaults	274 ms	268 ms	369 ms	0.73×
LF18 detected_level count, level not equal	1h	defaults	273 ms	264 ms	563 ms	0.47×
LF19 detected_level count, regexp with unknown	1h	defaults	255 ms	257 ms	329 ms	0.78×
LF2 logs, detected_level regexp	1h	defaults	140 ms	114 ms	26 ms	4.38×
LF20 logfmt count by detected_level	1h	defaults	32 ms	31 ms	39 ms	0.79×
LF21 logfmt count by detected_level, level filter	1h	defaults	20 ms	20 ms	33 ms	0.61×
LF22 json count by detected_level, level filter	1h	defaults	11 ms	10 ms	106 ms	0.09×
LF23 json count by level, level filter	1h	defaults	33 ms	32 ms	103 ms	0.31×
LF24 rate of errors by service	1h	defaults	228 ms	227 ms	360 ms	0.63×
LF25 count of warnings, error label dropped	1h	defaults	231 ms	228 ms	361 ms	0.63×
LF26 topk services by errors	1h	defaults	4 ms	4 ms	584 ms	0.01×
LF27 detected_level count, instant	instant	defaults	25 ms	24 ms	67 ms	0.36×
LF28 logs, keep detected_level	1h	defaults	29 ms	29 ms	16 ms †	—
LF29 logs, drop detected_level	1h	defaults	44 ms	48 ms	17 ms	2.82×
LF3 logs, detected_level not equal, all services	1h	defaults	113 ms	111 ms	63 ms	1.76×
LF30 logs, drop level	1h	defaults	42 ms	41 ms	16 ms	2.56×
LF31 count by detected_level, level dropped	1h	defaults	17 ms	17 ms	85 ms	0.20×
LF32 count by detected_level, detected_level kept	1h	defaults	16 ms	16 ms	97 ms	0.16×
LF33 drilldown logs count, level pill, instant	instant	defaults	6 ms	6 ms	15 ms	0.40×
LF4 logs, detected_level regexp with unknown	1h	defaults	230 ms	221 ms	83 ms	2.66×
LF5 logs, plain-text service, detected_level regexp	1h	defaults	43 ms	42 ms	11 ms	3.82×
LF6 logs, json then detected_level	1h	defaults	99 ms	82 ms	116 ms †	—
LF7 logs, json then level equals	1h	defaults	84 ms	79 ms	118 ms †	—
LF8 logs, logfmt then level regexp	1h	defaults	45 ms	48 ms	55 ms †	—
LF9 drilldown logs, level pill, logfmt service	1h	defaults	138 ms	112 ms	25 ms	4.48×
LS1 logs, level stream label	1h	defaults	44 ms	33 ms	55 ms	0.60×
LS2 logs, level stream label regexp, all services	1h	defaults	52 ms	56 ms	64 ms	0.88×
LS3 logs, level in the selector	1h	defaults	25 ms	26 ms	21 ms	1.24×
LS4 logs, json then level_extracted	1h	defaults	118 ms	99 ms	125 ms †	—
LS5 logs, logfmt then level_extracted regexp	1h	defaults	65 ms	58 ms	43 ms †	—
LS6 level stream label count	1h	defaults	7 ms	6 ms	271 ms	0.02×
LS7 rate of level stream label by service	1h	defaults	10 ms	10 ms	303 ms	0.03×
LS8 json count by level_extracted, collision	1h	defaults	678 ms	711 ms	100 ms	7.11×
O1 explore logs, underscore structured-metadata filter	1h	defaults	223 ms	263 ms	297 ms †	—
O2 explore log volume (Grafana form), underscore structured-metadata filter	1h	defaults	142 ms	125 ms	366 ms	0.34×
O3 explore logs, dotted structured-metadata filter	1h	defaults	400	400	400	—
O4 explore log volume (Grafana form), dotted structured-metadata filter	1h	defaults	400	400	400	—
label values env ending now	1h	defaults	68 ms	67 ms	44 ms †	—
label values env ending two days ago (control)	1h	defaults	3 ms	3 ms	1 ms †	—
labels ending now	1h	defaults	55 ms	36 ms	3 ms †	—
labels ending two days ago (control)	1h	defaults	3 ms	3 ms	2 ms †	—
drilldown detected fields	1h	defaults	25 ms	15 ms	21 ms †	—
drilldown detected labels	1h	defaults	27 ms	18 ms	2 ms †	—
drilldown landing volume	1h	defaults	31 ms	30 ms	77 ms	0.39×
explore label values env	1h	defaults	59 ms	44 ms	43 ms †	—
explore label values service_name	1h	defaults	22 ms	20 ms	51 ms †	—
explore labels (label browser)	1h	defaults	41 ms	30 ms	2 ms †	—
explore labels with selector	1h	defaults	58 ms	43 ms	52 ms †	—
refresh label values env	1h	defaults	1 ms	0 ms	37 ms †	—
refresh label values service_name	1h	defaults	1 ms	0 ms	45 ms †	—
refresh labels (label browser)	1h	defaults	0 ms	0 ms	1 ms †	—
refresh labels with selector	1h	defaults	0 ms	0 ms	49 ms †	—
DL1 drilldown field breakdown json-expr, level filter	1h	defaults	269 ms	288 ms	349 ms	0.83×
DL2 drilldown field breakdown, level + stream-label filter before json	1h	defaults	29 ms	28 ms	20 ms	1.40×
DL3 drilldown logfmt field breakdown, level filter	1h	defaults	353 ms	374 ms	796 ms	0.47×
DL4 instant count, level filter, json+logfmt, 1h	instant	defaults	296 ms	265 ms	733 ms	0.36×
DL6 explore volume, level filter before json	1h	defaults	104 ms	115 ms	422 ms	0.27×
N1 sum by app of max by app,pod, count	1h	defaults	32 ms	26 ms	457 ms	0.06×
N10 instant min of avg, rate	instant	defaults	5 ms	4 ms	71 ms	0.06×
N11 instant three levels	instant	defaults	5 ms	3 ms	57 ms	0.05×
N12 instant json inner, count of sum	instant	defaults	12 ms	9 ms	64 ms †	—
N13 instant without under by	instant	defaults	11 ms	10 ms	69 ms	0.14×
N14 sliding sum of max	1h	defaults	256 ms	265 ms	1.02 s	0.26×
N15 sliding sums only	1h	defaults	15 ms	15 ms	458 ms	0.03×
N16 innermost without under by, tumbling	1h	defaults	31 ms	31 ms	461 ms	0.07×
N2 max by namespace of sum by namespace,app, rate	1h	defaults	14 ms	13 ms	294 ms	0.04×
N3 count by namespace of sum by namespace,app	1h	defaults	11 ms	11 ms	472 ms †	—
N4 avg by app of min by app,pod	1h	defaults	28 ms	30 ms	491 ms	0.06×
N5 three levels, max min sum	1h	defaults	11 ms	12 ms	304 ms	0.04×
N6 json inner, sum of max	1h	defaults	103 ms	92 ms	451 ms	0.20×
N7 logfmt inner, max of sum	1h	defaults	16 ms	14 ms	47 ms	0.30×
N8 without over a named by level, tumbling	1h	defaults	26 ms	26 ms	551 ms	0.05×
N9 instant sum of max	instant	defaults	8 ms	5 ms	69 ms	0.07×
OR1 logs, or alternatives	1h	defaults	71 ms	85 ms	68 ms	1.25×
OR2 logs, negated or alternatives	1h	defaults	70 ms	60 ms	55 ms	1.09×
OR3 logs, regexp or alternatives	1h	defaults	41 ms	53 ms	65 ms	0.82×
OR4 count by level, or alternatives with parser	1h	defaults	74 ms	78 ms	265 ms	0.29×
OR5 rate, negated regexp or alternatives	1h	defaults	33 ms	33 ms	329 ms	0.10×
OR6 instant, or alternatives	instant	defaults	8 ms	6 ms	64 ms	0.09×
PR1 count by level, parenthesised json	1h	defaults	106 ms	101 ms	353 ms	0.29×
PR2 rate, parenthesised line filter	1h	defaults	27 ms	26 ms	282 ms	0.09×
PR3 bytes by app, parenthesised regexp	1h	defaults	50 ms	48 ms	353 ms	0.14×
PR4 count by level, parenthesised parser, offset	1h	defaults	27 ms	31 ms	359 ms	0.09×
PR5 rate by app, pipeline after the range	1h	defaults	108 ms	111 ms	395 ms	0.28×
PR6 doubly parenthesised range	1h	defaults	1 ms	1 ms	257 ms	0.00×
PR7 instant, parenthesised line filter	instant	defaults	10 ms	8 ms	63 ms	0.13×
P1 api-gateway empty pattern	1h	defaults	21 ms	15 ms	55 ms †	—
P2 api-gateway not empty pattern, line filter	1h	defaults	55 ms	57 ms	26 ms	2.19×
P3 nginx count_over_time, pattern with spaces	1h	defaults	11 ms	10 ms	17 ms †	—
P4 nginx count_over_time, GET request pattern	1h	defaults	9 ms	10 ms	18 ms	0.56×
A1 nginx include GET, log query	1h	defaults	20 ms	15 ms	10 ms	1.50×
A2 nginx exclude GET, log query	1h	defaults	17 ms	13 ms	9 ms	1.44×
A3 nginx include GET, count_over_time	1h	defaults	10 ms	6 ms	18 ms ↺	0.33×
A4 nginx exclude GET, count_over_time	1h	defaults	11 ms	10 ms	21 ms	0.48×
A5 drilldown exclude, volume by detected_level	1h	defaults	22 ms	22 ms	21 ms	1.05×
A6 nginx include ends on a literal, log query	1h	defaults	13 ms	13 ms	20 ms †	—
A7 nginx exclude ends on a literal, log query	1h	defaults	19 ms	20 ms	8 ms †	—
A8 nginx include two literals, count_over_time	1h	defaults	13 ms	13 ms	19 ms	0.68×
A9 nginx include two literals, ends on a literal, count_over_time	1h	defaults	12 ms	12 ms	17 ms †	—
P1 logs, json, level equals	1h	defaults	84 ms	65 ms	48 ms †	—
P2 logs, logfmt, level not equal	1h	defaults	19 ms	19 ms	22 ms †	—
P3 logs, json, level existence	1h	defaults	81 ms	82 ms	50 ms †	—
P4 logfmt count by level	1h	defaults	17 ms	19 ms	26 ms	0.73×
P5 json count by level	1h	defaults	47 ms	48 ms	95 ms	0.51×
P6 json count, level filter	1h	defaults	9 ms	8 ms	114 ms	0.07×
P7 json rate by level, level regexp	1h	defaults	10 ms	12 ms	102 ms	0.12×
P8 json count by a name that is no stream label	1h	defaults	51 ms	50 ms	80 ms	0.62×
P9 instant logfmt by level	instant	defaults	5 ms	8 ms	9 ms	0.89×
SW1 sum without pod, count_over_time	1h	defaults	52 ms	39 ms	359 ms	0.11×
SW2 sum without pod, bytes_over_time	1h	defaults	61 ms	58 ms	353 ms	0.16×
SW3 sum without pod and version, line filter	1h	defaults	54 ms	54 ms	593 ms	0.09×
SW4 trailing sum without pod	1h	defaults	9 ms	10 ms	335 ms	0.03×
SW5 sum without pod, rate (control)	1h	defaults	167 ms	158 ms	312 ms	0.51×
SW6 max without pod (control)	1h	defaults	111 ms	106 ms	486 ms	0.22×
SF1 logs, json extraction list	1h	defaults	83 ms	65 ms	89 ms	0.73×
SF10 volume, structured-metadata filter	1h	defaults	14 ms	14 ms	240 ms	0.06×
SF2 logs, logfmt extraction list	1h	defaults	107 ms	101 ms	54 ms	1.87×
SF3 logs, json then line_format	1h	defaults	69 ms	67 ms	67 ms	1.00×
SF4 logs, line_format without a parser	1h	defaults	52 ms	64 ms	53 ms	1.21×
SF5 logs, regexp capture	1h	defaults	98 ms	94 ms	60 ms	1.57×
SF6 logs, structured-metadata filter	1h	defaults	41 ms	37 ms	179 ms †	—
SF7 logs, filter on a line key without a parser	1h	defaults	56 ms	50 ms	322 ms †	—
SF8 volume, json extraction list	1h	defaults	13 ms	14 ms	543 ms	0.03×
SF9 volume, logfmt label_format rename	1h	defaults	310 ms	289 ms	976 ms	0.30×
E1 json duration_ms sum by service	1h	defaults	53 ms	46 ms	124 ms	0.37×
E2 json duration_ms p95 by service	1h	defaults	111 ms	98 ms	215 ms	0.46×
E3 logfmt latency_ms rate, production	1h	defaults	272 ms	310 ms	628 ms	0.49×
E4 logfmt size_bytes max by service	1h	defaults	25 ms	25 ms	139 ms	0.18×
E5 json duration_ms sum, instant	instant	defaults	21 ms	30 ms	24 ms	1.25×
E6 logfmt ttl unit string (Loki 400)	1h	defaults	400	400	400	—
E7 json hexadecimal trace_id (Loki 400)	1h	defaults	400	400	400	—
E8 json duration_ms sum by service, namespace prod	1h	defaults	128 ms	108 ms	402 ms	0.27×
E9 logfmt latency_ms sum by service, namespace prod	1h	defaults	182 ms	186 ms	745 ms	0.25×
D1 logfmt ttl duration_seconds rate by service	1h	defaults	58 ms	52 ms	33 ms	1.58×
D2 logfmt ttl duration rate by service	1h	defaults	2 ms	2 ms	38 ms	0.05×
D3 logfmt ttl duration_seconds rate, instant	instant	defaults	18 ms	15 ms	8 ms	1.88×
D4 logfmt ttl duration rate, instant	instant	defaults	13 ms	12 ms	7 ms	1.71×
G1 logfmt amount sum_over_time, no filter	1h	defaults	35 ms	25 ms	58 ms	0.43×
G10 logfmt latency_ms sum_over_time, production	1h	defaults	257 ms	260 ms	673 ms	0.39×
G11 logfmt amount sum_over_time, instant	instant	defaults	5 ms	4 ms	10 ms	0.40×
G12 json duration_ms sum_over_time, instant	instant	defaults	18 ms	16 ms	21 ms	0.76×
G2 json duration_ms sum_over_time, no filter	1h	defaults	45 ms	44 ms	134 ms	0.33×
G3 logfmt amount sum_over_time, error filter	1h	defaults	23 ms	22 ms	43 ms	0.51×
G4 json duration_ms sum_over_time, error filter	1h	defaults	45 ms	45 ms	134 ms	0.34×
G5 logfmt amount sum by rate	1h	defaults	21 ms	23 ms	41 ms ↺	0.56×
G6 json duration_ms sum by rate	1h	defaults	46 ms	50 ms	143 ms	0.35×
G7 duration() of a unit string	1h	defaults	37 ms	34 ms	32 ms	1.06×
G8 bytes() of a number	1h	defaults	34 ms	33 ms	29 ms	1.14×
G9 json duration_ms max_over_time	1h	defaults	47 ms	48 ms	1.39 s	0.03×
W1 sum without pod, max_over_time	1h	defaults	42 ms	37 ms	52 ms	0.71×
W2 sum without pod, sum_over_time	1h	defaults	36 ms	34 ms	55 ms	0.62×
W3 sum without pod, rate	1h	defaults	44 ms	40 ms	43 ms	0.93×
W4 max without pod, max_over_time	1h	defaults	19 ms	22 ms	49 ms	0.45×
W5 sum without pod, max_over_time, instant	instant	defaults	6 ms	6 ms	13 ms	0.46×
W6 sum without pod, rate, instant	instant	defaults	5 ms	6 ms	12 ms	0.50×
W7 topk 5 of sum without pod	1h	defaults	16 ms	18 ms	49 ms	0.37×
W8 sum without pod, count_over_time (production)	1h	defaults	43 ms	50 ms	276 ms	0.18×
W9 label_replace control, instant	instant	defaults	4 ms	3 ms	70 ms	0.04×
U1 logfmt amount rate	1h	defaults	27 ms	21 ms	30 ms	0.70×
U2 json duration_ms rate	1h	defaults	45 ms	44 ms	74 ms	0.59×
U3 json duration_ms rate, error filter	1h	defaults	44 ms	44 ms	90 ms	0.49×
U4 logfmt latency_ms rate, production	1h	defaults	193 ms	189 ms	481 ms	0.39×
U5 logfmt amount rate, instant	instant	defaults	5 ms	5 ms	13 ms	0.38×
U6 json duration_ms rate, instant	instant	defaults	7 ms	6 ms	18 ms	0.33×
U7 json duration_ms sum_over_time, instant	instant	defaults	7 ms	7 ms	18 ms	0.39×
U8 logfmt amount rate by service (grouped)	1h	defaults	26 ms	35 ms	41 ms	0.85×
U9 logfmt amount rate_counter (folded per series)	1h	defaults	32 ms	36 ms	165 ms †	—
U1 json duration_ms sum by service	1h	defaults	51 ms	46 ms	145 ms	0.32×
U10 instant sum_over_time unwrap	instant	defaults	19 ms	29 ms	23 ms	1.26×
U11 instant max_over_time missing label	instant	defaults	4 ms	4 ms	15 ms †	—
U12 json duration_ms over every service	1h	defaults	128 ms	130 ms	495 ms	0.26×
U13 logfmt latency_ms over every service	1h	defaults	285 ms	265 ms	824 ms	0.32×
U14 duration() of a unit string	1h	defaults	34 ms	32 ms	32 ms	1.00×
U2 json duration_ms max by service	1h	defaults	46 ms	46 ms	614 ms	0.07×
U3 json duration_ms sliding sum	1h	defaults	5 ms	10 ms	272 ms	0.04×
U4 logfmt size_bytes sum by service	1h	defaults	32 ms	24 ms	30 ms	0.80×
U5 logfmt ttl unit string (no sample)	1h	defaults	23 ms	24 ms	30 ms †	—
U6 missing label, no parser	1h	defaults	9 ms	19 ms	64 ms †	—
U7 missing label, ungrouped sum	1h	defaults	8 ms	8 ms	64 ms †	—
U8 rate of an unwrapped label	1h	defaults	45 ms	45 ms	133 ms	0.34×
U9 bytes() conversion	1h	defaults	35 ms	54 ms	33 ms	1.64×

† Loki's results cache answered this cold run (its own counters, or identical answers for windows a minute apart): not a computed time, so it is in no ratio.

↺ Loki cold taken from the first set that ran the same query; Loki's cache answered this one.

17 shape×range above 2× Loki already (listed, or as slow on the base build)

	profile	mode	shape	range	PR	Loki	PR ÷ Loki	base ÷ Loki	listed ÷ Loki	why
known	defaults	warm	SF2 logs, logfmt extraction list	1h	111 ms	55 ms	2.02×	1.93×	—	within noise of base
known	defaults	cold	E7 json count by level_extracted	1h	700 ms	98 ms	7.14×	7.11×	—	as slow on base
known	defaults	cold	X1 json filter level_extracted, by level	1h	1.72 s	532 ms	3.23×	3.30×	—	as slow on base
known	defaults	cold	DLV1 drilldown logs volume, json	1h	684 ms	108 ms	6.33×	6.44×	—	as slow on base
known	defaults	warm	DLV1 drilldown logs volume, json	1h	688 ms	21 ms	33×	33×	—	as slow on base
known	defaults	cold	DLV2 json error filter, detected_level	1h	3.02 s	470 ms	6.42×	6.80×	—	as slow on base
known	defaults	warm	DLV2 json error filter, detected_level	1h	3.01 s	57 ms	53×	53×	—	as slow on base
known	defaults	cold	DLV3 plain json, detected_level	1h	700 ms	104 ms	6.73×	6.80×	—	as slow on base
known	defaults	warm	DLV3 plain json, detected_level	1h	690 ms	16 ms	43×	43×	—	as slow on base
known	defaults	warm	LF1 logs, detected_level equals	1h	106 ms	49 ms	2.16×	2.10×	—	as slow on base
known	defaults	cold	LF2 logs, detected_level regexp	1h	114 ms	26 ms	4.38×	5.38×	—	as slow on base
known	defaults	warm	LF2 logs, detected_level regexp	1h	111 ms	25 ms	4.44×	4.44×	—	as slow on base
known	defaults	cold	LF4 logs, detected_level regexp with unknown	1h	221 ms	83 ms	2.66×	2.77×	—	as slow on base
known	defaults	warm	LF4 logs, detected_level regexp with unknown	1h	223 ms	85 ms	2.62×	2.78×	—	as slow on base
known	defaults	cold	LF9 drilldown logs, level pill, logfmt service	1h	112 ms	25 ms	4.48×	5.52×	—	as slow on base
known	defaults	warm	LF10 drilldown logs, level pill multi-select, json service	1h	82 ms	29 ms	2.83×	2.76×	—	as slow on base
known	defaults	cold	LS8 json count by level_extracted, collision	1h	711 ms	100 ms	7.11×	6.78×	—	as slow on base

2 shape×range×mode where Loki did not answer 200: not compared (not a proxy defect).

Why these shapes

baseline run: every shape of every set → 264: aggregation-over-binary/B1 sum by over rate divided by a literal, aggregation-over-binary/B2 sum by over a ratio of two per-stream metrics, aggregation-over-binary/B3 sum over a literal times count, instant, aggregation-over-binary/B4 outer sum over ignoring(machine), control/by service_name, control/bytes_over_time by app, control/detected_level volume, control/drilldown detected_level …

_{base 14e13dc6e5 → PR 20611faf6b · 4 runs (first is cold), warm p50 · ranges 1h · noise ±30% and 100 ms · VictoriaLogs restarts 0→0 · 10.1 min · phases stack_up 39s, seed_and_build 121s, loki-defaults:loki_profile 0s, loki-defaults:proxies 2s, loki-defaults:matrix 420s. Same runner, one fresh stack, targets interleaved per request. How to read and reproduce: bench/ab/README.md.}

🧭 New baseline candidate

This pull request got the baseline run. Every shape of every set was measured at the smoke's ranges (1h, 4 runs). Nothing was committed. The perf-baseline-candidate artifact of this run holds over-2x-loki.json, which a person can copy over bench/ab/history/over-2x-loki.json to accept it, and acceptance-day.json, a report: the committed acceptance.json is rebuilt from the daily history, so the next daily run after the merge writes the real new distribution.

profile	mode	rows compared	below Loki	close	1–2×	above 2×	share below	committed share below
loki-defaults	cold	213	118	84	2	9	55.4%	50.5% (2026-10-29)
loki-defaults	warm	258	70	179	1	8	27.1%	23.2% (2026-10-29)

The committed column is the newest daily run (7 runs, ranges up to 24h): the direction is comparable, the rows are not the same ones.

Above-2× list: 182 rows after this run, 17 added, 0 not above 2× in this run's profile (the daily run's two-run rule decides for the committed list). A profile that did not run leaves its committed rows as they are.

Added:

loki-defaults / extracted-pushdown / X1 json filter level_extracted, by level / 1h / cold
loki-defaults / extracted-suffix / E7 json count by level_extracted / 1h / cold
loki-defaults / json-detected-level / DLV1 drilldown logs volume, json / 1h / cold
loki-defaults / json-detected-level / DLV1 drilldown logs volume, json / 1h / warm
loki-defaults / json-detected-level / DLV2 json error filter, detected_level / 1h / cold
loki-defaults / json-detected-level / DLV2 json error filter, detected_level / 1h / warm
loki-defaults / json-detected-level / DLV3 plain json, detected_level / 1h / cold
loki-defaults / json-detected-level / DLV3 plain json, detected_level / 1h / warm
loki-defaults / level-filter / LF1 logs, detected_level equals / 1h / warm
loki-defaults / level-filter / LF10 drilldown logs, level pill multi-select, json service / 1h / warm
… and 7 more (in over-2x-loki.json)

_{10 min.}

github-actions · 2026-10-12T11:16:15Z

PR Quality Report

Compared against base branch main.

Coverage and tests

Signal	Base	PR	Delta
Test count	18723	18874	151
Coverage	89.0%	89.2%	+0.2% (improved)

Compatibility

Track	Base	PR	Delta
Loki API	100.0%	11/11 (100.0%)	0.0% (stable)
Logs Drilldown	100.0%	17/17 (100.0%)	0.0% (stable)
VictoriaLogs	100.0%	11/11 (100.0%)	0.0% (stable)

Performance smoke

Lower CPU cost (ns/op) is better. Lower benchmark memory cost (B/op, allocs/op) is better. Higher throughput is better. Lower load-test memory growth is better. Benchmark rows are medians from repeated samples.

Signal	Base	PR	Delta
QueryRange cache-hit CPU cost	3592.0 ns/op	3544.0 ns/op	-1.3% (stable)
QueryRange cache-hit memory	1256.0 B/op	1256.0 B/op	0.0% (stable)
QueryRange cache-hit allocations	28.0 allocs/op	28.0 allocs/op	0.0% (stable)
QueryRange cache-bypass CPU cost	4556.0 ns/op	4532.0 ns/op	-0.5% (stable)
QueryRange cache-bypass memory	1324.0 B/op	1327.0 B/op	+0.2% (stable)
QueryRange cache-bypass allocations	28.0 allocs/op	28.0 allocs/op	0.0% (stable)
Labels cache-hit CPU cost	895.3 ns/op	901.2 ns/op	+0.7% (stable)
Labels cache-hit memory	48.0 B/op	48.0 B/op	0.0% (stable)
Labels cache-hit allocations	3.0 allocs/op	3.0 allocs/op	0.0% (stable)
Labels cache-bypass CPU cost	1535.0 ns/op	1561.0 ns/op	+1.7% (stable)
Labels cache-bypass memory	399.0 B/op	400.0 B/op	+0.3% (stable)
Labels cache-bypass allocations	14.0 allocs/op	14.0 allocs/op	0.0% (stable)

State

Coverage, compatibility, and sampled performance are reported here from the same PR workflow.
This is a delta report, not a release gate by itself. Required checks still decide merge safety.
Performance is a smoke comparison, not a full benchmark lab run.
Delta states use the same noise guards as the quality gate (percent + absolute + low-baseline checks), so report labels match merge-gate behavior.

github-actions · 2026-10-12T11:21:22Z

✅ Visual smoke: passed with warnings

Base 14e13dc6e5 vs PR effafa8e2e, Grafana through the proxy builds and Loki on one fresh stack; run.
Loaded again, once and more patiently, after a first-pass difference or an unsettled page: explore-absent-over-time 15m, explore-absent-over-time 1h, explore-absent-over-time 6h.
Core set: 5 capture(s) at 1h. Detailed set: 111 entries (explore-logs-plain, explore-metric-plain, lvl-dd-logs-pill-json, lvl-dd-logs-pill-otel, lvl-dd-fields-pill, lvl-dd-field-method-pill, lvl-dd-field-pipeline-pill, lvl-dd-logs-pill-logfmt, ...). 237 capture(s), 15m/1h/6h, 24h and 7d stay out of PR runs.

Fix proof

Captures of the exact cases this pull request adds or fixes (registry case, base | PR | Loki). The verdict compares the PR with Loki where the base did not match. The gate fails a case unless one capture is fixed or improved, or, beyond the Loki data window, the base failed where Loki and the PR both answered with data (the label does not excuse it; only visual: none on the case does).

semantics/absent-over-time-semantics: 🟡 improved

page	range	base = PR (data)	vs Loki (PR)	verdict
explore-absent-over-time	15m	differs (1)	1 difference(s), 1 on base too	🟡 improved
explore-absent-over-time	1h	differs (1)	1 difference(s), 1 on base too	🟡 improved
explore-absent-over-time	6h	differs (1)	n/a (beyond the Loki window)	⚪ undecided

explore-absent-over-time-15m: improved

explore-absent-over-time-1h: improved

explore-absent-over-time-6h: undecided

Regression check

page	range	set	base = PR (data)	vs Loki (PR)	pixel diff	result
dd-fields	1h	core	✅ identical	identical (4 explained, 1 history-dependent)	0.12%	⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-landing	1h	core	✅ identical	identical (3 explained)	0.14%	✅
dd-service-logs	1h	core	✅ identical	identical (3 explained, 1 history-dependent)	0.12%	⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
explore-logs-plain	1h	core	✅ identical	identical (1 explained)	0.07%	✅
explore-metric-plain	1h	core	✅ identical	identical (1 explained)	0.04%	✅
dd-field-pipeline	15m	detailed	✅ identical	identical (3 explained, 1 history-dependent)	0.11%	✅
dd-field-pipeline	1h	detailed	✅ identical	identical (3 explained, 1 history-dependent)	0.12%	⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-field-pipeline	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.11%	✅
dd-field-service_version	15m	detailed	✅ identical	identical (3 explained, 1 history-dependent)	0.11%	✅
dd-field-service_version	1h	detailed	✅ identical	identical (3 explained, 1 history-dependent)	0.12%	⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-field-service_version	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.11%	✅
dd-fields	15m	detailed	✅ identical	identical (4 explained, 1 history-dependent)	0.11%	✅
dd-fields	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.11%	✅
dd-labels	15m	detailed	✅ identical	identical (3 explained, 1 history-dependent)	0.11%	✅
dd-labels	1h	detailed	✅ identical	identical (3 explained, 1 history-dependent)	0.12%	⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-labels	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.11%	✅
dd-labels-namespace	15m	detailed	✅ identical	identical (3 explained, 1 history-dependent)	0.11%	✅
dd-labels-namespace	1h	detailed	✅ identical	identical (3 explained, 1 history-dependent)	0.12%	⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-labels-namespace	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.11%	✅
dd-landing	15m	detailed	✅ identical	identical (3 explained)	0.14%	✅
dd-landing	6h	detailed	✅ identical	n/a (beyond the Loki window)	2.14%	✅
dd-patterns	15m	detailed	✅ identical	identical (3 explained, 1 history-dependent)	0.11%	✅
dd-patterns	1h	detailed	✅ identical	identical (3 explained, 1 history-dependent)	6.66%	⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
dd-patterns	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.11%	✅
dd-service-logs	15m	detailed	✅ identical	identical (3 explained, 1 history-dependent)	0.11%	✅
dd-service-logs	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.11%	✅
explore-A	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-A	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-A	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-B	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-B	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-B	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-C	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-C	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-C	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-D	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-D	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-D	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-DLV5	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-DLV5	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-DLV5	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-E	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-E	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-E	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-F	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-F	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-F	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-absent-over-time	15m	detailed	differs (1): query ('absent_over_time({service_name="api-gateway"} \| json	1 difference(s), 1 on base too	0.97%	✅ improved (closer to Loki)
explore-absent-over-time	1h	detailed	differs (1): query ('absent_over_time({service_name="api-gateway"} \| json	1 difference(s), 1 on base too	1.15%	✅ improved (closer to Loki)
explore-absent-over-time	6h	detailed	differs (1): query ('absent_over_time({service_name="api-gateway"} \| json	n/a (beyond the Loki window)	1.15%	✅ improved (closer to Loki)
explore-empty-pattern-filter	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.07%	✅
explore-empty-pattern-filter	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.07%	✅
explore-empty-pattern-filter	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.07%	✅
explore-extracted-filter-metric	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-extracted-filter-metric	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-extracted-filter-metric	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-extracted-group-metric	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-extracted-group-metric	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-extracted-group-metric	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-extracted-metric	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-extracted-metric	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-extracted-metric	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-json-error-filter-metric	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-json-error-filter-metric	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-json-error-filter-metric	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-label-browser	15m	detailed	✅ identical	identical	0.02%	✅
explore-label-browser	1h	detailed	✅ identical	identical	0.02%	✅
explore-label-browser	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.02%	✅
explore-label-replace-over-binary	15m	detailed	✅ identical	2 difference(s), 2 on base too	0.04%	✅
explore-label-replace-over-binary	1h	detailed	✅ identical	2 difference(s), 2 on base too	0.04%	✅
explore-label-replace-over-binary	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-log-context	15m	detailed	✅ identical	2 difference(s), 2 on base too	0.04%	✅
explore-log-context	1h	detailed	✅ identical	2 difference(s), 2 on base too	0.04%	✅
explore-log-context	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-logfmt-metric	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-logfmt-metric	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-logfmt-metric	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-logs-extracted	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.07%	✅
explore-logs-extracted	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.07%	✅
explore-logs-extracted	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.07%	✅
explore-logs-json	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.07%	✅
explore-logs-json	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.07%	✅
explore-logs-json	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.07%	✅
explore-logs-json-extraction	15m	detailed	✅ identical	1 difference(s), 1 on base too; 1 explained	0.07%	✅
explore-logs-json-extraction	1h	detailed	✅ identical	1 difference(s), 1 on base too; 1 explained	0.07%	✅
explore-logs-json-extraction	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.07%	✅
explore-logs-line-filter-or	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.07%	✅
explore-logs-line-filter-or	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.07%	✅
explore-logs-line-filter-or	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.07%	✅
explore-logs-line-format	15m	detailed	✅ identical	1 difference(s), 1 on base too; 1 explained	0.07%	✅
explore-logs-line-format	1h	detailed	✅ identical	1 difference(s), 1 on base too; 1 explained	0.07%	✅
explore-logs-line-format	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.07%	✅
explore-logs-multivalue-regexp	15m	detailed	✅ identical	identical (1 explained)	0.07%	✅
explore-logs-multivalue-regexp	1h	detailed	✅ identical	identical (1 explained)	0.07%	✅
explore-logs-multivalue-regexp	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.07%	✅
explore-logs-plain	15m	detailed	✅ identical	identical (1 explained)	0.07%	✅
explore-logs-plain	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.07%	✅
explore-logs-plain-level	15m	detailed	✅ identical	1 difference(s), 1 on base too; 1 explained	0.07%	✅
explore-logs-plain-level	1h	detailed	✅ identical	1 difference(s), 1 on base too; 1 explained	0.07%	✅
explore-logs-plain-level	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.07%	✅
explore-max-by-count-over-time-sliding	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-max-by-count-over-time-sliding	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-max-by-count-over-time-sliding	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-metric-line-filter-or	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-metric-line-filter-or	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-metric-line-filter-or	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-metric-parenthesized	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-metric-parenthesized	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-metric-parenthesized	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-metric-plain	15m	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-metric-plain	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-nested-aggregation-outer-by	15m	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-nested-aggregation-outer-by	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-nested-aggregation-outer-by	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-pattern-filter-anchored	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.07%	✅
explore-pattern-filter-anchored	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.07%	✅
explore-pattern-filter-anchored	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.07%	✅
explore-plain-level-metric	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-plain-level-metric	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-plain-level-metric	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-sliding-sum-without-stream-fold	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-sliding-sum-without-stream-fold	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-sliding-sum-without-stream-fold	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-sum-by-over-binary	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-sum-by-over-binary	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-sum-by-over-binary	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-sum-without-comparison-inside	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-sum-without-comparison-inside	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-sum-without-comparison-inside	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-sum-without-count	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-sum-without-count	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-sum-without-count	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-sum-without-rate	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-sum-without-rate	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-sum-without-rate	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-topk-by-per-step	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-topk-by-per-step	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-topk-by-per-step	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-topk-sum-without	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-topk-sum-without	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-topk-sum-without	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-topk-without-keeps-labels	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-topk-without-keeps-labels	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-topk-without-keeps-labels	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-unwrap-conversion-error	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-conversion-error	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-conversion-error	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-unwrap-duration-seconds	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-duration-seconds	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-duration-seconds	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-unwrap-first-last-under-sum	15m	detailed	✅ identical	2 difference(s), 2 on base too	0.04%	✅
explore-unwrap-first-last-under-sum	1h	detailed	✅ identical	2 difference(s), 2 on base too	0.04%	✅
explore-unwrap-first-last-under-sum	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-unwrap-max-over-time-end-sample	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-max-over-time-end-sample	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-max-over-time-end-sample	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-unwrap-missing-label	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-missing-label	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-missing-label	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-unwrap-quantile-over-time	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-quantile-over-time	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-quantile-over-time	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-unwrap-rate-counter-one-sample	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-rate-counter-one-sample	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-rate-counter-one-sample	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-unwrap-rate-sum	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-rate-sum	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-rate-sum	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-unwrap-sum	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-sum	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-sum	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-unwrap-sum-over-time-sliding-window	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-sum-over-time-sliding-window	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-sum-over-time-sliding-window	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-unwrap-unit-string	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-unit-string	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-unwrap-unit-string	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-volume-label-filter-regexp	15m	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-volume-label-filter-regexp	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-volume-label-filter-regexp	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-volume-multivalue-regexp	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-volume-multivalue-regexp	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-volume-multivalue-regexp	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-without-nested-under-aggregation	15m	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-without-nested-under-aggregation	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
explore-without-nested-under-aggregation	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
explore-without-over-inner-aggregation	15m	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-without-over-inner-aggregation	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
explore-without-over-inner-aggregation	6h	detailed	✅ identical	n/a (beyond the Loki window)	0.04%	✅
lvl-dd-field-method-pill	1h	detailed	✅ identical	1 difference(s), 1 on base too; 2 explained	0.11%	✅
lvl-dd-field-pipeline-pill	1h	detailed	✅ identical	1 difference(s), 1 on base too; 2 explained, 1 history-dependent	0.12%	⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
lvl-dd-fields-pill	1h	detailed	✅ identical	25 difference(s), 25 on base too; 4 explained	0.11%	✅
lvl-dd-label-pod-pill	1h	detailed	✅ identical	1 difference(s), 1 on base too; 2 explained, 1 history-dependent	0.12%	⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
lvl-dd-labels-pill	1h	detailed	✅ identical	2 difference(s), 2 on base too; 2 explained	0.11%	✅
lvl-dd-logs-pill-json	1h	detailed	✅ identical	1 difference(s), 1 on base too; 2 explained	0.11%	✅
lvl-dd-logs-pill-logfmt	1h	detailed	✅ identical	identical (4 explained, 1 history-dependent)	0.11%	⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
lvl-dd-logs-pill-multi-json	1h	detailed	✅ identical	2 difference(s), 2 on base too; 2 explained, 1 history-dependent	0.11%	⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
lvl-dd-logs-pill-otel	1h	detailed	✅ identical	1 difference(s), 1 on base too; 2 explained, 1 history-dependent	0.12%	⚠️ history-dependent difference, not gated (1): resource /api/datasources/uid/*/res
lvl-dd-logs-pill-plaintext-bodyonly	1h	detailed	✅ identical	2 difference(s), 2 on base too; 2 explained, 1 history-dependent	0.11%	✅
lvl-explore-count-warn-by-service	1h	detailed	✅ identical	2 difference(s), 2 on base too	0.04%	✅
lvl-explore-logs-dl-eq	1h	detailed	✅ identical	identical (1 explained)	0.07%	✅
lvl-explore-logs-dl-error-plaintext-bodyonly	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
lvl-explore-logs-dl-neq-all	1h	detailed	✅ identical	3 difference(s), 3 on base too	0.07%	✅
lvl-explore-logs-dl-plaintext	1h	detailed	✅ identical	2 difference(s), 2 on base too	0.07%	✅
lvl-explore-logs-dl-regexp	1h	detailed	✅ identical	identical (1 explained)	0.07%	✅
lvl-explore-logs-dl-unknown-bodyonly	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
lvl-explore-logs-dl-unknown-or-error	1h	detailed	✅ identical	3 difference(s), 3 on base too	0.07%	✅
lvl-explore-logs-drop-dl	1h	detailed	✅ identical	3 difference(s), 3 on base too	0.07%	✅
lvl-explore-logs-drop-level	1h	detailed	✅ identical	2 difference(s), 2 on base too	0.07%	✅
lvl-explore-logs-json-dl	1h	detailed	✅ identical	identical (1 explained)	0.07%	✅
lvl-explore-logs-json-level	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.07%	✅
lvl-explore-logs-json-level-extracted-stored	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.07%	✅
lvl-explore-logs-keep-dl	1h	detailed	✅ identical	2 difference(s), 2 on base too	0.07%	✅
lvl-explore-logs-level-label-stored	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.07%	✅
lvl-explore-logs-level-selector-stored	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.07%	✅
lvl-explore-logs-logfmt-level-extracted-stored	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.07%	✅
lvl-explore-logs-logfmt-level-regexp	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.07%	✅
lvl-explore-rate-errors-by-service	1h	detailed	✅ identical	2 difference(s), 2 on base too	0.04%	✅
lvl-explore-topk-errors-by-service	1h	detailed	✅ identical	2 difference(s), 2 on base too	0.04%	✅
lvl-explore-volume-dl-neq	1h	detailed	✅ identical	2 difference(s), 2 on base too	0.04%	✅
lvl-explore-volume-dl-prefilter-json	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
lvl-explore-volume-dl-regexp	1h	detailed	✅ identical	1 difference(s), 1 on base too; 1 explained	0.04%	✅
lvl-explore-volume-dl-unknown-bodyonly	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
lvl-explore-volume-dl-unknown-or-error	1h	detailed	✅ identical	2 difference(s), 2 on base too	0.04%	✅
lvl-explore-volume-drop-level	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
lvl-explore-volume-json-by-dl	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
lvl-explore-volume-json-dl-filter	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
lvl-explore-volume-json-error-by-dl	1h	detailed	✅ identical	identical (1 explained)	0.04%	✅
lvl-explore-volume-json-level-filter	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
lvl-explore-volume-keep-dl	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
lvl-explore-volume-level-extracted-stored	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
lvl-explore-volume-level-label-stored	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
lvl-explore-volume-logfmt-dl	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
lvl-explore-volume-logfmt-dl-filter	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
lvl-explore-volume-plaintext	1h	detailed	✅ identical	1 difference(s), 1 on base too	0.04%	✅
tail-env	live	detailed	✅ identical	entries in the window: PR 9304, Loki 2467, in both 2467	6.97%	✅
tail-json-filter	live	detailed	✅ identical	entries in the window: PR 1588, Loki 566, in both 566	5.48%	✅

Gate: a base-vs-PR data difference passes when the PR is closer to Loki, and fails when the PR diverges from Loki or Loki cannot decide it (at a range Loki does not hold, a difference passes when the same page proved that query to answer like Loki at a shorter range; otherwise label the pull request visual-change-expected to accept an intended change). Also failing: a panel empty on the PR but not on the base, an error banner, panel error, error answer or new "No data" panel on the PR, a PR side that never settled, a value difference that did not reproduce on the recapture. A delayed first-pass request and pixel differences only warn.

dd-fields-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3

history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3A00.000Z&query=%7Bservic: body: content differs (rows 5 vs 2)
vs Loki, explained: query ('sum by (service_name_extracted) (count_over_time({service_name="otel-collector"} | servi: request only on one side (right) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision)
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-service-logs-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3

history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3A00.000Z&query=%7Bservic: body: content differs (rows 5 vs 2)
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-field-pipeline-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3

history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3A00.000Z&query=%7Bservic: body: content differs (rows 5 vs 2)
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-field-service_version-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3

history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3A00.000Z&query=%7Bservic: body: content differs (rows 5 vs 2)
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-labels-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3

history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3A00.000Z&query=%7Bservic: body: content differs (rows 5 vs 2)
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-labels-namespace-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3

history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3A00.000Z&query=%7Bservic: body: content differs (rows 5 vs 2)
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

dd-patterns-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3

history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3A00.000Z&query=%7Bservic: body: content differs (rows 5 vs 2)
pixel diff 6.7%
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 13 vs 15) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

explore-absent-over-time-6h: no Loki data at this range; the same differences match Loki at a shorter range (1): query ('absent_over_time({service_na

no Loki data at this range; the same differences match Loki at a shorter range (1): query ('absent_over_time({service_name="api-gateway"} | json | nonexistant_field="foo" [$__auto]: series sets differ: 0 only left [], 1 only right ['(\'A\', \'\

lvl-dd-field-pipeline-pill-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3

history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3A00.000Z&query=%7Bservic: body: content differs (rows 5 vs 2)
vs Loki: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 36 vs 15)
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

lvl-dd-label-pod-pill-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3

history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3A00.000Z&query=%7Bservic: body: content differs (rows 5 vs 2)
vs Loki: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 36 vs 15)
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

lvl-dd-logs-pill-logfmt-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3

history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3A00.000Z&query=%7Bservic: body: content differs (rows 10 vs 4)
vs Loki, explained: query ('{service_name="payment-service"} | detected_level="warn" | json | logfmt | drop __er: ('logsPanelQuery', '', '{}'): content differs (rows 1000 vs 1000) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision)
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 8 vs 9) -- explained: Loki's _extracted suffix for a key named like a stream label, where the build does not return it (detected_fields: open, semantics/detected-fields-extracted-suffix; service/service.name: owner decision); the proxy also returns data, status; Loki lists fields in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 8 vs 8) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

lvl-dd-logs-pill-multi-json-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3

history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3A00.000Z&query=%7Bservic: body: content differs (rows 50 vs 4)
vs Loki: query ('sum(count_over_time({service_name="frontend-ssr"} [$__auto])) by (detected_level)',: request only on one side (left)
vs Loki: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 10 vs 0)
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 8 vs 8) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

lvl-dd-logs-pill-otel-1h: history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3

history-dependent difference, not gated (1): resource /api/datasources/uid/*/resources/patterns?end=2026-10-29T15%3A08%3A00.000Z&query=%7Bservic: body: content differs (rows 5 vs 2)
vs Loki: resource /api/datasources/uid/*/resources/detected_fields?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 36 vs 15)
vs Loki, explained: resource /api/datasources/uid/*/resources/detected_labels?end=2026-10-29T15%3A08%3A00.000Z&query=%7: body: content differs (rows 9 vs 9) -- explained: sampled cardinality: Loki counts every stream its ingesters hold, not the requested window (pkg/ingester/instance.go LabelsWithValues), the proxy the window's last 5 minutes (metadataMaxFieldNamesWindow); the label names match; the proxy also returns data, limit, status; Loki lists labels in no fixed order
vs Loki, explained: resource /api/datasources/uid/*/resources/drilldown-limits: body: content differs (rows 0 vs 0) -- explained: configuration probe: each backend publishes its own limits, version and settings (the proxy's pattern persistence)

Core set montages (3): base | PR | Loki

dd-landing-1h

explore-logs-plain-1h

explore-metric-plain-1h

All montages of this run: pr-visuals/pr-737.

🧭 New baseline candidate

This pull request got the baseline run: every page of bench/visual/spec.json was captured at its own ranges. The full candidate is the visual-baseline-candidate artifact of https://github.com/ReliablyObserve/loki-vl-proxy/actions/runs/40898298377; only bounded inline review images were published, and the candidate was not accepted as a baseline.

szibis · 2026-10-29T12:17:45Z

Proof for current HEAD effafa8e2e4ec297c9500eb4a5adac5df5b9dcda (rebased onto main 14e13dc6e5cea0222aa51f8bb06f67d6348b2889 after the v2.46.0 release):

All twelve PR commits are signed, the branch is conflict-free, and the worktree is clean. fix: absent_over_time is the series valued 1 at the steps without data, after a parser, under an aggregation and on Loki's step grid #737's changelog entries are in Unreleased; fix: an aggregation over a binary expression aggregates the whole binary, like Loki #741 remains in v2.46.0.
Current-head conformance gate passed (359 registry items; 228 tests declaring coverage; 190 wired). The quality summary was regenerated after reconciling fix: an aggregation over a binary expression aggregates the whole binary, like Loki #741's registry changes.
Current-head focused TestAbsentOverTime* Go tests, all 155 visual harness tests, 307 CI configuration tests, actionlint, and the conformance gate passed. Full proxy and translator tests passed on the preceding HEAD before the release-only rebase (319.894 s and 8.096 s). On the prior 231-capture fix: absent_over_time is the series valued 1 at the steps without data, after a parser, under an aggregation and on Loki's step grid #737 artifact, the trusted v2.46.0 base renderer selected exactly 15 inline main | PR | Loki montages; the new bounded selector staged those 15, and the trusted older publisher accepted all.
Hosted CI on this exact SHA finished with 72 successful checks, zero failures, one neutral advisory scan and four expected skips. Full Go, race, fuzz, e2e, Loki/VictoriaLogs/Grafana compatibility, performance A/B, quality, and security passed. Security / runtime passed on rerun after its first attempt reached a ZAP daemon startup timeout on the second target; no source change was needed.
Visual run 40898298377: visual-smoke, visual-publish, and visual-comment all passed. The CI bot comment shows base 14e13dc6e5 vs PR effafa8e2e, 237 captures, 18 unique inline main | PR | Loki images, and three open fix-proof images. It correctly grades the 15m and 1h absent-over-time cases “improved”: the absence result matches Loki, while one other difference is shared with the base. The current 1h PNG returned HTTP 200 image/png (30,552 bytes).
The previous HEAD 13e0f3b5 passed local full proxy/translator and fresh Loki/VictoriaLogs TestCompat_AbsentOverTime* e2e tests. Its visual, performance A/B, and quality results are historical controls, not acceptance proof for this rebased HEAD.

Current main 14e13dc6e5 | PR effafa8e2e | Loki montage (1h), published at immutable pr-visuals commit e610898290ddf9a89379e92a7b9b6357dde1c800. At 1h, the base showed “No data”; the PR and Loki both show the absence series at 1.

…a, after a parser, under an aggregation and on Loki's step grid Loki answers one series, labelled by the selector's single equality matchers, at each step whose window (t-range, t] holds no line the pipeline keeps, and nothing where it does. With a parser stage the proxy sent the query to the raw-row bare-parser evaluator and returned a series per stream and parsed label set; without one it read VictoriaLogs buckets as Loki windows, ignored a range that is not the step, stepped on an epoch grid, let older data hide an instant absence and answered the inner count under a vector aggregation. The presence of samples now comes from an ungrouped sum(count_over_time) (sum(rate) with unwrap) over VictoriaLogs stats, inverted on start, start+step, ... end, folded through vector aggregations as Loki folds one series; degraded answers are passed on, not inverted.

…s an unconvertible unwrap like rate, and says which route reads rows A start with milliseconds put the float-second presence points off the exact nanosecond step grid, which read every step as absent. The differential now starts a third of its scenarios on a millisecond. An unwrapped value Loki rejects now answers Loki's 400 pipeline error for absent_over_time, as for rate; the proxy test table and the e2e conversion-error cases cover it.

…of page

…ith a grouping, and on a warned answer A sort around absent_over_time keeps the series whole and a label_replace rewrites the absent series' labels instead of the inner count. A grouping on it is Loki's 400 parse error. A multi-tenant request is absent only where no tenant has a line, merged from each tenant's answer with no __tenant_id__ label, as Loki merges the tenants' samples before evaluating. A presence answer whose body carries a warnings array is passed on, not inverted.

Use the current label replacement helper for absent series after rebasing and regenerate quality summaries from saved evidence.

szibis added a commit that referenced this pull request Oct 12, 2026

chore(conformance): record #737 as the fix of absent-over-time-semantics

fff35f7

github-actions Bot added size/XL Extra large change scope/proxy Proxy core scope/docs Documentation scope/tests Tests bugfix Bug fix and removed size/XL Extra large change labels Oct 12, 2026

github-actions Bot added size/XL Extra large change scope/translator LogQL translator and removed size/XL Extra large change labels Oct 12, 2026

szibis marked this pull request as ready for review October 29, 2026 10:35

szibis added a commit that referenced this pull request Oct 29, 2026

chore(conformance): record #737 as the fix of absent-over-time-semantics

d2aaa48

szibis force-pushed the fix/absent-over-time-semantics branch from 14eb3b1 to 13e0f3b Compare October 29, 2026 11:29

github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 29, 2026

szibis added a commit that referenced this pull request Oct 29, 2026

chore(conformance): record #737 as the fix of absent-over-time-semantics

25ad282

szibis force-pushed the fix/absent-over-time-semantics branch from 13e0f3b to e5ea6a4 Compare October 29, 2026 14:36

github-actions Bot added size/XL Extra large change scope/ci CI/CD and removed size/XL Extra large change labels Oct 29, 2026

szibis force-pushed the fix/absent-over-time-semantics branch from 817f6ff to 377c2c0 Compare October 29, 2026 14:48

github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 29, 2026

szibis and others added 11 commits October 29, 2026 15:56

docs: absent_over_time conversion errors are detected; visual fix pro…

9c09af1

…of page

chore(conformance): record #737 as the fix of absent-over-time-semantics

2c57ca1

fix: align absent-over-time with current label transforms

4f61111

Use the current label replacement helper for absent series after rebasing and regenerate quality summaries from saved evidence.

docs: refresh conformance summary after rebasing absent-over-time fix

28ecb27

ci: publish bounded inline visual proof for baseline runs

69f6fb5

test: verify bounded baseline visual publishing scope

dc6fa4c

docs: link full visual artifact beside bounded inline proof

05b3ef0

docs: keep PR 737 fixes unreleased after v2.46.0

f134ef8

szibis force-pushed the fix/absent-over-time-semantics branch from 0e60eab to f134ef8 Compare October 29, 2026 14:57

github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 29, 2026

Publish bounded visual proof with legacy base renderer

effafa8

github-actions Bot added size/XL Extra large change and removed size/XL Extra large change labels Oct 29, 2026

szibis merged commit bb8f348 into main Oct 29, 2026
81 of 82 checks passed

szibis deleted the fix/absent-over-time-semantics branch October 29, 2026 16:01

szibis mentioned this pull request Oct 29, 2026

fix: reject group(), label_join() and other names Loki does not have, and accept keywords in any case, like Loki #748

Merged

Navigation Menu

Repository navigation

fix: absent_over_time is the series valued 1 at the steps without data, after a parser, under an aggregation and on Loki's step grid - #737

fix: absent_over_time is the series valued 1 at the steps without data, after a parser, under an aggregation and on Loki's step grid#737
szibis merged 12 commits into
mainfrom
fix/absent-over-time-semantics

szibis commented Oct 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 12, 2026 •

edited

Loading

Uh oh!

szibis commented Oct 29, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

szibis commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tests

Verification on rebased head

Main | PR | Loki visual comparison

Uh oh!

github-actions Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚪ Performance A/B: no change beyond noise

Uh oh!

github-actions Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Quality Report

Coverage and tests

Compatibility

Performance smoke

State

Uh oh!

github-actions Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Visual smoke: passed with warnings

Fix proof

Regression check

Uh oh!

szibis commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

szibis commented Oct 12, 2026 •

edited

Loading

github-actions Bot commented Oct 12, 2026 •

edited

Loading

github-actions Bot commented Oct 12, 2026 •

edited

Loading

github-actions Bot commented Oct 12, 2026 •

edited

Loading

szibis commented Oct 29, 2026 •

edited

Loading