Repository navigation
fix: Loki-compatible profile holds requests to Loki's contract; e2e stack tests it by default - #624
Merged
Merged
Conversation
szibis
force-pushed
the
feat/loki-compat-test-mode
branch
from
September 24, 2026 10:24
c77e3e2 to
ea8fc01
Compare
github-actions
Bot
added
size/XL
Contributor
🟢 Performance A/B: faster, no regression48 shape×range — 2 ✅ fixed · 46 ⚪ same · results vs Loki: 40 same, 0 new difference(s), 8 pre-existing
46 unchanged shape×range (within noise)
Cold (first-run) timings — what a user opening a panel sees
Why these shapes
base |
Contributor
PR Quality ReportCompared against base branch Coverage and tests
Compatibility
Performance smokeLower CPU cost (
State
|
szibis
force-pushed
the
feat/loki-compat-test-mode
branch
from
September 24, 2026 10:38
ea8fc01 to
ba497e0
Compare
github-actions
Bot
added
size/XL
szibis
force-pushed
the
feat/loki-compat-test-mode
branch
from
September 30, 2026 11:20
62e9ecf to
d14a807
Compare
github-actions
Bot
added
size/XL
szibis
force-pushed
the
feat/loki-compat-test-mode
branch
from
September 30, 2026 13:38
d14a807 to
bbfd677
Compare
github-actions
Bot
added
size/XL
…tack tests it by default - Dotted LogQL names (k8s.namespace.name) are Loki's exact 400 parse error (line, column, expected tokens) in the default underscores+translated profile, on query/query_range/tail and every selector parameter; hybrid, native and passthrough keep accepting them. - The label endpoints ignore limit/offset/search like Loki unless -label-values-indexed-cache opts into the browse window. - -label-values-max-response-bytes bounds label values reads with Loki's ResourceExhausted error; a VictoriaLogs response aborted after its headers is a 504/502, not a raw transport error. - Error bodies carry "message", so Grafana shows Loki's error text. - -labels-cache-warm turns off the label keep-warm scans. - e2e stack: Grafana Explore and Logs Drilldown datasources run the Loki-compatible profile, the hybrid profile is a named datasource, only three variants warm, the peer ring is behind the "peers" profile.
- ".5" right after a Loki keyword (offset.5h, > bool.5) is a number, not a dotted name. - The aborted-response error unwraps its transport cause, so deadline retries and graceful timeouts still see context.DeadlineExceeded. - The label values cap also bounds background refreshes, and coalesced reads with different caps no longer share a flight. - Docs name the entry points and the GET scope of the abort guard; open case for detected_fields offering dotted JSON-body keys; A/B control results saved.
…Loki-matching rejections A candidate that now fails exactly as Loki does counts as fixed in the A/B report; the Loki-compatible profile behaviour names its implementation so the per-PR shape selection reaches its shapes.
…trix tested -logql-dotted-names (auto|reject|accept) and -label-browse-extensions (auto|on|off) decouple dotted-name handling and the label browse window from the metadata mode; -error-response-message-field restores the previous error body. With dotted names rejected, detected_fields names a dotted JSON key by Loki's sanitized label with the key in jsonPath. The translated-metadata e2e variant is restored next to the new OTel hybrid variant, the option matrix is documented and walked by unit and e2e tests, and the @regression series-cap spec reads the published limit.
…lects Loki's json and logfmt parsers sanitize each key into a label name (http.method becomes http_method) while unpack_json keeps the original key, so | json | http_method="GET" matched nothing when the name was not a stored VictoriaLogs field. A label filter after a json or logfmt parser now accepts every original key that sanitizes to its name.
…, service_name values cap, peer ring Existence checks keep their single key for the Drilldown fast paths; candidate keys with adjacent or edge dots are not offered; pattern captures and label_format destinations are query-local; field names in ip() filters are quoted; detected_fields sanitizes logfmt keys; service_name values honour the response cap and report a failed read; the e2e stack starts its peer ring again.
…abel gives Loki's series A json expression label gets its value copied from the original key. A sanitized by() or unwrap label of a json or logfmt query is copied, right before the stats clause, from each original spelling, so VictoriaLogs stats aggregate a real field (pushdown for range queries). A parser-derived unwrap label drops the lines without it. A bare instant sum over an unwrap collapses its streams. The label-filter regexp anchoring is registered as an open item.
…the Drilldown field-breakdown rewrite
…ead; the label-values guard tests count bucket calls
szibis
force-pushed
the
feat/loki-compat-test-mode
branch
from
September 30, 2026 13:43
bbfd677 to
9c62082
Compare
github-actions
Bot
added
size/XL
…r decoding decodeCompressedHTTPResponse set ContentLength and Uncompressed on a gzip or zstd response after the transport had delivered it, while the transport's read loop reads ContentLength from another goroutine once the body is closed. A failed bucket of a label listing cancels its siblings, which made that a data race under -race. The response keeps the transport's values; the Content-Encoding and Content-Length headers are still dropped. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WqewtSrSe2W95zXFBXkam9
…sting The cap was checked against each VictoriaLogs bucket response, so a listing whose every bucket was under the cap but whose union was over it answered 200, and buckets another request had filled without a cap bypassed it. With the inventory on, the merged listing is now measured as VictoriaLogs would have sent it for one call over the whole range and fails with Loki's ResourceExhausted 500 above the cap, agreeing with the inventory-off path at the boundary. The inventory's singleflight key now carries the cap, so an uncapped request never inherits a capped leader's failure. The guard tests run with the inventory off again and assert the exact call count; new tests cover the inventory (a poisoned sealed bucket is never cached or merged into a partial answer). The e2e compose restores -metadata-default-lookback=1h on the translated-metadata proxy, with a guard test that no proxy service passes 0. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WqewtSrSe2W95zXFBXkam9
github-actions
Bot
added
size/XL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
The e2e stack's Grafana-facing proxies ran
-metadata-field-mode=hybrid, so Explore offeredk8s.namespace.namebesidek8s_namespace_name. A user built{env="production"} | json | pipeline=metrics/prometheus| k8s.namespace.name=monitoring``. Loki v3.7.7 rejects that with400 parse error at line 1, col 64: syntax error: unexpected .. The proxy accepted it (even in translated mode) and ran it on the raw-row path: 3.2 s at 1h, `502 manual range metric row limit exceeded` above ~3h.This PR makes Loki compatibility the stack's primary test target and holds the Loki-compatible profile to Loki's request contract as well as its response contract.
1. Loki compatibility mode is the primary testing target
Loki (via VL proxy)(+ multi-tenant,loki-vl-proxy-underscore) and the Logs Drilldown defaultLoki (via VL proxy patterns autodetect)now run-label-style=underscores -metadata-field-mode=translated. Datasource names, and so UIDs, are unchanged.Loki (via VL proxy OTel hybrid)(loki-vl-proxy-otel-hybrid, port 13111). It replacesloki-vl-proxy-translated-metadata, which duplicated the Loki profile.native metadatais unchanged.TestCompat_StackProfilesMatchDatasources(core group) fails CI if a Grafana-facing or parity datasource leaves the Loki profile, if warm-up flags drift, or if a member of the parity proxy's peer ring does not start with the stack.-compat-profileflag. The three flags already default to the Loki profile, and each also changes one surface on its own. A profile flag would need precedence rules against them. Documented indocs/configuration.md.2. Strict LogQL names in Loki mode
internal/logql/dotted_names.goemulates Loki's lexer (DOT token) and parser state. A dotted name gets Loki's exact message: line, character column, and the expected-token list goyacc prints there (expecting , or ),= or =~ or !~ or !=,=,RANGE or |,),IDENTIFIER or (...;name.5→unexpected NUMBER).match[], index/, patterns, detected_), before any VictoriaLogs call.limit/offset/searchare now ignored in Loki mode, as in Loki (loghttp.ParseLabelQuery), unless-label-values-indexed-cacheopts in. They used to truncate and were forwarded to VictoriaLogs.by()/without()/on()/ignoring()/group_*) are covered by the dotted-name check.message. Grafana shows the JSONmessagefield or the raw body, so Explore displayed the whole JSON instead of Loki's text./labels7d, 39 vs 24 names: VL-only fixture streams (data difference). The fixture helper now fails setup on a rejected push.service_namevalues: Loki over-reports by index-day granularity (TSDBIndex.LabelValuesignores from/through). Proxy answers the exact window. Documented inprofiles/label-endpoints-window-and-data./label/pod/values: new-label-values-max-response-bytes(default 64 MiB, per tenant). Over the cap it answers Loki's500 rpc error: code = ResourceExhausted ... (N vs. LIMIT)naming the flag, and never caches the result.502 chunked line ends with bare LF: VictoriaLogs' hijack-abort after headers is now Loki's504 request timed out...(or a sanitized 502) on GET reads.parsedlabels on queries without a parser.service_name_extractedfor structured metadata that collides with a stream label.3. Background warm-up in the test stack
-labels-cache-warm(default true; Helm, docs, registry).-labels-cache-warm=false.peer-a/-b,vmauth-ring) starts with the stack again: the parity proxy lists them in its static ring, and absent members filled its peer error metrics (214 errors in one core run).label_metadata.go(coalescer key + refresh cap) may need a trivial merge with fix: admit long-range metadata scans adaptively and list labels from a time-bucketed inventory #617.4. Parity suites target Loki mode
TestCompat_DottedNameParseErrorsMatchLoki(20 endpoint×position cases × parity + both Grafana-facing proxies) andTestCompat_LabelBrowseParamsIgnoredLikeLoki.datasourceshard go through Grafana/api/ds/query: dotted error text equals Loki's; structured-metadata keys are Loki's.Tests moved and why
TestDetectedFields_UnderscoreProxy/hybrid_fields_...,detected_field_values_service_dot_nameloki_profile_fields_are_loki_names_onlyon 13102TestGrafanaDrilldown_UnderscoreProxy/drilldown_detected_fields_for_service_otel_hybridsubtest on 13111TestOTelCompatibilityScoredetected_fieldsTestStructuredMetadata_HybridMode...TestStructuredMetadata_TranslatedMode...TestDrilldown_GrafanaResourceContracts/structured_metadata_fields_and_valuesLoki (via VL proxy)... OTel hybrid+ newstructured_metadata_fields_loki_profile.../datasource_proxy_query_range_accepts_dotted_field_filters.../label_values_honor_limitlimitTestOTelDots_ProxyPassthrough/query_with_dotted_*TestDrilldownTrackScorehybrid fieldTestPinnedCompatibilityMatrixMatchesComposeLive proof (shared stack, read-only; my builds on 13271/13273)
Builds: branch vs
main(4d517df), same stack flags +-backend=http://127.0.0.1:19428 -rate-limit-per-second=0 -rate-limit-burst=0, each on its own port with-admin-listen. Listener PIDs verified.Dotted names (1h):
col 64: unexpected .col 112: unexpected .sum by (k8s.namespace.name)col 12 ... expecting , or ){k8s.namespace.name="x"}col 5 ... expecting = or =~ or !~ or !=/labels?query=//series match[]dottedStructured-metadata keys (
{app="otel-collector"}, 15m, categorize-labels):deployment_environment, detected_level, k8s_cluster_name, k8s_namespace_name, k8s_pod_name, service_version, span_id, trace_id. No dotted keys.service_name_extractedvs the proxy'sservice_name, and the proxy's extraparsed.A/B on the final head (
bench/ab, 5 runs, warm p50 unless noted; main = origin/main at ba148c5, branch = this head, each on its own verified port with-admin-listen; Loki direct; shared-stack VictoriaLogs restarts 34→34, so every run is valid). Saved inbench/ab/results/2026-09-30-pr624-final-*.json.loki-profile-otel-metadata(changes: dotted filters answer Loki's 400 instead of 200/502; O2 24h 2.15x faster; every underscore shape equal to main):†= identical answers for windows a minute apart (cache or stale view), timing not like-for-like. O2differsrows: main and branch return identical series and sums, Loki lags at the live edge (it answers 0.00-0.02s, from its results cache). Verdict: 16 shape×range - 6 fixed, 4 faster, 6 same; 0 new differences from Loki.control, 37 shape×range): 3 faster, 34 same, 0 slower; 0 new differences from Loki (31 pre-existing, also on main). A first run of the previous head flaggedtopk rate by app24h 0.17s→0.23s; a 10-run recheck gave main 0.33s vs branch 0.31s (noise).drilldown-breakdown-exact, label and json field breakdowns 1h-7d, from fix: Logs Drilldown label and field breakdowns answer exactly like Loki #622): 16 of 16 same as main; 0 new differences from Loki. The 7d pod and batch_id rows fail on both main and branch (502 manual metric response exceeds 67108864 bytes, Loki also fails or returns nothing at 7d): pre-existing, not this change.stats_query_rangex4 +queryx2 per volume, no raw rows.benchstat (
-count=10, previous head; the code it covers is unchanged):Proxy_QueryRange_*,Proxy_LabelValues_Scale_*,NoCache_QueryRange_LogQueryandValidateLogQL/*.DottedNameErrorcosts ~0.7–1.0 µs and 1 alloc per query (cached validation is unaffected).ValidateLogQLns/op noise (±40%) is on unchanged code.Backend-safety load (previous head; 24 long-range requests at concurrency 8,
/labelsprobe every second):Local e2e on an isolated compose project (
-p lcmode, ports +10000, fresh stack per group, final rebased head, VictoriaLogs restarts 0 in every run, torn down):TestCompat_OptionMatrixAgainstStack,TestCompat_SanitizedJSONKeyFilterMatchesLokiandTestCompat_SanitizedJSONKeyGroupingMatchesLoki.patterns are visible in drilldown for autodetected datasourceshowed Patterns0 on the first attempt, passed on retry, same on both full runs), drilldown-mt, explore-ops, explore-mt, explore-regression (41 passed, 10 skipped), comprehensive all pass.Re-run:
python3 bench/ab/perf_matrix.py --set loki-profile-otel-metadata|control --runs 5 --target main=… --target branch=… --target loki=http://127.0.0.1:13101 --out …thenreport.py summarize … --save bench/ab/results/.Sanitized JSON keys: filters, grouping and unwrap
Loki's json/logfmt parsers sanitize
http.methodtohttp_method; VictoriaLogs'unpack_jsonkeeps the original key. So| json | http_method="GET"returned no lines, and grouping by or unwrapping such a name aggregated a field that does not exist (one unlabelled series, NaN).detected_fieldsof this PR names dotted JSON keys by the sanitized label with the key injsonPath, and Logs Drilldown builds| json label="[\"key\"]"from it, so all of these had to give Loki's series before merging.internal/translator/parsed_key_variants.go): after a json/logfmt parser a label filter accepts every original key that sanitizes to the name (each_read as_or., plus the all-hyphen spelling, up to 6 underscores);=/=~/comparisons hold when any spelling matches,!=/!~/empty checks when every spelling does. Mapped labels, captures,label_formattargets and json expressions keep one key.| copy "x.y" as a; a sanitized by()/unwrap label adds, right before the stats, one| format if ("x.y":*) "<x.y>" as label keep_original_fieldsper spelling (a stored or stream label wins, as in Loki); a parser-derived unwrap label also adds| filter label:*(Loki yields no sample for a line without it). VictoriaLogs stats aggregate a real field, so range queries stay pushed down, no raw rows. An instant baresum(sum_over_time(... | unwrap x))collapses its streams into one series like Loki's sum (count, avg, max, topk keep one value per stream).| json | f!="" ... by (f)(what fix: Logs Drilldown label and field breakdowns answer exactly like Loki #622's field-breakdown rewrite recognises) andlabel!=""keep the exact key; Drilldown names a dotted key through the expression form.TestCompat_SanitizedJSONKeyFilterMatchesLokiandTestCompat_SanitizedJSONKeyGroupingMatchesLokiagainst real Loki (range and instant, plain and Drilldown-tagged, three proxies, the fields-page sequence detected_fields -> breakdown by name and jsonPath, unwrap), registry casesprofiles/sanitized-json-key-filterandprofiles/sanitized-json-key-grouping.| msg=~"a"matches "cat";profiles/label-filter-regexp-anchoring, separate from fix(logql): anchor label-matcher regexps to the whole label value #572); a json-expression label is always present in Loki, so a line without the path forms{a=""}where the proxy returns{}(only without a followinga!="").Fresh-context reviews (Opus), each finding checked: first review fixed logfmt
detected_fieldskeys,ip()field quoting, dotted-edge candidates, pattern/label_formatnames,/label/service_name/valuesoutside the response cap, the e2e peer ring. Second review (grouping/unwrap) found and I fixed: instant outer operators collapsed wrongly (now sum of additive only), a splice into thedetected_levelparser pipes, an unwrap filter leaking across a binary expression, an unwrap filter dropping lines before a laterlabel_format, pipes inside a quoted literal, overwriting a stored label (nowkeep_original_fields); the resolution moved from a regex over the output to the stats clause. Left as is: error-bodymessageon VL-translated error envelopes (rare path);| unwrap xon a non-sanitized, non-expression label still yields a NaN series for lines without it.Gates
go build,go vet ./...(+-tags=e2e),gofmt,golangci-lint(0 issues),go test -count=1 ./....go run ./cmd/configdoc -check,validate_helm_flags.sh,check_conformance.pyok,docker compose configvalid.Breaking
limit/offset/searchunless-label-values-indexed-cache=true.