Repository navigation
fix: regexp label matchers select what Loki selects, on selectors and label filters - #909
Merged
Merged
Conversation
github-actions
Bot
added
size/XL
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
A LogQL label matcher is a full-value match: Loki builds it with
Prometheus' labels.NewMatcher, which compiles a MatchRegexp as
"^(?s:" + v + ")$". VictoriaLogs' field:~"re" is an unanchored match
and the proxy passed the pattern through verbatim, so every =~ / !~
matcher was silently widened: {namespace=~"nch"} also selected "anch"
and "nch-b", {app=~"a|b"} matched "ab".
AnchorLabelMatcherRegex wraps the pattern once, in internal/logsql, and
every emitter of a label-matcher regexp uses it: the stream selector,
pipeline label filters, the derived service_name matcher and label
filter (which already anchored with the same form) and the multi-tenant
__tenant_id__ fan-out, which compiled the bare pattern with Go regexp.
The form is Prometheus' own ^(?s:...)$, so the Go-compiled call site is
literally what Prometheus compiles too.
Line filters (|~, !~) are substring regexps in Loki and are left alone.
A parity case with a prefix matcher ({app=~"api"}, empty on Loki) joins
the query-semantics matrix, and the docs record that results shrink for
existing substring-style matchers.
…st paths Builds on the selector anchoring: a stream-selector matcher is a Prometheus matcher, ^(?s:re)$ (vendor/.../model/labels/regexp.go:76), but a label filter after a parser is not. Loki runs RegexSimplifier.Simplify first (pkg/logql/log/filter.go:645 parseRegexpFilter, :696 Simplify, :309 newOrFilter) and anchors only what it cannot simplify, as ^(?:re)$ without the s flag: a literal or a literal alternation compares whole values, api.*, .*api and a factored alternation (api|api-gateway, prod-eu|prod) compare by substring, =~"" keeps every line and GET| equals GET. Wrapping every label filter in ^(?s:re)$ answered 24 of 80 label-filter cases differently from Loki v3.7.7; lokiLabelFilterPattern ports the simplifier and every label filter site uses it: translateSingleLabelFilter, the sanitized-key variants (parsedKeyMatchesEmpty), the service_name label filter, and the ordered JSON metric pushdown and Loki profile conditions (NewLabelFilterCondition), which compiled ^(?:re)$. VictoriaLogs compiles '.' to match newlines (regexutil parses with DotNL), so an anchored label filter with a '.' is spelled ^(?-s:re)$. Where a cheaper filter means the same, the proxy sends it: a whole-value literal as an exact filter, .* and .+ as VictoriaLogs' own match-all and non-empty filters, a .*-wrapped literal as the substring regexp, and a label filter's literal alternation as exact filters joined by OR (Loki's OR of equality checks; VictoriaLogs' anchored alternation costs about 100 ns per value against 12 ns for main's substring match and 1 ns for an exact compare, measured on lib/regexutil v1.53.0). A literal with a control character keeps the regexp form, which is quoted with Go's rules. translatedFilterFieldOp reads the field of an OR group so the Drilldown same-field dedupe keeps its key. Evidence: the Loki answers were produced by Loki v3.7.7's own code (syntax.ParseMatchers and the label filter's pipeline) for 40 patterns over 28 values, and the emitted filters were run on VictoriaLogs v1.49.0 and v1.53.0 storage (stored fields, stream fields, | unpack_json | filter): 80/80 selector and 80/80 label-filter answers match on both versions; origin/main matched 26 and 36. On 200k rows VictoriaLogs answers the anchored selector forms faster than the unanchored ones (multi-value about -50%, single value -64 to -70%) because they select fewer rows and the single value is an exact filter; label-filter shapes over unpacked fields stay within the loaded host's noise. Every route that takes a selector is pinned by a recording-backend test, the __tenant_id__ matcher by an extended table, and the answers by TestLabelRegexpMatchesWhatLokiMatches (both VictoriaLogs capability sets), TestLabelFilterConditionAgreesWithLoki and FuzzSelectorRegexpFilter. TestCompat_LabelMatcherRegexpAnchoring compares Loki and the proxy over log, metric, label values, series, volume, detected labels and multi-tenant requests. Registry: profiles/stream-selector-regexp-anchoring (new) and profiles/label-filter-regexp-anchoring, both fixed; A/B set label-matcher-anchoring and three visual fix-proof pages.
…eports and the entry under Unreleased
szibis
force-pushed
the
fix/label-matcher-regexp-anchoring
branch
from
October 28, 2026 13:53
be7bb4d to
73b9ba4
Compare
Contributor
⚪ Performance A/B: no change beyond noise190 shape×range — 190 ⚪ same · results vs Loki: 181 same, 0 new difference(s), 9 pre-existing Loki profiles compared (like for like):
Every shape is within noise of the base build (±30% and 100 ms). 190 unchanged shape×range (within noise)
Cold (first-run) timings — what a user opening a panel sees
† Loki's results cache answered this cold run (its own counters, or identical answers for windows a minute apart): not a computed time, so it is in no ratio. ↺ Loki cold taken from the first set that ran the same query; Loki's cache answered this one. 14 shape×range above 2× Loki already (listed, or as slow on the base build)
Why these shapes
base |
github-actions
Bot
added
size/XL
Contributor
PR Quality ReportCompared against base branch Coverage and tests
Compatibility
Performance smokeLower CPU cost (
State
|
Contributor
✅ Visual smoke: passed with warningsBase Fix proofCaptures of the exact cases this pull request adds or fixes (registry case, base | PR | Loki). The verdict compares the PR with Loki where the base did not match. The gate fails a case unless one capture is fixed or improved, or, beyond the Loki data window, the base failed where Loki and the PR both answered with data (the label does not excuse it; only
explore-volume-label-filter-regexp-15m: improved
explore-volume-label-filter-regexp-1h: improved
explore-volume-label-filter-regexp-6h: undecided
explore-logs-multivalue-regexp-15m: improved
explore-logs-multivalue-regexp-1h: improved
explore-volume-multivalue-regexp-15m: improved
explore-volume-multivalue-regexp-1h: improved
Regression check
Gate: a base-vs-PR data difference passes when the PR is closer to Loki, and fails when the PR diverges from Loki or Loki cannot decide it (at a range Loki does not hold, a difference passes when the same page proved that query to answer like Loki at a shorter range; otherwise label the pull request explore-logs-multivalue-regexp-15m: pixel diff 3.2%
explore-logs-multivalue-regexp-1h: pixel diff 4.8%
explore-volume-label-filter-regexp-6h: no Loki data at this range; the same differences match Loki at a shorter range (1): query ('sum by (region) (count_over_
Core set montages (5): base | PR | Lokidd-fields-1h
dd-landing-1h
dd-service-logs-1h
explore-logs-plain-1h
explore-metric-plain-1h
All montages of this run: |
github-actions
Bot
added
size/XL
…s wrongly and pin the selector pages to the Loki ranges The label-filter Fix-proof page filtered region=~"us-east|eu-west-1". The base already anchors a label filter on the ordered JSON metric path (^(?:re)$), and for two unrelated literals that selects what Loki's equality-per-branch selects, so base and PR answered identically and the page could not show the fix. It now filters region=~"us-east-1|west.*": Loki ORs an equality check on us-east-1 with a substring check on west and selects all three seeded regions (us-east-1, us-west-2, eu-west-1); the base's anchored form selects us-east-1 only. Checked against Loki v3.7.9's pipeline and VictoriaLogs v1.49.0 and v1.53.0 storage, field and stream datasets: the PR's filter selects Loki's rows, the base's does not. The two multi-value selector pages showed the fix at 15m and 1h, but at 6h the index/stats request for the changed selector also differs base vs PR (the base counts the extra stream). The gate never carries a resource difference to the range Loki does not hold (test_a_resource_difference_at_6h_is_never_carried), so those rows failed. Their entries are pinned to 15m and 1h like the other fix pages, and the README says when to do so. Two tests pin the label-filter page's gate shape and the resource case.
github-actions
Bot
added
size/XL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #572 by @clain23: its commit is kept here as authored (
fix(logql): anchor label-matcher regexps to the whole label value), and the next commit completes it for label filters, where Loki does not simply anchor.Summary
Regexp label matchers now select what Loki selects. Contributed by @clain23 (first commit, kept as authored); the second commit completes it for label filters and keeps VictoriaLogs on its fast filters.
Selectors. Loki builds
{app=~"api|web"}with Prometheus'labels.NewMatcher, which compiles^(?s:api|web)$(vendor/github.com/prometheus/prometheus/model/labels/regexp.go:76). It selects appapiandwebonly. The proxy sentapp:~"api|web", and VictoriaLogs' regexp filter is a substring match (lib/logstorage/filter_regexp.go,matchRow). So a Grafana multi-value variable also selectedapi-gateway,my-apiandwebhook, and{namespace=~"nch"}selectedanch. Nothing errored; panels and log lists just held more. Only the derivedservice_namematcher was anchored.Where it applied. Every route that takes a selector was affected:
Multi-tenant fan-out.
{__tenant_id__=~"prod"}also queriedprod-euandnonprod. Loki matches tenant ids with the anchored matcher (pkg/querier/multi_tenant_querier.go:421,filterValuesByMatchers).Label filters are different. Loki does not anchor
| label =~ "re"as written.parseRegexpFilter(pkg/logql/log/filter.go:645, reached fromNewStringLabelFilterinlabel_filter.go:360) first runsRegexSimplifier.Simplify(filter.go:696). It anchors only what it cannot simplify, as^(?:re)$without thesflag:(?i);api|web) is equality per branch;api.*and.*apiare substring checks;api|api-gatewayandprod-eu|prodare factored;=~""and.*keep every line, and.+keeps non-empty values;newOrFilter,filter.go:309), soGET|is an equality check onGET.The contributor's commit wrapped label filters in
^(?s:re)$too. That answered 24 of 80 label-filter cases differently from Loki. The second commit ports the simplifier.VictoriaLogs semantics checked
field:~"re"is an unanchored match (filter_regexp.go) in which.matches a newline:regexutilparses withsyntax.DotNL. Exactly""and.*become a no-op filter, and.+becomes the non-empty prefix filter (parser.go,newFilterRegexpOpt).{f=~"re"}is already anchored.regexutil.NewPromRegexcompiles^(?:re)$, andindexdb.gogetStreamIDsForTagFiltertreats a regexp that matches the empty value like Prometheus does.canUseStreamSelectoraccepts only=on configured stream fields, on main and here. Routing=~to_stream:{}would be a separate improvement (listed under follow-ups).regexutilis the same file in both.Fix
internal/translator/label_matcher_regexp.go:selectorRegexpFieldFilter, for selectors. It matches the whole value with.matching newlines. It sends:{app=~"api"}→app:="api");.*and.+as they are (VictoriaLogs' match-all and non-empty filters);.*literal.*as the substring regexp;^(?s:re)$.lokiLabelFilterPatternpluslabelFilterRegexpFieldFilter, for label filters. They emit the filter Loki's simplification describes:OR(| method!~"GET|HEAD"→-(method:="GET" OR method:="HEAD")), at most 16;^(?-s:re)$when it contains a., because VictoriaLogs compiles.to match newlines.translatedFilterFieldOpreads the field inside anORgroup, so the same-field dedupe of repeated Drilldown filters keeps its key.Escaping. A literal with a control character keeps the regexp form, quoted with Go's rules.
Proxy-side conditions.
NewLabelFilterConditioncompiles a label filter with the same semantics. The ordered JSON metric pushdown and the Loki profile now use it to evaluate rows and to build their| filter(they compiled^(?:re)$throughNewDropCondition).NewDropConditionstays for drop and keep matchers.Every label-filter site uses one of these:
translateSingleLabelFilter;parsedKeyMatchesEmpty);service_namelabel filter;logsQLLabelCondition.internal/proxy/multitenant.go(contributor):__tenant_id__regexps are anchored as^(?s:re)$.Line filters (
|~,!~) are unchanged. They are substring regexps in Loki too, and the tests pin that.Breaking: a selector written for substring matching selects fewer streams, as in Loki.
{namespace=~"prod"}no longer selectsproduction; write.*prod.*where a substring is meant. Rules migrated withrules-migrateshould be checked for the same.Evidence
Loki's answers. Loki v3.7.7's own code produced them:
syntax.ParseMatchersfor selectors, and the stream pipeline of{app="x"} | json | v=~"…"for label filters. The table covers 40 patterns and both operators over 28 values: values that are prefixes of each other, the empty value, and one with a newline. It is embedded inTestLabelRegexpMatchesWhatLokiMatches.VictoriaLogs' answers. The emitted filters were run on VictoriaLogs storage at v1.49.0 and v1.53.0: stored fields, stream fields,
| unpack_json | filterand the stored-field search path.VictoriaLogs query cost. 200k rows, 8 apps,
GOMAXPROCS=1, median of 6 (15 for the label-filter alternation rows). Rows: what each form returns (main's are wrong).app=~"api-gateway|auth-service"api-.*.*(same query, noise)ORof exact filters (sent)5..us-.*(substring)The host was loaded (load average 40 to 70). The
.*row sends the same query on both sides, and across runs it moved from -27% to +11%. So the label-filter rows over unpacked fields, within ±15%, are noise here; the A/B on the stack decides them.Matching cost. Isolated
lib/regexutilv1.53.0, one value,GOMAXPROCS=1:That is why a label filter's literal alternation is sent as
ORof exact filters and not as the anchored regexp.Translation cost.
benchstat,-cpu 1 -count 10, interleaved builds, loaded host:BenchmarkTranslate(internal/logql): unchanged, exceptmany_labels+1 alloc/op (+40 B) from the anchoring string.BenchmarkTranslateLogQL_RegexpMatchers(five regexp matchers): 84 → 234 allocs/op and 2.5 → 11.1 KiB/op, from theregexp/syntaxparses that decide the filter form. Wall time 16.6 → 23.2 µs (p=0.063, ±66% on the loaded host).Simple,MultiLabel,JSONPipeline): identical allocs, no significant time change.Tests
TestLabelRegexpMatchesWhatLokiMatches(internal/translator): the Loki table above, for selector and label filter,=~and!~, translated with the v1.49.0 and v1.53.0 capability sets. Answers are evaluated with VictoriaLogs' filter semantics, which the storage run above validated. On origin/main it fails 196 subtests.TestLabelFilterConditionAgreesWithLoki: the same table for the proxy-side conditions and the LogsQL the ordered JSON path sends.FuzzSelectorRegexpFilter: the exact, substring and match-all forms against the reference^(?s:re)$. Its 1120 seeds run in-short, plus 60 s of fuzzing locally.TestLabelRegexpEmittedForms: the emitted forms, including line filters left unanchored.TestLabelMatcherRegexpAnchoredOnEveryRoute(internal/proxy): a recording backend on 13 routes, plus the query tail sends. No unanchored matcher reaches VictoriaLogs. On origin/main all 13 routes fail.TestTenantMatcherRegexpIsAnchored: extended with.*,.+,"",(?i),.*prodand a factored alternation.Contributor tests:
TestAnchorLabelMatcherRegexandTestAnchorLabelMatcherRegexMatchesLokiSemantics(the^(?s:re)$helper).TestLabelMatchersAreAnchoredLineFiltersAreNot,TestRegexpMatchersReachVictoriaLogsUnescaped,FuzzRegexpMatcherEscaping(it now skips the exact and substring forms, whichFuzzSelectorRegexpFilterchecks).e2e vs Loki,
TestCompat_LabelMatcherRegexpAnchoring(test/e2e-compat, group core). Its own fixture is five streams, with app api / api-gateway / my-api / web / webhook and env prod / prod-eu / nonprod. It covers:=~and!~,(?i),.*apiandapi.+;| json, including Loki's substring answer forsvc=~"api|api-gateway";X-Scope-OrgID: 0|fake). Loki runs single-tenant in the compose stack, so this subtest is checked against Loki's matcher semantics.Each comparison fails when Loki answers nothing or answers the widened set.
Registry
profiles/stream-selector-regexp-anchoring(new,status: fixed).profiles/label-filter-regexp-anchoring(was open, nowfixed; title andwhyrewritten for Loki's simplifier).label-matcher-anchoring(M1 to M6): multi-value and single-value selectors, an alternation over many streams, and label filters after| json.explore-logs-multivalue-regexp:{app=~"api-gateway|auth"}. On main it also showsauth-service.explore-volume-multivalue-regexp.explore-volume-label-filter-regexp:| json | region=~"us-east|eu-west-1". On main it also countsus-east-1.Checks
go vet ./...,go vet -tags e2e ./test/e2e-compat/andgofmt -l: clean. golangci-lint: 0 issues.-raceon the touched tests in internal/translator, internal/proxy, internal/logql and internal/logsql: pass.GOMEMLIMIT=3GiB GOMAXPROCS=4 go test -short -count=1 -p 1 ./cmd/... ./internal/...): every package ok. Peak RSS on the final head: 2.12 GB (1.92 GB on an earlier run); origin/main on the same host: 2.07 GB.check_conformance.py: conformance gate ok.check_changelog_pr.py --base origin/main --head HEAD: ok.Pending before ready
TestCompat_LabelMatcherRegexpAnchoringagainst Loki on VictoriaLogs v1.49.0 and v1.53.0 (-run '^(TestSetup_IngestLogs|TestCompat_LabelMatcherRegexpAnchoring)$').label-matcher-anchoringplus the control set, main vs branch vs Loki.Follow-ups (not in this PR)
-stream-fieldslabel could use VictoriaLogs' stream index ({f=~"re"}, which VictoriaLogs anchors the same way). Today only=on stream fields does.| detected_level=~on a stored level field (detected_level_logsql.go) keeps its own^(?:re)$. It differs from Loki only for an alternation with an empty branch, such aserror|.splitDropSpec(only used by a test) compiles drop and keep matcher regexps unanchored.Update: rebased onto main (2026-10-28)
perf:blocks main's conformance gate requires (bound 2× Loki, target 1×, unlocked until the daily run measures them).FuzzSelectorRegexpFilteris registered in the CIfuzz-smokejob (local run: 2 min, about 650k execs, no failure).[Unreleased], and the generated reports and quality section are regenerated.GOMEMLIMIT=2GiB GOMAXPROCS=4 go test -short -count=1 -p 1oninternal/proxy,internal/translator,internal/logqlandinternal/logsqlpasses.go vetpasses, including-tags e2e.