Skip to content

fix(logql): anchor label-matcher regexps to the whole label value - #572

Closed
clain23 wants to merge 1 commit into
ReliablyObserve:mainfrom
maestra-io:fix/anchor-label-matcher-regexps
Closed

clain23 wants to merge 1 commit into
ReliablyObserve:mainfrom
maestra-io:fix/anchor-label-matcher-regexps

Conversation

@clain23

@clain23 clain23 commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A LogQL label matcher is a full-value match. Loki builds its matchers with Prometheus' labels.NewMatcher, and MatchRegexp compiles the pattern as "^(?s:" + v + ")$" (prometheus/prometheus, model/labels/regexp.go), so {namespace=~"nch"} selects exactly the value nch.

VictoriaLogs' field:~"re" is an unanchored match, and the proxy passed the pattern through verbatim. Measured against the pinned VictoriaLogs v1.52.0, over a row whose app is anchor-test-xyz:

LogsQL rows
app:~"anchor" 1 (the anchor-test-xyz row)
app:~"^(?s:anchor)$" 0

So every =~ / !~ matcher the proxy emitted was silently widened. A widened matcher inflates results rather than erroring, which is why nothing surfaced it:

query Loki proxy before proxy after
{namespace=~"nch"} on values nch, anch, nch-b nch all three nch
{app=~"a|b"} on ab no match match no match
{ns=~"prod"} on production no match match no match

AnchorLabelMatcherRegex wraps the pattern once, in internal/logsql, and every emitter that carries a label-matcher regexp uses it:

  • internal/logql translateLabelMatcher (MatchRe, MatchNotRe);
  • internal/translator streamMatcherToFieldFilter for the stream selector, and translateSingleLabelFilter for a pipeline label filter such as | status =~ "5..";
  • the derived service_name matcher and label filter from fix: derive service_name as Loki assigns it, for selectors and grouping #579, which already anchored their patterns with the same form and now share the helper;
  • the multi-tenant __tenant_id__=~ fan-out, which compiled the bare pattern with Go regexp and so also matched prod-eu for prod.

The emitted form is Prometheus' ^(?s:…)$. On VictoriaLogs the s flag changes nothing (it compiles with DotNL on, verified on v1.52.0), but the multi-tenant path compiles the same helper's output with Go regexp, where (?s) is not the default, so both call sites are now literally what Prometheus compiles. The non-capturing group is load-bearing: ^a|b$ parses as (^a)|(b$), so an alternation has to be grouped before it is anchored.

Line filters (|~, !~) are deliberately left alone: those are substring regexps in Loki, which is what VictoriaLogs already does. The pipeline label filter anchors after the empty-value branch, so | level =~ "" keeps its absent-or-empty translation.

Changelog

One ### Fixed entry (label matchers, __tenant_id__) and one ### Changed entry (substring-style =~ stops matching) under ## [Unreleased].

Docs

  • docs/translation-reference.md: the stream-selector and label-filter rows show the anchored form, with one sentence on why.
  • docs/real-window-compatibility-gaps.md: one correction line, since results shrink for existing queries.

Validation

  • TestAnchorLabelMatcherRegexMatchesLokiSemantics (new): a (pattern, value, shouldMatch) table, including the reported nch/anch/nch-b case, the alternation leak and the empty pattern.
  • TestAnchorLabelMatcherRegex (new) pins the emitted shape.
  • TestLabelMatchersAreAnchoredLineFiltersAreNot (new, internal/translator) covers the boundary end to end, including one query carrying both kinds: {app=~"a|b"} |~ "substring" → app:~"^(?s:a|b)$" ~"substring". It fails on main on every anchored case.
  • TestRegexpMatchersReachVictoriaLogsUnescaped and FuzzRegexpMatcherEscaping now assert the anchored pattern as the one Loki compiles; the other updated expectations across internal/logql, internal/translator and internal/proxy all encoded the widened form.
  • Parity case in a CI shard: selector_regex_prefix_is_not_substring in test/e2e-compat/query-semantics-matrix.json ({app=~"api",namespace="prod",env="production"}, line count compared with Loki, empty on both sides), tracked under selector_regex in the operations inventory and run by TestQuerySemanticsMatrix.
  • Local: gofmt -s -l empty, go build ./..., go vet ./..., golangci-lint run (v2.13.2, 0 issues), scripts/ci/check_changelog_pr.py and scripts/ci/check_conformance.py ok, go test ./... -count=1 green across all packages, on top of current main (v1.86.0).

@szibis

szibis commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator

Reviewed end to end against the live stack (Loki 3.7.7 direct vs this branch's binary vs current main). This is the strongest of the four and the fix is confirmed live.

Confirmed fixed (Loki / main / this branch):

query Loki main branch
{app=~"otel"} empty 8 streams empty
{app=~"api|otel-collector"} only otel-collector also api-gateway only otel-collector
{app="otel-collector"} | level=~"wa" empty 5 streams empty
{app="otel-collector", namespace=~"monitor"} empty 5 streams empty

label!~"" also goes from matching everything to Loki's behaviour. Dropping the inline-flag scoping transform is right: RE2 scopes an inline flag to its enclosing group, so ^(?:(?m)foo)$ keeps whole-value anchors. Nothing in the proxy inspects the emitted regexp body, so no fast path or cache key is silently disabled.

Before merge:

  1. service_name wording. The CHANGELOG reads as if the selector fan-out is fixed, but {service_name=~"otel"} still returns 1 stream where Loki returns 0: the translator ORs the anchored pattern across 17 candidate fields, and container="otel" matches exactly while Loki's derived service_name is otel-collector. The pipeline form | service_name=~… does not fan out, so the two spellings now disagree. Please reword, or note the gap; PR fix: derive service_name as Loki assigns it, for selectors and grouping #579 is fixing the derivation itself.
  2. ^(?: vs ^(?s:. Prometheus compiles ^(?s:…)$, and this repo already spells that form elsewhere. Equivalent on VictoriaLogs (DotNL is on by default, verified), but the same helper is compiled with Go regexp in the multi-tenant path where (?s) is absent. Emitting ^(?s: would make both call sites literally Prometheus. The CHANGELOG also says ^(?: where the code comment says ^(?s:.
  3. No parity case. This is a user-visible narrowing of results; one {app=~"<prefix>"} proxy-vs-Loki case wired into a CI shard would lock it.
  4. Docs. Results shrink for existing queries; docs/translation-reference.md and docs/real-window-compatibility-gaps.md are untouched.
  5. Attribution. The commits carry Co-Authored-By: Claude … and the PR body has a "Generated with Claude Code" line and a session link. This repo's rule is no AI attribution anywhere. Since this comes from a fork we will strip it in the squash message, but please drop it from the source too. Two commits are also unsigned.
  6. The PR body is stale: it still presents the inline-flag scoping as load-bearing, which the final commit removed.

Merge order across the four: this one first, then #570, then #574. #571 should not merge as it stands (see my review there).

@szibis szibis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision: accept, merge first of the four, after three small changes.

The fix is real and confirmed live ({app=~"otel"}: Loki empty, main 8 streams, this branch empty).

Please do:

  1. Reword the service_name line in the CHANGELOG: the selector fan-out is not fixed here ({service_name=~"otel"} still returns 1 stream where Loki returns 0). PR #579 fixes the derivation.
  2. Emit ^(?s:…)$ instead of ^(?:…)$ to match Prometheus exactly, and make the CHANGELOG and the code comment agree.
  3. Add one proxy-vs-Loki parity case for {app=~"<prefix>"} wired into a CI shard, and a line in docs/real-window-compatibility-gaps.md since results shrink for existing queries.

Also remove Co-Authored-By: Claude … from the commits and the "Generated with Claude Code" line from the PR body: this repo allows no AI attribution. We squash-merge, so we can strip it from the squash message if it stays, but please drop it at the source.

A LogQL label matcher is a full-value match: Loki builds it with
Prometheus' labels.NewMatcher, which compiles a MatchRegexp as
"^(?s:" + v + ")$". VictoriaLogs' field:~"re" is an unanchored match
and the proxy passed the pattern through verbatim, so every =~ / !~
matcher was silently widened: {namespace=~"nch"} also selected "anch"
and "nch-b", {app=~"a|b"} matched "ab".

AnchorLabelMatcherRegex wraps the pattern once, in internal/logsql, and
every emitter of a label-matcher regexp uses it: the stream selector,
pipeline label filters, the derived service_name matcher and label
filter (which already anchored with the same form) and the multi-tenant
__tenant_id__ fan-out, which compiled the bare pattern with Go regexp.
The form is Prometheus' own ^(?s:...)$, so the Go-compiled call site is
literally what Prometheus compiles too.

Line filters (|~, !~) are substring regexps in Loki and are left alone.

A parity case with a prefix matcher ({app=~"api"}, empty on Loki) joins
the query-semantics matrix, and the docs record that results shrink for
existing substring-style matchers.
@clain23
clain23 force-pushed the fix/anchor-label-matcher-regexps branch from ed3fa40 to 4e6afbe Compare September 22, 2026 14:53
@clain23

clain23 commented Sep 22, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the live confirmation. Rebased onto current main (v1.86.0, so on top of #579) and squashed to one commit, addressing each point:

  1. service_name wording. The CHANGELOG no longer claims the selector fan-out; it says the derived service_name matcher from fix: derive service_name as Loki assigns it, for selectors and grouping #579 already anchored its pattern and now shares the helper. serviceNameMatcherFilter / serviceNameLabelFilterPattern call AnchorLabelMatcherRegex instead of spelling the prefix themselves, so there is one place for it.
  2. ^(?s:. The helper emits ^(?s:…)$; code comment, CHANGELOG, docs and every test expectation agree. TestRegexpMatchersReachVictoriaLogsUnescaped and FuzzRegexpMatcherEscaping now assert the anchored pattern as the one Loki compiles.
  3. Parity case in a CI shard. selector_regex_prefix_is_not_substring in query-semantics-matrix.json: {app=~"api",namespace="prod",env="production"}, line count compared against Loki, empty on both sides (fails on main, which returns the api-gateway lines). Tracked under selector_regex in the operations inventory, so it runs in the TestQuerySemanticsMatrix shard.
  4. Docs. docs/translation-reference.md rows show the anchored form with one sentence on why; one correction line in docs/real-window-compatibility-gaps.md.
  5. Attribution. Gone from the commit and the body. The commit is signed.
  6. The body is rewritten to the final shape (no flag-scoping claim).

Local: gofmt, build, vet, golangci-lint (0 issues), changelog and conformance gates, go test ./... -count=1 green.

@szibis

szibis commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

Thank you @clain23. Your commit landed in #909 as authored: regexp stream selectors are now anchored to the whole value, as Loki's Prometheus matchers are. #909 also completes it for label filters after a parser, which Loki first simplifies (an equality check, a substring check, or anchored otherwise), and registers a fuzz target for both. Closing this one as merged through #909.

@szibis szibis closed this Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants