feat(postgres): lex @@ and @@@ as full text search operators - #5
Open
OliverCulleyDeLange wants to merge 1 commit into
Open
OliverCulleyDeLange wants to merge 1 commit into
OliverCulleyDeLange wants to merge 1 commit into
Conversation
The postgres dialect lexed `@@` as two separate `@` tokens, so any statement using it failed to parse. Layout rules then ran over the fallback token stream and rewrote `@@` to `@ @` and `||` to `| |`, corrupting the SQL. `FullTextSearchOperator` already existed in the syntax kinds but was never lexed or parsed. It is now produced by a lexer matcher ahead of the single `@` matcher, parsed by `FullTextSearchOperatorSegment`, and accepted wherever a binary operator is, mirroring `JsonOperatorSegment`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The postgres dialect lexed
@@as two separate@tokens, so any statement using it failed to parse. Layout rules then ran over the fallback token stream and rewrote the SQL —@@became@ @and, in the same statement,||became| |. Both are invalid SQL, so formatting a file silently broke it.FullTextSearchOperatoralready existed inSyntaxKindbut was never lexed or parsed. It is now:@matcher, covering@@and the legacy@@@;FullTextSearchOperatorSegment, aTypedParseryieldingBinaryOperator, mirroringJsonOperatorSegment;BinaryOperatorGrammaris.Repro
Before:
LT01 | Expected single whitespace between "@" and "@", andsqruff fixrewrites it toa @ b.After: parses cleanly, no diagnostics.
Why this matters
Full text search is not reachable in postgres without
@@. The equivalent function callts_match_vq(v, q)parses fine, but the planner only matches GIN operator classes against the operator, so writing the function loses the index — measured over 60k rows, 9ms indexed via@@versus 29ms sequential scan viats_match_vq.Testing
dialects/postgres/sqruff/full_text_search_operator.sqlcovering the bare operator, a cast operand, a subquery operand with a cast, and@@@. Its generated tree showsbinary_operator: '@@'with no unparsable segments.cargo test --workspace --exclude sqruff-cli-pythonwithSQRUFF_SKIP_UNSUPPORTED_TEMPLATERS=1: 146 passed, 0 failed. The two exclusions are environmental — the python bindings test needs a.venv, and the rules test needs the jinja templater; both fail the same way on the unpatched revision.Version
Bumped to 0.38.2 so consumers pinning the fork by version can tell this revision from 0.38.1.
🤖 Generated with Claude Code