Skip to content

feat(postgres): lex @@ and @@@ as full text search operators - #5

Open
OliverCulleyDeLange wants to merge 1 commit into
mainfrom
feat/postgres-full-text-search-operator
Open

OliverCulleyDeLange wants to merge 1 commit into
mainfrom
feat/postgres-full-text-search-operator

Conversation

@OliverCulleyDeLange

Copy link
Copy Markdown

What

The postgres dialect lexed @@ as two separate @ tokens, so any statement using it failed to parse. Layout rules then ran over the fallback token stream and rewrote the SQL — @@ became @ @ and, in the same statement, || became | |. Both are invalid SQL, so formatting a file silently broke it.

FullTextSearchOperator already existed in SyntaxKind but was never lexed or parsed. It is now:

  • produced by a lexer matcher placed ahead of the single @ matcher, covering @@ and the legacy @@@;
  • parsed by FullTextSearchOperatorSegment, a TypedParser yielding BinaryOperator, mirroring JsonOperatorSegment;
  • accepted anywhere BinaryOperatorGrammar is.

Repro

SELECT 1 FROM t WHERE a @@ b;

Before: LT01 | Expected single whitespace between "@" and "@", and sqruff fix rewrites it to a @ b.
After: parses cleanly, no diagnostics.

Why this matters

Full text search is not reachable in postgres without @@. The equivalent function call ts_match_vq(v, q) parses fine, but the planner only matches GIN operator classes against the operator, so writing the function loses the index — measured over 60k rows, 9ms indexed via @@ versus 29ms sequential scan via ts_match_vq.

Testing

  • New fixture dialects/postgres/sqruff/full_text_search_operator.sql covering the bare operator, a cast operand, a subquery operand with a cast, and @@@. Its generated tree shows binary_operator: '@@' with no unparsable segments.
  • cargo test --workspace --exclude sqruff-cli-python with SQRUFF_SKIP_UNSUPPORTED_TEMPLATERS=1: 146 passed, 0 failed. The two exclusions are environmental — the python bindings test needs a .venv, and the rules test needs the jinja templater; both fail the same way on the unpatched revision.

Version

Bumped to 0.38.2 so consumers pinning the fork by version can tell this revision from 0.38.1.

🤖 Generated with Claude Code

The postgres dialect lexed `@@` as two separate `@` tokens, so any statement
using it failed to parse. Layout rules then ran over the fallback token stream
and rewrote `@@` to `@ @` and `||` to `| |`, corrupting the SQL.

`FullTextSearchOperator` already existed in the syntax kinds but was never
lexed or parsed. It is now produced by a lexer matcher ahead of the single `@`
matcher, parsed by `FullTextSearchOperatorSegment`, and accepted wherever a
binary operator is, mirroring `JsonOperatorSegment`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant