You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 6961511
Browse filesBrowse the repository at this point in the historyBrowse files
feat: parse MySQL casts, conversions, charset literals and collations
Replace the permissive MySQL CAST-as-function path with validated native
cast_type grammar. Retain numeric, temporal, character/binary, JSON and
spatial targets, accepted modifiers, character attributes, and lexical
aliases such as FLOAT4, FLOAT8 and INT4. Distinguish broad field-length
numeric tokens from the bounded NUM production used by temporal and
paired decimal precision, leaving runtime range checks to the server.
Represent typed and USING CONVERT forms, charset-introduced text/hex/bit
literals and expression COLLATE with appended AST node kinds. Preserve
operands, raw metadata spelling, adjacent introduced strings and native
collation precedence. Recognize the shared native charset names and utf8
alias without consuming unknown underscore-prefixed identifiers. Reject
reserved or introducer metadata tokens, incomplete operands, malformed
modifiers and unsupported conversion tails.
Emit the new syntax without losing charset, collation or type information.
Preserve original double-quote delimiters for MySQL string values so
embedded apostrophes and doubled double quotes remain valid. Continue to
emit the current AST value rather than stale source text. Reject odd-length
quoted hex literals and uppercase numeric introducers where native MySQL
rejects them. Keep PostgreSQL behavior unchanged.
Guard local planning against semantics the engine does not implement.
Keep parameterization conservative for the new node kinds; existing CAST
parameterization binds values while retaining type precision and scale.
Add regression coverage for strict rejection, exact emission and reparse,
AST cloning, planning guards, parameterization and native lexical edges.
Expand the curated MySQL corpus from 170 to 212 witnesses and promote the
CAST-decimal and charset-literal gaps to CHECKED. The baseline now records
110 CHECKED, 92 REJECTED and 10 remaining documented gaps; these counts
are specific to the selected corpus, not a dialect-wide coverage metric.
Verify 212 exact upstream source anchors for each pinned release and
update the compatibility guide with supported forms and explicit limits.
Validation:
- Forced full rebuild: make -B -j6 all build-corpus-test; make test.
1,611 tests passed, with 37 external-database tests skipped.
- ASan/UBSan with leak detection: 95 focused MySQL tests passed.
- MySQL harness: 11 tests passed; all 212 expected outcomes matched.
- Native MySQL 8.4.8: 110 accepted, 94 syntax rejections, five server
errors and three skipped DDL/procedure witnesses.
- Native MySQL 9.7.2: 112 accepted, 90 syntax rejections, seven server
errors and three skipped DDL/procedure witnesses.
- Both native runs had zero expectation mismatches, verified source
anchors, and cleaned up their owned schemas and server instances.
- Four double-quote regression queries passed native checks on both pins.
- PostgreSQL harness: 87 tests passed; all 5,122 committed CI cases and
the full 49,848/51,415 accepted-statement baseline remain unchanged.
- Seven parser/emitter benchmark smoke cases passed; this is not a
controlled performance comparison and makes no new speed claim.
- Independent review verified native grammar edge fixes and the final
delimiter-preservation change.
Keep PR #67 a work in progress. Benchmark code, reports and benchmark-only
README/.gitignore changes are deliberately excluded from this commit.
Copy file name to clipboardExpand all lines: docs/mysql-compatibility.md
+6-4Lines changed: 6 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,9 @@ This harness tracks a small, deliberately selected set of MySQL syntax witnesses
4
4
5
5
ParserSQL now preserves MySQL JSON arrows, named/inherited windows, `GROUP_CONCAT` modifiers and separators, and `GROUP BY ... WITH ROLLUP` as structured ASTs. Database-qualified column names are retained. The local engine rejects these unsupported semantics rather than executing a simplified interpretation; parameterization remains conservative for JSON arrows and `GROUP_CONCAT`. Empty MySQL SELECT target lists are rejected.
6
6
7
-
MySQL `EXTRACT(unit FROM expr)`, keyword forms of `SUBSTRING`/`SUBSTR`/`MID`, and `MATCH ... AGAINST` have structured ASTs. EXTRACT accepts the twenty native interval keywords, including compound units, and rejects quoted or unknown units. Substring calls retain the alias spelling and support both comma arguments and `FROM ... [FOR ...]`. MATCH preserves qualified column lists, default/natural/boolean modes and query expansion; its optional column-list parentheses are emitted canonically. The search operand enforces the native `bit_expr` precedence boundary, so a comparison must be parenthesized before a search mode. General expression gaps still apply inside these forms: unary `!` and `COLLATE` operands are accepted by both pinned servers but remain unsupported by ParserSQL. Complete-operand parsing rejects empty tuples, invalid ROW arity, missing grouping/query delimiters, invalid EXISTS bodies and incomplete CASE branches. The local engine rejects these special AST semantics; parameterization remains conservative for the new node kinds. The [8.4.8 special-function productions](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/sql_yacc.yy#L10661) and [9.7.2 special-function productions](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L11070), together with the corpus's MATCH, interval, and identifier anchors, define this bounded syntax coverage.
7
+
MySQL `EXTRACT(unit FROM expr)`, keyword forms of `SUBSTRING`/`SUBSTR`/`MID`, and `MATCH ... AGAINST` have structured ASTs. EXTRACT accepts the twenty native interval keywords, including compound units, and rejects quoted or unknown units. Substring calls retain the alias spelling and support both comma arguments and `FROM ... [FOR ...]`. MATCH preserves qualified column lists, default/natural/boolean modes and query expansion; its optional column-list parentheses are emitted canonically. The search operand enforces the native `bit_expr` precedence boundary, so a comparison must be parenthesized before a search mode. General expression gaps still apply inside these forms: unary `!` operands are accepted by both pinned servers but remain unsupported by ParserSQL. Complete-operand parsing rejects empty tuples, invalid ROW arity, missing grouping/query delimiters, invalid EXISTS bodies and incomplete CASE branches. The local engine rejects these special AST semantics; parameterization remains conservative for the new node kinds. The [8.4.8 special-function productions](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/sql_yacc.yy#L10661) and [9.7.2 special-function productions](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L11070), together with the corpus's MATCH, interval, and identifier anchors, define this bounded syntax coverage.
8
+
9
+
MySQL `CAST(expr AS cast_type)`, both `CONVERT(expr, cast_type)` and `CONVERT(expr USING charset)`, charset-introduced text/hex/bit literals, and expression `COLLATE` retain structured ASTs. Cast types follow the native cast-specific productions rather than general column definitions: numeric, temporal, character/binary, JSON, and spatial targets preserve their accepted modifiers and source spelling. Character targets retain charset and binary attributes. Introducers recognize the shared native charset names (including the `utf8` alias), support intervening whitespace, and retain literal kind. Literal emission preserves MySQL double-quote delimiters, including embedded apostrophes and doubled double quotes, with `ANSI_QUOTES` off. Expression `COLLATE` accepts a simple identifier or quoted name, with native precedence; bare `BINARY`, `DEFAULT`, qualified names, and missing operands are rejected. `CONVERT ... USING` additionally accepts bare `BINARY`. The [8.4.8 cast/conversion productions](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/sql_yacc.yy#L10425) and [9.7.2 cast/conversion productions](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L10745), plus the corpus's exact type, literal, charset, and collation anchors, define this bounded grammar coverage. Cast `ARRAY`, `AT LOCAL`, and `AT TIME ZONE` forms remain outside this batch. Native lexical aliases such as `FLOAT4`, `FLOAT8`, and `INT4` are retained. Single field lengths permit native decimal-number tokens; two-part decimal precision and temporal precision use a native integer `NUM` token from 0 through 2147483647. Runtime precision limits remain native semantic checks. These nodes are syntax/AST support: the local engine rejects unsupported semantics, and parameterization remains conservative.
8
10
9
11
With `IGNORE_SPACE` off, native EXTRACT and substring aliases require `(` immediately after the function name. ParserSQL follows that lexical boundary for these special forms; `MATCH (` remains valid. Spaced comma calls are ordinary function-call syntax and can resolve differently on a native server. This is not general SQL-mode emulation. The [native lexer](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/sql_lex.cc#L1567) distinguishes adjacent function tokens, and [the function-token table](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/lex.h#L831) maps `MID` and `SUBSTR` to `SUBSTRING`.
10
12
@@ -52,7 +54,7 @@ Malformed controls are handled separately: a complete AST is `MALFORMED_ACCEPTED
52
54
53
55
Every fixture freezes one exact expected outcome. Known gaps do not make arbitrary failures acceptable. Both regression and unexpected improvement cause exit status 1, so a fixed gap must be reviewed and explicitly promoted in the corpus. When promoting, add an independently derived `expected_sql`, set `known_gap=false`, and require `CHECKED`. Harness/transport/source errors return 2. Exit status 0 means the expected baseline matched, including its documented known gaps.
54
56
55
-
The 170-case baseline has 90`CHECKED` witnesses and 68`REJECTED` negative controls. Its 12 remaining known gaps are seven`TRAILING_INPUT`, three `NO_AST`, one `PARTIAL`, and one `EMISSION_MISMATCH`. CTE-prefixed UPDATE/DELETE, EXTRACT, substring keyword forms, and MATCH now pass complete parse, exact emission, and stable reparse checks. Source verification checks 167 exact grammar anchors per pinned release. These counts describe this corpus only.
57
+
The 212-case baseline has 110`CHECKED` witnesses and 92`REJECTED` negative controls. Its 10 remaining known gaps are five`TRAILING_INPUT`, three `NO_AST`, one `PARTIAL`, and one `EMISSION_MISMATCH`. CTE-prefixed UPDATE/DELETE, EXTRACT, substring keyword forms, MATCH, casts/conversions, charset-introduced literals, and COLLATE now pass complete parse, exact emission, and stable reparse checks. Source verification checks 212 exact source anchors per pinned release. These counts describe this corpus only.
56
58
57
59
The JSON report includes all original and reparsed records, source verification counts, pinned revisions, SQL-mode metadata, and SHA-256 hashes of the corpus and probe. Counts apply only to these witnesses. The probe uses hex-line input and hex fields in its TSV response, preserving tabs, newlines and embedded NUL bytes. It emits while both the input string and parser arena are alive; Python reparses owned copies in a second probe invocation.
58
60
@@ -71,9 +73,9 @@ Each curated case explicitly chooses direct query execution, `EXPLAIN` for DML (
71
73
72
74
Native results are stored separately in `native_oracle`: `ACCEPTED`, `SYNTAX_REJECTED` (1064/1149), `SERVER_ERROR`, `CLIENT_ERROR`, or **NOT VERIFIED**. For example, a derived table missing its required alias returns error 1248; this remains a server error, even though the corpus requires ParserSQL to reject that input. Accepted invalid controls and syntax-rejected valid controls count as expectation mismatches, using per-version validity when specified. Server errors remain explicitly unresolved; client errors return status 2. Thus exit status 0 still does not mean all native cases were verified. The default corpus metadata remains **NOT VERIFIED**, while the optional report describes the actual native observations.
73
75
74
-
A native check of all 170 witnesses under `ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION` found no native expectation mismatches. MySQL 8.4.8 accepted 92, syntax-rejected 70, and returned five server errors (two 1248 missing-alias errors, 3942, 3943, and 3569); MySQL 9.7.2 accepted 94, syntax-rejected 66, and returned seven server errors (the same five plus 1235 and 3889). Both runs skipped the three DDL/procedure witnesses and cleaned up their owned schemas. All shared positive query/DML witnesses except the three explicitly grammar-valid semantic-error controls were accepted, including partition selection, index hints, query constructors, recursive CTEs, CTE-prefixed UPDATE/DELETE, all twenty EXTRACT units, substring aliases, and MATCH modes. Re-run the command on each isolated socket to reproduce current observations; reports are intentionally not committed.
76
+
A native check of all 212 witnesses under `ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION` found no native expectation mismatches. MySQL 8.4.8 accepted 110, syntax-rejected 94, and returned five server errors (two 1248 missing-alias errors, 3942, 3943, and 3569); MySQL 9.7.2 accepted 112, syntax-rejected 90, and returned seven server errors (the same five plus 1235 and 3889). Both runs skipped the three DDL/procedure witnesses and cleaned up their owned schemas. All shared positive query/DML witnesses except the three explicitly grammar-valid semantic-error controls were accepted, including partition selection, index hints, query constructors, recursive CTEs, CTE-prefixed UPDATE/DELETE, all twenty EXTRACT units, substring aliases, MATCH modes, casts/conversions, charset-introduced literals, and COLLATE operands. Re-run the command on each isolated socket to reproduce current observations; reports are intentionally not committed.
75
77
76
-
The corpus is deliberately selected and is not a representative workload. It contains 166 shared witnesses and four version-specific query witnesses; this does not imply identical dialects in 8.4.8 and 9.7.2. The version-specific entries record `applicability`, source productions for both versions, `native_validity` per version, and observed native outcomes. Their generic `validity=valid` denotes the 9.7.2 grammar target. The native helper selects the intended validity using the actual server version, so an expected 8.4.8 syntax rejection is not treated as a corpus error. Stored native observations are context, not substitutes for a current run. These additions remain ParserSQL gaps. The following differences include those four query witnesses and two separate DDL spot checks:
78
+
The corpus is deliberately selected and is not a representative workload. It contains 208 shared witnesses and four version-specific query witnesses; this does not imply identical dialects in 8.4.8 and 9.7.2. The version-specific entries record `applicability`, source productions for both versions, `native_validity` per version, and observed native outcomes. Their generic `validity=valid` denotes the 9.7.2 grammar target. The native helper selects the intended validity using the actual server version, so an expected 8.4.8 syntax rejection is not treated as a corpus error. Stored native observations are context, not substitutes for a current run. These additions remain ParserSQL gaps. The following differences include those four query witnesses and two separate DDL spot checks:
77
79
78
80
| Reduced feature | MySQL 8.4.8 observation | MySQL 9.7.2 observation |
0 commit comments