Skip to content

Commit 6961511

Browse files
committed
feat: parse MySQL casts, conversions, charset literals and collations
Replace the permissive MySQL CAST-as-function path with validated native cast_type grammar. Retain numeric, temporal, character/binary, JSON and spatial targets, accepted modifiers, character attributes, and lexical aliases such as FLOAT4, FLOAT8 and INT4. Distinguish broad field-length numeric tokens from the bounded NUM production used by temporal and paired decimal precision, leaving runtime range checks to the server. Represent typed and USING CONVERT forms, charset-introduced text/hex/bit literals and expression COLLATE with appended AST node kinds. Preserve operands, raw metadata spelling, adjacent introduced strings and native collation precedence. Recognize the shared native charset names and utf8 alias without consuming unknown underscore-prefixed identifiers. Reject reserved or introducer metadata tokens, incomplete operands, malformed modifiers and unsupported conversion tails. Emit the new syntax without losing charset, collation or type information. Preserve original double-quote delimiters for MySQL string values so embedded apostrophes and doubled double quotes remain valid. Continue to emit the current AST value rather than stale source text. Reject odd-length quoted hex literals and uppercase numeric introducers where native MySQL rejects them. Keep PostgreSQL behavior unchanged. Guard local planning against semantics the engine does not implement. Keep parameterization conservative for the new node kinds; existing CAST parameterization binds values while retaining type precision and scale. Add regression coverage for strict rejection, exact emission and reparse, AST cloning, planning guards, parameterization and native lexical edges. Expand the curated MySQL corpus from 170 to 212 witnesses and promote the CAST-decimal and charset-literal gaps to CHECKED. The baseline now records 110 CHECKED, 92 REJECTED and 10 remaining documented gaps; these counts are specific to the selected corpus, not a dialect-wide coverage metric. Verify 212 exact upstream source anchors for each pinned release and update the compatibility guide with supported forms and explicit limits. Validation: - Forced full rebuild: make -B -j6 all build-corpus-test; make test. 1,611 tests passed, with 37 external-database tests skipped. - ASan/UBSan with leak detection: 95 focused MySQL tests passed. - MySQL harness: 11 tests passed; all 212 expected outcomes matched. - Native MySQL 8.4.8: 110 accepted, 94 syntax rejections, five server errors and three skipped DDL/procedure witnesses. - Native MySQL 9.7.2: 112 accepted, 90 syntax rejections, seven server errors and three skipped DDL/procedure witnesses. - Both native runs had zero expectation mismatches, verified source anchors, and cleaned up their owned schemas and server instances. - Four double-quote regression queries passed native checks on both pins. - PostgreSQL harness: 87 tests passed; all 5,122 committed CI cases and the full 49,848/51,415 accepted-statement baseline remain unchanged. - Seven parser/emitter benchmark smoke cases passed; this is not a controlled performance comparison and makes no new speed claim. - Independent review verified native grammar edge fixes and the final delimiter-preservation change. Keep PR #67 a work in progress. Benchmark code, reports and benchmark-only README/.gitignore changes are deliberately excluded from this commit.
1 parent 7015c8e commit 6961511

12 files changed

Lines changed: 1701 additions & 32 deletions

‎Makefile‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,8 @@ TEST_SRCS = $(TEST_DIR)/test_main.cpp \
7070
$(TEST_DIR)/test_mysql_query_expressions.cpp \
7171
$(TEST_DIR)/test_mysql_cte_dml.cpp \
7272
$(TEST_DIR)/test_mysql_special_expressions.cpp \
73+
$(TEST_DIR)/test_mysql_cast_types.cpp \
74+
$(TEST_DIR)/test_mysql_conversion_expressions.cpp \
7375
$(TEST_DIR)/test_review_ast.cpp \
7476
$(TEST_DIR)/test_review_transactions.cpp \
7577
$(TEST_DIR)/test_review_runtime.cpp \

‎docs/mysql-compatibility.md‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,9 @@ This harness tracks a small, deliberately selected set of MySQL syntax witnesses
44

55
ParserSQL now preserves MySQL JSON arrows, named/inherited windows, `GROUP_CONCAT` modifiers and separators, and `GROUP BY ... WITH ROLLUP` as structured ASTs. Database-qualified column names are retained. The local engine rejects these unsupported semantics rather than executing a simplified interpretation; parameterization remains conservative for JSON arrows and `GROUP_CONCAT`. Empty MySQL SELECT target lists are rejected.
66

7-
MySQL `EXTRACT(unit FROM expr)`, keyword forms of `SUBSTRING`/`SUBSTR`/`MID`, and `MATCH ... AGAINST` have structured ASTs. EXTRACT accepts the twenty native interval keywords, including compound units, and rejects quoted or unknown units. Substring calls retain the alias spelling and support both comma arguments and `FROM ... [FOR ...]`. MATCH preserves qualified column lists, default/natural/boolean modes and query expansion; its optional column-list parentheses are emitted canonically. The search operand enforces the native `bit_expr` precedence boundary, so a comparison must be parenthesized before a search mode. General expression gaps still apply inside these forms: unary `!` and `COLLATE` operands are accepted by both pinned servers but remain unsupported by ParserSQL. Complete-operand parsing rejects empty tuples, invalid ROW arity, missing grouping/query delimiters, invalid EXISTS bodies and incomplete CASE branches. The local engine rejects these special AST semantics; parameterization remains conservative for the new node kinds. The [8.4.8 special-function productions](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/sql_yacc.yy#L10661) and [9.7.2 special-function productions](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L11070), together with the corpus's MATCH, interval, and identifier anchors, define this bounded syntax coverage.
7+
MySQL `EXTRACT(unit FROM expr)`, keyword forms of `SUBSTRING`/`SUBSTR`/`MID`, and `MATCH ... AGAINST` have structured ASTs. EXTRACT accepts the twenty native interval keywords, including compound units, and rejects quoted or unknown units. Substring calls retain the alias spelling and support both comma arguments and `FROM ... [FOR ...]`. MATCH preserves qualified column lists, default/natural/boolean modes and query expansion; its optional column-list parentheses are emitted canonically. The search operand enforces the native `bit_expr` precedence boundary, so a comparison must be parenthesized before a search mode. General expression gaps still apply inside these forms: unary `!` operands are accepted by both pinned servers but remain unsupported by ParserSQL. Complete-operand parsing rejects empty tuples, invalid ROW arity, missing grouping/query delimiters, invalid EXISTS bodies and incomplete CASE branches. The local engine rejects these special AST semantics; parameterization remains conservative for the new node kinds. The [8.4.8 special-function productions](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/sql_yacc.yy#L10661) and [9.7.2 special-function productions](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L11070), together with the corpus's MATCH, interval, and identifier anchors, define this bounded syntax coverage.
8+
9+
MySQL `CAST(expr AS cast_type)`, both `CONVERT(expr, cast_type)` and `CONVERT(expr USING charset)`, charset-introduced text/hex/bit literals, and expression `COLLATE` retain structured ASTs. Cast types follow the native cast-specific productions rather than general column definitions: numeric, temporal, character/binary, JSON, and spatial targets preserve their accepted modifiers and source spelling. Character targets retain charset and binary attributes. Introducers recognize the shared native charset names (including the `utf8` alias), support intervening whitespace, and retain literal kind. Literal emission preserves MySQL double-quote delimiters, including embedded apostrophes and doubled double quotes, with `ANSI_QUOTES` off. Expression `COLLATE` accepts a simple identifier or quoted name, with native precedence; bare `BINARY`, `DEFAULT`, qualified names, and missing operands are rejected. `CONVERT ... USING` additionally accepts bare `BINARY`. The [8.4.8 cast/conversion productions](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/sql_yacc.yy#L10425) and [9.7.2 cast/conversion productions](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L10745), plus the corpus's exact type, literal, charset, and collation anchors, define this bounded grammar coverage. Cast `ARRAY`, `AT LOCAL`, and `AT TIME ZONE` forms remain outside this batch. Native lexical aliases such as `FLOAT4`, `FLOAT8`, and `INT4` are retained. Single field lengths permit native decimal-number tokens; two-part decimal precision and temporal precision use a native integer `NUM` token from 0 through 2147483647. Runtime precision limits remain native semantic checks. These nodes are syntax/AST support: the local engine rejects unsupported semantics, and parameterization remains conservative.
810

911
With `IGNORE_SPACE` off, native EXTRACT and substring aliases require `(` immediately after the function name. ParserSQL follows that lexical boundary for these special forms; `MATCH (` remains valid. Spaced comma calls are ordinary function-call syntax and can resolve differently on a native server. This is not general SQL-mode emulation. The [native lexer](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/sql_lex.cc#L1567) distinguishes adjacent function tokens, and [the function-token table](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/lex.h#L831) maps `MID` and `SUBSTR` to `SUBSTRING`.
1012

@@ -52,7 +54,7 @@ Malformed controls are handled separately: a complete AST is `MALFORMED_ACCEPTED
5254

5355
Every fixture freezes one exact expected outcome. Known gaps do not make arbitrary failures acceptable. Both regression and unexpected improvement cause exit status 1, so a fixed gap must be reviewed and explicitly promoted in the corpus. When promoting, add an independently derived `expected_sql`, set `known_gap=false`, and require `CHECKED`. Harness/transport/source errors return 2. Exit status 0 means the expected baseline matched, including its documented known gaps.
5456

55-
The 170-case baseline has 90 `CHECKED` witnesses and 68 `REJECTED` negative controls. Its 12 remaining known gaps are seven `TRAILING_INPUT`, three `NO_AST`, one `PARTIAL`, and one `EMISSION_MISMATCH`. CTE-prefixed UPDATE/DELETE, EXTRACT, substring keyword forms, and MATCH now pass complete parse, exact emission, and stable reparse checks. Source verification checks 167 exact grammar anchors per pinned release. These counts describe this corpus only.
57+
The 212-case baseline has 110 `CHECKED` witnesses and 92 `REJECTED` negative controls. Its 10 remaining known gaps are five `TRAILING_INPUT`, three `NO_AST`, one `PARTIAL`, and one `EMISSION_MISMATCH`. CTE-prefixed UPDATE/DELETE, EXTRACT, substring keyword forms, MATCH, casts/conversions, charset-introduced literals, and COLLATE now pass complete parse, exact emission, and stable reparse checks. Source verification checks 212 exact source anchors per pinned release. These counts describe this corpus only.
5658

5759
The JSON report includes all original and reparsed records, source verification counts, pinned revisions, SQL-mode metadata, and SHA-256 hashes of the corpus and probe. Counts apply only to these witnesses. The probe uses hex-line input and hex fields in its TSV response, preserving tabs, newlines and embedded NUL bytes. It emits while both the input string and parser arena are alive; Python reparses owned copies in a second probe invocation.
5860

@@ -71,9 +73,9 @@ Each curated case explicitly chooses direct query execution, `EXPLAIN` for DML (
7173

7274
Native results are stored separately in `native_oracle`: `ACCEPTED`, `SYNTAX_REJECTED` (1064/1149), `SERVER_ERROR`, `CLIENT_ERROR`, or **NOT VERIFIED**. For example, a derived table missing its required alias returns error 1248; this remains a server error, even though the corpus requires ParserSQL to reject that input. Accepted invalid controls and syntax-rejected valid controls count as expectation mismatches, using per-version validity when specified. Server errors remain explicitly unresolved; client errors return status 2. Thus exit status 0 still does not mean all native cases were verified. The default corpus metadata remains **NOT VERIFIED**, while the optional report describes the actual native observations.
7375

74-
A native check of all 170 witnesses under `ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION` found no native expectation mismatches. MySQL 8.4.8 accepted 92, syntax-rejected 70, and returned five server errors (two 1248 missing-alias errors, 3942, 3943, and 3569); MySQL 9.7.2 accepted 94, syntax-rejected 66, and returned seven server errors (the same five plus 1235 and 3889). Both runs skipped the three DDL/procedure witnesses and cleaned up their owned schemas. All shared positive query/DML witnesses except the three explicitly grammar-valid semantic-error controls were accepted, including partition selection, index hints, query constructors, recursive CTEs, CTE-prefixed UPDATE/DELETE, all twenty EXTRACT units, substring aliases, and MATCH modes. Re-run the command on each isolated socket to reproduce current observations; reports are intentionally not committed.
76+
A native check of all 212 witnesses under `ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION` found no native expectation mismatches. MySQL 8.4.8 accepted 110, syntax-rejected 94, and returned five server errors (two 1248 missing-alias errors, 3942, 3943, and 3569); MySQL 9.7.2 accepted 112, syntax-rejected 90, and returned seven server errors (the same five plus 1235 and 3889). Both runs skipped the three DDL/procedure witnesses and cleaned up their owned schemas. All shared positive query/DML witnesses except the three explicitly grammar-valid semantic-error controls were accepted, including partition selection, index hints, query constructors, recursive CTEs, CTE-prefixed UPDATE/DELETE, all twenty EXTRACT units, substring aliases, MATCH modes, casts/conversions, charset-introduced literals, and COLLATE operands. Re-run the command on each isolated socket to reproduce current observations; reports are intentionally not committed.
7577

76-
The corpus is deliberately selected and is not a representative workload. It contains 166 shared witnesses and four version-specific query witnesses; this does not imply identical dialects in 8.4.8 and 9.7.2. The version-specific entries record `applicability`, source productions for both versions, `native_validity` per version, and observed native outcomes. Their generic `validity=valid` denotes the 9.7.2 grammar target. The native helper selects the intended validity using the actual server version, so an expected 8.4.8 syntax rejection is not treated as a corpus error. Stored native observations are context, not substitutes for a current run. These additions remain ParserSQL gaps. The following differences include those four query witnesses and two separate DDL spot checks:
78+
The corpus is deliberately selected and is not a representative workload. It contains 208 shared witnesses and four version-specific query witnesses; this does not imply identical dialects in 8.4.8 and 9.7.2. The version-specific entries record `applicability`, source productions for both versions, `native_validity` per version, and observed native outcomes. Their generic `validity=valid` denotes the 9.7.2 grammar target. The native helper selects the intended validity using the actual server version, so an expected 8.4.8 syntax rejection is not treated as a corpus error. Stored native observations are context, not substitutes for a current run. These additions remain ParserSQL gaps. The following differences include those four query witnesses and two separate DDL spot checks:
7779

7880
| Reduced feature | MySQL 8.4.8 observation | MySQL 9.7.2 observation |
7981
| --- | --- | --- |

‎include/sql_engine/plan_builder.h‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -191,6 +191,9 @@ class PlanBuilder {
191191
break;
192192
case NodeType::NODE_NAMED_ARGUMENT:
193193
case NodeType::NODE_TYPE_CAST:
194+
case NodeType::NODE_MYSQL_CHARSET_LITERAL:
195+
case NodeType::NODE_MYSQL_COLLATE:
196+
case NodeType::NODE_MYSQL_CONVERT:
194197
case NodeType::NODE_MYSQL_EXTRACT:
195198
case NodeType::NODE_MYSQL_SUBSTRING:
196199
case NodeType::NODE_MYSQL_MATCH:

‎include/sql_parser/common.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -358,6 +358,10 @@ enum class NodeType : uint16_t {
358358
NODE_MYSQL_MATCH_COLUMNS, // nonempty list of simple or qualified columns
359359
NODE_MYSQL_MATCH, // value=validated search mode; column list, search expression
360360

361+
NODE_MYSQL_CHARSET_LITERAL, // value=charset introducer; string/hex/bit literal children
362+
NODE_MYSQL_COLLATE, // value=collation source spelling; one expression child
363+
NODE_MYSQL_CONVERT, // expression, type/charset metadata; flag 1=USING form
364+
361365
};
362366

363367
} // namespace sql_parser

‎include/sql_parser/emitter.h‎

Lines changed: 23 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -179,6 +179,21 @@ class Emitter {
179179
break;
180180
case NodeType::NODE_UNARY_OP: emit_unary_op(node); break;
181181
case NodeType::NODE_FUNCTION_CALL: emit_function_call(node); break;
182+
case NodeType::NODE_MYSQL_CHARSET_LITERAL:
183+
emit_value(node);
184+
for (const auto* literal = node->first_child; literal; literal = literal->next_sibling) {
185+
if (literal != node->first_child || literal->type != NodeType::NODE_LITERAL_STRING)
186+
sb_.append_char(' ');
187+
emit_mysql_syntax_string(literal);
188+
}
189+
break;
190+
case NodeType::NODE_MYSQL_COLLATE:
191+
emit_node(node->first_child); sb_.append(" COLLATE "); emit_value(node); break;
192+
case NodeType::NODE_MYSQL_CONVERT:
193+
sb_.append("CONVERT("); emit_node(node->first_child);
194+
sb_.append(node->flags & 1 ? " USING " : ", ");
195+
emit_node(node->first_child ? node->first_child->next_sibling : nullptr);
196+
sb_.append_char(')'); break;
182197
case NodeType::NODE_MYSQL_EXTRACT:
183198
case NodeType::NODE_MYSQL_SUBSTRING:
184199
case NodeType::NODE_MYSQL_MATCH: emit_mysql_special(node); break;
@@ -651,9 +666,15 @@ class Emitter {
651666
return;
652667
}
653668
}
654-
sb_.append_char('\'');
669+
char delimiter = '\'';
670+
if constexpr (D == Dialect::MySQL) {
671+
// Token values retain lexical escapes. Changing a double-quoted
672+
// delimiter can invalidate apostrophes or doubled double quotes.
673+
if (!node->source().empty() && node->source_ptr[0] == '"') delimiter = '"';
674+
}
675+
sb_.append_char(delimiter);
655676
sb_.append(node->value_ptr, node->value_len);
656-
sb_.append_char('\'');
677+
sb_.append_char(delimiter);
657678
}
658679

659680
void emit_placeholder(const AstNode* node) {

0 commit comments

Comments
 (0)