Skip to content

Commit aa7da89

Browse files
committed
feat: expand MySQL INSERT sources and preserve incoming row aliases
Parse INSERT VALUES and SET row aliases, including optional nonempty column alias lists before ON DUPLICATE KEY UPDATE. Append a dedicated AST node containing identifier children so aliases retain quoting, survive cloning and remain visible to traversal. Reject alias suffixes on REPLACE and query sources, and keep parameterization conservative for this node. Replace the permissive INSERT helper path with native production checks: validate option order, table targets and partitions, qualified column names, row delimiters, complete values, SET assignments and every keyword and assignment in ON DUPLICATE KEY UPDATE. Support VALUE as a synonym for VALUES, empty native target/row constructors, DEFAULT row values and := assignments, with canonical emission. Do not treat SELECT table aliases, index hints or arbitrary trailing tokens as INSERT target syntax. Use the existing query callback for INSERT query sources instead of a standalone SELECT parser. Preserve set operations, parenthesized sources, WITH queries, TABLE and VALUES ROW, plus nested scalar queries and trailing duplicate-key updates. PostgreSQL continues through its existing dedicated DML parser; remove unreachable legacy PostgreSQL helpers from InsertParser. Fix backtick tokenization so doubled backticks remain inside one delimited identifier. Preserve lexical separation before row-alias column lists: AS count (a) is valid native syntax, while AS count(a) becomes a function token. Reject adjacent unquoted aliases for the 35 function names shared by both pinned native lexers; retain quoted, spaced and comment-separated forms. Document the remaining version-specific function-token distinction. Reject unsupported aliases, partitions and query sources in direct MySQL INSERT planning as well as the common feature guard. Scope this check to INSERT so existing MySQL UPDATE/DELETE original-AST routing remains intact. The full suite caught and verified correction of an initially broader planner check that blocked the multi-table UPDATE path. Expand the selected compatibility corpus from 212 to 260 witnesses and promote insert_alias from TRAILING_INPUT to CHECKED. The corpus now has 127 CHECKED cases, 124 REJECTED malformed controls and nine documented remaining gaps. Verify 268 exact upstream grammar/lexer anchors for each release; these counts do not measure dialect-wide SQL coverage. Validation: - Forced make -B -j6 all build-corpus-test and make test: 1,621 passed, with 37 external-database tests skipped. - ASan/UBSan with leak detection: 105 focused MySQL tests passed. - MySQL Python harness: 11 tests passed; all 260 expected corpus outcomes matched, with source anchors verified for MySQL 8.4.8 and 9.7.2. - Native 8.4.8: 126 accepted, 126 syntax-rejected, five server errors and three skipped DDL/procedure witnesses. Native 9.7.2: 128 accepted, 122 syntax-rejected, seven server errors and three skipped witnesses. No native expectation mismatches; owned schemas and servers cleaned up. - PostgreSQL harness: 87 tests passed; 5,122 CI cases and the full pinned 49,848/51,415 accepted-statement baseline remain unchanged. - Independent review verified 175 alias probes; all 140 accepted emissions remained stable after reparsing, with no remaining actionable findings. - Seven parser/emitter benchmark smoke cases passed; no new performance advantage or controlled benchmark comparison is claimed. Keep PR #67 a work in progress. Exclude benchmark code, reports and benchmark-only README/.gitignore changes from the commit.
1 parent 6961511 commit aa7da89

10 files changed

Lines changed: 1716 additions & 392 deletions

File tree

‎Makefile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@ TEST_SRCS = $(TEST_DIR)/test_main.cpp \
7272
$(TEST_DIR)/test_mysql_special_expressions.cpp \
7373
$(TEST_DIR)/test_mysql_cast_types.cpp \
7474
$(TEST_DIR)/test_mysql_conversion_expressions.cpp \
75+
$(TEST_DIR)/test_mysql_insert_sources.cpp \
7576
$(TEST_DIR)/test_review_ast.cpp \
7677
$(TEST_DIR)/test_review_transactions.cpp \
7778
$(TEST_DIR)/test_review_runtime.cpp \

‎docs/mysql-compatibility.md‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,11 @@ MySQL query expressions also preserve explicit `TABLE name` and `VALUES ROW(...)
1616

1717
Top-level MySQL `WITH` and `WITH RECURSIVE` also retain structured UPDATE and DELETE main statements, including join/comma UPDATEs, both multi-table DELETE target-list forms, quoted CTE names, partitions, statement options, and single-table ordering/limits. CTE bodies and subqueries remain query-only; DML inside them and a leading `WITH ... INSERT` are rejected. UPDATE assignments require one-, two-, or three-part column names and accept both `=` and `:=` (emitted as `=`), including `DEFAULT` values. UPDATE options follow `LOW_PRIORITY` then `IGNORE`; DELETE options may appear in any order and repeat. The [8.4.8 UPDATE/DELETE productions](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/sql_yacc.yy#L13389) and [9.7.2 UPDATE/DELETE productions](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L13945) attach an optional WITH clause to each main statement. Routing metadata identifies the main DML target, including wildcard DELETE targets. JOIN conditions require complete ON/USING operands and retain quoted USING column names. These are parser/AST guarantees; the local engine rejects CTE-prefixed DML.
1818

19+
MySQL INSERT constructors and SET sources retain `AS row_alias` and optional column aliases before `ON DUPLICATE KEY UPDATE`, including quoted names and doubled-backtick escapes. Alias emission separates the row name from its column-list parentheses. With `IGNORE_SPACE` off, native function names such as `COUNT` and `MAX` require this separator when used as unquoted row aliases; adjacent parentheses turn them into function tokens and are rejected. Explicit alias-column parentheses require a nonempty identifier list; the native empty production means that the entire list is omitted. `VALUE` emits canonically as `VALUES`, and `:=` assignment operators emit as `=`. SET and duplicate-key assignments require complete column targets and values, including qualified targets and `DEFAULT`. INSERT also accepts partition lists and query sources containing set operations, parentheses, WITH, TABLE, and `VALUES ROW(...)`. Row aliases belong to INSERT's VALUES/SET source productions; REPLACE and query sources do not accept this suffix. REPLACE also excludes duplicate-key updates. The [8.4.8 INSERT/REPLACE productions](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/sql_yacc.yy#L13064) and [9.7.2 INSERT/REPLACE productions](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L13620), with the corpus's constructor, alias-list, and assignment anchors, define these bounded parser/AST guarantees. Parameterization remains conservative for row aliases. Local INSERT execution and direct INSERT planning reject unsupported row aliases, query sources, and partition selection. Native checks use EXPLAIN; they do not establish local-engine execution support.
20+
1921
The native grammar permits empty `VALUES ROW()`, `VALUES ROW(DEFAULT)`, and repeated locking clauses. Both pinned servers return semantic errors 3942, 3943, and 3569 respectively for the curated standalone or overlapping-lock witnesses. These remain `SERVER_ERROR` observations, not syntax rejections, while ParserSQL preserves the grammar-valid ASTs. `TABLE` takes a table identifier and query-level tails; aliases, partitions, index hints, and WHERE clauses directly after its table name are malformed controls. The [8.4.8 query-primary productions](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/sql_yacc.yy#L9831) and [9.7.2 query-primary productions](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L10145) include both constructors. The corpus records exact anchors for their rows, subquery contexts, locking, CTEs, and malformed controls.
2022

21-
Table identifiers and aliases use the common 259-word reserved baseline observed in the pinned 8.4.8 and 9.7.2 releases. ParserSQL has no MySQL version setting, so the five 9.7-only reserved additions (`CUBE`, `EXTERNAL`, `LIBRARY`, `QUALIFY`, and `TABLESAMPLE`) remain accepted as names. Strict version-specific identifier rules remain a gap.
23+
Table identifiers and aliases use the common 259-word reserved baseline observed in the pinned 8.4.8 and 9.7.2 releases. ParserSQL has no MySQL version setting, so the five 9.7-only reserved additions (`CUBE`, `EXTERNAL`, `LIBRARY`, `QUALIFY`, and `TABLESAMPLE`) remain accepted as names. Strict version-specific identifier rules remain a gap. The INSERT alias function-token check likewise uses the 35 names shared by both pins; the additional 9.7 `JSON_DUALITY_OBJECT` function-token distinction remains outside that common baseline.
2224

2325
Numeric-looking unquoted qualified table names such as `db.123` are valid natively but remain unsupported: the tokenizer reads `.123` as a numeric literal. ParserSQL rejects these forms; earlier behavior could emit an incorrect alias. Lossless parsing of this identifier edge remains a gap.
2426

@@ -54,7 +56,7 @@ Malformed controls are handled separately: a complete AST is `MALFORMED_ACCEPTED
5456

5557
Every fixture freezes one exact expected outcome. Known gaps do not make arbitrary failures acceptable. Both regression and unexpected improvement cause exit status 1, so a fixed gap must be reviewed and explicitly promoted in the corpus. When promoting, add an independently derived `expected_sql`, set `known_gap=false`, and require `CHECKED`. Harness/transport/source errors return 2. Exit status 0 means the expected baseline matched, including its documented known gaps.
5658

57-
The 212-case baseline has 110 `CHECKED` witnesses and 92 `REJECTED` negative controls. Its 10 remaining known gaps are five `TRAILING_INPUT`, three `NO_AST`, one `PARTIAL`, and one `EMISSION_MISMATCH`. CTE-prefixed UPDATE/DELETE, EXTRACT, substring keyword forms, MATCH, casts/conversions, charset-introduced literals, and COLLATE now pass complete parse, exact emission, and stable reparse checks. Source verification checks 212 exact source anchors per pinned release. These counts describe this corpus only.
59+
The 260-case baseline has 127 `CHECKED` witnesses and 124 `REJECTED` negative controls. Its nine remaining known gaps are four `TRAILING_INPUT`, three `NO_AST`, one `PARTIAL`, and one `EMISSION_MISMATCH`. CTE-prefixed UPDATE/DELETE, EXTRACT, substring keyword forms, MATCH, casts/conversions, charset-introduced literals, COLLATE, and the curated INSERT/REPLACE forms now pass complete parse, exact emission, and stable reparse checks. Source verification checks 268 exact source anchors per pinned release. These counts describe this corpus only.
5860

5961
The JSON report includes all original and reparsed records, source verification counts, pinned revisions, SQL-mode metadata, and SHA-256 hashes of the corpus and probe. Counts apply only to these witnesses. The probe uses hex-line input and hex fields in its TSV response, preserving tabs, newlines and embedded NUL bytes. It emits while both the input string and parser arena are alive; Python reparses owned copies in a second probe invocation.
6062

@@ -73,9 +75,9 @@ Each curated case explicitly chooses direct query execution, `EXPLAIN` for DML (
7375

7476
Native results are stored separately in `native_oracle`: `ACCEPTED`, `SYNTAX_REJECTED` (1064/1149), `SERVER_ERROR`, `CLIENT_ERROR`, or **NOT VERIFIED**. For example, a derived table missing its required alias returns error 1248; this remains a server error, even though the corpus requires ParserSQL to reject that input. Accepted invalid controls and syntax-rejected valid controls count as expectation mismatches, using per-version validity when specified. Server errors remain explicitly unresolved; client errors return status 2. Thus exit status 0 still does not mean all native cases were verified. The default corpus metadata remains **NOT VERIFIED**, while the optional report describes the actual native observations.
7577

76-
A native check of all 212 witnesses under `ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION` found no native expectation mismatches. MySQL 8.4.8 accepted 110, syntax-rejected 94, and returned five server errors (two 1248 missing-alias errors, 3942, 3943, and 3569); MySQL 9.7.2 accepted 112, syntax-rejected 90, and returned seven server errors (the same five plus 1235 and 3889). Both runs skipped the three DDL/procedure witnesses and cleaned up their owned schemas. All shared positive query/DML witnesses except the three explicitly grammar-valid semantic-error controls were accepted, including partition selection, index hints, query constructors, recursive CTEs, CTE-prefixed UPDATE/DELETE, all twenty EXTRACT units, substring aliases, MATCH modes, casts/conversions, charset-introduced literals, and COLLATE operands. Re-run the command on each isolated socket to reproduce current observations; reports are intentionally not committed.
78+
A native check of all 260 witnesses under `ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION` found no native expectation mismatches. MySQL 8.4.8 accepted 126, syntax-rejected 126, and returned five server errors (two 1248 missing-alias errors, 3942, 3943, and 3569); MySQL 9.7.2 accepted 128, syntax-rejected 122, and returned seven server errors (the same five plus 1235 and 3889). Both runs skipped the three DDL/procedure witnesses and cleaned up their owned schemas. All shared positive query/DML witnesses except the three explicitly grammar-valid semantic-error controls were accepted, including partition selection, index hints, query constructors, recursive CTEs, CTE-prefixed UPDATE/DELETE, all twenty EXTRACT units, substring aliases, MATCH modes, casts/conversions, charset-introduced literals, COLLATE operands, and INSERT constructors, aliases, assignments, and query sources. The 48 added INSERT/REPLACE witnesses contributed 16 accepted EXPLAIN statements and 32 syntax errors in each version. Re-run the command on each isolated socket to reproduce current observations; reports are intentionally not committed.
7779

78-
The corpus is deliberately selected and is not a representative workload. It contains 208 shared witnesses and four version-specific query witnesses; this does not imply identical dialects in 8.4.8 and 9.7.2. The version-specific entries record `applicability`, source productions for both versions, `native_validity` per version, and observed native outcomes. Their generic `validity=valid` denotes the 9.7.2 grammar target. The native helper selects the intended validity using the actual server version, so an expected 8.4.8 syntax rejection is not treated as a corpus error. Stored native observations are context, not substitutes for a current run. These additions remain ParserSQL gaps. The following differences include those four query witnesses and two separate DDL spot checks:
80+
The corpus is deliberately selected and is not a representative workload. It contains 256 shared witnesses and four version-specific query witnesses; this does not imply identical dialects in 8.4.8 and 9.7.2. The version-specific entries record `applicability`, source productions for both versions, `native_validity` per version, and observed native outcomes. Their generic `validity=valid` denotes the 9.7.2 grammar target. The native helper selects the intended validity using the actual server version, so an expected 8.4.8 syntax rejection is not treated as a corpus error. Stored native observations are context, not substitutes for a current run. These additions remain ParserSQL gaps. The following differences include those four query witnesses and two separate DDL spot checks:
7981

8082
| Reduced feature | MySQL 8.4.8 observation | MySQL 9.7.2 observation |
8183
| --- | --- | --- |

‎include/sql_engine/dml_plan_builder.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -206,6 +206,10 @@ class DmlPlanBuilder {
206206
if (!ast) return false;
207207
if constexpr (D == sql_parser::Dialect::PostgreSQL) {
208208
if (!PlanBuilder<D>::supports_dml_features(ast) || unsupported_dml(ast)) return false;
209+
} else if (ast->type == sql_parser::NodeType::NODE_INSERT_STMT) {
210+
// INSERT plans must not discard newly parsed aliases or sources.
211+
// UPDATE/DELETE retain their existing original-AST routing paths.
212+
if (!PlanBuilder<D>::supports_dml_features(ast) || unsupported_dml(ast)) return false;
209213
}
210214
return true;
211215
}

‎include/sql_engine/plan_builder.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -194,6 +194,7 @@ class PlanBuilder {
194194
case NodeType::NODE_MYSQL_CHARSET_LITERAL:
195195
case NodeType::NODE_MYSQL_COLLATE:
196196
case NodeType::NODE_MYSQL_CONVERT:
197+
case NodeType::NODE_MYSQL_INSERT_ALIAS:
197198
case NodeType::NODE_MYSQL_EXTRACT:
198199
case NodeType::NODE_MYSQL_SUBSTRING:
199200
case NodeType::NODE_MYSQL_MATCH:

‎include/sql_parser/common.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -361,6 +361,7 @@ enum class NodeType : uint16_t {
361361
NODE_MYSQL_CHARSET_LITERAL, // value=charset introducer; string/hex/bit literal children
362362
NODE_MYSQL_COLLATE, // value=collation source spelling; one expression child
363363
NODE_MYSQL_CONVERT, // expression, type/charset metadata; flag 1=USING form
364+
NODE_MYSQL_INSERT_ALIAS, // row identifier, optional nonempty column identifier list
364365

365366
};
366367

‎include/sql_parser/emitter.h‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,21 @@ class Emitter {
9898
case NodeType::NODE_VALUES_CLAUSE: emit_values_clause(node); break;
9999
case NodeType::NODE_VALUES_ROW: emit_values_row(node); break;
100100
case NodeType::NODE_INSERT_SET_CLAUSE: emit_insert_set_clause(node); break;
101+
case NodeType::NODE_MYSQL_INSERT_ALIAS: {
102+
sb_.append("AS ");
103+
const auto* name = node->first_child;
104+
if (name) emit_node(name);
105+
if (name && name->next_sibling) {
106+
// COUNT (a) is an alias; COUNT(a) is a native function token.
107+
sb_.append(" (");
108+
for (auto* column = name->next_sibling; column; column = column->next_sibling) {
109+
if (column != name->next_sibling) sb_.append(", ");
110+
emit_node(column);
111+
}
112+
sb_.append_char(')');
113+
}
114+
break;
115+
}
101116
case NodeType::NODE_ON_DUPLICATE_KEY: emit_on_duplicate_key(node); break;
102117
case NodeType::NODE_ON_CONFLICT: emit_on_conflict(node); break;
103118
case NodeType::NODE_CONFLICT_TARGET: emit_conflict_target(node); break;

0 commit comments

Comments
 (0)