Skip to content

Commit c51a246

Browse files
committed
feat: support MySQL TABLE and VALUES queries with query CTEs
Parse TABLE name and VALUES ROW(...) as structured query expressions, including set operations, ordered/limited branches, scalar subqueries, IN/EXISTS predicates, derived tables, and LATERAL sources. Retain explicit ROW syntax with a node-local flag so ordinary INSERT VALUES emission is unchanged. Preserve set-operation precedence and parenthesized grouping. Replace the permissive MySQL WITH path with a shared query-only parser that requires names, AS, delimiters, complete bodies, and a main query. Preserve quoted CTE names, output-column lists, and RECURSIVE. CTE-prefixed UPDATE and DELETE remain explicit unsupported errors rather than partial query trees. Production parsing has no MySQL server dependency. Extend query locking tails to TABLE/VALUES and compound queries, retaining repeated clauses and LOCK IN SHARE MODE. Distinguish VALUES(col) in existing INSERT updates from a VALUES query. Inspect parenthesized query boundaries in a linear scan without discarding EXISTS bodies or consuming enclosing scalar arithmetic and IN-list expressions as queries. Validate MySQL LIMIT operands as unsigned 64-bit integers, placeholders, or identifier references. Reject wildcard values and DEFAULT embedded in expressions while retaining standalone DEFAULT and empty ROW constructors: those forms are grammar-valid but native semantic errors. Keep quoted star identifiers and COUNT(*) intact. Guard new query constructors and legacy share-lock syntax against unsupported local execution; parameterization preserves ROW spelling and ORDER BY constants. Expand the compatibility corpus from 74 to 113 independently specified witnesses. Promote TABLE, VALUES, and recursive CTE coverage, retain the remaining 17 known gaps, and verify 110 grammar anchors per pinned version. Document native semantic errors separately from syntax rejection. Add 13 focused C++ regression tests for strict parsing, faithful emission, nested query boundaries, parameterization, and planner guards. Validation: - Forced full build and make test: 1,552 passed, 37 external DB tests skipped. - ASan/UBSan with leak detection: all 36 focused MySQL tests passed. - All 113 corpus expectations match: 62 CHECKED, 34 malformed controls REJECTED, and 17 explicitly tracked gaps; 11 harness tests passed. - Native MySQL 8.4.8 and 9.7.2 report zero validity mismatches. Semantic errors and three skipped DDL cases remain separately unresolved. - PostgreSQL harness: 87 passed; 5,122 committed fixtures and the full 51,415-statement pinned baseline match unchanged. - Independent final review verifies faithful stable emission for 14 affected queries; seven parser/emitter benchmark smoke cases passed. Benchmark code, reports, and benchmark-only README/.gitignore edits are excluded from this commit.
1 parent c2a092a commit c51a246

13 files changed

Lines changed: 1430 additions & 130 deletions

File tree

‎Makefile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,7 @@ TEST_SRCS = $(TEST_DIR)/test_main.cpp \
6767
$(TEST_DIR)/test_review_expressions.cpp \
6868
$(TEST_DIR)/test_mysql_grammar.cpp \
6969
$(TEST_DIR)/test_mysql_tables.cpp \
70+
$(TEST_DIR)/test_mysql_query_expressions.cpp \
7071
$(TEST_DIR)/test_review_ast.cpp \
7172
$(TEST_DIR)/test_review_transactions.cpp \
7273
$(TEST_DIR)/test_review_runtime.cpp \

‎docs/mysql-compatibility.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,11 @@ This harness tracks a small, deliberately selected set of MySQL syntax witnesses
44

55
ParserSQL now preserves MySQL JSON arrows, named/inherited windows, `GROUP_CONCAT` modifiers and separators, and `GROUP BY ... WITH ROLLUP` as structured ASTs. Database-qualified column names are retained. The local engine rejects these unsupported semantics rather than executing a simplified interpretation; parameterization remains conservative for JSON arrows and `GROUP_CONCAT`. Empty MySQL SELECT target lists are rejected.
66

7-
ParserSQL also preserves partition lists, `USE`/`FORCE`/`IGNORE INDEX` or `KEY`, hint scopes and empty `USE INDEX ()`, and `LATERAL` derived SELECTs with required aliases and optional alias-column lists. The local engine rejects these unsupported table-reference semantics. `SELECT` and `UPDATE` place `PARTITION` before the table alias; single-table `DELETE` places it after the alias. The current LATERAL implementation scope is SELECT-based queries, including UNION: both native releases also accept `LATERAL (TABLE t) AS q` and `LATERAL (VALUES ROW(1)) AS q`, but those forms remain outside this implementation and are not malformed controls.
7+
ParserSQL also preserves partition lists, `USE`/`FORCE`/`IGNORE INDEX` or `KEY`, hint scopes and empty `USE INDEX ()`, and `LATERAL` derived SELECTs with required aliases and optional alias-column lists. The local engine rejects these unsupported table-reference semantics. `SELECT` and `UPDATE` place `PARTITION` before the table alias; single-table `DELETE` places it after the alias. LATERAL derived queries include `SELECT`, `TABLE`, and `VALUES ROW(...)` expressions and their set operations.
8+
9+
MySQL query expressions also preserve explicit `TABLE name` and `VALUES ROW(...)` constructors in standalone, compound, derived, LATERAL, scalar, IN, EXISTS, and query-only CTE contexts. `INTERSECT` binds more tightly than `UNION` and `EXCEPT`; explicit parentheses and branch-local ordering/limits are retained. Recursive CTEs retain their keyword and optional column lists. Query locking clauses, including `LOCK IN SHARE MODE`, remain structured. This is parser/AST coverage; the local engine rejects unsupported query forms.
10+
11+
The native grammar permits empty `VALUES ROW()`, `VALUES ROW(DEFAULT)`, and repeated locking clauses. Both pinned servers return semantic errors 3942, 3943, and 3569 respectively for the curated standalone or overlapping-lock witnesses. These remain `SERVER_ERROR` observations, not syntax rejections, while ParserSQL preserves the grammar-valid ASTs. `TABLE` takes a table identifier and query-level tails; aliases, partitions, index hints, and WHERE clauses directly after its table name are malformed controls. The [8.4.8 query-primary productions](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/sql_yacc.yy#L9831) and [9.7.2 query-primary productions](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L10145) include both constructors. The corpus records exact anchors for their rows, subquery contexts, locking, CTEs, and malformed controls.
812

913
Table identifiers and aliases use the common 259-word reserved baseline observed in the pinned 8.4.8 and 9.7.2 releases. ParserSQL has no MySQL version setting, so the five 9.7-only reserved additions (`CUBE`, `EXTERNAL`, `LIBRARY`, `QUALIFY`, and `TABLESAMPLE`) remain accepted as names. Strict version-specific identifier rules remain a gap.
1014

@@ -42,7 +46,7 @@ Malformed controls are handled separately: a complete AST is `MALFORMED_ACCEPTED
4246

4347
Every fixture freezes one exact expected outcome. Known gaps do not make arbitrary failures acceptable. Both regression and unexpected improvement cause exit status 1, so a fixed gap must be reviewed and explicitly promoted in the corpus. When promoting, add an independently derived `expected_sql`, set `known_gap=false`, and require `CHECKED`. Harness/transport/source errors return 2. Exit status 0 means the expected baseline matched, including its documented known gaps.
4448

45-
The 74-case baseline has 33 `CHECKED` witnesses and 21 `REJECTED` negative controls. Its remaining known gaps are 13 `TRAILING_INPUT`, five `NO_AST`, one `PARTIAL`, and one `EMISSION_MISMATCH`. Source verification checks 71 exact grammar anchors per pinned release. These counts describe this corpus only.
49+
The 113-case baseline has 62 `CHECKED` witnesses and 34 `REJECTED` negative controls. Its remaining known gaps are ten `TRAILING_INPUT`, three `NO_AST`, two `ERROR`, one `PARTIAL`, and one `EMISSION_MISMATCH`. The two `ERROR` witnesses are CTE-prefixed UPDATE/DELETE: the query-only WITH parser now rejects these unsupported main statements directly. Source verification checks 110 exact grammar anchors per pinned release. These counts describe this corpus only.
4650

4751
The JSON report includes all original and reparsed records, source verification counts, pinned revisions, SQL-mode metadata, and SHA-256 hashes of the corpus and probe. Counts apply only to these witnesses. The probe uses hex-line input and hex fields in its TSV response, preserving tabs, newlines and embedded NUL bytes. It emits while both the input string and parser arena are alive; Python reparses owned copies in a second probe invocation.
4852

@@ -61,9 +65,9 @@ Each curated case explicitly chooses direct query execution, `EXPLAIN` for DML (
6165

6266
Native results are stored separately in `native_oracle`: `ACCEPTED`, `SYNTAX_REJECTED` (1064/1149), `SERVER_ERROR`, `CLIENT_ERROR`, or **NOT VERIFIED**. For example, a derived table missing its required alias returns error 1248; this remains a server error, even though the corpus requires ParserSQL to reject that input. Accepted invalid controls and syntax-rejected valid controls count as expectation mismatches, using per-version validity when specified. Server errors remain explicitly unresolved; client errors return status 2. Thus exit status 0 still does not mean all native cases were verified. The default corpus metadata remains **NOT VERIFIED**, while the optional report describes the actual native observations.
6367

64-
A native check of all 74 witnesses under `ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION` found no native expectation mismatches. MySQL 8.4.8 accepted 46, syntax-rejected 24, and returned error 1248 for the missing-alias control; MySQL 9.7.2 accepted 48, syntax-rejected 20, and returned three server errors (1248, 1235, and 3889). Both runs skipped the three DDL/procedure witnesses and cleaned up their owned schemas. All shared positive query/DML witnesses, including partition selection and index hints, were accepted. Re-run the command on each isolated socket to reproduce current observations; reports are intentionally not committed.
68+
A native check of all 113 witnesses under `ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION` found no native expectation mismatches. MySQL 8.4.8 accepted 69, syntax-rejected 36, and returned five server errors (two 1248 missing-alias errors, 3942, 3943, and 3569); MySQL 9.7.2 accepted 71, syntax-rejected 32, and returned seven server errors (the same five plus 1235 and 3889). Both runs skipped the three DDL/procedure witnesses and cleaned up their owned schemas. All shared positive query/DML witnesses except the three explicitly grammar-valid semantic-error controls were accepted, including partition selection, index hints, query constructors, and recursive CTEs. Re-run the command on each isolated socket to reproduce current observations; reports are intentionally not committed.
6569

66-
The corpus is deliberately selected and is not a representative workload. It contains 70 shared witnesses and four version-specific query witnesses; this does not imply identical dialects in 8.4.8 and 9.7.2. The version-specific entries record `applicability`, source productions for both versions, `native_validity` per version, and observed native outcomes. Their generic `validity=valid` denotes the 9.7.2 grammar target. The native helper selects the intended validity using the actual server version, so an expected 8.4.8 syntax rejection is not treated as a corpus error. Stored native observations are context, not substitutes for a current run. These additions remain ParserSQL gaps. The following differences include those four query witnesses and two separate DDL spot checks:
70+
The corpus is deliberately selected and is not a representative workload. It contains 109 shared witnesses and four version-specific query witnesses; this does not imply identical dialects in 8.4.8 and 9.7.2. The version-specific entries record `applicability`, source productions for both versions, `native_validity` per version, and observed native outcomes. Their generic `validity=valid` denotes the 9.7.2 grammar target. The native helper selects the intended validity using the actual server version, so an expected 8.4.8 syntax rejection is not treated as a corpus error. Stored native observations are context, not substitutes for a current run. These additions remain ParserSQL gaps. The following differences include those four query witnesses and two separate DDL spot checks:
6771

6872
| Reduced feature | MySQL 8.4.8 observation | MySQL 9.7.2 observation |
6973
| --- | --- | --- |

‎include/sql_engine/plan_builder.h‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -176,6 +176,12 @@ class PlanBuilder {
176176
case NodeType::NODE_CTE_COLUMNS:
177177
case NodeType::NODE_TABLE_QUERY:
178178
return true;
179+
case NodeType::NODE_VALUES_ROW:
180+
if (node->flags & sql_parser::FLAG_VALUES_EXPLICIT_ROW) return true;
181+
break;
182+
case NodeType::NODE_LOCKING_CLAUSE:
183+
if (!node->value().empty()) return true;
184+
break;
179185
case NodeType::NODE_VALUES_CLAUSE:
180186
if (!dml_values) return true;
181187
break;

‎include/sql_parser/common.h‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,8 @@ static constexpr uint16_t FLAG_PG_OPERATOR = 0x01;
7373
static constexpr uint16_t FLAG_WINDOW_BETWEEN = 0x01;
7474
static constexpr uint16_t FLAG_ORDER_NULLS = 0x01;
7575
static constexpr uint16_t FLAG_LIMIT_COMMA = 0x01;
76+
// MySQL table-value constructors spell each NODE_VALUES_ROW as ROW(...).
77+
static constexpr uint16_t FLAG_VALUES_EXPLICIT_ROW = 0x01;
7678
static constexpr uint16_t FLAG_CTE_RECURSIVE = 0x01;
7779
static constexpr uint16_t FLAG_CTE_MATERIALIZED = 0x02;
7880
static constexpr uint16_t FLAG_CTE_NOT_MATERIALIZED = 0x04;

0 commit comments

Comments
 (0)