You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 798c6dc
Browse filesBrowse the repository at this point in the historyBrowse files
feat: expand MySQL aggregates, grouping sets and EXPLAIN syntax
Parse native JSON_ARRAYAGG and JSON_OBJECTAGG with their required argument
counts, complete expression operands, optional ALL on each argument and
window clauses. Preserve MySQL 9.7 NULL ON NULL and ABSENT ON NULL on
JSON_ARRAYAGG, rejecting those clauses on JSON_OBJECTAGG and rejecting
DISTINCT, bare stars, incomplete operands and unsupported aggregate tails.
Append dedicated aggregate/argument AST nodes and retain conservative
local-execution and parameterization guards.
Keep native adjacent calls distinct from ordinary spaced or quoted calls.
A dedicated function flag preserves one lexical separator when emitting
spaced/comment-separated JSON aggregate-name calls, without changing the
function name stored in the AST. This avoids turning a valid ordinary
function call into a native aggregate with different syntax or arity.
Represent GROUP BY ROLLUP/CUBE and MySQL 9.7 GROUPING SETS as structured
grouping operations. Preserve expression lists, parentheses and empty
sets, while rejecting mixed top-level alternatives, malformed sets and
nested unquoted ROLLUP/CUBE calls. Quoted ordinary function names retain
their separate meaning. ROLLUP and CUBE are present in both pinned native
releases; grouping syntax support does not imply engine execution support.
Parse MySQL EXPLAIN options in native order and retain INTO destinations
as user-variable AST children. Preserve format and variable quoting,
strict query/DML bodies, and DESC/DESCRIBE statement synonyms. Use the
strict query callback so incomplete inner SELECT operands cannot appear
successful, including parenthesized query bodies ending in a semicolon.
Plain INTO exists in both pins; ANALYZE INTO is a MySQL 9.7 addition.
Retain table-description syntax and routing names with quoted identifiers,
string/hex/bit column filters, and FORMAT as an ordinary table name when
not followed by '='. Remove the unreachable legacy EXPLAIN implementation
while leaving the dedicated PostgreSQL option path unchanged. Native
format restrictions and unsupported engine features remain separately
reported semantic observations rather than invented parser execution.
Add 80 curated witnesses and promote four tracked gaps: both JSON array
null clauses, EXPLAIN ANALYZE INTO and GROUPING SETS. The 340-case corpus
now requires 166 CHECKED results and 169 REJECTED malformed controls, with
five documented remaining gaps. Verify 363 exact source anchors per pin
and preserve version-specific validity expectations. These figures apply
only to the selected corpus, not to the percentage of MySQL supported.
Validation:
- Forced make -B -j6 all build-corpus-test and make test: 1,643 passed;
37 external-database tests skipped.
- ASan/UBSan with leak detection: 127 focused MySQL tests passed.
- MySQL harness: 11 tests passed; all 340 corpus outcomes and source
anchors matched with exact expected emission and stable reparse.
- Native MySQL 8.4.8: 147 accepted, 175 syntax-rejected, 15 server errors
and three skipped DDL/procedure witnesses. MySQL 9.7.2: 152 accepted,
165 syntax-rejected, 20 server errors and three skipped witnesses.
Zero expectation mismatches; owned fixture schemas and servers cleaned.
- PostgreSQL harness: 87 tests passed; all 5,122 committed CI cases and
the full pinned 49,848/51,415 accepted-statement baseline are unchanged.
- Independent review identified and verified fixes for nested grouping,
FORMAT-table ambiguity and hex/bit description filters. Final review
checked 18 cases; all 13 successful emissions reparsed stably.
- Seven parser/emitter benchmark smoke cases passed. No new performance
comparison or speedup is claimed.
Keep PR #67 a work in progress. Benchmark code, reports and benchmark-only
README/.gitignore modifications are excluded from this commit.
Copy file name to clipboardExpand all lines: docs/mysql-compatibility.md
+10-4Lines changed: 10 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,13 @@
2
2
3
3
This harness tracks a small, deliberately selected set of MySQL syntax witnesses and known ParserSQL gaps. It does not measure the percentage of the MySQL dialect supported, implement mysqltest/MTR extraction, or establish query-engine execution support.
4
4
5
-
ParserSQL now preserves MySQL JSON arrows, named/inherited windows, `GROUP_CONCAT` modifiers and separators, and `GROUP BY ... WITH ROLLUP` as structured ASTs. Database-qualified column names are retained. The local engine rejects these unsupported semantics rather than executing a simplified interpretation; parameterization remains conservative for JSON arrows and `GROUP_CONCAT`. Empty MySQL SELECT target lists are rejected.
5
+
ParserSQL now preserves MySQL JSON arrows, named/inherited windows, `GROUP_CONCAT` modifiers and separators, and grouping alternatives including `GROUP BY ... WITH ROLLUP` as structured ASTs. Database-qualified column names are retained. The local engine rejects these unsupported semantics rather than executing a simplified interpretation; parameterization remains conservative for JSON arrows and `GROUP_CONCAT`. Empty MySQL SELECT target lists are rejected.
6
+
7
+
MySQL JSON aggregates use structured ASTs with native argument counts: `JSON_ARRAYAGG` takes one expression and `JSON_OBJECTAGG` takes two. Each argument accepts an explicit `ALL`; `DISTINCT`, a bare star, in-call ordering, and wrong arities are rejected. Both functions preserve optional window clauses. With `IGNORE_SPACE` off, these built-in forms require an adjacent opening parenthesis; spaced or quoted names remain ordinary function calls. The 9.7.2 `JSON_ARRAYAGG` extension preserves `NULL ON NULL` and `ABSENT ON NULL`; neither clause belongs to `JSON_OBJECTAGG`. These clauses are absent from the 8.4.8 production. Native 9.7.2 accepts `NULL ON NULL` but returns unsupported-feature error 1235 for the curated `ABSENT ON NULL` witnesses outside JSON duality views. The [8.4.8 JSON aggregate productions](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/sql_yacc.yy#L10988) and [9.7.2 JSON aggregate productions](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L11397) define this bounded coverage. These are syntax/AST guarantees; the local engine rejects the new aggregate nodes, and parameterization remains conservative.
8
+
9
+
`GROUP BY ROLLUP(...)` and `GROUP BY CUBE(...)` exist in both pinned versions; `GROUP BY GROUPING SETS ((...), ())` is a 9.7.2 addition. These standalone grouping alternatives retain their expression lists and empty grouping sets. Mixed top-level lists such as `id, ROLLUP(n)`, missing per-set parentheses, and combining `ROLLUP(...)` with `WITH ROLLUP` are rejected. The [8.4.8 grouping productions](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/sql_yacc.yy#L12455) and [9.7.2 grouping productions](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L12879) distinguish syntax from execution prerequisites: the native fixture accepts ROLLUP, while CUBE and nonempty GROUPING SETS witnesses return secondary-engine error 3889. A row-valued ROLLUP operand is grammar-valid but returns semantic error 1241. Unquoted nested ROLLUP/CUBE calls in grouping expressions are rejected; quoted names remain ordinary function calls and produce server error 1630 on the fixture because no such stored functions exist. ParserSQL preserves these ASTs and rejects unsupported local execution.
10
+
11
+
MySQL `EXPLAIN` retains options in native order: optional `ANALYZE`, optional `FORMAT = name`, then optional `INTO @variable`. Plain EXPLAIN INTO is present in both pins; ANALYZE INTO is a 9.7.2 addition. Formats and destination names retain their source quoting, including escaped quotes/backticks. Query bodies include WITH, TABLE, and VALUES expressions. Table-description forms emit as DESCRIBE, accept FORMAT as an ordinary table identifier, and preserve hex/bit column-filter literals. The two missing FORMAT-table witnesses return native errors 1146 and 1049, while the four literal-filter witnesses are accepted. Duplicate or reordered options, system-variable destinations, and missing query bodies are rejected. INTO without an explicit format and INTO with TREE remain grammar-valid observations with native semantic errors 6006 and 6007. The [8.4.8 EXPLAIN option productions](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/sql_yacc.yy#L14034) and [9.7.2 EXPLAIN option productions](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L14611) define this bounded syntax support. Native EXPLAIN ANALYZE checks execute only SELECT queries on owned fixture connections; these observations do not establish ParserSQL execution support. EXPLAIN FOR CONNECTION and FOR DATABASE remain outside this batch.
6
12
7
13
MySQL `EXTRACT(unit FROM expr)`, keyword forms of `SUBSTRING`/`SUBSTR`/`MID`, and `MATCH ... AGAINST` have structured ASTs. EXTRACT accepts the twenty native interval keywords, including compound units, and rejects quoted or unknown units. Substring calls retain the alias spelling and support both comma arguments and `FROM ... [FOR ...]`. MATCH preserves qualified column lists, default/natural/boolean modes and query expansion; its optional column-list parentheses are emitted canonically. The search operand enforces the native `bit_expr` precedence boundary, so a comparison must be parenthesized before a search mode. General expression gaps still apply inside these forms: unary `!` operands are accepted by both pinned servers but remain unsupported by ParserSQL. Complete-operand parsing rejects empty tuples, invalid ROW arity, missing grouping/query delimiters, invalid EXISTS bodies and incomplete CASE branches. The local engine rejects these special AST semantics; parameterization remains conservative for the new node kinds. The [8.4.8 special-function productions](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/sql_yacc.yy#L10661) and [9.7.2 special-function productions](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L11070), together with the corpus's MATCH, interval, and identifier anchors, define this bounded syntax coverage.
8
14
@@ -56,7 +62,7 @@ Malformed controls are handled separately: a complete AST is `MALFORMED_ACCEPTED
56
62
57
63
Every fixture freezes one exact expected outcome. Known gaps do not make arbitrary failures acceptable. Both regression and unexpected improvement cause exit status 1, so a fixed gap must be reviewed and explicitly promoted in the corpus. When promoting, add an independently derived `expected_sql`, set `known_gap=false`, and require `CHECKED`. Harness/transport/source errors return 2. Exit status 0 means the expected baseline matched, including its documented known gaps.
58
64
59
-
The 260-case baseline has 127`CHECKED` witnesses and 124`REJECTED` negative controls. Its nine remaining known gaps are four`TRAILING_INPUT`, three `NO_AST`, one `PARTIAL`, and one `EMISSION_MISMATCH`. CTE-prefixed UPDATE/DELETE, EXTRACT, substring keyword forms, MATCH, casts/conversions, charset-introduced literals, COLLATE, and the curated INSERT/REPLACE forms now pass complete parse, exact emission, and stable reparse checks. Source verification checks 268 exact source anchors per pinned release. These counts describe this corpus only.
65
+
The 340-case baseline has 166`CHECKED` witnesses and 169`REJECTED` negative controls. Its five remaining known gaps are one`TRAILING_INPUT`, three `NO_AST`, and one `EMISSION_MISMATCH`: JSON_TABLE, CREATE TABLE, ALTER TABLE, CREATE PROCEDURE, and optimizer-hint preservation. JSON aggregates, grouping alternatives, and EXPLAIN INTO now join the previously supported witnesses in passing complete parse, exact emission, and stable reparse checks. Source verification checks 363 exact source anchors per pinned release. These counts describe this corpus only.
60
66
61
67
The JSON report includes all original and reparsed records, source verification counts, pinned revisions, SQL-mode metadata, and SHA-256 hashes of the corpus and probe. Counts apply only to these witnesses. The probe uses hex-line input and hex fields in its TSV response, preserving tabs, newlines and embedded NUL bytes. It emits while both the input string and parser arena are alive; Python reparses owned copies in a second probe invocation.
62
68
@@ -75,9 +81,9 @@ Each curated case explicitly chooses direct query execution, `EXPLAIN` for DML (
75
81
76
82
Native results are stored separately in `native_oracle`: `ACCEPTED`, `SYNTAX_REJECTED` (1064/1149), `SERVER_ERROR`, `CLIENT_ERROR`, or **NOT VERIFIED**. For example, a derived table missing its required alias returns error 1248; this remains a server error, even though the corpus requires ParserSQL to reject that input. Accepted invalid controls and syntax-rejected valid controls count as expectation mismatches, using per-version validity when specified. Server errors remain explicitly unresolved; client errors return status 2. Thus exit status 0 still does not mean all native cases were verified. The default corpus metadata remains **NOT VERIFIED**, while the optional report describes the actual native observations.
77
83
78
-
A native check of all 260 witnesses under `ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION` found no native expectation mismatches. MySQL 8.4.8 accepted 126, syntax-rejected 126, and returned five server errors (two 1248 missing-alias errors, 3942, 3943, and 3569); MySQL 9.7.2 accepted 128, syntax-rejected 122, and returned seven server errors (the same five plus 1235 and 3889). Both runs skipped the three DDL/procedure witnesses and cleaned up their owned schemas. All shared positive query/DML witnesses except the three explicitly grammar-valid semantic-error controls were accepted, including partition selection, index hints, query constructors, recursive CTEs, CTE-prefixed UPDATE/DELETE, all twenty EXTRACT units, substring aliases, MATCH modes, casts/conversions, charset-introduced literals, COLLATE operands, and INSERT constructors, aliases, assignments, and query sources. The 48 added INSERT/REPLACE witnesses contributed 16 accepted EXPLAIN statements and 32 syntax errors in each version. Re-run the command on each isolated socket to reproduce current observations; reports are intentionally not committed.
84
+
A native check of all 340 witnesses under `ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION` found no native expectation mismatches. MySQL 8.4.8 accepted 147, syntax-rejected 175, and returned fifteen server errors; MySQL 9.7.2 accepted 152, syntax-rejected 165, and returned twenty server errors. Both runs skipped the three DDL/procedure witnesses and cleaned up their owned schemas. Server errors retain the existing missing-alias, empty/default VALUES, and repeated-lock observations, plus the documented unsupported JSON null behavior, secondary-engine requirements, row-valued grouping operand, quoted grouping-function names, missing FORMAT tables/databases, and EXPLAIN format restrictions. Two malformed EXPLAIN controls fail at semantic checks before native parsing reaches the later error: misplaced FORMAT returns 6006, and a missing destination name returns 3061. These remain unresolved server errors, not syntax rejections. Re-run the command on each isolated socket to reproduce current observations; reports are intentionally not committed.
79
85
80
-
The corpus is deliberately selected and is not a representative workload. It contains 256 shared witnesses and four version-specific query witnesses; this does not imply identical dialects in 8.4.8 and 9.7.2. The version-specific entries record `applicability`, source productions for both versions, `native_validity` per version, and observed native outcomes. Their generic `validity=valid` denotes the 9.7.2 grammar target. The native helper selects the intended validity using the actual server version, so an expected 8.4.8 syntax rejection is not treated as a corpus error. Stored native observations are context, not substitutes for a current run. These additions remain ParserSQL gaps. The following differences include those four query witnesses and two separate DDL spot checks:
86
+
The corpus is deliberately selected and is not a representative workload. It contains 330 shared witnesses and ten version-specific query witnesses; this does not imply identical dialects in 8.4.8 and 9.7.2. The version-specific entries record `applicability`, source productions for both versions, `native_validity` per version, and observed native outcomes. Their generic `validity=valid` denotes the 9.7.2 grammar target. The native helper selects the intended validity using the actual server version, so an expected 8.4.8 syntax rejection is not treated as a corpus error. Stored native observations are context, not substitutes for a current run. These version-specific query additions now pass ParserSQL parse/emission/reparse checks; native runtime restrictions remain separately reported. The following differences include the original four query witnesses and two separate DDL spot checks:
81
87
82
88
| Reduced feature | MySQL 8.4.8 observation | MySQL 9.7.2 observation |
0 commit comments