Skip to content

Commit 798c6dc

Browse files
committed
feat: expand MySQL aggregates, grouping sets and EXPLAIN syntax
Parse native JSON_ARRAYAGG and JSON_OBJECTAGG with their required argument counts, complete expression operands, optional ALL on each argument and window clauses. Preserve MySQL 9.7 NULL ON NULL and ABSENT ON NULL on JSON_ARRAYAGG, rejecting those clauses on JSON_OBJECTAGG and rejecting DISTINCT, bare stars, incomplete operands and unsupported aggregate tails. Append dedicated aggregate/argument AST nodes and retain conservative local-execution and parameterization guards. Keep native adjacent calls distinct from ordinary spaced or quoted calls. A dedicated function flag preserves one lexical separator when emitting spaced/comment-separated JSON aggregate-name calls, without changing the function name stored in the AST. This avoids turning a valid ordinary function call into a native aggregate with different syntax or arity. Represent GROUP BY ROLLUP/CUBE and MySQL 9.7 GROUPING SETS as structured grouping operations. Preserve expression lists, parentheses and empty sets, while rejecting mixed top-level alternatives, malformed sets and nested unquoted ROLLUP/CUBE calls. Quoted ordinary function names retain their separate meaning. ROLLUP and CUBE are present in both pinned native releases; grouping syntax support does not imply engine execution support. Parse MySQL EXPLAIN options in native order and retain INTO destinations as user-variable AST children. Preserve format and variable quoting, strict query/DML bodies, and DESC/DESCRIBE statement synonyms. Use the strict query callback so incomplete inner SELECT operands cannot appear successful, including parenthesized query bodies ending in a semicolon. Plain INTO exists in both pins; ANALYZE INTO is a MySQL 9.7 addition. Retain table-description syntax and routing names with quoted identifiers, string/hex/bit column filters, and FORMAT as an ordinary table name when not followed by '='. Remove the unreachable legacy EXPLAIN implementation while leaving the dedicated PostgreSQL option path unchanged. Native format restrictions and unsupported engine features remain separately reported semantic observations rather than invented parser execution. Add 80 curated witnesses and promote four tracked gaps: both JSON array null clauses, EXPLAIN ANALYZE INTO and GROUPING SETS. The 340-case corpus now requires 166 CHECKED results and 169 REJECTED malformed controls, with five documented remaining gaps. Verify 363 exact source anchors per pin and preserve version-specific validity expectations. These figures apply only to the selected corpus, not to the percentage of MySQL supported. Validation: - Forced make -B -j6 all build-corpus-test and make test: 1,643 passed; 37 external-database tests skipped. - ASan/UBSan with leak detection: 127 focused MySQL tests passed. - MySQL harness: 11 tests passed; all 340 corpus outcomes and source anchors matched with exact expected emission and stable reparse. - Native MySQL 8.4.8: 147 accepted, 175 syntax-rejected, 15 server errors and three skipped DDL/procedure witnesses. MySQL 9.7.2: 152 accepted, 165 syntax-rejected, 20 server errors and three skipped witnesses. Zero expectation mismatches; owned fixture schemas and servers cleaned. - PostgreSQL harness: 87 tests passed; all 5,122 committed CI cases and the full pinned 49,848/51,415 accepted-statement baseline are unchanged. - Independent review identified and verified fixes for nested grouping, FORMAT-table ambiguity and hex/bit description filters. Final review checked 18 cases; all 13 successful emissions reparsed stably. - Seven parser/emitter benchmark smoke cases passed. No new performance comparison or speedup is claimed. Keep PR #67 a work in progress. Benchmark code, reports and benchmark-only README/.gitignore modifications are excluded from this commit.
1 parent aa7da89 commit 798c6dc

13 files changed

Lines changed: 3322 additions & 155 deletions

File tree

‎Makefile‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,9 @@ TEST_SRCS = $(TEST_DIR)/test_main.cpp \
7373
$(TEST_DIR)/test_mysql_cast_types.cpp \
7474
$(TEST_DIR)/test_mysql_conversion_expressions.cpp \
7575
$(TEST_DIR)/test_mysql_insert_sources.cpp \
76+
$(TEST_DIR)/test_mysql_json_aggregates.cpp \
77+
$(TEST_DIR)/test_mysql_grouping.cpp \
78+
$(TEST_DIR)/test_mysql_explain.cpp \
7679
$(TEST_DIR)/test_review_ast.cpp \
7780
$(TEST_DIR)/test_review_transactions.cpp \
7881
$(TEST_DIR)/test_review_runtime.cpp \

‎docs/mysql-compatibility.md‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,13 @@
22

33
This harness tracks a small, deliberately selected set of MySQL syntax witnesses and known ParserSQL gaps. It does not measure the percentage of the MySQL dialect supported, implement mysqltest/MTR extraction, or establish query-engine execution support.
44

5-
ParserSQL now preserves MySQL JSON arrows, named/inherited windows, `GROUP_CONCAT` modifiers and separators, and `GROUP BY ... WITH ROLLUP` as structured ASTs. Database-qualified column names are retained. The local engine rejects these unsupported semantics rather than executing a simplified interpretation; parameterization remains conservative for JSON arrows and `GROUP_CONCAT`. Empty MySQL SELECT target lists are rejected.
5+
ParserSQL now preserves MySQL JSON arrows, named/inherited windows, `GROUP_CONCAT` modifiers and separators, and grouping alternatives including `GROUP BY ... WITH ROLLUP` as structured ASTs. Database-qualified column names are retained. The local engine rejects these unsupported semantics rather than executing a simplified interpretation; parameterization remains conservative for JSON arrows and `GROUP_CONCAT`. Empty MySQL SELECT target lists are rejected.
6+
7+
MySQL JSON aggregates use structured ASTs with native argument counts: `JSON_ARRAYAGG` takes one expression and `JSON_OBJECTAGG` takes two. Each argument accepts an explicit `ALL`; `DISTINCT`, a bare star, in-call ordering, and wrong arities are rejected. Both functions preserve optional window clauses. With `IGNORE_SPACE` off, these built-in forms require an adjacent opening parenthesis; spaced or quoted names remain ordinary function calls. The 9.7.2 `JSON_ARRAYAGG` extension preserves `NULL ON NULL` and `ABSENT ON NULL`; neither clause belongs to `JSON_OBJECTAGG`. These clauses are absent from the 8.4.8 production. Native 9.7.2 accepts `NULL ON NULL` but returns unsupported-feature error 1235 for the curated `ABSENT ON NULL` witnesses outside JSON duality views. The [8.4.8 JSON aggregate productions](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/sql_yacc.yy#L10988) and [9.7.2 JSON aggregate productions](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L11397) define this bounded coverage. These are syntax/AST guarantees; the local engine rejects the new aggregate nodes, and parameterization remains conservative.
8+
9+
`GROUP BY ROLLUP(...)` and `GROUP BY CUBE(...)` exist in both pinned versions; `GROUP BY GROUPING SETS ((...), ())` is a 9.7.2 addition. These standalone grouping alternatives retain their expression lists and empty grouping sets. Mixed top-level lists such as `id, ROLLUP(n)`, missing per-set parentheses, and combining `ROLLUP(...)` with `WITH ROLLUP` are rejected. The [8.4.8 grouping productions](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/sql_yacc.yy#L12455) and [9.7.2 grouping productions](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L12879) distinguish syntax from execution prerequisites: the native fixture accepts ROLLUP, while CUBE and nonempty GROUPING SETS witnesses return secondary-engine error 3889. A row-valued ROLLUP operand is grammar-valid but returns semantic error 1241. Unquoted nested ROLLUP/CUBE calls in grouping expressions are rejected; quoted names remain ordinary function calls and produce server error 1630 on the fixture because no such stored functions exist. ParserSQL preserves these ASTs and rejects unsupported local execution.
10+
11+
MySQL `EXPLAIN` retains options in native order: optional `ANALYZE`, optional `FORMAT = name`, then optional `INTO @variable`. Plain EXPLAIN INTO is present in both pins; ANALYZE INTO is a 9.7.2 addition. Formats and destination names retain their source quoting, including escaped quotes/backticks. Query bodies include WITH, TABLE, and VALUES expressions. Table-description forms emit as DESCRIBE, accept FORMAT as an ordinary table identifier, and preserve hex/bit column-filter literals. The two missing FORMAT-table witnesses return native errors 1146 and 1049, while the four literal-filter witnesses are accepted. Duplicate or reordered options, system-variable destinations, and missing query bodies are rejected. INTO without an explicit format and INTO with TREE remain grammar-valid observations with native semantic errors 6006 and 6007. The [8.4.8 EXPLAIN option productions](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/sql_yacc.yy#L14034) and [9.7.2 EXPLAIN option productions](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L14611) define this bounded syntax support. Native EXPLAIN ANALYZE checks execute only SELECT queries on owned fixture connections; these observations do not establish ParserSQL execution support. EXPLAIN FOR CONNECTION and FOR DATABASE remain outside this batch.
612

713
MySQL `EXTRACT(unit FROM expr)`, keyword forms of `SUBSTRING`/`SUBSTR`/`MID`, and `MATCH ... AGAINST` have structured ASTs. EXTRACT accepts the twenty native interval keywords, including compound units, and rejects quoted or unknown units. Substring calls retain the alias spelling and support both comma arguments and `FROM ... [FOR ...]`. MATCH preserves qualified column lists, default/natural/boolean modes and query expansion; its optional column-list parentheses are emitted canonically. The search operand enforces the native `bit_expr` precedence boundary, so a comparison must be parenthesized before a search mode. General expression gaps still apply inside these forms: unary `!` operands are accepted by both pinned servers but remain unsupported by ParserSQL. Complete-operand parsing rejects empty tuples, invalid ROW arity, missing grouping/query delimiters, invalid EXISTS bodies and incomplete CASE branches. The local engine rejects these special AST semantics; parameterization remains conservative for the new node kinds. The [8.4.8 special-function productions](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/sql_yacc.yy#L10661) and [9.7.2 special-function productions](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L11070), together with the corpus's MATCH, interval, and identifier anchors, define this bounded syntax coverage.
814

@@ -56,7 +62,7 @@ Malformed controls are handled separately: a complete AST is `MALFORMED_ACCEPTED
5662

5763
Every fixture freezes one exact expected outcome. Known gaps do not make arbitrary failures acceptable. Both regression and unexpected improvement cause exit status 1, so a fixed gap must be reviewed and explicitly promoted in the corpus. When promoting, add an independently derived `expected_sql`, set `known_gap=false`, and require `CHECKED`. Harness/transport/source errors return 2. Exit status 0 means the expected baseline matched, including its documented known gaps.
5864

59-
The 260-case baseline has 127 `CHECKED` witnesses and 124 `REJECTED` negative controls. Its nine remaining known gaps are four `TRAILING_INPUT`, three `NO_AST`, one `PARTIAL`, and one `EMISSION_MISMATCH`. CTE-prefixed UPDATE/DELETE, EXTRACT, substring keyword forms, MATCH, casts/conversions, charset-introduced literals, COLLATE, and the curated INSERT/REPLACE forms now pass complete parse, exact emission, and stable reparse checks. Source verification checks 268 exact source anchors per pinned release. These counts describe this corpus only.
65+
The 340-case baseline has 166 `CHECKED` witnesses and 169 `REJECTED` negative controls. Its five remaining known gaps are one `TRAILING_INPUT`, three `NO_AST`, and one `EMISSION_MISMATCH`: JSON_TABLE, CREATE TABLE, ALTER TABLE, CREATE PROCEDURE, and optimizer-hint preservation. JSON aggregates, grouping alternatives, and EXPLAIN INTO now join the previously supported witnesses in passing complete parse, exact emission, and stable reparse checks. Source verification checks 363 exact source anchors per pinned release. These counts describe this corpus only.
6066

6167
The JSON report includes all original and reparsed records, source verification counts, pinned revisions, SQL-mode metadata, and SHA-256 hashes of the corpus and probe. Counts apply only to these witnesses. The probe uses hex-line input and hex fields in its TSV response, preserving tabs, newlines and embedded NUL bytes. It emits while both the input string and parser arena are alive; Python reparses owned copies in a second probe invocation.
6268

@@ -75,9 +81,9 @@ Each curated case explicitly chooses direct query execution, `EXPLAIN` for DML (
7581

7682
Native results are stored separately in `native_oracle`: `ACCEPTED`, `SYNTAX_REJECTED` (1064/1149), `SERVER_ERROR`, `CLIENT_ERROR`, or **NOT VERIFIED**. For example, a derived table missing its required alias returns error 1248; this remains a server error, even though the corpus requires ParserSQL to reject that input. Accepted invalid controls and syntax-rejected valid controls count as expectation mismatches, using per-version validity when specified. Server errors remain explicitly unresolved; client errors return status 2. Thus exit status 0 still does not mean all native cases were verified. The default corpus metadata remains **NOT VERIFIED**, while the optional report describes the actual native observations.
7783

78-
A native check of all 260 witnesses under `ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION` found no native expectation mismatches. MySQL 8.4.8 accepted 126, syntax-rejected 126, and returned five server errors (two 1248 missing-alias errors, 3942, 3943, and 3569); MySQL 9.7.2 accepted 128, syntax-rejected 122, and returned seven server errors (the same five plus 1235 and 3889). Both runs skipped the three DDL/procedure witnesses and cleaned up their owned schemas. All shared positive query/DML witnesses except the three explicitly grammar-valid semantic-error controls were accepted, including partition selection, index hints, query constructors, recursive CTEs, CTE-prefixed UPDATE/DELETE, all twenty EXTRACT units, substring aliases, MATCH modes, casts/conversions, charset-introduced literals, COLLATE operands, and INSERT constructors, aliases, assignments, and query sources. The 48 added INSERT/REPLACE witnesses contributed 16 accepted EXPLAIN statements and 32 syntax errors in each version. Re-run the command on each isolated socket to reproduce current observations; reports are intentionally not committed.
84+
A native check of all 340 witnesses under `ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION` found no native expectation mismatches. MySQL 8.4.8 accepted 147, syntax-rejected 175, and returned fifteen server errors; MySQL 9.7.2 accepted 152, syntax-rejected 165, and returned twenty server errors. Both runs skipped the three DDL/procedure witnesses and cleaned up their owned schemas. Server errors retain the existing missing-alias, empty/default VALUES, and repeated-lock observations, plus the documented unsupported JSON null behavior, secondary-engine requirements, row-valued grouping operand, quoted grouping-function names, missing FORMAT tables/databases, and EXPLAIN format restrictions. Two malformed EXPLAIN controls fail at semantic checks before native parsing reaches the later error: misplaced FORMAT returns 6006, and a missing destination name returns 3061. These remain unresolved server errors, not syntax rejections. Re-run the command on each isolated socket to reproduce current observations; reports are intentionally not committed.
7985

80-
The corpus is deliberately selected and is not a representative workload. It contains 256 shared witnesses and four version-specific query witnesses; this does not imply identical dialects in 8.4.8 and 9.7.2. The version-specific entries record `applicability`, source productions for both versions, `native_validity` per version, and observed native outcomes. Their generic `validity=valid` denotes the 9.7.2 grammar target. The native helper selects the intended validity using the actual server version, so an expected 8.4.8 syntax rejection is not treated as a corpus error. Stored native observations are context, not substitutes for a current run. These additions remain ParserSQL gaps. The following differences include those four query witnesses and two separate DDL spot checks:
86+
The corpus is deliberately selected and is not a representative workload. It contains 330 shared witnesses and ten version-specific query witnesses; this does not imply identical dialects in 8.4.8 and 9.7.2. The version-specific entries record `applicability`, source productions for both versions, `native_validity` per version, and observed native outcomes. Their generic `validity=valid` denotes the 9.7.2 grammar target. The native helper selects the intended validity using the actual server version, so an expected 8.4.8 syntax rejection is not treated as a corpus error. Stored native observations are context, not substitutes for a current run. These version-specific query additions now pass ParserSQL parse/emission/reparse checks; native runtime restrictions remain separately reported. The following differences include the original four query witnesses and two separate DDL spot checks:
8187

8288
| Reduced feature | MySQL 8.4.8 observation | MySQL 9.7.2 observation |
8389
| --- | --- | --- |

‎include/sql_engine/plan_builder.h‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -195,6 +195,9 @@ class PlanBuilder {
195195
case NodeType::NODE_MYSQL_COLLATE:
196196
case NodeType::NODE_MYSQL_CONVERT:
197197
case NodeType::NODE_MYSQL_INSERT_ALIAS:
198+
case NodeType::NODE_MYSQL_JSON_AGGREGATE:
199+
case NodeType::NODE_MYSQL_JSON_AGG_ARGUMENT:
200+
case NodeType::NODE_MYSQL_EXPLAIN_INTO:
198201
case NodeType::NODE_MYSQL_EXTRACT:
199202
case NodeType::NODE_MYSQL_SUBSTRING:
200203
case NodeType::NODE_MYSQL_MATCH:
@@ -212,7 +215,7 @@ class PlanBuilder {
212215
case NodeType::NODE_FUNCTION_CALL:
213216
if (node->flags & (sql_parser::FLAG_FUNCTION_TABLE | sql_parser::FLAG_FUNCTION_DISTINCT |
214217
sql_parser::FLAG_FUNCTION_ALL | sql_parser::FLAG_FUNCTION_QUALIFIED |
215-
sql_parser::FLAG_FUNCTION_WITHIN_GROUP)) return true;
218+
sql_parser::FLAG_FUNCTION_WITHIN_GROUP | sql_parser::FLAG_FUNCTION_MYSQL_SPACE)) return true;
216219
break;
217220
case NodeType::NODE_ORDER_BY_ITEM:
218221
if (node->flags & sql_parser::FLAG_ORDER_NULLS) return true;

‎include/sql_parser/common.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,7 @@ static constexpr uint16_t FLAG_FUNCTION_DISTINCT = 0x02;
6868
static constexpr uint16_t FLAG_FUNCTION_ALL = 0x04;
6969
static constexpr uint16_t FLAG_FUNCTION_QUALIFIED = 0x08;
7070
static constexpr uint16_t FLAG_FUNCTION_WITHIN_GROUP = 0x10;
71+
static constexpr uint16_t FLAG_FUNCTION_MYSQL_SPACE = 0x20; // keep a native function name in ordinary-call syntax
7172
// On binary/unary expressions: PostgreSQL operation without local engine support.
7273
static constexpr uint16_t FLAG_PG_OPERATOR = 0x01;
7374
static constexpr uint16_t FLAG_WINDOW_BETWEEN = 0x01;
@@ -362,6 +363,9 @@ enum class NodeType : uint16_t {
362363
NODE_MYSQL_COLLATE, // value=collation source spelling; one expression child
363364
NODE_MYSQL_CONVERT, // expression, type/charset metadata; flag 1=USING form
364365
NODE_MYSQL_INSERT_ALIAS, // row identifier, optional nonempty column identifier list
366+
NODE_MYSQL_JSON_AGGREGATE, // value=function; argument children; flags 1=NULL ON NULL, 2=ABSENT ON NULL
367+
NODE_MYSQL_JSON_AGG_ARGUMENT, // explicit ALL argument; expression child
368+
NODE_MYSQL_EXPLAIN_INTO, // user-variable destination child
365369

366370
};
367371

‎include/sql_parser/emitter.h‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -194,6 +194,15 @@ class Emitter {
194194
break;
195195
case NodeType::NODE_UNARY_OP: emit_unary_op(node); break;
196196
case NodeType::NODE_FUNCTION_CALL: emit_function_call(node); break;
197+
case NodeType::NODE_MYSQL_JSON_AGGREGATE:
198+
emit_value(node); sb_.append_char('('); emit_list(node, ", ");
199+
if (node->flags & 1) sb_.append(" NULL ON NULL");
200+
else if (node->flags & 2) sb_.append(" ABSENT ON NULL");
201+
sb_.append_char(')'); break;
202+
case NodeType::NODE_MYSQL_JSON_AGG_ARGUMENT:
203+
sb_.append("ALL "); emit_node(node->first_child); break;
204+
case NodeType::NODE_MYSQL_EXPLAIN_INTO:
205+
sb_.append("INTO "); emit_node(node->first_child); break;
197206
case NodeType::NODE_MYSQL_CHARSET_LITERAL:
198207
emit_value(node);
199208
for (const auto* literal = node->first_child; literal; literal = literal->next_sibling) {
@@ -1409,7 +1418,8 @@ class Emitter {
14091418
c->type == NodeType::NODE_INSERT_STMT ||
14101419
c->type == NodeType::NODE_UPDATE_STMT ||
14111420
c->type == NodeType::NODE_DELETE_STMT ||
1412-
c->type == NodeType::NODE_COMPOUND_QUERY) {
1421+
c->type == NodeType::NODE_COMPOUND_QUERY ||
1422+
c->type == NodeType::NODE_CTE || c->type == NodeType::NODE_TABLE_QUERY) {
14131423
has_inner_stmt = true;
14141424
}
14151425
if (c->type == NodeType::NODE_EXPLAIN_OPTIONS) {
@@ -1708,6 +1718,9 @@ class Emitter {
17081718
emit_node(arg);
17091719
arg = arg ? arg->next_sibling : nullptr;
17101720
} else emit_value(node);
1721+
if constexpr (D == Dialect::MySQL) {
1722+
if (node->flags & FLAG_FUNCTION_MYSQL_SPACE) sb_.append_char(' ');
1723+
}
17111724
sb_.append_char('(');
17121725
if (node->flags & FLAG_FUNCTION_DISTINCT) sb_.append("DISTINCT ");
17131726
else if (node->flags & FLAG_FUNCTION_ALL) sb_.append("ALL ");

0 commit comments

Comments
 (0)