feat: preserve MySQL table modifiers and lateral derived queries

Parse PARTITION selections and USE, FORCE, and IGNORE INDEX/KEY hints into
structured table-reference children, retaining hint scope and empty USE
lists. Preserve the distinct clause ordering required by SELECT/UPDATE
and single-table DELETE, and reject malformed lists and misplaced clauses.

Support SELECT-based LATERAL derived tables, UNION sources, mandatory
MySQL derived-table aliases, and optional alias-column lists. Require
complete nested query operands. Share the locking-clause parser so nested
queries retain NOWAIT, SKIP LOCKED, and structured OF targets, including
qualified and wildcard targets. Require BY after GROUP and LOCKED after
SKIP instead of silently repairing malformed input.

Validate table identifiers and aliases against the common reserved-word
baseline verified on pinned MySQL 8.4.8 and 9.7.2. Preserve delimited names
and the special PRIMARY index name. Keep version-specific reserved words
outside this unversioned parser policy. Append AST kinds without changing
existing values or the node layout, and reject unsupported table modifiers
and derived column renaming in local planning.

Expand the pinned compatibility corpus from 43 to 74 hand-reduced
witnesses, verifying 71 exact grammar anchors per release. Promote the
three previous table-reference gaps and require independently derived
emission expectations. Give native checks a separate populated partitioned
fixture, preserving the existing fulltext fixture for unrelated cases.
Document SELECT-only LATERAL scope and the remaining identifier limitations;
these selected witnesses do not measure complete dialect coverage.

Validation:
- Forced full build, corpus-test build, and make test: 1,539 passed,
  37 external-database tests skipped.
- Nine new focused regressions plus 14 existing MySQL grammar tests pass
  under ASan/UBSan with leak detection.
- All 74 compatibility expectations match: 33 CHECKED, 21 malformed
  controls REJECTED, and 20 explicitly tracked gaps; 11 Python tests pass.
- Native MySQL 8.4.8 and 9.7.2 runs have zero validity mismatches. Server
  errors and three skipped DDL cases remain separately unresolved.
- PostgreSQL harness: 87 tests pass; 5,122 committed fixtures and the full
  51,415-statement pinned baseline match unchanged.
- Independent review verifies nested locking and reserved-name fixes;
  seven parser/emitter benchmark smoke cases run successfully.

Benchmark sources, reports, and benchmark-only README/.gitignore changes
are excluded. Production parsing retains no native MySQL dependency.

Author: renecannao

Makefile

@@ -66,6 +66,7 @@ TEST_SRCS = $(TEST_DIR)/test_main.cpp \
             $(TEST_DIR)/test_pg_maintenance.cpp \
             $(TEST_DIR)/test_review_expressions.cpp \
             $(TEST_DIR)/test_mysql_grammar.cpp \
+            $(TEST_DIR)/test_mysql_tables.cpp \
             $(TEST_DIR)/test_review_ast.cpp \
             $(TEST_DIR)/test_review_transactions.cpp \
             $(TEST_DIR)/test_review_runtime.cpp \

docs/mysql-compatibility.md

@@ -4,6 +4,12 @@ This harness tracks a small, deliberately selected set of MySQL syntax witnesses
 
 ParserSQL now preserves MySQL JSON arrows, named/inherited windows, `GROUP_CONCAT` modifiers and separators, and `GROUP BY ... WITH ROLLUP` as structured ASTs. Database-qualified column names are retained. The local engine rejects these unsupported semantics rather than executing a simplified interpretation; parameterization remains conservative for JSON arrows and `GROUP_CONCAT`. Empty MySQL SELECT target lists are rejected.
 
+ParserSQL also preserves partition lists, `USE`/`FORCE`/`IGNORE INDEX` or `KEY`, hint scopes and empty `USE INDEX ()`, and `LATERAL` derived SELECTs with required aliases and optional alias-column lists. The local engine rejects these unsupported table-reference semantics. `SELECT` and `UPDATE` place `PARTITION` before the table alias; single-table `DELETE` places it after the alias. The current LATERAL implementation scope is SELECT-based queries, including UNION: both native releases also accept `LATERAL (TABLE t) AS q` and `LATERAL (VALUES ROW(1)) AS q`, but those forms remain outside this implementation and are not malformed controls.
+
+Table identifiers and aliases use the common 259-word reserved baseline observed in the pinned 8.4.8 and 9.7.2 releases. ParserSQL has no MySQL version setting, so the five 9.7-only reserved additions (`CUBE`, `EXTERNAL`, `LIBRARY`, `QUALIFY`, and `TABLESAMPLE`) remain accepted as names. Strict version-specific identifier rules remain a gap.
+
+Numeric-looking unquoted qualified table names such as `db.123` are valid natively but remain unsupported: the tokenizer reads `.123` as a numeric literal. ParserSQL rejects these forms; earlier behavior could emit an incorrect alias. Lossless parsing of this identifier edge remains a gap.
+
 The corpus pins the official `mysql/mysql-server` sources:
 
 | Version | Commit |
@@ -36,6 +42,8 @@ Malformed controls are handled separately: a complete AST is `MALFORMED_ACCEPTED
 
 Every fixture freezes one exact expected outcome. Known gaps do not make arbitrary failures acceptable. Both regression and unexpected improvement cause exit status 1, so a fixed gap must be reviewed and explicitly promoted in the corpus. When promoting, add an independently derived `expected_sql`, set `known_gap=false`, and require `CHECKED`. Harness/transport/source errors return 2. Exit status 0 means the expected baseline matched, including its documented known gaps.
 
+The 74-case baseline has 33 `CHECKED` witnesses and 21 `REJECTED` negative controls. Its remaining known gaps are 13 `TRAILING_INPUT`, five `NO_AST`, one `PARTIAL`, and one `EMISSION_MISMATCH`. Source verification checks 71 exact grammar anchors per pinned release. These counts describe this corpus only.
+
 The JSON report includes all original and reparsed records, source verification counts, pinned revisions, SQL-mode metadata, and SHA-256 hashes of the corpus and probe. Counts apply only to these witnesses. The probe uses hex-line input and hex fields in its TSV response, preserving tabs, newlines and embedded NUL bytes. It emits while both the input string and parser arena are alive; Python reparses owned copies in a second probe invocation.
 
 The default corpus oracle state is **NOT VERIFIED**. Its intended `sql_mode` is the empty string; ParserSQL accepts no mode configuration here and the harness does not apply or emulate MySQL session modes. A native server check requires its exact version, actual session mode, schema fixtures, SQL sent, and result/error code. Missing tables, mode-dependent errors, or unsupported runtime features must not be relabeled as syntax rejection. Without native options the harness does not connect to a database. An optional native run uses an explicit absolute socket path to an already-running, isolated server:
@@ -47,15 +55,15 @@ python3 scripts/mysql_compat/run.py --probe /tmp/parsersql-mysql-probe \
   --report /tmp/parsersql-mysql-native.json
 ```
 
-The client uses `--no-defaults`, the supplied socket, and `root` unless `--native-user` is specified. There is no host discovery or automatic connection. Use an isolated test server: this option explicitly creates a uniquely named `parsersql_compat_<uuid>` database, installs the small fixture in [native.py](../scripts/mysql_compat/native.py), and drops only that owned database in cleanup. The supplied user needs permission to create/drop that database and its tables. The helper requires server version 8.4.8 or 9.7.2, records the exact reported version and session SQL mode, and reapplies that mode for each query. It does not assert that the native mode equals the corpus's empty intended mode.
+The client uses `--no-defaults`, the supplied socket, and `root` unless `--native-user` is specified. There is no host discovery or automatic connection. Use an isolated test server: this option explicitly creates a uniquely named `parsersql_compat_<uuid>` database, installs the small fixture in [native.py](../scripts/mysql_compat/native.py), and drops only that owned database in cleanup. The partition witnesses use a separate `pt` table with populated `p0` and `p1` range partitions; `t` retains its fulltext index for the existing search witness. The supplied user needs permission to create/drop that database and its tables. The helper requires server version 8.4.8 or 9.7.2, records the exact reported version and session SQL mode, and reapplies that mode for each query. It does not assert that the native mode equals the corpus's empty intended mode.
 
 Each curated case explicitly chooses direct query execution, `EXPLAIN` for DML (including CTE-prefixed DML), or no native action. DDL and procedures remain **NOT VERIFIED**. These are query-execution or EXPLAIN observations, not access to MySQL's raw parser. Returned row values are evidence only; they are not compared with ParserSQL execution. The helper neither executes emitted SQL nor claims semantic round-trip validation.
 
-Native results are stored separately in `native_oracle`: `ACCEPTED`, `SYNTAX_REJECTED` (1064/1149), `SERVER_ERROR`, `CLIENT_ERROR`, or **NOT VERIFIED**. For example, the PARTITION witness hits error 1747 because the fixture table is not partitioned; that is an unresolved server error, never an accepted syntax result. Accepted invalid controls and syntax-rejected valid controls count as expectation mismatches, using per-version validity when specified. Server errors remain explicitly unresolved; client errors return status 2. Thus exit status 0 still does not mean all native cases were verified. The default corpus metadata remains **NOT VERIFIED**, while the optional report describes the actual native observations.
+Native results are stored separately in `native_oracle`: `ACCEPTED`, `SYNTAX_REJECTED` (1064/1149), `SERVER_ERROR`, `CLIENT_ERROR`, or **NOT VERIFIED**. For example, a derived table missing its required alias returns error 1248; this remains a server error, even though the corpus requires ParserSQL to reject that input. Accepted invalid controls and syntax-rejected valid controls count as expectation mismatches, using per-version validity when specified. Server errors remain explicitly unresolved; client errors return status 2. Thus exit status 0 still does not mean all native cases were verified. The default corpus metadata remains **NOT VERIFIED**, while the optional report describes the actual native observations.
 
-A native check of the shared witnesses against both pinned releases found acceptance of the positive query/DML witnesses and syntax rejection of the negative controls under `ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION`. The three DDL/procedure witnesses were skipped and PARTITION remained unresolved. Re-run the command on each isolated socket to reproduce current observations; reports are intentionally not committed.
+A native check of all 74 witnesses under `ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION` found no native expectation mismatches. MySQL 8.4.8 accepted 46, syntax-rejected 24, and returned error 1248 for the missing-alias control; MySQL 9.7.2 accepted 48, syntax-rejected 20, and returned three server errors (1248, 1235, and 3889). Both runs skipped the three DDL/procedure witnesses and cleaned up their owned schemas. All shared positive query/DML witnesses, including partition selection and index hints, were accepted. Re-run the command on each isolated socket to reproduce current observations; reports are intentionally not committed.
 
-The corpus is deliberately selected and is not a representative workload. It contains 39 shared witnesses and four version-specific query witnesses; this does not imply identical dialects in 8.4.8 and 9.7.2. The version-specific entries record `applicability`, source productions for both versions, `native_validity` per version, and observed native outcomes. Their generic `validity=valid` denotes the 9.7.2 grammar target. The native helper selects the intended validity using the actual server version, so an expected 8.4.8 syntax rejection is not treated as a corpus error. Stored native observations are context, not substitutes for a current run. These additions remain ParserSQL gaps. The following differences include those four query witnesses and two separate DDL spot checks:
+The corpus is deliberately selected and is not a representative workload. It contains 70 shared witnesses and four version-specific query witnesses; this does not imply identical dialects in 8.4.8 and 9.7.2. The version-specific entries record `applicability`, source productions for both versions, `native_validity` per version, and observed native outcomes. Their generic `validity=valid` denotes the 9.7.2 grammar target. The native helper selects the intended validity using the actual server version, so an expected 8.4.8 syntax rejection is not treated as a corpus error. Stored native observations are context, not substitutes for a current run. These additions remain ParserSQL gaps. The following differences include those four query witnesses and two separate DDL spot checks:
 
 | Reduced feature | MySQL 8.4.8 observation | MySQL 9.7.2 observation |
 | --- | --- | --- |

include/sql_engine/plan_builder.h

@@ -70,6 +70,15 @@ class PlanBuilder {
                                                bool dml_values = false) {
         using sql_parser::NodeType;
         switch (node->type) {
+            case NodeType::NODE_ALIAS:
+                if constexpr (D == sql_parser::Dialect::MySQL) {
+                    if (node->first_child) return true; // Derived column renaming needs engine support.
+                }
+                break;
+            case NodeType::NODE_MYSQL_LOCK_TARGETS:
+            case NodeType::NODE_MYSQL_PARTITION_SELECTION:
+            case NodeType::NODE_MYSQL_INDEX_HINT:
+                return true;
             case NodeType::NODE_QUALIFIED_NAME:
                 // The local resolver combines only table.column; it would
                 // otherwise silently drop the final part of db.table.column.

include/sql_parser/common.h

@@ -347,6 +347,9 @@ enum class NodeType : uint16_t {
     NODE_MYSQL_JSON_EXTRACT, // value=-> or ->>; column reference, literal path
     NODE_MYSQL_GROUP_CONCAT, // arguments, optional aggregate ORDER BY and separator
     NODE_MYSQL_SEPARATOR, // literal string child
+    NODE_MYSQL_PARTITION_SELECTION, // nonempty identifier list on a table reference
+    NODE_MYSQL_LOCK_TARGETS, // OF table identifiers on a locking clause
+    NODE_MYSQL_INDEX_HINT, // value=USE/FORCE/IGNORE; flags KEY=1, JOIN=2, ORDER=4, GROUP=8; index children
 
 };
 

include/sql_parser/compound_query_parser.h

@@ -53,11 +53,9 @@ class CompoundQueryParser {
         }
         if (D == Dialect::MySQL && result->type == NodeType::NODE_SELECT_STMT &&
             tok_.peek().type == TokenType::TK_FOR) {
-            tok_.skip();
-            AstNode* lock = make_node(arena_, NodeType::NODE_LOCKING_CLAUSE);
+            SelectParser<D> select(tok_, arena_, false, require_operands);
+            AstNode* lock = select.parse_locking();
             if (!lock) return nullptr;
-            Token strength = tok_.next_token();
-            lock->add_child(make_node(arena_, NodeType::NODE_IDENTIFIER, strength.text));
             result->add_child(lock);
         }
         return result;

include/sql_parser/delete_parser.h

@@ -75,8 +75,18 @@ class DeleteParser {
             AstNode* first_table = parse_simple_table_ref();
             if (!first_table) return root;
 
+            table_ref_parser_.parse_optional_alias(first_table);
+            const bool has_partition = tok_.peek().type == TokenType::TK_PARTITION;
+            if (has_partition) {
+                auto* partition = table_ref_parser_.parse_mysql_partition_selection();
+                if (!partition) return expr_parser_.syntax_error();
+                first_table->add_child(partition);
+            }
+
             // Check if comma follows (target list) or if USING follows
             if (tok_.peek().type == TokenType::TK_COMMA || tok_.peek().type == TokenType::TK_USING) {
+                if (has_partition || (first_table->first_child &&
+                    first_table->first_child->next_sibling)) return expr_parser_.syntax_error();
                 // Could be multi-table form 2: DELETE FROM t1[, t2] USING ...
                 // Or single-table with comma would be unusual. Check for USING after table list.
                 // Collect all target tables

include/sql_parser/emitter.h

@@ -143,6 +143,17 @@ class Emitter {
 
             // ---- Table references ----
             case NodeType::NODE_TABLE_REF:       emit_table_ref(node); break;
+            case NodeType::NODE_MYSQL_LOCK_TARGETS:
+                sb_.append("OF "); emit_list(node, ", "); break;
+            case NodeType::NODE_MYSQL_PARTITION_SELECTION:
+                sb_.append(" PARTITION ("); emit_list(node, ", "); sb_.append_char(')'); break;
+            case NodeType::NODE_MYSQL_INDEX_HINT:
+                sb_.append_char(' '); emit_value(node);
+                sb_.append((node->flags & 1) ? " KEY" : " INDEX");
+                if (node->flags & 2) sb_.append(" FOR JOIN");
+                else if (node->flags & 4) sb_.append(" FOR ORDER BY");
+                else if (node->flags & 8) sb_.append(" FOR GROUP BY");
+                sb_.append(" ("); emit_list(node, ", "); sb_.append_char(')'); break;
             case NodeType::NODE_ALIAS:           emit_alias(node); break;
             case NodeType::NODE_QUALIFIED_NAME:  emit_qualified_name(node); break;
 

include/sql_parser/mysql_identifier.h

@@ -0,0 +1,97 @@
+#ifndef SQL_PARSER_MYSQL_IDENTIFIER_H
+#define SQL_PARSER_MYSQL_IDENTIFIER_H
+
+#include "sql_parser/token.h"
+#include <cstddef>
+
+namespace sql_parser {
+
+// Keywords may be identifiers after a qualification dot, including reserved ones.
+inline bool mysql_identifier_word(const Token& token) {
+    return token.type == TokenType::TK_IDENTIFIER ||
+        (token.type >= TokenType::TK_SELECT && token.type <= TokenType::TK_RECURSIVE);
+}
+
+inline bool mysql_identifier_token(const Token& token) {
+    if (!mysql_identifier_word(token)) return false;
+    if (token.source.ptr != token.text.ptr) return true; // delimited identifier
+    // Sorted reserved-word baseline from INFORMATION_SCHEMA.KEYWORDS, MySQL
+    // 8.4.8 (0896fcd61dec11a0904166911a0126f59daaa1bf), also reserved in
+    // 9.7.2 (008e09c2834b98143a8c067d4d225c90953050cf). Version-specific
+    // additions are not rejected without a server-version parser setting.
+    static constexpr const char* reserved[] = {
+        "ACCESSIBLE", "ADD", "ALL", "ALTER", "ANALYZE",
+        "AND", "AS", "ASC", "ASENSITIVE", "BEFORE",
+        "BETWEEN", "BIGINT", "BINARY", "BLOB", "BOTH",
+        "BY", "CALL", "CASCADE", "CASE", "CHANGE",
+        "CHAR", "CHARACTER", "CHECK", "COLLATE", "COLUMN",
+        "CONDITION", "CONSTRAINT", "CONTINUE", "CONVERT", "CREATE",
+        "CROSS", "CUME_DIST", "CURRENT_DATE", "CURRENT_TIME", "CURRENT_TIMESTAMP",
+        "CURRENT_USER", "CURSOR", "DATABASE", "DATABASES", "DAY_HOUR",
+        "DAY_MICROSECOND", "DAY_MINUTE", "DAY_SECOND", "DEC", "DECIMAL",
+        "DECLARE", "DEFAULT", "DELAYED", "DELETE", "DENSE_RANK",
+        "DESC", "DESCRIBE", "DETERMINISTIC", "DISTINCT", "DISTINCTROW",
+        "DIV", "DOUBLE", "DROP", "DUAL", "EACH",
+        "ELSE", "ELSEIF", "EMPTY", "ENCLOSED", "ESCAPED",
+        "EXCEPT", "EXISTS", "EXIT", "EXPLAIN", "FALSE",
+        "FETCH", "FIRST_VALUE", "FLOAT", "FLOAT4", "FLOAT8",
+        "FOR", "FORCE", "FOREIGN", "FROM", "FULLTEXT",
+        "FUNCTION", "GENERATED", "GET", "GRANT", "GROUP",
+        "GROUPING", "GROUPS", "HAVING", "HIGH_PRIORITY", "HOUR_MICROSECOND",
+        "HOUR_MINUTE", "HOUR_SECOND", "IF", "IGNORE", "IN",
+        "INDEX", "INFILE", "INNER", "INOUT", "INSENSITIVE",
+        "INSERT", "INT", "INT1", "INT2", "INT3",
+        "INT4", "INT8", "INTEGER", "INTERSECT", "INTERVAL",
+        "INTO", "IO_AFTER_GTIDS", "IO_BEFORE_GTIDS", "IS", "ITERATE",
+        "JOIN", "JSON_TABLE", "KEY", "KEYS", "KILL",
+        "LAG", "LAST_VALUE", "LATERAL", "LEAD", "LEADING",
+        "LEAVE", "LEFT", "LIKE", "LIMIT", "LINEAR",
+        "LINES", "LOAD", "LOCALTIME", "LOCALTIMESTAMP", "LOCK",
+        "LONG", "LONGBLOB", "LONGTEXT", "LOOP", "LOW_PRIORITY",
+        "MATCH", "MAXVALUE", "MEDIUMBLOB", "MEDIUMINT", "MEDIUMTEXT",
+        "MIDDLEINT", "MINUTE_MICROSECOND", "MINUTE_SECOND", "MOD", "MODIFIES",
+        "NATURAL", "NOT", "NO_WRITE_TO_BINLOG", "NTH_VALUE", "NTILE",
+        "NULL", "NUMERIC", "OF", "ON", "OPTIMIZE",
+        "OPTIMIZER_COSTS", "OPTION", "OPTIONALLY", "OR", "ORDER",
+        "OUT", "OUTER", "OUTFILE", "OVER", "PARTITION",
+        "PERCENT_RANK", "PRECISION", "PRIMARY", "PROCEDURE", "PURGE",
+        "RANGE", "RANK", "READ", "READS", "READ_WRITE",
+        "REAL", "RECURSIVE", "REFERENCES", "REGEXP", "RELEASE",
+        "RENAME", "REPEAT", "REPLACE", "REQUIRE", "RESIGNAL",
+        "RESTRICT", "RETURN", "REVOKE", "RIGHT", "RLIKE",
+        "ROW", "ROWS", "ROW_NUMBER", "SCHEMA", "SCHEMAS",
+        "SECOND_MICROSECOND", "SELECT", "SENSITIVE", "SEPARATOR", "SET",
+        "SHOW", "SIGNAL", "SMALLINT", "SPATIAL", "SPECIFIC",
+        "SQL", "SQLEXCEPTION", "SQLSTATE", "SQLWARNING", "SQL_BIG_RESULT",
+        "SQL_CALC_FOUND_ROWS", "SQL_SMALL_RESULT", "SSL", "STARTING", "STORED",
+        "STRAIGHT_JOIN", "SYSTEM", "TABLE", "TERMINATED", "THEN",
+        "TINYBLOB", "TINYINT", "TINYTEXT", "TO", "TRAILING",
+        "TRIGGER", "TRUE", "UNDO", "UNION", "UNIQUE",
+        "UNLOCK", "UNSIGNED", "UPDATE", "USAGE", "USE",
+        "USING", "UTC_DATE", "UTC_TIME", "UTC_TIMESTAMP", "VALUES",
+        "VARBINARY", "VARCHAR", "VARCHARACTER", "VARYING", "VIRTUAL",
+        "WHEN", "WHERE", "WHILE", "WINDOW", "WITH",
+        "WRITE", "XOR", "YEAR_MONTH", "ZEROFILL",
+    };
+    size_t lo = 0, hi = sizeof(reserved) / sizeof(reserved[0]);
+    while (lo < hi) {
+        size_t mid = lo + (hi - lo) / 2;
+        const char* word = reserved[mid];
+        uint32_t i = 0;
+        int difference = 0;
+        for (; i < token.text.len && word[i]; ++i) {
+            unsigned char ch = static_cast<unsigned char>(token.text.ptr[i]);
+            if (ch >= 'a' && ch <= 'z') ch -= 'a' - 'A';
+            difference = static_cast<int>(ch) - static_cast<unsigned char>(word[i]);
+            if (difference) break;
+        }
+        if (!difference) difference = i < token.text.len ? 1 : word[i] ? -1 : 0;
+        if (!difference) return false;
+        if (difference < 0) hi = mid;
+        else lo = mid + 1;
+    }
+    return true;
+}
+
+} // namespace sql_parser
+#endif // SQL_PARSER_MYSQL_IDENTIFIER_H