Skip to content

Commit 1a5c49e

Browse files
committed
test: pin MySQL 8.4 and 9.7 compatibility witnesses
Add a strict compatibility probe and 43 curated witnesses derived from pinned official MySQL 8.4.8 and 9.7.2 grammar sources. Each case retains source anchors and explicit expected outcomes, including known gaps. Require OK status, complete input and an AST; then reparse emitted SQL, check emission stability, and compare independently specified SQL where available. Malformed controls and hint-preservation failures are tracked separately, so classification-only or partial parsing cannot masquerade as grammar support. Unexpected improvements also require baseline review. Add an optional native check using an explicitly selected isolated Unix socket. Verify the server version, record and reapply its actual sql_mode, create a uniquely owned fixture schema, EXPLAIN DML, execute configured query witnesses and clean up the owned schema. Keep syntax errors, client errors, unresolved server/feature errors and skipped DDL distinct. Version-specific validity records distinguish four new 9.7 productions from 8.4 syntax rejections; raw-parser and semantic-equivalence claims are explicitly out of scope. Wire the probe and Python regression tests into make test-mysql-compat and a dedicated CI job. Document pins, commands, source verification, remaining gaps, optional native side effects and the limits of the small hand-reduced corpus. Generated reports and executables stay outside Git. Validation: 11 Python tests passed; all 43 expected ParserSQL outcomes matched; 40 source anchors per version verified. Optional native runs matched intended validity on both releases, retaining partition and 9.7 feature restrictions as unresolved and three DDL cases as unverified. All temporary schemas and both disposable native servers were cleaned up. Benchmark scripts, reports and benchmark-only documentation are excluded.
1 parent d0e54fd commit 1a5c49e

10 files changed

Lines changed: 1733 additions & 0 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,13 @@ jobs:
123123
- name: Test PostgreSQL compatibility gate
124124
run: make -f Makefile test-pg-compat
125125

126+
mysql-compat:
127+
runs-on: ubuntu-latest
128+
steps:
129+
- uses: actions/checkout@v4
130+
- name: Check curated MySQL compatibility baseline
131+
run: make -f Makefile test-mysql-compat
132+
126133
pg-compat-full:
127134
if: github.event_name == 'schedule'
128135
runs-on: ubuntu-24.04

‎Makefile‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ MYSQL_LIBS = $(shell mysql_config --libs 2>/dev/null)
88
PG_CFLAGS = -I$(shell pg_config --includedir 2>/dev/null || echo /usr/include/postgresql)
99
PG_LIBS = -L$(shell pg_config --libdir 2>/dev/null || echo /usr/lib/x86_64-linux-gnu) -lpq
1010
PG_COMPAT_CACHE ?= /tmp/parsersql-pg-compat
11+
MYSQL_COMPAT_CACHE ?= /tmp/parsersql-mysql-compat
1112

1213
PROJECT_ROOT = .
1314
SRC_DIR = $(PROJECT_ROOT)/src/sql_parser
@@ -168,6 +169,12 @@ pg-compat: build-pg-compat
168169
pg-compat-refresh: build-pg-compat
169170
python3 ./scripts/pg_compat/run_compat.py refresh --cache $(PG_COMPAT_CACHE)
170171

172+
.PHONY: test-mysql-compat
173+
test-mysql-compat: lib
174+
python3 scripts/mysql_compat/build_probe.py --output "$(MYSQL_COMPAT_CACHE)/probe"
175+
python3 -m unittest discover -s tests/mysql_compat -p 'test_*.py' -v
176+
python3 scripts/mysql_compat/run.py --probe "$(MYSQL_COMPAT_CACHE)/probe" --report "$(MYSQL_COMPAT_CACHE)/report.json"
177+
171178
all: lib test
172179

173180
lib: $(LIB_TARGET)

‎docs/mysql-compatibility.md‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# Curated MySQL compatibility checks
2+
3+
This harness tracks a small, deliberately selected set of MySQL syntax witnesses and known ParserSQL gaps. It does not measure the percentage of the MySQL dialect supported, implement mysqltest/MTR extraction, or establish query-engine execution support.
4+
5+
ParserSQL now preserves MySQL JSON arrows, named/inherited windows, `GROUP_CONCAT` modifiers and separators, and `GROUP BY ... WITH ROLLUP` as structured ASTs. Database-qualified column names are retained. The local engine rejects these unsupported semantics rather than executing a simplified interpretation; parameterization remains conservative for JSON arrows and `GROUP_CONCAT`. Empty MySQL SELECT target lists are rejected.
6+
7+
The corpus pins the official `mysql/mysql-server` sources:
8+
9+
| Version | Commit |
10+
| --- | --- |
11+
| MySQL 8.4.8 | `0896fcd61dec11a0904166911a0126f59daaa1bf` |
12+
| MySQL 9.7.2 | `008e09c2834b98143a8c067d4d225c90953050cf` |
13+
14+
[The corpus](../tests/mysql_compat/cases.json) contains hand-reduced SQL, intended validity, explicit expected ParserSQL outcomes, and source grammar paths, lines and excerpts for both versions. A grammar anchor identifies related upstream syntax; it does not claim that the reduced query is verbatim upstream test SQL or that source inspection is a native parser oracle. Hand-written controls are identified separately. The runner can verify the source commits and excerpts locally.
15+
16+
Run from the repository root:
17+
18+
```sh
19+
make -B lib
20+
# The CI gate is also available as: make test-mysql-compat
21+
python3 scripts/mysql_compat/build_probe.py --output /tmp/parsersql-mysql-probe
22+
python3 -m unittest discover -s tests/mysql_compat -v
23+
python3 scripts/mysql_compat/run.py --probe /tmp/parsersql-mysql-probe \
24+
--report /tmp/parsersql-mysql-report.json
25+
```
26+
27+
`build_probe.py` compiles a standalone C++17 executable against the existing `libsqlparser.a`; it does not rebuild that archive. Rebuild the library with `make -B lib` before checking parser changes: the Makefile does not track all header dependencies, so a plain incremental build can leave a stale archive. `CXX` selects the compiler. Keep binaries and generated reports outside the repository.
28+
29+
To also verify the source provenance, add `--source84 /path/to/mysql-8.4.8` and `--source97 /path/to/mysql-9.7.2`. Each checkout must have exactly the pinned HEAD and contain `sql/sql_yacc.yy`; no native MySQL build is required for this check. These are the official source links for the [8.4.8 grammar](https://github.com/mysql/mysql-server/blob/0896fcd61dec11a0904166911a0126f59daaa1bf/sql/sql_yacc.yy) and [9.7.2 grammar](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy).
30+
31+
The strict parse gate requires status `OK`, `full_input=true`, and a non-null AST. Failure categories are `ERROR`, `PARTIAL`, `NO_AST`, and `TRAILING_INPUT`. `NO_AST` takes precedence over trailing input; the report retains the raw flags and remaining SQL so neither fact is lost. In particular, classification-only DDL must not count as structured parsing.
32+
33+
Complete parses have their emitted SQL reparsed through the same strict gate. The emitted text must stabilize after that reparse. Positive controls also specify exact expected SQL, including JSON `->` and `->>` operators. These checks can expose dropped or changed syntax even when the altered SQL still parses. They are bounded emission checks, not a proof of AST or execution equivalence. `CHECKED` means only that these checks passed. The optimizer-hint witness records a known `EMISSION_MISMATCH`, because dropping a hint is not faithful preservation.
34+
35+
Malformed controls are handled separately: a complete AST is `MALFORMED_ACCEPTED`, even when emission changes. The `SELECT FROM` and malformed JSON-arrow controls require rejection. A rejection is `REJECTED`; the original record still identifies whether it was an error, incomplete input, or a missing AST.
36+
37+
Every fixture freezes one exact expected outcome. Known gaps do not make arbitrary failures acceptable. Both regression and unexpected improvement cause exit status 1, so a fixed gap must be reviewed and explicitly promoted in the corpus. When promoting, add an independently derived `expected_sql`, set `known_gap=false`, and require `CHECKED`. Harness/transport/source errors return 2. Exit status 0 means the expected baseline matched, including its documented known gaps.
38+
39+
The JSON report includes all original and reparsed records, source verification counts, pinned revisions, SQL-mode metadata, and SHA-256 hashes of the corpus and probe. Counts apply only to these witnesses. The probe uses hex-line input and hex fields in its TSV response, preserving tabs, newlines and embedded NUL bytes. It emits while both the input string and parser arena are alive; Python reparses owned copies in a second probe invocation.
40+
41+
The default corpus oracle state is **NOT VERIFIED**. Its intended `sql_mode` is the empty string; ParserSQL accepts no mode configuration here and the harness does not apply or emulate MySQL session modes. A native server check requires its exact version, actual session mode, schema fixtures, SQL sent, and result/error code. Missing tables, mode-dependent errors, or unsupported runtime features must not be relabeled as syntax rejection. Without native options the harness does not connect to a database. An optional native run uses an explicit absolute socket path to an already-running, isolated server:
42+
43+
```sh
44+
python3 scripts/mysql_compat/run.py --probe /tmp/parsersql-mysql-probe \
45+
--native-socket /absolute/path/to/isolated-mysql.sock \
46+
--mysql-client /absolute/path/to/mysql \
47+
--report /tmp/parsersql-mysql-native.json
48+
```
49+
50+
The client uses `--no-defaults`, the supplied socket, and `root` unless `--native-user` is specified. There is no host discovery or automatic connection. Use an isolated test server: this option explicitly creates a uniquely named `parsersql_compat_<uuid>` database, installs the small fixture in [native.py](../scripts/mysql_compat/native.py), and drops only that owned database in cleanup. The supplied user needs permission to create/drop that database and its tables. The helper requires server version 8.4.8 or 9.7.2, records the exact reported version and session SQL mode, and reapplies that mode for each query. It does not assert that the native mode equals the corpus's empty intended mode.
51+
52+
Each curated case explicitly chooses direct query execution, `EXPLAIN` for DML (including CTE-prefixed DML), or no native action. DDL and procedures remain **NOT VERIFIED**. These are query-execution or EXPLAIN observations, not access to MySQL's raw parser. Returned row values are evidence only; they are not compared with ParserSQL execution. The helper neither executes emitted SQL nor claims semantic round-trip validation.
53+
54+
Native results are stored separately in `native_oracle`: `ACCEPTED`, `SYNTAX_REJECTED` (1064/1149), `SERVER_ERROR`, `CLIENT_ERROR`, or **NOT VERIFIED**. For example, the PARTITION witness hits error 1747 because the fixture table is not partitioned; that is an unresolved server error, never an accepted syntax result. Accepted invalid controls and syntax-rejected valid controls count as expectation mismatches, using per-version validity when specified. Server errors remain explicitly unresolved; client errors return status 2. Thus exit status 0 still does not mean all native cases were verified. The default corpus metadata remains **NOT VERIFIED**, while the optional report describes the actual native observations.
55+
56+
A native check of the shared witnesses against both pinned releases found acceptance of the positive query/DML witnesses and syntax rejection of the negative controls under `ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION`. The three DDL/procedure witnesses were skipped and PARTITION remained unresolved. Re-run the command on each isolated socket to reproduce current observations; reports are intentionally not committed.
57+
58+
The corpus is deliberately selected and is not a representative workload. It contains 39 shared witnesses and four version-specific query witnesses; this does not imply identical dialects in 8.4.8 and 9.7.2. The version-specific entries record `applicability`, source productions for both versions, `native_validity` per version, and observed native outcomes. Their generic `validity=valid` denotes the 9.7.2 grammar target. The native helper selects the intended validity using the actual server version, so an expected 8.4.8 syntax rejection is not treated as a corpus error. Stored native observations are context, not substitutes for a current run. These additions remain ParserSQL gaps. The following differences include those four query witnesses and two separate DDL spot checks:
59+
60+
| Reduced feature | MySQL 8.4.8 observation | MySQL 9.7.2 observation |
61+
| --- | --- | --- |
62+
| `CREATE TABLE vector_test(v VECTOR(3))` | Syntax error 1064 | Accepted |
63+
| `CREATE VIEW IF NOT EXISTS view_test AS SELECT 1 AS a` | Syntax error 1064 | Accepted |
64+
| `EXPLAIN ANALYZE FORMAT=JSON INTO @plan SELECT 1` | Syntax error 1064 | Accepted |
65+
| `JSON_ARRAYAGG(a NULL ON NULL)` | Syntax error 1064 | Accepted |
66+
| `JSON_ARRAYAGG(a ABSENT ON NULL)` | Syntax error 1064 | Error 1235 outside JSON duality views |
67+
| `GROUP BY GROUPING SETS ((id),())` | Syntax error 1064 | Error 3889 without a secondary engine on the fixture table |
68+
69+
The pinned 9.7.2 grammar contains the corresponding [VECTOR type](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L7240), [view clause](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L18217), [EXPLAIN options](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L14612), [JSON aggregate clause](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L11397), and [GROUPING SETS](https://github.com/mysql/mysql-server/blob/008e09c2834b98143a8c067d4d225c90953050cf/sql/sql_yacc.yy#L12893). Grammar presence and successful execution are different observations: unsupported features and missing engine prerequisites remain unresolved, not accepted by the native helper. Likewise, window `GROUPS` and `EXCLUDE` appear in upstream grammar but returned unsupported-feature errors in both versions; this batch does not claim to implement them.
Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
#!/usr/bin/env python3
2+
"""Build only the standalone probe; first build libsqlparser.a with make -B lib."""
3+
import argparse
4+
import os
5+
from pathlib import Path
6+
import shlex
7+
import subprocess
8+
9+
ROOT = Path(__file__).resolve().parents[2]
10+
11+
12+
def main():
13+
parser = argparse.ArgumentParser(description=__doc__)
14+
parser.add_argument('--output', type=Path, required=True)
15+
args = parser.parse_args()
16+
args.output.parent.mkdir(parents=True, exist_ok=True)
17+
subprocess.run(shlex.split(os.environ.get('CXX', 'c++')) + [
18+
'-std=c++17', '-O2', '-Wall', '-Wextra', '-I' + str(ROOT / 'include'),
19+
str(Path(__file__).with_name('probe.cpp')), str(ROOT / 'libsqlparser.a'),
20+
'-pthread', '-o', str(args.output)], check=True)
21+
22+
23+
if __name__ == '__main__':
24+
main()

‎scripts/mysql_compat/native.py‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
"""Optional native checks on an explicitly selected isolated local MySQL server."""
2+
import re
3+
import subprocess
4+
import uuid
5+
6+
FIXTURE_SQL = """CREATE TABLE t(
7+
id INT PRIMARY KEY, n INT, j JSON, data JSON, body TEXT, d DATETIME,
8+
KEY idx(n), FULLTEXT KEY(body));
9+
CREATE TABLE u(id INT, n INT);
10+
INSERT INTO t VALUES(1,2,JSON_OBJECT('name','mysql','n',1),
11+
JSON_OBJECT('name','mysql','n',1),'mysql','2026-01-01');
12+
"""
13+
14+
15+
def classify_native(returncode, stderr):
16+
if returncode == 0:
17+
return dict(state='ACCEPTED', error_code=None)
18+
match = re.search(r'ERROR (\d+)\b', stderr)
19+
code = int(match[1]) if match else None
20+
if code in (1064, 1149):
21+
state = 'SYNTAX_REJECTED'
22+
elif code is None or 2000 <= code < 3000:
23+
state = 'CLIENT_ERROR'
24+
else:
25+
state = 'SERVER_ERROR'
26+
return dict(state=state, error_code=code)
27+
28+
29+
def run_native(cases, client, socket, user):
30+
if not socket or not socket.startswith('/'):
31+
raise ValueError('Native oracle requires an explicit absolute path to an isolated server socket')
32+
command = [client, '--no-defaults', '--protocol=SOCKET', '--socket=' + socket,
33+
'--user=' + user, '--batch', '--raw', '--skip-column-names',
34+
'--default-character-set=utf8mb4', '--connect-timeout=5']
35+
36+
def query(sql, schema=None):
37+
args = command + (['--database', schema] if schema else [])
38+
return subprocess.run(args, input=sql, text=True, capture_output=True, timeout=30)
39+
40+
metadata = query('SELECT @@version, @@session.sql_mode;')
41+
if metadata.returncode:
42+
raise RuntimeError('Cannot query native server metadata: ' + metadata.stderr.strip())
43+
version, mode = metadata.stdout.rstrip('\n').split('\t')
44+
if not re.match(r'^(8\.4\.8|9\.7\.2)(?:-|$)', version):
45+
raise ValueError('Native server must be MySQL 8.4.8 or 9.7.2; found ' + version)
46+
schema = 'parsersql_compat_' + uuid.uuid4().hex
47+
create = query(f'CREATE DATABASE `{schema}`;')
48+
if create.returncode:
49+
raise RuntimeError('Cannot create isolated native fixture schema: ' + create.stderr.strip())
50+
report = dict(state='RUN', version=version, sql_mode=mode, socket=socket,
51+
fixture_sql=FIXTURE_SQL, fixture_schema=schema, schema_cleaned_up=False,
52+
scope='SQL execution/EXPLAIN observations; not a raw-parser or equivalence oracle',
53+
cases=[])
54+
try:
55+
fixture = query(FIXTURE_SQL, schema)
56+
if fixture.returncode:
57+
raise RuntimeError('Native fixture setup failed: ' + fixture.stderr.strip())
58+
for case in cases:
59+
action = case.get('native_action', 'skip')
60+
if action == 'skip':
61+
report['cases'].append(dict(id=case['id'], state='NOT VERIFIED',
62+
reason='No native action configured; DDL/procedures excluded'))
63+
continue
64+
if action not in ('execute', 'explain'):
65+
raise ValueError('Unknown native action: ' + action)
66+
sql = ('EXPLAIN ' if action == 'explain' else '') + case['sql']
67+
# Reapply the captured mode for each connection, keeping every witness comparable.
68+
escaped_mode = mode.replace("'", "''")
69+
result = query(f"SET SESSION sql_mode='{escaped_mode}';\n" + sql, schema)
70+
record = classify_native(result.returncode, result.stderr)
71+
intended_validity = case.get('native_validity', {}).get(version.split('-')[0], case['validity'])
72+
record['intended_validity'] = intended_validity
73+
record['matches_intended_validity'] = (
74+
intended_validity == 'valid' if record['state'] == 'ACCEPTED' else
75+
intended_validity == 'invalid' if record['state'] == 'SYNTAX_REJECTED' else None)
76+
record.update(id=case['id'], sql=case['sql'], action=action, executed_sql=sql,
77+
returncode=result.returncode, stdout=result.stdout.rstrip('\n'),
78+
stderr=result.stderr.rstrip('\n'))
79+
report['cases'].append(record)
80+
finally:
81+
cleanup = query(f'DROP DATABASE `{schema}`;')
82+
report['schema_cleaned_up'] = cleanup.returncode == 0
83+
if cleanup.returncode:
84+
raise RuntimeError(f'Native schema cleanup failed for {schema}: ' + cleanup.stderr.strip())
85+
return report

‎scripts/mysql_compat/probe.cpp‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
// One hex-encoded SQL byte string per line; one TSV record per input.
2+
#include "sql_parser/parser.h"
3+
#include "sql_parser/emitter.h"
4+
#include <iostream>
5+
#include <string>
6+
7+
using namespace sql_parser;
8+
9+
static int hex_digit(char c) {
10+
if (c >= '0' && c <= '9') return c - '0';
11+
if (c >= 'a' && c <= 'f') return c - 'a' + 10;
12+
if (c >= 'A' && c <= 'F') return c - 'A' + 10;
13+
return -1;
14+
}
15+
16+
static void print_hex(StringRef text) {
17+
static constexpr char digits[] = "0123456789abcdef";
18+
for (uint32_t i = 0; i < text.len; ++i) {
19+
const auto c = static_cast<unsigned char>(text.ptr[i]);
20+
std::cout << digits[c >> 4] << digits[c & 15];
21+
}
22+
}
23+
24+
int main() {
25+
std::string line;
26+
while (std::getline(std::cin, line)) {
27+
if (line.size() % 2) {
28+
std::cerr << "Odd-length hex input\n";
29+
return 2;
30+
}
31+
std::string sql;
32+
sql.reserve(line.size() / 2);
33+
for (size_t i = 0; i < line.size(); i += 2) {
34+
const int hi = hex_digit(line[i]);
35+
const int lo = hex_digit(line[i + 1]);
36+
if (hi < 0 || lo < 0) {
37+
std::cerr << "Invalid hex input\n";
38+
return 2;
39+
}
40+
sql.push_back(static_cast<char>((hi << 4) | lo));
41+
}
42+
// Input, parser arena and emitter remain alive until all spans are printed.
43+
Parser<Dialect::MySQL> parser;
44+
const auto result = parser.parse(sql.data(), sql.size());
45+
std::cout << static_cast<int>(result.status) << '\t' << result.full_input
46+
<< '\t' << (result.ast != nullptr) << '\t';
47+
print_hex(result.remaining);
48+
std::cout << '\t';
49+
if (result.ast) {
50+
Emitter<Dialect::MySQL> emitter(parser.arena());
51+
emitter.emit(result.ast);
52+
print_hex(emitter.result());
53+
}
54+
std::cout << '\n';
55+
}
56+
return std::cin.bad() || !std::cout ? 2 : 0;
57+
}

0 commit comments

Comments
 (0)