Skip to content

fix(compat): close 21 defects blocking real MySQL clients - #171

Merged
maxpert merged 2 commits into
masterfrom
fix/ddl-implicit-commit
Sep 2, 2026
Merged

maxpert merged 2 commits into
masterfrom
fix/ddl-implicit-commit

Conversation

@maxpert

@maxpert maxpert commented Sep 2, 2026

Copy link
Copy Markdown
Owner

Found by driving LLDAP 0.6.3 (Rust, sqlx + sea-orm) against a live Marmot node and auditing the code paths it exercised. Each fix is covered by tests that fail on the prior code.

Result: LLDAP went from dying on its second schema migration to completing all 11 migrations. It is still blocked afterward by #170 (generated ids overflow its 32-bit id type), which is a design decision, not a defect in this branch.

CDC correctness

  • Tables with no declared PRIMARY KEY could not replicate UPDATE/DELETE at all. The capture side never emitted the rowid the apply side required. Rowid is now the replicated identity; capture is refused when a column shadows rowid/oid/_rowid_.
  • BLOB columns silently became TEXT on every replica. go-sqlite3 hands TEXT and BLOB back identically as []byte, so the distinction was already lost before decoding. The choice is now made at capture time from column affinity (only BLOB affinity stays msgpack Bin), with a strict CDC decode that no longer coerces Bin to string. Would have corrupted password hashes and UUID columns.
  • Segfault on any DML against a table with a VIRTUAL generated column — go-sqlite3's preupdate row() dereferences a NULL value for virtual column indices. Capture now refuses such tables before touching Old()/New().
  • PRAGMA table_info renumbers cid when hidden columns exist, which silently broke PK positional indexing. Now uses table_xinfo true positions. (Latent pre-existing bug.)
  • Schema-cache reload failures after DDL are propagated instead of logged-and-ignored; schema/value count mismatches fail loud instead of truncating.
  • Zero-rows-affected CDC applies are logged at Debug so divergence is diagnosable; added FK ON DELETE CASCADE convergence tests.

Interactive transactions

DML inside BEGIN...COMMIT was buffered until COMMIT and answered with a fabricated RowsAffected: 1, so ORMs got no real last_insert_id and no read-your-own-writes.

  • DML now executes eagerly against a pinned SQLite transaction; CDC is captured incrementally and replayed through the existing 2PC path at COMMIT. The pinned transaction is always rolled back locally — the write lands only via CDC replay.
  • Pinned sessions are released before 2PC (holding them through the local commit deadlocks SQLite's single writer).
  • Pinned state is released on disconnect, rollback and 2PC rejection; fixed a forwarded-session leak that never released transaction state.
  • Forward-session eviction now serializes against in-flight statements. An eviction racing a slow COMMIT could discard captured CDC, after which COMMIT reported success with the write silently lost. COMMIT additionally fails loudly if pinned state disappears rather than treating it as an empty transaction.
  • New transaction.ddl_implicit_commit (default true): DDL commits the open transaction, as MySQL does.

SQL transpilation

  • Partially-parsed DDL was forwarded truncated into 2PC — Vitess returns a partial AST on syntax errors, so alter table users add CONSTRAINT ... UNIQUE (email) became alter table users. Now rejected with a clean parse error.
  • Every HAVING was emitted as WHERE. Vitess models both with the same node and the serializer ignored the discriminator, so any GROUP BY ... HAVING (including inside subqueries) produced invalid SQL.
  • Index DDL now flows through one path: ADD CONSTRAINT UNIQUE, ADD INDEX, ADD UNIQUE and standalone CREATE [UNIQUE] INDEX all emit SQLite CREATE INDEX; DROP INDEX ... ON drops the ON clause and handles backtick-quoted names.
  • CHARACTER SET/COLLATE are stripped on ALTER TABLE ADD/MODIFY/CHANGE COLUMN, sharing the CREATE TABLE implementation.

MySQL wire protocol

  • COM_STMT_PREPARE reported zero columns. sqlx builds its column-name index from the prepare response, so every prepared query failed to find its columns — this was the actual cause of the reported client panic. Now sends real column count, names and types.
  • Column types are inferred from data and declared types instead of always VAR_STRING; time.Time and binary DATETIME encode correctly.
  • SERVER_STATUS_IN_TRANS is now set so clients can see transaction state.
  • Implemented COM_STMT_SEND_LONG_DATA (no response, per-parameter accumulation, cleared only by RESET/CLOSE) and COM_STMT_RESET; the unhandled default sent an ERR packet and desynced the connection.
  • The unsigned parameter flag is honored; values above MaxInt64 return ER_WARN_DATA_OUT_OF_RANGE rather than a confusing driver error or a negative value.

Parameter handling

  • not enough args to execute query: want 6 got 5 — literal extraction turns the injected auto-increment id into a placeholder, but every execution site chose either wire params or extracted params, never both. Added Statement.MergeExecParams, which interleaves them by per-placeholder provenance (proven by a test where the literal precedes the bound param).
  • Added Statement.WithResolvedParams so rewritten statements cannot carry stale placeholder provenance and misinterleave already-resolved args.

Test harness

The coordinator test package initialized the pipeline with a nil ID generator and left transpilation disabled, so auto-increment injection and BEGIN were both inert in tests — which is why several of these defects shipped. Both fixed; full suite still green.

Verification

Follow-ups filed

#164, #165, #166, #167, #168, #169, #170

🤖 Generated with Claude Code

https://claude.ai/code/session_015eBWcTYK93Zy5KAAwLMzZc

Zohaib Sibte Hassan and others added 2 commits September 1, 2026 14:23
MySQL has no transactional DDL. CREATE, ALTER, and DROP are on its
implicit-commit list: each ends the transaction a session has open before
it runs, then runs on its own, leaving no transaction open. Clients
written against MySQL depend on that, and schema migration tools rely on
it directly - a migration adds a column and then updates it in what looks
like one transaction, which works only because the DDL committed first.

Marmot buffered every statement in an explicit transaction until COMMIT,
so DML could not see a column added earlier in the same transaction. The
statement failed with "no such column", and no wire-protocol work could
fix it. LLDAP's v2 migration is exactly this shape and could not run.

DDL in an open transaction now commits the buffered statements and closes
the transaction before executing, controlled by transaction.
ddl_implicit_commit and defaulting to true for MySQL compatibility. Set
it false to keep DDL inside the transaction and replicate it atomically
with the surrounding statements, accepting that DML in that transaction
cannot see the pending schema change; a test pins that trade-off.

Write forwarding needed no change: ForwardQueryResponse.InTransaction is
read from the leader's session state after execution, so a replica
observes the implicit commit without being told about it.

Also fixes the coordinator test harness, which never initialized the
query pipeline and left TranspilationEnabled false on its sessions. BEGIN
parsed as StatementUnsupported there, so tests meaning to exercise the
explicit-transaction path silently ran in autocommit and passed for the
wrong reason. Both are corrected, which is what surfaced this defect.

The setting is documented in every shipped config with a [transaction]
section and in the configuration reference.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015eBWcTYK93Zy5KAAwLMzZc
Found by driving LLDAP 0.6.3 (Rust, sqlx + sea-orm) against a live node
and auditing the paths it exercised. Each fix is covered by tests that
fail on the prior code.

CDC correctness
- Replicate rowid as identity for tables with no declared PRIMARY KEY;
  UPDATE/DELETE on such tables previously failed to replicate at all.
  Refuse capture when a column shadows rowid/oid/_rowid_.
- Preserve BLOB vs TEXT storage class across replication. go-sqlite3
  hands both back as []byte, so the choice is now made at capture time
  from column affinity (only BLOB affinity stays msgpack Bin), with a
  strict CDC decode that no longer coerces Bin to string.
- Refuse capture on tables with VIRTUAL generated columns instead of
  segfaulting: go-sqlite3's preupdate row() dereferences a NULL value
  for virtual column indices.
- Use PRAGMA table_xinfo true column positions. table_info renumbers
  cid when hidden columns exist, which silently broke PK indexing.
- Propagate schema-cache reload failures after DDL instead of logging
  and continuing with a stale cache.
- Fail loud on schema/value count mismatch rather than truncating.
- Log zero-rows-affected CDC applies at Debug so divergence is
  diagnosable; add FK ON DELETE CASCADE convergence tests.

Interactive transactions
- Execute DML eagerly against a pinned SQLite transaction instead of
  buffering until COMMIT, so clients get real rows-affected and
  last_insert_id and can read their own writes. CDC is captured
  incrementally and replayed through the existing 2PC path at COMMIT;
  the pinned transaction is always rolled back locally.
- Release pinned sessions before 2PC to avoid a single-writer deadlock.
- Roll back pinned state on client disconnect, 2PC rejection and
  rollback; fix a forwarded-session leak that never released
  transaction state.
- Serialize forward-session eviction against in-flight statements. An
  eviction racing a slow COMMIT could discard captured CDC and let the
  commit report success with the write lost. COMMIT now fails loudly if
  pinned state disappears rather than reporting an empty transaction.
- Add transaction.ddl_implicit_commit (default true): DDL commits the
  open transaction as MySQL does.

SQL transpilation
- Reject partially-parsed DDL instead of forwarding a truncated
  statement into 2PC. Vitess returns a partial AST on syntax errors.
- Serialize HAVING as HAVING. Vitess models WHERE and HAVING with the
  same node, and every HAVING was being emitted as WHERE.
- Transpile index DDL through one path: ADD CONSTRAINT UNIQUE, ADD
  INDEX, ADD UNIQUE and standalone CREATE [UNIQUE] INDEX all emit
  SQLite CREATE INDEX; DROP INDEX ... ON drops the ON clause and
  handles backtick-quoted names.
- Strip CHARACTER SET/COLLATE on ALTER TABLE ADD/MODIFY/CHANGE COLUMN,
  sharing the CREATE TABLE implementation.

MySQL wire protocol
- Report real column count, names and types in COM_STMT_PREPARE. sqlx
  builds its column index from the prepare response, so an empty one
  made every prepared query fail to find its columns.
- Infer column types from data and declared types instead of always
  VAR_STRING; encode time.Time and binary DATETIME correctly.
- Set SERVER_STATUS_IN_TRANS so clients can see transaction state.
- Implement COM_STMT_SEND_LONG_DATA (no response, per-parameter
  accumulation) and COM_STMT_RESET; the unhandled default previously
  sent an ERR packet and desynced the stream.
- Honor the unsigned parameter flag; values above MaxInt64 now return
  ER_WARN_DATA_OUT_OF_RANGE instead of a driver error or a negative.

Parameter handling
- Merge wire parameters with extracted literals positionally via
  Statement.MergeExecParams. Literal extraction turns the injected
  auto-increment id into a placeholder, so a prepared INSERT needed
  more arguments than the client bound ("want 6 got 5"). Every
  execution site now uses the shared helper.
- Add Statement.WithResolvedParams for rewritten statements so stale
  placeholder provenance cannot misinterleave already-resolved args.

Test harness
- Initialize the coordinator test pipeline with a real ID generator and
  enable transpilation. Auto-increment injection and BEGIN were both
  inert in tests, which is why several of these defects shipped.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015eBWcTYK93Zy5KAAwLMzZc
@maxpert
maxpert merged commit 2d871bc into master Sep 2, 2026
10 checks passed
@maxpert
maxpert deleted the fix/ddl-implicit-commit branch September 2, 2026 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant