Narrow AUTO_INCREMENT ids from cluster range claims; log-cursor anti-entropy; per-row last-writer-wins (fixes #170)

[maxpert]

Fixes #170.

Summary

Marmot stores every MySQL integer type as SQLite INTEGER, and generated ids came from a 53-bit HLC-derived space. Any application that declares INT AUTO_INCREMENT (or a narrower type) and reads ids into a 32-bit type therefore broke; the issue was reported with LLDAP, but the class is general.

This PR does three things:

• It keeps each column's declared width.
• It mints ids for narrow AUTO_INCREMENT columns from cluster-wide range claims, so ids fit the column and no two nodes hand out the same id.
• It fixes two pre-existing replication bugs that surfaced while testing this under node outages:
  • a lagging or restarted node could miss committed rows;
  • catch-up could let an older row image overwrite a newer one.

Commits

c063286 fix(protocol): deliver AUTO_INCREMENT insert ids and rule errors on every client path

• Rule errors from query transforms now reach the client with their MySQL code. Before, they were swallowed.
• One analysis drives both id injection and the AST rewrite.
• INSERT ... SELECT and VALUES ROW(...) into an injected column are refused with 1235, instead of letting SQLite pick a key.
• The OK packet's insert id is now per-statement, where before it was read from the shared connection's last-rowid.

1d9d887 fix(cluster): a lone restarted node must not be its own write quorum

• Membership (node id, address, incarnation and status) is persisted atomically in the data directory.
• A node restarted while its peers were unreachable no longer computes a membership of one and commits alone.

ef6d388 feat(schema): record declared MySQL integer widths beside SQLite's INTEGER

• A /*M:<bits>[u][a]*/ marker in the stored CREATE TABLE text keeps the declared width and signedness of TINYINT, SMALLINT, MEDIUMINT and INT.
• INTEGER PRIMARY KEY still aliases rowid. BIGINT DDL is byte-identical to before.

541fc0f feat(autoinc): cluster-wide range claims for narrow AUTO_INCREMENT columns

• A node claims a range of ids through a quorum, keyed by (database, table), in a hidden system table that clients cannot write and CDC never replicates.
• Participants vote under the claim key's intent lock and check that the range fits the column width from their own schema.

7a66596 feat(autoinc): mint narrow AUTO_INCREMENT ids from cluster-wide range claims

• Inserts into narrow AUTO_INCREMENT columns take ids from the node's claimed range, and a new range is claimed when it runs out.
• Explicit ids:
  • Above the cluster base: raises the base.
  • At or below it: refused with 1235, unless the id is in the node's own range.
  • Beyond the column width: 1264.
  • When the column is exhausted: 1062.
• Each claim keeps three floors: committed, seed and merged. Disjointness rests on the committed floor alone.
• A node that has just started holds its claim votes until its claim bases are merged with its peers. Inserts during that window get a retryable 1205.
• Ids are never reset by DROP/CREATE of a table or database.

d0d47f0 feat(replication): anti-entropy pulls every peer's commit log from persisted cursors

Delta sync compared watermarks rather than the transactions each peer actually committed, so a lagging or restarted node could miss ACKed rows. It is replaced as follows:

Pulling

• Each node pulls from every alive peer's local commit log. There are two new RPCs, ListCommittedLog and FetchTransactions.
• A durable per-(peer, database) cursor, keyed (seq, txnID), advances only over a contiguous prefix of applied entries.

Idempotent replay

• An __marmot_applied_txn marker is written in the same SQLite transaction as the effect, on every commit path.
• The schema version lives in __marmot_schema_version, in the DDL's own transaction.

Unfinished local transactions

• A transaction that a peer's log holds is treated as committed.
• A locally prepared copy is committed through the local path.
• A begin-only record is discarded and replayed.
• A late PREPARE for it is refused.
• A commit whose captured rows no longer match its durable prepare record is refused.
• The stale-transaction GC aborts only under the per-transaction commit guard.

Log GC and catch-up

• The log is garbage-collected by the minimum position every member has consumed, bounded by a maximum retention.
• A cursor behind the truncation point falls back to a snapshot, then re-applies the node's own log.
• A node turns ALIVE only once every database is caught up.

Rolling upgrade

• DDL and CREATE/DROP DATABASE are refused cluster-wide (1213, retryable) while any member does not yet serve the log-pull protocol.
• Upgrade every node, then run DDL.

Test harness

• Cluster tests gain a log-based stall watchdog and poll for readiness instead of sleeping.

8914a41 feat(replication): per-row last-writer-wins on the coordinator's commit timestamp

A node catching up from several peers applied each transaction once, but not in commit order. An older row image could overwrite a newer one, or an older INSERT could resurrect a deleted row.

Commit timestamp

• The coordinator merges every participant's PREPARE-time clock, decides one commit HLC, and sends it on COMMIT (unary and streamed) and in the log listing.
• Every node commits with that HLC and merges it into its clock.
• The clock is seeded at startup from the highest recorded commit.

Row versions

• __marmot_row_version (table, intent key, wall, logical, origin node, deleted) is stamped in the same SQLite transaction as each row change, on every commit path.
• A change applies only if its version is not older than the stored one.
• DELETE leaves a tombstone.
• An UPDATE that moves the primary key tombstones the old key independently of the new one.
• LOAD DATA rows are stamped per row, in batches under one savepoint.

DDL

• DROP TABLE removes a table's versions.
• RENAME re-keys them.
• A primary-key change clears them.

Tombstone purge

• Tombstones are purged in bounded chunks through a partial index once they are older than every known member's log retention.

5ebafa0 fix(cluster): a gracefully stopped node rejoins on restart; document narrow AUTO_INCREMENT ids

Graceful restart rejoins. On master, a node restarted after a graceful stop shuts itself down again about half a second later. Its peers keep the LEAVING record from its own departure, and the node reads that as a decommission.

• The fix: a restarted node resumes one incarnation above its persisted one.
• Incarnation bumps from refutation and from turning ALIVE are persisted, so a restart never comes back at or below an incarnation it already announced.
• REMOVED is never refuted.
• An admin decommission of a live node still takes effect, including while that node is still catching up.
• A node restarted on a wiped data directory after a graceful stop keeps the old behaviour. The operations guide says to treat it as a replaced node.
• There are unit tests and a graceful stop/restart cluster test.

Docs

• compatibility.mdx: narrow AUTO_INCREMENT behaviour, including every error code above.
• operations.mdx: the restart note and the procedure for changing membership one node at a time.
• sql-reference.mdx: corrects the INT AUTO_INCREMENT row, which said BIGINT.
• A new application compatibility page, lldap.mdx, for a tested application (see the end-to-end check below).

98eafbc fix(replication): recover cleanly from a kill mid-commit; end a rolled-back explicit transaction

The rewrite below exposed these defects. Each was fixed test first.

• Stuck prepared transaction. A local prepare whose captured rows were lost to SIGKILL wedged catch-up, and the
  node stayed JOINING. The puller now discards the local prepare under the commit guard and replays the peer's
  committed copy.
• Leaked row locks. A transaction committed just before SIGKILL had its commit record rewritten at open without
  releasing its restored row locks. Every later change to those rows was refused on replay, so the node kept a
  stale value for an ACKed row. The repair now cleans up like any commit.
• Claim deadlock. A narrow claim inside an explicit transaction deadlocked on the batch-committer flush, which
  waited on the writer that the same transaction's pinned session holds. A claim-only commit no longer flushes.
• Rolled-back explicit transaction. A pinned statement whose SQLite transaction the lock wait rolled back now
  answers 1213 and ends the transaction. A later COMMIT commits nothing. Before, it answered 1105 and COMMIT
  applied the earlier statements.
• Race. XsyncTransactionStore updates are copy-on-write. This fixes a data race with the stale-transaction GC
  that is also on master.
• Config. The AUTO_INCREMENT merge and base-sync intervals become config knobs; the defaults are unchanged.
• Docs. The operations guide says to rebuild wiped nodes one at a time.

ca2ca4a test(cluster): rewrite the integration suite to run in under two minutes

The cluster suite took about 11.5 minutes and had flaky tests. It is rebuilt on one harness:

• Binaries are built once per run. Each test gets its own ports and directory, and a cleanup kills its nodes and
  prints log tails on failure.
• Readiness is checked, not assumed: MySQL answers, every node sees the others ALIVE, and no node holds its claim
  votes. The votes are read over gRPC, including from the old-release node.
• Fixed sleeps go from 71 to none; the only time.Sleep left is inside the poll helper, and every wait has a
  deadline. SIGKILL and SIGTERM stay real.
• Assertions check exact rows, values, ids and error codes.
• Retries are typed and use jittered backoff. None masks the defect a test guards.

The 23 tests over 10s are rewritten from scratch, and a coverage ledger maps each guarded behaviour to its
replacement.

• Each test that guards a fix fails when that fix is reverted.
• Where an end-to-end test cannot force the interleaving (per-row LWW, the GC safe position, idempotent replay), a
  unit test that fails under the same mutation pins it.

Flaky tests, root-caused:

• Rolling upgrade. The harness never waited for the old node's claim votes. It now passes 20/20.
• TestGarbageCollectionIntegration -race. A product race, fixed above.

718e334 fix(autoinc): a narrow insert waits out the startup vote hold instead of failing at once

A narrow insert through a node that had just started, before it released its claim votes, got 1205 at once: the
node's own prepare declined the claim and the claim path gave up without waiting. Any application started together
with the cluster could fail on its first insert.

• The local hold is now a typed decline, ErrLocalVotesHeld. The claim waits for the release, which is signalled,
  not polled. The wait is bounded by transaction.lock_wait_timeout_seconds, and the claim returns 1205 only when
  that expires.
• A peer turning ALIVE triggers an immediate merge round, which shortens the hold.
• A narrow insert inside a transaction that already holds a pinned writer still fails fast, as before. Waiting
  there would deadlock with a joiner's snapshot checkpoint. An insert that is the first write of its transaction,
  the common ORM shape, holds no writer and waits.

276e96f test: describe narrow AUTO_INCREMENT tests by behaviour, not by one application

Test names, fixtures and comments now describe the general behaviour they check (a narrow insert as the first write
of an ORM-style transaction; a client starting during the vote hold). Statements and assertions are unchanged.

e5be232 fix(gc): gc_max_retention_hours = 0 means no maximum

The config validation and the tombstone horizon treat 0 as unlimited, but the log GC computed its cutoff as "now",
so every pass deleted every committed log entry whatever peers had consumed, forcing snapshot fallbacks. The same
behaviour is on master. With max <= 0 only the safe position and min retention now delete entries; a new test failed
before the fix.

ea2bae8 docs: describe the log-pull anti-entropy, row versions and narrow ids

• The README and replication.mdx described delta sync as the catch-up mechanism. They now describe the per-peer log
  pull, applied markers, snapshot fallback, GC by consumed position, the single commit timestamp, per-row versions,
  persisted membership and narrow AUTO_INCREMENT beside BIGINT's compact/extended ids.
• reference.mdx documents what the legacy delta_sync_* keys still control (read-replica bootstrap and GC
  validation only), the two autoinc interval keys, [metastore] strict_prepare_sync and retention 0.
• metrics.mdx documents the log-pull metrics and drops a delta sync counter that never existed.
• replication.mdx also corrects older errors: a nonexistent /intent/ key, node states and gossip timeouts, and the
  TxnID, LWW and HLC examples.
• Each claim was checked against the code by the writers, and 18 were spot-checked by an independent reviewer.

a02b9e6 fix(cluster): a joiner's promotion to ALIVE wakes a held seed's claim merge

A joiner refutes the seed's JOINING record as ALIVE before it marks itself JOINING, and its later promotion then read
as ALIVE -> ALIVE on the seed, so the immediate merge never fired and a held seed waited a full merge interval. An ALIVE
node announcing ALIVE at a higher incarnation now signals the merge. This was the root cause of a flaky hold-wait
cluster test (now 20/20). The harness also sets batch_commit.max_wait_ms = 1, since its clients write one statement
at a time; the suite dropped from about 130s to about 106s.

91bf5cf feat(metrics): report replication lag per peer from the log pull

marmot_v2_replication_lag_txns was registered but never set. It now holds, per peer, the committed transactions past
this node's pull cursor, summed over databases, as of the last pull round (an upper bound on remaining catch-up).
It counts entries rather than subtracting sequence numbers, which skip on restart. A round stopped at the page cap asks
the peer for its remainder through a new optional field; the count reads keys only, honours cancellation and never
fails the listing. The gauge is never set to a false 0, and a removed peer's series is deleted. Reviewed: PASS.

End-to-end check with a real application

The fixes above are general. As an end-to-end check, LLDAP 0.6.3 (the application in #170, which reads group ids
into an i32) was run against Marmot in Docker on the final build. Every run passed:

Scenario	Runs
1 node: migrations, users, groups, memberships, password, LDAP bind, restarts	2/2
3 nodes: two app instances on different nodes, one node gracefully restarted, checksums equal on every node	5/5
Same, with the node SIGKILLed	2/2
3 nodes, cold start on a fresh database: migrations replicate, schema equal everywhere	3/3

Every id fits its declared width, no id is issued twice across nodes, and no application log contains an error.

Compatibility and operations

• Existing BIGINT and 64-bit id columns are unchanged.
• Rolling upgrade: upgrade every node first. DDL and CREATE/DROP DATABASE are refused with a retryable 1213 until every member runs this release. DML keeps working.
• Mixed versions: while nodes run different releases, a commit coordinated by an older node has no shared commit timestamp. The row-version guarantee holds once every member is upgraded.
• Membership: range claims are disjoint while membership is stable. Change membership one node at a time using the documented procedure.
• Startup: until a node that has just started releases its claim votes, a narrow insert waits for the release,
  up to the lock wait. Inside a transaction that already wrote, it returns a retryable 1205 at once.

Testing

All results below are from local runs on this branch.

Unit and static checks

• ./... with the required build tags, run twice at every freeze: all 28 packages pass.
• -race on cfg, coordinator, db, grpc and protocol: clean. The xsync transaction-store race, which also occurs on master, is fixed in 98eafbc.
• go vet reports only 2 findings, both also on master: grpc/forward_session.go:356 and db/vector_index_manager_test.go:315.

Cluster suite (after ca2ca4a)

• go test -p 1 ./test/ three times in a row at ca2ca4a: 76 tests pass each time; twice more at 718e334 (117–119s). One is skipped; it is a subprocess helper.
• Each run took 105–111s, and no test took more than 10s; the slowest took about 6s.
• Former flaky or failing tests, run 20 times each: 20/20.
  • The rolling upgrade.
  • Explicit transaction across a node outage.
  • Prepared transaction on a killed node.
  • Same rows overwritten across an outage.
• Before the rewrite, the suite took about 689s and the slowest test 43s.

Mutation testing

• Every new rule was disabled in turn in a scratch copy, 17 arms for per-row LWW and 18 for its review fixes. Each arm fails at least one test.

Regression tests (each fails on the code before its fix)

• Missing rows:
  • a node down under load across two databases;
  • an explicit transaction held across an outage;
  • database operations while a node is down;
  • kill -9 during catch-up;
  • a prepared transaction on a killed node.
• A stale-transaction GC racing a local commit.
• Rolling upgrade, run against a binary built from the pre-log-pull commit.
• Stale rows:
  • out-of-order update, delete, and update-before-insert;
  • a primary-key move;
  • LOAD DATA;
  • two peers listing the same transactions in different orders.

Benchmarks

Local, alternated A/B runs against the parent commit.

Benchmark	Result
BenchmarkUserDatabaseWriteCommit	No measurable difference
BenchmarkAutoIncrementInsert	About 10 ms/op on both sides (dominated by the commit batching delay); about 15–23 more allocs/op
Single-row apply and commit (new micro-benchmark)	About 18 µs to 33 µs. The row-version stamp is one extra indexed upsert in the same transaction, with no lock and no round trip.
100-row transaction	About 0.6 ms, with the stamp statement prepared once per transaction
10,000-row LOAD DATA replay	About 9.8 ms before, 23.1 ms after. About 9 ms of the difference is writing 10,000 version rows.

Prepared-statement INSERT throughput was not measured; there is no benchmark for it yet.

Known limits and follow-ups (not in this PR)

Retryable errors

• An explicit transaction can get a retryable 1213 at COMMIT when its narrow claim goes stale under concurrent claimants on other nodes.
• An explicit transaction open longer than the lock wait is rolled back and gets 1213. A MySQL transaction has no
  such lifetime limit.

Failure model

• If two of three nodes lose their data at once, which is more failures than a 3-node cluster tolerates, they can
  release their claim votes before hearing from the surviving member.
  • A narrow insert then fails with 1062. A duplicate id is never stored.
  • The operations guide says to rebuild wiped nodes one at a time.

Documented limits

• A table rebuild (CREATE new; INSERT ... SELECT; DROP; RENAME) drops that table's tombstones.
• The read-replica stream applies rows unversioned; it replays one source in order.

Test coverage

• Admin decommission and removal are covered by unit tests only; there is no cluster test for them yet.

Pre-existing, unrelated to this PR

• REPLACE semantics, for replicated inserts and for the versioned update path, delete rows that conflict on a secondary unique index without checking their versions.
• Commit log after SIGKILL.
  • The commit log is written with pebble.NoSync, so a SIGKILLed node can lose its unsynced tail.
  • Its SQLite still holds those rows, but its log can no longer serve them to peers.
  • This is only observable with more than f simultaneous failures.
• Convergence after more than f failures (not root-caused).
  • Setup: nodes 2 and 3 were SIGKILLed, node 1 was wiped, and all three were restarted.
  • In 2 of 5 runs, a row that node 1 committed with node 3 after the restart never reached node 2.
  • Node 2's pull cursor for node 1 stayed at 0 while it reported itself caught up.
  • With the same scenario inside the failure model (graceful stops, one wipe), every node converges.
• Joining nodes stay ALIVE to peers. MarkJoining does not bump the incarnation, so peers that already saw a node
  ALIVE keep it ALIVE while it catches up.
• Harness. go test -count>1 reuses per-test directories; repeated runs use a loop of -count=1.
• Failure detection. A SIGKILLed node is never marked SUSPECT while its survivors gossip, because every rumour
  refreshes lastSeen (grpc/node_registry.go, "Rule 3").
• LOAD DATA row count. LOAD DATA LOCAL reports RowsAffected = 1 for any number of rows (coordinator/handler.go,
  the handleMutation default). Narrow AUTO_INCREMENT tables report the exact count.
• Snapshot and open transactions. A snapshot holds the database manager lock while its TRUNCATE checkpoint waits
  on any open pinned writer from a client transaction.
• Conflicts with a node down. With 2 of 3 nodes up, two transactions that conflict on one row can both abort
  with 1213, with no winner. Clients that retry at a fixed interval can livelock; jittered backoff resolves it.
  The same code path exists on master; the runtime behaviour there was not verified.

🤖 Generated with Claude Code