Skip to content

feat: optional distributed dispatch layer (MySQL leases, outbox + RabbitMQ, Redis admission) - #3

Merged
sclfcz merged 20 commits into
mainfrom
feat/distributed-runs
Nov 2, 2026
Merged

sclfcz merged 20 commits into
mainfrom
feat/distributed-runs

Conversation

@sclfcz

@sclfcz sclfcz commented Nov 2, 2026

Copy link
Copy Markdown
Owner

概述

新增一套可选的分布式派发层,用于多 Worker 场景。两个开关(DIST_SCHEDULER、DIST_IM_OWNERSHIP)默认关闭,关闭时行为与未引入该层完全一致;MySQL 是运行状态的唯一权威,Redis 只做门控,RabbitMQ 只负责唤醒。

分阶段内容

  • MySQL 租约 + fencing 认领:FOR UPDATE SKIP LOCKED 认领、lease_token 递增围栏、心跳续租与 reaper 回收。
  • Transactional Outbox + RabbitMQ:状态变更与 outbox 行同事务写入,relay 用 publisher confirms 投递,消费端去重,含重试阶梯与 DLQ。
  • Redis 准入:租户并发槽位、模型令牌桶(全部 Lua + 服务端时间),以及按租户配额的 claimFair。
  • 步骤 checkpoint 与外部副作用对账:intent → committed/unknown → needs_review,发送前写 send_started_at,结算窗口后判定,不静默重放。
  • 定时任务接入派发层:SQLite 仍是结果权威,MySQL 只是派发信封;MySQL 不可用时自动回退到 legacy 认领循环。
  • IM 凭证跨实例单持有者:dist:im:owner:{credential} 租约 + fencing token,入站/出站逐条门控,丢租约后重新进入备用。
  • 工程化:compose 环境(MySQL 8.4 / RabbitMQ 4 / Redis 7)、一键演示 npm run dist:demo、被回收任务的优先级通道。

对主服务的改动

只涉及 src/db.ts、src/im-manager.ts、src/index.ts、src/task-scheduler.ts,且全部在开关之后(新增导出函数与钩子,不改既有行为)。

新增依赖

mysql2、amqplib、ioredis、@types/amqplib。

如何验证

docker compose -f docker-compose.distributed.yml up -d
export DIST_MYSQL_TEST_URL=mysql://dist:dist@127.0.0.1:3307/dist
export DIST_AMQP_TEST_URL=amqp://dist:dist@127.0.0.1:5672
export DIST_REDIS_TEST_URL=redis://127.0.0.1:6379
npx vitest run tests/distributed
npm run dist:demo

已知边界

  • 执行层仍是单机(SQLite + 本地工作区);本层解决派发、租约与副作用一致性,不是把执行拆成分布式。
  • 没有按会话的 serial_key 租约:同一会话内任务的顺序仍由单机 GroupQueue 保证。
  • IM 回复未接幂等键:buffered 类平台下是至少一次语义,重复回复需要回复侧去重。
  • 三实例 IM 僵尸混沌场景在 mock 平台下不可观测(两实例场景已验证并断言通过)。

测试结果

make test 与合并前基线一致:3190 通过 / 7 失败 / 78 跳过,7 个失败全部是既有的 tests/container-session-permissions.test.ts 环境问题(/proc/self/fd 在当前 Docker 镜像下的行为)。

sclfcz added 20 commits November 1, 2026 15:39
…kens

Adds an isolated distributed subsystem under src/distributed/ that claims
task runs from MySQL 8.0+ with FOR UPDATE SKIP LOCKED, keeps them alive with
heartbeats, and fences every worker write on (status, lease_owner,
lease_token) so a worker that lost its lease cannot overwrite a takeover
worker's result.

- config: env parsing with heartbeatMs <= leaseMs/3 enforced
- migrations: versioned SQL, GET_LOCK-serialised, refuses newer schemas
- repository: submit/claim/heartbeat/complete/fail/reap/cancel, all SQL
  parameterised and every state change atomic with its event row
- worker/reaper: abort on lost lease, bounded graceful shutdown, heartbeat
  budget so transient DB errors do not abandon a healthy run
- cli-worker: SIGTERM/SIGINT shutdown for standalone processes
- chaos: 500 runs, 10 SIGKILLs with replacements, one SIGSTOP zombie,
  SQL assertions and a JSON report with queue/takeover latencies
- tests: real-MySQL integration suite, skipped when DIST_MYSQL_TEST_URL is
  unset so make test never requires a database

Not wired into task-scheduler or the main service.
… stop

- reapExpired no longer rewrites available_at, so a recovered run keeps its
  original queue position; the reaped event now records the lease expiry it
  replaced (used for lease-expiry-detection metrics).
- Every pooled connection pins `time_zone = '+00:00'`, so a server whose
  default zone is not UTC cannot shift stored timestamps. Covered by a test
  that sets the global zone to +08:00 and reads started_at back.
- Migrations moved next to the subsystem (src/distributed/migrations).
- New repository.release(): hands a claimed run back to the queue with
  attempt - 1, token + 1 and a `released` event. Worker.stop() releases runs
  it cannot finish before aborting, so a graceful shutdown no longer burns an
  attempt.
- Worker config gains DIST_CLAIM_BATCH (default = concurrency) and reports
  claim latency + deadlock retries on shutdown for the new benchmark.
- Chaos: 6 workers x 4, sleep 100-1000ms, maxAttempts 3; asserts zombie
  stale complete/fail writes and that the zombie never succeeded with a stale
  token; reports leaseExpiryDetectionMs and crashRecoveryMs instead of
  takeover latency.
- New scripts/dist-bench.mjs (`npm run dist:bench`) measures claim-path
  throughput for 1/2/4/8 workers plus an 8-worker claim-batch=1 control.
Phase 2 adds an at-least-once wake-up path while MySQL stays the only
authority on execution rights.

- 002_outbox.sql: dist_outbox + dist_consumed_messages
- every state change that makes a run executable (submit, fail->retry_wait,
  reap->retry_wait, release) inserts its outbox row in the same transaction,
  copying available_at from the run so a backoff can never be published early
- relay: SKIP LOCKED batch -> ConfirmChannel publish -> waitForConfirms ->
  mark published -> COMMIT; failure bumps attempts/last_error and rolls back;
  fault injection for crash-after-confirm and duplicate publish
- topology: quorum main queue, classic TTL retry queues that dead-letter back
  to the exchange, DLQ; DLQ list/replay CLI
- worker: DIST_DISPATCH=poll|mq; mq consumes run.ready, claims in MySQL under
  the same fencing rules (claimById + dist_consumed_messages), acks right
  after the claim commits, and keeps a slow fallback poll for lost messages
- claim/reap now run under READ COMMITTED (SET TRANSACTION before BEGIN)
- new scripts: dist:relay, dist:dlq, dist:chaos:mq, dist:bench:dispatch
- dist-bench gained DIST_BENCH_ROUNDS / DIST_BENCH_REPEAT (median of N)

Not wired into task-scheduler or the main service.
The purge ran at 8s while the broker was already restarting, so it never
happened, and the restart landed after the work had drained, leaving the
recovery metric empty. Purge at 3s, restart at 7s, and report commandMs.
Redis now gates admission; MySQL stays the only authority on run state.

- tenant-slots: one ZSET per tenant (member=runId, score=lease expiry). Acquire
  removes expired members before the capacity check, so a crashed worker's
  slots are reclaimed by the next acquirer (no janitor, no exit hook). Renew
  refuses to resurrect an expired slot; release is best effort.
- worker: poll and mq paths acquire a slot before executing; a rejection
  returns the run to the queue without consuming an attempt and with a short
  re-queue delay (otherwise the throttled tenant keeps the global queue head
  and starves everyone else — found by dist:bench:fairness). Heartbeats renew
  the slot; a missing slot aborts like a lost lease. A worker that already lost
  its lease never frees the slot (the new owner may hold it).
- rate-limiter: Lua token bucket with injectable clock, tryAcquire +
  acquireBlocking; documented call site in src/distributed/README.md.
- DIST_TENANT_CONCURRENCY (0 = disabled), DIST_REDIS_URL,
  DIST_ADMISSION_FAIL_MODE=open|closed (default open, logs admission_degraded).
- scripts: dist:chaos:tenant (3x200 runs, 6 workers, 5 SIGKILLs, slot-leak
  assertions) and dist:bench:fairness (BIG 5000 vs SMALL 200, baseline vs
  isolated, 3 repeats).
- compose: redis:7-alpine with healthcheck.
… claiming

- tenant-slots/rate-limiter: all Lua scripts take the clock from Redis TIME;
  clients no longer pass nowMs, so worker clock skew cannot change capacity.
- slot members are `${runId}:${leaseToken}`: release only removes its own
  generation, and an acquire from a newer token deletes the older one instead
  of counting the run twice.
- claimFair(): candidates are grouped by tenant (unlocked read), then claimed
  per tenant within one transaction, skipping tenants Redis reports as
  saturated. Workers compute that view from a single Lua snapshot over
  dist:slots:active. DIST_TENANT_CONCURRENCY=0 keeps the old claim path.
- migration 003 adds idx_tenant_due (status, tenant_id, available_at).
- release() records reason=admission|shutdown, so admission churn is countable
  in SQL (admissionRejects/wastedClaims).
- chaos-tenant: asserts claimedEvents/runs <= 1.5 and reports the churn.
- bench-fairness: enforces and records BIG-then-SMALL submission order, runs
  baseline / isolated-10 / isolated-32, and reports SMALL p50-p99, spans,
  cluster utilization (100ms sampling) and wasted claims.
Phase 4 makes a run replayable at step granularity.

- 004 migration: needs_review run status + dist_run_steps ledger
  (intent/committed/unknown/failed, idempotency_key, request_hash, provider_ref)
- durable.ts: RunContext with step()/effect(); every ledger write is fenced on
  (status='running', lease_owner, lease_token) and a lost fence raises
  LeaseLostError before anything is written
- effect protocol: intent -> committed | unknown -> (lookup) -> committed |
  needs_review; timeouts/5xx mark unknown and retry, 4xx marks failed and the
  run goes dead, request changes are rejected
- worker: per-run DurableContext, NeedsReviewError never overwrites the run,
  per-worker durable counters; reaper scans stale unknown steps (monitoring)
- HttpEffectProvider + scripts/mock-effect-provider.mjs (node:http) with
  idempotency/drop-response/error/latency knobs
- scripts/dist-chaos-effects.mjs: naive vs durable vs durable-nolookup
- README documents the state machine and the fencing limitation (a fence
  cannot stop an outbound request; only provider idempotency can)
…sification

- 005: dist_run_steps.send_started_at, written (fenced) immediately before the
  provider call together with attempts+1; attempts now counts sends only.
- Rule: `intent` with send_started_at IS NULL means the request provably never
  left, so the retry sends directly (no lookup, no needs_review). This covers a
  worker killed between the intent write and the send.
- Reconcile misses now wait: while NOW(3)-send_started_at <
  effectTimeoutMs + DIST_EFFECT_SETTLE_MS the retry is deferred
  (reconcileDeferred) with available_at pushed past the window (fail gained
  minDelayMs) instead of sending again.
- EffectNotAppliedError: connection refused / DNS failure and provider-declared
  notAppliedStatuses (default 429/503) revert the step to intent with a cleared
  send_started_at and retry with normal backoff; timeouts, post-send resets and
  500/other 5xx stay `unknown`.
- needs_review events carry a reason (unknown_outcome /
  send_started_unknown_outcome) so the report can split them.
- chaos: LATENCY_MS=300, 5 repeats; report adds reconcileDeferred,
  notAppliedRetries and needsReviewByReason.
Behind DIST_SCHEDULER=1 (default off, legacy pump untouched when unset).

- SQLite task_runs stays the result/history/notification authority; the dist
  run is only a dispatch envelope (idempotency_key = task_run.id,
  tenant_id = workspace owner, max_attempts = 20 dispatch attempts).
- pump stops claiming and wakes a forwarder instead; the forwarder submits
  queued/retry_wait runs with the SQLite available_at (outbox copies it).
- embedded in-process worker + ScheduledTaskExecutor: every claim re-reads
  SQLite and mirrors terminal / not-due / running-elsewhere / claimable;
  after execution it re-reads again and mirrors success/dead or releases.
- claimTaskRunById mirrors claimNextTaskRun exactly, only fixed on id.
- executeClaimedTaskRun now returns Promise<void> (all settle paths resolve).
- crash semantics unchanged: started runs are never replayed, the dist mirror
  follows whatever failExpiredStartedTaskRuns wrote.
- MySQL lease fences dispatch only; the SQLite lease fences execution, so a
  lost dist lease never aborts a running legacy task.
- host tasks skip tenant slots (CLAUDE.md §4); container tasks keep them.
- scripts: dist:e2e:scheduler (latency + kill -9 recovery) and
  dist:smoke:agent (manual, spends real model calls, not run).
… is down

- submit() accepts an explicit id; the forwarder passes the SQLite
  task_run.id so the cancel hook (which only has the run id) addresses the
  right envelope instead of silently doing nothing.
- the bridge tracks dispatch health: 3 consecutive forwarder failures or a
  10s outage uninstall the hooks so the pump falls back to claimNextTaskRun,
  logging scheduler_dist_degraded; a 5s SELECT 1 probe reinstalls them after
  two successes and logs scheduler_dist_recovered.
  Safe because both paths claim the same SQLite row with the same conditions
  and lease fencing, so a run can never execute twice.
- pumpTaskScheduler is exported and getDistSchedulerHooks() added so the
  fallback is testable without driving timers.
- dist:e2e:scheduler gains a mysql-outage round: 100 tasks spread over 5-40s,
  mysqladmin shutdown for 15s, restart, three-phase start latency and a
  once-only side effect check.
- tests: cancel of a running and of a queued run, and degrade/fallback/recover.
Behind DIST_IM_OWNERSHIP=1 (default off: the manager behaves exactly as before).

- ConnectionOwnership: `dist:im:owner:{credentialKey}` holds
  `{instanceId}:{token}` with PX leaseMs, `dist:im:fence:{credentialKey}` is a
  monotonic token. acquire/renew/release are single Lua scripts, so a stale
  owner can neither extend nor delete the new owner's key. The credential key
  reuses claimCredential's provider+sha256 fingerprint; no secret reaches Redis.
- local self-fencing uses the monotonic clock (leaseMs - renewMs window), so a
  process that was frozen notices the expired lease before handling anything.
- connectChannel acquires after the local credential claim and before
  channel.connect; a loser stays standby, keeps the channel object and retries
  every renewMs. onLost tears down through the existing
  per-(userId, channelType) lock and returns to standby.
- inbound is gated per message (frozen processes get queued events first):
  drop + inboundDroppedNotOwner. outbound (sendMessage/sendFile) throws
  ConnectionOwnershipLostError (a DefinitiveChannelDeliveryError, i.e. "the
  provider never saw it") so the outbox retries instead of freezing the row.
- normal disconnect/shutdown release the lease so a standby takes over without
  waiting for expiry; Redis outages fence everyone off within one lease window.
- scripts/dist-chaos-im.mjs + dist-chaos-im-worker.mjs: worst-case mock
  platform (every message pushed to every connection), 3 instances, SIGKILL /
  SIGSTOP / SIGTERM injections, overlap and duplicate-reply accounting.
…owner

- IMConnectionManager.setChannelRebuilder(): a lost lease now rebuilds a fresh
  transport (index.ts resolves the channel account and calls the same
  create*Channel factories as reconnect) and retries connectChannel. Reusing
  the disconnected object was the availability bug: the process dropped the
  connection and never came back. A null rebuild (account deleted/disabled)
  clears the standby state instead of retrying forever.
- sendImage now shares the outbound ownership gate with sendMessage/sendFile;
  streaming cards and setTyping stay ungated (documented in the README).
- ConnectionOwnership bounds every Redis round-trip: with an offline queue a
  renew can hang forever and the self-fence would never fire.
- chaos: scenarios run per platform mode (push / buffered with offset acks) and
  a 2-instance freeze-handover case; injection targets come from the actual
  Redis lease holders (10s wait, hard failure if none), SIGSTOP assertions,
  Redis outage via `docker stop commerceagent-dist-redis`, and lost replies
  bucketed by phase.
- docker-compose.distributed.yml: host ports are configurable
  (DIST_MYSQL_HOST_PORT / DIST_AMQP_HOST_PORT / DIST_REDIS_HOST_PORT) so the
  stack can coexist with the local dev MySQL and Langfuse's Redis. Verified on
  MySQL 8.4 / RabbitMQ 4 / Redis 7 with `docker compose up -d` and healthy
  healthchecks; tests/distributed and the four chaos scripts pass there.
- dist-chaos-mq: a restarted broker may not have re-declared the DLQ yet; a 404
  closed the admin channel and crashed the run. Channels now carry error
  handlers and the DLQ check uses a throwaway channel.
- scheduler-bridge: startForwardingOnly() lets tests install the hooks without
  the embedded worker racing the SQLite run they are inspecting.
- scripts/dist-demo.mjs (npm run dist:demo): compose up -> migrate -> 3 workers
  + 2 relays -> 300 runs -> kill -9 a worker + RabbitMQ restart -> summary
  table; --down also stops the stack.
- dist-chaos-im: push rate is configurable (IM_CHAOS_RATE_PER_SEC), publishing
  waits for the fleet to be connected, backlog drains before losses are
  counted, capacity probing (IM_CHAOS_PROBE=1) and per-credential overlap.
- README: subsystem overview with a mermaid diagram, per-phase invariants, a
  fault -> behaviour -> verification table and the boundary list; numbers are
  referenced by report name only.
…mo recovery

- chaos buffered mode is now a real queue: the platform keeps an
  `ackedSeq` per credential and a new connection resumes there, so messages a
  killed owner never acknowledged wait for the next owner. The worker
  acknowledges only after the reply request succeeded. Result: 0 lost across
  five SIGKILLs; the 102 duplicates are exactly the "replied but killed before
  acking" at-least-once case and are counted/asserted as cross-instance
  re-delivery (production needs an idempotency key on the reply side).
- zombie injection picks the instance that both holds leases *and* has live
  platform sockets, so the freeze actually has something to observe.
- createRedis caps the reconnect backoff at 500ms; the IM outage recovery
  bound is documented as leaseMs + renewMs + 500ms.
- dist:demo runs 1-3s tasks, kills the worker holding the most running runs,
  restarts RabbitMQ while <50% is done, and reports per-run recovery
  (p50/max), reaped count and claims after the restart; it fails loudly when
  any of those invariants break.
…onnect

- chaos push mode is a real stream: /connect returns a never-ending chunked
  response and the worker reads JSON lines, so a frozen process keeps its
  sockets open (zombie overlap is now observable). buffered mode keeps the
  offset-acknowledged long poll.
- ConnectionOwnership records a diagnostic timeline (redis_ready,
  acquire_start/end, rebuild_start/end) and the chaos report turns it into
  per-stage timings relative to the Redis PING; createRedis bounds the connect
  attempt at 1s and the backoff at 500ms.
- RabbitMQ 4 quorum queues reject x-max-priority (406), so recovered work gets
  its own quorum lane `dist.runs.ready.hp`: outbox rows for reap/release are
  `run.recovered`, the relay routes them there, and the consumer subscribes to
  the priority lane before the backlog. Test: 200 queued messages + 1 recovery
  -> claimed within the first 20 deliveries.
- compose redis runs with --appendonly yes (fencing tokens must stay monotonic);
  README documents the recovery bound and the token-vs-instance caveat.
- root cause of the push-mode reply regression: the streaming fan-out was
  implemented only in the HTTP /publish endpoint, but the driver publishes
  in-process (state.publish), so streamWrites stayed 0 and the worker never
  received a line. Publishing now goes through one helper used by both paths;
  minimal scenario (1 credential, 1 instance, no injections) replies 884/884.
- ownership timeline events are written as JSON lines on worker stdout and the
  driver collects them, so a truncated /stats buffer cannot hide the recovery
  stages. healthyAt is the first successful PING (25ms cadence) and recovery
  stays signed: a negative value is reported, never clamped.
- steady-phase replies must exceed half of steady publishes (a scenario with
  zero replies can no longer pass "0 duplicates").
- README: recovery bound formula with the measured stages report pointer; the
  priority lane is documented as a bounded wait, not strict priority.
@sclfcz
sclfcz merged commit 3fa3ece into main Nov 2, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant