Repository navigation
feat: optional distributed dispatch layer (MySQL leases, outbox + RabbitMQ, Redis admission) - #3
Merged
Merged
Conversation
…kens Adds an isolated distributed subsystem under src/distributed/ that claims task runs from MySQL 8.0+ with FOR UPDATE SKIP LOCKED, keeps them alive with heartbeats, and fences every worker write on (status, lease_owner, lease_token) so a worker that lost its lease cannot overwrite a takeover worker's result. - config: env parsing with heartbeatMs <= leaseMs/3 enforced - migrations: versioned SQL, GET_LOCK-serialised, refuses newer schemas - repository: submit/claim/heartbeat/complete/fail/reap/cancel, all SQL parameterised and every state change atomic with its event row - worker/reaper: abort on lost lease, bounded graceful shutdown, heartbeat budget so transient DB errors do not abandon a healthy run - cli-worker: SIGTERM/SIGINT shutdown for standalone processes - chaos: 500 runs, 10 SIGKILLs with replacements, one SIGSTOP zombie, SQL assertions and a JSON report with queue/takeover latencies - tests: real-MySQL integration suite, skipped when DIST_MYSQL_TEST_URL is unset so make test never requires a database Not wired into task-scheduler or the main service.
… stop - reapExpired no longer rewrites available_at, so a recovered run keeps its original queue position; the reaped event now records the lease expiry it replaced (used for lease-expiry-detection metrics). - Every pooled connection pins `time_zone = '+00:00'`, so a server whose default zone is not UTC cannot shift stored timestamps. Covered by a test that sets the global zone to +08:00 and reads started_at back. - Migrations moved next to the subsystem (src/distributed/migrations). - New repository.release(): hands a claimed run back to the queue with attempt - 1, token + 1 and a `released` event. Worker.stop() releases runs it cannot finish before aborting, so a graceful shutdown no longer burns an attempt. - Worker config gains DIST_CLAIM_BATCH (default = concurrency) and reports claim latency + deadlock retries on shutdown for the new benchmark. - Chaos: 6 workers x 4, sleep 100-1000ms, maxAttempts 3; asserts zombie stale complete/fail writes and that the zombie never succeeded with a stale token; reports leaseExpiryDetectionMs and crashRecoveryMs instead of takeover latency. - New scripts/dist-bench.mjs (`npm run dist:bench`) measures claim-path throughput for 1/2/4/8 workers plus an 8-worker claim-batch=1 control.
Phase 2 adds an at-least-once wake-up path while MySQL stays the only authority on execution rights. - 002_outbox.sql: dist_outbox + dist_consumed_messages - every state change that makes a run executable (submit, fail->retry_wait, reap->retry_wait, release) inserts its outbox row in the same transaction, copying available_at from the run so a backoff can never be published early - relay: SKIP LOCKED batch -> ConfirmChannel publish -> waitForConfirms -> mark published -> COMMIT; failure bumps attempts/last_error and rolls back; fault injection for crash-after-confirm and duplicate publish - topology: quorum main queue, classic TTL retry queues that dead-letter back to the exchange, DLQ; DLQ list/replay CLI - worker: DIST_DISPATCH=poll|mq; mq consumes run.ready, claims in MySQL under the same fencing rules (claimById + dist_consumed_messages), acks right after the claim commits, and keeps a slow fallback poll for lost messages - claim/reap now run under READ COMMITTED (SET TRANSACTION before BEGIN) - new scripts: dist:relay, dist:dlq, dist:chaos:mq, dist:bench:dispatch - dist-bench gained DIST_BENCH_ROUNDS / DIST_BENCH_REPEAT (median of N) Not wired into task-scheduler or the main service.
The purge ran at 8s while the broker was already restarting, so it never happened, and the restart landed after the work had drained, leaving the recovery metric empty. Purge at 3s, restart at 7s, and report commandMs.
Redis now gates admission; MySQL stays the only authority on run state. - tenant-slots: one ZSET per tenant (member=runId, score=lease expiry). Acquire removes expired members before the capacity check, so a crashed worker's slots are reclaimed by the next acquirer (no janitor, no exit hook). Renew refuses to resurrect an expired slot; release is best effort. - worker: poll and mq paths acquire a slot before executing; a rejection returns the run to the queue without consuming an attempt and with a short re-queue delay (otherwise the throttled tenant keeps the global queue head and starves everyone else — found by dist:bench:fairness). Heartbeats renew the slot; a missing slot aborts like a lost lease. A worker that already lost its lease never frees the slot (the new owner may hold it). - rate-limiter: Lua token bucket with injectable clock, tryAcquire + acquireBlocking; documented call site in src/distributed/README.md. - DIST_TENANT_CONCURRENCY (0 = disabled), DIST_REDIS_URL, DIST_ADMISSION_FAIL_MODE=open|closed (default open, logs admission_degraded). - scripts: dist:chaos:tenant (3x200 runs, 6 workers, 5 SIGKILLs, slot-leak assertions) and dist:bench:fairness (BIG 5000 vs SMALL 200, baseline vs isolated, 3 repeats). - compose: redis:7-alpine with healthcheck.
… claiming
- tenant-slots/rate-limiter: all Lua scripts take the clock from Redis TIME;
clients no longer pass nowMs, so worker clock skew cannot change capacity.
- slot members are `${runId}:${leaseToken}`: release only removes its own
generation, and an acquire from a newer token deletes the older one instead
of counting the run twice.
- claimFair(): candidates are grouped by tenant (unlocked read), then claimed
per tenant within one transaction, skipping tenants Redis reports as
saturated. Workers compute that view from a single Lua snapshot over
dist:slots:active. DIST_TENANT_CONCURRENCY=0 keeps the old claim path.
- migration 003 adds idx_tenant_due (status, tenant_id, available_at).
- release() records reason=admission|shutdown, so admission churn is countable
in SQL (admissionRejects/wastedClaims).
- chaos-tenant: asserts claimedEvents/runs <= 1.5 and reports the churn.
- bench-fairness: enforces and records BIG-then-SMALL submission order, runs
baseline / isolated-10 / isolated-32, and reports SMALL p50-p99, spans,
cluster utilization (100ms sampling) and wasted claims.
Phase 4 makes a run replayable at step granularity. - 004 migration: needs_review run status + dist_run_steps ledger (intent/committed/unknown/failed, idempotency_key, request_hash, provider_ref) - durable.ts: RunContext with step()/effect(); every ledger write is fenced on (status='running', lease_owner, lease_token) and a lost fence raises LeaseLostError before anything is written - effect protocol: intent -> committed | unknown -> (lookup) -> committed | needs_review; timeouts/5xx mark unknown and retry, 4xx marks failed and the run goes dead, request changes are rejected - worker: per-run DurableContext, NeedsReviewError never overwrites the run, per-worker durable counters; reaper scans stale unknown steps (monitoring) - HttpEffectProvider + scripts/mock-effect-provider.mjs (node:http) with idempotency/drop-response/error/latency knobs - scripts/dist-chaos-effects.mjs: naive vs durable vs durable-nolookup - README documents the state machine and the fencing limitation (a fence cannot stop an outbound request; only provider idempotency can)
…sification - 005: dist_run_steps.send_started_at, written (fenced) immediately before the provider call together with attempts+1; attempts now counts sends only. - Rule: `intent` with send_started_at IS NULL means the request provably never left, so the retry sends directly (no lookup, no needs_review). This covers a worker killed between the intent write and the send. - Reconcile misses now wait: while NOW(3)-send_started_at < effectTimeoutMs + DIST_EFFECT_SETTLE_MS the retry is deferred (reconcileDeferred) with available_at pushed past the window (fail gained minDelayMs) instead of sending again. - EffectNotAppliedError: connection refused / DNS failure and provider-declared notAppliedStatuses (default 429/503) revert the step to intent with a cleared send_started_at and retry with normal backoff; timeouts, post-send resets and 500/other 5xx stay `unknown`. - needs_review events carry a reason (unknown_outcome / send_started_unknown_outcome) so the report can split them. - chaos: LATENCY_MS=300, 5 repeats; report adds reconcileDeferred, notAppliedRetries and needsReviewByReason.
… migration versions
Behind DIST_SCHEDULER=1 (default off, legacy pump untouched when unset). - SQLite task_runs stays the result/history/notification authority; the dist run is only a dispatch envelope (idempotency_key = task_run.id, tenant_id = workspace owner, max_attempts = 20 dispatch attempts). - pump stops claiming and wakes a forwarder instead; the forwarder submits queued/retry_wait runs with the SQLite available_at (outbox copies it). - embedded in-process worker + ScheduledTaskExecutor: every claim re-reads SQLite and mirrors terminal / not-due / running-elsewhere / claimable; after execution it re-reads again and mirrors success/dead or releases. - claimTaskRunById mirrors claimNextTaskRun exactly, only fixed on id. - executeClaimedTaskRun now returns Promise<void> (all settle paths resolve). - crash semantics unchanged: started runs are never replayed, the dist mirror follows whatever failExpiredStartedTaskRuns wrote. - MySQL lease fences dispatch only; the SQLite lease fences execution, so a lost dist lease never aborts a running legacy task. - host tasks skip tenant slots (CLAUDE.md §4); container tasks keep them. - scripts: dist:e2e:scheduler (latency + kill -9 recovery) and dist:smoke:agent (manual, spends real model calls, not run).
… is down - submit() accepts an explicit id; the forwarder passes the SQLite task_run.id so the cancel hook (which only has the run id) addresses the right envelope instead of silently doing nothing. - the bridge tracks dispatch health: 3 consecutive forwarder failures or a 10s outage uninstall the hooks so the pump falls back to claimNextTaskRun, logging scheduler_dist_degraded; a 5s SELECT 1 probe reinstalls them after two successes and logs scheduler_dist_recovered. Safe because both paths claim the same SQLite row with the same conditions and lease fencing, so a run can never execute twice. - pumpTaskScheduler is exported and getDistSchedulerHooks() added so the fallback is testable without driving timers. - dist:e2e:scheduler gains a mysql-outage round: 100 tasks spread over 5-40s, mysqladmin shutdown for 15s, restart, three-phase start latency and a once-only side effect check. - tests: cancel of a running and of a queued run, and degrade/fallback/recover.
Behind DIST_IM_OWNERSHIP=1 (default off: the manager behaves exactly as before).
- ConnectionOwnership: `dist:im:owner:{credentialKey}` holds
`{instanceId}:{token}` with PX leaseMs, `dist:im:fence:{credentialKey}` is a
monotonic token. acquire/renew/release are single Lua scripts, so a stale
owner can neither extend nor delete the new owner's key. The credential key
reuses claimCredential's provider+sha256 fingerprint; no secret reaches Redis.
- local self-fencing uses the monotonic clock (leaseMs - renewMs window), so a
process that was frozen notices the expired lease before handling anything.
- connectChannel acquires after the local credential claim and before
channel.connect; a loser stays standby, keeps the channel object and retries
every renewMs. onLost tears down through the existing
per-(userId, channelType) lock and returns to standby.
- inbound is gated per message (frozen processes get queued events first):
drop + inboundDroppedNotOwner. outbound (sendMessage/sendFile) throws
ConnectionOwnershipLostError (a DefinitiveChannelDeliveryError, i.e. "the
provider never saw it") so the outbox retries instead of freezing the row.
- normal disconnect/shutdown release the lease so a standby takes over without
waiting for expiry; Redis outages fence everyone off within one lease window.
- scripts/dist-chaos-im.mjs + dist-chaos-im-worker.mjs: worst-case mock
platform (every message pushed to every connection), 3 instances, SIGKILL /
SIGSTOP / SIGTERM injections, overlap and duplicate-reply accounting.
…owner - IMConnectionManager.setChannelRebuilder(): a lost lease now rebuilds a fresh transport (index.ts resolves the channel account and calls the same create*Channel factories as reconnect) and retries connectChannel. Reusing the disconnected object was the availability bug: the process dropped the connection and never came back. A null rebuild (account deleted/disabled) clears the standby state instead of retrying forever. - sendImage now shares the outbound ownership gate with sendMessage/sendFile; streaming cards and setTyping stay ungated (documented in the README). - ConnectionOwnership bounds every Redis round-trip: with an offline queue a renew can hang forever and the self-fence would never fire. - chaos: scenarios run per platform mode (push / buffered with offset acks) and a 2-instance freeze-handover case; injection targets come from the actual Redis lease holders (10s wait, hard failure if none), SIGSTOP assertions, Redis outage via `docker stop commerceagent-dist-redis`, and lost replies bucketed by phase.
- docker-compose.distributed.yml: host ports are configurable (DIST_MYSQL_HOST_PORT / DIST_AMQP_HOST_PORT / DIST_REDIS_HOST_PORT) so the stack can coexist with the local dev MySQL and Langfuse's Redis. Verified on MySQL 8.4 / RabbitMQ 4 / Redis 7 with `docker compose up -d` and healthy healthchecks; tests/distributed and the four chaos scripts pass there. - dist-chaos-mq: a restarted broker may not have re-declared the DLQ yet; a 404 closed the admin channel and crashed the run. Channels now carry error handlers and the DLQ check uses a throwaway channel. - scheduler-bridge: startForwardingOnly() lets tests install the hooks without the embedded worker racing the SQLite run they are inspecting. - scripts/dist-demo.mjs (npm run dist:demo): compose up -> migrate -> 3 workers + 2 relays -> 300 runs -> kill -9 a worker + RabbitMQ restart -> summary table; --down also stops the stack. - dist-chaos-im: push rate is configurable (IM_CHAOS_RATE_PER_SEC), publishing waits for the fleet to be connected, backlog drains before losses are counted, capacity probing (IM_CHAOS_PROBE=1) and per-credential overlap. - README: subsystem overview with a mermaid diagram, per-phase invariants, a fault -> behaviour -> verification table and the boundary list; numbers are referenced by report name only.
…mo recovery - chaos buffered mode is now a real queue: the platform keeps an `ackedSeq` per credential and a new connection resumes there, so messages a killed owner never acknowledged wait for the next owner. The worker acknowledges only after the reply request succeeded. Result: 0 lost across five SIGKILLs; the 102 duplicates are exactly the "replied but killed before acking" at-least-once case and are counted/asserted as cross-instance re-delivery (production needs an idempotency key on the reply side). - zombie injection picks the instance that both holds leases *and* has live platform sockets, so the freeze actually has something to observe. - createRedis caps the reconnect backoff at 500ms; the IM outage recovery bound is documented as leaseMs + renewMs + 500ms. - dist:demo runs 1-3s tasks, kills the worker holding the most running runs, restarts RabbitMQ while <50% is done, and reports per-run recovery (p50/max), reaped count and claims after the restart; it fails loudly when any of those invariants break.
…onnect - chaos push mode is a real stream: /connect returns a never-ending chunked response and the worker reads JSON lines, so a frozen process keeps its sockets open (zombie overlap is now observable). buffered mode keeps the offset-acknowledged long poll. - ConnectionOwnership records a diagnostic timeline (redis_ready, acquire_start/end, rebuild_start/end) and the chaos report turns it into per-stage timings relative to the Redis PING; createRedis bounds the connect attempt at 1s and the backoff at 500ms. - RabbitMQ 4 quorum queues reject x-max-priority (406), so recovered work gets its own quorum lane `dist.runs.ready.hp`: outbox rows for reap/release are `run.recovered`, the relay routes them there, and the consumer subscribes to the priority lane before the backlog. Test: 200 queued messages + 1 recovery -> claimed within the first 20 deliveries. - compose redis runs with --appendonly yes (fencing tokens must stay monotonic); README documents the recovery bound and the token-vs-instance caveat.
- root cause of the push-mode reply regression: the streaming fan-out was implemented only in the HTTP /publish endpoint, but the driver publishes in-process (state.publish), so streamWrites stayed 0 and the worker never received a line. Publishing now goes through one helper used by both paths; minimal scenario (1 credential, 1 instance, no injections) replies 884/884. - ownership timeline events are written as JSON lines on worker stdout and the driver collects them, so a truncated /stats buffer cannot hide the recovery stages. healthyAt is the first successful PING (25ms cadence) and recovery stays signed: a negative value is reported, never clamped. - steady-phase replies must exceed half of steady publishes (a scenario with zero replies can no longer pass "0 duplicates"). - README: recovery bound formula with the measured stages report pointer; the priority lane is documented as a bounded wait, not strict priority.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概述
新增一套可选的分布式派发层,用于多 Worker 场景。两个开关(
DIST_SCHEDULER、DIST_IM_OWNERSHIP)默认关闭,关闭时行为与未引入该层完全一致;MySQL 是运行状态的唯一权威,Redis 只做门控,RabbitMQ 只负责唤醒。分阶段内容
FOR UPDATE SKIP LOCKED认领、lease_token递增围栏、心跳续租与 reaper 回收。claimFair。intent → committed/unknown → needs_review,发送前写send_started_at,结算窗口后判定,不静默重放。dist:im:owner:{credential}租约 + fencing token,入站/出站逐条门控,丢租约后重新进入备用。npm run dist:demo、被回收任务的优先级通道。对主服务的改动
只涉及
src/db.ts、src/im-manager.ts、src/index.ts、src/task-scheduler.ts,且全部在开关之后(新增导出函数与钩子,不改既有行为)。新增依赖
mysql2、amqplib、ioredis、@types/amqplib。如何验证
已知边界
serial_key租约:同一会话内任务的顺序仍由单机GroupQueue保证。测试结果
make test与合并前基线一致:3190 通过 / 7 失败 / 78 跳过,7 个失败全部是既有的tests/container-session-permissions.test.ts环境问题(/proc/self/fd在当前 Docker 镜像下的行为)。