Recovery hangs if a worker dies while a recovery is running #46

Problem

If a worker dies while a recovery is already running, nothing detects it and the recovery hangs forever.

The heartbeat monitor runs the recovery itself (heartbeat_monitor_coroutine → _perform_recovery → wait_cluster_healthy). While it waits for every participant to report ReadyAfterRecovery, it doesn't check heartbeats. The coordinator still counts a worker that died after the recovery started as a participant, sends it InitRecovery, and waits for its ReadyAfterRecovery, which never comes. Its heartbeat process is gone too, but nothing is looking.

The cluster stays stopped: no epochs, no snapshots, and no error is logged.

How it was found

While reproducing a CI failure of PR #35 locally, an injected worker exit at the migration Phase B stop and the FT YCSB client's own docker kill styx-worker-1 hit two different workers about 10 ms apart. The coordinator detected the first death, started recovery with 3 participants, and waited for the second dead worker indefinitely.

From the coordinator log: only one failed to register a heartbeat, then [RECOVERY] Sending InitRecovery to 3 workers, and no Recovery process completed. The two live workers stopped at Egress started; the dead one never logged a recovery line.

This doesn't come from PR #35's changes. Any second failure inside the recovery window triggers it: a correlated crash, or a node hosting two workers.

Possible direction

• While waiting for the cluster to become healthy, keep checking heartbeats of the participating workers (for example, wait on the readiness events with a timeout, and run worker_pool.check_heartbeats on each tick).
• If a participant dies, abort this recovery and start a new one with that worker removed and its partitions rescheduled.
• The difficulty: _perform_recovery has to be safe to abort partway and restart. It reverts or keeps migration state, resets protocol metadata and sends InitRecovery, and a restarted recovery must not double-apply any of it. A new incarnation number per attempt already makes workers drop messages from the aborted attempt.
• A cheaper stopgap is a timeout on wait_cluster_healthy that logs which workers never reported. That makes the hang visible but doesn't fix it.