Repository navigation
S26 E3: faster multi-island weight sync (pipelined sweeps, WAN stream robustness) - #284
Closed
michaellchung wants to merge 9 commits into
Closed
michaellchung wants to merge 9 commits into
michaellchung wants to merge 9 commits into
Conversation
Allow --pipeline 1..P with --policy-sweep-fragments so fragment k+1 uploads while fragment k merges, checkpoints and broadcasts; commits stay in strict step order. Add --async-checkpoint (policy sweep only): the committed state is snapshotted and written on a blocking task, and the round is broadcast only after the checkpoint is durable. Checkpoint bytes, versions and the semantic profile hash are unchanged; the tape gains checkpoint_ms. Launcher flags --rl-sync-fragment-pipeline and --rl-sync-async-checkpoint (default off). Adds bit-equality, failure and real-syncer tests and a CPU micro-benchmark (tools/bench_sync_boundary.py). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add session dtype 4 (f32 session, bf16 PUSH deltas): INIT/BCAST/FINAL, masters and momentum stay f32; the client rounds f32 delta bytes to bf16 (round-to-nearest-even, streamed across part boundaries). The dtype is bound into the role-stream session contract. Launcher --rl-sync-push-dtype, island --sweep-push-dtype (default f32). Allow --checkpoint-every N (1 or a multiple of the fragment count) for policy sweeps: only checkpointed steps are durable before their broadcast, on both the synchronous and async paths; the interval is part of the semantic profile hash. Launcher --rl-sync-checkpoint-every (default 1). E3 launcher options are refused outside the full-parameter preset. Adds bf16 error-bound, interval/resume and real-syncer tests, and a third micro-benchmark profile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add session dtype 5 (f32 session, block-scaled int8 PUSH deltas): u32 block size, then per block an f32 absmax/127 scale and one i8 per value; error is at most absmax/254 per element. Launcher --rl-sync-push-int8-block (default 128) is bound into the role-stream contract. Add session flag 0x80 (bf16 downlink deltas) for f32-bulk sessions. Syncer and learners hold a bit-identical f32 anchor per fragment; BCAST is FULL f32 or bf16(theta - anchor), with a FULL refresh every N versions (--bcast-delta-full-every), FULL re-sends on HELLO/reconnect, and a SHA-256 check of the anchor after every update. A mismatch drops the connection and the reconnect restores the anchor. SyncerClient decodes before adapters see the payload: the Miles streaming sink is wrapped, the strict bridge (verl and Miles LoRA) gets f32 bytes. Masters, checkpoints and FINAL stay exact. Add --rl-sync-overlap for the full-parameter sweep: an island pushes round k, trains round k+1 at once, and at the next boundary rebases in place, theta <- G_k + (local - base), with G_k as the anchor so no delta is counted twice. The launcher raises --rl-max-policy-age to at least 1 before its checks; the island refuses overlap below that age. Adds Rust and Python codec/anchor tests, real-syncer int8/downlink/resync tests, a strict-bridge downlink test, and coordinator overlap tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sweep Pre-GPU check of --rl-sync-overlap on the Miles role stream found three problems: - The launcher raised --rl-max-policy-age to 1, but the dense sweep island refuses any age other than 0, so an overlap run could never start; and an age >= 1 also turns on Miles' fully_async rollout (partial rollout when colocated). Overlap is not rollout staleness: the driver publishes the boundary's cut before the next generation, so rollouts stay on-policy and only the local weights lag the global policy by one outer step. Keep the age at 0 (refuse others at launch) and bind the lag into the session contract only. - After a rebase the applied cut has local step generation 0, and Miles pushes only cuts with progress, so the second sweep's push failed. The submission is now the rebased cut relabeled with the rounds trained since the round base (same values, hashes and refs). - The boundary submitted once; a PULL that arrived later waited a whole local round. The overlap boundary now submits the whole sweep before training on, and the launcher requires --rl-sync-fragment-pipeline equal to --fragments so the syncer sends all PULLs at once. Adds Miles role-stream tests (real adapter and role stream, CPU tensors, fake trainer/Ray/syncer): exact-once rounds through rebase_fragment, and syncer restarts while the overlapped round trains (before commit, after commit with a lost broadcast, partly committed sweep). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The overlap rebase uses the bf16-tracked anchor and pushes theta - anchor against that same anchor, so the syncer's f32 masters still receive every round's progress exactly once and no DownlinkResync happens. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With bf16 downlink deltas, the last BCAST of a fragment can carry the same
version as its FINAL. Islands store one copy per (fragment, version) and
refuse a second copy with other bytes, so the delta-tracked BCAST and the
exact FINAL collided ('authoritative fragment replay changed bytes'), the
islands redialed in a loop and finalization timed out (s26-e3-p06-D8).
Broadcasts of a fragment's last version now always go out FULL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…stics E3 showed sweeps waiting on the slowest island's upload and one C32 run where a stream stopped moving with every socket still open. Island-side SyncerClient changes (all default off): - --wan-stall-timeout S: a watchdog fails the connection group when an outbound stream holds bytes, or an inbound chunked message stops growing, for S seconds; sockets also get TCP_USER_TIMEOUT=S. Recovery uses the existing redial path: the syncer replays its PULLs and the island pushes the fragment again at the same (global_step, round_attempt). - --wan-stream-schedule adaptive: each 4 MiB chunk goes to the data stream with the earliest estimated finish (backlog / EWMA rate) instead of round robin, so slow or stalled streams carry less. No wire or syncer change. - rl_full_parameter_round now records per-stream upload rates, the wait after this island's upload drained, and stall counts; scalars go to W&B on rl/step. A real-syncer test stalls one stream mid-sweep (rr and adaptive) and checks the redial keeps versions and finals bit for bit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…elaunch s26-e3b-p06-C32a: learner 0's connection group died before the syncer applied its INIT, the client never sent INIT again, and no island got the initial cut. s26-e3b runs also showed that a slow island is slow on every stream (container level), which no stream scheduling fixes. - SyncerClient.send_initial_fragments/init_replay_due/ mark_initial_cut_received: learner 0 replays INIT on every new connection group until the initial cut arrives (the syncer ignores INIT it already applied). TCP_USER_TIMEOUT is set only after the initial cut, so the multi-GB INIT and first broadcast are never reset by the kernel. - MSG_BW_PROBE (26): an HMAC-sealed probe connection to the legacy syncer uploads N bytes, gets an ack, downloads M bytes; never part of a session, at most 64 concurrent, 1 GiB per direction, 300 s. - --wan-probe-min-mbps / --wan-probe-min-down-mbps (default off): Miles islands probe the actor syncer before loading the model, record rl_wan_probe, and exit 75 below the minimum. The fleet controller requests a new container up to --wan-probe-relaunches times (default 2), also under --no-island-relaunch, since the island never joined the session. Real-syncer test: INIT interrupted between fragments is replayed and the finals match an undisturbed run bit for bit; the probe works on the same port and a wrong key is refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t mid-INIT Only the legacy syncer serves MSG_BW_PROBE. With --rl-elastic every probe would fail, every island would be judged slow and relaunched until its budget ran out, so the launch now refuses the probe flags there. Tests: - real syncer: one data socket of learner 0's group is shut down between the INIT of fragment 0 and fragment 1; the client replaces the group, replays INIT, and the final versions and bytes equal an undisturbed run. - slow-container decision: threshold equality accepts, upload just below rejects, slow download rejects, a zero download minimum is not checked, and a stream error rejects a fast link. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Closing: all commits of |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Speeds up multi-island weight sync (S26 E3):
Defaults
All new options are off by default (stall timeout, adaptive striping, link probe, overlap, int8/bf16 pushes).
Branch dependencies
Based on s25-final. The range s25-final..HEAD holds only the 9 E3 commits. No other branch is included.
Test evidence
GPU evidence
Qwen3-0.6B, 2 islands, Modal H100, C config (8 streams).
Not verified
🤖 Generated with Claude Code