[michaellchung]

Summary

This PR moves multi-island RL to one syncer service. The syncer serves LoRA runs and full-parameter runs. The base stays s26-e11-cluster (#288).

This update moves the head from 869078a5 to 718b2f14. The new head adds these parts:

1. Faster asynchronous full-parameter rounds: background pushes, a per-fragment boundary, staggered first pushes and an ACK-only final cut.
2. Full-parameter sync at data parallel (DP) > 1.
3. Engines on a multi-node island, with the Miles NCCL weight update to the engine nodes.
4. An automatic per-island data split, data positions and one tape event per round with the trained data rows.
5. The language-only bridge for the Nemotron-H Super VL checkpoint.
6. --rl-export-hf-rounds, which writes an island's model as HF directories, and a fix for its upload to a Modal Volume.
7. Ten branches from other open draft PRs (S26-E0.5: remove the self-written Codex harness #280, S26-E9: robustness fixes (preemption marker, verl cuts, data slots, one-command verl pairs) #282, S26-E7: reject verl with expert parallelism; require fsdp2 strategy #283, S26 E2: async buffer depth defaults to the policy-age limit; opt-in adaptive TIS #285, S26 E0.3: bias-stacking ablation tooling for Qwen3-0.6B on Miles #286, S26 E1: single-island overhead (background round cuts, publish fast path, profilers) #287, S26 E4/E5: phase timing, analysis tools, release writer Volume optimization #289, S26 E6: FP8 quantized rollout and full-parameter dedicated-engine island #290, Move fork and image references from michaellchung to agentenv #291).

A 2-island asynchronous full-parameter run of Nemotron-3 Nano 30B-A3B ran to the end on 32 x H200 at 33cce368. Its first GPU checks of the new mechanisms passed. One HF export of the 1-island control run failed (see "GPU validation"). Held-out pass@1 went from 0.521 (base) to 0.739 (merged global model). 718b2f14 adds only the HF export fix. That fix has CPU tests only.

What the new head merges

The first-parent history from 869078a5 to 718b2f14 has these merges and commits.

Branches from other open draft PRs. Where a branch conflicts with this PR, the merge keeps this PR's behavior.

Branch	PR	What it brings
s26-e04-imgassert	none	Harbor layer version lock and a start check.
s26-e7-verl-megatron	#283	verl refuses EP > 1 and strategies other than FSDP2.
s26-e45	#289	Release phase timings. A release writer that keeps its own LATEST and commits once. An optional LoRA copy check (--rl-sampler-fused-stage). Scale report tools.
s26-e1-overhead	#287	Merge record only. The tree already had this work. The default cut interval did not change to 4.
s26-e6-quant	#290	Rollout quantization tiers (--rl-rollout-quant-tier, default 0 = BF16), an FP8 GPU allow list and tier launch checks. This PR keeps its own Volume writer guard, pins and single-island full-parameter rule.
s26-e03-ablation	#286	Ablation drivers and task files (tools and data only).
s26-e2-async	#285	--rl-async-buffer-rounds. With overlapped generation the buffer depth defaults to --rl-max-policy-age. An optional adaptive TIS.
s26-e9-robust	#282	A rank > 0 preemption signal goes to rank 0. verl fully_async cuts save only optimizer-held parameters. Sampler data slots. FINISHED on an exhausted buffer. The launcher follows rescheduled Modal containers. One verl launch runs trainer and sampler islands. All roster checks count the syncer's learners.
s26-agentic-merge	none (contains #280)	Removes the self-written Codex harness. Adds the Harbor agentic path (suspend between rounds, curriculum, compaction), a boat reuse pool, Codex through the yeto model provider and verl fully_async agentic runs.
s26-agentenv-migrate	#291	Only the origin compatibility of the pins (empty former list, no behavior change). Fork and image references still point at michaellchung.

Feature branches and direct commits:

Branch or commit	What it brings
s27-sync-overlap (922f3880)	Background pushes, zero-copy full-parameter chunks, a faster int4 encoder.
s27-task-assign (f9318d15)	Automatic per-island row split, data positions, tools/eval/data_split_check.py.
s27-dp-sync (d93c97fe)	Full-parameter sync at DP > 1. Nemotron-H training defaults.
s27-120b-gaw with s27-120b-prep (b95e1f96)	Nemotron-H Super VL bridge wiring, a latent MoE check, multi-node full-parameter islands with dedicated engines, an offline Nemotron-H layout tool, a --moe-layer-freq mask without spaces.
s27-disagg-fast (4dc9acac, with 4c702a55 reverted)	Staged background releases, narrow integer buffer payloads, parallel prefetch, a Modal port range fix, no relaunch into a stopped app, a timed default cut interval. The revert keeps sampler islands on data slots only.
s27-sync-boundary (0c4750a2)	Per-fragment boundary, staggered first pushes, ACK-only final cut.
3b0bafc6	The offline layout tool supports DP replica owners (--dp).
63afafaf	The rl_round_data_rows tape event.
76f067d3, 72e9eef4	Miles and image pins (see "Pins and image").
fadbffc3, 33cce368	--rl-export-hf-rounds. The warning goes to stderr, so the --dry-run JSON stays valid.
718b2f14	HF export fix: the first export of each round to a Modal Volume no longer fails.

Behavior changes and new options

Asynchronous full-parameter rounds

• Background push. A worker thread prepares and sends each planned push while the island trains. The island applies broadcasts once per boundary. This path runs when every role encodes ahead (no int4 error feedback, no stochastic rounding). YETO_FULL_PARAMETER_BACKGROUND_PUSH=0 keeps the synchronous push. YETO_FULL_PARAMETER_PUSH_WORKERS (default 3) sets the number of workers.
• Zero-copy chunks. With Miles 8edea35a7, full-parameter chunks go through Ray as numpy buffers (one copy on put, mapped on get). Before, Ray pickled CPU tensors at about 0.30 GB/s per thread.
• int4 encoder. The encoder works in cache-sized row batches and splits large inputs over YETO_PUSH_ENCODE_THREADS threads (default 4). The bytes are identical to the old encoder. For a 0.5B-value fragment the encoder took 4.3 s before and 0.65 s after (CPU).
• Per-fragment boundary. When the Miles transport reports the lazy_export feature, a local round reads no whole master. The island hashes, NaN-checks and model-checks each pushed fragment when it materializes the fragment. Miles commits per parameter (partial commit). Every 16th round still checks the whole model (YETO_FULL_PARAMETER_FULL_CHECK_EVERY, 0 = never). YETO_FULL_PARAMETER_LAZY_CAPTURE=0 turns the lazy path off.
• Staggered first pushes. At stream start every fragment has the same anchor. The first push of fragment f now waits f * H // P more rounds. Later pushes keep the H gate. YETO_FULL_PARAMETER_STAGGER_FIRST_PUSH=0 turns this off.
• ACK-only final cut. New syncer option --final-cut-payload {full,manifest}. New launcher option --rl-sync-final-cut {auto,ack,download} (full-parameter preset only).
  • auto gives ack in --rl-sync-mode async without --rl-eval-interval. It gives download in all other cases. The strict mode keeps download, so its commands do not change.
  • With ack, the syncer sends only the final manifest. Each island drains its pushes and sends its final ACK. No island downloads the f32 final cut (126 GB for a 30B model).
  • The merged final parameters stay in the syncer's final checkpoint. Each island keeps its own local model.
  • The option is part of the semantic profile hash only when it is on. Existing hashes do not change.

Full-parameter sync at DP > 1

• Miles 921cde8c7 gives each parameter one owner rank per replica group. The replicas apply the owner's checked bytes. The layout lists each parameter once.
• yeto accepts DP > 1 only when the pinned Miles lineage has 921cde8c7.
• At DP > 1 the island keeps the non-distributed optimizer (whole FP32 masters on every replica). It refuses --rl-optimizer-offload-fraction.
• A sampler island builds no trainer, so it skips this check.
• New Nemotron-H training defaults: selective recompute, 8192 x TP tokens per GPU, and the flex dispatcher with DeepEP when DeepEP is present.
• tools/nemotron_h_offline_layout.py --dp N computes the layout hash before launch (scope: TP 1, EP = DP). Nano at TP1/EP8/DP8 gives 64 fragments and layout c2c3cb49…. An island on GPU accepted this hash.

Engines on a multi-node island

• A full-parameter island with dedicated engines may span several nodes (--rl-island-gpus-per-node). The launcher lays out the trainer and the engines over the nodes, as for a LoRA fixed partition.
• The trainer updates the engines with the Miles NCCL weight update. The engines can be on other containers of the same island. The weights do not go through a store.
• In a Modal container the island sets ip_local_port_range to 32768-60999 when the range starts lower. Before, an ephemeral port could take a port that Miles picks for an engine, a router or the trainer master. Then one engine of several failed with "address already in use".
• One island uses one GPU type. Trainer nodes and engine nodes must use the same card.

Data split and data rows

• --rl-data-split auto (new default, for N >= 2 training islands). Island k of N reads the rows i with i % N == k. The launcher sends --rl-data-slot k/N. --rl-island-tasks overrides the split. shared keeps the old behavior and prints a warning. External learners fall back to shared with the same warning.

• Passes. Pass 0 reads the rows in data order. Miles reshuffles each later pass with a seed from --seed and the pass number.

• Data positions. A synced full-parameter island with --rl-checkpoint-store gets --rl-data-position-store. It records its data cursor after each sync boundary, one file per record. A process that joins a running pool continues after the last record. A refused write goes on the tape, and the island trains on.

• rl_round_data_rows event. The driver tapes one event per island per round when --rl-observe-timeline is on and the island reads a row slot. Fields: rollout_id, policy_version, samples, row_count, rows (arithmetic runs [[first, last, step], ...]). verl does not report the rows. Then the driver writes no event.

• tools/eval/data_split_check.py. The tool reads the island tapes of a run. It checks these conditions:

  1. The islands hold distinct slots.
  2. No two islands train the same row.
  3. Each trained row lies in its island's slot.
  4. A relaunched island restored its data position.

  The tool prefers rl_round_data_rows and falls back to rl_trajectory_reward for older tapes.

Nemotron-H Super VL

• yeto reads a NemotronH_Omni_Reasoning_V3 VL config as its language model. The FLOPs count only the language model. The frozen vision names now include vision_projector.* and sound_model.*.
• For a VL checkpoint or a checkpoint with MTP, the island loads the Miles bridge plugins before AutoBridge. The bridge drops MTP (unless MILES_NEMOTRONH_KEEP_MTP is set). It reads a VL checkpoint as its language model and does not build the vision part.
• The island refuses an image whose Miles has no language-only VL bridge. The error names the Miles commit to use.
• Nemotron-3 Nano keeps the upstream provider.
• models.py registers nemotron35-super-vl (249 GB).

--rl-export-hf-rounds

• Syntax: --rl-export-hf-rounds R[,R...][,final]. Round R is the island's R-th trained round.
• final is the round whose boundary stops the island. On a synced full-parameter island, the export runs after the final manifest arrives and the pushes drain, before the final ACK. The syncer waits for the island within its final-ACK timeout. Without sync, final is the last round.
• Requirements: --rl-checkpoint-store, Miles, full parameter, sample role both. The launcher refuses verl, --rl-elastic and other sample roles. The island refuses LoRA. The launcher prints a warning on stderr for a round past --total-steps x --local-rl-rounds-per-sync.
• Layout: <store>/hf/island<k>/round-<r>/. Each complete directory has a .complete marker. The final export also writes <store>/hf/island<k>/final.json.
• Order: the ranks write into a local staging directory. The island puts the directory (one commit). Then it puts the marker and, for final, final.json (second commit). So a directory with the marker is committed.
• One rl_hf_export tape event per export. A failed export goes on the tape, and the island trains on.
• Fix in 718b2f14. A Modal Volume reports the removal of a missing path as InvalidError: No such file or directory. The store did not treat this as "missing". So the first export of each round failed at its upload, and the island removed the staging copy. Now:
  1. remove_tree raises only when the path is still there after the removal.
  2. The upload runs up to 3 times (waits of 10 s and 30 s). The tape event gets upload_attempts and upload_errors.
  3. When the last attempt fails, the staging copy stays with its marker (staging_kept). Each island process keeps at most 2 such copies (staging_dropped).

Trainer and sampler islands

• A trainer island exports each full-weight release into a local staging directory. Every rank writes a share. A background thread uploads the release while the next step trains. --rl-release-inline keeps the old inline path. YETO_RL_RELEASE_EXPORT=miles uses the Miles export.
• A Modal Volume store uploads a directory with Volume.batch_upload.
• An unsynced full-parameter trainer releases nothing for version 0. The samplers start from the base model.
• Buffer payloads store integer arrays in the narrowest type (R3 expert ids as uint8). The trainer prefetches in parallel.
• The default round-cut interval is now timed: prior MTBF 21600 s, with cuts capped at 5% of the wall time. --rl-cut-every N and --rl-cut-auto-mtbf-s replace the default as before. verl keeps one cut per round.

Island relaunch

• The default does not change. The launcher relaunches a failed island for --recover-timeout seconds (default 1200). --no-island-relaunch or --modal-retries 0 turns this off.
• New: when a relaunch fails with "App not found" and the app is stopped, the launcher abandons the island at once. Before, it retried until the recovery timeout ended.
• New: a relaunched synced full-parameter island with a checkpoint store continues at its next data row.

Pins and image

• Miles 59aea07f9 (michaellchung/miles, branch yeto/s27-closing). It adds to the previous pin d1616ba12: 8edea35a7 (zero-copy chunks), 921cde8c7 (DP replicas), d78bf39e4 (pinned CPU masters), b7868074c (language-only VL bridge) and 63cb372b3 (per-fragment boundary).
• SGLang 2fa880182 (unchanged).
• Image ghcr.io/michaellchung/yeto-miles-ports:59aea07-2fa8801@sha256:d54dd4458d08fb2c6012278ac48c285b89bb51b1e50584ad9a032fc18b6ef18a. The image check (scripts/check_miles_ports_image.sh) passes all 13 checks.
• 718b2f14 changes only yeto files. It needs no new image.

GPU validation of the new head

Closing run (code 33cce368, image 59aea07-2fa8801)

Model: Nemotron-3 Nano 30B-A3B. Backend: Miles, full parameter. Both runs started within one minute of each other on Modal H200.

Item	A (s27-bb-a1)	B (s27-bb-b1)
Islands	2. Each island has one 8 x H200 trainer container (TP1/EP8/DP8, no optimizer offload) and one 8 x H200 engine container (4 TP2 SGLang engines).	1, same island shape.
Sync	One syncer on an AWS head. Async, quorum 1, adaptive grace up to 300 s, max base lag 1, H = 8, pipeline 20, rda, tokens2-over-steps weights, int4 pushes, 64 fragments. 768 outer merges.	None (--rl-single-island-no-sync).
GPUs	32	16
Rounds	152 / 157 per island	105 (fixed in advance)
Training	32 prompts x 16 samples per island round. Responses up to 2048 tokens. Constant lr 1e-6. Rollout routing replay. numeric_answer reward. First 8192 rows of DAPO-Math-17k (A: split by i % 2).	Same.

Evaluation: 675 held-out problems (AIME24 29, AIME25 25, DAPO held-out 283, MATH-500 numeric answers 338). pass@1 uses n = 8 at temperature 1.0. Errors are paired bootstrap standard errors over problems. Two evaluations of the base model differ by -0.0004 ± 0.0068.

Model	Rounds per island	pass@1	Greedy
Base	0	0.521	0.643
A island-local, round 56	56	0.609 / 0.606	0.664 / 0.652
A global, merge 385 (mid-run syncer checkpoint)	78-82	0.639	0.686
B final	105	0.697	0.724
A island-local, final	152 / 157	0.729 / 0.727	0.735 / 0.759
A global, merge 768 (final syncer checkpoint)	152-157	0.739	0.756

• Every trained model is above the base by more than 2 standard errors.
• A global final minus B final: +0.042 ± 0.006. A trained more: 152/157 rounds per island against 105, and 158,208 samples against 53,760. A took 187.7 min to merge 768, and B took 129.6 min to round 105.
• A global mid minus B final: -0.058 ± 0.006. A's mid point came 107.1 min after start, and B's final point came 129.6 min after start.
• A global final minus A island-local final: +0.010 ± 0.005 and +0.012 ± 0.005.

Mechanisms that ran on GPU for the first time (run A):

Mechanism	Result
Per-fragment boundary	Steady median boundary 10.8 s and 10.4 s per round. Whole-model checks ran exactly every 16th round. 1176 of 1184 commits were partial, with 0 refusals.
Staggered first pushes	No first push came earlier than its planned round.
ACK-only final cut	The syncer wrote the final checkpoint at merge 768. Both islands sent the final ACK 184 s and 186 s later. The head job ended with rc = 0.
Mid-run syncer checkpoint (--rl-sync-checkpoint-every 385)	The write took 233 s. The next merge came 262 s after the previous merge. No stream stalled, no island reconnected and no island exited.
DP > 1 sync with in-island NCCL weight update	update_weights succeeded every round (153 and 158 calls per island). The engine weight_version matched the round number. The publish phase took a median 3.3-3.5 s per round.
Data split and rl_round_data_rows	data_split_check passed. All rows came from the new event. No row was trained by both islands.
--rl-export-hf-rounds	4 of 4 exports in A succeeded (63.2 GB and 23 files each, write 54-63 s, upload 124-136 s). This needed a workaround: placeholder directories on the Volume before the first export. B lost its round-56 export to the bug that 718b2f14 fixes.
Global model check	The step-385 and step-768 syncer checkpoints convert to HF with the base names, shapes and dtypes, and all values are finite. Teacher-forced logprobs, global against island-local final: mean abs diff 0.0112 and 0.0116, p99 0.179 and 0.188.

Health:

• No OOM, ENOSPC, NCCL error or Modal reschedule (exit 7) in A or B.
• Train/rollout logprob gap: A 0.007-0.011, B 0.008-0.012.
• Trainer GPU memory peak: A 139.7 GiB of 140 GiB, B 112.1 GiB.
• Health check: B had 0 FAIL. A had 1 FAIL and 1 WARN. The FAIL is one of 280 merge boundaries. At round 146, island 1 accuracy fell by 0.41 (0.72 to 0.31), more than the 0.35 limit. The logprob gap did not change at that boundary. We infer batch variance, not a merge fault.
• Merges: 710 of 768 had both islands. 46 late pushes went into later merges. The syncer rejected no push.
• Steady round wall time: A 63.9 s and 60.7 s, B 57.4 s. Actor MFU: A 11.5-11.7%, B 19.4%. B uses the distributed optimizer. A cannot use it at DP > 1.

Other GPU runs since 869078a5

• s27-ap-c tested the background push, zero-copy chunks and int4 encoder on 2 islands x 4 x H200 (Nano). The median sync per round was 63 s and 54 s. In s27-gr-j it was 403 s and 538 s. The 180 merges completed. Island 1 did not ACK in time while it downloaded the 126 GB final cut. This result led to the ACK-only final cut.
• s27-ba-a (closing shape with 2 engine containers per island, DP8, Nano): the island accepted the offline layout (64 fragments, c2c3cb49…). The DP replica checks had 0 errors. The NCCL weight update took a median 3.0 s. Health: 13 PASS, 0 WARN, 0 FAIL.
• s27-aw-b (Nemotron-3.5 Super VL 120B-A12B, 1 island of 5 x 8 H200, TP8/PP4/EP8, no sync, 2 steps): the island built the language-only bridge. The per-rank parameter counts matched the offline layout. All 28 update_weights_from_distributed calls returned 200. Logprob gap 0.0074 and 0.0095. Trainer GPU memory peak 84.7 GiB.

Testing

• 33cce368, remote full CPU suite without Miles: 7431 passed, 0 failed, 128 skipped, 25 deselected, 3 xfailed.
• 33cce368, cargo test: 226 passed.
• Local targeted tests at 63afafaf..33cce368:
  • files that use the pins: 1113 passed, 8 skipped.
  • data rows (85 files): 1698 passed, 1 skipped. 4 errors came from a missing cargo on PATH. With cargo on PATH they passed.
  • launcher, run flags, island entry, driver and HF export (230 files): 4332 passed. 2 failures came from new help and error texts. After the fix the affected files passed.
  • tests/test_rl_hf_export.py: 20 passed, 1 skipped (the skipped test passes with Miles on the path).
• With Miles 59aea07f9 on the path (25 files): 335 passed, 14 failed. All 14 failures have known causes:
  • 6 upstream parse tests need sglang_router.
  • 1 manifest probe test fails on its evidence path check.
  • 1 test needs ray.
  • 3 tests fail because the Miles tools package hides the yeto tools package.
  • 3 tests in test_neutral_rewards fail the same way at the merge base 6504d537.
• 718b2f14: 25 targeted files (Modal store, remove_tree, HF export, release store): 483 passed, 3 skipped, 0 failed. The full suite did not run on 718b2f14.
• After the ten merges from other draft PRs (467d1d91), the remote full suite had 7202 passed and 0 failed. On the same host, 869078a5 had 2 failures in tests that depend on timing.

Known limitations

Results:

• Each GPU result is one run with one seed.
• At about the same wall time, the single unsynced island (B final) scored higher than the 2-island global model (A mid). A was ahead only at its final point, with more rounds and samples. At an equal number of rounds per island, we infer that A is close to B, possibly 1-3 points lower. This is an interpolation, not a measurement.
• A two-island merge moved the weights about as far as a one-island merge (median norm ratio 1.07). We infer that the second island mostly lowers variance and does not double the progress per round.
• The syncer merged about 5 times per island round, not 8. Most pushes carried 10-11 local steps, not 8. We infer that the cause is the step order of the merges. We did not check it.
• 58 of 768 merges had one island. They fell at the start (grace not calibrated yet), during the round-56 HF export and after the mid-run checkpoint. 12 of them have no matching late push. The cause is not known.

Sync and layout:

• The syncer writes a checkpoint on its main loop (233 s mid-run, 248 s final). It does not merge during the write.
• DP > 1 full-parameter sync needs the non-distributed optimizer. In the closing run the actor step took 13.5 s at DP8, and 7.6 s on the unsynced island with the distributed optimizer.
• The trainer GPU memory peak in the closing shape is 139.7 of 140 GiB at 8192 tokens per GPU.
• The background push applies only to codecs that encode ahead (no int4 error feedback, no stochastic rounding).
• The lead cap (--rl-sync-max-local-rounds-ahead) counts the island's own unmerged pushes. With a deep pipeline this makes the rounds synchronous (H = 1). It is off by default.
• No tool in this PR converts the syncer's final checkpoint into an HF model. The closing run used a script outside this repository.

Shapes not tested on GPU:

• A trainer that spans nodes with sync. The 120B island with sync (272 fragments, offline layout 05c1fc0c…). Only the 120B one-island probe without sync ran.
• Island relaunch and peer recovery in the closing shape. Run A allowed them (--modal-retries 1 --recover-timeout 1200, peer recovery on), but no island exited.
• 262k-token sequences on two islands. Only one island without sync ran at 262k.
• Context parallel > 1 with Mamba sequence packing.

Refused or missing:

• The launcher refuses sampler islands together with full-parameter DiLoCo trainer islands. Three launch rules conflict:
  1. The full-parameter preset needs a fixed partition.
  2. A fixed partition needs engine GPUs.
  3. A trainer island must have 0 engine GPUs.
• One island cannot mix GPU types.
• For Nemotron-H the default tokens per GPU is 8192 x TP. At TP8 this gives 65536, so a 120B run must set --rl-max-tokens-per-gpu 8192.
• verl writes no rl_round_data_rows and cannot export HF rounds. verl has the Nemotron-H parameter rules and a smoke tool only. Full-parameter multi-island RL runs on Miles only.
• --local-rl-rounds-per-sync has no effect on a single island.
• The FP8 rollout tiers 1 and 2 on Miles need SGLang 754605964 and a new image. The current pin does not have them.
• Fork and image references still point at michaellchung. The move to agentenv needs the Miles commits and the image in agentenv first.

Unsynced and HF export paths:

• B's launch ended with rc = 4. After its final export, B committed a round cut in its finish phase. No training step came for 900 s, so the stall guard of the run script stopped the app.
• The launcher adds --rl-resume-store to an unsynced full-parameter island. So run B wrote 3 round cuts of 473 GB each (about 140 s per save). 2 of 3 cut commits hit the 600 s timeout.
• An unsynced full-parameter island still computes a full digest every round (about 8.5 s).
• B's train phase had about 13 s per round that the Miles timers do not explain. The cause is not known.
• The 718b2f14 fix has CPU tests only.
• We infer that a relaunched synced full-parameter island counts its export rounds from 0 again.

Merged branches:

• Git treats s26-e1-overhead and s26-agentenv-migrate as merged. The first merge kept the tree unchanged, so the default cut interval of 4 is not in the tree. The second merge took only the origin compatibility. A later change that wants the interval of 4 or the reference move must make it by hand.
• The one-commit release writer caches LATEST and assumes one writer per release directory. We infer a risk of a duplicate sequence number after a fenced publisher restarts. No test covers this.

Other:

• The launcher counts a Modal app that is still "stopping..." during teardown as a failed head job (rc = 1 after a successful run).

Earlier content of this PR (head 869078a)

Nemotron-H async full-parameter RL

• Nemotron-H recipe (Miles). yeto/rl/adapters/miles/models/nemotron_h.py writes the Megatron model flags from the Megatron-Bridge HybridModelProvider. RL keeps the routing statistics fixed (--moe-router-bias-update-rate 0, --moe-aux-loss-coeff 0). The backend registry loads the recipe (role nemotron_h_recipe), so the import boundary check passes.
• Ungated experts. The expert partition check accepts squared-ReLU experts, whose fc1 has FFN rows instead of 2 x FFN.
• Hybrid parameter rules and FLOPs. yeto/rl/hybrid_params.py names the Mamba fp32 parameters, the frozen vision tower and the 512-expert layout. yeto/rl/hybrid_flops.py gives the Nemotron-H model FLOPs for MFU. A verl Megatron GRPO plan and tools/nemotron_verl_smoke.py are included. The verl path is LoRA with one GPU per island.
• Per-fragment apply. An island applies a merged fragment by touching only that fragment. It skips fragments that the masters already hold (skip_resident).
• Deferred capture and batched applies. A local round asks Miles for descriptors only. The island materializes only the fragments that the next push needs. Distinct fragments of one partial-quorum commit go to the trainer in one apply. All three changes are bit-identical to the old path in tests. YETO_FULL_PARAMETER_DEFER_CAPTURE=0 and YETO_FULL_PARAMETER_BATCH_APPLY=0 turn them off.
• Wait timeouts, larger pipelines, terminal retire. --rl-sync-wait-timeout-s (default 900) sets the island wait. Every push restarts the wait. The async fragment pipeline limit is 60. After an island trains no more, the island closes the last round of a fragment without a trainer apply.
• Island lead cap (opt-in). --rl-sync-max-local-rounds-ahead K makes an island wait at a boundary instead of training on its own unmerged push.
• Sync diagnostics. Each Miles trainer-group call prints its elapsed time. Each push prints its delta, error-feedback, encode and enqueue seconds. Each island prints its host facts.
• Trainer speed flags. YETO_MILES_TRAIN_EXTRA_ARGV appends allow-listed Megatron flags. None of them changes the parallel layout.
• Long sequences. --context-parallel N reaches the island, the trainer layout (DP = world / (TP x PP x CP)), the placement request and the Miles --context-parallel-size.
• Numeric answer reward. yeto.rl.numeric_answer_reward:score (numeric_answer) grades one-number labels from the last \boxed{} or Answer: line. On 153,210 recorded responses the old gsm8k_grade credited 962 wrong answers and missed 200 right ones. gsm8k_grade did not change.
• Modal container-set guard. The guard maps containers to island slots from each island's own container line.
• Observability flags. --observe-training-entropy and --mamba-training-ssm-states-dtype reach the Miles argv.

How it is wired

• One syncer. A fragment-round server holds a fixed roster and an island pool. Islands join, leave and re-JOIN under a lease. A syncer-epoch fence rejects stale islands.
• Sync mode. --rl-sync-mode async|strict. Async is the default for multi-island RL. Strict is the control mode. --rl-sync-max-base-lag counts rounds.
• Full-parameter async sharded rounds. Policy versions, catch-up and quorum start are in place. --syncer-shards shards the head.
• Late-push carry and grace pacing. In async mode a late push goes into the next round. Grace pacing starts from every push.
• Island pool with peer recovery. --rl-island-state-source {peer,catch-up,cut} sets where a new island takes its training state. peer copies parameters and inner optimizer state from a healthy peer. catch-up pulls all fragments from the syncer with a fresh optimizer. cut is refused for synced islands. Peer recovery is on by default where it applies.
• Stop condition with a sharded head. An island stops local training only when the last round of every fragment is pushable or closed.
• Rejoin catch-up pacing. The syncer sends a rejoin catch-up one fragment per scheduler pass.
• Island disk and Ray storage. --modal-ephemeral-disk-gib, --rl-island-ray-object-store-gib and --rl-island-ray-spill-dir set the island disk and the Ray object store.
• Head memory. Pushes fold in a streaming weighted fold. Checkpoints are sharded and use copy-on-write. --rl-sync-checkpoint-every sets the checkpoint interval for async rounds.
• Push compression. Pushes can use int4 or int8. Islands can keep error feedback.
• Distributed optimizer and CPU offload. At DP = 1 the full-parameter sync works with the Miles distributed optimizer and optional optimizer CPU offload.
• MoE. The sync layout handles expert parallelism. Optional per-expert routed-token weights apply to the merge.
• Round cuts. Synced islands cut rounds in multi-island runs. Offline cut evaluation includes cut_to_hf with a TP/PP/EP merge.
• Sampler. The full-parameter sampler has an engine partition. SGLang applies ordered weight updates during generation.
• Outer optimizer. An optional delayed Nesterov optimizer runs in the syncer.

Earlier GPU validation (Nemotron-3 Nano 30B-A3B, 2 islands x 4 x H200, Miles, full parameter, async)

Run	Code / image	Result
s27-gr-f	fdd468ab	First run to the final cut. Each of the 60 fragments merged once with both islands. Both islands acknowledged the final cut.
s27-gr-g2	2b25c30f, image aa6b887-2fa8801	120 outer merges, all with both islands. Health: 13 PASS, 1 WARN, 0 FAIL.
s27-gr-h2	lead cap K=1	Both islands acknowledged the final cut. The lead cap made the rounds synchronous.
s27-gr-i2	abac7785, image aa6b887-2fa8801	180 merges, all with both islands. Health: 12 PASS, 2 WARN, 0 FAIL. Logprob gap up to 0.118 (Mamba packing leak).
s27-gr-j	869078a5, image d1616ba-2fa8801 (Mamba seq_idx fix)	180 merges, all with both islands. Health: 13 PASS, 1 WARN, 0 FAIL. Logprob gap 0.010-0.013. Train/rollout KL at most 0.0005. Both islands acknowledged the final cut.

• Mamba conv/SSM state leaked across packed samples in training before Miles d1616ba12. Runs before s27-gr-j trained with the leak.
• Long sequences (one island, 8 x H200, TP2 x CP2 / EP4): 262,144 tokens ran for 4 steps with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. Peak training memory was about 138 of 140 GiB.
• With optimizer CPU offload, island host memory peaked at 1034-1155 GiB.

Earlier testing (head 869078a)

• cargo test: 226 passed.
• Full CPU suite without Miles: 7406 passed, 52 skipped, 3 xfailed, 1 failed. The failure is a 10 s receive deadline in test_delta_protocol.py. The same test failed on the base and passed 3 of 3 times alone.
• Targeted tests with Miles on the path: 259 passed, 1 failed (test_rda_manifest_probe, a known evidence-file problem).
• The LoRA and Qwen3 full-parameter GPU smoke checks of earlier updates still apply.

Draft PR. Review is not requested yet.

🤖 Generated with Claude Code