Repository navigation
S26-E9: robustness fixes (preemption marker, verl cuts, data slots, one-command verl pairs) - #282
Closed
michaellchung wants to merge 23 commits into
Closed
michaellchung wants to merge 23 commits into
michaellchung wants to merge 23 commits into
Conversation
…y loosened value The C1b eval scripts passed --preflight-threads off. They now keep the default (wait) and may loosen it only with explicit flags (cfg preflight_args / PREFLIGHT env); the launcher prints each loosened value with its default and records it under preflight.threads.relaxed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ted buffer; launcher follows rescheduled Modal containers
- sample_roles.DataSlot: sampler i of n keeps prompt rows r % n == i. The
launcher sets --rl-sampler-data-slot per pair sampler island; the Miles
buffer writer splits its RolloutDataSource before the first draw and any
resume seek; the engine sampler spec takes prompts.data_slot (stage 1
sets it with several sampler nodes).
- BatchReader raises SampleBufferExhausted when the buffer ended with no
trainable group; IslandDriver ends the run FINISHED before that round
(event rl_buffer_exhausted), except under a strict roster.
- ContainerLogFollower: when a new container is expected, the launcher
lists the app's containers and streams each new one's log
('modal container logs'), with a warning; the call log stream misses it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gh the marker file Modal signals only the container's main process (the runner); a verl driver runs in the task runner actor, where install_preempt_handler cannot install (not the main thread). The launcher now sets the reclaim marker path on resumable Modal islands, so the runner writes the marker; resume.watch_preempt_marker polls it on a daemon thread in the driver's process and writes the same store preemption notice. Wired on verl (rl_round_cuts_wired.preempt_marker_watch) and on Miles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ly_async path - cut.save_rank / restore_rank keep only the parameters the optimizer holds. verl's state dict carried a requires_grad lm_head.weight that no optimizer steps (865 MB per rank file on GPU T-C4). - cut.CutPorts: the cut verbs shared by VerlTrainerGroup and FullyAsyncTrainerGroup (through the trainer actor's worker group). - fully_async_cut: the data cursor counts prompts the rounds took from the queue; on resume the rollouter skips them in its own actor before fit (same continuous iterator, so the epoch order is kept); queued prompts at the cut are generated again. run_fully_async_island wires the resume store, ledger and round cuts; resume_config no longer refuses --rl-max-policy-age > 0. Not run on a GPU yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s resume store), not the island flag Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s the engine sampler for role sampler - verl island_entry accepts the pair flags the launcher forwards (--rl-sampler-node, --rl-buffer-shards, --rl-sampler-data-slot, backlog and throttle) and, for --rl-sample-role sampler, runs adapters.verl.sampler_entry: an engine sampler spec on the image's vLLM (vllm-server) over the run store, prompts rendered with the chat template. - engine sampler spec: optional buffer_shards (several samplers on one shared directory write their own shard); open_trainer_buffer merges the named shards; the verl buffer trainer passes them. - Dry-run tests through the real launcher task builder: a verl pair, two samplers (shards, disjoint data slots, split rounds), and synced verl trainers with a separate sampler launch over the same store. Not run on a GPU yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r path (E9.1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…apter import) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ced trainers and their samplers - modal_runner: on a multi-node island rank 0's runner listens on a small relay port; a rank>0 runner that gets SIGINT/SIGTERM sends the signal there, and rank 0 writes the reclaim marker its driver watcher turns into the store's preemption notice (signal names the source rank). - sample_roles: a synced group (--rl-sample-role trainer with a syncer plus --rl-island-sample-role M:sampler) starts K synced trainers (learners 0..K-1, lanes K) and no-sync sampler islands over the same store in one launch; the samplers split K x total-steps rounds, take data slots and buffer shards. - launcher: the syncer counts only the trainers of a synced group. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mes the rank Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eral samplers) The verl buffer trainer opens its buffer with create_index=False. With named shards no sampler writes index.jsonl, so the first merge failed with FileNotFoundError (S26 E9 G1 on Modal). open_trainer_buffer now creates the index whenever this trainer merges shards: it is the index's only writer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A sampler that writes its own buffer shard counts its backlog from the release's buffer_shards. The verl publisher released only buffer_selected, so on Modal (S26 E9 G1) both shard samplers stayed at their backlog cap and the trainer timed out after four rounds. The release (and the release log) now carries the read cursor per shard, as the Miles trainer does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…alsy) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a buffer trainer the queue is the sample buffer. When the samplers ended and a round could not be filled (groups passed over as older than max_policy_age), FullyAsyncRolloutPool raised "rollouter ended the queue" and the island left INFRA_FAILED (S26 E9 G1 on Modal, launcher rc=4). The buffer ports now mark the pool so that end raises SampleBufferExhausted, which the driver turns into rl_buffer_exhausted and a FINISHED run (as on Miles). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…inalized The launcher requires every island tape to end with rl_learner_finalized; a verl pair's sampler islands run the engine sampler, whose tape never had it, so a run whose islands all succeeded exited 3 (S26 E9 G1 on Modal). A finished incarnation now writes it; a reclaimed one does not (the next incarnation continues the island). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The launcher completes a Modal island's event tape from the island's tape file and accepts only records with island_id. The verl pair's sampler islands write engine sampler records (sampler_id only), so every record, the new rl_learner_finalized included, was discarded and the run exited 3 although all islands succeeded (S26 E9 G1). The verl sampler spec now names the island (island_id = learner id) and the tape writer adds it to each record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With --rl-max-policy-age > 0 a round cut asks the rollout pool for its in-flight work. FullyAsyncRolloutPool had no export_in_flight, so every cut of a fully_async verl island was refused (RebuildRefused; S26 E9 G2 on Modal: rl_round_cut ok=false for each round). A round keeps or drops every group it pulls, so nothing pulled is in flight at the cut; the groups still queued in the rollouter are generated again after a resume from the data cursor. The export is now empty and the cut is written. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dal verl islands default NCCL_SHM_DISABLE=1 The sync path pinned trainer.n_gpus_per_node=1, so a 2-card container trained on one rank (S26 E9 e9-g3-a). The island now takes the card count from its island-local Ray. A 2-card verl island on Modal failed attaching NCCL shared memory (e9-g2-a); the launcher sets NCCL_SHM_DISABLE=1 for multi-card verl Modal islands unless --modal-env sets it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…samples record their data row e9-g4-a: both Miles samplers drew the same 16 prompts (prompt text overlap 16 of 16/24). DATA_SLOT_ENV was set in the island process but missing from ISLAND_ENVS, so the rollout worker never split its data source. Each kept row now also records its index in the unsplit data as metadata["yeto_data_row"], which reaches the sample buffer payload. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a 2-card verl sync island (e9b-g3-a) both vLLM replicas wrote readback-v0.json through the same tmp name; the second os.replace raised FileNotFoundError inside vLLM and the weight update failed. The tmp name now carries the pid, and a write failure is printed instead of raised into vLLM. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… islands cmd_head sized the head-local syncer with every --gpu island, so a synced group (2 trainers + 1 sampler, e9b-sync-a) waited for 3 learners and the trainers hung after connecting. It now uses sample_roles.syncer_learner_count, the same count launcher.run uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The strict and decoupled presets set --quorum to every --gpu island; in a synced group (2 trainers + 1 sampler, e9b-sync-b) that is quorum 3 over --learners 2, and the syncer exits at start. Both now use sample_roles.syncer_learner_count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Closing: all commits of |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Robustness items carried over from S25.
Verified on Modal A10G: G0 multi-node forwarding, G1 verl pair, G2 fully_async cut + preemption, G3 2-rank NCCL cut and resume, G4 Miles disjoint data slots, synced group with syncer. Full suite 6701 passed / 0 failed.
🤖 Generated with Claude Code
GPU evidence
All runs on Modal A10G, GPU pinned and asserted ("NVIDIA A10"). Raw data: SESSION26/logs/E9// (internal). Costs are estimates, not checked against billing.
SIGINT (reclaim marker, rank 1). e9-g2-e wrote preempt-notices/0001-*.json withSIGINT (reclaim marker). Not verified: Miles in-actor preemption.following its logpresent, rl_buffer_exhausted once, rc=0. e9b-g4-a (Miles) row overlap 0, prompt-text overlap 0, rc=0.--controller headcommand started 2 synced trainers + 1 sampler (syncer--learners 2 --quorum 2, outer steps 1-3, learner 0 published v1-v3, 48 read positions without duplicates, rc=0). Not verified on GPU: publisher stop mid-run and island backfill; OSS real run.RELAXED ... start 2950andthreads OK(2663 and 2724, both below 3000).