Repository navigation
S26 E0.3: bias-stacking ablation tooling for Qwen3-0.6B on Miles - #286
Closed
michaellchung wants to merge 85 commits into
Closed
michaellchung wants to merge 85 commits into
michaellchung wants to merge 85 commits into
Conversation
Eight arms per model preset ({bare,yeto} x {verl,miles} x {sync,async})
generated from one shared spec with a control fingerprint. Bare arms run
the frameworks' own entry points (verl main_ppo / fully_async_main,
Miles train.py / train_async.py) in the same images yeto uses; yeto arms
run verl island_entry and the yeto launcher. Parsers normalize verl
FileLogger, Miles perf lines and yeto tapes into per-round records
(missing fields are marked, never estimated); the report gives
yeto-vs-bare deltas with min-max range verdicts. Presets: Qwen3-0.6B
(E0.1) and Qwen3-4B-Instruct-2507 (E0.2). CLI: eff-plan, eff-run,
eff-parse, eff-report.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Allow --pipeline 1..P with --policy-sweep-fragments so fragment k+1 uploads while fragment k merges, checkpoints and broadcasts; commits stay in strict step order. Add --async-checkpoint (policy sweep only): the committed state is snapshotted and written on a blocking task, and the round is broadcast only after the checkpoint is durable. Checkpoint bytes, versions and the semantic profile hash are unchanged; the tape gains checkpoint_ms. Launcher flags --rl-sync-fragment-pipeline and --rl-sync-async-checkpoint (default off). Adds bit-equality, failure and real-syncer tests and a CPU micro-benchmark (tools/bench_sync_boundary.py). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ric recipe; TE flash fails on sm86) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add session dtype 4 (f32 session, bf16 PUSH deltas): INIT/BCAST/FINAL, masters and momentum stay f32; the client rounds f32 delta bytes to bf16 (round-to-nearest-even, streamed across part boundaries). The dtype is bound into the role-stream session contract. Launcher --rl-sync-push-dtype, island --sweep-push-dtype (default f32). Allow --checkpoint-every N (1 or a multiple of the fragment count) for policy sweeps: only checkpointed steps are durable before their broadcast, on both the synchronous and async paths; the interval is part of the semantic profile hash. Launcher --rl-sync-checkpoint-every (default 1). E3 launcher options are refused outside the full-parameter preset. Adds bf16 error-bound, interval/resume and real-syncer tests, and a third micro-benchmark profile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…spec Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add session dtype 5 (f32 session, block-scaled int8 PUSH deltas): u32 block size, then per block an f32 absmax/127 scale and one i8 per value; error is at most absmax/254 per element. Launcher --rl-sync-push-int8-block (default 128) is bound into the role-stream contract. Add session flag 0x80 (bf16 downlink deltas) for f32-bulk sessions. Syncer and learners hold a bit-identical f32 anchor per fragment; BCAST is FULL f32 or bf16(theta - anchor), with a FULL refresh every N versions (--bcast-delta-full-every), FULL re-sends on HELLO/reconnect, and a SHA-256 check of the anchor after every update. A mismatch drops the connection and the reconnect restores the anchor. SyncerClient decodes before adapters see the payload: the Miles streaming sink is wrapped, the strict bridge (verl and Miles LoRA) gets f32 bytes. Masters, checkpoints and FINAL stay exact. Add --rl-sync-overlap for the full-parameter sweep: an island pushes round k, trains round k+1 at once, and at the next boundary rebases in place, theta <- G_k + (local - base), with G_k as the anchor so no delta is counted twice. The launcher raises --rl-max-policy-age to at least 1 before its checks; the island refuses overlap below that age. Adds Rust and Python codec/anchor tests, real-syncer int8/downlink/resync tests, a strict-bridge downlink test, and coordinator overlap tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a backend-neutral rollout quantization setting (--rl-rollout-quant-tier 0..3, one flag definition for launcher and island): - tier 0: BF16 (default, no change); - tier 1: the sampler engine quantizes in-scope linears to FP8 itself and keeps an FP8 KV cache. Miles/SGLang: --sglang-quantization fp8 and --sglang-kv-cache-dtype fp8_e4m3, with the ignore list passed through SGLANG_FP8_IGNORED_LAYERS in the Ray job runtime_env. verl/vLLM: verl's own FP8 rollout config with our explicit ignored_layers through engine_kwargs.vllm.hf_overrides; - tier 2: the trainer side converts each BF16 release to FP8 W8A8 per channel (compressed-tensors float-quantized, the FP8_DYNAMIC preset) and sampler islands load it. Reuses the N8 requant pipeline with a new scheme whose target layout is derived from the BF16 base and the scope rule; - tier 3: 4-bit, interface only (refused at launch). Scope rule (yeto/rl/quant/scope.py): a dense model quantizes only its FFN linears, a MoE model only its routed experts; router, shared experts, lm_head, embeddings, norms and attention stay BF16. The launcher refuses tier 3, cards without FP8 and tier 2 without sampler islands or with LoRA; island entries refuse a tier 1 plan that would quantize an out-of-scope module; sampler loaders refuse an FP8 release outside the scope before touching the engine. Add yeto.rl.quant.gate: logprob difference (mean, p50/p90/p99), TIS truncation share and tokens per second of a quantized engine against BF16 on one prompt set, with configurable criteria and the report written to disk. CPU only. FP8 values equal compressed-tensors 0.18 bit for bit; the ignore lists checked against SGLang 2fa88018's own skip functions. Not run on a GPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Read the cards of --gpu with yeto.gpu_spec and accept tiers 1 and 2 only on cards with FP8 tensor cores (L4, L40S, RTX-6000-Ada, RTX-PRO-6000, H100, H200, B200); every other card is refused before launch. - Add the yeto_rollout_quant_tier runtime attribute only for a quantized tier, so tier-0 Miles launches keep their golden argv and runtime attributes. - Test the Miles translation and the island-task forwarding of the tier. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…LoRA and /dev/shm NCCL host buffers The verl image applies yeto's patch_verl, whose sender hook imports yeto, so the bare arms now build the image without it. The fork's disaggregated checkpoint engine drops the PEFT config of unmerged LoRA params; the bare async arm sends merged weights instead. On Modal the trainer allocated NCCL host buffers with cuMem while the receiver fell back to /dev/shm and failed to attach (NCCL_DEBUG=INFO run); NCCL_CUMEM_HOST_ENABLE=0 keeps both on /dev/shm. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sweep Pre-GPU check of --rl-sync-overlap on the Miles role stream found three problems: - The launcher raised --rl-max-policy-age to 1, but the dense sweep island refuses any age other than 0, so an overlap run could never start; and an age >= 1 also turns on Miles' fully_async rollout (partial rollout when colocated). Overlap is not rollout staleness: the driver publishes the boundary's cut before the next generation, so rollouts stay on-policy and only the local weights lag the global policy by one outer step. Keep the age at 0 (refuse others at launch) and bind the lag into the session contract only. - After a rebase the applied cut has local step generation 0, and Miles pushes only cuts with progress, so the second sweep's push failed. The submission is now the rebased cut relabeled with the rounds trained since the round base (same values, hashes and refs). - The boundary submitted once; a PULL that arrived later waited a whole local round. The overlap boundary now submits the whole sweep before training on, and the launcher requires --rl-sync-fragment-pipeline equal to --fragments so the syncer sends all PULLs at once. Adds Miles role-stream tests (real adapter and role stream, CPU tensors, fake trainer/Ray/syncer): exact-once rounds through rebase_fragment, and syncer restarts while the overlapped round trains (before commit, after commit with a lost broadcast, partly committed sweep). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Engine sampler islands: the launch layer writes rollout_quant {tier, apply}
into the sampler spec (stage1.sampler_spec); the entry applies it on the
node (sampler_engines.quant_tier). Tier 1 on sglang-server fills
--quantization fp8, --kv-cache-dtype fp8_e4m3 and SGLANG_FP8_IGNORED_LAYERS
from the checked online plan; vllm-server is refused (stand-alone vLLM online
FP8 has no ignore list). Tier 2 converts the base to the FP8 scheme and sets
quant_scheme, expected_format, release_dir and the routed-expert count. A spec
that already quantizes is refused. Stage 1 refuses a tier on top of its own
quant section.
- Post-load expert_count: an expected count of 0 (dense model) passes without
an engine report; the release index count is checked before the load.
- requant: drop a tied lm_head.weight that an exporter writes although the
base ties it to the embeddings (tie_word_embeddings).
- Tests: the ignore list reaches the process started by Miles'
launch_bound_subprocess; Miles' convert_qwen2_to_hf names equal the
Qwen3-4B-Instruct-2507 index names (fixture from the real index).
- Probe (Modal CPU, verl image): Hydra overrides -> RolloutConfig -> verl's
CLI builder -> vLLM 0.29 argument parser -> Fp8Config. The received
ignored_layers equal the sent list; qkv_proj and o_proj skipped,
gate_up_proj and down_proj quantized; kv_cache_dtype fp8.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Importing it as a package put the real miles package into sys.modules and broke a later test that fakes miles (test_rl_reward_success). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…over rounds that report the value The fork logs a step's training metrics one parameter version late and never logs the last step's, so rows are keyed by training/global_step and the step without metrics keeps its trained batch from the config. Throughput rates use the steady rounds that report the value; the report lists partial counts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… does not) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y Ray worker On Modal one weight-sync process could allocate NCCL host buffers with cuMem while its peer fell back to /dev/shm and then failed to attach (eff01-yeto-verl-async-e; NCCL_DEBUG=INFO in eff01-bare-verl-async-d). NCCL_CUMEM_HOST_ENABLE=0 unless the caller sets it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The overlap rebase uses the bf16-tracked anchor and pushes theta - anchor against that same anchor, so the syncer's f32 masters still receive every round's progress exactly once and no DownlinkResync happens. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ains without dropout --rl-gradient-accumulation-fusion lets a run keep Megatron's fused wgrad (the Miles recipes' default); off keeps the historical argv. The generic recipe passed no dropout, so Megatron's 0.1 applied during RL training while the Miles recipes pass 0.0; it now passes 0.0 as the Qwen3.5 recipe does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No deterministic inference, constant LR, full recompute, dynamic batching, gradient-accumulation fusion, and an algorithm spec with the recipe's clip range (0.2 / 0.28); S26 GOAL §6 makes the bare framework's settings the reference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… store) miles_local.sh pulls a copy of the repo whose test fixtures are tapes of other runs; their rl_local_round rows replaced the run's rewards (all four yeto+Miles runs showed the same fixture curve with reward 0 in round 0). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… missing The trainer's timing_s/gen and yeto's driver generate phase only pull finished samples from the rollouter queue. Bare verl logs the rollouter's active time per parameter version; yeto logs none per round, so its value stays missing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…from the rollouter E0.1 found yeto+Miles 77 % (sync) and 252 % (async) slower than bare Miles. The tape shows the gap is the per-round round cut: the arm sets a checkpoint store, so a cut runs after every publish (5.7-12.9 s: trainer wake, export, sleep, Modal volume commit), and in fully_async mode the cut also stops Miles' background worker, so generation never runs ahead of training. The bare arms keep no checkpoint; cut only at the end (--rl-cut-every rounds). yeto+verl async generation time was missing: the tape's rl_publication_check.rollouter_timing carries the rollouter's active time per policy version, the same metric the bare arm reports. Read it (version v closes round v - 1). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… path VerlPolicyState.export_digest returns a trainer-resident handle: the workers validate the LoRA like canonical_state, compute the policy digests and the bf16 send checksum, and only those cross Ray. Both verl publishers take the send checksum from the handle; other states are hashed on the driver as before. materialize falls back to the full export; YETO_RL_PUBLISH_FASTPATH=0 keeps the old path. Tests pin hash and checksum equality with the full export. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… background E0.1 measured a periodic round cut at 5.7-12.9 s per round on the critical path of a colocated Miles island: wake the offloaded trainer (0.7 s), export (1.8 s), sleep it again (1.0 s), then a synchronous store commit (1.5-8 s). - prepare(): when a cut is due at the next safe point, the driver writes the trainer part right before it offloads a colocated trainer for the publication (phase "checkpoint"), for the state it is about to publish. The safe point keeps it only if its own context names the same cut (id, progress incl. policy hash, data cursor, ledger, outer token, in-flight); otherwise the unpointed files are dropped and the cut is taken as before. - Background commit: a periodic cut writes the snapshot and LATEST, then a thread commits and prunes (ResumeController.commit_checkpoint_store). The next cut joins it first and reports rl_cut_committed. Pruning only follows a successful commit, so a failed or lost commit leaves the previous cut as the committed one. The final cut, and a run with a sample buffer that commits the same store, stay inline. YETO_RL_CUT_BACKGROUND_COMMIT=0 turns it off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A round cut on a Miles fully_async island stopped the background worker and requeued every untrained prompt into the live data source (P0-S decision 3). With a cut every round the worker restarted each round, so generation never ran ahead of training (E0.1: 8 worker starts in 8 rounds, policy age 0, 2.15 s of serial generation per round). snapshot_untrained() reads the data source buffer, every drawn untrained group (reset copies) and the data cursor in one step on the worker's event loop, where the worker draws; the cut carries them and the cursor read with them, and the live run is untouched. A resume still regenerates exactly the drawn, untrained prompts (decision 3 unchanged); the resume manifest's data watermark is that cursor. requeue_untrained() stays for its other callers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
e0f833d made the generic Miles recipe pass --attention-dropout 0.0 and --hidden-dropout 0.0 (the bare Miles recipe's values); the seven decoupling golden samples were not regenerated and failed on miles_argv. The only difference is those four argv items (and the argv hash); no algorithm or contract hash changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
E0.1: the yeto+Miles sync generate span was 0.46 s longer than Miles' perf/rollout_time. 0.44 s of it is --offload-rollout (release_memory_occupation), which Miles times outside rollout_time but MilesRolloutPool.generate ran inside the driver's generate phase. The driver now asks the pool to defer it and runs it as phase "rollout_offload"; callers that use the pool directly keep the old in-generate offload. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
final (default) cuts once at the end like the bare arms keep no checkpoint; every-round is the product default. Kept out of the control fingerprint so the bare runs stay comparable. The yeto+verl arm has no checkpoint store in either mode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With bf16 downlink deltas, the last BCAST of a fragment can carry the same
version as its FINAL. Islands store one copy per (fragment, version) and
refuse a second copy with other bytes, so the delta-tracked BCAST and the
exact FINAL collided ('authoritative fragment replay changed bytes'), the
islands redialed in a loop and finalization timed out (s26-e3-p06-D8).
Broadcasts of a fragment's last version now always go out FULL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GPU canary e1a-yeto-miles-async-1 (every-round cuts, after 74016eb and 3b00fe5): overlap is back (one fully_async worker start), but each cut still held the round for 2.1 s writing and hashing the 283 MB shard, so a round took 5.3 s against 4.15 s bare (78 %). MilesTrainerGroup.start_cut captures every rank's state at the step boundary (CPU copies) and the ranks write, fsync and hash their shards in a thread (cut_plugin.save_cut_shard background=True); finish_cut collects the summaries and commits the manifest. A periodic cut now only captures at its safe point; the shard write, manifest, pointer, store snapshot and commit complete before the next cut (join_commit, rl_cut_committed). Until then the previous cut stays the pointed and committed one; a failed completion drops the unpointed files. The store's first cut, the final cut, runs with a sample buffer or a critic, and YETO_RL_CUT_BACKGROUND_COMMIT=0 stay inline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rollout actors that reload the Volume are other processes than the tier 2 releaser thread (e6-t2-pair-c lost LATEST.json.tmp with the in-process guard). Writers now hold a shared flock on a container-local file per Volume; a reload takes it exclusive without blocking and is skipped while a writer holds it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hash_seconds and export_seconds are wall times and may differ between two calls on the same tensors (seen on yeto-cpu-test). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With --offload-train the colocated trainer sleeps right after the cut is prepared, and the writer thread then failed to read the device copies (CUDA error: invalid argument; 5 of 6 cuts refused in GPU run e1g-yeto-miles-sync-1, the run itself went on). Such a trainer now takes the host path, reported as reason offload_train. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Miles' native block FP8 with the ignore list has the larger gate margin (p99 |diff| 0.43 vs 0.46-0.49 for the engine's online FP8, limit 0.5). --rl-rollout-quant-fp8-source now defaults to trainer for tier 1 on Miles and to engine otherwise; engine stays selectable. The launcher resolves the default once so the island flags and the run config carry it explicitly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--tuning full was accepted only on sampler / trainer buffer islands. One island with --rl-single-island-no-sync and --rl-placement fixed-partition now also trains every parameter: Miles' own full-parameter weight update feeds its dedicated engine cards each round (S26 E6 uses it to check that Miles block-quantizes every update for the tier 1 trainer source). A colocated full-parameter island is still refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SGLang fork 754605964 (yeto/fp8-act-cast): TENSOR / CHANNEL FP8 linears reload in place through update_weights_from_disk, which the tier 2 sampler needs for each FP8 release. Image 9462de5-7546059 @sha256:37acaca3...; the image check also asserts the fix and runs its unit test in the image. The E6 driver gains the full-parameter dedicated-engine groups and a SEED knob, and the pair trainer island uses 2 cards instead of host Adam state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The background cut writer ran torch.save and four sha256 passes in a thread of the trainer process while the next step trained; each trained token was about 8 % slower in those rounds (GPU runs e1g/e1h). The trainer thread now copies the device snapshot into shared host memory and a per-rank spawn writer process (nice 10, one intra-op thread) serializes, hashes and summarizes it. A writer-process failure removes its partial file and the thread writes the same shard; the summary's writer field and rl_cut_committed record the mode and the fallback. YETO_RL_CUT_WRITER_PROCESS=0 keeps the thread. Strings are interned before torch.save so the shard bytes depend on values only: a shard written in the writer process is byte-identical to the inline one (tests compare file bytes, digests and the N7.6 state). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… 1 train + 1 engine card Engines on the island need DP=1 (e6-t*-full-a refused 2 training cards), so the Adam state lives on the host (200 GiB container). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Modal A10G containers are an NVIDIA A10 or an A10G, and the A10G trains
about 12 % slower (E1 run e1a9). --modal-gpu-name MODEL fails an island
at container start unless every GPU nvidia-smi reports is that model
("NVIDIA A10" refuses "NVIDIA A10G"). The efficiency eval passes it
for the yeto+Miles arm with the model the bare arms require, so every
arm runs on the same card.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In sync colocated mode the trainer hashed the staged policy (two sha256 streams, about 0.1 s) before it could offload. export_digest now takes deferred=True: the trainer returns once the tensors are staged in its pinned buffer and hashes them in a thread; the state's DeferredDigest collects the result (collect_digest plugin) on first read, at the publication. The values are those of the eager digest; a new export waits for an uncollected hash before it reuses the buffer. The cut context reads the policy hash last. The driver defers only for a colocated trainer that offloads to publish (YETO_RL_DEFERRED_DIGEST=0 turns it off). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…elaunch s26-e3b-p06-C32a: learner 0's connection group died before the syncer applied its INIT, the client never sent INIT again, and no island got the initial cut. s26-e3b runs also showed that a slow island is slow on every stream (container level), which no stream scheduling fixes. - SyncerClient.send_initial_fragments/init_replay_due/ mark_initial_cut_received: learner 0 replays INIT on every new connection group until the initial cut arrives (the syncer ignores INIT it already applied). TCP_USER_TIMEOUT is set only after the initial cut, so the multi-GB INIT and first broadcast are never reset by the kernel. - MSG_BW_PROBE (26): an HMAC-sealed probe connection to the legacy syncer uploads N bytes, gets an ack, downloads M bytes; never part of a session, at most 64 concurrent, 1 GiB per direction, 300 s. - --wan-probe-min-mbps / --wan-probe-min-down-mbps (default off): Miles islands probe the actor syncer before loading the model, record rl_wan_probe, and exit 75 below the minimum. The fleet controller requests a new container up to --wan-probe-relaunches times (default 2), also under --no-island-relaunch, since the island never joined the session. Real-syncer test: INIT interrupted between fragments is replayed and the finals match an undisturbed run bit for bit; the probe works on the same port and a wrong key is refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Observation only, for E1: around each train_one_step the trainer process prints the step's wall time, main-thread and process CPU time, garbage collector pauses, CUDA caching-allocator counters and stack samples of every Python thread. The launcher forwards the flag to the island, the island sets it on the Miles args, and eff-run --profile-trainer passes it for the yeto+Miles arms (not in the control fingerprint). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ith the link probe Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… step The writer thread copied the device snapshot to the host tensor by tensor while the next round trained: about 400 synchronous device-to-host copies interleaved with the training kernels, the suspected cause of the slower training (profile run e1j-yeto-miles-async-a: the step grew while GC, allocator and main-thread CPU stayed as in the final-only run). The capture now copies the device snapshot into one reused pinned host buffer with a single synchronization, drops the device copies, and the writer thread copies the staged views into shared memory (the buffer is reused only after that). Since nothing reads device memory after the capture returns, a trainer that sleeps right after the cut (--offload-train) takes the device path too. Tests compare the shard bytes and summaries of staged, unstaged and inline cuts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eps its background groups Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t at the step" This reverts commit fe5e612. GPU runs e1k-yeto-miles-async-2 and e1k-yeto-miles-sync-1: the trained tokens were not faster (0.553 ms per token, as before) and the step-boundary capture got slower (async 0.033 -> 0.08 s, sync host path 0.16-0.28 -> 0.30 s). The interleaved device-to-host copies were not the cause of the slower training. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Moving each tensor to shared memory passed one file descriptor per tensor (about 400 per cut) through the trainer process's resource sharer, and its queue feeder and server threads ran while the next round trained (profile run e1j-yeto-miles-async-a: about 15 % of the step's stack samples; none in the final-only run). The writer thread now copies every contiguous tensor that owns its whole storage into one shared uint8 buffer and sends offsets; the writer process rebuilds each tensor as its own copy. Other tensors are shared one by one as before. The shard bytes and summaries are unchanged (tests compare them with the inline cut). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Closing: all commits of |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
S26 E0.3: bias-stacking ablation tooling (no product-code change in this task's own commits).
data/s26_e03/(inputs, not run logs).--rl-profile-trainer).Defaults
Tooling only. New flags are off by default.
Branch dependencies
Stacked branch. s25-final..HEAD also contains merges of s26-e01-effeval (pushed), s26-e3-sync (up to 2509fde), s26-e6-quant and s26-e1-overhead (the last two are not pushed separately). It depends on the s26-e3-sync PR; the E0.3-only commits are 2e1d8d3, a78e73a, 91adbef, 3c9ee84, 92cb83c, b799606, 1a3014f and related profiling commits.
Test evidence
Remote full suite (yeto-cpu-test): 6877 passed, 0 failed, 101 skipped at 1a3014f. Earlier failures were run-log-guard hits on the task .jsonl files (moved to data/) and a stale git index on the remote worktree.
GPU evidence
Qwen3-0.6B, full-parameter, Miles, Modal H100, 10 rounds, token-level TIS (bound 2.0), seeds 17/29, range criterion.
Not verified
🤖 Generated with Claude Code