Repository navigation

S26 E4/E5: phase timing, analysis tools, release writer Volume optimization - #289

Closed

michaellchung wants to merge 64 commits into

s25-finalfrom

s26-e45

Closed

S26 E4/E5: phase timing, analysis tools, release writer Volume optimization#289
michaellchung wants to merge 64 commits into
s25-finalfrom
s26-e45

Commits on Oct 20, 2026
- tools/eval: add efficiency evaluation (bare vs yeto, verl and Miles)
  
  chowhsu
  and
  claude
  committed
  5a01732
- syncer: pipeline policy-sweep fragments and async per-round checkpoints
  
  chowhsu
  and
  claude
  committed
  8358c1b
- tools/eval: bare Miles trains with unfused attention (match yeto generic recipe; TE flash fails on sm86)
  
  chowhsu
  and
  claude
  committed
  261ba2e
- syncer: bf16 sweep pushes and sweep-aligned checkpoint intervals
  
  chowhsu
  and
  claude
  committed
  eb9e229
- tools/eval: yeto+verl async arm passes the staleness-2 TIS algorithm spec
  
  chowhsu
  and
  claude
  committed
  c3c9d09
- syncer: int8 sweep pushes, bf16 downlink deltas, sync/train overlap
  
  chowhsu
  and
  claude
  committed
  1a783d0
- tools/eval: bare verl arms run the unpatched fork; async uses merged LoRA and /dev/shm NCCL host buffers
  
  chowhsu
  and
  claude
  committed
  63dc064
- overlap: keep rollout policy age 0, relabel rebased cuts, push whole sweep
  
  chowhsu
  and
  claude
  committed
  8af78fa
- tools/eval: parse bare verl fully_async rows by training step; rates over rounds that report the value
  
  chowhsu
  and
  claude
  committed
  052c31e
- tools/eval: bare Miles arms do not shuffle prompts (yeto's Miles argv does not)
  
  chowhsu
  and
  claude
  committed
  c9245e5
- verl adapter: fully_async keeps NCCL host buffers on /dev/shm in every Ray worker
  
  chowhsu
  and
  claude
  committed
  73442ad
- tools/eval: read the yeto+Miles launcher exit code into the parsed run
  
  chowhsu
  and
  claude
  committed
  0122f4e
- test: overlap island with bf16 downlink deltas against the real syncer
  
  chowhsu
  and
  claude
  committed
  152374b
- miles adapter: opt-in gradient-accumulation fusion; generic recipe trains without dropout
  
  chowhsu
  and
  claude
  committed
  e0f833d
- tools/eval: yeto+Miles arm takes the bare Miles recipe settings
  
  chowhsu
  and
  claude
  committed
  6a4263e
- tools/eval: read only the run's own tapes (skip the repo copy and the store)
  
  chowhsu
  and
  claude
  committed
  ff893bc
- tools/eval: verl fully_async generation time from the rollouter, else missing
  
  chowhsu
  and
  claude
  committed
  ba17953
Commits on Oct 21, 2026
- eff eval: final round cut only for yeto+Miles; verl async generation from the rollouter
  
  chowhsu
  and
  claude
  committed
  a4b1104
- verl: hash the LoRA on the trainer for the single-island publish fast path
  
  chowhsu
  and
  claude
  committed
  77ce9cc
- round cut: write before the trainer offloads; commit the store in the background
  
  chowhsu
  and
  claude
  committed
  74016eb
- fully_async cut: record untrained prompts without stopping the worker
  
  chowhsu
  and
  claude
  committed
  3b00fe5
- golden: regenerate decoupling samples for the explicit Miles dropout 0.0
  
  chowhsu
  and
  claude
  committed
  84d43ce
- driver: time the rollout offload after generation as its own phase
  
  chowhsu
  and
  claude
  committed
  36eb805
- eff eval: --checkpoint final|every-round for the yeto+Miles arm
  
  chowhsu
  and
  claude
  committed
  dfe8b91
- syncer: send each fragment's FINAL version as an exact FULL broadcast
  
  chowhsu
  and
  claude
  committed
  bc8f53b
- round cut: write the trainer shards in the background for periodic cuts
  
  chowhsu
  and
  claude
  committed
  8ef1018
- round cut: skip a periodic cut while the previous commit runs; time the sample filter hook
  
  chowhsu
  and
  claude
  committed
  de275ef
- cut digests: hash tensor buffers without a bytes copy
  
  chowhsu
  and
  claude
  committed
  77b460c
- round cut: a periodic cut does not wait for the store commit it starts
  
  chowhsu
  and
  claude
  committed
  86d8872
- rollout metadata hook: time its steps (E1 diagnosis)
  
  chowhsu
  and
  claude
  committed
  0f615ab
- fully_async: estimate the cross-version IS truncation off the critical path
  
  chowhsu
  and
  claude
  committed
  427d142
- test: the fully_async round report fake takes build_metadata's new keyword
  
  chowhsu
  and
  claude
  committed
  4015cc5
- round cuts: commit the deferred cut fully off the critical path
  
  chowhsu
  and
  claude
  committed
  1502e7d
- driver: take the terminal load probe off the round's critical path
  
  chowhsu
  and
  claude
  committed
  ae82993
- round cuts: collect a skipped prepared cut's writer at the next cut
  
  chowhsu
  and
  claude
  committed
  31a951a
- round cuts: do not wake an offloaded trainer for a declined periodic cut
  
  chowhsu
  and
  claude
  committed
  0a11aa1
- eval: give the yeto+Miles arm the bare arms' container resources
  
  chowhsu
  and
  claude
  committed
  f19a055
- round cuts: sync a deferred cut's state dir as it was at the cut
  
  chowhsu
  and
  claude
  committed
  19e595a
- wan: per-stream stall timeout, rate-aware chunk striping, sync diagnostics
  
  chowhsu
  and
  claude
  committed
  3dcbec8
- round cuts: copy a background cut's state on the device first
  
  chowhsu
  and
  claude
  committed
  8154509
- publish fast path: stage the digest's tensors with one GPU wait
  
  chowhsu
  and
  claude
  committed
  13d53ab
- rollout metadata hook: read each sink value once per hook run
  
  chowhsu
  and
  claude
  committed
  19f2648
- eval: tell an NVIDIA A10 from an A10G
  
  chowhsu
  and
  claude
  committed
  4a1c245
- test: compare staged digests without their timing fields
  
  chowhsu
  and
  claude
  committed
  1fde82f
- round cuts: keep a sleeping trainer's cut on the host path
  
  chowhsu
  and
  claude
  committed
  d583c59
- round cuts: serialize and hash background shards in a writer process
  
  chowhsu
  and
  claude
  committed
  2f3d79c
- launcher: pin a Modal island to one GPU model with --modal-gpu-name
  
  chowhsu
  and
  claude
  committed
  27f66dd
- publish fastpath: hash the policy while a colocated trainer offloads
  
  chowhsu
  and
  claude
  committed
  d8d159c
- wan: replay INIT after a mid-init redial, link probe with container relaunch
  
  chowhsu
  and
  claude
  committed
  2509fde
- Merge s26-e3-sync into s26-e11-cluster
  
  chowhsu
  and
  claude
  committed
  329472f
- eval efficiency: spec.cards sets the per-run card total (E11 whole-cluster arms)
  
  chowhsu
  and
  claude
  committed
  87c298a
- wan: refuse the link probe with the elastic syncer; test a lost socket mid-INIT
  
  chowhsu
  and
  claude
  committed
  823b2e8
- eval efficiency: E11 preset and the 2-island yeto+Miles launcher args
  
  chowhsu
  and
  claude
  committed
  140aefa
- eval efficiency: JSONL datasets, a train-row cap and local task files for the yeto arms
  
  chowhsu
  and
  claude
  committed
  b0dfb1c
- eval efficiency: derived Miles model scripts and the yeto+verl rollout memory fraction
  
  chowhsu
  and
  claude
  committed
  392368b
- eval efficiency: the E11 2-island arm uses elastic island scheduling
  
  chowhsu
  and
  claude
  committed
  5593787
- eval efficiency: cluster.py, reward against samples and time for the E11 arms
  
  chowhsu
  and
  claude
  committed
  ae1a26e
- Merge commit '823b2e82' into s26-e11-cluster
  chowhsu
  committed
  f1b26ff
- Full-parameter no-sync single island and the E11 0.6B full-parameter preset
  
  chowhsu
  and
  claude
  committed
  fb520a6
- eval efficiency: no-progress watchdog and the E11 full-parameter 2-island arm
  
  chowhsu
  and
  claude
  committed
  8783b25
- E5: time the policy release phases; optional fused LoRA stage
  
  chowhsu
  and
  claude
  committed
  189df10
- E4: trainer driver phase seconds and buffer wait share in the scale report
  
  chowhsu
  and
  claude
  committed
  f183d75
Commits on Oct 22, 2026
- E4 tool: merge launch-log events; a preempted round with no groups is not a round
  
  chowhsu
  and
  claude
  committed
  72adc7c
- E5: a release writer keeps the LATEST it wrote and commits once
  
  chowhsu
  and
  claude
  committed
  dd88b26

Commits

Commits on Oct 20, 2026

Commits on Oct 21, 2026

Commits on Oct 22, 2026