Skip to content

S26 E6: FP8 quantized rollout and full-parameter dedicated-engine island - #290

Draft
michaellchung wants to merge 23 commits into
s25-finalfrom
s26-e6-quant
Draft

S26 E6: FP8 quantized rollout and full-parameter dedicated-engine island#290
michaellchung wants to merge 23 commits into
s25-finalfrom
s26-e6-quant

Conversation

@michaellchung

@michaellchung michaellchung commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

What changed

E6 quantized rollout (FP8) on Miles.

  • Tier 1 on Miles defaults to the trainer FP8 source (native FP8 plus an ignore list); verl is refused for tier 1.
  • Ports image pinned with the SGLang compressed-tensors FP8 reload.
  • Full-parameter on one no-sync island with dedicated engine cards (1 train + 1 engine card), run without round cuts or with cuts every 5 rounds; the store writer guard is held across a round cut.
  • Modal store: write guard is a container-wide flock; no reload while a writer holds uncommitted files.
  • Miles RL drivers for the tier groups, per-round parser of Miles RL runs, native-FP8 gate probes.

Defaults

FP8 tiers and the dedicated-engine full-parameter mode are opt-in. The Miles tier 1 FP8 source default applies only when a tier 1 run is requested.

Branch dependencies

Check the commit range against s25-final; it may include commits shared with other S26 branches (E1, E01). Merge order is the coordinator's decision.

Tests

  • Remote full suite (yeto-cpu-test) at 2ab01f8: 6704 passed / 0 failed / 108 skipped.

GPU evidence (Modal H100; SESSION26/E6-REPORT.md rounds 5-6)

  • Range criterion (4.2) passed for the Miles full-parameter group (0-tier [0.9375, 0.9401] vs 1-tier [0.9141, 0.9453], overlapping) and the later rerun (0-tier [0.9141, 0.9505] vs 2-tier [0.9297, 0.9297]).
  • Check 4.3 passed: TIS clip at most 0.49% (limit 2%), |ratio-1| at most 0.032 (limit 0.1).
  • The exit 134 fix passed on card (10 rounds, 2 cuts, no abort).

Not verified

  • Only 2 repeats per group, seeds 17 and 29.
  • No successful reload was observed inside a cut window.
  • verl is not covered.

🤖 Generated with Claude Code

chowhsu and others added 23 commits October 20, 2026 15:48
@chowhsu @claude
E6: quantized rollout tiers with an FFN/routed-expert scope rule
ce3822d 
Add a backend-neutral rollout quantization setting
(--rl-rollout-quant-tier 0..3, one flag definition for launcher and island):

- tier 0: BF16 (default, no change);
- tier 1: the sampler engine quantizes in-scope linears to FP8 itself and keeps
  an FP8 KV cache. Miles/SGLang: --sglang-quantization fp8 and
  --sglang-kv-cache-dtype fp8_e4m3, with the ignore list passed through
  SGLANG_FP8_IGNORED_LAYERS in the Ray job runtime_env. verl/vLLM: verl's own
  FP8 rollout config with our explicit ignored_layers through
  engine_kwargs.vllm.hf_overrides;
- tier 2: the trainer side converts each BF16 release to FP8 W8A8 per channel
  (compressed-tensors float-quantized, the FP8_DYNAMIC preset) and sampler
  islands load it. Reuses the N8 requant pipeline with a new scheme whose
  target layout is derived from the BF16 base and the scope rule;
- tier 3: 4-bit, interface only (refused at launch).

Scope rule (yeto/rl/quant/scope.py): a dense model quantizes only its FFN
linears, a MoE model only its routed experts; router, shared experts, lm_head,
embeddings, norms and attention stay BF16. The launcher refuses tier 3, cards
without FP8 and tier 2 without sampler islands or with LoRA; island entries
refuse a tier 1 plan that would quantize an out-of-scope module; sampler
loaders refuse an FP8 release outside the scope before touching the engine.

Add yeto.rl.quant.gate: logprob difference (mean, p50/p90/p99), TIS
truncation share and tokens per second of a quantized engine against BF16 on
one prompt set, with configurable criteria and the report written to disk.

CPU only. FP8 values equal compressed-tensors 0.18 bit for bit; the ignore
lists checked against SGLang 2fa88018's own skip functions. Not run on a GPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
E6: FP8 card allowlist from the GPU spec; keep tier-0 argv golden
20f603e 
- Read the cards of --gpu with yeto.gpu_spec and accept tiers 1 and 2 only on
  cards with FP8 tensor cores (L4, L40S, RTX-6000-Ada, RTX-PRO-6000, H100,
  H200, B200); every other card is refused before launch.
- Add the yeto_rollout_quant_tier runtime attribute only for a quantized tier,
  so tier-0 Miles launches keep their golden argv and runtime attributes.
- Test the Miles translation and the island-task forwarding of the tier.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
E6: engine sampler tier settings, dense expert count, CPU proofs
6b74178 
- Engine sampler islands: the launch layer writes rollout_quant {tier, apply}
  into the sampler spec (stage1.sampler_spec); the entry applies it on the
  node (sampler_engines.quant_tier). Tier 1 on sglang-server fills
  --quantization fp8, --kv-cache-dtype fp8_e4m3 and SGLANG_FP8_IGNORED_LAYERS
  from the checked online plan; vllm-server is refused (stand-alone vLLM online
  FP8 has no ignore list). Tier 2 converts the base to the FP8 scheme and sets
  quant_scheme, expected_format, release_dir and the routed-expert count. A spec
  that already quantizes is refused. Stage 1 refuses a tier on top of its own
  quant section.
- Post-load expert_count: an expected count of 0 (dense model) passes without
  an engine report; the release index count is checked before the load.
- requant: drop a tied lm_head.weight that an exporter writes although the
  base ties it to the embeddings (tie_word_embeddings).
- Tests: the ignore list reaches the process started by Miles'
  launch_bound_subprocess; Miles' convert_qwen2_to_hf names equal the
  Qwen3-4B-Instruct-2507 index names (fixture from the real index).
- Probe (Modal CPU, verl image): Hydra overrides -> RolloutConfig -> verl's
  CLI builder -> vLLM 0.29 argument parser -> Fp8Config. The received
  ignored_layers equal the sent list; qkv_proj and o_proj skipped,
  gate_up_proj and down_proj quantized; kv_cache_dtype fp8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
E6: load Miles process_utils from its file in the env test
a215af7 
Importing it as a package put the real miles package into sys.modules and
broke a later test that fakes miles (test_rl_reward_success).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
E6: fixes from the H100 gate runs (fused ignore aliases, token scorin…
4854dda 
…g, BF16 KV)

- scope.with_fused_aliases: the tier 2 compressed-tensors ignore list also names
  qkv_proj / gate_up_proj when all shards are ignored. SGLang 2fa88018 matches
  fused names only through packed_modules_mapping, which Qwen3ForCausalLM does
  not set, so qkv_proj got the FP8 scheme on the card (e6-gate-t2).
- gate: score by token ids when the engine can (SGLangServerEngine.score_tokens);
  text scoring dropped the trailing EOS and left 96% of completions unaligned.
- tier 1: KV cache stays BF16 (SGLang and vLLM). Uncalibrated FP8 KV failed the
  gate on Qwen3-4B (p99 |diff| 0.82 vs 0.46 with BF16 KV, limit 0.5).
- tools/probes/s26_e6: Modal gate and verl RL drivers, a lock wrapper, and an
  image overlay for the SGLang fork's FP8 activation cast in Qwen2/Qwen3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
E6: refuse tier 1 on verl, set verl max_model_len, native-FP8 gate pr…
ffe86d0 
…obes

- verl max_model_len = prompt + response: Qwen3-4B-Instruct-2507's 262144
  context needed 36 GiB of KV next to the actor and vLLM refused to start.
- Tier 1 on verl is refused at launch and on the island. With verl's FP8
  config vLLM treats the BF16 checkpoint as FP8-serialized and generates
  garbage until a base sync re-quantizes it; yeto verl is LoRA-only, so the
  base is never sent (e6-t1-verl-a-r2: 512-token answers, NaN log-probs;
  e6-vllm-fp8-probe). Plain vLLM online FP8 works with LoRA but quantizes
  attention, which failed the gate as SGLang online FP8 of every linear.
- Gate driver: framework-native variants (Miles block FP8 converter, SGLang
  online FP8 without an ignore list); vLLM probe; verl metrics parser.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
E6: pin the FP8-fixed ports image; tier 1 trainer FP8 source (Miles n…
d9a2703 
…ative + ignore list)

- Pins: SGLang fork 10afe11a7 (Qwen2 MLP / Qwen3 attention no longer cast the
  activation to an FP8 projection's dtype), Miles fork 9462de532 (quantize_params_fp8
  and convert_hf_to_fp8 honor the FP8 ignore list; --rollout-hf-checkpoint), image
  yeto-miles-ports:9462de5-10afe11 @sha256:6e0a97d2. Build record under
  openspec/changes/rl-infra-spec/evidence/ports-image/2026-10-21-9462de5-10afe11/.
- make_layer: Miles tools/ scripts may change in the overlay (nothing is built).
- --rl-rollout-quant-fp8-source {engine,trainer} (tier 1): "trainer" starts the
  SGLang engines on Miles' 128x128 block-FP8 conversion of the base with our
  ignore list in BF16 (entry.prepare_native_fp8_base, scope-checked by
  tiers.check_native_base) and Miles block-quantizes every weight update by that
  config. No SGLang online flag or ignore-list environment in that case.
- tools/probes/s26_e6: image check on a Modal CPU container; gate variant
  miles-native-block-ignore.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
E6: Miles RL drivers for the tier groups (Modal H100!)
7f100f4 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
E6: full-parameter pair needs a fixed-partition sampler island
8d1df81 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
E6: Dockerfile and tests follow the new image pins; neutral flag help
e3da116 
The sampler-snapshot G-L8 records stay at SGLang 2fa880182, where they were
verified; the new pin is refused for that opt-in mode until G-L8 is re-run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
E6: pair islands are 2xH100 each (fixed partition applies to every is…
2011906 
…land)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
E6: tier 2 also runs as one launch per role (--rl-sample-role); pair …
aa3b97c 
…driver uses two launches

A single launch with --rl-island-sample-role cannot give a full-parameter
trainer island (no engine) and sampler island (dedicated engine card)
different placements, so the E6 pair runs as trainer and sampler launches on
one store, as S25 C3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
E6: per-round parser of the Miles RL runs
d7c79be 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Modal store: no reload while a writer holds uncommitted files
be553cf 
A Modal Volume reload remounts the volume and drops local files that are not
committed yet. On the tier 2 trainer island (e6-t2-pair-a) the main thread's
reload removed the FP8 releaser thread's half-written release
(FileNotFoundError on RELEASE.json.tmp) and the releaser stopped.
CheckpointStore.writing() (no-op by default) now brackets a policy release;
ModalVolumeStore skips a reload while any writer of the same volume holds it
and a writer waits for a reload in flight.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
E6: pair trainer keeps the Adam state on the host (OOM on one H100)
beaf4e0 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
Modal store: the write guard is a container-wide flock
1fe2c76 
The rollout actors that reload the Volume are other processes than the tier 2
releaser thread (e6-t2-pair-c lost LATEST.json.tmp with the in-process guard).
Writers now hold a shared flock on a container-local file per Volume; a reload
takes it exclusive without blocking and is skipped while a writer holds it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
E6: tier 1 on Miles defaults to the trainer FP8 source
f4cd14e 
Miles' native block FP8 with the ignore list has the larger gate margin
(p99 |diff| 0.43 vs 0.46-0.49 for the engine's online FP8, limit 0.5).
--rl-rollout-quant-fp8-source now defaults to trainer for tier 1 on Miles
and to engine otherwise; engine stays selectable. The launcher resolves the
default once so the island flags and the run config carry it explicitly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
RL: full parameters on one no-sync island with dedicated engine cards
bce94af 
--tuning full was accepted only on sampler / trainer buffer islands. One
island with --rl-single-island-no-sync and --rl-placement fixed-partition
now also trains every parameter: Miles' own full-parameter weight update
feeds its dedicated engine cards each round (S26 E6 uses it to check that
Miles block-quantizes every update for the tier 1 trainer source). A
colocated full-parameter island is still refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
E6: pin the ports image with the SGLang compressed-tensors FP8 reload
8fe48c9 
SGLang fork 754605964 (yeto/fp8-act-cast): TENSOR / CHANNEL FP8 linears
reload in place through update_weights_from_disk, which the tier 2 sampler
needs for each FP8 release. Image 9462de5-7546059 @sha256:37acaca3...; the
image check also asserts the fix and runs its unit test in the image. The
E6 driver gains the full-parameter dedicated-engine groups and a SEED knob,
and the pair trainer island uses 2 cards instead of host Adam state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
E6: neutral fp8-source help; full-parameter dedicated-engine group is…
f74ac99 
… 1 train + 1 engine card

Engines on the island need DP=1 (e6-t*-full-a refused 2 training cards), so the
Adam state lives on the host (200 GiB container).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
E6: full-parameter dedicated-engine group runs without round cuts
dfa4a58 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
E6: hold the store writer guard across a round cut; cuts with host Adam
84011ee 
e6-t2-pair-e: the trainer island exited 134 after six rounds. Its rank
0 save_cut_shard failed with EPERM on makedirs of the store directory
on the Modal Volume; Miles marked the cell errored, the next train call
raised "Cannot recover when all cells are dead" and the process aborted
at exit ("FATAL: exception not rethrown"). The cut held no writer guard,
so a concurrent Volume reload (sample-buffer / release polling in the
same container) could remount the Volume while the shard was written.

- RoundCutCheckpoint.save holds store.writing() from the first shard to
  the store commit; reloads are skipped meanwhile (they are best effort).
- A successful Volume reload is logged, so a remount can be lined up
  with a writer failure next time.
- The full-parameter dedicated-engine probe group keeps the checkpoint
  store again: the Miles fork (ed5f64fd7) now exports and restores the
  host (HybridDeviceOptimizer) optimizer state; the DistOpt protocol
  test also runs against that fork commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chowhsu @claude
E6: 3c rerun cuts every 5 rounds; stop both apps after 15 min without…
2ab01f8 
… a new step

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@michaellchung michaellchung mentioned this pull request Oct 24, 2026
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@michaellchung @chowhsu