S26 E2: async buffer depth defaults to the policy-age limit; opt-in adaptive TIS

docker/miles-ports/Dockerfile

@@ -19,12 +19,12 @@
 FROM docker.io/radixark/miles@sha256:4d69750720eb1a4b99976a3fed45ed018e1fc4fff3d240667fa17d258dfc6e0c
 
 ARG MILES_REPOSITORY=https://github.com/michaellchung/miles
-ARG MILES_COMMIT=44d41efd0a0428cb12b4ee34422de6f5e1ccb9d9
+ARG MILES_COMMIT=9462de532f747cf688e090bbaf6b24f80e378fe8
 ARG SGLANG_REPOSITORY=https://github.com/michaellchung/sglang
-ARG SGLANG_COMMIT=2fa880182eefdb31e64d8ea70317be27f9f32121
-# setuptools-scm version of SGLANG_COMMIT (describe v0.5.20-69-g4e4148f1b);
+ARG SGLANG_COMMIT=754605964776322226a073763a6f636aa8233932
+# setuptools-scm version of SGLANG_COMMIT (describe v0.5.20-72-g754605964);
 # pinned because the one-commit clone has no tags to describe.
-ARG SGLANG_VERSION=0.5.21.dev70+g2fa8801
+ARG SGLANG_VERSION=0.5.21.dev72+g7546059
 
 # Miles fork -> /root/miles (same path the base installed editable).
 RUN rm -rf /root/miles \

docker/miles-ports/make_layer.py

@@ -173,7 +173,8 @@ def main() -> None:
     p.add_argument("--out", type=Path, required=True)
     a = p.parse_args()
 
-    check_pure_python(a.miles_src, a.miles_base_commit, a.miles_commit, ("miles/", "miles_plugins/"))
+    # tools/: stand-alone scripts run from the checkout (e.g. convert_hf_to_fp8.py), nothing built
+    check_pure_python(a.miles_src, a.miles_base_commit, a.miles_commit, ("miles/", "miles_plugins/", "tools/"))
     check_pure_python(a.sglang_src, a.sglang_base_commit, a.sglang_commit, ("python/sglang/",))
 
     entries: dict[str, tuple[str, bytes | None, int, str | None]] = {}

docs/PROTOCOL.md

@@ -32,7 +32,7 @@ or allocation overflows are errors rather than panics.
 
 | type | name | direction | payload |
 |---:|---|---|---|
-| 1 | HELLO | learner → syncer | protocol_version:u16 (=4), learner_id:u32, connection_generation:u64, dtype:u8 (1=f32, 2=bf16, 3=q4), num_fragments:u32, per-fragment layout, layout_fingerprint:[u8;32], session_contract_hash:[u8;32], optional syncer_profile_hash:[u8;32], num_streams:u16, optional compat_group trailer (`b"YCG1"`, u32 length, 1–64 bytes utf-8) |
+| 1 | HELLO | learner → syncer | protocol_version:u16 (=4), learner_id:u32, connection_generation:u64, dtype:u8 (1=f32, 2=bf16, 3=q4, 4=f32 with bf16 pushes, 5=f32 with int8 pushes; flag 0x80 = bf16 downlink deltas), num_fragments:u32, per-fragment layout, layout_fingerprint:[u8;32], session_contract_hash:[u8;32], optional syncer_profile_hash:[u8;32], num_streams:u16, optional compat_group trailer (`b"YCG1"`, u32 length, 1–64 bytes utf-8) |
 | 2 | INIT_PARAMS | learner → syncer | fragment_id:u32, full tensor bytes; only learner 0 may initialize |
 | 3 | PULL_REQ | syncer → learner | fragment_id:u32, global_step:u64, round_attempt:u32 |
 | 4 | PUSH_FRAGMENT | learner → syncer | learner_id:u32, fragment_id:u32, global_step:u64, round_attempt:u32, base_version:u64, local_step:u64, c_steps:u32, c_tokens:u64, base-relative learner-delta bytes |
@@ -223,6 +223,37 @@ The coordinator is one sequential scheduler actor. Pipelined rounds target
 distinct fragments, and merge + version update + broadcast for a completion
 is serialized. No two completions mutate one fragment concurrently.
 
+## bf16 pushes over an f32 session (dtype 4)
+
+Dtype 4 keeps INIT, BCAST and FINAL exact f32 and sends only PUSH deltas as
+bf16 (round-to-nearest-even). The syncer decodes and negates them like dtype 2
+and keeps f32 masters and momentum. The full-parameter policy sweep accepts
+dtype 1 or 4 (S26 E3, `--rl-sync-push-dtype bf16`).
+
+## int8 pushes over an f32 session (dtype 5)
+
+Dtype 5 sends PUSH deltas as block-scaled int8: u32 block size B
+(32..1048576), then per block of B values (the last block may be shorter) an
+f32 scale (block absmax / 127) and one i8 per value (`-128` is invalid).
+A value decodes to `q * scale`, so its error is at most absmax / 254. The
+learner picks B (launcher `--rl-sync-push-int8-block`, default 128); it is
+bound into the role-stream session contract. INIT, BCAST and FINAL stay f32.
+
+## bf16 downlink deltas (dtype flag 0x80)
+
+On an f32-bulk session (dtype 1, 4 or 5) the flag 0x80 changes BCAST_FRAGMENT
+to `fragment_id:u32, version:u64, kind:u8, base_version:u64,
+anchor_sha256:[u8;32]`, then the tensor: kind 0 (FULL) carries f32 values,
+kind 1 (DELTA) carries `bf16(theta - anchor)`. Syncer and learner hold the
+same f32 anchor per fragment: FULL sets it, DELTA adds the decoded delta in
+f32; the digest is checked after every update. Every N-th version of a
+fragment is FULL (`--bcast-delta-full-every N`, N >= 2; the syncer refuses a
+flagged session without it and the reverse). Initial and reconnect re-sends
+are always FULL with the shared anchor. A learner with a missing or wrong
+anchor drops its connection, and the reconnect re-send restores it. The
+syncer's masters, momentum, checkpoints and FINAL are unchanged and exact;
+anchors are not checkpointed, so a resumed syncer starts with FULL.
+
 ## Q4 pushes
 
 Q4 uses the same learner-delta semantics. INIT and BCAST stay bf16; PUSH is

openspec/changes/rl-infra-spec/evidence/ports-image/2026-10-21-9462de5-10afe11/build-record.json

@@ -0,0 +1,12 @@
+{
+  "tag": "ghcr.io/michaellchung/yeto-miles-ports:9462de5-10afe11",
+  "digest": "sha256:6e0a97d203cdd062c9f2fa35b6b834575cf05ee1f71fb45e921d8803b620f072",
+  "layer_tar_sha256": "08874741ace9733a576a2fa2b87736a6ebc46c8896806a3e025768b84e27e4ed",
+  "base": "docker.io/radixark/miles@sha256:4d69750720eb1a4b99976a3fed45ed018e1fc4fff3d240667fa17d258dfc6e0c",
+  "base_index": "sha256:90940828dcd4d54fd907ff668b43537cbd94778047580e4160d6560af548b74d",
+  "miles_commit": "9462de532f747cf688e090bbaf6b24f80e378fe8",
+  "sglang_commit": "10afe11a7ef2b53cedd9895cc4450474d85108b9",
+  "sglang_version": "0.5.21.dev71+g10afe11",
+  "peft_version": "0.20.0",
+  "peft_wheel_sha256": "0fbba16ffebfad3de96e06f2da6860fd860292324b85b6141909fa1e26ea9233"
+}

openspec/changes/rl-infra-spec/evidence/ports-image/2026-10-21-9462de5-10afe11/image-manifest.json

@@ -0,0 +1,50 @@
+{
+  "base": {
+    "index_digest": "sha256:90940828dcd4d54fd907ff668b43537cbd94778047580e4160d6560af548b74d",
+    "manifest_digest": "sha256:4d69750720eb1a4b99976a3fed45ed018e1fc4fff3d240667fa17d258dfc6e0c",
+    "miles_commit": "9e4260de047a704208535c0e90c531929879ab40",
+    "platform": "linux/amd64",
+    "repository": "docker.io/radixark/miles",
+    "sglang_commit": "e5bba1f9c433da5352e521a8bb826bab960e7836"
+  },
+  "builder": "scripts/build_miles_ports_image.sh (crane append); docker/miles-ports/Dockerfile equivalent",
+  "image": "ghcr.io/michaellchung/yeto-miles-ports",
+  "megatron_bridge": {
+    "commit": "8cd3466d14d2337c8492827b3712482c2b3e4866",
+    "patches": [
+      "lazy-modelopt (docker/miles-ports/megatron_bridge_init.py)"
+    ]
+  },
+  "megatron_lm": {
+    "commit": "a84b105473eca9c9c6e49dbf1c8aa460c5cf796e",
+    "patches": [
+      "lazy-modelopt (docker/miles-ports/megatron_lm_lazy_modelopt.patch)"
+    ],
+    "path": "/root/Megatron-LM",
+    "repository": "https://github.com/radixark/Megatron-LM"
+  },
+  "miles": {
+    "commit": "9462de532f747cf688e090bbaf6b24f80e378fe8",
+    "install": "base's editable install (pip install -e /root/miles --no-deps) kept (same path); /root is on the base's sys.path, so without PYTHONPATH import miles is a namespace package over the /root/miles checkout -- yeto runs with PYTHONPATH=$HOME/miles (regular package)",
+    "path": "/root/miles",
+    "repository": "https://github.com/michaellchung/miles",
+    "upstream_commit": "9e4260de047a704208535c0e90c531929879ab40"
+  },
+  "peft": {
+    "install": "wheel unpacked into the base's site-packages (pip install --no-deps equivalent)",
+    "version": "0.20.0",
+    "wheel": "peft-0.20.0-py3-none-any.whl",
+    "wheel_sha256": "0fbba16ffebfad3de96e06f2da6860fd860292324b85b6141909fa1e26ea9233"
+  },
+  "pure_python_overlay": true,
+  "schema": 1,
+  "sglang": {
+    "commit": "10afe11a7ef2b53cedd9895cc4450474d85108b9",
+    "install": "pip install -e /sgl-workspace/sglang/python[all] --no-deps (base's editable install, same path)",
+    "path": "/sgl-workspace/sglang",
+    "repository": "https://github.com/michaellchung/sglang",
+    "upstream_commit": "571212b636baca45e10fa3b4da11a289123f3235",
+    "version": "0.5.21.dev71+g10afe11"
+  },
+  "tag": "9462de5-10afe11"
+}

openspec/changes/rl-infra-spec/evidence/ports-image/2026-10-21-9462de5-7546059/build-record.json

@@ -0,0 +1,12 @@
+{
+  "tag": "ghcr.io/michaellchung/yeto-miles-ports:9462de5-7546059",
+  "digest": "sha256:37acaca3e1eee8adc61bf3edbbd706ece36e40e9b7fd5463e54f56e3f4cc9b2b",
+  "layer_tar_sha256": "4eee8887e4d85fe79fad9da706bbcef5b734d0deb74206427651a9fd8cf270b4",
+  "base": "docker.io/radixark/miles@sha256:4d69750720eb1a4b99976a3fed45ed018e1fc4fff3d240667fa17d258dfc6e0c",
+  "base_index": "sha256:90940828dcd4d54fd907ff668b43537cbd94778047580e4160d6560af548b74d",
+  "miles_commit": "9462de532f747cf688e090bbaf6b24f80e378fe8",
+  "sglang_commit": "754605964776322226a073763a6f636aa8233932",
+  "sglang_version": "0.5.21.dev72+g7546059",
+  "peft_version": "0.20.0",
+  "peft_wheel_sha256": "0fbba16ffebfad3de96e06f2da6860fd860292324b85b6141909fa1e26ea9233"
+}

openspec/changes/rl-infra-spec/evidence/ports-image/2026-10-21-9462de5-7546059/image-manifest.json

@@ -0,0 +1,50 @@
+{
+  "base": {
+    "index_digest": "sha256:90940828dcd4d54fd907ff668b43537cbd94778047580e4160d6560af548b74d",
+    "manifest_digest": "sha256:4d69750720eb1a4b99976a3fed45ed018e1fc4fff3d240667fa17d258dfc6e0c",
+    "miles_commit": "9e4260de047a704208535c0e90c531929879ab40",
+    "platform": "linux/amd64",
+    "repository": "docker.io/radixark/miles",
+    "sglang_commit": "e5bba1f9c433da5352e521a8bb826bab960e7836"
+  },
+  "builder": "scripts/build_miles_ports_image.sh (crane append); docker/miles-ports/Dockerfile equivalent",
+  "image": "ghcr.io/michaellchung/yeto-miles-ports",
+  "megatron_bridge": {
+    "commit": "8cd3466d14d2337c8492827b3712482c2b3e4866",
+    "patches": [
+      "lazy-modelopt (docker/miles-ports/megatron_bridge_init.py)"
+    ]
+  },
+  "megatron_lm": {
+    "commit": "a84b105473eca9c9c6e49dbf1c8aa460c5cf796e",
+    "patches": [
+      "lazy-modelopt (docker/miles-ports/megatron_lm_lazy_modelopt.patch)"
+    ],
+    "path": "/root/Megatron-LM",
+    "repository": "https://github.com/radixark/Megatron-LM"
+  },
+  "miles": {
+    "commit": "9462de532f747cf688e090bbaf6b24f80e378fe8",
+    "install": "base's editable install (pip install -e /root/miles --no-deps) kept (same path); /root is on the base's sys.path, so without PYTHONPATH import miles is a namespace package over the /root/miles checkout -- yeto runs with PYTHONPATH=$HOME/miles (regular package)",
+    "path": "/root/miles",
+    "repository": "https://github.com/michaellchung/miles",
+    "upstream_commit": "9e4260de047a704208535c0e90c531929879ab40"
+  },
+  "peft": {
+    "install": "wheel unpacked into the base's site-packages (pip install --no-deps equivalent)",
+    "version": "0.20.0",
+    "wheel": "peft-0.20.0-py3-none-any.whl",
+    "wheel_sha256": "0fbba16ffebfad3de96e06f2da6860fd860292324b85b6141909fa1e26ea9233"
+  },
+  "pure_python_overlay": true,
+  "schema": 1,
+  "sglang": {
+    "commit": "754605964776322226a073763a6f636aa8233932",
+    "install": "pip install -e /sgl-workspace/sglang/python[all] --no-deps (base's editable install, same path)",
+    "path": "/sgl-workspace/sglang",
+    "repository": "https://github.com/michaellchung/sglang",
+    "upstream_commit": "571212b636baca45e10fa3b4da11a289123f3235",
+    "version": "0.5.21.dev72+g7546059"
+  },
+  "tag": "9462de5-7546059"
+}

openspec/changes/rl-resume-from-checkpoint/design.md

@@ -79,6 +79,7 @@ Modal 文档：退出处理函数有 30 秒宽限，被抢占时也调用；GPU
 - **单岛切点**（沿用格式）：adapter、优化器、学习率调度器、随机数、iteration、consumed_samples、pointer（下一轮号、策略版本、策略哈希、启动序号）+ 新增 `run_fingerprint`（算法、模型、LoRA 形状、并行布局、学习率配置、数据集指纹与洗牌种子）与 `lr_at_next_round`（下一轮应使用的学习率）与 `consumed_prompt_ids_digest`（已用题号集合的摘要，详细列表放账本）。
 - **多岛**：每个岛存自己的切点（同上，路径 `<store>/islands/<岛名>/`）；syncer 检查点同步到 `<store>/syncer/`；切点 pointer 多记 `outer_version`。
 - **频率**：`--rl-cut-every N`（默认：0.6B 为 1，FN 为 2）；最后一轮、收到"我方停机"时总是存一次。
+- **Default since 2026-10-21 (E1)**: `--rl-cut-every` defaults to 4; publication checks (policy hash, engine read-back) run on the first publication and every 4th round; `--rl-verify-every-round` restores per-round checks. See the requirement "Round-cut interval and publication checks" in specs/rl-resume-from-checkpoint/spec.md.
 - **保留**：`--rl-cut-keep K`（默认 2）；新切点 MANIFEST 落盘并提交后才删旧的。
 - **目录**：`<store>/<运行名>/cuts/<轮号>/{shards…, manifest.json, MANIFEST.sha256}` + `<store>/<运行名>/LATEST`（最后写，指向最近完整切点）。
 

openspec/changes/rl-resume-from-checkpoint/specs/rl-resume-from-checkpoint/spec.md

@@ -49,3 +49,10 @@ launcher SHALL 接受 `--rl-checkpoint-store modal-volume://<卷名>[/前缀]`
 #### Scenario: 续训后的 tape
 - **WHEN** 第二次启动重跑了第一次未进切点的第 4 轮
 - **THEN** 第一次的第 4 轮记录仍在，`rl_resume.discarded_rounds` 含 4
+
+### Requirement: Round-cut interval and publication checks (E1, 2026-10-21)
+The product default round-cut interval SHALL be 4 rounds (`--rl-cut-every 4`; `--rl-cut-every 1` cuts every round), plus the last round and our own stop. A publication SHALL hash the policy and read the engine weight checksums back on the first publication, on every round whose number is a multiple of the cut interval (which includes every round that keeps a round cut), and on every round with `--rl-verify-every-round`. Every other publication SHALL still check that the trainer holds the published policy (trainer weights version) and SHALL be recorded with `sync/publication_verified: false` (verified publications carry no such field). A publication that was not verified MUST NOT be committed as a round cut.
+
+#### Scenario: an unverified round at a cut
+- **WHEN** a cut is due at a safe point whose published policy was not verified
+- **THEN** the cut is refused ("not verified") and the previous pointer stays the committed one