S25-J: round-3 integration — sharded requant, images, verl cuts, device families, stage-1 launcher

docker/README.md

@@ -0,0 +1,43 @@
+# Engine images
+
+| Directory | Image | CPU arch | Device family | Roles | Status |
+|---|---|---|---|---|---|
+| `miles-ports/` | `yeto-miles-ports` | x86_64 | cuda | trainer + sampler | built and pushed (see the Dockerfile header) |
+| `yeto-release/` | release layer on `yeto-miles-ports` | x86_64 | cuda | — | built once (2026-10-19) |
+| `ppu-train/` | `yeto-ppu-train` | x86_64 | ppu | trainer + sampler | written, never built (S25 N0.2) |
+| `npu-sampler/` | `yeto-npu-sampler` | aarch64 | npu | sampler | written, never built (S25 N0.2) |
+
+`yeto/launch/engine_images.py` is the catalog the launcher reads. A node of
+the SSH inventory declares `cpu_arch`; a node with no `image` takes the inventory's
+`images[cpu_arch]`, and a node whose image is a catalog image of another
+architecture or device family is refused before any SSH (N1.6).
+
+## Version locks
+
+Each image directory has `versions.lock.json`. It is the single source of the
+versions the image asserts, and it holds the decisions behind them (the
+`decisions` list). `docker/image-assert/assert_image_versions.py` reads it:
+
+* the image entrypoint (`docker/image-assert/entrypoint.sh`) runs it at every
+  container start and exits 1 before the engine starts when a check fails;
+* `yeto-ppu-train` also runs it at build time, so an image whose own lock does
+  not hold is never pushed. `yeto-npu-sampler` cannot: `import torch_npu`
+  needs the NPU driver, which no build host has.
+
+Checked: CPU architecture, Python, package versions and imports, build-time
+commits (`/opt/yeto/image/commits.json`), the CANN toolkit version, the host
+driver version and required files. An entry with `"version": null` is recorded
+in the report but not compared; `"optional": true` reports a missing package or
+failed import without failing (TransformerEngine and tilelang on PPU, whose
+support the first PPU probe decides).
+
+## Open points (both new images)
+
+1. Neither image was built: no base digest, no `pip check` output, no real
+   version report. Every version in the two locks comes from a survey or from
+   upstream documentation, not from a built image.
+2. `yeto-ppu-train`: the base image's registry path and digest are unknown
+   (N0.1 read the tag `inference-xpu-pytorch-opt:v0.5.17` on ppu1 only), and
+   `MEGATRON_COMMIT` has no value yet.
+3. `yeto-npu-sampler`: the ModelArts fixes come from a build context that does
+   not exist in this repository yet, and the arm64 release layer is a follow-up.

docker/image-assert/assert_image_versions.py

@@ -0,0 +1,180 @@
+#!/usr/bin/env python3
+"""Check an engine image against its version lock at container start (S25 N0.2).
+
+Usage: ``assert_image_versions.py LOCK.json`` (the image entrypoint runs it
+before the command). Exit 0 when every check passes, 1 otherwise; the report
+goes to stderr, one line per check, and to ``--report`` as JSON when given.
+
+The lock (``docker/<image>/versions.lock.json``) lists:
+
+* ``cpu_arch``: must equal ``platform.machine()``;
+* ``python``: ``major.minor`` prefix of the interpreter version (null: record);
+* ``packages``: ``{dist: {"version", "import", "optional"}}``. ``version`` is
+  compared with the public part of the installed version (``2.10.0+abc`` ->
+  ``2.10.0``); null means "record only". ``import`` names a module the image
+  must import. ``optional`` true turns a missing package or failed import into
+  a report line instead of a failure (used where the probe on the real machine
+  decides, e.g. TransformerEngine on PPU);
+* ``commits``: ``{name: sha}`` compared with ``/opt/yeto/image/commits.json``
+  written at build time (null: record);
+* ``cann``: ``{"version", "version_files"}``: the toolkit version read from the
+  first existing file (``version=`` / ``Version=`` line) must start with
+  ``version``;
+* ``driver``: ``{"min", "enforce", "version_file"}``: the host driver mounted
+  into the container; ``min`` null means record only; ``enforce`` false
+  reports a driver below ``min`` without failing (kept while a probe on the
+  real host decides, see the NPU lock);
+* ``required_files``: paths that must exist (patches, stubs).
+
+Stdlib only: it runs before anything else in the image is trusted.
+"""
+
+from __future__ import annotations
+
+import argparse
+import importlib
+import importlib.metadata
+import json
+import platform
+import re
+import sys
+from pathlib import Path
+
+COMMITS_FILE = Path("/opt/yeto/image/commits.json")
+_VERSION_LINE = re.compile(r"^\s*(?:version|Version)\s*=\s*(\S+)", re.MULTILINE)
+
+
+class Report:
+    def __init__(self) -> None:
+        self.rows: list[dict] = []
+
+    def add(self, check: str, ok: bool, detail: str, *, optional: bool = False) -> None:
+        status = "ok" if ok else ("not-enforced" if optional else "FAIL")
+        self.rows.append({"check": check, "status": status, "detail": detail})
+        print(f"[image-assert] {status:16} {check}: {detail}", file=sys.stderr)
+
+    @property
+    def failed(self) -> list[dict]:
+        return [row for row in self.rows if row["status"] == "FAIL"]
+
+
+def public_version(version: str) -> str:
+    return version.split("+", 1)[0]
+
+
+def _version_tuple(version: str) -> tuple[int, ...]:
+    return tuple(int(part) for part in re.findall(r"\d+", version))
+
+
+def check_packages(packages: dict, report: Report) -> None:
+    for dist, spec in packages.items():
+        optional = bool(spec.get("optional"))
+        want = spec.get("version")
+        try:
+            have = importlib.metadata.version(dist)
+        except importlib.metadata.PackageNotFoundError:
+            report.add(f"package {dist}", False, "not installed", optional=optional)
+            continue
+        if want is None:
+            report.add(f"package {dist}", True, f"{have} (recorded, not pinned)")
+        else:
+            ok = public_version(have) == want
+            report.add(f"package {dist}", ok, f"{have} (lock {want})", optional=optional)
+            if not ok:
+                continue
+        module = spec.get("import")
+        if module:
+            try:
+                importlib.import_module(module)
+            except Exception as exc:  # noqa: BLE001 - any import failure is the finding
+                report.add(f"import {module}", False, f"{type(exc).__name__}: {exc}", optional=optional)
+            else:
+                report.add(f"import {module}", True, "imported")
+
+
+def check_commits(commits: dict, report: Report, commits_file: Path) -> None:
+    if not commits:
+        return
+    try:
+        built = json.loads(commits_file.read_text())
+    except (OSError, ValueError) as exc:
+        report.add("commits", False, f"cannot read {commits_file}: {exc}")
+        return
+    for name, want in commits.items():
+        have = built.get(name)
+        if want is None:
+            report.add(f"commit {name}", have is not None, f"{have} (recorded, not pinned)")
+        else:
+            report.add(f"commit {name}", have == want, f"{have} (lock {want})")
+
+
+def _read_version(files: list[str]) -> tuple[str | None, str | None]:
+    for name in files:
+        path = Path(name)
+        if path.is_file():
+            match = _VERSION_LINE.search(path.read_text(errors="replace"))
+            if match:
+                return match.group(1), name
+    return None, None
+
+
+def check_cann(cann: dict | None, report: Report) -> None:
+    if not cann:
+        return
+    have, source = _read_version(cann["version_files"])
+    if have is None:
+        report.add("cann", False, f"no version in {cann['version_files']}")
+    else:
+        report.add("cann", have.startswith(cann["version"]), f"{have} from {source} (lock {cann['version']})")
+
+
+def check_driver(driver: dict | None, report: Report) -> None:
+    if not driver:
+        return
+    have, source = _read_version([driver["version_file"]])
+    want = driver.get("min")
+    if have is None:
+        report.add("driver", False, f"no version in {driver['version_file']} (is the host driver mounted?)")
+    elif want is None:
+        report.add("driver", True, f"{have} from {source} (recorded; no minimum in the lock)")
+    else:
+        report.add("driver", _version_tuple(have) >= _version_tuple(want), f"{have} (minimum {want})",
+                   optional=not driver.get("enforce", True))
+
+
+def run(lock: dict, *, commits_file: Path = COMMITS_FILE, machine: str | None = None) -> Report:
+    report = Report()
+    machine = machine or platform.machine()
+    report.add("cpu_arch", machine == lock["cpu_arch"], f"{machine} (lock {lock['cpu_arch']})")
+    want_py = lock.get("python")
+    have_py = platform.python_version()
+    report.add("python", want_py is None or have_py.startswith(want_py + ".") or have_py == want_py,
+               f"{have_py} (lock {want_py})")
+    check_packages(lock.get("packages", {}), report)
+    check_commits(lock.get("commits", {}), report, commits_file)
+    check_cann(lock.get("cann"), report)
+    check_driver(lock.get("driver"), report)
+    for name in lock.get("required_files", []):
+        report.add(f"file {name}", Path(name).exists(), "present" if Path(name).exists() else "absent")
+    return report
+
+
+def main(argv: list[str] | None = None) -> int:
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    ap.add_argument("lock", type=Path)
+    ap.add_argument("--report", type=Path, help="also write the rows as JSON here")
+    ap.add_argument("--commits-file", type=Path, default=COMMITS_FILE)
+    args = ap.parse_args(argv)
+    lock = json.loads(args.lock.read_text())
+    report = run(lock, commits_file=args.commits_file)
+    if args.report:
+        args.report.write_text(json.dumps({"image": lock.get("image"), "rows": report.rows}, indent=2))
+    if report.failed:
+        print(f"[image-assert] {len(report.failed)} check(s) failed; refusing to start "
+              f"image {lock.get('image')}", file=sys.stderr)
+        return 1
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

docker/image-assert/entrypoint.sh

@@ -0,0 +1,8 @@
+#!/bin/sh
+# Engine image entrypoint (S25 N0.2): check the image against its version lock,
+# then run the command. A failed check stops the container (exit 1) before any
+# engine starts. The report is kept at /tmp/yeto-image-assert.json.
+set -e
+python3 /opt/yeto/image/assert_image_versions.py /opt/yeto/image/versions.lock.json \
+  --report /tmp/yeto-image-assert.json
+exec "$@"

docker/npu-sampler/Dockerfile

@@ -0,0 +1,60 @@
+# yeto-npu-sampler: Ascend 910B (A2) vllm-ascend sampler image, linux/arm64
+# (S25 N0.2, stage 2). WRITTEN, NEVER BUILT.
+#
+# Versions: versions.lock.json in this directory is the single source.
+#
+#   docker build --platform linux/arm64 -f docker/npu-sampler/Dockerfile \
+#     --build-arg BASE_IMAGE=quay.io/ascend/vllm-ascend:v0.30.0rc1@sha256:<digest> \
+#     --build-context fixes=<dir with apply.sh> .
+#
+# Base: the official vllm-ascend v0.30.0rc1 A2 image. It carries the CANN
+# user-space (toolkit, nnal), torch / torch_npu, vLLM and vllm-ascend; the host
+# provides only the driver and firmware, mounted at run time
+# (/usr/local/Ascend/driver). The user ran GLM-5.3-Flash W4A8 on ModelArts with
+# vllm-ascend 0.30.0rc1 (as a rootfs, CANN 9.1); the vllm-ascend compatibility
+# matrix (mkdocs.yml, commit 0ca6a8b) names CANN 9.2.0 and Ascend HDK 26.0.RC1
+# for that release. Host npu1 has driver 24.1.0.3 and CANN 8.0.1: see
+# versions.lock.json "decisions" for how that gap is handled.
+#
+# yeto code is not copied: the sampler adapter runs from /opt/yeto/src (the
+# release layer, as for the other images). The release build script is
+# linux/amd64 only today, so an arm64 release layer is a follow-up.
+ARG BASE_IMAGE
+FROM ${BASE_IMAGE}
+
+ARG BASE_IMAGE
+RUN case "${BASE_IMAGE}" in *@sha256:*) ;; \
+      *) echo "BASE_IMAGE must be pinned by digest: '${BASE_IMAGE}'" >&2; exit 1;; esac
+
+# Freeze the base exactly as docker/ppu-train does: nothing below may replace a
+# base package (torch, torch_npu, vllm, vllm_ascend, triton_ascend, ...).
+RUN mkdir -p /opt/yeto/image \
+ && pip freeze > /opt/yeto/image/base-freeze.txt \
+ && python3 -c 'import importlib.metadata as m; print("\n".join(sorted({f"{d.name}=={d.version}" for d in m.distributions()})))' \
+      > /opt/yeto/image/base-constraints.txt
+ENV PIP_CONSTRAINT=/opt/yeto/image/base-constraints.txt
+
+# yeto sampler-side runtime (HTTP client, weight follower, post-load checks).
+RUN pip install --no-cache-dir cloudpickle "huggingface-hub>=0.34" "safetensors>=0.4" pyyaml "httpx[http2]"
+
+# The two run-time fixes the user needed on ModelArts (A-INF items 15 and 17):
+# the libprofapi symbol stub (CANN 9.1 HCCL hang; whether CANN 9.2 still needs
+# it is unknown) and the W4A8 host-memory staging change (the adapter turns it
+# on with VLLM_ASCEND_W4A8_CPU_STAGE=1). Their source is the user's ModelArts
+# environment, not this repository, so they arrive as a build context:
+# apply.sh installs them and writes /opt/yeto/image/fixes.json
+# ({"fixes": [{"name", "sha256"}, ...]}). The PT_INTERP change is for the
+# no-root rootfs only; a container brings its own loader. The build stops when
+# the context is missing or apply.sh writes nothing.
+COPY --from=fixes . /tmp/yeto-fixes
+RUN sh /tmp/yeto-fixes/apply.sh \
+ && test -s /opt/yeto/image/fixes.json \
+ && rm -rf /tmp/yeto-fixes
+
+COPY docker/npu-sampler/versions.lock.json /opt/yeto/image/versions.lock.json
+COPY docker/image-assert/assert_image_versions.py docker/image-assert/entrypoint.sh /opt/yeto/image/
+ENV PYTHONPATH=/opt/yeto/src
+# No build-time check here: torch_npu imports need the driver, which no build
+# host has. The entrypoint runs the whole check on the node instead.
+ENTRYPOINT ["/opt/yeto/image/entrypoint.sh"]
+CMD ["bash"]

docker/npu-sampler/versions.lock.json

@@ -0,0 +1,47 @@
+{
+  "image": "yeto-npu-sampler",
+  "cpu_arch": "aarch64",
+  "device_family": "npu",
+  "base_image": {
+    "name": "quay.io/ascend/vllm-ascend:v0.30.0rc1",
+    "reference": null,
+    "note": "digest unknown: no pull was made from this host (npu1 reaches hf-mirror, quay was not tested)"
+  },
+  "python": "3.12",
+  "packages": {
+    "torch": {"version": "2.10.0", "import": "torch"},
+    "torch-npu": {"version": "2.10.0.post4", "import": "torch_npu"},
+    "vllm": {"version": "0.30.0", "import": "vllm"},
+    "vllm-ascend": {"version": "0.30.0rc1", "import": "vllm_ascend"},
+    "triton-ascend": {"version": "3.2.2", "import": null, "optional": true},
+    "transformers": {"version": null, "import": "transformers"},
+    "compressed-tensors": {"version": null, "import": "compressed_tensors"},
+    "safetensors": {"version": null, "import": "safetensors"}
+  },
+  "commits": {},
+  "cann": {
+    "version": "9.",
+    "version_files": [
+      "/usr/local/Ascend/ascend-toolkit/latest/version.cfg",
+      "/usr/local/Ascend/ascend-toolkit/latest/aarch64-linux/ascend_toolkit_install.info"
+    ]
+  },
+  "driver": {
+    "min": "26.0",
+    "enforce": false,
+    "version_file": "/usr/local/Ascend/driver/version.info"
+  },
+  "required_files": [
+    "/usr/local/Ascend/driver/version.info",
+    "/opt/yeto/image/fixes.json"
+  ],
+  "decisions": [
+    "CANN gap, chosen option A: the container carries its own CANN user-space (the official vllm-ascend image) and uses the host driver as it is. Reasons: (1) the user already ran this stack on ModelArts without touching a host driver; (2) npu1 is shared with a colleague and is in use, so a driver upgrade needs the owner's consent and a reboot; (3) a container can be removed, a driver cannot. The host keeps CANN 8.0.1; nothing in the container reads it.",
+    "CANN gap, option B (not chosen, kept as the fallback): upgrade the npu1 driver and firmware to the Ascend HDK release that matches CANN 9.2 (26.0.RC1 per the vllm-ascend matrix) and then run either the container or a rootfs. Take this only if option A fails on the driver, and only with the machine owner's agreement.",
+    "driver minimum: recorded, not enforced (enforce false). The container's CANN user-space usually runs on an older driver, but the exact floor for CANN 9.2 on driver 24.1.0.3 is unknown. The first NPU probe reads the driver version and tries `import torch_npu` plus one small device operation; if it fails, option B applies and enforce becomes true with the measured floor.",
+    "cann version: the check only requires a 9.x toolkit in the container, because the base image may ship 9.1 or 9.2 depending on when it was built.",
+    "triton-ascend: optional; the GLM5.3 sampler path uses vllm-ascend's own kernels, and this entry only records what the base carries.",
+    "W4A8 checkpoint: the sampler loads the int-quantized `checkpoint/` directory, not `checkpoint-packed/` (the user's 10-15 result). This is an engine-adapter rule, not an image check.",
+    "transformers / compressed-tensors: recorded, not pinned, until the base image's versions are read."
+  ]
+}