diff --git a/.agent/skills/migrating-webster-models/SKILL.md b/.agent/skills/migrating-webster-models/SKILL.md new file mode 100644 index 0000000..d6caf8c --- /dev/null +++ b/.agent/skills/migrating-webster-models/SKILL.md @@ -0,0 +1,48 @@ +--- +name: migrating-webster-models +description: Use when replacing, upgrading, qualifying, publishing, rolling back, or recovering a model on Webster's fully occupied multi-node GPU pairs, especially vLLM TP deployments behind LiteLLM where public aliases and capability contracts must survive a bounded cold cutover. Triggers include model migration, limited downtime, in-place GPU replacement, AIPerf Weka tuning, TP rank recovery, alias preservation, and rollback planning. +--- + +# Migrating Webster Models + +## Core principle + +Treat a tensor-parallel pair as one fault and deployment unit. Reduce downtime by staging every reversible artifact before GPU handoff, not by rolling ranks independently. + +**REQUIRED BACKGROUND:** Use `webster-cluster` for current nodes, routes, ports, credentials, and maintenance rules. + +## Start with invariants + +Write a migration ledger from [assets/migration-ledger.md](assets/migration-ledger.md). Record: + +- public alias, context accounting, modalities, parameters, response model name, auth behavior, and prices; +- rank roles, TP/PP, listeners, fabric, image/checkpoint/tokenizer hashes, and secrets placement; +- current health, rollback artifacts, owner, timestamps, and sanitized evidence paths. + +Classify each action before executing it: + +| Class | Examples | Rule | +|---|---|---| +| Reversible staging | download, checksum, render disabled config | perform while the old backend is live | +| Cold boundary | stop the old fully occupied TP pair | announce that rollback now requires a full paired boot | +| Publication | enable/repoint a LiteLLM alias | validate offline, batch into one restart, verify immediately | +| Destructive | delete checkpoint, credential, backup, container, or prior route | exclude from migration; require separate authority | + +## Execute in gates + +1. Capture baseline contracts and live probes. Never infer a public contract from the replacement model. +2. Pre-stage immutable weights, image, tokenizer, config, keys, tests, and rollback state on both ranks without initializing the candidate GPU runtime. +3. Establish continuity. If the public alias can route to a compatible existing backend, validate it before retiring the hot backend. If no complete spare pair exists, call the handoff a bounded outage, not blue/green. +4. Drain bounded in-flight work, then stop and fence both ranks together. Confirm listeners and GPU processes are gone. +5. Start one pinned generation with rank 0 authenticated on its private interface and rank 1 headless/keyless. Do not publish until inference, auth rejection, listener isolation, hashes, and NCCL evidence pass. +6. Tune with cache-controlled real workloads. Read [references/playbook.md](references/playbook.md) before using AIPerf or comparing profiles. +7. Qualify the winner for feature correctness, long context, errors, OOM/preemption, restarts, observability, and rollback. For the DeepSeek V4.1 package, require its 17-case suite with repeated determinism, complete tool modes, both JSON modes, negative requests, disconnect queue drain, and failure-log capture armed on both ranks. +8. Generate `private-acceptance.json` last from hashed evidence, validate it with `verify-private-acceptance.py`, then run publication preflight. Render and validate the complete LiteLLM config offline. Preserve old alias metadata explicitly, add the new alias, keep LiteLLM on `127.0.0.1:4446`, back up, restart once, and run direct/private/public probes. +9. Soak, then install coordinated recovery. A failure of either rank removes the pair from service, fences both identities, and starts both only after authoritative absence is proven. +10. Synchronize `webster-cluster` and this skill wherever the fleet state changed. Require exact rendered Hermes content; use `--force` only after reviewing and intentionally replacing divergent consumer edits. + +## Stop conditions + +Fail closed when a rank is unreachable, identities/hashes differ, the worker has a key or HTTP listener, the previous generation cannot be fenced, a benchmark was cancelled/partial, acceptance evidence is stale or captured after `accepted_at`, LiteLLM validation differs from the live file, or rollback artifacts are not readable. + +Prefix every Webster cluster command with `env -u SSH_AUTH_SOCK`. Never print credentials, run two station models together, restart one TP rank alone, or touch Baker while migrating the Webster stations. diff --git a/.agent/skills/migrating-webster-models/agents/openai.yaml b/.agent/skills/migrating-webster-models/agents/openai.yaml new file mode 100644 index 0000000..ce3d6c8 --- /dev/null +++ b/.agent/skills/migrating-webster-models/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Migrating Webster Models" + short_description: "Plan bounded-downtime model cutovers" + default_prompt: "Use $migrating-webster-models to plan and execute a safe Webster model migration." diff --git a/.agent/skills/migrating-webster-models/assets/migration-ledger.md b/.agent/skills/migrating-webster-models/assets/migration-ledger.md new file mode 100644 index 0000000..f318a60 --- /dev/null +++ b/.agent/skills/migrating-webster-models/assets/migration-ledger.md @@ -0,0 +1,70 @@ +# Model migration ledger + +## Scope + +- Change: +- Owner: +- Started (UTC): +- Public aliases: +- Occupied GPU pair: +- Excluded systems: + +## Immutable contract + +| Field | Before | Required after | Evidence | +|---|---|---|---| +| Alias / response model | | | | +| Shared context window | | | | +| Modalities | | | | +| Reasoning / tools / structured output | | | | +| Authentication | | | | +| Timeout / retries / pricing | | | | + +## Artifacts and rollback + +| Artifact | Node/path or identity | Hash/mode | Preserved? | +|---|---|---|---| +| Previous image/checkpoint/tokenizer | | | | +| Previous launch/config | | | | +| Candidate image/checkpoint/tokenizer | | | | +| Rank-0 secret source | | redacted | | +| LiteLLM before-edit backup | | | | + +## Reversibility ledger + +| UTC | Action | Class | Gate/evidence | Rollback time now | +|---|---|---|---|---| +| | | reversible staging / cold boundary / publication / destructive | | | + +## Benchmark matrix + +| Profile | Repetition | Cache state | Requests/errors | Root trees | Peak overlap | Throughput | TTFT p90 | Queue/preemption | Rank health | Result path | +|---|---:|---|---:|---:|---:|---:|---:|---|---|---| +| | 1 | warmup | | | | | | | | | +| | 2 | comparison | | | | | | | | | + +Predeclared disqualifiers: + +Winner and rationale: + +## Cutover and recovery verification + +- Rank 0 private/authenticated: +- Missing/wrong key rejected: +- Rank 1 headless/keyless/no HTTP: +- Matching generation/artifacts/fabric: +- LiteLLM loopback bind and one restart: +- Legacy alias contract probe: +- New alias probe: +- Public ingress policy: +- Langfuse/Prometheus trace: +- Either-rank recovery test: +- Cold rollback rehearsal: + +## Outcome + +- Published (UTC): +- Measured continuity interruption: +- Current cold rollback estimate: +- Preserved evidence paths: +- Known limitations / follow-ups: diff --git a/.agent/skills/migrating-webster-models/references/playbook.md b/.agent/skills/migrating-webster-models/references/playbook.md new file mode 100644 index 0000000..4b77bac --- /dev/null +++ b/.agent/skills/migrating-webster-models/references/playbook.md @@ -0,0 +1,321 @@ +# Limited-downtime model migration playbook + +## Contents + +1. Predeclare success +2. Preserve the public contract +3. Stage before the GPU boundary +4. Handoff and private canary +5. Tune with AIPerf Weka traces +6. Select and qualify +7. Publish through LiteLLM +8. Recover and roll back +9. Close the migration + +## 1. Predeclare success + +Define pass/fail thresholds before results exist: + +- zero request errors, engine deaths, OOMs, and unexpected restarts; +- context-window and completion-budget behavior match the intended alias contract; +- required chat, streaming, reasoning, tool, structured-output, and long-context probes pass; +- rank 0 accepts the private upstream key and rejects missing/wrong keys; +- rank 1 is headless, has no inference key, and exposes no HTTP listener; +- queue depth is bounded at target load and both ranks use the pinned fabric; +- LiteLLM stays loopback-only and public ingress rejects non-inference/admin paths; +- previous artifacts remain present, checksum-valid, and startable. + +Record a rollback trigger and maximum soak window. Do not choose thresholds after seeing candidate results. + +## 2. Preserve the public contract + +Snapshot the old alias with authenticated and unauthenticated probes plus LiteLLM model metadata. Treat these as independent fields: + +- alias and response `model` name; +- shared prompt-plus-completion limit and over-limit status; +- vision/audio/tool/reasoning/structured-output behavior; +- defaults, ignored parameters, timeout/retry policy, and prices. + +A replacement backend may have a larger context or more modalities. That does not authorize exposing them through an existing alias. Use a separate alias for the candidate's native capabilities. + +When an already-running compatible backend can carry the old alias, repoint and verify that continuity route before stopping the occupied pair. This converts user-visible downtime into a backend cold-rollback interval. It does not make the retired backend hot. + +## 3. Stage before the GPU boundary + +While the old model is live: + +1. Pin image digest, checkpoint revision, tokenizer revision, runtime dependencies, and hashes. +2. Copy artifacts to both nodes and verify identical manifests. +3. Render rank-specific start commands. Rank 1 must be separately constructed with `--headless` and without any API-key source. +4. Stage mode-0600 secrets only on rank 0 and the authorized proxy host. +5. Prepare profiles, probes, benchmark client, result summarizer, stop/start scripts, and recovery controller. +6. Preserve old container inspection, launch parameters, config, credentials, images, weights, and a tested restore command. Exclude all of them from cleanup. + +Preserve executable bits when staging controllers and Python entry points. Before the +first live invocation, execute each staged entry point with `--help` or an equivalent +read-only check; a checksum alone does not prove that the remote copy can run. + +Treat a recovery controller as a **dependency closure**, not a shortlist of top-level +scripts. Starting from every systemd `ExecStart`, recursively inventory each sourced +file, relative executable, Python entry point, and generated profile; stage and hash +the complete set. Run a packaging gate from the deployed directory that resolves and +executes every dependency through a read-only path. A top-level `--help` check is not +enough when it exits before loading a helper. Do not enable the timer or invoke its +healthy path until that gate passes. On Webster, an omitted pair-verifier made the +fail-closed watchdog fence and reload both ranks, creating roughly five minutes of +model-only downtime even though the serving pair had been healthy. + +Measure `getent ahosts "$(hostname)"` and `sudo -n true` on any worker that +requires sudo. Slow hostname resolution can add a fixed delay to every remote Docker +operation and make healthy coordination look hung; repair the host's self-resolution +before timing lifecycle or recovery behavior. + +If a Brev alias stalls during RSA public-key negotiation or suddenly resolves to a +different gateway key/port, run `brev refresh` and inspect `ssh -G ` before +changing controller logic. Refresh rotating gateway metadata before the downtime +window; keep `SSH_AUTH_SOCK` unset for every cluster command. + +Make preflight capture the phase-specific owner. Baseline/stage/alias inspect the old +model, private canary may inspect both stopped identities, and publication inspects the +running candidate. Never make publication depend on inspecting a stopped old container. + +Do not initialize a second model on a fully occupied pair. Weight presence is safe staging; loading them into GPU memory is the cutover. + +## 4. Handoff and private canary + +1. Capture pre-stop evidence and stop both old ranks concurrently. +2. Verify both recorded container identities are stopped, serving/rendezvous listeners are absent, and no workload GPU process remains. +3. Start rank 0 and rank 1 as one generation with fixed roles and a pinned TP fabric. PP stays 1 unless a measured memory or partitioning requirement justifies PP>1; PP adds pipeline bubbles and operational state, while TP=2 already spans the two station GPUs. +4. Keep the candidate private. Require both containers running with zero restarts, a real authenticated completion, 401 for missing/wrong credentials, exact rank-0 bind, no rank-1 listener/key, matching artifacts, and NCCL evidence. +5. If a gate fails, stop both candidate ranks before retrying. Never repair one rank in place while its peer survives. + +Add generation/profile/rank labels before the first candidate launch. If stronger +evidence code arrives after an otherwise exact pair is already serving, do not spend a +cold restart only to attach metadata: accept a **legacy pair identity** only after exact +image, argv/profile, role, security, fabric, zero-restart, and bounded start-skew checks, +then derive it from both immutable container IDs and start timestamps. Reject partial +labels or a mismatch on either rank. Future launches must carry the explicit labels. + +## 5. Tune with AIPerf Weka traces + +Treat Weka as dependency-aware workflow replay, not an RPS microbenchmark. + +### Validate scheduling first + +Run a small fixture and calculate exact request overlap and root-tree count. Fixed schedule can release many roots at nearly the same recorded instant; if it creates an artificial burst, do not use it for the topology sweep. + +For Webster's 393-trace corpus, the validated production-like mode is: + +```text +--no-fixed-schedule --concurrency 1 +``` + +Concurrency limits root session trees; nested children still fan out naturally. Observe vLLM running and waiting request gauges to confirm that behavior. + +### Place the corpus and driver safely + +Keep `corpus-manifest.json` beside `traces/`, never inside it. AIPerf treats every +JSON file inside the trace directory as a workload trace, so an identity manifest in +that directory corrupts the workload before traffic begins. Verify the source JSONL, +all split files, repository revision, trace count, and manifest hashes on the machine +that will drive the run. + +Size the client host for the expanded corpus, not the compressed source. The 393-trace +Weka replay expands into a 5.70 GiB mmap containing 9,843 conversations and 98,827 +turns, while cache-miss reconstruction uses roughly 12 GiB across AIPerf's dataset +manager and workers before steady traffic. Webster's 8 GiB head plus 8 GiB swap is not +a qualified driver for this corpus: 300-, 900-, and 1,200-second configuration guards +all expired before request dispatch. The 900-second attempt parsed all files in 889.3 +seconds and missed its notification deadline; the 1,200-second attempt parsed them in +924.9 seconds but remained swap-bound in post-parse materialization until the command +deadline. All three failures sent zero model requests. + +Run the full corpus from a high-memory controller such as rank 0. Keep both +`AIPERF_DATASET_CONFIGURATION_TIMEOUT` and +`AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT` pinned to at least 1,200 seconds and include +both values in the rendered command, but treat those guards as failure bounds rather +than substitutes for memory. Gate the runner on at least 32 GiB of physical controller +memory before corpus hashing or AIPerf launch so an undersized driver fails immediately +instead of consuming a 20-minute evidence window. A timeout before dataset +configuration completes does not publish the mmap cache, so the next cache-miss +attempt must parse from cold again. On +the station, the preserved mmap cache restored the 5.70 GiB backing store by hardlink +and configured the full 9,843-conversation dataset in 2.46 seconds. Do not run a second +driver while one benchmark or grace-period drain is active. + +The station-side runner must keep rank-0 Docker inspection local and reach only rank 1 +over SSH. Set `WEBSTER_LOCAL_SHAMU=1`, point `WEBSTER_RUNS_ROOT` at the staged station +run root, and require the lifecycle verifier's `--verify-only` path before traffic. +That verifier must fail on a missing or mismatched rank instead of restarting the +serving pair. Preserve the AIPerf checkout's `.git` metadata so the runner can verify +the pinned commit rather than trusting the installed version string. If a required +runtime patch makes that checkout intentionally dirty, require the exact dirty path set +and the SHA-256 of `git diff --binary HEAD -- ` in every repetition's +provenance. Acceptance must reject a missing or different runtime patch hash. + +### Control cache state + +For each candidate profile: + +1. stop both ranks; +2. start the profile and verify health; +3. run the same full trace pool twice with the same seed and root-tree concurrency; +4. label repetition 1 warmup and repetition 2 comparison; +5. compare only repetition 2; +6. restart before the next profile so candidates cannot inherit KV/prefix cache. + +Keep model revision, tokenizer, quantization, runtime, max length, batched-token cap, GPU utilization, speculative method, sampling, trace set, and output limits fixed. Change one tuning dimension at a time. + +Treat a FlashInfer autotune failure as a profile failure first. If the pinned runtime +exposes a supported no-autotune flag, validate that flag under test, then rerun every +compared profile with autotune disabled; never compare mixed autotune modes. + +Never trust process exit status alone. Reject a run when `.aiperf_results_ready.json` reports `was_cancelled: true`, when `EXIT` is nonzero/missing, when records are partial, or when request errors exist. Preserve cancelled evidence under an unmistakable non-benchmark name. Never rewrite a failed `EXIT`; move the entire closed run directory to a reason-and-timestamp suffix and create a new canonical run. A summarizer bug, including failure to parse grouped request counts such as `completed=1,036`, requires a regression fix plus a genuine benchmark rerun. It never authorizes repairing old evidence in place. + +For every publishable repetition, capture and hash AIPerf stdout/stderr, the benchmark +window, raw request records, exported profile and server metrics, rank 0/1 state before +and after, and bounded logs from both serving containers. Reject HTTP failures recorded +in raw Weka requests and fatal rank markers including `EngineDeadError`, NCCL +errors/timeouts, and RPC timeouts. Rebuild each summary from those raw inputs during +acceptance instead of trusting a stored summary. + +Repetition 1 is warmup only. Repetition 2 is scored only when its pre-run rank states +exactly match repetition 1's post-run states and it records the exact warmup provenance +hash. Promote only repetition 2 into the scored-summary path. An earlier or externally +started run that lacks this evidence remains exploratory even if AIPerf exits zero. + +Record root workflows completed, request throughput, root latency if available, TTFT p50/p90/p99, TPOT/ITL, input/output tokens per second, exact peak request overlap, error types, cache hits/cached tokens, running/waiting queue, KV use, preemptions, GPU power/utilization/memory, and throttling. + +### Choose lexicographically + +1. Disqualify correctness failures, errors, OOMs, restarts, unbounded queues, or SLO breaches. +2. Prefer the highest completed root-workflow rate at target load. +3. If throughput differs by less than the predeclared tolerance, prefer lower p95 root latency, then p99 TTFT, fewer preemptions, and more memory headroom. +4. If still tied, choose the smaller `max-num-seqs`. + +Price output tokens from measured active power divided by qualified output throughput. +Price input tokens from a demonstrably cold-prefill request. Do not use aggregate input +throughput when prefix-cache hits dominate; that measures cache effectiveness rather +than prefill compute. Record whether the result covers energy only or also includes +hardware, labor, network, and margin. + +Perform the dynamic pricing derivation only after the final scored repetition exists. +Use `energy_usd_per_second = facility_power_watts / 1000 * rate_usd_per_kwh / 3600`, +then divide by the measured cold-prefill throughput for input and by the selected Weka +output throughput for output. Preserve the unrounded direct values and record any +rounded published values separately. Recompute the evidence after any generation, +cold-prefill, Weka, power, or electricity-rate change; never copy numbers from an older +successful run. + +## 6. Select and qualify + +Restart the warm-run winner cleanly, then run: + +- representative chat and streaming requests; +- reasoning on/off and usage-accounting probes; +- tool auto/required/named/parallel behavior and tool-result continuation; +- JSON schema and `json_object` output with documented thinking requirements; +- malformed-JSON and wrong-model rejection; +- three identical deterministic requests with one normalized output; +- a streamed client disconnect followed by running/waiting queue drain; +- near-limit valid requests and clean over-limit rejection; +- sustained representative load with rank logs, GPU metrics, restarts, and queue checks. + +For this repository, the 17-case qualification command is: + +```bash +webster/deepseek-v41/scripts/feature-qualification.py \ + --base-url http://100.73.140.127:8000/v1 \ + --metrics-url http://100.73.140.127:8000/metrics \ + --model deepseek-ai/DeepSeek-V4.1-Flash \ + --key-file "$RUN_ROOT/credentials/deepseek-v41.key" \ + --stage dspark \ + --output-json "$RUN_ROOT/feature-dspark-1m-seqs4-final.json" \ + --failure-log-output "$RUN_ROOT/logs/feature-final-server-failure.log" \ + --server-log-node shamu --server-log-node tilikum +``` + +Keep `--failure-log-output` armed even on an expected GO. A failure captures redacted, +bounded logs from both ranks; a success records that capture was armed and unnecessary. + +Document unsupported behavior rather than silently widening or narrowing an existing alias contract. + +## 7. Publish through LiteLLM + +Build a **freshness budget** backward from the six-hour publication gate. Complete long +Weka and context jobs first, generate dynamic pricing from their final artifacts, then +capture short-lived rank/API evidence and write `private-acceptance.json` last. If a +late correction invalidates a dependency, recapture every downstream artifact; never +touch timestamps merely to make evidence appear fresh. + +1. Capture fresh rank-0, rank-1, and authenticated API evidence after all qualification artifacts are stable. +2. Run `generate-private-acceptance.py` with those three evidence paths and the recorded selection/PP=2 rationale. It writes `accepted_at` after the captures, hashes every referenced artifact, writes mode 0600, and invokes `verify-private-acceptance.py` itself. +3. Run `preflight.sh --phase publish`; it must independently invoke `verify-private-acceptance.py` with a six-hour freshness limit. A file merely named `private-acceptance.json` is not authority to publish. +4. Back up the exact live config immediately before rendering. +5. Render from that live snapshot so unrelated routes and callbacks cannot disappear. +6. Add the candidate alias and preserve the legacy alias contract fields explicitly. +7. Validate syntax, complete model inventory, upstream auth source, loopback bind, Caddy policy, and rollback diff offline. +8. Restart LiteLLM once. +9. Verify liveliness/readiness, restart count, `127.0.0.1:4446`, unauthenticated 401, public root/admin denial, authenticated model metadata, direct candidate inference, legacy alias inference, and the new alias. +10. Confirm a fresh trace and metrics sample in Langfuse/Prometheus without exposing credentials. + +Build publication probes from the live registry, then require the declared +**migration-critical aliases** explicitly. Scope the temporary publication key and +public requests to those aliases; keep exhaustive deployment coverage in the private +preflight. This prevents an unrelated retired model from blocking publication while +still failing when a migration-owned alias disappears. + +Validate errors at the same serialization boundary clients consume. An HTTP 400 and a +nested provider code do not preserve a public contract when LiteLLM rewrites the +client-visible top-level error code. Exercise the exact live image and public ingress, +and assert the status plus the top-level code and message before declaring the legacy +contract intact. + +If any gate fails, restore the exact backup and perform one validated restart. Do not hand-edit forward on a sick proxy. + +## 8. Recover and roll back + +Use one external coordinated controller with a singleton lock and cooldown. Container restart policy remains `no` when it could replay only one rank. + +Test the controller once with a real paired fault before publication. Keep its +recent-log matcher narrow enough that ordinary stopped-container history does not +manufacture an engine/NCCL reason; rank or health failure remains sufficient to trigger +recovery even when the diagnostic reason is imperfect. + +On either-rank failure: + +1. make the backend unavailable or use a separately validated fallback; +2. acquire the lock and record both container identities/generation; +3. stop/fence both ranks; +4. if a node is unreachable, fail closed until fencing or power state proves the old process dead; +5. confirm workload listeners and GPU processes are absent; +6. start both ranks from one pinned profile; +7. run the entire private-canary gate before restoring traffic; +8. log cause, actions, duration, and old/new identities. + +Cold rollback follows the same state machine with the preserved prior manifest. Stop/fence both candidate ranks, start both previous ranks, verify them fully, then atomically restore the prior route. Never delete the failed candidate during rollback. + +When a model leaves the live registry, disable every stale recovery controller that can +recreate it. Registry removal, stopped containers, and a clean listener are not enough +while an enabled timer still owns the old model. Verify the stale recovery controller +is disabled/inactive and that no replacement container appears across its next normal +interval. Preserve its scripts and rollback artifacts unless cleanup was separately +authorized. + +## 9. Close the migration + +Run one acceptance entry point that checks direct authenticated inference and 401 +rejection, every private model, the new and legacy public aliases, metadata, ingress +denials, rank roles, the watchdog, Prometheus, Langfuse, and the hash-verified rollback +path. Revoke its scoped key and verify the revoked key returns 401. + +Update the ledger with measured continuity and cold-rollback intervals, final hashes/profile, benchmark evidence, LiteLLM backup, watchdog result, and unresolved limitations. Run `sync-runbook.sh --check` and require byte-exact local mirrors plus exact rendered Hermes trees. If a regular consumer directory diverged, inspect the diff and preserve useful edits; only then run `sync-runbook.sh --force`. Commit only scoped files, run repository validation, push, and attach the evidence paths to the PR. + +Reconcile downstream consumers after retiring a model: virtual-key scopes, UI model +lists, Prometheus targets, metrics proxies, rollback probes, and documentation must +match the live registry. Remove a retired model from those surfaces and add the +replacement only where that consumer is authorized to use it. Validate both the +backing configuration and the running process's view. A read-only file bind can keep a +running container on a deleted inode after an atomic host-file replacement; if a reload +still exposes the old state, confirm host/container hashes before performing a narrowly +scoped service recreate and measure its interruption. diff --git a/.github/workflows/bootstrap.yml b/.github/workflows/bootstrap.yml index d88131d..b7a11cd 100644 --- a/.github/workflows/bootstrap.yml +++ b/.github/workflows/bootstrap.yml @@ -50,6 +50,14 @@ jobs: EXPECTED_ARCHITECTURE: ${{ matrix.architecture }} run: test "$(uname -m)" = "$EXPECTED_ARCHITECTURE" + - name: Prepare Python test environment + run: | + test_venv="$RUNNER_TEMP/setup-test-venv" + python3 -m venv "$test_venv" + "$test_venv/bin/python" -m pip install --disable-pip-version-check \ + -r webster/deepseek-v41/tests/requirements.txt + echo "$test_venv/bin" >> "$GITHUB_PATH" + - name: Check shell syntax run: | bash -n install.sh @@ -83,6 +91,7 @@ jobs: run: | { echo "$HOME/.local/bin" + echo "$HOME/.local/share/setup/go/current/bin" echo "$HOME/.opencode/bin" echo "/usr/local/go/bin" } >> "$GITHUB_PATH" diff --git a/docs/superpowers/plans/2026-09-11-glm52-to-deepseek-v41-cutover.md b/docs/superpowers/plans/2026-09-11-glm52-to-deepseek-v41-cutover.md new file mode 100644 index 0000000..fbb9693 --- /dev/null +++ b/docs/superpowers/plans/2026-09-11-glm52-to-deepseek-v41-cutover.md @@ -0,0 +1,1256 @@ +# GLM-5.2 Compatibility Alias and DeepSeek V4.1 Flash Cutover Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Preserve the original public `glm-5.2` contract while routing it to GLM-5.3, then qualify and publish `deepseek-v4.1-flash` on the Shamu/Tilikum DGX Station pair with a cache-controlled AIPerf Weka workload. + +**Architecture:** LiteLLM keeps `glm-5.2` as a compatibility deployment whose request guard uses the original tokenizer and chat template to enforce the legacy 320,000-token shared window before forwarding to the existing GLM-5.3 backend. After a hot rollback soak, the station pair is released and runs a pinned DeepSeek V4.1 vLLM TP2/PP1 canary over the 400 Gb/s ConnectX-8 rail, with Engram embeddings in each node's local Grace memory; features and performance changes advance one gate at a time before a separate public registration. + +**Tech Stack:** Bash, Python 3 standard library, `unittest`, PyYAML, Jinja2, Hugging Face `tokenizers`/Hub, Docker, vLLM, CUDA, NCCL, LiteLLM 1.95.0, Caddy, NetBird, Prometheus, Langfuse, AIPerf, systemd, tmux. + +**Spec:** `docs/superpowers/specs/2026-09-11-glm52-to-deepseek-v41-cutover-design.md` + +## Global Constraints + +- Keep the external endpoint exactly `https://webster-models-extnode-3gdrajbr0hiykknxzitck9yaiwo.apps.run.brev.nvidia.com`. +- Keep LiteLLM bound only to `127.0.0.1:4446`; Caddy remains the sole ingress and its `:4444`/`:4445` policy does not change. +- Keep the public model name `glm-5.2`, all existing virtual-key scopes, and its original 320,000-token shared prompt-and-completion contract. +- `glm-5.2` continues to advertise reasoning and function calling, does not advertise vision, and does not advertise the GLM-5.3 1,048,576-token window. +- Exact generations are allowed to change because the alias changes weights; the API surface, capability metadata, limits, and limit behavior are not allowed to drift. +- Attribute `glm-5.2` input/output cost to the GLM-5.3 backend values actually serving it: `6e-08` and `1.7e-06` dollars per token until a newer measured value is approved. +- Leave native `glm-5.3-flash` on `http://100.73.165.55:8000/v1` with its independent 1,048,576-token and vision contract. +- Do not remove, rename, repoint, pool, retune, or restart Baker's `deepseek-v4-flash` deployment at `http://100.73.127.129:8888/v1`. +- The alias cutover and the final DeepSeek registration each receive one planned LiteLLM restart, targeting less than 30 seconds with a 60-second rollback decision threshold. +- Before either restart, capture and validate the live LiteLLM container command shape: the immutable image supplies `docker/prod_entrypoint.sh` through `Config.Entrypoint`, so `Config.Cmd` must contain only `--config /app/config.yaml --host 127.0.0.1 --port 4446` and must never repeat the entrypoint. Any recovery recreation must pass the same check on its candidate inspect record before the old container is stopped or renamed. +- Never use `LITELLM_MASTER_KEY` for a completion, probe, replay, or benchmark. Mint a named, model-scoped virtual key and revoke it at the end of the work block. +- Keep the station GLM-5.2 ranks hot for 30–60 minutes after the alias restart; do not shorten this soak to start DeepSeek sooner. +- Stop and start both ranks together. Tilikum is rank 1, always uses `--headless`, receives no serving API key, and exposes no HTTP listener. +- Shamu is rank 0 and the only HTTP listener. Bind port 8000 only to Shamu's NetBird address `100.73.140.127`, never `0.0.0.0`, loopback, or the Webster LAN address. +- Run station NCCL only over Shamu `10.10.1.1` and Tilikum `10.10.1.2`, interface `enP1p3s0f1np1`, HCA `mlx5_1`; do not use NetBird for collectives. +- Preserve `--network host`, `--ipc host`, `--ulimit memlock=-1:-1`, `--ulimit stack=67108864:67108864`, `--cap-add CAP_IPC_LOCK`, and the rank-1 headless invariant. +- Use DeepSeek V4.1 checkpoint `deepseek-ai/DeepSeek-V4.1-Flash` at revision `dba1be0a40aa45a94ad051997016db3960a90277`, exactly 48 weight shards, 510,286,023,000 logical tensor bytes from the safetensors index, and 510,296,708,312 on-disk shard-file bytes including headers. +- Keep a complete checkpoint on local NVMe on both stations and each rank's Engram embedding shard in that station's local pinned Grace memory; remote Engram access is a failed gate. +- Require at least 805,306,368,000 free bytes (750 GiB) before staging on each station and at least 214,748,364,800 free bytes (200 GiB) after the checkpoint, immutable runtime, build cache, and evidence are present. +- Do not compile GPU kernels or launch a second GPU workload while station GLM-5.2 is serving. +- Recheck vLLM PR `#56214` at runtime-pin time. Its observed head `0bfb653d3b5161660db9ada0d84c2cdd60961de7` and unstable state are evidence, not an approved pin. +- Do not run a floating vLLM branch, image tag, Hugging Face revision, AIPerf revision, or Weka dataset revision. Record the source commit, image ID, saved-image SHA-256, dependency lock, CUDA, NCCL, driver, kernel, and local diff. +- Start DeepSeek at TP=2, PP=1. EP is optional after the baseline; PP=2 is a matched fallback experiment only when its trigger criteria are met. +- Introduce features in this order: eager text, 320K/API contract, CUDA graphs, DSpark, vision, 1M. A failed feature returns to the preceding known-good profile. +- Run Weka from `head`, with the full subagent corpus, a pinned dataset revision, a persistent terminal, explicit time bounds, controlled cache state, and a completion record matching `^EXIT=`. +- Never put AIPerf mmap files on station NVMe or evict the serving checkpoint's page cache. Give each run an exact `mktemp` directory on `head` and delete only that resolved directory after artifacts are copied. +- Do not delete the GLM checkpoint, credentials, launch/restore scripts, stopped containers, LiteLLM backups, DeepSeek checkpoint, failed-variant logs, or benchmark evidence in this project. +- Keep secrets out of Git, argv, logs, terminal output, rank 1, benchmark artifacts, and documentation. Credential files are mode `0600`; evidence records only paths, modes, ownership, and non-reversible fingerprints. +- Update `/home/ubuntu/.claude/skills/webster-cluster/SKILL.md` only after live verification, then run `/home/ubuntu/ops-bot/sync-skills.sh` and resolve reverse-direction drift before any forced sync. +- Preserve the unrelated modified and untracked files already present in `/home/ubuntu/.setup`; each commit stages only paths named by its task. + +--- + +## Characterization Decision: Preserve the Shared 320K Contract with a Narrow Guard + +Read-only characterization on the live LiteLLM 1.95.0 container established: + +```text +Router(enable_pre_call_checks=True) + prompt > max_input_tokens -> REJECTED + short prompt + max_tokens=1_000_000 -> ACCEPTED BY ROUTER PRE-CHECK +``` + +The installed `_pre_call_checks()` counts only input tokens. It never combines +`max_tokens` or `max_completion_tokens` with the prompt. Enabling it globally would add +CPU cost and behavioral changes to every model without enforcing the legacy shared +window. + +That result characterizes only LiteLLM's optional router pre-check, not the original +backend contract. Live direct requests to the still-hot station vLLM 0.25.1 backend on +2026-09-11 established the actual boundary behavior: + +```text +319999 prompt + max_tokens 1 -> HTTP 200 +319999 prompt + max_tokens 2 -> HTTP 400 +320000 prompt + max_tokens 1 -> HTTP 400 +320000 prompt + omitted max_tokens -> HTTP 400 (effective max_tokens 0) +320001 prompt + max_tokens 1 -> HTTP 400 +short prompt + max_tokens 1000000 -> HTTP 400 +``` + +The source explains the split: `TokenizeParams` reserves an explicitly requested output +budget before tokenization and rejects a prompt that would make the combined total exceed +320,000. `get_max_tokens()` applies only after rendering and supplies the remaining budget +when the output field was omitted. Explicit over-budget output is rejected, not clamped. +Therefore the compatibility guard must: + +1. run only when the incoming public model is exactly `glm-5.2`; +2. render with the original GLM-5.2 `chat_template.jinja` and `tokenizer.json`; +3. return HTTP 400 when the rendered prompt plus an explicit active output limit exceeds + 320,000 tokens; +4. preserve an explicit active output limit when the combined total fits; +5. set `max_tokens` to the positive remaining budget when no output-limit field is + present, and return HTTP 400 when no output token remains; +6. fail closed for an alias request it cannot count; and +7. leave native `glm-5.3-flash` and all other models byte-for-byte untouched. + +The guard is a `CustomLogger.async_pre_call_hook`, placed first in +`litellm_settings.callbacks`, so rejection or defaulting happens before routing and before +the GLM-5.3 backend sees the request. Its renderer is accepted only after its token IDs +match the still-hot GLM-5.2 `/tokenize` behavior on plain, reasoning-disabled, tool, +tool-result, structured-output, and long-boundary fixtures. + +## File Map + +Create this repository-managed operational package: + +```text +webster/deepseek-v41/ + README.md limited-downtime sequence, gates, rollback index + .gitignore rejects live manifests, secrets, logs, images, checkpoints + manifest.env.example immutable and execution-time pin field names + litellm/ + glm52_contract_guard.py exact legacy prompt count and shared-budget guard + test_glm52_contract_guard.py unit and golden-token contract tests + runtime/ + Dockerfile pinned-source vLLM OpenAI image build wrapper + constraints.txt frozen Python dependency output copied from the build + README.md pin decision and reproducible build procedure + scripts/ + common.sh shared constants, redaction, evidence, remote helpers + preflight.sh read-only baseline and fail-closed invariants + stage-artifacts.sh low-priority checkpoint/tokenizer/image staging + verify-checkpoint.py shard, byte, revision, and cross-node manifest checks + pin-runtime.py GitHub PR/CI/release evidence and immutable pin selection + build-runtime.sh reproducible arm64 image build/save/copy/load + render-litellm-cutover.py semantic alias/callback rewrite with no other changes + verify-litellm-config.py allowed semantic diff and invariant validation + verify-litellm-container.py fail-closed entrypoint/Cmd and recreation-shape validation + install-glm52-guard.sh idempotent callback/tokenizer install on spark-1 + restore-litellm-config.sh phase-specific validated proxy rollback + stop-glm52-tp2.sh coordinated old-engine stop with evidence capture + start-glm52-tp2.sh coordinated cold rollback with explicit 320000/8 + start-deepseek-v41-tp2.sh TP2/PP1 launch and startup gate + stop-deepseek-v41-tp2.sh coordinated DeepSeek stop + deepseek-v41-watchdog.sh split-brain-safe health and two-rank recovery + contract-probe.py alias, API, listener, auth, and boundary probes + feature-qualification.py progressive deterministic correctness suite + run-weka.sh pinned full-subagent cold/warm replay runner + summarize-aiperf.py metric, telemetry, error, and sample report + acceptance.sh end-to-end final and cross-model checks + sync-runbook.sh canonical Webster edit/sync verification wrapper + tests/ + test_static.py repository safety and invariant tests + test_failure_paths.sh fake-host negative/idempotency tests + fixtures/ + chat.json plain chat token fixture + reasoning-none.json reasoning-disabled fixture + tools.json tool definition and forced-auto fixture + tool-result.json assistant tool call plus observation fixture + structured.json response-format fixture + tokenize-golden.json token IDs/counts captured from old vLLM + aiperf-profile.json representative successful summary + aiperf-errors.jsonl representative mixed error records +``` + +Modify during implementation: + +- `test.sh` — invoke the new Python and shell suites. +- `/home/ubuntu/.claude/skills/webster-cluster/SKILL.md` — final verified live state only. +- `/home/ubuntu/.codex/skills/webster-cluster/` and + `ops-bot:~/.hermes/skills/devops/webster-cluster/` — only through + `/home/ubuntu/ops-bot/sync-skills.sh`. + +Large and possibly sensitive live evidence stays outside Git under the exact run root +`/home/ubuntu/deepseek-v41-runs/$CHANGE_ID/`. The redacted decision summary and +paths to that evidence are appended to `webster/deepseek-v41/README.md` after each +completed milestone. + +## Operational Milestone Mapping + +| Approved milestone | Plan tasks | Irreversibility boundary | +|---|---|---| +| 1. Baseline/freeze | 1–2, 6 | None; read-only capture and recoverable backups | +| 2. Pre-stage | 3, 8, 6 | None; additive files only | +| 3. Alias cutover | 4–7 | First LiteLLM restart; hot GLM makes rollback fast | +| 4. Soak/release | 7 | Stopping GLM changes rollback from seconds to a 5–15 minute cold load | +| 5. Minimal TP2 canary | 8–9 | Port/GPU ownership changes; old artifacts remain recoverable | +| 6. Feature qualification | 10 | None public; failed private features revert to last passing profile | +| 7. TP2 tuning/PP2 fallback | 11 | None public; each variant is reproducible and reversible | +| 8. Weka qualification | 12 | None public; benchmark state is isolated on `head` | +| 9. Register/publish record | 13 | Second LiteLLM restart; rollback removes only the new model entry | + +Required execution order is Tasks 1–5, Task 6 Steps 1–2, Task 8, Task 6 Steps 3–5, +then Tasks 7 and 9–13. Task 6 deliberately brackets Task 8: its first two steps freeze +the live baseline and rollback state before any staging; Task 8 pins/builds/stages the +runtime; its remaining steps stage and cross-check the checkpoint and guard before the +milestone-2 GO record. The alias is not cut over until both the checkpoint and selected +immutable runtime image are present on both stations. Tasks 4 and 5 can be implemented +and tested while upstream runtime CI is still pending. + +### Task 1: Add Repository Contract and Failure-Path Tests + +**Files:** + +- Create: `webster/deepseek-v41/tests/test_static.py` +- Create: `webster/deepseek-v41/tests/test_failure_paths.sh` +- Create: `webster/deepseek-v41/tests/fixtures/aiperf-profile.json` +- Create: `webster/deepseek-v41/tests/fixtures/aiperf-errors.jsonl` +- Modify: `test.sh` + +**Interfaces:** + +- Consumes: the approved design and the global constraints above. +- Produces: `python3 -m unittest discover -s webster/deepseek-v41/tests -p 'test_*.py'` and `bash webster/deepseek-v41/tests/test_failure_paths.sh`. + +- [ ] **Step 1: Write the failing foundation contract suite** + +Create a standard-library `unittest` suite with helpers `read(relative_path) -> str` +and `shell_script_paths() -> list[Path]`. Include these exact core assertions and extend +them to every file in the file map: + +```python +def test_station_topology_is_pinned(self): + common = read("scripts/common.sh") + for literal in ( + 'SHAMU_NETBIRD="100.73.140.127"', + 'SHAMU_RAIL="10.10.1.1"', + 'TILIKUM_RAIL="10.10.1.2"', + 'NCCL_IFACE="enP1p3s0f1np1"', + 'NCCL_HCA="mlx5_1"', + ): + self.assertIn(literal, common) + +def test_baker_route_is_never_a_mutation_target(self): + for path in shell_script_paths(): + text = path.read_text() + self.assertNotIn("100.73.127.129:8888", text) + self.assertNotRegex(text, r"ssh\s+baker-spark-[12]") + +def test_no_secret_on_command_line(self): + text = "\n".join(path.read_text() for path in shell_script_paths()) + self.assertNotRegex(text, r"--api-key(?:=|\s+)[^\"'$]") + self.assertNotRegex(text, r"Authorization:\s*Bearer\s+\$\{") +``` + +Initially assert the checkpoint revision/count/bytes, pre/post free-space thresholds, +mode-0600 evidence creation, exact two-restart wording in README, and no mutation command +aimed at Baker. Each subsequent task adds the assertions for the assets it introduces; +the suite must not refer to a future task's absent files. + +- [ ] **Step 2: Add negative and idempotency tests with fake executables** + +`test_failure_paths.sh` creates a scratch directory with `mktemp -d`, installs fake +`ssh`, `docker`, `curl`, `systemctl`, `sha256sum`, and `nvidia-smi` ahead of `PATH`, and +records exact invocations in `$TEST_LOG`. Initially prove these paths exit nonzero: + +- preflight sees less than 750 GiB free before staging; +- Shamu or Tilikum rail/HCA/interface is absent; +- the run root resolves outside `/home/ubuntu/deepseek-v41-runs/`; +- a credential metadata capture reports a mode other than `0600`. + +Subsequent tasks add their checkpoint, runtime, LiteLLM, lifecycle, AIPerf, and skill +drift failures before implementing those behaviors. Run each successful fake path twice +and compare `$TEST_LOG` so idempotent operations are harmless on a second run. + +- [ ] **Step 3: Wire the suites into the repository runner** + +Append exactly: + +```bash +python3 -m unittest discover -s webster/deepseek-v41/tests -p 'test_*.py' +bash webster/deepseek-v41/tests/test_failure_paths.sh +if [ -f webster/deepseek-v41/litellm/test_glm52_contract_guard.py ]; then + python3 webster/deepseek-v41/litellm/test_glm52_contract_guard.py +fi +``` + +- [ ] **Step 4: Confirm the intended red state** + +Run: + +```bash +python3 -m unittest discover -s webster/deepseek-v41/tests -p 'test_*.py' +bash webster/deepseek-v41/tests/test_failure_paths.sh +``` + +Expected: failures name missing operational files; failures must not touch any remote +host. + +Do not commit this red state. Task 2 supplies the foundation files, reruns the suite to +green, and commits the tests with their first implementation. + +### Task 2: Implement the Manifest, Evidence, and Preflight Foundation + +**Files:** + +- Create: `webster/deepseek-v41/.gitignore` +- Create: `webster/deepseek-v41/manifest.env.example` +- Create: `webster/deepseek-v41/scripts/common.sh` +- Create: `webster/deepseek-v41/scripts/preflight.sh` +- Create: `webster/deepseek-v41/README.md` +- Test: `webster/deepseek-v41/tests/test_static.py` +- Test: `webster/deepseek-v41/tests/test_failure_paths.sh` + +**Interfaces:** + +- Consumes: `CHANGE_ID` in UTC form `YYYYMMDDTHHMMSSZ` and an optional read-only `RUN_ROOT` override. +- Produces: a mode-0700 run root with `manifest.env`, `events.tsv`, `baseline/`, `logs/`, `metrics/`, `aiperf/`, and redacted `summary.md`. + +- [ ] **Step 1: Define immutable constants and safe evidence helpers** + +`common.sh` uses `set -euo pipefail`, rejects a `RUN_ROOT` outside +`/home/ubuntu/deepseek-v41-runs/`, creates directories with umask `077`, and exports: + +```bash +SHAMU_NETBIRD="100.73.140.127" +SHAMU_LAN="192.168.1.75" +SHAMU_RAIL="10.10.1.1" +TILIKUM_NETBIRD="100.73.89.150" +TILIKUM_RAIL="10.10.1.2" +NCCL_IFACE="enP1p3s0f1np1" +NCCL_HCA="mlx5_1" +CANARY_PORT="8000" +MIN_FREE_BEFORE_STAGE_BYTES="805306368000" +MIN_FREE_AFTER_STAGE_BYTES="214748364800" +CHECKPOINT_REPO="deepseek-ai/DeepSeek-V4.1-Flash" +CHECKPOINT_REVISION="dba1be0a40aa45a94ad051997016db3960a90277" +CHECKPOINT_SHARDS="48" +CHECKPOINT_TENSOR_BYTES="510286023000" +CHECKPOINT_SHARD_FILE_BYTES="510296708312" +``` + +Implement `event milestone decision reason`, `capture name command...`, +`require_eq label expected actual`, `require_file_mode_600 host path`, and +`redact_stream`. `capture` writes stdout/stderr to the run root and prints only the +artifact path and exit status to the terminal. + +- [ ] **Step 2: Define manifest fields without approving a floating runtime** + +`manifest.env.example` contains the fixed checkpoint values plus empty, required +execution-time fields `VLLM_COMMIT`, `VLLM_BASE_IMAGE_DIGEST`, `VLLM_IMAGE_ID`, +`VLLM_IMAGE_TAR_SHA256`, `AIPERF_COMMIT`, `WEKA_REPOSITORY`, and `WEKA_REVISION`. +Comments state that `pin-runtime.py` or `run-weka.sh --pin` writes each value and that +preflight rejects an empty value. The template is documentation, never sourced as a +live manifest. + +`.gitignore` rejects `manifest.env`, `*.key`, `*.env`, `runs/`, `artifacts/`, `*.tar`, +`*.tar.zst`, checkpoint directories, Hugging Face caches, and Python caches. + +- [ ] **Step 3: Implement read-only preflight capture** + +`preflight.sh --phase baseline|stage|alias|canary|publish` captures without printing +secret values: + +- Git commit/status and the approved spec hash; +- `brev ls` and `brev ls nodes`; +- hostnames, UTC time, uptime, kernel, driver, GPU, RAM, root/NVMe bytes, NetBird + address, and interface/HCA/link state on Shamu and Tilikum; +- jumbo/non-jumbo rail reachability without changing MTU; +- GLM container ID/image/start/restart/health, launch/restore script SHA-256, listener, + rank arguments, and GPU memory on both stations; +- credential path, owner, mode, size, and SHA-256 only through files written beneath + the mode-0700 run root; terminal output shows no fingerprint; +- LiteLLM container ID/image/start/restart/network/bind, callbacks, router plugin, + `Config.Entrypoint` and `Config.Cmd` as separate fields, the effective command shape, + redacted model entries, config mode/hash, Caddy root 403, unauthenticated models 401, + and all current model completion probes through a temporary scoped key; +- Prometheus target health and a Langfuse trace window. + +`--phase stage` additionally enforces 750 GiB free. `--phase canary` requires GLM ranks +stopped and no listener. `--phase publish` requires the private DeepSeek acceptance +record. Every failed requirement appends `NO-GO` and exits nonzero. + +- [ ] **Step 4: Document the limited-downtime strategy** + +README's first page records this exact control sequence: + +```text +freeze rollback -> stage additively -> route alias -> restart once -> hot soak +-> stop both GLM ranks -> private canary/tuning -> register new name -> restart once +``` + +It includes the 60-second LiteLLM decision threshold, the point where rollback grows +from seconds to 5–15 minutes, exact stop/start scripts, the two-backend port collision, +and the rule that cleanup needs separate approval. + +- [ ] **Step 5: Run tests and commit** + +```bash +python3 -m unittest discover -s webster/deepseek-v41/tests -p 'test_*.py' +bash webster/deepseek-v41/tests/test_failure_paths.sh +git add test.sh webster/deepseek-v41/.gitignore webster/deepseek-v41/manifest.env.example \ + webster/deepseek-v41/README.md webster/deepseek-v41/scripts/common.sh \ + webster/deepseek-v41/scripts/preflight.sh webster/deepseek-v41/tests +git commit -m "feat: add cutover preflight and evidence capture" +``` + +Expected: both suites pass; no SSH fake log contains a secret value. + +### Task 3: Implement Low-Priority Immutable Artifact Staging + +**Files:** + +- Create: `webster/deepseek-v41/scripts/stage-artifacts.sh` +- Create: `webster/deepseek-v41/scripts/verify-checkpoint.py` +- Test: `webster/deepseek-v41/tests/test_static.py` +- Test: `webster/deepseek-v41/tests/test_failure_paths.sh` + +**Interfaces:** + +- Consumes: fixed checkpoint constants and `--node shamu|tilikum`. +- Produces: `/home/alecfong/deepseek-v41/models/DeepSeek-V4.1-Flash-dba1be0a` and a sorted SHA-256 manifest on each node; `/home/nvidia/litellm/energy-pricing/glm52-tokenizer/` on spark-1. + +- [ ] **Step 1: Write checkpoint verification before the downloader** + +`verify-checkpoint.py PATH` loads `model.safetensors.index.json`, resolves every +referenced weight shard, and exits nonzero unless the unique shard count is 48 and the +index metadata declares 510,286,023,000 logical tensor bytes while the sum of shard +`st_size` values is 510,296,708,312. It requires `config.json`, `tokenizer.json`, +`tokenizer_config.json`, the published `encoding/encoding.py` reference and fixtures, +and `inference/config.json`; rejects symlinks escaping PATH; and emits sorted +`sha256 bytes relative/path` lines to stdout. The release intentionally contains no +`generation_config.json`, processor config, or Jinja chat template; the candidate vLLM +source must register the native `deepseek_v41` renderer instead of staging invented +files. + +Before download, query the pinned Hugging Face revision's LFS metadata and compute +`missing_file_bytes + largest_missing_file_bytes + 214748364800`. Require free space to +be at least the greater of that exact value and 805306368000. This accounts for the +downloader's largest partial-file peak and the 200 GiB post-stage reserve; the already +staged runtime image/build cache is reflected in the live free-space measurement rather +than guessed from a nominal image size. Save the LFS layout used in the calculation. + +Tests create sparse fake shards with injected expected-size metadata so the suite does +not allocate 475 GiB. Cover missing, duplicate, extra, size-mismatch, traversal, and +cross-node manifest mismatch cases. + +- [ ] **Step 2: Implement safe, resumable staging** + +For one node at a time, `stage-artifacts.sh`: + +1. checks production GLM health and the 750 GiB floor; +2. downloads revision-pinned files into the sibling directory + `.DeepSeek-V4.1-Flash-dba1be0a.incomplete` using `ionice -c3 nice -n 19` and one Hub + worker; +3. records one-minute GLM error/latency and disk-pressure snapshots every five minutes; +4. stops and returns nonzero if success rate moves below 99% or p90 exceeds twice the + captured baseline for two consecutive windows; +5. verifies the checkpoint and writes `MANIFEST.sha256`; +6. atomically renames the incomplete directory to its final name and removes write bits; +7. enforces the 200 GiB post-stage floor. + +The command is resumable: an incomplete directory resumes, an identical final directory +verifies and exits zero, and a divergent final directory is never overwritten. + +- [ ] **Step 3: Stage the legacy tokenizer guard assets additively** + +Copy only `tokenizer.json`, `tokenizer_config.json`, and `chat_template.jinja` from +Shamu's existing GLM checkpoint to a temporary spark-1 directory, compare SHA-256 to +the source over SSH, then atomically rename it to +`/home/nvidia/litellm/energy-pricing/glm52-tokenizer`. The directory is readable by the +LiteLLM container but contains no credential. + +- [ ] **Step 4: Validate on fixtures and commit** + +```bash +python3 -m unittest discover -s webster/deepseek-v41/tests -p 'test_*.py' +bash webster/deepseek-v41/tests/test_failure_paths.sh +git add webster/deepseek-v41/scripts/stage-artifacts.sh \ + webster/deepseek-v41/scripts/verify-checkpoint.py webster/deepseek-v41/tests +git commit -m "feat: add immutable DeepSeek artifact staging" +``` + +Expected: idempotent fixture staging passes and every mismatch exits nonzero without +removing the divergent directory. + +### Task 4: Implement and Prove the Exact `glm-5.2` Contract Guard + +**Files:** + +- Create: `webster/deepseek-v41/litellm/glm52_contract_guard.py` +- Create: `webster/deepseek-v41/litellm/test_glm52_contract_guard.py` +- Create: `webster/deepseek-v41/tests/fixtures/chat.json` +- Create: `webster/deepseek-v41/tests/fixtures/reasoning-none.json` +- Create: `webster/deepseek-v41/tests/fixtures/tools.json` +- Create: `webster/deepseek-v41/tests/fixtures/tool-result.json` +- Create: `webster/deepseek-v41/tests/fixtures/structured.json` +- Create: `webster/deepseek-v41/tests/fixtures/tokenize-golden.json` + +**Interfaces:** + +- Consumes: incoming LiteLLM request dicts and `/app/custom_callbacks/glm52-tokenizer/`. +- Produces: `glm52_contract_guard = GLM52ContractGuard()` returning a request dict or raising HTTP 400. + +- [ ] **Step 1: Write failing unit tests for public-model isolation and shared-budget behavior** + +Use an injected `count_prompt(data) -> int` in tests. Cover: + +```python +def test_prompt_over_limit_is_rejected_before_routing(): + guard = GLM52ContractGuard(counter=lambda _: 320_001) + with self.assertRaises(HTTPException) as error: + asyncio.run(guard.async_pre_call_hook(None, None, {"model": "glm-5.2", "messages": []}, "completion")) + self.assertEqual(error.exception.status_code, 400) + +def test_explicit_output_over_remaining_budget_is_rejected_like_old_vllm(): + guard = GLM52ContractGuard(counter=lambda _: 319_900) + data = {"model": "glm-5.2", "messages": [], "max_tokens": 500} + with self.assertRaises(HTTPException) as error: + asyncio.run(guard.async_pre_call_hook(None, None, data, "completion")) + self.assertEqual(error.exception.status_code, 400) + +def test_native_glm53_is_untouched(): + data = {"model": "glm-5.3-flash", "messages": [], "max_tokens": 500_000} + original = copy.deepcopy(data) + self.assertEqual(run_guard(data), original) +``` + +Also test `max_completion_tokens` precedence, Responses `max_output_tokens`, absent output limit, exact-limit prompt, +negative remaining budget, invalid/boolean/zero output fields, text-only content arrays, +tool definitions, tool-result history, `reasoning_effort`, `chat_template_kwargs`, +Responses input normalization, concurrent lazy initialization, missing tokenizer files, +and renderer failure. Counting failures must fail closed only for `glm-5.2`. + +- [ ] **Step 2: Implement the tokenizer and template renderer** + +Use `tokenizers.Tokenizer.from_file`, Jinja2's immutable sandbox with +`trim_blocks=True`/`lstrip_blocks=True`, and a deterministic `tojson` filter matching +the original Transformers renderer. Render `messages`, `tools`, +`add_generation_prompt=True`, `reasoning_effort`, and every supported +`chat_template_kwargs` key, then count `encode(rendered, add_special_tokens=False).ids`. + +Normalize LiteLLM Responses input through its installed Responses-to-chat transform. +Load the tokenizer once under a lock. Do not make a network request or silently fall +back to tiktoken. Raise a 400 body with code `context_length_exceeded` for prompt +overflow and `contract_tokenization_failed` if the legacy contract cannot be evaluated. + +- [ ] **Step 3: Capture golden tokens from the still-hot old backend** + +Run each fixture against Shamu's authenticated `/tokenize` route without logging the +key. Store only rendered token IDs/counts and the non-secret fixture name in +`tokenize-golden.json`. If this vLLM build does not expose `/tokenize`, run a temporary +CPU-only Python process inside the existing GLM image with the read-only checkpoint +mount and Transformers tokenizer; do not start an engine or allocate the GPU. + +Require exact token-ID equality for all short fixtures. Generate boundary fixtures by +binary-searching repeated non-special text until local counts are 319,999, 320,000, and +320,001. With `max_tokens: 1`, validate the old Chat Completions route's +accept/reject/reject behavior, plus rejection of an explicit output request larger than +the remaining shared budget. + +- [ ] **Step 4: Run tests in the exact LiteLLM image** + +Run the module tests locally, then mount the repository module and staged tokenizer into +an ephemeral `ghcr.io/berriai/litellm:main-latest` container by the currently running +image ID, with `--network none`, and run the same tests. Expected: all pass and no +network socket is opened. + +- [ ] **Step 5: Commit the guard** + +```bash +git add webster/deepseek-v41/litellm webster/deepseek-v41/tests/fixtures +git commit -m "feat: preserve the GLM-5.2 shared token contract" +``` + +### Task 5: Implement Semantic LiteLLM Rendering and Offline Validation + +**Files:** + +- Create: `webster/deepseek-v41/scripts/render-litellm-cutover.py` +- Create: `webster/deepseek-v41/scripts/verify-litellm-config.py` +- Create: `webster/deepseek-v41/scripts/verify-litellm-container.py` +- Create: `webster/deepseek-v41/scripts/install-glm52-guard.sh` +- Create: `webster/deepseek-v41/scripts/restore-litellm-config.sh` +- Create: `webster/deepseek-v41/scripts/contract-probe.py` +- Test: `webster/deepseek-v41/tests/test_static.py` +- Test: `webster/deepseek-v41/tests/test_failure_paths.sh` + +**Interfaces:** + +- Consumes: `config.yaml.before`, a staged guard, and the staged legacy tokenizer. +- Produces: `config.yaml.candidate`, an allowed-diff report, and read-only/live contract probe JSON. + +- [ ] **Step 1: Write config fixtures and failing allowed-diff tests** + +Tests use configs containing unrelated models, callbacks, router plugins, virtual-key +settings, and secret-shaped values. `verify-litellm-config.py BEFORE AFTER --phase alias` +must allow only: + +```text +model_list[model_name=glm-5.2].litellm_params.model +model_list[model_name=glm-5.2].litellm_params.api_base +model_list[model_name=glm-5.2].litellm_params.api_key +model_list[model_name=glm-5.2].model_info.input_cost_per_token +model_list[model_name=glm-5.2].model_info.output_cost_per_token +model_list[model_name=glm-5.2].model_info.description +litellm_settings.callbacks (one first-position guard insertion) +``` + +It requires exactly one deployment for each existing public name, preserves every +non-GLM-5.2 object by deep equality, leaves `router_settings.plugins` equal to +`["custom_callbacks.kv_router.plugin"]`, and rejects global +`enable_pre_call_checks: true`. + +- [ ] **Step 2: Implement the alias renderer** + +The renderer parses YAML, finds exactly one `glm-5.2` and one `glm-5.3-flash` entry, +and copies the native entry's `model`, `api_base`, and `api_key` into the compatibility +entry without printing them. It modifies the following keys in place, preserving +`mode` and every other existing metadata field: + +```yaml +max_input_tokens: 320000 +max_output_tokens: 320000 +supports_function_calling: true +supports_reasoning: true +input_cost_per_token: 6.0e-08 +output_cost_per_token: 1.7e-06 +description: Deprecated GLM-5.2 compatibility alias served by GLM-5.3-Flash; legacy 320000-token shared window, reasoning, and function-calling contract retained; no advertised vision capability. +``` + +It removes `supports_vision` from only that model info and inserts +`custom_callbacks.glm52_contract_guard.glm52_contract_guard` exactly once at callback +index zero. The output is mode `0600`, is never written over the source, and is accepted +only after `verify-litellm-config.py` returns zero. + +- [ ] **Step 3: Implement fail-closed container-shape and recreation checks** + +`verify-litellm-container.py INSPECT_JSON` reads a Docker inspect object without +printing environment values. It requires the immutable live image ID, a nonempty +`Config.Entrypoint` containing `docker/prod_entrypoint.sh` exactly once, and +`Config.Cmd` equal to: + +```text +--config /app/config.yaml --host 127.0.0.1 --port 4446 +``` + +It rejects any entrypoint path in `Config.Cmd`, any duplicate effective entrypoint, +and any change to the expected host or port. Failure-path tests must reproduce the +2026-09-11 duplicate-entrypoint crash shape and prove it is rejected. Before a planned +restart, save the redacted live inspect and successful check beneath the run root. If a +recreation is ever required, render its candidate inspect-equivalent JSON first and +require the same check plus equality of image ID, mounts, network mode, restart policy, +published ports, user, capabilities, and ulimits against the captured baseline; only +the explicitly approved command/config delta may differ. + +- [ ] **Step 4: Implement idempotent guard installation and offline import checks** + +`install-glm52-guard.sh --candidate` copies the module and tokenizer to timestamped +temporary paths beneath `/home/nvidia/litellm/energy-pricing`, verifies hashes and +ownership, and atomically promotes them. A second identical install exits zero; a +divergent live file is backed up, never erased. Use the running LiteLLM image ID with +`--network none` to parse YAML, import every dotted callback, assert each custom object +subclasses `CustomLogger`, and invoke the guard fixture tests. + +- [ ] **Step 5: Implement boundary and routing probes** + +`contract-probe.py` reads a key from a mode-0600 file, never argv. It supports +`--base-url`, `--models`, `--backend-metrics-before`, and `--output-json`. It checks +chat, stream, reasoning off/on, tool auto, tool-result continuation, structured JSON, +the response `model` field against the captured pre-cutover behavior, and a long prompt. +Its over-limit case submits a 320,001-token rendered prompt and requires 400; backend +request counters must not move. Any response-model drift is resolved in the compatibility +callback and covered by non-streaming and streaming tests before the live restart. + +Implement `restore-litellm-config.sh --phase alias|publish --run-root PATH`. It reads the +phase-specific absolute backup path from mode-0600 `rollback.env`, rejects paths outside +`/home/nvidia/litellm/` or without the recorded SHA-256, restores mode `0600`, restarts +LiteLLM once, and runs bind/readiness/root/401/cross-model checks. No rollback command +depends on shell history or a manually substituted path. + +- [ ] **Step 6: Run tests and commit** + +```bash +python3 -m unittest discover -s webster/deepseek-v41/tests -p 'test_*.py' +bash webster/deepseek-v41/tests/test_failure_paths.sh +git add webster/deepseek-v41/scripts webster/deepseek-v41/tests +git commit -m "feat: add validated LiteLLM alias cutover tooling" +``` + +### Task 6: Execute Milestones 1 and 2 — Freeze Rollback and Stage Artifacts + +**Files:** + +- Modify: `webster/deepseek-v41/README.md` with redacted milestone records. +- Create outside Git: `/home/ubuntu/deepseek-v41-runs/$CHANGE_ID/`. + +**Interfaces:** + +- Consumes: Tasks 1–5 and the live cluster. +- Produces: a GO record for milestones 1 and 2; no route, container, or service change. + +- [ ] **Step 1: Create the evidence root and baseline** + +```bash +cd /home/ubuntu/.setup +CHANGE_ID="$(date -u +%Y%m%dT%H%M%SZ)" +RUN_ROOT="/home/ubuntu/deepseek-v41-runs/$CHANGE_ID" +umask 077 +mkdir -p "$RUN_ROOT" +webster/deepseek-v41/scripts/preflight.sh --phase baseline --run-root "$RUN_ROOT" +``` + +Expected: GO; all existing public model probes pass, root is 403, unauthenticated models +is 401, LiteLLM bind is only `127.0.0.1:4446`, and both GLM ranks are healthy. + +- [ ] **Step 2: Back up exact rollback state without exposing credentials** + +Use one UTC timestamp for `spark-1:/home/nvidia/litellm/config.yaml`, callback files, +both GLM scripts, both container inspections, and credential metadata. Copy redacted +config/inspect output to the run root; leave secret-bearing backups mode `0600` only on +their source host. Record exact restore paths in `events.tsv`. + +- [ ] **Step 3: Stage Shamu, observe, then stage Tilikum** + +```bash +webster/deepseek-v41/scripts/stage-artifacts.sh --node shamu --run-root "$RUN_ROOT" +webster/deepseek-v41/scripts/stage-artifacts.sh --node tilikum --run-root "$RUN_ROOT" +``` + +Do not run these concurrently. Expected: identical manifests, 48 shards, +510,286,023,000 logical tensor bytes, 510,296,708,312 on-disk shard-file bytes, and GLM +success/latency inside the abort thresholds. + +- [ ] **Step 4: Stage and prove the guard without restarting LiteLLM** + +Install candidate guard/tokenizer assets, run the exact-image offline checks, render the +candidate config, and verify its semantic diff. Do not replace live config and do not +restart LiteLLM. + +- [ ] **Step 5: Record the gate and commit only the redacted record** + +```bash +git add webster/deepseek-v41/README.md +git commit -m "docs: record DeepSeek cutover staging gate" +``` + +### Task 7: Execute Milestones 3 and 4 — Alias Cutover, Hot Soak, and GLM Stop + +**Files:** + +- Create: `webster/deepseek-v41/scripts/stop-glm52-tp2.sh` +- Create: `webster/deepseek-v41/scripts/start-glm52-tp2.sh` +- Modify: `webster/deepseek-v41/README.md` +- Test: `webster/deepseek-v41/tests/test_failure_paths.sh` + +**Interfaces:** + +- Consumes: verified candidate config and hot GLM ranks. +- Produces: `glm-5.2` routed to GLM-5.3 under the legacy contract; old ranks stopped but recoverable. + +- [ ] **Step 1: Implement coordinated stop and cold rollback wrappers** + +`stop-glm52-tp2.sh` captures both logs/inspects, stops Shamu and Tilikum as one +operation, and verifies both containers stopped, both GPUs released, and no station +port-8000 listener. If one stop fails it retries the other stop and exits nonzero; it +never leaves a single live rank intentionally. + +`start-glm52-tp2.sh` first stops both DeepSeek ranks, then launches Shamu and Tilikum +with the existing scripts and explicit values: + +```bash +GLM_NODE_RANK=0 GLM_HOST_IP=10.10.1.1 GLM_MAX_MODEL_LEN=320000 GLM_MAX_NUM_SEQS=8 /home/alecfong/serve_glm52.sh +GLM_NODE_RANK=1 GLM_HOST_IP=10.10.1.2 GLM_MAX_MODEL_LEN=320000 GLM_MAX_NUM_SEQS=8 GLM_DOCKER='sudo -n docker' /home/alecfong/serve_glm52.sh +``` + +It waits up to 20 minutes, samples rank-1 GPU memory, requires authenticated direct +health, `--headless`, no rank-1 listener/key, and zero restart count. + +- [ ] **Step 2: Install the guard and atomically promote the candidate config** + +Run the install script, copy live config to its timestamped mode-0600 backup, re-run +allowed-diff validation against that exact live file, and rename the candidate over +`config.yaml`. Abort before restart on any mismatch. + +- [ ] **Step 3: Restart LiteLLM once and enforce the 60-second decision** + +Capture the live Docker inspect, run `verify-litellm-container.py`, and store the +redacted command-shape evidence under the run root. Abort before the restart if the +entrypoint/Cmd check fails. Then record `date +%s%N`, run `docker restart litellm`, and +poll readiness every second. +When ready, record duration, ID, start time, bind, restart count, callback imports, and +logs. If not ready by 60 seconds, inspect logs once, restore the named backup, restart +once, and stop the milestone. + +- [ ] **Step 4: Prove both aliases and the legacy boundary** + +Mint `cutover-$CHANGE_ID` scoped only to `glm-5.2`, `glm-5.3-flash`, and one unrelated +control model. Run `contract-probe.py` through loopback Caddy and the public URL. +Require old-station request counters unchanged, GLM-5.3 counters/traces increased, +legacy metadata unchanged, native GLM-5.3 metadata unchanged, and the 320,001-token +request rejected before backend counters move. Revoke the key. + +- [ ] **Step 5: Soak for 30–60 minutes** + +For at least 30 minutes and up to 60, collect five-minute success, p50/p90, stream, +reasoning, tool, structured, limit, cost, and trace checks. Any unexplained regression +restores the LiteLLM backup and performs one validated restart while station GLM stays +hot. + +- [ ] **Step 6: Cross the cold-rollback boundary deliberately** + +After a clean soak, record `GO release-stations`, execute `stop-glm52-tp2.sh`, then +re-run both alias probes. Record that rollback now requires stopping DeepSeek, loading +both GLM ranks for up to 20 minutes, validating direct health, restoring config, and one +LiteLLM restart. + +- [ ] **Step 7: Run tests and commit** + +```bash +git add webster/deepseek-v41/scripts/stop-glm52-tp2.sh \ + webster/deepseek-v41/scripts/start-glm52-tp2.sh \ + webster/deepseek-v41/README.md webster/deepseek-v41/tests +git commit -m "ops: complete the GLM compatibility alias cutover" +``` + +### Task 8: Pin and Package the DeepSeek V4.1 vLLM Runtime + +**Files:** + +- Create: `webster/deepseek-v41/runtime/Dockerfile` +- Create: `webster/deepseek-v41/runtime/constraints.txt` +- Create: `webster/deepseek-v41/runtime/README.md` +- Create: `webster/deepseek-v41/scripts/pin-runtime.py` +- Create: `webster/deepseek-v41/scripts/build-runtime.sh` +- Test: `webster/deepseek-v41/tests/test_static.py` +- Test: `webster/deepseek-v41/tests/test_failure_paths.sh` + +**Interfaces:** + +- Consumes: GitHub PR/release API evidence and the still-serving station pair. +- Produces: one immutable arm64 vLLM image loaded identically on both nodes and a source bundle in the run root. + +- [ ] **Step 1: Implement a fail-closed runtime pin decision** + +`pin-runtime.py --pr 56214 --manifest "$RUN_ROOT/manifest.env"` saves raw GitHub PR, +review, commit-status, and check-run JSON. It selects: + +1. the containing released vLLM commit when a published release includes merged V4.1 + support; otherwise +2. PR merge commit when merged and all required checks pass; otherwise +3. the exact PR head only when not draft, review requirements pass, every required CI + check succeeds, and `mergeable_state` is clean. + +It exits nonzero for `unstable`, pending, failing, missing, or changing evidence. With +the observed state `open/unstable` at head +`0bfb653d3b5161660db9ada0d84c2cdd60961de7`, the expected result is NO-GO until upstream +state improves or a separately reviewed local patch decision is recorded. + +- [ ] **Step 2: Freeze source, dependencies, and base image** + +Clone by commit, verify HEAD, save `git bundle`, `git diff --binary`, submodule states, +and the source tar SHA-256. Resolve every `FROM` reference in the selected upstream +Dockerfile to a registry digest and write it to the manifest. Build output writes a +complete `pip freeze` to `runtime/constraints.txt` and captures CUDA/NCCL versions. + +- [ ] **Step 3: Build once on Shamu and transfer immutably** + +Use the upstream arm64 `vllm-openai` Docker target at the selected commit, with the +repository Dockerfile acting only as a digest-pinned wrapper. Run the build beneath a +scope capped at four CPUs, 96 GiB RAM, and low I/O weight; no GPU device enters the +builder. Tag for human readability but launch only by local image ID. Save the image, +SHA-256 the tar, copy over `10.10.1.2` with `rsync --checksum --bwlimit=250000`, verify +before load, and assert both nodes report the same image ID and internal vLLM commit. +Monitor GLM during build/save/copy/load and abort on the same two-window success/latency +threshold used for checkpoint staging. + +- [ ] **Step 4: Prove architecture and Engram flags without serving** + +Run `vllm serve --help` in the image and require `DeepseekV41ForCausalLM` registry +support plus `--engram-config`. Parse the checkpoint config and require nonempty +`engram_layer_ids`. Validate this exact baseline flag: + +```text +--engram-config {"cpu_offload":true,"embedding_across_dp":false} +``` + +Do not include PRs #56220, #56227, or #56344 in the baseline image. + +Do not run model import, kernel JIT, GPU architecture probes, or a serving process in +this task. Those begin only after the hot alias soak and coordinated GLM stop. + +- [ ] **Step 5: Run tests and commit the reproducible runtime record** + +```bash +python3 -m unittest discover -s webster/deepseek-v41/tests -p 'test_*.py' +bash webster/deepseek-v41/tests/test_failure_paths.sh +git add webster/deepseek-v41/runtime webster/deepseek-v41/scripts \ + webster/deepseek-v41/tests webster/deepseek-v41/README.md +git commit -m "build: pin the DeepSeek V4.1 vLLM runtime" +``` + +### Task 9: Execute Milestone 5 — Bring Up the Minimal TP2/PP1 Canary + +**Files:** + +- Create: `webster/deepseek-v41/scripts/start-deepseek-v41-tp2.sh` +- Create: `webster/deepseek-v41/scripts/stop-deepseek-v41-tp2.sh` +- Create: `webster/deepseek-v41/scripts/deepseek-v41-watchdog.sh` +- Modify: `webster/deepseek-v41/README.md` +- Test: `webster/deepseek-v41/tests/test_failure_paths.sh` + +**Interfaces:** + +- Consumes: pinned image ID, complete local checkpoints, separate rank-0 mode-0600 key. +- Produces: authenticated eager text canary at `http://100.73.140.127:8000/v1`. + +- [ ] **Step 1: Implement coordinated lifecycle scripts** + +The start script validates both checkpoints/image IDs, GLM stopped, port free, 200 GiB +free, rail/HCA active, and passwordless Shamu-to-Tilikum SSH. It starts rank 0 then rank +1 as detached containers with host network/IPC, InfiniBand device, memory-lock/stack +ulimits, CAP_IPC_LOCK, image ID, read-only checkpoint, node-local cache, and: + +```text +NCCL_IB_HCA=mlx5_1 +NCCL_SOCKET_IFNAME=enP1p3s0f1np1 +GLOO_SOCKET_IFNAME=enP1p3s0f1np1 +VLLM_HOST_IP=10.10.1.1|10.10.1.2 +--nnodes 2 --node-rank 0|1 --master-addr 10.10.1.1 --master-port 29511 +--tensor-parallel-size 2 --pipeline-parallel-size 1 +--engram-config {"cpu_offload":true,"embedding_across_dp":false} +--enforce-eager --max-model-len 131072 --max-num-seqs 1 +``` + +Rank 0 adds `--host 100.73.140.127 --port 8000` and reads its key from an env file. +Rank 1 adds `--headless`, explicitly unsets all serving-key variables, and has no HTTP +argument. Both use restart `no`; recovery belongs only to the coordinated watchdog. + +- [ ] **Step 2: Implement split-brain-safe health ownership** + +The watchdog checks both container start times, both process/rank arguments, direct +health, one correct listener, one headless rank, NCCL timeout/engine-death logs, and GPU +memory. A failure captures logs and invokes coordinated stop/start only after a +20-minute startup grace and 30-minute cooldown. It never calls `docker restart` or +starts one rank. + +- [ ] **Step 3: Launch and verify security/lifecycle** + +Require correct key 200, missing/wrong key 401, `/health` and `/metrics` mesh-only, +Shamu LAN port refused, Tilikum NetBird/LAN port refused, rank 1 no credential, and no +listener on any unintended interface. Verify NCCL logs name only `mlx5_1` and direct-rail +addresses; sample Grace and GPU allocations to prove local Engram placement. + +- [ ] **Step 4: Run the eager semantic smoke suite** + +Run three repetitions each of deterministic plain text, UTF-8, stop sequence, +streaming, and usage accounting. Require coherent nonempty output, no replacement-token +rate above the fixture threshold, exact stream reconstruction, and prompt/completion +usage fields. + +- [ ] **Step 5: Run tests and commit** + +```bash +git add webster/deepseek-v41/scripts webster/deepseek-v41/tests \ + webster/deepseek-v41/README.md +git commit -m "feat: launch the private DeepSeek V4.1 TP2 canary" +``` + +### Task 10: Execute Milestone 6 — Progressive Correctness Qualification + +**Files:** + +- Create: `webster/deepseek-v41/scripts/feature-qualification.py` +- Modify: `webster/deepseek-v41/README.md` +- Test: `webster/deepseek-v41/tests/test_static.py` + +**Interfaces:** + +- Consumes: direct private canary and named profile manifests. +- Produces: one JSON result per feature stage and `last-known-good.env`. + +- [ ] **Step 1: Encode the reusable API/correctness suite** + +Implement cases for deterministic text, UTF-8, streaming equivalence, stop sequences, +reasoning control, tool auto, named/required tool behavior recording, parallel tool +shape, tool-result continuation, strict schema JSON, `json_object`, usage details, +disconnect cancellation/queue drain, malformed input, wrong model, and over-limit +errors. Vision uses a repository-owned 2x2 PNG fixture encoded in the request; no remote +image URL is fetched. + +Each case records request hash, status, finish reason, token usage, latency, normalized +output hash, semantic validator result, and the last 100 server log lines on failure. + +- [ ] **Step 2: Qualify eager text and the 320K floor** + +Run short cases, then tokenizer-sized 128K, 256K, 319K, 320K, and 320K+1 prompts with +small outputs. Require no rank death, stable memory, and clean 400 above the configured +window. Save the exact profile as the first last-known-good candidate. + +- [ ] **Step 3: Enable compilation/CUDA graphs and compare output** + +Remove `--enforce-eager`, warm every intended batch shape, and rerun the full text/API +suite. Compare eager and graph outputs at temperature zero plus semantic validators. +Any garbling, graph break, uncaptured production shape, or unacceptable KV loss returns +to eager. + +- [ ] **Step 4: Enable DSpark alone and compare output/acceptance** + +Add only the pinned runtime's documented DSpark configuration. Capture accepted draft +length by position, step time, rejection mode, output hashes, and quality validators. +Synthetic/forced acceptance is forbidden. Any correctness drift or engine instability +returns to graphs-without-DSpark. + +- [ ] **Step 5: Enable and qualify vision alone** + +Run local image description, image-plus-text, malformed image, streaming multimodal, +and usage cases. Advertise vision in the final model only if all pass. + +- [ ] **Step 6: Raise the context ceiling to 1,048,576 last** + +Test 512K, 768K, 1M-minus-headroom, exact 1M, and over-limit with concurrency one. +Require no host swap storm, OOM, rank loss, NCCL timeout, or recovery failure. If 1M +fails, retain the highest passing ceiling and record that exact value for publication. + +- [ ] **Step 7: Select and commit the highest passing profile** + +```bash +git add webster/deepseek-v41/scripts/feature-qualification.py \ + webster/deepseek-v41/README.md webster/deepseek-v41/tests +git commit -m "test: qualify DeepSeek V4.1 serving features" +``` + +### Task 11: Execute Milestone 7 — Tune TP2 and Conditionally Benchmark PP2 + +**Files:** + +- Modify: `webster/deepseek-v41/scripts/start-deepseek-v41-tp2.sh` +- Modify: `webster/deepseek-v41/scripts/feature-qualification.py` +- Modify: `webster/deepseek-v41/README.md` + +**Interfaces:** + +- Consumes: highest passing correctness profile. +- Produces: frozen TP2 production candidate and an optional matched PP2 decision record. + +- [ ] **Step 1: Freeze the one-factor matrix before running it** + +Record baseline and candidate values for GPU memory utilization, max sequences, batched +tokens, chunked prefill, async scheduling, graph capture sizes, prefix caching, DSpark, +and EP. Change exactly one factor, restart both ranks, run correctness smoke, warm once, +measure three times, and return to baseline before the next factor. + +- [ ] **Step 2: Measure the variables that explain throughput** + +For every run capture request/session rate, TTFT p50/p90/p99, ITL, decode/user, +aggregate tokens, prefix-hit rate, KV occupancy, queue depth, preemptions, accepted draft +length, step time, GPU/Grace memory, NCCL bytes/packets, power, errors, and engine logs. +Do not infer progress from GPU utilization alone. + +- [ ] **Step 3: Test EP only after stable non-EP TP2** + +Use the same image/checkpoint and change only the documented EP switch. Require the full +API suite plus three matched warm workload repetitions. Reject EP if it introduces +semantic drift, imbalance, instability, or no Pareto improvement. + +- [ ] **Step 4: Apply the PP2 trigger and matched decision rule** + +Run PP=2/TP=1 only if TP2 fails correctness/memory/throughput, or NCCL synchronization +accounts for at least 15% of steady-state step time. PP2 must use the same context, +features, seed, cache state, concurrency, duration, and output checks. Keep PP2 only if +it improves sessions/s by at least 10% with no more than 5% p90/p99 regression, or +improves p90 by at least 10% with no more than 5% sessions/s regression. Otherwise +record `PP2 rejected` and retain TP2. + +- [ ] **Step 5: Re-run correctness and commit the chosen profile** + +```bash +git add webster/deepseek-v41/scripts webster/deepseek-v41/README.md +git commit -m "perf: select the DeepSeek V4.1 station profile" +``` + +### Task 12: Execute Milestone 8 — Run Controlled Full-Subagent Weka Replay + +**Files:** + +- Create: `webster/deepseek-v41/scripts/run-weka.sh` +- Create: `webster/deepseek-v41/scripts/summarize-aiperf.py` +- Modify: `webster/deepseek-v41/README.md` +- Test: `webster/deepseek-v41/tests/test_static.py` + +**Interfaces:** + +- Consumes: direct private canary and a mode-0600 benchmark key file on `head`. +- Produces: raw AIPerf/telemetry bundles and a Pareto report for the frozen matrix. + +- [ ] **Step 1: Pin the client and corpus** + +Create an isolated venv from an exact `ai-dynamo/aiperf` commit, record its commit and +lock, and resolve `semianalysisai/cc-traces-weka-062126` to an immutable Hugging Face +revision. Require the full 393-trace subagent corpus. The rolling +`semianalysis_cc_traces_weka_with_subagents` alias may resolve discovery metadata but is +not accepted as the recorded pin. + +- [ ] **Step 2: Implement safe persistent runs** + +`run-weka.sh` reads the API key file into an exported environment variable, never argv; +uses `mktemp -d -p /home/ubuntu/deepseek-v41-tmp`; writes the fully rendered non-secret +command; and runs under tmux. It traps exit to write exactly `EXIT=0` or the nonzero +integer exit status and copies all +artifacts before recursively deleting only the resolved temporary directory after +validating its prefix and ownership. + +Every run uses the pinned tokenizer, direct URL `http://100.73.140.127:8000`, chat, +streaming, full subagents, an explicit 900-second benchmark duration, 300-second grace, +900-second request timeout, and fixed seed. No request-count termination is allowed. + +- [ ] **Step 3: Run the frozen closed-loop matrix** + +Run concurrency 1, 2, 4, and 8 with identical trace sequence and +`--no-fixed-schedule`; add 16 only if 8 has 99% success, zero engine deaths/timeouts, +and at least 20% free KV headroom. For warm comparisons run each point twice and report +only repetition two. For cold comparisons call the coordinated restart, prove cache +counters reset, and run each candidate from the same cold state. + +- [ ] **Step 4: Run the fixed-schedule and arrival-rate qualification** + +Run one full recorded-timeline replay with subagent SPAWN/JOIN behavior. Derive three +open-loop arrival rates from the seven-day observed p50, p90, and peak five-minute +request rates and freeze them in the manifest before execution. Keep trace delay rules, +duration, grace, timeout, and seed matched across candidates. + +- [ ] **Step 5: Summarize correctness and performance** + +`summarize-aiperf.py` counts failed JSONL rows before reading aggregates and emits: +completion/error rate, requests/s, sessions/s, input/output tok/s, per-user decode, +TTFT p50/p90/p99, ITL, end-to-end latency, ISL/OSL, cache hit, KV occupancy, queue, +preemptions, GPU/host memory, NCCL, and power. Scan representative plain, reasoning, +tool, JSON, vision, and long-context output for garbling. + +The selected profile requires at least 99% server-side success, zero engine deaths, +zero NCCL timeouts, zero corruption flags, and reproducible warm results. Choose the +Pareto point; do not hide latency to maximize aggregate tokens. + +- [ ] **Step 6: Commit the runner and redacted decision record** + +```bash +git add webster/deepseek-v41/scripts/run-weka.sh \ + webster/deepseek-v41/scripts/summarize-aiperf.py \ + webster/deepseek-v41/README.md webster/deepseek-v41/tests +git commit -m "perf: qualify DeepSeek V4.1 on Weka traces" +``` + +### Task 13: Execute Milestone 9 — Register the New Model and Publish the Record + +**Files:** + +- Create: `webster/deepseek-v41/scripts/acceptance.sh` +- Create: `webster/deepseek-v41/scripts/sync-runbook.sh` +- Modify: `webster/deepseek-v41/scripts/render-litellm-cutover.py` +- Modify: `webster/deepseek-v41/scripts/verify-litellm-config.py` +- Modify: `webster/deepseek-v41/README.md` +- Modify live after verification: `/home/ubuntu/.claude/skills/webster-cluster/SKILL.md` + +**Interfaces:** + +- Consumes: selected passing profile, measured costs, and pre-registration LiteLLM backup. +- Produces: one new public `deepseek-v4.1-flash` entry plus synchronized Webster operating documentation. + +- [ ] **Step 1: Extend the renderer/verifier for one additive model** + +`--phase publish` allows exactly one new `model_name: deepseek-v4.1-flash`; it rejects +any change to `deepseek-v4-flash`, the GLM aliases, callbacks, router plugin, global +settings, and Caddy. The entry points to authenticated Shamu NetBird port 8000, uses the +actual served model name/image profile, sets the highest passing context, advertises +only passing reasoning/tool/structured/vision features, and uses measured marginal +input/output energy costs from the accepted Weka profile. + +- [ ] **Step 2: Validate offline and perform the second planned restart** + +Back up the exact live config, render to a candidate, run semantic diff, parse/import in +the pinned LiteLLM image with `--network none`, atomically promote, capture the live +inspect, and require `verify-litellm-container.py` to pass before restarting once. +If readiness is absent at 60 seconds or any cross-model probe fails, restore the backup, +restart once, and leave DeepSeek private. + +- [ ] **Step 3: Run end-to-end and security acceptance** + +Mint `publish-$CHANGE_ID` scoped to the new model plus unrelated controls. Require +direct canary, private Caddy, and public completion; Langfuse trace; Prometheus series; +403 root; 401 unauthenticated models; loopback-only LiteLLM; zero restart count after +the planned restart; rank isolation/auth; no Baker metric/container/config change; and +successful `glm-5.2`, `glm-5.3-flash`, `deepseek-v4-flash`, and Inkling control probes. +Revoke the key. + +- [ ] **Step 4: Update canonical operational truth only from live evidence** + +Edit the canonical Webster skill's current-state summary, Models table, station restart +procedure, DeepSeek topology/profile, known gaps, and verification date. Do not copy +planned-but-failed features into the skill. Run: + +```bash +/home/ubuntu/ops-bot/check-skill-drift.sh +/home/ubuntu/ops-bot/sync-skills.sh +``` + +If drift is reported, review and fold valid ops-bot changes into the canonical copy, +then sync. Verify Claude, Codex, and Hermes copies describe the same live state and +`hermes skills list` still includes Webster. + +- [ ] **Step 5: Publish the repeatable playbook evidence** + +README receives actual timings, backup paths, gate decisions, failed variants, selected +profile, rollback commands, benchmark report path, and the limited-downtime lessons: +control-plane route first, hot soak before data-plane release, private qualification, +coordinated ranks, exactly two front-door restarts, immutable artifacts, and no cleanup +inside the change. The subsequent skill-creation change will derive from this exercised +record rather than copying untested commands. + +- [ ] **Step 6: Run full verification and commit** + +```bash +bash -n webster/deepseek-v41/scripts/*.sh +python3 -m unittest discover -s webster/deepseek-v41/tests -p 'test_*.py' +bash webster/deepseek-v41/tests/test_failure_paths.sh +./test.sh +git diff --check +git add webster/deepseek-v41 test.sh +git commit -m "docs: publish the DeepSeek V4.1 cutover playbook" +``` + +Expected: all repository checks pass, all three Webster skill copies agree, no secret +scanner finding exists, and the final acceptance record is GO. + +## Rollback Commands by Boundary + +Before the alias restart, discard the candidate; no service action is required. + +During the hot-GLM soak: + +```bash +webster/deepseek-v41/scripts/restore-litellm-config.sh --phase alias --run-root "$RUN_ROOT" +``` + +After station release, stop DeepSeek, restore both GLM ranks, verify direct health, then +restore the LiteLLM backup and restart once: + +```bash +webster/deepseek-v41/scripts/stop-deepseek-v41-tp2.sh --run-root "$RUN_ROOT" +webster/deepseek-v41/scripts/start-glm52-tp2.sh --run-root "$RUN_ROOT" +webster/deepseek-v41/scripts/restore-litellm-config.sh --phase alias --run-root "$RUN_ROOT" +``` + +After new-model registration, restore only the pre-registration LiteLLM backup and +restart once while leaving the private canary available for diagnosis: + +```bash +webster/deepseek-v41/scripts/restore-litellm-config.sh --phase publish --run-root "$RUN_ROOT" +``` + +## Final Verification Checklist + +- [ ] `glm-5.2` routes to GLM-5.3 but retains its public name, 320K shared window, + reasoning/function metadata, no vision, explicit over-budget rejection, and the + remaining-budget default when no output limit is supplied. +- [ ] Native `glm-5.3-flash` retains 1M and vision; Baker remains unchanged. +- [ ] Old GLM ranks are stopped together and all rollback artifacts remain. +- [ ] DeepSeek V4.1 runs from identical pinned checkpoint/runtime artifacts on both + stations, TP2/PP1 unless the matched PP2 decision says otherwise. +- [ ] Engram is local Grace memory, NCCL is `mlx5_1`, rank 1 is headless/keyless, and + rank 0 is authenticated/NetBird-only. +- [ ] The highest enabled feature profile passed the full correctness suite. +- [ ] Full-subagent Weka meets 99% success, zero death/timeout/corruption, controlled + cache, and reproducibility gates. +- [ ] Exactly two planned LiteLLM restarts and their durations/rollback decisions are + recorded. +- [ ] `deepseek-v4.1-flash` is independently registered and observed in Prometheus and + Langfuse. +- [ ] Repository tests, `./test.sh`, shell syntax, and `git diff --check` pass. +- [ ] Canonical Claude, Codex, and Hermes Webster skills agree with verified live state. +- [ ] No secret exists in Git or the published evidence. diff --git a/docs/superpowers/specs/2026-09-11-glm52-to-deepseek-v41-cutover-design.md b/docs/superpowers/specs/2026-09-11-glm52-to-deepseek-v41-cutover-design.md new file mode 100644 index 0000000..b745d6a --- /dev/null +++ b/docs/superpowers/specs/2026-09-11-glm52-to-deepseek-v41-cutover-design.md @@ -0,0 +1,586 @@ +# GLM-5.2 Deprecation and DeepSeek V4.1 Flash Station Cutover Design + +**Date:** 2026-09-11 +**Status:** Approved + +## Summary + +This change deprecates the public `glm-5.2` implementation without removing its +client-visible model name. LiteLLM will route both `glm-5.2` and +`glm-5.3-flash` requests to the existing `glm-5.3-flash` backend. The current +GLM-5.2 TP2 deployment on `shamu` and `tilikum` will remain hot for a 30–60 +minute rollback soak, then both ranks will be stopped together. + +The freed DGX Station GB300 pair will be used to qualify +`deepseek-ai/DeepSeek-V4.1-Flash` on vLLM. The primary topology is TP=2, PP=1 +over the stations' direct 400 Gb/s ConnectX-8 link. Engram tables will reside in +each station's local Grace memory, while the non-Engram weights are tensor +parallel across the two GPUs. PP=2 is a measured fallback, not the starting +configuration. + +DeepSeek V4.1 will remain on a private, authenticated canary endpoint while its +runtime, API behavior, long-context behavior, and real-workload performance are +qualified. Features will be introduced progressively: eager text, the 320K +contract, CUDA graphs, DSpark speculative decoding, vision, and finally the 1M +context target. Only a passing configuration will be registered in LiteLLM as +the new model name `deepseek-v4.1-flash`. The existing Baker +`deepseek-v4-flash` route is not replaced or modified. + +The work is organized as nine gated milestones so the public endpoint sees at +most two planned LiteLLM restart blips: one to repoint the legacy GLM alias and, +later, one to expose the qualified DeepSeek model. Station model restarts do not +require LiteLLM restarts. + +## Goals + +- Preserve the `glm-5.2` client-visible name and existing virtual-key scopes + and its original 320K client contract while serving those requests from the + current `glm-5.3-flash` backend. +- Reclaim `shamu` and `tilikum` without a GLM-5.2 client outage beyond the + single planned LiteLLM restart. +- Establish a reproducible, pinned vLLM runtime for DeepSeek V4.1 Flash. +- Fit and serve the model on two DGX Station GB300 systems using local NVMe, + local Grace memory, and the direct ConnectX-8 fabric. +- Qualify correctness before adding performance features. +- Optimize against AIPerf's Weka trace, including the full subagent replay, + under controlled cache conditions. +- Preserve a fast rollback until the alias cutover has soaked and a recoverable + rollback after the stations have been repurposed. +- Capture commands, timestamps, artifacts, measurements, failure modes, and + rollback decisions in a form that can become a repeatable playbook and Codex + skill. + +## Non-goals + +- Remove the public `glm-5.2` name or force clients to change configuration. +- Change the external inference URL, Caddy policy, or LiteLLM bind address. +- Replace, pool with, or retune Baker's `deepseek-v4-flash` deployment. +- Expose an unqualified DeepSeek V4.1 backend through LiteLLM. +- Treat PP=2 as the default merely because there are two physical stations. +- Tune every upstream or experimental vLLM patch in one image. +- Claim a 1M production contract before correctness and resource headroom pass + at that length. +- Delete the GLM-5.2 checkpoint, rollback scripts, stopped containers, or + configuration backups during this project. +- Package the final skill before the operational procedure has been exercised + and corrected against live evidence. + +## Existing State and Constraints + +### Production routing + +| Public model | Current backend | Intended state after this change | +|---|---|---| +| `glm-5.2` | `http://100.73.140.127:8000/v1` on the station TP2 pair | Existing `glm-5.3-flash` backend, retained as a deprecated compatibility alias | +| `glm-5.3-flash` | `http://100.73.165.55:8000/v1` | Unchanged backend and native model name | +| `deepseek-v4-flash` | Baker at `http://100.73.127.129:8888/v1` | Unchanged | +| `deepseek-v4.1-flash` | Not registered | Added only after private qualification | + +LiteLLM runs on `spark-1`, binds only to `127.0.0.1:4446`, and is fronted by +Caddy on ports 4444 and 4445. The public inference endpoint is load-bearing. +Configuration changes must therefore be validated offline, batched, backed up, +and deployed with one restart per planned routing event. + +The latest seven-day traffic snapshot recorded 1,344 `glm-5.2` calls with a +prompt-length p90 of approximately 198,850 tokens, and 22,748 +`glm-5.3-flash` calls with a p90 of approximately 177,962 tokens. These data +support a compatibility floor of 320K for the replacement station service and +make long-context, multi-turn behavior a primary benchmark regime rather than +an edge case. + +The legacy alias retains its original compatibility contract: model name +`glm-5.2`; a 320,000-token shared prompt-and-completion ceiling; existing +reasoning, streaming, structured-output, and function-calling metadata; and no +advertised vision or 1M capability. Requests above the shared 320K ceiling must +continue to fail before reaching the GLM-5.3 backend. This preserves the option +to restore the former backend without first unwinding client dependencies on +GLM-5.3-only capabilities. Exact generations cannot remain identical because +the underlying weights change; “original contract” refers to the API surface, +advertised capabilities, limits, and failure behavior. Energy-cost metadata +will describe the backend that actually serves the request so accounting stays +truthful. + +### Station topology + +- `shamu` is rank 0 and will host the only HTTP listener. +- `tilikum` is rank 1 and must remain headless. +- NCCL traffic uses `10.10.1.1` and `10.10.1.2` on + `enP1p3s0f1np1` / `mlx5_1`, the configured 400 Gb/s ConnectX-8 rail. +- NetBird is the service plane. The HTTP listener binds to Shamu's NetBird + address, never `0.0.0.0` or the Webster LAN. +- Both stations have approximately 744 GiB of host memory and one approximately + 250.69 GiB GPU. +- Measured Grace CPU-to-GPU bandwidth is approximately 375–379 GB/s, making + local Grace memory the intended Engram tier. Cross-station Engram lookup is + not part of the design. + +The existing GLM-5.2 deployment uses the same rank ownership and direct rail. +Its rank-1 `--headless` requirement, memory-lock settings, API isolation, and +coordinated stop/start behavior remain the rollback template. + +### DeepSeek artifact sizing + +The selected checkpoint revision is: + +```text +repository: deepseek-ai/DeepSeek-V4.1-Flash +revision: dba1be0a40aa45a94ad051997016db3960a90277 +``` + +The revision's safetensors index declares 510,286,023,000 logical tensor bytes +(475.24 GiB). Its 48 shard files total 510,296,708,312 on-disk bytes after +safetensors headers. Engram tables account for approximately 189.13 GiB and the +remaining weights for approximately 286.11 GiB. With TP=2, the ideal +non-Engram GPU share is approximately 143.06 GiB per rank, leaving about +107.63 GiB of each GPU for runtime state, graphs, activations, and KV cache. +Each station must keep a complete local checkpoint on NVMe and its local Engram +tables in local Grace memory. + +Before staging, each station must demonstrate enough free NVMe for the complete +checkpoint, the runtime image, caches, and a temporary download margin. The +implementation plan will set the exact minimum from the downloader's real +layout; it may not assume that the nominal 475.24 GiB checkpoint is the only +disk consumer. + +The pinned release intentionally has no Jinja chat template, generation config, +or processor config. Its protocol reference is `encoding/encoding.py` plus the +published encoding fixtures, and the candidate vLLM source registers +`deepseek_v41` with its native `DeepseekV4Renderer`. Checkpoint verification must +require those published assets rather than inventing conventional Transformers +files that do not exist at this revision. + +### Runtime support state + +The installed vLLM 0.25.1 build does not recognize +`DeepseekV41ForCausalLM`. Upstream support is being developed in +[vLLM PR #56214](https://github.com/vllm-project/vllm/pull/56214). Its last +observed head was `0bfb653d3b5161660db9ada0d84c2cdd60961de7`, with an unstable CI +state. That hash is evidence, not an approved runtime pin. + +The runtime milestone must recheck the PR, its CI, review status, and merge or +release state immediately before pinning. The chosen source commit, base image, +container digest, dependency lock, CUDA version, NCCL version, driver versions, +and local patch diff must be recorded. No deployment may use a floating branch +or tag. + +Potential follow-on optimizations include Engram overlap +([#56220](https://github.com/vllm-project/vllm/pull/56220)), bounded SWA replay +([#56227](https://github.com/vllm-project/vllm/pull/56227)), and attention/FP4 KV +work ([#56344](https://github.com/vllm-project/vllm/pull/56344)). They are +separate experimental deltas. None may be bundled into the first known-good +runtime, and each must prove its own correctness and performance effect. + +## Target Architecture + +```text +Existing public clients + | + | model = glm-5.2 or glm-5.3-flash + v +Caddy :4444 -> LiteLLM 127.0.0.1:4446 + | + v + GLM-5.3 Flash backend + +Private qualification client on head + | + | authenticated OpenAI-compatible API over NetBird + v +Shamu rank 0, DeepSeek V4.1, TP=2 / PP=1 + | + | NCCL only: 10.10.1.1 <-> 10.10.1.2, mlx5_1 + v +Tilikum rank 1, headless + +Each station: + local NVMe -> pinned checkpoint + local Grace RAM -> local Engram tables + local GPU -> TP shard + activations + graphs + KV cache +``` + +The canary will reuse the station model port `8000` after GLM-5.2 is stopped. +This preserves the existing NetBird-only service shape and the interface-scoped +LAN guard. DeepSeek credentials live in separate mode-0600 files and do not +overwrite GLM rollback credentials. Reusing the port means only one station +model can run at a time; every transition must stop both ranks before starting +the other model. + +## Nine Milestones + +Each milestone ends at a gate. A failed gate stops progression and invokes the +stated rollback or leaves the current production path untouched. + +### Milestone 1 — Capture baseline and freeze rollback state + +Record the exact live LiteLLM configuration, container ID, image, start time, +restart count, loopback bind, model list, callbacks, routing plugin, virtual-key +behavior, and public security responses. Capture the GLM-5.2 and GLM-5.3 +backend model lists, health, container inspections, launch scripts, GPU/host +memory, direct-rail state, and recent traffic/error metrics. + +Create timestamped backups without printing credentials. Record file ownership, +mode, and hashes for credential-bearing files. Confirm that the current +GLM-5.2 coordinated launch and restore paths exist on both stations. No config, +container, or route changes occur in this milestone. + +**Gate:** the baseline is internally consistent, all current models pass a small +authenticated completion, the external root returns 403, unauthenticated +`/v1/models` returns 401, LiteLLM is on `127.0.0.1:4446`, and rollback artifacts +can be located without relying on shell history. + +### Milestone 2 — Pre-stage immutable artifacts without a serving change + +While GLM-5.2 still serves, stage the checkpoint revision and the selected +vLLM image on both stations. Use low-priority, rate-limited, sequential I/O so +checkpoint downloads or image extraction do not contend materially with the +production engine. Do not compile GPU kernels or start a second GPU workload +alongside GLM. + +The model resides on local NVMe on both nodes. Validate the Hugging Face +revision, shard inventory, per-file hashes, total size, tokenizer and processor +files, and byte-for-byte agreement between nodes. Validate image digests and +source revision on both nodes. Keep temporary download state out of the model's +final immutable directory. + +**Gate:** both stations have identical pinned artifacts, enough remaining disk +and memory headroom, and no material GLM health or latency regression during +staging. Any unexplained production degradation aborts staging before the alias +cutover. + +### Milestone 3 — Repoint the deprecated `glm-5.2` alias + +Back up `spark-1:~/litellm/config.yaml`. Change only the legacy `glm-5.2` +deployment so it targets the existing authenticated GLM-5.3 backend. Keep the +native `glm-5.3-flash` entry. Preserve the legacy name, virtual-key scopes, +callbacks, Caddy policy, all unrelated model entries, and the legacy alias's +320K text/reasoning/function-calling contract. Do not advertise GLM-5.3's 1M or +vision capabilities under `glm-5.2`. Update only the alias's backend and +energy-cost attribution, and mark its description as a deprecated compatibility +alias. Enforce the 320K shared prompt-and-completion ceiling at LiteLLM so an +over-limit request cannot begin depending on behavior the rollback backend does +not support. + +Validate YAML structure, callback imports, the expected model/deployment +counts, backend authentication, and the rendered target before restarting. +Restart LiteLLM exactly once. + +Use a temporary named virtual key scoped to the GLM aliases for real probes; +never use the master key as workload traffic. Test both model names and verify +from backend metrics or request metadata that they reached GLM-5.3, not the +station GLM-5.2 process. Revoke the temporary key after validation. + +**Gate:** LiteLLM returns within the downtime budget; both GLM names complete, +the old backend receives no new alias probe, unrelated models still complete, +a Langfuse trace records the correct alias and backend, the root/401 security +checks are unchanged, the bind remains loopback-only, and restart count is +zero after the deliberate restart. `/model/info` and `/v1/models` still expose +the original `glm-5.2` limits and capabilities, a request above the shared 320K +limit fails without reaching GLM-5.3, and the native `glm-5.3-flash` name retains +its independent 1M and vision contract. + +**Rollback:** restore the timestamped config and perform one validated LiteLLM +restart. Because the station GLM remains hot, this rollback does not require a +model reload. + +### Milestone 4 — Soak the alias, then release the stations + +Keep both GLM-5.2 station ranks running for 30–60 minutes after the successful +alias change. During the soak, observe request success, p50/p90 latency, +streaming, reasoning controls, tool calling, structured outputs, long prompts, +over-limit rejection, cost attribution, and Langfuse traces. Compare +`glm-5.2` compatibility-alias traffic with native `glm-5.3-flash` traffic and +confirm that no request reaches the old station backend. + +At the end of a clean soak, stop the two GLM-5.2 ranks as one coordinated +operation. Do not delete their containers, launch scripts, credentials, +checkpoint, or backups. Confirm that `glm-5.2` continues to serve through +GLM-5.3 after the stop. + +**Gate:** the soak contains no unexplained compatibility or availability +regression, both old ranks are stopped, no station process listens on port +8000, GPU memory is released on both stations, and the public aliases remain +healthy. + +**Rollback:** before the stop, restore the LiteLLM config as in Milestone 3. +After the stop, first stop any DeepSeek ranks, then restore GLM-5.2 on both +stations with the coordinated scripts and only then restore the LiteLLM route. +The latter path may take 5–15 minutes because a cold GLM checkpoint load has +previously taken 14m41s. + +### Milestone 5 — Bring up the minimal TP2 DeepSeek canary + +Launch the pinned V4.1 runtime with TP=2, PP=1 and the smallest useful eager, +text-only profile. Keep CUDA graphs, DSpark speculative decoding, vision, and +the 1M target disabled. Start rank 0 first and rank 1 headless, using the +station rail for NCCL. Engram data must be served from each node's local Grace +memory; the canary fails if a rank reads Engram remotely or silently falls back +to an unintended device. + +Rank 0 binds only to Shamu's NetBird address on port 8000 and requires a private +vLLM bearer key for OpenAI routes. Health and metrics may remain unauthenticated +only if they are restricted to the intended mesh listener. Rank 1 receives no +API credential, has no HTTP listener, and exposes no model API. Both ranks must +have explicit, coordinated lifecycle scripts and a restart policy or watchdog +that cannot restart a single rank into split brain. + +**Gate:** the architecture is recognized without a local model-code fork unless +that fork is pinned and documented; weights and Engram tables land in the +intended tiers; NCCL uses only `mlx5_1`; both ranks remain healthy; unauthenticated +and wrong-key OpenAI calls return 401; a correct-key text completion is +semantically valid and repeatable; the LAN address refuses port 8000; and rank +1 is headless with no credential or listener. + +### Milestone 6 — Qualify features progressively + +Advance through the following sequence without combining steps: + +1. Eager text correctness and short-context OpenAI compatibility. +2. A 320K shared prompt/completion contract, matching or exceeding the retired + GLM route's practical context requirement. +3. CUDA graphs and compilation, with eager mode retained as the diagnostic + fallback. +4. DSpark speculative decoding, including acceptance-length and output-quality + checks. +5. Vision inputs and multimodal request parsing. +6. The checkpoint's 1M context target, including memory headroom and recovery + behavior. + +At every step, rerun deterministic prompts, streaming, stop sequences, +reasoning controls, tool calls and tool-result continuation, structured JSON, +usage fields, cancellation, and over-limit errors. Add new batch shapes only +after warmup covers them. Community reports of garbled output under CUDA graphs +and DSpark make eager-vs-graph and DSpark-off-vs-on output comparisons mandatory. + +**Gate:** each newly enabled feature passes before the next is introduced. A +feature that corrupts output, destabilizes a rank, or consumes unacceptable KV +headroom is disabled without blocking qualification of the last known-good +profile. The production candidate is the highest passing profile, not +necessarily the feature-maximal profile. + +### Milestone 7 — Optimize TP2 and measure PP2 only as fallback + +Tune the TP=2, PP=1 candidate one factor at a time. Candidate variables include +GPU memory utilization, maximum sequences, batched-token budget, chunked +prefill, async scheduling, graph capture sizes, prefix caching, DSpark settings, +and expert parallelism. EP is optional and enters only after the non-EP TP2 +baseline is stable. Each variant must preserve the same runtime and checkpoint +pins so its effect is attributable. + +PP=2 uses one pipeline stage per station with TP=1 and is benchmarked only if +TP2 cannot meet a correctness, memory, or throughput gate, or if measurements +show collectives dominate. It may reduce per-layer cross-node collectives, but +it introduces pipeline bubbles, makes low-concurrency latency vulnerable, and +can create stage or Engram imbalance. PP2 is accepted only if it passes the same +API/correctness suite and materially improves the target Weka workload without +an unacceptable p90/p99 regression. + +**Gate:** a reproducible TP2 candidate is selected from controlled measurements. +If PP2 is tested, the decision record contains matched TP2/PP2 results and an +explicit keep/reject decision. Configuration folklore or GPU utilization alone +is not evidence. + +### Milestone 8 — Run the controlled AIPerf Weka qualification + +Run AIPerf from `head`, not on either serving station, under `tmux` or an +equivalent persistent session. Pin the AIPerf source/version and dataset +manifest. The primary workload is the full Weka trace including subagent +replay; the no-subagent variant may be used only as a diagnostic control. +Preserve conversation/session identifiers so parent and subagent turns that +share a prefix exercise the intended prefix-cache behavior. + +Use duration-bounded runs with a finite grace period and an explicit long +request timeout. Do not use a request-count termination that can cut a +multi-turn session and wait forever. Completion watchers must match an exact +terminal record such as `^EXIT=`, never the word `DONE`. Retain raw records, +aggregate JSON, client logs, server logs, Prometheus snapshots, and the rendered +command for every run. + +Freeze the comparison matrix in the run manifest before testing variants. It +starts with matched closed-loop concurrency points 1, 2, 4, and 8; concurrency +16 is added only if 8 is stable and has resource headroom. Add an open-loop +arrival-rate sweep derived from the observed seven-day traffic distribution. +Every candidate receives the same sequence, duration, grace period, timeout, +and randomization seed. + +Cache state is an experimental variable: + +- A cold comparison requires a coordinated two-rank restart and proof that the + relevant cache counters began cold. +- A warm comparison runs the same configuration twice and reports the second + run. +- Never compare one warm candidate with one cold candidate. +- Report request rate and completed sessions alongside token throughput. +- Record prefix-cache hit rate, input sequence length, and output sequence + length for every result. +- Remove abandoned AIPerf mmap files on `head` between runs; do not place them + on station NVMe or evict the serving checkpoint's page cache. + +At minimum, report completion/error rate, requests/s, sessions/s, input and +output token throughput, per-user decode rate, TTFT p50/p90/p99, inter-token +latency, end-to-end latency, cache-hit rate, KV occupancy, queue depth, GPU and +host-memory headroom, NCCL traffic, and power. Review a representative output +sample for tool-call, JSON, reasoning, multimodal, and garbling failures. + +**Gate:** the selected profile completes the frozen concurrency and arrival-rate +matrix with at least 99% server-side success, zero engine deaths, zero NCCL +timeouts, no detected output corruption, and reproducible warm results. The +report identifies the Pareto choice rather than hiding latency to maximize one +throughput number. If no candidate passes, DeepSeek remains private and the +public routing state is unchanged. + +### Milestone 9 — Register the new model and publish the operating record + +After qualification, add exactly one new LiteLLM model name: +`deepseek-v4.1-flash`. It points to the authenticated station rank-0 endpoint. +Do not modify or alias Baker's `deepseek-v4-flash`. Populate model limits, +reasoning, tool, structured-output, and vision capability metadata only for +features that passed Milestone 6. Set energy pricing from measured marginal +power and measured throughput, not a hardware peak estimate. + +Validate the full LiteLLM config and custom callbacks offline, then perform the +second and final planned LiteLLM restart. Use a temporary named virtual key +scoped to the new model for end-to-end probes and revoke it afterward. Verify a +Langfuse trace and all security invariants. + +Update the canonical Webster cluster skill only after live verification. Update +its current-state summary, Models table, restart procedure, topology details, +known gaps, and verification date, then run `head:~/ops-bot/sync-skills.sh` to +refresh both the Codex and Hermes mirrors. Resolve reverse-direction drift +before forcing a sync. + +Publish the change record and benchmark report. Extract the validated procedure +into a repeatable model-cutover playbook, then package it as a narrowly scoped +skill in a separate reviewed change. The skill must encode preflight, +low-downtime routing, credential handling, coordinated-rank lifecycle, +progressive feature gates, cache-controlled AIPerf methodology, rollback, and +anti-drift documentation checks. + +**Gate:** the new model completes direct, private-Caddy, and public endpoint +probes; unrelated models still complete; LiteLLM remains loopback-only with +zero restart count; public root and unauthenticated model-list responses remain +403/401; Baker is unchanged; Prometheus and Langfuse observe the new route; all +three Webster skill copies agree; and the operating record contains no secret. + +## Downtime Budget and Change Windows + +The goal is continuity, not literal zero process restarts. + +| Event | Expected user impact | Budget | Failure action | +|---|---|---:|---| +| Artifact staging | None | No endpoint downtime | Throttle or stop staging if production health moves | +| Legacy alias change | One global LiteLLM blip | Target <30 seconds; 60-second decision threshold | If readiness is not restored by 60 seconds, inspect once, then restore the backed-up config and restart | +| GLM station stop | None for `glm-5.2`; it is already on GLM-5.3 | No LiteLLM restart | Restart both GLM ranks only if alias rollback is required | +| DeepSeek bring-up and tuning | None; private canary only | No LiteLLM restart | Stop both ranks, preserve logs, return to last known-good profile | +| DeepSeek registration | One global LiteLLM blip | Target <30 seconds; 60-second decision threshold | Restore the backed-up config and restart if readiness or cross-model probes fail | + +The two planned LiteLLM restarts are separated by the full private +qualification period and are never spent on tuning hunches. The target total +planned front-door interruption is under 60 seconds. If either restart crosses +its 60-second decision threshold, allow up to one additional 60-second recovery +restart for that event; do not begin another change while the front door is +unhealthy. + +The 30–60 minute hot-GLM soak is the most important downtime control. Before it +ends, alias rollback needs only a short proxy restart. After it ends, restoring +the physical GLM backend can require a long cold model load, so the soak may not +be shortened merely to begin DeepSeek work sooner. + +## Acceptance and Abort Rules + +These rules apply across milestones: + +- Never proceed on a partially healthy two-rank engine. Stop or start both + ranks as one operation. +- Any semantic corruption, repeated garbling, engine death, NCCL timeout, + unexplained credential exposure, listener on the wrong interface, or + regression of Caddy's 403/401 policy is an immediate abort. +- A container being `Up` is not health evidence. Require a direct model-list or + completion probe and inspect recent logs for engine failure. +- Do not lower correctness, security, or rollback gates to rescue a performance + result. +- A feature or optimization that fails returns to the last known-good private + profile. It does not force restoration of GLM routing because `glm-5.2` + remains served by GLM-5.3. +- A public registration failure removes only `deepseek-v4.1-flash`; it does not + repoint `deepseek-v4-flash` or alter the GLM aliases. +- A decision to restore the old GLM route after its ranks are stopped requires + stopping DeepSeek first because the deployments share GPUs and port 8000. + +## Artifact and Evidence Record + +Each milestone appends to one UTC-timestamped change record. It must include: + +- operator, start/end time, milestone, go/no-go decision, and reason; +- config backup paths and hashes, excluding secret contents; +- checkpoint repository, exact revision, shard manifest, and total bytes; +- vLLM commit, PR state at pin time, local diff, image digest, Python lock, + CUDA, NCCL, driver, kernel, and firmware versions; +- launch commands with secret values redacted and the fully rendered + non-secret engine configuration; +- listener, firewall, credential-file mode, and rank-isolation checks; +- LiteLLM container identity, bind, restart duration/count, health, callbacks, + routes, and security responses before and after each restart; +- direct and end-to-end probe results plus Langfuse trace identifiers; +- AIPerf version, dataset identity, full command, cache state, raw artifacts, + aggregate metrics, and server telemetry window; +- every failed variant, its logs, and why it was rejected; +- exact rollback command, estimated duration, and the last point at which it + was rehearsed or proven. + +Secrets must never be printed, committed, placed in command-line arguments, +copied to rank 1, or stored in benchmark artifacts. Record only their file +locations, ownership, modes, and non-reversible fingerprints where comparison +is necessary. + +## Rollback Matrix + +| Failure point | Production route | Recovery | +|---|---|---| +| Before alias restart | Unchanged | Discard staged config; no service action | +| Alias restart fails | Intended alias may be unavailable | Restore LiteLLM backup and restart once; station GLM is still hot | +| Alias soak fails | Alias is on GLM-5.3 | Restore LiteLLM backup and restart once; keep station GLM running | +| Private DeepSeek startup or tuning fails | GLM aliases remain on GLM-5.3 | Stop both DeepSeek ranks, capture logs, restore last known-good private profile or pause | +| Need old GLM after station release | GLM alias still on GLM-5.3 until changed | Stop DeepSeek, restore both GLM ranks, verify direct health, restore LiteLLM config, restart once | +| New public DeepSeek route fails | Other models remain unchanged | Restore pre-registration LiteLLM config and restart once; diagnose canary privately | + +No rollback deletes evidence or checkpoints. Cleanup is a separate, explicitly +approved operation after a stable production soak. + +## Documentation and Repeatability Deliverables + +The finished project produces four durable artifacts: + +1. This approved design and its implementation plan. +2. A chronological cutover record containing actual commands, observed + durations, gate results, and rollback decisions. +3. A benchmark bundle with machine-readable AIPerf outputs and a concise + TP2/PP2/feature decision report. +4. A tested model-cutover playbook suitable for conversion into a Codex skill. + +The eventual skill is derived from successful evidence, not from this design's +untested commands. It should separate invariant safety rules from +model-specific parameters, accept an old alias, replacement backend, target +nodes, artifact pins, canary port, soak duration, and benchmark profile as +inputs, and fail closed at every gate. It must support both Claude Code and +Codex or document a deliberate exception, and it must include validation and +safe re-run behavior consistent with the setup repository's style guide. + +## Final Definition of Done + +The project is complete only when: + +- `glm-5.2` is a verified deprecated alias to the current GLM-5.3 backend while + retaining its original 320K client contract; +- the old station GLM ranks are stopped but recoverable; +- DeepSeek V4.1 Flash runs on both stations from pinned artifacts under a + documented, reproducible configuration; +- the selected profile passes API, correctness, security, long-context, + stability, and Weka workload gates; +- `deepseek-v4.1-flash` is independently exposed without changing Baker's + `deepseek-v4-flash`; +- both planned LiteLLM restarts stayed within the recorded change budget or + their rollback paths were exercised successfully; +- the Webster source-of-truth skill and both mirrors reflect verified live + state; and +- the operational notes are sufficient to create and test the repeatable + cutover skill without reconstructing decisions from chat history. diff --git a/install.sh b/install.sh index d89d39c..0fdc23a 100755 --- a/install.sh +++ b/install.sh @@ -136,6 +136,131 @@ add_path() { printf '\nexport PATH="%s:$PATH"\n' "$dir" >> "$profile" } +verify_sha256() { + local file="$1" expected="$2" actual + actual=$(shasum -a 256 "$file" | awk '{print $1}') || return 1 + [ -n "$expected" ] && [ "$actual" = "$expected" ] +} + +# Intel macOS 15 is a Homebrew Tier 3 platform. Use upstream release artifacts +# for Go and fzf instead of compiling through Homebrew's unsupported dependency +# path. Both downloads are checked against checksums published by their owners. +install_go_darwin_archive() { + local metadata version asset expected tmpd install_base install_root + metadata=$(curl -fsSL 'https://go.dev/dl/?mode=json') || return 1 + version=$(printf '%s' "$metadata" | jq -er 'map(select(.stable))[0].version') || return 1 + asset="${version}.darwin-amd64.tar.gz" + expected=$(printf '%s' "$metadata" | jq -er --arg asset "$asset" \ + 'map(select(.stable))[0].files[] | select(.filename == $asset) | .sha256') || return 1 + tmpd=$(mktemp -d) + if ! curl -fsSL "https://go.dev/dl/${asset}" -o "$tmpd/$asset" || + ! verify_sha256 "$tmpd/$asset" "$expected" || + ! tar -C "$tmpd" -xzf "$tmpd/$asset"; then + warn "failed to download or verify Go release $version" + rm -rf "$tmpd" + return 1 + fi + install_base="$HOME/.local/share/setup/go" + install_root="$install_base/$version" + mkdir -p "$install_base" + if [ -e "$install_root" ] && [ ! -x "$install_root/bin/go" ]; then + rm -rf "$install_root" + fi + if [ ! -e "$install_root" ]; then + mv "$tmpd/go" "$install_root" + fi + ln -sfn "$version" "$install_base/current" + rm -rf "$tmpd" + add_path "$install_base/current/bin" +} + +install_fzf_darwin_archive() { + local metadata version asset url checksums_url expected tmpd + metadata=$(curl -fsSL 'https://formulae.brew.sh/api/formula/fzf.json') || return 1 + version=$(printf '%s' "$metadata" | jq -er '.versions.stable') || return 1 + asset="fzf-${version}-darwin_amd64.tar.gz" + url="https://github.com/junegunn/fzf/releases/download/v${version}/${asset}" + checksums_url="https://github.com/junegunn/fzf/releases/download/v${version}/fzf_${version}_checksums.txt" + tmpd=$(mktemp -d) + expected=$(curl -fsSL "$checksums_url" | awk -v asset="$asset" '$2 == asset {print $1; exit}') || true + if [ -z "$expected" ] || + ! curl -fsSL "$url" -o "$tmpd/$asset" || + ! verify_sha256 "$tmpd/$asset" "$expected" || + ! tar -C "$tmpd" -xzf "$tmpd/$asset" fzf; then + warn "failed to download or verify fzf release $version" + rm -rf "$tmpd" + return 1 + fi + mkdir -p "$HOME/.local/bin" + install -m 0755 "$tmpd/fzf" "$HOME/.local/bin/fzf" + rm -rf "$tmpd" + add_path "$HOME/.local/bin" +} + +# tmux does not publish macOS binaries. Build its official release directly +# against existing Homebrew libraries, avoiding Homebrew's forced upgrades of +# unrelated stale dependencies on this unsupported platform. +install_tmux_darwin_source() { + local metadata version url expected tmpd dep prefix pkg_path cppflags ldflags jobs + for dep in libevent ncurses utf8proc jemalloc; do + if ! brew list --versions "$dep" >/dev/null 2>&1; then + brew install "$dep" || true + fi + prefix=$(brew --prefix "$dep" 2>/dev/null) || return 1 + [ -d "$prefix" ] || { warn "Homebrew dependency $dep is unavailable for tmux"; return 1; } + done + if ! have pkg-config; then + brew install pkgconf || brew install --build-from-source pkgconf || return 1 + fi + + metadata=$(curl -fsSL 'https://formulae.brew.sh/api/formula/tmux.json') || return 1 + version=$(printf '%s' "$metadata" | jq -er '.versions.stable') || return 1 + url=$(printf '%s' "$metadata" | jq -er '.urls.stable.url') || return 1 + expected=$(printf '%s' "$metadata" | jq -er '.urls.stable.checksum') || return 1 + tmpd=$(mktemp -d) + if ! curl -fsSL "$url" -o "$tmpd/tmux.tar.gz" || + ! verify_sha256 "$tmpd/tmux.tar.gz" "$expected" || + ! tar -C "$tmpd" -xzf "$tmpd/tmux.tar.gz"; then + warn "failed to download or verify tmux release $version" + rm -rf "$tmpd" + return 1 + fi + + pkg_path="" + cppflags="" + ldflags="" + for dep in libevent ncurses utf8proc jemalloc; do + prefix=$(brew --prefix "$dep") + pkg_path="${pkg_path:+$pkg_path:}$prefix/lib/pkgconfig" + cppflags="${cppflags:+$cppflags }-I$prefix/include" + ldflags="${ldflags:+$ldflags }-L$prefix/lib" + done + jobs=$(sysctl -n hw.logicalcpu 2>/dev/null || printf '2') + mkdir -p "$HOME/.local" + if ! ( + cd "$tmpd/tmux-$version" + env PKG_CONFIG_PATH="$pkg_path" CPPFLAGS="$cppflags" LDFLAGS="$ldflags" \ + ./configure --prefix="$HOME/.local" --enable-sixel --enable-utf8proc + make -j "$jobs" + make install + ); then + warn "failed to build tmux release $version" + rm -rf "$tmpd" + return 1 + fi + rm -rf "$tmpd" + add_path "$HOME/.local/bin" +} + +install_intel_macos_fallback() { + case "$1" in + fzf) install_fzf_darwin_archive ;; + go) install_go_darwin_archive ;; + tmux) install_tmux_darwin_source ;; + *) return 1 ;; + esac +} + # --------------------------------------------------------------------------- # package installs # --------------------------------------------------------------------------- @@ -150,7 +275,19 @@ install_one() { # when an unrelated/post-install hook fails (Homebrew ca-certificates has # done this on Intel CI). Treat the on-disk assertion below as authoritative; # otherwise a healthy tool remains permanently recorded as a failure. - if ! pm_install "$tool"; then warn "package manager reported an error while installing $tool"; fi + if ! pm_install "$tool"; then + # Homebrew's Intel macOS 15 runners are Tier 3 and some formulae no longer + # have bottles there. Use deterministic per-tool fallbacks only when the + # normal install did not create the requested executable. + if ! have "$bin" && [ "$PM" = "brew" ] && [ "$OS" = "darwin" ] && [ "$ARCH" = "x86_64" ]; then + warn "Homebrew binary install failed for $tool; using Intel macOS fallback" + if ! install_intel_macos_fallback "$tool"; then + warn "Intel macOS fallback also failed for $tool" + fi + else + warn "package manager reported an error while installing $tool" + fi + fi assert_installed "$tool" "$bin" "$tool" } diff --git a/test.sh b/test.sh index cc2b6ae..209b8b8 100755 --- a/test.sh +++ b/test.sh @@ -66,6 +66,54 @@ mkdir -p "$scratch/install-one-bin" exit 1 } +# Homebrew no longer publishes Intel macOS bottles for every formula, and its +# unsupported source path can fail on dependency resolution or transient module +# fetches. The three affected tools must use their deterministic vendor/source +# fallbacks when the normal bottle install leaves no executable behind. +mkdir -p "$scratch/intel-fallback-bin" +( + export PATH="$scratch/intel-fallback-bin:$PATH" + export SETUP_TEST_BIN="$scratch/intel-fallback-bin" + OS=darwin + ARCH=x86_64 + PM=brew + FAILED="" + have() { [ -x "$SETUP_TEST_BIN/$1" ]; } + pm_install() { return 1; } + install_fzf_darwin_archive() { + printf '#!/bin/sh\nexit 0\n' > "$SETUP_TEST_BIN/fzf" + chmod +x "$SETUP_TEST_BIN/fzf" + } + install_go_darwin_archive() { + printf '#!/bin/sh\nexit 0\n' > "$SETUP_TEST_BIN/go" + chmod +x "$SETUP_TEST_BIN/go" + } + install_tmux_darwin_source() { + printf '#!/bin/sh\nexit 0\n' > "$SETUP_TEST_BIN/tmux" + chmod +x "$SETUP_TEST_BIN/tmux" + } + install_one fzf >/dev/null 2>&1 + install_one tmux >/dev/null 2>&1 + install_one go >/dev/null 2>&1 + [ -z "$FAILED" ] +) || { + echo "FAIL: install_one did not use the Intel macOS tool fallbacks" >&2 + exit 1 +} + +# Vendor artifacts must be rejected when their published digest does not match. +printf 'verified artifact\n' > "$scratch/checksum-fixture" +checksum=$(shasum -a 256 "$scratch/checksum-fixture" | awk '{print $1}') +if ! verify_sha256 "$scratch/checksum-fixture" "$checksum"; then + echo "FAIL: verify_sha256 rejected a matching artifact digest" >&2 + exit 1 +fi +if verify_sha256 "$scratch/checksum-fixture" \ + '0000000000000000000000000000000000000000000000000000000000000000'; then + echo "FAIL: verify_sha256 accepted a mismatched artifact digest" >&2 + exit 1 +fi + # configure_claude must fail on an unparseable settings file and leave it # untouched rather than clobbering it. mkdir -p "$scratch/home/.claude" @@ -1425,3 +1473,9 @@ if ! printf '%s' "$claude_help" | grep -q 'WEBSTER_API_KEY'; then fi log "all negative tests passed" + +python3 -m unittest discover -s webster/deepseek-v41/tests -p 'test_*.py' +bash webster/deepseek-v41/tests/test_failure_paths.sh +if [ -f webster/deepseek-v41/litellm/test_glm52_contract_guard.py ]; then + python3 webster/deepseek-v41/litellm/test_glm52_contract_guard.py +fi diff --git a/webster/deepseek-v41/.gitignore b/webster/deepseek-v41/.gitignore new file mode 100644 index 0000000..0d00238 --- /dev/null +++ b/webster/deepseek-v41/.gitignore @@ -0,0 +1,12 @@ +manifest.env +*.key +*.env +runs/ +artifacts/ +*.tar +*.tar.zst +models/ +checkpoints/ +.cache/ +__pycache__/ +*.pyc diff --git a/webster/deepseek-v41/README.md b/webster/deepseek-v41/README.md new file mode 100644 index 0000000..46a3db8 --- /dev/null +++ b/webster/deepseek-v41/README.md @@ -0,0 +1,304 @@ +# DeepSeek V4.1 Flash station cutover + +This package records and executes the limited-downtime migration from the +station-hosted GLM-5.2 engine to a separately published DeepSeek V4.1 Flash +service. The public `glm-5.2` name remains a compatibility contract served by +GLM-5.3: a shared 320,000-token prompt-plus-completion window, reasoning and +function calling, no advertised vision, rejection of explicit over-budget output, +and the legacy remaining-budget default when no output limit is supplied. + +## Control sequence + +```text +freeze rollback -> stage additively -> route alias -> restart once -> hot soak +-> stop both GLM ranks -> private canary/tuning -> register new name -> restart once +``` + +There are exactly two planned LiteLLM restarts: the compatibility alias cutover +and the independent `deepseek-v4.1-flash` registration. Each uses a 60-second +readiness decision threshold and a named, hash-verified rollback file. LiteLLM +continues to bind only `127.0.0.1:4446`; Caddy remains the sole ingress. + +The old Shamu/Tilikum GLM ranks stay hot for a 30–60 minute soak after the first +restart. During that interval rollback is a config restore and one proxy restart. +Stopping both GLM ranks crosses the important boundary: rollback then includes a +coordinated cold model load and grows to a 5–15 minute operation (with a 20-minute +timeout for a cold page cache). + +Port `8000` has two mutually exclusive station owners. Shamu is the only HTTP +rank and binds its NetBird address; Tilikum is always headless and keyless. Never +run GLM and DeepSeek concurrently, and never restart only one rank. + +## Commands and rollback index + +The implementation creates these coordinated entry points: + +- `scripts/stop-glm52-tp2.sh` and `scripts/start-glm52-tp2.sh` +- `scripts/stop-deepseek-v41-tp2.sh` and `scripts/start-deepseek-v41-tp2.sh` +- `scripts/restore-litellm-config.sh --phase alias|publish --run-root PATH` +- `scripts/acceptance.sh --run-root PATH` +- `scripts/generate-private-acceptance.py --help` +- `scripts/verify-private-acceptance.py --help` +- `scripts/sync-runbook.sh [--check|--force]` + +Before the alias restart, discard the candidate and take no service action. +During the hot soak, restore the alias backup. After station release, stop both +DeepSeek ranks, cold-start both GLM ranks, validate direct health, then restore +the alias backup. After publication, restore only the pre-publication LiteLLM +backup while leaving the private canary available for diagnosis. + +No cleanup belongs to this change. Checkpoints, immutable images, credentials, +stopped containers, backups, failure logs, and benchmark evidence remain until a +separate cleanup approval. + +## Milestone records + +Append only redacted GO/NO-GO decisions here. Full evidence remains outside Git +under `/home/ubuntu/deepseek-v41-runs/$CHANGE_ID/`. + +- **Milestone 1 — GO (2026-09-11):** evidence root + `/home/ubuntu/deepseek-v41-runs/20260911T075859Z`. The read-only baseline + passed for all eight existing public models with its scoped key revoked. + Exact byte-verified rollback state was frozen with timestamp + `20260911T081659Z` at + `spark-1:/home/nvidia/litellm/deepseek-v41-rollback-20260911T081659Z` and + `shamu|tilikum:/home/alecfong/deepseek-v41-rollback-20260911T081659Z`. + Rank 1 remained headless and keyless; no service, route, container, or model + process changed. +- **Runtime pin gate — NO-GO (2026-09-11):** vLLM PR `#56214` was open, + non-draft, merge-blocked, unapproved, and still had 157 of 246 current commit + statuses pending at head `7d81d62702b41885e2ff3ebc7ad9dfb638cc429c`. + `VLLM_COMMIT` remains empty. No image build, checkpoint download, GPU kernel + compilation, or station workload was started. +- **Runtime package gate — GO (2026-09-12):** PR `#56214` merged and the + selected source is its merge commit + `e77daef89e18e08321ae7b8b24827eedd5fe8673`. The arm64 image + `sha256:3863bf0f59bd4df4012b7b7aed8a7d2be6ef9b42ee3a5299d014f83a1f1ba6ea` + and saved-image checksum match on Shamu and Tilikum. The upstream four-family + architecture build produced a 613.97 MiB wheel, so the private build retains + the upstream size check with a recorded 700 MiB ceiling below its 800 MiB + quota. GPU-blind checks verified `DeepseekV41ForCausalLM`, + `--engram-config`, and nonempty Engram layers. The evidence preserves the + earlier uv-cache, missing-tag, wheel-limit, help-device, and registry-glob + NO-GO failures. No route, listener, checkpoint, or GPU owner changed; GLM + remained live throughout. +- **Milestone 2 — GO (2026-09-12):** the pinned 48-shard checkpoint at revision + `dba1be0a40aa45a94ad051997016db3960a90277` passed two complete hash passes + per station and cross-node byte comparison. Both 9,257-byte manifests have + SHA-256 `aad652a2b601f711599298493229f32fcc7216ca17d6bb0ccd451fd2fdd01fef`; + Shamu retained 1,187,905,433,600 free bytes and Tilikum retained + 2,241,969,221,632. The low-priority rail seed and verifier windows recorded + no production regression. The compatibility guard and original tokenizer + then passed 18 tests inside the exact live LiteLLM image without a restart. +- **Milestone 3 — GO (2026-09-12):** the `glm-5.2` public deployment now uses + the existing GLM-5.3 backend while retaining the original shared 320,000-token + contract and public response name. The first planned LiteLLM restart became + ready in 10.273 seconds with restart count zero and the required loopback-only + command shape. Its exact rollback file is + `spark-1:/home/nvidia/litellm/config.yaml.bak-deepseek-v41-alias-20260912T075404Z`. + Private and public probes covered text, streaming, reasoning, tools, tool + results, structured output, and a pre-backend 320,001-token rejection; the + old station chat counter remained 16,400 while the GLM-5.3 counter advanced. +- **Milestone 3 metadata gate — NO-GO, rolled back, and corrected + (2026-09-12):** the first alias candidate omitted `supports_vision`, so + LiteLLM inferred `supports_vision: true` from the GLM-5.3 backend. The alias + was restored immediately from the exact backup while both station GLM ranks + remained hot. The corrected candidate explicitly sets `supports_vision: + false`; its retry became ready in 9.057 seconds and live `/model/info` now + reports the legacy 320,000-token, reasoning, function-calling, no-vision + contract while native `glm-5.3-flash` retains its 1,048,576-token vision + contract. Operationally, the alias phase has used three proxy restarts: the + initial planned restart, one hot rollback restart, and one corrected retry. + The separate planned publication restart remains outstanding. This history + is intentionally not collapsed into the two-planned-restart target. +- **Milestone 4 — GO (2026-09-12):** after the corrected seven-window hot soak, + both station GLM ranks were stopped together. Post-stop inspection confirmed + both GPUs and port 8000 were free. Private and public alias suites again + passed the full legacy contract, including a pre-backend 320,001-token 400; + all eight pre-existing public models returned 200; private and public ingress + remained deny-by-default; and the scoped cutover key was revoked and returned + 401. The old station counter was frozen at 16,401 before shutdown, while all + post-stop GLM traffic continued through the external GLM-5.3 backend. This is + the cold-rollback boundary: restoring station GLM now requires a coordinated + two-rank model load before the alias rollback can be applied. +- **Milestone 5 eager canary — GO (2026-09-12):** DeepSeek V4.1 Flash is live + privately on Shamu/Tilikum with TP=2, PP=1, eager execution, a 131,072-token + window, one sequence, and 0.90 GPU-memory utilization. Shamu is the sole + authenticated HTTP rank on its NetBird address; Tilikum is headless and has + no serving key. The pinned CX8 path initialized with GPUDirect RDMA, both + containers remained running with zero restarts/OOMs, and vLLM reported a + 6,953,138-token KV pool. The runtime's effective defaults enabled prefix + caching and 16,384 maximum batched tokens even though the initial profile did + not make those values explicit. Three consecutive expanded qualification + runs passed all ten cases: deterministic text, UTF-8, stop sequences, + streaming, usage accounting, reasoning, tool calls, tool-result continuation, + strict structured output, and native vision. Evidence is in + `feature-eager-expanded-{1,2,3}.json` under the run root. DeepSeek reasoning + is enabled by default and can consume the entire completion budget, returning + `content: null`; contract and surface probes must explicitly send + `reasoning_effort: "none"` unless reasoning behavior itself is under test. +- **Milestone 6 graph canary — GO (2026-09-12):** both DeepSeek ranks were + stopped together and relaunched with the `graphs-128k` profile. Relative to + the effective eager configuration, the only runtime change was removing + `--enforce-eager`; prefix caching and the 16,384 maximum-batched-token value + were made explicit. Breakable CUDA graph capture completed in 42 seconds, + both containers remained running with zero restarts, and the KV pool grew to + 8,263,348 tokens (63.04 full 131,072-token requests), up 18.8% from eager. + The correctly labeled `feature-graphs-128k-1.json` artifact passed all ten + qualification cases. The first graph launch events were incorrectly labeled + `eager` by hard-coded lifecycle text; a TDD regression fix now records the + selected profile, and an append-only correction event supersedes those two + labels without rewriting operational history. +- **Milestone 7 DSpark canary — GO (2026-09-12):** the strict profile schema + now renders a checkpoint-native speculative configuration and rejects a + draft block that does not match this pinned checkpoint. The exact resolved + configuration is method `dspark`, model `/model`, five speculative tokens, + greedy draft sampling, and standard rejection sampling. vLLM resolved + `DSparkV41DraftModel` from the target checkpoint and confirmed block size 5, + target layers `[37, 38, 39]`, and Markov rank 256. Both ranks were replaced + together with the `dspark-128k` TP=2/PP=1 profile; DSpark graph capture took + 22 seconds, the KV pool was 8,282,419 tokens, and both containers remained + running with zero restarts/OOMs. `feature-dspark-128k-1.json` passed all ten + surface cases. Live metrics prove the drafter was active: the qualification + workload accepted 103 of 144 draft tokens, not merely a no-op configuration. +- **Milestone 8 initial Weka sweep — partial GO, then NO-GO + (2026-09-12):** `dspark-1m-seqs1` completed a paired warmup and scored run + with zero errors, restarts, or preemptions. The scored run completed 383 + requests at 0.42554 requests/s and 189.95 output tokens/s, with 2.32-second + p90 TTFT and three completed root workflows in 900.03 seconds. The next + profile, `dspark-1m-seqs2`, failed before readiness during FlashInfer's + 16,384-token autotune dummy run with `Invalid MXFP8 split-K tactic`; rank 0 + exited without OOM while rank 1 remained resident, so both ranks were fenced + together. A new cache hash with no saved cache file rules out stale-cache + replay, and NCCL had initialized successfully. The historical profiles, + error excerpt, container state, and NO-GO event are preserved. A later + paired stop exposed that fixed `*-prestop.log` names could overwrite prior + evidence; the original full failure logs were replaced before that flaw was + caught. `capture` now allocates a numbered path when a name already exists, + with a regression test proving repeated captures retain both files. The + supported workaround is now a strict + `ENABLE_FLASHINFER_AUTOTUNE=0` profile field that renders + `--no-enable-flashinfer-autotune`. Fair replacement profiles use this same + setting for `max-num-seqs` 1, 2, 4, and 8; mixed autotune results will not be + compared. +- **Milestone 8 completed — GO (2026-09-12):** the fair no-autotune Weka sweep + completed paired warmup/scored runs at `max-num-seqs` 1, 2, 4, and 8 with + zero request errors, preemptions, engine deaths, OOMs, or restarts. Scored + request throughput was 0.42400, 0.43550, 0.43986, and 0.44377 requests/s; + output throughput was 189.30, 199.80, 202.03, and 207.01 tokens/s; p90 TTFT + was 2.211, 1.181, 1.068, and 1.037 seconds. `seqs=8` gained only 0.89% + request throughput over `seqs=4`, while observed concurrency peaked at three, + so `dspark-1m-noautotune-seqs4` won on KV headroom. The final clean winner + passed 10/10 feature cases, the exact 1,048,576-token boundary suite, and the + mode-0600 acceptance manifest. The scored run averaged 1,477.41 W. Published + energy-only prices use `1.3e-8` input from a 15,047-token/s cold-prefill + observation and `9.6e-7` output from measured Weka throughput; the 99.866% + cache-hit aggregate input rate was explicitly rejected as pricing evidence. +- **Coordinated recovery — GO (2026-09-12):** the head-hosted systemd watchdog + is enabled every minute. A healthy run preserved both container identities. + A real paired-stop fault produced `BREACH`, fenced both ranks, relaunched the + exact winning profile, and returned authenticated readiness in 5m09s with new + identities and zero restarts, followed by another healthy no-op. Tilikum's + fixed ten-second sudo delay was independently traced to missing hostname + resolution and removed by adding `127.0.1.1 tilikum`; the prior hosts file is + preserved as `/etc/hosts.bak-deepseek-v41-20260912T163400Z`. +- **Milestone 9 publication — GO (2026-09-12):** the exact pre-publication + LiteLLM config was preserved at + `spark-1:/home/nvidia/litellm/config.yaml.bak-deepseek-v41-publish-20260912T165510Z` + with SHA-256 + `84a0dea58c561f991c3870bcc6eb5bbd8e3d47aee4be59b457e40dcc17f0ee30`. + The candidate appended only `deepseek-v4.1-flash`, advertising the native + 1,048,576-token vision/reasoning/tools/structured-output contract. Semantic + validation and exact-image offline callback/router construction passed before + one atomic promotion and one proxy restart. Liveliness/readiness returned in + 19.616 seconds; the container remained at zero restart failures and bound only + to `127.0.0.1:4446`. The automated acceptance entry point passed direct auth + rejection/success, all nine private models, five public controls, legacy + `glm-5.2` metadata, native DeepSeek metadata, Caddy 403/401 policy, revoked-key + rejection, 11/11 Prometheus targets, a fresh Langfuse trace, active watchdog, + rank isolation, and the hash-verified rollback path. GLM station containers + and all Baker containers remain untouched and preserved. +- **Post-publication review hardening — GO (2026-09-12):** publication preflight + now validates the complete mode-0600 acceptance manifest, artifact hashes, + six-hour freshness, evidence ordering, exact runtime/profile/topology, feature + and long-context results, qualified Weka metrics, rank identities, and PP=2 + rationale. The direct feature suite expanded from 10 to 17 cases, adding + three-repeat determinism, named/required/parallel tools, `json_object`, malformed + input, wrong-model rejection, disconnect queue drain, and automatic redacted + two-rank log capture on failure. All 17 cases passed live without a restart. + Fresh rank/API evidence was captured before the generated acceptance timestamp + `2026-09-12T18:43:41Z`; the generator and an independent live publish preflight + both accepted it. AIPerf now rejects missing, malformed, cancelled, partial, or + unqualified results and includes Shamu vLLM metrics. Serving fast paths verify + the exact paired generation and profile before reuse. Skill synchronization now + requires explicit `--force` for divergent regular directories and verifies exact + rendered Hermes content. These checks introduced no model or LiteLLM downtime. +- **Recovery-package omission — NO-GO, recovered, and corrected + (2026-09-12):** the first deployed watchdog bundle omitted the newly required + `verify-serving-pair.py` dependency. Its fail-closed immutable-profile check + consequently reported `BREACH` against a healthy pair and performed a coordinated + fence/reload. The model was avoidably unavailable from approximately `23:07:19Z` + until authenticated readiness returned at `23:12:48Z`; the public LiteLLM service + and compatibility-routed `glm-5.2` remained available. The installed bundle now + contains the verifier and its subsequent healthy paths preserve both container + identities. Recovery packaging is therefore treated as a transitive dependency + closure: stage and hash every sourced helper and executable, then exercise a + read-only path from the installed directory before enabling the timer. +- **Hardened full-subagent Weka qualification — GO (2026-09-13):** the 393-trace + corpus expands to a 5.70 GiB mmap, 9,843 conversations, and 98,827 turns, with + roughly 12 GiB required during reconstruction. Head's 8 GiB of physical RAM plus + swap failed before sending model traffic at 300-, 900-, and 1,200-second dataset + configuration limits. Those three directories retain their exact `EXIT=1` files + (all SHA-256 + `4bf4fca02e6eeaf7f1c510b82c756e7f8a3c567a230bae1094b254adc774f38b`); + failed evidence is never rewritten. Moving the controller to Shamu and preserving + the mmap cache reduced configuration to 2.46 seconds. The genuine warmup completed + 682 requests and three root workflows at 0.74757 requests/s and 288.50 output + tokens/s, with 11.546-second p90 TTFT, 20 peak waiting requests, zero errors, + cancellations, preemptions, or restarts, and 1,955.64 W average facility power. + The genuine scored repetition completed 974 requests and five root workflows at + 0.89461 requests/s and 382.40 output tokens/s, with 10.542-second p90 TTFT, 28 + peak request concurrency (four running, 24 waiting), 4.84% peak KV use, 97.61% + prefix-cache hits, zero errors, cancellations, preemptions, or restarts, and + 1,812.61 W average facility power. A comma-grouped completion count had exposed a + summarizer defect; the parser received a regression fix, but acceptance required + these new runs instead of repairing the old summary. Repetition 2 is promoted only + because its rank states match repetition 1 and its provenance names the exact + warmup hash. +- **Final evidence refresh — GO (2026-09-13):** pricing was recomputed only after + the hardened scored run, using its measured power/output throughput and the final + 319,999-token cold prefill in 20.996604 seconds. The unrounded direct energy values + are `1.552747272598342e-8` input and `6.188455008963091e-7` output dollars/token; + published prices remain `1.3e-8` and `9.6e-7`. Fresh generation-bound rank and API + captures then preceded the mode-0600 acceptance timestamp + `2026-09-13T01:01:27Z`. Independent publication preflight and the complete + acceptance matrix passed again against generation + `20260912T230747Z-3280615`, with LiteLLM and both model ranks unrestarted. +- **Legacy-contract correction and final publication acceptance — GO + (2026-09-13):** an exact image request proved the first live guard returned HTTP + 400 but exposed top-level error code `"400"`; `unsupported_vision` survived only + in nested provider fields. The guard now raises a LiteLLM-native proxy exception + whose serialized top-level code is `unsupported_vision`. Twenty exact-image guard + tests and 23 focused local tests passed before one corrective LiteLLM restart. + Readiness returned in **9.122 seconds**, restart count remained zero, the config + SHA-256 remained + `f8a9ddf5d1634015d6a9b167756cc3c56dfa19cae852fb4896191dbfe5182c30`, + and the deployed guard SHA-256 became + `2557630b10890c74d61ef3f7bc8d58558e894b04e05c9ed9847b9bd5d07a7dfe`. + Publication probes now derive the current registry and require only the four + migration-critical aliases. The complete live acceptance entry point returned + `GO` for run root `/home/ubuntu/deepseek-v41-runs/20260911T075859Z`. +- **Retired Inkling reconciliation — GO (2026-09-13):** the live and on-disk + LiteLLM registries already omitted `inkling-small-nvfp4`, both Spark nodes had + zero Inkling containers, and the only Spark-2 port-8000 listener belonged to the + unrelated Reachy Mini simulator. The stale `inkling-watchdog.timer` was disabled + and stopped so it can no longer recreate the retired model. OpenWebUI's existing + `openwebui-shamu-5` key now removes Inkling and adds + `deepseek-v4.1-flash`; an authenticated completion through that key returned 200 + without an OpenWebUI or LiteLLM restart. Prometheus's dead Inkling `:9401` target + was removed and the Prometheus-only recreate produced **643 ms** of observed + metrics unavailability; all **10/10 Prometheus targets** are now up. The prior + Prometheus config is recoverable at + `/home/alecfong/observability/prometheus/prometheus.yml.bak-inkling-retirement-20260913T025930Z`. + A fresh complete acceptance run returned `GO` after the reconciliation. Neither + DeepSeek rank changed identity or restart count, and Baker remained untouched. diff --git a/webster/deepseek-v41/litellm/glm52_contract_guard.py b/webster/deepseek-v41/litellm/glm52_contract_guard.py new file mode 100644 index 0000000..ce64681 --- /dev/null +++ b/webster/deepseek-v41/litellm/glm52_contract_guard.py @@ -0,0 +1,381 @@ +"""Preserve the original GLM-5.2 shared 320K token contract. + +The public ``glm-5.2`` name may route to a backend with a larger context window. This +pre-call hook renders requests with the original tokenizer/template, rejects prompts +whose prompt-plus-output budget exceeds the old limit, and supplies the positive +remaining budget when no output field was requested. +All other public model names bypass the guard without initialization or mutation. +""" + +from __future__ import annotations + +import asyncio +import copy +import inspect +import json +import threading +from pathlib import Path +from typing import Any, Callable + +try: + from litellm.proxy._types import ProxyException +except ImportError: # pragma: no cover - exercised by the dependency-light local suite + class ProxyException(Exception): + def __init__( + self, + message: str, + type: str, + param: str | None, + code: int | str, + openai_code: str | None = None, + ) -> None: + self.message = message + self.type = type + self.param = param + self.code = str(code) + self.openai_code = openai_code or str(code) + super().__init__(message) + + def to_dict(self) -> dict[str, Any]: + return { + "message": self.message, + "type": self.type, + "param": self.param, + "code": self.code, + } + + +try: + from litellm.integrations.custom_logger import CustomLogger +except ImportError: # pragma: no cover - exact-image tests require the real base class + class CustomLogger: + pass + + +PUBLIC_MODEL = "glm-5.2" +LEGACY_CONTEXT_TOKENS = 320_000 +DEFAULT_TOKENIZER_DIR = Path("/app/custom_callbacks/glm52-tokenizer") +VISION_CONTENT_TYPES = frozenset({"image", "image_url", "input_image"}) + + +class GLM52ContractError(ProxyException): + """Keep the OpenAI error code distinct from LiteLLM's HTTP status field.""" + + def __init__(self, code: str, message: str, param: str | None) -> None: + self.status_code = 400 + self.detail = { + "error": { + "message": message, + "type": "invalid_request_error", + "param": param, + "code": code, + } + } + super().__init__( + message=message, + type="invalid_request_error", + param=param, + code=self.status_code, + openai_code=code, + ) + + def to_dict(self) -> dict[str, Any]: + payload = super().to_dict() + payload["code"] = self.openai_code + return payload + + +def _plain(value: Any) -> Any: + if isinstance(value, dict): + return {key: _plain(item) for key, item in value.items()} + if isinstance(value, (list, tuple)): + return [_plain(item) for item in value] + model_dump = getattr(value, "model_dump", None) + if callable(model_dump): + return _plain(model_dump(exclude_none=True)) + return value + + +class LegacyGLM52Renderer: + """Thread-safe-after-construction tokenizer and immutable Jinja renderer.""" + + REQUIRED_FILES = ("tokenizer.json", "tokenizer_config.json", "chat_template.jinja") + + def __init__(self, tokenizer_dir: Path): + root = tokenizer_dir.resolve(strict=True) + if not root.is_dir(): + raise RuntimeError(f"legacy tokenizer path is not a directory: {root}") + for name in self.REQUIRED_FILES: + path = root / name + if not path.is_file() or path.is_symlink(): + raise RuntimeError(f"missing regular legacy tokenizer file: {name}") + + try: + from jinja2 import ext + from jinja2.sandbox import ImmutableSandboxedEnvironment + from tokenizers import Tokenizer + except ImportError as error: + raise RuntimeError(f"legacy tokenizer dependency unavailable: {error}") from error + + self._tokenizer = Tokenizer.from_file(str(root / "tokenizer.json")) + template_text = (root / "chat_template.jinja").read_text(encoding="utf-8") + + def tojson( + value: Any, + ensure_ascii: bool = False, + indent: int | None = None, + separators: tuple[str, str] | None = None, + sort_keys: bool = False, + ) -> str: + return json.dumps( + value, + ensure_ascii=ensure_ascii, + indent=indent, + separators=separators, + sort_keys=sort_keys, + ) + + environment = ImmutableSandboxedEnvironment( + trim_blocks=True, + lstrip_blocks=True, + extensions=[ext.loopcontrols], + ) + environment.filters["tojson"] = tojson + self._template = environment.from_string(template_text) + + @staticmethod + def _normalize_messages(messages: Any) -> list[dict[str, Any]]: + if not isinstance(messages, list): + raise RuntimeError("messages must be a list") + normalized = _plain(copy.deepcopy(messages)) + for message in normalized: + if not isinstance(message, dict) or not isinstance(message.get("role"), str): + raise RuntimeError("each message must contain a role") + for tool_call in message.get("tool_calls") or []: + if not isinstance(tool_call, dict): + raise RuntimeError("tool call must be an object") + function = tool_call.get("function", tool_call) + if not isinstance(function, dict): + raise RuntimeError("tool-call function must be an object") + arguments = function.get("arguments") + if isinstance(arguments, str): + try: + parsed = json.loads(arguments) + except json.JSONDecodeError as error: + raise RuntimeError("tool-call arguments are not valid JSON") from error + if not isinstance(parsed, dict): + raise RuntimeError("tool-call arguments must decode to an object") + function["arguments"] = parsed + return normalized + + @staticmethod + def _template_kwargs(data: dict[str, Any]) -> dict[str, Any]: + user_kwargs = data.get("chat_template_kwargs") or {} + if not isinstance(user_kwargs, dict): + raise RuntimeError("chat_template_kwargs must be an object") + result = copy.deepcopy(user_kwargs) + extra_kwargs = { + "add_generation_prompt": data.get("add_generation_prompt", True), + "continue_final_message": data.get("continue_final_message"), + "documents": data.get("documents"), + "reasoning_effort": data.get("reasoning_effort"), + } + if ( + data.get("reasoning_effort") is not None + and "enable_thinking" not in user_kwargs + ): + extra_kwargs["enable_thinking"] = data["reasoning_effort"] != "none" + for key, value in extra_kwargs.items(): + if value not in (None, "auto"): + result[key] = value + result["messages"] = LegacyGLM52Renderer._normalize_messages( + data.get("messages") + ) + tools = _plain(copy.deepcopy(data.get("tools"))) + result["tools"] = tools + result["add_generation_prompt"] = bool(result.get("add_generation_prompt", True)) + return result + + def token_ids(self, data: dict[str, Any]) -> list[int]: + rendered = self._template.render(**self._template_kwargs(data)) + return self._tokenizer.encode(rendered, add_special_tokens=False).ids + + def count(self, data: dict[str, Any]) -> int: + return len(self.token_ids(data)) + + +class GLM52ContractGuard(CustomLogger): + def __init__( + self, + *, + counter: Callable[[dict[str, Any]], int] | None = None, + tokenizer_dir: Path = DEFAULT_TOKENIZER_DIR, + renderer_factory: Callable[[Path], Any] = LegacyGLM52Renderer, + responses_normalizer: Callable[[dict[str, Any]], list[Any]] | None = None, + ) -> None: + try: + super().__init__() + except TypeError: + # Older LiteLLM CustomLogger releases did not require initialization. + pass + self._counter = counter + self._tokenizer_dir = Path(tokenizer_dir) + self._renderer_factory = renderer_factory + self._responses_normalizer = responses_normalizer + self._renderer: Any | None = None + self._renderer_lock = threading.Lock() + + @staticmethod + def _error( + code: str, message: str, param: str | None = None + ) -> GLM52ContractError: + return GLM52ContractError(code, message, param) + + def _get_renderer(self) -> Any: + if self._renderer is None: + with self._renderer_lock: + if self._renderer is None: + self._renderer = self._renderer_factory(self._tokenizer_dir) + return self._renderer + + def _count_prompt(self, data: dict[str, Any]) -> int: + return self._get_renderer().count(data) + + def _normalize_responses(self, data: dict[str, Any]) -> list[Any]: + if self._responses_normalizer is not None: + return self._responses_normalizer(data) + try: + from litellm.responses.litellm_completion_transformation.transformation import ( + LiteLLMCompletionResponsesConfig, + ) + except ImportError as error: + raise RuntimeError("LiteLLM Responses normalizer is unavailable") from error + return LiteLLMCompletionResponsesConfig.transform_responses_api_input_to_messages( + input=data.get("input", ""), + responses_api_request=data, + ) + + def _count_data(self, data: dict[str, Any], call_type: str) -> dict[str, Any]: + normalized = copy.deepcopy(data) + if call_type in ("responses", "aresponses") or ( + "input" in normalized and "messages" not in normalized + ): + normalized["messages"] = _plain(self._normalize_responses(normalized)) + normalized.pop("input", None) + return normalized + + @staticmethod + def _contains_vision(messages: Any) -> bool: + if not isinstance(messages, list): + return False + for message in messages: + if not isinstance(message, dict): + continue + content = message.get("content") + if not isinstance(content, list): + continue + for part in content: + part = _plain(part) + if not isinstance(part, dict): + continue + if part.get("type") in VISION_CONTENT_TYPES: + return True + if "image_url" in part: + return True + return False + + @staticmethod + def _output_field(data: dict[str, Any], call_type: str) -> str: + if call_type in ("responses", "aresponses") or ( + "input" in data and "messages" not in data + ): + return "max_output_tokens" + if "max_completion_tokens" in data: + return "max_completion_tokens" + return "max_tokens" + + async def async_pre_call_hook( + self, + user_api_key_dict: Any, + cache: Any, + data: dict[str, Any], + call_type: str, + ) -> dict[str, Any]: + del user_api_key_dict, cache + if data.get("model") != PUBLIC_MODEL: + return data + + try: + count_data = self._count_data(data, call_type) + if self._contains_vision(count_data.get("messages")): + raise self._error( + "unsupported_vision", + "The legacy glm-5.2 contract does not support image input.", + "messages" if "messages" in data else "input", + ) + if self._counter is None: + prompt_tokens = await asyncio.to_thread(self._count_prompt, count_data) + else: + prompt_tokens = self._counter(count_data) + if inspect.isawaitable(prompt_tokens): + prompt_tokens = await prompt_tokens + if isinstance(prompt_tokens, bool) or not isinstance(prompt_tokens, int): + raise RuntimeError("prompt counter did not return an integer") + if prompt_tokens < 0: + raise RuntimeError("prompt counter returned a negative value") + except GLM52ContractError: + raise + except Exception as error: + raise self._error( + "contract_tokenization_failed", + "The legacy glm-5.2 token contract could not be evaluated.", + "messages" if "messages" in data else "input", + ) from error + + if prompt_tokens > LEGACY_CONTEXT_TOKENS: + raise self._error( + "context_length_exceeded", + ( + f"This model's maximum context length is {LEGACY_CONTEXT_TOKENS} " + f"tokens. The rendered prompt has {prompt_tokens} tokens." + ), + "messages" if "messages" in data else "input", + ) + + remaining = LEGACY_CONTEXT_TOKENS - prompt_tokens + output_field = self._output_field(data, call_type) + if output_field in data: + requested = data[output_field] + if isinstance(requested, bool) or not isinstance(requested, int) or requested <= 0: + raise self._error( + "invalid_request_error", + f"{output_field} must be a positive integer.", + output_field, + ) + if requested > remaining: + raise self._error( + "context_length_exceeded", + ( + f"This model's maximum context length is " + f"{LEGACY_CONTEXT_TOKENS} tokens. The rendered prompt has " + f"{prompt_tokens} tokens and {requested} output tokens were " + "requested." + ), + output_field, + ) + else: + if remaining <= 0: + raise self._error( + "context_length_exceeded", + ( + f"This model's maximum context length is " + f"{LEGACY_CONTEXT_TOKENS} tokens. The rendered prompt has " + f"{prompt_tokens} tokens, leaving no room for output." + ), + "messages" if "messages" in data else "input", + ) + data[output_field] = remaining + return data + + +glm52_contract_guard = GLM52ContractGuard() diff --git a/webster/deepseek-v41/litellm/test_glm52_contract_guard.py b/webster/deepseek-v41/litellm/test_glm52_contract_guard.py new file mode 100644 index 0000000..46134df --- /dev/null +++ b/webster/deepseek-v41/litellm/test_glm52_contract_guard.py @@ -0,0 +1,284 @@ +#!/usr/bin/env python3 +"""Behavior tests for the GLM-5.2 compatibility contract guard.""" + +from __future__ import annotations + +import asyncio +import copy +import importlib.util +import json +import os +import tempfile +import threading +import unittest +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path + + +HERE = Path(__file__).resolve().parent +MODULE_PATH = HERE / "glm52_contract_guard.py" +spec = importlib.util.spec_from_file_location("glm52_contract_guard", MODULE_PATH) +if spec is None or spec.loader is None: + raise RuntimeError("cannot load GLM-5.2 guard module") +module = importlib.util.module_from_spec(spec) +spec.loader.exec_module(module) +GLM52ContractGuard = module.GLM52ContractGuard +GLM52ContractError = module.GLM52ContractError +LegacyGLM52Renderer = module.LegacyGLM52Renderer + + +def run_guard(data: dict, *, count: int = 10, call_type: str = "completion") -> dict: + guard = GLM52ContractGuard(counter=lambda _: count) + return asyncio.run(guard.async_pre_call_hook(None, None, data, call_type)) + + +class GLM52ContractGuardTests(unittest.TestCase): + def assert_error_code(self, error: GLM52ContractError, code: str) -> None: + self.assertEqual(error.status_code, 400) + self.assertEqual(error.detail["error"]["code"], code) + + def test_prompt_over_limit_is_rejected_before_routing(self) -> None: + guard = GLM52ContractGuard(counter=lambda _: 320_001) + with self.assertRaises(GLM52ContractError) as caught: + asyncio.run( + guard.async_pre_call_hook( + None, + None, + {"model": "glm-5.2", "messages": []}, + "completion", + ) + ) + self.assert_error_code(caught.exception, "context_length_exceeded") + + def test_legacy_alias_rejects_image_content_before_routing(self) -> None: + data = { + "model": "glm-5.2", + "messages": [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Describe this image."}, + { + "type": "image_url", + "image_url": { + "url": "data:image/png;base64,iVBORw0KGgo=" + }, + }, + ], + } + ], + "max_tokens": 16, + } + with self.assertRaises(GLM52ContractError) as caught: + run_guard(data) + self.assert_error_code(caught.exception, "unsupported_vision") + self.assertEqual(caught.exception.detail["error"]["param"], "messages") + + def test_contract_error_serializes_public_code_separately_from_http_status(self) -> None: + error = GLM52ContractGuard._error( + "unsupported_vision", + "The legacy glm-5.2 contract does not support image input.", + "messages", + ) + + self.assertEqual(error.status_code, 400) + self.assertEqual(getattr(error, "code", None), "400") + self.assertEqual(error.to_dict()["code"], "unsupported_vision") + + def test_explicit_output_over_remaining_budget_is_rejected_like_old_vllm(self) -> None: + data = {"model": "glm-5.2", "messages": [], "max_tokens": 500} + with self.assertRaises(GLM52ContractError) as caught: + run_guard(data, count=319_900) + self.assert_error_code(caught.exception, "context_length_exceeded") + self.assertEqual(data["max_tokens"], 500) + + def test_explicit_output_at_remaining_budget_is_preserved(self) -> None: + data = {"model": "glm-5.2", "messages": [], "max_tokens": 100} + result = run_guard(data, count=319_900) + self.assertEqual(result["max_tokens"], 100) + + def test_max_completion_tokens_takes_precedence(self) -> None: + data = { + "model": "glm-5.2", + "messages": [], + "max_tokens": 75, + "max_completion_tokens": 100, + } + result = run_guard(data, count=319_900) + self.assertEqual(result["max_completion_tokens"], 100) + self.assertEqual(result["max_tokens"], 75) + + def test_responses_output_over_remaining_budget_is_rejected(self) -> None: + normalized = [{"role": "user", "content": "hello"}] + guard = GLM52ContractGuard( + counter=lambda data: 319_900, + responses_normalizer=lambda data: normalized, + ) + data = {"model": "glm-5.2", "input": "hello", "max_output_tokens": 500} + with self.assertRaises(GLM52ContractError) as caught: + asyncio.run(guard.async_pre_call_hook(None, None, data, "responses")) + self.assert_error_code(caught.exception, "context_length_exceeded") + + def test_absent_output_limit_uses_remaining_budget(self) -> None: + result = run_guard({"model": "glm-5.2", "messages": []}, count=319_900) + self.assertEqual(result["max_tokens"], 100) + + def test_exact_limit_prompt_is_rejected_when_no_output_token_remains(self) -> None: + with self.assertRaises(GLM52ContractError) as caught: + run_guard({"model": "glm-5.2", "messages": []}, count=320_000) + self.assert_error_code(caught.exception, "context_length_exceeded") + + def test_native_glm53_is_untouched_and_not_counted(self) -> None: + calls = [] + guard = GLM52ContractGuard(counter=lambda data: calls.append(data) or 1) + data = {"model": "glm-5.3-flash", "messages": [], "max_tokens": 500_000} + original = copy.deepcopy(data) + result = asyncio.run( + guard.async_pre_call_hook(None, None, data, "completion") + ) + self.assertEqual(result, original) + self.assertIs(result, data) + self.assertEqual(calls, []) + + def test_exported_guard_is_a_custom_logger(self) -> None: + self.assertIsInstance(module.glm52_contract_guard, module.CustomLogger) + + def test_other_models_are_untouched(self) -> None: + data = {"model": "inkling-small-nvfp4", "messages": [{"role": "user", "content": "hi"}]} + original = copy.deepcopy(data) + self.assertEqual(run_guard(data), original) + + def test_invalid_output_limits_are_rejected(self) -> None: + for value in (True, False, 0, -1, 1.5, "100", None): + with self.subTest(value=value): + data = {"model": "glm-5.2", "messages": [], "max_tokens": value} + with self.assertRaises(GLM52ContractError) as caught: + run_guard(data) + self.assert_error_code(caught.exception, "invalid_request_error") + + def test_counter_failure_fails_closed_only_for_alias(self) -> None: + def fail(_: dict) -> int: + raise RuntimeError("renderer unavailable") + + guard = GLM52ContractGuard(counter=fail) + with self.assertRaises(GLM52ContractError) as caught: + asyncio.run( + guard.async_pre_call_hook( + None, + None, + {"model": "glm-5.2", "messages": []}, + "completion", + ) + ) + self.assert_error_code(caught.exception, "contract_tokenization_failed") + control = {"model": "glm-5.3-flash", "messages": []} + self.assertIs( + asyncio.run(guard.async_pre_call_hook(None, None, control, "completion")), + control, + ) + + def test_responses_input_is_normalized_before_counting(self) -> None: + observed = [] + normalized = [{"role": "user", "content": "hello"}] + guard = GLM52ContractGuard( + counter=lambda data: observed.append(copy.deepcopy(data)) or 10, + responses_normalizer=lambda data: normalized, + ) + data = {"model": "glm-5.2", "input": "hello"} + asyncio.run(guard.async_pre_call_hook(None, None, data, "aresponses")) + self.assertEqual(observed[0]["messages"], normalized) + self.assertNotIn("input", observed[0]) + self.assertEqual(data["input"], "hello") + + @unittest.skipUnless( + module.CustomLogger.__module__.startswith("litellm."), + "requires the installed LiteLLM Responses converter", + ) + def test_installed_litellm_responses_converter_produces_chat_messages(self) -> None: + data = {"model": "glm-5.2", "input": "hello", "max_output_tokens": 32} + normalized = module.glm52_contract_guard._count_data(data, "responses") + self.assertEqual( + normalized["messages"], + [{"role": "user", "content": "hello"}], + ) + self.assertNotIn("input", normalized) + + def test_lazy_renderer_initializes_once_under_threads(self) -> None: + created = 0 + created_lock = threading.Lock() + + class FakeRenderer: + def count(self, _: dict) -> int: + return 1 + + def factory(_: Path) -> FakeRenderer: + nonlocal created + with created_lock: + created += 1 + return FakeRenderer() + + guard = GLM52ContractGuard(renderer_factory=factory) + with ThreadPoolExecutor(max_workers=8) as executor: + counts = list(executor.map(lambda _: guard._count_prompt({}), range(32))) + self.assertEqual(counts, [1] * 32) + self.assertEqual(created, 1) + + def test_missing_tokenizer_files_fail_closed(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + guard = GLM52ContractGuard(tokenizer_dir=Path(temporary)) + with self.assertRaises(GLM52ContractError) as caught: + asyncio.run( + guard.async_pre_call_hook( + None, + None, + {"model": "glm-5.2", "messages": []}, + "completion", + ) + ) + self.assert_error_code(caught.exception, "contract_tokenization_failed") + + def test_fixture_shapes_reach_the_counter_unchanged(self) -> None: + fixtures = Path(__file__).resolve().parents[1] / "tests" / "fixtures" + for name in ( + "chat.json", + "reasoning-none.json", + "tools.json", + "tool-result.json", + "structured.json", + ): + with self.subTest(name=name): + payload = json.loads((fixtures / name).read_text(encoding="utf-8")) + observed = [] + guard = GLM52ContractGuard( + counter=lambda data: observed.append(copy.deepcopy(data)) or 10 + ) + asyncio.run( + guard.async_pre_call_hook(None, None, payload, "completion") + ) + self.assertEqual(observed[0]["messages"], payload["messages"]) + self.assertEqual(observed[0].get("tools"), payload.get("tools")) + self.assertEqual( + observed[0].get("chat_template_kwargs"), + payload.get("chat_template_kwargs"), + ) + + @unittest.skipUnless( + os.environ.get("GLM52_TOKENIZER_DIR"), + "set GLM52_TOKENIZER_DIR to run the legacy tokenizer golden test", + ) + def test_renderer_matches_old_vllm_tokenize_for_short_fixtures(self) -> None: + fixtures = Path(__file__).resolve().parents[1] / "tests" / "fixtures" + goldens = json.loads( + (fixtures / "tokenize-golden.json").read_text(encoding="utf-8") + ) + renderer = LegacyGLM52Renderer(Path(os.environ["GLM52_TOKENIZER_DIR"])) + for name, expected in goldens.items(): + with self.subTest(name=name): + payload = json.loads((fixtures / name).read_text(encoding="utf-8")) + actual = renderer.token_ids(payload) + self.assertEqual(actual, expected["token_ids"]) + self.assertEqual(len(actual), expected["count"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/webster/deepseek-v41/manifest.env.example b/webster/deepseek-v41/manifest.env.example new file mode 100644 index 0000000..2d293b7 --- /dev/null +++ b/webster/deepseek-v41/manifest.env.example @@ -0,0 +1,22 @@ +# Fixed checkpoint identity. This file documents fields only and is never sourced +# as a live manifest. pin-runtime.py and run-weka.sh --pin populate execution-time +# values in the mode-0600 manifest.env under a run root. +CHECKPOINT_REPO=deepseek-ai/DeepSeek-V4.1-Flash +CHECKPOINT_REVISION=dba1be0a40aa45a94ad051997016db3960a90277 +CHECKPOINT_SHARDS=48 +CHECKPOINT_TENSOR_BYTES=510286023000 +CHECKPOINT_SHARD_FILE_BYTES=510296708312 + +# Required execution-time pins. Preflight rejects empty values in phases that use +# them; floating branches, tags, and dataset aliases are not accepted. +VLLM_COMMIT= +VLLM_BASE_IMAGE_DIGEST= +VLLM_BUILD_BASE_IMAGE_DIGEST= +VLLM_FINAL_BASE_IMAGE_DIGEST= +VLLM_SOURCE_ARCHIVE_SHA256= +VLLM_MAX_WHEEL_SIZE_MB=700 +VLLM_IMAGE_ID= +VLLM_IMAGE_TAR_SHA256= +AIPERF_COMMIT= +WEKA_REPOSITORY= +WEKA_REVISION= diff --git a/webster/deepseek-v41/runtime/Dockerfile b/webster/deepseek-v41/runtime/Dockerfile new file mode 100644 index 0000000..9e2a2bf --- /dev/null +++ b/webster/deepseek-v41/runtime/Dockerfile @@ -0,0 +1,11 @@ +# syntax=docker/dockerfile:1 + +ARG UPSTREAM_VLLM_IMAGE +FROM ${UPSTREAM_VLLM_IMAGE} + +ARG VLLM_COMMIT +ARG VLLM_BUILD_BASE_IMAGE_DIGEST +ARG VLLM_FINAL_BASE_IMAGE_DIGEST +LABEL ai.webster.vllm.commit="${VLLM_COMMIT}" \ + ai.webster.vllm.build-base-digest="${VLLM_BUILD_BASE_IMAGE_DIGEST}" \ + ai.webster.vllm.final-base-digest="${VLLM_FINAL_BASE_IMAGE_DIGEST}" diff --git a/webster/deepseek-v41/runtime/README.md b/webster/deepseek-v41/runtime/README.md new file mode 100644 index 0000000..8a86d77 --- /dev/null +++ b/webster/deepseek-v41/runtime/README.md @@ -0,0 +1,67 @@ +# DeepSeek V4.1 vLLM runtime + +The runtime is built from vLLM merge commit +`e77daef89e18e08321ae7b8b24827eedd5fe8673`, selected by the fail-closed PR +gate for [vLLM #56214](https://github.com/vllm-project/vllm/pull/56214). + +The default upstream Dockerfile build image is amd64-only. The selected commit's +own arm64 CI and release lanes use +`pytorch/manylinuxaarch64-builder:cuda13.0-b8b5f17a7d9ccfc25bbc5cf17b3fcea12964a042` +and compile architecture families `9.0 10.0 11.0 12.0`, explicitly including +DGX Spark/GB10 and the Grace/Blackwell targets. This package follows that lane, +with the arm64 platform manifests pinned directly: + +- build base: + `sha256:994bed2b225a9ff0f6fbe85c85fe84fbeac9031bb909442e18178484798529df` +- final CUDA base: + `sha256:56d9d8183e2181a20be6b0d3801d1f056a0e75c17706df939ba207b126e1cb9c` + +`Dockerfile` is deliberately an identity wrapper: it adds provenance labels but +has no `RUN`, `COPY`, or `ADD` instruction. All software comes from the selected +commit's upstream `docker/Dockerfile` and its `vllm-openai` target. + +Run from the setup repository after the pin gate has populated the manifest: + +```bash +webster/deepseek-v41/scripts/build-runtime.sh \ + --run-root /home/ubuntu/deepseek-v41-runs/CHANGE_ID +``` + +The script freezes a bundle, binary diff, submodule state, deterministic source +archive, registry evidence, and dependency freeze. It builds once on Shamu under +a four-CPU, 96-GiB, low-I/O cgroup; no GPU device is granted to BuildKit. It then +saves, checksums, copies over the `10.10.1.2` rail with `rsync --checksum +--bwlimit=250000`, loads on Tilikum, and requires identical image IDs and commit +labels. GLM health is probed directly over NetBird from the orchestrator, so an +unrelated SSH transport stall cannot block or misclassify the serving endpoint; +latency is monitored from Prometheus. Existing divergent +source trees, image tags, archives, or manifest values are never overwritten. + +The constrained build first completes upstream's independent +`vllm-runtime-base` target and then builds `vllm-openai`. This preserves the +upstream source and final target while preventing its manylinux and Ubuntu +branches from racing for the same uv distribution-cache lock. At four CPUs, an +arm64 native dependency build can legitimately hold that lock longer than uv's +default 300-second timeout. + +The upstream CUDA 13 Dockerfile also applies a 500 MiB wheel-size policy for +artifacts uploaded to PyPI, whose project quota is 800 MiB. The upstream arm64 +architecture lane produces a 613.97 MiB wheel at this pinned commit. This +private container build keeps the size check enabled with a recorded 700 MiB +ceiling; it does not discard the check or narrow the upstream architecture set +merely to satisfy the PyPI-oriented default. + +The post-build check uses runtime `runc`, `NVIDIA_VISIBLE_DEVICES=void`, and no +network to inspect CLI help and model registry metadata. Because current vLLM +constructs `DeviceConfig` before rendering help, the probe explicitly sets +`VLLM_TARGET_DEVICE=cpu` and requests `--help=engram-config`; the default help +view contains only configuration groups. It requires `DeepseekV41ForCausalLM`, +`--engram-config`, and the baseline +`{"cpu_offload":true,"embedding_across_dp":false}` without loading weights or +compiling a GPU kernel. The model-registry probe discovers the installed vLLM +package with Python rather than globbing multiple possible environment paths; +GNU grep reports exit 2 when any glob is absent even if another path matches. + +Optimization PRs `#56220`, `#56227`, and `#56344` are explicitly excluded from +this baseline. Their live PR records and ancestry decisions are captured before +the build; any excluded head found in the selected commit fails the gate. diff --git a/webster/deepseek-v41/runtime/constraints.txt b/webster/deepseek-v41/runtime/constraints.txt new file mode 100644 index 0000000..8cc1ab2 --- /dev/null +++ b/webster/deepseek-v41/runtime/constraints.txt @@ -0,0 +1,239 @@ +accelerate==1.15.0 +agent-detector==2.0.0 +aiohappyeyeballs==2.7.1 +aiohttp==3.14.3 +aiosignal==1.4.0 +annotated-doc==0.0.5 +annotated-types==0.8.0 +anthropic==1.5.0 +anyio==4.15.1 +apache-tvm-ffi==0.1.11 +astor==0.8.1 +attrs==26.1.0 +azure-core==1.41.0 +azure-identity==1.25.3 +azure-storage-blob==12.30.1 +blake3==1.0.9 +blinker==1.7.0 +boto3==1.43.93 +botocore==1.43.93 +cachetools==7.1.8 +cbor2==6.1.4 +certifi==2026.7.22 +cffi==2.1.1 +charset-normalizer==3.5.1 +click==8.5.0 +cloudpickle==3.1.2 +compressed-tensors==0.17.0 +cryptography==50.0.1 +cuda-bindings==13.4.1 +cuda-core==1.2.0 +cuda-pathfinder==1.8.1 +cuda-python==13.4.1 +cuda-tile==1.6.0 +cuda-toolkit==13.0.3.0 +dbus-python==1.3.2 +deep_ep @ file:///vllm-workspace/ep_kernels/dist/deep_ep-2.0.0+local-cp312-cp312-linux_aarch64.whl +depyf==0.20.0 +detect-installer==0.2.1 +dill==0.4.1 +distro==1.9.0 +dnspython==2.8.0 +docstring_parser==0.18.0 +einops==0.8.2 +email-validator==2.3.0 +fastapi==0.136.3 +fastapi-cli==0.0.32 +fastapi-cloud-cli==0.26.0 +fastar==0.12.0 +fastsafetensors==0.4.0 +filelock==3.32.6 +flashinfer-cubin==0.6.18.post1 +flashinfer-jit-cache==0.6.18.post1+cu130 +flashinfer-python==0.6.18.post1 +frozenlist==1.8.0 +fsspec==2026.7.0 +google-api-core==2.36.0 +google-auth==2.58.0 +google-cloud-core==2.7.0 +google-cloud-storage==3.14.1 +google-crc32c==1.8.0 +google-resumable-media==2.10.2 +googleapis-common-protos==1.75.3 +grpcio==1.83.1 +h11==0.16.0 +hf-xet==1.6.0 +httpcore==1.0.9 +httpcore2==2.12.0 +httplib2==0.20.4 +httptools==0.8.0 +httpx==0.28.1 +httpx2==2.12.0 +huggingface_hub==1.31.0 +humanize==4.16.0 +humming-kernels==0.1.12 +idna==3.19 +ijson==3.5.1 +instanttensor==0.1.9 +interegular==0.3.3 +isodate==0.7.2 +Jinja2==3.1.6 +jiter==0.16.0 +jmespath==1.1.0 +jsonschema==4.26.0 +jsonschema-specifications==2025.9.1 +lark==1.2.2 +launchpadlib==1.11.0 +lazr.restfulclient==0.14.6 +lazr.uri==1.0.6 +llguidance==1.7.6 +llvmlite==0.47.0 +lm-format-enforcer==0.11.3 +loguru==0.7.3 +markdown-it-py==4.2.0 +MarkupSafe==3.0.3 +mcp==2.2.0 +mcp-types==2.2.0 +mdurl==0.1.2 +mistral_common==1.11.7 +ml_dtypes==0.6.0 +model-hosting-container-standards==0.1.16 +modelscope==1.37.1 +mpmath==1.3.0 +msal==1.38.0 +msal-extensions==1.3.1 +msgspec==0.21.1 +multidict==6.8.0 +nccl4py==0.5.0 +networkx==3.6.1 +ninja==1.13.2 +numba==0.65.0 +numpy==2.3.5 +nvidia-cublas==13.1.1.3 +nvidia-cuda-cccl==13.3.4.2.1 +nvidia-cuda-crt==13.4.59 +nvidia-cuda-cupti==13.0.85 +nvidia-cuda-nvcc==13.4.59 +nvidia-cuda-nvdisasm==13.4.49 +nvidia-cuda-nvrtc==13.0.88 +nvidia-cuda-runtime==13.0.96 +nvidia-cudnn-cu13==9.20.0.48 +nvidia-cudnn-frontend==1.28.0 +nvidia-cufft==12.0.0.61 +nvidia-cufile==1.15.1.6 +nvidia-curand==10.4.0.35 +nvidia-cusolver==12.0.4.66 +nvidia-cusparse==12.6.3.3 +nvidia-cusparselt-cu13==0.8.1 +nvidia-cutlass-dsl==4.6.2 +nvidia-cutlass-dsl-libs-base==4.6.2 +nvidia-cutlass-dsl-libs-core==4.6.2 +nvidia-cutlass-dsl-libs-cu12==4.6.2 +nvidia-cutlass-dsl-libs-cu13==4.6.2 +nvidia-ml-py==13.610.43 +nvidia-nccl-cu13==2.30.7 +nvidia-nvjitlink==13.4.52 +nvidia-nvshmem-cu13==3.4.5 +nvidia-nvtx==13.0.85 +nvidia-nvvm==13.4.59 +nvtx==0.2.15 +oauthlib==3.2.2 +openai==3.13.0 +openai-harmony==0.0.8 +opencv-python-headless==5.0.0.93 +opentelemetry-api==1.44.0 +opentelemetry-exporter-otlp==1.44.0 +opentelemetry-exporter-otlp-proto-common==1.44.0 +opentelemetry-exporter-otlp-proto-grpc==1.44.0 +opentelemetry-exporter-otlp-proto-http==1.44.0 +opentelemetry-proto==1.44.0 +opentelemetry-sdk==1.44.0 +opentelemetry-semantic-conventions==0.65b0 +opentelemetry-semantic-conventions-ai==0.5.1 +outlines_core==0.2.14 +packaging==26.3 +partial-json-parser==0.2.1.1.post7 +pillow==12.3.0 +prometheus-fastapi-instrumentator==8.1.0 +prometheus_client==0.26.0 +propcache==0.5.2 +proto-plus==1.28.4 +protobuf==6.33.6 +psutil==7.2.2 +py-cpuinfo==9.0.0 +pyasn1==0.6.4 +pyasn1_modules==0.4.2 +pybase64==1.5.0 +pycountry==26.2.16 +pycparser==3.0 +pydantic==2.13.5 +pydantic-extra-types==2.11.1 +pydantic-settings==2.15.0 +pydantic_core==2.46.5 +Pygments==2.21.0 +PyGObject==3.48.2 +PyJWT==2.14.0 +pynvvideocodec==2.0.4 +pyparsing==3.1.1 +python-apt==2.7.7+ubuntu5.3 +python-dateutil==2.9.0.post0 +python-dotenv==1.2.3 +python-json-logger==4.2.0 +python-multipart==0.0.32 +PyYAML==6.0.3 +pyzmq==27.2.0 +quack-kernels==0.6.4 +referencing==0.37.0 +regex==2026.9.10 +requests==2.34.2 +rich==15.0.0 +rich-toolkit==0.20.5 +rignore==0.8.1 +rpds-py==2026.6.3 +runai-model-streamer==0.16.1 +runai-model-streamer-azure==0.16.1 +runai-model-streamer-gcs==0.16.1 +runai-model-streamer-s3==0.16.1 +s3transfer==0.19.2 +safetensors==0.8.0 +sentencepiece==0.2.2 +sentry-sdk==2.69.1 +setproctitle==1.3.7 +setuptools==80.10.2 +shellingham==1.5.4 +six==1.17.0 +sniffio==1.3.1 +sse-starlette==3.4.11 +starlette==1.6.0 +supervisor==4.3.0 +sympy==1.14.0 +tabulate==0.10.0 +tiktoken==0.14.0 +tilelang==0.1.12 +timm==1.0.29 +tokenizers==0.23.2 +tokenspeed-mla==0.1.8 +tokenspeed-triton==3.8.10.post20260906 +torch==2.13.0+cu130 +torch_c_dlpack_ext==0.1.5 +torchaudio==2.11.0+cu130 +torchcodec==0.16.0+cu130 +torchvision==0.28.0+cu130 +tqdm==4.70.1 +transformers==5.17.0 +triton==3.7.1 +truststore==0.10.4 +typer==0.27.2 +typing-inspection==0.4.4 +typing_extensions==4.16.0 +urllib3==2.7.0 +uv==0.12.13 +uvicorn==0.52.4 +uvloop==0.22.1 +vllm @ file:///vllm-workspace/dist/vllm-0.1.1.dev5+ge77daef89-cp38-abi3-linux_aarch64.whl +wadllib==1.3.6 +watchfiles==1.2.0 +websockets==17.1 +xgrammar==0.2.6 +yarl==1.24.5 +z3-solver==4.15.4.0 diff --git a/webster/deepseek-v41/scripts/acceptance.sh b/webster/deepseek-v41/scripts/acceptance.sh new file mode 100755 index 0000000..e786e9a --- /dev/null +++ b/webster/deepseek-v41/scripts/acceptance.sh @@ -0,0 +1,578 @@ +#!/usr/bin/env bash +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$script_dir/common.sh" +unset SSH_AUTH_SOCK + +run_root="" +while (( $# )); do + case "$1" in + --run-root) run_root="${2:-}"; shift 2 ;; + *) die "unknown argument: $1" ;; + esac +done +validated_root="$(validate_run_root "$run_root")" +init_run_root "$validated_root" + +test_acceptance() { + require_eq "Shamu DeepSeek running" true \ + "${TEST_ACCEPTANCE_DEEPSEEK_SHAMU_RUNNING:-true}" + require_eq "Tilikum DeepSeek running" true \ + "${TEST_ACCEPTANCE_DEEPSEEK_TILIKUM_RUNNING:-true}" + require_eq "Shamu GLM stopped" false \ + "${TEST_ACCEPTANCE_GLM_SHAMU_RUNNING:-false}" + require_eq "Tilikum GLM stopped" false \ + "${TEST_ACCEPTANCE_GLM_TILIKUM_RUNNING:-false}" + require_eq "direct authenticated completion" 200 \ + "${TEST_ACCEPTANCE_DIRECT_STATUS:-200}" + require_eq "missing direct key rejection" 401 \ + "${TEST_ACCEPTANCE_MISSING_KEY_STATUS:-401}" + require_eq "wrong direct key rejection" 401 \ + "${TEST_ACCEPTANCE_WRONG_KEY_STATUS:-401}" + require_eq "private model matrix" ok \ + "${TEST_ACCEPTANCE_PRIVATE_MODELS:-ok}" + require_eq "public model matrix" ok \ + "${TEST_ACCEPTANCE_PUBLIC_MODELS:-ok}" + require_eq "published metadata" ok \ + "${TEST_ACCEPTANCE_METADATA:-ok}" + require_eq "Prometheus targets" ok \ + "${TEST_ACCEPTANCE_PROMETHEUS:-ok}" + require_eq "Prometheus DeepSeek probe route" matched \ + "${TEST_ACCEPTANCE_PROMETHEUS_ROUTE:-matched}" + require_eq "Langfuse trace" ok \ + "${TEST_ACCEPTANCE_LANGFUSE:-ok}" + require_eq "Langfuse acceptance probe" matched \ + "${TEST_ACCEPTANCE_LANGFUSE_PROBE:-matched}" + require_eq "Baker semantic snapshot" equal \ + "${TEST_ACCEPTANCE_BAKER_EQUAL:-equal}" + require_eq "rollback path" ok \ + "${TEST_ACCEPTANCE_ROLLBACK:-ok}" +} + +if [[ "${WEBSTER_ACCEPTANCE_TEST_MODE:-0}" == "1" ]]; then + test_acceptance + event publication-acceptance GO "test-mode publication matrix passed" + printf 'publication acceptance validated in test mode\n' + exit 0 +fi + +capture_baker_node() { + local host="$1" role="$2" output="$3" + shift 3 + ssh "$host" python3 - "$role" "$BAKER_CONTAINER" "$@" <<'PY' >"$output" +import hashlib +import json +import os +import subprocess +import sys +from pathlib import Path + +role, container_name, *docker = sys.argv[1:] + + +def require(condition, message): + if not condition: + raise RuntimeError(message) + + +result = subprocess.run( + docker + ["inspect", container_name], capture_output=True, text=True +) +require(result.returncode == 0, f"Baker {role} container inspect failed") +records = json.loads(result.stdout) +require(isinstance(records, list) and len(records) == 1, "Baker inspect is malformed") +record = records[0] +config_root = Path.home() / "dspark-recipe" +require(config_root.is_dir() and not config_root.is_symlink(), "Baker config root is missing") +config_hashes = {} +config_links = {} +config_names = { + ".env.dspark", + "Dockerfile", + "compose.yaml", + "compose.yml", + "docker-compose.yaml", + "docker-compose.yml", +} +config_suffixes = {".conf", ".env", ".json", ".py", ".sh", ".toml", ".yaml", ".yml"} +for path in sorted(config_root.rglob("*")): + relative = str(path.relative_to(config_root)) + if path.is_symlink(): + config_links[relative] = os.readlink(path) + elif path.is_file() and ( + path.name in config_names or path.suffix.lower() in config_suffixes + ): + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + config_hashes[relative] = digest.hexdigest() +require(".env.dspark" in config_hashes, "Baker .env.dspark identity is missing") +state = record.get("State") or {} +host_config = record.get("HostConfig") or {} +container_config = record.get("Config") or {} +mounts = [ + { + "type": mount.get("Type"), + "source": mount.get("Source"), + "destination": mount.get("Destination"), + "mode": mount.get("Mode"), + "rw": mount.get("RW"), + "propagation": mount.get("Propagation"), + } + for mount in record.get("Mounts") or [] +] +mounts.sort(key=lambda item: (str(item["destination"]), str(item["source"]))) +snapshot = { + "role": role, + "container": { + "id": record.get("Id"), + "image_id": record.get("Image"), + "configured_image": container_config.get("Image"), + "entrypoint": container_config.get("Entrypoint"), + "cmd": container_config.get("Cmd"), + "args": record.get("Args"), + "state": { + "status": state.get("Status"), + "running": state.get("Running"), + "paused": state.get("Paused"), + "restarting": state.get("Restarting"), + "oom_killed": state.get("OOMKilled"), + "dead": state.get("Dead"), + }, + "restart_count": record.get("RestartCount"), + "network_mode": host_config.get("NetworkMode"), + "ipc_mode": host_config.get("IpcMode"), + "mounts": mounts, + }, + "config_root": str(config_root), + "config_sha256": config_hashes, + "config_links": config_links, +} +require(snapshot["container"]["id"], "Baker container ID is missing") +require(snapshot["container"]["image_id"], "Baker image identity is missing") +require(snapshot["container"]["state"]["running"] is True, "Baker container is not running") +print(json.dumps(snapshot, sort_keys=True)) +PY + chmod 0600 "$output" +} + +write_baker_snapshot() { + local phase="$1" rank0 rank1 output + rank0="$RUN_ROOT/logs/publication-baker-$phase-rank0.json" + rank1="$RUN_ROOT/logs/publication-baker-$phase-rank1.json" + output="$RUN_ROOT/logs/publication-baker-$phase.json" + capture_baker_node "$BAKER_RANK0_HOST" rank0 "$rank0" docker + capture_baker_node "$BAKER_RANK1_HOST" rank1 "$rank1" sudo -n docker + python3 - "$rank0" "$rank1" "$BAKER_RANK0_NETBIRD" "$BAKER_API_PORT" <<'PY' >"$output" +import json +import re +import sys +import urllib.request +from pathlib import Path + +rank0_path, rank1_path, host, port = sys.argv[1:] + + +def require(condition, message): + if not condition: + raise RuntimeError(message) + + +with urllib.request.urlopen(f"http://{host}:{port}/health", timeout=15) as response: + health_status = response.status +with urllib.request.urlopen(f"http://{host}:{port}/metrics", timeout=15) as response: + metrics_status = response.status + metrics = response.read().decode("utf-8") +required = { + "vllm:kv_cache_usage_perc", + "vllm:num_requests_running", + "vllm:num_requests_waiting", + "vllm:request_success_total", +} +seen = set() +model_names = {name: set() for name in required} +for line in metrics.splitlines(): + if not line or line.startswith("#"): + continue + sample = line.split(None, 1)[0] + name = sample.split("{", 1)[0] + if name not in required: + continue + seen.add(name) + labels = sample[len(name):] + for match in re.finditer(r'(?:^|[,{}])model_name="((?:\\.|[^"])*)"', labels): + model_names[name].add(match.group(1)) +require(health_status == 200 and metrics_status == 200, "Baker health or metrics failed") +require(seen == required, "Baker required metrics are incomplete") +snapshot = { + "rank0": json.loads(Path(rank0_path).read_text(encoding="utf-8")), + "rank1": json.loads(Path(rank1_path).read_text(encoding="utf-8")), + "metrics": { + "health_status": health_status, + "metrics_status": metrics_status, + "required_families": sorted(required), + "model_names": {name: sorted(model_names[name]) for name in sorted(required)}, + }, +} +print(json.dumps(snapshot, sort_keys=True)) +PY + chmod 0600 "$output" +} + +write_prometheus_snapshot() { + local previous="${1:-}" + python3 - "$PROMETHEUS_URL" "$SHAMU_NETBIRD:$CANARY_PORT" "$previous" <<'PY' +import json +import math +import sys +import time +import urllib.parse +import urllib.request +from pathlib import Path + +prometheus, instance, previous_path = sys.argv[1:] + + +def require(condition, message): + if not condition: + raise RuntimeError(message) + + +def fetch_json(path, parameters=None): + suffix = "" + if parameters: + suffix = "?" + urllib.parse.urlencode(parameters) + with urllib.request.urlopen(prometheus.rstrip("/") + path + suffix, timeout=15) as response: + return json.load(response) + + +def inspect(): + targets_payload = fetch_json("/api/v1/targets") + targets = targets_payload.get("data", {}).get("activeTargets", []) + matched = [ + target + for target in targets + if target.get("labels", {}).get("instance") == instance + and urllib.parse.urlparse(target.get("scrapeUrl", "")).netloc == instance + ] + require(matched, "Shamu :8000 is not an active Prometheus scrape target") + require( + all(target.get("health") == "up" for target in matched), + "Shamu :8000 Prometheus scrape target is not up", + ) + query = f'sum(vllm:request_success_total{{instance="{instance}"}})' + counter_payload = fetch_json("/api/v1/query", {"query": query}) + require(counter_payload.get("status") == "success", "Prometheus counter query failed") + rows = counter_payload.get("data", {}).get("result", []) + require(len(rows) == 1, "Prometheus DeepSeek request counter is absent") + try: + counter = float(rows[0]["value"][1]) + except (KeyError, IndexError, TypeError, ValueError) as error: + raise RuntimeError("Prometheus DeepSeek request counter is malformed") from error + require(math.isfinite(counter) and counter >= 0, "Prometheus DeepSeek request counter is invalid") + return matched, counter + + +baseline = None +if previous_path: + previous = json.loads(Path(previous_path).read_text(encoding="utf-8")) + baseline = previous.get("request_success_total") + require(isinstance(baseline, (int, float)) and not isinstance(baseline, bool), "Prometheus baseline is malformed") +deadline = time.monotonic() + 90 +while True: + matched, counter = inspect() + if baseline is None or counter > baseline: + break + if time.monotonic() >= deadline: + raise RuntimeError("Prometheus DeepSeek counter did not increase after the acceptance probe") + time.sleep(5) +print(json.dumps({ + "instance": instance, + "request_success_total": counter, + "active_up_targets": [ + {"health": target.get("health"), "scrape_url": target.get("scrapeUrl")} + for target in matched + ], +}, sort_keys=True)) +PY +} + +"$script_dir/preflight.sh" --phase publish --run-root "$RUN_ROOT" + +[[ -f "$RUN_ROOT/rollback.env" && ! -L "$RUN_ROOT/rollback.env" ]] || + die "rollback.env is missing" +require_eq "rollback.env mode" 600 "$(stat -c %a "$RUN_ROOT/rollback.env")" +declare -A rollback +while IFS='=' read -r key value; do + [[ -z "$key" || "$key" == \#* ]] && continue + case "$key" in + ALIAS_CONFIG_BACKUP|ALIAS_CONFIG_SHA256|PUBLISH_CONFIG_BACKUP|PUBLISH_CONFIG_SHA256) + rollback["$key"]="$value" + ;; + *) die "unknown rollback.env field: $key" ;; + esac +done <"$RUN_ROOT/rollback.env" +publish_backup="${rollback[PUBLISH_CONFIG_BACKUP]:-}" +publish_hash="${rollback[PUBLISH_CONFIG_SHA256]:-}" +[[ "$publish_backup" == /home/nvidia/litellm/* && "$publish_backup" != *'/../'* ]] || + die "PUBLISH_CONFIG_BACKUP escaped the LiteLLM directory" +[[ "$publish_hash" =~ ^[0-9a-f]{64}$ ]] || die "PUBLISH_CONFIG_SHA256 is invalid" +require_eq "publication rollback mode" 600 \ + "$(ssh spark-1 "stat -c %a -- '$publish_backup'")" +require_eq "publication rollback hash" "$publish_hash" \ + "$(ssh spark-1 "sha256sum -- '$publish_backup' | awk '{print \$1}'")" +[[ -x "$script_dir/restore-litellm-config.sh" ]] || die "rollback entry point is missing" + +source "$RUN_ROOT/manifest.env" +[[ "$VLLM_IMAGE_ID" =~ ^sha256:[0-9a-f]{64}$ ]] || die "VLLM_IMAGE_ID is invalid" +rank_summary() { + local host="$1" role="$2" + shift 2 + ssh "$host" python3 - "$role" "$VLLM_IMAGE_ID" "$@" <<'PY' +import json +import subprocess +import sys + +role, expected_image, *docker = sys.argv[1:] + + +def inspect(name): + result = subprocess.run(docker + ["inspect", name], capture_output=True, text=True) + if result.returncode: + return None + return json.loads(result.stdout)[0] + + +def require(condition, message): + if not condition: + raise RuntimeError(message) + + +deepseek = inspect("deepseek-v41-flash-tp2") +glm = inspect("glm52-full-mtp") +require(deepseek is not None and glm is not None, "serving containers are missing") +args = deepseek["Args"] +environment = deepseek["Config"].get("Env") or [] +key_present = any( + item.startswith(("VLLM_API_KEY=", "DSPARK_API_KEYS=")) + and item.split("=", 1)[1] + for item in environment +) +require(deepseek["Image"] == expected_image, "DeepSeek image is wrong") +require(deepseek["State"]["Running"] is True, "DeepSeek is not running") +require(deepseek["RestartCount"] == 0, "DeepSeek restart count is nonzero") +require(glm["State"]["Running"] is False, "station GLM is still running") +require(("--headless" in args) is (role == "worker"), "rank role is wrong") +require(key_present is (role == "head"), "rank credential isolation is wrong") +require("--tensor-parallel-size" in args and args[args.index("--tensor-parallel-size") + 1] == "2", "TP size is wrong") +require("--pipeline-parallel-size" in args and args[args.index("--pipeline-parallel-size") + 1] == "1", "PP size is wrong") +require("--max-model-len" in args and args[args.index("--max-model-len") + 1] == "1048576", "context is wrong") +require("--max-num-seqs" in args and args[args.index("--max-num-seqs") + 1] == "4", "max sequences is wrong") +print(json.dumps({ + "role": role, + "id": deepseek["Id"], + "image": deepseek["Image"], + "running": True, + "restarts": 0, + "headless": "--headless" in args, + "key_present": bool(key_present), +}, sort_keys=True)) +PY +} +rank_summary shamu head docker >"$RUN_ROOT/logs/publication-rank-shamu.json" +rank_summary tilikum worker sudo -n docker >"$RUN_ROOT/logs/publication-rank-tilikum.json" +chmod 0600 "$RUN_ROOT/logs/publication-rank-"*.json +require_eq "Shamu listener" "$SHAMU_NETBIRD:$CANARY_PORT" \ + "$(ssh shamu "ss -lntH '( sport = :$CANARY_PORT )' | awk '{print \$4}'")" +if ssh tilikum "ss -lntH '( sport = :$CANARY_PORT )' | grep -q ."; then + die "Tilikum exposes an HTTP listener" +fi +require_eq "watchdog enabled" enabled "$(systemctl is-enabled deepseek-v41-watchdog.timer)" +require_eq "watchdog active" active "$(systemctl is-active deepseek-v41-watchdog.timer)" + +python3 - "$RUN_ROOT" "$SHAMU_NETBIRD" "$CANARY_PORT" <<'PY' +import json +import os +import stat +import sys +import time +import urllib.error +import urllib.request +from pathlib import Path + +run_root, host, port = sys.argv[1:] +key_path = Path(run_root) / "credentials/deepseek-v41.key" +def require(condition, message): + if not condition: + raise RuntimeError(message) + + +require(key_path.is_file() and not key_path.is_symlink(), "credential is missing or unsafe") +require(stat.S_IMODE(key_path.stat().st_mode) == 0o600, "credential mode is not 0600") +key = key_path.read_text(encoding="utf-8").strip() +base = f"http://{host}:{port}" + + +def request(path, body=None, token=None): + headers = {} + data = None + if token is not None: + headers["Authorization"] = "Bearer " + token + if body is not None: + headers["Content-Type"] = "application/json" + data = json.dumps(body, separators=(",", ":")).encode() + value = urllib.request.Request( + base + path, + data=data, + headers=headers, + method="POST" if body is not None else "GET", + ) + started = time.monotonic() + try: + with urllib.request.urlopen(value, timeout=180) as response: + return response.status, json.load(response), time.monotonic() - started + except urllib.error.HTTPError as error: + return error.code, None, time.monotonic() - started + + +missing, _, _ = request("/v1/models") +wrong, _, _ = request("/v1/models", token="intentionally-wrong") +models_status, models, _ = request("/v1/models", token=key) +status, payload, latency = request( + "/v1/chat/completions", + { + "model": "deepseek-ai/DeepSeek-V4.1-Flash", + "messages": [{"role": "user", "content": "Reply with exactly OK."}], + "max_tokens": 16, + "temperature": 0, + "reasoning_effort": "none", + }, + key, +) +require((missing, wrong, models_status, status) == (401, 401, 200, 200), "station authentication responses are wrong") +require(any(item.get("id") == "deepseek-ai/DeepSeek-V4.1-Flash" for item in models["data"]), "served model is absent") +require(payload.get("choices"), "station completion has no choices") +result = { + "missing_key_status": missing, + "wrong_key_status": wrong, + "models_status": models_status, + "completion_status": status, + "response_model": payload.get("model"), + "latency_seconds": round(latency, 6), +} +output = Path(run_root) / "logs/publication-direct.json" +temporary = output.with_name(output.name + f".tmp.{os.getpid()}") +temporary.write_text(json.dumps(result, sort_keys=True) + "\n", encoding="utf-8") +os.chmod(temporary, 0o600) +os.replace(temporary, output) +PY + +ssh spark-1 python3 - "$(basename "$RUN_ROOT")" \ + <"$script_dir/current-model-probes.py" \ + >"$RUN_ROOT/logs/publication-private-models.json" +chmod 0600 "$RUN_ROOT/logs/publication-private-models.json" + +probe_id="deepseek-v41-acceptance-$(basename "$RUN_ROOT")-$(date -u +%s)-$$" +probe_started_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +write_baker_snapshot before +write_prometheus_snapshot >"$RUN_ROOT/logs/publication-prometheus-before.json" +chmod 0600 "$RUN_ROOT/logs/publication-prometheus-before.json" + +ssh spark-1 python3 - "$probe_id" \ + <"$script_dir/publication-model-probes.py" \ + >"$RUN_ROOT/logs/publication-public-models.json" +chmod 0600 "$RUN_ROOT/logs/publication-public-models.json" + +write_prometheus_snapshot \ + "$RUN_ROOT/logs/publication-prometheus-before.json" \ + >"$RUN_ROOT/logs/publication-prometheus.json" +chmod 0600 "$RUN_ROOT/logs/publication-prometheus.json" + +ssh spark-1 python3 - "$probe_id" "$probe_started_at" <<'PY' >"$RUN_ROOT/logs/publication-langfuse.json" +import base64 +import json +import subprocess +import sys +import time +import urllib.parse +import urllib.request + +probe_id, started_at = sys.argv[1:] +environment = subprocess.check_output( + ["docker", "inspect", "litellm", "--format", "{{range .Config.Env}}{{println .}}{{end}}"], + text=True, +).splitlines() +values = {item.split("=", 1)[0]: item.split("=", 1)[1] for item in environment if "=" in item} +host = values["LANGFUSE_HOST"].rstrip("/") +credentials = base64.b64encode( + (values["LANGFUSE_PUBLIC_KEY"] + ":" + values["LANGFUSE_SECRET_KEY"]).encode() +).decode() + + +def contains_exact(value, expected): + if isinstance(value, str): + return value == expected + if isinstance(value, list): + return any(contains_exact(item, expected) for item in value) + if isinstance(value, dict): + return any(contains_exact(item, expected) for item in value.values()) + return False + + +parameters = urllib.parse.urlencode({ + "limit": 100, + "orderBy": "timestamp.desc", + "fromTimestamp": started_at, + "userId": probe_id, +}) +deadline = time.monotonic() + 120 +matched = None +while time.monotonic() < deadline: + request = urllib.request.Request( + host + "/api/public/traces?" + parameters, + headers={"Authorization": "Basic " + credentials}, + ) + with urllib.request.urlopen(request, timeout=20) as response: + payload = json.load(response) + for row in payload.get("data", []): + if contains_exact(row, probe_id) and contains_exact(row, "deepseek-v4.1-flash"): + matched = row + break + if matched is not None: + break + time.sleep(5) +if matched is None: + raise RuntimeError("no Langfuse trace matched the exact acceptance probe and DeepSeek route") +print(json.dumps({ + "host": host, + "probe_id": probe_id, + "trace": { + "id": matched.get("id"), + "timestamp": matched.get("timestamp") or matched.get("createdAt"), + "name": matched.get("name"), + "deepseek_route": "deepseek-v4.1-flash", + }, +}, sort_keys=True)) +PY +chmod 0600 "$RUN_ROOT/logs/publication-langfuse.json" + +write_baker_snapshot after +python3 - \ + "$RUN_ROOT/logs/publication-baker-before.json" \ + "$RUN_ROOT/logs/publication-baker-after.json" <<'PY' +import json +import sys +from pathlib import Path + +before_path, after_path = map(Path, sys.argv[1:]) +before = json.loads(before_path.read_text(encoding="utf-8")) +after = json.loads(after_path.read_text(encoding="utf-8")) +if before != after: + raise RuntimeError("Baker container, config, identity, or metric semantics drifted") +PY + +event publication-acceptance GO \ + "direct/private/public models, metadata, ingress, observability, watchdog, and rollback passed" +printf 'GO publication acceptance run_root=%s\n' "$RUN_ROOT" diff --git a/webster/deepseek-v41/scripts/build-runtime.sh b/webster/deepseek-v41/scripts/build-runtime.sh new file mode 100755 index 0000000..643eeb3 --- /dev/null +++ b/webster/deepseek-v41/scripts/build-runtime.sh @@ -0,0 +1,741 @@ +#!/usr/bin/env bash +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +package_root="$(cd "$script_dir/.." && pwd)" +source "$script_dir/common.sh" +registry_probe="$script_dir/verify-runtime-registry.py" +[[ -x "$registry_probe" ]] || die "runtime registry probe is not executable" + +run_root="" +source_dir="" +while (($#)); do + case "$1" in + --run-root) + [[ $# -ge 2 ]] || die "--run-root requires a value" + run_root="$2" + shift 2 + ;; + --source-dir) + [[ $# -ge 2 ]] || die "--source-dir requires a value" + source_dir="$2" + shift 2 + ;; + *) die "unknown argument: $1" ;; + esac +done + +validated_root="$(validate_run_root "$run_root")" +[[ -d "$validated_root" ]] || die "run root does not exist" +RUN_ROOT="$validated_root" +export RUN_ROOT +manifest="$RUN_ROOT/manifest.env" +[[ -f "$manifest" ]] || die "manifest.env does not exist" +require_eq "manifest mode" "600" "$(stat -c %a "$manifest")" + +manifest_value() { + local key="$1" count value + count="$(grep -c "^${key}=" "$manifest" || true)" + require_eq "manifest field count for $key" "1" "$count" + value="$(sed -n "s/^${key}=//p" "$manifest")" + printf '%s\n' "$value" +} + +set_manifest_value() { + local key="$1" value="$2" + python3 - "$manifest" "$key" "$value" <<'PY' +import os +import re +import sys +from pathlib import Path + +path = Path(sys.argv[1]) +key = sys.argv[2] +value = sys.argv[3] +if not re.fullmatch(r"[A-Z][A-Z0-9_]*", key): + raise SystemExit("invalid manifest field") +lines = path.read_text(encoding="utf-8").splitlines() +matches = [index for index, line in enumerate(lines) if line.startswith(key + "=")] +if len(matches) > 1: + raise SystemExit(f"manifest contains duplicate {key}") +if matches: + index = matches[0] + existing = lines[index].split("=", 1)[1] + if existing and existing != value: + raise SystemExit(f"manifest already pins a different {key}") + lines[index] = f"{key}={value}" +else: + lines.append(f"{key}={value}") +temporary = path.with_name(path.name + f".tmp.{os.getpid()}") +descriptor = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) +try: + with os.fdopen(descriptor, "w", encoding="utf-8") as stream: + stream.write("\n".join(lines) + "\n") + stream.flush() + os.fsync(stream.fileno()) + os.replace(temporary, path) + os.chmod(path, 0o600) +finally: + if temporary.exists(): + temporary.unlink() +PY +} + +require_sha256() { + [[ "$2" =~ ^sha256:[0-9a-f]{64}$ ]] || die "$1 is not a sha256 digest" +} + +validate_cgroup_parent() { + local parent="$1" + [[ "$parent" == /* ]] || { + printf 'ERROR: BuildKit cgroup parent must be an absolute ControlGroup\n' >&2 + return 1 + } + [[ "$parent" != *".."* ]] || { + printf 'ERROR: BuildKit cgroup parent contains traversal\n' >&2 + return 1 + } +} + +validate_wheel_ceiling() { + local ceiling="$1" + [[ "$ceiling" =~ ^[1-9][0-9]*$ ]] || + die "runtime wheel ceiling must be a positive integer MiB value" + (( ceiling <= 800 )) || + die "runtime wheel ceiling exceeds the upstream 800 MiB quota" +} + +validate_help_probe() { + local visible_devices="$1" target_device="$2" selector="$3" + require_eq "runtime help probe GPU visibility" "void" "$visible_devices" + require_eq "runtime help probe target device" "cpu" "$target_device" + require_eq "runtime help probe selector" "engram-config" "$selector" +} + +vllm_commit="$(manifest_value VLLM_COMMIT)" +[[ "$vllm_commit" =~ ^[0-9a-f]{40}$ ]] || die "VLLM_COMMIT is not pinned" +source_dir="${source_dir:-$RUN_ROOT/runtime-source/vllm}" +[[ -d "$source_dir/.git" ]] || die "vLLM source is not a Git checkout" +require_eq "vLLM source HEAD" "$vllm_commit" \ + "$(git -C "$source_dir" rev-parse HEAD)" +[[ -z "$(git -C "$source_dir" status --porcelain --untracked-files=all)" ]] || + die "vLLM source checkout is dirty" +if git -C "$source_dir" symbolic-ref -q HEAD >/dev/null; then + die "vLLM source checkout must be detached" +fi + +validate_runtime_gate() { + local arch="$1" build_platform="$2" final_platform="$3" + local transfer_match="$4" image_ids="$5" internal_commits="$6" + local help_text="$7" registry_text="$8" engram_layers="$9" + local -a ids commits + require_eq "runtime builder architecture" "arm64" "$arch" + require_eq "build base platform" "linux/arm64" "$build_platform" + require_eq "final base platform" "linux/arm64" "$final_platform" + require_eq "saved-image transfer checksum" "1" "$transfer_match" + mapfile -t ids <<<"$image_ids" + mapfile -t commits <<<"$internal_commits" + require_eq "station image count" "2" "${#ids[@]}" + require_eq "station commit count" "2" "${#commits[@]}" + require_sha256 "Shamu image ID" "${ids[0]}" + require_sha256 "Tilikum image ID" "${ids[1]}" + require_eq "station image ID" "${ids[0]}" "${ids[1]}" + require_eq "Shamu internal vLLM commit" "$vllm_commit" "${commits[0]}" + require_eq "Tilikum internal vLLM commit" "$vllm_commit" "${commits[1]}" + [[ "$help_text" == *"--engram-config"* ]] || die "runtime lacks --engram-config" + [[ "$registry_text" == *"DeepseekV41ForCausalLM"* ]] || + die "runtime lacks DeepseekV41ForCausalLM registry support" + [[ -n "$engram_layers" ]] || die "checkpoint has no engram_layer_ids" +} + +if [[ "${WEBSTER_RUNTIME_TEST_MODE:-0}" == "1" ]]; then + runtime_wheel_ceiling="${TEST_RUNTIME_MAX_WHEEL_SIZE_MB:-$VLLM_MAX_WHEEL_SIZE_MB}" + validate_wheel_ceiling "$runtime_wheel_ceiling" + validate_help_probe \ + "${TEST_RUNTIME_HELP_VISIBLE_DEVICES:-void}" \ + "${TEST_RUNTIME_HELP_TARGET_DEVICE:-cpu}" \ + "${TEST_RUNTIME_HELP_SELECTOR:-engram-config}" + validate_runtime_gate \ + "${TEST_RUNTIME_REMOTE_ARCH:-}" \ + "${TEST_RUNTIME_BUILD_BASE_PLATFORM:-}" \ + "${TEST_RUNTIME_FINAL_BASE_PLATFORM:-}" \ + "${TEST_RUNTIME_TRANSFER_SHA_MATCH:-}" \ + "${TEST_RUNTIME_IMAGE_IDS:-}" \ + "${TEST_RUNTIME_INTERNAL_COMMITS:-}" \ + "${TEST_RUNTIME_HELP_TEXT:-}" \ + "${TEST_RUNTIME_REGISTRY_TEXT:-}" \ + "${TEST_RUNTIME_ENGRAM_LAYER_IDS:-}" + validate_cgroup_parent "${TEST_RUNTIME_CGROUP_PARENT:-}" + test_image_id="${TEST_RUNTIME_IMAGE_IDS%%$'\n'*}" + set_manifest_value VLLM_BASE_IMAGE_DIGEST "$VLLM_BUILD_BASE_IMAGE_DIGEST" + set_manifest_value VLLM_BUILD_BASE_IMAGE_DIGEST "$VLLM_BUILD_BASE_IMAGE_DIGEST" + set_manifest_value VLLM_FINAL_BASE_IMAGE_DIGEST "$VLLM_FINAL_BASE_IMAGE_DIGEST" + set_manifest_value VLLM_SOURCE_ARCHIVE_SHA256 \ + "${TEST_RUNTIME_SOURCE_ARCHIVE_SHA256:-$(printf test-source | sha256sum | awk '{print $1}')}" + set_manifest_value VLLM_MAX_WHEEL_SIZE_MB "$runtime_wheel_ceiling" + set_manifest_value VLLM_IMAGE_ID "$test_image_id" + set_manifest_value VLLM_IMAGE_TAR_SHA256 \ + "${TEST_RUNTIME_IMAGE_TAR_SHA256:-$(printf test-image | sha256sum | awk '{print $1}')}" + printf 'GO runtime-package test-mode\n' + exit 0 +fi + +validate_wheel_ceiling "$VLLM_MAX_WHEEL_SIZE_MB" + +[[ "$(uname -m)" == "x86_64" || "$(uname -m)" == "aarch64" ]] || + die "unsupported orchestrator architecture" +arm_lane="$source_dir/.buildkite/image_build/image_build_arm64.sh" +release_lane="$source_dir/.buildkite/release-pipeline.yaml" +[[ -f "$arm_lane" && -f "$release_lane" ]] || die "upstream arm64 build lanes missing" +grep -Fq "$VLLM_BUILD_BASE_IMAGE_TAG" "$arm_lane" || + die "selected source no longer names the reviewed arm64 builder" +grep -Fq -- '--platform linux/arm64' "$arm_lane" || + die "selected source no longer builds the arm64 platform" +grep -Fq 'torch_cuda_arch_list="9.0 10.0 11.0 12.0"' "$arm_lane" || + die "selected source arm64 architecture list changed" +grep -Fq -- '--target vllm-openai' "$release_lane" || + die "selected source lacks an arm64 vllm-openai release lane" +check_glm_health || + die "GLM health failed before runtime packaging" + +evidence_dir="$RUN_ROOT/runtime-source/evidence" +mkdir -p "$evidence_dir" +chmod 0700 "$RUN_ROOT/runtime-source" "$evidence_dir" + +resolve_platform_manifest() { + local label="$1" tag="$2" expected="$3" output="$4" selected + docker manifest inspect --verbose "$tag" >"$output" + chmod 0600 "$output" + selected="$(python3 - "$output" <<'PY' +import json +import sys + +payload = json.load(open(sys.argv[1], encoding="utf-8")) +if isinstance(payload, dict): + payload = [payload] +matches = [] +for item in payload: + descriptor = item.get("Descriptor", {}) + platform = descriptor.get("platform", {}) + if platform.get("os") == "linux" and platform.get("architecture") == "arm64": + matches.append(descriptor.get("digest")) +if len(matches) != 1 or not isinstance(matches[0], str): + raise SystemExit("expected exactly one linux/arm64 manifest") +print(matches[0]) +PY +)" + require_eq "$label platform digest" "$expected" "$selected" +} + +resolve_platform_manifest "build base" "$VLLM_BUILD_BASE_IMAGE_TAG" \ + "$VLLM_BUILD_BASE_IMAGE_DIGEST" "$evidence_dir/build-base-manifest.json" +resolve_platform_manifest "final base" "$VLLM_FINAL_BASE_IMAGE_TAG" \ + "$VLLM_FINAL_BASE_IMAGE_DIGEST" "$evidence_dir/final-base-manifest.json" +docker buildx imagetools inspect "$VLLM_BUILD_BASE_IMAGE_TAG" \ + >"$evidence_dir/build-base-index.txt" +docker buildx imagetools inspect "$VLLM_FINAL_BASE_IMAGE_TAG" \ + >"$evidence_dir/final-base-index.txt" +chmod 0600 "$evidence_dir"/* + +for excluded_pr in 56220 56227 56344; do + excluded_file="$evidence_dir/excluded-pr-$excluded_pr.json" + python3 - "$excluded_pr" "$excluded_file" <<'PY' +import json +import os +import sys +import urllib.request + +number, output = sys.argv[1:] +request = urllib.request.Request( + f"https://api.github.com/repos/vllm-project/vllm/pulls/{number}", + headers={ + "Accept": "application/vnd.github+json", + "User-Agent": "webster-runtime-gate", + "X-GitHub-Api-Version": "2022-11-28", + }, +) +with urllib.request.urlopen(request, timeout=30) as response: + payload = json.load(response) +temporary = output + f".tmp.{os.getpid()}" +with open(temporary, "w", encoding="utf-8") as stream: + json.dump(payload, stream, indent=2, sort_keys=True) + stream.write("\n") +os.chmod(temporary, 0o600) +os.replace(temporary, output) +PY + excluded_head="$(python3 - "$excluded_file" <<'PY' +import json +import sys + +payload = json.load(open(sys.argv[1], encoding="utf-8")) +head = payload.get("head", {}).get("sha") +if not isinstance(head, str) or len(head) != 40: + raise SystemExit("excluded PR has no immutable head") +print(head) +PY +)" + if git -C "$source_dir" cat-file -e "$excluded_head^{commit}" 2>/dev/null && + git -C "$source_dir" merge-base --is-ancestor "$excluded_head" "$vllm_commit"; then + die "excluded PR #$excluded_pr is present in the baseline commit" + fi +done + +bundle="$evidence_dir/vllm-$vllm_commit.bundle" +create_tagged_git_bundle "$source_dir" "$bundle" "$vllm_commit" +git -C "$source_dir" bundle verify "$bundle" >"$evidence_dir/bundle-verify.txt" 2>&1 +git -C "$source_dir" diff --binary --no-ext-diff HEAD \ + >"$evidence_dir/local.diff" +git -C "$source_dir" submodule status --recursive \ + >"$evidence_dir/submodules.txt" +archive="$evidence_dir/vllm-$vllm_commit.tar.gz" +if [[ ! -f "$archive" ]]; then + temporary="$archive.tmp.$$" + git -C "$source_dir" archive --format=tar \ + --prefix="vllm-$vllm_commit/" "$vllm_commit" | gzip -n >"$temporary" + mv "$temporary" "$archive" +fi +archive_sha="$(sha256sum "$archive" | awk '{print $1}')" +printf '%s %s\n' "$archive_sha" "$(basename "$archive")" \ + >"$archive.sha256" +chmod 0600 "$evidence_dir"/* + +set_manifest_value VLLM_BASE_IMAGE_DIGEST "$VLLM_BUILD_BASE_IMAGE_DIGEST" +set_manifest_value VLLM_BUILD_BASE_IMAGE_DIGEST "$VLLM_BUILD_BASE_IMAGE_DIGEST" +set_manifest_value VLLM_FINAL_BASE_IMAGE_DIGEST "$VLLM_FINAL_BASE_IMAGE_DIGEST" +set_manifest_value VLLM_SOURCE_ARCHIVE_SHA256 "$archive_sha" +set_manifest_value VLLM_MAX_WHEEL_SIZE_MB "$VLLM_MAX_WHEEL_SIZE_MB" + +change_id="$(basename "$RUN_ROOT")" +short_commit="${vllm_commit:0:12}" +source_description="$(git -C "$source_dir" describe --tags "$vllm_commit")" +remote_root="/home/alecfong/deepseek-v41/runtime/$change_id" +remote_source="/home/alecfong/deepseek-v41/runtime-source/vllm-$short_commit" +remote_bundle="$remote_root/$(basename "$bundle")" +remote_registry_probe="$remote_root/verify-runtime-registry.py" +runtime_tag="webster/deepseek-v41-vllm:$short_commit" +upstream_tag="webster/deepseek-v41-vllm-upstream:$short_commit" +slice="deepseek-v41-build-${change_id,,}.slice" +service="deepseek-v41-build-${change_id,,}.service" +keeper="deepseek-v41-build-${change_id,,}-keeper.service" + +event runtime-package START "freeze and arm64 image build" +ssh shamu "umask 077; mkdir -p '$remote_root' /home/alecfong/deepseek-v41/runtime-source" +rsync --archive --checksum "$bundle" "shamu:$remote_bundle" +rsync --archive --checksum "$package_root/runtime/Dockerfile" \ + "shamu:$remote_root/Dockerfile" +rsync --archive --checksum "$registry_probe" "shamu:$remote_registry_probe" +ssh shamu bash -s -- "$remote_bundle" "$remote_source" "$vllm_commit" \ + "$source_description" <<'REMOTE' +set -euo pipefail +bundle="$1" +source_dir="$2" +commit="$3" +expected_description="$4" +incomplete="${source_dir}.incomplete" +if [[ -d "$source_dir/.git" ]]; then + [[ "$(git -C "$source_dir" rev-parse HEAD)" == "$commit" ]] + [[ -z "$(git -C "$source_dir" status --porcelain --untracked-files=all)" ]] + ! git -C "$source_dir" symbolic-ref -q HEAD >/dev/null + git -C "$source_dir" fetch --quiet "$bundle" \ + '+refs/tags/*:refs/tags/*' + [[ "$(git -C "$source_dir" describe --tags "$commit")" == \ + "$expected_description" ]] + exit 0 +fi +[[ ! -e "$source_dir" ]] || { printf 'divergent final source path\n' >&2; exit 1; } +[[ ! -e "$incomplete" ]] || { printf 'existing incomplete source path\n' >&2; exit 1; } +git clone --quiet "$bundle" "$incomplete" +git -C "$incomplete" checkout --quiet --detach "$commit" +[[ -z "$(git -C "$incomplete" status --porcelain --untracked-files=all)" ]] +[[ "$(git -C "$incomplete" describe --tags "$commit")" == \ + "$expected_description" ]] +mv "$incomplete" "$source_dir" +REMOTE + +baseline_p90="" +for candidate_window in 15m 1h 6h 24h 7d; do + baseline_file="$RUN_ROOT/metrics/runtime-baseline-$candidate_window.json" + write_glm_window_snapshot "$baseline_file" "$candidate_window" + read -r requests candidate_p90 < <(python3 - "$baseline_file" <<'PY' +import json +import sys + +data = json.load(open(sys.argv[1], encoding="utf-8")) +print(data.get("request_count"), data.get("p90_seconds")) +PY +) + if python3 - "$requests" "$candidate_p90" "$STAGE_BASELINE_MIN_REQUESTS" <<'PY' +import math +import sys + +try: + requests, p90, minimum = map(float, sys.argv[1:]) +except (TypeError, ValueError): + raise SystemExit(1) +raise SystemExit(0 if requests >= minimum and math.isfinite(p90) and p90 > 0 else 1) +PY + then + baseline_p90="$candidate_p90" + break + fi +done +[[ -n "$baseline_p90" ]] || die "no GLM latency baseline for runtime build" + +runner="$remote_root/build.runner.sh" +status_file="$remote_root/build.status" +log_file="$remote_root/build.log" +pid_file="$remote_root/build.pid" +ssh shamu "sudo -n systemctl set-property --runtime '$slice' \ + CPUQuota=400% MemoryMax=96G IOWeight=10" +ssh shamu "sudo -n systemctl stop '$keeper' >/dev/null 2>&1 || true; \ + sudo -n systemd-run --unit='$keeper' --slice='$slice' \ + --property=RuntimeMaxSec=10min /usr/bin/sleep infinity >/dev/null" +cgroup_parent="$(ssh shamu "systemctl show '$slice' --property ControlGroup --value")" +if ! validate_cgroup_parent "$cgroup_parent"; then + ssh shamu "sudo -n systemctl stop '$keeper' >/dev/null 2>&1 || true" + exit 1 +fi +ssh shamu "systemctl show '$slice' -p ControlGroup -p CPUQuotaPerSecUSec \ + -p MemoryMax -p IOWeight" >"$evidence_dir/build-cgroup.txt" +chmod 0600 "$evidence_dir/build-cgroup.txt" +ssh shamu bash -s -- "$runner" <<'REMOTE' +set -euo pipefail +runner="$1" +cat >"$runner" <<'RUNNER' +#!/usr/bin/env bash +set -euo pipefail +source_dir="$1" +wrapper="$2" +commit="$3" +upstream_tag="$4" +runtime_tag="$5" +build_ref="$6" +final_ref="$7" +build_digest="$8" +final_digest="$9" +cuda_version="${10}" +nccl_version="${11}" +arch_list="${12}" +slice="${13}" +cgroup_parent="${14}" +status_file="${15}" +constraints="${16}" +log_file="${17}" +wheel_ceiling="${18}" +exec >"$log_file" 2>&1 + +finish() { + local status=$? + printf 'EXIT=%s\n' "$status" >"$status_file" +} +trap finish EXIT +trap 'exit 143' TERM INT + +sudo -n systemctl show "$slice" -p CPUQuotaPerSecUSec -p MemoryMax -p IOWeight +printf 'VLLM_MAX_SIZE_MB=%s\n' "$wheel_ceiling" + +if docker image inspect "$runtime_tag" >/dev/null 2>&1; then + image_commit="$(docker image inspect "$runtime_tag" \ + --format '{{index .Config.Labels "ai.webster.vllm.commit"}}')" + image_build_digest="$(docker image inspect "$runtime_tag" \ + --format '{{index .Config.Labels "ai.webster.vllm.build-base-digest"}}')" + image_final_digest="$(docker image inspect "$runtime_tag" \ + --format '{{index .Config.Labels "ai.webster.vllm.final-base-digest"}}')" + [[ "$image_commit" == "$commit" ]] + [[ "$image_build_digest" == "$build_digest" ]] + [[ "$image_final_digest" == "$final_digest" ]] +else + buildx_state="$(docker buildx inspect --bootstrap)" + [[ "$buildx_state" != *"Automatically allowed: true"* ]] + # The upstream graph builds the manylinux wheel branch and Ubuntu runtime + # branch concurrently. Both branches mount the same uv distribution cache. + # Under this four-CPU production-safe quota, a native dependency build can + # hold that cache lock longer than uv's 300-second timeout. Complete the + # independent runtime branch first so the full build consumes cached layers + # instead of racing the wheel branch for the same lock. + ionice -c3 nice -n 19 docker buildx build --load --pull \ + --platform linux/arm64 \ + --cgroup-parent "$cgroup_parent" \ + --build-arg CUDA_VERSION="$cuda_version" \ + --build-arg NCCL_VERSION="$nccl_version" \ + --build-arg FINAL_BASE_IMAGE="$final_ref" \ + --target vllm-runtime-base \ + --tag "$upstream_tag-runtime-base" \ + --progress plain \ + --file "$source_dir/docker/Dockerfile" "$source_dir" + ionice -c3 nice -n 19 docker buildx build --load --pull \ + --platform linux/arm64 \ + --cgroup-parent "$cgroup_parent" \ + --build-arg max_jobs=4 \ + --build-arg nvcc_threads=1 \ + --build-arg GIT_REPO_CHECK=1 \ + --build-arg CUDA_VERSION="$cuda_version" \ + --build-arg NCCL_VERSION="$nccl_version" \ + --build-arg BUILD_BASE_IMAGE="$build_ref" \ + --build-arg FINAL_BASE_IMAGE="$final_ref" \ + --build-arg torch_cuda_arch_list="$arch_list" \ + --build-arg VLLM_MAX_SIZE_MB="$wheel_ceiling" \ + --build-arg VLLM_BUILD_COMMIT="$commit" \ + --build-arg VLLM_BUILD_PIPELINE=webster-deepseek-v41 \ + --build-arg VLLM_IMAGE_TAG="$runtime_tag" \ + --target vllm-openai \ + --tag "$upstream_tag" \ + --progress plain \ + --file "$source_dir/docker/Dockerfile" "$source_dir" + ionice -c3 nice -n 19 docker buildx build --load \ + --platform linux/arm64 \ + --cgroup-parent "$cgroup_parent" \ + --build-arg UPSTREAM_VLLM_IMAGE="$upstream_tag" \ + --build-arg VLLM_COMMIT="$commit" \ + --build-arg VLLM_BUILD_BASE_IMAGE_DIGEST="$build_digest" \ + --build-arg VLLM_FINAL_BASE_IMAGE_DIGEST="$final_digest" \ + --tag "$runtime_tag" \ + --progress plain \ + --file "$wrapper" "$(dirname "$wrapper")" +fi + +image_id="$(docker image inspect "$runtime_tag" --format '{{.Id}}')" +architecture="$(docker image inspect "$runtime_tag" --format '{{.Architecture}}')" +image_commit="$(docker image inspect "$runtime_tag" \ + --format '{{index .Config.Labels "ai.webster.vllm.commit"}}')" +[[ "$image_id" =~ ^sha256:[0-9a-f]{64}$ ]] +[[ "$architecture" == arm64 ]] +[[ "$image_commit" == "$commit" ]] +temporary="${constraints}.tmp.$$" +docker run --rm --runtime runc --network none \ + --env NVIDIA_VISIBLE_DEVICES=void --entrypoint python3 "$image_id" \ + -m pip freeze >"$temporary" +[[ -s "$temporary" ]] +grep -Eqi '^vllm(==| @ )' "$temporary" +mv "$temporary" "$constraints" +printf 'IMAGE_ID=%s\n' "$image_id" +RUNNER +chmod 0700 "$runner" +REMOTE + +if ! ssh shamu "rm -f -- '$status_file' '$pid_file'; : >'$log_file'; \ + sudo -n systemctl stop '$service' >/dev/null 2>&1 || true; \ + sudo -n systemctl reset-failed '$service' >/dev/null 2>&1 || true; \ + sudo -n systemd-run --unit='$service' --slice='$slice' --uid=alecfong \ + --property=Nice=19 --property=IOSchedulingClass=idle --collect '$runner' \ + '$remote_source' '$remote_root/Dockerfile' '$vllm_commit' '$upstream_tag' \ + '$runtime_tag' '${VLLM_BUILD_BASE_IMAGE_TAG}@${VLLM_BUILD_BASE_IMAGE_DIGEST}' \ + '${VLLM_FINAL_BASE_IMAGE_TAG}@${VLLM_FINAL_BASE_IMAGE_DIGEST}' \ + '$VLLM_BUILD_BASE_IMAGE_DIGEST' '$VLLM_FINAL_BASE_IMAGE_DIGEST' \ + '$VLLM_CUDA_VERSION' '$VLLM_NCCL_VERSION' '$VLLM_ARM64_ARCH_LIST' '$slice' \ + '$cgroup_parent' '$status_file' '$remote_root/constraints.txt' '$log_file' \ + '$VLLM_MAX_WHEEL_SIZE_MB' \ + >/dev/null; systemctl show '$service' --property MainPID --value >'$pid_file'"; then + ssh shamu "sudo -n systemctl stop '$keeper' >/dev/null 2>&1 || true" + die "failed to start the constrained runtime build service" +fi +ssh shamu "sudo -n systemctl stop '$keeper' >/dev/null 2>&1 || true" + +sleep 3 +if ! ssh shamu "test \"\$(systemctl show '$slice' --property ActiveState --value)\" = active && \ + test \"\$(systemctl show '$slice' --property MemoryCurrent --value)\" -gt 0"; then + ssh shamu "sudo -n systemctl stop '$service' >/dev/null 2>&1 || true" + die "runtime build service did not enter the limited systemd slice" +fi + +monitor_window=0 +consecutive_failures=0 +while ! ssh shamu test -s "$status_file"; do + sleep 60 + monitor_window=$((monitor_window + 1)) + snapshot="$RUN_ROOT/metrics/runtime-build-$(printf '%05d' "$monitor_window").json" + write_glm_window_snapshot "$snapshot" 1m + read -r requests success p90 < <(python3 - "$snapshot" <<'PY' +import json +import sys + +data = json.load(open(sys.argv[1], encoding="utf-8")) +values = [data.get("request_count"), data.get("success_rate"), data.get("p90_seconds")] +print(*(value if value is not None else "null" for value in values)) +PY +) + decision="$(stage_window_decision "$baseline_p90" "$requests" "$success" "$p90")" + if ! check_glm_health; then + decision="BREACH direct-health;$decision" + fi + if [[ "$decision" == BREACH\ * ]]; then + consecutive_failures=$((consecutive_failures + 1)) + else + consecutive_failures=0 + fi + event "runtime-build-window-$monitor_window" MONITOR "$decision" + if (( consecutive_failures >= 2 )); then + ssh shamu "sudo -n systemctl stop '$service' >/dev/null 2>&1 || true" + die "GLM success/latency breached for two runtime-build windows" + fi +done +status="$(ssh shamu "cat '$status_file'")" +ssh shamu "cat '$log_file'" >"$RUN_ROOT/logs/runtime-build.log" +chmod 0600 "$RUN_ROOT/logs/runtime-build.log" +ssh shamu "sudo -n systemctl stop '$service' >/dev/null 2>&1 || true; \ + sudo -n systemctl stop '$slice' >/dev/null 2>&1 || true" +require_eq "runtime build" "EXIT=0" "$status" + +ssh shamu "cat '$remote_root/constraints.txt'" >"$package_root/runtime/constraints.txt" +[[ -s "$package_root/runtime/constraints.txt" ]] || die "empty runtime dependency freeze" + +image_id="$(ssh shamu "docker image inspect '$runtime_tag' --format '{{.Id}}'")" +require_sha256 "runtime image ID" "$image_id" +internal_commit_shamu="$(ssh shamu "docker image inspect '$runtime_tag' --format '{{index .Config.Labels \"ai.webster.vllm.commit\"}}'")" + +image_tar="$remote_root/deepseek-v41-vllm-$short_commit.tar" +tar_status="$remote_root/save.status" +tar_log="$remote_root/save.log" +tar_pid="$remote_root/save.pid" + +monitor_runtime_job() { + local host="$1" status_path="$2" pid_path="$3" label="$4" + local tick=0 consecutive_health_bad=0 consecutive_metric_bad=0 + local job_pid snapshot requests success p90 decision + while ! ssh "$host" test -s "$status_path"; do + sleep 30 + tick=$((tick + 1)) + if ! check_glm_health; then + consecutive_health_bad=$((consecutive_health_bad + 1)) + elif ((tick % 2 == 0)); then + consecutive_health_bad=0 + snapshot="$RUN_ROOT/metrics/$label-$(printf '%05d' "$((tick / 2))").json" + write_glm_window_snapshot "$snapshot" 1m + read -r requests success p90 < <(python3 - "$snapshot" <<'PY' +import json +import sys + +data = json.load(open(sys.argv[1], encoding="utf-8")) +values = [data.get("request_count"), data.get("success_rate"), data.get("p90_seconds")] +print(*(value if value is not None else "null" for value in values)) +PY +) + decision="$(stage_window_decision "$baseline_p90" "$requests" "$success" "$p90")" + if [[ "$decision" == BREACH\ * ]]; then + consecutive_metric_bad=$((consecutive_metric_bad + 1)) + else + consecutive_metric_bad=0 + fi + event "$label-window-$((tick / 2))" MONITOR "$decision" + else + consecutive_health_bad=0 + fi + if ((consecutive_health_bad >= 2 || consecutive_metric_bad >= 2)); then + job_pid="$(ssh "$host" "cat '$pid_path'")" + ssh "$host" "/bin/kill -TERM -- '-$job_pid' 2>/dev/null || true" + die "GLM health/latency breached while $label was active" + fi + done + check_glm_health || + die "GLM health failed after $label" +} + +if ! ssh shamu test -f "$image_tar"; then + if ssh shamu test -e "$image_tar.tmp"; then + die "incomplete saved-image archive exists; preserving it for diagnosis" + fi + ssh shamu "rm -f '$tar_status'; nohup setsid bash -c 'set -o pipefail; \ + ionice -c3 nice -n 19 docker image save \"$runtime_tag\" --output \"$image_tar.tmp\" && \ + mv \"$image_tar.tmp\" \"$image_tar\"; s=\$?; printf \"EXIT=%s\\n\" \"\$s\" >\"$tar_status\"; exit \"\$s\"' \ + >'$tar_log' 2>&1 '$tar_pid'" + monitor_runtime_job shamu "$tar_status" "$tar_pid" runtime-save + require_eq "runtime image save" "EXIT=0" "$(ssh shamu "cat '$tar_status'")" +fi +tar_sha="$(ssh shamu "sha256sum '$image_tar' | awk '{print \$1}'")" +[[ "$tar_sha" =~ ^[0-9a-f]{64}$ ]] || die "invalid saved-image checksum" +printf '%s %s\n' "$tar_sha" "$(basename "$image_tar")" \ + >"$evidence_dir/image-tar.sha256" +chmod 0600 "$evidence_dir/image-tar.sha256" + +tilikum_dir="/home/alecfong/deepseek-v41/runtime/$change_id" +ssh tilikum "umask 077; mkdir -p '$tilikum_dir'" +tilikum_tar="$tilikum_dir/$(basename "$image_tar")" +if ssh tilikum test -f "$tilikum_tar"; then + existing_tilikum_sha="$(ssh tilikum "sha256sum '$tilikum_tar' | awk '{print \$1}'")" + require_eq "existing Tilikum image archive" "$tar_sha" "$existing_tilikum_sha" +else + copy_status="$remote_root/copy.status" + copy_log="$remote_root/copy.log" + copy_pid="$remote_root/copy.pid" + ssh shamu "rm -f '$copy_status'; nohup setsid bash -c 'set -o pipefail; \ + env -u SSH_AUTH_SOCK rsync --archive --checksum --bwlimit=250000 \"$image_tar\" \"10.10.1.2:$tilikum_dir/\"; \ + s=\$?; printf \"EXIT=%s\\n\" \"\$s\" >\"$copy_status\"; exit \"\$s\"' \ + >'$copy_log' 2>&1 '$copy_pid'" + monitor_runtime_job shamu "$copy_status" "$copy_pid" runtime-copy + require_eq "runtime image copy" "EXIT=0" "$(ssh shamu "cat '$copy_status'")" +fi +tilikum_sha="$(ssh tilikum "sha256sum '$tilikum_dir/$(basename "$image_tar")' | awk '{print \$1}'")" +require_eq "saved-image transfer checksum" "$tar_sha" "$tilikum_sha" +load_status="$tilikum_dir/load.status" +load_log="$tilikum_dir/load.log" +load_pid="$tilikum_dir/load.pid" +ssh tilikum "rm -f '$load_status'; nohup setsid bash -c 'set -o pipefail; \ + sudo -n docker image load --input \"$tilikum_tar\" >/dev/null; \ + s=\$?; printf \"EXIT=%s\\n\" \"\$s\" >\"$load_status\"; exit \"\$s\"' \ + >'$load_log' 2>&1 '$load_pid'" +monitor_runtime_job tilikum "$load_status" "$load_pid" runtime-load +require_eq "runtime image load" "EXIT=0" "$(ssh tilikum "cat '$load_status'")" +tilikum_image_id="$(ssh tilikum "sudo -n docker image inspect '$runtime_tag' --format '{{.Id}}'")" +internal_commit_tilikum="$(ssh tilikum "sudo -n docker image inspect '$runtime_tag' --format '{{index .Config.Labels \"ai.webster.vllm.commit\"}}'")" + +help_visible_devices="void" +help_target_device="cpu" +help_selector="engram-config" +validate_help_probe "$help_visible_devices" "$help_target_device" "$help_selector" +help_text="$(ssh shamu "docker run --rm --runtime runc --network none \ + --env NVIDIA_VISIBLE_DEVICES='$help_visible_devices' \ + --env VLLM_TARGET_DEVICE='$help_target_device' \ + --entrypoint vllm '$image_id' serve --help='$help_selector' 2>&1")" +registry_text="$(ssh shamu "docker run --rm --runtime runc --network none \ + --env NVIDIA_VISIBLE_DEVICES=void \ + --volume '$remote_registry_probe:/opt/webster/verify-runtime-registry.py:ro' \ + --entrypoint python3 '$image_id' /opt/webster/verify-runtime-registry.py \ + --require-model DeepseekV41ForCausalLM")" +config_file="$evidence_dir/checkpoint-config.json" +python3 - "$CHECKPOINT_REPO" "$CHECKPOINT_REVISION" "$config_file" <<'PY' +import json +import os +import sys +import urllib.parse +import urllib.request + +repository, revision, output = sys.argv[1:] +url = ( + "https://huggingface.co/" + + repository + + "/resolve/" + + urllib.parse.quote(revision, safe="") + + "/config.json" +) +request = urllib.request.Request(url, headers={"User-Agent": "webster-runtime-gate"}) +with urllib.request.urlopen(request, timeout=60) as response: + payload = json.load(response) +temporary = output + f".tmp.{os.getpid()}" +with open(temporary, "w", encoding="utf-8") as stream: + json.dump(payload, stream, indent=2, sort_keys=True) + stream.write("\n") +os.chmod(temporary, 0o600) +os.replace(temporary, output) +PY +engram_layers="$(python3 - "$config_file" <<'PY' +import json +import sys + +payload = json.load(open(sys.argv[1], encoding="utf-8")) +layers = payload.get("text_config", payload).get("engram_layer_ids") +if not isinstance(layers, list) or not layers: + raise SystemExit("checkpoint has no engram_layer_ids") +print(",".join(str(layer) for layer in layers)) +PY +)" + +validate_runtime_gate arm64 linux/arm64 linux/arm64 1 \ + "$image_id"$'\n'"$tilikum_image_id" \ + "$internal_commit_shamu"$'\n'"$internal_commit_tilikum" \ + "$help_text" "$registry_text" "$engram_layers" + +set_manifest_value VLLM_IMAGE_ID "$image_id" +set_manifest_value VLLM_IMAGE_TAR_SHA256 "$tar_sha" +event runtime-package GO \ + "arm64 image=$image_id source=$vllm_commit engram baseline verified" +printf 'GO runtime-package image=%s\n' "$image_id" diff --git a/webster/deepseek-v41/scripts/build-weka-corpus-manifest.py b/webster/deepseek-v41/scripts/build-weka-corpus-manifest.py new file mode 100755 index 0000000..c17e886 --- /dev/null +++ b/webster/deepseek-v41/scripts/build-weka-corpus-manifest.py @@ -0,0 +1,126 @@ +#!/usr/bin/env python3 +"""Create a private split-trace manifest from one pinned Weka source JSONL.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import re +import sys +import tempfile +from pathlib import Path +from typing import BinaryIO + + +SHA256 = re.compile(r"^[0-9a-f]{64}$") +REVISION = re.compile(r"^[0-9a-f]{40}$") + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--dataset", required=True, type=Path) + parser.add_argument("--output", required=True, type=Path) + parser.add_argument("--repository", required=True) + parser.add_argument("--revision", required=True) + parser.add_argument("--source-sha256", required=True) + parser.add_argument("--source-size", required=True, type=int) + parser.add_argument("--trace-count", required=True, type=int) + return parser.parse_args() + + +def sha256_stream(handle: BinaryIO) -> str: + digest = hashlib.sha256() + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def fail(message: str) -> None: + raise RuntimeError(message) + + +def build(args: argparse.Namespace) -> dict[str, object]: + if not args.repository or REVISION.fullmatch(args.revision) is None: + fail("Weka repository or revision is malformed") + if SHA256.fullmatch(args.source_sha256) is None: + fail("Weka source hash is malformed") + if args.source_size <= 0 or args.trace_count <= 0: + fail("Weka source size or trace count is invalid") + if not args.dataset.is_dir() or args.dataset.is_symlink(): + fail("Weka dataset must be a regular non-symlink directory") + dataset = args.dataset.resolve(strict=True) + output = args.output.absolute() + if output.parent.resolve(strict=True) != dataset.parent or output.is_symlink(): + fail("Weka corpus manifest output must be beside the trace directory") + source = dataset.parent / "traces.jsonl" + if not source.is_file() or source.is_symlink(): + fail("pinned Weka source JSONL is missing") + if source.stat().st_size != args.source_size: + fail("pinned Weka source size differs") + with source.open("rb") as handle: + if sha256_stream(handle) != args.source_sha256: + fail("pinned Weka source hash differs") + + expected_names = [f"trace-{index:03d}.json" for index in range(args.trace_count)] + actual_names = sorted(path.name for path in dataset.glob("*.json")) + if actual_names != expected_names: + fail("Weka dataset does not contain the exact JSON trace set") + files: dict[str, str] = {} + with source.open("rb") as source_handle: + for name in expected_names: + source_line = source_handle.readline() + if not source_line: + fail("pinned Weka source has fewer traces than expected") + path = dataset / name + if not path.is_file() or path.is_symlink(): + fail(f"Weka split trace is missing or unsafe: {name}") + with path.open("rb") as trace_handle: + digest = sha256_stream(trace_handle) + if hashlib.sha256(source_line).hexdigest() != digest: + fail(f"Weka split trace does not match pinned source: {name}") + files[name] = digest + if source_handle.readline(): + fail("pinned Weka source has more traces than expected") + + return { + "schema_version": 1, + "repository": args.repository, + "revision": args.revision, + "source_sha256": args.source_sha256, + "source_size": args.source_size, + "trace_count": args.trace_count, + "full_subagents": True, + "files": files, + } + + +def write_private_json(path: Path, value: object) -> None: + descriptor, temporary_name = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent) + temporary = Path(temporary_name) + try: + with os.fdopen(descriptor, "w", encoding="utf-8") as handle: + json.dump(value, handle, indent=2, sort_keys=True) + handle.write("\n") + os.chmod(temporary, 0o600) + os.replace(temporary, path) + os.chmod(path, 0o600) + finally: + temporary.unlink(missing_ok=True) + + +def main() -> int: + args = parse_args() + try: + value = build(args) + write_private_json(args.output, value) + except (OSError, RuntimeError, UnicodeError, ValueError) as error: + print(f"ERROR: {error}", file=sys.stderr) + return 1 + print(f"Weka corpus manifest written: {args.output}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/webster/deepseek-v41/scripts/common.sh b/webster/deepseek-v41/scripts/common.sh new file mode 100755 index 0000000..c692c5e --- /dev/null +++ b/webster/deepseek-v41/scripts/common.sh @@ -0,0 +1,422 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Cluster commands must never inherit or forward an operator's SSH agent. The +# wrappers cover direct calls from every package script; composed remote paths +# additionally scrub the remote shell before starting their nested transport. +unset SSH_AUTH_SOCK +ssh() { + if [[ "${WEBSTER_LOCAL_SHAMU:-0}" == 1 && "${1:-}" == shamu ]]; then + if [[ "${WEBSTER_WEKA_TEST_MODE:-0}" != 1 && "$(hostname -s)" != shamu ]]; then + printf 'ERROR: local Shamu command mode may run only on shamu\n' >&2 + return 1 + fi + shift + if (( $# == 1 )); then + env -u SSH_AUTH_SOCK bash -c "$1" + else + env -u SSH_AUTH_SOCK "$@" + fi + return + fi + env -u SSH_AUTH_SOCK "$(type -P ssh)" -o ForwardAgent=no "$@" +} +scp() { + env -u SSH_AUTH_SOCK "$(type -P scp)" -o ForwardAgent=no "$@" +} +rsync() { + env -u SSH_AUTH_SOCK "$(type -P rsync)" "$@" +} +brev() { + env -u SSH_AUTH_SOCK "$(type -P brev)" "$@" +} + +canonical_path() { + python3 - "$1" <<'PY' +from pathlib import Path +import sys + +print(Path(sys.argv[1]).resolve(strict=False)) +PY +} + +canonical_existing_path() { + python3 - "$1" <<'PY' +from pathlib import Path +import sys + +try: + print(Path(sys.argv[1]).resolve(strict=True)) +except (OSError, RuntimeError): + raise SystemExit(1) +PY +} + +file_mode() { + python3 - "$1" <<'PY' +from pathlib import Path +import stat +import sys + +print(f"{stat.S_IMODE(Path(sys.argv[1]).stat().st_mode):o}") +PY +} + +file_uid() { + python3 - "$1" <<'PY' +from pathlib import Path +import sys + +print(Path(sys.argv[1]).stat().st_uid) +PY +} + +SHAMU_NETBIRD="100.73.140.127" +SHAMU_LAN="192.168.1.75" +SHAMU_RAIL="10.10.1.1" +TILIKUM_NETBIRD="100.73.89.150" +TILIKUM_RAIL="10.10.1.2" +NCCL_IFACE="enP1p3s0f1np1" +NCCL_HCA="mlx5_1" +CANARY_PORT="8000" +MIN_FREE_BEFORE_STAGE_BYTES="805306368000" +MIN_FREE_AFTER_STAGE_BYTES="214748364800" +CHECKPOINT_REPO="deepseek-ai/DeepSeek-V4.1-Flash" +CHECKPOINT_REVISION="dba1be0a40aa45a94ad051997016db3960a90277" +CHECKPOINT_SHARDS="48" +CHECKPOINT_TENSOR_BYTES="510286023000" +CHECKPOINT_SHARD_FILE_BYTES="510296708312" +VLLM_BUILD_BASE_IMAGE_TAG="pytorch/manylinuxaarch64-builder:cuda13.0-b8b5f17a7d9ccfc25bbc5cf17b3fcea12964a042" +VLLM_BUILD_BASE_IMAGE_DIGEST="sha256:994bed2b225a9ff0f6fbe85c85fe84fbeac9031bb909442e18178484798529df" +VLLM_FINAL_BASE_IMAGE_TAG="nvidia/cuda:13.0.3-base-ubuntu24.04" +VLLM_FINAL_BASE_IMAGE_DIGEST="sha256:56d9d8183e2181a20be6b0d3801d1f056a0e75c17706df939ba207b126e1cb9c" +VLLM_CUDA_VERSION="13.0.3" +VLLM_NCCL_VERSION="2.30.7" +VLLM_ARM64_ARCH_LIST="9.0 10.0 11.0 12.0" +VLLM_MAX_WHEEL_SIZE_MB="700" +PROMETHEUS_URL="http://100.73.140.127:9095" +GLM_METRICS_INSTANCE="100.73.140.127:8000" +BAKER_RANK0_HOST="baker-spark-1" +BAKER_RANK1_HOST="baker-spark-2" +BAKER_RANK0_NETBIRD="100.73.127.129" +BAKER_API_PORT="8888" +BAKER_CONTAINER="deepseek-v4-flash-vllm-dspark-1" +STAGE_MIN_SUCCESS_RATE="0.99" +STAGE_LATENCY_MULTIPLIER="2.0" +STAGE_BASELINE_MIN_REQUESTS="20" +if [[ "${WEBSTER_PREFLIGHT_TEST_MODE:-0}" == "1" || + "${WEBSTER_STAGE_TEST_MODE:-0}" == "1" || + "${WEBSTER_RUNTIME_TEST_MODE:-0}" == "1" || + "${WEBSTER_LIFECYCLE_TEST_MODE:-0}" == "1" || + "${WEBSTER_WEKA_TEST_MODE:-0}" == "1" || + "${WEBSTER_ACCEPTANCE_TEST_MODE:-0}" == "1" ]]; then + RUNS_ROOT="${TEST_RUNS_ROOT:-/home/ubuntu/deepseek-v41-runs}" +elif [[ -n "${WEBSTER_RUNS_ROOT:-}" ]]; then + [[ -d "$WEBSTER_RUNS_ROOT" && ! -L "$WEBSTER_RUNS_ROOT" ]] || { + printf 'ERROR: Webster runs root override must be an existing non-symlink directory\n' >&2 + return 1 2>/dev/null || exit 1 + } + RUNS_ROOT="$(canonical_existing_path "$WEBSTER_RUNS_ROOT")" +else + RUNS_ROOT="/home/ubuntu/deepseek-v41-runs" +fi + +GLM_CONTAINER="glm52-full-mtp" +DEEPSEEK_CONTAINER="deepseek-v41-flash-tp2" +DEEPSEEK_CHECKPOINT_DIR="DeepSeek-V4.1-Flash-${CHECKPOINT_REVISION:0:10}" +DEEPSEEK_REMOTE_ROOT="/home/alecfong/deepseek-v41" +DEEPSEEK_CHECKPOINT_PATH="$DEEPSEEK_REMOTE_ROOT/models/$DEEPSEEK_CHECKPOINT_DIR" +DEEPSEEK_CACHE_PATH="$DEEPSEEK_REMOTE_ROOT/cache" +DEEPSEEK_API_ENV_FILE="$DEEPSEEK_REMOTE_ROOT/credentials/vllm-api.env" +DEEPSEEK_SERVED_MODEL="deepseek-ai/DeepSeek-V4.1-Flash" + +die() { + printf 'ERROR: %s\n' "$*" >&2 + exit 1 +} + +validate_phase() { + case "${1:-}" in + baseline|stage|alias|canary|publish) ;; + *) die "invalid phase: ${1:-}" ;; + esac +} + +validate_run_root() { + local requested resolved relative + requested="${1:-}" + [[ -n "$requested" ]] || die "run root is required" + resolved="$(canonical_path "$requested")" + case "$resolved" in + "$RUNS_ROOT"/*) ;; + *) die "run root must resolve beneath $RUNS_ROOT" ;; + esac + relative="${resolved#"$RUNS_ROOT"/}" + [[ "$relative" != */* && "$relative" =~ ^[0-9]{8}T[0-9]{6}Z$ ]] || + die "run root must end in one UTC change id" + printf '%s\n' "$resolved" +} + +init_run_root() { + RUN_ROOT="$(validate_run_root "$1")" + export RUN_ROOT + umask 077 + mkdir -p -- "$RUN_ROOT" "$RUN_ROOT/baseline" "$RUN_ROOT/logs" \ + "$RUN_ROOT/metrics" "$RUN_ROOT/aiperf" + chmod 0700 "$RUN_ROOT" "$RUN_ROOT/baseline" "$RUN_ROOT/logs" \ + "$RUN_ROOT/metrics" "$RUN_ROOT/aiperf" + touch "$RUN_ROOT/events.tsv" "$RUN_ROOT/summary.md" + chmod 0600 "$RUN_ROOT/events.tsv" "$RUN_ROOT/summary.md" + if [[ ! -f "$RUN_ROOT/manifest.env" ]]; then + { + printf 'CHECKPOINT_REPO=%s\n' "$CHECKPOINT_REPO" + printf 'CHECKPOINT_REVISION=%s\n' "$CHECKPOINT_REVISION" + printf 'CHECKPOINT_SHARDS=%s\n' "$CHECKPOINT_SHARDS" + printf 'CHECKPOINT_TENSOR_BYTES=%s\n' "$CHECKPOINT_TENSOR_BYTES" + printf 'CHECKPOINT_SHARD_FILE_BYTES=%s\n' "$CHECKPOINT_SHARD_FILE_BYTES" + printf 'VLLM_COMMIT=\nVLLM_BASE_IMAGE_DIGEST=\n' + printf 'VLLM_BUILD_BASE_IMAGE_DIGEST=\nVLLM_FINAL_BASE_IMAGE_DIGEST=\n' + printf 'VLLM_SOURCE_ARCHIVE_SHA256=\nVLLM_IMAGE_ID=\n' + printf 'VLLM_MAX_WHEEL_SIZE_MB=%s\n' "$VLLM_MAX_WHEEL_SIZE_MB" + printf 'VLLM_IMAGE_TAR_SHA256=\nAIPERF_COMMIT=\nWEKA_REPOSITORY=\nWEKA_REVISION=\n' + } >"$RUN_ROOT/manifest.env" + chmod 0600 "$RUN_ROOT/manifest.env" + fi +} + +event() { + local milestone="$1" decision="$2" reason="$3" timestamp + timestamp="$(date -u +%Y-%m-%dT%H:%M:%SZ)" + reason="${reason//$'\t'/ }" + reason="${reason//$'\n'/ }" + printf '%s\t%s\t%s\t%s\n' "$timestamp" "$milestone" "$decision" "$reason" \ + >>"$RUN_ROOT/events.tsv" +} + +redact_stream() { + sed -E \ + -e 's/(Authorization:[[:space:]]*Bearer)[[:space:]]+[^[:space:]]+/\1 [REDACTED]/Ig' \ + -e "s/((api[_-]?keys?|master[_-]?key|password|secret|token)[\"']?[[:space:]]*[:=][[:space:]]*)[^,[:space:]\"']+/\\1[REDACTED]/Ig" \ + -e 's/sk-[A-Za-z0-9._-]{8,}/[REDACTED-KEY]/g' +} + +capture() { + local name="$1" output temporary status suffix=2 + shift + [[ "$name" =~ ^[A-Za-z0-9._-]+$ ]] || die "unsafe capture name: $name" + output="$RUN_ROOT/logs/$name.log" + while [[ -e "$output" || -L "$output" ]]; do + output="$RUN_ROOT/logs/$name-$suffix.log" + ((suffix += 1)) + done + temporary="$output.tmp.$$" + set +e + "$@" >"$temporary" 2>&1 + status=$? + set -e + redact_stream <"$temporary" >"$output" + rm -f -- "$temporary" + chmod 0600 "$output" + printf 'capture %s status=%s artifact=%s\n' "$name" "$status" "$output" + return "$status" +} + +check_glm_health() { + curl -fsS --connect-timeout 3 --max-time 5 \ + "http://$SHAMU_NETBIRD:$CANARY_PORT/health" >/dev/null +} + +remote_available_bytes() { + local node="$1" requested_path="$2" output available + if ! output="$( + ssh "$node" df -B1 --output=avail "$requested_path" 2>/dev/null + )"; then + output="$(ssh "$node" df -B1 --output=avail /home/alecfong)" + fi + available="$(awk 'NR == 2 {print $1}' <<<"$output")" + [[ "$available" =~ ^[0-9]+$ ]] || + die "remote free-space query returned a non-integer for $node" + printf '%s\n' "$available" +} + +require_eq() { + local label="$1" expected="$2" actual="$3" + [[ "$actual" == "$expected" ]] || + die "$label mismatch (expected $expected, got $actual)" +} + +require_ge() { + local label="$1" minimum="$2" actual="$3" + [[ "$actual" =~ ^[0-9]+$ ]] || die "$label is not an integer" + (( actual >= minimum )) || die "$label is below required floor" +} + +create_tagged_git_bundle() { + local source_dir="$1" output="$2" expected_commit="$3" + local temporary bundle_head tag_count + require_eq "source bundle HEAD" "$expected_commit" \ + "$(git -C "$source_dir" rev-parse HEAD)" + git -C "$source_dir" describe --tags "$expected_commit" >/dev/null + if [[ -f "$output" ]]; then + git -C "$source_dir" bundle verify "$output" >/dev/null 2>&1 + bundle_head="$(git bundle list-heads "$output" | awk '$2 == "HEAD" {print $1}')" + require_eq "existing source bundle HEAD" "$expected_commit" "$bundle_head" + tag_count="$(git bundle list-heads "$output" | awk '$2 ~ /^refs\/tags\// {count++} END {print count+0}')" + (( tag_count > 0 )) || die "existing source bundle contains no tags" + return 0 + fi + temporary="$output.tmp.$$" + git -C "$source_dir" bundle create "$temporary" HEAD --tags + bundle_head="$(git bundle list-heads "$temporary" | awk '$2 == "HEAD" {print $1}')" + require_eq "new source bundle HEAD" "$expected_commit" "$bundle_head" + tag_count="$(git bundle list-heads "$temporary" | awk '$2 ~ /^refs\/tags\// {count++} END {print count+0}')" + (( tag_count > 0 )) || die "new source bundle contains no tags" + mv "$temporary" "$output" +} + +write_glm_window_snapshot() { + local output="$1" window="$2" + [[ "$window" =~ ^[1-9][0-9]*[mhd]$ ]] || die "invalid Prometheus window: $window" + python3 - "$PROMETHEUS_URL" "$GLM_METRICS_INSTANCE" "$window" "$output" <<'PY' +import datetime +import json +import math +import os +import sys +import urllib.parse +import urllib.request + +prometheus_url, instance, window, output = sys.argv[1:] +selector = f'instance="{instance}"' +queries = { + "request_count": ( + f'sum(increase(vllm:request_success_total{{{selector}}}[{window}]))' + ), + "successful_requests": ( + 'sum(increase(vllm:request_success_total{' + f'{selector},finished_reason!~"abort|error"}}[{window}]))' + ), + "p90_seconds": ( + 'histogram_quantile(0.90, sum by (le) ' + f'(rate(vllm:e2e_request_latency_seconds_bucket{{{selector}}}[{window}])))' + ), +} + + +def query(expression): + url = prometheus_url.rstrip("/") + "/api/v1/query?query=" + urllib.parse.quote(expression) + with urllib.request.urlopen(url, timeout=15) as response: + payload = json.load(response) + if payload.get("status") != "success": + raise RuntimeError(f"Prometheus query failed: {payload.get('error', 'unknown error')}") + rows = payload.get("data", {}).get("result", []) + if len(rows) != 1: + return None + raw = rows[0].get("value", [None, None])[1] + try: + value = float(raw) + except (TypeError, ValueError): + return None + return value if math.isfinite(value) else None + + +values = {name: query(expression) for name, expression in queries.items()} +request_count = values["request_count"] +successful = values["successful_requests"] +success_rate = None +if request_count is not None and request_count > 0 and successful is not None: + success_rate = min(1.0, max(0.0, successful / request_count)) +result = { + "captured_at": datetime.datetime.now(datetime.timezone.utc).isoformat(), + "instance": instance, + "window": window, + "request_count": request_count, + "successful_requests": successful, + "success_rate": success_rate, + "p90_seconds": values["p90_seconds"], + "queries": queries, +} +temporary = output + f".tmp.{os.getpid()}" +with open(temporary, "w", encoding="utf-8") as handle: + json.dump(result, handle, indent=2, sort_keys=True) + handle.write("\n") +os.chmod(temporary, 0o600) +os.replace(temporary, output) +PY +} + +stage_window_decision() { + local baseline_p90="$1" request_count="$2" success_rate="$3" p90="$4" + python3 - "$baseline_p90" "$request_count" "$success_rate" "$p90" \ + "$STAGE_MIN_SUCCESS_RATE" "$STAGE_LATENCY_MULTIPLIER" <<'PY' +import math +import sys + +baseline_raw, requests_raw, success_raw, p90_raw, minimum_raw, multiplier_raw = sys.argv[1:] + + +def finite_number(raw, label): + try: + value = float(raw) + except ValueError as error: + raise SystemExit(f"invalid {label}: {raw}") from error + if not math.isfinite(value): + raise SystemExit(f"invalid {label}: {raw}") + return value + + +baseline = finite_number(baseline_raw, "baseline p90") +requests = finite_number(requests_raw, "request count") +minimum = finite_number(minimum_raw, "minimum success rate") +multiplier = finite_number(multiplier_raw, "latency multiplier") +if baseline <= 0 or requests < 0 or not 0 < minimum <= 1 or multiplier <= 0: + raise SystemExit("invalid staging health threshold") +if requests == 0: + print("OK no-completed-requests") + raise SystemExit(0) +if success_raw == "null" or p90_raw == "null": + print("BREACH incomplete-window-metrics") + raise SystemExit(0) +success = finite_number(success_raw, "success rate") +p90 = finite_number(p90_raw, "p90") +reasons = [] +if success < minimum: + reasons.append(f"success-rate={success:.6f}<{minimum:.6f}") +if p90 > baseline * multiplier: + reasons.append(f"p90={p90:.6f}>{baseline * multiplier:.6f}") +if reasons: + print("BREACH " + ",".join(reasons)) +else: + print(f"OK success-rate={success:.6f},p90={p90:.6f}") +PY +} + +capture_credential_metadata() { + local host="$1" path="$2" + ssh "$host" stat -c 'mode=%a,owner=%U,size=%s,path=%n' -- "$path" + ssh "$host" sha256sum -- "$path" +} + +require_file_mode_600() { + local host="$1" path="$2" metadata mode + if [[ "${WEBSTER_PREFLIGHT_TEST_MODE:-0}" == "1" ]]; then + mode="${TEST_CREDENTIAL_MODE:-0600}" + else + metadata="$(ssh "$host" stat -c '%a:%U:%s' -- "$path")" || + die "cannot inspect credential metadata on $host" + mode="${metadata%%:*}" + capture "credential-${host}" capture_credential_metadata "$host" "$path" + fi + mode="${mode#0}" + require_eq "$host credential mode" "600" "$mode" +} + +write_summary_header() { + { + printf '# Cutover evidence %s\n\n' "$(basename "$RUN_ROOT")" + printf -- '- Phase: `%s`\n' "$1" + printf -- '- Created: `%s`\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" + printf -- '- Full evidence is mode-restricted under this run root.\n' + } >"$RUN_ROOT/summary.md" + chmod 0600 "$RUN_ROOT/summary.md" +} diff --git a/webster/deepseek-v41/scripts/contract-probe.py b/webster/deepseek-v41/scripts/contract-probe.py new file mode 100755 index 0000000..c652c37 --- /dev/null +++ b/webster/deepseek-v41/scripts/contract-probe.py @@ -0,0 +1,408 @@ +#!/usr/bin/env python3 +"""Exercise the GLM compatibility contract without placing credentials in argv.""" + +from __future__ import annotations + +import argparse +import importlib.util +import json +import os +import re +import stat +import sys +import tempfile +import time +import urllib.error +import urllib.request +from pathlib import Path +from typing import Any + + +HERE = Path(__file__).resolve().parent +PACKAGE_ROOT = HERE.parent +GUARD_MODULE = PACKAGE_ROOT / "litellm" / "glm52_contract_guard.py" + + +class ProbeError(RuntimeError): + """A probe failure safe to record without request credentials.""" + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--base-url", required=True) + parser.add_argument("--models", nargs="+", required=True) + parser.add_argument("--key-file", type=Path, required=True) + parser.add_argument("--expected-response-model", action="append", default=[]) + parser.add_argument("--backend-metrics-before", type=Path) + parser.add_argument("--output-json", type=Path, required=True) + parser.add_argument("--tokenizer-dir", type=Path) + parser.add_argument("--skip-boundary", action="store_true") + return parser.parse_args() + + +def read_key(path: Path) -> str: + if not path.is_file() or path.is_symlink(): + raise ProbeError("key file must be a regular file") + if stat.S_IMODE(path.stat().st_mode) != 0o600: + raise ProbeError("key file mode must be 0600") + key = path.read_text(encoding="utf-8").strip() + if not key: + raise ProbeError("key file is empty") + return key + + +def expected_models(values: list[str], models: list[str]) -> dict[str, str]: + result: dict[str, str] = {} + for value in values: + public, separator, response = value.partition("=") + if not separator or not public or not response or public in result: + raise ProbeError("expected response models must use unique PUBLIC=RESPONSE pairs") + result[public] = response + missing = [model for model in models if model not in result] + if missing: + raise ProbeError("every probed model needs an expected response-model mapping") + return result + + +def request_json( + url: str, + key: str, + case: str, + body: dict[str, Any], + *, + expected_status: int = 200, +) -> tuple[int, Any, float]: + request = urllib.request.Request( + url, + data=json.dumps(body, separators=(",", ":")).encode("utf-8"), + headers={ + "Authorization": f"Bearer {key}", + "Content-Type": "application/json", + "X-Contract-Probe-Case": case, + }, + method="POST", + ) + started = time.monotonic() + try: + with urllib.request.urlopen(request, timeout=600) as response: + raw = response.read() + status = response.status + content_type = response.headers.get("Content-Type", "") + except urllib.error.HTTPError as error: + raw = error.read() + status = error.code + content_type = error.headers.get("Content-Type", "") + elapsed = time.monotonic() - started + if status != expected_status: + raise ProbeError(f"{case} returned HTTP {status}, expected {expected_status}") + if "text/event-stream" in content_type: + return status, raw.decode("utf-8", errors="replace"), elapsed + try: + return status, json.loads(raw), elapsed + except json.JSONDecodeError as error: + raise ProbeError(f"{case} did not return JSON") from error + + +def response_record(case: str, payload: dict[str, Any], elapsed: float) -> dict[str, Any]: + choices = payload.get("choices") + if not isinstance(choices, list) or not choices or not isinstance(choices[0], dict): + raise ProbeError(f"{case} response has no choice") + return { + "case": case, + "status": 200, + "elapsed_seconds": round(elapsed, 6), + "response_model": payload.get("model"), + "finish_reason": choices[0].get("finish_reason"), + } + + +def check_stream(case: str, payload: str, expected_model: str) -> dict[str, Any]: + done = False + models: set[str] = set() + finish_reasons: list[str] = [] + for line in payload.splitlines(): + if not line.startswith("data: "): + continue + data = line.removeprefix("data: ") + if data == "[DONE]": + done = True + continue + try: + chunk = json.loads(data) + except json.JSONDecodeError as error: + raise ProbeError("stream returned malformed JSON data") from error + if isinstance(chunk.get("model"), str): + models.add(chunk["model"]) + for choice in chunk.get("choices") or []: + if choice.get("finish_reason"): + finish_reasons.append(choice["finish_reason"]) + if not done or models != {expected_model}: + raise ProbeError("stream did not preserve the expected response model and DONE marker") + return { + "case": case, + "status": 200, + "response_model": expected_model, + "finish_reason": finish_reasons[-1] if finish_reasons else None, + } + + +def tool() -> dict[str, Any]: + return { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get current weather.", + "parameters": { + "type": "object", + "properties": {"city": {"type": "string"}}, + "required": ["city"], + }, + }, + } + + +def run_model(base_url: str, key: str, model: str, expected_model: str) -> list[dict[str, Any]]: + endpoint = base_url.rstrip("/") + "/chat/completions" + cases: list[tuple[str, dict[str, Any]]] = [ + ( + "plain", + {"model": model, "messages": [{"role": "user", "content": "Reply OK."}], "max_tokens": 8, "temperature": 0}, + ), + ( + "reasoning-none", + {"model": model, "messages": [{"role": "user", "content": "Reply OK."}], "max_completion_tokens": 16, "reasoning_effort": "none", "temperature": 0}, + ), + ( + "reasoning-on", + {"model": model, "messages": [{"role": "user", "content": "What is 2+2?"}], "max_completion_tokens": 32, "reasoning_effort": "medium", "temperature": 0}, + ), + ( + "tool-auto", + {"model": model, "messages": [{"role": "user", "content": "Weather in San Francisco?"}], "tools": [tool()], "tool_choice": "auto", "max_tokens": 64, "temperature": 0}, + ), + ( + "tool-result", + { + "model": model, + "messages": [ + {"role": "user", "content": "Weather in San Francisco?"}, + {"role": "assistant", "content": "", "tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "get_weather", "arguments": '{"city":"San Francisco"}'}}]}, + {"role": "tool", "tool_call_id": "call_1", "content": "15 C and foggy"}, + ], + "tools": [tool()], + "max_tokens": 64, + "temperature": 0, + }, + ), + ( + "structured", + { + "model": model, + "messages": [{"role": "user", "content": "Return city and temperature JSON."}], + "response_format": { + "type": "json_schema", + "json_schema": { + "name": "weather", + "strict": True, + "schema": { + "type": "object", + "properties": {"city": {"type": "string"}, "temperature_c": {"type": "number"}}, + "required": ["city", "temperature_c"], + "additionalProperties": False, + }, + }, + }, + "chat_template_kwargs": {"enable_thinking": False}, + "max_tokens": 64, + "temperature": 0, + }, + ), + ] + records: list[dict[str, Any]] = [] + plain = cases.pop(0) + status, payload, elapsed = request_json(endpoint, key, plain[0], plain[1]) + del status + if not isinstance(payload, dict) or payload.get("model") != expected_model: + raise ProbeError(f"{model} plain response model drifted") + records.append(response_record(plain[0], payload, elapsed)) + + stream_body = dict(plain[1], stream=True) + _, stream_payload, elapsed = request_json(endpoint, key, "stream", stream_body) + if not isinstance(stream_payload, str): + raise ProbeError("stream did not return an event stream") + stream_record = check_stream("stream", stream_payload, expected_model) + stream_record["elapsed_seconds"] = round(elapsed, 6) + records.append(stream_record) + + for case, body in cases: + _, payload, elapsed = request_json(endpoint, key, case, body) + if not isinstance(payload, dict) or payload.get("model") != expected_model: + raise ProbeError(f"{model} {case} response model drifted") + choice = (payload.get("choices") or [{}])[0] + message = choice.get("message") or {} + if case == "tool-auto" and not message.get("tool_calls"): + raise ProbeError("tool-auto response did not contain a tool call") + if case == "structured": + try: + structured = json.loads(message.get("content", "")) + except json.JSONDecodeError as error: + raise ProbeError("structured response was not JSON") from error + if not {"city", "temperature_c"} <= set(structured): + raise ProbeError("structured response omitted required fields") + records.append(response_record(case, payload, elapsed)) + return records + + +def load_guard_module(): + spec = importlib.util.spec_from_file_location("glm52_contract_guard_probe", GUARD_MODULE) + if spec is None or spec.loader is None: + raise ProbeError("cannot load legacy renderer") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def boundary_repeats(renderer: Any, target: int) -> int: + data = {"model": "glm-5.2", "messages": [{"role": "user", "content": ""}]} + low, high = 0, target + 1024 + while low <= high: + middle = (low + high) // 2 + data["messages"][0]["content"] = "x " * middle + count = renderer.count(data) + if count < target: + low = middle + 1 + elif count > target: + high = middle - 1 + else: + return middle + raise ProbeError(f"could not construct a {target}-token legacy prompt") + + +def backend_chat_request_total(url: str) -> float: + with urllib.request.urlopen(url, timeout=15) as response: + text = response.read().decode("utf-8", errors="replace") + total = 0.0 + matched = False + for line in text.splitlines(): + if ( + line.startswith("http_requests_total{") + and 'handler="/v1/chat/completions"' in line + and 'method="POST"' in line + ): + matched = True + total += float(line.rsplit(" ", 1)[1]) + if not matched: + raise ProbeError("backend metrics contained no chat-completions request counter") + return total + + +def snapshot_backend_chat_requests(data: dict[str, Any]) -> dict[str, float]: + backends = data.get("backends") + if not isinstance(backends, dict): + raise ProbeError("backend metrics input must contain a backends mapping") + totals: dict[str, float] = {} + for name, record in backends.items(): + if not isinstance(name, str) or not isinstance(record, dict): + raise ProbeError("backend metrics record is invalid") + url = record.get("url") + if not isinstance(url, str) or not url: + raise ProbeError("backend metrics URL is invalid") + totals[name] = backend_chat_request_total(url) + return totals + + +def check_backend_metrics(path: Path | None) -> tuple[dict[str, Any], dict[str, float]]: + if path is None: + return {}, {} + data = json.loads(path.read_text(encoding="utf-8")) + backends = data.get("backends") + if not isinstance(backends, dict): + raise ProbeError("backend metrics input must contain a backends mapping") + after: dict[str, float] = {} + for name, record in backends.items(): + if not isinstance(record, dict) or record.get("expect") not in ("unchanged", "increase"): + raise ProbeError("backend metrics expectation is invalid") + if "chat_request_total" not in record: + raise ProbeError("backend metrics record is missing chat_request_total") + after[name] = backend_chat_request_total(record["url"]) + before = float(record["chat_request_total"]) + if record["expect"] == "unchanged" and after[name] != before: + raise ProbeError(f"backend {name} changed unexpectedly") + if record["expect"] == "increase" and after[name] <= before: + raise ProbeError(f"backend {name} did not receive probe traffic") + return data, after + + +def atomic_json(path: Path, value: dict[str, Any]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + descriptor, temporary = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent) + temporary_path = Path(temporary) + try: + with os.fdopen(descriptor, "w", encoding="utf-8") as handle: + json.dump(value, handle, indent=2, sort_keys=True) + handle.write("\n") + os.chmod(temporary_path, 0o600) + os.replace(temporary_path, path) + os.chmod(path, 0o600) + finally: + temporary_path.unlink(missing_ok=True) + + +def main() -> int: + args = parse_args() + artifact: dict[str, Any] = {"ok": False, "models": {}} + try: + if args.skip_boundary and os.environ.get("WEBSTER_CONTRACT_PROBE_TEST_MODE") != "1": + raise ProbeError("--skip-boundary is test-only") + key = read_key(args.key_file) + mappings = expected_models(args.expected_response_model, args.models) + for model in args.models: + artifact["models"][model] = run_model( + args.base_url, key, model, mappings[model] + ) + + before_metrics, after_metrics = check_backend_metrics(args.backend_metrics_before) + artifact["backend_metrics_after"] = after_metrics + + if not args.skip_boundary: + if args.tokenizer_dir is None: + raise ProbeError("--tokenizer-dir is required for the legacy boundary probe") + module = load_guard_module() + renderer = module.LegacyGLM52Renderer(args.tokenizer_dir) + repeats = boundary_repeats(renderer, 320001) + boundary_body = { + "model": "glm-5.2", + "messages": [{"role": "user", "content": "x " * repeats}], + "max_tokens": 1, + "temperature": 0, + } + boundary_before = snapshot_backend_chat_requests(before_metrics) + status, _, elapsed = request_json( + args.base_url.rstrip("/") + "/chat/completions", + key, + "boundary-320001", + boundary_body, + expected_status=400, + ) + boundary_after = snapshot_backend_chat_requests(before_metrics) + if boundary_before != boundary_after: + raise ProbeError("320001-token rejection reached a serving backend") + artifact["boundary"] = { + "prompt_tokens": 320001, + "status": status, + "elapsed_seconds": round(elapsed, 6), + "backend_counters_unchanged": True, + } + artifact["ok"] = True + atomic_json(args.output_json, artifact) + print(f"contract probe passed; artifact={args.output_json}") + return 0 + except (OSError, ValueError, json.JSONDecodeError, ProbeError) as error: + artifact["error"] = str(error) + atomic_json(args.output_json, artifact) + print(f"ERROR: {error}", file=sys.stderr) + return 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/webster/deepseek-v41/scripts/current-model-probes.py b/webster/deepseek-v41/scripts/current-model-probes.py new file mode 100755 index 0000000..6d55f20 --- /dev/null +++ b/webster/deepseek-v41/scripts/current-model-probes.py @@ -0,0 +1,170 @@ +#!/usr/bin/env python3 +"""Run safe, model-scoped completion probes from the LiteLLM host.""" + +from __future__ import annotations + +import json +import subprocess +import sys +import time +import urllib.error +import urllib.request +from pathlib import Path +from typing import Callable + +import yaml + + +ADMIN_BASE_URL = "http://127.0.0.1:4446" +COMPLETION_URL = "http://127.0.0.1:4444/v1/chat/completions" + + +def post_json(url: str, body: dict, key: str, timeout: int) -> tuple[int, dict]: + request = urllib.request.Request( + url, + data=json.dumps(body, separators=(",", ":")).encode(), + headers={ + "Authorization": "Bearer " + key, + "Content-Type": "application/json", + }, + method="POST", + ) + with urllib.request.urlopen(request, timeout=timeout) as response: + return response.status, json.load(response) + + +def probe_body(model_name: str, provider_model: str) -> dict: + body = { + "model": model_name, + "messages": [{"role": "user", "content": "Reply OK."}], + } + if provider_model.startswith("hosted_vllm/"): + body["max_tokens"] = 8 + return body + + +def run_scoped_probes( + deployments: list[tuple[str, str]], + change_id: str, + master_key: str, + *, + post: Callable[[str, dict, str, int], tuple[int, dict]] = post_json, +) -> dict: + virtual_key = None + results = [] + try: + _, created = post( + ADMIN_BASE_URL + "/key/generate", + { + "key_alias": f"baseline-{change_id}", + "models": [model_name for model_name, _ in deployments], + "max_budget": 0.5, + "budget_duration": "1d", + }, + master_key, + 30, + ) + virtual_key = created["key"] + for model_name, provider_model in deployments: + started = time.monotonic() + try: + status, payload = post( + COMPLETION_URL, + probe_body(model_name, provider_model), + virtual_key, + 180, + ) + except urllib.error.HTTPError as error: + raise RuntimeError( + f"completion probe failed model={model_name} status={error.code}" + ) from None + except Exception as error: + raise RuntimeError( + "completion probe failed " + f"model={model_name} exception={type(error).__name__}" + ) from None + + choices = payload.get("choices") + if status != 200 or not isinstance(choices, list) or not choices: + raise RuntimeError( + f"completion probe failed model={model_name} status={status}" + ) + results.append( + { + "model": model_name, + "status": status, + "response_model": payload.get("model"), + "elapsed_seconds": round(time.monotonic() - started, 6), + } + ) + finally: + if virtual_key is not None: + try: + delete_status, _ = post( + ADMIN_BASE_URL + "/key/delete", + {"keys": [virtual_key]}, + master_key, + 30, + ) + except urllib.error.HTTPError as error: + raise RuntimeError( + f"temporary scoped key revocation failed status={error.code}" + ) from None + except Exception as error: + raise RuntimeError( + "temporary scoped key revocation failed " + f"exception={type(error).__name__}" + ) from None + if delete_status != 200: + raise RuntimeError( + f"temporary scoped key revocation failed status={delete_status}" + ) + return {"models": results, "key_revoked": True} + + +def current_master_key() -> str: + environment = subprocess.check_output( + [ + "docker", + "inspect", + "litellm", + "--format", + "{{range .Config.Env}}{{println .}}{{end}}", + ], + text=True, + ).splitlines() + return next( + line.split("=", 1)[1] + for line in environment + if line.startswith("LITELLM_MASTER_KEY=") + ) + + +def current_deployments() -> list[tuple[str, str]]: + config = yaml.safe_load(Path("/home/nvidia/litellm/config.yaml").read_text()) + deployments = [] + seen = set() + for item in config["model_list"]: + model_name = item["model_name"] + if model_name in seen: + continue + provider_model = item["litellm_params"]["model"] + deployments.append((model_name, provider_model)) + seen.add(model_name) + return deployments + + +def main(argv: list[str]) -> int: + if len(argv) != 2: + raise SystemExit("usage: current-model-probes.py CHANGE_ID") + result = run_scoped_probes( + current_deployments(), + argv[1], + current_master_key(), + ) + print(json.dumps(result, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv)) diff --git a/webster/deepseek-v41/scripts/deepseek-v41-watchdog.sh b/webster/deepseek-v41/scripts/deepseek-v41-watchdog.sh new file mode 100755 index 0000000..1bedc82 --- /dev/null +++ b/webster/deepseek-v41/scripts/deepseek-v41-watchdog.sh @@ -0,0 +1,226 @@ +#!/usr/bin/env bash +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$script_dir/common.sh" + +run_root="" +profile_file="" +while (($#)); do + case "$1" in + --run-root) + [[ $# -ge 2 ]] || die "--run-root requires a value" + run_root="$2" + shift 2 + ;; + --profile-file) + [[ $# -ge 2 ]] || die "--profile-file requires a value" + profile_file="$2" + shift 2 + ;; + *) die "unknown argument: $1" ;; + esac +done +init_run_root "$(validate_run_root "$run_root")" +exec {watchdog_lock_fd}>"$RUN_ROOT/watchdog.lock" +chmod 0600 "$RUN_ROOT/watchdog.lock" +flock -n "$watchdog_lock_fd" || exit 0 +[[ ! -e "$RUN_ROOT/watchdog.disabled" ]] || exit 0 + +healthy=1 +reason=() +if [[ "${WEBSTER_LIFECYCLE_TEST_MODE:-0}" == "1" ]]; then + shamu_running="${TEST_LIFECYCLE_DEEPSEEK_SHAMU_RUNNING:-false}" + tilikum_running="${TEST_LIFECYCLE_DEEPSEEK_TILIKUM_RUNNING:-false}" + [[ "$shamu_running" == true && "$tilikum_running" == true ]] || { + healthy=0 + reason+=("rank-state-skew") + } + for pair in \ + "${TEST_LIFECYCLE_AUTH_OK:-1}:direct-health" \ + "${TEST_LIFECYCLE_AUTH_SEQUENCE_OK:-1}:auth-sequence" \ + "${TEST_LIFECYCLE_LISTENERS_OK:-1}:listeners" \ + "${TEST_LIFECYCLE_RANK1_KEYLESS:-1}:rank1-key" \ + "${TEST_LIFECYCLE_NCCL_OK:-1}:nccl" \ + "${TEST_LIFECYCLE_DEEPSEEK_IMAGE_MATCH:-1}:image" \ + "${TEST_LIFECYCLE_DEEPSEEK_PROFILE_MATCH:-1}:profile" \ + "${TEST_LIFECYCLE_DEEPSEEK_RUNTIME_MATCH:-1}:runtime" \ + "${TEST_LIFECYCLE_DEEPSEEK_RESTARTS_OK:-1}:restart-state" \ + "${TEST_LIFECYCLE_DEEPSEEK_GENERATION_MATCH:-1}:generation" \ + "${TEST_LIFECYCLE_DEEPSEEK_MOUNTS_OK:-1}:mounts" \ + "${TEST_LIFECYCLE_DEEPSEEK_ULIMITS_OK:-1}:ulimits" \ + "${TEST_LIFECYCLE_DEEPSEEK_FABRIC_OK:-1}:fabric"; do + [[ "${pair%%:*}" == 1 ]] || { + healthy=0 + reason+=("${pair#*:}") + } + done + if [[ "$shamu_running" == true && "$tilikum_running" == true && + -n "$profile_file" ]]; then + if ! TEST_LIFECYCLE_DEEPSEEK_SHAMU_RUNNING="${TEST_WATCHDOG_VERIFY_SHAMU_RUNNING:-$shamu_running}" \ + TEST_LIFECYCLE_DEEPSEEK_TILIKUM_RUNNING="${TEST_WATCHDOG_VERIFY_TILIKUM_RUNNING:-$tilikum_running}" \ + "$script_dir/start-deepseek-v41-tp2.sh" --run-root "$RUN_ROOT" \ + --profile-file "$profile_file" --verify-only >/dev/null; then + healthy=0 + reason+=("immutable-runtime-profile") + fi + fi +else + shamu_running="$(ssh shamu "docker inspect '$DEEPSEEK_CONTAINER' --format '{{.State.Running}}' 2>/dev/null || printf false")" + tilikum_running="$(ssh tilikum "sudo -n docker inspect '$DEEPSEEK_CONTAINER' --format '{{.State.Running}}' 2>/dev/null || printf false")" + shamu_started="$(ssh shamu "docker inspect '$DEEPSEEK_CONTAINER' --format '{{.State.StartedAt}}' 2>/dev/null" || true)" + tilikum_started="$(ssh tilikum "sudo -n docker inspect '$DEEPSEEK_CONTAINER' --format '{{.State.StartedAt}}' 2>/dev/null" || true)" + [[ "$shamu_running" == true && "$tilikum_running" == true ]] || { + healthy=0 + reason+=("rank-state-skew") + } + if [[ "$shamu_running" == true && "$tilikum_running" == true ]]; then + if [[ -z "$profile_file" || ! -f "$profile_file" || -L "$profile_file" ]] || + ! "$script_dir/start-deepseek-v41-tp2.sh" --run-root "$RUN_ROOT" \ + --profile-file "$profile_file" --verify-only >/dev/null; then + healthy=0 + reason+=("immutable-runtime-profile") + fi + fi + if [[ -n "$shamu_started" && -n "$tilikum_started" ]]; then + shamu_started_epoch="$(date -d "$shamu_started" +%s)" || die "invalid Shamu container start time" + tilikum_started_epoch="$(date -d "$tilikum_started" +%s)" || die "invalid Tilikum container start time" + start_skew=$((shamu_started_epoch - tilikum_started_epoch)) + (( start_skew < 0 )) && start_skew=$((-start_skew)) + if (( start_skew > 300 )); then + healthy=0 + reason+=("rank-start-time-skew") + fi + fi + curl -fsS --connect-timeout 3 --max-time 5 "http://$SHAMU_NETBIRD:$CANARY_PORT/health" >/dev/null || { + healthy=0 + reason+=("direct-health") + } + shamu_listener="$(ssh shamu "ss -lntH '( sport = :$CANARY_PORT )' | awk '{print \$4}'" || true)" + tilikum_listener="$(ssh tilikum "ss -lntH '( sport = :$CANARY_PORT )'" || true)" + [[ "$shamu_listener" == "$SHAMU_NETBIRD:$CANARY_PORT" && -z "$tilikum_listener" ]] || { + healthy=0 + reason+=("listeners") + } + ssh tilikum "sudo -n docker inspect '$DEEPSEEK_CONTAINER' --format '{{json .Args}}' | grep -q -- --headless" || { + healthy=0 + reason+=("rank1-not-headless") + } + ssh tilikum "sudo -n docker inspect '$DEEPSEEK_CONTAINER' --format '{{range .Config.Env}}{{println .}}{{end}}' | grep -Eq '^(VLLM_API_KEY|DSPARK_API_KEYS)=.+'" && { + healthy=0 + reason+=("rank1-key-present") + } + if ! ssh shamu python3 - "$DEEPSEEK_API_ENV_FILE" "$SHAMU_NETBIRD" \ + "$CANARY_PORT" <<'PY' +import sys +import urllib.error +import urllib.request +from pathlib import Path + +key_path, host, port = sys.argv[1:] +values = {} +for line in Path(key_path).read_text(encoding="utf-8").splitlines(): + if "=" in line: + name, value = line.split("=", 1) + values[name] = value +key = values.get("VLLM_API_KEY", "") +if not key: + raise SystemExit("rank-0 API key is missing") +url = f"http://{host}:{port}/v1/models" + + +def status(token): + headers = {} if token is None else {"Authorization": "Bearer " + token} + request = urllib.request.Request(url, headers=headers) + try: + with urllib.request.urlopen(request, timeout=10) as response: + return response.status + except urllib.error.HTTPError as error: + return error.code + + +if (status(None), status("intentionally-wrong"), status(key)) != (401, 401, 200): + raise SystemExit("station auth sequence failed") +PY + then + healthy=0 + reason+=("auth-sequence") + fi + if ssh shamu "docker logs --since 10m '$DEEPSEEK_CONTAINER' 2>&1 | grep -Eq 'EngineDeadError|NCCL.*(timeout|error)|RPC call.*timed out'" || + ssh tilikum "sudo -n docker logs --since 10m '$DEEPSEEK_CONTAINER' 2>&1 | grep -Eq 'EngineDeadError|NCCL.*(timeout|error)|RPC call.*timed out'"; then + healthy=0 + reason+=("engine-or-nccl-error") + fi +fi + +if (( healthy )); then + event deepseek-v41-watchdog OK "two-rank canary healthy" + exit 0 +fi + +reason_text="${reason[*]}" +event deepseek-v41-watchdog BREACH "$reason_text" + +now="$(date +%s)" +startup_age="" +if [[ "${WEBSTER_LIFECYCLE_TEST_MODE:-0}" == "1" ]]; then + startup_age="${TEST_LIFECYCLE_START_AGE_SECONDS:-1200}" +else + for started in "${shamu_started:-}" "${tilikum_started:-}"; do + [[ -n "$started" ]] || continue + started_epoch="$(date -d "$started" +%s)" || die "invalid container start time" + candidate_age=$((now - started_epoch)) + if [[ -z "$startup_age" || candidate_age -lt startup_age ]]; then + startup_age="$candidate_age" + fi + done +fi +if [[ -n "$startup_age" ]]; then + [[ "$startup_age" =~ ^[0-9]+$ ]] || die "invalid startup age" + if (( startup_age < 1200 )); then + event deepseek-v41-watchdog DEFER "breach inside 20-minute startup grace" + exit 0 + fi +fi +if [[ "${TEST_WATCHDOG_NO_RECOVERY:-0}" == 1 ]]; then + die "watchdog breach: $reason_text" +fi + +state_file="$RUN_ROOT/watchdog.last-recovery" +if [[ -f "$state_file" ]]; then + last="$(<"$state_file")" + [[ "$last" =~ ^[0-9]+$ ]] || die "invalid watchdog cooldown state" + if (( now - last < 1800 )); then + event deepseek-v41-watchdog DEFER "breach inside 30-minute recovery cooldown" + exit 0 + fi +fi + +validate_recovery_profile() { + local requested="$1" resolved + [[ -n "$requested" ]] || die "profile file is required for watchdog recovery" + [[ -f "$requested" && ! -L "$requested" ]] || + die "profile file must be a regular non-symlink" + resolved="$(canonical_existing_path "$requested")" || die "profile file does not resolve" + case "$resolved" in + "$RUN_ROOT"/*) ;; + *) die "profile file must resolve beneath the run root" ;; + esac + require_eq "profile file mode" 600 "$(stat -c %a "$resolved")" + require_eq "profile file owner" "$(id -u)" "$(stat -c %u "$resolved")" + printf '%s\n' "$resolved" +} +validated_profile_file="$(validate_recovery_profile "$profile_file")" + +if [[ "${WEBSTER_LIFECYCLE_TEST_MODE:-0}" != "1" ]]; then + capture deepseek-v41-watchdog-shamu ssh shamu \ + "docker inspect '$DEEPSEEK_CONTAINER' 2>&1 || true; docker logs --tail 500 '$DEEPSEEK_CONTAINER' 2>&1 || true; nvidia-smi" + capture deepseek-v41-watchdog-tilikum ssh tilikum \ + "sudo -n docker inspect '$DEEPSEEK_CONTAINER' 2>&1 || true; sudo -n docker logs --tail 500 '$DEEPSEEK_CONTAINER' 2>&1 || true; nvidia-smi" +fi +printf '%s\n' "$now" >"$state_file" +chmod 0600 "$state_file" +"$script_dir/stop-deepseek-v41-tp2.sh" --run-root "$RUN_ROOT" +"$script_dir/start-deepseek-v41-tp2.sh" --run-root "$RUN_ROOT" \ + --profile-file "$validated_profile_file" +event deepseek-v41-watchdog RECOVERED "coordinated stop/start completed" diff --git a/webster/deepseek-v41/scripts/feature-qualification.py b/webster/deepseek-v41/scripts/feature-qualification.py new file mode 100755 index 0000000..5d2879f --- /dev/null +++ b/webster/deepseek-v41/scripts/feature-qualification.py @@ -0,0 +1,942 @@ +#!/usr/bin/env python3 +"""Run deterministic, redacted API qualification cases against the private canary.""" + +from __future__ import annotations + +import argparse +import base64 +from datetime import datetime, timezone +import hashlib +import json +import os +import re +import stat +import struct +import subprocess +import sys +import tempfile +import time +import unicodedata +import urllib.error +import urllib.request +import zlib +from pathlib import Path +from typing import Any, Callable + + +Validator = Callable[[dict[str, Any], dict[str, Any]], bool] +Payload = dict[str, Any] | bytes + +EXPECTED_STATUSES = { + "malformed-input": 400, + "wrong-model": 404, +} +REPETITIONS = {"deterministic-text": 3} + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--base-url", required=True) + parser.add_argument("--model", required=True) + parser.add_argument("--key-file", required=True, type=Path) + parser.add_argument( + "--stage", required=True, choices=("eager", "graphs", "dspark") + ) + parser.add_argument("--output-json", required=True, type=Path) + parser.add_argument("--timeout", type=float, default=900.0) + parser.add_argument("--metrics-url") + parser.add_argument("--failure-log-output", type=Path) + parser.add_argument( + "--server-log-node", + action="append", + choices=("shamu", "tilikum"), + default=[], + ) + parser.add_argument( + "--server-container", default="deepseek-v41-flash-tp2" + ) + return parser.parse_args() + + +def read_secret(path: Path) -> str: + if not path.is_file() or path.is_symlink(): + raise RuntimeError("key file must be a regular non-symlink file") + if stat.S_IMODE(path.stat().st_mode) != 0o600: + raise RuntimeError("key file mode must be 0600") + value = path.read_text(encoding="utf-8").strip() + if not value: + raise RuntimeError("key file is empty") + return value + + +def canonical_hash(value: Any) -> str: + encoded = json.dumps( + value, ensure_ascii=False, sort_keys=True, separators=(",", ":") + ).encode("utf-8") + return hashlib.sha256(encoded).hexdigest() + + +def request_hash(value: Payload) -> str: + if isinstance(value, bytes): + return hashlib.sha256(value).hexdigest() + return canonical_hash(value) + + +def file_hash(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def normalize_output(value: str) -> str: + return unicodedata.normalize("NFC", value).replace("\r\n", "\n").strip() + + +def request_json( + url: str, + key: str, + case: str, + payload: Payload, + timeout: float, +) -> tuple[int, dict[str, Any], str | None, dict[str, Any]]: + request = urllib.request.Request( + url, + data=( + payload + if isinstance(payload, bytes) + else json.dumps(payload, ensure_ascii=False).encode("utf-8") + ), + headers={ + "Authorization": "Bearer " + key, + "Content-Type": "application/json", + "X-Feature-Qualification-Case": case, + }, + method="POST", + ) + try: + with urllib.request.urlopen(request, timeout=timeout) as response: + status = response.status + body = response.read().decode("utf-8") + except urllib.error.HTTPError as error: + return error.code, {}, None, {} + + if isinstance(payload, dict) and payload.get("stream"): + content: list[str] = [] + reasoning: list[str] = [] + finish_reason: str | None = None + usage: dict[str, Any] = {} + for line in body.splitlines(): + if not line.startswith("data: ") or line == "data: [DONE]": + continue + chunk = json.loads(line[6:]) + if isinstance(chunk.get("usage"), dict): + usage = chunk["usage"] + choices = chunk.get("choices") or [] + if not choices: + continue + choice = choices[0] + delta = choice.get("delta") or {} + if isinstance(delta.get("content"), str): + content.append(delta["content"]) + delta_reasoning = delta.get("reasoning") or delta.get( + "reasoning_content" + ) + if isinstance(delta_reasoning, str): + reasoning.append(delta_reasoning) + if choice.get("finish_reason") is not None: + finish_reason = choice["finish_reason"] + return ( + status, + {"content": "".join(content), "reasoning": "".join(reasoning)}, + finish_reason, + usage, + ) + + parsed = json.loads(body) + choices = parsed.get("choices") or [] + if not choices: + return status, {}, None, parsed.get("usage") or {} + choice = choices[0] + message = choice.get("message") or {} + return ( + status, + message if isinstance(message, dict) else {}, + choice.get("finish_reason"), + parsed.get("usage") or {}, + ) + + +def content_of(message: dict[str, Any]) -> str: + content = message.get("content") + return content if isinstance(content, str) else "" + + +def reasoning_of(message: dict[str, Any]) -> str: + reasoning = message.get("reasoning") or message.get("reasoning_content") + return reasoning if isinstance(reasoning, str) else "" + + +def reasoning_tokens(usage: dict[str, Any]) -> int: + details = usage.get("completion_tokens_details") + if not isinstance(details, dict): + return 0 + value = details.get("reasoning_tokens") + return value if isinstance(value, int) else 0 + + +def add_tool() -> dict[str, Any]: + return { + "type": "function", + "function": { + "name": "add", + "description": "Add two integers.", + "parameters": { + "type": "object", + "properties": { + "a": {"type": "integer"}, + "b": {"type": "integer"}, + }, + "required": ["a", "b"], + "additionalProperties": False, + }, + }, + } + + +def multiply_tool() -> dict[str, Any]: + return { + "type": "function", + "function": { + "name": "multiply", + "description": "Multiply two integers.", + "parameters": { + "type": "object", + "properties": { + "a": {"type": "integer"}, + "b": {"type": "integer"}, + }, + "required": ["a", "b"], + "additionalProperties": False, + }, + }, + } + + +def valid_add_call(message: dict[str, Any], _usage: dict[str, Any]) -> bool: + calls = message.get("tool_calls") + if not isinstance(calls, list) or len(calls) != 1: + return False + function = calls[0].get("function") if isinstance(calls[0], dict) else None + if not isinstance(function, dict) or function.get("name") != "add": + return False + try: + arguments = json.loads(function.get("arguments", "")) + except (TypeError, json.JSONDecodeError): + return False + return arguments == {"a": 17, "b": 25} + + +def valid_parallel_add_calls( + message: dict[str, Any], _usage: dict[str, Any] +) -> bool: + calls = message.get("tool_calls") + if not isinstance(calls, list) or len(calls) != 2: + return False + parsed: set[tuple[int, int]] = set() + for call in calls: + function = call.get("function") if isinstance(call, dict) else None + if not isinstance(function, dict) or function.get("name") != "add": + return False + try: + arguments = json.loads(function.get("arguments", "")) + except (TypeError, json.JSONDecodeError): + return False + if not isinstance(arguments, dict): + return False + a, b = arguments.get("a"), arguments.get("b") + if not isinstance(a, int) or not isinstance(b, int): + return False + parsed.add((a, b)) + return parsed == {(17, 25), (9, 11)} + + +def valid_place(message: dict[str, Any], _usage: dict[str, Any]) -> bool: + try: + value = json.loads(content_of(message)) + except json.JSONDecodeError: + return False + return value == {"city": "Paris", "country": "France"} + + +def png_chunk(kind: bytes, data: bytes) -> bytes: + checksum = zlib.crc32(kind + data) & 0xFFFFFFFF + return struct.pack(">I", len(data)) + kind + data + struct.pack(">I", checksum) + + +def red_png_data_url() -> str: + width = height = 560 + raw = b"".join(b"\x00" + b"\xff\x00\x00" * width for _ in range(height)) + png = ( + b"\x89PNG\r\n\x1a\n" + + png_chunk(b"IHDR", struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0)) + + png_chunk(b"IDAT", zlib.compress(raw, 9)) + + png_chunk(b"IEND", b"") + ) + return "data:image/png;base64," + base64.b64encode(png).decode("ascii") + + +def eager_cases(model: str) -> list[tuple[str, Payload, Validator]]: + common = { + "model": model, + "temperature": 0, + "max_tokens": 32, + "reasoning_effort": "none", + } + return [ + ( + "deterministic-text", + { + **common, + "messages": [ + { + "role": "user", + "content": "Reply with exactly QUALIFICATION_OK.", + } + ], + }, + lambda message, _usage: normalize_output(content_of(message)) + == "QUALIFICATION_OK", + ), + ( + "utf8", + { + **common, + "messages": [ + {"role": "user", "content": "Reply with exactly: café 東京 🚀"} + ], + }, + lambda message, _usage: normalize_output(content_of(message)) + == "café 東京 🚀", + ), + ( + "stop-sequence", + { + **common, + "messages": [ + { + "role": "user", + "content": "Write 'beforeafter' exactly.", + } + ], + "stop": [""], + }, + lambda message, _usage: "before" in content_of(message) + and "" not in content_of(message), + ), + ( + "streaming", + { + **common, + "messages": [ + {"role": "user", "content": "Reply with exactly: stream ok"} + ], + "stream": True, + "stream_options": {"include_usage": True}, + }, + lambda message, usage: normalize_output(content_of(message)) + == "stream ok" + and valid_usage(usage), + ), + ( + "usage-accounting", + { + **common, + "messages": [ + {"role": "user", "content": "Reply with exactly: usage ok"} + ], + }, + lambda message, usage: normalize_output(content_of(message)) + == "usage ok" + and valid_usage(usage), + ), + ( + "reasoning-on", + { + **common, + "messages": [ + { + "role": "user", + "content": "What is 17 + 25? Answer with the number.", + } + ], + "reasoning_effort": "low", + "max_tokens": 256, + }, + lambda message, usage: normalize_output(content_of(message)) == "42" + and bool(reasoning_of(message)) + and reasoning_tokens(usage) > 0, + ), + ( + "tool-auto", + { + **common, + "messages": [ + { + "role": "user", + "content": "Use the add tool to add 17 and 25.", + } + ], + "tools": [add_tool()], + "tool_choice": "auto", + "max_tokens": 128, + }, + valid_add_call, + ), + ( + "tool-named", + { + **common, + "messages": [ + { + "role": "user", + "content": "Use the add tool to add 17 and 25.", + } + ], + "tools": [add_tool(), multiply_tool()], + "tool_choice": { + "type": "function", + "function": {"name": "add"}, + }, + "max_tokens": 128, + }, + valid_add_call, + ), + ( + "tool-required", + { + **common, + "messages": [ + { + "role": "user", + "content": "Use a tool to add 17 and 25.", + } + ], + "tools": [add_tool()], + "tool_choice": "required", + "max_tokens": 128, + }, + valid_add_call, + ), + ( + "tool-parallel", + { + **common, + "messages": [ + { + "role": "user", + "content": ( + "Use the add tool twice in parallel: add 17 and 25, " + "and separately add 9 and 11." + ), + } + ], + "tools": [add_tool()], + "tool_choice": "required", + "parallel_tool_calls": True, + "max_tokens": 256, + }, + valid_parallel_add_calls, + ), + ( + "tool-result", + { + **common, + "messages": [ + { + "role": "user", + "content": "Use the add tool to add 17 and 25.", + }, + { + "role": "assistant", + "content": "", + "tool_calls": [ + { + "id": "call_1", + "type": "function", + "function": { + "name": "add", + "arguments": '{"a":17,"b":25}', + }, + } + ], + }, + {"role": "tool", "tool_call_id": "call_1", "content": "42"}, + ], + "tools": [add_tool()], + "max_tokens": 128, + }, + lambda message, _usage: "42" in content_of(message), + ), + ( + "structured", + { + **common, + "messages": [ + { + "role": "user", + "content": "Return the city Paris and country France.", + } + ], + "response_format": { + "type": "json_schema", + "json_schema": { + "name": "place", + "strict": True, + "schema": { + "type": "object", + "properties": { + "city": {"type": "string"}, + "country": {"type": "string"}, + }, + "required": ["city", "country"], + "additionalProperties": False, + }, + }, + }, + "max_tokens": 128, + }, + valid_place, + ), + ( + "json-object", + { + **common, + "messages": [ + { + "role": "user", + "content": ( + "Return a JSON object with city Paris and country France." + ), + } + ], + "response_format": {"type": "json_object"}, + "max_tokens": 128, + }, + valid_place, + ), + ( + "vision", + { + **common, + "messages": [ + { + "role": "user", + "content": [ + { + "type": "text", + "text": ( + "What is the dominant color? Reply one lowercase word." + ), + }, + { + "type": "image_url", + "image_url": {"url": red_png_data_url()}, + }, + ], + } + ], + }, + lambda message, _usage: normalize_output(content_of(message)) == "red", + ), + ( + "malformed-input", + b'{"model":', + lambda _message, _usage: True, + ), + ( + "wrong-model", + { + **common, + "model": "qualification-wrong-model", + "messages": [{"role": "user", "content": "Reply with ok."}], + }, + lambda _message, _usage: True, + ), + ( + "disconnect-queue-drain", + { + **common, + "messages": [ + { + "role": "user", + "content": "Count upward, one integer per line.", + } + ], + "stream": True, + "max_tokens": 2048, + "min_tokens": 512, + }, + lambda _message, _usage: True, + ), + ] + + +def valid_usage(usage: dict[str, Any]) -> bool: + values = [ + usage.get("prompt_tokens"), + usage.get("completion_tokens"), + usage.get("total_tokens"), + ] + return all(isinstance(value, int) and value >= 0 for value in values) and ( + values[0] + values[1] == values[2] + ) + + +def read_queue_metrics(url: str, timeout: float) -> dict[str, float]: + request = urllib.request.Request(url, method="GET") + with urllib.request.urlopen(request, timeout=timeout) as response: + body = response.read().decode("utf-8") + names = { + "running": "vllm:num_requests_running", + "waiting": "vllm:num_requests_waiting", + } + values: dict[str, float] = {} + for label, metric in names.items(): + samples: list[float] = [] + for line in body.splitlines(): + if not (line.startswith(metric + " ") or line.startswith(metric + "{")): + continue + try: + samples.append(float(line.rsplit(None, 1)[1])) + except (IndexError, ValueError) as error: + raise RuntimeError(f"malformed queue metric {metric}") from error + if not samples: + raise RuntimeError(f"missing queue metric {metric}") + values[label] = sum(samples) + return values + + +def disconnect_and_wait_for_drain( + endpoint: str, + metrics_url: str, + key: str, + case: str, + payload: dict[str, Any], + timeout: float, +) -> tuple[int, dict[str, Any]]: + metric_timeout = max(1.0, min(timeout, 10.0)) + baseline = read_queue_metrics(metrics_url, metric_timeout) + request = urllib.request.Request( + endpoint, + data=json.dumps(payload, ensure_ascii=False).encode("utf-8"), + headers={ + "Authorization": "Bearer " + key, + "Content-Type": "application/json", + "X-Feature-Qualification-Case": case, + }, + method="POST", + ) + observed_chunk = False + with urllib.request.urlopen(request, timeout=timeout) as response: + status = response.status + while True: + raw_line = response.readline() + if not raw_line: + break + line = raw_line.decode("utf-8").strip() + if line.startswith("data: ") and line != "data: [DONE]": + observed_chunk = True + break + + started = time.monotonic() + deadline = started + min(timeout, 30.0) + final = read_queue_metrics(metrics_url, metric_timeout) + while ( + final["running"] > baseline["running"] + or final["waiting"] > baseline["waiting"] + ) and time.monotonic() < deadline: + time.sleep(0.1) + final = read_queue_metrics(metrics_url, metric_timeout) + queue_drained = ( + final["running"] <= baseline["running"] + and final["waiting"] <= baseline["waiting"] + ) + return status, { + "client_disconnected": observed_chunk, + "queue_drained": queue_drained, + "queue_baseline": baseline, + "queue_final": final, + "queue_drain_seconds": round(time.monotonic() - started, 6), + } + + +def write_private_json(path: Path, value: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + descriptor, temporary_name = tempfile.mkstemp( + prefix=f".{path.name}.", dir=path.parent + ) + temporary = Path(temporary_name) + try: + with os.fdopen(descriptor, "w", encoding="utf-8") as handle: + json.dump(value, handle, ensure_ascii=False, indent=2, sort_keys=True) + handle.write("\n") + os.chmod(temporary, 0o600) + os.replace(temporary, path) + os.chmod(path, 0o600) + finally: + temporary.unlink(missing_ok=True) + + +def write_private_text(path: Path, value: str) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + descriptor, temporary_name = tempfile.mkstemp( + prefix=f".{path.name}.", dir=path.parent + ) + temporary = Path(temporary_name) + try: + with os.fdopen(descriptor, "w", encoding="utf-8") as handle: + handle.write(value) + os.chmod(temporary, 0o600) + os.replace(temporary, path) + os.chmod(path, 0o600) + finally: + temporary.unlink(missing_ok=True) + + +def redact_log(value: str) -> str: + value = re.sub( + r"(?i)(Authorization:\s*Bearer)\s+\S+", + r"\1 [REDACTED]", + value, + ) + value = re.sub( + r"(?i)((?:api[_-]?keys?|master[_-]?key|password|secret)" + r"[\"']?\s*[:=]\s*)[^,\s\"']+", + r"\1[REDACTED]", + value, + ) + value = re.sub(r"sk-[A-Za-z0-9._-]{8,}", "[REDACTED-KEY]", value) + return value + + +def capture_failure_logs( + path: Path, + nodes: list[str], + container: str, + since: str, +) -> dict[str, Any]: + environment = os.environ.copy() + environment.pop("SSH_AUTH_SOCK", None) + sections: list[str] = [] + ok = True + for node in nodes: + docker = ["sudo", "-n", "docker"] if node == "tilikum" else ["docker"] + command = [ + "ssh", + node, + *docker, + "logs", + "--since", + since, + "--timestamps", + "--tail", + "5000", + container, + ] + try: + result = subprocess.run( + command, + text=True, + capture_output=True, + check=False, + timeout=60, + env=environment, + ) + ok = ok and result.returncode == 0 + sections.append( + f"===== {node} exit={result.returncode} =====\n" + + result.stdout + + result.stderr + ) + except (OSError, subprocess.TimeoutExpired) as error: + ok = False + sections.append( + f"===== {node} capture_error={type(error).__name__} =====\n" + ) + write_private_text(path, redact_log("".join(sections))) + return { + "requested": True, + "captured": True, + "ok": ok, + "path": str(path.resolve()), + "sha256": file_hash(path), + "nodes": nodes, + "container": container, + "since": since, + } + + +def main() -> int: + args = parse_args() + if bool(args.failure_log_output) != bool(args.server_log_node): + print( + "ERROR: failure-log-output and server-log-node must be used together", + file=sys.stderr, + ) + return 1 + if re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,127}", args.server_container) is None: + print("ERROR: unsafe server container name", file=sys.stderr) + return 1 + qualification_started = datetime.now(timezone.utc).strftime( + "%Y-%m-%dT%H:%M:%SZ" + ) + try: + key = read_secret(args.key_file) + except (OSError, RuntimeError, UnicodeError) as error: + print(f"ERROR: {error}", file=sys.stderr) + return 1 + + normalized_base = args.base_url.rstrip("/") + endpoint = normalized_base + "/chat/completions" + metrics_url = args.metrics_url + if not metrics_url: + metrics_base = normalized_base[:-3] if normalized_base.endswith("/v1") else normalized_base + metrics_url = metrics_base + "/metrics" + results: list[dict[str, Any]] = [] + for case, payload, validator in eager_cases(args.model): + started = time.monotonic() + repetitions = REPETITIONS.get(case, 1) + attempts: list[dict[str, Any]] = [] + disconnect_details: dict[str, Any] = {} + for _ in range(repetitions): + try: + if case == "disconnect-queue-drain": + require_payload = payload if isinstance(payload, dict) else {} + status, disconnect_details = disconnect_and_wait_for_drain( + endpoint, + metrics_url, + key, + case, + require_payload, + args.timeout, + ) + message: dict[str, Any] = {} + finish_reason = "client_disconnect" + usage: dict[str, Any] = {} + semantic_ok = ( + status == 200 + and disconnect_details["client_disconnected"] is True + and disconnect_details["queue_drained"] is True + ) + else: + status, message, finish_reason, usage = request_json( + endpoint, key, case, payload, args.timeout + ) + semantic_ok = ( + status == EXPECTED_STATUSES.get(case, 200) + and validator(message, usage) + ) + error = None + except ( + OSError, + RuntimeError, + UnicodeError, + ValueError, + json.JSONDecodeError, + ) as exc: + status = 0 + message = {} + finish_reason = None + usage = {} + semantic_ok = False + error = type(exc).__name__ + normalized = normalize_output(content_of(message)) + attempts.append( + { + "status": status, + "finish_reason": finish_reason, + "token_usage": usage, + "normalized_output_hash": canonical_hash(normalized), + "reasoning_present": bool(reasoning_of(message)), + "tool_call_count": ( + len(message.get("tool_calls")) + if isinstance(message.get("tool_calls"), list) + else 0 + ), + "semantic_ok": semantic_ok, + "error": error, + } + ) + hashes = {attempt["normalized_output_hash"] for attempt in attempts} + semantic_ok = all(attempt["semantic_ok"] for attempt in attempts) + if repetitions > 1: + semantic_ok = semantic_ok and len(hashes) == 1 + first = attempts[0] + result = { + "case": case, + "request_hash": request_hash(payload), + "status": first["status"], + "finish_reason": first["finish_reason"], + "token_usage": first["token_usage"], + "latency_seconds": round(time.monotonic() - started, 6), + "normalized_output_hash": first["normalized_output_hash"], + "reasoning_present": first["reasoning_present"], + "tool_call_count": first["tool_call_count"], + "semantic_ok": semantic_ok, + "error": next( + (attempt["error"] for attempt in attempts if attempt["error"]), + None, + ), + "repetitions": repetitions, + "distinct_normalized_outputs": len(hashes), + } + if repetitions > 1: + result["attempts"] = attempts + result.update(disconnect_details) + results.append(result) + + artifact = { + "schema_version": 1, + "stage": args.stage, + "model": args.model, + "ok": all(result["semantic_ok"] for result in results), + "cases": results, + "failure_log_capture": { + "requested": bool(args.failure_log_output), + "captured": False, + "ok": True, + "path": ( + str(args.failure_log_output.resolve()) + if args.failure_log_output + else None + ), + "nodes": args.server_log_node, + "container": args.server_container, + }, + } + if not artifact["ok"] and args.failure_log_output: + artifact["failure_log_capture"] = capture_failure_logs( + args.failure_log_output, + args.server_log_node, + args.server_container, + qualification_started, + ) + try: + write_private_json(args.output_json, artifact) + except OSError as error: + print(f"ERROR: cannot write result: {type(error).__name__}", file=sys.stderr) + return 1 + if not artifact["ok"]: + failed = ",".join( + result["case"] for result in results if not result["semantic_ok"] + ) + print(f"NO-GO feature qualification failed cases={failed}", file=sys.stderr) + return 1 + print(f"GO feature qualification stage={args.stage} cases={len(results)}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/webster/deepseek-v41/scripts/generate-private-acceptance.py b/webster/deepseek-v41/scripts/generate-private-acceptance.py new file mode 100755 index 0000000..179a408 --- /dev/null +++ b/webster/deepseek-v41/scripts/generate-private-acceptance.py @@ -0,0 +1,356 @@ +#!/usr/bin/env python3 +"""Build the private publication manifest only after all evidence is stable.""" + +from __future__ import annotations + +import argparse +from datetime import datetime, timezone +import hashlib +import json +import os +import re +import stat +import subprocess +import sys +import tempfile +from pathlib import Path +from typing import Any + + +PROFILE_NAME = "dspark-1m-noautotune-seqs4" +MODEL = "deepseek-ai/DeepSeek-V4.1-Flash" +PUBLIC_MODEL = "deepseek-v4.1-flash" +STATION_ENDPOINT = "http://100.73.140.127:8000/v1" +KV_PATTERN = re.compile( + r"GPU KV cache size:\s*([0-9,]+) tokens, Maximum concurrency for " + r"1,048,576 tokens per request:\s*([0-9.]+)x" +) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--run-root", required=True, type=Path) + parser.add_argument("--rank-0-evidence", required=True, type=Path) + parser.add_argument("--rank-1-evidence", required=True, type=Path) + parser.add_argument("--api-evidence", required=True, type=Path) + parser.add_argument( + "--seqs-8-request-throughput-gain-percent", required=True, type=float + ) + parser.add_argument("--selection-reason", required=True) + parser.add_argument("--pp2-reason", required=True) + parser.add_argument("--max-age-seconds", type=float, default=21600) + return parser.parse_args() + + +def fail(message: str) -> None: + raise RuntimeError(message) + + +def private_file(path: Path, run_root: Path, label: str) -> Path: + if not path.is_absolute() or not path.is_file() or path.is_symlink(): + fail(f"{label} must be an absolute regular non-symlink file") + resolved = path.resolve(strict=True) + if not resolved.is_relative_to(run_root): + fail(f"{label} escapes the run root") + if stat.S_IMODE(resolved.stat().st_mode) != 0o600: + fail(f"{label} must have mode 0600") + if resolved.stat().st_uid != run_root.stat().st_uid: + fail(f"{label} owner does not match the run root") + return resolved + + +def sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def load_json(path: Path, label: str) -> dict[str, Any]: + try: + value = json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeDecodeError, json.JSONDecodeError) as error: + raise RuntimeError(f"{label} is malformed") from error + if not isinstance(value, dict): + fail(f"{label} must be an object") + return value + + +def load_env(path: Path, label: str) -> dict[str, str]: + values: dict[str, str] = {} + for number, raw in enumerate(path.read_text(encoding="utf-8").splitlines(), 1): + line = raw.strip() + if not line or line.startswith("#"): + continue + if "=" not in line: + fail(f"{label} line {number} is malformed") + key, value = line.split("=", 1) + if not key or key in values: + fail(f"{label} contains a duplicate or empty field") + values[key] = value + return values + + +def reference(path: Path) -> dict[str, str]: + return {"path": str(path), "sha256": sha256(path)} + + +def write_private_json(path: Path, value: Any) -> None: + descriptor, temporary_name = tempfile.mkstemp( + prefix=f".{path.name}.", dir=path.parent + ) + temporary = Path(temporary_name) + try: + with os.fdopen(descriptor, "w", encoding="utf-8") as handle: + json.dump(value, handle, ensure_ascii=False, indent=2, sort_keys=True) + handle.write("\n") + os.chmod(temporary, 0o600) + os.replace(temporary, path) + os.chmod(path, 0o600) + finally: + temporary.unlink(missing_ok=True) + + +def generate(args: argparse.Namespace) -> tuple[Path, str]: + run_root = args.run_root.resolve(strict=True) + if not run_root.is_dir() or args.run_root.is_symlink(): + fail("run root must be a regular directory") + + profile = private_file( + run_root / "profiles" / f"{PROFILE_NAME}.env", + run_root, + "selected profile", + ) + manifest = private_file(run_root / "manifest.env", run_root, "runtime manifest") + checkpoint_shamu = private_file( + run_root / "baseline" / "shamu-checkpoint.manifest", + run_root, + "Shamu checkpoint manifest", + ) + checkpoint_tilikum = private_file( + run_root / "baseline" / "tilikum-checkpoint.manifest", + run_root, + "Tilikum checkpoint manifest", + ) + if sha256(checkpoint_shamu) != sha256(checkpoint_tilikum): + fail("checkpoint manifests differ between ranks") + + feature_path = private_file( + run_root / "feature-dspark-1m-seqs4-final.json", + run_root, + "feature qualification", + ) + context_path = private_file( + run_root / "long-context-dspark-1m-seqs4-final.json", + run_root, + "long-context qualification", + ) + weka_path = private_file( + run_root / "aiperf" / "scored-summaries" / "seqs4.json", + run_root, + "Weka qualification", + ) + pricing_path = private_file( + run_root / "pricing-evidence.json", + run_root, + "pricing evidence", + ) + weka_repetitions = [ + private_file( + run_root + / "aiperf" + / f"{PROFILE_NAME}-c4-r{repetition}-open" + / "provenance.json", + run_root, + f"Weka repetition {repetition} provenance", + ) + for repetition in (1, 2) + ] + rank_0 = private_file(args.rank_0_evidence, run_root, "rank 0 evidence") + rank_1 = private_file(args.rank_1_evidence, run_root, "rank 1 evidence") + api = private_file(args.api_evidence, run_root, "API evidence") + + feature = load_json(feature_path, "feature qualification") + context = load_json(context_path, "long-context qualification") + weka = load_json(weka_path, "Weka qualification") + pricing = load_json(pricing_path, "pricing evidence") + manifest_values = load_env(manifest, "runtime manifest") + profile_values = load_env(profile, "selected profile") + rank_0_text = rank_0.read_text(encoding="utf-8") + rank_1_text = rank_1.read_text(encoding="utf-8") + kv = KV_PATTERN.search(rank_0_text) + if kv is None: + fail("rank 0 evidence is missing KV-cache capacity") + fatal = re.search(r"fatal_error_matches=([0-9]+)", rank_1_text) + if fatal is None: + fail("rank 1 evidence is missing fatal-error count") + rank_0_generation = re.search(r"^generation=(\S+)$", rank_0_text, re.MULTILINE) + rank_1_generation = re.search(r"^generation=(\S+)$", rank_1_text, re.MULTILINE) + if ( + rank_0_generation is None + or rank_1_generation is None + or rank_0_generation.group(1) != rank_1_generation.group(1) + ): + fail("rank evidence does not identify one serving generation") + serving_generation = rank_0_generation.group(1) + + cases = feature.get("cases") + accepted = context.get("accepted") + if not isinstance(cases, list) or not isinstance(accepted, list): + fail("qualification artifacts are incomplete") + accepted_totals = [row.get("target_context_tokens") for row in accepted] + over_limit = context.get("over_limit") + if not isinstance(over_limit, dict): + fail("long-context over-limit result is missing") + pricing_input = pricing.get("input") + pricing_output = pricing.get("output") + if not isinstance(pricing_input, dict) or not isinstance(pricing_output, dict): + fail("pricing evidence is incomplete") + + mirrored_weka = ( + "requests", + "errors", + "server_success_rate", + "request_throughput", + "output_token_throughput", + "ttft_p90_ms", + "request_peak_concurrency", + "preemptions_total", + ) + accepted_at = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + output = run_root / "private-acceptance.json" + value = { + "schema_version": 1, + "accepted_at": accepted_at, + "decision": "GO", + "model": MODEL, + "public_model_name": PUBLIC_MODEL, + "topology": { + "tensor_parallel_size": 2, + "pipeline_parallel_size": 1, + "rank_0": "shamu", + "rank_1": "tilikum", + "rank_1_headless_and_keyless": True, + }, + "selected_profile": { + "name": PROFILE_NAME, + "path": str(profile), + "sha256": sha256(profile), + "max_model_len": int(profile_values["MAX_MODEL_LEN"]), + "max_num_seqs": int(profile_values["MAX_NUM_SEQS"]), + "speculative_method": profile_values["SPECULATIVE_METHOD"], + "flashinfer_autotune": ( + profile_values["ENABLE_FLASHINFER_AUTOTUNE"] == "1" + ), + }, + "runtime": { + "manifest_path": str(manifest), + "manifest_sha256": sha256(manifest), + "vllm_commit": manifest_values["VLLM_COMMIT"], + "image_id": manifest_values["VLLM_IMAGE_ID"], + "checkpoint_revision": manifest_values["CHECKPOINT_REVISION"], + "checkpoint_manifest_sha256": sha256(checkpoint_shamu), + }, + "feature_qualification": { + **reference(feature_path), + "passed": sum( + 1 + for case in cases + if isinstance(case, dict) and case.get("semantic_ok") is True + ), + "total": len(cases), + "ok": feature.get("ok"), + }, + "long_context_qualification": { + **reference(context_path), + "accepted_context_totals": accepted_totals, + "rejected_context_total": over_limit.get("target_context_tokens"), + "ok": context.get("ok"), + }, + "weka_qualification": { + **reference(weka_path), + **{name: weka.get(name) for name in mirrored_weka}, + "qualified": weka.get("qualified"), + "repetitions": [reference(path) for path in weka_repetitions], + }, + "live_state": { + "rank_0_evidence": reference(rank_0), + "rank_1_evidence": reference(rank_1), + "api_evidence": reference(api), + "kv_cache_tokens": int(kv.group(1).replace(",", "")), + "full_context_concurrency": float(kv.group(2)), + "restart_counts_zero": ( + "running=true" in rank_0_text + and "restart=0" in rank_0_text + and "running=true" in rank_1_text + and "restart=0" in rank_1_text + ), + "fatal_error_matches": int(fatal.group(1)), + "serving_generation": serving_generation, + }, + "selection": { + "seqs_8_request_throughput_gain_percent": ( + args.seqs_8_request_throughput_gain_percent + ), + "reason": args.selection_reason, + "pp2_triggered": False, + "pp2_reason": args.pp2_reason, + }, + "pricing_evidence": reference(pricing_path), + "publication": { + "public_model_name": PUBLIC_MODEL, + "served_model_name": MODEL, + "max_context": int(profile_values["MAX_MODEL_LEN"]), + "station_endpoint": STATION_ENDPOINT, + "supports_function_calling": True, + "supports_reasoning": True, + "supports_response_schema": True, + "supports_vision": any( + isinstance(case, dict) + and case.get("case") == "vision" + and case.get("semantic_ok") is True + and case.get("status") == 200 + for case in cases + ), + "input_cost_per_token": pricing_input.get("published_cost_per_token"), + "output_cost_per_token": pricing_output.get("published_cost_per_token"), + }, + } + write_private_json(output, value) + return output, accepted_at + + +def main() -> int: + args = parse_args() + try: + output, accepted_at = generate(args) + verifier = Path(__file__).with_name("verify-private-acceptance.py") + result = subprocess.run( + [ + sys.executable, + str(verifier), + "--run-root", + str(args.run_root), + "--acceptance", + str(output), + "--max-age-seconds", + str(args.max_age_seconds), + "--now", + accepted_at, + ], + text=True, + capture_output=True, + check=False, + ) + if result.returncode != 0: + fail(result.stderr.strip() or "generated acceptance failed validation") + except (KeyError, OSError, RuntimeError, UnicodeError, ValueError) as error: + print(f"ERROR: {error}", file=sys.stderr) + return 1 + print(f"private acceptance generated and validated: {output}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/webster/deepseek-v41/scripts/install-glm52-guard.sh b/webster/deepseek-v41/scripts/install-glm52-guard.sh new file mode 100755 index 0000000..0b27c92 --- /dev/null +++ b/webster/deepseek-v41/scripts/install-glm52-guard.sh @@ -0,0 +1,229 @@ +#!/usr/bin/env bash +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +package_dir="$(cd "$script_dir/.." && pwd)" +source "$script_dir/common.sh" + +candidate=0 +run_root="" +candidate_config="" +while (($#)); do + case "$1" in + --candidate) + candidate=1 + shift + ;; + --run-root) + [[ $# -ge 2 ]] || die "--run-root requires a value" + run_root="$2" + shift 2 + ;; + --candidate-config) + [[ $# -ge 2 ]] || die "--candidate-config requires a value" + candidate_config="$2" + shift 2 + ;; + *) die "unknown argument: $1" ;; + esac +done + +[[ "$candidate" == "1" ]] || die "--candidate is required" +validated_root="$(validate_run_root "$run_root")" +init_run_root "$validated_root" +[[ -f "$candidate_config" && ! -L "$candidate_config" ]] || + die "candidate config must be a regular file" +[[ "$(stat -c %a "$candidate_config")" == "600" ]] || + die "candidate config mode must be 0600" + +module_source="${TEST_GUARD_SOURCE:-$package_dir/litellm/glm52_contract_guard.py}" +test_source="$package_dir/litellm/test_glm52_contract_guard.py" +verifier_source="$script_dir/verify-litellm-callbacks.py" +fixture_source="$package_dir/tests/fixtures" +tokenizer_source="${TEST_TOKENIZER_SOURCE:-$RUN_ROOT/baseline/glm52-tokenizer}" +[[ -f "$module_source" && ! -L "$module_source" ]] || die "guard module is missing" +[[ -f "$verifier_source" && ! -L "$verifier_source" ]] || + die "callback verifier is missing" +for name in tokenizer.json tokenizer_config.json chat_template.jinja; do + [[ -f "$tokenizer_source/$name" && ! -L "$tokenizer_source/$name" ]] || + die "legacy tokenizer asset is missing: $name" +done + +backup_suffix="$(date -u +%Y%m%dT%H%M%SZ)-$$" + +promote_local_file() { + local source_path="$1" final_path="$2" temporary_path + mkdir -p "$(dirname "$final_path")" + if [[ -f "$final_path" ]] && cmp -s "$source_path" "$final_path"; then + chmod 0644 "$final_path" + return + fi + if [[ -e "$final_path" ]]; then + mv -- "$final_path" "$final_path.bak-$backup_suffix" + fi + temporary_path="$final_path.tmp-$backup_suffix" + install -m 0644 "$source_path" "$temporary_path" + mv -- "$temporary_path" "$final_path" +} + +promote_local_tokenizer() { + local source_path="$1" final_path="$2" temporary_path source_manifest live_manifest + source_manifest="$(mktemp)" + live_manifest="$(mktemp)" + ( + cd "$source_path" + sha256sum tokenizer.json tokenizer_config.json chat_template.jinja | sort + ) >"$source_manifest" + if [[ -d "$final_path" ]]; then + ( + cd "$final_path" + sha256sum tokenizer.json tokenizer_config.json chat_template.jinja | sort + ) >"$live_manifest" 2>/dev/null || true + if cmp -s "$source_manifest" "$live_manifest"; then + chmod 0444 "$final_path/tokenizer.json" \ + "$final_path/tokenizer_config.json" "$final_path/chat_template.jinja" + chmod 0555 "$final_path" + rm -f -- "$source_manifest" "$live_manifest" + return + fi + mv -- "$final_path" "$final_path.bak-$backup_suffix" + elif [[ -e "$final_path" ]]; then + mv -- "$final_path" "$final_path.bak-$backup_suffix" + fi + temporary_path="$final_path.tmp-$backup_suffix" + mkdir -m 0755 "$temporary_path" + for name in tokenizer.json tokenizer_config.json chat_template.jinja; do + install -m 0444 "$source_path/$name" "$temporary_path/$name" + done + chmod 0555 "$temporary_path" + mv -- "$temporary_path" "$final_path" + rm -f -- "$source_manifest" "$live_manifest" +} + +if [[ "${WEBSTER_LITELLM_TEST_MODE:-0}" == "1" ]]; then + test_root="${TEST_LITELLM_ROOT:?TEST_LITELLM_ROOT is required in test mode}" + promote_local_file "$module_source" "$test_root/glm52_contract_guard.py" + promote_local_tokenizer "$tokenizer_source" "$test_root/glm52-tokenizer" + printf 'guard candidate installed in test root\n' + exit 0 +fi + +change_id="$(basename "$RUN_ROOT")" +remote_parent="/home/nvidia/litellm/energy-pricing" +remote_module="$remote_parent/glm52_contract_guard.py" +remote_tokenizer="$remote_parent/glm52-tokenizer" +remote_stage="$remote_parent/.glm52-guard-stage-$change_id-$backup_suffix" +remote_offline="$remote_parent/.glm52-guard-offline-$change_id-$backup_suffix" + +cleanup_remote_candidates() { + local original_status=$? + trap - EXIT + if ! ssh spark-1 "set -eu +parent='$remote_parent' +for path in '$remote_stage' '$remote_offline'; do + case \"\$path\" in + \"\$parent\"/.glm52-guard-stage-*|\"\$parent\"/.glm52-guard-offline-*) ;; + *) exit 1 ;; + esac + if test -e \"\$path\"; then + resolved=\$(realpath -e -- \"\$path\") + case \"\$resolved\" in + \"\$parent\"/.glm52-guard-stage-*|\"\$parent\"/.glm52-guard-offline-*) ;; + *) exit 1 ;; + esac + chmod -R u+w -- \"\$resolved\" + rm -rf -- \"\$resolved\" + fi +done"; then + printf 'ERROR: failed to remove remote guard candidate paths\n' >&2 + [[ "$original_status" != "0" ]] || original_status=1 + fi + exit "$original_status" +} +trap cleanup_remote_candidates EXIT + +local_module_hash="$(sha256sum "$module_source" | awk '{print $1}')" +local_tokenizer_manifest="$RUN_ROOT/baseline/glm52-tokenizer.install.manifest" +( + cd "$tokenizer_source" + sha256sum tokenizer.json tokenizer_config.json chat_template.jinja | sort +) >"$local_tokenizer_manifest" +chmod 0600 "$local_tokenizer_manifest" + +ssh spark-1 "set -eu; umask 077; test ! -e '$remote_stage'; mkdir -p '$remote_stage/tokenizer'" +scp "$module_source" "spark-1:$remote_stage/glm52_contract_guard.py" +tar -C "$tokenizer_source" -cf - tokenizer.json tokenizer_config.json chat_template.jinja | + ssh spark-1 "tar -C '$remote_stage/tokenizer' -xf -" +remote_module_hash="$(ssh spark-1 "sha256sum '$remote_stage/glm52_contract_guard.py' | awk '{print \$1}'")" +require_eq "guard module transfer hash" "$local_module_hash" "$remote_module_hash" +ssh spark-1 "cd '$remote_stage/tokenizer' && sha256sum tokenizer.json tokenizer_config.json chat_template.jinja | sort" \ + >"$RUN_ROOT/baseline/glm52-tokenizer.install.remote.manifest" +cmp "$local_tokenizer_manifest" "$RUN_ROOT/baseline/glm52-tokenizer.install.remote.manifest" || + die "legacy tokenizer transfer hash mismatch" + +ssh spark-1 "set -eu +backup_suffix='$backup_suffix' +if test -f '$remote_module'; then + live_hash=\$(sha256sum '$remote_module' | awk '{print \$1}') + if test \"\$live_hash\" != '$local_module_hash'; then + mv '$remote_module' '$remote_module.bak-$backup_suffix' + fi +fi +if ! test -f '$remote_module' || test \"\$(sha256sum '$remote_module' | awk '{print \$1}')\" != '$local_module_hash'; then + install -m 0644 '$remote_stage/glm52_contract_guard.py' '$remote_module.tmp-$backup_suffix' + mv '$remote_module.tmp-$backup_suffix' '$remote_module' +fi +if test -d '$remote_tokenizer'; then + cd '$remote_tokenizer' + sha256sum tokenizer.json tokenizer_config.json chat_template.jinja | sort >'/tmp/glm52-live-$backup_suffix.manifest' + cd '$remote_stage/tokenizer' + sha256sum tokenizer.json tokenizer_config.json chat_template.jinja | sort >'/tmp/glm52-new-$backup_suffix.manifest' + if ! cmp -s '/tmp/glm52-live-$backup_suffix.manifest' '/tmp/glm52-new-$backup_suffix.manifest'; then + mv '$remote_tokenizer' '$remote_tokenizer.bak-$backup_suffix' + fi +elif test -e '$remote_tokenizer'; then + mv '$remote_tokenizer' '$remote_tokenizer.bak-$backup_suffix' +fi +if ! test -d '$remote_tokenizer'; then + mv '$remote_stage/tokenizer' '$remote_tokenizer' +fi +chmod 0644 '$remote_module' +chmod 0444 '$remote_tokenizer/tokenizer.json' '$remote_tokenizer/tokenizer_config.json' '$remote_tokenizer/chat_template.jinja' +chmod 0555 '$remote_tokenizer' +rm -f '/tmp/glm52-live-$backup_suffix.manifest' '/tmp/glm52-new-$backup_suffix.manifest' +rm -rf '$remote_stage' +test \"\$(sha256sum '$remote_module' | awk '{print \$1}')\" = '$local_module_hash' +test \"\$(stat -c %a:%U '$remote_module')\" = 644:nvidia +test \"\$(stat -c %a:%U '$remote_tokenizer')\" = 555:nvidia +for path in '$remote_tokenizer/tokenizer.json' '$remote_tokenizer/tokenizer_config.json' '$remote_tokenizer/chat_template.jinja'; do + test \"\$(stat -c %a:%U \"\$path\")\" = 444:nvidia +done" + +ssh spark-1 "set -eu; test ! -e '$remote_offline'; mkdir -p '$remote_offline/litellm' '$remote_offline/tests/fixtures'" +scp "$test_source" "spark-1:$remote_offline/litellm/test_glm52_contract_guard.py" +scp "$verifier_source" "spark-1:$remote_offline/verify-litellm-callbacks.py" +scp "$candidate_config" "spark-1:$remote_offline/config.yaml" +scp "$fixture_source/chat.json" "$fixture_source/reasoning-none.json" \ + "$fixture_source/tools.json" "$fixture_source/tool-result.json" \ + "$fixture_source/structured.json" "$fixture_source/tokenize-golden.json" \ + "spark-1:$remote_offline/tests/fixtures/" +ssh spark-1 "cp '$remote_module' '$remote_offline/litellm/glm52_contract_guard.py'; chmod -R a-w '$remote_offline'" + +ssh spark-1 "set -euo pipefail +image_id=\$(docker inspect litellm --format '{{.Image}}') +docker run --rm --network none --entrypoint python \ + -e LITELLM_LOCAL_MODEL_COST_MAP=true \ + -v '$remote_parent:/app/custom_callbacks:ro' \ + -v '$remote_offline:/work:ro' \ + -w /app \ + \"\$image_id\" /work/verify-litellm-callbacks.py /work/config.yaml +docker run --rm --network none --entrypoint python \ + -e GLM52_TOKENIZER_DIR=/app/custom_callbacks/glm52-tokenizer \ + -e LITELLM_LOCAL_MODEL_COST_MAP=true \ + -v '$remote_parent:/app/custom_callbacks:ro' \ + -v '$remote_offline:/work:ro' \ + -w /work/litellm \ + \"\$image_id\" test_glm52_contract_guard.py" + +event "alias-guard" GO "candidate guard and tokenizer installed; exact-image offline checks passed" +printf 'guard candidate installed and validated offline\n' diff --git a/webster/deepseek-v41/scripts/long-context-qualification.py b/webster/deepseek-v41/scripts/long-context-qualification.py new file mode 100755 index 0000000..e1846ed --- /dev/null +++ b/webster/deepseek-v41/scripts/long-context-qualification.py @@ -0,0 +1,321 @@ +#!/usr/bin/env python3 +"""Qualify exact DeepSeek context boundaries without retaining prompt contents.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import stat +import sys +import tempfile +import time +import urllib.error +import urllib.request +from pathlib import Path +from typing import Any + + +PADDING_UNIT = " x" +COMPLETION_TOKENS = 1 + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--base-url", required=True) + parser.add_argument("--model", required=True) + parser.add_argument("--key-file", required=True, type=Path) + parser.add_argument("--max-model-len", required=True, type=int) + parser.add_argument( + "--target-context", required=True, action="append", type=int + ) + parser.add_argument("--output-json", required=True, type=Path) + parser.add_argument("--timeout", type=float, default=1800.0) + return parser.parse_args() + + +def read_secret(path: Path) -> str: + if not path.is_file() or path.is_symlink(): + raise RuntimeError("key file must be a regular non-symlink file") + if stat.S_IMODE(path.stat().st_mode) != 0o600: + raise RuntimeError("key file mode must be 0600") + value = path.read_text(encoding="utf-8").strip() + if not value: + raise RuntimeError("key file is empty") + return value + + +def write_private_json(path: Path, value: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + descriptor, temporary_name = tempfile.mkstemp( + prefix=f".{path.name}.", dir=path.parent + ) + temporary = Path(temporary_name) + try: + with os.fdopen(descriptor, "w", encoding="utf-8") as handle: + json.dump(value, handle, indent=2, sort_keys=True) + handle.write("\n") + os.chmod(temporary, 0o600) + os.replace(temporary, path) + os.chmod(path, 0o600) + finally: + temporary.unlink(missing_ok=True) + + +def post_json( + url: str, + key: str, + payload: dict[str, Any], + timeout: float, +) -> tuple[int, dict[str, Any], str]: + encoded = json.dumps( + payload, ensure_ascii=False, separators=(",", ":") + ).encode("utf-8") + request_hash = hashlib.sha256(encoded).hexdigest() + request = urllib.request.Request( + url, + data=encoded, + headers={ + "Authorization": "Bearer " + key, + "Content-Type": "application/json", + "X-Long-Context-Qualification": "1", + }, + method="POST", + ) + try: + with urllib.request.urlopen(request, timeout=timeout) as response: + body = json.loads(response.read().decode("utf-8")) + return ( + response.status, + body if isinstance(body, dict) else {}, + request_hash, + ) + except urllib.error.HTTPError as error: + try: + body = json.loads(error.read().decode("utf-8")) + except (UnicodeError, json.JSONDecodeError): + body = {} + return error.code, body if isinstance(body, dict) else {}, request_hash + + +def tokenize( + base_url: str, + model: str, + key: str, + repetitions: int, + timeout: float, +) -> tuple[int, int | None, int | None, str]: + status, body, request_hash = post_json( + base_url.rstrip("/") + "/tokenize", + key, + { + "model": model, + "messages": [{"role": "user", "content": PADDING_UNIT * repetitions}], + "add_generation_prompt": True, + "chat_template_kwargs": {"enable_thinking": False}, + }, + timeout, + ) + count = body.get("count") + max_model_len = body.get("max_model_len") + return ( + status, + count if isinstance(count, int) and not isinstance(count, bool) else None, + ( + max_model_len + if isinstance(max_model_len, int) and not isinstance(max_model_len, bool) + else None + ), + request_hash, + ) + + +def chat_payload(model: str, repetitions: int) -> dict[str, Any]: + return { + "model": model, + "messages": [{"role": "user", "content": PADDING_UNIT * repetitions}], + "temperature": 0, + "max_tokens": COMPLETION_TOKENS, + "reasoning_effort": "none", + } + + +def validate_targets(targets: list[int], max_model_len: int) -> None: + if max_model_len < 2: + raise RuntimeError("max model length must be at least 2") + if not targets: + raise RuntimeError("at least one target context is required") + if len(set(targets)) != len(targets): + raise RuntimeError("target contexts must be unique") + for target in targets: + if target < 2 or target > max_model_len: + raise RuntimeError( + "target contexts must be between 2 and max model length" + ) + + +def usage_counts(body: dict[str, Any]) -> tuple[int | None, int | None, int | None]: + usage = body.get("usage") + if not isinstance(usage, dict): + return None, None, None + values: list[int | None] = [] + for name in ("prompt_tokens", "completion_tokens", "total_tokens"): + value = usage.get(name) + values.append(value if isinstance(value, int) and not isinstance(value, bool) else None) + return values[0], values[1], values[2] + + +def main() -> int: + args = parse_args() + artifact: dict[str, Any] = { + "schema_version": 1, + "model": args.model, + "max_model_len": args.max_model_len, + "padding_unit_sha256": hashlib.sha256(PADDING_UNIT.encode()).hexdigest(), + "accepted": [], + "over_limit": {}, + "ok": False, + } + try: + validate_targets(args.target_context, args.max_model_len) + key = read_secret(args.key_file) + base_url = args.base_url.rstrip("/") + + baseline_status, baseline_count, reported_max, baseline_hash = tokenize( + base_url, args.model, key, 0, args.timeout + ) + unit_status, unit_count, unit_max, unit_hash = tokenize( + base_url, args.model, key, 64, args.timeout + ) + tokenizer_ready = ( + baseline_status == 200 + and unit_status == 200 + and baseline_count is not None + and unit_count is not None + and unit_count - baseline_count == 64 + and reported_max == args.max_model_len + and unit_max == args.max_model_len + ) + artifact["tokenizer_calibration"] = { + "baseline_status": baseline_status, + "baseline_prompt_tokens": baseline_count, + "baseline_request_hash": baseline_hash, + "unit_probe_status": unit_status, + "unit_probe_prompt_tokens": unit_count, + "unit_probe_request_hash": unit_hash, + "unit_probe_repetitions": 64, + "reported_max_model_len": reported_max, + "exact_linear_unit": tokenizer_ready, + } + if not tokenizer_ready or baseline_count is None: + write_private_json(args.output_json, artifact) + return 1 + + accepted_ok = True + for target in args.target_context: + expected_prompt = target - COMPLETION_TOKENS + repetitions = expected_prompt - baseline_count + if repetitions < 0: + raise RuntimeError("target is smaller than the chat-template overhead") + started = time.monotonic() + tokenize_status, tokenize_count, target_max, tokenize_hash = tokenize( + base_url, args.model, key, repetitions, args.timeout + ) + token_count_exact = ( + tokenize_status == 200 + and tokenize_count == expected_prompt + and target_max == args.max_model_len + ) + result: dict[str, Any] = { + "target_context_tokens": target, + "expected_prompt_tokens": expected_prompt, + "padding_repetitions": repetitions, + "tokenize_status": tokenize_status, + "tokenize_request_hash": tokenize_hash, + "token_count_exact": token_count_exact, + } + if token_count_exact: + status, body, request_hash = post_json( + base_url + "/v1/chat/completions", + key, + chat_payload(args.model, repetitions), + args.timeout, + ) + prompt_tokens, completion_tokens, total_tokens = usage_counts(body) + result.update( + { + "status": status, + "request_hash": request_hash, + "server_prompt_tokens": prompt_tokens, + "server_completion_tokens": completion_tokens, + "server_total_tokens": total_tokens, + "latency_seconds": round(time.monotonic() - started, 6), + } + ) + result["semantic_ok"] = ( + status == 200 + and prompt_tokens == expected_prompt + and completion_tokens == COMPLETION_TOKENS + and total_tokens == target + ) + else: + result["semantic_ok"] = False + artifact["accepted"].append(result) + accepted_ok = accepted_ok and bool(result["semantic_ok"]) + if not token_count_exact: + break + + over_limit_ok = False + if accepted_ok: + over_limit_target = args.max_model_len + 1 + expected_prompt = args.max_model_len + repetitions = expected_prompt - baseline_count + started = time.monotonic() + tokenize_status, tokenize_count, target_max, tokenize_hash = tokenize( + base_url, args.model, key, repetitions, args.timeout + ) + token_count_exact = ( + tokenize_status == 200 + and tokenize_count == expected_prompt + and target_max == args.max_model_len + ) + status: int | None = None + request_hash: str | None = None + if token_count_exact: + status, _body, request_hash = post_json( + base_url + "/v1/chat/completions", + key, + chat_payload(args.model, repetitions), + args.timeout, + ) + over_limit_ok = token_count_exact and status == 400 + artifact["over_limit"] = { + "target_context_tokens": over_limit_target, + "expected_prompt_tokens": expected_prompt, + "padding_repetitions": repetitions, + "tokenize_status": tokenize_status, + "tokenize_request_hash": tokenize_hash, + "token_count_exact": token_count_exact, + "status": status, + "request_hash": request_hash, + "latency_seconds": round(time.monotonic() - started, 6), + "semantic_ok": over_limit_ok, + } + + artifact["ok"] = accepted_ok and over_limit_ok + write_private_json(args.output_json, artifact) + return 0 if artifact["ok"] else 1 + except (OSError, RuntimeError, UnicodeError, json.JSONDecodeError) as error: + artifact["error_type"] = type(error).__name__ + artifact["error"] = str(error) + try: + write_private_json(args.output_json, artifact) + except OSError: + pass + print(f"ERROR: {error}", file=sys.stderr) + return 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/webster/deepseek-v41/scripts/patch-aiperf-weka-timing.py b/webster/deepseek-v41/scripts/patch-aiperf-weka-timing.py new file mode 100755 index 0000000..750fe07 --- /dev/null +++ b/webster/deepseek-v41/scripts/patch-aiperf-weka-timing.py @@ -0,0 +1,66 @@ +#!/usr/bin/env python3 +"""Apply the pinned AIPerf Weka fixed-schedule timing fix safely.""" + +from __future__ import annotations + +import argparse +import os +import sys +from pathlib import Path + + +RELATIVE_RESOLVER = Path("src/aiperf/config/dataset/resolver.py") +UNPATCHED = ( + " CustomDatasetType.BURST_GPT_TRACE,\n" + " CustomDatasetType.BASETEN_TRACE,\n" + " CustomDatasetType.TRACELAB,\n" +) +PATCHED = UNPATCHED + " CustomDatasetType.WEKA_TRACE,\n" + + +def apply_patch(source: Path) -> None: + source = source.resolve(strict=True) + resolver = source / RELATIVE_RESOLVER + if resolver.is_symlink() or not resolver.is_file(): + raise ValueError(f"AIPerf resolver is not a regular file: {resolver}") + if not resolver.resolve(strict=True).is_relative_to(source): + raise ValueError(f"AIPerf resolver escapes source root: {resolver}") + + original = resolver.read_text(encoding="utf-8") + if original.count(PATCHED) == 1: + print(f"GO AIPerf Weka timing patch already active: {resolver}") + return + if original.count(UNPATCHED) != 1: + raise ValueError( + "unsupported AIPerf resolver source; refusing an ambiguous patch" + ) + + updated = original.replace(UNPATCHED, PATCHED, 1) + temporary = resolver.with_name(f".{resolver.name}.tmp.{os.getpid()}") + try: + temporary.write_text(updated, encoding="utf-8") + temporary.chmod(resolver.stat().st_mode & 0o777) + os.replace(temporary, resolver) + finally: + temporary.unlink(missing_ok=True) + + verified = resolver.read_text(encoding="utf-8") + if verified.count(PATCHED) != 1: + raise RuntimeError("AIPerf Weka timing patch verification failed") + print(f"GO applied AIPerf Weka timing patch: {resolver}") + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--source", required=True, type=Path) + args = parser.parse_args() + try: + apply_patch(args.source) + except (OSError, RuntimeError, ValueError) as exc: + print(f"ERROR: {exc}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/webster/deepseek-v41/scripts/pin-runtime.py b/webster/deepseek-v41/scripts/pin-runtime.py new file mode 100755 index 0000000..81e74ca --- /dev/null +++ b/webster/deepseek-v41/scripts/pin-runtime.py @@ -0,0 +1,406 @@ +#!/usr/bin/env python3 +"""Select an immutable vLLM source pin from fail-closed GitHub evidence.""" + +from __future__ import annotations + +import argparse +import datetime as dt +import json +import os +import re +import subprocess +import sys +import urllib.error +import urllib.parse +import urllib.request +from pathlib import Path + + +REPOSITORY = "vllm-project/vllm" +RUNS_ROOT = Path("/home/ubuntu/deepseek-v41-runs") +FULL_SHA = re.compile(r"^[0-9a-f]{40}$") +PASSING_CHECK_CONCLUSIONS = {"success", "neutral", "skipped"} + + +class PinRejected(RuntimeError): + """The available evidence does not authorize an immutable runtime pin.""" + + +def require_sha(value: object, label: str) -> str: + if not isinstance(value, str) or not FULL_SHA.fullmatch(value): + raise PinRejected(f"{label} is not a full commit SHA") + return value + + +def checks_pass(commit_status: dict, check_runs: dict) -> None: + status_state = commit_status.get("state") + if status_state != "success": + raise PinRejected(f"commit status is {status_state or 'missing'}") + statuses = commit_status.get("statuses") + total_statuses = commit_status.get("total_count") + if not isinstance(statuses, list) or not isinstance(total_statuses, int): + raise PinRejected("commit status evidence is malformed") + if total_statuses < 1 or len(statuses) != total_statuses: + raise PinRejected("commit status evidence is incomplete") + for item in statuses: + if item.get("state") != "success": + raise PinRejected( + "commit status did not pass: " + f"{item.get('context', '')}={item.get('state', 'missing')}" + ) + + runs = check_runs.get("check_runs") + total_runs = check_runs.get("total_count") + if not isinstance(runs, list) or not isinstance(total_runs, int): + raise PinRejected("check-run evidence is malformed") + if total_runs < 1 or len(runs) != total_runs: + raise PinRejected("check-run evidence is incomplete") + for item in runs: + status = item.get("status") + conclusion = item.get("conclusion") + if status != "completed" or conclusion not in PASSING_CHECK_CONCLUSIONS: + raise PinRejected( + "check run did not pass: " + f"{item.get('name', '')}={status}/{conclusion}" + ) + + +def latest_statuses_by_context(history: list[dict]) -> list[dict]: + latest = {} + for item in history: + context = item.get("context") + if not isinstance(context, str) or not context: + raise PinRejected("commit status history contains an unnamed context") + previous = latest.get(context) + if previous is None or (item.get("created_at") or "") > ( + previous.get("created_at") or "" + ): + latest[context] = item + return [latest[context] for context in sorted(latest)] + + +def require_stable_pr(before: dict, after: dict) -> None: + before_state = { + "state": before.get("state"), + "draft": before.get("draft"), + "merged": before.get("merged"), + "mergeable_state": before.get("mergeable_state"), + "updated_at": before.get("updated_at"), + "head_sha": (before.get("head") or {}).get("sha"), + } + after_state = { + "state": after.get("state"), + "draft": after.get("draft"), + "merged": after.get("merged"), + "mergeable_state": after.get("mergeable_state"), + "updated_at": after.get("updated_at"), + "head_sha": (after.get("head") or {}).get("sha"), + } + if before_state != after_state: + raise PinRejected("PR changed during evidence collection") + + +def reviews_pass(reviews: list[dict], head_sha: str) -> None: + latest_by_reviewer = {} + for review in reviews: + user = review.get("user") or {} + login = user.get("login") + if login: + latest_by_reviewer[login] = review + current = [ + review + for review in latest_by_reviewer.values() + if review.get("commit_id") == head_sha + ] + if any(review.get("state") == "CHANGES_REQUESTED" for review in current): + raise PinRejected("current head has a changes-requested review") + if not any(review.get("state") == "APPROVED" for review in current): + raise PinRejected("current head has no approving review") + + +def select_runtime_pin( + pr: dict, + reviews: list[dict], + commit_status: dict, + check_runs: dict, + releases: list[dict], +) -> dict[str, str]: + head_sha = require_sha((pr.get("head") or {}).get("sha"), "PR head") + if pr.get("merged") is True: + for release in releases: + if release.get("draft") or release.get("prerelease"): + continue + if release.get("contains_merge_commit") is True: + tag = release.get("tag_name") + commit = require_sha(release.get("target_commit"), "release commit") + if not isinstance(tag, str) or not tag: + raise PinRejected("containing release has no tag") + return {"commit": commit, "source": f"release:{tag}"} + checks_pass(commit_status, check_runs) + reviews_pass(reviews, head_sha) + return { + "commit": require_sha(pr.get("merge_commit_sha"), "PR merge commit"), + "source": "pr-merge", + } + + if pr.get("state") != "open": + raise PinRejected(f"PR state is {pr.get('state') or 'missing'}") + if pr.get("draft") is not False: + raise PinRejected("PR is draft or draft state is missing") + mergeable_state = pr.get("mergeable_state") + if mergeable_state != "clean": + raise PinRejected(f"PR mergeable_state={mergeable_state or 'missing'}") + checks_pass(commit_status, check_runs) + reviews_pass(reviews, head_sha) + return {"commit": head_sha, "source": "pr-head"} + + +def update_manifest(path: Path, commit: str) -> None: + require_sha(commit, "selected runtime commit") + if path.stat().st_mode & 0o777 != 0o600: + raise PinRejected("manifest mode must be 0600") + lines = path.read_text(encoding="utf-8").splitlines(keepends=True) + indexes = [index for index, line in enumerate(lines) if line.startswith("VLLM_COMMIT=")] + if len(indexes) != 1: + raise PinRejected("manifest must contain exactly one VLLM_COMMIT field") + index = indexes[0] + existing = lines[index].rstrip("\r\n").split("=", 1)[1] + if existing and existing != commit: + raise PinRejected("manifest already pins a different VLLM_COMMIT") + lines[index] = f"VLLM_COMMIT={commit}\n" + temporary = path.with_name(path.name + f".tmp.{os.getpid()}") + descriptor = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + try: + with os.fdopen(descriptor, "w", encoding="utf-8") as stream: + stream.writelines(lines) + stream.flush() + os.fsync(stream.fileno()) + os.replace(temporary, path) + os.chmod(path, 0o600) + finally: + if temporary.exists(): + temporary.unlink() + + +def write_json(path: Path, payload: object) -> None: + temporary = path.with_name(path.name + f".tmp.{os.getpid()}") + descriptor = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + try: + with os.fdopen(descriptor, "w", encoding="utf-8") as stream: + json.dump(payload, stream, indent=2, sort_keys=True) + stream.write("\n") + stream.flush() + os.fsync(stream.fileno()) + os.replace(temporary, path) + os.chmod(path, 0o600) + finally: + if temporary.exists(): + temporary.unlink() + + +def append_event(run_root: Path, decision: str, reason: str) -> None: + clean_reason = reason.replace("\t", " ").replace("\n", " ") + timestamp = dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + with (run_root / "events.tsv").open("a", encoding="utf-8") as stream: + stream.write(f"{timestamp}\truntime-pin\t{decision}\t{clean_reason}\n") + + +def resolve_run_root(manifest: Path) -> Path: + manifest = manifest.resolve() + run_root = manifest.parent + try: + relative = run_root.relative_to(RUNS_ROOT) + except ValueError as error: + raise PinRejected("manifest must be beneath the approved run root") from error + if len(relative.parts) != 1 or not re.fullmatch( + r"[0-9]{8}T[0-9]{6}Z", relative.parts[0] + ): + raise PinRejected("manifest run root must end in one UTC change id") + if manifest.name != "manifest.env" or not manifest.is_file(): + raise PinRejected("manifest.env does not exist") + if manifest.stat().st_mode & 0o777 != 0o600: + raise PinRejected("manifest mode must be 0600") + return run_root + + +class GitHubClient: + def __init__(self, token: str | None): + self.token = token + + def get(self, path: str) -> object: + headers = { + "Accept": "application/vnd.github+json", + "User-Agent": "webster-deepseek-v41-runtime-pin", + "X-GitHub-Api-Version": "2022-11-28", + } + if self.token: + headers["Authorization"] = "Bearer " + self.token + request = urllib.request.Request( + "https://api.github.com" + path, + headers=headers, + ) + try: + with urllib.request.urlopen(request, timeout=30) as response: + return json.load(response) + except urllib.error.HTTPError as error: + raise PinRejected( + f"GitHub API request failed path={path} status={error.code}" + ) from None + + def paginated_list(self, path: str) -> list[dict]: + separator = "&" if "?" in path else "?" + rows = [] + page = 1 + while True: + payload = self.get(f"{path}{separator}per_page=100&page={page}") + if not isinstance(payload, list): + raise PinRejected(f"GitHub API list is malformed path={path}") + rows.extend(payload) + if len(payload) < 100: + return rows + page += 1 + + def all_check_runs(self, commit: str) -> dict: + runs = [] + total = None + page = 1 + while True: + payload = self.get( + f"/repos/{REPOSITORY}/commits/{commit}/check-runs?per_page=100&page={page}" + ) + if not isinstance(payload, dict) or not isinstance(payload.get("check_runs"), list): + raise PinRejected("GitHub check-run evidence is malformed") + if total is None: + total = payload.get("total_count") + page_runs = payload["check_runs"] + runs.extend(page_runs) + if len(page_runs) < 100: + break + page += 1 + return {"total_count": total, "check_runs": runs} + + +def github_token() -> str | None: + for name in ("GITHUB_TOKEN", "GH_TOKEN"): + if os.environ.get(name): + return os.environ[name] + try: + result = subprocess.run( + ["gh", "auth", "token"], + check=True, + capture_output=True, + text=True, + ) + except (FileNotFoundError, subprocess.CalledProcessError): + return None + return result.stdout.strip() or None + + +def release_containment(client: GitHubClient, pr: dict, releases: list[dict]) -> list[dict]: + if pr.get("merged") is not True: + return [] + merge_commit = require_sha(pr.get("merge_commit_sha"), "PR merge commit") + merged_at = pr.get("merged_at") or "" + evidence = [] + for release in releases: + if release.get("draft") or release.get("prerelease"): + continue + if merged_at and (release.get("published_at") or "") < merged_at: + continue + tag = release.get("tag_name") + if not isinstance(tag, str) or not tag: + continue + encoded_tag = urllib.parse.quote(tag, safe="") + target = client.get(f"/repos/{REPOSITORY}/commits/{encoded_tag}") + if not isinstance(target, dict): + raise PinRejected(f"release target evidence is malformed tag={tag}") + target_commit = require_sha(target.get("sha"), f"release {tag} commit") + comparison = client.get( + f"/repos/{REPOSITORY}/compare/{merge_commit}...{target_commit}" + ) + if not isinstance(comparison, dict): + raise PinRejected(f"release containment evidence is malformed tag={tag}") + evidence.append( + { + "tag_name": tag, + "target_commit": target_commit, + "contains_merge_commit": comparison.get("status") in {"ahead", "identical"}, + "compare_status": comparison.get("status"), + "draft": False, + "prerelease": False, + } + ) + return evidence + + +def collect_evidence(client: GitHubClient, pr_number: int) -> dict[str, object]: + pr_path = f"/repos/{REPOSITORY}/pulls/{pr_number}" + pr = client.get(pr_path) + if not isinstance(pr, dict): + raise PinRejected("GitHub PR evidence is malformed") + head_sha = require_sha((pr.get("head") or {}).get("sha"), "PR head") + reviews = client.paginated_list(pr_path + "/reviews") + combined_status = client.get(f"/repos/{REPOSITORY}/commits/{head_sha}/status") + if not isinstance(combined_status, dict): + raise PinRejected("GitHub commit-status evidence is malformed") + status_history = client.paginated_list( + f"/repos/{REPOSITORY}/commits/{head_sha}/statuses" + ) + combined_status["statuses"] = latest_statuses_by_context(status_history) + check_runs = client.all_check_runs(head_sha) + releases = client.paginated_list(f"/repos/{REPOSITORY}/releases") + containment = release_containment(client, pr, releases) + pr_recheck = client.get(pr_path) + if not isinstance(pr_recheck, dict): + raise PinRejected("GitHub PR recheck evidence is malformed") + require_stable_pr(pr, pr_recheck) + return { + "pr": pr, + "pr-recheck": pr_recheck, + "reviews": reviews, + "commit-status": combined_status, + "check-runs": check_runs, + "releases": releases, + "release-containment": containment, + } + + +def parse_args(argv: list[str]) -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--pr", type=int, required=True) + parser.add_argument("--manifest", type=Path, required=True) + return parser.parse_args(argv) + + +def main(argv: list[str]) -> int: + args = parse_args(argv) + run_root = resolve_run_root(args.manifest) + evidence_dir = run_root / "runtime-pin" + evidence_dir.mkdir(mode=0o700, exist_ok=True) + os.chmod(evidence_dir, 0o700) + try: + evidence = collect_evidence(GitHubClient(github_token()), args.pr) + for name, payload in evidence.items(): + write_json(evidence_dir / f"{name}.json", payload) + selected = select_runtime_pin( + evidence["pr"], + evidence["reviews"], + evidence["commit-status"], + evidence["check-runs"], + evidence["release-containment"], + ) + update_manifest(args.manifest, selected["commit"]) + except PinRejected as error: + reason = str(error) + write_json(evidence_dir / "decision.json", {"decision": "NO-GO", "reason": reason}) + append_event(run_root, "NO-GO", reason) + print(f"NO-GO runtime-pin: {reason}", file=sys.stderr) + return 1 + write_json(evidence_dir / "decision.json", {"decision": "GO", **selected}) + append_event(run_root, "GO", f"source={selected['source']} commit={selected['commit']}") + print(f"GO runtime-pin source={selected['source']} commit={selected['commit']}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv[1:])) diff --git a/webster/deepseek-v41/scripts/preflight.sh b/webster/deepseek-v41/scripts/preflight.sh new file mode 100755 index 0000000..b2d76fb --- /dev/null +++ b/webster/deepseek-v41/scripts/preflight.sh @@ -0,0 +1,226 @@ +#!/usr/bin/env bash +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$script_dir/common.sh" + +phase="" +run_root="" +while (($#)); do + case "$1" in + --phase) + [[ $# -ge 2 ]] || die "--phase requires a value" + phase="$2" + shift 2 + ;; + --run-root) + [[ $# -ge 2 ]] || die "--run-root requires a value" + run_root="$2" + shift 2 + ;; + *) die "unknown argument: $1" ;; + esac +done + +validate_phase "$phase" +validated_root="$(validate_run_root "$run_root")" + +station_free_bytes() { + local node="$1" value + value="$(ssh "$node" df -PB1 /home/alecfong | awk 'NR == 2 {print $4}')" + [[ "$value" =~ ^[0-9]+$ ]] || die "$node free-space query returned a non-integer" + printf '%s\n' "$value" +} + +validate_station_owner_state() { + local checked_phase="$1" glm_shamu="$2" glm_tilikum="$3" + local deepseek_shamu="$4" deepseek_tilikum="$5" + case "$checked_phase" in + baseline|stage|alias) + require_eq "Shamu GLM running" "true" "$glm_shamu" + require_eq "Tilikum GLM running" "true" "$glm_tilikum" + require_eq "Shamu DeepSeek stopped" "false" "$deepseek_shamu" + require_eq "Tilikum DeepSeek stopped" "false" "$deepseek_tilikum" + ;; + canary) + require_eq "Shamu GLM stopped" "false" "$glm_shamu" + require_eq "Tilikum GLM stopped" "false" "$glm_tilikum" + require_eq "Shamu DeepSeek stopped" "false" "$deepseek_shamu" + require_eq "Tilikum DeepSeek stopped" "false" "$deepseek_tilikum" + ;; + publish) + require_eq "Shamu GLM stopped" "false" "$glm_shamu" + require_eq "Tilikum GLM stopped" "false" "$glm_tilikum" + require_eq "Shamu DeepSeek running" "true" "$deepseek_shamu" + require_eq "Tilikum DeepSeek running" "true" "$deepseek_tilikum" + ;; + esac +} + +validate_publish_acceptance() { + [[ "$phase" == "publish" ]] || return 0 + local verifier="$script_dir/verify-private-acceptance.py" + if [[ "${WEBSTER_PREFLIGHT_TEST_MODE:-0}" == "1" && + -n "${TEST_PRIVATE_ACCEPTANCE_VERIFIER:-}" ]]; then + verifier="$TEST_PRIVATE_ACCEPTANCE_VERIFIER" + fi + [[ -f "$validated_root/private-acceptance.json" ]] || + die "publish phase requires private-acceptance.json" + "$verifier" \ + --run-root "$validated_root" \ + --acceptance "$validated_root/private-acceptance.json" \ + --max-age-seconds 21600 +} + +test_mode_checks() { + if [[ "$phase" == "stage" ]]; then + require_ge "free bytes before staging" "$MIN_FREE_BEFORE_STAGE_BYTES" \ + "${TEST_FREE_BYTES:-$MIN_FREE_BEFORE_STAGE_BYTES}" + fi + [[ -z "${TEST_MISSING_TOPOLOGY:-}" ]] || + die "missing required topology: $TEST_MISSING_TOPOLOGY" + local mode="${TEST_CREDENTIAL_MODE:-0600}" + mode="${mode#0}" + require_eq "credential mode" "600" "$mode" + local inspect_path="${TEST_LITELLM_INSPECT:?TEST_LITELLM_INSPECT is required}" + "$script_dir/verify-litellm-container.py" "$inspect_path" >/dev/null + require_eq "current model probe status" "ok" \ + "${TEST_CURRENT_MODEL_PROBE_STATUS:-ok}" + local glm_default=true deepseek_default=false + if [[ "$phase" == "canary" || "$phase" == "publish" ]]; then + glm_default=false + fi + if [[ "$phase" == "publish" ]]; then + deepseek_default=true + fi + validate_station_owner_state "$phase" \ + "${TEST_PREFLIGHT_GLM_SHAMU_RUNNING:-$glm_default}" \ + "${TEST_PREFLIGHT_GLM_TILIKUM_RUNNING:-$glm_default}" \ + "${TEST_PREFLIGHT_DEEPSEEK_SHAMU_RUNNING:-$deepseek_default}" \ + "${TEST_PREFLIGHT_DEEPSEEK_TILIKUM_RUNNING:-$deepseek_default}" + validate_publish_acceptance + if [[ "${TEST_VALIDATE_STATION_FREE:-0}" == "1" ]]; then + station_free_bytes shamu >/dev/null + station_free_bytes tilikum >/dev/null + fi + ssh shamu preflight-test >/dev/null + ssh tilikum preflight-test >/dev/null +} + +if [[ "${WEBSTER_PREFLIGHT_TEST_MODE:-0}" == "1" ]]; then + test_mode_checks + exit 0 +fi + +init_run_root "$validated_root" +write_summary_header "$phase" +event "$phase" START "read-only preflight" + +inspect_path="$RUN_ROOT/baseline/litellm.inspect.redacted.json" +inspect_temporary="$inspect_path.tmp.$$" +ssh spark-1 'docker inspect litellm' | + python3 -c 'import json,sys; data=json.load(sys.stdin); data[0].get("Config", {})["Env"]=["REDACTED"]; json.dump(data,sys.stdout); print()' \ + >"$inspect_temporary" +chmod 0600 "$inspect_temporary" +mv -- "$inspect_temporary" "$inspect_path" +capture litellm-container-verify "$script_dir/verify-litellm-container.py" "$inspect_path" + +capture git-state bash -c \ + 'git rev-parse HEAD; git status --short --branch; sha256sum docs/superpowers/specs/2026-09-11-glm52-to-deepseek-v41-cutover-design.md' +capture brev-instances brev ls +capture brev-nodes brev ls nodes + +station_inspect_format='--format "id={{.Id}} image={{.Image}} started={{.State.StartedAt}} restarts={{.RestartCount}} running={{.State.Running}} args={{json .Args}}"' +station_inspect_shamu="docker inspect '$GLM_CONTAINER'" +station_inspect_tilikum="sudo -n docker inspect '$GLM_CONTAINER'" +if [[ "$phase" == "publish" ]]; then + station_inspect_shamu="docker inspect '$DEEPSEEK_CONTAINER'" + station_inspect_tilikum="sudo -n docker inspect '$DEEPSEEK_CONTAINER'" +elif [[ "$phase" == "canary" ]]; then + station_inspect_format="" + station_inspect_shamu="docker ps -a --filter name='$GLM_CONTAINER' --filter name='$DEEPSEEK_CONTAINER' --format 'id={{.ID}} image={{.Image}} status={{.Status}} names={{.Names}}'" + station_inspect_tilikum="sudo -n docker ps -a --filter name='$GLM_CONTAINER' --filter name='$DEEPSEEK_CONTAINER' --format 'id={{.ID}} image={{.Image}} status={{.Status}} names={{.Names}}'" +fi + +capture shamu-host ssh shamu \ + "set -eu; hostname; date -u +%Y-%m-%dT%H:%M:%SZ; uptime; uname -a; nvidia-smi --query-gpu=name,driver_version,memory.total,memory.used --format=csv,noheader; free -b; df -PB1 /home/alecfong; ip -o addr show; cat /sys/class/infiniband/mlx5_1/ports/1/state; $station_inspect_shamu $station_inspect_format; ss -lntp" +capture tilikum-host ssh tilikum \ + "set -eu; hostname; date -u +%Y-%m-%dT%H:%M:%SZ; uptime; uname -a; nvidia-smi --query-gpu=name,driver_version,memory.total,memory.used --format=csv,noheader; free -b; df -PB1 /home/alecfong; ip -o addr show; cat /sys/class/infiniband/mlx5_1/ports/1/state; $station_inspect_tilikum $station_inspect_format; ss -lntp" + +shamu_free="$(station_free_bytes shamu)" +tilikum_free="$(station_free_bytes tilikum)" +if [[ "$phase" == "stage" ]]; then + require_ge "Shamu free bytes before staging" "$MIN_FREE_BEFORE_STAGE_BYTES" "$shamu_free" + require_ge "Tilikum free bytes before staging" "$MIN_FREE_BEFORE_STAGE_BYTES" "$tilikum_free" +fi + +for node in shamu tilikum; do + rail="$SHAMU_RAIL" + [[ "$node" == "tilikum" ]] && rail="$TILIKUM_RAIL" + ssh "$node" "ip link show '$NCCL_IFACE' >/dev/null; ip addr show dev '$NCCL_IFACE' | grep -F '$rail/' >/dev/null; grep -q ACTIVE /sys/class/infiniband/'$NCCL_HCA'/ports/1/state" || + die "$node station rail/HCA validation failed" +done +capture rail-ping-shamu ssh shamu "ping -c 3 -W 2 -I '$NCCL_IFACE' '$TILIKUM_RAIL'" +capture rail-ping-tilikum ssh tilikum "ping -c 3 -W 2 -I '$NCCL_IFACE' '$SHAMU_RAIL'" + +glm_shamu_running="$(ssh shamu "docker inspect glm52-full-mtp --format '{{.State.Running}}' 2>/dev/null || printf false")" +glm_tilikum_running="$(ssh tilikum "sudo -n docker inspect glm52-full-mtp --format '{{.State.Running}}' 2>/dev/null || printf false")" +deepseek_shamu_running="$(ssh shamu "docker inspect '$DEEPSEEK_CONTAINER' --format '{{.State.Running}}' 2>/dev/null || printf false")" +deepseek_tilikum_running="$(ssh tilikum "sudo -n docker inspect '$DEEPSEEK_CONTAINER' --format '{{.State.Running}}' 2>/dev/null || printf false")" +validate_station_owner_state "$phase" \ + "$glm_shamu_running" "$glm_tilikum_running" \ + "$deepseek_shamu_running" "$deepseek_tilikum_running" +if [[ "$phase" == "baseline" || "$phase" == "stage" || "$phase" == "alias" ]]; then + ssh shamu "curl -fsS --max-time 5 http://$SHAMU_NETBIRD:$CANARY_PORT/health >/dev/null" || + die "GLM rank 0 health check failed" + ssh tilikum "sudo -n docker inspect glm52-full-mtp --format '{{json .Args}}' | grep -q -- --headless" || + die "GLM rank 1 is not headless" +fi + +require_file_mode_600 shamu /home/alecfong/.config/glm52/vllm-api-key +require_file_mode_600 spark-1 /home/nvidia/litellm/config.yaml + +capture litellm-shape ssh spark-1 \ + 'docker inspect litellm --format "id={{.Id}} image={{.Image}} started={{.State.StartedAt}} restarts={{.RestartCount}} running={{.State.Running}} network={{.HostConfig.NetworkMode}} entrypoint={{json .Config.Entrypoint}} cmd={{json .Config.Cmd}} ports={{json .HostConfig.PortBindings}}"; ss -lntp | grep 4446' +capture litellm-config ssh spark-1 \ + 'python3 - <<'"'"'PY'"'"' +import json +from pathlib import Path +import yaml +path = Path("/home/nvidia/litellm/config.yaml") +data = yaml.safe_load(path.read_text()) +models = [] +for item in data.get("model_list", []): + info = item.get("model_info", {}) + params = item.get("litellm_params", {}) + models.append({ + "model_name": item.get("model_name"), + "model": params.get("model"), + "api_base": params.get("api_base"), + "max_input_tokens": info.get("max_input_tokens"), + "max_output_tokens": info.get("max_output_tokens"), + }) +print(json.dumps({ + "callbacks": data.get("litellm_settings", {}).get("callbacks", []), + "plugins": data.get("router_settings", {}).get("plugins", []), + "models": models, +}, sort_keys=True)) +PY' + +root_status="$(ssh spark-1 "curl -sS -o /dev/null -w '%{http_code}' http://127.0.0.1:4444/")" +models_status="$(ssh spark-1 "curl -sS -o /dev/null -w '%{http_code}' http://127.0.0.1:4444/v1/models")" +require_eq "Caddy root status" "403" "$root_status" +require_eq "unauthenticated model-list status" "401" "$models_status" + +capture current-model-probes ssh spark-1 python3 - "$(basename "$RUN_ROOT")" \ + <"$script_dir/current-model-probes.py" + +capture prometheus-targets curl -fsS --max-time 10 \ + "http://$SHAMU_NETBIRD:9095/api/v1/targets?state=active" +capture langfuse-health curl -fsS --max-time 10 \ + "http://$SHAMU_NETBIRD:3000/api/public/health" + +validate_publish_acceptance + +event "$phase" GO "all read-only preflight requirements passed" +printf 'GO phase=%s run_root=%s\n' "$phase" "$RUN_ROOT" diff --git a/webster/deepseek-v41/scripts/publication-model-probes.py b/webster/deepseek-v41/scripts/publication-model-probes.py new file mode 100755 index 0000000..1bc84d3 --- /dev/null +++ b/webster/deepseek-v41/scripts/publication-model-probes.py @@ -0,0 +1,275 @@ +#!/usr/bin/env python3 +"""Run the authenticated public model and ingress publication probes.""" + +from __future__ import annotations + +import base64 +import json +import os +import struct +import subprocess +import sys +import urllib.error +import urllib.request +import zlib +from typing import Any + + +PUBLIC_API = ( + "https://webster-models-extnode-3gdrajbr0hiykknxzitck9yaiwo.apps.run.brev.nvidia.com" +) +ADMIN_API = "http://127.0.0.1:4446" +PRIVATE_API = "http://127.0.0.1:4444" +MIGRATION_CRITICAL_MODELS = ( + "deepseek-v4.1-flash", + "glm-5.2", + "glm-5.3-flash", + "deepseek-v4-flash", +) + + +def require(condition: bool, message: str) -> None: + if not condition: + raise RuntimeError(message) + + +def read_json_payload(response: Any) -> Any: + raw = response.read() + if not raw: + return None + try: + return json.loads(raw) + except (UnicodeDecodeError, json.JSONDecodeError): + return None + + +def request( + base: str, + path: str, + body: dict[str, Any] | None = None, + key: str | None = None, + timeout: int = 180, +) -> tuple[int, Any]: + headers = {} + data = None + if key is not None: + headers["Authorization"] = "Bearer " + key + if body is not None: + headers["Content-Type"] = "application/json" + data = json.dumps(body, separators=(",", ":")).encode() + value = urllib.request.Request( + base + path, + data=data, + headers=headers, + method="POST" if body is not None else "GET", + ) + try: + with urllib.request.urlopen(value, timeout=timeout) as response: + return response.status, read_json_payload(response) + except urllib.error.HTTPError as error: + return error.code, read_json_payload(error) + + +def png_chunk(kind: bytes, data: bytes) -> bytes: + checksum = zlib.crc32(kind + data) & 0xFFFFFFFF + return struct.pack(">I", len(data)) + kind + data + struct.pack(">I", checksum) + + +def red_png_data_url() -> str: + width = height = 560 + raw = b"".join(b"\x00" + b"\xff\x00\x00" * width for _ in range(height)) + png = ( + b"\x89PNG\r\n\x1a\n" + + png_chunk(b"IHDR", struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0)) + + png_chunk(b"IDAT", zlib.compress(raw, 9)) + + png_chunk(b"IEND", b"") + ) + return "data:image/png;base64," + base64.b64encode(png).decode("ascii") + + +def probe_legacy_vision_rejection(public_api: str, key: str) -> int: + status, payload = request( + public_api, + "/v1/chat/completions", + { + "model": "glm-5.2", + "messages": [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Describe this image."}, + { + "type": "image_url", + "image_url": {"url": red_png_data_url()}, + }, + ], + } + ], + "max_tokens": 16, + "reasoning_effort": "none", + }, + key, + ) + require(status == 400, "legacy alias accepted image input") + error = payload.get("error") if isinstance(payload, dict) else None + code = error.get("code") if isinstance(error, dict) else None + require( + code == "unsupported_vision", + "legacy alias image rejection did not come from the compatibility guard " + f"(expected unsupported_vision, got {code!r})", + ) + return status + + +def master_key() -> str: + override = os.environ.get("WEBSTER_PUBLICATION_MASTER_KEY") + if override: + return override + environment = subprocess.check_output( + [ + "docker", + "inspect", + "litellm", + "--format", + "{{range .Config.Env}}{{println .}}{{end}}", + ], + text=True, + ).splitlines() + return next( + item.split("=", 1)[1] + for item in environment + if item.startswith("LITELLM_MASTER_KEY=") + ) + + +def run(probe_id: str) -> dict[str, Any]: + public_api = os.environ.get("WEBSTER_PUBLICATION_PUBLIC_API", PUBLIC_API).rstrip("/") + admin_api = os.environ.get("WEBSTER_PUBLICATION_ADMIN_API", ADMIN_API).rstrip("/") + private_api = os.environ.get("WEBSTER_PUBLICATION_PRIVATE_API", PRIVATE_API).rstrip("/") + master = master_key() + + def admin(path: str, body: dict[str, Any]) -> tuple[int, Any]: + return request(admin_api, path, body, master, 30) + + key = None + results: dict[str, Any] = {} + try: + info_status, info = request(admin_api, "/model/info", key=master) + require( + info_status == 200 and isinstance(info, dict), + "model info request failed", + ) + rows = info.get("data", info) + require(isinstance(rows, list), "model info response is malformed") + by_name = { + row.get("model_name"): row + for row in rows + if isinstance(row, dict) + } + missing_models = [ + model for model in MIGRATION_CRITICAL_MODELS if model not in by_name + ] + require( + not missing_models, + "migration-critical models are absent from the current registry: " + + ", ".join(missing_models), + ) + models = list(MIGRATION_CRITICAL_MODELS) + status, created = admin( + "/key/generate", + { + "key_alias": "deepseek-v41-publication-acceptance", + "models": models, + "max_budget": 1.0, + "budget_duration": "1d", + }, + ) + require(status == 200, "temporary publication key creation failed") + require(isinstance(created, dict) and created.get("key"), "temporary publication key response is malformed") + key = created["key"] + for model in models: + body: dict[str, Any] = { + "model": model, + "messages": [{"role": "user", "content": "Reply with exactly OK."}], + "max_tokens": 16, + "temperature": 0, + "reasoning_effort": "none", + } + if model == "deepseek-v4.1-flash": + body["user"] = probe_id + body["metadata"] = {"acceptance_probe_id": probe_id} + response_status, payload = request( + public_api, "/v1/chat/completions", body, key + ) + require( + response_status == 200 + and isinstance(payload, dict) + and payload.get("choices"), + f"public probe failed for {model}", + ) + require(payload.get("model") == model, f"public response model differs for {model}") + results[model] = response_status + if model == "deepseek-v4.1-flash": + results["acceptance_probe_id"] = probe_id + results["glm-5.2-vision-rejection"] = probe_legacy_vision_rejection( + public_api, key + ) + root, _ = request(public_api, "/") + unauthenticated, _ = request(public_api, "/v1/models") + admin_denied, _ = request(public_api, "/key/info") + require( + (root, unauthenticated, admin_denied) == (403, 401, 403), + "ingress security responses are wrong", + ) + deepseek = by_name["deepseek-v4.1-flash"]["model_info"] + require( + deepseek["max_input_tokens"] + == deepseek["max_output_tokens"] + == 1048576, + "DeepSeek context metadata is wrong", + ) + require(deepseek["supports_vision"] is True, "DeepSeek vision metadata is wrong") + require( + deepseek["supports_function_calling"] is True, + "DeepSeek tool metadata is wrong", + ) + require( + deepseek["supports_reasoning"] is True, + "DeepSeek reasoning metadata is wrong", + ) + require( + deepseek["supports_response_schema"] is True, + "DeepSeek schema metadata is wrong", + ) + legacy = by_name["glm-5.2"]["model_info"] + require( + legacy["max_input_tokens"] + == legacy["max_output_tokens"] + == 320000, + "legacy context metadata is wrong", + ) + require(legacy["supports_vision"] is False, "legacy vision metadata is wrong") + results["ingress"] = { + "root": root, + "unauthenticated_models": unauthenticated, + "admin": admin_denied, + } + finally: + if key is not None: + delete_status, _ = admin("/key/delete", {"keys": [key]}) + require(delete_status == 200, "temporary publication key revocation failed") + revoked, _ = request(private_api, "/v1/models", key=key) + require(revoked == 401, "revoked publication key is still accepted") + results["revoked_key_status"] = revoked + return results + + +def main() -> int: + if len(sys.argv) != 2: + raise SystemExit("usage: publication-model-probes.py PROBE_ID") + print(json.dumps(run(sys.argv[1]), sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/webster/deepseek-v41/scripts/render-litellm-cutover.py b/webster/deepseek-v41/scripts/render-litellm-cutover.py new file mode 100755 index 0000000..1a25bd3 --- /dev/null +++ b/webster/deepseek-v41/scripts/render-litellm-cutover.py @@ -0,0 +1,172 @@ +#!/usr/bin/env python3 +"""Render the GLM-5.2 compatibility alias without exposing backend credentials.""" + +from __future__ import annotations + +import argparse +import importlib.util +import json +import os +import stat +import sys +import tempfile +from pathlib import Path + +import yaml + + +HERE = Path(__file__).resolve().parent +VERIFIER_PATH = HERE / "verify-litellm-config.py" + + +def load_verifier(): + spec = importlib.util.spec_from_file_location("verify_litellm_config", VERIFIER_PATH) + if spec is None or spec.loader is None: + raise RuntimeError("cannot load LiteLLM configuration verifier") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("before", type=Path) + parser.add_argument("output", type=Path) + parser.add_argument("--phase", choices=("alias", "publish"), default="alias") + parser.add_argument("--api-key-file", type=Path) + parser.add_argument("--private-acceptance", type=Path) + parser.add_argument("--served-model-name") + parser.add_argument("--max-context", type=int) + parser.add_argument("--input-cost-per-token", type=float) + parser.add_argument("--output-cost-per-token", type=float) + parser.add_argument("--supports-vision", action="store_true") + return parser.parse_args() + + +def read_private_key(path: Path | None, verifier) -> str: + if path is None: + raise verifier.ValidationError("--api-key-file is required for publish") + if not path.is_file() or path.is_symlink(): + raise verifier.ValidationError("backend key must be a regular non-symlink file") + if stat.S_IMODE(path.stat().st_mode) != 0o600: + raise verifier.ValidationError("backend key file mode must be 0600") + value = path.read_text(encoding="utf-8").strip() + if not value: + raise verifier.ValidationError("backend key file is empty") + return value + + +def read_accepted_publication(path: Path | None, verifier) -> dict[str, object]: + if path is None: + raise verifier.ValidationError("--private-acceptance is required for publish") + if not path.is_file() or path.is_symlink(): + raise verifier.ValidationError( + "private acceptance must be a regular non-symlink file" + ) + if stat.S_IMODE(path.stat().st_mode) != 0o600: + raise verifier.ValidationError("private acceptance file mode must be 0600") + try: + root = json.loads(path.read_text(encoding="utf-8")) + except (UnicodeDecodeError, json.JSONDecodeError) as error: + raise verifier.ValidationError("private acceptance is malformed") from error + if not isinstance(root, dict) or root.get("decision") != "GO": + raise verifier.ValidationError("private acceptance decision is not GO") + publication = root.get("publication") + expected_keys = { + "public_model_name", + "served_model_name", + "max_context", + "station_endpoint", + "supports_function_calling", + "supports_reasoning", + "supports_response_schema", + "supports_vision", + "input_cost_per_token", + "output_cost_per_token", + } + if not isinstance(publication, dict) or set(publication) != expected_keys: + raise verifier.ValidationError("accepted publication record is incomplete") + if ( + publication.get("public_model_name") != verifier.PUBLISHED_MODEL + or publication.get("station_endpoint") != verifier.PUBLISHED_API_BASE + or publication.get("served_model_name") + != "deepseek-ai/DeepSeek-V4.1-Flash" + or publication.get("max_context") != 1_048_576 + or publication.get("supports_function_calling") is not True + or publication.get("supports_reasoning") is not True + or publication.get("supports_response_schema") is not True + or publication.get("supports_vision") is not True + or publication.get("input_cost_per_token") != 1.3e-8 + or publication.get("output_cost_per_token") != 9.6e-7 + ): + raise verifier.ValidationError( + "accepted publication record does not match the qualified profile" + ) + return publication + + +def main() -> int: + args = parse_args() + verifier = load_verifier() + temporary_path: Path | None = None + try: + before_resolved = args.before.resolve(strict=True) + output_resolved = args.output.resolve(strict=False) + if before_resolved == output_resolved: + raise verifier.ValidationError("refusing to overwrite the source configuration") + before = verifier.load_config(before_resolved) + if args.phase == "alias": + candidate = verifier.expected_alias_config(before) + else: + publication = read_accepted_publication(args.private_acceptance, verifier) + supplied = { + "served_model_name": args.served_model_name, + "max_context": args.max_context, + "input_cost_per_token": args.input_cost_per_token, + "output_cost_per_token": args.output_cost_per_token, + } + for name, value in supplied.items(): + if value is not None and value != publication[name]: + raise verifier.ValidationError( + f"caller {name} differs from the accepted publication record" + ) + if args.supports_vision and publication["supports_vision"] is not True: + raise verifier.ValidationError( + "caller vision flag differs from the accepted publication record" + ) + candidate = verifier.expected_publish_config( + before, + api_key=read_private_key(args.api_key_file, verifier), + served_model_name=str(publication["served_model_name"]), + max_context=int(publication["max_context"]), + input_cost_per_token=float(publication["input_cost_per_token"]), + output_cost_per_token=float(publication["output_cost_per_token"]), + supports_vision=bool(publication["supports_vision"]), + ) + args.output.parent.mkdir(parents=True, exist_ok=True) + descriptor, temporary = tempfile.mkstemp( + prefix=f".{args.output.name}.", dir=args.output.parent + ) + temporary_path = Path(temporary) + with os.fdopen(descriptor, "w", encoding="utf-8") as handle: + yaml.safe_dump(candidate, handle, sort_keys=False) + os.chmod(temporary_path, 0o600) + rendered = verifier.load_config(temporary_path) + if args.phase == "alias": + verifier.validate_alias(before, rendered) + else: + verifier.validate_publish(before, rendered) + os.replace(temporary_path, args.output) + temporary_path = None + os.chmod(args.output, 0o600) + except (OSError, RuntimeError, yaml.YAMLError) as error: + if temporary_path is not None: + temporary_path.unlink(missing_ok=True) + print(f"ERROR: {error}", file=sys.stderr) + return 1 + print(f"rendered validated LiteLLM {args.phase} candidate at {args.output}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/webster/deepseek-v41/scripts/restore-litellm-config.sh b/webster/deepseek-v41/scripts/restore-litellm-config.sh new file mode 100755 index 0000000..fb12092 --- /dev/null +++ b/webster/deepseek-v41/scripts/restore-litellm-config.sh @@ -0,0 +1,180 @@ +#!/usr/bin/env bash +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$script_dir/common.sh" + +phase="" +run_root="" +while (($#)); do + case "$1" in + --phase) + [[ $# -ge 2 ]] || die "--phase requires a value" + phase="$2" + shift 2 + ;; + --run-root) + [[ $# -ge 2 ]] || die "--run-root requires a value" + run_root="$2" + shift 2 + ;; + *) die "unknown argument: $1" ;; + esac +done + +case "$phase" in alias|publish) ;; *) die "phase must be alias or publish" ;; esac +validated_root="$(validate_run_root "$run_root")" +init_run_root "$validated_root" +rollback_file="$validated_root/rollback.env" +[[ -f "$rollback_file" && ! -L "$rollback_file" ]] || die "rollback.env is missing" +[[ "$(stat -c %a "$rollback_file")" == "600" ]] || die "rollback.env mode must be 0600" + +declare -A rollback +while IFS='=' read -r key value; do + [[ -z "$key" || "$key" == \#* ]] && continue + case "$key" in + ALIAS_CONFIG_BACKUP|ALIAS_CONFIG_SHA256|PUBLISH_CONFIG_BACKUP|PUBLISH_CONFIG_SHA256) + rollback["$key"]="$value" + ;; + *) die "unknown rollback.env field: $key" ;; + esac +done <"$rollback_file" + +prefix="${phase^^}" +backup_path="${rollback[${prefix}_CONFIG_BACKUP]:-}" +expected_hash="${rollback[${prefix}_CONFIG_SHA256]:-}" +[[ "$backup_path" == /home/nvidia/litellm/* && "$backup_path" != *'/../'* ]] || + die "rollback backup path must remain under /home/nvidia/litellm" +[[ "$expected_hash" =~ ^[0-9a-f]{64}$ ]] || die "rollback SHA-256 is invalid" + +if [[ "${WEBSTER_LITELLM_TEST_MODE:-0}" == "1" ]]; then + test_root="${TEST_LITELLM_ROOT:?TEST_LITELLM_ROOT is required in test mode}" + mapped_backup="$test_root/${backup_path#/home/nvidia/litellm/}" + live_config="$test_root/config.yaml" + [[ -f "$mapped_backup" && ! -L "$mapped_backup" ]] || die "rollback backup is missing" + [[ "$(stat -c %a "$mapped_backup")" == "600" ]] || die "rollback backup mode must be 0600" + actual_hash="$(sha256sum "$mapped_backup" | awk '{print $1}')" + require_eq "rollback backup SHA-256" "$expected_hash" "$actual_hash" + if [[ ! -f "$live_config" ]] || ! cmp -s "$mapped_backup" "$live_config"; then + install -m 0600 "$mapped_backup" "$live_config.tmp-$$" + mv -- "$live_config.tmp-$$" "$live_config" + docker restart litellm >/dev/null + fi + chmod 0600 "$live_config" + [[ "$(stat -c %a "$live_config")" == "600" ]] || + die "restored config mode must be 0600" + [[ "${TEST_RESTORE_HEALTH:-ok}" == "ok" ]] || die "test restore health failed" + [[ "${TEST_RESTORE_READINESS:-ok}" == "ok" ]] || + die "test restore readiness failed" + event "$phase-rollback" GO "hash-verified config restored and ingress checks passed" + printf 'LiteLLM %s rollback validated in test mode\n' "$phase" + exit 0 +fi + +remote_backup="$(ssh spark-1 "realpath -e -- '$backup_path'")" || die "rollback backup is missing" +[[ "$remote_backup" == /home/nvidia/litellm/* ]] || die "resolved rollback path escaped" +remote_mode="$(ssh spark-1 "stat -c %a -- '$remote_backup'")" +require_eq "rollback backup mode" "600" "$remote_mode" +actual_hash="$(ssh spark-1 "sha256sum -- '$remote_backup' | awk '{print \$1}'")" +require_eq "rollback backup SHA-256" "$expected_hash" "$actual_hash" +live_hash="$(ssh spark-1 "sha256sum /home/nvidia/litellm/config.yaml | awk '{print \$1}'")" +if [[ "$live_hash" != "$expected_hash" ]]; then + ssh spark-1 "set -eu; install -m 0600 '$remote_backup' /home/nvidia/litellm/config.yaml.restore-$$; mv /home/nvidia/litellm/config.yaml.restore-$$ /home/nvidia/litellm/config.yaml; docker restart litellm >/dev/null" + ready=0 + for _ in $(seq 1 60); do + if ssh spark-1 "curl -fsS --max-time 2 http://127.0.0.1:4446/health/liveliness >/dev/null && curl -fsS --max-time 2 http://127.0.0.1:4446/health/readiness >/dev/null"; then + ready=1 + break + fi + sleep 1 + done + [[ "$ready" == "1" ]] || die "LiteLLM did not recover within 60 seconds" +fi +ssh spark-1 "chmod 0600 /home/nvidia/litellm/config.yaml" +live_mode_owner="$(ssh spark-1 "stat -c %a:%U /home/nvidia/litellm/config.yaml")" +require_eq "restored config mode and owner" "600:nvidia" "$live_mode_owner" +for endpoint in liveliness readiness; do + status="$(ssh spark-1 "curl -sS --max-time 3 -o /dev/null -w '%{http_code}' http://127.0.0.1:4446/health/$endpoint")" + require_eq "LiteLLM $endpoint status" "200" "$status" +done + +root_status="$(ssh spark-1 "curl -sS -o /dev/null -w '%{http_code}' http://127.0.0.1:4444/")" +models_status="$(ssh spark-1 "curl -sS -o /dev/null -w '%{http_code}' http://127.0.0.1:4444/v1/models")" +require_eq "Caddy root status" "403" "$root_status" +require_eq "unauthenticated model-list status" "401" "$models_status" +binds="$(ssh spark-1 "ss -lntH '( sport = :4446 )' | awk '{print \$4}'")" +[[ -n "$binds" ]] || die "LiteLLM is not listening on port 4446" +if grep -vE '^(127\.0\.0\.1|\[::1\]):4446$' <<<"$binds" | grep -q .; then + die "LiteLLM port 4446 is not loopback-only" +fi +ssh spark-1 python3 - "$phase" "$(basename "$validated_root")" <<'PY' +import json +import subprocess +import sys +import urllib.request + +phase, change_id = sys.argv[1:] +environment = subprocess.check_output( + [ + "docker", + "inspect", + "litellm", + "--format", + "{{range .Config.Env}}{{println .}}{{end}}", + ], + text=True, +).splitlines() +master = next( + line.split("=", 1)[1] + for line in environment + if line.startswith("LITELLM_MASTER_KEY=") +) + + +def post(url, body, key): + request = urllib.request.Request( + url, + data=json.dumps(body, separators=(",", ":")).encode("utf-8"), + headers={ + "Authorization": "Bearer " + key, + "Content-Type": "application/json", + }, + method="POST", + ) + with urllib.request.urlopen(request, timeout=120) as response: + return response.status, json.load(response) + + +virtual = None +try: + _, created = post( + "http://127.0.0.1:4446/key/generate", + { + "key_alias": f"rollback-{phase}-{change_id}", + "models": ["glm-5.2", "glm-5.3-flash", "inkling-small-nvfp4"], + "max_budget": 0.1, + "budget_duration": "1d", + }, + master, + ) + virtual = created["key"] + for model in ("glm-5.2", "glm-5.3-flash", "inkling-small-nvfp4"): + status, payload = post( + "http://127.0.0.1:4444/v1/chat/completions", + { + "model": model, + "messages": [{"role": "user", "content": "Reply OK."}], + "max_tokens": 8, + "temperature": 0, + }, + virtual, + ) + if status != 200 or payload.get("model") != model: + raise RuntimeError(f"cross-model rollback probe failed for {model}") +finally: + if virtual is not None: + post("http://127.0.0.1:4446/key/delete", {"keys": [virtual]}, master) +print("cross-model rollback probes passed") +PY +event "$phase-rollback" GO "hash-verified config restored and ingress checks passed" +printf 'LiteLLM %s rollback complete\n' "$phase" diff --git a/webster/deepseek-v41/scripts/run-weka.sh b/webster/deepseek-v41/scripts/run-weka.sh new file mode 100755 index 0000000..1c6ff4b --- /dev/null +++ b/webster/deepseek-v41/scripts/run-weka.sh @@ -0,0 +1,936 @@ +#!/usr/bin/env bash +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$script_dir/common.sh" + +AIPERF_COMMIT="ea28b2e81c7367f8403c6f6ebe0837a508795a8d" +AIPERF_WEKA_PATCH_SHA256="9557a46afac031b64f66ea64ebac4e8ed4d1a0e142354388c19c086a9377a36e" +AIPERF_DATASET_CONFIGURATION_TIMEOUT_SECONDS="1200" +AIPERF_PROFILE_CONFIGURE_TIMEOUT_SECONDS="1200" +MIN_CONTROLLER_MEMORY_BYTES="34359738368" +WEKA_REPOSITORY="semianalysisai/cc-traces-weka-062126" +WEKA_REVISION="23f152f6f0f9399a85901b89a6458def0ef16729" +WEKA_TRACE_COUNT="393" +WEKA_SOURCE_SHA256="29b6a19e751ff5230771519aab755f80a0f43a4ba9cf96b72d3a6a437ec99276" +WEKA_SOURCE_SIZE="1847151435" + +if [[ -n "${WEBSTER_WEKA_RUNS_ROOT:-}" ]]; then + [[ -d "$WEBSTER_WEKA_RUNS_ROOT" && ! -L "$WEBSTER_WEKA_RUNS_ROOT" ]] || + die "Weka runs root override must be an existing non-symlink directory" + RUNS_ROOT="$(canonical_existing_path "$WEBSTER_WEKA_RUNS_ROOT")" +fi +export WEBSTER_RUNS_ROOT="$RUNS_ROOT" + +run_root="" +profile_name="" +concurrency="" +repetition="" +key_file="" +schedule="" +internal=0 +while (($#)); do + case "$1" in + --run-root) run_root="${2:?}"; shift 2 ;; + --profile-name) profile_name="${2:?}"; shift 2 ;; + --concurrency) concurrency="${2:?}"; shift 2 ;; + --repetition) repetition="${2:?}"; shift 2 ;; + --key-file) key_file="${2:?}"; shift 2 ;; + --no-fixed-schedule) schedule="open"; shift ;; + --fixed-schedule) schedule="fixed"; shift ;; + --internal) internal=1; shift ;; + *) die "unknown argument: $1" ;; + esac +done + +init_run_root "$(validate_run_root "$run_root")" +[[ "$profile_name" =~ ^[a-z0-9][a-z0-9._-]*$ ]] || die "unsafe profile name" +[[ "$repetition" =~ ^[1-9][0-9]*$ ]] || die "repetition must be positive" +[[ "$repetition" == 1 || "$repetition" == 2 ]] || + die "repetition must be 1 (warmup) or 2 (scored)" +[[ "$schedule" == open || "$schedule" == fixed ]] || + die "exactly one schedule mode is required" +if [[ "$schedule" == open ]]; then + [[ "$concurrency" =~ ^[1-9][0-9]*$ ]] || die "concurrency must be positive" +elif [[ -n "$concurrency" ]]; then + [[ "$concurrency" =~ ^[1-9][0-9]*$ ]] || die "concurrency must be positive" +fi +[[ -f "$key_file" && ! -L "$key_file" ]] || die "benchmark key must be a regular file" +[[ "$(file_mode "$key_file")" == 600 ]] || die "benchmark key mode must be 0600" +[[ -s "$key_file" ]] || die "benchmark key is empty" + +manifest="$RUN_ROOT/manifest.env" +[[ -f "$manifest" && ! -L "$manifest" ]] || die "manifest.env is missing" +[[ "$(file_mode "$manifest")" == 600 ]] || die "manifest.env mode must be 0600" +manifest_value() { + local key="$1" value + value="$(awk -F= -v key="$key" '$1 == key {sub(/^[^=]*=/, ""); print; exit}' "$manifest")" + [[ -n "$value" ]] || die "manifest field $key is empty" + printf '%s\n' "$value" +} +require_eq "manifest AIPERF_COMMIT" "$AIPERF_COMMIT" \ + "$(manifest_value AIPERF_COMMIT)" +require_eq "manifest WEKA_REPOSITORY" "$WEKA_REPOSITORY" \ + "$(manifest_value WEKA_REPOSITORY)" +require_eq "manifest WEKA_REVISION" "$WEKA_REVISION" \ + "$(manifest_value WEKA_REVISION)" + +if [[ "$schedule" == open ]]; then + run_name="$profile_name-c$concurrency-r$repetition-open" +else + run_name="$profile_name-r$repetition-fixed" +fi +destination="$RUN_ROOT/aiperf/$run_name" + +validate_weka_artifacts() { + local artifacts="$1" + local preserve_summary="${2:-0}" + local results_ready="$artifacts/.aiperf_results_ready.json" + local readiness_code=0 readiness_message="" + local summary_code=0 summary_message="" + local summary_output="$artifacts/summary.json" + local rederived_summary="" + + if [[ ! -f "$results_ready" || -L "$results_ready" ]]; then + printf 'AIPerf results readiness marker is missing or not a regular file\n' >&2 + return 1 + fi + readiness_message="$(python3 - "$results_ready" <<'PY' +import json +import sys + +try: + with open(sys.argv[1], encoding="utf-8") as handle: + result = json.load(handle) +except (OSError, json.JSONDecodeError) as error: + print(f"AIPerf results readiness marker is malformed: {error}", file=sys.stderr) + raise SystemExit(1) +if not isinstance(result, dict): + print("AIPerf results readiness marker must be an object", file=sys.stderr) + raise SystemExit(1) +if result.get("was_cancelled") is True: + print("AIPerf results readiness marker reports cancellation", file=sys.stderr) + raise SystemExit(130) +if result.get("ready") is not True: + print("AIPerf results readiness marker does not report ready=true", file=sys.stderr) + raise SystemExit(1) +if result.get("was_cancelled") is not False: + print("AIPerf results readiness marker lacks was_cancelled=false", file=sys.stderr) + raise SystemExit(1) +if result.get("partial") is not False: + print("AIPerf results readiness marker does not report partial=false", file=sys.stderr) + raise SystemExit(1) +failed_exporters = result.get("failed_exporters", []) +if not isinstance(failed_exporters, list) or any( + not isinstance(value, str) or not value for value in failed_exporters +): + print("AIPerf results readiness failed_exporters must be a string list", file=sys.stderr) + raise SystemExit(1) +if failed_exporters: + print( + "AIPerf results readiness marker reports failed exporters: " + + ", ".join(failed_exporters), + file=sys.stderr, + ) + raise SystemExit(1) +PY +)" || readiness_code=$? + if (( readiness_code != 0 )); then + [[ -z "$readiness_message" ]] || printf '%s\n' "$readiness_message" >&2 + return "$readiness_code" + fi + + if [[ "$preserve_summary" == preserve ]]; then + rederived_summary="$(mktemp "$artifacts/.summary-rederived.XXXXXXXX")" + chmod 0600 "$rederived_summary" + summary_output="$rederived_summary" + fi + summary_message="$(python3 "$script_dir/summarize-aiperf.py" \ + --profile "$artifacts/profile_export_aiperf.json" \ + --records "$artifacts/profile_export.jsonl" \ + --server-metrics "$artifacts/server_metrics_export.json" \ + --timing-log "$artifacts/aiperf.stdout.log" \ + --output-json "$summary_output" 2>&1)" || summary_code=$? + if (( summary_code != 0 )); then + if [[ -f "$summary_output" ]] && python3 - "$summary_output" <<'PY' +import json +import sys + +try: + with open(sys.argv[1], encoding="utf-8") as handle: + summary = json.load(handle) +except (OSError, json.JSONDecodeError): + raise SystemExit(1) +raise SystemExit(0 if summary.get("qualified") is False else 1) +PY + then + [[ -z "$rederived_summary" ]] || rm -f -- "$rederived_summary" + printf 'AIPerf request results are not qualified\n' >&2 + return 1 + fi + [[ -z "$rederived_summary" ]] || rm -f -- "$rederived_summary" + printf 'AIPerf result summary failed: %s\n' "$summary_message" >&2 + return 1 + fi + + if ! python3 - "$summary_output" <<'PY' +import json +import sys + +with open(sys.argv[1], encoding="utf-8") as handle: + summary = json.load(handle) +if summary.get("qualified") is not True: + print("AIPerf request results are not qualified", file=sys.stderr) + raise SystemExit(1) +required = ( + "prefix_cache_hits", + "prefix_cache_queries", + "prefix_cache_hit_ratio", + "kv_cache_usage_max", + "preemptions_total", + "running_requests_max", + "waiting_requests_max", + "wall_power_avg_watts", + "facility_power_avg_watts", + "facility_energy_kwh", + "facility_cost_dollars", + "completed_root_workflows", + "cancelled_root_workflows", + "root_workflow_elapsed_seconds", + "completed_root_workflows_per_second", +) +for name in required: + if summary.get(name) is None: + print(f"required AIPerf summary metric is missing: {name}", file=sys.stderr) + raise SystemExit(1) +if summary["completed_root_workflows"] <= 0: + print("AIPerf completed no root workflows", file=sys.stderr) + raise SystemExit(1) +if summary["cancelled_root_workflows"] != 0: + print("AIPerf cancelled one or more root workflows", file=sys.stderr) + raise SystemExit(1) +PY + then + [[ -z "$rederived_summary" ]] || rm -f -- "$rederived_summary" + return 1 + fi + if [[ -n "$rederived_summary" ]] && + ! cmp -s -- "$rederived_summary" "$artifacts/summary.json"; then + rm -f -- "$rederived_summary" + printf 'AIPerf rederived summary differs from recorded evidence\n' >&2 + return 1 + fi + [[ -z "$rederived_summary" ]] || rm -f -- "$rederived_summary" +} + +verify_controller_capacity() { + local physical_memory_bytes + if [[ "${WEBSTER_WEKA_TEST_MODE:-0}" == 1 ]]; then + [[ -n "${TEST_WEKA_CONTROLLER_MEMORY_BYTES:-}" ]] || return 0 + physical_memory_bytes="$TEST_WEKA_CONTROLLER_MEMORY_BYTES" + else + physical_memory_bytes="$({ + awk '/^MemTotal:/ {printf "%.0f\n", $2 * 1024; exit}' /proc/meminfo + })" + fi + [[ "$physical_memory_bytes" =~ ^[1-9][0-9]*$ ]] || + die "cannot determine Weka controller physical memory" + (( physical_memory_bytes >= MIN_CONTROLLER_MEMORY_BYTES )) || + die "Weka controller physical memory is too small: need at least \ +$MIN_CONTROLLER_MEMORY_BYTES bytes, found $physical_memory_bytes" +} + +aiperf_bin="${WEBSTER_WEKA_AIPERF_BIN:-${TEST_AIPERF_BIN:-$RUN_ROOT/aiperf-venv/bin/aiperf}}" +aiperf_source="${WEBSTER_WEKA_AIPERF_SOURCE:-$RUN_ROOT/aiperf-source}" +dataset="${WEBSTER_WEKA_DATASET:-${TEST_WEKA_DATASET:-$RUN_ROOT/weka-dataset/traces}}" +tokenizer="${WEBSTER_WEKA_TOKENIZER:-${TEST_DEEPSEEK_TOKENIZER:-$RUN_ROOT/deepseek-tokenizer}}" + +verify_client_and_corpus() { + local actual_aiperf actual_patch actual_revision trace_count full_subagents + local expected_trace_count="$WEKA_TRACE_COUNT" + if [[ "${WEBSTER_WEKA_TEST_MODE:-0}" == 1 ]]; then + actual_aiperf="${TEST_WEKA_AIPERF_COMMIT:-$AIPERF_COMMIT}" + actual_patch="${TEST_WEKA_AIPERF_PATCH_SHA256:-$AIPERF_WEKA_PATCH_SHA256}" + actual_revision="${TEST_WEKA_REVISION:-$WEKA_REVISION}" + trace_count="${TEST_WEKA_TRACE_COUNT:-$WEKA_TRACE_COUNT}" + full_subagents="${TEST_WEKA_FULL_SUBAGENTS:-1}" + fi + if [[ "${WEBSTER_WEKA_TEST_MODE:-0}" != 1 ]]; then + [[ -d "$aiperf_source/.git" && ! -L "$aiperf_source" ]] || + die "pinned AIPerf source checkout is missing" + actual_aiperf="$(git -C "$aiperf_source" rev-parse HEAD)" + [[ "$(git -C "$aiperf_source" status --porcelain --untracked-files=all -- src)" == \ + " M src/aiperf/config/dataset/resolver.py" ]] || + die "AIPerf runtime source differs from the pinned Weka timing patch" + actual_patch="$( + git -C "$aiperf_source" diff --binary --no-ext-diff HEAD -- src | + sha256sum | awk '{print $1}' + )" + fi + if [[ "${WEBSTER_WEKA_TEST_MODE:-0}" != 1 || "${TEST_WEKA_VERIFY_CORPUS:-0}" == 1 ]]; then + local corpus_manifest source_sha256 source_size + corpus_manifest="${WEBSTER_WEKA_CORPUS_MANIFEST:-$(dirname -- "$dataset")/corpus-manifest.json}" + [[ -f "$corpus_manifest" && ! -L "$corpus_manifest" ]] || + die "pinned Weka corpus manifest is missing" + source_sha256="$WEKA_SOURCE_SHA256" + source_size="$WEKA_SOURCE_SIZE" + if [[ "${WEBSTER_WEKA_TEST_MODE:-0}" == 1 ]]; then + source_sha256="${TEST_WEKA_SOURCE_SHA256:?}" + source_size="${TEST_WEKA_SOURCE_SIZE:?}" + expected_trace_count="${TEST_WEKA_SOURCE_TRACE_COUNT:?}" + fi + python3 "$script_dir/verify-weka-corpus.py" \ + --dataset "$dataset" \ + --manifest "$corpus_manifest" \ + --repository "$WEKA_REPOSITORY" \ + --revision "$WEKA_REVISION" \ + --source-sha256 "$source_sha256" \ + --source-size "$source_size" \ + --trace-count "$expected_trace_count" >/dev/null || + die "Weka corpus identity validation failed" + actual_revision="$WEKA_REVISION" + trace_count="$expected_trace_count" + full_subagents=1 + fi + require_eq "checked-out AIPERF_COMMIT" "$AIPERF_COMMIT" "$actual_aiperf" + require_eq "AIPerf Weka timing patch" "$AIPERF_WEKA_PATCH_SHA256" \ + "$actual_patch" + require_eq "resolved WEKA_REVISION" "$WEKA_REVISION" "$actual_revision" + require_eq "exact $expected_trace_count traces" "$expected_trace_count" "$trace_count" + require_eq "full subagents" 1 "$full_subagents" +} +verify_client_and_corpus + +validate_weka_provenance() { + local directory="$1" + python3 - "$directory/provenance.json" "$profile_name" \ + "${concurrency:-0}" "$repetition" "$AIPERF_COMMIT" \ + "$AIPERF_WEKA_PATCH_SHA256" "$WEKA_REPOSITORY" "$WEKA_REVISION" \ + "$WEKA_TRACE_COUNT" <<'PY' +import hashlib +import json +import re +import sys +from pathlib import Path + +( + provenance_raw, + profile_name, + concurrency, + repetition, + aiperf_commit, + aiperf_patch, + weka_repository, + weka_revision, + trace_count, +) = sys.argv[1:] +provenance_path = Path(provenance_raw) +if not provenance_path.is_file() or provenance_path.is_symlink(): + raise SystemExit("completed Weka run provenance is missing or unsafe") +try: + provenance = json.loads(provenance_path.read_text(encoding="utf-8")) +except (OSError, json.JSONDecodeError) as error: + raise SystemExit(f"completed Weka run provenance is malformed: {error}") +if not isinstance(provenance, dict): + raise SystemExit("completed Weka run provenance must be an object") + +expected_repetition = int(repetition) +expected_cache_state = "warmup" if expected_repetition == 1 else "warm" +expected = ( + ("schema_version", 1, "recorded provenance schema"), + ("profile_name", profile_name, "recorded profile name"), + ("concurrency", int(concurrency), "recorded concurrency"), + ("repetition", expected_repetition, "recorded repetition"), + ("cache_state", expected_cache_state, "recorded cache state"), + ("aiperf_commit", aiperf_commit, "recorded AIPerf commit"), + ("aiperf_weka_patch_sha256", aiperf_patch, "recorded AIPerf patch"), + ("weka_repository", weka_repository, "recorded Weka repository"), + ("weka_revision", weka_revision, "recorded Weka revision"), + ("trace_count", int(trace_count), "recorded Weka trace count"), + ("full_subagents", True, "recorded full-subagent mode"), +) +for field, wanted, label in expected: + recorded = provenance.get(field) + if type(recorded) is not type(wanted) or recorded != wanted: + raise SystemExit(f"{label} differs from the current qualification contract") + +if expected_repetition == 1: + if ( + provenance.get("warmup_and_scored_comparable") is not False + or provenance.get("warmup_provenance_sha256") is not None + ): + raise SystemExit("recorded warmup cache provenance is malformed") +else: + warmup_hash = provenance.get("warmup_provenance_sha256") + if ( + provenance.get("warmup_and_scored_comparable") is not True + or not isinstance(warmup_hash, str) + or re.fullmatch(r"[0-9a-f]{64}", warmup_hash) is None + ): + raise SystemExit("recorded scored cache provenance is malformed") + +generation = provenance.get("serving_generation") +if not isinstance(generation, str) or not generation: + raise SystemExit("recorded serving generation is missing") + +required_artifacts = ( + "EXIT", + ".aiperf_results_ready.json", + "profile_export.jsonl", + "profile_export_aiperf.json", + "server_metrics_export.json", + "aiperf.stdout.log", + "aiperf.stderr.log", + "rendered-command.txt", + "summary.json", + "benchmark-window.json", + "rank0-before.json", + "rank1-before.json", + "rank0-after.json", + "rank1-after.json", + "rank0-container.log", + "rank1-container.log", +) +artifacts = provenance.get("artifacts") +if not isinstance(artifacts, dict) or set(artifacts) != set(required_artifacts): + raise SystemExit("recorded required artifact hash map is incomplete") +directory = provenance_path.parent +for name in required_artifacts: + expected_hash = artifacts.get(name) + if not isinstance(expected_hash, str) or re.fullmatch(r"[0-9a-f]{64}", expected_hash) is None: + raise SystemExit(f"recorded artifact hash is malformed: {name}") + path = directory / name + if not path.is_file() or path.is_symlink(): + raise SystemExit(f"recorded artifact is missing or unsafe: {name}") + actual_hash = hashlib.sha256(path.read_bytes()).hexdigest() + if actual_hash != expected_hash: + raise SystemExit(f"recorded artifact hash differs: {name}") +PY +} + +if [[ -f "$destination/EXIT" ]]; then + [[ "$(<"$destination/EXIT")" == "EXIT=0" ]] || + die "existing Weka run is not successful: $destination" + validate_weka_provenance "$destination" || + die "existing Weka run failed provenance validation: $destination" + validate_weka_artifacts "$destination" preserve || + die "existing Weka run failed artifact validation: $destination" + printf 'GO Weka run already complete: %s\n' "$destination" + exit 0 +fi + +verify_controller_capacity + +if [[ "$internal" == 0 && -z "${TMUX:-}" && + ( "${WEBSTER_WEKA_TEST_MODE:-0}" != 1 || "${TEST_WEKA_ENABLE_TMUX:-0}" == 1 ) ]]; then + session="dsv41-${run_name//./-}" + command=(env "WEBSTER_WEKA_RUNS_ROOT=$RUNS_ROOT" + "WEBSTER_WEKA_AIPERF_BIN=$aiperf_bin" + "WEBSTER_WEKA_AIPERF_SOURCE=$aiperf_source" + "WEBSTER_WEKA_DATASET=$dataset" + "WEBSTER_WEKA_TOKENIZER=$tokenizer" + "WEBSTER_LOCAL_SHAMU=${WEBSTER_LOCAL_SHAMU:-0}" + bash "$0" --run-root "$RUN_ROOT" --profile-name "$profile_name" + --repetition "$repetition" --key-file "$key_file" --internal) + if [[ "$schedule" == open ]]; then + command+=(--concurrency "$concurrency" --no-fixed-schedule) + else + command+=(--fixed-schedule) + fi + log="$RUN_ROOT/logs/$run_name.tmux.log" + : >"$log" + chmod 0600 "$log" + shell_command="$(printf '%q ' "${command[@]}")" + tmux new-session -d -s "$session" "$shell_command >$(printf '%q' "$log") 2>&1" + while tmux has-session -t "$session" 2>/dev/null; do sleep 15; done + [[ -f "$destination/EXIT" ]] || die "tmux Weka run ended without EXIT record" + [[ "$(<"$destination/EXIT")" == "EXIT=0" ]] || die "tmux Weka run failed" + printf 'GO Weka run complete: %s\n' "$destination" + exit 0 +fi + +collect_rank_state() { + local node="$1" docker_mode="$2" expected_image="$3" + ssh "$node" python3 - "$docker_mode" "$DEEPSEEK_CONTAINER" \ + "$expected_image" <<'PY' +import json +import subprocess +import sys + +docker_mode, container, expected_image = sys.argv[1:] +docker = ["docker"] if docker_mode == "direct" else ["sudo", "-n", "docker"] +try: + value = json.loads( + subprocess.check_output(docker + ["inspect", container], text=True) + )[0] +except (subprocess.CalledProcessError, json.JSONDecodeError, IndexError) as error: + raise SystemExit(f"cannot inspect serving rank: {error}") +state = value.get("State") or {} +if state.get("Running") is not True: + raise SystemExit("serving rank is not running") +if value.get("RestartCount") != 0: + raise SystemExit("serving rank has a nonzero restart count") +if value.get("Image") != expected_image: + raise SystemExit("serving rank uses the wrong image") +labels = (value.get("Config") or {}).get("Labels") or {} +print( + json.dumps( + { + "container_id": value.get("Id"), + "generation": labels.get("ai.webster.generation"), + "image": value.get("Image"), + "restart_count": value.get("RestartCount"), + "started_at": state.get("StartedAt"), + }, + separators=(",", ":"), + ) +) +PY +} + +resolve_generation_from_states() { + local rank0_state="$1" rank1_state="$2" + python3 "$script_dir/verify-serving-pair.py" \ + --rank0-state-json "$rank0_state" \ + --rank1-state-json "$rank1_state" \ + --max-start-skew-seconds 120 \ + --require-matching-image \ + "--print-generation" +} + +pre_run_generation="" +pre_rank0_state="" +pre_rank1_state="" +if [[ "${WEBSTER_WEKA_TEST_MODE:-0}" != 1 ]]; then + live_profile="$RUN_ROOT/profiles/$profile_name.env" + [[ -f "$live_profile" && ! -L "$live_profile" ]] || + die "selected Weka profile is missing" + "$script_dir/start-deepseek-v41-tp2.sh" --run-root "$RUN_ROOT" \ + --profile-file "$live_profile" --verify-only >/dev/null + expected_image="$(manifest_value VLLM_IMAGE_ID)" + pre_rank0_state="$(collect_rank_state shamu direct "$expected_image")" + pre_rank1_state="$(collect_rank_state tilikum sudo "$expected_image")" + pre_run_generation="$(resolve_generation_from_states \ + "$pre_rank0_state" "$pre_rank1_state")" +else + pre_run_generation="${TEST_WEKA_SERVING_GENERATION:-test-serving-generation}" + pre_rank0_state="{\"container_id\":\"$(printf '0%.0s' {1..64})\",\"generation\":\"$pre_run_generation\",\"image\":\"sha256:$(printf 'a%.0s' {1..64})\",\"restart_count\":0,\"started_at\":\"2099-01-01T00:00:00Z\"}" + pre_rank1_state="{\"container_id\":\"$(printf '1%.0s' {1..64})\",\"generation\":\"$pre_run_generation\",\"image\":\"sha256:$(printf 'a%.0s' {1..64})\",\"restart_count\":0,\"started_at\":\"2099-01-01T00:00:01Z\"}" +fi + +temp_parent="${TEST_WEKA_TMP_ROOT:-$RUN_ROOT/tmp}" +[[ -x "$aiperf_bin" ]] || die "pinned AIPerf executable is missing" +[[ ( -f "$dataset" || -d "$dataset" ) && ! -L "$dataset" ]] || + die "pinned Weka dataset is missing" +if [[ "${WEBSTER_WEKA_TEST_MODE:-0}" != 1 ]]; then + [[ -d "$tokenizer" && ! -L "$tokenizer" ]] || die "pinned tokenizer is missing" +fi +mkdir -p "$temp_parent" +chmod 0700 "$temp_parent" +temporary="$(mktemp -d "$temp_parent/weka-$run_name.XXXXXXXX")" + +cleanup() { + local status=$? + trap - EXIT + unset OPENAI_API_KEY || true + if [[ -n "${temporary:-}" && -d "$temporary" ]]; then + resolved="$(canonical_existing_path "$temporary")" + parent="$(canonical_existing_path "$temp_parent")" + case "$resolved" in "$parent"/weka-*) ;; *) exit 1 ;; esac + [[ "$(file_uid "$resolved")" == "$(id -u)" ]] || exit 1 + rm -rf -- "$resolved" + fi + exit "$status" +} +trap cleanup EXIT + +artifacts="$temporary/artifacts" +mkdir -m 0700 "$artifacts" +printf '%s\n' "$pre_rank0_state" >"$artifacts/rank0-before.json" +printf '%s\n' "$pre_rank1_state" >"$artifacts/rank1-before.json" +chmod 0600 "$artifacts/rank0-before.json" "$artifacts/rank1-before.json" +cache_state="warmup" +warmup_and_scored_comparable=false +warmup_provenance_sha256="" +if [[ "$repetition" == 2 ]]; then + if [[ "$schedule" == open ]]; then + warmup_name="$profile_name-c$concurrency-r1-open" + else + warmup_name="$profile_name-r1-fixed" + fi + warmup_dir="$RUN_ROOT/aiperf/$warmup_name" + warmup_provenance="$warmup_dir/provenance.json" + [[ -f "$warmup_provenance" && ! -L "$warmup_provenance" ]] || + die "successful warmup repetition provenance is missing" + [[ -f "$warmup_dir/EXIT" && "$(<"$warmup_dir/EXIT")" == "EXIT=0" ]] || + die "warmup repetition is not successful" + warmup_provenance_sha256="$(python3 - "$warmup_provenance" "$profile_name" \ + "${concurrency:-0}" "$pre_run_generation" "$AIPERF_COMMIT" \ + "$AIPERF_WEKA_PATCH_SHA256" "$WEKA_REPOSITORY" "$WEKA_REVISION" \ + "$artifacts/rank0-before.json" "$artifacts/rank1-before.json" <<'PY' +import hashlib +import json +import sys +from pathlib import Path + +provenance_path = Path(sys.argv[1]) +profile_name, concurrency, generation = sys.argv[2:5] +aiperf_commit, aiperf_patch, weka_repository, weka_revision = sys.argv[5:9] +current_states = [Path(path) for path in sys.argv[9:11]] +provenance = json.loads(provenance_path.read_text(encoding="utf-8")) +if ( + provenance.get("schema_version") != 1 + or provenance.get("profile_name") != profile_name + or provenance.get("concurrency") != int(concurrency) + or provenance.get("repetition") != 1 + or provenance.get("cache_state") != "warmup" + or provenance.get("serving_generation") != generation +): + raise SystemExit("warmup repetition provenance is not comparable") +if ( + provenance.get("aiperf_commit") != aiperf_commit + or provenance.get("weka_repository") != weka_repository + or provenance.get("weka_revision") != weka_revision +): + raise SystemExit("warmup repetition client or dataset pin is not comparable") +if provenance.get("aiperf_weka_patch_sha256") != aiperf_patch: + raise SystemExit("warmup repetition runtime patch is not comparable") +artifacts = provenance.get("artifacts") +if not isinstance(artifacts, dict): + raise SystemExit("warmup repetition artifact manifest is missing") +for rank, current in enumerate(current_states): + previous = provenance_path.parent / f"rank{rank}-after.json" + if not previous.is_file() or previous.is_symlink(): + raise SystemExit("warmup repetition rank state is missing") + digest = hashlib.sha256(previous.read_bytes()).hexdigest() + if artifacts.get(previous.name) != digest: + raise SystemExit("warmup repetition rank state hash differs") + if json.loads(previous.read_text(encoding="utf-8")) != json.loads( + current.read_text(encoding="utf-8") + ): + raise SystemExit("warmup repetition used a different serving rank") +print(hashlib.sha256(provenance_path.read_bytes()).hexdigest()) +PY +)" || die "warmup repetition is not generation-comparable" + cache_state="warm" + warmup_and_scored_comparable=true +fi +config="$artifacts/aiperf-config.yaml" +python3 - "$config" "$dataset" "$tokenizer" "$artifacts" "$schedule" \ + "$concurrency" "$SHAMU_NETBIRD" "$TILIKUM_NETBIRD" "$CANARY_PORT" \ + "$DEEPSEEK_SERVED_MODEL" <<'PY' +import json +import os +import sys +from pathlib import Path + +( + config_path, + dataset, + tokenizer, + artifacts, + schedule, + concurrency, + shamu, + tilikum, + port, + model, +) = sys.argv[1:] + +phase = { + "name": "profiling", + "kind": "profiling", + "duration": 900.0, + "grace_period": 300.0, +} +dataset_config = { + "name": "main", + "type": "file", + "path": dataset, + "format": "weka_trace", + "random_seed": 20260911, +} +if schedule == "fixed": + phase.update({"type": "fixed_schedule", "auto_offset": True}) +else: + phase.update({"type": "concurrency", "concurrency": int(concurrency)}) + dataset_config["ignore_trace_delays"] = True + +config = { + "schemaVersion": "2.0", + "benchmark": { + "models": {"items": [{"name": model}]}, + "endpoint": { + "urls": [f"http://{shamu}:{port}"], + "type": "chat", + "api_key": "${OPENAI_API_KEY}", + "timeout": 900.0, + "streaming": True, + "use_server_token_count": True, + }, + "datasets": [dataset_config], + "phases": [phase], + "artifacts": { + "dir": artifacts, + "records": ["jsonl"], + "export_outputs_json": True, + }, + "tokenizer": {"name": tokenizer}, + "gpu_telemetry": { + "enabled": True, + "urls": [ + f"http://{shamu}:9400/metrics", + f"http://{tilikum}:9400/metrics", + ], + "collector": "dcgm", + "mode": "summary", + }, + "server_metrics": { + "enabled": True, + "urls": [ + f"http://{shamu}:{port}/metrics", + f"http://{shamu}:9877/metrics", + f"http://{tilikum}:9877/metrics", + ], + "formats": ["json", "csv"], + }, + }, + "random_seed": 20260911, +} + +path = Path(config_path) +path.write_text(json.dumps(config, indent=2) + "\n", encoding="utf-8") +os.chmod(path, 0o600) +PY + +command=(env + "AIPERF_DATASET_CONFIGURATION_TIMEOUT=$AIPERF_DATASET_CONFIGURATION_TIMEOUT_SECONDS" + "AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT=$AIPERF_PROFILE_CONFIGURE_TIMEOUT_SECONDS" + "$aiperf_bin" profile --config "$config") + +printf '%q ' "${command[@]}" >"$artifacts/rendered-command.txt" +printf '\n' >>"$artifacts/rendered-command.txt" +chmod 0600 "$artifacts/rendered-command.txt" +OPENAI_API_KEY="$(<"$key_file")" +OPENAI_API_KEY="${OPENAI_API_KEY%$'\n'}" +export OPENAI_API_KEY +benchmark_started_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +validation_message="" +set +e +"${command[@]}" >"$artifacts/aiperf.stdout.log" 2>"$artifacts/aiperf.stderr.log" +status=$? +set -e +unset OPENAI_API_KEY +benchmark_ended_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" + +capture_rank_log() { + local node="$1" docker_mode="$2" output="$3" result=0 + if [[ "${WEBSTER_WEKA_TEST_MODE:-0}" == 1 ]]; then + if [[ "$node" == shamu ]]; then + printf '%s\n' "${TEST_WEKA_RANK0_LOG:-healthy rank 0}" >"$output" + else + printf '%s\n' "${TEST_WEKA_RANK1_LOG:-healthy rank 1}" >"$output" + fi + elif [[ "$docker_mode" == direct ]]; then + ssh "$node" docker logs --since "$benchmark_started_at" --timestamps \ + "$DEEPSEEK_CONTAINER" >"$output" 2>&1 || result=$? + else + ssh "$node" sudo -n docker logs --since "$benchmark_started_at" --timestamps \ + "$DEEPSEEK_CONTAINER" >"$output" 2>&1 || result=$? + fi + chmod 0600 "$output" + return "$result" +} + +rank_capture_failed=0 +capture_rank_log shamu direct "$artifacts/rank0-container.log" || rank_capture_failed=1 +capture_rank_log tilikum sudo "$artifacts/rank1-container.log" || rank_capture_failed=1 +if (( rank_capture_failed != 0 )); then + status=1 + validation_message="failed to capture one or more serving-rank logs" +fi + +post_rank0_state="$pre_rank0_state" +post_rank1_state="$pre_rank1_state" +if [[ "${WEBSTER_WEKA_TEST_MODE:-0}" != 1 ]]; then + post_rank0_state="$(collect_rank_state shamu direct "$expected_image")" || status=1 + post_rank1_state="$(collect_rank_state tilikum sudo "$expected_image")" || status=1 +fi +printf '%s\n' "$post_rank0_state" >"$artifacts/rank0-after.json" +printf '%s\n' "$post_rank1_state" >"$artifacts/rank1-after.json" +python3 - "$artifacts/benchmark-window.json" "$benchmark_started_at" \ + "$benchmark_ended_at" <<'PY' +import json +import os +import sys +from pathlib import Path + +path = Path(sys.argv[1]) +path.write_text( + json.dumps({"started_at": sys.argv[2], "ended_at": sys.argv[3]}, sort_keys=True) + + "\n", + encoding="utf-8", +) +os.chmod(path, 0o600) +PY +chmod 0600 "$artifacts/rank0-after.json" "$artifacts/rank1-after.json" + +rank_fatal_matches="$(python3 - \ + "$artifacts/rank0-container.log" "$artifacts/rank1-container.log" <<'PY' +import re +import sys +from pathlib import Path + +pattern = re.compile( + r"EngineDeadError|NCCL.*(?:timeout|error)|RPC call.*timed out", + re.IGNORECASE, +) +print( + sum( + len(pattern.findall(Path(path).read_text(encoding="utf-8", errors="replace"))) + for path in sys.argv[1:] + ) +) +PY +)" +if (( rank_fatal_matches != 0 )); then + status=1 + validation_message="fatal serving-rank log markers detected: $rank_fatal_matches" +fi +cancelled=0 +if (( status == 0 )); then + validation_code=0 + validation_message="$(validate_weka_artifacts "$artifacts" 2>&1)" || + validation_code=$? + if (( validation_code != 0 )); then + status="$validation_code" + (( validation_code == 130 )) && cancelled=1 + fi +fi +printf 'EXIT=%s\n' "$status" >"$artifacts/EXIT" +chmod -R go-rwx "$artifacts" + +serving_generation="$pre_run_generation" +if (( status == 0 )); then + serving_generation="$(resolve_generation_from_states \ + "$post_rank0_state" "$post_rank1_state")" + require_eq "serving generation remained stable" "$pre_run_generation" \ + "$serving_generation" +fi +if (( status == 0 )); then + python3 - "$artifacts" "$profile_name" "${concurrency:-0}" "$repetition" \ + "$AIPERF_COMMIT" "$WEKA_REPOSITORY" "$WEKA_REVISION" \ + "$WEKA_TRACE_COUNT" "$serving_generation" "$cache_state" \ + "$AIPERF_WEKA_PATCH_SHA256" \ + "$warmup_and_scored_comparable" "$warmup_provenance_sha256" <<'PY' +import hashlib +import json +import os +import re +import sys +from pathlib import Path + +( + artifacts_raw, + profile_name, + concurrency, + repetition, + aiperf_commit, + repository, + revision, + trace_count, + serving_generation, + cache_state, + aiperf_weka_patch_sha256, + warmup_and_scored_comparable, + warmup_provenance_sha256, +) = sys.argv[1:] +artifacts = Path(artifacts_raw) +names = ( + "EXIT", + ".aiperf_results_ready.json", + "profile_export.jsonl", + "profile_export_aiperf.json", + "server_metrics_export.json", + "aiperf.stdout.log", + "aiperf.stderr.log", + "rendered-command.txt", + "summary.json", + "benchmark-window.json", + "rank0-before.json", + "rank1-before.json", + "rank0-after.json", + "rank1-after.json", + "rank0-container.log", + "rank1-container.log", +) +missing = [name for name in names if not (artifacts / name).is_file()] +if missing: + raise SystemExit("missing raw AIPerf artifacts: " + ", ".join(missing)) +combined_logs = "\n".join( + (artifacts / name).read_text(encoding="utf-8", errors="replace") + for name in ( + "aiperf.stdout.log", + "aiperf.stderr.log", + "rank0-container.log", + "rank1-container.log", + ) + if (artifacts / name).is_file() +) +records_text = (artifacts / "profile_export.jsonl").read_text( + encoding="utf-8", errors="replace" +) +value = { + "schema_version": 1, + "profile_name": profile_name, + "concurrency": int(concurrency), + "repetition": int(repetition), + "cache_state": cache_state, + "warmup_and_scored_comparable": warmup_and_scored_comparable == "true", + "warmup_provenance_sha256": warmup_provenance_sha256 or None, + "aiperf_commit": aiperf_commit, + "aiperf_weka_patch_sha256": aiperf_weka_patch_sha256, + "weka_repository": repository, + "weka_revision": revision, + "trace_count": int(trace_count), + "full_subagents": True, + "serving_generation": serving_generation, + "engine_deaths": combined_logs.count("EngineDeadError"), + "nccl_timeouts": sum( + combined_logs.lower().count(marker) + for marker in ("nccl timeout", "nccl operation timed out") + ), + "rank_fatal_error_matches": sum( + len( + re.findall( + r"EngineDeadError|NCCL.*(?:timeout|error)|RPC call.*timed out", + (artifacts / name).read_text(encoding="utf-8", errors="replace"), + re.IGNORECASE, + ) + ) + for name in ("rank0-container.log", "rank1-container.log") + ), + "corruption_flags": records_text.lower().count('"corruption":true'), + "artifacts": { + name: hashlib.sha256((artifacts / name).read_bytes()).hexdigest() + for name in names + }, +} +temporary = artifacts / f"provenance.json.tmp.{os.getpid()}" +temporary.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n", encoding="utf-8") +os.chmod(temporary, 0o600) +os.replace(temporary, artifacts / "provenance.json") +PY +fi + +partial="$destination.partial.$$" +[[ ! -e "$destination" && ! -e "$partial" ]] || die "Weka destination already exists" +mkdir -m 0700 "$partial" +cp -a "$artifacts/." "$partial/" +mv "$partial" "$destination" +event weka-run "$([[ "$status" == 0 ]] && printf GO || printf NO-GO)" \ + "profile=$profile_name concurrency=${concurrency:-trace-driven} repetition=$repetition schedule=$schedule" +[[ -z "$validation_message" ]] || printf '%s\n' "$validation_message" >&2 +(( cancelled == 0 )) || die "AIPerf run was cancelled" +(( status == 0 )) || die "AIPerf exited with status $status" +printf 'GO Weka run complete: %s\n' "$destination" diff --git a/webster/deepseek-v41/scripts/stage-artifacts.sh b/webster/deepseek-v41/scripts/stage-artifacts.sh new file mode 100755 index 0000000..d66d7cf --- /dev/null +++ b/webster/deepseek-v41/scripts/stage-artifacts.sh @@ -0,0 +1,518 @@ +#!/usr/bin/env bash +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$script_dir/common.sh" + +node="" +run_root="" +while (($#)); do + case "$1" in + --node) + [[ $# -ge 2 ]] || die "--node requires a value" + node="$2" + shift 2 + ;; + --run-root) + [[ $# -ge 2 ]] || die "--run-root requires a value" + run_root="$2" + shift 2 + ;; + *) die "unknown argument: $1" ;; + esac +done + +case "$node" in + shamu|tilikum) ;; + *) die "node must be shamu or tilikum" ;; +esac +validated_root="$(validate_run_root "$run_root")" + +STAGE_SSH_TIMEOUT_SECONDS="${STAGE_SSH_TIMEOUT_SECONDS:-20}" +[[ "$STAGE_SSH_TIMEOUT_SECONDS" =~ ^[1-9][0-9]*$ ]] || + die "STAGE_SSH_TIMEOUT_SECONDS must be a positive integer" +(( STAGE_SSH_TIMEOUT_SECONDS <= 120 )) || + die "STAGE_SSH_TIMEOUT_SECONDS must not exceed 120" + +stage_quick_ssh() { + timeout --foreground --signal=TERM --kill-after=5 \ + "$STAGE_SSH_TIMEOUT_SECONDS" \ + ssh -o BatchMode=yes -o ConnectTimeout=5 \ + -o ServerAliveInterval=5 -o ServerAliveCountMax=2 "$@" +} + +if [[ "${WEBSTER_STAGE_TEST_MODE:-0}" == "1" ]]; then + check_glm_health + require_ge "free bytes before staging" "$MIN_FREE_BEFORE_STAGE_BYTES" \ + "${TEST_FREE_BYTES:-$MIN_FREE_BEFORE_STAGE_BYTES}" + [[ "${TEST_STAGE_FINAL_STATE:-absent}" != "divergent" ]] || + die "divergent final checkpoint must not be overwritten" + if [[ -n "${TEST_STAGE_HEALTH_WINDOWS:-}" ]]; then + baseline_p90="${TEST_STAGE_BASELINE_P90:?test baseline p90 is required}" + consecutive_failures=0 + while read -r request_count success_rate p90; do + [[ -n "${request_count:-}" ]] || continue + decision="$(stage_window_decision \ + "$baseline_p90" "$request_count" "$success_rate" "$p90")" + if [[ "$decision" == BREACH\ * ]]; then + consecutive_failures=$((consecutive_failures + 1)) + else + consecutive_failures=0 + fi + (( consecutive_failures < 2 )) || + die "GLM success/latency breached for two consecutive staging windows" + done <<<"$TEST_STAGE_HEALTH_WINDOWS" + fi + stage_quick_ssh "$node" stage-artifacts-test >/dev/null || + die "bounded staging SSH probe failed" + exit 0 +fi + +[[ -d "$validated_root" ]] || die "run root does not exist; run baseline preflight first" +RUN_ROOT="$validated_root" +export RUN_ROOT +checkpoint_name="DeepSeek-V4.1-Flash-${CHECKPOINT_REVISION:0:10}" +model_parent="/home/alecfong/deepseek-v41/models" +final_path="$model_parent/$checkpoint_name" +incomplete_path="$model_parent/.$checkpoint_name.incomplete" +verifier="$script_dir/verify-checkpoint.py" +[[ -f "$verifier" ]] || die "missing checkpoint verifier" + +event "stage-$node" START "immutable checkpoint staging" +check_glm_health || + die "GLM health failed before staging on $node" + +baseline_p90="" +baseline_window="" +for candidate_window in 15m 1h 6h 24h 7d; do + baseline_candidate="$RUN_ROOT/metrics/stage-$node-baseline-$candidate_window.json" + write_glm_window_snapshot "$baseline_candidate" "$candidate_window" || + die "cannot capture GLM staging baseline from Prometheus" + read -r baseline_requests candidate_p90 < <( + python3 - "$baseline_candidate" <<'PY' +import json +import sys + +data = json.load(open(sys.argv[1], encoding="utf-8")) +requests = data.get("request_count") +p90 = data.get("p90_seconds") +print("null" if requests is None else requests, "null" if p90 is None else p90) +PY + ) + if python3 - "$baseline_requests" "$candidate_p90" \ + "$STAGE_BASELINE_MIN_REQUESTS" <<'PY' +import math +import sys + +try: + requests, p90, minimum = map(float, sys.argv[1:]) +except ValueError: + raise SystemExit(1) +raise SystemExit(0 if requests >= minimum and math.isfinite(p90) and p90 > 0 else 1) +PY + then + baseline_p90="$candidate_p90" + baseline_window="$candidate_window" + break + fi +done +[[ -n "$baseline_p90" ]] || + die "no GLM latency baseline with at least $STAGE_BASELINE_MIN_REQUESTS requests" +event "stage-$node" GO \ + "monitor baseline window=$baseline_window p90_seconds=$baseline_p90" + +layout_file="$RUN_ROOT/baseline/$node-lfs-layout.json" +ssh "$node" python3 - "$CHECKPOINT_REPO" "$CHECKPOINT_REVISION" \ + "$incomplete_path" >"$layout_file" <<'PY' +import json +import sys +import urllib.parse +import urllib.request +from pathlib import Path + +repo, revision, incomplete = sys.argv[1:] +url = ( + "https://huggingface.co/api/models/" + + urllib.parse.quote(repo, safe="/") + + "/revision/" + + urllib.parse.quote(revision, safe="") + + "?blobs=true" +) +with urllib.request.urlopen(url, timeout=60) as response: + metadata = json.load(response) +if metadata.get("sha") != revision: + raise SystemExit("resolved Hugging Face revision does not match requested revision") +root = Path(incomplete) +files = [] +missing = [] +for sibling in metadata.get("siblings", []): + name = sibling.get("rfilename") + if not isinstance(name, str) or not name: + continue + size = sibling.get("lfs", {}).get("size", sibling.get("size")) + if not isinstance(size, int) or size < 0: + raise SystemExit(f"missing size metadata for {name}") + files.append({"path": name, "bytes": size}) + candidate = root / name + if not candidate.is_file() or candidate.stat().st_size != size: + missing.append({"path": name, "bytes": size}) +print(json.dumps({ + "repository": repo, + "revision": revision, + "files": files, + "missing_files": missing, + "missing_file_bytes": sum(item["bytes"] for item in missing), + "largest_missing_file_bytes": max((item["bytes"] for item in missing), default=0), +}, sort_keys=True, indent=2)) +PY +chmod 0600 "$layout_file" + +read -r missing_bytes largest_missing < <( + python3 - "$layout_file" <<'PY' +import json +import sys +data = json.load(open(sys.argv[1], encoding="utf-8")) +print(data["missing_file_bytes"], data["largest_missing_file_bytes"]) +PY +) +required_bytes=$((missing_bytes + largest_missing + MIN_FREE_AFTER_STAGE_BYTES)) +(( required_bytes < MIN_FREE_BEFORE_STAGE_BYTES )) && + required_bytes="$MIN_FREE_BEFORE_STAGE_BYTES" +free_bytes="$(remote_available_bytes "$node" "$model_parent")" +require_ge "$node free bytes for staging" "$required_bytes" "$free_bytes" + +verify_remote() { + local path="$1" output="$2" + ssh "$node" python3 - "$path" <"$verifier" >"$output" + chmod 0600 "$output" +} + +download_log="$incomplete_path/download.log" +download_status="$incomplete_path/download.status" +download_pid="$incomplete_path/download.pid" +download_runner="$incomplete_path/download.runner.sh" +download_container="deepseek-v41-artifact-stage-${CHECKPOINT_REVISION:0:10}" +if [[ "$node" == "tilikum" ]]; then + download_docker_mode="sudo" + download_docker="sudo -n docker" +else + download_docker_mode="plain" + download_docker="docker" +fi +shamu_downloader_image="$(ssh shamu \ + 'docker inspect glm52-full-mtp --format "{{.Image}}"')" +tilikum_downloader_image="$(ssh tilikum \ + 'sudo -n docker inspect glm52-full-mtp --format "{{.Image}}"')" +[[ "$shamu_downloader_image" =~ ^sha256:[0-9a-f]{64}$ ]] || + die "invalid Shamu GLM image ID for artifact downloader" +require_eq "station GLM image ID" "$shamu_downloader_image" \ + "$tilikum_downloader_image" +download_image="$shamu_downloader_image" +event "stage-$node" GO "downloader image=$download_image runtime=runc gpu=none" +download_active=0 +stage_complete=0 +preserve_download_on_exit=0 + +remote_download_pid() { + local pid + pid="$(stage_quick_ssh "$node" "test -s '$download_pid' && cat '$download_pid'" 2>/dev/null || true)" + [[ "$pid" =~ ^[1-9][0-9]*$ ]] || return 1 + printf '%s\n' "$pid" +} + +remote_download_is_running() { + local pid + pid="$(remote_download_pid)" || return 1 + stage_quick_ssh "$node" "/bin/kill -0 -- '-$pid' 2>/dev/null" +} + +remote_download_state() { + stage_quick_ssh "$node" \ + "if test -s '$download_status'; then printf complete; elif test -s '$download_pid'; then pid=\$(cat '$download_pid'); if test -n \"\$pid\" && /bin/kill -0 -- \"-\$pid\" 2>/dev/null; then printf running; else printf stopped; fi; else printf missing; fi" +} + +capture_download_evidence() { + local suffix="$1" artifact + artifact="$RUN_ROOT/logs/stage-$node-download-$suffix.log" + if ssh "$node" test -f "$download_log"; then + ssh "$node" "cat '$download_log'" >"$artifact" + chmod 0600 "$artifact" + fi + artifact="$RUN_ROOT/logs/stage-$node-download-$suffix.status" + if ssh "$node" test -f "$download_status"; then + ssh "$node" "cat '$download_status'" >"$artifact" + chmod 0600 "$artifact" + fi + artifact="$RUN_ROOT/logs/stage-$node-download-$suffix.runner.sh" + if ssh "$node" test -f "$download_runner"; then + ssh "$node" "cat '$download_runner'" >"$artifact" + chmod 0600 "$artifact" + fi + artifact="$RUN_ROOT/logs/stage-$node-hf-cache-$suffix.tar" + if ssh "$node" test -d "$incomplete_path/.cache"; then + ssh "$node" "tar -C '$incomplete_path' -cf - .cache" >"$artifact" + chmod 0600 "$artifact" + fi +} + +start_remote_download() { + ssh "$node" bash -s -- \ + "$download_docker_mode" "$download_container" "$download_image" \ + "$CHECKPOINT_REPO" "$CHECKPOINT_REVISION" "$incomplete_path" \ + "$download_status" "$download_log" "$download_pid" "$download_runner" <<'REMOTE' +set -euo pipefail +docker_mode="$1" +container="$2" +image="$3" +repository="$4" +revision="$5" +incomplete="$6" +status_file="$7" +log_file="$8" +pid_file="$9" +runner="${10}" + +case "$docker_mode" in + plain) docker_command=(docker) ;; + sudo) docker_command=(sudo -n docker) ;; + *) printf 'invalid Docker mode\n' >&2; exit 2 ;; +esac + +"${docker_command[@]}" rm -f "$container" >/dev/null 2>&1 || true +rm -f -- "$pid_file" +cat >"$runner" <<'RUNNER' +#!/usr/bin/env bash +set -uo pipefail +docker_mode="$1" +container="$2" +image="$3" +repository="$4" +revision="$5" +incomplete="$6" +status_file="$7" +case "$docker_mode" in + plain) docker_command=(docker) ;; + sudo) docker_command=(sudo -n docker) ;; + *) printf 'invalid Docker mode\n' >&2; exit 2 ;; +esac +"${docker_command[@]}" run --rm --runtime runc --network bridge \ + --name "$container" \ + --user "$(id -u):$(id -g)" \ + --env HOME=/tmp \ + --env HF_HUB_DISABLE_TELEMETRY=1 \ + --env NVIDIA_VISIBLE_DEVICES=void \ + --volume "$incomplete:$incomplete" \ + --entrypoint ionice \ + "$image" -c3 nice -n 19 python3 -c \ + 'from huggingface_hub import snapshot_download; import sys; snapshot_download(repo_id=sys.argv[1], revision=sys.argv[2], local_dir=sys.argv[3], max_workers=1)' \ + "$repository" "$revision" "$incomplete" +status=$? +printf 'EXIT=%s\n' "$status" >"$status_file" +exit "$status" +RUNNER +chmod 0700 "$runner" +nohup setsid "$runner" "$docker_mode" "$container" "$image" "$repository" \ + "$revision" "$incomplete" "$status_file" >"$log_file" 2>&1 "$pid_file" +REMOTE +} + +cancel_remote_download() { + local pid attempt + pid="$(remote_download_pid)" || return 0 + ssh "$node" "$download_docker stop --time 15 '$download_container' >/dev/null 2>&1 || true; /bin/kill -TERM -- '-$pid' 2>/dev/null || true" + for attempt in 1 2 3 4 5; do + ssh "$node" "/bin/kill -0 -- '-$pid' 2>/dev/null" || return 0 + sleep 1 + done + ssh "$node" "/bin/kill -KILL -- '-$pid' 2>/dev/null || true" +} + +cleanup_on_exit() { + local status=$? + trap - EXIT + if (( status != 0 && download_active == 1 && stage_complete == 0 && preserve_download_on_exit == 0 )); then + cancel_remote_download || true + capture_download_evidence "abort-$(date -u +%Y%m%dT%H%M%SZ)" || true + event "stage-$node" NO-GO "download cancelled after staging gate failure" || true + elif (( status != 0 && preserve_download_on_exit == 1 )); then + event "stage-$node" DEFER "monitor transport unavailable; durable downloader left running" || true + fi + exit "$status" +} +trap cleanup_on_exit EXIT + +local_manifest="$RUN_ROOT/baseline/$node-checkpoint.manifest" +if ssh "$node" test -d "$final_path"; then + ssh "$node" test -f "$final_path/MANIFEST.sha256" || + die "existing final checkpoint has no stored manifest at $node:$final_path" + if ! verify_remote "$final_path" "$local_manifest"; then + die "divergent final checkpoint at $node:$final_path; refusing overwrite" + fi + event "stage-$node" GO "existing immutable checkpoint verified" +else + ssh "$node" "umask 077; mkdir -p '$incomplete_path'" + if ssh "$node" test -s "$download_status"; then + status_line="$(ssh "$node" "cat '$download_status'")" + if [[ "$status_line" != "EXIT=0" ]]; then + capture_download_evidence "retry-$(date -u +%Y%m%dT%H%M%SZ)" + ssh "$node" "rm -f -- '$download_log' '$download_status' '$download_pid' '$download_runner'" + fi + fi + download_state="$(remote_download_state)" || { + preserve_download_on_exit=1 + die "cannot observe checkpoint downloader over bounded SSH on $node" + } + case "$download_state" in + complete) ;; + running) download_active=1 ;; + missing|stopped) + start_remote_download + download_state="$(remote_download_state)" || { + preserve_download_on_exit=1 + die "cannot observe newly started checkpoint downloader on $node" + } + require_eq "$node checkpoint downloader state" running "$download_state" + download_active=1 + ;; + *) die "invalid checkpoint downloader state: $download_state" ;; + esac + + window=0 + consecutive_failures=0 + while [[ "$download_state" != complete ]]; do + sleep 60 + window=$((window + 1)) + download_state="$(remote_download_state)" || { + preserve_download_on_exit=1 + die "cannot observe checkpoint downloader over bounded SSH on $node" + } + [[ "$download_state" != stopped && "$download_state" != missing ]] || + die "checkpoint downloader exited without a completion record on $node" + [[ "$download_state" != complete ]] || break + window_snapshot="$RUN_ROOT/metrics/stage-$node-window-$(printf '%05d' "$window").json" + write_glm_window_snapshot "$window_snapshot" 1m || + die "cannot capture GLM staging window from Prometheus" + read -r request_count success_rate p90 < <( + python3 - "$window_snapshot" <<'PY' +import json +import sys + +data = json.load(open(sys.argv[1], encoding="utf-8")) +values = [data.get("request_count"), data.get("success_rate"), data.get("p90_seconds")] +print(*(value if value is not None else "null" for value in values)) +PY + ) + decision="$(stage_window_decision \ + "$baseline_p90" "$request_count" "$success_rate" "$p90")" || + die "invalid GLM staging-window metrics" + window_bad=0 + [[ "$decision" != BREACH\ * ]] || window_bad=1 + if ! check_glm_health; then + decision="BREACH direct-health-check-failed;$decision" + window_bad=1 + fi + if (( window_bad == 1 )); then + consecutive_failures=$((consecutive_failures + 1)) + else + consecutive_failures=0 + fi + event "stage-$node-window-$window" MONITOR "$decision" + if (( window % 5 == 0 )); then + capture "stage-$node-window-$window" bash -c \ + "ssh '$node' 'df -PB1 /home/alecfong; cat /proc/pressure/io'" + fi + (( consecutive_failures < 2 )) || + die "GLM success/latency failed for two consecutive staging windows" + done + download_active=0 + status_line="$(stage_quick_ssh "$node" "cat '$download_status'")" || { + preserve_download_on_exit=1 + die "cannot read checkpoint completion record over bounded SSH on $node" + } + capture_download_evidence "complete-$(date -u +%Y%m%dT%H%M%SZ)" + require_eq "$node checkpoint download" "EXIT=0" "$status_line" + ssh "$node" "rm -f -- '$download_log' '$download_status' '$download_pid' '$download_runner'; if test -d '$incomplete_path/.cache'; then rm -rf -- '$incomplete_path/.cache'; fi" + + verify_remote "$incomplete_path" "$local_manifest" + ssh "$node" "tee '$incomplete_path/MANIFEST.sha256' >/dev/null" <"$local_manifest" + ssh "$node" "mv '$incomplete_path' '$final_path'; chmod -R a-w '$final_path'" + verify_remote "$final_path" "$local_manifest.after" + cmp "$local_manifest" "$local_manifest.after" || + die "checkpoint manifest changed during atomic promotion" + mv "$local_manifest.after" "$local_manifest" + event "stage-$node" GO "checkpoint downloaded, verified, and made immutable" +fi + +free_bytes="$(remote_available_bytes "$node" "$final_path")" +require_ge "$node post-stage free bytes" "$MIN_FREE_AFTER_STAGE_BYTES" "$free_bytes" + +other_node="shamu" +[[ "$node" == "shamu" ]] && other_node="tilikum" +other_manifest="$RUN_ROOT/baseline/$other_node-checkpoint.manifest" +if [[ -f "$other_manifest" ]]; then + cmp "$other_manifest" "$local_manifest" || + die "cross-node checkpoint manifest mismatch" +fi + +if [[ "$node" == "shamu" ]]; then + source_dir="$(ssh shamu 'python3 - <<'"'"'PY'"'"' +import json +import subprocess +from pathlib import Path +inspect = json.loads(subprocess.check_output(["docker", "inspect", "glm52-full-mtp"]))[0] +for mount in inspect.get("Mounts", []): + source = Path(mount.get("Source", "")) + if source.is_dir() and all((source / name).is_file() for name in ("tokenizer.json", "tokenizer_config.json", "chat_template.jinja")): + print(source) + break +else: + raise SystemExit("GLM tokenizer mount not found") +PY')" + tokenizer_local="$RUN_ROOT/baseline/glm52-tokenizer" + mkdir -p "$tokenizer_local" + ssh shamu "cd '$source_dir' && sha256sum tokenizer.json tokenizer_config.json chat_template.jinja | sort" \ + >"$RUN_ROOT/baseline/glm52-tokenizer.source.manifest" + ssh shamu "tar -C '$source_dir' -cf - tokenizer.json tokenizer_config.json chat_template.jinja" | + tar -C "$tokenizer_local" -xf - + (cd "$tokenizer_local" && sha256sum tokenizer.json tokenizer_config.json chat_template.jinja | sort) \ + >"$RUN_ROOT/baseline/glm52-tokenizer.manifest" + cmp "$RUN_ROOT/baseline/glm52-tokenizer.source.manifest" \ + "$RUN_ROOT/baseline/glm52-tokenizer.manifest" || + die "GLM tokenizer hash mismatch after copying from Shamu" + chmod 0600 "$RUN_ROOT/baseline/glm52-tokenizer.source.manifest" + chmod 0600 "$RUN_ROOT/baseline/glm52-tokenizer.manifest" + + tokenizer_parent="/home/nvidia/litellm/energy-pricing" + tokenizer_final="$tokenizer_parent/glm52-tokenizer" + tokenizer_temp="$tokenizer_parent/.glm52-tokenizer-${CHECKPOINT_REVISION:0:10}.tmp" + if ssh spark-1 test -d "$tokenizer_final"; then + ssh spark-1 "cd '$tokenizer_final' && sha256sum tokenizer.json tokenizer_config.json chat_template.jinja | sort" \ + >"$RUN_ROOT/baseline/glm52-tokenizer.remote.manifest" + cmp "$RUN_ROOT/baseline/glm52-tokenizer.manifest" \ + "$RUN_ROOT/baseline/glm52-tokenizer.remote.manifest" || + die "divergent staged GLM tokenizer; refusing overwrite" + else + if ssh spark-1 test -d "$tokenizer_temp"; then + ssh spark-1 "cd '$tokenizer_temp' && sha256sum tokenizer.json tokenizer_config.json chat_template.jinja | sort" \ + >"$RUN_ROOT/baseline/glm52-tokenizer.temp.manifest" || + die "incomplete staged GLM tokenizer cannot be verified" + cmp "$RUN_ROOT/baseline/glm52-tokenizer.manifest" \ + "$RUN_ROOT/baseline/glm52-tokenizer.temp.manifest" || + die "divergent incomplete GLM tokenizer; refusing overwrite" + else + ssh spark-1 "umask 077; mkdir '$tokenizer_temp'" + tar -C "$tokenizer_local" -cf - . | + ssh spark-1 "tar -C '$tokenizer_temp' -xf -" + fi + ssh spark-1 "cd '$tokenizer_temp' && sha256sum tokenizer.json tokenizer_config.json chat_template.jinja | sort" \ + >"$RUN_ROOT/baseline/glm52-tokenizer.remote.manifest" + cmp "$RUN_ROOT/baseline/glm52-tokenizer.manifest" \ + "$RUN_ROOT/baseline/glm52-tokenizer.remote.manifest" || + die "GLM tokenizer hash mismatch after transfer" + ssh spark-1 "mv '$tokenizer_temp' '$tokenizer_final'; chmod -R a-w '$tokenizer_final'" + fi +fi + +stage_complete=1 +printf 'GO node=%s checkpoint=%s\n' "$node" "$final_path" diff --git a/webster/deepseek-v41/scripts/start-deepseek-v41-tp2.sh b/webster/deepseek-v41/scripts/start-deepseek-v41-tp2.sh new file mode 100755 index 0000000..e1d387f --- /dev/null +++ b/webster/deepseek-v41/scripts/start-deepseek-v41-tp2.sh @@ -0,0 +1,552 @@ +#!/usr/bin/env bash +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$script_dir/common.sh" + +run_root="" +profile_file="" +verify_only=0 +while (($#)); do + case "$1" in + --run-root) + [[ $# -ge 2 ]] || die "--run-root requires a value" + run_root="$2" + shift 2 + ;; + --profile-file) + [[ $# -ge 2 ]] || die "--profile-file requires a value" + profile_file="$2" + shift 2 + ;; + --verify-only) + verify_only=1 + shift + ;; + *) die "unknown argument: $1" ;; + esac +done +init_run_root "$(validate_run_root "$run_root")" + +profile_name="eager-128k" +max_model_len="131072" +max_num_seqs="1" +gpu_memory_utilization="0.90" +enforce_eager="1" +enable_prefix_caching="0" +max_num_batched_tokens="" +enable_expert_parallel="0" +speculative_method="none" +num_speculative_tokens="0" +draft_sample_method="greedy" +rejection_sample_method="standard" +enable_flashinfer_autotune="1" + +if [[ -n "$profile_file" ]]; then + profile_file="$(canonical_path "$profile_file")" + case "$profile_file" in + "$RUN_ROOT"/*) ;; + *) die "profile file must resolve beneath the run root" ;; + esac + [[ -f "$profile_file" && ! -L "$profile_file" ]] || + die "profile file must be a regular non-symlink" + require_eq "profile file mode" 600 "$(stat -c %a "$profile_file")" + require_eq "profile file owner" "$(id -un)" "$(stat -c %U "$profile_file")" + + declare -A profile_values=() + declare -A profile_seen=() + while IFS='=' read -r key value; do + value="${value%$'\r'}" + [[ -n "$key" ]] || continue + [[ "$key" != \#* ]] || continue + case "$key" in + PROFILE_NAME|MAX_MODEL_LEN|MAX_NUM_SEQS|GPU_MEMORY_UTILIZATION|\ + ENFORCE_EAGER|ENABLE_PREFIX_CACHING|MAX_NUM_BATCHED_TOKENS|\ + ENABLE_EXPERT_PARALLEL|SPECULATIVE_METHOD|NUM_SPECULATIVE_TOKENS|\ + DRAFT_SAMPLE_METHOD|REJECTION_SAMPLE_METHOD|ENABLE_FLASHINFER_AUTOTUNE) ;; + *) die "unknown profile field: $key" ;; + esac + [[ -z "${profile_seen[$key]:-}" ]] || die "duplicate profile field: $key" + [[ -n "$value" ]] || die "profile field $key is empty" + profile_seen[$key]=1 + profile_values[$key]="$value" + done <"$profile_file" + + for key in PROFILE_NAME MAX_MODEL_LEN MAX_NUM_SEQS GPU_MEMORY_UTILIZATION \ + ENFORCE_EAGER ENABLE_PREFIX_CACHING MAX_NUM_BATCHED_TOKENS \ + ENABLE_EXPERT_PARALLEL; do + [[ -n "${profile_seen[$key]:-}" ]] || die "profile field $key is missing" + done + profile_name="${profile_values[PROFILE_NAME]}" + max_model_len="${profile_values[MAX_MODEL_LEN]}" + max_num_seqs="${profile_values[MAX_NUM_SEQS]}" + gpu_memory_utilization="${profile_values[GPU_MEMORY_UTILIZATION]}" + enforce_eager="${profile_values[ENFORCE_EAGER]}" + enable_prefix_caching="${profile_values[ENABLE_PREFIX_CACHING]}" + max_num_batched_tokens="${profile_values[MAX_NUM_BATCHED_TOKENS]}" + enable_expert_parallel="${profile_values[ENABLE_EXPERT_PARALLEL]}" + speculative_method="${profile_values[SPECULATIVE_METHOD]:-$speculative_method}" + num_speculative_tokens="${profile_values[NUM_SPECULATIVE_TOKENS]:-$num_speculative_tokens}" + draft_sample_method="${profile_values[DRAFT_SAMPLE_METHOD]:-$draft_sample_method}" + rejection_sample_method="${profile_values[REJECTION_SAMPLE_METHOD]:-$rejection_sample_method}" + enable_flashinfer_autotune="${profile_values[ENABLE_FLASHINFER_AUTOTUNE]:-$enable_flashinfer_autotune}" + + [[ "$profile_name" =~ ^[a-z0-9][a-z0-9._-]{0,63}$ ]] || + die "PROFILE_NAME is invalid" + [[ "$max_model_len" =~ ^[1-9][0-9]*$ ]] || die "MAX_MODEL_LEN is invalid" + [[ "$max_num_seqs" =~ ^[1-9][0-9]*$ ]] || die "MAX_NUM_SEQS is invalid" + [[ "$max_num_batched_tokens" =~ ^[1-9][0-9]*$ ]] || + die "MAX_NUM_BATCHED_TOKENS is invalid" + [[ "$gpu_memory_utilization" =~ ^0\.[0-9]+$ ]] || + die "GPU_MEMORY_UTILIZATION is invalid" + for toggle in "$enforce_eager" "$enable_prefix_caching" \ + "$enable_expert_parallel" "$enable_flashinfer_autotune"; do + [[ "$toggle" == 0 || "$toggle" == 1 ]] || die "profile toggle must be 0 or 1" + done + case "$speculative_method" in + none) + require_eq "NUM_SPECULATIVE_TOKENS for method none" 0 \ + "$num_speculative_tokens" + ;; + dspark) + require_eq "DeepSeek V4.1 DSpark block size" 5 \ + "$num_speculative_tokens" + [[ "$enforce_eager" == 0 ]] || + die "DSpark profile must enable CUDA graphs" + ;; + *) die "SPECULATIVE_METHOD is invalid" ;; + esac + case "$draft_sample_method" in + greedy|probabilistic) ;; + *) die "DRAFT_SAMPLE_METHOD is invalid" ;; + esac + case "$rejection_sample_method" in + standard|block) ;; + *) die "REJECTION_SAMPLE_METHOD is invalid" ;; + esac +fi + +manifest="$RUN_ROOT/manifest.env" +[[ -f "$manifest" && ! -L "$manifest" ]] || die "manifest.env is missing" +require_eq "manifest mode" 600 "$(stat -c %a "$manifest")" +manifest_value() { + local key="$1" value + value="$(awk -F= -v key="$key" '$1 == key {sub(/^[^=]*=/, ""); print; exit}' "$manifest")" + [[ -n "$value" ]] || die "manifest field $key is empty" + printf '%s\n' "$value" +} +image_id="$(manifest_value VLLM_IMAGE_ID)" +[[ "$image_id" =~ ^sha256:[0-9a-f]{64}$ ]] || die "manifest image ID is invalid" + +common_args="/model --served-model-name $DEEPSEEK_SERVED_MODEL --nnodes 2 --master-addr $SHAMU_RAIL --master-port 29511 --distributed-executor-backend mp --tensor-parallel-size 2 --pipeline-parallel-size 1 --engram-config '{\"cpu_offload\":true,\"embedding_across_dp\":false}'" +[[ "$enforce_eager" == 0 ]] || common_args+=" --enforce-eager" +common_args+=" --max-model-len $max_model_len --max-num-seqs $max_num_seqs --gpu-memory-utilization $gpu_memory_utilization --disable-custom-all-reduce --enable-auto-tool-choice --tool-call-parser deepseek_v41 --reasoning-parser deepseek_v41" +[[ "$enable_prefix_caching" == 0 ]] || common_args+=" --enable-prefix-caching" +[[ -z "$max_num_batched_tokens" ]] || + common_args+=" --max-num-batched-tokens $max_num_batched_tokens" +[[ "$enable_expert_parallel" == 0 ]] || common_args+=" --enable-expert-parallel" +[[ "$enable_flashinfer_autotune" == 0 ]] && + common_args+=" --no-enable-flashinfer-autotune" +if [[ "$speculative_method" == dspark ]]; then + common_args+=" --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":$num_speculative_tokens,\"draft_sample_method\":\"$draft_sample_method\",\"rejection_sample_method\":\"$rejection_sample_method\"}'" +fi +docker_args="--network host --ipc host --shm-size 34359738368 --gpus all --device /dev/infiniband --restart no --ulimit memlock=-1:-1 --ulimit stack=67108864:67108864 --cap-add CAP_IPC_LOCK --security-opt label=disable -w /vllm-workspace -v $DEEPSEEK_CHECKPOINT_PATH:/model:ro -v $DEEPSEEK_CACHE_PATH:/root/.cache -e NCCL_IB_HCA=$NCCL_HCA -e NCCL_SOCKET_IFNAME=$NCCL_IFACE -e GLOO_SOCKET_IFNAME=$NCCL_IFACE -e NCCL_NET_GDR_LEVEL=5 -e NCCL_IB_GID_INDEX=3 -e NCCL_CUMEM_ENABLE=1 -e NCCL_DEBUG=INFO -e NCCL_DEBUG_SUBSYS=INIT,NET -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800" +profile_fingerprint="$(python3 - "$image_id" "$common_args" "$docker_args" \ + "$SHAMU_RAIL" "$TILIKUM_RAIL" "$SHAMU_NETBIRD" "$CANARY_PORT" <<'PY' +import hashlib +import json +import sys + +payload = json.dumps(sys.argv[1:], ensure_ascii=True, separators=(",", ":")) +print(hashlib.sha256(payload.encode("utf-8")).hexdigest()) +PY +)" +expected_rank0_cmd="$(python3 - "$common_args --node-rank 0 --host $SHAMU_NETBIRD --port $CANARY_PORT" <<'PY' +import base64 +import json +import shlex +import sys + +print(base64.b64encode(json.dumps(shlex.split(sys.argv[1])).encode()).decode()) +PY +)" +expected_rank1_cmd="$(python3 - "$common_args --node-rank 1 --headless" <<'PY' +import base64 +import json +import shlex +import sys + +print(base64.b64encode(json.dumps(shlex.split(sys.argv[1])).encode()).decode()) +PY +)" + +if [[ "${WEBSTER_LIFECYCLE_TEST_MODE:-0}" == "1" ]]; then + existing_shamu="${TEST_LIFECYCLE_DEEPSEEK_SHAMU_RUNNING:-false}" + existing_tilikum="${TEST_LIFECYCLE_DEEPSEEK_TILIKUM_RUNNING:-false}" +else + existing_shamu="$(ssh shamu "docker inspect '$DEEPSEEK_CONTAINER' --format '{{.State.Running}}' 2>/dev/null || printf false")" + existing_tilikum="$(ssh tilikum "sudo -n docker inspect '$DEEPSEEK_CONTAINER' --format '{{.State.Running}}' 2>/dev/null || printf false")" +fi + +validate_live_deepseek_rank() { + local node="$1" docker_mode="$2" rank="$3" expected_cmd="$4" + local host_ip="$5" state + state="$(ssh "$node" python3 - "$docker_mode" "$DEEPSEEK_CONTAINER" \ + "$image_id" "$expected_cmd" "$profile_fingerprint" "$rank" "$host_ip" \ + "$DEEPSEEK_CHECKPOINT_PATH" "$DEEPSEEK_CACHE_PATH" "$NCCL_HCA" \ + "$NCCL_IFACE" <<'PY' +import base64 +import json +import subprocess +import sys + +( + docker_mode, + container, + image_id, + expected_cmd_b64, + profile_fingerprint, + rank, + host_ip, + checkpoint_path, + cache_path, + nccl_hca, + nccl_iface, +) = sys.argv[1:] +docker = ["docker"] if docker_mode == "direct" else ["sudo", "-n", "docker"] +try: + data = json.loads( + subprocess.check_output(docker + ["inspect", container], text=True) + )[0] +except (subprocess.CalledProcessError, json.JSONDecodeError, IndexError) as error: + raise SystemExit(f"cannot inspect existing DeepSeek rank {rank}: {error}") + +def require(condition: bool, message: str) -> None: + if not condition: + raise SystemExit(f"existing DeepSeek rank {rank} {message}") + +expected_cmd = json.loads(base64.b64decode(expected_cmd_b64)) +config = data.get("Config") or {} +host = data.get("HostConfig") or {} +state = data.get("State") or {} +require(state.get("Running") is True, "is not running") +require(data.get("Image") == image_id, "uses the wrong image ID") +require(data.get("RestartCount") == 0, "has a nonzero restart count") +require(config.get("Entrypoint") == ["vllm", "serve"], "has the wrong entrypoint") +require(config.get("Cmd") == expected_cmd, "does not match the requested profile") +require(config.get("WorkingDir") == "/vllm-workspace", "has the wrong working directory") +require(host.get("NetworkMode") == "host", "is not on host networking") +require(host.get("IpcMode") == "host", "does not use host IPC") +require(host.get("ShmSize") == 34359738368, "has the wrong shared-memory size") +require((host.get("RestartPolicy") or {}).get("Name") == "no", "has the wrong restart policy") +require("label=disable" in (host.get("SecurityOpt") or []), "lacks SELinux label isolation") +require( + any(value in {"IPC_LOCK", "CAP_IPC_LOCK"} for value in (host.get("CapAdd") or [])), + "lacks IPC_LOCK", +) +ulimits = {item.get("Name"): item for item in (host.get("Ulimits") or [])} +require( + (ulimits.get("memlock") or {}).get("Soft") == -1 + and (ulimits.get("memlock") or {}).get("Hard") == -1, + "has the wrong memlock ulimit", +) +require( + (ulimits.get("stack") or {}).get("Soft") == 67108864 + and (ulimits.get("stack") or {}).get("Hard") == 67108864, + "has the wrong stack ulimit", +) +mounts = {item.get("Destination"): item for item in (data.get("Mounts") or [])} +require( + (mounts.get("/model") or {}).get("Source") == checkpoint_path + and (mounts.get("/model") or {}).get("RW") is False, + "does not use the requested read-only checkpoint", +) +require( + (mounts.get("/root/.cache") or {}).get("Source") == cache_path, + "does not use the requested cache path", +) +require( + any( + item.get("PathOnHost") == "/dev/infiniband" + for item in (host.get("Devices") or []) + ), + "lacks the InfiniBand device", +) +require(bool(host.get("DeviceRequests")), "lacks a GPU device request") +environment = {} +for item in config.get("Env") or []: + if "=" in item: + key, value = item.split("=", 1) + environment[key] = value +required_environment = { + "VLLM_HOST_IP": host_ip, + "NCCL_IB_HCA": nccl_hca, + "NCCL_SOCKET_IFNAME": nccl_iface, + "GLOO_SOCKET_IFNAME": nccl_iface, + "NCCL_NET_GDR_LEVEL": "5", + "NCCL_IB_GID_INDEX": "3", + "NCCL_CUMEM_ENABLE": "1", + "NCCL_DEBUG": "INFO", + "NCCL_DEBUG_SUBSYS": "INIT,NET", + "HF_HUB_OFFLINE": "1", + "TRANSFORMERS_OFFLINE": "1", + "VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS": "1800", +} +for key, expected in required_environment.items(): + require(environment.get(key) == expected, f"has the wrong {key}") +if rank == "0": + require(bool(environment.get("VLLM_API_KEY")), "lacks the rank-0 API key") +else: + require(not environment.get("VLLM_API_KEY"), "received VLLM_API_KEY") + require(not environment.get("DSPARK_API_KEYS"), "received DSPARK_API_KEYS") +labels = config.get("Labels") or {} +profile_label = labels.get("ai.webster.profile-sha256") +rank_label = labels.get("ai.webster.rank") +generation = labels.get("ai.webster.generation") +labels_present = tuple( + isinstance(label, str) and bool(label) + for label in (profile_label, rank_label, generation) +) +if any(labels_present): + require(all(labels_present), "has partial serving labels") + require(profile_label == profile_fingerprint, "has a mismatched profile fingerprint") + require(rank_label == rank, "has a mismatched rank label") +print( + json.dumps( + { + "container_id": data.get("Id"), + "generation": generation, + "started_at": state.get("StartedAt"), + }, + separators=(",", ":"), + ) +) +PY +)" || die "$node existing DeepSeek generation validation failed" + printf '%s\n' "$state" +} + +validate_deepseek_pair_generation() { + "$script_dir/verify-serving-pair.py" \ + --rank0-state-json "$1" \ + --rank1-state-json "$2" \ + --max-start-skew-seconds 120 >/dev/null +} + +if [[ "$existing_shamu" == true && "$existing_tilikum" == true ]]; then + if [[ "${WEBSTER_LIFECYCLE_TEST_MODE:-0}" == "1" ]]; then + require_eq "existing authenticated API" 1 "${TEST_LIFECYCLE_AUTH_OK:-1}" + require_eq "existing listener isolation" 1 "${TEST_LIFECYCLE_LISTENERS_OK:-1}" + require_eq "existing rank 1 keyless" 1 "${TEST_LIFECYCLE_RANK1_KEYLESS:-1}" + require_eq "existing NCCL rail" 1 "${TEST_LIFECYCLE_NCCL_OK:-1}" + require_eq "existing DeepSeek image" 1 \ + "${TEST_LIFECYCLE_DEEPSEEK_IMAGE_MATCH:-1}" + require_eq "existing DeepSeek profile" 1 \ + "${TEST_LIFECYCLE_DEEPSEEK_PROFILE_MATCH:-1}" + require_eq "existing DeepSeek runtime shape" 1 \ + "${TEST_LIFECYCLE_DEEPSEEK_RUNTIME_MATCH:-1}" + require_eq "existing DeepSeek restart state" 1 \ + "${TEST_LIFECYCLE_DEEPSEEK_RESTARTS_OK:-1}" + require_eq "existing DeepSeek pair generation" 1 \ + "${TEST_LIFECYCLE_DEEPSEEK_GENERATION_MATCH:-1}" + else + shamu_generation="$(validate_live_deepseek_rank \ + shamu direct 0 "$expected_rank0_cmd" "$SHAMU_RAIL")" + tilikum_generation="$(validate_live_deepseek_rank \ + tilikum sudo 1 "$expected_rank1_cmd" "$TILIKUM_RAIL")" + validate_deepseek_pair_generation "$shamu_generation" "$tilikum_generation" + curl -fsS --connect-timeout 3 --max-time 5 \ + "http://$SHAMU_NETBIRD:$CANARY_PORT/health" >/dev/null || + die "both DeepSeek ranks run but direct health is unavailable" + ssh tilikum "sudo -n docker inspect '$DEEPSEEK_CONTAINER' --format '{{json .Args}}' | grep -q -- --headless" || + die "both DeepSeek ranks run but rank 1 is not headless" + fi + event start-deepseek-v41 GO "existing healthy two-rank deployment retained" + printf 'DeepSeek V4.1 TP2 already healthy; no relaunch performed\n' + exit 0 +fi +if [[ "$verify_only" == 1 ]]; then + die "DeepSeek pair is not already healthy for the requested profile" +fi +if [[ "$existing_shamu" == true || "$existing_tilikum" == true ]]; then + "$script_dir/stop-deepseek-v41-tp2.sh" --run-root "$RUN_ROOT" +fi + +test_lifecycle_gate() { + require_eq "Shamu GLM stopped" false "${TEST_LIFECYCLE_GLM_SHAMU_RUNNING:-false}" + require_eq "Tilikum GLM stopped" false "${TEST_LIFECYCLE_GLM_TILIKUM_RUNNING:-false}" + require_eq "checkpoint manifests" 1 "${TEST_LIFECYCLE_CHECKPOINTS_MATCH:-1}" + require_eq "runtime image IDs" 1 "${TEST_LIFECYCLE_IMAGES_MATCH:-1}" + require_ge "station free bytes" "$MIN_FREE_AFTER_STAGE_BYTES" \ + "${TEST_LIFECYCLE_FREE_BYTES:-$MIN_FREE_AFTER_STAGE_BYTES}" + require_eq "station rails" 1 "${TEST_LIFECYCLE_RAILS_OK:-1}" + require_eq "Shamu-to-Tilikum SSH" 1 "${TEST_LIFECYCLE_SSH_OK:-1}" +} + +live_lifecycle_gate() { + local shamu_glm tilikum_glm shamu_manifest tilikum_manifest + shamu_glm="$(ssh shamu "docker inspect '$GLM_CONTAINER' --format '{{.State.Running}}' 2>/dev/null || printf false")" + tilikum_glm="$(ssh tilikum "sudo -n docker inspect '$GLM_CONTAINER' --format '{{.State.Running}}' 2>/dev/null || printf false")" + require_eq "Shamu GLM stopped" false "$shamu_glm" + require_eq "Tilikum GLM stopped" false "$tilikum_glm" + for node in shamu tilikum; do + ssh "$node" "test -d '$DEEPSEEK_CHECKPOINT_PATH' && test ! -L '$DEEPSEEK_CHECKPOINT_PATH' && test -r '$DEEPSEEK_CHECKPOINT_PATH/MANIFEST.sha256'" || + die "$node checkpoint is incomplete" + require_ge "$node free bytes" "$MIN_FREE_AFTER_STAGE_BYTES" \ + "$(remote_available_bytes "$node" "$DEEPSEEK_CHECKPOINT_PATH")" + ssh "$node" "ip addr show dev '$NCCL_IFACE' | grep -q '$([[ "$node" == shamu ]] && printf %s "$SHAMU_RAIL" || printf %s "$TILIKUM_RAIL")/' && grep -q ACTIVE /sys/class/infiniband/'$NCCL_HCA'/ports/1/state" || + die "$node rail/HCA validation failed" + done + shamu_manifest="$(ssh shamu "sha256sum '$DEEPSEEK_CHECKPOINT_PATH/MANIFEST.sha256' | awk '{print \$1}'")" + tilikum_manifest="$(ssh tilikum "sha256sum '$DEEPSEEK_CHECKPOINT_PATH/MANIFEST.sha256' | awk '{print \$1}'")" + require_eq "checkpoint manifest hash" "$shamu_manifest" "$tilikum_manifest" + require_eq "Shamu runtime image" "$image_id" \ + "$(ssh shamu "docker image inspect '$image_id' --format '{{.Id}}'")" + require_eq "Tilikum runtime image" "$image_id" \ + "$(ssh tilikum "sudo -n docker image inspect '$image_id' --format '{{.Id}}'")" + ssh shamu "env -u SSH_AUTH_SOCK ssh -o ForwardAgent=no -o BatchMode=yes -o ConnectTimeout=5 -o StrictHostKeyChecking=accept-new alecfong@$TILIKUM_RAIL true" || + die "Shamu cannot reach Tilikum passwordlessly over the direct rail" + ssh shamu "test -s '$DEEPSEEK_API_ENV_FILE' && test \"\$(stat -c %a '$DEEPSEEK_API_ENV_FILE')\" = 600 && grep -q '^VLLM_API_KEY=' '$DEEPSEEK_API_ENV_FILE'" || + die "DeepSeek rank-0 API env file is missing or unsafe" + ssh shamu "test -z \"\$(ss -lntH '( sport = :$CANARY_PORT )')\"" || + die "Shamu port $CANARY_PORT is already in use" + ssh tilikum "test -z \"\$(ss -lntH '( sport = :$CANARY_PORT )')\"" || + die "Tilikum port $CANARY_PORT is already in use" + ssh shamu "mkdir -p '$DEEPSEEK_CACHE_PATH'" + ssh tilikum "mkdir -p '$DEEPSEEK_CACHE_PATH'" +} + +event start-deepseek-v41 START "profile $profile_name TP2/PP1 canary" +if [[ "${WEBSTER_LIFECYCLE_TEST_MODE:-0}" == "1" ]]; then + test_lifecycle_gate +else + live_lifecycle_gate +fi + +generation_id="$(date -u +%Y%m%dT%H%M%SZ)-$$" +generation_labels="--label ai.webster.profile-sha256=$profile_fingerprint --label ai.webster.generation=$generation_id" +# Preserve any stopped or failed generation before docker rm can discard its +# inspect metadata or logs. capture() redacts credential-shaped environment +# fields and assigns a unique mode-0600 evidence path on every attempt. +capture deepseek-v41-shamu-prior-generation ssh shamu \ + "docker inspect '$DEEPSEEK_CONTAINER' 2>&1 || true; docker logs --tail 1000 '$DEEPSEEK_CONTAINER' 2>&1 || true" +capture deepseek-v41-tilikum-prior-generation ssh tilikum \ + "sudo -n docker inspect '$DEEPSEEK_CONTAINER' 2>&1 || true; sudo -n docker logs --tail 1000 '$DEEPSEEK_CONTAINER' 2>&1 || true" +shamu_launch="docker rm -f $DEEPSEEK_CONTAINER >/dev/null 2>&1 || true; docker run -d --name $DEEPSEEK_CONTAINER $docker_args $generation_labels --label ai.webster.rank=0 -e VLLM_HOST_IP=$SHAMU_RAIL --env-file $DEEPSEEK_API_ENV_FILE $image_id $common_args --node-rank 0 --host $SHAMU_NETBIRD --port $CANARY_PORT" +tilikum_launch="sudo -n docker rm -f $DEEPSEEK_CONTAINER >/dev/null 2>&1 || true; sudo -n env -u VLLM_API_KEY -u DSPARK_API_KEYS docker run -d --name $DEEPSEEK_CONTAINER $docker_args $generation_labels --label ai.webster.rank=1 -e VLLM_HOST_IP=$TILIKUM_RAIL $image_id $common_args --node-rank 1 --headless" + +if [[ "${WEBSTER_LIFECYCLE_TEST_MODE:-0}" == "1" ]]; then + { + printf 'deepseek-profile %s --tensor-parallel-size 2 --pipeline-parallel-size 1 ' "$profile_name" + printf '%s ' '--engram-config' '{"cpu_offload":true,"embedding_across_dp":false}' + printf '%s --host %s --port %s\n' "$common_args" "$SHAMU_NETBIRD" "$CANARY_PORT" + } >>"$TEST_LOG" +fi + +if ! ssh shamu "$shamu_launch" >/dev/null; then + event start-deepseek-v41 NO-GO "rank 0 launch failed" + die "DeepSeek rank 0 launch failed" +fi +if ! ssh tilikum "$tilikum_launch" >/dev/null; then + set +e + "$script_dir/stop-deepseek-v41-tp2.sh" --run-root "$RUN_ROOT" + set -e + event start-deepseek-v41 NO-GO "rank 1 launch failed; both ranks stopped" + die "DeepSeek rank 1 launch failed" +fi + +if [[ "${WEBSTER_LIFECYCLE_TEST_MODE:-0}" == "1" ]]; then + require_eq "authenticated API" 1 "${TEST_LIFECYCLE_AUTH_OK:-1}" + require_eq "listener isolation" 1 "${TEST_LIFECYCLE_LISTENERS_OK:-1}" + require_eq "rank 1 keyless" 1 "${TEST_LIFECYCLE_RANK1_KEYLESS:-1}" + require_eq "NCCL rail" 1 "${TEST_LIFECYCLE_NCCL_OK:-1}" + event start-deepseek-v41 GO \ + "profile $profile_name authenticated; rank 1 headless/keyless; NCCL HCA pinned" + printf 'DeepSeek V4.1 profile %s TP2/PP1 canary validated\n' "$profile_name" + exit 0 +fi + +ready=0 +memory_log="$RUN_ROOT/metrics/deepseek-v41-startup-memory.tsv" +printf 'captured_at\tshamu_mib\ttilikum_mib\n' >"$memory_log" +chmod 0600 "$memory_log" +for _ in $(seq 1 60); do + shamu_memory="$(ssh shamu "nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits" 2>/dev/null || true)" + tilikum_memory="$(ssh tilikum "nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits" 2>/dev/null || true)" + printf '%s\t%s\t%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + "${shamu_memory:-unknown}" "${tilikum_memory:-unknown}" >>"$memory_log" + shamu_running="$(ssh shamu "docker inspect '$DEEPSEEK_CONTAINER' --format '{{.State.Running}}' 2>/dev/null" || true)" + tilikum_running="$(ssh tilikum "sudo -n docker inspect '$DEEPSEEK_CONTAINER' --format '{{.State.Running}}' 2>/dev/null" || true)" + if [[ "$shamu_running" == true && "$tilikum_running" == true ]] && + ssh shamu python3 - "$DEEPSEEK_API_ENV_FILE" "$SHAMU_NETBIRD" "$CANARY_PORT" <<'PY' >/dev/null 2>&1 +from pathlib import Path +import sys +import urllib.request + +path, host, port = sys.argv[1:] +line = Path(path).read_text().strip() +key = line.split("=", 1)[1] +request = urllib.request.Request( + f"http://{host}:{port}/v1/models", + headers={"Authorization": "Bearer " + key}, +) +with urllib.request.urlopen(request, timeout=5) as response: + if response.status != 200: + raise SystemExit(1) +PY + then + ready=1 + break + fi + sleep 20 +done +if [[ "$ready" != 1 ]]; then + capture deepseek-v41-shamu-failed-start ssh shamu "docker logs --tail 500 '$DEEPSEEK_CONTAINER'" + capture deepseek-v41-tilikum-failed-start ssh tilikum "sudo -n docker logs --tail 500 '$DEEPSEEK_CONTAINER'" + set +e + "$script_dir/stop-deepseek-v41-tp2.sh" --run-root "$RUN_ROOT" + set -e + event start-deepseek-v41 NO-GO "authenticated readiness absent after 20 minutes" + die "DeepSeek did not become ready within 20 minutes" +fi + +require_eq "unauthenticated OpenAI status" 401 \ + "$(curl -sS --max-time 5 -o /dev/null -w '%{http_code}' "http://$SHAMU_NETBIRD:$CANARY_PORT/v1/models")" +wrong_key_header="$(printf 'Authorization: %s %s' Bearer invalid-canary-key)" +require_eq "wrong-key OpenAI status" 401 \ + "$(curl -sS --max-time 5 -o /dev/null -w '%{http_code}' -H "$wrong_key_header" "http://$SHAMU_NETBIRD:$CANARY_PORT/v1/models")" +require_eq "health status" 200 \ + "$(curl -sS --max-time 5 -o /dev/null -w '%{http_code}' "http://$SHAMU_NETBIRD:$CANARY_PORT/health")" +require_eq "metrics status" 200 \ + "$(curl -sS --max-time 5 -o /dev/null -w '%{http_code}' "http://$SHAMU_NETBIRD:$CANARY_PORT/metrics")" +shamu_listener="$(ssh shamu "ss -lntH '( sport = :$CANARY_PORT )' | awk '{print \$4}'")" +require_eq "DeepSeek rank 0 listener" "$SHAMU_NETBIRD:$CANARY_PORT" "$shamu_listener" +[[ -z "$(ssh tilikum "ss -lntH '( sport = :$CANARY_PORT )'")" ]] || + die "DeepSeek rank 1 exposes port $CANARY_PORT" +ssh tilikum python3 - "$DEEPSEEK_CONTAINER" <<'PY' +import json +import subprocess +import sys + +name = sys.argv[1] +data = json.loads(subprocess.check_output(["sudo", "-n", "docker", "inspect", name]))[0] +args = data.get("Args", []) +if "--headless" not in args and "--headless" not in " ".join(args): + raise SystemExit("rank 1 is not headless") +for item in data.get("Config", {}).get("Env", []): + if item.startswith(("VLLM_API_KEY=", "DSPARK_API_KEYS=")) and item.split("=", 1)[1]: + raise SystemExit("rank 1 received a serving key") +if data.get("RestartCount") != 0: + raise SystemExit("rank 1 restart count is nonzero") +PY +ssh shamu "docker logs '$DEEPSEEK_CONTAINER' 2>&1 | grep -Eq '($NCCL_HCA|NCCL_IB_HCA)'" || + die "rank 0 logs do not prove the pinned NCCL HCA" +ssh tilikum "sudo -n docker logs '$DEEPSEEK_CONTAINER' 2>&1 | grep -Eq '($NCCL_HCA|NCCL_IB_HCA)'" || + die "rank 1 logs do not prove the pinned NCCL HCA" +capture deepseek-v41-shamu-started ssh shamu \ + "docker inspect '$DEEPSEEK_CONTAINER'; docker logs --tail 500 '$DEEPSEEK_CONTAINER'; free -b; nvidia-smi" +capture deepseek-v41-tilikum-started ssh tilikum \ + "sudo -n docker inspect '$DEEPSEEK_CONTAINER'; sudo -n docker logs --tail 500 '$DEEPSEEK_CONTAINER'; free -b; nvidia-smi" +event start-deepseek-v41 GO \ + "profile $profile_name authenticated; rank 1 headless/keyless; NCCL HCA pinned" +printf 'DeepSeek V4.1 profile %s TP2/PP1 canary validated\n' "$profile_name" diff --git a/webster/deepseek-v41/scripts/start-glm52-tp2.sh b/webster/deepseek-v41/scripts/start-glm52-tp2.sh new file mode 100755 index 0000000..6145bb5 --- /dev/null +++ b/webster/deepseek-v41/scripts/start-glm52-tp2.sh @@ -0,0 +1,350 @@ +#!/usr/bin/env bash +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$script_dir/common.sh" + +run_root="" +while (($#)); do + case "$1" in + --run-root) + [[ $# -ge 2 ]] || die "--run-root requires a value" + run_root="$2" + shift 2 + ;; + *) die "unknown argument: $1" ;; + esac +done +init_run_root "$(validate_run_root "$run_root")" + +event start-glm52 START "coordinated cold rollback" + +if [[ "${WEBSTER_LIFECYCLE_TEST_MODE:-0}" == "1" ]]; then + existing_shamu="${TEST_LIFECYCLE_GLM_SHAMU_RUNNING:-false}" + existing_tilikum="${TEST_LIFECYCLE_GLM_TILIKUM_RUNNING:-false}" +else + existing_shamu="$(ssh shamu "docker inspect '$GLM_CONTAINER' --format '{{.State.Running}}' 2>/dev/null || printf false")" + existing_tilikum="$(ssh tilikum "sudo -n docker inspect '$GLM_CONTAINER' --format '{{.State.Running}}' 2>/dev/null || printf false")" +fi + +validate_live_glm_rank() { + local node="$1" docker_mode="$2" rank="$3" host_ip="$4" state + state="$(ssh "$node" python3 - "$docker_mode" "$GLM_CONTAINER" "$rank" \ + "$host_ip" "$SHAMU_RAIL" "$CANARY_PORT" <<'PY' +import json +import shlex +import subprocess +import sys + +docker_mode, container, rank, host_ip, master_addr, port = sys.argv[1:] +docker = ["docker"] if docker_mode == "direct" else ["sudo", "-n", "docker"] +try: + data = json.loads( + subprocess.check_output(docker + ["inspect", container], text=True) + )[0] +except (subprocess.CalledProcessError, json.JSONDecodeError, IndexError) as error: + raise SystemExit(f"cannot inspect existing GLM rank {rank}: {error}") + +def require(condition: bool, message: str) -> None: + if not condition: + raise SystemExit(f"existing GLM rank {rank} {message}") + +config = data.get("Config") or {} +host = data.get("HostConfig") or {} +runtime = data.get("State") or {} +require(runtime.get("Running") is True, "is not running") +require(data.get("RestartCount") == 0, "has a nonzero restart count") +require(config.get("Entrypoint") == ["bash"], "has the wrong entrypoint") +command = config.get("Cmd") +require( + isinstance(command, list) and len(command) == 2 and command[0] == "-lc", + "has the wrong container command", +) +serve_lines = [ + line.strip() + for line in command[1].splitlines() + if line.strip().startswith("exec vllm serve ") +] +require(len(serve_lines) == 1, "does not contain one vLLM serve command") +tokens = shlex.split(serve_lines[0]) +require(tokens[:4] == ["exec", "vllm", "serve", "/model"], "has the wrong serve prefix") +value_flags = { + "--served-model-name", + "--host", + "--port", + "--nnodes", + "--node-rank", + "--master-addr", + "--master-port", + "--tensor-parallel-size", + "--speculative-config", + "--max-model-len", + "--block-size", + "--max-num-seqs", + "--kv-cache-dtype", + "--gpu-memory-utilization", + "--cpu-offload-gb", + "--kernel-config", + "--compilation-config", + "--tool-call-parser", + "--reasoning-parser", +} +boolean_flags = { + "--trust-remote-code", + "--disable-custom-all-reduce", + "--enable-prefix-caching", + "--enable-auto-tool-choice", + "--headless", +} +values = {} +flags = set() +index = 4 +while index < len(tokens): + token = tokens[index] + require(token not in values and token not in flags, f"duplicates {token}") + if token in value_flags: + require(index + 1 < len(tokens), f"omits the value for {token}") + values[token] = tokens[index + 1] + index += 2 + elif token in boolean_flags: + flags.add(token) + index += 1 + else: + require(False, f"uses unexpected serve argument {token}") +expected_values = { + "--served-model-name": "nvidia/GLM-5.2-NVFP4", + "--host": host_ip, + "--port": port, + "--nnodes": "2", + "--node-rank": rank, + "--master-addr": master_addr, + "--master-port": "29501", + "--tensor-parallel-size": "2", + "--max-model-len": "320000", + "--block-size": "128", + "--max-num-seqs": "8", + "--kv-cache-dtype": "fp8", + "--gpu-memory-utilization": "0.97", + "--cpu-offload-gb": "0", + "--tool-call-parser": "glm47", + "--reasoning-parser": "glm47", +} +for name, expected in expected_values.items(): + require(values.get(name) == expected, f"has the wrong {name}") +require(set(values) == value_flags, "does not match the complete requested profile") +required_flags = boolean_flags - {"--headless"} +require(required_flags.issubset(flags), "omits a required serve flag") +require(("--headless" in flags) == (rank == "1"), "has the wrong headless role") +try: + speculative = json.loads(values["--speculative-config"]) + kernel = json.loads(values["--kernel-config"]) + compilation = json.loads(values["--compilation-config"]) +except json.JSONDecodeError as error: + raise SystemExit(f"existing GLM rank {rank} has malformed JSON config: {error}") +require( + speculative == {"method": "mtp", "num_speculative_tokens": 4}, + "has the wrong speculative profile", +) +require( + kernel + == {"enable_flashinfer_autotune": False, "moe_backend": "flashinfer_cutlass"}, + "has the wrong kernel profile", +) +require( + compilation + == { + "cudagraph_mode": "FULL_DECODE_ONLY", + "cudagraph_capture_sizes": [5, 10, 20, 40], + "pass_config": {"fuse_allreduce_rms": False}, + }, + "has the wrong compilation profile", +) +require(host.get("NetworkMode") == "host", "is not on host networking") +require(host.get("IpcMode") == "host", "does not use host IPC") +require(config.get("WorkingDir") == "/vllm-workspace", "has the wrong working directory") +require(host.get("ShmSize") == 34359738368, "has the wrong shared-memory size") +require((host.get("RestartPolicy") or {}).get("Name") == "unless-stopped", "has the wrong restart policy") +require("label=disable" in (host.get("SecurityOpt") or []), "lacks SELinux label isolation") +require( + any(value in {"IPC_LOCK", "CAP_IPC_LOCK"} for value in (host.get("CapAdd") or [])), + "lacks IPC_LOCK", +) +ulimits = {item.get("Name"): item for item in (host.get("Ulimits") or [])} +require( + (ulimits.get("memlock") or {}).get("Soft") == -1 + and (ulimits.get("memlock") or {}).get("Hard") == -1, + "has the wrong memlock ulimit", +) +require( + (ulimits.get("stack") or {}).get("Soft") == 67108864 + and (ulimits.get("stack") or {}).get("Hard") == 67108864, + "has the wrong stack ulimit", +) +require( + any( + item.get("PathOnHost") == "/dev/infiniband" + for item in (host.get("Devices") or []) + ), + "lacks the InfiniBand device", +) +require(bool(host.get("DeviceRequests")), "lacks a GPU device request") +mounts = {item.get("Destination"): item for item in (data.get("Mounts") or [])} +require( + (mounts.get("/model") or {}).get("Source") + == "/home/alecfong/glm52/models/GLM-5.2-NVFP4-full" + and (mounts.get("/model") or {}).get("RW") is False, + "does not use the exact read-only checkpoint", +) +require( + (mounts.get("/root/.cache") or {}).get("Source") + == "/home/alecfong/glm52/exo-repro/cache" + and (mounts.get("/root/.cache") or {}).get("RW") is True, + "does not use the exact cache mount", +) +environment = {} +for item in config.get("Env") or []: + if "=" in item: + key, value = item.split("=", 1) + environment[key] = value +require(environment.get("VLLM_HOST_IP") == host_ip, "has the wrong VLLM_HOST_IP") +if rank == "0": + require(bool(environment.get("VLLM_API_KEY")), "lacks the rank-0 API key") +else: + require(not environment.get("VLLM_API_KEY"), "received VLLM_API_KEY") + require(not environment.get("DSPARK_API_KEYS"), "received DSPARK_API_KEYS") +print( + json.dumps( + {"image": data.get("Image"), "started_at": runtime.get("StartedAt")}, + separators=(",", ":"), + ) +) +PY +)" || die "$node existing GLM generation validation failed" + printf '%s\n' "$state" +} + +validate_glm_pair_generation() { + "$script_dir/verify-serving-pair.py" \ + --rank0-state-json "$1" \ + --rank1-state-json "$2" \ + --max-start-skew-seconds 120 \ + --require-matching-image >/dev/null +} + +if [[ "$existing_shamu" == true && "$existing_tilikum" == true ]]; then + if [[ "${WEBSTER_LIFECYCLE_TEST_MODE:-0}" == "1" ]]; then + require_eq "existing GLM authenticated health" 1 "${TEST_LIFECYCLE_AUTH_OK:-1}" + require_eq "existing GLM listener isolation" 1 "${TEST_LIFECYCLE_LISTENERS_OK:-1}" + require_eq "existing GLM rank 1 keyless" 1 "${TEST_LIFECYCLE_RANK1_KEYLESS:-1}" + require_eq "existing GLM image" 1 "${TEST_LIFECYCLE_GLM_IMAGE_MATCH:-1}" + require_eq "existing GLM profile" 1 "${TEST_LIFECYCLE_GLM_PROFILE_MATCH:-1}" + require_eq "existing GLM runtime shape" 1 \ + "${TEST_LIFECYCLE_GLM_RUNTIME_MATCH:-1}" + require_eq "existing GLM restart state" 1 \ + "${TEST_LIFECYCLE_GLM_RESTARTS_OK:-1}" + require_eq "existing GLM pair generation" 1 \ + "${TEST_LIFECYCLE_GLM_GENERATION_MATCH:-1}" + else + shamu_generation="$(validate_live_glm_rank \ + shamu direct 0 "$SHAMU_NETBIRD")" + tilikum_generation="$(validate_live_glm_rank \ + tilikum sudo 1 "$TILIKUM_NETBIRD")" + validate_glm_pair_generation "$shamu_generation" "$tilikum_generation" + check_glm_health || die "both GLM ranks run but direct health is unavailable" + ssh tilikum "sudo -n docker inspect '$GLM_CONTAINER' --format '{{json .Args}}' | grep -q -- --headless" || + die "both GLM ranks run but rank 1 is not headless" + fi + event start-glm52 GO "existing healthy two-rank deployment retained" + printf 'GLM-5.2 TP2 already healthy; no relaunch performed\n' + exit 0 +fi +if [[ "$existing_shamu" == true || "$existing_tilikum" == true ]]; then + "$script_dir/stop-glm52-tp2.sh" --run-root "$RUN_ROOT" +fi +"$script_dir/stop-deepseek-v41-tp2.sh" --run-root "$RUN_ROOT" + +shamu_command="GLM_NODE_RANK=0 GLM_HOST_IP=$SHAMU_RAIL GLM_MAX_MODEL_LEN=320000 GLM_MAX_NUM_SEQS=8 /home/alecfong/serve_glm52.sh" +tilikum_command="GLM_NODE_RANK=1 GLM_HOST_IP=$TILIKUM_RAIL GLM_MAX_MODEL_LEN=320000 GLM_MAX_NUM_SEQS=8 GLM_DOCKER='sudo -n docker' /home/alecfong/serve_glm52.sh" + +if ! ssh shamu "$shamu_command" >/dev/null; then + event start-glm52 NO-GO "rank 0 launch failed" + die "GLM rank 0 launch failed" +fi +if ! ssh tilikum "$tilikum_command" >/dev/null; then + set +e + "$script_dir/stop-glm52-tp2.sh" --run-root "$RUN_ROOT" + set -e + event start-glm52 NO-GO "rank 1 launch failed; both ranks stopped" + die "GLM rank 1 launch failed" +fi + +if [[ "${WEBSTER_LIFECYCLE_TEST_MODE:-0}" == "1" ]]; then + require_eq "GLM authenticated health" 1 "${TEST_LIFECYCLE_AUTH_OK:-1}" + require_eq "GLM listener isolation" 1 "${TEST_LIFECYCLE_LISTENERS_OK:-1}" + require_eq "GLM rank 1 keyless" 1 "${TEST_LIFECYCLE_RANK1_KEYLESS:-1}" + event start-glm52 GO "test-mode rollback checks passed" + printf 'GLM-5.2 TP2 rollback validated\n' + exit 0 +fi + +ready=0 +memory_log="$RUN_ROOT/metrics/glm52-rank1-startup-memory.tsv" +printf 'captured_at\tmemory_used_mib\n' >"$memory_log" +chmod 0600 "$memory_log" +for _ in $(seq 1 60); do + memory="$(ssh tilikum "nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits" 2>/dev/null || true)" + printf '%s\t%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "${memory:-unknown}" >>"$memory_log" + shamu_running="$(ssh shamu "docker inspect '$GLM_CONTAINER' --format '{{.State.Running}}' 2>/dev/null" || true)" + tilikum_running="$(ssh tilikum "sudo -n docker inspect '$GLM_CONTAINER' --format '{{.State.Running}}' 2>/dev/null" || true)" + if [[ "$shamu_running" == true && "$tilikum_running" == true ]] && + ssh shamu python3 - <<'PY' >/dev/null 2>&1 +from pathlib import Path +import urllib.request + +line = Path("/home/alecfong/.config/glm52/vllm-api.env").read_text().strip() +key = line.split("=", 1)[1] +request = urllib.request.Request( + "http://100.73.140.127:8000/v1/models", + headers={"Authorization": "Bearer " + key}, +) +with urllib.request.urlopen(request, timeout=5) as response: + if response.status != 200: + raise SystemExit(1) +PY + then + ready=1 + break + fi + sleep 20 +done +if [[ "$ready" != 1 ]]; then + capture glm52-shamu-failed-start ssh shamu "docker logs --tail 400 '$GLM_CONTAINER'" + capture glm52-tilikum-failed-start ssh tilikum "sudo -n docker logs --tail 400 '$GLM_CONTAINER'" + set +e + "$script_dir/stop-glm52-tp2.sh" --run-root "$RUN_ROOT" + set -e + event start-glm52 NO-GO "direct authenticated health absent after 20 minutes" + die "GLM rollback did not become healthy within 20 minutes" +fi + +ssh tilikum "sudo -n docker inspect '$GLM_CONTAINER' --format '{{json .Args}}' | grep -q -- --headless" || + die "GLM rank 1 is not headless" +ssh tilikum python3 - "$GLM_CONTAINER" <<'PY' +import json +import subprocess +import sys + +name = sys.argv[1] +data = json.loads(subprocess.check_output(["sudo", "-n", "docker", "inspect", name]))[0] +for item in data.get("Config", {}).get("Env", []): + if item.startswith(("VLLM_API_KEY=", "DSPARK_API_KEYS=")) and item.split("=", 1)[1]: + raise SystemExit("rank 1 received a serving key") +if data.get("RestartCount") != 0: + raise SystemExit("rank 1 restart count is nonzero") +PY +shamu_listener="$(ssh shamu "ss -lntH '( sport = :$CANARY_PORT )' | awk '{print \$4}'")" +require_eq "GLM rank 0 listener" "$SHAMU_NETBIRD:$CANARY_PORT" "$shamu_listener" +[[ -z "$(ssh tilikum "ss -lntH '( sport = :$CANARY_PORT )'")" ]] || + die "GLM rank 1 exposes port $CANARY_PORT" +event start-glm52 GO "both ranks healthy; rank 1 headless/keyless; listener isolated" +printf 'GLM-5.2 TP2 rollback validated\n' diff --git a/webster/deepseek-v41/scripts/stop-deepseek-v41-tp2.sh b/webster/deepseek-v41/scripts/stop-deepseek-v41-tp2.sh new file mode 100755 index 0000000..17dc56d --- /dev/null +++ b/webster/deepseek-v41/scripts/stop-deepseek-v41-tp2.sh @@ -0,0 +1,95 @@ +#!/usr/bin/env bash +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$script_dir/common.sh" + +run_root="" +while (($#)); do + case "$1" in + --run-root) + [[ $# -ge 2 ]] || die "--run-root requires a value" + run_root="$2" + shift 2 + ;; + *) die "unknown argument: $1" ;; + esac +done +init_run_root "$(validate_run_root "$run_root")" + +stop_rank() { + local node="$1" status=0 exists + if [[ "${WEBSTER_LIFECYCLE_TEST_MODE:-0}" != "1" ]]; then + if [[ "$node" == "shamu" ]]; then + exists="$(ssh shamu "docker inspect '$DEEPSEEK_CONTAINER' --format '{{.Id}}' 2>/dev/null" || true)" + else + exists="$(ssh tilikum "sudo -n docker inspect '$DEEPSEEK_CONTAINER' --format '{{.Id}}' 2>/dev/null" || true)" + fi + [[ -n "$exists" ]] || return 0 + fi + if [[ "$node" == "shamu" ]]; then + ssh shamu docker stop --time 120 "$DEEPSEEK_CONTAINER" >/dev/null || status=$? + else + ssh tilikum sudo -n docker stop --time 120 "$DEEPSEEK_CONTAINER" >/dev/null || status=$? + fi + if [[ "${WEBSTER_LIFECYCLE_TEST_MODE:-0}" == "1" && + "${TEST_LIFECYCLE_STOP_FAIL_NODE:-}" == "$node" ]]; then + return 9 + fi + return "$status" +} + +container_running() { + local node="$1" + if [[ "${WEBSTER_LIFECYCLE_TEST_MODE:-0}" == "1" ]]; then + if [[ "$node" == "shamu" ]]; then + printf '%s\n' "${TEST_LIFECYCLE_DEEPSEEK_SHAMU_RUNNING:-false}" + else + printf '%s\n' "${TEST_LIFECYCLE_DEEPSEEK_TILIKUM_RUNNING:-false}" + fi + return + fi + if [[ "$node" == "shamu" ]]; then + ssh shamu "docker inspect '$DEEPSEEK_CONTAINER' --format '{{.State.Running}}' 2>/dev/null || printf false" + else + ssh tilikum "sudo -n docker inspect '$DEEPSEEK_CONTAINER' --format '{{.State.Running}}' 2>/dev/null || printf false" + fi +} + +event stop-deepseek-v41 START "coordinated two-rank stop" +if [[ "${WEBSTER_LIFECYCLE_TEST_MODE:-0}" != "1" ]]; then + capture deepseek-v41-shamu-prestop ssh shamu \ + "if docker inspect '$DEEPSEEK_CONTAINER' >/dev/null 2>&1; then docker inspect '$DEEPSEEK_CONTAINER'; docker logs --tail 400 '$DEEPSEEK_CONTAINER'; fi" + capture deepseek-v41-tilikum-prestop ssh tilikum \ + "if sudo -n docker inspect '$DEEPSEEK_CONTAINER' >/dev/null 2>&1; then sudo -n docker inspect '$DEEPSEEK_CONTAINER'; sudo -n docker logs --tail 400 '$DEEPSEEK_CONTAINER'; fi" +fi + +set +e +stop_rank shamu & shamu_pid=$! +stop_rank tilikum & tilikum_pid=$! +wait "$shamu_pid"; shamu_status=$? +wait "$tilikum_pid"; tilikum_status=$? +set -e +if (( shamu_status != 0 || tilikum_status != 0 )); then + set +e + stop_rank shamu + stop_rank tilikum + set -e + event stop-deepseek-v41 NO-GO "at least one rank stop failed; both stops retried" + die "coordinated DeepSeek stop failed; both ranks received a retry" +fi + +require_eq "Shamu DeepSeek stopped" false "$(container_running shamu)" +require_eq "Tilikum DeepSeek stopped" false "$(container_running tilikum)" +if [[ "${WEBSTER_LIFECYCLE_TEST_MODE:-0}" == "1" ]]; then + require_eq "station listeners released" 1 "${TEST_LIFECYCLE_LISTENERS_OK:-1}" +else + for node in shamu tilikum; do + ssh "$node" "test -z \"\$(ss -lntH '( sport = :$CANARY_PORT )')\"" || + die "$node still has a port-$CANARY_PORT listener" + ssh "$node" "test -z \"\$(nvidia-smi --query-compute-apps=pid --format=csv,noheader 2>/dev/null)\"" || + die "$node GPU still has a compute process" + done +fi +event stop-deepseek-v41 GO "both ranks stopped; GPUs and port released" +printf 'DeepSeek V4.1 TP2 ranks stopped\n' diff --git a/webster/deepseek-v41/scripts/stop-glm52-tp2.sh b/webster/deepseek-v41/scripts/stop-glm52-tp2.sh new file mode 100755 index 0000000..b996687 --- /dev/null +++ b/webster/deepseek-v41/scripts/stop-glm52-tp2.sh @@ -0,0 +1,95 @@ +#!/usr/bin/env bash +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$script_dir/common.sh" + +run_root="" +while (($#)); do + case "$1" in + --run-root) + [[ $# -ge 2 ]] || die "--run-root requires a value" + run_root="$2" + shift 2 + ;; + *) die "unknown argument: $1" ;; + esac +done +init_run_root "$(validate_run_root "$run_root")" + +stop_rank() { + local node="$1" status=0 + if [[ "$node" == "shamu" ]]; then + ssh shamu docker stop --time 120 "$GLM_CONTAINER" >/dev/null || status=$? + else + ssh tilikum sudo -n docker stop --time 120 "$GLM_CONTAINER" >/dev/null || status=$? + fi + if [[ "${WEBSTER_LIFECYCLE_TEST_MODE:-0}" == "1" && + "${TEST_LIFECYCLE_STOP_FAIL_NODE:-}" == "$node" ]]; then + return 9 + fi + return "$status" +} + +container_running() { + local node="$1" + if [[ "${WEBSTER_LIFECYCLE_TEST_MODE:-0}" == "1" ]]; then + if [[ "$node" == "shamu" ]]; then + printf '%s\n' "${TEST_LIFECYCLE_GLM_SHAMU_RUNNING:-false}" + else + printf '%s\n' "${TEST_LIFECYCLE_GLM_TILIKUM_RUNNING:-false}" + fi + return + fi + if [[ "$node" == "shamu" ]]; then + ssh shamu "docker inspect '$GLM_CONTAINER' --format '{{.State.Running}}' 2>/dev/null || printf false" + else + ssh tilikum "sudo -n docker inspect '$GLM_CONTAINER' --format '{{.State.Running}}' 2>/dev/null || printf false" + fi +} + +verify_released() { + require_eq "Shamu GLM stopped" false "$(container_running shamu)" + require_eq "Tilikum GLM stopped" false "$(container_running tilikum)" + if [[ "${WEBSTER_LIFECYCLE_TEST_MODE:-0}" == "1" ]]; then + require_eq "station listeners released" 1 "${TEST_LIFECYCLE_LISTENERS_OK:-1}" + return + fi + for node in shamu tilikum; do + docker_prefix="docker" + [[ "$node" == "tilikum" ]] && docker_prefix="sudo -n docker" + ssh "$node" "test -z \"\$($docker_prefix ps -q --filter name='^/${GLM_CONTAINER}\$')\"" || + die "$node GLM container is still running" + ssh "$node" "test -z \"\$(nvidia-smi --query-compute-apps=pid --format=csv,noheader 2>/dev/null)\"" || + die "$node GPU still has a compute process" + ssh "$node" "test -z \"\$(ss -lntH '( sport = :$CANARY_PORT )')\"" || + die "$node still has a port-$CANARY_PORT listener" + done +} + +event stop-glm52 START "coordinated two-rank stop" +if [[ "${WEBSTER_LIFECYCLE_TEST_MODE:-0}" != "1" ]]; then + capture glm52-shamu-prestop ssh shamu \ + "docker inspect '$GLM_CONTAINER'; docker logs --tail 400 '$GLM_CONTAINER'" + capture glm52-tilikum-prestop ssh tilikum \ + "sudo -n docker inspect '$GLM_CONTAINER'; sudo -n docker logs --tail 400 '$GLM_CONTAINER'" +fi + +set +e +stop_rank shamu & shamu_pid=$! +stop_rank tilikum & tilikum_pid=$! +wait "$shamu_pid"; shamu_status=$? +wait "$tilikum_pid"; tilikum_status=$? +set -e +if (( shamu_status != 0 || tilikum_status != 0 )); then + set +e + stop_rank shamu + stop_rank tilikum + set -e + event stop-glm52 NO-GO "at least one rank stop failed; both stops retried" + die "coordinated GLM stop failed; both ranks received a retry" +fi + +verify_released +event stop-glm52 GO "both ranks stopped; GPUs and port released" +printf 'GLM-5.2 TP2 ranks stopped\n' diff --git a/webster/deepseek-v41/scripts/summarize-aiperf.py b/webster/deepseek-v41/scripts/summarize-aiperf.py new file mode 100755 index 0000000..956f8c0 --- /dev/null +++ b/webster/deepseek-v41/scripts/summarize-aiperf.py @@ -0,0 +1,385 @@ +#!/usr/bin/env python3 +"""Summarize AIPerf records before trusting aggregate profile metrics.""" + +from __future__ import annotations + +import argparse +import json +import math +import os +import re +import sys +import tempfile +from collections import Counter +from pathlib import Path +from typing import Any + + +PROFILING_COMPLETE_MARKER = "Phase profiling (profiling) complete" +DISPLAYED_COUNT = r"(?:\d+|\d{1,3}(?:,\d{3})+)" +PROFILING_COMPLETE_PATTERN = re.compile( + r"(?:^|\s)Phase profiling \(profiling\) complete \| " + rf"completed={DISPLAYED_COUNT}, cancelled={DISPLAYED_COUNT}, " + rf"errors={DISPLAYED_COUNT} \| " + r"sessions: completed=(?P\d+), " + r"cancelled=(?P\d+) \| " + r"elapsed=(?P\d+(?:\.\d+)?)s(?: \(runner\.py:\d+\))?$" +) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--profile", required=True, type=Path) + parser.add_argument("--records", required=True, type=Path) + parser.add_argument("--server-metrics", type=Path) + parser.add_argument("--timing-log", type=Path) + parser.add_argument("--output-json", required=True, type=Path) + return parser.parse_args() + + +def load_json(path: Path) -> Any: + if not path.is_file() or path.is_symlink(): + raise RuntimeError(f"input must be a regular non-symlink file: {path}") + return json.loads(path.read_text(encoding="utf-8")) + + +def load_records(path: Path) -> list[dict[str, Any]]: + if not path.is_file() or path.is_symlink(): + raise RuntimeError(f"records must be a regular non-symlink file: {path}") + records: list[dict[str, Any]] = [] + for line_number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1): + if not line.strip(): + continue + value = json.loads(line) + if not isinstance(value, dict): + raise RuntimeError(f"record line {line_number} is not an object") + records.append(value) + return records + + +def metric_stat( + profile: dict[str, Any], name: str, stat: str, legacy_name: str +) -> float | None: + current = profile.get(name) + if isinstance(current, dict): + value = current.get(stat) + if isinstance(value, (int, float)) and not isinstance(value, bool): + return float(value) + metrics = profile.get("metrics") + if not isinstance(metrics, dict): + return None + value = metrics.get(legacy_name) + if isinstance(value, dict): + value = value.get("mean") + if isinstance(value, (int, float)) and not isinstance(value, bool): + return float(value) + return None + + +def error_type(value: Any) -> str: + if isinstance(value, dict): + value = value.get("type") or value.get("message") or "unknown" + text = str(value or "unknown").strip() + match = re.search(r"([A-Za-z_][A-Za-z0-9_]*(?:Error|Exception|Timeout))", text) + return match.group(1) if match else text[:160] + + +def record_failed(record: dict[str, Any]) -> bool: + if record.get("error"): + return True + status = record.get("status_code", record.get("status")) + if status is not None: + return not isinstance(status, int) or not 200 <= status < 300 + return not isinstance(record.get("metadata"), dict) or not isinstance( + record.get("metrics"), dict + ) + + +def peak_concurrency(records: list[dict[str, Any]]) -> int | None: + events: list[tuple[int, int]] = [] + for record in records: + metadata = record.get("metadata") + if not isinstance(metadata, dict): + continue + start = metadata.get("request_start_ns") + end = metadata.get("request_end_ns") + if ( + isinstance(start, int) + and not isinstance(start, bool) + and isinstance(end, int) + and not isinstance(end, bool) + and end >= start + ): + events.extend(((start, 1), (end, -1))) + if not events: + return None + active = maximum = 0 + for _, change in sorted(events, key=lambda item: (item[0], item[1])): + active += change + maximum = max(maximum, active) + return maximum + + +def finite_number(value: Any, context: str) -> float: + if ( + not isinstance(value, (int, float)) + or isinstance(value, bool) + or not math.isfinite(value) + ): + raise RuntimeError(f"{context} must be a finite number") + return float(value) + + +def metric_series( + metrics: dict[str, Any], name: str, expected_type: str +) -> list[dict[str, Any]] | None: + if name not in metrics: + return None + metric = metrics[name] + if not isinstance(metric, dict): + raise RuntimeError(f"server metric {name} must be an object") + if metric.get("type") != expected_type: + raise RuntimeError( + f"server metric {name} must have type {expected_type}" + ) + series = metric.get("series") + if not isinstance(series, list): + raise RuntimeError(f"server metric {name} series must be a list") + for index, item in enumerate(series): + if not isinstance(item, dict): + raise RuntimeError(f"server metric {name} series {index} must be an object") + endpoint = item.get("endpoint_url") + if not isinstance(endpoint, str) or not endpoint: + raise RuntimeError( + f"server metric {name} series {index} endpoint_url must be a string" + ) + labels = item.get("labels") + if labels is not None and ( + not isinstance(labels, dict) + or any( + not isinstance(key, str) or not isinstance(value, str) + for key, value in labels.items() + ) + ): + raise RuntimeError( + f"server metric {name} series {index} labels must be a string map" + ) + stats = item.get("stats") + if stats is not None and not isinstance(stats, dict): + raise RuntimeError( + f"server metric {name} series {index} stats must be an object" + ) + return series + + +def series_stat(item: dict[str, Any], name: str, stat: str) -> float | None: + stats = item.get("stats") + if stats is None or stats.get(stat) is None: + return None + return finite_number(stats[stat], f"server metric {name} {stat}") + + +def sum_counter_totals( + metrics: dict[str, Any], name: str, scope: str | None = None +) -> float | None: + series = metric_series(metrics, name, "counter") + if series is None: + return None + values: list[float] = [] + seen: set[tuple[str, tuple[tuple[str, str], ...]]] = set() + for item in series: + labels = item.get("labels") or {} + if scope is not None and labels.get("scope") != scope: + continue + identity = (item["endpoint_url"], tuple(sorted(labels.items()))) + if identity in seen: + raise RuntimeError(f"server metric {name} contains duplicate series") + seen.add(identity) + value = series_stat(item, name, "total") + if value is None: + return None + values.append(value) + return sum(values) if values else None + + +def gauge_stat( + metrics: dict[str, Any], name: str, stat: str, scope: str | None = None +) -> float | None: + series = metric_series(metrics, name, "gauge") + if series is None: + return None + values: list[float] = [] + seen: set[tuple[str, tuple[tuple[str, str], ...]]] = set() + for item in series: + labels = item.get("labels") or {} + if scope is not None and labels.get("scope") != scope: + continue + identity = (item["endpoint_url"], tuple(sorted(labels.items()))) + if identity in seen: + raise RuntimeError(f"server metric {name} contains duplicate series") + seen.add(identity) + value = series_stat(item, name, stat) + if value is None: + return None + values.append(value) + if not values: + return None + return sum(values) if stat == "avg" else max(values) + + +def summarize_server_metrics(value: Any) -> dict[str, float | None]: + if not isinstance(value, dict): + raise RuntimeError("server metrics root must be an object") + metrics = value.get("metrics") + if not isinstance(metrics, dict): + raise RuntimeError("server metrics metrics must be an object") + hits = sum_counter_totals(metrics, "vllm:prefix_cache_hits") + queries = sum_counter_totals(metrics, "vllm:prefix_cache_queries") + hit_ratio = ( + hits / queries if hits is not None and queries not in (None, 0.0) else None + ) + return { + "prefix_cache_hits": hits, + "prefix_cache_queries": queries, + "prefix_cache_hit_ratio": hit_ratio, + "kv_cache_usage_max": gauge_stat( + metrics, "vllm:kv_cache_usage_perc", "max" + ), + "preemptions_total": sum_counter_totals( + metrics, "vllm:num_preemptions" + ), + "running_requests_max": gauge_stat( + metrics, "vllm:num_requests_running", "max" + ), + "waiting_requests_max": gauge_stat( + metrics, "vllm:num_requests_waiting", "max" + ), + "wall_power_avg_watts": gauge_stat( + metrics, "power_total_watts", "avg", "wall" + ), + "facility_power_avg_watts": gauge_stat( + metrics, "power_total_watts", "avg", "facility" + ), + "facility_energy_kwh": sum_counter_totals( + metrics, "power_energy_kwh", "facility" + ), + "facility_cost_dollars": sum_counter_totals(metrics, "power_cost_dollars"), + } + + +def summarize_timing_log(path: Path) -> dict[str, float | int]: + if not path.is_file() or path.is_symlink(): + raise RuntimeError(f"timing log must be a regular non-symlink file: {path}") + candidates = [ + line + for line in path.read_text(encoding="utf-8").splitlines() + if PROFILING_COMPLETE_MARKER in line + ] + if len(candidates) != 1: + raise RuntimeError( + "timing log must contain exactly one profiling-complete line" + ) + match = PROFILING_COMPLETE_PATTERN.search(candidates[0]) + if match is None: + raise RuntimeError("profiling-complete timing log line is malformed") + completed = int(match.group("completed")) + cancelled = int(match.group("cancelled")) + elapsed = finite_number( + float(match.group("elapsed")), "root workflow elapsed seconds" + ) + if elapsed <= 0.0: + raise RuntimeError( + "root workflow elapsed seconds must be greater than zero" + ) + return { + "completed_root_workflows": completed, + "cancelled_root_workflows": cancelled, + "root_workflow_elapsed_seconds": elapsed, + "completed_root_workflows_per_second": completed / elapsed, + } + + +def write_private_json(path: Path, value: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + descriptor, temporary_name = tempfile.mkstemp( + prefix=f".{path.name}.", dir=path.parent + ) + temporary = Path(temporary_name) + try: + with os.fdopen(descriptor, "w", encoding="utf-8") as handle: + json.dump(value, handle, indent=2, sort_keys=True) + handle.write("\n") + os.chmod(temporary, 0o600) + os.replace(temporary, path) + os.chmod(path, 0o600) + finally: + temporary.unlink(missing_ok=True) + + +def main() -> int: + args = parse_args() + try: + profile = load_json(args.profile) + records = load_records(args.records) + if not isinstance(profile, dict): + raise RuntimeError("profile root must be an object") + failures = [record for record in records if record_failed(record)] + counts = Counter(error_type(record.get("error")) for record in failures) + depths = Counter( + str(metadata["agent_depth"]) + for record in records + if isinstance((metadata := record.get("metadata")), dict) + and isinstance(metadata.get("agent_depth"), int) + and not isinstance(metadata["agent_depth"], bool) + ) + root_trees = { + metadata["root_correlation_id"] + for record in records + if isinstance((metadata := record.get("metadata")), dict) + and isinstance(metadata.get("root_correlation_id"), str) + and metadata["root_correlation_id"] + } + completed = len(records) - len(failures) + success_rate = completed / len(records) if records else 0.0 + summary = { + "schema_version": 1, + "requests": len(records), + "completed": completed, + "errors": len(failures), + "server_success_rate": success_rate, + "error_types": dict(sorted(counts.items())), + "request_throughput": metric_stat( + profile, "request_throughput", "avg", "request_throughput_avg" + ), + "ttft_p90_ms": metric_stat( + profile, "time_to_first_token", "p90", "time_to_first_token_p90" + ), + "output_token_throughput": metric_stat( + profile, + "output_token_throughput", + "avg", + "output_token_throughput_avg", + ), + "request_peak_concurrency": peak_concurrency(records), + "root_session_trees": len(root_trees), + "request_count_by_agent_depth": dict(sorted(depths.items())), + "qualified": bool(records) and success_rate >= 0.99 and not failures, + } + if args.server_metrics is not None: + summary.update(summarize_server_metrics(load_json(args.server_metrics))) + if args.timing_log is not None: + summary.update(summarize_timing_log(args.timing_log)) + write_private_json(args.output_json, summary) + except (OSError, RuntimeError, UnicodeError, json.JSONDecodeError) as error: + print(f"ERROR: {error}", file=sys.stderr) + return 1 + print( + "AIPerf summary " + f"requests={summary['requests']} errors={summary['errors']} " + f"qualified={str(summary['qualified']).lower()}" + ) + return 0 if summary["qualified"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/webster/deepseek-v41/scripts/sync-runbook.sh b/webster/deepseek-v41/scripts/sync-runbook.sh new file mode 100755 index 0000000..1c13409 --- /dev/null +++ b/webster/deepseek-v41/scripts/sync-runbook.sh @@ -0,0 +1,261 @@ +#!/usr/bin/env bash +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +repo_root="$(cd "$script_dir/../../.." && pwd)" +webster_source="${WEBSTER_CLUSTER_SOURCE:-$HOME/.claude/skills/webster-cluster}" +migration_source="${WEBSTER_MIGRATION_SOURCE:-$repo_root/.agent/skills/migrating-webster-models}" +check_only=0 +force=0 +while (( $# )); do + case "$1" in + --check) check_only=1; shift ;; + --force) force=1; shift ;; + *) printf 'ERROR: unknown argument: %s\n' "$1" >&2; exit 1 ;; + esac +done + +skills=(webster-cluster migrating-webster-models) +canonical_path() { + python3 - "$1" <<'PY' +from pathlib import Path +import sys + +print(Path(sys.argv[1]).resolve(strict=False)) +PY +} + +skill_source() { + case "$1" in + webster-cluster) printf '%s\n' "$webster_source" ;; + migrating-webster-models) printf '%s\n' "$migration_source" ;; + *) printf 'ERROR: unsupported skill: %s\n' "$1" >&2; return 1 ;; + esac +} +for skill in "${skills[@]}"; do + source="$(skill_source "$skill")" + [[ -f "$source/SKILL.md" ]] || { + printf 'ERROR: source skill missing: %s (%s)\n' "$skill" "$source" >&2 + exit 1 + } +done + +render_hermes() { + local skill="$1" source="$2" output="$3" + mkdir -p "$output/$skill" + rsync -a --delete "$source/" "$output/$skill/" + python3 - "$skill" "$output/$skill/SKILL.md" <<'PY' +import re +import sys +from pathlib import Path + +name, raw_path = sys.argv[1:] +path = Path(raw_path) +text = path.read_text(encoding="utf-8") +match = re.match(r"^---\n(.*?)\n---\n", text, re.S) +if not match: + raise SystemExit("source skill has no frontmatter") +description_match = re.search(r"^description:\s*(.*)$", match.group(1), re.M) +if not description_match: + raise SystemExit("source skill has no description") +description = description_match.group(1).strip() +tags = ( + "[webster, cluster, gpu, netbird, litellm, prometheus, grafana, monitoring]" + if name == "webster-cluster" + else "[webster, migration, gpu, vllm, litellm, aiperf, rollback]" +) +header = f"""--- +name: {name} +description: {description} +version: 1.0.0 +license: MIT +platforms: [linux, macos] +metadata: + hermes: + category: devops + tags: {tags} +--- +""" +path.write_text(header + text[match.end():], encoding="utf-8") +PY +} + +sync_local() { + local destination="$1" skill source destination_path + for skill in "${skills[@]}"; do + source="$(skill_source "$skill")" + destination_path="$destination/$skill" + if [[ "$(canonical_path "$destination_path")" == "$(canonical_path "$source")" ]]; then + continue + fi + if [[ -L "$destination_path" ]]; then + verify_tree "$source" "$destination_path" "existing $skill symlink" || { + printf 'ERROR: refusing to overwrite divergent skill symlink: %s\n' \ + "$destination_path" >&2 + return 1 + } + continue + fi + if [[ -e "$destination_path" ]] && + ! diff -qr "$source" "$destination_path" >/dev/null; then + if (( ! force )); then + printf 'ERROR: refusing to overwrite divergent skill directory: %s\n' \ + "$destination_path" >&2 + return 1 + fi + fi + mkdir -p "$destination_path" + rsync -a --delete "$source/" "$destination_path/" + done +} + +verify_tree() { + local expected="$1" actual="$2" label="$3" + diff -qr "$expected" "$actual" >/dev/null || { + printf 'ERROR: %s skill tree drifted\n' "$label" >&2 + return 1 + } +} + +verify_hermes_skills() { + local listing skill + listing="$(env -u SSH_AUTH_SOCK ssh "$remote_host" \ + 'export PATH=$HOME/.local/bin:$PATH; COLUMNS=240 hermes skills list --source local 2>/dev/null')" + for skill in "${skills[@]}"; do + grep -Fq "$skill" <<<"$listing" || { + printf 'ERROR: Hermes does not list skill: %s\n' "$skill" >&2 + return 1 + } + done +} + +verify_remote_hermes_tree() { + local expected="$1" skill="$2" changes + changes="$(env -u SSH_AUTH_SOCK rsync -rlnic --delete \ + --out-format='%i %n%L' -e ssh \ + "$expected/$skill/" "$remote_host:$remote_root/$skill/")" || return 1 + [[ -z "$changes" ]] || { + printf 'ERROR: Hermes %s content drifted:\n%s\n' "$skill" "$changes" >&2 + return 1 + } +} + +if [[ "${WEBSTER_SYNC_TEST_MODE:-0}" == "1" ]]; then + claude_root="${TEST_SYNC_CLAUDE_ROOT:?TEST_SYNC_CLAUDE_ROOT is required}" + codex_root="${TEST_SYNC_CODEX_ROOT:?TEST_SYNC_CODEX_ROOT is required}" + hermes_root="${TEST_SYNC_HERMES_ROOT:?TEST_SYNC_HERMES_ROOT is required}" + state_file="${TEST_SYNC_STATE_FILE:?TEST_SYNC_STATE_FILE is required}" + expected_hermes="$(mktemp -d)" + trap 'rm -rf "$expected_hermes"' EXIT + for skill in "${skills[@]}"; do + render_hermes "$skill" "$(skill_source "$skill")" "$expected_hermes" + done + if (( check_only )); then + for skill in "${skills[@]}"; do + source="$(skill_source "$skill")" + verify_tree "$source" "$claude_root/$skill" "Claude $skill" + verify_tree "$source" "$codex_root/$skill" "Codex $skill" + verify_tree "$expected_hermes/$skill" "$hermes_root/$skill" "Hermes $skill" + done + printf 'skill mirrors verified\n' + exit 0 + fi + sync_local "$claude_root" + sync_local "$codex_root" + mkdir -p "$hermes_root" + for skill in "${skills[@]}"; do + if [[ -e "$hermes_root/$skill" ]] && + ! diff -qr "$expected_hermes/$skill" "$hermes_root/$skill" >/dev/null; then + if (( ! force )); then + printf 'ERROR: refusing to overwrite divergent Hermes skill directory: %s\n' \ + "$hermes_root/$skill" >&2 + exit 1 + fi + fi + mkdir -p "$hermes_root/$skill" + rsync -a --delete "$expected_hermes/$skill/" "$hermes_root/$skill/" + done + mkdir -p "$(dirname "$state_file")" + sha256sum "$webster_source/SKILL.md" "$migration_source/SKILL.md" >"$state_file" + printf 'skill mirrors synchronized in test mode\n' + exit 0 +fi + +claude_root="$HOME/.claude/skills" +codex_root="$HOME/.codex/skills" +remote_host="ops-bot" +remote_root="/home/ubuntu/.hermes/skills/devops" +drift_check="$HOME/ops-bot/check-skill-drift.sh" +existing_sync="$HOME/ops-bot/sync-skills.sh" +state_dir="$HOME/ops-bot/.skill-push-hashes.d" +migration_state="$state_dir/migrating-webster-models" +expected_hermes="$(mktemp -d)" +trap 'rm -rf "$expected_hermes"' EXIT +for skill in "${skills[@]}"; do + render_hermes "$skill" "$(skill_source "$skill")" "$expected_hermes" +done + +[[ -x "$drift_check" && -x "$existing_sync" ]] || { + printf 'ERROR: existing Webster drift/sync controls are missing\n' >&2 + exit 1 +} +if (( ! force )); then + env -u SSH_AUTH_SOCK "$drift_check" + remote_hash="$(env -u SSH_AUTH_SOCK ssh "$remote_host" \ + "if test -f '$remote_root/migrating-webster-models/SKILL.md'; then awk 'BEGIN{n=0} /^---\$/{n++; if(n<=2) next} n>=2' '$remote_root/migrating-webster-models/SKILL.md' | sha256sum | awk '{print \$1}'; fi" 2>/dev/null || true)" + if [[ -f "$migration_state" && -n "$remote_hash" ]]; then + require_hash="$(<"$migration_state")" + [[ "$remote_hash" == "$require_hash" ]] || { + printf 'ERROR: remote migrating-webster-models changed since the last push\n' >&2 + exit 2 + } + elif [[ -n "$remote_hash" ]]; then + local_hash="$(awk 'BEGIN{n=0} /^---$/{n++; if(n<=2) next} n>=2' \ + "$migration_source/SKILL.md" | sha256sum | awk '{print $1}')" + [[ "$remote_hash" == "$local_hash" ]] || { + printf 'ERROR: untracked remote migrating-webster-models differs from source\n' >&2 + exit 2 + } + fi +fi + +if (( check_only )); then + for skill in "${skills[@]}"; do + source="$(skill_source "$skill")" + verify_tree "$source" "$claude_root/$skill" "Claude $skill" + verify_tree "$source" "$codex_root/$skill" "Codex $skill" + done + for skill in "${skills[@]}"; do + verify_remote_hermes_tree "$expected_hermes" "$skill" + done + verify_hermes_skills + printf 'live skill mirrors verified\n' + exit 0 +fi + +sync_local "$claude_root" +if (( force )); then + env -u SSH_AUTH_SOCK "$existing_sync" --force +else + env -u SSH_AUTH_SOCK "$existing_sync" +fi +sync_local "$codex_root" + +if (( ! force )) && env -u SSH_AUTH_SOCK ssh "$remote_host" \ + "test -d '$remote_root/migrating-webster-models'"; then + verify_remote_hermes_tree "$expected_hermes" migrating-webster-models +fi +env -u SSH_AUTH_SOCK ssh "$remote_host" \ + "mkdir -p '$remote_root/migrating-webster-models'" +env -u SSH_AUTH_SOCK rsync -a --delete -e ssh \ + "$expected_hermes/migrating-webster-models/" \ + "$remote_host:$remote_root/migrating-webster-models/" +for skill in "${skills[@]}"; do + verify_remote_hermes_tree "$expected_hermes" "$skill" +done +verify_hermes_skills +mkdir -p "$state_dir" +env -u SSH_AUTH_SOCK ssh "$remote_host" \ + "awk 'BEGIN{n=0} /^---\$/{n++; if(n<=2) next} n>=2' '$remote_root/migrating-webster-models/SKILL.md' | sha256sum | awk '{print \$1}'" \ + >"$migration_state" +printf 'live Webster runbooks synchronized\n' diff --git a/webster/deepseek-v41/scripts/verify-checkpoint.py b/webster/deepseek-v41/scripts/verify-checkpoint.py new file mode 100755 index 0000000..de219ed --- /dev/null +++ b/webster/deepseek-v41/scripts/verify-checkpoint.py @@ -0,0 +1,173 @@ +#!/usr/bin/env python3 +"""Verify an immutable DeepSeek V4.1 checkpoint and emit its manifest.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import sys +from pathlib import Path, PurePosixPath + + +DEFAULT_SHARDS = 48 +DEFAULT_TENSOR_BYTES = 510_286_023_000 +DEFAULT_SHARD_FILE_BYTES = 510_296_708_312 +REQUIRED_FILES = ( + "config.json", + "tokenizer.json", + "tokenizer_config.json", + "encoding/encoding.py", + "encoding/test_encoding.py", + "encoding/tests/test_input_1.json", + "encoding/tests/test_output_1.txt", + "inference/config.json", +) + + +class VerificationError(RuntimeError): + pass + + +def contained_file(root: Path, relative_name: str) -> Path: + relative = PurePosixPath(relative_name) + if relative.is_absolute() or ".." in relative.parts: + raise VerificationError(f"unsafe checkpoint path: {relative_name}") + candidate = root.joinpath(*relative.parts) + try: + resolved = candidate.resolve(strict=True) + except FileNotFoundError as error: + raise VerificationError(f"missing referenced shard: {relative_name}") from error + try: + resolved.relative_to(root) + except ValueError as error: + raise VerificationError(f"path escapes checkpoint: {relative_name}") from error + if not resolved.is_file(): + raise VerificationError(f"checkpoint path is not a file: {relative_name}") + return resolved + + +def sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for block in iter(lambda: handle.read(8 * 1024 * 1024), b""): + digest.update(block) + return digest.hexdigest() + + +def build_manifest( + checkpoint: Path, + expected_shards: int, + expected_tensor_bytes: int, + expected_shard_file_bytes: int, +) -> list[str]: + root = checkpoint.resolve(strict=True) + if not root.is_dir(): + raise VerificationError(f"checkpoint is not a directory: {checkpoint}") + + index_path = contained_file(root, "model.safetensors.index.json") + try: + index = json.loads(index_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as error: + raise VerificationError(f"invalid safetensors index: {error}") from error + weight_map = index.get("weight_map") + if not isinstance(weight_map, dict) or not weight_map: + raise VerificationError("safetensors index has no weight_map") + + shard_names = sorted(set(weight_map.values())) + if not all(isinstance(name, str) and name for name in shard_names): + raise VerificationError("safetensors index contains an invalid shard name") + if len(shard_names) != expected_shards: + raise VerificationError( + f"shard count mismatch: expected {expected_shards}, got {len(shard_names)}" + ) + + shards = [contained_file(root, name) for name in shard_names] + shard_file_bytes = sum(path.stat().st_size for path in shards) + if shard_file_bytes != expected_shard_file_bytes: + raise VerificationError( + "shard file bytes mismatch: " + f"expected {expected_shard_file_bytes}, got {shard_file_bytes}" + ) + metadata_size = index.get("metadata", {}).get("total_size") + if metadata_size != expected_tensor_bytes: + raise VerificationError( + "index total_size mismatch: " + f"expected {expected_tensor_bytes}, got {metadata_size}" + ) + + present_safetensors = { + path.relative_to(root).as_posix() + for path in root.rglob("*.safetensors") + if path.is_file() + } + referenced_safetensors = { + path.relative_to(root).as_posix() for path in shards + } + extras = sorted(present_safetensors - referenced_safetensors) + if extras: + raise VerificationError(f"unreferenced safetensors shards: {', '.join(extras)}") + + for required in REQUIRED_FILES: + try: + contained_file(root, required) + except VerificationError as error: + raise VerificationError(f"missing required checkpoint file: {required}") from error + + files: list[tuple[str, Path]] = [] + for candidate in root.rglob("*"): + if not candidate.is_file() and not candidate.is_symlink(): + continue + relative = candidate.relative_to(root).as_posix() + resolved = contained_file(root, relative) + files.append((relative, resolved)) + + return sorted( + f"{sha256_file(path)} {path.stat().st_size} {relative}" + for relative, path in files + if relative != "MANIFEST.sha256" + ) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("checkpoint", type=Path) + parser.add_argument("--expected-shards", type=int, default=DEFAULT_SHARDS) + parser.add_argument("--expected-tensor-bytes", type=int, default=DEFAULT_TENSOR_BYTES) + parser.add_argument( + "--expected-shard-file-bytes", type=int, default=DEFAULT_SHARD_FILE_BYTES + ) + parser.add_argument("--compare-manifest", type=Path) + return parser.parse_args() + + +def main() -> int: + args = parse_args() + try: + manifest = build_manifest( + args.checkpoint, + args.expected_shards, + args.expected_tensor_bytes, + args.expected_shard_file_bytes, + ) + rendered = "\n".join(manifest) + "\n" + stored_manifest_path = args.checkpoint / "MANIFEST.sha256" + if stored_manifest_path.exists() or stored_manifest_path.is_symlink(): + stored_manifest = contained_file( + args.checkpoint.resolve(strict=True), "MANIFEST.sha256" + ).read_text(encoding="utf-8") + if stored_manifest != rendered: + raise VerificationError("stored manifest mismatch") + if args.compare_manifest is not None: + comparison = args.compare_manifest.read_text(encoding="utf-8") + if comparison != rendered: + raise VerificationError("manifest mismatch") + sys.stdout.write(rendered) + except (OSError, VerificationError) as error: + print(f"checkpoint verification failed: {error}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/webster/deepseek-v41/scripts/verify-litellm-callbacks.py b/webster/deepseek-v41/scripts/verify-litellm-callbacks.py new file mode 100644 index 0000000..953aa07 --- /dev/null +++ b/webster/deepseek-v41/scripts/verify-litellm-callbacks.py @@ -0,0 +1,99 @@ +#!/usr/bin/env python3 +"""Validate LiteLLM callback ordering and exact-image importability.""" + +from __future__ import annotations + +import argparse +import importlib +import sys +from pathlib import Path +from typing import Any + +import yaml + + +GUARD_CALLBACK = "custom_callbacks.glm52_contract_guard.glm52_contract_guard" +RESPONSES_SANITIZER_CALLBACK = ( + "custom_callbacks.responses_sanitizer.responses_encrypted_content_sanitizer" +) + + +class ValidationError(RuntimeError): + """An intentionally non-secret callback validation failure.""" + + +def load_config(path: Path) -> dict[str, Any]: + if not path.is_file() or path.is_symlink(): + raise ValidationError(f"configuration is not a regular file: {path}") + try: + value = yaml.safe_load(path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, yaml.YAMLError) as error: + raise ValidationError(f"cannot parse configuration: {path}") from error + if not isinstance(value, dict): + raise ValidationError("configuration root must be a mapping") + return value + + +def validate_guard_order(config: dict[str, Any]) -> list[str]: + settings = config.get("litellm_settings") + if not isinstance(settings, dict): + raise ValidationError("litellm_settings must be a mapping") + callbacks = settings.get("callbacks") + if not isinstance(callbacks, list) or any( + not isinstance(callback, str) for callback in callbacks + ): + raise ValidationError("litellm_settings.callbacks must be a list of strings") + if callbacks.count(GUARD_CALLBACK) != 1: + raise ValidationError("guard callback must appear exactly once") + + guard_index = callbacks.index(GUARD_CALLBACK) + prefix = callbacks[:guard_index] + if prefix not in ([], [RESPONSES_SANITIZER_CALLBACK]): + raise ValidationError("only the responses sanitizer may precede the guard") + return callbacks + + +def validate_custom_callbacks(config: dict[str, Any]) -> None: + callbacks = validate_guard_order(config) + working_directory = str(Path.cwd()) + if not sys.path or sys.path[0] != working_directory: + sys.path.insert(0, working_directory) + + try: + from litellm.integrations.custom_logger import CustomLogger + except Exception as error: + raise ValidationError("cannot import LiteLLM CustomLogger") from error + + for callback in callbacks: + if not callback.startswith("custom_callbacks."): + continue + module_name, attribute_name = callback.rsplit(".", 1) + try: + module = importlib.import_module(module_name) + instance = getattr(module, attribute_name) + except Exception as error: + raise ValidationError(f"cannot import custom callback: {callback}") from error + if not isinstance(instance, CustomLogger): + raise ValidationError(f"custom callback is not a CustomLogger: {callback}") + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("config", type=Path) + return parser.parse_args() + + +def main() -> int: + args = parse_args() + try: + config = load_config(args.config) + validate_custom_callbacks(config) + except ValidationError as error: + print(f"ERROR: {error}", file=sys.stderr) + return 1 + print("LiteLLM callback order and imports are valid") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/webster/deepseek-v41/scripts/verify-litellm-config.py b/webster/deepseek-v41/scripts/verify-litellm-config.py new file mode 100755 index 0000000..9060062 --- /dev/null +++ b/webster/deepseek-v41/scripts/verify-litellm-config.py @@ -0,0 +1,308 @@ +#!/usr/bin/env python3 +"""Fail closed unless a LiteLLM candidate contains only the approved alias change.""" + +from __future__ import annotations + +import argparse +import copy +import math +import stat +import sys +from collections import Counter +from pathlib import Path +from typing import Any + +import yaml + + +ALIAS = "glm-5.2" +NATIVE = "glm-5.3-flash" +GUARD_CALLBACK = "custom_callbacks.glm52_contract_guard.glm52_contract_guard" +KV_ROUTER_PLUGINS = ["custom_callbacks.kv_router.plugin"] +DESCRIPTION = ( + "Deprecated GLM-5.2 compatibility alias served by GLM-5.3-Flash; legacy " + "320000-token shared window, reasoning, and function-calling contract retained; " + "no advertised vision capability." +) +PUBLISHED_MODEL = "deepseek-v4.1-flash" +PUBLISHED_API_BASE = "http://100.73.140.127:8000/v1" +PUBLISHED_DESCRIPTION = ( + "DeepSeek V4.1 Flash on the Shamu/Tilikum station pair; " + "qualified private TP2 profile." +) + + +class ValidationError(RuntimeError): + """An intentionally non-secret configuration validation failure.""" + + +def load_config(path: Path) -> dict[str, Any]: + if not path.is_file() or path.is_symlink(): + raise ValidationError(f"configuration is not a regular file: {path}") + try: + value = yaml.safe_load(path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, yaml.YAMLError) as error: + raise ValidationError(f"cannot parse configuration: {path}") from error + if not isinstance(value, dict): + raise ValidationError("configuration root must be a mapping") + return value + + +def model_map(config: dict[str, Any]) -> dict[str, dict[str, Any]]: + models = config.get("model_list") + if not isinstance(models, list): + raise ValidationError("model_list must be a list") + names = [item.get("model_name") if isinstance(item, dict) else None for item in models] + if any(not isinstance(name, str) or not name for name in names): + raise ValidationError("every model deployment must have a nonempty public name") + duplicates = [name for name, count in Counter(names).items() if count != 1] + if duplicates: + raise ValidationError("every existing public model name must have one deployment") + return {item["model_name"]: item for item in models} + + +def expected_alias_config(before: dict[str, Any]) -> dict[str, Any]: + expected = copy.deepcopy(before) + models = model_map(expected) + if ALIAS not in models or NATIVE not in models: + raise ValidationError("configuration must contain one GLM-5.2 alias and one native GLM-5.3") + + alias = models[ALIAS] + native = models[NATIVE] + alias_params = alias.get("litellm_params") + native_params = native.get("litellm_params") + if not isinstance(alias_params, dict) or not isinstance(native_params, dict): + raise ValidationError("GLM deployments must contain litellm_params mappings") + for key in ("model", "api_base", "api_key"): + if key not in native_params: + raise ValidationError(f"native GLM-5.3 deployment is missing {key}") + alias_params[key] = copy.deepcopy(native_params[key]) + + model_info = alias.get("model_info") + if not isinstance(model_info, dict): + raise ValidationError("GLM-5.2 model_info must be a mapping") + model_info.update( + { + "max_input_tokens": 320000, + "max_output_tokens": 320000, + "supports_function_calling": True, + "supports_reasoning": True, + "supports_vision": False, + "input_cost_per_token": 6.0e-08, + "output_cost_per_token": 1.7e-06, + "description": DESCRIPTION, + } + ) + settings = expected.get("litellm_settings") + if not isinstance(settings, dict): + raise ValidationError("litellm_settings must be a mapping") + callbacks = settings.get("callbacks") + if not isinstance(callbacks, list) or any( + not isinstance(callback, str) for callback in callbacks + ): + raise ValidationError("litellm_settings.callbacks must be a list of strings") + settings["callbacks"] = [ + GUARD_CALLBACK, + *(callback for callback in callbacks if callback != GUARD_CALLBACK), + ] + return expected + + +def diff_paths(expected: Any, actual: Any, prefix: str = "root") -> list[str]: + if type(expected) is not type(actual): + return [prefix] + if isinstance(expected, dict): + paths: list[str] = [] + for key in sorted(set(expected) | set(actual), key=str): + child = f"{prefix}.{key}" + if key not in expected or key not in actual: + paths.append(child) + else: + paths.extend(diff_paths(expected[key], actual[key], child)) + return paths + if isinstance(expected, list): + if len(expected) != len(actual): + return [prefix] + paths = [] + for index, (expected_item, actual_item) in enumerate(zip(expected, actual)): + paths.extend(diff_paths(expected_item, actual_item, f"{prefix}[{index}]")) + return paths + return [] if expected == actual else [prefix] + + +def validate_alias(before: dict[str, Any], after: dict[str, Any]) -> None: + before_models = model_map(before) + after_models = model_map(after) + if list(before_models) != list(after_models): + raise ValidationError("public model names or deployment order changed") + + for config in (before, after): + router = config.get("router_settings") + if not isinstance(router, dict): + raise ValidationError("router_settings must be a mapping") + if router.get("plugins") != KV_ROUTER_PLUGINS: + raise ValidationError("KV router plugin contract changed") + if router.get("enable_pre_call_checks") is True: + raise ValidationError("global enable_pre_call_checks is forbidden") + + expected = expected_alias_config(before) + differences = diff_paths(expected, after) + if differences: + preview = ", ".join(differences[:8]) + suffix = "" if len(differences) <= 8 else ", ..." + raise ValidationError(f"candidate contains disallowed semantic changes at {preview}{suffix}") + + +def expected_publish_config( + before: dict[str, Any], + *, + api_key: str, + served_model_name: str, + max_context: int, + input_cost_per_token: float, + output_cost_per_token: float, + supports_vision: bool, +) -> dict[str, Any]: + if not api_key: + raise ValidationError("published backend API key is empty") + if not served_model_name or served_model_name.startswith("hosted_vllm/"): + raise ValidationError("served model name must be the backend model identifier") + if not 1 <= max_context <= 1_048_576: + raise ValidationError("published context must be between 1 and 1048576") + for label, value in ( + ("input cost", input_cost_per_token), + ("output cost", output_cost_per_token), + ): + if not math.isfinite(value) or value <= 0: + raise ValidationError(f"published {label} must be positive and finite") + + expected = copy.deepcopy(before) + models = model_map(expected) + if PUBLISHED_MODEL in models: + raise ValidationError(f"{PUBLISHED_MODEL} already exists") + expected["model_list"].append( + { + "model_name": PUBLISHED_MODEL, + "litellm_params": { + "model": "hosted_vllm/" + served_model_name, + "api_base": PUBLISHED_API_BASE, + "api_key": api_key, + }, + "model_info": { + "mode": "chat", + "max_input_tokens": max_context, + "max_output_tokens": max_context, + "supports_function_calling": True, + "supports_reasoning": True, + "supports_response_schema": True, + **({"supports_vision": True} if supports_vision else {}), + "input_cost_per_token": input_cost_per_token, + "output_cost_per_token": output_cost_per_token, + "description": PUBLISHED_DESCRIPTION, + }, + } + ) + return expected + + +def validate_publish(before: dict[str, Any], after: dict[str, Any]) -> None: + before_models = model_map(before) + after_models = model_map(after) + if PUBLISHED_MODEL in before_models: + raise ValidationError(f"{PUBLISHED_MODEL} already exists before publication") + if list(after_models) != [*before_models, PUBLISHED_MODEL]: + raise ValidationError("publish phase must append exactly one approved public model") + for name, deployment in before_models.items(): + if after_models[name] != deployment: + raise ValidationError(f"existing model changed during publication: {name}") + for top_level_key in set(before) | set(after): + if top_level_key == "model_list": + continue + if before.get(top_level_key) != after.get(top_level_key): + raise ValidationError( + f"non-model configuration changed during publication: {top_level_key}" + ) + + published = after_models[PUBLISHED_MODEL] + params = published.get("litellm_params") + info = published.get("model_info") + if not isinstance(params, dict) or set(params) != {"model", "api_base", "api_key"}: + raise ValidationError("published litellm_params shape is invalid") + if params.get("api_base") != PUBLISHED_API_BASE: + raise ValidationError("published model must use the Shamu NetBird canary") + if not isinstance(params.get("model"), str) or not params["model"].startswith( + "hosted_vllm/" + ): + raise ValidationError("published backend model identifier is invalid") + if not isinstance(params.get("api_key"), str) or not params["api_key"]: + raise ValidationError("published backend key is absent") + required_info = { + "mode", + "max_input_tokens", + "max_output_tokens", + "supports_function_calling", + "supports_reasoning", + "supports_response_schema", + "input_cost_per_token", + "output_cost_per_token", + "description", + } + if not isinstance(info, dict) or set(info) not in ( + required_info, + required_info | {"supports_vision"}, + ): + raise ValidationError("published model_info shape is invalid") + if info.get("mode") != "chat" or info.get("description") != PUBLISHED_DESCRIPTION: + raise ValidationError("published mode or description is invalid") + if info.get("max_input_tokens") != info.get("max_output_tokens") or not isinstance( + info.get("max_input_tokens"), int + ): + raise ValidationError("published context fields must match") + if not 1 <= info["max_input_tokens"] <= 1_048_576: + raise ValidationError("published context is outside the qualified range") + for capability in ( + "supports_function_calling", + "supports_reasoning", + "supports_response_schema", + ): + if info.get(capability) is not True: + raise ValidationError(f"published capability is invalid: {capability}") + if "supports_vision" in info and info["supports_vision"] is not True: + raise ValidationError("supports_vision may only be advertised when true") + for cost_key in ("input_cost_per_token", "output_cost_per_token"): + value = info.get(cost_key) + if not isinstance(value, (int, float)) or isinstance(value, bool): + raise ValidationError(f"published {cost_key} is invalid") + if not math.isfinite(float(value)) or float(value) <= 0: + raise ValidationError(f"published {cost_key} must be positive and finite") + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("before", type=Path) + parser.add_argument("after", type=Path) + parser.add_argument("--phase", required=True, choices=("alias", "publish")) + return parser.parse_args() + + +def main() -> int: + args = parse_args() + try: + before = load_config(args.before) + after = load_config(args.after) + mode = stat.S_IMODE(args.after.stat().st_mode) + if mode != 0o600: + raise ValidationError("candidate configuration mode must be 0600") + if args.phase == "alias": + validate_alias(before, after) + else: + validate_publish(before, after) + except ValidationError as error: + print(f"ERROR: {error}", file=sys.stderr) + return 1 + print(f"allowed semantic diff validated for {args.phase} phase") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/webster/deepseek-v41/scripts/verify-litellm-container.py b/webster/deepseek-v41/scripts/verify-litellm-container.py new file mode 100755 index 0000000..5285193 --- /dev/null +++ b/webster/deepseek-v41/scripts/verify-litellm-container.py @@ -0,0 +1,119 @@ +#!/usr/bin/env python3 +"""Validate LiteLLM's immutable image and non-duplicated entrypoint command shape.""" + +from __future__ import annotations + +import argparse +import json +import re +import sys +from pathlib import Path +from typing import Any + + +EXPECTED_ENTRYPOINT = "docker/prod_entrypoint.sh" +EXPECTED_COMMAND = [ + "--config", + "/app/config.yaml", + "--host", + "127.0.0.1", + "--port", + "4446", +] +IMAGE_PATTERN = re.compile(r"^sha256:[0-9a-f]{64}$") + + +class ValidationError(RuntimeError): + """A non-secret inspect validation failure.""" + + +def load_inspect(path: Path) -> dict[str, Any]: + if not path.is_file() or path.is_symlink(): + raise ValidationError(f"inspect input is not a regular file: {path}") + try: + payload = json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, json.JSONDecodeError) as error: + raise ValidationError(f"cannot parse inspect JSON: {path}") from error + if not isinstance(payload, list) or len(payload) != 1 or not isinstance(payload[0], dict): + raise ValidationError("inspect JSON must contain exactly one container object") + return payload[0] + + +def require_mapping(value: Any, name: str) -> dict[str, Any]: + if not isinstance(value, dict): + raise ValidationError(f"{name} must be an object") + return value + + +def validate_shape(inspect: dict[str, Any], expected_image: str | None = None) -> None: + image = inspect.get("Image") + if not isinstance(image, str) or not IMAGE_PATTERN.fullmatch(image): + raise ValidationError("container Image must be an immutable sha256 ID") + if expected_image is not None and image != expected_image: + raise ValidationError("container image ID does not match the approved image") + + config = require_mapping(inspect.get("Config"), "Config") + entrypoint = config.get("Entrypoint") + if not isinstance(entrypoint, list) or any( + not isinstance(token, str) for token in entrypoint + ): + raise ValidationError("Config.Entrypoint must be a string list") + if entrypoint.count(EXPECTED_ENTRYPOINT) != 1: + raise ValidationError("Config.Entrypoint must contain prod_entrypoint exactly once") + command = config.get("Cmd") + if command != EXPECTED_COMMAND: + raise ValidationError("Config.Cmd does not match the loopback-only LiteLLM contract") + if any("prod_entrypoint.sh" in token for token in command): + raise ValidationError("Config.Cmd must not repeat the image entrypoint") + + +def recreation_shape(inspect: dict[str, Any]) -> dict[str, Any]: + config = require_mapping(inspect.get("Config"), "Config") + host = require_mapping(inspect.get("HostConfig"), "HostConfig") + return { + "Image": inspect.get("Image"), + "Entrypoint": config.get("Entrypoint"), + "Cmd": config.get("Cmd"), + "Mounts": inspect.get("Mounts"), + "NetworkMode": host.get("NetworkMode"), + "RestartPolicy": host.get("RestartPolicy"), + "PortBindings": host.get("PortBindings"), + "User": config.get("User"), + "Capabilities": { + "CapAdd": host.get("CapAdd"), + "CapDrop": host.get("CapDrop"), + "Privileged": host.get("Privileged"), + "ReadonlyRootfs": host.get("ReadonlyRootfs"), + "SecurityOpt": host.get("SecurityOpt"), + }, + "Ulimits": host.get("Ulimits"), + } + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("inspect_json", type=Path) + parser.add_argument("--baseline", type=Path) + parser.add_argument("--expected-image") + return parser.parse_args() + + +def main() -> int: + args = parse_args() + try: + candidate = load_inspect(args.inspect_json) + validate_shape(candidate, args.expected_image) + if args.baseline is not None: + baseline = load_inspect(args.baseline) + validate_shape(baseline, args.expected_image) + if recreation_shape(candidate) != recreation_shape(baseline): + raise ValidationError("recreation shape differs from captured baseline") + except ValidationError as error: + print(f"ERROR: {error}", file=sys.stderr) + return 1 + print("container shape valid") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/webster/deepseek-v41/scripts/verify-private-acceptance.py b/webster/deepseek-v41/scripts/verify-private-acceptance.py new file mode 100755 index 0000000..0ae2e66 --- /dev/null +++ b/webster/deepseek-v41/scripts/verify-private-acceptance.py @@ -0,0 +1,1207 @@ +#!/usr/bin/env python3 +"""Validate private qualification evidence before publication.""" + +from __future__ import annotations + +import argparse +from datetime import datetime, timezone +import hashlib +import json +import math +import os +import re +import stat +import subprocess +import sys +import tempfile +from pathlib import Path +from typing import Any + + +MODEL = "deepseek-ai/DeepSeek-V4.1-Flash" +PUBLIC_MODEL = "deepseek-v4.1-flash" +PROFILE_NAME = "dspark-1m-noautotune-seqs4" +CHECKPOINT_REVISION = "dba1be0a40aa45a94ad051997016db3960a90277" +CHECKPOINT_MANIFEST_SHA256 = ( + "aad652a2b601f711599298493229f32fcc7216ca17d6bb0ccd451fd2fdd01fef" +) +VLLM_COMMIT = "e77daef89e18e08321ae7b8b24827eedd5fe8673" +VLLM_IMAGE_ID = ( + "sha256:3863bf0f59bd4df4012b7b7aed8a7d2be6ef9b42ee3a5299d014f83a1f1ba6ea" +) +AIPERF_COMMIT = "ea28b2e81c7367f8403c6f6ebe0837a508795a8d" +AIPERF_WEKA_PATCH_SHA256 = ( + "9557a46afac031b64f66ea64ebac4e8ed4d1a0e142354388c19c086a9377a36e" +) +WEKA_REPOSITORY = "semianalysisai/cc-traces-weka-062126" +WEKA_REVISION = "23f152f6f0f9399a85901b89a6458def0ef16729" +WEKA_TRACE_COUNT = 393 +CLOCK_SKEW_SECONDS = 5 +STATION_ENDPOINT = "http://100.73.140.127:8000/v1" +INPUT_COST_PER_TOKEN = 1.3e-8 +OUTPUT_COST_PER_TOKEN = 9.6e-7 +ELECTRICITY_RATE_USD_PER_KWH = 0.47 +EXPECTED_PROFILE = { + "PROFILE_NAME": PROFILE_NAME, + "MAX_MODEL_LEN": "1048576", + "MAX_NUM_SEQS": "4", + "GPU_MEMORY_UTILIZATION": "0.90", + "ENFORCE_EAGER": "0", + "ENABLE_PREFIX_CACHING": "1", + "MAX_NUM_BATCHED_TOKENS": "16384", + "ENABLE_EXPERT_PARALLEL": "0", + "SPECULATIVE_METHOD": "dspark", + "NUM_SPECULATIVE_TOKENS": "5", + "DRAFT_SAMPLE_METHOD": "greedy", + "REJECTION_SAMPLE_METHOD": "standard", + "ENABLE_FLASHINFER_AUTOTUNE": "0", +} +EXPECTED_COMMON_SERVING_ARGS = [ + "serve", + "/model", + "--served-model-name", + MODEL, + "--nnodes", + "2", + "--master-addr", + "10.10.1.1", + "--master-port", + "29511", + "--distributed-executor-backend", + "mp", + "--tensor-parallel-size", + "2", + "--pipeline-parallel-size", + "1", + "--engram-config", + '{"cpu_offload":true,"embedding_across_dp":false}', + "--max-model-len", + "1048576", + "--max-num-seqs", + "4", + "--gpu-memory-utilization", + "0.90", + "--disable-custom-all-reduce", + "--enable-auto-tool-choice", + "--tool-call-parser", + "deepseek_v41", + "--reasoning-parser", + "deepseek_v41", + "--enable-prefix-caching", + "--max-num-batched-tokens", + "16384", + "--no-enable-flashinfer-autotune", + "--speculative-config", + ( + '{"method":"dspark","num_speculative_tokens":5,' + '"draft_sample_method":"greedy",' + '"rejection_sample_method":"standard"}' + ), +] +FEATURE_CASES = { + "deterministic-text", + "utf8", + "stop-sequence", + "streaming", + "usage-accounting", + "reasoning-on", + "tool-auto", + "tool-named", + "tool-required", + "tool-parallel", + "tool-result", + "structured", + "json-object", + "vision", + "malformed-input", + "wrong-model", + "disconnect-queue-drain", +} +CONTEXT_TARGETS = [320000, 600000, 990016, 996579] +SHA256 = re.compile(r"^[0-9a-f]{64}$") +IMAGE_ID = re.compile(r"^sha256:[0-9a-f]{64}$") + + +class ValidationError(RuntimeError): + pass + + +def require(condition: bool, message: str) -> None: + if not condition: + raise ValidationError(message) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--run-root", required=True, type=Path) + parser.add_argument("--acceptance", required=True, type=Path) + parser.add_argument("--max-age-seconds", required=True, type=float) + parser.add_argument("--now") + return parser.parse_args() + + +def timestamp(value: Any, label: str) -> datetime: + require(isinstance(value, str), f"{label} timestamp must be a string") + try: + return datetime.strptime(value, "%Y-%m-%dT%H:%M:%SZ").replace( + tzinfo=timezone.utc + ) + except ValueError as error: + raise ValidationError(f"{label} timestamp is malformed") from error + + +def load_json(path: Path, label: str) -> Any: + try: + return json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeDecodeError, json.JSONDecodeError) as error: + raise ValidationError(f"{label} is malformed: {error}") from error + + +def private_file(path: Path, run_root: Path, label: str) -> Path: + require(path.is_absolute(), f"{label} path must be absolute") + require( + path.is_file() and not path.is_symlink(), + f"{label} must be a regular non-symlink file", + ) + resolved = path.resolve(strict=True) + require(resolved.is_relative_to(run_root), f"{label} path escapes the run root") + metadata = resolved.stat() + require(stat.S_IMODE(metadata.st_mode) == 0o600, f"{label} must have mode 0600") + require( + metadata.st_uid == run_root.stat().st_uid, + f"{label} owner does not match the run root", + ) + return resolved + + +def file_hash(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def weka_record_failed(record: dict[str, Any]) -> bool: + if record.get("error"): + return True + status = record.get("status_code", record.get("status")) + if status is not None: + return ( + not isinstance(status, int) + or isinstance(status, bool) + or not 200 <= status < 300 + ) + return not isinstance(record.get("metadata"), dict) or not isinstance( + record.get("metrics"), dict + ) + + +def require_fresh( + path: Path, + label: str, + accepted: datetime, + now: datetime, + max_age_seconds: float, +) -> None: + captured = path.stat().st_mtime + require( + captured >= now.timestamp() - max_age_seconds, + f"{label} evidence is stale", + ) + require( + captured + <= min(accepted.timestamp(), now.timestamp()) + CLOCK_SKEW_SECONDS, + f"{label} evidence is future-dated beyond the " + f"{CLOCK_SKEW_SECONDS}-second clock-skew allowance", + ) + + +def expected_checkpoint_manifest_hash() -> str: + if os.environ.get("WEBSTER_ACCEPTANCE_TEST_MODE") == "1": + value = os.environ.get("TEST_CHECKPOINT_MANIFEST_SHA256", "") + require( + SHA256.fullmatch(value) is not None, + "test checkpoint manifest hash is invalid", + ) + return value + return CHECKPOINT_MANIFEST_SHA256 + + +def child( + root: Any, key: str, label: str = "acceptance" +) -> dict[str, Any]: + require(isinstance(root, dict), f"{label} must be an object") + value = root.get(key) + require(isinstance(value, dict), f"{label}.{key} must be an object") + return value + + +def reference( + value: dict[str, Any], + run_root: Path, + label: str, + expected_path: Path, + *, + parse_json: bool = True, +) -> tuple[Path, Any]: + raw_path = value.get("path") + require(isinstance(raw_path, str), f"{label} path must be a string") + path = private_file(Path(raw_path), run_root, label) + require(path == expected_path, f"{label} path does not match its artifact") + expected_hash = value.get("sha256") + require( + isinstance(expected_hash, str) and SHA256.fullmatch(expected_hash) is not None, + f"{label} hash is invalid", + ) + require(file_hash(path) == expected_hash, f"{label} hash does not match content") + return path, load_json(path, label) if parse_json else None + + +def env_file(path: Path, label: str) -> dict[str, str]: + values: dict[str, str] = {} + for number, raw in enumerate(path.read_text(encoding="utf-8").splitlines(), 1): + line = raw.strip() + if not line or line.startswith("#"): + continue + require("=" in line, f"{label} line {number} is malformed") + key, value = line.split("=", 1) + require(key and key not in values, f"{label} has a duplicate field") + values[key] = value + return values + + +def validate_profile(root: dict[str, Any], run_root: Path) -> None: + selected = child(root, "selected_profile") + require( + selected.get("name") == PROFILE_NAME + and selected.get("max_model_len") == 1048576 + and selected.get("max_num_seqs") == 4 + and selected.get("speculative_method") == "dspark" + and selected.get("flashinfer_autotune") is False, + "selected profile does not match the qualified production profile", + ) + expected = run_root / "profiles" / f"{PROFILE_NAME}.env" + path, _ = reference( + selected, run_root, "selected profile", expected, parse_json=False + ) + require( + env_file(path, "selected profile") == EXPECTED_PROFILE, + "selected profile file content is wrong", + ) + + +def validate_runtime(root: dict[str, Any], run_root: Path) -> str: + runtime = child(root, "runtime") + require( + runtime.get("checkpoint_revision") == CHECKPOINT_REVISION, + "runtime checkpoint revision is wrong", + ) + image = runtime.get("image_id") + require( + isinstance(image, str) and IMAGE_ID.fullmatch(image) is not None, + "runtime image is invalid", + ) + require(image == VLLM_IMAGE_ID, "runtime image is wrong") + require(runtime.get("vllm_commit") == VLLM_COMMIT, "runtime vLLM commit is wrong") + raw_manifest = runtime.get("manifest_path") + require(isinstance(raw_manifest, str), "runtime manifest path must be a string") + manifest = private_file(Path(raw_manifest), run_root, "runtime manifest") + require(manifest == run_root / "manifest.env", "runtime manifest path is wrong") + manifest_hash = runtime.get("manifest_sha256") + require( + isinstance(manifest_hash, str) and SHA256.fullmatch(manifest_hash) is not None, + "runtime manifest hash is invalid", + ) + require( + file_hash(manifest) == manifest_hash, + "runtime manifest hash does not match content", + ) + values = env_file(manifest, "runtime manifest") + require(values.get("CHECKPOINT_REPO") == MODEL, "runtime manifest model is wrong") + require( + values.get("CHECKPOINT_REVISION") == CHECKPOINT_REVISION, + "runtime manifest checkpoint is wrong", + ) + require(values.get("VLLM_IMAGE_ID") == image, "runtime image does not match manifest") + require( + values.get("VLLM_COMMIT") == runtime.get("vllm_commit"), + "runtime vLLM commit does not match manifest", + ) + checkpoint_hash = runtime.get("checkpoint_manifest_sha256") + require( + isinstance(checkpoint_hash, str) + and SHA256.fullmatch(checkpoint_hash) is not None, + "checkpoint manifest hash is invalid", + ) + require( + checkpoint_hash == expected_checkpoint_manifest_hash(), + "checkpoint manifest hash is wrong", + ) + for rank in ("shamu", "tilikum"): + path = private_file( + run_root / "baseline" / f"{rank}-checkpoint.manifest", + run_root, + f"{rank} checkpoint manifest", + ) + require( + file_hash(path) == checkpoint_hash, + f"{rank} checkpoint manifest hash is wrong", + ) + return image + + +def validate_feature(root: dict[str, Any], run_root: Path) -> None: + item = child(root, "feature_qualification") + _, payload = reference( + item, + run_root, + "capability qualification", + run_root / "feature-dspark-1m-seqs4-final.json", + ) + require( + isinstance(payload, dict) and payload.get("schema_version") == 1, + "capability qualification schema is wrong", + ) + require( + payload.get("model") == MODEL and payload.get("stage") == "dspark", + "capability qualification target is wrong", + ) + require( + payload.get("ok") is True and item.get("ok") is True, + "capability qualification is not successful", + ) + cases = payload.get("cases") + require(isinstance(cases, list) and cases, "capability cases are missing") + names = [case.get("case") for case in cases if isinstance(case, dict)] + require( + len(names) == len(cases) and len(set(names)) == len(names), + "capability cases are malformed", + ) + require(FEATURE_CASES.issubset(set(names)), "capability is missing required cases") + require( + all(case.get("semantic_ok") is True for case in cases), + "capability contains a failed case", + ) + by_name = {case["case"]: case for case in cases} + require( + all( + by_name[name].get("status") + == {"malformed-input": 400, "wrong-model": 404}.get(name, 200) + for name in FEATURE_CASES + ), + "capability contains an unexpected response status", + ) + deterministic = by_name["deterministic-text"] + require( + deterministic.get("repetitions", 0) >= 3 + and deterministic.get("distinct_normalized_outputs") == 1, + "capability deterministic repetitions are incomplete", + ) + disconnect = by_name["disconnect-queue-drain"] + baseline = disconnect.get("queue_baseline") + final = disconnect.get("queue_final") + require( + disconnect.get("client_disconnected") is True + and disconnect.get("queue_drained") is True + and isinstance(baseline, dict) + and isinstance(final, dict) + and all( + isinstance(baseline.get(name), (int, float)) + and isinstance(final.get(name), (int, float)) + and final[name] <= baseline[name] + for name in ("running", "waiting") + ), + "capability disconnect queue drain is incomplete", + ) + failure_capture = payload.get("failure_log_capture") + require( + isinstance(failure_capture, dict) + and failure_capture.get("requested") is True + and failure_capture.get("captured") is False + and failure_capture.get("ok") is True + and failure_capture.get("nodes") == ["shamu", "tilikum"] + and failure_capture.get("container") == "deepseek-v41-flash-tp2", + "capability failure-log capture was not armed", + ) + require( + item.get("passed") == len(cases) == item.get("total"), + "capability passed/total is wrong", + ) + + +def validate_context(root: dict[str, Any], run_root: Path) -> dict[str, Any]: + item = child(root, "long_context_qualification") + _, payload = reference( + item, + run_root, + "context qualification", + run_root / "long-context-dspark-1m-seqs4-final.json", + ) + require( + isinstance(payload, dict) and payload.get("schema_version") == 1, + "context qualification schema is wrong", + ) + require( + payload.get("model") == MODEL and payload.get("max_model_len") == 1048576, + "context qualification target is wrong", + ) + require( + payload.get("ok") is True and item.get("ok") is True, + "context qualification is not successful", + ) + require( + item.get("accepted_context_totals") == CONTEXT_TARGETS, + "context acceptance totals are wrong", + ) + require( + item.get("rejected_context_total") == 1048577, + "context rejection boundary is wrong", + ) + accepted = payload.get("accepted") + require(isinstance(accepted, list), "context accepted cases are missing") + require( + [row.get("target_context_tokens") for row in accepted] == CONTEXT_TARGETS, + "context accepted cases target the wrong totals", + ) + for row, target in zip(accepted, CONTEXT_TARGETS): + require( + row.get("status") == 200 + and row.get("semantic_ok") is True + and row.get("token_count_exact") is True + and row.get("server_total_tokens") == target, + "context accepted case is not qualified", + ) + over = payload.get("over_limit") + require( + isinstance(over, dict) + and over.get("target_context_tokens") == 1048577 + and over.get("status") == 400 + and over.get("semantic_ok") is True + and over.get("token_count_exact") is True, + "context over-limit rejection is not qualified", + ) + calibration = payload.get("tokenizer_calibration") + require( + isinstance(calibration, dict) + and calibration.get("reported_max_model_len") == 1048576 + and calibration.get("exact_linear_unit") is True, + "context tokenizer calibration is wrong", + ) + return payload + + +def validate_weka( + root: dict[str, Any], run_root: Path, serving_generation: str +) -> dict[str, Any]: + item = child(root, "weka_qualification") + _, payload = reference( + item, + run_root, + "Weka qualification", + run_root / "aiperf" / "scored-summaries" / "seqs4.json", + ) + require( + isinstance(payload, dict) + and payload.get("schema_version") == 1 + and payload.get("qualified") is True + and item.get("qualified") is True, + "Weka qualification is not successful", + ) + mirrored = ( + "requests", + "errors", + "server_success_rate", + "request_throughput", + "output_token_throughput", + "ttft_p90_ms", + "request_peak_concurrency", + "preemptions_total", + ) + require( + all(item.get(name) == payload.get(name) for name in mirrored), + "Weka qualification summary does not match its artifact", + ) + required = ( + "kv_cache_usage_max", + "prefix_cache_hits", + "prefix_cache_queries", + "prefix_cache_hit_ratio", + "running_requests_max", + "waiting_requests_max", + "wall_power_avg_watts", + "facility_power_avg_watts", + "facility_energy_kwh", + "facility_cost_dollars", + "completed_root_workflows", + "cancelled_root_workflows", + "root_workflow_elapsed_seconds", + "completed_root_workflows_per_second", + ) + require( + all(payload.get(name) is not None for name in required), + "Weka qualification is missing required metrics", + ) + require( + payload.get("requests", 0) > 0 + and payload.get("errors") == 0 + and payload.get("server_success_rate", 0) >= 0.99 + and payload.get("preemptions_total") == 0 + and payload.get("cancelled_root_workflows") == 0 + and payload.get("completed_root_workflows", 0) > 0, + "Weka qualification contains disqualifying results", + ) + repetitions = item.get("repetitions") + require( + isinstance(repetitions, list) and len(repetitions) == 2, + "Weka qualification requires both repetitions", + ) + selected_rederived: dict[str, Any] | None = None + warmup_provenance_hash: str | None = None + warmup_rank_states: list[dict[str, Any]] | None = None + for expected_repetition, repetition in enumerate(repetitions, 1): + require(isinstance(repetition, dict), "Weka repetition reference is malformed") + raw_path = repetition.get("path") + require(isinstance(raw_path, str), "Weka repetition path must be a string") + expected = ( + run_root + / "aiperf" + / f"{PROFILE_NAME}-c4-r{expected_repetition}-open" + / "provenance.json" + ) + provenance_path, provenance = reference( + repetition, + run_root, + f"Weka repetition {expected_repetition}", + expected, + ) + require( + isinstance(provenance, dict) and provenance.get("schema_version") == 1, + f"Weka repetition {expected_repetition} provenance is malformed", + ) + require( + provenance.get("profile_name") == PROFILE_NAME + and provenance.get("concurrency") == 4 + and provenance.get("repetition") == expected_repetition, + f"Weka repetition {expected_repetition} selected point is wrong", + ) + require( + provenance.get("aiperf_commit") == AIPERF_COMMIT + and provenance.get("weka_repository") == WEKA_REPOSITORY + and provenance.get("weka_revision") == WEKA_REVISION, + f"Weka repetition {expected_repetition} client or dataset pin is wrong", + ) + require( + provenance.get("aiperf_weka_patch_sha256") + == AIPERF_WEKA_PATCH_SHA256, + f"Weka repetition {expected_repetition} AIPerf runtime patch is wrong", + ) + require( + provenance.get("trace_count") == WEKA_TRACE_COUNT + and provenance.get("full_subagents") is True, + f"Weka repetition {expected_repetition} does not prove the exact " + "393-trace full-subagent corpus", + ) + if expected_repetition == 1: + require( + provenance.get("cache_state") == "warmup" + and provenance.get("warmup_and_scored_comparable") is False + and provenance.get("warmup_provenance_sha256") is None, + "Weka repetition 1 is not identified as the warmup", + ) + warmup_provenance_hash = file_hash(provenance_path) + else: + require( + provenance.get("cache_state") == "warm" + and provenance.get("warmup_and_scored_comparable") is True + and provenance.get("warmup_provenance_sha256") + == warmup_provenance_hash, + "Weka scored repetition is not bound to the warmup provenance", + ) + require( + provenance.get("serving_generation") == serving_generation, + f"Weka repetition {expected_repetition} serving generation does not match", + ) + require( + provenance.get("engine_deaths") == 0 + and provenance.get("nccl_timeouts") == 0 + and provenance.get("corruption_flags") == 0 + and provenance.get("rank_fatal_error_matches") == 0, + f"Weka repetition {expected_repetition} contains engine, NCCL, or corruption failures", + ) + artifacts = provenance.get("artifacts") + require(isinstance(artifacts, dict), "Weka raw artifact manifest is missing") + required_artifacts = ( + "EXIT", + ".aiperf_results_ready.json", + "profile_export.jsonl", + "profile_export_aiperf.json", + "server_metrics_export.json", + "aiperf.stdout.log", + "aiperf.stderr.log", + "rendered-command.txt", + "summary.json", + "benchmark-window.json", + "rank0-before.json", + "rank1-before.json", + "rank0-after.json", + "rank1-after.json", + "rank0-container.log", + "rank1-container.log", + ) + directory = provenance_path.parent + for name in required_artifacts: + kind = "rank log" if name.endswith("-container.log") else "raw artifact" + artifact = private_file(directory / name, run_root, f"Weka {kind} {name}") + digest = artifacts.get(name) + require( + isinstance(digest, str) + and SHA256.fullmatch(digest) is not None + and file_hash(artifact) == digest, + f"Weka raw artifact {name} is missing or has the wrong identity", + ) + require( + (directory / "EXIT").read_text(encoding="utf-8") == "EXIT=0\n", + f"Weka repetition {expected_repetition} does not have exact EXIT=0 readiness", + ) + readiness = load_json( + directory / ".aiperf_results_ready.json", "Weka readiness" + ) + require( + isinstance(readiness, dict) + and readiness.get("ready") is True + and readiness.get("was_cancelled") is False + and readiness.get("partial") is False + and readiness.get("failed_exporters", []) == [], + f"Weka repetition {expected_repetition} readiness is incomplete", + ) + raw_lines = [ + line + for line in (directory / "profile_export.jsonl") + .read_text(encoding="utf-8") + .splitlines() + if line.strip() + ] + require(raw_lines, f"Weka repetition {expected_repetition} raw records are empty") + try: + raw_records = [json.loads(line) for line in raw_lines] + except json.JSONDecodeError as error: + raise ValidationError("Weka raw records are malformed") from error + require( + all(isinstance(row, dict) for row in raw_records), + "Weka raw records must be objects", + ) + require( + not any(weka_record_failed(row) for row in raw_records), + f"Weka repetition {expected_repetition} raw records contain failed requests", + ) + require( + '"corruption":true' + not in (directory / "profile_export.jsonl") + .read_text(encoding="utf-8") + .lower(), + f"Weka repetition {expected_repetition} raw records contain corruption flags", + ) + fatal_pattern = re.compile( + r"EngineDeadError|NCCL.*(?:timeout|error)|RPC call.*timed out", + re.IGNORECASE, + ) + require( + all( + fatal_pattern.search( + (directory / name).read_text( + encoding="utf-8", errors="replace" + ) + ) + is None + for name in ("rank0-container.log", "rank1-container.log") + ), + f"Weka repetition {expected_repetition} rank logs contain fatal errors", + ) + rank_states: list[dict[str, Any]] = [] + for rank in (0, 1): + before = load_json( + directory / f"rank{rank}-before.json", + f"Weka rank {rank} pre-run state", + ) + after = load_json( + directory / f"rank{rank}-after.json", + f"Weka rank {rank} post-run state", + ) + require( + isinstance(before, dict) and before == after, + f"Weka repetition {expected_repetition} rank {rank} changed during the run", + ) + require( + before.get("image") == VLLM_IMAGE_ID + and before.get("restart_count") == 0 + and isinstance(before.get("container_id"), str) + and SHA256.fullmatch(before["container_id"]) is not None, + f"Weka repetition {expected_repetition} rank {rank} state is not generation-bound", + ) + rank_states.append(after) + generation_result = subprocess.run( + [ + sys.executable, + str(Path(__file__).with_name("verify-serving-pair.py")), + "--rank0-state-json", + json.dumps(rank_states[0], separators=(",", ":")), + "--rank1-state-json", + json.dumps(rank_states[1], separators=(",", ":")), + "--max-start-skew-seconds", + "120", + "--require-matching-image", + "--print-generation", + ], + text=True, + capture_output=True, + check=False, + ) + require( + generation_result.returncode == 0 + and generation_result.stdout.strip() == serving_generation, + f"Weka repetition {expected_repetition} rank state is not generation-bound", + ) + if expected_repetition == 1: + warmup_rank_states = rank_states + else: + require( + warmup_rank_states == rank_states, + "Weka scored repetition did not continue on the warmup serving ranks", + ) + window = load_json( + directory / "benchmark-window.json", "Weka benchmark window" + ) + require( + isinstance(window, dict) + and timestamp(window.get("started_at"), "Weka benchmark start") + <= timestamp(window.get("ended_at"), "Weka benchmark end"), + f"Weka repetition {expected_repetition} benchmark window is invalid", + ) + with tempfile.TemporaryDirectory(prefix="webster-weka-verify-") as temporary: + derived_path = Path(temporary) / "summary.json" + result = subprocess.run( + [ + sys.executable, + str(Path(__file__).with_name("summarize-aiperf.py")), + "--profile", + str(directory / "profile_export_aiperf.json"), + "--records", + str(directory / "profile_export.jsonl"), + "--server-metrics", + str(directory / "server_metrics_export.json"), + "--timing-log", + str(directory / "aiperf.stdout.log"), + "--output-json", + str(derived_path), + ], + text=True, + capture_output=True, + check=False, + ) + require( + result.returncode == 0, + f"Weka repetition {expected_repetition} raw artifacts are not qualified: " + f"{result.stderr.strip()}", + ) + derived = load_json( + derived_path, f"Weka repetition {expected_repetition} rederived summary" + ) + stored_summary = load_json( + directory / "summary.json", + f"Weka repetition {expected_repetition} stored summary", + ) + require( + stored_summary == derived, + f"Weka repetition {expected_repetition} stored summary differs from raw artifacts", + ) + if expected_repetition == 2: + selected_rederived = derived + require(selected_rederived is not None, "Weka repetition 2 was not rederived") + selected_fields = mirrored + required + ( + "completed", + "qualified", + ) + require( + all(payload.get(name) == selected_rederived.get(name) for name in selected_fields), + "Weka selected score differs from rederived repetition 2", + ) + return payload + + +def validate_pricing( + root: dict[str, Any], + run_root: Path, + serving_generation: str, + context: dict[str, Any], + weka: dict[str, Any], +) -> tuple[float, float]: + item = child(root, "pricing_evidence") + expected_path = run_root / "pricing-evidence.json" + require(expected_path.is_file(), "pricing evidence is absent") + _, pricing = reference( + item, + run_root, + "pricing evidence", + expected_path, + ) + require(isinstance(pricing, dict), "pricing evidence must be an object") + require( + set(pricing) + == { + "schema_version", + "model", + "serving_generation", + "electricity_rate_usd_per_kwh", + "facility_power_watts", + "input", + "output", + }, + "pricing evidence schema is wrong", + ) + require( + pricing.get("schema_version") == 1 and pricing.get("model") == MODEL, + "pricing evidence identity is wrong", + ) + require( + pricing.get("serving_generation") == serving_generation, + "pricing evidence references a different serving generation", + ) + rate = pricing.get("electricity_rate_usd_per_kwh") + power = pricing.get("facility_power_watts") + require( + isinstance(rate, (int, float)) + and not isinstance(rate, bool) + and rate == ELECTRICITY_RATE_USD_PER_KWH + and isinstance(power, (int, float)) + and not isinstance(power, bool) + and power == weka.get("facility_power_avg_watts"), + "pricing evidence power or electricity rate is wrong", + ) + input_basis = pricing.get("input") + output_basis = pricing.get("output") + require( + isinstance(input_basis, dict) + and set(input_basis) + == { + "basis", + "prompt_tokens", + "elapsed_seconds", + "throughput_tokens_per_second", + "direct_energy_cost_per_token", + "published_cost_per_token", + }, + "pricing input evidence schema is wrong", + ) + require( + isinstance(output_basis, dict) + and set(output_basis) + == { + "basis", + "throughput_tokens_per_second", + "direct_energy_cost_per_token", + "published_cost_per_token", + }, + "pricing output evidence schema is wrong", + ) + accepted = context.get("accepted") + require( + isinstance(accepted, list) and accepted and isinstance(accepted[0], dict), + "pricing cold-prefill source is missing", + ) + cold = accepted[0] + prompt_tokens = input_basis.get("prompt_tokens") + elapsed = input_basis.get("elapsed_seconds") + input_throughput = input_basis.get("throughput_tokens_per_second") + output_throughput = output_basis.get("throughput_tokens_per_second") + numeric = (int, float) + require( + input_basis.get("basis") == "cold-prefill" + and prompt_tokens == cold.get("expected_prompt_tokens") == 319999 + and prompt_tokens == cold.get("server_prompt_tokens") + and elapsed == cold.get("latency_seconds") + and isinstance(elapsed, numeric) + and not isinstance(elapsed, bool) + and elapsed > 0 + and isinstance(input_throughput, numeric) + and not isinstance(input_throughput, bool) + and math.isclose( + input_throughput, + prompt_tokens / elapsed, + rel_tol=1e-12, + abs_tol=0.0, + ), + "pricing input value does not prove the measured cold-prefill throughput", + ) + require( + output_basis.get("basis") == "selected-weka-profile" + and isinstance(output_throughput, numeric) + and not isinstance(output_throughput, bool) + and output_throughput == weka.get("output_token_throughput") + and output_throughput > 0, + "pricing output value does not match measured Weka throughput", + ) + energy_per_second = power / 1000 * rate / 3600 + expected_input_direct = energy_per_second / input_throughput + expected_output_direct = energy_per_second / output_throughput + input_direct = input_basis.get("direct_energy_cost_per_token") + output_direct = output_basis.get("direct_energy_cost_per_token") + input_published = input_basis.get("published_cost_per_token") + output_published = output_basis.get("published_cost_per_token") + require( + isinstance(input_direct, numeric) + and not isinstance(input_direct, bool) + and math.isclose( + input_direct, expected_input_direct, rel_tol=1e-12, abs_tol=0.0 + ) + and input_published == INPUT_COST_PER_TOKEN, + "pricing input value is wrong", + ) + require( + isinstance(output_direct, numeric) + and not isinstance(output_direct, bool) + and math.isclose( + output_direct, expected_output_direct, rel_tol=1e-12, abs_tol=0.0 + ) + and output_published == OUTPUT_COST_PER_TOKEN, + "pricing output value is wrong", + ) + return input_published, output_published + + +def validate_rank_evidence( + root: dict[str, Any], + run_root: Path, + image: str, + accepted: datetime, +) -> None: + live = child(root, "live_state") + serving_generation = live.get("serving_generation") + require( + isinstance(serving_generation, str) and serving_generation, + "live serving generation is missing", + ) + require( + live.get("restart_counts_zero") is True + and live.get("fatal_error_matches") == 0 + and isinstance(live.get("kv_cache_tokens"), int) + and live["kv_cache_tokens"] > 0 + and isinstance(live.get("full_context_concurrency"), (int, float)) + and live["full_context_concurrency"] > 0, + "live state is not qualified", + ) + texts: dict[str, str] = {} + for key, label in ( + ("rank_0_evidence", "rank 0 evidence"), + ("rank_1_evidence", "rank 1 evidence"), + ("api_evidence", "API evidence"), + ): + item = child(live, key, "acceptance.live_state") + raw_path = item.get("path") + require(isinstance(raw_path, str), f"{label} path must be a string") + path = private_file(Path(raw_path), run_root, label) + require(path.parent == run_root / "logs", f"{label} path is wrong") + digest = item.get("sha256") + require( + isinstance(digest, str) + and SHA256.fullmatch(digest) is not None + and file_hash(path) == digest, + f"{label} hash does not match content", + ) + require( + path.stat().st_mtime <= accepted.timestamp() + 5, + f"{label} was captured after acceptance", + ) + texts[key] = path.read_text(encoding="utf-8") + rank0 = texts["rank_0_evidence"] + rank1 = texts["rank_1_evidence"] + + def serving_args(text: str, label: str) -> list[str]: + rows = [line.split(" args=", 1)[1] for line in text.splitlines() if " args=" in line] + require(len(rows) == 1, f"{label} serving arguments are missing or duplicated") + try: + value = json.loads(rows[0]) + except json.JSONDecodeError as error: + raise ValidationError(f"{label} serving arguments are malformed") from error + require( + isinstance(value, list) and all(isinstance(arg, str) for arg in value), + f"{label} serving arguments must be a string array", + ) + return value + + rank0_args = EXPECTED_COMMON_SERVING_ARGS + [ + "--node-rank", + "0", + "--host", + "100.73.140.127", + "--port", + "8000", + ] + rank1_args = EXPECTED_COMMON_SERVING_ARGS + ["--node-rank", "1", "--headless"] + require( + serving_args(rank0, "rank 0") == rank0_args + and f"image={image}" in rank0 + and "running=true" in rank0 + and "restart=0" in rank0 + and "serving_key_vars=1" in rank0 + and "health=pass" in rank0 + and "100.73.140.127:8000" in rank0, + "rank 0 serving arguments or identity for shamu are wrong", + ) + require( + serving_args(rank1, "rank 1") == rank1_args + and f"image={image}" in rank1 + and "running=true" in rank1 + and "restart=0" in rank1 + and "serving_key_vars=0" in rank1 + and "listener=none" in rank1 + and "fatal_error_matches=0" in rank1, + "rank 1 serving arguments or identity for tilikum are wrong", + ) + require( + f"generation={serving_generation}" in rank0 + and f"generation={serving_generation}" in rank1, + "rank evidence does not match the accepted serving generation", + ) + api_rows = [ + json.loads(line) + for line in texts["api_evidence"].splitlines() + if line.strip().startswith("{") + ] + require( + len(api_rows) == 1 + and api_rows[0].get("missing_key_status") == 401 + and api_rows[0].get("wrong_key_status") == 401 + and api_rows[0].get("correct_key_status") == 200 + and api_rows[0].get("status") == 200 + and api_rows[0].get("model") == MODEL + and api_rows[0].get("has_content") is True, + "API auth evidence must prove missing/wrong 401 and correct-key 200", + ) + require( + api_rows[0].get("serving_generation") == serving_generation, + "API auth evidence references a different serving generation", + ) + + +def validate() -> None: + args = parse_args() + require(args.max_age_seconds > 0, "maximum age must be positive") + run_root = args.run_root.resolve(strict=True) + require( + run_root.is_dir() and not args.run_root.is_symlink(), + "run root must be a regular directory", + ) + acceptance = private_file(args.acceptance, run_root, "private acceptance") + require( + acceptance == run_root / "private-acceptance.json", + "private acceptance path is wrong", + ) + root = load_json(acceptance, "private acceptance") + require(isinstance(root, dict), "private acceptance must be an object") + require(root.get("schema_version") == 1, "private acceptance schema is wrong") + require(root.get("decision") == "GO", "private acceptance decision is not GO") + require( + root.get("model") == MODEL + and root.get("public_model_name") == PUBLIC_MODEL, + "private acceptance model identity is wrong", + ) + accepted = timestamp(root.get("accepted_at"), "accepted_at") + now = timestamp(args.now, "now") if args.now else datetime.now(timezone.utc) + age = (now - accepted).total_seconds() + require( + age >= -CLOCK_SKEW_SECONDS, + "private acceptance timestamp is beyond the documented clock-skew allowance", + ) + require(age <= args.max_age_seconds, "private acceptance is stale") + topology = child(root, "topology") + require( + topology + == { + "tensor_parallel_size": 2, + "pipeline_parallel_size": 1, + "rank_0": "shamu", + "rank_1": "tilikum", + "rank_1_headless_and_keyless": True, + }, + "rank identity or topology is wrong", + ) + require( + (run_root / "pricing-evidence.json").is_file(), + "pricing evidence is absent", + ) + for section, label in ( + (child(root, "feature_qualification"), "feature qualification"), + (child(root, "long_context_qualification"), "context qualification"), + (child(root, "weka_qualification"), "Weka qualification"), + (child(root, "pricing_evidence"), "pricing evidence"), + (child(child(root, "live_state"), "rank_0_evidence", "acceptance.live_state"), "Shamu rank"), + (child(child(root, "live_state"), "rank_1_evidence", "acceptance.live_state"), "Tilikum rank"), + (child(child(root, "live_state"), "api_evidence", "acceptance.live_state"), "station API auth"), + ): + raw_path = section.get("path") + require(isinstance(raw_path, str), f"{label} path must be a string") + evidence_path = private_file(Path(raw_path), run_root, label) + require_fresh(evidence_path, label, accepted, now, args.max_age_seconds) + validate_profile(root, run_root) + image = validate_runtime(root, run_root) + validate_feature(root, run_root) + context = validate_context(root, run_root) + serving_generation = child(root, "live_state").get("serving_generation") + require(isinstance(serving_generation, str), "live serving generation is missing") + weka = validate_weka(root, run_root, serving_generation) + input_cost, output_cost = validate_pricing( + root, + run_root, + serving_generation, + context, + weka, + ) + validate_rank_evidence(root, run_root, image, accepted) + publication = child(root, "publication") + require( + publication + == { + "public_model_name": PUBLIC_MODEL, + "served_model_name": MODEL, + "max_context": 1048576, + "station_endpoint": STATION_ENDPOINT, + "supports_function_calling": True, + "supports_reasoning": True, + "supports_response_schema": True, + "supports_vision": True, + "input_cost_per_token": input_cost, + "output_cost_per_token": output_cost, + }, + "publication parameters do not match the accepted record", + ) + selection = child(root, "selection") + require( + selection.get("pp2_triggered") is False + and isinstance(selection.get("pp2_reason"), str) + and selection["pp2_reason"] + and isinstance(selection.get("reason"), str) + and selection["reason"], + "selection rationale is incomplete", + ) + gain = selection.get("seqs_8_request_throughput_gain_percent") + require( + isinstance(gain, (int, float)) + and not isinstance(gain, bool) + and gain == 0.89, + "selection measurement is wrong", + ) + + +def main() -> int: + try: + validate() + except (ValidationError, OSError, json.JSONDecodeError) as error: + print(f"ERROR: {error}", file=sys.stderr) + return 1 + print("private acceptance validated") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/webster/deepseek-v41/scripts/verify-runtime-registry.py b/webster/deepseek-v41/scripts/verify-runtime-registry.py new file mode 100755 index 0000000..b588988 --- /dev/null +++ b/webster/deepseek-v41/scripts/verify-runtime-registry.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python3 +"""Verify that an installed vLLM registry exposes a required model class.""" + +from __future__ import annotations + +import argparse +import importlib.util +import re +from pathlib import Path + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--package-root", type=Path) + parser.add_argument("--require-model", required=True) + return parser.parse_args() + + +def discover_package_roots() -> list[Path]: + spec = importlib.util.find_spec("vllm") + if spec is None or spec.submodule_search_locations is None: + raise SystemExit("installed vLLM package was not found") + return [Path(location).resolve() for location in spec.submodule_search_locations] + + +def main() -> None: + args = parse_args() + if not re.fullmatch(r"[A-Za-z][A-Za-z0-9_]*", args.require_model): + raise SystemExit("required model is not a Python identifier") + + roots = ( + [args.package_root.resolve()] + if args.package_root is not None + else discover_package_roots() + ) + registry_paths = [ + root / "model_executor" / "models" / "registry.py" for root in roots + ] + registry_paths = [path for path in registry_paths if path.is_file()] + if len(registry_paths) != 1: + raise SystemExit( + f"expected exactly one installed vLLM registry, found {len(registry_paths)}" + ) + + registry_path = registry_paths[0] + registry = registry_path.read_text(encoding="utf-8") + entry = re.compile(rf'''["']{re.escape(args.require_model)}["']\s*:''') + if entry.search(registry) is None: + raise SystemExit(f"vLLM registry lacks {args.require_model}") + print(f"registry-model={args.require_model} path={registry_path}") + + +if __name__ == "__main__": + main() diff --git a/webster/deepseek-v41/scripts/verify-serving-pair.py b/webster/deepseek-v41/scripts/verify-serving-pair.py new file mode 100755 index 0000000..2f3a2ac --- /dev/null +++ b/webster/deepseek-v41/scripts/verify-serving-pair.py @@ -0,0 +1,121 @@ +#!/usr/bin/env python3 +"""Verify that two validated serving ranks belong to one generation.""" + +from __future__ import annotations + +import argparse +from datetime import datetime, timezone +import hashlib +import json +import re +import sys +from typing import Any + + +DOCKER_TIMESTAMP = re.compile( + r"^(?P\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})" + r"(?:\.(?P\d{1,9}))?Z$" +) +IMAGE_ID = re.compile(r"^sha256:[0-9a-f]{64}$") +CONTAINER_ID = re.compile(r"^[0-9a-f]{64}$") + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--rank0-state-json", required=True) + parser.add_argument("--rank1-state-json", required=True) + parser.add_argument("--max-start-skew-seconds", required=True, type=float) + parser.add_argument("--require-matching-image", action="store_true") + parser.add_argument("--print-generation", action="store_true") + return parser.parse_args() + + +def load_state(raw: str, rank: int) -> dict[str, Any]: + try: + value = json.loads(raw) + except json.JSONDecodeError as error: + raise RuntimeError(f"rank {rank} state is malformed JSON: {error}") from error + if not isinstance(value, dict): + raise RuntimeError(f"rank {rank} state must be an object") + return value + + +def parse_docker_timestamp(value: Any, rank: int) -> datetime: + if not isinstance(value, str): + raise RuntimeError(f"rank {rank} start timestamp must be a string") + match = DOCKER_TIMESTAMP.fullmatch(value) + if match is None: + raise RuntimeError(f"rank {rank} start timestamp is invalid") + fraction = (match.group("fraction") or "")[:6].ljust(6, "0") + normalized = f"{match.group('whole')}.{fraction}+00:00" + try: + return datetime.fromisoformat(normalized).astimezone(timezone.utc) + except ValueError as error: + raise RuntimeError(f"rank {rank} start timestamp is invalid") from error + + +def main() -> int: + args = parse_args() + try: + if args.max_start_skew_seconds < 0: + raise RuntimeError("maximum start skew must not be negative") + states = [ + load_state(args.rank0_state_json, 0), + load_state(args.rank1_state_json, 1), + ] + if args.require_matching_image: + images = [state.get("image") for state in states] + if ( + not all(isinstance(image, str) and IMAGE_ID.fullmatch(image) for image in images) + or images[0] != images[1] + ): + raise RuntimeError("serving ranks use different image generations") + + generations = [state.get("generation") for state in states] + generation_id = "" + if any(generation not in (None, "") for generation in generations): + if ( + not all(isinstance(generation, str) and generation for generation in generations) + or generations[0] != generations[1] + ): + raise RuntimeError("serving ranks are from different generations") + generation_id = generations[0] + else: + started = [ + parse_docker_timestamp(state.get("started_at"), rank) + for rank, state in enumerate(states) + ] + if abs((started[0] - started[1]).total_seconds()) > args.max_start_skew_seconds: + raise RuntimeError("legacy serving ranks were not started as one generation") + if args.print_generation: + container_ids = [state.get("container_id") for state in states] + if not all( + isinstance(container_id, str) + and CONTAINER_ID.fullmatch(container_id) + for container_id in container_ids + ): + raise RuntimeError("legacy serving ranks lack container identities") + identity = json.dumps( + [ + { + "container_id": container_ids[rank], + "started_at": states[rank].get("started_at"), + } + for rank in range(2) + ], + separators=(",", ":"), + sort_keys=True, + ).encode("utf-8") + generation_id = "legacy-" + hashlib.sha256(identity).hexdigest() + except RuntimeError as error: + print(f"ERROR: {error}", file=sys.stderr) + return 1 + if args.print_generation: + print(generation_id) + else: + print("serving pair generation validated") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/webster/deepseek-v41/scripts/verify-weka-corpus.py b/webster/deepseek-v41/scripts/verify-weka-corpus.py new file mode 100755 index 0000000..c153ce8 --- /dev/null +++ b/webster/deepseek-v41/scripts/verify-weka-corpus.py @@ -0,0 +1,144 @@ +#!/usr/bin/env python3 +"""Bind split Weka traces to one immutable source JSONL identity.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import re +import stat +import sys +from pathlib import Path +from typing import BinaryIO + + +SHA256 = re.compile(r"^[0-9a-f]{64}$") +REVISION = re.compile(r"^[0-9a-f]{40}$") + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--dataset", required=True, type=Path) + parser.add_argument("--manifest", required=True, type=Path) + parser.add_argument("--repository", required=True) + parser.add_argument("--revision", required=True) + parser.add_argument("--source-sha256", required=True) + parser.add_argument("--source-size", required=True, type=int) + parser.add_argument("--trace-count", required=True, type=int) + return parser.parse_args() + + +def sha256_stream(handle: BinaryIO) -> str: + digest = hashlib.sha256() + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def fail(message: str) -> None: + raise RuntimeError(message) + + +def verify(args: argparse.Namespace) -> None: + if not args.repository or REVISION.fullmatch(args.revision) is None: + fail("expected Weka repository or revision is malformed") + if SHA256.fullmatch(args.source_sha256) is None: + fail("expected Weka source hash is malformed") + if args.source_size <= 0 or args.trace_count <= 0: + fail("expected Weka source size or trace count is invalid") + + if not args.dataset.is_dir() or args.dataset.is_symlink(): + fail("Weka dataset must be a regular non-symlink directory") + dataset = args.dataset.resolve(strict=True) + source = dataset.parent / "traces.jsonl" + if not source.is_file() or source.is_symlink(): + fail("pinned Weka source JSONL is missing") + if source.stat().st_size != args.source_size: + fail("pinned Weka source size differs") + with source.open("rb") as handle: + if sha256_stream(handle) != args.source_sha256: + fail("pinned Weka source hash differs") + + if not args.manifest.is_file() or args.manifest.is_symlink(): + fail("Weka corpus manifest must be a regular non-symlink file") + manifest_path = args.manifest.resolve(strict=True) + if manifest_path.parent != dataset.parent: + fail("Weka corpus manifest must be beside the trace directory") + if stat.S_IMODE(manifest_path.stat().st_mode) != 0o600: + fail("Weka corpus manifest must have mode 0600") + value = json.loads(manifest_path.read_text(encoding="utf-8")) + if not isinstance(value, dict) or value.get("schema_version") != 1: + fail("Weka corpus manifest is malformed") + if ( + value.get("repository") != args.repository + or value.get("revision") != args.revision + or value.get("trace_count") != args.trace_count + or value.get("full_subagents") is not True + or value.get("source_sha256") != args.source_sha256 + or value.get("source_size") != args.source_size + ): + fail("Weka corpus manifest does not identify the pinned source") + files = value.get("files") + if not isinstance(files, dict): + fail("Weka corpus manifest has no file identities") + expected_names = { + f"trace-{index:03d}.json" for index in range(args.trace_count) + } + if set(files) != expected_names: + fail("Weka corpus manifest does not list the exact trace set") + actual_names = {path.name for path in dataset.glob("*.json")} + if actual_names != expected_names: + fail("Weka dataset does not contain the exact JSON trace set") + + with source.open("rb") as source_handle: + for index in range(args.trace_count): + source_line = source_handle.readline() + if not source_line: + fail("pinned Weka source has fewer traces than expected") + name = f"trace-{index:03d}.json" + expected_hash = files[name] + if not isinstance(expected_hash, str) or SHA256.fullmatch(expected_hash) is None: + fail(f"Weka corpus file identity is malformed: {name}") + path = dataset / name + if not path.is_file() or path.is_symlink(): + fail(f"Weka corpus trace is missing or unsafe: {name}") + resolved = path.resolve(strict=True) + if not resolved.is_relative_to(dataset): + fail(f"Weka corpus trace escapes the dataset: {name}") + with resolved.open("rb") as trace_handle: + actual_hash = sha256_stream(trace_handle) + if actual_hash != expected_hash: + fail(f"Weka corpus file hash differs: {name}") + if hashlib.sha256(source_line).hexdigest() != actual_hash: + fail(f"Weka corpus trace does not match pinned source: {name}") + if source_handle.readline(): + fail("pinned Weka source has more traces than expected") + + +def main() -> int: + args = parse_args() + try: + verify(args) + except (OSError, RuntimeError, UnicodeError, json.JSONDecodeError) as error: + print(f"ERROR: {error}", file=sys.stderr) + return 1 + print( + json.dumps( + { + "repository": args.repository, + "revision": args.revision, + "source_sha256": args.source_sha256, + "source_size": args.source_size, + "trace_count": args.trace_count, + "full_subagents": True, + }, + sort_keys=True, + ) + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/webster/deepseek-v41/tests/fixtures/aiperf-errors.jsonl b/webster/deepseek-v41/tests/fixtures/aiperf-errors.jsonl new file mode 100644 index 0000000..02ae09b --- /dev/null +++ b/webster/deepseek-v41/tests/fixtures/aiperf-errors.jsonl @@ -0,0 +1,2 @@ +{"request_id":"ok-1","status_code":200,"error":null} +{"request_id":"bad-1","status_code":500,"error":"EngineDeadError"} diff --git a/webster/deepseek-v41/tests/fixtures/aiperf-profile.json b/webster/deepseek-v41/tests/fixtures/aiperf-profile.json new file mode 100644 index 0000000..c711299 --- /dev/null +++ b/webster/deepseek-v41/tests/fixtures/aiperf-profile.json @@ -0,0 +1,7 @@ +{ + "metrics": { + "request_throughput_avg": {"mean": 0.25}, + "time_to_first_token_p90": {"mean": 1200.0}, + "output_token_throughput_avg": {"mean": 42.5} + } +} diff --git a/webster/deepseek-v41/tests/fixtures/chat.json b/webster/deepseek-v41/tests/fixtures/chat.json new file mode 100644 index 0000000..a81eea5 --- /dev/null +++ b/webster/deepseek-v41/tests/fixtures/chat.json @@ -0,0 +1,9 @@ +{ + "model": "glm-5.2", + "messages": [ + {"role": "system", "content": "Answer precisely."}, + {"role": "user", "content": [{"type": "text", "text": "What is 2 + 2?"}]} + ], + "max_tokens": 32, + "temperature": 0 +} diff --git a/webster/deepseek-v41/tests/fixtures/reasoning-none.json b/webster/deepseek-v41/tests/fixtures/reasoning-none.json new file mode 100644 index 0000000..6f883e0 --- /dev/null +++ b/webster/deepseek-v41/tests/fixtures/reasoning-none.json @@ -0,0 +1,9 @@ +{ + "model": "glm-5.2", + "messages": [ + {"role": "user", "content": "Return only the word cobalt."} + ], + "max_completion_tokens": 32, + "reasoning_effort": "none", + "temperature": 0 +} diff --git a/webster/deepseek-v41/tests/fixtures/structured.json b/webster/deepseek-v41/tests/fixtures/structured.json new file mode 100644 index 0000000..cbaab0b --- /dev/null +++ b/webster/deepseek-v41/tests/fixtures/structured.json @@ -0,0 +1,25 @@ +{ + "model": "glm-5.2", + "messages": [ + {"role": "user", "content": "Return the city and temperature as JSON."} + ], + "response_format": { + "type": "json_schema", + "json_schema": { + "name": "weather", + "strict": true, + "schema": { + "type": "object", + "properties": { + "city": {"type": "string"}, + "temperature_c": {"type": "number"} + }, + "required": ["city", "temperature_c"], + "additionalProperties": false + } + } + }, + "chat_template_kwargs": {"enable_thinking": false}, + "max_tokens": 64, + "temperature": 0 +} diff --git a/webster/deepseek-v41/tests/fixtures/tokenize-golden.json b/webster/deepseek-v41/tests/fixtures/tokenize-golden.json new file mode 100644 index 0000000..0a7e42b --- /dev/null +++ b/webster/deepseek-v41/tests/fixtures/tokenize-golden.json @@ -0,0 +1,420 @@ +{ + "chat.json": { + "count": 24, + "token_ids": [ + 154822, + 154824, + 154826, + 25062, + 287, + 29905, + 371, + 25, + 7487, + 154826, + 16127, + 23573, + 13, + 154827, + 3838, + 374, + 220, + 17, + 488, + 220, + 17, + 30, + 154828, + 154841 + ] + }, + "reasoning-none.json": { + "count": 13, + "token_ids": [ + 154822, + 154824, + 154827, + 5598, + 1172, + 279, + 3409, + 33666, + 3145, + 13, + 154828, + 154841, + 154842 + ] + }, + "structured.json": { + "count": 14, + "token_ids": [ + 154822, + 154824, + 154827, + 5598, + 279, + 3283, + 323, + 9312, + 438, + 4718, + 13, + 154828, + 154841, + 154842 + ] + }, + "tool-result.json": { + "count": 182, + "token_ids": [ + 154822, + 154824, + 154826, + 25062, + 287, + 29905, + 371, + 25, + 7487, + 154826, + 198, + 2, + 13843, + 271, + 2610, + 1231, + 1618, + 825, + 476, + 803, + 5746, + 311, + 7789, + 448, + 279, + 1196, + 3239, + 382, + 2610, + 525, + 3897, + 448, + 729, + 32479, + 2878, + 366, + 15906, + 1472, + 15906, + 29, + 11868, + 9488, + 510, + 27, + 15906, + 397, + 4913, + 606, + 788, + 330, + 455, + 68852, + 497, + 330, + 4684, + 788, + 330, + 1949, + 1482, + 9101, + 10460, + 330, + 13777, + 788, + 5212, + 1313, + 788, + 330, + 1700, + 497, + 330, + 13185, + 788, + 5212, + 8923, + 788, + 5212, + 1313, + 788, + 330, + 917, + 9204, + 2137, + 330, + 6279, + 788, + 4383, + 8923, + 1341, + 11243, + 522, + 15906, + 1339, + 2461, + 1817, + 729, + 1618, + 11, + 2550, + 279, + 729, + 829, + 323, + 5977, + 2878, + 279, + 2701, + 11868, + 3561, + 510, + 154843, + 90, + 1688, + 11489, + 92, + 154847, + 90, + 858, + 16159, + 12, + 16, + 92, + 154848, + 154849, + 90, + 858, + 19048, + 12, + 16, + 92, + 154850, + 154847, + 90, + 858, + 16159, + 12, + 17, + 92, + 154848, + 154849, + 90, + 858, + 19048, + 12, + 17, + 92, + 154850, + 1112, + 154844, + 154827, + 3838, + 374, + 279, + 9101, + 304, + 5836, + 12872, + 30, + 154828, + 154841, + 154842, + 154843, + 455, + 68852, + 154847, + 8923, + 154848, + 154849, + 23663, + 12872, + 154850, + 154844, + 154829, + 154845, + 99082, + 356, + 323, + 30132, + 4577, + 154846, + 154828, + 154841 + ] + }, + "tools.json": { + "count": 160, + "token_ids": [ + 154822, + 154824, + 154826, + 25062, + 287, + 29905, + 371, + 25, + 7487, + 154826, + 198, + 2, + 13843, + 271, + 2610, + 1231, + 1618, + 825, + 476, + 803, + 5746, + 311, + 7789, + 448, + 279, + 1196, + 3239, + 382, + 2610, + 525, + 3897, + 448, + 729, + 32479, + 2878, + 366, + 15906, + 1472, + 15906, + 29, + 11868, + 9488, + 510, + 27, + 15906, + 397, + 4913, + 606, + 788, + 330, + 455, + 68852, + 497, + 330, + 4684, + 788, + 330, + 1949, + 1482, + 9101, + 10460, + 330, + 13777, + 788, + 5212, + 1313, + 788, + 330, + 1700, + 497, + 330, + 13185, + 788, + 5212, + 8923, + 788, + 5212, + 1313, + 788, + 330, + 917, + 9204, + 2137, + 330, + 6279, + 788, + 4383, + 8923, + 1341, + 11243, + 522, + 15906, + 1339, + 2461, + 1817, + 729, + 1618, + 11, + 2550, + 279, + 729, + 829, + 323, + 5977, + 2878, + 279, + 2701, + 11868, + 3561, + 510, + 154843, + 90, + 1688, + 11489, + 92, + 154847, + 90, + 858, + 16159, + 12, + 16, + 92, + 154848, + 154849, + 90, + 858, + 19048, + 12, + 16, + 92, + 154850, + 154847, + 90, + 858, + 16159, + 12, + 17, + 92, + 154848, + 154849, + 90, + 858, + 19048, + 12, + 17, + 92, + 154850, + 1112, + 154844, + 154827, + 3838, + 374, + 279, + 9101, + 304, + 5836, + 12872, + 30, + 154828, + 154841 + ] + } +} diff --git a/webster/deepseek-v41/tests/fixtures/tool-result.json b/webster/deepseek-v41/tests/fixtures/tool-result.json new file mode 100644 index 0000000..c7e40b8 --- /dev/null +++ b/webster/deepseek-v41/tests/fixtures/tool-result.json @@ -0,0 +1,34 @@ +{ + "model": "glm-5.2", + "messages": [ + {"role": "user", "content": "What is the weather in San Francisco?"}, + { + "role": "assistant", + "content": "", + "tool_calls": [ + { + "id": "call_fixture_1", + "type": "function", + "function": {"name": "get_weather", "arguments": "{\"city\":\"San Francisco\"}"} + } + ] + }, + {"role": "tool", "tool_call_id": "call_fixture_1", "content": "15 C and foggy"} + ], + "tools": [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get current weather.", + "parameters": { + "type": "object", + "properties": {"city": {"type": "string"}}, + "required": ["city"] + } + } + } + ], + "max_tokens": 64, + "temperature": 0 +} diff --git a/webster/deepseek-v41/tests/fixtures/tools.json b/webster/deepseek-v41/tests/fixtures/tools.json new file mode 100644 index 0000000..12c0274 --- /dev/null +++ b/webster/deepseek-v41/tests/fixtures/tools.json @@ -0,0 +1,23 @@ +{ + "model": "glm-5.2", + "messages": [ + {"role": "user", "content": "What is the weather in San Francisco?"} + ], + "tools": [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get current weather.", + "parameters": { + "type": "object", + "properties": {"city": {"type": "string"}}, + "required": ["city"] + } + } + } + ], + "tool_choice": "auto", + "max_tokens": 64, + "temperature": 0 +} diff --git a/webster/deepseek-v41/tests/requirements.txt b/webster/deepseek-v41/tests/requirements.txt new file mode 100644 index 0000000..f62ce0c --- /dev/null +++ b/webster/deepseek-v41/tests/requirements.txt @@ -0,0 +1 @@ +PyYAML==6.0.3 diff --git a/webster/deepseek-v41/tests/test_aiperf_tools.py b/webster/deepseek-v41/tests/test_aiperf_tools.py new file mode 100644 index 0000000..c68e2e8 --- /dev/null +++ b/webster/deepseek-v41/tests/test_aiperf_tools.py @@ -0,0 +1,2339 @@ +#!/usr/bin/env python3 +"""Behavior tests for cache-controlled Weka result summarization.""" + +from __future__ import annotations + +import json +import hashlib +import os +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + + +REPOSITORY_ROOT = Path(__file__).resolve().parents[3] +PACKAGE_ROOT = REPOSITORY_ROOT / "webster" / "deepseek-v41" +SUMMARIZER = PACKAGE_ROOT / "scripts" / "summarize-aiperf.py" +RUNNER = PACKAGE_ROOT / "scripts" / "run-weka.sh" +PATCHER = PACKAGE_ROOT / "scripts" / "patch-aiperf-weka-timing.py" +CORPUS_VERIFIER = PACKAGE_ROOT / "scripts" / "verify-weka-corpus.py" +CORPUS_BUILDER = PACKAGE_ROOT / "scripts" / "build-weka-corpus-manifest.py" +FIXTURES = PACKAGE_ROOT / "tests" / "fixtures" + + +def write_valid_runner_exports( + directory: Path, + *, + readiness: object | None = None, + readiness_raw: str | None = None, + record_error: bool = False, + missing_server_metric: str | None = None, +) -> None: + directory.mkdir(parents=True, exist_ok=True) + if readiness_raw is not None: + (directory / ".aiperf_results_ready.json").write_text( + readiness_raw, encoding="utf-8" + ) + elif readiness is not None: + (directory / ".aiperf_results_ready.json").write_text( + json.dumps(readiness) + "\n", encoding="utf-8" + ) + + (directory / "profile_export_aiperf.json").write_text( + json.dumps( + { + "request_throughput": {"unit": "requests/sec", "avg": 1.0}, + "time_to_first_token": {"unit": "ms", "p90": 25.0}, + "output_token_throughput": {"unit": "tokens/sec", "avg": 8.0}, + } + ) + + "\n", + encoding="utf-8", + ) + record: dict[str, object] = { + "metadata": { + "request_start_ns": 100, + "request_end_ns": 200, + "agent_depth": 0, + "root_correlation_id": "root-1", + }, + "metrics": {}, + } + if record_error: + record["error"] = {"type": "EngineDeadError"} + (directory / "profile_export.jsonl").write_text( + json.dumps(record) + "\n", encoding="utf-8" + ) + + def series( + metric_type: str, + value_name: str, + value: float, + *, + endpoint: str = "http://100.73.140.127:8000/metrics", + labels: dict[str, str] | None = None, + ) -> dict[str, object]: + return { + "type": metric_type, + "series": [ + { + "endpoint_url": endpoint, + "labels": labels or {}, + "stats": {value_name: value}, + } + ], + } + + metrics = { + "vllm:prefix_cache_hits": series("counter", "total", 8.0), + "vllm:prefix_cache_queries": series("counter", "total", 10.0), + "vllm:kv_cache_usage_perc": series("gauge", "max", 0.5), + "vllm:num_preemptions": series("counter", "total", 0.0), + "vllm:num_requests_running": series("gauge", "max", 1.0), + "vllm:num_requests_waiting": series("gauge", "max", 0.0), + "power_total_watts": { + "type": "gauge", + "series": [ + { + "endpoint_url": "http://100.73.140.127:9877/metrics", + "labels": {"scope": "wall"}, + "stats": {"avg": 900.0}, + }, + { + "endpoint_url": "http://100.73.89.150:9877/metrics", + "labels": {"scope": "facility"}, + "stats": {"avg": 1000.0}, + }, + ], + }, + "power_energy_kwh": series( + "counter", "total", 0.25, labels={"scope": "facility"} + ), + "power_cost_dollars": series( + "counter", "total", 0.12, labels={"scope": "facility"} + ), + } + if missing_server_metric is not None: + del metrics[missing_server_metric] + (directory / "server_metrics_export.json").write_text( + json.dumps({"metrics": metrics}) + "\n", encoding="utf-8" + ) + + +def run_weka_fixture( + root: Path, + fixture: Path, + *, + profile_name: str, + environment_overrides: dict[str, str] | None = None, + verify_corpus: bool = False, + corrupt_corpus: bool = False, + repetition: int = 1, + concurrency: int | None = None, +) -> tuple[subprocess.CompletedProcess[str], Path]: + runs_root = root / "runs" + run_root = runs_root / "20990101T000000Z" + run_root.mkdir(parents=True, exist_ok=True) + manifest = run_root / "manifest.env" + manifest.write_text( + "AIPERF_COMMIT=ea28b2e81c7367f8403c6f6ebe0837a508795a8d\n" + "WEKA_REPOSITORY=semianalysisai/cc-traces-weka-062126\n" + "WEKA_REVISION=23f152f6f0f9399a85901b89a6458def0ef16729\n", + encoding="utf-8", + ) + os.chmod(manifest, 0o600) + dataset = root / "traces" + dataset.mkdir(exist_ok=True) + (dataset / "trace.json").write_text( + '{"id":"fixture","requests":[]}\n', encoding="utf-8" + ) + corpus_environment: dict[str, str] = {} + if verify_corpus: + (dataset / "trace.json").unlink() + source = dataset.parent / "traces.jsonl" + source_line = b'{"id":"fixture","requests":[]}\n' + source.write_bytes(source_line) + split = dataset / "trace-000.json" + split.write_bytes( + b'{"id":"forged","requests":[]}\n' if corrupt_corpus else source_line + ) + source_hash = hashlib.sha256(source.read_bytes()).hexdigest() + corpus_manifest = dataset.parent / "corpus-manifest.json" + corpus_manifest.write_text( + json.dumps( + { + "schema_version": 1, + "repository": "semianalysisai/cc-traces-weka-062126", + "revision": "23f152f6f0f9399a85901b89a6458def0ef16729", + "trace_count": 1, + "full_subagents": True, + "source_sha256": source_hash, + "source_size": source.stat().st_size, + "files": { + split.name: hashlib.sha256(split.read_bytes()).hexdigest() + }, + } + ) + + "\n", + encoding="utf-8", + ) + os.chmod(corpus_manifest, 0o600) + corpus_environment = { + "TEST_WEKA_VERIFY_CORPUS": "1", + "TEST_WEKA_SOURCE_SHA256": source_hash, + "TEST_WEKA_SOURCE_SIZE": str(source.stat().st_size), + "TEST_WEKA_SOURCE_TRACE_COUNT": "1", + } + tokenizer = root / "tokenizer" + tokenizer.mkdir(exist_ok=True) + key_file = root / "benchmark.key" + key_file.write_text("test-weka-secret\n", encoding="utf-8") + os.chmod(key_file, 0o600) + fake = root / "aiperf" + fake.write_text( + "#!/usr/bin/env bash\n" + "set -euo pipefail\n" + "if [[ -n \"${TEST_AIPERF_ENV_CAPTURE:-}\" ]]; then\n" + " printf '%s\\n' \"${AIPERF_DATASET_CONFIGURATION_TIMEOUT:-unset}\" " + "\"${AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT:-unset}\" " + ">\"$TEST_AIPERF_ENV_CAPTURE\"\n" + "fi\n" + "config=\n" + "while (($#)); do\n" + " if [[ $1 == --config ]]; then config=$2; shift 2; else shift; fi\n" + "done\n" + "cp -a \"$AIPERF_OUTPUT_FIXTURE_ROOT/.\" \"$(dirname \"$config\")/\"\n" + "printf '%s\\n' '05:31:54.473 NOTICE Phase profiling (profiling) complete | completed=1, cancelled=0, errors=0 | sessions: completed=1, cancelled=0 | elapsed=1.00s (runner.py:1227)'\n" + "exit \"${TEST_AIPERF_EXIT_CODE:-0}\"\n", + encoding="utf-8", + ) + os.chmod(fake, 0o700) + command = [ + "bash", + str(RUNNER), + "--run-root", + str(run_root), + "--profile-name", + profile_name, + "--repetition", + str(repetition), + "--key-file", + str(key_file), + ] + if concurrency is None: + command.append("--fixed-schedule") + run_name = f"{profile_name}-r{repetition}-fixed" + else: + command.extend(["--concurrency", str(concurrency), "--no-fixed-schedule"]) + run_name = f"{profile_name}-c{concurrency}-r{repetition}-open" + command.append("--internal") + result = subprocess.run( + command, + cwd=REPOSITORY_ROOT, + env={ + **os.environ, + "WEBSTER_WEKA_RUNS_ROOT": str(runs_root), + "WEBSTER_WEKA_TEST_MODE": "1", + "WEBSTER_WEKA_AIPERF_BIN": str(fake), + "WEBSTER_WEKA_DATASET": str(dataset), + "WEBSTER_WEKA_TOKENIZER": str(tokenizer), + "AIPERF_OUTPUT_FIXTURE_ROOT": str(fixture), + **corpus_environment, + **(environment_overrides or {}), + }, + text=True, + capture_output=True, + check=False, + ) + return result, run_root / "aiperf" / run_name + + +class AIPerfToolTests(unittest.TestCase): + def test_weka_runner_accepts_a_bsd_realpath_without_gnu_flags(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fixture = root / "fixture" + write_valid_runner_exports( + fixture, + readiness={ + "ready": True, + "was_cancelled": False, + "partial": False, + }, + ) + fake_bin = root / "bin" + fake_bin.mkdir() + fake_realpath = fake_bin / "realpath" + fake_realpath.write_text( + "#!/usr/bin/env bash\n" + "set -euo pipefail\n" + "if [[ ${1:-} == -* ]]; then\n" + " printf 'realpath: illegal option -- %s\\n' \"${1#-}\" >&2\n" + " exit 64\n" + "fi\n" + "python3 - \"$1\" <<'PY'\n" + "import os\n" + "import sys\n" + "print(os.path.realpath(sys.argv[1]))\n" + "PY\n", + encoding="utf-8", + ) + os.chmod(fake_realpath, 0o700) + + result, destination = run_weka_fixture( + root, + fixture, + profile_name="bsd-realpath", + environment_overrides={ + "PATH": f"{fake_bin}:{os.environ['PATH']}" + }, + ) + + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual((destination / "EXIT").read_text(), "EXIT=0\n") + + def test_weka_runner_accepts_a_bsd_stat_without_gnu_format(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fixture = root / "fixture" + write_valid_runner_exports( + fixture, + readiness={ + "ready": True, + "was_cancelled": False, + "partial": False, + }, + ) + fake_bin = root / "bin" + fake_bin.mkdir() + fake_stat = fake_bin / "stat" + fake_stat.write_text( + "#!/usr/bin/env bash\n" + "set -euo pipefail\n" + "if [[ ${1:-} == -c ]]; then\n" + " printf 'stat: illegal option -- c\\n' >&2\n" + " exit 64\n" + "fi\n" + "exec /usr/bin/stat \"$@\"\n", + encoding="utf-8", + ) + os.chmod(fake_stat, 0o700) + + result, destination = run_weka_fixture( + root, + fixture, + profile_name="bsd-stat", + environment_overrides={ + "PATH": f"{fake_bin}:{os.environ['PATH']}" + }, + ) + + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual((destination / "EXIT").read_text(), "EXIT=0\n") + + def test_weka_runner_accepts_a_bsd_mktemp_without_parent_option(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fixture = root / "fixture" + write_valid_runner_exports( + fixture, + readiness={ + "ready": True, + "was_cancelled": False, + "partial": False, + }, + ) + fake_bin = root / "bin" + fake_bin.mkdir() + fake_mktemp = fake_bin / "mktemp" + fake_mktemp.write_text( + "#!/usr/bin/env bash\n" + "set -euo pipefail\n" + "for argument in \"$@\"; do\n" + " if [[ $argument == -p ]]; then\n" + " printf 'mktemp: illegal option -- p\\n' >&2\n" + " exit 64\n" + " fi\n" + "done\n" + "exec /usr/bin/mktemp \"$@\"\n", + encoding="utf-8", + ) + os.chmod(fake_mktemp, 0o700) + + result, destination = run_weka_fixture( + root, + fixture, + profile_name="bsd-mktemp", + environment_overrides={ + "PATH": f"{fake_bin}:{os.environ['PATH']}" + }, + ) + + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual((destination / "EXIT").read_text(), "EXIT=0\n") + + def test_weka_runner_rejects_memory_limited_controller_before_aiperf(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fixture = root / "fixture" + aiperf_invocation = root / "aiperf-environment.log" + write_valid_runner_exports( + fixture, + readiness={ + "ready": True, + "was_cancelled": False, + "partial": False, + }, + ) + + result, destination = run_weka_fixture( + root, + fixture, + profile_name="memory-limited-controller", + environment_overrides={ + "TEST_WEKA_CONTROLLER_MEMORY_BYTES": str(8 * 1024**3), + "TEST_AIPERF_ENV_CAPTURE": str(aiperf_invocation), + }, + ) + + self.assertNotEqual(result.returncode, 0, result.stderr) + self.assertIn("controller physical memory", result.stderr) + self.assertFalse(aiperf_invocation.exists()) + self.assertFalse(destination.exists()) + + def test_weka_runner_pins_timeouts_for_cold_corpus_configuration(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fixture = root / "fixture" + environment_capture = root / "aiperf-environment.log" + write_valid_runner_exports( + fixture, + readiness={ + "ready": True, + "was_cancelled": False, + "partial": False, + }, + ) + + result, _ = run_weka_fixture( + root, + fixture, + profile_name="cold-corpus-timeout", + environment_overrides={ + "TEST_AIPERF_ENV_CAPTURE": str(environment_capture) + }, + ) + + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual( + environment_capture.read_text(encoding="utf-8").splitlines(), + ["1200", "1200"], + ) + + def test_corpus_builder_writes_private_manifest_bound_to_source(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + dataset = root / "traces" + dataset.mkdir() + source = root / "traces.jsonl" + lines = [b'{"id":"one","requests":[]}\n', b'{"id":"two","requests":[]}\n'] + source.write_bytes(b"".join(lines)) + for index, line in enumerate(lines): + (dataset / f"trace-{index:03d}.json").write_bytes(line) + source_hash = hashlib.sha256(source.read_bytes()).hexdigest() + manifest = dataset.parent / "corpus-manifest.json" + + result = subprocess.run( + [ + sys.executable, + str(CORPUS_BUILDER), + "--dataset", + str(dataset), + "--output", + str(manifest), + "--repository", + "fixture/repository", + "--revision", + "a" * 40, + "--source-sha256", + source_hash, + "--source-size", + str(source.stat().st_size), + "--trace-count", + "2", + ], + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + ) + + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual(os.stat(manifest).st_mode & 0o777, 0o600) + value = json.loads(manifest.read_text(encoding="utf-8")) + self.assertEqual(value["source_sha256"], source_hash) + self.assertEqual(set(value["files"]), {"trace-000.json", "trace-001.json"}) + verify = subprocess.run( + [ + sys.executable, + str(CORPUS_VERIFIER), + "--dataset", + str(dataset), + "--manifest", + str(manifest), + "--repository", + "fixture/repository", + "--revision", + "a" * 40, + "--source-sha256", + source_hash, + "--source-size", + str(source.stat().st_size), + "--trace-count", + "2", + ], + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertEqual(verify.returncode, 0, verify.stderr) + + def test_corpus_builder_rejects_unmanifested_json_loader_input(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + dataset = root / "traces" + dataset.mkdir() + source = root / "traces.jsonl" + source_line = b'{"id":"one","requests":[]}\n' + source.write_bytes(source_line) + (dataset / "trace-000.json").write_bytes(source_line) + (dataset / "unexpected.json").write_text( + '{"id":"injected","requests":[]}\n', encoding="utf-8" + ) + source_hash = hashlib.sha256(source.read_bytes()).hexdigest() + + result = subprocess.run( + [ + sys.executable, + str(CORPUS_BUILDER), + "--dataset", + str(dataset), + "--output", + str(root / "corpus-manifest.json"), + "--repository", + "fixture/repository", + "--revision", + "a" * 40, + "--source-sha256", + source_hash, + "--source-size", + str(source.stat().st_size), + "--trace-count", + "1", + ], + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + ) + + self.assertNotEqual(result.returncode, 0) + self.assertIn("exact JSON trace set", result.stderr) + + def test_corpus_verifier_rejects_rehashed_split_not_from_pinned_source(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + dataset = root / "traces" + dataset.mkdir() + source = root / "traces.jsonl" + source_lines = [b'{"id":"one","requests":[]}\n', b'{"id":"two","requests":[]}\n'] + source.write_bytes(b"".join(source_lines)) + for index, line in enumerate(source_lines): + (dataset / f"trace-{index:03d}.json").write_bytes(line) + tampered = dataset / "trace-001.json" + tampered.write_bytes(b'{"id":"forged","requests":[]}\n') + source_hash = hashlib.sha256(source.read_bytes()).hexdigest() + manifest = dataset.parent / "corpus-manifest.json" + manifest.write_text( + json.dumps( + { + "schema_version": 1, + "repository": "fixture/repository", + "revision": "a" * 40, + "trace_count": 2, + "full_subagents": True, + "source_sha256": source_hash, + "source_size": source.stat().st_size, + "files": { + path.name: hashlib.sha256(path.read_bytes()).hexdigest() + for path in sorted(dataset.glob("trace-*.json")) + }, + } + ) + + "\n", + encoding="utf-8", + ) + os.chmod(manifest, 0o600) + + result = subprocess.run( + [ + sys.executable, + str(CORPUS_VERIFIER), + "--dataset", + str(dataset), + "--manifest", + str(manifest), + "--repository", + "fixture/repository", + "--revision", + "a" * 40, + "--source-sha256", + source_hash, + "--source-size", + str(source.stat().st_size), + "--trace-count", + "2", + ], + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + ) + + self.assertNotEqual(result.returncode, 0) + self.assertIn("pinned source", result.stderr) + + def test_corpus_verifier_rejects_json_added_after_manifest_creation(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + dataset = root / "traces" + dataset.mkdir() + source = root / "traces.jsonl" + source_line = b'{"id":"one","requests":[]}\n' + source.write_bytes(source_line) + split = dataset / "trace-000.json" + split.write_bytes(source_line) + source_hash = hashlib.sha256(source.read_bytes()).hexdigest() + manifest = root / "corpus-manifest.json" + manifest.write_text( + json.dumps( + { + "schema_version": 1, + "repository": "fixture/repository", + "revision": "a" * 40, + "trace_count": 1, + "full_subagents": True, + "source_sha256": source_hash, + "source_size": source.stat().st_size, + "files": { + split.name: hashlib.sha256(split.read_bytes()).hexdigest() + }, + } + ) + + "\n", + encoding="utf-8", + ) + os.chmod(manifest, 0o600) + (dataset / "unexpected.json").write_text( + '{"id":"injected","requests":[]}\n', encoding="utf-8" + ) + + result = subprocess.run( + [ + sys.executable, + str(CORPUS_VERIFIER), + "--dataset", + str(dataset), + "--manifest", + str(manifest), + "--repository", + "fixture/repository", + "--revision", + "a" * 40, + "--source-sha256", + source_hash, + "--source-size", + str(source.stat().st_size), + "--trace-count", + "1", + ], + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + ) + + self.assertNotEqual(result.returncode, 0) + self.assertIn("exact JSON trace set", result.stderr) + + def test_weka_runner_invokes_immutable_corpus_verification(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fixture = root / "fixture" + write_valid_runner_exports( + fixture, + readiness={ + "ready": True, + "was_cancelled": False, + "partial": False, + }, + ) + + result, destination = run_weka_fixture( + root, + fixture, + profile_name="forged-corpus", + verify_corpus=True, + corrupt_corpus=True, + ) + + self.assertNotEqual(result.returncode, 0) + self.assertIn("pinned source", result.stderr) + self.assertFalse(destination.exists()) + + def test_weka_runner_rejects_fatal_worker_rank_log(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fixture = root / "fixture" + write_valid_runner_exports( + fixture, + readiness={ + "ready": True, + "was_cancelled": False, + "partial": False, + }, + ) + + result, destination = run_weka_fixture( + root, + fixture, + profile_name="worker-engine-death", + environment_overrides={ + "TEST_WEKA_RANK1_LOG": "EngineDeadError: worker failed" + }, + ) + + self.assertNotEqual(result.returncode, 0) + self.assertEqual((destination / "EXIT").read_text(), "EXIT=1\n") + self.assertIn("fatal serving-rank log", result.stderr) + self.assertFalse((destination / "provenance.json").exists()) + + def test_weka_runner_preserves_direct_aiperf_failure(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fixture = root / "fixture" + write_valid_runner_exports( + fixture, + readiness={ + "ready": True, + "was_cancelled": False, + "partial": False, + }, + ) + + result, destination = run_weka_fixture( + root, + fixture, + profile_name="aiperf-exit-failure", + environment_overrides={"TEST_AIPERF_EXIT_CODE": "7"}, + ) + + self.assertNotEqual(result.returncode, 0) + self.assertEqual((destination / "EXIT").read_text(), "EXIT=7\n") + self.assertIn("AIPerf exited with status 7", result.stderr) + self.assertNotIn("unbound variable", result.stderr) + self.assertFalse((destination / "provenance.json").exists()) + + def test_weka_scored_repetition_requires_same_generation_warmup(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fixture = root / "fixture" + write_valid_runner_exports( + fixture, + readiness={ + "ready": True, + "was_cancelled": False, + "partial": False, + }, + ) + + result, destination = run_weka_fixture( + root, + fixture, + profile_name="scored-without-warmup", + repetition=2, + concurrency=4, + ) + + self.assertNotEqual(result.returncode, 0) + self.assertIn("warmup repetition", result.stderr) + self.assertFalse(destination.exists()) + + def test_weka_scored_repetition_binds_exact_warmup_provenance(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fixture = root / "fixture" + write_valid_runner_exports( + fixture, + readiness={ + "ready": True, + "was_cancelled": False, + "partial": False, + }, + ) + warmup, warmup_destination = run_weka_fixture( + root, + fixture, + profile_name="sequential", + repetition=1, + concurrency=4, + ) + self.assertEqual(warmup.returncode, 0, warmup.stderr) + + scored, scored_destination = run_weka_fixture( + root, + fixture, + profile_name="sequential", + repetition=2, + concurrency=4, + ) + + self.assertEqual(scored.returncode, 0, scored.stderr) + warmup_provenance = warmup_destination / "provenance.json" + scored_provenance = json.loads( + (scored_destination / "provenance.json").read_text(encoding="utf-8") + ) + self.assertEqual(scored_provenance["cache_state"], "warm") + self.assertTrue(scored_provenance["warmup_and_scored_comparable"]) + self.assertEqual( + scored_provenance["warmup_provenance_sha256"], + hashlib.sha256(warmup_provenance.read_bytes()).hexdigest(), + ) + + def test_weka_scored_repetition_rejects_different_warmup_runtime_patch(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fixture = root / "fixture" + write_valid_runner_exports( + fixture, + readiness={ + "ready": True, + "was_cancelled": False, + "partial": False, + }, + ) + warmup, warmup_destination = run_weka_fixture( + root, + fixture, + profile_name="runtime-patch-continuity", + repetition=1, + concurrency=4, + ) + self.assertEqual(warmup.returncode, 0, warmup.stderr) + warmup_provenance = warmup_destination / "provenance.json" + payload = json.loads(warmup_provenance.read_text(encoding="utf-8")) + payload["aiperf_weka_patch_sha256"] = "0" * 64 + warmup_provenance.write_text( + json.dumps(payload, sort_keys=True) + "\n", + encoding="utf-8", + ) + + scored, scored_destination = run_weka_fixture( + root, + fixture, + profile_name="runtime-patch-continuity", + repetition=2, + concurrency=4, + ) + + self.assertNotEqual(scored.returncode, 0) + self.assertIn("runtime patch", scored.stderr) + self.assertFalse(scored_destination.exists()) + + def test_weka_timing_patcher_is_idempotent(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + source = Path(temporary) / "aiperf" + resolver = source / "src" / "aiperf" / "config" / "dataset" / "resolver.py" + resolver.parent.mkdir(parents=True) + resolver.write_text( + " CustomDatasetType.BURST_GPT_TRACE,\n" + " CustomDatasetType.BASETEN_TRACE,\n" + " CustomDatasetType.TRACELAB,\n", + encoding="utf-8", + ) + + first = subprocess.run( + [sys.executable, str(PATCHER), "--source", str(source)], + text=True, + capture_output=True, + check=False, + ) + self.assertEqual(first.returncode, 0, first.stderr) + after_first = resolver.read_text(encoding="utf-8") + self.assertEqual(after_first.count("CustomDatasetType.WEKA_TRACE"), 1) + + second = subprocess.run( + [sys.executable, str(PATCHER), "--source", str(source)], + text=True, + capture_output=True, + check=False, + ) + self.assertEqual(second.returncode, 0, second.stderr) + self.assertEqual(resolver.read_text(encoding="utf-8"), after_first) + + def test_weka_timing_patcher_rejects_divergent_source(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + source = Path(temporary) / "aiperf" + resolver = source / "src" / "aiperf" / "config" / "dataset" / "resolver.py" + resolver.parent.mkdir(parents=True) + resolver.write_text("unexpected upstream source\n", encoding="utf-8") + + result = subprocess.run( + [sys.executable, str(PATCHER), "--source", str(source)], + text=True, + capture_output=True, + check=False, + ) + self.assertNotEqual(result.returncode, 0) + self.assertIn("unsupported AIPerf resolver source", result.stderr) + self.assertEqual( + resolver.read_text(encoding="utf-8"), "unexpected upstream source\n" + ) + + def test_summarizer_counts_request_errors_before_accepting_aggregates(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + output = Path(temporary) / "summary.json" + result = subprocess.run( + [ + sys.executable, + str(SUMMARIZER), + "--profile", + str(FIXTURES / "aiperf-profile.json"), + "--records", + str(FIXTURES / "aiperf-errors.jsonl"), + "--output-json", + str(output), + ], + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertNotEqual(result.returncode, 0, result.stderr) + summary = json.loads(output.read_text(encoding="utf-8")) + self.assertEqual(summary["requests"], 2) + self.assertEqual(summary["completed"], 1) + self.assertEqual(summary["errors"], 1) + self.assertEqual(summary["server_success_rate"], 0.5) + self.assertEqual(summary["request_throughput"], 0.25) + self.assertEqual(summary["ttft_p90_ms"], 1200.0) + self.assertEqual(summary["output_token_throughput"], 42.5) + self.assertEqual(summary["error_types"], {"EngineDeadError": 1}) + self.assertFalse(summary["qualified"]) + + def test_summarizer_reads_current_aiperf_schema_and_request_overlap(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + profile = root / "profile.json" + records = root / "records.jsonl" + output = root / "summary.json" + profile.write_text( + json.dumps( + { + "request_throughput": {"unit": "requests/sec", "avg": 1.5}, + "time_to_first_token": {"unit": "ms", "p90": 900.0}, + "output_token_throughput": {"unit": "tokens/sec", "avg": 88.0}, + } + ) + + "\n", + encoding="utf-8", + ) + rows = [ + { + "metadata": { + "request_start_ns": 100, + "request_end_ns": 300, + "agent_depth": 0, + "root_correlation_id": "root-1", + }, + "metrics": {}, + }, + { + "metadata": { + "request_start_ns": 200, + "request_end_ns": 400, + "agent_depth": 1, + "root_correlation_id": "root-1", + }, + "metrics": {}, + }, + ] + records.write_text( + "".join(json.dumps(row) + "\n" for row in rows), encoding="utf-8" + ) + + result = subprocess.run( + [ + sys.executable, + str(SUMMARIZER), + "--profile", + str(profile), + "--records", + str(records), + "--output-json", + str(output), + ], + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + summary = json.loads(output.read_text(encoding="utf-8")) + self.assertEqual(summary["requests"], 2) + self.assertEqual(summary["errors"], 0) + self.assertEqual(summary["request_throughput"], 1.5) + self.assertEqual(summary["ttft_p90_ms"], 900.0) + self.assertEqual(summary["output_token_throughput"], 88.0) + self.assertEqual(summary["request_peak_concurrency"], 2) + self.assertEqual(summary["root_session_trees"], 1) + self.assertEqual(summary["request_count_by_agent_depth"], {"0": 1, "1": 1}) + self.assertNotIn("prefix_cache_hits", summary) + self.assertNotIn("completed_root_workflows", summary) + self.assertNotIn("cancelled_root_workflows", summary) + self.assertNotIn("root_workflow_elapsed_seconds", summary) + self.assertNotIn("completed_root_workflows_per_second", summary) + self.assertTrue(summary["qualified"]) + + def test_summarizer_reads_completed_root_workflows_from_timing_log(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + timing_log = root / "aiperf.stdout.log" + output = root / "summary.json" + timing_log.write_text( + "05:31:00.000 INFO profiling in progress\n" + "05:31:54.473 NOTICE Phase profiling (profiling) complete | " + "completed=354, cancelled=0, errors=0 | sessions: completed=3, " + "cancelled=1 | elapsed=60.00s (runner.py:1227)\n", + encoding="utf-8", + ) + + result = subprocess.run( + [ + sys.executable, + str(SUMMARIZER), + "--profile", + str(FIXTURES / "aiperf-profile.json"), + "--records", + str(FIXTURES / "aiperf-errors.jsonl"), + "--timing-log", + str(timing_log), + "--output-json", + str(output), + ], + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertNotEqual(result.returncode, 0, result.stderr) + summary = json.loads(output.read_text(encoding="utf-8")) + self.assertEqual(summary["completed_root_workflows"], 3) + self.assertEqual(summary["cancelled_root_workflows"], 1) + self.assertEqual(summary["root_workflow_elapsed_seconds"], 60.0) + self.assertEqual(summary["completed_root_workflows_per_second"], 0.05) + self.assertFalse(summary["qualified"]) + + def test_summarizer_accepts_grouped_aiperf_request_counts(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + timing_log = root / "aiperf.log" + output = root / "summary.json" + timing_log.write_text( + "2026-09-12 15:16:53.471 - PhaseRunner - NOTICE - " + "Phase profiling (profiling) complete | completed=1,036, " + "cancelled=0, errors=0 | sessions: completed=6, cancelled=0 | " + "elapsed=916.22s\n", + encoding="utf-8", + ) + + result = subprocess.run( + [ + sys.executable, + str(SUMMARIZER), + "--profile", + str(FIXTURES / "aiperf-profile.json"), + "--records", + str(FIXTURES / "aiperf-errors.jsonl"), + "--timing-log", + str(timing_log), + "--output-json", + str(output), + ], + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertNotEqual(result.returncode, 0, result.stderr) + summary = json.loads(output.read_text(encoding="utf-8")) + self.assertEqual(summary["completed_root_workflows"], 6) + self.assertEqual(summary["cancelled_root_workflows"], 0) + self.assertEqual(summary["root_workflow_elapsed_seconds"], 916.22) + + def test_summarizer_rejects_timing_log_without_profiling_completion(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + timing_log = root / "aiperf.stdout.log" + output = root / "summary.json" + timing_log.write_text( + "05:31:00.000 INFO profiling in progress\n", encoding="utf-8" + ) + + result = subprocess.run( + [ + sys.executable, + str(SUMMARIZER), + "--profile", + str(FIXTURES / "aiperf-profile.json"), + "--records", + str(FIXTURES / "aiperf-errors.jsonl"), + "--timing-log", + str(timing_log), + "--output-json", + str(output), + ], + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertNotEqual(result.returncode, 0) + self.assertIn( + "timing log must contain exactly one profiling-complete line", + result.stderr, + ) + self.assertFalse(output.exists()) + + def test_summarizer_rejects_multiple_profiling_completion_lines(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + timing_log = root / "aiperf.log" + output = root / "summary.json" + completion = ( + "Phase profiling (profiling) complete | completed=354, cancelled=0, " + "errors=0 | sessions: completed=3, cancelled=0 | elapsed=900.03s\n" + ) + timing_log.write_text( + "05:31:54.473 NOTICE " + completion + + "2099-01-01 05:31:54.473 | aiperf | NOTICE | " + + completion, + encoding="utf-8", + ) + + result = subprocess.run( + [ + sys.executable, + str(SUMMARIZER), + "--profile", + str(FIXTURES / "aiperf-profile.json"), + "--records", + str(FIXTURES / "aiperf-errors.jsonl"), + "--timing-log", + str(timing_log), + "--output-json", + str(output), + ], + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertNotEqual(result.returncode, 0) + self.assertIn( + "timing log must contain exactly one profiling-complete line", + result.stderr, + ) + self.assertFalse(output.exists()) + + def test_summarizer_rejects_malformed_profiling_completion_line(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + timing_log = root / "aiperf.stdout.log" + output = root / "summary.json" + timing_log.write_text( + "05:31:54.473 NOTICE NOTPhase profiling (profiling) complete | " + "completed=354, cancelled=0, errors=0 | sessions: completed=3, " + "cancelled=0 | elapsed=900.03s\n", + encoding="utf-8", + ) + + result = subprocess.run( + [ + sys.executable, + str(SUMMARIZER), + "--profile", + str(FIXTURES / "aiperf-profile.json"), + "--records", + str(FIXTURES / "aiperf-errors.jsonl"), + "--timing-log", + str(timing_log), + "--output-json", + str(output), + ], + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertNotEqual(result.returncode, 0) + self.assertIn( + "profiling-complete timing log line is malformed", result.stderr + ) + self.assertFalse(output.exists()) + + def test_summarizer_rejects_zero_root_workflow_elapsed_time(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + timing_log = root / "aiperf.stdout.log" + output = root / "summary.json" + timing_log.write_text( + "05:31:54.473 NOTICE Phase profiling (profiling) complete | " + "completed=0, cancelled=0, errors=0 | sessions: completed=0, " + "cancelled=0 | elapsed=0.00s\n", + encoding="utf-8", + ) + + result = subprocess.run( + [ + sys.executable, + str(SUMMARIZER), + "--profile", + str(FIXTURES / "aiperf-profile.json"), + "--records", + str(FIXTURES / "aiperf-errors.jsonl"), + "--timing-log", + str(timing_log), + "--output-json", + str(output), + ], + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertNotEqual(result.returncode, 0) + self.assertIn( + "root workflow elapsed seconds must be greater than zero", + result.stderr, + ) + self.assertFalse(output.exists()) + + def test_summarizer_rejects_nonfinite_root_workflow_elapsed_time(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + timing_log = root / "aiperf.stdout.log" + output = root / "summary.json" + timing_log.write_text( + "05:31:54.473 NOTICE Phase profiling (profiling) complete | " + "completed=1, cancelled=0, errors=0 | sessions: completed=1, " + f"cancelled=0 | elapsed={'9' * 400}.00s\n", + encoding="utf-8", + ) + + result = subprocess.run( + [ + sys.executable, + str(SUMMARIZER), + "--profile", + str(FIXTURES / "aiperf-profile.json"), + "--records", + str(FIXTURES / "aiperf-errors.jsonl"), + "--timing-log", + str(timing_log), + "--output-json", + str(output), + ], + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertNotEqual(result.returncode, 0) + self.assertIn( + "root workflow elapsed seconds must be a finite number", + result.stderr, + ) + self.assertFalse(output.exists()) + + def test_summarizer_reads_server_metrics_for_profile_comparison(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + profile = root / "profile.json" + records = root / "records.jsonl" + server_metrics = root / "server_metrics_export.json" + output = root / "summary.json" + profile.write_text( + json.dumps( + { + "request_throughput": {"unit": "requests/sec", "avg": 0.5}, + "time_to_first_token": {"unit": "ms", "p90": 1200.0}, + "output_token_throughput": { + "unit": "tokens/sec", + "avg": 40.0, + }, + } + ) + + "\n", + encoding="utf-8", + ) + records.write_text( + json.dumps( + { + "metadata": { + "request_start_ns": 100, + "request_end_ns": 200, + "agent_depth": 0, + "root_correlation_id": "root-1", + }, + "metrics": {}, + } + ) + + "\n", + encoding="utf-8", + ) + server_metrics.write_text( + json.dumps( + { + "schema_version": "1.1", + "aiperf_version": "0.13.0", + "benchmark_id": "benchmark-fixture", + "summary": { + "endpoints_configured": [ + "http://head:8000/metrics", + "http://shamu:9877/metrics", + "http://tilikum:9877/metrics", + ], + "endpoints_successful": [ + "http://head:8000/metrics", + "http://shamu:9877/metrics", + "http://tilikum:9877/metrics", + ], + "start_time": "2099-01-01T00:00:00Z", + "end_time": "2099-01-01T00:10:00Z", + }, + "metrics_phase": "profiling", + "metrics": { + "vllm:prefix_cache_hits": { + "type": "counter", + "description": "Prefix cache hits", + "series": [ + { + "endpoint_url": "http://head:8000/metrics", + "labels": {"model_name": "deepseek-v4.1-flash"}, + "stats": {"total": 720.0, "rate": 1.2}, + } + ], + }, + "vllm:prefix_cache_queries": { + "type": "counter", + "description": "Prefix cache queries", + "series": [ + { + "endpoint_url": "http://head:8000/metrics", + "labels": {"model_name": "deepseek-v4.1-flash"}, + "stats": {"total": 800.0, "rate": 1.333}, + } + ], + }, + "vllm:kv_cache_usage_perc": { + "type": "gauge", + "description": "KV-cache usage", + "series": [ + { + "endpoint_url": "http://head:8000/metrics", + "labels": {"model_name": "deepseek-v4.1-flash"}, + "stats": { + "avg": 0.55, + "min": 0.1, + "max": 0.875, + }, + } + ], + }, + "vllm:num_preemptions": { + "type": "counter", + "description": "Request preemptions", + "series": [ + { + "endpoint_url": "http://head:8000/metrics", + "labels": {"model_name": "deepseek-v4.1-flash"}, + "stats": {"total": 3.0, "rate": 0.005}, + } + ], + }, + "vllm:num_requests_running": { + "type": "gauge", + "description": "Running requests", + "series": [ + { + "endpoint_url": "http://head:8000/metrics", + "labels": {"model_name": "deepseek-v4.1-flash"}, + "stats": {"avg": 0.8, "min": 0.0, "max": 1.0}, + } + ], + }, + "vllm:num_requests_waiting": { + "type": "gauge", + "description": "Waiting requests", + "series": [ + { + "endpoint_url": "http://head:8000/metrics", + "labels": {"model_name": "deepseek-v4.1-flash"}, + "stats": {"avg": 4.0, "min": 0.0, "max": 11.0}, + } + ], + }, + "power_total_watts": { + "type": "gauge", + "description": "Total power by scope", + "unit": "watts", + "series": [ + { + "endpoint_url": "http://shamu:9877/metrics", + "labels": {"scope": "measured"}, + "stats": {"avg": 425.0, "min": 400.0, "max": 450.0}, + }, + { + "endpoint_url": "http://shamu:9877/metrics", + "labels": {"scope": "wall"}, + "stats": {"avg": 510.0, "min": 490.0, "max": 530.0}, + }, + { + "endpoint_url": "http://shamu:9877/metrics", + "labels": {"scope": "facility"}, + "stats": {"avg": 550.0, "min": 525.0, "max": 575.0}, + }, + { + "endpoint_url": "http://tilikum:9877/metrics", + "labels": {"scope": "measured"}, + "stats": {"avg": 440.0, "min": 410.0, "max": 470.0}, + }, + { + "endpoint_url": "http://tilikum:9877/metrics", + "labels": {"scope": "wall"}, + "stats": {"avg": 530.0, "min": 500.0, "max": 560.0}, + }, + { + "endpoint_url": "http://tilikum:9877/metrics", + "labels": {"scope": "facility"}, + "stats": {"avg": 572.0, "min": 540.0, "max": 604.0}, + }, + ], + }, + "power_energy_kwh": { + "type": "counter", + "description": "Accumulated energy by scope", + "unit": "kilowatt-hours", + "series": [ + { + "endpoint_url": "http://shamu:9877/metrics", + "labels": {"scope": "measured"}, + "stats": {"total": 0.5, "rate": 0.0008}, + }, + { + "endpoint_url": "http://shamu:9877/metrics", + "labels": {"scope": "wall"}, + "stats": {"total": 0.7, "rate": 0.0011}, + }, + { + "endpoint_url": "http://shamu:9877/metrics", + "labels": {"scope": "facility"}, + "stats": {"total": 0.75, "rate": 0.0012}, + }, + { + "endpoint_url": "http://tilikum:9877/metrics", + "labels": {"scope": "measured"}, + "stats": {"total": 0.55, "rate": 0.0009}, + }, + { + "endpoint_url": "http://tilikum:9877/metrics", + "labels": {"scope": "wall"}, + "stats": {"total": 0.74, "rate": 0.0012}, + }, + { + "endpoint_url": "http://tilikum:9877/metrics", + "labels": {"scope": "facility"}, + "stats": {"total": 0.8, "rate": 0.0013}, + }, + ], + }, + "power_cost_dollars": { + "type": "counter", + "description": "Accumulated facility energy cost", + "unit": "dollars", + "series": [ + { + "endpoint_url": "http://shamu:9877/metrics", + "stats": {"total": 0.3525, "rate": 0.0006}, + }, + { + "endpoint_url": "http://tilikum:9877/metrics", + "stats": {"total": 0.376, "rate": 0.0006}, + }, + ], + }, + }, + "input_config": {"benchmark": {"model": "deepseek-v4.1-flash"}}, + } + ) + + "\n", + encoding="utf-8", + ) + + result = subprocess.run( + [ + sys.executable, + str(SUMMARIZER), + "--profile", + str(profile), + "--records", + str(records), + "--server-metrics", + str(server_metrics), + "--output-json", + str(output), + ], + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + summary = json.loads(output.read_text(encoding="utf-8")) + self.assertEqual(summary["prefix_cache_hits"], 720.0) + self.assertEqual(summary["prefix_cache_queries"], 800.0) + self.assertEqual(summary["prefix_cache_hit_ratio"], 0.9) + self.assertEqual(summary["kv_cache_usage_max"], 0.875) + self.assertEqual(summary["preemptions_total"], 3.0) + self.assertEqual(summary["running_requests_max"], 1.0) + self.assertEqual(summary["waiting_requests_max"], 11.0) + self.assertEqual(summary["wall_power_avg_watts"], 1040.0) + self.assertEqual(summary["facility_power_avg_watts"], 1122.0) + self.assertEqual(summary["facility_energy_kwh"], 1.55) + self.assertAlmostEqual(summary["facility_cost_dollars"], 0.7285) + + def test_summarizer_returns_null_for_unexported_server_metrics(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + server_metrics = root / "server_metrics_export.json" + output = root / "summary.json" + server_metrics.write_text( + json.dumps( + { + "schema_version": "1.1", + "aiperf_version": "0.13.0", + "benchmark_id": "benchmark-fixture", + "summary": { + "endpoints_configured": ["http://head:8000/metrics"], + "endpoints_successful": ["http://head:8000/metrics"], + "start_time": "2099-01-01T00:00:00Z", + "end_time": "2099-01-01T00:10:00Z", + }, + "metrics_phase": "profiling", + "metrics": { + "vllm:num_preemptions": { + "type": "counter", + "description": "Request preemptions", + "series": [ + { + "endpoint_url": "http://head:8000/metrics", + "labels": {"model_name": "deepseek-v4.1-flash"}, + "stats": {"total": 2.0, "rate": 0.003}, + } + ], + } + }, + "input_config": {}, + } + ) + + "\n", + encoding="utf-8", + ) + + result = subprocess.run( + [ + sys.executable, + str(SUMMARIZER), + "--profile", + str(FIXTURES / "aiperf-profile.json"), + "--records", + str(FIXTURES / "aiperf-errors.jsonl"), + "--server-metrics", + str(server_metrics), + "--output-json", + str(output), + ], + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertNotEqual(result.returncode, 0, result.stderr) + summary = json.loads(output.read_text(encoding="utf-8")) + self.assertIsNone(summary["prefix_cache_hits"]) + self.assertIsNone(summary["prefix_cache_queries"]) + self.assertIsNone(summary["prefix_cache_hit_ratio"]) + self.assertIsNone(summary["kv_cache_usage_max"]) + self.assertEqual(summary["preemptions_total"], 2.0) + self.assertIsNone(summary["running_requests_max"]) + self.assertIsNone(summary["waiting_requests_max"]) + self.assertIsNone(summary["wall_power_avg_watts"]) + self.assertIsNone(summary["facility_power_avg_watts"]) + self.assertIsNone(summary["facility_energy_kwh"]) + self.assertIsNone(summary["facility_cost_dollars"]) + self.assertFalse(summary["qualified"]) + + def test_summarizer_rejects_malformed_exported_server_metric(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + profile = root / "profile.json" + records = root / "records.jsonl" + server_metrics = root / "server_metrics_export.json" + output = root / "summary.json" + profile.write_text("{}\n", encoding="utf-8") + records.write_text( + '{"metadata":{"request_start_ns":100,"request_end_ns":200},' + '"metrics":{}}\n', + encoding="utf-8", + ) + server_metrics.write_text( + json.dumps( + { + "schema_version": "1.1", + "aiperf_version": "0.13.0", + "benchmark_id": "benchmark-fixture", + "summary": { + "endpoints_configured": ["http://head:8000/metrics"], + "endpoints_successful": ["http://head:8000/metrics"], + "start_time": "2099-01-01T00:00:00Z", + "end_time": "2099-01-01T00:10:00Z", + }, + "metrics_phase": "profiling", + "metrics": {"vllm:num_preemptions": None}, + "input_config": {}, + } + ) + + "\n", + encoding="utf-8", + ) + + result = subprocess.run( + [ + sys.executable, + str(SUMMARIZER), + "--profile", + str(profile), + "--records", + str(records), + "--server-metrics", + str(server_metrics), + "--output-json", + str(output), + ], + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertNotEqual(result.returncode, 0) + self.assertIn( + "server metric vllm:num_preemptions must be an object", + result.stderr, + ) + self.assertFalse(output.exists()) + + def test_weka_runner_keeps_the_key_out_of_argv_and_cleans_exact_temp(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + runs_root = root / "runs" + run_root = runs_root / "20990101T000000Z" + run_root.mkdir(parents=True) + manifest = run_root / "manifest.env" + manifest.write_text( + "AIPERF_COMMIT=ea28b2e81c7367f8403c6f6ebe0837a508795a8d\n" + "WEKA_REPOSITORY=semianalysisai/cc-traces-weka-062126\n" + "WEKA_REVISION=23f152f6f0f9399a85901b89a6458def0ef16729\n", + encoding="utf-8", + ) + os.chmod(manifest, 0o600) + dataset = root / "traces" + dataset.mkdir() + (dataset / "trace.json").write_text( + '{"id":"fixture","requests":[]}\n', encoding="utf-8" + ) + key_file = root / "benchmark.key" + key_file.write_text("test-weka-secret\n", encoding="utf-8") + os.chmod(key_file, 0o600) + tokenizer = root / "tokenizer" + tokenizer.mkdir() + temp_root = run_root / "tmp" + fake = root / "aiperf" + config_capture = root / "aiperf-config.json" + output_fixture = root / "aiperf-output-fixture" + write_valid_runner_exports( + output_fixture, + readiness={ + "ready": True, + "was_cancelled": False, + "partial": False, + }, + ) + fake.write_text( + "#!/usr/bin/env bash\n" + "set -euo pipefail\n" + "printf '%s\\n' \"$@\" >\"$AIPERF_TEST_ARGV\"\n" + "test \"${OPENAI_API_KEY:-}\" = test-weka-secret\n" + "config=\n" + "while (($#)); do\n" + " if [[ $1 == --config ]]; then config=$2; shift 2; else shift; fi\n" + "done\n" + "test -n \"$config\"\n" + "python3 - \"$config\" <<'PY'\n" + "from pathlib import Path\n" + "import stat\n" + "import sys\n" + "raise SystemExit(\n" + " 0 if stat.S_IMODE(Path(sys.argv[1]).stat().st_mode) == 0o600 " + "else 1\n" + ")\n" + "PY\n" + "cp \"$config\" \"$AIPERF_TEST_CONFIG\"\n" + "cp -a \"$AIPERF_OUTPUT_FIXTURE_ROOT/.\" \"$(dirname \"$config\")/\"\n" + "printf '%s\\n' '05:31:54.473 NOTICE Phase profiling (profiling) complete | completed=1, cancelled=0, errors=0 | sessions: completed=1, cancelled=0 | elapsed=1.00s (runner.py:1227)'\n", + encoding="utf-8", + ) + os.chmod(fake, 0o700) + argv_log = root / "argv.log" + environment = { + **os.environ, + "WEBSTER_WEKA_RUNS_ROOT": str(runs_root), + "WEBSTER_WEKA_TEST_MODE": "1", + "WEBSTER_WEKA_AIPERF_BIN": str(fake), + "WEBSTER_WEKA_DATASET": str(dataset), + "WEBSTER_WEKA_TOKENIZER": str(tokenizer), + "AIPERF_TEST_ARGV": str(argv_log), + "AIPERF_TEST_CONFIG": str(config_capture), + "AIPERF_OUTPUT_FIXTURE_ROOT": str(output_fixture), + } + result = subprocess.run( + [ + "bash", + str(RUNNER), + "--run-root", + str(run_root), + "--profile-name", + "baseline", + "--repetition", + "1", + "--key-file", + str(key_file), + "--fixed-schedule", + "--internal", + ], + cwd=REPOSITORY_ROOT, + env=environment, + text=True, + capture_output=True, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + argv = argv_log.read_text(encoding="utf-8") + self.assertNotIn("test-weka-secret", argv) + self.assertNotIn("--api-key", argv) + self.assertEqual(argv.splitlines()[0], "profile") + self.assertEqual(argv.splitlines()[1], "--config") + self.assertEqual(len(argv.splitlines()), 3) + config = json.loads(config_capture.read_text(encoding="utf-8")) + benchmark = config["benchmark"] + self.assertEqual(benchmark["endpoint"]["api_key"], "${OPENAI_API_KEY}") + self.assertEqual(benchmark["datasets"][0]["format"], "weka_trace") + self.assertEqual(benchmark["phases"][0]["type"], "fixed_schedule") + self.assertNotIn("concurrency", benchmark["phases"][0]) + self.assertTrue(benchmark["phases"][0]["auto_offset"]) + self.assertEqual( + benchmark["server_metrics"]["urls"], + [ + "http://100.73.140.127:8000/metrics", + "http://100.73.140.127:9877/metrics", + "http://100.73.89.150:9877/metrics", + ], + ) + destination = run_root / "aiperf" / "baseline-r1-fixed" + self.assertEqual((destination / "EXIT").read_text(), "EXIT=0\n") + self.assertTrue( + json.loads((destination / "summary.json").read_text())["qualified"] + ) + provenance = json.loads( + (destination / "provenance.json").read_text(encoding="utf-8") + ) + for name in ( + "aiperf.stderr.log", + "summary.json", + "benchmark-window.json", + "rank0-before.json", + "rank1-before.json", + "rank0-after.json", + "rank1-after.json", + "rank0-container.log", + "rank1-container.log", + ): + self.assertEqual( + provenance["artifacts"][name], + hashlib.sha256((destination / name).read_bytes()).hexdigest(), + ) + self.assertEqual( + json.loads((destination / "rank0-before.json").read_text()), + json.loads((destination / "rank0-after.json").read_text()), + ) + self.assertEqual( + json.loads((destination / "rank1-before.json").read_text()), + json.loads((destination / "rank1-after.json").read_text()), + ) + self.assertEqual( + os.stat(destination / "aiperf-config.yaml").st_mode & 0o777, 0o600 + ) + rendered = (destination / "rendered-command.txt").read_text() + self.assertNotIn("test-weka-secret", rendered) + self.assertNotIn("--api-key", rendered) + self.assertIn("--config", rendered) + self.assertTrue(temp_root.is_dir()) + self.assertEqual(list(temp_root.iterdir()), []) + + def test_weka_runner_carries_path_overrides_into_detached_tmux_run(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + injection_canary = root / "tmux-injection-canary" + runs_root = ( + root + / 'runs with spaces $(touch "$AIPERF_INJECTION_CANARY") [fixture]' + ) + run_root = runs_root / "20990101T000000Z" + run_root.mkdir(parents=True) + manifest = run_root / "manifest.env" + manifest.write_text( + "AIPERF_COMMIT=ea28b2e81c7367f8403c6f6ebe0837a508795a8d\n" + "WEKA_REPOSITORY=semianalysisai/cc-traces-weka-062126\n" + "WEKA_REVISION=23f152f6f0f9399a85901b89a6458def0ef16729\n", + encoding="utf-8", + ) + os.chmod(manifest, 0o600) + dataset = root / "traces" + dataset.mkdir() + (dataset / "trace.json").write_text( + '{"id":"fixture","requests":[]}\n', encoding="utf-8" + ) + tokenizer = root / "tokenizer" + tokenizer.mkdir() + key_file = root / "benchmark.key" + key_file.write_text("test-weka-secret\n", encoding="utf-8") + os.chmod(key_file, 0o600) + + fake_aiperf = root / "aiperf" + environment_capture = root / "aiperf-environment.log" + output_fixture = root / "aiperf-output-fixture" + write_valid_runner_exports( + output_fixture, + readiness={ + "ready": True, + "was_cancelled": False, + "partial": False, + }, + ) + fake_aiperf.write_text( + "#!/usr/bin/env bash\n" + "set -euo pipefail\n" + "printf '%s\\n' \"$WEBSTER_WEKA_RUNS_ROOT\" " + "\"$WEBSTER_WEKA_AIPERF_BIN\" \"$WEBSTER_WEKA_DATASET\" " + "\"$WEBSTER_WEKA_TOKENIZER\" \"${WEBSTER_LOCAL_SHAMU:-}\" " + "\"${WEBSTER_WEKA_AIPERF_SOURCE:-}\" " + ">\"$AIPERF_ENV_CAPTURE\"\n" + "cp -a \"$AIPERF_OUTPUT_FIXTURE_ROOT/.\" \"$(dirname \"${!#}\")/\"\n" + "printf '%s\\n' '05:31:54.473 NOTICE Phase profiling (profiling) complete | completed=1, cancelled=0, errors=0 | sessions: completed=1, cancelled=0 | elapsed=1.00s (runner.py:1227)'\n", + encoding="utf-8", + ) + os.chmod(fake_aiperf, 0o700) + + fake_bin = root / "bin" + fake_bin.mkdir() + fake_tmux = fake_bin / "tmux" + tmux_argv_capture = root / "tmux-argv.log" + fake_tmux.write_text( + "#!/usr/bin/env bash\n" + "set -euo pipefail\n" + "case ${1:-} in\n" + " new-session)\n" + " printf '%s\\n' \"$@\" >\"$TMUX_TEST_ARGV\"\n" + " command=${!#}\n" + " umask 002\n" + " env -u WEBSTER_WEKA_RUNS_ROOT " + "-u WEBSTER_WEKA_AIPERF_BIN " + "-u WEBSTER_WEKA_DATASET " + "-u WEBSTER_WEKA_TOKENIZER " + "-u WEBSTER_LOCAL_SHAMU " + "-u WEBSTER_WEKA_AIPERF_SOURCE " + "TMUX=fake bash -c \"$command\" || true\n" + " ;;\n" + " has-session) exit 1 ;;\n" + " *) exit 2 ;;\n" + "esac\n", + encoding="utf-8", + ) + os.chmod(fake_tmux, 0o700) + + environment = { + **os.environ, + "PATH": f"{fake_bin}:{os.environ['PATH']}", + "WEBSTER_WEKA_RUNS_ROOT": str(runs_root), + "WEBSTER_WEKA_TEST_MODE": "1", + "TEST_WEKA_ENABLE_TMUX": "1", + "WEBSTER_WEKA_AIPERF_BIN": str(fake_aiperf), + "WEBSTER_WEKA_DATASET": str(dataset), + "WEBSTER_WEKA_TOKENIZER": str(tokenizer), + "WEBSTER_LOCAL_SHAMU": "1", + "WEBSTER_WEKA_AIPERF_SOURCE": str(root / "aiperf-source"), + "AIPERF_ENV_CAPTURE": str(environment_capture), + "AIPERF_INJECTION_CANARY": str(injection_canary), + "AIPERF_OUTPUT_FIXTURE_ROOT": str(output_fixture), + "TMUX_TEST_ARGV": str(tmux_argv_capture), + } + environment.pop("TMUX", None) + result = subprocess.run( + [ + "bash", + str(RUNNER), + "--run-root", + str(run_root), + "--profile-name", + "detached", + "--repetition", + "1", + "--key-file", + str(key_file), + "--fixed-schedule", + ], + cwd=REPOSITORY_ROOT, + env=environment, + text=True, + capture_output=True, + check=False, + ) + tmux_log = run_root / "logs" / "detached-r1-fixed.tmux.log" + tmux_output = ( + tmux_log.read_text(encoding="utf-8") if tmux_log.exists() else "" + ) + self.assertEqual(os.stat(tmux_log).st_mode & 0o777, 0o600) + self.assertFalse(injection_canary.exists(), "tmux command injection ran") + self.assertEqual(result.returncode, 0, result.stderr + tmux_output) + self.assertEqual( + environment_capture.read_text(encoding="utf-8").splitlines(), + [ + str(runs_root.resolve()), + str(fake_aiperf), + str(dataset), + str(tokenizer), + "1", + str(root / "aiperf-source"), + ], + ) + tmux_argv = tmux_argv_capture.read_text(encoding="utf-8") + self.assertNotIn("test-weka-secret", tmux_argv) + self.assertNotIn("--api-key", tmux_argv) + rendered = ( + run_root + / "aiperf" + / "detached-r1-fixed" + / "rendered-command.txt" + ).read_text(encoding="utf-8") + self.assertNotIn("test-weka-secret", rendered) + self.assertNotIn("--api-key", rendered) + + def test_weka_runner_rejects_incomplete_result_publication(self) -> None: + cases: list[tuple[str, dict[str, object]]] = [ + ("missing-ready", {}), + ("malformed-ready", {"readiness_raw": "{not-json\n"}), + ( + "not-ready", + { + "readiness": { + "ready": False, + "was_cancelled": False, + "partial": False, + } + }, + ), + ( + "partial", + { + "readiness": { + "ready": True, + "was_cancelled": False, + "partial": True, + } + }, + ), + ( + "failed-exporter", + { + "readiness": { + "ready": True, + "was_cancelled": False, + "partial": True, + "failed_exporters": ["ServerMetricsJsonExporter"], + } + }, + ), + ] + for profile_name, fixture_options in cases: + with self.subTest(profile_name=profile_name), tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fixture = root / "fixture" + write_valid_runner_exports(fixture, **fixture_options) + result, destination = run_weka_fixture( + root, fixture, profile_name=profile_name + ) + self.assertNotEqual(result.returncode, 0, result.stderr) + self.assertIn("AIPerf results readiness", result.stderr) + self.assertNotEqual( + (destination / "EXIT").read_text(encoding="utf-8"), + "EXIT=0\n", + ) + + def test_weka_runner_rejects_unqualified_request_results(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fixture = root / "fixture" + write_valid_runner_exports( + fixture, + readiness={ + "ready": True, + "was_cancelled": False, + "partial": False, + }, + record_error=True, + ) + result, destination = run_weka_fixture( + root, fixture, profile_name="unqualified" + ) + self.assertNotEqual(result.returncode, 0, result.stderr) + summary = json.loads( + (destination / "summary.json").read_text(encoding="utf-8") + ) + self.assertFalse(summary["qualified"]) + self.assertIn("request results are not qualified", result.stderr) + + def test_weka_runner_verifies_exact_client_revision_and_full_corpus(self) -> None: + cases = ( + ("AIPERF_COMMIT", {"TEST_WEKA_AIPERF_COMMIT": "f" * 40}), + ( + "AIPerf Weka timing patch", + {"TEST_WEKA_AIPERF_PATCH_SHA256": "f" * 64}, + ), + ("WEKA_REVISION", {"TEST_WEKA_REVISION": "e" * 40}), + ("393 traces", {"TEST_WEKA_TRACE_COUNT": "392"}), + ("full subagents", {"TEST_WEKA_FULL_SUBAGENTS": "0"}), + ) + for expected, overrides in cases: + with self.subTest(expected=expected), tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fixture = root / "fixture" + write_valid_runner_exports( + fixture, + readiness={ + "ready": True, + "was_cancelled": False, + "partial": False, + }, + ) + result, _ = run_weka_fixture( + root, + fixture, + profile_name="pin-check", + environment_overrides=overrides, + ) + self.assertNotEqual(result.returncode, 0) + self.assertIn(expected, result.stderr) + + def test_weka_runner_revalidates_pins_before_reusing_completed_run(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fixture = root / "fixture" + write_valid_runner_exports( + fixture, + readiness={ + "ready": True, + "was_cancelled": False, + "partial": False, + }, + ) + first, destination = run_weka_fixture( + root, + fixture, + profile_name="resume-pin-check", + ) + self.assertEqual(first.returncode, 0, first.stderr) + self.assertEqual( + (destination / "EXIT").read_text(encoding="utf-8"), "EXIT=0\n" + ) + + resumed, _ = run_weka_fixture( + root, + fixture, + profile_name="resume-pin-check", + environment_overrides={"TEST_WEKA_AIPERF_COMMIT": "f" * 40}, + ) + + self.assertNotEqual(resumed.returncode, 0) + self.assertIn("checked-out AIPERF_COMMIT", resumed.stderr) + + def test_weka_runner_rejects_tampered_completed_run_provenance(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fixture = root / "fixture" + write_valid_runner_exports( + fixture, + readiness={ + "ready": True, + "was_cancelled": False, + "partial": False, + }, + ) + first, destination = run_weka_fixture( + root, + fixture, + profile_name="resume-provenance-check", + ) + self.assertEqual(first.returncode, 0, first.stderr) + provenance_path = destination / "provenance.json" + provenance = json.loads(provenance_path.read_text(encoding="utf-8")) + provenance["aiperf_commit"] = "f" * 40 + provenance_path.write_text( + json.dumps(provenance, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + + resumed, _ = run_weka_fixture( + root, + fixture, + profile_name="resume-provenance-check", + ) + + self.assertNotEqual(resumed.returncode, 0) + self.assertIn("recorded AIPerf commit", resumed.stderr) + + def test_weka_runner_rejects_type_confused_provenance_scalars(self) -> None: + cases: tuple[tuple[str, object], ...] = ( + ("schema_version", True), + ("concurrency", False), + ("repetition", True), + ("trace_count", 393.0), + ("full_subagents", 1), + ) + for field, forged in cases: + with self.subTest(field=field), tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fixture = root / "fixture" + write_valid_runner_exports( + fixture, + readiness={ + "ready": True, + "was_cancelled": False, + "partial": False, + }, + ) + first, destination = run_weka_fixture( + root, + fixture, + profile_name="resume-provenance-types", + ) + self.assertEqual(first.returncode, 0, first.stderr) + provenance_path = destination / "provenance.json" + provenance = json.loads(provenance_path.read_text(encoding="utf-8")) + provenance[field] = forged + provenance_path.write_text( + json.dumps(provenance, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + + resumed, _ = run_weka_fixture( + root, + fixture, + profile_name="resume-provenance-types", + ) + + self.assertNotEqual(resumed.returncode, 0, field) + self.assertIn("recorded", resumed.stderr) + + def test_weka_runner_rederives_summary_without_rewriting_recorded_evidence(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fixture = root / "fixture" + write_valid_runner_exports( + fixture, + readiness={ + "ready": True, + "was_cancelled": False, + "partial": False, + }, + ) + first, destination = run_weka_fixture( + root, + fixture, + profile_name="resume-summary-immutability", + ) + self.assertEqual(first.returncode, 0, first.stderr) + summary_path = destination / "summary.json" + summary = json.loads(summary_path.read_text(encoding="utf-8")) + summary["forged"] = True + forged_summary = json.dumps(summary, indent=2, sort_keys=True) + "\n" + summary_path.write_text(forged_summary, encoding="utf-8") + provenance_path = destination / "provenance.json" + provenance = json.loads(provenance_path.read_text(encoding="utf-8")) + provenance["artifacts"]["summary.json"] = hashlib.sha256( + summary_path.read_bytes() + ).hexdigest() + provenance_path.write_text( + json.dumps(provenance, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + + resumed, _ = run_weka_fixture( + root, + fixture, + profile_name="resume-summary-immutability", + ) + + self.assertNotEqual(resumed.returncode, 0) + self.assertIn("rederived summary differs", resumed.stderr) + self.assertEqual(summary_path.read_text(encoding="utf-8"), forged_summary) + + def test_weka_runner_requires_exported_engine_metrics(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fixture = root / "fixture" + write_valid_runner_exports( + fixture, + readiness={ + "ready": True, + "was_cancelled": False, + "partial": False, + }, + missing_server_metric="vllm:num_requests_waiting", + ) + result, destination = run_weka_fixture( + root, fixture, profile_name="missing-metric" + ) + self.assertNotEqual(result.returncode, 0, result.stderr) + self.assertTrue((destination / "summary.json").is_file()) + self.assertIn("waiting_requests_max", result.stderr) + + def test_weka_runner_revalidates_an_existing_success_directory(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fixture = root / "fixture" + write_valid_runner_exports( + fixture, + readiness={ + "ready": True, + "was_cancelled": False, + "partial": False, + }, + ) + first, destination = run_weka_fixture( + root, fixture, profile_name="existing" + ) + self.assertEqual(first.returncode, 0, first.stderr) + (destination / ".aiperf_results_ready.json").unlink() + second, _ = run_weka_fixture(root, fixture, profile_name="existing") + self.assertNotEqual(second.returncode, 0, second.stderr) + self.assertIn( + "recorded artifact is missing or unsafe: .aiperf_results_ready.json", + second.stderr, + ) + + def test_weka_runner_rejects_aiperf_cancelled_success(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + runs_root = root / "runs" + run_root = runs_root / "20990101T000000Z" + run_root.mkdir(parents=True) + manifest = run_root / "manifest.env" + manifest.write_text( + "AIPERF_COMMIT=ea28b2e81c7367f8403c6f6ebe0837a508795a8d\n" + "WEKA_REPOSITORY=semianalysisai/cc-traces-weka-062126\n" + "WEKA_REVISION=23f152f6f0f9399a85901b89a6458def0ef16729\n", + encoding="utf-8", + ) + os.chmod(manifest, 0o600) + dataset = root / "traces" + dataset.mkdir() + (dataset / "trace.json").write_text( + '{"id":"fixture","requests":[]}\n', encoding="utf-8" + ) + tokenizer = root / "tokenizer" + tokenizer.mkdir() + key_file = root / "benchmark.key" + key_file.write_text("test-weka-secret\n", encoding="utf-8") + os.chmod(key_file, 0o600) + fake = root / "aiperf" + fake.write_text( + "#!/usr/bin/env bash\n" + "set -euo pipefail\n" + "config=\n" + "while (($#)); do\n" + " if [[ $1 == --config ]]; then config=$2; shift 2; else shift; fi\n" + "done\n" + "printf '%s\\n' '{\"ready\":true,\"was_cancelled\":true,\"partial\":false}' " + ">\"$(dirname \"$config\")/.aiperf_results_ready.json\"\n", + encoding="utf-8", + ) + os.chmod(fake, 0o700) + result = subprocess.run( + [ + "bash", + str(RUNNER), + "--run-root", + str(run_root), + "--profile-name", + "cancelled", + "--repetition", + "1", + "--key-file", + str(key_file), + "--fixed-schedule", + "--internal", + ], + cwd=REPOSITORY_ROOT, + env={ + **os.environ, + "WEBSTER_WEKA_RUNS_ROOT": str(runs_root), + "WEBSTER_WEKA_TEST_MODE": "1", + "WEBSTER_WEKA_AIPERF_BIN": str(fake), + "WEBSTER_WEKA_DATASET": str(dataset), + "WEBSTER_WEKA_TOKENIZER": str(tokenizer), + }, + text=True, + capture_output=True, + check=False, + ) + self.assertNotEqual(result.returncode, 0) + destination = run_root / "aiperf" / "cancelled-r1-fixed" + self.assertEqual((destination / "EXIT").read_text(), "EXIT=130\n") + self.assertIn("AIPerf run was cancelled", result.stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/webster/deepseek-v41/tests/test_checkpoint.py b/webster/deepseek-v41/tests/test_checkpoint.py new file mode 100644 index 0000000..dab44bf --- /dev/null +++ b/webster/deepseek-v41/tests/test_checkpoint.py @@ -0,0 +1,185 @@ +#!/usr/bin/env python3 +"""Behavior tests for immutable checkpoint verification.""" + +from __future__ import annotations + +import json +import subprocess +import tempfile +import unittest +from pathlib import Path + + +REPOSITORY_ROOT = Path(__file__).resolve().parents[3] +VERIFIER = ( + REPOSITORY_ROOT + / "webster" + / "deepseek-v41" + / "scripts" + / "verify-checkpoint.py" +) + + +class CheckpointVerifierTests(unittest.TestCase): + def make_checkpoint(self, root: Path, sizes: tuple[int, ...] = (5, 7)) -> Path: + checkpoint = root / "checkpoint" + checkpoint.mkdir() + weight_map: dict[str, str] = {} + for index, size in enumerate(sizes, start=1): + name = f"model-{index:05d}-of-{len(sizes):05d}.safetensors" + (checkpoint / name).write_bytes(bytes([index]) * size) + weight_map[f"layer.{index}.weight"] = name + (checkpoint / "model.safetensors.index.json").write_text( + json.dumps({"metadata": {"total_size": sum(sizes)}, "weight_map": weight_map}), + encoding="utf-8", + ) + for name in ( + "config.json", + "tokenizer.json", + "tokenizer_config.json", + "encoding/encoding.py", + "encoding/test_encoding.py", + "encoding/tests/test_input_1.json", + "encoding/tests/test_output_1.txt", + "inference/config.json", + ): + path = checkpoint / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(f"fixture:{name}\n", encoding="utf-8") + return checkpoint + + def run_verifier(self, checkpoint: Path, *extra: str) -> subprocess.CompletedProcess[str]: + return subprocess.run( + [ + "python3", + str(VERIFIER), + str(checkpoint), + "--expected-shards", + "2", + "--expected-shard-file-bytes", + "12", + "--expected-tensor-bytes", + "12", + *extra, + ], + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=False, + ) + + def test_valid_checkpoint_emits_sorted_manifest(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + checkpoint = self.make_checkpoint(Path(temporary)) + result = self.run_verifier(checkpoint) + self.assertEqual(result.returncode, 0, result.stderr) + rows = result.stdout.splitlines() + self.assertEqual(rows, sorted(rows)) + self.assertTrue(any(row.endswith("model-00001-of-00002.safetensors") for row in rows)) + self.assertTrue(any(" 5 model-00001-of-00002.safetensors" in row for row in rows)) + + def test_missing_referenced_shard_is_rejected(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + checkpoint = self.make_checkpoint(Path(temporary)) + (checkpoint / "model-00002-of-00002.safetensors").unlink() + result = self.run_verifier(checkpoint) + self.assertNotEqual(result.returncode, 0) + self.assertIn("missing referenced shard", result.stderr) + + def test_weight_byte_mismatch_is_rejected(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + checkpoint = self.make_checkpoint(Path(temporary), sizes=(5, 6)) + result = self.run_verifier(checkpoint) + self.assertNotEqual(result.returncode, 0) + self.assertIn("shard file bytes mismatch", result.stderr) + + def test_index_tensor_byte_mismatch_is_rejected(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + checkpoint = self.make_checkpoint(Path(temporary)) + index_path = checkpoint / "model.safetensors.index.json" + index = json.loads(index_path.read_text(encoding="utf-8")) + index["metadata"]["total_size"] = 11 + index_path.write_text(json.dumps(index), encoding="utf-8") + result = self.run_verifier(checkpoint) + self.assertNotEqual(result.returncode, 0) + self.assertIn("index total_size mismatch", result.stderr) + + def test_duplicate_index_shard_is_rejected(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + checkpoint = self.make_checkpoint(Path(temporary)) + index_path = checkpoint / "model.safetensors.index.json" + index = json.loads(index_path.read_text(encoding="utf-8")) + index["weight_map"]["layer.2.weight"] = "model-00001-of-00002.safetensors" + index_path.write_text(json.dumps(index), encoding="utf-8") + result = self.run_verifier(checkpoint) + self.assertNotEqual(result.returncode, 0) + self.assertIn("shard count mismatch", result.stderr) + + def test_unreferenced_extra_shard_is_rejected(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + checkpoint = self.make_checkpoint(Path(temporary)) + (checkpoint / "unreferenced.safetensors").write_bytes(b"extra") + result = self.run_verifier(checkpoint) + self.assertNotEqual(result.returncode, 0) + self.assertIn("unreferenced safetensors", result.stderr) + + def test_parent_traversal_is_rejected(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + checkpoint = self.make_checkpoint(root) + outside = root / "outside.safetensors" + outside.write_bytes(b"1234567") + index_path = checkpoint / "model.safetensors.index.json" + index = json.loads(index_path.read_text(encoding="utf-8")) + index["weight_map"]["layer.2.weight"] = "../outside.safetensors" + index_path.write_text(json.dumps(index), encoding="utf-8") + result = self.run_verifier(checkpoint) + self.assertNotEqual(result.returncode, 0) + self.assertIn("unsafe checkpoint path", result.stderr) + + def test_symlink_that_escapes_checkpoint_is_rejected(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + checkpoint = self.make_checkpoint(root) + outside = root / "outside.safetensors" + outside.write_bytes(b"1234567") + shard = checkpoint / "model-00002-of-00002.safetensors" + shard.unlink() + shard.symlink_to(outside) + result = self.run_verifier(checkpoint) + self.assertNotEqual(result.returncode, 0) + self.assertIn("escapes checkpoint", result.stderr) + + def test_compare_manifest_rejects_cross_node_mismatch(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + checkpoint = self.make_checkpoint(root) + comparison = root / "other.manifest" + comparison.write_text("different\n", encoding="utf-8") + result = self.run_verifier( + checkpoint, "--compare-manifest", str(comparison) + ) + self.assertNotEqual(result.returncode, 0) + self.assertIn("manifest mismatch", result.stderr) + + def test_stored_manifest_detects_post_promotion_tampering(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + checkpoint = self.make_checkpoint(Path(temporary)) + initial = self.run_verifier(checkpoint) + self.assertEqual(initial.returncode, 0, initial.stderr) + (checkpoint / "MANIFEST.sha256").write_text( + initial.stdout, encoding="utf-8" + ) + verified = self.run_verifier(checkpoint) + self.assertEqual(verified.returncode, 0, verified.stderr) + + (checkpoint / "config.json").write_text( + "tampered\n", encoding="utf-8" + ) + result = self.run_verifier(checkpoint) + self.assertNotEqual(result.returncode, 0) + self.assertIn("stored manifest mismatch", result.stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/webster/deepseek-v41/tests/test_current_model_probes.py b/webster/deepseek-v41/tests/test_current_model_probes.py new file mode 100644 index 0000000..ec6b16d --- /dev/null +++ b/webster/deepseek-v41/tests/test_current_model_probes.py @@ -0,0 +1,94 @@ +#!/usr/bin/env python3 +"""Behavior tests for the scoped current-model baseline probe.""" + +from __future__ import annotations + +import importlib.util +import io +import unittest +import urllib.error +from pathlib import Path + + +PACKAGE_ROOT = Path(__file__).resolve().parents[1] +MODULE_PATH = PACKAGE_ROOT / "scripts" / "current-model-probes.py" + + +def load_module(): + spec = importlib.util.spec_from_file_location("current_model_probes", MODULE_PATH) + if spec is None or spec.loader is None: + raise RuntimeError(f"cannot import {MODULE_PATH}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +class CurrentModelProbeTests(unittest.TestCase): + def test_bedrock_converse_omits_output_limit_while_vllm_keeps_it(self) -> None: + module = load_module() + calls = [] + + def post(url, body, key, timeout): + calls.append((url, body, key, timeout)) + if url.endswith("/key/generate"): + return 200, {"key": "sk-fixture-secret"} + if url.endswith("/key/delete"): + return 200, {} + return 200, {"choices": [{"message": {"content": "OK"}}], "model": body["model"]} + + module.run_scoped_probes( + [ + ("glm-5.2", "hosted_vllm/nvidia/GLM-5.2-NVFP4"), + ("aws/openai-gpt-5.6-sol", "bedrock/converse/us.openai.gpt-5.6-sol"), + ], + "20990101T000000Z", + "master-fixture-secret", + post=post, + ) + + completion_bodies = [ + body for url, body, _, _ in calls if url.endswith("/v1/chat/completions") + ] + self.assertEqual(completion_bodies[0]["max_tokens"], 8) + self.assertNotIn("max_tokens", completion_bodies[1]) + + def test_http_error_names_model_and_status_then_revokes_without_body_or_key(self) -> None: + module = load_module() + calls = [] + + def post(url, body, key, timeout): + calls.append((url, body, key, timeout)) + if url.endswith("/key/generate"): + return 200, {"key": "sk-fixture-secret"} + if url.endswith("/key/delete"): + return 200, {} + raise urllib.error.HTTPError( + url, + 400, + "Bad Request", + {}, + io.BytesIO(b"sensitive provider response"), + ) + + with self.assertRaises(RuntimeError) as caught: + module.run_scoped_probes( + [("aws/xai-grok-4.6", "bedrock/converse/us.xai.grok-4.6")], + "20990101T000000Z", + "master-fixture-secret", + post=post, + ) + + message = str(caught.exception) + self.assertEqual( + message, + "completion probe failed model=aws/xai-grok-4.6 status=400", + ) + self.assertNotIn("sensitive provider response", message) + self.assertNotIn("fixture-secret", message) + delete_calls = [call for call in calls if call[0].endswith("/key/delete")] + self.assertEqual(len(delete_calls), 1) + self.assertEqual(delete_calls[0][1], {"keys": ["sk-fixture-secret"]}) + + +if __name__ == "__main__": + unittest.main() diff --git a/webster/deepseek-v41/tests/test_failure_paths.sh b/webster/deepseek-v41/tests/test_failure_paths.sh new file mode 100755 index 0000000..6d35874 --- /dev/null +++ b/webster/deepseek-v41/tests/test_failure_paths.sh @@ -0,0 +1,1212 @@ +#!/usr/bin/env bash +set -euo pipefail + +if [[ "$(uname -s)" != Linux ]]; then + printf 'skip: Webster cluster shell integration tests require Linux utilities\n' + exit 0 +fi + +cd "$(dirname "$0")/../../.." + +preflight="webster/deepseek-v41/scripts/preflight.sh" +if [[ ! -x "$preflight" ]]; then + echo "FAIL: missing executable $preflight" >&2 + exit 1 +fi +stage_artifacts="webster/deepseek-v41/scripts/stage-artifacts.sh" +if [[ ! -x "$stage_artifacts" ]]; then + echo "FAIL: missing executable $stage_artifacts" >&2 + exit 1 +fi +install_guard="webster/deepseek-v41/scripts/install-glm52-guard.sh" +restore_config="webster/deepseek-v41/scripts/restore-litellm-config.sh" +acceptance="webster/deepseek-v41/scripts/acceptance.sh" +sync_runbook="webster/deepseek-v41/scripts/sync-runbook.sh" +build_runtime="webster/deepseek-v41/scripts/build-runtime.sh" +registry_probe="webster/deepseek-v41/scripts/verify-runtime-registry.py" +lifecycle_scripts=( + webster/deepseek-v41/scripts/stop-glm52-tp2.sh + webster/deepseek-v41/scripts/start-glm52-tp2.sh + webster/deepseek-v41/scripts/stop-deepseek-v41-tp2.sh + webster/deepseek-v41/scripts/start-deepseek-v41-tp2.sh + webster/deepseek-v41/scripts/deepseek-v41-watchdog.sh +) +for required_script in \ + "$install_guard" "$restore_config" "$acceptance" "$sync_runbook" \ + "$build_runtime" "$registry_probe" \ + "${lifecycle_scripts[@]}"; do + if [[ ! -x "$required_script" ]]; then + echo "FAIL: missing executable $required_script" >&2 + exit 1 + fi +done + +scratch="$(mktemp -d)" +trap 'chmod -R u+w "$scratch" 2>/dev/null || true; rm -rf "$scratch"' EXIT +fake_bin="$scratch/bin" +mkdir -p "$fake_bin" +export TEST_LOG="$scratch/invocations.log" +: >"$TEST_LOG" + +for command in ssh docker curl systemctl sha256sum nvidia-smi; do + command_path="$fake_bin/$command" + printf '%s\n' \ + '#!/usr/bin/env bash' \ + 'display=("$@")' \ + 'if [[ "$(basename "$0")" == ssh && "${display[0]:-}" == -o && "${display[1]:-}" == ForwardAgent=no ]]; then display=("${display[@]:2}"); fi' \ + 'printf "%s %s\\n" "$(basename "$0")" "${display[*]}" >>"$TEST_LOG"' \ + 'if [[ "$(basename "$0")" == "ssh" && "$*" == *"df -PB1 /home/alecfong"* && -n "${TEST_SSH_DF_FREE:-}" ]]; then' \ + ' printf "Filesystem 1-blocks Used Available Capacity Mounted\\n/dev/test 1000 1 %s 1%% /home/alecfong\\n" "$TEST_SSH_DF_FREE"' \ + 'fi' \ + 'if [[ "$(basename "$0")" == "ssh" && "$*" == *"df -B1 --output=avail"* && -n "${TEST_SSH_DF_FREE:-}" ]]; then' \ + ' printf "Avail\\n%s\\n" "$TEST_SSH_DF_FREE"' \ + 'fi' \ + 'if [[ "$(basename "$0")" == "ssh" && "$*" == *"stat -c %a:%U:%s"* && -n "${TEST_SSH_CREDENTIAL_METADATA:-}" ]]; then' \ + ' printf "%s\\n" "$TEST_SSH_CREDENTIAL_METADATA"' \ + 'fi' \ + 'if [[ "$(basename "$0")" == "ssh" && "$*" == *".glm52-guard-stage-"*"/glm52_contract_guard.py"* && -n "${TEST_SSH_GUARD_HASH:-}" ]]; then' \ + ' printf "%s\\n" "$TEST_SSH_GUARD_HASH"' \ + 'fi' \ + 'if [[ "$(basename "$0")" == "ssh" && "$*" == *".glm52-guard-stage-"*"/tokenizer"*"sha256sum tokenizer.json"* && -n "${TEST_SSH_TOKENIZER_MANIFEST:-}" ]]; then' \ + ' /bin/cat "$TEST_SSH_TOKENIZER_MANIFEST"' \ + 'fi' \ + 'if [[ "$(basename "$0")" == "ssh" && "${TEST_SSH_HANG:-0}" == "1" ]]; then' \ + ' sleep 30' \ + 'fi' \ + 'exit 0' >"$command_path" + chmod +x "$command_path" +done +printf '%s\n' \ + '#!/usr/bin/env bash' \ + 'display=("$@")' \ + 'if [[ "${display[0]:-}" == -o && "${display[1]:-}" == ForwardAgent=no ]]; then display=("${display[@]:2}"); fi' \ + 'printf "%s %s\n" "$(basename "$0")" "${display[*]}" >>"$TEST_LOG"' \ + '[[ "${TEST_SCP_FAIL:-0}" != "1" ]] || exit 9' \ + 'exit 0' >"$fake_bin/scp" +chmod +x "$fake_bin/scp" +printf '%s\n' \ + '#!/usr/bin/env bash' \ + 'printf "%s %s\\n" "$(basename "$0")" "$*" >>"$TEST_LOG"' \ + 'exec /usr/bin/sha256sum "$@"' >"$fake_bin/sha256sum" +chmod +x "$fake_bin/sha256sum" + +export PATH="$fake_bin:$PATH" +export WEBSTER_PREFLIGHT_TEST_MODE=1 +export WEBSTER_STAGE_TEST_MODE=1 + +expect_failure() { + local name="$1" + shift + if "$@" >"$scratch/$name.stdout" 2>"$scratch/$name.stderr"; then + echo "FAIL: $name unexpectedly succeeded" >&2 + exit 1 + fi +} + +valid_root="/home/ubuntu/deepseek-v41-runs/20990101T000000Z" +publish_root="$scratch/runs/20990101T000001Z" +mkdir -p "$publish_root" +printf '%s\n' '{"decision":"GO"}' >"$publish_root/private-acceptance.json" +chmod 0600 "$publish_root/private-acceptance.json" +publish_verifier="$scratch/verify-private-acceptance" +printf '%s\n' \ + '#!/usr/bin/env bash' \ + 'printf "private-acceptance-verifier %s\n" "$*" >>"$TEST_LOG"' \ + '[[ "${TEST_PRIVATE_ACCEPTANCE_VERIFY_FAIL:-0}" != "1" ]]' \ + >"$publish_verifier" +chmod +x "$publish_verifier" +valid_inspect="$scratch/litellm-inspect-valid.json" +bad_inspect="$scratch/litellm-inspect-bad.json" +printf '%s\n' \ + '[{"Image":"sha256:0000000000000000000000000000000000000000000000000000000000000000","Config":{"Entrypoint":["docker/prod_entrypoint.sh"],"Cmd":["--config","/app/config.yaml","--host","127.0.0.1","--port","4446"]}}]' \ + >"$valid_inspect" +printf '%s\n' \ + '[{"Image":"sha256:0000000000000000000000000000000000000000000000000000000000000000","Config":{"Entrypoint":["docker/prod_entrypoint.sh"],"Cmd":["--config","/app/config.yaml","--host","0.0.0.0","--port","4446"]}}]' \ + >"$bad_inspect" +export TEST_LITELLM_INSPECT="$valid_inspect" + +( + export TEST_RUNS_ROOT="$scratch/runs" + source webster/deepseek-v41/scripts/common.sh + local_root="$TEST_RUNS_ROOT/20990101T000000Z" + init_run_root "$local_root" + printf 'sentinel\n' >>"$local_root/events.tsv" + init_run_root "$local_root" + grep -qx sentinel "$local_root/events.tsv" + [[ "$(stat -c %a "$local_root")" == "700" ]] + [[ "$(stat -c %a "$local_root/manifest.env")" == "600" ]] + capture repeated-evidence /bin/sh -c 'printf "first\n"' + capture repeated-evidence /bin/sh -c 'printf "second\n"' + grep -qx first "$local_root/logs/repeated-evidence.log" + grep -qx second "$local_root/logs/repeated-evidence-2.log" +) + +custom_runs_root="$scratch/custom-runs" +mkdir -p "$custom_runs_root" +( + unset WEBSTER_PREFLIGHT_TEST_MODE WEBSTER_STAGE_TEST_MODE + unset WEBSTER_RUNTIME_TEST_MODE WEBSTER_LIFECYCLE_TEST_MODE + unset WEBSTER_WEKA_TEST_MODE WEBSTER_ACCEPTANCE_TEST_MODE TEST_RUNS_ROOT + export WEBSTER_RUNS_ROOT="$custom_runs_root" + source webster/deepseek-v41/scripts/common.sh + [[ "$RUNS_ROOT" == "$(realpath -e "$custom_runs_root")" ]] +) + +: >"$TEST_LOG" +local_shamu_output="$( + export TEST_RUNS_ROOT="$scratch/runs" + export WEBSTER_LOCAL_SHAMU=1 + export WEBSTER_WEKA_TEST_MODE=1 + source webster/deepseek-v41/scripts/common.sh + ssh shamu /usr/bin/printf '%s' local-rank0 +)" +[[ "$local_shamu_output" == local-rank0 ]] +if grep -q '^ssh ' "$TEST_LOG"; then + echo "FAIL: local Shamu mode still traversed SSH for rank 0" >&2 + exit 1 +fi + +: >"$TEST_LOG" +( + export TEST_RUNS_ROOT="$scratch/runs" + source webster/deepseek-v41/scripts/common.sh + check_glm_health +) +grep -Fxq \ + 'curl -fsS --connect-timeout 3 --max-time 5 http://100.73.140.127:8000/health' \ + "$TEST_LOG" +if grep -q '^ssh ' "$TEST_LOG"; then + echo "FAIL: direct GLM health check unexpectedly used SSH" >&2 + exit 1 +fi + +available_bytes="$( + export TEST_RUNS_ROOT="$scratch/runs" + source webster/deepseek-v41/scripts/common.sh + TEST_SSH_DF_FREE=123456789 remote_available_bytes \ + shamu /home/alecfong/test +)" +[[ "$available_bytes" == "123456789" ]] +grep -Fxq \ + 'ssh shamu df -B1 --output=avail /home/alecfong/test' \ + "$TEST_LOG" + +credential_root="$scratch/credential-evidence" +mkdir -p "$credential_root/logs" +chmod 0700 "$credential_root" "$credential_root/logs" +env WEBSTER_PREFLIGHT_TEST_MODE=0 WEBSTER_STAGE_TEST_MODE=0 \ + TEST_SSH_CREDENTIAL_METADATA=600:nvidia:32 \ + RUN_ROOT="$credential_root" bash -c ' + source webster/deepseek-v41/scripts/common.sh + require_file_mode_600 shamu /home/alecfong/.config/glm52/vllm-api-key + ' + +TEST_FREE_BYTES=805306367999 expect_failure low_free \ + "$preflight" --phase stage --run-root "$valid_root" + +TEST_MISSING_TOPOLOGY=shamu-hca expect_failure missing_hca \ + "$preflight" --phase baseline --run-root "$valid_root" + +expect_failure invalid_root \ + "$preflight" --phase baseline --run-root "$scratch/outside" + +TEST_CREDENTIAL_MODE=0644 expect_failure bad_credential_mode \ + "$preflight" --phase baseline --run-root "$valid_root" + +expect_failure bad_litellm_shape \ + env TEST_LITELLM_INSPECT="$bad_inspect" \ + "$preflight" --phase baseline --run-root "$valid_root" + +expect_failure model_probe_failure \ + env TEST_CURRENT_MODEL_PROBE_STATUS=failed \ + "$preflight" --phase baseline --run-root "$valid_root" + +# Publication is only safe after DeepSeek is the exclusive station owner. This +# catches a preflight that accidentally retains the pre-cutover GLM requirement. +expect_failure publish_wrong_station_owner \ + env TEST_RUNS_ROOT="$scratch/runs" \ + TEST_PRIVATE_ACCEPTANCE_VERIFIER="$publish_verifier" \ + TEST_PREFLIGHT_GLM_SHAMU_RUNNING=true \ + TEST_PREFLIGHT_GLM_TILIKUM_RUNNING=true \ + TEST_PREFLIGHT_DEEPSEEK_SHAMU_RUNNING=false \ + TEST_PREFLIGHT_DEEPSEEK_TILIKUM_RUNNING=false \ + "$preflight" --phase publish --run-root "$publish_root" + +expect_failure publish_invalid_private_acceptance \ + env TEST_RUNS_ROOT="$scratch/runs" \ + TEST_PRIVATE_ACCEPTANCE_VERIFIER="$publish_verifier" \ + TEST_PRIVATE_ACCEPTANCE_VERIFY_FAIL=1 \ + TEST_PREFLIGHT_GLM_SHAMU_RUNNING=false \ + TEST_PREFLIGHT_GLM_TILIKUM_RUNNING=false \ + TEST_PREFLIGHT_DEEPSEEK_SHAMU_RUNNING=true \ + TEST_PREFLIGHT_DEEPSEEK_TILIKUM_RUNNING=true \ + "$preflight" --phase publish --run-root "$publish_root" + +env TEST_RUNS_ROOT="$scratch/runs" \ + TEST_PRIVATE_ACCEPTANCE_VERIFIER="$publish_verifier" \ + TEST_PREFLIGHT_GLM_SHAMU_RUNNING=false \ + TEST_PREFLIGHT_GLM_TILIKUM_RUNNING=false \ + TEST_PREFLIGHT_DEEPSEEK_SHAMU_RUNNING=true \ + TEST_PREFLIGHT_DEEPSEEK_TILIKUM_RUNNING=true \ + "$preflight" --phase publish --run-root "$publish_root" +grep -Fxq \ + "private-acceptance-verifier --run-root $publish_root --acceptance $publish_root/private-acceptance.json --max-age-seconds 21600" \ + "$TEST_LOG" + +expect_failure malformed_station_free \ + env TEST_VALIDATE_STATION_FREE=1 TEST_SSH_DF_FREE=not-an-integer \ + "$preflight" --phase baseline --run-root "$valid_root" + +: >"$TEST_LOG" +"$preflight" --phase baseline --run-root "$valid_root" +cp "$TEST_LOG" "$scratch/first-success.log" +: >"$TEST_LOG" +"$preflight" --phase baseline --run-root "$valid_root" +cmp "$scratch/first-success.log" "$TEST_LOG" + +TEST_FREE_BYTES=805306367999 expect_failure stage_low_free \ + "$stage_artifacts" --node shamu --run-root "$valid_root" + +acceptance_root="$scratch/runs/20990101T000008Z" +WEBSTER_ACCEPTANCE_TEST_MODE=1 TEST_RUNS_ROOT="$scratch/runs" \ + "$acceptance" --run-root "$acceptance_root" +grep -Fq $'\tpublication-acceptance\tGO\t' "$acceptance_root/events.tsv" +expect_failure acceptance_rejects_rank_failure \ + env WEBSTER_ACCEPTANCE_TEST_MODE=1 TEST_RUNS_ROOT="$scratch/runs" \ + TEST_ACCEPTANCE_DEEPSEEK_TILIKUM_RUNNING=false \ + "$acceptance" --run-root "$acceptance_root" +expect_failure acceptance_rejects_unrelated_prometheus_activity \ + env WEBSTER_ACCEPTANCE_TEST_MODE=1 TEST_RUNS_ROOT="$scratch/runs" \ + TEST_ACCEPTANCE_PROMETHEUS_ROUTE=unrelated \ + "$acceptance" --run-root "$acceptance_root" +expect_failure acceptance_rejects_unrelated_langfuse_trace \ + env WEBSTER_ACCEPTANCE_TEST_MODE=1 TEST_RUNS_ROOT="$scratch/runs" \ + TEST_ACCEPTANCE_LANGFUSE_PROBE=unrelated \ + "$acceptance" --run-root "$acceptance_root" +expect_failure acceptance_rejects_baker_semantic_drift \ + env WEBSTER_ACCEPTANCE_TEST_MODE=1 TEST_RUNS_ROOT="$scratch/runs" \ + TEST_ACCEPTANCE_BAKER_EQUAL=drifted \ + "$acceptance" --run-root "$acceptance_root" + +sync_root="$scratch/sync" +webster_sync_source="$sync_root/sources/webster-cluster" +migration_sync_source="$sync_root/sources/migrating-webster-models" +mkdir -p "$webster_sync_source/reference" "$migration_sync_source/references" +printf '%s\n' \ + '---' \ + 'name: webster-cluster' \ + 'description: Webster test source.' \ + '---' \ + 'canonical-webster-source-marker' \ + >"$webster_sync_source/SKILL.md" +printf '%s\n' 'canonical-webster-reference-marker' \ + >"$webster_sync_source/reference/topology.md" +printf '%s\n' \ + '---' \ + 'name: migrating-webster-models' \ + 'description: Migration test source.' \ + '---' \ + 'repository-migration-source-marker' \ + >"$migration_sync_source/SKILL.md" +printf '%s\n' 'repository-migration-reference-marker' \ + >"$migration_sync_source/references/playbook.md" +WEBSTER_SYNC_TEST_MODE=1 \ + WEBSTER_CLUSTER_SOURCE="$webster_sync_source" \ + WEBSTER_MIGRATION_SOURCE="$migration_sync_source" \ + TEST_SYNC_CLAUDE_ROOT="$sync_root/claude" \ + TEST_SYNC_CODEX_ROOT="$sync_root/codex" \ + TEST_SYNC_HERMES_ROOT="$sync_root/hermes" \ + TEST_SYNC_STATE_FILE="$sync_root/state" \ + "$sync_runbook" +for consumer in claude codex hermes; do + test -f "$sync_root/$consumer/webster-cluster/SKILL.md" + test -f "$sync_root/$consumer/migrating-webster-models/SKILL.md" +done +grep -Fq 'canonical-webster-source-marker' \ + "$sync_root/claude/webster-cluster/SKILL.md" +grep -Fq 'repository-migration-source-marker' \ + "$sync_root/codex/migrating-webster-models/SKILL.md" +grep -Fq 'metadata:' "$sync_root/hermes/migrating-webster-models/SKILL.md" +WEBSTER_SYNC_TEST_MODE=1 \ + WEBSTER_CLUSTER_SOURCE="$webster_sync_source" \ + WEBSTER_MIGRATION_SOURCE="$migration_sync_source" \ + TEST_SYNC_CLAUDE_ROOT="$sync_root/claude" \ + TEST_SYNC_CODEX_ROOT="$sync_root/codex" \ + TEST_SYNC_HERMES_ROOT="$sync_root/hermes" \ + TEST_SYNC_STATE_FILE="$sync_root/state" \ + "$sync_runbook" --check + +printf 'consumer-local-edit\n' \ + >>"$sync_root/codex/migrating-webster-models/SKILL.md" +expect_failure sync_refuses_divergent_regular_destination \ + env WEBSTER_SYNC_TEST_MODE=1 \ + WEBSTER_CLUSTER_SOURCE="$webster_sync_source" \ + WEBSTER_MIGRATION_SOURCE="$migration_sync_source" \ + TEST_SYNC_CLAUDE_ROOT="$sync_root/claude" \ + TEST_SYNC_CODEX_ROOT="$sync_root/codex" \ + TEST_SYNC_HERMES_ROOT="$sync_root/hermes" \ + TEST_SYNC_STATE_FILE="$sync_root/state" \ + "$sync_runbook" +grep -Fq 'consumer-local-edit' \ + "$sync_root/codex/migrating-webster-models/SKILL.md" +WEBSTER_SYNC_TEST_MODE=1 \ + WEBSTER_CLUSTER_SOURCE="$webster_sync_source" \ + WEBSTER_MIGRATION_SOURCE="$migration_sync_source" \ + TEST_SYNC_CLAUDE_ROOT="$sync_root/claude" \ + TEST_SYNC_CODEX_ROOT="$sync_root/codex" \ + TEST_SYNC_HERMES_ROOT="$sync_root/hermes" \ + TEST_SYNC_STATE_FILE="$sync_root/state" \ + "$sync_runbook" --force +if grep -Fq 'consumer-local-edit' \ + "$sync_root/codex/migrating-webster-models/SKILL.md"; then + echo "FAIL: --force did not restore the regular destination" >&2 + exit 1 +fi + +printf 'hermes-reference-drift\n' \ + >>"$sync_root/hermes/migrating-webster-models/references/playbook.md" +expect_failure sync_check_detects_exact_hermes_drift \ + env WEBSTER_SYNC_TEST_MODE=1 \ + WEBSTER_CLUSTER_SOURCE="$webster_sync_source" \ + WEBSTER_MIGRATION_SOURCE="$migration_sync_source" \ + TEST_SYNC_CLAUDE_ROOT="$sync_root/claude" \ + TEST_SYNC_CODEX_ROOT="$sync_root/codex" \ + TEST_SYNC_HERMES_ROOT="$sync_root/hermes" \ + TEST_SYNC_STATE_FILE="$sync_root/state" \ + "$sync_runbook" --check +expect_failure sync_refuses_divergent_hermes_destination \ + env WEBSTER_SYNC_TEST_MODE=1 \ + WEBSTER_CLUSTER_SOURCE="$webster_sync_source" \ + WEBSTER_MIGRATION_SOURCE="$migration_sync_source" \ + TEST_SYNC_CLAUDE_ROOT="$sync_root/claude" \ + TEST_SYNC_CODEX_ROOT="$sync_root/codex" \ + TEST_SYNC_HERMES_ROOT="$sync_root/hermes" \ + TEST_SYNC_STATE_FILE="$sync_root/state" \ + "$sync_runbook" +WEBSTER_SYNC_TEST_MODE=1 \ + WEBSTER_CLUSTER_SOURCE="$webster_sync_source" \ + WEBSTER_MIGRATION_SOURCE="$migration_sync_source" \ + TEST_SYNC_CLAUDE_ROOT="$sync_root/claude" \ + TEST_SYNC_CODEX_ROOT="$sync_root/codex" \ + TEST_SYNC_HERMES_ROOT="$sync_root/hermes" \ + TEST_SYNC_STATE_FILE="$sync_root/state" \ + "$sync_runbook" --force +WEBSTER_SYNC_TEST_MODE=1 \ + WEBSTER_CLUSTER_SOURCE="$webster_sync_source" \ + WEBSTER_MIGRATION_SOURCE="$migration_sync_source" \ + TEST_SYNC_CLAUDE_ROOT="$sync_root/claude" \ + TEST_SYNC_CODEX_ROOT="$sync_root/codex" \ + TEST_SYNC_HERMES_ROOT="$sync_root/hermes" \ + TEST_SYNC_STATE_FILE="$sync_root/state" \ + "$sync_runbook" --check + +# Production's canonical Webster source is also the Claude destination. The +# synchronizer must treat that tree as authoritative and never rsync it onto itself. +WEBSTER_SYNC_TEST_MODE=1 \ + WEBSTER_CLUSTER_SOURCE="$webster_sync_source" \ + WEBSTER_MIGRATION_SOURCE="$migration_sync_source" \ + TEST_SYNC_CLAUDE_ROOT="$sync_root/sources" \ + TEST_SYNC_CODEX_ROOT="$sync_root/self-source-codex" \ + TEST_SYNC_HERMES_ROOT="$sync_root/self-source-hermes" \ + TEST_SYNC_STATE_FILE="$sync_root/self-source-state" \ + "$sync_runbook" +grep -Fq 'canonical-webster-source-marker' "$webster_sync_source/SKILL.md" +grep -Fq 'repository-migration-source-marker' "$migration_sync_source/SKILL.md" + +expect_failure stage_invalid_node \ + "$stage_artifacts" --node baker-spark-1 --run-root "$valid_root" + +stage_timeout_started="$(date +%s)" +expect_failure stage_ssh_probe_timeout \ + timeout 5 env TEST_SSH_HANG=1 STAGE_SSH_TIMEOUT_SECONDS=1 \ + "$stage_artifacts" --node shamu --run-root "$valid_root" +stage_timeout_elapsed=$(( $(date +%s) - stage_timeout_started )) +(( stage_timeout_elapsed < 5 )) || { + echo "FAIL: staging quick SSH probe was not bounded" >&2 + exit 1 +} + +TEST_STAGE_FINAL_STATE=divergent expect_failure stage_divergent_final \ + "$stage_artifacts" --node shamu --run-root "$valid_root" + +TEST_STAGE_BASELINE_P90=10 \ +TEST_STAGE_HEALTH_WINDOWS=$'12 0.98 5\n9 0.97 4' \ + expect_failure stage_two_bad_windows \ + "$stage_artifacts" --node shamu --run-root "$valid_root" + +TEST_STAGE_BASELINE_P90=10 \ +TEST_STAGE_HEALTH_WINDOWS=$'12 1.0 21\n9 1.0 25' \ + expect_failure stage_two_slow_windows \ + "$stage_artifacts" --node shamu --run-root "$valid_root" + +: >"$TEST_LOG" +TEST_STAGE_BASELINE_P90=10 \ +TEST_STAGE_HEALTH_WINDOWS=$'12 0.98 5\n12 1.0 6' \ + "$stage_artifacts" --node shamu --run-root "$valid_root" + +: >"$TEST_LOG" +"$stage_artifacts" --node shamu --run-root "$valid_root" +cp "$TEST_LOG" "$scratch/first-stage.log" +: >"$TEST_LOG" +"$stage_artifacts" --node shamu --run-root "$valid_root" +cmp "$scratch/first-stage.log" "$TEST_LOG" + +test_runs_root="$scratch/runs" +test_run_root="$test_runs_root/20990101T000001Z" +( + export WEBSTER_PREFLIGHT_TEST_MODE=1 + export TEST_RUNS_ROOT="$test_runs_root" + source webster/deepseek-v41/scripts/common.sh + init_run_root "$test_run_root" +) + +tokenizer_source="$scratch/tokenizer" +mkdir -p "$tokenizer_source" +for name in tokenizer.json tokenizer_config.json chat_template.jinja; do + printf 'fixture-%s\n' "$name" >"$tokenizer_source/$name" +done +candidate_config="$scratch/candidate.yaml" +printf 'model_list: []\n' >"$candidate_config" +chmod 0600 "$candidate_config" +test_litellm_root="$scratch/litellm" + +: >"$TEST_LOG" +expect_failure install_transfer_cleanup \ + env TEST_SCP_FAIL=1 TEST_RUNS_ROOT="$test_runs_root" \ + TEST_TOKENIZER_SOURCE="$tokenizer_source" \ + "$install_guard" --candidate --run-root "$test_run_root" \ + --candidate-config "$candidate_config" +if ! grep -F 'for path in '\''/home/nvidia/litellm/energy-pricing/.glm52-guard-stage-' \ + "$TEST_LOG" | + grep -F "' '/home/nvidia/litellm/energy-pricing/.glm52-guard-offline-" >/dev/null || + ! grep -F 'rm -rf -- "$resolved"' "$TEST_LOG" >/dev/null; then + echo "FAIL: transfer failure did not clean both remote candidate trees" >&2 + sed -n '1,20p' "$TEST_LOG" >&2 + exit 1 +fi + +guard_hash="$(sha256sum webster/deepseek-v41/litellm/glm52_contract_guard.py | awk '{print $1}')" +tokenizer_manifest="$scratch/tokenizer.manifest" +( + cd "$tokenizer_source" + sha256sum tokenizer.json tokenizer_config.json chat_template.jinja | sort +) >"$tokenizer_manifest" +: >"$TEST_LOG" +TEST_SSH_GUARD_HASH="$guard_hash" \ +TEST_SSH_TOKENIZER_MANIFEST="$tokenizer_manifest" \ +TEST_RUNS_ROOT="$test_runs_root" \ +TEST_TOKENIZER_SOURCE="$tokenizer_source" \ + "$install_guard" --candidate --run-root "$test_run_root" \ + --candidate-config "$candidate_config" +grep -Fq \ + "verify-litellm-callbacks.py spark-1:/home/nvidia/litellm/energy-pricing/.glm52-guard-offline-" \ + "$TEST_LOG" +grep -Fq -- "-v '/home/nvidia/litellm/energy-pricing:/app/custom_callbacks:ro'" \ + "$TEST_LOG" +grep -Fq -- '-w /app' "$TEST_LOG" +grep -Fq '/work/verify-litellm-callbacks.py /work/config.yaml' "$TEST_LOG" +if grep -Fq 'docker restart litellm' "$TEST_LOG"; then + echo "FAIL: guard installer restarted LiteLLM" >&2 + exit 1 +fi + +install_test_env=( + env + WEBSTER_LITELLM_TEST_MODE=1 + TEST_RUNS_ROOT="$test_runs_root" + TEST_LITELLM_ROOT="$test_litellm_root" + TEST_TOKENIZER_SOURCE="$tokenizer_source" +) +"${install_test_env[@]}" "$install_guard" --candidate \ + --run-root "$test_run_root" --candidate-config "$candidate_config" +first_module_hash="$(sha256sum "$test_litellm_root/glm52_contract_guard.py")" +"${install_test_env[@]}" "$install_guard" --candidate \ + --run-root "$test_run_root" --candidate-config "$candidate_config" +[[ "$(sha256sum "$test_litellm_root/glm52_contract_guard.py")" == "$first_module_hash" ]] +if find "$test_litellm_root" -maxdepth 1 -name 'glm52_contract_guard.py.bak-*' | grep -q .; then + echo "FAIL: identical guard reinstall created a backup" >&2 + exit 1 +fi + +chmod 0666 "$test_litellm_root/glm52_contract_guard.py" +chmod 0755 "$test_litellm_root/glm52-tokenizer" +chmod 0644 "$test_litellm_root/glm52-tokenizer/"* +"${install_test_env[@]}" "$install_guard" --candidate \ + --run-root "$test_run_root" --candidate-config "$candidate_config" +[[ "$(stat -c %a "$test_litellm_root/glm52_contract_guard.py")" == "644" ]] +[[ "$(stat -c %a "$test_litellm_root/glm52-tokenizer")" == "555" ]] +for tokenizer_file in "$test_litellm_root/glm52-tokenizer/"*; do + [[ "$(stat -c %a "$tokenizer_file")" == "444" ]] +done + +printf 'divergent\n' >"$test_litellm_root/glm52_contract_guard.py" +"${install_test_env[@]}" "$install_guard" --candidate \ + --run-root "$test_run_root" --candidate-config "$candidate_config" +find "$test_litellm_root" -maxdepth 1 -name 'glm52_contract_guard.py.bak-*' | grep -q . +[[ "$(sha256sum "$test_litellm_root/glm52_contract_guard.py")" == "$first_module_hash" ]] + +mv "$tokenizer_source/chat_template.jinja" "$scratch/missing-chat-template" +expect_failure install_missing_tokenizer \ + "${install_test_env[@]}" "$install_guard" --candidate \ + --run-root "$test_run_root" --candidate-config "$candidate_config" +mv "$scratch/missing-chat-template" "$tokenizer_source/chat_template.jinja" + +rollback_backup="$test_litellm_root/config.yaml.bak-alias" +printf 'restored: true\n' >"$rollback_backup" +chmod 0600 "$rollback_backup" +printf 'restored: false\n' >"$test_litellm_root/config.yaml" +chmod 0600 "$test_litellm_root/config.yaml" +rollback_hash="$(sha256sum "$rollback_backup" | awk '{print $1}')" +{ + printf 'ALIAS_CONFIG_BACKUP=/home/nvidia/litellm/config.yaml.bak-alias\n' + printf 'ALIAS_CONFIG_SHA256=%s\n' "$rollback_hash" +} >"$test_run_root/rollback.env" +chmod 0600 "$test_run_root/rollback.env" +: >"$TEST_LOG" +restore_test_env=( + env + WEBSTER_LITELLM_TEST_MODE=1 + TEST_RUNS_ROOT="$test_runs_root" + TEST_LITELLM_ROOT="$test_litellm_root" +) +"${restore_test_env[@]}" "$restore_config" --phase alias --run-root "$test_run_root" +grep -qx 'restored: true' "$test_litellm_root/config.yaml" +grep -Fq $'\talias-rollback\tGO\t' "$test_run_root/events.tsv" +first_restart_count="$(grep -c '^docker restart litellm$' "$TEST_LOG" || true)" +"${restore_test_env[@]}" "$restore_config" --phase alias --run-root "$test_run_root" +second_restart_count="$(grep -c '^docker restart litellm$' "$TEST_LOG" || true)" +[[ "$first_restart_count" == "1" && "$second_restart_count" == "1" ]] + +chmod 0644 "$test_litellm_root/config.yaml" +"${restore_test_env[@]}" "$restore_config" --phase alias --run-root "$test_run_root" +[[ "$(stat -c %a "$test_litellm_root/config.yaml")" == "600" ]] +third_restart_count="$(grep -c '^docker restart litellm$' "$TEST_LOG" || true)" +[[ "$third_restart_count" == "1" ]] + +expect_failure restore_not_ready \ + env WEBSTER_LITELLM_TEST_MODE=1 TEST_RUNS_ROOT="$test_runs_root" \ + TEST_LITELLM_ROOT="$test_litellm_root" TEST_RESTORE_READINESS=failed \ + "$restore_config" --phase alias --run-root "$test_run_root" + +sed -i 's#^ALIAS_CONFIG_BACKUP=.*#ALIAS_CONFIG_BACKUP=/tmp/outside.yaml#' \ + "$test_run_root/rollback.env" +expect_failure restore_outside_path \ + "${restore_test_env[@]}" "$restore_config" --phase alias --run-root "$test_run_root" +sed -i 's#^ALIAS_CONFIG_BACKUP=.*#ALIAS_CONFIG_BACKUP=/home/nvidia/litellm/config.yaml.bak-alias#' \ + "$test_run_root/rollback.env" +sed -i 's/^ALIAS_CONFIG_SHA256=.*/ALIAS_CONFIG_SHA256=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff/' \ + "$test_run_root/rollback.env" +expect_failure restore_hash_mismatch \ + "${restore_test_env[@]}" "$restore_config" --phase alias --run-root "$test_run_root" + +runtime_runs_root="$scratch/runtime-runs" +runtime_run_root="$runtime_runs_root/20990101T000002Z" +registry_fixture="$scratch/registry-fixture/vllm/model_executor/models" +mkdir -p "$registry_fixture" +printf '%s\n' \ + 'MODEL_REGISTRY = {' \ + ' "DeepseekV41ForCausalLM": ("deepseek_v4_1", "DeepseekV41ForCausalLM"),' \ + '}' >"$registry_fixture/registry.py" +"$registry_probe" --package-root "$scratch/registry-fixture/vllm" \ + --require-model DeepseekV41ForCausalLM \ + >"$scratch/registry-probe.stdout" +grep -Fq 'DeepseekV41ForCausalLM' "$scratch/registry-probe.stdout" +expect_failure runtime_registry_missing_model \ + "$registry_probe" --package-root "$scratch/registry-fixture/vllm" \ + --require-model MissingForCausalLM +runtime_source="$scratch/vllm-source" +mkdir -p "$runtime_source" +git -C "$runtime_source" init -q +git -C "$runtime_source" config user.name test +git -C "$runtime_source" config user.email test@example.invalid +printf 'fixture\n' >"$runtime_source/source.txt" +git -C "$runtime_source" add source.txt +git -C "$runtime_source" commit -qm fixture +runtime_commit="$(git -C "$runtime_source" rev-parse HEAD)" +git -C "$runtime_source" tag v0.1.0 +printf 'second fixture\n' >>"$runtime_source/source.txt" +git -C "$runtime_source" commit -qam second-fixture +runtime_commit="$(git -C "$runtime_source" rev-parse HEAD)" +git -C "$runtime_source" checkout -q --detach "$runtime_commit" +runtime_bundle="$scratch/runtime-source.bundle" +( + export WEBSTER_RUNTIME_TEST_MODE=1 + export TEST_RUNS_ROOT="$runtime_runs_root" + source webster/deepseek-v41/scripts/common.sh + create_tagged_git_bundle "$runtime_source" "$runtime_bundle" "$runtime_commit" + mkdir "$scratch/non-git-caller" + cd "$scratch/non-git-caller" + create_tagged_git_bundle "$runtime_source" "$runtime_bundle" "$runtime_commit" +) +runtime_bundle_clone="$scratch/runtime-bundle-clone" +git -c advice.detachedHead=false clone -q "$runtime_bundle" "$runtime_bundle_clone" +[[ "$(git -C "$runtime_bundle_clone" describe --tags)" == v0.1.0-1-g* ]] +( + export WEBSTER_RUNTIME_TEST_MODE=1 + export TEST_RUNS_ROOT="$runtime_runs_root" + source webster/deepseek-v41/scripts/common.sh + init_run_root "$runtime_run_root" +) +sed -i "s/^VLLM_COMMIT=.*/VLLM_COMMIT=$runtime_commit/" \ + "$runtime_run_root/manifest.env" + +runtime_test_env=( + env + WEBSTER_RUNTIME_TEST_MODE=1 + TEST_RUNS_ROOT="$runtime_runs_root" + TEST_RUNTIME_REMOTE_ARCH=arm64 + TEST_RUNTIME_BUILD_BASE_PLATFORM=linux/arm64 + TEST_RUNTIME_FINAL_BASE_PLATFORM=linux/arm64 + TEST_RUNTIME_TRANSFER_SHA_MATCH=1 + TEST_RUNTIME_IMAGE_IDS=$'sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\nsha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' + TEST_RUNTIME_INTERNAL_COMMITS=$'REPLACE_RUNTIME_COMMIT\nREPLACE_RUNTIME_COMMIT' + TEST_RUNTIME_HELP_TEXT=--engram-config + TEST_RUNTIME_REGISTRY_TEXT=DeepseekV41ForCausalLM + TEST_RUNTIME_ENGRAM_LAYER_IDS=1,2 + TEST_RUNTIME_CGROUP_PARENT=/deepseek.slice/deepseek-v41.slice/build.slice +) +runtime_test_env=("${runtime_test_env[@]//REPLACE_RUNTIME_COMMIT/$runtime_commit}") + +chmod 0644 "$runtime_run_root/manifest.env" +expect_failure runtime_manifest_mode \ + "${runtime_test_env[@]}" "$build_runtime" --run-root "$runtime_run_root" \ + --source-dir "$runtime_source" +chmod 0600 "$runtime_run_root/manifest.env" + +expect_failure runtime_wrong_arch \ + env "${runtime_test_env[@]:1}" TEST_RUNTIME_REMOTE_ARCH=amd64 \ + "$build_runtime" --run-root "$runtime_run_root" --source-dir "$runtime_source" + +printf 'dirty\n' >>"$runtime_source/source.txt" +expect_failure runtime_dirty_source \ + "${runtime_test_env[@]}" "$build_runtime" --run-root "$runtime_run_root" \ + --source-dir "$runtime_source" +git -C "$runtime_source" checkout -q -- source.txt + +expect_failure runtime_wrong_base_platform \ + env "${runtime_test_env[@]:1}" TEST_RUNTIME_BUILD_BASE_PLATFORM=linux/amd64 \ + "$build_runtime" --run-root "$runtime_run_root" --source-dir "$runtime_source" + +expect_failure runtime_transfer_mismatch \ + env "${runtime_test_env[@]:1}" TEST_RUNTIME_TRANSFER_SHA_MATCH=0 \ + "$build_runtime" --run-root "$runtime_run_root" --source-dir "$runtime_source" + +expect_failure runtime_image_mismatch \ + env "${runtime_test_env[@]:1}" \ + TEST_RUNTIME_IMAGE_IDS=$'sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\nsha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb' \ + "$build_runtime" --run-root "$runtime_run_root" --source-dir "$runtime_source" + +expect_failure runtime_missing_engram_flag \ + env "${runtime_test_env[@]:1}" TEST_RUNTIME_HELP_TEXT='usage: vllm serve' \ + "$build_runtime" --run-root "$runtime_run_root" --source-dir "$runtime_source" + +expect_failure runtime_help_probe_gpu_visible \ + env "${runtime_test_env[@]:1}" TEST_RUNTIME_HELP_VISIBLE_DEVICES=all \ + "$build_runtime" --run-root "$runtime_run_root" --source-dir "$runtime_source" + +expect_failure runtime_help_probe_cuda_target \ + env "${runtime_test_env[@]:1}" TEST_RUNTIME_HELP_TARGET_DEVICE=cuda \ + "$build_runtime" --run-root "$runtime_run_root" --source-dir "$runtime_source" + +expect_failure runtime_help_probe_default_selector \ + env "${runtime_test_env[@]:1}" TEST_RUNTIME_HELP_SELECTOR=default \ + "$build_runtime" --run-root "$runtime_run_root" --source-dir "$runtime_source" + +expect_failure runtime_relative_cgroup \ + env "${runtime_test_env[@]:1}" TEST_RUNTIME_CGROUP_PARENT=deepseek-v41-build.slice \ + "$build_runtime" --run-root "$runtime_run_root" --source-dir "$runtime_source" + +expect_failure runtime_wheel_ceiling_above_upstream_quota \ + env "${runtime_test_env[@]:1}" TEST_RUNTIME_MAX_WHEEL_SIZE_MB=801 \ + "$build_runtime" --run-root "$runtime_run_root" --source-dir "$runtime_source" + +: >"$TEST_LOG" +"${runtime_test_env[@]}" "$build_runtime" --run-root "$runtime_run_root" \ + --source-dir "$runtime_source" +wheel_manifest_entry="$(grep '^VLLM_MAX_WHEEL_SIZE_MB=' "$runtime_run_root/manifest.env")" +[[ "$wheel_manifest_entry" == "VLLM_MAX_WHEEL_SIZE_MB=700" ]] +first_runtime_manifest="$(sha256sum "$runtime_run_root/manifest.env")" +"${runtime_test_env[@]}" "$build_runtime" --run-root "$runtime_run_root" \ + --source-dir "$runtime_source" +[[ "$(sha256sum "$runtime_run_root/manifest.env")" == "$first_runtime_manifest" ]] + +lifecycle_runs_root="$scratch/lifecycle-runs" +lifecycle_run_root="$lifecycle_runs_root/20990101T000003Z" +( + export WEBSTER_PREFLIGHT_TEST_MODE=1 + export TEST_RUNS_ROOT="$lifecycle_runs_root" + source webster/deepseek-v41/scripts/common.sh + init_run_root "$lifecycle_run_root" +) +sed -i \ + 's#^VLLM_IMAGE_ID=.*#VLLM_IMAGE_ID=sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa#' \ + "$lifecycle_run_root/manifest.env" + +lifecycle_test_env=( + env + WEBSTER_LIFECYCLE_TEST_MODE=1 + TEST_RUNS_ROOT="$lifecycle_runs_root" + TEST_LIFECYCLE_GLM_SHAMU_RUNNING=false + TEST_LIFECYCLE_GLM_TILIKUM_RUNNING=false + TEST_LIFECYCLE_DEEPSEEK_SHAMU_RUNNING=false + TEST_LIFECYCLE_DEEPSEEK_TILIKUM_RUNNING=false + TEST_LIFECYCLE_CHECKPOINTS_MATCH=1 + TEST_LIFECYCLE_IMAGES_MATCH=1 + TEST_LIFECYCLE_FREE_BYTES=214748364800 + TEST_LIFECYCLE_RAILS_OK=1 + TEST_LIFECYCLE_SSH_OK=1 + TEST_LIFECYCLE_AUTH_OK=1 + TEST_LIFECYCLE_LISTENERS_OK=1 + TEST_LIFECYCLE_RANK1_KEYLESS=1 + TEST_LIFECYCLE_NCCL_OK=1 +) + +stop_glm="${lifecycle_scripts[0]}" +start_glm="${lifecycle_scripts[1]}" +stop_deepseek="${lifecycle_scripts[2]}" +start_deepseek="${lifecycle_scripts[3]}" +watchdog="${lifecycle_scripts[4]}" + +: >"$TEST_LOG" +"${lifecycle_test_env[@]}" "$stop_glm" --run-root "$lifecycle_run_root" +grep -Fq 'ssh shamu docker stop --time 120 glm52-full-mtp' "$TEST_LOG" +grep -Fq 'ssh tilikum sudo -n docker stop --time 120 glm52-full-mtp' "$TEST_LOG" + +redacted_inspect="$( + export TEST_RUNS_ROOT="$scratch/runs" + source webster/deepseek-v41/scripts/common.sh + printf '%s\n' \ + '{"Config":{"Env":["VLLM_API_KEY=singular-secret","DSPARK_API_KEYS=plural-secret"]}}' | + redact_stream +)" +[[ "$redacted_inspect" != *singular-secret* ]] +if [[ "$redacted_inspect" == *plural-secret* ]]; then + echo "FAIL: Docker inspect evidence leaked a plural API key field" >&2 + exit 1 +fi + +: >"$TEST_LOG" +expect_failure glm_partial_stop \ + env "${lifecycle_test_env[@]:1}" TEST_LIFECYCLE_STOP_FAIL_NODE=tilikum \ + "$stop_glm" --run-root "$lifecycle_run_root" +[[ "$(grep -Fc 'ssh shamu docker stop --time 120 glm52-full-mtp' "$TEST_LOG")" -ge 2 ]] +[[ "$(grep -Fc 'ssh tilikum sudo -n docker stop --time 120 glm52-full-mtp' "$TEST_LOG")" -ge 2 ]] + +: >"$TEST_LOG" +"${lifecycle_test_env[@]}" "$start_glm" --run-root "$lifecycle_run_root" +grep -Fq 'GLM_NODE_RANK=0 GLM_HOST_IP=10.10.1.1 GLM_MAX_MODEL_LEN=320000 GLM_MAX_NUM_SEQS=8' "$TEST_LOG" +grep -Fq 'GLM_NODE_RANK=1 GLM_HOST_IP=10.10.1.2 GLM_MAX_MODEL_LEN=320000 GLM_MAX_NUM_SEQS=8' "$TEST_LOG" +grep -Fq "GLM_DOCKER='sudo -n docker'" "$TEST_LOG" + +: >"$TEST_LOG" +env "${lifecycle_test_env[@]:1}" \ + TEST_LIFECYCLE_GLM_SHAMU_RUNNING=true \ + TEST_LIFECYCLE_GLM_TILIKUM_RUNNING=true \ + "$start_glm" --run-root "$lifecycle_run_root" +if grep -Fq '/home/alecfong/serve_glm52.sh' "$TEST_LOG"; then + echo "FAIL: healthy GLM re-run relaunched ranks" >&2 + exit 1 +fi +: >"$TEST_LOG" +expect_failure glm_existing_wrong_profile \ + env "${lifecycle_test_env[@]:1}" \ + TEST_LIFECYCLE_GLM_SHAMU_RUNNING=true \ + TEST_LIFECYCLE_GLM_TILIKUM_RUNNING=true \ + TEST_LIFECYCLE_GLM_PROFILE_MATCH=0 \ + "$start_glm" --run-root "$lifecycle_run_root" +if grep -Eq 'serve_glm52\.sh|docker stop' "$TEST_LOG"; then + echo "FAIL: mismatched running GLM ranks were mutated before rejection" >&2 + exit 1 +fi +: >"$TEST_LOG" +expect_failure glm_existing_split_generation \ + env "${lifecycle_test_env[@]:1}" \ + TEST_LIFECYCLE_GLM_SHAMU_RUNNING=true \ + TEST_LIFECYCLE_GLM_TILIKUM_RUNNING=true \ + TEST_LIFECYCLE_GLM_GENERATION_MATCH=0 \ + "$start_glm" --run-root "$lifecycle_run_root" +if grep -Eq 'serve_glm52\.sh|docker stop' "$TEST_LOG"; then + echo "FAIL: split-generation GLM ranks were mutated before rejection" >&2 + exit 1 +fi + +: >"$TEST_LOG" +"${lifecycle_test_env[@]}" "$start_deepseek" --run-root "$lifecycle_run_root" +prior_shamu_line="$(grep -n 'ssh shamu .*docker inspect.*deepseek-v41-flash-tp2.*docker logs' "$TEST_LOG" | head -1 | cut -d: -f1)" +prior_tilikum_line="$(grep -n 'ssh tilikum .*docker inspect.*deepseek-v41-flash-tp2.*docker logs' "$TEST_LOG" | head -1 | cut -d: -f1)" +rank0_launch_line="$(grep -n 'ssh shamu .*docker rm -f deepseek-v41-flash-tp2' "$TEST_LOG" | head -1 | cut -d: -f1)" +rank1_launch_line="$(grep -n 'ssh tilikum .*docker rm -f deepseek-v41-flash-tp2' "$TEST_LOG" | head -1 | cut -d: -f1)" +[[ -n "$prior_shamu_line" && "$prior_shamu_line" -lt "$rank0_launch_line" ]] +[[ -n "$prior_tilikum_line" && "$prior_tilikum_line" -lt "$rank1_launch_line" ]] +grep -Fq -- '--tensor-parallel-size 2 --pipeline-parallel-size 1' "$TEST_LOG" +grep -Fq -- '--engram-config {"cpu_offload":true,"embedding_across_dp":false}' "$TEST_LOG" +grep -Fq -- '--enforce-eager --max-model-len 131072 --max-num-seqs 1' "$TEST_LOG" +grep -Fq -- '--host 100.73.140.127 --port 8000' "$TEST_LOG" +grep -Fq -- '--headless' "$TEST_LOG" +rank0_launch="$(grep 'ssh shamu .*deepseek-v41-flash-tp2' "$TEST_LOG")" +rank1_launch="$(grep 'ssh tilikum .*deepseek-v41-flash-tp2' "$TEST_LOG")" +[[ "$rank1_launch" != *'--env-file'* ]] +[[ "$rank1_launch" != *'VLLM_API_KEY='* ]] +rank0_generation="$(sed -n 's/.*--label ai\.webster\.generation=\([^ ]*\).*/\1/p' <<<"$rank0_launch")" +rank1_generation="$(sed -n 's/.*--label ai\.webster\.generation=\([^ ]*\).*/\1/p' <<<"$rank1_launch")" +[[ -n "$rank0_generation" && "$rank0_generation" == "$rank1_generation" ]] +grep -Eq -- '--label ai\.webster\.profile-sha256=[0-9a-f]{64}' <<<"$rank0_launch" +grep -Eq -- '--label ai\.webster\.profile-sha256=[0-9a-f]{64}' <<<"$rank1_launch" + +profile_file="$lifecycle_run_root/profile-graphs-320k.env" +printf '%s\n' \ + 'PROFILE_NAME=graphs-320k' \ + 'MAX_MODEL_LEN=320000' \ + 'MAX_NUM_SEQS=2' \ + 'GPU_MEMORY_UTILIZATION=0.90' \ + 'ENFORCE_EAGER=0' \ + 'ENABLE_PREFIX_CACHING=1' \ + 'MAX_NUM_BATCHED_TOKENS=8192' \ + 'ENABLE_EXPERT_PARALLEL=0' \ + >"$profile_file" +chmod 0600 "$profile_file" +: >"$TEST_LOG" +"${lifecycle_test_env[@]}" "$start_deepseek" \ + --run-root "$lifecycle_run_root" --profile-file "$profile_file" +grep -Fq $'\tstart-deepseek-v41\tSTART\tprofile graphs-320k TP2/PP1 canary' \ + "$lifecycle_run_root/events.tsv" +grep -Fq $'\tstart-deepseek-v41\tGO\tprofile graphs-320k authenticated; rank 1 headless/keyless; NCCL HCA pinned' \ + "$lifecycle_run_root/events.tsv" +grep -Fq -- '--max-model-len 320000 --max-num-seqs 2' "$TEST_LOG" +grep -Fq -- '--gpu-memory-utilization 0.90' "$TEST_LOG" +grep -Fq -- '--enable-prefix-caching' "$TEST_LOG" +grep -Fq -- '--max-num-batched-tokens 8192' "$TEST_LOG" +if grep -Fq -- '--enforce-eager' "$TEST_LOG"; then + echo "FAIL: graph profile retained enforce-eager" >&2 + exit 1 +fi + +dspark_profile_file="$lifecycle_run_root/profile-dspark-128k.env" +printf '%s\n' \ + 'PROFILE_NAME=dspark-128k' \ + 'MAX_MODEL_LEN=131072' \ + 'MAX_NUM_SEQS=1' \ + 'GPU_MEMORY_UTILIZATION=0.90' \ + 'ENFORCE_EAGER=0' \ + 'ENABLE_PREFIX_CACHING=1' \ + 'MAX_NUM_BATCHED_TOKENS=16384' \ + 'ENABLE_EXPERT_PARALLEL=0' \ + 'SPECULATIVE_METHOD=dspark' \ + 'NUM_SPECULATIVE_TOKENS=5' \ + 'DRAFT_SAMPLE_METHOD=greedy' \ + 'REJECTION_SAMPLE_METHOD=standard' \ + 'ENABLE_FLASHINFER_AUTOTUNE=0' \ + >"$dspark_profile_file" +chmod 0600 "$dspark_profile_file" +: >"$TEST_LOG" +"${lifecycle_test_env[@]}" "$start_deepseek" \ + --run-root "$lifecycle_run_root" --profile-file "$dspark_profile_file" +grep -Fq -- \ + "--speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":5,\"draft_sample_method\":\"greedy\",\"rejection_sample_method\":\"standard\"}'" \ + "$TEST_LOG" +grep -Fq -- '--no-enable-flashinfer-autotune' "$TEST_LOG" +grep -Fq $'\tstart-deepseek-v41\tGO\tprofile dspark-128k authenticated; rank 1 headless/keyless; NCCL HCA pinned' \ + "$lifecycle_run_root/events.tsv" +invalid_autotune_profile="$lifecycle_run_root/profile-dspark-invalid-autotune.env" +sed 's/^ENABLE_FLASHINFER_AUTOTUNE=0$/ENABLE_FLASHINFER_AUTOTUNE=2/' \ + "$dspark_profile_file" >"$invalid_autotune_profile" +chmod 0600 "$invalid_autotune_profile" +expect_failure deepseek_dspark_invalid_autotune \ + "${lifecycle_test_env[@]}" "$start_deepseek" \ + --run-root "$lifecycle_run_root" --profile-file "$invalid_autotune_profile" +invalid_dspark_profile="$lifecycle_run_root/profile-dspark-invalid-block.env" +sed 's/^NUM_SPECULATIVE_TOKENS=5$/NUM_SPECULATIVE_TOKENS=4/' \ + "$dspark_profile_file" >"$invalid_dspark_profile" +chmod 0600 "$invalid_dspark_profile" +expect_failure deepseek_dspark_block_size \ + "${lifecycle_test_env[@]}" "$start_deepseek" \ + --run-root "$lifecycle_run_root" --profile-file "$invalid_dspark_profile" + +chmod 0644 "$profile_file" +expect_failure deepseek_profile_mode \ + "${lifecycle_test_env[@]}" "$start_deepseek" \ + --run-root "$lifecycle_run_root" --profile-file "$profile_file" +chmod 0600 "$profile_file" +printf 'UNREVIEWED_FLAG=1\n' >>"$profile_file" +expect_failure deepseek_profile_unknown_field \ + "${lifecycle_test_env[@]}" "$start_deepseek" \ + --run-root "$lifecycle_run_root" --profile-file "$profile_file" + +: >"$TEST_LOG" +env "${lifecycle_test_env[@]:1}" \ + TEST_LIFECYCLE_DEEPSEEK_SHAMU_RUNNING=true \ + TEST_LIFECYCLE_DEEPSEEK_TILIKUM_RUNNING=true \ + "$start_deepseek" --run-root "$lifecycle_run_root" +if grep -Fq 'docker run -d --name deepseek-v41-flash-tp2' "$TEST_LOG"; then + echo "FAIL: healthy DeepSeek re-run relaunched ranks" >&2 + exit 1 +fi +: >"$TEST_LOG" +expect_failure deepseek_verify_only_requires_healthy_pair \ + env "${lifecycle_test_env[@]:1}" \ + TEST_LIFECYCLE_DEEPSEEK_SHAMU_RUNNING=false \ + TEST_LIFECYCLE_DEEPSEEK_TILIKUM_RUNNING=false \ + "$start_deepseek" --run-root "$lifecycle_run_root" \ + --profile-file "$dspark_profile_file" --verify-only +grep -Fq 'not already healthy for the requested profile' \ + "$scratch/deepseek_verify_only_requires_healthy_pair.stderr" +if grep -Eq 'docker (run|stop)' "$TEST_LOG"; then + echo "FAIL: verify-only DeepSeek check mutated the serving pair" >&2 + exit 1 +fi +: >"$TEST_LOG" +expect_failure deepseek_existing_wrong_profile \ + env "${lifecycle_test_env[@]:1}" \ + TEST_LIFECYCLE_DEEPSEEK_SHAMU_RUNNING=true \ + TEST_LIFECYCLE_DEEPSEEK_TILIKUM_RUNNING=true \ + TEST_LIFECYCLE_DEEPSEEK_PROFILE_MATCH=0 \ + "$start_deepseek" --run-root "$lifecycle_run_root" \ + --profile-file "$dspark_profile_file" +if grep -Eq 'docker (run|stop)' "$TEST_LOG"; then + echo "FAIL: mismatched running DeepSeek ranks were mutated before rejection" >&2 + exit 1 +fi +: >"$TEST_LOG" +expect_failure deepseek_existing_split_generation \ + env "${lifecycle_test_env[@]:1}" \ + TEST_LIFECYCLE_DEEPSEEK_SHAMU_RUNNING=true \ + TEST_LIFECYCLE_DEEPSEEK_TILIKUM_RUNNING=true \ + TEST_LIFECYCLE_DEEPSEEK_GENERATION_MATCH=0 \ + "$start_deepseek" --run-root "$lifecycle_run_root" \ + --profile-file "$dspark_profile_file" +if grep -Eq 'docker (run|stop)' "$TEST_LOG"; then + echo "FAIL: split-generation DeepSeek ranks were mutated before rejection" >&2 + exit 1 +fi + +expect_failure deepseek_glm_collision \ + env "${lifecycle_test_env[@]:1}" TEST_LIFECYCLE_GLM_SHAMU_RUNNING=true \ + "$start_deepseek" --run-root "$lifecycle_run_root" +expect_failure deepseek_checkpoint_mismatch \ + env "${lifecycle_test_env[@]:1}" TEST_LIFECYCLE_CHECKPOINTS_MATCH=0 \ + "$start_deepseek" --run-root "$lifecycle_run_root" +expect_failure deepseek_image_mismatch \ + env "${lifecycle_test_env[@]:1}" TEST_LIFECYCLE_IMAGES_MATCH=0 \ + "$start_deepseek" --run-root "$lifecycle_run_root" + +: >"$TEST_LOG" +"${lifecycle_test_env[@]}" "$stop_deepseek" --run-root "$lifecycle_run_root" +grep -Fq 'ssh shamu docker stop --time 120 deepseek-v41-flash-tp2' "$TEST_LOG" +grep -Fq 'ssh tilikum sudo -n docker stop --time 120 deepseek-v41-flash-tp2' "$TEST_LOG" + +expect_failure watchdog_rank_skew \ + env "${lifecycle_test_env[@]:1}" \ + TEST_LIFECYCLE_DEEPSEEK_SHAMU_RUNNING=true \ + TEST_LIFECYCLE_DEEPSEEK_TILIKUM_RUNNING=false \ + TEST_WATCHDOG_NO_RECOVERY=1 \ + "$watchdog" --run-root "$lifecycle_run_root" + +for watchdog_case in \ + TEST_LIFECYCLE_DEEPSEEK_IMAGE_MATCH \ + TEST_LIFECYCLE_DEEPSEEK_PROFILE_MATCH \ + TEST_LIFECYCLE_DEEPSEEK_RUNTIME_MATCH \ + TEST_LIFECYCLE_DEEPSEEK_RESTARTS_OK \ + TEST_LIFECYCLE_DEEPSEEK_GENERATION_MATCH \ + TEST_LIFECYCLE_DEEPSEEK_MOUNTS_OK \ + TEST_LIFECYCLE_DEEPSEEK_ULIMITS_OK \ + TEST_LIFECYCLE_DEEPSEEK_FABRIC_OK \ + TEST_LIFECYCLE_AUTH_SEQUENCE_OK; do + expect_failure "watchdog_rejects_${watchdog_case,,}" \ + env "${lifecycle_test_env[@]:1}" \ + TEST_LIFECYCLE_DEEPSEEK_SHAMU_RUNNING=true \ + TEST_LIFECYCLE_DEEPSEEK_TILIKUM_RUNNING=true \ + TEST_WATCHDOG_NO_RECOVERY=1 \ + "$watchdog_case"=0 \ + "$watchdog" --run-root "$lifecycle_run_root" +done + +: >"$TEST_LOG" +env "${lifecycle_test_env[@]:1}" \ + TEST_LIFECYCLE_DEEPSEEK_SHAMU_RUNNING=true \ + TEST_LIFECYCLE_DEEPSEEK_TILIKUM_RUNNING=false \ + TEST_LIFECYCLE_START_AGE_SECONDS=300 \ + "$watchdog" --run-root "$lifecycle_run_root" +if grep -Eq 'docker (stop|run).*deepseek-v41-flash-tp2' "$TEST_LOG"; then + echo "FAIL: watchdog restarted inside the 20-minute startup grace" >&2 + exit 1 +fi + +init_watchdog_test_root() { + local test_root="$1" + ( + export WEBSTER_PREFLIGHT_TEST_MODE=1 + export TEST_RUNS_ROOT="$lifecycle_runs_root" + source webster/deepseek-v41/scripts/common.sh + init_run_root "$test_root" + ) + sed -i \ + 's#^VLLM_IMAGE_ID=.*#VLLM_IMAGE_ID=sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa#' \ + "$test_root/manifest.env" +} + +watchdog_regression_failures=0 +record_watchdog_failure() { + echo "FAIL: $*" >&2 + watchdog_regression_failures=$((watchdog_regression_failures + 1)) +} + +watchdog_verify_race_root="$lifecycle_runs_root/20990101T000011Z" +init_watchdog_test_root "$watchdog_verify_race_root" +watchdog_verify_race_profile="$watchdog_verify_race_root/profile-dspark-128k.env" +cp "$dspark_profile_file" "$watchdog_verify_race_profile" +chmod 0600 "$watchdog_verify_race_profile" +: >"$TEST_LOG" +if env "${lifecycle_test_env[@]:1}" \ + TEST_LIFECYCLE_DEEPSEEK_SHAMU_RUNNING=true \ + TEST_LIFECYCLE_DEEPSEEK_TILIKUM_RUNNING=true \ + TEST_WATCHDOG_VERIFY_SHAMU_RUNNING=false \ + TEST_WATCHDOG_VERIFY_TILIKUM_RUNNING=false \ + TEST_WATCHDOG_NO_RECOVERY=1 \ + "$watchdog" --run-root "$watchdog_verify_race_root" \ + --profile-file "$watchdog_verify_race_profile"; then + record_watchdog_failure "watchdog verification ignored a serving-state race" +fi +if ! grep -Fq $'\tdeepseek-v41-watchdog\tBREACH\timmutable-runtime-profile' \ + "$watchdog_verify_race_root/events.tsv"; then + record_watchdog_failure "watchdog did not classify the raced verification failure" +fi +if grep -Eq 'docker (stop|run).*deepseek-v41-flash-tp2' "$TEST_LOG"; then + record_watchdog_failure "watchdog verification relaunched the raced serving pair" +fi + +watchdog_lock_root="$lifecycle_runs_root/20990101T000004Z" +init_watchdog_test_root "$watchdog_lock_root" +watchdog_lock_events_before="$(wc -l <"$watchdog_lock_root/events.tsv")" +: >"$TEST_LOG" +exec {watchdog_lock_fd}>"$watchdog_lock_root/watchdog.lock" +flock -n "$watchdog_lock_fd" +if ! env "${lifecycle_test_env[@]:1}" \ + TEST_LIFECYCLE_DEEPSEEK_SHAMU_RUNNING=false \ + TEST_LIFECYCLE_DEEPSEEK_TILIKUM_RUNNING=false \ + TEST_LIFECYCLE_START_AGE_SECONDS=invalid \ + "$watchdog" --run-root "$watchdog_lock_root"; then + record_watchdog_failure "contending watchdog invocation did not exit cleanly" +fi +flock -u "$watchdog_lock_fd" +exec {watchdog_lock_fd}>&- +if [[ "$(wc -l <"$watchdog_lock_root/events.tsv")" != "$watchdog_lock_events_before" ]]; then + record_watchdog_failure "contending watchdog invocation recorded an event" +fi +if [[ -s "$TEST_LOG" || -e "$watchdog_lock_root/watchdog.last-recovery" ]]; then + record_watchdog_failure "contending watchdog invocation attempted recovery" +fi + +watchdog_missing_profile_root="$lifecycle_runs_root/20990101T000005Z" +init_watchdog_test_root "$watchdog_missing_profile_root" +: >"$TEST_LOG" +if env "${lifecycle_test_env[@]:1}" \ + TEST_LIFECYCLE_DEEPSEEK_SHAMU_RUNNING=false \ + TEST_LIFECYCLE_DEEPSEEK_TILIKUM_RUNNING=false \ + TEST_LIFECYCLE_START_AGE_SECONDS=1200 \ + "$watchdog" --run-root "$watchdog_missing_profile_root" \ + >"$scratch/watchdog_missing_profile.stdout" \ + 2>"$scratch/watchdog_missing_profile.stderr"; then + record_watchdog_failure "unhealthy watchdog recovered without a profile file" +fi +if grep -Eq 'docker (stop|run).*deepseek-v41-flash-tp2' "$TEST_LOG" || + [[ -e "$watchdog_missing_profile_root/watchdog.last-recovery" ]]; then + record_watchdog_failure "missing watchdog profile was detected after recovery began" +fi + +watchdog_unsafe_profile_root="$lifecycle_runs_root/20990101T000006Z" +init_watchdog_test_root "$watchdog_unsafe_profile_root" +unsafe_watchdog_profile="$watchdog_unsafe_profile_root/profile-dspark-128k.env" +ln -s "$dspark_profile_file" "$unsafe_watchdog_profile" +: >"$TEST_LOG" +if env "${lifecycle_test_env[@]:1}" \ + TEST_LIFECYCLE_DEEPSEEK_SHAMU_RUNNING=false \ + TEST_LIFECYCLE_DEEPSEEK_TILIKUM_RUNNING=false \ + TEST_LIFECYCLE_START_AGE_SECONDS=1200 \ + "$watchdog" --run-root "$watchdog_unsafe_profile_root" \ + --profile-file "$unsafe_watchdog_profile" \ + >"$scratch/watchdog_unsafe_profile.stdout" \ + 2>"$scratch/watchdog_unsafe_profile.stderr"; then + record_watchdog_failure "watchdog accepted a symlinked profile outside its run root" +fi +if ! grep -Fq 'profile file' "$scratch/watchdog_unsafe_profile.stderr"; then + record_watchdog_failure "watchdog did not identify the unsafe profile" +fi +if grep -Eq 'docker (stop|run).*deepseek-v41-flash-tp2' "$TEST_LOG" || + [[ -e "$watchdog_unsafe_profile_root/watchdog.last-recovery" ]]; then + record_watchdog_failure "unsafe watchdog profile was detected after recovery began" +fi + +expect_watchdog_profile_rejection_before_stop() { + local name="$1" test_root="$2" tested_profile="$3" + : >"$TEST_LOG" + if env "${lifecycle_test_env[@]:1}" \ + TEST_LIFECYCLE_DEEPSEEK_SHAMU_RUNNING=false \ + TEST_LIFECYCLE_DEEPSEEK_TILIKUM_RUNNING=false \ + TEST_LIFECYCLE_START_AGE_SECONDS=1200 \ + "$watchdog" --run-root "$test_root" --profile-file "$tested_profile" \ + >"$scratch/$name.stdout" 2>"$scratch/$name.stderr"; then + record_watchdog_failure "watchdog accepted $name" + fi + if grep -Eq 'docker (stop|run).*deepseek-v41-flash-tp2' "$TEST_LOG" || + [[ -e "$test_root/watchdog.last-recovery" ]]; then + record_watchdog_failure "$name was detected after recovery began" + fi +} + +watchdog_bad_mode_root="$lifecycle_runs_root/20990101T000008Z" +init_watchdog_test_root "$watchdog_bad_mode_root" +watchdog_bad_mode_profile="$watchdog_bad_mode_root/profile-dspark-128k.env" +cp "$dspark_profile_file" "$watchdog_bad_mode_profile" +chmod 0644 "$watchdog_bad_mode_profile" +expect_watchdog_profile_rejection_before_stop \ + "mode-0644 watchdog profile" "$watchdog_bad_mode_root" \ + "$watchdog_bad_mode_profile" + +watchdog_bad_owner_root="$lifecycle_runs_root/20990101T000009Z" +init_watchdog_test_root "$watchdog_bad_owner_root" +watchdog_bad_owner_profile="$watchdog_bad_owner_root/profile-dspark-128k.env" +cp "$dspark_profile_file" "$watchdog_bad_owner_profile" +chmod 0600 "$watchdog_bad_owner_profile" +sudo -n chown 65534 "$watchdog_bad_owner_profile" +expect_watchdog_profile_rejection_before_stop \ + "foreign-owner watchdog profile" "$watchdog_bad_owner_root" \ + "$watchdog_bad_owner_profile" + +watchdog_outside_profile_root="$lifecycle_runs_root/20990101T000010Z" +init_watchdog_test_root "$watchdog_outside_profile_root" +expect_watchdog_profile_rejection_before_stop \ + "profile from another run root" "$watchdog_outside_profile_root" \ + "$dspark_profile_file" + +watchdog_recovery_root="$lifecycle_runs_root/20990101T000007Z" +init_watchdog_test_root "$watchdog_recovery_root" +watchdog_recovery_profile="$watchdog_recovery_root/profile-dspark-128k.env" +cp "$dspark_profile_file" "$watchdog_recovery_profile" +chmod 0600 "$watchdog_recovery_profile" +: >"$TEST_LOG" +if ! env "${lifecycle_test_env[@]:1}" \ + TEST_LIFECYCLE_DEEPSEEK_SHAMU_RUNNING=false \ + TEST_LIFECYCLE_DEEPSEEK_TILIKUM_RUNNING=false \ + TEST_LIFECYCLE_START_AGE_SECONDS=1200 \ + "$watchdog" --run-root "$watchdog_recovery_root" \ + --profile-file "$watchdog_recovery_profile"; then + record_watchdog_failure "watchdog rejected a safe DSpark recovery profile" +fi +if ! grep -Fq 'ssh shamu docker stop --time 120 deepseek-v41-flash-tp2' "$TEST_LOG" || + ! grep -Fq 'ssh tilikum sudo -n docker stop --time 120 deepseek-v41-flash-tp2' "$TEST_LOG" || + ! grep -Fq -- \ + '--speculative-config '\''{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"greedy","rejection_sample_method":"standard"}'\''' \ + "$TEST_LOG"; then + record_watchdog_failure "watchdog did not perform coordinated DSpark recovery" +fi +if ! grep -Fq $'\tdeepseek-v41-watchdog\tRECOVERED\tcoordinated stop/start completed' \ + "$watchdog_recovery_root/events.tsv"; then + record_watchdog_failure "watchdog did not record successful DSpark recovery" +fi + +if (( watchdog_regression_failures > 0 )); then + echo "FAIL: $watchdog_regression_failures watchdog regression test(s) failed" >&2 + exit 1 +fi + +echo "cutover failure-path tests passed" diff --git a/webster/deepseek-v41/tests/test_feature_qualification.py b/webster/deepseek-v41/tests/test_feature_qualification.py new file mode 100644 index 0000000..8293c54 --- /dev/null +++ b/webster/deepseek-v41/tests/test_feature_qualification.py @@ -0,0 +1,493 @@ +#!/usr/bin/env python3 +"""Behavior tests for the private DeepSeek feature qualification runner.""" + +from __future__ import annotations + +import json +import os +import subprocess +import sys +import tempfile +import threading +import unittest +from collections import Counter +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path + + +REPOSITORY_ROOT = Path(__file__).resolve().parents[3] +SCRIPT = ( + REPOSITORY_ROOT + / "webster" + / "deepseek-v41" + / "scripts" + / "feature-qualification.py" +) + + +class FeatureQualificationTests(unittest.TestCase): + def setUp(self) -> None: + self.temporary = tempfile.TemporaryDirectory() + self.root = Path(self.temporary.name) + self.key_file = self.root / "canary.key" + self.key_file.write_text("test-deepseek-key\n", encoding="utf-8") + os.chmod(self.key_file, 0o600) + + def tearDown(self) -> None: + self.temporary.cleanup() + + def run_runner( + self, + base_url: str, + output: Path, + *, + expect_success: bool, + stage: str = "eager", + extra_args: list[str] | None = None, + environment: dict[str, str] | None = None, + ) -> subprocess.CompletedProcess[str]: + command = [ + sys.executable, + str(SCRIPT), + "--base-url", + base_url, + "--model", + "deepseek-ai/DeepSeek-V4.1-Flash", + "--key-file", + str(self.key_file), + "--stage", + stage, + "--output-json", + str(output), + ] + command.extend(extra_args or []) + result = subprocess.run( + command, + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + env=environment, + ) + if expect_success and result.returncode != 0: + self.fail(result.stderr) + if not expect_success and result.returncode == 0: + self.fail(result.stdout) + self.assertNotIn("test-deepseek-key", result.stdout + result.stderr) + return result + + def test_eager_stage_records_the_full_required_capability_contract(self) -> None: + seen_cases: list[str] = [] + seen_requests: dict[str, dict[str, object]] = {} + + class Handler(BaseHTTPRequestHandler): + def log_message(self, *_: object) -> None: + return + + def do_POST(self) -> None: # noqa: N802 - stdlib handler API + length = int(self.headers.get("Content-Length", "0")) + raw = self.rfile.read(length) + if self.headers.get("Authorization") != "Bearer test-deepseek-key": + self.send_error(401) + return + case = self.headers["X-Feature-Qualification-Case"] + seen_cases.append(case) + if case == "malformed-input": + if raw != b'{"model":': + self.send_error(422) + return + self.send_error(400) + return + request = json.loads(raw) + seen_requests[case] = request + if case == "wrong-model": + self.send_error(404) + return + expected_effort = "low" if case == "reasoning-on" else "none" + if request.get("reasoning_effort") != expected_effort: + response = { + "model": "deepseek-ai/DeepSeek-V4.1-Flash", + "choices": [ + { + "message": { + "role": "assistant", + "content": None, + "reasoning": "default reasoning consumed the budget", + }, + "finish_reason": "length", + } + ], + "usage": { + "prompt_tokens": 7, + "completion_tokens": 32, + "total_tokens": 39, + "completion_tokens_details": { + "reasoning_tokens": 32 + }, + }, + } + encoded = json.dumps(response).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(encoded))) + self.end_headers() + self.wfile.write(encoded) + return + if request.get("stream"): + chunks = ( + {"choices": [{"delta": {"content": "stream "}}]}, + { + "choices": [ + { + "delta": {"content": "ok"}, + "finish_reason": "stop", + } + ], + "usage": { + "prompt_tokens": 7, + "completion_tokens": 2, + "total_tokens": 9, + }, + }, + ) + body = "".join( + f"data: {json.dumps(chunk)}\n\n" for chunk in chunks + ) + "data: [DONE]\n\n" + encoded = body.encode() + self.send_response(200) + self.send_header("Content-Type", "text/event-stream") + self.send_header("Content-Length", str(len(encoded))) + self.end_headers() + self.wfile.write(encoded) + return + if case == "disconnect-queue-drain": + chunk = { + "choices": [{"delta": {"content": "started"}}] + } + encoded = f"data: {json.dumps(chunk)}\n\n".encode() + self.send_response(200) + self.send_header("Content-Type", "text/event-stream") + self.send_header("Content-Length", str(len(encoded))) + self.end_headers() + self.wfile.write(encoded) + return + + content = { + "deterministic-text": "QUALIFICATION_OK", + "utf8": "café 東京 🚀", + "stop-sequence": "before", + "usage-accounting": "usage ok", + "reasoning-on": "42", + "tool-auto": "", + "tool-named": "", + "tool-required": "", + "tool-parallel": "", + "tool-result": "17 + 25 = 42", + "structured": '{"city":"Paris","country":"France"}', + "json-object": '{"city":"Paris","country":"France"}', + "vision": "red", + }[case] + message = {"role": "assistant", "content": content} + if case == "reasoning-on": + message["reasoning"] = "17 + 25 equals 42." + if case in {"tool-auto", "tool-named", "tool-required"}: + message["tool_calls"] = [ + { + "id": "call_1", + "type": "function", + "function": { + "name": "add", + "arguments": '{"a":17,"b":25}', + }, + } + ] + if case == "tool-parallel": + message["tool_calls"] = [ + { + "id": "call_1", + "type": "function", + "function": { + "name": "add", + "arguments": '{"a":17,"b":25}', + }, + }, + { + "id": "call_2", + "type": "function", + "function": { + "name": "add", + "arguments": '{"a":9,"b":11}', + }, + }, + ] + response = { + "model": "deepseek-ai/DeepSeek-V4.1-Flash", + "choices": [ + { + "message": message, + "finish_reason": ( + "tool_calls" + if case.startswith("tool-") + and case != "tool-result" + else "stop" + ), + } + ], + "usage": { + "prompt_tokens": 7, + "completion_tokens": 2, + "total_tokens": 9, + "completion_tokens_details": { + "reasoning_tokens": 1 if case == "reasoning-on" else 0 + }, + }, + } + encoded = json.dumps(response).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(encoded))) + self.end_headers() + self.wfile.write(encoded) + + def do_GET(self) -> None: # noqa: N802 - stdlib handler API + if self.path != "/metrics": + self.send_error(404) + return + encoded = ( + "vllm:num_requests_running 0\n" + "vllm:num_requests_waiting 0\n" + ).encode() + self.send_response(200) + self.send_header("Content-Type", "text/plain") + self.send_header("Content-Length", str(len(encoded))) + self.end_headers() + self.wfile.write(encoded) + + server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + output = self.root / "eager.json" + failure_log = self.root / "eager-failure-server.log" + try: + self.run_runner( + f"http://127.0.0.1:{server.server_port}/v1", + output, + expect_success=True, + extra_args=[ + "--failure-log-output", + str(failure_log), + "--server-log-node", + "shamu", + "--server-log-node", + "tilikum", + ], + ) + finally: + server.shutdown() + thread.join(timeout=5) + server.server_close() + + self.assertEqual( + set(seen_cases), + { + "deterministic-text", + "utf8", + "stop-sequence", + "streaming", + "usage-accounting", + "reasoning-on", + "tool-auto", + "tool-named", + "tool-required", + "tool-parallel", + "tool-result", + "structured", + "json-object", + "vision", + "malformed-input", + "wrong-model", + "disconnect-queue-drain", + }, + ) + counts = Counter(seen_cases) + self.assertEqual(counts["deterministic-text"], 3) + self.assertTrue(all(count == 1 for case, count in counts.items() if case != "deterministic-text")) + self.assertEqual( + seen_requests["tool-named"]["tool_choice"], + {"type": "function", "function": {"name": "add"}}, + ) + self.assertEqual(seen_requests["tool-required"]["tool_choice"], "required") + self.assertIs(seen_requests["tool-parallel"]["parallel_tool_calls"], True) + self.assertEqual( + seen_requests["json-object"]["response_format"], + {"type": "json_object"}, + ) + self.assertEqual(seen_requests["wrong-model"]["model"], "qualification-wrong-model") + artifact = json.loads(output.read_text(encoding="utf-8")) + self.assertEqual(output.stat().st_mode & 0o777, 0o600) + self.assertTrue(artifact["ok"]) + self.assertFalse(failure_log.exists()) + self.assertEqual( + artifact["failure_log_capture"], + { + "requested": True, + "captured": False, + "ok": True, + "path": str(failure_log.resolve()), + "nodes": ["shamu", "tilikum"], + "container": "deepseek-v41-flash-tp2", + }, + ) + self.assertEqual(artifact["stage"], "eager") + self.assertEqual(len(artifact["cases"]), 17) + for result in artifact["cases"]: + self.assertTrue(result["semantic_ok"], result) + expected_status = { + "malformed-input": 400, + "wrong-model": 404, + }.get(result["case"], 200) + self.assertEqual(result["status"], expected_status) + self.assertRegex(result["request_hash"], r"^[0-9a-f]{64}$") + self.assertRegex(result["normalized_output_hash"], r"^[0-9a-f]{64}$") + self.assertGreaterEqual(result["latency_seconds"], 0) + if expected_status == 200 and result["case"] != "disconnect-queue-drain": + self.assertEqual(result["token_usage"]["total_tokens"], 9) + deterministic = next( + result + for result in artifact["cases"] + if result["case"] == "deterministic-text" + ) + self.assertEqual(deterministic["repetitions"], 3) + self.assertEqual(deterministic["distinct_normalized_outputs"], 1) + disconnect = next( + result + for result in artifact["cases"] + if result["case"] == "disconnect-queue-drain" + ) + self.assertTrue(disconnect["client_disconnected"]) + self.assertTrue(disconnect["queue_drained"]) + self.assertNotIn("test-deepseek-key", output.read_text(encoding="utf-8")) + + def test_group_readable_key_is_rejected_before_network_access(self) -> None: + os.chmod(self.key_file, 0o640) + result = self.run_runner( + "http://127.0.0.1:1/v1", + self.root / "unsafe.json", + expect_success=False, + ) + self.assertIn("0600", result.stderr) + + def test_graph_stage_is_recorded_for_a_failed_network_qualification(self) -> None: + output = self.root / "graphs.json" + self.run_runner( + "http://127.0.0.1:1/v1", + output, + expect_success=False, + stage="graphs", + ) + artifact = json.loads(output.read_text(encoding="utf-8")) + self.assertEqual(artifact["stage"], "graphs") + self.assertEqual(len(artifact["cases"]), 17) + + def test_http_error_is_recorded_instead_of_crashing_on_message_access(self) -> None: + class Handler(BaseHTTPRequestHandler): + def log_message(self, *_: object) -> None: + return + + def do_POST(self) -> None: # noqa: N802 - stdlib handler API + self.send_error(429) + + server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + output = self.root / "http-error.json" + try: + self.run_runner( + f"http://127.0.0.1:{server.server_port}/v1", + output, + expect_success=False, + ) + finally: + server.shutdown() + thread.join(timeout=5) + server.server_close() + + artifact = json.loads(output.read_text(encoding="utf-8")) + self.assertFalse(artifact["ok"]) + self.assertEqual(len(artifact["cases"]), 17) + regular = [ + result + for result in artifact["cases"] + if result["case"] != "disconnect-queue-drain" + ] + self.assertTrue(all(result["status"] == 429 for result in regular)) + disconnect = next( + result + for result in artifact["cases"] + if result["case"] == "disconnect-queue-drain" + ) + self.assertEqual(disconnect["status"], 0) + self.assertEqual(disconnect["error"], "HTTPError") + self.assertTrue( + all(not result["semantic_ok"] for result in artifact["cases"]) + ) + + def test_failure_captures_both_rank_logs_without_credentials(self) -> None: + fake_bin = self.root / "bin" + fake_bin.mkdir() + invocation_log = self.root / "ssh-invocations.log" + fake_ssh = fake_bin / "ssh" + fake_ssh.write_text( + "#!/usr/bin/env bash\n" + 'printf "%s auth_sock=%s\\n" "$*" "${SSH_AUTH_SOCK-unset}" ' + '>>"$TEST_SSH_INVOCATIONS"\n' + 'printf "Authorization: Bearer sk-should-never-survive\\n"\n' + 'printf "VLLM_API_KEY=raw-hex-secret-should-not-survive\\n"\n' + 'printf "rank diagnostic for %s\\n" "$1"\n', + encoding="utf-8", + ) + os.chmod(fake_ssh, 0o755) + environment = os.environ.copy() + environment["PATH"] = f"{fake_bin}:{environment['PATH']}" + environment["TEST_SSH_INVOCATIONS"] = str(invocation_log) + environment["SSH_AUTH_SOCK"] = "/tmp/agent.sock" + output = self.root / "failed.json" + failure_log = self.root / "failure-server.log" + self.run_runner( + "http://127.0.0.1:1/v1", + output, + expect_success=False, + extra_args=[ + "--failure-log-output", + str(failure_log), + "--server-log-node", + "shamu", + "--server-log-node", + "tilikum", + "--server-container", + "deepseek-v41-flash-tp2", + ], + environment=environment, + ) + captured = failure_log.read_text(encoding="utf-8") + self.assertEqual(failure_log.stat().st_mode & 0o777, 0o600) + self.assertNotIn("sk-should-never-survive", captured) + self.assertNotIn("raw-hex-secret-should-not-survive", captured) + self.assertIn("Authorization: Bearer [REDACTED]", captured) + invocations = invocation_log.read_text(encoding="utf-8") + self.assertIn( + "shamu docker logs --since", invocations + ) + self.assertIn( + "tilikum sudo -n docker logs --since", invocations + ) + self.assertNotIn("/tmp/agent.sock", invocations) + artifact = json.loads(output.read_text(encoding="utf-8")) + capture = artifact["failure_log_capture"] + self.assertTrue(capture["ok"]) + self.assertEqual(capture["path"], str(failure_log.resolve())) + self.assertRegex(capture["sha256"], r"^[0-9a-f]{64}$") + + +if __name__ == "__main__": + unittest.main() diff --git a/webster/deepseek-v41/tests/test_litellm_cutover.py b/webster/deepseek-v41/tests/test_litellm_cutover.py new file mode 100644 index 0000000..769c780 --- /dev/null +++ b/webster/deepseek-v41/tests/test_litellm_cutover.py @@ -0,0 +1,830 @@ +#!/usr/bin/env python3 +"""Behavior tests for semantic LiteLLM alias rendering and shape checks.""" + +from __future__ import annotations + +import copy +import importlib.util +import json +import os +import subprocess +import sys +import tempfile +import threading +import unittest +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path + +import yaml + + +REPOSITORY_ROOT = Path(__file__).resolve().parents[3] +PACKAGE_ROOT = REPOSITORY_ROOT / "webster" / "deepseek-v41" +SCRIPTS = PACKAGE_ROOT / "scripts" +RENDER = SCRIPTS / "render-litellm-cutover.py" +VERIFY_CONFIG = SCRIPTS / "verify-litellm-config.py" +VERIFY_CALLBACKS = SCRIPTS / "verify-litellm-callbacks.py" +VERIFY_CONTAINER = SCRIPTS / "verify-litellm-container.py" +CONTRACT_PROBE = SCRIPTS / "contract-probe.py" +GUARD_CALLBACK = "custom_callbacks.glm52_contract_guard.glm52_contract_guard" +SECRET_VALUES = ("os.environ/OLD_BACKEND_KEY", "os.environ/NEW_BACKEND_KEY") + + +def before_config() -> dict: + return { + "model_list": [ + { + "model_name": "glm-5.2", + "litellm_params": { + "model": "hosted_vllm/nvidia/GLM-5.2-NVFP4", + "api_base": "http://100.73.140.127:8000/v1", + "api_key": SECRET_VALUES[0], + }, + "model_info": { + "mode": "chat", + "max_input_tokens": 320000, + "max_output_tokens": 320000, + "supports_function_calling": True, + "supports_reasoning": True, + "input_cost_per_token": 5.8e-08, + "output_cost_per_token": 3.8e-06, + "description": "Station GLM-5.2", + "compatibility_note": "preserve-me", + }, + }, + { + "model_name": "glm-5.3-flash", + "litellm_params": { + "model": "hosted_vllm/zai-org/GLM-5.3-Flash", + "api_base": "http://100.73.165.55:8000/v1", + "api_key": SECRET_VALUES[1], + }, + "model_info": { + "mode": "chat", + "max_input_tokens": 1048576, + "max_output_tokens": 1048576, + "supports_function_calling": True, + "supports_reasoning": True, + "supports_vision": True, + "input_cost_per_token": 6.0e-08, + "output_cost_per_token": 1.7e-06, + "description": "Native GLM-5.3", + }, + }, + { + "model_name": "unrelated-model", + "litellm_params": { + "model": "openai/unrelated", + "api_key": "os.environ/UNRELATED_KEY", + }, + "model_info": {"mode": "chat", "description": "untouched"}, + }, + ], + "general_settings": {"store_prompts_in_spend_logs": True}, + "litellm_settings": { + "callbacks": [ + "custom_callbacks.session_mapper.session_mapper", + "langfuse_otel", + "custom_callbacks.cost_in_body_logger.logger", + ] + }, + "router_settings": {"plugins": ["custom_callbacks.kv_router.plugin"]}, + "key_management_settings": { + "default_team_settings": [{"models": ["glm-5.2"]}] + }, + } + + +def valid_inspect() -> list[dict]: + return [ + { + "Id": "container-id", + "Image": "sha256:" + "0" * 64, + "Config": { + "Image": "ghcr.io/berriai/litellm:main-latest", + "Entrypoint": ["docker/prod_entrypoint.sh"], + "Cmd": [ + "--config", + "/app/config.yaml", + "--host", + "127.0.0.1", + "--port", + "4446", + ], + "Env": ["LITELLM_MASTER_KEY=never-print-this"], + "User": "", + }, + "HostConfig": { + "NetworkMode": "host", + "RestartPolicy": {"Name": "unless-stopped", "MaximumRetryCount": 0}, + "PortBindings": {}, + "CapAdd": None, + "Ulimits": [], + }, + "Mounts": [ + { + "Type": "bind", + "Source": "/home/nvidia/litellm/config.yaml", + "Destination": "/app/config.yaml", + "Mode": "ro", + "RW": False, + } + ], + } + ] + + +class LiteLLMCutoverTests(unittest.TestCase): + def setUp(self) -> None: + self.temporary = tempfile.TemporaryDirectory() + self.root = Path(self.temporary.name) + self.before = self.root / "before.yaml" + self.candidate = self.root / "candidate.yaml" + self.before.write_text( + yaml.safe_dump(before_config(), sort_keys=False), encoding="utf-8" + ) + os.chmod(self.before, 0o600) + self.private_acceptance = self.root / "private-acceptance.json" + self.private_acceptance.write_text( + json.dumps( + { + "decision": "GO", + "publication": { + "public_model_name": "deepseek-v4.1-flash", + "served_model_name": "deepseek-ai/DeepSeek-V4.1-Flash", + "max_context": 1048576, + "station_endpoint": "http://100.73.140.127:8000/v1", + "supports_function_calling": True, + "supports_reasoning": True, + "supports_response_schema": True, + "supports_vision": True, + "input_cost_per_token": 1.3e-8, + "output_cost_per_token": 9.6e-7, + }, + }, + sort_keys=True, + ) + + "\n", + encoding="utf-8", + ) + os.chmod(self.private_acceptance, 0o600) + + def tearDown(self) -> None: + self.temporary.cleanup() + + def run_script( + self, script: Path, *arguments: str, expect_success: bool + ) -> subprocess.CompletedProcess[str]: + result = subprocess.run( + [sys.executable, str(script), *arguments], + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + ) + if expect_success and result.returncode != 0: + self.fail(f"{script.name} failed: {result.stderr}") + if not expect_success and result.returncode == 0: + self.fail(f"{script.name} unexpectedly succeeded: {result.stdout}") + for secret in SECRET_VALUES: + self.assertNotIn(secret, result.stdout + result.stderr) + return result + + def render_candidate(self) -> dict: + self.run_script( + RENDER, + str(self.before), + str(self.candidate), + expect_success=True, + ) + return yaml.safe_load(self.candidate.read_text(encoding="utf-8")) + + def test_guard_loader_accepts_only_the_known_sanitizer_before_guard(self) -> None: + spec = importlib.util.spec_from_file_location( + "verify_litellm_callbacks", VERIFY_CALLBACKS + ) + if spec is None or spec.loader is None: + self.fail("cannot load LiteLLM callback verifier") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + sanitizer = ( + "custom_callbacks.responses_sanitizer." + "responses_encrypted_content_sanitizer" + ) + suffix = [ + "custom_callbacks.session_mapper.session_mapper", + "langfuse_otel", + ] + for callbacks in ([GUARD_CALLBACK, *suffix], [sanitizer, GUARD_CALLBACK, *suffix]): + with self.subTest(callbacks=callbacks): + module.validate_guard_order({"litellm_settings": {"callbacks": callbacks}}) + for callbacks in ( + [suffix[0], GUARD_CALLBACK, suffix[1]], + [GUARD_CALLBACK, GUARD_CALLBACK, *suffix], + [sanitizer, suffix[0], GUARD_CALLBACK], + [sanitizer, *suffix], + [], + [GUARD_CALLBACK, 7], + ): + with self.subTest(callbacks=callbacks), self.assertRaises( + module.ValidationError + ): + module.validate_guard_order( + {"litellm_settings": {"callbacks": callbacks}} + ) + + def test_callback_loader_uses_working_directory_and_sanitizes_failures(self) -> None: + runtime = self.root / "runtime" + integrations = runtime / "litellm" / "integrations" + callbacks_root = runtime / "custom_callbacks" + integrations.mkdir(parents=True) + callbacks_root.mkdir() + for package in ( + runtime / "litellm", + integrations, + callbacks_root, + ): + (package / "__init__.py").write_text("", encoding="utf-8") + (integrations / "custom_logger.py").write_text( + "class CustomLogger:\n pass\n", encoding="utf-8" + ) + modules = { + "responses_sanitizer.py": ( + "from litellm.integrations.custom_logger import CustomLogger\n" + "responses_encrypted_content_sanitizer = CustomLogger()\n" + ), + "glm52_contract_guard.py": ( + "from litellm.integrations.custom_logger import CustomLogger\n" + "glm52_contract_guard = CustomLogger()\n" + ), + "good.py": ( + "from litellm.integrations.custom_logger import CustomLogger\n" + "callback = CustomLogger()\n" + ), + "bad.py": "callback = object()\n", + "explodes.py": "raise RuntimeError('SENTINEL_SECRET')\n", + } + for name, source in modules.items(): + (callbacks_root / name).write_text(source, encoding="utf-8") + + sanitizer = ( + "custom_callbacks.responses_sanitizer." + "responses_encrypted_content_sanitizer" + ) + + def invoke(callbacks: list[str]) -> subprocess.CompletedProcess[str]: + config = runtime / "config.yaml" + config.write_text( + yaml.safe_dump({"litellm_settings": {"callbacks": callbacks}}), + encoding="utf-8", + ) + return subprocess.run( + [sys.executable, str(VERIFY_CALLBACKS), str(config)], + cwd=runtime, + text=True, + capture_output=True, + check=False, + ) + + result = invoke( + [sanitizer, GUARD_CALLBACK, "langfuse_otel", "custom_callbacks.good.callback"] + ) + self.assertEqual(result.returncode, 0, result.stderr) + + result = invoke([GUARD_CALLBACK, "custom_callbacks.bad.callback"]) + self.assertEqual(result.returncode, 1) + self.assertIn("custom callback is not a CustomLogger", result.stderr) + + result = invoke([GUARD_CALLBACK, "custom_callbacks.explodes.callback"]) + self.assertEqual(result.returncode, 1) + self.assertNotIn("SENTINEL_SECRET", result.stdout + result.stderr) + self.assertNotIn("Traceback", result.stderr) + + def test_renderer_changes_only_the_legacy_alias_contract(self) -> None: + original = self.before.read_bytes() + candidate = self.render_candidate() + self.assertEqual(self.before.read_bytes(), original) + self.assertEqual(self.candidate.stat().st_mode & 0o777, 0o600) + + models = {item["model_name"]: item for item in candidate["model_list"]} + alias = models["glm-5.2"] + native = models["glm-5.3-flash"] + self.assertEqual(alias["litellm_params"], native["litellm_params"]) + self.assertEqual( + alias["model_info"], + { + "mode": "chat", + "max_input_tokens": 320000, + "max_output_tokens": 320000, + "supports_function_calling": True, + "supports_reasoning": True, + "supports_vision": False, + "input_cost_per_token": 6.0e-08, + "output_cost_per_token": 1.7e-06, + "description": ( + "Deprecated GLM-5.2 compatibility alias served by " + "GLM-5.3-Flash; legacy 320000-token shared window, reasoning, " + "and function-calling contract retained; no advertised vision " + "capability." + ), + "compatibility_note": "preserve-me", + }, + ) + self.assertEqual(models["unrelated-model"], before_config()["model_list"][2]) + self.assertEqual( + candidate["litellm_settings"]["callbacks"], + [GUARD_CALLBACK, *before_config()["litellm_settings"]["callbacks"]], + ) + self.assertEqual( + candidate["router_settings"], before_config()["router_settings"] + ) + self.assertEqual( + candidate["key_management_settings"], + before_config()["key_management_settings"], + ) + + def test_renderer_is_idempotent(self) -> None: + self.render_candidate() + second = self.root / "second.yaml" + self.run_script( + RENDER, + str(self.candidate), + str(second), + expect_success=True, + ) + self.assertEqual( + yaml.safe_load(second.read_text(encoding="utf-8")), + yaml.safe_load(self.candidate.read_text(encoding="utf-8")), + ) + + def test_allowed_diff_verifier_accepts_the_rendered_candidate(self) -> None: + self.render_candidate() + result = self.run_script( + VERIFY_CONFIG, + str(self.before), + str(self.candidate), + "--phase", + "alias", + expect_success=True, + ) + self.assertIn("allowed semantic diff", result.stdout) + + def test_allowed_diff_verifier_rejects_unrelated_model_mutation(self) -> None: + candidate = self.render_candidate() + candidate["model_list"][2]["model_info"]["description"] = "changed" + self.candidate.write_text( + yaml.safe_dump(candidate, sort_keys=False), encoding="utf-8" + ) + self.run_script( + VERIFY_CONFIG, + str(self.before), + str(self.candidate), + "--phase", + "alias", + expect_success=False, + ) + + def test_allowed_diff_verifier_rejects_missing_kv_router(self) -> None: + candidate = self.render_candidate() + candidate["router_settings"]["plugins"] = [] + self.candidate.write_text( + yaml.safe_dump(candidate, sort_keys=False), encoding="utf-8" + ) + self.run_script( + VERIFY_CONFIG, + str(self.before), + str(self.candidate), + "--phase", + "alias", + expect_success=False, + ) + + def test_allowed_diff_verifier_rejects_global_pre_call_checks(self) -> None: + candidate = self.render_candidate() + candidate["router_settings"]["enable_pre_call_checks"] = True + self.candidate.write_text( + yaml.safe_dump(candidate, sort_keys=False), encoding="utf-8" + ) + self.run_script( + VERIFY_CONFIG, + str(self.before), + str(self.candidate), + "--phase", + "alias", + expect_success=False, + ) + + def test_allowed_diff_verifier_rejects_duplicate_public_name(self) -> None: + candidate = self.render_candidate() + candidate["model_list"].append(copy.deepcopy(candidate["model_list"][0])) + self.candidate.write_text( + yaml.safe_dump(candidate, sort_keys=False), encoding="utf-8" + ) + self.run_script( + VERIFY_CONFIG, + str(self.before), + str(self.candidate), + "--phase", + "alias", + expect_success=False, + ) + + def test_container_shape_accepts_the_exact_live_command_contract(self) -> None: + inspect_path = self.root / "inspect.json" + inspect_path.write_text(json.dumps(valid_inspect()), encoding="utf-8") + result = self.run_script( + VERIFY_CONTAINER, str(inspect_path), expect_success=True + ) + self.assertIn("container shape valid", result.stdout) + self.assertNotIn("never-print-this", result.stdout) + + def test_container_shape_rejects_duplicate_entrypoint_in_cmd(self) -> None: + inspect = valid_inspect() + inspect[0]["Config"]["Cmd"] = [ + "docker/prod_entrypoint.sh", + *inspect[0]["Config"]["Cmd"], + ] + inspect_path = self.root / "inspect.json" + inspect_path.write_text(json.dumps(inspect), encoding="utf-8") + self.run_script(VERIFY_CONTAINER, str(inspect_path), expect_success=False) + + def test_container_shape_rejects_wrong_bind_or_port(self) -> None: + for token, replacement in (("127.0.0.1", "0.0.0.0"), ("4446", "4444")): + with self.subTest(token=token): + inspect = valid_inspect() + command = inspect[0]["Config"]["Cmd"] + command[command.index(token)] = replacement + inspect_path = self.root / f"inspect-{token.replace('.', '-')}.json" + inspect_path.write_text(json.dumps(inspect), encoding="utf-8") + self.run_script( + VERIFY_CONTAINER, str(inspect_path), expect_success=False + ) + + def test_container_recreation_must_match_the_baseline_shape(self) -> None: + baseline = self.root / "baseline.json" + candidate = self.root / "recreation.json" + baseline.write_text(json.dumps(valid_inspect()), encoding="utf-8") + changed = valid_inspect() + changed[0]["HostConfig"]["NetworkMode"] = "bridge" + candidate.write_text(json.dumps(changed), encoding="utf-8") + self.run_script( + VERIFY_CONTAINER, + str(candidate), + "--baseline", + str(baseline), + expect_success=False, + ) + + def test_contract_probe_exercises_real_chat_shapes_without_leaking_key(self) -> None: + seen_cases: list[str] = [] + + class Handler(BaseHTTPRequestHandler): + def log_message(self, *_: object) -> None: + return + + def do_POST(self) -> None: # noqa: N802 - stdlib handler API + length = int(self.headers.get("Content-Length", "0")) + body = json.loads(self.rfile.read(length)) + if self.headers.get("Authorization") != "Bearer test-contract-key": + self.send_error(401) + return + case = self.headers.get("X-Contract-Probe-Case", "missing") + seen_cases.append(case) + if body.get("stream"): + chunks = [ + { + "model": "legacy-response-model", + "choices": [ + {"delta": {"content": "ok"}, "finish_reason": None} + ], + }, + { + "model": "legacy-response-model", + "choices": [{"delta": {}, "finish_reason": "stop"}], + }, + ] + payload = "".join( + f"data: {json.dumps(chunk)}\n\n" for chunk in chunks + ) + "data: [DONE]\n\n" + encoded = payload.encode() + self.send_response(200) + self.send_header("Content-Type", "text/event-stream") + self.send_header("Content-Length", str(len(encoded))) + self.end_headers() + self.wfile.write(encoded) + return + + message: dict = {"role": "assistant", "content": "ok"} + if case == "tool-auto": + message = { + "role": "assistant", + "content": None, + "tool_calls": [ + { + "id": "call_1", + "type": "function", + "function": { + "name": "get_weather", + "arguments": '{"city":"San Francisco"}', + }, + } + ], + } + elif case == "structured": + message["content"] = '{"city":"San Francisco","temperature_c":15}' + response = { + "model": "legacy-response-model", + "choices": [{"message": message, "finish_reason": "stop"}], + "usage": { + "prompt_tokens": 10, + "completion_tokens": 2, + "total_tokens": 12, + }, + } + encoded = json.dumps(response).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(encoded))) + self.end_headers() + self.wfile.write(encoded) + + server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + try: + key_file = self.root / "probe.key" + key_file.write_text("test-contract-key\n", encoding="utf-8") + os.chmod(key_file, 0o600) + output = self.root / "probe.json" + environment = os.environ.copy() + environment["WEBSTER_CONTRACT_PROBE_TEST_MODE"] = "1" + result = subprocess.run( + [ + sys.executable, + str(CONTRACT_PROBE), + "--base-url", + f"http://127.0.0.1:{server.server_port}/v1", + "--models", + "glm-5.2", + "--key-file", + str(key_file), + "--expected-response-model", + "glm-5.2=legacy-response-model", + "--output-json", + str(output), + "--skip-boundary", + ], + cwd=REPOSITORY_ROOT, + env=environment, + text=True, + capture_output=True, + check=False, + ) + finally: + server.shutdown() + thread.join(timeout=5) + server.server_close() + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual( + set(seen_cases), + { + "plain", + "stream", + "reasoning-none", + "reasoning-on", + "tool-auto", + "tool-result", + "structured", + }, + ) + artifact = json.loads(output.read_text(encoding="utf-8")) + self.assertTrue(artifact["ok"]) + self.assertEqual(output.stat().st_mode & 0o777, 0o600) + self.assertNotIn("test-contract-key", output.read_text(encoding="utf-8")) + + def test_contract_probe_rejects_a_group_readable_key(self) -> None: + key_file = self.root / "probe.key" + key_file.write_text("test-contract-key\n", encoding="utf-8") + os.chmod(key_file, 0o640) + result = subprocess.run( + [ + sys.executable, + str(CONTRACT_PROBE), + "--base-url", + "http://127.0.0.1:1/v1", + "--models", + "glm-5.2", + "--key-file", + str(key_file), + "--expected-response-model", + "glm-5.2=legacy-response-model", + "--output-json", + str(self.root / "probe.json"), + "--skip-boundary", + ], + cwd=REPOSITORY_ROOT, + env={**os.environ, "WEBSTER_CONTRACT_PROBE_TEST_MODE": "1"}, + text=True, + capture_output=True, + check=False, + ) + self.assertNotEqual(result.returncode, 0) + self.assertIn("0600", result.stderr) + + def test_backend_counter_includes_rejected_chat_requests(self) -> None: + class Handler(BaseHTTPRequestHandler): + def log_message(self, *_: object) -> None: + return + + def do_GET(self) -> None: # noqa: N802 - stdlib handler API + payload = b"""\ +http_requests_total{handler=\"/v1/chat/completions\",method=\"POST\",status=\"2xx\"} 7 +http_requests_total{handler=\"/v1/chat/completions\",method=\"POST\",status=\"4xx\"} 3 +http_requests_total{handler=\"/v1/models\",method=\"GET\",status=\"2xx\"} 100 +vllm:request_success_total{finished_reason=\"stop\"} 1000 +""" + self.send_response(200) + self.send_header("Content-Length", str(len(payload))) + self.end_headers() + self.wfile.write(payload) + + server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + try: + spec = importlib.util.spec_from_file_location("contract_probe", CONTRACT_PROBE) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + total = module.backend_chat_request_total( + f"http://127.0.0.1:{server.server_port}/metrics" + ) + finally: + server.shutdown() + thread.join(timeout=5) + server.server_close() + self.assertEqual(total, 10.0) + + def test_boundary_snapshot_covers_unchanged_and_increasing_backends(self) -> None: + class Handler(BaseHTTPRequestHandler): + def log_message(self, *_: object) -> None: + return + + def do_GET(self) -> None: # noqa: N802 - stdlib handler API + count = 11 if self.path == "/old" else 22 + payload = ( + "http_requests_total{handler=\"/v1/chat/completions\"," + f"method=\"POST\",status=\"2xx\"}} {count}\n" + ).encode() + self.send_response(200) + self.send_header("Content-Length", str(len(payload))) + self.end_headers() + self.wfile.write(payload) + + server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + try: + spec = importlib.util.spec_from_file_location("contract_probe", CONTRACT_PROBE) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + base = f"http://127.0.0.1:{server.server_port}" + totals = module.snapshot_backend_chat_requests( + { + "backends": { + "old": {"url": base + "/old", "expect": "unchanged"}, + "new": {"url": base + "/new", "expect": "increase"}, + } + } + ) + finally: + server.shutdown() + thread.join(timeout=5) + server.server_close() + self.assertEqual(totals, {"old": 11.0, "new": 22.0}) + + def test_publish_renderer_adds_only_the_qualified_private_canary(self) -> None: + api_key_file = self.root / "deepseek-v41.key" + api_key_file.write_text("test-publish-backend-key\n", encoding="utf-8") + os.chmod(api_key_file, 0o600) + result = subprocess.run( + [ + sys.executable, + str(RENDER), + str(self.before), + str(self.candidate), + "--phase", + "publish", + "--api-key-file", + str(api_key_file), + "--private-acceptance", + str(self.private_acceptance), + ], + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertNotIn("test-publish-backend-key", result.stdout + result.stderr) + candidate = yaml.safe_load(self.candidate.read_text(encoding="utf-8")) + self.assertEqual(candidate["model_list"][:-1], before_config()["model_list"]) + published = candidate["model_list"][-1] + self.assertEqual(published["model_name"], "deepseek-v4.1-flash") + self.assertEqual( + published["litellm_params"], + { + "model": "hosted_vllm/deepseek-ai/DeepSeek-V4.1-Flash", + "api_base": "http://100.73.140.127:8000/v1", + "api_key": "test-publish-backend-key", + }, + ) + self.assertEqual( + published["model_info"], + { + "mode": "chat", + "max_input_tokens": 1048576, + "max_output_tokens": 1048576, + "supports_function_calling": True, + "supports_reasoning": True, + "supports_response_schema": True, + "supports_vision": True, + "input_cost_per_token": 1.3e-8, + "output_cost_per_token": 9.6e-7, + "description": ( + "DeepSeek V4.1 Flash on the Shamu/Tilikum station pair; " + "qualified private TP2 profile." + ), + }, + ) + verify = self.run_script( + VERIFY_CONFIG, + str(self.before), + str(self.candidate), + "--phase", + "publish", + expect_success=True, + ) + self.assertIn("publish phase", verify.stdout) + + def test_publish_verifier_rejects_an_existing_model_change(self) -> None: + api_key_file = self.root / "deepseek-v41.key" + api_key_file.write_text("test-publish-backend-key\n", encoding="utf-8") + os.chmod(api_key_file, 0o600) + result = subprocess.run( + [ + sys.executable, + str(RENDER), + str(self.before), + str(self.candidate), + "--phase", + "publish", + "--api-key-file", + str(api_key_file), + "--private-acceptance", + str(self.private_acceptance), + ], + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + candidate = yaml.safe_load(self.candidate.read_text(encoding="utf-8")) + candidate["model_list"][2]["model_info"]["description"] = "mutated" + self.candidate.write_text( + yaml.safe_dump(candidate, sort_keys=False), encoding="utf-8" + ) + self.run_script( + VERIFY_CONFIG, + str(self.before), + str(self.candidate), + "--phase", + "publish", + expect_success=False, + ) + + def test_publish_renderer_rejects_caller_values_that_differ_from_acceptance(self) -> None: + api_key_file = self.root / "deepseek-v41.key" + api_key_file.write_text("test-publish-backend-key\n", encoding="utf-8") + os.chmod(api_key_file, 0o600) + result = self.run_script( + RENDER, + str(self.before), + str(self.candidate), + "--phase", + "publish", + "--api-key-file", + str(api_key_file), + "--private-acceptance", + str(self.private_acceptance), + "--max-context", + "320000", + expect_success=False, + ) + self.assertIn("accepted publication record", result.stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/webster/deepseek-v41/tests/test_long_context_qualification.py b/webster/deepseek-v41/tests/test_long_context_qualification.py new file mode 100644 index 0000000..ed4e6d7 --- /dev/null +++ b/webster/deepseek-v41/tests/test_long_context_qualification.py @@ -0,0 +1,241 @@ +#!/usr/bin/env python3 +"""Behavior tests for exact-token DeepSeek long-context qualification.""" + +from __future__ import annotations + +import json +import os +import subprocess +import sys +import tempfile +import threading +import unittest +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path + + +REPOSITORY_ROOT = Path(__file__).resolve().parents[3] +SCRIPT = ( + REPOSITORY_ROOT + / "webster" + / "deepseek-v41" + / "scripts" + / "long-context-qualification.py" +) + + +class LongContextQualificationTests(unittest.TestCase): + def setUp(self) -> None: + self.temporary = tempfile.TemporaryDirectory() + self.root = Path(self.temporary.name) + self.key_file = self.root / "canary.key" + self.key_file.write_text("test-long-context-key\n", encoding="utf-8") + os.chmod(self.key_file, 0o600) + + def tearDown(self) -> None: + self.temporary.cleanup() + + def run_runner( + self, + base_url: str, + output: Path, + *, + expect_success: bool, + targets: tuple[int, ...] = (64, 96), + max_model_len: int = 128, + ) -> subprocess.CompletedProcess[str]: + command = [ + sys.executable, + str(SCRIPT), + "--base-url", + base_url, + "--model", + "deepseek-ai/DeepSeek-V4.1-Flash", + "--key-file", + str(self.key_file), + "--max-model-len", + str(max_model_len), + "--output-json", + str(output), + ] + for target in targets: + command.extend(("--target-context", str(target))) + result = subprocess.run( + command, + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + ) + if expect_success and result.returncode != 0: + self.fail(result.stderr) + if not expect_success and result.returncode == 0: + self.fail(result.stdout) + self.assertNotIn("test-long-context-key", result.stdout + result.stderr) + return result + + def test_exact_context_targets_and_over_limit_rejection_are_qualified(self) -> None: + chat_totals: list[int] = [] + + class Handler(BaseHTTPRequestHandler): + def log_message(self, *_: object) -> None: + return + + def send_json(self, status: int, value: object) -> None: + encoded = json.dumps(value).encode() + self.send_response(status) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(encoded))) + self.end_headers() + self.wfile.write(encoded) + + def do_POST(self) -> None: # noqa: N802 - stdlib handler API + length = int(self.headers.get("Content-Length", "0")) + request = json.loads(self.rfile.read(length)) + if self.headers.get("Authorization") != "Bearer test-long-context-key": + self.send_json(401, {"error": {"message": "unauthorized"}}) + return + content = request["messages"][0]["content"] + prompt_tokens = 4 + content.count(" x") + if self.path == "/tokenize": + if request.get("chat_template_kwargs") != { + "enable_thinking": False + }: + self.send_json( + 400, + {"error": {"message": "thinking template mismatch"}}, + ) + return + self.send_json( + 200, + { + "count": prompt_tokens, + "max_model_len": 128, + "tokens": [], + "token_strs": [], + }, + ) + return + self.assert_chat_request(request) + total = prompt_tokens + request["max_tokens"] + chat_totals.append(total) + if total > 128: + self.send_json( + 400, + {"error": {"message": "maximum context length is 128 tokens"}}, + ) + return + self.send_json( + 200, + { + "choices": [ + { + "message": {"role": "assistant", "content": "x"}, + "finish_reason": "length", + } + ], + "usage": { + "prompt_tokens": prompt_tokens, + "completion_tokens": 1, + "total_tokens": total, + }, + }, + ) + + def assert_chat_request(self, request: dict[str, object]) -> None: + testcase.assertEqual(self.path, "/v1/chat/completions") + testcase.assertEqual(request["reasoning_effort"], "none") + testcase.assertEqual(request["temperature"], 0) + testcase.assertEqual(request["max_tokens"], 1) + + testcase = self + server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + output = self.root / "long-context.json" + try: + self.run_runner( + f"http://127.0.0.1:{server.server_port}", + output, + expect_success=True, + ) + finally: + server.shutdown() + thread.join(timeout=5) + server.server_close() + + self.assertEqual(chat_totals, [64, 96, 129]) + artifact = json.loads(output.read_text(encoding="utf-8")) + self.assertEqual(output.stat().st_mode & 0o777, 0o600) + self.assertTrue(artifact["ok"]) + self.assertEqual(artifact["max_model_len"], 128) + self.assertEqual( + [case["target_context_tokens"] for case in artifact["accepted"]], + [64, 96], + ) + self.assertTrue(all(case["status"] == 200 for case in artifact["accepted"])) + self.assertTrue( + all(case["server_prompt_tokens"] == case["expected_prompt_tokens"] for case in artifact["accepted"]) + ) + self.assertEqual(artifact["over_limit"]["target_context_tokens"], 129) + self.assertEqual(artifact["over_limit"]["status"], 400) + self.assertNotIn( + "test-long-context-key", output.read_text(encoding="utf-8") + ) + + def test_tokenizer_mismatch_fails_before_chat_and_records_evidence(self) -> None: + chat_requests = 0 + + class Handler(BaseHTTPRequestHandler): + def log_message(self, *_: object) -> None: + return + + def do_POST(self) -> None: # noqa: N802 - stdlib handler API + nonlocal chat_requests + length = int(self.headers.get("Content-Length", "0")) + request = json.loads(self.rfile.read(length)) + if self.path != "/tokenize": + chat_requests += 1 + repetitions = request["messages"][0]["content"].count(" x") + count = 4 + repetitions + if repetitions == 59: + count -= 1 + encoded = json.dumps( + { + "count": count, + "max_model_len": 128, + "tokens": [], + "token_strs": [], + } + ).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(encoded))) + self.end_headers() + self.wfile.write(encoded) + + server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + output = self.root / "mismatch.json" + try: + self.run_runner( + f"http://127.0.0.1:{server.server_port}", + output, + expect_success=False, + targets=(64,), + ) + finally: + server.shutdown() + thread.join(timeout=5) + server.server_close() + + self.assertEqual(chat_requests, 0) + artifact = json.loads(output.read_text(encoding="utf-8")) + self.assertFalse(artifact["ok"]) + self.assertEqual(artifact["accepted"][0]["tokenize_status"], 200) + self.assertFalse(artifact["accepted"][0]["token_count_exact"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/webster/deepseek-v41/tests/test_private_acceptance.py b/webster/deepseek-v41/tests/test_private_acceptance.py new file mode 100644 index 0000000..895f074 --- /dev/null +++ b/webster/deepseek-v41/tests/test_private_acceptance.py @@ -0,0 +1,1323 @@ +#!/usr/bin/env python3 +"""Behavior tests for the publication private-acceptance gate.""" + +from __future__ import annotations + +import hashlib +import json +import os +import subprocess +import sys +import tempfile +import time +import unittest +from pathlib import Path + + +REPOSITORY_ROOT = Path(__file__).resolve().parents[3] +VERIFIER = ( + REPOSITORY_ROOT + / "webster" + / "deepseek-v41" + / "scripts" + / "verify-private-acceptance.py" +) +GENERATOR = ( + REPOSITORY_ROOT + / "webster" + / "deepseek-v41" + / "scripts" + / "generate-private-acceptance.py" +) +NOW = "2099-01-01T12:00:00Z" +ACCEPTED_AT = "2099-01-01T11:55:00Z" +MODEL = "deepseek-ai/DeepSeek-V4.1-Flash" +IMAGE = "sha256:3863bf0f59bd4df4012b7b7aed8a7d2be6ef9b42ee3a5299d014f83a1f1ba6ea" +CHECKPOINT_REVISION = "dba1be0a40aa45a94ad051997016db3960a90277" +VLLM_COMMIT = "e77daef89e18e08321ae7b8b24827eedd5fe8673" +AIPERF_COMMIT = "ea28b2e81c7367f8403c6f6ebe0837a508795a8d" +AIPERF_WEKA_PATCH_SHA256 = ( + "9557a46afac031b64f66ea64ebac4e8ed4d1a0e142354388c19c086a9377a36e" +) +WEKA_REPOSITORY = "semianalysisai/cc-traces-weka-062126" +WEKA_REVISION = "23f152f6f0f9399a85901b89a6458def0ef16729" +SERVING_GENERATION = "deepseek-v41-accepted-generation" +LEGACY_SERVING_GENERATION = ( + "legacy-e11a1b895929ef85ea53d5014becaeeb7dee3099dbb578325cfc94aaf78d2cd1" +) +FIXTURE_CHECKPOINT_SHA256 = hashlib.sha256(b"checkpoint fixture\n").hexdigest() +EVIDENCE_EPOCH = time.mktime( + time.strptime("2099-01-01T11:54:00Z", "%Y-%m-%dT%H:%M:%SZ") +) +FEATURE_CASES = { + "deterministic-text", + "utf8", + "stop-sequence", + "streaming", + "usage-accounting", + "reasoning-on", + "tool-auto", + "tool-named", + "tool-required", + "tool-parallel", + "tool-result", + "structured", + "json-object", + "vision", + "malformed-input", + "wrong-model", + "disconnect-queue-drain", +} + + +def sha256(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def write_private(path: Path, value: object) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(value, sort_keys=True) + "\n", encoding="utf-8") + os.chmod(path, 0o600) + + +def mark_evidence_time(path: Path) -> None: + os.utime(path, (EVIDENCE_EPOCH, EVIDENCE_EPOCH)) + + +def build_fixture(base: Path) -> tuple[Path, Path]: + run_root = base / "runs" / "20990101T000000Z" + (run_root / "profiles").mkdir(parents=True) + (run_root / "baseline").mkdir() + (run_root / "aiperf" / "scored-summaries").mkdir(parents=True) + (run_root / "logs").mkdir() + + manifest = run_root / "manifest.env" + manifest.write_text( + f"CHECKPOINT_REPO={MODEL}\n" + f"CHECKPOINT_REVISION={CHECKPOINT_REVISION}\n" + "CHECKPOINT_SHARDS=48\n" + f"VLLM_COMMIT={VLLM_COMMIT}\n" + f"VLLM_IMAGE_ID={IMAGE}\n" + f"AIPERF_COMMIT={AIPERF_COMMIT}\n" + f"WEKA_REPOSITORY={WEKA_REPOSITORY}\n" + f"WEKA_REVISION={WEKA_REVISION}\n", + encoding="utf-8", + ) + os.chmod(manifest, 0o600) + + profile = run_root / "profiles" / "dspark-1m-noautotune-seqs4.env" + profile.write_text( + "PROFILE_NAME=dspark-1m-noautotune-seqs4\n" + "MAX_MODEL_LEN=1048576\n" + "MAX_NUM_SEQS=4\n" + "GPU_MEMORY_UTILIZATION=0.90\n" + "ENFORCE_EAGER=0\n" + "ENABLE_PREFIX_CACHING=1\n" + "MAX_NUM_BATCHED_TOKENS=16384\n" + "ENABLE_EXPERT_PARALLEL=0\n" + "SPECULATIVE_METHOD=dspark\n" + "NUM_SPECULATIVE_TOKENS=5\n" + "DRAFT_SAMPLE_METHOD=greedy\n" + "REJECTION_SAMPLE_METHOD=standard\n" + "ENABLE_FLASHINFER_AUTOTUNE=0\n", + encoding="utf-8", + ) + os.chmod(profile, 0o600) + + checkpoint = run_root / "baseline" / "shamu-checkpoint.manifest" + checkpoint.write_text("checkpoint fixture\n", encoding="utf-8") + os.chmod(checkpoint, 0o600) + checkpoint_peer = run_root / "baseline" / "tilikum-checkpoint.manifest" + checkpoint_peer.write_bytes(checkpoint.read_bytes()) + os.chmod(checkpoint_peer, 0o600) + + feature = run_root / "feature-dspark-1m-seqs4-final.json" + feature_payload = { + "schema_version": 1, + "model": MODEL, + "stage": "dspark", + "ok": True, + "failure_log_capture": { + "requested": True, + "captured": False, + "ok": True, + "path": str(run_root / "logs" / "feature-failure-server.log"), + "nodes": ["shamu", "tilikum"], + "container": "deepseek-v41-flash-tp2", + }, + "cases": [ + { + "case": name, + "status": { + "malformed-input": 400, + "wrong-model": 404, + }.get(name, 200), + "semantic_ok": True, + "error": None, + "repetitions": 3 if name == "deterministic-text" else 1, + "distinct_normalized_outputs": 1, + **( + { + "client_disconnected": True, + "queue_drained": True, + "queue_baseline": {"running": 0.0, "waiting": 0.0}, + "queue_final": {"running": 0.0, "waiting": 0.0}, + } + if name == "disconnect-queue-drain" + else {} + ), + } + for name in sorted(FEATURE_CASES) + ], + } + write_private(feature, feature_payload) + + long_context = run_root / "long-context-dspark-1m-seqs4-final.json" + targets = [320000, 600000, 990016, 996579] + write_private( + long_context, + { + "schema_version": 1, + "model": MODEL, + "max_model_len": 1048576, + "ok": True, + "accepted": [ + { + "target_context_tokens": target, + "server_total_tokens": target, + "status": 200, + "semantic_ok": True, + "token_count_exact": True, + **( + { + "expected_prompt_tokens": 319999, + "server_prompt_tokens": 319999, + "server_completion_tokens": 1, + "latency_seconds": 21.266248, + } + if target == 320000 + else {} + ), + } + for target in targets + ], + "over_limit": { + "target_context_tokens": 1048577, + "status": 400, + "semantic_ok": True, + "token_count_exact": True, + }, + "tokenizer_calibration": { + "reported_max_model_len": 1048576, + "exact_linear_unit": True, + }, + }, + ) + + weka = run_root / "aiperf" / "scored-summaries" / "seqs4.json" + weka_payload = { + "schema_version": 1, + "qualified": True, + "requests": 396, + "completed": 396, + "errors": 0, + "server_success_rate": 1.0, + "request_throughput": 0.44, + "output_token_throughput": 202.0287585576069, + "ttft_p90_ms": 1067.0, + "request_peak_concurrency": 3, + "preemptions_total": 0.0, + "waiting_requests_max": 0.0, + "kv_cache_usage_max": 0.02, + "prefix_cache_hits": 100.0, + "prefix_cache_queries": 110.0, + "prefix_cache_hit_ratio": 100.0 / 110.0, + "running_requests_max": 3.0, + "wall_power_avg_watts": 1477.4115518641565, + "facility_power_avg_watts": 1477.4115518641565, + "facility_energy_kwh": 0.37, + "facility_cost_dollars": 0.17, + "completed_root_workflows": 3, + "cancelled_root_workflows": 0, + "root_workflow_elapsed_seconds": 900.0, + "completed_root_workflows_per_second": 3.0 / 900.0, + "input_cost_per_token": 1.3e-8, + "output_cost_per_token": 9.6e-7, + "engine_deaths": 0, + "nccl_timeouts": 0, + "corruption_flags": 0, + } + write_private(weka, weka_payload) + + pricing = run_root / "pricing-evidence.json" + pricing_payload = { + "schema_version": 1, + "model": MODEL, + "serving_generation": SERVING_GENERATION, + "electricity_rate_usd_per_kwh": 0.47, + "facility_power_watts": 1477.4115518641565, + "input": { + "basis": "cold-prefill", + "prompt_tokens": 319999, + "elapsed_seconds": 21.266248, + "throughput_tokens_per_second": 15047.271150040195, + "direct_energy_cost_per_token": 1.2818555870663976e-8, + "published_cost_per_token": 1.3e-8, + }, + "output": { + "basis": "selected-weka-profile", + "throughput_tokens_per_second": 202.0287585576069, + "direct_energy_cost_per_token": 9.547367776495098e-7, + "published_cost_per_token": 9.6e-7, + }, + } + write_private(pricing, pricing_payload) + + rederived_weka = { + key: value + for key, value in weka_payload.items() + if key + not in { + "input_cost_per_token", + "output_cost_per_token", + "engine_deaths", + "nccl_timeouts", + "corruption_flags", + } + } + rederived_weka.update( + { + "error_types": {}, + "root_session_trees": 3, + "request_count_by_agent_depth": {"0": 396}, + } + ) + + def metric_series( + metric_type: str, + stat: str, + value: float, + *, + labels: dict[str, str] | None = None, + ) -> dict[str, object]: + return { + "type": metric_type, + "series": [ + { + "endpoint_url": "http://fixture/metrics", + "labels": labels or {}, + "stats": {stat: value}, + } + ], + } + + server_metrics = { + "metrics": { + "vllm:prefix_cache_hits": metric_series("counter", "total", 100.0), + "vllm:prefix_cache_queries": metric_series("counter", "total", 110.0), + "vllm:kv_cache_usage_perc": metric_series("gauge", "max", 0.02), + "vllm:num_preemptions": metric_series("counter", "total", 0.0), + "vllm:num_requests_running": metric_series("gauge", "max", 3.0), + "vllm:num_requests_waiting": metric_series("gauge", "max", 0.0), + "power_total_watts": { + "type": "gauge", + "series": [ + { + "endpoint_url": "http://fixture/metrics", + "labels": {"scope": scope}, + "stats": {"avg": 1477.4115518641565}, + } + for scope in ("wall", "facility") + ], + }, + "power_energy_kwh": metric_series( + "counter", "total", 0.37, labels={"scope": "facility"} + ), + "power_cost_dollars": metric_series("counter", "total", 0.17), + } + } + + weka_repetitions = [] + warmup_provenance_sha256: str | None = None + for repetition in (1, 2): + repetition_dir = ( + run_root + / "aiperf" + / f"dspark-1m-noautotune-seqs4-c4-r{repetition}-open" + ) + repetition_dir.mkdir() + artifacts = { + "EXIT": "EXIT=0\n", + ".aiperf_results_ready.json": json.dumps( + { + "ready": True, + "was_cancelled": False, + "partial": False, + "failed_exporters": [], + }, + sort_keys=True, + ) + + "\n", + "profile_export.jsonl": "".join( + json.dumps( + { + "status": 200, + "error": None, + "output": "OK", + "metadata": { + "request_start_ns": index, + "request_end_ns": index + 3, + "agent_depth": 0, + "root_correlation_id": f"root-{index % 3}", + }, + "metrics": {}, + }, + sort_keys=True, + ) + + "\n" + for index in range(396) + ), + "profile_export_aiperf.json": json.dumps( + { + "request_throughput": {"avg": 0.44}, + "time_to_first_token": {"p90": 1067.0}, + "output_token_throughput": {"avg": 202.0287585576069}, + }, + sort_keys=True, + ) + + "\n", + "server_metrics_export.json": json.dumps(server_metrics, sort_keys=True) + + "\n", + "aiperf.stdout.log": ( + "05:31:54.473 NOTICE Phase profiling (profiling) complete | " + "completed=396, cancelled=0, errors=0 | sessions: completed=3, " + "cancelled=0 | elapsed=900.00s (runner.py:1227)\n" + ), + "aiperf.stderr.log": "", + "rendered-command.txt": "aiperf profile --config aiperf-config.yaml\n", + "summary.json": json.dumps(rederived_weka, sort_keys=True) + "\n", + "benchmark-window.json": json.dumps( + { + "started_at": "2099-01-01T11:30:00Z", + "ended_at": "2099-01-01T11:45:00Z", + }, + sort_keys=True, + ) + + "\n", + "rank0-before.json": json.dumps( + { + "container_id": "0" * 64, + "generation": SERVING_GENERATION, + "image": IMAGE, + "restart_count": 0, + "started_at": "2099-01-01T00:00:00Z", + }, + sort_keys=True, + ) + + "\n", + "rank1-before.json": json.dumps( + { + "container_id": "1" * 64, + "generation": SERVING_GENERATION, + "image": IMAGE, + "restart_count": 0, + "started_at": "2099-01-01T00:00:01Z", + }, + sort_keys=True, + ) + + "\n", + "rank0-after.json": json.dumps( + { + "container_id": "0" * 64, + "generation": SERVING_GENERATION, + "image": IMAGE, + "restart_count": 0, + "started_at": "2099-01-01T00:00:00Z", + }, + sort_keys=True, + ) + + "\n", + "rank1-after.json": json.dumps( + { + "container_id": "1" * 64, + "generation": SERVING_GENERATION, + "image": IMAGE, + "restart_count": 0, + "started_at": "2099-01-01T00:00:01Z", + }, + sort_keys=True, + ) + + "\n", + "rank0-container.log": "healthy rank 0\n", + "rank1-container.log": "healthy rank 1\n", + } + for name, content in artifacts.items(): + path = repetition_dir / name + path.write_text(content, encoding="utf-8") + os.chmod(path, 0o600) + provenance = repetition_dir / "provenance.json" + write_private( + provenance, + { + "schema_version": 1, + "profile_name": "dspark-1m-noautotune-seqs4", + "concurrency": 4, + "repetition": repetition, + "cache_state": "warmup" if repetition == 1 else "warm", + "warmup_and_scored_comparable": repetition == 2, + "warmup_provenance_sha256": ( + warmup_provenance_sha256 if repetition == 2 else None + ), + "aiperf_commit": AIPERF_COMMIT, + "aiperf_weka_patch_sha256": AIPERF_WEKA_PATCH_SHA256, + "weka_repository": WEKA_REPOSITORY, + "weka_revision": WEKA_REVISION, + "trace_count": 393, + "full_subagents": True, + "serving_generation": SERVING_GENERATION, + "engine_deaths": 0, + "nccl_timeouts": 0, + "corruption_flags": 0, + "rank_fatal_error_matches": 0, + "artifacts": { + name: sha256(repetition_dir / name) for name in artifacts + }, + }, + ) + weka_repetitions.append( + {"path": str(provenance), "sha256": sha256(provenance)} + ) + if repetition == 1: + warmup_provenance_sha256 = sha256(provenance) + + serving_args = [ + "serve", + "/model", + "--served-model-name", + MODEL, + "--nnodes", + "2", + "--master-addr", + "10.10.1.1", + "--master-port", + "29511", + "--distributed-executor-backend", + "mp", + "--tensor-parallel-size", + "2", + "--pipeline-parallel-size", + "1", + "--engram-config", + '{"cpu_offload":true,"embedding_across_dp":false}', + "--max-model-len", + "1048576", + "--max-num-seqs", + "4", + "--gpu-memory-utilization", + "0.90", + "--disable-custom-all-reduce", + "--enable-auto-tool-choice", + "--tool-call-parser", + "deepseek_v41", + "--reasoning-parser", + "deepseek_v41", + "--enable-prefix-caching", + "--max-num-batched-tokens", + "16384", + "--no-enable-flashinfer-autotune", + "--speculative-config", + ( + '{"method":"dspark","num_speculative_tokens":5,' + '"draft_sample_method":"greedy",' + '"rejection_sample_method":"standard"}' + ), + ] + rank0 = run_root / "logs" / "private-acceptance-shamu.log" + rank0.write_text( + f"running=true restart=0 image={IMAGE} args=" + + json.dumps( + serving_args + + [ + "--node-rank", + "0", + "--host", + "100.73.140.127", + "--port", + "8000", + ], + separators=(",", ":"), + ) + + "\n" + f"generation={SERVING_GENERATION}\n" + "serving_key_vars=1\nhealth=pass\n" + "LISTEN 0 2048 100.73.140.127:8000 0.0.0.0:*\n" + "GPU KV cache size: 26,028,861 tokens, Maximum concurrency for " + "1,048,576 tokens per request: 24.82x\n", + encoding="utf-8", + ) + os.chmod(rank0, 0o600) + rank1 = run_root / "logs" / "private-acceptance-tilikum.log" + rank1.write_text( + f"running=true restart=0 image={IMAGE} args=" + + json.dumps( + serving_args + ["--node-rank", "1", "--headless"], + separators=(",", ":"), + ) + + "\n" + f"generation={SERVING_GENERATION}\n" + "serving_key_vars=0\nlistener=none\nfatal_error_matches=0\n", + encoding="utf-8", + ) + os.chmod(rank1, 0o600) + api = run_root / "logs" / "private-acceptance-api.log" + api.write_text( + json.dumps( + { + "status": 200, + "missing_key_status": 401, + "wrong_key_status": 401, + "correct_key_status": 200, + "model": MODEL, + "has_content": True, + "finish_reason": "stop", + "serving_generation": SERVING_GENERATION, + } + ) + + "\n", + encoding="utf-8", + ) + os.chmod(api, 0o600) + for evidence in ( + feature, + long_context, + weka, + rank0, + rank1, + api, + pricing, + *(Path(item["path"]) for item in weka_repetitions), + ): + mark_evidence_time(evidence) + + acceptance = run_root / "private-acceptance.json" + write_private( + acceptance, + { + "schema_version": 1, + "accepted_at": ACCEPTED_AT, + "decision": "GO", + "model": MODEL, + "public_model_name": "deepseek-v4.1-flash", + "selected_profile": { + "name": "dspark-1m-noautotune-seqs4", + "path": str(profile), + "sha256": sha256(profile), + "max_model_len": 1048576, + "max_num_seqs": 4, + "speculative_method": "dspark", + "flashinfer_autotune": False, + }, + "runtime": { + "checkpoint_revision": CHECKPOINT_REVISION, + "checkpoint_manifest_sha256": sha256(checkpoint), + "vllm_commit": VLLM_COMMIT, + "image_id": IMAGE, + "manifest_path": str(manifest), + "manifest_sha256": sha256(manifest), + }, + "topology": { + "tensor_parallel_size": 2, + "pipeline_parallel_size": 1, + "rank_0": "shamu", + "rank_1": "tilikum", + "rank_1_headless_and_keyless": True, + }, + "feature_qualification": { + "path": str(feature), + "sha256": sha256(feature), + "ok": True, + "passed": len(FEATURE_CASES), + "total": len(FEATURE_CASES), + }, + "long_context_qualification": { + "path": str(long_context), + "sha256": sha256(long_context), + "ok": True, + "accepted_context_totals": targets, + "rejected_context_total": 1048577, + }, + "weka_qualification": { + "path": str(weka), + "sha256": sha256(weka), + "qualified": True, + "repetitions": weka_repetitions, + **{ + name: weka_payload[name] + for name in ( + "requests", + "errors", + "server_success_rate", + "request_throughput", + "output_token_throughput", + "ttft_p90_ms", + "request_peak_concurrency", + "preemptions_total", + ) + }, + }, + "live_state": { + "restart_counts_zero": True, + "fatal_error_matches": 0, + "kv_cache_tokens": 26028861, + "full_context_concurrency": 24.82, + "rank_0_evidence": {"path": str(rank0), "sha256": sha256(rank0)}, + "rank_1_evidence": {"path": str(rank1), "sha256": sha256(rank1)}, + "api_evidence": {"path": str(api), "sha256": sha256(api)}, + "serving_generation": SERVING_GENERATION, + }, + "selection": { + "pp2_triggered": False, + "pp2_reason": "TP2/PP1 passed the qualification gates", + "seqs_8_request_throughput_gain_percent": 0.89, + "reason": "seqs=4 retains KV headroom within the throughput tolerance", + }, + "pricing_evidence": { + "path": str(pricing), + "sha256": sha256(pricing), + }, + "publication": { + "public_model_name": "deepseek-v4.1-flash", + "served_model_name": MODEL, + "max_context": 1048576, + "station_endpoint": "http://100.73.140.127:8000/v1", + "supports_function_calling": True, + "supports_reasoning": True, + "supports_response_schema": True, + "supports_vision": True, + "input_cost_per_token": 1.3e-8, + "output_cost_per_token": 9.6e-7, + }, + }, + ) + return run_root, acceptance + + +def verify(run_root: Path, acceptance: Path) -> subprocess.CompletedProcess[str]: + return subprocess.run( + [ + sys.executable, + str(VERIFIER), + "--run-root", + str(run_root), + "--acceptance", + str(acceptance), + "--now", + NOW, + "--max-age-seconds", + "21600", + ], + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + env={ + **os.environ, + "WEBSTER_ACCEPTANCE_TEST_MODE": "1", + "TEST_CHECKPOINT_MANIFEST_SHA256": FIXTURE_CHECKPOINT_SHA256, + }, + ) + + +class PrivateAcceptanceTests(unittest.TestCase): + def test_generator_builds_a_validator_accepted_manifest_last(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + original = json.loads(acceptance.read_text(encoding="utf-8")) + acceptance.unlink() + evidence = original["live_state"] + current_epoch = time.time() + for item in evidence.values(): + if isinstance(item, dict) and "path" in item: + os.utime(item["path"], (current_epoch, current_epoch)) + for section in ( + "feature_qualification", + "long_context_qualification", + "weka_qualification", + ): + os.utime( + original[section]["path"], (current_epoch, current_epoch) + ) + for item in original["weka_qualification"]["repetitions"]: + os.utime(item["path"], (current_epoch, current_epoch)) + os.utime( + original["pricing_evidence"]["path"], + (current_epoch, current_epoch), + ) + result = subprocess.run( + [ + sys.executable, + str(GENERATOR), + "--run-root", + str(run_root), + "--rank-0-evidence", + evidence["rank_0_evidence"]["path"], + "--rank-1-evidence", + evidence["rank_1_evidence"]["path"], + "--api-evidence", + evidence["api_evidence"]["path"], + "--seqs-8-request-throughput-gain-percent", + "0.89", + "--selection-reason", + original["selection"]["reason"], + "--pp2-reason", + original["selection"]["pp2_reason"], + ], + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + env={ + **os.environ, + "WEBSTER_ACCEPTANCE_TEST_MODE": "1", + "TEST_CHECKPOINT_MANIFEST_SHA256": FIXTURE_CHECKPOINT_SHA256, + }, + ) + self.assertEqual(result.returncode, 0, result.stderr) + generated = json.loads(acceptance.read_text(encoding="utf-8")) + self.assertEqual(acceptance.stat().st_mode & 0o777, 0o600) + self.assertEqual(generated["feature_qualification"]["total"], 17) + self.assertEqual(generated["live_state"]["kv_cache_tokens"], 26028861) + self.assertEqual(generated["live_state"]["full_context_concurrency"], 24.82) + accepted_at = time.mktime( + time.strptime(generated["accepted_at"], "%Y-%m-%dT%H:%M:%SZ") + ) + self.assertGreaterEqual(accepted_at + 5, max( + Path(item["path"]).stat().st_mtime + for item in evidence.values() + if isinstance(item, dict) and "path" in item + )) + + def test_complete_private_acceptance_is_valid(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + result = verify(run_root, acceptance) + self.assertEqual(result.returncode, 0, result.stderr) + + def test_unlabeled_legacy_weka_rank_states_use_the_derived_pair_generation(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + payload = json.loads(acceptance.read_text(encoding="utf-8")) + + warmup_hash = None + for repetition in payload["weka_qualification"]["repetitions"]: + provenance_path = Path(repetition["path"]) + provenance = json.loads( + provenance_path.read_text(encoding="utf-8") + ) + for rank in (0, 1): + for phase in ("before", "after"): + name = f"rank{rank}-{phase}.json" + state_path = provenance_path.parent / name + state = json.loads(state_path.read_text(encoding="utf-8")) + self.assertEqual(state["generation"], SERVING_GENERATION) + state["generation"] = None + write_private(state_path, state) + provenance["artifacts"][name] = sha256(state_path) + provenance["serving_generation"] = LEGACY_SERVING_GENERATION + if provenance["repetition"] == 2: + provenance["warmup_provenance_sha256"] = warmup_hash + write_private(provenance_path, provenance) + mark_evidence_time(provenance_path) + repetition["sha256"] = sha256(provenance_path) + if provenance["repetition"] == 1: + warmup_hash = repetition["sha256"] + + live = payload["live_state"] + live["serving_generation"] = LEGACY_SERVING_GENERATION + for key in ("rank_0_evidence", "rank_1_evidence"): + item = live[key] + path = Path(item["path"]) + text = path.read_text(encoding="utf-8").replace( + f"generation={SERVING_GENERATION}", + f"generation={LEGACY_SERVING_GENERATION}", + ) + path.write_text(text, encoding="utf-8") + os.chmod(path, 0o600) + mark_evidence_time(path) + item["sha256"] = sha256(path) + + api_item = live["api_evidence"] + api_path = Path(api_item["path"]) + api = json.loads(api_path.read_text(encoding="utf-8")) + api["serving_generation"] = LEGACY_SERVING_GENERATION + write_private(api_path, api) + mark_evidence_time(api_path) + api_item["sha256"] = sha256(api_path) + + pricing_item = payload["pricing_evidence"] + pricing_path = Path(pricing_item["path"]) + pricing = json.loads(pricing_path.read_text(encoding="utf-8")) + pricing["serving_generation"] = LEGACY_SERVING_GENERATION + write_private(pricing_path, pricing) + mark_evidence_time(pricing_path) + pricing_item["sha256"] = sha256(pricing_path) + write_private(acceptance, payload) + + result = verify(run_root, acceptance) + + self.assertEqual(result.returncode, 0, result.stderr) + + def test_pricing_does_not_require_cost_fields_in_weka_summary(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + payload = json.loads(acceptance.read_text(encoding="utf-8")) + item = payload["weka_qualification"] + path = Path(item["path"]) + weka = json.loads(path.read_text(encoding="utf-8")) + weka.pop("input_cost_per_token") + weka.pop("output_cost_per_token") + write_private(path, weka) + mark_evidence_time(path) + item["sha256"] = sha256(path) + write_private(acceptance, payload) + + result = verify(run_root, acceptance) + + self.assertEqual(result.returncode, 0, result.stderr) + + def test_pricing_accepts_a_fresh_remeasured_cold_prefill(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + payload = json.loads(acceptance.read_text(encoding="utf-8")) + + context_item = payload["long_context_qualification"] + context_path = Path(context_item["path"]) + context = json.loads(context_path.read_text(encoding="utf-8")) + context["accepted"][0]["latency_seconds"] = 20.907312 + write_private(context_path, context) + mark_evidence_time(context_path) + context_item["sha256"] = sha256(context_path) + + pricing_item = payload["pricing_evidence"] + pricing_path = Path(pricing_item["path"]) + pricing = json.loads(pricing_path.read_text(encoding="utf-8")) + pricing["input"]["elapsed_seconds"] = 20.907312 + pricing["input"]["throughput_tokens_per_second"] = 15305.602174014526 + pricing["input"]["direct_energy_cost_per_token"] = ( + 1.2602201713127931e-8 + ) + write_private(pricing_path, pricing) + mark_evidence_time(pricing_path) + pricing_item["sha256"] = sha256(pricing_path) + write_private(acceptance, payload) + + result = verify(run_root, acceptance) + + self.assertEqual(result.returncode, 0, result.stderr) + + def test_live_rank_arguments_must_match_the_selected_profile_exactly(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + payload = json.loads(acceptance.read_text(encoding="utf-8")) + item = payload["live_state"]["rank_0_evidence"] + path = Path(item["path"]) + evidence = path.read_text(encoding="utf-8").replace( + '"--max-num-batched-tokens","16384"', + '"--max-num-batched-tokens","8192"', + ) + self.assertNotEqual(evidence, path.read_text(encoding="utf-8")) + path.write_text(evidence, encoding="utf-8") + os.chmod(path, 0o600) + mark_evidence_time(path) + item["sha256"] = sha256(path) + write_private(acceptance, payload) + + result = verify(run_root, acceptance) + + self.assertNotEqual(result.returncode, 0) + self.assertIn("serving arguments", result.stderr) + + def test_malformed_private_acceptance_is_rejected(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + acceptance.write_text("{not-json\n", encoding="utf-8") + result = verify(run_root, acceptance) + self.assertNotEqual(result.returncode, 0) + self.assertIn("malformed", result.stderr) + + def test_private_acceptance_must_be_mode_0600(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + os.chmod(acceptance, 0o644) + result = verify(run_root, acceptance) + self.assertNotEqual(result.returncode, 0) + self.assertIn("mode 0600", result.stderr) + + def test_stale_private_acceptance_is_rejected(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + payload = json.loads(acceptance.read_text(encoding="utf-8")) + payload["accepted_at"] = "2099-01-01T00:00:00Z" + write_private(acceptance, payload) + result = verify(run_root, acceptance) + self.assertNotEqual(result.returncode, 0) + self.assertIn("stale", result.stderr) + + def test_every_decision_artifact_requires_both_freshness_bounds(self) -> None: + for relation, epoch, expected in ( + ("too old", "2099-01-01T05:54:59Z", "stale"), + ("future", "2099-01-01T11:55:06Z", "future"), + ): + for section in ( + "feature_qualification", + "long_context_qualification", + "weka_qualification", + ): + with self.subTest(relation=relation, section=section), tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + payload = json.loads(acceptance.read_text(encoding="utf-8")) + path = Path(payload[section]["path"]) + timestamp = time.mktime(time.strptime(epoch, "%Y-%m-%dT%H:%M:%SZ")) + os.utime(path, (timestamp, timestamp)) + result = verify(run_root, acceptance) + self.assertNotEqual(result.returncode, 0) + self.assertIn(expected, result.stderr) + + def test_future_acceptance_cannot_extend_evidence_clock_skew(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + payload = json.loads(acceptance.read_text(encoding="utf-8")) + payload["accepted_at"] = "2099-01-01T12:00:05Z" + feature = Path(payload["feature_qualification"]["path"]) + future = time.mktime( + time.strptime("2099-01-01T12:00:10Z", "%Y-%m-%dT%H:%M:%SZ") + ) + os.utime(feature, (future, future)) + write_private(acceptance, payload) + + result = verify(run_root, acceptance) + + self.assertNotEqual(result.returncode, 0) + self.assertIn("future", result.stderr) + + def test_weka_publication_requires_two_complete_pinned_raw_repetitions(self) -> None: + mutations = ( + ("both repetitions", lambda root, value: value["weka_qualification"].pop("repetitions")), + ( + "393-trace", + lambda root, value: self._mutate_weka_provenance( + value, 0, "trace_count", 392 + ), + ), + ( + "raw artifact", + lambda root, value: ( + Path(value["weka_qualification"]["repetitions"][0]["path"]).parent + / "profile_export.jsonl" + ).unlink(), + ), + ( + "serving generation", + lambda root, value: self._mutate_weka_provenance( + value, 1, "serving_generation", "other-generation" + ), + ), + ) + for expected, mutate in mutations: + with self.subTest(expected=expected), tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + payload = json.loads(acceptance.read_text(encoding="utf-8")) + mutate(run_root, payload) + write_private(acceptance, payload) + result = verify(run_root, acceptance) + self.assertNotEqual(result.returncode, 0) + self.assertIn(expected, result.stderr) + + def test_weka_publication_requires_exact_aiperf_runtime_patch(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + payload = json.loads(acceptance.read_text(encoding="utf-8")) + self._mutate_weka_provenance( + payload, + 1, + "aiperf_weka_patch_sha256", + "0" * 64, + ) + write_private(acceptance, payload) + + result = verify(run_root, acceptance) + + self.assertNotEqual(result.returncode, 0) + self.assertIn("runtime patch", result.stderr) + + def test_weka_publication_rederives_request_success_from_raw_records(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + payload = json.loads(acceptance.read_text(encoding="utf-8")) + item = payload["weka_qualification"]["repetitions"][0] + provenance_path = Path(item["path"]) + provenance = json.loads(provenance_path.read_text(encoding="utf-8")) + records = provenance_path.parent / "profile_export.jsonl" + write_private( + records, + {"status": 500, "error": "server failed", "corruption": False}, + ) + provenance["artifacts"]["profile_export.jsonl"] = sha256(records) + write_private(provenance_path, provenance) + mark_evidence_time(provenance_path) + item["sha256"] = sha256(provenance_path) + write_private(acceptance, payload) + + result = verify(run_root, acceptance) + + self.assertNotEqual(result.returncode, 0) + self.assertIn("raw records contain failed requests", result.stderr) + + def test_weka_publication_requires_hashed_paired_rank_logs(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + payload = json.loads(acceptance.read_text(encoding="utf-8")) + provenance_path = Path( + payload["weka_qualification"]["repetitions"][0]["path"] + ) + (provenance_path.parent / "rank1-container.log").unlink() + + result = verify(run_root, acceptance) + + self.assertNotEqual(result.returncode, 0) + self.assertIn("rank log", result.stderr) + + def test_weka_publication_matches_selected_score_to_rederived_repetition(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + payload = json.loads(acceptance.read_text(encoding="utf-8")) + item = payload["weka_qualification"] + scored_path = Path(item["path"]) + scored = json.loads(scored_path.read_text(encoding="utf-8")) + scored["request_throughput"] = 99.0 + write_private(scored_path, scored) + mark_evidence_time(scored_path) + item["request_throughput"] = 99.0 + item["sha256"] = sha256(scored_path) + write_private(acceptance, payload) + + result = verify(run_root, acceptance) + + self.assertNotEqual(result.returncode, 0) + self.assertIn("rederived repetition 2", result.stderr) + + def test_weka_scored_repetition_is_bound_to_warmup_provenance(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + payload = json.loads(acceptance.read_text(encoding="utf-8")) + item = payload["weka_qualification"]["repetitions"][1] + provenance_path = Path(item["path"]) + provenance = json.loads(provenance_path.read_text(encoding="utf-8")) + provenance["warmup_provenance_sha256"] = "0" * 64 + write_private(provenance_path, provenance) + mark_evidence_time(provenance_path) + item["sha256"] = sha256(provenance_path) + write_private(acceptance, payload) + + result = verify(run_root, acceptance) + + self.assertNotEqual(result.returncode, 0) + self.assertIn("warmup provenance", result.stderr) + + def test_station_auth_evidence_requires_missing_wrong_and_correct_key_proofs(self) -> None: + for field, value in ( + ("missing_key_status", None), + ("wrong_key_status", 200), + ("correct_key_status", 401), + ("serving_generation", "other-generation"), + ): + with self.subTest(field=field), tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + payload = json.loads(acceptance.read_text(encoding="utf-8")) + item = payload["live_state"]["api_evidence"] + path = Path(item["path"]) + api = json.loads(path.read_text(encoding="utf-8")) + if value is None: + api.pop(field) + else: + api[field] = value + write_private(path, api) + mark_evidence_time(path) + item["sha256"] = sha256(path) + write_private(acceptance, payload) + result = verify(run_root, acceptance) + self.assertNotEqual(result.returncode, 0) + self.assertIn("auth", result.stderr.lower()) + + def test_pricing_publication_requires_fresh_hashed_generation_bound_evidence(self) -> None: + def mutate_hash(root: Path, payload: dict[str, object]) -> None: + payload["pricing_evidence"]["sha256"] = "0" * 64 + + def mutate_value(root: Path, payload: dict[str, object]) -> None: + item = payload["pricing_evidence"] + path = Path(item["path"]) + pricing = json.loads(path.read_text(encoding="utf-8")) + pricing["input"]["published_cost_per_token"] = 1.4e-8 + write_private(path, pricing) + mark_evidence_time(path) + item["sha256"] = sha256(path) + + def mutate_stale(root: Path, payload: dict[str, object]) -> None: + path = Path(payload["pricing_evidence"]["path"]) + stale = time.mktime( + time.strptime("2099-01-01T05:54:59Z", "%Y-%m-%dT%H:%M:%SZ") + ) + os.utime(path, (stale, stale)) + + def mutate_generation(root: Path, payload: dict[str, object]) -> None: + item = payload["pricing_evidence"] + path = Path(item["path"]) + pricing = json.loads(path.read_text(encoding="utf-8")) + pricing["serving_generation"] = "other-generation" + write_private(path, pricing) + mark_evidence_time(path) + item["sha256"] = sha256(path) + + mutations = ( + ("absent", lambda root, payload: (root / "pricing-evidence.json").unlink()), + ("hash", mutate_hash), + ("value", mutate_value), + ("stale", mutate_stale), + ("generation", mutate_generation), + ) + for expected, mutate in mutations: + with self.subTest(expected=expected), tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + payload = json.loads(acceptance.read_text(encoding="utf-8")) + mutate(run_root, payload) + write_private(acceptance, payload) + + result = verify(run_root, acceptance) + + self.assertNotEqual(result.returncode, 0) + self.assertIn(expected, result.stderr.lower()) + + @staticmethod + def _mutate_weka_provenance( + acceptance: dict[str, object], repetition: int, field: str, value: object + ) -> None: + item = acceptance["weka_qualification"]["repetitions"][repetition] + path = Path(item["path"]) + provenance = json.loads(path.read_text(encoding="utf-8")) + provenance[field] = value + write_private(path, provenance) + mark_evidence_time(path) + item["sha256"] = sha256(path) + + def test_wrong_profile_and_image_are_rejected(self) -> None: + for field, value, expected in ( + (("selected_profile", "max_num_seqs"), 8, "selected profile"), + (("runtime", "image_id"), "sha256:" + "b" * 64, "image"), + ): + with self.subTest(field=field), tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + payload = json.loads(acceptance.read_text(encoding="utf-8")) + payload[field[0]][field[1]] = value + write_private(acceptance, payload) + result = verify(run_root, acceptance) + self.assertNotEqual(result.returncode, 0) + self.assertIn(expected, result.stderr) + + def test_wrong_context_capability_hash_and_rank_identity_are_rejected(self) -> None: + mutations = ( + ("context", lambda payload: payload["long_context_qualification"].update(accepted_context_totals=[320000])), + ("capability", lambda payload: payload["feature_qualification"].update(passed=9)), + ("hash", lambda payload: payload["weka_qualification"].update(sha256="0" * 64)), + ( + "rank identity", + lambda payload: payload["topology"].update(rank_0="tilikum"), + ), + ) + for expected, mutate in mutations: + with self.subTest(expected=expected), tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + payload = json.loads(acceptance.read_text(encoding="utf-8")) + mutate(payload) + write_private(acceptance, payload) + result = verify(run_root, acceptance) + self.assertNotEqual(result.returncode, 0) + self.assertIn(expected, result.stderr) + + def test_missing_expanded_capability_case_is_rejected(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + payload = json.loads(acceptance.read_text(encoding="utf-8")) + feature_path = Path(payload["feature_qualification"]["path"]) + feature = json.loads(feature_path.read_text(encoding="utf-8")) + feature["cases"] = [ + case + for case in feature["cases"] + if case["case"] != "tool-parallel" + ] + write_private(feature_path, feature) + mark_evidence_time(feature_path) + payload["feature_qualification"].update( + sha256=sha256(feature_path), + passed=len(feature["cases"]), + total=len(feature["cases"]), + ) + write_private(acceptance, payload) + result = verify(run_root, acceptance) + self.assertNotEqual(result.returncode, 0) + self.assertIn("missing required cases", result.stderr) + + def test_runtime_evidence_cannot_repin_known_immutable_artifacts(self) -> None: + mutations = ("image", "vLLM commit", "checkpoint manifest") + for mutation in mutations: + with self.subTest(mutation=mutation), tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + payload = json.loads(acceptance.read_text(encoding="utf-8")) + manifest_path = Path(payload["runtime"]["manifest_path"]) + manifest = manifest_path.read_text(encoding="utf-8") + if mutation == "image": + replacement = "sha256:" + "b" * 64 + payload["runtime"]["image_id"] = replacement + manifest = manifest.replace(IMAGE, replacement) + for key in ("rank_0_evidence", "rank_1_evidence"): + item = payload["live_state"][key] + path = Path(item["path"]) + path.write_text( + path.read_text(encoding="utf-8").replace(IMAGE, replacement), + encoding="utf-8", + ) + os.chmod(path, 0o600) + mark_evidence_time(path) + item["sha256"] = sha256(path) + elif mutation == "vLLM commit": + replacement = "f" * 40 + payload["runtime"]["vllm_commit"] = replacement + manifest = manifest.replace(VLLM_COMMIT, replacement) + else: + checkpoint = run_root / "baseline" / "shamu-checkpoint.manifest" + peer = run_root / "baseline" / "tilikum-checkpoint.manifest" + checkpoint.write_text("different checkpoint fixture\n", encoding="utf-8") + peer.write_bytes(checkpoint.read_bytes()) + os.chmod(checkpoint, 0o600) + os.chmod(peer, 0o600) + payload["runtime"]["checkpoint_manifest_sha256"] = sha256(checkpoint) + manifest_path.write_text(manifest, encoding="utf-8") + os.chmod(manifest_path, 0o600) + payload["runtime"]["manifest_sha256"] = sha256(manifest_path) + write_private(acceptance, payload) + result = verify(run_root, acceptance) + self.assertNotEqual(result.returncode, 0) + self.assertIn(mutation, result.stderr) + + def test_profile_selection_measurement_is_pinned(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + run_root, acceptance = build_fixture(Path(temporary)) + payload = json.loads(acceptance.read_text(encoding="utf-8")) + payload["selection"]["seqs_8_request_throughput_gain_percent"] = 42.0 + write_private(acceptance, payload) + result = verify(run_root, acceptance) + self.assertNotEqual(result.returncode, 0) + self.assertIn("selection", result.stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/webster/deepseek-v41/tests/test_publication_model_probes.py b/webster/deepseek-v41/tests/test_publication_model_probes.py new file mode 100755 index 0000000..bef3fd5 --- /dev/null +++ b/webster/deepseek-v41/tests/test_publication_model_probes.py @@ -0,0 +1,220 @@ +#!/usr/bin/env python3 +"""Behavior tests for the public publication model probes.""" + +from __future__ import annotations + +import base64 +import importlib.util +import json +import os +import threading +import unittest +import zlib +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path +from unittest.mock import patch + + +PACKAGE_ROOT = Path(__file__).resolve().parents[1] +PROBE_PATH = PACKAGE_ROOT / "scripts" / "publication-model-probes.py" +spec = importlib.util.spec_from_file_location("publication_model_probes", PROBE_PATH) +if spec is None or spec.loader is None: + raise RuntimeError("cannot load publication model probes") +module = importlib.util.module_from_spec(spec) +spec.loader.exec_module(module) + + +class VisionErrorHandler(BaseHTTPRequestHandler): + error_code = "unsupported_vision" + request_body: dict[str, object] | None = None + + def do_POST(self) -> None: # noqa: N802 - BaseHTTPRequestHandler API + length = int(self.headers["Content-Length"]) + type(self).request_body = json.loads(self.rfile.read(length)) + body = json.dumps( + {"error": {"code": type(self).error_code, "message": "fixture"}} + ).encode("utf-8") + self.send_response(400) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def log_message(self, _format: str, *_args: object) -> None: + return + + +class PublicationHandler(BaseHTTPRequestHandler): + registry = ( + "deepseek-v4.1-flash", + "glm-5.2", + "glm-5.3-flash", + "deepseek-v4-flash", + "aws/openai-gpt-5.6-sol", + ) + created_models: list[str] = [] + probed_models: list[str] = [] + deleted_keys: list[str] = [] + + def send_json(self, status: int, payload: dict[str, object]) -> None: + body = json.dumps(payload).encode("utf-8") + self.send_response(status) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def read_body(self) -> dict[str, object]: + length = int(self.headers["Content-Length"]) + return json.loads(self.rfile.read(length)) + + def do_GET(self) -> None: # noqa: N802 - BaseHTTPRequestHandler API + if self.path == "/model/info": + rows = [] + for model in self.registry: + model_info: dict[str, object] = {} + if model == "deepseek-v4.1-flash": + model_info = { + "max_input_tokens": 1048576, + "max_output_tokens": 1048576, + "supports_vision": True, + "supports_function_calling": True, + "supports_reasoning": True, + "supports_response_schema": True, + } + elif model == "glm-5.2": + model_info = { + "max_input_tokens": 320000, + "max_output_tokens": 320000, + "supports_vision": False, + } + rows.append({"model_name": model, "model_info": model_info}) + self.send_json(200, {"data": rows}) + return + if self.path == "/": + self.send_json(403, {"error": "denied"}) + return + if self.path == "/v1/models": + self.send_json(401, {"error": "unauthorized"}) + return + if self.path == "/key/info": + self.send_json(403, {"error": "denied"}) + return + self.send_json(404, {"error": "not found"}) + + def do_POST(self) -> None: # noqa: N802 - BaseHTTPRequestHandler API + body = self.read_body() + if self.path == "/key/generate": + type(self).created_models = list(body["models"]) + self.send_json(200, {"key": "sk-publication-fixture"}) + return + if self.path == "/key/delete": + type(self).deleted_keys = list(body["keys"]) + self.send_json(200, {"deleted_keys": body["keys"]}) + return + if self.path == "/v1/chat/completions": + model = str(body["model"]) + type(self).probed_models.append(model) + if model not in self.registry: + self.send_json(404, {"error": {"code": "model_not_found"}}) + return + content = body["messages"][0]["content"] + if model == "glm-5.2" and isinstance(content, list): + self.send_json( + 400, + {"error": {"code": "unsupported_vision", "message": "fixture"}}, + ) + return + self.send_json( + 200, + { + "model": model, + "choices": [{"message": {"role": "assistant", "content": "OK"}}], + }, + ) + return + self.send_json(404, {"error": "not found"}) + + def log_message(self, _format: str, *_args: object) -> None: + return + + +class PublicationModelProbeTests(unittest.TestCase): + def setUp(self) -> None: + VisionErrorHandler.error_code = "unsupported_vision" + VisionErrorHandler.request_body = None + self.server = ThreadingHTTPServer(("127.0.0.1", 0), VisionErrorHandler) + self.thread = threading.Thread(target=self.server.serve_forever, daemon=True) + self.thread.start() + self.base = f"http://127.0.0.1:{self.server.server_port}" + + def tearDown(self) -> None: + self.server.shutdown() + self.server.server_close() + self.thread.join(timeout=5) + + def test_vision_probe_requires_the_guard_specific_error(self) -> None: + VisionErrorHandler.error_code = "invalid_image" + + with self.assertRaisesRegex(RuntimeError, "unsupported_vision"): + module.probe_legacy_vision_rejection(self.base, "fixture-key") + + def test_vision_probe_sends_a_valid_png_and_accepts_only_the_guard_error(self) -> None: + status = module.probe_legacy_vision_rejection(self.base, "fixture-key") + + self.assertEqual(status, 400) + request = VisionErrorHandler.request_body + self.assertIsInstance(request, dict) + content = request["messages"][0]["content"] + encoded = content[1]["image_url"]["url"].split(",", 1)[1] + png = base64.b64decode(encoded, validate=True) + self.assertEqual(png[:8], b"\x89PNG\r\n\x1a\n") + self.assertEqual(png[12:16], b"IHDR") + idat_offset = png.index(b"IDAT") + idat_length = int.from_bytes(png[idat_offset - 4 : idat_offset], "big") + pixels = zlib.decompress(png[idat_offset + 4 : idat_offset + 4 + idat_length]) + self.assertEqual(len(pixels), 560 * (1 + 560 * 3)) + self.assertIn(b"IEND", png) + + def test_run_probes_only_current_migration_critical_models(self) -> None: + PublicationHandler.created_models = [] + PublicationHandler.probed_models = [] + PublicationHandler.deleted_keys = [] + server = ThreadingHTTPServer(("127.0.0.1", 0), PublicationHandler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + base = f"http://127.0.0.1:{server.server_port}" + critical = [ + "deepseek-v4.1-flash", + "glm-5.2", + "glm-5.3-flash", + "deepseek-v4-flash", + ] + environment = { + "WEBSTER_PUBLICATION_MASTER_KEY": "fixture-master-key", + "WEBSTER_PUBLICATION_PUBLIC_API": base, + "WEBSTER_PUBLICATION_ADMIN_API": base, + "WEBSTER_PUBLICATION_PRIVATE_API": base, + } + try: + with patch.dict(os.environ, environment, clear=False): + result = module.run("fixture-probe-id") + finally: + server.shutdown() + server.server_close() + thread.join(timeout=5) + + self.assertEqual(PublicationHandler.created_models, critical) + self.assertEqual(PublicationHandler.probed_models[:-1], critical) + self.assertEqual(PublicationHandler.probed_models[-1], "glm-5.2") + self.assertNotIn("inkling-small-nvfp4", result) + self.assertNotIn("aws/openai-gpt-5.6-sol", result) + self.assertEqual( + PublicationHandler.deleted_keys, + ["sk-publication-fixture"], + ) + self.assertEqual(result["revoked_key_status"], 401) + + +if __name__ == "__main__": + unittest.main() diff --git a/webster/deepseek-v41/tests/test_runtime_pin.py b/webster/deepseek-v41/tests/test_runtime_pin.py new file mode 100644 index 0000000..c4ffb1c --- /dev/null +++ b/webster/deepseek-v41/tests/test_runtime_pin.py @@ -0,0 +1,168 @@ +#!/usr/bin/env python3 +"""Behavior tests for the fail-closed vLLM runtime pin decision.""" + +from __future__ import annotations + +import importlib.util +import os +import tempfile +import unittest +from pathlib import Path + + +PACKAGE_ROOT = Path(__file__).resolve().parents[1] +MODULE_PATH = PACKAGE_ROOT / "scripts" / "pin-runtime.py" +HEAD_SHA = "7d81d62702b41885e2ff3ebc7ad9dfb638cc429c" +MERGE_SHA = "41fef9e0ed8b61342b232a3cf776a9cb695dadfe" +RELEASE_SHA = "a" * 40 + + +def load_module(): + spec = importlib.util.spec_from_file_location("pin_runtime", MODULE_PATH) + if spec is None or spec.loader is None: + raise RuntimeError(f"cannot import {MODULE_PATH}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def passing_evidence() -> tuple[dict, list[dict], dict, dict]: + pr = { + "state": "open", + "draft": False, + "merged": False, + "mergeable_state": "clean", + "head": {"sha": HEAD_SHA}, + "merge_commit_sha": MERGE_SHA, + } + reviews = [ + {"state": "APPROVED", "user": {"login": "reviewer"}, "commit_id": HEAD_SHA} + ] + status = { + "state": "success", + "total_count": 1, + "statuses": [{"context": "ci/test", "state": "success"}], + } + checks = { + "total_count": 1, + "check_runs": [ + {"name": "unit", "status": "completed", "conclusion": "success"} + ], + } + return pr, reviews, status, checks + + +class RuntimePinDecisionTests(unittest.TestCase): + def test_operational_script_is_directly_executable(self) -> None: + self.assertTrue(os.access(MODULE_PATH, os.X_OK)) + + def test_open_blocked_pr_is_rejected(self) -> None: + module = load_module() + pr, reviews, status, checks = passing_evidence() + pr["mergeable_state"] = "blocked" + + with self.assertRaisesRegex(module.PinRejected, "mergeable_state=blocked"): + module.select_runtime_pin(pr, reviews, status, checks, []) + + def test_pending_ci_is_rejected(self) -> None: + module = load_module() + pr, reviews, status, checks = passing_evidence() + status["state"] = "pending" + status["statuses"][0]["state"] = "pending" + + with self.assertRaisesRegex(module.PinRejected, "commit status is pending"): + module.select_runtime_pin(pr, reviews, status, checks, []) + + def test_only_latest_status_per_context_is_evaluated(self) -> None: + module = load_module() + history = [ + { + "context": "ci/test", + "state": "success", + "created_at": "2026-09-11T08:00:00Z", + }, + { + "context": "ci/test", + "state": "pending", + "created_at": "2026-09-11T07:00:00Z", + }, + { + "context": "ci/lint", + "state": "success", + "created_at": "2026-09-11T07:30:00Z", + }, + ] + + current = module.latest_statuses_by_context(history) + + self.assertEqual( + [(item["context"], item["state"]) for item in current], + [("ci/lint", "success"), ("ci/test", "success")], + ) + + def test_clean_reviewed_open_pr_selects_exact_head(self) -> None: + module = load_module() + pr, reviews, status, checks = passing_evidence() + + selected = module.select_runtime_pin(pr, reviews, status, checks, []) + + self.assertEqual(selected, {"commit": HEAD_SHA, "source": "pr-head"}) + + def test_release_containing_merged_change_is_preferred(self) -> None: + module = load_module() + pr, reviews, status, checks = passing_evidence() + pr.update({"state": "closed", "merged": True}) + releases = [ + { + "tag_name": "v0.99.0", + "target_commit": RELEASE_SHA, + "contains_merge_commit": True, + "draft": False, + "prerelease": False, + } + ] + + selected = module.select_runtime_pin(pr, reviews, status, checks, releases) + + self.assertEqual( + selected, + {"commit": RELEASE_SHA, "source": "release:v0.99.0"}, + ) + + def test_existing_different_manifest_pin_is_never_overwritten(self) -> None: + module = load_module() + with tempfile.TemporaryDirectory() as temporary: + manifest = Path(temporary) / "manifest.env" + manifest.write_text("VLLM_COMMIT=" + ("b" * 40) + "\nOTHER=value\n") + os.chmod(manifest, 0o600) + + with self.assertRaisesRegex(module.PinRejected, "already pins a different"): + module.update_manifest(manifest, HEAD_SHA) + + self.assertEqual( + manifest.read_text(), + "VLLM_COMMIT=" + ("b" * 40) + "\nOTHER=value\n", + ) + + def test_pr_change_during_evidence_collection_is_rejected(self) -> None: + module = load_module() + before = { + "state": "open", + "draft": False, + "merged": False, + "mergeable_state": "clean", + "updated_at": "2026-09-11T08:00:00Z", + "head": {"sha": HEAD_SHA}, + } + after = { + **before, + "updated_at": "2026-09-11T08:01:00Z", + "head": {"sha": "c" * 40}, + } + + with self.assertRaisesRegex(module.PinRejected, "changed during evidence collection"): + module.require_stable_pr(before, after) + + +if __name__ == "__main__": + unittest.main() diff --git a/webster/deepseek-v41/tests/test_serving_pair.py b/webster/deepseek-v41/tests/test_serving_pair.py new file mode 100644 index 0000000..70ced93 --- /dev/null +++ b/webster/deepseek-v41/tests/test_serving_pair.py @@ -0,0 +1,124 @@ +#!/usr/bin/env python3 +"""Behavior tests for matching two-rank serving generations.""" + +from __future__ import annotations + +import json +import subprocess +import sys +import unittest +from pathlib import Path + + +REPOSITORY_ROOT = Path(__file__).resolve().parents[3] +VERIFIER = ( + REPOSITORY_ROOT + / "webster" + / "deepseek-v41" + / "scripts" + / "verify-serving-pair.py" +) + + +def verify( + rank0: dict[str, object], + rank1: dict[str, object], + *, + require_matching_image: bool = False, + print_generation: bool = False, +) -> subprocess.CompletedProcess[str]: + command = [ + sys.executable, + str(VERIFIER), + "--rank0-state-json", + json.dumps(rank0, separators=(",", ":")), + "--rank1-state-json", + json.dumps(rank1, separators=(",", ":")), + "--max-start-skew-seconds", + "120", + ] + if require_matching_image: + command.append("--require-matching-image") + if print_generation: + command.append("--print-generation") + return subprocess.run( + command, + cwd=REPOSITORY_ROOT, + text=True, + capture_output=True, + check=False, + ) + + +class ServingPairTests(unittest.TestCase): + def test_legacy_pair_emits_a_stable_derived_generation(self) -> None: + rank0 = { + "container_id": "a" * 64, + "generation": None, + "started_at": "2026-09-12T16:41:07.263876192Z", + } + rank1 = { + "container_id": "b" * 64, + "generation": None, + "started_at": "2026-09-12T16:41:08.329565990Z", + } + first = verify(rank0, rank1, print_generation=True) + second = verify(rank0, rank1, print_generation=True) + self.assertEqual(first.returncode, 0, first.stderr) + self.assertEqual(second.returncode, 0, second.stderr) + self.assertRegex(first.stdout.strip(), r"^legacy-[0-9a-f]{64}$") + self.assertEqual(first.stdout, second.stdout) + + def test_legacy_pair_accepts_docker_nanosecond_timestamps(self) -> None: + result = verify( + { + "generation": None, + "started_at": "2026-09-12T16:41:07.263876192Z", + }, + { + "generation": None, + "started_at": "2026-09-12T16:41:08.329565990Z", + }, + ) + self.assertEqual(result.returncode, 0, result.stderr) + + def test_labeled_pair_requires_the_same_nonempty_generation(self) -> None: + result = verify( + { + "generation": "generation-a", + "started_at": "2026-09-12T16:41:07Z", + }, + { + "generation": "generation-b", + "started_at": "2026-09-12T16:41:08Z", + }, + ) + self.assertNotEqual(result.returncode, 0) + self.assertIn("different generations", result.stderr) + + def test_unlabeled_pair_rejects_excessive_start_skew(self) -> None: + result = verify( + {"generation": None, "started_at": "2026-09-12T16:40:00Z"}, + {"generation": None, "started_at": "2026-09-12T16:43:00Z"}, + ) + self.assertNotEqual(result.returncode, 0) + self.assertIn("not started as one generation", result.stderr) + + def test_glm_pair_requires_matching_nonempty_image_ids(self) -> None: + result = verify( + { + "image": "sha256:" + "a" * 64, + "started_at": "2026-09-12T16:41:07Z", + }, + { + "image": "sha256:" + "b" * 64, + "started_at": "2026-09-12T16:41:08Z", + }, + require_matching_image=True, + ) + self.assertNotEqual(result.returncode, 0) + self.assertIn("different image generations", result.stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/webster/deepseek-v41/tests/test_static.py b/webster/deepseek-v41/tests/test_static.py new file mode 100644 index 0000000..593abab --- /dev/null +++ b/webster/deepseek-v41/tests/test_static.py @@ -0,0 +1,361 @@ +#!/usr/bin/env python3 +"""Repository contract tests for the DeepSeek V4.1 cutover package.""" + +from __future__ import annotations + +import json +import os +import re +import subprocess +import tempfile +import unittest +from pathlib import Path + + +REPOSITORY_ROOT = Path(__file__).resolve().parents[3] +PACKAGE_ROOT = REPOSITORY_ROOT / "webster" / "deepseek-v41" +MIGRATION_SKILL_ROOT = REPOSITORY_ROOT / ".agent" / "skills" / "migrating-webster-models" + + +def read(relative_path: str) -> str: + return (PACKAGE_ROOT / relative_path).read_text(encoding="utf-8") + + +def shell_script_paths() -> list[Path]: + scripts = PACKAGE_ROOT / "scripts" + return sorted(scripts.glob("*.sh")) if scripts.exists() else [] + + +class FoundationContractTests(unittest.TestCase): + def test_foundation_files_exist(self) -> None: + for relative_path in ( + ".gitignore", + "manifest.env.example", + "README.md", + "scripts/common.sh", + "scripts/preflight.sh", + ): + with self.subTest(path=relative_path): + self.assertTrue((PACKAGE_ROOT / relative_path).is_file()) + + def test_station_topology_is_pinned(self) -> None: + common = read("scripts/common.sh") + for literal in ( + 'SHAMU_NETBIRD="100.73.140.127"', + 'SHAMU_RAIL="10.10.1.1"', + 'TILIKUM_RAIL="10.10.1.2"', + 'NCCL_IFACE="enP1p3s0f1np1"', + 'NCCL_HCA="mlx5_1"', + ): + self.assertIn(literal, common) + + def test_checkpoint_and_capacity_contract_is_pinned(self) -> None: + common = read("scripts/common.sh") + for literal in ( + 'MIN_FREE_BEFORE_STAGE_BYTES="805306368000"', + 'MIN_FREE_AFTER_STAGE_BYTES="214748364800"', + 'CHECKPOINT_REPO="deepseek-ai/DeepSeek-V4.1-Flash"', + 'CHECKPOINT_REVISION="dba1be0a40aa45a94ad051997016db3960a90277"', + 'CHECKPOINT_SHARDS="48"', + 'CHECKPOINT_TENSOR_BYTES="510286023000"', + 'CHECKPOINT_SHARD_FILE_BYTES="510296708312"', + ): + self.assertIn(literal, common) + + def test_manifest_requires_execution_time_pins(self) -> None: + manifest = read("manifest.env.example") + for field in ( + "VLLM_COMMIT", + "VLLM_BASE_IMAGE_DIGEST", + "VLLM_BUILD_BASE_IMAGE_DIGEST", + "VLLM_FINAL_BASE_IMAGE_DIGEST", + "VLLM_SOURCE_ARCHIVE_SHA256", + "VLLM_IMAGE_ID", + "VLLM_IMAGE_TAR_SHA256", + "AIPERF_COMMIT", + "WEKA_REPOSITORY", + "WEKA_REVISION", + ): + self.assertRegex(manifest, rf"(?m)^{field}=\s*$") + + def test_limited_downtime_sequence_is_documented(self) -> None: + documentation = read("README.md") + sequence = ( + "freeze rollback -> stage additively -> route alias -> restart once -> hot soak\n" + "-> stop both GLM ranks -> private canary/tuning -> register new name -> restart once" + ) + self.assertIn(sequence, documentation) + self.assertIn("60-second", documentation) + self.assertIn("5–15 minute", documentation) + + def test_baker_route_is_never_a_mutation_target(self) -> None: + for path in shell_script_paths(): + script = path.read_text(encoding="utf-8") + self.assertNotIn("100.73.127.129:8888", script, path) + self.assertNotRegex(script, r"ssh\s+baker-spark-[12]", path) + + def test_no_secret_on_command_line(self) -> None: + scripts = "\n".join( + path.read_text(encoding="utf-8") for path in shell_script_paths() + ) + self.assertNotRegex(scripts, r"--api-key(?:=|\s+)[^\"'$]") + self.assertNotRegex(scripts, r"Authorization:\s*Bearer\s+\$\{") + self.assertNotRegex(scripts, r"Authorization:\s*Bearer\s+[A-Za-z0-9]") + + def test_safety_critical_embedded_python_does_not_depend_on_assert(self) -> None: + for relative_path in ( + "scripts/acceptance.sh", + "scripts/install-glm52-guard.sh", + ): + with self.subTest(path=relative_path): + self.assertNotRegex(read(relative_path), r"\bassert\s") + + def test_cluster_command_wrapper_scrubs_ssh_agent_and_disables_forwarding(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fake = root / "ssh" + fake.write_text( + "#!/usr/bin/env bash\n" + "set -euo pipefail\n" + "test -z \"${SSH_AUTH_SOCK:-}\"\n" + "test \"$1\" = -o\n" + "test \"$2\" = ForwardAgent=no\n" + "printf '%s\\n' \"$@\"\n", + encoding="utf-8", + ) + os.chmod(fake, 0o700) + result = subprocess.run( + [ + "bash", + "-c", + 'source "$1"; SSH_AUTH_SOCK=/tmp/inherited-agent; export SSH_AUTH_SOCK; ssh shamu true', + "bash", + str(PACKAGE_ROOT / "scripts" / "common.sh"), + ], + env={**os.environ, "PATH": f"{root}:{os.environ['PATH']}"}, + text=True, + capture_output=True, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual( + result.stdout.splitlines(), + ["-o", "ForwardAgent=no", "shamu", "true"], + ) + start = read("scripts/start-deepseek-v41-tp2.sh") + self.assertIn( + "env -u SSH_AUTH_SOCK ssh -o ForwardAgent=no", + start, + "the Shamu-to-Tilikum composed path must scrub the remote shell too", + ) + + def test_aiperf_fixtures_are_parseable(self) -> None: + profile = json.loads(read("tests/fixtures/aiperf-profile.json")) + self.assertIn("metrics", profile) + rows = [ + json.loads(line) + for line in read("tests/fixtures/aiperf-errors.jsonl").splitlines() + if line.strip() + ] + self.assertTrue(any(row.get("error") for row in rows)) + + def test_root_runner_invokes_cutover_suites(self) -> None: + runner = (REPOSITORY_ROOT / "test.sh").read_text(encoding="utf-8") + self.assertIn( + "python3 -m unittest discover -s webster/deepseek-v41/tests -p 'test_*.py'", + runner, + ) + self.assertIn( + "bash webster/deepseek-v41/tests/test_failure_paths.sh", runner + ) + self.assertTrue( + re.search( + r"if \[ -f webster/deepseek-v41/litellm/" + r"test_glm52_contract_guard\.py \]", + runner, + ) + ) + + def test_staging_files_exist(self) -> None: + for relative_path in ( + "scripts/stage-artifacts.sh", + "scripts/verify-checkpoint.py", + ): + with self.subTest(path=relative_path): + self.assertTrue((PACKAGE_ROOT / relative_path).is_file()) + + def test_checkpoint_downloader_keeps_xet_enabled(self) -> None: + staging = read("scripts/stage-artifacts.sh") + self.assertNotIn("HF_HUB_DISABLE_XET=1", staging) + + def test_litellm_cutover_files_exist(self) -> None: + for relative_path in ( + "scripts/acceptance.sh", + "scripts/render-litellm-cutover.py", + "scripts/verify-litellm-config.py", + "scripts/verify-litellm-container.py", + "scripts/install-glm52-guard.sh", + "scripts/restore-litellm-config.sh", + "scripts/sync-runbook.sh", + "scripts/contract-probe.py", + ): + with self.subTest(path=relative_path): + self.assertTrue((PACKAGE_ROOT / relative_path).is_file()) + + def test_closeout_scripts_encode_publication_and_skill_sync_contracts(self) -> None: + acceptance = read("scripts/acceptance.sh") + publication_probes = read("scripts/publication-model-probes.py") + publication_entrypoint = acceptance + "\n" + publication_probes + for literal in ( + "deepseek-v4.1-flash", + "glm-5.2", + "glm-5.3-flash", + "deepseek-v4-flash", + "deepseek-v41-watchdog.timer", + "PUBLISH_CONFIG_BACKUP", + ): + self.assertIn(literal, publication_entrypoint) + sync = read("scripts/sync-runbook.sh") + for literal in ( + "migrating-webster-models", + "WEBSTER_CLUSTER_SOURCE", + "WEBSTER_MIGRATION_SOURCE", + ".claude/skills", + ".codex/skills", + ".hermes/skills/devops", + "check-skill-drift.sh", + "COLUMNS=240", + "skills list --source local", + ): + self.assertIn(literal, sync) + + def test_migration_skill_retains_observed_closeout_lessons(self) -> None: + playbook = (MIGRATION_SKILL_ROOT / "references" / "playbook.md").read_text( + encoding="utf-8" + ) + for lesson in ( + "phase-specific owner", + "hostname resolution", + "FlashInfer autotune", + "cold-prefill", + "recent-log matcher", + "acceptance entry point", + "legacy pair identity", + "runtime patch", + "executable bits", + "dependency closure", + "AIPERF_DATASET_CONFIGURATION_TIMEOUT", + "Never rewrite a failed `EXIT`", + "freshness budget", + "dynamic pricing", + "`brev refresh`", + "grouped request counts", + "stale recovery controller", + "migration-critical aliases", + "client-visible top-level error", + "retired model", + ): + self.assertIn(lesson, playbook) + + def test_readme_records_final_reconciliation(self) -> None: + documentation = read("README.md") + for fact in ( + "9.122 seconds", + "inkling-watchdog.timer", + "10/10 Prometheus targets", + "openwebui-shamu-5", + ): + self.assertIn(fact, documentation) + + def test_migration_skill_requires_fresh_cryptographic_publication_evidence(self) -> None: + skill = (MIGRATION_SKILL_ROOT / "SKILL.md").read_text(encoding="utf-8") + playbook = (MIGRATION_SKILL_ROOT / "references" / "playbook.md").read_text( + encoding="utf-8" + ) + combined = skill + "\n" + playbook + for requirement in ( + "17-case", + "failure-log-output", + "generate-private-acceptance.py", + "verify-private-acceptance.py", + "accepted_at", + "exact rendered Hermes", + "--force", + ): + with self.subTest(requirement=requirement): + self.assertIn(requirement, combined) + + def test_runtime_package_files_exist(self) -> None: + for relative_path in ( + "runtime/Dockerfile", + "runtime/constraints.txt", + "runtime/README.md", + "scripts/build-runtime.sh", + ): + with self.subTest(path=relative_path): + self.assertTrue((PACKAGE_ROOT / relative_path).is_file()) + + def test_runtime_wrapper_cannot_mutate_upstream_image(self) -> None: + dockerfile = read("runtime/Dockerfile") + instructions = [ + line.split(maxsplit=1)[0].upper() + for line in dockerfile.splitlines() + if line and not line.startswith("#") + ] + self.assertEqual(instructions.count("FROM"), 1) + self.assertIn("ARG", instructions) + self.assertIn("LABEL", instructions) + for forbidden in ("ADD", "COPY", "RUN"): + self.assertNotIn(forbidden, instructions) + + def test_preflight_captures_the_phase_owner_without_requiring_stopped_glm(self) -> None: + preflight = read("scripts/preflight.sh") + self.assertIn('station_inspect_shamu="docker inspect \'$GLM_CONTAINER\'"', preflight) + self.assertIn('station_inspect_tilikum="sudo -n docker inspect \'$GLM_CONTAINER\'"', preflight) + self.assertIn('if [[ "$phase" == "publish" ]]; then', preflight) + self.assertIn('station_inspect_shamu="docker inspect \'$DEEPSEEK_CONTAINER\'"', preflight) + self.assertIn('station_inspect_tilikum="sudo -n docker inspect \'$DEEPSEEK_CONTAINER\'"', preflight) + capture_block = preflight.split("station_inspect_format=", 1)[1].split( + 'shamu_free="', 1 + )[0] + self.assertNotIn( + "docker inspect glm52-full-mtp --format", + capture_block, + "publication host capture must not unconditionally inspect stopped GLM", + ) + + def test_glm_fast_path_validates_the_complete_rollback_runtime_shape(self) -> None: + start = read("scripts/start-glm52-tp2.sh") + for requirement in ( + 'config.get("WorkingDir") == "/vllm-workspace"', + 'host.get("ShmSize") == 34359738368', + '"label=disable" in (host.get("SecurityOpt") or [])', + '(ulimits.get("memlock") or {}).get("Soft") == -1', + '(ulimits.get("stack") or {}).get("Soft") == 67108864', + 'item.get("PathOnHost") == "/dev/infiniband"', + 'bool(host.get("DeviceRequests"))', + 'mounts.get("/model") or {}', + 'mounts.get("/root/.cache") or {}', + ): + with self.subTest(requirement=requirement): + self.assertIn(requirement, start) + + def test_deepseek_live_evidence_supports_the_exact_unlabeled_legacy_pair(self) -> None: + start = read("scripts/start-deepseek-v41-tp2.sh") + for requirement in ( + 'require(all(labels_present), "has partial serving labels")', + '"container_id": data.get("Id")', + ): + with self.subTest(requirement=requirement): + self.assertIn(requirement, start) + self.assertNotIn( + 'require(isinstance(generation, str) and generation, "has no serving generation")', + start, + ) + runner = read("scripts/run-weka.sh") + self.assertIn('"--print-generation"', runner) + self.assertIn('"container_id": value.get("Id")', runner) + + +if __name__ == "__main__": + unittest.main()