diff --git a/docs/RLF_CODEX_RUN.md b/docs/RLF_CODEX_RUN.md deleted file mode 100644 index 1c82a2d50..000000000 --- a/docs/RLF_CODEX_RUN.md +++ /dev/null @@ -1,177 +0,0 @@ -# RLF → Yeto Codex RL runbook - -The online RL harness is Codex only. rlenvfactory/AgentFlow may produce or review offline task packages, but its node output is never used as task identity, grader evidence, reward, or rollout input. SWE and the legacy `rl-agentic-reward-env` change remain deferred. - -## Apply order - -1. Generate or install an rlenvfactory CTF package and retain `public/`, `task.yaml`, `MANIFEST.json`, source commit, package/image digests, and runtime quality-gate evidence. -2. Admit the package with `yeto.rl.rlf.admit_task_package`; reject packages without a trusted runtime gate. -3. Acquire one Modal VM lease per `(task_id, episode_id, trajectory_id)`. Keep secrets on the trusted side and tear down the lease after grading. -4. Run the existing Codex subprocess harness. Copy RLF identities and policy identity into every chain/compaction segment; preserve tokens, logprobs, loss masks, and segment metadata. -5. Finish and grade on the trusted side, then pass the resulting metadata through the Miles or generic verl preflight. Bare float rewards are rejected. -6. Run the CPU gate before any paid GPU run. A healthy RL gate must show nonzero reward variance, success count, absolute advantage sum, and policy update. - -The baseline profile is Qwen3-30B-A3B thinking mode with a 32K window, 1024 generated tokens per turn, at most 48 turns, and 8192-byte tool output. - -## Qwen3 cyber baseline profile (5.6) - -The registered backend profile is `qwen3_30b_a3b` (`yeto.rl.codex_backend`, -allowlisted in `yeto/rl/harness/codex/pins.py`). Pass it per launch with -`--codex-backend-profile qwen3_30b_a3b`. Its budgets are declared on the -profile; three of them are launch-time knobs that must be set explicitly -because their global defaults deliberately stay unchanged: - -| Budget | Profile value | How it takes effect | -| --- | --- | --- | -| context window | 32768 tokens | `--agent-max-seq-len 32768` (run config `agent_max_seq_len`) | -| generated tokens per turn | 1024 | `YETO_CODEX_BACKEND_MAX_TOKENS=1024` | -| turns | 48 | `SECRLENV_MAX_TURNS=48` (worker-passthrough env; global default stays 40) | -| tool output | 8192 bytes | read from the profile by `codex_harness_agent` (`MAX_TOOL_OUTPUT_BYTES`); profiles without the field keep 32768 | - -Keep the 20-30 minute episode deadline (`OPENENV_MAX_ROLLOUT_TIME_SECONDS`, -default 1800s) and record `max_seq_len_hit`, `end_reason`, turns, model -tokens, tool calls, and tool-output bytes per episode for the baseline -analysis. If a flag is reached only after truncation, enable the registered compaction plan: 24K active window, 4K–8K reserve, and at most 2–3 compactions while preserving segment identity. - -## Baseline runner (5.6): RLF 形态的 Qwen3-30B-A3B - -The runner is `yeto.rl.eval.rlf_baseline` (RLF counterpart of -`yeto/rl/eval/codex_infer.py`, which covers the TB2 attempt shape). Per run: -admit the task packages into the RLF catalog, render prompt rows, bring up -serving, then per row acquire one RLF CTF lease (W4 `RlfModalProvider`), -drive the training Codex subprocess worker, finish on the trusted side, and -write the trace. Codex-only; AgentFlow artifacts are refused at admission. - -### Serving shape (inside the GPU container) - -- Modal `A100-80GB` x1, image = the pinned Miles-ports image - (`MILES_NEXT_IMAGE`), model read from the `yeto-fn-models` Volume: - `/mnt/yeto-models/hf/Qwen3-30B-A3B/ad44e777` (bf16). -- `python -m sglang.launch_server --model-path /mnt/yeto-models/hf/Qwen3-30B-A3B/ad44e777 \ - --trust-remote-code --context-length 32768 --mem-fraction-static 0.9` - plus the Miles session server (`session_server_config`, LoRA-free). The - profile's parsers/chat template come from Miles' `resolve_serving("qwen3")`. -- Serving that is not healthy within 15 min aborts the whole run and the - summary is written with `aborted.classification = "infrastructure"` - (never scored). HARD per-task deadline: 30 min - (`--episode-seconds`, default 1800; a tighter lease deadline wins). - -### Run - -Driver side (needs a local modal client): - - export YETO_CODEX_BUNDLE_DIR=/path/to/codex-bundle # scripts/fetch_codex_bundle.py - export YETO_RLF_CTF_TASKS_DIR=/path/to/tasks # admitted packages (task_id subdirs) - export YETO_RLF_BASELINE_ENVS_JSON=/path/to/envs.json # see below - modal run yeto.cloud.modal_rlf_baseline --config-json run.json - -`envs.json` holds the container env, same inputs as the eval island smoke: -the Codex contract env (`yeto.launcher.codex_container_env( -codex_bundle_contract(...))`), `TBENCH_REWARD_HMAC_KEY`, the sandbox-only -`MODAL_TOKEN_ID`/`MODAL_TOKEN_SECRET` (the RLF provider opens one VM per -episode from inside the container), and `YETO_RLF_CTF_FLAGS_JSON` -(`{"task_id": "flag{...}"}` — trusted; lives only in the Modal Secret, -never in the image, the traces, or the logs). `run.json`: - - {"run_name": "qwen3-30b-baseline-01", "repeat": 1, - "evidence": {"runtime_quality_gate": {"healthy": true}}, - "baseline": {"episode_seconds": 1800}} - -Evidence lands in the `yeto-rlf-baseline-runs` Volume under the run name. -In-container / CPU-smoke entry without Modal: - - python -m yeto.rl.eval.rlf_baseline --tasks-dir /root/rlf-tasks \ - --flags-json @flags.json --evidence '{"runtime_quality_gate": {"healthy": true}}' \ - --out /tmp/rlf-baseline-out [--repeat N] [--task ID] [--serve external --session-url URL] - -### Evidence checklist (per run) - -- `traces.jsonl`: one record per episode — RLF identity quadruple, - `policy_id` (`Qwen/Qwen3-30B-A3B@ad44e777...`), the task prompt, per-turn - completion/context tokens and tool-output bytes, per-tool events - (scrubbed command, bounded scrubbed output preview, `flag_seen` marker), - submission markers (`after_tool_calls`, `has_flag`, `flag_correct`; the - submitted flag value is never recorded), `submit_calls`, - `max_model_total_tokens`, `max_seq_len_hit`, compaction count, - `end_reason`/`end_kind`, `grader_evidence`, reward/success and the - `classification` (`success` / `failed` / `infrastructure` / `rejected`). -- `summary.json`: success rate over evaluated episodes (infrastructure and - rejected episodes are excluded, never scored 0), reward distribution, - end-reason counts, `max_seq_len_hit` rate, total compactions, - turns/tool-calls/token means and maxima, per-task breakdown. - -Stop conditions: serving start timeout (infrastructure abort), the 30-min -per-task deadline (`timeout` episodes keep reward 0 via the signed outcome -envelope and are never graded), or every prompt row processed. A -`max_seq_len_hit` rate above zero is the evidence for enabling the -registered 24K/4K compaction plan (5.7). - -Known unverified (first GPU run must confirm): Miles `resolve_serving` -coverage of the `qwen3` tito family, LoRA-free `SessionServerConfig` -(`lora_rank=None`), and A100-80GB scheduling latency on Modal. - - -## Real-GPU RL gate (5.8): Miles GRPO on `qwen35_08b` - -The gate proves the RLF -> Miles GRPO -> GAE -> policy-update chain on real -GPU with the small `qwen35_08b` profile (0.8B, the S17-verified harness -stack; LoRA, 1xGPU). The 30B capability surface is covered separately by the -5.6 baseline; 30B RL training is a follow-up item (design decision -2026-10-11). Tasks: `fixture-ctf-ezshell` (solvable in ~2 turns, provides -success) and `fixture-ctf-ret2win` (provides reward variance). Codex-only; -AgentFlow is offline-only and refused at admission. - -### Runner and commands - -The gate runner is `yeto.rl.eval.rlf_rl_gate`. Island side, in order: - - # 1. admit the packages, write Miles --prompt-data (flags never logged) - python -m yeto.rl.eval.rlf_rl_gate prompt-data \ - --tasks-dir "$YETO_RLF_CTF_TASKS_DIR" --flags-json /trusted/flags.json \ - --out /run/rlf_gate_prompt.jsonl --repeat 2 - - # 2. launch the Miles GRPO island (qwen35_08b, LoRA, 1xGPU): - # --data /run/rlf_gate_prompt.jsonl --codex-backend-profile qwen35_08b - # --reward-function rlf_success (-> --custom-rm-path yeto.rl.adapters.miles.rewards.rlf_success) - # --advantage-estimator grpo (fixed by island_entry) - # env: YETO_HARNESS_ENVIRONMENT_PROVIDER=yeto.cloud.modal_reward_env:rlf_modal_provider - # YETO_RLF_CTF_TASKS_DIR / YETO_RLF_CTF_FLAGS_JSON (Modal Secret, never in logs) - - # 3. after training, collect real artifacts, judge, write evidence - python -m yeto.rl.eval.rlf_rl_gate gate \ - --completed-groups /run/completed_groups.pt \ - --adapter-before /run/adapter_init --adapter-after /run/adapter_final \ - --step 2 --flags-json /trusted/flags.json --out /run/rlf_gate_evidence.json - -### Hard criteria (`yeto.rl.rlf.contracts.rl_gate`) - -Non-empty reward, `reward_std > 0`, `success_count > 0`, -`advantage_abs_sum > 0`, `policy_update == true`. All-zero or constant -reward MUST fail. The verdict is announced only for evidence collected -from the real island artifacts (`collector="miles_island"`); CPU/injected -collections exercise the orchestration but can never announce a pass. - -### Evidence chain (`yeto.rlf.rl_gate.evidence/v1`) - -- reward/success: the island completed-groups checkpoint - (`--completed-groups-path`) — the samples the learner actually trained - on, each with the `rlf_success` reward and `metadata.success` (W4 grade); -- advantage: recomputed per GRPO group with `contracts.group_advantages` - (the designated computation of record; never hand-filled); -- policy update: `contracts.policy_update_evidence` over a per-tensor - summary of the initial vs final LoRA adapter (代拍板: the Miles fork - exports no per-step snapshot hook; this proves parameters changed without - a fork patch, at the cost of not attributing the change to a specific - optimizer step, which the gate does not require); -- the JSON carries reward detail, success, `advantage_abs_sum`, and the - before/after policy digests; trusted flags are scrubbed - (`rlf.modal.scrub`) and only their sha256 appears in `config_digest`. - -### Stop conditions - -Stop and treat as infrastructure (never scored): serving not healthy within -15 min, or per-episode HARD deadline (30 min) exceeded repeatedly. Stop -with gate failure (the chain is broken, investigate before re-running): -`rl_gate` raising on any criterion, a sample missing `metadata.success`, -or unchanged adapter digests after training. A passing run archives the -evidence JSON next to the island artifacts. diff --git a/scripts/fetch_codex_bundle.py b/scripts/fetch_codex_bundle.py deleted file mode 100644 index c063a8aac..000000000 --- a/scripts/fetch_codex_bundle.py +++ /dev/null @@ -1,184 +0,0 @@ -#!/usr/bin/env python3 -"""Fetch and attest the pinned stock Codex Linux artifact (design R-SCOPE). - -Reproducible, pin-driven: every expected digest comes from ``yeto.rl`` -(``CODEX_NPM_PACKAGE``, ``CODEX_NPM_TARBALL_SHA256``, ``CODEX_LINUX_BINARY_SHA256``, -``CODEX_LINUX_BINARY_SIZE_BYTES``, ``CODEX_PACKAGE_MANIFEST_SHA256``, -``CODEX_APP_SERVER_SCHEMA_SHA256``). Steps: - -1. ``npm pack `` (or ``--tarball`` to reuse a download) and - check the tarball sha256. -2. Extract ``package/vendor//bin/codex`` and - ``package/vendor//codex-package.json``; check size + sha256. -3. Run ``codex --version`` and ``codex app-server generate-json-schema - --experimental`` (the same command ``yeto.rl.adapters.miles.island_entry._preflight_codex_harness`` - runs in the container) and write the v2 schema into the bundle. Codex - serialises the schema with unstable key order, so the file is written in - canonical form (``sort_keys=True, indent=2``) and compared against the pin - by sha256; a mismatch is reported (exit 3) unless ``--allow-schema-drift``. - -Output layout (``--out``, default ``~/work/codex-bundle``), the run asset dir -the launcher mounts at ``/opt/yeto/codex`` (``YETO_CODEX_BUNDLE_DIR``): - - /npm/.tgz - /codex/codex-x86_64-unknown-linux-musl - /codex/codex-package.json - /codex/codex_app_server_protocol.v2.schemas.json - /codex/BUNDLE-SHA256.json (what was verified, with the pins) -""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import os -import shutil -import subprocess -import sys -import tarfile -import tempfile -from pathlib import Path - -REPO_ROOT = Path(__file__).resolve().parents[1] -sys.path.insert(0, str(REPO_ROOT)) - -from yeto.rl import ( # noqa: E402 - CODEX_APP_SERVER_SCHEMA_SHA256, - CODEX_CLI_VERSION, - CODEX_LINUX_BINARY_SHA256, - CODEX_LINUX_BINARY_SIZE_BYTES, - CODEX_LINUX_TARGET, - CODEX_NPM_PACKAGE, - CODEX_NPM_TARBALL_SHA256, - CODEX_PACKAGE_MANIFEST_SHA256, -) - -BINARY_NAME = f"codex-{CODEX_LINUX_TARGET}" -MANIFEST_NAME = "codex-package.json" -SCHEMA_NAME = "codex_app_server_protocol.v2.schemas.json" -RECORD_NAME = "BUNDLE-SHA256.json" - - -def sha256_of(path: Path) -> str: - digest = hashlib.sha256() - with path.open("rb") as handle: - for chunk in iter(lambda: handle.read(1 << 20), b""): - digest.update(chunk) - return digest.hexdigest() - - -def npm_pack(package: str, into: Path, npm: str) -> Path: - into.mkdir(parents=True, exist_ok=True) - out = subprocess.run( - [npm, "pack", package, "--pack-destination", str(into)], - check=True, - capture_output=True, - text=True, - ).stdout.strip().splitlines() - name = out[-1].strip() if out else "" - tarball = into / name - if not name.endswith(".tgz") or not tarball.is_file(): - raise SystemExit(f"npm pack did not report a tarball (stdout: {out!r})") - return tarball - - -def extract_member(tarball: Path, member: str, destination: Path) -> None: - with tarfile.open(tarball, "r:gz") as archive: - try: - info = archive.getmember(member) - except KeyError as exc: - raise SystemExit(f"{member} missing from {tarball}") from exc - if not info.isfile(): - raise SystemExit(f"{member} is not a regular file in {tarball}") - source = archive.extractfile(info) - assert source is not None - destination.parent.mkdir(parents=True, exist_ok=True) - with destination.open("wb") as handle: - shutil.copyfileobj(source, handle) - - -def generate_schema(binary: Path) -> dict: - with tempfile.TemporaryDirectory(prefix="yeto-codex-schema-") as temporary: - out = Path(temporary) / "schema" - subprocess.run( - [str(binary), "app-server", "generate-json-schema", "--experimental", "--out", str(out)], - check=True, - capture_output=True, - text=True, - timeout=120, - env={**os.environ, "HOME": temporary}, - ) - with (out / SCHEMA_NAME).open(encoding="utf-8") as handle: - return json.load(handle) - - -def main(argv: list[str] | None = None) -> int: - parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - parser.add_argument("--out", default=os.path.expanduser("~/work/codex-bundle")) - parser.add_argument("--tarball", default=None, help="reuse an already downloaded npm tarball") - parser.add_argument("--npm", default=shutil.which("npm") or "npm") - parser.add_argument("--allow-schema-drift", action="store_true", - help="write the bundle even if the canonical schema sha256 differs from the pin") - args = parser.parse_args(argv) - - out = Path(args.out).expanduser().resolve() - codex_dir = out / "codex" - tarball = Path(args.tarball).expanduser().resolve() if args.tarball else npm_pack(CODEX_NPM_PACKAGE, out / "npm", args.npm) - tarball_sha = sha256_of(tarball) - if tarball_sha != CODEX_NPM_TARBALL_SHA256: - raise SystemExit(f"tarball sha256 {tarball_sha} != pin {CODEX_NPM_TARBALL_SHA256} ({tarball})") - - binary = codex_dir / BINARY_NAME - manifest = codex_dir / MANIFEST_NAME - extract_member(tarball, f"package/vendor/{CODEX_LINUX_TARGET}/bin/codex", binary) - extract_member(tarball, f"package/vendor/{CODEX_LINUX_TARGET}/{MANIFEST_NAME}", manifest) - binary.chmod(0o755) - size = binary.stat().st_size - if size != CODEX_LINUX_BINARY_SIZE_BYTES: - raise SystemExit(f"binary size {size} != pin {CODEX_LINUX_BINARY_SIZE_BYTES}") - binary_sha = sha256_of(binary) - if binary_sha != CODEX_LINUX_BINARY_SHA256: - raise SystemExit(f"binary sha256 {binary_sha} != pin {CODEX_LINUX_BINARY_SHA256}") - manifest_sha = sha256_of(manifest) - if manifest_sha != CODEX_PACKAGE_MANIFEST_SHA256: - raise SystemExit(f"manifest sha256 {manifest_sha} != pin {CODEX_PACKAGE_MANIFEST_SHA256}") - - version = subprocess.run([str(binary), "--version"], check=True, capture_output=True, text=True, timeout=30).stdout.strip() - if version != CODEX_CLI_VERSION: - raise SystemExit(f"codex --version {version!r} != pin {CODEX_CLI_VERSION!r}") - - schema_path = codex_dir / SCHEMA_NAME - schema_path.write_text(json.dumps(generate_schema(binary), sort_keys=True, indent=2) + "\n", encoding="utf-8") - schema_sha = sha256_of(schema_path) - schema_ok = schema_sha == CODEX_APP_SERVER_SCHEMA_SHA256 - - record = { - "npm_package": CODEX_NPM_PACKAGE, - "tarball": str(tarball), - "tarball_sha256": tarball_sha, - "binary": str(binary), - "binary_sha256": binary_sha, - "binary_size_bytes": size, - "cli_version": version, - "package_manifest_sha256": manifest_sha, - "app_server_schema_sha256": schema_sha, - "app_server_schema_sha256_pin": CODEX_APP_SERVER_SCHEMA_SHA256, - "app_server_schema_matches_pin": schema_ok, - } - (codex_dir / RECORD_NAME).write_text(json.dumps(record, indent=2, sort_keys=True) + "\n", encoding="utf-8") - print(json.dumps(record, indent=2, sort_keys=True)) - if not schema_ok: - print( - f"WARNING: canonical app-server schema sha256 {schema_sha} != pin " - f"{CODEX_APP_SERVER_SCHEMA_SHA256}; the container preflight " - "(yeto.rl.adapters.miles.island_entry._preflight_codex_harness) will reject this bundle until the pin is re-derived", - file=sys.stderr, - ) - if not args.allow_schema_drift: - return 3 - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/tests/decoupling_golden.py b/tests/decoupling_golden.py index 83563992c..021b34d9c 100644 --- a/tests/decoupling_golden.py +++ b/tests/decoupling_golden.py @@ -234,47 +234,6 @@ def _fn_cli(_tmp: Path) -> tuple[str, ...]: return tuple(fta._argv("2x8")) -def _codex_cli(_tmp: Path) -> tuple[str, ...]: - from yeto.rl import CODEX_OPENENV_AGENT - from yeto.rl.codex_backend import QWEN35_08B_MODEL, QWEN35_08B_REVISION - - return BASE + ( - "--model", QWEN35_08B_MODEL, "--model-revision", QWEN35_08B_REVISION, - "--lora-targets", "attention", - "--custom-generate-function-path", "miles.rollout.generate_hub.agentic_tool_call.generate", - "--custom-agent-function-path", CODEX_OPENENV_AGENT, - "--codex-reasoning-effort", "xhigh", "--codex-backend-profile", "qwen35_08b", - "--use-session-server", "--tito-model", "qwen35", - "--tito-allowed-append-roles", "tool", "user", "--agent-max-seq-len", "128", - ) - - -def _codex_bundle(mp, tmp: Path) -> None: - """The fake Codex bundle of ``test_rl_launcher_codex_bundle.bundle`` (contract - attestation stubbed to the pinned values; no 310 MB binary, no network).""" - import test_rl_launcher_codex_bundle as cb - import yeto.rl.ssh_harness as sh - from yeto import launcher as L - - root = tmp / "codex-bundle" / "codex" - root.mkdir(parents=True) - for name in ("codex-x86_64-unknown-linux-musl", "codex-package.json", - "codex_app_server_protocol.v2.schemas.json"): - (root / name).write_bytes(b"x") - mp.setattr(sh, "_codex_harness_contract", lambda _ns, args: cb._synthetic_contract(args)) - mp.setenv(L.CODEX_BUNDLE_DIR_ENV, str(root)) - mp.setenv(L.HARNESS_ENVIRONMENT_PROVIDER_ENV, cb.PROVIDER) - mp.delenv(L.CODEX_COMPACTION_ENV, raising=False) - mp.delenv(L.HARNESS_PREFLIGHT_ENV, raising=False) - # The reasoning / tool-call parser names are resolved by Miles - # (miles.utils.chat_template_utils), which this host must not import: the two - # values are recorded as explicit placeholders, everything else is yeto's own. - from yeto.rl.adapters.miles import config as mc - - mp.setattr(mc, "_resolve_tito_parsers", lambda model: ( - "", "")) - - CONFIGS: tuple[Config, ...] = ( Config("grpo_default", "GRPO 默认（ports，固定分区，strict-avg）", lambda t: BASE), Config("grpo_tis", "GRPO + TIS 失配修正（tis_clip 2.0）", @@ -285,8 +244,6 @@ def _codex_bundle(mp, tmp: Path) -> None: lambda t: BASE + ("--rl-algorithm-spec", str(REPO / DRGRPO_EVIDENCE))), Config("seq_adv_maxrl", "seq_adv：MaxRL 优势变换（含奖励流水线插件）", lambda t: BASE + ("--rl-algorithm-spec", _spec_file(t, "maxrl", _maxrl_spec))), - Config("codex_harness", "codex harness（ports，Qwen3.5-0.8B，假 bundle 合同）", _codex_cli, - patch=_codex_bundle), Config("elastic", "岛内弹性模式（--rl-elastic，单配置资源表）", lambda t: BASE + _elastic(t)), Config("fn_2x8", "Flash-Next 2×8 正式训练形状（fntrain.sh print 2x8）", _fn_cli, fn=True), diff --git a/tests/fixtures/trajectory_evidence/generate_signed_era.py b/tests/fixtures/trajectory_evidence/generate_signed_era.py index 0868e8c29..9a3d20278 100644 --- a/tests/fixtures/trajectory_evidence/generate_signed_era.py +++ b/tests/fixtures/trajectory_evidence/generate_signed_era.py @@ -22,7 +22,7 @@ os.environ.pop("TBENCH_REWARD_HMAC_KEY_FILE", None) from yeto.rl import tbench_outcome # noqa: E402 -from yeto.rl.harness.codex import reward # noqa: E402 +from yeto.rl import secrlenv_outcome as reward # noqa: E402 from yeto.rl.trajectory_evidence import ( # noqa: E402 build_trajectory_batch_evidence, write_trajectory_batch_evidence, diff --git a/tests/golden/decoupling/codex_harness.json b/tests/golden/decoupling/codex_harness.json deleted file mode 100644 index b957d6a5d..000000000 --- a/tests/golden/decoupling/codex_harness.json +++ /dev/null @@ -1,273 +0,0 @@ -{ - "algorithm_semantic_sha256": "1e843db7fd7e17df813ae42634f2b1f724082cbc29b2a475104e793a820d7eca", - "algorithm_sha256": "27df1133c924e7a337e90246c7e1e30699017f461e9f35b1dd165a0263e65bea", - "algorithm_sha256_launcher_expected": "27df1133c924e7a337e90246c7e1e30699017f461e9f35b1dd165a0263e65bea", - "algorithm_sha256_learner_ports_spec": "27df1133c924e7a337e90246c7e1e30699017f461e9f35b1dd165a0263e65bea", - "algorithm_spec": { - "advantage_estimator": "grpo", - "dynamic_sampling_filter": null, - "dynamic_sampling_max_replacements": null, - "kl_coef": null, - "loss": "policy_loss", - "schema": "yeto-rl-algorithm-spec-v1" - }, - "backend_identity": { - "compat_group": "nvidia-h100", - "device_family": "nvidia", - "engine": "miles", - "param_map_sha256": "36c37d691b6372ee86485cf0f7d4f8d5e185a6346b875aac33380518ee8f259a", - "schema": "yeto-backend-identity-v2" - }, - "execution_profile": { - "contract_hash": "sha256:67aee42a1a530e9b32695039266e1cef5393816354f3dd9363a6467a31823d43", - "profile": { - "algorithm_contract": "on-policy", - "algorithm_spec_sha256": "27df1133c924e7a337e90246c7e1e30699017f461e9f35b1dd165a0263e65bea", - "allowed_overlap": [], - "execution_mode": "partitioned-serial", - "groups_per_batch": 4, - "lr_schedule_sha256": "1a5872e05b262db8c1b8fd48b5b7249b41e479181097e724ae6200ed122d0e99", - "max_inflight_batches": 1, - "max_policy_age": 0, - "name": "miles-lora-partitioned-serial", - "optimizer_steps_per_round": 1, - "outer_protocol": "strict-avg", - "publish_rule": "after-every-update", - "ready_buffer_groups": 0, - "samples_per_group": 2, - "schema": "yeto-rl-execution-profile-v1" - } - }, - "launch_algorithm_sha256": "27df1133c924e7a337e90246c7e1e30699017f461e9f35b1dd165a0263e65bea", - "launch_cli": [ - "--rl-placement", - "fixed-partition", - "--rl-rollout-gpus", - "1", - "--gpu", - "aws:2xa100@us-east-1", - "--model", - "Qwen/Qwen3.5-0.8B", - "--model-revision", - "2fc06364715b967f1860aea9cf38778875588b17", - "--lora-targets", - "attention", - "--custom-generate-function-path", - "miles.rollout.generate_hub.agentic_tool_call.generate", - "--custom-agent-function-path", - "yeto.rl.harness.codex.codex_openenv_subprocess_agent_function.run", - "--codex-reasoning-effort", - "xhigh", - "--codex-backend-profile", - "qwen35_08b", - "--use-session-server", - "--tito-model", - "qwen35", - "--tito-allowed-append-roles", - "tool", - "user", - "--agent-max-seq-len", - "128" - ], - "miles_argv": [ - "train.py", - "--train-backend", - "megatron", - "--hf-checkpoint", - "/model", - "--ref-load", - "/model", - "--megatron-to-hf-mode", - "bridge", - "--model-name", - "SimpleNamespace", - "--num-layers", - "2", - "--hidden-size", - "16", - "--num-attention-heads", - "4", - "--num-query-groups", - "2", - "--kv-channels", - "4", - "--ffn-hidden-size", - "32", - "--max-position-embeddings", - "32768", - "--seq-length", - "2048", - "--normalization", - "RMSNorm", - "--norm-epsilon", - "1e-06", - "--position-embedding-type", - "rope", - "--rotary-base", - "1000000", - "--rotary-percent", - "1.0", - "--vocab-size", - "64", - "--lora-rank", - "16", - "--lora-alpha", - "16", - "--lora-dropout", - "0", - "--lora-type", - "canonical_lora", - "--target-modules", - "q_proj,k_proj,v_proj,o_proj", - "--sglang-max-lora-rank", - "16", - "--actor-num-nodes", - "1", - "--actor-num-gpus-per-node", - "1", - "--num-gpus-per-node", - "2", - "--rollout-num-gpus-per-engine", - "1", - "--rollout-num-gpus", - "1", - "--update-weight-transfer-mode", - "broadcast", - "--no-offload-train", - "--sglang-mem-fraction-static", - "0.4", - "--tensor-model-parallel-size", - "1", - "--pipeline-model-parallel-size", - "1", - "--context-parallel-size", - "1", - "--expert-model-parallel-size", - "1", - "--expert-tensor-parallel-size", - "1", - "--prompt-data", - "/root/yeto-rl/prompts.jsonl", - "--input-key", - "messages", - "--label-key", - "label", - "--metadata-key", - "metadata", - "--rollout-seed", - "0", - "--num-rollout", - "3", - "--rollout-batch-size", - "4", - "--n-samples-per-prompt", - "2", - "--over-sampling-batch-size", - "4", - "--num-steps-per-rollout", - "1", - "--global-batch-size", - "8", - "--balance-data", - "--rollout-max-context-len", - "2048", - "--rollout-max-response-len", - "128", - "--rollout-sample-filter-path", - "yeto.rl.adapters.miles.rollout_meta_hook.record_trained_groups", - "--rollout-all-samples-process-path", - "yeto.rl.adapters.miles.rollout_meta_hook.extract_rollout_metadata", - "--buffer-filter-path", - "yeto.rl.adapters.miles.rollout_meta_hook.policy_buffer_filter", - "--custom-rm-path", - "pkg.reward.score", - "--advantage-estimator", - "grpo", - "--lr", - "0.0003", - "--accumulate-allreduce-grads-in-fp32", - "--attention-softmax-in-fp32", - "--attention-backend", - "unfused", - "--no-gradient-accumulation-fusion", - "--bf16", - "--no-load-optim", - "--no-load-rng", - "--no-save-optim", - "--no-save-rng", - "--finetune", - "--seed", - "0", - "--pin-rollout-manager-to-head", - "--lr-decay-style", - "linear", - "--lr-decay-iters", - "3", - "--lr-warmup-iters", - "0", - "--min-lr", - "0", - "--sglang-enable-deterministic-inference", - "--distributed-timeout-minutes", - "10", - "--apply-chat-template-kwargs", - "{\"clear_thinking\":false}", - "--custom-generate-function-path", - "miles.rollout.generate_hub.agentic_tool_call.generate", - "--custom-agent-function-path", - "yeto.rl.harness.codex.codex_openenv_subprocess_agent_function.run", - "--max-seq-len", - "128", - "--use-session-server", - "--session-server-port", - "9301", - "--tito-model", - "qwen35", - "--sglang-reasoning-parser", - "", - "--sglang-tool-call-parser", - "", - "--group-query-attention" - ], - "miles_argv_sha256": "279d3bae4c7c8381c9dfbea4e0e83ffdbfcd0afbeabb0f8931d7c3312984f771", - "name": "codex_harness", - "placement": { - "allow_cross_node_engine_tp": false, - "allow_cross_node_tp": false, - "bundle_map": null, - "expert_parallel": 1, - "gpus_per_engine": 1, - "gpus_per_node": null, - "kind": "fixed-partition", - "model_parallel": 1, - "node_parallel": 1, - "rollout_cell_names": [], - "rollout_gpus": 1, - "standby_gpus": 0, - "trainer_gpus": 1 - }, - "plugins": [ - { - "module_file": "yeto/rl/adapters/miles/rollout_meta_hook.py", - "path": "yeto.rl.adapters.miles.rollout_meta_hook.extract_rollout_metadata" - }, - { - "module_file": "yeto/rl/adapters/miles/rollout_meta_hook.py", - "path": "yeto.rl.adapters.miles.rollout_meta_hook.policy_buffer_filter" - }, - { - "module_file": "yeto/rl/adapters/miles/rollout_meta_hook.py", - "path": "yeto.rl.adapters.miles.rollout_meta_hook.record_trained_groups" - }, - { - "module_file": "yeto/rl/harness/codex/codex_openenv_subprocess_agent_function.py", - "path": "yeto.rl.harness.codex.codex_openenv_subprocess_agent_function.run" - } - ], - "runtime_attrs": { - "yeto_rl_dynamic_sampling_max_replacements": null - }, - "session_contract_hash": null, - "session_contract_hash_note": "运行时由 LoRA 张量布局算出（yeto.protocol.layout_fingerprint），再与岛契约 v3（后端身份、学习率计划、语义哈希、代码包摘要）绑定（yeto.rl.engine.backend_identity），离线不可得", - "title": "codex harness（ports，Qwen3.5-0.8B，假 bundle 合同）" -} diff --git a/tests/import_boundary_allowlist.txt b/tests/import_boundary_allowlist.txt index 9d257f9a8..6d8455c07 100644 --- a/tests/import_boundary_allowlist.txt +++ b/tests/import_boundary_allowlist.txt @@ -12,11 +12,6 @@ yeto/rl/algos/reducers.py megatron # A3 :35,41 —— 暂不动（任务 4.8， yeto/rl/algos/vendor/miles_mis.py miles # A3 :350 cp_utils —— 暂不动（任务 4.8） yeto/rl/grad_audit.py megatron # A3 :209,320 —— 暂不动（任务 4.8） # -# --- harness（A6、A7、R4-R7、R14） -yeto/rl/harness/codex/reward.py miles # R4 :159,222 Sample.Status / DynamicFilterOutput -yeto/rl/harness/codex/tbench_reward.py miles # R5 :32,101 -yeto/rl/harness/codex/tb2_provider.py modal # 审计未列（扫描新发现）：:362,368,679 tb2 环境用 Modal 沙箱；云层接口时再定归属 -# # --- 奖励与过滤（R1-R3） yeto/rl/math_reward.py miles # R1 :24 math_utils —— 保留（D9c，运行时 math 奖励需装 Miles） yeto/rl/filters.py miles # R3 :30 DynamicFilterOutput diff --git a/tests/test_codex_bridge_compaction.py b/tests/test_codex_bridge_compaction.py deleted file mode 100644 index 504c61fa6..000000000 --- a/tests/test_codex_bridge_compaction.py +++ /dev/null @@ -1,505 +0,0 @@ -"""CompactionRL compaction at the Codex Responses bridge (design D8; CPU only). - -Fake Codex client (direct POSTs to the bridge, as Codex app-server would send -them) + fake session server (aiohttp, per-session routes). No Ray, no Miles. -""" - -from __future__ import annotations - -import asyncio -import copy -import json -from dataclasses import asdict, dataclass -from types import SimpleNamespace -from typing import Any - -import aiohttp -import pytest -from aiohttp import web - -from test_secrlenv_codex_harness import _codex_body, _completion, _fake_miles_sse, _sse_events -from yeto.rl.compaction import RESUME_TEMPLATE, SUMMARY_PROMPT -from yeto.rl.harness.codex import codex_harness_agent as harness -from yeto.rl.harness.codex import codex_openenv_agent_function as adapter -from yeto.rl.harness.codex import codex_openenv_generate as generate_wrapper -from yeto.rl.harness.codex import compaction_bridge as cb - -C, T = 8000, 4000 - - -def _run(coro): - return asyncio.run(coro) - - -def _summary(text: str, total: int, finish: str = "stop") -> dict[str, Any]: - return { - "id": "chatcmpl-summary", - "object": "chat.completion", - "created": 1, - "model": harness.BACKEND_MODEL, - "choices": [{"index": 0, "finish_reason": finish, "message": { - "role": "assistant", "content": text, "reasoning_content": "compress"}}], - "usage": {"prompt_tokens": total - 10, "completion_tokens": 10, "total_tokens": total}, - } - - -class FakeSessionServer: - """Per-session Chat Completions; records (session, body, headers).""" - - def __init__(self, replies: list[dict[str, Any]]): - self.replies = list(replies) - self.requests: list[tuple[str, dict[str, Any], dict[str, str]]] = [] - - async def __aenter__(self): - async def chat(request: web.Request) -> web.Response: - self.requests.append((request.match_info["sid"], await request.json(), dict(request.headers))) - return web.Response(body=_fake_miles_sse(self.replies.pop(0)), content_type="text/event-stream") - - app = web.Application() - app.router.add_post("/sessions/{sid}/v1/chat/completions", chat) - self.runner = web.AppRunner(app, access_log=None) - await self.runner.setup() - site = web.TCPSite(self.runner, "127.0.0.1", 0) - await site.start() - self.router = f"http://127.0.0.1:{site._server.sockets[0].getsockname()[1]}" - return self - - async def __aexit__(self, *_): - await self.runner.cleanup() - - def url(self, sid: str) -> str: - return f"{self.router}/sessions/{sid}" - - -class FakeCodex: - """Sends what Codex app-server sends: the full append-only history every turn.""" - - def __init__(self, bridge: harness._ResponsesBridge, http: aiohttp.ClientSession): - self.bridge, self.http, self.raw = bridge, http, [] - - async def turn(self, tool_output: str | None = None) -> tuple[int, dict[str, Any] | None]: - if tool_output is not None: - self.bridge.expect_tool_output(self.bridge._pending.call_id, tool_output) - history = copy.deepcopy(self.bridge._expected_input) - else: - history = [{"type": "message", "role": "user", "content": [{"type": "input_text", "text": "solve target"}]}] - async with self.http.post(f"{self.bridge.url}/v1/responses", json=_codex_body(history), - headers={"Authorization": f"Bearer {self.bridge.token}"}) as response: - text = await response.text() - self.raw.append((response.status, text)) - if response.status != 200: - return response.status, None - return 200, _sse_events(text)[-1]["response"] - - -def _env(monkeypatch, *, on: bool = True, t_comp: int = T, max_tokens: int = 512): - monkeypatch.setenv("YETO_CODEX_BACKEND_MAX_TOKENS", str(max_tokens)) - monkeypatch.delenv("YETO_CODEX_COMPACTION_ENABLED", raising=False) - if on: - monkeypatch.setenv(cb.COMPACTIONRL_ENV, "1") - monkeypatch.setenv(cb.T_COMP_ENV, str(t_comp)) - else: - monkeypatch.delenv(cb.COMPACTIONRL_ENV, raising=False) - monkeypatch.delenv(cb.T_COMP_ENV, raising=False) - - -def _exec(i: int, total: int) -> dict[str, Any]: - return _completion("terminal.exec", json.dumps({"command": f"step {i}"}), f"call-{i}", f"think {i}", total) - - -def _submit(total: int) -> dict[str, Any]: - return _completion("submit", '{"evidence":"done"}', "call-submit", "finish", total) - - -def _bridge(server, metrics, *, cls=cb.CompactionRLBridge, **kw): - if cls is cb.CompactionRLBridge: - kw.setdefault("segment_base_urls", [server.url(f"S{i}") for i in (1, 2, 3)]) - kw.setdefault("config", cb.compaction_config(C)) - return cls(server.url("S0"), "solve target", {"max_tokens": 512}, metrics, max_seq_len=C, **kw) - - -def _by_session(server): - out: dict[str, list[dict[str, Any]]] = {} - for sid, body, _headers in server.requests: - out.setdefault(sid, []).append(body) - return out - - -# --- trigger, trainable summary, rebuild, session switch -------------------- - -def test_trigger_summary_rebuild_and_new_session(monkeypatch): - _env(monkeypatch) - replies = [_exec(1, 500), _exec(2, 3900), _summary("scratchS-ONE", 4300), - _submit(900)] - - async def scenario(): - metrics = harness.legacy.AgentMetrics() - async with FakeSessionServer(replies) as server: - async with _bridge(server, metrics) as bridge, aiohttp.ClientSession() as http: - codex = FakeCodex(bridge, http) - assert (await codex.turn())[0] == 200 - assert (await codex.turn('{"output":"a"}'))[0] == 200 # 500+bound: no trigger - status, resp = await codex.turn('{"output":"b"}') # 3900+bound > C-T: compaction - assert status == 200 and resp["output"][1]["name"] == "submit" - assert bridge._expected_input is not None and len(bridge._history_after_model) == 1 + 3 * 2 + 2 # Codex keeps the full history - return server, metrics - - server, metrics = _run(scenario()) - sessions = _by_session(server) - assert [len(sessions[s]) for s in ("S0", "S1")] == [3, 1] and set(sessions) == {"S0", "S1"} - first, second, summary_req = sessions["S0"] - # Summary: same policy, same session (trainable turn with logprobs in the session record). - assert summary_req["messages"][:-1] == second["messages"] + summary_req["messages"][len(second["messages"]):-1] - assert summary_req["messages"][-1] == {"role": "user", "content": SUMMARY_PROMPT} - assert summary_req["tool_choice"] == "none" and summary_req["model"] == second["model"] - # Explicit D8 choice: the summary request carries no tool table (execution turns do). - assert summary_req["tools"] == list(cb.SUMMARY_REQUEST_TOOLS) == [] - assert summary_req["tool_choice"] == cb.SUMMARY_REQUEST_TOOL_CHOICE and second["tools"] - assert all(not k.lower().startswith("x-miles-compaction") for _s, _b, h in server.requests for k in h) - # Rebuilt context = system + u_resume(summary without ) + last k=2 atomic steps, in S1. - resumed = sessions["S1"][0]["messages"] - assert resumed[0] == {"role": "system", "content": harness.BASE_INSTRUCTIONS} - assert resumed[1] == {"role": "user", "content": RESUME_TEMPLATE.format(summary="S-ONE")} - assert resumed[2:] == summary_req["messages"][2:-1] - assert len(resumed[2:]) == 4 and "scratch" not in json.dumps(resumed) - record = cb.compaction_counters(metrics)[cb.METRICS_KEY] - assert record["compactions"] == 1 and record["t_comp"] == T and record["context_budget"] == C - assert record["segments"][0]["summary_ok"] is True and record["segments"][0]["kept_steps"] == 2 - assert metrics.turns == 3 # the summary is not a Codex turn - - -def test_trigger_boundary_is_remaining_context_below_t_comp(monkeypatch): - _env(monkeypatch) - bridge = cb.CompactionRLBridge("http://127.0.0.1:1/sessions/S0", "p", {"max_tokens": 512}, - harness.legacy.AgentMetrics(), max_seq_len=C, - segment_base_urls=["http://127.0.0.1:1/sessions/S1"] * 3) - bridge._atomic_steps = [({}, {})] - assert not bridge._should_compact(C - T) # C-|h| == T_comp: no trigger - assert bridge._should_compact(C - T + 1) - bridge._compaction_count = 3 - assert not bridge._should_compact(C) # cap reached - with pytest.raises(harness.CodexHarnessError, match="one pre-created session"): - cb.CompactionRLBridge("http://h/sessions/S0", "p", {"max_tokens": 512}, harness.legacy.AgentMetrics(), - max_seq_len=C, segment_base_urls=["http://h/sessions/S1"]) - with pytest.raises(harness.CodexHarnessError, match="config rejected"): - cb.compaction_config(T) # C must exceed T_comp - - -def test_kept_steps_shrink_when_rebuilt_context_would_trigger(monkeypatch): - _env(monkeypatch) - big = "x" * 1500 - replies = [_exec(1, 300), _exec(2, 2500), _summary("S", 5900), _submit(1200)] - - async def scenario(): - metrics = harness.legacy.AgentMetrics() - async with FakeSessionServer(replies) as server: - async with _bridge(server, metrics) as bridge, aiohttp.ClientSession() as http: - codex = FakeCodex(bridge, http) - await codex.turn() - await codex.turn(big) - assert (await codex.turn(big))[0] == 200 - return server, metrics - - server, metrics = _run(scenario()) - resumed = _by_session(server)["S1"][0]["messages"] - summary_req = _by_session(server)["S0"][-1]["messages"] - k = cb.compaction_counters(metrics)[cb.METRICS_KEY]["segments"][0]["kept_steps"] - assert k < 2 # two 1.5 kB steps would make the rebuilt context trigger again - assert resumed[2:] == (summary_req[2:-1][-2 * k:] if k else []) - - -def test_cap_of_three_then_run_to_full_truncation(monkeypatch): - _env(monkeypatch) - replies = [_exec(1, 3900)] - for i in range(3): - replies += [_summary(f"S{i}", 4200), _exec(i + 2, 3900)] - replies += [_exec(9, 7995)] # 4th trigger is ignored; the context then fills up - - async def scenario(): - metrics = harness.legacy.AgentMetrics() - async with FakeSessionServer(replies) as server: - async with _bridge(server, metrics) as bridge, aiohttp.ClientSession() as http: - codex = FakeCodex(bridge, http) - assert (await codex.turn())[0] == 200 - for _ in range(4): - assert (await codex.turn('{"output":"o"}'))[0] == 200 - status, _resp = await codex.turn('{"output":"o"}') - fatal = bridge.fatal.result() if bridge.fatal.done() else None - return server, metrics, status, fatal - - server, metrics, status, fatal = _run(scenario()) - assert status == 400 and isinstance(fatal, harness.CodexSequenceLimit) - assert metrics.max_seq_len_hit == 1 # -> policy status max_seq_len (truncated, still rewarded) - sessions = _by_session(server) - assert [len(sessions[f"S{i}"]) for i in range(4)] == [2, 2, 2, 2] - assert sessions["S3"][0]["messages"][1]["content"] == RESUME_TEMPLATE.format(summary="S2") - record = cb.compaction_counters(metrics)[cb.METRICS_KEY] - assert record["compactions"] == 3 and len(record["segments"]) == 3 - - -def test_summary_without_tag_is_kept_and_flagged(monkeypatch): - _env(monkeypatch) - replies = [_exec(1, 3900), _summary("aplain state", 4400, finish="length"), _submit(500)] - - async def scenario(): - metrics = harness.legacy.AgentMetrics() - async with FakeSessionServer(replies) as server: - async with _bridge(server, metrics) as bridge, aiohttp.ClientSession() as http: - codex = FakeCodex(bridge, http) - await codex.turn() - assert (await codex.turn('{"output":"o"}'))[0] == 200 - return server, metrics - - server, metrics = _run(scenario()) - seg = cb.compaction_counters(metrics)[cb.METRICS_KEY]["segments"][0] - assert seg["summary_ok"] is False and seg["summary_finish_reason"] == "length" - assert _by_session(server)["S1"][0]["messages"][1]["content"] == RESUME_TEMPLATE.format(summary="plain state") - - -def test_no_room_for_summary_truncates(monkeypatch): - _env(monkeypatch) - replies = [_exec(1, 7000)] - - async def scenario(): - async with FakeSessionServer(replies) as server: - async with _bridge(server, harness.legacy.AgentMetrics()) as bridge, aiohttp.ClientSession() as http: - codex = FakeCodex(bridge, http) - await codex.turn() - status, _ = await codex.turn("y" * 800) - return status, bridge.fatal.result(), len(server.requests) - - status, fatal, n = _run(scenario()) - assert status == 400 and isinstance(fatal, harness.CodexSequenceLimit) and n == 1 - - -def test_codex_argv_disables_codex_auto_compaction(monkeypatch): - _env(monkeypatch) - bridge = SimpleNamespace(url="http://127.0.0.1:9") - argv = cb.codex_argv(harness.Path("/codex"), bridge) - stock = harness._codex_argv(harness.Path("/codex"), bridge) - assert argv == [*stock, "-c", "model_auto_compact_token_limit=9223372036854775807"] - assert not any("auto_compact" in a for a in stock) - - -# --- compaction off: stock path byte-identical ------------------------------ - -def _script(): - return [_exec(1, 500), _exec(2, 2900), _exec(3, 3300), _submit(3600)] - - -async def _drive(cls, monkeypatch, **kw): - async with FakeSessionServer(_script()) as server: - metrics = harness.legacy.AgentMetrics() - async with _bridge(server, metrics, cls=cls, **kw) as bridge, aiohttp.ClientSession() as http: - codex = FakeCodex(bridge, http) - await codex.turn() - for out in ('{"o":1}', '{"o":2}', '{"o":3}'): - await codex.turn(out) - return [(s, b) for s, b, _h in server.requests], codex.raw, asdict(metrics) - - -def test_off_stock_bridge_unchanged_and_compactionrl_without_trigger_is_byte_identical(monkeypatch): - # created_at is wall-clock seconds; a run that crosses a second boundary - # would differ, so pin the clock this module reads (monotonic stays real). - monkeypatch.setattr(harness, "time", SimpleNamespace(time=lambda: 1_700_000_000.0, - monotonic=harness.time.monotonic)) - _env(monkeypatch, on=False) - stock_requests, stock_raw, stock_metrics = _run(_drive(harness._ResponsesBridge, monkeypatch)) - _env(monkeypatch, on=True, t_comp=10) # never triggers below C-10 - rl_requests, rl_raw, rl_metrics = _run(_drive(cb.CompactionRLBridge, monkeypatch, - config=cb.compaction_config(C, {cb.T_COMP_ENV: "10"}))) - stock_metrics.pop("total_generation_time"), rl_metrics.pop("total_generation_time") # wall clock - assert rl_requests == stock_requests and rl_raw == stock_raw and rl_metrics == stock_metrics - - -def test_off_drive_untrusted_trusted_side_and_metrics_are_stock(monkeypatch): - _env(monkeypatch, on=False) - calls = [] - - async def fake_drive(*args, **kwargs): - calls.append((args[1], args[4], kwargs)) - return "completed" - - monkeypatch.setattr(harness, "_drive_codex", fake_drive) - job = {"base_url": "http://r/sessions/S0", "prompt": "p", "request_kwargs": {"temperature": 1.0}, - "episode_id": "ep-1", "max_seq_len": C} - before = copy.deepcopy(job) - assert _run(adapter.prepare_segment_sessions(job, post=None)) == {} and job == before - result = _run(adapter.drive_untrusted(job, SimpleNamespace(), binary=harness.Path("/codex"))) - assert calls == [("http://r/sessions/S0", {"temperature": 1.0}, {"max_seq_len": C})] - metrics = harness.legacy.AgentMetrics() - assert set(result["metrics"]) == set(asdict(metrics)) | set(harness.tito_counters(metrics)) - with pytest.raises(harness.CodexHarnessError, match="without CompactionRL"): - _run(adapter.drive_untrusted({**job, cb.SEGMENT_URLS_KEY: ["u"] * 3}, SimpleNamespace(), - binary=harness.Path("/codex"))) - - -def test_on_trusted_side_precreates_sessions_and_worker_requires_them(monkeypatch): - _env(monkeypatch) - posts = [] - - async def post(url, body): - posts.append((url, body)) - return {"session_id": f"seg{len(posts)}"} - - job = {"base_url": "http://r:1/sessions/S0", "prompt": "p", - "request_kwargs": {"temperature": 0.7, "top_p": 1, "top_k": 20, "max_tokens": 9}, "episode_id": "e", - "max_seq_len": C} - meta = _run(adapter.prepare_segment_sessions(job, post=post)) - assert meta == {cb.SESSIONS_METADATA_KEY: ["seg1", "seg2", "seg3"], cb.SESSIONS_ROUTER_METADATA_KEY: "http://r:1"} - assert cb.SESSIONS_ROUTER_METADATA_KEY == generate_wrapper.ROUTER_KEY - assert job[cb.SEGMENT_URLS_KEY] == [f"http://r:1/sessions/seg{i}" for i in (1, 2, 3)] - assert posts == [("http://r:1/sessions", {"evaluation": False, "temperature": 0.7, "top_p": 1.0, "top_k": 20})] * 3 - with pytest.raises(harness.CodexHarnessError, match="max_seq_len"): - _run(adapter.prepare_segment_sessions({**job, "max_seq_len": None}, post=post)) - with pytest.raises(harness.CodexHarnessError, match="no pre-created segment sessions"): - _run(adapter.drive_untrusted({k: v for k, v in job.items() if k != cb.SEGMENT_URLS_KEY}, - SimpleNamespace(), binary=harness.Path("/codex"))) - - -# --- generate wrapper: one sample per segment ------------------------------- - -def _sample(n_prompt, n_gen, version="pv", **meta): - tokens = list(range(n_prompt + n_gen)) - span = SimpleNamespace(version=version, abs_start=n_prompt, abs_end=n_prompt + n_gen) - return SimpleNamespace(tokens=tokens, loss_mask=[1] * n_gen, rollout_log_probs=[-0.1] * n_gen, - response_length=n_gen, weight_versions=[SimpleNamespace(spans=[span])], - metadata=dict(meta), status=None) - - -def _seg0_meta(compactions, sessions=("a", "b", "c"), **extra): - return {"expected_policy_version": "pv", "session_server_id": "10.0.0.1:9", "exit_status": "completed", - "tbench_trusted_outcome": {"reward": 1.0}, - "chain_index": 0, "chains_total": 1, "segment_id": 0, "chain_break_reason": None, - generate_wrapper.SESSIONS_KEY: list(sessions), - "agent_metrics": {cb.METRICS_KEY: {"compactions": compactions}}, **extra} - - -@dataclass(frozen=True) -class _Out: - samples: Any - - -def _wrap(seg0, collected, *, raise_on=None): - inp = SimpleNamespace(sample=SimpleNamespace(group_index=7, index=11, metadata={"expected_policy_version": "pv"})) - seen = [] - - async def upstream(_input): - return _Out(samples=seg0) - - async def collect(_input, router, sid): - seen.append((router, sid)) - if sid == raise_on: - raise TimeoutError("boom") - return collected.get(sid, ([], {})) - - return _run(generate_wrapper.generate(inp, upstream=upstream, collect=collect)), seen - - -def test_wrapper_splits_segments_with_metadata_and_shared_reward(): - seg0 = _sample(10, 30, **_seg0_meta(2)) - s1, s2 = _sample(8, 20), _sample(8, 5) - out, seen = _wrap(seg0, {"a": ([s1], {"sm": 1}), "b": ([s2], {})}) - assert seen == [("http://10.0.0.1:9", "a"), ("http://10.0.0.1:9", "b"), ("http://10.0.0.1:9", "c")] - assert out.samples == [seg0, s1, s2] and all(s.status is None for s in out.samples) - md = [s.metadata for s in out.samples] - assert [m["segment_index"] for m in md] == [0, 1, 2] and {m["num_segments"] for m in md} == {3} - assert [m["segment_tokens"] for m in md] == [30, 20, 5] - assert [m["tokens_after"] for m in md] == [25, 5, 0] and {m["gae_length"] for m in md} == {55} - assert [m["chain_index"] for m in md] == [0, 1, 2] and {m["chains_total"] for m in md} == {3} - assert [m["chain_break_reason"] for m in md] == [None, "compaction_window", "compaction_window"] - assert {json.dumps(m["tbench_trusted_outcome"]) for m in md} == {'{"reward": 1.0}'} - assert {s.rollout_id for s in out.samples} == {11} and {s.group_index for s in out.samples} == {7} - assert md[1]["sm"] == 1 and all(m["truncated"] is False and m["compactions"] == 2 for m in md) - - -def test_wrapper_no_compaction_keeps_single_sample_and_drains_sessions(): - seg0 = _sample(10, 30, **_seg0_meta(0, exit_status="max_seq_len")) - out, seen = _wrap(seg0, {}) - assert out.samples is seg0 and len(seen) == 3 - assert seg0.metadata["num_segments"] == 1 and seg0.metadata["tokens_after"] == 0 - assert seg0.metadata["truncated"] is True - - -@pytest.mark.parametrize("case", ["mismatch_used_empty", "mismatch_unused_nonempty", "collect_error", "no_record"]) -def test_wrapper_aborts_on_inconsistent_segments(case): - meta = _seg0_meta(1) - collected = {"a": ([_sample(4, 4)], {})} - raise_on = None - if case == "mismatch_used_empty": - collected = {} - elif case == "mismatch_unused_nonempty": - collected["b"] = ([_sample(4, 4)], {}) - elif case == "collect_error": - raise_on = "a" - else: - meta["agent_metrics"] = {} - seg0 = _sample(10, 30, **meta) - out, seen = _wrap(seg0, collected, raise_on=raise_on) - assert len(seen) == 3 # every pre-created session is still collected (deleted) - assert out.samples is seg0 and seg0.status == "ABORTED" - assert "tbench_trusted_outcome" not in seg0.metadata - - -def test_wrapper_infrastructure_failure_only_drains(): - seg0 = _sample(10, 30, **_seg0_meta(1, **{generate_wrapper.INFRASTRUCTURE_KEY: "x"})) - out, seen = _wrap(seg0, {"a": ([_sample(4, 4)], {})}) - assert len(seen) == 3 and out.samples is seg0 and "segment_index" not in seg0.metadata - - -def test_wrapper_without_compaction_metadata_is_untouched(): - seg0 = _sample(10, 30, expected_policy_version="pv") - before = copy.deepcopy(seg0.metadata) - out, seen = _wrap(seg0, {}) - assert seen == [] and out.samples is seg0 - assert {k: v for k, v in seg0.metadata.items() if k not in before} == { - "chain_index": 0, "chains_total": 1, "policy_versions_actual": ["pv"]} - - -def test_wrapper_collects_on_the_recorded_router_and_aborts_without_one(): - """s19-compaction-g1-20261010d: the returned metadata had no session_server_id; - the collect URL was "http:///sessions/" and retried forever.""" - meta = _seg0_meta(1, **{generate_wrapper.ROUTER_KEY: "http://10.0.0.2:7"}) - del meta["session_server_id"] - seg0, s1 = _sample(10, 30, **meta), _sample(8, 20) - out, seen = _wrap(seg0, {"a": ([s1], {})}) - assert {r for r, _ in seen} == {"http://10.0.0.2:7"} and [m.metadata["tokens_after"] for m in out.samples] == [20, 0] - bare = _seg0_meta(1) - del bare["session_server_id"] - seg0 = _sample(10, 30, **bare) - out, seen = _wrap(seg0, {"a": ([s1], {})}) - assert seen == [] and generate_wrapper.INFRASTRUCTURE_KEY in seg0.metadata - - -def test_wrapper_bounds_each_segment_collect(monkeypatch): - """A hanging collect (Miles retries) aborts the rollout after the budget.""" - import asyncio - - monkeypatch.setenv(generate_wrapper.SEGMENT_COLLECT_TIMEOUT_ENV, "0.05") - seg0 = _sample(10, 30, **_seg0_meta(1)) - inp = SimpleNamespace(sample=SimpleNamespace(group_index=7, index=11, metadata={"expected_policy_version": "pv"})) - - async def upstream(_input): - return _Out(samples=seg0) - - async def hang(_input, _router, _sid): - await asyncio.sleep(30) - - out = _run(generate_wrapper.generate(inp, upstream=upstream, collect=hang)) - assert "timed out" in seg0.metadata[generate_wrapper.INFRASTRUCTURE_KEY] - assert generate_wrapper.segment_collect_timeout_s() == 0.05 - monkeypatch.setenv(generate_wrapper.SEGMENT_COLLECT_TIMEOUT_ENV, "bad") - assert generate_wrapper.segment_collect_timeout_s() == generate_wrapper.DEFAULT_SEGMENT_COLLECT_TIMEOUT_S - - -def test_wrapper_returns_a_list_for_every_rollout_when_compactionrl_is_on(monkeypatch): - """s19-compaction-g1-20261010e: mixed list / bare-Sample outputs broke Miles' - flattening ("'Sample' object is not iterable").""" - seg0 = _sample(10, 30, **_seg0_meta(0)) - monkeypatch.setenv(cb.COMPACTIONRL_ENV, "1") - out, _ = _wrap(seg0, {}) - assert out.samples == [seg0] - seg0 = _sample(10, 30, **_seg0_meta(0)) - monkeypatch.setenv(cb.COMPACTIONRL_ENV, "0") - out, _ = _wrap(seg0, {}) - assert out.samples is seg0 # CompactionRL off: unchanged diff --git a/tests/test_codex_compaction_wiring.py b/tests/test_codex_compaction_wiring.py deleted file mode 100644 index 2b37fe33c..000000000 --- a/tests/test_codex_compaction_wiring.py +++ /dev/null @@ -1,356 +0,0 @@ -"""CompactionRL switch wiring (progress.md "S13 Codex 压缩接线"; CPU only, no Ray). - -- spec <-> switch: launcher env injection, preflight consistency, control arm; -- pre-created segment sessions are deleted when they cannot reach the - generate wrapper (half-way creation, worker crash, finish_trusted error); -- R-D5a chain-count judge with/without compaction; -- module list / summary tool-table constants. -""" - -from __future__ import annotations - -import asyncio -from types import SimpleNamespace - -import pytest -from aiohttp import web - -from yeto import launcher as L -from yeto.rl import CODEX_HARNESS_AGENT, CODEX_OPENENV_AGENT_MODULES -from yeto.rl.algos import compactionrl as crl -from yeto.rl.harness.codex import chain_judge -from yeto.rl.harness.codex import codex_openenv_agent_function as adapter -from yeto.rl.harness.codex import codex_openenv_subprocess_agent_function as subprocess_agent -from yeto.rl.harness.codex import compaction_bridge as cb -from yeto.rl.harness.codex import preflight -from yeto.rl.harness.codex.environment import FakeTerminalEnvironment - -from test_harness_codex_openenv import _Provider, _metadata -from test_rl_launcher_codex_bundle import PROVIDER, _codex_args, bundle # noqa: F401 - fixture - -SWITCH, TCOMP = crl.COMPACTION_SWITCH_ENV, crl.COMPACTION_T_COMP_ENV - - -def _run(coro): - return asyncio.run(coro) - - -def _env(root, **extra): - return {L.CODEX_BUNDLE_DIR_ENV: str(root), L.HARNESS_ENVIRONMENT_PROVIDER_ENV: PROVIDER, **extra} - - -def _with_spec(args, spec): - args.rl_algorithm_spec_json = spec.canonical_json() - return args - - -# --- constants shared by bridge / spec ------------------------------------- - -def test_switch_names_match_the_bridge_and_module_list_includes_it(): - assert crl.COMPACTION_SWITCH_ENV == cb.COMPACTIONRL_ENV and crl.COMPACTION_T_COMP_ENV == cb.T_COMP_ENV - assert crl.compactionrl_spec().canonical_json() and crl.COMPACTED_GAE_VARIANT in crl.compactionrl_spec().canonical_json() - assert "compaction_bridge.py" in CODEX_OPENENV_AGENT_MODULES - assert preflight.required_pin_updates()["CODEX_OPENENV_AGENT_MODULES"] == CODEX_OPENENV_AGENT_MODULES - assert cb.SUMMARY_REQUEST_TOOLS == () and cb.SUMMARY_REQUEST_TOOL_CHOICE == "none" - - -# --- launcher --------------------------------------------------------------- - -def test_launcher_compactionrl_spec_switches_rollout_compaction_on(bundle): # noqa: F811 - root, _ = bundle - args = _with_spec(_codex_args(), crl.compactionrl_spec()) - # upstream agentic_tool_call.generate never collects the segment sessions - # (s19-compaction-g1-20261010b: no tokens_after, trainer failed at round 0) - with pytest.raises(ValueError, match="codex_openenv_generate"): - L.codex_harness_launch(args, environ=_env(root)) - from yeto.rl.adapters.miles import config as miles_config - - args.custom_generate_function_path = miles_config.CODEX_OPENENV_GENERATE - check = miles_config._requires_agentic_generate("--custom-agent-function-path") - assert check("x.y", SimpleNamespace(agent=args)) is None # the wrapper reads the agent flags - assert check("x.y", SimpleNamespace(agent=SimpleNamespace(custom_generate_function_path="a.b"))) - _flags, envs, _ = L.codex_harness_launch(args, environ=_env(root)) - assert envs[SWITCH] == "1" and TCOMP not in envs - # The island preflight forwards YETO_CODEX_* to the Ray rollout workers. - assert preflight.worker_runtime_env(SimpleNamespace(), envs)[SWITCH] == "1" - _flags, envs, _ = L.codex_harness_launch(args, environ=_env(root, **{TCOMP: "4096", SWITCH: "1"})) - assert envs[SWITCH] == "1" and envs[TCOMP] == "4096" - with pytest.raises(ValueError, match="contradicts"): - L.codex_harness_launch(args, environ=_env(root, **{SWITCH: "0"})) - with pytest.raises(ValueError, match="positive integer"): - L.codex_harness_launch(args, environ=_env(root, **{TCOMP: "-3"})) - - -def test_launcher_default_specs_unchanged_and_switch_rejected(bundle): # noqa: F811 - root, _ = bundle - args = _codex_args() # default spec - _f, base, _ = L.codex_harness_launch(args, environ=_env(root)) - assert SWITCH not in base and TCOMP not in base - _f, explicit_off, _ = L.codex_harness_launch(args, environ=_env(root, **{SWITCH: "0"})) - assert explicit_off == base - with pytest.raises(ValueError, match="only valid with cross_segment_per_sample"): - L.codex_harness_launch(args, environ=_env(root, **{SWITCH: "1"})) - with pytest.raises(ValueError, match="without"): - L.codex_harness_launch(args, environ=_env(root, **{TCOMP: "4096"})) - with pytest.raises(ValueError, match="boolean"): - L.codex_harness_launch(args, environ=_env(root, **{SWITCH: "maybe"})) - - -def test_launcher_control_arm_rejected_on_codex(bundle): # noqa: F811 - root, _ = bundle - args = _with_spec(_codex_args(), crl.compactionrl_whole_rollout_control_spec()) - for extra in ({}, {SWITCH: "1"}): - with pytest.raises(ValueError, match="cannot be concatenated"): - L.codex_harness_launch(args, environ=_env(root, **extra)) - - -def test_launcher_compactionrl_needs_the_compacting_harness(bundle): # noqa: F811 - root, _ = bundle - args = _with_spec(_codex_args(), crl.compactionrl_spec()) - args.custom_agent_function_path = CODEX_HARNESS_AGENT # signed, but cannot compact - with pytest.raises(ValueError, match="only the Codex OpenEnv harness"): - L.codex_harness_launch(args, environ=_env(root)) - args.custom_agent_function_path = None # not a Codex run at all - with pytest.raises(ValueError, match="only the Codex OpenEnv harness"): - L.codex_harness_launch(args, environ=_env(root)) - - -# --- island preflight ------------------------------------------------------- - -@pytest.mark.parametrize( - "variant,env,error", - [ - ("cross_segment_per_sample", {SWITCH: "1"}, None), - ("cross_segment_per_sample", {SWITCH: "1", TCOMP: "10240"}, None), - (None, {}, None), - ("vanilla", {SWITCH: "0"}, None), - ("cross_segment_per_sample", {}, "requires YETO_CODEX_COMPACTIONRL=1"), - (None, {SWITCH: "1"}, "only valid with"), - ("vanilla", {TCOMP: "100"}, "without"), - ("cross_segment_whole_rollout", {SWITCH: "1"}, "cannot be concatenated"), - ("cross_segment_whole_rollout", {}, "cannot be concatenated"), - ("cross_segment_per_sample", {SWITCH: "1", TCOMP: "x"}, "positive integer"), - ], -) -def test_preflight_switch_matches_gae_variant(variant, env, error): - args = SimpleNamespace(gae_variant=variant) - if error is None: - preflight.assert_compactionrl_consistent(args, env) - else: - with pytest.raises(preflight.PreflightError, match=error): - preflight.assert_compactionrl_consistent(args, env) - - -# --- segment session cleanup ------------------------------------------------ - -class _SessionServer: - """Fake router: POST /sessions + DELETE /sessions/{id} (pin: 204).""" - - def __init__(self, fail_after: int | None = None): - self.fail_after = fail_after - self.live: set[str] = set() - self.deleted: list[str] = [] - self.n = 0 - - async def __aenter__(self): - async def create(_request): - self.n += 1 - if self.fail_after is not None and self.n > self.fail_after: - return web.Response(status=500) - sid = f"seg{self.n}" - self.live.add(sid) - return web.json_response({"session_id": sid}) - - async def delete(request): - sid = request.match_info["sid"] - if sid not in self.live: - return web.Response(status=404) - self.live.discard(sid) - self.deleted.append(sid) - return web.Response(status=204) - - app = web.Application() - app.router.add_post("/sessions", create) - app.router.add_delete("/sessions/{sid}", delete) - self.runner = web.AppRunner(app, access_log=None) - await self.runner.setup() - site = web.TCPSite(self.runner, "127.0.0.1", 0) - await site.start() - self.url = f"http://127.0.0.1:{site._server.sockets[0].getsockname()[1]}" - return self - - async def __aexit__(self, *exc): - await self.runner.cleanup() - - -def _job(url): - return {"base_url": f"{url}/sessions/S0", "prompt": "p", "request_kwargs": {}, "episode_id": "e", - "max_seq_len": 8000} - - -def test_half_way_precreation_deletes_what_was_created(monkeypatch): - monkeypatch.setenv(SWITCH, "1") - monkeypatch.setenv(TCOMP, "4000") - - async def scenario(): - async with _SessionServer(fail_after=2) as server: - with pytest.raises(RuntimeError, match="HTTP 500"): - await adapter.prepare_segment_sessions(_job(server.url)) - return server - - server = _run(scenario()) - assert server.deleted == ["seg1", "seg2"] and server.live == set() - - -def test_delete_segment_sessions_is_best_effort(): - calls = [] - - async def delete(url): - calls.append(url) - if url.endswith("/b"): - raise OSError("down") - - failed = _run(adapter.delete_segment_sessions("http://r:1/sessions/S0", ["a", "b", "c"], delete=delete)) - assert failed == ["b"] and calls == [f"http://r:1/sessions/{s}" for s in "abc"] - assert _run(adapter.delete_segment_sessions("http://r:1/sessions/S0", None, delete=delete)) == [] - - -def _in_process(monkeypatch, server, *, drive=None, finish=None): - monkeypatch.setenv(SWITCH, "1") - monkeypatch.setenv(TCOMP, "4000") - if drive is not None: - monkeypatch.setattr(adapter, "drive_untrusted", drive) - if finish is not None: - monkeypatch.setattr(adapter, "finish_trusted", finish) - env = FakeTerminalEnvironment(passed=True) - return adapter.run(f"{server.url}/sessions/S0", "p", {}, {"task_id": "fix-git", "trajectory_id": "t", - "max_seq_len": 8000}, - environment=env, verifier=env) - - -@pytest.mark.parametrize("where", ["drive", "finish"]) -def test_in_process_run_deletes_sessions_on_unexpected_errors(monkeypatch, where): - async def crash(*_a, **_k): - raise ValueError("worker crashed") - - async def ok_drive(*_a, **_k): - return {"status": "completed", "episode_id": "e", "metrics": {}} - - async def scenario(): - async with _SessionServer() as server: - with pytest.raises(ValueError, match="worker crashed"): - await _in_process(monkeypatch, server, drive=crash if where == "drive" else ok_drive, - finish=crash if where == "finish" else None) - return server - - server = _run(scenario()) - assert sorted(server.deleted) == ["seg1", "seg2", "seg3"] and server.live == set() - - -def test_in_process_run_hands_sessions_off_on_success(monkeypatch): - async def ok_drive(*_a, **_k): - return {"status": "completed", "episode_id": "e", "metrics": {}} - - async def finish(*_a, **_k): - return {"signed": 1} - - async def scenario(): - async with _SessionServer() as server: - result = await _in_process(monkeypatch, server, drive=ok_drive, finish=finish) - return server, result - - server, result = _run(scenario()) - assert result[cb.SESSIONS_METADATA_KEY] == ["seg1", "seg2", "seg3"] - assert server.deleted == [] and server.live == {"seg1", "seg2", "seg3"} # the wrapper collects them - - -def _subprocess(monkeypatch, *, drive, finish=None): - monkeypatch.setenv(SWITCH, "1") - monkeypatch.setenv(TCOMP, "4000") - provider = _Provider(FakeTerminalEnvironment(passed=True)) - subprocess_agent.configure(provider=provider, tool_wait_board=None, harness_board=None, member="m0") - monkeypatch.setattr(subprocess_agent, "_drive_worker", drive) - if finish is not None: - monkeypatch.setattr(adapter, "finish_trusted", finish) - return provider - - -@pytest.mark.parametrize("where", ["worker", "finish", "harness_error"]) -def test_subprocess_run_cleans_up_or_hands_off_segment_sessions(monkeypatch, where): - async def crash(*_a, **_k): - raise RuntimeError("worker crashed") - - async def harness_error(*_a, **_k): - raise adapter.harness.CodexHarnessError("worker exited without a result") - - async def ok_drive(*_a, **_k): - return {"status": "completed", "episode_id": "e", "metrics": {}} - - drive = {"worker": crash, "finish": ok_drive, "harness_error": harness_error}[where] - provider = _subprocess(monkeypatch, drive=drive, finish=crash if where == "finish" else None) - - async def scenario(): - async with _SessionServer() as server: - md = _metadata(max_seq_len=8000) - # S24 T-TB2: a worker or verifier failure is infrastructure (no reward), not a crash - result = await subprocess_agent.run(f"{server.url}/sessions/S0", "p", {}, md) - return server, result - - try: - server, result = _run(scenario()) - finally: - subprocess_agent.configure(provider=None) - assert provider.destroyed == 1 - if where == "harness_error": # metadata returned -> the generate wrapper drains them - assert result[cb.SESSIONS_METADATA_KEY] == ["seg1", "seg2", "seg3"] and server.deleted == [] - else: - assert "worker crashed" in result["tbench_infrastructure_error"] - assert cb.SESSIONS_METADATA_KEY not in result - assert sorted(server.deleted) == ["seg1", "seg2", "seg3"] and server.live == set() - - -# --- R-D5a chain-count judge ------------------------------------------------ - -def _seg(i, n, traj="t", **kw): - return {"trajectory_id": traj, "chain_index": i, "segment_index": i, "chains_total": n, - "num_segments": n, "compactions": n - 1, - "chain_break_reason": "compaction_window" if i else None, **kw} - - -def test_chain_judge_off_keeps_chains_total_one(): - ok = [{"trajectory_id": "a", "chains_total": 1}, {"trajectory_id": "b", "chains_total": 1}] - assert chain_judge.judge_chain_counts(ok, compaction_enabled=False) == [] - assert chain_judge.judge_chain_counts([_seg(0, 2), _seg(1, 2)], compaction_enabled=False) - assert chain_judge.judge_chain_counts([{"trajectory_id": "a", "chains_total": 2}], compaction_enabled=False) - - -def test_chain_judge_on_counts_segments(): - good = [_seg(0, 3), _seg(1, 3), _seg(2, 3), _seg(0, 1, traj="u"), - {"trajectory_id": "aborted", "chains_total": 1}] - assert chain_judge.judge_chain_counts(good, compaction_enabled=True) == [] - assert chain_judge.judge_chain_counts([_seg(0, 3), _seg(1, 3)], compaction_enabled=True) # missing segment - bad_comp = [_seg(0, 2), {**_seg(1, 2), "compactions": 2}] - assert any("compactions" in p for p in chain_judge.judge_chain_counts(bad_comp, compaction_enabled=True)) - bad_reason = [_seg(0, 2), {**_seg(1, 2), "chain_break_reason": "retry_fork"}] - assert chain_judge.judge_chain_counts(bad_reason, compaction_enabled=True) - bad_index = [_seg(0, 2), {**_seg(1, 2), "chain_index": 0}] - assert chain_judge.judge_chain_counts(bad_index, compaction_enabled=True) - - -def test_generate_wrapper_registers_upstream_agent_flags(monkeypatch): - """Miles parse_args calls the generate function's add_arguments; the wrapper - must register upstream agentic_tool_call's flags (s19-compaction-g1-20261010c).""" - import argparse - - from yeto.rl.harness.codex import codex_openenv_generate as gen - - def upstream(): - pass - - upstream.add_arguments = lambda p: (p.add_argument("--custom-agent-function-path"), - p.add_argument("--max-seq-len", type=int)) - monkeypatch.setattr(gen, "_load_upstream", lambda: upstream) - parser = argparse.ArgumentParser() - gen.generate.add_arguments(parser) - ns = parser.parse_args(["--custom-agent-function-path", "a.b", "--max-seq-len", "16384"]) - assert ns.custom_agent_function_path == "a.b" and ns.max_seq_len == 16384 diff --git a/tests/test_codex_end_kind.py b/tests/test_codex_end_kind.py deleted file mode 100644 index c420d371a..000000000 --- a/tests/test_codex_end_kind.py +++ /dev/null @@ -1,83 +0,0 @@ -"""S17 C9: Codex end reason split + per-turn lengths reach the trajectory event.""" - -from __future__ import annotations - -import asyncio - -import pytest - -from yeto.rl.engine import timeline -from yeto.rl.adapters.miles.rollout_meta_hook import trajectory_diagnostics -from yeto.rl.harness.codex import agent as legacy -from yeto.rl.harness.codex import codex_harness_agent as H -from yeto.rl.harness.codex import codex_openenv_agent_function as F - - -def _driver(exc=None, submit_calls=0): - metrics = legacy.AgentMetrics(submit_calls=submit_calls) - d = H._AppServerDriver.__new__(H._AppServerDriver) - d._metrics = metrics - - async def proto(): - if exc is not None: - raise exc - return "completed" - - d._drive_protocol = proto - return d, metrics - - -@pytest.mark.parametrize("exc,status,kind", [ - (H.CodexModelFailure("DSV4 must return exactly one tool call"), "max_turns", "protocol_error"), - (H.CodexTurnLimit("Codex reached the signed turn budget"), "max_turns", "turn_limit"), - (H.CodexSequenceLimit("Miles returned a truncated sample"), "max_seq_len", "response_truncated"), - (H.CodexSequenceLimit("Miles sequence limit reached"), "max_seq_len", "context_limit"), - (None, "completed", "completed_no_submit"), -]) -def test_drive_splits_end_kind(exc, status, kind): - d, m = _driver(exc) - assert asyncio.run(d.drive()) == status # signed status unchanged - assert m.end_kind == kind and kind in H.END_KINDS - - -def test_drive_submit(): - d, m = _driver(None, submit_calls=1) - assert asyncio.run(d.drive()) == "completed" and m.end_kind == "submit" - - -def test_end_kind_for_rejects_infra_errors(): - with pytest.raises(TypeError): - H.end_kind_for(H.CodexHarnessError("x")) - - -def test_turn_lengths_recorded_and_capped(): - m = legacy.AgentMetrics() - choice = {"finish_reason": "tool_calls", "message": {"content": "", "reasoning_content": "r"}} - H._note_last_completion(m, choice, {"usage": {"completion_tokens": 120, "total_tokens": 900}}) - H._note_last_completion(m, choice, {}) - H._note_turn_tool_output(m, "héllo") - assert m.turn_completion_tokens == [120, None] - assert m.turn_context_tokens == [900, None] - assert m.turn_tool_output_bytes == [6] - for _ in range(H.TURN_LENGTHS_MAX + 5): - H._note_turn_tool_output(m, "x") - assert len(m.turn_tool_output_bytes) == H.TURN_LENGTHS_MAX - d = F._metrics_dict(m) - assert d["turn_completion_tokens"] == [120, None] and "end_kind" not in d - m.end_kind = "protocol_error" - assert F._metrics_dict(m)["end_kind"] == "protocol_error" - - -def test_event_carries_fields_and_validates(): - meta = {"exit_status": "max_turns", - "agent_metrics": {"turns": 1, "parse_failures": 1, "end_kind": "protocol_error", - "turn_completion_tokens": [4100, True, None] + [1] * 300, - "turn_context_tokens": [5000], "turn_tool_output_bytes": []}} - out = trajectory_diagnostics(meta) - assert out["exit_status"] == "max_turns" and out["end_kind"] == "protocol_error" - assert out["turn_completion_tokens"][:3] == [4100, None, None] - assert len(out["turn_completion_tokens"]) == 256 - record = {"rollout_id": 0, "policy_version": 0, "sample_index": 0, "group_index": 0, "task_id": "t", - "trajectory_id": "x", "reward": 0.0, "success": False, "aborted": False, **out} - assert timeline.validate_trajectory_reward(record) == [] - assert timeline.validate_trajectory_reward({**record, "end_kind": 3}) diff --git a/tests/test_codex_evaluate_time.py b/tests/test_codex_evaluate_time.py deleted file mode 100644 index 7f44eee93..000000000 --- a/tests/test_codex_evaluate_time.py +++ /dev/null @@ -1,52 +0,0 @@ -"""S17 N13: per-trajectory grading time (``evaluate_time``) reaches the event.""" - -from __future__ import annotations - -import asyncio - -from yeto.rl.engine import timeline -from yeto.rl.adapters.miles.rollout_meta_hook import trajectory_diagnostics -from yeto.rl.harness.codex import codex_openenv_agent_function as F - - -class _Verifier: - def __init__(self, delay: float) -> None: - self.delay = delay - self.calls = 0 - - async def evaluate(self, episode_id): - self.calls += 1 - await asyncio.sleep(self.delay) - return {"passed": True, "testsh_rc": 0, "log": "ok"} - - -def _finish(status, verifier): - untrusted = {"status": status, "episode_id": "ep", "metrics": {"turns": 2, "evaluate_time": 0.0}} - return asyncio.run(F.finish_trusted(untrusted, verifier, task_id="t", sample_id="s")) - - -def test_finish_trusted_records_evaluate_time(): - v = _Verifier(0.05) - meta = _finish("completed", v) - et = meta["agent_metrics"]["evaluate_time"] - assert v.calls == 1 and isinstance(et, float) and 0.04 <= et < 5 - assert meta["agent_metrics"]["turns"] == 2 - out = trajectory_diagnostics(meta) - assert out["evaluate_time"] == round(et, 3) - record = {"rollout_id": 0, "policy_version": 0, "sample_index": 0, "group_index": 0, "task_id": "t", - "trajectory_id": "x", "reward": 1.0, "success": True, "aborted": False, **out} - assert timeline.validate_trajectory_reward(record) == [] - assert timeline.validate_trajectory_reward({**record, "evaluate_time": "1"}) - - -def test_timeout_skips_grading_and_drops_worker_value(): - v = _Verifier(0.0) - meta = _finish("timeout", v) - assert v.calls == 0 and "evaluate_time" not in meta["agent_metrics"] - assert "evaluate_time" not in trajectory_diagnostics(meta) - - -def test_hook_ignores_bad_evaluate_time(): - for bad in (True, -1.0, "3", None): - assert "evaluate_time" not in trajectory_diagnostics({"agent_metrics": {"evaluate_time": bad}}) - assert trajectory_diagnostics({"agent_metrics": {"evaluate_time": 2}})["evaluate_time"] == 2.0 diff --git a/tests/test_codex_head_staging.py b/tests/test_codex_head_staging.py deleted file mode 100644 index 89f18f6b6..000000000 --- a/tests/test_codex_head_staging.py +++ /dev/null @@ -1,40 +0,0 @@ -"""S17 M1: a signed Codex run under --controller head stages its bundle and harness env on the head.""" - -from types import SimpleNamespace - -from yeto import launcher - - -def test_non_codex_run_stages_nothing(monkeypatch): - monkeypatch.setattr("yeto.launch.codex.codex_harness_launch", lambda args, environ=None: None) - assert launcher.codex_head_staging(SimpleNamespace(), {"HF_TOKEN": "k"}) == ({}, {}) - - -def test_codex_run_mounts_bundle_and_forwards_harness_env_as_secrets(monkeypatch, tmp_path): - monkeypatch.setattr("yeto.launch.codex.codex_harness_launch", lambda args, environ=None: ("", {}, {})) - env = { - launcher.CODEX_BUNDLE_DIR_ENV: str(tmp_path), - launcher.HARNESS_ENVIRONMENT_PROVIDER_ENV: "yeto.rl.harness.codex.tb2_provider:modal_provider", - "MODAL_TOKEN_ID": "id", - "MODAL_TOKEN_SECRET": "sec", - "YETO_SANDBOX_MODAL_TOKEN_ID": "sid", - "YETO_SANDBOX_MODAL_TOKEN_SECRET": "ssec", - "OPENENV_RUN_ID": "run", - "YETO_HARNESS_TB2_TASKS_DIR": "/opt/yeto/codex/tb2-tasks", - "YETO_HARNESS_TB2_MODAL_APP": "yeto-tbench2", - "UNRELATED": "x", - "TBENCH_REWARD_HMAC_KEY": "retired", - "SECRLENV_MAX_TURNS": "", - } - mounts, secrets = launcher.codex_head_staging(SimpleNamespace(), env) - assert mounts == {launcher.HEAD_CODEX_BUNDLE_PATH: str(tmp_path.resolve())} - assert secrets == {k: env[k] for k in ( - launcher.HARNESS_ENVIRONMENT_PROVIDER_ENV, - "YETO_SANDBOX_MODAL_TOKEN_ID", "YETO_SANDBOX_MODAL_TOKEN_SECRET", "OPENENV_RUN_ID", "YETO_HARNESS_TB2_TASKS_DIR", "YETO_HARNESS_TB2_MODAL_APP")} - - -def test_args_bundle_dir_wins_over_env(monkeypatch, tmp_path): - monkeypatch.setattr("yeto.launch.codex.codex_harness_launch", lambda args, environ=None: ("", {}, {})) - mounts, _ = launcher.codex_head_staging(SimpleNamespace(codex_bundle_dir=str(tmp_path)), - {launcher.CODEX_BUNDLE_DIR_ENV: "/nope"}) - assert list(mounts.values()) == [str(tmp_path.resolve())] diff --git a/tests/test_harness_codex_exec_replay.py b/tests/test_harness_codex_exec_replay.py deleted file mode 100644 index c86d68701..000000000 --- a/tests/test_harness_codex_exec_replay.py +++ /dev/null @@ -1,89 +0,0 @@ -"""G8 / task 10.3 (reduced): replay the black-box Codex 0.145.0 binary on CPU. - -Real stock Codex binary (``YETO_CODEX_BINARY_PATH``) -> in-process -``_ResponsesBridge`` -> fake Miles ``/v1/chat/completions`` endpoint, with the -real TB2 relay (``tb2_provider.LocalProcessBackend``) and trusted verifier. -Skipped when the 0.145.0 binary is not available locally. The request-shape -summary is written to ``/tmp/t3-g8-codex-replay.json``. -""" - -from __future__ import annotations - -import asyncio -import hashlib -import json -from pathlib import Path - -import test_harness_tb2_provider as tb2_tests -import test_secrlenv_codex_harness as legacy_tests -from yeto.rl import tbench_outcome -from yeto.rl.engine.tool_wait import HarnessBoard -from yeto.rl.harness.codex import codex_harness_agent as harness -from yeto.rl.harness.codex import codex_openenv_subprocess_agent_function as subprocess_agent - -SUMMARY = Path("/tmp/t3-g8-codex-replay.json") - - -def _stock_env(monkeypatch, binary: Path) -> None: - for name, value in harness._IDENTITY_ENV.items(): - monkeypatch.setenv(name, value) - monkeypatch.setenv("YETO_CODEX_BINARY_PATH", str(binary)) - monkeypatch.setenv("YETO_CODEX_BINARY_SHA256", hashlib.sha256(binary.read_bytes()).hexdigest()) - monkeypatch.setenv("YETO_CODEX_BINARY_SIZE_BYTES", str(binary.stat().st_size)) - monkeypatch.setenv("YETO_CODEX_VERSION", harness.CODEX_CLI_VERSION) - monkeypatch.setenv("YETO_CODEX_BACKEND_MAX_TOKENS", "4096") - monkeypatch.delenv("YETO_CODEX_COMPACTION_ENABLED", raising=False) - - -def test_stock_codex_0145_single_chain_replay_through_tb2_relay(monkeypatch, tmp_path): - binary = legacy_tests._stock_codex_binary() - _stock_env(monkeypatch, binary) - hb = HarnessBoard() - provider = tb2_tests._provider(tmp_path) - tb2_tests._configure(monkeypatch, provider, harness_board=hb) - monkeypatch.delenv("YETO_CODEX_OPENENV_ALLOW_SCRIPTED_DRIVER", raising=False) - - async def scenario(): - runner, miles_url, requests = await legacy_tests._fake_miles( - [ - legacy_tests._completion("terminal.exec", '{"command":"echo ok > fixed.txt && ls"}', "call-1", "fix it", 2_000), - legacy_tests._completion("terminal.exec", '{"command":"cat fixed.txt"}', "call-2", "check", 2_500), - legacy_tests._completion("submit", '{"evidence":"fixed.txt written"}', "call-3", "done", 3_000), - ] - ) - try: - metadata = { - "task_id": "fix-git", "trajectory_id": "traj-g8", "prompt": "write ok into fixed.txt", - "expected_policy_version": "pv-g8", - } - result = await asyncio.wait_for(subprocess_agent.run(miles_url, "p", {}, metadata), timeout=120) - finally: - await runner.cleanup() - return result, requests - - result, requests = asyncio.run(scenario()) - outcome, reward = tbench_outcome.verified_outcome(result) - assert outcome["status"] == "completed", result - assert reward == 1.0 - assert result["chains_total"] == 1 and result["chain_break_reason"] is None - assert provider.destroyed == 1 and provider.live == {} - snap = hb.snapshot() - assert snap.env_live == 0 and snap.in_flight == 0 - assert snap.tito_session_mismatch == 0 - breaks = snap.tito_chain_breaks - assert not any(breaks.values()) if isinstance(breaks, dict) else breaks == 0 - # black-box Codex request shape: exactly 3 samples, one chain, tool set fixed, history grows by tool turns - assert len(requests) == 3 - assert [t["function"]["name"] for t in requests[0]["tools"]] == ["terminal.exec", "submit"] - roles = [[m["role"] for m in r["messages"]] for r in requests] - assert roles[1][-1] == "tool" and roles[2][-1] == "tool" - assert all(len(roles[i]) < len(roles[i + 1]) for i in range(2)) - SUMMARY.write_text(json.dumps({ - "codex_version": harness.CODEX_CLI_VERSION, - "binary_sha256": hashlib.sha256(binary.read_bytes()).hexdigest(), - "requests": len(requests), - "tools": [t["function"]["name"] for t in requests[0]["tools"]], - "message_roles_per_request": roles, - "status": outcome["status"], "reward": reward, "chains_total": result["chains_total"], - "tito_session_mismatch": snap.tito_session_mismatch, - }, indent=2) + "\n") diff --git a/tests/test_harness_codex_openenv.py b/tests/test_harness_codex_openenv.py deleted file mode 100644 index 9d2104efc..000000000 --- a/tests/test_harness_codex_openenv.py +++ /dev/null @@ -1,935 +0,0 @@ -"""Ports-path acceptance for the Codex Terminal-Bench harness (change rl-codex-harness-rollout). - -Covers the six contracts listed in CODEX-PROGRESS §1.3 on top of the moved -legacy suite (tests/test_secrlenv_codex_harness.py): -1. Responses tool call -> TITO append-only prefix reuse on the Miles wire; -2. rollout-level cancellation cleanup (worker process group, lease, tool-wait); -3. reward verification at all three points; -4. mask / token / logprob alignment; -5. sibling segments share one reward and count once; -6. compaction enabled -> preflight fails closed. - -Stage 4 (IR-1..IR-4 landed, INFRA 8347351): the agent entry is driven through -the real ``tool_wait.HarnessBoard`` (leases / sessions / admission), -``rollout_meta_hook.expected_policy_version`` (IR-3), ``entry.preflight_stage`` -with the ``(miles_args, launch)`` hook (IR-1) and the 1.7 schema names (IR-4). -""" - -from __future__ import annotations - -import asyncio -import copy -import json -from pathlib import Path -from types import SimpleNamespace -from typing import Any - -import aiohttp -import pytest - -import test_secrlenv_codex_harness as legacy_tests -from yeto.rl import tbench_outcome -from yeto.rl.engine.algorithm import AlgorithmSpec -from yeto.rl.adapters.miles import entry, rollout_meta_hook -from yeto.rl.adapters.miles.config import MilesConfigError -from yeto.rl.engine.timeline import HARNESS_METRIC_KEYS, LOAD_SAMPLE_SCHEMA -from yeto.rl.engine.tool_wait import HARNESS_ZERO, HarnessBoard, HarnessSnapshot, ToolWaitBoard, drain_blockers -from yeto.rl.harness.codex import ( - alignment, - codex_harness_agent as harness, - codex_openenv_agent_function as adapter, - codex_openenv_generate as generate_wrapper, - codex_openenv_subprocess_agent_function as subprocess_agent, - preflight, - tbench_reward, -) -from yeto.rl.harness.codex.environment import FakeTerminalEnvironment, serve_environment - - - -class _Board: - def __init__(self) -> None: - self.events: list[tuple[str, str]] = [] - - def enter(self, trajectory_id: str) -> None: - self.events.append(("enter", trajectory_id)) - - def exit(self, trajectory_id: str) -> None: - self.events.append(("exit", trajectory_id)) - - @property - def open(self) -> int: - return sum(1 for e, _ in self.events if e == "enter") - sum(1 for e, _ in self.events if e == "exit") - - -class _Provider: - def __init__(self, env: FakeTerminalEnvironment, *, deadline: float = 30.0, fail: bool = False) -> None: - self.env = env - self.deadline = deadline - self.fail = fail - self.destroyed = 0 - self.runner = None - - async def acquire(self, task_id: str, trajectory_id: str) -> subprocess_agent.EnvironmentLease: - if self.fail: - raise OSError("sandbox quota exhausted") - self.runner, url = await serve_environment(self.env, "tok") - - async def destroy() -> None: - self.destroyed += 1 - await self.runner.cleanup() - - async def describe() -> str: - return "gone" if self.destroyed else "live" - - return subprocess_agent.EnvironmentLease( - env_url=url, env_token="tok", verifier=self.env, destroy=destroy, describe=describe, - deadline_seconds=self.deadline, - ) - - -def _run(coro): - return asyncio.run(coro) - - -# ---------------------------------------------------------------- identity / preflight (6) - -def test_identity_matches_stock_harness_and_profile(): - identity = adapter.codex_openenv_harness_identity() - assert set(identity) == { - "base_instructions_sha256", "terminal_exec_tool_schema_sha256", - "submit_tool_schema_sha256", "dynamic_tools_schema_sha256", - } - assert adapter._OPENENV_IDENTITY_ENV["YETO_CODEX_OPENENV_BACKEND_PROFILE"] == "qwen35_08b" - assert adapter.stock.BACKEND_MODEL == "qwen35" - assert callable(adapter.stock._attest_runtime) - - -def _good_env() -> dict[str, str]: - return dict(adapter._OPENENV_IDENTITY_ENV) - - -def test_preflight_fails_closed_on_compaction_scripted_driver_identity_and_version(tmp_path): - ok = lambda: tmp_path / "codex" # noqa: E731 - report = preflight.preflight_codex_openenv(_good_env(), attest_runtime=ok) - assert report["compaction"] == "disabled" and report["codex_version"] == "codex-cli 0.145.0" - for env, match in ( - ({**_good_env(), "YETO_CODEX_COMPACTION_ENABLED": "1"}, "incompatible"), - ({**_good_env(), "YETO_CODEX_MAX_COMPACTIONS": "3"}, "compaction tuning"), - ({**_good_env(), "YETO_CODEX_OPENENV_ALLOW_SCRIPTED_DRIVER": "1"}, "must not be set"), - ({k: v for k, v in _good_env().items() if "MODEL_REVISION" not in k}, "identity drifted"), - ): - with pytest.raises(preflight.PreflightError, match=match): - preflight.preflight_codex_openenv(env, attest_runtime=ok) - - def bad_version() -> Path: - raise harness.CodexHarnessError("pinned Codex version contract drifted") - - with pytest.raises(preflight.PreflightError, match="version contract drifted"): - preflight.preflight_codex_openenv(_good_env(), attest_runtime=bad_version) - - -# ---------------------------------------------------------------- Responses -> TITO prefix (1) - -def test_responses_tool_call_round_trip_is_append_only_on_the_miles_wire(monkeypatch): - legacy_tests._set_bridge_env(monkeypatch) - - async def scenario() -> None: - runner, miles_url, miles_requests = await legacy_tests._fake_miles([ - legacy_tests._completion("terminal.exec", '{ "command": "git status" }', "call-1", "look", 25), - legacy_tests._completion("submit", '{"evidence":"done"}', "call-2", "finish", 40), - ]) - metrics = harness.legacy.AgentMetrics() - try: - async with harness._ResponsesBridge( - miles_url, "fix the repo", {"temperature": 0.7, "top_p": 0.9, "max_tokens": 128, "stream": True, "n": 1}, - metrics, max_seq_len=512, - ) as bridge: - headers = {"Authorization": f"Bearer {bridge.token}"} - initial = [{"type": "message", "role": "user", "content": [{"type": "input_text", "text": "fix the repo"}]}] - async with aiohttp.ClientSession() as session: - async with session.post(f"{bridge.url}/v1/responses", json=legacy_tests._codex_body(initial), headers=headers) as r: - assert r.status == 200 - legacy_tests._sse_events(await r.text()) - bridge.expect_tool_output("call-1", '{"exit_code":0,"output":"clean","timed_out":false,"truncated":false}') - second = copy.deepcopy(bridge._expected_input) - async with session.post(f"{bridge.url}/v1/responses", json=legacy_tests._codex_body(second), headers=headers) as r: - assert r.status == 200 - # Append-only: request 2's messages == request 1's + [assistant, tool] - m1, m2 = miles_requests[0]["messages"], miles_requests[1]["messages"] - assert m2[: len(m1)] == m1 - appended = m2[len(m1):] - assert [m["role"] for m in appended] == ["assistant", "tool"] - assert appended[0]["tool_calls"][0]["id"] == "call-1" - assert appended[1]["tool_call_id"] == "call-1" - # Mutated history is refused (no third Miles sample is taken) - mutated = copy.deepcopy(second) - mutated[0]["content"][0]["text"] = "fix the repo!" - async with session.post(f"{bridge.url}/v1/responses", json=legacy_tests._codex_body(mutated), headers=headers) as r: - assert r.status == 400 - assert len(miles_requests) == 2 - finally: - await runner.cleanup() - - _run(scenario()) - - -# ---------------------------------------------------------------- subprocess entry: outcome, tool-wait, cleanup (2) - -def _configure(monkeypatch, env: FakeTerminalEnvironment, harness_board: Any = None, **kw) -> tuple[_Provider, _Board]: - monkeypatch.setenv("YETO_CODEX_OPENENV_ALLOW_SCRIPTED_DRIVER", "1") - provider = _Provider(env, **kw) - board = _Board() - subprocess_agent.configure(provider=provider, tool_wait_board=board, harness_board=harness_board, member="m0") - return provider, board - - -def _metadata(**extra) -> dict[str, Any]: - # IR-3: the driver's target token travels in the prompt metadata (or the sink). - return {"task_id": "fix-git", "trajectory_id": "traj-1", "prompt": "fix the repo", - "codex_openenv_driver": "scripted", "script": ["git status", "git checkout -- ."], - "expected_policy_version": "pv-7", **extra} - - -def test_subprocess_run_relays_tool_wait_and_builds_outcome(monkeypatch): - env = FakeTerminalEnvironment(passed=True) - provider, board = _configure(monkeypatch, env) - result = _run(subprocess_agent.run("http://miles", "p", {}, _metadata(expected_policy_version="pv-7"))) - assert result is not None and tbench_outcome.OUTCOME_KEY in result - outcome, reward = tbench_outcome.verified_outcome(result) - assert reward == 1.0 and outcome["status"] == "completed" and outcome["task_id"] == "fix-git" - assert env.evaluations == 1 and [c for _, c in env.calls] == ["git status", "git checkout -- ."] - assert board.events == [("enter", "traj-1"), ("exit", "traj-1")] * 3 and board.open == 0 - assert provider.destroyed == 1 - assert result["chains_total"] == 1 and result["reward_scope"] == "trajectory" - assert result["expected_policy_version"] == "pv-7" - - -def test_subprocess_negative_reward_and_timeout_are_policy_boundaries(monkeypatch): - env = FakeTerminalEnvironment(passed=False) - _configure(monkeypatch, env) - result = _run(subprocess_agent.run("http://miles", "p", {}, _metadata())) - outcome, reward = tbench_outcome.verified_outcome(result) - assert reward == 0.0 and outcome["status"] == "completed" - - env = FakeTerminalEnvironment(passed=True) - provider, board = _configure(monkeypatch, env, deadline=0.5) - result = _run(subprocess_agent.run("http://miles", "p", {}, _metadata(hang_seconds=20))) - outcome, reward = tbench_outcome.verified_outcome(result) - assert reward == 0.0 and outcome["status"] == "timeout" and outcome["verifier"] == tbench_outcome.TIMEOUT_VERIFIER - assert env.evaluations == 0 and provider.destroyed == 1 and board.open == 0 - - -def test_subprocess_infrastructure_failures_have_no_outcome_and_abort(monkeypatch): - env = FakeTerminalEnvironment(passed=True) - _configure(monkeypatch, env, fail=True) - result = _run(subprocess_agent.run("http://miles", "p", {}, _metadata())) - assert tbench_reward.INFRASTRUCTURE_KEY in result and tbench_outcome.OUTCOME_KEY not in result - sample = SimpleNamespace(metadata=result, status=None) - assert _run(tbench_reward.reward_func(None, sample)) == 0.0 and sample.status == "ABORTED" - - -def test_rollout_cancellation_kills_worker_and_releases_everything(monkeypatch): - env = FakeTerminalEnvironment(passed=True) - provider, board = _configure(monkeypatch, env) - terminated: list[int | None] = [] - original = subprocess_agent._terminate - - async def recording(process): - await original(process) - terminated.append(process.returncode) - - monkeypatch.setattr(subprocess_agent, "_terminate", recording) - - async def scenario() -> None: - task = asyncio.create_task(subprocess_agent.run("http://miles", "p", {}, _metadata(hang_seconds=30))) - while not board.events: - await asyncio.sleep(0.05) - task.cancel() - with pytest.raises(asyncio.CancelledError): - await task - - _run(scenario()) - assert terminated and terminated[0] is not None # worker reaped - assert board.open == 0 and provider.destroyed == 1 and env.evaluations == 0 - - -def test_abort_hook_cancels_surplus_workers_and_releases_everything(monkeypatch): - """S17 N17: Miles' call_agent_abort_hook finds ``.abort``; it cancels the - in-flight Codex workers so an over-sampled rollout does not wait for surplus - trajectories. The aborted trajectory is infrastructure (never a reward) and - its worker, lease and board entries are released.""" - env = FakeTerminalEnvironment(passed=True) - provider, board = _configure(monkeypatch, env) - - async def scenario(): - task = asyncio.create_task(subprocess_agent.run("http://miles", "p", {}, _metadata(hang_seconds=30))) - while not board.events: - await asyncio.sleep(0.05) - assert await subprocess_agent.abort(None) == 1 - return await asyncio.wait_for(task, 20) - - result = _run(scenario()) - assert tbench_reward.INFRASTRUCTURE_KEY in result and tbench_outcome.OUTCOME_KEY not in result - assert subprocess_agent.ABORTED_STATUS in json.dumps(result) - assert board.open == 0 and provider.destroyed == 1 and env.evaluations == 0 - assert subprocess_agent._INFLIGHT == {} and subprocess_agent._ABORTED_IDS == set() - assert _run(subprocess_agent.abort(None)) == 0 # nothing in flight: no-op - - -def test_abort_during_acquire_waits_for_the_sandbox_then_destroys_it(monkeypatch): - """S17 N17: acquire is never cancelled (a sandbox made by a cancelled acquire would - leak); the trajectory is flagged and stops as soon as the sandbox exists.""" - env = FakeTerminalEnvironment(passed=True) - provider, board = _configure(monkeypatch, env) - gate = asyncio.Event() - real_acquire = provider.acquire - - async def slow_acquire(task_id, trajectory_id): - await gate.wait() - return await real_acquire(task_id, trajectory_id) - - provider.acquire = slow_acquire - - async def scenario(): - task = asyncio.create_task(subprocess_agent.run("http://miles", "p", {}, _metadata(hang_seconds=30))) - while "traj-1" not in subprocess_agent._INFLIGHT: - await asyncio.sleep(0.01) - assert subprocess_agent._INFLIGHT["traj-1"]["stage"] == "acquire" - assert await subprocess_agent.abort(None) == 1 - gate.set() - return await asyncio.wait_for(task, 20) - - result = _run(scenario()) - assert subprocess_agent.ABORTED_STATUS in json.dumps(result) and tbench_outcome.OUTCOME_KEY not in result - assert provider.destroyed == 1 and env.calls == [] and env.evaluations == 0 - assert subprocess_agent._INFLIGHT == {} and subprocess_agent._ABORTED_IDS == set() - - -def test_abort_during_verification_cancels_it_and_destroys_the_sandbox(monkeypatch): - env = FakeTerminalEnvironment(passed=True) - provider, board = _configure(monkeypatch, env) - started = asyncio.Event() - - async def slow_finish(untrusted, verifier, **kw): - started.set() - await asyncio.sleep(60) # a judge that hangs (detached sandbox) - - monkeypatch.setattr(subprocess_agent.adapter, "finish_trusted", slow_finish) - - async def scenario(): - task = asyncio.create_task(subprocess_agent.run("http://miles", "p", {}, _metadata())) - await asyncio.wait_for(started.wait(), 20) - assert subprocess_agent._INFLIGHT["traj-1"]["stage"] == "verify" - assert await subprocess_agent.abort(None) == 1 - return await asyncio.wait_for(task, 20) - - result = _run(scenario()) - assert subprocess_agent.ABORTED_STATUS in json.dumps(result) and tbench_outcome.OUTCOME_KEY not in result - assert provider.destroyed == 1 and board.open == 0 - assert subprocess_agent._INFLIGHT == {} and subprocess_agent._ABORTED_IDS == set() - - -def test_lease_force_released_by_the_board_is_infrastructure_not_a_crash(monkeypatch): - """S17 N17 A/B arm a: the board force-released an expired lease; the late - lease_released raised ToolWaitError("... is not live") out of run() and the - reward function then failed the whole rollout. Now: infrastructure sample.""" - from yeto.rl.engine.tool_wait import HarnessBoard - - hb = HarnessBoard() - env = FakeTerminalEnvironment(passed=True) - provider, _board = _configure(monkeypatch, env, harness_board=hb, deadline=30.0) - real_destroy_calls = [] - - orig_acquire = provider.acquire - - async def acquire(task_id, trajectory_id): - lease = await orig_acquire(task_id, trajectory_id) - orig_destroy = lease.destroy - - async def destroy(): - hb._leases.pop(trajectory_id, None) # the board expired it meanwhile - real_destroy_calls.append(trajectory_id) - await orig_destroy() - - return subprocess_agent.EnvironmentLease( - env_url=lease.env_url, env_token=lease.env_token, verifier=lease.verifier, - destroy=destroy, describe=lease.describe, deadline_seconds=lease.deadline_seconds) - - provider.acquire = acquire - result = _run(subprocess_agent.run("http://miles", "p", {}, _metadata())) - assert tbench_reward.INFRASTRUCTURE_KEY in result and tbench_outcome.OUTCOME_KEY not in result - assert subprocess_agent.LEASE_EXPIRED_ON_BOARD in json.dumps(result) - assert real_destroy_calls == ["traj-1"] and provider.destroyed == 1 - assert hb.snapshot().in_flight == 0 # the session was still closed - sample = SimpleNamespace(metadata=result, status=None) - assert _run(tbench_reward.reward_func(None, sample)) == 0.0 and sample.status == "ABORTED" - - -# ---------------------------------------------------------------- IR-2/IR-3: HarnessBoard, admission, policy token source - -def test_subprocess_run_counts_sessions_and_leases_on_the_harness_board_and_drains_to_zero(monkeypatch): - import time - - hb = HarnessBoard() # monotonic clock, as the island actor - env = FakeTerminalEnvironment(passed=True) - provider, _board = _configure(monkeypatch, env, harness_board=hb, deadline=30.0) - seen: list[HarnessSnapshot] = [] - - async def scenario() -> dict[str, Any]: - task = asyncio.create_task(subprocess_agent.run("http://miles", "p", {}, _metadata(hang_seconds=0.5))) - while not (hb.snapshot().env_live and hb.snapshot().in_flight): - await asyncio.sleep(0.02) - seen.append((hb.snapshot(), time.monotonic())) - return await task - - result = _run(scenario()) - busy, at = seen[0] - # the lease carries its hard deadline (acquire time + provider deadline): drain waits are bounded - assert busy.in_flight == 1 and busy.env_live == 1 and busy.latest_lease_deadline == pytest.approx(at + 30.0, abs=5.0) - # drain probe (IR-2, three-arg): a live sandbox / open session blocks even with router=0, tool_wait=0 - idle_tools = ToolWaitBoard().snapshot() - assert drain_blockers(0, idle_tools, busy) == ["1 harness sessions in flight", "1 sandboxes live"] - assert drain_blockers(0, idle_tools, None) == ["harness counts unknown"] - assert drain_blockers(0, idle_tools, HARNESS_ZERO) == [] - done = hb.snapshot() - assert done.in_flight == 0 and done.env_live == 0 and done.leases_expired_total == 0 - assert drain_blockers(0, idle_tools, done) == [] and provider.destroyed == 1 - assert tbench_outcome.OUTCOME_KEY in result and result["expected_policy_version"] == "pv-7" - - -def test_subprocess_run_refuses_new_session_when_admission_is_closed(monkeypatch): - hb = HarnessBoard() - env = FakeTerminalEnvironment(passed=True) - provider, _board = _configure(monkeypatch, env, harness_board=hb) - hb.close_admission(["m0"]) # drain of our member, first step of MilesRolloutPool.drain - assert hb.allow_new_session("m0") is False - result = _run(subprocess_agent.run("http://miles", "p", {}, _metadata())) - assert result[tbench_reward.INFRASTRUCTURE_KEY] == subprocess_agent.ADMISSION_CLOSED - assert tbench_outcome.OUTCOME_KEY not in result and provider.runner is None # nothing acquired - assert hb.snapshot() == HarnessSnapshot(0, 0, hb.snapshot().generation) - hb.open_admission(["m0"]) - result = _run(subprocess_agent.run("http://miles", "p", {}, _metadata())) - assert tbench_outcome.OUTCOME_KEY in result and hb.snapshot().in_flight == 0 and hb.snapshot().env_live == 0 - - -def test_subprocess_cancellation_releases_board_lease_and_session(monkeypatch): - hb = HarnessBoard() - env = FakeTerminalEnvironment(passed=True) - provider, board = _configure(monkeypatch, env, harness_board=hb) - - async def scenario() -> None: - task = asyncio.create_task(subprocess_agent.run("http://miles", "p", {}, _metadata(hang_seconds=30))) - while not board.events: - await asyncio.sleep(0.05) - assert hb.snapshot().in_flight == 1 and hb.snapshot().env_live == 1 - task.cancel() - with pytest.raises(asyncio.CancelledError): - await task - - _run(scenario()) - snap = hb.snapshot() - assert snap.in_flight == 0 and snap.env_live == 0 and provider.destroyed == 1 and board.open == 0 - - -def test_subprocess_policy_token_comes_from_metadata_or_driver_sink_and_missing_refuses(monkeypatch, tmp_path): - env = FakeTerminalEnvironment(passed=True) - _configure(monkeypatch, env) - monkeypatch.setenv(rollout_meta_hook.META_SINK_ENV, f"dir:{tmp_path}") - md = _metadata() - del md["expected_policy_version"] - # nothing published by the driver -> refuse before any environment is acquired - with pytest.raises(subprocess_agent.PolicyVersionMissing, match="expected_policy_version missing"): - _run(subprocess_agent.run("http://miles", "p", {}, dict(md))) - assert env.evaluations == 0 - rollout_meta_hook.put_policy_token("yeto:3:abc") # what MilesRolloutPool.generate publishes - result = _run(subprocess_agent.run("http://miles", "p", {}, dict(md))) - assert result["expected_policy_version"] == "yeto:3:abc" - result = _run(subprocess_agent.run("http://miles", "p", {}, _metadata())) # metadata wins over the sink - assert result["expected_policy_version"] == "pv-7" - - -def test_generate_wrapper_refuses_without_policy_token_and_reads_sink(monkeypatch, tmp_path): - monkeypatch.setenv(rollout_meta_hook.META_SINK_ENV, f"dir:{tmp_path}") - calls = [] - - async def upstream(_input): - calls.append(1) - return SimpleNamespace(samples=[_sample([1, 2, 3], [0, 1, 1], [-0.1, -0.2], 2, [("yeto:3:abc", 1, 3)])]) - - inp = SimpleNamespace(sample=SimpleNamespace(group_index=1, index=2, metadata={})) - with pytest.raises(generate_wrapper.PolicyVersionMissing): - _run(generate_wrapper.generate(inp, upstream=upstream)) - assert calls == [] - rollout_meta_hook.put_policy_token("yeto:3:abc") - out = _run(generate_wrapper.generate(inp, upstream=upstream)) - s = out.samples[0] - assert s.status is None and s.metadata["expected_policy_version"] == "yeto:3:abc" - assert rollout_meta_hook.harness_counters([[s]]) == {} - rollout_meta_hook.put_policy_token("yeto:4:def") - out = _run(generate_wrapper.generate(inp, upstream=upstream)) - s = out.samples[0] - assert s.status == "ABORTED" and rollout_meta_hook.harness_counters([[s]]) == {"policy_age_violation": 1} - - -# ---------------------------------------------------------------- IR-1: preflight hook before any allocation; 7.1 reward_scope - -def _island_args(**over) -> SimpleNamespace: - base = dict( - yeto_harness_preflight=preflight.HARNESS_PREFLIGHT_SPEC, - custom_agent_function_path=preflight.EXPECTED_AGENT_FUNCTION, - yeto_harness_reward_scope=None, - yeto_harness_environment_provider="test_harness_codex_openenv:_provider_factory", - yeto_rl_learner_id=0, - ) - return SimpleNamespace(**{**base, **over}) - - -def _provider_factory(miles_args): - return _Provider(FakeTerminalEnvironment(passed=True)) - - -def _stage(monkeypatch): - """Stub the A1 contract pieces so entry.preflight_stage runs without Miles (as test_rl_ir_harness).""" - monkeypatch.setattr(entry, "ports_runtime_fingerprint", lambda launch: "fp") - monkeypatch.setattr(entry, "miles_capabilities", lambda fp, unverified_mechanisms=(): "caps") - monkeypatch.setattr(entry, "with_partitioned_serial", lambda caps: caps) - monkeypatch.setattr(entry, "execution_profile_for", lambda *a, **k: "profile") - monkeypatch.setattr(entry, "expected_algorithm_sha256", lambda a: None) - monkeypatch.setattr(entry, "preflight", lambda profile, algorithm, caps: None) - monkeypatch.setattr(entry, "elastic_wiring_for", lambda a, profile, fingerprint: None) - allocations = [] - monkeypatch.setattr(entry, "connect_island_ray", lambda *a, **k: allocations.append("ray")) - monkeypatch.setattr(entry, "resolve_harness_preflight", - lambda a, environ=None: preflight.harness_preflight if a.yeto_harness_preflight else None) - return allocations - - -def test_entry_preflight_stage_runs_codex_preflight_before_allocation_and_installs_boards(monkeypatch, tmp_path): - allocations = _stage(monkeypatch) - for name, value in _good_env().items(): - monkeypatch.setenv(name, value) - monkeypatch.delenv("YETO_CODEX_OPENENV_ALLOW_SCRIPTED_DRIVER", raising=False) - attested = [] - monkeypatch.setattr(preflight, "preflight_codex_openenv", - lambda env=None, **kw: attested.append(env.get("YETO_CODEX_OPENENV_MODEL_REVISION")) or {}) - subprocess_agent.configure(provider=None) - # the real spec resolves through entry.resolve_harness_preflight (module:callable) - assert entry.resolve_harness_preflight(SimpleNamespace(yeto_harness_preflight=preflight.HARNESS_PREFLIGHT_SPEC)) is preflight.harness_preflight - - # failure (compaction on) -> no Ray connect / placement / allocation, no provider installed - monkeypatch.setenv("YETO_CODEX_COMPACTION_ENABLED", "1") - monkeypatch.setattr(preflight, "preflight_codex_openenv", lambda env=None, **kw: preflight.assert_compaction_disabled(env)) - with pytest.raises(preflight.PreflightError, match="incompatible"): - entry.preflight_stage(_island_args(), "launch", AlgorithmSpec(), yeto_policy_sync=False) - assert allocations == [] and subprocess_agent.configured()["provider"] is None - monkeypatch.delenv("YETO_CODEX_COMPACTION_ENABLED") - - # 7.1: reward_scope=segment fails at startup through the same hook (and in validate_parsed_args) - with pytest.raises(MilesConfigError, match="reward_scope='segment'"): - entry.preflight_stage(_island_args(yeto_harness_reward_scope="segment"), "launch", AlgorithmSpec(), yeto_policy_sync=False) - assert allocations == [] and subprocess_agent.configured()["provider"] is None - - # wrong agent function / no provider -> fail closed - with pytest.raises(preflight.PreflightError, match="expects"): - entry.preflight_stage(_island_args(custom_agent_function_path="x.run"), "launch", AlgorithmSpec(), yeto_policy_sync=False) - with pytest.raises(preflight.PreflightError, match="no environment provider"): - preflight.harness_preflight(_island_args(yeto_harness_environment_provider=None), "launch", env=_good_env()) - - # success: provider + island boards installed (lazy actors named like entry.harness_source uses) - monkeypatch.setattr(preflight, "preflight_codex_openenv", lambda env=None, **kw: attested.append("ok") or {}) - assert entry.preflight_stage(_island_args(yeto_rl_learner_id=3), "launch", AlgorithmSpec(), yeto_policy_sync=False) == ("fp", "caps", "profile", None) - installed = subprocess_agent.configured() - assert isinstance(installed["provider"], _Provider) and attested[-1] == "ok" - assert installed["tool_wait_board"].learner_id == 3 and installed["harness_board"].learner_id == 3 - from yeto.rl.engine.tool_wait import harness_board_actor - - assert installed["harness_board"]._factory is harness_board_actor and installed["tool_wait_board"]._factory is None - assert allocations == [] # preflight_stage never connects; run_ports_island does after it returns - subprocess_agent.configure(provider=None) - - -def test_ir4_schema_names_match_the_harness_payload_keys(): - # IR-4 as implemented: counters are registered without a *_total suffix (kind=counter carries the semantics). - assert set(HARNESS_METRIC_KEYS) == {"harness_in_flight", "env_live", "tito_session_mismatch", "tito_chain_breaks", "policy_age_violation"} - assert all(not k.endswith("_total") for k in LOAD_SAMPLE_SCHEMA) - assert LOAD_SAMPLE_SCHEMA["policy_age_violation"][0] == "counter" and LOAD_SAMPLE_SCHEMA["env_live"][0] == "gauge" - assert generate_wrapper.POLICY_AGE_KEY == "policy_age_violation" == rollout_meta_hook.POLICY_AGE_VIOLATION_KEY - - -# ---------------------------------------------------------------- reward validation at three points (3) - -def _outcome_metadata(reward: float = 1.0, **over) -> dict[str, Any]: - md = tbench_outcome.build_outcome_metadata( - task_id="fix-git", sample_id="s1", episode_id="ep1", status="completed", reward=reward, - verifier=tbench_outcome.TEST_SH_VERIFIER, testsh_rc=0 if reward else 1, - ) - md.update(over) - return md - - -def test_reward_is_validated_at_reward_function_parent_and_evidence(): - good = SimpleNamespace(metadata=_outcome_metadata(), status=None, rollout_id=1) - assert _run(tbench_reward.reward_func(None, [good])) == [1.0] - tampered = copy.deepcopy(good.metadata) - tampered[tbench_outcome.OUTCOME_KEY]["reward"] = 0.0 # pass bit now disagrees - for bad in (tampered, {**_outcome_metadata(), "extra": 1} | {tbench_outcome.OUTCOME_KEY: {**_outcome_metadata()[tbench_outcome.OUTCOME_KEY], "x": 1}}): - with pytest.raises(tbench_outcome.UntrustedTBenchOutcome): - _run(tbench_reward.reward_func(None, SimpleNamespace(metadata=bad, status=None))) - with pytest.raises(tbench_outcome.UntrustedTBenchOutcome): - tbench_outcome.verified_outcome(bad) # parent-side verification point - from yeto.rl import trajectory_evidence - - # evidence point: a legacy MAC field is ignored, structure is still checked - legacy = {**good.metadata, "tbench_trusted_outcome_hmac": "0" * 64} - assert trajectory_evidence._verified_outcome(legacy, "terminal-bench-2.1").reward == 1.0 - with pytest.raises(tbench_outcome.UntrustedTBenchOutcome): - trajectory_evidence._verified_outcome(tampered, "terminal-bench-2.1") - - -# ---------------------------------------------------------------- siblings share one reward (5) - -def test_sibling_segments_share_reward_and_count_once(monkeypatch): - a = SimpleNamespace(metadata=_outcome_metadata(1.0, trajectory_id="t1"), status=None, rollout_id=3) - b = SimpleNamespace(metadata=_outcome_metadata(1.0, trajectory_id="t1"), status=None, rollout_id=3) - c = SimpleNamespace(metadata=_outcome_metadata(0.0, trajectory_id="t2"), status=None, rollout_id=4) - rewards = tbench_reward.check_siblings([a, b, c]) - assert rewards == {("rollout", 3): 1.0, ("rollout", 4): 0.0} # two trajectories, not three samples - assert tbench_reward.check_group(None, [[a, b], c]).keep is True - assert tbench_reward.check_group(None, [[a, b]]).keep is False # zero std over trajectories - b_bad = SimpleNamespace(metadata=_outcome_metadata(0.0, trajectory_id="t1"), status=None, rollout_id=3) - with pytest.raises(tbench_outcome.UntrustedTBenchOutcome, match="different outcomes"): - tbench_reward.check_group(None, [[a, b_bad], c]) - infra = SimpleNamespace(metadata={tbench_reward.INFRASTRUCTURE_KEY: "x"}, status=None) - out = tbench_reward.check_group(None, [a, infra]) - assert out.keep is False and out.reason == "group_has_aborted" and infra.status == "ABORTED" - - -def _sample(tokens, mask, logprobs, response_length, spans=None, **extra): - wv = [SimpleNamespace(spans=[SimpleNamespace(version=v, abs_start=s, abs_end=e) for v, s, e in spans])] if spans else [] - return SimpleNamespace(tokens=tokens, loss_mask=mask, rollout_log_probs=logprobs, response_length=response_length, - weight_versions=wv, metadata={}, status=None, **extra) - - -def test_generate_wrapper_sets_sibling_keys_checks_policy_version_and_alignment(): - inp = SimpleNamespace(sample=SimpleNamespace(group_index=9, index=42, metadata={"expected_policy_version": "pv"})) - s1 = _sample([1, 2, 3, 4], [0, 0, 1, 1], [-0.1, -0.2], 2, [("pv", 2, 4)]) - s2 = _sample([1, 5, 6], [0, 1, 1], [-0.3, -0.4], 2, [("pv", 1, 3)]) - - async def upstream(_input): - return SimpleNamespace(samples=[s1, s2]) - - out = _run(generate_wrapper.generate(inp, upstream=upstream)) - assert [s.group_index for s in out.samples] == [9, 9] and [s.rollout_id for s in out.samples] == [42, 42] - assert [s.metadata["chain_index"] for s in out.samples] == [0, 1] and s1.metadata["chains_total"] == 2 - assert s1.status is None and s1.metadata["policy_versions_actual"] == ["pv"] - - stale = _sample([1, 2, 3], [0, 1, 1], [-0.1, -0.2], 2, [("pv-old", 1, 3)]) - _run(generate_wrapper.generate(inp, upstream=lambda _i: _coro(SimpleNamespace(samples=[stale])))) - assert stale.status == "ABORTED" and stale.metadata["policy_age_violation"] == 1 - assert tbench_outcome.OUTCOME_KEY not in stale.metadata and "policy_age" in stale.metadata[tbench_reward.INFRASTRUCTURE_KEY] - - -async def _coro(value): - return value - - -# ---------------------------------------------------------------- mask / token / logprob alignment (4) - -def test_alignment_contract_fails_closed(): - good = _sample([7, 8, 9, 10, 11], [0, 0, 1, 0, 1], [-0.5, 0.0, -0.1], 3, [("v", 2, 3), ("v", 4, 5)]) - alignment.assert_sample_alignment(good) - cases = { - "neither": _sample([7, 8, 9], [1, 1], [-0.1, -0.1, -0.1], 3), - "rollout_log_probs": _sample([7, 8, 9], [0, 1, 1], None, 2), - "prompt position": _sample([7, 8, 9], [1, 0, 1], [-0.1, -0.1], 2), - "finite rollout logprob": _sample([7, 8, 9], [0, 1, 1], [None, -0.1], 2), - "outside generated": _sample([7, 8, 9, 10], [0, 1, 1, 1], [-0.1, -0.1, -0.1], 3, [("v", 1, 3)]), - } - for match, sample in cases.items(): - with pytest.raises(alignment.AlignmentError, match=match): - alignment.assert_sample_alignment(sample) - - -# ---------------------------------------------------------------- 2.2 legacy forwarding / 6.3 boundary - -def test_legacy_preflight_forwarder_matches_legacy_failure_classes(monkeypatch): - env = dict(adapter._OPENENV_IDENTITY_ENV) - preflight.forward_legacy_openenv_preflight(None, "qwen35_08b", env) - with pytest.raises(ValueError, match="requires backend profile"): - preflight.forward_legacy_openenv_preflight(None, "deepseekv4", env) - with pytest.raises(ValueError, match="environment drifted"): - preflight.forward_legacy_openenv_preflight(None, "qwen35_08b", {k: v for k, v in env.items() if "MODEL_REVISION" not in k}) - pins = preflight.required_pin_updates() - assert pins["CODEX_HARNESS_AGENT_SHA256"] == "459855e8845ab3b3cc1a4e2f47e5a9e8cc6cad79197208e43aa1d822a7c300ad" - assert pins["CODEX_OPENENV_AGENT"].endswith("codex_openenv_subprocess_agent_function.run") - - -# ---------------------------------------------------------------- A-T3-4: rollout workers configure themselves - -def _worker_provider_factory(miles_args): - assert miles_args.yeto_rl_learner_id == 5 - return _Provider(FakeTerminalEnvironment()) - - -def test_run_configures_the_rollout_worker_from_its_environment(monkeypatch): - """codex-smoke-20261003-6: every trajectory failed with "configure(provider=...) was not - called" because upstream Miles calls the agent function inside RolloutExecutor actors.""" - subprocess_agent.configure(provider=None) - monkeypatch.delenv(preflight.ENVIRONMENT_PROVIDER_ENV, raising=False) - # no provider in the environment -> still fails closed, nothing installed - with pytest.raises(RuntimeError, match="configure\\(provider=...\\) was not called"): - asyncio.run(subprocess_agent.run("http://x", "p", metadata={"task_id": "fix-git", "trajectory_id": "t"})) - assert subprocess_agent.configured()["provider"] is None - - monkeypatch.setenv(preflight.ENVIRONMENT_PROVIDER_ENV, f"{__name__}:_worker_provider_factory") - monkeypatch.setenv(preflight.LEARNER_ID_ENV, "5") - monkeypatch.setenv(preflight.MEMBER_CELL_ENV, "cell9") - assert preflight.configure_rollout_worker() is True - installed = subprocess_agent.configured() - assert isinstance(installed["provider"], _Provider) - assert installed["tool_wait_board"].learner_id == 5 and installed["harness_board"].learner_id == 5 - assert installed["member"] == "engine:cell9" - - # run() itself performs that configure when the module is still bare (and then - # refuses for IR-3 on the missing policy token, i.e. it got past the provider check) - subprocess_agent.configure(provider=None) - monkeypatch.setattr(subprocess_agent, "resolve_expected_policy_version", lambda metadata: None) - with pytest.raises(subprocess_agent.PolicyVersionMissing): - asyncio.run(subprocess_agent.run("http://x", "p", metadata={"task_id": "fix-git", "trajectory_id": "t"})) - assert isinstance(subprocess_agent.configured()["provider"], _Provider) - subprocess_agent.configure(provider=None) - - -def test_repeated_provider_failures_tear_the_rollout_down_instead_of_resampling(monkeypatch): - """A-T3-5: Miles has no abort cap (-7 looped 17k ABORTED samples); after the - threshold the worker raises a BaseException that escapes Miles' except Exception.""" - env = FakeTerminalEnvironment(passed=True) - provider, _ = _configure(monkeypatch, env, fail=True) - monkeypatch.setenv(subprocess_agent.PROVIDER_OUTAGE_THRESHOLD_ENV, "3") - for _ in range(2): - result = _run(subprocess_agent.run("http://miles", "p", {}, _metadata())) - assert tbench_reward.INFRASTRUCTURE_KEY in result - with pytest.raises(subprocess_agent.EnvironmentProviderOutage, match="3 consecutive"): - _run(subprocess_agent.run("http://miles", "p", {}, _metadata())) - assert not isinstance(subprocess_agent.EnvironmentProviderOutage("x"), Exception) - - # a success resets the counter; injected create_fail faults never count - provider.fail = False - assert tbench_outcome.OUTCOME_KEY in _run(subprocess_agent.run("http://miles", "p", {}, _metadata())) - from yeto.rl.harness.codex.tb2_provider import InjectedCreateFailure - - async def injected(task_id, trajectory_id): - raise InjectedCreateFailure("injected") - - provider.acquire = injected - for _ in range(5): - assert tbench_reward.INFRASTRUCTURE_KEY in _run(subprocess_agent.run("http://miles", "p", {}, _metadata())) - subprocess_agent.configure(provider=None) - - -def test_tool_wait_board_calls_are_awaited_in_order_and_never_fail_the_trajectory(monkeypatch): - """A-T3-7 (codex-smoke-20261003-11): the boards are max_concurrency actors, so - fire-and-forget exit/enter of successive tool calls raced; the agent now awaits each.""" - env = FakeTerminalEnvironment(passed=True) - - class _Ref: # stands in for ray.ObjectRef - def __init__(self, value): - self.value = value - - _Ref.__name__ = "ObjectRef" - order: list[tuple[str, str]] = [] - - class _RemoteBoard: - def __init__(self): - self.enter = SimpleNamespace(remote=lambda tid: order.append(("enter", tid)) or _Ref(1)) - self.exit = SimpleNamespace(remote=lambda tid: order.append(("exit", tid)) or _Ref(2)) - - resolved: list[Any] = [] - monkeypatch.setattr(subprocess_agent, "_board_resolve", lambda ref: resolved.append(ref) or ref.value) - monkeypatch.setenv("YETO_CODEX_OPENENV_ALLOW_SCRIPTED_DRIVER", "1") - subprocess_agent.configure(provider=_Provider(env), tool_wait_board=_RemoteBoard(), harness_board=None, member="m0") - result = _run(subprocess_agent.run("http://miles", "p", {}, _metadata())) - assert tbench_outcome.OUTCOME_KEY in result - kinds = [k for k, _ in order] - assert kinds and kinds == ["enter", "exit"] * (len(kinds) // 2) # strictly alternating, same trajectory - assert len(resolved) == len(order) # every board call was awaited - - class _FailingBoard: - def __init__(self): - self.enter = SimpleNamespace(remote=lambda tid: (_ for _ in ()).throw(RuntimeError("board down"))) - self.exit = SimpleNamespace(remote=lambda tid: (_ for _ in ()).throw(RuntimeError("board down"))) - - subprocess_agent.configure(provider=_Provider(env), tool_wait_board=_FailingBoard(), harness_board=None, member="m0") - assert tbench_outcome.OUTCOME_KEY in _run(subprocess_agent.run("http://miles", "p", {}, _metadata())) - subprocess_agent.configure(provider=None) - - -# ---------------------------------------------------------------- rl-fn-codex-rollout 1.0: runtime backend profile - -def test_openenv_profile_pin_relaxed_to_allowlist(monkeypatch): - from yeto.rl import codex_backend as cb - from yeto.rl.harness.codex import pins - - assert adapter.IMAGE_BACKEND_PROFILE_NAME == pins.OPENENV_BACKEND_PROFILE == "qwen35_08b" - assert set(pins.OPENENV_BACKEND_PROFILES) == {"qwen35_08b", "qwen35", "qwen38", "qwen38_next", "qwen38_next_4layer", "qwen3_30b_a3b"} # rlf-codex-rl 5.6; S24 27B - # the identity derives from the selected profile (codex_backend allowlist), per profile - assert adapter.profile_identity("qwen35_08b") == (cb.QWEN35_08B_MODEL, cb.QWEN35_08B_REVISION) - assert adapter.profile_identity("qwen38_next_4layer") == (cb.QWEN38_NEXT_4LAYER_MODEL, cb.QWEN38_NEXT_4LAYER_REVISION) - assert adapter.profile_identity("qwen38_next") == (cb.QWEN38_NEXT_MODEL, cb.QWEN38_NEXT_REVISION) - assert adapter.profile_identity("qwen35") == (cb.QWEN35_MODEL, cb.QWEN35_REVISION) # S17 M1: 4B allowlisted - assert adapter.profile_identity("qwen38") == (cb.QWEN38_MODEL, cb.QWEN38_REVISION) # S24: 27B allowlisted - with pytest.raises(ValueError, match="requires backend profile"): - adapter.profile_identity("deepseekv4") - # drift of an allowlisted profile is still refused - drifted = dict(cb.stock_codex_backend_profile("qwen38_next_4layer"), model_revision="0" * 40) - monkeypatch.setattr(adapter, "stock_codex_backend_profile", lambda name: drifted) - with pytest.raises(RuntimeError, match="profile drifted"): - adapter.profile_identity("qwen38_next_4layer") - - -def test_openenv_runtime_profile_resolution(): - from yeto.rl.codex_backend import stock_codex_backend_contract - - # the launch declaration the learner publishes (YETO_CODEX_CHAT_TEMPLATE) names the profile - for name in adapter.OPENENV_BACKEND_PROFILES: - assert stock_codex_backend_contract(name, 1)["chat_template"] == name - assert adapter.resolve_backend_profile(env={adapter.RUNTIME_PROFILE_ENV: name}) == name - assert adapter.resolve_backend_profile(name, env={}) == name - assert adapter.resolve_backend_profile(env={}) == "qwen35_08b" - assert adapter.resolve_backend_profile(env={adapter.RUNTIME_PROFILE_ENV: "deepseekv4"}) == "qwen35_08b" - with pytest.raises(ValueError, match="requires backend profile"): - adapter.resolve_backend_profile("deepseekv4", env={}) - # the image record is independent of the runtime profile - assert adapter._OPENENV_IDENTITY_ENV["YETO_CODEX_OPENENV_BACKEND_PROFILE"] == "qwen35_08b" - assert adapter._OPENENV_IDENTITY_ENV["YETO_CODEX_OPENENV_MODEL_ID"] == "Qwen/Qwen3.5-0.8B" - - -def test_legacy_preflight_accepts_fn_profiles_against_image_env(): - env = dict(adapter._OPENENV_IDENTITY_ENV) # the container env the image line bakes (qwen35_08b record) - for name in ("qwen38_next_4layer", "qwen38_next", "qwen35_08b"): - preflight.forward_legacy_openenv_preflight(None, name, env) - with pytest.raises(ValueError, match="requires backend profile"): - preflight.forward_legacy_openenv_preflight(None, "deepseekv4", env) - # the image record is still enforced for the FN profile - with pytest.raises(ValueError, match="environment drifted"): - preflight.forward_legacy_openenv_preflight( - None, "qwen38_next_4layer", {**env, "YETO_CODEX_OPENENV_BACKEND_PROFILE": "qwen38_next_4layer"}) - with pytest.raises(ValueError, match="environment drifted"): - preflight.forward_legacy_openenv_preflight(None, "qwen38_next_4layer", {k: v for k, v in env.items() if "MODEL_ID" not in k}) - - -def test_subprocess_run_records_phase_timing(monkeypatch): - """agentic-rollout-utilization 1.3: per-turn generation/tool seconds sum to the - worker wall time; judge = evaluate_time; sandbox cold start and wall start/end.""" - from yeto.rl.adapters.miles.rollout_meta_hook import trajectory_diagnostics - from yeto.rl.engine.trajectory_timing import phase_totals - - env = FakeTerminalEnvironment(passed=True) - _configure(monkeypatch, env) - result = _run(subprocess_agent.run("http://miles", "p", {}, _metadata())) - record = trajectory_diagnostics(result) - assert len(record["turn_tool_seconds"]) == 3 # two commands + submit - assert len(record["turn_generation_seconds"]) == 4 - totals = phase_totals(record) - assert totals["generation_seconds"] + totals["tool_seconds"] == pytest.approx( - record["worker_seconds"], abs=0.01) - assert totals["judge_seconds"] is not None and totals["judge_seconds"] >= 0 - assert record["sandbox_start_seconds"] >= 0 - assert record["trajectory_started_at"] <= record["trajectory_ended_at"] - - -# ---------------------------------------------------------------- S24 T-TB2: infrastructure never reaches reward or advantage - -class ConflictError(Exception): - """Stand-in for ``modal.exception.ConflictError`` ("Modal Sandbox is shutting down").""" - - -class NotFoundError(Exception): - """Stand-in for ``modal.exception.NotFoundError`` (sandbox already shut down).""" - - -def _assert_infrastructure(result: dict[str, Any], reason: str) -> None: - assert tbench_reward.INFRASTRUCTURE_KEY in result and tbench_outcome.OUTCOME_KEY not in result - assert reason in result[tbench_reward.INFRASTRUCTURE_KEY] - sample = SimpleNamespace(metadata=result, status=None) - assert _run(tbench_reward.reward_func(None, sample)) == 0.0 and sample.status == "ABORTED" - assert "success" not in result - - -def test_sandbox_shutdown_at_acquire_is_infrastructure(monkeypatch): - provider, _ = _configure(monkeypatch, FakeTerminalEnvironment(passed=True)) - - async def gone(task_id, trajectory_id): - raise ConflictError("Modal Sandbox is shutting down.") - - monkeypatch.setattr(provider, "acquire", gone) - _assert_infrastructure(_run(subprocess_agent.run("http://miles", "p", {}, _metadata())), "ConflictError") - - -@pytest.mark.parametrize("exc,reason", [ - (lambda: subprocess_agent.adapter.harness.CodexHarnessError( - "terminal_exec output exceeded its signed boundary"), "exceeded its signed boundary"), - (lambda: NotFoundError("Modal Sandbox with container ID ta-x not found"), "worker: NotFoundError"), -]) -def test_worker_failures_are_infrastructure_and_release_the_sandbox(monkeypatch, exc, reason): - provider, board = _configure(monkeypatch, FakeTerminalEnvironment(passed=True)) - - async def fail(*_a, **_k): - raise exc() - - monkeypatch.setattr(subprocess_agent, "_drive_worker", fail) - _assert_infrastructure(_run(subprocess_agent.run("http://miles", "p", {}, _metadata())), reason) - assert provider.destroyed == 1 and board.open == 0 - - -def test_verifier_sandbox_gone_is_infrastructure_not_a_crash(monkeypatch): - env = FakeTerminalEnvironment(passed=True) - provider, board = _configure(monkeypatch, env) - - async def gone(*_a, **_k): - raise NotFoundError("Modal Sandbox with container ID ta-x not found") - - monkeypatch.setattr(subprocess_agent.adapter, "finish_trusted", gone) - _assert_infrastructure(_run(subprocess_agent.run("http://miles", "p", {}, _metadata())), "verify: NotFoundError") - assert provider.destroyed == 1 and board.open == 0 - - -def test_a_group_with_an_infrastructure_member_never_reaches_the_advantage(): - """The TB2 group filter (``--dynamic-sampling-filter-path ...tbench_reward.check_group``) drops the - whole group before Miles computes rewards-to-advantages; Miles itself does not mask ABORTED samples.""" - from yeto.rl.harness.codex.codex_openenv_agent_function import infrastructure_metadata - - def passed(i): - env = FakeTerminalEnvironment(passed=bool(i % 2)) - return dict(_run(subprocess_agent.adapter.finish_trusted( - {"status": "completed", "metrics": {}, "episode_id": f"e{i}"}, env, task_id="fix-git", - sample_id=f"s{i}"))) - - good = [SimpleNamespace(metadata={**passed(i), "trajectory_id": f"t{i}"}, status=None, rollout_id=None) - for i in range(3)] - infra = SimpleNamespace(metadata={**infrastructure_metadata("verify: NotFoundError: gone", episode_id="e9"), - "trajectory_id": "t9"}, status=None, rollout_id=None) - assert tbench_reward.check_group(None, good).keep is True # mixed rewards: kept - out = tbench_reward.check_group(None, [*good, infra]) - assert out.keep is False and out.reason == "group_has_aborted" and infra.status == "ABORTED" - - -def test_tb2_group_check_is_a_bound_neutral_filter(): - import json as _json - - from yeto.rl.adapters.miles import binding - from yeto.rl.engine.algorithm import TBENCH_GROUP_CHECK_FILTER, AlgorithmSpec, normalize_filter_name - - assert normalize_filter_name("yeto.rl.harness.codex.tbench_reward.check_group") == TBENCH_GROUP_CHECK_FILTER - assert binding.filter_argv(TBENCH_GROUP_CHECK_FILTER) == [ - "--dynamic-sampling-filter-path", "yeto.rl.harness.codex.tbench_reward.check_group"] - from importlib import import_module - - module, _, name = binding.filter_path(TBENCH_GROUP_CHECK_FILTER).rpartition(".") - assert getattr(import_module(module), name) is tbench_reward.check_group - raw = _json.loads((Path(__file__).resolve().parents[1] - / "examples/rl_algorithms/decoupled-ppo-c2-agentic-tb2.json").read_text()) - spec = AlgorithmSpec.from_json(_json.dumps(raw)) if hasattr(AlgorithmSpec, "from_json") else AlgorithmSpec.from_dict(raw) - assert spec.sampling.filter == TBENCH_GROUP_CHECK_FILTER and spec.sampling.over_sampling_batch_size == 8 - assert spec.correction.upper == 2.0 and spec.execution.max_policy_staleness == 4 diff --git a/tests/test_harness_codex_preflight_member.py b/tests/test_harness_codex_preflight_member.py deleted file mode 100644 index b1b2c3ae2..000000000 --- a/tests/test_harness_codex_preflight_member.py +++ /dev/null @@ -1,106 +0,0 @@ -"""T3 G4: the Codex harness preflight derives the INFRA rollout member key.""" - -from __future__ import annotations - -from types import SimpleNamespace - -from yeto.rl.adapters.miles.rollout import member_id -from yeto.rl.adapters.miles import entry -from yeto.rl.harness.codex import preflight - - -def test_resolve_member_accepts_member_key_cell_attr_or_env(): - # explicit full member key - assert preflight.resolve_member(SimpleNamespace(yeto_rl_member_id="engine:cell-7"), {}) == "engine:cell-7" - # bare cell id in yeto_rl_member_id -> INFRA key - assert preflight.resolve_member(SimpleNamespace(yeto_rl_member_id="cell-7"), {}) == member_id("cell-7") - # cell id attr set by entry/INFRA - assert preflight.resolve_member(SimpleNamespace(yeto_rl_cell_id=3), {}) == "engine:3" - # environment fallback - assert preflight.resolve_member(SimpleNamespace(), {preflight.MEMBER_CELL_ENV: "c0"}) == "engine:c0" - # nothing -> global admission key (single-island smoke) - assert preflight.resolve_member(SimpleNamespace(), {}) is None - assert preflight.resolve_member(SimpleNamespace(yeto_rl_member_id=""), {}) is None - - -def test_harness_preflight_configures_the_member(monkeypatch): - seen = {} - from yeto.rl.harness.codex import codex_openenv_subprocess_agent_function as agent - - monkeypatch.setattr(agent, "configure", lambda **kw: seen.update(kw)) - monkeypatch.setattr(preflight, "preflight_codex_openenv", lambda env: {}) - monkeypatch.setattr(preflight, "resolve_environment_provider", lambda a, e: object()) - monkeypatch.setattr(preflight, "island_boards", lambda a: (None, None)) - args = SimpleNamespace(custom_agent_function_path=preflight.EXPECTED_AGENT_FUNCTION, - yeto_harness_reward_scope="trajectory", yeto_rl_cell_id="cell-1") - preflight.harness_preflight(args, None, env={}) - assert seen["member"] == "engine:cell-1" - - -# --- rl-fn-codex-rollout 0.6 (A16 / D5): the island entry publishes its own cell --- - -def test_entry_publishes_cell_from_learner_id_single_island_is_engine_0(): - env = {} - args = SimpleNamespace(yeto_rl_learner_id=0) - assert entry.publish_member_cell(args, 0, env) == "0" - assert args.yeto_rl_cell_id == "0" and env[preflight.MEMBER_CELL_ENV] == "0" - assert preflight.resolve_member(args, {}) == "engine:0" # field, no env needed - assert preflight.resolve_member(SimpleNamespace(), env) == "engine:0" # env fallback - assert preflight.worker_runtime_env(args, env)[preflight.MEMBER_CELL_ENV] == "0" - - -def test_entry_publishes_distinct_cells_for_two_islands_and_admission_is_per_member(): - from yeto.rl.engine.tool_wait import HarnessBoard - - envs = [{}, {}] - islands = [SimpleNamespace(yeto_rl_learner_id=i) for i in range(2)] - members = [preflight.resolve_member(a, e) for a, e in zip(islands, envs) - for _ in [entry.publish_member_cell(a, i, e) for i in [a.yeto_rl_learner_id]]] - assert members == ["engine:0", "engine:1"] - assert [e[preflight.MEMBER_CELL_ENV] for e in envs] == ["0", "1"] - hb = HarnessBoard() # the shared fake board: closing one island's member leaves the other open - hb.close_admission([members[1]]) - assert hb.allow_new_session(members[1]) is False - assert hb.allow_new_session(members[0]) is True - - -def test_entry_keeps_an_explicit_cell_and_falls_back_to_learner_field(): - env = {} - args = SimpleNamespace(yeto_rl_cell_id="c7", yeto_rl_learner_id=3) - assert entry.publish_member_cell(args, 3, env) == "c7" and env[preflight.MEMBER_CELL_ENV] == "c7" - args = SimpleNamespace(yeto_rl_learner_id=2) - assert entry.publish_member_cell(args, environ=env) == "2" - assert preflight.resolve_member(args, {}) == "engine:2" - - -def test_configure_rollout_worker_reads_cell_from_env(monkeypatch): - seen = {} - from yeto.rl.harness.codex import codex_openenv_subprocess_agent_function as agent - - monkeypatch.setattr(agent, "configure", lambda **kw: seen.update(kw)) - monkeypatch.setattr(preflight, "resolve_environment_provider", lambda a, e: object()) - monkeypatch.setattr(preflight, "island_boards", lambda a: (None, None)) - assert preflight.configure_rollout_worker({preflight.ENVIRONMENT_PROVIDER_ENV: "x:y", - preflight.MEMBER_CELL_ENV: "1"}) is True - assert seen["member"] == "engine:1" - assert preflight.configure_rollout_worker({}) is False - - -def test_preflight_stage_publishes_engine_0_on_a_single_island(monkeypatch): - from yeto.rl.engine.algorithm import AlgorithmSpec - - for name, val in [("ports_runtime_fingerprint", lambda launch: "fp"), - ("miles_capabilities", lambda fp, unverified_mechanisms=(): "caps"), - ("with_partitioned_serial", lambda caps: caps), - ("execution_profile_for", lambda *a, **k: "profile"), - ("expected_algorithm_sha256", lambda a: None), - ("preflight", lambda profile, algorithm, caps: None), - ("elastic_wiring_for", lambda a, profile, fingerprint: None), - ("resolve_harness_preflight", lambda a, environ=None: None)]: - monkeypatch.setattr(entry, name, val) - monkeypatch.delenv(preflight.MEMBER_CELL_ENV, raising=False) - args = SimpleNamespace(yeto_rl_learner_id=0) - entry.preflight_stage(args, "launch", AlgorithmSpec(), yeto_policy_sync=False) - import os - assert args.yeto_rl_cell_id == "0" and os.environ[preflight.MEMBER_CELL_ENV] == "0" - assert preflight.resolve_member(args) == "engine:0" diff --git a/tests/test_harness_session_protocol.py b/tests/test_harness_session_protocol.py deleted file mode 100644 index e9e112806..000000000 --- a/tests/test_harness_session_protocol.py +++ /dev/null @@ -1,88 +0,0 @@ -"""codex harness against the session-service protocol test double (decoupling 3.8). - -The trusted-layer session code (``codex_openenv_agent_function``) talks real -HTTP to ``tests/session_service_double.py`` instead of a Miles session server; -the protocol is ``yeto/rl/harness/session_protocol.md``. CPU only, localhost. -""" - -from __future__ import annotations - -import asyncio - -import aiohttp -import pytest - -from session_service_double import SessionServiceDouble -from yeto.rl.harness.codex import codex_openenv_agent_function as agent_fn -from yeto.rl.harness.codex import compaction_bridge - - -def _run(coro): - return asyncio.run(coro) - - -async def _main_session(double: SessionServiceDouble) -> str: - async with aiohttp.ClientSession() as http: - async with http.post(f"{double.router}/sessions", json={}) as reply: - assert reply.status == 200 - return double.session_url((await reply.json())["session_id"]) - - -def test_create_chat_collect_delete_through_the_protocol(): - async def scenario(): - async with SessionServiceDouble() as double: - base = await _main_session(double) - ids, urls = await agent_fn.create_segment_sessions( - base, {"temperature": 0.7, "top_p": 0.9, "top_k": 20, "max_tokens": 5}, 2) - assert urls == [double.session_url(i) for i in ids] - for sid in ids: # sampling defaults went into CreateSessionRequest, typed - assert double.sessions[sid].params == {"evaluation": False, "temperature": 0.7, - "top_p": 0.9, "top_k": 20} - async with aiohttp.ClientSession() as http: - async with http.post(f"{urls[0]}/v1/chat/completions", - json={"messages": [{"role": "user", "content": "hi"}]}) as r: - body = await r.json() - async with http.post(f"{urls[0]}/samples", json={"max_seq_len": 64}) as r: - samples = await r.json() - choice = body["choices"][0] - assert [t[1] for t in choice["meta_info"]["output_token_logprobs"]] == [11, 12, 13] - assert double.sessions[ids[0]].records[0]["request"]["logprobs"] is True - assert samples["samples"][0]["tokens"] == [11, 12, 13] - # Unreturned segments are released; a second release sees 404 and is still fine. - segments = {compaction_bridge.SESSIONS_METADATA_KEY: ids} - assert await agent_fn.release_unreturned_segments(base, segments) == [] - assert double.deleted == ids - assert await agent_fn.delete_segment_sessions(base, ids) == [] - _run(scenario()) - - -def test_half_failed_pre_creation_deletes_what_it_made(): - async def scenario(): - async with SessionServiceDouble() as double: - base = await _main_session(double) - calls = {"n": 0} - - async def flaky_post(url, body): - calls["n"] += 1 - if calls["n"] == 2: - raise RuntimeError("router went away") - return await agent_fn._post_json(url, body) - - with pytest.raises(RuntimeError, match="router went away"): - await agent_fn.create_segment_sessions(base, {}, 3, post=flaky_post) - assert double.deleted == ["s2"] # the one segment created before the failure - assert set(double.sessions) == {"s1"} # only the main session remains - _run(scenario()) - - -def test_bad_create_body_is_rejected_by_the_double(): - async def scenario(): - async with SessionServiceDouble() as double: - base = await _main_session(double) - with pytest.raises(RuntimeError, match="HTTP 400"): - await agent_fn._post_json(f"{double.router}/sessions", {"top_k": "20"}) - with pytest.raises(RuntimeError, match="no session_id"): - async def no_id(url, body): - return {} - await agent_fn.create_segment_sessions(base, {}, 1, post=no_id) - _run(scenario()) diff --git a/tests/test_harness_tb2_provider.py b/tests/test_harness_tb2_provider.py deleted file mode 100644 index d5cd82f84..000000000 --- a/tests/test_harness_tb2_provider.py +++ /dev/null @@ -1,707 +0,0 @@ -"""G1 (real TB2 EnvironmentProvider, CPU backend) and G6(a) (tito counter mirroring). - -The ``LocalProcessBackend`` stands in for a Modal sandbox: commands run in a -scratch root on this host, ``/tests`` and ``/logs/verifier`` are redirected -through ``TB2_TESTS_DIR`` / ``TB2_VERIFIER_LOGS_DIR``. Everything above the -backend (relay wire, verifier, lease, faults, trusted-side counters) is the -production code path. -""" - -from __future__ import annotations - -import asyncio -import copy -import json -import os -import textwrap -from pathlib import Path -from types import SimpleNamespace -from typing import Any - -import aiohttp -import pytest - -import test_secrlenv_codex_harness as legacy_tests -from yeto.rl import tbench_outcome -from yeto.rl.engine.tool_wait import HarnessBoard -from yeto.rl.harness.codex import ( - codex_harness_agent as harness, - codex_openenv_agent_function as adapter, - codex_openenv_subprocess_agent_function as subprocess_agent, - preflight, - tb2_provider, - tbench_reward, -) -from yeto.rl.harness.codex.environment import HttpTerminalEnvironment - - -TEST_SH = textwrap.dedent( - """\ - #!/bin/bash - L=${TB2_VERIFIER_LOGS_DIR:-/logs/verifier} - mkdir -p "$L" - if [ -f fixed.txt ] && grep -q ok fixed.txt; then - echo 1 > "$L/reward.txt"; exit 0 - else - echo 0 > "$L/reward.txt"; exit 1 - fi - """ -) - - -def _make_task(root: Path, task_id: str = "fix-git") -> Path: - task_dir = root / task_id - (task_dir / "tests").mkdir(parents=True) - (task_dir / "environment").mkdir() - (task_dir / "task.toml").write_text( - '[environment]\ndocker_image = "alexgshaw/fix-git:20251031"\ncpus = 1\nmemory_mb = 2048\n' - "[agent]\ntimeout_sec = 900.0\n[verifier]\ntimeout_sec = 60.0\n" - ) - (task_dir / "environment" / "Dockerfile").write_text("FROM ubuntu:24.04\nWORKDIR /app\n") - (task_dir / "tests" / "test.sh").write_text(TEST_SH) - (task_dir / "tests" / "test_outputs.py").write_text("# fixture\n") - return root - - -def _run(coro): - return asyncio.run(coro) - - -def _provider(tmp_path: Path, **kw: Any) -> tb2_provider.Tb2EnvironmentProvider: - tasks_dir = _make_task(tmp_path / "tb2") - backend = tb2_provider.LocalProcessBackend(tmp_path / "sandboxes") - return tb2_provider.Tb2EnvironmentProvider(backend, tasks_dir, **kw) - - -def _configure(monkeypatch, provider, harness_board: Any = None) -> None: - monkeypatch.setenv("YETO_CODEX_OPENENV_ALLOW_SCRIPTED_DRIVER", "1") - subprocess_agent.configure(provider=provider, tool_wait_board=None, harness_board=harness_board, member="m0") - - -def _metadata(script: list[str], **extra) -> dict[str, Any]: - return {"task_id": "fix-git", "trajectory_id": "traj-1", "prompt": "fix the repo", - "codex_openenv_driver": "scripted", "script": script, "expected_policy_version": "pv-7", **extra} - - -# ---------------------------------------------------------------- task resolution / faults - - -def test_resolve_task_reads_task_toml_and_dockerfile_workdir(tmp_path): - tasks_dir = _make_task(tmp_path) - task = tb2_provider.resolve_task("fix-git", tasks_dir) - assert task.docker_image == "alexgshaw/fix-git:20251031" and task.workdir == "/app" - assert task.cpus == 1 and task.memory_mb == 2048 and task.verifier_timeout_s == 60.0 - with pytest.raises(FileNotFoundError): - tb2_provider.resolve_task("nope", tasks_dir) - with pytest.raises(ValueError): - tb2_provider.resolve_task("../fix-git", tasks_dir) - - -def test_parse_faults_accepts_matrix_entries_and_rejects_unknown_kinds(): - faults = tb2_provider.parse_faults("create_fail:0, deadline:traj-9=2, max_turns:2=2") - assert [f.kind for f in faults] == ["create_fail", "deadline", "max_turns"] - assert faults[0].matches(0, "x") and not faults[0].matches(1, "x") - assert faults[1].matches(5, "traj-9") and faults[1].value == "2" - assert tb2_provider.parse_faults("") == () - with pytest.raises(ValueError, match="invalid fault"): - tb2_provider.parse_faults("explode:0") - with pytest.raises(ValueError, match="selector"): - tb2_provider.parse_faults("create_fail:") - - -# ---------------------------------------------------------------- lease lifecycle on the wire - - -def test_lease_relays_execute_submit_verifies_and_is_gone_after_destroy(tmp_path): - provider = _provider(tmp_path) - - async def scenario() -> None: - lease = await provider.acquire("fix-git", "traj-1") - assert lease.deadline_seconds == 900.0 and await lease.describe() == "live" - async with HttpTerminalEnvironment(lease.env_url, lease.env_token) as env: - out = await env.execute("ep-1", "echo hello; pwd; exit 3", timeout_seconds=10, output_bytes=4096) - assert out["exit_code"] == 3 and "hello" in out["output"] and out["output"].rstrip().endswith("/app") - assert out["timed_out"] is False and out["truncated"] is False - big = await env.execute("ep-1", "head -c 100 /dev/zero | tr '\\0' x", timeout_seconds=10, output_bytes=16) - assert big["truncated"] is True and len(big["output"]) == 16 - slow = await env.execute("ep-1", "sleep 5", timeout_seconds=1, output_bytes=64) - assert slow["timed_out"] is True - # the verifier is not reachable before evaluate: /tests is empty - probe = await env.execute("ep-1", "ls $TB2_TESTS_DIR 2>/dev/null | wc -l", timeout_seconds=10, output_bytes=64) - assert probe["output"].strip() == "0" - unknown = HttpTerminalEnvironment(lease.env_url, "wrong") - async with unknown: - with pytest.raises(Exception): - await unknown.execute("ep-1", "true", timeout_seconds=1, output_bytes=1) - assert (await env.execute("ep-1", "echo ok > fixed.txt", timeout_seconds=10, output_bytes=64))["exit_code"] == 0 - assert (await env.submit("ep-1", {"evidence": "done"})) == {"accepted": True} - evaluation = await lease.verifier.evaluate("ep-1") - assert {k: v for k, v in evaluation.items() if k != "log"} == {"passed": True, "testsh_rc": 0, "timed_out": False} - assert "YETO_TB2_REWARD=1" in evaluation["log"] - assert lease.environment.commands[0].startswith("echo hello") and lease.environment.submitted - await lease.destroy() - assert await lease.describe() == "gone" and provider.destroyed == 1 and provider.live == {} - await lease.destroy() # idempotent - assert provider.destroyed == 1 - - _run(scenario()) - - -def test_verifier_reports_failure_and_binds_to_its_episode(tmp_path): - provider = _provider(tmp_path) - - async def scenario() -> None: - lease = await provider.acquire("fix-git", "traj-2") - evaluation = await lease.verifier.evaluate("ep-1") # unbound: no command ran (e.g. timeout path) - assert evaluation["passed"] is False and evaluation["testsh_rc"] == 1 - async with HttpTerminalEnvironment(lease.env_url, lease.env_token) as env: - await env.execute("ep-1", "true", timeout_seconds=5, output_bytes=16) - with pytest.raises(Exception): # a second episode on the same sandbox is refused - await env.execute("ep-2", "true", timeout_seconds=5, output_bytes=16) - with pytest.raises(ValueError): - await lease.verifier.evaluate("other-episode") - await lease.destroy() - - _run(scenario()) - - -# ---------------------------------------------------------------- full subprocess path (R-TB matrix) - - -def test_subprocess_run_positive_and_negative_rewards_through_the_real_relay(monkeypatch, tmp_path): - hb = HarnessBoard() - provider = _provider(tmp_path) - _configure(monkeypatch, provider, harness_board=hb) - result = _run(subprocess_agent.run("http://miles", "p", {}, _metadata(["ls", "echo ok > fixed.txt"]))) - outcome, reward = tbench_outcome.verified_outcome(result) - assert reward == 1.0 and outcome["status"] == "completed" and outcome["verifier"] == tbench_outcome.TEST_SH_VERIFIER - assert result["chains_total"] == 1 and result["chain_break_reason"] is None - assert provider.destroyed == 1 and provider.live == {} - snap = hb.snapshot() - assert snap.env_live == 0 and snap.in_flight == 0 and snap.tito_session_mismatch == 0 - - result = _run(subprocess_agent.run("http://miles", "p", {}, _metadata(["ls", "echo nope > fixed.txt"], trajectory_id="traj-neg"))) - outcome, reward = tbench_outcome.verified_outcome(result) - assert reward == 0.0 and outcome["status"] == "completed" and tbench_outcome.OUTCOME_KEY in result - assert provider.destroyed == 2 and hb.snapshot().env_live == 0 - - -def test_create_fail_fault_is_infrastructure_aborted_and_touches_no_sandbox(monkeypatch, tmp_path): - hb = HarnessBoard() - provider = _provider(tmp_path, faults=tb2_provider.parse_faults("create_fail:traj-1")) - _configure(monkeypatch, provider, harness_board=hb) - result = _run(subprocess_agent.run("http://miles", "p", {}, _metadata(["ls"]))) - assert tbench_reward.INFRASTRUCTURE_KEY in result and tbench_outcome.OUTCOME_KEY not in result - assert "InjectedCreateFailure" in result[tbench_reward.INFRASTRUCTURE_KEY] - sample = SimpleNamespace(metadata=result, status=None) - assert _run(tbench_reward.reward_func(None, sample)) == 0.0 and sample.status == "ABORTED" - assert provider.backend.created == [] and provider.fault_log == [("create_fail", "traj-1", 0)] - assert hb.snapshot().env_live == 0 and hb.snapshot().in_flight == 0 - # ordinal selectors: the second acquire is unaffected - ok = _run(subprocess_agent.run("http://miles", "p", {}, _metadata(["echo ok > fixed.txt"], trajectory_id="traj-2"))) - assert tbench_outcome.verified_outcome(ok)[1] == 1.0 - - -def test_deadline_fault_cancels_the_episode_destroys_and_drains_env_live(monkeypatch, tmp_path): - hb = HarnessBoard() - provider = _provider(tmp_path, faults=tb2_provider.parse_faults("deadline:0=1")) - _configure(monkeypatch, provider, harness_board=hb) - seen: list[int] = [] - - async def scenario() -> dict[str, Any]: - task = asyncio.create_task(subprocess_agent.run("http://miles", "p", {}, _metadata(["sleep 30"]))) - while not hb.snapshot().env_live: - await asyncio.sleep(0.02) - seen.append(hb.snapshot().env_live) - return await task - - result = _run(scenario()) - outcome, reward = tbench_outcome.verified_outcome(result) - assert seen == [1] and reward == 0.0 and outcome["status"] == "timeout" - assert outcome["verifier"] == tbench_outcome.TIMEOUT_VERIFIER - assert provider.destroyed == 1 and not provider.backend.created[0].alive() - snap = hb.snapshot() - assert snap.env_live == 0 and snap.in_flight == 0 and snap.leases_expired_total == 0 - - -def test_max_turns_fault_sets_the_worker_turn_budget(monkeypatch, tmp_path): - provider = _provider(tmp_path, faults=tb2_provider.parse_faults("max_turns:traj-1=2")) - _configure(monkeypatch, provider) - captured: dict[str, Any] = {} - original = asyncio.create_subprocess_exec - - async def spy(*args, **kwargs): - captured["env"] = dict(kwargs["env"]) - return await original(*args, **kwargs) - - monkeypatch.setattr(asyncio, "create_subprocess_exec", spy) - result = _run(subprocess_agent.run("http://miles", "p", {}, _metadata(["echo ok > fixed.txt"]))) - assert tbench_outcome.verified_outcome(result)[1] == 1.0 - assert captured["env"][tb2_provider.TURN_BUDGET_ENV] == "2" - # the budget is the bridge's signed turn limit - monkeypatch.setenv(tb2_provider.TURN_BUDGET_ENV, "2") - legacy_tests._set_bridge_env(monkeypatch) - bridge = harness._ResponsesBridge("http://miles", "p", {"temperature": 0.7, "top_p": 0.9, "max_tokens": 128, "stream": True, "n": 1}, - harness.legacy.AgentMetrics(), max_seq_len=512) - assert bridge._max_turns == 2 - - -# ---------------------------------------------------------------- factories (module:callable) - - -def test_local_provider_factory_reads_env_and_resolves_through_preflight(monkeypatch, tmp_path): - tasks_dir = _make_task(tmp_path / "tb2") - monkeypatch.delenv(tb2_provider.TASKS_DIR_ENV, raising=False) - with pytest.raises(RuntimeError, match=tb2_provider.TASKS_DIR_ENV): - tb2_provider.local_provider(None) - monkeypatch.setenv(tb2_provider.TASKS_DIR_ENV, str(tasks_dir)) - monkeypatch.setenv(tb2_provider.FAULT_ENV, "create_fail:3") - monkeypatch.setenv(tb2_provider.LEASE_SECONDS_ENV, "120") - provider = preflight.resolve_environment_provider( - SimpleNamespace(yeto_harness_environment_provider=None), - {preflight.ENVIRONMENT_PROVIDER_ENV: "yeto.rl.harness.codex.tb2_provider:local_provider"}, - ) - assert isinstance(provider, tb2_provider.Tb2EnvironmentProvider) - assert provider.lease_seconds == 120.0 and provider.faults[0].kind == "create_fail" - monkeypatch.setattr(tb2_provider, "require_modal_client", lambda: None) # no client in the test venv - modal = tb2_provider.modal_provider(None) - assert isinstance(modal.backend, tb2_provider.ModalSandboxBackend) and modal.backend.app_name == tb2_provider.DEFAULT_MODAL_APP - - -def test_modal_backend_maps_task_to_sandbox_create(monkeypatch, tmp_path): - import sys - import types - - calls: dict[str, Any] = {} - - class _Sandbox: - object_id = "sb-1" - - def __init__(self) -> None: - self.polls = 0 - - def exec(self, *args, **kwargs): - calls["exec"] = (args, kwargs) - return SimpleNamespace(stdout=SimpleNamespace(read=lambda: "out\n"), stderr=SimpleNamespace(read=lambda: ""), wait=lambda: 0) - - def terminate(self): - calls["terminated"] = True - - def poll(self): - return None if "terminated" not in calls else 0 - - @classmethod - def create(cls, *args, **kwargs): - calls["create"] = (args, kwargs) - return cls() - - fake = types.ModuleType("modal") - fake.Sandbox = _Sandbox - fake.Image = SimpleNamespace(from_registry=lambda tag: ("image", tag)) - fake.App = SimpleNamespace(lookup=lambda name, create_if_missing: ("app", name)) - monkeypatch.setitem(sys.modules, "modal", fake) - - task = tb2_provider.resolve_task("fix-git", _make_task(tmp_path)) - backend = tb2_provider.ModalSandboxBackend(app_name="t", ttl_s=600, idle_timeout_s=60, run_id="r1") - handle = backend.create(task, "traj-1") - args, kwargs = calls["create"] - assert args == ("sleep", "infinity") and kwargs["image"] == ("image", "alexgshaw/fix-git:20251031") - assert kwargs["app"] == ("app", "t") and kwargs["timeout"] == 600 and kwargs["idle_timeout"] == 60 - assert kwargs["cpu"] == 1.0 and kwargs["memory"] == 2048 and kwargs["workdir"] == "/app" - assert kwargs["tags"] == {"yeto-tb2-task": "fix-git", "yeto-trajectory": "traj-1", "yeto-run-id": "r1"} - result = handle.exec("echo out", timeout_s=5) - assert result.exit_code == 0 and result.output == "out\n" and calls["exec"][1]["workdir"] == "/app" - assert handle.alive() - handle.terminate() - assert not handle.alive() - - -# ---------------------------------------------------------------- G6(a): tito counters - - -def test_bridge_counts_session_mismatch_and_retry_fork(monkeypatch): - legacy_tests._set_bridge_env(monkeypatch) - - async def scenario() -> None: - runner, miles_url, _requests = await legacy_tests._fake_miles([ - legacy_tests._completion("terminal.exec", '{ "command": "git status" }', "call-1", "look", 25), - ]) - metrics = harness.legacy.AgentMetrics() - try: - async with harness._ResponsesBridge( - miles_url, "fix the repo", {"temperature": 0.7, "top_p": 0.9, "max_tokens": 128, "stream": True, "n": 1}, - metrics, max_seq_len=512, - ) as bridge: - headers = {"Authorization": f"Bearer {bridge.token}"} - initial = [{"type": "message", "role": "user", "content": [{"type": "input_text", "text": "fix the repo"}]}] - async with aiohttp.ClientSession() as session: - async with session.post(f"{bridge.url}/v1/responses", json=legacy_tests._codex_body(initial), headers=headers) as r: - assert r.status == 200 - assert harness.tito_counters(metrics) == {"tito_session_mismatch": 0, "tito_chain_breaks": {}} - # byte-identical retry -> retry_fork chain break - async with session.post(f"{bridge.url}/v1/responses", json=legacy_tests._codex_body(initial), headers=headers) as r: - assert r.status == 400 - mutated = copy.deepcopy(initial) - mutated[0]["content"][0]["text"] = "fix the repo!" - async with session.post(f"{bridge.url}/v1/responses", json=legacy_tests._codex_body(mutated), headers=headers) as r: - assert r.status == 400 - finally: - await runner.cleanup() - assert harness.tito_counters(metrics) == {"tito_session_mismatch": 1, "tito_chain_breaks": {"retry_fork": 1}} - assert adapter._metrics_dict(metrics)["tito_chain_breaks"] == {"retry_fork": 1} - - _run(scenario()) - - -def test_mirror_tito_counters_uses_gateway_board_semantics(): - hb = HarnessBoard() - fields = adapter.mirror_tito_counters({"tito_session_mismatch": 2, "tito_chain_breaks": {"retry_fork": 1, "history_rewrite": 0}}, hb) - assert fields == {"chain_break_reason": "retry_fork"} - snap = hb.snapshot() - assert snap.tito_session_mismatch == 2 and snap.tito_chain_breaks == {"retry_fork": 1} - assert adapter.mirror_tito_counters(None, hb) == {"chain_break_reason": None} - assert adapter.mirror_tito_counters({"tito_session_mismatch": 1}, None) == {"chain_break_reason": None} - with pytest.raises(ValueError): - adapter.mirror_tito_counters({"tito_chain_breaks": {"made_up": 1}}, hb) - - -def test_mirror_tito_counters_awaits_remote_board_calls_in_order(monkeypatch, capsys): - """S14/A19 (FINAL-REPORT-S7 §6.1, A-T3-7 leftovers): record_session_mismatch / - record_chain_break on an actor board were fire-and-forget; each call is now - resolved in issue order, and a remote failure never fails the trajectory.""" - - class _Ref: # stands in for ray.ObjectRef - def __init__(self, value): - self.value = value - - _Ref.__name__ = "ObjectRef" - order: list[tuple[str, tuple[Any, ...]]] = [] - resolved: list[Any] = [] - monkeypatch.setattr("yeto.rl.engine.tool_wait._resolve", lambda ref: resolved.append(ref) or ref.value) - - class _RemoteBoard: - def __init__(self): - self.record_session_mismatch = SimpleNamespace( - remote=lambda *a: order.append(("record_session_mismatch", a)) or _Ref(None)) - self.record_chain_break = SimpleNamespace( - remote=lambda *a: order.append(("record_chain_break", a)) or _Ref(None)) - - metrics = {"tito_session_mismatch": 2, "tito_chain_breaks": {"retry_fork": 1, "history_rewrite": 3}} - assert adapter.mirror_tito_counters(metrics, _RemoteBoard()) == {"chain_break_reason": "retry_fork"} - assert order == [("record_session_mismatch", (2,)), ("record_chain_break", ("retry_fork", 1)), - ("record_chain_break", ("history_rewrite", 3))] - assert len(resolved) == len(order) # every remote call was awaited, in issue order - - class _FailingBoard: - def __init__(self): - self.record_session_mismatch = SimpleNamespace( - remote=lambda *a: (_ for _ in ()).throw(RuntimeError("board down"))) - self.record_chain_break = SimpleNamespace( - remote=lambda *a: (_ for _ in ()).throw(RuntimeError("board down"))) - - assert adapter.mirror_tito_counters(metrics, _FailingBoard()) == {"chain_break_reason": "retry_fork"} - assert "harness board record_session_mismatch failed: RuntimeError: board down" in capsys.readouterr().err - - -def test_subprocess_rejection_counters_reach_the_board_and_abort(monkeypatch, tmp_path): - hb = HarnessBoard() - provider = _provider(tmp_path) - _configure(monkeypatch, provider, harness_board=hb) - - async def rejected(job, trajectory_id, board, worker_env=None): - error = harness.CodexHarnessError("Codex truncated or mutated episode history") - error.metrics = {"turns": 1, "tito_session_mismatch": 1, "tito_chain_breaks": {"retry_fork": 1}} - raise error - - monkeypatch.setattr(subprocess_agent, "_drive_worker", rejected) - result = _run(subprocess_agent.run("http://miles", "p", {}, _metadata(["ls"]))) - assert tbench_reward.INFRASTRUCTURE_KEY in result and tbench_outcome.OUTCOME_KEY not in result - assert result["chains_total"] == 1 and result["chain_break_reason"] == "retry_fork" - assert result["agent_metrics"]["tito_session_mismatch"] == 1 - sample = SimpleNamespace(metadata=result, status=None) - assert _run(tbench_reward.reward_func(None, sample)) == 0.0 and sample.status == "ABORTED" - snap = hb.snapshot() - assert snap.tito_session_mismatch == 1 and snap.tito_chain_breaks == {"retry_fork": 1} - assert snap.env_live == 0 and snap.in_flight == 0 and provider.destroyed == 1 - - -def test_worker_error_event_carries_metrics(monkeypatch): - import subprocess - import sys - - monkeypatch.setenv("YETO_CODEX_OPENENV_ALLOW_SCRIPTED_DRIVER", "1") - - async def boom(job, env, tool_wait=None): - error = harness.CodexHarnessError("Codex retried a Responses sample") - error.metrics = {"tito_chain_breaks": {"retry_fork": 1}} - raise error - - code = ( - "import asyncio, json, sys\n" - "from yeto.rl.harness.codex import codex_openenv_agent_worker as w, codex_openenv_agent_function as a\n" - "async def boom(job, env, tool_wait=None):\n" - " e = a.harness.CodexHarnessError('Codex retried a Responses sample'); e.metrics = {'tito_chain_breaks': {'retry_fork': 1}}; raise e\n" - "a.drive_untrusted = boom\n" - "sys.exit(w.main())\n" - ) - job = {"env_url": "http://127.0.0.1:9", "env_token": "t", "episode_id": "e", "base_url": "b", "prompt": "p", "request_kwargs": {}} - proc = subprocess.run([sys.executable, "-c", code], input=json.dumps(job) + "\n", capture_output=True, text=True, - env={k: v for k, v in os.environ.items() if "HMAC" not in k}) - event = json.loads(proc.stdout.strip().splitlines()[-1]) - assert proc.returncode == 1 and event["event"] == "error" and event["metrics"] == {"tito_chain_breaks": {"retry_fork": 1}} - del boom - - -def test_modal_provider_fails_closed_without_an_importable_modal_client(monkeypatch): - """A-T3-5 (codex-smoke-20261003-7): the ports image has no Modal client; the - driver preflight must refuse instead of every rollout worker failing at acquire.""" - import sys - - monkeypatch.setitem(sys.modules, "modal", None) # import raises ImportError - with pytest.raises(RuntimeError, match="importable `modal` client"): - tb2_provider.modal_provider(None) - assert tb2_provider.InjectedCreateFailure.injected_fault is True - - -# ---------------------------------------------------------------- task statement (S15 stage-2 root cause) - - -SMOKE6_PROMPT = [{"role": "system", "content": "You are an autonomous terminal agent solving a Terminal-Bench task."}] - - -def test_resolve_task_reads_instruction_md(tmp_path): - tasks_dir = _make_task(tmp_path / "tb2") - assert tb2_provider.resolve_task("fix-git", tasks_dir).instruction is None - (tasks_dir / "fix-git" / "instruction.md").write_text("\nFind my lost changes and merge them into master.\n") - assert tb2_provider.resolve_task("fix-git", tasks_dir).instruction == "Find my lost changes and merge them into master." - - -def test_task_prompt_precedence_and_fail_closed(): - lease = SimpleNamespace(task=SimpleNamespace(instruction="TASK STATEMENT")) - no_task = SimpleNamespace(task=None) - assert subprocess_agent.task_prompt({"prompt": "explicit"}, SMOKE6_PROMPT, lease) == "explicit" - # the smoke6 row shape: system-only chat prompt, no metadata prompt -> instruction.md - assert subprocess_agent.task_prompt({"task_id": "fix-git"}, SMOKE6_PROMPT, lease) == "TASK STATEMENT" - assert subprocess_agent.task_prompt({}, "plain", no_task) == "plain" - chat = SMOKE6_PROMPT + [{"role": "user", "content": "do it"}] - assert subprocess_agent.task_prompt({}, chat, no_task) == "do it" - with pytest.raises(subprocess_agent.TaskPromptMissing): - subprocess_agent.task_prompt({"task_id": "fix-git"}, SMOKE6_PROMPT, no_task) - - -def test_subprocess_run_sends_instruction_md_not_the_stringified_chat_prompt(monkeypatch, tmp_path): - provider = _provider(tmp_path) - (provider.tasks_dir / "fix-git" / "instruction.md").write_text("Find my lost changes and merge them into master.\n") - _configure(monkeypatch, provider) - seen: dict[str, Any] = {} - original = subprocess_agent._drive_worker - - async def spy(job, *args, **kwargs): - seen["prompt"] = job["prompt"] - return await original(job, *args, **kwargs) - - monkeypatch.setattr(subprocess_agent, "_drive_worker", spy) - metadata = _metadata(["echo ok > fixed.txt"]) - del metadata["prompt"] - result = _run(subprocess_agent.run("http://miles", SMOKE6_PROMPT, {}, metadata)) - assert seen["prompt"] == "Find my lost changes and merge them into master." - assert "'role'" not in seen["prompt"] - assert tbench_outcome.verified_outcome(result)[1] == 1.0 - assert provider.destroyed == 1 and provider.live == {} - - -# ---------------------------------------------------------------- TB2 system prompt (2026-10-07) / prompt preflight - - -def test_legacy_instructions_hash_unchanged_and_tb2_prompt_pinned(monkeypatch): - import hashlib - import yeto.rl as rl_config - - monkeypatch.delenv(harness.INSTRUCTIONS_FAMILY_ENV, raising=False) - assert harness.base_instructions() is harness.BASE_INSTRUCTIONS - assert harness.BASE_INSTRUCTIONS_SHA256 == rl_config.CODEX_BASE_INSTRUCTIONS_SHA256 == ( - "1c183656ca1319142cba9e76baa199b7ab59f770a51a76660622a087e74ba846") - assert harness.codex_harness_identity()["base_instructions_sha256"] == rl_config.CODEX_BASE_INSTRUCTIONS_SHA256 - assert harness._tb2_surface_sha256() == harness.TB2_BASE_INSTRUCTIONS_SHA256 == rl_config.CODEX_TB2_BASE_INSTRUCTIONS_SHA256 - # input schemas are kept verbatim (optional "flag" property); prompt + descriptions are reworded - tb2_text = harness.TB2_BASE_INSTRUCTIONS + " ".join(t["description"] for t in harness.TB2_DYNAMIC_TOOLS) - for ctf in ("TARGET_URL", "flag", "DEBUG_URL", "security", "attack", "scored target"): - assert ctf not in tb2_text, ctf - # the prompt names the model-facing tools and requires submit - for needle in ("`terminal.exec`", "`submit`", "exactly ONE tool"): - assert needle in " ".join(harness.TB2_BASE_INSTRUCTIONS.split()) - # legacy surface unchanged: legacy tools / Miles tools - assert harness.dynamic_tools() is harness.DYNAMIC_TOOLS and harness.miles_tools() is harness._MILES_TOOLS - assert harness._miles_tools_for(harness.DYNAMIC_TOOLS) == harness._MILES_TOOLS - # compaction head bound was sized for the legacy prompt - assert len(harness.TB2_BASE_INSTRUCTIONS) <= len(harness.BASE_INSTRUCTIONS) - - -def test_tb2_instructions_require_the_signed_hash(monkeypatch): - for name, value in tb2_provider.tb2_instructions_env().items(): - monkeypatch.setenv(name, value) - assert harness.base_instructions() == harness.TB2_BASE_INSTRUCTIONS - # TB2 keeps tool names and input schemas; only descriptions change - tb2_tools = harness.miles_tools() - assert [t["function"]["name"] for t in tb2_tools] == ["terminal.exec", "submit"] - assert [t["function"]["parameters"] for t in tb2_tools] == [t["function"]["parameters"] for t in harness._MILES_TOOLS] - assert harness.dynamic_tools() is harness.TB2_DYNAMIC_TOOLS - assert [t["name"] for t in harness.dynamic_tools()] == [t["name"] for t in harness.DYNAMIC_TOOLS] - with pytest.raises(harness.CodexHarnessError, match="base instructions drifted"): - harness._validate_codex_request({"model": "m", "instructions": harness.BASE_INSTRUCTIONS}, "m") - monkeypatch.setenv(harness.TB2_INSTRUCTIONS_SHA_ENV, "0" * 64) - with pytest.raises(harness.CodexHarnessError, match="signed TB2"): - harness.base_instructions() - monkeypatch.setenv(harness.INSTRUCTIONS_FAMILY_ENV, "ctf2") - with pytest.raises(harness.CodexHarnessError, match="not a signed"): - harness.base_instructions() - - -def test_tb2_lease_selects_tb2_instructions_and_keeps_fault_turn_budget(tmp_path): - provider = _provider(tmp_path, faults=tb2_provider.parse_faults("max_turns:traj-9=3")) - - async def go(): - a = await provider.acquire("fix-git", "traj-1") - b = await provider.acquire("fix-git", "traj-9") - envs = (dict(a.worker_env), dict(b.worker_env)) - await a.destroy(); await b.destroy() - return envs - - a, b = _run(go()) - assert a == tb2_provider.tb2_instructions_env() - assert a[harness.INSTRUCTIONS_FAMILY_ENV] == "tb2" - assert b == {**a, tb2_provider.TURN_BUDGET_ENV: "3"} - - -def test_preflight_task_prompts_catches_the_stage2_dataset(tmp_path): - tasks_dir = _make_task(tmp_path / "tb2") - data = tmp_path / "smoke.jsonl" - data.write_text(json.dumps({"prompt": SMOKE6_PROMPT, "metadata": {"task_id": "fix-git"}}) + "\n") - with pytest.raises(tb2_provider.TaskPromptPreflightError, match="fix-git"): - tb2_provider.preflight_task_prompts(data, tasks_dir) - (tasks_dir / "fix-git" / "instruction.md").write_text("Merge my lost changes into master.\n") - assert tb2_provider.preflight_task_prompts(data, tasks_dir) == [("fix-git", "Merge my lost changes into master.")] - data.write_text(json.dumps({"prompt": SMOKE6_PROMPT, "metadata": {"task_id": "fix-git", "prompt": str(SMOKE6_PROMPT)}}) + "\n") - with pytest.raises(tb2_provider.TaskPromptPreflightError, match="stringified"): - tb2_provider.preflight_task_prompts(data, tasks_dir) - - -def test_harness_preflight_task_prompt_hook(tmp_path): - from yeto.rl.harness.codex import preflight as pf - - tasks_dir = _make_task(tmp_path / "tb2") - data = tmp_path / "smoke.jsonl" - data.write_text(json.dumps({"prompt": SMOKE6_PROMPT, "metadata": {"task_id": "fix-git"}}) + "\n") - provider = SimpleNamespace(tasks_dir=tasks_dir) - with pytest.raises(pf.PreflightError): - pf.assert_task_prompts(SimpleNamespace(prompt_data=str(data)), provider) - pf.assert_task_prompts(SimpleNamespace(prompt_data=str(tmp_path / "absent.jsonl")), provider) # not readable here - (tasks_dir / "fix-git" / "instruction.md").write_text("x\n") - pf.assert_task_prompts(SimpleNamespace(prompt_data=str(data)), provider) - - -def test_last_completion_and_end_reason_reach_the_metrics(monkeypatch): - metrics = harness.legacy.AgentMetrics() - choice = {"finish_reason": "stop", "message": {"role": "assistant", "content": "I will run ls." * 30, - "reasoning_content": "think", "tool_calls": [{"id": "c", "function": {"name": "terminal.exec", "arguments": "{}"}}]}} - harness._note_last_completion(metrics, choice, {"usage": {"completion_tokens": 77}}) - metrics.end_reason = "CodexModelFailure: DSV4 mixed prose with the required tool call" - d = adapter._metrics_dict(metrics) - assert d["end_reason"].startswith("CodexModelFailure") and d["last_completion"]["tool_calls"] == 1 - assert d["last_completion"]["content_chars"] == len("I will run ls." * 30) - assert len(d["last_completion"]["content_head"]) == harness.LAST_COMPLETION_HEAD_CHARS - assert d["last_completion"]["completion_tokens"] == 77 - assert "end_reason" not in adapter._metrics_dict(harness.legacy.AgentMetrics()) - - -def test_verifier_log_tail_reaches_the_trajectory_metadata_and_tape(monkeypatch, tmp_path): - from yeto.rl.adapters.miles import rollout_meta_hook as hook - - provider = _provider(tmp_path) - _configure(monkeypatch, provider) - result = _run(subprocess_agent.run("http://miles", "p", {}, _metadata(["echo nope > fixed.txt"]))) - assert tbench_outcome.verified_outcome(result)[1] == 0.0 - assert "YETO_TB2_REWARD=0" in result["verifier_log"] - diag = hook.trajectory_diagnostics(result) - assert diag["verifier_log"] == result["verifier_log"] and diag["testsh_rc"] == 1 - # S17 M1: observe-only timing line (verifier wall time, agent phase times, per-turn shapes) - timing = json.loads(result["verifier_log"].rsplit("YETO_TIMING ", 1)[1]) - assert timing["verify_s"] >= 0 - long = "x" * 5000 + "TAIL" - excerpt = tb2_provider.verifier_log_excerpt(long) - assert len(excerpt) == tb2_provider.VERIFIER_LOG_CHARS and excerpt.endswith("TAIL") - - -# --- S17 G3: tests/ larger than Modal's 64 KiB exec-argv limit --------------------- - -class _ArgvLimited: - """Wraps a sandbox handle and refuses any command longer than Modal's ARG_MAX.""" - - def __init__(self, inner): - self.inner = inner - self.lengths: list[int] = [] - - def exec(self, command, *, timeout_s, workdir=None): - self.lengths.append(len(command)) - if len(command) > 65536: - raise RuntimeError("Total length of CMD arguments cannot exceed 65536 bytes (ARG_MAX)") - return self.inner.exec(command, timeout_s=timeout_s, workdir=workdir) - - -@pytest.mark.parametrize("big", [False, True]) -def test_verifier_stages_large_tests_dir(tmp_path, big): - import os as _os - - tasks_dir = _make_task(tmp_path / "tasks") - if big: # ~200 KB of incompressible data -> ~270 KB base64 - (tasks_dir / "fix-git" / "tests" / "blob.bin").write_bytes(_os.urandom(200_000)) - task = tb2_provider.resolve_task("fix-git", tasks_dir) - stage = tb2_provider.verifier_stage_commands(task) - assert bool(stage) == big - backend = tb2_provider.LocalProcessBackend(tmp_path / "sandboxes") - handle = _ArgvLimited(backend.create(task, "t1")) - handle.exec("echo ok > fixed.txt", timeout_s=30) - verdict = _run(tb2_provider.Tb2Verifier(handle, task).evaluate("e")) - assert verdict["passed"] is True and verdict["testsh_rc"] == 0 - assert max(handle.lengths) <= 65536 - if big: - assert len(handle.lengths) >= 1 + len(stage) - - -def test_reward_env_run_judge_stages_large_tests(tmp_path): - import os as _os - - from yeto.rl.harness.reward_env import benchmark as bm - from yeto.rl.harness.reward_env import tb2 as rt - - tasks_dir = _make_task(tmp_path / "tasks") - (tasks_dir / "fix-git" / "tests" / "blob.bin").write_bytes(_os.urandom(120_000)) - adapter = rt.Tb2Benchmark(tasks_dir, version="tb2@test") - assert adapter.judge_setup_commands("fix-git") - task = tb2_provider.resolve_task("fix-git", tasks_dir) - handle = _ArgvLimited(tb2_provider.LocalProcessBackend(tmp_path / "sb").create(task, "t1")) - handle.exec("echo ok > fixed.txt", timeout_s=30) - result = bm.run_judge(adapter, "fix-git", handle, prebaked=False) - assert result.passed and not result.infra_error and max(handle.lengths) <= 65536 - -def test_modal_sandbox_exec_reads_bytes_and_decodes_leniently(): - calls = {} - - class _Proc: - stdout = SimpleNamespace(read=lambda: b"ok \xf0\x9f") # cut mid UTF-8 sequence - stderr = SimpleNamespace(read=lambda: b"") - - @staticmethod - def wait(): - return 0 - - class _Sandbox: - object_id = "sb" - - def exec(self, *args, **kwargs): - calls["kwargs"] = kwargs - return _Proc() - - result = tb2_provider.ModalSandbox(_Sandbox(), "/app").exec("echo", timeout_s=5) - assert calls["kwargs"]["text"] is False - assert result.output.startswith("ok ") and result.exit_code == 0 diff --git a/tests/test_import_boundaries.py b/tests/test_import_boundaries.py index 7994a6840..2a3603f8c 100644 --- a/tests/test_import_boundaries.py +++ b/tests/test_import_boundaries.py @@ -30,7 +30,7 @@ ALLOWLIST = REPO / "tests" / "import_boundary_allowlist.txt" README = "yeto/rl/engine/README.md" # Upper bound of allowlist entries; may only go down (D4). -ALLOWLIST_CAP = 8 # rl-code-quality-cleanup 1.10: legacy engine and launcher->overlay gone +ALLOWLIST_CAP = 5 # S26 E0.5: Codex harness removed; rl-code-quality-cleanup 1.10: legacy engine and launcher->overlay gone FRAMEWORKS = ("miles", "miles_plugins", "megatron", "sglang", "verl", "vllm", "torch_npu") ADAPTERS = ("yeto.rl.adapters",) diff --git a/tests/test_qwen3_cyber_profile.py b/tests/test_qwen3_cyber_profile.py index 1fa97fd7f..becdff303 100644 --- a/tests/test_qwen3_cyber_profile.py +++ b/tests/test_qwen3_cyber_profile.py @@ -8,11 +8,6 @@ from __future__ import annotations -import os -import subprocess -import sys -from pathlib import Path - import pytest from yeto.rl import codex_backend as cb @@ -25,8 +20,6 @@ stock_codex_reasoning_markers, validate_stock_codex_fields, ) -from yeto.rl.harness.codex import codex_openenv_agent_function as adapter -from yeto.rl.harness.codex import pins PROFILE = "qwen3_30b_a3b" @@ -37,17 +30,7 @@ } -def test_profile_is_allowlisted_and_resolvable(): - assert PROFILE in pins.OPENENV_BACKEND_PROFILES - assert adapter.profile_identity(PROFILE) == ( - QWEN3_30B_A3B_MODEL, - QWEN3_30B_A3B_REVISION, - ) - assert adapter.resolve_backend_profile(PROFILE, env={}) == PROFILE - assert ( - adapter.resolve_backend_profile(env={adapter.RUNTIME_PROFILE_ENV: PROFILE}) - == PROFILE - ) +def test_profile_chat_template_is_its_name(): # every runtime profile's chat template equals its profile name assert stock_codex_backend_contract(PROFILE, 1)["chat_template"] == PROFILE @@ -99,36 +82,6 @@ def test_validate_stock_codex_fields_accepts_then_rejects_drift(): _validate_qwen3(rollout_model="Qwen/Qwen3-32B") -def _harness_tool_output_bytes(env_extra: dict[str, str]) -> int: - """Resolve MAX_TOOL_OUTPUT_BYTES in a child interpreter (it is bound at - import time from the profile named by YETO_CODEX_CHAT_TEMPLATE).""" - repo = Path(__file__).resolve().parents[1] - env = dict(os.environ) - env.pop("YETO_CODEX_CHAT_TEMPLATE", None) - env.update(env_extra) - env["PYTHONPATH"] = str(repo) - out = subprocess.run( - [ - sys.executable, - "-c", - "from yeto.rl.harness.codex import codex_harness_agent as h;" - " print(h.MAX_TOOL_OUTPUT_BYTES)", - ], - env=env, - capture_output=True, - text=True, - check=True, - ) - return int(out.stdout.strip()) - - -def test_tool_output_budget_is_profile_driven(): - assert _harness_tool_output_bytes({"YETO_CODEX_CHAT_TEMPLATE": PROFILE}) == 8_192 - # profiles without a declaration keep the signed 32 KiB default - assert _harness_tool_output_bytes({"YETO_CODEX_CHAT_TEMPLATE": "qwen35_08b"}) == 32_768 - assert _harness_tool_output_bytes({}) == 32_768 - - def test_existing_profiles_are_unchanged(): for name in ("qwen35_08b", "qwen35", "qwen38_next", "qwen38_next_4layer"): profile = cb.stock_codex_backend_profile(name) diff --git a/tests/test_reasoning_loss_mask.py b/tests/test_reasoning_loss_mask.py index 9069d99c1..a2ae242f8 100644 --- a/tests/test_reasoning_loss_mask.py +++ b/tests/test_reasoning_loss_mask.py @@ -20,7 +20,6 @@ from yeto.rl import codex_backend as cb from yeto.rl.harness import reasoning_loss_mask as rlm -from yeto.rl.harness.codex.alignment import assert_sample_alignment OPEN, CLOSE = 900, 901 IM_START, ASSISTANT, NL, IM_END, TOOL = 10, 11, 12, 13, 14 @@ -113,7 +112,6 @@ def test_masks_reasoning_keeps_close_answers_and_tool(): assert changed == 3 assert s.metadata == {"k": "v", "reasoning_in_loss": False, "reasoning_tokens_masked": 3} assert len(s.rollout_log_probs) == s.response_length # nothing else changed - assert_sample_alignment(s) def test_generated_open_marker_is_masked_and_truncation_masks_to_end(): @@ -216,25 +214,7 @@ def test_miles_calls_hook_before_returning_train_samples(rel): assert "_convert_samples_to_train_data" in executor or "convert_samples_to_train_data" in executor -# --- flag plumbing: CLI -> launcher -> learner argv; default argv unchanged ---- -from test_rl_launcher_codex_bundle import bundle # noqa: E402,F401 (pytest fixture) - -def test_launcher_forwards_flag_only_when_set(bundle): - import test_rl_launcher_codex_bundle as tb # noqa: PLC0415 - from yeto.gpu_spec import parse_gpu_spec - from yeto.launcher import _prepare_rl_args, make_miles_island_task - - args = tb._codex_args() - spec = parse_gpu_spec(args.gpu)[0] - default_run = make_miles_island_task(args, spec, 0, 1, "127.0.0.1:29400").run - assert "--codex-exclude-reasoning-from-loss" not in default_run - - args.codex_exclude_reasoning_from_loss = True - _prepare_rl_args(args) - run = make_miles_island_task(args, spec, 0, 1, "127.0.0.1:29400").run - assert " --codex-exclude-reasoning-from-loss" in run - assert run.replace(" --codex-exclude-reasoning-from-loss", "") == default_run - +# --- flag plumbing: CLI -> launcher ---------------------------------------------- def test_launcher_rejects_flag_without_signed_codex_agent(): from test_rl_engine_selection import _cli # noqa: PLC0415 diff --git a/tests/test_reward_env.py b/tests/test_reward_env.py index 345b89d01..29fe7b07f 100644 --- a/tests/test_reward_env.py +++ b/tests/test_reward_env.py @@ -1,367 +1,19 @@ -"""CPU tests for rl-agentic-reward-env (benchmark-neutral reward sandbox, TB2 first).""" +"""CPU tests for the benchmark-neutral reward-env helpers (rl-agentic-reward-env). -from __future__ import annotations +The TB2 adapter and its prebake / judge tests left with the self-written Codex +harness (S26 E0.5). +""" -import json -import sys -import types -from pathlib import Path -from types import SimpleNamespace +from __future__ import annotations import pytest from yeto.rl.harness.reward_env import benchmark as bm -from yeto.rl.harness.reward_env import tb2 as rt - -UVX_TEST_SH = """#!/bin/bash -# Install curl -apt-get update -apt-get install -y curl - -curl -LsSf https://astral.sh/uv/0.9.5/install.sh | sh -source $HOME/.local/bin/env -if [ "$PWD" = "/" ]; then - echo "Error: No working directory set." - exit 1 -fi -uvx \\ - -p 3.13 \\ - -w pytest==8.4.1 \\ - -w pytest-json-ctrf==0.3.5 \\ - pytest --ctrf /logs/verifier/ctrf.json /tests/test_outputs.py -rA -if [ $? -eq 0 ]; then - echo 1 > /logs/verifier/reward.txt -else - echo 0 > /logs/verifier/reward.txt -fi -""" - - -def make_task(root: Path, task_id: str, test_sh: str = UVX_TEST_SH, image: str = "org/img:1", - difficulty: str = "medium") -> Path: - d = root / task_id - (d / "tests").mkdir(parents=True) - (d / "environment").mkdir() - (d / "task.toml").write_text( - f'[metadata]\ndifficulty = "{difficulty}"\n' - f'[environment]\ndocker_image = "{image}"\ncpus = 2\nmemory_mb = 4096\n' - "[agent]\ntimeout_sec = 600\n[verifier]\ntimeout_sec = 300\n" - ) - (d / "tests" / "test.sh").write_text(test_sh) - (d / "environment" / "Dockerfile").write_text("FROM x\nWORKDIR /app\n") - (d / "instruction.md").write_text(f"do {task_id}\n") - return d - - -# --- prebake plan extraction --------------------------------------------------- - -def test_uvx_plan_is_setup_plus_warmup(): - plan = rt.prebake_from_test_sh(UVX_TEST_SH, "img") - assert plan.complete - assert plan.commands == ( - "apt-get update", - "apt-get install -y curl", - "curl -LsSf https://astral.sh/uv/0.9.5/install.sh | sh", - "source $HOME/.local/bin/env", - "uvx -p 3.13 -w pytest==8.4.1 -w pytest-json-ctrf==0.3.5 pytest --version", - ) - script = rt.prebake_script(plan) - assert script.startswith("set -e\n") and script.rstrip().endswith("pytest --version") - - -def test_pip_plan(): - sh = 'pip install pytest==8.4.1 pytest-json-ctrf==0.3.5 --break-system-packages\npython -m pytest /tests -rA\n' - plan = rt.prebake_from_test_sh(sh, "img") - assert plan.complete and plan.commands == ("pip install pytest==8.4.1 pytest-json-ctrf==0.3.5 --break-system-packages",) - - -def test_uv_run_is_test_line_and_venv_setup_kept(): - sh = "uv venv -p 3.12 .tb\nsource .tb/bin/activate\nuv pip install pytest==8.4.1\nuv run pytest /tests -rA\n" - plan = rt.prebake_from_test_sh(sh, "img") - assert plan.complete and plan.commands[-1] == "uv pip install pytest==8.4.1" - - -def test_task_specific_lines_stay_at_judge_time_and_vars_block_warmup(): - sh = ("apt-get update\nX='abc'\ncurl -L -o w.pt https://h/w.pt\ncp /tests/a .\n" - 'uvx -p 3.11 -w "git+https://g@${X}" pytest /tests\n') - plan = rt.prebake_from_test_sh(sh, "img") - assert plan.commands == ("apt-get update",) - assert not plan.complete - assert "curl -L -o w.pt https://h/w.pt" in plan.skipped and "cp /tests/a ." in plan.skipped - assert any(s.startswith("uvx ") for s in plan.skipped) - - -def test_uvx_value_flags(): - line = "uvx --index https://x/cpu --index-strategy unsafe-best-match -p 3.11 -w torch==2.5.1 pytest -q" - assert rt._uvx_warmup(line).endswith("-w torch==2.5.1 pytest --version") - - -def test_no_test_line_means_no_guessing(): - plan = rt.prebake_from_test_sh("apt-get update\nmake test\n", "img") - assert plan.commands == () and not plan.complete -def test_digest_is_content_hash(): - a = rt.prebake_from_test_sh(UVX_TEST_SH, "img") - assert a.digest() == rt.prebake_from_test_sh(UVX_TEST_SH, "img").digest() - assert a.digest() != rt.prebake_from_test_sh(UVX_TEST_SH, "img2").digest() - assert a.digest() != rt.prebake_from_test_sh(UVX_TEST_SH.replace("8.4.1", "8.4.2"), "img").digest() - - -# --- registry + adapter ---------------------------------------------------------- - -def test_registry_lazy_import_and_errors(tmp_path): - make_task(tmp_path, "t1") - adapter = bm.get_adapter("tb2", tasks_dir=tmp_path) - assert adapter.name == "tb2" and "tb2" in bm.registered() +def test_unknown_benchmark_is_refused(): with pytest.raises(KeyError, match="unknown benchmark"): bm.get_adapter("no_such_bench") - with pytest.raises(ValueError, match="already registered"): - bm.register("tb2", lambda **_: None) - - -def test_adapter_spec_and_plans(tmp_path): - make_task(tmp_path, "t1") - make_task(tmp_path, "t2", image="org/other:2") - adapter = rt.Tb2Benchmark(tmp_path) - assert adapter.task_ids() == ["t1", "t2"] - spec = adapter.task_spec("t2") - assert (spec.base_image, spec.cpus, spec.memory_mb, spec.workdir, spec.judge_timeout_s) == ("org/other:2", 2, 4096, "/app", 300.0) - assert spec.instruction.startswith("do t2") - assert set(bm.plans(adapter)) == {"t1", "t2"} - assert "test.sh" in adapter.judge_command("t1") - - -# --- judge runner ---------------------------------------------------------------- - -class FakeHandle: - def __init__(self, output, exit_code=0, timed_out=False): - self.result = SimpleNamespace(output=output, exit_code=exit_code, timed_out=timed_out) - self.commands = [] - - def exec(self, command, *, timeout_s, workdir=None): - self.commands.append((command, timeout_s)) - return self.result - - -@pytest.mark.parametrize("reward,passed", [("1", True), ("0", False), ("", False)]) -def test_run_judge(tmp_path, reward, passed): - make_task(tmp_path, "t1") - adapter = rt.Tb2Benchmark(tmp_path) - handle = FakeHandle(f"pytest ok\nYETO_TB2_TESTSH_RC=0\nYETO_TB2_REWARD={reward}\n") - ticks = iter([10.0, 12.5]) - result = bm.run_judge(adapter, "t1", handle, prebaked=True, clock=lambda: next(ticks)) - assert (result.passed, result.reward, result.seconds, result.prebaked) == (passed, float(passed), 2.5, True) - assert result.extra["testsh_rc"] == 0 and handle.commands[0][1] == 300.0 - json.dumps(result.to_dict()) - - -# --- Modal backend (fake modal module) ------------------------------------------- - -class _FakeImage: - calls: list = [] - - @classmethod - def from_registry(cls, ref): - img = cls() - img.ops = [("from_registry", ref)] - return img - - def run_commands(self, *cmds): - self.ops.append(("run_commands", cmds)) - return self - - -@pytest.fixture -def fake_modal(monkeypatch): - created = {} - - class Sandbox: - @staticmethod - def create(*args, **kwargs): - created.update(kwargs, args=args) - return SimpleNamespace(object_id="sb-1") - - class App: - @staticmethod - def lookup(name, create_if_missing=False): - created["app_name"] = name - return SimpleNamespace(name=name) - - monkeypatch.setitem(sys.modules, "modal", types.SimpleNamespace(Image=_FakeImage, Sandbox=Sandbox, App=App)) - return created - - -@pytest.mark.parametrize("prebake", [True, False]) -def test_prebaked_backend(tmp_path, fake_modal, prebake): - from yeto.rl.harness.codex import tb2_provider as tb2 - - make_task(tmp_path, "t1") - task = tb2.resolve_task("t1", tmp_path) - from yeto.cloud import modal_reward_env - - backend = modal_reward_env.PrebakedModalSandboxBackend(prebake=prebake) - handle = backend.create(task, "traj-1") - assert fake_modal["app_name"] == "yeto-reward-env" and fake_modal["app"].name == "yeto-reward-env" - assert handle.workdir == "/app" - ops = fake_modal["image"].ops - assert ops[0] == ("from_registry", "org/img:1") - if prebake: - assert ops[1][0] == "run_commands" and len(ops[1][1]) == 1 - run = ops[1][1][0] - assert "\n" not in run and run.startswith("echo ") # one Dockerfile RUN line (S17 G3) - import base64 - - script = base64.b64decode(run.split()[1]).decode() - assert script == rt.prebake_script(rt.prebake_from_test_sh(UVX_TEST_SH, "org/img:1")) - assert "pytest --version" in script - assert fake_modal["tags"]["yeto-prebake"] == rt.prebake_from_test_sh(UVX_TEST_SH, "org/img:1").digest()[:16] - else: - assert len(ops) == 1 and fake_modal["tags"]["yeto-prebake"] == "none" - assert fake_modal["cpu"] == 2.0 and fake_modal["memory"] == 4096 - - -def test_build_never_targets_production_app(): - with pytest.raises(ValueError, match="production"): - rt.check_build_app("yeto-tbench2") - assert rt.check_build_app("yeto-reward-env-build") == "yeto-reward-env-build" - - -def test_build_tool_plan_only(tmp_path, capsys): - sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tools" / "reward_env")) - import build_images - - make_task(tmp_path, "t1") - out = tmp_path / "plan.json" - assert build_images.main(["--tasks-dir", str(tmp_path), "--out", str(out)]) == 0 - report = json.loads(out.read_text()) - assert report["total"] == 1 and report["tasks"][0]["complete"] - with pytest.raises(SystemExit): - build_images.main(["--tasks-dir", str(tmp_path), "--build"]) # no --app - - -# --- hold-out list (WP3 D6b) + metadata (D6a) ------------------------------------ - -def _tb2_pool(root): - for i in range(4): - make_task(root, f"e{i}", difficulty="easy") - for i in range(20): - make_task(root, f"m{i:02d}", difficulty="medium") - for i in range(12): - make_task(root, f"h{i:02d}", difficulty="hard") - return rt.Tb2Benchmark(root, version="tb2@test") - - -def test_tb2_metadata_shape(tmp_path): - make_task(tmp_path, "t1", difficulty="hard") - meta = rt.Tb2Benchmark(tmp_path, version="tb2@abc1234").task_spec("t1").metadata() - assert meta == {"task_id": "t1", "benchmark": "tb2", "benchmark_version": "tb2@abc1234", - "difficulty": "hard", "difficulty_source": "tb2-task.toml", "eval_bucket": "tb2-hard"} - - -def test_tb2_holdout_stratified_and_reproducible(tmp_path): - adapter = _tb2_pool(tmp_path) - h = rt.build_holdout(adapter) - assert h["schema"] == "yeto-eval-holdout/1" and h["seed"] == 20261008 and h["benchmark_version"] == "tb2@test" - from collections import Counter - - assert Counter(i["eval_bucket"] for i in h["items"]) == {"tb2-easy": 2, "tb2-medium": 18, "tb2-hard": 10} - assert rt.build_holdout(adapter, task_ids=list(reversed(adapter.task_ids()))) == h - assert bm.holdout_sha256(h) == bm.holdout_sha256(rt.build_holdout(adapter)) - ids = bm.holdout_ids(h) - excluded = rt.build_holdout(adapter, exclude=ids[:1] if ids[0].startswith("m") else ids[2:3]) - assert len(excluded["items"]) == 30 - with pytest.raises(ValueError, match="quota"): - rt.build_holdout(adapter, exclude=["e0", "e1", "e2"]) - - -def test_tb2_holdout_excludes_smoke6_by_default(tmp_path): - adapter = _tb2_pool(tmp_path) - for t in rt.SMOKE6_TASK_IDS: - make_task(tmp_path, t, difficulty="medium") - adapter = rt.Tb2Benchmark(tmp_path, version="tb2@test") - h = rt.build_holdout(adapter) - ids = set(bm.holdout_ids(h)) - assert not ids & set(rt.SMOKE6_TASK_IDS) and len(ids) == 30 - assert h["excluded"] == [{"task_id": t, "reason": rt.SMOKE6_REASON} for t in sorted(rt.SMOKE6_TASK_IDS)] - assert "smoke6" in h["rule"] - none = rt.build_holdout(adapter, exclude={}) - assert "excluded" not in none and len(none["items"]) == 30 - - -def test_holdout_cli_exclusions(tmp_path): - sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tools" / "reward_env")) - import holdout - - tasks = tmp_path / "tasks" - tasks.mkdir() - _tb2_pool(tasks) - out = tmp_path / "out" - # built-in smoke-6 list, but the checkout lacks them -> fail closed - with pytest.raises(SystemExit, match="not in the TB2 checkout"): - holdout.main(["--tb2-tasks-dir", str(tasks), "--out-dir", str(out)]) - assert holdout.main(["--tb2-tasks-dir", str(tasks), "--tb2-no-exclude", "--out-dir", str(out)]) == 0 - assert "excluded" not in json.loads((out / "tb2-holdout.json").read_text()) - jl = tmp_path / "ex.jsonl" - jl.write_text(json.dumps({"metadata": {"task_id": "m00"}}) + "\n\n") - assert holdout.main(["--tb2-tasks-dir", str(tasks), "--tb2-exclude-jsonl", str(jl), "--tb2-no-unusable", - "--tb2-exclude-reason", "r", "--out-dir", str(out)]) == 0 - h = json.loads((out / "tb2-holdout.json").read_text()) - assert h["excluded"] == [{"task_id": "m00", "reason": "r"}] and "m00" not in bm.holdout_ids(h) - jl.write_text(json.dumps({"prompt": "x"}) + "\n") - with pytest.raises(SystemExit, match="metadata.task_id"): - holdout.main(["--tb2-tasks-dir", str(tasks), "--tb2-exclude-jsonl", str(jl), "--out-dir", str(out)]) - - -def test_tb2_unusable_table_shipped(): - table = rt.load_unusable() - assert len(table) == 13 and "qemu-startup" in table - assert all(r.startswith("unusable/") for r in table.values()) - assert not set(table) & set(rt.SMOKE6_TASK_IDS) - - -def test_tb2_unusable_bad_file(tmp_path): - bad = tmp_path / "u.json" - bad.write_text(json.dumps({"schema": "nope", "benchmark": "tb2", "items": []})) - with pytest.raises(ValueError, match="not a"): - rt.load_unusable(bad) - bad.write_text(json.dumps({"schema": rt.UNUSABLE_SCHEMA, "benchmark": "tb2", "items": [ - {"task_id": "a", "category": "c", "reason": "r"}, {"task_id": "a", "category": "c", "reason": "r"}]})) - with pytest.raises(ValueError, match="duplicate"): - rt.load_unusable(bad) - - -def test_holdout_cli_unusable_leaves_train_and_eval(tmp_path): - sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tools" / "reward_env")) - import holdout - - tasks = tmp_path / "tasks" - tasks.mkdir() - _tb2_pool(tasks) - for t in rt.SMOKE6_TASK_IDS: - make_task(tasks, t, difficulty="medium") - table = tmp_path / "u.json" - table.write_text(json.dumps({"schema": rt.UNUSABLE_SCHEMA, "benchmark": "tb2", "items": [ - {"task_id": "m01", "category": "timeout", "reason": "slow"}, - {"task_id": "h03", "category": "env", "reason": "404"}]})) - out = tmp_path / "out" - assert holdout.main(["--tb2-tasks-dir", str(tasks), "--tb2-unusable", str(table), "--out-dir", str(out)]) == 0 - h = json.loads((out / "tb2-holdout.json").read_text()) - tr = json.loads((out / "tb2-train.json").read_text()) - held, train = set(bm.holdout_ids(h)), {i["task_id"] for i in tr["items"]} - assert len(held) == 30 and h["rule"] == rt.HOLDOUT_RULE_UNUSABLE - assert {"m01", "h03"} <= {e["task_id"] for e in h["excluded"]} - assert not {"m01", "h03"} & (held | train) and not held & train - assert set(rt.SMOKE6_TASK_IDS) <= train # smoke-6 only leaves the eval pool - assert len(held) + len(train) + 2 == 36 + 6 - assert tr["holdout_sha256"] == bm.holdout_sha256(h) and tr["schema"] == bm.TRAIN_SPLIT_SCHEMA - assert tr["excluded"] == [{"task_id": "h03", "reason": "unusable/env: 404"}, - {"task_id": "m01", "reason": "unusable/timeout: slow"}] - # an unusable task missing from the checkout fails closed - table.write_text(json.dumps({"schema": rt.UNUSABLE_SCHEMA, "benchmark": "tb2", "items": [ - {"task_id": "zz", "category": "c", "reason": "r"}]})) - with pytest.raises(SystemExit, match="unusable tasks not in the TB2 checkout"): - holdout.main(["--tb2-tasks-dir", str(tasks), "--tb2-unusable", str(table), "--out-dir", str(out)]) def test_rows_split_and_leak_check(): @@ -375,23 +27,3 @@ def test_rows_split_and_leak_check(): bm.assert_disjoint(["a"], ["a"]) with pytest.raises(ValueError, match="duplicate"): bm.holdout_ids({"schema": bm.HOLDOUT_SCHEMA, "items": [{"task_id": "a"}, {"task_id": "a"}]}) - - -def test_run_judge_infra_error(tmp_path): - make_task(tmp_path, "t1") - - class Gone: - def exec(self, command, *, timeout_s, workdir=None): - raise RuntimeError("sandbox terminated") - - result = bm.run_judge(rt.Tb2Benchmark(tmp_path), "t1", Gone(), prebaked=False) - assert result.infra_error and not result.passed and "terminated" in result.log - - -# --- launcher recognises the new provider ----------------------------------------- - -def test_launcher_treats_prebaked_provider_as_modal(): - from yeto import launcher as L - - assert L.PREBAKED_MODAL_SANDBOX_PROVIDER in L.MODAL_SANDBOX_PROVIDERS - assert L.MODAL_SANDBOX_PROVIDER in L.MODAL_SANDBOX_PROVIDERS diff --git a/tests/test_rl_agentic_suspend.py b/tests/test_rl_agentic_suspend.py deleted file mode 100644 index 4dd1c6712..000000000 --- a/tests/test_rl_agentic_suspend.py +++ /dev/null @@ -1,491 +0,0 @@ -"""agentic-rollout-utilization stage 3 (5.1-5.3) and the Miles in-flight cut wiring (3.3). - -CPU only: fake session server (aiohttp), fake Codex client, fake executor. -No Ray, no Miles process, no GPU. -""" - -from __future__ import annotations - -import asyncio -import json -import os -from dataclasses import asdict -from types import SimpleNamespace -from typing import Any - -import aiohttp -import pytest -from aiohttp import web - -from test_codex_bridge_compaction import FakeCodex, _exec, _submit -from test_rl_miles_carry_over import Call, FakePartialEngine, Sample, Span, tok -from test_secrlenv_codex_harness import _fake_miles_sse -from yeto.rl.adapters.miles import carry_over, in_flight -from yeto.rl.adapters.miles import policy_age as miles_policy_age -from yeto.rl.engine.cut import context_problems -from yeto.rl.engine.policy_age import PolicyAgeError -from yeto.rl.engine.trajectory_timing import PhaseClock -from yeto.rl.harness.codex import codex_harness_agent as harness -from yeto.rl.harness.codex import codex_openenv_generate as wrapper -from yeto.rl.harness.codex import codex_openenv_subprocess_agent_function as trusted - -AGENT = "yeto.rl.harness.codex.codex_openenv_subprocess_agent_function.run" - - -# ------------------------------------------------------------ 5.1 the gate - - -class GatedSessionServer: - """Fake Miles session server; ``on_request(n, body)`` may return an HTTP status - to answer instead of the next completion (503 = engine aborted the turn).""" - - def __init__(self, replies, on_request=None): - self.replies = list(replies) - self.bodies: list[dict[str, Any]] = [] - self.on_request = on_request - - async def __aenter__(self): - async def chat(request: web.Request) -> web.Response: - body = await request.json() - self.bodies.append(body) - status = self.on_request(len(self.bodies), body) if self.on_request else None - if status is not None: - return web.json_response({"error": "upstream generation aborted"}, status=status) - return web.Response(body=_fake_miles_sse(self.replies.pop(0)), content_type="text/event-stream") - - app = web.Application() - app.router.add_post("/sessions/{sid}/v1/chat/completions", chat) - self.runner = web.AppRunner(app, access_log=None) - await self.runner.setup() - site = web.TCPSite(self.runner, "127.0.0.1", 0) - await site.start() - self.url = f"http://127.0.0.1:{site._server.sockets[0].getsockname()[1]}/sessions/S0" - return self - - async def __aexit__(self, *_): - await self.runner.cleanup() - - -def _gate_env(monkeypatch, gate, max_s="30"): - monkeypatch.setenv("YETO_CODEX_BACKEND_MAX_TOKENS", "512") - monkeypatch.delenv("YETO_CODEX_COMPACTION_ENABLED", raising=False) - monkeypatch.setenv(harness.SUSPEND_GATE_ENV, str(gate)) - monkeypatch.setenv(harness.SUSPEND_MAX_SECONDS_ENV, max_s) - monkeypatch.setattr(harness, "SUSPEND_POLL_SECONDS", 0.02) - - -def _bridge(url, metrics): - return harness._ResponsesBridge(url, "solve target", {"max_tokens": 512}, metrics, max_seq_len=8000) - - -def test_cutoff_during_a_tool_call_parks_the_next_model_turn_until_resume(monkeypatch, tmp_path): - gate = tmp_path / "t.gate" - _gate_env(monkeypatch, gate) - edges: list[str] = [] - harness.set_suspend_event_sink(edges.append) - - async def scenario(): - metrics = harness.legacy.AgentMetrics() - async with GatedSessionServer([_exec(1, 500), _submit(900)]) as server: - async with _bridge(server.url, metrics) as bridge, aiohttp.ClientSession() as http: - codex = FakeCodex(bridge, http) - assert (await codex.turn())[0] == 200 # model turn 1 -> tool call - gate.write_text("suspended\n") # cut-off while the tool runs - # the tool returns and its result goes into the next request; the - # bridge holds that request (no model call while suspended) - pending = asyncio.ensure_future(codex.turn('{"output":"tool result"}')) - await asyncio.sleep(0.3) - assert not pending.done() and len(server.bodies) == 1 - gate.unlink() # resume - status, resp = await pending - assert status == 200 and resp["output"][1]["name"] == "submit" - return server, metrics - - try: - server, metrics = asyncio.run(scenario()) - finally: - harness.set_suspend_event_sink(None) - # the request after resume carries the tool result written before the suspension - assert any("tool result" in json.dumps(m) for m in server.bodies[1]["messages"]) - assert metrics.suspensions == 1 and metrics.suspended_seconds >= 0.25 - assert metrics.suspend_retried_turns == 0 and metrics.suspend_expired == 0 - assert edges == ["enter", "exit"] - # parked time is not generation time - assert metrics.total_generation_time < metrics.suspended_seconds - - -def test_a_model_turn_aborted_by_the_cutoff_is_redone_after_resume(monkeypatch, tmp_path): - gate = tmp_path / "t.gate" - _gate_env(monkeypatch, gate) - - def on_request(n, _body): - if n == 2: # the cut-off: gate closed first, then the engine aborts this turn - gate.write_text("suspended\n") - asyncio.get_event_loop().call_later(0.2, gate.unlink) - return 503 - return None - - async def scenario(): - metrics = harness.legacy.AgentMetrics() - async with GatedSessionServer([_exec(1, 500), _submit(900)], on_request) as server: - async with _bridge(server.url, metrics) as bridge, aiohttp.ClientSession() as http: - codex = FakeCodex(bridge, http) - assert (await codex.turn())[0] == 200 - status, resp = await codex.turn('{"output":"r"}') - assert status == 200 and resp["output"][1]["name"] == "submit" - return server, metrics - - server, metrics = asyncio.run(scenario()) - assert len(server.bodies) == 3 and server.bodies[1] == server.bodies[2] # same turn redone - assert metrics.suspend_retried_turns == 1 and metrics.suspensions == 1 - - -def test_a_503_without_a_suspension_is_still_an_error(monkeypatch, tmp_path): - _gate_env(monkeypatch, tmp_path / "never.gate") - - async def scenario(): - metrics = harness.legacy.AgentMetrics() - async with GatedSessionServer([_exec(1, 500)], lambda n, _b: 503 if n == 2 else None) as server: - async with _bridge(server.url, metrics) as bridge, aiohttp.ClientSession() as http: - codex = FakeCodex(bridge, http) - assert (await codex.turn())[0] == 200 - status, _ = await codex.turn('{"output":"r"}') - return status, bridge, metrics - - status, bridge, metrics = asyncio.run(scenario()) - assert status == 400 and "HTTP 503" in str(bridge.fatal.result()) - assert metrics.suspensions == 0 - - -def test_a_suspension_past_the_survival_limit_discards_the_trajectory(monkeypatch, tmp_path): - gate = tmp_path / "t.gate" - _gate_env(monkeypatch, gate, max_s="0.2") - - async def scenario(): - metrics = harness.legacy.AgentMetrics() - async with GatedSessionServer([_exec(1, 500), _submit(900)]) as server: - async with _bridge(server.url, metrics) as bridge, aiohttp.ClientSession() as http: - codex = FakeCodex(bridge, http) - assert (await codex.turn())[0] == 200 - gate.write_text("suspended\n") - status, _ = await codex.turn('{"output":"r"}') - return status, bridge, metrics, server - - status, bridge, metrics, server = asyncio.run(scenario()) - assert status == 400 and isinstance(bridge.fatal.result(), harness.CodexSuspendExpired) - assert metrics.suspend_expired == 1 and len(server.bodies) == 1 # never sampled again - - -def test_codex_waits_for_a_held_request_only_when_suspension_is_on(monkeypatch, tmp_path): - monkeypatch.delenv(harness.SUSPEND_GATE_ENV, raising=False) - assert harness._suspend_settings() == [] # default command line unchanged - monkeypatch.setenv(harness.SUSPEND_GATE_ENV, str(tmp_path / "g")) - monkeypatch.setenv(harness.SUSPEND_MAX_SECONDS_ENV, "600") - assert harness._suspend_settings() == ["model_providers.miles.stream_idle_timeout_ms=1200000"] - - -def test_phase_clock_keeps_parked_time_out_of_generation(): - now = [0.0] - clock = PhaseClock(lambda: now[0]) - now[0] = 2.0 - clock.enter_tool() - now[0] = 3.0 - clock.exit_tool() - now[0] = 4.0 - clock.enter_suspend() # parked before model turn 2 - now[0] = 104.0 - clock.exit_suspend() - now[0] = 106.0 - out = clock.finish() - assert out["turn_generation_seconds"] == [2.0, 3.0] and out["turn_tool_seconds"] == [1.0] - assert out["suspended_seconds"] == 100.0 and out["worker_seconds"] == 106.0 - assert "suspended_seconds" not in PhaseClock(lambda: 0.0).finish() # unchanged when never parked - - -# ------------------------------------------------- 5.1/5.3 trusted-side hooks - - -def test_suspend_closes_the_gates_of_trajectories_in_flight_and_resume_opens_them(monkeypatch, tmp_path): - monkeypatch.setattr(trusted, "_SUSPEND_DIR", None) - monkeypatch.setattr(trusted, "_SUSPEND_STATE", {"suspended_at": None, "gates": {}}) - monkeypatch.setattr(trusted, "_INFLIGHT", {"a": {"stage": "worker"}, "b": {"stage": "verify"}, - "c": {"stage": "teardown"}}) - assert trusted._suspend_worker_env("x") == {} # no suspension in use: env unchanged - asyncio.run(trusted.resume()) # Miles calls resume at the start of every rollout - assert trusted._suspend_worker_env("x") == {harness.SUSPEND_GATE_ENV: trusted._gate_path("x")} - assert asyncio.run(trusted.suspend()) == 2 # teardown is not suspended - gates = trusted._SUSPEND_STATE["gates"] - assert set(gates) == {"a", "b"} and all(os.path.exists(p) for p in gates.values()) - # a trajectory started while suspended is parked too - env = trusted._suspend_worker_env("late") - assert os.path.exists(env[harness.SUSPEND_GATE_ENV]) - paths = list(trusted._SUSPEND_STATE["gates"].values()) - assert asyncio.run(trusted.resume()) == 3 - assert not any(os.path.exists(p) for p in paths) and trusted._SUSPEND_STATE["gates"] == {} - - -# ------------------------------------------------------- 5.2 version segments - - -def _traj(versions): - calls = [{"spans": [{"version": v}]} for v in versions] - return SimpleNamespace(metadata={}, weight_versions=calls, status="completed", loss_mask=None) - - -def test_codex_versions_are_recorded_as_segments_within_the_window(monkeypatch): - monkeypatch.setattr(wrapper, "assert_sample_alignment", lambda s: None) - ok = _traj([tok(3), tok(3), tok(4)]) - wrapper.apply_trajectory_bookkeeping(SimpleNamespace(group_index=1, index=9), [ok], - expected_version=tok(3), max_policy_age=1, current_version=tok(4)) - assert ok.status == "completed" and wrapper.POLICY_AGE_KEY not in ok.metadata - assert ok.metadata[wrapper.VERSION_SEGMENTS_KEY] == sorted({tok(3), tok(4)}) - old = _traj([tok(2), tok(4)]) - wrapper.apply_trajectory_bookkeeping(SimpleNamespace(group_index=1, index=9), [old], - expected_version=tok(2), max_policy_age=1, current_version=tok(4)) - reason = old.metadata[wrapper.INFRASTRUCTURE_KEY] - assert reason.startswith(wrapper.POLICY_AGE_EXCEEDED) and wrapper.POLICY_AGE_KEY not in old.metadata - # limit 0: drift is still a policy-identity violation (unchanged) - drift = _traj([tok(3), tok(4)]) - wrapper.apply_trajectory_bookkeeping(SimpleNamespace(group_index=1, index=9), [drift], - expected_version=tok(3)) - assert drift.metadata[wrapper.POLICY_AGE_KEY] == 1 - - -def test_window_problem_rules(): - assert wrapper.window_problem([tok(3)], tok(4), 1) is None - assert "newer" in wrapper.window_problem([tok(5)], tok(4), 1) - assert "unparsable" in wrapper.window_problem(["default"], tok(4), 1) - assert "no generation" in wrapper.window_problem([], tok(4), 1) - - -# ----------------------------------------------------- policy age (launcher) - - -def test_miles_stage_three_accepts_the_suspendable_codex_agent_only(): - assert miles_policy_age.SUPPORT.stage == 3 - miles_policy_age.check_task(1, custom_generate=miles_policy_age.AGENTIC_GENERATE, custom_agent=AGENT) - with pytest.raises(PolicyAgeError, match="suspended between turns"): - miles_policy_age.check_task(1, custom_generate=miles_policy_age.AGENTIC_GENERATE, - custom_agent="some.other.agent.run") - with pytest.raises(PolicyAgeError, match="stage 3"): - miles_policy_age.check_task(1, custom_generate="miles.rollout.generate_hub.multi_turn.generate") - assert miles_policy_age.policy_age_argv(1, agentic=True) == ( - "--agentic-suspend-between-turns", "--agentic-suspend-max-rounds", "1") - assert miles_policy_age.policy_age_argv(0, agentic=True) == () - - -# ------------------------------------------- carry-over metrics (5.3 / 5.5) - - -def test_agentic_tool_tokens_are_not_cross_version_and_ratios_are_summarised(): - s = Sample(index=0, group_index=0) - FakePartialEngine().generate(s, 3, 2, finish=False) - s.tokens = s.tokens + [9, 9] # tool output (no version span) - s.response_length += 2 - s.rollout_log_probs = s.rollout_log_probs + [0.0, 0.0] - FakePartialEngine().generate(s, 4, 1, finish=True) - s.loss_mask = [1, 1, 0, 0, 1] - assert carry_over.response_token_versions(s) == [3, 3, None, None, 4] - assert carry_over.trained_token_versions(s, 4) == [3, 3, 4, 4, 4] - out = carry_over.estimate_cross_version_truncation( - SimpleNamespace(tis_clip_low=0.0, tis_clip=2.0), [s], 4, lambda _s: [-0.5, 0.5, 0.0, 0.0, -0.5]) - assert out["cross_version_scored_tokens"] == 2 and out["cross_version_unscored_samples"] == 0 - assert out["cross_version_truncated_fraction"] == 0.5 # ratios 1 and e (>2: truncated) - assert out["cross_version_ratio_max"] == pytest.approx(2.718281828, rel=1e-6) - - -def test_suspend_fields_report_the_fork_stats_and_the_survival_cost(): - args = SimpleNamespace(rollout_suspend_stats={"suspended_groups": 3, "over_age_cancelled_groups": 1}, - rollout_resume_stats={"resumed_groups": 2, "resumed_done_groups": 0}) - expired = SimpleNamespace(metadata={"tbench_infrastructure_error": "CodexSuspendExpired: x", - "agent_metrics": {"suspensions": 1, "suspended_seconds": 600.0}}) - parked = SimpleNamespace(metadata={"agent_metrics": {"suspensions": 1, "suspended_seconds": 40.5, - "suspend_retried_turns": 1}}) - plain = SimpleNamespace(metadata={}) - out = carry_over.suspend_fields(args, [[expired, parked], [plain]]) - assert out["suspended_groups"] == 3 and out["resumed_groups"] == 2 - assert out["suspend_expired_trajectories"] == 1 and out["suspended_trajectories"] == 2 - assert out["suspended_env_seconds"] == 640.5 and out["suspend_retried_turns"] == 1 - assert args.rollout_suspend_stats is None # consumed - assert carry_over.suspend_fields(args, []) == {} - - -# ----------------------------------------- 3.3 Miles in-flight cut wiring - - -class MilesLikeSample(Sample): - def to_dict(self): - d = asdict(self) - d["status"] = self.status.value - return d - - -def _buffered_group(gi, version=3): - group = [MilesLikeSample(index=gi * 10 + i, group_index=gi) for i in range(2)] - for s in group: - FakePartialEngine().generate(s, version, 3, finish=False) - s.metadata["task_id"] = f"task-{gi}" - return group - - -def test_export_and_restore_buffered_groups_and_agentic_references(monkeypatch): - executor = SimpleNamespace(data_source=SimpleNamespace(buffer=[_buffered_group(1), _buffered_group(2)])) - monkeypatch.setattr(in_flight, "_codex_suspension", - lambda: ({"t7": "/tmp/t7.gate"}, {"t7": {"stage": "worker"}})) - exported = in_flight.export_in_flight(executor) - assert exported["buffer_groups"] == 2 and len(exported["entries"]) == 5 - assert in_flight.check_export(exported) == [] - agentic = [e for e in exported["entries"] if e.get("session_ref")] - assert agentic[0]["session_ref"] == "codex-suspended:t7" and agentic[0]["stage"] == "worker" - # the cut accepts the buffer because the in-flight section carries it - base = dict(cut_id="r000004-x", progress=SimpleNamespace(problems=lambda: []), - algorithm=SimpleNamespace(algorithm_spec_sha256="a" * 64), - data={"sample_offset": 0, "epoch_id": 0, "sample_group_index": 3, "sample_index": 6, - "buffer_length": 2}, - outer={"settled": True}, runtime={}) - entries = tuple(exported["entries"]) - ok_ledger = {"carried_over": 5, "ready_unconsumed": 0, "max_policy_age": 1} - assert not [p for p in context_problems(**base, ledger=ok_ledger, in_flight=entries, max_policy_age=1) - if p.startswith(("data", "in_flight", "ledger"))] - zero = context_problems(**base, ledger={"carried_over": 0, "ready_unconsumed": 0}) - assert any("not carried by a cut" in p for p in zero) # limit 0: unchanged refusal - short = context_problems(**base, ledger=ok_ledger, in_flight=entries[:2], max_policy_age=1) - assert any("carries 1" in p for p in short) - - # restore at version 4 with limit 1: group 2 already trained, agentic lost - target = SimpleNamespace(data_source=SimpleNamespace(buffer=[], args=None)) - target.data_source.add_samples = lambda groups: target.data_source.buffer.extend(groups) - report = in_flight.import_in_flight(target, exported["entries"], max_policy_age=1, current_version=4, - completed_group_ids=["g2"], sample_from_dict=dict) - assert report["resumed_groups"] == 1 and report["resumed"] == 2 - assert report["resumed_group_ids"] == ["g1"] - assert report["reasons"] == {"group_already_completed": 2, in_flight.AGENTIC_SESSION_LOST: 1} - restored = target.data_source.buffer[0] - assert [m["index"] for m in restored] == [10, 11] and restored[0]["metadata"]["task_id"] == "task-1" - # too old at version 5 (limit 1): discarded - target.data_source.buffer.clear() - report = in_flight.import_in_flight(target, exported["entries"], max_policy_age=1, current_version=5, - sample_from_dict=dict) - assert report["resumed_groups"] == 0 and report["reasons"]["policy_age_exceeded"] == 4 - assert report["resumed_group_ids"] == [] - # restored under limit 0: everything discarded and reported - report = in_flight.import_in_flight(target, exported["entries"], max_policy_age=0, current_version=4, - sample_from_dict=dict) - assert report["discarded"] == 5 and target.data_source.buffer == [] - - -def test_export_refuses_a_buffered_token_of_unknown_version(): - group = _buffered_group(1) - group[0].weight_versions.append(Call([Span("default", 0, len(group[0].tokens))])) - group[0].tokens = group[0].tokens + [1] - group[0].response_length += 1 - group[0].rollout_log_probs.append(-0.1) - group[0].weight_versions = [Call([Span("default", len(group[0].tokens) - 4, len(group[0].tokens))])] - with pytest.raises(in_flight.InFlightExportError, match="unknown"): - in_flight.buffered_entries([group]) - - -def test_cut_source_writes_the_in_flight_section_under_a_limit(): - from yeto.rl.adapters.miles.rebuild_wiring import CutSource - - exported = {"entries": in_flight.buffered_entries([_buffered_group(1)]), "buffer_groups": 1} - state = SimpleNamespace(policy_version=4, policy_tensor_hash=lambda: "h" * 64) - driver = SimpleNamespace(published_state=state, published_version=4, local_step=4, - at_safe_point=True, expected_token=tok(4), - profile=SimpleNamespace(max_policy_age=1)) - rollout = SimpleNamespace(data_cursor=lambda: {"sample_offset": 0, "epoch_id": 0, - "sample_group_index": 2, "sample_index": 4}, - export_in_flight=lambda: exported) - ledger = SimpleNamespace(cut_summary=lambda: {"carried_over": 0, "ready_unconsumed": 0, - "engine_buffer_length": 1}) - source = CutSource.__new__(CutSource) - source.driver_ref, source.rollout, source.ledger = driver, rollout, ledger - source.cut_root, source.backend_fingerprint, source.global_batch_size = "/tmp", "f", 4 - source.identity, source.shared_filesystem = SimpleNamespace(), True - context = source.context("r000004-x") - assert len(context.in_flight) == 2 and context.data["buffer_length"] == 1 - assert context.ledger["carried_over"] == 2 and context.ledger["max_policy_age"] == 1 - driver.profile.max_policy_age = 0 - assert source.context("r000004-y").in_flight == () # limit 0: section absent - - -def test_translate_emits_the_suspend_flags_for_an_agentic_run(): - import dataclasses - - from tests.test_rl_miles_adapter_config import make_config # noqa: PLC0415 - from yeto.rl.adapters.miles.config import translate_run_config - from yeto.rl.engine.algorithm import AlgorithmSpec, CorrectionSpec, ExecutionSpec - - tolerant = AlgorithmSpec(execution=ExecutionSpec(max_policy_staleness=1), - correction=CorrectionSpec(method="tis", tis_clip=2.0, tis_clip_low=0.0)) - config = make_config() - agent = dataclasses.replace(config.agent, custom_generate_function_path=miles_policy_age.AGENTIC_GENERATE, - custom_agent_function_path=AGENT, use_session_server=True) - config = dataclasses.replace(config, agent=agent) - zero = translate_run_config(config, tolerant).argv - one = translate_run_config(config, tolerant, max_policy_age=1).argv - assert [a for a in one if a not in zero] == ["--agentic-suspend-between-turns", "--agentic-suspend-max-rounds"] - assert "--partial-rollout" not in one - i = one.index("--agentic-suspend-max-rounds") - assert one[i + 1] == "1" - - -def test_estimate_reports_why_samples_are_unscored_and_skips_unknown_trained_tokens(): - s = Sample(index=0, group_index=0) - FakePartialEngine().generate(s, 3, 2, finish=False) - FakePartialEngine().generate(s, 4, 1, finish=True) - s.weight_versions = s.weight_versions[1:] # the first turn's span is missing - s.weight_versions.insert(0, Call([Span(tok(3), len(s.tokens) - 3, len(s.tokens) - 2)])) - args = SimpleNamespace(tis_clip_low=0.0, tis_clip=2.0) - out = carry_over.estimate_cross_version_truncation(args, [s], 4, lambda _s: [-0.5, -0.5, -0.5]) - assert out["cross_version_unknown_version_tokens"] == 1 and out["cross_version_scored_tokens"] == 1 - - def failing(_s): - failing.last_error = "ValueError: alignment" - return None - - out = carry_over.estimate_cross_version_truncation(args, [s], 4, failing) - assert out["cross_version_unscored_samples"] == 1 - assert out["cross_version_unscored_reasons"] == {"ValueError: alignment": 1} - - -def _fake_prefill_module(monkeypatch): - import sys - import types - - mod = types.ModuleType("miles.rollout.generate_utils.prefill_logprobs") - mod._build_prefill_scoring_payload = lambda args, sample, params: {"input_ids": list(sample.tokens)} - mod._extract_response_logprobs = lambda sample, meta: meta["input_token_logprobs"] - for name in ("miles", "miles.rollout", "miles.rollout.generate_utils"): - monkeypatch.setitem(sys.modules, name, sys.modules.get(name) or types.ModuleType(name)) - monkeypatch.setitem(sys.modules, mod.__name__, mod) - - -def test_scorer_sends_routing_key_under_manual_router_policy(monkeypatch): - """S18 ARU-3 M1: the session server sets the router policy to ``manual``, - which rejects /generate without X-SMG-Routing-Key (one router client error - per unscored sample). The scorer must send the key.""" - import io - - _fake_prefill_module(monkeypatch) - seen: list[dict[str, str]] = [] - - def fake_urlopen(request, timeout): - seen.append({k.lower(): v for k, v in request.header_items()}) - return io.BytesIO(json.dumps({"meta_info": {"input_token_logprobs": [-0.1, -0.2]}}).encode()) - - monkeypatch.setattr("urllib.request.urlopen", fake_urlopen) - args = SimpleNamespace(sglang_router_ip="r", sglang_router_port=1, sglang_router_policy="manual") - score = carry_over.router_prefill_scorer(args) - - keyed = SimpleNamespace(tokens=[1, 2, 3], response_length=2, routing_key="sess-7") - assert score(keyed) == [-0.1, -0.2] and score.last_error is None - assert seen[-1]["x-smg-routing-key"] == "sess-7" - - unkeyed = SimpleNamespace(tokens=[1, 2, 3], response_length=2, routing_key=None) - assert score(unkeyed) == [-0.1, -0.2] - assert seen[-1]["x-smg-routing-key"] == carry_over.SCORING_ROUTING_KEY - - args.sglang_router_policy = "cache_aware" - assert score(unkeyed) == [-0.1, -0.2] - assert "x-smg-routing-key" not in seen[-1] diff --git a/tests/test_rl_agentic_tool_call_args.py b/tests/test_rl_agentic_tool_call_args.py index 54b07e6d8..93a1e0abf 100644 --- a/tests/test_rl_agentic_tool_call_args.py +++ b/tests/test_rl_agentic_tool_call_args.py @@ -79,7 +79,7 @@ def upstream_add_arguments(monkeypatch): def _agent_argv() -> list[str]: c = sub(make_config(), "agent", custom_generate_function_path=cfg.AGENTIC_TOOL_CALL_GENERATE, - custom_agent_function_path="yeto.rl.harness.codex.codex_openenv_subprocess_agent_function.run", + custom_agent_function_path="yeto.rl.adapters.miles.harbor_agent.run", agent_max_seq_len=8192, use_session_server=True, tito_model="qwen35") return list(cfg.translate_run_config(c, AlgorithmSpec(), tito_parser_resolver=lambda m: ("qwen3", "qwen")).argv) @@ -93,7 +93,7 @@ def test_pinned_add_arguments_recognises_the_ir1_flags(upstream_add_arguments): agent_flags = argv[i:i + 2] + argv[j:j + 2] ns, unknown = parser.parse_known_args(agent_flags) assert unknown == [] - assert ns.custom_agent_function_path == "yeto.rl.harness.codex.codex_openenv_subprocess_agent_function.run" + assert ns.custom_agent_function_path == "yeto.rl.adapters.miles.harbor_agent.run" assert ns.max_seq_len == 8192 # The launcher-level spelling (--agent-max-seq-len) is NOT an upstream flag: # config.py must keep mapping agent_max_seq_len to --max-seq-len. diff --git a/tests/test_rl_algorithm_capabilities.py b/tests/test_rl_algorithm_capabilities.py index 84a6e29b7..df4de8b3c 100644 --- a/tests/test_rl_algorithm_capabilities.py +++ b/tests/test_rl_algorithm_capabilities.py @@ -136,10 +136,6 @@ def test_old_declaration_reads_as_r0_mechanisms(): f"dynamic_sampling_filters:{alg.BOUNDED_NONZERO_STD_FILTER}", f"dynamic_sampling_filters:{alg.STOCK_NONZERO_STD_FILTER}", } -# Moved from the removed legacy engine (P0-1.7b.4-Q1); declared with the R0 -# filters, it needs SecRLEnv episode evidence, so no CPU candidate exercises it. -SECRLENV_FILTER_P0 = {f"dynamic_sampling_filters:{alg.SECRLENV_GROUP_CHECK_FILTER}", - f"dynamic_sampling_filters:{alg.TBENCH_GROUP_CHECK_FILTER}"} # S24 T-TB2 # P0 built-ins without a candidate, with the reason. EXEMPT = {"advantage_estimators:ppo": "critic: refused by the rejection matrix"} @@ -147,7 +143,7 @@ def test_old_declaration_reads_as_r0_mechanisms(): def test_candidates_cover_the_p0_mechanism_registry(): registered = alg.mechanism_names() assert set(CANDIDATES) <= registered - p0_builtin = set(CANDIDATES) | set(EXEMPT) | R0_MECHANISMS_P0 | SECRLENV_FILTER_P0 + p0_builtin = set(CANDIDATES) | set(EXEMPT) | R0_MECHANISMS_P0 # every P0 built-in beyond R0 has a candidate (extension mechanisms may add more) builtin = { f"{m.dimension}:{m.name}" for m in alg.registered_mechanisms() @@ -382,7 +378,7 @@ def test_miles_and_fake_declarations(): miles = miles_capabilities(FP) assert set(MILES_DECLARED) == EXPECTED_MILES_DECLARED assert miles.declared_mechanisms() == ( - R0_MECHANISMS_P0 | SECRLENV_FILTER_P0 | EXPECTED_MILES_DECLARED + R0_MECHANISMS_P0 | EXPECTED_MILES_DECLARED ) assert all(MILES_DECLARED[m] for m in MILES_DECLARED) # every entry cites evidence # the fake declares every correction for CPU tests (and nothing else extra) diff --git a/tests/test_rl_binding_check.py b/tests/test_rl_binding_check.py index 5b8c41c02..e43220a05 100644 --- a/tests/test_rl_binding_check.py +++ b/tests/test_rl_binding_check.py @@ -5,7 +5,6 @@ import pytest -from yeto.rl import SECRLENV_GROUP_FILTER from yeto.rl.adapters.miles import binding from yeto.rl.engine import algorithm as alg from yeto.rl.engine.algorithm import AlgorithmSpec, SamplingSpec @@ -33,8 +32,7 @@ def test_every_legacy_name_normalizes_to_a_neutral_name(): - assert set(alg.LEGACY_FILTER_NAMES) == {B_OLD, S_OLD, SECRLENV_GROUP_FILTER, - "yeto.rl.harness.codex.tbench_reward.check_group"} + assert set(alg.LEGACY_FILTER_NAMES) == {B_OLD, S_OLD} for old, new in alg.LEGACY_FILTER_NAMES.items(): assert alg.normalize_filter_name(old) == new assert alg.normalize_filter_name(new) == new diff --git a/tests/test_rl_codex_schema.py b/tests/test_rl_codex_schema.py deleted file mode 100644 index 9425fd421..000000000 --- a/tests/test_rl_codex_schema.py +++ /dev/null @@ -1,116 +0,0 @@ -"""Regression tests for live stock Codex app-server schema attestation.""" - -from __future__ import annotations - -import hashlib -import importlib -from types import SimpleNamespace - -import pytest - -from yeto.rl import ( - CODEX_OPENENV_AGENT_MODULES, - CODEX_OPENENV_IDENTITY_ENV, -) -from yeto.rl.adapters.miles import island_entry as learner - - -def test_live_codex_schema_accepts_reordered_identical_json(tmp_path): - pinned = tmp_path / "pinned.json" - generated = tmp_path / "generated.json" - pinned.write_text( - '{"methods":{"initialize":{"params":{"b":2,"a":1}}},"revision":2}', - encoding="utf-8", - ) - generated.write_text( - '{\n "revision": 2,\n "methods": {"initialize": {"params": ' - '{"a": 1, "b": 2}}}\n}\n', - encoding="utf-8", - ) - assert ( - hashlib.sha256(pinned.read_bytes()).digest() - != hashlib.sha256(generated.read_bytes()).digest() - ) - - learner._verify_live_codex_app_server_schema(pinned, generated) - - -def test_live_codex_schema_rejects_semantic_drift(tmp_path): - pinned = tmp_path / "pinned.json" - generated = tmp_path / "generated.json" - pinned.write_text('{"revision":2,"methods":{}}', encoding="utf-8") - generated.write_text('{"methods":{},"revision":3}', encoding="utf-8") - - with pytest.raises(ValueError, match="schema drifted"): - learner._verify_live_codex_app_server_schema(pinned, generated) - - -def test_live_codex_schema_rejects_exact_numeric_drift(tmp_path): - pinned = tmp_path / "pinned.json" - generated = tmp_path / "generated.json" - pinned.write_text('{"revision":9007199254740992.0}', encoding="utf-8") - generated.write_text('{"revision":9007199254740993.0}', encoding="utf-8") - - with pytest.raises(ValueError, match="schema drifted"): - learner._verify_live_codex_app_server_schema(pinned, generated) - - -@pytest.mark.parametrize( - "malformed", - [ - '{"revision":', - '{"revision":2,"revision":2}', - '{"revision":NaN}', - ], -) -def test_live_codex_schema_rejects_malformed_json(tmp_path, malformed): - pinned = tmp_path / "pinned.json" - generated = tmp_path / "generated.json" - pinned.write_text('{"revision":2}', encoding="utf-8") - generated.write_text(malformed, encoding="utf-8") - - with pytest.raises(ValueError, match="schema drifted"): - learner._verify_live_codex_app_server_schema(pinned, generated) - - -def test_codex_openenv_preflight_attests_pinned_adapter_and_environment( - tmp_path, monkeypatch -): - adapter_dir = tmp_path / "examples" / "experimental" / "openenv" - adapter_dir.mkdir(parents=True) - for name in CODEX_OPENENV_AGENT_MODULES: - (adapter_dir / name).write_text("# pinned adapter\n", encoding="utf-8") - identity = { - name.removeprefix("YETO_CODEX_OPENENV_").lower(): value - for name, value in CODEX_OPENENV_IDENTITY_ENV.items() - if name.endswith("_SHA256") - } - adapter = SimpleNamespace( - __file__=str(adapter_dir / "codex_openenv_agent_function.py"), - _OPENENV_IDENTITY_ENV=dict(CODEX_OPENENV_IDENTITY_ENV), - codex_openenv_harness_identity=lambda: identity, - ) - subprocess_adapter = SimpleNamespace( - __file__=str(adapter_dir / "codex_openenv_subprocess_agent_function.py"), - run=lambda: None, - ) - - def import_module(name): - return { - "codex_openenv_agent_function": adapter, - "codex_openenv_subprocess_agent_function": subprocess_adapter, - }[name] - - monkeypatch.setattr(importlib, "import_module", import_module) - for name, value in CODEX_OPENENV_IDENTITY_ENV.items(): - monkeypatch.setenv(name, value) - - args = SimpleNamespace(miles_root=str(tmp_path)) - learner._preflight_codex_openenv_adapter(args, "qwen35_08b") - - monkeypatch.delenv("YETO_CODEX_OPENENV_MODEL_REVISION") - with pytest.raises(ValueError, match="environment drifted"): - learner._preflight_codex_openenv_adapter(args, "qwen35_08b") - # qwen35 joined OPENENV_BACKEND_PROFILES in S17 M1, so use a profile outside the allowlist. - with pytest.raises(ValueError, match="requires backend profile"): - learner._preflight_codex_openenv_adapter(args, "not_an_openenv_profile") diff --git a/tests/test_rl_compactionrl_counts_receipt.py b/tests/test_rl_compactionrl_counts_receipt.py index eff525f7c..2eb0aba77 100644 --- a/tests/test_rl_compactionrl_counts_receipt.py +++ b/tests/test_rl_compactionrl_counts_receipt.py @@ -6,7 +6,7 @@ from yeto.rl.adapters.miles import rollout_meta_hook as hook from yeto.rl.adapters.miles.trainer import receipt_trajectory_ids -from yeto.rl.harness.codex.codex_openenv_generate import refresh_segment_counts +from yeto.rl.compaction import refresh_segment_counts def _batch(): diff --git a/tests/test_rl_engine_ports.py b/tests/test_rl_engine_ports.py index 8fcfc6aff..e808a7649 100644 --- a/tests/test_rl_engine_ports.py +++ b/tests/test_rl_engine_ports.py @@ -138,35 +138,3 @@ def test_connect_island_ray_forwards_elastic_metadata_env_only_when_on(): assert ELASTIC_METADATA_ENV not in off.init_calls[0][1]["env_vars"] -def test_connect_island_ray_forwards_codex_harness_env_and_learner_id_to_workers(): - """A-T3-4 (codex-smoke-20261003-6): the agent function runs inside RolloutExecutor - actors and must self-configure from the job runtime_env, not the raylet env.""" - from types import SimpleNamespace - - from yeto.rl.adapters.miles.entry import connect_island_ray - - machine = _TwoRayMachine(raylet_env={}) - environ = { - "RAY_ADDRESS": "a:6379", - "YETO_HARNESS_ENVIRONMENT_PROVIDER": "yeto.rl.harness.codex.tb2_provider:modal_provider", - "YETO_HARNESS_TB2_FAULT": "create_fail:2", - "YETO_CODEX_OPENENV_MODEL_REVISION": "abc", - "TBENCH_REWARD_HMAC_KEY": "k", - "YETO_SANDBOX_MODAL_TOKEN_SECRET": "s", - "MODAL_TOKEN_SECRET": "mainTok", - "UNRELATED": "x", - } - connect_island_ray(environ=environ, ray_module=machine, - miles_args=SimpleNamespace(yeto_rl_learner_id=3, yeto_rl_cell_id="c7")) - env_vars = machine.init_calls[0][1]["env_vars"] - for key in ("YETO_HARNESS_ENVIRONMENT_PROVIDER", "YETO_HARNESS_TB2_FAULT", "YETO_CODEX_OPENENV_MODEL_REVISION", - "YETO_SANDBOX_MODAL_TOKEN_SECRET"): - assert env_vars[key] == environ[key] - assert "MODAL_TOKEN_SECRET" not in env_vars # secret-handling-hardening - assert "TBENCH_REWARD_HMAC_KEY" not in env_vars # retired reward key - assert env_vars["YETO_RL_LEARNER_ID"] == "3" and env_vars["YETO_RL_CELL_ID"] == "c7" - assert "UNRELATED" not in env_vars - # without miles_args (legacy callers) nothing harness-specific is forwarded - bare = _TwoRayMachine(raylet_env={}) - connect_island_ray(environ=environ, ray_module=bare) - assert "YETO_HARNESS_ENVIRONMENT_PROVIDER" not in bare.init_calls[0][1]["env_vars"] diff --git a/tests/test_rl_eval_infer.py b/tests/test_rl_eval_infer.py deleted file mode 100644 index 184740ea9..000000000 --- a/tests/test_rl_eval_infer.py +++ /dev/null @@ -1,450 +0,0 @@ -"""rl-eval-difficulty-buckets 5.9/5.10/5.1/1.1/1.3: eval inference agent, Modal function, roles, data (CPU only).""" - -from __future__ import annotations - -import asyncio -import hashlib -import json -import re -from pathlib import Path -from types import SimpleNamespace - -import pytest - -from yeto.rl.eval import codex_infer as ci -from yeto.rl.eval.export import StoreEvalExporter -from yeto.rl.eval.island import EvalTask -from yeto.rl.eval.store import EvalStore -from yeto.rl.eval.tb2_attempt import tb2_attempt - -REPO = Path(__file__).resolve().parents[1] -MILES_SESSION_CONFIG = Path("/home/michael/work/s17-infra-fork-m/miles/rollout/session/config.py") - - -# --- PEFT writer pieces --------------------------------------------------------------------- - - -def test_rank_and_targets_from_tensor_shapes(): - shapes = {"base_model.model.layers.0.self_attn.o_proj.lora_A.weight": (16, 1024), - "base_model.model.layers.0.self_attn.o_proj.lora_B.weight": (1024, 16), - "base_model.model.layers.0.linear_attn.out_proj.lora_A.weight": (16, 2048), - "base_model.model.layers.0.linear_attn.out_proj.lora_B.weight": (1024, 16)} - assert ci.lora_rank_and_targets(shapes) == (16, ["o_proj", "out_proj"]) - with pytest.raises(ValueError, match="single LoRA rank"): - ci.lora_rank_and_targets({**shapes, "x.q_proj.lora_A.weight": (8, 4)}) - - -def test_peft_config_is_what_initial_adapter_accepts(): - cfg = ci.peft_adapter_config(rank=16, targets=["o_proj"], base_model="Qwen/Qwen3.5-0.8B", revision="r") - assert cfg["peft_type"] == "LORA" and cfg["task_type"] == "CAUSAL_LM" - assert cfg["lora_alpha"] == cfg["r"] == 16 and cfg["lora_dropout"] == 0.0 and cfg["bias"] == "none" - assert cfg["rank_pattern"] == cfg["alpha_pattern"] == {} and not cfg["use_rslora"] and not cfg["use_dora"] - - -def test_write_peft_dir_checks_tensor_hash(tmp_path): - torch = pytest.importorskip("torch") - pytest.importorskip("safetensors") - from yeto.rl.core import canonical_lora_config_hash, canonical_state - from yeto.rl.engine.trainable_state import TrainableState - from yeto.rl.eval.export import canonical_policy_files - from yeto.rl.eval.store import EvalIntegrityError - - tensors = {"m.layers.0.self_attn.o_proj.lora_A.weight": torch.ones(4, 8), - "m.layers.0.self_attn.o_proj.lora_B.weight": torch.zeros(8, 4)} - state = TrainableState.from_lora(canonical_state( - 3, tensors, base_model_revision="a" * 40, - lora_config_hash=canonical_lora_config_hash(rank=4, target_modules=["o_proj"]))) - store = EvalStore(tmp_path / "s") - m = store.put_version(3, files=canonical_policy_files(state), policy_tensor_hash=state.policy_tensor_hash(), - policy_token="yeto:3:x", sampling={}) - info = ci.write_peft_dir(store.files_dir(3), m, tmp_path / "peft", base_model="B") - assert info["rank"] == 4 and info["targets"] == ["o_proj"] - assert json.loads((tmp_path / "peft/adapter_config.json").read_text())["revision"] == "a" * 40 - with pytest.raises(EvalIntegrityError): - ci.write_peft_dir(store.files_dir(3), {**m, "policy_tensor_hash": "0" * 64}, tmp_path / "p2", base_model="B") - - -# --- SGLang / session server supervision ------------------------------------------------------ - - -def test_sglang_argv_and_session_config_match_training(): - cfg = ci.InferConfig(base_model="/models/q", revision="r", sglang_extra_args=("--x", "1")) - serving = {"reasoning_parser": "qwen3", "tool_call_parser": "qwen3_coder", "chat_template_path": "/t.jinja", - "chat_template_kwargs": {"enable_thinking": True}} - argv = ci.sglang_argv(cfg, rank=16, targets=["o_proj", "out_proj"], python="py", serving=serving) - joined = " ".join(argv) - assert "--reasoning-parser qwen3 --tool-call-parser qwen3_coder --chat-template /t.jinja" in joined - assert "parser" not in " ".join(ci.sglang_argv(cfg, rank=16, targets=["o_proj"], python="py")) - assert "--enable-lora --max-lora-rank 16 --lora-target-modules o_proj out_proj" in joined - assert "--context-length 8192" in joined and argv[-2:] == ["--x", "1"] - sc2 = ci.session_server_config(cfg, rank=16, serving=serving) - assert sc2["chat_template_path"] == "/t.jinja" - assert sc2["apply_chat_template_kwargs"] == {"clear_thinking": False, "enable_thinking": True} - with pytest.raises(ValueError, match="conflicts"): - ci.session_server_config(cfg, rank=16, serving={"chat_template_kwargs": {"clear_thinking": True}}) - assert "--revision" not in argv or not Path("/models/q").exists() - sc = ci.session_server_config(cfg, rank=16) - assert sc["tito_model"] == "qwen35" and sc["apply_chat_template_kwargs"] == {"clear_thinking": False} - assert sc["lora_rank"] == 16 and sc["backend_url"] == "http://127.0.0.1:30000" - assert sc["use_rollout_routing_replay"] is False and sc["session_message_matcher"] == "strict" - if MILES_SESSION_CONFIG.is_file(): # every SessionServerConfig field, nothing else (strict model) - text = MILES_SESSION_CONFIG.read_text().split("def compute_session_server_config")[0] - fields = set(re.findall(r"^ (\w+): ", text, re.M)) - assert set(sc) == fields - - -class _Proc: - def __init__(self): - self.returncode = None - self.terminated = False - - def poll(self): - return self.returncode - - def terminate(self): - self.terminated = True - self.returncode = 0 - - def wait(self, timeout=None): - return 0 - - -class _Http: - def __init__(self, refuse_load=False): - self.calls = [] - self.refuse_load = refuse_load - - def __call__(self, method, url, body=None, timeout=60.0): - self.calls.append((method, url.split(":", 2)[-1], body)) - if url.endswith("/load_lora_adapter") and self.refuse_load: - return 400, {"success": False, "message": "bad"} - if url.endswith("/sessions") and method == "POST": - return 200, {"session_id": "s1"} - return 200, {} - - -def _loader(monkeypatch, tmp_path, *, rank=16, targets=("o_proj",), refuse=False): - spawned = [] - - def popen(argv, **kw): - spawned.append(argv) - return _Proc() - - def fake_write(files_dir, manifest, out_dir, *, base_model): - return {"rank": rank, "targets": list(targets), "tensors": 2, "path": str(out_dir), "seconds": 0.1} - - monkeypatch.setattr(ci, "write_peft_dir", fake_write) - events = [] - http = _Http(refuse_load=refuse) - loader = ci.SglangSessionLoader(ci.InferConfig(base_model="B", work_dir=str(tmp_path)), popen=popen, - http=http, sleep=lambda s: None, - emit=lambda e, **f: events.append((e, f)), - serving=lambda tito: {"reasoning_parser": "qwen3", - "tool_call_parser": "qwen3_coder", - "chat_template_path": "/t.jinja"}) - return loader, spawned, http, events - - -def test_loader_starts_once_then_swaps_adapter(monkeypatch, tmp_path): - loader, spawned, http, events = _loader(monkeypatch, tmp_path) - assert loader.load({"policy_version": 0, "rl/policy_token": "yeto:0:h"}, tmp_path) == "yeto:0:h" - assert len(spawned) == 2 and "sglang.launch_server" in spawned[0] - assert spawned[0][spawned[0].index("--tool-call-parser") + 1] == "qwen3_coder" - assert any(c[1].endswith("/health_generate") for c in http.calls) - assert loader.load({"policy_version": 10, "rl/policy_token": "yeto:10:h"}, tmp_path) == "yeto:10:h" - assert len(spawned) == 2 # no restart - paths = [c[1] for c in http.calls if c[0] == "POST"] - assert paths.count("30000/load_lora_adapter") == 2 and paths.count("30000/unload_lora_adapter") == 1 - load_bodies = [c[2] for c in http.calls if c[1].endswith("/load_lora_adapter")] - assert {b["lora_name"] for b in load_bodies} == {ci.SERVED_LORA_NAME} - assert [e for e, _ in events] == ["rl_eval_policy_load"] * 2 and "sglang_start_s" in events[0][1] - assert events[0][1]["serving"]["tool_call_parser"] == "qwen3_coder" - assert json.loads(spawned[1][-1])["chat_template_path"] == "/t.jinja" - loader.close() - assert all(p is not None for p in loader.procs.values()) - - -def test_loader_start_base_serves_without_lora(monkeypatch, tmp_path): - loader, spawned, http, _ = _loader(monkeypatch, tmp_path) - out = loader.start_base() - assert "sglang_start_s" in out and len(spawned) == 2 - assert not any(a.startswith("--enable-lora") or "lora" in a for a in spawned[0]) - session = json.loads(spawned[1][-1]) - assert session["lora_rank"] == 0 and session["lora_adapter_path"] is None - assert not [c for c in http.calls if "lora" in c[1]] - with pytest.raises(RuntimeError, match="already started"): - loader.start_base() - - -def test_loader_fails_closed(monkeypatch, tmp_path): - loader, *_ = _loader(monkeypatch, tmp_path, refuse=True) - with pytest.raises(RuntimeError, match="refused the adapter"): - loader.load({"policy_version": 0, "rl/policy_token": "t"}, tmp_path) - loader, *_ = _loader(monkeypatch, tmp_path) - loader.load({"policy_version": 0, "rl/policy_token": "t"}, tmp_path) - monkeypatch.setattr(ci, "write_peft_dir", lambda *a, **k: {"rank": 8, "targets": ["o_proj"], "path": "p", - "seconds": 0}) - with pytest.raises(RuntimeError, match="differs from the served"): - loader.load({"policy_version": 1, "rl/policy_token": "t"}, tmp_path) - - -def test_loader_reports_a_dead_server(monkeypatch, tmp_path): - loader, *_ = _loader(monkeypatch, tmp_path) - - def popen(argv, **kw): - p = _Proc() - p.returncode = 1 - return p - - loader.popen = popen - with pytest.raises(RuntimeError, match="sglang exited"): - loader.load({"policy_version": 0, "rl/policy_token": "t"}, tmp_path) - - -# --- the agent port ----------------------------------------------------------------------------- - - -def _lease(tmp_path, instruction="Do the thing."): - task = SimpleNamespace(instruction=instruction) - return SimpleNamespace(env_url="http://127.0.0.1:9", env_token="tok", deadline_seconds=5.0, - worker_env={"W": "1"}, task=task) - - -def _agent(drive): - http = _Http() - return ci.CodexEvalAgent(ci.InferConfig(base_model="B", top_k=20), drive=drive, http=http), http - - -def test_agent_runs_the_training_worker_job(tmp_path): - seen = {} - - async def drive(job, trajectory, board, worker_env): - seen.update(job=job, board=board, worker_env=worker_env) - return {"status": "max_turns", "metrics": {"turns": 12, "max_model_total_tokens": 7000, - "end_reason": "CodexModelFailure: x", "secret": 1}, - "episode_id": job["episode_id"]} - - agent, http = _agent(drive) - out = asyncio.run(agent(_lease(tmp_path), EvalTask("a", "tb2-easy"), 0, policy_token="t")) - job = seen["job"] - assert job["base_url"] == "http://127.0.0.1:30100/sessions/s1" and job["prompt"] == "Do the thing." - assert job["request_kwargs"] == {"temperature": 1.0, "top_p": 1.0, "max_tokens": 4096, "top_k": 20} - assert job["max_seq_len"] == 8192 and job["env_token"] == "tok" and job["driver"] == "stock" - assert seen["worker_env"] == {"W": "1"} and seen["board"] is None - assert out["end_reason"] == "max_turns" and out["turns"] == 12 and out["tokens"] == 7000 - assert out["agent_detail"] == {"end_reason": "CodexModelFailure: x", "turns": 12, "max_model_total_tokens": 7000} - create = [c for c in http.calls if c[1].endswith("/sessions")][0] - assert create[2]["evaluation"] is True and create[2]["top_k"] == 20 - assert http.calls[-1][0] == "DELETE" - - -def test_agent_maps_statuses_and_harness_errors(tmp_path): - from yeto.rl.harness.codex import codex_harness_agent as harness - - async def timeout_drive(job, *a): - return {"status": "timeout", "metrics": {}, "episode_id": job["episode_id"]} - - agent, _ = _agent(timeout_drive) - assert asyncio.run(agent(_lease(tmp_path), EvalTask("a", "x"), 0, policy_token="t"))["end_reason"] == "timed_out" - - async def broken(job, *a): - raise harness.CodexHarnessError("protocol violation") - - agent, http = _agent(broken) - out = asyncio.run(agent(_lease(tmp_path), EvalTask("a", "x"), 0, policy_token="t")) - assert out["end_reason"] == "infra_error" and "protocol" in out["error"] - assert http.calls[-1][0] == "DELETE" # session released on error too - - async def hang(job, *a): - await asyncio.sleep(10) - - agent, _ = _agent(hang) - lease = _lease(tmp_path) - lease.deadline_seconds = 0.05 - assert asyncio.run(agent(lease, EvalTask("a", "x"), 0, policy_token="t"))["end_reason"] == "timed_out" - - -def test_tb2_attempt_records_timing_and_results_hash_ignores_it(tmp_path): - class Verifier: - async def evaluate(self, episode_id): - return {"passed": True} - - class Provider: - async def acquire(self, task_id, tid): - async def destroy(): - return None - return SimpleNamespace(verifier=Verifier(), destroy=destroy) - - async def agent(lease, task, trial, *, policy_token): - return {"episode_id": "e", "end_reason": "completed", "turns": 2, "agent_detail": {"end_reason": "x"}} - - out = tb2_attempt(Provider(), agent)(EvalTask("a", "x"), 0, policy_version=0, policy_token="t") - assert set(out["timing"]) == {"sandbox_s", "agent_s", "judge_s", "destroy_s"} - from yeto.rl.eval.island import EvalIsland - - rec = EvalIsland._record(SimpleNamespace(island_id="e"), EvalTask("a", "tb2-easy"), 0, 0, "t", out, 1.0) - assert rec["timing"] == out["timing"] and rec["agent_detail"] == {"end_reason": "x"} - s1, s2 = EvalStore(tmp_path / "1"), EvalStore(tmp_path / "2") - s1.append_unit(rec) - s2.append_unit({**rec, "timing": {"agent_s": 99.0}, "agent_detail": {"end_reason": "y"}}) - assert s1.results_sha256(0) == s2.results_sha256(0) - - -# --- training-finished marker (5.10) -------------------------------------------------------------- - - -def test_final_export_marks_training_finished(tmp_path): - store = EvalStore(tmp_path) - exp = StoreEvalExporter(store, sampling={}, interval=10, - policy_files=lambda s: {"policy.safetensors": b"w"}) - state = SimpleNamespace(policy_tensor_hash=lambda: "h") - exp(10, state, "yeto:10:h") - assert not store.training_finished() - exp(13, state, "yeto:13:h", final=True) - assert store.training_finished() - assert json.loads((tmp_path / "training-finished.json").read_text()) == {"final_version": 13} - - -# --- Modal function definition (fake modal) ------------------------------------------------------ - - -def test_eval_function_spec_env_and_client(): - from yeto.cloud import modal_eval_island as mei - - spec = mei.EvalFunctionSpec(app_name="a", volume="vol", workdir=".", codex_dir="c", tb2_dir="t", - envs={"YETO_CODEX_VERSION": "x"}) - env = spec.container_env() - assert env["YETO_RL_ISLAND_ROLE"] == "eval" and env["YETO_RL_EVAL_STORE"] == mei.EVAL_STORE_MOUNT - assert env["YETO_RL_EVAL_STORE_MOUNTED"] == "1" and env["YETO_RL_EVAL_VOLUME"] == "vol" - assert env["YETO_HARNESS_TB2_TASKS_DIR"] == mei.TB2_MOUNT and env["YETO_CODEX_VERSION"] == "x" - assert spec.image_ref.startswith("ghcr.io/") and "@sha256:" in spec.image_ref - - class Call: - def get(self): - return {"evaluated": [0]} - - class Fn: - def __init__(self): - self.args = [] - - def spawn(self, cfg): - self.args.append(cfg) - return Call() - - fn = Fn() - events = [] - launcher = mei.EvalIslandLauncher(client=mei.ModalFunctionClient(fn), emit=lambda e, **f: events.append(e), - config={"store": "s"}, gpu="H100!", gpus=1) - assert launcher.run() == {"evaluated": [0]} - assert fn.args == [{"store": "s"}] and events == ["rl_eval_island", "rl_eval_island"] - - -# --- roles: launcher + island ledger (5.1) --------------------------------------------------------- - - -def test_eval_role_never_joins_the_merge_pool(): - from yeto.launcher import ISLAND_ROLES, island_role_env - from yeto.rl.engine.island_ledger import CrossIslandLedger, LedgerError - from yeto.rl.eval.island import EVAL_ISLAND_ROLE - - assert EVAL_ISLAND_ROLE in ISLAND_ROLES and island_role_env("eval") == {"YETO_RL_ISLAND_ROLE": "eval"} - with pytest.raises(ValueError): - island_role_env("judge") - ledger = CrossIslandLedger(mode="elastic") - ledger.join("i0", now=0.0) - with pytest.raises(LedgerError, match="cannot join the merge pool"): - ledger.join("eval-0", now=0.0, role="eval") - assert set(ledger.members) == {"i0"} - - -# --- eval / train jsonl (1.1 second half, 1.3) ------------------------------------------------------- - - -def _fake_tb2(root: Path, spec): - for tid, diff in spec: - d = root / tid - (d / "tests").mkdir(parents=True) - (d / "tests/test.sh").write_text("echo ok\n") - (d / "instruction.md").write_text(f"solve {tid}\n") - (d / "task.toml").write_text(f'[metadata]\ndifficulty = "{diff}"\n[environment]\ndocker_image = "img/{tid}"\n') - - -def test_build_eval_data_split_and_determinism(tmp_path): - import importlib.util - - from yeto.rl.harness.reward_env.benchmark import HOLDOUT_SCHEMA - - mod_spec = importlib.util.spec_from_file_location("build_eval_data", REPO / "tools/build_eval_data.py") - tool = importlib.util.module_from_spec(mod_spec) - mod_spec.loader.exec_module(tool) - _fake_tb2(tmp_path / "tb2", [("a", "easy"), ("b", "medium"), ("c", "hard"), ("d", "medium")]) - hold = {"schema": HOLDOUT_SCHEMA, "benchmark": "tb2", "benchmark_version": "tb2@unknown", "seed": 1, - "rule": "t", "items": [{"task_id": "b", "difficulty": "medium", "eval_bucket": "tb2-medium"}, - {"task_id": "c", "difficulty": "hard", "eval_bucket": "tb2-hard"}]} - out = tool.build(tmp_path / "tb2", hold) - assert out == tool.build(tmp_path / "tb2", hold) # same inputs, same bytes - ev = [json.loads(x) for x in out[tool.EVAL_JSONL].decode().splitlines()] - tr = [json.loads(x) for x in out[tool.TRAIN_JSONL].decode().splitlines()] - assert [r["metadata"]["task_id"] for r in ev] == ["b", "c"] and [r["metadata"]["task_id"] for r in tr] == ["a", "d"] - assert ev[1]["metadata"] == {"task_id": "c", "benchmark": "tb2", "benchmark_version": "tb2@unknown", - "difficulty": "hard", "difficulty_source": "tb2-task.toml", "eval_bucket": "tb2-hard"} - assert "eval_bucket" not in tr[0]["metadata"] and tr[0]["metadata"]["difficulty"] == "easy" - sha = json.loads(out[tool.SHA_JSON]) - assert sha[tool.EVAL_JSONL]["sha256"] == hashlib.sha256(out[tool.EVAL_JSONL]).hexdigest() - with pytest.raises(ValueError, match="checkout"): - tool.build(tmp_path / "tb2", {**hold, "benchmark_version": "tb2@other"}) - # S17 N13: the unusable table leaves training; an unusable task in the hold-out list is refused - tr = [json.loads(x) for x in tool.build(tmp_path / "tb2", hold, unusable={"d": "unusable/x: y"})[ - tool.TRAIN_JSONL].decode().splitlines()] - assert [r["metadata"]["task_id"] for r in tr] == ["a"] - with pytest.raises(ValueError, match="hold-out"): - tool.build(tmp_path / "tb2", hold, unusable={"b": "unusable/x: y"}) - - -def test_shipped_eval_and_train_jsonl_match_their_sha_record(): - sha = json.loads((REPO / "data/eval/tb2-data.sha256.json").read_text()) - for rel, n in (("eval/tb2-holdout-eval.jsonl", 30), ("tb2/tb2-train.jsonl", 46)): - data = (REPO / "data" / rel).read_bytes() - assert sha[rel] == {"rows": n, "sha256": hashlib.sha256(data).hexdigest()} - hold = json.loads((REPO / "data/eval/tb2-holdout.json").read_text()) - ev = {json.loads(x)["metadata"]["task_id"] for x in (REPO / "data/eval/tb2-holdout-eval.jsonl").read_text().splitlines()} - tr = {json.loads(x)["metadata"]["task_id"] for x in (REPO / "data/tb2/tb2-train.jsonl").read_text().splitlines()} - assert ev == {i["task_id"] for i in hold["items"]} and not ev & tr - unusable = {i["task_id"] for i in json.loads((REPO / "data/eval/tb2-unusable.json").read_text())["items"]} - assert not unusable & (ev | tr) # S17 N13: unusable tasks leave both sets - assert {e["task_id"] for e in hold["excluded"]} - unusable <= tr # smoke-6: out of the eval pool, still trainable - split = json.loads((REPO / "data/eval/tb2-train.json").read_text()) - assert tr == {i["task_id"] for i in split["items"]} and sha["unusable_excluded"] == sorted(unusable) - - -def test_eval_island_body_keeps_logs_off_the_volume_until_the_end(tmp_path, monkeypatch): - import subprocess - - from yeto.cloud import modal_eval_island as mei - - import os - import sys - - monkeypatch.setenv("PYTHONPATH", os.environ.get("PYTHONPATH", "")) # restored after the test - monkeypatch.setenv("YETO_MODAL_RUNTIME_PYTHONPATH", "") - monkeypatch.setattr(sys, "path", list(sys.path)) - monkeypatch.setattr(subprocess, "run", lambda *a, **k: SimpleNamespace(stdout="NVIDIA H100 80GB HBM3\n")) - seen = {} - - def fake_main(config): - seen["infer"] = config["infer"] - Path(config["infer"]["log_dir"]).mkdir(parents=True) - (Path(config["infer"]["log_dir"]) / "sglang.log").write_text("up") - return {"evaluated": [0], "loads": []} - - monkeypatch.setattr(mei, "island_main", fake_main) - out = mei.eval_island_body({"store": str(tmp_path / "vol"), "island_id": "e1", "assert_gpu_name": "H100", - "local_run_dir": str(tmp_path / "local"), "infer": {"base_model": "B"}}) - assert out["evaluated"] == [0] and out["gpu_names"] == ["NVIDIA H100 80GB HBM3"] - assert seen["infer"]["log_dir"].startswith(str(tmp_path / "local")) - assert (tmp_path / "vol/runs/e1/logs/sglang.log").read_text() == "up" - assert (tmp_path / "vol/runs/e1/body.json").is_file() - assert os.environ["PYTHONPATH"] == mei.IMAGE_PYTHONPATH - with pytest.raises(RuntimeError, match="asked for A100"): - mei.eval_island_body({"store": str(tmp_path / "vol"), "assert_gpu_name": "A100", - "local_run_dir": str(tmp_path / "l2")}) diff --git a/tests/test_rl_eval_island.py b/tests/test_rl_eval_island.py index e8e0c1c6e..6a242ae75 100644 --- a/tests/test_rl_eval_island.py +++ b/tests/test_rl_eval_island.py @@ -2,7 +2,6 @@ from __future__ import annotations -import asyncio import hashlib import json from pathlib import Path @@ -12,9 +11,8 @@ from yeto.rl.eval import guard, stats from yeto.rl.eval.export import StoreEvalExporter, is_eval_version -from yeto.rl.eval.island import EvalIsland, EvalPlan, EvalSettingError, EvalTask, Preempted, jsonl_emitter +from yeto.rl.eval.island import EvalIsland, EvalPlan, EvalSettingError, Preempted, jsonl_emitter from yeto.rl.eval.store import EvalIntegrityError, EvalStore -from yeto.rl.eval.tb2_attempt import tb2_attempt, trajectory_id from yeto.rl.harness.reward_env.benchmark import HOLDOUT_SCHEMA, holdout_sha256 REPO = Path(__file__).resolve().parents[1] @@ -359,52 +357,6 @@ def test_driver_hook_emits_rl_eval_export(tmp_path): assert len(events) == 1 -# --- TB2 attempt over a fake provider ----------------------------------------------------------- - - -class _Verifier: - def __init__(self, passed): - self.passed = passed - - async def evaluate(self, episode_id): - return {"passed": self.passed, "timed_out": False} - - -class _Provider: - def __init__(self, passed=True, fail=False): - self.passed, self.fail, self.destroyed, self.ids = passed, fail, 0, [] - - async def acquire(self, task_id, tid): - if self.fail: - raise RuntimeError("no sandbox") - self.ids.append(tid) - - async def destroy(): - self.destroyed += 1 - return SimpleNamespace(verifier=_Verifier(self.passed), destroy=destroy) - - -def test_tb2_attempt_judges_and_always_destroys(): - async def agent(lease, task, trial, *, policy_token): - return {"episode_id": "e", "end_reason": "max_turns", "turns": 30, "tokens": 900} - - task = EvalTask("a", "tb2-easy") - prov = _Provider(passed=True) - out = tb2_attempt(prov, agent)(task, 1, policy_version=10, policy_token="t") - assert out["success"] and out["reward"] == 1.0 and out["end_reason"] == "max_turns" - assert prov.ids == [trajectory_id(10, "a", 1)] and prov.destroyed == 1 - - async def broken(lease, task, trial, *, policy_token): - raise Preempted("gone") - - prov2 = _Provider() - with pytest.raises(Preempted): - tb2_attempt(prov2, broken)(task, 0, policy_version=10, policy_token="t") - assert prov2.destroyed == 1 - with pytest.raises(RuntimeError, match="no sandbox"): - tb2_attempt(_Provider(fail=True), agent)(task, 0, policy_version=10, policy_token="t") - - # --- Modal launcher / store (fake client) ------------------------------------------------------ @@ -507,15 +459,6 @@ def test_island_main_with_dotted_fakes(tmp_path, monkeypatch): assert tape[0]["event"] == "rl_eval" and tape[0]["eval/units"] == 2 -def test_asyncio_default_runner_is_used(): - # tb2_attempt defaults to asyncio.run: works from sync code - async def agent(lease, task, trial, *, policy_token): - return {"end_reason": "completed"} - - assert tb2_attempt(_Provider(passed=False), agent, run=asyncio.run)( - EvalTask("a", "x"), 0, policy_version=0, policy_token="t")["success"] is False - - def test_canonical_policy_files_round_trip_policy_tensor_hash(tmp_path): torch = pytest.importorskip("torch") pytest.importorskip("safetensors") diff --git a/tests/test_rl_ir_harness.py b/tests/test_rl_ir_harness.py index a30cde8f7..41a3a68d3 100644 --- a/tests/test_rl_ir_harness.py +++ b/tests/test_rl_ir_harness.py @@ -7,7 +7,6 @@ from __future__ import annotations import asyncio -import dataclasses import json import time from types import SimpleNamespace @@ -15,7 +14,6 @@ import pytest import torch -from yeto.rl.engine import run_config as rc from yeto.rl.engine.algorithm import AlgorithmSpec from yeto.rl.engine.bridges import LocalOnlySync from yeto.rl.engine.driver import EventTape, IslandDriver, PolicyIdentityError, policy_token @@ -53,11 +51,11 @@ # --------------------------------------------------------------------------- def test_ir1_agent_flags_pass_through_with_agentic_generate(): c = sub(make_config(), "agent", custom_generate_function_path=AGENTIC, - custom_agent_function_path="yeto.rl.harness.codex.codex_openenv_agent_function", + custom_agent_function_path="yeto.rl.adapters.miles.harbor_agent.run", agent_max_seq_len=8192) argv = cfg.translate_run_config(c, AlgorithmSpec()).argv i = argv.index("--custom-agent-function-path") - assert argv[i + 1] == "yeto.rl.harness.codex.codex_openenv_agent_function" + assert argv[i + 1] == "yeto.rl.adapters.miles.harbor_agent.run" assert argv[argv.index("--max-seq-len") + 1] == "8192" assert argv[argv.index("--custom-generate-function-path") + 1] == AGENTIC @@ -82,20 +80,6 @@ def test_ir1_append_roles_rejected_with_template_reason(): cfg.translate_run_config(c, AlgorithmSpec()) -def test_ir1_append_roles_accepted_for_signed_codex_agent_when_equal_to_profile(): - from yeto.rl import CODEX_OPENENV_AGENT - c = sub(make_config(), "agent", use_session_server=True, tito_model="qwen35", - custom_generate_function_path=AGENTIC, custom_agent_function_path=CODEX_OPENENV_AGENT, - tito_allowed_append_roles=("tool", "user")) - stub = lambda _m: (None, None) # noqa: E731 - no miles checkout in the CPU venv - launch = cfg.translate_run_config(c, AlgorithmSpec(), tito_parser_resolver=stub) - assert "--tito-allowed-append-roles" not in " ".join(map(str, launch.argv)) - assert "--tito-model" in launch.argv - bad = sub(c, "agent", tito_allowed_append_roles=("tool",)) - with pytest.raises(cfg.UnmappedConfigError, match=r"fix the\s+append roles"): - cfg.translate_run_config(bad, AlgorithmSpec(), tito_parser_resolver=stub) - - def test_ir1_session_server_and_partial_rollout_are_mutually_exclusive(): with pytest.raises(cfg.UnmappedConfigError) as err: cfg.check_session_server_partial_rollout(True, True) diff --git a/tests/test_rl_launcher.py b/tests/test_rl_launcher.py index 8dd19f953..ad2c7203b 100644 --- a/tests/test_rl_launcher.py +++ b/tests/test_rl_launcher.py @@ -20,20 +20,14 @@ syncer_command, ) from yeto.rl import ( - CODEX_OPENENV_AGENT, MILES_NEXT_COMMIT, MILES_NEXT_REPOSITORY, MILES_PEFT_VERSION, - SECRLENV_GENERATE, - SECRLENV_GROUP_FILTER, - SECRLENV_REWARD, SGLANG_NEXT_COMMIT, SGLANG_NEXT_REPOSITORY, ) from yeto.rl.adapters.miles import island_entry as rl_learner from yeto.rl.codex_backend import ( - QWEN35_08B_MODEL, - QWEN35_08B_REVISION, QWEN35_MODEL, QWEN35_REVISION, ) @@ -158,48 +152,6 @@ def test_over_sampling_below_the_batch_is_rejected(): _prepare_rl_args(args) -def test_legacy_secrlenv_agent_auto_binds_exact_replacement_contract(): - args = _args( - [ - "--reward-function", - SECRLENV_REWARD, - "--custom-generate-function-path", - SECRLENV_GENERATE, - "--custom-agent-function-path", - "yeto.rl.harness.codex.agent.run", - "--use-session-server", - "--tito-model", - "org/model", - ] - ) - - _prepare_rl_args(args) - - assert args.dynamic_sampling_filter_path == SECRLENV_GROUP_FILTER - assert args.dynamic_sampling_max_replacements == 0 - assert args.secrlenv_max_infrastructure_replacements == 1 - assert args.over_sampling_batch_size == args.rollout_batch_size + 1 - - -def test_secrlenv_agent_rejects_the_unwrapped_miles_generator(): - args = _args( - [ - "--reward-function", - SECRLENV_REWARD, - "--custom-generate-function-path", - "miles.rollout.generate_hub.agentic_tool_call.generate", - "--custom-agent-function-path", - "yeto.rl.harness.codex.agent.run", - "--use-session-server", - "--tito-model", - "org/model", - ] - ) - - with pytest.raises(ValueError, match="signed SecRLEnv generate wrapper"): - _prepare_rl_args(args) - - def test_decoupled_rl_preset_fixes_the_fragment_outer_contract(): args = _args( ( @@ -404,45 +356,6 @@ def test_rl_maps_long_task_and_oversampling_options_to_miles(): assert args.tito_allowed_append_roles == ["tool", "user"] -def test_codex_openenv_uses_exact_08b_profile_with_qwen35_tito_family(): - args = _args( - ( - "--model", - QWEN35_08B_MODEL, - "--model-revision", - QWEN35_08B_REVISION, - "--lora-targets", - "attention", - "--custom-generate-function-path", - "miles.rollout.generate_hub.agentic_tool_call.generate", - "--custom-agent-function-path", - CODEX_OPENENV_AGENT, - "--codex-reasoning-effort", - "xhigh", - "--codex-backend-profile", - "qwen35_08b", - "--use-session-server", - "--tito-model", - "qwen35", - "--tito-allowed-append-roles", - "tool", - "user", - ) - ) - - _prepare_rl_args(args) - - assert args.codex_backend_profile == "qwen35_08b" - assert args.tito_model == "qwen35" - assert args.model == QWEN35_08B_MODEL - assert args.model_revision == QWEN35_08B_REVISION - assert args.apply_chat_template_kwargs == {"clear_thinking": False} - - args.tito_model = "qwen35_08b" - with pytest.raises(ValueError, match="Miles TITO family"): - _prepare_rl_args(args) - - def test_rl_parses_cybergym_worker_configuration(): args = _args( ( @@ -1762,3 +1675,13 @@ def test_run_miles_carries_the_island_task_record(monkeypatch, tmp_path): record = {"task_ids": ["web1"], "data_sha256": "a" * 64} rl_learner.run_miles(args, model_path=model, prompt_path=prompts, island_tasks=record) assert captured["miles_args"].yeto_rl_island_tasks == record + + +@pytest.mark.parametrize("flag, value", [ + ("--codex-reasoning-effort", "xhigh"), + ("--codex-backend-profile", "qwen35_08b"), +]) +def test_removed_codex_agent_flags_are_refused(flag, value): + args = _args((flag, value, "--use-session-server", "--tito-model", "qwen35")) + with pytest.raises(ValueError, match="signed Codex agent"): + _prepare_rl_args(args) diff --git a/tests/test_rl_launcher_codex_bundle.py b/tests/test_rl_launcher_codex_bundle.py deleted file mode 100644 index d4193c518..000000000 --- a/tests/test_rl_launcher_codex_bundle.py +++ /dev/null @@ -1,401 +0,0 @@ -"""Stock Codex run bundle on the ports launch paths (T3 G2; design R-SCOPE). - -CPU only. The contract attestation is exercised against the real bundle when -``YETO_CODEX_BUNDLE_DIR`` points at a ``scripts/fetch_codex_bundle.py`` output -(skipped otherwise); everything else uses a synthetic contract built from the -Yeto pins so the launcher wiring (learner flag, ``YETO_CODEX_*`` env, sky -file_mounts, Modal add_local_dir) is checked without a 310 MB binary. -""" - -from __future__ import annotations - -import json -import os -import shlex -import sys -import types -from pathlib import Path - -import pytest - -from yeto import launcher as L -from yeto import modal_runner as mr -from yeto.gpu_spec import parse_gpu_spec -from yeto.launcher import _prepare_rl_args, build_modal_island_config, make_miles_island_task -from yeto.rl import ( - CODEX_CONTAINER_BINARY_PATH, - CODEX_OPENENV_AGENT, - CODEX_OPENENV_IDENTITY_ENV, -) -from yeto.rl.codex_backend import QWEN35_08B_MODEL, QWEN35_08B_REVISION -from yeto.rl.ssh_harness import _plan_digest - -from test_modal_runner import fake_modal -from test_rl_engine_selection import _cli as _args # ports engine (CLI default) -from test_rl_launcher import _Resources, _Storage, _StorageMode, _Task - -PROVIDER = "yeto.rl.harness.codex.tb2_provider:provider" - - -def _codex_args(): - args = _args( - ( - "--model", QWEN35_08B_MODEL, - "--model-revision", QWEN35_08B_REVISION, - "--lora-targets", "attention", - "--custom-generate-function-path", - "miles.rollout.generate_hub.agentic_tool_call.generate", - "--custom-agent-function-path", CODEX_OPENENV_AGENT, - "--codex-reasoning-effort", "xhigh", - "--codex-backend-profile", "qwen35_08b", - "--use-session-server", - "--tito-model", "qwen35", - "--tito-allowed-append-roles", "tool", "user", - "--agent-max-seq-len", "128", - ) - ) - args.data_revision = "d" * 40 - args.source_sha256 = "e" * 64 - args.reward_sha256 = "f" * 64 - _prepare_rl_args(args) - return args - - -def _synthetic_contract(args) -> dict: - """What ``ssh_harness._codex_harness_contract`` returns for a pin-exact bundle.""" - from yeto.rl import ssh_harness as sh - from yeto.rl.codex_backend import stock_codex_backend_contract - - identity = sh._pinned_codex_adapter_identity() - return { - "agent_function_path": sh.CODEX_HARNESS_AGENT, - "agent_source_sha256": sh.CODEX_HARNESS_AGENT_SHA256, - "controller_binary_path": "/bundle/codex/codex-x86_64-unknown-linux-musl", - "controller_package_manifest_path": "/bundle/codex/codex-package.json", - "controller_app_server_schema_path": "/bundle/codex/codex_app_server_protocol.v2.schemas.json", - "bundle_binary_path": sh.CODEX_REMOTE_BUNDLE_PATH, - "bundle_package_manifest_path": sh.CODEX_REMOTE_MANIFEST_PATH, - "bundle_app_server_schema_path": sh.CODEX_REMOTE_SCHEMA_PATH, - "container_binary_path": sh.CODEX_CONTAINER_BINARY_PATH, - "container_app_server_schema_path": sh.CODEX_CONTAINER_APP_SERVER_SCHEMA_PATH, - "binary_sha256": sh.CODEX_LINUX_BINARY_SHA256, - "binary_size_bytes": sh.CODEX_LINUX_BINARY_SIZE_BYTES, - "cli_version": sh.CODEX_CLI_VERSION, - "npm_package": sh.CODEX_NPM_PACKAGE, - "target": sh.CODEX_LINUX_TARGET, - "npm_tarball_sha256": sh.CODEX_NPM_TARBALL_SHA256, - "package_manifest_sha256": sh.CODEX_PACKAGE_MANIFEST_SHA256, - "app_server_protocol_revision": sh.CODEX_APP_SERVER_PROTOCOL_REVISION, - "app_server_schema_sha256": sh.CODEX_APP_SERVER_SCHEMA_SHA256, - **identity, - "reasoning_effort": "xhigh", - "backend": stock_codex_backend_contract("qwen35_08b", args.rollout_max_response_len), - "openenv_identity_env": dict(CODEX_OPENENV_IDENTITY_ENV), - } - - -@pytest.fixture -def bundle(tmp_path, monkeypatch): - """A fake bundle dir + the contract attestation stubbed to the pin values.""" - # An empty HOME keeps machine state (e.g. a cached HF token, which the - # launcher mounts when present) out of the mount lists these tests pin. - monkeypatch.setenv("HOME", str(tmp_path / "home")) - monkeypatch.setitem(sys.modules, "sky", types.SimpleNamespace( - Task=_Task, Resources=_Resources, Storage=_Storage, StorageMode=_StorageMode)) - root = tmp_path / "codex-bundle" / "codex" - root.mkdir(parents=True) - for name in ("codex-x86_64-unknown-linux-musl", "codex-package.json", - "codex_app_server_protocol.v2.schemas.json"): - (root / name).write_bytes(b"x") - seen = {} - - def fake_contract(namespace, args): - seen["namespace"] = vars(namespace) - return _synthetic_contract(args) - - import yeto.rl.ssh_harness as sh - - monkeypatch.setattr(sh, "_codex_harness_contract", fake_contract) - monkeypatch.setenv(L.CODEX_BUNDLE_DIR_ENV, str(root)) - monkeypatch.setenv(L.HARNESS_ENVIRONMENT_PROVIDER_ENV, PROVIDER) - monkeypatch.delenv(L.CODEX_COMPACTION_ENV, raising=False) - monkeypatch.delenv(L.HARNESS_PREFLIGHT_ENV, raising=False) - return root, seen - - -# The learner's container-side expected_env keys (yeto.rl.adapters.miles.island_entry._preflight_codex_harness) -# plus what codex_harness_agent._attest_runtime / preflight.assert_openenv_identity read. -LEARNER_EXPECTED_ENV = { - "YETO_CODEX_BINARY_PATH", "YETO_CODEX_BINARY_SHA256", "YETO_CODEX_BINARY_SIZE_BYTES", - "YETO_CODEX_VERSION", "YETO_CODEX_APP_SERVER_PROTOCOL_REVISION", - "YETO_CODEX_APP_SERVER_SCHEMA_SHA256", "YETO_CODEX_BASE_INSTRUCTIONS_SHA256", - "YETO_CODEX_TERMINAL_EXEC_TOOL_SCHEMA_SHA256", "YETO_CODEX_SUBMIT_TOOL_SCHEMA_SHA256", - "YETO_CODEX_DYNAMIC_TOOLS_SCHEMA_SHA256", "YETO_CODEX_REASONING_EFFORT", - "YETO_CODEX_BACKEND_MAX_TOKENS", "YETO_CODEX_BACKEND_REASONING_EFFORT", - "YETO_CODEX_BACKEND_THINKING", "YETO_CODEX_CHAT_TEMPLATE", "YETO_CODEX_CHAT_TEMPLATE_KWARGS", - "YETO_CODEX_TITO_ALLOWED_APPEND_ROLES", "YETO_CODEX_HARNESS_CONTRACT_SHA256", -} - - -def test_non_codex_runs_are_untouched(): - args = _args() - args.data_revision = "d" * 40 - args.source_sha256 = "e" * 64 - args.reward_sha256 = "f" * 64 - assert L.codex_harness_launch(args, environ={}) is None - - -def test_codex_on_ports_fails_closed_without_bundle_provider_or_with_compaction(bundle, monkeypatch): - args = _codex_args() - with pytest.raises(ValueError, match=L.CODEX_BUNDLE_DIR_ENV): - L.codex_harness_launch(args, environ={}) - root, _ = bundle - with pytest.raises(ValueError, match=L.HARNESS_ENVIRONMENT_PROVIDER_ENV): - L.codex_harness_launch(args, environ={L.CODEX_BUNDLE_DIR_ENV: str(root)}) - with pytest.raises(ValueError, match=L.CODEX_COMPACTION_ENV): - L.codex_harness_launch(args, environ={ - L.CODEX_BUNDLE_DIR_ENV: str(root), L.HARNESS_ENVIRONMENT_PROVIDER_ENV: PROVIDER, - L.CODEX_COMPACTION_ENV: "1"}) - with pytest.raises(ValueError, match="not a directory"): - L.codex_harness_launch(args, environ={ - L.CODEX_BUNDLE_DIR_ENV: str(root / "missing"), L.HARNESS_ENVIRONMENT_PROVIDER_ENV: PROVIDER}) - - -def test_codex_launch_flags_env_and_mount_come_from_the_contract(bundle): - root, seen = bundle - args = _codex_args() - flags, envs, mounts = L.codex_harness_launch(args) - # The attestation sees the three bundle files, named as the SSH harness CLI does. - assert seen["namespace"] == { - "codex_harness_binary": str(root / "codex-x86_64-unknown-linux-musl"), - "codex_package_manifest": str(root / "codex-package.json"), - "codex_app_server_schema": str(root / "codex_app_server_protocol.v2.schemas.json"), - } - contract = _synthetic_contract(args) - parts = shlex.split(flags) - assert parts[0] == "--codex-harness-contract" and json.loads(parts[1]) == contract - assert mounts == {L.CODEX_CONTAINER_DIR: str(root.resolve())} - assert L.CODEX_CONTAINER_DIR == str(Path(CODEX_CONTAINER_BINARY_PATH).parent) - # Every env the container preflights compare is present, with contract values. - assert LEARNER_EXPECTED_ENV <= set(envs) - assert set(CODEX_OPENENV_IDENTITY_ENV) <= set(envs) - assert envs["YETO_CODEX_BINARY_PATH"] == CODEX_CONTAINER_BINARY_PATH - assert envs["YETO_CODEX_BINARY_SIZE_BYTES"] == "310730800" - assert envs["YETO_CODEX_VERSION"] == "codex-cli 0.145.0" - assert envs["YETO_CODEX_BACKEND_MAX_TOKENS"] == str(contract["backend"]["max_tokens"]) - assert envs["YETO_CODEX_CHAT_TEMPLATE_KWARGS"] == json.dumps( - contract["backend"]["chat_template_kwargs"], sort_keys=True, separators=(",", ":")) - assert envs["YETO_CODEX_HARNESS_CONTRACT_SHA256"] == _plan_digest(contract) - assert envs["YETO_CODEX_REASONING_EFFORT"] == "xhigh" - # Harness hook + provider reach the island; compaction stays unset (R-D5a). - assert envs[L.HARNESS_PREFLIGHT_ENV] == "yeto.rl.harness.codex.preflight:harness_preflight" - assert envs[L.HARNESS_ENVIRONMENT_PROVIDER_ENV] == PROVIDER - assert L.CODEX_COMPACTION_ENV not in envs - - -def test_sky_island_task_mounts_the_bundle_and_exports_the_env(bundle): - root, _ = bundle - args = _codex_args() - task = make_miles_island_task(args, parse_gpu_spec(args.gpu)[0], 0, 1, "127.0.0.1:29400") - assert "--codex-harness-contract " in task.run - assert f"--custom-agent-function-path {CODEX_OPENENV_AGENT}" in task.run - assert "--agent-max-seq-len 128" in task.run and "--use-session-server" in task.run - assert "--tito-model qwen35" in task.run and "--codex-backend-profile qwen35_08b" in task.run - assert "--partial-rollout" not in task.run - assert task.file_mounts[L.CODEX_CONTAINER_DIR] == str(root.resolve()) - assert task.envs["YETO_CODEX_BINARY_PATH"] == CODEX_CONTAINER_BINARY_PATH - assert task.envs[L.HARNESS_PREFLIGHT_ENV].endswith(":harness_preflight") - assert task.envs[L.HARNESS_ENVIRONMENT_PROVIDER_ENV] == PROVIDER - assert "YETO_CODEX_COMPACTION_ENABLED" not in task.envs - - -def test_modal_island_mounts_the_bundle_through_the_image(bundle, monkeypatch): - root, _ = bundle - args = _codex_args() - args.gpu = "modal:1xh100" - args.rl_image = "docker:ghcr.io/x/miles@sha256:" + "c" * 64 - spec = parse_gpu_spec(args.gpu)[0] - task = make_miles_island_task(args, spec, 0, 1, "127.0.0.1:29400") - cfg = build_modal_island_config(args, spec, 0, task, "1.2.3.4:29400") - assert cfg.codex_dir == str(root.resolve()) and cfg.codex_mount == L.CODEX_CONTAINER_DIR - assert cfg.envs["YETO_CODEX_BINARY_PATH"] == CODEX_CONTAINER_BINARY_PATH - state = fake_modal(monkeypatch) - monkeypatch.setattr(mr, "registry_credentials", lambda *_a, **_k: None) - mr.ModalOps("yeto-run").define(cfg) - (img,) = state["images"] - names = [c[0] for c in img.calls] - assert names == ["from_registry", "env", "add_local_dir", "add_local_dir"] - assert img.calls[-1][1] == (cfg.codex_dir, L.CODEX_CONTAINER_DIR) and img.calls[-1][2] == {"copy": False} - - -def test_modal_config_requires_an_existing_bundle_dir(tmp_path): - with pytest.raises(ValueError, match="go together"): - mr.ModalIslandConfig(app_name="a", learner_id=0, training_mode="sft", gpu="H100", - gpus_per_node=8, num_nodes=1, run_script="x", codex_dir=str(tmp_path)).validate() - with pytest.raises(ValueError, match="not a directory"): - mr.ModalIslandConfig(app_name="a", learner_id=0, training_mode="sft", gpu="H100", - gpus_per_node=8, num_nodes=1, run_script="x", - codex_dir=str(tmp_path / "nope"), codex_mount="/opt/yeto/codex").validate() - - -@pytest.mark.skipif( - not os.environ.get(L.CODEX_BUNDLE_DIR_ENV) or not os.path.isdir(os.environ.get(L.CODEX_BUNDLE_DIR_ENV, "")), - reason="YETO_CODEX_BUNDLE_DIR not set to a fetched bundle (scripts/fetch_codex_bundle.py)", -) -def test_real_bundle_attests_against_the_pins(): - args = _codex_args() - try: - contract = L.codex_bundle_contract(args, os.environ[L.CODEX_BUNDLE_DIR_ENV]) - except ValueError as exc: - if "app-server schema" in str(exc): - pytest.xfail(f"pinned app-server schema file not reproducible from the npm artifact: {exc}") - raise - assert contract["binary_size_bytes"] == 310_730_800 - assert contract["openenv_identity_env"] == CODEX_OPENENV_IDENTITY_ENV - - -def test_modal_island_mounts_local_data_and_passes_tb2_env(bundle, monkeypatch, tmp_path): - root, _ = bundle - data = tmp_path / "smoke.jsonl" - data.write_text('{"prompt": "x", "metadata": {"task_id": "fix-git"}}\n') - monkeypatch.setenv("MODAL_TOKEN_ID", "ak-test") - monkeypatch.setenv("MODAL_TOKEN_SECRET", "as-test") - monkeypatch.setenv("YETO_SANDBOX_MODAL_TOKEN_ID", "sb-id") - monkeypatch.setenv("YETO_SANDBOX_MODAL_TOKEN_SECRET", "sb-secret") - monkeypatch.setenv("YETO_HARNESS_TB2_TASKS_DIR", "/opt/yeto/codex/tb2-tasks") - monkeypatch.setenv("YETO_HARNESS_TB2_FAULT", "create_fail:2") - args = _codex_args() - args.data = str(data) - args.gpu = "modal:1xl40s" - args.rl_image = "docker:ghcr.io/x/miles@sha256:" + "c" * 64 - spec = parse_gpu_spec(args.gpu)[0] - task = make_miles_island_task(args, spec, 0, 1, "127.0.0.1:29400") - cfg = build_modal_island_config(args, spec, 0, task, "1.2.3.4:29400") - assert cfg.extra_mounts == {"/root/yeto-data.jsonl": str(data)} - for name in ("YETO_SANDBOX_MODAL_TOKEN_ID", "YETO_SANDBOX_MODAL_TOKEN_SECRET", - "YETO_HARNESS_TB2_TASKS_DIR", "YETO_HARNESS_TB2_FAULT"): - assert cfg.envs[name] == os.environ[name] - # secret-handling-hardening: the main Modal token never reaches the island. - assert not {"MODAL_TOKEN_ID", "MODAL_TOKEN_SECRET"} & set(cfg.envs) - assert not {"YETO_SANDBOX_MODAL_TOKEN_ID", "YETO_SANDBOX_MODAL_TOKEN_SECRET"} & set(task.envs) - assert mr.ModalIslandConfig.from_json(cfg.to_json()).extra_mounts == cfg.extra_mounts - state = fake_modal(monkeypatch) - monkeypatch.setattr(mr, "registry_credentials", lambda *_a, **_k: None) - mr.ModalOps("yeto-run").define(cfg) - (img,) = state["images"] - assert img.calls[-1][0] == "add_local_file" - assert img.calls[-1][1] == (str(data), "/root/yeto-data.jsonl") and img.calls[-1][2] == {"copy": False} - - -def test_learner_command_forwards_the_codex_reasoning_effort(bundle): - args = _codex_args() - spec = parse_gpu_spec(args.gpu)[0] - task = make_miles_island_task(args, spec, 0, 1, "127.0.0.1:29400") - assert " --codex-reasoning-effort xhigh" in task.run - assert " --codex-backend-profile qwen35_08b" in task.run - - -def test_island_setup_installs_the_modal_client_only_for_the_modal_sandbox_provider(bundle, monkeypatch): - """A-T3-5 (codex-smoke-20261003-7): rollout workers create task sandboxes with - the Modal client, which the ports image does not ship; `import modal` there - resolved Modal's dependency-less runtime mount and failed on every acquire.""" - args = _codex_args() - spec = parse_gpu_spec(args.gpu)[0] - plain = make_miles_island_task(args, spec, 0, 1, "127.0.0.1:29400") - assert L.MODAL_CLIENT_SETUP not in plain.setup # local/other providers: untouched - - monkeypatch.setenv(L.HARNESS_ENVIRONMENT_PROVIDER_ENV, L.MODAL_SANDBOX_PROVIDER) - with pytest.raises(ValueError, match="YETO_SANDBOX_MODAL_TOKEN_ID"): - make_miles_island_task(args, spec, 0, 1, "127.0.0.1:29400") - monkeypatch.setenv("YETO_SANDBOX_MODAL_TOKEN_ID", "sb-id") - monkeypatch.setenv("YETO_SANDBOX_MODAL_TOKEN_SECRET", "sb-secret") - task = make_miles_island_task(args, spec, 0, 1, "127.0.0.1:29400") - assert task.envs[L.HARNESS_ENVIRONMENT_PROVIDER_ENV] == L.MODAL_SANDBOX_PROVIDER - assert L.MODAL_CLIENT_SETUP in task.setup - assert "--no-deps --ignore-installed" in L.MODAL_CLIENT_SETUP and "modal==1.5.5" in L.MODAL_CLIENT_SETUP - assert 'sysconfig.get_paths()["purelib"]' in L.MODAL_CLIENT_SETUP # must land in the island python itself - # the check runs with the run script's PYTHONPATH, where Modal's own runtime mount is not enough - assert "$HOME/miles:$HOME/sglang/python:$HOME/sky_workdir" in L.MODAL_CLIENT_SETUP - assert "import modal, grpclib, synchronicity" in L.MODAL_CLIENT_SETUP - # the client install precedes the prefetch and fails the setup when unusable - assert task.setup.index(L.MODAL_CLIENT_SETUP) > task.setup.index("pip install -q --no-deps -e ~/miles") - assert "exit 1" in L.MODAL_CLIENT_SETUP - cfg = build_modal_island_config(args, spec, 0, task, "1.2.3.4:29400") - assert L.MODAL_CLIENT_SETUP in cfg.setup_script - - -def test_modal_sandbox_secret_attaches_a_named_secret_and_checks_in_the_island(bundle, monkeypatch): - """--modal-sandbox-secret: the sandbox token comes from a named Modal Secret - (never through the launcher); the island setup fails closed without it.""" - for name in L.SANDBOX_MODAL_TOKEN_ENVS: - monkeypatch.delenv(name, raising=False) - monkeypatch.setenv("MODAL_TOKEN_ID", "ak-test") - monkeypatch.setenv("MODAL_TOKEN_SECRET", "as-test") - monkeypatch.setenv(L.HARNESS_ENVIRONMENT_PROVIDER_ENV, L.MODAL_SANDBOX_PROVIDER) - args = _codex_args() - args.gpu = "modal:1xl40s" - args.rl_image = "docker:ghcr.io/x/miles@sha256:" + "c" * 64 - spec = parse_gpu_spec(args.gpu)[0] - with pytest.raises(ValueError, match="YETO_SANDBOX_MODAL_TOKEN_ID"): - make_miles_island_task(args, spec, 0, 1, "127.0.0.1:29400") # unchanged without the flag - args.modal_sandbox_secret = "yeto-sandbox-modal" - task = make_miles_island_task(args, spec, 0, 1, "127.0.0.1:29400") - assert L.SANDBOX_TOKEN_IN_ISLAND_CHECK in task.setup - assert task.setup.index(L.SANDBOX_TOKEN_IN_ISLAND_CHECK) < task.setup.index(L.MODAL_CLIENT_SETUP) - cfg = build_modal_island_config(args, spec, 0, task, "1.2.3.4:29400") - assert cfg.named_secrets == ("yeto-sandbox-modal",) - assert not set(L.SANDBOX_MODAL_TOKEN_ENVS) & set(cfg.envs) - assert L.SANDBOX_TOKEN_IN_ISLAND_CHECK in cfg.setup_script - assert mr.ModalIslandConfig.from_json(cfg.to_json()).named_secrets == ("yeto-sandbox-modal",) - state = fake_modal(monkeypatch) - monkeypatch.setattr(mr, "registry_credentials", lambda *_a, **_k: None) - mr.ModalOps("yeto-run").define(cfg) - (_fn, kwargs), = state["functions"].values() - assert ("named-secret", "yeto-sandbox-modal") in kwargs["secrets"] - - -def test_modal_sandbox_secret_in_island_check_fails_closed_without_values(tmp_path): - import subprocess - - env = {"PATH": os.environ["PATH"], "YETO_SANDBOX_MODAL_TOKEN_ID": "ak-x"} - r = subprocess.run(["bash", "-c", L.SANDBOX_TOKEN_IN_ISLAND_CHECK], env=env, - capture_output=True, text=True) - assert r.returncode == 1 and "YETO_SANDBOX_MODAL_TOKEN_SECRET missing" in r.stderr - assert "ak-x" not in r.stderr + r.stdout - env["YETO_SANDBOX_MODAL_TOKEN_SECRET"] = "as-y" - r = subprocess.run(["bash", "-c", L.SANDBOX_TOKEN_IN_ISLAND_CHECK], env=env, - capture_output=True, text=True) - assert r.returncode == 0 and r.stdout + r.stderr == "" - - -def test_modal_sandbox_secret_is_refused_for_non_modal_islands(bundle, monkeypatch): - args = _codex_args() - args.modal_sandbox_secret = "yeto-sandbox-modal" - spec = parse_gpu_spec(args.gpu)[0] - assert spec.cloud != "modal" - with pytest.raises(ValueError, match="--modal-sandbox-secret"): - make_miles_island_task(args, spec, 0, 1, "127.0.0.1:29400") - with pytest.raises(ValueError, match="cannot use it"): - L.check_cloud_prerequisites([spec], project_ids={}, args=args, modal_ok=True) - args.modal_sandbox_secret = "bad name;rm" - with pytest.raises(ValueError, match="not a Modal Secret name"): - L.modal_sandbox_secret_name(args, parse_gpu_spec("modal:1xl40s")[0]) - - -def test_modal_island_ships_real_secret_values_not_sky_masks(bundle, monkeypatch, tmp_path): - """sky.Task keeps secrets as pydantic SecretStr ("**********"); the Modal - island must get the plain value (s19-compaction-g1-20261010a).""" - pydantic = pytest.importorskip("pydantic") - masked = pydantic.SecretStr("k" * 48) # what sky>=0.10 Task.secrets holds - assert str(masked) != "k" * 48 - task = types.SimpleNamespace(envs={"A": "1"}, secrets={"HF_TOKEN": masked}, - setup="", run="echo") - monkeypatch.setenv("MODAL_TOKEN_ID", "ak-test") - monkeypatch.setenv("MODAL_TOKEN_SECRET", "as-test") - args = _codex_args() - args.gpu = "modal:1xl40s" - args.rl_image = "docker:ghcr.io/x/miles@sha256:" + "c" * 64 - spec = parse_gpu_spec(args.gpu)[0] - cfg = build_modal_island_config(args, spec, 0, task, "1.2.3.4:29400") - assert cfg.envs["HF_TOKEN"] == "k" * 48 - assert L._secret_value("plain") == "plain" diff --git a/tests/test_rl_launcher_dense_full.py b/tests/test_rl_launcher_dense_full.py index 1ce3a062a..a246dac3b 100644 --- a/tests/test_rl_launcher_dense_full.py +++ b/tests/test_rl_launcher_dense_full.py @@ -132,27 +132,6 @@ def test_island_entry_accepts_the_dense_island_command(): island_entry._check_dense_full_sweep(parsed, yeto_policy_sync=True) -@pytest.mark.parametrize( - ("reward_function", "kind"), - ( - ("yeto.rl.harness.codex.reward:reward_func", "secrlenv"), - ("yeto.rl.gsm8k_reward:score", "reward-function"), - ("yeto.rl.math_reward:reward_func", "reward-function"), - ), -) -def test_dense_island_evidence_kind_follows_its_reward_function(reward_function, kind): - from types import SimpleNamespace - - from yeto.rl.adapters.miles import island_entry - from yeto.rl.trajectory_evidence import EVIDENCE_KINDS - - chosen = island_entry._trajectory_evidence_kind( - SimpleNamespace(reward_function=reward_function) - ) - assert chosen == kind - assert chosen in EVIDENCE_KINDS - - EXPERT_FULL = ( "--rl-sync-preset", "full-parameter", "--rl-placement", "fixed-partition", "--rl-rollout-gpus", "8", "--rl-parameter-layout-sha256", LAYOUT, "--fragments", "4", diff --git a/tests/test_rl_miles_adapter_config.py b/tests/test_rl_miles_adapter_config.py index 503fe2e28..825e2fd0e 100644 --- a/tests/test_rl_miles_adapter_config.py +++ b/tests/test_rl_miles_adapter_config.py @@ -647,31 +647,18 @@ def test_lr_schedule_flags_are_adapter_owned(): mc.check_extra_argv((flag, "1")) -# ---------------------------------------------------------------- rl-fn-codex-rollout 1.0: append roles by tokenizer family +# ---------------------------------------------------------------- append roles are decided by the --tito-model template -@pytest.mark.parametrize("tito_model,ok", [("qwen35", True), ("qwen4exp", True), ("qwen38small", True), ("deepseekv4", True), - ("nosuchfamily", False)]) -def test_stock_codex_append_roles_resolve_by_tito_family(tito_model, ok): - """``--tito-model`` is a tokenizer family (``qwen4exp`` for Flash-Next), not a - profile name; the s15 4-layer smoke failed with UnmappedConfigError here.""" +def test_explicit_append_roles_are_refused(): + """Since S26 E0.5 no signed Codex agent may state the append roles.""" from types import SimpleNamespace - from yeto.rl import CODEX_OPENENV_AGENT from yeto.rl.adapters.miles.config import LEAF_POLICY check = LEAF_POLICY["agent.tito_allowed_append_roles"] - agent = SimpleNamespace(custom_agent_function_path=CODEX_OPENENV_AGENT, tito_model=tito_model) - cfg = SimpleNamespace(agent=agent) + cfg = SimpleNamespace(agent=SimpleNamespace(custom_agent_function_path="x:y", tito_model="qwen35")) assert check(None, cfg) is None - if ok: - assert check(("tool", "user"), cfg) is None - assert "fix the append roles" in check(("tool",), cfg) - else: - assert "upstream Miles has no --tito-allowed-append-roles" in check(("tool", "user"), cfg) - # an unsigned agent is still refused - cfg.agent.custom_agent_function_path = "x:y" - assert check(("tool", "user"), cfg) is not None - + assert "upstream Miles has no --tito-allowed-append-roles" in check(("tool", "user"), cfg) # --- S17 N16: --rl-lr-schedule ---------------------------------------------------- diff --git a/tests/test_rl_miles_carry_over.py b/tests/test_rl_miles_carry_over.py index e94f3d267..a83ad2a90 100644 --- a/tests/test_rl_miles_carry_over.py +++ b/tests/test_rl_miles_carry_over.py @@ -249,7 +249,7 @@ def test_miles_stage_two_support_and_single_turn_check(): check_task(1) check_task(0, custom_agent="x.y", custom_generate="x.z") with pytest.raises(PolicyAgeError, match="stage 3"): - check_task(1, custom_generate="yeto.rl.harness.codex.codex_openenv_generate.generate") + check_task(1, custom_generate="yeto.rl.adapters.miles.records_generate.generate") with pytest.raises(PolicyAgeError, match="recompute"): check_task(1, recompute_prefill=True) @@ -259,7 +259,7 @@ def test_launcher_refuses_agentic_carry_over_before_launch(): from yeto.launcher import _prepare_rl_args args = _args(["--rl-max-policy-age", "1", "--custom-generate-function-path", - "yeto.rl.harness.codex.codex_openenv_generate.generate"]) + "yeto.rl.adapters.miles.records_generate.generate"]) with pytest.raises(PolicyAgeError, match="stage 3"): _prepare_rl_args(args) diff --git a/tests/test_rl_neutral_rewards.py b/tests/test_rl_neutral_rewards.py index 748a644a7..b150750db 100644 --- a/tests/test_rl_neutral_rewards.py +++ b/tests/test_rl_neutral_rewards.py @@ -139,73 +139,6 @@ def test_generic_miles_wrapper_equals_original_entry_point(name, entry): assert a.snapshot() == b.snapshot(), (name, i) -# -- 3.3 codex: abort as a reward result -------------------------------------------- - -def _outcome_metadata(status: str, reward: float) -> dict: - from yeto.rl.harness.codex import reward as codex_reward - - outcome = {"schema": 1, "status": status, "episode_id": "ep", "task_id": "t", "reward": reward, - "passed": reward == 1.0, "class": None, - codex_reward.INFRASTRUCTURE_503_RETRIES_KEY: 0} - return {codex_reward.OUTCOME_KEY: outcome} - - -@pytest.fixture -def fake_miles_sample(monkeypatch): - names = ("miles", "miles.utils", "miles.utils.types") - for n in names[:-1]: - monkeypatch.setitem(sys.modules, n, types.ModuleType(n)) - mod = types.ModuleType(names[-1]) - mod.Sample = cases.FakeSample - monkeypatch.setitem(sys.modules, names[-1], mod) - - -def test_codex_infrastructure_outcome_aborts_in_neutral_and_miles(monkeypatch, fake_miles_sample): - from yeto.rl.harness.codex import reward as codex_reward - - metadata = _outcome_metadata("infrastructure_error", 0.0) - result = codex_reward.secrlenv_reward(Trajectory(metadata=metadata)) - assert (result.value, result.aborted, result.reason) == (0.0, True, "secrlenv_infrastructure_failure") - sample = cases.FakeSample(metadata=dict(metadata)) - assert _run(codex_reward.reward_func(None, sample)) == 0.0 - # The SecRLEnv entry marks FAILED, not ABORTED: Miles drops an aborted - # group before the SecRLEnv group check could schedule its retry. - assert sample.status is cases.Status.FAILED - generic = cases.FakeSample(metadata=dict(metadata)) - assert _run(miles_rewards.miles_reward(codex_reward.secrlenv_reward)(None, generic)) == 0.0 - assert generic.status is cases.Status.ABORTED - - -def test_codex_verdict_and_invalid_outcome(monkeypatch, fake_miles_sample): - from yeto.rl.harness.codex import reward as codex_reward - - metadata = _outcome_metadata("completed", 1.0) - result = codex_reward.secrlenv_reward(Trajectory(metadata=metadata)) - assert (result.value, result.aborted) == (1.0, False) - sample = cases.FakeSample(metadata=dict(metadata)) - assert _run(codex_reward.reward_func(None, [sample])) == [1.0] - assert sample.status is cases.Status.COMPLETED - legacy = {**metadata, "secrlenv_trusted_outcome_hmac": "0" * 64} - assert codex_reward.secrlenv_reward(Trajectory(metadata=legacy)) == result - tampered = {codex_reward.OUTCOME_KEY: {**metadata[codex_reward.OUTCOME_KEY], "reward": 1.5}} - with pytest.raises(codex_reward.UntrustedOutcome): - codex_reward.secrlenv_reward(Trajectory(metadata=tampered)) - with pytest.raises(codex_reward.UntrustedOutcome): - _run(codex_reward.reward_func(None, cases.FakeSample(metadata=tampered))) - - -def test_tbench_infrastructure_marker_aborts(monkeypatch): - from yeto.rl.harness.codex import tbench_reward - - metadata = {tbench_reward.INFRASTRUCTURE_KEY: "x", "success": True} - result = tbench_reward.tbench_reward(Trajectory(metadata=metadata)) - assert result.aborted and result.value == 0.0 and "success" not in metadata - sample = SimpleNamespace(metadata={tbench_reward.INFRASTRUCTURE_KEY: "x", "success": True}, - status=None) - assert _run(tbench_reward.reward_func(None, sample)) == 0.0 - assert sample.status == "ABORTED" and "success" not in sample.metadata - - # -- 3.10 registry and user rewards ------------------------------------------------ def test_custom_reward_via_miles_equals_direct_call(): diff --git a/tests/test_rl_policy_versions.py b/tests/test_rl_policy_versions.py index f600d50dc..b994c6cbb 100644 --- a/tests/test_rl_policy_versions.py +++ b/tests/test_rl_policy_versions.py @@ -191,19 +191,3 @@ def test_fully_async_trained_groups_hook_reports_the_round(monkeypatch): assert len(reported) == 1 # the all-samples hook reports otherwise -def test_rollout_side_matches_plain_versions_against_the_published_label(): - from yeto.rl.adapters.miles import carry_over - from yeto.rl.adapters.miles.rollout_meta_hook import group_matches_policy - from yeto.rl.harness.codex.codex_openenv_generate import window_problem - - def sample(version): - span = SimpleNamespace(version=version, abs_start=1, abs_end=2) - return SimpleNamespace(status=SimpleNamespace(value="completed"), tokens=[0, 1], - response_length=1, weight_versions=[SimpleNamespace(spans=[span])]) - - label = f"yeto:6:{H6}" - assert group_matches_policy([sample("6"), sample("6")], label, 2) - assert not group_matches_policy([sample("6"), sample("5")], label, 2) - assert carry_over.response_token_versions(sample("5")) == [5] - assert window_problem(["5", "6"], label, 1) is None - assert "behind" in window_problem(["4"], label, 1) diff --git a/tests/test_rl_reward_success.py b/tests/test_rl_reward_success.py index 84ab6b78d..13b556dc2 100644 --- a/tests/test_rl_reward_success.py +++ b/tests/test_rl_reward_success.py @@ -2,7 +2,6 @@ import asyncio import importlib.util -import re import sys import types from pathlib import Path @@ -110,24 +109,3 @@ def _tb_sample(monkeypatch, reward, status="completed", verifier=None, rc=None): return SimpleNamespace(metadata=meta, status=None) -def test_tbench_success_is_outcome_pass_bit(monkeypatch): - from yeto.rl import tbench_outcome - from yeto.rl.harness.codex import tbench_reward - - ok = _tb_sample(monkeypatch, 1.0, verifier=tbench_outcome.TEST_SH_VERIFIER, rc=0) - bad = _tb_sample(monkeypatch, 0.0) - timeout = _tb_sample(monkeypatch, 0.0, status="timeout", verifier=tbench_outcome.TIMEOUT_VERIFIER) - bad.metadata["success"] = True # untrusted pre-set flag is overwritten - assert _run(tbench_reward.reward_func(None, [ok, bad, timeout])) == [1.0, 0.0, 0.0] - assert [s.metadata["success"] for s in (ok, bad, timeout)] == [True, False, False] - assert all(_fork_flag(s.metadata) in (0, 1) for s in (ok, bad, timeout)) - # siblings / group filter still accept the extra key - tbench_reward.check_siblings([ok, bad]) - - -def test_tbench_infrastructure_sample_has_no_success(monkeypatch): - from yeto.rl.harness.codex import tbench_reward - - sample = SimpleNamespace(metadata={tbench_reward.INFRASTRUCTURE_KEY: "boom", "success": True}, status=None) - assert _run(tbench_reward.reward_func(None, sample)) == 0.0 - assert sample.status == "ABORTED" and "success" not in sample.metadata diff --git a/tests/test_rl_sao_streaming_runtime.py b/tests/test_rl_sao_streaming_runtime.py index ced8e08eb..737c786cd 100644 --- a/tests/test_rl_sao_streaming_runtime.py +++ b/tests/test_rl_sao_streaming_runtime.py @@ -595,7 +595,7 @@ def test_context_schema_label_2_still_writes_trajectory_evidence_schema_4(tmp_pa persist_published_round_evidence, ) from yeto.rl.engine.policy_versions import PublishedPolicies - from yeto.rl.harness.codex import reward + from yeto.rl import secrlenv_outcome as reward from yeto.rl.trajectory_evidence import ( read_trajectory_batch_evidence, trajectory_batch_evidence_path, diff --git a/tests/test_rl_secrlenv_ports_retry.py b/tests/test_rl_secrlenv_ports_retry.py deleted file mode 100644 index f64ff17bf..000000000 --- a/tests/test_rl_secrlenv_ports_retry.py +++ /dev/null @@ -1,509 +0,0 @@ -"""P0-1.7b.4-Q1: the SecRLEnv group check and same-task retry on ports. - -CPU only: the algorithm spec accepts the group check, the Miles translation -binds the retry classes, the retry data source keeps the legacy contract, the -terminal cleanup sanitizes errors, and the Miles classes load against stub -Miles modules. -""" - -from __future__ import annotations - -import asyncio -import importlib -import sys -import types -from types import SimpleNamespace - -import pytest - -from test_rl_miles_adapter_config import flag_value, make_config, sub -from yeto.rl import ( - SECRLENV_AGENT, - SECRLENV_GENERATE, - SECRLENV_GROUP_FILTER, -) -from yeto.rl.adapters.miles import config as cfg -from yeto.rl.adapters.miles import secrlenv_retry as retry -from yeto.rl.adapters.miles.binding import check_filter -from yeto.rl.adapters.miles.entry import miles_capabilities -from yeto.rl.engine.algorithm import SECRLENV_GROUP_CHECK_FILTER, AlgorithmSpec -from yeto.rl.harness.codex.reward import ( - SecRLEnvReplacementExhausted, - SecRLEnvRolloutCleanupError, - SecRLEnvUntrustedEvidence, - UntrustedOutcome, -) - -TARGET = 2 - - -# ------------------------------------------------------------------ spec -def _secrlenv_spec() -> AlgorithmSpec: - return AlgorithmSpec( - advantage_estimator="grpo", - dynamic_sampling_filter=SECRLENV_GROUP_FILTER, - dynamic_sampling_max_replacements=0, - ) - - -def test_spec_accepts_the_group_check_under_its_neutral_name(): - spec = _secrlenv_spec() - assert spec.dynamic_sampling_filter == SECRLENV_GROUP_CHECK_FILTER - assert spec.sha256() == AlgorithmSpec( - advantage_estimator="grpo", - dynamic_sampling_filter=SECRLENV_GROUP_CHECK_FILTER, - dynamic_sampling_max_replacements=0, - ).sha256() - legacy_args = SimpleNamespace(dynamic_sampling_filter_path=SECRLENV_GROUP_FILTER, - dynamic_sampling_max_replacements=0, kl_coef=None) - assert AlgorithmSpec.from_legacy_args(legacy_args).sha256() == spec.sha256() - capabilities = miles_capabilities("sha256:" + "0" * 64) - assert SECRLENV_GROUP_CHECK_FILTER in capabilities.dynamic_sampling_filters - assert ("dynamic_sampling_filters", SECRLENV_GROUP_CHECK_FILTER) in spec.required_mechanisms() - - -def test_binding_check_skips_only_the_evidence_call(): - row = check_filter(SECRLENV_GROUP_CHECK_FILTER) - assert (row["mapped"], row["argv"]) == (SECRLENV_GROUP_FILTER, "same") - assert row["identity"]["source_sha256"] - assert row["call"] == "skipped: needs environment evidence" - - -# ----------------------------------------------------------- translation -def _secrlenv_config(**batch): - config = sub(make_config(), "agent", - custom_generate_function_path=SECRLENV_GENERATE, - custom_agent_function_path=SECRLENV_AGENT, - dynamic_sampling_filter_path=SECRLENV_GROUP_FILTER) - groups = make_config().batch.groups_per_round - return sub(config, "batch", **{"over_sampling_batch_size": groups + 1, **batch}) - - -def test_translation_binds_the_retry_classes(): - argv = list(cfg.translate_run_config(_secrlenv_config(), _secrlenv_spec()).argv) - assert flag_value(argv, "--dynamic-sampling-filter-path") == SECRLENV_GROUP_FILTER - assert flag_value(argv, "--data-source-path") == retry.DATA_SOURCE_PATH - assert flag_value(argv, "--rollout-function-path") == retry.ROLLOUT_FUNCTION_PATH - assert flag_value(argv, "--custom-generate-function-path") == SECRLENV_GENERATE - assert flag_value(argv, "--custom-agent-function-path") == SECRLENV_AGENT - - -def test_translation_leaves_other_runs_unchanged(): - argv = cfg.translate_run_config(make_config(), AlgorithmSpec()).argv - assert "--data-source-path" not in argv and "--rollout-function-path" not in argv - - -def test_retry_classes_are_adapter_owned(): - assert {"--data-source-path", "--rollout-function-path"} <= cfg.ADAPTER_OWNED_FLAGS - - -@pytest.mark.parametrize("config, spec, match", [ - (sub(_secrlenv_config(), "agent", dynamic_sampling_filter_path=None), AlgorithmSpec(), - "go together"), - (sub(make_config(), "agent", dynamic_sampling_filter_path=SECRLENV_GROUP_FILTER), - _secrlenv_spec(), "go together"), - (_secrlenv_config(over_sampling_batch_size=make_config().batch.groups_per_round), - _secrlenv_spec(), "exactly one refill slot"), -]) -def test_translation_refuses_a_broken_retry_contract(config, spec, match): - with pytest.raises(cfg.UnmappedConfigError, match=match): - cfg.translate_run_config(config, spec) - - -def test_retry_refuses_a_policy_age_limit(): - with pytest.raises(ValueError, match="policy-age limit"): - retry.rollout_argv(custom_agent=SECRLENV_AGENT, - dynamic_filter=SECRLENV_GROUP_CHECK_FILTER, - rollout_batch_size=TARGET, over_sampling_batch_size=TARGET + 1, - max_policy_age=1) - - -# ------------------------------------------------------- retry data source -class _Sample: - def __init__(self, index, status="completed"): - self.index = index - self.status = status - self.metadata = {"secrlenv_trusted_outcome": {"index": index}} - self.resets = 0 - - def reset_for_retry(self): - self.status = "pending" - self.resets += 1 - - -class _Source: - """Stands in for Miles' RolloutDataSourceWithBuffer.""" - - def __init__(self, args): - self.args = args - self.next_index = 0 - self.buffer = [] - self.draws = [] - - def get_samples(self, num_samples): - self.draws.append(num_samples) - groups = [[_Sample(self.next_index + g * 10 + s) for s in range(2)] - for g in range(num_samples)] - self.next_index += 100 - return groups - - def add_samples(self, samples): - self.buffer.extend(samples) - - -class _RetrySource(retry.SecRLEnvTaskRetry, _Source): - pass - - -def _args(**overrides): - values = dict(rollout_batch_size=TARGET, over_sampling_batch_size=TARGET + 1, - yeto_rl_secrlenv_max_infrastructure_replacements=1) - values.update(overrides) - return SimpleNamespace(**values) - - -def _ledger(group, value=0): - return {sample.index: value for sample in group} - - -def _request(source, group, *, key="evidence", reason="secrlenv_infrastructure_failure", - ledger=None): - source.request_retry(group, evidence_key=key, reason=reason, - infrastructure_503_retries=_ledger(group) if ledger is None else ledger) - - -@pytest.mark.parametrize("overrides", [ - dict(over_sampling_batch_size=TARGET), - dict(rollout_batch_size=0, over_sampling_batch_size=1), - dict(yeto_rl_secrlenv_max_infrastructure_replacements=2), - dict(yeto_rl_secrlenv_max_infrastructure_replacements=None), -]) -def test_data_source_refuses_a_broken_contract(overrides): - with pytest.raises(SecRLEnvReplacementExhausted, match="contract is invalid"): - _RetrySource(_args(**overrides)) - - -def test_round_draws_the_exact_batch_then_only_the_queued_retry(): - args = _args() - source = _RetrySource(args) - assert getattr(args, retry.RETRY_CALLBACK_ATTR) == source.request_retry - groups = source.get_samples(TARGET + 1) - assert len(groups) == TARGET and source.draws == [TARGET] - failed = groups[1] - failed[0].status = "failed" - failed[0].metadata["secrlenv_trusted_outcome"] = {"status": "infrastructure_error"} - _request(source, failed, ledger={failed[0].index: 1, failed[1].index: 0}) - _request(source, failed, ledger={failed[0].index: 1, failed[1].index: 0}) # repeated callback - (rerun,) = source.get_samples(TARGET + 1) - assert [s.index for s in rerun] == [s.index for s in failed] - assert [s.resets for s in rerun] == [1, 1] - assert [s.metadata for s in rerun] == [ - {"secrlenv_infrastructure_503_retries": 1}, - {"secrlenv_infrastructure_503_retries": 0}, - ] - assert source.draws == [TARGET] # no new task was drawn for the retry - assert source.retry_stats() == dict( - replacement_attempts=1, nontrainable_groups=1, infrastructure_groups=1, - pending_groups=0, failed_groups=1) - assert retry.retry_metrics(args._yeto_secrlenv_retry_stats)[ - "rl/secrlenv/replacement_attempts"] == 1.0 - - -def test_refill_without_a_queued_retry_is_terminal(): - source = _RetrySource(_args()) - source.get_samples(TARGET + 1) - with pytest.raises(SecRLEnvReplacementExhausted, match="no authenticated task retry"): - source.get_samples(TARGET + 1) - - -def test_refill_size_is_part_of_the_contract(): - source = _RetrySource(_args()) - with pytest.raises(SecRLEnvReplacementExhausted, match="refill contract"): - source.get_samples(TARGET) - - -def test_second_retry_in_a_round_is_terminal(): - source = _RetrySource(_args()) - groups = source.get_samples(TARGET + 1) - _request(source, groups[0], key="a") - source.get_samples(TARGET + 1) - with pytest.raises(SecRLEnvReplacementExhausted, match="budget was exhausted"): - _request(source, groups[1], key="b") - - -@pytest.mark.parametrize("ledger, error, match", [ - ({}, SecRLEnvUntrustedEvidence, "ledger authentication"), - ("bad", SecRLEnvUntrustedEvidence, "ledger authentication"), - (None, SecRLEnvReplacementExhausted, "identity did not match"), -]) -def test_retry_must_name_a_drawn_task_and_its_ledger(ledger, error, match): - source = _RetrySource(_args()) - groups = source.get_samples(TARGET + 1) - group = groups[0] if ledger is not None else [_Sample(9999), _Sample(9998)] - with pytest.raises(error, match=match): - _request(source, group, ledger=ledger) - - -def test_ledger_values_are_bounded(): - source = _RetrySource(_args()) - group = source.get_samples(TARGET + 1)[0] - with pytest.raises(SecRLEnvUntrustedEvidence): - _request(source, group, ledger=_ledger(group, value=2)) - - -def test_round_end_resets_the_budget_and_the_filter_memo(): - args = _args() - source = _RetrySource(args) - groups = source.get_samples(TARGET + 1) - _request(source, groups[0]) - source.get_samples(TARGET + 1) - args._yeto_secrlenv_filter_state = {"rollout_id": None, "decisions": {"k": (False, "x")}} - source.add_samples([]) # Miles returns the round's aborted groups - next_round = source.get_samples(TARGET + 1) - assert len(next_round) == TARGET and source.draws == [TARGET, TARGET] - assert args._yeto_secrlenv_filter_state is None - assert source.retry_stats()["replacement_attempts"] == 0 - with pytest.raises(SecRLEnvReplacementExhausted, match="identity did not match"): - _request(source, groups[1], key="old-round") - - -def test_each_round_has_a_fresh_budget(): - source = _RetrySource(_args()) - _request(source, source.get_samples(TARGET + 1)[0]) - source.get_samples(TARGET + 1) - source.add_samples([]) - _request(source, source.get_samples(TARGET + 1)[0]) - assert len(source.get_samples(TARGET + 1)) == 1 - - -def test_group_check_queues_the_retry_through_the_data_source(monkeypatch): - """The real group check, fed a signed infrastructure outcome, asks the - data source for the rerun (the callback attribute is the contract).""" - - from yeto.rl.harness.codex import reward - - monkeypatch.setattr(reward, "_verified_outcome", lambda metadata: (metadata["outcome"], 0.0)) - output_module = types.ModuleType("miles.rollout.filter_hub.base_types") - output_module.DynamicFilterOutput = lambda keep, reason: SimpleNamespace(keep=keep, reason=reason) - for name in ("miles", "miles.rollout", "miles.rollout.filter_hub"): - monkeypatch.setitem(sys.modules, name, types.ModuleType(name)) - monkeypatch.setitem(sys.modules, "miles.rollout.filter_hub.base_types", output_module) - - args = _args(yeto_rl_dynamic_sampling_max_replacements=0) - source = _RetrySource(args) - group = source.get_samples(TARGET + 1)[0] - for sample in group: - sample.status = SimpleNamespace(value="failed") - sample.metadata = {"outcome": {"status": reward.INFRASTRUCTURE_STATUS, - reward.INFRASTRUCTURE_503_RETRIES_KEY: 0}} - result = reward.check_group(args, group) - assert (result.keep, result.reason) == (False, "secrlenv_infrastructure_failure") - (rerun,) = source.get_samples(TARGET + 1) - assert [s.index for s in rerun] == [s.index for s in group] - - -# ------------------------------------------------------- terminal cleanup -def _cleanup(error, *, abort=None, require_clean=lambda: None, timeout_s=1.0): - calls = [] - - async def default_abort(): - calls.append("abort") - - asyncio.run(retry.raise_after_terminal_cleanup( - error, abort=abort or default_abort, in_flight=list, require_clean=require_clean, - timeout_s=timeout_s)) - return calls - - -def test_terminal_error_is_raised_after_the_abort(): - error = SecRLEnvReplacementExhausted("budget") - with pytest.raises(SecRLEnvReplacementExhausted) as raised: - _cleanup(error) - assert raised.value is error and raised.value.__cause__ is None - - -def test_untrusted_outcome_is_sanitized(): - with pytest.raises(SecRLEnvUntrustedEvidence, match="authentication failed") as raised: - _cleanup(UntrustedOutcome("payload with episode secrets")) - assert "secrets" not in str(raised.value) and raised.value.__cause__ is None - - -@pytest.mark.parametrize("kind", ["abort", "residue", "timeout"]) -def test_cleanup_failure_replaces_the_error(kind): - async def failing_abort(): - raise RuntimeError("engine said no") - - async def slow_abort(): - await asyncio.sleep(5) - - def residue(): - raise RuntimeError("episode left") - - options = {"abort": dict(abort=failing_abort), "residue": dict(require_clean=residue), - "timeout": dict(abort=slow_abort, timeout_s=0.01)}[kind] - with pytest.raises(SecRLEnvRolloutCleanupError) as raised: - _cleanup(SecRLEnvReplacementExhausted("x"), **options) - assert raised.value.__cause__ is None - - -# ------------------------------------------------------- Miles classes -@pytest.fixture -def stub_miles(monkeypatch): - """Minimal Miles modules the binding module imports.""" - - calls = [] - - class RolloutDataSourceWithBuffer(_Source): - pass - - class InferenceRolloutFn: - def __init__(self, input): - self.state = SimpleNamespace(args=input.args) - self.outcome = input.outcome - - async def _call_train(self, input): - if isinstance(self.outcome, BaseException): - raise self.outcome - if callable(self.outcome): # a driver standing in for generate_rollout_async - return await self.outcome(self.state.args) - return self.outcome - - async def abort(state, pendings, rollout_id): - calls.append(("abort", pendings, rollout_id)) - return [] - - modules = { - "miles": {}, "miles.rollout": {}, "miles.rollout.inference_rollout": {}, - "miles.rollout.data_source": {"RolloutDataSourceWithBuffer": RolloutDataSourceWithBuffer}, - "miles.rollout.inference_rollout.inference_rollout_common": { - "InferenceRolloutFn": InferenceRolloutFn}, - "miles.rollout.inference_rollout.inference_rollout_train": {"abort": abort}, - } - for name, attrs in modules.items(): - module = types.ModuleType(name) - for key, value in attrs.items(): - setattr(module, key, value) - monkeypatch.setitem(sys.modules, name, module) - monkeypatch.delitem(sys.modules, "yeto.rl.adapters.miles.secrlenv_miles", raising=False) - module = importlib.import_module("yeto.rl.adapters.miles.secrlenv_miles") - yield module, calls - sys.modules.pop("yeto.rl.adapters.miles.secrlenv_miles", None) - - -def test_miles_paths_name_the_binding_classes(stub_miles): - module, _ = stub_miles - for path in (retry.DATA_SOURCE_PATH, retry.ROLLOUT_FUNCTION_PATH): - module_name, _, name = path.rpartition(".") - assert module_name == module.__name__ and hasattr(module, name) - source = module.SecRLEnvRetryDataSource(_args()) - assert len(source.get_samples(TARGET + 1)) == TARGET - - -def test_rollout_function_reports_the_retry_counters(stub_miles): - module, _ = stub_miles - args = _args() - module.SecRLEnvRetryDataSource(args) - output = SimpleNamespace(metrics={"rollout/x": 1.0}) - fn = module.SecRLEnvRolloutFn(SimpleNamespace(args=args, outcome=output)) - result = asyncio.run(fn._call_train(SimpleNamespace(rollout_id=3))) - assert result.metrics["rl/secrlenv/replacement_attempts"] == 0.0 - assert result.metrics["rollout/x"] == 1.0 - - -def test_rollout_function_drains_before_a_terminal_error(stub_miles, monkeypatch): - module, calls = stub_miles - monkeypatch.setattr(module, "_require_no_episode_residue", lambda: calls.append("clean")) - fn = module.SecRLEnvRolloutFn(SimpleNamespace( - args=_args(), outcome=SecRLEnvReplacementExhausted("budget"))) - with pytest.raises(SecRLEnvReplacementExhausted): - asyncio.run(fn._call_train(SimpleNamespace(rollout_id=7))) - assert calls == [("abort", set(), 7), "clean"] - - -# ------------------------------------------------------------- launcher -def test_launcher_ports_preflight_accepts_a_secrlenv_run(): - """Before P0-1.7b.4-Q1 the ports spec refused the group check here; the - expected spec hash is the one the island derives from the bound filter.""" - - from test_rl_launcher import _args as launcher_args - - from yeto.launcher import _prepare_rl_args - - args = launcher_args([ - "--reward-function", "yeto.rl.harness.codex.reward:reward_func", - "--custom-generate-function-path", SECRLENV_GENERATE, - "--custom-agent-function-path", SECRLENV_AGENT, - "--use-session-server", "--tito-model", "org/model", - ]) - _prepare_rl_args(args) - assert args.dynamic_sampling_filter_path == SECRLENV_GROUP_FILTER - assert args.rl_expected_algorithm_sha256 == _secrlenv_spec().sha256() - # Regression: computed before the contract bound the filter, the hash was - # that of the filterless spec and the island refused the run. - assert args.rl_expected_algorithm_sha256 != AlgorithmSpec(advantage_estimator="grpo").sha256() - assert args.over_sampling_batch_size == args.rollout_batch_size + 1 - - -def _miles_like_round(source, *, stuck): - """Stands in for Miles' generate_rollout_async: group tasks run as - ``generate_and_rm_group``; an exception inside a task is swallowed (the - group only counts as failed) and a dropped group triggers a refill.""" - - from yeto.rl.harness.codex.reward import remember_terminal_error - - async def generate_and_rm_group(args, group, fail): - if fail: - error = UntrustedOutcome("forged outcome with episode payload") - remember_terminal_error(args, error) # what reward_func/generate do - raise error - await asyncio.sleep(3600) - - async def driver(args): - groups = source.get_samples(TARGET + 1) - pendings = {asyncio.create_task(generate_and_rm_group(args, g, i == 0)) - for i, g in enumerate(groups)} - stuck.extend(pendings) - done, pendings = await asyncio.wait(pendings, return_when=asyncio.FIRST_COMPLETED) - for task in done: - try: - task.result() - except Exception: # noqa: BLE001 - Miles: logged, group counted as failed - continue - source.get_samples(TARGET + 1) # refill for the dropped group: no retry queued - - return driver - - -def test_rollout_function_raises_the_first_terminal_error_and_stops_group_tasks( - stub_miles, monkeypatch): - module, calls = stub_miles - monkeypatch.setattr(module, "_require_no_episode_residue", lambda: calls.append("clean")) - args = _args() - source = module.SecRLEnvRetryDataSource(args) - stuck = [] - fn = module.SecRLEnvRolloutFn(SimpleNamespace( - args=args, outcome=_miles_like_round(source, stuck=stuck))) - - async def run(): - with pytest.raises(SecRLEnvUntrustedEvidence) as raised: - await fn._call_train(SimpleNamespace(rollout_id=4)) - # checked inside the loop: asyncio.run would cancel leftovers on exit - assert len(stuck) == TARGET and all(task.done() for task in stuck) - return raised.value - - error = asyncio.run(run()) - assert "payload" not in str(error) # sanitized first error, not the refill error - assert calls == [("abort", set(), 4), "clean"] - assert sum(task.cancelled() for task in stuck) == TARGET - 1 - - -def test_round_start_forgets_the_previous_terminal_error(): - from yeto.rl.harness.codex.reward import TERMINAL_ERROR_ATTR - - args = _args() - source = _RetrySource(args) - setattr(args, TERMINAL_ERROR_ATTR, UntrustedOutcome("old")) - source.get_samples(TARGET + 1) - assert getattr(args, TERMINAL_ERROR_ATTR) is None - fallback = SecRLEnvReplacementExhausted("x") - assert retry.first_terminal_error(args, fallback) is fallback diff --git a/tests/test_rl_ssh_harness.py b/tests/test_rl_ssh_harness.py index 1d15010ce..b5fb662c7 100644 --- a/tests/test_rl_ssh_harness.py +++ b/tests/test_rl_ssh_harness.py @@ -1,4 +1,3 @@ -import copy import hashlib import json import os @@ -10,8 +9,6 @@ import pytest import torch -import yeto.rl as rl_config -from yeto.provenance import python_spec_sha256 from yeto.rl import ( MILES_NEXT_COMMIT, MILES_NEXT_REPOSITORY, @@ -21,13 +18,6 @@ ssh_harness, ) from yeto.rl.bridge import _write_round_audit -from yeto.rl.codex_backend import ( - QWEN35_08B_MODEL, - QWEN35_08B_REVISION, - QWEN38_MODEL, - QWEN38_REVISION, - stock_codex_backend_contract, -) from yeto.rl.core import ( CanonicalTensorSpec, canonical_layout_hash, @@ -151,38 +141,16 @@ def _plan(): } -def _secrlenv_daemon_contract(run_id="acceptance"): - return { - "source_root": "/data/yeto-rl/src/secrlenv-v24", - "source_sha256": "2" * 64, - "task_pack": "/data/yeto-rl/taskpacks/" + "3" * 64, - "task_pack_sha256": "3" * 64, - "state_root": f"/data/yeto-rl/secrlenv-runs/{run_id}", - "bind": "127.0.0.1", - "port": 28765, - "operator_image": "secrlenv-operator:pinned", - "operator_image_id": "sha256:" + "4" * 64, - "max_active_episodes": 16, - "enable_dind_debug": False, - "dind_image": None, - "dind_image_id": None, - "dind_debug_script_sha256": None, - } - - -def _enable_secrlenv_agent(plan): - plan["remote_env_file"] = ".config/yeto/rl.env" - plan["learner"].update( - custom_agent_function_path="yeto.rl.harness.codex.agent.run", - custom_generate_function_path=ssh_harness.SECRLENV_GENERATE, - use_session_server=True, - tito_model="org/model", - reward_function=ssh_harness.SECRLENV_REWARD, - dynamic_sampling_filter_path=ssh_harness.SECRLENV_GROUP_FILTER, - dynamic_sampling_max_replacements=0, - secrlenv_max_infrastructure_replacements=1, - over_sampling_batch_size=plan["learner"]["groups_per_round"] + 1, - ) +@pytest.mark.parametrize("field, value, message", [ + ("secrlenv_daemon", {"port": 28765}, "secrlenv daemon plans"), + ("codex_harness", {"cli_version": "x"}, "stock Codex contract"), + ("eval_checkpoint", {"sha256": "f" * 64}, "removed secrlenv agent"), +]) +def test_plans_of_the_removed_codex_harness_are_refused(field, value, message): + plan = _plan() + plan[field] = value + with pytest.raises(ssh_harness.HarnessError, match=message): + ssh_harness._validate_plan(plan) def _local_data_plan(path: Path): @@ -198,75 +166,6 @@ def _local_data_plan(path: Path): return plan -def test_secrlenv_plan_round_trip_requires_exact_replacement_contract(): - plan = _plan() - _enable_secrlenv_agent(plan) - plan["secrlenv_daemon"] = _secrlenv_daemon_contract(plan["run_id"]) - - ssh_harness._validate_plan(plan) - parsed = parse_learner_args(_learner_argv(plan, 0)[3:]) - - assert parsed.dynamic_sampling_filter_path == ssh_harness.SECRLENV_GROUP_FILTER - assert parsed.dynamic_sampling_max_replacements == 0 - assert parsed.secrlenv_max_infrastructure_replacements == 1 - assert parsed.over_sampling_batch_size == parsed.groups_per_round + 1 - - mutations = [ - ( - "custom_generate_function_path", - "miles.rollout.generate_hub.agentic_tool_call.generate", - ), - ("reward_function", "other.reward:score"), - ("dynamic_sampling_filter_path", None), - ("dynamic_sampling_filter_path", "other.filter"), - ("dynamic_sampling_max_replacements", None), - ("dynamic_sampling_max_replacements", False), - ("dynamic_sampling_max_replacements", 1), - ("secrlenv_max_infrastructure_replacements", None), - ("secrlenv_max_infrastructure_replacements", True), - ("secrlenv_max_infrastructure_replacements", 2), - ("over_sampling_batch_size", plan["learner"]["groups_per_round"]), - ] - for name, value in mutations: - invalid = copy.deepcopy(plan) - invalid["learner"][name] = value - with pytest.raises(ssh_harness.HarnessError): - ssh_harness._validate_plan(invalid) - - -def _secrlenv_eval_plan(path: Path): - pytest.importorskip("yeto.rl.harness.codex") - plan = _local_data_plan(path) - _enable_secrlenv_agent(plan) - plan["learner"]["reward_function"] = ssh_harness.SECRLENV_REWARD - plan["reward_sha256"] = python_spec_sha256(ssh_harness.SECRLENV_REWARD) - plan["secrlenv_daemon"] = _secrlenv_daemon_contract(plan["run_id"]) - plan["final_ack_timeout_s"] = 21600 - plan["eval_checkpoint"] = ssh_harness._eval_checkpoint_contract( - "root@h200-n6", - "yeto-rl-ssh-data/train-run/state/state.ckpt", - "f" * 64, - 138_632_000_000, - plan["learner"]["global_rounds"], - ) - plan["learner"]["evaluation"] = { - "eval_only": True, - "dataset_name": "flaky100", - "data": plan["learner"]["data"], - "data_sha256": plan["learner"]["data_sha256"], - "interval": 1, - "samples_per_prompt": 2, - "prompt_count": 1, - "skip_before_train": True, - "temperature": 0.7, - "top_p": 0.95, - "max_prompt_len": 256, - "max_response_len": 256, - "max_context_len": 512, - } - return plan - - def _tms_patch_contract(): return { "repository": ssh_harness.TMS_PRELOAD_PATCH_REPOSITORY, @@ -690,61 +589,6 @@ def test_plan_rejects_previous_pipeline_identity_schema(): ssh_harness._validate_plan(plan) -def test_prepare_cli_exposes_final_secrlenv_eval_controls(): - args = ssh_harness.build_parser().parse_args( - [ - "prepare", - "--host", - "root@h200-n1,root@h200-n2", - "--run-id", - "flaky100", - "--final-ack-timeout-s", - "21600", - "--eval-checkpoint-host", - "root@h200-n6", - "--eval-checkpoint-path", - "yeto-rl-ssh-data/train/state/state.ckpt", - "--eval-checkpoint-sha256", - "f" * 64, - "--eval-checkpoint-size-bytes", - "138632000000", - "--eval-checkpoint-global-step", - "8", - "--eval-dataset-name", - "flaky100", - "--eval-interval", - "1", - "--eval-samples-per-prompt", - "2", - "--eval-temperature", - "0.7", - "--eval-top-p", - "0.95", - "--eval-max-prompt-len", - "4096", - "--eval-max-response-len", - "32768", - "--eval-max-context-len", - "32768", - ] - ) - - assert args.final_ack_timeout_s == 21600 - assert args.eval_checkpoint_host == "root@h200-n6" - assert args.eval_checkpoint_path.endswith("train/state/state.ckpt") - assert args.eval_checkpoint_sha256 == "f" * 64 - assert args.eval_checkpoint_size_bytes == 138_632_000_000 - assert args.eval_checkpoint_global_step == 8 - assert args.eval_dataset_name == "flaky100" - assert args.eval_interval == 1 - assert args.eval_samples_per_prompt == 2 - assert args.eval_temperature == 0.7 - assert args.eval_top_p == 0.95 - assert args.eval_max_prompt_len == 4096 - assert args.eval_max_response_len == 32768 - assert args.eval_max_context_len == 32768 - - def test_plan_requires_the_patched_sglang_pin(): plan = _plan() plan.pop("sglang") @@ -769,84 +613,6 @@ def test_local_prompt_plan_is_content_bound_without_a_hub_revision(tmp_path): ) -def test_secrlenv_eval_reuses_the_attested_training_dataset_and_extends_final_ack( - tmp_path, -): - prompts = tmp_path / "flaky100.jsonl" - prompts.write_text('{"messages": []}\n', encoding="utf-8") - plan = _secrlenv_eval_plan(prompts) - - ssh_harness._validate_plan(plan) - evaluation = plan["learner"]["evaluation"] - assert evaluation["data"] == plan["learner"]["data"] - assert evaluation["data_sha256"] == plan["learner"]["data_sha256"] - assert evaluation["eval_only"] is True - assert evaluation["interval"] == 1 - assert evaluation["prompt_count"] == 1 - assert plan["learner"]["reward_function"] == ssh_harness.SECRLENV_REWARD - assert plan["reward_sha256"] == python_spec_sha256(ssh_harness.SECRLENV_REWARD) - assert plan["eval_checkpoint"]["global_step"] == plan["learner"]["global_rounds"] - syncer = _syncer_argv(plan) - assert syncer[syncer.index("--final-ack-timeout-s") + 1] == "21600" - - argv = _learner_argv(plan, 0) - parsed = parse_learner_args(argv[3:]) - assert parsed.eval_only is True - assert parsed.eval_dataset_name == "flaky100" - assert parsed.eval_data_sha256 == plan["learner"]["data_sha256"] - assert parsed.eval_interval == 1 - assert parsed.eval_samples_per_prompt == 2 - assert parsed.reward_function == ssh_harness.SECRLENV_REWARD - assert parsed.eval_temperature == 0.7 - assert parsed.eval_top_p == 0.95 - assert parsed.eval_max_prompt_len == 256 - assert parsed.eval_max_response_len == 256 - assert parsed.eval_max_context_len == 512 - - -@pytest.mark.parametrize("field", ["data", "data_sha256"]) -def test_secrlenv_eval_rejects_a_distinct_training_dataset_identity(tmp_path, field): - prompts = tmp_path / "flaky100.jsonl" - prompts.write_text('{"messages": []}\n', encoding="utf-8") - plan = _secrlenv_eval_plan(prompts) - plan["learner"]["evaluation"][field] = ( - "/workspace/data/other.jsonl" if field == "data" else "0" * 64 - ) - - with pytest.raises(HarnessError, match="exact training dataset"): - ssh_harness._validate_plan(plan) - - -def test_secrlenv_eval_requires_an_explicitly_extended_final_ack_timeout(tmp_path): - prompts = tmp_path / "flaky100.jsonl" - prompts.write_text('{"messages": []}\n', encoding="utf-8") - plan = _secrlenv_eval_plan(prompts) - plan.pop("final_ack_timeout_s") - - with pytest.raises(HarnessError, match="extended final_ack_timeout"): - ssh_harness._validate_plan(plan) - - -def test_secrlenv_eval_rejects_a_different_reward_contract(tmp_path): - prompts = tmp_path / "flaky100.jsonl" - prompts.write_text('{"messages": []}\n', encoding="utf-8") - plan = _secrlenv_eval_plan(prompts) - plan["learner"]["reward_function"] = "pkg.reward:score" - - with pytest.raises(HarnessError, match="signed secrlenv reward"): - ssh_harness._validate_plan(plan) - - -def test_secrlenv_eval_checkpoint_step_must_match_terminal_training_step(tmp_path): - prompts = tmp_path / "flaky100.jsonl" - prompts.write_text('{"messages": []}\n', encoding="utf-8") - plan = _secrlenv_eval_plan(prompts) - plan["eval_checkpoint"]["global_step"] -= 1 - - with pytest.raises(HarnessError, match="eval checkpoint identity"): - ssh_harness._validate_plan(plan) - - def test_prepare_maps_a_local_prompt_file_into_the_remote_plan(tmp_path, monkeypatch): prompts = tmp_path / "prompts.jsonl" prompts.write_text('{"messages": []}\n', encoding="utf-8") @@ -898,97 +664,6 @@ def test_prepare_maps_a_local_prompt_file_into_the_remote_plan(tmp_path, monkeyp ) -def test_prepare_wires_final_secrlenv_eval_and_same_dataset_sha(tmp_path, monkeypatch): - pytest.importorskip("yeto.rl.harness.codex") - prompts = tmp_path / "flaky100.jsonl" - prompts.write_text('{"messages": []}\n', encoding="utf-8") - learner = _plan()["learner"] - learner.update( - custom_agent_function_path="yeto.rl.harness.codex.agent.run", - custom_generate_function_path=ssh_harness.SECRLENV_GENERATE, - use_session_server=True, - tito_model="org/model", - reward_function=ssh_harness.SECRLENV_REWARD, - dynamic_sampling_filter_path=ssh_harness.SECRLENV_GROUP_FILTER, - dynamic_sampling_max_replacements=0, - secrlenv_max_infrastructure_replacements=1, - over_sampling_batch_size=learner["groups_per_round"] + 1, - ) - args = SimpleNamespace( - **{ - **learner, - "data": str(prompts), - "data_revision": None, - "total_steps": learner["global_rounds"], - "rollout_batch_size": learner["groups_per_round"], - "n_samples_per_prompt": learner["samples_per_group"], - "rl_image": "docker:" + IMAGE, - "source_sha256": "c" * 64, - "reward_sha256": python_spec_sha256(ssh_harness.SECRLENV_REWARD), - "_provenance": {"dataset": {"source": "local"}}, - } - ) - daemon = _secrlenv_daemon_contract("final-eval") - checkpoint_sha256 = "f" * 64 - namespace = SimpleNamespace( - run_id="final-eval", - host=["alice@a0", "alice@b0"], - gpus_per_node=1, - remote_root=".cache/yeto-rl-ssh", - remote_env_file=".config/yeto/rl.env", - model_manifest_sha256=None, - syncer_address="100.64.0.6:29400", - syncer_host="root@syncer0", - output_dir=tmp_path / "run", - ssh_option=[], - final_ack_timeout_s=21600, - eval_checkpoint_host="root@h200-n6", - eval_checkpoint_path="yeto-rl-ssh-data/train/state/state.ckpt", - eval_checkpoint_sha256=checkpoint_sha256, - eval_checkpoint_size_bytes=138_632_000_000, - eval_checkpoint_global_step=learner["global_rounds"], - eval_dataset_name="flaky100", - eval_interval=1, - eval_samples_per_prompt=2, - eval_temperature=0.7, - eval_top_p=0.95, - eval_max_prompt_len=256, - eval_max_response_len=256, - eval_max_context_len=512, - secrlenv_source_root=daemon["source_root"], - secrlenv_source_sha256=daemon["source_sha256"], - secrlenv_task_pack=daemon["task_pack"], - secrlenv_task_pack_sha256=daemon["task_pack_sha256"], - secrlenv_operator_image=daemon["operator_image"], - secrlenv_operator_image_id=daemon["operator_image_id"], - secrlenv_port=daemon["port"], - secrlenv_max_active_episodes=daemon["max_active_episodes"], - secrlenv_enable_dind_debug=False, - secrlenv_dind_image=None, - secrlenv_dind_image_id=None, - secrlenv_dind_debug_script_sha256=None, - ) - monkeypatch.setattr(ssh_harness, "_resolved_launch_args", lambda *unused: args) - monkeypatch.setattr(ssh_harness, "_syncer_source_sha256", lambda: "1" * 64) - - plan_path = ssh_harness.prepare(namespace) - _, plan = load_plan(plan_path) - evaluation = plan["learner"]["evaluation"] - - assert evaluation["data"] == plan["learner"]["data"] - assert evaluation["data_sha256"] == plan["learner"]["data_sha256"] - assert evaluation["eval_only"] is True - assert evaluation["interval"] == 1 - assert evaluation["prompt_count"] == 1 - assert evaluation["skip_before_train"] is True - assert plan["final_ack_timeout_s"] == 21600 - assert plan["eval_checkpoint"]["source_host"] == "root@h200-n6" - assert plan["eval_checkpoint"]["source_path"].endswith("train/state/state.ckpt") - assert plan["eval_checkpoint"]["sha256"] == checkpoint_sha256 - assert plan["eval_checkpoint"]["global_step"] == learner["global_rounds"] - assert plan["secrlenv_daemon"]["placement"] == "island-heads" - - def test_local_prompt_directory_hash_is_stable_and_rejects_symlinks(tmp_path): first = tmp_path / "first" second = tmp_path / "second" @@ -1106,183 +781,6 @@ def test_deploy_copies_local_prompts_to_every_host_and_mounts_read_only( assert all(command[-1].endswith("/data/dataset.jsonl") for command in copies) -def test_eval_deploy_copies_terminal_checkpoint_only_to_fresh_syncer( - tmp_path, monkeypatch -): - prompts = tmp_path / "flaky100.jsonl" - prompts.write_text('{"messages": []}\n', encoding="utf-8") - plan = _secrlenv_eval_plan(prompts) - plan["syncer_host"] = "root@syncer0" - plan_path = tmp_path / "plan.json" - _write_plan(plan_path, plan) - commands = [] - scripts = [] - monkeypatch.setattr(ssh_harness, "_require_program", lambda name: None) - monkeypatch.setattr(ssh_harness, "_attest_local", lambda value: None) - monkeypatch.setattr( - ssh_harness, "_run", lambda command, **kwargs: commands.append(command) - ) - monkeypatch.setattr( - ssh_harness, - "_ssh", - lambda _plan, target, script, **kwargs: scripts.append((target, script)), - ) - - ssh_harness.deploy(plan_path) - - checkpoint_copies = [ - command for command in commands if command[:2] == ["scp", "-3"] - ] - assert len(checkpoint_copies) == 1 - assert checkpoint_copies[0][-2] == ( - "root@h200-n6:yeto-rl-ssh-data/train-run/state/state.ckpt" - ) - assert checkpoint_copies[0][-1].endswith( - "root@syncer0:.cache/yeto-rl-ssh/acceptance/state/state.ckpt" - ) - source_attestation = next( - script for target, script in scripts if target == "root@h200-n6" - ) - assert plan["eval_checkpoint"]["sha256"] in source_attestation - assert str(plan["eval_checkpoint"]["size_bytes"]) in source_attestation - assert "syncer checkpoint global step mismatch" in source_attestation - syncer_script = ssh_harness._syncer_start_script(plan) - assert plan["eval_checkpoint"]["sha256"] in syncer_script - assert "sha256sum --check -" in syncer_script - - -def test_eval_collect_whitelists_small_evidence_and_excludes_checkpoint( - tmp_path, monkeypatch -): - prompts = tmp_path / "flaky100.jsonl" - prompts.write_text('{"messages": []}\n', encoding="utf-8") - plan = _secrlenv_eval_plan(prompts) - plan["islands"] = [plan["islands"][0]] - plan_path = tmp_path / "plan.json" - _write_plan(plan_path, plan) - commands = [] - ssh_scripts = [] - - def ssh(_plan, _target, script, **_kwargs): - ssh_scripts.append(script) - if script.startswith("docker inspect"): - stdout = json.dumps( - { - "Status": "exited", - "ExitCode": 0, - "OOMKilled": False, - "RestartCount": 0, - } - ) - elif "final_ack_count" in script: - stdout = json.dumps({"schema": 1, "resume_count": 1, "final_ack_count": 1}) - else: - stdout = "" - return SimpleNamespace(stdout=stdout, stderr="", returncode=0) - - monkeypatch.setattr(ssh_harness, "_require_program", lambda _name: None) - monkeypatch.setattr(ssh_harness, "_ssh", ssh) - monkeypatch.setattr( - ssh_harness, "_run", lambda command, **_kwargs: commands.append(command) - ) - - ssh_harness.collect(plan_path) - - remote_sources = [command[-2] for command in commands] - assert not any("/island-0/state/" in source for source in remote_sources) - assert not any("/island-0/audit/" in source for source in remote_sources) - assert not any("/island-0/output/" in source for source in remote_sources) - learner_filters = [script for script in ssh_scripts if "eval-learner" in script] - assert len(learner_filters) == 1 - syncer = next(command for command in commands if command[-2].endswith("/state/")) - assert "--include=/syncer.exit" in syncer - assert "--include=/syncer.log" not in syncer - assert "--include=/events.jsonl" not in syncer - assert "--exclude=*" in syncer - assert not any("state.ckpt" in value for value in syncer) - assert not any("docker logs" in script for script in ssh_scripts) - inspect_scripts = [ - script for script in ssh_scripts if script.startswith("docker inspect") - ] - assert inspect_scripts - assert all("--format" in script for script in inspect_scripts) - for inspection in (tmp_path / "artifacts").glob("island-*/node-*.inspect.json"): - assert set(json.loads(inspection.read_text())) == { - "Status", - "ExitCode", - "OOMKilled", - "RestartCount", - } - lifecycle = json.loads( - (tmp_path / "artifacts/syncer/lifecycle-evidence.json").read_text() - ) - assert lifecycle == {"schema": 1, "resume_count": 1, "final_ack_count": 1} - assert not list((tmp_path / "artifacts").glob("island-*/state")) - - -def test_eval_verify_uses_scalar_final_ack_evidence_without_checkpoint( - tmp_path, capsys -): - prompts = tmp_path / "flaky100.jsonl" - prompts.write_text('{"messages": []}\n', encoding="utf-8") - plan = _secrlenv_eval_plan(prompts) - plan["islands"] = [plan["islands"][0]] - plan_path = tmp_path / "plan.json" - _write_plan(plan_path, plan) - artifacts = tmp_path / "artifacts" - island = artifacts / "island-0" - output = island / "output" - output.mkdir(parents=True) - for node_id in range(len(plan["islands"][0]["hosts"])): - (island / f"node-{node_id}.inspect.json").write_text( - json.dumps( - { - "Status": "exited", - "ExitCode": 0, - "OOMKilled": False, - "RestartCount": 0, - } - ) - ) - _write_jsonl( - output / "events.jsonl", - [ - { - "event": "rl_policy_apply", - "island_id": 0, - "policy_version": plan["learner"]["global_rounds"], - "sync/global_policy_hash": "a" * 64, - }, - { - "event": "rl_eval_result", - "island_id": 0, - "rollout_id": 0, - "policy_version": plan["learner"]["global_rounds"], - "dataset_name": "flaky100", - "sample_count": 2, - "rl/eval/result": 0.5, - "rl/eval/pass_at_1": 0.5, - }, - ], - ) - syncer = artifacts / "syncer" - syncer.mkdir() - (syncer / "syncer.exit").write_text( - "result=success\n" - "exit_code=exited\n" - "exit_status=0\n" - "timestamp=2026-08-17T00:00:00Z\n" - ) - (syncer / "lifecycle-evidence.json").write_text( - json.dumps({"schema": 1, "resume_count": 1, "final_ack_count": 1}) - ) - - ssh_harness.verify(plan_path) - - assert "verified terminal eval flaky100" in capsys.readouterr().out - assert not (syncer / "state.ckpt").exists() - - def test_local_prompt_learner_command_omits_hub_revision_and_mounts_read_only( tmp_path, ): @@ -2037,264 +1535,6 @@ def test_harness_forwards_agentic_session_contract_to_the_learner(): assert "--env MILES_EXPERIMENTAL_ROLLOUT_REFACTOR=1" in script -def test_harness_freezes_stock_codex_identity_and_binary_mount(): - plan = _plan() - plan["remote_env_file"] = ".config/yeto/rl.env" - plan["secrlenv_daemon"] = _secrlenv_daemon_contract() - plan["learner"].update( - { - "rl_model_recipe": "deepseek-v4-flash", - "custom_generate_function_path": rl_config.SECRLENV_GENERATE, - "custom_agent_function_path": rl_config.CODEX_HARNESS_AGENT, - "codex_reasoning_effort": "xhigh", - "use_session_server": True, - "tito_model": "deepseekv4", - "tito_allowed_append_roles": ["tool", "user"], - "tensor_parallel": 8, - "expert_parallel": 8, - "rollout_num_gpus_per_engine": 8, - "sglang_tp_size": 8, - "sglang_dp_size": 1, - "sglang_ep_size": 8, - "lora_targets": "attention-routed-experts", - "apply_chat_template_kwargs": { - "thinking_mode": "thinking", - "reasoning_effort": "max", - "drop_thinking": False, - }, - "reward_function": ssh_harness.SECRLENV_REWARD, - "dynamic_sampling_filter_path": ssh_harness.SECRLENV_GROUP_FILTER, - "dynamic_sampling_max_replacements": 0, - "secrlenv_max_infrastructure_replacements": 1, - "over_sampling_batch_size": plan["learner"]["groups_per_round"] + 1, - } - ) - plan["codex_harness"] = { - "agent_function_path": rl_config.CODEX_HARNESS_AGENT, - "agent_source_sha256": rl_config.CODEX_HARNESS_AGENT_SHA256, - "controller_binary_path": "/controller/codex", - "controller_package_manifest_path": "/controller/codex-package.json", - "controller_app_server_schema_path": "/controller/app-server-v2.json", - "bundle_binary_path": ssh_harness.CODEX_REMOTE_BUNDLE_PATH, - "bundle_package_manifest_path": ssh_harness.CODEX_REMOTE_MANIFEST_PATH, - "bundle_app_server_schema_path": ssh_harness.CODEX_REMOTE_SCHEMA_PATH, - "container_binary_path": rl_config.CODEX_CONTAINER_BINARY_PATH, - "container_app_server_schema_path": ( - rl_config.CODEX_CONTAINER_APP_SERVER_SCHEMA_PATH - ), - "binary_sha256": rl_config.CODEX_LINUX_BINARY_SHA256, - "binary_size_bytes": rl_config.CODEX_LINUX_BINARY_SIZE_BYTES, - "cli_version": rl_config.CODEX_CLI_VERSION, - "npm_package": rl_config.CODEX_NPM_PACKAGE, - "target": rl_config.CODEX_LINUX_TARGET, - "npm_tarball_sha256": rl_config.CODEX_NPM_TARBALL_SHA256, - "package_manifest_sha256": rl_config.CODEX_PACKAGE_MANIFEST_SHA256, - "app_server_protocol_revision": (rl_config.CODEX_APP_SERVER_PROTOCOL_REVISION), - "app_server_schema_sha256": rl_config.CODEX_APP_SERVER_SCHEMA_SHA256, - "base_instructions_sha256": rl_config.CODEX_BASE_INSTRUCTIONS_SHA256, - "terminal_exec_tool_schema_sha256": ( - rl_config.CODEX_TERMINAL_EXEC_TOOL_SCHEMA_SHA256 - ), - "submit_tool_schema_sha256": (rl_config.CODEX_SUBMIT_TOOL_SCHEMA_SHA256), - "dynamic_tools_schema_sha256": (rl_config.CODEX_DYNAMIC_TOOLS_SCHEMA_SHA256), - "reasoning_effort": "xhigh", - "backend": { - "profile": "deepseekv4", - "tito_model": "deepseekv4", - "model": "deepseekv4", - "max_tokens": plan["learner"]["rollout_max_response_len"], - "reasoning_effort": "max", - "thinking": {"type": "enabled"}, - "chat_template": "deepseekv4", - "chat_template_kwargs": { - "thinking_mode": "thinking", - "reasoning_effort": "max", - "drop_thinking": False, - }, - "tito_allowed_append_roles": ["tool", "user"], - }, - } - - ssh_harness._validate_plan(plan) - argv = _learner_argv(plan, 0) - parsed = parse_learner_args(argv[3:]) - - assert parsed.codex_harness_contract == plan["codex_harness"] - for node_id in (0, 1): - script = _node_start_script(plan, 0, node_id) - assert rl_config.CODEX_CONTAINER_BINARY_PATH in script - assert "YETO_CODEX_REASONING_EFFORT=xhigh" in script - assert "YETO_CODEX_APP_SERVER_SCHEMA_SHA256=" in script - assert "YETO_CODEX_HARNESS_CONTRACT_SHA256=" in script - syntax = subprocess.run( - ["bash", "-n"], - input=script, - text=True, - capture_output=True, - check=False, - ) - assert syntax.returncode == 0, syntax.stderr - - drifted = json.loads(json.dumps(plan)) - drifted["codex_harness"]["backend"]["chat_template_kwargs"]["drop_thinking"] = True - with pytest.raises(HarnessError, match="backend/TITO contract drifted"): - ssh_harness._validate_plan(drifted) - - for field in ( - "agent_source_sha256", - "base_instructions_sha256", - "terminal_exec_tool_schema_sha256", - "submit_tool_schema_sha256", - "dynamic_tools_schema_sha256", - ): - drifted = json.loads(json.dumps(plan)) - drifted["codex_harness"][field] = "0" * 64 - with pytest.raises(HarnessError, match="pinned stock Codex runtime"): - ssh_harness._validate_plan(drifted) - - qwen = copy.deepcopy(plan) - qwen["learner"].update( - { - "model": QWEN38_MODEL, - "model_revision": QWEN38_REVISION, - "rollout_model": None, - "rollout_model_revision": None, - "rl_model_recipe": "generic", - "tito_model": "qwen38small", - "codex_backend_profile": "qwen38", - "apply_chat_template_kwargs": { - "enable_thinking": True, - "preserve_thinking": True, - "reasoning_effort": "xhigh", - }, - "lora_targets": "attention", - "expert_full_count": 0, - "expert_parallel": 1, - "tensor_parallel": 4, - "pipeline_parallel": 2, - "rollout_num_gpus_per_engine": 1, - "sglang_tp_size": 1, - "sglang_dp_size": 1, - "sglang_ep_size": 1, - "sglang_attention_backend": None, - "sglang_page_size": None, - } - ) - qwen["codex_harness"]["backend"] = stock_codex_backend_contract( - "qwen38", - qwen["learner"]["rollout_max_response_len"], - ) - - ssh_harness._validate_plan(qwen) - qwen_args = parse_learner_args(_learner_argv(qwen, 0)[3:]) - assert qwen_args.model == QWEN38_MODEL - assert qwen_args.model_revision == QWEN38_REVISION - assert qwen_args.tito_model == "qwen38small" - assert qwen_args.apply_chat_template_kwargs == { - "enable_thinking": True, - "preserve_thinking": True, - "reasoning_effort": "xhigh", - } - - openenv = copy.deepcopy(qwen) - openenv.pop("secrlenv_daemon") - openenv["learner"].update( - { - "model": QWEN35_08B_MODEL, - "model_revision": QWEN35_08B_REVISION, - "tito_model": "qwen35", - "codex_backend_profile": "qwen35_08b", - "apply_chat_template_kwargs": {"clear_thinking": False}, - "custom_generate_function_path": ( - "miles.rollout.generate_hub.agentic_tool_call.generate" - ), - "custom_agent_function_path": rl_config.CODEX_OPENENV_AGENT, - "reward_function": "openenv_generate.reward_func", - "dynamic_sampling_filter_path": ( - "openenv_generate.check_terminal_bench_episode" - ), - "secrlenv_max_infrastructure_replacements": None, - } - ) - openenv["codex_harness"]["backend"] = stock_codex_backend_contract( - "qwen35_08b", - openenv["learner"]["rollout_max_response_len"], - ) - openenv["codex_harness"]["openenv_identity_env"] = dict( - rl_config.CODEX_OPENENV_IDENTITY_ENV - ) - - ssh_harness._validate_plan(openenv) - openenv_args = parse_learner_args(_learner_argv(openenv, 0)[3:]) - assert openenv_args.tito_model == "qwen35" - assert openenv_args.codex_backend_profile == "qwen35_08b" - openenv_script = _node_start_script(openenv, 0, 0) - assert "YETO_CODEX_OPENENV_BACKEND_PROFILE=qwen35_08b" in openenv_script - assert "YETO_CODEX_OPENENV_MODEL_ID=Qwen/Qwen3.5-0.8B" in openenv_script - - drifted_openenv = copy.deepcopy(openenv) - drifted_openenv["codex_harness"]["openenv_identity_env"][ - "YETO_CODEX_OPENENV_MODEL_REVISION" - ] = "0" * 40 - with pytest.raises(HarnessError, match="pinned Codex OpenEnv surface"): - ssh_harness._validate_plan(drifted_openenv) - - qwen["learner"]["model_revision"] = "0" * 40 - with pytest.raises(HarnessError, match="model profile drifted"): - ssh_harness._validate_plan(qwen) - - -def test_codex_controller_artifacts_are_attested_before_plan_write( - tmp_path, monkeypatch -): - binary = tmp_path / "codex" - manifest = tmp_path / "codex-package.json" - schema = tmp_path / "app-server-v2.json" - binary.write_bytes(b"stock-linux-codex") - binary.chmod(0o555) - manifest.write_bytes(b"signed-package-manifest") - schema.write_bytes(b'{"b": 1, "a": [2]}') - digest = lambda path: hashlib.sha256(path.read_bytes()).hexdigest() - monkeypatch.setattr( - ssh_harness, "CODEX_LINUX_BINARY_SIZE_BYTES", binary.stat().st_size - ) - monkeypatch.setattr(ssh_harness, "CODEX_LINUX_BINARY_SHA256", digest(binary)) - monkeypatch.setattr(ssh_harness, "CODEX_PACKAGE_MANIFEST_SHA256", digest(manifest)) - from yeto.provenance import canonical_json_sha256 - - schema_digest = canonical_json_sha256(schema) - assert schema_digest != digest(schema) # pin is canonical-form, not byte-form - monkeypatch.setattr(ssh_harness, "CODEX_APP_SERVER_SCHEMA_SHA256", schema_digest) - namespace = SimpleNamespace( - codex_harness_binary=str(binary), - codex_package_manifest=str(manifest), - codex_app_server_schema=str(schema), - ) - args = SimpleNamespace( - custom_agent_function_path=rl_config.CODEX_OPENENV_AGENT, - codex_reasoning_effort="xhigh", - codex_backend_profile="qwen35_08b", - tito_model="qwen35", - rollout_max_response_len=4096, - tito_allowed_append_roles=["tool", "user"], - apply_chat_template_kwargs={"clear_thinking": False}, - ) - - contract = ssh_harness._codex_harness_contract(namespace, args) - - assert contract["controller_binary_path"] == str(binary) - assert contract["binary_sha256"] == digest(binary) - assert contract["app_server_schema_sha256"] == schema_digest - assert contract["backend"]["reasoning_effort"] == "xhigh" - assert contract["openenv_identity_env"] == rl_config.CODEX_OPENENV_IDENTITY_ENV - - binary.chmod(0o755) - binary.write_bytes(b"drifted") - binary.chmod(0o555) - with pytest.raises(HarnessError, match="binary size"): - ssh_harness._codex_harness_contract(namespace, args) - - def test_harness_forwards_deepseek_v4_recipe_to_the_learner(): plan = _plan() plan["learner"]["rl_model_recipe"] = "deepseek-v4-flash" @@ -2400,206 +1640,6 @@ def test_decoupled_plan_propagates_fragment_and_variance_filter_settings(): assert learner[learner.index("--rl-distributed-timeout-minutes") + 1] == "7" -def test_secrlenv_daemon_contract_is_required_and_propagated_to_nodes(): - plan = _plan() - _enable_secrlenv_agent(plan) - - with pytest.raises(HarnessError, match="pinned daemon contract"): - ssh_harness._validate_plan(plan) - - plan["secrlenv_daemon"] = _secrlenv_daemon_contract() - script = ssh_harness._secrlenv_daemon_script(plan) - node = _node_start_script(plan, 0, 0) - - assert "python3 -m secrlenv_rl.server" in script - assert "\n --enable-dind-debug" not in script - assert "\n --dind-image" not in script - assert "task_pack_sha256" in script - assert "python3 -m yeto.rl.secrlenv_task_images" in script - assert "secrlenv_task_images=ready" not in script - assert ( - script.index('test "$SOURCE_SHA"') - < script.index("python3 -m yeto.rl.secrlenv_task_images") - < script.index("python3 -m secrlenv_rl.server") - ) - assert "secrlenv_daemon=ready" in script - assert 'ENV_FILE="$HOME/.config/yeto/rl.env"' in script - assert 'TOKEN_FILE="$STATE_ROOT/daemon.token"' in script - assert "secrets.token_hex(32)" in script - assert '--token-file "$TOKEN_FILE"' in script - assert "--env SECRLENV_DAEMON_URL=http://127.0.0.1:28765" in node - assert ("--env SECRLENV_TASK_PACK_SHA256=" + "3" * 64) in node - assert "--env SECRLENV_BEARER_TOKEN_FILE=/run/secrlenv/daemon.token" in node - assert ( - "/data/yeto-rl/secrlenv-runs/acceptance/daemon.token:" - "/run/secrlenv/daemon.token:ro" in node - ) - - -def test_secrlenv_dind_debug_is_explicit_pinned_and_propagated(): - plan = _plan() - _enable_secrlenv_agent(plan) - plan["secrlenv_daemon"] = { - **_secrlenv_daemon_contract(), - "enable_dind_debug": True, - "dind_image": "docker:24-dind", - "dind_image_id": "sha256:" + "5" * 64, - "dind_debug_script_sha256": "6" * 64, - } - - ssh_harness._validate_plan(plan) - script = ssh_harness._secrlenv_daemon_script(plan) - - assert "--enable-dind-debug" in script - assert "--dind-image sha256:" + "5" * 64 in script - assert "DIND_IMAGE_ID=" in script - assert "sha256:" + "5" * 64 in script - assert "DIND_DEBUG_SCRIPT=" in script - assert "6" * 64 in script - assert "existing secrlenv daemon has the wrong DinD debug identity" in script - - -@pytest.mark.parametrize( - ("updates", "message"), - [ - ({"enable_dind_debug": 1}, "explicit boolean"), - ( - {"enable_dind_debug": True}, - "requires a pinned image and image ID", - ), - ( - { - "enable_dind_debug": True, - "dind_image": "docker:24-dind", - "dind_image_id": "sha256:" + "5" * 64, - }, - "requires a pinned debug script", - ), - ( - {"dind_image": "docker:24-dind"}, - "must not carry unused debug identities", - ), - ], -) -def test_secrlenv_dind_debug_contract_fails_closed(updates, message): - plan = _plan() - _enable_secrlenv_agent(plan) - plan["secrlenv_daemon"] = {**_secrlenv_daemon_contract(), **updates} - - with pytest.raises(HarnessError, match=message): - ssh_harness._validate_plan(plan) - - -def test_secrlenv_island_head_placement_is_explicit_and_fail_closed( - tmp_path, monkeypatch -): - plan = _plan() - _enable_secrlenv_agent(plan) - plan["secrlenv_daemon"] = { - **_secrlenv_daemon_contract(), - "placement": "island-heads", - } - ssh_harness._validate_plan(plan) - - assert ssh_harness._secrlenv_daemon_hosts(plan) == [ - "alice@a0", - "alice@b0", - ] - for learner_id in range(2): - head = _node_start_script(plan, learner_id, 0) - worker = _node_start_script(plan, learner_id, 1) - assert "SECRLENV_DAEMON_URL=" in head - assert "/run/secrlenv/daemon.token:ro" in head - assert "SECRLENV_DAEMON_URL=" not in worker - assert "/run/secrlenv/daemon.token:ro" not in worker - - calls = [] - - def fake_ssh(plan, target, script, **kwargs): - calls.append((target, script)) - return SimpleNamespace(stdout="", stderr="") - - monkeypatch.setattr(ssh_harness, "_ssh", fake_ssh) - ssh_harness._start_secrlenv_daemons(plan) - assert [target for target, _script in calls] == ["alice@a0", "alice@b0"] - assert all( - "python3 -m yeto.rl.secrlenv_task_images" in script for _target, script in calls - ) - - calls.clear() - plan_path = _write_plan(tmp_path / "plan.json", plan) - ssh_harness.status(plan_path) - daemon_status_targets = [ - target for target, script in calls if "secrlenv_daemon=running" in script - ] - assert daemon_status_targets == ["alice@a0", "alice@b0"] - - calls.clear() - ssh_harness._stop_secrlenv_daemons(plan) - assert [target for target, _script in calls] == ["alice@a0", "alice@b0"] - - plan["secrlenv_daemon"]["placement"] = "unknown" - with pytest.raises(HarnessError, match="daemon contract"): - ssh_harness._validate_plan(plan) - with pytest.raises(HarnessError, match="invalid placement"): - ssh_harness._secrlenv_daemon_hosts(plan) - - -def test_legacy_secrlenv_plan_without_placement_retains_all_host_semantics( - monkeypatch, -): - plan = _plan() - _enable_secrlenv_agent(plan) - plan["secrlenv_daemon"] = _secrlenv_daemon_contract() - ssh_harness._validate_plan(plan) - - assert ssh_harness._secrlenv_daemon_hosts(plan) == ssh_harness._all_hosts(plan) - assert "SECRLENV_DAEMON_URL=" in _node_start_script(plan, 0, 1) - - calls = [] - monkeypatch.setattr( - ssh_harness, - "_ssh", - lambda plan, target, script, **kwargs: calls.append((target, script)), - ) - ssh_harness._start_secrlenv_daemons(plan) - assert [target for target, _script in calls] == ssh_harness._all_hosts(plan) - - calls.clear() - ssh_harness._stop_secrlenv_daemons(plan) - assert [target for target, _script in calls] == ssh_harness._all_hosts(plan) - - -def test_start_attests_secrlenv_daemons_before_host_gpu_setup(tmp_path, monkeypatch): - plan = _plan() - _enable_secrlenv_agent(plan) - plan["secrlenv_daemon"] = _secrlenv_daemon_contract() - path = _write_plan(tmp_path / "plan.json", plan) - events = [] - - monkeypatch.setattr(ssh_harness, "deploy", lambda _path: events.append("deploy")) - monkeypatch.setattr( - ssh_harness, - "_start_secrlenv_daemons", - lambda _plan: events.append("daemon"), - ) - monkeypatch.setattr(ssh_harness, "_check_island_cards", lambda _plan: None) - monkeypatch.setattr(ssh_harness, "_host_setup_script", lambda *_args: "setup") - monkeypatch.setattr( - ssh_harness, - "_ssh", - lambda *_args, **_kwargs: events.append("host_setup"), - ) - monkeypatch.setattr(ssh_harness, "_build_syncer", lambda _plan: None) - monkeypatch.setattr(ssh_harness, "_start_syncer", lambda _plan: None) - monkeypatch.setattr(ssh_harness, "_wait_for_syncer", lambda _plan: None) - monkeypatch.setattr(ssh_harness, "_start_island", lambda *_args: None) - - ssh_harness.start(path) - - assert events[:3] == ["deploy", "daemon", "host_setup"] - - def test_start_sets_up_a_dedicated_syncer_without_gpu_requirements( tmp_path, monkeypatch ): @@ -2609,7 +1649,6 @@ def test_start_sets_up_a_dedicated_syncer_without_gpu_requirements( calls = [] monkeypatch.setattr(ssh_harness, "deploy", lambda _path: None) - monkeypatch.setattr(ssh_harness, "_start_secrlenv_daemons", lambda _plan: None) monkeypatch.setattr(ssh_harness, "_check_island_cards", lambda _plan: None) monkeypatch.setattr( ssh_harness, "_host_setup_script", lambda *_args: "learner-setup" diff --git a/tests/test_rl_synthetic_segments.py b/tests/test_rl_synthetic_segments.py index 3f2f5e1fa..e08075451 100644 --- a/tests/test_rl_synthetic_segments.py +++ b/tests/test_rl_synthetic_segments.py @@ -27,8 +27,8 @@ def test_labels_alternate_and_keep_gsm8k_reward(): def test_launcher_accepts_synthetic_only_with_per_sample_variant(): - assert L.codex_harness_launch(_args("cross_segment_per_sample"), environ={}) is None + assert L.check_compaction_launch(_args("cross_segment_per_sample"), environ={}) is None with pytest.raises(ValueError, match="only valid"): - L.codex_harness_launch(_args("length_adaptive"), environ={}) + L.check_compaction_launch(_args("length_adaptive"), environ={}) with pytest.raises(ValueError, match="needs compacted rollouts"): - L.codex_harness_launch(_args("cross_segment_per_sample", "yeto.rl.gsm8k_reward:score"), environ={}) + L.check_compaction_launch(_args("cross_segment_per_sample", "yeto.rl.gsm8k_reward:score"), environ={}) diff --git a/tests/test_rl_tb2_baseline.py b/tests/test_rl_tb2_baseline.py deleted file mode 100644 index 1d42c7cf7..000000000 --- a/tests/test_rl_tb2_baseline.py +++ /dev/null @@ -1,127 +0,0 @@ -"""S24 T-TB2: TB2 base-model baseline runner (CPU only; fake serving, provider and agent).""" - -from __future__ import annotations - -import asyncio -import json -from pathlib import Path -from types import SimpleNamespace - -import pytest - -from yeto.rl.eval import tb2_baseline as tb -from yeto.rl.eval.island import EvalTask - -REPO = Path(__file__).resolve().parents[1] -SPLIT = {"benchmark": "tb2", "items": [{"task_id": "a", "difficulty": "easy", "eval_bucket": "tb2-easy"}, - {"task_id": "b", "difficulty": "hard", "eval_bucket": "tb2-hard"}]} -ROWS = [{"metadata": {"task_id": "a"}}, {"metadata": {"task_id": "b"}}] - - -def test_load_tasks_requires_rows_and_known_ids(): - tasks = tb.load_tasks(SPLIT, ROWS) - assert [(t.task_id, t.difficulty) for t in tasks] == [("a", "easy"), ("b", "hard")] - assert tb.load_tasks(SPLIT, ROWS, ["b"])[0].row == ROWS[1] - with pytest.raises(ValueError, match="without a jsonl row"): - tb.load_tasks(SPLIT, ROWS[:1]) - with pytest.raises(ValueError, match="not in the split"): - tb.load_tasks(SPLIT, ROWS, ["a", "zz"]) - - -def test_shipped_train_split_has_46_tasks_with_rows(): - split = json.loads((REPO / "data/eval/tb2-train.json").read_text()) - rows = [json.loads(x) for x in (REPO / "data/tb2/tb2-train.jsonl").read_text().splitlines() if x.strip()] - assert len(tb.load_tasks(split, rows)) == 46 - - -def test_run_attempts_bounds_concurrency_and_keeps_failures(): - live, peak, seen = [0], [0], [] - - async def attempt(task, trial, *, policy_version, policy_token): - assert policy_version == tb.BASE_POLICY_VERSION and policy_token == "base" - live[0] += 1 - peak[0] = max(peak[0], live[0]) - await asyncio.sleep(0.01) - live[0] -= 1 - if task.task_id == "b" and trial == 1: - raise RuntimeError("sandbox create failed") - return {"success": task.task_id == "a", "reward": float(task.task_id == "a"), "end_reason": "completed"} - - tasks = tb.load_tasks(SPLIT, ROWS) - traces = asyncio.run(tb.run_attempts(tasks, 3, attempt, concurrency=2, on_trace=seen.append)) - assert len(traces) == len(seen) == 6 and peak[0] == 2 - infra = [t for t in traces if t["end_reason"] == "infra_error"] - assert len(infra) == 1 and "sandbox create failed" in infra[0]["error"] - - -def test_summarize_excludes_infra_and_counts_mixed_tasks(): - def t(task, ok, end="completed", diff="easy"): - return {"task_id": task, "difficulty": diff, "success": ok, "end_reason": end, "turns": 3, - "timing": {"agent_s": 10.0}} - - traces = [t("a", True), {**t("a", False), "agent_detail": {"end_reason": "CodexModelFailure: x"}}, t("b", False, diff="hard"), t("b", False, "infra_error", "hard"), - t("c", True), t("c", True)] - s = tb.summarize(traces, trials=2) - assert (s["attempts"], s["judged"], s["infra_errors"], s["passed"]) == (6, 5, 1, 3) - assert s["pass_rate"] == 0.6 and s["by_difficulty"]["hard"] == {"passed": 0, "judged": 1, "pass_rate": 0.0} - assert s["mixed_task_ids"] == ["a"] and s["tasks_any_pass"] == 2 and s["tasks_all_pass"] == 1 - assert s["end_reasons"] == {"completed": 5, "infra_error": 1} and s["agent_s"]["n"] == 5 - assert s["driver_end_reasons"] == {"None": 5, "CodexModelFailure: x": 1} - - -def test_run_from_config_writes_traces_and_summary_and_closes(tmp_path): - (tmp_path / "split.json").write_text(json.dumps(SPLIT)) - (tmp_path / "rows.jsonl").write_text("".join(json.dumps(r) + "\n" for r in ROWS)) - closed = [] - loader = SimpleNamespace(start_base=lambda: {"sglang_start_s": 1.0}, close=lambda: closed.append(1)) - - class Lease: - verifier = SimpleNamespace(evaluate=None) - - async def destroy(self): - pass - - async def evaluate(_episode): - return {"passed": True} - - Lease.verifier.evaluate = evaluate - - async def acquire(task_id, tid): - return Lease() - - async def agent(lease, task, trial, *, policy_token): - return {"episode_id": "e", "end_reason": "completed", "turns": 2, "tokens": 100} - - out = tmp_path / "out" - summary = tb.run_from_config({"split": str(tmp_path / "split.json"), "rows": str(tmp_path / "rows.jsonl"), - "out_dir": str(out), "trials": 2, "concurrency": 3, - "infer": {"base_model": "B"}}, - provider=SimpleNamespace(acquire=acquire), loader=loader, agent=agent) - assert closed == [1] and summary["pass_rate"] == 1.0 and summary["attempts"] == 4 - assert len((out / tb.TRACES).read_text().splitlines()) == 4 - assert json.loads((out / tb.SUMMARY).read_text())["serving"] == {"sglang_start_s": 1.0} - - -def test_run_from_config_closes_serving_when_start_fails(tmp_path): - (tmp_path / "split.json").write_text(json.dumps(SPLIT)) - (tmp_path / "rows.jsonl").write_text("".join(json.dumps(r) + "\n" for r in ROWS)) - closed = [] - - def boom(): - raise RuntimeError("sglang exited") - - loader = SimpleNamespace(start_base=boom, close=lambda: closed.append(1)) - with pytest.raises(RuntimeError, match="sglang exited"): - tb.run_from_config({"split": str(tmp_path / "split.json"), "rows": str(tmp_path / "rows.jsonl"), - "out_dir": str(tmp_path / "o"), "infer": {"base_model": "B"}}, provider=object(), loader=loader, agent=object()) - assert closed == [1] - - -def test_function_spec_env_pins_tb2_dir(): - from yeto.cloud.modal_tb2_baseline import BaselineFunctionSpec - - spec = BaselineFunctionSpec(app_name="a", workdir="w", codex_dir="c", tb2_dir="t", image_ref="img", - envs={"SECRLENV_MAX_TURNS": "32"}) - env = spec.container_env() - assert env["YETO_HARNESS_TB2_TASKS_DIR"] == "/root/tb2-data" and env["SECRLENV_MAX_TURNS"] == "32" - assert "PYTHONPATH" not in env and spec.gpu == "H200" diff --git a/tests/test_rl_tbench_direct_preflight.py b/tests/test_rl_tbench_direct_preflight.py deleted file mode 100644 index 638cb3e50..000000000 --- a/tests/test_rl_tbench_direct_preflight.py +++ /dev/null @@ -1,137 +0,0 @@ -from __future__ import annotations - -from types import SimpleNamespace - -import pytest - -from yeto.rl import CODEX_OPENENV_AGENT, CODEX_OPENENV_IDENTITY_ENV -from yeto.rl import tbench_direct_preflight as preflight -from yeto.rl.codex_backend import QWEN35_08B_MODEL, QWEN35_08B_REVISION - - -def _argv() -> list[str]: - return [ - "--custom-generate-function-path", - "miles.rollout.generate_hub.agentic_tool_call.generate", - "--custom-agent-function-path", - CODEX_OPENENV_AGENT, - "--custom-rm-path", - "openenv_generate.reward_func", - "--dynamic-sampling-filter-path", - "openenv_generate.check_terminal_bench_episode", - "--input-key", - "messages", - "--tito-model", - "qwen35", - "--sao-online-recipe", - "coding", - "--max-seq-len", - "8192", - "--num-gpus-per-node", - "1", - "--sglang-mem-fraction-static", - "0.15", - "--sglang-max-total-tokens", - "393216", - "--sglang-max-mamba-cache-size", - "256", - "--sao-compaction", - "--sao-one-gpu-island", - "--colocate", - ] - - -@pytest.fixture(autouse=True) -def _environment(monkeypatch): - values = { - **CODEX_OPENENV_IDENTITY_ENV, - "YETO_CODEX_COMPACTION_ENABLED": "1", - "YETO_CODEX_COMPACTION_TRIGGER_TOKENS": "6144", - "YETO_CODEX_COMPACTION_SUMMARY_MAX_TOKENS": "1024", - "YETO_CODEX_MAX_COMPACTIONS": "3", - "OPENENV_MAX_ROLLOUT_TIME_SECONDS": "1800", - "SECRLENV_MAX_ROLLOUT_TIME_SECONDS": "1800", - } - for name, value in values.items(): - monkeypatch.setenv(name, value) - - -def _context() -> dict[str, str]: - return { - "benchmark": "terminal-bench-2.1", - "model": QWEN35_08B_MODEL, - "base_model_revision": QWEN35_08B_REVISION, - "rollout_model_revision": QWEN35_08B_REVISION, - } - - -def _runtime(): - return SimpleNamespace( - trajectory_evidence_kind="terminal-bench-2.1", - trajectory_evidence_schema_version=2, - ) - - -def test_exact_contract_attests_before_launch(tmp_path, monkeypatch): - calls = [] - monkeypatch.setattr(preflight, "_attest_adapter", lambda root: calls.append(root)) - preflight.preflight_tbench_codex_streaming( - _argv(), - sao_context=_context(), - streaming_runtime=_runtime(), - miles_root=tmp_path, - ) - assert calls == [tmp_path.resolve()] - - -@pytest.mark.parametrize( - ("old", "new", "message"), - ( - ("messages", "prompt", "input_key"), - ("qwen35", "qwen35_08b", "tito_model"), - ("8192", "16384", "max_seq_len"), - ("0.15", "0.16", "sglang_mem_fraction_static"), - ("393216", "393217", "sglang_max_total_tokens"), - ("256", "257", "sglang_max_mamba_cache_size"), - ), -) -def test_argument_drift_fails_closed(tmp_path, monkeypatch, old, new, message): - monkeypatch.setattr(preflight, "_attest_adapter", lambda _root: None) - argv = _argv() - argv[argv.index(old)] = new - with pytest.raises(ValueError, match=message): - preflight.preflight_tbench_codex_streaming( - argv, - sao_context=_context(), - streaming_runtime=_runtime(), - miles_root=tmp_path, - ) - - -def test_environment_and_model_drift_fail_before_adapter_attestation( - tmp_path, monkeypatch -): - monkeypatch.setattr( - preflight, - "_attest_adapter", - lambda _root: pytest.fail("adapter attestation ran after an earlier gate failed"), - ) - monkeypatch.setenv("OPENENV_MAX_ROLLOUT_TIME_SECONDS", "1799") - with pytest.raises(ValueError, match="OPENENV_MAX_ROLLOUT_TIME_SECONDS"): - preflight.preflight_tbench_codex_streaming( - _argv(), - sao_context=_context(), - streaming_runtime=_runtime(), - miles_root=tmp_path, - ) - - monkeypatch.setenv("OPENENV_MAX_ROLLOUT_TIME_SECONDS", "1800") - context = _context() - context["rollout_model_revision"] = "0" * 40 - with pytest.raises(ValueError, match="model identity drifted"): - preflight.preflight_tbench_codex_streaming( - _argv(), - sao_context=context, - streaming_runtime=_runtime(), - miles_root=tmp_path, - ) diff --git a/tests/test_rl_trajectory_evidence.py b/tests/test_rl_trajectory_evidence.py index 105c81c51..8253a7cab 100644 --- a/tests/test_rl_trajectory_evidence.py +++ b/tests/test_rl_trajectory_evidence.py @@ -7,7 +7,7 @@ import pytest from yeto.rl import trajectory_evidence -from yeto.rl.harness.codex import reward +from yeto.rl import secrlenv_outcome as reward from yeto.rl.trajectory_evidence import ( build_trajectory_batch_evidence, read_trajectory_batch_evidence, diff --git a/tests/test_rl_trajectory_timing.py b/tests/test_rl_trajectory_timing.py index 22fdf2113..073e90420 100644 --- a/tests/test_rl_trajectory_timing.py +++ b/tests/test_rl_trajectory_timing.py @@ -1,6 +1,5 @@ """agentic-rollout-utilization 1.3: phase clock and tool-wait wiring.""" -from types import SimpleNamespace from yeto.rl.engine.trajectory_timing import PhaseClock, phase_totals, timing_fields @@ -28,18 +27,6 @@ def test_timing_fields_pick_typed_values_from_metadata_and_agent_metrics(): assert timing_fields(meta) == {"trajectory_started_at": 1.0, "turn_tool_seconds": [1.0, 2.5]} -def test_codex_openenv_agent_reads_the_island_tool_wait_board(): - from yeto.rl import CODEX_OPENENV_AGENT - from yeto.rl.adapters.miles.elastic_wiring import LazyBoardActor - from yeto.rl.adapters.miles.entry import load_tool_wait_source - - args = SimpleNamespace(custom_generate_function_path="miles.x.generate", - custom_agent_function_path=CODEX_OPENENV_AGENT, yeto_rl_learner_id=1) - assert isinstance(load_tool_wait_source(args), LazyBoardActor) - args.custom_agent_function_path = "other.agent.run" - assert load_tool_wait_source(args) is None - - def test_engine_kv_fields_and_round_peaks(): from yeto.rl.adapters.miles.rollout import _engine_kv_capacity, _engine_kv_used from yeto.rl.engine.timeline import load_peaks, validate_load_sample diff --git a/tests/test_rlf_baseline.py b/tests/test_rlf_baseline.py deleted file mode 100644 index bfb0b3e23..000000000 --- a/tests/test_rlf_baseline.py +++ /dev/null @@ -1,484 +0,0 @@ -"""RLF Qwen3-30B-A3B baseline runner (W5, rlf-codex-rl 5.6), CPU only. - -Fake provider / fake drive / fake session-server HTTP: serving argv, budget -env, episode orchestration, trace content, flag hygiene (the flag never lands -in any persisted record), classification (success / failed / timeout / -infrastructure / rejected) and the run summary. No Modal, no GPU, no SGLang. -""" - -from __future__ import annotations - -import asyncio -import json -from pathlib import Path -from types import SimpleNamespace -from typing import Any - -import pytest - -from yeto.rl.eval import rlf_baseline as rb -from yeto.rl.codex_backend import stock_codex_backend_profile -from yeto.rl.harness.codex import codex_harness_agent as harness -from yeto.rl.harness.codex.environment import HttpTerminalEnvironment, serve_environment -from yeto.rl.rlf import modal as rlf_modal - -FLAG = "flag{rlf_baseline_cpu_test_9f2e}" -TASK = "fixture-ctf-ret2win" - - -def _run(coro): - return asyncio.run(coro) - - -def _cfg(**over) -> rb.BaselineConfig: - kwargs: dict[str, Any] = {"tasks_dir": "/nonexistent", "flags": {TASK: FLAG}, - "model_path": "/models/qwen3-30b", "start_timeout_s": 5.0} - kwargs.update(over) - return rb.BaselineConfig(**kwargs) - - -# --- serving shape ------------------------------------------------------------- - -def test_sglang_argv_is_lora_free_32k_shape(tmp_path): - local = tmp_path / "model" - local.mkdir() - cfg = _cfg(model_path=str(local)) - argv = rb.sglang_argv(cfg, python="python3", - serving={"reasoning_parser": "qwen3", "tool_call_parser": "qwen3", - "chat_template_path": "/tpl.jinja"}) - assert argv[:4] == ["python3", "-m", "sglang.launch_server", "--model-path"] - assert not any(a.startswith(("--enable-lora", "--max-lora", "--lora-")) for a in argv) - assert argv[argv.index("--context-length") + 1] == "32768" - assert argv[argv.index("--mem-fraction-static") + 1] == "0.9" - assert "--revision" not in argv # local snapshot dir: no hub revision - assert argv[argv.index("--reasoning-parser") + 1] == "qwen3" - missing = _cfg(model_path="Qwen/Qwen3-30B-A3B") # not a local path - argv2 = rb.sglang_argv(missing, serving={}) - assert argv2[argv2.index("--revision") + 1] == rb.QWEN3_30B_A3B_REVISION - - -def test_session_server_config_is_lora_free(): - cfg = _cfg() - conf = rb.session_server_config(cfg, serving={"chat_template_path": "/tpl.jinja", - "chat_template_kwargs": {"enable_thinking": True}}) - # lora_rank is an inert int (pydantic rejects None; no adapter is ever loaded) - assert isinstance(conf["lora_rank"], int) and conf["lora_adapter_path"] is None - assert conf["lora_train_only"] is False - assert conf["backend_url"] == cfg.sglang_url and conf["tito_model"] == "qwen3" - assert conf["hf_checkpoint"] == cfg.model_path and conf["use_session_server"] is True - assert conf["apply_chat_template_kwargs"] == {"enable_thinking": True} - - -def test_config_defaults_come_from_the_profile(): - cfg = rb.BaselineConfig.for_profile(tasks_dir="/t", flags={}) - profile = stock_codex_backend_profile("qwen3_30b_a3b") - assert (cfg.context_length, cfg.max_tokens, cfg.max_turns) == \ - (profile["max_seq_len"], profile["max_tokens_per_turn"], profile["max_turns"]) == \ - (32768, 8192, 48) - assert cfg.mem_fraction_static == 0.9 and cfg.episode_seconds == 1800.0 - assert cfg.start_timeout_s == 900.0 # 15 min serving budget - assert cfg.model_path.endswith("/hf/Qwen3-30B-A3B/ad44e777") - assert cfg.policy_id == f"{profile['model_identifier']}@{profile['model_revision']}" - env = cfg.worker_budget_env() - assert env == {"YETO_CODEX_CHAT_TEMPLATE": "qwen3_30b_a3b", - "SECRLENV_MAX_TURNS": "48", "YETO_CODEX_BACKEND_MAX_TOKENS": "8192"} - - -class _FakeProc: - def __init__(self): - self.killed = 0 - - def poll(self): - return None - - def terminate(self): - self.killed += 1 - - def wait(self, timeout=None): - return 0 - - -def test_serving_stack_ready_and_start_timeout(): - cfg = _cfg() - spawned = [] - stack = rb.ServingStack(cfg, popen=lambda argv, **kw: spawned.append(argv) or _FakeProc(), - http=lambda *a, **kw: (200, {}), sleep=lambda s: None, - clock=iter([0, 1, 2, 3]).__next__, serving=lambda t: {}) - info = stack.start() - assert info["sglang_start_s"] == 1 and "--context-length" in spawned[0] - assert "session.server" in spawned[1][2] - stack.close() - - # never ready within start_timeout_s -> InfrastructureError (the run aborts) - clock_state = {"t": 0.0} - - def clock(): - clock_state["t"] += 400.0 - return clock_state["t"] - - stack2 = rb.ServingStack(_cfg(), popen=lambda argv, **kw: _FakeProc(), - http=lambda *a, **kw: (None, None), sleep=lambda s: None, - clock=clock, serving=lambda t: {}) - with pytest.raises(rb.InfrastructureError, match="infrastructure"): - stack2.start() - - # process exits before ready -> InfrastructureError too - class _Dead(_FakeProc): - returncode = 3 - - def poll(self): - return 3 - - stack3 = rb.ServingStack(_cfg(), popen=lambda argv, **kw: _Dead(), - http=lambda *a, **kw: (None, None), sleep=lambda s: None, - serving=lambda t: {}) - with pytest.raises(rb.InfrastructureError, match="exited"): - stack3.start() - - -# --- fakes: provider / session server / drive ------------------------------------ - -class _FakeExec: - def __init__(self, output=""): - self.output = output - self.commands: list[str] = [] - - def __call__(self, command: str, *, timeout_s: float) -> rlf_modal.VmExecResult: - self.commands.append(command) - return rlf_modal.VmExecResult(0, self.output) - - -class _FakeLease: - def __init__(self, relay, runner, url, token, *, flag=FLAG, trajectory_id=""): - self.env_url = url - self.env_token = token - self.environment = relay - self.verifier = rlf_modal.RlfFlagVerifier(relay, flag=flag, task_id=TASK, - episode_id="ep-lease-1", - trajectory_id=trajectory_id or "traj-lease-1") - self.episode_id = "ep-lease-1" - self.trajectory_id = trajectory_id or "traj-lease-1" - self.deadline_seconds = 1800.0 - self.worker_env = None - self._runner = runner - self.destroyed = 0 - - async def destroy(self): - self.destroyed += 1 - await self._runner.cleanup() - - async def describe(self): - return "gone" - - -class FakeProvider: - """One freshly-served relay environment per acquire (the real wire, no VM).""" - - def __init__(self, *, exec_output="", flag=FLAG): - self.exec_output = exec_output - self.flag = flag - self.leases: list[_FakeLease] = [] - self.acquired: list[tuple[str, str]] = [] - - async def acquire(self, task_id: str, trajectory_id: str = "") -> _FakeLease: - self.acquired.append((task_id, trajectory_id)) - relay = rlf_modal.CtfRelayEnvironment(_FakeExec(self.exec_output)) - runner, url = await serve_environment(relay, "tok-" + trajectory_id[:8], host="127.0.0.1") - lease = _FakeLease(relay, runner, url, "tok-" + trajectory_id[:8], - flag=self.flag, trajectory_id=trajectory_id) - self.leases.append(lease) - return lease - - -class FakeSessions: - """The session server's /sessions routes; the drive never reaches chat completions.""" - - def __init__(self): - self.created = 0 - self.deleted: list[str] = [] - - def __call__(self, method: str, url: str, body: Any = None, *, timeout: float = 60.0): - if method == "POST" and url.endswith("/sessions"): - self.created += 1 - return 200, {"session_id": f"s-{self.created}"} - if method == "DELETE" and "/sessions/" in url: - self.deleted.append(url.rsplit("/", 1)[-1]) - return 200, {} - raise AssertionError(f"unexpected {method} {url}") - - -GOOD_METRICS = {"turns": 3, "tool_calls": 3, "submit_calls": 1, "parse_failures": 0, - "max_model_total_tokens": 12345, "max_seq_len_hit": 0, - "turn_completion_tokens": [100, 200, 300], - "turn_context_tokens": [1000, 2000, 3000], - "turn_tool_output_bytes": [50, 60, 70], - "end_reason": "submitted", "end_kind": "submit"} - - -def make_drive(*, submit=FLAG, commands=("cat /challenge/flag.txt",), metrics=None, - recorded=None): - """The untrusted worker: talks to the lease over the real HTTP wire, then reports.""" - - async def drive(job, trajectory_id, board, worker_env): - if recorded is not None: - recorded.append({"job": dict(job), "worker_env": dict(worker_env or {})}) - async with HttpTerminalEnvironment(job["env_url"], job["env_token"]) as env: - for command in commands: - await env.execute(job["episode_id"], command, timeout_seconds=5, - output_bytes=8192) - if submit is not None: - await env.submit(job["episode_id"], {"flag": submit}) - return {"status": "completed", "episode_id": job["episode_id"], - "metrics": dict(GOOD_METRICS if metrics is None else metrics)} - - return drive - - -ROW = {"metadata": {"task_id": TASK, "rlf_task_id": TASK, "grader_mode": "flag", - "prompt": "Connect to the target and obtain the flag."}, - "prompt": "Connect to the target and obtain the flag."} - - -# --- episode orchestration -------------------------------------------------------- - -def test_episode_success_trace_and_flag_hygiene(): - provider = FakeProvider(exec_output=f"... {FLAG} ...") # the flag appears in tool output - sessions = FakeSessions() - recorded: list[dict] = [] - trace = _run(rb.run_episode(_cfg(), provider, ROW, session_url="http://sess", - drive=make_drive(recorded=recorded), http=sessions,)) - assert trace["classification"] == "success" and trace["success"] is True - assert trace["reward"] == 1.0 and trace["end_reason"] == "completed" - # RLF identity quadruple + baseline policy identity - assert trace["rlf_task_id"] == TASK and trace["rlf_episode_id"] == "ep-lease-1" - assert trace["rlf_trajectory_id"] and trace["policy_id"].startswith("Qwen/Qwen3-30B-A3B@") - assert trace["grader_evidence"]["grader"] == "rlf-flag" - # per-turn token rows and per-tool trace events - assert [t["completion_tokens"] for t in trace["turns"]] == [100, 200, 300] - assert trace["turn_count"] == 3 and trace["tool_calls"] == 3 and trace["submit_calls"] == 1 - assert len(trace["tool_events"]) == 1 - event = trace["tool_events"][0] - assert event["exit_code"] == 0 and event["flag_seen"] is True - assert event["output_preview"] == "... *** ..." # scrubbed, never the raw flag - # the submission turn marker: 1 tool call happened before the submit - assert trace["submissions"] == [{"after_tool_calls": 1, "has_flag": True, "flag_correct": True}] - assert trace["max_model_total_tokens"] == 12345 and trace["max_seq_len_hit"] == 0 - assert trace["compactions"] == 0 - # the flag is nowhere in anything that would be persisted - assert FLAG not in json.dumps(trace) - # the job and the worker budget env reached the drive - job, worker_env = recorded[0]["job"], recorded[0]["worker_env"] - assert job["episode_id"] == "ep-lease-1" and job["max_seq_len"] == 32768 - assert job["request_kwargs"]["max_tokens"] == 8192 and job["driver"] == "stock" - assert job["base_url"] == "http://sess/sessions/s-1" - assert worker_env["SECRLENV_MAX_TURNS"] == "48" - assert worker_env["YETO_CODEX_BACKEND_MAX_TOKENS"] == "8192" - assert worker_env["YETO_CODEX_CHAT_TEMPLATE"] == "qwen3_30b_a3b" - # teardown: session deleted, lease destroyed exactly once - assert sessions.deleted == ["s-1"] and provider.leases[0].destroyed == 1 - - -def test_episode_wrong_flag_is_failed_with_zero_reward(): - provider = FakeProvider() - trace = _run(rb.run_episode(_cfg(), provider, ROW, session_url="http://sess", - drive=make_drive(submit="flag{nope}"), http=FakeSessions(),)) - assert trace["classification"] == "failed" and trace["success"] is False - assert trace["reward"] == 0.0 - assert trace["submissions"][0]["flag_correct"] is False - assert "flag{nope}" not in json.dumps(trace) # submitted values are never recorded - - -def test_episode_hard_deadline_times_out_and_skips_grading(): - provider = FakeProvider() - - async def slow_drive(job, trajectory_id, board, worker_env): - await asyncio.sleep(30) - - cfg = _cfg(episode_seconds=0.2) - trace = _run(rb.run_episode(cfg, provider, ROW, session_url="http://sess", - drive=slow_drive, http=FakeSessions())) - assert trace["status"] == "timeout" and trace["end_reason"] == "timed_out" - assert trace["classification"] == "failed" and trace["reward"] == 0.0 - assert provider.leases[0].verifier.evaluations == 0 # a timeout never reaches the grader - assert provider.leases[0].destroyed == 1 - - -def test_episode_lease_deadline_tightens_the_hard_limit(): - provider = FakeProvider() - seen = {} - - async def slow_drive(job, trajectory_id, board, worker_env): - await asyncio.sleep(30) - - cfg = _cfg(episode_seconds=1800.0) - orig_acquire = provider.acquire - - async def acquire(task_id, trajectory_id=""): - lease = await orig_acquire(task_id, trajectory_id) - lease.deadline_seconds = 0.2 # the lease's own deadline wins over 30 min - return lease - - provider.acquire = acquire - trace = _run(rb.run_episode(cfg, provider, ROW, session_url="http://sess", - drive=slow_drive, http=FakeSessions())) - assert trace["status"] == "timeout" - - -def test_episode_harness_error_is_infrastructure(): - provider = FakeProvider() - - async def broken_drive(job, trajectory_id, board, worker_env): - raise harness.CodexHarnessError("worker exited without a result") - - trace = _run(rb.run_episode(_cfg(), provider, ROW, session_url="http://sess", - drive=broken_drive, http=FakeSessions())) - assert trace["classification"] == "infrastructure" and trace["reward"] is None - assert "worker exited" in trace["error"] - assert provider.leases[0].destroyed == 1 - - -def test_episode_acquire_failure_is_infrastructure_and_never_graded(): - class DenyProvider: - async def acquire(self, task_id, trajectory_id=""): - raise rlf_modal.OrchestrationError(f"no trusted flag configured, FLAG={FLAG}") - - trace = _run(rb.run_episode(_cfg(), DenyProvider(), ROW, session_url="http://sess", - drive=make_drive(), http=FakeSessions())) - assert trace["classification"] == "infrastructure" - assert trace.get("rlf_episode_id") is None # no lease, no identity - assert "acquire" in trace["error"] and FLAG not in json.dumps(trace) # scrubbed - - -def test_episode_malformed_grade_is_rejected_not_failed(): - provider = FakeProvider() - - async def drive(job, trajectory_id, board, worker_env): - return {"status": "completed", "episode_id": job["episode_id"], "metrics": {}} - - orig_acquire = provider.acquire - - async def acquire(task_id, trajectory_id=""): - lease = await orig_acquire(task_id, trajectory_id) - lease.verifier = SimpleNamespace(evaluate=lambda episode_id: _bad_grade()) - return lease - - async def _bad_grade(): - return {"rlf_grade": {"status": "completed"}} # malformed: no reward/identity - - provider.acquire = acquire - trace = _run(rb.run_episode(_cfg(), provider, ROW, session_url="http://sess", - drive=drive, http=FakeSessions())) - assert trace["classification"] == "rejected" and trace["reward"] is None - assert "grade rejected" in trace["error"] - - -# --- run loop + summary ----------------------------------------------------------- - -def _catalog_package(root: Path) -> None: - pkg = root / TASK - (pkg / "public").mkdir(parents=True) - (pkg / "public" / "README.md").write_text("# ret2win\nConnect and obtain the flag.") - (pkg / "Dockerfile").write_text("FROM scratch\n") - (pkg / "task.yaml").write_text(f"task_id: {TASK}\ngrader_mode: flag\ntarget_port: 31337\n") - (pkg / "MANIFEST.json").write_text(json.dumps({"task_id": TASK, "grader_mode": "flag"})) - - -def test_run_from_config_end_to_end(tmp_path): - _catalog_package(tmp_path) - provider = FakeProvider() - config = {"tasks_dir": str(tmp_path), "flags": {TASK: FLAG}, - "evidence": {"runtime_quality_gate": {"healthy": True}}, - "out_dir": str(tmp_path / "out"), "serve": "external", - "session_url": "http://sess", "baseline": {"repeat": 2, "model_path": "/m"}} - result = rb.run_from_config(config, provider=provider, drive=make_drive(), - http=FakeSessions(), progress=None) - assert result["ok"] is True - summary = result["summary"] - assert summary["episodes"] == 2 and summary["evaluated"] == 2 - assert summary["success_count"] == 2 and summary["success_rate"] == 1.0 - assert summary["reward_distribution"] == {"mean": 1.0, "counts": {"1.0": 2}} - assert summary["end_reasons"] == {"completed": 2} - assert summary["max_seq_len_hit"] == {"count": 0, "rate": 0.0} - assert summary["per_task"][TASK]["successes"] == 2 - assert summary["policy_id"].startswith("Qwen/Qwen3-30B-A3B@") - traces = [json.loads(line) for line in - Path(result["paths"]["traces"]).read_text().splitlines()] - assert len(traces) == 2 - assert {t["rlf_trajectory_id"] for t in traces} and \ - len({t["rlf_episode_id"] for t in traces}) == 1 # fake provider reuses its lease id - assert len({t["rlf_trajectory_id"] for t in traces}) == 2 # but trajectories are unique - on_disk = Path(result["paths"]["traces"]).read_text() + \ - Path(result["paths"]["summary"]).read_text() - assert FLAG not in on_disk - - -def test_run_from_config_serving_timeout_aborts_as_infrastructure(tmp_path, monkeypatch): - _catalog_package(tmp_path) - - class NeverReady: - def __init__(self, cfg): - self.cfg = cfg - - def start(self): - raise rb.InfrastructureError("sglang not ready after 900.0 s: infrastructure") - - def close(self): - pass - - monkeypatch.setattr(rb, "ServingStack", NeverReady) - config = {"tasks_dir": str(tmp_path), "flags": {TASK: FLAG}, - "evidence": {"runtime_quality_gate": {"healthy": True}}, - "out_dir": str(tmp_path / "out"), "serve": "start", - "baseline": {"model_path": "/m"}} - result = rb.run_from_config(config, provider=FakeProvider(), drive=make_drive(), - progress=None) - assert result["ok"] is False - aborted = result["summary"]["aborted"] - assert aborted["classification"] == "infrastructure" and aborted["stage"] == "serving" - assert result["summary"]["episodes"] == 0 - assert Path(result["paths"]["summary"]).is_file() - - -def test_summarize_mixed_classifications(): - traces = [ - {"task_id": "a", "classification": "success", "success": True, "reward": 1.0, - "end_reason": "completed", "max_seq_len_hit": 0, "turn_count": 10, "tool_calls": 9, - "max_model_total_tokens": 20000, "compactions": 0}, - {"task_id": "a", "classification": "failed", "success": False, "reward": 0.0, - "end_reason": "max_seq_len", "max_seq_len_hit": 1, "turn_count": 48, "tool_calls": 47, - "max_model_total_tokens": 32768, "compactions": 2}, - {"task_id": "b", "classification": "infrastructure", "success": False, "reward": None}, - {"task_id": "b", "classification": "rejected", "success": False, "reward": None}, - ] - summary = rb.summarize(traces, policy_id="p@1") - assert summary["episodes"] == 4 and summary["evaluated"] == 2 - assert summary["infrastructure"] == 1 and summary["rejected"] == 1 - assert summary["success_rate"] == 0.5 and summary["success_count"] == 1 - assert summary["reward_distribution"]["counts"] == {"0.0": 1, "1.0": 1} - assert summary["end_reasons"] == {"completed": 1, "max_seq_len": 1} - assert summary["max_seq_len_hit"] == {"count": 1, "rate": 0.5} - assert summary["compactions_total"] == 2 - assert summary["turns"] == {"mean": 29.0, "max": 48.0} - assert summary["per_task"]["a"]["success_rate"] == 0.5 - assert summary["per_task"]["b"]["infrastructure"] == 1 - # an all-infrastructure run has a null success rate, never a fake 0 - empty = rb.summarize([traces[2]]) - assert empty["success_rate"] is None and empty["reward_distribution"]["mean"] is None - - -def test_trace_recorder_bounds_previews_and_scrubs(): - relay = rlf_modal.CtfRelayEnvironment(_FakeExec(output=(FLAG + "x" * 100))) - - async def go(): - recorder = rb.TraceRecorder(relay, secrets=(FLAG,), preview_bytes=20).attach() - out = await relay.execute("ep-1", f"echo {FLAG}", timeout_seconds=5, output_bytes=8192) - await relay.submit("ep-1", {"flag": FLAG}) - return recorder, out - - recorder, _ = _run(go()) - event = recorder.tool_events[0] - assert event["flag_seen"] is True and len(event["output_preview"]) <= 20 - assert event["command"] == "echo ***" - assert recorder.submissions[0]["flag_correct"] is True - assert FLAG not in json.dumps({"tool_events": recorder.tool_events, - "submissions": recorder.submissions}) diff --git a/tests/test_rlf_bridge.py b/tests/test_rlf_bridge.py deleted file mode 100644 index e30891255..000000000 --- a/tests/test_rlf_bridge.py +++ /dev/null @@ -1,33 +0,0 @@ -import json -from pathlib import Path -import pytest -from yeto.rl.rlf import * - -def package(tmp_path, mode='flag'): - (tmp_path/'public').mkdir(); (tmp_path/'public'/'README').write_text('x') - (tmp_path/'task.yaml').write_text(f'task_id: demo\ngrader_mode: {mode}\n') - (tmp_path/'MANIFEST.json').write_text(json.dumps({'task_id':'demo','grader_mode':mode})) - -def test_admission_requires_runtime_evidence(tmp_path): - package(tmp_path) - with pytest.raises(AdmissionError): admit_task_package(tmp_path) - a=admit_task_package(tmp_path,evidence={'runtime_quality_gate':{'healthy':True}},image_digest='sha256:x') - assert a.task_id=='demo' and a.grader_mode=='flag' - -def test_agentflow_trace_cannot_admit(tmp_path): - package(tmp_path) - with pytest.raises(AdmissionError, match='AgentFlow'): admit_task_package(tmp_path,evidence={'runtime_quality_gate':1,'agentflow_node_output':'x'}) - -def test_finish_preserves_identity_and_cleanup(): - l=EpisodeLease('demo','traj-1',secret='private') - r=finish_episode(l,TrustedGrade('completed',1,True,{'flag':'matched'})) - assert r.value==1 and r.metadata['success'] and l.closed and l.secret is None - -def test_gate_rejects_constant_reward_and_accepts_update(): - with pytest.raises(AdmissionError): rl_gate([{'reward':0,'advantage':0,'policy_update':False}]) - out=rl_gate([{'reward':0,'advantage':-.1,'metadata':{'success':False},'policy_update':True},{'reward':1,'advantage':.2,'metadata':{'success':True}}]) - assert out['success_count']==1 and out['advantage_abs_sum']>.0 - -def test_gate_requires_success_count(): - with pytest.raises(AdmissionError, match='success_count_positive'): - rl_gate([{'reward':0,'advantage':-.5,'metadata':{'success':False},'policy_update':True},{'reward':1,'advantage':.5,'metadata':{'success':False},'policy_update':True}]) diff --git a/tests/test_rlf_compaction_profile.py b/tests/test_rlf_compaction_profile.py deleted file mode 100644 index 593c2d8af..000000000 --- a/tests/test_rlf_compaction_profile.py +++ /dev/null @@ -1,7 +0,0 @@ -from yeto.rl.harness.codex.compaction_bridge import qwen3_cyber_compaction_config - -def test_qwen3_cyber_profile_reserves_context_and_segments(): - c=qwen3_cyber_compaction_config() - assert c.context_budget==24576 - assert c.t_comp==4096 - assert c.max_compactions==3 diff --git a/tests/test_rlf_env_lease.py b/tests/test_rlf_env_lease.py deleted file mode 100644 index ec54fc104..000000000 --- a/tests/test_rlf_env_lease.py +++ /dev/null @@ -1,446 +0,0 @@ -"""RLF CTF environment lease (W4, breakpoints 3+5), CPU only. - -Fake VM exec / fake verifier: orchestration order, flag hygiene (never in -stdout/logs/exceptions), fail-closed grade parsing, idempotent teardown, and -the finish_trusted dual path (RLF grade vs byte-identical TB2 fallback). -No Modal, no GPU, no nested Docker. -""" - -from __future__ import annotations - -import asyncio -import base64 -import io -import json -import tarfile -from pathlib import Path -from types import SimpleNamespace -from typing import Any - -import pytest - -from yeto.rl.rlf import modal as rlf_modal -from yeto.rl.rlf.contracts import TrustedGrade -from yeto.rl.rlf.preflight import RolloutRejected -from yeto.rl.harness.codex import codex_openenv_agent_function as adapter -from yeto.rl import tbench_outcome - -FLAG = "flag{trusted_vm_acceptance_7b3d}" -TASK = "fixture-ctf-ret2win" - - -# --- fakes ------------------------------------------------------------------- - -class FakeExec: - """Scripted VmExec: records every command, replays canned results.""" - - def __init__(self, responses: dict[str, tuple[int, str]] | None = None): - self.commands: list[str] = [] - self.responses = responses or {} - - def __call__(self, command: str, *, timeout_s: float) -> rlf_modal.VmExecResult: - self.commands.append(command) - for needle, (rc, out) in self.responses.items(): - if needle in command: - return rlf_modal.VmExecResult(rc, out) - return rlf_modal.VmExecResult(0, "") - - -def healthy_exec() -> FakeExec: - return FakeExec({ - "dockerd --host": (0, ""), - " info ": (0, "Server: Docker"), - " build --network host": (0, "Successfully built"), - " run -d ": (0, "c0ffee123\n"), - " inspect -f ": (0, "running healthy\n"), - "rm -rf": (0, ""), - "printf %s": (0, ""), - "base64 -d": (0, ""), - }) - - -def make_package(root: Path) -> Path: - pkg = root / TASK - (pkg / "public").mkdir(parents=True) - (pkg / "public" / "README.md").write_text("# ret2win\nConnect and obtain the flag.") - (pkg / "public" / "challenge.c").write_text("int main(){}") - (pkg / "Dockerfile").write_text("FROM scratch\n") - (pkg / "entrypoint.sh").write_text("#!/bin/sh\n") - (pkg / "task.yaml").write_text(f"task_id: {TASK}\ngrader_mode: flag\ntarget_port: 31337\n") - (pkg / "private").mkdir() - (pkg / "private" / "solution.py").write_text(f"print({FLAG!r})") - (pkg / "MANIFEST.json").write_text(json.dumps({"task_id": TASK})) - return pkg - - -def ctf_task(root: Path) -> rlf_modal.RlfCtfTask: - return rlf_modal.resolve_ctf_task(TASK, root, {TASK: FLAG}) - - -# --- orchestration (breakpoint 3) -------------------------------------------- - -def test_orchestration_order_and_flag_env_injection(tmp_path): - make_package(tmp_path) - fake = healthy_exec() - orch = rlf_modal.CtfOrchestration(fake, ctf_task(tmp_path), sleep=lambda s: None, - auxiliary_containers=("rlf-ctf-solver", "rlf-ctf-grader")) - cid = orch.up() - assert cid == "c0ffee123" - cmds = fake.commands - - def idx(needle): - return next(i for i, c in enumerate(cmds) if needle in c) - - assert idx("dockerd --host=unix:///run/docker/docker.sock --iptables=false --bridge=none") \ - < idx(" info ") < idx("build --network host") < idx("run -d") < idx("inspect -f") - run_cmd = cmds[idx("run -d")] - import shlex as _sh - assert f"-e FLAG={_sh.quote(FLAG)}" in run_cmd and "--network host" in run_cmd - # the flag crosses onto the VM exactly once, on the target run command - assert sum(FLAG in c for c in cmds) == 1 - # dockerd was reset before launch - assert "rm -rf /var/lib/docker /run/docker" in cmds[0] - - -def test_flag_never_in_logs_or_exceptions(tmp_path): - make_package(tmp_path) - fake = healthy_exec() - fake.responses[" build --network host"] = (1, f"build exploded, FLAG={FLAG} leaked by docker") - orch = rlf_modal.CtfOrchestration(fake, ctf_task(tmp_path), sleep=lambda s: None) - with pytest.raises(rlf_modal.OrchestrationError) as excinfo: - orch.up() - assert FLAG not in str(excinfo.value) and FLAG not in repr(excinfo.value) - assert "***" in str(excinfo.value) - - fake2 = healthy_exec() - fake2.responses[" run -d "] = (1, f"docker: {FLAG} boom") - orch2 = rlf_modal.CtfOrchestration(fake2, ctf_task(tmp_path), sleep=lambda s: None) - with pytest.raises(rlf_modal.OrchestrationError) as excinfo2: - orch2.up() - assert FLAG not in str(excinfo2.value) and FLAG not in repr(excinfo2.value) - - -def test_stage_package_excludes_private_and_admission_metadata(tmp_path): - pkg = make_package(tmp_path) - blob = base64.b64decode(rlf_modal.package_tar_b64(pkg)) - with tarfile.open(fileobj=io.BytesIO(blob), mode="r:gz") as tar: - names = tar.getnames() - assert "Dockerfile" in names and "public/challenge.c" in names - assert not any("private" in n or "MANIFEST" in n or "task.yaml" in n for n in names) - assert FLAG.encode() not in blob - - -def test_health_wait_fails_fast_on_exited_container(tmp_path): - make_package(tmp_path) - fake = healthy_exec() - fake.responses[" inspect -f "] = (0, "exited\n") - orch = rlf_modal.CtfOrchestration(fake, ctf_task(tmp_path), sleep=lambda s: None) - with pytest.raises(rlf_modal.OrchestrationError, match="exited"): - orch.up() - - -def test_teardown_is_idempotent_and_ordered(tmp_path): - make_package(tmp_path) - fake = healthy_exec() - orch = rlf_modal.CtfOrchestration(fake, ctf_task(tmp_path), sleep=lambda s: None, - auxiliary_containers=("rlf-ctf-solver", "rlf-ctf-grader")) - orch.up() - orch.down() - orch.down() # second call is a no-op - rm = [c for c in fake.commands if "rm -f" in c and "docker -H" in c] - assert len(rm) == 3 # solver, grader, target — exactly once each - assert "rm -f rlf-ctf-solver " in rm[0] and "rm -f rlf-ctf-grader " in rm[1] \ - and "rm -f rlf-ctf-target " in rm[2] - - -def test_failed_up_cleans_containers(tmp_path): - make_package(tmp_path) - fake = healthy_exec() - fake.responses[" inspect -f "] = (0, "running starting\n") - orch = rlf_modal.CtfOrchestration(fake, ctf_task(tmp_path), sleep=lambda s: None, - health_attempts=2) - with pytest.raises(rlf_modal.OrchestrationError, match="healthy"): - orch.up() - assert any("rm -f" in c and "rlf-ctf-target" in c for c in fake.commands) - - -def test_resolve_ctf_task_fails_closed(tmp_path): - make_package(tmp_path) - with pytest.raises(rlf_modal.OrchestrationError, match="no trusted flag"): - rlf_modal.resolve_ctf_task(TASK, tmp_path, {}) - with pytest.raises(rlf_modal.OrchestrationError, match="no task.yaml"): - rlf_modal.resolve_ctf_task("nope", tmp_path, {"nope": FLAG}) - pkg = make_package(tmp_path / "hacked") - (pkg / "task.yaml").write_text(f"task_id: {TASK}\ngrader_mode: hacked\n") - with pytest.raises(rlf_modal.OrchestrationError, match="flag only"): - rlf_modal.resolve_ctf_task(TASK, tmp_path / "hacked", {TASK: FLAG}) - - -# --- relay + verifier ---------------------------------------------------------- - -def _run(coro): - return asyncio.run(coro) - - -def test_flag_verifier_grades_submission_and_never_leaks(tmp_path): - relay = rlf_modal.CtfRelayEnvironment(FakeExec()) - verifier = rlf_modal.RlfFlagVerifier(relay, flag=FLAG, task_id=TASK, - episode_id="ep-1", trajectory_id="traj-1") - - async def go(): - # no submission -> fail closed, error_class set - empty = await verifier.evaluate("ep-1") - assert empty["passed"] is False - assert empty["rlf_grade"]["error_class"] == "missing_submission" - # wrong flag -> 0 reward - await relay.submit("ep-1", {"flag": "flag{wrong}"}) - wrong = await verifier.evaluate("ep-1") - assert wrong["passed"] is False and wrong["rlf_grade"]["reward"] == 0.0 - # right flag -> 1 reward - relay2 = rlf_modal.CtfRelayEnvironment(FakeExec()) - verifier2 = rlf_modal.RlfFlagVerifier(relay2, flag=FLAG, task_id=TASK, - episode_id="ep-2", trajectory_id="traj-1") - await relay2.submit("ep-2", {"flag": FLAG}) - right = await verifier2.evaluate("ep-2") - assert right["passed"] is True and right["rlf_grade"]["reward"] == 1.0 - ev = right["rlf_grade"]["grader_evidence"] - assert (ev["rlf_task_id"], ev["rlf_episode_id"], ev["rlf_trajectory_id"]) == (TASK, "ep-2", "traj-1") - # the flag never appears in any payload/log/evidence - blob = json.dumps(right) - assert FLAG not in blob - # verifier is bound to its episode - with pytest.raises(ValueError, match="bound"): - await verifier2.evaluate("ep-other") - - _run(go()) - - -def test_relay_binds_one_episode_and_executes_on_vm(): - fake = FakeExec({"uname": (0, "Linux vm\n")}) - relay = rlf_modal.CtfRelayEnvironment(fake) - - async def go(): - out = await relay.execute("ep-1", "uname", timeout_seconds=5, output_bytes=100) - assert out["exit_code"] == 0 and "Linux vm" in out["output"] - assert any("uname" in c and "timeout -k 2 5" in c for c in fake.commands) - with pytest.raises(Exception, match="unknown episode"): - await relay.execute("ep-2", "uname", timeout_seconds=5, output_bytes=100) - await relay.submit("ep-1", {"flag": "x"}) - with pytest.raises(Exception, match="terminal"): - await relay.execute("ep-1", "uname", timeout_seconds=5, output_bytes=100) - - _run(go()) - - -# --- full provider lease contract (fake VM) ------------------------------------ - -class _FakeProc: - def __init__(self, result): - self._result = result - self.stdout = io.BytesIO(result.output.encode()) - self.stderr = io.BytesIO(b"") - - def wait(self): - return self._result.exit_code - - -class _FakeSandbox: - object_id = "sb-fake" - - def __init__(self, exec_fn): - self._exec = exec_fn - self.terminated = 0 - - def exec(self, *args, **kwargs): - return _FakeProc(self._exec(args[2], timeout_s=30)) - - def poll(self): - return 1 if self.terminated else None - - def terminate(self): - self.terminated += 1 - - -def _fake_vm_provider(monkeypatch, tmp_path): - make_package(tmp_path) - fake = healthy_exec() - sandbox = _FakeSandbox(fake) - monkeypatch.setattr(rlf_modal, "acquire_vm", - lambda task_id, episode_id, *, config=None, image=None, trajectory_id=None: - rlf_modal.RlfVmLease(sandbox, task_id=task_id, episode_id=episode_id, - trajectory_id=trajectory_id)) - provider = rlf_modal.RlfModalProvider( - ctf=rlf_modal.RlfCtfConfig(tasks_dir=str(tmp_path), flags={TASK: FLAG})) - return provider, fake, sandbox - - -def test_provider_full_environment_lease_contract(monkeypatch, tmp_path): - provider, fake, sandbox = _fake_vm_provider(monkeypatch, tmp_path) - - async def go(): - lease = await provider.acquire(TASK, "traj-9") - # full EnvironmentLease surface - assert lease.env_url.startswith("http://127.0.0.1:") and lease.env_token - assert lease.deadline_seconds > 0 and callable(lease.destroy) and callable(lease.describe) - assert lease.task_id == TASK and lease.episode_id and lease.trajectory_id == "traj-9" - # target orchestrated before the lease is handed out - assert any("run -d" in c and FLAG in c for c in fake.commands) - # the worker wire reaches the VM (host namespace) and grading is trusted - from yeto.rl.harness.codex.environment import HttpTerminalEnvironment - async with HttpTerminalEnvironment(lease.env_url, lease.env_token) as env: - out = await env.execute("ep-wire-1", "id", timeout_seconds=5, output_bytes=100) - assert out["exit_code"] == 0 - await env.submit("ep-wire-1", {"flag": FLAG}) - evaluation = await lease.verifier.evaluate("ep-wire-1") - assert evaluation["passed"] is True - # teardown: containers -> runner -> VM, idempotent - await lease.destroy() - await lease.destroy() - assert await lease.describe() == "gone" - - _run(go()) - assert sandbox.terminated == 1 - assert [c for c in fake.commands if "rm -f" in c and "docker -H" in c] - - -def test_provider_acquire_failure_cleans_vm(monkeypatch, tmp_path): - provider, fake, sandbox = _fake_vm_provider(monkeypatch, tmp_path) - fake.responses[" build --network host"] = (1, "nope") - with pytest.raises(rlf_modal.OrchestrationError): - _run(provider.acquire(TASK, "traj-10")) - assert sandbox.terminated == 1 # failed acquire never leaks a VM - assert any("rm -f" in c and "rlf-ctf-target" in c for c in fake.commands) - - -def test_provider_without_ctf_keeps_vm_only_lease(monkeypatch): - sandbox = _FakeSandbox(FakeExec()) - monkeypatch.setattr(rlf_modal, "acquire_vm", - lambda task_id, episode_id, *, config=None, image=None, trajectory_id=None: - rlf_modal.RlfVmLease(sandbox, task_id=task_id, episode_id=episode_id, - trajectory_id=trajectory_id)) - lease = _run(rlf_modal.RlfModalProvider().acquire(TASK, "traj-11")) - assert isinstance(lease, rlf_modal.RlfVmLease) and not hasattr(lease, "env_url") - lease.teardown() - lease.teardown() - assert sandbox.terminated == 1 - - -def test_cloud_factory_env_switch(monkeypatch, tmp_path): - from yeto.cloud import modal_reward_env as mre - - make_package(tmp_path) - monkeypatch.delenv(mre.RLF_CTF_TASKS_DIR_ENV, raising=False) - monkeypatch.delenv(mre.RLF_CTF_FLAGS_ENV, raising=False) - assert mre.rlf_modal_provider(SimpleNamespace()).ctf is None - monkeypatch.setenv(mre.RLF_CTF_TASKS_DIR_ENV, str(tmp_path)) - with pytest.raises(RuntimeError, match="FLAGS"): - mre.rlf_modal_provider(SimpleNamespace()) # fail closed without flags - monkeypatch.setenv(mre.RLF_CTF_FLAGS_ENV, json.dumps({TASK: FLAG})) - provider = mre.rlf_modal_provider(SimpleNamespace()) - assert provider.ctf is not None and provider.ctf.flags[TASK] == FLAG - - -# --- finish_trusted dual path (breakpoint 5) ----------------------------------- - -class _GradeVerifier: - def __init__(self, payload): - self.payload = payload - - async def evaluate(self, episode_id): - return self.payload - - -def _rlf_payload(passed=True): - return {"passed": passed, "log": "flag submission matched", - "rlf_grade": {"status": "completed", "reward": 1.0 if passed else 0.0, - "passed": passed, - "grader_evidence": {"grader": "rlf-flag", "rlf_task_id": TASK, - "rlf_episode_id": "ep-rlf-1", - "rlf_trajectory_id": "traj-rlf-1"}}} - - -def test_finish_trusted_rlf_grade_writes_back_episode(): - untrusted = {"status": "completed", "metrics": {"turns": 3}, "episode_id": "ep-wire"} - md = _run(adapter.finish_trusted(untrusted, _GradeVerifier(_rlf_payload()), - task_id=TASK, sample_id="s-1")) - # RLF write-back: reward/success/grader evidence + identity quadruple - assert md["reward"] == 1.0 and md["success"] is True - assert md["grader_evidence"]["grader"] == "rlf-flag" - assert (md["rlf_task_id"], md["rlf_episode_id"], md["rlf_trajectory_id"]) == \ - (TASK, "ep-rlf-1", "traj-rlf-1") - assert md["rlf_status"] == "completed" and md["exit_status"] == "completed" - # the outcome envelope still validates (tape consumers keep working) - outcome = md[tbench_outcome.OUTCOME_KEY] - assert outcome["reward"] == 1.0 and outcome["episode_id"] == "ep-wire" - assert tbench_outcome.verified_outcome(md) == (outcome, 1.0) - assert "tbench_trusted_outcome_hmac" not in md - assert md["agent_metrics"]["turns"] == 3 and "evaluate_time" in md["agent_metrics"] - - -def test_finish_trusted_malformed_grade_fails_closed(): - untrusted = {"status": "completed", "metrics": {}, "episode_id": "ep-wire"} - bad = {"rlf_grade": {"status": "completed"}} # present but malformed - with pytest.raises(RolloutRejected, match="malformed"): - _run(adapter.finish_trusted(untrusted, _GradeVerifier(bad), - task_id=TASK, sample_id="s-1")) - missing_identity = _rlf_payload() - del missing_identity["rlf_grade"]["grader_evidence"]["rlf_episode_id"] - with pytest.raises(RolloutRejected, match="identity"): - _run(adapter.finish_trusted(untrusted, _GradeVerifier(missing_identity), - task_id=TASK, sample_id="s-1")) - wrong_task = _rlf_payload() - wrong_task["rlf_grade"]["grader_evidence"]["rlf_task_id"] = "other-task" - with pytest.raises(RolloutRejected, match="does not match"): - _run(adapter.finish_trusted(untrusted, _GradeVerifier(wrong_task), - task_id=TASK, sample_id="s-1")) - - -def test_finish_trusted_tb2_fallback_byte_unchanged(): - untrusted = {"status": "completed", "metrics": {"turns": 2}, "episode_id": "ep-wire"} - tb2_payload = {"passed": True, "testsh_rc": 0, "log": "ok"} - md = _run(adapter.finish_trusted(untrusted, _GradeVerifier(tb2_payload), - task_id="tb2-task", sample_id="s-1")) - # no RLF keys at all on the fallback path - assert not any(k.startswith("rlf_") or k == "grader_evidence" for k in md) - assert "success" not in md - outcome = md[tbench_outcome.OUTCOME_KEY] - assert outcome["reward"] == 1.0 and outcome["verifier"] == tbench_outcome.TEST_SH_VERIFIER - assert tbench_outcome.verified_outcome(md) == (outcome, 1.0) - assert "tbench_trusted_outcome_hmac" not in md - # timeout path never calls the verifier and never sees RLF keys - md_timeout = _run(adapter.finish_trusted({"status": "timeout", "metrics": {}, "episode_id": "ep"}, - _GradeVerifier(tb2_payload), - task_id="tb2-task", sample_id="s-1")) - assert md_timeout[tbench_outcome.OUTCOME_KEY]["verifier"] == tbench_outcome.TIMEOUT_VERIFIER - assert not any(k.startswith("rlf_") for k in md_timeout) - - -def test_acquire_vm_idle_timeout_follows_ttl(monkeypatch): - """Sandbox idle kill must not reclaim a VM mid-generation (S21 gate-5).""" - import sys - created: dict[str, Any] = {} - - class _Sb: - object_id = "sb-x" - - def poll(self): - return None - - def terminate(self): - pass - - fake_modal = SimpleNamespace( - App=SimpleNamespace(lookup=lambda *a, **k: object()), - Image=SimpleNamespace(from_registry=lambda *a, **k: object()), - Sandbox=SimpleNamespace(create=lambda *a, **k: (created.update(k) or _Sb())), - ) - monkeypatch.setitem(sys.modules, "modal", fake_modal) - - rlf_modal.acquire_vm("t", "ep") - assert created["timeout"] == 1800 - assert created["idle_timeout"] == 1800 # not the old min(ttl, 300) - - created.clear() - cfg = rlf_modal.RlfModalVmConfig(ttl_s=600, idle_timeout_s=900) - rlf_modal.acquire_vm("t", "ep", config=cfg) - assert created["timeout"] == 600 and created["idle_timeout"] == 900 diff --git a/tests/test_rlf_gate_evidence.py b/tests/test_rlf_gate_evidence.py deleted file mode 100644 index 7270e9d91..000000000 --- a/tests/test_rlf_gate_evidence.py +++ /dev/null @@ -1,151 +0,0 @@ -"""RL gate criteria, structured gate-evidence collection, and the RLF Miles -reward wrapping contract (task 5.8 readiness fixes).""" -import asyncio -import importlib -from types import SimpleNamespace - -import pytest - -from yeto.rl.rlf import AdmissionError, rl_gate -from yeto.rl.rlf.contracts import group_advantages, policy_update_evidence -from yeto.rl.adapters.miles import rewards as miles_rewards -from yeto.rl.adapters.verl import reward_fn as verl_reward_fn -from yeto.rl.rewards.types import Trajectory - - -def _row(reward, advantage, success, policy_update=True): - return {'reward': reward, 'advantage': advantage, - 'metadata': {'success': success}, 'policy_update': policy_update} - - -# -- rl_gate criteria (spec: 非空 reward、std>0、success_count>0、adv_abs_sum>0、update) -- - -def test_gate_passes_with_success_and_variance(): - out = rl_gate([_row(0.0, -0.5, False), _row(1.0, 0.5, True)]) - assert out['passed'] and out['success_count'] == 1 and out['advantage_abs_sum'] == 1.0 - - -def test_gate_rejects_empty_rows(): - with pytest.raises(AdmissionError, match='non_empty'): - rl_gate([]) - - -def test_gate_rejects_all_zero_rewards(): - with pytest.raises(AdmissionError): - rl_gate([_row(0.0, 0.0, False), _row(0.0, 0.0, False)]) - - -def test_gate_rejects_constant_nonzero_rewards(): - with pytest.raises(AdmissionError, match='reward_std_positive'): - rl_gate([_row(1.0, -0.5, True), _row(1.0, 0.5, True)]) - - -def test_gate_rejects_zero_success_count(): - # std>0, advantage>0, policy update=true, but nobody succeeded. - with pytest.raises(AdmissionError, match='success_count_positive'): - rl_gate([_row(0.0, -0.5, False), _row(1.0, 0.5, False)]) - - -def test_gate_rejects_zero_advantage_abs_sum(): - with pytest.raises(AdmissionError, match='advantage_abs_sum_positive'): - rl_gate([_row(0.0, 0.0, False), _row(1.0, 0.0, True)]) - - -def test_gate_rejects_missing_policy_update(): - with pytest.raises(AdmissionError, match='policy_update'): - rl_gate([_row(0.0, -0.5, False, policy_update=False), - _row(1.0, 0.5, True, policy_update=False)]) - - -# -- structured evidence collection -- - -def test_group_advantages_are_reward_minus_group_mean(): - assert group_advantages([0.0, 1.0]) == [-0.5, 0.5] - assert group_advantages([]) == [] - assert sum(group_advantages([0.2, 0.7, 1.0, 0.0])) == pytest.approx(0.0) - - -def test_policy_update_evidence_detects_real_change(): - ev = policy_update_evidence({'bias': 0.0}, {'bias': 0.01}, step=3) - assert ev.updated and ev.step == 3 and ev.changed == ('bias',) - assert ev.before_digest != ev.after_digest - - -def test_policy_update_evidence_rejects_unchanged_params(): - ev = policy_update_evidence({'bias': 0.5}, {'bias': 0.5}, step=0) - assert not ev.updated and ev.before_digest == ev.after_digest and ev.changed == () - - -# -- Miles wrapping of the RLF success reward -- - -def _rlf_metadata(success=True, reward=1.0): - return {'rlf_episode_id': 'ep-0', 'rlf_task_id': 'fixture-ctf-ret2win', - 'rlf_trajectory_id': 'traj-0', 'grader_evidence': {'trusted': True}, - 'success': success, 'reward': reward} - - -def _fake_sample(metadata): - return SimpleNamespace(metadata=dict(metadata), group_index=0, index=0, - prompt='p', response='r', label=None, tokens=[1], - rollout_log_probs=[-0.1], loss_mask=[1], - status=None, reward=None) - - -@pytest.mark.parametrize('success,reward', [(True, 1.0), (False, 0.0)]) -def test_miles_wrapper_matches_neutral_rlf_reward(success, reward): - md = _rlf_metadata(success, reward) - neutral = miles_rewards.rlf_success_reward(Trajectory(metadata=dict(md))) - wrapped = getattr(miles_rewards, 'rlf_success') # module __getattr__ wraps it - sample = _fake_sample(md) - value = asyncio.run(wrapped(None, sample)) - assert value == neutral.value == reward - # Same input, same output: the adapter writes exactly the neutral metadata. - assert sample.metadata['success'] == neutral.metadata['success'] == success - - -def test_miles_wrapper_handles_sample_list(): - wrapped = getattr(miles_rewards, 'rlf_success') - samples = [_fake_sample(_rlf_metadata(True, 1.0)), _fake_sample(_rlf_metadata(False, 0.0))] - assert asyncio.run(wrapped(None, samples)) == [1.0, 0.0] - assert [s.metadata['success'] for s in samples] == [True, False] - - -def test_custom_rm_path_resolves_to_wrapped_callable(): - path = miles_rewards.miles_custom_rm_path('rlf_success') - assert path == 'yeto.rl.adapters.miles.rewards.rlf_success' - # What Miles' load_function does: split at the last dot, getattr the module. - module, _, attr = path.rpartition('.') - fn = getattr(importlib.import_module(module), attr) - assert getattr(fn, '__wrapped_neutral__') is miles_rewards.rlf_success_reward - sample = _fake_sample(_rlf_metadata(True, 1.0)) - assert asyncio.run(fn(None, sample)) == 1.0 and sample.metadata['success'] is True - - -def test_raw_neutral_name_is_not_a_valid_miles_entry(): - # The unwrapped neutral attribute takes one Trajectory; Miles calls fn(args, samples). - with pytest.raises(TypeError): - miles_rewards.rlf_success_reward(None, _fake_sample(_rlf_metadata())) - - -# -- verl path behavior unchanged: bare float rejected, RLF row validation kept -- - -def test_verl_rlf_row_requires_trusted_metadata(): - with pytest.raises(ValueError, match='missing trusted metadata'): - verl_reward_fn.compute_score_rlf('rlf', 'answer', extra_info={'metadata': {}}) - - -def test_verl_rlf_row_accepts_full_metadata(): - score = verl_reward_fn.compute_score_rlf('rlf', 'answer', - extra_info={'metadata': _rlf_metadata(True, 1.0)}) - assert score == 1.0 - - -def test_verl_rlf_row_rejects_bool_reward(): - md = _rlf_metadata(True, True) - with pytest.raises(ValueError, match='must be numeric'): - verl_reward_fn.compute_score_rlf('rlf', 'answer', extra_info={'metadata': md}) - - -def test_verl_unmapped_reward_still_refused(): - with pytest.raises(verl_reward_fn.UnmappedReward): - verl_reward_fn.function_for('not-a-reward') diff --git a/tests/test_rlf_rl_gate.py b/tests/test_rlf_rl_gate.py deleted file mode 100644 index 7c064d216..000000000 --- a/tests/test_rlf_rl_gate.py +++ /dev/null @@ -1,350 +0,0 @@ -"""RLF real-GPU RL gate runner orchestration (task 5.8): fake learner / fake -provider only — no GPU, no Miles, no torch, no Modal. - -These tests cover the orchestration (collection -> gate rows -> rl_gate -> -evidence). The injected collection can never announce a passing verdict; -only the real ``miles_island`` collector on the GPU run may do that. -""" - -from __future__ import annotations - -import json -from pathlib import Path - -import pytest - -from yeto.rl.eval.rlf_rl_gate import ( - EVIDENCE_SCHEMA, - GateConfig, - GateEvidenceError, - build_gate_rows, - collect_policy_update, - collect_policy_update_from_tape, - collect_rollout_samples, - prepare_prompt_data, - run_gate, - write_evidence, -) - -FLAGS = {"fixture-ctf-ezshell": "flag{ezshell-secret}", "fixture-ctf-ret2win": "flag{ret2win-secret}"} - - -def _sample(group, index, reward, success, task_id="fixture-ctf-ezshell"): - return {"reward": reward, - "metadata": {"success": success, "rlf_task_id": task_id, - "rlf_episode_id": f"ep-{group}-{index}", - "rlf_trajectory_id": f"traj-{group}-{index}", - "grader_evidence": {"trusted": True}}} - - -def _payload(groups): - return {"schema_version": 1, "completed_groups": groups} - - -def _passing_groups(): - # ezshell group: one success (variance + success_count); ret2win group: all fail. - return [[_sample(0, 0, 1.0, True), _sample(0, 1, 0.0, False), - _sample(0, 2, 0.0, False), _sample(0, 3, 0.0, False)], - [_sample(1, 0, 0.0, False, "fixture-ctf-ret2win"), - _sample(1, 1, 0.0, False, "fixture-ctf-ret2win")]] - - -def _update_loader(before, after): - tables = {"before": before, "after": after} - return lambda path: tables[str(path)] - - -def _update(updated=True): - before = {"lora_a": 0.5, "lora_b": -0.25} - after = {"lora_a": 0.5 + (0.01 if updated else 0.0), "lora_b": -0.25} - return collect_policy_update("before", "after", step=2, - loader=_update_loader(before, after)) - - -def _config(tmp_path): - return GateConfig(tasks_dir=str(tmp_path), flags=FLAGS) - - -# -- collection from the completed-groups checkpoint ------------------------- - -def test_collect_rollout_samples_reads_real_checkpoint_shape(): - samples = collect_rollout_samples("ckpt", loader=lambda _p: _payload(_passing_groups())) - assert len(samples) == 6 - assert samples[0] == {"group_index": 0, "index": 0, "task_id": "fixture-ctf-ezshell", - "reward": 1.0, "success": True} - assert sum(s["success"] for s in samples) == 1 - - -def test_collect_flattens_compaction_segments_sharing_one_outcome(): - group = [[_sample(0, 0, 1.0, True), _sample(0, 0, 1.0, True)], _sample(0, 1, 0.0, False)] - samples = collect_rollout_samples("ckpt", loader=lambda _p: _payload([group])) - assert [s["reward"] for s in samples] == [1.0, 0.0] - - -def test_collect_fails_closed_on_missing_success_or_reward(): - bad = _sample(0, 0, 1.0, True) - del bad["metadata"]["success"] - with pytest.raises(GateEvidenceError, match="metadata.success"): - collect_rollout_samples("ckpt", loader=lambda _p: _payload([[bad]])) - bad = _sample(0, 0, 1.0, True) - bad["reward"] = None - with pytest.raises(GateEvidenceError, match="numeric reward"): - collect_rollout_samples("ckpt", loader=lambda _p: _payload([[bad]])) - with pytest.raises(GateEvidenceError, match="no completed groups"): - collect_rollout_samples("ckpt", loader=lambda _p: _payload([])) - - -# -- advantages and policy update are computed, never hand-filled ------------ - -def test_gate_rows_compute_grpo_group_advantages(): - samples = collect_rollout_samples("ckpt", loader=lambda _p: _payload(_passing_groups())) - rows = build_gate_rows(samples, _update()) - first = [r["advantage"] for r in rows if r["metadata"]["group_index"] == 0] - assert first == [0.75, -0.25, -0.25, -0.25] # reward minus group mean - assert all(r["policy_update"] for r in rows) - - -def test_policy_update_unchanged_params_marks_rows_not_updated(): - samples = collect_rollout_samples("ckpt", loader=lambda _p: _payload(_passing_groups())) - rows = build_gate_rows(samples, _update(updated=False)) - assert not any(r["policy_update"] for r in rows) - - -# -- the five gate criteria --------------------------------------------------- - -def test_all_five_criteria_pass_but_injected_collection_never_announces(tmp_path): - samples = collect_rollout_samples("ckpt", loader=lambda _p: _payload(_passing_groups())) - update = _update() - rows = build_gate_rows(samples, update) - evidence = run_gate(rows, collector="injected", config=_config(tmp_path), update=update) - gate = evidence["gate"] - assert gate["passed"] and gate["result"]["reward_std"] > 0 - assert gate["result"]["success_count"] == 1 - assert gate["result"]["advantage_abs_sum"] > 0 - # CPU orchestration only: the 5.8 gate is never announced off the real run. - assert evidence["verdict"] is False - assert "never announced" in evidence["verdict_note"] - - -def test_all_zero_rewards_fail(): - groups = [[_sample(0, i, 0.0, False) for i in range(4)]] - samples = collect_rollout_samples("ckpt", loader=lambda _p: _payload(groups)) - evidence = run_gate(build_gate_rows(samples, _update()), collector="miles_island") - assert evidence["verdict"] is False and evidence["gate"]["passed"] is False - assert "reward_std_positive" in evidence["gate"]["error"] - assert "success_count_positive" in evidence["gate"]["error"] - assert "advantage_abs_sum_positive" in evidence["gate"]["error"] - - -def test_constant_nonzero_rewards_fail(): - groups = [[_sample(0, i, 1.0, True) for i in range(4)]] - samples = collect_rollout_samples("ckpt", loader=lambda _p: _payload(groups)) - evidence = run_gate(build_gate_rows(samples, _update()), collector="miles_island") - assert evidence["verdict"] is False - assert "reward_std_positive" in evidence["gate"]["error"] - - -def test_no_policy_update_fails_even_with_real_collector(): - samples = collect_rollout_samples("ckpt", loader=lambda _p: _payload(_passing_groups())) - update = _update(updated=False) - rows = build_gate_rows(samples, update) - evidence = run_gate(rows, collector="miles_island", update=update) - assert evidence["verdict"] is False - assert "policy_update" in evidence["gate"]["error"] - pu = evidence["policy_update_evidence"] - assert pu["updated"] is False and pu["before_digest"] == pu["after_digest"] - - -def test_real_collector_passing_run_announces_verdict(tmp_path): - samples = collect_rollout_samples("ckpt", loader=lambda _p: _payload(_passing_groups())) - update = _update() - rows = build_gate_rows(samples, update) - evidence = run_gate(rows, collector="miles_island", config=_config(tmp_path), update=update) - assert evidence["verdict"] is True and evidence["schema"] == EVIDENCE_SCHEMA - assert evidence["policy_update_evidence"]["before_digest"] != \ - evidence["policy_update_evidence"]["after_digest"] - - -# -- policy-update proof from the island event tape (代拍板 v2) ----------------- - -def _tape(tmp_path, events): - path = tmp_path / "rl-island-0.jsonl" - path.write_text("".join(json.dumps(e) + "\n" for e in events), encoding="utf-8") - return path - - -def _publication(version, digest): - return {"event": "rl_publication", "policy_version": version, - "sync/publication_payload_hash": digest} - - -def test_tape_policy_update_detects_changed_payload(tmp_path): - tape = _tape(tmp_path, [ - {"event": "rl_driver_start"}, - _publication(0, "aaa"), - "not json at all", - _publication(1, "bbb"), - _publication(2, "ccc"), - ]) - update = collect_policy_update_from_tape(tape, step=2) - assert update.updated is True - assert update.before_digest == "aaa" and update.after_digest == "ccc" - assert update.changed == ("policy_payload",) - - -def test_tape_policy_update_unchanged_payload_marks_not_updated(tmp_path): - tape = _tape(tmp_path, [_publication(0, "aaa"), _publication(1, "aaa")]) - update = collect_policy_update_from_tape(tape, step=1) - assert update.updated is False and update.changed == () - - -def test_tape_policy_update_fails_closed_on_single_version(tmp_path): - tape = _tape(tmp_path, [_publication(0, "aaa")]) - with pytest.raises(GateEvidenceError, match="published policy version"): - collect_policy_update_from_tape(tape, step=1) - - -def test_tape_policy_update_fails_closed_on_missing_hash(tmp_path): - tape = _tape(tmp_path, [_publication(0, "aaa"), {"event": "rl_publication", - "policy_version": 1}]) - with pytest.raises(GateEvidenceError, match="payload hash"): - collect_policy_update_from_tape(tape, step=1) - - -def test_tape_policy_update_passes_full_gate(tmp_path): - samples = collect_rollout_samples("ckpt", loader=lambda _p: _payload(_passing_groups())) - tape = _tape(tmp_path, [_publication(0, "aaa"), _publication(1, "bbb")]) - update = collect_policy_update_from_tape(tape, step=1) - rows = build_gate_rows(samples, update) - evidence = run_gate(rows, collector="miles_island", update=update) - assert evidence["verdict"] is True - - -# -- evidence on disk: content and flag hygiene -------------------------------- - -def test_write_evidence_scrubs_flags_and_keeps_detail(tmp_path): - config = _config(tmp_path) - samples = collect_rollout_samples("ckpt", loader=lambda _p: _payload(_passing_groups())) - rows = build_gate_rows(samples, _update()) - rows[0]["metadata"]["note"] = f"grader saw {FLAGS['fixture-ctf-ezshell']} then submitted" - evidence = run_gate(rows, collector="injected", config=config, update=_update()) - out = write_evidence(evidence, tmp_path / "evidence.json", secrets=config.flags.values()) - text = out.read_text() - for flag in FLAGS.values(): - assert flag not in text - assert "***" in text - loaded = json.loads(text) - assert loaded["schema"] == EVIDENCE_SCHEMA - assert {r["reward"] for r in loaded["rows"]} == {0.0, 1.0} - assert "flag{" not in json.dumps(loaded["config_digest"]) # digest only - - -def test_config_requires_flags_for_gate_tasks(tmp_path): - with pytest.raises(GateEvidenceError, match="no trusted flag"): - GateConfig(tasks_dir=str(tmp_path), flags={"fixture-ctf-ezshell": "flag{x}"}) - - -# -- prompt data preparation ---------------------------------------------------- - -def _package(root, task_id): - (root / "public").mkdir(parents=True) - (root / "public" / "README.md").write_text(f"# {task_id}\nObtain the flag.") - (root / "task.yaml").write_text(f"task_id: {task_id}\ngrader_mode: flag\n") - (root / "MANIFEST.json").write_text(json.dumps({"task_id": task_id, "grader_mode": "flag"})) - - -def test_prepare_prompt_data_admits_gate_tasks(tmp_path): - for task_id in FLAGS: - _package(tmp_path / task_id, task_id) - out = prepare_prompt_data(_config(tmp_path), tmp_path / "prompt.jsonl") - rows = [json.loads(line) for line in Path(out).read_text().splitlines() if line.strip()] - assert len(rows) == 2 * len(FLAGS) # repeat=2 per task - assert {r["metadata"]["rlf_task_id"] for r in rows} == set(FLAGS) - assert all(r["metadata"]["benchmark"] == "rlf" for r in rows) - - -def test_prepare_prompt_data_rejects_unadmitted_package(tmp_path): - _package(tmp_path / "fixture-ctf-ezshell", "fixture-ctf-ezshell") - from yeto.rl.rlf.contracts import AdmissionError - with pytest.raises(AdmissionError): - prepare_prompt_data(_config(tmp_path), tmp_path / "prompt.jsonl") - - -# --- rl_sample_outcome tape collection (ports engine, S21 gate-7) ----------- - -def _outcome(group, index, reward, success, task="fixture-ctf-ezshell"): - return {"event": "rl_sample_outcome", "policy_version": 0, - "group_index": group, "index": index, - "task_id": task, "reward": reward, "success": success} - - -def test_tape_sample_outcomes_collect(tmp_path): - from yeto.rl.eval.rlf_rl_gate import collect_rollout_samples_from_tape - tape = _tape(tmp_path, [ - {"event": "rl_driver_start"}, - _outcome(0, 0, 1.0, True), - _outcome(0, 1, 0.0, False), - "garbage", - _outcome(1, 0, 0.0, False, task="fixture-ctf-ret2win"), - ]) - records = collect_rollout_samples_from_tape(tape) - assert len(records) == 3 - assert records[0]["reward"] == 1.0 and records[0]["success"] is True - assert records[2]["task_id"] == "fixture-ctf-ret2win" - - -def test_tape_sample_outcomes_fail_closed(tmp_path): - from yeto.rl.eval.rlf_rl_gate import collect_rollout_samples_from_tape - from yeto.rl.eval.rlf_rl_gate import GateEvidenceError - # no outcome records at all - tape = _tape(tmp_path, [{"event": "rl_driver_start"}]) - with pytest.raises(GateEvidenceError, match="no rl_sample_outcome"): - collect_rollout_samples_from_tape(tape) - # missing success must abort, never default to False - tape = _tape(tmp_path, [_outcome(0, 0, 1.0, None)]) - with pytest.raises(GateEvidenceError, match="lacks success"): - collect_rollout_samples_from_tape(tape) - # non-numeric reward aborts - bad = _outcome(0, 0, "1.0", True) - tape = _tape(tmp_path, [bad]) - with pytest.raises(GateEvidenceError, match="numeric reward"): - collect_rollout_samples_from_tape(tape) - - -def _trajectory_reward(group, sample, reward, success, task="fixture-ctf-ezshell"): - return {"event": "rl_trajectory_reward", "rollout_id": 0, "policy_version": 0, - "group_index": group, "sample_index": sample, "trajectory_id": f"t-{sample}", - "task_id": task, "reward": reward, "success": success, "aborted": False} - - -def test_tape_trajectory_rewards_collect(tmp_path): - """rlf-codex-rl 5.8 attempt 8 lesson: on the ports engine the all-samples - hook is rollout_meta_hook (queue_completed_groups never runs); the driver - tapes its per-sample records as rl_trajectory_reward.""" - from yeto.rl.eval.rlf_rl_gate import collect_rollout_samples_from_tape - tape = _tape(tmp_path, [ - {"event": "rl_driver_start"}, - _trajectory_reward(0, 0, 1.0, True), - _trajectory_reward(0, 1, 0.0, False), - _trajectory_reward(1, 0, 0.0, False, task="fixture-ctf-ret2win"), - ]) - records = collect_rollout_samples_from_tape(tape) - assert len(records) == 3 - assert records[0]["reward"] == 1.0 and records[0]["success"] is True - assert records[0]["index"] == 0 and records[1]["index"] == 1 - assert records[2]["task_id"] == "fixture-ctf-ret2win" - - -def test_tape_trajectory_rewards_fail_closed(tmp_path): - from yeto.rl.eval.rlf_rl_gate import collect_rollout_samples_from_tape - from yeto.rl.eval.rlf_rl_gate import GateEvidenceError - # success None (no signed verdict) must abort, never default to False - tape = _tape(tmp_path, [_trajectory_reward(0, 0, 1.0, None)]) - with pytest.raises(GateEvidenceError, match="lacks success"): - collect_rollout_samples_from_tape(tape) - # reward None (non-finite) must abort - tape = _tape(tmp_path, [_trajectory_reward(0, 0, None, True)]) - with pytest.raises(GateEvidenceError, match="numeric reward"): - collect_rollout_samples_from_tape(tape) - - diff --git a/tests/test_rlf_rollout_entry.py b/tests/test_rlf_rollout_entry.py deleted file mode 100644 index 55616e59a..000000000 --- a/tests/test_rlf_rollout_entry.py +++ /dev/null @@ -1,226 +0,0 @@ -"""RLF rollout entry points (W2): catalog -> prompt data, identity producer, -Modal provider factory contract, compaction profile registration, RLF grade -interface, and RLF metadata preservation across compaction segments. - -CPU only: fake samples/fake VM, no Ray, no Miles, no Modal, no torch. -""" - -from __future__ import annotations - -import asyncio -import dataclasses -import json -from types import SimpleNamespace -from typing import Any - -import pytest - -from yeto.rl.compaction import T_COMP -from yeto.rl.harness.codex import codex_harness_agent as harness -from yeto.rl.harness.codex import codex_openenv_generate as generate_wrapper -from yeto.rl.harness.codex import compaction_bridge as cb -from yeto.rl.rlf.catalog import ( - TaskCatalog, import_catalog, prompt_rows, rlf_identity, task_statement, write_prompt_data, -) -from yeto.rl.rlf.contracts import AdmissionError, EpisodeLease, TrustedGrade -from yeto.rl.rlf.preflight import RolloutRejected, trusted_grade_from_evaluation - - -# --- helpers ----------------------------------------------------------------- - -def _package(root, task_id="fixture-ctf-ret2win"): - (root / "public").mkdir(parents=True) - (root / "public" / "README.md").write_text("# ret2win\nConnect to the target and obtain the flag.") - (root / "task.yaml").write_text(f"task_id: {task_id}\ngrader_mode: flag\n") - (root / "MANIFEST.json").write_text(json.dumps({"task_id": task_id, "grader_mode": "flag"})) - - -def _catalog(tmp_path): - _package(tmp_path / "pkg") - return import_catalog([tmp_path / "pkg"], evidence={"runtime_quality_gate": {"healthy": True}}) - - -def _sample(n_prompt, n_gen, version="pv", **meta): - tokens = list(range(n_prompt + n_gen)) - span = SimpleNamespace(version=version, abs_start=n_prompt, abs_end=n_prompt + n_gen) - return SimpleNamespace(tokens=tokens, loss_mask=[1] * n_gen, rollout_log_probs=[-0.1] * n_gen, - response_length=n_gen, weight_versions=[SimpleNamespace(spans=[span])], - metadata=dict(meta), status=None) - - -# --- breakpoint 2: catalog -> rollout prompt data ---------------------------- - -def test_catalog_prompt_rows_feed_miles_prompt_data(tmp_path): - catalog = _catalog(tmp_path) - rows = prompt_rows(catalog) - assert len(rows) == 1 - row = rows[0] - md = row["metadata"] - # harness task_identity reads metadata.task_id; the RLF producer reads rlf_task_id. - assert md["task_id"] == "fixture-ctf-ret2win" == md["rlf_task_id"] - assert md["grader_mode"] == "flag" and md["benchmark"] == "rlf" - assert md["package_digest"] == catalog.get("fixture-ctf-ret2win").package_digest - assert "obtain the flag" in md["prompt"] and row["prompt"] == md["prompt"] - out = write_prompt_data(catalog, tmp_path / "prompt.jsonl", repeat=2) - loaded = [json.loads(line) for line in out.read_text().splitlines() if line.strip()] - assert len(loaded) == 2 and loaded[0] == row - - -def test_catalog_refuses_fabricated_statement_and_unknown_task(tmp_path): - root = tmp_path / "pkg" - _package(root) - (root / "public" / "README.md").unlink() - catalog = import_catalog([root], evidence={"runtime_quality_gate": {"healthy": True}}) - with pytest.raises(AdmissionError, match="task statement"): - prompt_rows(catalog) - with pytest.raises(KeyError): - prompt_rows(catalog, ["no-such-task"]) - - -def test_catalog_episode_produces_rlf_identity(tmp_path): - catalog = _catalog(tmp_path) - lease = catalog.episode("fixture-ctf-ret2win") - assert isinstance(lease, EpisodeLease) and lease.task_id == "fixture-ctf-ret2win" - assert lease.episode_id and lease.trajectory_id - other = catalog.episode("fixture-ctf-ret2win") - assert lease.episode_id != other.episode_id # episodes are never shared - identity = rlf_identity(lease, policy_id="pv") - assert identity == {"rlf_task_id": "fixture-ctf-ret2win", "rlf_episode_id": lease.episode_id, - "rlf_trajectory_id": lease.trajectory_id, "policy_id": "pv"} - - -# --- breakpoint 1: identity producer on the real generate path --------------- - -def test_bookkeeping_produces_rlf_identity_for_catalog_rows(): - inp = SimpleNamespace(group_index=3, index=5, - metadata={"rlf_task_id": "fixture-ctf-ret2win"}) - chains = [_sample(4, 6), _sample(4, 6)] - generate_wrapper.apply_trajectory_bookkeeping(inp, chains, expected_version="pv") - md = [c.metadata for c in chains] - assert all(m["rlf_task_id"] == "fixture-ctf-ret2win" for m in md) - assert len({m["rlf_episode_id"] for m in md}) == 1 # one episode per rollout - assert len({m["rlf_trajectory_id"] for m in md}) == 1 - assert all(m["policy_id"] == "pv" for m in md) - assert all(c.status is None for c in chains) # versions match: no abort - - -def test_bookkeeping_keeps_existing_identity_and_ignores_non_rlf_rows(): - inp = SimpleNamespace(group_index=3, index=5, metadata={ - "rlf_task_id": "fixture-ctf-ret2win", "rlf_episode_id": "ep-fixed", - "rlf_trajectory_id": "traj-fixed", "policy_id": "pv-old"}) - chain = _sample(4, 6) - generate_wrapper.apply_trajectory_bookkeeping(inp, [chain], expected_version="pv") - assert chain.metadata["rlf_episode_id"] == "ep-fixed" - assert chain.metadata["rlf_trajectory_id"] == "traj-fixed" - assert chain.metadata["policy_id"] == "pv-old" # never overwritten - plain_inp = SimpleNamespace(group_index=3, index=5, metadata={"task_id": "tb2-task"}) - plain = _sample(4, 6) - generate_wrapper.apply_trajectory_bookkeeping(plain_inp, [plain], expected_version="pv") - assert not any(k in plain.metadata for k in generate_wrapper.RLF_IDENTITY_KEYS) - - -# --- breakpoint: rlf_modal_provider factory contract -------------------------- - -def test_rlf_modal_provider_matches_preflight_factory_contract(monkeypatch): - from yeto.cloud import modal_reward_env as mre - from yeto.rl.rlf import modal as rlf_modal - - class FakeSandbox: - object_id = "sb-fake" - - def terminate(self): - pass - - calls = {} - - def fake_acquire_vm(task_id, episode_id, *, config=None, image=None, trajectory_id=None): - calls.update(task_id=task_id, episode_id=episode_id, trajectory_id=trajectory_id) - return rlf_modal.RlfVmLease(FakeSandbox(), task_id=task_id, episode_id=episode_id, - trajectory_id=trajectory_id) - - monkeypatch.setattr(rlf_modal, "acquire_vm", fake_acquire_vm) - factory = mre.rlf_modal_provider - provider = factory(SimpleNamespace()) # preflight calls factory(miles_args) - assert hasattr(provider, "acquire") # resolve_environment_provider's only check - lease = asyncio.run(provider.acquire("fixture-ctf-ret2win", "traj-1")) - assert lease.task_id == "fixture-ctf-ret2win" - assert lease.episode_id and lease.trajectory_id == "traj-1" - assert calls["trajectory_id"] == "traj-1" and calls["episode_id"] == lease.episode_id - lease.teardown() # idempotent cleanup path stays intact - assert lease.closed - - -# --- compaction profile registration (5.7) ------------------------------------ - -def test_qwen3_cyber_profile_is_registered_and_selectable(): - assert "qwen3_cyber" in cb.compaction_profiles() - cfg = cb.compaction_config(24_576, env={cb.PROFILE_ENV: "qwen3_cyber"}) - assert (cfg.context_budget, cfg.t_comp, cfg.max_compactions) == (24_576, 4_096, 3) - with pytest.raises(harness.CodexHarnessError, match="context_budget"): - cb.compaction_config(32_768, env={cb.PROFILE_ENV: "qwen3_cyber"}) - with pytest.raises(harness.CodexHarnessError, match="not a registered"): - cb.compaction_config(24_576, env={cb.PROFILE_ENV: "no-such-profile"}) - - -def test_compaction_config_default_derivation_unchanged(): - cfg = cb.compaction_config(32_768, env={}) - assert cfg.context_budget == 32_768 and cfg.t_comp == T_COMP - cfg = cb.compaction_config(32_768, env={cb.T_COMP_ENV: "4096"}) - assert cfg.t_comp == 4_096 - with pytest.raises(harness.CodexHarnessError): - cb.compaction_config(32_768, env={cb.T_COMP_ENV: "x"}) - - -# --- trusted finish: RLF grade interface (breakpoint 5, interface only) ------- - -def test_trusted_grade_from_evaluation_prefers_rlf_and_falls_back(): - assert trusted_grade_from_evaluation({"passed": True, "testsh_rc": 0}) is None # TB2 HMAC fallback - payload = {"status": "completed", "reward": 1.0, "passed": True, - "grader_evidence": {"flag": "matched"}} - grade = trusted_grade_from_evaluation({"rlf_grade": payload, "log": "..."}) - assert isinstance(grade, TrustedGrade) and grade.passed and grade.reward == 1.0 - assert grade.grader_evidence == {"flag": "matched"} - direct = trusted_grade_from_evaluation(payload) # a bare full-grade mapping also parses - assert direct is not None and direct.status == "completed" - with pytest.raises(RolloutRejected, match="malformed"): - trusted_grade_from_evaluation({"rlf_grade": {"status": "completed"}}) - - -# --- RLF metadata survives compaction segment splitting ------------------------ - -@dataclasses.dataclass(frozen=True) -class _Out: - samples: Any - - -def test_compaction_segments_preserve_rlf_identity_and_reward_metadata(): - seg0_meta = { - "expected_policy_version": "pv", "session_server_id": "10.0.0.1:9", - "exit_status": "completed", - "rlf_task_id": "fixture-ctf-ret2win", "rlf_episode_id": "ep-1", - "rlf_trajectory_id": "traj-1", "policy_id": "pv", - "success": True, "reward": 1.0, "grader_evidence": {"flag": "matched"}, - generate_wrapper.SESSIONS_KEY: ["a", "b"], - "agent_metrics": {cb.METRICS_KEY: {"compactions": 1}}, - } - seg0 = _sample(10, 30, **seg0_meta) - s1 = _sample(8, 20) - inp = SimpleNamespace(sample=SimpleNamespace(group_index=7, index=11, - metadata={"expected_policy_version": "pv"})) - - async def upstream(_input): - return _Out(samples=seg0) - - async def collect(_input, router, sid): - return (([s1], {"seg": 1}) if sid == "a" else ([], {})) - - out = asyncio.run(generate_wrapper.generate(inp, upstream=upstream, collect=collect)) - assert out.samples == [seg0, s1] - for sample in out.samples: - md = sample.metadata - assert md["rlf_task_id"] == "fixture-ctf-ret2win" - assert md["rlf_episode_id"] == "ep-1" and md["rlf_trajectory_id"] == "traj-1" - assert md["policy_id"] == "pv" - assert md["success"] is True and md["grader_evidence"] == {"flag": "matched"} - assert [s.metadata["segment_index"] for s in out.samples] == [0, 1] - assert s1.metadata["seg"] == 1 diff --git a/tests/test_run_flags.py b/tests/test_run_flags.py index 9ef5c8eab..90995176e 100644 --- a/tests/test_run_flags.py +++ b/tests/test_run_flags.py @@ -251,10 +251,10 @@ def test_forward_tables_name_flags_of_both_layers(): _NON_RL_LITERALS = frozenset({ "--abbrev-ref", "--adapter-dir", "--add", "--address", "--allow-unattested-legacy-adapter", "--allow-unauthenticated-islands", "--bind", "--cap-add", "--carry-gamma", - "--checkpoint-every", "--checkpoint-path", "--codex-harness-binary", "--config-json", + "--checkpoint-every", "--checkpoint-path", "--config-json", "--create", "--dashboard-host", "--dearmor", "--depth", "--detach", "--dtype", "--entrypoint", "--expected-island-contract", "--final-ack-timeout-s", "--final-grace-s", "--follow", "--force", - "--format", "--get", "--global", "--gpus", "--head", "--ignore-installed", + "--format", "--get", "--global", "--gpus", "--head", "--include-dashboard", "--index-url", "--ipc", "--island-lease-s", "--island-scheduling-mode", "--json", "--learner-weight", "--learners", "--level", "--manifest-path", "--mark-final-checkpoint", "--master", "--matrix-merge", diff --git a/tests/test_sandbox_isolation.py b/tests/test_sandbox_isolation.py deleted file mode 100644 index b7b04f1b5..000000000 --- a/tests/test_sandbox_isolation.py +++ /dev/null @@ -1,142 +0,0 @@ -"""rl-agentic-reward-env 5.1/5.2: sandbox egress closed by default, minimal local env.""" - -from __future__ import annotations - -import json -import sys -import types -from pathlib import Path - -import pytest - -from yeto.rl.harness.codex import tb2_provider as tb2 -from yeto.rl.harness.codex.tb2_network_grants import TB2_OPEN_EGRESS_TASKS - - -def _task(tmp_path: Path, task_id: str = "t1") -> tb2.Tb2Task: - return tb2.Tb2Task(task_id=task_id, task_dir=tmp_path, docker_image="img:1", cpus=1, memory_mb=2048, - workdir="/app", agent_timeout_s=60, verifier_timeout_s=60) - - -def test_default_policy_is_closed(): - grant = tb2.NetworkPolicy().grant("anything") - assert grant.closed - assert grant.modal_kwargs() == {"block_network": True} - - -def test_grant_forms(): - policy = tb2.NetworkPolicy.from_mapping({"tasks": { - "pip": {"domains": ["pypi.org", "*.pythonhosted.org"]}, - "net": {"cidrs": ["10.0.0.0/8"]}, - "web": "open", - "off": "closed", - }}) - assert policy.grant("pip").modal_kwargs() == {"outbound_domain_allowlist": ["pypi.org", "*.pythonhosted.org"]} - assert policy.grant("net").modal_kwargs() == {"outbound_cidr_allowlist": ["10.0.0.0/8"]} - assert policy.grant("web").modal_kwargs() == {} - assert policy.grant("off").modal_kwargs() == {"block_network": True} - assert policy.grant("unlisted").modal_kwargs() == {"block_network": True} - - -@pytest.mark.parametrize("bad", [ - {"tasks": {"x": {"domains": ["*"]}}}, - {"tasks": {"x": {"cidrs": ["not-a-cidr"]}}}, - {"tasks": {"x": {"hosts": ["a"]}}}, - {"tasks": {"x": "yes"}}, - {"tasks": [], "default": "closed"}, - {"extra": 1}, -]) -def test_bad_policy_fails_closed(bad): - with pytest.raises(ValueError): - tb2.NetworkPolicy.from_mapping(bad) - - -def test_builtin_grants_every_tb2_task_open_explicitly(): - assert len(TB2_OPEN_EGRESS_TASKS) == 89 == len(set(TB2_OPEN_EGRESS_TASKS)) - policy = tb2.NetworkPolicy.builtin() - assert policy.default.closed - for task_id in ("adaptive-rejection-sampler", "build-pmars", "qemu-startup"): - assert policy.grant(task_id).open - assert policy.grant("not-a-tb2-task").closed - - -def test_policy_from_env(tmp_path, monkeypatch): - monkeypatch.delenv(tb2.NETWORK_POLICY_ENV, raising=False) - assert tb2.NetworkPolicy.from_env() == tb2.NetworkPolicy.builtin() - path = tmp_path / "p.json" - path.write_text(json.dumps({"tasks": {"pip": {"domains": ["pypi.org"]}}})) - monkeypatch.setenv(tb2.NETWORK_POLICY_ENV, str(path)) - policy = tb2.NetworkPolicy.from_env() - assert policy.grant("pip").domains == ("pypi.org",) - assert policy.grant("adaptive-rejection-sampler").closed - - -def _fake_modal(monkeypatch, calls): - mod = types.ModuleType("modal") - - class Sandbox: - @staticmethod - def create(*args, **kwargs): - calls.append(kwargs) - return types.SimpleNamespace(object_id="sb-1") - - class Image: - @staticmethod - def from_registry(name): - return types.SimpleNamespace(name=name, run_commands=lambda *a: "img") - - class App: - @staticmethod - def lookup(name, create_if_missing=False): - return "app" - - mod.Sandbox, mod.Image, mod.App = Sandbox, Image, App - monkeypatch.setitem(sys.modules, "modal", mod) - - -def test_modal_backend_passes_network_kwargs(tmp_path, monkeypatch): - calls: list[dict] = [] - _fake_modal(monkeypatch, calls) - tb2.ModalSandboxBackend().create(_task(tmp_path), "traj") - assert calls[-1]["block_network"] is True - policy = tb2.NetworkPolicy.from_mapping({"tasks": {"t1": {"domains": ["pypi.org"]}}}) - tb2.ModalSandboxBackend(network_policy=policy).create(_task(tmp_path), "traj") - assert calls[-1]["outbound_domain_allowlist"] == ["pypi.org"] - assert "block_network" not in calls[-1] - - -def test_prebaked_modal_backend_passes_network_kwargs(tmp_path, monkeypatch): - from yeto.cloud import modal_reward_env - - calls: list[dict] = [] - _fake_modal(monkeypatch, calls) - (tmp_path / "tests").mkdir() - (tmp_path / "tests" / "test.sh").write_text("#!/bin/bash\necho 1\n") - modal_reward_env.PrebakedModalSandboxBackend(prebake=False).create(_task(tmp_path), "traj") - assert calls[-1]["block_network"] is True - backend = modal_reward_env.PrebakedModalSandboxBackend(prebake=False, network_policy=tb2.NetworkPolicy.builtin()) - backend.create(_task(tmp_path, "adaptive-rejection-sampler"), "traj") - assert not {"block_network", "outbound_domain_allowlist", "outbound_cidr_allowlist"} & set(calls[-1]) - - -def test_minimal_env_drops_tokens(): - parent = {"PATH": "/x/bin", "MODAL_TOKEN_SECRET": "s", "HF_TOKEN": "h", "TBENCH_REWARD_HMAC_KEY": "k", - "WANDB_API_KEY": "w", "LANG": "en_US.UTF-8", "HOME": "/home/u"} - assert tb2.minimal_env(parent) == {"PATH": "/x/bin", "LANG": "en_US.UTF-8"} - assert tb2.minimal_env({})["PATH"] == tb2.DEFAULT_PATH - - -def test_local_sandbox_does_not_inherit_parent_env(tmp_path, monkeypatch): - monkeypatch.setenv("MODAL_TOKEN_SECRET", "secret-value") - monkeypatch.setenv("YETO_FAKE_TOKEN", "x") - backend = tb2.LocalProcessBackend(tmp_path) - sandbox = backend.create(_task(tmp_path), "traj") - try: - result = sandbox.exec("env", timeout_s=10) - finally: - sandbox.terminate() - assert result.exit_code == 0 - names = {line.split("=", 1)[0] for line in result.output.splitlines() if "=" in line} - assert "MODAL_TOKEN_SECRET" not in names and "YETO_FAKE_TOKEN" not in names - assert {"PATH", "HOME", "TB2_TESTS_DIR", "TB2_VERIFIER_LOGS_DIR"} <= names - assert sandbox.network.closed diff --git a/tests/test_secret_handling_hardening.py b/tests/test_secret_handling_hardening.py index a9def9dcb..294fde129 100644 --- a/tests/test_secret_handling_hardening.py +++ b/tests/test_secret_handling_hardening.py @@ -199,37 +199,6 @@ def test_secret_envs_are_split_out_of_plain_envs(): assert secrets == {"HF_TOKEN": "h", "WANDB_API_KEY": "w", "YETO_SANDBOX_MODAL_TOKEN_SECRET": "s"} -def test_main_modal_token_is_not_a_harness_passthrough(): - from yeto import launcher - - assert not set(launcher.HARNESS_PASSTHROUGH_ENV) & launcher.CLOUD_CREDENTIAL_ENV_NAMES - with pytest.raises(ValueError, match="YETO_SANDBOX_MODAL_TOKEN_ID"): - launcher.require_sandbox_modal_token({"MODAL_TOKEN_ID": "a", "MODAL_TOKEN_SECRET": "b"}) - launcher.require_sandbox_modal_token( - {"YETO_SANDBOX_MODAL_TOKEN_ID": "a", "YETO_SANDBOX_MODAL_TOKEN_SECRET": "b"}) - - -def test_sandbox_client_uses_only_the_sandbox_token(monkeypatch): - from yeto.rl.harness.codex import tb2_provider - - assert tb2_provider.sandbox_modal_client({"MODAL_TOKEN_ID": "a", "MODAL_TOKEN_SECRET": "b"}) is None - seen = {} - - class FakeClient: - @staticmethod - def from_credentials(token_id, token_secret): - seen["args"] = (token_id, token_secret) - return "client" - - import sys - import types - - monkeypatch.setitem(sys.modules, "modal", types.SimpleNamespace(Client=FakeClient)) - env = {"YETO_SANDBOX_MODAL_TOKEN_ID": "sid", "YETO_SANDBOX_MODAL_TOKEN_SECRET": "ssec"} - assert tb2_provider.sandbox_modal_client(env) == "client" - assert seen["args"] == ("sid", "ssec") - - # --------------------------------------------------------------------------- D4 island start-up check def test_guard_names_match_the_launcher_list(): diff --git a/tests/test_secrlenv_adapter.py b/tests/test_secrlenv_adapter.py deleted file mode 100644 index 02d81d455..000000000 --- a/tests/test_secrlenv_adapter.py +++ /dev/null @@ -1,1937 +0,0 @@ -from __future__ import annotations - -import asyncio -import copy -import json -import sys -import types -from io import BytesIO -from types import SimpleNamespace - -import pytest - -pytest.importorskip( - "yeto_miles_secrlenv", - reason="the Miles SecRLEnv adapter is an optional external integration", -) - -from yeto_miles_secrlenv import agent, reward -from yeto_miles_secrlenv import generate as secrlenv_generate -from yeto_miles_secrlenv.client import ( - EpisodeAPIError, - EpisodeClient, - EpisodeClientError, - EpisodeTransportError, - require_daemon_ready, -) - -from yeto.rl.harness.codex import reward as secrlenv_reward - - -class FakeMessage: - def __init__(self, value): - self.value = value - - def model_dump(self, *, exclude_none=True): - return copy.deepcopy(self.value) - - -class FakeCompletions: - def __init__(self, messages): - self.responses = list(messages) - self.requests = [] - - async def create(self, **kwargs): - self.requests.append(copy.deepcopy(kwargs)) - return SimpleNamespace( - choices=[SimpleNamespace(message=FakeMessage(self.responses.pop(0)))] - ) - - -class FakePolicy: - def __init__(self, messages): - self.chat = SimpleNamespace(completions=FakeCompletions(messages)) - - -class FakeEpisodeClient: - def __init__(self): - self.exec_calls = [] - self.submissions = [] - - async def execute( - self, episode_id, command, *, timeout_seconds, output_bytes - ): - self.exec_calls.append( - (episode_id, command, timeout_seconds, output_bytes) - ) - return { - "exit_code": 0, - "output": "command output", - "timed_out": False, - "truncated": False, - } - - async def submit(self, episode_id, submission): - self.submissions.append((episode_id, submission)) - return {"accepted": True} - - -def _install_run_openai(monkeypatch): - class RunPolicy: - async def close(self): - return None - - openai = types.ModuleType("openai") - openai.AsyncOpenAI = lambda **_kwargs: RunPolicy() - monkeypatch.setitem(sys.modules, "openai", openai) - - -def _run_metadata(): - return {"task_id": "CVE-2024-1234", "prompt_tier": "l2"} - - -def test_exhausted_precreate_infrastructure_503_returns_signed_admission( - monkeypatch, -): - attempts = [] - - class AdmissionClient: - def __init__(self, **_kwargs): - pass - - async def __aenter__(self): - return self - - async def __aexit__(self, *_args): - return None - - async def create(self, task_id, tier): - attempts.append((task_id, tier)) - raise EpisodeAPIError(503, "infrastructure_error", "unavailable") - - _install_run_openai(monkeypatch) - monkeypatch.setattr(agent, "EpisodeClient", AdmissionClient) - monkeypatch.setattr(agent.random, "uniform", lambda *_args: 0.0) - monkeypatch.setenv("SECRLENV_REWARD_HMAC_KEY", "a" * 48) - monkeypatch.setenv("SECRLENV_CAPACITY_MAX_WAIT_SECONDS", "30") - - result = asyncio.run( - agent.run("http://127.0.0.1:8000/v1", None, metadata=_run_metadata()) - ) - - assert result is not None - outcome, value = reward._verified_outcome(result) - assert len(attempts) == 2 - assert outcome["schema"] == reward.ADMISSION_SCHEMA - assert outcome["status"] == reward.INFRASTRUCTURE_STATUS - assert outcome["episode_id"] is None - assert outcome[reward.INFRASTRUCTURE_503_RETRIES_KEY] == 1 - assert value == 0.0 - agent.require_no_episode_residue() - - -def test_precreate_transport_failure_remains_unsigned_and_fatal(monkeypatch): - class TransportFailingClient: - def __init__(self, **_kwargs): - pass - - async def __aenter__(self): - return self - - async def __aexit__(self, *_args): - return None - - async def create(self, _task_id, _tier): - raise EpisodeTransportError("unavailable") - - _install_run_openai(monkeypatch) - monkeypatch.setattr(agent, "EpisodeClient", TransportFailingClient) - monkeypatch.setenv("SECRLENV_REWARD_HMAC_KEY", "a" * 48) - - assert ( - asyncio.run( - agent.run( - "http://127.0.0.1:8000/v1", None, metadata=_run_metadata() - ) - ) - is None - ) - agent.require_no_episode_residue() - - -def test_precreate_infrastructure_retry_timeout_remains_unsigned(monkeypatch): - attempts = 0 - - class AdmissionTimeoutClient: - def __init__(self, **_kwargs): - pass - - async def __aenter__(self): - return self - - async def __aexit__(self, *_args): - return None - - async def create(self, _task_id, _tier): - nonlocal attempts - attempts += 1 - if attempts == 1: - raise EpisodeAPIError( - 503, "infrastructure_error", "unavailable" - ) - await asyncio.Event().wait() - - _install_run_openai(monkeypatch) - monkeypatch.setattr(agent, "EpisodeClient", AdmissionTimeoutClient) - monkeypatch.setattr(agent.random, "uniform", lambda *_args: 0.0) - monkeypatch.setenv("SECRLENV_REWARD_HMAC_KEY", "a" * 48) - monkeypatch.setenv("SECRLENV_CAPACITY_MAX_WAIT_SECONDS", "0.01") - - result = asyncio.run( - agent.run("http://127.0.0.1:8000/v1", None, metadata=_run_metadata()) - ) - - assert result is None - assert attempts == 2 - agent.require_no_episode_residue() - - -def test_episode_finalization_cannot_be_claimed_by_abort(): - episode_id = "f" * 24 - agent._register_episode(episode_id) - try: - assert agent._claim_episode_finalization(episode_id) - assert agent._claim_driving_episodes_for_abort() == [] - finally: - agent._release_episode(episode_id) - - -def test_abort_claim_preempts_finalization_once(): - episode_id = "a" * 24 - agent._register_episode(episode_id) - try: - assert agent._claim_driving_episodes_for_abort() == [episode_id] - assert not agent._claim_episode_finalization(episode_id) - assert not agent._claim_episode_cleanup(episode_id) - finally: - agent._release_episode(episode_id) - - -def test_finalizing_episode_cannot_be_claimed_for_cancelled_cleanup(): - episode_id = "e" * 24 - agent._register_episode(episode_id) - try: - assert agent._claim_episode_finalization(episode_id) - assert not agent._claim_episode_cleanup(episode_id) - assert agent._claim_episode_finalization(episode_id) - finally: - agent._release_episode(episode_id) - - -def test_policy_completion_claims_finalization_before_abort_callback(): - episode_id = "b" * 24 - abort_claims = [] - - async def complete_policy(): - asyncio.get_running_loop().call_soon( - lambda: abort_claims.extend( - agent._claim_driving_episodes_for_abort() - ) - ) - return "completed" - - async def scenario(): - result = await agent._await_policy_and_claim_finalization( - episode_id, complete_policy() - ) - await asyncio.sleep(0) - return result - - agent._register_episode(episode_id) - try: - assert asyncio.run(scenario()) == "completed" - assert abort_claims == [] - assert agent._claim_episode_finalization(episode_id) - finally: - agent._release_episode(episode_id) - - -def test_registered_policy_task_includes_finalization_boundary(): - episode_id = "6" * 24 - abort_claims = [] - - async def complete_policy(): - asyncio.get_running_loop().call_soon( - lambda: abort_claims.extend( - agent._claim_driving_episodes_for_abort() - ) - ) - return "completed" - - async def scenario(): - policy_task = asyncio.create_task( - agent._await_policy_and_claim_finalization( - episode_id, complete_policy() - ) - ) - agent._register_episode(episode_id, policy_task) - result = await asyncio.wait_for(policy_task, timeout=1.0) - await asyncio.sleep(0) - return result - - try: - assert asyncio.run(scenario()) == "completed" - assert abort_claims == [] - assert agent._claim_episode_finalization(episode_id) - finally: - agent._release_episode(episode_id) - - -def test_cancelled_run_can_transfer_its_drained_boundary_to_cleanup(): - episode_id = "5" * 24 - - async def scenario(): - policy_task = asyncio.create_task( - agent._await_policy_and_claim_finalization( - episode_id, asyncio.sleep(0, result="completed") - ) - ) - agent._register_episode(episode_id, policy_task) - await policy_task - assert not agent._claim_episode_cleanup(episode_id) - assert agent._claim_cancelled_policy_cleanup(episode_id, policy_task) - - try: - asyncio.run(scenario()) - assert agent._EPISODE_PHASES[episode_id] == "aborting" - finally: - agent._release_episode(episode_id) - - -def test_run_cancellation_drains_evaluation_before_cleanup_close(monkeypatch): - episode_id = "3" * 24 - evaluation_started = None - evaluation_live = False - close_observations = [] - - class RunClient: - def __init__(self, **_kwargs): - pass - - async def __aenter__(self): - return self - - async def __aexit__(self, _exc_type, _exc, _traceback): - return False - - async def create(self, _task_id, _tier): - return {"episode_id": episode_id, "prompt": "task"} - - async def evaluate(self, value): - assert value == episode_id - nonlocal evaluation_live - evaluation_live = True - evaluation_started.set() - try: - await asyncio.Event().wait() - finally: - evaluation_live = False - - async def close(self, value): - assert value == episode_id - close_observations.append(evaluation_live) - return {} - - async def complete_policy(*_args, **_kwargs): - return "completed" - - async def scenario(): - nonlocal evaluation_started - evaluation_started = asyncio.Event() - run_task = asyncio.create_task( - agent.run("http://127.0.0.1:8000/v1", None, metadata=_run_metadata()) - ) - await evaluation_started.wait() - run_task.cancel() - with pytest.raises(asyncio.CancelledError): - await run_task - - _install_run_openai(monkeypatch) - monkeypatch.setattr(agent, "EpisodeClient", RunClient) - monkeypatch.setattr(agent, "_drive_policy", complete_policy) - asyncio.run(scenario()) - assert close_observations == [False] - assert episode_id not in agent._EPISODE_PHASES - - -def test_run_cancellation_drains_normal_close_before_cleanup_retry(monkeypatch): - episode_id = "2" * 24 - first_close_started = None - first_close_live = False - close_observations = [] - - class RunClient: - def __init__(self, **_kwargs): - pass - - async def __aenter__(self): - return self - - async def __aexit__(self, _exc_type, _exc, _traceback): - return False - - async def create(self, _task_id, _tier): - return {"episode_id": episode_id, "prompt": "task"} - - async def evaluate(self, value): - assert value == episode_id - return {} - - async def close(self, value): - assert value == episode_id - nonlocal first_close_live - close_observations.append(first_close_live) - if len(close_observations) == 1: - first_close_live = True - first_close_started.set() - try: - await asyncio.Event().wait() - finally: - first_close_live = False - assert not first_close_live - return {} - - async def complete_policy(*_args, **_kwargs): - return "completed" - - async def scenario(): - nonlocal first_close_started - first_close_started = asyncio.Event() - run_task = asyncio.create_task( - agent.run("http://127.0.0.1:8000/v1", None, metadata=_run_metadata()) - ) - await first_close_started.wait() - run_task.cancel() - with pytest.raises(asyncio.CancelledError): - await run_task - - _install_run_openai(monkeypatch) - monkeypatch.setattr(agent, "EpisodeClient", RunClient) - monkeypatch.setattr(agent, "_drive_policy", complete_policy) - asyncio.run(scenario()) - assert close_observations == [False, False] - assert episode_id not in agent._EPISODE_PHASES - - -def test_run_close_failure_retains_cleanup_pending_ownership(monkeypatch): - episode_id = "1" * 24 - - class RunClient: - def __init__(self, **_kwargs): - pass - - async def __aenter__(self): - return self - - async def __aexit__(self, _exc_type, _exc, _traceback): - return False - - async def create(self, _task_id, _tier): - return {"episode_id": episode_id, "prompt": "task"} - - async def evaluate(self, value): - assert value == episode_id - return { - "task_id": "CVE-2024-1234", - "episode_id": episode_id, - "reward": 0.0, - "passed": False, - } - - async def close(self, value): - assert value == episode_id - raise EpisodeTransportError("close failed") - - async def complete_policy(*_args, **_kwargs): - return "completed" - - _install_run_openai(monkeypatch) - monkeypatch.setattr(agent, "EpisodeClient", RunClient) - monkeypatch.setattr(agent, "_drive_policy", complete_policy) - monkeypatch.setenv("SECRLENV_REWARD_HMAC_KEY", "k" * 48) - try: - result = asyncio.run( - agent.run("http://127.0.0.1:8000/v1", None, metadata=_run_metadata()) - ) - assert result is not None - assert result["exit_status"] == reward.CLEANUP_ERROR_STATUS - assert ( - result[reward.OUTCOME_KEY]["status"] - == reward.CLEANUP_ERROR_STATUS - ) - assert agent._EPISODE_PHASES[episode_id] == "cleanup_pending" - assert agent._claim_episodes_and_tasks_for_abort() == [ - (episode_id, agent._EPISODE_POLICY_TASKS[episode_id]) - ] - finally: - agent._release_episode(episode_id) - - -def test_run_recovers_close_before_returning_retryable_infrastructure(monkeypatch): - episode_id = "2" * 24 - close_calls = [] - - class RunClient: - def __init__(self, **_kwargs): - pass - - async def __aenter__(self): - return self - - async def __aexit__(self, _exc_type, _exc, _traceback): - return False - - async def create(self, _task_id, _tier): - return {"episode_id": episode_id, "prompt": "task"} - - async def evaluate(self, value): - assert value == episode_id - return { - "task_id": "CVE-2024-1234", - "episode_id": episode_id, - "reward": 0.0, - "passed": False, - } - - async def close(self, value): - assert value == episode_id - close_calls.append(value) - if len(close_calls) == 1: - raise EpisodeTransportError("close failed once") - return {"closed": True} - - async def complete_policy(*_args, **_kwargs): - return "completed" - - _install_run_openai(monkeypatch) - monkeypatch.setattr(agent, "EpisodeClient", RunClient) - monkeypatch.setattr(agent, "_drive_policy", complete_policy) - monkeypatch.setenv("SECRLENV_REWARD_HMAC_KEY", "k" * 48) - - result = asyncio.run( - agent.run("http://127.0.0.1:8000/v1", None, metadata=_run_metadata()) - ) - - assert result is not None - assert result["exit_status"] == reward.INFRASTRUCTURE_STATUS - assert len(close_calls) == 2 - agent.require_no_episode_residue() - - -def test_second_run_cancellation_drains_cleanup_close_before_handoff(monkeypatch): - episode_id = "0" * 24 - policy_started = None - cleanup_close_started = None - close_live = False - close_observations = [] - - class RunClient: - def __init__(self, **_kwargs): - pass - - async def __aenter__(self): - return self - - async def __aexit__(self, _exc_type, _exc, _traceback): - return False - - async def create(self, _task_id, _tier): - return {"episode_id": episode_id, "prompt": "task"} - - async def close(self, value): - assert value == episode_id - nonlocal close_live - close_observations.append(close_live) - if len(close_observations) == 1: - close_live = True - cleanup_close_started.set() - try: - await asyncio.Event().wait() - finally: - close_live = False - assert not close_live - return {} - - async def blocked_policy(*_args, **_kwargs): - policy_started.set() - await asyncio.Event().wait() - - async def scenario(): - nonlocal policy_started, cleanup_close_started - policy_started = asyncio.Event() - cleanup_close_started = asyncio.Event() - run_task = asyncio.create_task( - agent.run("http://127.0.0.1:8000/v1", None, metadata=_run_metadata()) - ) - await policy_started.wait() - run_task.cancel() - await cleanup_close_started.wait() - run_task.cancel() - with pytest.raises(asyncio.CancelledError): - await run_task - assert not close_live - assert agent._EPISODE_PHASES[episode_id] == "cleanup_pending" - await agent.abort() - - _install_run_openai(monkeypatch) - monkeypatch.setattr(agent, "EpisodeClient", RunClient) - monkeypatch.setattr(agent, "_drive_policy", blocked_policy) - asyncio.run(scenario()) - assert close_observations == [False, False] - assert episode_id not in agent._EPISODE_PHASES - - -def test_abort_drains_pending_policy_before_close(monkeypatch): - episode_id = "d" * 24 - events = [] - - class AbortClient: - def __init__(self, *, total_timeout_seconds): - assert total_timeout_seconds == 180.0 - - async def __aenter__(self): - return self - - async def __aexit__(self, _exc_type, _exc, _traceback): - return False - - async def close(self, value): - assert value == episode_id - assert policy_task.done() - events.append("close") - return {} - - async def scenario(): - started = asyncio.Event() - - async def pending_policy(): - events.append("policy_started") - started.set() - try: - await asyncio.Event().wait() - finally: - events.append("policy_drained") - - nonlocal policy_task - policy_task = asyncio.create_task(pending_policy()) - agent._register_episode(episode_id, policy_task) - await started.wait() - await agent.abort() - assert policy_task.cancelled() - - policy_task = None - monkeypatch.setattr(agent, "EpisodeClient", AbortClient) - try: - asyncio.run(scenario()) - assert events == ["policy_started", "policy_drained", "close"] - assert episode_id not in agent._EPISODE_PHASES - finally: - agent._release_episode(episode_id) - - -def test_abort_close_retries_episode_conflict_without_releasing_ownership( - monkeypatch, -): - episode_id = "9" * 24 - phases = [] - - class ConflictClient: - async def close(self, value): - assert value == episode_id - phases.append(agent._EPISODE_PHASES.get(episode_id)) - if len(phases) < 3: - raise EpisodeAPIError(409, "episode_conflict", "still active") - return {} - - async def no_wait(_delay): - return None - - monkeypatch.setattr(agent.asyncio, "sleep", no_wait) - agent._register_episode(episode_id) - try: - assert agent._claim_driving_episodes_for_abort() == [episode_id] - assert asyncio.run( - agent._close_aborted_episode( - ConflictClient(), - episode_id, - retry_timeout_seconds=1.0, - ) - ) - assert phases == ["aborting", "aborting", "aborting"] - assert episode_id not in agent._EPISODE_PHASES - finally: - agent._release_episode(episode_id) - - -@pytest.mark.parametrize( - ("status", "code"), - [(409, "episode_conflict"), (500, "internal_error")], -) -def test_failed_abort_close_retains_retryable_cleanup_ownership(status, code): - episode_id = "8" * 24 - - class FailingClient: - async def close(self, value): - assert value == episode_id - raise EpisodeAPIError(status, code, "not closed") - - agent._register_episode(episode_id) - try: - assert agent._claim_driving_episodes_for_abort() == [episode_id] - assert not asyncio.run( - agent._close_aborted_episode( - FailingClient(), - episode_id, - retry_timeout_seconds=0.0, - ) - ) - assert agent._EPISODE_PHASES[episode_id] == "aborting" - assert agent._claim_episodes_and_tasks_for_abort() == [] - agent._mark_episode_cleanup_pending(episode_id) - assert agent._EPISODE_PHASES[episode_id] == "cleanup_pending" - assert agent._claim_episodes_and_tasks_for_abort() == [(episode_id, None)] - assert agent._EPISODE_PHASES[episode_id] == "aborting" - finally: - agent._release_episode(episode_id) - - -def test_failed_close_cannot_be_reclaimed_before_owner_finishes(monkeypatch): - episode_id = "4" * 24 - - class FailingClient: - async def close(self, value): - assert value == episode_id - raise EpisodeAPIError(500, "internal_error", "not closed") - - class UnexpectedAbortClient: - def __init__(self, **_kwargs): - raise AssertionError("overlapping abort reclaimed live cleanup ownership") - - async def scenario(): - assert not await agent._close_aborted_episode( - FailingClient(), - episode_id, - retry_timeout_seconds=0.0, - ) - assert agent._EPISODE_PHASES[episode_id] == "aborting" - await agent.abort() - assert agent._EPISODE_PHASES[episode_id] == "aborting" - agent._mark_episode_cleanup_pending(episode_id) - - monkeypatch.setattr(agent, "EpisodeClient", UnexpectedAbortClient) - agent._register_episode(episode_id) - try: - assert agent._claim_driving_episodes_for_abort() == [episode_id] - asyncio.run(scenario()) - assert agent._EPISODE_PHASES[episode_id] == "cleanup_pending" - finally: - agent._release_episode(episode_id) - - -def test_timed_out_policy_drain_retains_live_owner_until_abort_returns(): - episode_id = "a" * 23 + "0" - - async def scenario(): - release = asyncio.Event() - - async def slow_cancellation(): - try: - await asyncio.Event().wait() - except asyncio.CancelledError: - await release.wait() - - task = asyncio.create_task(slow_cancellation()) - agent._register_episode(episode_id, task) - await asyncio.sleep(0) - assert agent._claim_episodes_and_tasks_for_abort() == [(episode_id, task)] - assert not await agent._drain_aborted_policy_task(task, timeout_seconds=0.0) - assert agent._EPISODE_PHASES[episode_id] == "aborting" - assert agent._claim_episodes_and_tasks_for_abort() == [] - release.set() - await task - - try: - asyncio.run(scenario()) - finally: - agent._release_episode(episode_id) - - -def test_abort_without_claimable_episodes_does_not_initialize_client(monkeypatch): - class UnexpectedClient: - def __init__(self, **_kwargs): - raise AssertionError("abort client initialized without cleanup work") - - monkeypatch.setattr(agent, "EpisodeClient", UnexpectedClient) - asyncio.run(agent.abort()) - - -def test_cancelled_abort_retains_cleanup_pending_ownership(monkeypatch): - episode_id = "7" * 24 - - class BlockingClient: - def __init__(self, *, total_timeout_seconds): - assert total_timeout_seconds == 180.0 - - async def __aenter__(self): - return self - - async def __aexit__(self, _exc_type, _exc, _traceback): - return False - - async def close(self, value): - assert value == episode_id - close_started.set() - await asyncio.Event().wait() - - async def scenario(): - nonlocal close_started - close_started = asyncio.Event() - abort_task = asyncio.create_task(agent.abort()) - await close_started.wait() - abort_task.cancel() - with pytest.raises(asyncio.CancelledError): - await abort_task - - close_started = None - monkeypatch.setattr(agent, "EpisodeClient", BlockingClient) - agent._register_episode(episode_id) - try: - asyncio.run(scenario()) - assert agent._EPISODE_PHASES[episode_id] == "cleanup_pending" - assert agent._claim_episodes_and_tasks_for_abort() == [(episode_id, None)] - finally: - agent._release_episode(episode_id) - - -def test_abort_client_initialization_failure_retains_cleanup_ownership(monkeypatch): - episode_id = "c" * 24 - - class FailingEpisodeClient: - def __init__(self, *, total_timeout_seconds): - assert total_timeout_seconds == 180.0 - - async def __aenter__(self): - raise RuntimeError("client unavailable") - - async def __aexit__(self, _exc_type, _exc, _traceback): - return False - - monkeypatch.setattr(agent, "EpisodeClient", FailingEpisodeClient) - agent._register_episode(episode_id) - try: - asyncio.run(agent.abort()) - assert not agent._claim_episode_finalization(episode_id) - assert agent._EPISODE_PHASES[episode_id] == "cleanup_pending" - assert agent._claim_episodes_and_tasks_for_abort() == [(episode_id, None)] - finally: - agent._release_episode(episode_id) - - -def _tool_call(identifier, name, arguments): - return { - "id": identifier, - "type": "function", - "function": {"name": name, "arguments": json.dumps(arguments)}, - } - - -def test_adapter_is_network_only_and_enforces_l2(): - prompt_surface = agent.SYSTEM_PROMPT + json.dumps(agent.TOOLS) - assert "DOCKER_HOST" not in prompt_surface - assert "DEBUG_URL" not in prompt_surface - assert "debug daemon" not in prompt_surface - assert agent._task_identity({"task_id": "CVE-2024-1234"}) == ( - "CVE-2024-1234", - "l2", - ) - assert agent._task_identity( - {"task_id": "CVE-2024-1234", "prompt_tier": "l2"} - ) == ("CVE-2024-1234", "l2") - for tier in ("l0", "l1"): - with pytest.raises(ValueError, match="must use prompt_tier l2"): - agent._task_identity( - {"task_id": "CVE-2024-1234", "prompt_tier": tier} - ) - - -def test_agent_preserves_full_assistant_message_and_uses_tool_roles(monkeypatch): - monkeypatch.setenv("SECRLENV_MAX_TURNS", "4") - policy = FakePolicy( - [ - { - "role": "assistant", - "content": "", - "reasoning_content": "inspect first", - "tool_calls": [ - _tool_call("call-1", "terminal.exec", {"command": "id"}) - ], - }, - { - "role": "assistant", - "content": "", - "reasoning_content": "submit evidence", - "tool_calls": [ - _tool_call( - "call-2", "submit", {"evidence": "target returned nonce"} - ) - ], - }, - ] - ) - client = FakeEpisodeClient() - metrics = agent.AgentMetrics() - status = asyncio.run( - agent._drive_policy( - policy, - client, - {"episode_id": "a" * 24, "prompt": "solve this"}, - {"temperature": 0.7, "stream": True}, - metrics, - ) - ) - assert status == "completed" - assert client.exec_calls[0][1] == "id" - assert client.submissions == [ - ("a" * 24, {"evidence": "target returned nonce"}) - ] - second_messages = policy.chat.completions.requests[1]["messages"] - first_assistant = second_messages[2] - assert first_assistant["reasoning_content"] == "inspect first" - assert first_assistant["tool_calls"][0]["id"] == "call-1" - assert second_messages[3] == { - "role": "tool", - "tool_call_id": "call-1", - "content": json.dumps( - { - "exit_code": 0, - "output": "command output", - "timed_out": False, - "truncated": False, - }, - separators=(",", ":"), - ), - } - assert "stream" not in policy.chat.completions.requests[0]["extra_body"] - assert metrics.terminal_calls == 1 - assert metrics.submit_calls == 1 - - -def test_agent_retries_non_tool_prose_as_user_turn(monkeypatch): - monkeypatch.setenv("SECRLENV_MAX_TURNS", "3") - monkeypatch.setenv("SECRLENV_MAX_NO_TOOL_RETRIES", "2") - policy = FakePolicy( - [ - {"role": "assistant", "content": "I would run id"}, - {"role": "assistant", "content": "Still no tool"}, - ] - ) - metrics = agent.AgentMetrics() - status = asyncio.run( - agent._drive_policy( - policy, - FakeEpisodeClient(), - {"episode_id": "b" * 24, "prompt": "solve"}, - {}, - metrics, - ) - ) - assert status == "max_turns" - assert policy.chat.completions.requests[1]["messages"][-1]["role"] == "user" - assert metrics.parse_failures == 2 - - -def test_signed_zero_reward_is_valid_and_tampering_is_rejected(monkeypatch): - key = "k" * 48 - monkeypatch.setenv("SECRLENV_REWARD_HMAC_KEY", key) - outcome = { - "schema": 1, - "status": "completed", - "episode_id": "c" * 24, - "task_id": "CVE-2024-1234", - "reward": 0.0, - "passed": False, - "class": "rce", - reward.INFRASTRUCTURE_503_RETRIES_KEY: 0, - } - metadata = { - reward.OUTCOME_KEY: outcome, - reward.MAC_KEY: reward.sign_outcome(outcome), - } - assert reward.verify_outcome(metadata) == 0.0 - sample = SimpleNamespace(metadata=metadata) - assert asyncio.run(reward.reward_func(None, sample)) == 0.0 - - forged = copy.deepcopy(metadata) - forged[reward.OUTCOME_KEY]["reward"] = 1.0 - with pytest.raises(reward.UntrustedOutcome, match="signature mismatch"): - reward.verify_outcome(forged) - - -def test_sequence_cap_outcome_contract_and_unknown_status(monkeypatch): - key = "s" * 48 - monkeypatch.setenv("SECRLENV_REWARD_HMAC_KEY", key) - task_id = "CVE-2024-1234" - episode_id = "m" * 24 - outcome = agent._validated_outcome( - { - "task_id": task_id, - "episode_id": episode_id, - "reward": 0.25, - "passed": False, - "grader": {"class": "rce"}, - }, - task_id, - episode_id, - "max_seq_len", - ) - metadata = { - reward.OUTCOME_KEY: outcome, - reward.MAC_KEY: reward.sign_outcome(outcome), - } - - assert reward.verify_outcome(metadata) == 0.25 - - unknown = copy.deepcopy(outcome) - unknown["status"] = "unknown_termination" - unknown_metadata = { - reward.OUTCOME_KEY: unknown, - reward.MAC_KEY: reward.sign_outcome(unknown), - } - with pytest.raises(reward.UntrustedOutcome, match="invalid schema or status"): - reward.verify_outcome(unknown_metadata) - - -def test_missing_reward_never_defaults_to_zero(monkeypatch): - monkeypatch.setenv("SECRLENV_REWARD_HMAC_KEY", "z" * 48) - sample = SimpleNamespace(metadata={"reward": 0.0}) - with pytest.raises(reward.UntrustedOutcome, match="no signed"): - asyncio.run(reward.reward_func(None, sample)) - - -def _install_filter_types(monkeypatch): - class DynamicFilterOutput: - def __init__(self, *, keep, reason): - self.keep = keep - self.reason = reason - - class Sample: - class Status: - ABORTED = "aborted" - PENDING = "pending" - COMPLETED = "completed" - TRUNCATED = "truncated" - FAILED = "failed" - - miles = types.ModuleType("miles") - rollout = types.ModuleType("miles.rollout") - filter_hub = types.ModuleType("miles.rollout.filter_hub") - base_types = types.ModuleType("miles.rollout.filter_hub.base_types") - utils = types.ModuleType("miles.utils") - sample_types = types.ModuleType("miles.utils.types") - base_types.DynamicFilterOutput = DynamicFilterOutput - sample_types.Sample = Sample - monkeypatch.setitem(sys.modules, "miles", miles) - monkeypatch.setitem(sys.modules, "miles.rollout", rollout) - monkeypatch.setitem(sys.modules, "miles.rollout.filter_hub", filter_hub) - monkeypatch.setitem( - sys.modules, "miles.rollout.filter_hub.base_types", base_types - ) - monkeypatch.setitem(sys.modules, "miles.utils", utils) - monkeypatch.setitem(sys.modules, "miles.utils.types", sample_types) - - -def _signed_sample( - index, value, *, status="completed", outcome_status="completed" -): - outcome = { - "schema": 1, - "status": outcome_status, - "episode_id": f"episode-{index:024d}", - "task_id": "CVE-2019-7859", - "reward": value, - reward.INFRASTRUCTURE_503_RETRIES_KEY: 0, - } - return SimpleNamespace( - index=index, - status=status, - metadata={ - reward.OUTCOME_KEY: outcome, - reward.MAC_KEY: reward.sign_outcome(outcome), - }, - ) - - -def test_signed_infrastructure_outcome_is_only_an_abort_signal(monkeypatch): - monkeypatch.setenv("SECRLENV_REWARD_HMAC_KEY", "i" * 48) - _install_filter_types(monkeypatch) - outcome = agent._infrastructure_outcome( - "CVE-2024-1234", "f" * 24 - ) - metadata = { - reward.OUTCOME_KEY: outcome, - reward.MAC_KEY: reward.sign_outcome(outcome), - } - sample = SimpleNamespace(index=30, status="completed", metadata=metadata) - - assert asyncio.run(reward.reward_func(None, sample)) == 0.0 - assert sample.status == "aborted" - with pytest.raises( - reward.UntrustedOutcome, match="not a grader verdict" - ): - reward.verify_outcome(metadata) - - result = reward.check_group( - SimpleNamespace( - yeto_rl_policy_version=10, - yeto_rl_dynamic_sampling_max_replacements=0, - _yeto_secrlenv_retry_callback=lambda *_args, **_kwargs: None, - ), - [sample], - ) - assert (result.keep, result.reason) == ( - False, - "secrlenv_infrastructure_failure", - ) - - direct_sample = SimpleNamespace( - index=31, status="completed", metadata=copy.deepcopy(metadata) - ) - direct = reward.check_group( - SimpleNamespace( - yeto_rl_policy_version=10, - yeto_rl_dynamic_sampling_max_replacements=0, - _yeto_secrlenv_retry_callback=lambda *_args, **_kwargs: None, - ), - [direct_sample], - ) - assert (direct.keep, direct.reason) == ( - False, - "secrlenv_infrastructure_failure", - ) - - forged = copy.deepcopy(metadata) - forged[reward.OUTCOME_KEY]["task_id"] = "CVE-2024-9999" - forged_sample = SimpleNamespace( - index=32, status="completed", metadata=forged - ) - with pytest.raises(reward.UntrustedOutcome, match="signature mismatch"): - asyncio.run(reward.reward_func(None, forged_sample)) - assert forged_sample.status == "completed" - - -def test_admission_outcome_schema_is_strict_and_nonce_distinguishes_attempts( - monkeypatch, -): - monkeypatch.setenv("SECRLENV_REWARD_HMAC_KEY", "n" * 48) - first = agent._admission_infrastructure_outcome( - "CVE-2024-1234", 1, nonce="1" * 32 - ) - second = agent._admission_infrastructure_outcome( - "CVE-2024-1234", 1, nonce="2" * 32 - ) - first_metadata = { - reward.OUTCOME_KEY: first, - reward.MAC_KEY: reward.sign_outcome(first), - } - second_metadata = { - reward.OUTCOME_KEY: second, - reward.MAC_KEY: reward.sign_outcome(second), - } - - assert reward._verified_outcome(first_metadata)[1] == 0.0 - assert reward._group_key( - [SimpleNamespace(index=9, metadata=first_metadata)] - ) != reward._group_key([SimpleNamespace(index=9, metadata=second_metadata)]) - - for field, invalid in ( - (reward.ADMISSION_PHASE_KEY, "post_create"), - (reward.ADMISSION_ERROR_CODE_KEY, "capacity_reached"), - (reward.ADMISSION_NONCE_KEY, "bad"), - (reward.INFRASTRUCTURE_503_RETRIES_KEY, 0), - ("episode_id", "not-null"), - ): - forged = copy.deepcopy(first) - forged[field] = invalid - metadata = { - reward.OUTCOME_KEY: forged, - reward.MAC_KEY: reward.sign_outcome(forged), - } - with pytest.raises(reward.UntrustedOutcome, match="admission outcome"): - reward._verified_outcome(metadata) - - extra = copy.deepcopy(first) - extra["unexpected"] = True - with pytest.raises(reward.UntrustedOutcome, match="admission outcome"): - reward._verified_outcome( - { - reward.OUTCOME_KEY: extra, - reward.MAC_KEY: reward.sign_outcome(extra), - } - ) - - -def _install_generate_upstream(monkeypatch, upstream): - package = types.ModuleType("miles") - rollout = types.ModuleType("miles.rollout") - hub = types.ModuleType("miles.rollout.generate_hub") - upstream_module = types.ModuleType( - "miles.rollout.generate_hub.agentic_tool_call" - ) - upstream_module.generate = upstream - monkeypatch.setitem(sys.modules, "miles", package) - monkeypatch.setitem(sys.modules, "miles.rollout", rollout) - monkeypatch.setitem(sys.modules, "miles.rollout.generate_hub", hub) - monkeypatch.setitem( - sys.modules, "miles.rollout.generate_hub.agentic_tool_call", upstream_module - ) - - -def test_generate_wrapper_preserves_only_verified_metadata_on_aborted_fallback( - monkeypatch, -): - monkeypatch.setenv("SECRLENV_REWARD_HMAC_KEY", "g" * 48) - sample = SimpleNamespace(status=SimpleNamespace(value="aborted"), metadata={}) - outcome = agent._admission_infrastructure_outcome( - "CVE-2024-1234", 1, nonce="3" * 32 - ) - signed = { - reward.OUTCOME_KEY: outcome, - reward.MAC_KEY: reward.sign_outcome(outcome), - "exit_status": reward.INFRASTRUCTURE_STATUS, - "agent_metrics": {}, - } - - async def upstream(_input): - secrlenv_generate.capture_agent_metadata(signed) - return SimpleNamespace(samples=sample) - - _install_generate_upstream(monkeypatch, upstream) - output = asyncio.run(secrlenv_generate.generate(SimpleNamespace())) - - assert output.samples is sample - assert sample.status.value == "aborted" - assert reward._verified_outcome(sample.metadata)[0] == outcome - assert set(sample.metadata) == {reward.OUTCOME_KEY, reward.MAC_KEY} - - -def test_generate_wrapper_rejects_forged_or_missing_fallback_metadata(monkeypatch): - monkeypatch.setenv("SECRLENV_REWARD_HMAC_KEY", "g" * 48) - captured = {"value": None} - sample_metadata = {"value": {}} - - async def upstream(_input): - if captured["value"] is not None: - secrlenv_generate.capture_agent_metadata(captured["value"]) - return SimpleNamespace( - samples=SimpleNamespace( - status=SimpleNamespace(value="aborted"), - metadata=copy.deepcopy(sample_metadata["value"]), - ) - ) - - _install_generate_upstream(monkeypatch, upstream) - with pytest.raises(reward.UntrustedOutcome): - asyncio.run(secrlenv_generate.generate(SimpleNamespace())) - - outcome = agent._admission_infrastructure_outcome( - "CVE-2024-1234", 1, nonce="4" * 32 - ) - captured["value"] = { - reward.OUTCOME_KEY: outcome, - reward.MAC_KEY: "0" * 64, - } - with pytest.raises(reward.UntrustedOutcome, match="signature mismatch"): - asyncio.run(secrlenv_generate.generate(SimpleNamespace())) - - sample_metadata["value"] = {reward.OUTCOME_KEY: outcome} - with pytest.raises(reward.UntrustedOutcome, match="lost signed outcome metadata"): - asyncio.run(secrlenv_generate.generate(SimpleNamespace())) - - -def test_admission_replacement_nonce_preserves_shared_retry_exhaustion(monkeypatch): - monkeypatch.setenv("SECRLENV_REWARD_HMAC_KEY", "r" * 48) - _install_filter_types(monkeypatch) - calls = [] - - def retry(group, **kwargs): - assert kwargs["infrastructure_503_retries"] == {201: 1} - if calls: - raise secrlenv_reward.SecRLEnvReplacementExhausted( - "SecRLEnv rollout replacement budget was exhausted" - ) - calls.append((group, kwargs)) - - def admission_sample(nonce): - outcome = agent._admission_infrastructure_outcome( - "CVE-2024-1234", 1, nonce=nonce - ) - return SimpleNamespace( - index=201, - status="aborted", - metadata={ - reward.OUTCOME_KEY: outcome, - reward.MAC_KEY: reward.sign_outcome(outcome), - }, - ) - - args = SimpleNamespace( - yeto_rl_policy_version=92, - yeto_rl_dynamic_sampling_max_replacements=0, - _yeto_secrlenv_retry_callback=retry, - ) - first = [admission_sample("5" * 32)] - exhausted = [admission_sample("6" * 32)] - - initial = reward.check_group(args, first) - with pytest.raises( - secrlenv_reward.SecRLEnvReplacementExhausted, - match="budget was exhausted", - ): - reward.check_group(args, exhausted) - - assert (initial.keep, initial.reason) == ( - False, - "secrlenv_infrastructure_failure", - ) - assert len(calls) == 1 - - -def test_signed_cleanup_failure_is_terminal_without_replacement(monkeypatch): - monkeypatch.setenv("SECRLENV_REWARD_HMAC_KEY", "c" * 48) - _install_filter_types(monkeypatch) - outcome = agent._cleanup_error_outcome("CVE-2024-1234", "e" * 24) - metadata = { - reward.OUTCOME_KEY: outcome, - reward.MAC_KEY: reward.sign_outcome(outcome), - } - sample = SimpleNamespace(index=33, status="completed", metadata=metadata) - replacements = [] - - assert asyncio.run(reward.reward_func(None, sample)) == 0.0 - with pytest.raises( - secrlenv_reward.SecRLEnvRolloutCleanupError, - match="cleanup could not be verified", - ): - reward.check_group( - SimpleNamespace( - yeto_rl_policy_version=10, - yeto_rl_dynamic_sampling_max_replacements=0, - _yeto_secrlenv_retry_callback=lambda *_args, **_kwargs: ( - replacements.append(True) - ), - ), - [sample], - ) - - assert replacements == [] - - -def test_signed_group_filter_accepts_valid_reward_variance(monkeypatch): - monkeypatch.setenv("SECRLENV_REWARD_HMAC_KEY", "v" * 48) - _install_filter_types(monkeypatch) - args = SimpleNamespace( - yeto_rl_policy_version=7, - yeto_rl_dynamic_sampling_max_replacements=2, - ) - - result = reward.check_group( - args, [_signed_sample(1, 0.0), _signed_sample(2, 1.0)] - ) - - assert result.keep is True - assert result.reason is None - assert args._yeto_secrlenv_filter_state["rejections"] == 0 - - -def test_signed_group_filter_accepts_completed_and_truncated_statuses(monkeypatch): - monkeypatch.setenv("SECRLENV_REWARD_HMAC_KEY", "t" * 48) - _install_filter_types(monkeypatch) - args = SimpleNamespace( - yeto_rl_policy_version=71, - yeto_rl_dynamic_sampling_max_replacements=0, - ) - - result = reward.check_group( - args, - [ - _signed_sample(101, 0.0, status="completed"), - _signed_sample(102, 1.0, status="truncated"), - ], - ) - - assert (result.keep, result.reason) == (True, None) - - -def test_signed_zero_variance_groups_have_a_memoized_replacement_bound(monkeypatch): - monkeypatch.setenv("SECRLENV_REWARD_HMAC_KEY", "b" * 48) - _install_filter_types(monkeypatch) - args = SimpleNamespace( - yeto_rl_policy_version=8, - yeto_rl_dynamic_sampling_max_replacements=2, - ) - first = [_signed_sample(10, 0.0), _signed_sample(11, 0.0)] - second = [_signed_sample(12, 1.0), _signed_sample(13, 1.0)] - fallback = [_signed_sample(14, 0.0), _signed_sample(15, 0.0)] - - decisions = [ - reward.check_group(args, first), - reward.check_group(args, second), - reward.check_group(args, fallback), - reward.check_group(args, fallback), - ] - - assert [decision.keep for decision in decisions] == [False, False, True, True] - assert decisions[2].reason == "secrlenv_bounded_fallback_after_2_replacements" - assert args._yeto_secrlenv_filter_state["rejections"] == 2 - assert args._yeto_secrlenv_filter_state["forced"] == 1 - - -def test_aborted_and_untrusted_groups_never_use_bounded_fallback(monkeypatch): - monkeypatch.setenv("SECRLENV_REWARD_HMAC_KEY", "u" * 48) - _install_filter_types(monkeypatch) - args = SimpleNamespace( - yeto_rl_policy_version=9, - yeto_rl_dynamic_sampling_max_replacements=0, - _yeto_secrlenv_retry_callback=lambda *_args, **_kwargs: None, - ) - aborted = [_signed_sample(20, 0.0, status="aborted")] - forged = _signed_sample(21, 0.0) - forged.metadata[reward.OUTCOME_KEY]["reward"] = 1.0 - - aborted_result = reward.check_group(args, aborted) - with pytest.raises( - secrlenv_reward.SecRLEnvUntrustedEvidence, - match="authentication failed", - ): - reward.check_group(args, [forged]) - - assert (aborted_result.keep, aborted_result.reason) == ( - False, - "secrlenv_aborted", - ) - assert aborted[0].status == "aborted" - assert args._yeto_secrlenv_filter_state["nontrainable_rejections"] == 1 - - -def test_signed_infrastructure_replacement_is_memoized_and_exhausts(monkeypatch): - monkeypatch.setenv("SECRLENV_REWARD_HMAC_KEY", "r" * 48) - _install_filter_types(monkeypatch) - calls = [] - - def retry(group, **kwargs): - if calls: - raise secrlenv_reward.SecRLEnvReplacementExhausted( - "SecRLEnv rollout replacement budget was exhausted" - ) - calls.append((group, kwargs)) - - args = SimpleNamespace( - yeto_rl_policy_version=91, - yeto_rl_dynamic_sampling_max_replacements=0, - _yeto_secrlenv_retry_callback=retry, - ) - first = [ - _signed_sample( - 201, - 0.0, - status="aborted", - outcome_status=reward.INFRASTRUCTURE_STATUS, - ) - ] - second = [ - _signed_sample( - 202, - 0.0, - status="aborted", - outcome_status=reward.INFRASTRUCTURE_STATUS, - ) - ] - - initial = reward.check_group(args, first) - duplicate = reward.check_group(args, first) - with pytest.raises( - secrlenv_reward.SecRLEnvReplacementExhausted, - match="budget was exhausted", - ): - reward.check_group(args, second) - - assert (initial.keep, duplicate.keep) == (False, False) - assert len(calls) == 1 - assert first[0].status == "aborted" - - -def test_evaluation_identity_and_range_are_checked(): - with pytest.raises(EpisodeClientError, match="identity"): - agent._validated_outcome( - { - "task_id": "CVE-2024-9999", - "episode_id": "d" * 24, - "reward": 0.5, - "passed": False, - }, - "CVE-2024-1234", - "d" * 24, - "completed", - ) - with pytest.raises(EpisodeClientError, match="invalid verdict"): - agent._validated_outcome( - { - "task_id": "CVE-2024-1234", - "episode_id": "d" * 24, - "reward": 2.0, - "passed": False, - }, - "CVE-2024-1234", - "d" * 24, - "completed", - ) - - -def test_client_refuses_non_loopback_or_credentialed_origins(): - token = "t" * 48 - for value in ( - "http://example.com:8765", - "https://127.0.0.1:8765", - "http://user:pass@127.0.0.1:8765", - "http://127.0.0.1:8765/path", - ): - with pytest.raises(EpisodeClientError): - EpisodeClient(value, token) - - -def test_daemon_readiness_fails_fast_without_a_listener(monkeypatch): - monkeypatch.setenv("SECRLENV_DAEMON_URL", "http://127.0.0.1:8765") - - def unavailable(*_args, **_kwargs): - raise OSError - - monkeypatch.setattr("yeto_miles_secrlenv.client.urlopen", unavailable) - with pytest.raises(EpisodeTransportError, match="not healthy"): - require_daemon_ready(timeout_seconds=0.1) - - -def test_daemon_readiness_attests_health_and_task_pack(monkeypatch): - expected = "a" * 64 - monkeypatch.setenv("SECRLENV_DAEMON_URL", "http://127.0.0.1:8765") - monkeypatch.setenv("SECRLENV_TASK_PACK_SHA256", expected) - - class Response: - status = 200 - - def __enter__(self): - return self - - def __exit__(self, *_args): - return None - - def read(self): - return BytesIO( - json.dumps({"ok": True, "task_pack_sha256": expected}).encode() - ).read() - - monkeypatch.setattr( - "yeto_miles_secrlenv.client.urlopen", lambda *_a, **_k: Response() - ) - require_daemon_ready(timeout_seconds=0.1) - - monkeypatch.setenv("SECRLENV_TASK_PACK_SHA256", "b" * 64) - with pytest.raises(EpisodeTransportError, match="task-pack identity"): - require_daemon_ready(timeout_seconds=0.1) - - -def test_create_retry_default_covers_sixty_rollouts_at_capacity_sixteen( - monkeypatch, -): - monkeypatch.delenv("SECRLENV_CAPACITY_MAX_WAIT_SECONDS", raising=False) - observed = [] - - def positive_env(name, default): - observed.append((name, default)) - return 0.001 - - class UnavailableClient: - calls = 0 - - async def create(self, *_args): - self.calls += 1 - raise EpisodeTransportError("offline") - - client = UnavailableClient() - monkeypatch.setattr(agent, "_positive_env", positive_env) - with pytest.raises(EpisodeTransportError, match="offline"): - asyncio.run( - agent._create_with_capacity_retry( - client, "CVE-2024-1234", "l2" - ) - ) - assert client.calls == 1 - assert observed == [("SECRLENV_CAPACITY_MAX_WAIT_SECONDS", 14400.0)] - rollout_count = 60 - max_active = 16 - rollout_timeout_seconds = 3600.0 - admission_waves = (rollout_count + max_active - 1) // max_active - assert observed[0][1] >= admission_waves * rollout_timeout_seconds - - -def test_create_retries_capacity_then_one_infrastructure_error_same_identity( - monkeypatch, -): - monkeypatch.setenv("SECRLENV_CAPACITY_MAX_WAIT_SECONDS", "30") - expected = {"episode_id": "a" * 24} - responses = [ - EpisodeAPIError(503, "capacity_reached", "busy"), - EpisodeAPIError(503, "infrastructure_error", "provisioning failed"), - expected, - ] - calls = [] - delays = [] - - class RecoveringClient: - async def create(self, task_id, tier): - calls.append((task_id, tier)) - response = responses.pop(0) - if isinstance(response, Exception): - raise response - return response - - async def no_wait(delay): - delays.append(delay) - - monkeypatch.setattr(agent.asyncio, "sleep", no_wait) - monkeypatch.setattr(agent.random, "uniform", lambda *_args: 0.5) - result = asyncio.run( - agent._create_with_capacity_retry( - RecoveringClient(), "CVE-2024-1234", "l2" - ) - ) - assert result == expected - assert calls == [("CVE-2024-1234", "l2")] * 3 - assert delays == [0.5, 0.5] - - -def test_capacity_retry_stops_at_the_configured_deadline(monkeypatch): - monkeypatch.setenv("SECRLENV_CAPACITY_MAX_WAIT_SECONDS", "0.02") - calls = 0 - - class BusyClient: - async def create(self, *_args): - nonlocal calls - calls += 1 - raise EpisodeAPIError(503, "capacity_reached", "busy") - - monkeypatch.setattr(agent.random, "uniform", lambda *_args: 0.005) - with pytest.raises(EpisodeAPIError) as caught: - asyncio.run( - agent._create_with_capacity_retry( - BusyClient(), "CVE-2024-1234", "l2" - ) - ) - assert caught.value.code == "capacity_reached" - assert 2 <= calls < 20 - - -def test_capacity_retry_accepts_success_exactly_at_deadline(monkeypatch): - monkeypatch.setenv("SECRLENV_CAPACITY_MAX_WAIT_SECONDS", "1") - clock = [0.0] - failure = EpisodeAPIError(503, "capacity_reached", "busy") - calls = [] - - class DeadlineClient: - async def create(self, *_args): - calls.append(clock[0]) - if len(calls) == 1: - raise failure - clock[0] = 1.0 - return {"episode_id": "a" * 24} - - async def advance(delay): - clock[0] += delay - - monkeypatch.setattr( - agent, "time", SimpleNamespace(monotonic=lambda: clock[0]) - ) - monkeypatch.setattr(agent.asyncio, "sleep", advance) - monkeypatch.setattr(agent.random, "uniform", lambda *_args: 0.1) - result = asyncio.run( - agent._create_with_capacity_retry( - DeadlineClient(), "CVE-2024-1234", "l2" - ) - ) - assert result == {"episode_id": "a" * 24} - assert calls == [0.0, 0.1] - - -def test_infrastructure_retry_accepts_late_create_instead_of_stale_503( - monkeypatch, -): - monkeypatch.setenv("SECRLENV_CAPACITY_MAX_WAIT_SECONDS", "1") - clock = [0.0] - calls = [] - ledger = {reward.INFRASTRUCTURE_503_RETRIES_KEY: 0} - - class DeadlineClient: - async def create(self, *_args): - calls.append(clock[0]) - if len(calls) == 1: - raise EpisodeAPIError( - 503, "infrastructure_error", "unavailable" - ) - clock[0] = 1.0 - return {"episode_id": "b" * 24} - - async def advance(delay): - clock[0] += delay - - monkeypatch.setattr( - agent, "time", SimpleNamespace(monotonic=lambda: clock[0]) - ) - monkeypatch.setattr(agent.asyncio, "sleep", advance) - monkeypatch.setattr(agent.random, "uniform", lambda *_args: 0.1) - - result = asyncio.run( - agent._create_with_capacity_retry( - DeadlineClient(), - "CVE-2024-1234", - "l2", - retry_ledger=ledger, - ) - ) - - assert result == {"episode_id": "b" * 24} - assert calls == [0.0, 0.1] - assert ledger == {reward.INFRASTRUCTURE_503_RETRIES_KEY: 1} - - -def test_capacity_retry_does_not_create_at_the_deadline(monkeypatch): - monkeypatch.setenv("SECRLENV_CAPACITY_MAX_WAIT_SECONDS", "1") - clock = [0.0] - failure = EpisodeAPIError(503, "capacity_reached", "busy") - calls = 0 - - class DeadlineClient: - async def create(self, *_args): - nonlocal calls - calls += 1 - if calls > 1: - raise AssertionError("a retry began at or after its deadline") - raise failure - - async def advance(delay): - clock[0] += delay - - monkeypatch.setattr( - agent, "time", SimpleNamespace(monotonic=lambda: clock[0]) - ) - monkeypatch.setattr(agent.asyncio, "sleep", advance) - monkeypatch.setattr(agent.random, "uniform", lambda *_args: 3.0) - with pytest.raises(EpisodeAPIError) as caught: - asyncio.run( - agent._create_with_capacity_retry( - DeadlineClient(), "CVE-2024-1234", "l2" - ) - ) - assert caught.value is failure - assert calls == 1 - - -def test_capacity_retry_bounds_a_blocked_retry_by_remaining_time(monkeypatch): - monkeypatch.setenv("SECRLENV_CAPACITY_MAX_WAIT_SECONDS", "0.05") - failure = EpisodeAPIError(503, "capacity_reached", "busy") - calls = 0 - retry_cancelled = False - - class BlockingClient: - async def create(self, *_args): - nonlocal calls, retry_cancelled - calls += 1 - if calls == 1: - raise failure - try: - await asyncio.Event().wait() - except asyncio.CancelledError: - retry_cancelled = True - raise - - async def no_wait(_delay): - return None - - monkeypatch.setattr(agent.asyncio, "sleep", no_wait) - with pytest.raises(EpisodeAPIError) as caught: - asyncio.run( - agent._create_with_capacity_retry( - BlockingClient(), "CVE-2024-1234", "l2" - ) - ) - assert caught.value is failure - assert calls == 2 - assert retry_cancelled is True - - -def test_capacity_retry_preserves_external_cancellation(monkeypatch): - monkeypatch.setenv("SECRLENV_CAPACITY_MAX_WAIT_SECONDS", "30") - retry_started = asyncio.Event() - retry_cancelled = False - calls = 0 - - class BlockingClient: - async def create(self, *_args): - nonlocal calls, retry_cancelled - calls += 1 - if calls == 1: - raise EpisodeAPIError(503, "capacity_reached", "busy") - retry_started.set() - try: - await asyncio.Event().wait() - except asyncio.CancelledError: - retry_cancelled = True - raise - - async def no_wait(_delay): - return None - - async def drive_and_cancel(): - task = asyncio.create_task( - agent._create_with_capacity_retry( - BlockingClient(), "CVE-2024-1234", "l2" - ) - ) - await retry_started.wait() - task.cancel() - with pytest.raises(asyncio.CancelledError): - await task - - monkeypatch.setattr(agent.asyncio, "sleep", no_wait) - asyncio.run(drive_and_cancel()) - assert calls == 2 - assert retry_cancelled is True - - -def test_infrastructure_error_is_retried_at_most_once(monkeypatch): - monkeypatch.setenv("SECRLENV_CAPACITY_MAX_WAIT_SECONDS", "30") - calls = 0 - delays = [] - - class BrokenClient: - async def create(self, *_args): - nonlocal calls - calls += 1 - raise EpisodeAPIError( - 503, "infrastructure_error", "provisioning failed" - ) - - async def no_wait(delay): - delays.append(delay) - - monkeypatch.setattr(agent.asyncio, "sleep", no_wait) - monkeypatch.setattr(agent.random, "uniform", lambda *_args: 0.5) - ledger = {reward.INFRASTRUCTURE_503_RETRIES_KEY: 0} - with pytest.raises(EpisodeAPIError) as caught: - asyncio.run( - agent._create_with_capacity_retry( - BrokenClient(), - "CVE-2024-1234", - "l2", - retry_ledger=ledger, - ) - ) - assert caught.value.code == "infrastructure_error" - assert calls == 2 - assert delays == [0.5] - assert ledger == {reward.INFRASTRUCTURE_503_RETRIES_KEY: 1} - - -def test_same_task_retry_shares_infrastructure_503_attempt_ledger(monkeypatch): - monkeypatch.setenv("SECRLENV_CAPACITY_MAX_WAIT_SECONDS", "30") - calls = 0 - - class BrokenClient: - async def create(self, *_args): - nonlocal calls - calls += 1 - raise EpisodeAPIError( - 503, "infrastructure_error", "provisioning failed" - ) - - async def unexpected_wait(_delay): - raise AssertionError("a consumed infrastructure retry must not back off") - - monkeypatch.setattr(agent.asyncio, "sleep", unexpected_wait) - ledger = {reward.INFRASTRUCTURE_503_RETRIES_KEY: 1} - - with pytest.raises(EpisodeAPIError) as caught: - asyncio.run( - agent._create_with_capacity_retry( - BrokenClient(), - "CVE-2024-1234", - "l2", - retry_ledger=ledger, - ) - ) - - assert caught.value.code == "infrastructure_error" - assert calls == 1 - assert ledger == {reward.INFRASTRUCTURE_503_RETRIES_KEY: 1} - - -@pytest.mark.parametrize( - "failure", - [ - EpisodeAPIError(400, "invalid_request", "bad task"), - EpisodeAPIError(400, "capacity_reached", "wrong status"), - EpisodeAPIError(500, "internal_error", "unexpected failure"), - EpisodeTransportError("ambiguous transport failure"), - ], - ids=("invalid-request", "wrong-capacity-status", "internal-error", "transport"), -) -def test_create_does_not_retry_nonretryable_or_transport_errors( - monkeypatch, failure -): - monkeypatch.setenv("SECRLENV_CAPACITY_MAX_WAIT_SECONDS", "30") - calls = 0 - - class FailingClient: - async def create(self, *_args): - nonlocal calls - calls += 1 - raise failure - - async def unexpected_wait(_delay): - raise AssertionError("single-shot failures must not back off") - - monkeypatch.setattr(agent.asyncio, "sleep", unexpected_wait) - with pytest.raises(type(failure)) as caught: - asyncio.run( - agent._create_with_capacity_retry( - FailingClient(), "CVE-2024-1234", "l2" - ) - ) - assert caught.value is failure - assert calls == 1 - - -@pytest.mark.parametrize("value", ["0", "-1", "nan", "inf", "invalid"]) -def test_create_retry_rejects_invalid_capacity_deadline_before_create( - monkeypatch, value -): - monkeypatch.setenv("SECRLENV_CAPACITY_MAX_WAIT_SECONDS", value) - - class UnexpectedClient: - async def create(self, *_args): - raise AssertionError("invalid retry configuration must fail first") - - with pytest.raises(ValueError, match="SECRLENV_CAPACITY_MAX_WAIT_SECONDS"): - asyncio.run( - agent._create_with_capacity_retry( - UnexpectedClient(), "CVE-2024-1234", "l2" - ) - ) diff --git a/tests/test_secrlenv_codex_harness.py b/tests/test_secrlenv_codex_harness.py deleted file mode 100644 index c7b1136ba..000000000 --- a/tests/test_secrlenv_codex_harness.py +++ /dev/null @@ -1,1790 +0,0 @@ -from __future__ import annotations - -import asyncio -import copy -import errno -import hashlib -import json -import os -import shutil -import subprocess -import sys -from collections.abc import Iterable -from pathlib import Path -from types import SimpleNamespace -from typing import Any - -import aiohttp -import pytest -from aiohttp import web - - -from yeto.rl.harness.codex import codex_harness_agent as harness -from yeto.rl.harness.codex import reward as secrlenv_reward - - -def test_stock_codex_qwen38_adapter_process_binds_exact_xhigh_profile(): - environment = dict(os.environ) - environment["YETO_CODEX_CHAT_TEMPLATE"] = "qwen38" - result = subprocess.run( - [ - sys.executable, - "-c", - """ -from yeto.rl.harness.codex import codex_harness_agent as adapter -assert adapter.BACKEND_MODEL == "qwen38" -assert adapter.BACKEND_REASONING_EFFORT == "xhigh" -assert adapter.BACKEND_CHAT_TEMPLATE_KWARGS == { - "enable_thinking": True, - "preserve_thinking": True, - "reasoning_effort": "xhigh", -} -""", - ], - check=False, - capture_output=True, - text=True, - timeout=15, - env=environment, - ) - assert result.returncode == 0, result.stderr - - -def test_stock_codex_qwen35_adapter_process_binds_exact_fixed_profile(): - environment = dict(os.environ) - environment["YETO_CODEX_CHAT_TEMPLATE"] = "qwen35" - result = subprocess.run( - [ - sys.executable, - "-c", - """ -from yeto.rl.harness.codex import codex_harness_agent as adapter -assert adapter.BACKEND_MODEL == "qwen35" -assert adapter.BACKEND_REASONING_EFFORT == "xhigh" -assert adapter.BACKEND_CHAT_TEMPLATE_KWARGS == {"clear_thinking": False} -""", - ], - check=False, - capture_output=True, - text=True, - timeout=15, - env=environment, - ) - assert result.returncode == 0, result.stderr - - -def _completion( - name: str, - raw_arguments: str, - call_id: str, - reasoning: str, - total_tokens: int, -) -> dict[str, Any]: - return { - "id": f"chatcmpl-{call_id}", - "object": "chat.completion", - "created": 1, - "model": harness.BACKEND_MODEL, - "choices": [ - { - "index": 0, - "finish_reason": "tool_calls", - "message": { - "role": "assistant", - "content": "", - "reasoning_content": reasoning, - "tool_calls": [ - { - "id": call_id, - "type": "function", - "function": { - "name": name, - "arguments": raw_arguments, - }, - } - ], - }, - } - ], - "usage": { - "prompt_tokens": total_tokens - 5, - "completion_tokens": 5, - "total_tokens": total_tokens, - }, - } - - -def _fake_miles_sse(completion: dict[str, Any]) -> bytes: - choice = completion["choices"][0] - message = choice["message"] - delta = { - "role": message.get("role", "assistant"), - "content": message.get("content"), - } - if message.get("reasoning_content") is not None: - delta["reasoning_content"] = message["reasoning_content"] - if message.get("tool_calls"): - delta["tool_calls"] = [ - {**tool_call, "index": index} - for index, tool_call in enumerate(message["tool_calls"]) - ] - chunk = { - "id": completion.get("id"), - "object": "chat.completion.chunk", - "created": completion.get("created", 1), - "model": completion.get("model", harness.BACKEND_MODEL), - "choices": [ - { - "index": 0, - "delta": delta, - "finish_reason": choice.get("finish_reason"), - } - ], - "usage": completion.get("usage"), - } - return ( - b"data: " - + json.dumps( - chunk, - ensure_ascii=False, - allow_nan=False, - separators=(",", ":"), - ).encode() - + b"\n\ndata: [DONE]\n\n" - ) - - -async def _fake_miles( - completions: Iterable[dict[str, Any]], -) -> tuple[web.AppRunner, str, list[dict[str, Any]]]: - queue = list(completions) - requests: list[dict[str, Any]] = [] - - async def chat(request: web.Request) -> web.Response: - requests.append(await request.json()) - if not queue: - return web.json_response({"error": "unexpected sample"}, status=409) - return web.Response( - body=_fake_miles_sse(queue.pop(0)), - content_type="text/event-stream", - ) - - app = web.Application() - app.router.add_post("/v1/chat/completions", chat) - runner = web.AppRunner(app, access_log=None) - await runner.setup() - site = web.TCPSite(runner, "127.0.0.1", 0) - await site.start() - sockets = site._server.sockets if site._server is not None else [] - assert len(sockets) == 1 - return runner, f"http://127.0.0.1:{sockets[0].getsockname()[1]}", requests - - -async def _fake_miles_wire( - body: bytes, - *, - content_type: str = "text/event-stream", -) -> tuple[web.AppRunner, str]: - async def chat(_request: web.Request) -> web.Response: - return web.Response(body=body, headers={"Content-Type": content_type}) - - app = web.Application() - app.router.add_post("/v1/chat/completions", chat) - runner = web.AppRunner(app, access_log=None) - await runner.setup() - site = web.TCPSite(runner, "127.0.0.1", 0) - await site.start() - sockets = site._server.sockets if site._server is not None else [] - assert len(sockets) == 1 - return runner, f"http://127.0.0.1:{sockets[0].getsockname()[1]}" - - -def _codex_body(input_items: list[dict[str, Any]]) -> dict[str, Any]: - return { - "model": harness.BACKEND_MODEL, - "instructions": harness.BASE_INSTRUCTIONS, - "input": input_items, - "parallel_tool_calls": False, - "reasoning": {"effort": "xhigh", "summary": "none"}, - "store": False, - "stream": True, - "tool_choice": "auto", - "tools": [ - { - "type": "function", - "name": "update_plan", - "description": "Update the task plan.", - "strict": False, - "parameters": {"type": "object"}, - }, - *copy.deepcopy(harness._EXPECTED_RESPONSE_TOOLS), - ], - } - - -def _sse_events(raw: str) -> list[dict[str, Any]]: - events: list[dict[str, Any]] = [] - for block in raw.split("\n\n"): - data = [line[6:] for line in block.splitlines() if line.startswith("data: ")] - if data and data[0] != "[DONE]": - events.append(json.loads("\n".join(data))) - return events - - -def _set_bridge_env(monkeypatch: pytest.MonkeyPatch, max_tokens: int = 256) -> None: - monkeypatch.setenv("YETO_CODEX_BACKEND_MAX_TOKENS", str(max_tokens)) - - -def test_identity_is_live_exact_and_fail_closed(monkeypatch): - identity = harness.codex_harness_identity() - assert set(identity) == { - "base_instructions_sha256", - "terminal_exec_tool_schema_sha256", - "submit_tool_schema_sha256", - "dynamic_tools_schema_sha256", - } - assert all(len(value) == 64 and int(value, 16) >= 0 for value in identity.values()) - assert harness._MILES_TOOLS == harness.legacy.TOOLS - assert "model-facing tools, `terminal.exec` and `submit`" in ( - harness.BASE_INSTRUCTIONS - ) - assert "Codex's internal\n`terminal_exec` dynamic-tool alias" in ( - harness.BASE_INSTRUCTIONS - ) - monkeypatch.setattr(harness, "BASE_INSTRUCTIONS", harness.BASE_INSTRUCTIONS + " drift") - with pytest.raises(harness.CodexHarnessError, match="identity constants drifted"): - harness.codex_harness_identity() - - -def test_runtime_attests_live_binary_version_and_all_env(tmp_path, monkeypatch): - harness._RUNTIME_ATTESTATION_CACHE.clear() - binary = tmp_path / "codex-fake" - binary.write_text( - "#!/bin/sh\n" - "if [ \"$1\" = \"--version\" ]; then\n" - " printf 'codex-cli 0.145.0\\n'\n" - " exit 0\n" - "fi\n" - "exit 1\n" - ) - binary.chmod(0o700) - for name, value in harness._IDENTITY_ENV.items(): - monkeypatch.setenv(name, value) - monkeypatch.setenv("YETO_CODEX_BINARY_PATH", str(binary)) - monkeypatch.setenv( - "YETO_CODEX_BINARY_SHA256", hashlib.sha256(binary.read_bytes()).hexdigest() - ) - monkeypatch.setenv("YETO_CODEX_BINARY_SIZE_BYTES", str(binary.stat().st_size)) - monkeypatch.setenv("YETO_CODEX_VERSION", harness.CODEX_CLI_VERSION) - monkeypatch.setenv("YETO_CODEX_BACKEND_MAX_TOKENS", "256") - real_run = harness.subprocess.run - version_calls = 0 - - def counted_run(*args, **kwargs): - nonlocal version_calls - version_calls += 1 - return real_run(*args, **kwargs) - - monkeypatch.setattr(harness.subprocess, "run", counted_run) - assert harness._attest_runtime() == binary - assert harness._attest_runtime() == binary - assert version_calls == 1 - - binary.write_text(binary.read_text().replace("0.145.0", "0.145.1")) - monkeypatch.setenv( - "YETO_CODEX_BINARY_SHA256", hashlib.sha256(binary.read_bytes()).hexdigest() - ) - monkeypatch.setenv("YETO_CODEX_BINARY_SIZE_BYTES", str(binary.stat().st_size)) - with pytest.raises(harness.CodexHarnessError, match="version drifted"): - harness._attest_runtime() - assert version_calls == 2 - harness._RUNTIME_ATTESTATION_CACHE.clear() - - -def test_history_and_responses_contract_reject_compaction_and_extra_tools(): - with pytest.raises(harness.CodexHarnessError, match="compaction"): - harness._canonical_history_item({"type": "compaction"}) - body = _codex_body( - [ - { - "type": "message", - "role": "user", - "content": [{"type": "input_text", "text": "task"}], - } - ] - ) - body["tools"].append( - { - "type": "function", - "name": "shell", - "strict": False, - "parameters": {"type": "object"}, - } - ) - with pytest.raises(harness.CodexHarnessError, match="tool surface drifted"): - harness._validate_codex_request(body, harness.BACKEND_MODEL) - body = _codex_body(body["input"]) - body["reasoning"]["effort"] = "max" - with pytest.raises(harness.CodexHarnessError, match="xhigh"): - harness._validate_codex_request(body, harness.BACKEND_MODEL) - - -def test_sampling_cannot_override_signed_miles_fields(monkeypatch): - _set_bridge_env(monkeypatch) - metrics = harness.legacy.AgentMetrics() - with pytest.raises(harness.CodexHarnessError, match="override"): - harness._ResponsesBridge( - "http://127.0.0.1:1", - "task", - {"max_tokens": 64, "parallel_tool_calls": True}, - metrics, - max_seq_len=None, - ) - with pytest.raises(harness.CodexHarnessError, match="unknown sampling"): - harness._ResponsesBridge( - "http://127.0.0.1:1", - "task", - {"max_tokens": 64, "untrusted_body_field": "value"}, - metrics, - max_seq_len=None, - ) - - -def test_miles_fake_stream_round_trips_usage_reasoning_and_tool_call(): - raw_arguments = '{ "command": "printf café" }' - completion = _completion( - "terminal.exec", - raw_arguments, - "call-stream", - "inspect café", - 321, - ) - - normalized = harness._parse_miles_sse(_fake_miles_sse(completion)) - - assert normalized["object"] == "chat.completion" - assert normalized["usage"] == completion["usage"] - choice = normalized["choices"][0] - assert choice["finish_reason"] == "tool_calls" - assert choice["message"] == { - "role": "assistant", - "content": "", - "reasoning_content": "inspect café", - "tool_calls": [ - { - "id": "call-stream", - "type": "function", - "function": { - "name": "terminal.exec", - "arguments": raw_arguments, - }, - } - ], - } - - -def test_completion_accepts_parser_whitespace_but_rejects_real_prose(monkeypatch): - _set_bridge_env(monkeypatch) - whitespace_completion = _completion( - "terminal.exec", - '{"command":"id"}', - "call-whitespace", - "inspect", - 32, - ) - whitespace_completion["choices"][0]["message"]["content"] = "\n\n" - bridge = harness._ResponsesBridge( - "http://127.0.0.1:1", - "task", - {"max_tokens": 64}, - harness.legacy.AgentMetrics(), - max_seq_len=None, - ) - - output = bridge._translate_completion(whitespace_completion) - - assert output[-1]["type"] == "function_call" - assert output[-1]["name"] == "terminal_exec" - assert bridge._messages[-1]["content"] == "\n\n" - - prose_completion = copy.deepcopy(whitespace_completion) - prose_completion["choices"][0]["message"]["content"] = "I will describe it." - prose_bridge = harness._ResponsesBridge( - "http://127.0.0.1:1", - "task", - {"max_tokens": 64}, - harness.legacy.AgentMetrics(), - max_seq_len=None, - ) - with pytest.raises(harness.CodexModelFailure, match="mixed prose"): - prose_bridge._translate_completion(prose_completion) - - -def test_only_opted_in_profiles_keep_prose_before_the_tool_call(monkeypatch): - from yeto.rl.codex_backend import _PROFILES - - assert [name for name, p in _PROFILES.items() if p.get("allow_prose_with_tool_call")] == ["qwen38"] - assert harness.ALLOW_PROSE_WITH_TOOL_CALL is False # module default profile stays strict - _set_bridge_env(monkeypatch) - completion = _completion("terminal.exec", '{"command":"id"}', "call-prose", "inspect", 12) - completion["choices"][0]["message"]["content"] = "I found the cause; checking it now." - monkeypatch.setattr(harness, "ALLOW_PROSE_WITH_TOOL_CALL", True) - bridge = harness._ResponsesBridge("http://127.0.0.1:1", "task", {"max_tokens": 64}, - harness.legacy.AgentMetrics(), max_seq_len=None) - output = bridge._translate_completion(copy.deepcopy(completion)) - assert output[-1]["type"] == "function_call" and output[-1]["name"] == "terminal_exec" - assert bridge._messages[-1]["content"] == "I found the cause; checking it now." - - -def test_miles_fake_stream_preserves_length_as_sequence_policy_boundary(monkeypatch): - _set_bridge_env(monkeypatch) - completion = _completion( - "terminal.exec", - '{"command":"id"}', - "call-length", - "inspect", - 321, - ) - completion["choices"][0]["finish_reason"] = "length" - - normalized = harness._parse_miles_sse(_fake_miles_sse(completion)) - - assert normalized["choices"][0]["finish_reason"] == "length" - bridge = harness._ResponsesBridge( - "http://127.0.0.1:1", - "task", - {"max_tokens": 64}, - harness.legacy.AgentMetrics(), - max_seq_len=512, - ) - with pytest.raises(harness.CodexSequenceLimit, match="truncated"): - bridge._translate_completion(normalized) - - -def test_miles_fake_stream_rejects_malformed_framing_and_shapes(): - valid = _fake_miles_sse( - _completion( - "terminal.exec", - '{"command":"id"}', - "call-stream", - "inspect", - 32, - ) - ) - json_event, done_event, _empty = valid.split(b"\n\n") - chunk = json.loads(json_event[len(b"data: ") :]) - invalid_chunks = [] - for mutate in ( - lambda value: value.update({"extra": True}), - lambda value: value.update({"choices": []}), - lambda value: value["choices"][0].update({"index": False}), - lambda value: value["choices"][0]["delta"]["tool_calls"][0].update( - {"index": False} - ), - lambda value: value["choices"][0].update({"delta": {"role": "assistant"}}), - ): - value = copy.deepcopy(chunk) - mutate(value) - invalid_chunks.append( - b"data: " - + json.dumps(value, separators=(",", ":")).encode() - + b"\n\ndata: [DONE]\n\n" - ) - malformed = [ - b"", - b"data: [DONE]\n\n", - json_event + b"\n\n", - done_event + b"\n\n" + json_event + b"\n\n", - json_event + b"\n\n" + done_event + b"\n\n" + done_event + b"\n\n", - json_event + b"\n\n" + json_event + b"\n\n" + done_event + b"\n\n", - b"data: \xff\n\ndata: [DONE]\n\n", - b"data: {not-json}\n\ndata: [DONE]\n\n", - b'data: {"id":"first","id":"second"}\n\ndata: [DONE]\n\n', - b'data: {"created":NaN}\n\ndata: [DONE]\n\n', - *invalid_chunks, - ] - - for raw in malformed: - with pytest.raises(harness.CodexHarnessError, match="Miles session returned"): - harness._parse_miles_sse(raw) - - -def test_miles_fake_stream_omits_large_tito_metadata_from_client(monkeypatch): - _set_bridge_env(monkeypatch, max_tokens=32_768) - - async def scenario() -> None: - completion = _completion( - "terminal.exec", - '{"command":"id"}', - "call-large-record", - "inspect", - 32_768, - ) - completion["choices"][0]["meta_info"] = { - "output_token_logprobs": "x" * (harness.MAX_MILES_RESPONSE_BYTES + 1) - } - assert len(json.dumps(completion).encode()) > (harness.MAX_MILES_RESPONSE_BYTES) - runner, miles_url, miles_requests = await _fake_miles([completion]) - try: - async with harness._ResponsesBridge( - miles_url, - "task", - {"max_tokens": 32_768}, - harness.legacy.AgentMetrics(), - max_seq_len=65_536, - ) as bridge: - normalized = await bridge._sample_miles() - finally: - await runner.cleanup() - - assert normalized["choices"][0]["message"]["reasoning_content"] == ("inspect") - assert "meta_info" not in normalized["choices"][0] - assert miles_requests[0]["stream"] is True - - asyncio.run(scenario()) - - -def test_miles_response_reader_rejects_exact_cap_plus_one_without_unbounded_read(): - class NeverReadContent: - calls = 0 - - def iter_chunked(self, _size): - self.calls += 1 - raise AssertionError("oversized Content-Length must not be read") - - async def read(self): - raise AssertionError("unbounded read is forbidden") - - declared_content = NeverReadContent() - declared = SimpleNamespace( - content_length=harness.MAX_MILES_RESPONSE_BYTES + 1, - content=declared_content, - ) - with pytest.raises(harness.CodexHarnessError, match="oversized"): - asyncio.run(harness._read_bounded_miles_response(declared)) - assert declared_content.calls == 0 - - class ChunkedContent: - def __init__(self): - self.yields = 0 - self.read_past_limit = False - - def iter_chunked(self, size): - assert size == harness._MILES_RESPONSE_READ_CHUNK_BYTES - - async def chunks(): - block = b"x" * size - for _ in range(harness.MAX_MILES_RESPONSE_BYTES // size): - self.yields += 1 - yield block - self.yields += 1 - yield b"x" - self.read_past_limit = True - yield b"must-not-be-read" - - return chunks() - - async def read(self): - raise AssertionError("unbounded read is forbidden") - - chunked_content = ChunkedContent() - chunked = SimpleNamespace(content_length=None, content=chunked_content) - with pytest.raises(harness.CodexHarnessError, match="oversized"): - asyncio.run(harness._read_bounded_miles_response(chunked)) - assert chunked_content.yields == 65 - assert chunked_content.read_past_limit is False - - -def test_miles_non_200_response_is_closed_without_reading_body(monkeypatch): - _set_bridge_env(monkeypatch) - - class NeverReadContent: - calls = 0 - - def iter_chunked(self, _size): - self.calls += 1 - raise AssertionError("non-200 response body must not be read") - - async def read(self): - raise AssertionError("unbounded read is forbidden") - - class Response: - status = 503 - content_type = "application/json" - content_length = harness.MAX_MILES_RESPONSE_BYTES + 1 - content = NeverReadContent() - closed = False - - async def __aenter__(self): - return self - - async def __aexit__(self, *_exc): - self.closed = True - - class Session: - def __init__(self, response): - self.response = response - self.payload = None - self.headers = None - - def post(self, _url, *, json, headers): - self.payload = json - self.headers = headers - return self.response - - async def scenario() -> None: - response = Response() - session = Session(response) - bridge = harness._ResponsesBridge( - "http://127.0.0.1:1", - "task", - {"max_tokens": 64}, - harness.legacy.AgentMetrics(), - max_seq_len=512, - ) - bridge._session = session - with pytest.raises(harness.CodexHarnessError, match="HTTP 503"): - await bridge._sample_miles() - assert session.payload["stream"] is True - assert session.headers == {} - assert response.content.calls == 0 - assert response.closed is True - - asyncio.run(scenario()) - - -def test_miles_sample_accepts_content_type_parameters_and_enforces_cap(monkeypatch): - _set_bridge_env(monkeypatch) - - async def accepted() -> None: - completion = _completion( - "terminal.exec", - '{"command":"id"}', - "call-media-type", - "inspect", - 32, - ) - runner, miles_url = await _fake_miles_wire( - _fake_miles_sse(completion), - content_type="text/event-stream; charset=utf-8", - ) - try: - async with harness._ResponsesBridge( - miles_url, - "task", - {"max_tokens": 64}, - harness.legacy.AgentMetrics(), - max_seq_len=512, - ) as bridge: - assert await bridge._sample_miles() == harness._parse_miles_sse( - _fake_miles_sse(completion) - ) - finally: - await runner.cleanup() - - async def rejected(body: bytes, content_type: str, match: str) -> None: - runner, miles_url = await _fake_miles_wire( - body, - content_type=content_type, - ) - try: - async with harness._ResponsesBridge( - miles_url, - "task", - {"max_tokens": 64}, - harness.legacy.AgentMetrics(), - max_seq_len=512, - ) as bridge: - with pytest.raises(harness.CodexHarnessError, match=match): - await bridge._sample_miles() - finally: - await runner.cleanup() - - asyncio.run(accepted()) - asyncio.run(rejected(b"{}", "application/json", "media type")) - asyncio.run( - rejected( - b"x" * (harness.MAX_MILES_RESPONSE_BYTES + 1), - "text/event-stream", - "oversized", - ) - ) - - -def test_fake_codex_bridge_preserves_history_and_filters_only_update_plan(monkeypatch): - _set_bridge_env(monkeypatch) - - async def scenario() -> None: - raw_terminal = '{ "command": "id" }' - runner, miles_url, miles_requests = await _fake_miles( - [ - _completion( - "terminal.exec", raw_terminal, "call-terminal", "inspect", 25 - ), - _completion( - "submit", - '{"evidence":"uid=1000"}', - "call-submit", - "finish", - 40, - ), - ] - ) - metrics = harness.legacy.AgentMetrics() - try: - async with harness._ResponsesBridge( - miles_url, - "solve target", - { - "temperature": 0.7, - "top_p": 0.9, - "max_tokens": 128, - "stream": True, - "n": 1, - }, - metrics, - max_seq_len=512, - ) as bridge: - headers = {"Authorization": f"Bearer {bridge.token}"} - initial = [ - { - "type": "message", - "role": "user", - "content": [ - {"type": "input_text", "text": "solve target"} - ], - } - ] - async with aiohttp.ClientSession() as session: - async with session.post( - f"{bridge.url}/v1/responses", - json=_codex_body(initial), - headers=headers, - ) as response: - assert response.status == 200 - first_events = _sse_events(await response.text()) - completed = first_events[-1]["response"] - function_call = completed["output"][1] - assert function_call["name"] == "terminal_exec" - assert function_call["arguments"] == raw_terminal - assert function_call["call_id"] == "call-terminal" - - rendered = '{"exit_code":0,"output":"uid=1000","timed_out":false,"truncated":false}' - bridge.expect_tool_output("call-terminal", rendered) - second_input = copy.deepcopy(bridge._expected_input) - assert second_input is not None - async with session.post( - f"{bridge.url}/v1/responses", - json=_codex_body(second_input), - headers=headers, - ) as response: - assert response.status == 200 - second_events = _sse_events(await response.text()) - submit = second_events[-1]["response"]["output"][1] - assert submit["name"] == "submit" - assert submit["call_id"] == "call-submit" - bridge.mark_terminal() - finally: - await runner.cleanup() - - assert len(miles_requests) == 2 - assert [tool["function"]["name"] for tool in miles_requests[0]["tools"]] == [ - "terminal.exec", - "submit", - ] - assert miles_requests[0]["max_tokens"] == 128 - assert miles_requests[0]["temperature"] == 0.7 - assert miles_requests[0]["top_p"] == 0.9 - assert miles_requests[0]["stream"] is True - assert miles_requests[0]["parallel_tool_calls"] is False - assert miles_requests[0]["reasoning_effort"] == "max" - assert miles_requests[0]["thinking"] == {"type": "enabled"} - assert miles_requests[0]["chat_template_kwargs"] == ( - harness.BACKEND_CHAT_TEMPLATE_KWARGS - ) - assert miles_requests[1]["messages"][2] == { - "role": "assistant", - "content": "", - "reasoning_content": "inspect", - "tool_calls": [ - { - "id": "call-terminal", - "type": "function", - "function": { - "name": "terminal.exec", - "arguments": raw_terminal, - }, - } - ], - } - assert miles_requests[1]["messages"][3] == { - "role": "tool", - "tool_call_id": "call-terminal", - "content": rendered, - } - assert metrics.turns == 2 - assert metrics.tool_calls == 2 - - asyncio.run(scenario()) - - -def test_bridge_compaction_is_same_session_trainable_and_keeps_atomic_tail(monkeypatch): - _set_bridge_env(monkeypatch, max_tokens=128) - monkeypatch.setenv("YETO_CODEX_COMPACTION_ENABLED", "1") - monkeypatch.setenv("YETO_CODEX_COMPACTION_TRIGGER_TOKENS", "300") - monkeypatch.setenv("YETO_CODEX_COMPACTION_SUMMARY_MAX_TOKENS", "128") - monkeypatch.setenv("YETO_CODEX_MAX_COMPACTIONS", "3") - - summary = { - "id": "chatcmpl-summary", - "object": "chat.completion", - "created": 1, - "model": harness.BACKEND_MODEL, - "choices": [ - { - "index": 0, - "finish_reason": "stop", - "message": { - "role": "assistant", - "content": "Goal: solve target. Evidence: uid=1000. Next: submit.", - "reasoning_content": "compress state", - }, - } - ], - "usage": {"prompt_tokens": 50, "completion_tokens": 10, "total_tokens": 60}, - } - - async def scenario() -> None: - requests: list[tuple[dict[str, Any], dict[str, str]]] = [] - queue = [ - _completion("terminal.exec", '{"command":"id"}', "call-1", "inspect", 25), - summary, - _completion("submit", '{"evidence":"uid=1000"}', "call-2", "finish", 40), - ] - - async def chat(request: web.Request) -> web.Response: - requests.append((await request.json(), dict(request.headers))) - return web.Response( - body=_fake_miles_sse(queue.pop(0)), - content_type="text/event-stream", - ) - - app = web.Application() - app.router.add_post("/v1/chat/completions", chat) - runner = web.AppRunner(app, access_log=None) - await runner.setup() - site = web.TCPSite(runner, "127.0.0.1", 0) - await site.start() - sockets = site._server.sockets if site._server is not None else [] - miles_url = f"http://127.0.0.1:{sockets[0].getsockname()[1]}" - try: - async with harness._ResponsesBridge( - miles_url, - "solve target", - {"max_tokens": 128}, - harness.legacy.AgentMetrics(), - max_seq_len=2048, - ) as bridge: - headers = {"Authorization": f"Bearer {bridge.token}"} - initial = [ - { - "type": "message", - "role": "user", - "content": [{"type": "input_text", "text": "solve target"}], - } - ] - async with aiohttp.ClientSession() as session: - async with session.post( - f"{bridge.url}/v1/responses", - json=_codex_body(initial), - headers=headers, - ) as response: - assert response.status == 200 - bridge.expect_tool_output("call-1", '{"output":"uid=1000"}') - second_input = copy.deepcopy(bridge._expected_input) - async with session.post( - f"{bridge.url}/v1/responses", - json=_codex_body(second_input), - headers=headers, - ) as response: - assert response.status == 200 - finally: - await runner.cleanup() - - assert len(requests) == 3 - _first, first_headers = requests[0] - summary_request, summary_headers = requests[1] - resumed, resumed_headers = requests[2] - assert first_headers["X-Miles-Compaction-Segment-Type"] == "execution" - assert summary_headers["X-Miles-Compaction-Segment-Type"] == "summary" - assert summary_headers["X-Miles-Compaction-Segment-Index"] == "1" - assert summary_headers["X-Miles-Compaction-Context-Budget"] == "2048" - assert summary_request["tools"] == [] - assert summary_request["max_tokens"] == 128 - assert summary_request["messages"][-1]["content"] == harness.COMPACTION_SUMMARY_INSTRUCTION - assert resumed_headers["X-Miles-Compaction-Context-Window"] == "1" - assert resumed_headers["X-Miles-Compaction-Segment-Index"] == "2" - assert resumed_headers["X-Miles-Compaction-Segment-Type"] == "execution" - assert resumed["messages"][0] == {"role": "system", "content": harness.BASE_INSTRUCTIONS} - assert "uid=1000" in resumed["messages"][1]["content"] - assert resumed["messages"][2:] == summary_request["messages"][2:-1] - - asyncio.run(scenario()) - - -def test_bridge_compaction_trigger_is_consumed_context_boundary(monkeypatch): - _set_bridge_env(monkeypatch, max_tokens=1024) - monkeypatch.setenv("YETO_CODEX_COMPACTION_ENABLED", "1") - monkeypatch.setenv("YETO_CODEX_COMPACTION_TRIGGER_TOKENS", "6144") - monkeypatch.setenv("YETO_CODEX_COMPACTION_SUMMARY_MAX_TOKENS", "1024") - monkeypatch.setenv("YETO_CODEX_MAX_COMPACTIONS", "3") - - bridge = harness._ResponsesBridge( - "http://127.0.0.1:1", - "boundary", - {"max_tokens": 1024}, - harness.legacy.AgentMetrics(), - max_seq_len=8192, - ) - - assert not bridge._should_compact(6143) - assert bridge._should_compact(6144) - assert ( - 6144 - + harness.COMPACTION_SUMMARY_INSTRUCTION_TOKEN_RESERVE - + bridge._compaction_summary_max_tokens - <= 8192 - ) - - -def test_bridge_compaction_rejects_trigger_without_summary_reserve(monkeypatch): - _set_bridge_env(monkeypatch, max_tokens=512) - monkeypatch.setenv("YETO_CODEX_COMPACTION_ENABLED", "1") - monkeypatch.setenv("YETO_CODEX_COMPACTION_TRIGGER_TOKENS", "500") - monkeypatch.setenv("YETO_CODEX_COMPACTION_SUMMARY_MAX_TOKENS", "512") - - with pytest.raises(harness.CodexHarnessError, match="insufficient room"): - harness._ResponsesBridge( - "http://127.0.0.1:1", - "reserve", - {"max_tokens": 512}, - harness.legacy.AgentMetrics(), - max_seq_len=1024, - ) - - -def test_bridge_rejects_history_mutation_without_an_extra_sample(monkeypatch): - _set_bridge_env(monkeypatch) - - async def scenario() -> None: - runner, miles_url, miles_requests = await _fake_miles( - [_completion("terminal.exec", '{"command":"id"}', "call-1", "think", 10)] - ) - try: - async with harness._ResponsesBridge( - miles_url, - "immutable task", - {"max_tokens": 64}, - harness.legacy.AgentMetrics(), - max_seq_len=None, - ) as bridge: - headers = {"Authorization": f"Bearer {bridge.token}"} - initial = [ - { - "type": "message", - "role": "user", - "content": [ - {"type": "input_text", "text": "immutable task"} - ], - } - ] - async with aiohttp.ClientSession() as session: - async with session.post( - f"{bridge.url}/v1/responses", - json=_codex_body(initial), - headers=headers, - ) as response: - assert response.status == 200 - bridge.expect_tool_output("call-1", "tool result") - mutated = copy.deepcopy(bridge._expected_input) - assert mutated is not None - mutated[0]["content"][0]["text"] = "changed task" - async with session.post( - f"{bridge.url}/v1/responses", - json=_codex_body(mutated), - headers=headers, - ) as response: - assert response.status == 400 - assert isinstance(bridge.fatal.result(), harness.CodexHarnessError) - finally: - await runner.cleanup() - assert len(miles_requests) == 1 - - asyncio.run(scenario()) - - -class _FakeEpisodeClient: - def __init__(self, *, output: str = "uid=1000", truncated: bool = False) -> None: - self.output = output - self.truncated = truncated - self.exec_calls: list[tuple[str, str]] = [] - self.submissions: list[tuple[str, dict[str, Any]]] = [] - - async def execute(self, episode_id: str, command: str, **_kwargs: Any): - self.exec_calls.append((episode_id, command)) - return { - "exit_code": 0, - "output": self.output, - "timed_out": False, - "truncated": self.truncated, - } - - async def submit(self, episode_id: str, value: dict[str, Any]): - self.submissions.append((episode_id, value)) - return {"accepted": True} - - -class _RecoveringEpisodeClient(_FakeEpisodeClient): - def __init__(self) -> None: - super().__init__() - self.execute_attempts = 0 - self.submit_attempts = 0 - - async def execute(self, episode_id: str, command: str, **kwargs: Any): - self.execute_attempts += 1 - if self.execute_attempts == 1: - self.exec_calls.append((episode_id, command)) - raise harness.EpisodeAPIError(400, "invalid_command", "command rejected") - return await super().execute(episode_id, command, **kwargs) - - async def submit(self, episode_id: str, value: dict[str, Any]): - self.submit_attempts += 1 - if self.submit_attempts == 1: - self.submissions.append((episode_id, value)) - raise harness.EpisodeAPIError(400, "invalid_submission", "evidence rejected") - return await super().submit(episode_id, value) - - -def test_codex_exhausted_precreate_503_returns_signed_admission(monkeypatch): - attempts = [] - - class AdmissionClient: - async def __aenter__(self): - return self - - async def __aexit__(self, *_args): - return None - - async def create(self, task_id, tier): - attempts.append((task_id, tier)) - raise harness.EpisodeAPIError( - 503, "infrastructure_error", "unavailable" - ) - - monkeypatch.setenv("SECRLENV_CAPACITY_MAX_WAIT_SECONDS", "30") - monkeypatch.setattr(harness, "EpisodeClient", AdmissionClient) - monkeypatch.setattr(harness, "_attest_runtime", lambda: Path("/codex")) - monkeypatch.setattr(harness.legacy.random, "uniform", lambda *_args: 0.0) - - result = asyncio.run( - harness.run( - "http://127.0.0.1:1", - None, - metadata={"task_id": "CVE-2024-1234", "prompt_tier": "l2"}, - ) - ) - - assert result is not None - outcome, value = secrlenv_reward._verified_outcome(result) - assert len(attempts) == 2 - assert outcome["schema"] == 2 - assert outcome["episode_id"] is None - assert outcome[harness.INFRASTRUCTURE_503_RETRIES_KEY] == 1 - assert value == 0.0 - harness.legacy.require_no_episode_residue() - - -def test_codex_precreate_infrastructure_retry_timeout_is_not_a_verdict( - monkeypatch, -): - attempts = 0 - - class AdmissionTimeoutClient: - async def __aenter__(self): - return self - - async def __aexit__(self, *_args): - return None - - async def create(self, _task_id, _tier): - nonlocal attempts - attempts += 1 - if attempts == 1: - raise harness.EpisodeAPIError( - 503, "infrastructure_error", "unavailable" - ) - await asyncio.Event().wait() - - monkeypatch.setenv("SECRLENV_CAPACITY_MAX_WAIT_SECONDS", "0.01") - monkeypatch.setattr(harness, "EpisodeClient", AdmissionTimeoutClient) - monkeypatch.setattr(harness, "_attest_runtime", lambda: Path("/codex")) - monkeypatch.setattr(harness.legacy.random, "uniform", lambda *_args: 0.0) - - result = asyncio.run( - harness.run( - "http://127.0.0.1:1", - None, - metadata={"task_id": "CVE-2024-1234", "prompt_tier": "l2"}, - ) - ) - - assert result is None - assert attempts == 2 - harness.legacy.require_no_episode_residue() - - -def test_run_does_not_start_codex_before_capacity_retry_creates_episode( - monkeypatch, -): - episode_id = "e" * 24 - events: list[str] = [] - - class CapacityThenReadyClient: - def __init__(self) -> None: - self.create_calls = 0 - - async def __aenter__(self): - return self - - async def __aexit__(self, *_args): - return None - - async def create(self, task_id: str, tier: str): - assert (task_id, tier) == ("CVE-2024-1234", "l2") - self.create_calls += 1 - events.append("create") - if self.create_calls == 1: - raise harness.EpisodeAPIError( - 503, "capacity_reached", "busy" - ) - return { - "episode_id": episode_id, - "prompt": "immutable task", - } - - async def evaluate(self, value: str): - assert value == episode_id - events.append("evaluate") - return { - "task_id": "CVE-2024-1234", - "episode_id": episode_id, - "reward": 1.0, - "passed": True, - } - - async def close(self, value: str): - assert value == episode_id - events.append("close") - return {"closed": True} - - async def drive_after_create( - _binary, - _base_url, - _client, - episode, - _request_kwargs, - _metrics, - *, - max_seq_len, - ): - assert episode["episode_id"] == episode_id - assert max_seq_len is None - assert events == ["create", "create"] - events.append("drive") - return "completed" - - monkeypatch.setenv("SECRLENV_CAPACITY_MAX_WAIT_SECONDS", "30") - monkeypatch.setattr(harness, "EpisodeClient", CapacityThenReadyClient) - monkeypatch.setattr(harness, "_attest_runtime", lambda: Path("/codex")) - monkeypatch.setattr(harness, "_drive_codex", drive_after_create) - monkeypatch.setattr(harness.legacy.random, "uniform", lambda *_args: 0.0) - try: - result = asyncio.run( - harness.run( - "http://127.0.0.1:1", - None, - metadata={"task_id": "CVE-2024-1234", "prompt_tier": "l2"}, - ) - ) - finally: - harness.legacy._release_episode(episode_id) - - assert result is not None - assert result["exit_status"] == "completed" - assert events == ["create", "create", "drive", "evaluate", "close"] - - -def test_run_close_failure_returns_terminal_cleanup_evidence(monkeypatch): - episode_id = "d" * 24 - close_calls = [] - - class CloseFailingClient: - async def __aenter__(self): - return self - - async def __aexit__(self, *_args): - return None - - async def create(self, _task_id: str, _tier: str): - return {"episode_id": episode_id, "prompt": "immutable task"} - - async def evaluate(self, value: str): - assert value == episode_id - return { - "task_id": "CVE-2024-1234", - "episode_id": episode_id, - "reward": 1.0, - "passed": True, - } - - async def close(self, value: str): - assert value == episode_id - close_calls.append(value) - raise harness.EpisodeClientError("close failed") - - async def complete_policy(*_args, **_kwargs): - return "completed" - - monkeypatch.setattr(harness, "EpisodeClient", CloseFailingClient) - monkeypatch.setattr(harness, "_attest_runtime", lambda: Path("/codex")) - monkeypatch.setattr(harness, "_drive_codex", complete_policy) - try: - result = asyncio.run( - harness.run( - "http://127.0.0.1:1", - None, - metadata={"task_id": "CVE-2024-1234", "prompt_tier": "l2"}, - ) - ) - assert result is not None - assert result["exit_status"] == harness.legacy.CLEANUP_ERROR_STATUS - assert ( - result[harness.OUTCOME_KEY]["status"] - == harness.legacy.CLEANUP_ERROR_STATUS - ) - assert len(close_calls) == 2 - assert harness.legacy._EPISODE_PHASES[episode_id] == "cleanup_pending" - finally: - harness.legacy._release_episode(episode_id) - - -def test_driver_reaps_isolated_process_group_before_home_cleanup(monkeypatch): - events: list[object] = [] - - class FakeProcess: - pid = 4321 - returncode = None - - async def wait(self): - events.append("wait-parent") - self.returncode = 0 - return 0 - - def terminate(self): - raise AssertionError("process-group TERM should be used") - - def kill(self): - raise AssertionError("process-group KILL should be used") - - async def scenario() -> None: - driver = harness._AppServerDriver( - Path("/unused"), - SimpleNamespace(), # type: ignore[arg-type] - _FakeEpisodeClient(), - "episode", - "task", - harness.legacy.AgentMetrics(), - ) - driver._process = FakeProcess() # type: ignore[assignment] - driver._process_group_id = 4321 - monkeypatch.setattr(harness, "_CODEX_DESCENDANT_TERM_GRACE_SECONDS", 0.0) - monkeypatch.setattr( - harness.os, - "killpg", - lambda process_group_id, sig: events.append((process_group_id, sig)), - ) - - await driver._stop() - - assert events == [ - (4321, harness.signal.SIGTERM), - "wait-parent", - (4321, harness.signal.SIGKILL), - ] - assert driver._process_group_id is None - - asyncio.run(scenario()) - - -def test_codex_argv_disables_background_plugin_sync(): - bridge = SimpleNamespace(url="http://127.0.0.1:1234") - - argv = harness._codex_argv(Path("/codex"), bridge) # type: ignore[arg-type] - - assert argv.count("features.plugins=false") == 1 - - -def test_driver_retries_transient_nonempty_isolated_home_cleanup(monkeypatch, tmp_path): - home_path = tmp_path / "codex-home" - home_path.mkdir() - - class RacingHome: - name = str(home_path) - attempts = 0 - - def cleanup(self): - self.attempts += 1 - if self.attempts < 3: - raise OSError(errno.ENOTEMPTY, "Directory not empty", "plugins") - home_path.rmdir() - - async def scenario() -> None: - home = RacingHome() - driver = harness._AppServerDriver( - Path("/unused"), - SimpleNamespace(), # type: ignore[arg-type] - _FakeEpisodeClient(), - "episode", - "task", - harness.legacy.AgentMetrics(), - ) - driver._isolated_home = home # type: ignore[assignment] - monkeypatch.setattr( - harness, - "_CODEX_HOME_CLEANUP_RETRY_DELAYS_SECONDS", - (0.0, 0.0, 0.0), - ) - - await driver.__aexit__(None, None, None) - - assert home.attempts == 3 - assert not home_path.exists() - - asyncio.run(scenario()) - - -def test_driver_home_cleanup_exhaustion_is_best_effort_and_nonfatal( - monkeypatch, tmp_path -): - home_path = tmp_path / "codex-home" - (home_path / "plugins").mkdir(parents=True) - - class PersistentlyRacingHome: - name = str(home_path) - attempts = 0 - - def cleanup(self): - self.attempts += 1 - raise OSError(errno.ENOTEMPTY, "Directory not empty", "plugins") - - async def scenario() -> None: - home = PersistentlyRacingHome() - driver = harness._AppServerDriver( - Path("/unused"), - SimpleNamespace(), # type: ignore[arg-type] - _FakeEpisodeClient(), - "episode", - "task", - harness.legacy.AgentMetrics(), - ) - driver._isolated_home = home # type: ignore[assignment] - monkeypatch.setattr( - harness, - "_CODEX_HOME_CLEANUP_RETRY_DELAYS_SECONDS", - (0.0, 0.0), - ) - - # A resource-hygiene race must not change a successful episode result. - await driver.__aexit__(None, None, None) - - assert home.attempts == 2 - assert not home_path.exists() - - asyncio.run(scenario()) - - -def test_driver_teardown_failures_do_not_mask_active_episode_error( - monkeypatch, tmp_path -): - class NoLaunchDriver(harness._AppServerDriver): - async def __aenter__(self): - return self - - home_path = tmp_path / "codex-home" - home_path.mkdir() - - class FailedHome: - name = str(home_path) - - def cleanup(self): - raise RuntimeError("secondary home cleanup failure") - - async def scenario() -> None: - driver = NoLaunchDriver( - Path("/unused"), - SimpleNamespace(), # type: ignore[arg-type] - _FakeEpisodeClient(), - "episode", - "task", - harness.legacy.AgentMetrics(), - ) - driver._isolated_home = FailedHome() # type: ignore[assignment] - - async def failed_stop(): - raise RuntimeError("secondary shutdown failure") - - monkeypatch.setattr(driver, "_stop", failed_stop) - with pytest.raises(ValueError, match="primary episode failure"): - async with driver: - raise ValueError("primary episode failure") - assert not home_path.exists() - - asyncio.run(scenario()) - - -def test_driver_rejects_mutated_arguments_but_preserves_bounded_truncation(monkeypatch): - _set_bridge_env(monkeypatch) - - async def scenario() -> None: - metrics = harness.legacy.AgentMetrics() - bridge = harness._ResponsesBridge( - "http://127.0.0.1:1", "task", {"max_tokens": 64}, metrics, max_seq_len=None - ) - bridge._pending = harness._PendingTool( - "call-1", "terminal_exec", "terminal.exec", '{"command":"id"}' - ) - bridge._history_after_model = [] - client = _FakeEpisodeClient(truncated=True) - driver = harness._AppServerDriver( - Path("/unused"), bridge, client, "episode", "task", metrics - ) - driver._thread_id = "thread" - driver._turn_id = "turn" - with pytest.raises(harness.CodexHarnessError, match="mutated raw"): - await driver._handle_tool_request( - { - "id": 1, - "method": "item/tool/call", - "params": { - "threadId": "thread", - "turnId": "turn", - "callId": "call-1", - "namespace": None, - "tool": "terminal_exec", - "arguments": {"command": "whoami"}, - }, - } - ) - observation = await driver._execute_terminal({"command": "id"}) - assert observation == { - "exit_code": 0, - "output": "uid=1000", - "timed_out": False, - "truncated": True, - } - client.output = "x" * (harness.MAX_TOOL_OUTPUT_BYTES + 1) - with pytest.raises(harness.CodexHarnessError, match="signed boundary"): - await driver._execute_terminal({"command": "id"}) - - asyncio.run(scenario()) - - -def test_driver_prefers_policy_boundary_set_while_stdout_read_is_pending(monkeypatch): - _set_bridge_env(monkeypatch) - - async def scenario() -> None: - bridge = harness._ResponsesBridge( - "http://127.0.0.1:1", - "task", - {"max_tokens": 64}, - harness.legacy.AgentMetrics(), - max_seq_len=None, - ) - bridge._fatal = asyncio.get_running_loop().create_future() - boundary = harness.CodexModelFailure("sampled policy failure") - - class RaceStdout: - async def readline(self) -> bytes: - bridge._fatal.set_result(boundary) - return b"" - - class RaceProcess: - stdout = RaceStdout() - returncode = None - - driver = harness._AppServerDriver( - Path("/unused"), - bridge, - _FakeEpisodeClient(), - "episode", - "task", - harness.legacy.AgentMetrics(), - ) - driver._process = RaceProcess() # type: ignore[assignment] - with pytest.raises(harness.CodexModelFailure) as caught: - await driver._read_message() - assert caught.value is boundary - - asyncio.run(scenario()) - - -def _stock_codex_binary() -> Path: - # YETO_CODEX_BINARY_PATH (the fetched bundle's binary, see - # scripts/fetch_codex_bundle.py) wins over the PATH install. - binary_name = os.environ.get("YETO_CODEX_BINARY_PATH") or shutil.which("codex") - if not binary_name or not Path(binary_name).is_file(): - pytest.skip("stock Codex is not installed on this controller") - version = subprocess.run( - [binary_name, "--version"], capture_output=True, text=True, check=True - ).stdout.strip() - if version != harness.CODEX_CLI_VERSION: - pytest.skip(f"requires {harness.CODEX_CLI_VERSION}, found {version}") - return Path(binary_name) - - -def test_stock_codex_0145_app_server_fake_miles_integration(monkeypatch): - binary = _stock_codex_binary() - _set_bridge_env(monkeypatch, max_tokens=32_768) - - async def scenario() -> None: - long_reasoning = "r" * 100_000 - maximum_output = "x" * harness.MAX_TOOL_OUTPUT_BYTES - runner, miles_url, miles_requests = await _fake_miles( - [ - _completion( - "terminal.exec", - '{"command":"id"}', - "call-1", - long_reasoning, - 25_000, - ), - _completion( - "submit", - '{"evidence":"uid=1000"}', - "call-2", - "submit", - 60_000, - ), - ] - ) - metrics = harness.legacy.AgentMetrics() - client = _FakeEpisodeClient(output=maximum_output, truncated=True) - try: - async with harness._ResponsesBridge( - miles_url, - "solve target", - {"temperature": 0.7, "max_tokens": 32_768}, - metrics, - max_seq_len=200_000, - ) as bridge, harness._AppServerDriver( - binary, - bridge, - client, - "episode-stock", - "solve target", - metrics, - ) as driver: - status = await asyncio.wait_for(driver.drive(), timeout=20.0) - assert status == "completed" - finally: - await runner.cleanup() - - assert client.exec_calls == [("episode-stock", "id")] - assert client.submissions == [ - ("episode-stock", {"evidence": "uid=1000"}) - ] - assert len(miles_requests) == 2 - assert [tool["function"]["name"] for tool in miles_requests[0]["tools"]] == [ - "terminal.exec", - "submit", - ] - tool_message = miles_requests[1]["messages"][-1] - assert tool_message["role"] == "tool" - assert json.loads(tool_message["content"])["output"] == maximum_output - assert json.loads(tool_message["content"])["truncated"] is True - assert len(json.loads(tool_message["content"])["output"]) == ( - harness.MAX_TOOL_OUTPUT_BYTES - ) - - asyncio.run(scenario()) - - -def test_stock_codex_turn_limit_is_a_normal_policy_boundary(monkeypatch): - binary = _stock_codex_binary() - _set_bridge_env(monkeypatch) - monkeypatch.setenv("SECRLENV_MAX_TURNS", "1") - - async def scenario() -> None: - runner, miles_url, miles_requests = await _fake_miles( - [ - _completion( - "terminal.exec", '{"command":"id"}', "call-1", "inspect", 20 - ) - ] - ) - metrics = harness.legacy.AgentMetrics() - client = _FakeEpisodeClient() - try: - async with harness._ResponsesBridge( - miles_url, - "solve target", - {"max_tokens": 128}, - metrics, - max_seq_len=512, - ) as bridge, harness._AppServerDriver( - binary, - bridge, - client, - "episode-limit", - "solve target", - metrics, - ) as driver: - assert await asyncio.wait_for(driver.drive(), timeout=20.0) == "max_turns" - finally: - await runner.cleanup() - - assert client.exec_calls == [("episode-limit", "id")] - assert client.submissions == [] - assert len(miles_requests) == 1 - assert metrics.turns == 1 - - asyncio.run(scenario()) - - -def test_stock_codex_model_format_failure_retains_sample_as_normal_failure(monkeypatch): - binary = _stock_codex_binary() - _set_bridge_env(monkeypatch) - - async def scenario() -> None: - completion = { - "id": "chatcmpl-policy-failure", - "object": "chat.completion", - "choices": [ - { - "index": 0, - "finish_reason": "stop", - "message": { - "role": "assistant", - "content": "I will only describe what to do.", - "reasoning_content": "decline tool use", - "tool_calls": [], - }, - } - ], - "usage": { - "prompt_tokens": 10, - "completion_tokens": 5, - "total_tokens": 15, - }, - } - runner, miles_url, miles_requests = await _fake_miles([completion]) - metrics = harness.legacy.AgentMetrics() - client = _FakeEpisodeClient() - try: - async with harness._ResponsesBridge( - miles_url, - "solve target", - {"max_tokens": 128}, - metrics, - max_seq_len=512, - ) as bridge, harness._AppServerDriver( - binary, - bridge, - client, - "episode-policy-failure", - "solve target", - metrics, - ) as driver: - assert await asyncio.wait_for(driver.drive(), timeout=20.0) == "max_turns" - finally: - await runner.cleanup() - - assert client.exec_calls == [] - assert client.submissions == [] - assert len(miles_requests) == 1 - assert metrics.turns == 1 - assert metrics.parse_failures == 1 - - asyncio.run(scenario()) - - -def test_stock_codex_daemon_400_tool_errors_recover_without_dropping_sample(monkeypatch): - binary = _stock_codex_binary() - _set_bridge_env(monkeypatch) - - async def scenario() -> None: - runner, miles_url, miles_requests = await _fake_miles( - [ - _completion( - "terminal.exec", '{"command":"bad"}', "call-1", "try", 15 - ), - _completion( - "terminal.exec", '{"command":"id"}', "call-2", "recover", 30 - ), - _completion( - "submit", - '{"evidence":"too vague"}', - "call-3", - "submit", - 45, - ), - _completion( - "submit", - '{"evidence":"uid=1000"}', - "call-4", - "repair", - 60, - ), - ] - ) - metrics = harness.legacy.AgentMetrics() - client = _RecoveringEpisodeClient() - try: - async with harness._ResponsesBridge( - miles_url, - "solve target", - {"max_tokens": 128}, - metrics, - max_seq_len=1024, - ) as bridge, harness._AppServerDriver( - binary, - bridge, - client, - "episode-recovery", - "solve target", - metrics, - ) as driver: - assert await asyncio.wait_for(driver.drive(), timeout=20.0) == "completed" - finally: - await runner.cleanup() - - assert client.exec_calls == [ - ("episode-recovery", "bad"), - ("episode-recovery", "id"), - ] - assert client.submissions == [ - ("episode-recovery", {"evidence": "too vague"}), - ("episode-recovery", {"evidence": "uid=1000"}), - ] - assert len(miles_requests) == 4 - assert json.loads(miles_requests[1]["messages"][-1]["content"]) == { - "error": "command rejected" - } - assert json.loads(miles_requests[3]["messages"][-1]["content"]) == { - "error": "evidence rejected" - } - assert metrics.tool_calls == 4 - assert metrics.terminal_calls == 1 - assert metrics.submit_calls == 1 - - asyncio.run(scenario()) diff --git a/tests/test_tbench21_sao_diloco_plan.py b/tests/test_tbench21_sao_diloco_plan.py deleted file mode 100644 index 21aa2a8be..000000000 --- a/tests/test_tbench21_sao_diloco_plan.py +++ /dev/null @@ -1,234 +0,0 @@ -from __future__ import annotations - -import importlib.util -import json -from pathlib import Path - -import pytest - -from yeto.rl import CODEX_OPENENV_IDENTITY_ENV, SECRLENV_AGENTS - -TOOL = ( - Path(__file__).resolve().parents[1] - / "tools" - / "probes" - / "build_tbench21_sao_diloco_plan.py" -) -SPEC = importlib.util.spec_from_file_location("build_tbench21_sao_diloco_plan", TOOL) -assert SPEC is not None and SPEC.loader is not None -plan = importlib.util.module_from_spec(SPEC) -SPEC.loader.exec_module(plan) - - -def _task_tree(root: Path, count: int = 89) -> Path: - tasks = root / "tasks" - for index in range(count): - task = tasks / f"task-{index:03d}" - (task / "environment").mkdir(parents=True) - (task / "instruction.md").write_text("untrusted task text\n") - (task / "environment" / "Dockerfile").write_text("FROM scratch\n") - (task / "task.toml").write_text( - f"""schema_version = "1.1" -[task] -name = "terminal-bench/task-{index:03d}" -[metadata] -category = "category-{index % 7}" -[environment] -docker_image = "example/task-{index:03d}:fixed" -cpus = 1 -memory_mb = 2048 -storage_mb = 10240 -""" - ) - return tasks - - -def _jsonl(path: Path) -> list[dict]: - return [json.loads(line) for line in path.read_text().splitlines() if line] - - -def test_builds_exact_full_suite_plan(tmp_path): - tasks = _task_tree(tmp_path / "source") - output = tmp_path / "plan" - - manifest_path = plan.build_plan( - tasks, - output, - seed="fixed-seed", - run_root="/run/tbench21", - ) - manifest = json.loads(manifest_path.read_text()) - - assert manifest["schema"] == plan.SCHEMA - assert manifest["split"]["train_task_count"] == 44 - assert manifest["split"]["eval_task_count"] == 45 - train_ids = set(manifest["split"]["train_task_ids"]) - eval_ids = set(manifest["split"]["eval_task_ids"]) - assert not train_ids & eval_ids - assert len(train_ids | eval_ids) == 89 - assert manifest["rollouts"] == { - "per_task": 4, - "baseline": 356, - "train": 176, - "eval": 180, - "episode_timeout_seconds": 1800, - "target_concurrency": 304, - "per_island_concurrency": 38, - "seed_base": 82621, - "per_island_seeds": list(range(82621, 82629)), - "baseline_achievable_concurrency": 304, - "train_cardinality_limited_concurrency": 176, - "eval_cardinality_limited_concurrency": 180, - } - - baseline = _jsonl(output / "baseline" / "all.jsonl") - train = _jsonl(output / "train" / "all.jsonl") - evaluation = _jsonl(output / "eval" / "all.jsonl") - assert (len(baseline), len(train), len(evaluation)) == (356, 176, 180) - assert {row["metadata"]["task_id"] for row in train} == train_ids - assert {row["metadata"]["task_id"] for row in evaluation} == eval_ids - assert all(row["metadata"]["episode_timeout_seconds"] == 1800 for row in baseline) - assert all( - row["metadata"]["rollout_seed"] - == 82621 + row["metadata"]["island_id"] - for row in baseline + train + evaluation - ) - assert all( - len([row for row in baseline if row["metadata"]["task_id"] == task_id]) == 4 - for task_id in train_ids | eval_ids - ) - - assert [len(_jsonl(output / "train" / f"island-{i}.jsonl")) for i in range(8)] == [ - 22 - ] * 8 - baseline_counts = [ - len(_jsonl(output / "baseline" / f"island-{i}.jsonl")) for i in range(8) - ] - assert sum(baseline_counts) == 356 - assert max(baseline_counts) - min(baseline_counts) == 1 - - value = json.loads((output / "value-pretraining-selection.json").read_text()) - assert value["selection"] == "all" - assert value["sample_count"] == 356 - assert value["actor_eval_task_count_exposed_to_critic"] == 45 - assert value["actor_eval_trajectory_count_exposed_to_critic"] == 180 - assert value["critic_pretraining_exposes_actor_eval_tasks"] is True - assert manifest["compaction"] == { - "enabled": True, - "trainer_objective": "sao", - "max_seq_len": 8192, - "trigger_semantics": "consumed-context-tokens-at-least", - "trigger_tokens": 6144, - "summary_max_tokens": 1024, - "max_compactions_per_episode": 3, - } - - topology = manifest["topology"] - assert topology["islands"] == topology["physical_gpus"] == 8 - assert topology["one_physical_gpu_per_island"] is True - assert topology["cross_island_collectives"] is False - for island in topology["island_contracts"]: - assert island["rollout_seed"] == 82621 + island["island_id"] - assert island["entrypoint"].endswith("train_sao_streaming_secrlenv.py") - assert island["runtime_contracts"]["actor_syncer_port"] == 29400 - assert island["runtime_contracts"]["critic_syncer_port"] == 29401 - assert island["openenv_agent_contract"] == { - "custom_agent_function_path": ( - "yeto.rl.harness.codex." - "codex_openenv_subprocess_agent_function.run" - ), - "custom_rm_path": "openenv_generate.reward_func", - "dynamic_sampling_filter_path": ( - "openenv_generate.check_terminal_bench_episode" - ), - "input_key": "messages", - "tito_model": "qwen35", - "codex_backend_profile": "qwen35_08b", - "model": "Qwen/Qwen3.5-0.8B", - "model_revision": "2fc06364715b967f1860aea9cf38778875588b17", - "max_seq_len": 8192, - "must_not_enter_secrlenv_retry_path": True, - } - assert ( - island["openenv_agent_contract"]["custom_agent_function_path"] - not in SECRLENV_AGENTS - ) - assert island["ray_fractional_gpu"] == { - "actor": 0.4, - "critic": 0.4, - "sglang": 0.2, - } - args = island["miles_topology_args"] - assert "--sao-one-gpu-island" in args - assert "--sao-compaction" in args - assert args[args.index("--input-key") + 1] == "messages" - assert "--colocate" in args - assert args[args.index("--sglang-mem-fraction-static") + 1] == "0.15" - assert args[args.index("--sglang-max-total-tokens") + 1] == "393216" - assert args[args.index("--sglang-max-mamba-cache-size") + 1] == "256" - environment = island["environment"] - assert environment["SECRLENV_MAX_ROLLOUT_TIME_SECONDS"] == "1800" - assert environment["OPENENV_MAX_ROLLOUT_TIME_SECONDS"] == "1800" - assert environment["YETO_CODEX_COMPACTION_ENABLED"] == "1" - assert environment["YETO_CODEX_COMPACTION_TRIGGER_TOKENS"] == "6144" - assert environment["YETO_CODEX_COMPACTION_SUMMARY_MAX_TOKENS"] == "1024" - assert environment["YETO_CODEX_MAX_COMPACTIONS"] == "3" - assert all( - environment[name] == value - for name, value in CODEX_OPENENV_IDENTITY_ENV.items() - ) - assert "PYTORCH_CUDA_ALLOC_CONF" not in environment - assert ( - manifest["required_gates"]["openenv_tbench_max_active_episodes_at_least"] == 304 - ) - assert ( - manifest["required_gates"]["openenv_compaction_trajectory_evidence_v2"] is True - ) - assert "terminal_bench_signed_outcome_hmac_key_file" not in manifest["required_gates"] - assert manifest["required_gates"]["direct_codex_qwen35_08b_preflight"] is True - assert manifest["launch_readiness"]["ready"] is False - assert [ - issue["code"] for issue in manifest["launch_readiness"]["blocking_issues"] - ] == [ - "task-images-not-attested", - "openenv-capacity-not-attested", - "one-gpu-island-runtime-not-smoked", - "final-layout-attestations-not-generated", - ] - assert manifest["integration_notes"] == [ - { - "code": "streaming-entrypoint-retains-legacy-cli-name", - "blocking": False, - "detail": manifest["integration_notes"][0]["detail"], - } - ] - - -def test_split_is_reproducible(tmp_path): - tasks = _task_tree(tmp_path / "source") - first = plan.build_plan( - tasks, - tmp_path / "first", - seed="fixed-seed", - run_root="/run/one", - ) - second = plan.build_plan( - tasks, - tmp_path / "second", - seed="fixed-seed", - run_root="/run/two", - ) - one = json.loads(first.read_text())["split"] - two = json.loads(second.read_text())["split"] - assert one == two - - -def test_rejects_incomplete_suite(tmp_path): - tasks = _task_tree(tmp_path / "source", count=88) - with pytest.raises(plan.PlanError, match="exactly 89"): - plan.build_plan( - tasks, - tmp_path / "output", - seed="fixed-seed", - run_root="/run/tbench21", - ) diff --git a/tools/build_eval_data.py b/tools/build_eval_data.py deleted file mode 100644 index f88270dfe..000000000 --- a/tools/build_eval_data.py +++ /dev/null @@ -1,124 +0,0 @@ -#!/usr/bin/env python3 -"""TB2 eval / train jsonl from the hold-out list (rl-eval-difficulty-buckets 1.1 second half, 1.3). - - python tools/build_eval_data.py --tb2-tasks-dir ~/work/tb2-data \ - --holdout data/eval/tb2-holdout.json --out-dir data - -Writes -* ``/eval/tb2-holdout-eval.jsonl`` -- the 30 hold-out tasks (eval only); -* ``/tb2/tb2-train.jsonl`` -- every other TB2 task except the unusable table - ``data/eval/tb2-unusable.json`` (S17 N13; 46 at tb2@2fd12b8 = 89 - 30 hold-out - 13 - unusable; the S15 smoke-6 included: they were excluded from the eval pool, not from training); -* ``/eval/tb2-data.sha256.json`` -- sha256 of the hold-out list and both files - (pin these in the run config). - -Row format = design D6.a: the smoke-6 system prompt (the task statement itself -comes from the task's ``instruction.md`` at rollout time, ``task_prompt``) and -``metadata`` with ``task_id``, ``benchmark``, ``benchmark_version``, -``difficulty``, ``difficulty_source`` and, on eval rows, ``eval_bucket``. -Deterministic: rows sorted by ``task_id``; the same inputs give the same bytes. -""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import sys -from pathlib import Path -from typing import Any - -sys.path.insert(0, str(Path(__file__).resolve().parents[1])) - -SYSTEM_PROMPT = ( - "You are an autonomous terminal agent solving a Terminal-Bench task. You will be given the task " - "instruction, then interact with a real Linux shell. On each turn respond with EXACTLY ONE shell " - "command inside a single ```bash code block and nothing else. Inspect the environment, make the " - "required changes, and verify your work. When you are confident the task is fully complete, reply " - "with TASK_COMPLETE (with no code block)." -) -EVAL_JSONL = "eval/tb2-holdout-eval.jsonl" -TRAIN_JSONL = "tb2/tb2-train.jsonl" -SHA_JSON = "eval/tb2-data.sha256.json" - - -def _row(spec: Any, *, with_bucket: bool) -> dict[str, Any]: - meta = {"task_id": spec.task_id, "benchmark": spec.benchmark, "benchmark_version": spec.benchmark_version, - "difficulty": spec.difficulty, "difficulty_source": spec.difficulty_source} - if with_bucket: - if not spec.eval_bucket: - raise ValueError(f"{spec.task_id}: hold-out task without an eval bucket") - meta["eval_bucket"] = spec.eval_bucket - return {"prompt": [{"role": "system", "content": SYSTEM_PROMPT}], "metadata": meta} - - -def _jsonl(rows: list[dict[str, Any]]) -> bytes: - return "".join(json.dumps(r, sort_keys=True, ensure_ascii=False) + "\n" for r in rows).encode() - - -def build(tasks_dir: str | Path, holdout: dict[str, Any], *, - unusable: dict[str, str] | None = None) -> dict[str, bytes]: - """{relative path: bytes} for the eval jsonl, the train jsonl and the sha256 record.""" - from yeto.rl.harness.reward_env import benchmark as bm - from yeto.rl.harness.reward_env import tb2 - - adapter = tb2.Tb2Benchmark(tasks_dir) - if holdout.get("benchmark_version") != adapter.version: - raise ValueError(f"hold-out list is for {holdout.get('benchmark_version')}, checkout is {adapter.version}") - held = bm.holdout_ids(holdout) - every = adapter.task_ids() - missing = sorted(set(held) - set(every)) - if missing: - raise ValueError(f"hold-out tasks not in the checkout: {missing}") - by_item = {i["task_id"]: i for i in holdout["items"]} - eval_rows = [] - for tid in sorted(held): - spec = adapter.task_spec(tid) - if spec.eval_bucket != by_item[tid]["eval_bucket"]: - raise ValueError(f"{tid}: bucket {spec.eval_bucket} != hold-out list {by_item[tid]['eval_bucket']}") - eval_rows.append(_row(spec, with_bucket=True)) - unusable = {} if unusable is None else unusable - unknown = sorted(set(unusable) - set(every)) - if unknown: - raise ValueError(f"unusable tasks not in the checkout: {unknown}") - clash = sorted(set(unusable) & set(held)) - if clash: - raise ValueError(f"unusable tasks in the hold-out list (rebuild it with tools/reward_env/holdout.py): {clash}") - train_rows = [_row(adapter.task_spec(t), with_bucket=False) for t in every - if t not in set(held) and t not in unusable] - bm.assert_disjoint([r["metadata"]["task_id"] for r in train_rows], held) - eval_bytes, train_bytes = _jsonl(eval_rows), _jsonl(train_rows) - record = { - "benchmark_version": adapter.version, - "holdout_sha256": bm.holdout_sha256(holdout), - "unusable_excluded": sorted(unusable), - EVAL_JSONL: {"rows": len(eval_rows), "sha256": hashlib.sha256(eval_bytes).hexdigest()}, - TRAIN_JSONL: {"rows": len(train_rows), "sha256": hashlib.sha256(train_bytes).hexdigest()}, - } - return {EVAL_JSONL: eval_bytes, TRAIN_JSONL: train_bytes, - SHA_JSON: (json.dumps(record, indent=1, sort_keys=True) + "\n").encode()} - - -def main(argv: list[str] | None = None) -> int: - p = argparse.ArgumentParser() - p.add_argument("--tb2-tasks-dir", required=True) - p.add_argument("--holdout", required=True) - p.add_argument("--out-dir", required=True) - p.add_argument("--tb2-unusable", default=None, - help="unusable-task table (default: data/eval/tb2-unusable.json); its tasks leave training") - p.add_argument("--tb2-no-unusable", action="store_true", help="do not apply the unusable-task table") - a = p.parse_args(argv) - from yeto.rl.harness.reward_env import tb2 - - unusable = {} if a.tb2_no_unusable else tb2.load_unusable(a.tb2_unusable) - out = Path(a.out_dir) - for rel, data in build(a.tb2_tasks_dir, json.loads(Path(a.holdout).read_text()), - unusable=unusable).items(): - (out / rel).parent.mkdir(parents=True, exist_ok=True) - (out / rel).write_bytes(data) - print(rel, len(data), hashlib.sha256(data).hexdigest()[:12]) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/tools/probes/build_tbench21_sao_diloco_plan.py b/tools/probes/build_tbench21_sao_diloco_plan.py deleted file mode 100755 index fd9d7759a..000000000 --- a/tools/probes/build_tbench21_sao_diloco_plan.py +++ /dev/null @@ -1,520 +0,0 @@ -#!/usr/bin/env python3 -"""Build the immutable Terminal-Bench 2.1 SAO/DiLoCo run inputs. - -This tool only prepares data and a launch contract. It never starts Docker, -Ray, a model server, or a training process. The resulting plan intentionally -records that value pretraining consumes all four baseline trajectories from -all tasks, including the actor-evaluation half, while actor RL sees only the -44-task training split. -""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import os -import subprocess -from collections import Counter -from collections.abc import Iterable -from pathlib import Path -from typing import Any - -import tomllib - -from yeto.rl import CODEX_OPENENV_AGENT, CODEX_OPENENV_IDENTITY_ENV -from yeto.rl.codex_backend import QWEN35_08B_MODEL, QWEN35_08B_REVISION - -SCHEMA = "yeto.tbench21-sao-diloco-plan.v1" -SPLIT_DOMAIN = b"yeto-tbench21-sao-split-v1\0" -ROLLOUTS_PER_TASK = 4 -EPISODE_TIMEOUT_SECONDS = 1800 -ISLANDS = 8 -TARGET_CONCURRENCY = 304 -PER_ISLAND_CONCURRENCY = 38 -SGLANG_MEM_FRACTION_STATIC = 0.15 -SGLANG_MAX_TOTAL_TOKENS = 393216 -SGLANG_MAX_MAMBA_CACHE_SIZE = 256 -MAX_SEQ_LEN = 8192 -COMPACTION_TRIGGER_TOKENS = 6144 -COMPACTION_SUMMARY_MAX_TOKENS = 1024 -MAX_COMPACTIONS = 3 -ROLLOUT_SEED_BASE = 82621 -MIN_H200_MEMORY_BYTES = 120 * 1024**3 -SYSTEM_MESSAGE = ( - "The authenticated Terminal-Bench daemon supplies the immutable task " - "instruction. Use the Codex terminal tools to solve it and submit exactly once." -) - - -class PlanError(RuntimeError): - pass - - -def _sha256_bytes(value: bytes) -> str: - return hashlib.sha256(value).hexdigest() - - -def _sha256_file(path: Path) -> str: - digest = hashlib.sha256() - with path.open("rb") as handle: - for block in iter(lambda: handle.read(1024 * 1024), b""): - digest.update(block) - return digest.hexdigest() - - -def _canonical_bytes(value: object) -> bytes: - return json.dumps( - value, sort_keys=True, separators=(",", ":"), ensure_ascii=False - ).encode("utf-8") - - -def _write_json(path: Path, value: object) -> None: - path.write_bytes(_canonical_bytes(value) + b"\n") - - -def _write_jsonl(path: Path, rows: Iterable[dict[str, Any]]) -> None: - with path.open("wb") as handle: - for row in rows: - handle.write(_canonical_bytes(row) + b"\n") - - -def _git_revision(root: Path) -> str | None: - try: - value = subprocess.run( - [ - "git", - "-c", - f"safe.directory={root}", - "-C", - str(root), - "rev-parse", - "HEAD", - ], - check=True, - stdout=subprocess.PIPE, - stderr=subprocess.DEVNULL, - text=True, - ).stdout.strip() - except (OSError, subprocess.CalledProcessError): - return None - return value if len(value) == 40 else None - - -def _load_tasks(tasks_dir: Path) -> list[dict[str, Any]]: - if not tasks_dir.is_dir() or tasks_dir.is_symlink(): - raise PlanError("tasks directory must be a real directory") - tasks: list[dict[str, Any]] = [] - for task_dir in sorted(tasks_dir.iterdir(), key=lambda path: path.name): - if not task_dir.is_dir() or task_dir.is_symlink(): - continue - task_toml = task_dir / "task.toml" - instruction = task_dir / "instruction.md" - dockerfile = task_dir / "environment" / "Dockerfile" - if ( - not task_toml.is_file() - or not instruction.is_file() - or not dockerfile.is_file() - ): - raise PlanError(f"task {task_dir.name!r} is incomplete") - try: - raw = tomllib.loads(task_toml.read_text(encoding="utf-8")) - except (OSError, UnicodeError, tomllib.TOMLDecodeError) as error: - raise PlanError(f"task {task_dir.name!r} has invalid task.toml") from error - metadata = raw.get("metadata", {}) - environment = raw.get("environment", {}) - task = raw.get("task", {}) - if not all(isinstance(value, dict) for value in (metadata, environment, task)): - raise PlanError(f"task {task_dir.name!r} metadata is malformed") - docker_image = environment.get("docker_image") - if not isinstance(docker_image, str) or not docker_image: - raise PlanError(f"task {task_dir.name!r} has no Docker image identity") - category = metadata.get("category", "unknown") - if not isinstance(category, str) or not category: - raise PlanError(f"task {task_dir.name!r} has no category") - tasks.append( - { - "task_id": task_dir.name, - "task_name": task.get("name"), - "category": category, - "docker_image": docker_image, - "cpus": environment.get("cpus"), - "memory_mb": environment.get("memory_mb"), - "storage_mb": environment.get("storage_mb"), - "task_toml_sha256": _sha256_file(task_toml), - "instruction_sha256": _sha256_file(instruction), - "dockerfile_sha256": _sha256_file(dockerfile), - } - ) - if len(tasks) != 89: - raise PlanError( - f"Terminal-Bench 2.1 must contain exactly 89 tasks, got {len(tasks)}" - ) - ids = [task["task_id"] for task in tasks] - if len(set(ids)) != len(ids): - raise PlanError("Terminal-Bench task IDs are not unique") - return tasks - - -def _split_tasks( - tasks: list[dict[str, Any]], seed: str -) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]: - """Return an exact deterministic 44-train/45-eval partition.""" - - if not seed: - raise PlanError("split seed must not be empty") - - def key(task: dict[str, Any]) -> tuple[bytes, str]: - payload = ( - SPLIT_DOMAIN - + seed.encode("utf-8") - + b"\0" - + task["task_id"].encode("utf-8") - ) - return hashlib.sha256(payload).digest(), task["task_id"] - - ranked = sorted(tasks, key=key) - return ranked[:44], ranked[44:] - - -def _rollout_rows(tasks: list[dict[str, Any]], *, phase: str) -> list[dict[str, Any]]: - rows: list[dict[str, Any]] = [] - for task_index, task in enumerate(tasks): - for replica in range(ROLLOUTS_PER_TASK): - island_id = (task_index * ROLLOUTS_PER_TASK + replica) % ISLANDS - rollout_seed = ROLLOUT_SEED_BASE + island_id - sample_id = f"{phase}:{task['task_id']}:r{replica}" - rows.append( - { - "messages": [{"role": "system", "content": SYSTEM_MESSAGE}], - "metadata": { - "task_id": task["task_id"], - "prompt_tier": "l2", - "max_seq_len": MAX_SEQ_LEN, - "split": phase, - "rollout_replica": replica, - "sample_id": sample_id, - "island_id": island_id, - "rollout_seed": rollout_seed, - "episode_timeout_seconds": EPISODE_TIMEOUT_SECONDS, - }, - } - ) - return rows - - -def _island_contract(island_id: int, run_root: str) -> dict[str, Any]: - base = 32000 + island_id * 100 - return { - "island_id": island_id, - "physical_gpu_index": island_id, - "cuda_visible_devices": str(island_id), - "rollout_seed": ROLLOUT_SEED_BASE + island_id, - "run_dir": f"{run_root}/island-{island_id}", - "entrypoint": "/root/miles/tools/probes/train_sao_streaming_secrlenv.py", - "runtime_contracts": { - "sao_context": f"{run_root}/island-{island_id}/sao-context.json", - "streaming_context": f"{run_root}/island-{island_id}/sao-streaming-context.json", - "actor_syncer_port": 29400, - "critic_syncer_port": 29401, - }, - "openenv_agent_contract": { - "custom_agent_function_path": CODEX_OPENENV_AGENT, - "custom_rm_path": "openenv_generate.reward_func", - "dynamic_sampling_filter_path": ( - "openenv_generate.check_terminal_bench_episode" - ), - "input_key": "messages", - "tito_model": "qwen35", - "codex_backend_profile": "qwen35_08b", - "model": QWEN35_08B_MODEL, - "model_revision": QWEN35_08B_REVISION, - "max_seq_len": MAX_SEQ_LEN, - "must_not_enter_secrlenv_retry_path": True, - }, - "ray_gcs_port": 26000 + island_id, - "ports": { - "session_server": 31000 + island_id, - "rollout_engine_base": base, - "sglang_router": base + 20, - "sglang_router_prometheus": base + 21, - "train_master_base": base + 40, - }, - "environment": { - "SECRLENV_MAX_ROLLOUT_TIME_SECONDS": str(EPISODE_TIMEOUT_SECONDS), - "OPENENV_MAX_ROLLOUT_TIME_SECONDS": str(EPISODE_TIMEOUT_SECONDS), - "YETO_CODEX_COMPACTION_ENABLED": "1", - "YETO_CODEX_COMPACTION_TRIGGER_TOKENS": str(COMPACTION_TRIGGER_TOKENS), - "YETO_CODEX_COMPACTION_SUMMARY_MAX_TOKENS": str( - COMPACTION_SUMMARY_MAX_TOKENS - ), - "YETO_CODEX_MAX_COMPACTIONS": str(MAX_COMPACTIONS), - **CODEX_OPENENV_IDENTITY_ENV, - "NCCL_NVLS_ENABLE": "0", - "RAY_DEDUP_LOGS": "0", - }, - "miles_topology_args": [ - "--sao-one-gpu-island", - "--sao-compaction", - "--input-key", - "messages", - "--colocate", - "--num-gpus-per-node", - "1", - "--actor-num-nodes", - "1", - "--actor-num-gpus-per-node", - "1", - "--critic-num-nodes", - "1", - "--critic-num-gpus-per-node", - "1", - "--rollout-num-gpus", - "1", - "--rollout-num-gpus-per-engine", - "1", - "--tensor-model-parallel-size", - "1", - "--pipeline-model-parallel-size", - "1", - "--context-parallel-size", - "1", - "--expert-model-parallel-size", - "1", - "--sglang-server-concurrency", - str(PER_ISLAND_CONCURRENCY), - "--sglang-max-running-requests", - str(PER_ISLAND_CONCURRENCY), - "--async-max-concurrent-samples", - str(PER_ISLAND_CONCURRENCY), - "--sglang-mem-fraction-static", - str(SGLANG_MEM_FRACTION_STATIC), - "--sglang-max-total-tokens", - str(SGLANG_MAX_TOTAL_TOKENS), - "--sglang-max-mamba-cache-size", - str(SGLANG_MAX_MAMBA_CACHE_SIZE), - ], - "ray_fractional_gpu": {"actor": 0.4, "critic": 0.4, "sglang": 0.2}, - } - - -def _category_counts(tasks: list[dict[str, Any]]) -> dict[str, int]: - return dict(sorted(Counter(task["category"] for task in tasks).items())) - - -def build_plan(tasks_dir: Path, output_dir: Path, *, seed: str, run_root: str) -> Path: - if output_dir.exists() or output_dir.is_symlink(): - raise PlanError("output directory must be fresh") - if not os.path.isabs(run_root) or ".." in Path(run_root).parts: - raise PlanError("run root must be an absolute safe path") - - tasks = _load_tasks(tasks_dir) - train_tasks, eval_tasks = _split_tasks(tasks, seed) - baseline_tasks = sorted(tasks, key=lambda task: task["task_id"]) - rows_by_phase = { - "baseline": _rollout_rows(baseline_tasks, phase="baseline"), - "train": _rollout_rows(train_tasks, phase="train"), - "eval": _rollout_rows(eval_tasks, phase="eval"), - } - if [len(rows_by_phase[name]) for name in ("baseline", "train", "eval")] != [ - 356, - 176, - 180, - ]: - raise AssertionError("Terminal-Bench rollout cardinality changed") - - output_dir.mkdir(mode=0o755, parents=True) - files: dict[str, str] = {} - for phase, rows in rows_by_phase.items(): - phase_dir = output_dir / phase - phase_dir.mkdir() - all_path = phase_dir / "all.jsonl" - _write_jsonl(all_path, rows) - files[str(all_path.relative_to(output_dir))] = _sha256_file(all_path) - for island_id in range(ISLANDS): - island_path = phase_dir / f"island-{island_id}.jsonl" - _write_jsonl( - island_path, - (row for row in rows if row["metadata"]["island_id"] == island_id), - ) - files[str(island_path.relative_to(output_dir))] = _sha256_file(island_path) - - baseline_ids = [row["metadata"]["sample_id"] for row in rows_by_phase["baseline"]] - value_selection = { - "schema": "yeto.sao-value-pretraining-selection.v1", - "source_phase": "baseline", - "selection": "all", - "sample_count": len(baseline_ids), - "task_count": len(tasks), - "rollouts_per_task": ROLLOUTS_PER_TASK, - "baseline_sample_ids": baseline_ids, - "critic_pretraining_exposes_actor_eval_tasks": True, - "actor_eval_task_count_exposed_to_critic": len(eval_tasks), - "actor_eval_trajectory_count_exposed_to_critic": len(eval_tasks) - * ROLLOUTS_PER_TASK, - } - value_path = output_dir / "value-pretraining-selection.json" - _write_json(value_path, value_selection) - files[value_path.name] = _sha256_file(value_path) - - task_contracts = { - task["task_id"]: {key: value for key, value in task.items() if key != "task_id"} - for task in sorted(tasks, key=lambda item: item["task_id"]) - } - inventory_sha256 = _sha256_bytes(_canonical_bytes(task_contracts)) - split_payload = { - "seed": seed, - "algorithm": "sha256-domain-ranked-first-44", - "train_task_ids": [task["task_id"] for task in train_tasks], - "eval_task_ids": [task["task_id"] for task in eval_tasks], - } - split_sha256 = _sha256_bytes(_canonical_bytes(split_payload)) - - manifest = { - "schema": SCHEMA, - "terminal_bench": { - "version": "2.1", - "revision": _git_revision(tasks_dir.parent), - "task_count": len(tasks), - "task_inventory_sha256": inventory_sha256, - "task_contracts": task_contracts, - }, - "split": { - **split_payload, - "sha256": split_sha256, - "train_task_count": len(train_tasks), - "eval_task_count": len(eval_tasks), - "train_category_counts": _category_counts(train_tasks), - "eval_category_counts": _category_counts(eval_tasks), - }, - "rollouts": { - "per_task": ROLLOUTS_PER_TASK, - "baseline": 356, - "train": 176, - "eval": 180, - "episode_timeout_seconds": EPISODE_TIMEOUT_SECONDS, - "target_concurrency": TARGET_CONCURRENCY, - "per_island_concurrency": PER_ISLAND_CONCURRENCY, - "seed_base": ROLLOUT_SEED_BASE, - "per_island_seeds": [ - ROLLOUT_SEED_BASE + island_id for island_id in range(ISLANDS) - ], - "baseline_achievable_concurrency": TARGET_CONCURRENCY, - "train_cardinality_limited_concurrency": 176, - "eval_cardinality_limited_concurrency": 180, - }, - "value_pretraining": { - "selection_path": value_path.name, - "selection_sha256": files[value_path.name], - "uses_all_baseline_trajectories": True, - "sample_count": 356, - "critic_pretraining_exposes_actor_eval_tasks": True, - }, - "compaction": { - "enabled": True, - "trainer_objective": "sao", - "max_seq_len": MAX_SEQ_LEN, - "trigger_semantics": "consumed-context-tokens-at-least", - "trigger_tokens": COMPACTION_TRIGGER_TOKENS, - "summary_max_tokens": COMPACTION_SUMMARY_MAX_TOKENS, - "max_compactions_per_episode": MAX_COMPACTIONS, - }, - "topology": { - "islands": ISLANDS, - "physical_gpus": ISLANDS, - "one_physical_gpu_per_island": True, - "model": QWEN35_08B_MODEL, - "model_revision": QWEN35_08B_REVISION, - "minimum_gpu_memory_bytes": MIN_H200_MEMORY_BYTES, - "cross_island_collectives": False, - "synchronization": "dual-role streaming DiLoCo", - "syncer_sessions": {"actor_port": 29400, "critic_port": 29401}, - "island_contracts": [ - _island_contract(island_id, run_root) for island_id in range(ISLANDS) - ], - }, - "required_gates": { - "codex_harness": True, - "compaction": True, - "all_task_images_available": True, - "openenv_tbench_max_active_episodes_at_least": TARGET_CONCURRENCY, - "openenv_compaction_trajectory_evidence_v2": True, - "direct_codex_qwen35_08b_preflight": True, - "host_open_file_limit_at_least": 65536, - "fresh_per_island_ray_clusters": True, - "actor_and_critic_streaming_syncers_healthy": True, - "layout_attestation_per_role": True, - }, - "launch_readiness": { - "ready": False, - "blocking_issues": [ - { - "code": "task-images-not-attested", - "detail": ( - "All 89 immutable Terminal-Bench task images must be present " - "and capacity-preflighted before the baseline or online phase." - ), - }, - { - "code": "openenv-capacity-not-attested", - "detail": ( - "The trusted OpenEnv service has not yet attested at least 304 " - "simultaneous Terminal-Bench episodes on this host." - ), - }, - { - "code": "one-gpu-island-runtime-not-smoked", - "detail": ( - "Actor, critic, and SGLang fractional placement is validated " - "fail-closed but still needs a physical one-GPU memory/runtime " - "smoke before the eight-island launch." - ), - }, - { - "code": "final-layout-attestations-not-generated", - "detail": ( - "Each island still needs its final actor/critic layout probe " - "and hash-bound streaming runtime after fragment count P is " - "discovered." - ), - }, - ], - }, - "integration_notes": [ - { - "code": "streaming-entrypoint-retains-legacy-cli-name", - "blocking": False, - "detail": ( - "The retained entrypoint and flag names mention SecRLEnv for " - "compatibility, but the v2 base context binds the benchmark as " - "terminal-bench-2.1 without synthetic SecRLEnv fields. Direct " - "preflight attests the exact Codex/OpenEnv Qwen3.5-0.8B adapter " - "before Miles parses arguments or allocates a GPU." - ), - } - ], - "files": dict(sorted(files.items())), - } - manifest_path = output_dir / "manifest.json" - _write_json(manifest_path, manifest) - return manifest_path - - -def main(argv: list[str] | None = None) -> None: - parser = argparse.ArgumentParser() - parser.add_argument("--tasks-dir", type=Path, required=True) - parser.add_argument("--output-dir", type=Path, required=True) - parser.add_argument("--split-seed", default="tbench21-sao-20260826") - parser.add_argument("--run-root", default="/data/sft/tbench21-sao-diloco-full") - args = parser.parse_args(argv) - manifest = build_plan( - args.tasks_dir.resolve(), - args.output_dir.resolve(), - seed=args.split_seed, - run_root=args.run_root, - ) - print(manifest) - - -if __name__ == "__main__": - main() diff --git a/tools/probes/s17_eval_island_gpu_smoke.py b/tools/probes/s17_eval_island_gpu_smoke.py deleted file mode 100644 index 953b662ae..000000000 --- a/tools/probes/s17_eval_island_gpu_smoke.py +++ /dev/null @@ -1,85 +0,0 @@ -"""(codex-env.json in OUT: yeto.launcher.codex_container_env(codex_bundle_contract(args, /codex)) for qwen35_08b / 4096 / xhigh.) -S17 N11 eval-island GPU smoke: real model (Qwen3.5-0.8B, codex qwen35_08b) on TB2 hold-out 3 tasks. - -seed (Modal CPU, same image): version 0 into Volume through the training export path -island (Modal H100!:1): SGLang + Miles session server + codex worker -> TB2 sandboxes -> judge. -Run with reward-env-venv python (modal 1.6.1). Raw output stays in this directory. -""" -import json, os, sys, time, tomllib, threading -from pathlib import Path - -REPO = Path("/home/michael/work/s17-eval-infer") -sys.path.insert(0, str(REPO)) -OUT = Path(os.environ.get("N11_OUT", "s1-runs/s17-n11-evalinfer")).resolve(); OUT.mkdir(parents=True, exist_ok=True) -APP = "yeto-s17-n11-evalinfer" -VOLUME = os.environ.get("N11_VOLUME", "yeto-eval-store-n11") -TASKS = ["prove-plus-comm", "constraints-scheduling", "largest-eigenval"] -MODEL, REV = "Qwen/Qwen3.5-0.8B", "2fc06364715b967f1860aea9cf38778875588b17" -HARD_S = int(os.environ.get("N11_HARD_S", "2400")) -SAMPLING = {"rollout_temperature": 1.0, "rollout_top_p": 1.0, "rollout_top_k": None, - "rollout_max_response_len": 4096, "rollout_max_context_len": 8192, - "yeto_codex_reasoning_effort": "xhigh", "yeto_codex_max_turns": "12"} - -import modal -from yeto.cloud import modal_eval_island as mei - -log = open(OUT / "smoke.log", "a") -def say(*a): - line = " ".join(str(x) for x in a) - print(time.strftime("%H:%M:%SZ", time.gmtime()), line, file=log, flush=True) - print(line, flush=True) - -prof = tomllib.loads(Path("~/.modal.toml").expanduser().read_text()) -prof = next(v for v in prof.values() if isinstance(v, dict) and v.get("active")) if any( - isinstance(v, dict) and v.get("active") for v in prof.values()) else next(iter(prof.values())) -envs = json.loads((OUT / "codex-env.json").read_text()) -envs.update({"SECRLENV_MAX_TURNS": "12", "YETO_HARNESS_TB2_LEASE_SECONDS": "600", - "YETO_CODEX_WORKER_STDERR": "1", - "MODAL_TOKEN_ID": prof["token_id"], "MODAL_TOKEN_SECRET": prof["token_secret"]}) -spec = mei.EvalFunctionSpec(app_name=APP, volume=VOLUME, workdir=str(REPO), - codex_dir="/home/michael/work/codex-bundle/codex", tb2_dir="/home/michael/work/tb2-data", - gpu="H100!", gpus=1, cpu=8.0, memory_mib=65536, timeout_s=HARD_S, envs=envs) -app, island_fn, seed_fn = mei.build_eval_app(spec) - -config = { - "store": mei.EVAL_STORE_MOUNT, "volume": VOLUME, "island_id": "eval-n11", - "holdout": f"{mei.WORKDIR_MOUNT}/data/eval/tb2-holdout.json", - "eval_data": f"{mei.WORKDIR_MOUNT}/data/eval/tb2-holdout-eval.jsonl", - "task_ids": TASKS, "trials_v0": int(os.environ.get("N11_TRIALS", "1")), "trials": 1, "assert_gpu_name": "H100", - "loader": "yeto.rl.eval.codex_infer:loader_factory", "attempt": "yeto.rl.eval.codex_infer:attempt_factory", - "infer": {"base_model": MODEL, "revision": REV, "tito_model": "qwen35", "context_length": 8192, - "max_tokens": 4096, "mem_fraction_static": 0.8}, -} -events = [] -def emit(event, **f): - events.append({"event": event, "ts": time.time(), **f}) - with open(OUT / "launcher-events.jsonl", "a") as fh: - fh.write(json.dumps(events[-1], default=str) + "\n") - -def watchdog(): - time.sleep(HARD_S + 300) - say("WATCHDOG: hard limit passed, stopping app") - os.system(f"/home/michael/work/reward-env-venv/bin/modal app stop {APP} >> {OUT}/watchdog.out 2>&1") - os._exit(3) -threading.Thread(target=watchdog, daemon=True).start() - -say("start", APP, "hard", HARD_S) -with modal.enable_output(): - with app.run(): - say("app run", app.app_id) - (OUT / "app_id.txt").write_text(str(app.app_id) + "\n") - t = time.time() - seed = seed_fn.remote({"volume": VOLUME, "store": mei.EVAL_STORE_MOUNT, "model": MODEL, "revision": REV, - "rank": 16, "targets": "attention", "sampling": SAMPLING, "seed": 0, - "training_finished": True}) - say("seed done", round(time.time() - t, 1), "s", json.dumps(seed, default=str)[:600]) - (OUT / "seed.json").write_text(json.dumps(seed, indent=1, default=str)) - launcher = mei.EvalIslandLauncher(client=mei.ModalFunctionClient(island_fn), emit=emit, config=config, - gpu="H100!", gpus=1, price_per_hour=None, max_starts=1) - try: - summary = launcher.run() - say("island done", json.dumps(summary, default=str)[:2000]) - (OUT / "summary.json").write_text(json.dumps(summary, indent=1, default=str)) - except Exception as exc: # noqa: BLE001 - say("island FAILED", type(exc).__name__, str(exc)[:2000]) -say("app exited") diff --git a/tools/reward_env/build_images.py b/tools/reward_env/build_images.py deleted file mode 100644 index 2988e7d06..000000000 --- a/tools/reward_env/build_images.py +++ /dev/null @@ -1,85 +0,0 @@ -#!/usr/bin/env python3 -"""Plan (default) or build prebaked reward-environment images on Modal (CPU only). - - # plan only: no network, no Modal; writes a JSON report - python tools/reward_env/build_images.py --benchmark tb2 --tasks-dir ~/work/tb2-data --out plan.json - - # build (needs a Modal token; costs CPU build time; needs approval first) - python tools/reward_env/build_images.py ... --build --app yeto-reward-env-build [--tasks a,b] - -``--build`` never targets the production app ``yeto-tbench2``. Each image is -built by Modal from ``Image.from_registry(base).run_commands()``; -Modal caches layers by content, so sandboxes later created from the same -definition (``yeto.cloud.modal_reward_env.PrebakedModalSandboxBackend``) reuse the build. -""" - -from __future__ import annotations - -import argparse -import json -import sys -import time -from pathlib import Path - -sys.path.insert(0, str(Path(__file__).resolve().parents[2])) - -from yeto.rl.harness.reward_env import benchmark as bm # noqa: E402 - - -def plan_report(adapter, task_ids): - rows = [] - for tid in task_ids: - plan = adapter.prebake_plan(tid) - rows.append({"task_id": tid, "base_image": plan.base_image, "digest": plan.digest(), - "complete": plan.complete, "commands": list(plan.commands), "skipped": list(plan.skipped)}) - return {"benchmark": adapter.name, "schema": bm.PREBAKE_SCHEMA, "tasks": rows, - "complete": sum(r["complete"] for r in rows), "total": len(rows)} - - -def build(adapter, task_ids, app_name): - import modal - - from yeto.cloud.modal_reward_env import modal_image - from yeto.rl.harness.reward_env.tb2 import check_build_app - - app = modal.App.lookup(check_build_app(app_name), create_if_missing=True) - results = [] - for tid in task_ids: - plan = adapter.prebake_plan(tid) - started = time.monotonic() - try: - with modal.enable_output(): - modal_image(plan).build(app) - results.append({"task_id": tid, "ok": True, "seconds": round(time.monotonic() - started, 1)}) - except Exception as exc: # noqa: BLE001 - report and continue - results.append({"task_id": tid, "ok": False, "error": f"{type(exc).__name__}: {exc}"[:500], - "seconds": round(time.monotonic() - started, 1)}) - print(json.dumps(results[-1]), flush=True) - return results - - -def main(argv=None) -> int: - p = argparse.ArgumentParser() - p.add_argument("--benchmark", default="tb2") - p.add_argument("--tasks-dir") - p.add_argument("--tasks", help="comma separated subset") - p.add_argument("--out") - p.add_argument("--build", action="store_true") - p.add_argument("--app") - a = p.parse_args(argv) - adapter = bm.get_adapter(a.benchmark, **({"tasks_dir": a.tasks_dir} if a.tasks_dir else {})) - ids = a.tasks.split(",") if a.tasks else adapter.task_ids() - report = plan_report(adapter, ids) - if a.build: - if not a.app: - p.error("--build needs an explicit --app (never yeto-tbench2)") - report["build"] = build(adapter, ids, a.app) - text = json.dumps(report, indent=2, ensure_ascii=False) - if a.out: - Path(a.out).write_text(text) - print(f"{report['complete']}/{report['total']} plans complete" if a.out else text) - return 0 if all(r.get("ok", True) for r in report.get("build", [])) else 1 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/tools/reward_env/holdout.py b/tools/reward_env/holdout.py index c9f5fcd9e..a271776c5 100644 --- a/tools/reward_env/holdout.py +++ b/tools/reward_env/holdout.py @@ -1,11 +1,11 @@ #!/usr/bin/env python3 """Write WP3 (rl-eval-difficulty-buckets D6b) hold-out lists for the built-in benchmarks. - python tools/reward_env/holdout.py --tb2-tasks-dir ~/work/tb2-data \ - --swev-data swev-78f471bf.parquet --out-dir data/eval + python tools/reward_env/holdout.py --swev-data swev-78f471bf.parquet --out-dir data/eval -TB2 writes tb2-holdout.json (eval) and tb2-train.json (training task list cut -against that hold-out). Tasks in data/eval/tb2-unusable.json leave both. +The TB2 lists (data/eval/tb2-holdout.json, tb2-train.json) were written by the +TB2 task loader of the self-written Codex harness, removed in S26 E0.5 (git tag +s26-pre-harness-removal has the generator). """ from __future__ import annotations @@ -22,56 +22,12 @@ def main(argv=None) -> int: p = argparse.ArgumentParser() - p.add_argument("--tb2-tasks-dir") p.add_argument("--swev-data") - p.add_argument("--tb2-exclude-jsonl", action="append", default=[], - help="dataset jsonl whose metadata.task_id rows leave the TB2 eval pool " - "(default: the built-in S15 smoke-6 list)") - p.add_argument("--tb2-exclude-reason", default=None, - help="reason recorded for --tb2-exclude-jsonl tasks (default: S15 smoke training)") - p.add_argument("--tb2-unusable", default=None, - help="unusable-task table (default: data/eval/tb2-unusable.json); its tasks leave " - "both the hold-out pool and the training list") - p.add_argument("--tb2-no-unusable", action="store_true", help="do not apply the unusable-task table") - p.add_argument("--tb2-no-exclude", action="store_true", - help="do not exclude any TB2 task (neither smoke-6 nor the unusable table)") p.add_argument("--out-dir", required=True) a = p.parse_args(argv) out = Path(a.out_dir) out.mkdir(parents=True, exist_ok=True) written = {} - if a.tb2_tasks_dir: - from yeto.rl.harness.reward_env import tb2 - - if a.tb2_no_exclude: - exclude = {} - elif a.tb2_exclude_jsonl: - reason = a.tb2_exclude_reason or tb2.SMOKE6_REASON - exclude = {} - for path in a.tb2_exclude_jsonl: - for line in Path(path).read_text().splitlines(): - if line.strip(): - tid = bm.task_id_of(json.loads(line)) - if tid is None: - raise SystemExit(f"{path}: row without metadata.task_id") - exclude[tid] = reason - else: - exclude = tb2.smoke6_exclusions() - unusable = {} if (a.tb2_no_exclude or a.tb2_no_unusable) else tb2.load_unusable(a.tb2_unusable) - clash = sorted(set(unusable) & set(exclude)) - if clash: - raise SystemExit(f"tasks both in the exclusion list and the unusable table: {clash}") - adapter = tb2.Tb2Benchmark(a.tb2_tasks_dir) - unknown = sorted(set(unusable) - set(adapter.task_ids())) - if unknown: - raise SystemExit(f"unusable tasks not in the TB2 checkout: {unknown}") - holdout = tb2.build_holdout(adapter, exclude={**exclude, **unusable}, - **({"rule": tb2.HOLDOUT_RULE_UNUSABLE} if unusable else {})) - unknown = sorted({e["task_id"] for e in holdout.get("excluded", [])} - set(adapter.task_ids())) - if unknown: - raise SystemExit(f"excluded tasks not in the TB2 checkout: {unknown}") - written["tb2-holdout.json"] = holdout - written["tb2-train.json"] = bm.build_train_split(adapter, holdout, exclude=unusable) if a.swev_data: from yeto.rl.harness.reward_env import swebench_verified as sv diff --git a/yeto/cli.py b/yeto/cli.py index 7c68497f8..d03d1b5a4 100644 --- a/yeto/cli.py +++ b/yeto/cli.py @@ -1256,10 +1256,6 @@ def _make_head_task(args, extra_mounts: dict | None = None): # The head's controller defines, deploys and spawns modal: islands # through the Modal SDK (yeto.modal_runner), so it needs the package. head_pip += ' && pip install -q "modal>=1.0"' - if getattr(args, "codex_bundle_dir", None): - # S17 M1: the head attests the Codex bundle and imports the harness - # preflight (aiohttp / dill) when it builds each island. - head_pip += " && pip install -q aiohttp dill" if getattr(args, "wandb", False): # The head tails the syncer's event tape into W&B (yeto.wandb_tape). head_pip += " && pip install -q wandb" @@ -1429,16 +1425,6 @@ def cmd_launch_head(args) -> int: args.rl_initial_adapter ) args.rl_initial_adapter = launcher.RL_HEAD_INITIAL_ADAPTER_PATH - # S17 M1: a signed Codex run needs its bundle and harness env on the head, - # which builds the islands (attested here first; fails before provisioning). - try: - codex_mounts, codex_secrets = launcher.codex_head_staging(args) - except ValueError as exc: - print(f"[yeto] {exc}", file=sys.stderr) - return 1 - if codex_mounts: - data_mounts.update(codex_mounts) - args.codex_bundle_dir = launcher.HEAD_CODEX_BUNDLE_PATH args_dict = _serializable_args(args) runs.create_run(name, args_dict) learner_names = launcher.learner_cluster_names(name, specs) @@ -1488,7 +1474,6 @@ def cmd_launch_head(args) -> int: # 0.20: the head's LocalSyncer and the islands it launches read the # island HMAC key from this secret env (elastic only; {} in legacy). secrets.update(launcher.island_hmac_secret(args)) - secrets.update(codex_secrets) # secret-handling-hardening: HF/W&B/CyberGym tokens ride as secrets too. envs, env_secrets = launcher.split_secret_envs(envs) secrets.update(env_secrets) diff --git a/yeto/cloud/modal_ckpt_store.py b/yeto/cloud/modal_ckpt_store.py index f7a4bf3fa..f98140865 100644 --- a/yeto/cloud/modal_ckpt_store.py +++ b/yeto/cloud/modal_ckpt_store.py @@ -2,7 +2,7 @@ (rl-resume-from-checkpoint). Moved out of ``yeto/rl/engine/resume.py`` (the neutral core must not import -``modal``; same pattern as ``yeto.cloud.modal_reward_env``). The core only knows +``modal``; same pattern as the other ``yeto.cloud.modal_*`` modules). The core only knows ``CheckpointStore`` and the ``YETO_RL_STORE_IMPL`` hook; the launcher exports ``YETO_RL_STORE_IMPL=yeto.cloud.modal_ckpt_store:store_from_env`` plus ``YETO_RL_STORE_MODAL_VOLUME=`` on a Modal island with a diff --git a/yeto/cloud/modal_eval_island.py b/yeto/cloud/modal_eval_island.py index 2e9fe3972..08845ab4e 100644 --- a/yeto/cloud/modal_eval_island.py +++ b/yeto/cloud/modal_eval_island.py @@ -253,7 +253,6 @@ def timed_commit() -> None: # --- the Modal GPU function (5.10) ------------------------------------------------- EVAL_STORE_MOUNT = "/mnt/yeto-eval-store" # the eval store Volume inside the eval island -CODEX_MOUNT = "/opt/yeto/codex" # yeto.rl.CODEX_CONTAINER_DIR (signed codex bundle) TB2_MOUNT = "/root/tb2-data" # terminal-bench-2 checkout (YETO_HARNESS_TB2_TASKS_DIR) WORKDIR_MOUNT = "/root/sky_workdir" # yeto source, same place as the training islands IMAGE_PYTHONPATH = "/root/miles:/root/sglang/python:/root/sky_workdir" @@ -270,13 +269,12 @@ def default_image_ref() -> str: @dataclass class EvalFunctionSpec: """Shape of the eval-island Modal function. ``gpu`` is a Modal GPU string - (``H100!`` = exactly H100); ``envs`` become one Modal secret (codex - contract env, Modal token for the judge sandboxes, TB2 knobs).""" + (``H100!`` = exactly H100); ``envs`` become one Modal secret (Modal + token for the judge sandboxes, TB2 knobs).""" app_name: str volume: str workdir: str - codex_dir: str tb2_dir: str gpu: str = "H100!" gpus: int = 1 @@ -315,7 +313,7 @@ def _gpu_sampler(path: Path, stop: Any, interval_s: float = 5.0) -> None: def _image_paths(environ: Any = None) -> None: """Miles / SGLang / yeto source on ``sys.path``; subprocesses (SGLang, session - server, codex worker) get exactly the training island's ``PYTHONPATH``, without + server) get exactly the training island's ``PYTHONPATH``, without Modal's runtime copies (their protobuf/grpclib would shadow the image's).""" import sys @@ -416,7 +414,6 @@ def build_eval_app(spec: EvalFunctionSpec) -> tuple[Any, Any, Any]: .env({"HOME": "/root", "PYTHONUNBUFFERED": "1"}) .add_local_dir(spec.workdir, WORKDIR_MOUNT, copy=False, ignore=[".git", ".venv", ".claude", "**/__pycache__", "**/*.pyc", "s1-runs"]) - .add_local_dir(spec.codex_dir, CODEX_MOUNT, copy=False) .add_local_dir(spec.tb2_dir, TB2_MOUNT, copy=False, ignore=[".git"])) app = modal.App(spec.app_name) volume = modal.Volume.from_name(spec.volume, create_if_missing=True) diff --git a/yeto/cloud/modal_reward_env.py b/yeto/cloud/modal_reward_env.py deleted file mode 100644 index 104fbc687..000000000 --- a/yeto/cloud/modal_reward_env.py +++ /dev/null @@ -1,142 +0,0 @@ -"""Cloud layer: Modal sandbox backend for the reward environment's TB2 adapter. - -Moved out of ``yeto/rl/harness/reward_env/tb2.py`` (neutral core must not -import ``modal``; spec rl-framework-neutral-core). The neutral adapter only -knows ``tb2_provider.SandboxBackend`` / ``Tb2EnvironmentProvider``; this module -is the Modal implementation, injected at launch with -``YETO_HARNESS_ENVIRONMENT_PROVIDER=yeto.cloud.modal_reward_env:modal_provider``. -Behaviour is unchanged from the original location. -""" - -from __future__ import annotations - -import base64 -import os -from typing import Any - -from yeto.rl.harness.codex import tb2_provider as tb2 -from yeto.rl.harness.reward_env import tb2 as rt -from yeto.rl.harness.reward_env.benchmark import PrebakePlan -from yeto.rl.rlf.modal import RlfCtfConfig, RlfModalProvider, RlfModalVmConfig - - - -def modal_image(plan: PrebakePlan, *, prebake: bool = True) -> Any: - from modal import Image - - image = Image.from_registry(plan.base_image) - if prebake and not plan.empty: - image = image.run_commands(prebake_run_command(plan)) - return image - - -def prebake_run_command(plan: PrebakePlan) -> str: - """One Dockerfile ``RUN`` line for the multi-line prebake script. - - A Dockerfile instruction cannot span raw newlines (S17 G3: Modal rejected - ``bash -lc ''`` with "could not parse Dockerfile"), so the - script travels base64-encoded and runs as a login shell from a temp file. - """ - b64 = base64.b64encode(rt.prebake_script(plan).encode()).decode() - return (f"echo {b64} | base64 -d > /tmp/yeto-prebake.sh && bash -l /tmp/yeto-prebake.sh" - " && rm -f /tmp/yeto-prebake.sh") - - -def _prebake_enabled() -> bool: - return os.environ.get(rt.PREBAKE_ENV, "1") != "0" - - -class PrebakedModalSandboxBackend(tb2.ModalSandboxBackend): - """``tb2_provider.ModalSandboxBackend`` with the task's prebaked image.""" - - def __init__(self, *, prebake: bool = True, **kwargs: Any) -> None: - kwargs.setdefault("app_name", rt.DEFAULT_APP) - super().__init__(**kwargs) - self.prebake = prebake - - def create(self, task: tb2.Tb2Task, trajectory_id: str) -> tb2.ModalSandbox: - from modal import Sandbox - - plan = rt.prebake_from_test_sh((task.tests_dir / "test.sh").read_text(), task.docker_image) - tags = {"yeto-tb2-task": task.task_id, "yeto-trajectory": trajectory_id[:63], - "yeto-prebake": plan.digest()[:16] if self.prebake and not plan.empty else "none"} - if self.run_id: - tags["yeto-run-id"] = self.run_id[:63] - sandbox = Sandbox.create( - "sleep", "infinity", app=self._get_app(), image=modal_image(plan, prebake=self.prebake), - timeout=self.ttl_s, idle_timeout=self.idle_timeout_s, cpu=float(task.cpus), - memory=int(task.memory_mb), workdir=task.workdir, tags=tags, - **self._client_kwargs(), - **self.network_policy.grant(task.task_id).modal_kwargs(), - ) - return tb2.ModalSandbox(sandbox, task.workdir) - - -def modal_provider(miles_args: Any = None) -> tb2.Tb2EnvironmentProvider: - """``YETO_HARNESS_ENVIRONMENT_PROVIDER=yeto.cloud.modal_reward_env:modal_provider``.""" - del miles_args - tb2.require_modal_client() - backend = PrebakedModalSandboxBackend( - prebake=_prebake_enabled(), - app_name=os.environ.get(rt.APP_ENV, rt.DEFAULT_APP), - ttl_s=int(tb2._env_float(tb2.SANDBOX_TTL_ENV, tb2.DEFAULT_SANDBOX_TTL_S) or tb2.DEFAULT_SANDBOX_TTL_S), - idle_timeout_s=int(tb2._env_float(tb2.IDLE_TIMEOUT_ENV, tb2.DEFAULT_IDLE_TIMEOUT_S) - or tb2.DEFAULT_IDLE_TIMEOUT_S), - network_policy=tb2.NetworkPolicy.from_env(), - ) - return tb2._provider(backend) - - -RLF_CTF_TASKS_DIR_ENV = "YETO_RLF_CTF_TASKS_DIR" -RLF_CTF_FLAGS_ENV = "YETO_RLF_CTF_FLAGS_JSON" # {"task_id": "flag{...}"} (trusted only) -RLF_CTF_VM_IMAGE_ENV = "YETO_RLF_CTF_VM_IMAGE" -RLF_CTF_LEASE_SECONDS_ENV = "YETO_RLF_CTF_LEASE_SECONDS" - - -def rlf_ctf_config_from_env(environ: Any = None) -> RlfCtfConfig | None: - """CTF packaging config, or None when the launch did not configure it. - - Fail closed: setting ``YETO_RLF_CTF_TASKS_DIR`` without flags (or with - invalid JSON) raises instead of silently degrading to the VM-only lease. - """ - import json - from pathlib import Path - - environ = os.environ if environ is None else environ - tasks_dir = environ.get(RLF_CTF_TASKS_DIR_ENV) - if not tasks_dir: - return None - if not Path(tasks_dir).is_dir(): - raise RuntimeError(f"{RLF_CTF_TASKS_DIR_ENV}={tasks_dir}: not a directory") - raw_flags = environ.get(RLF_CTF_FLAGS_ENV) - if not raw_flags: - raise RuntimeError(f"{RLF_CTF_TASKS_DIR_ENV} is set but {RLF_CTF_FLAGS_ENV} is not " - "(the trusted flags never live in the task package)") - flags = json.loads(raw_flags) - if not isinstance(flags, dict) or not all(isinstance(k, str) and isinstance(v, str) and v - for k, v in flags.items()): - raise RuntimeError(f"{RLF_CTF_FLAGS_ENV} must be a JSON object of task_id -> flag") - lease = float(environ.get(RLF_CTF_LEASE_SECONDS_ENV) or 1800.0) - return RlfCtfConfig(tasks_dir=tasks_dir, flags=flags, - vm_image=environ.get(RLF_CTF_VM_IMAGE_ENV) or "docker:27-dind", - lease_seconds=lease) - - -def rlf_modal_provider(miles_args: Any = None, *, - config: RlfModalVmConfig | None = None, - ctf: RlfCtfConfig | None = None) -> RlfModalProvider: - """``YETO_HARNESS_ENVIRONMENT_PROVIDER=yeto.cloud.modal_reward_env:rlf_modal_provider``. - - Same factory contract as ``modal_provider`` (preflight calls - ``factory(miles_args)`` and requires ``.acquire`` on the product), but each - acquire opens one isolated Modal VM carrying RLF task/episode/trajectory - identity instead of a TB2 sandbox. - - When CTF packaging is configured (``ctf=`` or ``YETO_RLF_CTF_TASKS_DIR`` + - ``YETO_RLF_CTF_FLAGS_JSON``), the provider is a full ``EnvironmentProvider``: - each acquire additionally runs the packaged nested-Docker CTF orchestration - and the lease exposes env_url/env_token/verifier/destroy/describe/deadline - (audit breakpoint 3). Without it, the S20 VM-only lease is returned. - """ - del miles_args - return RlfModalProvider(config=config, ctf=ctf if ctf is not None else rlf_ctf_config_from_env()) diff --git a/yeto/cloud/modal_rlf_baseline.py b/yeto/cloud/modal_rlf_baseline.py deleted file mode 100644 index 6ede1d4e3..000000000 --- a/yeto/cloud/modal_rlf_baseline.py +++ /dev/null @@ -1,193 +0,0 @@ -"""Modal deployment shape for the RLF Qwen3-30B-A3B baseline (rlf-codex-rl 5.6). - -1xA100-80GB container running ``yeto.rl.eval.rlf_baseline.run_from_config``: -SGLang bf16 from the ``yeto-fn-models`` Volume -(``hf/Qwen3-30B-A3B/ad44e777/``) + Miles session server + the Codex harness -against per-episode RLF CTF VMs (created by the in-container Modal client). - -Mirrors ``modal_eval_island.build_eval_app``: same pinned Miles-ports image, -same workdir/codex mounts. Launch (like the S17 N11 smoke): - - python tools/probes/... style driver, or: - - modal run yeto.cloud.modal_rlf_baseline --config-json run.json - -Local entrypoint inputs (never baked into the image): - -* ``YETO_CODEX_BUNDLE_DIR`` — the attested Codex bundle dir - (``scripts/fetch_codex_bundle.py``), mounted at ``/opt/yeto/codex``; -* ``YETO_RLF_CTF_TASKS_DIR`` — local dir of admitted task packages, mounted - at ``/root/rlf-tasks`` (the container-side value overrides it); -* ``YETO_RLF_BASELINE_ENVS_JSON`` — JSON file with the container env: - the Codex contract env (``yeto.launcher.codex_container_env( - codex_bundle_contract(...))``, same as the eval island's codex-env.json), - ``YETO_RLF_CTF_FLAGS_JSON`` (trusted flags; - live only in the Modal Secret, never in the image or the traces) and the - sandbox-only ``MODAL_TOKEN_ID`` / ``MODAL_TOKEN_SECRET`` the RLF provider - needs to open episode VMs. -""" - -from __future__ import annotations - -import json -import os -from dataclasses import dataclass, field -from pathlib import Path -from typing import Any, Mapping - -APP_NAME = "yeto-rlf-baseline" -BASELINE_GPU = "A100-80GB" -MODELS_VOLUME = "yeto-fn-models" # hf/// + MANIFEST.json -RESULTS_VOLUME = "yeto-rlf-baseline-runs" -MODELS_MOUNT = "/mnt/yeto-models" # yeto.launcher.MODEL_STORE_MOUNT -RESULTS_MOUNT = "/mnt/yeto-rlf-runs" -WORKDIR_MOUNT = "/root/sky_workdir" # same as the training/eval islands -CODEX_MOUNT = "/opt/yeto/codex" # yeto.launcher.CODEX_CONTAINER_DIR -TASKS_MOUNT = "/root/rlf-tasks" -ENVS_JSON_ENV = "YETO_RLF_BASELINE_ENVS_JSON" - -_SECRET_ENV_KEYS = ("YETO_RLF_CTF_FLAGS_JSON", "MODAL_TOKEN_ID", "MODAL_TOKEN_SECRET") - - -def default_image_ref() -> str: - from yeto.rl import MILES_NEXT_IMAGE # re-exported pin (same route as the eval island) - - return MILES_NEXT_IMAGE.removeprefix("docker:") - - -@dataclass -class BaselineFunctionSpec: - """Shape of the baseline Modal function; ``gpu`` is a Modal GPU string.""" - - app_name: str - workdir: str - codex_dir: str - tasks_dir: str - models_volume: str = MODELS_VOLUME - results_volume: str = RESULTS_VOLUME - gpu: str = BASELINE_GPU - cpu: float = 8.0 - memory_mib: int = 65536 - timeout_s: int = 6 * 3600 - image_ref: str = field(default_factory=default_image_ref) - envs: Mapping[str, str] = field(default_factory=dict) - - def container_env(self) -> dict[str, str]: - # No PYTHONPATH override (S17 N11: it breaks Modal's container entrypoint). - return {"HOME": "/root", "PYTHONUNBUFFERED": "1", "YETO_RLF_CTF_TASKS_DIR": TASKS_MOUNT, - **{str(k): str(v) for k, v in self.envs.items()}} - - -def build_baseline_app(spec: BaselineFunctionSpec) -> tuple[Any, Any]: - """(app, baseline GPU function). Models Volume read-only; results Volume committed.""" - import modal - - image = (modal.Image.from_registry(spec.image_ref) - .env({"HOME": "/root", "PYTHONUNBUFFERED": "1"}) - .add_local_dir(spec.workdir, WORKDIR_MOUNT, copy=False, - ignore=[".git", ".venv", ".claude", "**/__pycache__", "**/*.pyc", "s1-runs"]) - .add_local_dir(spec.codex_dir, CODEX_MOUNT, copy=False) - .add_local_dir(spec.tasks_dir, TASKS_MOUNT, copy=False)) - app = modal.App(spec.app_name) - models = modal.Volume.from_name(spec.models_volume, create_if_missing=True) - results = modal.Volume.from_name(spec.results_volume, create_if_missing=True) - secret = modal.Secret.from_dict(spec.container_env()) - run = app.function(image=image, gpu=spec.gpu, cpu=spec.cpu, memory=spec.memory_mib, - timeout=spec.timeout_s, secrets=[secret], - volumes={MODELS_MOUNT: models, RESULTS_MOUNT: results}, - name="rlf_baseline")(baseline_body) - return app, run - - -def baseline_body(config: Mapping[str, Any]) -> dict[str, Any]: - """In-container entry: run the baseline, then copy the evidence onto the results Volume.""" - import shutil - import sys - import time - - t0 = time.time() - sys.path.insert(0, WORKDIR_MOUNT) - from yeto.rl.eval import rlf_baseline - - config = dict(config) - local_out = "/tmp/rlf-baseline-out" - config["out_dir"] = local_out - config.setdefault("tasks_dir", TASKS_MOUNT) - # capture sglang/session-server logs into the results volume copy - config.setdefault("baseline", {}) - config["baseline"].setdefault("log_dir", local_out + "/logs") - # Trusted flags arrive via the Modal Secret env, never via run.json on disk. - if not config.get("flags"): - config["flags"] = json.loads(os.environ.get("YETO_RLF_CTF_FLAGS_JSON") or "{}") - run_name = str(config.get("run_name") or time.strftime("rlf-%Y%m%d-%H%M%S", time.gmtime())) - result = rlf_baseline.run_from_config(config) - final_dir = Path(RESULTS_MOUNT) / run_name - shutil.copytree(local_out, final_dir, dirs_exist_ok=True) - import modal - - modal.Volume.from_name(str(config.get("results_volume") or RESULTS_VOLUME), - create_if_missing=True).commit() - return {"run_name": run_name, "results": str(final_dir), "wall_s": round(time.time() - t0, 3), - **result} - - -def spec_from_env(environ: Mapping[str, str] | None = None) -> BaselineFunctionSpec: - """Local-side spec from the environment (see the module docstring).""" - environ = os.environ if environ is None else environ - repo = Path(__file__).resolve().parents[2] - codex_dir = environ.get("YETO_CODEX_BUNDLE_DIR") - tasks_dir = environ.get("YETO_RLF_CTF_TASKS_DIR") - if not codex_dir or not Path(codex_dir).is_dir(): - raise RuntimeError("YETO_CODEX_BUNDLE_DIR must point at the attested Codex bundle dir") - if not tasks_dir or not Path(tasks_dir).is_dir(): - raise RuntimeError("YETO_RLF_CTF_TASKS_DIR must point at the local admitted task packages") - envs: dict[str, str] = {} - envs_json = environ.get(ENVS_JSON_ENV) - if envs_json: - envs.update({str(k): str(v) for k, v in json.loads(Path(envs_json).read_text()).items()}) - for key in _SECRET_ENV_KEYS: - if environ.get(key) and key not in envs: - envs[key] = str(environ[key]) - return BaselineFunctionSpec(app_name=environ.get("YETO_RLF_BASELINE_APP", APP_NAME), - workdir=str(repo), codex_dir=codex_dir, tasks_dir=tasks_dir, - envs=envs) - - -try: # the Modal entry point (only usable with the modal client installed) - import modal as _modal - - try: - app, baseline_run = build_baseline_app(spec_from_env()) - except RuntimeError: - # In-container hydration re-imports this module; the local-only spec - # env (bundle dir, tasks dir) is absent there. The container finds - # the function body (``baseline_body``) by qualname, not via ``app``. - app = None - baseline_run = None - - def main(argv: "list[str] | None" = None) -> int: - """Local driver: ``python -m yeto.cloud.modal_rlf_baseline run.json``. - - Driver pattern (same as the S17 eval-island probes): the app is built - locally and the run is spawned inside ``app.run()``; ``modal run -m`` - is NOT usable here because the spec needs local-only paths. - """ - import sys - - args = sys.argv[1:] if argv is None else argv - if app is None or baseline_run is None: - raise RuntimeError("baseline app is only built locally (spec env missing)") - payload = json.loads(Path(args[0]).read_text()) - with app.run(): - result = baseline_run.remote(payload) - print(json.dumps(result, indent=1, default=str)) - return 0 - -except ImportError: # pragma: no cover - CPU tests have no modal client - _modal = None - app = None - baseline_run = None - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/yeto/cloud/modal_tb2_baseline.py b/yeto/cloud/modal_tb2_baseline.py deleted file mode 100644 index 04eec3b89..000000000 --- a/yeto/cloud/modal_tb2_baseline.py +++ /dev/null @@ -1,129 +0,0 @@ -"""Modal deployment shape for the TB2 base-model baseline (``yeto.rl.eval.tb2_baseline``). - -One GPU container: the pinned Miles-ports image (same mounts and env as the -eval island), the base model read-only from the models Volume -(``hf///``), TB2 sandboxes created by the in-container Modal -client. Evidence (traces, summary, SGLang / session-server logs, nvidia-smi -samples) is copied to the results Volume every ``sync_s`` seconds and once at -the end, so a container killed by its timeout still leaves its traces. - -The driver builds the app locally (the spec needs local-only paths) and must -``spawn`` the call inside ``app.run(detach=True)`` and return: killing a client -that blocks on ``.remote()`` cancels the input. -""" - -from __future__ import annotations - -import json -import os -import shutil -import threading -import time -from dataclasses import dataclass, field -from pathlib import Path -from typing import Any, Mapping - -from .modal_eval_island import (CODEX_MOUNT, TB2_MOUNT, WORKDIR_MOUNT, _gpu_sampler, _image_paths, - default_image_ref) - -MODELS_VOLUME = "yeto-fn-models" -MODELS_MOUNT = "/mnt/yeto-models" -RESULTS_VOLUME = "yeto-tb2-baseline-runs" -RESULTS_MOUNT = "/mnt/yeto-tb2-baseline" -FUNCTION_NAME = "tb2_baseline" - - -@dataclass -class BaselineFunctionSpec: - """``gpu`` is a Modal GPU string pinned to one type (``H200``; ``H100!`` keeps Modal from upgrading H100 to H200); ``envs`` become one secret - (codex contract env, Modal token for the TB2 sandboxes, TB2 / worker knobs).""" - - app_name: str - workdir: str - codex_dir: str - tb2_dir: str - gpu: str = "H200" - gpus: int = 1 - cpu: float = 16.0 - memory_mib: int = 131072 - timeout_s: int = 4 * 3600 - image_ref: str = field(default_factory=default_image_ref) - envs: Mapping[str, str] = field(default_factory=dict) - - def container_env(self) -> dict[str, str]: - # No PYTHONPATH here (see EvalFunctionSpec.container_env). - return {"HOME": "/root", "PYTHONUNBUFFERED": "1", "YETO_HARNESS_TB2_TASKS_DIR": TB2_MOUNT, - **{str(k): str(v) for k, v in self.envs.items()}} - - -def _sync(src: Path, dst: Path, volume: Any) -> None: - shutil.copytree(src, dst, dirs_exist_ok=True) - volume.commit() - - -def baseline_body(config: Mapping[str, Any]) -> dict[str, Any]: - """Inside the container: assert the GPU type, run the baseline, keep the evidence on the results Volume.""" - import subprocess - - import modal - - _image_paths() - from yeto.rl.eval import tb2_baseline - - names = subprocess.run(["nvidia-smi", "--query-gpu=name", "--format=csv,noheader"], - capture_output=True, text=True, timeout=30).stdout.split("\n") - names = [n.strip() for n in names if n.strip()] - want = str(config["assert_gpu_name"]) - if not names or not all(want in n for n in names): - raise RuntimeError(f"baseline asked for {want}, got {names}") - run_name = str(config["run_name"]) - local = Path("/tmp/tb2-baseline") / run_name - final = Path(RESULTS_MOUNT) / run_name - local.mkdir(parents=True, exist_ok=True) - (local / "body.json").write_text(json.dumps({"gpu_names": names, "container": os.environ.get("MODAL_TASK_ID"), - "start_ts": time.time()}, sort_keys=True)) - results = modal.Volume.from_name(RESULTS_VOLUME, create_if_missing=True) - stop = threading.Event() - sampler = threading.Thread(target=_gpu_sampler, args=(local / "gpu.jsonl", stop), daemon=True) - sampler.start() - - def syncer() -> None: - while not stop.wait(float(config.get("sync_s", 120))): - try: - _sync(local, final, results) - except Exception as exc: # noqa: BLE001 - the next tick retries - print(f"[tb2-baseline] sync failed: {exc}", flush=True) - - threading.Thread(target=syncer, daemon=True).start() - infer = dict(config.get("infer") or {}) - infer.setdefault("log_dir", str(local / "logs")) - try: - summary = tb2_baseline.run_from_config({**config, "infer": infer, "out_dir": str(local)}) - finally: - stop.set() - sampler.join(timeout=10) - _sync(local, final, results) - return {"run_name": run_name, "results": str(final), **{k: summary.get(k) for k in - ("attempts", "judged", "passed", "pass_rate", "infra_errors", "tasks_mixed", "attempts_wall_s")}} - - -def build_baseline_app(spec: BaselineFunctionSpec) -> tuple[Any, Any]: - """(app, baseline GPU function). Models Volume read-only.""" - import modal - - image = (modal.Image.from_registry(spec.image_ref) - .env({"HOME": "/root", "PYTHONUNBUFFERED": "1"}) - .add_local_dir(spec.workdir, WORKDIR_MOUNT, copy=False, - ignore=[".git", ".venv", ".claude", "**/__pycache__", "**/*.pyc", "s1-runs", - "syncer/target"]) - .add_local_dir(spec.codex_dir, CODEX_MOUNT, copy=False) - .add_local_dir(spec.tb2_dir, TB2_MOUNT, copy=False, ignore=[".git"])) - app = modal.App(spec.app_name) - models = modal.Volume.from_name(MODELS_VOLUME) - results = modal.Volume.from_name(RESULTS_VOLUME, create_if_missing=True) - gpu = spec.gpu if spec.gpus == 1 else f"{spec.gpu}:{spec.gpus}" - run = app.function(image=image, gpu=gpu, cpu=spec.cpu, memory=spec.memory_mib, timeout=spec.timeout_s, - secrets=[modal.Secret.from_dict(spec.container_env())], retries=0, - volumes={MODELS_MOUNT: models.read_only(), RESULTS_MOUNT: results}, - name=FUNCTION_NAME)(baseline_body) - return app, run diff --git a/yeto/launch/args.py b/yeto/launch/args.py index 81e0de97d..cac76db7e 100644 --- a/yeto/launch/args.py +++ b/yeto/launch/args.py @@ -10,7 +10,7 @@ from .loss import pickled_loss_path, resolve_loss_function from .storage import modal_checkpoint_store from .rl_args import _prepare_rl_args -from .codex import modal_sandbox_secret_name +from .harness_env import modal_sandbox_secret_name def prepare_launch_args( diff --git a/yeto/launch/codex.py b/yeto/launch/codex.py deleted file mode 100644 index 18c63f1e3..000000000 --- a/yeto/launch/codex.py +++ /dev/null @@ -1,314 +0,0 @@ -"""Codex harness and reward-sandbox environment for RL islands.""" - -from __future__ import annotations - -import json -import os -import re -import shlex -from pathlib import Path - - -# --- Stock Codex run bundle (ports engine; design R-SCOPE / T3 G2) --------------- -# -# The legacy direct SSH harness freezes the attested Codex artifact into the run -# bundle and ``--volume``-mounts it at /opt/yeto/codex (ssh_harness.py). The -# sky / Modal launch paths do the same from a local bundle directory -# (``scripts/fetch_codex_bundle.py``): the launcher attests it with the same -# ``_codex_harness_contract`` the SSH harness uses, mounts the directory at -# ``CODEX_CONTAINER_DIR`` (sky file_mounts / Modal add_local_dir) and injects -# the ``YETO_CODEX_*`` environment the container preflights check -# (``yeto.rl.adapters.miles.island_entry._preflight_codex_harness`` expected_env and -# ``codex_harness_agent._attest_runtime``); every value comes from the contract. -CODEX_BUNDLE_DIR_ENV = "YETO_CODEX_BUNDLE_DIR" -CODEX_CONTAINER_DIR = "/opt/yeto/codex" -CODEX_BUNDLE_FILES = ( - "codex_harness_binary", - "codex_package_manifest", - "codex_app_server_schema", -) -# Harness hook / provider env the ports island preflight resolves -# (entry.HARNESS_PREFLIGHT_ENV, preflight.ENVIRONMENT_PROVIDER_ENV); the -# provider must come from the launching environment (no in-tree default). -HARNESS_PREFLIGHT_ENV = "YETO_HARNESS_PREFLIGHT" -HARNESS_ENVIRONMENT_PROVIDER_ENV = "YETO_HARNESS_ENVIRONMENT_PROVIDER" -HARNESS_PASSTHROUGH_ENV = ( - HARNESS_ENVIRONMENT_PROVIDER_ENV, - # tb2_provider Modal Sandbox backend: the island creates task sandboxes - # itself. secret-handling-hardening: it gets a separate sandbox-only Modal - # token (YETO_SANDBOX_MODAL_TOKEN_*), never the main MODAL_TOKEN_*. - "YETO_SANDBOX_MODAL_TOKEN_ID", - "YETO_SANDBOX_MODAL_TOKEN_SECRET", - "OPENENV_RUN_ID", - "SECRLENV_MAX_TURNS", -) -HARNESS_PASSTHROUGH_ENV_PREFIXES = ("YETO_HARNESS_TB2_", "YETO_RLF_") -# tb2_provider's Modal Sandbox backend runs inside the island (rollout -# workers create the task sandboxes), so the island python needs the Modal -# client. The ports image does not ship it; install it in setup without -# touching the image's protobuf 7 (modal pins <7 but works; verified in-image -# on ghcr 12fcd9e5: Sandbox create/exec/terminate, ray/sglang/miles import). -MODAL_SANDBOX_PROVIDER = "yeto.rl.harness.codex.tb2_provider:modal_provider" -# rl-agentic-reward-env: same TB2 provider, sandboxes from prebaked images. -PREBAKED_MODAL_SANDBOX_PROVIDER = "yeto.cloud.modal_reward_env:modal_provider" -RLF_MODAL_SANDBOX_PROVIDER = "yeto.cloud.modal_reward_env:rlf_modal_provider" -MODAL_SANDBOX_PROVIDERS = frozenset({MODAL_SANDBOX_PROVIDER, PREBAKED_MODAL_SANDBOX_PROVIDER, RLF_MODAL_SANDBOX_PROVIDER}) -SANDBOX_MODAL_TOKEN_ENVS = ("YETO_SANDBOX_MODAL_TOKEN_ID", "YETO_SANDBOX_MODAL_TOKEN_SECRET") - - -def require_sandbox_modal_token(envs) -> None: - """secret-handling-hardening: a Modal Sandbox provider on the island needs - the sandbox-only token; the main MODAL_TOKEN_* never goes to an island.""" - missing = [n for n in SANDBOX_MODAL_TOKEN_ENVS if not envs.get(n)] - if missing: - raise ValueError( - f"the Modal Sandbox provider needs {', '.join(missing)} in the launching " - "environment: a separate Modal token used only for task sandboxes (the main " - "MODAL_TOKEN_ID/SECRET is never shipped to a learner island)") -# --modal-sandbox-secret: in-island fail-closed check (names only, never values). -SANDBOX_TOKEN_IN_ISLAND_CHECK = ( - "for _v in " + " ".join(SANDBOX_MODAL_TOKEN_ENVS) + "; do " - '[ -n "$(printenv "$_v")" ] || { echo "[yeto-setup] $_v missing in the island: ' - 'the --modal-sandbox-secret Modal Secret must hold it" >&2; exit 1; }; done' -) - - -def modal_sandbox_secret_name(args, spec) -> str | None: - """--modal-sandbox-secret NAME, or None when unset. Only Modal islands can - attach a named Modal Secret: any other cloud is refused, never ignored.""" - name = getattr(args, "modal_sandbox_secret", None) - if not name: - return None - if getattr(spec, "cloud", None) != "modal": - raise ValueError( - f"--modal-sandbox-secret attaches a Modal Secret to a Modal island; " - f"island cloud {getattr(spec, 'cloud', None)!r} cannot use it") - if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._-]{0,63}", str(name)): - raise ValueError(f"--modal-sandbox-secret {name!r} is not a Modal Secret name") - return str(name) - - -MODAL_CLIENT_SETUP = ( - # --ignore-installed: in a Modal Function container the setup shell - # already sees Modal's runtime copies (/pkg, /__modal/deps), so a plain - # install reports "already satisfied" and writes nothing into the island - # python's site-packages, which is all its own processes can import - # (codex-smoke-20261003-8/-9). - "python3 -m pip install -q --no-deps --ignore-installed " - "'modal==1.5.5' 'grpclib>=0.4.7,<0.4.10' 'synchronicity~=0.12.5' cbor2 toml " - "types-certifi types-toml watchfiles\n" - # Verify the packages physically landed in site-packages and import under - # the run script's PYTHONPATH (what the learner and Ray workers see). - 'PYTHONPATH="$HOME/miles:$HOME/sglang/python:$HOME/sky_workdir${PYTHONPATH:+:$PYTHONPATH}" ' - "python3 -c 'import os, sys, sysconfig; import modal, grpclib, synchronicity; " - "sp = sysconfig.get_paths()[\"purelib\"]; " - "missing = [m for m in (\"modal\", \"grpclib\", \"synchronicity\") if not os.path.isdir(os.path.join(sp, m))]; " - "sys.exit(f\"not in {sp}: {missing}\" if missing else 0)' " - "|| { echo '[yeto-setup] Modal client unusable in the island python' >&2; exit 1; }" -) -CODEX_COMPACTION_ENV = "YETO_CODEX_COMPACTION_ENABLED" - - -def codex_bundle_paths(bundle_dir: str) -> dict[str, str]: - """The three attested files inside a fetched bundle dir, keyed like the SSH - harness CLI (``--codex-harness-binary`` ...).""" - from ..rl import CODEX_CONTAINER_APP_SERVER_SCHEMA_PATH, CODEX_CONTAINER_BINARY_PATH - - root = Path(bundle_dir).expanduser() - return { - "codex_harness_binary": str(root / os.path.basename(CODEX_CONTAINER_BINARY_PATH)), - "codex_package_manifest": str(root / "codex-package.json"), - "codex_app_server_schema": str( - root / os.path.basename(CODEX_CONTAINER_APP_SERVER_SCHEMA_PATH) - ), - } - - -def codex_bundle_contract(args, bundle_dir: str) -> dict: - """Attest ``bundle_dir`` against the Yeto pins (same checks as the SSH harness).""" - from argparse import Namespace - - from ..rl.ssh_harness import HarnessError, _codex_harness_contract - - if not os.path.isdir(os.path.expanduser(bundle_dir)): - raise ValueError(f"{CODEX_BUNDLE_DIR_ENV}={bundle_dir!r} is not a directory") - try: - return _codex_harness_contract(Namespace(**codex_bundle_paths(bundle_dir)), args) - except HarnessError as exc: - raise ValueError(f"Codex bundle {bundle_dir}: {exc}") from exc - - -def codex_container_env(contract: dict) -> dict[str, str]: - """``YETO_CODEX_*`` for the island container, mirroring the SSH harness - ``codex_env`` template and the learner's expected_env, from the contract.""" - from ..rl.ssh_harness import _canonical_json, _plan_digest - - backend = contract["backend"] - env = { - "YETO_CODEX_BINARY_PATH": contract["container_binary_path"], - "YETO_CODEX_BINARY_SHA256": contract["binary_sha256"], - "YETO_CODEX_BINARY_SIZE_BYTES": str(contract["binary_size_bytes"]), - "YETO_CODEX_VERSION": contract["cli_version"], - "YETO_CODEX_APP_SERVER_PROTOCOL_REVISION": contract["app_server_protocol_revision"], - "YETO_CODEX_APP_SERVER_SCHEMA_SHA256": contract["app_server_schema_sha256"], - "YETO_CODEX_BASE_INSTRUCTIONS_SHA256": contract["base_instructions_sha256"], - "YETO_CODEX_TERMINAL_EXEC_TOOL_SCHEMA_SHA256": contract["terminal_exec_tool_schema_sha256"], - "YETO_CODEX_SUBMIT_TOOL_SCHEMA_SHA256": contract["submit_tool_schema_sha256"], - "YETO_CODEX_DYNAMIC_TOOLS_SCHEMA_SHA256": contract["dynamic_tools_schema_sha256"], - "YETO_CODEX_REASONING_EFFORT": contract["reasoning_effort"], - "YETO_CODEX_BACKEND_MAX_TOKENS": str(backend["max_tokens"]), - "YETO_CODEX_BACKEND_REASONING_EFFORT": backend["reasoning_effort"], - "YETO_CODEX_BACKEND_THINKING": backend["thinking"]["type"], - "YETO_CODEX_CHAT_TEMPLATE": backend["chat_template"], - "YETO_CODEX_CHAT_TEMPLATE_KWARGS": _canonical_json(backend["chat_template_kwargs"]), - "YETO_CODEX_TITO_ALLOWED_APPEND_ROLES": "tool,user", - } - env.update({str(k): str(v) for k, v in contract.get("openenv_identity_env", {}).items()}) - env["YETO_CODEX_HARNESS_CONTRACT_SHA256"] = _plan_digest(contract) - return env - - -def rl_gae_variant(args) -> str | None: - """``advantage.gae_variant`` of the prepared RL algorithm spec (None if absent).""" - raw = getattr(args, "rl_algorithm_spec_json", None) - if getattr(args, "training_mode", "sft") != "rl" or not raw: - return None - try: - payload = json.loads(raw) - except (TypeError, ValueError): - return None - value = ((payload or {}).get("advantage") or {}).get("gae_variant") - return None if value is None else str(value) - - -def compactionrl_launch_env(args, environ, harness_compacts: bool) -> dict[str, str]: - """Rollout-worker env for CompactionRL (design D8; progress "S13 Codex 压缩接线"). - - A ``cross_segment_per_sample`` spec switches ``YETO_CODEX_COMPACTIONRL=1`` on - (and forwards ``YETO_CODEX_COMPACTIONRL_T_COMP`` from the launching - environment when set); the island preflight forwards every ``YETO_CODEX_*`` - to the Ray rollout workers. Any other spec returns {} (env unchanged). - Raises when spec, harness and the launching environment disagree - (``compactionrl.check_rollout_compaction``).""" - from ..rl.algos import compactionrl as crl - - variant = rl_gae_variant(args) - raw_switch = environ.get(crl.COMPACTION_SWITCH_ENV) - t_comp = environ.get(crl.COMPACTION_T_COMP_ENV) - if variant is None and not raw_switch and not t_comp: - return {} - try: - explicit = crl.compaction_switch(raw_switch) - wanted = variant == crl.COMPACTED_GAE_VARIANT - if raw_switch not in (None, "") and explicit != wanted: - if wanted: - raise ValueError( - f"{crl.COMPACTION_SWITCH_ENV}={raw_switch!r} in the launching environment " - f"contradicts the CompactionRL spec ({crl.COMPACTED_GAE_VARIANT})" - ) - crl.check_rollout_compaction( - variant, wanted or explicit, harness_compacts=harness_compacts, t_comp=t_comp - ) - except ValueError as error: - raise ValueError(f"CompactionRL launch: {error}") from error - if not wanted: - return {} - generate = getattr(args, "custom_generate_function_path", None) - if getattr(args, "custom_agent_function_path", None) and generate != crl.CODEX_OPENENV_GENERATE: - # s19-compaction-g1-20261010b: with upstream agentic_tool_call.generate the - # segment sessions are never collected, samples carry no tokens_after and - # the trainer fails at round 0 (cross_segment_per_sample needs tokens_after). - raise ValueError( - f"CompactionRL launch: {crl.COMPACTED_GAE_VARIANT} needs " - f"--custom-generate-function-path {crl.CODEX_OPENENV_GENERATE} (got {generate!r})") - env = {crl.COMPACTION_SWITCH_ENV: "1"} - if t_comp: - env[crl.COMPACTION_T_COMP_ENV] = str(t_comp).strip() - return env - - -def codex_harness_launch(args, environ=None) -> tuple[str, dict[str, str], dict[str, str]] | None: - """(learner flags, container envs, file_mounts) for a signed Codex agent on - the ports engine, or None when the run is not a Codex run. - - Fails closed: no bundle dir, a bundle that does not match the pins, no - environment provider, or trainable compaction switched on in the launching - environment (R-D5a) all raise before any cloud resource is requested.""" - from ..rl import CODEX_OPENENV_AGENT, SIGNED_CODEX_AGENTS - from ..rl.harness.codex.preflight import HARNESS_PREFLIGHT_SPEC, _FALSE - - environ = os.environ if environ is None else environ - custom_agent = getattr(args, "custom_agent_function_path", None) - from ..rl.synthetic_segments import SYNTHETIC_SEGMENTS_REWARD - - # TEST ONLY (task 6.4 G1): the synthetic two-segment reward stands in for a - # compacting harness; it is accepted only with cross_segment_per_sample. - synthetic = getattr(args, "reward_function", None) == SYNTHETIC_SEGMENTS_REWARD - if synthetic and rl_gae_variant(args) != "cross_segment_per_sample": - raise ValueError(f"{SYNTHETIC_SEGMENTS_REWARD} is only valid with gae_variant cross_segment_per_sample") - compaction_env = compactionrl_launch_env( - args, environ, custom_agent == CODEX_OPENENV_AGENT or (synthetic and custom_agent is None)) - if custom_agent not in SIGNED_CODEX_AGENTS: - return None - bundle_dir = getattr(args, "codex_bundle_dir", None) or environ.get(CODEX_BUNDLE_DIR_ENV) - if not bundle_dir: - raise ValueError( - f"the signed Codex harness on the ports engine needs {CODEX_BUNDLE_DIR_ENV} " - "(a bundle from scripts/fetch_codex_bundle.py) to mount at " - f"{CODEX_CONTAINER_DIR}" - ) - if environ.get(CODEX_COMPACTION_ENV, "").strip().lower() not in _FALSE: - raise ValueError( - f"{CODEX_COMPACTION_ENV} is set in the launching environment; the fork pin " - "session server has no trainable compaction (R-D5a)" - ) - contract = codex_bundle_contract(args, bundle_dir) - flags = f" --codex-harness-contract {shlex.quote(_json_compact(contract))}" - envs = codex_container_env(contract) - if custom_agent == CODEX_OPENENV_AGENT: - envs[HARNESS_PREFLIGHT_ENV] = environ.get(HARNESS_PREFLIGHT_ENV) or HARNESS_PREFLIGHT_SPEC - if not environ.get(HARNESS_ENVIRONMENT_PROVIDER_ENV): - raise ValueError( - f"{HARNESS_ENVIRONMENT_PROVIDER_ENV}=module:callable is required: the " - "Codex Terminal-Bench preflight fails closed without an environment provider" - ) - for name in HARNESS_PASSTHROUGH_ENV: - if environ.get(name): - envs[name] = environ[name] - for name, value in environ.items(): - if name.startswith(HARNESS_PASSTHROUGH_ENV_PREFIXES) and value: - envs[name] = value - envs.update(compaction_env) # {} unless the spec is CompactionRL - mounts = {CODEX_CONTAINER_DIR: str(Path(bundle_dir).expanduser().resolve())} - return flags, envs, mounts - - -HEAD_CODEX_BUNDLE_PATH = "~/yeto-codex-bundle" - - -def codex_head_staging(args, environ=None) -> tuple[dict[str, str], dict[str, str]]: - """(head file_mounts, head-job secret envs) for a signed Codex run under - ``--controller head`` (S17 M1). - - The head VM replays the launch and builds every island itself, so it needs - what ``codex_harness_launch`` reads on the submitter: the bundle directory - (staged at ``HEAD_CODEX_BUNDLE_PATH``; the caller points - ``args.codex_bundle_dir`` there) and the harness env (provider, TB2 knobs, - reward HMAC key, Modal token). The env goes as sky secrets (redacted), - never as plain envs. Returns ({}, {}) for a non-Codex run; fails closed - exactly like ``codex_harness_launch`` (attests the local bundle first). - """ - environ = os.environ if environ is None else environ - if codex_harness_launch(args, environ) is None: - return {}, {} - bundle_dir = getattr(args, "codex_bundle_dir", None) or environ.get(CODEX_BUNDLE_DIR_ENV) - mounts = {HEAD_CODEX_BUNDLE_PATH: str(Path(bundle_dir).expanduser().resolve())} - secrets = {name: environ[name] for name in (HARNESS_PREFLIGHT_ENV, *HARNESS_PASSTHROUGH_ENV) - if environ.get(name)} - secrets.update({name: value for name, value in environ.items() - if name.startswith(HARNESS_PASSTHROUGH_ENV_PREFIXES) and value}) - return mounts, secrets - - -def _json_compact(value) -> str: - return json.dumps(value, sort_keys=True, separators=(",", ":")) diff --git a/yeto/launch/harness_env.py b/yeto/launch/harness_env.py new file mode 100644 index 000000000..6821a9478 --- /dev/null +++ b/yeto/launch/harness_env.py @@ -0,0 +1,90 @@ +"""Launch-side island environment: the Modal sandbox secret and the CompactionRL switch.""" + +from __future__ import annotations + +import json +import os +import re + + +def modal_sandbox_secret_name(args, spec) -> str | None: + """--modal-sandbox-secret NAME, or None when unset. Only Modal islands can + attach a named Modal Secret: any other cloud is refused, never ignored.""" + name = getattr(args, "modal_sandbox_secret", None) + if not name: + return None + if getattr(spec, "cloud", None) != "modal": + raise ValueError( + f"--modal-sandbox-secret attaches a Modal Secret to a Modal island; " + f"island cloud {getattr(spec, 'cloud', None)!r} cannot use it") + if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._-]{0,63}", str(name)): + raise ValueError(f"--modal-sandbox-secret {name!r} is not a Modal Secret name") + return str(name) + + +def rl_gae_variant(args) -> str | None: + """``advantage.gae_variant`` of the prepared RL algorithm spec (None if absent).""" + raw = getattr(args, "rl_algorithm_spec_json", None) + if getattr(args, "training_mode", "sft") != "rl" or not raw: + return None + try: + payload = json.loads(raw) + except (TypeError, ValueError): + return None + value = ((payload or {}).get("advantage") or {}).get("gae_variant") + return None if value is None else str(value) + + +def compactionrl_launch_env(args, environ, harness_compacts: bool) -> dict[str, str]: + """Rollout-worker env for CompactionRL (design D8; progress "S13 Codex 压缩接线"). + + A ``cross_segment_per_sample`` spec switches ``YETO_CODEX_COMPACTIONRL=1`` on + (and forwards ``YETO_CODEX_COMPACTIONRL_T_COMP`` from the launching + environment when set). Any other spec returns {} (env unchanged). + Raises when spec, harness and the launching environment disagree + (``compactionrl.check_rollout_compaction``).""" + from ..rl.algos import compactionrl as crl + + variant = rl_gae_variant(args) + raw_switch = environ.get(crl.COMPACTION_SWITCH_ENV) + t_comp = environ.get(crl.COMPACTION_T_COMP_ENV) + if variant is None and not raw_switch and not t_comp: + return {} + try: + explicit = crl.compaction_switch(raw_switch) + wanted = variant == crl.COMPACTED_GAE_VARIANT + if raw_switch not in (None, "") and explicit != wanted: + if wanted: + raise ValueError( + f"{crl.COMPACTION_SWITCH_ENV}={raw_switch!r} in the launching environment " + f"contradicts the CompactionRL spec ({crl.COMPACTED_GAE_VARIANT})" + ) + crl.check_rollout_compaction( + variant, wanted or explicit, harness_compacts=harness_compacts, t_comp=t_comp + ) + except ValueError as error: + raise ValueError(f"CompactionRL launch: {error}") from error + if not wanted: + return {} + env = {crl.COMPACTION_SWITCH_ENV: "1"} + if t_comp: + env[crl.COMPACTION_T_COMP_ENV] = str(t_comp).strip() + return env + + +def check_compaction_launch(args, environ=None) -> None: + """Refuse a CompactionRL launch that cannot work, before any cloud resource. + + No agent harness compacts rollouts (the Codex OpenEnv bridge was removed in + S26 E0.5). TEST ONLY (task 6.4 G1): the synthetic two-segment reward stands + in for a compacting harness; it is accepted only with + ``cross_segment_per_sample`` and without a custom agent. + """ + from ..rl.synthetic_segments import SYNTHETIC_SEGMENTS_REWARD + + environ = os.environ if environ is None else environ + synthetic = getattr(args, "reward_function", None) == SYNTHETIC_SEGMENTS_REWARD + if synthetic and rl_gae_variant(args) != "cross_segment_per_sample": + raise ValueError(f"{SYNTHETIC_SEGMENTS_REWARD} is only valid with gae_variant cross_segment_per_sample") + harness_compacts = synthetic and getattr(args, "custom_agent_function_path", None) is None + compactionrl_launch_env(args, environ, harness_compacts) diff --git a/yeto/launch/modal_islands.py b/yeto/launch/modal_islands.py index a7f12af6f..b96808845 100644 --- a/yeto/launch/modal_islands.py +++ b/yeto/launch/modal_islands.py @@ -21,7 +21,7 @@ _rl_checkpoint_storage_name, droppable_rejoins_in_new_container, modal_checkpoint_store, resumes_in_new_container, ) -from .codex import CODEX_CONTAINER_DIR, modal_sandbox_secret_name +from .harness_env import modal_sandbox_secret_name from .fleet import TEARDOWN_UNVERIFIED_EXIT @@ -74,16 +74,13 @@ def build_modal_island_config(args, spec: ClusterSpec, learner_id: int, task, sy if rl and island_keeps_spot_volume(args): volume_name = _rl_checkpoint_storage_name(args.cluster_prefix, learner_id) volume_mount = _rl_checkpoint_mount(args.rl_completed_groups_path).replace("~", "/root", 1) - # Codex run bundle: the sky task mounts it at CODEX_CONTAINER_DIR; Modal - # mounts the same directory through the image (add_local_dir). + # Every sky file_mount (local prompt file, initial adapter, HF token file) + # is mounted into the container the same way; `~` is /root in Modal. all_mounts = dict(getattr(task, "file_mounts", None) or {}) - codex_dir = all_mounts.get(CODEX_CONTAINER_DIR) - # Every other sky file_mount (local prompt file, initial adapter, HF token - # file) is mounted into the container the same way; `~` is /root in Modal. extra_mounts = { ("/root" + target[1:] if target.startswith("~") else target): str(source) for target, source in all_mounts.items() - if target != CODEX_CONTAINER_DIR and not str(source).startswith(("s3://", "gs://", "r2://")) + if not str(source).startswith(("s3://", "gs://", "r2://")) } requirements: tuple[str, ...] = () if not rl: @@ -111,8 +108,6 @@ def build_modal_island_config(args, spec: ClusterSpec, learner_id: int, task, sy pip_requirements=requirements, volume_name=volume_name, volume_mount=volume_mount, - codex_dir=codex_dir, - codex_mount=CODEX_CONTAINER_DIR if codex_dir else None, extra_mounts=extra_mounts, checkpoint_store_volume_name=None if store_volume is None else store_volume[0], checkpoint_store_mount=None if store_volume is None else MODAL_CHECKPOINT_STORE_MOUNT, diff --git a/yeto/launch/rl_args.py b/yeto/launch/rl_args.py index 0c21669b9..53df042ee 100644 --- a/yeto/launch/rl_args.py +++ b/yeto/launch/rl_args.py @@ -343,16 +343,6 @@ def _prepare_rl_args( _rl_miles_function(args.custom_generate_function_path) custom_agent = getattr(args, "custom_agent_function_path", None) _rl_miles_function(custom_agent, "--custom-agent-function-path") - from ..rl import ( - SECRLENV_AGENTS, - SECRLENV_GENERATE, - SECRLENV_GROUP_FILTER, - SECRLENV_INFRASTRUCTURE_REPLACEMENTS, - SECRLENV_REWARD, - SECRLENV_ZERO_VARIANCE_REPLACEMENTS, - SIGNED_CODEX_AGENTS, - ) - if custom_agent is not None: if args.custom_generate_function_path is None: raise ValueError( @@ -367,120 +357,23 @@ def _prepare_rl_args( raise ValueError( "--custom-agent-function-path requires --tito-model" ) - if ( - args.apply_chat_template_kwargs is not None - and custom_agent not in SIGNED_CODEX_AGENTS - ): + if args.apply_chat_template_kwargs is not None: raise ValueError( "agentic session rollouts preserve raw messages and do not accept " "--apply-chat-template-kwargs" ) - codex_reasoning_effort = getattr(args, "codex_reasoning_effort", None) - codex_backend_profile = getattr(args, "codex_backend_profile", None) - if custom_agent in SIGNED_CODEX_AGENTS: - from ..rl.codex_backend import ( - stock_codex_backend_profile, - validate_stock_codex_fields, - ) - - codex_backend_profile = codex_backend_profile or str(args.tito_model) - args.codex_backend_profile = codex_backend_profile - codex_profile = stock_codex_backend_profile(codex_backend_profile) - codex_chat_template_kwargs = codex_profile["chat_template_kwargs"] - if args.apply_chat_template_kwargs is None: - args.apply_chat_template_kwargs = codex_chat_template_kwargs - validate_stock_codex_fields( - tito_model=str(args.tito_model), - codex_backend_profile=codex_backend_profile, - rl_model_recipe=getattr(args, "rl_model_recipe", "generic"), - model=str(args.model), - model_revision=str(args.model_revision), - rollout_model=getattr(args, "rollout_model", None), - rollout_model_revision=getattr(args, "rollout_model_revision", None), - apply_chat_template_kwargs=args.apply_chat_template_kwargs, - tito_allowed_append_roles=list( - getattr(args, "tito_allowed_append_roles", None) or () - ), - codex_reasoning_effort=codex_reasoning_effort, - lora_targets=str(args.lora_targets), - expert_full_count=int(getattr(args, "expert_full_count", 0)), - lora_expert_rank=int(getattr(args, "rl_lora_expert_rank", None) or 0), - ) - elif codex_reasoning_effort is not None or codex_backend_profile is not None: - raise ValueError( - "--codex-reasoning-effort/--codex-backend-profile requires a signed Codex agent" - ) - if getattr(args, "codex_exclude_reasoning_from_loss", False): - if custom_agent not in SIGNED_CODEX_AGENTS: - raise ValueError( - "--codex-exclude-reasoning-from-loss requires a signed Codex agent" - ) - from ..rl.codex_backend import stock_codex_reasoning_markers - - if stock_codex_reasoning_markers(codex_backend_profile) is None: - raise ValueError( - f"Codex profile {codex_backend_profile!r} declares no reasoning " - "markers; --codex-exclude-reasoning-from-loss is unavailable" - ) - secrlenv_infrastructure_replacements = getattr( - args, "secrlenv_max_infrastructure_replacements", None - ) - if custom_agent in SECRLENV_AGENTS: - if args.custom_generate_function_path != SECRLENV_GENERATE: - raise ValueError( - "the SecRLEnv agents require the signed SecRLEnv generate wrapper" - ) - if args.reward_function != SECRLENV_REWARD: - raise ValueError( - "the SecRLEnv agents require the signed SecRLEnv reward function" - ) - configured_filter = getattr(args, "dynamic_sampling_filter_path", None) - if configured_filter not in {None, SECRLENV_GROUP_FILTER}: - raise ValueError( - "the SecRLEnv agents require the exact signed group filter" - ) - zero_variance_replacements = getattr( - args, "dynamic_sampling_max_replacements", None - ) - if zero_variance_replacements not in { - None, - SECRLENV_ZERO_VARIANCE_REPLACEMENTS, - } or isinstance(zero_variance_replacements, bool): - raise ValueError( - "the SecRLEnv agents require zero variance replacements" - ) - if secrlenv_infrastructure_replacements not in { - None, - SECRLENV_INFRASTRUCTURE_REPLACEMENTS, - } or isinstance(secrlenv_infrastructure_replacements, bool): - raise ValueError( - "the SecRLEnv agents require exactly one infrastructure replacement" - ) - expected_oversampling = args.rollout_batch_size + 1 - if args.over_sampling_batch_size == args.rollout_batch_size: - args.over_sampling_batch_size = expected_oversampling - elif args.over_sampling_batch_size != expected_oversampling: - # agentic-rollout-utilization 1.1 (10-09 ruling): the SecRLEnv +1 - # is a reserved same-task retry slot (_SecRLEnvRetryDataSource), not - # over-sampling; over-sample/cut-off applies to the other agents. - raise ValueError( - "the SecRLEnv agents require oversampling equal to the training " - "batch plus one (a reserved same-task retry slot; over-sampling " - "with cut-off is not available to SecRLEnv agents)" - ) - args.dynamic_sampling_filter_path = SECRLENV_GROUP_FILTER - args.dynamic_sampling_max_replacements = ( - SECRLENV_ZERO_VARIANCE_REPLACEMENTS - ) - args.secrlenv_max_infrastructure_replacements = ( - SECRLENV_INFRASTRUCTURE_REPLACEMENTS - ) - elif secrlenv_infrastructure_replacements is not None: - raise ValueError( - "--secrlenv-max-infrastructure-replacements requires a signed " - "SecRLEnv agent" - ) + # The signed stock-Codex and SecRLEnv agents these flags configured were + # removed in S26 E0.5 (tag s26-pre-harness-removal); refuse them. + for flag, attr in ( + ("--codex-reasoning-effort", "codex_reasoning_effort"), + ("--codex-backend-profile", "codex_backend_profile"), + ("--codex-exclude-reasoning-from-loss", "codex_exclude_reasoning_from_loss"), + ("--secrlenv-max-infrastructure-replacements", "secrlenv_max_infrastructure_replacements"), + ): + if getattr(args, attr, None) not in (None, False): + raise ValueError(f"{flag} requires a signed Codex agent; the signed Codex and " + "SecRLEnv agents were removed (S26 E0.5)") agent_max_seq_len = getattr(args, "agent_max_seq_len", None) if agent_max_seq_len is not None: if custom_agent is None: @@ -517,16 +410,13 @@ def _prepare_rl_args( bounded_filter = "yeto.rl.filters.bounded_nonzero_reward_std" if dynamic_filter == stock_filter: args.dynamic_sampling_filter_path = bounded_filter - elif dynamic_filter not in { - bounded_filter, - SECRLENV_GROUP_FILTER, - }: + elif dynamic_filter != bounded_filter: raise ValueError( "--dynamic-sampling-max-replacements is only supported with " "an approved bounded nonzero-variance filter" ) - # After the agent contract binds its filter (SecRLEnv) and the over-sampling - # count: the expected spec hash must be the one the island computes. + # After the over-sampling count is final: the expected spec hash must be + # the one the island computes. _prepare_ports_algorithm(args) if not args.use_session_server and ( args.session_server_ip is not None diff --git a/yeto/launch/rl_island_task.py b/yeto/launch/rl_island_task.py index 1185f4c9a..ce862f1dd 100644 --- a/yeto/launch/rl_island_task.py +++ b/yeto/launch/rl_island_task.py @@ -38,11 +38,7 @@ from .in_vm_docker import ( DOCKER_LOGIN_UNSET, IN_VM_DOCKER_CLOUDS, in_vm_docker_run, in_vm_docker_setup, ) -from .codex import ( - HARNESS_ENVIRONMENT_PROVIDER_ENV, MODAL_CLIENT_SETUP, MODAL_SANDBOX_PROVIDERS, - SANDBOX_TOKEN_IN_ISLAND_CHECK, codex_harness_launch, modal_sandbox_secret_name, - require_sandbox_modal_token, -) +from .harness_env import check_compaction_launch def submitted_unix() -> float: @@ -96,7 +92,7 @@ def make_miles_island_task( ): raise ValueError("RL task requires prepared source and reward provenance") island_data, island_data_sha256, data_mounts = island_task_data(args, learner_id) - codex_launch = codex_harness_launch(args) # None unless a signed Codex agent + check_compaction_launch(args) # --rl-island-use-gpus-per-node: island rules see M GPUs/node; resources stay physical. island = rl_island_spec(args, spec) @@ -195,11 +191,6 @@ def make_miles_island_task( " --dynamic-sampling-max-replacements " f"{args.dynamic_sampling_max_replacements}" ) - if getattr(args, "secrlenv_max_infrastructure_replacements", None) is not None: - flags += ( - " --secrlenv-max-infrastructure-replacements " - f"{args.secrlenv_max_infrastructure_replacements}" - ) if args.rl_offload_train: flags += " --rl-offload-train" algorithm_prelude, algorithm_flags = _ports_algorithm_flags(args) @@ -254,20 +245,6 @@ def make_miles_island_task( flags += f" --session-server-workers {int(args.session_server_workers)}" if args.tito_model: flags += f" --tito-model {shlex.quote(args.tito_model)}" - if getattr(args, "codex_backend_profile", None): - flags += ( - " --codex-backend-profile " - f"{shlex.quote(args.codex_backend_profile)}" - ) - if getattr(args, "codex_exclude_reasoning_from_loss", False): - flags += " --codex-exclude-reasoning-from-loss" - if getattr(args, "codex_reasoning_effort", None): - # The learner's _preflight_codex_harness re-validates the stock - # profile (xhigh) and fails closed without it. - flags += ( - " --codex-reasoning-effort " - f"{shlex.quote(args.codex_reasoning_effort)}" - ) if getattr(args, "tito_allowed_append_roles", None): roles = " ".join( shlex.quote(role) for role in args.tito_allowed_append_roles @@ -319,10 +296,6 @@ def make_miles_island_task( file_mounts[HF_TOKEN_PATH] = local_token if rl_sao_streaming(args): file_mounts.update(sao_contract_mounts(args, learner_id)) - if codex_launch is not None: - codex_flags, codex_envs, codex_mounts = codex_launch - flags += codex_flags - file_mounts.update(codex_mounts) envs = { "SYNCER_ADDR": syncer_addr, **({"CRITIC_SYNCER_ADDR": critic_syncer_address(syncer_addr)} @@ -378,8 +351,6 @@ def make_miles_island_task( for name in ("CYBERGYM_REWARD_SCHEME", "CYBERGYM_REWARD_VIEW"): if os.environ.get(name): envs[name] = os.environ[name] - if codex_launch is not None: - envs.update(codex_envs) envs.update(elastic_debug_delay_env(args)) # 0.25 ({} unless the test switch is given) envs.update(elastic_debug_pause_env(args)) # link-pause test switch ({} by default) if getattr(args, "wandb", False): @@ -401,15 +372,6 @@ def make_miles_island_task( store_envs = store_object_envs(args, spec.cloud, spec.num_nodes) if store_envs: # the store's object API (yeto.cloud.s3_store) needs boto3 in the image setup_steps.append(STORE_CLIENT_SETUP) - sandbox_secret = modal_sandbox_secret_name(args, spec) - if codex_launch is not None and envs.get(HARNESS_ENVIRONMENT_PROVIDER_ENV) in MODAL_SANDBOX_PROVIDERS: - if sandbox_secret is None: - require_sandbox_modal_token(envs) - else: - # The token comes from the named Modal Secret at container start; - # the island re-checks it before anything else runs. - setup_steps.append(SANDBOX_TOKEN_IN_ISLAND_CHECK) - setup_steps.append(MODAL_CLIENT_SETUP) store_fs = model_store_filesystem(getattr(args, "model_store", None), spec.cloud, spec.region) store_env = "" if (spec.cloud == "modal" and getattr(args, "modal_model_volume", None) diff --git a/yeto/launcher.py b/yeto/launcher.py index 0abd2f4b5..f104fef1e 100644 --- a/yeto/launcher.py +++ b/yeto/launcher.py @@ -125,13 +125,8 @@ from .launch.in_vm_docker import ( # noqa: F401 DOCKER_LOGIN_UNSET, IN_VM_DOCKER_CLOUDS, in_vm_docker_image, in_vm_docker_run, ) -from .launch.codex import ( # noqa: F401 - CODEX_BUNDLE_DIR_ENV, CODEX_COMPACTION_ENV, CODEX_CONTAINER_DIR, - HARNESS_ENVIRONMENT_PROVIDER_ENV, HARNESS_PASSTHROUGH_ENV, HARNESS_PREFLIGHT_ENV, - HEAD_CODEX_BUNDLE_PATH, MODAL_CLIENT_SETUP, MODAL_SANDBOX_PROVIDER, MODAL_SANDBOX_PROVIDERS, - PREBAKED_MODAL_SANDBOX_PROVIDER, SANDBOX_MODAL_TOKEN_ENVS, SANDBOX_TOKEN_IN_ISLAND_CHECK, - codex_bundle_contract, codex_harness_launch, codex_head_staging, modal_sandbox_secret_name, - require_sandbox_modal_token, +from .launch.harness_env import ( # noqa: F401 + check_compaction_launch, compactionrl_launch_env, modal_sandbox_secret_name, ) from .launch.rl_island_task import _worker_node_ip_flag # noqa: F401 from .launch.diffusion_sample import make_diffusion_sample_task, run_diffusion_sample # noqa: F401 diff --git a/yeto/modal_runner.py b/yeto/modal_runner.py index b25be3c4f..ce3d6fefc 100644 --- a/yeto/modal_runner.py +++ b/yeto/modal_runner.py @@ -216,12 +216,6 @@ class ModalIslandConfig: # links CONTAINER_WORKDIR to it). workdir: str | None = str(REPO_ROOT) python_version: str = "3.12" - # Stock Codex run bundle (ports engine, signed Codex agent): a local dir - # holding the attested binary / package manifest / app-server schema, - # mounted read-only at ``codex_mount`` (= ``/opt/yeto/codex``, the - # CODEX_CONTAINER_BINARY_PATH parent) the same way sky file_mounts does. - codex_dir: str | None = None - codex_mount: str | None = None # Other sky file_mounts (container path -> local file or dir), mounted # read-only at start-up like the workdir. extra_mounts: dict[str, str] = field(default_factory=dict) @@ -294,10 +288,6 @@ def validate(self) -> None: raise ValueError("model_volume_name and model_volume_mount go together") if self.model_volume_name and self.model_volume_name in (self.volume_name, self.tape_volume_name): raise ValueError("the model volume must differ from the checkpoint/tape volumes") - if (self.codex_dir is None) != (self.codex_mount is None): - raise ValueError("codex_dir and codex_mount go together") - if self.codex_dir is not None and not os.path.isdir(self.codex_dir): - raise ValueError(f"codex_dir {self.codex_dir} is not a directory") for target, source in self.extra_mounts.items(): if not target.startswith("/"): raise ValueError(f"extra_mounts target {target} must be absolute") @@ -1022,9 +1012,6 @@ def build_image(self, cfg: ModalIslandConfig): copy=False, ignore=MODAL_WORKDIR_IGNORE, ) - if cfg.codex_dir and cfg.codex_mount: - # Codex run bundle (sky: file_mounts[codex_mount] = codex_dir). - image = image.add_local_dir(cfg.codex_dir, cfg.codex_mount, copy=False) for target, source in sorted(cfg.extra_mounts.items()): source = os.path.expanduser(source) if os.path.isdir(source): diff --git a/yeto/rl/__init__.py b/yeto/rl/__init__.py index f83714f06..4c920a88e 100644 --- a/yeto/rl/__init__.py +++ b/yeto/rl/__init__.py @@ -35,96 +35,3 @@ def __getattr__(name: str): return getattr(pins, name) raise AttributeError(f"module {__name__!r} has no attribute {name!r}") -SECRLENV_AGENT_PATH = "yeto/rl/harness/codex/agent.py" -SECRLENV_AGENT_SHA256 = ( - "84c6c708cba2142e3d043ae047c90f529b90b953b739261466b8a6e28b2c7411" -) -SECRLENV_AGENT = "yeto.rl.harness.codex.agent.run" -SECRLENV_REWARD = "yeto.rl.harness.codex.reward:reward_func" -SECRLENV_GROUP_FILTER = "yeto.rl.harness.codex.reward.check_group" -# Moved to the Miles adapter (yeto-framework-decoupling 5.7); was yeto.rl.harness.codex.generate. -SECRLENV_GENERATE = "yeto.rl.adapters.miles.harness_glue.codex_generate.generate" -SECRLENV_GENERATE_SHA256 = ( - "1630dc8a3aae3cbbd4708a81c9803920791ee58d6725c3fcaa4736373f8f9e48" -) -SECRLENV_ZERO_VARIANCE_REPLACEMENTS = 0 -SECRLENV_INFRASTRUCTURE_REPLACEMENTS = 1 - -# Stock Codex is part of the signed Yeto security-environment harness. These -# pins identify the official Linux artifact, not the controller's host binary. -CODEX_HARNESS_AGENT = "yeto.rl.harness.codex.codex_harness_agent.run" -CODEX_HARNESS_AGENT_PATH = "yeto/rl/harness/codex/codex_harness_agent.py" -CODEX_HARNESS_AGENT_SHA256 = ( - "459855e8845ab3b3cc1a4e2f47e5a9e8cc6cad79197208e43aa1d822a7c300ad" -) -CODEX_BASE_INSTRUCTIONS_SHA256 = ( - "1c183656ca1319142cba9e76baa199b7ab59f770a51a76660622a087e74ba846" -) -# 2026-10-07 (S15 stage 2): signed TB2 system prompt (codex_harness_agent. -# TB2_BASE_INSTRUCTIONS); the legacy CODEX_BASE_INSTRUCTIONS_SHA256 is unchanged. -CODEX_TB2_BASE_INSTRUCTIONS_SHA256 = ( - "707e144f4c41dd791f5875ea6693be4738d21e7dec73a3ea09ccbf92106bdeb3" -) -CODEX_TERMINAL_EXEC_TOOL_SCHEMA_SHA256 = ( - "868dbbff9fe2f5a57573826cae1ae1f4ceac04eff8689a522d9af7ef1b589c5a" -) -CODEX_SUBMIT_TOOL_SCHEMA_SHA256 = ( - "162980cf1de2346e6a246a739c10a31d0f5bd30c62b27c080887b298ecad1a6f" -) -CODEX_DYNAMIC_TOOLS_SCHEMA_SHA256 = ( - "06142f7664a668c11149b9410af6438423654ac7ee85fa382226bc7fbbf101af" -) -CODEX_CLI_VERSION = "codex-cli 0.145.0" -CODEX_NPM_PACKAGE = "@openai/codex@0.145.0-linux-x64" -CODEX_LINUX_TARGET = "x86_64-unknown-linux-musl" -CODEX_LINUX_BINARY_SHA256 = ( - "a2a05dafaa1acb002a45eaec0a462de5b13694fcfcd7bc43305f14781ce7be14" -) -CODEX_LINUX_BINARY_SIZE_BYTES = 310_730_800 -CODEX_NPM_TARBALL_SHA256 = ( - "11239480f8e3efd1430f23bbe91c1a397856b8bbe6185ccbaee2382d25e03df2" -) -CODEX_PACKAGE_MANIFEST_SHA256 = ( - "8da5349aa5a4242f5e11c5ca8ff4a16d8f9f912cb8accebea4def94edbf30aee" -) -CODEX_APP_SERVER_PROTOCOL_REVISION = "v2" -CODEX_APP_SERVER_SCHEMA_SHA256 = ( - "a88d865c3ca41fc63672baf423e28b3bfdb85b989e1c95e6e268932daaac91a0" -) -CODEX_CONTAINER_BINARY_PATH = "/opt/yeto/codex/codex-x86_64-unknown-linux-musl" -CODEX_CONTAINER_APP_SERVER_SCHEMA_PATH = ( - "/opt/yeto/codex/codex_app_server_protocol.v2.schemas.json" -) - -# Terminal-Bench uses a thin isolated-process wrapper around the same attested -# stock Codex runtime. It is intentionally not a SecRLEnv agent: its reward, -# retry, and cleanup evidence contracts are Terminal-Bench-specific. -CODEX_OPENENV_AGENT = "yeto.rl.harness.codex.codex_openenv_subprocess_agent_function.run" -CODEX_OPENENV_AGENT_MODULES = ( - "codex_openenv_subprocess_agent_function.py", - "codex_openenv_agent_worker.py", - "codex_openenv_agent_function.py", - # CompactionRL bridge (opt-in, YETO_CODEX_COMPACTIONRL); imported by - # codex_openenv_agent_function. Name list only: no per-module hash pin. - "compaction_bridge.py", -) -CODEX_OPENENV_IDENTITY_ENV = { - "YETO_CODEX_OPENENV_BACKEND_PROFILE": "qwen35_08b", - "YETO_CODEX_OPENENV_MODEL_ID": "Qwen/Qwen3.5-0.8B", - "YETO_CODEX_OPENENV_MODEL_REVISION": ("2fc06364715b967f1860aea9cf38778875588b17"), - "YETO_CODEX_OPENENV_BASE_INSTRUCTIONS_SHA256": ( - "1c183656ca1319142cba9e76baa199b7ab59f770a51a76660622a087e74ba846" - ), - "YETO_CODEX_OPENENV_TERMINAL_EXEC_TOOL_SCHEMA_SHA256": ( - "868dbbff9fe2f5a57573826cae1ae1f4ceac04eff8689a522d9af7ef1b589c5a" - ), - "YETO_CODEX_OPENENV_SUBMIT_TOOL_SCHEMA_SHA256": ( - "162980cf1de2346e6a246a739c10a31d0f5bd30c62b27c080887b298ecad1a6f" - ), - "YETO_CODEX_OPENENV_DYNAMIC_TOOLS_SCHEMA_SHA256": ( - "06142f7664a668c11149b9410af6438423654ac7ee85fa382226bc7fbbf101af" - ), -} -SIGNED_CODEX_AGENTS = frozenset((CODEX_HARNESS_AGENT, CODEX_OPENENV_AGENT)) - -SECRLENV_AGENTS = frozenset((SECRLENV_AGENT, CODEX_HARNESS_AGENT)) diff --git a/yeto/rl/adapters/miles/binding.py b/yeto/rl/adapters/miles/binding.py index adc7b90d6..f638d0fb0 100644 --- a/yeto/rl/adapters/miles/binding.py +++ b/yeto/rl/adapters/miles/binding.py @@ -16,8 +16,7 @@ 3. call -- one fixed CPU input through the mapped loading path gives a result equal to the pre-rename path (skipped, and reported as skipped, when the implementation cannot be imported here, e.g. a ``miles.*`` built-in on a - machine without Miles; not run for a filter that needs environment - evidence the fixed input cannot carry, see ``EVIDENCE_FILTERS``). + machine without Miles). The result is a JSON-able dict for the runtime manifest. """ @@ -31,13 +30,10 @@ from types import SimpleNamespace from typing import Any -from yeto.rl import SECRLENV_GROUP_FILTER from yeto.rl.engine.algorithm import ( BOUNDED_NONZERO_STD_FILTER, LEGACY_FILTER_NAMES, - SECRLENV_GROUP_CHECK_FILTER, STOCK_NONZERO_STD_FILTER, - TBENCH_GROUP_CHECK_FILTER, ) BINDING_SCHEMA = "yeto-backend-binding-v1" @@ -47,15 +43,8 @@ BOUNDED_NONZERO_STD_FILTER: "yeto.rl.filters.bounded_nonzero_reward_std", STOCK_NONZERO_STD_FILTER: "miles.rollout.filter_hub.dynamic_sampling_filters.check_reward_nonzero_std", - SECRLENV_GROUP_CHECK_FILTER: SECRLENV_GROUP_FILTER, - TBENCH_GROUP_CHECK_FILTER: "yeto.rl.harness.codex.tbench_reward.check_group", } -# Filters whose input must carry signed environment evidence (the SecRLEnv -# group check refuses any group without it), so the fixed CPU input of check -# ③ cannot exercise them; checks ① and ② still run. -EVIDENCE_FILTERS = frozenset({SECRLENV_GROUP_CHECK_FILTER, TBENCH_GROUP_CHECK_FILTER}) - class BindingError(RuntimeError): """A neutral name does not bind to the same implementation as before the rename.""" @@ -134,9 +123,6 @@ def check_filter(name: str, *, paths: Mapping[str, str] | None = None, if ident != ident_old: raise BindingError(f"{name!r} ② identity: {ident} != pre-rename {ident_old}") result["identity"] = ident - if name in EVIDENCE_FILTERS: - result["call"] = "skipped: needs environment evidence" - return result try: new_fn, old_fn = loader(mapped), loader(old) except ImportError as exc: diff --git a/yeto/rl/adapters/miles/carry_over.py b/yeto/rl/adapters/miles/carry_over.py index 4e4eca4a0..6905fda98 100644 --- a/yeto/rl/adapters/miles/carry_over.py +++ b/yeto/rl/adapters/miles/carry_over.py @@ -462,11 +462,11 @@ def trained_token_versions(sample: Any, current_version: int) -> list[int | None RESUME_STATS_ATTR = "rollout_resume_stats" # Miles fork, set at rollout start SUSPEND_EXPIRED_PREFIX = "CodexSuspendExpired" POLICY_AGE_EXCEEDED_PREFIX = "policy_age_exceeded" +# Sample metadata key of an infrastructure failure reason (set by the agent function). +INFRASTRUCTURE_KEY = "tbench_infrastructure_error" def _infrastructure_reason(sample: Any) -> str: - from yeto.rl.harness.codex.tbench_reward import INFRASTRUCTURE_KEY - meta = getattr(sample, "metadata", None) reason = meta.get(INFRASTRUCTURE_KEY) if isinstance(meta, dict) else None return str(reason) if reason else "" diff --git a/yeto/rl/adapters/miles/config.py b/yeto/rl/adapters/miles/config.py index dadbb341d..7b1b627eb 100644 --- a/yeto/rl/adapters/miles/config.py +++ b/yeto/rl/adapters/miles/config.py @@ -95,7 +95,7 @@ "--rollout-all-samples-process-path", "--rollout-sample-filter-path", "--rollout-function-path", - "--data-source-path", # SecRLEnv same-task retry (secrlenv_retry) + "--data-source-path", "--buffer-filter-path", "--fully-async", "--debug-train-only", @@ -189,48 +189,6 @@ def check(value, _config): return check -def _stock_codex_append_roles_or_none(reason: str) -> _Check: - """Append roles are decided by the ``--tito-model`` template upstream, so - the option is refused -- except for a signed stock Codex agent, whose - launcher-side profile check (``validate_stock_codex_fields``) requires the - roles to be stated and equal to the profile's. Those are accepted when they - match the profile and are NOT emitted to argv (the template decides).""" - - def check(value, config): - if value is None: - return None - agent = getattr(config, "agent", None) - try: - from yeto.rl import SIGNED_CODEX_AGENTS - from yeto.rl.codex_backend import ( - stock_codex_backend_profile, - stock_codex_profiles_for_tito_model, - ) - except ImportError: # pragma: no cover - defensive - return reason - if getattr(agent, "custom_agent_function_path", None) not in SIGNED_CODEX_AGENTS: - return reason - if not getattr(agent, "tito_model", None): - return reason - # ``tito_model`` is the Miles tokenizer family, not the profile name - # (``qwen35`` happened to be both; ``qwen4exp`` serves the Flash-Next - # profiles, rl-fn-codex-rollout 1.0): the roles are fixed by every - # profile of that family, which the launcher already matched to the - # declared ``--codex-backend-profile`` (``validate_stock_codex_fields``). - names = stock_codex_profiles_for_tito_model(str(agent.tito_model)) - if not names: - return reason - fixed = {tuple(stock_codex_backend_profile(n)["tito_allowed_append_roles"]) for n in names} - if len(fixed) != 1 or tuple(value) != next(iter(fixed)): - return ( - f"stock Codex profiles {list(names)} (tito-model {agent.tito_model!r}) fix the " - f"append roles to {sorted(list(r) for r in fixed)} (got {list(value)})" - ) - return None - - return check - - def _must_be_false(reason: str) -> _Check: def check(value, _config): return reason if value else None @@ -249,14 +207,7 @@ def check(value, _config): ) -# yeto's Codex wrapper calls AGENTIC_TOOL_CALL_GENERATE with the same args (so it -# reads the same agent flags) and turns CompactionRL segment sessions into one -# sample per segment; without it a compacted rollout trains as one sample. -from yeto.rl.algos.compactionrl import CODEX_OPENENV_GENERATE # noqa: E402 (single source) -from yeto.rl import SECRLENV_GENERATE # noqa: E402 (wraps AGENTIC_TOOL_CALL_GENERATE) -AGENT_FLAG_READERS = frozenset( - {AGENTIC_TOOL_CALL_GENERATE, CODEX_OPENENV_GENERATE, SECRLENV_GENERATE} -) +AGENT_FLAG_READERS = frozenset({AGENTIC_TOOL_CALL_GENERATE}) def _requires_agentic_generate(flag: str) -> _Check: @@ -452,7 +403,7 @@ def _check_sglang_tp(value, config): "agent.session_server_port": _ok, "agent.session_server_workers": _ok, "agent.tito_model": _ok, - "agent.tito_allowed_append_roles": _stock_codex_append_roles_or_none( + "agent.tito_allowed_append_roles": _must_be_none( "upstream Miles has no --tito-allowed-append-roles; the append roles are " "decided by the --tito-model template (allowed_append_roles)" ), @@ -526,22 +477,6 @@ def _algorithm_filter(config, algorithm: AlgorithmSpec) -> str | None: return algorithm.dynamic_sampling_filter -def _secrlenv_retry_argv(agent, batch, dynamic_filter: str | None, - max_policy_age: int) -> tuple[str, ...]: - """P0-1.7b.4-Q1: the SecRLEnv agents bind the same-task retry classes.""" - - from .secrlenv_retry import rollout_argv - - try: - return rollout_argv(custom_agent=agent.custom_agent_function_path, - dynamic_filter=dynamic_filter, - rollout_batch_size=batch.groups_per_round, - over_sampling_batch_size=batch.over_sampling_batch_size, - max_policy_age=max_policy_age) - except ValueError as exc: - raise UnmappedConfigError("agent.custom_agent_function_path", str(exc)) from None - - def placement_request(config) -> PlacementRequest: parallel = config.parallel trainer = parallel.actor_num_nodes * parallel.actor_num_gpus_per_node @@ -1075,7 +1010,6 @@ def translate_run_config( values.extend(("--custom-generate-function-path", agent.custom_generate_function_path)) if agent.custom_agent_function_path: # IR-1 (leaf policy proved agentic generate) values.extend(("--custom-agent-function-path", agent.custom_agent_function_path)) - values.extend(_secrlenv_retry_argv(agent, batch, dynamic_filter, max_policy_age)) if agent.agent_max_seq_len is not None: values.extend(("--max-seq-len", str(int(agent.agent_max_seq_len)))) if agent.use_session_server: diff --git a/yeto/rl/adapters/miles/entry.py b/yeto/rl/adapters/miles/entry.py index f05f8ac4e..b9a1683a0 100644 --- a/yeto/rl/adapters/miles/entry.py +++ b/yeto/rl/adapters/miles/entry.py @@ -28,8 +28,6 @@ from yeto.rl.engine.algorithm import ( BOUNDED_NONZERO_STD_FILTER, - SECRLENV_GROUP_CHECK_FILTER, - TBENCH_GROUP_CHECK_FILTER, STOCK_NONZERO_STD_FILTER, AlgorithmSpec, ) @@ -273,10 +271,7 @@ def miles_capabilities( runtime_fingerprint=fingerprint, parameter_layouts={"lora"}, placements={"colocated"}, - dynamic_sampling_filters={ - BOUNDED_NONZERO_STD_FILTER, STOCK_NONZERO_STD_FILTER, SECRLENV_GROUP_CHECK_FILTER, - TBENCH_GROUP_CHECK_FILTER, - }, + dynamic_sampling_filters={BOUNDED_NONZERO_STD_FILTER, STOCK_NONZERO_STD_FILTER}, execution_modes={"colocated-serial"}, execution=ExecutionCapabilities( critic=False, max_policy_staleness=0, rollout_logprobs=True @@ -432,21 +427,13 @@ def load_tool_wait_source(miles_args: Any, elastic: Any = None) -> Any: The elastic drain board when wired; the island's named board when the tool-wait workload generate is configured (it counts on that board); 0 (``TOOL_WAIT_NO_BOARD_STOCK``) for Miles' stock generate, which makes no - tool calls; the island's named board for the Codex OpenEnv agent, which - enters/exits it around every tool call (agentic-rollout-utilization 1.3); - None (unknown) for any other custom generate. + tool calls; None (unknown) for any other custom generate. """ - from yeto.rl import CODEX_OPENENV_AGENT - from .rollout import TOOL_WAIT_NO_BOARD_STOCK board = getattr(elastic, "tool_wait_board", None) if elastic is not None else None if board is not None: return board - if getattr(miles_args, "custom_agent_function_path", None) == CODEX_OPENENV_AGENT: - from .elastic_wiring import LazyBoardActor - - return LazyBoardActor(int(getattr(miles_args, "yeto_rl_learner_id", 0) or 0)) custom = getattr(miles_args, "custom_generate_function_path", None) if not custom: return TOOL_WAIT_NO_BOARD_STOCK @@ -531,10 +518,8 @@ def publish_member_cell(miles_args: Any, learner_id: int | None = None, environ: Source, in order: an explicit ``miles_args.yeto_rl_cell_id`` (INFRA may pre-set it), ``learner_id`` (the island number the launcher already assigns; ``island_id`` in the - ``rl_engine_selected`` event), ``miles_args.yeto_rl_learner_id``, else 0. The env - export reaches rollout-worker subprocesses through ``worker_runtime_env`` and - ``preflight.configure_rollout_worker``; ``preflight.resolve_member`` then yields - ``engine:`` (``engine:0`` on a single island). + ``rl_engine_selected`` event), ``miles_args.yeto_rl_learner_id``, else 0. Rollout-side + code reads the env export (``engine:``; ``engine:0`` on a single island). """ environ = os.environ if environ is None else environ cell = getattr(miles_args, "yeto_rl_cell_id", None) @@ -1280,7 +1265,7 @@ def selection_event( return event -def connect_island_ray(*, environ=None, ray_module=None, miles_args=None) -> str | None: +def connect_island_ray(*, environ=None, ray_module=None) -> str | None: """Connect the driver to the island's own Ray and pin every actor to it. A SkyPilot machine runs two Ray instances: the island's (6379, started by @@ -1355,12 +1340,6 @@ def connect_island_ray(*, environ=None, ray_module=None, miles_args=None) -> str # --rl-elastic: the rollout metadata hook runs inside Ray workers, which # inherit the raylet's environment, not the driver's. env_vars[ELASTIC_METADATA_ENV] = "1" - if miles_args is not None: - # Codex harness: the agent function runs in RolloutExecutor actors and - # configures itself from this env (preflight.configure_rollout_worker). - from yeto.rl.harness.codex.preflight import worker_runtime_env - - env_vars.update(worker_runtime_env(miles_args, environ)) ray_module.init(address=address, runtime_env={"env_vars": env_vars}) return address @@ -1974,7 +1953,7 @@ def run_ports_island( start_phases = StartPhases( lambda event, **fields: append_tape_event(miles_args, {"event": event, **fields})) start_phases.replay() - connect_island_ray(miles_args=miles_args) + connect_island_ray() topology = getattr(launch.placement, "topology", None) if topology is not None and topology.nodes > 1: # tasks 3.3: a restarted learner on a partial island (worker node DEAD in the GCS) diff --git a/yeto/rl/adapters/miles/harness_glue/codex_generate.py b/yeto/rl/adapters/miles/harness_glue/codex_generate.py deleted file mode 100644 index 0ab0ba00e..000000000 --- a/yeto/rl/adapters/miles/harness_glue/codex_generate.py +++ /dev/null @@ -1,129 +0,0 @@ -"""SecRLEnv wrapper for pinned Miles' agentic generate function. - -Moved from ``yeto/rl/harness/codex/generate.py`` (yeto-framework-decoupling 5.7): -Miles loads ``generate`` by path (``--custom-generate-function-path``). The -agent-metadata capture it reads stays neutral in ``yeto.rl.harness.codex.generate``. -This file verifies itself against ``yeto.rl.harness.codex.pins.GENERATE_SHA256``. -""" - -from __future__ import annotations - -import copy -import hashlib -import hmac -from pathlib import Path -from typing import Any - -from yeto.rl.harness.codex.generate import _CAPTURED_AGENT_METADATA -from yeto.rl.harness.codex.pins import GENERATE_SHA256 as SECRLENV_GENERATE_SHA256 -from yeto.rl.harness.codex.reward import ( - OUTCOME_KEY, - UntrustedOutcome, - _verified_outcome, - remember_terminal_error, -) - - -def _verify_pinned_source() -> None: - actual = hashlib.sha256(Path(__file__).read_bytes()).hexdigest() - if not hmac.compare_digest(actual, SECRLENV_GENERATE_SHA256): - raise RuntimeError("secrlenv generate wrapper does not match its Yeto pin") - - -_verify_pinned_source() - - -def _generated_samples(output: Any) -> list[Any]: - samples = getattr(output, "samples", None) - if isinstance(samples, list): - flattened: list[Any] = [] - pending = list(samples) - while pending: - sample = pending.pop(0) - if isinstance(sample, list): - pending[0:0] = sample - else: - flattened.append(sample) - return flattened - return [] if samples is None else [samples] - - -def _status_value(sample: Any) -> str | None: - status = getattr(sample, "status", None) - value = getattr(status, "value", status) - return value if isinstance(value, str) else None - - -async def generate(input: Any) -> Any: - """Preserve verified evidence across Miles' no-record aborted fallback. - - Pinned Miles intentionally returns an aborted copy when a session emitted no - model records or all records were truncated, but it does so before merging - the custom agent's metadata. SecRLEnv may restore that metadata only after - validating the complete outcome; every other missing/invalid - evidence path remains fatal. - - The restored sample is marked ``failed``, not ``aborted``: Miles drops a - group with an aborted sample before the dynamic filter runs, so the - SecRLEnv group check would never see it and could not schedule the - same-task retry. A ``failed`` sample still never trains (the group check - rejects any status other than completed or truncated). - """ - - from miles.rollout.generate_hub.agentic_tool_call import generate as upstream - - token = _CAPTURED_AGENT_METADATA.set(None) - try: - return await _generate(upstream, input) - except UntrustedOutcome as error: - remember_terminal_error(getattr(getattr(input, "state", None), "args", None), error) - raise - finally: - _CAPTURED_AGENT_METADATA.reset(token) - - -async def _generate(upstream: Any, input: Any) -> Any: - """Validate the upstream output; restore evidence on the no-record fallback.""" - - output = await upstream(input) - samples = _generated_samples(output) - if not samples: - raise UntrustedOutcome("SecRLEnv generation returned no samples") - - has_outcome = [ - isinstance(getattr(sample, "metadata", None), dict) - and OUTCOME_KEY in sample.metadata - for sample in samples - ] - if all(has_outcome): - return output - if ( - any(has_outcome) - or len(samples) != 1 - or _status_value(samples[0]) != "aborted" - or getattr(samples[0], "response", None) not in {None, ""} - ): - raise UntrustedOutcome("SecRLEnv generation lost outcome metadata") - - captured = _CAPTURED_AGENT_METADATA.get() - _verified_outcome(captured) - if not isinstance(captured, dict): # Kept explicit for type narrowing. - raise UntrustedOutcome("SecRLEnv generation has no outcome") - evidence = {OUTCOME_KEY: copy.deepcopy(captured[OUTCOME_KEY])} - metadata = getattr(samples[0], "metadata", None) - if not isinstance(metadata, dict): - raise UntrustedOutcome("SecRLEnv sample metadata is not an object") - metadata.update(evidence) - from miles.utils.types import Sample - - samples[0].status = Sample.Status.FAILED - return output - - -def _add_arguments(parser: Any) -> None: - from miles.rollout.generate_hub.agentic_tool_call import generate as upstream - - upstream.add_arguments(parser) - - -generate.add_arguments = _add_arguments diff --git a/yeto/rl/adapters/miles/harness_glue/codex_openenv.py b/yeto/rl/adapters/miles/harness_glue/codex_openenv.py deleted file mode 100644 index a71e42b9a..000000000 --- a/yeto/rl/adapters/miles/harness_glue/codex_openenv.py +++ /dev/null @@ -1,40 +0,0 @@ -"""Miles pieces of the codex OpenEnv generate wrapper (yeto-framework-decoupling 5.7). - -``yeto.rl.harness.codex.codex_openenv_generate`` keeps the trajectory -bookkeeping (neutral) and reaches these through the backend registry -(role ``harness_glue``): upstream ``agentic_tool_call.generate``, the -``Sample.Status.ABORTED`` value and the per-session sample collection. -""" - -from __future__ import annotations - -from typing import Any, Awaitable, Callable - - -def load_agentic_upstream() -> Callable[[Any], Awaitable[Any]]: - from miles.rollout.generate_hub.agentic_tool_call import generate as upstream - - return upstream - - -def aborted_status() -> Any: - from miles.utils.types import Sample - - return Sample.Status.ABORTED - - -async def collect_segment_session(input: Any, router: str, session_id: str) -> tuple[list[Any], dict[str, Any]]: - """Collect (and delete) one extra session exactly like upstream's tracer does.""" - from miles.rollout.generate_utils.openai_endpoint_utils import ( - COMPUTED_FIELDS, - ROLLOUT_SAMPLING_MASK_FIELDS, - OpenAIEndpointTracer, - should_return_sampling_mask, - ) - - fields = COMPUTED_FIELDS - if should_return_sampling_mask(input.args, input.sampling_params, evaluation=input.evaluation): - fields += ROLLOUT_SAMPLING_MASK_FIELDS - tracer = OpenAIEndpointTracer(router_url=router, session_id=session_id, samples_wire_fields=fields) - reply = await tracer.collect_samples(input.sample, max_seq_len=getattr(input.args, "max_seq_len", None)) - return list(reply.samples), dict(reply.session_metadata or {}) diff --git a/yeto/rl/adapters/miles/in_flight.py b/yeto/rl/adapters/miles/in_flight.py index 44a32207b..ffa7adc9e 100644 --- a/yeto/rl/adapters/miles/in_flight.py +++ b/yeto/rl/adapters/miles/in_flight.py @@ -104,32 +104,6 @@ def buffered_entries(buffer: Iterable[Sequence[Any]]) -> list[dict[str, Any]]: return entries -def agentic_entries(gates: Mapping[str, Any], inflight: Mapping[str, Mapping[str, Any]]) -> list[dict[str, Any]]: - """One entry per suspended agentic trajectory (reference only).""" - entries = [] - for trajectory_id in sorted(gates): - stage = (inflight.get(trajectory_id) or {}).get("stage") - entry = InFlightTrajectory( - trajectory_id=str(trajectory_id), group_id=f"agentic:{trajectory_id}", - task_id=str(trajectory_id), provenance=TokenProvenance((), ()), - session_ref=f"{AGENTIC_SESSION_PREFIX}{trajectory_id}", - ).to_dict() - if stage: - entry["stage"] = str(stage) - entries.append(entry) - return entries - - -def _codex_suspension() -> tuple[Mapping[str, Any], Mapping[str, Any]]: - import sys - - module = sys.modules.get("yeto.rl.harness.codex.codex_openenv_subprocess_agent_function") - if module is None: - return {}, {} - state = getattr(module, "_SUSPEND_STATE", {}) or {} - return dict(state.get("gates") or {}), dict(getattr(module, "_INFLIGHT", {}) or {}) - - def export_in_flight(executor: Any) -> dict[str, Any]: """Runs INSIDE the rollout executor actor: the in-flight entries and the data buffer length they account for.""" @@ -143,8 +117,7 @@ def export_in_flight(executor: Any) -> dict[str, Any]: source = getattr(executor, "data_source", None) source = getattr(source, "__self__", source) buffer = list(getattr(source, "buffer", None) or []) - gates, inflight = _codex_suspension() - entries = buffered_entries(buffer) + agentic_entries(gates, inflight) + entries = buffered_entries(buffer) return {"entries": entries, "buffer_groups": len(buffer)} diff --git a/yeto/rl/adapters/miles/island_entry.py b/yeto/rl/adapters/miles/island_entry.py index 48585224f..0193f0c35 100644 --- a/yeto/rl/adapters/miles/island_entry.py +++ b/yeto/rl/adapters/miles/island_entry.py @@ -12,7 +12,6 @@ import tempfile import time from collections.abc import Mapping, Sequence -from decimal import Decimal, DecimalException from pathlib import Path from typing import Any @@ -764,24 +763,6 @@ def _check_dense_full_sweep(args, *, yeto_policy_sync: bool) -> None: raise ValueError("dense full-parameter mode requires --syncer-profile-sha256") -def _trajectory_evidence_kind(args) -> str: - """The evidence kind of a dense full-parameter island, chosen by its reward. - - The SecRLEnv reward binds the grader's signed verdict as evidence. Any - other reward function (for example a GSM8K answer check) binds its own - output, ``sample.reward``; a sample that still carries a grader verdict is - rejected there. Terminal-Bench 2.1 runs the SAO sync, whose runtime - contract names its own evidence kind. - """ - - from yeto.rl import SECRLENV_REWARD - from yeto.rl.trajectory_evidence import REWARD_FUNCTION_EVIDENCE_KIND - - if args.reward_function == SECRLENV_REWARD: - return "secrlenv" - return REWARD_FUNCTION_EVIDENCE_KIND - - def _private_trajectory_evidence_dir(args) -> Path: """A fresh owner-only directory for this island's trajectory evidence.""" @@ -1017,93 +998,13 @@ def _dense_full_training_contract_sha256( "yeto_source_sha256": args.source_sha256, "miles_backend_identity_sha256": _backend_identity_sha256(args), "reward_sha256": args.reward_sha256, - "codex_harness_contract": getattr( - args, - "codex_harness_contract", - None, - ), + # The Codex harness contract left with the harness (S26 E0.5); the + # key stays so the v6 contract hash of existing runs is unchanged. + "codex_harness_contract": None, } ) -def _strict_json_file_semantics(path: Path) -> tuple[Any, ...]: - """Parse exact, typed JSON semantics independent of object-key order.""" - - def unique_object(pairs: list[tuple[str, Any]]) -> dict[str, Any]: - value: dict[str, Any] = {} - for name, item in pairs: - if name in value: - raise ValueError(f"duplicate JSON object key: {name!r}") - value[name] = item - return value - - def finite_number(name: str) -> None: - raise ValueError(f"non-finite JSON number: {name}") - - try: - value = json.loads( - path.read_bytes(), - object_pairs_hook=unique_object, - parse_constant=finite_number, - parse_float=Decimal, - parse_int=Decimal, - ) - except (OSError, UnicodeError, ValueError, DecimalException) as exc: - raise ValueError("Codex app-server schema is malformed JSON") from exc - - def typed_semantics(item: Any) -> tuple[Any, ...]: - if item is None: - return ("null",) - if isinstance(item, bool): - return ("boolean", item) - if isinstance(item, Decimal): - return ("number", item) - if isinstance(item, str): - return ("string", item) - if isinstance(item, list): - return ("array", tuple(typed_semantics(value) for value in item)) - if isinstance(item, dict): - return ( - "object", - tuple( - sorted( - (name, typed_semantics(value)) - for name, value in item.items() - ) - ), - ) - raise TypeError(f"unsupported parsed JSON value: {type(item).__name__}") - - return typed_semantics(value) - - -def _canonical_schema_sha256(path: Path) -> str | None: - from yeto.provenance import canonical_json_sha256 - - try: - return canonical_json_sha256(path) - except (OSError, ValueError): - return None - - -def _verify_live_codex_app_server_schema(pinned: Path, generated: Path) -> None: - try: - pinned_semantics = _strict_json_file_semantics(pinned) - generated_semantics = _strict_json_file_semantics(generated) - except ValueError as exc: - raise ValueError("live stock Codex app-server schema drifted") from exc - if generated_semantics != pinned_semantics: - raise ValueError("live stock Codex app-server schema drifted") - - -def _preflight_codex_openenv_adapter(args, profile_name: str) -> None: - """Attest the in-tree OpenEnv adapter (forwarded to harness.codex.preflight).""" - - from yeto.rl.harness.codex.preflight import forward_legacy_openenv_preflight - - forward_legacy_openenv_preflight(args, profile_name) - - def miles_revision_pins(args: Any) -> Any: """The Miles commit the island checks for ``--miles-commit`` (full SHA); None without it (``verify_miles_revision`` then checks the pinned MILES_NEXT_PINS).""" @@ -1118,279 +1019,6 @@ def miles_revision_pins(args: Any) -> Any: return MilesRevisionPins(MILES_NEXT_REPOSITORY, str(commit).lower()) -def _preflight_codex_harness(args) -> None: - """Fail before model allocation if the signed Codex surface has drifted.""" - - from yeto.rl import ( - CODEX_APP_SERVER_PROTOCOL_REVISION, - CODEX_APP_SERVER_SCHEMA_SHA256, - CODEX_BASE_INSTRUCTIONS_SHA256, - CODEX_CLI_VERSION, - CODEX_CONTAINER_APP_SERVER_SCHEMA_PATH, - CODEX_CONTAINER_BINARY_PATH, - CODEX_DYNAMIC_TOOLS_SCHEMA_SHA256, - CODEX_HARNESS_AGENT, - CODEX_HARNESS_AGENT_SHA256, - CODEX_LINUX_BINARY_SHA256, - CODEX_LINUX_BINARY_SIZE_BYTES, - CODEX_LINUX_TARGET, - CODEX_NPM_PACKAGE, - CODEX_NPM_TARBALL_SHA256, - CODEX_OPENENV_AGENT, - CODEX_OPENENV_IDENTITY_ENV, - CODEX_PACKAGE_MANIFEST_SHA256, - CODEX_SUBMIT_TOOL_SCHEMA_SHA256, - CODEX_TERMINAL_EXEC_TOOL_SCHEMA_SHA256, - SIGNED_CODEX_AGENTS, - ) - - contract = getattr(args, "codex_harness_contract", None) - if args.custom_agent_function_path not in SIGNED_CODEX_AGENTS: - if contract is not None: - raise ValueError( - "--codex-harness-contract requires a signed Codex agent" - ) - return - if not isinstance(contract, dict): - raise ValueError("the signed Codex agent requires its harness contract") - required = { - "agent_function_path", - "agent_source_sha256", - "controller_binary_path", - "controller_package_manifest_path", - "controller_app_server_schema_path", - "bundle_binary_path", - "bundle_package_manifest_path", - "bundle_app_server_schema_path", - "container_binary_path", - "container_app_server_schema_path", - "binary_sha256", - "binary_size_bytes", - "cli_version", - "npm_package", - "target", - "npm_tarball_sha256", - "package_manifest_sha256", - "app_server_protocol_revision", - "app_server_schema_sha256", - "base_instructions_sha256", - "terminal_exec_tool_schema_sha256", - "submit_tool_schema_sha256", - "dynamic_tools_schema_sha256", - "reasoning_effort", - "backend", - } - if args.custom_agent_function_path == CODEX_OPENENV_AGENT: - required.add("openenv_identity_env") - if set(contract) != required: - raise ValueError("stock Codex harness contract shape drifted") - pinned = { - "agent_function_path": CODEX_HARNESS_AGENT, - "agent_source_sha256": CODEX_HARNESS_AGENT_SHA256, - "container_binary_path": CODEX_CONTAINER_BINARY_PATH, - "container_app_server_schema_path": ( - CODEX_CONTAINER_APP_SERVER_SCHEMA_PATH - ), - "binary_sha256": CODEX_LINUX_BINARY_SHA256, - "binary_size_bytes": CODEX_LINUX_BINARY_SIZE_BYTES, - "cli_version": CODEX_CLI_VERSION, - "npm_package": CODEX_NPM_PACKAGE, - "target": CODEX_LINUX_TARGET, - "npm_tarball_sha256": CODEX_NPM_TARBALL_SHA256, - "package_manifest_sha256": CODEX_PACKAGE_MANIFEST_SHA256, - "app_server_protocol_revision": CODEX_APP_SERVER_PROTOCOL_REVISION, - "app_server_schema_sha256": CODEX_APP_SERVER_SCHEMA_SHA256, - "base_instructions_sha256": CODEX_BASE_INSTRUCTIONS_SHA256, - "terminal_exec_tool_schema_sha256": ( - CODEX_TERMINAL_EXEC_TOOL_SCHEMA_SHA256 - ), - "submit_tool_schema_sha256": CODEX_SUBMIT_TOOL_SCHEMA_SHA256, - "dynamic_tools_schema_sha256": CODEX_DYNAMIC_TOOLS_SCHEMA_SHA256, - "reasoning_effort": "xhigh", - } - if any(contract.get(name) != expected for name, expected in pinned.items()): - raise ValueError("stock Codex runtime identity drifted") - if ( - args.custom_agent_function_path == CODEX_OPENENV_AGENT - and contract.get("openenv_identity_env") != CODEX_OPENENV_IDENTITY_ENV - ): - raise ValueError("stock Codex OpenEnv surface identity drifted") - from yeto.rl.codex_backend import ( - stock_codex_backend_contract_with_reasoning_policy, - validate_stock_codex_fields, - ) - - profile_name = getattr(args, "codex_backend_profile", None) or args.tito_model - expected_backend = stock_codex_backend_contract_with_reasoning_policy( - profile_name, - args.rollout_max_response_len, - not getattr(args, "codex_exclude_reasoning_from_loss", False), - ) - validate_stock_codex_fields( - tito_model=args.tito_model, - codex_backend_profile=profile_name, - rl_model_recipe=args.rl_model_recipe, - model=args.model, - model_revision=args.model_revision, - rollout_model=getattr(args, "rollout_model", None), - rollout_model_revision=getattr(args, "rollout_model_revision", None), - apply_chat_template_kwargs=args.apply_chat_template_kwargs, - tito_allowed_append_roles=args.tito_allowed_append_roles, - codex_reasoning_effort=args.codex_reasoning_effort, - # The signed Qwen rollout backend profile predates full-parameter - # training and uses ``attention`` only as its legacy tuning-profile - # discriminator. Full mode emits no LoRA runtime flags below. - lora_targets=( - args.lora_targets - if getattr(args, "parameter_mode", "lora") == "lora" - else "attention" - ), - expert_full_count=getattr(args, "expert_full_count", 0), - lora_expert_rank=( - int(getattr(args, "rl_lora_expert_rank", None) or 0) - if getattr(args, "parameter_mode", "lora") == "lora" - else 0 - ), - ) - if contract.get("backend") != expected_backend: - raise ValueError("stock Codex backend/TITO identity drifted") - from yeto.provenance import file_sha256 - - # OpenEnv owns and attests its adapter inside the pinned Miles source. The - # legacy SecRLEnv-backed Codex adapter remains an optional integration, but - # it must not be required for the independent Terminal-Bench path. - if args.custom_agent_function_path != CODEX_OPENENV_AGENT: - try: - from yeto.rl.harness.codex import codex_harness_agent - - live_identity = codex_harness_agent.codex_harness_identity() - except ( - ImportError, - AttributeError, - TypeError, - ValueError, - RuntimeError, - ) as exc: - raise ValueError("cannot attest the Yeto Codex harness adapter") from exc - identity_names = ( - "base_instructions_sha256", - "terminal_exec_tool_schema_sha256", - "submit_tool_schema_sha256", - "dynamic_tools_schema_sha256", - ) - if live_identity != {name: contract.get(name) for name in identity_names}: - raise ValueError("Yeto Codex adapter identity drifted") - - adapter_path = Path(codex_harness_agent.__file__) - if adapter_path.is_symlink() or not adapter_path.is_file(): - raise ValueError("Yeto Codex adapter source identity drifted") - adapter_path = adapter_path.resolve() - if file_sha256(adapter_path) != contract.get("agent_source_sha256"): - raise ValueError("Yeto Codex adapter source identity drifted") - binary = Path(CODEX_CONTAINER_BINARY_PATH) - manifest = Path("/opt/yeto/codex/codex-package.json") - schema = Path(CODEX_CONTAINER_APP_SERVER_SCHEMA_PATH) - if ( - binary.is_symlink() - or not binary.is_file() - or binary.stat().st_size != CODEX_LINUX_BINARY_SIZE_BYTES - or file_sha256(binary) != CODEX_LINUX_BINARY_SHA256 - or manifest.is_symlink() - or not manifest.is_file() - or file_sha256(manifest) != CODEX_PACKAGE_MANIFEST_SHA256 - or schema.is_symlink() - or not schema.is_file() - or _canonical_schema_sha256(schema) != CODEX_APP_SERVER_SCHEMA_SHA256 - ): - raise ValueError("mounted stock Codex artifact does not match its Yeto pin") - try: - version = subprocess.run( - [str(binary), "--version"], - check=True, - capture_output=True, - text=True, - timeout=15, - ).stdout.strip() - except (OSError, subprocess.SubprocessError) as exc: - raise ValueError("cannot execute the pinned stock Codex binary") from exc - if version != CODEX_CLI_VERSION: - raise ValueError("stock Codex executable version drifted") - try: - with tempfile.TemporaryDirectory(prefix="yeto-codex-schema-") as temporary: - root = Path(temporary) - output = root / "schema" - subprocess.run( - [ - str(binary), - "app-server", - "generate-json-schema", - "--experimental", - "--out", - str(output), - ], - check=True, - capture_output=True, - text=True, - timeout=30, - ) - generated_schema = ( - output / "codex_app_server_protocol.v2.schemas.json" - ) - if ( - generated_schema.is_symlink() - or not generated_schema.is_file() - ): - raise ValueError("live stock Codex app-server schema drifted") - _verify_live_codex_app_server_schema(schema, generated_schema) - except (OSError, subprocess.SubprocessError) as exc: - raise ValueError("cannot verify the stock Codex app-server schema") from exc - expected_env = { - "YETO_CODEX_BINARY_PATH": CODEX_CONTAINER_BINARY_PATH, - "YETO_CODEX_BINARY_SHA256": CODEX_LINUX_BINARY_SHA256, - "YETO_CODEX_BINARY_SIZE_BYTES": str(CODEX_LINUX_BINARY_SIZE_BYTES), - "YETO_CODEX_VERSION": CODEX_CLI_VERSION, - "YETO_CODEX_APP_SERVER_PROTOCOL_REVISION": ( - CODEX_APP_SERVER_PROTOCOL_REVISION - ), - "YETO_CODEX_APP_SERVER_SCHEMA_SHA256": CODEX_APP_SERVER_SCHEMA_SHA256, - "YETO_CODEX_BASE_INSTRUCTIONS_SHA256": contract[ - "base_instructions_sha256" - ], - "YETO_CODEX_TERMINAL_EXEC_TOOL_SCHEMA_SHA256": contract[ - "terminal_exec_tool_schema_sha256" - ], - "YETO_CODEX_SUBMIT_TOOL_SCHEMA_SHA256": contract[ - "submit_tool_schema_sha256" - ], - "YETO_CODEX_DYNAMIC_TOOLS_SCHEMA_SHA256": contract[ - "dynamic_tools_schema_sha256" - ], - "YETO_CODEX_REASONING_EFFORT": "xhigh", - "YETO_CODEX_BACKEND_MAX_TOKENS": str(args.rollout_max_response_len), - "YETO_CODEX_BACKEND_REASONING_EFFORT": expected_backend[ - "reasoning_effort" - ], - "YETO_CODEX_BACKEND_THINKING": "enabled", - "YETO_CODEX_CHAT_TEMPLATE": expected_backend["chat_template"], - "YETO_CODEX_CHAT_TEMPLATE_KWARGS": json.dumps( - expected_backend["chat_template_kwargs"], - sort_keys=True, - separators=(",", ":"), - ), - "YETO_CODEX_TITO_ALLOWED_APPEND_ROLES": "tool,user", - "YETO_CODEX_HARNESS_CONTRACT_SHA256": _canonical_sha256(contract), - } - mismatched = [ - name for name, expected in expected_env.items() if os.getenv(name) != expected - ] - if mismatched: - raise ValueError( - "stock Codex container environment drifted: " + ", ".join(mismatched) - ) - if args.custom_agent_function_path == CODEX_OPENENV_AGENT: - _preflight_codex_openenv_adapter(args, profile_name) - - def _miles_callable(spec: str) -> str: module, separator, function = spec.partition(":") if not separator or not module or not function.isidentifier(): @@ -1898,18 +1526,6 @@ def run_miles( write_boot_only_marker(args, ports_launch, run_config, ref_probe) return # boot-only: exit 0 before Ray/GPU work; no training - if getattr(args, "codex_exclude_reasoning_from_loss", False): - # S17 WP6: read by rollout_meta_hook.record_trained_groups in the - # rollout process (with or without Yeto policy sync); absent (default) == reasoning counted, no-op. - miles_args.yeto_rl_codex_reasoning_in_loss = False - from yeto.rl.codex_backend import stock_codex_reasoning_markers - - miles_args.yeto_rl_codex_reasoning_markers = list( - stock_codex_reasoning_markers( - getattr(args, "codex_backend_profile", None) or args.tito_model - ) - or () - ) # The full-parameter sync contracts bind the spec hash (4.8). The island # contract v3 (strict, decoupled, elastic) binds the semantic hash and the # run code pack instead (rl-identity-tiering L9.7); placement capabilities @@ -1958,22 +1574,9 @@ def run_miles( spec.name for spec in specs ) miles_args.yeto_rl_reward_sha256 = args.reward_sha256 - miles_args.yeto_rl_codex_harness_contract = getattr( - args, - "codex_harness_contract", - None, - ) - miles_args.yeto_rl_codex_backend_profile = getattr( - args, - "codex_backend_profile", - None, - ) miles_args.yeto_rl_dynamic_sampling_max_replacements = getattr( args, "dynamic_sampling_max_replacements", None ) - miles_args.yeto_rl_secrlenv_max_infrastructure_replacements = getattr( - args, "secrlenv_max_infrastructure_replacements", None - ) miles_args.yeto_rl_completed_groups_path = args.completed_groups_path miles_args.yeto_rl_event_tape = args.event_tape miles_args.yeto_rl_learner_id = args.learner_id @@ -1989,6 +1592,7 @@ def run_miles( miles_args.external_policy_sync_run_until_stop = sync_preset == "decoupled" if dense_full: from yeto.rl.adapters.miles.full_parameter.session import CONFIG_ATTRIBUTE + from yeto.rl.trajectory_evidence import REWARD_FUNCTION_EVIDENCE_KIND miles_args.external_policy_identity_setter_path = ( "yeto.rl.adapters.miles.publish.set_current_published_policy_identity" @@ -1996,7 +1600,7 @@ def run_miles( miles_args.yeto_rl_trajectory_evidence_dir = str( _private_trajectory_evidence_dir(args) ) - miles_args.yeto_rl_trajectory_evidence_kind = _trajectory_evidence_kind(args) + miles_args.yeto_rl_trajectory_evidence_kind = REWARD_FUNCTION_EVIDENCE_KIND miles_args.yeto_rl_trajectory_evidence_schema_version = 1 setattr( miles_args, @@ -2258,7 +1862,7 @@ def _run_test_debug_rollout_replay(args, miles_args) -> bool: from yeto.rl.adapters.miles.entry import connect_island_ray # same job runtime_env as the ports island (PYTHONPATH -> pinned Miles/Megatron in the actors) - connect_island_ray(miles_args=miles_args) + connect_island_ray() print(f"[rl] TEST ONLY: upstream Miles train loop on {miles_args.load_debug_rollout_data}", flush=True) asyncio.run(with_disposer(module.train, miles_args)) print("[rl] TEST ONLY: upstream Miles train loop finished", flush=True) @@ -2383,7 +1987,6 @@ def main(argv=None) -> None: verify_miles_revision(miles_root) # the pinned MILES_NEXT_PINS else: verify_miles_revision(miles_root, expected=pins) - _preflight_codex_harness(args) probe_syncers(args) from miles.utils.misc import load_function @@ -2393,16 +1996,6 @@ def main(argv=None) -> None: load_function(args.custom_generate_function_path) if args.custom_agent_function_path: load_function(args.custom_agent_function_path) - from yeto.rl import CODEX_HARNESS_AGENT, SECRLENV_AGENT - - if args.custom_agent_function_path in { - SECRLENV_AGENT, - CODEX_HARNESS_AGENT, - }: - from yeto.rl.harness.codex.client import require_daemon_ready - - require_daemon_ready() - print("[rl] secrlenv episode daemon ready") from huggingface_hub import snapshot_download from yeto.models import resolve diff --git a/yeto/rl/adapters/miles/policy_age.py b/yeto/rl/adapters/miles/policy_age.py index d52886feb..53bcab818 100644 --- a/yeto/rl/adapters/miles/policy_age.py +++ b/yeto/rl/adapters/miles/policy_age.py @@ -44,9 +44,9 @@ AGENTIC_GENERATE = "miles.rollout.generate_hub.agentic_tool_call.generate" # Agent functions whose module defines the suspend()/resume() hooks Miles calls # under --agentic-suspend-between-turns (model-turn gate, survival limit). -SUSPEND_CAPABLE_AGENTS = frozenset({ - "yeto.rl.harness.codex.codex_openenv_subprocess_agent_function.run", -}) +# Empty since S26 E0.5: the only such agent was the removed self-written Codex +# harness; the Harbor agent function does not implement the hooks. +SUSPEND_CAPABLE_AGENTS: frozenset[str] = frozenset() def policy_age_argv(limit: int, *, agentic: bool = False, overlap: bool = False) -> tuple[str, ...]: diff --git a/yeto/rl/adapters/miles/rewards.py b/yeto/rl/adapters/miles/rewards.py index 060e15b15..32be9ea51 100644 --- a/yeto/rl/adapters/miles/rewards.py +++ b/yeto/rl/adapters/miles/rewards.py @@ -15,7 +15,7 @@ from typing import Any, Callable -from yeto.rl.rewards.registry import RewardRegistryError, load_reward, register_reward +from yeto.rl.rewards.registry import RewardRegistryError, load_reward from yeto.rl.rewards.types import FilterDecision, RewardResult, Trajectory _MODULE = __name__ @@ -122,29 +122,6 @@ def miles_custom_rm_path(ref: str) -> str: return f"{_MODULE}.{attr}" -def rlf_success_reward(trajectory: Trajectory) -> RewardResult: - """Neutral success declaration for an authenticated RLF grader result. - - This is the framework-neutral ``Trajectory -> RewardResult`` function. - Do NOT pass ``yeto.rl.adapters.miles.rewards:rlf_success_reward`` - directly as Miles ``--custom-rm-path``: Miles invokes reward callables - as ``fn(args, samples)``, so the unwrapped neutral signature would not - match. Use ``miles_custom_rm_path("rlf_success")`` instead; it resolves - to the ``__getattr__``-wrapped attribute below, which adapts the neutral - function to the Miles calling convention via :func:`miles_reward`. - """ - from yeto.rl.rlf.preflight import validate_rollout - validate_rollout({"metadata": trajectory.metadata}) - value = float(trajectory.metadata.get("reward", 0.0)) - return RewardResult(value=value, metadata={"success": bool(trajectory.metadata["success"])}) - - -# Registered so Miles reaches the wrapped callable as -# ``yeto.rl.adapters.miles.rewards.rlf_success`` (module ``__getattr__`` below -# wraps registered neutral rewards with :func:`miles_reward`). -register_reward("rlf_success", rlf_success_reward) - - def __getattr__(attr: str): if attr.startswith("__"): raise AttributeError(attr) diff --git a/yeto/rl/adapters/miles/rollout_meta_hook.py b/yeto/rl/adapters/miles/rollout_meta_hook.py index d34e761d9..b10d7e902 100644 --- a/yeto/rl/adapters/miles/rollout_meta_hook.py +++ b/yeto/rl/adapters/miles/rollout_meta_hook.py @@ -154,7 +154,7 @@ def record_trained_groups(args: Any, data: Sequence[Sequence[Any]]) -> None: mask_placeholder_logprobs(data, tokenizer_loader=_load_miles_tokenizer, args=args)) # CompactionRL: the masks above may drop optimised tokens; the per-segment # counts (tokens_after, gae_length) must follow the final masks. - from yeto.rl.harness.codex.codex_openenv_generate import refresh_segment_counts + from yeto.rl.compaction import refresh_segment_counts refresh_segment_counts([s for group in data for s in _flat(group)]) setattr(args, _TRAINED_ATTR, {_group_key(group) for group in data}) diff --git a/yeto/rl/adapters/miles/run_flags.py b/yeto/rl/adapters/miles/run_flags.py index 8a9f52027..3319f5d76 100644 --- a/yeto/rl/adapters/miles/run_flags.py +++ b/yeto/rl/adapters/miles/run_flags.py @@ -9,7 +9,6 @@ from __future__ import annotations import argparse -import json from collections.abc import Mapping from types import MappingProxyType @@ -241,7 +240,7 @@ "on few-core single-GPU islands to avoid the parallel-import storm " "(32 sglang imports can miss the 300 s ready window)", ), - shared_flag( + cli_flag( # refused since S26 E0.5 (removed Codex/SecRLEnv agents) "--codex-backend-profile", default=None, help=( @@ -337,7 +336,6 @@ island_flag("--groups-per-round", type=int, required=True), island_flag("--samples-per-group", type=int, required=True), island_flag("--optimizer-steps", type=int, required=True), - island_flag("--codex-harness-contract", type=json.loads, default=None), island_flag("--completed-groups-path", required=True), island_flag("--event-tape", required=True), island_flag("--audit-dir", default=None), diff --git a/yeto/rl/adapters/miles/sampler_island.py b/yeto/rl/adapters/miles/sampler_island.py index 22da4c7d9..da8645bc4 100644 --- a/yeto/rl/adapters/miles/sampler_island.py +++ b/yeto/rl/adapters/miles/sampler_island.py @@ -310,7 +310,7 @@ def emit(event: str, **fields: Any) -> None: incarnation, resume, emit) progress = {"rollout_id": None, "rounds": 0} # the round in flight, rounds done here - connect_island_ray(miles_args=miles_args) + connect_island_ray() # The rollout process's metadata hook pushes each round's group records to this # named actor (on a trainer island the driver creates it and consumes them). metadata = RayMetadataSink() diff --git a/yeto/rl/adapters/miles/secrlenv_miles.py b/yeto/rl/adapters/miles/secrlenv_miles.py deleted file mode 100644 index 70e5add6e..000000000 --- a/yeto/rl/adapters/miles/secrlenv_miles.py +++ /dev/null @@ -1,74 +0,0 @@ -"""Miles classes for the SecRLEnv same-task retry (P0-1.7b.4-Q1). - -Loaded by Miles in the rollout process (``--data-source-path`` and -``--rollout-function-path``, set by :func:`.secrlenv_retry.rollout_argv`); -the contract and its logic live in :mod:`.secrlenv_retry`. -""" - -from __future__ import annotations - -import asyncio -from typing import Any - -from miles.rollout.data_source import RolloutDataSourceWithBuffer -from miles.rollout.inference_rollout.inference_rollout_common import InferenceRolloutFn -from miles.rollout.inference_rollout.inference_rollout_train import abort - -from .secrlenv_retry import ( - RETRY_STATS_ATTR, - TERMINAL_ERRORS, - SecRLEnvTaskRetry, - first_terminal_error, - raise_after_terminal_cleanup, - retry_metrics, -) - - -class SecRLEnvRetryDataSource(SecRLEnvTaskRetry, RolloutDataSourceWithBuffer): - """Miles' buffered data source with the reserved same-task retry slot.""" - - -class SecRLEnvRolloutFn(InferenceRolloutFn): - """Miles' rollout function; on a terminal SecRLEnv error it drains first. - - The training round also reports the retry counters as rollout metrics. - """ - - async def _call_train(self, input: Any) -> Any: - try: - output = await super()._call_train(input) - except TERMINAL_ERRORS as error: - await raise_after_terminal_cleanup( - first_terminal_error(self.state.args, error), - # Miles' end-of-round abort: stop every engine request and - # call the agent's abort hook (tears down local episodes). - abort=lambda: abort(self.state, set(), input.rollout_id), - in_flight=_in_flight_groups, - require_clean=_require_no_episode_residue, - ) - raise # unreachable: the cleanup always raises - stats = getattr(self.state.args, RETRY_STATS_ATTR, None) - metrics = getattr(output, "metrics", None) - if isinstance(metrics, dict): - metrics.update(retry_metrics(stats)) - return output - - -# Miles keeps a round's group tasks in a local of generate_rollout_async; when -# a terminal error leaves that loop they are only reachable on the event loop. -_GROUP_TASK = "generate_and_rm_group" - - -def _in_flight_groups() -> list[asyncio.Task[Any]]: - current = asyncio.current_task() - return [ - task for task in asyncio.all_tasks() - if task is not current and not task.done() - and getattr(task.get_coro(), "__name__", None) == _GROUP_TASK - ] - - -def _require_no_episode_residue() -> None: - from yeto.rl.harness.codex.agent import require_no_episode_residue - - require_no_episode_residue() diff --git a/yeto/rl/adapters/miles/secrlenv_retry.py b/yeto/rl/adapters/miles/secrlenv_retry.py deleted file mode 100644 index 504b1de76..000000000 --- a/yeto/rl/adapters/miles/secrlenv_retry.py +++ /dev/null @@ -1,318 +0,0 @@ -"""SecRLEnv same-task retry on the ports engine (P0-1.7b.4-Q1). - -The SecRLEnv group check (``yeto.rl.harness.codex.reward.check_group``) -verifies each group's signed episode outcomes. A non-trainable group (an -infrastructure failure, or a sample that did not complete) is rejected and the -check asks for one exact rerun of the same task. This module holds the -Miles-independent half of that contract; :mod:`.secrlenv_miles` binds it to -Miles' data source and rollout function classes. - -Rollout contract (unchanged from the removed legacy rollout): - -* the round draws exactly ``rollout_batch_size`` new task groups; the - ``over_sampling_batch_size = rollout_batch_size + 1`` slot is reserved for - the retry, it is not over-sampling; -* at most ``SECRLENV_INFRASTRUCTURE_REPLACEMENTS`` (1) retry per round; a - refill without a queued retry, a second retry, or a retry whose task or - 503-retry ledger does not match the drawn task ends the run with a - sanitized :class:`SecRLEnvRolloutTerminalError`; -* on a terminal error the in-flight work is aborted and the run checks that no - episode is left behind before it raises. - -Miles marks a rollout boundary for the data source: its rollout function -draws the round's groups with ``get_samples`` and returns the aborted groups -with ``add_samples`` once the round ends. -""" - -from __future__ import annotations - -import asyncio -import copy -from collections.abc import Awaitable, Callable, Iterable, Mapping -from typing import Any - -from yeto.rl import SECRLENV_AGENTS, SECRLENV_INFRASTRUCTURE_REPLACEMENTS -from yeto.rl.engine.algorithm import SECRLENV_GROUP_CHECK_FILTER -from yeto.rl.harness.codex.reward import ( - INFRASTRUCTURE_503_RETRIES_KEY, - OUTCOME_KEY, - SecRLEnvReplacementExhausted, - SecRLEnvRolloutCleanupError, - SecRLEnvRolloutTerminalError, - TERMINAL_ERROR_ATTR, - SecRLEnvUntrustedEvidence, - UntrustedOutcome, - sample_statuses, -) - -DATA_SOURCE_PATH = "yeto.rl.adapters.miles.secrlenv_miles.SecRLEnvRetryDataSource" -ROLLOUT_FUNCTION_PATH = "yeto.rl.adapters.miles.secrlenv_miles.SecRLEnvRolloutFn" -# Read by the group check (reward.check_group) to queue the retry. -RETRY_CALLBACK_ATTR = "_yeto_secrlenv_retry_callback" -RETRY_STATS_ATTR = "_yeto_secrlenv_retry_stats" -# The group check's per-round decision memo; reset at each round start. -FILTER_STATE_ATTR = "_yeto_secrlenv_filter_state" -METRIC_PREFIX = "rl/secrlenv/" -TERMINAL_CLEANUP_TIMEOUT_S = 240.0 -_STAT_NAMES = ( - "replacement_attempts", - "nontrainable_groups", - "infrastructure_groups", - "pending_groups", - "failed_groups", -) - - -def rollout_argv(*, custom_agent: str | None, dynamic_filter: str | None, - rollout_batch_size: int, over_sampling_batch_size: int, - max_policy_age: int) -> tuple[str, ...]: - """Miles flags that bind the SecRLEnv retry; empty for any other agent. - - Raises ``ValueError`` when the agent and the group check are not used - together, when the refill slot is not exactly one group, or when the run - carries rollouts across rounds (the retry slot is counted per round). - """ - - secrlenv = custom_agent in SECRLENV_AGENTS - if secrlenv != (dynamic_filter == SECRLENV_GROUP_CHECK_FILTER): - raise ValueError( - f"the SecRLEnv agents and the {SECRLENV_GROUP_CHECK_FILTER!r} filter go " - f"together (agent {custom_agent!r}, filter {dynamic_filter!r})" - ) - if not secrlenv: - return () - if over_sampling_batch_size != rollout_batch_size + 1: - raise ValueError( - "the SecRLEnv agents reserve exactly one refill slot for the same-task " - f"retry: over-sampling must be {rollout_batch_size + 1} " - f"(got {over_sampling_batch_size})" - ) - if max_policy_age: - raise ValueError( - "the SecRLEnv same-task retry is counted per round; it does not support " - f"a policy-age limit (got {max_policy_age})" - ) - return ( - "--data-source-path", DATA_SOURCE_PATH, - "--rollout-function-path", ROLLOUT_FUNCTION_PATH, - ) - - -def _task_indices(group: object) -> tuple[int, ...] | None: - """The sample indexes of a group (nested sample lists use their first item).""" - - if not isinstance(group, list) or not group: - return None - values = [] - for item in group: - while isinstance(item, list): - if not item: - return None - item = item[0] - try: - values.append(int(item.index)) - except (AttributeError, TypeError, ValueError): - return None - return tuple(values) - - -def _strict_int(value: Any) -> bool: - return type(value) is int - - -class SecRLEnvTaskRetry: - """Data-source mixin: reserve the spare refill slot for one same-task retry. - - Put it before a Miles data source class (``RolloutDataSourceWithBuffer``): - it overrides ``get_samples``/``add_samples`` and defers the rest. - """ - - def __init__(self, args: Any) -> None: - super().__init__(args) - target = getattr(args, "rollout_batch_size", None) - capacity = getattr(args, "over_sampling_batch_size", None) - limit = getattr(args, "yeto_rl_secrlenv_max_infrastructure_replacements", None) - if ( - not _strict_int(target) - or target <= 0 - or capacity != target + 1 - or not _strict_int(limit) - or limit != SECRLENV_INFRASTRUCTURE_REPLACEMENTS - ): - raise SecRLEnvReplacementExhausted( - "SecRLEnv rollout replacement contract is invalid" - ) - self._retry_args = args - self._retry_target = target - self._retry_capacity = capacity - self._retry_limit = limit - self._round_open = False - self._reset_round() - setattr(args, RETRY_CALLBACK_ATTR, self.request_retry) - - def _reset_round(self) -> None: - self._templates: dict[tuple[int, ...], list[Any]] = {} - self._queued: list[list[Any]] = [] - self._seen_evidence: set[object] = set() - self._retries_used = 0 - self._retry_stats = dict.fromkeys(_STAT_NAMES, 0) - setattr(self._retry_args, RETRY_STATS_ATTR, self._retry_stats) - setattr(self._retry_args, FILTER_STATE_ATTR, None) - setattr(self._retry_args, TERMINAL_ERROR_ATTR, None) - - def get_samples(self, num_samples: int) -> list[list[Any]]: - if not _strict_int(num_samples) or num_samples != self._retry_capacity: - raise SecRLEnvReplacementExhausted("SecRLEnv rollout refill contract is invalid") - if not self._round_open: - return self._open_round() - if not self._queued: - raise SecRLEnvReplacementExhausted( - "SecRLEnv rollout refill had no authenticated task retry" - ) - groups, self._queued = self._queued, [] - return groups - - def _open_round(self) -> list[list[Any]]: - self._reset_round() - groups = super().get_samples(self._retry_target) - if not isinstance(groups, list) or len(groups) != self._retry_target: - raise SecRLEnvReplacementExhausted( - "SecRLEnv rollout did not receive its exact task batch" - ) - for group in groups: - key = _task_indices(group) - if key is None: - raise SecRLEnvReplacementExhausted("SecRLEnv rollout task identity is unavailable") - self._templates[key] = copy.deepcopy(group) - self._round_open = True - return groups - - def add_samples(self, samples: list[list[Any]]) -> None: - """Miles returns the round's aborted groups here once the round ends.""" - - super().add_samples(samples) - self._round_open = False - - def retry_stats(self) -> dict[str, int]: - return dict(self._retry_stats) - - def request_retry( - self, - group: list[Any], - *, - evidence_key: object, - reason: str, - infrastructure_503_retries: Mapping[int, int], - ) -> None: - """Queue one exact rerun of ``group``'s task (called by the group check).""" - - if evidence_key in self._seen_evidence: - return - self._seen_evidence.add(evidence_key) - if self._retries_used >= self._retry_limit or self._queued: - raise SecRLEnvReplacementExhausted("SecRLEnv rollout replacement budget was exhausted") - key = _task_indices(group) - template = self._templates.get(key) if key is not None else None - if template is None: - raise SecRLEnvReplacementExhausted( - "SecRLEnv rollout replacement task identity did not match" - ) - if ( - not isinstance(infrastructure_503_retries, Mapping) - or set(infrastructure_503_retries) != set(key) - or any( - not _strict_int(index) or not _strict_int(value) or value not in {0, 1} - for index, value in infrastructure_503_retries.items() - ) - ): - raise SecRLEnvUntrustedEvidence("SecRLEnv rollout retry ledger authentication failed") - retry = copy.deepcopy(template) - for sample in retry: - reset = getattr(sample, "reset_for_retry", None) - if not callable(reset): - raise SecRLEnvReplacementExhausted("SecRLEnv rollout sample cannot be reset safely") - reset() - metadata = getattr(sample, "metadata", None) - if isinstance(metadata, dict): - metadata.pop(OUTCOME_KEY, None) - metadata[INFRASTRUCTURE_503_RETRIES_KEY] = ( - infrastructure_503_retries[int(sample.index)] - ) - statuses = sample_statuses(group) - stats = self._retry_stats - stats["nontrainable_groups"] += 1 - stats["pending_groups"] += int("pending" in statuses) - stats["failed_groups"] += int("failed" in statuses) - stats["infrastructure_groups"] += int(reason == "secrlenv_infrastructure_failure") - self._retries_used += 1 - stats["replacement_attempts"] = self._retries_used - self._queued.append(retry) - - -def retry_metrics(stats: Mapping[str, int] | None) -> dict[str, float]: - """The round's retry counters as rollout metrics (``rl/secrlenv/``).""" - - if not isinstance(stats, Mapping): - return {} - return {f"{METRIC_PREFIX}{name}": float(stats.get(name, 0)) for name in _STAT_NAMES} - - -TERMINAL_ERRORS = (SecRLEnvRolloutTerminalError, UntrustedOutcome) - - -def first_terminal_error(args: Any, error: BaseException) -> BaseException: - """The round's first terminal error, else ``error``. - - Miles swallows an exception raised inside a generate task (it only counts - the group as failed); the round then stops later on a different error, - e.g. a refill without a queued retry. The SecRLEnv code records the first - one on ``args`` (``reward.remember_terminal_error``). - """ - - recorded = getattr(args, TERMINAL_ERROR_ATTR, None) - return recorded if isinstance(recorded, TERMINAL_ERRORS) else error - - -async def cancel_and_wait(tasks: Iterable[asyncio.Task[Any]]) -> None: - """Cancel ``tasks`` and wait until every one has finished.""" - - pending = [task for task in tasks if not task.done()] - for task in pending: - task.cancel() - if pending: - await asyncio.gather(*pending, return_exceptions=True) - - -async def raise_after_terminal_cleanup( - error: BaseException, - *, - abort: Callable[[], Awaitable[Any]], - in_flight: Callable[[], Iterable[asyncio.Task[Any]]], - require_clean: Callable[[], None], - timeout_s: float = TERMINAL_CLEANUP_TIMEOUT_S, -) -> None: - """Stop the in-flight work after a terminal error, then raise it sanitized. - - Order: ``abort`` stops the engines and the agent's episodes; the group - tasks ``in_flight`` returns are cancelled and awaited (the backend loop - that owned them has already exited); ``require_clean`` raises if an - episode is left behind. A cleanup failure or timeout (one bound for the - whole sequence) raises :class:`SecRLEnvRolloutCleanupError` instead. - Error text never carries episode payloads (``from None``). - """ - - if isinstance(error, SecRLEnvRolloutTerminalError): - terminal: SecRLEnvRolloutTerminalError = error - else: - terminal = SecRLEnvUntrustedEvidence("SecRLEnv rollout evidence authentication failed") - - async def stop() -> None: - await abort() - await cancel_and_wait(in_flight()) - - try: - await asyncio.wait_for(stop(), timeout=timeout_s) - require_clean() - except Exception: - raise SecRLEnvRolloutCleanupError("SecRLEnv terminal rollout cleanup failed") from None - raise terminal from None diff --git a/yeto/rl/algos/compactionrl.py b/yeto/rl/algos/compactionrl.py index 31fc46315..bb35a8c8e 100644 --- a/yeto/rl/algos/compactionrl.py +++ b/yeto/rl/algos/compactionrl.py @@ -92,8 +92,9 @@ def compactionrl_spec(**overrides: Any) -> AlgorithmSpec: # --- Rollout-side switch (progress.md "S13 Codex 压缩接线") ------------------------ # The trainer-side gae variant decides whether rollouts must be compacted: # ``cross_segment_per_sample`` (this spec) needs one sample per compaction -# segment, which only the Codex bridge (``harness.codex.compaction_bridge``, -# switched on by ``YETO_CODEX_COMPACTIONRL``) produces. The control arm +# segment. The Codex bridge that produced them (switched on by +# ``YETO_CODEX_COMPACTIONRL``) was removed in S26 E0.5; only the TEST synthetic +# reward (``yeto.rl.synthetic_segments``) labels segments now. The control arm # ``cross_segment_whole_rollout`` needs the whole rollout as ONE sample with # per-token segment ids; the bridge runs every segment in its own session-server # session with a rebuilt context, so its segments cannot be concatenated into @@ -102,9 +103,6 @@ def compactionrl_spec(**overrides: Any) -> AlgorithmSpec: COMPACTION_SWITCH_ENV = "YETO_CODEX_COMPACTIONRL" COMPACTION_T_COMP_ENV = "YETO_CODEX_COMPACTIONRL_T_COMP" COMPACTED_GAE_VARIANT = "cross_segment_per_sample" -# yeto's Codex generate wrapper (segment collection); the launcher checks it -# without importing the adapter layer (import boundary, yeto/rl/engine/README.md). -CODEX_OPENENV_GENERATE = "yeto.rl.harness.codex.codex_openenv_generate.generate" WHOLE_ROLLOUT_CONTROL_GAE_VARIANT = "cross_segment_whole_rollout" _SWITCH_FALSE = frozenset({"", "0", "false", "no", "off"}) _SWITCH_TRUE = frozenset({"1", "true", "yes", "on"}) @@ -138,7 +136,7 @@ def check_rollout_compaction( """Fail closed unless spec, harness and switch agree. ``gae_variant``: the spec's ``advantage.gae_variant`` (= Miles ``--gae-variant``); - ``harness_compacts``: the run's agent can compact (the Codex OpenEnv agent). + ``harness_compacts``: the run produces compaction segments. """ if gae_variant == WHOLE_ROLLOUT_CONTROL_GAE_VARIANT and harness_compacts: raise ValueError( @@ -151,7 +149,7 @@ def check_rollout_compaction( if not harness_compacts: raise ValueError( f"gae_variant {COMPACTED_GAE_VARIANT} (CompactionRL) needs compacted rollouts; " - "only the Codex OpenEnv harness can compact" + "no agent harness compacts (only the TEST synthetic segments reward)" ) if not switch_on: raise ValueError( diff --git a/yeto/rl/compaction.py b/yeto/rl/compaction.py index c23e4b36a..dd4730e03 100644 --- a/yeto/rl/compaction.py +++ b/yeto/rl/compaction.py @@ -336,3 +336,40 @@ def validate_compaction_segments(segments: Sequence[Any]) -> None: for value in identity ): raise RuntimeError("Miles compaction segments changed trajectory identity") + + +def _optimized_tokens(sample: Any) -> int: + mask = getattr(sample, "loss_mask", None) or [] + return int(sum(1 for value in mask if value)) + + +def refresh_segment_counts(samples: Any) -> int: + """Recompute ``segment_tokens``/``tokens_after``/``gae_length`` from the final loss masks. + + Segment samples (``metadata.segment_index``) count optimised tokens when + they are assembled; later hooks still set loss_mask 1 -> 0 (S19 #8 + placeholder-logprob stop tokens, S17 WP6 reasoning tokens) before the + backend converts the samples. N_{>s} and l are optimised-token counts, so + they are refreshed after those hooks (s19-compaction-g1-20261010g: + tokens_after was 1-17 tokens high). Returns the number of segment samples + whose counts changed. + """ + rollouts: dict[Any, list[Any]] = {} + for sample in samples: + meta = getattr(sample, "metadata", None) + if not isinstance(meta, dict) or not isinstance(meta.get("segment_index"), int): + continue + key = (getattr(sample, "rollout_id", None), getattr(sample, "group_index", None), + getattr(sample, "index", None)) + rollouts.setdefault(key, []).append(sample) + changed = 0 + for segments in rollouts.values(): + segments.sort(key=lambda s: s.metadata["segment_index"]) + tokens = [_optimized_tokens(s) for s in segments] + for index, sample in enumerate(segments): + new = {"segment_tokens": tokens[index], "tokens_after": sum(tokens[index + 1:]), + "gae_length": sum(tokens)} + if any(sample.metadata.get(k) != v for k, v in new.items()): + changed += 1 + sample.metadata.update(new) + return changed diff --git a/yeto/rl/engine/algorithm.py b/yeto/rl/engine/algorithm.py index 95e67a837..67bc5283e 100644 --- a/yeto/rl/engine/algorithm.py +++ b/yeto/rl/engine/algorithm.py @@ -52,20 +52,10 @@ # *new* ``AlgorithmSpec.sha256()`` (hash-migration.md, 阶段 4 记录). BOUNDED_NONZERO_STD_FILTER = "nonzero_reward_std_bounded" STOCK_NONZERO_STD_FILTER = "nonzero_reward_std" -# SecRLEnv group check (P0-1.7b.4-Q1): verifies each group's signed episode -# outcomes, rejects non-trainable groups and asks the backend to rerun the -# same task once; zero-variance groups follow the bounded rule. -SECRLENV_GROUP_CHECK_FILTER = "secrlenv_group_check" -# Terminal-Bench group check (S24 T-TB2): validates the signed TB2 outcomes and -# drops a group with any infrastructure (aborted) member or zero reward -# variance, so an environment failure never reaches the advantage. -TBENCH_GROUP_CHECK_FILTER = "tbench_group_check" LEGACY_FILTER_NAMES = { "yeto.rl.filters.bounded_nonzero_reward_std": BOUNDED_NONZERO_STD_FILTER, "miles.rollout.filter_hub.dynamic_sampling_filters.check_reward_nonzero_std": STOCK_NONZERO_STD_FILTER, - "yeto.rl.harness.codex.reward.check_group": SECRLENV_GROUP_CHECK_FILTER, - "yeto.rl.harness.codex.tbench_reward.check_group": TBENCH_GROUP_CHECK_FILTER, } @@ -75,8 +65,7 @@ def normalize_filter_name(name: str | None) -> str | None: # v1 (R0) vocabulary: what the flat v1 fields can express. SUPPORTED_ADVANTAGE_ESTIMATORS = frozenset({"grpo", "ppo"}) # ppo: rl-algo-critic-family 2.2 SUPPORTED_LOSSES = frozenset({"policy_loss"}) -SUPPORTED_FILTERS = frozenset({BOUNDED_NONZERO_STD_FILTER, SECRLENV_GROUP_CHECK_FILTER, - TBENCH_GROUP_CHECK_FILTER}) +SUPPORTED_FILTERS = frozenset({BOUNDED_NONZERO_STD_FILTER}) # v2 vocabulary: what the structured spec can *express* (upstream Miles # ``arguments.py`` choices at MILES_NEXT_COMMIT). Expressible is not @@ -136,10 +125,7 @@ def normalize_filter_name(name: str | None) -> str | None: ) # Pre-D1 names of the bounds, still accepted on input (constructor and JSON). CORRECTION_FIELD_ALIASES = {"tis_clip": "upper", "tis_clip_low": "lower"} -DYNAMIC_SAMPLING_FILTERS = { - BOUNDED_NONZERO_STD_FILTER, STOCK_NONZERO_STD_FILTER, SECRLENV_GROUP_CHECK_FILTER, - TBENCH_GROUP_CHECK_FILTER, -} +DYNAMIC_SAMPLING_FILTERS = {BOUNDED_NONZERO_STD_FILTER, STOCK_NONZERO_STD_FILTER} # Plugins may only live in yeto or in Miles' own (built-in) namespace (D7). PLUGIN_NAMESPACES = ["yeto.", "miles."] diff --git a/yeto/rl/engine/backends.py b/yeto/rl/engine/backends.py index 243e07c98..ad12e92f2 100644 --- a/yeto/rl/engine/backends.py +++ b/yeto/rl/engine/backends.py @@ -49,7 +49,6 @@ class BackendEntry: # publish publisher (test-only fault injection env names) # algorithm_flags AlgorithmSpec -> backend argv (critic warmup) # selection_argv facts read from pass-through backend argv (--rl-engine matrix) - # harness_glue codex OpenEnv generate: upstream generate, aborted status, session collect # launch_flags island-entry flag spellings of the neutral layout (launcher) # run_config_rules backend-only checks of resolve_rl_run_config (ref-load format, divisibility) # binding neutral name -> backend binding check (4.4a) @@ -74,7 +73,6 @@ class BackendEntry: "publish": f"{_MILES}.publish", "algorithm_flags": f"{_MILES}.algorithm_flags", "selection_argv": f"{_MILES}.selection_argv", - "harness_glue": f"{_MILES}.harness_glue.codex_openenv", "launch_flags": f"{_MILES}.launch_flags", "run_flags": f"{_MILES}.run_flags", "run_config_rules": f"{_MILES}.run_config_rules", diff --git a/yeto/rl/engine/progress.py b/yeto/rl/engine/progress.py index 71441da40..9c1891f13 100644 --- a/yeto/rl/engine/progress.py +++ b/yeto/rl/engine/progress.py @@ -54,11 +54,9 @@ def _island_checkpoint_config(args) -> dict[str, Any]: "dynamic_sampling_max_replacements": getattr( args, "yeto_rl_dynamic_sampling_max_replacements", None ), - "secrlenv_max_infrastructure_replacements": getattr( - args, - "yeto_rl_secrlenv_max_infrastructure_replacements", - None, - ), + # The SecRLEnv and Codex agents left in S26 E0.5; the keys stay (None) + # so island checkpoints written before that still resume. + "secrlenv_max_infrastructure_replacements": None, "rl_offload_train": bool(getattr(args, "offload_train", False)), "rl_distributed_timeout_minutes": getattr( args, "distributed_timeout_minutes", 10 @@ -69,17 +67,8 @@ def _island_checkpoint_config(args) -> dict[str, Any]: "custom_generate_function_path": args.custom_generate_function_path, "use_session_server": args.use_session_server, "tito_model": args.tito_model, - "codex_backend_profile": getattr( - args, "yeto_rl_codex_backend_profile", None - ), + "codex_backend_profile": None, } - codex_harness_contract = getattr( - args, - "yeto_rl_codex_harness_contract", - None, - ) - if codex_harness_contract is not None: - config["codex_harness_contract"] = codex_harness_contract initial_adapter_sha256 = getattr( args, "yeto_rl_initial_adapter_sha256", diff --git a/yeto/rl/engine/run_flags.py b/yeto/rl/engine/run_flags.py index b4de91487..7ab563b2f 100644 --- a/yeto/rl/engine/run_flags.py +++ b/yeto/rl/engine/run_flags.py @@ -301,7 +301,7 @@ def rl_forward_pairs(*dests: str | tuple[str, str]) -> tuple[tuple[str, str], .. "fallback after this many rejected groups" ), ), - shared_flag( + cli_flag( # refused since S26 E0.5 (removed Codex/SecRLEnv agents) "--secrlenv-max-infrastructure-replacements", type=int, default=None, @@ -798,7 +798,7 @@ def rl_forward_pairs(*dests: str | tuple[str, str]) -> tuple[tuple[str, str], .. _AGENT = ( shared_flag("--apply-chat-template-kwargs", type=json.loads, default=None), shared_flag("--custom-generate-function-path", default=None), - shared_flag( + cli_flag( # refused since S26 E0.5 (removed Codex/SecRLEnv agents) "--codex-reasoning-effort", choices=["xhigh"], default=None, @@ -811,7 +811,7 @@ def rl_forward_pairs(*dests: str | tuple[str, str]) -> tuple[tuple[str, str], .. shared_flag("--session-server-ip", default=None), shared_flag("--session-server-port", type=int, nargs="+", default=None), shared_flag("--tito-model", default=None), - shared_flag( + cli_flag( # refused since S26 E0.5 (removed Codex/SecRLEnv agents) "--codex-exclude-reasoning-from-loss", action="store_true", help=( diff --git a/yeto/rl/eval/codex_infer.py b/yeto/rl/eval/codex_infer.py deleted file mode 100644 index 3adc1206e..000000000 --- a/yeto/rl/eval/codex_infer.py +++ /dev/null @@ -1,420 +0,0 @@ -"""Eval-island inference agent: SGLang + Miles session server + the training Codex driver -(rl-eval-difficulty-buckets 5.3/5.9). - -What runs inside the eval-island container, in this order: - -1. :func:`write_peft_dir` -- rebuild the canonical LoRA state from the stored - ``policy.safetensors`` / ``policy.json`` (the store already checked every - file's sha256), require its ``policy_tensor_hash`` to equal the manifest's, - and write a standard PEFT adapter directory; -2. :class:`SglangSessionLoader` (the island's ``PolicyLoader``) -- start one - SGLang server for the base model with LoRA enabled, load the adapter under - Miles' served adapter name (``miles_lora``), and start Miles' own standalone - session server (``miles.rollout.session.server``) in front of it with the - training TITO settings; the next version only swaps the adapter. ``load`` - returns the manifest's ``rl/policy_token`` only after the tensor-hash check - and a successful adapter load (fail closed otherwise); -3. :class:`CodexEvalAgent` (the ``tb2_attempt`` agent port) -- one evaluation - session per attempt, then exactly the training rollout path: the task - statement from ``task_prompt``, the Codex worker subprocess - (``codex_openenv_subprocess_agent_function._drive_worker``, the same - ``drive_untrusted`` + stock Responses bridge), the lease's worker env and - deadline. Judging stays on the trusted side (``tb2_attempt``). - -Nothing here re-implements the harness; the only new pieces are process -supervision and the PEFT writer. Factories for ``island_main``: -``yeto.rl.eval.codex_infer:loader_factory`` / ``:attempt_factory``. -""" - -from __future__ import annotations - -import asyncio -import json -import os -import subprocess -import sys -import time -import urllib.error -import urllib.request -from dataclasses import dataclass, field -from pathlib import Path -from typing import Any, Callable, Mapping - -SERVED_LORA_NAME = "miles_lora" # miles.utils.lora.utils.LORA_ADAPTER_NAME (the session server sends it) -STATUS_TO_END_REASON = {"completed": "completed", "timeout": "timed_out", - "max_turns": "max_turns", "max_seq_len": "max_seq_len"} - - -# --- 1. stored policy -> PEFT adapter dir ------------------------------------------------ - - -def peft_adapter_config(*, rank: int, targets: list[str], base_model: str, revision: str | None) -> dict[str, Any]: - """The fields ``yeto.rl.initial_adapter.load_initial_adapter`` accepts (alpha = rank, no dropout).""" - return {"peft_type": "LORA", "task_type": "CAUSAL_LM", "r": int(rank), "lora_alpha": int(rank), - "target_modules": sorted(targets), "lora_dropout": 0.0, "bias": "none", "use_rslora": False, - "use_dora": False, "fan_in_fan_out": False, "rank_pattern": {}, "alpha_pattern": {}, - "inference_mode": True, "base_model_name_or_path": base_model, "revision": revision} - - -def lora_rank_and_targets(shapes: Mapping[str, tuple[int, ...]]) -> tuple[int, list[str]]: - """Rank from the ``lora_A`` rows (must be one value); targets = module leaf names.""" - ranks = {int(shape[0]) for name, shape in shapes.items() if ".lora_A." in name} - if len(ranks) != 1: - raise ValueError(f"stored adapter has no single LoRA rank: {sorted(ranks)}") - targets = sorted({name.rsplit(".lora_", 1)[0].rsplit(".", 1)[-1] for name in shapes}) - return ranks.pop(), targets - - -def write_peft_dir(files_dir: str | Path, manifest: Mapping[str, Any], out_dir: str | Path, *, - base_model: str) -> dict[str, Any]: - """Recompute ``policy_tensor_hash`` from the stored bytes, then write the PEFT dir. Needs torch.""" - from safetensors.torch import load_file, save_file - - from .export import POLICY_IDENTITY, POLICY_TENSORS, verify_policy_tensor_hash - - t0 = time.monotonic() - verify_policy_tensor_hash(files_dir, manifest) # raises EvalIntegrityError on mismatch - root = Path(files_dir) - identity = json.loads((root / POLICY_IDENTITY).read_text()) - tensors = load_file(str(root / POLICY_TENSORS)) - rank, targets = lora_rank_and_targets({k: tuple(v.shape) for k, v in tensors.items()}) - out = Path(out_dir) - out.mkdir(parents=True, exist_ok=True) - save_file(tensors, str(out / "adapter_model.safetensors")) - config = peft_adapter_config(rank=rank, targets=targets, base_model=base_model, - revision=identity.get("base_model_revision")) - (out / "adapter_config.json").write_text(json.dumps(config, indent=1, sort_keys=True)) - return {"rank": rank, "targets": targets, "tensors": len(tensors), "path": str(out), - "seconds": round(time.monotonic() - t0, 3)} - - -# --- 2. SGLang + session server ---------------------------------------------------------- - - -def _http(method: str, url: str, body: Any = None, *, timeout: float = 60.0) -> tuple[int, Any]: - data = None if body is None else json.dumps(body).encode() - req = urllib.request.Request(url, data=data, method=method, headers={"Content-Type": "application/json"}) - try: - with urllib.request.urlopen(req, timeout=timeout) as resp: - raw = resp.read() - status = resp.status - except urllib.error.HTTPError as exc: - raw, status = exc.read(), exc.code - try: - return status, json.loads(raw) if raw else None - except ValueError: - return status, raw.decode("utf-8", "replace") - - -@dataclass -class InferConfig: - """Serving settings; the defaults are the S17 M1 codex shape (qwen35_08b, LoRA r16 attention).""" - - base_model: str # HF id or local snapshot dir - revision: str | None = None - tito_model: str = "qwen35" - chat_template_kwargs: Mapping[str, Any] = field(default_factory=lambda: {"clear_thinking": False}) - context_length: int = 8192 # --agent-max-seq-len (the bridge's max_seq_len) - max_tokens: int = 4096 # --rollout-max-response-len (YETO_CODEX_BACKEND_MAX_TOKENS) - temperature: float = 1.0 # Miles rollout defaults - top_p: float = 1.0 - top_k: int | None = None - mem_fraction_static: float = 0.8 - host: str = "127.0.0.1" - sglang_port: int = 30000 - session_port: int = 30100 - work_dir: str = "/tmp/yeto-eval-infer" - sglang_extra_args: tuple[str, ...] = () - start_timeout_s: float = 900.0 - log_dir: str | None = None - - @classmethod - def from_mapping(cls, raw: Mapping[str, Any]) -> "InferConfig": - known = {k: v for k, v in raw.items() if k in cls.__dataclass_fields__} - if "sglang_extra_args" in known: - known["sglang_extra_args"] = tuple(known["sglang_extra_args"]) - return cls(**known) - - @property - def sglang_url(self) -> str: - return f"http://{self.host}:{self.sglang_port}" - - @property - def session_url(self) -> str: - return f"http://{self.host}:{self.session_port}" - - def request_kwargs(self) -> dict[str, Any]: - out: dict[str, Any] = {"temperature": self.temperature, "top_p": self.top_p, "max_tokens": self.max_tokens} - if self.top_k is not None: - out["top_k"] = self.top_k - return out - - -def resolve_serving(tito_model: str) -> dict[str, Any]: - """What the training island derives from ``--tito-model`` (Miles' own resolvers): - the SGLang reasoning / tool-call parsers (``yeto.rl.adapters.miles.config`` adds - them as ``--sglang-*-parser``) and the family's fixed chat template + kwargs - (``miles.utils.arguments``: ``chat_template_path`` for the session server, - ``--sglang-chat-template`` for SGLang).""" - from miles.utils.chat_template_utils import resolve_fixed_chat_template, resolve_reasoning_and_tool_call_parser - - reasoning, tool_call = resolve_reasoning_and_tool_call_parser(tito_model) - template, kwargs = resolve_fixed_chat_template(tito_model) - return {"reasoning_parser": reasoning, "tool_call_parser": tool_call, - "chat_template_path": template, "chat_template_kwargs": dict(kwargs)} - - -def sglang_argv(cfg: InferConfig, *, rank: int, targets: list[str], python: str = sys.executable, - serving: Mapping[str, Any] | None = None) -> list[str]: - """``rank == 0`` serves the base model alone (no LoRA flags): the baseline shape.""" - argv = [python, "-m", "sglang.launch_server", "--model-path", cfg.base_model, - "--host", cfg.host, "--port", str(cfg.sglang_port), "--trust-remote-code", - "--context-length", str(cfg.context_length), "--mem-fraction-static", str(cfg.mem_fraction_static)] - if rank: - argv += ["--enable-lora", "--max-lora-rank", str(rank), "--lora-target-modules", *targets, - "--max-loaded-loras", "2", "--max-loras-per-batch", "1"] - if cfg.revision and not Path(cfg.base_model).exists(): - argv += ["--revision", cfg.revision] - serving = serving or {} - if serving.get("reasoning_parser") is not None: - argv += ["--reasoning-parser", serving["reasoning_parser"]] - if serving.get("tool_call_parser") is not None: - argv += ["--tool-call-parser", serving["tool_call_parser"]] - if serving.get("chat_template_path") is not None: - argv += ["--chat-template", serving["chat_template_path"]] - return argv + list(cfg.sglang_extra_args) - - -def session_server_config(cfg: InferConfig, *, rank: int, serving: Mapping[str, Any] | None = None) -> dict[str, Any]: - """``miles.rollout.session.config.SessionServerConfig`` fields, as the training island sets them - (``compute_session_server_config``) for a LoRA codex run without replay / speculative decoding. - ``rank == 0`` with no adapter path is Miles' only LoRA-free shape (``is_lora_enabled``).""" - serving = serving or {} - kwargs = dict(cfg.chat_template_kwargs) - for key, value in (serving.get("chat_template_kwargs") or {}).items(): - if key in kwargs and kwargs[key] != value: # miles.utils.arguments: conflicting values are invalid - raise ValueError(f"chat template kwarg {key}={kwargs[key]!r} conflicts with the {cfg.tito_model} " - f"family's fixed {value!r}") - kwargs[key] = value - return { - "host": cfg.host, "port": cfg.session_port, "instance_id": "eval-island", - "backend_url": cfg.sglang_url, "timeout": 1800.0, "hf_checkpoint": cfg.base_model, - "chat_template_path": serving.get("chat_template_path"), "tito_model": cfg.tito_model, - "apply_chat_template_kwargs": kwargs, - "use_rollout_routing_replay": False, "use_rollout_indexer_replay": False, - "use_sampling_support_replay": False, "sglang_speculative_algorithm": None, - "num_layers": None, "moe_router_topk": None, "save_debug_trajectory_data": None, - "lora_rank": int(rank), "lora_adapter_path": None, "lora_train_only": False, - "use_session_server": True, "session_message_matcher": "strict", "pause_generation_mode": None, - "session_sample_picker_path": None, "session_sample_postprocessor_path": None, - } - - -SESSION_SERVER_BOOT = ( - "import json, sys\n" - "from miles.rollout.session.config import SessionServerConfig\n" - "from miles.rollout.session.server import run_session_server\n" - "run_session_server(SessionServerConfig(**json.loads(sys.argv[1])))\n" -) - - -class SglangSessionLoader: - """``PolicyLoader`` for ``EvalIsland``: one SGLang + one session server per island.""" - - def __init__(self, cfg: InferConfig, *, popen: Callable[..., Any] = subprocess.Popen, - http: Callable[..., tuple[int, Any]] = _http, sleep: Callable[[float], None] = time.sleep, - clock: Callable[[], float] = time.monotonic, emit: Callable[..., None] | None = None, - serving: Callable[[str], Mapping[str, Any]] = resolve_serving) -> None: - self.cfg = cfg - self.serving = serving - self.popen = popen - self.http = http - self.sleep = sleep - self.clock = clock - self.emit = emit - self.procs: dict[str, Any] = {} - self.served: dict[str, Any] | None = None - self.loads: list[dict[str, Any]] = [] - - def _spawn(self, name: str, argv: list[str]) -> None: - log = None - if self.cfg.log_dir: - Path(self.cfg.log_dir).mkdir(parents=True, exist_ok=True) - log = open(Path(self.cfg.log_dir) / f"{name}.log", "ab") - self.procs[name] = self.popen(argv, stdout=log, stderr=subprocess.STDOUT if log else None, - start_new_session=True) - - def _wait(self, name: str, url: str) -> float: - t0 = self.clock() - while True: - proc = self.procs.get(name) - if proc is not None and proc.poll() is not None: - raise RuntimeError(f"{name} exited with {proc.returncode} before it was ready") - try: - status, _ = self.http("GET", url, timeout=10.0) - except OSError: - status = None - if status == 200: - return round(self.clock() - t0, 3) - if self.clock() - t0 > self.cfg.start_timeout_s: - raise TimeoutError(f"{name} not ready after {self.cfg.start_timeout_s} s ({url})") - self.sleep(2.0) - - def _start(self, rank: int, targets: list[str]) -> dict[str, float]: - serving = dict(self.serving(self.cfg.tito_model)) - self._spawn("sglang", sglang_argv(self.cfg, rank=rank, targets=targets, serving=serving)) - sglang_s = self._wait("sglang", f"{self.cfg.sglang_url}/health_generate") - self._spawn("session-server", [sys.executable, "-c", SESSION_SERVER_BOOT, - json.dumps(session_server_config(self.cfg, rank=rank, serving=serving))]) - session_s = self._wait("session-server", f"{self.cfg.session_url}/health") - self.served = {"rank": rank, "targets": list(targets)} - return {"sglang_start_s": sglang_s, "session_server_start_s": session_s, "serving": serving} - - def start_base(self) -> dict[str, Any]: - """Serve the base model without LoRA (baseline runs: no stored policy to load).""" - if self.served is not None: - raise RuntimeError(f"serving already started: {self.served}") - return self._start(0, []) - - def load(self, manifest: Mapping[str, Any], files_dir: Path) -> str: - version = int(manifest["policy_version"]) - t0 = self.clock() - adapter = write_peft_dir(files_dir, manifest, Path(self.cfg.work_dir) / f"v{version:06d}", - base_model=str(manifest.get("base_model") or self.cfg.base_model)) - record: dict[str, Any] = {"policy_version": version, "verify_and_write_s": adapter["seconds"], - "rank": adapter["rank"], "targets": adapter["targets"]} - if self.served is None: - record.update(self._start(adapter["rank"], adapter["targets"])) - elif [adapter["rank"], adapter["targets"]] != [self.served["rank"], self.served["targets"]]: - raise RuntimeError(f"v{version}: LoRA shape {adapter['rank']}/{adapter['targets']} differs from the " - f"served {self.served}; restart the island") - else: - self.http("POST", f"{self.cfg.sglang_url}/unload_lora_adapter", {"lora_name": SERVED_LORA_NAME}) - t1 = self.clock() - status, body = self.http("POST", f"{self.cfg.sglang_url}/load_lora_adapter", - {"lora_name": SERVED_LORA_NAME, "lora_path": adapter["path"]}, timeout=600.0) - if status != 200 or (isinstance(body, dict) and body.get("success") is False): - raise RuntimeError(f"v{version}: SGLang refused the adapter: HTTP {status} {str(body)[:300]}") - record["adapter_load_s"] = round(self.clock() - t1, 3) - record["load_total_s"] = round(self.clock() - t0, 3) - self.loads.append(record) - if self.emit is not None: - self.emit("rl_eval_policy_load", **record) - return str(manifest["rl/policy_token"]) - - def close(self) -> None: - for proc in self.procs.values(): - if proc.poll() is None: - proc.terminate() - for proc in self.procs.values(): - try: - proc.wait(timeout=30) - except Exception: # noqa: BLE001 - proc.kill() - - -# --- 3. the agent port -------------------------------------------------------------------- - -DETAIL_KEYS = ("end_reason", "last_completion", "parse_failures", "tool_calls", "turns", - "max_model_total_tokens", "timed_out", "max_seq_len_hit", "tito_session_mismatch") - - -def agent_detail(metrics: Mapping[str, Any] | None) -> dict[str, Any] | None: - """Why the episode ended, as the stock driver recorded it (bounded; raw data for the run record).""" - if not metrics: - return None - out = {k: metrics[k] for k in DETAIL_KEYS if k in metrics} - if isinstance(out.get("last_completion"), (str, dict, list)): - out["last_completion"] = json.dumps(out["last_completion"], default=str)[:1500] - return out - - -def _default_drive() -> Callable[..., Any]: - from yeto.rl.harness.codex.codex_openenv_subprocess_agent_function import _drive_worker - - return _drive_worker - - -class CodexEvalAgent: - """``agent(lease, task, trial, *, policy_token)`` for ``tb2_attempt``.""" - - def __init__(self, cfg: InferConfig, *, drive: Callable[..., Any] | None = None, - http: Callable[..., tuple[int, Any]] = _http, clock: Callable[[], float] = time.monotonic) -> None: - self.cfg = cfg - self.drive = drive - self.http = http - self.clock = clock - - async def _session(self) -> str: - body = {"evaluation": True, "temperature": float(self.cfg.temperature), "top_p": float(self.cfg.top_p)} - if self.cfg.top_k is not None: - body["top_k"] = int(self.cfg.top_k) - status, out = await asyncio.to_thread(self.http, "POST", f"{self.cfg.session_url}/sessions", body) - if status != 200 or not isinstance(out, dict) or not out.get("session_id"): - raise RuntimeError(f"session server refused a session: HTTP {status} {str(out)[:200]}") - return str(out["session_id"]) - - async def __call__(self, lease: Any, task: Any, trial: int, *, policy_token: str) -> dict[str, Any]: - from yeto.rl.harness.codex import codex_openenv_agent_function as adapter - from yeto.rl.harness.codex.codex_openenv_subprocess_agent_function import task_prompt - - drive = self.drive or _default_drive() - session_id = await self._session() - episode_id = adapter.new_episode_id() - job = { - "base_url": f"{self.cfg.session_url}/sessions/{session_id}", - "prompt": task_prompt(dict(getattr(task, "row", {}).get("metadata") or {}), None, lease), - "request_kwargs": self.cfg.request_kwargs(), - "episode_id": episode_id, - "max_seq_len": self.cfg.context_length, - "env_url": lease.env_url, - "env_token": lease.env_token, - "driver": "stock", - } - trajectory = f"eval-{task.task_id}-t{trial}-{episode_id}" - t0 = self.clock() - try: - try: - untrusted = await asyncio.wait_for( - drive(job, trajectory, None, getattr(lease, "worker_env", None)), - timeout=float(lease.deadline_seconds)) - except asyncio.TimeoutError: - untrusted = {"status": "timeout", "metrics": {"timed_out": 1}, "episode_id": episode_id} - except adapter.harness.CodexHarnessError as exc: # training: infrastructure, not a reward - return {"episode_id": episode_id, "end_reason": "infra_error", "error": str(exc)[:500], - "metrics": getattr(exc, "metrics", None), "agent_s": round(self.clock() - t0, 3), - "agent_detail": agent_detail(getattr(exc, "metrics", None))} - finally: - await asyncio.to_thread(self.http, "DELETE", f"{self.cfg.session_url}/sessions/{session_id}") - metrics = dict(untrusted.get("metrics") or {}) - status = str(untrusted.get("status")) - if status not in STATUS_TO_END_REASON: - return {"episode_id": episode_id, "end_reason": "infra_error", "error": f"unknown status {status!r}"} - return {"episode_id": untrusted.get("episode_id", episode_id), "end_reason": STATUS_TO_END_REASON[status], - "turns": metrics.get("turns"), "tokens": metrics.get("max_model_total_tokens"), - "metrics": metrics, "agent_s": round(self.clock() - t0, 3), "agent_detail": agent_detail(metrics)} - - -# --- island_main factories ---------------------------------------------------------------- - -_LOADERS: dict[int, SglangSessionLoader] = {} - - -def loader_factory(config: Mapping[str, Any]) -> SglangSessionLoader: - cfg = InferConfig.from_mapping(config.get("infer") or {}) - loader = SglangSessionLoader(cfg) - _LOADERS[id(config)] = loader - return loader - - -def attempt_factory(config: Mapping[str, Any]) -> Callable[..., Mapping[str, Any]]: - """``tb2_attempt`` over the Modal reward-env provider (prebaked images) and :class:`CodexEvalAgent`.""" - from .tb2_attempt import tb2_attempt - - provider_spec = config.get("provider") or os.environ.get("YETO_HARNESS_ENVIRONMENT_PROVIDER") \ - or "yeto.cloud.modal_reward_env:modal_provider" - module, _, name = provider_spec.partition(":") - import importlib - - provider = getattr(importlib.import_module(module), name)(None) - return tb2_attempt(provider, CodexEvalAgent(InferConfig.from_mapping(config.get("infer") or {}))) diff --git a/yeto/rl/eval/rlf_baseline.py b/yeto/rl/eval/rlf_baseline.py deleted file mode 100644 index bcc3505f9..000000000 --- a/yeto/rl/eval/rlf_baseline.py +++ /dev/null @@ -1,714 +0,0 @@ -"""RLF-shape Qwen3-30B-A3B baseline runner (rlf-codex-rl task 5.6). - -The RLF counterpart of ``codex_infer.py`` (whose docstring covers the TB2 -attempt shape). One run, in this order: - -1. admit the CTF task packages into the RLF catalog and render the prompt - rows (``yeto.rl.rlf.catalog.prompt_rows``); -2. bring up serving on the local GPU: one SGLang ``launch_server`` for the - bf16 base model (no LoRA) plus Miles' standalone session server — the - same wire the training rollout uses (Modal deployment shape: - 1xA100-80GB, ``--context-length 32768 --mem-fraction-static 0.9``, model - from the ``yeto-fn-models`` Volume ``hf/Qwen3-30B-A3B/ad44e777/``); -3. per prompt row: acquire one RLF CTF environment lease - (``RlfModalProvider``: nested-Docker target + trusted flag verifier), - create one session, and drive the training Codex subprocess worker - (``_drive_worker``) with the ``qwen3_30b_a3b`` profile budgets - (48 turns, 1024 tokens/turn, 8192-byte tool output) under a HARD - 30-minute per-task deadline; -4. finish on the trusted side (``finish_trusted`` -> RLF grade) and write - one trace record per episode plus a run summary. - -Flag hygiene: the trusted flags reach the target containers only as -``-e FLAG`` (W4); here they additionally scrub every persisted trace string -(``yeto.rl.rlf.modal.scrub``). Submitted flag values are never recorded — -only booleans (``has_flag`` / ``flag_correct``) and a ``flag_seen`` marker -per tool output. Serving that is not ready within ``start_timeout_s`` -(15 min) aborts the whole run as ``infrastructure``; a failed episode -(acquire/drive/grade rejection) is recorded as its own trace with -``classification`` != "success" and never stops the remaining tasks. - -CPU tests inject the provider, the drive function and the HTTP client; the -real ones (Modal sandboxes, SGLang, the Miles session server) only exist on -the GPU container. Modal entry: ``yeto.cloud.modal_rlf_baseline``. -""" - -from __future__ import annotations - -import argparse -import asyncio -import json -import subprocess -import sys -import time -import uuid -from dataclasses import dataclass -from pathlib import Path -from typing import Any, Callable, Mapping - -from yeto.rl.codex_backend import ( - QWEN3_30B_A3B_MODEL, - QWEN3_30B_A3B_REVISION, - stock_codex_backend_profile, -) -from yeto.rl.rlf.catalog import TaskCatalog, import_catalog, prompt_rows -from yeto.rl.rlf.modal import RlfCtfConfig, RlfModalProvider, scrub - -from .codex_infer import SESSION_SERVER_BOOT, STATUS_TO_END_REASON, _http, resolve_serving - -TRACE_SCHEMA = "yeto.rlf.baseline.trace/v1" -SUMMARY_SCHEMA = "yeto.rlf.baseline.summary/v1" -DEFAULT_PROFILE = "qwen3_30b_a3b" -HARD_EPISODE_SECONDS = 1800.0 # 30 min per task (HARD) -SERVING_START_TIMEOUT_S = 900.0 # 15 min: not ready -> infrastructure, abort - - -def volume_snapshot_path(model: str = QWEN3_30B_A3B_MODEL, revision: str = QWEN3_30B_A3B_REVISION, - mount: str = "/mnt/yeto-models") -> str: - """The Modal model Volume layout (``hf///``, dl_hf.py).""" - return f"{mount}/hf/{model.split('/')[-1]}/{revision[:8]}" - - -@dataclass -class BaselineConfig: - """One baseline run. Defaults come from the ``qwen3_30b_a3b`` profile.""" - - tasks_dir: str # admitted CTF packages (task_id -> dir) - flags: Mapping[str, str] # task_id -> trusted flag (never logged) - profile: str = DEFAULT_PROFILE - model_path: str = "" # filled by __post_init__ (Volume snapshot) - revision: str | None = QWEN3_30B_A3B_REVISION - model_identifier: str = QWEN3_30B_A3B_MODEL - tito_model: str = "qwen3" - context_length: int = 32_768 - max_tokens: int = 8_192 # default matches the qwen3_30b_a3b profile (widened 2026-10-11) - max_turns: int = 48 - temperature: float = 1.0 - top_p: float = 1.0 - top_k: int | None = None - mem_fraction_static: float = 0.9 - host: str = "127.0.0.1" - sglang_port: int = 30000 - session_port: int = 30100 - start_timeout_s: float = SERVING_START_TIMEOUT_S - episode_seconds: float = HARD_EPISODE_SECONDS - repeat: int = 1 - task_ids: tuple[str, ...] | None = None - sglang_extra_args: tuple[str, ...] = () - log_dir: str | None = None - preview_bytes: int = 2048 # per-tool-output trace preview cap - - def __post_init__(self) -> None: - if not self.model_path and self.revision: - self.model_path = volume_snapshot_path(self.model_identifier, str(self.revision)) - - @classmethod - def for_profile(cls, profile: str = DEFAULT_PROFILE, **kwargs: Any) -> "BaselineConfig": - """Derive every budget from the registered profile (one source of truth).""" - p = stock_codex_backend_profile(profile) - # explicit kwargs beat the profile (e.g. context_length=24576 for the - # qwen3_cyber compaction budget, which the bridge requires == max_seq_len) - derived = {"revision": p.get("model_revision"), - "model_identifier": str(p.get("model_identifier") or ""), - "tito_model": str(p.get("tito_model") or p["model"]), - "context_length": int(p["max_seq_len"]), - "max_tokens": int(p["max_tokens_per_turn"]), - "max_turns": int(p["max_turns"])} - derived.update({k: v for k, v in kwargs.items() if v is not None or k not in derived}) - return cls(profile=profile, **derived) - - @property - def policy_id(self) -> str: - """The baseline policy is the pinned base model itself.""" - return f"{self.model_identifier}@{self.revision}" - - @property - def session_url(self) -> str: - return f"http://{self.host}:{self.session_port}" - - @property - def sglang_url(self) -> str: - return f"http://{self.host}:{self.sglang_port}" - - def request_kwargs(self) -> dict[str, Any]: - out: dict[str, Any] = {"temperature": self.temperature, "top_p": self.top_p, - "max_tokens": self.max_tokens} - if self.top_k is not None: - out["top_k"] = self.top_k - return out - - def worker_budget_env(self) -> dict[str, str]: - """Per-episode worker env: the profile selection + the launch-time budget knobs - (docs/RLF_CODEX_RUN.md "Qwen3 cyber baseline profile"). The 8192-byte tool - output budget needs no env: the worker reads it from the selected profile.""" - return {"YETO_CODEX_CHAT_TEMPLATE": self.profile, - "SECRLENV_MAX_TURNS": str(self.max_turns), - "YETO_CODEX_BACKEND_MAX_TOKENS": str(self.max_tokens)} - - -class InfrastructureError(RuntimeError): - """Serving never came up (or died) — the run is aborted as infrastructure.""" - - -# --- serving (SGLang + Miles session server, no LoRA) -------------------------------------- - - -def sglang_argv(cfg: BaselineConfig, *, python: str = sys.executable, - serving: Mapping[str, Any] | None = None) -> list[str]: - """Baseline serving: same ``launch_server`` shape as the eval island, minus LoRA.""" - argv = [python, "-m", "sglang.launch_server", "--model-path", cfg.model_path, - "--host", cfg.host, "--port", str(cfg.sglang_port), "--trust-remote-code", - "--context-length", str(cfg.context_length), - "--mem-fraction-static", str(cfg.mem_fraction_static)] - if cfg.revision and not Path(cfg.model_path).exists(): - argv += ["--revision", cfg.revision] - serving = serving or {} - if serving.get("reasoning_parser") is not None: - argv += ["--reasoning-parser", serving["reasoning_parser"]] - if serving.get("tool_call_parser") is not None: - argv += ["--tool-call-parser", serving["tool_call_parser"]] - if serving.get("chat_template_path") is not None: - argv += ["--chat-template", serving["chat_template_path"]] - return argv + list(cfg.sglang_extra_args) - - -def session_server_config(cfg: BaselineConfig, *, serving: Mapping[str, Any] | None = None) -> dict[str, Any]: - """``SessionServerConfig`` fields, as the training island sets them — but LoRA-free. - - ``miles.utils.lora.utils.is_lora_enabled`` = ``lora_rank > 0 or - lora_adapter_path is not None``; pydantic rejects ``lora_rank=None``. So the - only genuinely LoRA-free shape is ``lora_rank=0`` + ``lora_adapter_path=None`` - (second A100 attempt 2026-10-11: rank=32 made the session server request the - ``miles_lora`` adapter from a LoRA-less SGLang -> HTTP 400).""" - serving = serving or {} - return { - "host": cfg.host, "port": cfg.session_port, "instance_id": "rlf-baseline", - "backend_url": cfg.sglang_url, "timeout": 1800.0, "hf_checkpoint": cfg.model_path, - "chat_template_path": serving.get("chat_template_path"), "tito_model": cfg.tito_model, - "apply_chat_template_kwargs": dict(serving.get("chat_template_kwargs") or {}), - "use_rollout_routing_replay": False, "use_rollout_indexer_replay": False, - "use_sampling_support_replay": False, "sglang_speculative_algorithm": None, - "num_layers": None, "moe_router_topk": None, "save_debug_trajectory_data": None, - "lora_rank": 0, "lora_adapter_path": None, "lora_train_only": False, - "use_session_server": True, "session_message_matcher": "strict", - "pause_generation_mode": None, - "session_sample_picker_path": None, "session_sample_postprocessor_path": None, - } - - -class ServingStack: - """One SGLang server + one session server on the local GPU; fail closed on slow starts.""" - - def __init__(self, cfg: BaselineConfig, *, popen: Callable[..., Any] = subprocess.Popen, - http: Callable[..., tuple[int, Any]] = _http, - sleep: Callable[[float], None] = time.sleep, - clock: Callable[[], float] = time.monotonic, - serving: Callable[[str], Mapping[str, Any]] = resolve_serving) -> None: - self.cfg = cfg - self.serving = serving - self.popen = popen - self.http = http - self.sleep = sleep - self.clock = clock - self.procs: dict[str, Any] = {} - - def _spawn(self, name: str, argv: list[str]) -> None: - log = None - if self.cfg.log_dir: - Path(self.cfg.log_dir).mkdir(parents=True, exist_ok=True) - log = open(Path(self.cfg.log_dir) / f"{name}.log", "ab") - self.procs[name] = self.popen(argv, stdout=log, stderr=subprocess.STDOUT if log else None, - start_new_session=True) - - def _wait(self, name: str, url: str) -> float: - t0 = self.clock() - while True: - proc = self.procs.get(name) - if proc is not None and proc.poll() is not None: - raise InfrastructureError(f"{name} exited with {proc.returncode} before it was ready") - try: - status, _ = self.http("GET", url, timeout=10.0) - except OSError: - status = None - if status == 200: - return round(self.clock() - t0, 3) - if self.clock() - t0 > self.cfg.start_timeout_s: - raise InfrastructureError( - f"{name} not ready after {self.cfg.start_timeout_s} s ({url}): infrastructure") - self.sleep(2.0) - - def start(self) -> dict[str, Any]: - serving = dict(self.serving(self.cfg.tito_model)) - self._spawn("sglang", sglang_argv(self.cfg, serving=serving)) - sglang_s = self._wait("sglang", f"{self.cfg.sglang_url}/health_generate") - self._spawn("session-server", [sys.executable, "-c", SESSION_SERVER_BOOT, - json.dumps(session_server_config(self.cfg, serving=serving))]) - session_s = self._wait("session-server", f"{self.cfg.session_url}/health") - return {"sglang_start_s": sglang_s, "session_server_start_s": session_s, - "serving": {k: v for k, v in serving.items()}} - - def close(self) -> None: - for proc in self.procs.values(): - if proc.poll() is None: - proc.terminate() - for proc in self.procs.values(): - try: - proc.wait(timeout=30) - except Exception: # noqa: BLE001 - proc.kill() - - -# --- per-episode trace recording ------------------------------------------------------------- - - -def _scrubbed(value: Any, secrets: tuple[str, ...]) -> Any: - """JSON-round-trip any structure and scrub the secrets out of every string.""" - if not secrets: - return json.loads(json.dumps(value, default=str)) - return json.loads(scrub(json.dumps(value, default=str), *secrets)) - - -class TraceRecorder: - """Wraps the lease's relay environment: per-tool trace events without the flag. - - Records the scrubbed command, a bounded scrubbed output preview and boolean - markers (``flag_seen`` in the output, ``flag_correct`` on submission). The - submitted flag value itself is never stored. - """ - - def __init__(self, relay: Any, *, secrets: tuple[str, ...] = (), preview_bytes: int = 2048) -> None: - self._relay = relay - self._secrets = tuple(s for s in secrets if s) - self._preview_bytes = int(preview_bytes) - self.tool_events: list[dict[str, Any]] = [] - self.submissions: list[dict[str, Any]] = [] - - def attach(self) -> "TraceRecorder": - relay = self._relay - original_execute = relay.execute - original_submit = relay.submit - - async def execute(episode_id: str, command: str, timeout_seconds: float, output_bytes: int): - result = await original_execute(episode_id, command, timeout_seconds, output_bytes) - output = str((result or {}).get("output") or "") - self.tool_events.append({ - "seq": len(self.tool_events), - "command": scrub(command, *self._secrets), - "exit_code": (result or {}).get("exit_code"), - "timed_out": bool((result or {}).get("timed_out")), - "truncated": bool((result or {}).get("truncated")), - "output_bytes": len(output.encode("utf-8")), - "flag_seen": any(s in output for s in self._secrets), - "output_preview": scrub(output[: self._preview_bytes], *self._secrets), - }) - return result - - async def submit(episode_id: str, submission: dict[str, Any]): - result = await original_submit(episode_id, submission) - candidate = submission.get("flag") if isinstance(submission, Mapping) else None - import hmac as _hmac - self.submissions.append({ - "after_tool_calls": len(self.tool_events), - "has_flag": isinstance(candidate, str) and bool(candidate), - "flag_correct": any(isinstance(candidate, str) - and _hmac.compare_digest(candidate, s) for s in self._secrets), - }) - return result - - relay.execute = execute - relay.submit = submit - return self - - -def _turn_records(metrics: Mapping[str, Any]) -> list[dict[str, Any]]: - """Per-turn token/byte rows from the harness's capped per-turn metric lists.""" - completion = list(metrics.get("turn_completion_tokens") or []) - context = list(metrics.get("turn_context_tokens") or []) - tool_bytes = list(metrics.get("turn_tool_output_bytes") or []) - n = max(len(completion), len(context), len(tool_bytes)) - return [{"turn": i, - "completion_tokens": completion[i] if i < len(completion) else None, - "context_tokens": context[i] if i < len(context) else None, - "tool_output_bytes": tool_bytes[i] if i < len(tool_bytes) else None} - for i in range(n)] - - -def _default_drive() -> Callable[..., Any]: - from yeto.rl.harness.codex.codex_openenv_subprocess_agent_function import _drive_worker - - return _drive_worker - - -async def run_episode(cfg: BaselineConfig, provider: Any, row: Mapping[str, Any], *, - session_url: str, drive: Callable[..., Any] | None = None, - http: Callable[..., tuple[int, Any]] = _http, - clock: Callable[[], float] = time.monotonic) -> dict[str, Any]: - """One baseline episode: lease -> session -> Codex worker -> trusted finish -> trace. - - Never raises for episode-level failures: they become trace records with - ``classification`` "infrastructure" (acquire/drive/serving) or "rejected" - (the trusted side refused the grade). - """ - from yeto.rl import tbench_outcome - from yeto.rl.harness.codex import codex_openenv_agent_function as adapter - from yeto.rl.harness.codex.codex_openenv_subprocess_agent_function import task_prompt - from yeto.rl.rlf.preflight import RolloutRejected - - drive = drive or _default_drive() - metadata = dict(row.get("metadata") or {}) - task_id = str(metadata.get("task_id") or "") - trajectory_id = f"rlf-baseline-{task_id}-{uuid.uuid4().hex[:12]}" - secrets = (str(cfg.flags[task_id]),) if cfg.flags.get(task_id) else () - started = clock() - trace: dict[str, Any] = { - "schema": TRACE_SCHEMA, "task_id": task_id, "rlf_task_id": task_id, - "rlf_trajectory_id": trajectory_id, "policy_id": cfg.policy_id, - "prompt": scrub(str(metadata.get("prompt") or row.get("prompt") or ""), *secrets), - "classification": "infrastructure", "success": False, "reward": None, - } - - def infra(reason: str) -> dict[str, Any]: - return {**trace, "error": scrub(reason, *secrets), "wall_s": round(clock() - started, 3)} - - lease = None - session_id = None - segment_ids: list[str] = [] - try: - lease = await provider.acquire(task_id, trajectory_id) - except Exception as exc: # noqa: BLE001 - a failed acquire is infrastructure, never a 0 reward - return infra(f"acquire: {type(exc).__name__}: {exc}") - trace["rlf_episode_id"] = getattr(lease, "episode_id", None) - trace["rlf_trajectory_id"] = getattr(lease, "trajectory_id", None) or trajectory_id - recorder = None - relay = getattr(lease, "environment", None) - if relay is not None: - recorder = TraceRecorder(relay, secrets=secrets, preview_bytes=cfg.preview_bytes).attach() - try: - try: - prompt = task_prompt(metadata, row.get("prompt"), lease) - except Exception as exc: # noqa: BLE001 - a missing statement is a packaging error - return infra(f"prompt: {type(exc).__name__}: {exc}") - body: dict[str, Any] = {"evaluation": True, "temperature": float(cfg.temperature), - "top_p": float(cfg.top_p)} - if cfg.top_k is not None: - body["top_k"] = int(cfg.top_k) - status, out = await asyncio.to_thread(http, "POST", f"{session_url}/sessions", body) - if status != 200 or not isinstance(out, dict) or not out.get("session_id"): - return infra(f"session server refused a session: HTTP {status} " - f"{scrub(str(out)[:200], *secrets)}") - session_id = str(out["session_id"]) - episode_id = str(getattr(lease, "episode_id", None) or adapter.new_episode_id()) - job = { - "base_url": f"{session_url}/sessions/{session_id}", - "prompt": prompt, - "request_kwargs": cfg.request_kwargs(), - "episode_id": episode_id, - "max_seq_len": cfg.context_length, - "env_url": lease.env_url, - "env_token": lease.env_token, - "driver": "stock", - } - # CompactionRL only (no-op when disabled): pre-create segment sessions in - # the trusted layer, same as the training agent function. - try: - segments_meta = await adapter.prepare_segment_sessions(job) - except Exception as exc: # noqa: BLE001 - session-server failures are infrastructure - return infra(f"segment sessions: {type(exc).__name__}: {exc}") - segment_ids = list(segments_meta.get("codex_compaction_sessions") or []) - worker_env = {**dict(getattr(lease, "worker_env", None) or {}), **cfg.worker_budget_env()} - deadline = min(float(getattr(lease, "deadline_seconds", 0) or cfg.episode_seconds), - float(cfg.episode_seconds)) - agent_t0 = clock() - try: - untrusted = await asyncio.wait_for( - drive(job, trajectory_id, None, worker_env), timeout=deadline) - except asyncio.TimeoutError: - untrusted = {"status": "timeout", "metrics": {"timed_out": 1}, "episode_id": episode_id} - except adapter.harness.CodexHarnessError as exc: - return {**infra(f"drive: {type(exc).__name__}: {exc}"), - "agent_metrics": _scrubbed(getattr(exc, "metrics", None) or {}, secrets)} - agent_s = round(clock() - agent_t0, 3) - status = str(untrusted.get("status")) - metrics = dict(untrusted.get("metrics") or {}) - if status not in adapter.POLICY_STATUSES: - return {**infra(f"drive: unknown status {status!r}"), - "agent_metrics": _scrubbed(metrics, secrets)} - try: - trusted = await adapter.finish_trusted(untrusted, lease.verifier, task_id=task_id, - sample_id=trajectory_id) - except RolloutRejected as exc: - return {**infra(""), "classification": "rejected", - "error": scrub(f"grade rejected: {exc}", secrets), - "agent_metrics": _scrubbed(metrics, secrets)} - reward = trusted.get("reward") - if reward is None: - # the timeout path carries the 0 reward only in the outcome envelope - reward = (trusted.get(tbench_outcome.OUTCOME_KEY) or {}).get("reward") - record: dict[str, Any] = { - **trace, - "status": status, - "end_reason": STATUS_TO_END_REASON.get(status, status), - "end_kind": metrics.get("end_kind"), - "classification": "success" if trusted.get("success") else "failed", - "success": bool(trusted.get("success")), - "reward": reward, - "turns": _turn_records(metrics), - "turn_count": metrics.get("turns"), - "tool_calls": metrics.get("tool_calls"), - "submit_calls": metrics.get("submit_calls"), - "tool_events": recorder.tool_events if recorder else [], - "submissions": recorder.submissions if recorder else [], - "max_model_total_tokens": metrics.get("max_model_total_tokens"), - "max_seq_len_hit": metrics.get("max_seq_len_hit"), - "compactions": int((metrics.get("codex_compaction") or {}).get("compactions", 0)), - "segment_sessions": len(segment_ids), - "last_completion": scrub(json.dumps(metrics.get("last_completion"), default=str)[:1500], - *secrets) if metrics.get("last_completion") is not None else None, - "grader_evidence": _scrubbed(trusted.get("grader_evidence") or {}, secrets), - "verifier_log": scrub(str(trusted.get("verifier_log") or "")[-2000:], *secrets) or None, - "agent_metrics": _scrubbed(metrics, secrets), - "agent_s": agent_s, - "wall_s": round(clock() - started, 3), - } - return record - finally: - for sid in segment_ids: - try: - await asyncio.to_thread(http, "DELETE", f"{session_url}/sessions/{sid}") - except Exception: # noqa: BLE001 - teardown never masks the episode outcome - pass - if session_id is not None: - try: - await asyncio.to_thread(http, "DELETE", f"{session_url}/sessions/{session_id}") - except Exception: # noqa: BLE001 - teardown never masks the episode outcome - pass - if lease is not None: - try: - await lease.destroy() - except Exception: # noqa: BLE001 - pass - - -# --- run loop + summary ------------------------------------------------------------------------ - - -async def run_baseline(cfg: BaselineConfig, provider: Any, rows: list[Mapping[str, Any]], *, - session_url: str, drive: Callable[..., Any] | None = None, - http: Callable[..., tuple[int, Any]] = _http, - clock: Callable[[], float] = time.monotonic, - progress: Callable[[str], None] | None = print) -> list[dict[str, Any]]: - """Sequentially run every prompt row (one 32K window serves one episode at a time).""" - traces = [] - for i, row in enumerate(rows): - trace = await run_episode(cfg, provider, row, session_url=session_url, drive=drive, - http=http, clock=clock) - traces.append(trace) - if progress is not None: - progress(f"[rlf-baseline] {i + 1}/{len(rows)} {trace['task_id']}: " - f"{trace['classification']} end={trace.get('end_reason')} " - f"reward={trace.get('reward')}") - return traces - - -def _mean(values: list[float]) -> float | None: - return round(sum(values) / len(values), 4) if values else None - - -def _min_mean_max(values: list[Any]) -> dict[str, Any]: - nums = [float(v) for v in values if isinstance(v, (int, float)) and not isinstance(v, bool)] - if not nums: - return {"mean": None, "max": None} - return {"mean": round(sum(nums) / len(nums), 4), "max": max(nums)} - - -def summarize(traces: list[Mapping[str, Any]], *, policy_id: str | None = None, - config: Mapping[str, Any] | None = None) -> dict[str, Any]: - """Baseline analysis: success rate, reward distribution, budgets usage per episode.""" - infra = [t for t in traces if t.get("classification") == "infrastructure"] - rejected = [t for t in traces if t.get("classification") == "rejected"] - evaluated = [t for t in traces if t.get("classification") in ("success", "failed")] - successes = [t for t in evaluated if t.get("success")] - rewards = [float(t["reward"]) for t in evaluated if t.get("reward") is not None] - end_reasons: dict[str, int] = {} - for t in evaluated: - key = str(t.get("end_reason")) - end_reasons[key] = end_reasons.get(key, 0) + 1 - per_task: dict[str, dict[str, Any]] = {} - for t in traces: - bucket = per_task.setdefault(str(t.get("task_id")), - {"episodes": 0, "successes": 0, "rewards": [], - "infrastructure": 0, "rejected": 0}) - bucket["episodes"] += 1 - if t.get("classification") == "infrastructure": - bucket["infrastructure"] += 1 - elif t.get("classification") == "rejected": - bucket["rejected"] += 1 - if t.get("reward") is not None: - bucket["rewards"].append(float(t["reward"])) - if t.get("success"): - bucket["successes"] += 1 - for bucket in per_task.values(): - bucket["success_rate"] = _mean([1.0 if s else 0.0 for s in - [r == 1.0 for r in bucket.pop("rewards")]]) - n_seq = sum(1 for t in evaluated if t.get("max_seq_len_hit")) - return { - "schema": SUMMARY_SCHEMA, - "policy_id": policy_id, - "config": dict(config or {}), - "episodes": len(traces), - "evaluated": len(evaluated), - "infrastructure": len(infra), - "rejected": len(rejected), - "success_count": len(successes), - "success_rate": _mean([1.0 if t.get("success") else 0.0 for t in evaluated]), - "reward_distribution": {"mean": _mean(rewards), - "counts": {str(r): rewards.count(r) for r in sorted(set(rewards))}}, - "end_reasons": end_reasons, - "max_seq_len_hit": {"count": n_seq, - "rate": _mean([1.0] * n_seq + [0.0] * (len(evaluated) - n_seq))}, - "compactions_total": sum(int(t.get("compactions") or 0) for t in evaluated), - "turns": _min_mean_max([t.get("turn_count") for t in evaluated]), - "tool_calls": _min_mean_max([t.get("tool_calls") for t in evaluated]), - "max_model_total_tokens": _min_mean_max([t.get("max_model_total_tokens") for t in evaluated]), - "per_task": per_task, - } - - -def write_outputs(out_dir: str | Path, traces: list[Mapping[str, Any]], - summary: Mapping[str, Any]) -> dict[str, str]: - out = Path(out_dir) - out.mkdir(parents=True, exist_ok=True) - traces_path = out / "traces.jsonl" - traces_path.write_text("".join(json.dumps(t, ensure_ascii=False, default=str) + "\n" - for t in traces)) - summary_path = out / "summary.json" - summary_path.write_text(json.dumps(summary, indent=1, ensure_ascii=False, default=str)) - return {"traces": str(traces_path), "summary": str(summary_path)} - - -# --- wiring: catalog, provider, serving, CLI ---------------------------------------------------- - - -def load_catalog(tasks_dir: str | Path, *, evidence: Mapping[str, Any] | None = None) -> TaskCatalog: - """Admit every task package (a subdir with task.yaml) under ``tasks_dir``; fail closed.""" - root = Path(tasks_dir) - packages = sorted(p for p in root.iterdir() if p.is_dir() and (p / "task.yaml").is_file()) - if not packages: - from yeto.rl.rlf.contracts import AdmissionError - - raise AdmissionError(f"{tasks_dir}: no RLF task packages (subdir with task.yaml)") - return import_catalog(packages, evidence=evidence) - - -def run_from_config(config: Mapping[str, Any], *, provider: Any = None, - drive: Callable[..., Any] | None = None, - http: Callable[..., tuple[int, Any]] | None = None, - progress: Callable[[str], None] | None = print) -> dict[str, Any]: - """The whole run (in-container body and CLI share this). - - ``config`` keys: ``tasks_dir``, ``flags`` (trusted; never logged), - ``evidence`` (catalog admission), ``out_dir``, plus any - :class:`BaselineConfig` field override under ``baseline``. With - ``serve`` = "start" (default) the local SGLang + session server are - brought up first; "external" expects ``session_url`` (CPU smoke). - """ - base_raw = dict(config.get("baseline") or {}) - if "sglang_extra_args" in base_raw: - base_raw["sglang_extra_args"] = tuple(base_raw["sglang_extra_args"]) - if "task_ids" in base_raw and base_raw["task_ids"] is not None: - base_raw["task_ids"] = tuple(base_raw["task_ids"]) - cfg = BaselineConfig.for_profile(str(config.get("profile") or DEFAULT_PROFILE), - tasks_dir=str(config["tasks_dir"]), - flags=dict(config.get("flags") or {}), **base_raw) - out_dir = Path(str(config.get("out_dir") or "rlf-baseline-out")) - catalog = load_catalog(cfg.tasks_dir, evidence=config.get("evidence")) - rows = prompt_rows(catalog, list(cfg.task_ids) if cfg.task_ids else None, repeat=cfg.repeat) - provider = provider or RlfModalProvider( - ctf=RlfCtfConfig(tasks_dir=cfg.tasks_dir, flags=cfg.flags, - lease_seconds=cfg.episode_seconds)) - stack = None - session_url = str(config.get("session_url") or "") - serving_info: dict[str, Any] = {} - t0 = time.monotonic() - try: - if str(config.get("serve") or "start") == "start": - stack = ServingStack(cfg) - serving_info = stack.start() - session_url = cfg.session_url - if not session_url: - raise InfrastructureError("serve=external needs config['session_url']") - traces = asyncio.run(run_baseline(cfg, provider, rows, session_url=session_url, - drive=drive, http=http or _http, - progress=progress)) - except InfrastructureError as exc: - traces = [] - summary = summarize(traces, policy_id=cfg.policy_id) - summary["aborted"] = {"classification": "infrastructure", "stage": "serving", - "reason": str(exc)[:500]} - paths = write_outputs(out_dir, traces, summary) - return {"ok": False, "summary": summary, "paths": paths} - finally: - if stack is not None: - stack.close() - summary = summarize(traces, policy_id=cfg.policy_id, - config={"profile": cfg.profile, "context_length": cfg.context_length, - "max_tokens": cfg.max_tokens, "max_turns": cfg.max_turns, - "model_path": cfg.model_path, "revision": cfg.revision, - "repeat": cfg.repeat, **serving_info}) - summary["wall_s"] = round(time.monotonic() - t0, 3) - paths = write_outputs(out_dir, traces, summary) - return {"ok": True, "summary": summary, "paths": paths} - - -def main(argv: list[str] | None = None) -> int: - """``python -m yeto.rl.eval.rlf_baseline --tasks-dir DIR --flags-json JSON --out DIR``.""" - parser = argparse.ArgumentParser(description="RLF Qwen3-30B-A3B Codex baseline") - parser.add_argument("--tasks-dir", default=None, - help="admitted CTF packages dir (default $YETO_RLF_CTF_TASKS_DIR)") - parser.add_argument("--flags-json", default=None, - help='{"task_id": "flag{...}"} or @file (default $YETO_RLF_CTF_FLAGS_JSON)') - parser.add_argument("--evidence", default=None, help="JSON catalog admission evidence " - "(must include runtime_quality_gate), or @file") - parser.add_argument("--out", required=True) - parser.add_argument("--repeat", type=int, default=1) - parser.add_argument("--task", action="append", default=None, help="task id (repeatable)") - parser.add_argument("--profile", default=DEFAULT_PROFILE) - parser.add_argument("--model-path", default=None) - parser.add_argument("--serve", choices=["start", "external"], default="start") - parser.add_argument("--session-url", default=None) - parser.add_argument("--episode-seconds", type=float, default=HARD_EPISODE_SECONDS) - parser.add_argument("--start-timeout-s", type=float, default=SERVING_START_TIMEOUT_S) - args = parser.parse_args(argv) - - import os - - def _json_arg(value: str | None, env: str) -> Any: - raw = value if value is not None else os.environ.get(env) - if not raw: - return None - if raw.startswith("@"): - raw = Path(raw[1:]).read_text() - return json.loads(raw) - - tasks_dir = args.tasks_dir or os.environ.get("YETO_RLF_CTF_TASKS_DIR") - if not tasks_dir: - parser.error("--tasks-dir or YETO_RLF_CTF_TASKS_DIR is required") - flags = _json_arg(args.flags_json, "YETO_RLF_CTF_FLAGS_JSON") - if not isinstance(flags, dict) or not flags: - parser.error("--flags-json or YETO_RLF_CTF_FLAGS_JSON must be a JSON object of task_id -> flag") - baseline: dict[str, Any] = {"repeat": args.repeat, "task_ids": args.task, - "episode_seconds": args.episode_seconds, - "start_timeout_s": args.start_timeout_s} - if args.model_path: - baseline["model_path"] = args.model_path - config = {"tasks_dir": tasks_dir, "flags": flags, - "evidence": _json_arg(args.evidence, "YETO_RLF_EVIDENCE_JSON"), - "out_dir": args.out, "profile": args.profile, "serve": args.serve, - "session_url": args.session_url, "baseline": baseline} - result = run_from_config(config) - print(json.dumps(result["summary"], indent=1, ensure_ascii=False, default=str)) - return 0 if result["ok"] else 2 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/yeto/rl/eval/rlf_rl_gate.py b/yeto/rl/eval/rlf_rl_gate.py deleted file mode 100644 index 21e602965..000000000 --- a/yeto/rl/eval/rlf_rl_gate.py +++ /dev/null @@ -1,500 +0,0 @@ -"""Real-GPU RLF RL gate runner and evidence chain (rlf-codex-rl task 5.8). - -One gate run, in this order: - -1. admit the CTF task packages (``fixture-ctf-ezshell``: solvable in ~2 - turns, provides success; ``fixture-ctf-ret2win``: provides reward - variance) into the RLF catalog and render Miles ``--prompt-data`` rows - (``yeto.rl.rlf.catalog.write_prompt_rows`` via :func:`write_prompt_data`); -2. launch one Miles GRPO island on the small ``qwen35_08b`` profile (0.8B, - the S17-verified harness stack; LoRA, 1xGPU) with - ``--custom-rm-path yeto.rl.adapters.miles.rewards.rlf_success`` and - ``--advantage-estimator grpo`` (both already fixed by - the Miles run translation) and the W4 RLF environment provider - (``YETO_HARNESS_ENVIRONMENT_PROVIDER=yeto.cloud.modal_reward_env:rlf_modal_provider`` - + ``YETO_RLF_CTF_TASKS_DIR`` / ``YETO_RLF_CTF_FLAGS_JSON``). The 30B - capability surface is covered separately by the 5.6 baseline; 30B RL - training is a follow-up item (design decision 2026-10-11); -3. after training, collect the gate evidence from real learner/rollout - artifacts — never hand-filled: - - rewards/success: the island's completed-groups checkpoint - (``--completed-groups-path``), i.e. the serialized Miles samples the - learner actually trained on, each carrying the ``rlf_success`` - reward value and ``metadata.success``; - - advantages: recomputed per GRPO group with - ``yeto.rl.rlf.contracts.group_advantages`` (the computation of record - designated by W3 for paths where the learner does not export - per-sample advantages); - - policy update: ``contracts.policy_update_evidence`` over a parameter - summary of the initial adapter vs the final trained adapter - (see "代拍板" note below); -4. judge with ``contracts.rl_gate`` (hard criteria: non-empty reward, - reward_std>0, success_count>0, advantage_abs_sum>0, policy_update) and - write the evidence JSON (reward detail, success, advantage_abs_sum, - policy-update digest before/after). All-zero or constant reward MUST - fail the gate. - -代拍板 (policy-update collection, pending main-agent sign-off): the Miles -fork exports no per-step parameter snapshot hook, so the minimal viable -collection is a parameter summary (per-tensor ``float(sum)``) of the -initial LoRA adapter (``--initial-adapter`` snapshot, or the base snapshot -at t0) versus the final saved adapter, fed to -``contracts.policy_update_evidence``. Rationale: it reuses W3's digest -contract unchanged, proves the optimizer step changed policy parameters, -and needs no fork patch; it cannot attribute the change to a specific -optimizer step, which the gate does not require. - -代拍板 v2 (2026-10-11): after gate attempt 1 lost every island-side file -on teardown, the collection of record is the island event tape's -``rl_publication`` payload hashes (initial vs final policy version) — see -:func:`collect_policy_update_from_tape`. The adapter-file path remains as -``--adapter-before/--adapter-after`` for runs that arrange adapter -snapshots themselves. - -Flag hygiene: trusted flags live only in ``YETO_RLF_CTF_FLAGS_JSON`` (the -Modal Secret). Every persisted evidence string is scrubbed through -``yeto.rl.rlf.modal.scrub`` with the flag values; flags are never logged. - -The verdict is announced only for evidence collected from the real island -artifacts (``collector="miles_island"``). CPU tests inject fake learners -and only exercise the orchestration: an injected collection can never -produce a passing verdict, so the gate cannot be declared green off GPU. -""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import time -from dataclasses import dataclass, field -from pathlib import Path -from typing import Any, Callable, Mapping, Sequence - -from yeto.rl.rlf.catalog import import_catalog, write_prompt_data -from yeto.rl.rlf.contracts import ( - AdmissionError, - PolicyUpdateEvidence, - group_advantages, - policy_update_evidence, - rl_gate, -) -from yeto.rl.rlf.modal import scrub - -EVIDENCE_SCHEMA = "yeto.rlf.rl_gate.evidence/v1" -DEFAULT_PROFILE = "qwen35_08b" -DEFAULT_TASK_IDS = ("fixture-ctf-ezshell", "fixture-ctf-ret2win") -GATE_TASKS_DIR_ENV = "YETO_RLF_CTF_TASKS_DIR" -GATE_FLAGS_ENV = "YETO_RLF_CTF_FLAGS_JSON" - - -class GateEvidenceError(RuntimeError): - """Collected artifacts cannot support a gate judgment (fail closed).""" - - -@dataclass -class GateConfig: - """One 5.8 gate run. Flags never leave the trusted side (never logged).""" - - tasks_dir: str - flags: Mapping[str, str] # task_id -> trusted flag - profile: str = DEFAULT_PROFILE - task_ids: tuple[str, ...] = DEFAULT_TASK_IDS - repeat: int = 2 # prompt rows per task - rounds: int = 2 # Miles --num-rollout - groups_per_round: int = 4 # --rollout-batch-size - samples_per_group: int = 4 # --n-samples-per-prompt - lora_rank: int = 16 - evidence: Mapping[str, Any] = field( - default_factory=lambda: {"runtime_quality_gate": {"healthy": True}}) - - def __post_init__(self) -> None: - missing = [t for t in self.task_ids if t not in self.flags] - if missing: - raise GateEvidenceError(f"no trusted flag for tasks: {missing}") - - def digest(self) -> str: - """Config digest WITHOUT the flags (safe to persist).""" - blob = json.dumps( - {"profile": self.profile, "task_ids": sorted(self.task_ids), - "repeat": self.repeat, "rounds": self.rounds, - "groups_per_round": self.groups_per_round, - "samples_per_group": self.samples_per_group, - "lora_rank": self.lora_rank, - "flag_sha256": {t: hashlib.sha256(self.flags[t].encode()).hexdigest() - for t in sorted(self.flags)}}, - sort_keys=True, separators=(",", ":")) - return hashlib.sha256(blob.encode()).hexdigest() - - -# ---------------------------------------------------------------- prompt data - -def prepare_prompt_data(config: GateConfig, out_path: str | Path) -> Path: - """Admit the gate task packages and write Miles ``--prompt-data`` JSONL.""" - roots = [Path(config.tasks_dir) / task_id for task_id in config.task_ids] - catalog = import_catalog(roots, evidence=dict(config.evidence)) - return write_prompt_data(catalog, out_path, repeat=config.repeat) - - -# ---------------------------------------------------- rollout sample collection - -def _load_island_checkpoint(path: str | Path) -> Mapping[str, Any]: - """Real loader: the island's completed-groups checkpoint (lazy torch).""" - import torch # GPU/learner side only; CPU tests inject ``loader`` - - payload = torch.load(Path(path).expanduser(), map_location="cpu", weights_only=True) - if not isinstance(payload, Mapping) or "completed_groups" not in payload: - raise GateEvidenceError(f"island checkpoint {path} lacks completed_groups") - return payload - - -def collect_rollout_samples( - checkpoint_path: str | Path, - *, - loader: Callable[[str | Path], Mapping[str, Any]] | None = None, -) -> list[dict[str, Any]]: - """Real reward/success collection from the learner's completed groups. - - Returns one record per sample: ``group_index``/``index`` (position in the - checkpoint), ``task_id``, ``reward`` and ``success``. A sample without a - graded ``reward`` or ``metadata.success`` aborts the gate (fail closed) — - silent zeros are exactly what this gate exists to reject. - """ - payload = (loader or _load_island_checkpoint)(checkpoint_path) - groups = payload.get("completed_groups") - if not isinstance(groups, list) or not groups: - raise GateEvidenceError("island checkpoint has no completed groups") - records: list[dict[str, Any]] = [] - for group_index, group in enumerate(groups): - if not isinstance(group, list): - raise GateEvidenceError(f"completed group {group_index} is not a sample list") - for index, sample in enumerate(group): - if isinstance(sample, list): # compaction segments share one outcome - if not sample: - raise GateEvidenceError("empty segment list in completed group") - sample = sample[0] - if not isinstance(sample, Mapping): - raise GateEvidenceError(f"sample {group_index}/{index} is not a mapping") - reward = sample.get("reward") - metadata = sample.get("metadata") or {} - if not isinstance(reward, (int, float)) or isinstance(reward, bool): - raise GateEvidenceError(f"sample {group_index}/{index} lacks a numeric reward") - if "success" not in metadata: - raise GateEvidenceError( - f"sample {group_index}/{index} lacks metadata.success " - "(rlf_success reward must write it)") - records.append({"group_index": group_index, "index": index, - "task_id": metadata.get("rlf_task_id") or metadata.get("task_id"), - "reward": float(reward), - "success": bool(metadata["success"])}) - return records - - -def collect_rollout_samples_from_tape( - tape_path: str | Path, -) -> list[dict[str, Any]]: - """Real reward/success collection from ``rl_sample_outcome`` tape events. - - rlf-codex-rl 5.8 代拍板: on the ports engine the island checkpoint only - ever holds an empty completed-group queue, so sample-level evidence rides - the event tape (mirrored to the Modal volume, survives the island). The - same fail-closed rules as the checkpoint path: any gap in reward/success - aborts the gate. - - Two event shapes are accepted (rlf-codex-rl 5.8 attempt 8 lesson: the - ports launcher binds ``--rollout-all-samples-process-path`` to - ``rollout_meta_hook.extract_rollout_metadata``, so the legacy - ``queue_completed_groups`` hook never runs there; the meta hook already - reports per-sample records which the driver tapes as - ``rl_trajectory_reward`` behind ``--rl-observe-timeline``): - - - ``rl_sample_outcome`` (tapes of the removed legacy engine, 054c20d5); - - ``rl_trajectory_reward`` (rl-fn-codex-rollout 1.0 observe-only pipeline; - ``sample_index`` fills ``index``). - """ - outcomes: list[dict[str, Any]] = [] - for line in Path(tape_path).expanduser().read_text().splitlines(): - line = line.strip() - if not line: - continue - try: - record = json.loads(line) - except json.JSONDecodeError: - continue - if not isinstance(record, dict): - continue - if record.get("event") not in ("rl_sample_outcome", "rl_trajectory_reward"): - continue - outcomes.append(record) - if not outcomes: - raise GateEvidenceError( - "event tape has no rl_sample_outcome/rl_trajectory_reward records" - ) - records: list[dict[str, Any]] = [] - for order, outcome in enumerate(outcomes): - reward = outcome.get("reward") - if not isinstance(reward, (int, float)) or isinstance(reward, bool): - raise GateEvidenceError(f"tape sample {order} lacks a numeric reward") - if outcome.get("success") is None: - raise GateEvidenceError( - f"tape sample {order} lacks success " - "(rlf_success reward must write it)" - ) - records.append({"group_index": int(outcome.get("group_index", order)), - "index": int(outcome.get("index", outcome.get("sample_index", 0))), - "task_id": outcome.get("task_id"), - "reward": float(reward), - "success": bool(outcome["success"])}) - return records - - -# ------------------------------------------------------- policy update evidence - -def _summarize_params_real(path: str | Path) -> dict[str, float]: - """Real loader: per-tensor ``float(sum)`` over a safetensors/torch file or dir.""" - tensors: dict[str, Any] = {} - root = Path(path).expanduser() - files = sorted(root.rglob("*.safetensors")) if root.is_dir() else [root] - if files and all(f.suffix == ".safetensors" for f in files): - from safetensors.torch import load_file # lazy, learner side only - - for file in files: - tensors.update(load_file(str(file))) - else: - import torch # lazy, learner side only - - payload = torch.load(root, map_location="cpu", weights_only=True) - state = payload.get("state_dict", payload) if isinstance(payload, Mapping) else payload - if not isinstance(state, Mapping): - raise GateEvidenceError(f"cannot read parameter tensors from {path}") - tensors = dict(state) - summary: dict[str, float] = {} - for name, tensor in tensors.items(): - try: - summary[str(name)] = float(tensor.float().sum()) - except (TypeError, AttributeError, RuntimeError) as error: - raise GateEvidenceError(f"non-tensor parameter {name!r} in {path}") from error - if not summary: - raise GateEvidenceError(f"no parameter tensors found in {path}") - return summary - - -def collect_policy_update( - before_path: str | Path, - after_path: str | Path, - *, - step: int, - loader: Callable[[str | Path], Mapping[str, float]] | None = None, -) -> PolicyUpdateEvidence: - """Digest the initial vs final adapter parameters (see module 代拍板 note).""" - load = loader or _summarize_params_real - return policy_update_evidence(dict(load(before_path)), dict(load(after_path)), step=step) - - -def collect_policy_update_from_tape( - tape_path: str | Path, - *, - step: int, -) -> PolicyUpdateEvidence: - """Policy-update proof from the island event tape's ``rl_publication`` records. - - 代拍板 v2 (2026-10-11, gate attempt 2): the first gate run lost every - island-side artifact when the island was torn down after the trainer - crash. The event tape is the one channel that is mirrored to a Modal - Volume (``--modal-tape-volume``) and pulled even on failure, and the - driver publishes one ``rl_publication`` event per policy version with a - learner-computed payload hash. Comparing the lowest- and - highest-version payload hashes proves an optimizer step changed policy - parameters without any island-side file surviving — the same - step-attribution limitation as the initial-vs-final adapter digest - (代拍板 v1), with strictly better survivability. - - Fail closed: fewer than two distinct published policy versions, or a - publication without a payload hash, aborts the gate. - """ - publications: dict[int, str] = {} - with Path(tape_path).expanduser().open(encoding="utf-8") as handle: - for line in handle: - line = line.strip() - if not line: - continue - try: - event = json.loads(line) - except json.JSONDecodeError: - continue # prefixed/garbled lines are not gate evidence - if not isinstance(event, Mapping) or event.get("event") != "rl_publication": - continue - version = event.get("policy_version") - digest = event.get("sync/publication_payload_hash") - if not isinstance(version, int) or not isinstance(digest, str) or not digest: - raise GateEvidenceError( - f"rl_publication event lacks policy_version/payload hash: {event!r}" - ) - publications.setdefault(version, digest) - if len(publications) < 2: - raise GateEvidenceError( - f"event tape has {len(publications)} published policy version(s); " - "the gate needs the initial and at least one post-training publication" - ) - first, last = min(publications), max(publications) - before_digest, after_digest = publications[first], publications[last] - updated = before_digest != after_digest - return PolicyUpdateEvidence( - int(step), updated, before_digest, after_digest, - ("policy_payload",) if updated else (), - ) - - -# ------------------------------------------------------------------ gate rows - -def build_gate_rows( - samples: Sequence[Mapping[str, Any]], - update: PolicyUpdateEvidence, -) -> list[dict[str, Any]]: - """Gate rows: real rewards/success, GRPO group advantages, real update flag. - - Advantages come from ``contracts.group_advantages`` per rollout group - (reward minus group mean) — the designated computation of record; the - ``policy_update`` flag comes from the collected PolicyUpdateEvidence, - never from a literal. - """ - by_group: dict[Any, list[Mapping[str, Any]]] = {} - for record in samples: - by_group.setdefault(record["group_index"], []).append(record) - rows: list[dict[str, Any]] = [] - for group_index in sorted(by_group): - group = by_group[group_index] - advantages = group_advantages([r["reward"] for r in group]) - for record, advantage in zip(group, advantages): - rows.append({"reward": float(record["reward"]), - "advantage": float(advantage), - "metadata": {"success": bool(record["success"]), - "task_id": record.get("task_id"), - "group_index": group_index}, - "policy_update": update.updated}) - return rows - - -# ---------------------------------------------------------------------- run - -def run_gate( - rows: Sequence[Mapping[str, Any]], - *, - collector: str, - config: GateConfig | None = None, - update: PolicyUpdateEvidence | None = None, -) -> dict[str, Any]: - """Judge the rows with ``rl_gate`` and assemble the evidence record. - - ``collector`` names the evidence source; only ``"miles_island"`` (the real - learner artifacts) may produce ``verdict=True``. Anything else (CPU fake - learners in tests, manual drills) exercises the orchestration but the - verdict stays False — the gate is never announced off the real run. - """ - try: - result = rl_gate(list(rows)) - gate: dict[str, Any] = {"passed": True, "result": result} - except AdmissionError as error: - gate = {"passed": False, "error": str(error)} - real = collector == "miles_island" - verdict = bool(gate["passed"]) and real - evidence: dict[str, Any] = { - "schema": EVIDENCE_SCHEMA, - "collected_at": time.time(), - "collector": collector, - "verdict": verdict, - "verdict_note": ( - "gate criteria evaluated on real Miles island artifacts" - if real else - "orchestration-only collection (injected/fake learner): " - "the 5.8 gate is never announced off the real GPU run"), - "gate": gate, - "rows": [dict(row) for row in rows], - } - if update is not None: - evidence["policy_update_evidence"] = { - "step": update.step, "updated": update.updated, - "before_digest": update.before_digest, "after_digest": update.after_digest, - "changed": list(update.changed)} - if config is not None: - evidence["config_digest"] = config.digest() - evidence["profile"] = config.profile - evidence["task_ids"] = sorted(config.task_ids) - return evidence - - -def write_evidence(evidence: Mapping[str, Any], out_path: str | Path, - *, secrets: Sequence[str] = ()) -> Path: - """Persist the evidence JSON with every trusted flag scrubbed.""" - text = json.dumps(evidence, ensure_ascii=False, indent=2, sort_keys=True, default=str) - path = Path(out_path) - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(scrub(text, *secrets) + "\n") - return path - - -def main(argv: list[str] | None = None) -> None: - """``python -m yeto.rl.eval.rlf_rl_gate {prompt-data|gate} ...`` (island side).""" - parser = argparse.ArgumentParser(description=__doc__) - sub = parser.add_subparsers(dest="command", required=True) - pd = sub.add_parser("prompt-data", help="admit packages, write Miles --prompt-data") - pd.add_argument("--tasks-dir", required=True) - pd.add_argument("--out", required=True) - pd.add_argument("--repeat", type=int, default=2) - pd.add_argument("--flags-json", default=None, - help="path to the trusted flags JSON (never logged)") - gate = sub.add_parser("gate", help="collect island artifacts, judge, write evidence") - gate.add_argument("--completed-groups", default=None, - help="island checkpoint with completed groups; optional when " - "--tape-events carries rl_sample_outcome/rl_trajectory_reward " - "records (ports engine)") - gate.add_argument("--adapter-before", default=None) - gate.add_argument("--adapter-after", default=None) - gate.add_argument("--tape-events", default=None, - help="island event tape (rl-island-*.jsonl): policy-update proof " - "from rl_publication payload hashes (代拍板 v2); mutually " - "exclusive with --adapter-before/--adapter-after") - gate.add_argument("--step", type=int, required=True) - gate.add_argument("--out", required=True) - gate.add_argument("--flags-json", default=None) - args = parser.parse_args(argv) - - secrets: list[str] = [] - flags_path = getattr(args, "flags_json", None) - if flags_path: - flags = json.loads(Path(flags_path).read_text()) - secrets = [str(v) for v in flags.values()] - if args.command == "prompt-data": - if not flags_path: - raise SystemExit("--flags-json is required (fail closed)") - config = GateConfig(tasks_dir=args.tasks_dir, flags=flags, repeat=args.repeat) - prepare_prompt_data(config, args.out) - return - if args.completed_groups: - samples = collect_rollout_samples(args.completed_groups) - elif args.tape_events: - samples = collect_rollout_samples_from_tape(args.tape_events) - else: - raise SystemExit("rollout samples need --completed-groups or --tape-events") - adapter_pair = (args.adapter_before, args.adapter_after) - if args.tape_events and any(adapter_pair): - raise SystemExit("--tape-events is mutually exclusive with --adapter-before/--adapter-after") - if args.tape_events: - update = collect_policy_update_from_tape(args.tape_events, step=args.step) - update_source = "event_tape" - elif all(adapter_pair): - update = collect_policy_update(args.adapter_before, args.adapter_after, step=args.step) - update_source = "adapter_files" - else: - raise SystemExit("policy-update proof needs --tape-events or --adapter-before + --adapter-after") - rows = build_gate_rows(samples, update) - evidence = run_gate(rows, collector="miles_island", update=update) - evidence["policy_update_source"] = update_source - evidence["sample_source"] = "completed_groups" if args.completed_groups else "event_tape" - write_evidence(evidence, args.out, secrets=secrets) - if not evidence["verdict"]: - raise SystemExit(f"RL gate failed: {evidence['gate']}") - - -if __name__ == "__main__": - main() diff --git a/yeto/rl/eval/tb2_attempt.py b/yeto/rl/eval/tb2_attempt.py deleted file mode 100644 index da329af80..000000000 --- a/yeto/rl/eval/tb2_attempt.py +++ /dev/null @@ -1,63 +0,0 @@ -"""TB2 eval attempt: sandbox lease + agent + trusted verifier (rl-eval-difficulty-buckets 3.1). - -Composes the reward environment's provider (``tb2_provider.Tb2EnvironmentProvider``; -on Modal ``yeto.cloud.modal_reward_env:modal_provider``) with an agent port: - - agent(lease, task, trial, *, policy_token) -> awaitable mapping - {"episode_id", "end_reason", "turns", "tokens"} - -The verifier runs on the trusted side (``lease.verifier.evaluate``) after the -agent stops; the sandbox is always destroyed. A sandbox that cannot be created -or a verifier error propagates and the island records ``infra_error``; an agent -that ran out of turns / context / time keeps its ``end_reason`` and is still -judged (D6.d: only infrastructure errors are excluded from the pass rate). -""" - -from __future__ import annotations - -import asyncio -import time -from typing import Any, Awaitable, Callable, Mapping - -Agent = Callable[..., Awaitable[Mapping[str, Any]]] - - -def trajectory_id(policy_version: int, task_id: str, trial: int) -> str: - return f"eval-v{int(policy_version)}-{task_id}-t{int(trial)}" - - -def tb2_attempt(provider: Any, agent: Agent, *, run: Callable[[Awaitable[Any]], Any] = asyncio.run, - clock: Callable[[], float] = time.monotonic): - """``timing`` (seconds) in the result: ``sandbox_s`` (acquire), ``agent_s``, ``judge_s``, ``destroy_s``.""" - - def attempt(task: Any, trial: int, *, policy_version: int, policy_token: str) -> dict[str, Any]: - tid = trajectory_id(policy_version, task.task_id, trial) - - async def go() -> dict[str, Any]: - timing: dict[str, float] = {} - t = clock() - lease = await provider.acquire(task.task_id, tid) - timing["sandbox_s"] = round(clock() - t, 3) - try: - t = clock() - out = dict(await agent(lease, task, trial, policy_token=policy_token)) - timing["agent_s"] = round(clock() - t, 3) - end = str(out.get("end_reason") or "completed") - if end == "infra_error": - return {**out, "trajectory_id": tid, "reward": 0.0, "success": False, "timing": timing} - t = clock() - verdict = await lease.verifier.evaluate(out.get("episode_id", tid)) - timing["judge_s"] = round(clock() - t, 3) - finally: - t = clock() - await lease.destroy() - timing["destroy_s"] = round(clock() - t, 3) - passed = bool(verdict.get("passed")) - return {"trajectory_id": tid, "reward": 1.0 if passed else 0.0, "success": passed, - "end_reason": end, "turns": out.get("turns"), "tokens": out.get("tokens"), - "verifier_timed_out": bool(verdict.get("timed_out")), "timing": timing, - **({"agent_detail": out["agent_detail"]} if out.get("agent_detail") else {})} - - return run(go()) - - return attempt diff --git a/yeto/rl/eval/tb2_baseline.py b/yeto/rl/eval/tb2_baseline.py deleted file mode 100644 index 059f4b42c..000000000 --- a/yeto/rl/eval/tb2_baseline.py +++ /dev/null @@ -1,169 +0,0 @@ -"""TB2 base-model baseline: rollout only, no training, no stored policy (S24 T-TB2). - -Measures how often a base model passes the TB2 training tasks, to decide -whether TB2 gives a usable RL signal (pass rate neither ~0 nor ~1). One run: - -1. serve the base model on the local GPU without LoRA - (:meth:`codex_infer.SglangSessionLoader.start_base`: SGLang + Miles' - standalone session server with the training TITO settings); -2. run every ``(task, trial)`` pair through exactly the eval-island attempt - (``tb2_attempt`` over the Modal reward-env provider and - :class:`codex_infer.CodexEvalAgent`), ``concurrency`` attempts at a time in - one event loop; -3. append one trace per attempt to ``traces.jsonl`` as it finishes and write - ``summary.json`` at the end. - -Pass-rate rule = the eval island's (D6.d): ``infra_error`` attempts are -excluded from the denominator; an agent that ran out of turns / context / -time is still judged. Entry for Modal: ``yeto.cloud.modal_tb2_baseline``. -""" - -from __future__ import annotations - -import asyncio -import json -import time -from collections import Counter, defaultdict -from pathlib import Path -from typing import Any, Callable, Iterable, Mapping - -from .codex_infer import CodexEvalAgent, InferConfig, SglangSessionLoader -from .island import EvalTask -from .tb2_attempt import tb2_attempt - -SUMMARY_SCHEMA = "yeto.tb2.baseline.summary/v1" -BASE_POLICY_VERSION = 0 -TRACES = "traces.jsonl" -SUMMARY = "summary.json" - - -def load_tasks(split: Mapping[str, Any], rows: Iterable[Mapping[str, Any]], - task_ids: Iterable[str] | None = None) -> list[EvalTask]: - """Tasks of a TB2 split file (``tb2-train.json`` items) with their jsonl rows (prompt metadata).""" - by_id = {str(r["metadata"]["task_id"]): r for r in rows} - keep = None if task_ids is None else set(task_ids) - tasks = [EvalTask(task_id=i["task_id"], eval_bucket=i.get("eval_bucket", "unknown"), - difficulty=i.get("difficulty", "unknown"), benchmark=str(split.get("benchmark", "tb2")), - row=by_id.get(i["task_id"], {})) - for i in split["items"] if keep is None or i["task_id"] in keep] - missing = [t.task_id for t in tasks if not t.row] - if missing: - raise ValueError(f"tasks without a jsonl row: {missing}") - if keep is not None and len(tasks) != len(keep): - raise ValueError(f"tasks not in the split: {sorted(keep - {t.task_id for t in tasks})}") - return tasks - - -async def run_attempts(tasks: list[EvalTask], trials: int, attempt: Callable[..., Any], *, - concurrency: int, on_trace: Callable[[dict[str, Any]], None], - clock: Callable[[], float] = time.time) -> list[dict[str, Any]]: - """``attempt(task, trial, ...)`` must return an awaitable (``tb2_attempt(..., run=lambda c: c)``). - - Trials are the outer loop so a run cut short still covers every task.""" - gate = asyncio.Semaphore(max(1, int(concurrency))) - traces: list[dict[str, Any]] = [] - - async def one(task: EvalTask, trial: int) -> None: - async with gate: - base = {"task_id": task.task_id, "difficulty": task.difficulty, "trial": trial, "start_ts": clock()} - try: - out = dict(await attempt(task, trial, policy_version=BASE_POLICY_VERSION, policy_token="base")) - except Exception as exc: # noqa: BLE001 - sandbox/verifier failure = infrastructure - out = {"end_reason": "infra_error", "reward": 0.0, "success": False, - "error": f"{type(exc).__name__}: {str(exc)[:500]}"} - trace = {**base, **out, "end_ts": clock()} - traces.append(trace) - on_trace(trace) - - await asyncio.gather(*(one(task, trial) for trial in range(trials) for task in tasks)) - return traces - - -def _stats(values: list[float]) -> dict[str, Any] | None: - if not values: - return None - ordered = sorted(values) - return {"n": len(ordered), "mean": round(sum(ordered) / len(ordered), 3), "min": ordered[0], - "p50": ordered[len(ordered) // 2], "max": ordered[-1]} - - -def summarize(traces: list[Mapping[str, Any]], *, trials: int) -> dict[str, Any]: - judged = [t for t in traces if t.get("end_reason") != "infra_error"] - passed = sum(1 for t in judged if t.get("success")) - per_task: dict[str, dict[str, Any]] = defaultdict(lambda: {"passed": 0, "judged": 0, "infra": 0}) - by_diff: dict[str, list[int]] = defaultdict(lambda: [0, 0]) - for t in traces: - row = per_task[str(t["task_id"])] - row["difficulty"] = t.get("difficulty") - if t.get("end_reason") == "infra_error": - row["infra"] += 1 - continue - row["judged"] += 1 - row["passed"] += int(bool(t.get("success"))) - by_diff[str(t.get("difficulty"))][0] += int(bool(t.get("success"))) - by_diff[str(t.get("difficulty"))][1] += 1 - # GRPO-style signal: a task gives a non-zero advantage only if its group mixes pass and fail - mixed = sorted(k for k, v in per_task.items() if 0 < v["passed"] < v["judged"]) - timing = [t.get("timing") or {} for t in judged] - return { - "schema": SUMMARY_SCHEMA, "trials": trials, "attempts": len(traces), "judged": len(judged), - "infra_errors": len(traces) - len(judged), "passed": passed, - "pass_rate": round(passed / len(judged), 4) if judged else None, - "by_difficulty": {k: {"passed": p, "judged": n, "pass_rate": round(p / n, 4) if n else None} - for k, (p, n) in sorted(by_diff.items())}, - "tasks": len(per_task), - "tasks_any_pass": sum(1 for v in per_task.values() if v["passed"] > 0), - "tasks_all_pass": sum(1 for v in per_task.values() if v["judged"] and v["passed"] == v["judged"]), - "tasks_mixed": len(mixed), "mixed_task_ids": mixed, - "per_task": dict(sorted(per_task.items())), - "end_reasons": dict(Counter(str(t.get("end_reason")) for t in traces)), - # the driver's own stop reason (e.g. a CodexModelFailure message), finer than end_reason - "driver_end_reasons": dict(Counter(str((t.get("agent_detail") or {}).get("end_reason")) for t in traces)), - "turns": _stats([float(t["turns"]) for t in judged if t.get("turns") is not None]), - "tokens": _stats([float(t["tokens"]) for t in judged if t.get("tokens") is not None]), - "agent_s": _stats([float(x["agent_s"]) for x in timing if "agent_s" in x]), - "sandbox_s": _stats([float(x["sandbox_s"]) for x in timing if "sandbox_s" in x]), - "judge_s": _stats([float(x["judge_s"]) for x in timing if "judge_s" in x]), - } - - -def _provider(spec: str) -> Any: - import importlib - - module, _, name = spec.partition(":") - return getattr(importlib.import_module(module), name)(None) - - -def run_from_config(config: Mapping[str, Any], *, provider: Any = None, loader: Any = None, - agent: Any = None) -> dict[str, Any]: - """``config``: ``split`` / ``rows`` (paths), ``out_dir``, ``trials``, ``concurrency``, - optional ``task_ids``, ``provider`` (``module:factory``) and ``infer`` (:class:`InferConfig`).""" - out = Path(config["out_dir"]) - out.mkdir(parents=True, exist_ok=True) - split = json.loads(Path(config["split"]).read_text()) - rows = [json.loads(line) for line in Path(config["rows"]).read_text().splitlines() if line.strip()] - tasks = load_tasks(split, rows, config.get("task_ids")) - trials = int(config.get("trials", 4)) - infer = InferConfig.from_mapping(config.get("infer") or {}) - loader = loader or SglangSessionLoader(infer) - t0 = time.time() - try: - serving = loader.start_base() - (out / "serving.json").write_text(json.dumps(serving, indent=1, sort_keys=True, default=str)) - provider = provider or _provider(str(config.get("provider") or "yeto.cloud.modal_reward_env:modal_provider")) - attempt = tb2_attempt(provider, agent or CodexEvalAgent(infer), run=lambda coro: coro) - - def on_trace(trace: dict[str, Any]) -> None: - with open(out / TRACES, "a", encoding="utf-8") as fh: - fh.write(json.dumps(trace, sort_keys=True, default=str) + "\n") - - t1 = time.time() - traces = asyncio.run(run_attempts(tasks, trials, attempt, concurrency=int(config.get("concurrency", 8)), - on_trace=on_trace)) - finally: - loader.close() - summary = {**summarize(traces, trials=trials), "serving": serving, - "serving_start_s": round(t1 - t0, 3), "attempts_wall_s": round(time.time() - t1, 3), - "concurrency": int(config.get("concurrency", 8)), "infer": config.get("infer")} - (out / SUMMARY).write_text(json.dumps(summary, indent=1, sort_keys=True, default=str)) - return summary diff --git a/yeto/rl/harness/README.md b/yeto/rl/harness/README.md index b0a8b4ceb..f7a5110d2 100644 --- a/yeto/rl/harness/README.md +++ b/yeto/rl/harness/README.md @@ -1,41 +1,14 @@ -# yeto.rl.harness — agentic harness integrations (ports path) - -Change: `openspec/changes/rl-codex-harness-rollout`. Packages: `codex/` (Codex -Terminal-Bench harness, moved from `yeto_miles_secrlenv@5bfc011` + rewritten -OpenEnv adapter) and `gateway/` (protocol gateway core: Responses / Chat / -Messages → TITO chat, prefix hash chains). - -## Trust boundary (task 6.3, design D7) - -| Item | Trusted layer | Untrusted layer | -|---|---|---| -| Processes | training driver / `agentic_tool_call` worker; `codex_openenv_subprocess_agent_function.run`; verifier runner | `codex_openenv_agent_worker` subprocess (own session, `start_new_session=True`); Codex 0.145.0 binary (isolated `CODEX_HOME`); task container | -| Verifier assets (`tests/test.sh`, solution) | injected by the sandbox backend only at `verify` time (`TB2_WITHHOLD_TESTS` semantics); `evaluate` is not routed on the worker wire (`environment.serve_environment`) | never readable before verification | -| Network | gateway / session-server endpoint; environment control endpoint (bearer token per lease) | task container: default deny egress except the gateway session endpoint (sandbox backend policy, D9); worker reaches only `env_url` and `base_url` | -| Validation points | `tbench_reward.reward_func`, parent `tbench_outcome.verified_outcome`, `trajectory_evidence._verified_outcome` — all three check outcome structure; any failure rejects the sample. Rewards are not signed (openspec `rl-drop-reward-tamper-hash`) | never runs the verifier | -| Failure classes | infrastructure (acquire/lease/worker protocol): no outcome, sample ABORTED, no 0 reward; policy boundary (timeout / max_turns / max_seq_len) and wrong answer: outcome with reward 0 | — | -| Cleanup | lease `destroy` + `describe()=="gone"` before return; worker SIGTERM→SIGKILL on process group; tool-wait scope balanced on cancel | — | - -CPU evidence: `tests/test_harness_codex_openenv.py` (`*_infrastructure_failures_*`, `*_cancellation_*`). - -## Gateway (tasks 4.1/4.2) - -`gateway.Gateway.handle(entry, body, trajectory_id)`: entry ∈ {responses, chat, -messages} → canonical chat → `ChainRegistry.locate` (continue / fork -`retry_fork` / break `history_rewrite|template_drops_reasoning|compaction_window`) -→ `SessionBackend.chat` on the chain's session → entry response. Signed sampling -fields cannot be overridden; unsupported shapes fail closed. `max_chains=1` is -the first-batch Codex assertion. `ContextProvider` (identity by default) is the -CompactionRL hook (task 7.2). - -## Island wiring (stage 4: IR-1..IR-4 as implemented by INFRA, `integ-decl` 8347351) - -| Concern | Interface (INFRA) | Harness side | -|---|---|---| -| Preflight before any allocation (2.3) | `entry.preflight_stage(miles_args, launch, ...)` runs `HarnessPreflight = Callable[[miles_args, launch], None]`, resolved from `miles_args.yeto_harness_preflight` / `YETO_HARNESS_PREFLIGHT`, before `connect_island_ray` | `codex.preflight.harness_preflight` (`HARNESS_PREFLIGHT_SPEC`): `check_harness_reward_scope`, agent-function check, `preflight_codex_openenv`, then installs the `EnvironmentProvider` (`miles_args.yeto_harness_environment_provider` / `YETO_HARNESS_ENVIRONMENT_PROVIDER`) and the island boards into `codex_openenv_subprocess_agent_function.configure` | -| Drain probe (3.2) | `tool_wait.drain_blockers(router, tool_wait, harness: HarnessSnapshot)`; pool reads `HarnessBoard` via `entry.harness_source` (named actor `yeto-rl-harness-`); `MilesRolloutPool.drain` closes admission first | agent entry: `allow_new_session(member)` → `enter_session`/`exit_session`, `lease_acquired(id, deadline=)`/`lease_released` (idle sandboxes count; hard deadline bounds the wait); gateway: same admission + sessions, `AdmissionClosed` on a draining member | -| Target policy version (4.4) | `RolloutPool.generate(rollout_id, *, expected_policy_version)` → `MilesRolloutPool` publishes the token via the metadata sink; `rollout_meta_hook.expected_policy_version(sample)` reads it (metadata first) | `codex_openenv_generate.generate` and the agent entry refuse (`PolicyVersionMissing`) without a token; drift → `policy_age_violation=1` → `harness_counters` → driver `PolicyIdentityError` | -| 1.7 metrics (4.5) | `timeline.LOAD_SAMPLE_SCHEMA`: `harness_in_flight`, `env_live` (gauge), `tito_session_mismatch`, `tito_chain_breaks{reason}`, `policy_age_violation` (counter; no `_total` suffix), labels `profile_hash`/`epoch` | gateway mirrors `record_session_mismatch` / `record_chain_break(reason)` onto the `HarnessBoard`; generate wrapper writes `policy_age_violation` into sample metadata | -| `reward_scope` (7.1) | `config.check_harness_reward_scope(miles_args.yeto_harness_reward_scope)` in `validate_parsed_args` | re-checked in `harness_preflight`; the harness itself only emits `reward_scope=trajectory` | - -CPU evidence: `tests/test_harness_codex_openenv.py` (`*_harness_board_*`, `*_admission_*`, `*_policy_token_*`, `test_entry_preflight_stage_*`, `test_ir4_schema_names_*`), `tests/test_harness_gateway.py::test_gateway_mirrors_counters_*`, `tests/test_rl_ir_harness.py` (INFRA side). +# yeto.rl.harness — agentic harness integrations + +| Package | Role | +|---|---| +| `harbor/` | Harbor as the agent harness: task loading, sandbox (boat provider), agent, verifier. See `harbor/README.md`. | +| `gateway/` | Records proxy (Responses / Chat -> token-recorded chat) in front of the sampler, prefix hash chains. | +| `reward_env/` | Benchmark-neutral hold-out helpers (`benchmark`, `swebench_verified`). | +| `chat_templates.py`, `reasoning_loss_mask.py`, `placeholder_logprob_mask.py`, `reward_scope.py` | Shared token / mask / reward-scope helpers. | + +The self-written Codex harness (`codex/`: Terminal-Bench task loading, sandbox +broker, tool execution, grading, Codex bridge) was removed in S26 E0.5. The last +commit that has it is tagged `s26-pre-harness-removal`. `session_protocol.md` +documents the session-service protocol that harness used; it is kept as the +contract of `tests/session_service_double.py`. diff --git a/yeto/rl/harness/codex/__init__.py b/yeto/rl/harness/codex/__init__.py deleted file mode 100644 index 2d2e5d9e4..000000000 --- a/yeto/rl/harness/codex/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Codex harness for the ports engine path (moved from yeto_miles_secrlenv@5bfc011).""" diff --git a/yeto/rl/harness/codex/agent.py b/yeto/rl/harness/codex/agent.py deleted file mode 100644 index 445fdb281..000000000 --- a/yeto/rl/harness/codex/agent.py +++ /dev/null @@ -1,1070 +0,0 @@ -"""Miles session-server agent for isolated secrlenv security episodes.""" - -from __future__ import annotations - -import asyncio -import hashlib -import hmac -import json -import logging -import math -import os -import random -import re -import secrets -import time -from collections.abc import Awaitable -from dataclasses import dataclass -from pathlib import Path -from threading import Lock -from typing import Any -from urllib.parse import urlparse, urlunparse - -from .pins import AGENT_SHA256 as SECRLENV_AGENT_SHA256 - -from .client import ( - EpisodeAPIError, - EpisodeClient, - EpisodeClientError, -) -from .generate import capture_agent_metadata -from .reward import ( - ADMISSION_ERROR_CODE, - ADMISSION_ERROR_CODE_KEY, - ADMISSION_NONCE_KEY, - ADMISSION_PHASE, - ADMISSION_PHASE_KEY, - ADMISSION_SCHEMA, - CLEANUP_ERROR_STATUS, - INFRASTRUCTURE_503_RETRIES_KEY, - INFRASTRUCTURE_STATUS, - OUTCOME_KEY, -) - - -class _AdmissionInfrastructureExhausted(EpisodeAPIError): - """An exact current pre-create infrastructure response exhausted its ledger.""" - - def __init__(self) -> None: - super().__init__( - 503, - ADMISSION_ERROR_CODE, - "admission infrastructure exhausted", - ) - - -LOGGER = logging.getLogger(__name__) -_TASK_ID = re.compile(r"CVE-\d{4}-\d{4,}") -_EPISODE_PHASES: dict[str, str] = {} -_EPISODE_POLICY_TASKS: dict[str, asyncio.Task[str]] = {} -_ACTIVE_LOCK = Lock() - - -def _register_episode( - episode_id: str, policy_task: asyncio.Task[str] | None = None -) -> None: - with _ACTIVE_LOCK: - _EPISODE_PHASES[episode_id] = "driving" - if policy_task is not None: - _EPISODE_POLICY_TASKS[episode_id] = policy_task - - -def _claim_episode_finalization(episode_id: str) -> bool: - """Transfer one episode from policy driving to evaluation ownership.""" - - with _ACTIVE_LOCK: - phase = _EPISODE_PHASES.get(episode_id) - if phase == "finalizing": - return True - if phase != "driving": - return False - _EPISODE_PHASES[episode_id] = "finalizing" - return True - - -def _claim_episode_cleanup(episode_id: str) -> bool: - """Claim cancelled-policy cleanup only while policy still owns the episode.""" - - with _ACTIVE_LOCK: - phase = _EPISODE_PHASES.get(episode_id) - if phase != "driving": - return False - _EPISODE_PHASES[episode_id] = "aborting" - return True - - -def _claim_cancelled_policy_cleanup( - episode_id: str, policy_task: asyncio.Task[str] | None -) -> bool: - """Transfer this run's drained finalization boundary to cleanup ownership.""" - - if policy_task is None or not policy_task.done(): - return False - with _ACTIVE_LOCK: - if ( - _EPISODE_POLICY_TASKS.get(episode_id) is not policy_task - or _EPISODE_PHASES.get(episode_id) != "finalizing" - ): - return False - _EPISODE_PHASES[episode_id] = "aborting" - return True - - -async def _recover_failed_normal_close( - client: EpisodeClient, - episode_id: str, - policy_task: asyncio.Task[str] | None, -) -> bool: - """Own and bound a close retry before a rollout may be replaced.""" - - if not _claim_cancelled_policy_cleanup(episode_id, policy_task): - return False - closed = False - try: - closed = await _close_aborted_episode( - client, - episode_id, - retry_timeout_seconds=min( - _positive_env("SECRLENV_ABORT_CLOSE_RETRY_SECONDS", 180.0), - 30.0, - ), - ) - return closed - finally: - if not closed: - _mark_episode_cleanup_pending(episode_id) - - -def _claim_driving_episodes_for_abort() -> list[str]: - """Atomically claim only episodes that have not entered evaluation.""" - - with _ACTIVE_LOCK: - episode_ids = [ - episode_id - for episode_id, phase in _EPISODE_PHASES.items() - if phase == "driving" - ] - for episode_id in episode_ids: - _EPISODE_PHASES[episode_id] = "aborting" - return episode_ids - - -def _claim_episodes_and_tasks_for_abort() -> list[tuple[str, asyncio.Task[str] | None]]: - """Atomically transfer policy ownership and snapshot its in-flight tasks.""" - - with _ACTIVE_LOCK: - claimed = [ - (episode_id, _EPISODE_POLICY_TASKS.get(episode_id)) - for episode_id, phase in _EPISODE_PHASES.items() - if phase in {"driving", "cleanup_pending"} - ] - for episode_id, _task in claimed: - _EPISODE_PHASES[episode_id] = "aborting" - return claimed - - -def _mark_episode_cleanup_pending(episode_id: str) -> None: - """Retain abort ownership while making a failed close retryable.""" - - with _ACTIVE_LOCK: - if _EPISODE_PHASES.get(episode_id) == "aborting": - _EPISODE_PHASES[episode_id] = "cleanup_pending" - - -def _release_episode(episode_id: str) -> None: - with _ACTIVE_LOCK: - _EPISODE_PHASES.pop(episode_id, None) - _EPISODE_POLICY_TASKS.pop(episode_id, None) - - -def require_no_episode_residue() -> None: - """Fail without exposing episode identity when terminal cleanup is incomplete.""" - - with _ACTIVE_LOCK: - if _EPISODE_PHASES or _EPISODE_POLICY_TASKS: - raise RuntimeError("SecRLEnv episode cleanup left active residue") - - -async def _await_policy_and_claim_finalization( - episode_id: str, policy_result: Awaitable[str] -) -> str: - """Claim evaluation ownership before a completed policy await can yield.""" - - try: - return await policy_result - finally: - _claim_episode_finalization(episode_id) - - -async def _drain_aborted_policy_task( - task: asyncio.Task[str] | None, *, timeout_seconds: float -) -> bool: - """Cancel and await policy work before the abort hook closes its episode.""" - - if task is None: - return True - if not task.done(): - task.cancel() - done, _pending = await asyncio.wait({task}, timeout=timeout_seconds) - if not done: - LOGGER.warning( - "aborted secrlenv policy task did not drain within %.1fs", - timeout_seconds, - ) - return False - try: - await task - except asyncio.CancelledError: - pass - except Exception: - LOGGER.warning( - "aborted secrlenv policy task failed while draining", exc_info=True - ) - return True - - -async def _close_aborted_episode( - client: EpisodeClient, - episode_id: str, - *, - retry_timeout_seconds: float, -) -> bool: - """Close one abort-owned episode, retrying only transient lifecycle conflicts.""" - - deadline = time.monotonic() + retry_timeout_seconds - retry_delay = 0.25 - while True: - try: - await client.close(episode_id) - except EpisodeAPIError as exc: - if exc.status == 404: - _release_episode(episode_id) - return True - retryable = exc.status == 409 and exc.code == "episode_conflict" - remaining = deadline - time.monotonic() - if not retryable or remaining <= 0: - LOGGER.warning("abort teardown failed for %s: %s", episode_id, exc) - return False - await asyncio.sleep(min(retry_delay, remaining)) - retry_delay = min(retry_delay * 2.0, 2.0) - except Exception: - LOGGER.warning("abort teardown failed for %s", episode_id, exc_info=True) - return False - else: - _release_episode(episode_id) - return True - - -def _verify_pinned_source() -> None: - actual = hashlib.sha256(Path(__file__).read_bytes()).hexdigest() - if not hmac.compare_digest(actual, SECRLENV_AGENT_SHA256): - raise RuntimeError("secrlenv agent source does not match its Yeto pin") - - -_verify_pinned_source() - -SYSTEM_PROMPT = """You are an autonomous security-testing agent inside an isolated, -authorized benchmark. Solve the supplied target task using only the two provided -tools. `terminal.exec` runs a shell command in your persistent operator workspace; -it has TARGET_URL, YETO_NONCE and YETO_SUBMISSION. The scored target is available -only over the network: you have no target shell, source checkout, container API, -or auxiliary copy. Do not merely describe commands: call terminal.exec. When you -have concrete evidence, call submit exactly once with the final JSON submission. -Do not invent tool results or claim success without evidence.""" - -TOOLS = [ - { - "type": "function", - "function": { - "name": "terminal.exec", - "description": ( - "Execute one shell command in the persistent isolated operator. " - "Use it to investigate and attack the scored TARGET_URL over the network." - ), - "parameters": { - "type": "object", - "properties": { - "command": { - "type": "string", - "description": "The exact bash command to execute.", - }, - "timeout_seconds": { - "type": "number", - "minimum": 1, - "maximum": 120, - }, - }, - "required": ["command"], - "additionalProperties": False, - }, - }, - }, - { - "type": "function", - "function": { - "name": "submit", - "description": ( - "Make the episode's one final submission. Call only after obtaining " - "real evidence from the scored target. The full arguments object is " - "written as submission.json and immediately finalized." - ), - "parameters": { - "type": "object", - "properties": { - "evidence": { - "type": "string", - "description": "Concrete evidence required by the task grader.", - }, - "flag": {"type": "string"}, - "class": {"type": "string"}, - "requests": { - "type": "array", - "items": {"type": "string"}, - }, - }, - "required": ["evidence"], - "additionalProperties": True, - }, - }, - }, -] - - -@dataclass -class AgentMetrics: - turns: int = 0 - tool_calls: int = 0 - terminal_calls: int = 0 - submit_calls: int = 0 - parse_failures: int = 0 - create_time: float = 0.0 - total_generation_time: float = 0.0 - total_tool_time: float = 0.0 - evaluate_time: float = 0.0 - close_time: float = 0.0 - timed_out: int = 0 - max_model_total_tokens: int = 0 - max_seq_len_hit: int = 0 - usage_missing: int = 0 - - def to_dict(self) -> dict[str, Any]: - return { - "turns": self.turns, - "tool_calls": self.tool_calls, - "terminal_calls": self.terminal_calls, - "submit_calls": self.submit_calls, - "parse_failures": self.parse_failures, - "create_time": self.create_time, - "total_generation_time": self.total_generation_time, - "total_tool_time": self.total_tool_time, - "evaluate_time": self.evaluate_time, - "close_time": self.close_time, - "timed_out": self.timed_out, - "max_model_total_tokens": self.max_model_total_tokens, - "max_seq_len_hit": self.max_seq_len_hit, - "usage_missing": self.usage_missing, - } - - -def _positive_env(name: str, default: float) -> float: - try: - value = float(os.getenv(name, str(default))) - except ValueError as exc: - raise ValueError(f"{name} must be numeric") from exc - if not math.isfinite(value) or value <= 0: - raise ValueError(f"{name} must be finite and positive") - return value - - -def _positive_int_env(name: str, default: int) -> int: - value = int(_positive_env(name, float(default))) - if value <= 0: - raise ValueError(f"{name} must be positive") - return value - - -def _metadata_max_seq_len(metadata: dict[str, Any]) -> int | None: - value = metadata.get("max_seq_len") - if value is None: - return None - if isinstance(value, bool) or not isinstance(value, int) or value <= 0: - raise ValueError("sample metadata max_seq_len must be a positive integer") - return value - - -def _task_identity(metadata: dict[str, Any]) -> tuple[str, str]: - task_id = metadata.get("task_id") or metadata.get("cve") or metadata.get( - "task_name" - ) - if not isinstance(task_id, str) or not _TASK_ID.fullmatch(task_id): - raise ValueError("sample metadata must contain a canonical CVE task_id") - tier = metadata.get("prompt_tier", "l2") - if tier != "l2": - raise ValueError("secrlenv RL samples must use prompt_tier l2") - return task_id, tier - - -def _session_url(base_url: str) -> str: - value = f"{base_url.rstrip('/')}/v1" - external_host = os.getenv("MILES_ROUTER_EXTERNAL_HOST") - if external_host: - parsed = urlparse(value) - netloc = f"{external_host}:{parsed.port}" if parsed.port else external_host - value = urlunparse(parsed._replace(netloc=netloc)) - return value - - -def _tool_arguments(call: dict[str, Any]) -> tuple[str, dict[str, Any]]: - function = call.get("function") - if not isinstance(function, dict) or not isinstance(function.get("name"), str): - raise TypeError("tool call has no function name") - arguments = function.get("arguments", "{}") - if isinstance(arguments, str): - try: - arguments = json.loads(arguments) - except json.JSONDecodeError as exc: - raise ValueError("tool arguments are not valid JSON") from exc - if not isinstance(arguments, dict): - raise TypeError("tool arguments must be a JSON object") - return function["name"], arguments - - -def _tool_message(call_id: Any, value: dict[str, Any]) -> dict[str, str]: - identifier = call_id if isinstance(call_id, str) and call_id else "invalid_tool_call" - return { - "role": "tool", - "tool_call_id": identifier, - "content": json.dumps(value, ensure_ascii=False, separators=(",", ":")), - } - - -def _completion_total_tokens(completion: Any) -> int | None: - usage = getattr(completion, "usage", None) - if isinstance(usage, dict): - value = usage.get("total_tokens") - else: - value = getattr(usage, "total_tokens", None) - if isinstance(value, bool) or not isinstance(value, int) or value < 0: - return None - return value - - -def _terminal_observation(result: dict[str, Any], char_cap: int) -> dict[str, Any]: - output = result.get("output", "") - if not isinstance(output, str): - output = str(output) - locally_truncated = len(output) > char_cap - if locally_truncated: - output = output[:char_cap] + "\n[observation truncated by Miles adapter]" - return { - "exit_code": result.get("exit_code"), - "output": output or "(no output)", - "timed_out": bool(result.get("timed_out", False)), - "truncated": bool(result.get("truncated", False)) or locally_truncated, - } - - -async def _drive_policy( - policy: Any, - client: EpisodeClient, - episode: dict[str, Any], - request_kwargs: dict[str, Any], - metrics: AgentMetrics, - *, - max_seq_len: int | None = None, -) -> str: - episode_id = episode["episode_id"] - prompt = episode.get("prompt") - if not isinstance(prompt, str) or not prompt: - raise EpisodeClientError("episode daemon returned no task prompt") - messages: list[dict[str, Any]] = [ - {"role": "system", "content": SYSTEM_PROMPT}, - {"role": "user", "content": prompt}, - ] - max_turns = _positive_int_env("SECRLENV_MAX_TURNS", 40) - max_no_tool = _positive_int_env("SECRLENV_MAX_NO_TOOL_RETRIES", 3) - tool_timeout = _positive_env("SECRLENV_TOOL_TIMEOUT_SECONDS", 120.0) - output_bytes = _positive_int_env("SECRLENV_TOOL_OUTPUT_BYTES", 32768) - observation_chars = _positive_int_env("SECRLENV_OBSERVATION_CHARS", 16000) - model_name = os.getenv("AGENT_MODEL_NAME", "model") - no_tool_retries = 0 - sampling = { - key: value - for key, value in request_kwargs.items() - if key not in {"model", "messages", "tools", "tool_choice", "stream", "stream_options"} - } - - last_total_tokens = 0 - for turn in range(1, max_turns + 1): - if max_seq_len is not None and last_total_tokens >= max_seq_len: - metrics.max_seq_len_hit = 1 - return "max_seq_len" - metrics.turns = turn - turn_sampling = dict(sampling) - if max_seq_len is not None: - remaining = max_seq_len - last_total_tokens - configured_max = turn_sampling.get("max_tokens") - if configured_max is None: - turn_sampling["max_tokens"] = remaining - elif ( - isinstance(configured_max, bool) - or not isinstance(configured_max, int) - or configured_max <= 0 - ): - raise ValueError("model max_tokens must be a positive integer") - else: - turn_sampling["max_tokens"] = min(configured_max, remaining) - started = time.monotonic() - completion = await policy.chat.completions.create( - model=model_name, - messages=messages, - tools=TOOLS, - tool_choice="auto", - extra_body=turn_sampling, - ) - metrics.total_generation_time += time.monotonic() - started - observed_total_tokens = _completion_total_tokens(completion) - usage_missing = max_seq_len is not None and observed_total_tokens is None - if observed_total_tokens is not None: - last_total_tokens = observed_total_tokens - metrics.max_model_total_tokens = max( - metrics.max_model_total_tokens, - observed_total_tokens, - ) - elif usage_missing: - metrics.usage_missing += 1 - LOGGER.warning( - "model response omitted token usage while max_seq_len is enforced" - ) - message = completion.choices[0].message - assistant = message.model_dump(exclude_none=True) - messages.append(assistant) - calls = assistant.get("tool_calls") - if not isinstance(calls, list) or not calls: - no_tool_retries += 1 - metrics.parse_failures += 1 - if max_seq_len is not None and ( - usage_missing or last_total_tokens >= max_seq_len - ): - metrics.max_seq_len_hit = 1 - return "max_seq_len" - if no_tool_retries >= max_no_tool: - return "max_turns" - messages.append( - { - "role": "user", - "content": ( - "You must act through a structured terminal.exec or submit " - "tool call. Do not answer with prose or a bare command." - ), - } - ) - continue - no_tool_retries = 0 - submitted = False - for raw_call in calls: - metrics.tool_calls += 1 - call = raw_call if isinstance(raw_call, dict) else {} - call_id = call.get("id") - try: - name, arguments = _tool_arguments(call) - except ValueError as exc: - metrics.parse_failures += 1 - messages.append(_tool_message(call_id, {"error": str(exc)})) - continue - if name == "terminal.exec": - command = arguments.get("command") - if not isinstance(command, str) or not command.strip(): - messages.append( - _tool_message(call_id, {"error": "command must be non-empty"}) - ) - continue - requested_timeout = arguments.get("timeout_seconds", tool_timeout) - try: - requested_timeout = min(float(requested_timeout), tool_timeout) - except (TypeError, ValueError): - requested_timeout = tool_timeout - started = time.monotonic() - try: - result = await client.execute( - episode_id, - command, - timeout_seconds=max(1.0, requested_timeout), - output_bytes=output_bytes, - ) - except EpisodeAPIError as exc: - if exc.status == 400: - messages.append(_tool_message(call_id, {"error": exc.message})) - continue - raise - finally: - metrics.total_tool_time += time.monotonic() - started - metrics.terminal_calls += 1 - messages.append( - _tool_message( - call_id, _terminal_observation(result, observation_chars) - ) - ) - elif name == "submit": - started = time.monotonic() - try: - result = await client.submit(episode_id, arguments) - except EpisodeAPIError as exc: - if exc.status == 400: - messages.append(_tool_message(call_id, {"error": exc.message})) - continue - raise - finally: - metrics.total_tool_time += time.monotonic() - started - metrics.submit_calls += 1 - messages.append(_tool_message(call_id, result)) - submitted = True - else: - messages.append( - _tool_message(call_id, {"error": f"unknown tool {name!r}"}) - ) - if submitted: - return "completed" - if max_seq_len is not None and ( - usage_missing or last_total_tokens >= max_seq_len - ): - metrics.max_seq_len_hit = 1 - return "max_seq_len" - return "max_turns" - - -def _validated_outcome( - result: dict[str, Any], - task_id: str, - episode_id: str, - status: str, - *, - infrastructure_503_retries: int = 0, -) -> dict[str, Any]: - if result.get("task_id") != task_id or result.get("episode_id") != episode_id: - raise EpisodeClientError("evaluation response identity mismatch") - reward = result.get("reward") - passed = result.get("passed") - if ( - isinstance(reward, bool) - or not isinstance(reward, (int, float)) - or not math.isfinite(float(reward)) - or not 0.0 <= float(reward) <= 1.0 - or not isinstance(passed, bool) - ): - raise EpisodeClientError("evaluation response has an invalid verdict") - grader = result.get("grader") if isinstance(result.get("grader"), dict) else {} - return { - "schema": 1, - "status": status, - "episode_id": episode_id, - "task_id": task_id, - "reward": float(reward), - "passed": passed, - "class": grader.get("class") if isinstance(grader.get("class"), str) else None, - INFRASTRUCTURE_503_RETRIES_KEY: infrastructure_503_retries, - } - - -def _infrastructure_outcome( - task_id: str, - episode_id: str, - infrastructure_503_retries: int = 0, -) -> dict[str, Any]: - """Build an abort signal, never a synthetic grader verdict.""" - - return { - "schema": 1, - "status": INFRASTRUCTURE_STATUS, - "episode_id": episode_id, - "task_id": task_id, - "reward": 0.0, - "passed": False, - "class": None, - INFRASTRUCTURE_503_RETRIES_KEY: infrastructure_503_retries, - } - - -def _admission_infrastructure_outcome( - task_id: str, - infrastructure_503_retries: int, - *, - nonce: str | None = None, -) -> dict[str, Any]: - """Build evidence for one exhausted authenticated pre-create 503.""" - - if infrastructure_503_retries != 1: - raise ValueError("invalid SecRLEnv admission retry ledger") - return { - "schema": ADMISSION_SCHEMA, - "status": INFRASTRUCTURE_STATUS, - "episode_id": None, - "task_id": task_id, - "reward": 0.0, - "passed": False, - "class": None, - INFRASTRUCTURE_503_RETRIES_KEY: infrastructure_503_retries, - ADMISSION_PHASE_KEY: ADMISSION_PHASE, - ADMISSION_ERROR_CODE_KEY: ADMISSION_ERROR_CODE, - ADMISSION_NONCE_KEY: nonce or secrets.token_hex(16), - } - - -def _agent_result( - outcome: dict[str, Any], outcome_status: str, metrics: AgentMetrics -) -> dict[str, Any]: - result = { - OUTCOME_KEY: outcome, - "exit_status": outcome_status, - "agent_metrics": metrics.to_dict(), - } - capture_agent_metadata(result) - return result - - -def _cleanup_error_outcome( - task_id: str, - episode_id: str, - infrastructure_503_retries: int = 0, -) -> dict[str, Any]: - """Build a terminal signal when episode release is unproven.""" - - outcome = _infrastructure_outcome( - task_id, - episode_id, - infrastructure_503_retries, - ) - outcome["status"] = CLEANUP_ERROR_STATUS - return outcome - - -async def _create_with_capacity_retry( - client: EpisodeClient, - task_id: str, - tier: str, - *, - retry_ledger: dict[str, int] | None = None, -) -> dict[str, Any]: - max_wait = _positive_env("SECRLENV_CAPACITY_MAX_WAIT_SECONDS", 14400.0) - started = time.monotonic() - deadline = started + max_wait - attempts = 0 - if retry_ledger is None: - retry_ledger = {INFRASTRUCTURE_503_RETRIES_KEY: 0} - infrastructure_retries = retry_ledger.get(INFRASTRUCTURE_503_RETRIES_KEY) - if type(infrastructure_retries) is not int or infrastructure_retries not in {0, 1}: - raise ValueError("invalid SecRLEnv infrastructure retry ledger") - next_log = started - last_retryable: EpisodeAPIError | None = None - while True: - remaining: float | None = None - if last_retryable is not None: - remaining = deadline - time.monotonic() - if remaining <= 0: - raise last_retryable - attempts += 1 - try: - if remaining is None: - result = await client.create(task_id, tier) - else: - create_task = asyncio.create_task(client.create(task_id, tier)) - try: - result = await asyncio.wait_for(create_task, timeout=remaining) - except asyncio.TimeoutError: - if last_retryable.code == ADMISSION_ERROR_CODE: - raise EpisodeClientError( - "secrlenv episode admission retry timed out" - ) from None - raise last_retryable from None - except EpisodeAPIError as exc: - now = time.monotonic() - if exc.status == 503 and exc.code == "capacity_reached": - reason = "capacity" - elif exc.status == 503 and exc.code == ADMISSION_ERROR_CODE: - if infrastructure_retries >= 1: - raise _AdmissionInfrastructureExhausted() from None - infrastructure_retries += 1 - retry_ledger[INFRASTRUCTURE_503_RETRIES_KEY] = infrastructure_retries - reason = "infrastructure" - else: - raise - if now >= deadline: - raise - last_retryable = exc - else: - # Once create returns an episode it must be tracked and closed. A - # deadline-edge success cannot be discarded as a stale prior 503. - return result - if now >= next_log: - LOGGER.warning( - "waiting for secrlenv episode daemon reason=%s attempts=%d elapsed_s=%.1f max_wait_s=%.1f", - reason, - attempts, - now - started, - max_wait, - ) - next_log = now + 15.0 - await asyncio.sleep(min(random.uniform(0.5, 3.0), max(0.0, deadline - now))) - - -async def run( - base_url: str, - prompt: Any, - request_kwargs: dict[str, Any] | None = None, - metadata: dict[str, Any] | None = None, - **_kwargs: Any, -) -> dict[str, Any] | None: - """Run one model-driven secrlenv episode and return reward metadata.""" - - del prompt # The authenticated daemon supplies the immutable task prompt. - metadata = dict(metadata or {}) - request_kwargs = dict(request_kwargs or {}) - try: - task_id, tier = _task_identity(metadata) - max_seq_len = _metadata_max_seq_len(metadata) - infrastructure_503_retries = metadata.get( - INFRASTRUCTURE_503_RETRIES_KEY, 0 - ) - if ( - type(infrastructure_503_retries) is not int - or infrastructure_503_retries not in {0, 1} - ): - raise ValueError("invalid SecRLEnv infrastructure retry ledger") - except ValueError as exc: - LOGGER.error("invalid secrlenv sample metadata: %s", exc) - return None - - metrics = AgentMetrics() - episode_id: str | None = None - outcome_status = "completed" - infrastructure_failure = False - cleanup_failure = False - evaluation: dict[str, Any] | None = None - policy = None - policy_task: asyncio.Task[str] | None = None - retry_ledger = { - INFRASTRUCTURE_503_RETRIES_KEY: infrastructure_503_retries - } - try: - from openai import AsyncOpenAI - - policy = AsyncOpenAI(base_url=_session_url(base_url), api_key="EMPTY") - async with EpisodeClient() as client: - started = time.monotonic() - episode = await _create_with_capacity_retry( - client, task_id, tier, retry_ledger=retry_ledger - ) - metrics.create_time = time.monotonic() - started - episode_id = episode.get("episode_id") - if not isinstance(episode_id, str): - raise EpisodeClientError("episode daemon returned no episode ID") - - rollout_timeout = _positive_env( - "SECRLENV_MAX_ROLLOUT_TIME_SECONDS", 3600.0 - ) - policy_task = asyncio.create_task( - _await_policy_and_claim_finalization( - episode_id, - _drive_policy( - policy, - client, - episode, - request_kwargs, - metrics, - max_seq_len=max_seq_len, - ), - ) - ) - _register_episode(episode_id, policy_task) - try: - outcome_status = await asyncio.wait_for( - policy_task, - timeout=rollout_timeout, - ) - except asyncio.TimeoutError: - metrics.timed_out = 1 - outcome_status = "timeout" - LOGGER.warning( - "secrlenv episode %s exceeded %.1fs; evaluating partial state", - episode_id, - rollout_timeout, - ) - except asyncio.CancelledError: - raise - except Exception: - infrastructure_failure = True - LOGGER.exception("secrlenv policy/tool loop failed") - - if _claim_episode_finalization(episode_id): - started = time.monotonic() - try: - evaluation = await client.evaluate(episode_id) - except Exception: - infrastructure_failure = True - LOGGER.exception("secrlenv evaluation failed") - finally: - metrics.evaluate_time = time.monotonic() - started - - started = time.monotonic() - try: - await client.close(episode_id) - except asyncio.CancelledError: - raise - except EpisodeAPIError as exc: - if exc.status == 404: - _release_episode(episode_id) - else: - infrastructure_failure = True - LOGGER.exception("secrlenv teardown failed") - cleanup_failure = not await _recover_failed_normal_close( - client, episode_id, policy_task - ) - except Exception: - infrastructure_failure = True - LOGGER.exception("secrlenv teardown failed") - cleanup_failure = not await _recover_failed_normal_close( - client, episode_id, policy_task - ) - else: - _release_episode(episode_id) - finally: - metrics.close_time = time.monotonic() - started - metrics.total_tool_time += ( - metrics.create_time - + metrics.evaluate_time - + metrics.close_time - ) - else: - infrastructure_failure = True - LOGGER.warning( - "secrlenv episode was claimed by the abort hook before evaluation" - ) - except asyncio.CancelledError: - LOGGER.warning("secrlenv rollout cancelled") - cleanup_claimed = episode_id is not None and ( - _claim_episode_cleanup(episode_id) - or _claim_cancelled_policy_cleanup(episode_id, policy_task) - ) - if cleanup_claimed: - try: - async with EpisodeClient(total_timeout_seconds=180.0) as cleanup_client: - retry_timeout = _positive_env( - "SECRLENV_ABORT_CLOSE_RETRY_SECONDS", 180.0 - ) - await _close_aborted_episode( - cleanup_client, - episode_id, - retry_timeout_seconds=retry_timeout, - ) - except Exception: - LOGGER.warning( - "cancelled episode cleanup failed for %s", episode_id, exc_info=True - ) - finally: - _mark_episode_cleanup_pending(episode_id) - raise - except _AdmissionInfrastructureExhausted: - if ( - episode_id is None - and retry_ledger[INFRASTRUCTURE_503_RETRIES_KEY] == 1 - ): - try: - outcome = _admission_infrastructure_outcome( - task_id, - retry_ledger[INFRASTRUCTURE_503_RETRIES_KEY], - ) - return _agent_result(outcome, INFRASTRUCTURE_STATUS, metrics) - except Exception: - LOGGER.exception("refusing untrusted secrlenv admission metadata") - return None - LOGGER.error("invalid secrlenv admission infrastructure state") - return None - except EpisodeAPIError: - LOGGER.exception("secrlenv episode failed before a trustworthy verdict") - return None - except Exception: - LOGGER.exception("secrlenv episode failed before a trustworthy verdict") - return None - finally: - if policy is not None: - try: - await policy.close() - except Exception: - LOGGER.warning("failed to close policy client", exc_info=True) - - if episode_id is None: - return None - try: - if cleanup_failure: - outcome_status = CLEANUP_ERROR_STATUS - outcome = _cleanup_error_outcome( - task_id, - episode_id, - retry_ledger[INFRASTRUCTURE_503_RETRIES_KEY], - ) - elif infrastructure_failure or evaluation is None: - outcome_status = INFRASTRUCTURE_STATUS - outcome = _infrastructure_outcome( - task_id, - episode_id, - retry_ledger[INFRASTRUCTURE_503_RETRIES_KEY], - ) - else: - outcome = _validated_outcome( - evaluation, - task_id, - episode_id, - outcome_status, - infrastructure_503_retries=retry_ledger[ - INFRASTRUCTURE_503_RETRIES_KEY - ], - ) - result = _agent_result(outcome, outcome_status, metrics) - except Exception: - LOGGER.exception("refusing untrusted secrlenv evaluation metadata") - return None - return result - - -async def abort(_args: Any = None) -> None: - """Miles oversampling abort hook: tear down every in-flight local episode.""" - - claimed = _claim_episodes_and_tasks_for_abort() - if not claimed: - return - try: - retry_timeout = _positive_env("SECRLENV_ABORT_CLOSE_RETRY_SECONDS", 180.0) - drain_timeout = _positive_env("SECRLENV_ABORT_DRAIN_TIMEOUT_SECONDS", 30.0) - drained = await asyncio.gather( - *( - _drain_aborted_policy_task( - task, - timeout_seconds=drain_timeout, - ) - for _episode_id, task in claimed - ) - ) - close_ids = [] - for (episode_id, _task), task_drained in zip(claimed, drained, strict=True): - if task_drained: - close_ids.append(episode_id) - if not close_ids: - return - async with EpisodeClient(total_timeout_seconds=180.0) as client: - await asyncio.gather( - *( - _close_aborted_episode( - client, - episode_id, - retry_timeout_seconds=retry_timeout, - ) - for episode_id in close_ids - ) - ) - except asyncio.CancelledError: - raise - except Exception: - LOGGER.warning("secrlenv abort cleanup failed", exc_info=True) - finally: - for episode_id, _task in claimed: - _mark_episode_cleanup_pending(episode_id) diff --git a/yeto/rl/harness/codex/alignment.py b/yeto/rl/harness/codex/alignment.py deleted file mode 100644 index 0432d100c..000000000 --- a/yeto/rl/harness/codex/alignment.py +++ /dev/null @@ -1,75 +0,0 @@ -"""Mask / token / logprob alignment contract for agentic samples (design D5 assertions). - -A sample produced by the session server (``compute_samples_from_openai_records`` -+ ``merge_samples``) must satisfy, fail closed: - -- ``len(tokens) == len(loss_mask)`` (loss_mask covers the whole sequence), or - ``len(loss_mask) == response_length`` (per-turn layout before merge); -- ``rollout_log_probs`` has one entry per response token; -- every mask=1 position lies in the response region and has a finite logprob; -- mask=1 positions are exactly the generated spans when ``weight_versions`` - spans are present (they delimit SGLang output); tool/template tokens are mask 0. - -Violations raise ``AlignmentError``; callers abort the trajectory and never -convert the failure into a 0 reward. -""" - -from __future__ import annotations - -import math -from typing import Any - - -class AlignmentError(ValueError): - pass - - -def _response_offset(sample: Any) -> tuple[int, int]: - tokens = list(getattr(sample, "tokens", None) or []) - response_length = getattr(sample, "response_length", None) - if not isinstance(response_length, int) or response_length < 0 or response_length > len(tokens): - raise AlignmentError("response_length is missing or out of range") - return len(tokens) - response_length, len(tokens) - - -def generated_spans(sample: Any) -> list[tuple[int, int]]: - spans: list[tuple[int, int]] = [] - for call in getattr(sample, "weight_versions", None) or []: - for span in getattr(call, "spans", None) or (call.get("spans", []) if isinstance(call, dict) else []): - start = getattr(span, "abs_start", None) if not isinstance(span, dict) else span.get("abs_start") - end = getattr(span, "abs_end", None) if not isinstance(span, dict) else span.get("abs_end") - if not isinstance(start, int) or not isinstance(end, int) or start < 0 or end < start: - raise AlignmentError("weight version span is malformed") - spans.append((start, end)) - return spans - - -def assert_sample_alignment(sample: Any) -> None: - tokens = list(getattr(sample, "tokens", None) or []) - loss_mask = getattr(sample, "loss_mask", None) - logprobs = getattr(sample, "rollout_log_probs", None) - if loss_mask is None: - raise AlignmentError("loss_mask is missing") - if logprobs is None: - raise AlignmentError("rollout_log_probs is missing") - start, end = _response_offset(sample) - if len(loss_mask) == len(tokens): - mask_offset = 0 - elif len(loss_mask) == end - start: - mask_offset = start - else: - raise AlignmentError(f"len(loss_mask)={len(loss_mask)} matches neither len(tokens)={len(tokens)} nor response_length={end - start}") - if len(logprobs) != end - start: - raise AlignmentError(f"len(rollout_log_probs)={len(logprobs)} != response_length={end - start}") - ones = {mask_offset + i for i, m in enumerate(loss_mask) if int(m) == 1} - for position in sorted(ones): - if position < start: - raise AlignmentError(f"loss_mask=1 at prompt position {position}") - value = logprobs[position - start] - if value is None or not isinstance(value, (int, float)) or not math.isfinite(float(value)): - raise AlignmentError(f"loss_mask=1 at {position} without a finite rollout logprob") - spans = generated_spans(sample) - if spans: - generated = {p for s, e in spans for p in range(s, e)} - if ones - generated: - raise AlignmentError(f"loss_mask=1 outside generated spans at {sorted(ones - generated)[:5]}") diff --git a/yeto/rl/harness/codex/chain_judge.py b/yeto/rl/harness/codex/chain_judge.py deleted file mode 100644 index b9c97bd8a..000000000 --- a/yeto/rl/harness/codex/chain_judge.py +++ /dev/null @@ -1,78 +0,0 @@ -"""R-D5a chain-count judge for Codex GPU runs (rl-codex-harness-rollout design 9.2). - -The GPU 9.2 hard criterion used to be stated only in prose: every trajectory -has ``chains_total == 1``. With CompactionRL rollout compaction on -(``YETO_CODEX_COMPACTIONRL``, progress.md "S13 Codex 压缩接线") a trajectory is -deliberately split into one sample per compaction segment, so the criterion -becomes, per trajectory: - -- ``chains_total == num_segments == compactions + 1 == number of samples``; -- ``chain_index`` / ``segment_index`` are exactly ``0..n-1``; -- ``chain_break_reason`` is ``compaction_window`` on every sample but the first - (None there). - -With compaction off the criterion is unchanged: ``chains_total == 1`` on every -sample (and one sample per trajectory). Aborted (infrastructure) samples are -judged the same way: the generate wrapper keeps an aborted compacted -trajectory as its single segment-0 sample, which satisfies the off rule. -""" - -from __future__ import annotations - -from collections.abc import Iterable, Mapping -from typing import Any - -COMPACTION_BREAK_REASON = "compaction_window" - - -def _key(meta: Mapping[str, Any], position: int) -> str: - trajectory = meta.get("trajectory_id") - return str(trajectory) if trajectory not in (None, "") else f"" - - -def judge_chain_counts( - samples_metadata: Iterable[Mapping[str, Any]], - *, - compaction_enabled: bool, -) -> list[str]: - """Return the violations (empty list = criterion holds).""" - groups: dict[str, list[Mapping[str, Any]]] = {} - for position, meta in enumerate(samples_metadata): - groups.setdefault(_key(meta, position), []).append(meta) - problems: list[str] = [] - for trajectory, metas in groups.items(): - totals = {m.get("chains_total") for m in metas} - if not compaction_enabled: - if totals != {1} or len(metas) != 1: - problems.append( - f"{trajectory}: chains_total {sorted(map(str, totals))} over {len(metas)} sample(s); " - "expected exactly one sample with chains_total==1" - ) - continue - # Aborted compacted trajectories collapse to one sample (no segment keys). - if len(metas) == 1 and metas[0].get("num_segments") is None: - if totals != {1}: - problems.append(f"{trajectory}: single sample with chains_total {sorted(map(str, totals))}") - continue - n = len(metas) - segments = {m.get("num_segments") for m in metas} - compactions = {m.get("compactions") for m in metas} - if totals != {n} or segments != {n}: - problems.append( - f"{trajectory}: {n} sample(s) but chains_total {sorted(map(str, totals))}, " - f"num_segments {sorted(map(str, segments))}" - ) - if compactions != {n - 1}: - problems.append( - f"{trajectory}: compactions {sorted(map(str, compactions))} != num_segments-1 ({n - 1})" - ) - order = sorted(metas, key=lambda m: (m.get("chain_index") is None, m.get("chain_index") or 0)) - if [m.get("chain_index") for m in order] != list(range(n)) or [ - m.get("segment_index") for m in order - ] != list(range(n)): - problems.append(f"{trajectory}: chain_index/segment_index are not 0..{n - 1}") - continue - reasons = [m.get("chain_break_reason") for m in order] - if reasons != [None] + [COMPACTION_BREAK_REASON] * (n - 1): - problems.append(f"{trajectory}: chain_break_reason {reasons}") - return problems diff --git a/yeto/rl/harness/codex/client.py b/yeto/rl/harness/codex/client.py deleted file mode 100644 index 138059fca..000000000 --- a/yeto/rl/harness/codex/client.py +++ /dev/null @@ -1,215 +0,0 @@ -"""Authenticated async client for the loopback secrlenv episode daemon.""" - -from __future__ import annotations - -import json -import math -import os -import stat -from pathlib import Path -from typing import Any, Self -from urllib.error import HTTPError, URLError -from urllib.parse import urlsplit -from urllib.request import urlopen - -import aiohttp - - -class EpisodeClientError(RuntimeError): - pass - - -class EpisodeAPIError(EpisodeClientError): - def __init__(self, status: int, code: str, message: str) -> None: - super().__init__(f"episode daemon returned {status} {code}: {message}") - self.status = status - self.code = code - self.message = message - - -class EpisodeTransportError(EpisodeClientError): - pass - - -def _read_token() -> str: - token_file = os.getenv("SECRLENV_BEARER_TOKEN_FILE") - if token_file: - path = Path(token_file) - try: - info = path.stat() - if not stat.S_ISREG(info.st_mode) or info.st_mode & 0o077: - raise EpisodeClientError( - "SECRLENV_BEARER_TOKEN_FILE must be a private regular file" - ) - token = path.read_text(encoding="utf-8").strip() - except OSError as exc: - raise EpisodeClientError(f"cannot read episode daemon token file: {exc}") from exc - else: - token = os.getenv("SECRLENV_BEARER_TOKEN", "").strip() - if len(token) < 32: - raise EpisodeClientError("episode daemon bearer token is missing or too short") - return token - - -def daemon_url_from_env() -> str: - value = os.getenv("SECRLENV_DAEMON_URL", "http://127.0.0.1:8765").rstrip("/") - parsed = urlsplit(value) - if ( - parsed.scheme != "http" - or parsed.hostname not in {"127.0.0.1", "::1", "localhost"} - or parsed.username is not None - or parsed.password is not None - or parsed.path not in {"", "/"} - or parsed.query - or parsed.fragment - or parsed.port is None - ): - raise EpisodeClientError( - "SECRLENV_DAEMON_URL must be an unauthenticated loopback HTTP origin" - ) - return value - - -def require_daemon_ready(*, timeout_seconds: float = 10.0) -> None: - """Fail before GPU initialization unless the expected daemon is healthy.""" - - if not math.isfinite(timeout_seconds) or timeout_seconds <= 0: - raise ValueError("episode daemon readiness timeout must be positive") - try: - with urlopen( - f"{daemon_url_from_env()}/healthz", timeout=timeout_seconds - ) as response: - if response.status != 200: - raise EpisodeTransportError( - f"secrlenv episode daemon health check returned {response.status}" - ) - value = json.loads(response.read()) - except EpisodeTransportError: - raise - except (HTTPError, URLError, OSError, json.JSONDecodeError) as exc: - raise EpisodeTransportError( - "secrlenv episode daemon is not healthy on its configured loopback origin" - ) from exc - if not isinstance(value, dict) or value.get("ok") is not True: - raise EpisodeTransportError("secrlenv episode daemon health response is invalid") - expected_task_pack = os.getenv("SECRLENV_TASK_PACK_SHA256", "").strip() - if expected_task_pack and value.get("task_pack_sha256") != expected_task_pack: - raise EpisodeTransportError( - "secrlenv episode daemon task-pack identity does not match the run plan" - ) - - -class EpisodeClient: - def __init__( - self, - base_url: str | None = None, - token: str | None = None, - *, - total_timeout_seconds: float = 1200.0, - ) -> None: - self.base_url = (base_url or daemon_url_from_env()).rstrip("/") - # Apply the same validation to explicit constructor values used in tests - # and direct-drive tools, not only to the environment-derived default. - parsed = urlsplit(self.base_url) - if ( - parsed.scheme != "http" - or parsed.hostname not in {"127.0.0.1", "::1", "localhost"} - or parsed.username is not None - or parsed.password is not None - or parsed.path not in {"", "/"} - or parsed.query - or parsed.fragment - or parsed.port is None - ): - raise EpisodeClientError("episode daemon URL must be a loopback HTTP origin") - self.token = token or _read_token() - if len(self.token) < 32: - raise EpisodeClientError("episode daemon bearer token is too short") - self.timeout = aiohttp.ClientTimeout(total=total_timeout_seconds) - self._session: aiohttp.ClientSession | None = None - - async def __aenter__(self) -> Self: - self._session = aiohttp.ClientSession( - timeout=self.timeout, - connector=aiohttp.TCPConnector(limit=4), - headers={"Authorization": f"Bearer {self.token}"}, - ) - return self - - async def __aexit__(self, *_exc: object) -> None: - if self._session is not None: - await self._session.close() - self._session = None - - async def _request( - self, - method: str, - path: str, - body: dict[str, Any] | None = None, - ) -> dict[str, Any]: - if self._session is None: - raise EpisodeClientError("EpisodeClient must be used as an async context manager") - try: - async with self._session.request( - method, - f"{self.base_url}{path}", - json=body, - headers={"Content-Type": "application/json"}, - ) as response: - raw = await response.text() - except (aiohttp.ClientError, TimeoutError) as exc: - raise EpisodeTransportError(f"episode daemon request failed: {exc}") from exc - try: - value = json.loads(raw) - except json.JSONDecodeError as exc: - raise EpisodeTransportError("episode daemon returned invalid JSON") from exc - if not isinstance(value, dict): - raise EpisodeTransportError("episode daemon returned a non-object response") - if not 200 <= response.status < 300: - error = value.get("error") if isinstance(value.get("error"), dict) else {} - raise EpisodeAPIError( - response.status, - str(error.get("code", "unknown_error")), - str(error.get("message", "request failed")), - ) - return value - - async def create(self, task_id: str, prompt_tier: str) -> dict[str, Any]: - return await self._request( - "POST", - "/v1/episodes", - {"task_id": task_id, "prompt_tier": prompt_tier}, - ) - - async def execute( - self, - episode_id: str, - command: str, - *, - timeout_seconds: float, - output_bytes: int, - ) -> dict[str, Any]: - return await self._request( - "POST", - f"/v1/episodes/{episode_id}/exec", - { - "command": command, - "timeout_seconds": timeout_seconds, - "output_bytes": output_bytes, - }, - ) - - async def submit(self, episode_id: str, submission: dict[str, Any]) -> dict[str, Any]: - return await self._request( - "POST", - f"/v1/episodes/{episode_id}/submit", - {"submission": submission}, - ) - - async def evaluate(self, episode_id: str) -> dict[str, Any]: - return await self._request( - "POST", f"/v1/episodes/{episode_id}/evaluate", {} - ) - - async def close(self, episode_id: str) -> dict[str, Any]: - return await self._request("DELETE", f"/v1/episodes/{episode_id}") diff --git a/yeto/rl/harness/codex/codex_harness_agent.py b/yeto/rl/harness/codex/codex_harness_agent.py deleted file mode 100644 index aab59eb2e..000000000 --- a/yeto/rl/harness/codex/codex_harness_agent.py +++ /dev/null @@ -1,2389 +0,0 @@ -"""Pinned stock-Codex harness for isolated Miles/SecRLEnv episodes. - -The stock Codex app-server owns the agent loop. A per-episode loopback bridge -adapts its Responses API calls to the one Miles session-server Chat Completions -endpoint so Miles remains the sole sampler and logprob/TITO owner. -""" - -from __future__ import annotations - -import asyncio -import copy -import errno -import hashlib -import hmac -import json -import logging -import math -import os -import re -import secrets -import shutil -import signal -import stat -import subprocess -import tempfile -import time -from dataclasses import dataclass -from pathlib import Path -from threading import Lock -from typing import Any, Self - -import aiohttp -from aiohttp import web - -from yeto.rl.codex_backend import stock_codex_backend_profile - -from . import agent as legacy -from .client import EpisodeAPIError, EpisodeClient, EpisodeClientError -from .generate import capture_agent_metadata -from .reward import ( - INFRASTRUCTURE_503_RETRIES_KEY, - INFRASTRUCTURE_STATUS, - OUTCOME_KEY, -) - -LOGGER = logging.getLogger(__name__) - -CODEX_APP_SERVER_PROTOCOL_REVISION = "v2" -CODEX_APP_SERVER_SCHEMA_SHA256 = ( - "a88d865c3ca41fc63672baf423e28b3bfdb85b989e1c95e6e268932daaac91a0" -) -CODEX_CLI_VERSION = "codex-cli 0.145.0" -_BACKEND_PROFILE = stock_codex_backend_profile( - os.getenv("YETO_CODEX_CHAT_TEMPLATE", "deepseekv4") -) -BACKEND_MODEL = _BACKEND_PROFILE["model"] -BACKEND_REASONING_EFFORT = _BACKEND_PROFILE["backend_reasoning_effort"] -BACKEND_CHAT_TEMPLATE_KWARGS = _BACKEND_PROFILE["chat_template_kwargs"] -# rlf-codex-rl 5.6: the tool-output budget is a profile field. The -# qwen3_30b_a3b cyber baseline pins 8192 bytes; profiles without a declaration -# keep the signed 32 KiB default. -MAX_TOOL_OUTPUT_BYTES = int(_BACKEND_PROFILE.get("tool_output_bytes", 32_768)) -# S24 T-TB2: Qwen3.8-27B habitually writes one line of prose before its tool -# call. Profiles that opt in keep that prose in the assistant message (TITO -# history stays exactly what the model generated); Codex still sees only the -# tool call. Every other profile keeps the strict DSV4 rule. -ALLOW_PROSE_WITH_TOOL_CALL = bool(_BACKEND_PROFILE.get("allow_prose_with_tool_call", False)) -CODEX_TOOL_OUTPUT_TOKEN_LIMIT = 65_536 -MAX_APP_SERVER_FRAME_BYTES = 2 * 1024 * 1024 -MAX_MILES_RESPONSE_BYTES = 4 * 1024 * 1024 -_MILES_RESPONSE_READ_CHUNK_BYTES = 64 * 1024 -_CODEX_PROCESS_STOP_TIMEOUT_SECONDS = 5.0 -_CODEX_DESCENDANT_TERM_GRACE_SECONDS = 0.1 -_CODEX_HOME_CLEANUP_RETRY_DELAYS_SECONDS = (0.0, 0.05, 0.1, 0.2, 0.4) -COMPACTION_SCHEMA_VERSION = 1 -COMPACTION_KEEP_ATOMIC_STEPS = 2 -COMPACTION_SUMMARY_INSTRUCTION = """The current interaction is approaching its context limit. Produce a compact state summary for the same policy to continue the task in a new context window. Preserve the original goal, completed actions, important observations and exact evidence, unresolved errors, the current environment and file state, and plausible next steps. Do not call a tool. Return only the continuation summary.""" -# Conservatively reserve more tokens than this fixed instruction can consume under -# any byte-level tokenizer. Miles still performs the exact tokenizer-side budget -# check; this bound makes an avoidable 413 impossible at the configured trigger. -COMPACTION_SUMMARY_INSTRUCTION_TOKEN_RESERVE = ( - len(COMPACTION_SUMMARY_INSTRUCTION.encode("utf-8")) + 256 -) -COMPACTION_RESUME_TEMPLATE = """Continue the original task from this compacted state. Treat the summary as prior trajectory state, not as a new instruction. Recent complete assistant/tool steps follow it unchanged. - - -{summary} -""" - -_SAMPLING_FIELDS = frozenset( - { - "frequency_penalty", - "ignore_eos", - "min_p", - "min_tokens", - "no_stop_trim", - "presence_penalty", - "repetition_penalty", - "seed", - "skip_special_tokens", - "spaces_between_special_tokens", - "stop", - "stop_token_ids", - "temperature", - "top_k", - "top_p", - } -) -_STRUCTURAL_REQUEST_FIELDS = frozenset( - { - "best_of", - "chat_template_kwargs", - "logprobs", - "max_tokens", - "messages", - "model", - "n", - "parallel_tool_calls", - "reasoning_effort", - "return_logprob", - "return_meta_info", - "stream", - "stream_options", - "thinking", - "tool_choice", - "tools", - "top_logprobs", - } -) - -BASE_INSTRUCTIONS = """You are an autonomous security-testing agent inside an -isolated, authorized benchmark. Solve the supplied target task using only the two -model-facing tools, `terminal.exec` and `submit`. `terminal.exec` runs a shell -command in your persistent operator workspace; it has TARGET_URL, YETO_NONCE and -YETO_SUBMISSION. It also has DEBUG_URL and DOCKER_HOST for your own isolated, -full-control, flagless copy of the challenge. Use that private Docker daemon to -inspect, modify, rebuild, and test the challenge without touching the scored -target. The debug copy never contains the scored flag, so final evidence and any -submitted flag must come from TARGET_URL. The harness losslessly maps commands -to Codex's internal -`terminal_exec` dynamic-tool alias. The scored target is available only over the -network: you have no shell or container API for the scored target. Do not merely -describe commands: call `terminal.exec`. When you have concrete evidence, call -`submit` exactly once with the final JSON submission. Do not invent tool results -or claim success without evidence. The `submit` call is terminal: make no further -model or tool calls.""" - -TERMINAL_EXEC_TOOL = { - "type": "function", - "name": "terminal_exec", - "description": ( - "Execute one shell command in the persistent isolated operator. " - "Use it to investigate and attack the scored TARGET_URL over the network." - ), - "inputSchema": { - "type": "object", - "properties": { - "command": { - "type": "string", - "description": "The exact bash command to execute.", - }, - "timeout_seconds": { - "type": "number", - "minimum": 1, - "maximum": 120, - }, - }, - "required": ["command"], - "additionalProperties": False, - }, -} - -SUBMIT_TOOL = { - "type": "function", - "name": "submit", - "description": ( - "Make the episode's one final submission. Call only after obtaining " - "real evidence from the scored target. The full arguments object is " - "written as submission.json and immediately finalized." - ), - "inputSchema": { - "type": "object", - "properties": { - "evidence": { - "type": "string", - "description": "Concrete evidence required by the task grader.", - }, - "flag": {"type": "string"}, - "class": {"type": "string"}, - "requests": { - "type": "array", - "items": {"type": "string"}, - }, - }, - "required": ["evidence"], - "additionalProperties": True, - }, -} - -DYNAMIC_TOOLS = [TERMINAL_EXEC_TOOL, SUBMIT_TOOL] - -# 2026-10-07 (S15 r2 follow-up): the legacy tool descriptions above are CTF -# wording ("attack the scored TARGET_URL", "evidence from the scored target"). -# The TB2 family keeps names and input schemas and only rewords descriptions. -TB2_TERMINAL_EXEC_TOOL = { - **TERMINAL_EXEC_TOOL, - "description": ( - "Execute one shell command in the task container (the graded " - "environment), starting in the task workdir. Returns exit code and output." - ), -} -TB2_SUBMIT_TOOL = { - **SUBMIT_TOOL, - "description": ( - "Finish the episode after the task is done and verified. The hidden " - "tests then grade the container state. Call exactly once." - ), -} -TB2_DYNAMIC_TOOLS = [TB2_TERMINAL_EXEC_TOOL, TB2_SUBMIT_TOOL] - - -def _canonical_json(value: Any) -> bytes: - return json.dumps( - value, - ensure_ascii=False, - allow_nan=False, - sort_keys=True, - separators=(",", ":"), - ).encode("utf-8") - - -def _sha256_json(value: Any) -> str: - return hashlib.sha256(_canonical_json(value)).hexdigest() - - -BASE_INSTRUCTIONS_SHA256 = hashlib.sha256(BASE_INSTRUCTIONS.encode()).hexdigest() - -# 2026-10-07 (S15 stage 2): Terminal-Bench 2 tasks get their own system prompt. -# The legacy text above is a security-testing/CTF prompt (TARGET_URL, flags, -# "no shell for the scored target"); in TB2 the shell *is* the scored task -# container. Selected per episode by the TB2 environment provider through -# ``YETO_CODEX_INSTRUCTIONS_FAMILY=tb2`` plus the signed -# ``YETO_CODEX_TB2_BASE_INSTRUCTIONS_SHA256`` (worker env); every other path -# keeps BASE_INSTRUCTIONS and its unchanged hash. -TB2_BASE_INSTRUCTIONS = """You are an autonomous terminal agent solving a -Terminal-Bench task. You have a shell inside the task container: the container -is the environment that will be graded, and the shell starts in the task -workdir. The task statement is the user message. - -Reply format (strict, every reply): think briefly, then make exactly ONE tool -call and write no other visible text. A reply with prose, no tool call or -several tool calls ends the episode with no credit. -- `terminal.exec`: run one shell command (inspect, edit, install, build, test). -- `submit`: finish the episode; arguments {"evidence": ""}. - -Budget: about ten model calls and a small context window; long thinking or -long outputs end the episode early. Act from the first reply, prefer compact -commands (write files with a heredoc, limit output with head/tail), verify the -result, then call `submit`. The grader runs the task's hidden tests against the -container state when the episode ends.""" -def _tb2_surface_sha256() -> str: - """Signed TB2 surface: system prompt + model/Codex-facing tool definitions.""" - return hashlib.sha256( - TB2_BASE_INSTRUCTIONS.encode() + b"\0" + _canonical_json(TB2_DYNAMIC_TOOLS) - ).hexdigest() - - -TB2_BASE_INSTRUCTIONS_SHA256 = _tb2_surface_sha256() -INSTRUCTIONS_FAMILY_ENV = "YETO_CODEX_INSTRUCTIONS_FAMILY" -TB2_INSTRUCTIONS_SHA_ENV = "YETO_CODEX_TB2_BASE_INSTRUCTIONS_SHA256" -TB2_INSTRUCTIONS_FAMILY = "tb2" - - -def base_instructions() -> str: - """System prompt for this process: legacy unless the TB2 family is signed in.""" - family = os.getenv(INSTRUCTIONS_FAMILY_ENV, "").strip() - if not family: - return BASE_INSTRUCTIONS - if family != TB2_INSTRUCTIONS_FAMILY: - raise CodexHarnessError(f"{INSTRUCTIONS_FAMILY_ENV}={family!r} is not a signed instructions family") - if _tb2_surface_sha256() != TB2_BASE_INSTRUCTIONS_SHA256 or not hmac.compare_digest( - os.getenv(TB2_INSTRUCTIONS_SHA_ENV, ""), TB2_BASE_INSTRUCTIONS_SHA256 - ): - raise CodexHarnessError(f"{TB2_INSTRUCTIONS_SHA_ENV} does not match the signed TB2 instructions") - return TB2_BASE_INSTRUCTIONS - - -def _tb2_selected() -> bool: - return base_instructions() is TB2_BASE_INSTRUCTIONS - - -def dynamic_tools() -> list[dict[str, Any]]: - """Codex dynamic tools for this process (TB2 wording when TB2 is signed in).""" - return TB2_DYNAMIC_TOOLS if _tb2_selected() else DYNAMIC_TOOLS - - -def _miles_tools_for(tools: list[dict[str, Any]]) -> list[dict[str, Any]]: - return [ - { - "type": "function", - "function": { - "name": "terminal.exec" if tool["name"] == "terminal_exec" else tool["name"], - "description": tool["description"], - "parameters": tool["inputSchema"], - }, - } - for tool in tools - ] - - -def miles_tools() -> list[dict[str, Any]]: - """Model-facing tools sent to Miles (``terminal.exec`` / ``submit``).""" - return _MILES_TOOLS if not _tb2_selected() else _miles_tools_for(TB2_DYNAMIC_TOOLS) -TERMINAL_EXEC_TOOL_SCHEMA_SHA256 = _sha256_json(TERMINAL_EXEC_TOOL) -SUBMIT_TOOL_SCHEMA_SHA256 = _sha256_json(SUBMIT_TOOL) -DYNAMIC_TOOLS_SCHEMA_SHA256 = _sha256_json(DYNAMIC_TOOLS) - -_MILES_TOOLS = [ - { - "type": "function", - "function": { - "name": "terminal.exec", - "description": TERMINAL_EXEC_TOOL["description"], - "parameters": TERMINAL_EXEC_TOOL["inputSchema"], - }, - }, - { - "type": "function", - "function": { - "name": "submit", - "description": SUBMIT_TOOL["description"], - "parameters": SUBMIT_TOOL["inputSchema"], - }, - }, -] - -_IDENTITY_ENV = { - "YETO_CODEX_APP_SERVER_PROTOCOL_REVISION": CODEX_APP_SERVER_PROTOCOL_REVISION, - "YETO_CODEX_APP_SERVER_SCHEMA_SHA256": CODEX_APP_SERVER_SCHEMA_SHA256, - "YETO_CODEX_BASE_INSTRUCTIONS_SHA256": BASE_INSTRUCTIONS_SHA256, - "YETO_CODEX_TERMINAL_EXEC_TOOL_SCHEMA_SHA256": TERMINAL_EXEC_TOOL_SCHEMA_SHA256, - "YETO_CODEX_SUBMIT_TOOL_SCHEMA_SHA256": SUBMIT_TOOL_SCHEMA_SHA256, - "YETO_CODEX_DYNAMIC_TOOLS_SCHEMA_SHA256": DYNAMIC_TOOLS_SCHEMA_SHA256, - "YETO_CODEX_REASONING_EFFORT": "xhigh", - "YETO_CODEX_BACKEND_REASONING_EFFORT": BACKEND_REASONING_EFFORT, - "YETO_CODEX_BACKEND_THINKING": "enabled", - "YETO_CODEX_CHAT_TEMPLATE": _BACKEND_PROFILE["chat_template"], - "YETO_CODEX_CHAT_TEMPLATE_KWARGS": json.dumps( - BACKEND_CHAT_TEMPLATE_KWARGS, - sort_keys=True, - separators=(",", ":"), - ), - "YETO_CODEX_TITO_ALLOWED_APPEND_ROLES": "tool,user", -} -_HEX64 = re.compile(r"[0-9a-f]{64}") -_RUNTIME_ATTESTATION_LOCK = Lock() -_RUNTIME_ATTESTATION_CACHE: dict[tuple[Any, ...], Path] = {} - - -class CodexHarnessError(RuntimeError): - """A fail-closed identity, transport, or protocol violation.""" - - -class CodexSequenceLimit(CodexHarnessError): - """The next sample would exceed the signed Miles sequence limit.""" - - -class _CompactionContextDoesNotFit(CodexHarnessError): - """Miles proved that a reconstructed context exceeds its declared budget.""" - - -class CodexTurnLimit(CodexHarnessError): - """Codex reached the signed model-call budget after a valid trajectory.""" - - -class CodexSuspendExpired(CodexHarnessError): - """agentic-rollout-utilization 5.3: a trajectory suspended between model turns - was not continued within the survival limit (discarded; environment released).""" - - -# agentic-rollout-utilization 5.1/5.3: model-turn gate of THIS trajectory (a file -# path; the file exists = gate closed). The trusted side (``suspend``/``resume`` -# hooks of the subprocess agent function) creates and removes it; the bridge -# checks it only before a model request, so a running tool call is never -# interrupted and its result is already in the history when the request waits. -SUSPEND_GATE_ENV = "YETO_CODEX_SUSPEND_GATE" -SUSPEND_MAX_SECONDS_ENV = "YETO_CODEX_SUSPEND_MAX_SECONDS" -SUSPEND_MAX_SECONDS_DEFAULT = 600.0 -SUSPEND_POLL_SECONDS = 0.25 -SUSPEND_METRIC_FIELDS = ("suspensions", "suspended_seconds", "suspend_retried_turns", "suspend_expired") - - -def suspend_max_seconds() -> float: - raw = os.environ.get(SUSPEND_MAX_SECONDS_ENV) - if raw is None or raw == "": - return SUSPEND_MAX_SECONDS_DEFAULT - value = float(raw) - if not math.isfinite(value) or value <= 0: - raise ValueError(f"{SUSPEND_MAX_SECONDS_ENV} must be a positive number of seconds") - return value - - -# The worker process installs a sink for gate edges ("enter"/"exit") so the -# trusted side's PhaseClock keeps parked time out of the generation phase. -_suspend_event_sink: Any = None - - -def set_suspend_event_sink(sink: Any) -> None: - global _suspend_event_sink - _suspend_event_sink = sink - - -def _suspend_edge(phase: str) -> None: - if _suspend_event_sink is not None: - _suspend_event_sink(phase) - - -class SuspendGate: - """The bridge side of one trajectory's model-turn gate.""" - - def __init__(self, path: str | None, metrics: Any, *, max_seconds: float | None = None, - poll_seconds: float = SUSPEND_POLL_SECONDS, clock: Any = time.monotonic) -> None: - self.path = path or None - self.metrics = metrics - self.max_seconds = suspend_max_seconds() if max_seconds is None and self.path else (max_seconds or 0.0) - self.poll_seconds = poll_seconds - self.clock = clock - for name in SUSPEND_METRIC_FIELDS: - if not hasattr(metrics, name): - setattr(metrics, name, 0.0 if name == "suspended_seconds" else 0) - - @property - def enabled(self) -> bool: - return self.path is not None - - def closed(self) -> bool: - return self.path is not None and os.path.exists(self.path) - - async def wait_open(self) -> None: - """Return at once when the gate is open; else wait (counted as one - suspension) until it opens, or raise :class:`CodexSuspendExpired` after - the survival limit.""" - if not self.closed(): - return - started = self.clock() - self.metrics.suspensions += 1 - _suspend_edge("enter") - try: - while self.closed(): - if self.clock() - started >= self.max_seconds: - self.metrics.suspend_expired = 1 - raise CodexSuspendExpired( - f"suspended trajectory not continued within {self.max_seconds:g} s") - await asyncio.sleep(self.poll_seconds) - finally: - self.metrics.suspended_seconds += self.clock() - started - _suspend_edge("exit") - - -class CodexModelFailure(CodexHarnessError): - """The sampled model turn could not produce one executable tool call.""" - - -def codex_harness_identity() -> dict[str, str]: - """Return live, recomputed identities consumed by Yeto plan attestation.""" - - live = { - "base_instructions_sha256": hashlib.sha256( - BASE_INSTRUCTIONS.encode() - ).hexdigest(), - "terminal_exec_tool_schema_sha256": _sha256_json(TERMINAL_EXEC_TOOL), - "submit_tool_schema_sha256": _sha256_json(SUBMIT_TOOL), - "dynamic_tools_schema_sha256": _sha256_json(DYNAMIC_TOOLS), - } - declared = { - "base_instructions_sha256": BASE_INSTRUCTIONS_SHA256, - "terminal_exec_tool_schema_sha256": TERMINAL_EXEC_TOOL_SCHEMA_SHA256, - "submit_tool_schema_sha256": SUBMIT_TOOL_SCHEMA_SHA256, - "dynamic_tools_schema_sha256": DYNAMIC_TOOLS_SCHEMA_SHA256, - } - if ( - live != declared - or _MILES_TOOLS != legacy.TOOLS - or any(not _HEX64.fullmatch(value) for value in live.values()) - ): - raise CodexHarnessError("Codex harness identity constants drifted") - return live - - -def _positive_int(name: str) -> int: - raw = os.getenv(name, "").strip() - try: - value = int(raw) - except ValueError as exc: - raise CodexHarnessError(f"{name} must be an integer") from exc - if value <= 0: - raise CodexHarnessError(f"{name} must be positive") - return value - - -def _strict_env_flag(name: str, default: bool = False) -> bool: - raw = os.getenv(name, "1" if default else "0").strip() - if raw not in {"0", "1"}: - raise CodexHarnessError(f"{name} must be exactly 0 or 1") - return raw == "1" - - -def _attest_runtime() -> Path: - codex_harness_identity() - for name, expected in _IDENTITY_ENV.items(): - if not hmac.compare_digest(os.getenv(name, ""), expected): - raise CodexHarnessError(f"{name} does not match the signed harness") - binary = Path(os.getenv("YETO_CODEX_BINARY_PATH", "")) - expected_size = _positive_int("YETO_CODEX_BINARY_SIZE_BYTES") - expected_sha = os.getenv("YETO_CODEX_BINARY_SHA256", "") - if not _HEX64.fullmatch(expected_sha): - raise CodexHarnessError("pinned Codex binary digest is invalid") - if not hmac.compare_digest(os.getenv("YETO_CODEX_VERSION", ""), CODEX_CLI_VERSION): - raise CodexHarnessError("pinned Codex version contract drifted") - if _positive_int("YETO_CODEX_BACKEND_MAX_TOKENS") <= 0: - raise CodexHarnessError("invalid Codex backend token budget") - with _RUNTIME_ATTESTATION_LOCK: - try: - info = binary.stat() - except OSError as exc: - raise CodexHarnessError("pinned Codex binary is unavailable") from exc - if ( - binary.is_symlink() - or not stat.S_ISREG(info.st_mode) - or not info.st_mode & 0o111 - ): - raise CodexHarnessError("pinned Codex binary is not a regular executable") - if info.st_size != expected_size: - raise CodexHarnessError("pinned Codex binary size drifted") - identity = ( - str(binary.resolve()), - info.st_dev, - info.st_ino, - info.st_size, - info.st_mtime_ns, - info.st_ctime_ns, - info.st_mode, - expected_sha, - CODEX_CLI_VERSION, - ) - if identity in _RUNTIME_ATTESTATION_CACHE: - return _RUNTIME_ATTESTATION_CACHE[identity] - digest = hashlib.sha256() - try: - with binary.open("rb") as stream: - for chunk in iter(lambda: stream.read(1024 * 1024), b""): - digest.update(chunk) - except OSError as exc: - raise CodexHarnessError("pinned Codex binary could not be read") from exc - if not hmac.compare_digest(digest.hexdigest(), expected_sha): - raise CodexHarnessError("pinned Codex binary digest drifted") - try: - version = subprocess.run( - [str(binary), "--version"], - check=True, - capture_output=True, - text=True, - timeout=15.0, - env={"LANG": "C.UTF-8", "PATH": "/usr/bin:/bin", "TZ": "UTC"}, - ).stdout.strip() - except (OSError, subprocess.SubprocessError) as exc: - raise CodexHarnessError("pinned Codex executable version check failed") from exc - if not hmac.compare_digest(version, CODEX_CLI_VERSION): - raise CodexHarnessError("pinned Codex executable version drifted") - after = binary.stat() - after_identity = ( - str(binary.resolve()), - after.st_dev, - after.st_ino, - after.st_size, - after.st_mtime_ns, - after.st_ctime_ns, - after.st_mode, - expected_sha, - CODEX_CLI_VERSION, - ) - if after_identity != identity: - raise CodexHarnessError("pinned Codex binary changed during attestation") - _RUNTIME_ATTESTATION_CACHE.clear() - _RUNTIME_ATTESTATION_CACHE[identity] = binary - return binary - - -def _canonical_history_item(item: Any) -> dict[str, Any]: - if not isinstance(item, dict) or not isinstance(item.get("type"), str): - raise CodexHarnessError("Codex supplied an invalid history item") - kind = item["type"] - if kind in {"compaction", "compaction_trigger", "context_compaction"}: - raise CodexHarnessError("Codex history compaction is forbidden") - if kind == "message": - role = item.get("role") - content = item.get("content") - if role not in {"developer", "user", "assistant"} or not isinstance(content, list): - raise CodexHarnessError("Codex supplied an invalid message history item") - normalized_content = [] - for part in content: - if not isinstance(part, dict) or part.get("type") not in { - "input_text", - "output_text", - }: - raise CodexHarnessError("Codex supplied non-text message history") - if not isinstance(part.get("text"), str): - raise CodexHarnessError("Codex supplied invalid message text") - normalized_content.append({"type": part["type"], "text": part["text"]}) - return {"type": "message", "role": role, "content": normalized_content} - if kind == "reasoning": - summary = item.get("summary", []) - content = item.get("content") - encrypted = item.get("encrypted_content") - if not isinstance(summary, list) or content is not None and not isinstance(content, list): - raise CodexHarnessError("Codex supplied invalid reasoning history") - normalized: dict[str, Any] = {"type": "reasoning", "summary": copy.deepcopy(summary)} - if content is not None: - normalized["content"] = copy.deepcopy(content) - if encrypted is not None: - if not isinstance(encrypted, str): - raise CodexHarnessError("Codex supplied invalid encrypted reasoning") - normalized["encrypted_content"] = encrypted - return normalized - if kind == "function_call": - name = item.get("name") - arguments = item.get("arguments") - call_id = item.get("call_id") - if ( - name not in {"terminal_exec", "submit"} - or not isinstance(arguments, str) - or not isinstance(call_id, str) - or not call_id - ): - raise CodexHarnessError("Codex supplied invalid function-call history") - return { - "type": "function_call", - "name": name, - "arguments": arguments, - "call_id": call_id, - } - if kind == "function_call_output": - call_id = item.get("call_id") - output = item.get("output") - if not isinstance(call_id, str) or not call_id or not isinstance(output, str): - raise CodexHarnessError("Codex supplied invalid function output history") - return {"type": kind, "call_id": call_id, "output": output} - raise CodexHarnessError(f"Codex supplied forbidden history type {kind!r}") - - -def _response_tool(tool: dict[str, Any]) -> dict[str, Any]: - parameters = copy.deepcopy(tool["inputSchema"]) - if tool["name"] == "terminal_exec": - # Stock 0.145.0's DynamicToolSpec conversion preserves the numeric type - # but intentionally drops JSON-Schema numeric bounds on the Responses wire. - timeout_schema = parameters["properties"]["timeout_seconds"] - timeout_schema.pop("minimum") - timeout_schema.pop("maximum") - return { - "type": "function", - "name": tool["name"], - "description": tool["description"], - "strict": False, - "parameters": parameters, - } - - -_EXPECTED_RESPONSE_TOOLS = [_response_tool(tool) for tool in DYNAMIC_TOOLS] - - -def _validate_codex_request(body: Any, expected_model: str) -> list[dict[str, Any]]: - if not isinstance(body, dict): - raise CodexHarnessError("Codex Responses request is not an object") - if body.get("model") != expected_model or body.get("instructions") != base_instructions(): - raise CodexHarnessError("Codex model or base instructions drifted") - if body.get("stream") is not True or body.get("store") is not False: - raise CodexHarnessError("Codex must use one non-persistent stream") - if body.get("parallel_tool_calls") is not False or body.get("tool_choice") != "auto": - raise CodexHarnessError("Codex sampling/tool-call policy drifted") - reasoning = body.get("reasoning") - if not isinstance(reasoning, dict) or reasoning.get("effort") != "xhigh": - raise CodexHarnessError("Codex did not request xhigh reasoning") - if reasoning.get("summary") not in {None, "none"}: - raise CodexHarnessError("Codex reasoning summary policy drifted") - if body.get("previous_response_id") is not None: - raise CodexHarnessError("server-side Responses history is forbidden") - if body.get("truncation") not in {None, "disabled"}: - raise CodexHarnessError("Codex Responses truncation is forbidden") - tools = body.get("tools") - if not isinstance(tools, list) or [tool.get("name") for tool in tools if isinstance(tool, dict)] != [ - "update_plan", - "terminal_exec", - "submit", - ]: - raise CodexHarnessError("Codex-side tool surface drifted") - if tools[1:] != [_response_tool(tool) for tool in dynamic_tools()]: - raise CodexHarnessError("Codex dynamic tool schemas drifted") - update_plan = tools[0] - if update_plan.get("type") != "function" or not isinstance( - update_plan.get("parameters"), dict - ): - raise CodexHarnessError("stock update_plan schema is invalid") - values = body.get("input") - if not isinstance(values, list) or not values: - raise CodexHarnessError("Codex supplied no explicit history") - return [_canonical_history_item(item) for item in values] - - -def _message_text(item: dict[str, Any]) -> str: - return "\n".join(part["text"] for part in item["content"]) - - -def _usage(completion: dict[str, Any]) -> tuple[dict[str, Any], int | None]: - usage = completion.get("usage") - if not isinstance(usage, dict): - return { - "input_tokens": 0, - "input_tokens_details": {"cached_tokens": 0}, - "output_tokens": 0, - "output_tokens_details": {"reasoning_tokens": 0}, - "total_tokens": 0, - }, None - prompt = usage.get("prompt_tokens", usage.get("input_tokens", 0)) - output = usage.get("completion_tokens", usage.get("output_tokens", 0)) - total = usage.get("total_tokens") - for value in (prompt, output): - if isinstance(value, bool) or not isinstance(value, int) or value < 0: - raise CodexHarnessError("Miles returned invalid token usage") - if total is not None and ( - isinstance(total, bool) or not isinstance(total, int) or total < 0 - ): - raise CodexHarnessError("Miles returned invalid token usage") - return { - "input_tokens": prompt, - "input_tokens_details": {"cached_tokens": 0}, - "output_tokens": output, - "output_tokens_details": {"reasoning_tokens": output}, - "total_tokens": total, - }, total - - -async def _read_bounded_miles_response(response: Any) -> bytes: - """Read one Miles response without ever buffering past the signed cap.""" - - content_length = response.content_length - if content_length is not None and content_length > MAX_MILES_RESPONSE_BYTES: - raise CodexHarnessError("Miles session response is oversized") - chunks: list[bytes] = [] - size = 0 - async for chunk in response.content.iter_chunked(_MILES_RESPONSE_READ_CHUNK_BYTES): - if len(chunk) > MAX_MILES_RESPONSE_BYTES - size: - raise CodexHarnessError("Miles session response is oversized") - chunks.append(chunk) - size += len(chunk) - return b"".join(chunks) - - -def _normalize_miles_stream_chunk(value: Any) -> dict[str, Any]: - """Normalize pinned Miles' one fake-stream chunk to a completion.""" - - if not isinstance(value, dict) or set(value) != { - "id", - "object", - "created", - "model", - "choices", - "usage", - }: - raise CodexHarnessError("Miles session returned an invalid SSE chunk") - identifier = value.get("id") - created = value.get("created") - if ( - not isinstance(identifier, str) - or not identifier - or value.get("object") != "chat.completion.chunk" - or value.get("model") != BACKEND_MODEL - or isinstance(created, bool) - or not isinstance(created, int) - or created < 0 - or not isinstance(value.get("usage"), dict) - ): - raise CodexHarnessError("Miles session returned an invalid SSE chunk") - choices = value.get("choices") - if not isinstance(choices, list) or len(choices) != 1: - raise CodexHarnessError("Miles session returned an invalid SSE chunk") - choice = choices[0] - if not isinstance(choice, dict) or set(choice) != { - "index", - "delta", - "finish_reason", - }: - raise CodexHarnessError("Miles session returned an invalid SSE chunk") - choice_index = choice.get("index") - finish_reason = choice.get("finish_reason") - if ( - isinstance(choice_index, bool) - or not isinstance(choice_index, int) - or choice_index != 0 - or finish_reason - not in { - "stop", - "length", - "tool_calls", - "content_filter", - "function_call", - "abort", - } - ): - raise CodexHarnessError("Miles session returned an invalid SSE chunk") - delta = choice.get("delta") - if ( - not isinstance(delta, dict) - or not {"role", "content"}.issubset(delta) - or not set(delta).issubset( - {"role", "content", "reasoning_content", "tool_calls"} - ) - or delta.get("role") != "assistant" - or not isinstance(delta.get("content"), str) - ): - raise CodexHarnessError("Miles session returned an invalid SSE delta") - reasoning = delta.get("reasoning_content") - if reasoning is not None and not isinstance(reasoning, str): - raise CodexHarnessError("Miles session returned an invalid SSE delta") - raw_calls = delta.get("tool_calls") - calls = None - if raw_calls is not None: - if not isinstance(raw_calls, list): - raise CodexHarnessError("Miles session returned an invalid SSE delta") - calls = [] - for index, raw_call in enumerate(raw_calls): - call_index = raw_call.get("index") if isinstance(raw_call, dict) else None - if ( - not isinstance(raw_call, dict) - or set(raw_call) != {"id", "index", "type", "function"} - or isinstance(call_index, bool) - or not isinstance(call_index, int) - or call_index != index - or raw_call.get("type") != "function" - or not isinstance(raw_call.get("id"), str) - or not raw_call["id"] - ): - raise CodexHarnessError("Miles session returned an invalid SSE delta") - function = raw_call.get("function") - if ( - not isinstance(function, dict) - or set(function) != {"name", "arguments"} - or not isinstance(function.get("name"), str) - or not isinstance(function.get("arguments"), str) - ): - raise CodexHarnessError("Miles session returned an invalid SSE delta") - calls.append( - { - "id": raw_call["id"], - "type": "function", - "function": copy.deepcopy(function), - } - ) - message: dict[str, Any] = { - "role": "assistant", - "content": delta["content"], - } - if reasoning is not None: - message["reasoning_content"] = reasoning - if calls is not None: - message["tool_calls"] = calls - return { - "id": identifier, - "object": "chat.completion", - "created": created, - "model": BACKEND_MODEL, - "choices": [ - { - "index": 0, - "message": message, - "finish_reason": finish_reason, - } - ], - "usage": copy.deepcopy(value["usage"]), - } - - -def _parse_miles_sse(raw: bytes) -> dict[str, Any]: - """Parse exactly one pinned Miles data event followed by ``[DONE]``.""" - - def object_without_duplicates(pairs: list[tuple[str, Any]]) -> dict[str, Any]: - value = {} - for key, item in pairs: - if key in value: - raise ValueError("duplicate JSON object key") - value[key] = item - return value - - def reject_nonfinite_constant(_value: str) -> None: - raise ValueError("non-finite JSON number") - - try: - text = raw.decode("utf-8") - except UnicodeDecodeError as exc: - raise CodexHarnessError("Miles session returned invalid UTF-8") from exc - blocks = text.split("\n\n") - if len(blocks) != 3 or blocks[-1] or any("\n" in block for block in blocks[:-1]): - raise CodexHarnessError("Miles session returned malformed SSE") - if not all(block.startswith("data: ") for block in blocks[:-1]): - raise CodexHarnessError("Miles session returned malformed SSE") - payload, done = (block[len("data: ") :] for block in blocks[:-1]) - if done != "[DONE]": - raise CodexHarnessError("Miles session returned malformed SSE") - try: - value = json.loads( - payload, - object_pairs_hook=object_without_duplicates, - parse_constant=reject_nonfinite_constant, - ) - except (json.JSONDecodeError, ValueError) as exc: - raise CodexHarnessError("Miles session returned invalid SSE JSON") from exc - return _normalize_miles_stream_chunk(value) - - -def _sse_response( - *, - request: dict[str, Any], - output: list[dict[str, Any]], - usage: dict[str, Any], - response_number: int, -) -> bytes: - response_id = f"resp_yeto_{response_number}" - completed = { - "id": response_id, - "object": "response", - "created_at": int(time.time()), - "status": "completed", - "error": None, - "incomplete_details": None, - "instructions": base_instructions(), - "max_output_tokens": None, - "max_tool_calls": None, - "model": request["model"], - "output": output, - "parallel_tool_calls": False, - "previous_response_id": None, - "prompt_cache_key": request.get("prompt_cache_key"), - "reasoning": {"effort": "xhigh", "summary": "none"}, - "safety_identifier": None, - "service_tier": "default", - "store": False, - "temperature": None, - "text": {"format": {"type": "text"}}, - "tool_choice": "auto", - "tools": request["tools"], - "top_logprobs": 0, - "top_p": None, - "truncation": "disabled", - "usage": usage, - "user": None, - "metadata": {}, - } - events: list[dict[str, Any]] = [ - { - "type": "response.created", - "response": {**completed, "status": "in_progress", "output": []}, - } - ] - for index, item in enumerate(output): - added = copy.deepcopy(item) - added["status"] = "in_progress" - if item["type"] == "function_call": - added["arguments"] = "" - events.append( - {"type": "response.output_item.added", "output_index": index, "item": added} - ) - if item["type"] == "function_call": - events.extend( - [ - { - "type": "response.function_call_arguments.delta", - "item_id": item["id"], - "output_index": index, - "delta": item["arguments"], - }, - { - "type": "response.function_call_arguments.done", - "item_id": item["id"], - "output_index": index, - "arguments": item["arguments"], - }, - ] - ) - events.append( - {"type": "response.output_item.done", "output_index": index, "item": item} - ) - events.append({"type": "response.completed", "response": completed}) - chunks = [] - for event in events: - chunks.append(f"event: {event['type']}\n".encode()) - chunks.append(b"data: " + _canonical_json(event) + b"\n\n") - chunks.append(b"data: [DONE]\n\n") - return b"".join(chunks) - - -@dataclass -class _PendingTool: - call_id: str - codex_name: str - miles_name: str - raw_arguments: str - - -def _tool_error(message: str) -> dict[str, str]: - compact = message if len(message) <= 1024 else message[:1000] + "...[truncated]" - return {"error": compact} - - -# G6(a): bridge rejections that are, in gateway terms, a session mismatch -# (history is not the expected prefix) or a chain break (reason names from -# ``yeto.rl.engine.timeline.TITO_CHAIN_BREAK_REASONS``). -TITO_SESSION_MISMATCH_MESSAGES = frozenset( - { - "Codex mutated the initial task history", - "Codex truncated or mutated episode history", - "Codex supplied no explicit history", - } -) -TITO_SESSION_MISMATCH_KEY = "tito_session_mismatch" -TITO_CHAIN_BREAKS_KEY = "tito_chain_breaks" -TITO_CHAIN_BREAK_MESSAGES = { - "Codex retried a Responses sample": "retry_fork", - "Codex history compaction is forbidden": "compaction_window", -} - - -def tito_counters(metrics: Any) -> dict[str, Any]: - """The G6(a) counters recorded on a metrics object (zero when none).""" - return { - TITO_SESSION_MISMATCH_KEY: int(getattr(metrics, TITO_SESSION_MISMATCH_KEY, 0) or 0), - TITO_CHAIN_BREAKS_KEY: dict(getattr(metrics, TITO_CHAIN_BREAKS_KEY, {}) or {}), - } - - -class _ResponsesBridge: - def __init__( - self, - miles_base_url: str, - prompt: str, - request_kwargs: dict[str, Any], - metrics: legacy.AgentMetrics, - *, - max_seq_len: int | None, - ) -> None: - self._miles_url = ( - f"{legacy._session_url(miles_base_url).rstrip('/')}/chat/completions" - ) - self._prompt = prompt - self._request_kwargs = dict(request_kwargs) - self._metrics = metrics - self._max_seq_len = max_seq_len - self._model = BACKEND_MODEL - self._max_turns = legacy._positive_int_env("SECRLENV_MAX_TURNS", 40) - self._backend_max_tokens = _positive_int("YETO_CODEX_BACKEND_MAX_TOKENS") - unexpected_sampling = set(self._request_kwargs) - ( - _SAMPLING_FIELDS | _STRUCTURAL_REQUEST_FIELDS - ) - if unexpected_sampling: - raise CodexHarnessError("Miles supplied an unknown sampling field") - forbidden_structural = set(self._request_kwargs) & ( - _STRUCTURAL_REQUEST_FIELDS - - {"best_of", "max_tokens", "n", "stream", "stream_options"} - ) - if forbidden_structural: - raise CodexHarnessError("Miles attempted to override a signed request field") - requested_max_tokens = self._request_kwargs.get( - "max_tokens", self._backend_max_tokens - ) - if ( - isinstance(requested_max_tokens, bool) - or not isinstance(requested_max_tokens, int) - or not 0 < requested_max_tokens <= self._backend_max_tokens - ): - raise CodexHarnessError( - "Miles sampling max_tokens exceeds the signed backend budget" - ) - self._requested_max_tokens = requested_max_tokens - for key in ("n", "best_of"): - if self._request_kwargs.get(key) not in {None, 1}: - raise CodexHarnessError("Miles requested more than one sample") - self._compaction_enabled = _strict_env_flag("YETO_CODEX_COMPACTION_ENABLED") - self._compaction_trigger_tokens = legacy._positive_int_env( - "YETO_CODEX_COMPACTION_TRIGGER_TOKENS", 6_144 - ) - self._compaction_summary_max_tokens = legacy._positive_int_env( - "YETO_CODEX_COMPACTION_SUMMARY_MAX_TOKENS", 1_024 - ) - self._max_compactions = legacy._positive_int_env("YETO_CODEX_MAX_COMPACTIONS", 3) - if self._compaction_enabled: - if self._max_seq_len is None: - raise CodexHarnessError("Codex compaction requires max_seq_len") - if self._compaction_trigger_tokens >= self._max_seq_len: - raise CodexHarnessError( - "Codex compaction trigger must be smaller than max_seq_len" - ) - summary_generation_tokens = min( - self._requested_max_tokens, - self._compaction_summary_max_tokens, - ) - if ( - self._compaction_trigger_tokens - + COMPACTION_SUMMARY_INSTRUCTION_TOKEN_RESERVE - + summary_generation_tokens - > self._max_seq_len - ): - raise CodexHarnessError( - "Codex compaction trigger leaves insufficient room for the " - "summary instruction and generation" - ) - self._token = secrets.token_urlsafe(32) - self._seen_bodies: set[str] = set() - self._expected_input: list[dict[str, Any]] | None = None - self._history_after_model: list[dict[str, Any]] | None = None - self._pending: _PendingTool | None = None - self._messages: list[dict[str, Any]] = [] - self._atomic_steps: list[tuple[dict[str, Any], dict[str, Any]]] = [] - self._last_assistant_message: dict[str, Any] | None = None - self._context_window = 0 - self._segment_index = 0 - self._segment_type = "execution" - self._compaction_count = 0 - self._current_context_tokens = 0 - self._resume_summary: str | None = None - self._retained_step_count = COMPACTION_KEEP_ATOMIC_STEPS - self._unobserved_tool_token_upper_bound = 0 - self._request_count = 0 - self._terminal = False - self._runner: web.AppRunner | None = None - self._session: aiohttp.ClientSession | None = None - self._lock = asyncio.Lock() - self._fatal: asyncio.Future[BaseException] | None = None - self.url: str | None = None - self._gate = SuspendGate(os.environ.get(SUSPEND_GATE_ENV), metrics) - - @property - def token(self) -> str: - return self._token - - @property - def fatal(self) -> asyncio.Future[BaseException]: - if self._fatal is None: - raise RuntimeError("bridge is not started") - return self._fatal - - async def __aenter__(self) -> Self: - loop = asyncio.get_running_loop() - self._fatal = loop.create_future() - self._session = aiohttp.ClientSession( - timeout=aiohttp.ClientTimeout(total=legacy._positive_env( - "SECRLENV_MAX_ROLLOUT_TIME_SECONDS", 3600.0 - )), - connector=aiohttp.TCPConnector(limit=1), - ) - app = web.Application(client_max_size=2 * 1024 * 1024) - app.router.add_post("/v1/responses", self._handle_responses) - app.router.add_route("*", "/{tail:.*}", self._reject_route) - self._runner = web.AppRunner(app, access_log=None) - await self._runner.setup() - site = web.TCPSite(self._runner, "127.0.0.1", 0) - await site.start() - sockets = site._server.sockets if site._server is not None else [] - if len(sockets) != 1: - raise CodexHarnessError("Responses bridge did not bind one loopback socket") - host, port = sockets[0].getsockname()[:2] - if host != "127.0.0.1": - raise CodexHarnessError("Responses bridge escaped loopback") - self.url = f"http://127.0.0.1:{port}" - return self - - async def __aexit__(self, *_exc: object) -> None: - if self._session is not None: - await self._session.close() - if self._runner is not None: - await self._runner.cleanup() - - def _fail(self, exc: BaseException) -> None: - if self._fatal is not None and not self._fatal.done(): - self._fatal.set_result(exc) - - async def _reject_route(self, _request: web.Request) -> web.Response: - self._fail(CodexHarnessError("Codex called an unexpected bridge route")) - return web.json_response({"error": "unsupported route"}, status=404) - - def expect_tool_output(self, call_id: str, output: str) -> None: - if self._terminal or self._pending is None or self._history_after_model is None: - raise CodexHarnessError("Codex tool output arrived outside a model call") - if not hmac.compare_digest(self._pending.call_id, call_id): - raise CodexHarnessError("Codex tool output call ID drifted") - self._expected_input = self._history_after_model + [ - {"type": "function_call_output", "call_id": call_id, "output": output} - ] - self._messages.append( - {"role": "tool", "tool_call_id": call_id, "content": output} - ) - if self._last_assistant_message is None: - raise CodexHarnessError("Codex lost the assistant half of an atomic tool step") - self._atomic_steps.append( - ( - copy.deepcopy(self._last_assistant_message), - copy.deepcopy(self._messages[-1]), - ) - ) - self._last_assistant_message = None - self._unobserved_tool_token_upper_bound += len(output.encode("utf-8")) + 256 - _note_turn_tool_output(self._metrics, output) - self._pending = None - - def mark_terminal(self) -> None: - self._terminal = True - - def take_pending(self) -> _PendingTool: - if self._pending is None: - raise CodexHarnessError("Codex requested a tool without a model call") - return self._pending - - async def _handle_responses(self, request: web.Request) -> web.Response: - authorization = request.headers.get("Authorization", "") - if not hmac.compare_digest(authorization, f"Bearer {self._token}"): - self._fail(CodexHarnessError("Codex bridge authentication failed")) - return web.json_response({"error": "unauthorized"}, status=401) - async with self._lock: - if self._terminal: - self._fail(CodexHarnessError("Codex sampled after terminal submit")) - return web.json_response({"error": "episode is terminal"}, status=409) - try: - body = await request.json() - fingerprint = hashlib.sha256(_canonical_json(body)).hexdigest() - if fingerprint in self._seen_bodies: - raise CodexHarnessError("Codex retried a Responses sample") - self._seen_bodies.add(fingerprint) - history = _validate_codex_request(body, self._model) - if self._request_count >= self._max_turns: - raise CodexTurnLimit("Codex reached the signed turn budget") - if self._expected_input is None: - expected = [ - { - "type": "message", - "role": "user", - "content": [{"type": "input_text", "text": self._prompt}], - } - ] - if history != expected: - raise CodexHarnessError("Codex mutated the initial task history") - self._messages = [ - {"role": "system", "content": base_instructions()}, - {"role": "user", "content": self._prompt}, - ] - elif history != self._expected_input: - raise CodexHarnessError("Codex truncated or mutated episode history") - estimated_context_tokens = ( - self._current_context_tokens - + self._unobserved_tool_token_upper_bound - ) - if self._should_compact(estimated_context_tokens): - await self._compact_context() - estimated_context_tokens = 0 - if self._max_seq_len is not None and estimated_context_tokens >= self._max_seq_len: - self._metrics.max_seq_len_hit = 1 - raise CodexSequenceLimit("Miles sequence limit reached") - completion = await self._sample_execution_with_tail_fallback() - self._request_count += 1 - self._metrics.turns = self._request_count - usage, total = _usage(completion) - if total is None: - if self._max_seq_len is not None: - self._metrics.usage_missing += 1 - raise CodexHarnessError("Miles omitted required token usage") - else: - self._metrics.max_model_total_tokens = max( - self._metrics.max_model_total_tokens, total - ) - self._current_context_tokens = total - self._unobserved_tool_token_upper_bound = 0 - output = self._translate_completion(completion) - self._history_after_model = history + [ - _canonical_history_item(item) for item in output - ] - return web.Response( - body=_sse_response( - request=body, - output=output, - usage=usage, - response_number=self._request_count, - ), - headers={ - "Content-Type": "text/event-stream", - "Cache-Control": "no-cache", - "X-Accel-Buffering": "no", - }, - ) - except (CodexSequenceLimit, CodexTurnLimit, CodexModelFailure) as exc: - if isinstance(exc, CodexSequenceLimit): - self._metrics.max_seq_len_hit = 1 - elif isinstance(exc, CodexModelFailure): - self._metrics.parse_failures += 1 - self._fail(exc) - return web.json_response({"error": "episode boundary"}, status=400) - except Exception as exc: # noqa: BLE001 - sanitize all parser failures - failure = exc if isinstance(exc, CodexHarnessError) else CodexHarnessError( - "Codex Responses bridge rejected the request" - ) - self._note_tito_rejection(failure) - self._fail(failure) - return web.json_response({"error": "protocol violation"}, status=400) - - def _note_tito_rejection(self, failure: CodexHarnessError) -> None: - """G6(a): count a rejection the way the gateway would (9.2 judgement). - - The bridge never forks a chain: a request whose history is not the - exact expected prefix is a session mismatch (``tito_session_mismatch``); - a byte-identical retry is a ``retry_fork`` chain break and a compaction - attempt a ``compaction_window`` one. The trusted layer mirrors these - onto the HarnessBoard (``record_session_mismatch`` / ``record_chain_break``). - """ - # ``AgentMetrics`` is sha-pinned legacy source, so the counters ride on - # the instance as extra attributes (read back by ``tito_counters``). - message = str(failure) - if message in TITO_SESSION_MISMATCH_MESSAGES: - setattr(self._metrics, TITO_SESSION_MISMATCH_KEY, getattr(self._metrics, TITO_SESSION_MISMATCH_KEY, 0) + 1) - reason = TITO_CHAIN_BREAK_MESSAGES.get(message) - if reason is not None: - breaks = dict(getattr(self._metrics, TITO_CHAIN_BREAKS_KEY, {}) or {}) - breaks[reason] = breaks.get(reason, 0) + 1 - setattr(self._metrics, TITO_CHAIN_BREAKS_KEY, breaks) - - def _compaction_headers(self) -> dict[str, str]: - if not self._compaction_enabled or self._max_seq_len is None: - return {} - return { - "X-Miles-Compaction-Schema-Version": str(COMPACTION_SCHEMA_VERSION), - "X-Miles-Compaction-Context-Window": str(self._context_window), - "X-Miles-Compaction-Segment-Index": str(self._segment_index), - "X-Miles-Compaction-Segment-Type": self._segment_type, - "X-Miles-Compaction-Context-Budget": str(self._max_seq_len), - } - - def _should_compact(self, estimated_context_tokens: int) -> bool: - """Return whether consumed context reached the configured boundary.""" - return ( - self._compaction_enabled - and self._compaction_count < self._max_compactions - and self._max_seq_len is not None - and estimated_context_tokens >= self._compaction_trigger_tokens - ) - - def _rebuild_compacted_messages(self, retained_step_count: int) -> None: - if self._resume_summary is None: - raise CodexHarnessError("compaction resume summary is missing") - retained = self._atomic_steps[-retained_step_count:] if retained_step_count else [] - messages: list[dict[str, Any]] = [ - {"role": "system", "content": base_instructions()}, - { - "role": "user", - "content": COMPACTION_RESUME_TEMPLATE.format(summary=self._resume_summary), - }, - ] - for assistant, tool in retained: - messages.extend((copy.deepcopy(assistant), copy.deepcopy(tool))) - self._messages = messages - self._atomic_steps = copy.deepcopy(retained) - self._retained_step_count = retained_step_count - - async def _compact_context(self) -> None: - """Sample a trainable summary, then start one reconstructed window.""" - if self._pending is not None or self._last_assistant_message is not None: - raise CodexHarnessError("compaction attempted inside an atomic tool step") - if not self._atomic_steps: - raise CodexHarnessError("compaction attempted without a complete tool step") - if self._max_seq_len is None: - raise CodexHarnessError("compaction requires a context budget") - summary_generation_tokens = min( - self._requested_max_tokens, - self._compaction_summary_max_tokens, - ) - if ( - self._current_context_tokens - + self._unobserved_tool_token_upper_bound - + COMPACTION_SUMMARY_INSTRUCTION_TOKEN_RESERVE - + summary_generation_tokens - > self._max_seq_len - ): - raise CodexSequenceLimit( - "compaction was triggered too late to reserve a complete summary" - ) - self._segment_index += 1 - self._segment_type = "summary" - summary_messages = copy.deepcopy(self._messages) - summary_messages.append( - {"role": "user", "content": COMPACTION_SUMMARY_INSTRUCTION} - ) - try: - completion = await self._sample_miles( - messages=summary_messages, - summary=True, - ) - except _CompactionContextDoesNotFit as exc: - raise CodexSequenceLimit( - "compaction summary instruction no longer fits the context budget" - ) from exc - usage, total = _usage(completion) - del usage - if total is None: - self._metrics.usage_missing += 1 - raise CodexHarnessError("Miles omitted summary token usage") - self._metrics.max_model_total_tokens = max( - self._metrics.max_model_total_tokens, total - ) - self._current_context_tokens = total - self._unobserved_tool_token_upper_bound = 0 - summary = self._translate_summary(completion) - - self._resume_summary = summary - self._context_window += 1 - self._segment_index += 1 - self._segment_type = "execution" - self._compaction_count += 1 - self._current_context_tokens = 0 - self._rebuild_compacted_messages( - min(COMPACTION_KEEP_ATOMIC_STEPS, len(self._atomic_steps)) - ) - - async def _sample_execution_with_tail_fallback(self) -> dict[str, Any]: - """Reduce the exact recent-step tail only after Miles proves it cannot fit.""" - while True: - try: - return await self._sample_miles() - except _CompactionContextDoesNotFit as exc: - if self._resume_summary is None or self._retained_step_count == 0: - raise CodexSequenceLimit( - "compacted resume context does not fit the context budget" - ) from exc - self._rebuild_compacted_messages(self._retained_step_count - 1) - - async def _sample_miles( - self, - *, - messages: list[dict[str, Any]] | None = None, - summary: bool = False, - ) -> dict[str, Any]: - if self._session is None: - raise RuntimeError("bridge is not started") - sampling = { - key: copy.deepcopy(value) - for key, value in self._request_kwargs.items() - if key in _SAMPLING_FIELDS - } - max_tokens = self._requested_max_tokens - if self._max_seq_len is not None: - remaining = ( - self._max_seq_len - - self._current_context_tokens - - self._unobserved_tool_token_upper_bound - ) - if summary: - remaining -= COMPACTION_SUMMARY_INSTRUCTION_TOKEN_RESERVE - if remaining <= 0: - raise CodexSequenceLimit("Miles sequence limit reached") - max_tokens = min(max_tokens, remaining) - if summary: - max_tokens = min(max_tokens, self._compaction_summary_max_tokens) - payload = { - **sampling, - "model": self._model, - "messages": copy.deepcopy(self._messages if messages is None else messages), - "parallel_tool_calls": False, - # Pinned Miles fake-streams one compact client event while retaining - # the full token/logprob response in its session record for TITO. - "stream": True, - "max_tokens": max_tokens, - "reasoning_effort": BACKEND_REASONING_EFFORT, - "thinking": {"type": "enabled"}, - "chat_template_kwargs": copy.deepcopy(BACKEND_CHAT_TEMPLATE_KWARGS), - } - if summary: - payload["tools"] = [] - payload["tool_choice"] = "none" - else: - payload["tools"] = copy.deepcopy(miles_tools()) - payload["tool_choice"] = "auto" - started = time.monotonic() - suspended_before = self._metrics.suspended_seconds - try: - while True: - # 5.1: a suspended trajectory waits here, between two model turns. - await self._gate.wait_open() - async with self._session.post( - self._miles_url, - json=payload, - headers=self._compaction_headers(), - ) as response: - status = response.status - if status == 503 and self._gate.closed(): - # The cut-off aborted this model turn in the engine (the gate - # closed first); the session server did not record it. Redo - # the same turn once the trajectory is resumed. - await _read_bounded_miles_response(response) - self._metrics.suspend_retried_turns += 1 - continue - if status == 413: - await _read_bounded_miles_response(response) - raise _CompactionContextDoesNotFit( - "Miles proved the compacted context exceeds its budget" - ) - if status != 200: - snippet = "" - if status != 503: - # bounded diagnostic read (503 keeps the retry-path - # no-read contract pinned by the sec harness tests) - detail = await _read_bounded_miles_response(response) - snippet = detail[:500].decode("utf-8", errors="replace") if isinstance(detail, bytes) else str(detail)[:500] - raise CodexHarnessError( - f"Miles session returned HTTP {status}" + (f": {snippet}" if snippet else "")) - if response.content_type != "text/event-stream": - raise CodexHarnessError( - "Miles session returned an unexpected media type" - ) - raw = await _read_bounded_miles_response(response) - break - except (aiohttp.ClientError, TimeoutError) as exc: - raise CodexHarnessError("Miles session transport failed") from exc - finally: - # time parked at the gate is not generation time - self._metrics.total_generation_time += ( - time.monotonic() - started - (self._metrics.suspended_seconds - suspended_before)) - completion = _parse_miles_sse(raw) - if not summary and self._resume_summary is not None: - # The session server tokenized this exact candidate before sampling. - # A 413 above is the only allowed reason to reduce k. - self._resume_summary = None - return completion - - def _translate_summary(self, completion: dict[str, Any]) -> str: - choices = completion.get("choices") - if not isinstance(choices, list) or len(choices) != 1: - raise CodexHarnessError("Miles returned extra or missing summary samples") - choice = choices[0] - if not isinstance(choice, dict) or choice.get("finish_reason") != "stop": - raise CodexModelFailure("compaction summary did not finish cleanly") - message = choice.get("message") - if not isinstance(message, dict) or message.get("role") != "assistant": - raise CodexModelFailure("compaction summary is not an assistant message") - content = message.get("content") - calls = message.get("tool_calls") - if ( - not isinstance(content, str) - or not content.strip() - or (calls is not None and calls != []) - ): - raise CodexModelFailure("compaction summary must be non-empty and tool-free") - return content.strip() - - def _translate_completion(self, completion: dict[str, Any]) -> list[dict[str, Any]]: - choices = completion.get("choices") - if not isinstance(choices, list) or len(choices) != 1: - raise CodexHarnessError("Miles returned extra or missing samples") - choice = choices[0] - if not isinstance(choice, dict): - raise CodexHarnessError("Miles returned an invalid sample") - _note_last_completion(self._metrics, choice, completion) - if choice.get("finish_reason") == "length": - raise CodexSequenceLimit("Miles returned a truncated sample") - message = choice.get("message") - if not isinstance(message, dict) or message.get("role") != "assistant": - raise CodexHarnessError("Miles returned an invalid assistant message") - content = message.get("content", "") - reasoning = message.get("reasoning_content") - calls = message.get("tool_calls") - if not isinstance(content, str) or not isinstance(reasoning, str) or not reasoning: - raise CodexModelFailure("DSV4 did not return thinking content") - if content.strip() and not ALLOW_PROSE_WITH_TOOL_CALL: - raise CodexModelFailure("DSV4 mixed prose with the required tool call") - if not isinstance(calls, list) or len(calls) != 1: - raise CodexModelFailure("DSV4 must return exactly one tool call") - call = calls[0] - function = call.get("function") if isinstance(call, dict) else None - call_id = call.get("id") if isinstance(call, dict) else None - if not isinstance(function, dict) or not isinstance(call_id, str) or not call_id: - raise CodexModelFailure("DSV4 returned an invalid tool call") - name = function.get("name") - raw_arguments = function.get("arguments") - if name not in {"terminal.exec", "submit"} or not isinstance(raw_arguments, str): - raise CodexModelFailure("DSV4 returned a forbidden tool") - try: - arguments = json.loads(raw_arguments) - except json.JSONDecodeError as exc: - raise CodexModelFailure("DSV4 returned malformed raw tool arguments") from exc - if not isinstance(arguments, dict): - raise CodexModelFailure("DSV4 tool arguments are not an object") - codex_name = "terminal_exec" if name == "terminal.exec" else "submit" - if self._pending is not None: - raise CodexHarnessError("DSV4 emitted a tool before prior output") - self._pending = _PendingTool(call_id, codex_name, name, raw_arguments) - self._metrics.tool_calls += 1 - self._messages.append(copy.deepcopy(message)) - self._last_assistant_message = copy.deepcopy(message) - index = self._request_count + 1 - return [ - { - "id": f"rs_yeto_{index}", - "type": "reasoning", - "summary": [], - "content": [{"type": "reasoning_text", "text": reasoning}], - "encrypted_content": None, - "status": "completed", - }, - { - "id": f"fc_yeto_{index}", - "type": "function_call", - "status": "completed", - "name": codex_name, - "call_id": call_id, - "arguments": raw_arguments, - }, - ] - - -LAST_COMPLETION_HEAD_CHARS = 200 - -# S17 C9 (observe only): why a Codex episode ended, finer than the signed -# ``exit_status`` (which folds protocol errors into ``max_turns`` and reply -# truncation into ``max_seq_len``). -END_KIND_SUBMIT = "submit" # episode ended after a submit call -END_KIND_COMPLETED = "completed_no_submit" # Codex stopped on its own, no submit -END_KIND_TURN_LIMIT = "turn_limit" # signed turn budget reached -END_KIND_PROTOCOL_ERROR = "protocol_error" # model reply violated the tool protocol -END_KIND_RESPONSE_TRUNCATED = "response_truncated" # one reply hit the per-reply cap -END_KIND_CONTEXT_LIMIT = "context_limit" # whole trajectory hit the sequence cap -END_KIND_TIMEOUT = "timeout" -END_KINDS = (END_KIND_SUBMIT, END_KIND_COMPLETED, END_KIND_TURN_LIMIT, END_KIND_PROTOCOL_ERROR, - END_KIND_RESPONSE_TRUNCATED, END_KIND_CONTEXT_LIMIT, END_KIND_TIMEOUT) -_TRUNCATED_SAMPLE_MESSAGE = "Miles returned a truncated sample" -# Per-turn length lists are capped so one runaway episode cannot bloat events. -TURN_LENGTHS_MAX = 256 - - -def end_kind_for(exc: BaseException) -> str: - if isinstance(exc, CodexSequenceLimit): - return END_KIND_RESPONSE_TRUNCATED if str(exc) == _TRUNCATED_SAMPLE_MESSAGE else END_KIND_CONTEXT_LIMIT - if isinstance(exc, CodexTurnLimit): - return END_KIND_TURN_LIMIT - if isinstance(exc, CodexModelFailure): - return END_KIND_PROTOCOL_ERROR - raise TypeError(f"not a policy boundary: {type(exc).__name__}") - - -def _append_capped(metrics: Any, name: str, value: Any) -> None: - values = getattr(metrics, name, None) - if not isinstance(values, list): - values = [] - setattr(metrics, name, values) - if len(values) < TURN_LENGTHS_MAX: - values.append(value) - - -def _note_turn_tool_output(metrics: Any, output: str) -> None: - """Observe only (S17 C9): bytes of each tool result fed back to the model.""" - _append_capped(metrics, "turn_tool_output_bytes", len(output.encode("utf-8"))) - - -def _note_last_completion(metrics: Any, choice: dict[str, Any], completion: dict[str, Any]) -> None: - """Observe only (S15 r2 follow-up): shape of the latest model reply, so a - tape can tell thinking exhaustion from prose / missing tool calls.""" - message = choice.get("message") if isinstance(choice.get("message"), dict) else {} - content = message.get("content") if isinstance(message.get("content"), str) else "" - reasoning = message.get("reasoning_content") if isinstance(message.get("reasoning_content"), str) else "" - calls = message.get("tool_calls") if isinstance(message.get("tool_calls"), list) else [] - usage = completion.get("usage") if isinstance(completion.get("usage"), dict) else {} - names = [] - for call in calls: - function = call.get("function") if isinstance(call, dict) else None - names.append(str(function.get("name"))[:32] if isinstance(function, dict) else "?") - # S17 C9: one entry per model reply (None when the backend sent no usage). - tokens = usage.get("completion_tokens") - _append_capped(metrics, "turn_completion_tokens", - tokens if isinstance(tokens, int) and not isinstance(tokens, bool) else None) - total = usage.get("total_tokens") - _append_capped(metrics, "turn_context_tokens", - total if isinstance(total, int) and not isinstance(total, bool) else None) - metrics.last_completion = { # type: ignore[attr-defined] - "finish_reason": str(choice.get("finish_reason"))[:16], - "content_chars": len(content), - "reasoning_chars": len(reasoning), - "tool_calls": len(calls), - "tool_names": ",".join(names)[:96], - "completion_tokens": usage.get("completion_tokens") if isinstance(usage.get("completion_tokens"), int) else None, - "content_head": content[:LAST_COMPLETION_HEAD_CHARS], - "reasoning_tail": reasoning[-LAST_COMPLETION_HEAD_CHARS:], - } - - -def _toml_string(value: str) -> str: - return json.dumps(value, ensure_ascii=True) - - -def _suspend_settings() -> list[str]: - """5.1: a suspended trajectory's model request is held by the bridge for up to - the survival limit (plus a training round); Codex must not give up on it.""" - if not os.environ.get(SUSPEND_GATE_ENV): - return [] # default command line unchanged - idle_ms = int((suspend_max_seconds() + 600.0) * 1000) - return [f"model_providers.miles.stream_idle_timeout_ms={idle_ms}"] - - -def _codex_argv(binary: Path, bridge: _ResponsesBridge) -> list[str]: - if bridge.url is None: - raise RuntimeError("bridge is not started") - model = BACKEND_MODEL - settings = [ - f"model={_toml_string(model)}", - 'model_provider="miles"', - 'model_reasoning_effort="xhigh"', - 'model_reasoning_summary="none"', - 'model_providers.miles.name="Yeto Miles"', - f"model_providers.miles.base_url={_toml_string(bridge.url + '/v1')}", - 'model_providers.miles.env_key="YETO_CODEX_BRIDGE_TOKEN"', - 'model_providers.miles.wire_api="responses"', - "model_providers.miles.requires_openai_auth=false", - "model_providers.miles.request_max_retries=0", - "model_providers.miles.stream_max_retries=0", - *_suspend_settings(), - "features.shell_tool=false", - "features.unified_exec=false", - "features.multi_agent=false", - "features.multi_agent_v2=false", - "features.apps=false", - "features.plugins=false", - "features.browser_use=false", - "features.computer_use=false", - "features.image_generation=false", - "features.default_mode_request_user_input=false", - "features.skill_search=false", - "features.workspace_dependencies=false", - "agents.enabled=false", - "orchestrator.skills.enabled=false", - "orchestrator.mcp.enabled=false", - "tools.experimental_request_user_input.enabled=false", - 'web_search="disabled"', - "include_permissions_instructions=false", - "include_environment_context=false", - "include_apps_instructions=false", - "include_collaboration_mode_instructions=false", - "skills.include_instructions=false", - "analytics.enabled=false", - "check_for_update_on_startup=false", - "project_doc_max_bytes=0", - f"tool_output_token_limit={CODEX_TOOL_OUTPUT_TOKEN_LIMIT}", - ] - values = [str(binary), "app-server", "--stdio", "--strict-config"] - for setting in settings: - values.extend(("-c", setting)) - return values - - -class _AppServerDriver: - def __init__( - self, - binary: Path, - bridge: _ResponsesBridge, - client: EpisodeClient, - episode_id: str, - prompt: str, - metrics: legacy.AgentMetrics, - ) -> None: - self._binary = binary - self._bridge = bridge - self._client = client - self._episode_id = episode_id - self._prompt = prompt - self._metrics = metrics - self._process: asyncio.subprocess.Process | None = None - self._process_group_id: int | None = None - self._request_id = 0 - self._thread_id: str | None = None - self._turn_id: str | None = None - self._stderr_task: asyncio.Task[None] | None = None - - async def __aenter__(self) -> Self: - isolated_home = tempfile.TemporaryDirectory(prefix="yeto-codex-") - self._isolated_home = isolated_home - Path(isolated_home.name).chmod(0o700) - environment = { - "CODEX_HOME": isolated_home.name, - "PATH": "/usr/bin:/bin", - "LANG": "C.UTF-8", - "TZ": "UTC", - "YETO_CODEX_BRIDGE_TOKEN": self._bridge.token, - } - self._process = await asyncio.create_subprocess_exec( - *_codex_argv(self._binary, self._bridge), - stdin=asyncio.subprocess.PIPE, - stdout=asyncio.subprocess.PIPE, - stderr=asyncio.subprocess.PIPE, - env=environment, - limit=MAX_APP_SERVER_FRAME_BYTES + 1, - start_new_session=True, - ) - self._process_group_id = self._process.pid - self._stderr_task = asyncio.create_task(self._drain_stderr()) - return self - - async def __aexit__( - self, - exc_type: object, - _exc: object, - _traceback: object, - ) -> None: - try: - await self._stop() - except Exception: - if exc_type is None: - raise - LOGGER.warning( - "Codex app-server shutdown also failed while handling an episode error", - exc_info=True, - ) - finally: - await self._cleanup_isolated_home() - - async def _drain_stderr(self) -> None: - if self._process is None or self._process.stderr is None: - return - while await self._process.stderr.read(8192): - pass - - async def _stop(self) -> None: - process = self._process - if process is None: - return - - process_group_id = self._process_group_id - group_signalled = self._signal_process_group(signal.SIGTERM) - if process.returncode is None and not group_signalled: - process.terminate() - if process.returncode is None: - try: - await asyncio.wait_for( - process.wait(), timeout=_CODEX_PROCESS_STOP_TIMEOUT_SECONDS - ) - except asyncio.TimeoutError: - if not self._signal_process_group(signal.SIGKILL): - process.kill() - await process.wait() - - # The app-server may have spawned plugin/helper processes which inherited - # CODEX_HOME. Waiting for only the direct child does not reap them. Give - # the isolated process group a short TERM grace period, then make sure no - # descendant remains able to recreate files while the home is removed. - if process_group_id is not None: - await asyncio.sleep(_CODEX_DESCENDANT_TERM_GRACE_SECONDS) - self._signal_process_group(signal.SIGKILL) - if self._stderr_task is not None: - await self._stderr_task - self._process_group_id = None - - def _signal_process_group(self, sig: signal.Signals) -> bool: - process_group_id = self._process_group_id - if process_group_id is None: - return False - try: - os.killpg(process_group_id, sig) - except ProcessLookupError: - return False - except OSError: - LOGGER.warning( - "failed to signal Codex app-server process group %s", - process_group_id, - exc_info=True, - ) - return False - return True - - async def _cleanup_isolated_home(self) -> None: - isolated_home = getattr(self, "_isolated_home", None) - if isolated_home is None: - return - - last_error: Exception | None = None - for delay_seconds in _CODEX_HOME_CLEANUP_RETRY_DELAYS_SECONDS: - if delay_seconds: - await asyncio.sleep(delay_seconds) - try: - isolated_home.cleanup() - return - except FileNotFoundError: - return - except OSError as exc: - last_error = exc - if exc.errno != errno.ENOTEMPTY: - break - except Exception as exc: # noqa: BLE001 - teardown must not mask outcome - last_error = exc - break - - # Temp-home teardown is resource hygiene, not episode validity. Once the - # app-server group has been reaped, make one non-throwing final removal so - # cleanup can neither invalidate a successful rollout nor mask its primary - # failure. A warning retains evidence if the path somehow remains. - if last_error is not None: - LOGGER.warning( - "Codex isolated HOME cleanup did not complete normally: %s", - last_error, - ) - try: - shutil.rmtree(isolated_home.name, ignore_errors=True) - if Path(isolated_home.name).exists(): - LOGGER.warning("Codex isolated HOME remains at %s", isolated_home.name) - except Exception: - LOGGER.warning( - "Codex isolated HOME final best-effort cleanup failed", - exc_info=True, - ) - - async def _send(self, value: dict[str, Any]) -> None: - if self._process is None or self._process.stdin is None: - raise CodexHarnessError("Codex app-server stdin is unavailable") - raw = _canonical_json(value) + b"\n" - self._process.stdin.write(raw) - await self._process.stdin.drain() - - async def _request(self, method: str, params: dict[str, Any]) -> dict[str, Any]: - self._request_id += 1 - request_id = self._request_id - await self._send({"id": request_id, "method": method, "params": params}) - while True: - message = await self._read_message() - if message.get("id") == request_id: - if "error" in message or not isinstance(message.get("result"), dict): - raise CodexHarnessError(f"Codex rejected app-server method {method}") - return message["result"] - await self._handle_message(message, allow_tools=False) - - async def _read_message(self) -> dict[str, Any]: - if self._process is None or self._process.stdout is None: - raise CodexHarnessError("Codex app-server stdout is unavailable") - while True: - if self._bridge.fatal.done(): - error = self._bridge.fatal.result() - raise error - if self._process.returncode is not None: - raise CodexHarnessError("Codex app-server exited before submit") - try: - line = await asyncio.wait_for( - self._process.stdout.readline(), timeout=0.1 - ) - except asyncio.TimeoutError: - continue - except (ValueError, asyncio.LimitOverrunError) as exc: - raise CodexHarnessError( - "Codex app-server emitted an oversized frame" - ) from exc - if self._bridge.fatal.done(): - raise self._bridge.fatal.result() - if not line: - raise CodexHarnessError("Codex app-server closed stdout before submit") - if len(line) > MAX_APP_SERVER_FRAME_BYTES: - raise CodexHarnessError("Codex app-server emitted an oversized frame") - try: - value = json.loads(line) - except json.JSONDecodeError as exc: - raise CodexHarnessError("Codex app-server emitted invalid JSON") from exc - if not isinstance(value, dict): - raise CodexHarnessError("Codex app-server emitted a non-object frame") - return value - - async def _handle_message(self, value: dict[str, Any], *, allow_tools: bool) -> str | None: - if self._bridge.fatal.done(): - raise self._bridge.fatal.result() - method = value.get("method") - if isinstance(value.get("id"), (int, str)) and method is not None: - if method != "item/tool/call" or not allow_tools: - raise CodexHarnessError("Codex app-server made an unexpected request") - return await self._handle_tool_request(value) - if method in {"thread/compacted", "error"}: - raise CodexHarnessError("Codex app-server reported a forbidden lifecycle event") - if method == "turn/completed": - raise CodexHarnessError("Codex completed without terminal submit") - return None - - async def _handle_tool_request(self, value: dict[str, Any]) -> str | None: - params = value.get("params") - request_id = value.get("id") - if not isinstance(params, dict): - raise CodexHarnessError("Codex supplied invalid tool request params") - pending = self._bridge.take_pending() - if ( - params.get("threadId") != self._thread_id - or params.get("turnId") != self._turn_id - or params.get("callId") != pending.call_id - or params.get("namespace") is not None - or params.get("tool") != pending.codex_name - ): - raise CodexHarnessError("Codex tool request identity drifted") - arguments = params.get("arguments") - if not isinstance(arguments, dict) or _canonical_json(arguments) != _canonical_json( - json.loads(pending.raw_arguments) - ): - raise CodexHarnessError("Codex mutated raw tool arguments") - started = time.monotonic() - terminal = False - if pending.codex_name == "terminal_exec": - output = await self._execute_terminal(arguments) - if "error" not in output: - self._metrics.terminal_calls += 1 - elif pending.codex_name == "submit": - output = await self._execute_submit(arguments) - if output.get("accepted") is True: - self._metrics.submit_calls += 1 - terminal = True - else: - raise CodexHarnessError("Codex requested a forbidden tool") - self._metrics.total_tool_time += time.monotonic() - started - rendered = json.dumps(output, ensure_ascii=False, separators=(",", ":")) - self._bridge.expect_tool_output(pending.call_id, rendered) - if terminal: - self._bridge.mark_terminal() - await self._send( - { - "id": request_id, - "result": { - "contentItems": [{"type": "inputText", "text": rendered}], - "success": True, - }, - } - ) - if terminal: - return "completed" - return None - - async def _execute_terminal(self, arguments: dict[str, Any]) -> dict[str, Any]: - if set(arguments) - {"command", "timeout_seconds"}: - return _tool_error("terminal_exec received unexpected arguments") - command = arguments.get("command") - if not isinstance(command, str) or not command.strip(): - return _tool_error("terminal_exec command must be non-empty") - maximum = legacy._positive_env("SECRLENV_TOOL_TIMEOUT_SECONDS", 120.0) - requested = arguments.get("timeout_seconds", maximum) - if isinstance(requested, bool) or not isinstance(requested, (int, float)): - return _tool_error("terminal_exec timeout is invalid") - timeout = float(requested) - if not math.isfinite(timeout) or not 1.0 <= timeout <= maximum: - return _tool_error("terminal_exec timeout is outside the signed range") - try: - result = await self._client.execute( - self._episode_id, - command, - timeout_seconds=timeout, - output_bytes=MAX_TOOL_OUTPUT_BYTES, - ) - except EpisodeAPIError as exc: - if exc.status == 400: - return _tool_error(exc.message) - raise - output = result.get("output", "") - truncated = result.get("truncated", False) - if ( - not isinstance(output, str) - or len(output.encode("utf-8")) > MAX_TOOL_OUTPUT_BYTES - or not isinstance(truncated, bool) - ): - raise CodexHarnessError("terminal_exec output exceeded its signed boundary") - return { - "exit_code": result.get("exit_code"), - "output": output or "(no output)", - "timed_out": bool(result.get("timed_out", False)), - "truncated": truncated, - } - - async def _execute_submit(self, arguments: dict[str, Any]) -> dict[str, Any]: - evidence = arguments.get("evidence") - if not isinstance(evidence, str) or not evidence: - return _tool_error("submit evidence must be non-empty") - try: - result = await self._client.submit(self._episode_id, arguments) - except EpisodeAPIError as exc: - if exc.status == 400: - return _tool_error(exc.message) - raise - if not isinstance(result, dict) or result.get("accepted") is not True: - raise CodexHarnessError("episode daemon did not accept submit") - return result - - async def drive(self) -> str: - try: - status = await self._drive_protocol() - except CodexSequenceLimit as exc: - self._metrics.end_reason = f"{type(exc).__name__}: {exc}"[:160] # type: ignore[attr-defined] - self._metrics.end_kind = end_kind_for(exc) # type: ignore[attr-defined] - return "max_seq_len" - except (CodexTurnLimit, CodexModelFailure) as exc: - # The signed status stays ``max_turns`` (reward 0, verifier still - # runs; changing it would change the signed outcome contract), but - # ``end_kind`` separates a real turn budget from a protocol error. - self._metrics.end_reason = f"{type(exc).__name__}: {exc}"[:160] # type: ignore[attr-defined] - self._metrics.end_kind = end_kind_for(exc) # type: ignore[attr-defined] - return "max_turns" - self._metrics.end_kind = ( # type: ignore[attr-defined] - END_KIND_SUBMIT if self._metrics.submit_calls else END_KIND_COMPLETED - ) - return status - - async def _drive_protocol(self) -> str: - initialized = await self._request( - "initialize", - { - "clientInfo": {"name": "yeto-secrlenv", "version": "1"}, - "capabilities": {"experimentalApi": True}, - }, - ) - user_agent = initialized.get("userAgent") - if not isinstance(user_agent, str) or "0.145.0" not in user_agent: - raise CodexHarnessError("Codex app-server reported the wrong version") - await self._send({"method": "initialized", "params": {}}) - started = await self._request( - "thread/start", - { - "model": BACKEND_MODEL, - "modelProvider": "miles", - "cwd": self._isolated_home.name, - "baseInstructions": base_instructions(), - "developerInstructions": None, - "dynamicTools": copy.deepcopy(dynamic_tools()), - "environments": [], - "ephemeral": True, - "approvalPolicy": "never", - "sandbox": "read-only", - "threadSource": "exec", - }, - ) - thread = started.get("thread") - self._thread_id = thread.get("id") if isinstance(thread, dict) else None - if not isinstance(self._thread_id, str) or not self._thread_id: - raise CodexHarnessError("Codex did not create an ephemeral thread") - turn = await self._request( - "turn/start", - { - "threadId": self._thread_id, - "input": [{"type": "text", "text": self._prompt, "text_elements": []}], - "effort": "xhigh", - "summary": "none", - "environments": [], - }, - ) - turn_value = turn.get("turn") - self._turn_id = turn_value.get("id") if isinstance(turn_value, dict) else None - if not isinstance(self._turn_id, str) or not self._turn_id: - raise CodexHarnessError("Codex did not start one turn") - while True: - message = await self._read_message() - result = await self._handle_message(message, allow_tools=True) - if result is not None: - return result - - -async def _drive_codex( - binary: Path, - miles_base_url: str, - client: EpisodeClient, - episode: dict[str, Any], - request_kwargs: dict[str, Any], - metrics: legacy.AgentMetrics, - *, - max_seq_len: int | None, -) -> str: - episode_id = episode.get("episode_id") - prompt = episode.get("prompt") - if not isinstance(episode_id, str) or not isinstance(prompt, str) or not prompt: - raise EpisodeClientError("episode daemon returned invalid episode identity") - async with _ResponsesBridge( - miles_base_url, - prompt, - request_kwargs, - metrics, - max_seq_len=max_seq_len, - ) as bridge, _AppServerDriver( - binary, bridge, client, episode_id, prompt, metrics - ) as driver: - return await driver.drive() - - -async def run( - base_url: str, - prompt: Any, - request_kwargs: dict[str, Any] | None = None, - metadata: dict[str, Any] | None = None, - **_kwargs: Any, -) -> dict[str, Any] | None: - """Run one stock-Codex/Miles SecRLEnv episode.""" - - del prompt - metadata = dict(metadata or {}) - request_kwargs = dict(request_kwargs or {}) - try: - task_id, tier = legacy._task_identity(metadata) - max_seq_len = legacy._metadata_max_seq_len(metadata) - infrastructure_503_retries = metadata.get( - INFRASTRUCTURE_503_RETRIES_KEY, 0 - ) - if ( - type(infrastructure_503_retries) is not int - or infrastructure_503_retries not in {0, 1} - ): - raise ValueError("invalid SecRLEnv infrastructure retry ledger") - binary = _attest_runtime() - except (ValueError, CodexHarnessError) as exc: - LOGGER.error("invalid signed Codex episode contract: %s", exc) - return None - - metrics = legacy.AgentMetrics() - episode_id: str | None = None - outcome_status = "completed" - infrastructure_failure = False - cleanup_failure = False - evaluation: dict[str, Any] | None = None - policy_task: asyncio.Task[str] | None = None - retry_ledger = { - INFRASTRUCTURE_503_RETRIES_KEY: infrastructure_503_retries - } - try: - async with EpisodeClient() as client: - started = time.monotonic() - episode = await legacy._create_with_capacity_retry( - client, task_id, tier, retry_ledger=retry_ledger - ) - metrics.create_time = time.monotonic() - started - episode_id = episode.get("episode_id") - if not isinstance(episode_id, str): - raise EpisodeClientError("episode daemon returned no episode ID") - policy_task = asyncio.create_task( - legacy._await_policy_and_claim_finalization( - episode_id, - _drive_codex( - binary, - base_url, - client, - episode, - request_kwargs, - metrics, - max_seq_len=max_seq_len, - ), - ) - ) - legacy._register_episode(episode_id, policy_task) - try: - outcome_status = await asyncio.wait_for( - policy_task, - timeout=legacy._positive_env( - "SECRLENV_MAX_ROLLOUT_TIME_SECONDS", 3600.0 - ), - ) - except asyncio.TimeoutError: - metrics.timed_out = 1 - metrics.end_kind = END_KIND_TIMEOUT # type: ignore[attr-defined] - outcome_status = "timeout" - except asyncio.CancelledError: - raise - except Exception: - infrastructure_failure = True - LOGGER.exception("stock Codex policy/tool loop failed") - - if legacy._claim_episode_finalization(episode_id): - started = time.monotonic() - try: - evaluation = await client.evaluate(episode_id) - except Exception: - infrastructure_failure = True - LOGGER.exception("secrlenv evaluation failed") - finally: - metrics.evaluate_time = time.monotonic() - started - started = time.monotonic() - try: - await client.close(episode_id) - except asyncio.CancelledError: - raise - except EpisodeAPIError as exc: - if exc.status == 404: - legacy._release_episode(episode_id) - else: - infrastructure_failure = True - cleanup_failure = ( - not await legacy._recover_failed_normal_close( - client, episode_id, policy_task - ) - ) - except Exception: # noqa: BLE001 - lifecycle cleanup is best-effort - infrastructure_failure = True - cleanup_failure = ( - not await legacy._recover_failed_normal_close( - client, episode_id, policy_task - ) - ) - else: - legacy._release_episode(episode_id) - finally: - metrics.close_time = time.monotonic() - started - metrics.total_tool_time += ( - metrics.create_time + metrics.evaluate_time + metrics.close_time - ) - else: - infrastructure_failure = True - except asyncio.CancelledError: - cleanup_claimed = episode_id is not None and ( - legacy._claim_episode_cleanup(episode_id) - or legacy._claim_cancelled_policy_cleanup(episode_id, policy_task) - ) - if cleanup_claimed: - try: - async with EpisodeClient(total_timeout_seconds=180.0) as cleanup_client: - await legacy._close_aborted_episode( - cleanup_client, - episode_id, - retry_timeout_seconds=legacy._positive_env( - "SECRLENV_ABORT_CLOSE_RETRY_SECONDS", 180.0 - ), - ) - except Exception: - LOGGER.warning("cancelled Codex episode cleanup failed", exc_info=True) - finally: - legacy._mark_episode_cleanup_pending(episode_id) - raise - except legacy._AdmissionInfrastructureExhausted: - if ( - episode_id is None - and retry_ledger[INFRASTRUCTURE_503_RETRIES_KEY] == 1 - ): - try: - outcome = legacy._admission_infrastructure_outcome( - task_id, - retry_ledger[INFRASTRUCTURE_503_RETRIES_KEY], - ) - return legacy._agent_result( - outcome, INFRASTRUCTURE_STATUS, metrics - ) - except Exception: - LOGGER.exception("refusing untrusted Codex admission metadata") - return None - LOGGER.error("invalid stock Codex admission infrastructure state") - return None - except EpisodeAPIError: - LOGGER.exception("stock Codex episode failed before a trustworthy verdict") - return None - except Exception: - LOGGER.exception("stock Codex episode failed before a trustworthy verdict") - return None - - if episode_id is None: - return None - try: - if cleanup_failure: - outcome_status = legacy.CLEANUP_ERROR_STATUS - outcome = legacy._cleanup_error_outcome( - task_id, - episode_id, - retry_ledger[INFRASTRUCTURE_503_RETRIES_KEY], - ) - elif infrastructure_failure or evaluation is None: - outcome_status = INFRASTRUCTURE_STATUS - outcome = legacy._infrastructure_outcome( - task_id, - episode_id, - retry_ledger[INFRASTRUCTURE_503_RETRIES_KEY], - ) - else: - outcome = legacy._validated_outcome( - evaluation, - task_id, - episode_id, - outcome_status, - infrastructure_503_retries=retry_ledger[ - INFRASTRUCTURE_503_RETRIES_KEY - ], - ) - result = { - OUTCOME_KEY: outcome, - "exit_status": outcome_status, - "agent_metrics": metrics.to_dict(), - } - capture_agent_metadata(result) - except Exception: - LOGGER.exception("refusing untrusted secrlenv evaluation metadata") - return None - return result - - -abort = legacy.abort diff --git a/yeto/rl/harness/codex/codex_openenv_agent_function.py b/yeto/rl/harness/codex/codex_openenv_agent_function.py deleted file mode 100644 index 74145a81a..000000000 --- a/yeto/rl/harness/codex/codex_openenv_agent_function.py +++ /dev/null @@ -1,681 +0,0 @@ -"""Terminal-Bench (OpenEnv) Codex adapter for the ports path. - -REWRITE, not a move: the legacy ``codex_openenv_*`` modules lived in a private -``agentenv/miles`` ``examples/experimental/openenv`` copy and were never found -(CODEX-PROGRESS §1.3). The interface below is the one the yeto consumers -expect (``yeto/rl/adapters/miles/island_entry.py``, ``yeto/rl/tbench_direct_preflight.py``, -``tests/test_rl_codex_schema.py``): ``stock``, ``_OPENENV_IDENTITY_ENV``, -``codex_openenv_harness_identity()`` and a module-level ``run``. - -Trust split (design D7): -- ``drive_untrusted`` runs Codex against a terminal environment and never runs - the verifier; -- ``finish_trusted`` runs the verifier and builds the ``tbench_outcome``. - -The stock driver and Responses bridge are reused unchanged from -``codex_harness_agent`` (moved from ``yeto_miles_secrlenv@5bfc011``). -""" - -from __future__ import annotations - -import asyncio -import json -import os -import secrets -import time -import uuid -from dataclasses import asdict -from pathlib import Path -from types import SimpleNamespace -from typing import Any, Awaitable, Callable, Mapping - -from yeto.rl.codex_backend import ( - QWEN3_30B_A3B_MODEL, - QWEN3_30B_A3B_REVISION, - QWEN35_08B_MODEL, - QWEN35_MODEL, - QWEN35_REVISION, - QWEN35_08B_REVISION, - QWEN38_MODEL, - QWEN38_REVISION, - QWEN38_NEXT_4LAYER_MODEL, - QWEN38_NEXT_4LAYER_REVISION, - QWEN38_NEXT_MODEL, - QWEN38_NEXT_REVISION, - stock_codex_backend_profile, -) -from yeto.rl.tbench_outcome import ( - NATIVE_VERIFIER, - TEST_SH_VERIFIER, - TIMEOUT_VERIFIER, - build_outcome_metadata, -) - -from . import agent as legacy -from . import codex_harness_agent as harness -from . import compaction_bridge -from .environment import TerminalEnvironment, TrustedVerifier - -# S17 M1: observe-only timing line appended to verifier_log (the tape keeps its 2000-char tail). -VERIFIER_LOG_TAIL_CHARS = 4000 -from .pins import OPENENV_BACKEND_PROFILE, OPENENV_BACKEND_PROFILES - -# rl-fn-codex-rollout 1.0: the backend profile is no longer pinned to the -# image default. Two distinct notions: -# * IMAGE_BACKEND_PROFILE_NAME -- the build-time default recorded in the image -# env (``YETO_CODEX_OPENENV_BACKEND_PROFILE`` / ``_MODEL_ID`` / ``_MODEL_REVISION``); -# * BACKEND_PROFILE_NAME -- the runtime profile declared by the launch -# (``--codex-backend-profile``), which the learner publishes to the container -# as ``YETO_CODEX_CHAT_TEMPLATE`` (``stock_codex_backend_contract(profile)["chat_template"]``, -# the same declaration ``codex_harness_agent`` reads) and ``validate_stock_codex_fields`` -# already checks against ``--model`` / ``--model-revision``. -# Why no image rebuild: the adapter's tool surface (``*_SHA256`` pins) does not -# depend on the model, so the image record only has to stay a faithful record of -# the build; the model identity of the runtime profile is derived here from the -# ``yeto.rl.codex_backend`` allowlist (``profile_identity``) instead of being -# compared against the image env. Boundary: the runtime profile must belong to -# ``pins.OPENENV_BACKEND_PROFILES`` and declare the exact HF identity listed in -# ``_PROFILE_IDENTITY``; anything else is refused at import / preflight. -IMAGE_BACKEND_PROFILE_NAME = OPENENV_BACKEND_PROFILE # "qwen35_08b" -RUNTIME_PROFILE_ENV = "YETO_CODEX_CHAT_TEMPLATE" -_PROFILE_IDENTITY: dict[str, tuple[str, str]] = { - "qwen35_08b": (QWEN35_08B_MODEL, QWEN35_08B_REVISION), - "qwen35": (QWEN35_MODEL, QWEN35_REVISION), - "qwen38": (QWEN38_MODEL, QWEN38_REVISION), - "qwen38_next": (QWEN38_NEXT_MODEL, QWEN38_NEXT_REVISION), - "qwen38_next_4layer": (QWEN38_NEXT_4LAYER_MODEL, QWEN38_NEXT_4LAYER_REVISION), - "qwen3_30b_a3b": (QWEN3_30B_A3B_MODEL, QWEN3_30B_A3B_REVISION), -} -assert set(_PROFILE_IDENTITY) == set(OPENENV_BACKEND_PROFILES) -INFRASTRUCTURE_STATUS = "infrastructure" -POLICY_STATUSES = frozenset({"completed", "timeout", "max_turns", "max_seq_len"}) -REWARD_SCOPE = "trajectory" - - -def profile_identity(profile_name: str) -> tuple[str, str]: - """``(model_identifier, model_revision)`` the adapter derives for one runtime profile. - - Raises ``ValueError`` ("requires backend profile") for a profile outside - ``OPENENV_BACKEND_PROFILES`` and ``RuntimeError`` ("profile drifted") when - the allowlisted ``codex_backend`` profile no longer declares the identity - pinned here (the legacy module-level qwen35_08b assertion, per profile). - """ - expected = _PROFILE_IDENTITY.get(profile_name) - if expected is None: - raise ValueError( - "the Codex OpenEnv adapter requires backend profile in " - + ", ".join(OPENENV_BACKEND_PROFILES) + f" (got {profile_name!r})" - ) - profile = stock_codex_backend_profile(profile_name) - if (profile.get("model_identifier"), profile.get("model_revision")) != expected: - raise RuntimeError(f"the Codex OpenEnv {profile_name} profile drifted") - return expected - - -def resolve_backend_profile(name: str | None = None, env: Mapping[str, str] | None = None) -> str: - """The runtime backend profile: explicit ``name``, else the launch declaration - ``YETO_CODEX_CHAT_TEMPLATE`` when it names an allowlisted profile (for every - member of ``OPENENV_BACKEND_PROFILES`` the chat template equals the profile - name), else the image default. An explicit name is validated.""" - if name is not None: - profile_identity(name) - return name - env = os.environ if env is None else env - declared = env.get(RUNTIME_PROFILE_ENV) - if declared in _PROFILE_IDENTITY: - return declared - return IMAGE_BACKEND_PROFILE_NAME - - -BACKEND_PROFILE_NAME = resolve_backend_profile() -profile_identity(BACKEND_PROFILE_NAME) # import-time drift check, per selected profile - -stock = SimpleNamespace( - _BACKEND_PROFILE=stock_codex_backend_profile(BACKEND_PROFILE_NAME), - BACKEND_MODEL=stock_codex_backend_profile(BACKEND_PROFILE_NAME)["model"], - _attest_runtime=harness._attest_runtime, -) - - -def codex_openenv_harness_identity() -> dict[str, str]: - """Live identity of the Codex surface this adapter drives (same tools as the stock harness).""" - live = harness.codex_harness_identity() - return {key: value for key, value in live.items() if key.endswith("_sha256")} - - -# Image record (what the image line bakes into the container env and -# ``yeto.rl.CODEX_OPENENV_IDENTITY_ENV`` mirrors): the *build-time* profile and -# its HF identity plus the profile-independent tool-surface hashes. It is -# deliberately NOT derived from BACKEND_PROFILE_NAME so the container env drift -# check stays a check of the image, whichever runtime profile the launch declares. -_OPENENV_IDENTITY_ENV: dict[str, str] = { - "YETO_CODEX_OPENENV_BACKEND_PROFILE": IMAGE_BACKEND_PROFILE_NAME, - "YETO_CODEX_OPENENV_MODEL_ID": _PROFILE_IDENTITY[IMAGE_BACKEND_PROFILE_NAME][0], - "YETO_CODEX_OPENENV_MODEL_REVISION": _PROFILE_IDENTITY[IMAGE_BACKEND_PROFILE_NAME][1], - **{ - f"YETO_CODEX_OPENENV_{name.upper()}": value - for name, value in codex_openenv_harness_identity().items() - }, -} - - -class ToolWaitEmitter: - """Per-tool-call tool-wait hooks (3.1). ``enter``/``exit`` are sync callables.""" - - def __init__(self, enter: Callable[[], None], exit: Callable[[], None]) -> None: - self.enter = enter - self.exit = exit - - -class _ToolWaitEnvironment: - """Wrap an environment so every tool execution is one tool-wait interval.""" - - def __init__(self, env: TerminalEnvironment, emitter: ToolWaitEmitter | None) -> None: - self._env = env - self._emitter = emitter - - async def _scoped(self, coro: Awaitable[dict[str, Any]]) -> dict[str, Any]: - if self._emitter is None: - return await coro - self._emitter.enter() - try: - return await coro - finally: - self._emitter.exit() - - async def execute(self, episode_id: str, command: str, timeout_seconds: float, output_bytes: int): - return await self._scoped( - self._env.execute(episode_id, command, timeout_seconds=timeout_seconds, output_bytes=output_bytes) - ) - - async def submit(self, episode_id: str, submission: dict[str, Any]): - return await self._scoped(self._env.submit(episode_id, submission)) - - -def _max_rollout_seconds() -> float: - return legacy._positive_env("OPENENV_MAX_ROLLOUT_TIME_SECONDS", 1800.0) - - -async def drive_untrusted( - job: dict[str, Any], - env: TerminalEnvironment, - *, - tool_wait: ToolWaitEmitter | None = None, - binary: Path | None = None, -) -> dict[str, Any]: - """Untrusted layer: run one Codex episode. Returns an UNSIGNED result. - - ``job`` = {base_url, prompt, request_kwargs, episode_id, max_seq_len?}. - Policy boundaries (turn/seq/timeout) are reported as statuses; harness - protocol failures raise ``CodexHarnessError`` and are infrastructure errors. - """ - metrics = legacy.AgentMetrics() - scoped_env = _ToolWaitEnvironment(env, tool_wait) - episode = {"episode_id": job["episode_id"], "prompt": job["prompt"]} - codex = binary if binary is not None else harness._attest_runtime() - segment_urls = job.get(compaction_bridge.SEGMENT_URLS_KEY) - if compaction_bridge.compactionrl_enabled(): - if not isinstance(segment_urls, list) or not all(isinstance(u, str) for u in segment_urls): - raise harness.CodexHarnessError("CompactionRL job has no pre-created segment sessions") - driving = compaction_bridge.drive_codex_compactionrl( - codex, - job["base_url"], - scoped_env, - episode, - dict(job.get("request_kwargs") or {}), - metrics, - max_seq_len=job.get("max_seq_len"), - segment_base_urls=segment_urls, - ) - elif segment_urls is not None: - raise harness.CodexHarnessError("segment sessions supplied without CompactionRL") - else: - driving = harness._drive_codex( - codex, - job["base_url"], - scoped_env, - episode, - dict(job.get("request_kwargs") or {}), - metrics, - max_seq_len=job.get("max_seq_len"), - ) - try: - status = await asyncio.wait_for(driving, timeout=_max_rollout_seconds()) - except asyncio.TimeoutError: - metrics.timed_out = 1 - metrics.end_kind = harness.END_KIND_TIMEOUT # type: ignore[attr-defined] - status = "timeout" - except harness.CodexHarnessError as exc: - # The rejection counters (G6a) must survive the failure: the trusted - # layer mirrors them onto the HarnessBoard before aborting the sample. - exc.metrics = _metrics_dict(metrics) # type: ignore[attr-defined] - raise - if status not in POLICY_STATUSES: - raise harness.CodexHarnessError(f"Codex driver returned unknown status {status!r}") - return {"status": status, "metrics": _metrics_dict(metrics), "episode_id": job["episode_id"]} - - -def _metrics_dict(metrics: legacy.AgentMetrics) -> dict[str, Any]: - # compaction_counters is {} unless the CompactionRL bridge ran. - extra: dict[str, Any] = {} - # Observe only (S15 r2 follow-up): why the Codex episode ended and the shape - # of the last model reply; set by the stock driver, absent otherwise. - # S17 C9: finer end reason and per-turn lengths (model reply tokens, context - # tokens after the reply, tool-result bytes). - for name in ("end_reason", "last_completion", "end_kind", "turn_completion_tokens", - "turn_context_tokens", "turn_tool_output_bytes", - # agentic-rollout-utilization 5.1/5.3: only when a gate was in use - *(harness.SUSPEND_METRIC_FIELDS if getattr(metrics, "suspensions", 0) else ())): - value = getattr(metrics, name, None) - if value is not None: - extra[name] = value - return {**asdict(metrics), **harness.tito_counters(metrics), **compaction_bridge.compaction_counters(metrics), **extra} - - -# --- CompactionRL segment sessions (trusted side; design D8) ---------------- -_SESSION_FIELDS = ("temperature", "top_p", "top_k") - - -def split_session_url(base_url: str) -> tuple[str, str]: - """``{router}/sessions/{id}`` -> (router, id).""" - head, sep, session_id = base_url.rstrip("/").rpartition("/sessions/") - if not sep or not head or not session_id or "/" in session_id: - raise ValueError(f"not a session-server session URL: {base_url!r}") - return head, session_id - - -def segment_session_body(request_kwargs: dict[str, Any]) -> dict[str, Any]: - """``CreateSessionRequest`` body with the rollout's own sampling defaults.""" - body: dict[str, Any] = {"evaluation": False} - for key in _SESSION_FIELDS: - value = request_kwargs.get(key) - if value is not None: - body[key] = int(value) if key == "top_k" else float(value) - return body - - -async def _post_json(url: str, body: dict[str, Any]) -> dict[str, Any]: - import aiohttp - - async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=120)) as session: - async with session.post(url, json=body) as response: - if response.status != 200: - raise RuntimeError(f"session server returned HTTP {response.status}") - return await response.json() - - -async def create_segment_sessions( - base_url: str, - request_kwargs: dict[str, Any], - count: int, - *, - post: Callable[[str, dict[str, Any]], Awaitable[dict[str, Any]]] | None = None, - delete: Callable[[str], Awaitable[None]] | None = None, -) -> tuple[list[str], list[str]]: - """Pre-create one session per allowed compaction; returns (ids, agent base URLs). - - Created here, in the trusted layer, so the untrusted worker can only use - sessions this rollout owns; ``codex_openenv_generate`` collects (and so - deletes) every listed session, used or not. - """ - router, _ = split_session_url(base_url) - body = segment_session_body(request_kwargs) - ids: list[str] = [] - try: - for _ in range(count): - reply = await (post or _post_json)(f"{router}/sessions", dict(body)) - session_id = reply.get("session_id") if isinstance(reply, dict) else None - if not isinstance(session_id, str) or not session_id or "/" in session_id: - raise RuntimeError("session server returned no session_id") - ids.append(session_id) - except BaseException: - # Pre-creation failed half-way: nobody will collect the ones already made. - await delete_segment_sessions(base_url, ids, delete=delete) - raise - return ids, [f"{router}/sessions/{session_id}" for session_id in ids] - - -async def _delete_session(url: str) -> None: - import aiohttp - - async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=60)) as session: - async with session.delete(url) as response: - # Pinned session server (miles/rollout/session/sessions.py): 204 on - # success; an already collected/deleted session is a 404. - if response.status not in (200, 204, 404): - raise RuntimeError(f"session server returned HTTP {response.status}") - - -async def delete_segment_sessions( - base_url: str, - session_ids: Any, - *, - delete: Callable[[str], Awaitable[None]] | None = None, -) -> list[str]: - """Best-effort ``DELETE {router}/sessions/{id}`` for every id; returns the failures. - - Used when pre-created segment sessions will not reach - ``codex_openenv_generate`` (which otherwise collects and so deletes them). - Never raises: a cleanup failure must not mask the original error. - """ - if not isinstance(session_ids, (list, tuple)) or not session_ids: - return [] - try: - router, _ = split_session_url(base_url) - except ValueError: - return [str(session_id) for session_id in session_ids] - failed: list[str] = [] - for session_id in session_ids: - try: - await (delete or _delete_session)(f"{router}/sessions/{session_id}") - except Exception: # noqa: BLE001 - keep deleting the rest - failed.append(str(session_id)) - return failed - - -async def release_unreturned_segments( - base_url: str, - segments: dict[str, Any], - *, - delete: Callable[[str], Awaitable[None]] | None = None, -) -> list[str]: - """Delete the sessions listed in ``segments`` (the trusted metadata from - :func:`prepare_segment_sessions`) when it is not being returned.""" - return await delete_segment_sessions( - base_url, segments.get(compaction_bridge.SESSIONS_METADATA_KEY), delete=delete - ) - - -def _rlf_grade_from_evaluation(evaluation: dict[str, Any]) -> Any: - """RLF grade interface (rlf-codex-rl breakpoint 5); lazy import keeps the - neutral RLF package out of the TB2 import graph. A malformed grade raises - ``RolloutRejected`` (fail closed) instead of downgrading to the TB2 path.""" - from yeto.rl.rlf.preflight import trusted_grade_from_evaluation - - return trusted_grade_from_evaluation(evaluation) - - -def _finish_rlf( - untrusted: dict[str, Any], - grade: Any, - *, - task_id: str, - sample_id: str, - evaluate_time: float, - evaluation: dict[str, Any], - status: str, -) -> dict[str, Any]: - """Trusted finish for an RLF grade: close the episode via ``finish_episode`` - (reward/success/grader evidence + the RLF identity quadruple, written by the - trusted side from the per-lease verifier's identity) on top of the usual - outcome envelope, so tape consumers keep working. The ``policy_id`` - leg of the quadruple is not produced here: it rides the input metadata and - is copied onto the chains by ``codex_openenv_generate`` bookkeeping.""" - from yeto.rl.rlf.contracts import EpisodeLease, finish_episode - from yeto.rl.rlf.preflight import RolloutRejected - - evidence_map = grade.grader_evidence - rlf_task_id = evidence_map.get("rlf_task_id") - rlf_episode_id = evidence_map.get("rlf_episode_id") - rlf_trajectory_id = evidence_map.get("rlf_trajectory_id") - if not all(isinstance(value, str) and value - for value in (rlf_task_id, rlf_episode_id, rlf_trajectory_id)): - raise RolloutRejected("RLF grade is missing the identity quadruple") - if rlf_task_id != task_id: - raise RolloutRejected(f"RLF grade task {rlf_task_id!r} does not match rollout task {task_id!r}") - lease = EpisodeLease(task_id=rlf_task_id, trajectory_id=rlf_trajectory_id, - episode_id=rlf_episode_id) - reward = finish_episode(lease, grade) - metadata = build_outcome_metadata( - task_id=task_id, - sample_id=sample_id, - episode_id=untrusted["episode_id"], - status=status, - reward=float(grade.reward), - verifier=NATIVE_VERIFIER, - testsh_rc=None, - ) - metadata.update(reward.metadata) # success, reward, quadruple, grader_evidence, rlf_status - metadata["agent_metrics"] = dict(untrusted.get("metrics") or {}) - metadata["agent_metrics"]["evaluate_time"] = evaluate_time - metadata["exit_status"] = status - if isinstance(evaluation.get("log"), str): - # Observe only; the grader contract forbids the flag in it. - metadata["verifier_log"] = evaluation["log"][-VERIFIER_LOG_TAIL_CHARS:] - return metadata - - -async def finish_trusted( - untrusted: dict[str, Any], - verifier: TrustedVerifier, - *, - task_id: str, - sample_id: str, -) -> dict[str, Any]: - """Trusted layer: verify and build the outcome (reward in {0,1}). - - When the verifier's evaluation carries an RLF grade the episode is closed - via ``finish_episode`` (``_finish_rlf``); anything else takes the TB2 - path below, byte-identical to before the RLF branch existed. - """ - status = untrusted["status"] - if status not in POLICY_STATUSES: - raise ValueError(f"untrusted result has invalid status {status!r}") - evaluation: dict[str, Any] = {} - evaluate_time: float | None = None - if status == "timeout": - passed, testsh_rc, verifier_name = False, None, TIMEOUT_VERIFIER - else: - started = time.monotonic() - evaluation = await verifier.evaluate(untrusted["episode_id"]) - evaluate_time = verify_seconds = time.monotonic() - started # N13 metric + M1 YETO_TIMING - grade = _rlf_grade_from_evaluation(evaluation) - if grade is not None: - return _finish_rlf(untrusted, grade, task_id=task_id, sample_id=sample_id, - evaluate_time=evaluate_time, evaluation=evaluation, status=status) - passed = bool(evaluation.get("passed")) - testsh_rc = evaluation.get("testsh_rc") - verifier_name = ( - TEST_SH_VERIFIER if isinstance(testsh_rc, int) else NATIVE_VERIFIER - ) - metadata = build_outcome_metadata( - task_id=task_id, - sample_id=sample_id, - episode_id=untrusted["episode_id"], - status=status, - reward=1.0 if passed else 0.0, - verifier=verifier_name, - testsh_rc=testsh_rc if isinstance(testsh_rc, int) else None, - ) - metadata["agent_metrics"] = dict(untrusted.get("metrics") or {}) - # S17 N13: the trusted verifier owns grading, so it owns the grading time - # (seconds, observe only). The worker's own ``evaluate_time`` is - # 0.0 on this path; a timeout skips grading and drops the key. - if evaluate_time is None: - metadata["agent_metrics"].pop("evaluate_time", None) - else: - metadata["agent_metrics"]["evaluate_time"] = evaluate_time - metadata["exit_status"] = status - if status != "timeout" and isinstance(evaluation.get("log"), str): - metadata["verifier_log"] = evaluation["log"] # observe only - # S17 M1 (observe only): verifier wall time and agent phase times - # (per-turn shapes would need the pinned codex_harness_agent.py), appended as the log's last line so the tape's - # verifier_log tail carries them. - metrics = untrusted.get("metrics") or {} - timing = {"verify_s": round(verify_seconds, 3)} - for name in ("create_time", "total_generation_time", "total_tool_time", "max_model_total_tokens"): - if isinstance(metrics.get(name), (int, float)) and not isinstance(metrics.get(name), bool): - timing[name] = round(float(metrics[name]), 3) - metadata["verifier_log"] = (metadata["verifier_log"] + "\nYETO_TIMING " - + json.dumps(timing, separators=(",", ":")))[-VERIFIER_LOG_TAIL_CHARS:] - return metadata - - -def infrastructure_metadata(reason: str, *, episode_id: str | None, metrics: dict[str, Any] | None = None) -> dict[str, Any]: - """Infrastructure marker: no reward, sample must be ABORTED (never a 0 reward).""" - return { - "tbench_infrastructure_error": reason, - "exit_status": INFRASTRUCTURE_STATUS, - "episode_id": episode_id, - "agent_metrics": dict(metrics or {}), - } - - -TITO_SESSION_MISMATCH_KEY = harness.TITO_SESSION_MISMATCH_KEY -TITO_CHAIN_BREAKS_KEY = harness.TITO_CHAIN_BREAKS_KEY - - -def mirror_tito_counters(metrics: dict[str, Any] | None, harness_board: Any) -> dict[str, Any]: - """G6(a): replay the bridge's rejection counters onto the HarnessBoard. - - Same method names and counter names as ``gateway/core.py`` so the 1.7 load - sample (``tito_session_mismatch`` / ``tito_chain_breaks``) reads the A path - too. Returns the trajectory-field overrides: the bridge holds one chain and - never forks, so ``chains_total`` stays 1 and ``chain_break_reason`` names the - first break (the sample is infrastructure-aborted, never trained on). - """ - from yeto.rl.engine.rollout_meta import counter_value - - metrics = metrics or {} - mismatches = counter_value(metrics.get(TITO_SESSION_MISMATCH_KEY)) - breaks = dict(metrics.get(TITO_CHAIN_BREAKS_KEY) or {}) - if harness_board is not None: - if mismatches: - _board_call(harness_board, "record_session_mismatch", mismatches) - for reason, count in breaks.items(): - if count: - _board_call(harness_board, "record_chain_break", reason, int(count)) - first_break = next((reason for reason, count in breaks.items() if count), None) - return {"chain_break_reason": first_break} - - -def _board_call(target: Any, method: str, *args: Any) -> Any: - """Call a board method and wait for it (A-T3-7 follow-up, S14/A19). - - The island boards are Ray actors with ``max_concurrency=64``: a - fire-and-forget ``record_session_mismatch`` / ``record_chain_break`` may - land after the round's ``snapshot()`` (counter missing from the rollout it - belongs to) and any error it raises is an unhandled actor error instead of - this trajectory's. Resolving the ObjectRef here keeps the calls in issue - order, like the tool-wait board's enter/exit. A failure of the remote - bookkeeping is logged and never fails the trajectory; a local board's own - exception (e.g. an unknown chain-break reason) propagates as before.""" - import sys - - from yeto.rl.engine.tool_wait import _resolve - - fn = getattr(target, method) - remote = getattr(fn, "remote", None) - if not callable(remote): - return fn(*args) - try: - return _resolve(remote(*args)) - except Exception as exc: # noqa: BLE001 - never fail a trajectory on telemetry - print(f"[codex-harness] harness board {method} failed: {type(exc).__name__}: {exc}", - file=sys.stderr) - return None - - -def trajectory_fields(trajectory_id: str) -> dict[str, Any]: - """D8 / R-D5a bookkeeping carried on every sample of the trajectory.""" - return { - "trajectory_id": trajectory_id, - "chain_index": 0, - "chains_total": 1, - "chain_break_reason": None, - "segment_id": 0, - "segment_boundary_reason": None, - "reward_scope": REWARD_SCOPE, - } - - -def task_identity(metadata: dict[str, Any]) -> tuple[str, str]: - task_id = metadata.get("task_id") - if not isinstance(task_id, str) or not task_id: - raise ValueError("Terminal-Bench sample metadata must carry task_id") - sample_id = metadata.get("sample_id") or metadata.get("trajectory_id") or f"tb-{uuid.uuid4().hex}" - return task_id, str(sample_id) - - -def new_episode_id() -> str: - return f"ep-{secrets.token_hex(8)}" - - -async def run( - base_url: str, - prompt: Any, - request_kwargs: dict[str, Any] | None = None, - metadata: dict[str, Any] | None = None, - *, - environment: TerminalEnvironment | None = None, - verifier: TrustedVerifier | None = None, - tool_wait: ToolWaitEmitter | None = None, - harness_board: Any = None, - **_kwargs: Any, -) -> dict[str, Any] | None: - """In-process entry (both layers in one process; use the subprocess entry in training).""" - metadata = dict(metadata or {}) - if environment is None or verifier is None: - raise ValueError("in-process run requires an environment and a verifier") - task_id, sample_id = task_identity(metadata) - trajectory_id = metadata.get("trajectory_id") or sample_id - episode_id = new_episode_id() - job = { - "base_url": base_url, - "prompt": metadata.get("prompt") if isinstance(metadata.get("prompt"), str) else str(prompt), - "request_kwargs": dict(request_kwargs or {}), - "episode_id": episode_id, - "max_seq_len": metadata.get("max_seq_len"), - } - try: - segments = await prepare_segment_sessions(job) - except Exception as exc: # noqa: BLE001 - session-server failures are infrastructure - return { - **infrastructure_metadata(f"segment sessions: {type(exc).__name__}: {exc}", episode_id=episode_id), - **trajectory_fields(trajectory_id), - } - handed_off = False # True once ``segments`` rides in the returned metadata - try: - try: - untrusted = await drive_untrusted(job, environment, tool_wait=tool_wait) - except (harness.CodexHarnessError, legacy.EpisodeClientError, OSError) as exc: - metrics = getattr(exc, "metrics", None) - tito = mirror_tito_counters(metrics, harness_board) - handed_off = True - return { - **infrastructure_metadata(f"{type(exc).__name__}: {exc}", episode_id=episode_id, metrics=metrics), - **trajectory_fields(trajectory_id), - **tito, - **segments, - } - tito = mirror_tito_counters(untrusted.get("metrics"), harness_board) - finished = await finish_trusted(untrusted, verifier, task_id=task_id, sample_id=sample_id) - handed_off = True - return {**finished, **trajectory_fields(trajectory_id), **tito, **segments} - finally: - if segments and not handed_off: - await release_unreturned_segments(job["base_url"], segments) - - -async def prepare_segment_sessions( - job: dict[str, Any], - *, - post: Callable[[str, dict[str, Any]], Awaitable[dict[str, Any]]] | None = None, - delete: Callable[[str], Awaitable[None]] | None = None, -) -> dict[str, Any]: - """CompactionRL only: add segment URLs to ``job``; return the trusted metadata. - - Returns ``{}`` (and leaves ``job`` untouched) when CompactionRL is off. - """ - if not compaction_bridge.compactionrl_enabled(): - return {} - if job.get("max_seq_len") is None: - raise harness.CodexHarnessError("CompactionRL requires max_seq_len") - cfg = compaction_bridge.compaction_config(int(job["max_seq_len"])) - ids, urls = await create_segment_sessions( - job["base_url"], dict(job.get("request_kwargs") or {}), cfg.max_compactions, post=post, delete=delete - ) - job[compaction_bridge.SEGMENT_URLS_KEY] = urls - router, _ = split_session_url(job["base_url"]) - return {compaction_bridge.SESSIONS_METADATA_KEY: ids, - compaction_bridge.SESSIONS_ROUTER_METADATA_KEY: router} diff --git a/yeto/rl/harness/codex/codex_openenv_agent_worker.py b/yeto/rl/harness/codex/codex_openenv_agent_worker.py deleted file mode 100644 index f9733be4b..000000000 --- a/yeto/rl/harness/codex/codex_openenv_agent_worker.py +++ /dev/null @@ -1,81 +0,0 @@ -"""Untrusted worker process for the Terminal-Bench Codex adapter (REWRITE). - -Protocol (stdin/stdout, one JSON object per line): -- stdin: one job ``{"base_url", "prompt", "request_kwargs", "episode_id", - "max_seq_len", "env_url", "env_token", "driver": "stock"|"scripted", ...}``; -- stdout events: ``{"event": "tool_wait", "phase": "enter"|"exit"}`` for every - tool execution, ``{"event": "suspend", "phase": "enter"|"exit"}`` around a - wait at the suspended model-turn gate, then exactly one ``{"event": "result", ...}`` or - ``{"event": "error", "reason": ...}``. - -The worker refuses to start when a reward HMAC key is visible (trust split). -``driver: "scripted"`` is a CPU-test driver (no Codex, no Miles) and is only -honoured when ``YETO_CODEX_OPENENV_ALLOW_SCRIPTED_DRIVER=1``; preflight fails -closed when that variable is set. -""" - -from __future__ import annotations - -import asyncio -import json -import os -import sys -from typing import Any - -from . import codex_openenv_agent_function as adapter -from .environment import HttpTerminalEnvironment - -SCRIPTED_DRIVER_ENV = "YETO_CODEX_OPENENV_ALLOW_SCRIPTED_DRIVER" - - -def _emit(payload: dict[str, Any]) -> None: - sys.stdout.write(json.dumps(payload, sort_keys=True) + "\n") - sys.stdout.flush() - - -async def _scripted_drive(job: dict[str, Any], env: Any, tool_wait: adapter.ToolWaitEmitter) -> dict[str, Any]: - """Execute ``job['script']`` commands then submit; mimics the driver's tool loop.""" - scoped = adapter._ToolWaitEnvironment(env, tool_wait) - for command in job.get("script") or []: - await scoped.execute(job["episode_id"], str(command), timeout_seconds=30.0, output_bytes=4096) - if job.get("hang_seconds"): - await asyncio.sleep(float(job["hang_seconds"])) - await scoped.submit(job["episode_id"], {"evidence": "scripted"}) - return {"status": job.get("final_status", "completed"), "metrics": {"turns": len(job.get("script") or []) + 1}, "episode_id": job["episode_id"]} - - -async def _main_async(job: dict[str, Any]) -> dict[str, Any]: - tool_wait = adapter.ToolWaitEmitter( - enter=lambda: _emit({"event": "tool_wait", "phase": "enter"}), - exit=lambda: _emit({"event": "tool_wait", "phase": "exit"}), - ) - # agentic-rollout-utilization 5.1: parked between model turns (gate closed) - adapter.harness.set_suspend_event_sink(lambda phase: _emit({"event": "suspend", "phase": phase})) - async with HttpTerminalEnvironment(job["env_url"], job["env_token"]) as env: - if job.get("driver") == "scripted": - if os.getenv(SCRIPTED_DRIVER_ENV) != "1": - raise RuntimeError("scripted driver is not allowed in this process") - return await _scripted_drive(job, env, tool_wait) - return await adapter.drive_untrusted(job, env, tool_wait=tool_wait) - - -def main(argv: list[str] | None = None) -> int: - del argv - try: - job = json.loads(sys.stdin.readline()) - if not isinstance(job, dict): - raise ValueError("worker job must be a JSON object") - result = asyncio.run(_main_async(job)) - except BaseException as exc: # noqa: BLE001 - every failure is reported on the wire - event = {"event": "error", "reason": f"{type(exc).__name__}: {exc}"} - metrics = getattr(exc, "metrics", None) - if isinstance(metrics, dict): - event["metrics"] = metrics # G6a: rejection counters reach the trusted layer - _emit(event) - return 1 - _emit({"event": "result", **result}) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/yeto/rl/harness/codex/codex_openenv_generate.py b/yeto/rl/harness/codex/codex_openenv_generate.py deleted file mode 100644 index e4f781d5a..000000000 --- a/yeto/rl/harness/codex/codex_openenv_generate.py +++ /dev/null @@ -1,420 +0,0 @@ -"""``--custom-generate-function-path`` wrapper around upstream ``agentic_tool_call.generate``. - -Adds the R-D5a / R-IR-3 bookkeeping that the upstream session path does not -know about: - -- sibling samples of one trajectory get the same ``group_index`` and - ``rollout_id`` so ``train_data_conversion`` shares one reward and counts one - baseline entry per trajectory (never per chain); -- ``expected_policy_version`` (IR-3: ``rollout_meta_hook.expected_policy_version``, - i.e. prompt metadata first, else the driver token published through the - metadata sink by ``MilesRolloutPool.generate``) is compared against the - weight versions SGLang reported for every generation; under the default - policy-age limit 0 any drift aborts the trajectory (metadata ``policy_age_violation=1`` -> ``harness_counters`` -> - driver ``PolicyIdentityError``) instead of giving it a 0 reward; a missing - token refuses generation before upstream is called; under a limit > 0 - (agentic-rollout-utilization 5.2: a trajectory suspended between model turns - continues under the next version) the versions are recorded as segments and - checked against the version current when the trajectory finishes: oldest at - most ``limit`` behind, none newer; outside the window the trajectory is - discarded (infrastructure ``policy_age_exceeded``, not a policy-identity - error); -- mask/token/logprob alignment is asserted on every sample (``alignment``). - -The upstream function is injected (``upstream``) so this wrapper is testable -without the Miles package. -""" - -from __future__ import annotations - -import asyncio -import dataclasses -import os -import uuid -from typing import Any, Awaitable, Callable - -from yeto.rl.engine import rollout_meta -from yeto.rl.engine.policy_versions import engine_version_matches, engine_version_number -from yeto.rl.rlf.contracts import EpisodeLease - -from .alignment import AlignmentError, assert_sample_alignment -from .tbench_reward import INFRASTRUCTURE_KEY - -UPSTREAM_PATH = "miles.rollout.generate_hub.agentic_tool_call.generate" -POLICY_AGE_KEY = rollout_meta.POLICY_AGE_VIOLATION_KEY # "policy_age_violation" -EXPECTED_VERSION_KEY = rollout_meta.EXPECTED_POLICY_VERSION_KEY -ACTUAL_VERSIONS_KEY = "policy_versions_actual" - - -class PolicyVersionMissing(RuntimeError): - """IR-3: no target policy token for this rollout (driver did not publish one).""" - - -def expected_policy_version(input_sample: Any) -> str | None: - """IR-3 target token for ``input_sample`` (metadata first, else the driver sink).""" - try: - return rollout_meta.expected_policy_version(input_sample) - except Exception: # noqa: BLE001 - unreachable sink == nothing published - meta = getattr(input_sample, "metadata", None) - return str(meta[EXPECTED_VERSION_KEY]) if isinstance(meta, dict) and meta.get(EXPECTED_VERSION_KEY) else None - - -def _glue(): - """Backend glue: upstream generate, aborted status, session collection (decoupling 5.7).""" - from yeto.rl.engine import backends - - return backends.module("harness_glue") - - -def _load_upstream() -> Callable[[Any], Awaitable[Any]]: - return _glue().load_agentic_upstream() - - -def _samples_of(output: Any) -> list[Any]: - samples = getattr(output, "samples", output) - return list(samples) if isinstance(samples, list) else [samples] - - -def _mark_aborted(sample: Any, reason: str) -> None: - try: - sample.status = _glue().aborted_status() - except ImportError: # backend not installed (CPU tests) - sample.status = "ABORTED" - metadata = sample.metadata if isinstance(getattr(sample, "metadata", None), dict) else {} - metadata.pop("tbench_trusted_outcome", None) - metadata[INFRASTRUCTURE_KEY] = reason - sample.metadata = metadata - - -def actual_versions(sample: Any) -> list[str]: - versions: list[str] = [] - for call in getattr(sample, "weight_versions", None) or []: - spans = getattr(call, "spans", None) - if spans is None and isinstance(call, dict): - spans = call.get("spans", []) - for span in spans or []: - version = getattr(span, "version", None) - if version is None and isinstance(span, dict): - version = span.get("version") - if version is not None: - versions.append(str(version)) - return versions - - -POLICY_AGE_EXCEEDED = "policy_age_exceeded" -VERSION_SEGMENTS_KEY = "policy_version_segments" - - -def window_problem(versions: list[str], current_token: str | None, limit: int) -> str | None: - """Limit > 0: why ``versions`` (one per generation span) cannot be trained at - ``current_token`` (None = within the window).""" - from yeto.rl.engine.policy_age import token_version - - current = token_version(str(current_token or "")) - if current is None: - return f"current policy token {current_token!r} has no version" - if not versions: - return "no generation version recorded" - numbers = [engine_version_number(v) for v in versions] - if None in numbers: - return f"unparsable generation versions {sorted(set(versions))}" - if max(numbers) > current: - return f"generation version {max(numbers)} is newer than the current {current}" - if current - min(numbers) > limit: - return f"oldest generation version {min(numbers)} is {current - min(numbers)} behind {current} (limit {limit})" - return None - - -RLF_TASK_KEY = "rlf_task_id" -RLF_IDENTITY_KEYS = ("rlf_task_id", "rlf_episode_id", "rlf_trajectory_id", "policy_id") - - -def produce_rlf_identity(input_meta: dict[str, Any], *, expected_version: str | None = None) -> dict[str, Any]: - """Producer for the RLF identity quadruple on catalog-sourced rows. - - Rows emitted by ``yeto.rl.rlf.catalog.prompt_rows`` carry ``rlf_task_id`` - only; the per-rollout episode/trajectory ids are produced here (one - ``EpisodeLease`` per prompt draw) so episodes are never shared across - rollouts, and ``policy_id`` defaults to the IR-3 expected policy version. - Non-RLF rows (no ``rlf_task_id``) pass through unchanged, and ids already - present (e.g. written by the trusted finish) are never overwritten. - """ - if not input_meta.get(RLF_TASK_KEY): - return input_meta - meta = dict(input_meta) - lease = EpisodeLease( - task_id=str(meta[RLF_TASK_KEY]), - trajectory_id=str(meta.get("rlf_trajectory_id") or f"rlf-traj-{uuid.uuid4().hex}"), - ) - if not meta.get("rlf_episode_id"): - meta["rlf_episode_id"] = lease.episode_id - if not meta.get("rlf_trajectory_id"): - meta["rlf_trajectory_id"] = lease.trajectory_id - if not meta.get("policy_id") and expected_version is not None: - meta["policy_id"] = str(expected_version) - return meta - - -def apply_trajectory_bookkeeping(input_sample: Any, samples: list[Any], *, expected_version: str | None = None, - max_policy_age: int = 0, current_version: str | None = None) -> None: - """Siblings share ``group_index``/``rollout_id``; policy version and alignment checks. - - ``max_policy_age`` > 0 (5.2): versions are recorded as segments and checked - against ``current_version`` (the policy token current when the trajectory - finished; default ``expected_version``) instead of requiring one version.""" - group_index = getattr(input_sample, "group_index", None) - rollout_key = getattr(input_sample, "index", None) - input_meta = getattr(input_sample, "metadata", None) - input_meta = input_meta if isinstance(input_meta, dict) else {} - if expected_version is None: - expected_version = expected_policy_version(input_sample) - input_meta = produce_rlf_identity(input_meta, expected_version=expected_version) - for position, sample in enumerate(samples): - if group_index is not None: - sample.group_index = group_index - if rollout_key is not None: - sample.rollout_id = rollout_key - metadata = sample.metadata if isinstance(getattr(sample, "metadata", None), dict) else {} - metadata.setdefault("chain_index", position) - metadata["chains_total"] = len(samples) - # RLF identity (produced above for catalog rows, or written upstream) - # is copied onto every Codex chain so trusted grading and segment - # aggregation cannot depend on framework-only sample fields. - for key in RLF_IDENTITY_KEYS: - if key in input_meta: - metadata.setdefault(key, input_meta[key]) - sample.metadata = metadata - if INFRASTRUCTURE_KEY in metadata: - continue - versions = actual_versions(sample) - metadata[ACTUAL_VERSIONS_KEY] = versions - if max_policy_age > 0: - current = current_version if current_version is not None else expected_version - metadata[EXPECTED_VERSION_KEY] = current - metadata[VERSION_SEGMENTS_KEY] = sorted(set(versions)) - problem = window_problem(versions, current, max_policy_age) - if problem is not None: - _mark_aborted(sample, f"{POLICY_AGE_EXCEEDED}: {problem}") - continue - elif expected_version is not None: - metadata[EXPECTED_VERSION_KEY] = expected_version - if not versions or not all( - engine_version_matches(v, str(expected_version)) for v in versions - ): - metadata[POLICY_AGE_KEY] = 1 - _mark_aborted(sample, f"policy_age_violation: expected {expected_version}, saw {versions}") - continue - try: - assert_sample_alignment(sample) - except AlignmentError as exc: - _mark_aborted(sample, f"alignment: {exc}") - - -# --- CompactionRL segments (design D8; progress.md "S13 Codex 桥压缩拦截") ---- -SESSIONS_KEY = "codex_compaction_sessions" # trusted: pre-created by the agent function -ROUTER_KEY = "codex_compaction_router" # trusted: router those sessions live on -SEGMENT_COLLECT_TIMEOUT_ENV = "YETO_CODEX_SEGMENT_COLLECT_TIMEOUT_S" -DEFAULT_SEGMENT_COLLECT_TIMEOUT_S = 120.0 - - -def segment_collect_timeout_s() -> float: - """Per-session collect budget (env override, positive seconds).""" - raw = os.environ.get(SEGMENT_COLLECT_TIMEOUT_ENV) - try: - value = float(raw) if raw else DEFAULT_SEGMENT_COLLECT_TIMEOUT_S - except ValueError: - value = DEFAULT_SEGMENT_COLLECT_TIMEOUT_S - return value if value > 0 else DEFAULT_SEGMENT_COLLECT_TIMEOUT_S -COMPACTION_METRICS_KEY = "codex_compaction" # untrusted bridge record in agent_metrics -SegmentCollector = Callable[[Any, str, str], Awaitable[tuple[list[Any], dict[str, Any]]]] - - -async def collect_segment_session(input: Any, router: str, session_id: str) -> tuple[list[Any], dict[str, Any]]: - """Collect (and delete) one extra session exactly like upstream's tracer does (backend glue).""" - return await _glue().collect_segment_session(input, router, session_id) - - -def _optimized_tokens(sample: Any) -> int: - mask = getattr(sample, "loss_mask", None) or [] - return int(sum(1 for value in mask if value)) - - -def refresh_segment_counts(samples: Any) -> int: - """Recompute ``segment_tokens``/``tokens_after``/``gae_length`` from the final loss masks. - - The wrapper counts optimised tokens when it assembles the segments; later - hooks still set loss_mask 1 -> 0 (S19 #8 placeholder-logprob stop tokens, - S17 WP6 reasoning tokens) before Miles converts the samples. N_{>s} and l - are optimised-token counts, so they are refreshed after those hooks - (s19-compaction-g1-20261010g: tokens_after was 1-17 tokens high). - Returns the number of segment samples whose counts changed. - """ - - rollouts: dict[Any, list[Any]] = {} - for sample in samples: - meta = getattr(sample, "metadata", None) - if not isinstance(meta, dict) or not isinstance(meta.get("segment_index"), int): - continue - key = (getattr(sample, "rollout_id", None), getattr(sample, "group_index", None), - getattr(sample, "index", None)) - rollouts.setdefault(key, []).append(sample) - changed = 0 - for segments in rollouts.values(): - segments.sort(key=lambda s: s.metadata["segment_index"]) - tokens = [_optimized_tokens(s) for s in segments] - for index, sample in enumerate(segments): - new = {"segment_tokens": tokens[index], "tokens_after": sum(tokens[index + 1:]), - "gae_length": sum(tokens)} - if any(sample.metadata.get(k) != v for k, v in new.items()): - changed += 1 - sample.metadata.update(new) - return changed - - -async def assemble_compaction_segments(input: Any, output: Any, *, collect: SegmentCollector | None = None) -> Any: - """Collect the pre-created segment sessions; one sample per segment. - - Every listed session is collected (which deletes it) even when the - trajectory failed. Segment ``s`` carries the rollout's metadata (signed - outcome -> shared reward) plus ``segment_index``/``num_segments``/ - ``segment_tokens``/``tokens_after`` (N_{>s}, optimised tokens of later - segments)/``gae_length`` (optimised tokens of the whole rollout). - """ - samples = _samples_of(output) - if len(samples) != 1: - raise ValueError("CompactionRL expects one upstream sample for segment 0") - first = samples[0] - meta0 = first.metadata if isinstance(getattr(first, "metadata", None), dict) else {} - session_ids = meta0.get(SESSIONS_KEY) - if session_ids is None: - return output - collect = collect or collect_segment_session - # The agent function records the router it created the sessions on - # (s19-compaction-g1-20261010d: session_server_id is not in the returned - # metadata, the collect URL became "http:///sessions/..." and retried forever). - router = meta0.get(ROUTER_KEY) - if not router and meta0.get("session_server_id"): - router = f"http://{meta0['session_server_id']}" - if not isinstance(router, str) or not router.startswith(("http://", "https://")) or router.endswith("//"): - _mark_aborted(first, "segment collect: no session-server router for the segment sessions") - return output - collected: list[tuple[list[Any], dict[str, Any]]] = [] - failure: str | None = None - for session_id in session_ids if isinstance(session_ids, list) else []: - try: - collected.append(await asyncio.wait_for(collect(input, router, str(session_id)), - timeout=segment_collect_timeout_s())) - except asyncio.TimeoutError: - # s19-compaction-g1-20261010d: Miles' http_utils retries a failing - # collect for a long time; bound it so one rollout cannot stall the round. - failure = failure or f"segment collect: timed out after {segment_collect_timeout_s():g} s" - collected.append(([], {})) - except Exception as exc: # noqa: BLE001 - keep deleting the remaining sessions - failure = failure or f"segment collect: {type(exc).__name__}: {exc}" - collected.append(([], {})) - if INFRASTRUCTURE_KEY in meta0: - return output # already aborted; extra sessions were only drained - if not isinstance(session_ids, list): - failure = "segment sessions metadata is not a list" - record = (meta0.get("agent_metrics") or {}).get(COMPACTION_METRICS_KEY) - compactions = record.get("compactions") if isinstance(record, dict) else None - if failure is None and (type(compactions) is not int or not 0 <= compactions <= len(collected)): - failure = "segment record missing or out of range" - if failure is None: - used, unused = collected[:compactions], collected[compactions:] - if any(len(got) != 1 for got, _ in used) or any(got for got, _ in unused): - failure = f"segment sessions do not match {compactions} compactions" - if failure is not None: - _mark_aborted(first, failure) - return output - segments = [first] - for got, session_metadata in collected[:compactions]: - sample = got[0] - sample.metadata = {**meta0, **session_metadata} - segments.append(sample) - tokens = [_optimized_tokens(sample) for sample in segments] - truncated = meta0.get("exit_status") == "max_seq_len" - for index, sample in enumerate(segments): - sample.metadata.update( - { - "segment_index": index, - "num_segments": len(segments), - "segment_tokens": tokens[index], - "tokens_after": sum(tokens[index + 1 :]), - "gae_length": sum(tokens), - "compactions": compactions, - "truncated": truncated, - "chain_index": index, - "chains_total": len(segments), - "segment_id": index, - "chain_break_reason": "compaction_window" if index else None, - "segment_boundary_reason": "compaction" if index < len(segments) - 1 else None, - } - ) - if len(segments) > 1: - # GenerateFnOutput is a frozen dataclass ("multiple samples" is its documented use). - output = dataclasses.replace(output, samples=segments) if dataclasses.is_dataclass(output) else segments - return output - - -async def generate( - input: Any, - *, - upstream: Callable[[Any], Awaitable[Any]] | None = None, - collect: SegmentCollector | None = None, -) -> Any: - expected_version = expected_policy_version(input.sample) - if expected_version is None: # IR-3: refuse before any generation happens - raise PolicyVersionMissing("expected_policy_version missing: the driver published no policy token") - output = await (upstream or _load_upstream())(input) - samples = _samples_of(output) - if len(samples) == 1 and isinstance(getattr(samples[0], "metadata", None), dict) and SESSIONS_KEY in samples[0].metadata: - output = await assemble_compaction_segments(input, output, collect=collect) - limit = int(getattr(getattr(input, "args", None), "yeto_rl_max_policy_age", 0) or 0) - current = None - if limit > 0: - # 5.2: the trajectory may have been suspended across a publish; it is - # trained at the version current NOW (driver token in the metadata sink). - try: - current = rollout_meta.current_policy_token() or expected_version - except Exception: # noqa: BLE001 - unreachable sink: the start token - current = expected_version - apply_trajectory_bookkeeping(input.sample, _samples_of(output), expected_version=expected_version, - max_policy_age=limit, current_version=current) - if _compactionrl_on(): - output = _as_sample_list(output) - return output - - -def _compactionrl_on() -> bool: - from .compaction_bridge import compactionrl_enabled - - return compactionrl_enabled() - - -def _as_sample_list(output: Any) -> Any: - """CompactionRL: every rollout's output carries a *list* of samples (one per - segment, possibly one), so Miles flattens one uniform nesting level - (s19-compaction-g1-20261010e: compacted rollouts returned lists, the others a - bare Sample -> rollout_data_conversion "'Sample' object is not iterable").""" - samples = getattr(output, "samples", output) - if isinstance(samples, list): - return output - if dataclasses.is_dataclass(output) and hasattr(output, "samples"): - return dataclasses.replace(output, samples=[samples]) - return [samples] - - -def _add_arguments(parser: Any) -> None: - """Miles parse_args registers a generate function's own flags through its - ``add_arguments`` attribute; this wrapper hands the agent flags - (--custom-agent-function-path, --max-seq-len) to upstream - agentic_tool_call.generate, so it must register upstream's flags too - (s19-compaction-g1-20261010c: "unrecognized arguments").""" - upstream = _load_upstream() - add = getattr(upstream, "add_arguments", None) - if callable(add): - add(parser) - - -generate.add_arguments = _add_arguments diff --git a/yeto/rl/harness/codex/codex_openenv_subprocess_agent_function.py b/yeto/rl/harness/codex/codex_openenv_subprocess_agent_function.py deleted file mode 100644 index de052c8f6..000000000 --- a/yeto/rl/harness/codex/codex_openenv_subprocess_agent_function.py +++ /dev/null @@ -1,616 +0,0 @@ -"""Trusted-side entry for the Terminal-Bench Codex adapter (REWRITE). - -``run`` is the ``--custom-agent-function-path`` target under upstream -``agentic_tool_call.generate``. It owns the environment lease, spawns the -untrusted worker with the reward key scrubbed from its environment, maps the -worker's tool-wait events onto the ToolWaitBoard (3.1), runs the verifier and -signs the outcome (D7), and guarantees cleanup on cancellation/timeouts: -worker process group, environment lease, tool-wait scope. - -Environment provisioning is injected (``EnvironmentProvider``) so the sandbox -backend (D9) and the CPU fake share one code path. The boards are installed -by the IR-1 preflight hook (``preflight.harness_preflight``): the island's -``ToolWaitBoard`` (3.1) and ``HarnessBoard`` (IR-2) actors. IR-2 wiring here: -``allow_new_session`` before any environment is acquired (drain closes -admission; fail closed -> infrastructure ABORTED, never a 0 reward), -``enter_session``/``exit_session`` around the worker, ``lease_acquired`` -(with the hard deadline) / ``lease_released`` around the environment lease so -``env_live`` counts idle sandboxes too. IR-3: the target policy token comes -from ``rollout_meta_hook.expected_policy_version`` (prompt metadata first, -else the driver's token in the metadata sink); a missing token refuses the -trajectory before any environment is acquired. -""" - -from __future__ import annotations - -import asyncio -import json -import os -import signal -import sys -import time -from dataclasses import dataclass -from types import SimpleNamespace -from typing import Any, Awaitable, Callable, Protocol - -from yeto.rl.engine import rollout_meta -from yeto.rl.engine.tool_wait import _call as _board_call -from yeto.rl.engine.trajectory_timing import PhaseClock -from yeto.rl.engine.tool_wait import _resolve as _board_resolve - -from . import codex_openenv_agent_function as adapter -from .environment import TerminalEnvironment, TrustedVerifier - -WORKER_MODULE = "yeto.rl.harness.codex.codex_openenv_agent_worker" -WORKER_GRACE_SECONDS = 5.0 - - -@dataclass -class EnvironmentLease: - """One environment bound to one trajectory (R-D9: deadline is a hard cap).""" - - env_url: str - env_token: str - verifier: TrustedVerifier - destroy: Callable[[], Awaitable[None]] - describe: Callable[[], Awaitable[str]] # "live" | "gone" - deadline_seconds: float - # Extra environment for the worker process (e.g. a per-trajectory - # ``SECRLENV_MAX_TURNS`` turn budget injected by the provider); never a - # reward key (the worker refuses to start with one). - worker_env: dict[str, str] | None = None - - -class EnvironmentProvider(Protocol): - async def acquire(self, task_id: str, trajectory_id: str) -> EnvironmentLease: ... - - -_provider: EnvironmentProvider | None = None -_tool_wait_board: Any = None -_harness_board: Any = None -_member: str | None = None - -ADMISSION_CLOSED = "harness_admission_closed" -# Consecutive non-injected acquire failures before the island is torn down. -# Each failure is reported as INFRASTRUCTURE (sample ABORTED); upstream -# Miles then draws the next group forever (``generate_rollout`` has no -# abort cap: codex-smoke-20261003-7 looped 17k samples in six minutes on a -# broken provider), so after this many the worker raises -# ``EnvironmentProviderOutage`` instead. -PROVIDER_OUTAGE_THRESHOLD_ENV = "YETO_HARNESS_PROVIDER_OUTAGE_THRESHOLD" -DEFAULT_PROVIDER_OUTAGE_THRESHOLD = 4 -_acquire_failures = 0 - - -class EnvironmentProviderOutage(BaseException): - """The environment provider failed repeatedly: stop the rollout, do not resample. - - A ``BaseException`` on purpose: upstream Miles wraps the agent function and - each rollout task in ``except Exception`` and keeps generating, so only a - BaseException escapes ``generate_rollout`` and fails the island. - """ - - -def _provider_outage_threshold() -> int: - try: - return max(1, int(os.environ.get(PROVIDER_OUTAGE_THRESHOLD_ENV) or DEFAULT_PROVIDER_OUTAGE_THRESHOLD)) - except ValueError: - return DEFAULT_PROVIDER_OUTAGE_THRESHOLD - - -def _note_acquire_failure(exc: BaseException, trajectory_id: str) -> None: - """Count real provisioning failures; injected ones (fault tests) do not count.""" - global _acquire_failures - if getattr(exc, "injected_fault", False): - return - _acquire_failures += 1 - threshold = _provider_outage_threshold() - if _acquire_failures >= threshold: - raise EnvironmentProviderOutage( - f"environment provider failed {_acquire_failures} consecutive acquires " - f"(threshold {threshold}); last for {trajectory_id!r}: {type(exc).__name__}: {exc}" - ) from exc - - -def _note_acquire_success() -> None: - global _acquire_failures - _acquire_failures = 0 -POLICY_VERSION_MISSING = "expected_policy_version missing" - - -class PolicyVersionMissing(RuntimeError): - """IR-3: the driver published no target policy token for this rollout (refuse).""" - - -def configure( - *, - provider: EnvironmentProvider | None, - tool_wait_board: Any = None, - harness_board: Any = None, - member: str | None = None, -) -> None: - """Install the environment provider and the island boards. - - ``tool_wait_board``: ``ToolWaitBoard`` (local or actor handle, 3.1). - ``harness_board``: ``HarnessBoard`` (local or actor handle, IR-2); None - means the island reports "harness counts unknown" and stays undrainable. - ``member``: the rollout member this process targets (admission key). - """ - global _provider, _tool_wait_board, _harness_board, _member, _acquire_failures - _acquire_failures = 0 - _provider = provider - _tool_wait_board = tool_wait_board - _harness_board = harness_board - _member = member - - -def configured() -> dict[str, Any]: - return {"provider": _provider, "tool_wait_board": _tool_wait_board, "harness_board": _harness_board, - "member": _member} - - -async def _board_await(target: Any, method: str, *args: Any) -> None: - """Call a board method and wait for it, so consecutive calls stay ordered. - - The island boards are Ray actors with ``max_concurrency=64``: back-to-back - fire-and-forget ``exit``/``enter`` of one trajectory's successive tool - calls ran out of order ("already in a tool call" / "is not in a tool call", - A-T3-7, codex-smoke-20261003-11). Board bookkeeping is telemetry, so its - errors are logged and never fail the trajectory. - """ - try: - value = _board_call(target, method, *args) - if type(value).__name__ == "ObjectRef": - await asyncio.to_thread(_board_resolve, value) - except Exception as exc: # noqa: BLE001 - never fail a trajectory on telemetry - print(f"[codex-harness] tool-wait board {method} failed: {type(exc).__name__}: {exc}", file=sys.stderr) - - -def _board_kwcall(target: Any, method: str, *args: Any, **kwargs: Any) -> Any: - """Like ``tool_wait._call`` but forwards keyword arguments (``lease_acquired(deadline=)``).""" - fn = getattr(target, method) - remote = getattr(fn, "remote", None) - return _board_resolve(remote(*args, **kwargs) if callable(remote) else fn(*args, **kwargs)) - - -def resolve_expected_policy_version(metadata: dict[str, Any]) -> str | None: - """IR-3 target token: prompt metadata first, else the driver token in the sink.""" - try: - return rollout_meta.expected_policy_version(SimpleNamespace(metadata=metadata)) - except Exception: # noqa: BLE001 - no sink reachable == no token published - return None - - -async def _terminate(process: asyncio.subprocess.Process) -> None: - if process.returncode is not None: - return - for sig, wait in ((signal.SIGTERM, WORKER_GRACE_SECONDS), (signal.SIGKILL, WORKER_GRACE_SECONDS)): - try: - os.killpg(process.pid, sig) - except ProcessLookupError: - return - # asyncio.wait (not wait_for): must work inside a cancelled task's finally. - waiter = asyncio.ensure_future(process.wait()) - done, _pending = await asyncio.wait({waiter}, timeout=wait) - if done: - return - waiter.cancel() - - -async def _drive_worker( - job: dict[str, Any], trajectory_id: str, board: Any, worker_env: dict[str, str] | None = None -) -> dict[str, Any]: - """Spawn the worker, relay tool-wait events, return its result event.""" - process = await asyncio.create_subprocess_exec( - sys.executable, - "-m", - WORKER_MODULE, - stdin=asyncio.subprocess.PIPE, - stdout=asyncio.subprocess.PIPE, - stderr=asyncio.subprocess.DEVNULL if os.getenv("YETO_CODEX_WORKER_STDERR") != "1" else None, - env={**os.environ, **{str(k): str(v) for k, v in (worker_env or {}).items()}}, - start_new_session=True, - ) - in_tool = False - phases = PhaseClock() # agentic-rollout-utilization 1.3 - try: - assert process.stdin is not None and process.stdout is not None - process.stdin.write((json.dumps(job) + "\n").encode()) - await process.stdin.drain() - process.stdin.close() - while True: - line = await process.stdout.readline() - if not line: - raise adapter.harness.CodexHarnessError("worker exited without a result") - event = json.loads(line) - kind = event.get("event") - if kind == "tool_wait": - if event.get("phase") == "enter" and not in_tool: - in_tool = True - phases.enter_tool() - if board is not None: - await _board_await(board, "enter", trajectory_id) - elif event.get("phase") == "exit" and in_tool: - in_tool = False - phases.exit_tool() - if board is not None: - await _board_await(board, "exit", trajectory_id) - elif kind == "suspend": # agentic-rollout-utilization 5.1 - if event.get("phase") == "enter": - phases.enter_suspend() - elif event.get("phase") == "exit": - phases.exit_suspend() - elif kind == "result": - if isinstance(event.get("metrics"), dict): # observe only, unsigned - event["metrics"].update(phases.finish()) - return event - elif kind == "error": - error = adapter.harness.CodexHarnessError(str(event.get("reason"))) - if isinstance(event.get("metrics"), dict): - error.metrics = event["metrics"] # type: ignore[attr-defined] - raise error - finally: - if in_tool and board is not None: - await _board_await(board, "exit", trajectory_id) - await _terminate(process) - - -class TaskPromptMissing(ValueError): - """No task statement could be resolved for the episode (fail closed).""" - - -def _last_user_text(prompt: Any) -> str | None: - if not isinstance(prompt, list): - return None - for message in reversed(prompt): - if isinstance(message, dict) and message.get("role") == "user": - content = message.get("content") - if isinstance(content, str) and content.strip(): - return content - return None - - -def task_prompt(metadata: dict[str, Any], prompt: Any, lease: Any) -> str: - """The task statement Codex receives as its first user message. - - Precedence: explicit ``metadata["prompt"]`` string; the provider task's - ``instruction.md``; a plain-string Miles prompt; the last user message of a - chat-format prompt. A chat prompt without a user message (e.g. only a - generic system message) is refused instead of being stringified -- S15 - stage 2 sent ``str([{'role': 'system', ...}])`` to every trajectory, so the - model never saw the task and all 48 rewards were 0. - """ - explicit = metadata.get("prompt") - if isinstance(explicit, str) and explicit.strip(): - return explicit - task = getattr(lease, "task", None) - instruction = getattr(task, "instruction", None) - if isinstance(instruction, str) and instruction.strip(): - return instruction - if isinstance(prompt, str) and prompt.strip(): - return prompt - user = _last_user_text(prompt) - if user is not None: - return user - raise TaskPromptMissing( - f"no task statement for task {metadata.get('task_id')!r}: metadata has no prompt, " - "the environment task has no instruction.md and the Miles prompt has no user message" - ) - - -# S17 N17 (over-sampling A/B): in-flight trajectories of this rollout process. -# Miles calls ``.abort(args)`` (``call_agent_abort_hook``) once -# enough groups are collected; without it the Codex loops keep issuing fresh -# turns after SGLang's abort_all and the rollout waits for every surplus group. -# Each entry: ``stage`` (acquire / segments / worker / verify / teardown), the -# abortable step's task (None in acquire and teardown) and the start time. -# acquire is never cancelled (a sandbox created by a cancelled acquire would -# leak): the trajectory is flagged and stops right after acquire returns; the -# other steps are cancelled and the lease is destroyed in ``run``'s finally. -_INFLIGHT: dict[str, dict[str, Any]] = {} -_ABORTED_IDS: set[str] = set() -ABORTED_STATUS = "aborted_surplus" -LEASE_EXPIRED_ON_BOARD = "lease force-released on the harness board before release" - - -class _AbortedSurplus(Exception): - """abort() stopped this trajectory (surplus of an over-sampled rollout).""" - - -class _LeaseExpiredOnBoard(Exception): - """The board force-released the lease at its hard deadline before ``run`` released it.""" - - -def _stage(trajectory_id: str, stage: str) -> None: - entry = _INFLIGHT.get(trajectory_id) - if entry is not None: - entry["stage"] = stage - - -async def _abortable(trajectory_id: str, stage: str, awaitable: Any) -> Any: - """Run one step of ``run`` as a task abort() may cancel; a cancel that abort() - issued surfaces as :class:`_AbortedSurplus` (any other cancel propagates).""" - if trajectory_id in _ABORTED_IDS: - if asyncio.iscoroutine(awaitable): - awaitable.close() - raise _AbortedSurplus(stage) - task = asyncio.ensure_future(awaitable) - entry = _INFLIGHT.get(trajectory_id) - if entry is not None: - entry.update(stage=stage, task=task) - try: - return await task - except asyncio.CancelledError: - if trajectory_id in _ABORTED_IDS and task.cancelled(): - raise _AbortedSurplus(stage) from None - raise - finally: - if entry is not None: - entry["task"] = None - - -async def abort(args: Any = None) -> int: - """Stop every in-flight trajectory of this process: cancel its current step - (segments / worker / verify) or, while it is still acquiring its sandbox, - flag it so it stops as soon as the sandbox exists. Leases are destroyed in - ``run``'s finally. Returns how many trajectories were stopped and logs one - ``rl_codex_abort`` line (count per stage, elapsed seconds).""" - now = time.monotonic() - stages: dict[str, int] = {} - elapsed = [] - for tid, entry in list(_INFLIGHT.items()): - if entry.get("stage") == "teardown": - continue - _ABORTED_IDS.add(tid) - task = entry.get("task") - if task is not None and not task.done(): - task.cancel() - stage = entry.get("stage", "?") - stages[stage] = stages.get(stage, 0) + 1 - elapsed.append(round(now - entry["t0"], 2)) - if elapsed: - print("YETO_CODEX_ABORT " + json.dumps({ - "event": "rl_codex_abort", "cancelled": len(elapsed), "stages": stages, - "elapsed_s": sorted(elapsed)}), flush=True) - return len(elapsed) - - -# agentic-rollout-utilization 5.1/5.3: suspension between model turns -# (Miles --agentic-suspend-between-turns calls ``resume`` at the start of every -# rollout and ``suspend`` at its cut-off). Each trajectory's worker gets its own -# gate file (``YETO_CODEX_SUSPEND_GATE``); ``suspend`` creates the files of the -# trajectories in flight (closed gate: the worker's bridge holds the next model -# request -- a running tool call is not interrupted), ``resume`` removes them. -# A trajectory parked longer than ``YETO_CODEX_SUSPEND_MAX_SECONDS`` (600 s) is -# discarded by its bridge (CodexSuspendExpired -> infrastructure) and its lease -# is destroyed in ``run``'s finally. -_SUSPEND_DIR: str | None = None -_SUSPEND_STATE: dict[str, Any] = {"suspended_at": None, "gates": {}} -SUSPEND_EXPIRED = "CodexSuspendExpired" - - -def _gate_path(trajectory_id: str) -> str | None: - if _SUSPEND_DIR is None: - return None - safe = "".join(c if c.isalnum() or c in "-_." else "_" for c in trajectory_id) - return os.path.join(_SUSPEND_DIR, f"{safe}.gate") - - -async def suspend(args: Any = None) -> int: - """Close the model-turn gate of every trajectory in flight; returns how many.""" - global _SUSPEND_DIR - if _SUSPEND_DIR is None: - import tempfile - - _SUSPEND_DIR = tempfile.mkdtemp(prefix="yeto-codex-suspend-") - gates = _SUSPEND_STATE["gates"] - stages: dict[str, int] = {} - for tid, entry in list(_INFLIGHT.items()): - if entry.get("stage") == "teardown": - continue - path = _gate_path(tid) - with open(path, "w", encoding="utf-8") as fh: - fh.write("suspended\n") - gates[tid] = path - stage = entry.get("stage", "?") - stages[stage] = stages.get(stage, 0) + 1 - _SUSPEND_STATE["suspended_at"] = time.time() - print("YETO_CODEX_SUSPEND " + json.dumps({ - "event": "rl_codex_suspend", "suspended": len(gates), "stages": stages}), flush=True) - return len(gates) - - -async def resume(args: Any = None) -> int: - """Open every closed gate (start of a rollout); returns how many were closed. - The first call also turns the gates on for the trajectories started after it.""" - global _SUSPEND_DIR - if _SUSPEND_DIR is None: - import tempfile - - _SUSPEND_DIR = tempfile.mkdtemp(prefix="yeto-codex-suspend-") - gates = _SUSPEND_STATE["gates"] - opened = 0 - for tid, path in list(gates.items()): - try: - os.unlink(path) - opened += 1 - except FileNotFoundError: - pass - gates.pop(tid, None) - since = _SUSPEND_STATE.get("suspended_at") - _SUSPEND_STATE["suspended_at"] = None - if opened or since is not None: - print("YETO_CODEX_RESUME " + json.dumps({ - "event": "rl_codex_resume", "resumed": opened, - "suspended_s": None if since is None else round(time.time() - since, 3)}), flush=True) - return opened - - -def _suspend_worker_env(trajectory_id: str) -> dict[str, str]: - path = _gate_path(trajectory_id) - if path is None: - return {} # suspension not in use: worker environment unchanged - if _SUSPEND_STATE.get("suspended_at") is not None and trajectory_id not in _SUSPEND_STATE["gates"]: - # started while the rollout is suspended (should not happen: Miles holds - # new samples) -- park it too rather than generate on offloaded engines - with open(path, "w", encoding="utf-8") as fh: - fh.write("suspended\n") - _SUSPEND_STATE["gates"][trajectory_id] = path - return {adapter.harness.SUSPEND_GATE_ENV: path} - - -async def run( - base_url: str, - prompt: Any, - request_kwargs: dict[str, Any] | None = None, - metadata: dict[str, Any] | None = None, - **kwargs: Any, -) -> dict[str, Any] | None: - metadata = dict(metadata or {}) - try: - return await _run(base_url, prompt, request_kwargs, metadata, **kwargs) - except _LeaseExpiredOnBoard as exc: - # The trajectory outlived its lease's hard deadline (e.g. a detached - # Modal sandbox) and the board already force-released the lease: - # infrastructure, never a reward, and never a failed rollout (S17 N17 A/B a). - _task_id, sample_id = adapter.task_identity(metadata) - trajectory_id = str(metadata.get("trajectory_id") or sample_id) - return {**adapter.infrastructure_metadata(f"{LEASE_EXPIRED_ON_BOARD}: {exc}", episode_id=None), - **adapter.trajectory_fields(trajectory_id)} - - -async def _run( - base_url: str, - prompt: Any, - request_kwargs: dict[str, Any] | None, - metadata: dict[str, Any], - **_kwargs: Any, -) -> dict[str, Any] | None: - if _provider is None: - # Rollout workers are separate Ray actors: the driver's configure() - # never ran here, so build the same wiring from the forwarded env. - from .preflight import configure_rollout_worker - - configure_rollout_worker() - if _provider is None: - raise RuntimeError("codex_openenv_subprocess_agent_function.configure(provider=...) was not called") - task_id, sample_id = adapter.task_identity(metadata) - trajectory_id = str(metadata.get("trajectory_id") or sample_id) - episode_id = adapter.new_episode_id() - fields = adapter.trajectory_fields(trajectory_id) - started_at = time.time() # agentic-rollout-utilization 1.3 (observe only) - expected_version = resolve_expected_policy_version(metadata) - if expected_version is None: # IR-3: refuse before any environment exists - raise PolicyVersionMissing(f"{POLICY_VERSION_MISSING} for trajectory {trajectory_id!r}") - board = _harness_board - if board is not None and not _board_kwcall(board, "allow_new_session", _member): - # IR-2: admission closed (drain of this member) -> infrastructure, not a reward. - return {**adapter.infrastructure_metadata(ADMISSION_CLOSED, episode_id=None), **fields} - lease: EnvironmentLease | None = None - session_open = lease_open = handed_off = False - _INFLIGHT[trajectory_id] = {"stage": "acquire", "task": None, "t0": time.monotonic()} - try: - if board is not None: - _board_kwcall(board, "enter_session", trajectory_id, _member) - session_open = True - try: - acquire_started = time.monotonic() - lease = await _provider.acquire(task_id, trajectory_id) - sandbox_start_seconds = round(time.monotonic() - acquire_started, 3) - except Exception as exc: # noqa: BLE001 - provisioning failures are infrastructure - _note_acquire_failure(exc, trajectory_id) # raises EnvironmentProviderOutage past the threshold - return {**adapter.infrastructure_metadata(f"acquire: {type(exc).__name__}: {exc}", episode_id=None), **fields} - _note_acquire_success() - if trajectory_id in _ABORTED_IDS: - # abort() during acquire: never started, the sandbox is destroyed below - return {**adapter.infrastructure_metadata(ABORTED_STATUS, episode_id=None), **fields} - if board is not None: - _board_kwcall(board, "lease_acquired", trajectory_id, deadline=time.monotonic() + lease.deadline_seconds) - lease_open = True - job = { - "base_url": base_url, - "prompt": task_prompt(metadata, prompt, lease), - "request_kwargs": dict(request_kwargs or {}), - "episode_id": episode_id, - "max_seq_len": metadata.get("max_seq_len"), - "env_url": lease.env_url, - "env_token": lease.env_token, - "driver": metadata.get("codex_openenv_driver", "stock"), - **{k: metadata[k] for k in ("script", "final_status", "hang_seconds") if k in metadata}, - } - try: - segments = await _abortable(trajectory_id, "segments", adapter.prepare_segment_sessions(job)) - except _AbortedSurplus: - return {**adapter.infrastructure_metadata(ABORTED_STATUS, episode_id=episode_id), **fields} - except Exception as exc: # noqa: BLE001 - session-server failures are infrastructure - return {**adapter.infrastructure_metadata(f"segment sessions: {type(exc).__name__}: {exc}", episode_id=episode_id), **fields} - try: - try: - untrusted = await _abortable(trajectory_id, "worker", asyncio.wait_for( - _drive_worker(job, trajectory_id, _tool_wait_board, - {**(getattr(lease, "worker_env", None) or {}), - **_suspend_worker_env(trajectory_id)}), - timeout=lease.deadline_seconds)) - except _AbortedSurplus: - # abort(): surplus trajectory of an over-sampled rollout; Miles - # discards it (no partial rollout). Infrastructure, never a reward. - return {**adapter.infrastructure_metadata(ABORTED_STATUS, episode_id=episode_id), **fields} - except asyncio.TimeoutError: - untrusted = {"status": "timeout", "metrics": {"timed_out": 1}, "episode_id": episode_id} - except adapter.harness.CodexHarnessError as exc: - metrics = getattr(exc, "metrics", None) - tito = adapter.mirror_tito_counters(metrics, board) - handed_off = True - return {**adapter.infrastructure_metadata(str(exc), episode_id=episode_id, metrics=metrics), **fields, **tito, **segments} - except Exception as exc: # noqa: BLE001 - e.g. the sandbox shut down under the worker - # Environment failure, not a policy outcome: infrastructure (no reward, group dropped) - # instead of an exception that fails the whole rollout step. The finally block - # deletes the pre-created sessions (not handed off). - return {**adapter.infrastructure_metadata(f"worker: {type(exc).__name__}: {exc}", episode_id=episode_id), **fields} - tito = adapter.mirror_tito_counters(untrusted.get("metrics"), board) - try: - signed = await _abortable(trajectory_id, "verify", adapter.finish_trusted( - untrusted, lease.verifier, task_id=task_id, sample_id=sample_id)) - except _AbortedSurplus: - return {**adapter.infrastructure_metadata(ABORTED_STATUS, episode_id=episode_id), **fields} - except Exception as exc: # noqa: BLE001 - verifier sandbox gone (Modal ConflictError / NotFoundError) - return {**adapter.infrastructure_metadata(f"verify: {type(exc).__name__}: {exc}", episode_id=episode_id), **fields} - signed["expected_policy_version"] = expected_version - signed.update(trajectory_started_at=started_at, trajectory_ended_at=time.time(), - sandbox_start_seconds=sandbox_start_seconds) - handed_off = True - return {**signed, **fields, **tito, **segments} - finally: - # Worker crash (any non-harness exception), cancellation or a - # finish_trusted error: the metadata never reaches - # codex_openenv_generate, so delete the pre-created sessions here. - if segments and not handed_off: - await adapter.release_unreturned_segments(base_url, segments) - finally: - _stage(trajectory_id, "teardown") - try: - if lease is not None: - await lease.destroy() - if await lease.describe() != "gone": - # Not confirmed gone: keep the lease on the board; it is - # force-released at its hard deadline (leases_expired_total). - raise RuntimeError(f"environment for {trajectory_id} was not confirmed destroyed") - if lease_open: - try: - _board_kwcall(board, "lease_released", trajectory_id) - except Exception as exc: # noqa: BLE001 - ray wraps the board's ToolWaitError - if "is not live" not in str(exc): - raise - # the board force-released it at its hard deadline already - raise _LeaseExpiredOnBoard(str(exc).splitlines()[-1][:300]) from exc - finally: - _INFLIGHT.pop(trajectory_id, None) - _ABORTED_IDS.discard(trajectory_id) - gate = _SUSPEND_STATE["gates"].pop(trajectory_id, None) - if gate is not None: - try: - os.unlink(gate) - except FileNotFoundError: - pass - if session_open: - _board_kwcall(board, "exit_session", trajectory_id) diff --git a/yeto/rl/harness/codex/compaction_bridge.py b/yeto/rl/harness/codex/compaction_bridge.py deleted file mode 100644 index 90a02a18a..000000000 --- a/yeto/rl/harness/codex/compaction_bridge.py +++ /dev/null @@ -1,379 +0,0 @@ -"""CompactionRL context compaction inside the Codex Responses bridge (design D8). - -Opt-in only (``YETO_CODEX_COMPACTIONRL=1``); with it unset nothing here runs and -the stock path (``codex_harness_agent._drive_codex``) is used unchanged. - -Why the bridge can rewrite the context although Codex keeps the full history: -the bridge never forwards Codex's Responses ``input``. It checks that input is -exactly the expected append-only history (``_expected_input``) and samples Miles -from its own Chat Completions message list (``_messages``). Codex requests use -``store:false`` and no ``previous_response_id`` (both enforced by -``_validate_codex_request``), so Codex is never told about the compaction: its -own history stays complete and append-only, while each Miles request carries -only the rebuilt context ``s + u_resume(S) + last k steps``. - -Training shape (progress.md "S13 cross_segment 每段 sample"): one sample per -segment. The fork-pinned v1 session server is append-only and would silently -roll back a rewritten history (R-D5a), so every segment uses its own session: -the summary is sampled in the current session (trainable, logprobs recorded by -the session server like any other turn), then the rebuilt context starts in the -next pre-created session. The trusted parent pre-creates those sessions -(``codex_openenv_agent_function.create_segment_sessions``) and -``codex_openenv_generate`` collects them and writes segment metadata. - -Codex 0.145.0's own auto-compaction is switched off explicitly -(``model_auto_compact_token_limit`` = i64 max); any Codex-side compaction still -fails closed in the stock bridge (history item types / ``thread/compacted``). -""" - -from __future__ import annotations - -import asyncio -import copy -import json -import os -import tempfile -from collections.abc import Mapping, Sequence -from pathlib import Path -from typing import Any, Callable, Self - -from yeto.rl.compaction import ( - MAX_COMPACTIONS, - RESUME_TEMPLATE, - SUMMARY_PROMPT, - T_COMP, - CompactionConfig, - extract_summary, - should_compact, -) - -from . import codex_harness_agent as harness -from . import agent as legacy - -COMPACTIONRL_ENV = "YETO_CODEX_COMPACTIONRL" -T_COMP_ENV = "YETO_CODEX_COMPACTIONRL_T_COMP" -# Explicit compaction budget registration: when set, the named profile fully -# determines the CompactionConfig (its context_budget must equal max_seq_len, -# which the bridge enforces anyway); unset keeps the max_seq_len+T_COMP -# derivation below. Starts with YETO_CODEX_ so workers inherit it. -PROFILE_ENV = "YETO_CODEX_COMPACTIONRL_PROFILE" -SEGMENT_URLS_KEY = "segment_base_urls" -METRICS_KEY = "codex_compaction" -SESSIONS_METADATA_KEY = "codex_compaction_sessions" -# Router URL the segment sessions were created on (trusted, next to the ids); -# the generate wrapper collects them there. -SESSIONS_ROUTER_METADATA_KEY = "codex_compaction_router" -# Codex 0.145.0 accepts this key under --strict-config (checked against the -# pinned binary: a string value fails with "expected i64"). -CODEX_AUTO_COMPACT_DISABLED = "model_auto_compact_token_limit=9223372036854775807" -# Summary request tool table (design D8 "摘要请求工具表"): explicit choice to -# send NO tool table -- exactly the stock ``codex_harness_agent._sample_miles -# (summary=True)`` payload (tools=[], tool_choice="none") -- so the policy -# cannot call a tool while summarising. Risk, unverified until GPU: the -# execution turns carry the Codex tool table, so the summary prompt renders a -# different system/tools prefix and TITO prefix reuse on the session server may -# break or re-tokenise. Changing this needs a stock-sampler change; tests pin -# that the wire payload matches these constants. -SUMMARY_REQUEST_TOOLS: tuple[Any, ...] = () -SUMMARY_REQUEST_TOOL_CHOICE = "none" -SUMMARY_PROMPT_TOKEN_RESERVE = len(SUMMARY_PROMPT.encode("utf-8")) + 256 -_MESSAGE_TOKEN_OVERHEAD = 256 -_FALSE = frozenset({"", "0", "false", "no", "off"}) -_TRUE = frozenset({"1", "true", "yes", "on"}) - - -def compactionrl_enabled(env: Mapping[str, str] | None = None) -> bool: - raw = (os.environ if env is None else env).get(COMPACTIONRL_ENV, "").strip().lower() - if raw in _FALSE: - return False - if raw in _TRUE: - return True - raise harness.CodexHarnessError(f"{COMPACTIONRL_ENV} must be a boolean flag") - - -_COMPACTION_PROFILES: dict[str, Callable[[], CompactionConfig]] = {} - - -def register_compaction_profile(name: str, factory: Callable[[], CompactionConfig]) -> None: - """Register a named, explicit compaction budget selectable via PROFILE_ENV.""" - if not name or not callable(factory): - raise harness.CodexHarnessError("compaction profiles need a name and a factory") - _COMPACTION_PROFILES[name] = factory - - -def compaction_profiles() -> tuple[str, ...]: - return tuple(sorted(_COMPACTION_PROFILES)) - - -def compaction_config(max_seq_len: int, env: Mapping[str, str] | None = None) -> CompactionConfig: - env = os.environ if env is None else env - name = env.get(PROFILE_ENV, "").strip() - if name: - factory = _COMPACTION_PROFILES.get(name) - if factory is None: - raise harness.CodexHarnessError( - f"{PROFILE_ENV}={name!r} is not a registered compaction profile " - f"(registered: {', '.join(compaction_profiles()) or 'none'})" - ) - cfg = factory() - if cfg.context_budget != int(max_seq_len): - raise harness.CodexHarnessError( - f"compaction profile {name!r} fixes context_budget={cfg.context_budget}; " - f"max_seq_len is {max_seq_len} (the bridge requires them equal)" - ) - return cfg - raw = env.get(T_COMP_ENV, "").strip() - t_comp = T_COMP - if raw: - if not raw.isdigit() or int(raw) <= 0: - raise harness.CodexHarnessError(f"{T_COMP_ENV} must be a positive integer") - t_comp = int(raw) - try: - return CompactionConfig(context_budget=int(max_seq_len), t_comp=t_comp) - except ValueError as exc: - raise harness.CodexHarnessError(f"CompactionRL config rejected: {exc}") from exc - - -def qwen3_cyber_compaction_config() -> CompactionConfig: - """Registered fallback for long cyber traces after a 32K baseline truncates. - - Keep 4K tokens available for the next response and permit at most three - segment boundaries; segment identity and shared reward are preserved by - ``assemble_compaction_segments``. Select it on the run path with - ``YETO_CODEX_COMPACTIONRL_PROFILE=qwen3_cyber`` together with - ``--max-seq-len 24576`` (spec 5.7: the Qwen budget is explicit, never a - default of an unregistered model). - """ - return CompactionConfig(context_budget=24_576, t_comp=4_096, max_compactions=3) - - -register_compaction_profile("qwen3_cyber", qwen3_cyber_compaction_config) - - -def _message_bound(message: Mapping[str, Any]) -> int: - """Upper bound on a chat message's tokens: bytes + template overhead.""" - return len(json.dumps(message, ensure_ascii=False).encode("utf-8")) + _MESSAGE_TOKEN_OVERHEAD - - -def compaction_counters(metrics: Any) -> dict[str, Any]: - record = getattr(metrics, METRICS_KEY, None) - return {METRICS_KEY: copy.deepcopy(record)} if isinstance(record, dict) else {} - - -class CompactionRLBridge(harness._ResponsesBridge): - """Stock bridge + D8 trigger/summary/rebuild, one session per segment.""" - - def __init__( - self, - miles_base_url: str, - prompt: str, - request_kwargs: dict[str, Any], - metrics: legacy.AgentMetrics, - *, - max_seq_len: int | None, - segment_base_urls: Sequence[str], - config: CompactionConfig | None = None, - ) -> None: - super().__init__( - miles_base_url, prompt, request_kwargs, metrics, max_seq_len=max_seq_len - ) - if self._compaction_enabled: - raise harness.CodexHarnessError( - "legacy YETO_CODEX_COMPACTION_ENABLED and CompactionRL are exclusive" - ) - if max_seq_len is None: - raise harness.CodexHarnessError("CompactionRL requires max_seq_len (C)") - self._cfg = config or compaction_config(max_seq_len) - if self._cfg.context_budget != max_seq_len: - raise harness.CodexHarnessError("CompactionRL budget must equal max_seq_len") - if len(segment_base_urls) != self._cfg.max_compactions: - raise harness.CodexHarnessError( - "CompactionRL needs one pre-created session per allowed compaction" - ) - self._segment_urls = [ - f"{legacy._session_url(url).rstrip('/')}/chat/completions" - for url in segment_base_urls - ] - # Paper sec. 5.1: one reply cap for every turn, summaries included. - self._compaction_summary_max_tokens = self._requested_max_tokens - self._max_compactions = self._cfg.max_compactions - self._step_bounds: list[int] = [] - self._record: dict[str, Any] = { - "schema": 1, - "context_budget": self._cfg.context_budget, - "t_comp": self._cfg.t_comp, - "max_compactions": self._cfg.max_compactions, - "compactions": 0, - "segments": [], - } - setattr(metrics, METRICS_KEY, self._record) - - # --- stock hooks ----------------------------------------------------- - def _compaction_headers(self) -> dict[str, str]: - return {} # the pinned session server has no compaction headers (R-D5a) - - def _should_compact(self, estimated_context_tokens: int) -> bool: - return ( - self._compaction_count < self._cfg.max_compactions - and bool(self._atomic_steps) - and self._pending is None - and should_compact(estimated_context_tokens, self._cfg) - ) - - def expect_tool_output(self, call_id: str, output: str) -> None: - super().expect_tool_output(call_id, output) - assistant, tool = self._atomic_steps[-1] - self._step_bounds.append(_message_bound(assistant) + _message_bound(tool)) - - def _rebuild_compacted_messages(self, retained_step_count: int) -> None: - if self._resume_summary is None: - raise harness.CodexHarnessError("compaction resume summary is missing") - retained = self._atomic_steps[-retained_step_count:] if retained_step_count else [] - bounds = self._step_bounds[-retained_step_count:] if retained_step_count else [] - messages: list[dict[str, Any]] = [ - {"role": "system", "content": harness.base_instructions()}, - {"role": "user", "content": RESUME_TEMPLATE.format(summary=self._resume_summary)}, - ] - for assistant, tool in retained: - messages.extend((copy.deepcopy(assistant), copy.deepcopy(tool))) - self._messages = messages - self._atomic_steps = copy.deepcopy(retained) - self._step_bounds = list(bounds) - self._retained_step_count = retained_step_count - if self._record["segments"]: - self._record["segments"][-1]["kept_steps"] = retained_step_count - # Nothing of the new window is tokenized yet: bound it from above. - self._current_context_tokens = 0 - self._unobserved_tool_token_upper_bound = sum(_message_bound(m) for m in messages) - - def _choose_kept_steps(self, summary: str) -> int: - """Eq. 9: largest k <= cfg.k whose rebuilt context does not trigger again.""" - head = _message_bound({"role": "system", "content": harness.base_instructions()}) + _message_bound( - {"role": "user", "content": RESUME_TEMPLATE.format(summary=summary)} - ) - for k in range(min(self._cfg.keep_recent_steps, len(self._atomic_steps)), 0, -1): - if not should_compact(head + sum(self._step_bounds[-k:]), self._cfg): - return k - return 0 - - async def _compact_context(self) -> None: - if self._pending is not None or self._last_assistant_message is not None: - raise harness.CodexHarnessError("compaction attempted inside an atomic tool step") - used = self._current_context_tokens + self._unobserved_tool_token_upper_bound - if self._cfg.context_budget - used - SUMMARY_PROMPT_TOKEN_RESERVE <= 0: - raise harness.CodexSequenceLimit("no room left for the compaction summary") - # The stock sampler reserves the legacy instruction size; top it up to ours. - self._unobserved_tool_token_upper_bound += max( - 0, SUMMARY_PROMPT_TOKEN_RESERVE - harness.COMPACTION_SUMMARY_INSTRUCTION_TOKEN_RESERVE - ) - summary_messages = copy.deepcopy(self._messages) - summary_messages.append({"role": "user", "content": SUMMARY_PROMPT}) - try: - # summary=True -> SUMMARY_REQUEST_TOOLS / SUMMARY_REQUEST_TOOL_CHOICE (see above). - completion = await self._sample_miles(messages=summary_messages, summary=True) - except harness._CompactionContextDoesNotFit as exc: - raise harness.CodexSequenceLimit("compaction summary does not fit") from exc - _usage, total = harness._usage(completion) - if total is None: - self._metrics.usage_missing += 1 - raise harness.CodexHarnessError("Miles omitted summary token usage") - self._metrics.max_model_total_tokens = max(self._metrics.max_model_total_tokens, total) - text, finish_reason = self._summary_text(completion) - summary, ok = extract_summary(text) - if not summary: - raise harness.CodexModelFailure("compaction summary is empty") - segment = self._compaction_count - self._record["segments"].append( - { - "segment_index": segment, - "trigger_context_tokens": used, - "summary_total_tokens": total, - "summary_finish_reason": finish_reason, - "summary_ok": ok, - "kept_steps": None, - } - ) - self._miles_url = self._segment_urls[segment] - self._compaction_count += 1 - self._record["compactions"] = self._compaction_count - self._context_window += 1 - self._resume_summary = summary - self._rebuild_compacted_messages(self._choose_kept_steps(summary)) - - @staticmethod - def _summary_text(completion: dict[str, Any]) -> tuple[str, str]: - choices = completion.get("choices") - if not isinstance(choices, list) or len(choices) != 1 or not isinstance(choices[0], dict): - raise harness.CodexHarnessError("Miles returned extra or missing summary samples") - choice = choices[0] - finish_reason = choice.get("finish_reason") - if finish_reason not in {"stop", "length"}: - raise harness.CodexModelFailure("compaction summary ended abnormally") - message = choice.get("message") - if not isinstance(message, dict) or message.get("role") != "assistant": - raise harness.CodexModelFailure("compaction summary is not an assistant message") - content = message.get("content") - if not isinstance(content, str) or message.get("tool_calls") not in (None, []): - raise harness.CodexModelFailure("compaction summary must be tool-free text") - return content, finish_reason - - -class _CompactionAppServerDriver(harness._AppServerDriver): - """Stock driver; only the Codex argv gains the auto-compaction off switch.""" - - async def __aenter__(self) -> Self: - isolated_home = tempfile.TemporaryDirectory(prefix="yeto-codex-") - self._isolated_home = isolated_home - Path(isolated_home.name).chmod(0o700) - environment = { - "CODEX_HOME": isolated_home.name, - "PATH": "/usr/bin:/bin", - "LANG": "C.UTF-8", - "TZ": "UTC", - "YETO_CODEX_BRIDGE_TOKEN": self._bridge.token, - } - self._process = await asyncio.create_subprocess_exec( - *codex_argv(self._binary, self._bridge), - stdin=asyncio.subprocess.PIPE, - stdout=asyncio.subprocess.PIPE, - stderr=asyncio.subprocess.PIPE, - env=environment, - limit=harness.MAX_APP_SERVER_FRAME_BYTES + 1, - start_new_session=True, - ) - self._process_group_id = self._process.pid - self._stderr_task = asyncio.create_task(self._drain_stderr()) - return self - - -def codex_argv(binary: Path, bridge: harness._ResponsesBridge) -> list[str]: - return [*harness._codex_argv(binary, bridge), "-c", CODEX_AUTO_COMPACT_DISABLED] - - -async def drive_codex_compactionrl( - binary: Path, - miles_base_url: str, - client: Any, - episode: dict[str, Any], - request_kwargs: dict[str, Any], - metrics: legacy.AgentMetrics, - *, - max_seq_len: int | None, - segment_base_urls: Sequence[str], -) -> str: - """``codex_harness_agent._drive_codex`` with the CompactionRL bridge.""" - episode_id = episode.get("episode_id") - prompt = episode.get("prompt") - if not isinstance(episode_id, str) or not isinstance(prompt, str) or not prompt: - raise legacy.EpisodeClientError("episode daemon returned invalid episode identity") - async with CompactionRLBridge( - miles_base_url, - prompt, - request_kwargs, - metrics, - max_seq_len=max_seq_len, - segment_base_urls=segment_base_urls, - ) as bridge, _CompactionAppServerDriver( - binary, bridge, client, episode_id, prompt, metrics - ) as driver: - return await driver.drive() diff --git a/yeto/rl/harness/codex/environment.py b/yeto/rl/harness/codex/environment.py deleted file mode 100644 index e45d0d8b8..000000000 --- a/yeto/rl/harness/codex/environment.py +++ /dev/null @@ -1,181 +0,0 @@ -"""Terminal environment surface used by the Codex driver (ports path). - -The stock driver (``codex_harness_agent._AppServerDriver``) only needs two -calls from its environment: ``execute`` and ``submit``. Legacy bound those to -the SecRLEnv ``EpisodeClient``; the Terminal-Bench path binds them to a sandbox -HTTP endpoint instead. Verification (``evaluate``) is *not* part of this -surface: it belongs to the trusted layer (design D7 / R-D9). - -``HttpTerminalEnvironment`` is the untrusted-side client (no HMAC key, bearer -token only). ``serve_environment`` exposes any in-process environment over the -same wire so the worker subprocess can be exercised on CPU with a fake. -""" - -from __future__ import annotations - -import asyncio -import hmac -import json -from dataclasses import dataclass, field -from typing import Any, Protocol - -import aiohttp -from aiohttp import web - -from .client import EpisodeAPIError, EpisodeTransportError - -MAX_WIRE_BYTES = 1 << 20 - - -class TerminalEnvironment(Protocol): - async def execute( - self, episode_id: str, command: str, timeout_seconds: float, output_bytes: int - ) -> dict[str, Any]: ... - - async def submit(self, episode_id: str, submission: dict[str, Any]) -> dict[str, Any]: ... - - -class TrustedVerifier(Protocol): - """Runs the task verifier for one episode; trusted layer only.""" - - async def evaluate(self, episode_id: str) -> dict[str, Any]: ... - - -@dataclass -class FakeTerminalEnvironment: - """Scripted environment for CPU tests: ``outputs[command]`` or default echo.""" - - outputs: dict[str, dict[str, Any]] = field(default_factory=dict) - passed: bool = False - calls: list[tuple[str, str]] = field(default_factory=list) - submissions: list[dict[str, Any]] = field(default_factory=list) - evaluations: int = 0 - exec_delay_seconds: float = 0.0 - - async def execute( - self, episode_id: str, command: str, timeout_seconds: float, output_bytes: int - ) -> dict[str, Any]: - self.calls.append((episode_id, command)) - if self.exec_delay_seconds: - await asyncio.sleep(self.exec_delay_seconds) - return dict( - self.outputs.get( - command, - {"exit_code": 0, "output": f"ran:{command}", "timed_out": False, "truncated": False}, - ) - ) - - async def submit(self, episode_id: str, submission: dict[str, Any]) -> dict[str, Any]: - self.submissions.append(dict(submission)) - return {"accepted": True} - - async def evaluate(self, episode_id: str) -> dict[str, Any]: - self.evaluations += 1 - return {"passed": self.passed, "testsh_rc": 0 if self.passed else 1} - - -class HttpTerminalEnvironment: - """Untrusted-side client: ``POST {base}/execute`` and ``POST {base}/submit``.""" - - def __init__(self, base_url: str, token: str, *, total_timeout_seconds: float = 600.0): - self._base = base_url.rstrip("/") - self._token = token - self._timeout = aiohttp.ClientTimeout(total=total_timeout_seconds) - self._session: aiohttp.ClientSession | None = None - - async def __aenter__(self) -> "HttpTerminalEnvironment": - self._session = aiohttp.ClientSession(timeout=self._timeout) - return self - - async def __aexit__(self, *_exc: Any) -> None: - if self._session is not None: - await self._session.close() - self._session = None - - async def _post(self, route: str, payload: dict[str, Any]) -> dict[str, Any]: - if self._session is None: - raise EpisodeTransportError("environment client is not started") - try: - async with self._session.post( - f"{self._base}/{route}", - json=payload, - headers={"Authorization": f"Bearer {self._token}"}, - ) as response: - raw = await response.content.read(MAX_WIRE_BYTES + 1) - if len(raw) > MAX_WIRE_BYTES: - raise EpisodeTransportError("environment response is oversized") - if response.status != 200: - raise EpisodeAPIError( - response.status, "environment_error", raw.decode("utf-8", "replace")[:512] - ) - except (aiohttp.ClientError, TimeoutError) as exc: - raise EpisodeTransportError("environment transport failed") from exc - try: - body = json.loads(raw) - except ValueError as exc: - raise EpisodeTransportError("environment returned invalid JSON") from exc - if not isinstance(body, dict): - raise EpisodeTransportError("environment returned a non-object") - return body - - async def execute( - self, episode_id: str, command: str, timeout_seconds: float, output_bytes: int - ) -> dict[str, Any]: - return await self._post( - "execute", - { - "episode_id": episode_id, - "command": command, - "timeout_seconds": timeout_seconds, - "output_bytes": output_bytes, - }, - ) - - async def submit(self, episode_id: str, submission: dict[str, Any]) -> dict[str, Any]: - return await self._post("submit", {"episode_id": episode_id, "submission": submission}) - - -async def serve_environment( - env: TerminalEnvironment, token: str, *, host: str = "127.0.0.1" -) -> tuple[web.AppRunner, str]: - """Expose ``env`` on the worker wire. ``evaluate`` is deliberately not routed.""" - - def _authorized(request: web.Request) -> bool: - return hmac.compare_digest( - request.headers.get("Authorization", ""), f"Bearer {token}" - ) - - async def execute(request: web.Request) -> web.Response: - if not _authorized(request): - return web.json_response({"error": "unauthorized"}, status=401) - body = await request.json() - try: - result = await env.execute( - str(body["episode_id"]), - str(body["command"]), - timeout_seconds=float(body["timeout_seconds"]), - output_bytes=int(body["output_bytes"]), - ) - except EpisodeAPIError as exc: - return web.json_response({"error": exc.message}, status=exc.status) - return web.json_response(result) - - async def submit(request: web.Request) -> web.Response: - if not _authorized(request): - return web.json_response({"error": "unauthorized"}, status=401) - body = await request.json() - try: - result = await env.submit(str(body["episode_id"]), dict(body["submission"])) - except EpisodeAPIError as exc: - return web.json_response({"error": exc.message}, status=exc.status) - return web.json_response(result) - - app = web.Application(client_max_size=MAX_WIRE_BYTES) - app.router.add_post("/execute", execute) - app.router.add_post("/submit", submit) - runner = web.AppRunner(app, shutdown_timeout=1.0) - await runner.setup() - site = web.TCPSite(runner, host, 0) - await site.start() - port = site._server.sockets[0].getsockname()[1] # noqa: SLF001 - ephemeral port - return runner, f"http://{host}:{port}" diff --git a/yeto/rl/harness/codex/generate.py b/yeto/rl/harness/codex/generate.py deleted file mode 100644 index fec404d21..000000000 --- a/yeto/rl/harness/codex/generate.py +++ /dev/null @@ -1,23 +0,0 @@ -"""SecRLEnv agent-metadata capture (neutral part of the old generate wrapper). - -The custom agent (``agent.py`` / ``codex_harness_agent.py``) records its signed -result here; the Miles generate wrapper that restores it after Miles' aborted -fallback moved to ``yeto/rl/adapters/miles/harness_glue/codex_generate.py`` -(yeto-framework-decoupling 5.7). -""" - -from __future__ import annotations - -import copy -from contextvars import ContextVar -from typing import Any - -_CAPTURED_AGENT_METADATA: ContextVar[dict[str, Any] | None] = ContextVar( - "secrlenv_agent_metadata", default=None -) - - -def capture_agent_metadata(metadata: dict[str, Any]) -> None: - """Retain one agent result inside its generation task only.""" - - _CAPTURED_AGENT_METADATA.set(copy.deepcopy(metadata)) diff --git a/yeto/rl/harness/codex/pins.py b/yeto/rl/harness/codex/pins.py deleted file mode 100644 index bd502f83f..000000000 --- a/yeto/rl/harness/codex/pins.py +++ /dev/null @@ -1,33 +0,0 @@ -"""Self-attestation pins for the Codex harness package (ports path). - -Provenance: moved from yeto commit 5bfc011 ``yeto_miles_secrlenv/`` (see -openspec change rl-codex-harness-rollout, design R-CTX). The legacy pins in -``yeto.rl`` (``SECRLENV_*_SHA256``, ``CODEX_HARNESS_AGENT_SHA256``) are owned by -the image line and still describe the legacy import path; the values below -describe the moved files. ``LEGACY_SOURCE_SHA256`` is kept for comparison. -""" - -AGENT_SHA256 = "84c6c708cba2142e3d043ae047c90f529b90b953b739261466b8a6e28b2c7411" -GENERATE_SHA256 = "1630dc8a3aae3cbbd4708a81c9803920791ee58d6725c3fcaa4736373f8f9e48" # adapters/miles/harness_glue/codex_generate.py (P0-1.7b.4-Q1 failed status + terminal-error record; was a3212183…) - -LEGACY_SOURCE_SHA256 = { - "agent.py": "0f76c7fbd81135bc5b02cab2488629aaff1bb58dc59eae9228ca317583d90c26", - "generate.py": "9e034d6b2e9fec642501ea4a638a8fe196819dacde614ce2903359fc54ea1713", - "codex_harness_agent.py": "94fa4c245b719d236ec1007b70d395adb12456b3ef04278592ae2d3c0d843947", -} -LEGACY_SOURCE_COMMIT = "5bfc011" - -# Build-time default backend profile of the Codex OpenEnv adapter. This is the -# value the image line bakes into the container as -# ``YETO_CODEX_OPENENV_BACKEND_PROFILE`` (``yeto/rl/__init__.py`` -# CODEX_OPENENV_IDENTITY_ENV) and is a *record of the image build*, not the -# profile a launch must use. rl-fn-codex-rollout 1.0: the runtime profile is -# declared per launch (``--codex-backend-profile``) and only has to belong to -# OPENENV_BACKEND_PROFILES; see codex_openenv_agent_function.resolve_backend_profile. -OPENENV_BACKEND_PROFILE = "qwen35_08b" -# Profiles the OpenEnv adapter may be driven with at runtime. Every member must -# declare ``model_identifier``/``model_revision`` in ``yeto.rl.codex_backend`` -# (checked by ``codex_openenv_agent_function.profile_identity``). The tool -# surface pins (``*_SHA256`` identity env) are profile-independent, so adding a -# profile here needs no image rebuild. -OPENENV_BACKEND_PROFILES = ("qwen35_08b", "qwen35", "qwen38", "qwen38_next", "qwen38_next_4layer", "qwen3_30b_a3b") # qwen35 (4B): S17 M1; qwen3_30b_a3b: rlf-codex-rl 5.6; qwen38 (27B): S24 diff --git a/yeto/rl/harness/codex/preflight.py b/yeto/rl/harness/codex/preflight.py deleted file mode 100644 index d6e4f7e94..000000000 --- a/yeto/rl/harness/codex/preflight.py +++ /dev/null @@ -1,360 +0,0 @@ -"""Ports-path preflight for the Codex Terminal-Bench harness (design D3, R-CTX, R-D5a). - -``harness_preflight(miles_args, launch)`` is the IR-1 hook (INFRA signature -``entry.HarnessPreflight = Callable[[miles_args, launch], None]``): ports -``entry.preflight_stage`` calls it before ``connect_island_ray`` / placement / -model allocation, resolved from ``miles_args.yeto_harness_preflight`` or -``YETO_HARNESS_PREFLIGHT`` (= ``HARNESS_PREFLIGHT_SPEC``). It fails closed on: -- ``reward_scope`` other than ``trajectory`` (task 7.1; IR-1 - ``config.check_harness_reward_scope`` on ``miles_args.yeto_harness_reward_scope``); -- an agent function other than this package's ``CODEX_OPENENV_AGENT``; -- no environment provider (``miles_args.yeto_harness_environment_provider`` or - ``YETO_HARNESS_ENVIRONMENT_PROVIDER`` = ``module:callable`` returning an - ``EnvironmentProvider``), then installs it together with the island's - ``ToolWaitBoard`` / ``HarnessBoard`` actors (same names ``entry.harness_source`` uses); -and, through ``preflight_codex_openenv``, on: -- legacy trainable compaction being enabled: the fork pin's session server does - not know the ``X-Miles-Compaction-*`` headers and would silently roll back - window 0 (R-D5a), so ``YETO_CODEX_COMPACTION_ENABLED`` must be unset/false; -- the CPU-only scripted worker driver being allowed in a training process; -- Codex binary / version / identity drift (``codex_harness_agent._attest_runtime``: - signed env, binary sha256 and size, ``codex --version`` == 0.145.0); -- OpenEnv identity env drift against ``codex_openenv_agent_function._OPENENV_IDENTITY_ENV``. -""" - -from __future__ import annotations - -import importlib -import os -from types import SimpleNamespace -from collections.abc import Callable, Mapping -from pathlib import Path -from typing import Any - -from yeto.rl.engine.members import MEMBER_PREFIX, member_id -from yeto.rl.engine.tool_wait import LazyBoardActor, harness_board_actor -from yeto.rl.harness.reward_scope import check_harness_reward_scope - -from . import codex_harness_agent as harness -from . import codex_openenv_agent_function as adapter -from .codex_openenv_agent_worker import SCRIPTED_DRIVER_ENV - -COMPACTION_ENV = ( - "YETO_CODEX_COMPACTION_ENABLED", - "YETO_CODEX_COMPACTION_TRIGGER_TOKENS", - "YETO_CODEX_COMPACTION_SUMMARY_MAX_TOKENS", - "YETO_CODEX_MAX_COMPACTIONS", -) -_FALSE = frozenset({"", "0", "false", "no", "off"}) - - -class PreflightError(RuntimeError): - pass - - -def assert_compaction_disabled(env: Mapping[str, str]) -> None: - if env.get("YETO_CODEX_COMPACTION_ENABLED", "").strip().lower() not in _FALSE: - raise PreflightError( - "YETO_CODEX_COMPACTION_ENABLED is set: legacy trainable compaction is " - "incompatible with the fork pin session server (R-D5a) and must stay off" - ) - extra = [name for name in COMPACTION_ENV[1:] if env.get(name)] - if extra: - raise PreflightError("compaction tuning without support: " + ", ".join(extra)) - - -def assert_compactionrl_consistent(miles_args: Any, env: Mapping[str, str]) -> None: - """CompactionRL switch vs the trainer's ``--gae-variant`` (progress "S13 Codex 压缩接线"). - - This hook only runs for the Codex OpenEnv agent, which can compact, so: - ``cross_segment_per_sample`` requires ``YETO_CODEX_COMPACTIONRL`` on; any - other variant requires it off; ``cross_segment_whole_rollout`` (control arm) - is rejected outright (per-segment sessions cannot form one sample). - """ - from yeto.rl.algos import compactionrl as crl - - try: - switch = crl.compaction_switch(env.get(crl.COMPACTION_SWITCH_ENV)) - crl.check_rollout_compaction( - getattr(miles_args, "gae_variant", None), - switch, - harness_compacts=True, - t_comp=env.get(crl.COMPACTION_T_COMP_ENV), - ) - except ValueError as exc: - raise PreflightError(f"CompactionRL: {exc}") from exc - - -def assert_no_scripted_driver(env: Mapping[str, str]) -> None: - if env.get(SCRIPTED_DRIVER_ENV): - raise PreflightError(f"{SCRIPTED_DRIVER_ENV} must not be set in a training process") - - -def assert_openenv_identity(env: Mapping[str, str]) -> None: - drift = [name for name, value in adapter._OPENENV_IDENTITY_ENV.items() if env.get(name) != value] - if drift: - raise PreflightError("Codex OpenEnv identity drifted: " + ", ".join(drift)) - - -def preflight_codex_openenv( - env: Mapping[str, str] | None = None, - *, - attest_runtime: Callable[[], Path] = harness._attest_runtime, -) -> dict[str, Any]: - """Return a small report on success; raise ``PreflightError`` otherwise.""" - env = os.environ if env is None else env - assert_compaction_disabled(env) - assert_no_scripted_driver(env) - assert_openenv_identity(env) - try: - binary = attest_runtime() - except harness.CodexHarnessError as exc: - raise PreflightError(f"Codex runtime attestation failed: {exc}") from exc - return { - "codex_binary": str(binary), - "codex_version": harness.CODEX_CLI_VERSION, - "identity": adapter.codex_openenv_harness_identity(), - "compaction": "disabled", - } - - -# --------------------------------------------------------------------------- -# IR-1 hook: (miles_args, launch) -> None, run by entry.preflight_stage - -HARNESS_PREFLIGHT_SPEC = "yeto.rl.harness.codex.preflight:harness_preflight" -ENVIRONMENT_PROVIDER_ENV = "YETO_HARNESS_ENVIRONMENT_PROVIDER" -EXPECTED_AGENT_FUNCTION = "yeto.rl.harness.codex.codex_openenv_subprocess_agent_function.run" - - -def _load_dotted(spec: Any, what: str) -> Any: - if callable(spec): - return spec - module, sep, name = str(spec).partition(":") - if not sep: - module, _, name = module.rpartition(".") - if not module or not name: - raise PreflightError(f"{what} {spec!r} is not module:callable") - try: - target = getattr(importlib.import_module(module), name) - except (ImportError, AttributeError) as exc: - raise PreflightError(f"{what} {spec!r} cannot be imported: {exc}") from exc - if not callable(target): - raise PreflightError(f"{what} {spec!r} is not callable") - return target - - -def resolve_environment_provider(miles_args: Any, env: Mapping[str, str]) -> Any: - """The sandbox provider factory for this island (fail closed when absent).""" - spec = getattr(miles_args, "yeto_harness_environment_provider", None) or env.get(ENVIRONMENT_PROVIDER_ENV) - if not spec: - raise PreflightError( - f"no environment provider: set miles_args.yeto_harness_environment_provider or {ENVIRONMENT_PROVIDER_ENV}" - ) - factory = _load_dotted(spec, "environment provider") - provider = factory(miles_args) if not hasattr(factory, "acquire") else factory - if not hasattr(provider, "acquire"): - raise PreflightError(f"environment provider {spec!r} has no acquire()") - return provider - - -def island_boards(miles_args: Any) -> tuple[Any, Any]: - """``(tool_wait_board, harness_board)``: the island's named actors, looked up lazily.""" - learner_id = int(getattr(miles_args, "yeto_rl_learner_id", 0) or 0) - return LazyBoardActor(learner_id), LazyBoardActor(learner_id, factory=harness_board_actor) - - -def assert_task_prompts(miles_args: Any, provider: Any) -> None: - """2026-10-07 (S15 stage 2): every row must resolve to a real task statement. - Runs when the provider exposes its TB2 ``tasks_dir`` and the prompt data file - is readable here; the launch script's PLAN_ONLY runs the same check locally.""" - tasks_dir = getattr(provider, "tasks_dir", None) - data = getattr(miles_args, "prompt_data", None) - if tasks_dir is None or not isinstance(data, str) or not Path(data).expanduser().is_file(): - return - from .tb2_provider import TaskPromptPreflightError, preflight_task_prompts - - try: - preflight_task_prompts(Path(data).expanduser(), Path(tasks_dir)) - except TaskPromptPreflightError as exc: - raise PreflightError(str(exc)) from exc - - -def harness_preflight(miles_args: Any, launch: Any, *, env: Mapping[str, str] | None = None) -> None: - """IR-1 hook body. Raises ``PreflightError``/``HarnessConfigError`` before any allocation.""" - del launch # identity / binary / key checks do not depend on the launch args - env = os.environ if env is None else env - # 7.1: a per-segment reward scope must not start (IR-1 path, same check as validate_parsed_args). - check_harness_reward_scope(getattr(miles_args, "yeto_harness_reward_scope", None)) - agent = getattr(miles_args, "custom_agent_function_path", None) - if agent != EXPECTED_AGENT_FUNCTION: - raise PreflightError(f"custom_agent_function_path={agent!r}; the Codex harness preflight expects {EXPECTED_AGENT_FUNCTION}") - preflight_codex_openenv(env) - assert_compactionrl_consistent(miles_args, env) - provider = resolve_environment_provider(miles_args, env) - assert_task_prompts(miles_args, provider) - from . import codex_openenv_subprocess_agent_function as subprocess_agent - - tool_wait_board, harness_board = island_boards(miles_args) - subprocess_agent.configure( - provider=provider, - tool_wait_board=tool_wait_board, - harness_board=harness_board, - member=resolve_member(miles_args, env), - ) - - -MEMBER_CELL_ENV = "YETO_RL_CELL_ID" -LEARNER_ID_ENV = "YETO_RL_LEARNER_ID" -# Harness env the island driver forwards into Ray's job runtime_env so the -# rollout workers (which inherit the raylet's environment, not the driver's) -# can self-configure (A-T3-4, codex-smoke-20261003-6). -WORKER_PASSTHROUGH_ENV = ( - ENVIRONMENT_PROVIDER_ENV, - # secret-handling-hardening: the sandbox-only Modal token, never MODAL_TOKEN_*. - "YETO_SANDBOX_MODAL_TOKEN_ID", - "YETO_SANDBOX_MODAL_TOKEN_SECRET", - "OPENENV_RUN_ID", - "SECRLENV_MAX_TURNS", - MEMBER_CELL_ENV, - LEARNER_ID_ENV, -) -WORKER_PASSTHROUGH_ENV_PREFIXES = ("YETO_HARNESS_TB2_", "YETO_CODEX_") - - -def worker_runtime_env(miles_args: Any, env: Mapping[str, str] | None = None) -> dict[str, str]: - """Env vars every Ray actor of this island must see for the agent function to run.""" - env = os.environ if env is None else env - forwarded = { - key: value - for key, value in env.items() - if value and (key in WORKER_PASSTHROUGH_ENV or key.startswith(WORKER_PASSTHROUGH_ENV_PREFIXES)) - } - learner_id = getattr(miles_args, "yeto_rl_learner_id", None) - if learner_id is not None and str(learner_id) != "": - forwarded[LEARNER_ID_ENV] = str(int(learner_id)) - cell = getattr(miles_args, "yeto_rl_cell_id", None) - if cell is not None and str(cell) != "": - forwarded[MEMBER_CELL_ENV] = str(cell) - return forwarded - - -def configure_rollout_worker(env: Mapping[str, str] | None = None) -> bool: - """Install the provider and boards in a rollout worker from its environment. - - ``harness_preflight`` configures the driver process only; upstream Miles - calls the agent function inside ``RolloutExecutor`` actors, where the - module globals start empty. Returns False (configures nothing) when the - environment names no provider, so ``run`` keeps failing closed. - """ - env = os.environ if env is None else env - if not env.get(ENVIRONMENT_PROVIDER_ENV): - return False - miles_args = SimpleNamespace( - yeto_harness_environment_provider=None, - yeto_rl_learner_id=int(env.get(LEARNER_ID_ENV) or 0), - yeto_rl_member_id=None, - # A16 (D5): the island entry exports its cell (publish_member_cell) and - # worker_runtime_env forwards it; read it back into the field so the - # worker resolves the same member key as the driver. - yeto_rl_cell_id=env.get(MEMBER_CELL_ENV) or None, - ) - provider = resolve_environment_provider(miles_args, env) - assert_task_prompts(miles_args, provider) - from . import codex_openenv_subprocess_agent_function as subprocess_agent - - tool_wait_board, harness_board = island_boards(miles_args) - subprocess_agent.configure( - provider=provider, - tool_wait_board=tool_wait_board, - harness_board=harness_board, - member=resolve_member(miles_args, env), - ) - return True - - -def resolve_member(miles_args: Any, env: Mapping[str, str] | None = None) -> str | None: - """The INFRA rollout member key this island's sessions are admitted under. - - INFRA keys members as ``rollout.member_id(cell_id)`` (= ``engine:``, - ``MilesRolloutPool.members`` / ``close_admission``). Sources, in order: - ``miles_args.yeto_rl_member_id`` (a full member key, or a bare cell id), - ``miles_args.yeto_rl_cell_id`` / ``YETO_RL_CELL_ID`` (a cell id). None - (no source) keeps the global admission key, which single-island smoke - runs rely on. - """ - env = os.environ if env is None else env - member = getattr(miles_args, "yeto_rl_member_id", None) - if member is not None and str(member) != "": - member = str(member) - return member if member.startswith(MEMBER_PREFIX) else member_id(member) - cell = getattr(miles_args, "yeto_rl_cell_id", None) - if cell is None or str(cell) == "": - cell = env.get(MEMBER_CELL_ENV) or None - return member_id(cell) if cell is not None else None - - -# --------------------------------------------------------------------------- -# Task 2.2: legacy preflight forwarding (injectable; legacy modules are not edited here) -# -# ``yeto/rl/adapters/miles/island_entry.py::_preflight_codex_openenv_adapter`` and -# ``yeto/rl/tbench_direct_preflight.py::_attest_adapter`` currently import the -# adapter from ``/examples/experimental/openenv`` and compare it -# against ``yeto.rl.CODEX_OPENENV_IDENTITY_ENV`` (image-line pins). Once the -# image line repoints those pins (see ``required_pin_updates``), the legacy -# functions become one-liners calling ``forward_legacy_openenv_preflight``. - -def forward_legacy_openenv_preflight(args: Any, profile_name: str, env: Mapping[str, str] | None = None) -> None: - """Drop-in body for ``learner._preflight_codex_openenv_adapter(args, profile)``. - - Same failure classes/messages as legacy so existing callers keep their - behaviour: ``ValueError`` with "requires backend profile", "identity drifted", - "environment drifted". - """ - del args # the adapter no longer lives under miles_root - # rl-fn-codex-rollout 1.0: the runtime profile only has to be an allowlisted - # OpenEnv profile with a declared HF identity (``adapter.profile_identity``); - # ``learner._preflight_codex_harness`` has already matched it against - # ``--model`` / ``--model-revision`` (``validate_stock_codex_fields``). The - # identity env below is the image's build-time record and is compared as - # such, so a new profile needs no image rebuild. - adapter.profile_identity(profile_name) - live = adapter.codex_openenv_harness_identity() - expected = { - name.removeprefix("YETO_CODEX_OPENENV_").lower(): value - for name, value in adapter._OPENENV_IDENTITY_ENV.items() - if name.endswith("_SHA256") - } - if live != expected: - raise ValueError("the Codex OpenEnv surface identity drifted") - env = os.environ if env is None else env - mismatched = [name for name, value in adapter._OPENENV_IDENTITY_ENV.items() if env.get(name) != value] - if mismatched: - raise ValueError("Codex OpenEnv container environment drifted: " + ", ".join(mismatched)) - - -def required_pin_updates() -> dict[str, Any]: - """What ``yeto/rl/__init__.py`` (image line) must change to point at this package.""" - import hashlib - - here = Path(__file__).resolve().parent - sha = lambda name: hashlib.sha256((here / name).read_bytes()).hexdigest() # noqa: E731 - return { - "SECRLENV_AGENT_PATH": "yeto/rl/harness/codex/agent.py", - "SECRLENV_AGENT_SHA256": sha("agent.py"), - "SECRLENV_AGENT": "yeto.rl.harness.codex.agent.run", - "SECRLENV_REWARD": "yeto.rl.harness.codex.reward:reward_func", - "SECRLENV_GROUP_FILTER": "yeto.rl.harness.codex.reward.check_group", - "SECRLENV_GENERATE": "yeto.rl.adapters.miles.harness_glue.codex_generate.generate", - "SECRLENV_GENERATE_SHA256": hashlib.sha256( # decoupling 5.7: glue lives in the adapter - (here.parents[1] / "adapters" / "miles" / "harness_glue" / "codex_generate.py").read_bytes() - ).hexdigest(), - "CODEX_HARNESS_AGENT": "yeto.rl.harness.codex.codex_harness_agent.run", - "CODEX_HARNESS_AGENT_PATH": "yeto/rl/harness/codex/codex_harness_agent.py", - "CODEX_HARNESS_AGENT_SHA256": sha("codex_harness_agent.py"), - "CODEX_OPENENV_AGENT": "yeto.rl.harness.codex.codex_openenv_subprocess_agent_function.run", - "CODEX_OPENENV_AGENT_MODULES": ( - "codex_openenv_subprocess_agent_function.py", "codex_openenv_agent_worker.py", "codex_openenv_agent_function.py", - "compaction_bridge.py", - ), - "CODEX_OPENENV_IDENTITY_ENV": dict(adapter._OPENENV_IDENTITY_ENV), - } diff --git a/yeto/rl/harness/codex/reward.py b/yeto/rl/harness/codex/reward.py deleted file mode 100644 index cc3a27762..000000000 --- a/yeto/rl/harness/codex/reward.py +++ /dev/null @@ -1,377 +0,0 @@ -"""Validated reward extraction for secrlenv Miles samples.""" - -from __future__ import annotations - -import json -import math -import re -import statistics -from typing import Any - -from yeto.rl.rewards.types import RewardResult, Trajectory - -OUTCOME_KEY = "secrlenv_trusted_outcome" -INFRASTRUCTURE_STATUS = "infrastructure_error" -CLEANUP_ERROR_STATUS = "cleanup_error" -INFRASTRUCTURE_503_RETRIES_KEY = "secrlenv_infrastructure_503_retries" -ADMISSION_SCHEMA = 2 -ADMISSION_PHASE_KEY = "admission_phase" -ADMISSION_ERROR_CODE_KEY = "admission_error_code" -ADMISSION_NONCE_KEY = "admission_nonce" -ADMISSION_PHASE = "pre_create" -ADMISSION_ERROR_CODE = "infrastructure_error" -_ADMISSION_NONCE = re.compile(r"[0-9a-f]{32}") -_OUTCOME_STATUSES = { - "completed", - "timeout", - "max_turns", - "max_seq_len", - INFRASTRUCTURE_STATUS, - CLEANUP_ERROR_STATUS, -} - - -class SecRLEnvRolloutTerminalError(RuntimeError): - """Sanitized base class for non-recoverable SecRLEnv rollout failures.""" - - -class SecRLEnvReplacementExhausted(SecRLEnvRolloutTerminalError): - """The one authenticated same-task replacement did not restore progress.""" - - -class SecRLEnvUntrustedEvidence(SecRLEnvRolloutTerminalError): - """A rollout group did not carry authentic SecRLEnv evidence.""" - - -class SecRLEnvRolloutCleanupError(SecRLEnvRolloutTerminalError): - """Pinned Miles could not drain a terminal SecRLEnv rollout.""" - - -class UntrustedOutcome(RuntimeError): - pass - - -def _canonical(value: dict[str, Any]) -> bytes: - return json.dumps(value, sort_keys=True, separators=(",", ":")).encode("utf-8") - - -def _verified_outcome(metadata: Any) -> tuple[dict[str, Any], float]: - if not isinstance(metadata, dict): - raise UntrustedOutcome("sample metadata is not an object") - outcome = metadata.get(OUTCOME_KEY) - if not isinstance(outcome, dict): - raise UntrustedOutcome("sample has no secrlenv outcome") - schema = outcome.get("schema") - status = outcome.get("status") - if schema not in {1, ADMISSION_SCHEMA} or status not in _OUTCOME_STATUSES: - raise UntrustedOutcome("secrlenv outcome has an invalid schema or status") - infrastructure_503_retries = outcome.get(INFRASTRUCTURE_503_RETRIES_KEY) - if ( - type(infrastructure_503_retries) is not int - or infrastructure_503_retries not in {0, 1} - ): - raise UntrustedOutcome("secrlenv outcome has an invalid retry ledger") - reward = outcome.get("reward") - if isinstance(reward, bool) or not isinstance(reward, (int, float)): - raise UntrustedOutcome("secrlenv outcome reward is not numeric") - value = float(reward) - if not math.isfinite(value) or not 0.0 <= value <= 1.0: - raise UntrustedOutcome("secrlenv outcome reward is outside [0, 1]") - if not isinstance(outcome.get("task_id"), str): - raise UntrustedOutcome("secrlenv outcome has no episode/task identity") - if schema == ADMISSION_SCHEMA: - expected_keys = { - "schema", - "status", - "episode_id", - "task_id", - "reward", - "passed", - "class", - INFRASTRUCTURE_503_RETRIES_KEY, - ADMISSION_PHASE_KEY, - ADMISSION_ERROR_CODE_KEY, - ADMISSION_NONCE_KEY, - } - if ( - set(outcome) != expected_keys - or status != INFRASTRUCTURE_STATUS - or outcome.get("episode_id") is not None - or not outcome["task_id"] - or outcome.get(ADMISSION_PHASE_KEY) != ADMISSION_PHASE - or outcome.get(ADMISSION_ERROR_CODE_KEY) != ADMISSION_ERROR_CODE - or not isinstance(outcome.get(ADMISSION_NONCE_KEY), str) - or _ADMISSION_NONCE.fullmatch(outcome[ADMISSION_NONCE_KEY]) is None - or infrastructure_503_retries != 1 - or value != 0.0 - or outcome.get("passed") is not False - or outcome.get("class") is not None - ): - raise UntrustedOutcome("secrlenv admission outcome is invalid") - elif ( - not isinstance(outcome.get("episode_id"), str) - or any( - key in outcome - for key in ( - ADMISSION_PHASE_KEY, - ADMISSION_ERROR_CODE_KEY, - ADMISSION_NONCE_KEY, - ) - ) - ): - raise UntrustedOutcome("secrlenv outcome has no episode/task identity") - return outcome, value - - -def verify_outcome(metadata: Any) -> float: - outcome, value = _verified_outcome(metadata) - if outcome.get("status") in {INFRASTRUCTURE_STATUS, CLEANUP_ERROR_STATUS}: - raise UntrustedOutcome( - "secrlenv non-verdict outcome is not a grader verdict" - ) - return value - - -INFRASTRUCTURE_ABORT_REASON = "secrlenv_infrastructure_failure" - - -def secrlenv_reward(trajectory: Trajectory) -> RewardResult: - """Neutral form (decoupling 3.3): only structurally valid rewards. - - Missing or invalid evidence raises :class:`UntrustedOutcome` (fatal). - An infrastructure marker is not a - verdict: it returns value 0.0 with ``aborted=True`` so the backend adapter - marks the sample with its abort status and the group is replaced. - """ - - outcome, value = _verified_outcome(trajectory.metadata) - if outcome.get("status") != INFRASTRUCTURE_STATUS: - return RewardResult(value=value) - return RewardResult(value=0.0, aborted=True, reason=INFRASTRUCTURE_ABORT_REASON) - - -def _sample_reward(sample: Any) -> float: - result = secrlenv_reward(Trajectory(metadata=getattr(sample, "metadata", None))) - if not result.aborted: - return result.value - - # Miles calls the reward function before the dynamic group filter. Mark an - # infrastructure failure FAILED here so the group check replaces it, while - # still returning the numeric value required by the reward-function API. - # The group check never permits this transient zero into training data. - # FAILED, not ABORTED: Miles drops a group with an aborted sample before - # the group check runs, so its same-task retry would never be scheduled. - from miles.utils.types import Sample - - sample.status = Sample.Status.FAILED - return result.value - - -async def reward_func(args: Any, samples: Any, **_kwargs: Any) -> float | list[float]: - """Return only structurally valid host-side rewards. - - Missing or invalid metadata remains fatal. An infrastructure marker - is returned as the numeric zero required by Miles only after marking its - sample FAILED, so the group check replaces it and it never enters training. - """ - - try: - if isinstance(samples, list): - return [_sample_reward(sample) for sample in samples] - return _sample_reward(samples) - except UntrustedOutcome as error: - remember_terminal_error(args, error) - raise - - -def sample_statuses(samples: Any) -> list[str | None]: - """Status value of every sample in ``samples`` (nested lists flattened).""" - - values: list[str | None] = [] - pending = list(samples) if isinstance(samples, list) else [] - while pending: - item = pending.pop(0) - if isinstance(item, list): - pending[0:0] = item - continue - status = getattr(item, "status", None) - value = getattr(status, "value", status) - values.append(value if isinstance(value, str) else None) - return values - - -# Miles catches an exception raised inside a generate task and only counts the -# group as failed; the rollout then fails later with a different error. The -# SecRLEnv code records the first terminal error here so the backend can -# raise it instead. -TERMINAL_ERROR_ATTR = "_yeto_secrlenv_terminal_error" - - -def remember_terminal_error(args: Any, error: BaseException) -> None: - if args is not None and getattr(args, TERMINAL_ERROR_ATTR, None) is None: - try: - setattr(args, TERMINAL_ERROR_ATTR, error) - except AttributeError: - pass - - -def _flatten_samples(samples: list[Any]) -> list[Any]: - flattened: list[Any] = [] - for item in samples: - flattened.extend(item if isinstance(item, list) else [item]) - return flattened - - -def _outcome_fingerprint(outcome: Any) -> bytes | None: - """Return a hashable outcome identity; malformed outcomes fail later.""" - - if not isinstance(outcome, dict): - return None - try: - return _canonical(outcome) - except (TypeError, ValueError): - return None - - -def _group_key(samples: list[Any]) -> tuple[Any, ...]: - """Identify a generated group across Miles' repeated callbacks. - - Sample indexes are stable across the generation-time and all-samples - callbacks. The episode identity and canonical outcome prevent an index - collision from reusing a decision for different external-environment - evidence. - """ - - values = [] - for sample in samples: - metadata = getattr(sample, "metadata", None) - outcome = metadata.get(OUTCOME_KEY) if isinstance(metadata, dict) else None - index = getattr(sample, "index", None) - values.append( - ( - index if index is not None else id(sample), - outcome.get("episode_id") if isinstance(outcome, dict) else None, - outcome.get("task_id") if isinstance(outcome, dict) else None, - _outcome_fingerprint(outcome), - ) - ) - return tuple(values) - - -def _replacement_limit(args: Any) -> int | None: - value = getattr(args, "yeto_rl_dynamic_sampling_max_replacements", None) - try: - limit = int(value) if value is not None else None - except (TypeError, ValueError): - return None - return limit if limit is not None and limit >= 0 else None - - -def check_group(args: Any, samples: list[Any], **_kwargs: Any): - """Validate groups and coordinate bounded same-task replacements.""" - - from miles.rollout.filter_hub.base_types import DynamicFilterOutput - - flattened = _flatten_samples(samples) - rollout_id = getattr(args, "yeto_rl_policy_version", None) - state = getattr(args, "_yeto_secrlenv_filter_state", None) - if state is None or state.get("rollout_id") != rollout_id: - state = { - "rollout_id": rollout_id, - "rejections": 0, - "nontrainable_rejections": 0, - "forced": 0, - "decisions": {}, - } - args._yeto_secrlenv_filter_state = state - - key = _group_key(flattened) - previous = state["decisions"].get(key) - if previous is not None: - return DynamicFilterOutput(keep=previous[0], reason=previous[1]) - - try: - verified = [_verified_outcome(sample.metadata) for sample in flattened] - except UntrustedOutcome: - raise SecRLEnvUntrustedEvidence( - "SecRLEnv rollout evidence validation failed" - ) from None - - statuses = sample_statuses(flattened) - infrastructure = any( - outcome.get("status") == INFRASTRUCTURE_STATUS - for outcome, _ in verified - ) - cleanup_failure = any( - outcome.get("status") == CLEANUP_ERROR_STATUS - for outcome, _ in verified - ) - if cleanup_failure: - raise SecRLEnvRolloutCleanupError( - "SecRLEnv episode cleanup could not be verified" - ) - incomplete = any( - status not in {"completed", "truncated"} for status in statuses - ) - if infrastructure or incomplete: - if infrastructure: - reason = "secrlenv_infrastructure_failure" - elif "pending" in statuses: - reason = "secrlenv_pending" - elif "failed" in statuses: - reason = "secrlenv_failed" - else: - reason = "secrlenv_incomplete_status" - decision = (False, reason) - state["decisions"][key] = decision - state["nontrainable_rejections"] += 1 - retry = getattr(args, "_yeto_secrlenv_retry_callback", None) - if not callable(retry): - raise SecRLEnvReplacementExhausted( - "SecRLEnv rollout replacement scheduler is unavailable" - ) - retry_ledger: dict[int, int] = {} - for sample, (outcome, _) in zip(flattened, verified, strict=True): - try: - index = int(sample.index) - except (AttributeError, TypeError, ValueError): - raise SecRLEnvReplacementExhausted( - "SecRLEnv rollout replacement task identity is unavailable" - ) from None - value = outcome[INFRASTRUCTURE_503_RETRIES_KEY] - previous_retry_count = retry_ledger.setdefault(index, value) - if previous_retry_count != value: - raise SecRLEnvUntrustedEvidence( - "SecRLEnv rollout retry ledger is inconsistent" - ) - retry( - flattened, - evidence_key=key, - reason=reason, - infrastructure_503_retries=retry_ledger, - ) - return DynamicFilterOutput(keep=False, reason=reason) - - rewards = [value for _, value in verified] - - std = statistics.pstdev(rewards) if rewards else 0.0 - if math.isfinite(std) and std > 1e-8: - decision = (True, None) - else: - limit = _replacement_limit(args) - if limit is None: - raise SecRLEnvReplacementExhausted( - "SecRLEnv zero-variance replacement contract is missing" - ) - if state["rejections"] < limit: - state["rejections"] += 1 - value = round(rewards[0], 3) if rewards else 0.0 - decision = (False, f"secrlenv_zero_std_{value}") - else: - state["forced"] += 1 - decision = ( - True, - f"secrlenv_bounded_fallback_after_{limit}_replacements", - ) - state["decisions"][key] = decision - return DynamicFilterOutput(keep=decision[0], reason=decision[1]) diff --git a/yeto/rl/harness/codex/tb2_network_grants.py b/yeto/rl/harness/codex/tb2_network_grants.py deleted file mode 100644 index d29bcac2c..000000000 --- a/yeto/rl/harness/codex/tb2_network_grants.py +++ /dev/null @@ -1,104 +0,0 @@ -"""Explicit network grants for the 89 Terminal-Bench-2 tasks (rl-agentic-reward-env task 5.1). - -Sandbox egress is closed unless a task is listed in a network policy. Every -TB2 task declares ``allow_internet = true`` upstream and its verifier -(``tests/test.sh``) installs uv/pip/apt packages at grading time, and the -agentic runs that passed so far (S17 G3 positive control, FN codex r3) ran -with open egress. To keep them working, each task is granted ``open`` -explicitly here. Narrowing these to a domain allowlist needs a Modal CPU -check first (not done; see design.md D-NET). -""" - -from __future__ import annotations - -TB2_OPEN_EGRESS_TASKS: tuple[str, ...] = ( - "adaptive-rejection-sampler", - "bn-fit-modify", - "break-filter-js-from-html", - "build-cython-ext", - "build-pmars", - "build-pov-ray", - "caffe-cifar-10", - "cancel-async-tasks", - "chess-best-move", - "circuit-fibsqrt", - "cobol-modernization", - "code-from-image", - "compile-compcert", - "configure-git-webserver", - "constraints-scheduling", - "count-dataset-tokens", - "crack-7z-hash", - "custom-memory-heap-crash", - "db-wal-recovery", - "distribution-search", - "dna-assembly", - "dna-insert", - "extract-elf", - "extract-moves-from-video", - "feal-differential-cryptanalysis", - "feal-linear-cryptanalysis", - "filter-js-from-html", - "financial-document-processor", - "fix-code-vulnerability", - "fix-git", - "fix-ocaml-gc", - "gcode-to-text", - "git-leak-recovery", - "git-multibranch", - "gpt2-codegolf", - "headless-terminal", - "hf-model-inference", - "install-windows-3.11", - "kv-store-grpc", - "large-scale-text-editing", - "largest-eigenval", - "llm-inference-batching-scheduler", - "log-summary-date-ranges", - "mailman", - "make-doom-for-mips", - "make-mips-interpreter", - "mcmc-sampling-stan", - "merge-diff-arc-agi-task", - "model-extraction-relu-logits", - "modernize-scientific-stack", - "mteb-leaderboard", - "mteb-retrieve", - "multi-source-data-merger", - "nginx-request-logging", - "openssl-selfsigned-cert", - "overfull-hbox", - "password-recovery", - "path-tracing", - "path-tracing-reverse", - "polyglot-c-py", - "polyglot-rust-c", - "portfolio-optimization", - "protein-assembly", - "prove-plus-comm", - "pypi-server", - "pytorch-model-cli", - "pytorch-model-recovery", - "qemu-alpine-ssh", - "qemu-startup", - "query-optimize", - "raman-fitting", - "regex-chess", - "regex-log", - "reshard-c4-data", - "rstan-to-pystan", - "sam-cell-seg", - "sanitize-git-repo", - "schemelike-metacircular-eval", - "sparql-university", - "sqlite-db-truncate", - "sqlite-with-gcov", - "torch-pipeline-parallelism", - "torch-tensor-parallelism", - "train-fasttext", - "tune-mjcf", - "video-processing", - "vulnerable-secret", - "winning-avg-corewars", - "write-compressor", -) diff --git a/yeto/rl/harness/codex/tb2_provider.py b/yeto/rl/harness/codex/tb2_provider.py deleted file mode 100644 index 5a46718e8..000000000 --- a/yeto/rl/harness/codex/tb2_provider.py +++ /dev/null @@ -1,980 +0,0 @@ -"""Terminal-Bench-2 ``EnvironmentProvider`` (design R-TB / gap G1). - -One sandbox per trajectory, started from the task's **official image** -(``task.toml [environment].docker_image``, the exact image the TB2 harness -itself runs). The sandbox runs nothing but ``sleep infinity``; every agent -command is relayed from the *trusted* side through the backend's ``exec``. -The worker-facing wire (``/execute`` ``/submit``) is therefore served by -``environment.serve_environment`` **in the trusted process**, and the untrusted -worker only ever sees ``http://127.0.0.1:`` plus a bearer token — exactly -the surface ``HttpTerminalEnvironment`` already speaks. Compared with the -upstream recipe (``tb2_sandbox_modal`` + an OpenEnv ``tbench2_env`` server -layer baked into every task image) this needs no extra image layers, no -tunnel and no Python inside the task image. - -Verification mirrors the official harness's stage-at-verify model: ``tests/`` -never enters the sandbox before ``evaluate`` (the agent cannot read the -checks), is staged at ``/tests`` from the trusted side when the episode is -over, ``bash /tests/test.sh`` runs in the task workdir and the reward is the -``/logs/verifier/reward.txt`` the script writes (``1``/``0``). ``solution/`` -is never shipped at all (it exists only for the R-TB positive-path proof, run -by the operator through the same relay). - -Backends implement ``SandboxBackend``; ``LocalProcessBackend`` (CPU tests and -dev) runs commands in a scratch directory on this host, ``ModalSandboxBackend`` -materialises a Modal Sandbox. ``module:callable`` factories for -``YETO_HARNESS_ENVIRONMENT_PROVIDER``:: - - yeto.rl.harness.codex.tb2_provider:modal_provider - yeto.rl.harness.codex.tb2_provider:local_provider - -Configuration (environment variables, read when the factory runs): - -``YETO_HARNESS_TB2_TASKS_DIR`` terminal-bench-2 checkout root (required); - ``task_id`` = task directory name (the - ``metadata.task_id`` make_tbench2_data.py writes) -``YETO_HARNESS_TB2_LEASE_SECONDS`` hard per-trajectory wall clock (default: - the task's ``[agent].timeout_sec``, 900) -``YETO_HARNESS_TB2_MODAL_APP`` Modal app the sandboxes are created under - (default ``yeto-tbench2``; sweep scope) -``YETO_HARNESS_TB2_SANDBOX_TTL_S`` / ``YETO_HARNESS_TB2_IDLE_TIMEOUT_S`` - Modal hard ceiling / idle reclamation -``YETO_HARNESS_TB2_NETWORK_POLICY`` path of a JSON network policy that - replaces the built-in one (see - ``NetworkPolicy``); sandbox egress is closed - for every task the policy does not grant -``YETO_HARNESS_TB2_FAULT`` fault injection for the R-TB coverage matrix, - comma separated ``kind:selector[=value]``: - ``create_fail:`` acquire raises - (-> INFRASTRUCTURE / ABORTED); - ``deadline:=`` lease deadline of - ``s`` seconds (wall-clock cancel -> timeout, - destroy, env_live back to 0); - ``max_turns:=`` worker gets - ``SECRLENV_MAX_TURNS=n`` (turn budget). - ```` is the 0-based acquire ordinal in - this process or an exact trajectory id. -""" - -from __future__ import annotations - -import asyncio -import base64 -import io -import os -import re -import secrets -import shlex -import shutil -import signal -import subprocess -import sys -import tarfile -import tempfile -import threading -import time -from dataclasses import dataclass, field -from pathlib import Path -from typing import Any, Callable, Protocol - -try: - import tomllib -except ImportError: # pragma: no cover - Python < 3.11 - import tomli as tomllib # type: ignore[no-redef] - -from .client import EpisodeAPIError -from .environment import serve_environment - -TASKS_DIR_ENV = "YETO_HARNESS_TB2_TASKS_DIR" -LEASE_SECONDS_ENV = "YETO_HARNESS_TB2_LEASE_SECONDS" -FAULT_ENV = "YETO_HARNESS_TB2_FAULT" -MODAL_APP_ENV = "YETO_HARNESS_TB2_MODAL_APP" -SANDBOX_TTL_ENV = "YETO_HARNESS_TB2_SANDBOX_TTL_S" -IDLE_TIMEOUT_ENV = "YETO_HARNESS_TB2_IDLE_TIMEOUT_S" -ENV_HOST_ENV = "YETO_HARNESS_TB2_ENV_HOST" -NETWORK_POLICY_ENV = "YETO_HARNESS_TB2_NETWORK_POLICY" - -DEFAULT_MODAL_APP = "yeto-tbench2" -DEFAULT_SANDBOX_TTL_S = 1800 -DEFAULT_IDLE_TIMEOUT_S = 600 -DEFAULT_WORKDIR = "/app" -TESTS_PATH = "/tests" -VERIFIER_LOGS_PATH = "/logs/verifier" -MAX_VERIFIER_OUTPUT_BYTES = 64 * 1024 -FAULT_KINDS = ("create_fail", "deadline", "max_turns") -TURN_BUDGET_ENV = "SECRLENV_MAX_TURNS" - - -# ----------------------------------------------------------------------------- task - - -@dataclass(frozen=True) -class Tb2Task: - task_id: str - task_dir: Path - docker_image: str - cpus: int - memory_mb: int - workdir: str - agent_timeout_s: float - verifier_timeout_s: float - # Task statement from ``instruction.md`` (the Terminal-Bench prompt). The - # smoke datasets carry only ``task_id`` and a generic system message, so - # this is the only place the model can learn what to do (S15 stage 2: all - # 48 trajectories ran without it and scored 0). - instruction: str | None = None - - @property - def tests_dir(self) -> Path: - return self.task_dir / "tests" - - @property - def solution_dir(self) -> Path: - return self.task_dir / "solution" - - -_TASK_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$") - - -def resolve_task(task_ref: str, tasks_dir: Path) -> Tb2Task: - """``task_ref`` is the task directory name (``metadata.task_id``); fail closed otherwise.""" - if not isinstance(task_ref, str) or not _TASK_ID_RE.match(task_ref): - raise ValueError(f"invalid TB2 task reference {task_ref!r}") - task_dir = Path(tasks_dir) / task_ref - toml_path = task_dir / "task.toml" - if not toml_path.is_file(): - raise FileNotFoundError(f"{task_ref}: no task.toml under {tasks_dir}") - config = tomllib.loads(toml_path.read_text()) - env_cfg = config.get("environment", {}) - image = env_cfg.get("docker_image") - if not image: - raise ValueError(f"{task_ref}: task.toml has no [environment].docker_image") - if not (task_dir / "tests" / "test.sh").is_file(): - raise FileNotFoundError(f"{task_ref}: tests/test.sh is missing") - return Tb2Task( - task_id=task_ref, - task_dir=task_dir, - docker_image=str(image), - cpus=max(1, int(env_cfg.get("cpus", 1))), - memory_mb=max(2048, int(env_cfg.get("memory_mb", 2048))), - workdir=_dockerfile_workdir(task_dir / "environment" / "Dockerfile"), - agent_timeout_s=float(config.get("agent", {}).get("timeout_sec", 900.0)), - verifier_timeout_s=float(config.get("verifier", {}).get("timeout_sec", 900.0)), - instruction=_read_instruction(task_dir / "instruction.md"), - ) - - -def tb2_instructions_env() -> dict[str, str]: - """Worker env selecting the signed TB2 Codex system prompt.""" - from . import codex_harness_agent as harness - - return { - harness.INSTRUCTIONS_FAMILY_ENV: harness.TB2_INSTRUCTIONS_FAMILY, - harness.TB2_INSTRUCTIONS_SHA_ENV: harness.TB2_BASE_INSTRUCTIONS_SHA256, - } - - -class TaskPromptPreflightError(ValueError): - """A dataset row would reach Codex without a real task statement.""" - - -def preflight_task_prompts(data_path: Path, tasks_dir: Path) -> list[tuple[str, str]]: - """Resolve the first user message of every dataset row exactly as the - subprocess agent will (``task_prompt``) and refuse rows without a task - statement or whose prompt is a stringified chat list (S15 stage 2 root - cause). Returns ``[(task_id, prompt), ...]`` in row order.""" - import json - from types import SimpleNamespace - - from .codex_openenv_subprocess_agent_function import TaskPromptMissing, task_prompt - - out: list[tuple[str, str]] = [] - for lineno, line in enumerate(Path(data_path).read_text().splitlines(), 1): - if not line.strip(): - continue - row = json.loads(line) - metadata = dict(row.get("metadata") or {}) - task_id = metadata.get("task_id") - try: - task = resolve_task(task_id, Path(tasks_dir)) - prompt = task_prompt(metadata, row.get("prompt"), SimpleNamespace(task=task)) - except (TaskPromptMissing, ValueError, FileNotFoundError) as exc: - raise TaskPromptPreflightError(f"{data_path}:{lineno} task {task_id!r}: {exc}") from exc - head = prompt.lstrip()[:2] - if head in ("[{", "{'", '{"') or "'role':" in prompt or '"role":' in prompt: - raise TaskPromptPreflightError(f"{data_path}:{lineno} task {task_id!r}: prompt is a stringified chat message") - out.append((str(task_id), prompt)) - if not out: - raise TaskPromptPreflightError(f"{data_path}: no dataset rows") - return out - - -def _read_instruction(path: Path) -> str | None: - if not path.is_file(): - return None - text = path.read_text(errors="replace").strip() - return text or None - - -def _dockerfile_workdir(dockerfile: Path) -> str: - workdir = DEFAULT_WORKDIR - if dockerfile.is_file(): - for line in dockerfile.read_text(errors="replace").splitlines(): - m = re.match(r"^\s*WORKDIR\s+(\S+)", line) - if m: - workdir = m.group(1).strip('"') - return workdir - - -def tests_tar_b64(tests_dir: Path) -> str: - buf = io.BytesIO() - with tarfile.open(fileobj=buf, mode="w:gz") as tar: - for path in sorted(tests_dir.rglob("*")): - if path.is_file() and "__pycache__" not in path.parts: - tar.add(path, arcname=str(path.relative_to(tests_dir))) - return base64.b64encode(buf.getvalue()).decode() - - -# ----------------------------------------------------------------------------- network policy - - -@dataclass(frozen=True) -class NetworkGrant: - """Outbound network access of one sandbox. - - The default grant is closed (no egress). ``open`` allows all egress. - Otherwise only ``domains`` (``*.`` wildcard prefixes allowed) and - ``cidrs`` are reachable. - """ - - open: bool = False - domains: tuple[str, ...] = () - cidrs: tuple[str, ...] = () - - @property - def closed(self) -> bool: - return not self.open and not self.domains and not self.cidrs - - @classmethod - def parse(cls, value: Any, *, where: str) -> "NetworkGrant": - if value == "open": - return cls(open=True) - if value in ("closed", None): - return cls() - if not isinstance(value, dict) or set(value) - {"domains", "cidrs"}: - raise ValueError(f"{where}: grant must be 'open', 'closed' or {{domains, cidrs}}, got {value!r}") - domains = tuple(value.get("domains") or ()) - cidrs = tuple(value.get("cidrs") or ()) - for item in domains + cidrs: - if not isinstance(item, str) or not item or item == "*": - raise ValueError(f"{where}: bad allowlist entry {item!r} (use 'open' for all egress)") - import ipaddress - - for cidr in cidrs: - ipaddress.ip_network(cidr, strict=False) - return cls(domains=domains, cidrs=cidrs) - - def modal_kwargs(self) -> dict[str, Any]: - """``modal.Sandbox.create`` network arguments (modal >= 1.5.5).""" - if self.open: - return {} - if self.closed: - return {"block_network": True} - kwargs: dict[str, Any] = {} - if self.cidrs: - kwargs["outbound_cidr_allowlist"] = list(self.cidrs) - if self.domains: - kwargs["outbound_domain_allowlist"] = list(self.domains) - return kwargs - - -CLOSED = NetworkGrant() - - -@dataclass(frozen=True) -class NetworkPolicy: - """Per-task network grants; every task not listed gets ``default`` (closed). - - JSON form (``YETO_HARNESS_TB2_NETWORK_POLICY``):: - - {"default": "closed", - "tasks": {"pip-task": {"domains": ["pypi.org", "files.pythonhosted.org"]}, - "web-task": "open"}} - """ - - tasks: dict[str, NetworkGrant] = field(default_factory=dict) - default: NetworkGrant = CLOSED - - def grant(self, task_id: str) -> NetworkGrant: - return self.tasks.get(task_id, self.default) - - @classmethod - def from_mapping(cls, data: Any, *, where: str = "network policy") -> "NetworkPolicy": - if not isinstance(data, dict) or set(data) - {"default", "tasks"}: - raise ValueError(f"{where}: expected {{'default', 'tasks'}}") - tasks = data.get("tasks", {}) - if not isinstance(tasks, dict): - raise ValueError(f"{where}: 'tasks' must be an object") - return cls( - tasks={str(k): NetworkGrant.parse(v, where=f"{where} task {k}") for k, v in tasks.items()}, - default=NetworkGrant.parse(data.get("default", "closed"), where=f"{where} default"), - ) - - @classmethod - def builtin(cls) -> "NetworkPolicy": - from .tb2_network_grants import TB2_OPEN_EGRESS_TASKS - - return cls(tasks={t: NetworkGrant(open=True) for t in TB2_OPEN_EGRESS_TASKS}) - - @classmethod - def from_env(cls) -> "NetworkPolicy": - path = os.environ.get(NETWORK_POLICY_ENV) - if not path: - return cls.builtin() - import json - - return cls.from_mapping(json.loads(Path(path).read_text()), where=path) - - -# ----------------------------------------------------------------------------- backend protocol - - -@dataclass(frozen=True) -class ExecResult: - exit_code: int - output: str - timed_out: bool = False - - -class SandboxHandle(Protocol): - """One running sandbox. ``exec`` is synchronous (run via ``asyncio.to_thread``).""" - - workdir: str - - def exec(self, command: str, *, timeout_s: float, workdir: str | None = None) -> ExecResult: ... - - def terminate(self) -> None: ... - - def alive(self) -> bool: ... - - -class SandboxBackend(Protocol): - def create(self, task: Tb2Task, trajectory_id: str) -> SandboxHandle: ... - - -def shell_with_limits(command: str, *, timeout_s: float, output_bytes: int) -> str: - """Wrap an agent command: wall clock via coreutils ``timeout``, output capped. - - Exit code 124 (``timeout``) / 137 (KILL after grace) mean the clock expired; - stdout and stderr are merged and capped at ``output_bytes + 1`` so the relay - can tell truncation from an exact fit without ever reading more. - """ - seconds = max(1, int(timeout_s)) - return ( - f"timeout -k 2 {seconds} bash -c {shlex.quote(command)} 2>&1 | head -c {int(output_bytes) + 1}; " - "exit ${PIPESTATUS[0]}" - ) - - -_TIMEOUT_EXIT_CODES = frozenset({124, 137}) - - -# ----------------------------------------------------------------------------- local backend - - -# Only these parent variables reach a local sandbox command (never tokens). -LOCAL_ENV_PASSTHROUGH = ("PATH", "LANG", "LC_ALL", "TZ") -DEFAULT_PATH = "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" - - -def minimal_env(parent: dict[str, str] | None = None) -> dict[str, str]: - """The base environment of a local sandbox command. - - The parent environment is not inherited: it holds Modal / HF / W&B - tokens and the reward HMAC key. Only ``LOCAL_ENV_PASSTHROUGH`` is - copied; ``PATH`` falls back to a standard value. - """ - parent = os.environ if parent is None else parent - env = {k: parent[k] for k in LOCAL_ENV_PASSTHROUGH if parent.get(k)} - env.setdefault("PATH", DEFAULT_PATH) - env.setdefault("LANG", "C.UTF-8") - return env - - -class LocalProcessSandbox: - """Commands run on this host in a scratch root; ``/tests`` and ``/logs`` are - redirected into that root through ``TB2_TESTS_DIR`` / ``TB2_VERIFIER_LOGS_DIR``. - - Commands get ``minimal_env`` only. The network grant is recorded but NOT - enforced: this backend is for CPU tests and dev on a trusted host. - """ - - def __init__(self, root: Path, workdir: str, network: NetworkGrant = CLOSED) -> None: - self.root = root - self.workdir = workdir - self.network = network - (root / workdir.lstrip("/")).mkdir(parents=True, exist_ok=True) - self._alive = True - self._processes: set[subprocess.Popen] = set() - self._lock = threading.Lock() - - def host_path(self, sandbox_path: str) -> Path: - return self.root / sandbox_path.lstrip("/") - - def exec(self, command: str, *, timeout_s: float, workdir: str | None = None) -> ExecResult: - if not self._alive: - raise RuntimeError("sandbox is gone") - cwd = self.host_path(workdir or self.workdir) - cwd.mkdir(parents=True, exist_ok=True) - env = { - **minimal_env(), - "TB2_TESTS_DIR": str(self.host_path(TESTS_PATH)), - "TB2_VERIFIER_LOGS_DIR": str(self.host_path(VERIFIER_LOGS_PATH)), - "HOME": str(self.root), - } - proc = subprocess.Popen( - ["bash", "-lc", command], cwd=cwd, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, - start_new_session=True, - ) - with self._lock: - self._processes.add(proc) - try: - try: - out, _ = proc.communicate(timeout=timeout_s + 5) - except subprocess.TimeoutExpired: - self._kill(proc) - out, _ = proc.communicate() - return ExecResult(exit_code=124, output=(out or b"").decode("utf-8", "replace"), timed_out=True) - finally: - with self._lock: - self._processes.discard(proc) - output = (out or b"").decode("utf-8", "replace") - return ExecResult(exit_code=proc.returncode, output=output, timed_out=proc.returncode in _TIMEOUT_EXIT_CODES) - - @staticmethod - def _kill(proc: subprocess.Popen) -> None: - try: - os.killpg(proc.pid, signal.SIGKILL) - except ProcessLookupError: - pass - - def terminate(self) -> None: - """Like a sandbox kill: every command still running dies with the sandbox.""" - self._alive = False - with self._lock: - running = list(self._processes) - for proc in running: - self._kill(proc) - shutil.rmtree(self.root, ignore_errors=True) - - def alive(self) -> bool: - return self._alive and self.root.exists() - - -class LocalProcessBackend: - """CPU backend: a scratch directory per sandbox, commands run as subprocesses. - - ``setup`` (optional) seeds the workdir from the task (tests use it to put - the fixture the verifier checks into place). - """ - - def __init__(self, base_dir: Path | None = None, *, setup: Callable[[Tb2Task, LocalProcessSandbox], None] | None = None, - network_policy: NetworkPolicy | None = None) -> None: - self.base_dir = Path(base_dir) if base_dir else Path(tempfile.gettempdir()) / "yeto-tb2-local" - self.network_policy = network_policy or NetworkPolicy() - self.setup = setup - self.created: list[LocalProcessSandbox] = [] - - def create(self, task: Tb2Task, trajectory_id: str) -> LocalProcessSandbox: - self.base_dir.mkdir(parents=True, exist_ok=True) - root = Path(tempfile.mkdtemp(prefix=f"{task.task_id}-", dir=self.base_dir)) - sandbox = LocalProcessSandbox(root, task.workdir, self.network_policy.grant(task.task_id)) - if self.setup is not None: - self.setup(task, sandbox) - self.created.append(sandbox) - return sandbox - - -# ----------------------------------------------------------------------------- Modal backend - - -def _decode_output(data: bytes | str | None) -> str: - if not data: - return "" - return data if isinstance(data, str) else data.decode("utf-8", errors="replace") - - -class ModalSandbox: - def __init__(self, sandbox: Any, workdir: str) -> None: - self._sandbox = sandbox - self.workdir = workdir - self.object_id = getattr(sandbox, "object_id", "") - - def exec(self, command: str, *, timeout_s: float, workdir: str | None = None) -> ExecResult: - # Bytes, decoded leniently: command output cut by the byte limit (or plain - # binary output) can end mid UTF-8 sequence, and Modal's text mode then - # raises UnicodeDecodeError inside the relay (S17 M1, 5 tool calls failed). - process = self._sandbox.exec( - "bash", "-lc", command, workdir=workdir or self.workdir, timeout=int(timeout_s) + 30, text=False - ) - output = _decode_output(process.stdout.read()) - err = _decode_output(process.stderr.read()) - code = process.wait() - return ExecResult(exit_code=int(code), output=output + err, timed_out=code in _TIMEOUT_EXIT_CODES) - - def terminate(self) -> None: - try: - self._sandbox.terminate() - finally: - detach = getattr(self._sandbox, "detach", None) - if callable(detach): - detach() - - def alive(self) -> bool: - try: - return self._sandbox.poll() is None - except Exception: # noqa: BLE001 - a sandbox the API no longer knows is gone - return False - - -SANDBOX_MODAL_TOKEN_ID_ENV = "YETO_SANDBOX_MODAL_TOKEN_ID" -SANDBOX_MODAL_TOKEN_SECRET_ENV = "YETO_SANDBOX_MODAL_TOKEN_SECRET" - - -def sandbox_modal_client(environ=None) -> Any: - """secret-handling-hardening: a Modal client built from the sandbox-only - token (YETO_SANDBOX_MODAL_TOKEN_*), or None when it is not set (then - Modal's default credentials apply, e.g. on a developer machine).""" - environ = os.environ if environ is None else environ - token_id = environ.get(SANDBOX_MODAL_TOKEN_ID_ENV) - token_secret = environ.get(SANDBOX_MODAL_TOKEN_SECRET_ENV) - if not (token_id and token_secret): - return None - from modal import Client - - return Client.from_credentials(token_id, token_secret) - - -class ModalSandboxBackend: - """Modal Sandbox from the task's official image. All ``modal`` imports are lazy.""" - - def __init__( - self, - *, - app_name: str = DEFAULT_MODAL_APP, - ttl_s: int = DEFAULT_SANDBOX_TTL_S, - idle_timeout_s: int = DEFAULT_IDLE_TIMEOUT_S, - run_id: str | None = None, - network_policy: NetworkPolicy | None = None, - ) -> None: - self.app_name = app_name - # Closed for every task unless a policy grants it (task 5.1). - self.network_policy = network_policy or NetworkPolicy() - self.ttl_s = int(ttl_s) - self.idle_timeout_s = int(idle_timeout_s) - self.run_id = run_id or os.environ.get("OPENENV_RUN_ID") or "" - self._app: Any = None - self._client: Any = None - self._lock = threading.Lock() - - def _client_kwargs(self) -> dict[str, Any]: - """``{"client": c}`` for the sandbox-only token, ``{}`` otherwise.""" - with self._lock: - if self._client is None: - self._client = sandbox_modal_client() or False - return {"client": self._client} if self._client else {} - - def _get_app(self) -> Any: - kwargs = self._client_kwargs() - with self._lock: - if self._app is None: - from modal import App - - self._app = App.lookup(self.app_name, create_if_missing=True, **kwargs) - return self._app - - def create(self, task: Tb2Task, trajectory_id: str) -> ModalSandbox: - from modal import Image, Sandbox - - tags = {"yeto-tb2-task": task.task_id, "yeto-trajectory": trajectory_id[:63]} - if self.run_id: - tags["yeto-run-id"] = self.run_id[:63] - sandbox = Sandbox.create( - "sleep", - "infinity", - app=self._get_app(), - image=Image.from_registry(task.docker_image), - timeout=self.ttl_s, - idle_timeout=self.idle_timeout_s, - cpu=float(task.cpus), - memory=int(task.memory_mb), - workdir=task.workdir, - tags=tags, - **self._client_kwargs(), - **self.network_policy.grant(task.task_id).modal_kwargs(), - ) - return ModalSandbox(sandbox, task.workdir) - - -# ----------------------------------------------------------------------------- relay environment + verifier - - -class EpisodeBinding: - """The one episode a sandbox serves: bound by the first request on the wire. - - The agent function mints the episode id after ``acquire`` (the worker job - carries it), so the lease cannot know it up front; it binds to the first id - it sees and refuses every other one (one sandbox == one episode). - """ - - def __init__(self) -> None: - self.episode_id: str | None = None - - def bind(self, episode_id: str) -> None: - if self.episode_id is None: - self.episode_id = episode_id - elif episode_id != self.episode_id: - raise EpisodeAPIError(404, "unknown_episode", "unknown episode") - - -class RelayTerminalEnvironment: - """``TerminalEnvironment`` served in the trusted process; commands go to the sandbox.""" - - def __init__(self, handle: SandboxHandle, binding: EpisodeBinding | None = None) -> None: - self._handle = handle - self.binding = binding or EpisodeBinding() - self.commands: list[str] = [] - self.submissions: list[dict[str, Any]] = [] - self.submitted = False - - def _check_episode(self, episode_id: str) -> None: - self.binding.bind(episode_id) - if self.submitted: - raise EpisodeAPIError(409, "episode_terminal", "episode already submitted") - - async def execute(self, episode_id: str, command: str, timeout_seconds: float, output_bytes: int) -> dict[str, Any]: - self._check_episode(episode_id) - self.commands.append(command) - result = await asyncio.to_thread( - self._handle.exec, - shell_with_limits(command, timeout_s=timeout_seconds, output_bytes=output_bytes), - timeout_s=timeout_seconds, - ) - truncated = len(result.output.encode("utf-8")) > output_bytes - return { - "exit_code": result.exit_code, - "output": result.output[:output_bytes], - "timed_out": result.timed_out, - "truncated": truncated, - } - - async def submit(self, episode_id: str, submission: dict[str, Any]) -> dict[str, Any]: - self._check_episode(episode_id) - self.submissions.append(dict(submission)) - self.submitted = True - return {"accepted": True} - - -# S17 G3: Modal rejects an exec whose argv exceeds 64 KiB (ARG_MAX), and 7 of -# the 89 TB2 tasks ship tests/ larger than that once tar+base64'd. Small tests -# stay inline (command unchanged); larger ones are first appended to a staging -# file in the sandbox in chunks (``verifier_stage_commands``), and the verifier -# command decodes that file instead. -MAX_INLINE_TESTS_B64 = 48_000 -TESTS_STAGE_CHUNK = 48_000 -TESTS_STAGE_PATH = "/tmp/yeto-tb2-tests.tgz.b64" - - -def verifier_stage_commands(task: Tb2Task) -> list[str]: - """Commands to run (in order) before ``verifier_command``; empty when inline.""" - b64 = tests_tar_b64(task.tests_dir) - if len(b64) <= MAX_INLINE_TESTS_B64: - return [] - commands = [f": > {TESTS_STAGE_PATH}"] - commands += [f"printf %s {b64[i:i + TESTS_STAGE_CHUNK]} >> {TESTS_STAGE_PATH}" - for i in range(0, len(b64), TESTS_STAGE_CHUNK)] - return commands - - -def verifier_command(task: Tb2Task) -> str: - """Stage ``tests/`` at ``/tests``, run ``test.sh`` in the workdir, print the reward. - - The reward line is tagged so it survives arbitrary test output in front of it. - Large ``tests/`` must be staged first with ``verifier_stage_commands``. - """ - b64 = tests_tar_b64(task.tests_dir) - source = (f"echo {b64} | base64 -d" if len(b64) <= MAX_INLINE_TESTS_B64 - else f"base64 -d {TESTS_STAGE_PATH}") - return ( - f"T=${{TB2_TESTS_DIR:-{TESTS_PATH}}}; L=${{TB2_VERIFIER_LOGS_DIR:-{VERIFIER_LOGS_PATH}}}; " - f"rm -rf \"$T\" && mkdir -p \"$T\" \"$L\" && {source} | tar xzf - -C \"$T\" && " - f"bash \"$T/test.sh\"; rc=$?; echo; echo \"YETO_TB2_TESTSH_RC=$rc\"; " - f"echo \"YETO_TB2_REWARD=$(cat \"$L/reward.txt\" 2>/dev/null | tr -d '[:space:]')\"" - ) - - -def stage_tests(handle: "SandboxHandle", task: Tb2Task) -> None: - """Run ``verifier_stage_commands`` in ``handle``; raise if any chunk fails.""" - for command in verifier_stage_commands(task): - result = handle.exec(command, timeout_s=120) - if result.exit_code != 0: - raise RuntimeError(f"staging tests/ failed (rc={result.exit_code}): {result.output[-200:]}") - - -_REWARD_RE = re.compile(r"YETO_TB2_REWARD=(\S*)") -_RC_RE = re.compile(r"YETO_TB2_TESTSH_RC=(\d+)") - - -VERIFIER_LOG_CHARS = 2000 - - -def verifier_log_excerpt(output: str, limit: int = VERIFIER_LOG_CHARS) -> str: - """Last ``limit`` chars of the verifier output (pytest prints its verdict last).""" - text = output if isinstance(output, str) else "" - return text if len(text) <= limit else "…" + text[-(limit - 1):] - - -class Tb2Verifier: - """Trusted-side verifier: ``tests/test.sh`` inside the sandbox, reward from ``reward.txt``.""" - - def __init__(self, handle: SandboxHandle, task: Tb2Task, binding: EpisodeBinding | None = None) -> None: - self._handle = handle - self._task = task - self.binding = binding or EpisodeBinding() - self.evaluations = 0 - self.last_output = "" - - async def evaluate(self, episode_id: str) -> dict[str, Any]: - bound = self.binding.episode_id - if bound is not None and episode_id != bound: - raise ValueError(f"verifier bound to {bound!r}, asked for {episode_id!r}") - self.evaluations += 1 - await asyncio.to_thread(stage_tests, self._handle, self._task) - result = await asyncio.to_thread( - self._handle.exec, verifier_command(self._task), timeout_s=self._task.verifier_timeout_s - ) - self.last_output = result.output[-MAX_VERIFIER_OUTPUT_BYTES:] - reward = _REWARD_RE.search(result.output) - rc = _RC_RE.search(result.output) - testsh_rc = int(rc.group(1)) if rc else result.exit_code - return { - "passed": bool(reward) and reward.group(1) == "1", - "testsh_rc": testsh_rc, - "timed_out": result.timed_out, - # Observe only (S15 verifier probe): the tail of test.sh output - # (pytest summary / install errors); never part of the signed outcome. - "log": verifier_log_excerpt(result.output), - } - - -# ----------------------------------------------------------------------------- faults - - -@dataclass(frozen=True) -class FaultSpec: - kind: str - selector: str - value: str | None = None - - def matches(self, ordinal: int, trajectory_id: str) -> bool: - return self.selector == str(ordinal) or self.selector == trajectory_id - - -def parse_faults(spec: str | None) -> tuple[FaultSpec, ...]: - faults: list[FaultSpec] = [] - for raw in (spec or "").split(","): - item = raw.strip() - if not item: - continue - kind, sep, rest = item.partition(":") - if not sep or kind not in FAULT_KINDS: - raise ValueError(f"{FAULT_ENV}: invalid fault {item!r} (kinds: {', '.join(FAULT_KINDS)})") - selector, _sep, value = rest.partition("=") - if not selector: - raise ValueError(f"{FAULT_ENV}: fault {item!r} needs a selector (acquire ordinal or trajectory id)") - faults.append(FaultSpec(kind, selector, value or None)) - return tuple(faults) - - -class InjectedCreateFailure(RuntimeError): - """Raised by ``acquire`` when ``create_fail`` selects the trajectory.""" - - injected_fault = True # not a provider outage (subprocess agent fail-fast counter) - - -# ----------------------------------------------------------------------------- provider - - -@dataclass -class Tb2Lease: - """Returned by ``acquire``: field names match ``subprocess_agent.EnvironmentLease``.""" - - env_url: str - env_token: str - verifier: Tb2Verifier - destroy: Callable[[], Any] - describe: Callable[[], Any] - deadline_seconds: float - worker_env: dict[str, str] | None = None - task: Tb2Task | None = None - environment: RelayTerminalEnvironment | None = None - handle: Any = None - - -class Tb2EnvironmentProvider: - def __init__( - self, - backend: SandboxBackend, - tasks_dir: Path, - *, - lease_seconds: float | None = None, - faults: tuple[FaultSpec, ...] = (), - host: str = "127.0.0.1", - ) -> None: - self.backend = backend - self.tasks_dir = Path(tasks_dir) - self.lease_seconds = lease_seconds - self.faults = tuple(faults) - self.host = host - self.acquired = 0 - self.destroyed = 0 - self.live: dict[str, Tb2Lease] = {} - self.fault_log: list[tuple[str, str, int]] = [] - - def _faults_for(self, ordinal: int, trajectory_id: str) -> dict[str, str | None]: - hits = {f.kind: f.value for f in self.faults if f.matches(ordinal, trajectory_id)} - for kind in hits: - self.fault_log.append((kind, trajectory_id, ordinal)) - return hits - - async def acquire(self, task_id: str, trajectory_id: str) -> Tb2Lease: - ordinal = self.acquired - self.acquired += 1 - task = resolve_task(task_id, self.tasks_dir) - faults = self._faults_for(ordinal, trajectory_id) - if "create_fail" in faults: - raise InjectedCreateFailure(f"injected sandbox creation failure for {trajectory_id!r} (acquire #{ordinal})") - handle = await asyncio.to_thread(self.backend.create, task, trajectory_id) - token = secrets.token_urlsafe(24) - binding = EpisodeBinding() - relay = RelayTerminalEnvironment(handle, binding) - try: - runner, url = await serve_environment(relay, token, host=self.host) - except BaseException: - await asyncio.to_thread(handle.terminate) - raise - deadline = self.lease_seconds if self.lease_seconds is not None else task.agent_timeout_s - # 2026-10-07: TB2 episodes run under the signed TB2 system prompt. - worker_env: dict[str, str] | None = tb2_instructions_env() - if "deadline" in faults: - deadline = float(faults["deadline"] or 1.0) - if "max_turns" in faults: - worker_env = {**(worker_env or {}), TURN_BUDGET_ENV: str(int(faults["max_turns"] or 2))} - state = {"destroyed": False} - - async def destroy() -> None: - if state["destroyed"]: - return - state["destroyed"] = True - try: - await runner.cleanup() - finally: - self.live.pop(trajectory_id, None) - self.destroyed += 1 - await asyncio.to_thread(handle.terminate) - - async def describe() -> str: - if not state["destroyed"]: - return "live" - # Modal reports the exit asynchronously; a short bounded wait keeps - # ``describe`` honest without stalling the rollout. - for _ in range(20): - if not await asyncio.to_thread(handle.alive): - return "gone" - await asyncio.sleep(0.25) - return "live" - - lease = Tb2Lease( - env_url=url, - env_token=token, - verifier=Tb2Verifier(handle, task, binding), - destroy=destroy, - describe=describe, - deadline_seconds=float(deadline), - worker_env=worker_env, - task=task, - environment=relay, - handle=handle, - ) - self.live[trajectory_id] = lease - return lease - - -# ----------------------------------------------------------------------------- factories (module:callable) - - -def _env_float(name: str, default: float | None) -> float | None: - raw = os.environ.get(name) - if raw is None or not raw.strip(): - return default - value = float(raw) - if value <= 0: - raise ValueError(f"{name} must be positive") - return value - - -def _tasks_dir_from_env() -> Path: - raw = os.environ.get(TASKS_DIR_ENV) - if not raw: - raise RuntimeError(f"{TASKS_DIR_ENV} is not set (terminal-bench-2 checkout root)") - tasks_dir = Path(raw).expanduser().resolve() - if not tasks_dir.is_dir(): - raise RuntimeError(f"{TASKS_DIR_ENV}={raw}: not a directory") - return tasks_dir - - -def _provider(backend: SandboxBackend) -> Tb2EnvironmentProvider: - return Tb2EnvironmentProvider( - backend, - _tasks_dir_from_env(), - lease_seconds=_env_float(LEASE_SECONDS_ENV, None), - faults=parse_faults(os.environ.get(FAULT_ENV)), - host=os.environ.get(ENV_HOST_ENV, "127.0.0.1"), - ) - - -def require_modal_client() -> None: - """The Modal client must import in *this* interpreter (the island's python). - - The ports image does not ship it; Modal mounts its own ``modal`` package - into Function containers without its dependencies, so a bare ``import - modal`` there fails only at first Sandbox use, inside every rollout - worker (A-T3-5, codex-smoke-20261003-7). The launcher installs the client - in the island setup (``MODAL_CLIENT_SETUP``); this check makes the driver - preflight fail closed before any GPU work when that did not happen. - """ - try: - import modal # noqa: F401 - except Exception as exc: # noqa: BLE001 - ImportError or Modal's own re-raise - raise RuntimeError( - f"the Modal Sandbox backend needs an importable `modal` client in {sys.executable}: {exc}" - ) from exc - - -def modal_provider(miles_args: Any = None) -> Tb2EnvironmentProvider: - """``YETO_HARNESS_ENVIRONMENT_PROVIDER=yeto.rl.harness.codex.tb2_provider:modal_provider``.""" - del miles_args - require_modal_client() - backend = ModalSandboxBackend( - app_name=os.environ.get(MODAL_APP_ENV, DEFAULT_MODAL_APP), - ttl_s=int(_env_float(SANDBOX_TTL_ENV, DEFAULT_SANDBOX_TTL_S) or DEFAULT_SANDBOX_TTL_S), - idle_timeout_s=int(_env_float(IDLE_TIMEOUT_ENV, DEFAULT_IDLE_TIMEOUT_S) or DEFAULT_IDLE_TIMEOUT_S), - network_policy=NetworkPolicy.from_env(), - ) - return _provider(backend) - - -def local_provider(miles_args: Any = None) -> Tb2EnvironmentProvider: - """CPU-only provider (commands run on this host): smoke tests and dev.""" - del miles_args - return _provider(LocalProcessBackend(network_policy=NetworkPolicy.from_env())) diff --git a/yeto/rl/harness/codex/tbench_reward.py b/yeto/rl/harness/codex/tbench_reward.py deleted file mode 100644 index a8df347b5..000000000 --- a/yeto/rl/harness/codex/tbench_reward.py +++ /dev/null @@ -1,130 +0,0 @@ -"""Reward and group filter for Terminal-Bench Codex samples on the ports path. - -- ``reward_func``: only structurally valid ``tbench_outcome`` rewards (first of - the three validation points, design D7). An infrastructure marker - marks the sample ABORTED and returns the numeric 0.0 Miles requires; the - group filter never lets it into training. A verified outcome also sets - ``sample.metadata["success"]`` to the outcome ``passed`` bit (all - Terminal-Bench tests passed); aborted samples get no flag. -- ``check_group``: validates every sample again, drops groups with aborted - members, and enforces R-D5a: samples of one trajectory (siblings) share one - reward and are counted once (``rollout_id``/``trajectory_id``). -""" - -from __future__ import annotations - -from typing import Any - -from yeto.rl.rewards.types import RewardResult, Trajectory -from yeto.rl.tbench_outcome import OUTCOME_KEY, UntrustedTBenchOutcome, verified_outcome - -INFRASTRUCTURE_KEY = "tbench_infrastructure_error" - - -def _metadata(sample: Any) -> dict[str, Any]: - metadata = getattr(sample, "metadata", None) - if not isinstance(metadata, dict): - raise UntrustedTBenchOutcome("sample has no metadata") - return metadata - - -def _mark_aborted(sample: Any) -> None: - try: - from miles.utils.types import Sample - - sample.status = Sample.Status.ABORTED - except ImportError: # CPU tests without the Miles package - sample.status = "ABORTED" - - -def is_infrastructure(sample: Any) -> bool: - metadata = _metadata(sample) - return INFRASTRUCTURE_KEY in metadata and OUTCOME_KEY not in metadata - - -def tbench_reward(trajectory: Trajectory) -> RewardResult: - """Neutral form (decoupling 3.3). - - Writes the full-success flag into ``trajectory.metadata`` in place (the - outcome pass bit, not reward > 0). An infrastructure marker is no - verdict: ``success`` is removed and the result is ``aborted`` so the - backend adapter marks the sample with its abort status. - """ - - metadata = trajectory.metadata - if not isinstance(metadata, dict): - raise UntrustedTBenchOutcome("sample has no metadata") - if INFRASTRUCTURE_KEY in metadata and OUTCOME_KEY not in metadata: - metadata.pop("success", None) # no verdict: never a positive - return RewardResult(value=0.0, aborted=True, reason=INFRASTRUCTURE_KEY) - outcome, value = verified_outcome(metadata) - metadata["success"] = outcome["passed"] is True - return RewardResult(value=value) - - -def _sample_reward(sample: Any) -> float: - result = tbench_reward(Trajectory(metadata=_metadata(sample))) - if result.aborted: - _mark_aborted(sample) - return result.value - - -async def reward_func(args: Any, samples: Any, **_kwargs: Any) -> float | list[float]: - del args - if isinstance(samples, list): - return [_sample_reward(sample) for sample in samples] - return _sample_reward(samples) - - -def iter_samples(group: list[Any]) -> list[Any]: - flat: list[Any] = [] - for item in group: - flat.extend(item if isinstance(item, list) else [item]) - return flat - - -def sibling_key(sample: Any) -> Any: - rollout_id = getattr(sample, "rollout_id", None) - if rollout_id is not None: - return ("rollout", rollout_id) - trajectory_id = _metadata(sample).get("trajectory_id") - if trajectory_id is not None: - return ("trajectory", trajectory_id) - return ("row", id(sample)) - - -def check_siblings(samples: list[Any]) -> dict[Any, float]: - """R-D5a: siblings share one reward; returns one reward per trajectory.""" - per_trajectory: dict[Any, float] = {} - outcomes: dict[Any, dict[str, Any]] = {} - for sample in samples: - outcome, value = verified_outcome(_metadata(sample)) - key = sibling_key(sample) - if key in per_trajectory: - if per_trajectory[key] != value or outcomes[key] != outcome: - raise UntrustedTBenchOutcome( - f"sibling samples of {key!r} carry different outcomes" - ) - else: - per_trajectory[key] = value - outcomes[key] = outcome - return per_trajectory - - -def check_group(args: Any, samples: list[Any], **_kwargs: Any): - """Dynamic group filter: validate, drop aborted groups, enforce sibling sharing.""" - try: - from miles.rollout.filter_hub.base_types import DynamicFilterOutput - except ImportError: # CPU tests without the Miles package - from types import SimpleNamespace as DynamicFilterOutput # type: ignore[assignment] - flat = iter_samples(samples) - if any(is_infrastructure(sample) for sample in flat): - for sample in flat: - if is_infrastructure(sample): - _mark_aborted(sample) - return DynamicFilterOutput(keep=False, reason="group_has_aborted") - rewards = check_siblings(flat) # raises on tampering / mismatch - distinct = set(rewards.values()) - if len(distinct) < 2: - return DynamicFilterOutput(keep=False, reason=f"zero_std_{next(iter(distinct)) if distinct else 'empty'}") - return DynamicFilterOutput(keep=True, reason=None) diff --git a/yeto/rl/harness/gateway/proxy.py b/yeto/rl/harness/gateway/proxy.py index ac6e7a342..d6a086040 100644 --- a/yeto/rl/harness/gateway/proxy.py +++ b/yeto/rl/harness/gateway/proxy.py @@ -17,8 +17,10 @@ from __future__ import annotations +import copy import json import secrets +import time from contextvars import ContextVar from typing import Any @@ -144,13 +146,66 @@ def _error(status: int, exc: Exception) -> Any: def _responses_sse(request: dict[str, Any], response: dict[str, Any], number: int) -> bytes: - # Reuse the Codex bridge's event sequence (created, item added/done, completed). - from yeto.rl.harness.codex.codex_harness_agent import _sse_response - - return _sse_response(request={"model": request.get("model"), "tools": request.get("tools") or [], - "prompt_cache_key": request.get("prompt_cache_key")}, - output=response["output"], usage=response["usage"], - response_number=number) + """Responses API event stream: created, per-item added/(arguments)/done, completed. + + One event per output item, no token streaming. ``instructions`` echoes the + request (the client owns its system prompt). + """ + output = response["output"] + completed = { + "id": f"resp_yeto_{number}", + "object": "response", + "created_at": int(time.time()), + "status": "completed", + "error": None, + "incomplete_details": None, + "instructions": request.get("instructions"), + "max_output_tokens": None, + "max_tool_calls": None, + "model": request.get("model"), + "output": output, + "parallel_tool_calls": False, + "previous_response_id": None, + "prompt_cache_key": request.get("prompt_cache_key"), + "reasoning": {"effort": "xhigh", "summary": "none"}, + "safety_identifier": None, + "service_tier": "default", + "store": False, + "temperature": None, + "text": {"format": {"type": "text"}}, + "tool_choice": "auto", + "tools": request.get("tools") or [], + "top_logprobs": 0, + "top_p": None, + "truncation": "disabled", + "usage": response["usage"], + "user": None, + "metadata": {}, + } + events: list[dict[str, Any]] = [ + {"type": "response.created", "response": {**completed, "status": "in_progress", "output": []}}] + for index, item in enumerate(output): + added = copy.deepcopy(item) + added["status"] = "in_progress" + if item["type"] == "function_call": + added["arguments"] = "" + events.append({"type": "response.output_item.added", "output_index": index, "item": added}) + if item["type"] == "function_call": + events.append({"type": "response.function_call_arguments.delta", "item_id": item["id"], + "output_index": index, "delta": item["arguments"]}) + events.append({"type": "response.function_call_arguments.done", "item_id": item["id"], + "output_index": index, "arguments": item["arguments"]}) + events.append({"type": "response.output_item.done", "output_index": index, "item": item}) + events.append({"type": "response.completed", "response": completed}) + chunks = [f"event: {event['type']}\n".encode() + b"data: " + _canonical_json(event) + b"\n\n" + for event in events] + chunks.append(b"data: [DONE]\n\n") + return b"".join(chunks) + + +def _canonical_json(value: Any) -> bytes: + return json.dumps(value, ensure_ascii=False, allow_nan=False, sort_keys=True, + separators=(",", ":")).encode("utf-8") def _chat_sse(completion: dict[str, Any]) -> bytes: diff --git a/yeto/rl/harness/harbor/README.md b/yeto/rl/harness/harbor/README.md index 8c7195e97..5dc706b7e 100644 --- a/yeto/rl/harness/harbor/README.md +++ b/yeto/rl/harness/harbor/README.md @@ -18,8 +18,8 @@ backend: not verified. | verl backend: `HarborAgentLoop(AgentLoopBase)`, registered as `yeto_harbor` | yeto (verl unchanged) | `yeto/rl/adapters/verl/harbor_agent_loop.py` | | Token recorder in front of the sampler (verl) | yeto, N10.5 (not written) | port: `yeto/rl/agentic/recorder.py` | -The self-written Codex harness (`yeto/rl/harness/codex/`) is not deleted (user -decision pending). New code does not import it. +The self-written Codex harness (`yeto/rl/harness/codex/`) was removed in S26 +E0.5; the last commit with it is tagged `s26-pre-harness-removal`. ## Outcome rule diff --git a/yeto/rl/harness/harbor/__init__.py b/yeto/rl/harness/harbor/__init__.py index f77cb09a2..3310a2265 100644 --- a/yeto/rl/harness/harbor/__init__.py +++ b/yeto/rl/harness/harbor/__init__.py @@ -1,8 +1,8 @@ """Harbor (harbor-framework/harbor) as the agent harness of yeto agentic RL (S25 N10.1). -Replaces layers L1/L2/L4/L8/L11 of the self-written Codex harness -(``yeto.rl.harness.codex``): task loading, sandbox, tool exec, verifier, -agent build. New code depends on this package, not on ``harness.codex``. +Harbor owns task loading, sandbox, tool exec, verifier and agent build. The +self-written Codex harness that did this before was removed in S26 E0.5 (git +tag ``s26-pre-harness-removal``). """ from .pins import CODEX_VERSION, HARBOR_COMMIT, HARBOR_VERSION, install_spec, installed_problem diff --git a/yeto/rl/harness/reward_env/__init__.py b/yeto/rl/harness/reward_env/__init__.py index debae3ff5..07f9bfb54 100644 --- a/yeto/rl/harness/reward_env/__init__.py +++ b/yeto/rl/harness/reward_env/__init__.py @@ -1,5 +1,6 @@ """Benchmark-neutral reward environments for agentic RL (change rl-agentic-reward-env). -``benchmark``: adapter protocol, registry, judge runner. ``tb2``: Terminal-Bench 2 -adapter, prebaked Modal sandboxes, held-out split helpers. +``benchmark``: adapter protocol, registry, judge runner, held-out split helpers. +``swebench_verified``: SWE-bench Verified adapter. The Terminal-Bench 2 adapter +left with the self-written Codex harness (S26 E0.5). """ diff --git a/yeto/rl/harness/reward_env/benchmark.py b/yeto/rl/harness/reward_env/benchmark.py index 7044ddf25..2af974dd3 100644 --- a/yeto/rl/harness/reward_env/benchmark.py +++ b/yeto/rl/harness/reward_env/benchmark.py @@ -96,7 +96,7 @@ def to_dict(self) -> dict[str, Any]: class JudgeHandle(Protocol): - """What the judge runner needs from a sandbox (``tb2_provider.SandboxHandle`` fits).""" + """What the judge runner needs from a sandbox.""" def exec(self, command: str, *, timeout_s: float, workdir: str | None = None) -> Any: ... @@ -124,7 +124,7 @@ def parse_judge(self, task_id: str, output: str, exit_code: int, timed_out: bool _REGISTRY: dict[str, Callable[..., BenchmarkAdapter]] = {} # benchmark name -> module of this package that registers it (lazy, no import cost) -_BUILTIN_MODULES = {"tb2": "tb2", "swebench-verified": "swebench_verified"} +_BUILTIN_MODULES = {"swebench-verified": "swebench_verified"} def register(name: str, factory: Callable[..., BenchmarkAdapter]) -> None: diff --git a/yeto/rl/harness/reward_env/tb2.py b/yeto/rl/harness/reward_env/tb2.py deleted file mode 100644 index 5fe2d2c73..000000000 --- a/yeto/rl/harness/reward_env/tb2.py +++ /dev/null @@ -1,252 +0,0 @@ -"""Terminal-Bench 2 adapter for the reward environment (change rl-agentic-reward-env). - -Reuses the task resolution and the judge command of the existing codex TB2 -provider (``yeto.rl.harness.codex.tb2_provider``; imported, never modified) and -adds what that provider lacks: - -- ``prebake_plan``: the dependency setup that each task's ``tests/test.sh`` - performs on every judge run (``apt-get install curl``, the uv installer, - ``uvx -p 3.13 -w pytest==… …`` or ``pip install pytest…``), extracted so it can - run once at image build time. ``test.sh`` itself is never changed: at judge - time it runs the same steps again but they hit the baked caches. -- the Modal app names/env knobs for the prebaked sandboxes; the Modal backend - itself (``PrebakedModalSandboxBackend`` + ``modal_provider``) lives in the - cloud layer ``yeto/cloud/modal_reward_env.py`` and is injected through the - neutral ``tb2_provider.SandboxBackend`` / ``YETO_HARNESS_ENVIRONMENT_PROVIDER``. -- held-out evaluation split helpers (30 tasks by default). -""" - -from __future__ import annotations - -import json -import os -import re -import shlex -from pathlib import Path -from typing import Any - -from yeto.rl.harness.codex import tb2_provider as tb2 - -from .benchmark import PrebakePlan, TaskSpec, register - -NAME = "tb2" -DEFAULT_APP = "yeto-reward-env" -PROTECTED_APPS = frozenset({tb2.DEFAULT_MODAL_APP}) # "yeto-tbench2": production, never built into -APP_ENV = "YETO_REWARD_ENV_MODAL_APP" -PREBAKE_ENV = "YETO_REWARD_ENV_PREBAKE" # "0" disables prebaking (A/B timing) - -_INSTALL_PREFIXES = ("apt-get ", "apt ", "curl ", "wget ", "source ", ". ", "export ", "pip install ", - "pip3 install ", "python -m pip install ", "python3 -m pip install ", "uv ") -_CONTROL = re.compile(r"^(if|then|else|elif|fi|echo|exit|set)\b") - - -def _logical_lines(script: str) -> list[str]: - lines: list[str] = [] - buffer = "" - for raw in script.splitlines(): - stripped = raw.strip() - if not buffer and (not stripped or stripped.startswith("#")): - continue - if stripped.endswith("\\"): - buffer += stripped[:-1] + " " - continue - lines.append((buffer + stripped).strip()) - buffer = "" - if buffer.strip(): - lines.append(buffer.strip()) - return lines - - -_UVX_VALUE_FLAGS = frozenset({"-p", "--python", "-w", "--with", "--from", "--index", "--default-index", - "--index-url", "--extra-index-url", "-i", "--with-requirements", - "--with-editable", "--index-strategy", "--prerelease"}) - - -def _uvx_warmup(line: str) -> str | None: - """``uvx -p 3.13 -w a -w b pytest --ctrf …`` -> ``uvx -p 3.13 -w a -w b pytest --version``.""" - try: - tokens = shlex.split(line) - except ValueError: - return None - if not tokens or tokens[0] != "uvx": - return None - i = 1 - while i < len(tokens): - token = tokens[i] - if token in _UVX_VALUE_FLAGS: - i += 2 - continue - if token.startswith("-"): - i += 1 - continue - return shlex.join(tokens[: i + 1] + ["--version"]) - return None - - -def _is_test_run(line: str) -> bool: - return line.startswith(("uvx ", "uv run ")) or bool(re.match(r"^(python3? -m )?pytest\b", line)) - - -def prebake_from_test_sh(script: str, base_image: str) -> PrebakePlan: - """Setup lines before the test run, verbatim, plus a cache warm-up of the uvx tool env.""" - commands: list[str] = [] - skipped: list[str] = [] - for line in _logical_lines(script): - if _is_test_run(line): - # shlex re-quoting would freeze shell variables (e.g. ${COMMIT_HASH} - # set by a task-specific line): no warm-up then, judge installs it. - warm = None if "$" in line else _uvx_warmup(line) - if warm: - commands.append(warm) - elif line.startswith("uvx "): - skipped.append(line) # tool env not warmed - break - if line.startswith(("curl ", "wget ")) and not re.search(r"\|\s*(ba)?sh\b", line): - skipped.append(line) # a data download into the workdir, not a dependency - elif line.startswith(_INSTALL_PREFIXES): - commands.append(line) - elif _CONTROL.match(line) or line.startswith("[") or line.startswith("cd "): - continue # PWD guards etc.: not dependencies - else: - skipped.append(line) - else: - # no recognisable test-run line: do not guess - return PrebakePlan(base_image=base_image, commands=(), complete=False, skipped=tuple(skipped) or ("",)) - return PrebakePlan(base_image=base_image, commands=tuple(commands), complete=not skipped, skipped=tuple(skipped)) - - -def prebake_script(plan: PrebakePlan) -> str: - """One bash script (env from ``source`` must survive into the warm-up).""" - return "set -e\ncd /tmp\n" + "\n".join(plan.commands) + "\n" - - -def _checkout_version(path: Path) -> str: - import subprocess - - try: - rev = subprocess.run(["git", "-C", str(path), "rev-parse", "--short=7", "HEAD"], capture_output=True, - text=True, timeout=10, check=True).stdout.strip() - except (OSError, subprocess.SubprocessError): - return "tb2@unknown" - return f"tb2@{rev}" if rev else "tb2@unknown" - - -def _official_difficulty(task_dir: Path) -> str: - try: - meta = tb2.tomllib.loads((task_dir / "task.toml").read_text()).get("metadata", {}) - except (OSError, ValueError): - return "unknown" - value = meta.get("difficulty") - return str(value) if value else "unknown" - - -class Tb2Benchmark: - name = NAME - - def __init__(self, tasks_dir: str | os.PathLike[str] | None = None, version: str | None = None) -> None: - raw = tasks_dir or os.environ.get(tb2.TASKS_DIR_ENV) - if not raw: - raise RuntimeError(f"{tb2.TASKS_DIR_ENV} is not set (terminal-bench-2 checkout root)") - self.tasks_dir = Path(raw).expanduser().resolve() - self.version = version or _checkout_version(self.tasks_dir) - - def task_ids(self) -> list[str]: - return sorted(p.parent.name for p in self.tasks_dir.glob("*/task.toml")) - - def _task(self, task_id: str) -> tb2.Tb2Task: - return tb2.resolve_task(task_id, self.tasks_dir) - - def task_spec(self, task_id: str) -> TaskSpec: - t = self._task(task_id) - difficulty = _official_difficulty(t.task_dir) - return TaskSpec(benchmark=NAME, task_id=t.task_id, base_image=t.docker_image, cpus=t.cpus, - memory_mb=t.memory_mb, workdir=t.workdir, agent_timeout_s=t.agent_timeout_s, - judge_timeout_s=t.verifier_timeout_s, instruction=t.instruction, - benchmark_version=self.version, difficulty=difficulty, - difficulty_source="tb2-task.toml", - eval_bucket=f"tb2-{difficulty}" if difficulty in ("easy", "medium", "hard") else None) - - def prebake_plan(self, task_id: str) -> PrebakePlan: - t = self._task(task_id) - return prebake_from_test_sh((t.tests_dir / "test.sh").read_text(), t.docker_image) - - def judge_setup_commands(self, task_id: str) -> list[str]: - """Commands run before ``judge_command`` (stage large tests/, S17 G3).""" - return tb2.verifier_stage_commands(self._task(task_id)) - - def judge_command(self, task_id: str, submission: str | None = None) -> str: - del submission # TB2 judges the sandbox the agent worked in (official stage-at-verify) - return tb2.verifier_command(self._task(task_id)) - - def parse_judge(self, task_id: str, output: str, exit_code: int, timed_out: bool) -> dict[str, Any]: - reward = tb2._REWARD_RE.search(output) - rc = tb2._RC_RE.search(output) - passed = bool(reward) and reward.group(1) == "1" - return {"reward": 1.0 if passed else 0.0, "passed": passed, - "testsh_rc": int(rc.group(1)) if rc else exit_code, - "reward_txt": reward.group(1) if reward else None} - - -register(NAME, Tb2Benchmark) - - -# ----------------------------------------------------------------------------- Modal backend -# The Modal sandbox backend for this adapter lives in the cloud layer -# (``yeto/cloud/modal_reward_env.py``; import boundary: neutral core never imports -# a cloud library). It is injected at launch through the neutral provider -# interface ``YETO_HARNESS_ENVIRONMENT_PROVIDER=yeto.cloud.modal_reward_env:modal_provider`` -# (a ``tb2_provider.SandboxBackend`` behind a ``Tb2EnvironmentProvider``). - - -def check_build_app(app_name: str) -> str: - if app_name in PROTECTED_APPS: - raise ValueError(f"refusing to build into production Modal app {app_name!r}") - return app_name - - -# ----------------------------------------------------------------------------- hold-out (WP3 D2/D6b) - -HOLDOUT_SEED = 20261008 -HOLDOUT_QUOTAS = {"tb2-easy": 2, "tb2-medium": 18, "tb2-hard": 10} -HOLDOUT_RULE_UNUSABLE = ("exclude smoke6 (S15 trained) and data/eval/tb2-unusable.json; " - "stratified by difficulty: easy 2 / medium 18 / hard 10") -HOLDOUT_RULE = "exclude smoke6 (S15 trained); stratified by difficulty: easy 2 / medium 18 / hard 10" -# WP3 rl-eval-difficulty-buckets D2 (#123 c8979bd2): the S15 smoke tasks -# (codex-bundle/data/tbench2_smoke6.jsonl) were trained on in both S15 GPU runs, -# so they leave the eval pool before the stratified draw (they may stay in training). -SMOKE6_TASK_IDS = ("fix-git", "regex-log", "sqlite-db-truncate", "log-summary-date-ranges", - "openssl-selfsigned-cert", "git-multibranch") -SMOKE6_REASON = "S15 smoke training" - - -def smoke6_exclusions() -> dict[str, str]: - return {t: SMOKE6_REASON for t in SMOKE6_TASK_IDS} - - -UNUSABLE_SCHEMA = "yeto.tb2-unusable/v1" -# data/eval/tb2-unusable.json: tasks whose official solution does not score 1 in -# our judge env (S17 G3 positive control). They leave training *and* the eval pool. -UNUSABLE_DEFAULT_PATH = Path(__file__).resolve().parents[4] / "data" / "eval" / "tb2-unusable.json" - - -def load_unusable(path: str | os.PathLike[str] | None = None) -> dict[str, str]: - """``task_id -> "unusable/: "`` from the unusable table (fail closed).""" - data = json.loads(Path(path or UNUSABLE_DEFAULT_PATH).read_text()) - if data.get("schema") != UNUSABLE_SCHEMA or data.get("benchmark") != "tb2": - raise ValueError(f"{path or UNUSABLE_DEFAULT_PATH}: not a {UNUSABLE_SCHEMA} tb2 file") - out: dict[str, str] = {} - for item in data["items"]: - tid, cat, why = item["task_id"], item["category"], item["reason"] - if not (tid and cat and why) or tid in out: - raise ValueError(f"bad or duplicate unusable entry: {item!r}") - out[tid] = f"unusable/{cat}: {why}" - return out - - -def build_holdout(adapter: "Tb2Benchmark", *, exclude: Any = None, **kwargs: Any) -> dict[str, Any]: - """TB2 hold-out; ``exclude`` defaults to the smoke-6 tasks (pass ``{}`` for none).""" - from .benchmark import build_holdout as _build - - kwargs.setdefault("rule", HOLDOUT_RULE) - return _build(adapter, HOLDOUT_QUOTAS, seed=HOLDOUT_SEED, - exclude=smoke6_exclusions() if exclude is None else exclude, **kwargs) diff --git a/yeto/rl/harness/session_protocol.md b/yeto/rl/harness/session_protocol.md index fc417f850..17f3e9cda 100644 --- a/yeto/rl/harness/session_protocol.md +++ b/yeto/rl/harness/session_protocol.md @@ -1,5 +1,7 @@ # 会话服务协议（decoupling 任务 3.8，spec `rl-neutral-rewards-harness`） +> S26 E0.5: the Codex harness this file describes was removed (tag `s26-pre-harness-removal`); the protocol stays as the contract of the test double. + codex harness 对"会话服务"的全部依赖。Miles 会话服务是这个协议的一种实现；以后换框架（如 verl）时，只要提供同样的五条路径，harness 不用改。测试替身：`tests/session_service_double.py`。 本协议只写文档与测试替身，**没有**实现 verl 侧。 diff --git a/yeto/rl/island_tasks.py b/yeto/rl/island_tasks.py index 89e485749..524979fe1 100644 --- a/yeto/rl/island_tasks.py +++ b/yeto/rl/island_tasks.py @@ -14,7 +14,7 @@ A token ``-`` names the tasks ```` ... ```` unless the token itself is a task id. A row's task id is -``metadata.task_id`` (what ``yeto.rl.rlf.catalog.prompt_rows`` writes) or a +``metadata.task_id`` or a top-level ``task_id``. This module is backend-neutral: it knows JSONL rows and task ids only. diff --git a/yeto/rl/rlf/__init__.py b/yeto/rl/rlf/__init__.py deleted file mode 100644 index 680abf802..000000000 --- a/yeto/rl/rlf/__init__.py +++ /dev/null @@ -1,10 +0,0 @@ -"""Small, trusted-side bridge for rlenvfactory CTF tasks.""" -from .contracts import ( - AdmissionError, TaskAdmission, admit_task_package, TrustedGrade, - EpisodeLease, RewardEvidence, finish_episode, rl_gate, -) -from .catalog import TaskCatalog, import_catalog, prompt_rows, write_prompt_data, task_statement, rlf_identity -from .preflight import RolloutRejected, validate_rollout, trusted_grade_from_evaluation -from .trajectory import CodexTrajectory, from_codex_result -__all__ = ["prompt_rows", "write_prompt_data", "task_statement", "rlf_identity", "trusted_grade_from_evaluation","AdmissionError", "TaskAdmission", "admit_task_package", "TrustedGrade", "EpisodeLease", "RewardEvidence", "finish_episode", "rl_gate", "TaskCatalog", "import_catalog", "RolloutRejected", "validate_rollout", "CodexTrajectory", "from_codex_result"] -from .modal import RlfModalVmConfig, RlfVmLease, acquire_vm diff --git a/yeto/rl/rlf/catalog.py b/yeto/rl/rlf/catalog.py deleted file mode 100644 index 705f9aec4..000000000 --- a/yeto/rl/rlf/catalog.py +++ /dev/null @@ -1,116 +0,0 @@ -from __future__ import annotations -import json, uuid -from dataclasses import dataclass, field -from pathlib import Path -from typing import Any -from .contracts import AdmissionError, EpisodeLease, TaskAdmission, admit_task_package - -@dataclass -class TaskCatalog: - tasks: dict[str, TaskAdmission] - roots: dict[str, Path] = field(default_factory=dict) - def add(self, package: str|Path, **kwargs: Any) -> TaskAdmission: - item = admit_task_package(package, **kwargs) - if item.task_id in self.tasks: raise ValueError(f'duplicate RLF task: {item.task_id}') - self.tasks[item.task_id] = item - self.roots[item.task_id] = Path(package) - return item - def get(self, task_id: str) -> TaskAdmission: - return self.tasks[task_id] - def episode(self, task_id: str, *, trajectory_id: str|None=None) -> EpisodeLease: - """Start one rollout episode for an admitted task (RLF identity producer).""" - admission = self.get(task_id) - return EpisodeLease(task_id=admission.task_id, - trajectory_id=trajectory_id or f'rlf-traj-{uuid.uuid4().hex}') - -def import_catalog(packages: list[str|Path], **kwargs: Any) -> TaskCatalog: - catalog=TaskCatalog({}) - for package in packages: catalog.add(package, **kwargs) - return catalog - -RLF_IDENTITY_KEYS = ('rlf_task_id', 'rlf_episode_id', 'rlf_trajectory_id', 'policy_id') - -def rlf_identity(lease: EpisodeLease, *, policy_id: str|None=None) -> dict[str, str]: - """The RLF identity quadruple for one episode lease (prompt/chain metadata).""" - identity = {'rlf_task_id': lease.task_id, 'rlf_episode_id': lease.episode_id, - 'rlf_trajectory_id': lease.trajectory_id} - if policy_id is not None: identity['policy_id'] = str(policy_id) - return identity - -def task_statement(catalog: TaskCatalog, task_id: str) -> str: - """The task's own statement (public README); never fabricated.""" - root = catalog.roots.get(task_id) - if root is None: raise AdmissionError(f'no package root recorded for task {task_id!r}') - for name in ('README.md', 'README'): - candidate = root/'public'/name - if candidate.is_file(): - text = candidate.read_text().strip() - if text: return text - raise AdmissionError(f'task {task_id!r} has no public README task statement') - -def target_endpoint(catalog: TaskCatalog, task_id: str) -> str | None: - """The agent-visible target address from the package's task.yaml. - - RLF CTF targets run on the episode VM's host network, so the address is - always 127.0.0.1:. Returns None when the package declares no - port (non-network task). - """ - root = catalog.roots.get(task_id) - if root is None: - return None - text = (root/'task.yaml').read_text() - for line in text.splitlines(): - if line.startswith('target_port:'): - port = line.split(':', 1)[1].strip() - return f'127.0.0.1:{port}' if port else None - return None - - -def prompt_rows(catalog: TaskCatalog, task_ids: list[str]|None=None, *, repeat: int=1) -> list[dict[str, Any]]: - """Catalog -> Miles ``--prompt-data`` rows (the catalog consumer for rollout). - - Each row carries ``metadata.task_id`` (read by the harness ``task_identity``) - and ``rlf_task_id``; the per-rollout episode/trajectory ids are produced - later by ``codex_openenv_generate.produce_rlf_identity`` so episodes are - never shared across rollouts. - """ - rows: list[dict[str, Any]] = [] - for task_id in (task_ids or list(catalog.tasks)): - admission = catalog.get(task_id) - statement = task_statement(catalog, task_id) - endpoint = target_endpoint(catalog, task_id) - if endpoint: - statement += ('\n\nThe target service listens on ' - + endpoint + ' (reachable from your shell).') - - metadata: dict[str, Any] = {'task_id': admission.task_id, 'rlf_task_id': admission.task_id, - 'benchmark': 'rlf', 'grader_mode': admission.grader_mode, - 'package_digest': admission.package_digest, 'prompt': statement} - if admission.image_digest: metadata['image_digest'] = admission.image_digest - for _ in range(max(1, repeat)): - rows.append({'metadata': dict(metadata), 'prompt': statement}) - return rows - -def write_prompt_data(catalog: TaskCatalog, path: str|Path, task_ids: list[str]|None=None, *, repeat: int=1) -> Path: - """Write ``prompt_rows`` as the JSONL file Miles ``--prompt-data`` expects.""" - path = Path(path) - rows = prompt_rows(catalog, task_ids, repeat=repeat) - if not rows: raise AdmissionError('catalog produced no prompt rows') - path.write_text(''.join(json.dumps(row, ensure_ascii=False)+'\n' for row in rows)) - return path - -def main(argv: list[str]|None=None) -> None: - """``python -m yeto.rl.rlf.catalog PKG... --out prompt.jsonl [--repeat N]``.""" - import argparse - parser = argparse.ArgumentParser(description='RLF catalog -> Miles prompt data') - parser.add_argument('packages', nargs='+') - parser.add_argument('--out', required=True) - parser.add_argument('--repeat', type=int, default=1) - parser.add_argument('--evidence', default=None, help='JSON admission evidence (must include runtime_quality_gate)') - args = parser.parse_args(argv) - evidence = json.loads(args.evidence) if args.evidence else None - catalog = import_catalog(args.packages, evidence=evidence) - write_prompt_data(catalog, args.out, repeat=args.repeat) - -if __name__ == '__main__': - main() diff --git a/yeto/rl/rlf/contracts.py b/yeto/rl/rlf/contracts.py deleted file mode 100644 index 00e0ae695..000000000 --- a/yeto/rl/rlf/contracts.py +++ /dev/null @@ -1,98 +0,0 @@ -from __future__ import annotations -from dataclasses import dataclass, field -from pathlib import Path -from typing import Any, Mapping -import hashlib, json, time, uuid - -class AdmissionError(ValueError): pass - -def _sha(path: Path) -> str: - h=hashlib.sha256() - if path.is_file(): h.update(path.read_bytes()) - else: - for p in sorted(path.rglob('*')): - if p.is_file(): h.update(str(p.relative_to(path)).encode()); h.update(p.read_bytes()) - return h.hexdigest() - -@dataclass(frozen=True) -class TaskAdmission: - task_id: str; grader_mode: str; package_digest: str; image_digest: str|None - source_commit: str|None; evidence: Mapping[str,Any]; manifest: Mapping[str,Any] - -def admit_task_package(root: str|Path, *, evidence: Mapping[str,Any]|None=None, image_digest: str|None=None, source_commit: str|None=None) -> TaskAdmission: - root=Path(root); public=root/'public'; yaml=root/'task.yaml'; manifest=root/'MANIFEST.json' - if not public.is_dir() or not yaml.is_file() or not manifest.is_file(): raise AdmissionError('package must contain public/, task.yaml and MANIFEST.json') - try: m=json.loads(manifest.read_text()); text=yaml.read_text() - except Exception as e: raise AdmissionError(f'invalid package metadata: {e}') from e - task_id=m.get('task_id') or next((x.split(':',1)[1].strip() for x in text.splitlines() if x.startswith('task_id:')),None) - mode=m.get('grader_mode') or next((x.split(':',1)[1].strip() for x in text.splitlines() if x.startswith('grader_mode:')),None) - if not task_id or mode not in {'flag','hacked'}: raise AdmissionError('task_id and grader_mode flag/hacked are required') - ev=dict(evidence or {}) - if not ev.get('runtime_quality_gate'): raise AdmissionError('runtime_quality_gate evidence is required') - if ev.get('agentflow_node_output'): raise AdmissionError('AgentFlow node output is not admission evidence') - return TaskAdmission(task_id, mode, _sha(root), image_digest, source_commit, ev, m) - -@dataclass -class EpisodeLease: - task_id: str; trajectory_id: str; episode_id: str=field(default_factory=lambda: str(uuid.uuid4())); started_at: float=field(default_factory=time.time); closed: bool=False - secret: str|None=None - def teardown(self): self.closed=True; self.secret=None - -@dataclass(frozen=True) -class TrustedGrade: - status: str; reward: float; passed: bool; grader_evidence: Mapping[str,Any]; error_class: str|None=None -@dataclass(frozen=True) -class RewardEvidence: - value: float; metadata: Mapping[str,Any] - -def finish_episode(lease: EpisodeLease, grade: TrustedGrade, *, timed_out: bool=False) -> RewardEvidence: - if lease.closed: raise RuntimeError('episode already torn down') - status='timeout' if timed_out else grade.status - md={'success': bool(grade.passed), 'reward': float(grade.reward), 'rlf_episode_id':lease.episode_id, 'rlf_task_id':lease.task_id, 'rlf_trajectory_id':lease.trajectory_id, 'rlf_status':status, 'grader_evidence':dict(grade.grader_evidence)} - if grade.error_class: md['rejected_data_reason']=grade.error_class - lease.teardown(); return RewardEvidence(float(grade.reward), md) - -def rl_gate(rows: list[Mapping[str,Any]]) -> dict[str,Any]: - rewards=[float(r.get('reward',0)) for r in rows]; adv=[float(r.get('advantage',0)) for r in rows] - total=sum(rewards); mean=total/len(rewards) if rewards else 0.0 - variance=sum((x-mean)**2 for x in rewards)/len(rewards) if rewards else 0.0 - result={'reward_sum':total,'reward_std':variance**0.5,'success_count':sum(bool(r.get('metadata',{}).get('success')) for r in rows),'advantage_abs_sum':sum(abs(x) for x in adv),'policy_update':any(bool(r.get('policy_update')) for r in rows)} - criteria={'non_empty':bool(rows),'reward_std_positive':result['reward_std']>0,'success_count_positive':result['success_count']>0,'advantage_abs_sum_positive':result['advantage_abs_sum']>0,'policy_update':result['policy_update']} - result['passed']=all(criteria.values()) - if not result['passed']: - failed=', '.join(k for k,v in criteria.items() if not v) - raise AdmissionError(f'RL gate failed ({failed}): {result}') - return result - -def group_advantages(rewards: list[float]) -> list[float]: - """GRPO-style group baseline: advantage_i = reward_i - mean(group). - - This is the gate-runner/evidence-entry computation of record for - ``advantage_abs_sum`` on paths where the learner does not export - per-sample advantages; callers must not hand-fill the value. - """ - values=[float(r) for r in rewards] - if not values: return [] - mean=sum(values)/len(values) - return [r-mean for r in values] - -def _params_digest(params: Mapping[str,float]) -> str: - blob=json.dumps({k:float(v) for k,v in sorted(params.items())},sort_keys=True,separators=(',',':')) - return hashlib.sha256(blob.encode()).hexdigest() - -@dataclass(frozen=True) -class PolicyUpdateEvidence: - """Structured proof that an optimizer step changed policy parameters.""" - step: int; updated: bool; before_digest: str; after_digest: str; changed: tuple[str,...] - -def policy_update_evidence(before: Mapping[str,float], after: Mapping[str,float], *, step: int) -> PolicyUpdateEvidence: - """Record a learner update by digesting parameters before and after the step. - - ``updated`` is true only when the parameter snapshot actually changed; - gate rows must read ``policy_update`` from this evidence, never from a - hand-written literal. - """ - keys=set(before)|set(after) - changed=tuple(sorted(k for k in keys if float(before.get(k,0.0))!=float(after.get(k,0.0)))) - bd=_params_digest(before); ad=_params_digest(after) - return PolicyUpdateEvidence(int(step), bd!=ad, bd, ad, changed) diff --git a/yeto/rl/rlf/modal.py b/yeto/rl/rlf/modal.py deleted file mode 100644 index 07124bb36..000000000 --- a/yeto/rl/rlf/modal.py +++ /dev/null @@ -1,579 +0,0 @@ -"""RLF Modal VM lifecycle + CTF environment orchestration (audit breakpoints 3/5). - -Learner GPUs are deliberately never allocated here. - -Two layers, both CPU-testable with a fake VM: - -* VM lifecycle (S20): ``RlfModalVmConfig`` / ``RlfVmLease`` / ``acquire_vm``. -* CTF orchestration (breakpoint 3): the nested-Docker sequence verified in - ``evidence/rlf/modal-ctf-trusted-recvfix2.log`` packaged as functions — - dockerd readiness, ``docker build --network host``, target run with - ``-e FLAG`` (env injection only), health wait, idempotent cleanup. - -Transport decision (代拍板, rationale): the worker-facing wire is served by -``serve_environment`` **in the trusted process** and commands are relayed into -the VM through the Modal exec channel — the exact pattern ``tb2_provider`` -uses — instead of running a token-protected exec HTTP service inside the VM. -Reasons: (1) the bearer token never enters the VM, so a compromised agent -command cannot read it; (2) nothing has to be baked into the VM image beyond -docker; (3) the worker speaks the unchanged ``HttpTerminalEnvironment`` wire. -The agent reaches the CTF target at ``127.0.0.1:31337`` because relay -commands exec in the VM's host network namespace and the target container -runs ``--network host``. - -Flag hygiene: the FLAG reaches the target container only as ``-e FLAG`` on -``docker run``. It is never written into stdout/stderr captures, logs, or -exception messages: every orchestration error passes through :func:`scrub` -with the flag, and :class:`OrchestrationError` never embeds the command line. - -Cleanup order (each step idempotent): solver/grader containers -> target -container -> images are kept (layer cache for the next episode) -> VM -terminate. -""" -from __future__ import annotations - -import base64 -import io -import json -import os -import shlex -import tarfile -import time -import uuid -from dataclasses import dataclass, field, replace -from pathlib import Path -from typing import Any, Callable, Mapping - -from .contracts import EpisodeLease - -# ----------------------------------------------------------------------------- VM lifecycle (S20) - - -@dataclass(frozen=True) -class RlfModalVmConfig: - app_name: str = 'yeto-rlf-env' - cpu: float = 2.0 - memory_mb: int = 4096 - ttl_s: int = 1800 - # Idle kill must not fire below ttl_s: agent turns can spend minutes in model - # generation with no exec running; a shorter idle timeout reclaims the VM - # mid-episode (Modal NotFoundError) and the fail-closed preflight then drops - # the whole group. Default: follow ttl_s (hard cap still bounds leaks). - idle_timeout_s: int | None = None - image: str = 'debian:bookworm-slim' - runtime: str = 'vm' - image_run_commands: tuple = () - - def validate(self): - if self.runtime != 'vm': - raise ValueError('RLF environment must use Modal VM runtime') - if self.cpu <= 0 or self.memory_mb < 512: - raise ValueError('invalid VM resources') - - -class RlfVmLease: - def __init__(self, sandbox: Any, *, task_id: str, episode_id: str, trajectory_id: str | None = None): - self.sandbox = sandbox - self.task_id = task_id - self.episode_id = episode_id - self.closed = False - self.trajectory_id = trajectory_id - - @property - def object_id(self): - return getattr(self.sandbox, 'object_id', '') - - def exec(self, *args, **kwargs): - if self.closed: - raise RuntimeError('RLF VM lease is closed') - return self.sandbox.exec(*args, **kwargs) - - def alive(self) -> bool: - try: - poll = getattr(self.sandbox, 'poll', None) - return bool(poll) and poll() is None - except Exception: # noqa: BLE001 - a sandbox the API no longer knows is gone - return False - - def teardown(self): - if not self.closed: - self.closed = True - self.sandbox.terminate() - - -def acquire_vm(task_id: str, episode_id: str, *, config: RlfModalVmConfig | None = None, image: Any = None, - trajectory_id: str | None = None) -> RlfVmLease: - cfg = config or RlfModalVmConfig() - cfg.validate() - import modal - app = modal.App.lookup(cfg.app_name, create_if_missing=True) - if image is None: - image = modal.Image.from_registry(cfg.image) - if cfg.image_run_commands: - image = image.run_commands(*cfg.image_run_commands) - idle_timeout_s = cfg.idle_timeout_s if cfg.idle_timeout_s is not None else cfg.ttl_s - sb = modal.Sandbox.create('sleep', 'infinity', app=app, image=image, runtime='vm', cpu=cfg.cpu, - memory=cfg.memory_mb, timeout=cfg.ttl_s, idle_timeout=idle_timeout_s, - tags={'yeto-purpose': 'rlf-environment', 'yeto-task': task_id[:63], - 'yeto-episode': episode_id[:63]}) - return RlfVmLease(sb, task_id=task_id, episode_id=episode_id, trajectory_id=trajectory_id) - - -# ----------------------------------------------------------------------------- exec surface (fakeable) - - -@dataclass(frozen=True) -class VmExecResult: - exit_code: int - output: str - timed_out: bool = False - - -# One synchronous VM command: (shell command, timeout) -> captured result. -VmExec = Callable[..., VmExecResult] - -_TIMEOUT_EXIT_CODES = frozenset({124, 137}) - - -def _decode_output(data: bytes | str | None) -> str: - if not data: - return '' - return data if isinstance(data, str) else data.decode('utf-8', errors='replace') - - -def vm_exec(lease: RlfVmLease) -> VmExec: - """Adapt a Modal VM lease to the synchronous ``VmExec`` surface. - - Commands run as ``bash -lc`` in the VM's host network namespace (this is - what lets the agent reach the ``--network host`` CTF target). - """ - - def run(command: str, *, timeout_s: float) -> VmExecResult: - process = lease.exec('bash', '-lc', command, timeout=int(timeout_s) + 30, text=False) - output = _decode_output(process.stdout.read()) + _decode_output(process.stderr.read()) - code = int(process.wait()) - return VmExecResult(code, output, code in _TIMEOUT_EXIT_CODES) - - return run - - -class OrchestrationError(RuntimeError): - """A CTF orchestration step failed. Messages are always flag-scrubbed.""" - - -def scrub(text: Any, *secrets: str | None) -> str: - """Replace every secret occurrence with ``***`` (defense in depth).""" - out = str(text) - for secret in secrets: - if secret: - out = out.replace(str(secret), '***') - return out - - -# ----------------------------------------------------------------------------- CTF task spec - - -@dataclass(frozen=True) -class RlfCtfTask: - """One admitted CTF task package as the orchestrator needs it (trusted side).""" - - task_id: str - package_dir: Path - flag: str - target_port: int = 31337 - grader_mode: str = 'flag' - - -def _task_yaml_field(text: str, name: str) -> str | None: - for line in text.splitlines(): - if line.startswith(name + ':'): - return line.split(':', 1)[1].strip() - return None - - -def resolve_ctf_task(task_id: str, tasks_dir: str | Path, flags: Mapping[str, str]) -> RlfCtfTask: - """Fail closed: unknown task, missing package files, non-flag mode or a - missing trusted flag all refuse the acquire (infrastructure, never a 0 reward).""" - package_dir = Path(tasks_dir) / task_id - yaml_path = package_dir / 'task.yaml' - if not yaml_path.is_file(): - raise OrchestrationError(f'{task_id}: no task.yaml under {tasks_dir}') - if not (package_dir / 'Dockerfile').is_file(): - raise OrchestrationError(f'{task_id}: package has no Dockerfile') - text = yaml_path.read_text() - mode = _task_yaml_field(text, 'grader_mode') or 'flag' - if mode != 'flag': - raise OrchestrationError(f'{task_id}: grader_mode {mode!r} is not packaged (flag only)') - flag = flags.get(task_id) - if not flag: - raise OrchestrationError(f'{task_id}: no trusted flag configured') - port = int(_task_yaml_field(text, 'target_port') or 31337) - return RlfCtfTask(task_id=task_id, package_dir=package_dir, flag=str(flag), target_port=port) - - -# ----------------------------------------------------------------------------- dockerd / build / run / health - -DOCKER_SOCK = 'unix:///run/docker/docker.sock' -_DOCKER = f'docker -H {DOCKER_SOCK}' -REMOTE_TASK_DIR = '/opt/task' -_STAGE_PATH = '/tmp/rlf-task.tgz.b64' -_STAGE_CHUNK = 48_000 -# Only these package entries form the docker build context; ``private/`` -# (reference solution) and admission metadata never enter the VM. -_BUILD_CONTEXT_ENTRIES = ('Dockerfile', 'entrypoint.sh', 'public') - - -def start_dockerd(exec_fn: VmExec, *, attempts: int = 40, sleep_s: float = 1.0, - sleep: Callable[[float], None] = time.sleep) -> None: - """Reset and start nested dockerd, then wait for ``docker info``. - - Parameters verified on Modal (``modal-dockerd-probe.log`` / - ``modal-ctf-trusted-recvfix2.log``): custom socket under /run/docker, - ``--iptables=false --bridge=none`` (the VM has no bridge network). - """ - result = exec_fn( - 'rm -rf /var/lib/docker /run/docker && mkdir -p /run/docker && ' - f'(dockerd --host={DOCKER_SOCK} --iptables=false --bridge=none >/tmp/dockerd.log 2>&1 &)', - timeout_s=30, - ) - if result.exit_code != 0: - raise OrchestrationError(f'dockerd launch failed (rc={result.exit_code}): {result.output[-200:]}') - for _ in range(max(1, attempts)): - if exec_fn(f'{_DOCKER} info >/dev/null 2>&1', timeout_s=15).exit_code == 0: - return - sleep(sleep_s) - raise OrchestrationError(f'dockerd did not become ready after {attempts} attempts') - - -def package_tar_b64(package_dir: str | Path) -> str: - """tar+base64 of the build context only (never ``private/``).""" - root = Path(package_dir) - buf = io.BytesIO() - with tarfile.open(fileobj=buf, mode='w:gz') as tar: - for entry in _BUILD_CONTEXT_ENTRIES: - path = root / entry - if path.is_file(): - tar.add(path, arcname=entry) - elif path.is_dir(): - for item in sorted(path.rglob('*')): - if item.is_file() and '__pycache__' not in item.parts: - tar.add(item, arcname=str(item.relative_to(root))) - return base64.b64encode(buf.getvalue()).decode() - - -def stage_package(exec_fn: VmExec, package_dir: str | Path, *, remote_dir: str = REMOTE_TASK_DIR) -> None: - """Ship the build context into the VM (chunked base64, like the TB2 tests staging).""" - b64 = package_tar_b64(package_dir) - commands = [f'rm -rf {remote_dir} && mkdir -p {remote_dir} && : > {_STAGE_PATH}'] - commands += [f'printf %s {b64[i:i + _STAGE_CHUNK]} >> {_STAGE_PATH}' - for i in range(0, len(b64), _STAGE_CHUNK)] - commands.append(f'base64 -d {_STAGE_PATH} | tar xzf - -C {remote_dir} && rm -f {_STAGE_PATH}') - for command in commands: - result = exec_fn(command, timeout_s=120) - if result.exit_code != 0: - raise OrchestrationError(f'staging the task package failed (rc={result.exit_code}): {result.output[-200:]}') - - -def build_target(exec_fn: VmExec, task: RlfCtfTask, *, image: str, remote_dir: str = REMOTE_TASK_DIR) -> None: - """``docker build --network host`` — the default bridge hangs and - ``--network=none`` fails in nested Docker (verified in recvfix2).""" - result = exec_fn(f'{_DOCKER} build --network host -t {shlex.quote(image)} {shlex.quote(remote_dir)}', - timeout_s=600) - if result.exit_code != 0: - raise OrchestrationError(scrub(f'target image build failed (rc={result.exit_code}): ' - f'{result.output[-200:]}', task.flag)) - - -def run_target(exec_fn: VmExec, task: RlfCtfTask, *, image: str, container: str) -> str: - """Start the target with the FLAG as a container env var (the only place - the flag crosses onto the VM). Output/errors are scrubbed; the flag is - never logged or raised.""" - command = (f'{_DOCKER} run -d --name {shlex.quote(container)} --network host ' - f'-e FLAG={shlex.quote(task.flag)} {shlex.quote(image)}') - result = exec_fn(command, timeout_s=60) - if result.exit_code != 0: - raise OrchestrationError(scrub(f'target container failed to start (rc={result.exit_code}): ' - f'{result.output[-200:]}', task.flag)) - cid = result.output.strip().splitlines()[-1] if result.output.strip() else '' - if not cid: - raise OrchestrationError('target container start returned no container id') - return cid - - -def wait_target_healthy(exec_fn: VmExec, task: RlfCtfTask, *, container: str, attempts: int = 30, - sleep_s: float = 1.0, sleep: Callable[[float], None] = time.sleep) -> None: - """Block until the target's HEALTHCHECK reports healthy; fail fast on exit.""" - probe = (f"{_DOCKER} inspect -f '{{{{.State.Status}}}} " - f"{{{{if .State.Health}}}}{{{{.State.Health.Status}}}}{{{{end}}}}' {shlex.quote(container)}") - for _ in range(max(1, attempts)): - result = exec_fn(probe, timeout_s=15) - fields = result.output.split() - status = fields[0] if fields else '' - health = fields[1] if len(fields) > 1 else '' - if status in ('exited', 'dead'): - raise OrchestrationError(f'target container {container} is {status} before becoming healthy') - if result.exit_code == 0 and (health == 'healthy' or (status == 'running' and not health)): - return - sleep(sleep_s) - raise OrchestrationError(f'target container did not become healthy after {attempts} attempts') - - -def remove_containers(exec_fn: VmExec, containers: list[str] | tuple[str, ...]) -> None: - """``docker rm -f`` in the given order; every step tolerates failure - (idempotent teardown — a missing container is already clean).""" - for name in containers: - try: - exec_fn(f'{_DOCKER} rm -f {shlex.quote(name)} >/dev/null 2>&1', timeout_s=30) - except Exception: # noqa: BLE001 - cleanup never masks the original outcome - pass - - -# ----------------------------------------------------------------------------- orchestration state machine - - -@dataclass -class CtfOrchestration: - """Verified nested-Docker sequence as an idempotent up/down state machine.""" - - exec_fn: VmExec - task: RlfCtfTask - target_container: str = 'rlf-ctf-target' - target_image: str = 'rlf-ctf-target' - # Solver/grader containers (cleaned before the target); empty in the online - # path (the agent relays into the VM host namespace and grading is - # trusted-side), but the ordering contract is kept for reference-check runs. - auxiliary_containers: tuple[str, ...] = () - dockerd_attempts: int = 40 - health_attempts: int = 30 - sleep: Callable[[float], None] = time.sleep - _up_done: bool = False - _down_done: bool = False - - def up(self) -> str: - """dockerd -> stage -> build -> run(FLAG) -> health. Returns the target cid.""" - if self._up_done: - raise OrchestrationError('orchestration already brought up') - try: - start_dockerd(self.exec_fn, attempts=self.dockerd_attempts, sleep=self.sleep) - stage_package(self.exec_fn, self.task.package_dir) - build_target(self.exec_fn, self.task, image=self.target_image) - cid = run_target(self.exec_fn, self.task, image=self.target_image, container=self.target_container) - wait_target_healthy(self.exec_fn, self.task, container=self.target_container, - attempts=self.health_attempts, sleep=self.sleep) - except BaseException: - self.down() # a half-up environment is always cleaned - raise - self._up_done = True - return cid - - def down(self) -> None: - """solver/grader containers -> target container; images stay; idempotent.""" - if self._down_done: - return - self._down_done = True - remove_containers(self.exec_fn, [*self.auxiliary_containers, self.target_container]) - - -# ----------------------------------------------------------------------------- relay environment + verifier - - -class CtfRelayEnvironment: - """``TerminalEnvironment`` served in the trusted process; commands exec on the VM. - - One relay == one episode: the first episode id on the wire binds it - (same contract as ``tb2_provider.EpisodeBinding``). ``submit`` records - the agent's submission for the trusted verifier; the flag comparison - never happens here. - """ - - def __init__(self, exec_fn: VmExec) -> None: - self._exec_fn = exec_fn - self.episode_id: str | None = None - self.commands: list[str] = [] - self.submissions: list[dict[str, Any]] = [] - self.submitted = False - - def _bind(self, episode_id: str) -> None: - if self.episode_id is None: - self.episode_id = episode_id - elif episode_id != self.episode_id: - from yeto.rl.harness.codex.client import EpisodeAPIError - raise EpisodeAPIError(404, 'unknown_episode', 'unknown episode') - - async def execute(self, episode_id: str, command: str, timeout_seconds: float, output_bytes: int): - import asyncio - self._bind(episode_id) - if self.submitted: - from yeto.rl.harness.codex.client import EpisodeAPIError - raise EpisodeAPIError(409, 'episode_terminal', 'episode already submitted') - self.commands.append(command) - seconds = max(1, int(timeout_seconds)) - capped = (f'timeout -k 2 {seconds} bash -c {shlex.quote(command)} 2>&1 | ' - f'head -c {int(output_bytes) + 1}; exit ${{PIPESTATUS[0]}}') - result = await asyncio.to_thread(self._exec_fn, capped, timeout_s=timeout_seconds) - truncated = len(result.output.encode('utf-8')) > output_bytes - return {'exit_code': result.exit_code, 'output': result.output[:output_bytes], - 'timed_out': result.timed_out, 'truncated': truncated} - - async def submit(self, episode_id: str, submission: dict[str, Any]): - self._bind(episode_id) - self.submissions.append(dict(submission)) - self.submitted = True - return {'accepted': True} - - -class RlfFlagVerifier: - """Trusted-side flag grader (``grader_mode: flag``). - - Compares the agent's submitted flag against the trusted flag in constant - time. The evaluation payload carries the RLF grade under ``rlf_grade`` - (consumed by ``finish_trusted`` via ``trusted_grade_from_evaluation``) - with the RLF identity quadruple in ``grader_evidence`` — the verifier is - built per-lease, so that identity is trusted-side. The flag itself never - appears in the payload, the evidence, or the log. - """ - - def __init__(self, relay: CtfRelayEnvironment, *, flag: str, task_id: str, episode_id: str, - trajectory_id: str) -> None: - self._relay = relay - self._flag = flag - self._identity = {'rlf_task_id': task_id, 'rlf_episode_id': episode_id, - 'rlf_trajectory_id': trajectory_id} - self.evaluations = 0 - - async def evaluate(self, episode_id: str) -> dict[str, Any]: - if self._relay.episode_id is not None and episode_id != self._relay.episode_id: - raise ValueError(f'verifier bound to {self._relay.episode_id!r}, asked for {episode_id!r}') - self.evaluations += 1 - import hmac as _hmac - submission = self._relay.submissions[-1] if self._relay.submissions else None - candidate = submission.get('flag') if isinstance(submission, Mapping) else None - submitted = isinstance(candidate, str) and bool(candidate) - matched = submitted and _hmac.compare_digest(candidate, self._flag) - evidence = {'grader': 'rlf-flag', 'grader_mode': 'flag', 'submitted': submitted, **self._identity} - grade: dict[str, Any] = {'status': 'completed', 'reward': 1.0 if matched else 0.0, - 'passed': bool(matched), 'grader_evidence': evidence} - if not submitted: - grade['error_class'] = 'missing_submission' - return {'passed': bool(matched), 'rlf_grade': grade, - 'log': 'flag submission matched' if matched else 'flag submission missing or did not match'} - - -# ----------------------------------------------------------------------------- full environment lease - - -@dataclass(frozen=True) -class RlfCtfConfig: - """CTF packaging for :class:`RlfModalProvider` (breakpoint 3).""" - - tasks_dir: str # directory of admitted task packages (task_id -> package dir) - flags: Mapping[str, str] # task_id -> trusted flag (never leaves the trusted side) - vm_image: str = 'docker:27-dind' # nested Docker runtime (verified on Modal) - lease_seconds: float = 1800.0 - dockerd_attempts: int = 40 - health_attempts: int = 30 - - -@dataclass -class RlfCtfLease: - """Full ``EnvironmentLease``: field names match the subprocess agent protocol.""" - - env_url: str - env_token: str - verifier: RlfFlagVerifier - destroy: Callable[[], Any] - describe: Callable[[], Any] - deadline_seconds: float - worker_env: dict[str, str] | None = None - task_id: str = '' - episode_id: str = '' - trajectory_id: str = '' - vm_lease: RlfVmLease | None = None - orchestration: CtfOrchestration | None = None - environment: CtfRelayEnvironment | None = None - - -class RlfModalProvider: - """Preflight-compatible RLF environment provider factory product. - - ``resolve_environment_provider`` calls ``factory(miles_args)`` and requires - the result to expose ``acquire(task_id, trajectory_id)``; this class is that - product. Each acquire opens one ``EpisodeLease`` (RLF identity producer) - and one isolated Modal VM bound to it. - - With ``ctf=None`` the returned lease is the S20 VM lease (exec/teardown + - RLF identity; no HTTP endpoint/verifier). With an :class:`RlfCtfConfig` - each acquire additionally runs the packaged CTF orchestration - (dockerd/target/flag/health) and returns a full :class:`RlfCtfLease` - (env_url/env_token/verifier/destroy/describe/deadline) — audit - breakpoint 3. - """ - - def __init__(self, *, config: RlfModalVmConfig | None = None, ctf: RlfCtfConfig | None = None): - self.config = config or RlfModalVmConfig() - self.ctf = ctf - - async def acquire(self, task_id: str, trajectory_id: str = '') -> RlfVmLease | RlfCtfLease: - episode = EpisodeLease(task_id=task_id, - trajectory_id=trajectory_id or f'rlf-traj-{uuid.uuid4().hex}') - if self.ctf is None: - return acquire_vm(task_id, episode.episode_id, config=self.config, - trajectory_id=episode.trajectory_id) - return await self._acquire_ctf(episode) - - async def _acquire_ctf(self, episode: EpisodeLease) -> RlfCtfLease: - import asyncio - import secrets - - from yeto.rl.harness.codex.environment import serve_environment - - ctf = self.ctf - assert ctf is not None - task = resolve_ctf_task(episode.task_id, ctf.tasks_dir, ctf.flags) - # vm_exec/relay use ``bash -lc``; the default docker:27-dind image is - # alpine-based and has no bash, so add it at image build time. - vm_config = replace(self.config, image=ctf.vm_image, - image_run_commands=('apk add --no-cache bash',)) - vm = await asyncio.to_thread(acquire_vm, episode.task_id, episode.episode_id, - config=vm_config, trajectory_id=episode.trajectory_id) - orchestration = CtfOrchestration(vm_exec(vm), task, - dockerd_attempts=ctf.dockerd_attempts, - health_attempts=ctf.health_attempts) - try: - await asyncio.to_thread(orchestration.up) - relay = CtfRelayEnvironment(vm_exec(vm)) - token = secrets.token_urlsafe(24) - runner, url = await serve_environment(relay, token, host='127.0.0.1') - except BaseException: - orchestration.down() - await asyncio.to_thread(vm.teardown) - raise - deadline = float(ctf.lease_seconds or self.config.ttl_s) - state = {'destroyed': False} - - async def destroy() -> None: - if state['destroyed']: - return - state['destroyed'] = True - # solver/grader containers -> target container -> images stay -> VM terminate - await asyncio.to_thread(orchestration.down) - await runner.cleanup() - await asyncio.to_thread(vm.teardown) - - async def describe() -> str: - if not state['destroyed']: - return 'live' - for _ in range(20): - if not await asyncio.to_thread(vm.alive): - return 'gone' - await asyncio.sleep(0.25) - return 'live' - - return RlfCtfLease(env_url=url, env_token=token, - verifier=RlfFlagVerifier(relay, flag=task.flag, task_id=episode.task_id, - episode_id=episode.episode_id, - trajectory_id=episode.trajectory_id), - destroy=destroy, describe=describe, deadline_seconds=deadline, - task_id=episode.task_id, episode_id=episode.episode_id, - trajectory_id=episode.trajectory_id, vm_lease=vm, - orchestration=orchestration, environment=relay) diff --git a/yeto/rl/rlf/preflight.py b/yeto/rl/rlf/preflight.py deleted file mode 100644 index e99db9437..000000000 --- a/yeto/rl/rlf/preflight.py +++ /dev/null @@ -1,41 +0,0 @@ -from __future__ import annotations -from typing import Any, Mapping -from .contracts import TrustedGrade -_REQUIRED=('rlf_episode_id','rlf_task_id','rlf_trajectory_id','grader_evidence','success') -class RolloutRejected(ValueError): pass -def validate_rollout(row: Mapping[str,Any]) -> None: - md=row.get('metadata') - if not isinstance(md, Mapping): raise RolloutRejected('missing metadata') - missing=[x for x in _REQUIRED if x not in md] - if missing: raise RolloutRejected('missing trusted fields: '+', '.join(missing)) - if not md.get('rlf_episode_id') or not md.get('rlf_task_id'): raise RolloutRejected('missing RLF identity') - if not isinstance(md['grader_evidence'], Mapping): raise RolloutRejected('invalid grader evidence') - if row.get('cleanup_ok') is False or md.get('cleanup_error'): raise RolloutRejected('cleanup evidence failed') - -RLF_GRADE_KEY = 'rlf_grade' -_GRADE_REQUIRED = ('status', 'reward', 'passed', 'grader_evidence') - -def trusted_grade_from_evaluation(evaluation: Mapping[str, Any]) -> TrustedGrade | None: - """Interface for the harness trusted finish (RLF grade first, TB2 HMAC fallback). - - A ``TrustedVerifier.evaluate()`` payload that carries ``rlf_grade`` (or is - itself a full grade mapping) is parsed into a ``TrustedGrade`` so the - trusted finish can close the episode via ``finish_episode``; anything else - returns ``None`` and the caller falls back to the TB2 HMAC-signed outcome. - A present but malformed grade fails closed (``RolloutRejected``) instead of - silently downgrading to HMAC. - - Not yet wired into ``codex_openenv_agent_function.finish_trusted``; that - call site is outside this change's write scope. - """ - if not isinstance(evaluation, Mapping): return None - payload = evaluation.get(RLF_GRADE_KEY) - if payload is None and all(k in evaluation for k in _GRADE_REQUIRED): payload = evaluation - if payload is None: return None - if not isinstance(payload, Mapping) or not all(k in payload for k in _GRADE_REQUIRED): - raise RolloutRejected('malformed RLF grade in trusted evaluation') - evidence = payload['grader_evidence'] - if not isinstance(evidence, Mapping): raise RolloutRejected('RLF grade evidence must be a mapping') - error = payload.get('error_class') - return TrustedGrade(str(payload['status']), float(payload['reward']), bool(payload['passed']), - dict(evidence), str(error) if error else None) diff --git a/yeto/rl/rlf/trajectory.py b/yeto/rl/rlf/trajectory.py deleted file mode 100644 index d00a34bce..000000000 --- a/yeto/rl/rlf/trajectory.py +++ /dev/null @@ -1,14 +0,0 @@ -from __future__ import annotations -from dataclasses import dataclass, field -from typing import Any, Mapping -@dataclass -class CodexTrajectory: - task_id: str; episode_id: str; trajectory_id: str - tokens: list[int]=field(default_factory=list); logprobs: list[float]=field(default_factory=list) - loss_mask: list[int]=field(default_factory=list); segments: list[Mapping[str,Any]]=field(default_factory=list) - policy_id: str|None=None; metadata: dict[str,Any]=field(default_factory=dict) - def as_metadata(self): - return {'rlf_task_id':self.task_id,'rlf_episode_id':self.episode_id,'rlf_trajectory_id':self.trajectory_id,'policy_id':self.policy_id,'segments':self.segments} - -def from_codex_result(result: Mapping[str,Any], *, task_id: str, episode_id: str, trajectory_id: str, policy_id: str|None=None) -> CodexTrajectory: - return CodexTrajectory(task_id,episode_id,trajectory_id,list(result.get('tokens',[])),list(result.get('logprobs',result.get('rollout_logprobs',[]))),list(result.get('loss_mask',[])),list(result.get('segments',[])),policy_id,dict(result.get('metadata',{}))) diff --git a/yeto/rl/secrlenv_outcome.py b/yeto/rl/secrlenv_outcome.py new file mode 100644 index 000000000..c713d14d4 --- /dev/null +++ b/yeto/rl/secrlenv_outcome.py @@ -0,0 +1,111 @@ +"""Structural check of a SecRLEnv grader verdict in trajectory metadata. + +Kept from the SecRLEnv reward of the removed self-written Codex harness (S26 +E0.5, tag s26-pre-harness-removal) so trajectory evidence of the ``secrlenv`` +kind still verifies; the same role ``tbench_outcome`` has for Terminal-Bench. +Rewards are not signed (openspec ``rl-drop-reward-tamper-hash``): this checks +shape, status and range only. +""" + +from __future__ import annotations + +import json +import math +import re +from typing import Any + +OUTCOME_KEY = "secrlenv_trusted_outcome" +INFRASTRUCTURE_STATUS = "infrastructure_error" +CLEANUP_ERROR_STATUS = "cleanup_error" +INFRASTRUCTURE_503_RETRIES_KEY = "secrlenv_infrastructure_503_retries" +ADMISSION_SCHEMA = 2 +ADMISSION_PHASE_KEY = "admission_phase" +ADMISSION_ERROR_CODE_KEY = "admission_error_code" +ADMISSION_NONCE_KEY = "admission_nonce" +ADMISSION_PHASE = "pre_create" +ADMISSION_ERROR_CODE = "infrastructure_error" +_ADMISSION_NONCE = re.compile(r"[0-9a-f]{32}") +_OUTCOME_STATUSES = { + "completed", + "timeout", + "max_turns", + "max_seq_len", + INFRASTRUCTURE_STATUS, + CLEANUP_ERROR_STATUS, +} + + +class UntrustedOutcome(RuntimeError): + pass + + +def canonical_outcome(value: dict[str, Any]) -> bytes: + return json.dumps(value, sort_keys=True, separators=(",", ":")).encode("utf-8") + + +def verified_outcome(metadata: Any) -> tuple[dict[str, Any], float]: + if not isinstance(metadata, dict): + raise UntrustedOutcome("sample metadata is not an object") + outcome = metadata.get(OUTCOME_KEY) + if not isinstance(outcome, dict): + raise UntrustedOutcome("sample has no secrlenv outcome") + schema = outcome.get("schema") + status = outcome.get("status") + if schema not in {1, ADMISSION_SCHEMA} or status not in _OUTCOME_STATUSES: + raise UntrustedOutcome("secrlenv outcome has an invalid schema or status") + infrastructure_503_retries = outcome.get(INFRASTRUCTURE_503_RETRIES_KEY) + if ( + type(infrastructure_503_retries) is not int + or infrastructure_503_retries not in {0, 1} + ): + raise UntrustedOutcome("secrlenv outcome has an invalid retry ledger") + reward = outcome.get("reward") + if isinstance(reward, bool) or not isinstance(reward, (int, float)): + raise UntrustedOutcome("secrlenv outcome reward is not numeric") + value = float(reward) + if not math.isfinite(value) or not 0.0 <= value <= 1.0: + raise UntrustedOutcome("secrlenv outcome reward is outside [0, 1]") + if not isinstance(outcome.get("task_id"), str): + raise UntrustedOutcome("secrlenv outcome has no episode/task identity") + if schema == ADMISSION_SCHEMA: + expected_keys = { + "schema", + "status", + "episode_id", + "task_id", + "reward", + "passed", + "class", + INFRASTRUCTURE_503_RETRIES_KEY, + ADMISSION_PHASE_KEY, + ADMISSION_ERROR_CODE_KEY, + ADMISSION_NONCE_KEY, + } + if ( + set(outcome) != expected_keys + or status != INFRASTRUCTURE_STATUS + or outcome.get("episode_id") is not None + or not outcome["task_id"] + or outcome.get(ADMISSION_PHASE_KEY) != ADMISSION_PHASE + or outcome.get(ADMISSION_ERROR_CODE_KEY) != ADMISSION_ERROR_CODE + or not isinstance(outcome.get(ADMISSION_NONCE_KEY), str) + or _ADMISSION_NONCE.fullmatch(outcome[ADMISSION_NONCE_KEY]) is None + or infrastructure_503_retries != 1 + or value != 0.0 + or outcome.get("passed") is not False + or outcome.get("class") is not None + ): + raise UntrustedOutcome("secrlenv admission outcome is invalid") + elif ( + not isinstance(outcome.get("episode_id"), str) + or any( + key in outcome + for key in ( + ADMISSION_PHASE_KEY, + ADMISSION_ERROR_CODE_KEY, + ADMISSION_NONCE_KEY, + ) + ) + ): + raise UntrustedOutcome("secrlenv outcome has no episode/task identity") + return outcome, value diff --git a/yeto/rl/ssh_harness.py b/yeto/rl/ssh_harness.py index 8c6975e98..686e73236 100644 --- a/yeto/rl/ssh_harness.py +++ b/yeto/rl/ssh_harness.py @@ -5,12 +5,10 @@ import argparse import hashlib import json -import math import os import re import shlex import shutil -import statistics import subprocess import sys import time @@ -22,37 +20,11 @@ import torch from . import ( - CODEX_APP_SERVER_PROTOCOL_REVISION, - CODEX_APP_SERVER_SCHEMA_SHA256, - CODEX_BASE_INSTRUCTIONS_SHA256, - CODEX_CLI_VERSION, - CODEX_CONTAINER_APP_SERVER_SCHEMA_PATH, - CODEX_CONTAINER_BINARY_PATH, - CODEX_DYNAMIC_TOOLS_SCHEMA_SHA256, - CODEX_HARNESS_AGENT, - CODEX_HARNESS_AGENT_SHA256, - CODEX_LINUX_BINARY_SHA256, - CODEX_LINUX_BINARY_SIZE_BYTES, - CODEX_LINUX_TARGET, - CODEX_NPM_PACKAGE, - CODEX_NPM_TARBALL_SHA256, - CODEX_OPENENV_AGENT, - CODEX_OPENENV_IDENTITY_ENV, - CODEX_PACKAGE_MANIFEST_SHA256, - CODEX_SUBMIT_TOOL_SCHEMA_SHA256, - CODEX_TERMINAL_EXEC_TOOL_SCHEMA_SHA256, MILES_NEXT_COMMIT, MILES_NEXT_REPOSITORY, MILES_PEFT_VERSION, - SECRLENV_AGENTS, - SECRLENV_GENERATE, - SECRLENV_GROUP_FILTER, - SECRLENV_INFRASTRUCTURE_REPLACEMENTS, - SECRLENV_REWARD, - SECRLENV_ZERO_VARIANCE_REPLACEMENTS, SGLANG_NEXT_COMMIT, SGLANG_NEXT_REPOSITORY, - SIGNED_CODEX_AGENTS, TMS_PRELOAD_BASE_BINARY_SHA256, TMS_PRELOAD_PATCH_BASE_COMMIT, TMS_PRELOAD_PATCH_COMMIT, @@ -111,10 +83,6 @@ }) _REVISION = re.compile(r"[0-9a-f]{40}\Z") _SAFE_IMAGE = re.compile(r"[a-zA-Z0-9][a-zA-Z0-9._/:@-]+\Z") -_EVAL_DATASET_NAME = re.compile(r"[a-zA-Z0-9][a-zA-Z0-9._-]{0,63}\Z") -CODEX_REMOTE_BUNDLE_PATH = "codex/codex-x86_64-unknown-linux-musl" -CODEX_REMOTE_MANIFEST_PATH = "codex/codex-package.json" -CODEX_REMOTE_SCHEMA_PATH = "codex/codex_app_server_protocol.v2.schemas.json" _CONTAINER_STATE_FORMAT = ( '{"Status":{{json .State.Status}},' '"ExitCode":{{.State.ExitCode}},' @@ -140,15 +108,6 @@ "rl_policy_apply": ("event", "sync/global_policy_hash"), "rl_final_cut": ("event",), }, - "eval-learner": { - "rl_policy_apply": ( - "event", "island_id", "policy_version", "sync/global_policy_hash" - ), - "rl_eval_result": ( - "event", "island_id", "rollout_id", "policy_version", "dataset_name", - "sample_count", "rl/eval/result", "rl/eval/pass_at_1" - ), - }, "syncer": { "syncer_commit": ( "step", "fragment", "launch_base_version", "expected", "responded", @@ -255,26 +214,6 @@ def scalar(name, value): except (OSError, UnicodeError, ValueError, json.JSONDecodeError): raise SystemExit("invalid event evidence source") """ -_SYNCER_LIFECYCLE_EVIDENCE = r""" -import json -import sys -from pathlib import Path - -path = Path(sys.argv[1]) -if path.is_symlink() or not path.is_file() or path.stat().st_size > 64 * 1024 * 1024: - raise SystemExit("invalid syncer lifecycle source") -resume = 0 -final_ack = 0 -try: - with path.open(encoding="utf-8", errors="strict") as source: - for line in source: - resume += line.count("resumed from checkpoint") - final_ack += line.count("learner finalized authoritative parameters") -except (OSError, UnicodeError): - raise SystemExit("invalid syncer lifecycle source") -print(json.dumps({"schema": 1, "resume_count": resume, "final_ack_count": final_ack}, - sort_keys=True, separators=(",", ":"))) -""" _EVENT_EVIDENCE_FIELDS = { "strict-learner": { "rl_local_round": { @@ -300,24 +239,6 @@ def scalar(name, value): }, "rl_final_cut": {"event"}, }, - "eval-learner": { - "rl_policy_apply": { - "event", - "island_id", - "policy_version", - "sync/global_policy_hash", - }, - "rl_eval_result": { - "event", - "island_id", - "rollout_id", - "policy_version", - "dataset_name", - "sample_count", - "rl/eval/result", - "rl/eval/pass_at_1", - }, - }, "syncer": { "syncer_commit": { "step", @@ -502,276 +423,6 @@ def _syncer_source_sha256() -> str: return digest.hexdigest() -def _pinned_codex_adapter_identity() -> dict[str, str]: - return { - "base_instructions_sha256": CODEX_BASE_INSTRUCTIONS_SHA256, - "terminal_exec_tool_schema_sha256": ( - CODEX_TERMINAL_EXEC_TOOL_SCHEMA_SHA256 - ), - "submit_tool_schema_sha256": CODEX_SUBMIT_TOOL_SCHEMA_SHA256, - "dynamic_tools_schema_sha256": CODEX_DYNAMIC_TOOLS_SCHEMA_SHA256, - } - - -def _codex_adapter_source_path() -> Path: - """Resolve the optional legacy adapter from its installed package.""" - - try: - from yeto.rl.harness.codex import codex_harness_agent - except ImportError as exc: - raise HarnessError("cannot attest the Yeto Codex harness adapter") from exc - value = getattr(codex_harness_agent, "__file__", None) - if not isinstance(value, str): - raise HarnessError("Yeto Codex harness adapter has no source identity") - source = Path(value) - if source.is_symlink() or not source.is_file(): - raise HarnessError("Yeto Codex harness adapter source is unavailable") - return source.resolve() - - -def _codex_adapter_identity() -> dict[str, str]: - """Read hashes derived by the live optional Codex adapter.""" - - try: - from yeto.rl.harness.codex import codex_harness_agent - - identity = codex_harness_agent.codex_harness_identity() - except (ImportError, AttributeError, TypeError, ValueError, RuntimeError) as exc: - raise HarnessError("cannot attest the Yeto Codex harness adapter") from exc - required = { - "base_instructions_sha256", - "terminal_exec_tool_schema_sha256", - "submit_tool_schema_sha256", - "dynamic_tools_schema_sha256", - } - if not isinstance(identity, dict) or set(identity) != required: - raise HarnessError("Yeto Codex adapter returned an invalid identity contract") - normalized = {name: str(value).lower() for name, value in identity.items()} - if any(not _SHA256.fullmatch(value) for value in normalized.values()): - raise HarnessError("Yeto Codex adapter identity contains an invalid SHA256") - expected = _pinned_codex_adapter_identity() - if normalized != expected: - raise HarnessError("Yeto Codex adapter identity does not match its Yeto pins") - return normalized - - -def _codex_harness_contract(namespace, args) -> dict[str, Any]: - """Attest the official Linux Codex artifact and the Yeto adapter surface.""" - - from ..provenance import canonical_json_sha256, file_sha256 - - binary_value = getattr(namespace, "codex_harness_binary", None) - manifest_value = getattr(namespace, "codex_package_manifest", None) - schema_value = getattr(namespace, "codex_app_server_schema", None) - if not binary_value or not manifest_value or not schema_value: - raise HarnessError( - "the stock Codex agent requires --codex-harness-binary and " - "--codex-package-manifest and --codex-app-server-schema" - ) - binary = Path(binary_value).expanduser() - manifest = Path(manifest_value).expanduser() - schema = Path(schema_value).expanduser() - for path, flag in ( - (binary, "--codex-harness-binary"), - (manifest, "--codex-package-manifest"), - (schema, "--codex-app-server-schema"), - ): - if path.is_symlink() or not path.is_file(): - raise HarnessError(f"{flag} must be a regular, non-symlink file") - binary = binary.resolve() - manifest = manifest.resolve() - schema = schema.resolve() - if not binary.stat().st_mode & 0o111: - raise HarnessError("--codex-harness-binary must be executable") - if binary.stat().st_size != CODEX_LINUX_BINARY_SIZE_BYTES: - raise HarnessError("Codex Linux binary size does not match its Yeto pin") - if file_sha256(binary) != CODEX_LINUX_BINARY_SHA256: - raise HarnessError("Codex Linux binary SHA256 does not match its Yeto pin") - if file_sha256(manifest) != CODEX_PACKAGE_MANIFEST_SHA256: - raise HarnessError("Codex package manifest SHA256 does not match its Yeto pin") - try: - schema_sha = canonical_json_sha256(schema) - except ValueError: - schema_sha = None - if schema_sha != CODEX_APP_SERVER_SCHEMA_SHA256: - raise HarnessError("Codex app-server schema SHA256 does not match its Yeto pin") - openenv = ( - getattr(args, "custom_agent_function_path", None) == CODEX_OPENENV_AGENT - ) - if openenv: - # The OpenEnv wrapper is shipped by the pinned Miles bundle and is - # attested inside the learner before model allocation. - identity = _pinned_codex_adapter_identity() - else: - identity = _codex_adapter_identity() - adapter_path = _codex_adapter_source_path() - if file_sha256(adapter_path) != CODEX_HARNESS_AGENT_SHA256: - raise HarnessError( - "Yeto Codex harness adapter source does not match its pin" - ) - from .codex_backend import stock_codex_backend_contract_with_reasoning_policy - - profile_name = getattr(args, "codex_backend_profile", None) or args.tito_model - backend = stock_codex_backend_contract_with_reasoning_policy( - profile_name, - args.rollout_max_response_len, - not getattr(args, "codex_exclude_reasoning_from_loss", False), - ) - contract = { - "agent_function_path": CODEX_HARNESS_AGENT, - "agent_source_sha256": CODEX_HARNESS_AGENT_SHA256, - "controller_binary_path": str(binary), - "controller_package_manifest_path": str(manifest), - "controller_app_server_schema_path": str(schema), - "bundle_binary_path": CODEX_REMOTE_BUNDLE_PATH, - "bundle_package_manifest_path": CODEX_REMOTE_MANIFEST_PATH, - "bundle_app_server_schema_path": CODEX_REMOTE_SCHEMA_PATH, - "container_binary_path": CODEX_CONTAINER_BINARY_PATH, - "container_app_server_schema_path": ( - CODEX_CONTAINER_APP_SERVER_SCHEMA_PATH - ), - "binary_sha256": CODEX_LINUX_BINARY_SHA256, - "binary_size_bytes": CODEX_LINUX_BINARY_SIZE_BYTES, - "cli_version": CODEX_CLI_VERSION, - "npm_package": CODEX_NPM_PACKAGE, - "target": CODEX_LINUX_TARGET, - "npm_tarball_sha256": CODEX_NPM_TARBALL_SHA256, - "package_manifest_sha256": CODEX_PACKAGE_MANIFEST_SHA256, - "app_server_protocol_revision": CODEX_APP_SERVER_PROTOCOL_REVISION, - "app_server_schema_sha256": CODEX_APP_SERVER_SCHEMA_SHA256, - **identity, - "reasoning_effort": args.codex_reasoning_effort, - "backend": backend, - } - if openenv: - contract["openenv_identity_env"] = dict(CODEX_OPENENV_IDENTITY_ENV) - return contract - - -def _validate_codex_harness(value: Any, learner: dict[str, Any]) -> None: - required = { - "agent_function_path", - "agent_source_sha256", - "controller_binary_path", - "controller_package_manifest_path", - "controller_app_server_schema_path", - "bundle_binary_path", - "bundle_package_manifest_path", - "bundle_app_server_schema_path", - "container_binary_path", - "container_app_server_schema_path", - "binary_sha256", - "binary_size_bytes", - "cli_version", - "npm_package", - "target", - "npm_tarball_sha256", - "package_manifest_sha256", - "app_server_protocol_revision", - "app_server_schema_sha256", - "base_instructions_sha256", - "terminal_exec_tool_schema_sha256", - "submit_tool_schema_sha256", - "dynamic_tools_schema_sha256", - "reasoning_effort", - "backend", - } - custom_agent = learner.get("custom_agent_function_path") - if custom_agent == CODEX_OPENENV_AGENT: - required.add("openenv_identity_env") - if not isinstance(value, dict) or set(value) != required: - raise HarnessError("plan has an invalid stock Codex harness contract") - pinned = { - "agent_function_path": CODEX_HARNESS_AGENT, - "agent_source_sha256": CODEX_HARNESS_AGENT_SHA256, - "bundle_binary_path": CODEX_REMOTE_BUNDLE_PATH, - "bundle_package_manifest_path": CODEX_REMOTE_MANIFEST_PATH, - "bundle_app_server_schema_path": CODEX_REMOTE_SCHEMA_PATH, - "container_binary_path": CODEX_CONTAINER_BINARY_PATH, - "container_app_server_schema_path": ( - CODEX_CONTAINER_APP_SERVER_SCHEMA_PATH - ), - "binary_sha256": CODEX_LINUX_BINARY_SHA256, - "binary_size_bytes": CODEX_LINUX_BINARY_SIZE_BYTES, - "cli_version": CODEX_CLI_VERSION, - "npm_package": CODEX_NPM_PACKAGE, - "target": CODEX_LINUX_TARGET, - "npm_tarball_sha256": CODEX_NPM_TARBALL_SHA256, - "package_manifest_sha256": CODEX_PACKAGE_MANIFEST_SHA256, - "app_server_protocol_revision": CODEX_APP_SERVER_PROTOCOL_REVISION, - "app_server_schema_sha256": CODEX_APP_SERVER_SCHEMA_SHA256, - "base_instructions_sha256": CODEX_BASE_INSTRUCTIONS_SHA256, - "terminal_exec_tool_schema_sha256": ( - CODEX_TERMINAL_EXEC_TOOL_SCHEMA_SHA256 - ), - "submit_tool_schema_sha256": CODEX_SUBMIT_TOOL_SCHEMA_SHA256, - "dynamic_tools_schema_sha256": CODEX_DYNAMIC_TOOLS_SCHEMA_SHA256, - "reasoning_effort": "xhigh", - } - if any(value.get(name) != expected for name, expected in pinned.items()): - raise HarnessError("plan does not use the pinned stock Codex runtime") - if ( - custom_agent == CODEX_OPENENV_AGENT - and value.get("openenv_identity_env") != CODEX_OPENENV_IDENTITY_ENV - ): - raise HarnessError("plan does not use the pinned Codex OpenEnv surface") - for name in ( - "controller_binary_path", - "controller_package_manifest_path", - "controller_app_server_schema_path", - ): - path = Path(str(value.get(name, ""))) - if not path.is_absolute() or ".." in path.parts: - raise HarnessError(f"stock Codex contract has an invalid {name}") - from .codex_backend import ( - stock_codex_backend_contract_with_reasoning_policy, - validate_stock_codex_fields, - ) - - backend = value.get("backend") - profile_name = str( - learner.get("codex_backend_profile") or learner.get("tito_model", "") - ) - expected_backend = stock_codex_backend_contract_with_reasoning_policy( - profile_name, - learner.get("rollout_max_response_len"), - not learner.get("codex_exclude_reasoning_from_loss", False), - ) - if backend != expected_backend: - raise HarnessError("stock Codex backend/TITO contract drifted") - if custom_agent not in SIGNED_CODEX_AGENTS: - raise HarnessError("stock Codex contract requires a signed agent path") - if ( - custom_agent == CODEX_OPENENV_AGENT - and profile_name != "qwen35_08b" - ): - raise HarnessError( - "the Codex OpenEnv adapter requires backend profile qwen35_08b" - ) - try: - validate_stock_codex_fields( - tito_model=str(learner.get("tito_model", "")), - codex_backend_profile=profile_name, - rl_model_recipe=str(learner.get("rl_model_recipe", "generic")), - model=str(learner.get("model", "")), - model_revision=str(learner.get("model_revision", "")), - rollout_model=learner.get("rollout_model"), - rollout_model_revision=learner.get("rollout_model_revision"), - apply_chat_template_kwargs=learner.get( - "apply_chat_template_kwargs" - ), - tito_allowed_append_roles=learner.get( - "tito_allowed_append_roles" - ), - codex_reasoning_effort=learner.get("codex_reasoning_effort"), - lora_targets=str(learner.get("lora_targets", "")), - expert_full_count=int(learner.get("expert_full_count", 0)), - lora_expert_rank=int(learner.get("rl_lora_expert_rank") or 0), - ) - except (TypeError, ValueError) as exc: - raise HarnessError("stock Codex model profile drifted") from exc - - def _tms_preload_patch(path: str | Path) -> dict[str, str]: """Attest the narrow upstream fix over the pinned Miles image binary.""" @@ -1008,49 +659,6 @@ def _local_data_sha256(path: str | Path) -> str: return digest.hexdigest() -def _local_jsonl_record_count(path: str | Path) -> int: - source = Path(path) - if source.is_symlink() or not source.is_file(): - raise HarnessError("evaluation dataset must be one regular JSONL file") - try: - with source.open("rb") as handle: - count = sum(1 for line in handle if line.strip()) - except OSError as error: - raise HarnessError("cannot read evaluation dataset") from error - if count <= 0: - raise HarnessError("evaluation dataset is empty") - return count - - -def _eval_checkpoint_contract( - source_host: str, - source_path: str, - sha256: str, - size_bytes: int, - global_step: int, -) -> dict[str, Any]: - """Pin a remote terminal checkpoint without staging it on the controller.""" - - source_host = _validate_target(source_host) - source_path = _validate_remote_path(source_path, "--eval-checkpoint-path") - sha256 = str(sha256).lower() - if not _SHA256.fullmatch(sha256): - raise HarnessError("--eval-checkpoint-sha256 must be a lowercase SHA256") - for flag, value in ( - ("--eval-checkpoint-size-bytes", size_bytes), - ("--eval-checkpoint-global-step", global_step), - ): - if not isinstance(value, int) or isinstance(value, bool) or value <= 0: - raise HarnessError(f"{flag} must be a positive integer") - return { - "source_host": source_host, - "source_path": source_path, - "sha256": sha256, - "size_bytes": size_bytes, - "global_step": global_step, - } - - def _canonical_json(value: Any) -> str: return json.dumps(value, sort_keys=True, separators=(",", ":")) @@ -1153,59 +761,8 @@ def _validate_plan(plan: dict[str, Any]) -> None: if plan.get("rl_engine") != "ports": raise HarnessError("plan rl_engine must be 'ports'") _require_ports_plan(learner) - daemon = plan.get("secrlenv_daemon") - if learner.get("custom_agent_function_path") in SECRLENV_AGENTS: - if not isinstance(daemon, dict): - raise HarnessError("secrlenv agent requires a pinned daemon contract") - for name in ("source_root", "task_pack", "state_root"): - _validate_data_path(str(daemon.get(name, "")), f"secrlenv {name}") - for name in ("source_sha256", "task_pack_sha256"): - if not _SHA256.fullmatch(str(daemon.get(name, ""))): - raise HarnessError(f"secrlenv daemon has an invalid {name}") - enable_dind_debug = daemon.get("enable_dind_debug") - if type(enable_dind_debug) is not bool: - raise HarnessError( - "secrlenv daemon enable_dind_debug must be an explicit boolean" - ) - dind_image = daemon.get("dind_image") - dind_image_id = daemon.get("dind_image_id") - dind_debug_script_sha256 = daemon.get("dind_debug_script_sha256") - if enable_dind_debug: - if not _SAFE_IMAGE.fullmatch(str(dind_image or "")) or not re.fullmatch( - r"sha256:[0-9a-f]{64}", str(dind_image_id or "") - ): - raise HarnessError( - "enabled secrlenv DinD debug requires a pinned image and image ID" - ) - if not _SHA256.fullmatch(str(dind_debug_script_sha256 or "")): - raise HarnessError( - "enabled secrlenv DinD debug requires a pinned debug script" - ) - elif any( - value is not None - for value in (dind_image, dind_image_id, dind_debug_script_sha256) - ): - raise HarnessError( - "disabled secrlenv DinD debug must not carry unused debug identities" - ) - if ( - daemon.get("bind") != "127.0.0.1" - or daemon.get("placement", "all-hosts") - not in {"all-hosts", "island-heads"} - or not isinstance(daemon.get("port"), int) - or not 1024 <= daemon["port"] <= 65535 - or not isinstance(daemon.get("max_active_episodes"), int) - or daemon["max_active_episodes"] <= 0 - or not _SAFE_IMAGE.fullmatch(str(daemon.get("operator_image", ""))) - or not re.fullmatch( - r"sha256:[0-9a-f]{64}", str(daemon.get("operator_image_id", "")) - ) - ): - raise HarnessError("secrlenv daemon contract is invalid") - if not plan.get("remote_env_file"): - raise HarnessError("secrlenv daemon requires --remote-env-file") - elif daemon is not None: - raise HarnessError("secrlenv daemon requires the secrlenv custom agent") + if plan.get("secrlenv_daemon") is not None: + raise HarnessError("secrlenv daemon plans are no longer supported (S26 E0.5)") tms_patch = plan.get("tms_preload_patch") tms_train_disk_backup = plan.get("tms_train_disk_backup") if learner.get("rl_offload_train"): @@ -1274,126 +831,9 @@ def _validate_plan(plan: dict[str, Any]) -> None: sync_preset = learner.get("sync_preset", "strict-avg") if sync_preset not in {"strict-avg", "decoupled"}: raise HarnessError("plan has an invalid learner sync_preset") - evaluation = learner.get("evaluation") - if evaluation is not None: - if not isinstance(evaluation, dict): - raise HarnessError("plan has an invalid learner evaluation") - if evaluation.get("eval_only") is not True: - raise HarnessError("SSH evaluation must use a separate eval-only run") - if learner.get("custom_agent_function_path") not in SECRLENV_AGENTS: - raise HarnessError("SSH evaluation is restricted to the secrlenv agent") - if learner.get("reward_function") != SECRLENV_REWARD: - raise HarnessError( - "SSH evaluation requires the signed secrlenv reward contract" - ) - if sync_preset != "strict-avg": - raise HarnessError( - "final SSH evaluation requires strict-avg synchronization" - ) - if data_local_path is None or not PurePosixPath( - str(learner.get("data", "")) - ).suffix: - raise HarnessError( - "SSH evaluation requires one local training dataset file" - ) - if ( - evaluation.get("data") != learner.get("data") - or evaluation.get("data_sha256") != data_sha256 - ): - raise HarnessError( - "evaluation must reuse the exact training dataset path and SHA256" - ) - if not _EVAL_DATASET_NAME.fullmatch( - str(evaluation.get("dataset_name", "")) - ): - raise HarnessError("plan has an invalid evaluation dataset_name") - interval = evaluation.get("interval") - samples = evaluation.get("samples_per_prompt") - prompt_count = evaluation.get("prompt_count") - if ( - not isinstance(interval, int) - or isinstance(interval, bool) - or interval != 1 - ): - raise HarnessError("eval-only Miles evaluation interval must be 1") - if ( - not isinstance(samples, int) - or isinstance(samples, bool) - or samples <= 0 - ): - raise HarnessError("plan has an invalid evaluation samples_per_prompt") - if ( - not isinstance(prompt_count, int) - or isinstance(prompt_count, bool) - or prompt_count <= 0 - ): - raise HarnessError("plan has an invalid evaluation prompt_count") - if evaluation.get("skip_before_train") is not True: - raise HarnessError( - "post-training evaluation must skip pre-train evaluation" - ) - if final_ack_timeout_s is None or final_ack_timeout_s <= 3600: - raise HarnessError( - "evaluation requires an explicitly extended final_ack_timeout_s" - ) - temperature = evaluation.get("temperature") - if temperature is not None and ( - not isinstance(temperature, (int, float)) - or isinstance(temperature, bool) - or not math.isfinite(temperature) - or temperature < 0 - ): - raise HarnessError("plan has an invalid evaluation temperature") - top_p = evaluation.get("top_p") - if top_p is not None and ( - not isinstance(top_p, (int, float)) - or isinstance(top_p, bool) - or not math.isfinite(top_p) - or not 0 < top_p <= 1 - ): - raise HarnessError("plan has an invalid evaluation top_p") - for name in ("max_prompt_len", "max_response_len", "max_context_len"): - value = evaluation.get(name) - if value is not None and ( - not isinstance(value, int) - or isinstance(value, bool) - or value <= 0 - or value > learner["seq_len"] - ): - raise HarnessError(f"plan has an invalid evaluation {name}") - checkpoint = plan.get("eval_checkpoint") - if not isinstance(checkpoint, dict) or set(checkpoint) != { - "source_host", - "source_path", - "sha256", - "size_bytes", - "global_step", - }: - raise HarnessError("eval-only plan requires an attested syncer checkpoint") - try: - _validate_target(str(checkpoint.get("source_host", ""))) - _validate_remote_path( - str(checkpoint.get("source_path", "")), - "eval checkpoint source_path", - ) - except HarnessError as error: - raise HarnessError("plan has an invalid eval checkpoint identity") from error - if ( - not _SHA256.fullmatch(str(checkpoint.get("sha256", ""))) - or not isinstance(checkpoint.get("size_bytes"), int) - or isinstance(checkpoint.get("size_bytes"), bool) - or checkpoint["size_bytes"] <= 12 - or checkpoint.get("global_step") != learner["global_rounds"] - ): - raise HarnessError("plan has an invalid eval checkpoint identity") - if ( - checkpoint["source_host"] == _syncer_host(plan) - and checkpoint["source_path"] - == f"{plan['remote_run']}/state/state.ckpt" - ): - raise HarnessError("eval checkpoint source and fresh destination coincide") - elif plan.get("eval_checkpoint") is not None: - raise HarnessError("eval checkpoint requires an eval-only learner") + if learner.get("evaluation") is not None or plan.get("eval_checkpoint") is not None: + # SSH evaluation ran only the SecRLEnv agent, removed in S26 E0.5. + raise HarnessError("SSH evaluation is restricted to the removed secrlenv agent") fragments = learner.get("fragments", 1) pipeline = learner.get("pipeline", 1) local_horizon = learner.get("local_horizon", 1) @@ -1435,38 +875,7 @@ def _validate_plan(plan: dict[str, Any]) -> None: type(max_replacements) is not int or max_replacements < 0 ): raise HarnessError("plan has an invalid dynamic_sampling_max_replacements") - infrastructure_replacements = learner.get( - "secrlenv_max_infrastructure_replacements" - ) - agent_function = learner.get("custom_agent_function_path") - if agent_function in SECRLENV_AGENTS: - if learner.get("custom_generate_function_path") != SECRLENV_GENERATE: - raise HarnessError( - "secrlenv agent requires the signed generate wrapper" - ) - if learner.get("reward_function") != SECRLENV_REWARD: - raise HarnessError("secrlenv agent requires the signed reward function") - if dynamic_filter != SECRLENV_GROUP_FILTER: - raise HarnessError("secrlenv agent requires the signed group filter") - if max_replacements != SECRLENV_ZERO_VARIANCE_REPLACEMENTS or isinstance( - max_replacements, bool - ): - raise HarnessError( - "secrlenv agent requires zero variance replacements" - ) - if ( - infrastructure_replacements - != SECRLENV_INFRASTRUCTURE_REPLACEMENTS - or isinstance(infrastructure_replacements, bool) - ): - raise HarnessError( - "secrlenv agent requires exactly one infrastructure replacement" - ) - if learner["over_sampling_batch_size"] != learner["groups_per_round"] + 1: - raise HarnessError( - "secrlenv agent requires one reserved same-task retry slot" - ) - elif infrastructure_replacements is not None: + if learner.get("secrlenv_max_infrastructure_replacements") is not None: raise HarnessError( "secrlenv infrastructure replacements require a secrlenv agent" ) @@ -1526,10 +935,7 @@ def _validate_plan(plan: dict[str, Any]) -> None: or not learner.get("tito_model") ): raise HarnessError("plan has an incomplete agentic session contract") - codex_harness = plan.get("codex_harness") - if custom_agent in SIGNED_CODEX_AGENTS: - _validate_codex_harness(codex_harness, learner) - elif codex_harness is not None: + if plan.get("codex_harness") is not None: raise HarnessError("stock Codex contract requires the signed Codex agent") roles = learner.get("tito_allowed_append_roles") if roles is not None and ( @@ -1677,7 +1083,6 @@ def prepare(namespace) -> Path: data = args.data data_local_path = None data_sha256 = None - data_prompt_count = None if dataset["source"] == "local": source = Path(args.data).expanduser().absolute() data_sha256 = _local_data_sha256(source) @@ -1686,8 +1091,6 @@ def prepare(namespace) -> Path: data = "/workspace/data/dataset" if source.is_file(): data += source.suffix - if getattr(namespace, "eval_checkpoint_host", None) is not None: - data_prompt_count = _local_jsonl_record_count(source) tms_patch_path = getattr(namespace, "tms_preload_patch", None) tms_preload_patch = ( @@ -1793,9 +1196,6 @@ def prepare(namespace) -> Path: "dynamic_sampling_max_replacements": getattr( args, "dynamic_sampling_max_replacements", None ), - "secrlenv_max_infrastructure_replacements": getattr( - args, "secrlenv_max_infrastructure_replacements", None - ), "rl_offload_train": bool(getattr(args, "rl_offload_train", False)), "rl_distributed_timeout_minutes": getattr( args, "rl_distributed_timeout_minutes", 10 @@ -1809,17 +1209,11 @@ def prepare(namespace) -> Path: "custom_agent_function_path": getattr( args, "custom_agent_function_path", None ), - "codex_reasoning_effort": getattr( - args, "codex_reasoning_effort", None - ), "agent_max_seq_len": getattr(args, "agent_max_seq_len", None), "use_session_server": args.use_session_server, "session_server_ip": args.session_server_ip, "session_server_port": args.session_server_port, "tito_model": args.tito_model, - "codex_backend_profile": getattr( - args, "codex_backend_profile", None - ), "tito_allowed_append_roles": getattr( args, "tito_allowed_append_roles", None ), @@ -1853,144 +1247,13 @@ def prepare(namespace) -> Path: "cybergym_reward_view": os.environ.get("CYBERGYM_REWARD_VIEW", "train"), }, } - if getattr(args, "codex_exclude_reasoning_from_loss", False): - # S17 WP6: only present when opted out, so default plans are unchanged. - plan["learner"]["codex_exclude_reasoning_from_loss"] = True final_ack_timeout_s = getattr(namespace, "final_ack_timeout_s", None) if final_ack_timeout_s is not None: plan["final_ack_timeout_s"] = final_ack_timeout_s - eval_checkpoint_host = getattr(namespace, "eval_checkpoint_host", None) - eval_checkpoint_path = getattr(namespace, "eval_checkpoint_path", None) - eval_checkpoint_sha256 = getattr(namespace, "eval_checkpoint_sha256", None) - eval_checkpoint_size_bytes = getattr( - namespace, "eval_checkpoint_size_bytes", None - ) - eval_checkpoint_global_step = getattr( - namespace, "eval_checkpoint_global_step", None - ) - checkpoint_values = ( - eval_checkpoint_host, - eval_checkpoint_path, - eval_checkpoint_sha256, - eval_checkpoint_size_bytes, - eval_checkpoint_global_step, - ) - if any(value is not None for value in checkpoint_values) and any( - value is None for value in checkpoint_values - ): - raise HarnessError( - "all remote eval checkpoint identity options are required together" - ) - eval_interval = getattr(namespace, "eval_interval", None) - eval_dataset_name = getattr(namespace, "eval_dataset_name", None) - eval_options = { - "temperature": getattr(namespace, "eval_temperature", None), - "top_p": getattr(namespace, "eval_top_p", None), - "max_prompt_len": getattr(namespace, "eval_max_prompt_len", None), - "max_response_len": getattr(namespace, "eval_max_response_len", None), - "max_context_len": getattr(namespace, "eval_max_context_len", None), - } - if eval_checkpoint_host is None: - if ( - eval_interval is not None - or eval_dataset_name is not None - or getattr(namespace, "eval_samples_per_prompt", None) is not None - or any(value is not None for value in eval_options.values()) - ): - raise HarnessError("evaluation options require --eval-checkpoint-host") - else: - plan["eval_checkpoint"] = _eval_checkpoint_contract( - eval_checkpoint_host, - eval_checkpoint_path, - eval_checkpoint_sha256, - eval_checkpoint_size_bytes, - eval_checkpoint_global_step, - ) - eval_samples_per_prompt = getattr( - namespace, "eval_samples_per_prompt", None - ) - plan["learner"]["evaluation"] = { - "eval_only": True, - "dataset_name": eval_dataset_name, - "data": data, - "data_sha256": data_sha256, - "interval": 1 if eval_interval is None else eval_interval, - "samples_per_prompt": ( - 1 if eval_samples_per_prompt is None else eval_samples_per_prompt - ), - "prompt_count": data_prompt_count, - "skip_before_train": True, - **eval_options, - } jit_cache_root = getattr(namespace, "jit_cache_root", None) if jit_cache_root is not None: _require_cuda_jit_cache(plan) plan["jit_cache"] = _jit_cache_contract(jit_cache_root, plan) - daemon_source_root = getattr(namespace, "secrlenv_source_root", None) - if args.custom_agent_function_path in SECRLENV_AGENTS: - required = { - "--secrlenv-source-root": daemon_source_root, - "--secrlenv-source-sha256": getattr( - namespace, "secrlenv_source_sha256", None - ), - "--secrlenv-task-pack": getattr(namespace, "secrlenv_task_pack", None), - "--secrlenv-task-pack-sha256": getattr( - namespace, "secrlenv_task_pack_sha256", None - ), - "--secrlenv-operator-image": getattr( - namespace, "secrlenv_operator_image", None - ), - "--secrlenv-operator-image-id": getattr( - namespace, "secrlenv_operator_image_id", None - ), - } - missing = [flag for flag, value in required.items() if not value] - if missing: - raise HarnessError( - "secrlenv custom agent requires " + ", ".join(missing) - ) - plan["secrlenv_daemon"] = { - "source_root": daemon_source_root, - "source_sha256": namespace.secrlenv_source_sha256, - "task_pack": namespace.secrlenv_task_pack, - "task_pack_sha256": namespace.secrlenv_task_pack_sha256, - "state_root": f"/data/yeto-rl/secrlenv-runs/{run_id}", - "bind": "127.0.0.1", - "placement": "island-heads", - "port": namespace.secrlenv_port, - "operator_image": namespace.secrlenv_operator_image, - "operator_image_id": namespace.secrlenv_operator_image_id, - "max_active_episodes": namespace.secrlenv_max_active_episodes, - "enable_dind_debug": bool(namespace.secrlenv_enable_dind_debug), - "dind_image": ( - namespace.secrlenv_dind_image - if namespace.secrlenv_enable_dind_debug - else None - ), - "dind_image_id": ( - namespace.secrlenv_dind_image_id - if namespace.secrlenv_enable_dind_debug - else None - ), - "dind_debug_script_sha256": ( - namespace.secrlenv_dind_debug_script_sha256 - if namespace.secrlenv_enable_dind_debug - else None - ), - } - elif daemon_source_root is not None: - raise HarnessError( - "--secrlenv-source-root requires the secrlenv custom agent" - ) - codex_binary = getattr(namespace, "codex_harness_binary", None) - codex_manifest = getattr(namespace, "codex_package_manifest", None) - codex_schema = getattr(namespace, "codex_app_server_schema", None) - if args.custom_agent_function_path in SIGNED_CODEX_AGENTS: - plan["codex_harness"] = _codex_harness_contract(namespace, args) - elif any(value is not None for value in (codex_binary, codex_manifest, codex_schema)): - raise HarnessError( - "Codex artifact options require the signed stock Codex agent" - ) if getattr(args, "expert_full_count", 0): # Recorded only so plan validation rejects it: expert-full needs the # full-parameter sync preset, which this harness does not run. @@ -2044,228 +1307,10 @@ def _remote_vars(plan: dict[str, Any]) -> str: return f'RUN="$HOME/{plan["remote_run"]}"' -def _secrlenv_daemon_script(plan: dict[str, Any]) -> str: - daemon = plan["secrlenv_daemon"] - dind_image_check = "" - dind_server_args = "" - dind_existing_identity = """ - case "$ARGS" in - *--enable-dind-debug*) - echo 'existing secrlenv daemon unexpectedly enables DinD debug' >&2 - exit 1 - ;; - esac -""" - if daemon["enable_dind_debug"]: - dind_image_check = f""" -DIND_IMAGE_ID="$(docker image inspect --format '{{{{.Id}}}}' {shlex.quote(daemon['dind_image'])})" -test "$DIND_IMAGE_ID" = {shlex.quote(daemon['dind_image_id'])} -DIND_DEBUG_SCRIPT="$SOURCE/pipeline/debug_instance.sh" -test -f "$DIND_DEBUG_SCRIPT" && test ! -L "$DIND_DEBUG_SCRIPT" -printf '%s %s\n' {shlex.quote(daemon['dind_debug_script_sha256'])} "$DIND_DEBUG_SCRIPT" | sha256sum --check - -""" - dind_server_args = ( - f" --dind-image {shlex.quote(daemon['dind_image_id'])} \\\n" - " --enable-dind-debug \\\n" - ) - dind_existing_identity = f""" - case "$ARGS" in - *--dind-image*{shlex.quote(daemon['dind_image_id'])}*--enable-dind-debug*) ;; - *) echo 'existing secrlenv daemon has the wrong DinD debug identity' >&2; exit 1 ;; - esac -""" - return f"""set -euo pipefail -{_remote_vars(plan)} -SOURCE={shlex.quote(daemon['source_root'])} -TASK_PACK={shlex.quote(daemon['task_pack'])} -STATE_ROOT={shlex.quote(daemon['state_root'])} -ENV_FILE="$HOME/{plan['remote_env_file']}" -test -d "$SOURCE/secrlenv_rl" -test -d "$TASK_PACK" -test -f "$ENV_FILE" && test ! -L "$ENV_FILE" -test "$(stat -c '%a' "$ENV_FILE")" = 600 -SOURCE_SHA="$(cd "$SOURCE" && find secrlenv_rl -type f -name '*.py' -print0 | sort -z | xargs -0 sha256sum | sha256sum | awk '{{print $1}}')" -test "$SOURCE_SHA" = {shlex.quote(daemon['source_sha256'])} -PYTHONPATH="$RUN/source:$SOURCE${{PYTHONPATH:+:$PYTHONPATH}}" \ - python3 -m yeto.rl.secrlenv_task_images \ - --task-pack "$TASK_PACK" \ - --expected-task-pack-sha256 {shlex.quote(daemon['task_pack_sha256'])} \ - --data-root /data -IMAGE_ID="$(docker image inspect --format '{{{{.Id}}}}' {shlex.quote(daemon['operator_image'])})" -test "$IMAGE_ID" = {shlex.quote(daemon['operator_image_id'])} -{dind_image_check}if [ -s "$STATE_ROOT/daemon.pid" ]; then - PID="$(cat "$STATE_ROOT/daemon.pid")" - ARGS="$(ps -p "$PID" -o args= 2>/dev/null || true)" - case "$ARGS" in - *secrlenv_rl.server*"$STATE_ROOT/state"*) ;; - *) echo 'existing secrlenv daemon PID has the wrong identity' >&2; exit 1 ;; - esac -{dind_existing_identity}else - if python3 - <<'PY' -import socket -s = socket.socket() -try: - s.bind(({daemon['bind']!r}, {daemon['port']})) -finally: - s.close() -PY - then :; else - echo 'secrlenv daemon port is occupied by an unattested process' >&2 - exit 1 - fi - mkdir -p "$STATE_ROOT/state" - TOKEN_FILE="$STATE_ROOT/daemon.token" - if [ ! -e "$TOKEN_FILE" ]; then - umask 077 - python3 - <<'PY' >"$TOKEN_FILE" -import secrets - -print(secrets.token_hex(32)) -PY - fi - test -f "$TOKEN_FILE" && test ! -L "$TOKEN_FILE" - test "$(stat -c '%a' "$TOKEN_FILE")" = 600 - cd "$SOURCE" - nohup setsid env PYTHONPATH="$SOURCE${{PYTHONPATH:+:$PYTHONPATH}}" \ - python3 -m secrlenv_rl.server \ - --task-pack "$TASK_PACK" \ - --state-dir "$STATE_ROOT/state" \ - --repository-root "$SOURCE" \ - --token-file "$TOKEN_FILE" \ - --bind {shlex.quote(daemon['bind'])} \ - --port {daemon['port']} \ - --operator-image {shlex.quote(daemon['operator_image'])} \ -{dind_server_args} --max-active-episodes {daemon['max_active_episodes']} \ - >"$STATE_ROOT/daemon.log" 2>&1 < /dev/null & - echo "$!" > "$STATE_ROOT/daemon.pid" -fi -python3 - <<'PY' -import json -import time -from urllib.request import urlopen - -deadline = time.monotonic() + 60 -last_error = None -while time.monotonic() < deadline: - try: - with urlopen("http://{daemon['bind']}:{daemon['port']}/healthz", timeout=2) as response: - value = json.load(response) - if ( - response.status == 200 - and value.get("ok") is True - and value.get("task_pack_sha256") == {daemon['task_pack_sha256']!r} - and value.get("max_active_episodes") == {daemon['max_active_episodes']} - ): - print("secrlenv_daemon=ready task_pack=" + value["task_pack_sha256"]) - raise SystemExit(0) - last_error = "health identity mismatch" - except Exception as exc: - last_error = str(exc) - time.sleep(1) -raise SystemExit("secrlenv daemon did not become ready: " + str(last_error)) -PY -""" - - -def _start_secrlenv_daemons(plan: dict[str, Any]) -> None: - if plan.get("secrlenv_daemon") is None: - return - for target in _secrlenv_daemon_hosts(plan): - _ssh(plan, target, _secrlenv_daemon_script(plan)) - - -def _stop_secrlenv_daemons(plan: dict[str, Any]) -> None: - daemon = plan.get("secrlenv_daemon") - if daemon is None: - return - for target in _secrlenv_daemon_hosts(plan): - _ssh( - plan, - target, - f"""set -euo pipefail -STATE_ROOT={shlex.quote(daemon['state_root'])} -if [ ! -s "$STATE_ROOT/daemon.pid" ]; then exit 0; fi -PID="$(cat "$STATE_ROOT/daemon.pid")" -ARGS="$(ps -p "$PID" -o args= 2>/dev/null || true)" -case "$ARGS" in - *secrlenv_rl.server*"$STATE_ROOT/state"*) ;; - '') exit 0 ;; - *) echo 'refusing to stop a process with the wrong identity' >&2; exit 1 ;; -esac -kill -TERM -- -"$PID" -for _ in {{1..600}}; do - kill -0 "$PID" 2>/dev/null || exit 0 - sleep 0.1 -done -echo 'secrlenv daemon did not exit after SIGTERM' >&2 -exit 1 -""", - ) - - -def _secrlenv_daemon_status_script(plan: dict[str, Any]) -> str: - daemon = plan.get("secrlenv_daemon") - if daemon is None: - return "" - return f"""STATE_ROOT={shlex.quote(daemon['state_root'])} -if [ ! -s "$STATE_ROOT/daemon.pid" ]; then - echo 'secrlenv_daemon=stopped' -else - PID="$(cat "$STATE_ROOT/daemon.pid")" - case "$PID" in - ''|*[!0-9]*) echo 'secrlenv_daemon=identity-drifted' >&2; exit 1 ;; - esac - ARGS="$(ps -p "$PID" -o args= 2>/dev/null || true)" - case "$ARGS" in - *secrlenv_rl.server*"$STATE_ROOT/state"*) ;; - '') echo 'secrlenv_daemon=stopped' ;; - *) echo 'secrlenv_daemon=identity-drifted' >&2; exit 1 ;; - esac - if [ -n "$ARGS" ]; then - python3 - "$PID" <<'PY' || exit 1 -import json -import sys -from urllib.request import urlopen - -try: - with urlopen("http://{daemon['bind']}:{daemon['port']}/healthz", timeout=2) as response: - value = json.load(response) -except Exception: - print("secrlenv_daemon=unhealthy") - raise SystemExit(1) -if not ( - response.status == 200 - and value.get("ok") is True - and value.get("task_pack_sha256") == {daemon['task_pack_sha256']!r} - and value.get("max_active_episodes") == {daemon['max_active_episodes']} -): - print("secrlenv_daemon=identity-drifted") - raise SystemExit(1) -print("secrlenv_daemon=running pid=" + sys.argv[1]) -PY - fi -fi -""" - - def _all_hosts(plan: dict[str, Any]) -> list[str]: return [host for island in plan["islands"] for host in island["hosts"]] -def _secrlenv_daemon_hosts(plan: dict[str, Any]) -> list[str]: - """Return the pinned daemon roster, retaining old-plan cleanup semantics.""" - - daemon = plan.get("secrlenv_daemon") - if daemon is None: - return [] - placement = daemon.get("placement", "all-hosts") - if placement == "all-hosts": - return _all_hosts(plan) - if placement == "island-heads": - return [island["hosts"][0] for island in plan["islands"]] - raise HarnessError("secrlenv daemon has an invalid placement") - - def _deployment_hosts(plan: dict[str, Any]) -> list[str]: hosts = _all_hosts(plan) syncer_host = _syncer_host(plan) @@ -2292,65 +1337,12 @@ def _attest_local(plan: dict[str, Any]) -> None: != plan["reward_sha256"] ): raise HarnessError("reward source changed after the plan was prepared") - learner = plan["learner"] - codex_harness = plan.get("codex_harness") - if codex_harness is not None: - _validate_codex_harness(codex_harness, learner) - binary = Path(codex_harness["controller_binary_path"]) - manifest = Path(codex_harness["controller_package_manifest_path"]) - schema = Path(codex_harness["controller_app_server_schema_path"]) - for path, expected_size, expected_sha256, executable in ( - ( - binary, - codex_harness["binary_size_bytes"], - codex_harness["binary_sha256"], - True, - ), - (manifest, None, codex_harness["package_manifest_sha256"], False), - (schema, None, codex_harness["app_server_schema_sha256"], False), - ): - if ( - path.is_symlink() - or not path.is_file() - or (executable and not path.stat().st_mode & 0o111) - or (expected_size is not None and path.stat().st_size != expected_size) - or file_sha256(path) != expected_sha256 - ): - raise HarnessError("Codex controller artifact changed after prepare") - if learner.get("custom_agent_function_path") != CODEX_OPENENV_AGENT: - try: - adapter_path = _codex_adapter_source_path() - adapter_identity = _codex_adapter_identity() - except HarnessError: - raise HarnessError( - "Yeto Codex adapter changed after prepare" - ) from None - if ( - file_sha256(adapter_path) - != codex_harness["agent_source_sha256"] - or adapter_identity - != { - name: codex_harness[name] - for name in ( - "base_instructions_sha256", - "terminal_exec_tool_schema_sha256", - "submit_tool_schema_sha256", - "dynamic_tools_schema_sha256", - ) - } - ): - raise HarnessError("Yeto Codex adapter changed after prepare") data_local_path = plan["learner"].get("data_local_path") if ( data_local_path is not None and _local_data_sha256(data_local_path) != plan["learner"]["data_sha256"] ): raise HarnessError("local dataset changed after the plan was prepared") - evaluation = plan["learner"].get("evaluation") - if evaluation is not None and _local_jsonl_record_count( - data_local_path - ) != evaluation.get("prompt_count"): - raise HarnessError("evaluation dataset record count changed after prepare") def deploy(plan_path: str | Path) -> None: @@ -2360,31 +1352,6 @@ def deploy(plan_path: str | Path) -> None: _attest_local(plan) digest = _plan_digest(plan) local_data = plan["learner"].get("data_local_path") - eval_checkpoint = plan.get("eval_checkpoint") - if eval_checkpoint is not None: - _require_program("scp") - source_path = eval_checkpoint["source_path"] - _ssh( - plan, - eval_checkpoint["source_host"], - f"""set -euo pipefail -SOURCE="$HOME/{source_path}" -test -f "$SOURCE" && test ! -L "$SOURCE" -test "$(stat -c %s "$SOURCE")" = {eval_checkpoint['size_bytes']} -printf '%s %s\n' {shlex.quote(eval_checkpoint['sha256'])} "$SOURCE" | sha256sum --check - -python3 - "$SOURCE" {eval_checkpoint['global_step']} <<'PY' -import sys -from pathlib import Path - -header = Path(sys.argv[1]).open("rb").read(12) -expected_step = int(sys.argv[2]) -if len(header) != 12 or int.from_bytes(header[:4], "little") != {SYNCER_CHECKPOINT_MAGIC}: - raise SystemExit("invalid syncer checkpoint header") -if int.from_bytes(header[4:12], "little") != expected_step: - raise SystemExit("syncer checkpoint global step mismatch") -PY -""", - ) excludes = ( ".git/", ".venv/", @@ -2404,11 +1371,7 @@ def deploy(plan_path: str | Path) -> None: learner_hosts = set(_all_hosts(plan)) for target in _deployment_hosts(plan): deploy_data = local_data is not None and target in learner_hosts - deploy_checkpoint = eval_checkpoint is not None and target == _syncer_host(plan) - deploy_codex = plan.get("codex_harness") is not None and target in learner_hosts data_directory = '\nmkdir -p "$RUN/data"' if deploy_data else "" - state_directory = '\nmkdir -p "$RUN/state"' if deploy_checkpoint else "" - codex_directory = '\nmkdir -p "$RUN/codex"' if deploy_codex else "" initialize = f"""set -euo pipefail {_remote_vars(plan)} if [ -f "$RUN/control/plan.sha256" ]; then @@ -2422,7 +1385,7 @@ def deploy(plan_path: str | Path) -> None: printf '%s\n' {shlex.quote(digest)} > "$RUN/control/deploying.sha256" fi fi -{data_directory}{state_directory}{codex_directory} +{data_directory} """ _ssh(plan, target, initialize) _run( @@ -2462,69 +1425,6 @@ def deploy(plan_path: str | Path) -> None: + ("/" if is_directory else ""), ] ) - if deploy_codex: - codex = plan["codex_harness"] - for source_path, bundle_path in ( - (codex["controller_binary_path"], codex["bundle_binary_path"]), - ( - codex["controller_package_manifest_path"], - codex["bundle_package_manifest_path"], - ), - ( - codex["controller_app_server_schema_path"], - codex["bundle_app_server_schema_path"], - ), - ): - _run( - [ - "rsync", - "-az", - "-e", - _rsync_shell(plan), - source_path, - f"{target}:{plan['remote_run']}/{bundle_path}", - ] - ) - _ssh( - plan, - target, - f"""set -euo pipefail -{_remote_vars(plan)} -CODEX="$RUN/{codex['bundle_binary_path']}" -MANIFEST="$RUN/{codex['bundle_package_manifest_path']}" -SCHEMA="$RUN/{codex['bundle_app_server_schema_path']}" -test -f "$CODEX" && test ! -L "$CODEX" && test -x "$CODEX" -test "$(stat -c %s "$CODEX")" = {codex['binary_size_bytes']} -printf '%s %s\n' {shlex.quote(codex['binary_sha256'])} "$CODEX" | sha256sum --check - -test -f "$MANIFEST" && test ! -L "$MANIFEST" -printf '%s %s\n' {shlex.quote(codex['package_manifest_sha256'])} "$MANIFEST" | sha256sum --check - -test -f "$SCHEMA" && test ! -L "$SCHEMA" -printf '%s %s\n' {shlex.quote(codex['app_server_schema_sha256'])} "$SCHEMA" | sha256sum --check - -""", - ) - if deploy_checkpoint: - if eval_checkpoint["source_host"] == target: - _ssh( - plan, - target, - f"""set -euo pipefail -{_remote_vars(plan)} -cp --reflink=auto -- "$HOME/{eval_checkpoint['source_path']}" "$RUN/state/state.ckpt" -""", - ) - else: - _run( - [ - "scp", - "-3", - *plan.get("ssh_options", []), - ( - f"{eval_checkpoint['source_host']}:" - f"{eval_checkpoint['source_path']}" - ), - f"{target}:{plan['remote_run']}/state/state.ckpt", - ] - ) _ssh( plan, target, @@ -2955,16 +1855,9 @@ def _syncer_runtime_identity_functions() -> str: def _syncer_start_script(plan: dict[str, Any]) -> str: unit = shlex.quote(_syncer_unit_name(plan)) - eval_checkpoint = plan.get("eval_checkpoint") - checkpoint_check = "" - if eval_checkpoint is not None: - checkpoint_check = f"""test -f "$RUN/state/state.ckpt" -test "$(stat -c %s "$RUN/state/state.ckpt")" = {eval_checkpoint['size_bytes']} -printf '%s %s\n' {shlex.quote(eval_checkpoint['sha256'])} "$RUN/state/state.ckpt" | sha256sum --check - -""" return f"""set -euo pipefail {_remote_vars(plan)} -{checkpoint_check}command -v systemctl >/dev/null +command -v systemctl >/dev/null command -v systemd-run >/dev/null test -d /run/systemd/system UNIT={unit} @@ -3224,30 +2117,6 @@ def _learner_argv(plan: dict[str, Any], learner_id: int) -> list[str]: values.extend(("--wandb-mode", str(learner.get("wandb_mode", "online")))) if learner.get("wandb_entity"): values.extend(("--wandb-entity", str(learner["wandb_entity"]))) - evaluation = learner.get("evaluation") - if evaluation is not None: - values.append("--eval-only") - values.extend( - ( - "--eval-dataset-name", - evaluation["dataset_name"], - "--eval-data-sha256", - evaluation["data_sha256"], - "--eval-interval", - str(evaluation["interval"]), - "--eval-samples-per-prompt", - str(evaluation["samples_per_prompt"]), - ) - ) - for flag, name in ( - ("--eval-temperature", "temperature"), - ("--eval-top-p", "top_p"), - ("--eval-max-prompt-len", "max_prompt_len"), - ("--eval-max-response-len", "max_response_len"), - ("--eval-max-context-len", "max_context_len"), - ): - if evaluation.get(name) is not None: - values.extend((flag, str(evaluation[name]))) if learner.get("rollout_model") is not None: values.extend(("--rollout-model", learner["rollout_model"])) values.extend( @@ -3274,13 +2143,6 @@ def _learner_argv(plan: dict[str, Any], learner_id: int) -> list[str]: str(learner["dynamic_sampling_max_replacements"]), ) ) - if learner.get("secrlenv_max_infrastructure_replacements") is not None: - values.extend( - ( - "--secrlenv-max-infrastructure-replacements", - str(learner["secrlenv_max_infrastructure_replacements"]), - ) - ) if learner.get("dynamic_sampling_filter_path"): values.extend( ( @@ -3326,13 +2188,6 @@ def _learner_argv(plan: dict[str, Any], learner_id: int) -> list[str]: learner["custom_agent_function_path"], ) ) - if plan.get("codex_harness") is not None: - values.extend( - ( - "--codex-harness-contract", - _canonical_json(plan["codex_harness"]), - ) - ) if learner.get("agent_max_seq_len") is not None: values.extend(("--agent-max-seq-len", str(learner["agent_max_seq_len"]))) if learner.get("use_session_server"): @@ -3344,15 +2199,9 @@ def _learner_argv(plan: dict[str, Any], learner_id: int) -> list[str]: values.extend(str(port) for port in learner["session_server_port"]) if learner.get("tito_model"): values.extend(("--tito-model", learner["tito_model"])) - if learner.get("codex_backend_profile"): - values.extend( - ("--codex-backend-profile", learner["codex_backend_profile"]) - ) if learner.get("tito_allowed_append_roles"): values.append("--tito-allowed-append-roles") values.extend(learner["tito_allowed_append_roles"]) - if learner.get("codex_exclude_reasoning_from_loss"): - values.append("--codex-exclude-reasoning-from-loss") if learner["trust_remote_code"]: values.append("--trust-remote-code") return values @@ -3366,7 +2215,6 @@ def _node_start_script(plan: dict[str, Any], learner_id: int, node_id: int) -> s island = plan["islands"][learner_id] if node_id not in range(len(island["hosts"])): raise HarnessError("node ID is outside the island topology") - daemon_on_node = island["hosts"][node_id] in _secrlenv_daemon_hosts(plan) name = _container_name(plan, learner_id, node_id) head = _target_host(island["hosts"][0]) gpus = island["gpus_per_node"] @@ -3389,22 +2237,6 @@ def _node_start_script(plan: dict[str, Any], learner_id: int, node_id: int) -> s if learner.get("wandb") else "" ) - daemon_env = ( - " --env SECRLENV_DAEMON_URL=" - f"http://127.0.0.1:{plan['secrlenv_daemon']['port']} " - "--env SECRLENV_TASK_PACK_SHA256=" - f"{plan['secrlenv_daemon']['task_pack_sha256']} " - "--env SECRLENV_BEARER_TOKEN_FILE=/run/secrlenv/daemon.token \\\n" - if daemon_on_node - else "" - ) - daemon_volume = ( - " --volume " - f"{shlex.quote(plan['secrlenv_daemon']['state_root'])}/daemon.token:" - "/run/secrlenv/daemon.token:ro \\\n" - if daemon_on_node - else "" - ) reward_scheme = shlex.quote(learner.get("cybergym_reward_scheme", "binary")) reward_view = shlex.quote(learner.get("cybergym_reward_view", "train")) data_volume = ( @@ -3416,69 +2248,6 @@ def _node_start_script(plan: dict[str, Any], learner_id: int, node_id: int) -> s f" --volume {shlex.quote(f'{mount}:{mount}:ro')} \\\n" for mount in learner.get("model_mounts", []) ) - codex = plan.get("codex_harness") - if codex is not None: - codex_contract_sha256 = _plan_digest(codex) - codex_setup = f"""CODEX_HOST="$RUN/{codex['bundle_binary_path']}" -CODEX_MANIFEST_HOST="$RUN/{codex['bundle_package_manifest_path']}" -CODEX_SCHEMA_HOST="$RUN/{codex['bundle_app_server_schema_path']}" -test -f "$CODEX_HOST" && test ! -L "$CODEX_HOST" && test -x "$CODEX_HOST" -test "$(stat -c %s "$CODEX_HOST")" = {codex['binary_size_bytes']} -printf '%s %s\n' {shlex.quote(codex['binary_sha256'])} "$CODEX_HOST" | sha256sum --check - -test -f "$CODEX_MANIFEST_HOST" && test ! -L "$CODEX_MANIFEST_HOST" -printf '%s %s\n' {shlex.quote(codex['package_manifest_sha256'])} "$CODEX_MANIFEST_HOST" | sha256sum --check - -test -f "$CODEX_SCHEMA_HOST" && test ! -L "$CODEX_SCHEMA_HOST" -printf '%s %s\n' {shlex.quote(codex['app_server_schema_sha256'])} "$CODEX_SCHEMA_HOST" | sha256sum --check - -""" - openenv_env = "".join( - f"--env {name}={shlex.quote(value)} " - for name, value in codex.get("openenv_identity_env", {}).items() - ) - codex_env = ( - " --env YETO_CODEX_BINARY_PATH=" - f"{shlex.quote(codex['container_binary_path'])} " - "--env YETO_CODEX_BINARY_SHA256=" - f"{codex['binary_sha256']} " - f"--env YETO_CODEX_BINARY_SIZE_BYTES={codex['binary_size_bytes']} " - f"--env YETO_CODEX_VERSION={shlex.quote(codex['cli_version'])} " - "--env YETO_CODEX_APP_SERVER_PROTOCOL_REVISION=" - f"{codex['app_server_protocol_revision']} " - "--env YETO_CODEX_APP_SERVER_SCHEMA_SHA256=" - f"{codex['app_server_schema_sha256']} " - "--env YETO_CODEX_BASE_INSTRUCTIONS_SHA256=" - f"{codex['base_instructions_sha256']} " - "--env YETO_CODEX_TERMINAL_EXEC_TOOL_SCHEMA_SHA256=" - f"{codex['terminal_exec_tool_schema_sha256']} " - "--env YETO_CODEX_SUBMIT_TOOL_SCHEMA_SHA256=" - f"{codex['submit_tool_schema_sha256']} " - "--env YETO_CODEX_DYNAMIC_TOOLS_SCHEMA_SHA256=" - f"{codex['dynamic_tools_schema_sha256']} " - f"--env YETO_CODEX_REASONING_EFFORT={codex['reasoning_effort']} " - "--env YETO_CODEX_BACKEND_MAX_TOKENS=" - f"{codex['backend']['max_tokens']} " - "--env YETO_CODEX_BACKEND_REASONING_EFFORT=" - f"{codex['backend']['reasoning_effort']} " - "--env YETO_CODEX_BACKEND_THINKING=" - f"{codex['backend']['thinking']['type']} " - f"--env YETO_CODEX_CHAT_TEMPLATE={codex['backend']['chat_template']} " - "--env YETO_CODEX_CHAT_TEMPLATE_KWARGS=" - f"{shlex.quote(_canonical_json(codex['backend']['chat_template_kwargs']))} " - "--env YETO_CODEX_TITO_ALLOWED_APPEND_ROLES=tool,user " - f"{openenv_env}" - f"--env YETO_CODEX_HARNESS_CONTRACT_SHA256={codex_contract_sha256} \\\n" - ) - codex_volumes = ( - " --volume \"$CODEX_HOST:" - f"{codex['container_binary_path']}:ro\" \\\n" - " --volume \"$CODEX_MANIFEST_HOST:/opt/yeto/codex/" - "codex-package.json:ro\" \\\n" - " --volume \"$CODEX_SCHEMA_HOST:" - f"{codex['container_app_server_schema_path']}:ro\" \\\n" - ) - else: - codex_setup = "" - codex_env = "" - codex_volumes = "" tms_patch = plan.get("tms_preload_patch") tms_patch_volume = ( " --volume " @@ -3617,7 +2386,7 @@ def _node_start_script(plan: dict[str, Any], learner_id: int, node_id: int) -> s echo 'refusing to reuse a same-name learner container; use status or explicit restart-learner' >&2 exit 1 fi -{codex_setup}{tms_disk_setup}{jit_cache_setup}mkdir -p "$RUN/island-{learner_id}"/{{state,output,audit,cores}} +{tms_disk_setup}{jit_cache_setup}mkdir -p "$RUN/island-{learner_id}"/{{state,output,audit,cores}} docker run --detach \ --name {shlex.quote(name)} \ --gpus {shlex.quote(gpu_request)} \ @@ -3630,12 +2399,12 @@ def _node_start_script(plan: dict[str, Any], learner_id: int, node_id: int) -> s --env NCCL_IB_DISABLE=1 --env NCCL_SOCKET_IFNAME={network_interface} \ --env GLOO_SOCKET_IFNAME={network_interface} --env TORCH_NCCL_ASYNC_ERROR_HANDLING=1 \ --env CUDA_DEVICE_MAX_CONNECTIONS=1 \ -{attention_env}{recipe_env}{diagnostics_env}{tms_disk_env}{agent_env}{wandb_env}{daemon_env}{codex_env} --env CYBERGYM_URL={shlex.quote(learner['cybergym_url'])} \ +{attention_env}{recipe_env}{diagnostics_env}{tms_disk_env}{agent_env}{wandb_env} --env CYBERGYM_URL={shlex.quote(learner['cybergym_url'])} \ --env CYBERGYM_AGENT_ID={shlex.quote(learner['cybergym_agent_id'])} \ --env CYBERGYM_TIMEOUT={shlex.quote(str(learner['cybergym_timeout']))} \ --env CYBERGYM_REWARD_SCHEME={reward_scheme} \ --env CYBERGYM_REWARD_VIEW={reward_view} \ -{data_volume}{model_volumes}{tms_patch_volume}{tms_disk_volume}{jit_cache_volumes}{daemon_volume}{codex_volumes} --volume "$RUN/source:/workspace/yeto:ro" \ +{data_volume}{model_volumes}{tms_patch_volume}{tms_disk_volume}{jit_cache_volumes} --volume "$RUN/source:/workspace/yeto:ro" \ --volume "$RUN/sglang:/workspace/sglang:ro" \ --volume "$RUN/miles:/workspace/miles" \ --volume "$RUN/island-{learner_id}/state:/workspace/state" \ @@ -3661,7 +2430,6 @@ def start(plan_path: str | Path) -> None: _, plan = load_plan(plan_path) deploy(plan_path) _check_island_cards(plan) - _start_secrlenv_daemons(plan) for island in plan["islands"]: for target in island["hosts"]: _ssh(plan, target, _host_setup_script(plan, island["gpus_per_node"])) @@ -3729,7 +2497,6 @@ def _syncer_status_script(plan: dict[str, Any]) -> str: def status(plan_path: str | Path) -> None: _, plan = load_plan(plan_path) - daemon_hosts = set(_secrlenv_daemon_hosts(plan)) if _syncer_host(plan) not in _all_hosts(plan): result = _ssh( plan, @@ -3751,18 +2518,13 @@ def status(plan_path: str | Path) -> None: syncer = "" if _syncer_host(plan) == target: syncer = _syncer_status_script(plan) - daemon = ( - _secrlenv_daemon_status_script(plan) - if target in daemon_hosts - else "" - ) result = _ssh( plan, target, f"""set -u {_remote_vars(plan)} echo host={shlex.quote(target)} island={learner_id} node={node_id} -{syncer}{daemon}if docker inspect {shlex.quote(name)} >/dev/null 2>&1; then +{syncer}if docker inspect {shlex.quote(name)} >/dev/null 2>&1; then printf 'container=' docker inspect --format {shlex.quote(_CONTAINER_STATE_FORMAT)} {shlex.quote(name)} else echo container=missing; fi @@ -3956,7 +2718,6 @@ def stop(plan_path: str | Path) -> None: _syncer_host(plan), _syncer_stop_script(plan), ) - _stop_secrlenv_daemons(plan) def _collect_event_evidence( @@ -4006,37 +2767,9 @@ def _collect_event_evidence( _atomic_bytes(destination, payload) -def _parse_syncer_lifecycle_evidence(raw: str) -> dict[str, int]: - if len(raw.encode()) > 4096: - raise HarnessError("syncer lifecycle evidence is unexpectedly large") - try: - evidence = json.loads(raw) - except (TypeError, json.JSONDecodeError) as error: - raise HarnessError("syncer lifecycle evidence is malformed") from error - if not isinstance(evidence, dict) or set(evidence) != { - "schema", - "resume_count", - "final_ack_count", - }: - raise HarnessError("syncer lifecycle evidence has unapproved fields") - if ( - evidence.get("schema") != 1 - or any( - isinstance(evidence.get(name), bool) - or not isinstance(evidence.get(name), int) - or not 0 <= evidence[name] <= 1_000_000 - for name in ("resume_count", "final_ack_count") - ) - ): - raise HarnessError("syncer lifecycle evidence has invalid counters") - return evidence - - def _collect_capacity_preflight(plan: dict[str, Any], artifacts: Path) -> None: """Refuse a large training transfer before it can fill the controller disk.""" - if plan["learner"].get("evaluation") is not None: - return total_bytes = (len(plan["islands"]) + 1) * 64 * 1024 * 1024 if plan["learner"].get("sync_preset", "strict-avg") != "decoupled": for learner_id, island in enumerate(plan["islands"]): @@ -4083,16 +2816,9 @@ def collect(plan_path: str | Path) -> Path: artifacts = plan_file.parent / "artifacts" artifacts.mkdir(parents=True, exist_ok=True) _collect_capacity_preflight(plan, artifacts) - eval_only = plan["learner"].get("evaluation") is not None sync_preset = plan["learner"].get("sync_preset", "strict-avg") learner_event_mode = ( - "eval-learner" - if eval_only - else ( - "decoupled-learner" - if sync_preset == "decoupled" - else "strict-learner" - ) + "decoupled-learner" if sync_preset == "decoupled" else "strict-learner" ) for learner_id, island in enumerate(plan["islands"]): destination = artifacts / f"island-{learner_id}" @@ -4122,7 +2848,7 @@ def collect(plan_path: str | Path) -> Path: learner_event_mode, output / "events.jsonl", ) - if not eval_only and sync_preset != "decoupled": + if sync_preset != "decoupled": audit = destination / "audit" audit.mkdir(parents=True, exist_ok=True) audit_includes = [ @@ -4144,34 +2870,17 @@ def collect(plan_path: str | Path) -> Path: ) syncer = artifacts / "syncer" syncer.mkdir(parents=True, exist_ok=True) - if eval_only: - lifecycle = _ssh( - plan, - _syncer_host(plan), - f"{_remote_vars(plan)}\n" - "python3 -c " - f"{shlex.quote(_SYNCER_LIFECYCLE_EVIDENCE)} " - '"$RUN/state/syncer.log"', - capture=True, - ) - evidence = _parse_syncer_lifecycle_evidence(lifecycle.stdout) - _atomic_bytes( - syncer / "lifecycle-evidence.json", - (_canonical_json(evidence) + "\n").encode(), - ) - syncer_includes = ("--include=/syncer.exit",) - else: - _collect_event_evidence( - plan, - _syncer_host(plan), - "state/events.jsonl", - "syncer", - syncer / "events.jsonl", - ) - syncer_includes = ( - "--include=/state.ckpt", - "--include=/syncer.exit", - ) + _collect_event_evidence( + plan, + _syncer_host(plan), + "state/events.jsonl", + "syncer", + syncer / "events.jsonl", + ) + syncer_includes = ( + "--include=/state.ckpt", + "--include=/syncer.exit", + ) _run( [ "rsync", @@ -4633,118 +3342,6 @@ def _container_succeeded(inspection: Any) -> bool: ) -def _read_successful_syncer_exit(path: Path) -> None: - try: - if path.stat().st_size > 4096: - raise HarnessError("eval syncer exit record is unexpectedly large") - lines = path.read_text(encoding="utf-8").splitlines() - pairs = [line.split("=", 1) for line in lines] - except OSError as error: - raise HarnessError("missing eval syncer exit record") from error - if any(len(pair) != 2 for pair in pairs) or len({pair[0] for pair in pairs}) != len( - pairs - ): - raise HarnessError("eval syncer exit record is malformed") - fields = dict(pairs) - if ( - fields.get("result") != "success" - or fields.get("exit_code") != "exited" - or fields.get("exit_status") != "0" - ): - raise HarnessError("eval syncer did not exit successfully after FINAL_ACK") - - -def _eval_metric(event: dict[str, Any], name: str) -> float: - value = event.get(name) - if ( - not isinstance(value, (int, float)) - or isinstance(value, bool) - or not math.isfinite(value) - or not 0.0 <= value <= 1.0 - ): - raise HarnessError(f"eval event has an invalid {name}") - return float(value) - - -def _verify_eval( - plan: dict[str, Any], artifacts: Path -) -> tuple[str, int, int, float, float]: - learner = plan["learner"] - evaluation = learner["evaluation"] - expected_version = learner["global_rounds"] - expected_samples = ( - evaluation["prompt_count"] * evaluation["samples_per_prompt"] - ) - if (artifacts / "syncer" / "state.ckpt").exists(): - raise HarnessError("eval artifact collection must not contain state.ckpt") - _read_successful_syncer_exit(artifacts / "syncer" / "syncer.exit") - try: - lifecycle = _parse_syncer_lifecycle_evidence( - (artifacts / "syncer" / "lifecycle-evidence.json").read_text( - encoding="utf-8" - ) - ) - except OSError as error: - raise HarnessError("missing eval syncer lifecycle evidence") from error - if lifecycle["resume_count"] != 1: - raise HarnessError("eval syncer lacks one checkpoint resume event") - if lifecycle["final_ack_count"] != _learner_count(plan): - raise HarnessError("eval syncer lacks one FINAL_ACK per learner") - - hashes = set() - results = [] - pass_at_1 = [] - for learner_id in range(_learner_count(plan)): - events = _json_lines(_event_path(artifacts, learner_id)) - apply_indices = [ - index - for index, event in enumerate(events) - if event.get("event") == "rl_policy_apply" - and event.get("policy_version") == expected_version - ] - eval_indices = [ - index - for index, event in enumerate(events) - if event.get("event") == "rl_eval_result" - and event.get("dataset_name") == evaluation["dataset_name"] - ] - if len(apply_indices) != 1 or len(eval_indices) != 1: - raise HarnessError( - f"eval learner {learner_id} lacks one final apply/result event" - ) - if apply_indices[0] >= eval_indices[0]: - raise HarnessError( - f"eval learner {learner_id} evaluated before final policy apply" - ) - apply_event = events[apply_indices[0]] - eval_event = events[eval_indices[0]] - policy_hash = apply_event.get("sync/global_policy_hash") - if not isinstance(policy_hash, str) or not _SHA256.fullmatch(policy_hash): - raise HarnessError(f"eval learner {learner_id} has no final policy hash") - if ( - apply_event.get("island_id") != learner_id - or eval_event.get("island_id") != learner_id - or eval_event.get("rollout_id") != 0 - or eval_event.get("policy_version") != expected_version - or eval_event.get("sample_count") != expected_samples - ): - raise HarnessError( - f"eval learner {learner_id} result has the wrong policy/sample identity" - ) - hashes.add(policy_hash) - results.append(_eval_metric(eval_event, "rl/eval/result")) - pass_at_1.append(_eval_metric(eval_event, "rl/eval/pass_at_1")) - if len(hashes) != 1: - raise HarnessError("eval learners applied different terminal policies") - return ( - evaluation["dataset_name"], - evaluation["prompt_count"], - evaluation["samples_per_prompt"], - statistics.fmean(results), - statistics.fmean(pass_at_1), - ) - - def verify(plan_path: str | Path, export_dir: str | None = None) -> None: plan_file, plan = load_plan(plan_path) artifacts = plan_file.parent / "artifacts" @@ -4766,15 +3363,6 @@ def verify(plan_path: str | Path, export_dir: str | None = None) -> None: raise HarnessError( f"island {learner_id} node {node_id} did not exit successfully" ) - if plan["learner"].get("evaluation") is not None: - if export_dir: - raise HarnessError("eval-only verification cannot export a checkpoint") - dataset, prompts, samples, result, pass_at_1 = _verify_eval(plan, artifacts) - print( - f"verified terminal eval {dataset}: {prompts} prompts x {samples} " - f"sample(s), result={result:.6f}, pass@1={pass_at_1:.6f}" - ) - return from ..export import parse_checkpoint checkpoint_path = artifacts / "syncer" / "state.ckpt" @@ -4840,55 +3428,7 @@ def build_parser() -> argparse.ArgumentParser: prepare_parser.add_argument("--remote-root", default=DEFAULT_REMOTE_ROOT) prepare_parser.add_argument("--remote-env-file", default=None) prepare_parser.add_argument("--final-ack-timeout-s", type=int, default=None) - prepare_parser.add_argument("--eval-checkpoint-host", default=None) - prepare_parser.add_argument("--eval-checkpoint-path", default=None) - prepare_parser.add_argument("--eval-checkpoint-sha256", default=None) - prepare_parser.add_argument("--eval-checkpoint-size-bytes", type=int, default=None) - prepare_parser.add_argument("--eval-checkpoint-global-step", type=int, default=None) - prepare_parser.add_argument("--eval-dataset-name", default=None) - prepare_parser.add_argument("--eval-interval", type=int, default=None) - prepare_parser.add_argument("--eval-samples-per-prompt", type=int, default=None) - prepare_parser.add_argument("--eval-temperature", type=float, default=None) - prepare_parser.add_argument("--eval-top-p", type=float, default=None) - prepare_parser.add_argument("--eval-max-prompt-len", type=int, default=None) - prepare_parser.add_argument("--eval-max-response-len", type=int, default=None) - prepare_parser.add_argument("--eval-max-context-len", type=int, default=None) prepare_parser.add_argument("--model-manifest-sha256", default=None) - prepare_parser.add_argument("--secrlenv-source-root", default=None) - prepare_parser.add_argument("--secrlenv-source-sha256", default=None) - prepare_parser.add_argument("--secrlenv-task-pack", default=None) - prepare_parser.add_argument("--secrlenv-task-pack-sha256", default=None) - prepare_parser.add_argument("--secrlenv-operator-image", default=None) - prepare_parser.add_argument("--secrlenv-operator-image-id", default=None) - prepare_parser.add_argument("--secrlenv-port", type=int, default=28765) - prepare_parser.add_argument( - "--secrlenv-max-active-episodes", type=int, default=16 - ) - prepare_parser.add_argument( - "--secrlenv-enable-dind-debug", - action="store_true", - help="give each solver a private flagless DinD challenge copy", - ) - prepare_parser.add_argument("--secrlenv-dind-image", default=None) - prepare_parser.add_argument("--secrlenv-dind-image-id", default=None) - prepare_parser.add_argument( - "--secrlenv-dind-debug-script-sha256", default=None - ) - prepare_parser.add_argument( - "--codex-harness-binary", - default=None, - help="controller path to the pinned official Codex Linux binary", - ) - prepare_parser.add_argument( - "--codex-package-manifest", - default=None, - help="controller path to the pinned codex-package.json identity", - ) - prepare_parser.add_argument( - "--codex-app-server-schema", - default=None, - help="controller path to the pinned experimental v2 app-server schema", - ) prepare_parser.add_argument("--tms-preload-patch", default=None) prepare_parser.add_argument( "--tms-train-disk-backup-root", diff --git a/yeto/rl/tbench_direct_preflight.py b/yeto/rl/tbench_direct_preflight.py deleted file mode 100644 index f8609c393..000000000 --- a/yeto/rl/tbench_direct_preflight.py +++ /dev/null @@ -1,129 +0,0 @@ -"""Pre-allocation attestation for direct Miles Terminal-Bench SAO launches.""" - -from __future__ import annotations - -import argparse -import importlib -import os -import sys -from pathlib import Path -from typing import Any - -from . import ( - CODEX_OPENENV_AGENT, - CODEX_OPENENV_AGENT_MODULES, - CODEX_OPENENV_IDENTITY_ENV, -) -from .codex_backend import QWEN35_08B_MODEL, QWEN35_08B_REVISION - -_REWARD_FUNCTION = "openenv_generate.reward_func" -_FILTER_FUNCTION = "openenv_generate.check_terminal_bench_episode" -_GENERATE_FUNCTION = "miles.rollout.generate_hub.agentic_tool_call.generate" -_COMPACTION_ENV = { - "YETO_CODEX_COMPACTION_ENABLED": "1", - "YETO_CODEX_COMPACTION_TRIGGER_TOKENS": "6144", - "YETO_CODEX_COMPACTION_SUMMARY_MAX_TOKENS": "1024", - "YETO_CODEX_MAX_COMPACTIONS": "3", -} -_TIMEOUT_ENV = { - "OPENENV_MAX_ROLLOUT_TIME_SECONDS": "1800", - "SECRLENV_MAX_ROLLOUT_TIME_SECONDS": "1800", -} - - -def _direct_options(argv: list[str]) -> argparse.Namespace: - parser = argparse.ArgumentParser(add_help=False, allow_abbrev=False) - parser.add_argument("--custom-generate-function-path") - parser.add_argument("--custom-agent-function-path") - parser.add_argument("--custom-rm-path") - parser.add_argument("--dynamic-sampling-filter-path") - parser.add_argument("--input-key") - parser.add_argument("--tito-model") - parser.add_argument("--sao-online-recipe") - parser.add_argument("--max-seq-len", type=int) - parser.add_argument("--num-gpus-per-node", type=int) - parser.add_argument("--sglang-mem-fraction-static", type=float) - parser.add_argument("--sglang-max-total-tokens", type=int) - parser.add_argument("--sglang-max-mamba-cache-size", type=int) - parser.add_argument("--sao-compaction", action="store_true") - parser.add_argument("--sao-one-gpu-island", action="store_true") - parser.add_argument("--colocate", action="store_true") - values, _ = parser.parse_known_args(argv) - return values - - -def _attest_adapter(miles_root: Path) -> None: - from yeto.rl.harness.codex.preflight import forward_legacy_openenv_preflight - - if miles_root.is_symlink() or not miles_root.is_dir(): - raise ValueError("the pinned Miles root is not a real directory") - forward_legacy_openenv_preflight(None, "qwen35_08b") - - -def preflight_tbench_codex_streaming( - argv: list[str], - *, - sao_context: dict[str, Any], - streaming_runtime: Any, - miles_root: str | Path, -) -> None: - """Fail before Miles parses arguments or creates any model/Ray actor.""" - - if ( - getattr(streaming_runtime, "trajectory_evidence_kind", None) - != "terminal-bench-2.1" - or getattr(streaming_runtime, "trajectory_evidence_schema_version", None) - != 2 - ): - raise ValueError( - "Terminal-Bench direct launch requires trajectory evidence v2" - ) - if ( - sao_context.get("benchmark") != "terminal-bench-2.1" - or sao_context.get("model") != QWEN35_08B_MODEL - or sao_context.get("base_model_revision") != QWEN35_08B_REVISION - or sao_context.get("rollout_model_revision") != QWEN35_08B_REVISION - ): - raise ValueError("Terminal-Bench direct launch model identity drifted") - - options = _direct_options(argv) - expected = { - "custom_generate_function_path": _GENERATE_FUNCTION, - "custom_agent_function_path": CODEX_OPENENV_AGENT, - "custom_rm_path": _REWARD_FUNCTION, - "dynamic_sampling_filter_path": _FILTER_FUNCTION, - "input_key": "messages", - "tito_model": "qwen35", - "sao_online_recipe": "coding", - "max_seq_len": 8192, - "num_gpus_per_node": 1, - "sglang_mem_fraction_static": 0.15, - "sglang_max_total_tokens": 393216, - "sglang_max_mamba_cache_size": 256, - "sao_compaction": True, - "sao_one_gpu_island": True, - "colocate": True, - } - mismatched = [ - name for name, value in expected.items() if getattr(options, name) != value - ] - if mismatched: - raise ValueError( - "Terminal-Bench direct Miles arguments drifted: " - + ", ".join(mismatched) - ) - - environment = { - **CODEX_OPENENV_IDENTITY_ENV, - **_COMPACTION_ENV, - **_TIMEOUT_ENV, - } - mismatched_environment = [ - name for name, value in environment.items() if os.getenv(name) != value - ] - if mismatched_environment: - raise ValueError( - "Terminal-Bench trusted container environment drifted: " - + ", ".join(mismatched_environment) - ) - _attest_adapter(Path(miles_root).expanduser().resolve()) diff --git a/yeto/rl/trajectory_evidence.py b/yeto/rl/trajectory_evidence.py index 7ee7c67da..3087c7e71 100644 --- a/yeto/rl/trajectory_evidence.py +++ b/yeto/rl/trajectory_evidence.py @@ -498,14 +498,12 @@ def _verified_outcome( if "tbench_trusted_outcome" in metadata: raise ValueError("trajectory metadata mixes benchmark evidence kinds") - from yeto.rl.harness.codex.reward import ( + from .secrlenv_outcome import ( CLEANUP_ERROR_STATUS, INFRASTRUCTURE_STATUS, - _canonical, - ) - from yeto.rl.harness.codex.reward import ( - _verified_outcome as verified_secrlenv_outcome, + canonical_outcome as _canonical, ) + from .secrlenv_outcome import verified_outcome as verified_secrlenv_outcome outcome, reward = verified_secrlenv_outcome(dict(metadata)) if outcome["status"] in {INFRASTRUCTURE_STATUS, CLEANUP_ERROR_STATUS}: