diff --git a/CHANGELOG.md b/CHANGELOG.md index d5bd633..224de25 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,11 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver ## Unreleased +### Fixed + +- Served OneJev uses one HTTP decision deadline across its permitted retry. A retry receives only the remaining + `ONEJEV_TIMEOUT_S` budget, and an exhausted budget fails without sending another request. + ### Changed - The direct TypeSafe path pins `jev-1.13.0` instead of the moving `jev-latest` alias (which moved to Jev 1.14.0 @@ -14,6 +19,12 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver ### Added +- `--model onejev`: [OneJev](https://github.com/OmniJev/OneJev) (Apache-2.0) as a **served** multimodal System One + client against `qev`/`flide` (`ONEJEV_URL`, optional `ONEJEV_MODEL` / `ONEJEV_API_KEY` / `ONEJEV_TIMEOUT_S`). Posts + choice/noul with TypeSafe-compatible `media` and `` state markers; refuses an observation without a + screenshot (OmniJev policy). Contract tests use a fake HTTP server; live GPU validation is not claimed. Browser + and `build_model` only (#88). + - `s1a probe --order reverse|shuffle:N` (with `--seed` and `--json PATH`): the option-order check. Every case is asked in file order and again with the same keys and texts in another dict order; the probe reports the flip rate, accuracy in each order and a row per case with every order asked. Verdict and exit code are unchanged diff --git a/NOTICE b/NOTICE index f2c58b0..d66d358 100644 --- a/NOTICE +++ b/NOTICE @@ -21,3 +21,5 @@ This product includes software developed by third parties: - evals/labelled/injection-public.jsonl: tool outputs from AgentDojo, ethz-spylab/agentdojo (MIT, Copyright (c) 2024 Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr). Full license text: THIRD_PARTY_LICENSES.md +- s1a/decision_models/onejev.py: HTTP client for OneJev / qev (OmniJev/OneJev), Apache-2.0; + the weights and server are not vendored. Details: THIRD_PARTY_LICENSES.md diff --git a/THIRD_PARTY_LICENSES.md b/THIRD_PARTY_LICENSES.md index 2005a81..ac25900 100644 --- a/THIRD_PARTY_LICENSES.md +++ b/THIRD_PARTY_LICENSES.md @@ -116,3 +116,13 @@ TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONIN THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +## OneJev / qev + +The `--model onejev` adapter (`s1a/decision_models/onejev.py`) talks to a locally served +[OneJev](https://github.com/OmniJev/OneJev) checkpoint through the project's `qev` or +[Flide](https://github.com/OmniJev/Flide) HTTP server (`POST /v1/systemone` with a `media` field). +OneJev and its serving code are Apache License 2.0 (Copyright OmniJev Team). This repository does not +vendor OneJev weights, training code, or the `qev`/`flide` packages; install and run them separately +per the model card. The unrelated PyPI package named `qev` is not the OneJev server — install from the +OneJev GitHub repository as the model card instructs. diff --git a/docs/agents.md b/docs/agents.md index 75f36f2..087cf1a 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -19,7 +19,7 @@ hook of a running agent. The injection guard rail fails closed: a decision error Every tool agent takes `--model jev|clm|laya|laya-served|cua|llm|random|rule`, `--rethink on|off`, `--episodes N`, `--seed S`, `--max-steps` and `--timeout`, and writes a Harbor-shaped job folder under `evals/results//`. A browser agent -takes `--model jev|clm|laya|laya-served|cua|omnijev|llm`, `--goal` and the same `--rethink` flag. +takes `--model jev|clm|laya|laya-served|cua|omnijev|onejev|llm`, `--goal` and the same `--rethink` flag. The desktop agent adds `--rethink-attempts` and `--rethink-timeout` and stalls after 3 actions without observed progress. The browser front takes the same three names; `docs/browser-front.md` decision 19 describes its branch. A rail takes `--model jev|clm|laya|laya-served`, the models that answer `noul`. diff --git a/docs/architecture.md b/docs/architecture.md index f393b2a..09e67a9 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -12,7 +12,7 @@ score, seconds, steps and dollars. sequenceDiagram participant A as openJiuwen agent participant E as page or environment - participant S1 as Jev, Laya, Cua-S1 Nano or OmniJev (System 1) + participant S1 as Jev, Laya, Cua-S1 Nano, OmniJev or OneJev (System 1) participant C as chat model (System 2) A->>E: observe E-->>A: state + enumerated options @@ -35,7 +35,7 @@ the description, the rules text the model reads, a budget, and the front-specifi - Rail front (`RailSpec`, `s1a/rails.py`): `DecisionModelRail` asks one question at one DeepAgent callback hook, acts on it above a threshold, and is evaluated on a labelled set. Agent: `injection_guard`. -Shared: `s1a/decision_models/` (the decision-model interface, the Jev, Laya, Cua-S1 and OmniJev adapters, the baselines, the answer +Shared: `s1a/decision_models/` (the decision-model interface, the Jev, Laya, Cua-S1, OmniJev and OneJev adapters, the baselines, the answer validation; `docs/decision-models.md`), `s1a/decision_models/wire.py` (the HTTP transport and its two backends), `s1a/config.py` (env loading, the chat model, the browser launch flags), `s1a/tool/rethink.py` (a rail that blocks repeats and asks for a plan on a stall), `s1a/jobs.py` @@ -50,6 +50,7 @@ validation; `docs/decision-models.md`), `s1a/decision_models/wire.py` (the HTTP | `laya` | [Laya](https://huggingface.co/convaiinnovations/laya) (`convaiinnovations/laya`, 0.4B) | in process, `uv sync --extra laya`; no key | a 512 to 1024 token window | | `cua` | [Cua-S1 Nano](https://huggingface.co/cua-ai/cua-s1-nano-0.1) (`cua-ai/cua-s1-nano-0.1`, 855K) | in process, `uv sync --extra cua`; no key | a 256-byte context (header, goal, state, then rules), 96 bytes per option | | `omnijev` | [OmniJev](https://github.com/tinnel123666888/OmniJev) (Qwen3.5 VL, 0.8B by default, 2B or 4B) | in process, browser agents only, `uv sync --extra omnijev` and a clone named by `OMNIJEV_REPO`; no key; a GPU for useful speed | a viewport screenshot per tick, plus the text state (recent actions first, page text capped at 2,000 characters) | +| `onejev` | [OneJev](https://github.com/OmniJev/OneJev) (0.8B–27B, Apache-2.0) via `qev`/`flide` | served HTTP, browser agents only; `ONEJEV_URL`; no cloud key | a viewport screenshot per tick as `media` with `` in state | | `random`, `rule` | the tool front's two baselines | in process | the candidates | | `llm` | the chat model | for the comparison columns | the transcript | @@ -63,7 +64,7 @@ the first import of both, routes the harness logs to files under `runs/logs` bef MCP stdio protocol only. Every `run` prints one JSON object on stdout: a tool agent's series summary with its `job_dir`, a browser agent's -answer, a rail's evaluation. A browser agent takes `--model jev|laya|cua|omnijev|llm`; its policy switches are run-time flags: +answer, a rail's evaluation. A browser agent takes `--model jev|laya|cua|omnijev|onejev|llm`; its policy switches are run-time flags: `--batch on|off`, `--prefetch on|off`, `--goal-values on|off`. `s1a-mcp` serves the same agents to an MCP host over stdio, one Runner for the server's lifetime and one run at a time. `uv run python -m evals.table evals/results` aggregates every job folder per eval and model into one table. `scripts/showcase.sh` plays one visual episode per @@ -72,7 +73,7 @@ eval and model outside the matrix and `python -m evals.replay` renders a pair si Every tool agent, `desktop` included, takes `--model jev|laya|cua|llm|random|rule`, `--rethink on|off`, `--episodes N`, `--seed S`, `--max-steps`, `--timeout` and `--headed`, and writes a Harbor-shaped job folder under -`evals/results//`. Every browser agent takes `--model jev|laya|cua|omnijev|llm` and `--goal`. A rail takes +`evals/results//`. Every browser agent takes `--model jev|laya|cua|omnijev|onejev|llm` and `--goal`. A rail takes `--model jev|laya`, the two models that answer `noul`. `decide` and `probe` take `--model jev|laya|cua`. On a browser agent `laya` needs `LAYA_MAX_LEN` raised to the page's size; `cua` reads a 256-byte context (header, goal, state, then rules) and 96 bytes per option, a baseline on any page. Exit codes: 0 for a finished run, including one whose diff --git a/docs/capabilities.md b/docs/capabilities.md index 00c4ad4..f427af6 100644 --- a/docs/capabilities.md +++ b/docs/capabilities.md @@ -16,6 +16,7 @@ For issue trackers and serving support, see [System1-Omni model tracker #83](htt | LAYA (`--model laya`; default `convaiinnovations/laya`, English; `LAYA_SUBFOLDER=multilingual` selects multilingual) | Text | `choice`, `noul` | Tool, browser, rails, `decide`/`probe`; browser state/question shaping is implemented | In-process Python `laya` | Historical English and multilingual CPU/MPS runs are documented below. Other/typed/custom checkpoints and CUDA are unverified here. | | LAYA (`--model laya-served`; served checkpoint/revision is reported by the endpoint) | Text | `choice`, `noul` | Tool, browser, rails, `decide`/`probe`, MCP | Agents HTTP client can connect directly to a Laya worker or through System1-Omni `omni-jev` at `/v1/systemone` | Historical direct-worker and Omni-front MPS ticket-router measurements are recorded below (90 decisions/configuration, 63 correct each); historical CPU served request replay is separate and is not task-success evidence. Other hardware/checkpoints unverified. | | OmniJev (`--model omnijev`; default `tinnel123/OmniJev-0.8B@v1.1` on `Qwen/Qwen3.5-0.8B`; configurable 2B/4B) | Screenshot plus text state/context | `choice`, `noul` | Browser front via `s1a run` and MCP `run_agent`; direct `build_model()` factory path. Not registered in tool/rails, CLI `decide`/`probe`, or MCP `decide` | In-process OmniJev clone via `MSO1.system_one`; no System1-Omni served vision path | Historical A100 80 GB 4B runs, Windows 11 CPU 0.8B runs, and A40 46 GB 4B runs are detailed below. They do not establish broad task success; current-main reproduction unverified. | +| OneJev (`--model onejev`; default request id `OmniJev/OneJev1.6-4B`; set `ONEJEV_MODEL`) | Screenshot(s) plus text state/context via TypeSafe `media` / `` | `choice`, `noul` | Browser front via `s1a run` and MCP `run_agent`; direct `build_model()` factory path. Not registered in tool/rails, CLI `decide`/`probe`, or MCP `decide` | HTTP client to `qev`/`flide` at `ONEJEV_URL` (`POST /v1/systemone` + `media`) | Contract tests use a fake HTTP server. Live GPU / GGUF hardware validation **not recorded in this checkout** (#88). | | Cua-S1 Nano (`cua-ai/cua-s1-nano-0.1/text`, `--model cua`, default variant) | Text | `choice` only | Tool/browser paths where registered; no screenshot input | In-process `cua_s1.nano.NanoScorer` | No real Nano inference artifact found in this checkout. **CPU/GPU/MPS unverified here.** | | Cua-S1 4B 0.2 text adapter (`Qwen/Qwen3.5-4B` + `cua-ai/cua-s1-4b-0.2/text`, `CUA_S1_VARIANT=4b`, `CUA_S1_MODALITY=text`) | Text | `choice` only | Tool decisions and text-only client paths; desktop option grounding is implemented | In-process Transformers/PEFT scorer | Contract tests use a fake scorer; real checkpoint/device validation **not present in this checkout**. | | Cua-S1 4B 0.2 multimodal adapter (`Qwen/Qwen3.5-4B` + `cua-ai/cua-s1-4b-0.2/multimodal`, variant `4b`) | One current screenshot plus text task/context | `choice` only; maximum 26 options in text and multimodal modes; multimodal requires exactly one screenshot | Screenshot-dependent desktop decision adapter; this does not itself complete the Omni capture/execute bridge | In-process Transformers/PEFT scorer with screenshot passed as a temporary file | Contract tests use a fake scorer; separate historical external evidence reports a bounded MPS fixture run (see below). Current-main reproduction and CUDA validation remain unverified. | diff --git a/docs/configuration.md b/docs/configuration.md index 37c2af9..e03db7c 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -53,6 +53,10 @@ Variables can be exported in your shell or placed in a `.env` file at the root o | `OMNIJEV_REVISION` | `omnijev` model | `v1.1` | Revision of a Hugging Face `OMNIJEV_CHECKPOINT`. | | `OMNIJEV_BASE` | `omnijev` model | `Qwen/Qwen3.5-0.8B` | Hugging Face ID or local directory of the Qwen3.5 backbone of the same size as the checkpoint. | | `OMNIJEV_PROMPT` | `omnijev` model | `full` | `full` sends the text state, goal and rules with each question; `short` sends the goal and the ask only, the screenshot carrying the page. | +| `ONEJEV_URL` | `onejev` model | *(unset, required)* | Base URL of a served OneJev (`qev`/`flide`), e.g. `http://127.0.0.1:8000`. Browser and `build_model` only; requires a screenshot in the observation. | +| `ONEJEV_MODEL` | `onejev` model | `OmniJev/OneJev1.6-4B` | Model id sent in the `/v1/systemone` body (checkpoint the server loaded, e.g. `OmniJev/OneJev1.6-0.8B`). | +| `ONEJEV_API_KEY` | `onejev` model | *(unset)* | Bearer token when the server expects one; loopback `qev`/`flide` usually needs none. | +| `ONEJEV_TIMEOUT_S` | `onejev` model | `30` | Deadline per decision in seconds, the one refused-connection / 502 / 504 retry included. A finite number above 0. Raise for large checkpoints or a cold load. | | `CUA_DRIVER_BIN` | `desktop` agent | `cua-driver` | Path to the `cua-driver` executable on Windows or macOS when not located on `PATH`. | | `CUA_DRIVER_PERMISSION_MODE` | `desktop` agent | `standard` | Permission mode passed to `cua-driver mcp` (`standard`, or `bounded` for restricted capability manifests). | | `HF_HOME` | Hugging Face runtime | `~/.cache/huggingface` | Cache directory where Laya and Cua-S1 checkpoints are downloaded on first run. | diff --git a/docs/decision-models.md b/docs/decision-models.md index 8cfb47b..0ca3c5b 100644 --- a/docs/decision-models.md +++ b/docs/decision-models.md @@ -12,7 +12,7 @@ TypeSafe "System One model". This repository uses the terms interchangeably. `-- `clm` (CLM's `clm-serve` over HTTP, see [clm.md](clm.md)), `laya` (in process, behind `uv sync --extra laya`), `laya-served` (Laya served over HTTP by system1-omni, see [served-laya.md](served-laya.md)), `cua` (Cua-S1 Nano in process, behind `uv sync --extra cua`), -`random` and `rule` (the tool front's baselines). +`onejev` (OneJev over HTTP via `qev`/`flide` with a screenshot), `random` and `rule` (the tool front's baselines). `build_model(model_name, seed=, rule=)` builds one from the environment; `llm` names the chat model, which `build_model` does not build. The registered paths, input modalities and question types are summarized in the [decision-model capability matrix](capabilities.md), separately from run evidence and hardware validation. @@ -65,6 +65,7 @@ shorthands; `warm()` and `close()` open and release the backend. | `laya-served` | `ServedLayaModel(client, model=, max_len=)` | `laya-served` | one `POST /v1/systemone` per request to `LAYA_SERVED_URL`, body built with `laya_question()` as for `laya`; one retry on a dropped connection, 502 or 504, and after `Retry-After` on 503, all within `LAYA_SERVED_TIMEOUT_S`; the same window check as `laya`; `model` is the served checkpoint and revision, and `raw` keeps `served_by`, `url`, `request_id` and `server_timing` (see [served-laya.md](served-laya.md)) | | `cua` | `CuaS1Model(scorer, collator, model=, context_bytes=, option_bytes=)` | `cua` | Cua-S1 Nano, one `score_elements` pass per request on a thread; choice questions only, text only, deterministic; the context is header, state and rules; the checkpoint reads its first 256 bytes, and the first overflowing request logs one warning; `from_env` reads `CUA_S1_*` (see [configuration.md](configuration.md)) | | `omnijev` | `OmniJevModel(agent, model=)` | `omnijev` | OmniJev (a Qwen3.5 vision-language decision model), one `MSO1.system_one` call per request on a thread; choice and noul questions over the observation's screenshot, deterministic; the text state, goal and rules go in front of each question (a browser state puts its recent actions first and caps the page text at 2,000 characters, so a long page cannot push the history out of the 6,000-character context), a browser element row becomes its label and value; a choice's probabilities, which OmniJev leaves `abstain` out of, are renormalised over the offered keys; an observation without an image is `MODEL_CALL_FAILED`; `from_env` reads `OMNIJEV_*` (see [configuration.md](configuration.md)); in process only: system1-omni has no OmniJev vision worker yet (its supported models are Laya, Open-Jev text and Cua-S1 screenshots), so there is no served `omnijev` path | +| `onejev` | `OneJevModel(client, model=)` | `onejev` | OneJev (OmniJev org, Apache-2.0) served by `qev`/`flide` over TypeSafe-compatible `POST /v1/systemone` plus a `media` field; choice and noul over the observation's screenshot(s), deterministic; state carries `` markers and each screenshot is a PNG/JPEG/WebP data URL in `media` (digest and size in `raw.sent_images`, never the bytes); an observation without an image is `MODEL_CALL_FAILED` (same policy as `omnijev`); one refused-connection / 502 / 504 retry within `ONEJEV_TIMEOUT_S`; `raw` keeps `served_by`, `url`, `request_id` and `server_timing`; `from_env` reads `ONEJEV_*` (see [configuration.md](configuration.md)); browser and `build_model` only — not tool/rails/`decide`. Local tests use a stub server; live GPU validation is **not recorded** in this checkout | | `random` | `RandomModel(seed)` | `random` | uniform over the offered keys (contract confidence = `1/|options|`), one seeded stream per episode; choice questions only | | `rule` | `RuleModel(name, rule)` | the rule's name | one-hot, confidence 1; a key outside the menu raises `RuntimeError`, a bug in the rule | @@ -80,6 +81,26 @@ and API keys are referenced in [configuration.md](configuration.md). Latency and price: `benchmarks.md`. Laya (Convai Innovations, open weights, 0.4B parameters) and Cua-S1 Nano (Cua, 855K parameters) answer the same `choice` question in process. + +### Served OneJev + +`onejev` is registered in the browser front and `build_model()`. Point `ONEJEV_URL` at a local +[`qev`](https://github.com/OmniJev/OneJev) or [`flide`](https://github.com/OmniJev/Flide) server that speaks +`/v1/systemone` with `media`. Example smoke (not run in CI; needs a GPU or GGUF build): + +```bash +# terminal 1 — install from the OneJev repo (pip install qev is a different package) +qev serve --model OmniJev/OneJev1.6-4B --port 8000 +# or: flide serve --model OmniJev/OneJev1.6-4B --port 8000 + +# terminal 2 +ONEJEV_URL=http://127.0.0.1:8000 ONEJEV_TIMEOUT_S=60 \ + uv run s1a run flights --model onejev --rethink off +``` + +This is a setup recipe only. Capability registration and stub-server contract tests do **not** establish task +success rates or hardware validation; see [capabilities.md](capabilities.md). + ### Served JEV-VL `jev-vl-served` is registered in the browser front and `build_model()`. It sends the observation's text state, diff --git a/s1a/browser/browse.py b/s1a/browser/browse.py index 43f05a6..81adfed 100644 --- a/s1a/browser/browse.py +++ b/s1a/browser/browse.py @@ -48,6 +48,7 @@ "laya-served", "cua", "omnijev", + "onejev", "llm", ) # a decision model over HTTP or in process, or the chat model RUNS_DIR = HOME / "runs" / "browser" @@ -253,7 +254,7 @@ async def browse( workspace = str(logs_dir / "workspace") # the harness scaffolds SOUL.md, memory/ and friends here, not in the cwd instance = BrowserInstanceConfig(launch_args=browser_launch_args(headless)) match model_name: - case "jev" | "jev-vl-served" | "clm" | "laya" | "laya-served" | "cua" | "omnijev": + case "jev" | "jev-vl-served" | "clm" | "laya" | "laya-served" | "cua" | "omnijev" | "onejev": if decision_model is None: raise RuntimeError(f"--model {model_name} needs a decision model") slot_model = BrowserDecisionModel( @@ -320,7 +321,7 @@ def parser(spec: BrowserAgentSpec) -> argparse.ArgumentParser: "--model", choices=BROWSER_MODEL_NAMES, required=True, - help="who decides each browser step: jev, jev-vl-served, clm or laya-served (over HTTP; jev-vl-served requires a screenshot), laya, cua or omnijev (in process; omnijev reads a screenshot), or llm (the chat model in MODEL_NAME)", + help="who decides each browser step: jev, jev-vl-served, clm, laya-served or onejev (over HTTP; jev-vl-served and onejev require a screenshot), laya, cua or omnijev (in process; omnijev reads a screenshot), or llm (the chat model in MODEL_NAME)", ) build.add_argument( "--goal", default=spec.goal, required=spec.goal is None, help="the task; the spec's goal when it has one" diff --git a/s1a/decision_models/__init__.py b/s1a/decision_models/__init__.py index 7af66ad..b3bdab2 100644 --- a/s1a/decision_models/__init__.py +++ b/s1a/decision_models/__init__.py @@ -15,6 +15,7 @@ from s1a.decision_models.jev import JevModel, jev_question from s1a.decision_models.laya import LayaModel from s1a.decision_models.omnijev import OmniJevModel +from s1a.decision_models.onejev import OneJevModel from s1a.decision_models.types import ( Answer, Choice, @@ -56,6 +57,7 @@ "Noul", "NoulQuestion", "OmniJevModel", + "OneJevModel", "Observation", "Question", "RandomModel", diff --git a/s1a/decision_models/factory.py b/s1a/decision_models/factory.py index cca659a..ea1cf33 100644 --- a/s1a/decision_models/factory.py +++ b/s1a/decision_models/factory.py @@ -14,6 +14,7 @@ from s1a.decision_models.jev_vl_served import ServedJevVLModel from s1a.decision_models.laya import LayaModel from s1a.decision_models.omnijev import OmniJevModel +from s1a.decision_models.onejev import OneJevModel from s1a.decision_models.served import ServedLayaModel DECISION_MODEL_NAMES = ( @@ -24,6 +25,7 @@ "laya-served", "cua", "omnijev", + "onejev", "random", "rule", ) # the names that build a decision model; ``llm`` is not one @@ -51,6 +53,8 @@ def build_model(model_name: str, *, seed: int = 0, rule: tuple[str, Rule] | None return CuaS1Model.from_env() # the cua_s1 import happens inside case "omnijev": return OmniJevModel.from_env() # the OmniJev import happens inside; it decides over a screenshot + case "onejev": + return OneJevModel.from_env() # OneJev/qev over HTTP with media; screenshot required case "random": return RandomModel(seed) case "rule": diff --git a/s1a/decision_models/onejev.py b/s1a/decision_models/onejev.py new file mode 100644 index 0000000..53d6edb --- /dev/null +++ b/s1a/decision_models/onejev.py @@ -0,0 +1,305 @@ +# coding: utf-8 +"""OneJev served over HTTP by ``qev`` / ``flide`` (TypeSafe-compatible ``/v1/systemone`` + ``media``). + +OneJev (https://github.com/OmniJev/OneJev, Apache-2.0) answers typed Choice / Noul / Score questions over +screenshots in one forward pass. The server speaks System One plus a ``media`` field; state references images +with ````. This adapter is the served counterpart of in-process ``omnijev``: it posts observation + +questions over HTTP and never loads weights. Start a server with ``qev serve --model OmniJev/OneJev1.6-4B`` +(or ``flide serve``) and set ``ONEJEV_URL``. +""" + +from __future__ import annotations + +import asyncio +import base64 +import hashlib +import logging +import math +import os +import time +import uuid +from collections.abc import Callable +from datetime import datetime, timezone +import httpx + +from openjiuwen.core.common.exception.codes import StatusCode +from openjiuwen.core.common.exception.errors import BaseError, build_error + +from s1a.decision_models.base import DecisionModel +from s1a.decision_models.omnijev import omnijev_answer, omnijev_question +from s1a.decision_models.served import parse_server_timing +from s1a.decision_models.types import Image, Json, Observation, Question, Reply, Usage + +logger = logging.getLogger(__name__) + +ONEJEV_TIMEOUT_S = 30.0 # one multimodal forward pass; raise for 27B or a cold load +ONEJEV_HEALTH_TIMEOUT_S = 2.0 +DEFAULT_ONEJEV_MODEL = "OmniJev/OneJev1.6-4B" +_RETRIED_TRANSPORT = (httpx.ConnectError, httpx.ConnectTimeout, httpx.RemoteProtocolError, httpx.ReadError) +_RETRIED_STATUSES = frozenset({502, 504}) +_NOT_UP = ( + "qev/flide listens only once the checkpoint is loaded; start it per " + "https://github.com/OmniJev/OneJev (qev serve --model OmniJev/OneJev1.6-4B) and point ONEJEV_URL at it" +) +_MEDIA_TYPES = {"image/png": "png", "image/jpeg": "jpeg", "image/webp": "webp"} + + +def onejev_media(image: Image, index: int) -> tuple[Json, Json]: + """One ``media`` entry (1-based ````) and the digest the step record keeps instead of the bytes.""" + if image.media_type not in _MEDIA_TYPES: + raise build_error( + StatusCode.MODEL_CALL_FAILED, + error_msg=f"onejev cannot send media type {image.media_type!r}; use PNG, JPEG or WebP", + ) + if not image.data: + raise build_error(StatusCode.MODEL_CALL_FAILED, error_msg="onejev cannot send an empty screenshot") + data_uri = f"data:{image.media_type};base64,{base64.b64encode(image.data).decode('ascii')}" + record: Json = { + "index": index, + "sha256": hashlib.sha256(image.data).hexdigest(), + "media_type": image.media_type, + "bytes": len(image.data), + } + return {"type": "image", "data": data_uri}, record + + +def onejev_state(observation: Observation) -> Json: + """Observation state with ```` markers for every screenshot (OneJev's documented convention).""" + markers = {f"image_{i}": f"" for i in range(1, len(observation.images) + 1)} + if len(observation.images) == 1: + markers = {"screen": ""} + if isinstance(observation.state, dict): + return {**observation.state, **markers} + return {"text": observation.state, **markers} + + +def onejev_question(question: Question) -> Json: + """Choice / noul in OneJev's TypeSafe shape: string instructions, flattened option labels. + + The observation's text state travels in ``state`` with the image markers; instructions carry the goal, rules + and ask (OmniJev's short prompt), so the screenshot is not duplicated as prose. + """ + return omnijev_question(question, context="", prompt="short") + + +class OneJevClient: + """HTTP to one ``qev`` / ``flide`` ``/v1/systemone`` server: one deadline per decision, one transport retry.""" + + def __init__( + self, + *, + url: str, + api_key: str | None = None, + timeout_s: float = ONEJEV_TIMEOUT_S, + transport: httpx.AsyncBaseTransport | None = None, + clock: Callable[[], float] = time.monotonic, + ) -> None: + if not math.isfinite(timeout_s) or timeout_s <= 0: + raise build_error( + StatusCode.MODEL_SERVICE_CONFIG_ERROR, + error_msg=f"timeout_s must be a finite number above 0, not {timeout_s!r}", + ) + self.url = url.rstrip("/") + self._timeout_s = timeout_s + self._clock = clock + headers = {"Authorization": f"Bearer {api_key}"} if api_key else {} + self._client = httpx.AsyncClient(timeout=timeout_s, headers=headers, transport=transport) + + async def warm(self) -> Json: + """``GET /health`` once: fail early when nothing is listening.""" + try: + response = await asyncio.wait_for( + self._client.get(f"{self.url}/health", timeout=ONEJEV_HEALTH_TIMEOUT_S), ONEJEV_HEALTH_TIMEOUT_S + ) + except (httpx.ConnectError, httpx.ConnectTimeout) as exc: + raise build_error( + StatusCode.MODEL_CALL_FAILED, cause=exc, error_msg=f"no served OneJev at {self.url}: {_NOT_UP}" + ) from exc + except (TimeoutError, httpx.HTTPError) as exc: + raise build_error( + StatusCode.MODEL_CALL_FAILED, cause=exc, error_msg=f"served OneJev at {self.url} did not answer: {exc}" + ) from exc + if response.is_error: + raise build_error( + StatusCode.MODEL_CALL_FAILED, + error_msg=f"served OneJev at {self.url} returned HTTP {response.status_code}", + ) + try: + body = response.json() + except ValueError as exc: + raise build_error( + StatusCode.MODEL_CALL_FAILED, cause=exc, error_msg="served OneJev returned a malformed body" + ) from exc + return body if isinstance(body, dict) else {} + + async def decide(self, body: Json, request_id: str) -> tuple[Json, int, dict[str, str]]: + """One decision: the payload, its round trip in ms and the response headers.""" + headers = {"X-Request-Id": request_id} + deadline = self._clock() + self._timeout_s + retried = False + while True: + remaining = deadline - self._clock() + if remaining <= 0: + raise self._no_answer(None) + started = time.perf_counter() + post = self._client.post(f"{self.url}/v1/systemone", json=body, headers=headers, timeout=remaining) + try: + response = await asyncio.wait_for(post, remaining) + except TimeoutError as exc: + raise self._no_answer(exc) from exc + except _RETRIED_TRANSPORT as exc: + if not retried: + retried = True + continue + if isinstance(exc, httpx.ConnectError): + raise build_error( + StatusCode.MODEL_CALL_FAILED, + cause=exc, + error_msg=f"no served OneJev at {self.url}: {_NOT_UP}", + ) from exc + raise build_error( + StatusCode.MODEL_CALL_FAILED, + cause=exc, + error_msg=f"served OneJev unreachable at {self.url}: {exc}", + ) from exc + except httpx.HTTPError as exc: + raise build_error( + StatusCode.MODEL_CALL_FAILED, + cause=exc, + error_msg=f"served OneJev call failed at {self.url}: {exc}", + ) from exc + ms = round((time.perf_counter() - started) * 1000) + if response.status_code in _RETRIED_STATUSES and not retried: + retried = True + continue + if response.is_error: + raise build_error( + StatusCode.MODEL_CALL_FAILED, + error_msg=f"served OneJev failed (HTTP {response.status_code}): {response.text[:200].strip()}", + ) + try: + payload = response.json() + except ValueError as exc: + raise build_error( + StatusCode.MODEL_CALL_FAILED, cause=exc, error_msg="served OneJev returned a malformed body" + ) from exc + if not isinstance(payload, dict) or not isinstance(payload.get("answers"), dict): + raise build_error(StatusCode.MODEL_CALL_FAILED, error_msg="served OneJev returned no answers object") + return payload, ms, dict(response.headers) + + def _no_answer(self, cause: Exception | None) -> Exception: + return build_error( + StatusCode.MODEL_CALL_FAILED, + cause=cause, + error_msg=f"no answer from served OneJev at {self.url} within {self._timeout_s:g} s; {_NOT_UP}", + ) + + async def close(self) -> None: + await self._client.aclose() + + +class OneJevModel(DecisionModel): + """OneJev behind ``/v1/systemone`` + ``media``: choice and noul over the observation's screenshot(s).""" + + name = "onejev" + supports_images = True + deterministic = True + bills_input_tokens = False # the client's own server, not Jev's pricing + + def __init__(self, client: OneJevClient, *, model: str = DEFAULT_ONEJEV_MODEL) -> None: + self._client = client + self._model = model + self._served: Json = {} + self._served_at: str | None = None + self._warmed = False + + @property + def model(self) -> str: + return self._model + + async def warm(self) -> None: + self._served = await self._client.warm() + self._served_at = datetime.now(timezone.utc).isoformat(timespec="seconds") + self._warmed = True + + def _served_by(self) -> Json: + return { + "url": self._client.url, + "checkpoint": self._served.get("model") or self._served.get("checkpoint") or self._model, + "device": self._served.get("device"), + "source": "health", + "read_at": self._served_at, + } + + async def _warm_once(self) -> None: + if self._warmed: + return + try: + await self.warm() + except BaseError as exc: + logger.warning("[onejev] could not read /health at %s: %s", self._client.url, exc) + + async def _decide(self, observation: Observation, questions: dict[str, Question]) -> Reply: + if not observation.images: + raise build_error( + StatusCode.MODEL_CALL_FAILED, + error_msg="onejev decides over a screenshot; this observation carries no image", + ) + media: list[Json] = [] + records: list[Json] = [] + for index, image in enumerate(observation.images, start=1): + entry, record = onejev_media(image, index) + media.append(entry) + records.append(record) + body: Json = { + "model": self._model, + "state": onejev_state(observation), + "media": media, + "questions": {name: onejev_question(question) for name, question in questions.items()}, + } + await self._warm_once() + request_id = uuid.uuid4().hex + payload, ms, headers = await self._client.decide(body, request_id) + answers = payload["answers"] + mapped = {name: omnijev_answer(answers[name]) for name in questions if name in answers} + lower = {key.lower(): value for key, value in headers.items()} + return Reply( + answers=mapped, + latency_ms=ms, + usage=Usage.from_payload(payload.get("usage")), + model=str(payload.get("model") or self._model), + raw={ + **payload, + "url": self._client.url, + "request_id": request_id, + "served_by": self._served_by(), + "server_timing": parse_server_timing(lower.get("server-timing")), + "sent_images": records, + }, + ) + + async def close(self) -> None: + await self._client.close() + + @classmethod + def from_env(cls) -> "OneJevModel": + """``ONEJEV_URL`` (required), ``ONEJEV_MODEL``, ``ONEJEV_API_KEY``, ``ONEJEV_TIMEOUT_S``. No cloud key.""" + url = os.getenv("ONEJEV_URL") + if not url: + raise build_error( + StatusCode.MODEL_SERVICE_CONFIG_ERROR, + error_msg="--model onejev needs ONEJEV_URL, e.g. http://127.0.0.1:8000 (qev/flide serve)", + ) + try: + timeout_s = float(os.getenv("ONEJEV_TIMEOUT_S") or ONEJEV_TIMEOUT_S) + except ValueError as exc: + raise build_error( + StatusCode.MODEL_SERVICE_CONFIG_ERROR, cause=exc, error_msg="ONEJEV_TIMEOUT_S must be a number" + ) from exc + if not math.isfinite(timeout_s) or timeout_s <= 0: + raise build_error( + StatusCode.MODEL_SERVICE_CONFIG_ERROR, + error_msg="ONEJEV_TIMEOUT_S must be a finite number above 0", + ) + client = OneJevClient(url=url, api_key=os.getenv("ONEJEV_API_KEY") or None, timeout_s=timeout_s) + return cls(client, model=os.getenv("ONEJEV_MODEL") or DEFAULT_ONEJEV_MODEL) diff --git a/tests/test_browse.py b/tests/test_browse.py index a6850ef..37365df 100644 --- a/tests/test_browse.py +++ b/tests/test_browse.py @@ -60,7 +60,8 @@ def test_a_timeout_or_max_steps_at_or_below_zero_is_a_usage_error(self) -> None: def test_the_model_flag_takes_a_decision_model_or_the_chat_model(self) -> None: self.assertEqual( - browse.BROWSER_MODEL_NAMES, ("jev", "jev-vl-served", "clm", "laya", "laya-served", "cua", "omnijev", "llm") + browse.BROWSER_MODEL_NAMES, + ("jev", "jev-vl-served", "clm", "laya", "laya-served", "cua", "omnijev", "onejev", "llm"), ) for model_name in browse.BROWSER_MODEL_NAMES: self.assertEqual(browse.parser(SPEC).parse_args(["--model", model_name, "--goal", "x"]).model, model_name) diff --git a/tests/test_decision_models_factory.py b/tests/test_decision_models_factory.py index 9339da1..cc51ab6 100644 --- a/tests/test_decision_models_factory.py +++ b/tests/test_decision_models_factory.py @@ -46,7 +46,7 @@ def test_every_name_builds_its_class(self) -> None: self.assertEqual(rule.name, "always-inc") self.assertEqual( DECISION_MODEL_NAMES, - ("jev", "jev-vl-served", "clm", "laya", "laya-served", "cua", "omnijev", "random", "rule"), + ("jev", "jev-vl-served", "clm", "laya", "laya-served", "cua", "omnijev", "onejev", "random", "rule"), ) def test_the_errors(self) -> None: diff --git a/tests/test_decision_models_onejev.py b/tests/test_decision_models_onejev.py new file mode 100644 index 0000000..86b4d18 --- /dev/null +++ b/tests/test_decision_models_onejev.py @@ -0,0 +1,304 @@ +# coding: utf-8 +"""``onejev`` served client: contract tests against a fake HTTP server (no GPU / weights).""" + +from __future__ import annotations + +import asyncio +import base64 +import json +import os +from typing import Any +from unittest import IsolatedAsyncioTestCase, TestCase +from unittest.mock import patch + +import httpx +from openjiuwen.core.common.exception.codes import StatusCode +from openjiuwen.core.common.exception.errors import BaseError + +import decision_model_contract as contract +from s1a.decision_models import ChoiceQuestion, DecisionModel, Image, Observation, build_model +from s1a.decision_models.onejev import ( + DEFAULT_ONEJEV_MODEL, + OneJevClient, + OneJevModel, + onejev_media, + onejev_question, + onejev_state, +) + +# Minimal valid-looking PNG header bytes for media encoding tests (not decoded by the stub). +SCREEN = Image(b"\x89PNG\r\n\x1a\n" + b"\x00" * 24, media_type="image/png") +PICTURED = Observation({"task": "Pay the invoice"}, images=(SCREEN,)) +URL = "http://onejev.test" + + +def _choice_answer(question: dict[str, Any]) -> dict[str, Any]: + keys = list(question["criteria"]) + rest = 0.2 / max(1, len(keys) - 1) + probabilities = {key: (0.8 if i == 0 else rest) for i, key in enumerate(keys)} if keys else {} + return {"choice": keys[0], "probabilities": probabilities, "confidence": 0.8} + + +def _noul_answer(_question: dict[str, Any]) -> dict[str, Any]: + return {"noul": 0.65, "confidence": 0.65} + + +class Server: + """Scripted ``/v1/systemone`` + ``/health`` stand-in for qev/flide.""" + + def __init__(self, *, health: Any = None, script: list[Any] | None = None) -> None: + self.health = health if health is not None else {"model": DEFAULT_ONEJEV_MODEL, "device": "cuda"} + self.script = list(script or []) + self.decisions: list[httpx.Request] = [] + self.health_calls = 0 + + def answer(self, request: httpx.Request) -> httpx.Response: + body = json.loads(request.content) + answers = {} + for name, question in body["questions"].items(): + answers[name] = _noul_answer(question) if question["type"] == "noul" else _choice_answer(question) + return httpx.Response( + 200, + json={ + "model": body.get("model") or DEFAULT_ONEJEV_MODEL, + "answers": answers, + "usage": {"input_tokens": 12, "output_tokens": 0}, + }, + ) + + def handler(self, request: httpx.Request) -> httpx.Response: + if request.url.path == "/health": + self.health_calls += 1 + if isinstance(self.health, Exception): + raise self.health + if isinstance(self.health, httpx.Response): + return self.health + return httpx.Response(200, json=self.health) + self.decisions.append(request) + if self.script: + step = self.script.pop(0) + if isinstance(step, Exception): + raise step + if callable(step): + return step(request) + return step + return self.answer(request) + + +def make_model(server: Server, *, timeout_s: float = 5.0, model: str = DEFAULT_ONEJEV_MODEL) -> OneJevModel: + client = OneJevClient(url=URL + "/", timeout_s=timeout_s, transport=httpx.MockTransport(server.handler)) + return OneJevModel(client, model=model) + + +class TestOneJevContract(contract.DecisionModelContract, IsolatedAsyncioTestCase): + def setUp(self) -> None: + patcher = patch.object(contract, "OBSERVATION", PICTURED) + patcher.start() + self.addCleanup(patcher.stop) + + def make(self) -> DecisionModel: + return make_model(Server()) + + def make_scripted(self, answers: list[dict[str, Any]]) -> DecisionModel: + script = [ + httpx.Response(200, json={"model": DEFAULT_ONEJEV_MODEL, "answers": a, "usage": {"input_tokens": 8}}) + for a in answers + ] + return make_model(Server(script=script)) + + +class TestRequestBody(IsolatedAsyncioTestCase): + async def test_posts_media_and_image_marker(self) -> None: + server = Server() + await make_model(server).decide_many(PICTURED, {"pick": contract.PICK, "check": contract.CHECK}) + request = server.decisions[0] + self.assertEqual(request.url, httpx.URL(URL + "/v1/systemone")) + body = json.loads(request.content) + self.assertEqual(body["model"], DEFAULT_ONEJEV_MODEL) + self.assertEqual(body["state"]["screen"], "") + self.assertEqual(body["state"]["task"], "Pay the invoice") + self.assertEqual(len(body["media"]), 1) + self.assertEqual(body["media"][0]["type"], "image") + self.assertTrue(body["media"][0]["data"].startswith("data:image/png;base64,")) + decoded = base64.b64decode(body["media"][0]["data"].split(",", 1)[1]) + self.assertEqual(decoded, SCREEN.data) + self.assertEqual(body["questions"]["pick"]["type"], "choice") + self.assertEqual(body["questions"]["check"]["type"], "noul") + self.assertIsInstance(body["questions"]["pick"]["instructions"], str) + self.assertIn("X-Request-Id", request.headers) + + async def test_provenance_fields_and_no_image_bytes_in_raw(self) -> None: + decision = await make_model(Server()).decide_many(PICTURED, {"pick": contract.PICK}) + self.assertEqual(decision.raw["url"], URL) + self.assertIsInstance(decision.raw["request_id"], str) + self.assertEqual(decision.raw["served_by"]["checkpoint"], DEFAULT_ONEJEV_MODEL) + self.assertEqual(decision.raw["sent_images"][0]["bytes"], len(SCREEN.data)) + dumped = json.dumps(decision.raw) + self.assertNotIn(base64.b64encode(SCREEN.data).decode("ascii"), dumped) + + async def test_browser_element_row_becomes_label(self) -> None: + server = Server() + question = ChoiceQuestion( + {"12": {"element": "[12] Where from?", "current_value": "Zurich", "role": "combobox"}}, + goal="book a flight", + ) + await make_model(server).decide_many(PICTURED, {"click_target": question}) + asked = json.loads(server.decisions[0].content)["questions"]["click_target"] + self.assertEqual(asked["criteria"], {"12": "Where from? = Zurich"}) + self.assertIn("book a flight", asked["instructions"]) + + +class TestFailures(IsolatedAsyncioTestCase): + async def test_missing_image_is_a_call_failure(self) -> None: + with self.assertRaises(BaseError) as caught: + await make_model(Server()).decide_many(Observation({"page": "x"}), {"pick": contract.PICK}) + self.assertEqual(caught.exception.status, StatusCode.MODEL_CALL_FAILED) + self.assertIn("no image", str(caught.exception).lower()) + + async def test_transport_error_is_a_call_failure(self) -> None: + server = Server(script=[httpx.ConnectError("refused"), httpx.ConnectError("refused")]) + with self.assertRaises(BaseError) as caught: + await make_model(server).decide_many(PICTURED, {"pick": contract.PICK}) + self.assertEqual(caught.exception.status, StatusCode.MODEL_CALL_FAILED) + self.assertIn("onejev", str(caught.exception).lower()) + + async def test_malformed_answers_object_is_a_call_failure(self) -> None: + server = Server(script=[httpx.Response(200, json={"model": "x", "answers": "nope"})]) + with self.assertRaises(BaseError) as caught: + await make_model(server).decide_many(PICTURED, {"pick": contract.PICK}) + self.assertEqual(caught.exception.status, StatusCode.MODEL_CALL_FAILED) + self.assertIn("answers", str(caught.exception).lower()) + + async def test_http_error_is_a_call_failure(self) -> None: + server = Server(script=[httpx.Response(500, text="boom")]) + with self.assertRaises(BaseError) as caught: + await make_model(server).decide_many(PICTURED, {"pick": contract.PICK}) + self.assertEqual(caught.exception.status, StatusCode.MODEL_CALL_FAILED) + self.assertIn("500", str(caught.exception)) + + +class TestDecisionDeadline(IsolatedAsyncioTestCase): + async def test_retry_uses_remaining_budget_for_httpx_and_wait_for(self) -> None: + for failure in (httpx.ConnectError("refused"), httpx.Response(502), httpx.Response(504)): + with self.subTest(failure=failure): + now = 0.0 + requests = [] + + def handler(request: httpx.Request) -> httpx.Response: + nonlocal now + requests.append(request) + if len(requests) == 1: + now += 1.25 + if isinstance(failure, Exception): + raise failure + return failure + return httpx.Response(200, json={"answers": {}}) + + client = OneJevClient(url=URL, timeout_s=2.0, transport=httpx.MockTransport(handler), clock=lambda: now) + self.addAsyncCleanup(client.close) + with patch("s1a.decision_models.onejev.asyncio.wait_for", wraps=asyncio.wait_for) as wait_for: + await client.decide({"questions": {}}, "same-request") + self.assertEqual([call.args[1] for call in wait_for.call_args_list], [2.0, 0.75]) + for request, timeout in zip(requests, (2.0, 0.75), strict=True): + self.assertEqual(set(request.extensions["timeout"].values()), {timeout}) + self.assertEqual(request.headers["X-Request-Id"], "same-request") + + async def test_exhausted_budget_prevents_retry(self) -> None: + for failure in (httpx.ConnectError("refused"), httpx.Response(502), httpx.Response(504)): + with self.subTest(failure=failure): + now = 0.0 + requests = [] + + def handler(request: httpx.Request) -> httpx.Response: + nonlocal now + requests.append(request) + now += 2.0 + if isinstance(failure, Exception): + raise failure + return failure + + client = OneJevClient(url=URL, timeout_s=2.0, transport=httpx.MockTransport(handler), clock=lambda: now) + self.addAsyncCleanup(client.close) + with self.assertRaises(BaseError) as caught: + await client.decide({"questions": {}}, "exhausted") + self.assertEqual(caught.exception.status, StatusCode.MODEL_CALL_FAILED) + self.assertIn("within 2 s", str(caught.exception)) + self.assertEqual(len(requests), 1) + + async def test_stalled_retry_is_cancelled_at_remaining_deadline(self) -> None: + now = 0.0 + attempts = 0 + stopped = asyncio.Event() + + async def handler(request: httpx.Request) -> httpx.Response: + nonlocal now, attempts + attempts += 1 + if attempts == 1: + now = 0.99 + raise httpx.ConnectError("refused", request=request) + try: + await asyncio.Event().wait() + finally: + stopped.set() + raise AssertionError("stalled retry was not cancelled") + + client = OneJevClient(url=URL, timeout_s=1.0, transport=httpx.MockTransport(handler), clock=lambda: now) + self.addAsyncCleanup(client.close) + with patch("s1a.decision_models.onejev.asyncio.wait_for", wraps=asyncio.wait_for) as wait_for: + with self.assertRaises(BaseError) as caught: + await client.decide({"questions": {}}, "stalled") + self.assertEqual(caught.exception.status, StatusCode.MODEL_CALL_FAILED) + self.assertIsInstance(caught.exception.__cause__, TimeoutError) + self.assertAlmostEqual(wait_for.call_args_list[1].args[1], 0.01) + self.assertEqual(attempts, 2) + self.assertTrue(stopped.is_set()) + + +class TestMappingHelpers(TestCase): + def test_state_wraps_text_and_marks_one_image(self) -> None: + self.assertEqual( + onejev_state(Observation("plain", images=(SCREEN,))), + {"text": "plain", "screen": ""}, + ) + + def test_state_marks_multiple_images(self) -> None: + second = Image(b"\x89PNG second", media_type="image/png") + state = onejev_state(Observation({"goal": "g"}, images=(SCREEN, second))) + self.assertEqual(state["image_1"], "") + self.assertEqual(state["image_2"], "") + + def test_media_rejects_empty_and_unknown_types(self) -> None: + with self.assertRaises(BaseError): + onejev_media(Image(b"", media_type="image/png"), 1) + with self.assertRaises(BaseError): + onejev_media(Image(b"x", media_type="image/gif"), 1) + + def test_question_uses_string_instructions(self) -> None: + asked = onejev_question(ChoiceQuestion({"a": "A"}, goal="g", rules="r")) + self.assertEqual(asked["type"], "choice") + self.assertIsInstance(asked["instructions"], str) + self.assertIn("g", asked["instructions"]) + + +class TestFromEnv(TestCase): + def test_missing_url_is_a_config_error(self) -> None: + with patch.dict(os.environ, {"ONEJEV_URL": ""}, clear=False): + os.environ.pop("ONEJEV_URL", None) + with self.assertRaises(BaseError) as caught: + OneJevModel.from_env() + self.assertEqual(caught.exception.status, StatusCode.MODEL_SERVICE_CONFIG_ERROR) + self.assertIn("ONEJEV_URL", str(caught.exception)) + + def test_build_model_onejev_from_env(self) -> None: + with patch.dict(os.environ, {"ONEJEV_URL": "http://127.0.0.1:8000", "ONEJEV_MODEL": "OmniJev/OneJev-0.8B"}): + model = build_model("onejev") + self.assertIsInstance(model, OneJevModel) + self.assertEqual(model.name, "onejev") + self.assertEqual(model.model, "OmniJev/OneJev-0.8B") + self.assertTrue(model.supports_images) + + def test_bad_timeout_is_a_config_error(self) -> None: + with patch.dict(os.environ, {"ONEJEV_URL": "http://127.0.0.1:8000", "ONEJEV_TIMEOUT_S": "0"}): + with self.assertRaises(BaseError) as caught: + OneJevModel.from_env() + self.assertEqual(caught.exception.status, StatusCode.MODEL_SERVICE_CONFIG_ERROR) diff --git a/tests/test_onejev_registration.py b/tests/test_onejev_registration.py new file mode 100644 index 0000000..cc195c7 --- /dev/null +++ b/tests/test_onejev_registration.py @@ -0,0 +1,86 @@ +# coding: utf-8 +"""``onejev`` is registered for image-bearing browser decisions and the model factory.""" + +from __future__ import annotations + +import json +import os +import tempfile +from pathlib import Path +from unittest import IsolatedAsyncioTestCase, TestCase, mock + +from s1a.browser import browse +from s1a.browser.decision_model import BrowserPolicy +from s1a.decision_models import DECISION_MODEL_NAMES, build_model +from s1a.decision_models.fakes import ScriptedModel +from s1a.spec import BrowserAgentSpec, Budget + + +def spec() -> BrowserAgentSpec: + return BrowserAgentSpec( + name="fixture", + description="A browser fixture", + rules="Choose an offered action", + language="en", + budget=Budget(max_steps=2, timeout_s=10, stall_after=0), + goal="Open the fixture", + ) + + +class TestOneJevRegistration(TestCase): + def test_factory_and_browser_offer_the_backend(self) -> None: + self.assertIn("onejev", DECISION_MODEL_NAMES) + args = browse.parser(spec()).parse_args(["--model", "onejev"]) + self.assertEqual(args.model, "onejev") + + +class TestOneJevEntrypoints(IsolatedAsyncioTestCase): + async def test_factory_builds_the_served_vision_model_from_the_environment(self) -> None: + with mock.patch.dict(os.environ, {"ONEJEV_URL": "http://127.0.0.1:8000", "ONEJEV_TIMEOUT_S": "5"}): + model = build_model("onejev") + try: + self.assertEqual(type(model).__name__, "OneJevModel") + self.assertEqual(model.name, "onejev") + self.assertTrue(model.supports_images) + self.assertTrue(model.deterministic) + finally: + await model.close() + + async def test_browser_routes_the_backend_to_the_decision_model_slot(self) -> None: + decision_model = ScriptedModel() + decision_model.name = "onejev" + slot = mock.MagicMock() + slot.report.return_value = {"jev_input_tokens": 0, "decisions": 1} + slot.ticks = [] + final = json.dumps({"status": "DONE", "answer": "Fixture answer"}) + with ( + mock.patch.object(browse, "CountingModel") as counting, + mock.patch.object(browse, "BrowserDecisionModel", return_value=slot) as slot_class, + mock.patch.object(browse, "create_browser_agent") as create_agent, + mock.patch.object( + browse, + "run_task", + new=mock.AsyncMock(return_value={"ok": True, "final": final, "error": None}), + ) as run_task, + mock.patch.object(browse, "usage_summary", return_value={"decisions": 1}), + ): + with tempfile.TemporaryDirectory() as directory: + answer = await browse.browse( + spec(), + BrowserPolicy(prefetch_values=False, batch_actions=False, goal_value_cache=False), + model_name="onejev", + goal="Open the fixture", + timeout_s=10, + max_steps=2, + logs_dir=Path(directory), + headless=True, + chat=mock.MagicMock(), + decision_model=decision_model, + ) + self.assertTrue((Path(directory) / "decision_ticks.json").is_file()) + self.assertIs(slot_class.call_args.args[2], counting.return_value) + self.assertIs(slot_class.call_args.kwargs["decision_model"], decision_model) + self.assertEqual(slot_class.call_args.kwargs["max_steps"], 2) + self.assertIs(create_agent.call_args.args[0], slot) + run_task.assert_awaited_once() + self.assertEqual(answer["final"], "Fixture answer")