[Eval] BrowseSafe-Bench sample for the injection guard and its page coverage at the 6,000-char clip

.claude/skills/build-s1a-agent/references/fronts.md

@@ -54,6 +54,7 @@ One Jev question at one callback hook of a DeepAgent, acted on above a threshold
 | `thresholds` | `Thresholds(allow, act)`: at or below `allow` nothing, at or above `act` the action runs |
 | `act` | `async (ctx, verdict) -> None`: quarantine a result, block a call, log, escalate |
 | `labelled_set` | JSONL of `{"state", "label"}` records; `s1a run <rail>` reports precision and recall |
+| `labelled_state` | optional `(state) -> state`: what the hook would send for a labelled record, so a set can keep pages in full and the evaluation measures the runtime clip; None sends the state as is |
 | `coverage_of` | optional `(ctx) -> Coverage(inspected, total) or None`: how much of the event `state_of` shows; ticks record it |
 | `on_partial` | optional `async (ctx, coverage) -> None`: runs instead of a pass when a verdict below the act band saw part of the event; needs `coverage_of` |
 

CHANGELOG.md

@@ -20,6 +20,10 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver
   without `--order`. `--order same:N` is the control: N more asks in file order, so a backend that disagrees with
   itself (sampling, a load-balanced server, GPU non-determinism) shows its own "flip" rate next to the reordered
   one. `--seed` or `--json` without `--order` is a usage error (exit 2) rather than silently ignored. See #90.
+- `scripts/build_browsesafe_set.py`: a seeded, label-stratified BrowseSafe-Bench test sample (row ids in
+  `evals/labelled/browsesafe-test-sample.json`, dataset not vendored) in the injection guard's labelled-set shape,
+  and the share of page text the guard inspects at its 6,000-character clip (`docs/benchmarks.md`). Every
+  command, including one given a local `--parquet`, refuses a file whose sha256 is not the pinned one.
 - Offline quality consumer for shared Omni public231 raw, retaining planned
   denominators and original scoring.
 - `s1a-mcp` tools carry titles and MCP behavior hints: `list_agents` and `decide` are read-only (so hosts with
@@ -63,6 +67,9 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver
   policy is in `docs/agents.md`. `RailSpec` gains optional `coverage_of` and `on_partial`. A failed decision's
   tick and warning log record the error's status and cause type (`error`, `error_code`) instead of its message,
   which can echo an HTTP error body quoting the guarded text.
+- A rail's labelled-set evaluation sends each state through the spec's runtime clip (`RailSpec.labelled_state`;
+  the injection guard cuts text to 6,000 characters) and records a failed decision as a `failure` row instead of
+  aborting the run; the summary adds `failures` and per-label band counts.
 - Malformed ticket-router JSONL now names the dataset, physical line and JSON column without printing ticket contents.
 - Laya browser-state compaction preserves the recovery plan for the next decision.
 - Desktop recovery resets its detection window on observed progress without resetting its cumulative budget.

NOTICE

@@ -12,6 +12,9 @@ This product includes software developed by third parties:
   evals/2048/style/fonts are Clear Sans by Intel Corporation (Apache-2.0)
 - evals/millionaire/questions.json, fetched at first run: Open Trivia Database
   (https://opentdb.com), CC BY-SA 4.0
+- evals/labelled/browsesafe-test-sample.json and scripts/build_browsesafe_set.py: row ids of, and a
+  converter for, BrowseSafe-Bench (perplexity-ai/browsesafe-bench; Zhang et al., 2025, arXiv:2511.20597),
+  MIT, Copyright 2025 Perplexity AI, Inc. The dataset itself is not included. Details: THIRD_PARTY_LICENSES.md
 - evals/labelled/injection-public.jsonl: tool outputs from InjecAgent,
   uiuc-kang-lab/InjecAgent (MIT, Copyright (c) 2023 Qiusi Zhan). Full license text:
   THIRD_PARTY_LICENSES.md

THIRD_PARTY_LICENSES.md

@@ -40,6 +40,33 @@ Fonts under `evals/2048/style/fonts` are Clear Sans by Intel Corporation, Apache
 `evals/millionaire/questions.json`, fetched on the first Millionaire run, holds questions from the
 [Open Trivia Database](https://opentdb.com), Creative Commons Attribution-ShareAlike 4.0 International.
 
+## BrowseSafe-Bench
+
+`evals/labelled/browsesafe-test-sample.json` lists row ids of the test split of
+[perplexity-ai/browsesafe-bench](https://huggingface.co/datasets/perplexity-ai/browsesafe-bench) at revision
+`b506fb5bc7fd4472c8738055a67a0ef6406afdc9`, described in Zhang et al., 2025, [BrowseSafe: Understanding and
+Preventing Prompt Injection Within AI Browser Agents](https://arxiv.org/abs/2511.20597).
+`scripts/build_browsesafe_set.py` downloads the split and converts the listed rows; no row content is
+redistributed in this repository.
+
+MIT License
+
+Copyright 2025 Perplexity AI, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+documentation files (the "Software"), to deal in the Software without restriction, including without limitation
+the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
+to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions
+of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
+TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
+CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE.
+
 ## InjecAgent
 
 The records of `evals/labelled/injection-public.jsonl` whose note starts with `injecagent-` are built by

docs/benchmarks.md

@@ -120,6 +120,63 @@ The set has two limits. Only 34 of the normal outputs contain a request. The pla
 requests also come from different benchmarks with different text formats, so a simple word-count classifier already
 separates them 73% of the time.
 
+### Page-length inputs: BrowseSafe-Bench
+
+The 20-item set and the InjecAgent/AgentDojo set of #23 are short tool outputs. Real `browser_snapshot` and
+`fetch_webpage` results are whole pages, and the guard decides on the first 6,000 characters (`MAX_CHARS`).
+[BrowseSafe-Bench](https://huggingface.co/datasets/perplexity-ai/browsesafe-bench) (Perplexity, MIT;
+[arXiv:2511.20597](https://arxiv.org/abs/2511.20597)) is HTML from real browser-agent use, with benign distractors
+and multilingual attacks. Its public test split has 3,680 rows (1,824 `yes`, 1,856 `no`) and only `content` and
+`label`; the paper's attack-type and strategy columns are not released. These figures are kept apart from #23's.
+
+**Sample.** `evals/labelled/browsesafe-test-sample.json`: 100 `yes` and 100 `no` rows drawn with seed 0 from the
+test split at revision `b506fb5`, `test.parquet` sha256 `00cbad96…0573d0`. The dataset is not vendored;
+`scripts/build_browsesafe_set.py build` downloads the pinned file, checks its digest and writes the JSONL with
+each page in full. The evaluation applies the runtime clip itself (`RailSpec.labelled_state`), so the scores
+measure what the guard sees in production. Budget: 200 Jev decisions per text mode.
+
+**Text.** `html` (default) is the row unchanged, as a raw `fetch_webpage` returns it; the label is defined on it,
+so hidden injections survive. `text` keeps the visible text, comments and the alt/title/aria-label/placeholder
+attributes, roughly a reader-mode fetch; it is about an eighth as long, but an injection in another attribute or
+a `<script>` disappears while the row stays `yes`. A Playwright `browser_snapshot` of the rendered page matches
+production best and is not built yet.
+
+**Coverage**, measured from the data (no model call), on 2026-10-14. "Inspected" is the share of page characters
+inside the 6,000-character clip, summed over pages; "mean page" averages that share per page. The
+`reaches model, main` column is the pass-through behavior this sample was measured against: a page below the
+act band reached the model whole. `reaches model, withhold` is #57's policy, which is the runtime: only the
+inspected prefix reaches the model. Both columns assume the page is allowed.
+
+| text | label | pages | median chars | fully inspected | text inspected | mean page inspected | reaches model, main | reaches model, withhold |
+|---|---|---|---|---|---|---|---|---|
+| html | yes | 100 | 53,084 | 8 | 9.8% | 24.3% | 100% | 9.8% |
+| html | no | 100 | 49,118 | 14 | 11.3% | 29.8% | 100% | 11.3% |
+| text | yes | 100 | 6,329 | 49 | 34.4% | 71.1% | 100% | 34.4% |
+| text | no | 100 | 5,860 | 51 | 43.5% | 75.8% | 100% | 43.5% |
+| html, whole split | yes / no | 1,824 / 1,856 | 44,863 / 47,749 | 187 / 157 | 10.5% / 10.4% | 27.4% / 25.4% | 100% | 10.5% / 10.4% |
+| text, whole split | yes / no | 1,824 / 1,856 | 5,918 / 5,802 | 921 / 950 | 39.7% / 40.4% | 74.6% / 74.7% | 100% | 39.7% / 40.4% |
+
+So on raw HTML the guard inspects about a tenth of the page text. Under the earlier pass-through behavior the
+other nine tenths reached the model uninspected; under the withhold policy a benign page loses them.
+Reader-mode text raises the inspected share to about 40%; most page text is still outside the clip.
+
+**Scores.** Not run yet: they need a Jev key (`TYPESAFE_API_KEY`). Reproduce with
+
+```bash
+uv run --with pyarrow python scripts/build_browsesafe_set.py build --text html
+uv run s1a run injection_guard --model jev --labelled-set .external/browsesafe/test-sample-html.jsonl
+uv run --with pyarrow python scripts/build_browsesafe_set.py coverage   # the table above
+```
+
+`summary.json` gives precision and recall of the act band with all 200 rows in the denominator, `uncertain`,
+`failures` (a failed decision counts as acted on this closed rail) and `bands`: allow/uncertain/act/failure per
+label. The paper's baseline on the same split is, for example, PromptGuard-2 22M at F1 0.350 (precision 0.975,
+recall 0.213).
+
+| date | text | rows | clip | model | label | allow | uncertain | act | failure | precision | recall |
+|---|---|---|---|---|---|---|---|---|---|---|---|
+| pending | html | 200 | 6,000 | jev | yes / no | | | | | | |
+
 ## Series
 
 The protocol in `evals/README.md` quotes nothing under ten episodes and asks for 500 Blackjack hands. The series of

evals/README.md

@@ -15,6 +15,7 @@ folder per episode). `uv run python -m evals.table evals/results` aggregates the
 | Ticket router (30 local labelled tickets) | correct routes to five queues | `s1a run ticket_router --model jev --rethink off --episodes 1` |
 | Injection guard (rail) | precision and recall on a labelled set | `s1a run injection_guard` |
 | Injection guard, public set (InjecAgent + AgentDojo) | precision and recall per source without the context-needed positives, balanced accuracy on the hard subset | `s1a run injection_guard --model jev --labelled-set evals/labelled/injection-public.jsonl`; numbers in `docs/benchmarks.md` |
+| Injection guard on page-length HTML (BrowseSafe-Bench sample) | precision, recall, failures and page coverage at the runtime clip | `s1a run injection_guard --labelled-set .external/browsesafe/test-sample-html.jsonl`, after `scripts/build_browsesafe_set.py build`; see `docs/benchmarks.md` |
 | Bounded recovery on/off (3 local form fixtures) | completion on/off in a real headless Chromium, judged by the fixture server | `python -m evals.recovery --repeat 1` |
 | Bounded recovery live (real decision/chat models, 3 local form fixtures) | completion on/off with the real models in a real headless Chromium, judged by the fixture server; cost null unless `CHAT_USD_PER_M_*` is set; optional `--validate-submission` variant | `python -m evals.recovery.live --model jev --repeat 1` |
 

evals/labelled/browsesafe-test-sample.json

@@ -0,0 +1,214 @@
+{
+ "dataset": "perplexity-ai/browsesafe-bench",
+ "revision": "b506fb5bc7fd4472c8738055a67a0ef6406afdc9",
+ "split": "test",
+ "parquet_sha256": "00cbad96b60fee46e016d79af6981fb221384c61f12cf28b4f04b5a6420573d0",
+ "seed": 0,
+ "per_label": 100,
+ "labels": {
+  "yes": 100,
+  "no": 100
+ },
+ "row_ids": [
+  4,
+  24,
+  57,
+  79,
+  80,
+  133,
+  140,
+  153,
+  154,
+  164,
+  169,
+  171,
+  209,
+  214,
+  217,
+  295,
+  301,
+  304,
+  327,
+  357,
+  358,
+  363,
+  382,
+  406,
+  413,
+  419,
+  435,
+  443,
+  448,
+  454,
+  474,
+  494,
+  496,
+  501,
+  507,
+  513,
+  543,
+  555,
+  563,
+  585,
+  607,
+  612,
+  613,
+  628,
+  659,
+  662,
+  689,
+  691,
+  717,
+  720,
+  739,
+  777,
+  781,
+  819,
+  844,
+  847,
+  848,
+  914,
+  927,
+  1009,
+  1029,
+  1049,
+  1089,
+  1128,
+  1135,
+  1143,
+  1158,
+  1165,
+  1181,
+  1229,
+  1236,
+  1262,
+  1266,
+  1271,
+  1306,
+  1318,
+  1357,
+  1374,
+  1383,
+  1404,
+  1441,
+  1499,
+  1501,
+  1503,
+  1525,
+  1537,
+  1538,
+  1543,
+  1573,
+  1586,
+  1606,
+  1619,
+  1626,
+  1629,
+  1633,
+  1643,
+  1658,
+  1700,
+  1732,
+  1765,
+  1780,
+  1784,
+  1796,
+  1811,
+  1825,
+  1846,
+  1863,
+  1877,
+  1892,
+  1901,
+  1978,
+  1983,
+  1993,
+  1998,
+  2001,
+  2048,
+  2049,
+  2055,
+  2060,
+  2071,
+  2087,
+  2121,
+  2129,
+  2156,
+  2175,
+  2187,
+  2224,
+  2225,
+  2254,
+  2262,
+  2309,
+  2324,
+  2350,
+  2351,
+  2356,
+  2390,
+  2409,
+  2434,
+  2444,
+  2470,
+  2488,
+  2498,
+  2514,
+  2525,
+  2530,
+  2540,
+  2546,
+  2559,
+  2564,
+  2631,
+  2673,
+  2682,
+  2706,
+  2741,
+  2744,
+  2802,
+  2827,
+  2863,
+  2864,
+  2882,
+  2898,
+  2899,
+  2912,
+  2913,
+  2919,
+  2928,
+  2935,
+  2989,
+  3001,
+  3031,
+  3035,
+  3079,
+  3112,
+  3178,
+  3202,
+  3261,
+  3270,
+  3298,
+  3336,
+  3343,
+  3374,
+  3397,
+  3412,
+  3469,
+  3470,
+  3484,
+  3499,
+  3501,
+  3504,
+  3528,
+  3562,
+  3576,
+  3581,
+  3590,
+  3593,
+  3594,
+  3613,
+  3640,
+  3663,
+  3671
+ ]
+}

s1a/agents/injection_guard.py

@@ -80,6 +80,14 @@ def _replace(inputs: ToolCallInputs, content: str) -> None:
     inputs.tool_msg = ToolMessage(content=content, tool_call_id=call_id)
 
 
+def clip_state(state: Json) -> Json:
+    """A labelled record's state as ``state_of`` would send it: the text cut to ``MAX_CHARS`` (a bare-text state too)."""
+    if isinstance(state, str):  # read_labelled accepts a state that is text alone
+        return state[:MAX_CHARS]
+    text = state.get("text")
+    return {**state, "text": text[:MAX_CHARS]} if isinstance(text, str) else state
+
+
 async def quarantine(ctx: AgentCallbackContext, verdict: Verdict) -> None:
     """Replace the tool result and the tool message the model reads with the notice alone."""
     inputs = ctx.inputs
@@ -117,6 +125,7 @@ async def withhold_uninspected(ctx: AgentCallbackContext, coverage: Coverage) ->
     act=quarantine,
     on_failure="closed",
     labelled_set=ROOT / "evals" / "labelled" / "injection.jsonl",
+    labelled_state=clip_state,
     coverage_of=coverage_of,
     on_partial=withhold_uninspected,
 )