[Birfy]

Required human contributor answers

Per the repository owner's standing instruction (Danyang, 2026-10-14) for scheduled agent runs, these are filled from issue #109 rather than waiting for separate human answers.

1. What outcome are you trying to achieve?

   Answer: Compare browser decision backends at the step level, offline and repeatably, instead of from a few live tasks ([Eval] Score browser decision backends step by step on a public Mind2Web/WebLINX set in the jev-ultrafast request format #109).

2. What is the highest-leverage variable for that outcome?

   Answer: A per-backend step success and confidence-calibration number on a fixed, website-disjoint public set, with denominators.

3. How will this work move that variable rather than create busywork?

   Answer: It turns a 3,345-step frozen benchmark and a 652-step calibration split into one command per backend, and produces the threshold table [Feat] Confidence-gated deferral: hand low-confidence browser steps to the chat model and measure the trade-off #94 / [Feat] Add confidence-gated browser deferral #107 otherwise need live runs for.

4. Who is the customer or user, and what do they actually need?

   Answer: Maintainers choosing a browser backend or a deferral threshold; they need numbers that are cheap to rerun and honest about failures.

After answering the four questions: is a PR needed at all to achieve this outcome?

Human decision and rationale (or explicit source reference): Yes; issue #109 scopes one offline PR (collector, re-scorer, threshold table, docs).

For a bug fix: link actual observed failure evidence.

Evidence: Not a bug fix.

Why

Every browser number we publish comes from a few live tasks, which is slow, noisy and drifts with the sites. #109 asks for a step-level score: given this page and goal, does the backend pick the right operation and element, and is its confidence any good?

How

benchmarks/legs/browser_steps.py (open first), with tests/benchmarks/test_browser_steps.py:

• fetch downloads Mlchemist/myjev-browser-decisions at pinned revision 8c0c5d53 and keeps a file only if its hash matches (LFS SHA-256 for the splits, git blob id for the Jev-answered id list). No rows are committed (CC BY-NC-SA 4.0; Mind2Web's no-republish request is stated in the doc).
• predict turns each record into an Observation plus ChoiceQuestions and calls DecisionModel.decide_many (any text backend from build_model), one request at a time, one attempt, no retries. Every step is saved with the request body, answers, latency and usage, or the error. Gold is never sent.
• score reports operation accuracy, element accuracy, step success, invalid/failed rate, Brier and 10-bin ECE of the operation confidence, multiclass Brier, p50/p95 latency and per-operation step success, each with numerator and denominator, on the full split and the Jev-answered subset. Failed and unattempted steps count as wrong.
• thresholds turns a calibration run into a keep/defer table per confidence threshold.

Request shape, verified rather than assumed: --shape dataset re-sends jev-ultrafast's body unchanged; over all 652 real calibration records, the body JevModel builds is identical to the dataset's. --shape ours (default) keeps the state, question names and element rows (they already match our action space) and swaps in our _OPERATION_LABELS and OPERATION_RULES / TARGET_RULES. The remaining differences (no PRESS_ENTER / text_value heads, SELECT option label folded into element, no probe region / blocked_by) are listed in the doc.

What

• New python -m benchmarks.legs.browser_steps {fetch,predict,score,thresholds}.
• docs/results/browser-steps/README.md: run, licence, metrics, the shape comparison table, results.
• benchmarks/README.md pointer and a CHANGELOG.md Unreleased entry. No existing behaviour changes.

Verification

• ruff format --check . && ruff check . && ty check: all pass.
• pytest -q: 1356 passed, 132 skipped (14 new tests on a synthetic fixture, no network). scripts/smoke.sh was not runnable on this box (uv sync cannot fetch the gitcode intelli-router dependency); CI runs it.
• CHANGELOG.md and docs updated.
• Real-data check (offline, not committed): the pinned calibration file verifies; the random backend (seed 0) over all 652 steps gives operation 98/652, element 51/608, step success 12/652, 0 failed, in both shapes. That is the chance floor in the results table.

Demo / evidence

• Video link, or exemption/blocker with follow-up: Exempt. This is an offline benchmark tool with no application run to record.
• Application/task → agent actions → independently checked result: N/A (step-level offline scoring).
• System1-Agents / System1-Omni commits, checkpoint and worker/frontend provenance: Agents base 69527d9. Still missing for [Eval] Score browser decision backends step by step on a public Mind2Web/WebLINX set in the jev-ultrafast request format #109 scope item 5: hosted Jev and one open backend on the benchmark split. They need a TypeSafe key and a GPU host, which this run did not have, so this PR stays a draft until those rows land.

Closes #109

[Birfy]

Friendly ping — this PR is ready for review. Could someone take a look when you have a moment? Thanks!

[Birfy]

Self-review (Birfy; cannot approve own PR).

Ready-to-merge-from-my-side for the offline tooling half of #109: pinned HF revision + hash checks, no vendored rows (CC BY-NC-SA / Mind2Web no-republish called out), predict/score/thresholds through DecisionModel, --shape dataset vs ours verified on the calibration split, and the random-backend floor (12/652 step success) is the right honesty check. Tests on a synthetic fixture without network are solid.

What's left before this fully closes #109 / leaves draft:

1. Scope item 5 still open: hosted Jev + at least one open text backend on the frozen benchmark split (needs TypeSafe key / GPU host). Until those rows land under docs/results/browser-steps/, keep draft or get an explicit maintainer OK to merge the collector first.
2. When those rows land, also emit the calibration threshold table that #94 / #107 need — that is the main consumer of this leg.

No blocking code nits from this pass; CI green. Not merge-ready as a completed #109 until the hosted/open backend rows exist.