[Birfy]

Required human contributor answers

Per the repository owner's standing instruction (Danyang, 2026-10-14) for scheduled agent runs, these are filled from issue #98 rather than waiting for separate human answers.

1. What outcome are you trying to achieve?

   Answer: Locate unknown desktop controls via Jev Box instead of hand-written --pixel-target coordinates ([Feat] Find desktop click targets with Jev's Box question instead of hand-written --pixel-target coordinates #98).

2. What is the highest-leverage variable for that outcome?

   Answer: Noul presence + Box confidence–gated targets offered each step, with Box answering only where.

3. How will this work move that variable rather than create busywork?

   Answer: Replaces hand-written --pixel-target for unknown layouts; choice among offered actions stays the existing decision step.

4. Who is the customer or user, and what do they actually need?

   Answer: Desktop agent users clicking controls without accessibility entries or known coordinates.

After answering the four questions: is a PR needed at all to achieve this outcome?

Human decision and rationale (or explicit source reference): Yes; issue #98 scopes one feature PR after #97 lands. This PR's base is issue-97-jev-images (PR #99) so the diff is only #98 work.

For a bug fix: link actual observed failure evidence.

Evidence: Not a bug fix.

Why

Desktop screenshot actions need hand-written --pixel-target coordinates today. That fails for apps whose layout is unknown in advance. Jev 1.14's Box question returns where something is; TypeSafe's pattern is to gate it with a presence noul because Box always returns a box.

How

Depends on #97 / PR #99 (S1A_JEV_IMAGES, danger-zone header). Open first:

• s1a/desktop/env.py — locate_box_targets, gated box: candidates, centre click with capture binding
• s1a/agents/desktop.py — --box-target, Jev 1.14 + images usage errors, uncalibrated thresholds
• s1a/decision_models/types.py / validation.py / jev.py — BoxQuestion / Box
• s1a/tool/models.py — locate before the choice decision; record box on the tick
• tests/test_desktop_box_target.py, scripts/measure_box_targets.py

What

• New desktop flag --box-target KEY=description (experimental).
• Settings S1A_BOX_PRESENCE_THRESHOLD / S1A_BOX_CONFIDENCE_THRESHOLD (default 0.5, uncalibrated).
• --pixel-target unchanged; --box-target requires --model jev, S1A_JEV_IMAGES=1, TYPESAFE_MODEL=jev-1.14.0+.
• Docs mark experimental; no general accuracy claim.
• Out of scope (per issue): browser Box; tiling/zoom search.

Verification

• ruff format --check / ruff check / ty check: pass (box env used pip workaround for gitcode intelli-router 418; same as prior weekday runs).
• pytest -q: 1364 passed, 132 skipped (7 new in tests/test_desktop_box_target.py). scripts/smoke.sh not run here (uv sync blocked on gitcode); CI runs it.
• CHANGELOG.md and docs updated.

Demo / evidence

• Video link, or exemption/blocker with follow-up: Blocker — live [RFC] Local Cua-S1 4B and screenshot targets for #14 and #15 #30 visual fixture measurement needs TypeSafe images + jev-1.14.0 (no key/GPU on this agent box). Offline unit tests + docs ship; PR stays draft until a fixture run or maintainer exemption.
• Application/task → agent actions → independently checked result: N/A offline; planned command sketched in evals/desktop/visual.md.
• System1-Agents / System1-Omni commits, checkpoint and worker/frontend provenance: Agents head of this PR efae4c3 on base issue-97-jev-images (8ec00f9).

Closes #98

[Birfy]

Self-review (Birfy; cannot approve own PR).

Ready-to-merge-from-my-side once the stack is ready: the #98 slice looks correctly scoped on top of #99 (issue-97-jev-images). --box-target is opt-in, gated by presence noul + Box confidence (defaults clearly marked uncalibrated), --pixel-target stays, and Jev 1.14 + S1A_JEV_IMAGES usage errors are the right hard gates. BoxQuestion / Box on the decision-model types, locate_box_targets binding to the current capture, and the new unit tests are what I wanted for an experimental path.

What's left before leaving draft / merge:

1. Depends on #99 merging first (base is issue-97-jev-images).
2. Live #30 visual fixture measurement (or maintainer exemption) — still blocked without TypeSafe images + jev-1.14.0 on a machine that can run the desktop fixture; offline tests alone are fine for draft, not for a quality claim.
3. Keep the docs/experimental wording; do not imply calibrated thresholds until scripts/measure_box_targets.py has a real capture run recorded.

No code-change request from this pass; CI green (core/full/benchmarks). Stay draft until (1)+(2).