Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions DEVIATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,3 +83,13 @@
`demo/` 同步角色化:run 文档 `selector_backend` → `fast_model`(新增 `slow_model` 名),
排序调用计数 `jev_calls` → `fast_calls`、`jev_latency_ms_total` → `fast_latency_ms_total`
(这两个是 demo 层自算字段,不在结果目录契约里)。

## 可选阶段执行(2026-10-24)

28. 【行为,可选】`execution_mode=stage` 时,慢模型一次规划当前阶段的多个里程碑,
快模型依据每帧新观察重新选择动作;规则后置条件优先,证据不足才调用模型 Verify。
阶段完成、目标无法绑定、执行失败或阶段动作上限触发后返回慢模型。
不采用严格只规划一次的模式;阶段完成不等于任务完成,原有独立终态检查保留。
默认 `step` 路径不变,动作/观察/回执协议不变。
此为旧阶段执行实验的组件化移植,不包含其传输重试、部署或环境准备逻辑。
详细实现边界见 [STAGE_EXECUTION.md](docs/STAGE_EXECUTION.md)。
6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -293,3 +293,9 @@ python -m http.server 8899 --directory demo/site # 本地预览
| [DEVIATIONS.md](DEVIATIONS.md) | 与旧仓库 SA-CUAharness 的已知行为差异清单 |
| [AGENTS.md](AGENTS.md) | 开发纪律与目录地图 |
| [demo/README.md](demo/README.md) | demo 站点说明 |

## 阶段执行(可选)

默认保持逐步规划;启用 `--execution-mode stage` 后,由慢模型规划阶段、快模型连续选择动作,
规则验证优先,仅在必要时调用模型验证或重规划。
配置、执行边界、开发入口和验收方法见 [阶段执行说明](docs/STAGE_EXECUTION.md)。
2 changes: 2 additions & 0 deletions docs/CLUSTER_RUNTIME.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ worker 入口、共享资源与已知坑。
| `harness.fast_model.apis.<名字>` | 快模型 API 条目:`backend`(`jev` / `kev`)+ `endpoint` / `endpoint_file` + `timeout_seconds` |
| `harness.fast_model.image_stage.write` / `.read` | 截图中转目录的**两侧视角**:写入侧在 adpc,读取侧在 evobox(服务端宿主机要能读到) |
| `harness.route.policy` / `.allow_cli` / `.cli_timeout_seconds` | 路由策略与 CLI 开关 |
| `harness.execution.mode` / `.stage_max_actions` | 可选:step(默认)或 stage;阶段内动作上限默认 24。见 [阶段执行说明](STAGE_EXECUTION.md) |
| `harness.budget.max_steps` / `.max_wall_seconds` | 单题预算默认值(命令行可覆盖) |

**慢模型与快模型都是命名 API 登记表**:登记多套大脑/小脑,启动命令用 `--slow-model` /
Expand Down Expand Up @@ -135,6 +136,7 @@ cleanup stop(释放资源)+ delete(永久删除 rjob 记录)——默
| `--slow-model NAME` | 慢模型(大脑)API 名,取自 `harness.slow_model.apis`;缺省用 `default` |
| `--fast-model NAME` | 快模型(小脑)API 名,取自 `harness.fast_model.apis`;缺省用 `default` |
| `--route-policy` / `--allow-cli` | 路由策略与 CLI 开关 |
| `--execution-mode` / `--stage-max-actions` | step/stage 模式与阶段动作上限 |
| `--max-steps` / `--max-wall` | 单题预算(步数 / 墙钟秒) |
| `--slow-reasoning` | `off` / `low`(默认)/ `medium` / `high` |
| `--slow-max-tokens` | 慢模型单次响应上限 |
Expand Down
10 changes: 10 additions & 0 deletions docs/RESULT_DIR_CONTRACT.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ OSWorld 2.1 显式配置替代 judge/用户模拟器时,`code_version.json.res
|------|--------|---------|
| `task_requirements` | verifier.task_requirements | episode 开始时抽取任务需求(1-12 条) |
| `plan_step` | planner.plan_step | 每步规划(截图+可见控件+需求+近期动作→下一步决策) |
| `plan_stage` | stage_planner.plan_stage | 仅 stage 模式:初始阶段、阶段完成或请求修复时生成阶段计划 |
| `verify_observation` | verifier.verify_observation | 每步动作执行后独立验证(BEFORE/AFTER 截图→achieved/not_achieved/uncertain)|

`verify_observation` 不信任 planner 的自我宣称:它拿前后截图+预期变化独立裁决,驱动 recent 历史、no_action 连击计数、toggle 检查、以及终态双确认(terminal_claim 模式下逐条核验 requirements + 输出文件存在性)。
Expand All @@ -75,8 +76,17 @@ OSWorld 2.1 显式配置替代 judge/用户模拟器时,`code_version.json.res
| `cli` | planner 的 route=cli 决策 + 命令 |
| `gui_verified` / `cli_verified` | 验证裁决(outcome/task_status/evidence/state_changed)+ 回执 |
| `terminal` | 终态裁决(planner 声称完成后的 verifier 双确认全文) |
| `stage_local` | 消费已生成的阶段计划;含 stage_goal、milestone 和当帧重绑定后的 plan,不代表新的模型调用 |
| `stage_progress` | 里程碑推进,含 stage_goal、milestone、验证 source |
| `stage_replan` | 阶段修复请求,含原因及当前里程碑;不等于调用已经完成 |
| `action`(kind 字段) | runner 写的执行记录 `{step, kind: "action", action: {…协议动作…}}` |

stage 模式的 planner 事件只在真实规划调用时写入,增加 mode、reason、seconds;
gui/cli 中的 plan 带 planner_called,表示本轮是否刚调用规划模型。
gui_verified/cli_verified 的 verification 增加 source(rule_receipt / rule_a11y / model)
与 seconds。规则检查不写 llm_calls,不能计为一次慢模型 Verify。
其他文件和动作/观察/回执协议不变;详细触发规则见 [STAGE_EXECUTION.md](STAGE_EXECUTION.md)。

## failed_unscored 纪律

API/VM/评分异常 ≠ 0 分——异常题不写 `result.txt`,`episode.json.status=failed_unscored`,run 继续下一题。`summary.json.failed_unscored` 单独计数,不计入均分。
Expand Down
3 changes: 3 additions & 0 deletions docs/STAGE_EXECUTION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# 阶段执行

通过 `--execution-mode stage` 启用:慢模型规划阶段,JEV/Kev 连续选择 GUI 动作,CLI 执行计划中的命令;优先使用规则验证,证据不足时调用模型 Verify,失败或状态偏离时按需重规划,最终完成仍需独立验证。默认保留原有 `step` 模式。
4 changes: 4 additions & 0 deletions infra/config.example.json
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,10 @@
"allow_cli": true,
"cli_timeout_seconds": 90
},
"execution": {
"mode": "step",
"stage_max_actions": 24
},
"budget": {
"max_steps": 30,
"max_wall_seconds": 1800
Expand Down
3 changes: 2 additions & 1 deletion src/tandem/components.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
from .planner import plan_step
from .proposals import focused_candidates
from .selector import JevOmniPolicy
from .stage_planner import plan_stage
from .targets import A11yTargetBuilder
from .verifier import task_requirements, verify_observation

Expand Down Expand Up @@ -42,7 +43,7 @@ def build_default(*, client, endpoint: str, config: SessionConfig,
image_stage=image_stage)
return Components(
planner=partial(
plan_step,
plan_stage if config.execution_mode == "stage" else plan_step,
client,
allow_cli=config.allow_cli and config.route_policy != "gui",
route_policy=config.route_policy,
Expand Down
6 changes: 6 additions & 0 deletions src/tandem/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ class SessionConfig:
allow_cli: bool = True
route_policy: str = "auto"
cli_timeout_seconds: int = 90
execution_mode: str = "step"
stage_max_actions: int = 24
slow_max_tokens: int = 16384
slow_timeout_seconds: float = 120
slow_direct_api: bool = False
Expand All @@ -23,6 +25,10 @@ class SessionConfig:
fast_timeout_seconds: float = 300

def __post_init__(self):
if self.execution_mode not in {"step", "stage"}:
raise ValueError("execution_mode must be step or stage")
if type(self.stage_max_actions) is not int or self.stage_max_actions < 1:
raise ValueError("stage_max_actions must be a positive integer")
if self.route_policy not in {"auto", "gui", "cli"}:
raise ValueError("route_policy must be auto, gui or cli")
if not 1 <= self.cli_timeout_seconds <= 120:
Expand Down
19 changes: 12 additions & 7 deletions src/tandem/episode.py
Original file line number Diff line number Diff line change
Expand Up @@ -117,12 +117,10 @@ def verify_pending(session, observation: Observation, tree: str | None) -> None:
receipt = observation.last_receipt
if receipt is None:
raise RuntimeError("环境未返回上一步执行回执(协议违例)")
screenshot_after = Path(observation.screenshot.path)
screenshot_before = Path(pending["before_screenshot"])
if pending["kind"] == "cli":
verification = session.components.verifier(
session.instruction,
screenshot_after,
verification = _verify_action(
session, observation, tree, pending,
previous=screenshot_before,
expected=pending["plan"]["expected_observation"],
action=pending["action"],
Expand Down Expand Up @@ -166,9 +164,8 @@ def verify_pending(session, observation: Observation, tree: str | None) -> None:
and tree is not None
else None
)
verification = session.components.verifier(
session.instruction,
screenshot_after,
verification = _verify_action(
session, observation, tree, pending,
previous=screenshot_before,
expected=pending["plan"]["expected_observation"],
action=winner.compact(),
Expand Down Expand Up @@ -217,3 +214,11 @@ def verify_pending(session, observation: Observation, tree: str | None) -> None:
"control_state_check": control_check,
}
)


def _verify_action(session, observation, tree, pending, **kwargs):
if session.stage is not None:
return session.stage.verify(session, observation, tree, pending, **kwargs)
return session.components.verifier(
session.instruction, Path(observation.screenshot.path), **kwargs,
)
34 changes: 22 additions & 12 deletions src/tandem/session.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
from .focus import prune_stalled
from .protocol.models import Action, CliAction, FailAction, GuiAction, Observation, WaitAction
from .selector import ranking_dict
from .stages import StageExecution
from .stopping import NO_ACTION_LIMIT, budget_exhausted


Expand All @@ -40,6 +41,9 @@ def begin(self, instruction: str = "", requirements: list | None = None) -> None
self.failed_toggles: dict[str, tuple[str, object]] = {}
self.pending: dict | None = None
self.done_probe: dict | None = None
self.stage = (
StageExecution(self.config.stage_max_actions) if self.config.execution_mode == "stage" else None
)

# ------------------------------------------------------------------ 主入口
def act(self, observation: Observation) -> Action:
Expand All @@ -64,17 +68,20 @@ def act(self, observation: Observation) -> Action:
# 同一观察内最多重规划 NO_ACTION_LIMIT+3 次:完成被拒/无候选会递增 no_action 并重规划,
# 与旧循环 continue 语义一致(旧版重取 obs,本帧无动作时内容等价)。
for _ in range(NO_ACTION_LIMIT + 3):
plan = self.components.planner(
self.instruction,
screenshot,
targets,
self.recent,
tree if tree is not None else "<empty/>",
self.verified_focus,
self.requirements,
observation.budget.steps_remaining,
)
self.events.append({"step": observation.step_idx, "route": "planner", "plan": plan})
if self.stage is not None:
plan = self.stage.plan(self, observation, targets, tree, screenshot)
if plan is None:
self.consecutive_no_action += 1
if self.consecutive_no_action >= NO_ACTION_LIMIT:
return FailAction(reason="no_action_limit")
continue
else:
plan = self.components.planner(
self.instruction, screenshot, targets, self.recent,
tree if tree is not None else "<empty/>", self.verified_focus,
self.requirements, observation.budget.steps_remaining,
)
self.events.append({"step": observation.step_idx, "route": "planner", "plan": plan})
if plan["done"]:
terminal = handle_done(self, plan, observation, tree)
if terminal is not None:
Expand All @@ -99,6 +106,7 @@ def _emit_cli(self, plan: dict, observation: Observation) -> CliAction:
"action": action.to_dict(),
"plan": plan,
"before_screenshot": observation.screenshot.path,
"before_tree": self._read_tree(observation),
}
self.consecutive_no_action = 0
self.events.append(
Expand Down Expand Up @@ -146,7 +154,7 @@ def _plan_gui(
for service_retry in range(3):
try:
ranking = self.components.selector.rank(
stage_goal=self.instruction,
stage_goal=plan.get("stage_goal", self.instruction),
intent=local_goal,
active_window="",
targets=targets,
Expand Down Expand Up @@ -175,6 +183,8 @@ def _plan_gui(
}
)
if compiled is None:
if self.stage is not None:
self.stage.invalidate(self, "no_executable_candidate", observation.step_idx)
self.consecutive_no_action += 1
self.recent.append(
{
Expand Down
Loading
Loading