Skip to content

Commit bf1bbdd

Browse files
authored
Merge pull request #435 from InferMatrixLab/fix/rebase-wrapper-signals-mcp-discovery-sigint
Graceful SIGINT/SIGTERM shutdown, resilient tool bridge, wrapper signal receipts
2 parents e26e1ee + 0cd77d6 commit bf1bbdd

33 files changed

Lines changed: 2395 additions & 132 deletions

‎adapters/vllm_omni/rebase/templates/module_rebase.live.prompt.tmpl‎

Lines changed: 28 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -239,13 +239,35 @@ every later call. Therefore:
239239
- Use `imx-omni-pytest …` with `TEST_TIMEOUT_SEC={MODULE_TEST_TIMEOUT_SEC}` for
240240
every required verification command. Keep the selected scope, assertions and
241241
granted GPU geometry; do not shrink or extend the test budget.
242-
- Never block a tool call on a long wait (`sleep 480 && tail …`). For anything
243-
longer than a short command, launch detached with stdin/stdout/stderr
244-
redirected and a PID file, e.g.
245-
`setsid -f sh -c 'echo $$ > "$1"; shift; exec "$@"' sh "$pidfile" imx-omni-pytest ... </dev/null >"$log" 2>&1`
242+
- Never block a tool call on a long wait (`sleep 480 && tail …`), and never run
243+
a verification command in the foreground of a tool call: a foreground run is
244+
killed at the tool's deadline together with its process group. Launch every
245+
`imx-omni-pytest` DETACHED — own session, stdin from /dev/null, stdout/stderr
246+
to a log, PID saved — in ONE tool call that returns immediately:
247+
```
248+
log=$IMX_LOG_DIR/tests/detached-<name>.out; pidfile=$IMX_LOG_DIR/tests/detached-<name>.pid
249+
setsid -f sh -c 'echo $$ > "$1"; shift; exec "$@"' sh "$pidfile" imx-omni-pytest ... </dev/null >"$log" 2>&1
250+
sleep 1; cat "$pidfile"; ps -o pid,sid,pgid,stat,etimes,cmd -p "$(cat "$pidfile")"
251+
```
246252
(a plain `nohup ... &` is insufficient: the non-interactive bridge shell waits
247-
for that job). Then poll the saved wrapper PID with short calls, at most one
248-
check per tool call and under 20 seconds each including any sleep.
253+
for that job). The `ps` line is the detach proof: the wrapper's `sid` must
254+
equal its own pid; if it does not, stop and relaunch, never fall back to a
255+
foreground run. The wrapper also re-sessions itself and ignores SIGHUP, so a
256+
tool deadline cannot end it, but it still must not be started in the
257+
foreground, where the tool's output pipe is its stdout.
258+
- Then POLL, at most one check per tool call and under 20 seconds each
259+
including any sleep: `kill -0 "$(cat "$pidfile")"` says live; once it is
260+
gone, read the newest `$IMX_LOG_DIR/tests/outcomes/wrap-*.json` receipt (it is
261+
written by the wrapper at exit) and `tail -n 40 "$log"`. The receipt's
262+
`rc`, `timed_out`, `watchdog_triggered` are the outcome; `signal` and
263+
`signal_origin` say whether a signal ended the child and who sent it.
264+
- A receipt with `signal_origin: "external"` (rc -2/-15/-9, `timed_out` and
265+
`watchdog_triggered` both false, typically ~30 s after start) means a
266+
supervisor OUTSIDE this process tree killed the test — on this host, the
267+
GPU reservation guard, because the GPUs in `CUDA_VISIBLE_DEVICES` were not
268+
reserved for the wrapper's process tree. That is not a test failure and not
269+
your code: record it in debug memory and report the module incomplete with
270+
the receipt path; do not retry in a loop and do not change the test scope.
249271
- Guard optional glob or command-substitution results before passing them to
250272
stdin-reading commands (`tail`, `head`, `cat`, `grep`); an empty expansion
251273
turns `tail -n 5 "$log"` into a bare stdin reader that holds the bridge.

‎doc/RUNBOOK-rebase.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,30 @@ requires a fresh module signal, no typed session/audit refusal, and the required
6969
plan decision from that module's current attempt. Plain completion prose does
7070
not replace these checks.
7171

72+
### Stopping a run (SIGINT/SIGTERM) and what it leaves behind
73+
74+
SIGINT (Ctrl-C, or a GPU scheduler's `--timeout`) and SIGTERM are a graceful
75+
shutdown, also when the launcher started the copilot with `nohup … &` and the
76+
process inherited an ignored SIGINT. The engine stops the running harness
77+
session (the vendor CLI and its MCP bridge), kills this run's marked background
78+
processes (including detached test wrappers), releases the GPU lock through the
79+
run finalizers, keeps the upstream scratch, and exits with `128 + signal`
80+
(130 for SIGINT, 143 for SIGTERM) within about a minute; a second signal does
81+
not shorten that. The in-flight module is recorded as `interrupted` in
82+
`substate.json` (never left `running` or promoted to `done`), `run_status.json`
83+
says `interrupted`, and `run_trace.jsonl` carries `module_interrupted` and
84+
`run_interrupted` events. `--resume <run-id>` re-runs interrupted modules and
85+
skips completed ones.
86+
87+
Module test receipts under `<run_dir>/tests/outcomes/wrap-*.json` name the
88+
signal that ended a test and its origin. `signal_origin: "external"` (rc
89+
-2/-15/-9 with `timed_out` and `watchdog_triggered` both false) means a
90+
supervisor outside the run — on a shared host, the GPU reservation guard —
91+
killed the test process: check that the reservation covers the wrapper's
92+
process tree before reading it as a test failure. The bridge's own lifecycle
93+
log, `<run_dir>/bridge/<step>.lifecycle.jsonl`, records why a tool server
94+
exited; a `bridge_unavailable` trace event summarises it per attempt.
95+
7296
## External checkout pin (EXT1-class guard)
7397

7498
Before cutover, the EXTERNAL orchestrator's canonical checkout gains a

‎doc/architecture/SPEC/app/core.md‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# app/core.py —— 规范
22

3-
<!-- verified-against: 2026-10-27 -->
3+
<!-- verified-against: 2026-10-28 -->
44

55
`Copilot` is the headless workflow application. It resolves a TaskSpec,
66
delegates reviewer verdict policy to `review.plan_gate`, confirms when needed,
@@ -39,3 +39,11 @@ run(`_contained_run_dir` 拒绝越界路径);`--repo/--playbook/--task-par
3939
在恢复时是对已保存 task.json 的**断言**而非修改,任一不匹配即 BLOCKED。
4040
发布 launcher 因此能把 resume 绑定到不可变的 campaign,而不会误入更新的
4141
无关任务。损坏的 task.json 同样 BLOCKED 而非崩溃。
42+
43+
## 被信号终止的预约 run(2026-10-28)
44+
45+
`_execute_reserved_locked` 的终态写入(`0 → done`、`BLOCKED_EXIT → blocked`、
46+
其他 → `failed`)在 `engine.lifecycle.shutdown_requested()` 非空时**跳过**:
47+
`workflow_execution._interrupted` 已经写下 `interrupted`,`128 + signal` 的
48+
退出码不得被这里改标成 `failed`(那会让 reconciler 把一个可恢复的 run 当作
49+
失败收尾)。

‎doc/architecture/SPEC/app/workflow_execution.md‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# app/workflow_execution.py — workflow execution boundary
22

3-
<!-- verified-against: 2026-09-26 -->
3+
<!-- verified-against: 2026-10-28 -->
44

55
`WorkflowExecution` runs an already selected playbook with the caller's
66
current LLM dependency. It rechecks the repository context before starting,
@@ -9,6 +9,20 @@ initializes tracing and notification, seeds executor state with checkout and
99
adapter policy, and drives `Executor` through `run_guarded`. It returns an
1010
immutable `ExecutionResult` containing the exit code and blocked reason.
1111

12+
Around `asyncio.run(run_guarded(...))` it installs the run's SIGINT/SIGTERM
13+
handlers (`engine/lifecycle.py::install_run_signal_handlers`) and restores
14+
them afterwards. A signalled run ends through `_interrupted`: run_status
15+
`interrupted` with this process as `child_pid` (`run_status.mark_interrupted`),
16+
a `run_interrupted` trace event (signal, exit code, completed steps), the
17+
per-step marks plus `run <id>: interrupted (<SIG>)`, and
18+
`ExecutionResult(128 + signal, reason)`. Both paths land there: the
19+
cooperative one (a step saw `shutdown_requested()` and returned blocked, so
20+
`executor.run` returned normally) and the forced one (the run task was
21+
cancelled; `CancelledError`/`KeyboardInterrupt` is caught only when a
22+
shutdown request exists, otherwise it propagates as before). Finalizers have
23+
already run inside `run_guarded` by then: GPU locks are released and the
24+
upstream scratch is kept, so the run is resumable with `--resume`.
25+
1226
A losing run lock leaves active run artifacts untouched. A failed knowledge
1327
lock releases an acquired run lock. A caller-provided lock stays owned by the
1428
caller. Finalizers and locks run on every executor outcome, including failure.

‎doc/architecture/SPEC/engine/lifecycle.md‎

Lines changed: 62 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,59 @@
11
# engine/lifecycle.py —— 规范
22

3-
<!-- verified-against: 2026-08-28 -->
3+
<!-- verified-against: 2026-10-28 -->
44

5-
`LOC ~141 · 引擎底座(run 生命周期原语) · refactor-status: ok`
5+
`LOC ~396 · 引擎底座(run 生命周期原语 + 优雅停机) · refactor-status: ok`
66

77
## 职责
8-
executor 自己给不了的两条**进程级**保证:同一 run 的互斥锁,与 run 离开事件
9-
循环时的 finalizer 挂钩点。
8+
executor 自己给不了的三条**进程级**保证:同一 run 的互斥锁,run 离开事件
9+
循环时的 finalizer 挂钩点,以及 SIGINT/SIGTERM 到达时**有界的优雅停机**
10+
(run-20261027-193251:引擎从 `nohup … &` 启动器继承了 SIGINT 的 `SIG_IGN`,
11+
GPU 调度器 `--timeout` 发出的 SIGINT 被整个忽略,随后 reservation guard 逐个
12+
杀掉其下的测试进程)。
1013

1114
## 功能
1215
`RunLock` 对 `<run_dir>/.lock` 取排他 advisory `flock`(第二个并发
1316
`--resume` fail fast,而不是交错写进度、共享 checkout);
1417
`register_finalizer`/`finalize` 维护逐 run 的异步 finalizer 表;
1518
`run_guarded` 包住 executor 协程,在完成/失败/抛异常的**每条**退出路径上、
16-
同一事件循环内执行 finalize。
19+
同一事件循环内执行 finalize,并把自己的 task 绑定给停机路径
20+
(`bind_run_task`),退出时解绑。
21+
22+
停机:`install_run_signal_handlers(run_dir, run_id) -> restorer` 为 SIGINT/
23+
SIGTERM 安装处理器(**无条件** `signal.signal`——继承来的 `SIG_IGN` 正是要
24+
覆盖的状态;`asyncio.Runner` 只在默认处理器仍在时才装自己的,所以本处理器
25+
在 `asyncio.run` 内保持权威);处理器调用 `request_shutdown(signum, …)`
26+
(幂等:第一次接受,之后只计 `repeats`),它启动守护线程 `imx-shutdown`
27+
并对绑定的 run task 调 `cancel`。工作线程按序:`terminate_children`
28+
(对所有已注册 harness 子进程 `kill_tree`,TERM→宽限→KILL,含其 MCP bridge
29+
后代)→ `cleanup_run_processes(run_dir, run_id)`(带 `IMX_RUN_ID`+`IMX_LOG_DIR`
30+
标记的后台链,含 detached 测试链)→ 注册的 `shutdown hooks`(模块循环用来
31+
把在飞模块标 `interrupted`)→ 等主线程正常退出事件循环;超过
32+
`SHUTDOWN_HARD_DEADLINE_S`(60 s)则再杀一次子进程并 `exit_fn(128+signum)`
33+
(默认 `os._exit`)。
1734

1835
## 公开契约
1936
`RunLock(run_dir).acquire()/release()`(context manager);`RunLockHeld`;
2037
`register_finalizer(run_dir, fn)`(`fn(outcome)`,outcome 是 RunOutcome,或
2138
executor 在产出之前就抛时为 None);`finalize(run_dir, outcome)`;
2239
`run_guarded(run, run_dir) -> outcome`;`require_file_locking()` 与
2340
`FileLockingUnavailable`(POSIX advisory 锁不可用时抛出的 fail-closed 门)。
24-
调用方:`cli/copilot.py` 在 run/resume 全程持锁并用 `run_guarded` 包
25-
`executor.run`;finalizer 注册者是 rebase 流水线(flock 释放、scratch 清理、
41+
停机面:`ShutdownRequest(signum, run_dir, run_id, requested_at, repeats)`
42+
(`signal_name`、`exit_code = 128 + signum`:SIGINT→130、SIGTERM→143);
43+
`shutdown_requested() -> ShutdownRequest | None`(协作式轮询点);
44+
`register_child(proc)/unregister_child(proc)/registered_children()`
45+
(transport 在一次 harness 调用期间注册 Popen,见 `providers/base.py::
46+
run_harness_cli`);`register_shutdown_hook(fn)/unregister_shutdown_hook(fn)`
47+
(同步、幂等、不得抛);`terminate_children(grace_s)`;
48+
`request_shutdown(signum, *, run_dir, run_id, hard_deadline_s, grace_s, exit_fn)`;
49+
`install_run_signal_handlers(run_dir, run_id, *, signals) -> restorer`
50+
(仅主线程;其他线程返回 no-op restorer);`_reset_shutdown_state_for_tests()`。
51+
调用方:`app/workflow_execution.py` 在 run/resume 全程持锁、在
52+
`asyncio.run` 外侧安装/恢复信号处理器并用 `run_guarded` 包 `executor.run`,
53+
捕获 `CancelledError`/`KeyboardInterrupt` 后若 `shutdown_requested()` 非空则走
54+
`_interrupted` 终局协议(run_status `interrupted`、`run_interrupted` trace、
55+
退出码 128+signum);`rebase_engine/module_rebase.py` 在每次 attempt 后轮询
56+
`shutdown_requested()` 并注册在飞钩子;finalizer 注册者是 rebase 流水线(flock 释放、scratch 清理、
2657
终局报告、CI abort 清理)与 `engine/worktrees.py` 的共享持有释放;
2758
`require_file_locking` 由 `engine/worktrees.py::materialize` 和
2859
`idempotency.py`(`key_lock`/`reap_stale`)消费。
@@ -47,24 +78,45 @@ executor 在产出之前就抛时为 None);`finalize(run_dir, outcome)`;
4778
finalize 真正完成,然后才让取消向外传播 —— 光 shield 会让外层先抛、
4879
loop 关闭时把孤儿 finalizer 拦腰取消。
4980
- 什么都没注册时整个挂钩点是 no-op(现有非 rebase playbook 行为不变)。
81+
- 停机**恰好一次**:第二个信号(任一种)只计 `repeats`,不重启序列、不改
82+
退出码;信号处理器只做"记状态 + 起线程 + `call_soon_threadsafe(cancel)`",
83+
不在处理器内做 I/O 等待。
84+
- 协作式与强制式两条路径留下**同一份**持久痕迹:模块 `interrupted`、run
85+
`interrupted`——前者由步骤自己在看到请求后返回 BLOCKED,后者由取消穿过
86+
finalizer(GPU 锁释放、scratch 保留)后在 `workflow_execution` 收口。
87+
- 硬截止是兜底不是常态:主线程正常退出事件循环(`_run_loop is None`)时
88+
工作线程直接返回,由主线程写终局并返回退出码;只有卡死的 executor 才会
89+
被 `os._exit(128+signum)` 终结。
90+
- 子进程注册面向 **cursor-agent/claude/codex 整棵树**:`subprocess.run` 隐藏
91+
Popen,所以 transport 必须经 `run_harness_cli` 注册;`terminate_children`
92+
用 `kill_tree` 走后代,bridge 不会被 reparent 到 init 后继续持有 run 的
93+
工具作用域。
5094

5195
## 边界 —— 不属于这里
5296
不含 teardown 的内容本身(各注册方自带);不做进度/checkpoint 持久化
5397
(executor);不是 run 状态查询(`run_status.py`);不含 step 逻辑。
5498

5599
## 依赖(允许)
56-
仅 stdlib(asyncio/os/pathlib/fcntl-守卫)。零 engine 内 import。
100+
stdlib(asyncio/os/signal/subprocess/threading/pathlib/fcntl-守卫);停机
101+
工作线程**延迟导入** `testing/process_tree.py::kill_tree` 与
102+
`rebase_engine/processes.py::cleanup_run_processes`。零 engine 内 import。
57103

58104
## 扩展点
59105
新的 run 级资源清理 = 调用方去 `register_finalizer`,不改这里;只有新的
60106
**进程级** run 保证才落在这里。
61107

62108
## 测试
63109
`test_run_lifecycle.py`(同进程/跨进程锁争用、release 后可重取、无 fcntl
64-
降级、finalizer 恰好一次 / 异常与取消隔离 / 失败路径也执行)。
110+
降级、finalizer 恰好一次 / 异常与取消隔离 / 失败路径也执行);
111+
`test_run_shutdown.py`(覆盖继承的 SIG_IGN 并恢复、请求幂等与 128+signum、
112+
工作线程杀注册子进程整棵树后在硬截止 `exit_fn`、无绑定 task 时跑钩子即返回、
113+
`run_guarded` 绑定/解绑与取消仍 finalize、`run_harness_cli` 注册/超时杀树、
114+
模块协作式/取消式 `interrupted`、步骤 BLOCKED、wave gate、resume 重跑)。
65115

66116
## 重构备注
67117
小而承重。`_finalizers` 是模块级进程状态 —— pop-before-run 就是它的自清
68118
机制;若将来单进程要并行多 run,把表挂到 run 对象而不是模块全局。
69119
`run_guarded` 的 shield 循环微妙(取消期间的 teardown 存活是评审加固过的)
70-
—— 改动前先重跑取消路径的测试。
120+
—— 改动前先重跑取消路径的测试。停机状态(`_shutdown`/`_children`/
121+
`_shutdown_hooks`/`_run_loop`)同样是模块级进程状态,与"一进程一 run"的
122+
前提绑定;测试经 `_reset_shutdown_state_for_tests()` 清场。

‎doc/architecture/SPEC/engine/steps/rebase_v3.md‎

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# engine/steps/rebase_v3.py —— 规范
22

3-
<!-- verified-against: 2026-10-27 -->
3+
<!-- verified-against: 2026-10-28 -->
44

55
`LOC ~2204 · step 库(v3 rebase 装配层) · refactor-status: oversized`
66

@@ -16,8 +16,8 @@ step —— 薄的受治理 wrapper,substate-first、类型化失败、发布
1616
| `rebase.v3_scan` | deterministic/read | `manifest_jobs`;`test_manifest.json` 产物 |
1717
| `rebase.v3_wheel` | deterministic/write_workspace | `upstream_commit`(选 commit → 装进目标 venv → **最后**才 pin Dockerfile:装失败绝不留脏树) |
1818
| `rebase.v3_assign` | deterministic/read | `active_modules`、`wave1_modules`、`wave2_modules`(path-sync 后按 wave 分派) |
19-
| `rebase.v3_wave_gate` | deterministic/read | wave-1 失败 ⇒ `wave2_modules=[]`;`halt_on_module_failure` ⇒ ESCALATE |
20-
| `rebase.v3_module_rebase` | script/write_workspace | `module_<name>_status`(foreach;同 run 串行锁 —— 同一 checkout;substate done/skipped 短路防重入二次执行) |
19+
| `rebase.v3_wave_gate` | deterministic/read | wave-1 失败**或 interrupted** ⇒ `wave2_modules=[]`;`halt_on_module_failure` ⇒ ESCALATE |
20+
| `rebase.v3_module_rebase` | script/write_workspace | `module_<name>_status`(foreach;同 run 串行锁 —— 同一 checkout;substate done/skipped 短路防重入二次执行;outcome `interrupted` ⇒ BLOCKED,可 `--resume`) |
2121
| `rebase.v3_test_loop` | script/write_workspace | `phase3_failed`;substate `tests.pipeline`/`infra_failures`;空 manifest ⇒ `manifest_empty` |
2222
| `rebase.v3_precommit` | script/write_workspace | 无发布 —— substate `tests.precommit`(passed/failed/failed_preexisting/not_declared) |
2323
| `rebase.v3_push_gate` | deterministic/read | `push_gate_flagged`、`push_gate_overrides`;不许 ⇒ FORBIDDEN;override 记 trace |
@@ -133,3 +133,16 @@ harness 会话通过 MCP tool bridge 拿到同一套工具面,plan gate 在派
133133
- baseline venv(`ensure_baseline_venv`):硬链接克隆 campaign venv 并换入
134134
baseline commit 的 wheel,让"omni main 对上次对齐的 wheel 是否也失败"可问;
135135
任何失败返回 None,保留保守的回归判定。
136+
137+
## 信号中断的模块:BLOCKED 收口、resume 重跑(2026-10-28)
138+
139+
`_v3_module_rebase` 在 `rebase_module` 返回 `status == "interrupted"`
140+
(引擎收到 SIGINT/SIGTERM,见 `engine/lifecycle` 与 `rebase_engine` 规范)时
141+
返回 `StepResult(False, BLOCKED, "<m>: interrupted by <SIG> — resumable with
142+
--resume", state_updates={module_<m>_status: interrupted})`:executor 不再推进到
143+
下一个 foreach 兄弟,run 由 `workflow_execution` 以 `interrupted` 收口。短路
144+
条件**不变**(只有 done/skipped),因此 `--resume` 重跑 interrupted 模块、跳过
145+
done;`_v3_wave_gate` 把 interrupted 与 failed 同等视为"wave-1 未完成"(半
146+
rebase 的模块不是 wave 2 的基础)。测试:`test_assembly.py::
147+
test_v3_module_rebase_reruns_interrupted_and_blocks_on_interrupt`、
148+
`test_run_shutdown.py::test_wave_gate_treats_interrupted_as_not_done`。

0 commit comments

Comments
 (0)