Repository navigation
Bench backstop kills the whole agent tree - #25
solvyxtech wants to merge 1 commit into
Conversation
solvyxtech
left a comment
There was a problem hiding this comment.
Grok review, verdict: APPROVE (posted as a comment because GitHub won't let this shared account approve its own PR). The nits below can land here or in the follow-up deadline PR.
Pairing rules: clean. Bench-only change on a claude/ branch, stacked on claude/sync-local-work (#23), not main. No subscription backends, no competitor names in product copy (the "Grok turn" mention is an internal bench comment), nothing touching fine-tune data.
Correctness: the fix is right. start_new_session=True + killpg + communicate() replaces the old subprocess.run(timeout=) that only killed the top process and then blocked on the inherited pipe. In the containers runuser -u ... -- puts the command in its own session (util-linux su/runuser -c-style behavior), so killpg on runuser's pgid alone would not reach Maat, and pkill -KILL -u $BENCH_AGENT_USER is what actually does the work there. Good that it's included. I reproduced the grandchild-holds-stdout case locally: killpg releases the pipe in ~0.5 s, which matches the new test.
Nits / follow-ups:
- Host lanes (no
BENCH_AGENT_USER): a descendant that callssetsid(the docstring's own case: a backend CLI starting its own session) surviveskillpgand keeps holding the pipe. I confirmed this withsh -c 'setsid sleep 3 & sleep 30'. The 30 s secondcommunicatenow bounds the run, which is the main win, but in that path the row recordsout, err = "", ""and the orphan keeps running into the next task. Consider at least logging a line to.errsaying the pipe was still held after the kill, so a blank log isn't mistaken for a silent run. - Old behavior kept partial stdout on timeout (
e.stdout). Now partial output survives only if the secondcommunicatereturns. Fine in practice, but worth knowing when reading timed-out rows. tests/test_kill_tree.pycalls the realrun.kill_tree, so if pytest ever runs whereBENCH_AGENT_USERis set (e.g. inside a bench container), it willpkill -KILLevery process of that user, including a bench running alongside. Suggestmonkeypatch.setattr(run, "AGENT_USER", None)in this test, and separately a small test for the pkill branch if you want it covered.- Style:
import signalcan move to the top-level imports.
CI (typecheck-and-test) was still running when I reviewed.
| try: | ||
| out, err = proc.communicate(timeout=30) | ||
| except subprocess.TimeoutExpired: | ||
| out, err = "", "" |
There was a problem hiding this comment.
If we reach this, a descendant escaped the session (setsid) and still holds the pipe, and its output is lost. Maybe write a short note to err here (e.g. "backstop: pipe still held 30 s after kill") so the .err log explains the blank row.
| try: | ||
| proc.communicate(timeout=0.5) | ||
| except subprocess.TimeoutExpired: | ||
| run.kill_tree(proc) |
There was a problem hiding this comment.
This hits the real pkill -u $BENCH_AGENT_USER branch when that env var is set. Suggest pinning run.AGENT_USER = None via monkeypatch so the test never kills a running bench's agent processes.
…past a 600 s limit
33d6e2b to
8dab78e
Compare
subprocess.run's timeout killed only runuser; Maat and its backend held the pipe, so one hung Grok turn ran 3570 s against a 600 s limit (ab1-base fix-git, 2026-10-07). Popen in its own session + killpg, plus pkill -u agent in containers. Test: tests/test_kill_tree.py (a grandchild holding stdout). bench/local pytest 21 green. The product-side deadline (Maat's --for not enforced while an ACP turn hangs) is a separate PR.