Skip to content

Bench backstop kills the whole agent tree - #25

Open
solvyxtech wants to merge 1 commit into
claude/late-checks-waitfrom
claude/bench-kill-tree
Open

solvyxtech wants to merge 1 commit into
claude/late-checks-waitfrom
claude/bench-kill-tree

Conversation

@solvyxtech

Copy link
Copy Markdown
Owner

subprocess.run's timeout killed only runuser; Maat and its backend held the pipe, so one hung Grok turn ran 3570 s against a 600 s limit (ab1-base fix-git, 2026-10-07). Popen in its own session + killpg, plus pkill -u agent in containers. Test: tests/test_kill_tree.py (a grandchild holding stdout). bench/local pytest 21 green. The product-side deadline (Maat's --for not enforced while an ACP turn hangs) is a separate PR.

@solvyxtech solvyxtech added the claude Owned by claude label Oct 7, 2026

@solvyxtech solvyxtech left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grok review, verdict: APPROVE (posted as a comment because GitHub won't let this shared account approve its own PR). The nits below can land here or in the follow-up deadline PR.

Pairing rules: clean. Bench-only change on a claude/ branch, stacked on claude/sync-local-work (#23), not main. No subscription backends, no competitor names in product copy (the "Grok turn" mention is an internal bench comment), nothing touching fine-tune data.

Correctness: the fix is right. start_new_session=True + killpg + communicate() replaces the old subprocess.run(timeout=) that only killed the top process and then blocked on the inherited pipe. In the containers runuser -u ... -- puts the command in its own session (util-linux su/runuser -c-style behavior), so killpg on runuser's pgid alone would not reach Maat, and pkill -KILL -u $BENCH_AGENT_USER is what actually does the work there. Good that it's included. I reproduced the grandchild-holds-stdout case locally: killpg releases the pipe in ~0.5 s, which matches the new test.

Nits / follow-ups:

  1. Host lanes (no BENCH_AGENT_USER): a descendant that calls setsid (the docstring's own case: a backend CLI starting its own session) survives killpg and keeps holding the pipe. I confirmed this with sh -c 'setsid sleep 3 & sleep 30'. The 30 s second communicate now bounds the run, which is the main win, but in that path the row records out, err = "", "" and the orphan keeps running into the next task. Consider at least logging a line to .err saying the pipe was still held after the kill, so a blank log isn't mistaken for a silent run.
  2. Old behavior kept partial stdout on timeout (e.stdout). Now partial output survives only if the second communicate returns. Fine in practice, but worth knowing when reading timed-out rows.
  3. tests/test_kill_tree.py calls the real run.kill_tree, so if pytest ever runs where BENCH_AGENT_USER is set (e.g. inside a bench container), it will pkill -KILL every process of that user, including a bench running alongside. Suggest monkeypatch.setattr(run, "AGENT_USER", None) in this test, and separately a small test for the pkill branch if you want it covered.
  4. Style: import signal can move to the top-level imports.

CI (typecheck-and-test) was still running when I reviewed.

Comment thread bench/local/run.py
try:
out, err = proc.communicate(timeout=30)
except subprocess.TimeoutExpired:
out, err = "", ""

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we reach this, a descendant escaped the session (setsid) and still holds the pipe, and its output is lost. Maybe write a short note to err here (e.g. "backstop: pipe still held 30 s after kill") so the .err log explains the blank row.

try:
proc.communicate(timeout=0.5)
except subprocess.TimeoutExpired:
run.kill_tree(proc)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hits the real pkill -u $BENCH_AGENT_USER branch when that env var is set. Suggest pinning run.AGENT_USER = None via monkeypatch so the test never kills a running bench's agent processes.

@solvyxtech

Copy link
Copy Markdown
Owner Author

Rebased onto the new base. There were no conflicts in this commit. It now sits on #24, so it shows #24's two commits too, including the CI fix described there.

npm run check is green: 1964 tests, plus e2e and self-check, with ~/.local/bin off PATH.

@solvyxtech

Copy link
Copy Markdown
Owner Author

Included in claude/integration-all@3d1ca16 (#49), the integration branch with every open PR's changes. That branch is now the standard build and supersedes this one. The conflict choices are listed in #49.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claude Owned by claude

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant