Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -1114,5 +1114,6 @@ Create and check whatever MM would ask about *before* the call (design/88 E1).
**Never read `is_alive()` after a `Thread.join` that Ctrl-C can interrupt.** On
CPython 3.12 an interrupted `join` releases the live thread's state lock and
marks it stopped. 88a's supervisor returned mid-run because of it. Wait on an
`Event` the worker sets in its `finally` instead. The pycro-manager teardown
wait still has the hazard (`R155`).
`Event` the worker sets in its `finally` instead. Both acquisition
supervisors now do (design/90 90a): `_wait_acquisition_completion` waits on
the event and the pending-acquisition record carries it, never the `Thread`.
12 changes: 7 additions & 5 deletions design/70-carried-forward-register.md
Original file line number Diff line number Diff line change
Expand Up @@ -233,8 +233,8 @@ Sorted by ease, then by importance. `→` names an existing block; do the block,
| `R95` | [Does live-mode ownership churn precede a lost terminal notification?](#r95) | MEDIUM | LARGE | |
| `R96` | [An unterminated acquisition with every frame accounted is summarised to the user as a success](#r96) | MEDIUM | SMALL | |
| `R97` | [The runtime deadline includes Acquisition() construction, and nothing records how much](#r97) | LOW | SMALL | |
| `R155` | [Ctrl-C during a pycro-manager teardown wait can make a live waiter look finished](#r155) | MEDIUM | SMALL | |
| `R158` | [84a's window test reads the final display once, without a wait](#r158) | LOW | SMALL | |
| ~~`R155`~~ | [Ctrl-C during a pycro-manager teardown wait can make a live waiter look finished](#r155) | MEDIUM | SMALL | **90a** |
| ~~`R158`~~ | [84a's window test reads the final display once, without a wait](#r158) | LOW | SMALL | **90a** |

**Demo machine**

Expand Down Expand Up @@ -1416,11 +1416,12 @@ This row comes from the register's table "Absorbed into a block above". Verbatim

**On CPython 3.12, a `Thread.join(timeout)` interrupted by `KeyboardInterrupt` releases the live thread's state lock (bpo-45274's cleanup) and marks it stopped, so `is_alive()` returns `False` while the thread still runs.**

- **Status** — **CLOSED 2026-10-19** by `design/90` 90a (PR #61). The defect was wider than this row said: an interrupt in `_acquire_with_hooks`'s wait wrote no pending record at all, so the refusal had nothing to read. The wait is now on an `Event`, the record carries it, and an interrupt hands teardown and reservation closure to the waiter. A real `interrupt_main()` test failed on the pre-fix tree and passed 30 of 30 after. The other sites below were read, not measured, as unreachable by a main-thread Ctrl-C (design/90's table).
- **What happened** — found by `design/88` 88a's gate selftest, 2026-10-17: `run_mda` returned mid-run with `outcome=None` while its worker kept going, and the next acquisition was not refused. Reproduced with a standalone probe on 3.12.14. 88a fixed `run_mda` by waiting on a completion `Event` instead of joining.
- **Where else** — `_acquire_with_hooks` waits with `_join_acquisition_waiter` and then reads `waiter.is_alive()`, and `execute_tool`'s pending-acquisition refusal reads the same flag, so a terminal Ctrl-C during a `run_timelapse` teardown wait can make the refusal believe teardown finished. 88a's runner also listed `controller._bridge_call`, `AcquisitionDiagnosticWriter.close` and `skill_supervisor`'s pipe-thread joins; those run on background threads, so a main-thread Ctrl-C reaching them is not established.
- **What a fix would look like** — the waiter sets an `Event` in its `finally`; foreground waits use `Event.wait`; nothing reads `is_alive()` after an interruptible `join`. Test with a real `_thread.interrupt_main()` delivered inside the wait.
- **Where** — LOCAL.
- **Block** — NONE.
- **Block** — `design/90` block 90a.
- **Importance** — MEDIUM. Only the terminal path delivers Ctrl-C into that wait; the web UI's Stop does not.
- **Effort** — SMALL
- **Provenance** — `design/88` 88a, 2026-10-17.
Expand Down Expand Up @@ -3380,11 +3381,12 @@ the model" as the actual blast radius.

**`test_84a_fixture_live_pause_final_drain_and_eof` failed on Windows CI at its last line, `assert displayed(output, 3)`: one read of `display.json`, 0.2 s after the result, where every earlier display check uses a bounded `wait_until`.**

- **Status** — **CLOSED 2026-10-19** by `design/90` 90a (PR #61): the line is now `wait_until(lambda: displayed(output, 3), describe)`. The cause of the one failure is still not identified; the next one will carry its evidence.
- **What happened** — PR #59, run 39113232886, `windows-latest`, 2026-10-18, at commit `100fd8b`. Since the previous green run (`5d418ec`) only files under `design/` had changed, so no code under test differed. 1 failed, 5954 passed. This is the test `R153` named, but a different line: R153 was the 8 s first-frame wait at `:1750`, which design/86 86a fixed.
- **Cause** — not identified. Either the worker had not yet drawn frame 3 within 0.2 s on the runner, or the read met the fixture's `display.json` replace (the race 86a found for the first frame). The assertion carries no `describe()` evidence, so the log cannot tell which.
- **What a fix would look like** — `wait_until(lambda: displayed(output, 3), describe)`, as the earlier checks do, so a failure carries the record, `poll()` and `display.json`.
- **Where** — LOCAL, then Windows CI.
- **Block** — NONE.
- **Block** — `design/90` block 90a.
- **Importance** — LOW. The test, not the product: the product's result had already been asserted.
- **Effort** — SMALL
- **Provenance** — `design/88` 88b's CI, 2026-10-18.
Expand All @@ -3398,6 +3400,6 @@ the model" as the actual blast radius.
- **What a fix would look like** — run P1 and P4 on M2 with the activation property as the probed target. `design/89-probes.py` already takes `--device`/`--property`. The `.ps1` launchers do not pass them yet, and the probe's limit check must accept that property. If SMAPpy runs beside them, they can carry `R142`.
- **Where** — M2.
- **Block** — NONE; 89b's M2 gate depends on it.
- **Importance** — MEDIUM. 89b should not be opened without it, if 89b is wanted.
- **Importance** — MEDIUM. Recommended before 89b, not required: 89b's own M2 gate compares cadence with feedback on and off, so this would only find a problem earlier and more cheaply (operator decision, 2026-10-19).
- **Effort** — SMALL
- **Provenance** — `design/89` 89a, 2026-10-19.
108 changes: 108 additions & 0 deletions design/90-interrupted-teardown-wait.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
# An interrupted teardown wait must not release the next acquisition

**Status: CLOSED with PR #61.** One block, 90a, LOCAL gate plus CI. Closes
`R155` and `R158` (design/70).

## The problem

**R155.** On CPython 3.12 a `Thread.join(timeout)` interrupted by
`KeyboardInterrupt` releases the live thread's state lock and marks it stopped,
so `is_alive()` returns `False` while the thread still runs (bpo-45274's
cleanup). design/88 88a found this in `run_mda` and fixed it there by waiting on
a completion `Event` (`_wait_mda_completion`). `_acquire_with_hooks`, the
pycro-manager path, still joins:

- `tools.py` `_join_acquisition_waiter` is `waiter.join(timeout_s)`, called
inside `while waiter.is_alive()`, and the expiry check reads
`waiter.is_alive()` again after it.
- `execute_tool`'s pending-acquisition refusal reads
`pending["waiter"].is_alive()`.

Reading the code on `0ffa44a` found the defect is wider than the row says. A
`KeyboardInterrupt` raised inside that wait leaves `_acquire_with_hooks`
through its `except Exception` untouched, so **no pending record is written at
all**. The refusal has nothing to read, and the next acquisition starts while
the daemon teardown waiter is still inside `Acquisition.__exit__`. The waiter
also does not inherit reservation closure, which only the expiry path grants.

**Who can reach it.** The terminal REPL does not catch `KeyboardInterrupt`
around `run_agent`, so today a Ctrl-C in that wait ends the process. What
reaches the refusal is any caller of `execute_tool` that survives the
interrupt, such as a notebook kernel. The contract the refusal states is
caller-independent, so it is fixed at the tool.

**The other `is_alive()` sites.** The hazard needs an interruptible join *on
the main thread* (CPython delivers `KeyboardInterrupt` only there) followed by a
later `is_alive()` read of the same `Thread`:

| Site | Thread | Reachable |
|---|---|---|
| `controller._bridge_call` | caller (main in the terminal) | No. The interrupt leaves before `is_alive()` and the `Thread` is local. |
| `AcquisitionDiagnosticWriter.close` (`conversation.py`) | main, from `__main__`'s `finally` | No, same shape. |
| `skill_supervisor` pipe-thread joins | dispatcher or reaper threads; `close()` joins dispatchers but reads no `is_alive()` | No. |
| `skill_store._start_discovery_recheck` | reads a thread nothing in the product joins | No. |

None is fixed. That a Ctrl-C cannot reach them is read from the code, not
measured.

**R158.** `tests/test_skill_supervisor.py`
`test_84a_fixture_live_pause_final_drain_and_eof` ends with
`assert displayed(output, 3)`, one read with no evidence. It failed once on
Windows CI (PR #59). Cause not identified.

## Decision

1. **One wait helper, on an `Event`.** `_join_acquisition_waiter` and
`_wait_mda_completion` fold into one helper that does `done.wait(timeout_s)`.
`_acquire_with_hooks`'s `finish()` sets a `done` event as the last thing in
its outermost `finally`. The loop and the expiry check read
`done.is_set()`, never `is_alive()`.
2. **The pending record carries the event.** Both writers (`_acquire_with_hooks`
expiry and `run_mda`'s `expired`) store `"done"`. `execute_tool` decides
`teardown_running` from `not done.is_set()`. No `waiter` key survives
(no legacy anchoring).
3. **An interrupt leaving the wait hands teardown to the waiter, as expiry
does.** If a `BaseException` leaves the supervision loop while `done` is
unset, record the pending entry, grant the waiter reservation closure, and
re-raise. No abort is requested. `run_mda`'s cancel-on-interrupt is that
tool's design and is unchanged.
4. **R158:** `wait_until(lambda: displayed(output, 3), describe)`. No deadline
change.

## Block 90a

Acceptance:

- A test drives a hooked or hookless `run_timelapse` through `execute_tool`,
delivers a real `_thread.interrupt_main()` while the foreground is inside its
blocking wait, and asserts that the next acquisition call is refused with
`teardown_running: true` while the teardown waiter is still blocked. After
release it is no longer refused for teardown. Watched failing on `0ffa44a`.
- 88a's interrupt regression in `test_mda_supervision.py` still passes on the
folded helper.
- R158's line uses `wait_until`.
- Full suite green locally and on both CI runners.

## Run ledger

| Block | Branch | Start commit | Status |
|-------|--------|--------------|--------|
| 90a | `block-90a` | `0ffa44a` | PR #61, 2026-10-19; local gate passed, CI green on both runners (run 39276478002); one start turn, no revision |

## What 90a built and measured (2026-10-19)

- **As decided.** One helper, `_wait_acquisition_completion(done, timeout_s)`,
serves both supervisors. The pending record carries `done`. An interrupt
leaving `_acquire_with_hooks`'s loop writes the record and grants the waiter
reservation closure.
- **Watched failing.** `test_real_interrupt_hands_blocked_teardown_and_reservation_to_waiter`
delivers a real `_thread.interrupt_main()` once the foreground is in its
wait. On `2e5b7e2`'s `tools.py` the next call was not refused
(`AssertionError: {'ok': True}`). Fixed: 30 of 30 passes. The test drives the
composite-child shape (`_reservation` passed in), so only the inherited
closure can close the reservation.
- **Not addressed.** A waiter already past `finish_owned_cleanup` but not yet
past `done.set()` does not see a late grant of reservation closure. The
expiry path has the same window. It was not measured.
- **Evidence.** Full suite locally (macOS, 3.12): 5964 passed, 107 skipped.
CI: run 39276478002 at `7bc7c21` green on both runners (Ubuntu 5964 passed, 107 skipped; Windows 5956 passed, 115 skipped). n=1.
26 changes: 26 additions & 0 deletions design/prompts.md
Original file line number Diff line number Diff line change
Expand Up @@ -11923,3 +11923,29 @@ report NOT EXERCISED.
- Seven slow attempts before the restart, each stopped at its first control: P1 12:35, 13:03, 13:28; P4 12:55, 12:57, 13:17, 13:30.
- CI at `68535a1`: Ubuntu and Windows green. No product code changed.
- Owed: M2 limbs and the M2 activation rig fact (`R159`).

## design/90 — block 90a (interrupted teardown wait, R155 + R158; PR #61)

**Trace the exception before fixing the mechanism a row names.** R155 named
an `is_alive()` lie after an interrupted join. Reading `_acquire_with_hooks`
showed the interrupt left through `except Exception` untouched and wrote no
pending record, so the refusal had nothing to read in the first place. Fixing
only the named read would have left the defect intact, and a test of the
`Event` alone would have passed.

**Say who can reach a defect before sizing it.** The terminal REPL does not
catch `KeyboardInterrupt`, so a Ctrl-C there ends the process. The refusal is
reached only by a caller that survives the interrupt. The fix is at the tool
because the contract is caller-independent, and the notebook says so instead
of claiming a terminal failure.

**Create the worktree, then the branch in it.** `git worktree add` refuses a
branch checked out in the primary. The coordinator had committed the notebook
on `block-90a` in the primary, and the worktree add failed. Only the `&&`
chain kept the venv and checkout steps out of the primary. Return the primary
to `main` before adding the block's worktree.

**Counts.** Runner: one start turn, no revision, targeted files only
(362 passed, 9 skipped). The new test was watched failing on `2e5b7e2`'s
`tools.py` (`AssertionError: {'ok': True}`) and passed 30 of 30 runs. Full
suite locally: 5964 passed, 107 skipped. CI: run 39276478002 at `7bc7c21` green on both runners (Ubuntu 5964 passed, 107 skipped; Windows 5956 passed, 115 skipped). n=1.
Loading
Loading