diff --git a/CLAUDE.md b/CLAUDE.md index 4049f284..79bd5afa 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1114,5 +1114,6 @@ Create and check whatever MM would ask about *before* the call (design/88 E1). **Never read `is_alive()` after a `Thread.join` that Ctrl-C can interrupt.** On CPython 3.12 an interrupted `join` releases the live thread's state lock and marks it stopped. 88a's supervisor returned mid-run because of it. Wait on an -`Event` the worker sets in its `finally` instead. The pycro-manager teardown -wait still has the hazard (`R155`). +`Event` the worker sets in its `finally` instead. Both acquisition +supervisors now do (design/90 90a): `_wait_acquisition_completion` waits on +the event and the pending-acquisition record carries it, never the `Thread`. diff --git a/design/70-carried-forward-register.md b/design/70-carried-forward-register.md index c17a3718..2af3711b 100644 --- a/design/70-carried-forward-register.md +++ b/design/70-carried-forward-register.md @@ -233,8 +233,8 @@ Sorted by ease, then by importance. `→` names an existing block; do the block, | `R95` | [Does live-mode ownership churn precede a lost terminal notification?](#r95) | MEDIUM | LARGE | | | `R96` | [An unterminated acquisition with every frame accounted is summarised to the user as a success](#r96) | MEDIUM | SMALL | | | `R97` | [The runtime deadline includes Acquisition() construction, and nothing records how much](#r97) | LOW | SMALL | | -| `R155` | [Ctrl-C during a pycro-manager teardown wait can make a live waiter look finished](#r155) | MEDIUM | SMALL | | -| `R158` | [84a's window test reads the final display once, without a wait](#r158) | LOW | SMALL | | +| ~~`R155`~~ | [Ctrl-C during a pycro-manager teardown wait can make a live waiter look finished](#r155) | MEDIUM | SMALL | **90a** | +| ~~`R158`~~ | [84a's window test reads the final display once, without a wait](#r158) | LOW | SMALL | **90a** | **Demo machine** @@ -1416,11 +1416,12 @@ This row comes from the register's table "Absorbed into a block above". Verbatim **On CPython 3.12, a `Thread.join(timeout)` interrupted by `KeyboardInterrupt` releases the live thread's state lock (bpo-45274's cleanup) and marks it stopped, so `is_alive()` returns `False` while the thread still runs.** +- **Status** — **CLOSED 2026-10-19** by `design/90` 90a (PR #61). The defect was wider than this row said: an interrupt in `_acquire_with_hooks`'s wait wrote no pending record at all, so the refusal had nothing to read. The wait is now on an `Event`, the record carries it, and an interrupt hands teardown and reservation closure to the waiter. A real `interrupt_main()` test failed on the pre-fix tree and passed 30 of 30 after. The other sites below were read, not measured, as unreachable by a main-thread Ctrl-C (design/90's table). - **What happened** — found by `design/88` 88a's gate selftest, 2026-10-17: `run_mda` returned mid-run with `outcome=None` while its worker kept going, and the next acquisition was not refused. Reproduced with a standalone probe on 3.12.14. 88a fixed `run_mda` by waiting on a completion `Event` instead of joining. - **Where else** — `_acquire_with_hooks` waits with `_join_acquisition_waiter` and then reads `waiter.is_alive()`, and `execute_tool`'s pending-acquisition refusal reads the same flag, so a terminal Ctrl-C during a `run_timelapse` teardown wait can make the refusal believe teardown finished. 88a's runner also listed `controller._bridge_call`, `AcquisitionDiagnosticWriter.close` and `skill_supervisor`'s pipe-thread joins; those run on background threads, so a main-thread Ctrl-C reaching them is not established. - **What a fix would look like** — the waiter sets an `Event` in its `finally`; foreground waits use `Event.wait`; nothing reads `is_alive()` after an interruptible `join`. Test with a real `_thread.interrupt_main()` delivered inside the wait. - **Where** — LOCAL. -- **Block** — NONE. +- **Block** — `design/90` block 90a. - **Importance** — MEDIUM. Only the terminal path delivers Ctrl-C into that wait; the web UI's Stop does not. - **Effort** — SMALL - **Provenance** — `design/88` 88a, 2026-10-17. @@ -3380,11 +3381,12 @@ the model" as the actual blast radius. **`test_84a_fixture_live_pause_final_drain_and_eof` failed on Windows CI at its last line, `assert displayed(output, 3)`: one read of `display.json`, 0.2 s after the result, where every earlier display check uses a bounded `wait_until`.** +- **Status** — **CLOSED 2026-10-19** by `design/90` 90a (PR #61): the line is now `wait_until(lambda: displayed(output, 3), describe)`. The cause of the one failure is still not identified; the next one will carry its evidence. - **What happened** — PR #59, run 39113232886, `windows-latest`, 2026-10-18, at commit `100fd8b`. Since the previous green run (`5d418ec`) only files under `design/` had changed, so no code under test differed. 1 failed, 5954 passed. This is the test `R153` named, but a different line: R153 was the 8 s first-frame wait at `:1750`, which design/86 86a fixed. - **Cause** — not identified. Either the worker had not yet drawn frame 3 within 0.2 s on the runner, or the read met the fixture's `display.json` replace (the race 86a found for the first frame). The assertion carries no `describe()` evidence, so the log cannot tell which. - **What a fix would look like** — `wait_until(lambda: displayed(output, 3), describe)`, as the earlier checks do, so a failure carries the record, `poll()` and `display.json`. - **Where** — LOCAL, then Windows CI. -- **Block** — NONE. +- **Block** — `design/90` block 90a. - **Importance** — LOW. The test, not the product: the product's result had already been asserted. - **Effort** — SMALL - **Provenance** — `design/88` 88b's CI, 2026-10-18. @@ -3398,6 +3400,6 @@ the model" as the actual blast radius. - **What a fix would look like** — run P1 and P4 on M2 with the activation property as the probed target. `design/89-probes.py` already takes `--device`/`--property`. The `.ps1` launchers do not pass them yet, and the probe's limit check must accept that property. If SMAPpy runs beside them, they can carry `R142`. - **Where** — M2. - **Block** — NONE; 89b's M2 gate depends on it. -- **Importance** — MEDIUM. 89b should not be opened without it, if 89b is wanted. +- **Importance** — MEDIUM. Recommended before 89b, not required: 89b's own M2 gate compares cadence with feedback on and off, so this would only find a problem earlier and more cheaply (operator decision, 2026-10-19). - **Effort** — SMALL - **Provenance** — `design/89` 89a, 2026-10-19. diff --git a/design/90-interrupted-teardown-wait.md b/design/90-interrupted-teardown-wait.md new file mode 100644 index 00000000..9e874e98 --- /dev/null +++ b/design/90-interrupted-teardown-wait.md @@ -0,0 +1,108 @@ +# An interrupted teardown wait must not release the next acquisition + +**Status: CLOSED with PR #61.** One block, 90a, LOCAL gate plus CI. Closes +`R155` and `R158` (design/70). + +## The problem + +**R155.** On CPython 3.12 a `Thread.join(timeout)` interrupted by +`KeyboardInterrupt` releases the live thread's state lock and marks it stopped, +so `is_alive()` returns `False` while the thread still runs (bpo-45274's +cleanup). design/88 88a found this in `run_mda` and fixed it there by waiting on +a completion `Event` (`_wait_mda_completion`). `_acquire_with_hooks`, the +pycro-manager path, still joins: + +- `tools.py` `_join_acquisition_waiter` is `waiter.join(timeout_s)`, called + inside `while waiter.is_alive()`, and the expiry check reads + `waiter.is_alive()` again after it. +- `execute_tool`'s pending-acquisition refusal reads + `pending["waiter"].is_alive()`. + +Reading the code on `0ffa44a` found the defect is wider than the row says. A +`KeyboardInterrupt` raised inside that wait leaves `_acquire_with_hooks` +through its `except Exception` untouched, so **no pending record is written at +all**. The refusal has nothing to read, and the next acquisition starts while +the daemon teardown waiter is still inside `Acquisition.__exit__`. The waiter +also does not inherit reservation closure, which only the expiry path grants. + +**Who can reach it.** The terminal REPL does not catch `KeyboardInterrupt` +around `run_agent`, so today a Ctrl-C in that wait ends the process. What +reaches the refusal is any caller of `execute_tool` that survives the +interrupt, such as a notebook kernel. The contract the refusal states is +caller-independent, so it is fixed at the tool. + +**The other `is_alive()` sites.** The hazard needs an interruptible join *on +the main thread* (CPython delivers `KeyboardInterrupt` only there) followed by a +later `is_alive()` read of the same `Thread`: + +| Site | Thread | Reachable | +|---|---|---| +| `controller._bridge_call` | caller (main in the terminal) | No. The interrupt leaves before `is_alive()` and the `Thread` is local. | +| `AcquisitionDiagnosticWriter.close` (`conversation.py`) | main, from `__main__`'s `finally` | No, same shape. | +| `skill_supervisor` pipe-thread joins | dispatcher or reaper threads; `close()` joins dispatchers but reads no `is_alive()` | No. | +| `skill_store._start_discovery_recheck` | reads a thread nothing in the product joins | No. | + +None is fixed. That a Ctrl-C cannot reach them is read from the code, not +measured. + +**R158.** `tests/test_skill_supervisor.py` +`test_84a_fixture_live_pause_final_drain_and_eof` ends with +`assert displayed(output, 3)`, one read with no evidence. It failed once on +Windows CI (PR #59). Cause not identified. + +## Decision + +1. **One wait helper, on an `Event`.** `_join_acquisition_waiter` and + `_wait_mda_completion` fold into one helper that does `done.wait(timeout_s)`. + `_acquire_with_hooks`'s `finish()` sets a `done` event as the last thing in + its outermost `finally`. The loop and the expiry check read + `done.is_set()`, never `is_alive()`. +2. **The pending record carries the event.** Both writers (`_acquire_with_hooks` + expiry and `run_mda`'s `expired`) store `"done"`. `execute_tool` decides + `teardown_running` from `not done.is_set()`. No `waiter` key survives + (no legacy anchoring). +3. **An interrupt leaving the wait hands teardown to the waiter, as expiry + does.** If a `BaseException` leaves the supervision loop while `done` is + unset, record the pending entry, grant the waiter reservation closure, and + re-raise. No abort is requested. `run_mda`'s cancel-on-interrupt is that + tool's design and is unchanged. +4. **R158:** `wait_until(lambda: displayed(output, 3), describe)`. No deadline + change. + +## Block 90a + +Acceptance: + +- A test drives a hooked or hookless `run_timelapse` through `execute_tool`, + delivers a real `_thread.interrupt_main()` while the foreground is inside its + blocking wait, and asserts that the next acquisition call is refused with + `teardown_running: true` while the teardown waiter is still blocked. After + release it is no longer refused for teardown. Watched failing on `0ffa44a`. +- 88a's interrupt regression in `test_mda_supervision.py` still passes on the + folded helper. +- R158's line uses `wait_until`. +- Full suite green locally and on both CI runners. + +## Run ledger + +| Block | Branch | Start commit | Status | +|-------|--------|--------------|--------| +| 90a | `block-90a` | `0ffa44a` | PR #61, 2026-10-19; local gate passed, CI green on both runners (run 39276478002); one start turn, no revision | + +## What 90a built and measured (2026-10-19) + +- **As decided.** One helper, `_wait_acquisition_completion(done, timeout_s)`, + serves both supervisors. The pending record carries `done`. An interrupt + leaving `_acquire_with_hooks`'s loop writes the record and grants the waiter + reservation closure. +- **Watched failing.** `test_real_interrupt_hands_blocked_teardown_and_reservation_to_waiter` + delivers a real `_thread.interrupt_main()` once the foreground is in its + wait. On `2e5b7e2`'s `tools.py` the next call was not refused + (`AssertionError: {'ok': True}`). Fixed: 30 of 30 passes. The test drives the + composite-child shape (`_reservation` passed in), so only the inherited + closure can close the reservation. +- **Not addressed.** A waiter already past `finish_owned_cleanup` but not yet + past `done.set()` does not see a late grant of reservation closure. The + expiry path has the same window. It was not measured. +- **Evidence.** Full suite locally (macOS, 3.12): 5964 passed, 107 skipped. + CI: run 39276478002 at `7bc7c21` green on both runners (Ubuntu 5964 passed, 107 skipped; Windows 5956 passed, 115 skipped). n=1. diff --git a/design/prompts.md b/design/prompts.md index e07e7e4a..29e2e493 100644 --- a/design/prompts.md +++ b/design/prompts.md @@ -11923,3 +11923,29 @@ report NOT EXERCISED. - Seven slow attempts before the restart, each stopped at its first control: P1 12:35, 13:03, 13:28; P4 12:55, 12:57, 13:17, 13:30. - CI at `68535a1`: Ubuntu and Windows green. No product code changed. - Owed: M2 limbs and the M2 activation rig fact (`R159`). + +## design/90 — block 90a (interrupted teardown wait, R155 + R158; PR #61) + +**Trace the exception before fixing the mechanism a row names.** R155 named +an `is_alive()` lie after an interrupted join. Reading `_acquire_with_hooks` +showed the interrupt left through `except Exception` untouched and wrote no +pending record, so the refusal had nothing to read in the first place. Fixing +only the named read would have left the defect intact, and a test of the +`Event` alone would have passed. + +**Say who can reach a defect before sizing it.** The terminal REPL does not +catch `KeyboardInterrupt`, so a Ctrl-C there ends the process. The refusal is +reached only by a caller that survives the interrupt. The fix is at the tool +because the contract is caller-independent, and the notebook says so instead +of claiming a terminal failure. + +**Create the worktree, then the branch in it.** `git worktree add` refuses a +branch checked out in the primary. The coordinator had committed the notebook +on `block-90a` in the primary, and the worktree add failed. Only the `&&` +chain kept the venv and checkout steps out of the primary. Return the primary +to `main` before adding the block's worktree. + +**Counts.** Runner: one start turn, no revision, targeted files only +(362 passed, 9 skipped). The new test was watched failing on `2e5b7e2`'s +`tools.py` (`AssertionError: {'ok': True}`) and passed 30 of 30 runs. Full +suite locally: 5964 passed, 107 skipped. CI: run 39276478002 at `7bc7c21` green on both runners (Ubuntu 5964 passed, 107 skipped; Windows 5956 passed, 115 skipped). n=1. diff --git a/microclaw/tools.py b/microclaw/tools.py index 4d169f4d..02ec86b3 100644 --- a/microclaw/tools.py +++ b/microclaw/tools.py @@ -209,8 +209,10 @@ def _acquisition_monotonic() -> float: return timing_clock() -def _join_acquisition_waiter(waiter: threading.Thread, timeout_s: float) -> None: - waiter.join(timeout_s) +def _wait_acquisition_completion(done, timeout_s): + # Event.wait does not touch Thread's state lock. An interrupted join can + # falsely mark a live worker stopped on CPython (bpo-45274). + return done.wait(timeout_s) def _runtime_ceiling_s( @@ -4969,47 +4971,51 @@ def finish_owned_cleanup() -> list[str]: }) acq.acquire(events) + done = threading.Event() outcome: dict[str, Any] = {} def finish() -> None: - with _acquisition_diagnostic_context(diagnostic_context): - try: + try: + with _acquisition_diagnostic_context(diagnostic_context): try: - _emit_acquisition_diagnostic({ - "type": "acquisition_mark_finished", - "dataset_path": dataset_path, - }, lifecycle=True, publish=False) - acq.__exit__(None, None, None) - except BaseException as exc: - outcome["exc"] = exc + try: + _emit_acquisition_diagnostic({ + "type": "acquisition_mark_finished", + "dataset_path": dataset_path, + }, lifecycle=True, publish=False) + acq.__exit__(None, None, None) + except BaseException as exc: + outcome["exc"] = exc + finally: + teardown_timing["acquisition"]["end_s"] = timing_clock() + failures = finish_owned_cleanup() + if failures: + prior = outcome.get("exc") + detail = "; ".join(failures) + if prior is not None: + prior.add_note(detail) + else: + outcome["exc"] = RuntimeError(detail) + flag = getattr(ctrl, "_microclaw_unterminated_acquisition", None) + if isinstance(flag, dict) and outcome.get("exc") is not None: + late = outcome["exc"] + _emit_acquisition_diagnostic({ + "type": "acquisition_diagnostic", + "message": f"Late teardown/cleanup failure: {type(late).__name__}: {late}", + "dataset_path": dataset_path, + }) finally: - teardown_timing["acquisition"]["end_s"] = timing_clock() - failures = finish_owned_cleanup() - if failures: - prior = outcome.get("exc") - detail = "; ".join(failures) - if prior is not None: - prior.add_note(detail) - else: - outcome["exc"] = RuntimeError(detail) - flag = getattr(ctrl, "_microclaw_unterminated_acquisition", None) - if isinstance(flag, dict) and outcome.get("exc") is not None: - late = outcome["exc"] + flag = getattr(ctrl, "_microclaw_unterminated_acquisition", None) + if isinstance(flag, dict): + flag["teardown_running"] = False + if analysis_job is not None: + analysis_job.writer_finished() _emit_acquisition_diagnostic({ - "type": "acquisition_diagnostic", - "message": f"Late teardown/cleanup failure: {type(late).__name__}: {late}", + "type": "acquisition_teardown_completion", "dataset_path": dataset_path, - }) - finally: - flag = getattr(ctrl, "_microclaw_unterminated_acquisition", None) - if isinstance(flag, dict): - flag["teardown_running"] = False - if analysis_job is not None: - analysis_job.writer_finished() - _emit_acquisition_diagnostic({ - "type": "acquisition_teardown_completion", - "dataset_path": dataset_path, - "frames_accounted": frame_state["count"], - }, lifecycle=True, publish=False) + "frames_accounted": frame_state["count"], + }, lifecycle=True, publish=False) + finally: + done.set() waiter = threading.Thread(target=finish, name="microclaw-acq-teardown", daemon=True) waiter_started = True @@ -5017,89 +5023,99 @@ def finish() -> None: error_deadline = None engine_exc = None last_metrics_progress = None - while waiter.is_alive(): - tick = _acquisition_monotonic() - if analysis_job is not None and (last_metrics_progress is None or tick - last_metrics_progress >= 1): - last_metrics_progress = tick - handle = analysis_job.handle - if handle is not None: - metrics = handle.metrics_snapshot(for_progress=True) - if metrics.get("declared"): - with frame_lock: - count, phase = frame_state["count"], progress_state["phase"] - _emit_acquisition_diagnostic(_analysis_metrics_progress( - analysis_job, metrics, dataset_path, count, - plan.frames if plan is not None else None, phase)) - # pycro-manager 1.0.2 abort() stores the fatal exception here. - # This private API and _dataset_disk_location below must both be - # re-verified whenever pycro-manager is upgraded. - observed = getattr(acq, "_exception", None) - if observed is not None and error_deadline is None: - engine_exc = observed - error_deadline = _acquisition_monotonic() + ERROR_TEARDOWN_GRACE_S - _emit_acquisition_diagnostic({ - "type": "acquisition_diagnostic", - "message": f"{type(observed).__name__}: {observed}", - "dataset_path": dataset_path, - "engine_exception": f"{type(observed).__name__}: {observed}", - }, lifecycle=True) - _join_acquisition_waiter(waiter, _ACQUISITION_POLL_S) - now = _acquisition_monotonic() - with frame_lock: - quiet_window, quiet_term = _stall_quiet_s( - frame_state["largest_gap"], policy, - ) - quiet = now - frame_state["last_saved"] >= quiet_window - frames_accounted = frame_state["count"] - error_expired = error_deadline is not None and now >= error_deadline - runtime_expired = now >= runtime_deadline and quiet - if waiter.is_alive() and (error_expired or runtime_expired): - # A composite caller normally owns a shared reservation across - # positions. Expiry ends that composite immediately, so the - # still-live waiter inherits final closure after its callbacks. + try: + while not done.is_set(): + tick = _acquisition_monotonic() + if analysis_job is not None and (last_metrics_progress is None or tick - last_metrics_progress >= 1): + last_metrics_progress = tick + handle = analysis_job.handle + if handle is not None: + metrics = handle.metrics_snapshot(for_progress=True) + if metrics.get("declared"): + with frame_lock: + count, phase = frame_state["count"], progress_state["phase"] + _emit_acquisition_diagnostic(_analysis_metrics_progress( + analysis_job, metrics, dataset_path, count, + plan.frames if plan is not None else None, phase)) + # pycro-manager 1.0.2 abort() stores the fatal exception here. + # This private API and _dataset_disk_location below must both be + # re-verified whenever pycro-manager is upgraded. + observed = getattr(acq, "_exception", None) + if observed is not None and error_deadline is None: + engine_exc = observed + error_deadline = _acquisition_monotonic() + ERROR_TEARDOWN_GRACE_S + _emit_acquisition_diagnostic({ + "type": "acquisition_diagnostic", + "message": f"{type(observed).__name__}: {observed}", + "dataset_path": dataset_path, + "engine_exception": f"{type(observed).__name__}: {observed}", + }, lifecycle=True) + _wait_acquisition_completion(done, _ACQUISITION_POLL_S) + now = _acquisition_monotonic() + with frame_lock: + quiet_window, quiet_term = _stall_quiet_s( + frame_state["largest_gap"], policy, + ) + quiet = now - frame_state["last_saved"] >= quiet_window + frames_accounted = frame_state["count"] + error_expired = error_deadline is not None and now >= error_deadline + runtime_expired = now >= runtime_deadline and quiet + if not done.is_set() and (error_expired or runtime_expired): + # A composite caller normally owns a shared reservation across + # positions. Expiry ends that composite immediately, so the + # still-live waiter inherits final closure after its callbacks. + waiter_must_close_reservation = True + expired_bound = "error_grace" if error_expired else policy.bound_name + bound_s = ERROR_TEARDOWN_GRACE_S if error_expired else runtime_bound + pending = { + "done": done, "teardown_running": True, + "camera_sequence_running": None, "dataset_path": dataset_path, + } + setattr(ctrl, "_microclaw_unterminated_acquisition", pending) + failure = AcquisitionUnterminated( + dataset_path=dataset_path, + frames_planned=(plan.frames if plan is not None else + reservation.plan.frames if reservation is not None else None), + frames_accounted=frames_accounted, engine_exception=engine_exc, + camera_sequence_running=None, teardown_running=True, + expired_bound=expired_bound, bound_s=bound_s, + fallback_ceiling=fallback, + phase=("finalizing" if policy.terminal_frames is not None + and frames_accounted >= policy.terminal_frames + else "acquiring_or_notifying"), + cadence=(_gap_summary_payload(cadence_summary) + if report_cadence else None), + ) + camera = _camera_sequence_running(ctrl) + pending["camera_sequence_running"] = camera + failure.camera_sequence_running = camera + failure.diagnostic_persisted = _emit_acquisition_diagnostic({ + "type": "acquisition_timeout", + "phase": failure.phase, + "dataset_path": dataset_path, + "frames_planned": (plan.frames if plan is not None else + reservation.plan.frames if reservation is not None else None), + "frames_accounted": frames_accounted, + "estimated_bytes": plan.estimated_bytes if plan is not None else None, + "active_bound": expired_bound, + "active_bound_s": bound_s, + "active_bound_term": ("error_grace" if error_expired else runtime_term), + "quiet_bound_s": quiet_window, + "quiet_bound_term": quiet_term, + "camera_sequence_running": camera, + "engine_exception": (f"{type(engine_exc).__name__}: {engine_exc}" + if engine_exc is not None else None), + }, lifecycle=True, acknowledge=True, publish=False) + raise failure + except BaseException as exc: + if not isinstance(exc, AcquisitionUnterminated) and not done.is_set(): waiter_must_close_reservation = True - expired_bound = "error_grace" if error_expired else policy.bound_name - bound_s = ERROR_TEARDOWN_GRACE_S if error_expired else runtime_bound - pending = { - "waiter": waiter, "teardown_running": True, + ctrl._microclaw_unterminated_acquisition = { + "done": done, "teardown_running": True, "camera_sequence_running": None, "dataset_path": dataset_path, } - setattr(ctrl, "_microclaw_unterminated_acquisition", pending) - failure = AcquisitionUnterminated( - dataset_path=dataset_path, - frames_planned=(plan.frames if plan is not None else - reservation.plan.frames if reservation is not None else None), - frames_accounted=frames_accounted, engine_exception=engine_exc, - camera_sequence_running=None, teardown_running=True, - expired_bound=expired_bound, bound_s=bound_s, - fallback_ceiling=fallback, - phase=("finalizing" if policy.terminal_frames is not None - and frames_accounted >= policy.terminal_frames - else "acquiring_or_notifying"), - cadence=(_gap_summary_payload(cadence_summary) - if report_cadence else None), - ) - camera = _camera_sequence_running(ctrl) - pending["camera_sequence_running"] = camera - failure.camera_sequence_running = camera - failure.diagnostic_persisted = _emit_acquisition_diagnostic({ - "type": "acquisition_timeout", - "phase": failure.phase, - "dataset_path": dataset_path, - "frames_planned": (plan.frames if plan is not None else - reservation.plan.frames if reservation is not None else None), - "frames_accounted": frames_accounted, - "estimated_bytes": plan.estimated_bytes if plan is not None else None, - "active_bound": expired_bound, - "active_bound_s": bound_s, - "active_bound_term": ("error_grace" if error_expired else runtime_term), - "quiet_bound_s": quiet_window, - "quiet_bound_term": quiet_term, - "camera_sequence_running": camera, - "engine_exception": (f"{type(engine_exc).__name__}: {engine_exc}" - if engine_exc is not None else None), - }, lifecycle=True, acknowledge=True, publish=False) - raise failure + raise + if "exc" in outcome: raise outcome["exc"] if analysis_job is not None: @@ -12163,12 +12179,6 @@ def _mda_prepare_save_dir(path): raise SafetyViolation(f"MDA save directory is not writable: {path}") -def _wait_mda_completion(done, timeout_s): - # Event.wait does not touch Thread's state lock. An interrupted join can - # falsely mark a live worker stopped on CPython (bpo-45274). - return done.wait(timeout_s) - - def _mda_poll_wait(seconds): threading.Event().wait(seconds) @@ -12419,7 +12429,7 @@ def worker(): "active_bound_s": runtime_s, **state["supervision"]}) if expired is not None: - expired(state, waiter) + expired(state, done) raise RuntimeError("MDA unterminated: " + state["phase"]) except KeyboardInterrupt: request_abort.set() @@ -12599,9 +12609,9 @@ def notify(event): acknowledge=event["type"] == "acquisition_timeout", publish=event["type"] != "acquisition_timeout") runtime, fallback, term = _runtime_ceiling_s(plan, DEFAULT) - def expired(state, waiter): + def expired(state, done): ctrl._microclaw_unterminated_acquisition = { - "waiter": waiter, "teardown_running": True, + "done": done, "teardown_running": True, "camera_sequence_running": None, "dataset_path": state["dataset_path"], } failure = AcquisitionUnterminated( @@ -12619,7 +12629,7 @@ def expired(state, waiter): manager, start, current, runtime_s=runtime, startup_s=plan.exposure_ms_per_frame / 1000.0, interval_s=interval_s, quiet_window=lambda gap: _stall_quiet_s(gap, DEFAULT)[0], - clock=_acquisition_monotonic, wait=_wait_mda_completion, + clock=_acquisition_monotonic, wait=_wait_acquisition_completion, poll_s=_ACQUISITION_POLL_S, cancel=getattr(_ACQUISITION_EVENT_CONTEXT, "cancel", None), account=reservation.commit_frame, close=reservation.close, @@ -12795,8 +12805,8 @@ def encode_result(result): if getattr(fn, "_microclaw_acquisition_entry_point", False): pending = getattr(ctrl, "_microclaw_unterminated_acquisition", None) if isinstance(pending, dict): - waiter = pending.get("waiter") - teardown = bool(waiter.is_alive()) if waiter is not None else bool( + done = pending.get("done") + teardown = not done.is_set() if done is not None else bool( pending.get("teardown_running") ) # Never queue a bridge probe behind a live, possibly blocked worker. diff --git a/tests/test_acquisition_order.py b/tests/test_acquisition_order.py index 8005873d..735b5e10 100644 --- a/tests/test_acquisition_order.py +++ b/tests/test_acquisition_order.py @@ -300,7 +300,7 @@ def blocked_exit(self,*_): release.set() pending=getattr(rig.ctrl,'_microclaw_unterminated_acquisition',None) if isinstance(pending,dict): - pending['waiter'].join(1) + pending['done'].wait(1) assert rig.reservations[0]._closed diff --git a/tests/test_bounded_acquisition_wait.py b/tests/test_bounded_acquisition_wait.py index 4cfbb5ed..7b870096 100644 --- a/tests/test_bounded_acquisition_wait.py +++ b/tests/test_bounded_acquisition_wait.py @@ -67,11 +67,11 @@ def __init__(self, *, step=100.0, saved_at=(), release_at=None, limit=None): def __call__(self): return self.now - def join(self, waiter, _timeout): + def wait(self, done, _timeout): target = self.now + self.step if self.limit is not None and target > self.limit: BlockingAcquisition.release.set() - waiter.join(1) + done.wait(1) raise AssertionError("supervisor exceeded test delivery ceiling") while self.saved_at and self.saved_at[0] <= target: self.now = self.saved_at.pop(0) @@ -79,12 +79,12 @@ def join(self, waiter, _timeout): self.now = target if self.release_at is not None and self.now >= self.release_at: BlockingAcquisition.release.set() - waiter.join(0.01) + done.wait(0.01) def _install_clock(monkeypatch, clock): monkeypatch.setattr(tools, "_acquisition_monotonic", clock) - monkeypatch.setattr(tools, "_join_acquisition_waiter", clock.join) + monkeypatch.setattr(tools, "_wait_acquisition_completion", clock.wait) def test_fatal_engine_error_returns_complete_unterminated_result_and_waiter_owns_cleanup( @@ -412,7 +412,7 @@ def authorize(_ctrl, _guard, plan, **kwargs): def wait_mda(done, _): assert manager.entered.wait(5) clock.now += 1000 - monkeypatch.setattr(tools, "_wait_mda_completion", wait_mda) + monkeypatch.setattr(tools, "_wait_acquisition_completion", wait_mda) if with_analysis: inputs[name]['analysis'] = dict(adapter='fixture-lab/package:observe_dataset', @@ -444,14 +444,14 @@ def wait_mda(done, _): if name == "run_mda": assert manager.calls.count("start") == 1 manager.release.set() - ctrl._microclaw_unterminated_acquisition['waiter'].join(5) + ctrl._microclaw_unterminated_acquisition['done'].wait(5) BlockingAcquisition.release.set() if with_analysis: assert len(result['analysis']['jobs']) == (2 if looping else 1) assert [job['dataset'] for job in result['analysis']['jobs']] == [ f'/data/run_{i + 1}' for i in range(len(jobs))] - ctrl._microclaw_unterminated_acquisition['waiter'].join(5) - assert not ctrl._microclaw_unterminated_acquisition['waiter'].is_alive() + ctrl._microclaw_unterminated_acquisition['done'].wait(5) + assert ctrl._microclaw_unterminated_acquisition['done'].is_set() messages = [entry['message'] for entry in jobs[-1].handle.record()['notifications']] assert [(message['type'], message.get('outcome'), message.get('writer'), message.get('state')) for message in messages] == [ @@ -630,9 +630,8 @@ def test_measured_camera_state_selects_distinct_recovery_text(camera): ) def test_every_marked_acquisition_entry_point_is_refused_while_teardown_lives(name): ctrl = _ctrl(False) - waiter = MagicMock() - waiter.is_alive.return_value = True - ctrl._microclaw_unterminated_acquisition = {"waiter": waiter} + done = threading.Event() + ctrl._microclaw_unterminated_acquisition = {"done": done} result = json.loads(tools.execute_tool(name, {}, ctrl, _guard())) assert result["acquisition"] == "refused" @@ -643,9 +642,9 @@ def test_acquisition_refusal_lifts_only_when_camera_and_teardown_are_clear(monke def run(ctrl, guard): return {"ok": True} ctrl = _ctrl(False) - waiter = MagicMock() - waiter.is_alive.return_value = False - ctrl._microclaw_unterminated_acquisition = {"waiter": waiter} + done = threading.Event() + done.set() + ctrl._microclaw_unterminated_acquisition = {"done": done} assert json.loads(tools.execute_tool("run", {}, ctrl, _guard(), {"run": run})) == {"ok": True} assert not hasattr(ctrl, "_microclaw_unterminated_acquisition") @@ -982,7 +981,7 @@ def test_short_fixed_timeout_survives_missing_notification(monkeypatch, saved): BlockingAcquisition.release.set() flag = getattr(ctrl, "_microclaw_unterminated_acquisition", None) if isinstance(flag, dict): - flag["waiter"].join(1) + flag["done"].wait(1) confirm.assert_not_called() policy = tools.SHORT_FIXED expiry = max(plan.estimated_duration_s * 1.5, @@ -1171,7 +1170,7 @@ def test_short_waiter_completion_owns_cleanup_once_and_refusal_lifetime(monkeypa finally: BlockingAcquisition.release.set() flag = getattr(ctrl, "_microclaw_unterminated_acquisition", None) - if isinstance(flag, dict): flag["waiter"].join(1) + if isinstance(flag, dict): flag["done"].wait(1) reservation.close.assert_called_once() assert json.loads(tools.execute_tool("run", {}, ctrl, _guard(), {"run": run})) == {"available": True} assert not hasattr(ctrl, "_microclaw_unterminated_acquisition") @@ -1386,3 +1385,85 @@ def __exit__(self, *args): assert len(frames) == 1 ctrl.refresh_gui.assert_not_called() assert not any("GUI controls can lag" in item.get("message", "") for item in received) + + +def test_real_interrupt_hands_blocked_teardown_and_reservation_to_waiter(monkeypatch): + import _thread + import sys + + entered = threading.Event() + interrupt_sent = threading.Event() + stop = threading.Event() + foreground = threading.get_ident() + + class InterruptedAcquisition(BlockingAcquisition): + def __exit__(self, *exc): + entered.set() + super().__exit__(*exc) + + def interrupt_inside_wait(): + if not entered.wait(5): + return + for _ in range(10000): + if stop.is_set(): + return + frame = sys._current_frames()[foreground] + frames = [] + while frame: + frames.append(frame) + frame = frame.f_back + # Thread.start also waits on an Event. The teardown may have + # entered before start returns; interrupt only the supervision wait. + if any(f.f_code.co_filename == threading.__file__ + and f.f_code.co_name == 'start' for f in frames): + stop.wait(0.001) + continue + if any(f.f_code.co_filename == threading.__file__ + and f.f_code.co_name in ('wait', '_wait_for_tstate_lock') + for f in frames): + interrupt_sent.set() + _thread.interrupt_main() + return + stop.wait(0.001) + + monkeypatch.setattr(tools, 'Acquisition', InterruptedAcquisition) + monkeypatch.setattr(tools, '_ACQUISITION_POLL_S', 0.05) + monkeypatch.setattr('microclaw.authorization.authorize_path', lambda *_: None) + monkeypatch.setattr(tools, '_build_acquisition_events', lambda **_: [{}]) + reservation = AcquisitionLedger().reserve(MagicMock(), AcquisitionPlan(1, 1, 1, 1)) + ctrl = _ctrl(False) + guard = _guard() + + @_entry + def next_acquisition(ctrl, guard): + return {'ok': True} + + registry = {'next_acquisition': next_acquisition} + interrupter = threading.Thread(target=interrupt_inside_wait, daemon=True) + interrupter.start() + pending = None + try: + with pytest.raises(KeyboardInterrupt): + tools.execute_tool('run_timelapse', { + 'n_frames': 1, 'interval_s': 0, 'save_dir': '/data', + '_reservation': reservation, + }, ctrl, guard) + assert interrupt_sent.is_set() + assert not InterruptedAcquisition.release.is_set() + refused = json.loads(tools.execute_tool('next_acquisition', {}, ctrl, guard, registry)) + assert refused.get('acquisition') == 'refused', refused + assert refused['teardown_running'] is True + pending = ctrl._microclaw_unterminated_acquisition + assert reservation.ledger.in_flight + ctrl.core.is_sequence_running.assert_not_called() + InterruptedAcquisition.release.set() + assert pending['done'].wait(5) + assert not reservation.ledger.in_flight + assert json.loads(tools.execute_tool('next_acquisition', {}, ctrl, guard, registry)) == {'ok': True} + ctrl.core.is_sequence_running.assert_called_once_with() + finally: + stop.set() + InterruptedAcquisition.release.set() + interrupter.join(5) + if pending is not None: + assert pending['done'].wait(5) diff --git a/tests/test_mda_analysis.py b/tests/test_mda_analysis.py index 5eb3c8ee..302bb3f2 100644 --- a/tests/test_mda_analysis.py +++ b/tests/test_mda_analysis.py @@ -127,7 +127,7 @@ def pause(_): monkeypatch.setattr(manager, 'is_acquisition_running', original) manager.release.set() pending = getattr(rig[0], '_microclaw_unterminated_acquisition', {}) - if pending: pending['waiter'].join(5) + if pending: pending['done'].wait(5) def test_no_analysis_no_observer_work(rig, observer, monkeypatch): diff --git a/tests/test_mda_supervision.py b/tests/test_mda_supervision.py index 75031f7c..e708a14f 100644 --- a/tests/test_mda_supervision.py +++ b/tests/test_mda_supervision.py @@ -56,7 +56,7 @@ def wait(done, _): # Synchronize with the fake bridge entry rather than measuring throughput. assert manager.entered.wait(5) now[0] += step - monkeypatch.setattr(tools, '_wait_mda_completion', wait) + monkeypatch.setattr(tools, '_wait_acquisition_completion', wait) @pytest.mark.parametrize('block', ['start', 'poll', 'abort']) @@ -69,8 +69,8 @@ def test_hung_call_keeps_one_worker_and_refuses_without_probe(rig, monkeypatch, result = json.loads(tools.execute_tool('run_mda', {'settings': settings}, ctrl, guard, cancel=cancel)) assert result.get('acquisition') == 'unterminated', result assert result['phase'] == {'start': 'starting', 'poll': 'acquiring', 'abort': 'stopping'}[block] - waiter = ctrl._microclaw_unterminated_acquisition['waiter'] - assert waiter.is_alive() + done = ctrl._microclaw_unterminated_acquisition['done'] + assert not done.is_set() before = list(manager.calls) refused = json.loads(tools.execute_tool('run_timelapse', {}, ctrl, guard)) assert refused['acquisition'] == 'refused' @@ -79,8 +79,8 @@ def test_hung_call_keeps_one_worker_and_refuses_without_probe(rig, monkeypatch, assert manager.calls.count('start') == 1 if block == 'start': assert 'poll' not in manager.calls if block == 'poll': assert manager.calls.count('poll') == 1 - manager.release.set(); waiter.join(5) - assert not waiter.is_alive() + manager.release.set(); done.wait(5) + assert done.is_set() if block == 'abort': assert manager.calls.count('abort') == 1 @@ -97,7 +97,7 @@ def wait(done, seconds): cancel.set() raise KeyboardInterrupt done.wait(5) - monkeypatch.setattr(tools, '_wait_mda_completion', wait) + monkeypatch.setattr(tools, '_wait_acquisition_completion', wait) # Ensure a poll after the first blocked call, so interrupt reaches worker. manager.observations = [(True, 0, False, False)] * 10 + [(False, 2, True, True)] result = json.loads(tools.execute_tool('run_mda', {'settings': settings}, ctrl, guard, @@ -147,16 +147,16 @@ def wait(done, _): assert tick.wait(5); tick.clear() now[0] += 1000 # leave worker parked at expiry so pending ownership is observable - monkeypatch.setattr(tools, '_wait_mda_completion', wait) + monkeypatch.setattr(tools, '_wait_acquisition_completion', wait) result = json.loads(tools.execute_tool('run_mda', {'settings': settings}, ctrl, guard)) assert result.get('acquisition') == 'unterminated', result assert result['phase'] == 'writer_finalizing' assert result['writer_completion']['state'] == 'not_reached' assert result['outcome'] == 'unterminated' - waiter = ctrl._microclaw_unterminated_acquisition['waiter'] + done = ctrl._microclaw_unterminated_acquisition['done'] assert tools._acquisition_ledger(ctrl).in_flight - release.set(); advance.set(); waiter.join(5) - assert not waiter.is_alive() + release.set(); advance.set(); done.wait(5) + assert done.is_set() assert not tools._acquisition_ledger(ctrl).in_flight @@ -183,7 +183,7 @@ def wait(done, _): done.wait(5) joins[0] += 1 monkeypatch.setattr(tools, '_mda_poll_wait', pause) - monkeypatch.setattr(tools, '_wait_mda_completion', wait) + monkeypatch.setattr(tools, '_wait_acquisition_completion', wait) monkeypatch.setattr(tools, '_acquisition_monotonic', lambda: now[0]) try: result = tools.run_mda(ctrl, guard, settings=settings) @@ -489,7 +489,7 @@ def wait(done, _): if now[0] < 1200: advance.set() monkeypatch.setattr(tools, '_mda_poll_wait', pause) monkeypatch.setattr(tools, '_acquisition_monotonic', lambda: now[0]) - monkeypatch.setattr(tools, '_wait_mda_completion', wait) + monkeypatch.setattr(tools, '_wait_acquisition_completion', wait) # Use a short policy seam so all 12 diagnostics are exercised before expiry. monkeypatch.setattr(tools, '_stall_quiet_s', lambda *args: (1200, 'test')) try: @@ -505,7 +505,7 @@ def wait(done, _): manager.observation = (False, 2, True, True) advance.set() pending = getattr(ctrl, '_microclaw_unterminated_acquisition', {}) - if pending: pending['waiter'].join(5) + if pending: pending['done'].wait(5) def test_interrupt_from_watchdog_clock_requests_abort_and_keeps_supervising(rig, monkeypatch): @@ -526,7 +526,7 @@ def wait(done, _): else: manager.release.set(); done.wait(5) monkeypatch.setattr(tools, '_acquisition_monotonic', clock) - monkeypatch.setattr(tools, '_wait_mda_completion', wait) + monkeypatch.setattr(tools, '_wait_acquisition_completion', wait) try: try: result = tools.run_mda(ctrl, guard, settings=settings) @@ -550,8 +550,7 @@ def test_real_interrupt_inside_wait_preserves_live_worker_and_cancels(rig, monke worker = [] observed = [] foreground = threading.get_ident() - original_wait = getattr(tools, '_wait_mda_completion', lambda event, seconds: event.wait(seconds)) - original_join = tools._join_acquisition_waiter + original_wait = getattr(tools, '_wait_acquisition_completion', lambda event, seconds: event.wait(seconds)) first = [True] def pause(_): if not first[0]: return @@ -581,10 +580,7 @@ def interrupted_wait(call): observed.append(alive) assert alive, 'Interrupted Thread.join falsely marked a still-running MDA worker stopped' raise - # The old join seam is used only when this regression is run against f79ed81. - monkeypatch.setattr(tools, '_join_acquisition_waiter', - lambda waiter, seconds: interrupted_wait(lambda: original_join(waiter, seconds))) - monkeypatch.setattr(tools, '_wait_mda_completion', + monkeypatch.setattr(tools, '_wait_acquisition_completion', lambda done, seconds: interrupted_wait(lambda: original_wait(done, seconds)), raising=False) monkeypatch.setattr(tools, '_mda_poll_wait', pause) monkeypatch.setattr(tools, '_ACQUISITION_POLL_S', 0.05) diff --git a/tests/test_skill_supervisor.py b/tests/test_skill_supervisor.py index b20cfc68..d7f81116 100644 --- a/tests/test_skill_supervisor.py +++ b/tests/test_skill_supervisor.py @@ -1783,7 +1783,7 @@ def describe(): process = sup._windows[h.job_id][1] time.sleep(0.2) assert process.poll() is None # stdin EOF after result leaves display alive. - assert displayed(output, 3) + wait_until(lambda: displayed(output, 3), describe) finally: writer.close()