feat: Record + Z-Stack acquisition mode

[hongquanli]

Summary

Adds a new Record + Z-Stack acquisition mode (its own tab, next to Wellplate Multipoint, gated by ENABLE_RECORDING). It walks selected wells × a per-well FOV grid × Nt time points and, at each FOV, runs two phases in sequence:

1. Recording — a continuous single-channel video: T = round(fps × duration) frames at one Z plane, saved as a per-FOV Zarr (T, 1, 1, Y, X).
2. Z-stack — a multi-channel stack over a Z range, saved as OME-Zarr (Nt, C, NZ, Y, X) per FOV.

Both phases are positioned in Z by offsets relative to a reference plane (laser reflection AF when enabled+referenced, otherwise the current Z). Each phase is independently enable-able.

Stacked on #691

This PR is based on feat/in-process-zarr-saving (#691) and shows only its own commits; merge #691 first, then retarget this one to master.

Why: every recording uses tensorstore in the main process (RecordingWriter), and tensorstore aborts any fork()ed child of a process that has used it. The z-stack used to be saved by a forked JobRunner subprocess that was forked at the start of the previous acquisition, so from the third Record + Z-Stack acquisition of a session on, the subprocess died on its first write and the z-stack was silently lost (reproduced in simulation: runs 1–2 complete, 3–4 lost; the recording itself was intact). #691 saves Zarr v3 in the acquisition process; on top of it this PR's last commit makes the z-stack use that InProcessZarrRunner, drops the record controller's pre-warm slot, polls save results after every FOV (a failed write ends the run as an error with the store sealed aborted), and removes the USE_MULTIPROCESSING=False test workarounds that only existed to dodge the abort. test_three_record_zstack_acquisitions_all_sealed pins the scenario; the full suite now runs in one process without the old stall at the worker smoke test.

Architecture

• MultiPointWorkerBase — behavior-preserving refactor lifting the shared capture mechanics (channel apply, single-frame capture, frame callback + SaveZarrJob dispatch, backpressure, abort/progress) out of MultiPointWorker. The acquisition-watchdog hooks (_abort_due_to_error, _compute_end_reason, the run-state writer) live on the base; both controllers start the run.json breadcrumb through one start_run_state_writer helper and both workers end it with the same reasons, so Record + Z-Stack runs are visible to the watchdog like multipoint runs.
• StreamingCapture (control/core/streaming_capture.py) — the recording primitive: a continuous frame source → bounded queue (count- and byte-capped) → writer thread → direct ZarrWriter. Generic frame_source/frame_router/stop_condition seam, shaped so a future hardware-sequenced source drops in.
• RecordZStackWorker / RecordZStackController — thin t×well×FOV loop reusing the inherited mechanics; recording via StreamingCapture, z-stack via SaveZarrJob on the InProcessZarrRunner from feat(acquisition): save Zarr v3 in the acquisition process #691 (no save subprocess). Timepoints are paced on the absolute t0 + k·dt grid with grid-preserving skip (mirrors MultiPointWorker); each experiment dir gets the acquisition_channels.yaml snapshot and, once every recording store is sealed, the .done marker like every multipoint acquisition.
• RecordZStackMultiPointWidget + gui_hcs tab wiring. Per-well FOV grid configurable inline and rebuilt on tab entry/well clicks; channels editable inline with Copy-from-Live; channel combos refresh on objective/profile/config changes. XY/Time controls mirror WellplateMultiPointWidget's tabbed row (see below).
• Camera frame-rate hint — AbstractCamera.set_frame_rate(fps) (no-op default; ToupTek PRECISE_FRAMERATE; simulated camera honors it). The recording sizes its dataset, pacing, and time_increment_s from the achievable rate the camera reports, with software downsampling as the portable safety net.

Widget UI polish

• XY/Time tabbed row, mirroring WellplateMultiPointWidget (checkbox + combo in a frame that highlights orange/green when active), skipping Z since the widget's own Z-Stack phase group already covers z-stacking:
  • combobox_xy_mode now offers two real modes: Select Wells (today's tile-a-grid-over-selected-wells behavior, default) and Current Position (single FOV at the live stage position, bypassing well selection entirely — validate() no longer requires a well selection in this mode). Unchecking XY forces Current Position and disables the combo; re-checking restores the previous mode.
  • checkbox_time wraps Nt/dt: unchecked forces a single timepoint and hides the controls, restoring the previous Nt/dt on re-check.
  • checkbox_laser_af moved into this row as a plain checkbox, matching how WellplateMultiPointWidget places its own Laser AF toggle.
• Channel-add seeds from configured settings: the z-stack "+ Add" button and the recording-channel combo (initial row and on selection change) now seed exposure/gain/illumination from the channel's own config via liveController.get_channels(), instead of a hardcoded 50 ms / 0 gain / 50%.
• Layout: removed the channel tables' fixed 530px width — it fought against whatever width the main window happened to lock in for this tab (centralWidget.setFixedWidth(minimumSizeHint()) in gui_hcs.py sizes the whole app once at startup from whichever tab is active then) — in favor of a responsive fill plus tightened field widths, so the panel no longer needs a horizontal scrollbar. The Z-Stack channel table now spans the group's full width to match the Recording table, and z-stack channel names get a tooltip with the full name since the Channel column truncates.
• ~20 new tests; caught in review and fixed with a reproducing test first: refresh_channel_list() was silently resetting the user's manually-edited recording exposure/gain/illumination back to the channel's base config (repopulating the combo transiently fires currentIndexChanged); a checkbox toggle could rebuild the scan region twice; dark-theme text contrast on the new Time tab; duplicated channel-lookup logic consolidated into a shared _find_channel() helper.

Data-integrity guarantees (recording pipeline)

These came out of the review rounds below and are worth stating as contract:

• acquisition_complete: true is only stamped when every expected frame was written. Write errors, backpressure drops, and under-delivery (camera stall) all seal the store incomplete with counts (write_errors, dropped_frames, captured_frames/expected_frames); user aborts additionally stamp aborted: true.
• The recording dataset is sized from one real processed frame (crop/rotation/ROI applied), not get_resolution() — sensor-vs-delivered mismatches previously produced silently blank recordings on real cameras.
• Frames map to the time slot nearest their arrival time: delivery jitter can't halve the capture rate, and a post-stall burst leaves the stall as fill holes instead of compressing the time axis. A camera that paces itself gets sequential slots instead; that decision is re-made from the rate the camera reports with the stream running (ToupTek rejects the stopped-state rate write after a binning change and reports its free-run maximum).
• The acquisition fails fast (aborts, store sealed incomplete) on systematic recording failure: write errors, sustained drops, or a wedged writer — instead of grinding through hours of blank FOVs. A wedged writer's captured backlog is flushed once the stall clears, never discarded. A recording whose drain thread outlives finalize() stays tracked: a write error that surfaces afterwards still fails the run, and .done is withheld until the store is sealed.
• Hardware state is restored after every run: trigger mode, channel configuration (exposure/gain/illumination), live view, and Z.

Review & hardening

Three review rounds (multi-agent adversarial review with independent verification, plus hand-verification when API limits interrupted round 2) — 40 confirmed findings, all fixed, each with a test that failed first:

• Round 1 (17): critical hardware-path bugs simulation couldn't surface — blank recordings from get_resolution() sizing, error-swallowing drain, finalize deadlock, unresponsive abort, trigger/channel state corruption after runs, fps-gate jitter halving capture rate, count-only queue bound (~13 GB RAM risk), per-FOV z coords dropped, dt pacing/metadata mismatch, missing cleanup/bookkeeping.
• Round 2 (10 + 13 after full verification): completeness-attribute lies (drops/under-delivery), wedged-finalize data discard and false-positive fail-fast, well-selector lost after acquisitions, inert well clicks on the tab, silent recording-channel swaps, monotonic pacing, seal-attribute honesty, snapshot dedupe, per-FOV double camera reconfiguration.
• Merge with master ported the acquisition-watchdog (feat: acquisition watchdog — Slack alert on prematurely-ended acquisitions #565) and z-offset logging (feat(worker): log actual z after each per-channel z-offset move #573) into the refactored base class.
• Copilot round on the stacked branch (5, all fixed with a failing test first): pacing decided from a stopped-state rate write, late write errors lost after finalize() returned, no run.json for Record + Z-Stack, and two cleanup regions that started after the first hardware step.

Testing

• ~120 new tests: StreamingCapture/RecordingWriter/RecordingRouter (error/OOB/abort/wedge/drop/jitter/burst paths), worker simulation (shapes, state restore, pacing, fail-fast, bookkeeping), widget (validation, Copy-from-Live, refresh, Start/Stop handoff, FOV-grid wiring), gui-level tab/selector behavior, camera fps.
• Runtime-verified end-to-end by driving the real GUI in simulation (screenshots + on-disk Zarr shape/attr verification): happy path, validation rejects, mid-recording abort, Nt=2 t-indexing, phase-solo runs, z-offset stage return, back-to-back acquisitions.
• CI green (full install-and-test suite) after resolving the master conflict — note the PR had been CONFLICTING, which silently skips pull_request workflows, so earlier pushes only ran lint.
• black clean (CI 25.12.0).

Hardware testing (pending — the remaining merge gate)

See the test-plan comment: instrument checklist + a camera-only bench mode (real ToupTek camera, everything else simulated via the [SIMULATION] INI section). Highest-value bench test: the PRECISE_FRAMERATE fps hint, which has never executed against real hardware. Also record once right after changing the binning: the log should show routing switched to camera-paced and the store must seal acquisition_complete: true.

Added by the stacking: on the instrument, run 4 Record + Z-Stack acquisitions back to back in one session (ToupTek, whose callback thread persists across runs); every z-stack zarr.json must have acquisition_complete: true and the terminal must show neither aborting: fork() nor Event loop is closed.

Deferred follow-ups (documented, out of scope here)

• GUI-visible error dialog when the fail-fast aborts an acquisition (currently loud in the log only; needs an error signal through the Qt controller).
• closeEvent acquisition guard: app exit during a run now waits 30 s for the worker to unwind, but a still-running worker can race teardown beyond that; a proper "acquisition in progress — abort and exit?" prompt is the full fix.
• NTP-step resilience for the recording router's absolute-slot anchor (wall-clock frame timestamps; a mid-recording clock step mispaces until re-anchored).
• From the original plan: replacing the duck-typed record-tab stubs (display_progress_bar, emit_selected_channels) with a protocol.

🤖 Generated with Claude Code

[hongquanli]

Hardware test plan

Simulation testing (GUI driven end-to-end, Zarr outputs verified — happy path, validation, Copy-from-Live, mid-recording abort, Nt=2 t-indexing, recording-only / z-stack-only, z-offset with stage return) is green as of 9b5be65. The items below are the paths that only run on real hardware and should be checked on an instrument before this ships.

Setup

1. Add enable_recording = True to the [GENERAL] section of configuration_*.ini and restart.
2. The Record + Z-Stack tab appears next to Simple Recording. Switching to it should keep the well selector visible and log no traceback (fixed in 9b5be65 — worth re-confirming on the instrument build).
3. Put a real sample (or fluorescent slide) on stage, focus, and select 1–2 wells.

Tests

• 1. Recording illumination (validates the critical fix). Recording phase only, BF channel, 10 fps × 5 s. Open the recording Zarr — frames must show the sample, not darkness. Repeat with a fluorescence channel. This exercises the illumination-energize fix that simulation can't prove (the sim camera synthesizes images regardless of illumination).
• 2. fps hint (ToupTek PRECISE_FRAMERATE). 10 fps × 10 s → expect exactly T=100 frames, _squid.time_increment_s = 0.1, wall time ≈ 10 s. Then request an impossible rate (e.g. 50 ms exposure @ 30 fps) — the portable safety net should downsample in software; check the log for dropped/queue-full warnings and confirm the Zarr still seals with acquisition_complete: true.
• 3. Hardware-trigger live interplay. Set live to Hardware Trigger, start live, stop live, then run an acquisition with both phases. The z-stack must not come back dark (the trigger-mode fix), and after the acquisition live view must restore and work in the previously selected mode.
• 4. Laser AF reference path. Initialize laser AF and set a reference, tick Laser AF in the tab, run with a recording z-offset (e.g. +10 µm) and a z-stack range. Both phases should be positioned relative to the reference plane — compare focus between the recording frames and the z-stack center plane. Also confirm Start is rejected with a clear message when Laser AF is ticked without a reference.
• 5. Abort mid-recording. 60 s recording, hit Stop after ~10 s. UI must unlock immediately, the recording Zarr must carry _squid.aborted: true / acquisition_complete: false, stage and illumination must return to a sane state, and a follow-up acquisition must work without restarting the app.
• 6. Sustained throughput / backpressure. Realistic load: full-resolution camera, ~30 fps × 60 s recording × 2 wells × a small FOV grid, plus a multi-channel z-stack. Watch the RAM indicator, check the log for queue full / drop warnings from the recording writer, and verify every per-FOV Zarr has the full expected frame count.
• 7. Multi-timepoint on hardware. Nt=3, dt ≥ per-timepoint duration. Expect recording/t0..t2/ trees and a z-stack Zarr (3, C, NZ, Y, X) with all three t-slices populated (verified in simulation; re-check with real stage settle/AF timing).

Inspecting outputs

Per-FOV shapes: recording (T, 1, 1, Y, X), z-stack (Nt, C, NZ, Y, X). Quick check without opening napari:

import json, glob
for p in glob.glob("<experiment_dir>/**/zarr.json", recursive=True):
    m = json.load(open(p))
    if m.get("node_type") == "array":
        print(p, m["shape"], json.load(open(p)).get("attributes", {}).get("_squid", {}))

Log signals worth watching: squid.control.core.zarr_writer warnings (abort path is intentionally loud), recording-writer drop warnings, and [MCU] >>> illumination commands around phase transitions.

---

Camera-only bench (no assembled scope)

If you only have the ToupTek camera on a dev machine, Squid's per-component simulation covers most of the list above. In configuration_*.ini:

[SIMULATION]
simulate_camera = false           ; real ToupTek camera
simulate_microcontroller = true   ; simulated MCU, stage, illumination
simulate_spinning_disk = true
simulate_filter_wheel = true
simulate_objective_changer = true
simulate_laser_af_camera = true

Then launch without --simulation (the flag forces everything simulated; without it the per-component settings apply — see _should_simulate in microscope.py). Stage moves become instant no-ops; every frame comes off the real sensor.

What this bench covers, in value order:

• fps hint (PRECISE_FRAMERATE) — the new camera_toupcam.py code has never executed against real hardware (the simulated camera has its own set_frame_rate). 10 fps × 10 s → exactly T=100 frames, wall time ≈ 10 s, _squid.time_increment_s = 0.1.
• StreamingCapture under real frame timing — real callback thread, real jitter, full-res frames into the bounded queue. Feasible rate: no drop warnings, exact count. Then request an infeasible rate (e.g. 100 ms exposure @ 30 fps) and observe: does the run stretch until T frames arrive, or do drops get logged loudly?
• Real pixels through the save path — point the camera at the room, wave a hand during recording, verify frame-to-frame differences in the Zarr are non-zero (a live temporal stream, not repeated buffers).
• Abort/recovery with a real device — Stop mid-recording → aborted: true seal → Start Live again streams normally without replug/restart.
• Multi-well/FOV fan-out at real data rates — 2 wells × 4 FOVs; simulated stage walks instantly, per-FOV Zarr routing carries real frames.

Still instrument-only after this bench: illumination energize (dark-frame check), hardware-trigger interplay, laser AF, real stage-settle timing (tests 1, 3, 4 above).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR adds a new Record + Z-Stack acquisition mode to the HCS GUI, enabling per-FOV sequential acquisition of (1) a continuous recording phase saved as per-FOV Zarr and (2) a multi-channel Z-stack saved as OME-Zarr, with shared acquisition mechanics refactored into a reusable worker base.

Changes:

• Introduces a recording pipeline (StreamingCapture) with bounded backpressure and integrity sealing for incomplete captures.
• Adds RecordZStackController/RecordZStackWorker plus GUI wiring for a new “Record + Z-Stack” tab.
• Refactors shared capture/job-dispatch mechanics into MultiPointWorkerBase and adds a best-effort camera FPS hint API (set_frame_rate) including Toupcam support.

Reviewed changes

Copilot reviewed 20 out of 21 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
software/control/acquisition_yaml_loader.py	Extends YAML parsing to support widget_type: record_zstack.
software/control/camera_toupcam.py	Adds Toupcam set_frame_rate() support and continuous-mode max FPS calculation.
software/control/core/acquisition_setup.py	Introduces shared helpers for pixel size calculation and experiment directory creation.
software/control/core/multi_point_controller.py	Uses shared experiment-dir creation helper.
software/control/core/multi_point_worker.py	Refactors shared acquisition mechanics into MultiPointWorkerBase.
software/control/core/record_zstack_controller.py	Adds controller for Record + Z-Stack acquisition setup/threading and YAML snapshotting.
software/control/core/record_zstack_worker.py	Adds worker implementing per-FOV recording + z-stack sequencing.
software/control/core/streaming_capture.py	Adds recording primitives (frame source/router/stop condition/writer) with bounded queues.
software/control/core/zarr_writer.py	Improves event-loop handling and expands abort sealing to distinguish incomplete vs aborted runs.
software/control/gui_hcs.py	Wires new Record + Z-Stack tab/widget/controller into the HCS GUI and well-selector behavior.
software/squid/abc.py	Adds AbstractCamera.set_frame_rate() API (default no-op returning max achievable).
software/squid/camera/utils.py	Implements set_frame_rate() and continuous pacing behavior in the simulated camera.
software/tests/control/test_HighContentScreeningGui.py	Adds GUI tests ensuring well-selector behavior on the new tab.
software/tests/control/test_acquisition_yaml_loader.py	Adds YAML loader tests for record_zstack acquisitions.
software/tests/core/init.py	Test package init (structure/support for new core tests).
software/tests/core/test_record_zstack_worker.py	Adds worker/controller smoke + behavior tests for Record + Z-Stack acquisition.
software/tests/core/test_streaming_capture.py	Adds extensive unit tests for the recording pipeline and its failure modes.
software/tests/test_acquisition_yaml_drop_mixin.py	Adds tests for YAML drop mixin generalizations supporting the 3rd widget type.
software/tests/test_record_zstack_camera_fps.py	Adds tests for camera FPS hinting/clamping (sim + Toupcam math).

Comments suppressed due to low confidence (1)

software/control/acquisition_yaml_loader.py:133

• parse_acquisition_yaml() is annotated/documented to return AcquisitionYAMLData, but it now returns RecordZStackYAMLData when widget_type == "record_zstack". This makes the public API misleading and can break type-aware tooling (and any downstream code that relies on AcquisitionYAMLData-specific fields without checking widget_type first).

def parse_acquisition_yaml(file_path: str) -> AcquisitionYAMLData:
    """Parse acquisition YAML file and return structured data.

    Args:
        file_path: Path to the acquisition.yaml file

    Returns:
        AcquisitionYAMLData with parsed values

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 21 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

software/squid/camera/utils.py:312

• SimulatedCamera.set_frame_rate() clamps achievable fps using get_total_frame_time() (exposure + strobe), but the CONTINUOUS streaming loop paces frames using only exposure time. This can make the simulated camera deliver frames faster than the achievable fps it reports, which can mis-size recordings and invalidate tests that rely on achievable-fps behavior.

                period_s = self._exposure_time_ms / 1000.0
                if self._target_frame_period_s is not None:
                    period_s = max(period_s, self._target_frame_period_s)
                time_since = time.time() - last_frame_time
                if time_since >= period_s and self._acquisition_mode == CameraAcquisitionMode.CONTINUOUS:

[Alpaca233]

fixed in lastest master

[Copilot]

Pull request overview

Copilot reviewed 20 out of 21 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

software/squid/camera/utils.py:312

• The simulated camera’s streaming loop paces frames using exposure time only, but set_frame_rate() clamps/returns an achievable rate using get_total_frame_time() (exposure + strobe). This makes the simulated camera’s actual free-run period inconsistent with the frame-time contract used elsewhere (and with the achievable fps that callers compute).

            last_frame_time = time.time()
            while self._continue_streaming:
                period_s = self._exposure_time_ms / 1000.0
                if self._target_frame_period_s is not None:
                    period_s = max(period_s, self._target_frame_period_s)
                time_since = time.time() - last_frame_time
                if time_since >= period_s and self._acquisition_mode == CameraAcquisitionMode.CONTINUOUS:

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved timing, cleanup, abort, and failure-classification paths can produce incorrect metadata or leave hardware resources in an inconsistent state.

Review effort: Balanced
Findings: 6 · 1

Open (7)

• Recheck abort state before starting the z-stack · New
• Always finalize the writer when source stop fails · New
• Preserve current-Z recording offsets without Laser AF · New
• SimulatedCamera.read_camera_frame() currently waits only until _current_frame is non-None. If the…
• RecordingWriter.finalize() is intended to respect a single timeout budget, but the sentinel put()…
• SimulatedCamera.set_frame_rate() clamps using get_total_frame_time() (exposure + strobe), but the…
• This log message uses braces literally (not an f-string), so the timeout value never appears in…

[Copilot]

🟡 Changes recommended

Outstanding lifecycle, pacing, cleanup, and watchdog gaps can leave stores unsealed or acquisitions incorrectly reported.

12 open findings

• Routing mode uses stale pre-start frame-rate result · New
• Finalize returns before in-flight writes complete · New
• Record and Z-Stack path omits RunStateWriter lifecycle · New
• Early illumination failure skips hardware-state restoration · New
• Camera leaks when stream setup fails · New
• Preserve current-Z recording offsets without Laser AF
• Always finalize the writer when source stop fails
• Recheck abort state before starting the z-stack
• SimulatedCamera.read_camera_frame() currently waits only until _current_frame is non-None. If the…
• RecordingWriter.finalize() is intended to respect a single timeout budget, but the sentinel put()…
• SimulatedCamera.set_frame_rate() clamps using get_total_frame_time() (exposure + strobe), but the…
• This log message uses braces literally (not an f-string), so the timeout value never appears in…

🧠 Review effort: Balanced

[Alpaca233]

[Claude Code] Fixed in 6c36b1a - ContinuousFrameSource.start() now hands the rate the camera reports with the stream running to RecordingRouter.set_paced_from_rate(), so a rejected stopped-state write (free-run maximum reported) no longer leaves an at-rate sensor on nearest-slot routing. Tests: test_frame_source_corrects_the_router_pacing_from_the_running_stream_rate, test_recording_switches_to_paced_routing_when_the_camera_only_paces_once_streaming.

[Alpaca233]

[Claude Code] Simplified in 54eb11c - the regime is now decided once, by StreamingCapture, from the rate ContinuousFrameSource.start() returns with the stream running (RecordingRouter.set_paced_from_rate); the worker no longer pre-decides from the stopped-state probe. Tests: test_router_decides_the_regime_from_the_running_stream_rate, test_recording_switches_to_paced_routing_when_the_camera_only_paces_once_streaming.

[Alpaca233]

[Claude Code] Fixed in 6c36b1a - finalize() returns whether the store is sealed and RecordingWriter.wait_sealed() waits for a drain that outlived it. RecordZStackWorker tracks those writers, fails the run on a write error that surfaces afterwards (checked at the next plane and before .done), and withholds .done while a store is still being written. Tests: test_finalize_reports_a_drain_that_outlived_it_and_wait_sealed_sees_its_errors, test_late_recording_write_error_ends_the_run_as_error, test_run_withholds_done_while_a_recording_is_still_being_written.

[Alpaca233]

[Claude Code] Fixed in 6c36b1a - RecordZStackController starts the run.json breadcrumb through a new start_run_state_writer helper (MultiPointController now uses the same one); the worker beats per FOV and ends it with the same reasons as MultiPointWorker (_compute_end_reason and the run-state writer moved to MultiPointWorkerBase). Test: test_record_zstack_run_writes_the_watchdog_breadcrumb.

[Alpaca233]

[Claude Code] Fixed in 6c36b1a - the first Z move and the illumination enable run inside record()'s try, so the finally restores Z and the camera mode when either raises. Test: test_record_restores_z_and_camera_mode_when_illumination_fails.

[Alpaca233]

[Claude Code] Fixed in 6c36b1a - start_streaming() and add_frame_callback() run inside zstack()'s try with cb_id initialized to None; the finally removes the callback only when it was registered and always stops streaming. Test: test_zstack_stops_streaming_when_callback_registration_fails.

[maragall]

Hardware test results (head 54eb11c)

Run on a Squid+ (ToupTek ITR3CMOS26000KMA, 20x, 2×2 binning, Teensy controller, Daheng laser AF camera) on 2026-10-12, with a 96-well glass-bottom plate of fluorescent beads: G10 has moving beads, G11 stuck beads. The tests are the ones in @hongquanli's test plan and @Alpaca233's PR description, driven through the Record + Z-Stack tab.

Result: 10 of 11 pass on 54eb11c. Trigger restore fails every time (3/3). With the one-line fix below, all 11 pass, including trigger restore 2/2.

Test	By	Result	Measured
Tab and setup	hongquanli	✅	The tab is present, the well selector stays visible, and no traceback appears.
Recording illumination	hongquanli	✅	BF frames average 2,261 and 488 frames 4,723, against about 870 with the light off. The sample is visible in both.
fps hint	hongquanli	✅	10 fps × 10 s gives T = 100, time_increment_s 0.1, and 10.02 s from stream start to finish. 30 fps at 50 ms exposure warns before Start and logs camera cannot deliver 30 fps at 50 ms exposure (max 20.00). That run recorded at 20 fps and sealed acquisition_complete: true.
Real pixels	hongquanli	✅	Moving beads, 100 frames, no identical consecutive pair.
Hardware-trigger live	hongquanli	❌ → ✅ with fix	See below.
Laser AF reference	hongquanli	✅	Start is refused without a reference: "Laser AF is enabled but no reference position has been captured." With a reference and a +10 µm offset, the sharpest z-stack plane is at 0 µm, the recording matches the +10 µm plane, and Z returns to the reference afterwards.
Abort mid-recording	hongquanli	✅	Stop at 10 s of 60 s. The UI unlocks in 0.25 s and the store is sealed aborted: true, acquisition_complete: false. Illumination is off, Z is back at the reference, and the next run completes without a restart.
Throughput and fan-out	hongquanli	✅	30 fps requested, 28 fps achieved (the camera's maximum at 20 ms exposure), × 60 s × 2 wells × 2 FOVs, plus a BF + 488 z-stack. All 4 recordings have T = 1680 and are complete, all 4 z-stacks are complete, and no drop or stall appears in the log. Process RSS stays between 0.61 and 0.73 GB.
Multi-timepoint	hongquanli	✅	Nt = 3 gives recordings t0..t2 in both wells. The z-stacks are (3, 2, 7, 2084, 2084), all 3 t-slices are filled, and every store is complete.
After a binning change	Alpaca233	✅	Recorded right after switching to 1×1. The log shows recording routed camera-paced, sequential slots (camera reports 6.80 fps ...) and the store is complete. The log line quoted in the description (routing switched to camera-paced) does not exist at this head, so I checked against this one.
Back to back	Alpaca233	✅	4 Record + Z-Stack runs in one session. Every z-stack and recording is complete, and neither aborting: fork() nor Event loop is closed appears.

Hardware-trigger live: the failure and its cause

Setting live view to Hardware Trigger, starting and stopping live, then running both phases: the recording completes (30/30 frames), then the z-stack's first software trigger never returns a frame. The worker logs Timed out waiting {non_hw_frame_timeout} [s] for a frame, aborting acquisition. and no z-stack store is written. This happened 3 of 3 times. Starting from Software Trigger, every z-stack works.

Cause. The ToupTek trigger source (IOCONTROLTYPE_TRIGGERSOURCE: 1 = GPIO0, 5 = software) follows OPTION_TRIGGER only while the stream runs. zstack() calls set_trigger_mode(SOFTWARE) on a stopped stream, so the option becomes 1 (software) while the source stays on GPIO0, and the camera ignores software triggers. Camera-level experiments, without Record + Z-Stack:

Sequence	Source after the switch	Software trigger
Hardware → Software, stream running	5	frame in 0.08 s
Hardware-trigger live, then Software, stream running	5	frame in 0.08 s
Hardware → Software, stream stopped	1	no frame
Same, with a continuous recording in between	1	no frame
Previous state, source set to 5 by hand	5	frame in 0.08 s

The recording itself is not involved, and hardware trigger keeps working after a recording (25 frames in 3 s of hardware-trigger live).

Fix: d5df796 on this branch. The run now makes its one switch to software trigger right after stopping live view, with the camera streaming (it starts the stream first if a previous run left it stopped). That is the case where the camera moves the source itself, as in the first two rows. zstack()'s own switch stays as the fallback, and the user's mode is still restored at the end. The shared ToupTek driver is unchanged. A driver-level version (#704, now closed) was dropped to keep the change inside this feature.

A unit test (test_hardware_trigger_run_switches_to_software_while_streaming_before_recording) fails without the change and passes with it. All 52 worker tests pass.

Verified with the fix on this microscope:

• Hardware-trigger live passes 2/2. The z-stack is lit (BF about 10,660 and 488 about 2,890, against about 870 dark), it seals complete, and live view comes back in Hardware Trigger.
• Two hardware-trigger runs back to back pass. The second run starts with the camera stopped, and both z-stacks are lit and complete.
• The other 10 tests still pass.

Not covered by these tests

• Level-mode hardware triggering (HARDWARE_TRIGGER_MODE = LEVEL). This microscope uses edge mode.
• Other ToupTek models.
• Hardware-path changes that no test in the plan reaches:
  • Live in Software Trigger before Start (2e7e2ba).
  • A hardware ROI (5bad2d6).
  • Multi-plane recording, Nz > 1 (a8735fa, c0c2a3e).
  • Fail-fast on write errors, drops or a failed save (da61d26, 739cba5).
  • Live preview during a recording (b914c21).
• Per-FOV stored Z was used implicitly, because every FOV here carried its own Z, but it was not checked as a test of its own.

Notes

• The timeout message above is missing its f prefix (multi_point_worker.py:701), so it prints {non_hw_frame_timeout} literally.
• With laser AF on, a FOV whose stored Z was 18.5 µm away from the current focus (the plate had drifted over an hour) landed 7.9 µm above the reference. With the FOV's Z at the current focus, laser AF landed exactly on the reference. This looks like the laser AF's working range, not this PR.
• Wellplate BF images in Hardware Trigger read about 7,680, against about 10,750 in Software Trigger at the same settings, with or without the fix. I have not looked into this.

🤖 Generated with Claude Code

[maragall]

I could not find any bugs in output.