feat(acquisition): save Zarr v3 in the acquisition process

[Alpaca233]

Problem

tensorstore aborts any fork()ed child of a process that has already used it, on the child's first tensorstore call:

aborting: fork() use detected, which is not allowed due to internal threading use by tensorstore

Zarr v3 is written by SaveZarrJob inside the forked JobRunner subprocess, and the subprocess for the next acquisition is forked at the start of the current one. So once the main process has used tensorstore, every later Zarr v3 acquisition is saved by a subprocess that dies on its first write. The main process does not notice: the backpressure wait times out per image (30 s after the 10th pending job), the run logs Pending jobs for SaveZarrJob abandoned!!! and ends "normally" with no data on disk.

On master today the trigger is the NDViewer (enable_ndviewer), which opens stores with tensorstore in the main process: Zarr v3 + viewer loses the third acquisition and every one after it. #564 (Record + Z-Stack) adds a recording writer that uses tensorstore in the main process on every recording, which makes this hit every session. Measured in simulation: runs 1–2 complete, runs 3–4 lose the z-stack entirely.

What changes

Zarr v3 is saved in the acquisition process; no subprocess is forked for it.

• InProcessZarrRunner (new, control/core/in_process_zarr_runner.py): presents the part of JobRunner's interface the worker uses (dispatch, output_queue, has_pending, shutdown, …) and runs SaveZarrJob here. dispatch submits the tensorstore write and returns (~1 ms in the camera callback; the first frame of each store also creates it there, ~2 ms median / 17 ms max on a local SSD); a drain thread waits on each future in order, keeps the same backpressure counters, and reports JobResults on the same output queue. shutdown(aborted=...) waits for outstanding writes and seals every store.
• ZarrWriterRegistry: the open writers and HCS plate/well bookkeeping were class-level on SaveZarrJob (per-acquisition only because each acquisition had its own subprocess). Now an injectable object; the in-process runner owns one per acquisition, the subprocess keeps SaveZarrJob.default_registry.
• ZarrWriter.submit_frame() / SaveZarrJob.submit(): the non-blocking halves of write_frame() / run(), which are now submit-then-wait with unchanged behavior.
• ZarrWriter no longer uses a per-thread asyncio event loop (it used one only for ts.open). With initialize on the camera callback thread and finalize on the worker thread, the loop state broke the second acquisition on cameras whose callback thread persists; tensorstore futures are waited on directly now.
• MultiPointWorker / MultiPointController.close(): SaveZarrJob gets the in-process runner (and an abnormal app shutdown seals its stores instead of trying to terminate() it); the TIFF job classes keep the forked subprocess exactly as before. MultiPointController.get_prewarmed_job_runner() hands nothing to a Zarr v3 run, so the warm subprocess stays ready for a later TIFF run.
• Loud failure: a save subprocess that dies with jobs pending, a save job of any class whose failure only surfaces in the final drain, or a save backlog that outlives the end-of-run wait ("Pending jobs … abandoned") now ends the acquisition as an error instead of a quiet "completed" with data missing.

What does not change

• The forked JobRunner path for SaveImageJob / SaveOMETiffJob: same start method, same pre-warm, same pickling.
• ZarrWriter is still the only Zarr writer and writes the same bytes. The multiprocessing start method stays fork.

Behavior changes to know about

• A Zarr v3 store of an acquisition that was aborted (user Stop, error, or a crash out of the worker) is now sealed acquisition_complete: false, aborted: true. The subprocess sealed every store acquisition_complete: true regardless.
• Acquisition.USE_MULTIPROCESSING = False with Zarr v3 now works and seals the store. On master that path aborted on the first image (zarr_writer_info was never injected in the in-process fallback).
• main_hcs_worker.log is no longer written for Zarr v3 runs (no subprocess); the Finalized zarr writer lines are in main_hcs.log.

Measurement (why in-process, not spawn/forkserver)

Z-stack phase only, 2 channels × 11 planes of 2048×2048 uint16, simulated camera, three runs each, fresh process per run:

Save path	Z-stack phase wall time	Camera callback blocked per image
Subprocess, as on master (fork + pickle each image)	1.28–1.34 s	~0 ms (copy on a feeder thread)
In-process, existing fallback (waits on every write)	1.99–2.08 s	36–38 ms
In-process, non-blocking submits (this PR)	1.06–1.09 s	1.0 ms

tensorstore compresses and writes on its own threads, so the subprocess only added a copy of every image. A spawn/forkserver subprocess would also fix the abort but costs 1.6 s to become ready (vs 8 ms) and 224 MB idle (vs 83 MB).

Tests

• New: tests/control/core/test_in_process_zarr_runner.py (9), registry / submit() / submit_frame() tests in tests/control/core/test_zarr_writer.py, and in tests/control/test_MultiPointController.py: Zarr v3 multipoint saves in-process and seals; sealed with multiprocessing off; abort mid-run seals aborted; a Zarr run neither consumes nor restarts the warm subprocess; three Zarr runs in one process all sealed; a write error ends the run as an error (per-FOV and end-of-run); a dead TIFF subprocess aborts the run.
• Full suite in one process passes locally (GitHub CI does not install tensorstore, so it skips every tensorstore-dependent test; a green CI says nothing about this path).

Merge gate: bench on the instrument

1. Multipoint saved as Zarr v3, 3 acquisitions back to back with the NDViewer enabled, on a camera whose frame-callback thread persists across runs (Hamamatsu, ToupTek): every OME group zarr.json has acquisition_complete: true; no aborting: fork() and no Event loop is closed in the terminal. (On today's master the third run loses its data.)
2. Multipoint OME-TIFF, 2 acquisitions: unchanged behavior (the subprocess path).
3. Per-image time of a Zarr v3 z-stack at 2×2 binning 16-bit, master vs this branch: must not be slower. Also a many-FOV run with 1–3 images per FOV (store creation now happens on the first frame of each FOV, in the callback): per-image time must not be slower.
4. main_hcs.log carries the Finalized zarr writer lines; no main_hcs_worker.log for the Zarr runs.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Shutdown can seal and clear a Zarr store while TensorStore writes remain active after the drain timeout.

1 open finding

• Timeout can finalize or abort while drain thread still writes · New

What changed in this PR

Moves Zarr v3 saving into the acquisition process to avoid TensorStore failures in forked subprocesses.

Changes:

• Adds asynchronous in-process Zarr writes and per-acquisition writer registries.
• Updates acquisition lifecycle, backpressure, failure handling, and store sealing.
• Adds comprehensive tests and documentation.

File	Description
software/​control/​core/​in_process_zarr_runner.py	Adds the in-process Zarr runner.
software/​control/​core/​job_processing.py	Adds writer registries and non-blocking job submission.
software/​control/​core/​zarr_writer.py	Uses TensorStore futures without asyncio.
software/​control/​core/​multi_point_worker.py	Routes Zarr saving in-process and propagates failures.
software/​control/​core/​multi_point_controller.py	Preserves prewarmed TIFF runners and handles Zarr shutdown.
software/​docs/​zarr-v3-format.md	Documents in-process Zarr saving.
software/​tests/​control/​core/​test_in_process_zarr_runner.py	Tests runner lifecycle, failures, and backpressure.
software/​tests/​control/​core/​test_zarr_writer.py	Tests registries, futures, and cross-thread use.
software/​tests/​control/​test_MultiPointController.py	Adds acquisition-level Zarr integration tests.
software/​tests/​control/​core/​test_backpressure.py	Updates subprocess shutdown mocks.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Alpaca233]

[Claude Code] Fixed in 79f5075 - shutdown() now returns False when a write is still in flight after the bounded join and seals the stores as aborted instead of stamping them complete; the worker treats False as an error and aborts the run. Covered by test_shutdown_with_a_write_still_in_flight_seals_aborted_and_reports_it.