From afaabcfee61e357c74e88d4c472cc055ada95796 Mon Sep 17 00:00:00 2001 From: Spencer Schwarz Date: Thu, 23 Jul 2026 01:45:59 -0400 Subject: [PATCH 01/13] Migrate GUI from PyQt5 to PyQt6 for napari 0.7 / Python 3.14 WHY --- The HCS GUI embeds napari viewers (Live, Multichannel, and the unified Mosaic/Plate view) for image display. napari 0.7 targets PyQt6, but the app was pinning the Qt binding to PyQt5 via `QT_API=pyqt5`. Running napari 0.7 + vispy 0.16 under PyQt5 made the Mosaic view glitchy/broken and blocked moving to Python 3.14 (PyQt5 has no 3.14 wheels; old napari pulled in pydantic v1, which is also incompatible with 3.14). See "Python 3.14 Compatibility Check.md" for the full dependency analysis behind these decisions. This migrates the app to PyQt6 so the embedded napari 0.7 viewers render correctly and the stack can run on Python 3.14 (napari 0.7.0, PyQt6 6.11, numpy 2.4, pydantic v2). WHAT CHANGED ------------ 1. Qt binding flipped PyQt5 -> PyQt6. `os.environ["QT_API"]` is set to "pyqt6" in every entry module that configures it before the first qtpy import: main_hcs.py, control/gui_hcs.py, control/widgets.py, control/console.py, control/single_instance.py, control/core/core.py, control/core_volumetric_imaging.py, control/core_usbspectrometer.py, control/core_PDAF.py, control/core_displacement_measurement.py, control/widgets_usbspectrometer.py, scripts/run_acquisition.py (child env), and tests/control/test_single_instance.py. Almost all Qt access already goes through qtpy, which shims the common PyQt5->PyQt6 differences (unscoped enums, .exec_(), QAction/QShortcut re-exports), so most call sites needed no change. 2. The handful of real PyQt6 breakers that qtpy does NOT shim: - QDesktopWidget (removed in Qt6) -> QApplication.primaryScreen() in control/gui_hcs.py and control/core_volumetric_imaging.py. - QVariant (removed in PyQt6): dropped the now-dead isinstance(x, QVariant) unwrap guards in control/core/core.py and control/gui_hcs.py (PyQt6 hands back plain Python objects). - QComboBox.activated[str] overload removed in Qt6 -> use the dedicated `textActivated` signal in control/widgets.py. - tools/view_laser_af_reference_image.py: converted its direct PyQt5 imports to qtpy and matplotlib backend_qt5agg -> backend_qtagg (binding-agnostic). 3. Shared OpenGL context for the multiple embedded napari viewers. main_hcs.py now sets AA_ShareOpenGLContexts before the QApplication is constructed. Several vispy/OpenGL canvases in one process must share a GL context; without it, rendering fails with "Cannot SIZE object N ..." / GL_INVALID_FRAMEBUFFER_OPERATION. 4. Single Qt binding requirement + setup scripts. The blank-canvas rendering seen during bring-up was root-caused to PyQt5 and PyQt6 being installed side by side (conflicting Qt libraries break napari's OpenGL rendering). setup_22.04.sh and setup_cuda_22.04.sh now install exactly one binding: PyQt6 (+ PyQt6-Qt6, PyQt6-sip) via pip, remove any pre-existing PyQt5, move pyqtgraph to pip (apt's python3-pyqtgraph drags in python3-pyqt5), bump napari to >=0.7,<0.8, and drop the "numpy<2" pin (napari 0.7 needs numpy>=2). Comments explain the one-binding rule so it isn't reintroduced. 5. Mosaic view: defer GL redraws while its tab is hidden (control/widgets_mosaic.py). Under PyQt6, painting a napari/vispy canvas that lives on a hidden QTabWidget page fails because the hidden QOpenGLWidget has an incomplete framebuffer, which corrupts the shared GL program and cascades into repeated "Cannot SIZE object N because it does not exist" errors during acquisition. The widget now writes tile pixel data unconditionally (so on-disk saves stay correct) but skips the GL refresh/reset/fit calls while hidden, records a pending-refresh flag, and flushes it in showEvent once the tab is visible. This makes live-during-acquisition mosaic updates work regardless of which tab is in front. TESTING ------- - Verified against the live dev environment: Python 3.14, napari 0.7.0, PyQt6 6.11, numpy 2.4.4, pydantic 2.13.4, qtpy 2.4.3, vispy 0.16.1. - qtpy resolves to PyQt6; full test suite collects cleanly (1435 tests). - GUI/pytest-qt suites pass, including tests/control/test_unified_mosaic_widget.py (its fixture builds a real napari.Viewer). The pre-existing full-suite single-process teardown crash (napari/vispy OpenGL shutdown) is unrelated to the binding change; run napari-heavy test files separately. - App launches and the Mosaic/Live/Multichannel napari views render; a full plate acquisition completes and the mosaic populates in both Full View and Plate View. KNOWN FOLLOW-UPS (pre-existing, not introduced here) ---------------------------------------------------- - Memory: full-view mosaic of a multi-well plate builds a large in-memory canvas (~2GB+), which napari displays at ~4x overhead, and the end-of-run save copies the whole canvas; back-to-back runs can stack these. Not an unbounded leak (canvas is reused), but peak RAM is high and causes GUI-thread stalls. Candidate fixes: bound the save-snapshot copy, pre-allocate the full-view canvas, optionally coarser full-view render resolution. - A napari 0.7 / vispy 0.16 layer-teardown ValueError ("list.remove(x): x not in list") can fire when clearing layers on a mode switch. Needs a guard. Co-Authored-By: Claude Opus 4.8 (1M context) --- software/Python 3.14 Compatibility Check.md | 66 +++++++++++++++++++ software/control/console.py | 2 +- software/control/core/core.py | 2 +- software/control/core_PDAF.py | 2 +- .../control/core_displacement_measurement.py | 2 +- software/control/core_usbspectrometer.py | 2 +- software/control/core_volumetric_imaging.py | 6 +- software/control/gui_hcs.py | 8 +-- software/control/single_instance.py | 2 +- software/control/widgets.py | 4 +- software/control/widgets_mosaic.py | 63 ++++++++++++++++-- software/control/widgets_usbspectrometer.py | 2 +- software/main_hcs.py | 11 +++- software/scripts/run_acquisition.py | 2 +- software/setup_22.04.sh | 24 ++++--- software/setup_cuda_22.04.sh | 2 - .../tests/control/test_single_instance.py | 2 +- .../tools/view_laser_af_reference_image.py | 10 ++- 18 files changed, 171 insertions(+), 41 deletions(-) create mode 100644 software/Python 3.14 Compatibility Check.md diff --git a/software/Python 3.14 Compatibility Check.md b/software/Python 3.14 Compatibility Check.md new file mode 100644 index 000000000..33804f13a --- /dev/null +++ b/software/Python 3.14 Compatibility Check.md @@ -0,0 +1,66 @@ +# Python 3.14 Compatibility -- Squid Dependencies + +## Supported (confirmed) +| Package | Notes | +|---------|-------| +| numpy | Wheels for 3.14 since v2.3+ | +| scipy | Supported | +| pandas | Supported | +| pydantic (v2) | Supported -- but **pydantic v1** (used by old napari) is **NOT** | +| Pillow | Supported | +| matplotlib | Supported | +| PyYAML | Supported | +| filelock | Supported | +| platformdirs | Supported | +| scikit-image | Wheels for 3.14 since v0.26 | +| napari (latest) | Supported -- requires PyQt6, not PyQt5 | +| opencv-python-headless | Uses stable ABI (cp37-abi3), works | +| opencv-contrib-python-headless | Same stable ABI, works | +| torch (PyTorch) | Supported since ~2.13 | +| dask | Active releases in 2026 | +| zarr | Supported | +| tifffile | Supported | +| imageio | Supported | +| pyqtgraph | Pure Python, likely works | +| qtpy | Pure Python abstraction layer, works | + +## Blocked / Incompatible +| Package | Issue | +|---------|-------| +| **PyQt5** | **No Python 3.14 support -- this is the main blocker** | +| **pydantic v1** | Explicitly incompatible with 3.14 (the `__slots__` error) | + +## Uncertain / Likely OK but unconfirmed +| Package | Notes | +|---------|-------| +| pyserial | Pure Python, last release 2020 -- probably works but untested | +| pyvisa | Likely works | +| hidapi | Cython extension -- no 3.14 wheels confirmed | +| GitPython | Marked as NOT supporting 3.14 on pyreadiness.org | +| aicsimageio | Unclear, niche package | +| basicpy | Unclear | +| ome-zarr | Unclear | +| pydantic-xml | Likely works if pydantic v2 is used | +| pyreadline3 | Windows-specific, unclear | +| qtconsole | Depends on Qt backend | +| lxml / lxml_html_clean | C extension -- needs checking | +| mcp | Unclear | +| Hardware drivers (PySpin, pyAndorSDK3, ids_peak, pyvcam, pm16) | Vendor-specific, unlikely to have 3.14 builds yet | + +## Bottom Line + +The **two hard blockers** are **PyQt5** and **pydantic v1** (via old napari). Migrating to Python 3.14 would require: +1. Switching from PyQt5 to PyQt6 (non-trivial API changes) +2. Upgrading napari to a version that uses pydantic v2 +3. Verifying all hardware vendor SDKs have 3.14 builds + +**Recommendation: Stay on Python 3.10** for production use. It's the sweet spot where everything works. + +## Sources +- [Python 3.14 Readiness](https://pyreadiness.org/3.14/) +- [NumPy 2.3.0 Release Notes](https://numpy.org/devdocs/release/2.3.0-notes.html) +- [PyQt5 on PyPI](https://pypi.org/project/PyQt5/) +- [napari Installation](https://napari.org/stable/getting_started/installation.html) +- [PyTorch torch.compile Python 3.14 support](https://dev-discuss.pytorch.org/t/torch-compile-support-for-python-3-14-completed/3276) +- [Pydantic v1 Python 3.14 issue](https://github.com/pydantic/pydantic/issues/12618) +- [Anaconda Python 3.14 overview](https://www.anaconda.com/blog/python-3-14-what-data-scientists-developers-need-know) diff --git a/software/control/console.py b/software/control/console.py index fed32978e..9accbb423 100644 --- a/software/control/console.py +++ b/software/control/console.py @@ -1,7 +1,7 @@ import os # set QT_API environment variable -os.environ["QT_API"] = "pyqt5" +os.environ["QT_API"] = "pyqt6" import qtpy from qtpy.QtWidgets import QApplication, QMainWindow, QWidget, QVBoxLayout, QPushButton, QLabel diff --git a/software/control/core/core.py b/software/control/core/core.py index 35ef4ad5a..6f4c0b623 100644 --- a/software/control/core/core.py +++ b/software/control/core/core.py @@ -4,7 +4,7 @@ import tempfile # qt libraries -os.environ["QT_API"] = "pyqt5" +os.environ["QT_API"] = "pyqt6" import qtpy import pyqtgraph as pg from qtpy.QtCore import * diff --git a/software/control/core_PDAF.py b/software/control/core_PDAF.py index 935955b81..f89e7f6d8 100644 --- a/software/control/core_PDAF.py +++ b/software/control/core_PDAF.py @@ -1,7 +1,7 @@ # set QT_API environment variable import os -os.environ["QT_API"] = "pyqt5" +os.environ["QT_API"] = "pyqt6" # qt libraries from qtpy.QtCore import * diff --git a/software/control/core_displacement_measurement.py b/software/control/core_displacement_measurement.py index c86794d3b..55336f55c 100644 --- a/software/control/core_displacement_measurement.py +++ b/software/control/core_displacement_measurement.py @@ -1,7 +1,7 @@ # set QT_API environment variable import os -os.environ["QT_API"] = "pyqt5" +os.environ["QT_API"] = "pyqt6" import qtpy # qt libraries diff --git a/software/control/core_usbspectrometer.py b/software/control/core_usbspectrometer.py index e67f2bd31..733065ec7 100644 --- a/software/control/core_usbspectrometer.py +++ b/software/control/core_usbspectrometer.py @@ -1,7 +1,7 @@ # set QT_API environment variable import os -os.environ["QT_API"] = "pyqt5" +os.environ["QT_API"] = "pyqt6" import qtpy # qt libraries diff --git a/software/control/core_volumetric_imaging.py b/software/control/core_volumetric_imaging.py index 9c15c81aa..ef9689522 100644 --- a/software/control/core_volumetric_imaging.py +++ b/software/control/core_volumetric_imaging.py @@ -1,7 +1,7 @@ # set QT_API environment variable import os -os.environ["QT_API"] = "pyqt5" +os.environ["QT_API"] = "pyqt6" import qtpy # qt libraries @@ -173,8 +173,8 @@ def __init__(self, window_title=""): self.setCentralWidget(self.widget) # set window size - desktopWidget = QDesktopWidget() - width = min(desktopWidget.height() * 0.9, 1000) # @@@TO MOVE@@@# + screen_height = QApplication.primaryScreen().size().height() + width = int(min(screen_height * 0.9, 1000)) # @@@TO MOVE@@@# height = width self.setFixedSize(width, height) diff --git a/software/control/gui_hcs.py b/software/control/gui_hcs.py index ba8c93e05..a9c558ec0 100644 --- a/software/control/gui_hcs.py +++ b/software/control/gui_hcs.py @@ -15,7 +15,7 @@ ClearedScanCoordinates, ) -os.environ["QT_API"] = "pyqt5" +os.environ["QT_API"] = "pyqt6" import re import time from enum import Enum, auto @@ -1411,8 +1411,7 @@ def _getMainWindowMinimumSize(self): We want our main window to fit on the primary screen, so grab the users primary screen and return something slightly smaller than that. """ - desktop_info = QDesktopWidget() - primary_screen_size = desktop_info.screen(desktop_info.primaryScreen()).size() + primary_screen_size = QApplication.primaryScreen().size() height_min = int(0.9 * primary_screen_size.height()) width_min = int(0.96 * primary_screen_size.width()) @@ -2398,9 +2397,6 @@ def onDisplayTabChanged(self, index): self.toggleWellSelector(False) def onWellplateChanged(self, format_): - if isinstance(format_, QVariant): - format_ = format_.value() - # TODO(imo): Not sure why glass slide is so special here? It seems like it's just a "1 well plate". if format_ == "glass slide": self.toggleWellSelector(False) diff --git a/software/control/single_instance.py b/software/control/single_instance.py index 309e2c896..60f2eee7c 100644 --- a/software/control/single_instance.py +++ b/software/control/single_instance.py @@ -11,7 +11,7 @@ # Pin the Qt binding before importing qtpy, matching the convention used by # every other Qt-using module in this codebase. -os.environ["QT_API"] = "pyqt5" +os.environ["QT_API"] = "pyqt6" import getpass from typing import NamedTuple, Optional diff --git a/software/control/widgets.py b/software/control/widgets.py index eaf79d884..50a74ab6a 100644 --- a/software/control/widgets.py +++ b/software/control/widgets.py @@ -52,7 +52,7 @@ from squid.config import CameraPixelFormat # set QT_API environment variable -os.environ["QT_API"] = "pyqt5" +os.environ["QT_API"] = "pyqt6" # qt libraries import qtpy @@ -4682,7 +4682,7 @@ def add_components(self, show_trigger_options, show_autolevel, autolevel, stretc # connections self.entry_triggerFPS.valueChanged.connect(self.liveController.set_trigger_fps) - self.dropdown_modeSelection.activated[str].connect(self.select_new_microscope_mode_by_name) + self.dropdown_modeSelection.textActivated.connect(self.select_new_microscope_mode_by_name) self.dropdown_triggerManu.currentIndexChanged.connect(self.update_trigger_mode) self.btn_live.clicked.connect(self.toggle_live) self.btn_snap.clicked.connect(self.snap) diff --git a/software/control/widgets_mosaic.py b/software/control/widgets_mosaic.py index 17a6ae662..068e5d9aa 100644 --- a/software/control/widgets_mosaic.py +++ b/software/control/widgets_mosaic.py @@ -135,6 +135,13 @@ def __init__(self, objectiveStore, camera, contrastManager, parent=None): self.mode = _load_last_view_mode() self.layers_initialized = False + # Deferred-redraw flag. Painting this napari/vispy canvas while its tab is + # hidden fails on PyQt6 (the hidden QOpenGLWidget has an incomplete + # framebuffer → GL_INVALID_FRAMEBUFFER_OPERATION, which corrupts the shared + # GL program and cascades into "Cannot SIZE object N" on later frames). Tile + # numpy data is still written while hidden (so saves stay correct); only the + # GL refresh is deferred and flushed in showEvent once the tab is visible. + self._pending_refresh = False self.mosaic_dtype = None self.viewer_pixel_size_mm = None @@ -282,12 +289,39 @@ def setPlateLayout(self, plate_view_init): if PLATE_BOUNDARIES_LAYER in self.viewer.layers: self.viewer.layers.remove(self.viewer.layers[PLATE_BOUNDARIES_LAYER]) if canvas_changed and self.mode == DisplayMode.PLATE: - self._fit_view_to_plate() + if self._display_active(): + self._fit_view_to_plate() + else: + self._pending_refresh = True def _image_layers(self): """Iterate napari image layers, skipping shape/boundary overlays.""" return [lyr for lyr in self.viewer.layers if lyr.name not in NON_IMAGE_LAYERS and hasattr(lyr, "data")] + def _display_active(self) -> bool: + """Whether the mosaic canvas is on-screen and safe to repaint. + + A napari/vispy canvas embedded in a hidden QTabWidget page cannot be + painted on PyQt6 (the hidden QOpenGLWidget has no valid framebuffer), so + callers defer GL refreshes to showEvent when this returns False. + """ + return self.isVisible() + + def showEvent(self, event): + """Flush any GL refreshes that were deferred while the tab was hidden. + + Tile data written while hidden is already in each layer's array; here we + repaint once the framebuffer is valid so the tab shows the latest mosaic. + """ + super().showEvent(event) + if self._pending_refresh: + self._pending_refresh = False + for lyr in self._image_layers(): + lyr.refresh() + if self.mode == DisplayMode.PLATE: + self._draw_plate_boundaries() + self.resetView() + def enable_shape_drawing(self, enable): """Set Manual-ROI drawing on/off. Idempotent: the upstream signal only emits True on entry to Manual mode (never False on exit), so @@ -535,8 +569,11 @@ def updateTile(self, update): layer = self.viewer.layers[channel_name] blit_tiles_to_canvas(layer.data, [(image, y_px, x_px)]) - layer.refresh() - self._draw_plate_boundaries() + if self._display_active(): + layer.refresh() + self._draw_plate_boundaries() + else: + self._pending_refresh = True # The fit-the-whole-plate camera reset only fires when the canvas # is (re)allocated — in setPlateLayout when slot dims/coverage # change and in _create_channel_layer when a layer is created. @@ -556,6 +593,8 @@ def _update_mosaic_layer(self, layer, image, tl_x_mm, tl_y_mm, prev_top_left): mosaic_height = int(math.ceil((self.viewer_extents[1] - self.viewer_extents[0]) / self.viewer_pixel_size_mm)) mosaic_width = int(math.ceil((self.viewer_extents[3] - self.viewer_extents[2]) / self.viewer_pixel_size_mm)) + display_active = self._display_active() + if layer.data.shape[:2] != (mosaic_height, mosaic_width): y_offset = int(math.floor((prev_top_left[0] - self.top_left_coordinate[0]) / self.viewer_pixel_size_mm)) x_offset = int(math.floor((prev_top_left[1] - self.top_left_coordinate[1]) / self.viewer_pixel_size_mm)) @@ -567,14 +606,20 @@ def _update_mosaic_layer(self, layer, image, tl_x_mm, tl_y_mm, prev_top_left): x_end = min(x_offset + lyr.data.shape[1], new_data.shape[1]) new_data[y_offset:y_end, x_offset:x_end] = lyr.data[: y_end - y_offset, : x_end - x_offset] lyr.data = new_data - self.resetView() # Keep ROI vertices anchored to their stage-coordinate positions after the shift. self._update_shape_layer_position() + if display_active: + self.resetView() y_pos = int(math.floor((tl_y_mm - self.top_left_coordinate[0]) / self.viewer_pixel_size_mm)) x_pos = int(math.floor((tl_x_mm - self.top_left_coordinate[1]) / self.viewer_pixel_size_mm)) blit_tiles_to_canvas(layer.data, [(image, y_pos, x_pos)]) - layer.refresh() + # Painting a hidden tab's canvas fails on PyQt6 (see _display_active); defer + # the GL refresh to showEvent. The tile is already in layer.data either way. + if display_active: + layer.refresh() + else: + self._pending_refresh = True def _create_channel_layer(self, channel_name, reference_image): """Create a new napari image layer for a channel. @@ -613,9 +658,13 @@ def _create_channel_layer(self, channel_name, reference_image): layer.mouse_double_click_callbacks.append(self._on_double_click) # Fit the view when the first plate-sized canvas is created so the user # immediately sees the full plate; subsequent tiles preserve any - # pan/zoom they've made since. + # pan/zoom they've made since. Deferred to showEvent when the tab is + # hidden — painting a hidden canvas fails on PyQt6 (see _display_active). if self.mode == DisplayMode.PLATE and self.num_rows > 0 and self.num_cols > 0: - self._fit_view_to_plate() + if self._display_active(): + self._fit_view_to_plate() + else: + self._pending_refresh = True def _convert_image_dtype(self, image, target_dtype): """Convert image to target dtype with range scaling.""" diff --git a/software/control/widgets_usbspectrometer.py b/software/control/widgets_usbspectrometer.py index 358348eb6..e5ff9c234 100644 --- a/software/control/widgets_usbspectrometer.py +++ b/software/control/widgets_usbspectrometer.py @@ -1,7 +1,7 @@ # set QT_API environment variable import os -os.environ["QT_API"] = "pyqt5" +os.environ["QT_API"] = "pyqt6" import qtpy # qt libraries diff --git a/software/main_hcs.py b/software/main_hcs.py index dddc81816..412f3a766 100644 --- a/software/main_hcs.py +++ b/software/main_hcs.py @@ -3,11 +3,12 @@ import logging import os -os.environ["QT_API"] = "pyqt5" +os.environ["QT_API"] = "pyqt6" import signal import sys # qt libraries +from qtpy.QtCore import Qt from qtpy.QtWidgets import * from qtpy.QtGui import * @@ -54,6 +55,14 @@ ) args = parser.parse_args() + # This GUI embeds several napari viewers (Live, Multichannel, Mosaic), each + # backed by its own vispy/OpenGL canvas, in a single process. Multiple vispy + # canvases must share one OpenGL context or vispy misroutes GLIR draw commands + # between them and rendering fails with "Cannot SIZE object N because it does + # not exist" (the mosaic image never appears). This attribute MUST be set + # before the QApplication is constructed. See napari's multiple-viewer example. + QApplication.setAttribute(Qt.ApplicationAttribute.AA_ShareOpenGLContexts, True) + # Construct QApplication first so the single-instance check can show a # QMessageBox before any other startup side effects (logging, migration). app = QApplication(["Squid"]) diff --git a/software/scripts/run_acquisition.py b/software/scripts/run_acquisition.py index aa5f824ca..47e90a79b 100755 --- a/software/scripts/run_acquisition.py +++ b/software/scripts/run_acquisition.py @@ -113,7 +113,7 @@ def launch_gui(simulation: bool = False, verbose: bool = False) -> subprocess.Po cmd.append("--verbose") env = os.environ.copy() - env["QT_API"] = "pyqt5" + env["QT_API"] = "pyqt6" if verbose: print(f"Launching GUI: {' '.join(cmd)}") diff --git a/software/setup_22.04.sh b/software/setup_22.04.sh index 8e4defd68..2f784dfc7 100755 --- a/software/setup_22.04.sh +++ b/software/setup_22.04.sh @@ -37,8 +37,11 @@ sudo apt update # install packages sudo apt install python3-pip -y -sudo apt install python3-pyqtgraph python3-pyqt5 -y -sudo apt install python3-pyqt5.qtsvg +# NOTE: do NOT install apt's python3-pyqtgraph / python3-pyqt5 here. python3-pyqtgraph +# pulls in python3-pyqt5 as a dependency, and having BOTH PyQt5 and PyQt6 present in the +# same environment causes conflicting Qt libraries that break napari's OpenGL rendering +# (blank/failed canvases, "Cannot SIZE object N because it does not exist"). We install +# pyqtgraph and PyQt6 via pip below so exactly one Qt binding is present. sudo apt-get install git -y ## clone the repo if we don't already have it. @@ -54,15 +57,20 @@ fi cd "$SQUID_SOFTWARE_ROOT" mkdir -p "$SQUID_SOFTWARE_ROOT/cache" -# Ubuntu 22.04 ships pip 22.0.2, whose resolver can't handle the -# napari==0.5.4 dependency graph on current PyPI (hits ResolutionTooDeep -# after hours of backtracking). Upgrade pip before installing libraries. +# Ubuntu 22.04 ships an old pip; upgrade it before resolving the dependency graph. python3 -m pip install --upgrade pip -# install libraries -pip3 install qtpy pyserial pandas imageio crc==1.3.0 lxml "numpy<2" tifffile scipy pyreadline3 +# Qt binding: napari 0.7 requires PyQt6. Exactly ONE Qt binding may be installed — +# PyQt5 and PyQt6 in the same environment conflict and break napari/vispy OpenGL +# rendering. Remove any pre-existing PyQt5 (e.g. pulled in by an apt package) first. +sudo apt remove -y python3-pyqt5 python3-pyqt5.qtsvg 2>/dev/null || true +pip3 uninstall -y PyQt5 PyQt5-Qt5 PyQt5-sip 2>/dev/null || true +pip3 install PyQt6 PyQt6-Qt6 PyQt6-sip + +# install libraries. napari 0.7 requires numpy>=2, so no "numpy<2" pin here. +pip3 install pyqtgraph qtpy pyserial pandas imageio crc==1.3.0 lxml numpy tifffile scipy pyreadline3 pip3 install opencv-python-headless opencv-contrib-python-headless -pip3 install napari==0.5.4 scikit-image dask_image ome_zarr aicsimageio basicpy pytest pytest-qt pytest-xvfb gitpython matplotlib pydantic_xml pyvisa hidapi filelock lxml_html_clean psutil mcp ndv +pip3 install "napari>=0.7,<0.8" scikit-image dask_image ome_zarr aicsimageio basicpy pytest pytest-qt pytest-xvfb gitpython matplotlib pydantic_xml pyvisa hidapi filelock lxml_html_clean psutil mcp ndv # Optional: PI V-308 / C-414 focus stage (USE_PI_FOCUS_STAGE). Safe to skip if unused; # squid.stage.pi imports it lazily and only needs it to connect to real hardware, so diff --git a/software/setup_cuda_22.04.sh b/software/setup_cuda_22.04.sh index 744763fb3..b0047d2ba 100755 --- a/software/setup_cuda_22.04.sh +++ b/software/setup_cuda_22.04.sh @@ -10,6 +10,4 @@ pip install cuda-python pip install cupy-cuda12x pip3 install torch torchvision torchaudio -pip3 install "numpy<2" - sudo apt autoremove -y diff --git a/software/tests/control/test_single_instance.py b/software/tests/control/test_single_instance.py index 4a05435ff..e021feb81 100644 --- a/software/tests/control/test_single_instance.py +++ b/software/tests/control/test_single_instance.py @@ -71,7 +71,7 @@ def test_stale_lock_from_dead_process_is_reclaimed(tmp_path): # Pin QT_API so qtpy in the child selects the same binding as the parent # regardless of what other Qt bindings are installed in the environment. - child_env = {**os.environ, "QT_API": "pyqt5"} + child_env = {**os.environ, "QT_API": "pyqt6"} result = subprocess.run( [sys.executable, "-c", child_script], capture_output=True, diff --git a/software/tools/view_laser_af_reference_image.py b/software/tools/view_laser_af_reference_image.py index fe6507c09..ae944235a 100644 --- a/software/tools/view_laser_af_reference_image.py +++ b/software/tools/view_laser_af_reference_image.py @@ -1,11 +1,15 @@ +import os + +os.environ["QT_API"] = "pyqt6" + import sys import json import base64 import numpy as np -from PyQt5.QtWidgets import QApplication, QMainWindow, QLabel, QVBoxLayout, QWidget, QPushButton, QFileDialog -from PyQt5.QtCore import Qt +from qtpy.QtWidgets import QApplication, QMainWindow, QLabel, QVBoxLayout, QWidget, QPushButton, QFileDialog +from qtpy.QtCore import Qt import matplotlib.pyplot as plt -from matplotlib.backends.backend_qt5agg import FigureCanvasQTAgg +from matplotlib.backends.backend_qtagg import FigureCanvasQTAgg try: import yaml From d9a1c7b99f601fba7bfcd541c1f1875bd0959a73 Mon Sep 17 00:00:00 2001 From: Spencer Schwarz Date: Thu, 23 Jul 2026 02:42:21 -0400 Subject: [PATCH 02/13] Performance mode: defer mosaic rendering during acquisition, render once at completion MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit WHY --- With the Mosaic view active, rendering every tile into the napari canvas during a multi-well acquisition stalls the GUI thread (repeated GL texture rebuilds and canvas reallocations), so the run visibly stutters and slows as it progresses. See "Mosaic Memory Diagnosis.md" for the full analysis. Previously "Performance Mode" avoided this by disconnecting the mosaic/multichannel data feeds entirely — the run stayed fast, but the mosaic was never populated (the image data only existed in the saved files on disk). This changes Performance Mode so the mosaic still assembles during the run but without per-tile rendering, then renders the finished mosaic once when the run completes — you get the speed of not rendering mid-run and still see the result. WHAT CHANGED (control/gui_hcs.py) --------------------------------- 1. updateNapariConnections: in performance mode the mosaic's data feed (mosaic_tile_update -> updateTile, plate_view_init -> setPlateLayout) now stays CONNECTED, so its canvas keeps building during acquisition. Rendering is what gets suppressed: UnifiedMosaicWidget already defers all GL refresh/reset/fit work while its tab is hidden (see UnifiedMosaicWidget.showEvent / _pending_refresh), and performance mode keeps that tab hidden for the whole run. The remaining napari views (e.g. multichannel) are still disconnected in performance mode as before. 2. toggleAcquisitionStart: the start/finish hook now manages the heavy tabs in performance mode. - start: re-apply toggleNapariTabs() so the mosaic/multichannel tabs are hidden and disabled for the duration of the run (a previous run's completion may have re-enabled them), guaranteeing rendering stays deferred. - finish: re-enable the napari tabs and switch to the Mosaic view; becoming visible fires showEvent, which flushes the single deferred refresh — the one-shot render of the assembled mosaic. TEST ---- tests/control/test_performance_mode.py (new): builds the simulated GUI with the mosaic view forced on, enters performance mode, and asserts the mosaic feed remains connected, the mosaic tab is disabled during the run, and is re-enabled + made current at completion. It lives in its own module on purpose: forcing the mosaic napari.Viewer on alongside the other full-GUI tests accumulates enough napari/vispy viewers to trigger the known STATUS_HEAP_CORRUPTION on OpenGL teardown at process exit (documented; run napari-heavy GUI test files separately). Split out, it and test_HighContentScreeningGui.py each exit cleanly. NOTES / TRADE-OFFS ------------------ - Performance mode now builds the full mosaic canvas in RAM during the run (same peak as normal mode), trading the old mode's lower memory for a usable final render and no mid-run stalls. Reducing that peak is separate follow-up work (see "Mosaic Memory Diagnosis.md"). - The acquisition RAM pre-check is still skipped in performance mode (check_ram_available_with_error_dialog(..., performance_mode=...)); since the canvas is now built, re-enabling that check for performance mode is a reasonable follow-up (left unchanged here to avoid scope creep and an existing test that asserts the skip). Co-Authored-By: Claude Opus 4.8 (1M context) --- software/control/gui_hcs.py | 37 ++++++++-- .../tests/control/test_performance_mode.py | 74 +++++++++++++++++++ 2 files changed, 106 insertions(+), 5 deletions(-) create mode 100644 software/tests/control/test_performance_mode.py diff --git a/software/control/gui_hcs.py b/software/control/gui_hcs.py index a9c558ec0..a99ceae59 100644 --- a/software/control/gui_hcs.py +++ b/software/control/gui_hcs.py @@ -1671,7 +1671,14 @@ def setup_movement_updater(self): self.movement_update_timer.start() def makeNapariConnections(self): - """Collect the (signal, slot[, connection_type]) pairs that feed the napari display widgets.""" + """Collect the (signal, slot[, connection_type]) pairs that feed the napari display widgets. + + Only the multichannel view goes into ``napari_connections``, which performance mode + disconnects. The mosaic's feed is wired here for good: its canvas still builds during + acquisition, and what performance mode defers is the rendering, by keeping its tab + hidden until the run finishes (see UnifiedMosaicWidget.showEvent and + toggleAcquisitionStart). + """ self.napari_connections = [] if not self.live_only_mode: @@ -1715,11 +1722,12 @@ def makeNapariConnections(self): ), ] - # Unified mosaic/plate view connections. + # Unified mosaic/plate view connections. These stay connected in performance + # mode (see the docstring), so they are wired once here rather than registered. # plate_view_init uses Qt.QueuedConnection because it can be emitted from # the acquisition worker thread; the slot needs to run on the main thread. if self.unifiedMosaicWidget is not None: - self.napari_connections += [ + mosaic_connections = [ (self.multipointController.mosaic_tile_update, self.unifiedMosaicWidget.updateTile), (self.unifiedMosaicWidget.signal_coordinates_clicked, self.move_from_click_mm), (self.unifiedMosaicWidget.signal_clear_viewer, self.navigationViewer.clear_slide), @@ -1732,7 +1740,7 @@ def makeNapariConnections(self): # ROI shape drawing in mosaic mode (wellplate flow only). if ENABLE_WELLPLATE_MULTIPOINT: - self.napari_connections += [ + mosaic_connections += [ ( self.wellplateMultiPointWidget.signal_manual_shape_mode, self.unifiedMosaicWidget.enable_shape_drawing, @@ -1743,11 +1751,14 @@ def makeNapariConnections(self): ), ] + for signal, slot, *connection_type in mosaic_connections: + signal.connect(slot, *connection_type) + # Make initial connections self.updateNapariConnections() def updateNapariConnections(self): - """Performance mode disconnects the napari display widgets; leaving it reconnects them.""" + """Performance mode disconnects the multichannel napari view; leaving it reconnects it.""" for signal, slot, *connection_type in self.napari_connections: try: if self.performance_mode: @@ -2498,6 +2509,13 @@ def toggleAcquisitionStart(self, acquisition_started): self.live_scan_grid_was_on = False # NOTE: RAM monitor widget is connected via multipointController.signal_acquisition_start # which fires AFTER the memory monitor is created (see make_connections) + + # Performance mode: keep the mosaic/multichannel napari tabs hidden and + # disabled for the duration of the run so the mosaic canvas builds without + # rendering (deferred), avoiding the per-tile GL stalls. A previous run's + # completion may have re-enabled/shown them, so re-apply the hidden state here. + if self.performance_mode: + self.toggleNapariTabs() else: self.log.info("FINISHED ACQUISITION") if self.live_scan_grid_was_on: @@ -2505,6 +2523,15 @@ def toggleAcquisitionStart(self, acquisition_started): self.live_scan_grid_was_on = False # NOTE: RAM monitor widget is disconnected via multipointController.acquisition_finished + # Performance mode: render the assembled mosaic once, now that the run is + # done. Re-enable the napari tabs and switch to the mosaic view; becoming + # visible flushes the single deferred refresh (UnifiedMosaicWidget.showEvent). + if self.performance_mode: + for i in range(self.imageDisplayTabs.count()): + self.imageDisplayTabs.setTabEnabled(i, True) + if self.unifiedMosaicWidget is not None: + self.imageDisplayTabs.setCurrentWidget(self.unifiedMosaicWidget) + # click to move off during acquisition self.navigationWidget.set_click_to_move(not acquisition_started) diff --git a/software/tests/control/test_performance_mode.py b/software/tests/control/test_performance_mode.py new file mode 100644 index 000000000..07d0f19f0 --- /dev/null +++ b/software/tests/control/test_performance_mode.py @@ -0,0 +1,74 @@ +import pytest + +import control.gui_hcs +import control.microscope +from qtpy.QtWidgets import QMessageBox + + +@pytest.fixture +def confirm_exit_yes(monkeypatch): + """Auto-accept the 'Confirm Exit' dialog GUI shutdown shows, or teardown hangs forever.""" + + def confirm_exit(parent, title, text, *args, **kwargs): + if title == "Confirm Exit": + return QMessageBox.Yes + raise RuntimeError(f"Unexpected QMessageBox: {title} - {text}") + + monkeypatch.setattr(QMessageBox, "question", confirm_exit) + + +def test_performance_mode_defers_mosaic_and_renders_at_completion(qtbot, confirm_exit_yes, monkeypatch): + """Performance mode keeps the mosaic's data feed connected (so its canvas still + builds during acquisition) but holds its tab hidden/disabled so rendering is + deferred; on completion the mosaic tab is re-enabled and shown so the single + deferred refresh flushes. + + Lives in its own module because it forces the mosaic napari.Viewer on; grouping + it with the other full-GUI tests accumulates enough napari/vispy viewers to hit + the known STATUS_HEAP_CORRUPTION on OpenGL teardown at process exit (documented; + run napari-heavy GUI test files separately). See gui_hcs.updateNapariConnections + and gui_hcs.toggleAcquisitionStart. + """ + # gui_hcs star-imports _def; force the mosaic view on before construction so the + # widget exists regardless of this machine's cached [VIEWS] config. + monkeypatch.setattr(control.gui_hcs, "USE_NAPARI_FOR_MOSAIC_DISPLAY", True) + + scope = control.microscope.Microscope.build_from_global_config(True) + win = control.gui_hcs.HighContentScreeningGui(microscope=scope, is_simulation=True) + qtbot.add_widget(win) + + mosaic = win.unifiedMosaicWidget + assert mosaic is not None, "mosaic widget should be created when USE_NAPARI_FOR_MOSAIC_DISPLAY is on" + mosaic_idx = win.imageDisplayTabs.indexOf(mosaic) + + def mosaic_feed_connected(): + """True if the controller's mosaic_tile_update is still wired to updateTile. + Probes by disconnect (raises TypeError if not connected) then restores.""" + try: + win.multipointController.mosaic_tile_update.disconnect(mosaic.updateTile) + except TypeError: + return False + win.multipointController.mosaic_tile_update.connect(mosaic.updateTile) + return True + + assert mosaic_feed_connected(), "mosaic feed should be connected in normal mode" + + # Enter performance mode via the toggle-button path. + win.performanceModeToggle.setChecked(True) + win.togglePerformanceMode() + assert win.performance_mode is True + + # Key change: the mosaic's data feed stays CONNECTED in performance mode (so its + # canvas still builds during acquisition; rendering is what gets deferred). + assert mosaic_feed_connected(), "mosaic feed must remain connected in performance mode" + + # Start of run: the mosaic tab is hidden/disabled so rendering defers (no per-tile GL). + win.toggleAcquisitionStart(True) + assert win.imageDisplayTabs.isTabEnabled(mosaic_idx) is False + + # Completion: the mosaic tab is re-enabled and made current so its showEvent + # flushes the single deferred render. + win.toggleAcquisitionStart(False) + qtbot.wait(20) + assert win.imageDisplayTabs.isTabEnabled(mosaic_idx) is True + assert win.imageDisplayTabs.currentWidget() is mosaic From 2d309787a4b7287a082c3b1bdb091ead6590bd9d Mon Sep 17 00:00:00 2001 From: You Yan Date: Wed, 26 Aug 2026 16:08:18 -0700 Subject: [PATCH 03/13] fix(gui): make updateNapariConnections idempotent across performance-mode toggles Plain signal.connect() does not raise when the connection already exists; it adds a second one. updateNapariConnections re-ran connect() for every widget that stays connected (always the mosaic widget, and every other napari view on the way back out of performance mode), so each toggle stacked another connection and slots like updateTile ran N times per emit. Disconnect before connecting so the method is idempotent. Qt.UniqueConnection was considered but cannot be OR'd with the requested connection type (plate_view_init uses QueuedConnection) under PyQt6, where Qt.ConnectionType is a plain Enum. Adds a unit test that drives the method against a minimal stand-in and asserts exact receiver counts after repeated toggles. Co-Authored-By: Claude Fable 5 --- software/control/gui_hcs.py | 17 ++-- .../tests/control/test_napari_connections.py | 84 +++++++++++++++++++ 2 files changed, 95 insertions(+), 6 deletions(-) create mode 100644 software/tests/control/test_napari_connections.py diff --git a/software/control/gui_hcs.py b/software/control/gui_hcs.py index a99ceae59..ada72757a 100644 --- a/software/control/gui_hcs.py +++ b/software/control/gui_hcs.py @@ -1758,15 +1758,20 @@ def makeNapariConnections(self): self.updateNapariConnections() def updateNapariConnections(self): - """Performance mode disconnects the multichannel napari view; leaving it reconnects it.""" + """Performance mode disconnects the multichannel napari view; leaving it reconnects it. + + Plain connect() does not raise on an existing connection; it adds a second one, so a + repeated refresh would stack connections and the slot would run N times per emit. + Disconnect first so this is idempotent. (Qt.UniqueConnection can't be OR'd with the + requested connection type under PyQt6, where Qt.ConnectionType is a plain Enum.) + """ for signal, slot, *connection_type in self.napari_connections: try: - if self.performance_mode: - signal.disconnect(slot) - else: - signal.connect(slot, *connection_type) + signal.disconnect(slot) except TypeError: - pass # not connected / already connected + pass # not connected yet + if not self.performance_mode: + signal.connect(slot, *connection_type) def toggleNapariTabs(self): # Enable/disable Napari tabs based on performance mode diff --git a/software/tests/control/test_napari_connections.py b/software/tests/control/test_napari_connections.py new file mode 100644 index 000000000..3689b839e --- /dev/null +++ b/software/tests/control/test_napari_connections.py @@ -0,0 +1,84 @@ +"""Unit tests for HighContentScreeningGui.updateNapariConnections. + +Runs the method against a minimal stand-in object instead of the full GUI so the +tests can assert exact connection counts: toggling performance mode back and +forth must never stack duplicate connections (plain ``signal.connect`` silently +adds a second connection; it does not raise). +""" + +from qtpy.QtCore import QObject, Qt, Signal + +from control.gui_hcs import HighContentScreeningGui + + +class _Emitter(QObject): + sig = Signal(int) + queued_sig = Signal(int) + + +class _Receiver(QObject): + def __init__(self): + super().__init__() + self.calls = 0 + self.queued_calls = 0 + + def slot(self, _value): + self.calls += 1 + + def queued_slot(self, _value): + self.queued_calls += 1 + + +class _GuiStub: + """Only the attributes updateNapariConnections reads.""" + + def __init__(self, emitter, multichannel): + self.performance_mode = False + self.napari_connections = [ + (emitter.sig, multichannel.slot), + (emitter.queued_sig, multichannel.queued_slot, Qt.QueuedConnection), + ] + + def update(self, performance_mode): + self.performance_mode = performance_mode + HighContentScreeningGui.updateNapariConnections(self) + + +def _make(): + emitter, multichannel = _Emitter(), _Receiver() + return emitter, multichannel, _GuiStub(emitter, multichannel) + + +def test_repeated_refreshes_keep_exactly_one_connection(qtbot): + emitter, multichannel, gui = _make() + + gui.update(performance_mode=False) # initial wiring (makeNapariConnections) + gui.update(performance_mode=False) # a redundant refresh must not double-connect + for mode in (True, False, True, False): # user toggles performance mode a few times + gui.update(performance_mode=mode) + + emitter.sig.emit(1) + emitter.queued_sig.emit(1) + qtbot.waitUntil(lambda: multichannel.queued_calls >= 1, timeout=1000) + qtbot.wait(20) # let any stacked queued deliveries land before counting + + assert multichannel.calls == 1, "direct-connection slot must fire once per emit" + assert multichannel.queued_calls == 1, "queued-connection slot must fire once per emit" + assert emitter.receivers(emitter.sig) == 1 + assert emitter.receivers(emitter.queued_sig) == 1 + + +def test_performance_mode_disconnects_and_leaving_it_reconnects_once(qtbot): + emitter, multichannel, gui = _make() + + gui.update(performance_mode=False) + gui.update(performance_mode=True) + gui.update(performance_mode=True) # a redundant refresh must not raise + emitter.sig.emit(1) + assert multichannel.calls == 0, "multichannel must be disconnected in performance mode" + assert emitter.receivers(emitter.sig) == 0 + + gui.update(performance_mode=False) + emitter.sig.emit(1) + assert multichannel.calls == 1 + assert emitter.receivers(emitter.sig) == 1 From 7e0dcbe4577bbce5f0c8e85cf0c9f6f1ee07a4f4 Mon Sep 17 00:00:00 2001 From: You Yan Date: Wed, 26 Aug 2026 16:08:34 -0700 Subject: [PATCH 04/13] build: pin PyQt6 6.11 + napari[pyqt6]==0.7.1; run CI GUI test under pyqt6 - setup_22.04.sh: install napari via the [pyqt6] extra so pip enforces napari's Qt blocklist (napari 0.7.1 blocks PyQt6-Qt6 6.11.0/6.11.1 for dock-widget and resizing bugs, napari/napari#9052; the unpinned install put 6.11.1 on CI). Pin napari 0.7.1 / PyQt6 6.11.0 / PyQt6-Qt6 6.11.2 / PyQt6-sip 13.12.0 so installs are reproducible. Correct the comments: napari 0.7 requires neither PyQt6 nor numpy>=2 (metadata: pyqt5 extra, numpy>=1.24.2). - main.yml: the separate GUI-test step pinned QT_API/PYTEST_QT_API to pyqt5, which the setup script now uninstalls; pytest-qt aborts at startup with ModuleNotFoundError in that case. Pin to pyqt6. - widgets.py: import matplotlib's binding-agnostic backend_qtagg instead of backend_qt5agg, matching tools/view_laser_af_reference_image.py. Co-Authored-By: Claude Fable 5 --- .github/workflows/main.yml | 11 ++++++----- software/control/widgets.py | 2 +- software/setup_22.04.sh | 22 ++++++++++++++++------ 3 files changed, 23 insertions(+), 12 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index d67d86fa9..867d3e546 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -69,12 +69,13 @@ jobs: # Own invocation: the full-application GUI test leaves Qt/thread state # behind that deterministically segfaults later Qt tests (e.g. # test_channel_sequence) when run in the same pytest process. - # Step-scoped pyqt5 pin: without it, pytest-qt picks PySide6 for qtbot - # while the app code (qtpy) uses PyQt5, and the mixed bindings fail - # with a QObject double-init error. + # Step-scoped pyqt6 pin: keeps pytest-qt's qtbot on the same binding as + # the app code (qtpy, QT_API=pyqt6). Mixed bindings fail with a QObject + # double-init error, and pointing this at a binding that is not installed + # (e.g. pyqt5, removed by setup_22.04.sh) aborts pytest at startup. run: python3 -m pytest tests/control/test_HighContentScreeningGui.py working-directory: ./software env: - QT_API: pyqt5 - PYTEST_QT_API: pyqt5 + QT_API: pyqt6 + PYTEST_QT_API: pyqt6 SQUID_PYTEST_HARD_EXIT: "1" diff --git a/software/control/widgets.py b/software/control/widgets.py index 50a74ab6a..281809951 100644 --- a/software/control/widgets.py +++ b/software/control/widgets.py @@ -13869,7 +13869,7 @@ def save_settings(self): json.dump(data, f) -from matplotlib.backends.backend_qt5agg import FigureCanvasQTAgg as FigureCanvas +from matplotlib.backends.backend_qtagg import FigureCanvasQTAgg as FigureCanvas from matplotlib.figure import Figure from mpl_toolkits.mplot3d import proj3d from scipy.interpolate import griddata diff --git a/software/setup_22.04.sh b/software/setup_22.04.sh index 2f784dfc7..40497695d 100755 --- a/software/setup_22.04.sh +++ b/software/setup_22.04.sh @@ -60,17 +60,27 @@ mkdir -p "$SQUID_SOFTWARE_ROOT/cache" # Ubuntu 22.04 ships an old pip; upgrade it before resolving the dependency graph. python3 -m pip install --upgrade pip -# Qt binding: napari 0.7 requires PyQt6. Exactly ONE Qt binding may be installed — -# PyQt5 and PyQt6 in the same environment conflict and break napari/vispy OpenGL -# rendering. Remove any pre-existing PyQt5 (e.g. pulled in by an apt package) first. +# Qt binding: the app targets PyQt6 (QT_API=pyqt6; napari itself supports either binding). +# Exactly ONE Qt binding may be installed — PyQt5 and PyQt6 in the same environment +# conflict and break napari/vispy OpenGL rendering. Remove any pre-existing PyQt5 +# (e.g. pulled in by an apt package) first. +# Pinned to the tested combination. napari 0.7.1 blocks PyQt6-Qt6 6.11.0 and 6.11.1 +# (dock-widget and resizing bugs, napari/napari#9052); the napari[pyqt6] extra installed +# below makes pip re-check these pins against napari's own Qt constraints, so bump them +# together. sudo apt remove -y python3-pyqt5 python3-pyqt5.qtsvg 2>/dev/null || true pip3 uninstall -y PyQt5 PyQt5-Qt5 PyQt5-sip 2>/dev/null || true -pip3 install PyQt6 PyQt6-Qt6 PyQt6-sip +pip3 install "PyQt6==6.11.0" "PyQt6-Qt6==6.11.2" "PyQt6-sip==13.12.0" -# install libraries. napari 0.7 requires numpy>=2, so no "numpy<2" pin here. +# install libraries. No "numpy<2" pin: napari 0.7 only needs numpy>=1.24, but the rest of +# the current stack (opencv-python 5.x, pyqtgraph 0.14, ...) targets NumPy 2 and the +# pinned-back packages aicsimageio drags in (tifffile 2023.2, zarr 2.15, lxml 4.9) were +# verified to work with it. pip3 install pyqtgraph qtpy pyserial pandas imageio crc==1.3.0 lxml numpy tifffile scipy pyreadline3 pip3 install opencv-python-headless opencv-contrib-python-headless -pip3 install "napari>=0.7,<0.8" scikit-image dask_image ome_zarr aicsimageio basicpy pytest pytest-qt pytest-xvfb gitpython matplotlib pydantic_xml pyvisa hidapi filelock lxml_html_clean psutil mcp ndv +# napari pinned to a tested release (patch releases change its vispy/Qt constraints). The +# [pyqt6] extra is what makes pip enforce napari's Qt blocklist against the PyQt6 above. +pip3 install "napari[pyqt6]==0.7.1" scikit-image dask_image ome_zarr aicsimageio basicpy pytest pytest-qt pytest-xvfb gitpython matplotlib pydantic_xml pyvisa hidapi filelock lxml_html_clean psutil mcp ndv # Optional: PI V-308 / C-414 focus stage (USE_PI_FOCUS_STAGE). Safe to skip if unused; # squid.stage.pi imports it lazily and only needs it to connect to real hardware, so From e170a6fb59ace460da6be63c8f3e29766a48bea5 Mon Sep 17 00:00:00 2001 From: You Yan Date: Wed, 26 Aug 2026 16:08:34 -0700 Subject: [PATCH 05/13] docs: correct Python 3.14 compatibility claims PyQt5 installs on 3.14 (5.15.11 wheels are cp38-abi3; PyQt5-sip 12.19.0 ships cp314 wheels), pydantic v1 is not used by our napari (0.5.x already ran on pydantic 2), and napari still ships a [pyqt5] extra. Point at the actual unknowns instead: the pinned-back packages aicsimageio and basicpy pull in, and vendor SDKs. Co-Authored-By: Claude Fable 5 --- software/Python 3.14 Compatibility Check.md | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/software/Python 3.14 Compatibility Check.md b/software/Python 3.14 Compatibility Check.md index 33804f13a..8aed45baa 100644 --- a/software/Python 3.14 Compatibility Check.md +++ b/software/Python 3.14 Compatibility Check.md @@ -6,14 +6,14 @@ | numpy | Wheels for 3.14 since v2.3+ | | scipy | Supported | | pandas | Supported | -| pydantic (v2) | Supported -- but **pydantic v1** (used by old napari) is **NOT** | +| pydantic (v2) | Supported. (pydantic v1 is not, but nothing in our stack needs it: napari 0.5.x already ran on pydantic 2.x) | | Pillow | Supported | | matplotlib | Supported | | PyYAML | Supported | | filelock | Supported | | platformdirs | Supported | | scikit-image | Wheels for 3.14 since v0.26 | -| napari (latest) | Supported -- requires PyQt6, not PyQt5 | +| napari (latest) | Supported -- ships both `[pyqt5]` and `[pyqt6]` extras (0.7.x and 0.9.x); we use PyQt6 | | opencv-python-headless | Uses stable ABI (cp37-abi3), works | | opencv-contrib-python-headless | Same stable ABI, works | | torch (PyTorch) | Supported since ~2.13 | @@ -23,12 +23,13 @@ | imageio | Supported | | pyqtgraph | Pure Python, likely works | | qtpy | Pure Python abstraction layer, works | +| PyQt5 | Installs on 3.14: PyQt5 5.15.11 wheels are `cp38-abi3` and PyQt5-sip 12.19.0 ships cp314 wheels. Not a blocker -- we moved to PyQt6 for other reasons | ## Blocked / Incompatible -| Package | Issue | -|---------|-------| -| **PyQt5** | **No Python 3.14 support -- this is the main blocker** | -| **pydantic v1** | Explicitly incompatible with 3.14 (the `__slots__` error) | +None confirmed in the pip stack. Earlier drafts listed PyQt5 and pydantic v1 here; both were wrong +(see the Supported table). The likely blockers are the old pins pulled in by unmaintained packages: +aicsimageio 4.14 pins `tifffile<2023.3.15`, `zarr<2.16`, `lxml<5`; basicpy pins `scipy<1.13` +(peng-lab/BaSiCPy#173). Whether those old releases have 3.14 wheels has not been checked. ## Uncertain / Likely OK but unconfirmed | Package | Notes | @@ -49,10 +50,10 @@ ## Bottom Line -The **two hard blockers** are **PyQt5** and **pydantic v1** (via old napari). Migrating to Python 3.14 would require: -1. Switching from PyQt5 to PyQt6 (non-trivial API changes) -2. Upgrading napari to a version that uses pydantic v2 -3. Verifying all hardware vendor SDKs have 3.14 builds +No confirmed hard blocker in the pip stack. The PyQt6 + napari 0.7 migration in this PR removes the +Qt-side uncertainty; what remains before a 3.14 move is: +1. Confirming 3.14 wheels (or dropping) the pinned-back packages aicsimageio and basicpy pull in +2. Verifying all hardware vendor SDKs have 3.14 builds **Recommendation: Stay on Python 3.10** for production use. It's the sweet spot where everything works. From f18fa9b72316d3a12b1613950b135949dc0fb5cc Mon Sep 17 00:00:00 2001 From: You Yan Date: Thu, 27 Aug 2026 07:41:25 -0700 Subject: [PATCH 06/13] fix: stop the two threads the full GUI leaked past shutdown SimulatedCamera.close() was a no-op, so a streaming camera (the laser-AF focus camera, which LiveController.stop_live() only stops in CONTINUOUS trigger mode) kept its stream thread alive after Microscope.close()/camera_focus.close(). Make close() stop streaming, as owners assume it does. The GUI created a SlackNotifier worker thread and never closed it; _cleanup_common now does. Found by probing threads left behind by tests/control/test_performance_mode.py (which builds the full GUI in-process). Regression test for the camera half. Co-Authored-By: Claude Fable 5 --- software/control/gui_hcs.py | 8 ++++++++ software/squid/camera/utils.py | 2 +- software/tests/squid/test_camera.py | 15 +++++++++++++++ 3 files changed, 24 insertions(+), 1 deletion(-) diff --git a/software/control/gui_hcs.py b/software/control/gui_hcs.py index ada72757a..2126b2303 100644 --- a/software/control/gui_hcs.py +++ b/software/control/gui_hcs.py @@ -2851,6 +2851,14 @@ def _cleanup_common(self, for_restart: bool = False): except Exception: self.log.exception(f"Error closing multipoint controller during {context}") + # Stop the Slack notifier's worker thread. It is a plain daemon thread, so + # without this it outlives the window (and, in the test suite, the test). + if self.slackNotifier is not None: + try: + self.slackNotifier.close() + except Exception: + self.log.exception(f"Error closing Slack notifier during {context}") + # Clean up NDViewer if self.ndviewerTab is not None: try: diff --git a/software/squid/camera/utils.py b/software/squid/camera/utils.py index 196de07f7..7f7deb4c0 100644 --- a/software/squid/camera/utils.py +++ b/software/squid/camera/utils.py @@ -462,4 +462,4 @@ def get_frame_id(self) -> int: @debug_log def close(self): - pass + self.stop_streaming() diff --git a/software/tests/squid/test_camera.py b/software/tests/squid/test_camera.py index 0f2ba932c..0698d1006 100644 --- a/software/tests/squid/test_camera.py +++ b/software/tests/squid/test_camera.py @@ -190,3 +190,18 @@ def test_not_ready_for_trigger_while_settings_change_in_flight(): with sim_cam._trigger_lock: assert sim_cam.get_ready_for_trigger() sim_cam.stop_streaming() + + +def test_simulated_camera_close_stops_streaming_thread(): + """close() must stop a running streaming thread. Owners (Microscope.close, the GUI + shutdown path) rely on close() as the final stop; a stream thread that survives it keeps + pushing frames into callbacks whose Qt receivers are gone, which segfaulted later Qt + tests in the same pytest process.""" + sim_cam = squid.camera.utils.get_camera(squid.config.get_camera_config(), simulated=True) + sim_cam.start_streaming() + assert sim_cam.get_is_streaming() + + sim_cam.close() + + assert not sim_cam.get_is_streaming() + assert not any("stream_fn" in t.name for t in threading.enumerate()) From ae5485f0579fed578fa86a61629989b1143f4d2c Mon Sep 17 00:00:00 2001 From: You Yan Date: Thu, 27 Aug 2026 07:41:25 -0700 Subject: [PATCH 07/13] ci: run test_performance_mode.py in its own pytest process MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #606's new test builds the full HCS GUI inside the main pytest process. Destroying that GUI segfaults (pre-existing destructor-order crash: QSplitter -> QTabWidget -> QFrame -> QVBoxLayout -> QGridLayout -> ~QHBoxLayout; reproducible by flushing DeferredDelete right after the test), and pytest-qt deleteLater()s the window at teardown. A later test's event-loop activity flushes that deferred delete — test_squid_laser_engine::test_status_updated_signal did, killing the run at 67% on every CI attempt for #606/#622 and locally under PyQt5 and PyQt6 alike. Same treatment test_HighContentScreeningGui.py already gets: own step, own process, SQUID_PYTEST_HARD_EXIT. Excluding the file from the main run, the full suite completes under PyQt6 (1551 passed). Co-Authored-By: Claude Fable 5 --- .github/workflows/main.yml | 19 ++++++++++++++++++- .../tests/control/test_performance_mode.py | 7 +++++-- 2 files changed, 23 insertions(+), 3 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 867d3e546..de86cc767 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -61,7 +61,9 @@ jobs: # Python GC — the same crash main_hcs.py avoids with os._exit()). The # conftest hook hard-exits with pytest's real status once the session # is complete, so real test failures still fail the step. - run: python3 -m pytest --ignore=tests/control/test_HighContentScreeningGui.py + # Full-application GUI tests are excluded here and run below, one per + # process: see the comments on those steps. + run: python3 -m pytest --ignore=tests/control/test_HighContentScreeningGui.py --ignore=tests/control/test_performance_mode.py working-directory: ./software env: SQUID_PYTEST_HARD_EXIT: "1" @@ -79,3 +81,18 @@ jobs: QT_API: pyqt6 PYTEST_QT_API: pyqt6 SQUID_PYTEST_HARD_EXIT: "1" + - name: Run the performance-mode GUI test in its own process + # Also constructs the full HCS GUI. Destroying that GUI segfaults (the + # destructor-order crash noted above; reproducible by flushing + # DeferredDelete events right after the test). pytest-qt deleteLater()s + # the window at teardown, and a later test's event-loop activity flushes + # that deferred delete — test_squid_laser_engine's cross-thread signal + # test did, taking the whole pytest process down at 67%. Each full-GUI + # test file must therefore be the last thing its process does: own + # step, own process, hard exit. + run: python3 -m pytest tests/control/test_performance_mode.py + working-directory: ./software + env: + QT_API: pyqt6 + PYTEST_QT_API: pyqt6 + SQUID_PYTEST_HARD_EXIT: "1" diff --git a/software/tests/control/test_performance_mode.py b/software/tests/control/test_performance_mode.py index 07d0f19f0..2475a4a10 100644 --- a/software/tests/control/test_performance_mode.py +++ b/software/tests/control/test_performance_mode.py @@ -26,8 +26,11 @@ def test_performance_mode_defers_mosaic_and_renders_at_completion(qtbot, confirm Lives in its own module because it forces the mosaic napari.Viewer on; grouping it with the other full-GUI tests accumulates enough napari/vispy viewers to hit the known STATUS_HEAP_CORRUPTION on OpenGL teardown at process exit (documented; - run napari-heavy GUI test files separately). See gui_hcs.updateNapariConnections - and gui_hcs.toggleAcquisitionStart. + run napari-heavy GUI test files separately). CI also runs this file in its own + pytest process (main.yml): destroying the full GUI segfaults, and the deferred + delete pytest-qt posts for the window at teardown would otherwise be flushed + inside a later test's event loop. See gui_hcs.updateNapariConnections and + gui_hcs.toggleAcquisitionStart. """ # gui_hcs star-imports _def; force the mosaic view on before construction so the # widget exists regardless of this machine's cached [VIEWS] config. From d1e6337aa4b431abf65e60382fa15e8a011a0fad Mon Sep 17 00:00:00 2001 From: Hongquan Li Date: Sun, 6 Sep 2026 15:36:19 -0800 Subject: [PATCH 08/13] fix: unblock the test suite on non-Linux hosts; install tensorstore JobRunner.shutdown() could hang forever: after terminate()ing a worker with unread queue data, the feeder thread stays blocked writing to a pipe that no longer has a reader, and join_thread() waits on it. cancel_join_thread() first discards those bytes, which is correct when both queues are being dropped. Deterministic on Windows; latent on Linux, where a small payload happens to fit the pipe buffer and the write completes before the child is gone. test_position_caching leaked the open descriptor from tempfile.mkstemp, so the os.replace() in cache_position() was denied while a handle to the destination was still held. test_zarr_writer compared os.path.join output against POSIX literals. Those assertions are about path structure, not separator style, so normalize first. setup_22.04.sh now installs tensorstore, required by control/ndviewer_light and by test_zarr_writer, which begins with pytest.importorskip(tensorstore) -- so those ~79 zarr tests were silently skipping rather than running. --- software/control/core/job_processing.py | 15 ++++++- software/setup_22.04.sh | 4 +- .../tests/control/core/test_zarr_writer.py | 43 ++++++++++++------- software/tests/squid/test_stage.py | 7 ++- 4 files changed, 50 insertions(+), 19 deletions(-) diff --git a/software/control/core/job_processing.py b/software/control/core/job_processing.py index ec27df65a..7ca413c39 100644 --- a/software/control/core/job_processing.py +++ b/software/control/core/job_processing.py @@ -935,9 +935,22 @@ def shutdown(self, timeout_s=1.0): if self.is_alive(): self.terminate() self.join(timeout=1.0) - # Clean up multiprocessing primitives to avoid semaphore leaks + # Clean up multiprocessing primitives to avoid semaphore leaks. + # + # cancel_join_thread() must come first. A Queue feeds the underlying pipe from a + # background thread; if we terminated the worker above while it still had unread + # data queued, that thread stays blocked in a pipe write that no longer has a + # reader, and join_thread() then waits on it forever. This deadlocks reliably on + # Windows, where the pipe is a named pipe whose write blocks once the buffer fills + # and the child is gone (on Linux a small payload fits the buffer, so the write + # completes and the hang is usually not observed). + # + # Discarding those bytes is correct here: we are shutting the runner down and + # dropping both queues, so nothing will ever read them. + self._input_queue.cancel_join_thread() self._input_queue.close() self._input_queue.join_thread() + self._output_queue.cancel_join_thread() self._output_queue.close() self._output_queue.join_thread() # Clear references to allow garbage collection of Event and Value semaphores diff --git a/software/setup_22.04.sh b/software/setup_22.04.sh index 40497695d..76b81af8d 100755 --- a/software/setup_22.04.sh +++ b/software/setup_22.04.sh @@ -80,7 +80,9 @@ pip3 install pyqtgraph qtpy pyserial pandas imageio crc==1.3.0 lxml numpy tifffi pip3 install opencv-python-headless opencv-contrib-python-headless # napari pinned to a tested release (patch releases change its vispy/Qt constraints). The # [pyqt6] extra is what makes pip enforce napari's Qt blocklist against the PyQt6 above. -pip3 install "napari[pyqt6]==0.7.1" scikit-image dask_image ome_zarr aicsimageio basicpy pytest pytest-qt pytest-xvfb gitpython matplotlib pydantic_xml pyvisa hidapi filelock lxml_html_clean psutil mcp ndv +# tensorstore is required by control/ndviewer_light and by tests/control/core/test_zarr_writer.py, +# which begins with pytest.importorskip("tensorstore") -- without it ~79 zarr tests silently skip. +pip3 install "napari[pyqt6]==0.7.1" scikit-image dask_image ome_zarr tensorstore aicsimageio basicpy pytest pytest-qt pytest-xvfb gitpython matplotlib pydantic_xml pyvisa hidapi filelock lxml_html_clean psutil mcp ndv # Optional: PI V-308 / C-414 focus stage (USE_PI_FOCUS_STAGE). Safe to skip if unused; # squid.stage.pi imports it lazily and only needs it to connect to real hardware, so diff --git a/software/tests/control/core/test_zarr_writer.py b/software/tests/control/core/test_zarr_writer.py index c4e79e9ff..3999bc373 100644 --- a/software/tests/control/core/test_zarr_writer.py +++ b/software/tests/control/core/test_zarr_writer.py @@ -31,6 +31,17 @@ pytest.importorskip("tensorstore") +def as_posix_path(path: str) -> str: + """Normalize a built path to forward slashes so it can be compared to a literal. + + The path builders use os.path.join, so they emit "\\" on Windows while the expected + paths in these tests are written in the POSIX form the code produces on Linux. The + assertions are about path *structure*, not about which separator the OS uses, so + normalize before comparing rather than duplicating every expectation per platform. + """ + return path.replace(os.sep, "/") + + def make_test_capture_info( region_id: str = "A1", fov: int = 0, @@ -476,18 +487,18 @@ def test_zarr_writer_info_hcs_output_path(self): # Test single-letter row path = info.get_output_path("A1", 0) - assert path == "/tmp/experiment/plate.ome.zarr/A/1/0/0" + assert as_posix_path(path) == "/tmp/experiment/plate.ome.zarr/A/1/0/0" path = info.get_output_path("A1", 2) - assert path == "/tmp/experiment/plate.ome.zarr/A/1/2/0" + assert as_posix_path(path) == "/tmp/experiment/plate.ome.zarr/A/1/2/0" # Test multi-digit column path = info.get_output_path("B12", 2) - assert path == "/tmp/experiment/plate.ome.zarr/B/12/2/0" + assert as_posix_path(path) == "/tmp/experiment/plate.ome.zarr/B/12/2/0" # Test double-letter row (e.g., AA, AB) path = info.get_output_path("AA3", 0) - assert path == "/tmp/experiment/plate.ome.zarr/AA/3/0/0" + assert as_posix_path(path) == "/tmp/experiment/plate.ome.zarr/AA/3/0/0" def test_zarr_writer_info_non_hcs_per_fov_output_path(self): """Test non-HCS default: per-FOV zarr files (OME-NGFF compliant).""" @@ -502,12 +513,12 @@ def test_zarr_writer_info_non_hcs_per_fov_output_path(self): ) # Each FOV gets its own zarr file - get_output_path returns ARRAY path (group + /0) - assert info.get_output_path("region_1", 0) == "/tmp/experiment/zarr/region_1/fov_0.ome.zarr/0" - assert info.get_output_path("region_1", 1) == "/tmp/experiment/zarr/region_1/fov_1.ome.zarr/0" - assert info.get_output_path("region_1", 2) == "/tmp/experiment/zarr/region_1/fov_2.ome.zarr/0" + assert as_posix_path(info.get_output_path("region_1", 0)) == "/tmp/experiment/zarr/region_1/fov_0.ome.zarr/0" + assert as_posix_path(info.get_output_path("region_1", 1)) == "/tmp/experiment/zarr/region_1/fov_1.ome.zarr/0" + assert as_posix_path(info.get_output_path("region_1", 2)) == "/tmp/experiment/zarr/region_1/fov_2.ome.zarr/0" # Different region - assert info.get_output_path("region_2", 0) == "/tmp/experiment/zarr/region_2/fov_0.ome.zarr/0" + assert as_posix_path(info.get_output_path("region_2", 0)) == "/tmp/experiment/zarr/region_2/fov_0.ome.zarr/0" def test_zarr_writer_info_non_hcs_6d_output_path(self): """Test non-HCS with 6D mode: single zarr per region (non-standard).""" @@ -525,12 +536,12 @@ def test_zarr_writer_info_non_hcs_6d_output_path(self): path_fov0 = info.get_output_path("region_1", 0) path_fov1 = info.get_output_path("region_1", 1) - assert path_fov0 == "/tmp/experiment/zarr/region_1/acquisition.zarr" - assert path_fov1 == "/tmp/experiment/zarr/region_1/acquisition.zarr" + assert as_posix_path(path_fov0) == "/tmp/experiment/zarr/region_1/acquisition.zarr" + assert as_posix_path(path_fov1) == "/tmp/experiment/zarr/region_1/acquisition.zarr" # Different region path_region2 = info.get_output_path("region_2", 0) - assert path_region2 == "/tmp/experiment/zarr/region_2/acquisition.zarr" + assert as_posix_path(path_region2) == "/tmp/experiment/zarr/region_2/acquisition.zarr" def test_zarr_writer_info_get_fov_count(self): """Test get_fov_count returns correct counts for regions.""" @@ -1926,7 +1937,7 @@ def test_hcs_paths_use_shared_utility(self): for well_id, fov, expected_group_path, expected_array_path in test_cases: # Utility returns GROUP path (for NDViewer/readers) group_path = build_hcs_zarr_fov_path(base_path, well_id, fov) - assert group_path == expected_group_path, ( + assert as_posix_path(group_path) == expected_group_path, ( f"Group path mismatch for well={well_id}, fov={fov}: " f"got={group_path}, expected={expected_group_path}" ) @@ -1940,7 +1951,7 @@ def test_hcs_paths_use_shared_utility(self): is_hcs=True, ) writer_path = zarr_info.get_output_path(well_id, fov) - assert writer_path == expected_array_path, ( + assert as_posix_path(writer_path) == expected_array_path, ( f"Writer path mismatch for well={well_id}, fov={fov}: " f"writer={writer_path}, expected={expected_array_path}" ) @@ -1986,7 +1997,7 @@ def test_non_hcs_per_fov_paths_use_shared_utility(self): for region_id, fov, expected_group_path, expected_array_path in test_cases: # Utility returns GROUP path (for NDViewer/readers) group_path = build_per_fov_zarr_path(base_path, region_id, fov) - assert group_path == expected_group_path, ( + assert as_posix_path(group_path) == expected_group_path, ( f"Group path mismatch for region={region_id}, fov={fov}: " f"got={group_path}, expected={expected_group_path}" ) @@ -2001,7 +2012,7 @@ def test_non_hcs_per_fov_paths_use_shared_utility(self): use_6d_fov=False, ) writer_path = zarr_info.get_output_path(region_id, fov) - assert writer_path == expected_array_path, ( + assert as_posix_path(writer_path) == expected_array_path, ( f"Writer path mismatch for region={region_id}, fov={fov}: " f"writer={writer_path}, expected={expected_array_path}" ) @@ -2020,7 +2031,7 @@ def test_6d_paths_use_shared_utility(self): for region_id, expected_path in test_cases: # Test utility function directly util_path = build_6d_zarr_path(base_path, region_id) - assert util_path == expected_path, ( + assert as_posix_path(util_path) == expected_path, ( f"Utility path mismatch for region={region_id}: " f"got={util_path}, expected={expected_path}" ) diff --git a/software/tests/squid/test_stage.py b/software/tests/squid/test_stage.py index 02a646590..6c1a0c70e 100644 --- a/software/tests/squid/test_stage.py +++ b/software/tests/squid/test_stage.py @@ -1,5 +1,6 @@ import builtins import logging +import os import pytest import tempfile import unittest.mock @@ -30,7 +31,11 @@ def test_simulated_cephla_stage_ops(): def test_position_caching(): - (unused_temp_fd, temp_cache_path) = tempfile.mkstemp(".cache", "squid_testing_") + (temp_fd, temp_cache_path) = tempfile.mkstemp(".cache", "squid_testing_") + # mkstemp hands back an *open* descriptor. cache_position() writes a temp file and + # os.replace()s it over this path, which Windows refuses with PermissionError while + # another handle to the destination is open, so close it before the call. + os.close(temp_fd) # Use 6 figures after the decimal so we test that we can capture nanometers p = squid.abc.Pos(x_mm=11.111111, y_mm=22.222222, z_mm=1.333333, theta_rad=None) From 0d092d9acd13055dcbfef4cae0a51dfbec0c8943 Mon Sep 17 00:00:00 2001 From: hongquanli Date: Sun, 6 Sep 2026 15:54:48 -0800 Subject: [PATCH 09/13] build: stop installing basicpy/aicsimageio, which break the NumPy 2 stack (#641) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit > **Stacked on #622.** Base is `feat/pyqt6-napari07-migration-v2` so the diff stays to a single file. Retarget to `master` once #622 lands. ## Problem The current install line produces a **broken environment**. `basicpy` pins `scipy<1.13` ([peng-lab/BaSiCPy#173](https://github.com/peng-lab/BaSiCPy/issues/173)), and scipy 1.12 ships no NumPy-2 wheel — so pip resolves the whole environment back to numpy 1.26. After running `setup_22.04.sh` as written: ``` $ pip check napari 0.7.1 has requirement scipy>=1.14.0, but you have scipy 1.12.0. ome-zarr-models 1.8.0 has requirement zarr>=3.1.1, but you have zarr 2.15.0. opencv-contrib-python-headless 5.0.0.93 has requirement numpy>=2, but you have numpy 1.26.4. opencv-python-headless 5.0.0.93 has requirement numpy>=2, but you have numpy 1.26.4. pydantic-zarr 0.10.0 has requirement numpy>=2.0.0, but you have numpy 1.26.4. ``` napari then fails to import. The comment being replaced states that the pinned-back packages "were verified to work" with NumPy 2 — that does not hold once `basicpy` is in the resolution. This isn't a "prefer newer packages" cleanup: **you cannot have both** `basicpy` and a working napari in one environment, because their scipy constraints don't intersect. ## Why removal is safe `basicpy` and `aicsimageio` are imported by exactly one file, `control/stitcher.py`, and **nothing imports that module**: - no static import — `control.stitcher` / `import stitcher` match nowhere - no dynamic import — the only `__import__` uses are the MCP server's sandbox builtins and a test mock - `tools/stitcher.py` defines its **own**, unrelated `Stitcher` class and imports neither package Stitching now lives in the separate [image-stitcher](https://github.com/Cephla-Lab/image-stitcher) repo, which the README already points users to. No code is deleted: `control/stitcher.py` stays on disk and remains usable from a dedicated venv, as the new comment explains. ## Alternative If you'd rather keep it installable in-tree, a `[stitching]` extras group would work as well — happy to rework it that way. Flagging this as the judgment call in the change, since it reverses an existing claim in the script. ## Verification With these two dropped, `pip check` reports **no broken requirements**, and the suite runs **1634 passed, 0 failed, 12 skipped** under Python 3.12 / PyQt6 6.11.0 / napari 0.7.1. --- software/setup_22.04.sh | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/software/setup_22.04.sh b/software/setup_22.04.sh index 76b81af8d..f9946fa9f 100755 --- a/software/setup_22.04.sh +++ b/software/setup_22.04.sh @@ -72,17 +72,23 @@ sudo apt remove -y python3-pyqt5 python3-pyqt5.qtsvg 2>/dev/null || true pip3 uninstall -y PyQt5 PyQt5-Qt5 PyQt5-sip 2>/dev/null || true pip3 install "PyQt6==6.11.0" "PyQt6-Qt6==6.11.2" "PyQt6-sip==13.12.0" -# install libraries. No "numpy<2" pin: napari 0.7 only needs numpy>=1.24, but the rest of -# the current stack (opencv-python 5.x, pyqtgraph 0.14, ...) targets NumPy 2 and the -# pinned-back packages aicsimageio drags in (tifffile 2023.2, zarr 2.15, lxml 4.9) were -# verified to work with it. +# install libraries. No "numpy<2" pin: napari 0.7 only needs numpy>=1.24, and the rest of +# the current stack (opencv-python 5.x, pyqtgraph 0.14, ...) targets NumPy 2. +# +# aicsimageio and basicpy are deliberately NOT installed. basicpy pins scipy<1.13 +# (peng-lab/BaSiCPy#173) and scipy 1.12 ships no NumPy-2 wheel, so pip resolves the whole +# environment back to numpy 1.26 -- which then violates napari's scipy>=1.14 and +# opencv 5.x's numpy>=2, leaving `pip check` failing and napari unimportable. Their only +# consumer is control/stitcher.py, which nothing in the application imports; stitching +# lives in the separate Cephla-Lab/image-stitcher repo. If you need that module, install +# these two into a dedicated venv rather than this one. pip3 install pyqtgraph qtpy pyserial pandas imageio crc==1.3.0 lxml numpy tifffile scipy pyreadline3 pip3 install opencv-python-headless opencv-contrib-python-headless # napari pinned to a tested release (patch releases change its vispy/Qt constraints). The # [pyqt6] extra is what makes pip enforce napari's Qt blocklist against the PyQt6 above. # tensorstore is required by control/ndviewer_light and by tests/control/core/test_zarr_writer.py, # which begins with pytest.importorskip("tensorstore") -- without it ~79 zarr tests silently skip. -pip3 install "napari[pyqt6]==0.7.1" scikit-image dask_image ome_zarr tensorstore aicsimageio basicpy pytest pytest-qt pytest-xvfb gitpython matplotlib pydantic_xml pyvisa hidapi filelock lxml_html_clean psutil mcp ndv +pip3 install "napari[pyqt6]==0.7.1" scikit-image dask_image ome_zarr tensorstore pytest pytest-qt pytest-xvfb gitpython matplotlib pydantic_xml pyvisa hidapi filelock lxml_html_clean psutil mcp ndv # Optional: PI V-308 / C-414 focus stage (USE_PI_FOCUS_STAGE). Safe to skip if unused; # squid.stage.pi imports it lazily and only needs it to connect to real hardware, so From 3ec6606d36c62f355d2e149e933d5d20f7a8ecef Mon Sep 17 00:00:00 2001 From: You Yan Date: Wed, 14 Oct 2026 11:03:44 -0700 Subject: [PATCH 10/13] refactor: one deferral helper in the mosaic widget, one tab-state setter, fewer copies Cleanup pass over the rebased branch; no behaviour change except where noted. - widgets_mosaic: the five "refresh now or defer until shown" branches collapse into _refresh_or_defer(); _display_active() and the four copies of its rationale go. showEvent now replays a plate-mode flush with _fit_view_to_plate() instead of resetView(), which the visible path already uses (resetView() would be re-clamped by _on_zoom_changed and show a fraction of the plate). - gui_hcs: toggleNapariTabs() becomes _set_napari_tabs_enabled(enabled) and the end-of-run branch uses it instead of a hand-rolled loop that also re-enabled the laser-AF and fluidics tabs. The mosaic's permanent connections move to make_connections as plain connect() calls; makeNapariConnections only collects the set performance mode toggles. - test_performance_mode.py folds into test_HighContentScreeningGui.py, which already runs in its own hard-exit process and builds the same GUI ten times; the extra CI step, --ignore and the duplicated confirm_exit_yes fixture go with it. The test asserts the feed with QObject.receivers() instead of disconnecting and reconnecting the real signal, and toggles via the button. - test_napari_connections: receiver counts only (no emit/wait timing), stub is a SimpleNamespace. - job_processing: the queue teardown is a loop over the two queues with a shorter note; test_stage uses tmp_path instead of mkstemp; setup_22.04.sh installs the PyQt6 pins in the same pip call as napari[pyqt6] and says the one-binding rule once. Co-Authored-By: Claude Fable 5.1 --- .github/workflows/main.yml | 24 +---- software/control/core/job_processing.py | 28 ++---- software/control/gui_hcs.py | 93 +++++++------------ software/control/widgets_mosaic.py | 66 +++++-------- software/setup_22.04.sh | 37 +++----- .../tests/control/core/test_zarr_writer.py | 8 +- .../control/test_HighContentScreeningGui.py | 30 ++++++ .../tests/control/test_napari_connections.py | 82 +++++++--------- .../tests/control/test_performance_mode.py | 77 --------------- .../widgets_fluidics/test_gui_wiring.py | 2 +- software/tests/squid/test_stage.py | 14 +-- 11 files changed, 153 insertions(+), 308 deletions(-) delete mode 100644 software/tests/control/test_performance_mode.py diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index de86cc767..0c361207e 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -61,16 +61,17 @@ jobs: # Python GC — the same crash main_hcs.py avoids with os._exit()). The # conftest hook hard-exits with pytest's real status once the session # is complete, so real test failures still fail the step. - # Full-application GUI tests are excluded here and run below, one per - # process: see the comments on those steps. - run: python3 -m pytest --ignore=tests/control/test_HighContentScreeningGui.py --ignore=tests/control/test_performance_mode.py + run: python3 -m pytest --ignore=tests/control/test_HighContentScreeningGui.py working-directory: ./software env: SQUID_PYTEST_HARD_EXIT: "1" - name: Run the GUI test in its own process # Own invocation: the full-application GUI test leaves Qt/thread state # behind that deterministically segfaults later Qt tests (e.g. - # test_channel_sequence) when run in the same pytest process. + # test_channel_sequence) when run in the same pytest process: pytest-qt + # deleteLater()s each window at teardown, and destroying the full GUI + # crashes once a later test's event loop flushes that delete, so this + # file must be the last thing its process does (and hard-exit). # Step-scoped pyqt6 pin: keeps pytest-qt's qtbot on the same binding as # the app code (qtpy, QT_API=pyqt6). Mixed bindings fail with a QObject # double-init error, and pointing this at a binding that is not installed @@ -81,18 +82,3 @@ jobs: QT_API: pyqt6 PYTEST_QT_API: pyqt6 SQUID_PYTEST_HARD_EXIT: "1" - - name: Run the performance-mode GUI test in its own process - # Also constructs the full HCS GUI. Destroying that GUI segfaults (the - # destructor-order crash noted above; reproducible by flushing - # DeferredDelete events right after the test). pytest-qt deleteLater()s - # the window at teardown, and a later test's event-loop activity flushes - # that deferred delete — test_squid_laser_engine's cross-thread signal - # test did, taking the whole pytest process down at 67%. Each full-GUI - # test file must therefore be the last thing its process does: own - # step, own process, hard exit. - run: python3 -m pytest tests/control/test_performance_mode.py - working-directory: ./software - env: - QT_API: pyqt6 - PYTEST_QT_API: pyqt6 - SQUID_PYTEST_HARD_EXIT: "1" diff --git a/software/control/core/job_processing.py b/software/control/core/job_processing.py index 7ca413c39..408c1a78c 100644 --- a/software/control/core/job_processing.py +++ b/software/control/core/job_processing.py @@ -935,24 +935,16 @@ def shutdown(self, timeout_s=1.0): if self.is_alive(): self.terminate() self.join(timeout=1.0) - # Clean up multiprocessing primitives to avoid semaphore leaks. - # - # cancel_join_thread() must come first. A Queue feeds the underlying pipe from a - # background thread; if we terminated the worker above while it still had unread - # data queued, that thread stays blocked in a pipe write that no longer has a - # reader, and join_thread() then waits on it forever. This deadlocks reliably on - # Windows, where the pipe is a named pipe whose write blocks once the buffer fills - # and the child is gone (on Linux a small payload fits the buffer, so the write - # completes and the hang is usually not observed). - # - # Discarding those bytes is correct here: we are shutting the runner down and - # dropping both queues, so nothing will ever read them. - self._input_queue.cancel_join_thread() - self._input_queue.close() - self._input_queue.join_thread() - self._output_queue.cancel_join_thread() - self._output_queue.close() - self._output_queue.join_thread() + # Clean up multiprocessing primitives to avoid semaphore leaks. cancel_join_thread() + # first: a Queue feeds its pipe from a background thread, and if the worker was + # terminated with unread data queued that thread is stuck in a write nobody will + # read, so join_thread() would wait forever (deterministic on Windows, where the + # write blocks once the pipe buffer fills). Both queues are being dropped, so + # discarding those bytes is fine. + for queue in (self._input_queue, self._output_queue): + queue.cancel_join_thread() + queue.close() + queue.join_thread() # Clear references to allow garbage collection of Event and Value semaphores self._input_queue = None self._output_queue = None diff --git a/software/control/gui_hcs.py b/software/control/gui_hcs.py index 2126b2303..d7efa882f 100644 --- a/software/control/gui_hcs.py +++ b/software/control/gui_hcs.py @@ -1496,10 +1496,26 @@ def make_connections(self): self.movement_updater.piezo_z_um.connect(self.piezoWidget.update_displacement_um_display) self.multipointController.signal_set_display_tabs.connect(self.setAcquisitionDisplayTabs) - # Unified mosaic widget save hooks: route the controller's start/finish - # signals over here, where self.unifiedMosaicWidget is reachable. + # Unified mosaic widget: save hooks and its data feed. The feed is not part of + # napari_connections because performance mode keeps it connected: the canvas still + # builds during the run and only its redraw is deferred, by keeping the tab hidden + # until the run finishes (UnifiedMosaicWidget.showEvent, toggleAcquisitionStart). self.multipointController.signal_acquisition_save_target.connect(self._on_acquisition_save_target) self.multipointController.timepoint_finished.connect(self._on_timepoint_finished) + if self.unifiedMosaicWidget is not None: + self.multipointController.mosaic_tile_update.connect(self.unifiedMosaicWidget.updateTile) + self.unifiedMosaicWidget.signal_coordinates_clicked.connect(self.move_from_click_mm) + self.unifiedMosaicWidget.signal_clear_viewer.connect(self.navigationViewer.clear_slide) + # plate_view_init can be emitted from the acquisition worker thread; the slot + # needs to run on the main thread. + self.multipointController.plate_view_init.connect( + self.unifiedMosaicWidget.setPlateLayout, Qt.QueuedConnection + ) + if ENABLE_WELLPLATE_MULTIPOINT: # ROI shape drawing in mosaic mode + self.wellplateMultiPointWidget.signal_manual_shape_mode.connect( + self.unifiedMosaicWidget.enable_shape_drawing + ) + self.unifiedMosaicWidget.signal_shape_drawn.connect(self.wellplateMultiPointWidget.update_manual_shape) # Laser engine readiness gate — modal progress dialog while waiting at timepoint boundaries. self.multipointController.signal_laser_engine_waiting.connect(self._show_laser_engine_dialog) @@ -1671,13 +1687,9 @@ def setup_movement_updater(self): self.movement_update_timer.start() def makeNapariConnections(self): - """Collect the (signal, slot[, connection_type]) pairs that feed the napari display widgets. + """Collect the (signal, slot[, connection_type]) pairs that performance mode disconnects. - Only the multichannel view goes into ``napari_connections``, which performance mode - disconnects. The mosaic's feed is wired here for good: its canvas still builds during - acquisition, and what performance mode defers is the rendering, by keeping its tab - hidden until the run finishes (see UnifiedMosaicWidget.showEvent and - toggleAcquisitionStart). + That is the multichannel view's feed; the mosaic's is permanent (see make_connections). """ self.napari_connections = [] @@ -1722,48 +1734,15 @@ def makeNapariConnections(self): ), ] - # Unified mosaic/plate view connections. These stay connected in performance - # mode (see the docstring), so they are wired once here rather than registered. - # plate_view_init uses Qt.QueuedConnection because it can be emitted from - # the acquisition worker thread; the slot needs to run on the main thread. - if self.unifiedMosaicWidget is not None: - mosaic_connections = [ - (self.multipointController.mosaic_tile_update, self.unifiedMosaicWidget.updateTile), - (self.unifiedMosaicWidget.signal_coordinates_clicked, self.move_from_click_mm), - (self.unifiedMosaicWidget.signal_clear_viewer, self.navigationViewer.clear_slide), - ( - self.multipointController.plate_view_init, - self.unifiedMosaicWidget.setPlateLayout, - Qt.QueuedConnection, - ), - ] - - # ROI shape drawing in mosaic mode (wellplate flow only). - if ENABLE_WELLPLATE_MULTIPOINT: - mosaic_connections += [ - ( - self.wellplateMultiPointWidget.signal_manual_shape_mode, - self.unifiedMosaicWidget.enable_shape_drawing, - ), - ( - self.unifiedMosaicWidget.signal_shape_drawn, - self.wellplateMultiPointWidget.update_manual_shape, - ), - ] - - for signal, slot, *connection_type in mosaic_connections: - signal.connect(slot, *connection_type) - # Make initial connections self.updateNapariConnections() def updateNapariConnections(self): """Performance mode disconnects the multichannel napari view; leaving it reconnects it. - Plain connect() does not raise on an existing connection; it adds a second one, so a - repeated refresh would stack connections and the slot would run N times per emit. - Disconnect first so this is idempotent. (Qt.UniqueConnection can't be OR'd with the - requested connection type under PyQt6, where Qt.ConnectionType is a plain Enum.) + Disconnect first so repeated calls are idempotent: plain connect() stacks a second + connection instead of raising. (Qt.UniqueConnection would do this but can't be OR'd + with the requested connection type under PyQt6.) """ for signal, slot, *connection_type in self.napari_connections: try: @@ -1773,13 +1752,13 @@ def updateNapariConnections(self): if not self.performance_mode: signal.connect(slot, *connection_type) - def toggleNapariTabs(self): - # Enable/disable Napari tabs based on performance mode + def _set_napari_tabs_enabled(self, enabled: bool): + """Enable or disable the napari display tabs; disabling also returns to Live View.""" for i in range(1, self.imageDisplayTabs.count()): if self.imageDisplayTabs.tabText(i) not in (self.LASER_BASED_FOCUS_TAB_NAME, self.FLUIDICS_TAB_NAME): - self.imageDisplayTabs.setTabEnabled(i, not self.performance_mode) + self.imageDisplayTabs.setTabEnabled(i, enabled) - if self.performance_mode: + if not enabled: self.imageDisplayTabs.setCurrentIndex(0) # Live View def togglePerformanceMode(self): @@ -1787,7 +1766,7 @@ def togglePerformanceMode(self): button_txt = "Disable" if self.performance_mode else "Enable" self.performanceModeToggle.setText(button_txt + " Performance Mode") self.updateNapariConnections() - self.toggleNapariTabs() + self._set_napari_tabs_enabled(not self.performance_mode) self.signal_performance_mode_changed.emit(self.performance_mode) print(f"Performance mode {'enabled' if self.performance_mode else 'disabled'}") @@ -2515,12 +2494,10 @@ def toggleAcquisitionStart(self, acquisition_started): # NOTE: RAM monitor widget is connected via multipointController.signal_acquisition_start # which fires AFTER the memory monitor is created (see make_connections) - # Performance mode: keep the mosaic/multichannel napari tabs hidden and - # disabled for the duration of the run so the mosaic canvas builds without - # rendering (deferred), avoiding the per-tile GL stalls. A previous run's - # completion may have re-enabled/shown them, so re-apply the hidden state here. + # Performance mode: hide the napari tabs for the run so the mosaic canvas builds + # without rendering (the previous run's completion re-enabled them). if self.performance_mode: - self.toggleNapariTabs() + self._set_napari_tabs_enabled(False) else: self.log.info("FINISHED ACQUISITION") if self.live_scan_grid_was_on: @@ -2528,12 +2505,10 @@ def toggleAcquisitionStart(self, acquisition_started): self.live_scan_grid_was_on = False # NOTE: RAM monitor widget is disconnected via multipointController.acquisition_finished - # Performance mode: render the assembled mosaic once, now that the run is - # done. Re-enable the napari tabs and switch to the mosaic view; becoming - # visible flushes the single deferred refresh (UnifiedMosaicWidget.showEvent). + # Performance mode: show the assembled mosaic once, now that the run is done; + # becoming visible flushes its deferred redraw (UnifiedMosaicWidget.showEvent). if self.performance_mode: - for i in range(self.imageDisplayTabs.count()): - self.imageDisplayTabs.setTabEnabled(i, True) + self._set_napari_tabs_enabled(True) if self.unifiedMosaicWidget is not None: self.imageDisplayTabs.setCurrentWidget(self.unifiedMosaicWidget) diff --git a/software/control/widgets_mosaic.py b/software/control/widgets_mosaic.py index 068e5d9aa..0eeb86c09 100644 --- a/software/control/widgets_mosaic.py +++ b/software/control/widgets_mosaic.py @@ -135,13 +135,7 @@ def __init__(self, objectiveStore, camera, contrastManager, parent=None): self.mode = _load_last_view_mode() self.layers_initialized = False - # Deferred-redraw flag. Painting this napari/vispy canvas while its tab is - # hidden fails on PyQt6 (the hidden QOpenGLWidget has an incomplete - # framebuffer → GL_INVALID_FRAMEBUFFER_OPERATION, which corrupts the shared - # GL program and cascades into "Cannot SIZE object N" on later frames). Tile - # numpy data is still written while hidden (so saves stay correct); only the - # GL refresh is deferred and flushed in showEvent once the tab is visible. - self._pending_refresh = False + self._pending_refresh = False # a redraw was skipped while hidden; see _refresh_or_defer self.mosaic_dtype = None self.viewer_pixel_size_mm = None @@ -289,30 +283,27 @@ def setPlateLayout(self, plate_view_init): if PLATE_BOUNDARIES_LAYER in self.viewer.layers: self.viewer.layers.remove(self.viewer.layers[PLATE_BOUNDARIES_LAYER]) if canvas_changed and self.mode == DisplayMode.PLATE: - if self._display_active(): - self._fit_view_to_plate() - else: - self._pending_refresh = True + self._refresh_or_defer(self._fit_view_to_plate) def _image_layers(self): """Iterate napari image layers, skipping shape/boundary overlays.""" return [lyr for lyr in self.viewer.layers if lyr.name not in NON_IMAGE_LAYERS and hasattr(lyr, "data")] - def _display_active(self) -> bool: - """Whether the mosaic canvas is on-screen and safe to repaint. + def _refresh_or_defer(self, gl_call): + """Run a GL-touching call now, or once the tab is next shown. - A napari/vispy canvas embedded in a hidden QTabWidget page cannot be - painted on PyQt6 (the hidden QOpenGLWidget has no valid framebuffer), so - callers defer GL refreshes to showEvent when this returns False. + Painting this napari/vispy canvas while its tab is hidden fails on PyQt6: the + hidden QOpenGLWidget has no valid framebuffer, which corrupts the shared GL + program and cascades into "Cannot SIZE object N" on later frames. Tile data is + written to the layers regardless (so saves stay correct); only the redraw waits + for showEvent, which replays every kind of deferred redraw at once. """ - return self.isVisible() + if self.isVisible(): + gl_call() + else: + self._pending_refresh = True def showEvent(self, event): - """Flush any GL refreshes that were deferred while the tab was hidden. - - Tile data written while hidden is already in each layer's array; here we - repaint once the framebuffer is valid so the tab shows the latest mosaic. - """ super().showEvent(event) if self._pending_refresh: self._pending_refresh = False @@ -320,7 +311,9 @@ def showEvent(self, event): lyr.refresh() if self.mode == DisplayMode.PLATE: self._draw_plate_boundaries() - self.resetView() + self._fit_view_to_plate() # not resetView(): bypasses the plate zoom clamp + else: + self.resetView() def enable_shape_drawing(self, enable): """Set Manual-ROI drawing on/off. Idempotent: the upstream signal @@ -569,11 +562,8 @@ def updateTile(self, update): layer = self.viewer.layers[channel_name] blit_tiles_to_canvas(layer.data, [(image, y_px, x_px)]) - if self._display_active(): - layer.refresh() - self._draw_plate_boundaries() - else: - self._pending_refresh = True + self._refresh_or_defer(layer.refresh) + self._refresh_or_defer(self._draw_plate_boundaries) # The fit-the-whole-plate camera reset only fires when the canvas # is (re)allocated — in setPlateLayout when slot dims/coverage # change and in _create_channel_layer when a layer is created. @@ -593,8 +583,6 @@ def _update_mosaic_layer(self, layer, image, tl_x_mm, tl_y_mm, prev_top_left): mosaic_height = int(math.ceil((self.viewer_extents[1] - self.viewer_extents[0]) / self.viewer_pixel_size_mm)) mosaic_width = int(math.ceil((self.viewer_extents[3] - self.viewer_extents[2]) / self.viewer_pixel_size_mm)) - display_active = self._display_active() - if layer.data.shape[:2] != (mosaic_height, mosaic_width): y_offset = int(math.floor((prev_top_left[0] - self.top_left_coordinate[0]) / self.viewer_pixel_size_mm)) x_offset = int(math.floor((prev_top_left[1] - self.top_left_coordinate[1]) / self.viewer_pixel_size_mm)) @@ -608,18 +596,12 @@ def _update_mosaic_layer(self, layer, image, tl_x_mm, tl_y_mm, prev_top_left): lyr.data = new_data # Keep ROI vertices anchored to their stage-coordinate positions after the shift. self._update_shape_layer_position() - if display_active: - self.resetView() + self._refresh_or_defer(self.resetView) y_pos = int(math.floor((tl_y_mm - self.top_left_coordinate[0]) / self.viewer_pixel_size_mm)) x_pos = int(math.floor((tl_x_mm - self.top_left_coordinate[1]) / self.viewer_pixel_size_mm)) blit_tiles_to_canvas(layer.data, [(image, y_pos, x_pos)]) - # Painting a hidden tab's canvas fails on PyQt6 (see _display_active); defer - # the GL refresh to showEvent. The tile is already in layer.data either way. - if display_active: - layer.refresh() - else: - self._pending_refresh = True + self._refresh_or_defer(layer.refresh) def _create_channel_layer(self, channel_name, reference_image): """Create a new napari image layer for a channel. @@ -658,13 +640,9 @@ def _create_channel_layer(self, channel_name, reference_image): layer.mouse_double_click_callbacks.append(self._on_double_click) # Fit the view when the first plate-sized canvas is created so the user # immediately sees the full plate; subsequent tiles preserve any - # pan/zoom they've made since. Deferred to showEvent when the tab is - # hidden — painting a hidden canvas fails on PyQt6 (see _display_active). + # pan/zoom they've made since. if self.mode == DisplayMode.PLATE and self.num_rows > 0 and self.num_cols > 0: - if self._display_active(): - self._fit_view_to_plate() - else: - self._pending_refresh = True + self._refresh_or_defer(self._fit_view_to_plate) def _convert_image_dtype(self, image, target_dtype): """Convert image to target dtype with range scaling.""" diff --git a/software/setup_22.04.sh b/software/setup_22.04.sh index f9946fa9f..bd3e3dee4 100755 --- a/software/setup_22.04.sh +++ b/software/setup_22.04.sh @@ -37,11 +37,8 @@ sudo apt update # install packages sudo apt install python3-pip -y -# NOTE: do NOT install apt's python3-pyqtgraph / python3-pyqt5 here. python3-pyqtgraph -# pulls in python3-pyqt5 as a dependency, and having BOTH PyQt5 and PyQt6 present in the -# same environment causes conflicting Qt libraries that break napari's OpenGL rendering -# (blank/failed canvases, "Cannot SIZE object N because it does not exist"). We install -# pyqtgraph and PyQt6 via pip below so exactly one Qt binding is present. +# Do not apt-install python3-pyqtgraph: it pulls in python3-pyqt5 (see the Qt-binding note +# below). pyqtgraph comes from pip. sudo apt-get install git -y ## clone the repo if we don't already have it. @@ -60,35 +57,27 @@ mkdir -p "$SQUID_SOFTWARE_ROOT/cache" # Ubuntu 22.04 ships an old pip; upgrade it before resolving the dependency graph. python3 -m pip install --upgrade pip -# Qt binding: the app targets PyQt6 (QT_API=pyqt6; napari itself supports either binding). -# Exactly ONE Qt binding may be installed — PyQt5 and PyQt6 in the same environment -# conflict and break napari/vispy OpenGL rendering. Remove any pre-existing PyQt5 -# (e.g. pulled in by an apt package) first. -# Pinned to the tested combination. napari 0.7.1 blocks PyQt6-Qt6 6.11.0 and 6.11.1 -# (dock-widget and resizing bugs, napari/napari#9052); the napari[pyqt6] extra installed -# below makes pip re-check these pins against napari's own Qt constraints, so bump them -# together. +# Qt binding: the app targets PyQt6 (QT_API=pyqt6). Exactly ONE Qt binding may be installed: +# PyQt5 and PyQt6 side by side break napari/vispy OpenGL rendering (blank canvases, "Cannot +# SIZE object N because it does not exist"), so remove any pre-existing PyQt5 first. sudo apt remove -y python3-pyqt5 python3-pyqt5.qtsvg 2>/dev/null || true pip3 uninstall -y PyQt5 PyQt5-Qt5 PyQt5-sip 2>/dev/null || true -pip3 install "PyQt6==6.11.0" "PyQt6-Qt6==6.11.2" "PyQt6-sip==13.12.0" # install libraries. No "numpy<2" pin: napari 0.7 only needs numpy>=1.24, and the rest of # the current stack (opencv-python 5.x, pyqtgraph 0.14, ...) targets NumPy 2. # -# aicsimageio and basicpy are deliberately NOT installed. basicpy pins scipy<1.13 -# (peng-lab/BaSiCPy#173) and scipy 1.12 ships no NumPy-2 wheel, so pip resolves the whole -# environment back to numpy 1.26 -- which then violates napari's scipy>=1.14 and -# opencv 5.x's numpy>=2, leaving `pip check` failing and napari unimportable. Their only -# consumer is control/stitcher.py, which nothing in the application imports; stitching -# lives in the separate Cephla-Lab/image-stitcher repo. If you need that module, install -# these two into a dedicated venv rather than this one. +# aicsimageio and basicpy are deliberately NOT installed: basicpy pins scipy<1.13, which +# drags the environment back to numpy 1.26 and breaks napari and opencv (see the commit +# "stop installing basicpy/aicsimageio"). Their only consumer, control/stitcher.py, is not +# imported by the application; install them into a separate venv if you need it. pip3 install pyqtgraph qtpy pyserial pandas imageio crc==1.3.0 lxml numpy tifffile scipy pyreadline3 pip3 install opencv-python-headless opencv-contrib-python-headless -# napari pinned to a tested release (patch releases change its vispy/Qt constraints). The -# [pyqt6] extra is what makes pip enforce napari's Qt blocklist against the PyQt6 above. +# napari and PyQt6 pinned to the tested combination and resolved in one pip call, so the +# [pyqt6] extra checks the PyQt6 pins against napari's Qt blocklist (napari 0.7.1 blocks +# PyQt6-Qt6 6.11.0 and 6.11.1, napari/napari#9052): bump them together. # tensorstore is required by control/ndviewer_light and by tests/control/core/test_zarr_writer.py, # which begins with pytest.importorskip("tensorstore") -- without it ~79 zarr tests silently skip. -pip3 install "napari[pyqt6]==0.7.1" scikit-image dask_image ome_zarr tensorstore pytest pytest-qt pytest-xvfb gitpython matplotlib pydantic_xml pyvisa hidapi filelock lxml_html_clean psutil mcp ndv +pip3 install "napari[pyqt6]==0.7.1" "PyQt6==6.11.0" "PyQt6-Qt6==6.11.2" "PyQt6-sip==13.12.0" scikit-image dask_image ome_zarr tensorstore pytest pytest-qt pytest-xvfb gitpython matplotlib pydantic_xml pyvisa hidapi filelock lxml_html_clean psutil mcp ndv # Optional: PI V-308 / C-414 focus stage (USE_PI_FOCUS_STAGE). Safe to skip if unused; # squid.stage.pi imports it lazily and only needs it to connect to real hardware, so diff --git a/software/tests/control/core/test_zarr_writer.py b/software/tests/control/core/test_zarr_writer.py index 3999bc373..3890b526e 100644 --- a/software/tests/control/core/test_zarr_writer.py +++ b/software/tests/control/core/test_zarr_writer.py @@ -32,13 +32,7 @@ def as_posix_path(path: str) -> str: - """Normalize a built path to forward slashes so it can be compared to a literal. - - The path builders use os.path.join, so they emit "\\" on Windows while the expected - paths in these tests are written in the POSIX form the code produces on Linux. The - assertions are about path *structure*, not about which separator the OS uses, so - normalize before comparing rather than duplicating every expectation per platform. - """ + """The builders use os.path.join; compare path structure, not the OS separator.""" return path.replace(os.sep, "/") diff --git a/software/tests/control/test_HighContentScreeningGui.py b/software/tests/control/test_HighContentScreeningGui.py index 7e8cd5eaf..148b2158a 100644 --- a/software/tests/control/test_HighContentScreeningGui.py +++ b/software/tests/control/test_HighContentScreeningGui.py @@ -240,3 +240,33 @@ def test_pi_focus_stage_is_a_referenced_z_even_without_cephla_z_homing(qtbot, mo cache_position.assert_called_once() assert cache_position.call_args.kwargs["validate_z"] is True + + +def test_performance_mode_keeps_mosaic_fed_and_shows_it_at_completion(qtbot, monkeypatch, confirm_exit_yes): + """Performance mode keeps the mosaic's data feed connected (its canvas builds during the + run) and defers only the rendering: the tab is disabled for the run, then re-enabled and + shown at completion, which flushes the deferred redraw (UnifiedMosaicWidget.showEvent).""" + # gui_hcs star-imports _def; force the mosaic view on so the widget exists regardless + # of this machine's cached [VIEWS] config. + monkeypatch.setattr(control.gui_hcs, "USE_NAPARI_FOR_MOSAIC_DISPLAY", True) + + scope = control.microscope.Microscope.build_from_global_config(True) + win = control.gui_hcs.HighContentScreeningGui(microscope=scope, is_simulation=True) + qtbot.add_widget(win) + + mosaic = win.unifiedMosaicWidget + assert mosaic is not None, "mosaic widget should be created when USE_NAPARI_FOR_MOSAIC_DISPLAY is on" + mosaic_idx = win.imageDisplayTabs.indexOf(mosaic) + controller = win.multipointController + assert controller.receivers(controller.mosaic_tile_update) == 1 + + win.performanceModeToggle.click() + assert win.performance_mode is True + assert controller.receivers(controller.mosaic_tile_update) == 1, "mosaic feed must stay connected" + + win.toggleAcquisitionStart(True) + assert not win.imageDisplayTabs.isTabEnabled(mosaic_idx) + + win.toggleAcquisitionStart(False) + assert win.imageDisplayTabs.isTabEnabled(mosaic_idx) + assert win.imageDisplayTabs.currentWidget() is mosaic diff --git a/software/tests/control/test_napari_connections.py b/software/tests/control/test_napari_connections.py index 3689b839e..ca5715d04 100644 --- a/software/tests/control/test_napari_connections.py +++ b/software/tests/control/test_napari_connections.py @@ -1,11 +1,12 @@ """Unit tests for HighContentScreeningGui.updateNapariConnections. -Runs the method against a minimal stand-in object instead of the full GUI so the -tests can assert exact connection counts: toggling performance mode back and -forth must never stack duplicate connections (plain ``signal.connect`` silently -adds a second connection; it does not raise). +Runs the method against a stand-in object instead of the full GUI so the tests can +assert exact receiver counts: toggling performance mode back and forth must never stack +duplicate connections (plain ``signal.connect`` silently adds a second one; it does not raise). """ +from types import SimpleNamespace + from qtpy.QtCore import QObject, Qt, Signal from control.gui_hcs import HighContentScreeningGui @@ -16,69 +17,52 @@ class _Emitter(QObject): queued_sig = Signal(int) -class _Receiver(QObject): - def __init__(self): - super().__init__() - self.calls = 0 - self.queued_calls = 0 - +class _View(QObject): def slot(self, _value): - self.calls += 1 + pass def queued_slot(self, _value): - self.queued_calls += 1 - - -class _GuiStub: - """Only the attributes updateNapariConnections reads.""" + pass - def __init__(self, emitter, multichannel): - self.performance_mode = False - self.napari_connections = [ - (emitter.sig, multichannel.slot), - (emitter.queued_sig, multichannel.queued_slot, Qt.QueuedConnection), - ] - def update(self, performance_mode): - self.performance_mode = performance_mode - HighContentScreeningGui.updateNapariConnections(self) +def _make(): + emitter, view = _Emitter(), _View() + gui = SimpleNamespace( + performance_mode=False, + napari_connections=[ + (emitter.sig, view.slot), + (emitter.queued_sig, view.queued_slot, Qt.QueuedConnection), + ], + ) + return emitter, view, gui -def _make(): - emitter, multichannel = _Emitter(), _Receiver() - return emitter, multichannel, _GuiStub(emitter, multichannel) +def _update(gui, performance_mode): + gui.performance_mode = performance_mode + HighContentScreeningGui.updateNapariConnections(gui) def test_repeated_refreshes_keep_exactly_one_connection(qtbot): - emitter, multichannel, gui = _make() + emitter, _view, gui = _make() - gui.update(performance_mode=False) # initial wiring (makeNapariConnections) - gui.update(performance_mode=False) # a redundant refresh must not double-connect - for mode in (True, False, True, False): # user toggles performance mode a few times - gui.update(performance_mode=mode) + _update(gui, performance_mode=False) # initial wiring (makeNapariConnections) + _update(gui, performance_mode=False) # a redundant refresh must not double-connect + for mode in (True, False, True, False): # the user toggles performance mode a few times + _update(gui, performance_mode=mode) - emitter.sig.emit(1) - emitter.queued_sig.emit(1) - qtbot.waitUntil(lambda: multichannel.queued_calls >= 1, timeout=1000) - qtbot.wait(20) # let any stacked queued deliveries land before counting - - assert multichannel.calls == 1, "direct-connection slot must fire once per emit" - assert multichannel.queued_calls == 1, "queued-connection slot must fire once per emit" assert emitter.receivers(emitter.sig) == 1 assert emitter.receivers(emitter.queued_sig) == 1 def test_performance_mode_disconnects_and_leaving_it_reconnects_once(qtbot): - emitter, multichannel, gui = _make() + emitter, _view, gui = _make() - gui.update(performance_mode=False) - gui.update(performance_mode=True) - gui.update(performance_mode=True) # a redundant refresh must not raise - emitter.sig.emit(1) - assert multichannel.calls == 0, "multichannel must be disconnected in performance mode" + _update(gui, performance_mode=False) + _update(gui, performance_mode=True) + _update(gui, performance_mode=True) # a redundant refresh must not raise assert emitter.receivers(emitter.sig) == 0 + assert emitter.receivers(emitter.queued_sig) == 0 - gui.update(performance_mode=False) - emitter.sig.emit(1) - assert multichannel.calls == 1 + _update(gui, performance_mode=False) assert emitter.receivers(emitter.sig) == 1 + assert emitter.receivers(emitter.queued_sig) == 1 diff --git a/software/tests/control/test_performance_mode.py b/software/tests/control/test_performance_mode.py deleted file mode 100644 index 2475a4a10..000000000 --- a/software/tests/control/test_performance_mode.py +++ /dev/null @@ -1,77 +0,0 @@ -import pytest - -import control.gui_hcs -import control.microscope -from qtpy.QtWidgets import QMessageBox - - -@pytest.fixture -def confirm_exit_yes(monkeypatch): - """Auto-accept the 'Confirm Exit' dialog GUI shutdown shows, or teardown hangs forever.""" - - def confirm_exit(parent, title, text, *args, **kwargs): - if title == "Confirm Exit": - return QMessageBox.Yes - raise RuntimeError(f"Unexpected QMessageBox: {title} - {text}") - - monkeypatch.setattr(QMessageBox, "question", confirm_exit) - - -def test_performance_mode_defers_mosaic_and_renders_at_completion(qtbot, confirm_exit_yes, monkeypatch): - """Performance mode keeps the mosaic's data feed connected (so its canvas still - builds during acquisition) but holds its tab hidden/disabled so rendering is - deferred; on completion the mosaic tab is re-enabled and shown so the single - deferred refresh flushes. - - Lives in its own module because it forces the mosaic napari.Viewer on; grouping - it with the other full-GUI tests accumulates enough napari/vispy viewers to hit - the known STATUS_HEAP_CORRUPTION on OpenGL teardown at process exit (documented; - run napari-heavy GUI test files separately). CI also runs this file in its own - pytest process (main.yml): destroying the full GUI segfaults, and the deferred - delete pytest-qt posts for the window at teardown would otherwise be flushed - inside a later test's event loop. See gui_hcs.updateNapariConnections and - gui_hcs.toggleAcquisitionStart. - """ - # gui_hcs star-imports _def; force the mosaic view on before construction so the - # widget exists regardless of this machine's cached [VIEWS] config. - monkeypatch.setattr(control.gui_hcs, "USE_NAPARI_FOR_MOSAIC_DISPLAY", True) - - scope = control.microscope.Microscope.build_from_global_config(True) - win = control.gui_hcs.HighContentScreeningGui(microscope=scope, is_simulation=True) - qtbot.add_widget(win) - - mosaic = win.unifiedMosaicWidget - assert mosaic is not None, "mosaic widget should be created when USE_NAPARI_FOR_MOSAIC_DISPLAY is on" - mosaic_idx = win.imageDisplayTabs.indexOf(mosaic) - - def mosaic_feed_connected(): - """True if the controller's mosaic_tile_update is still wired to updateTile. - Probes by disconnect (raises TypeError if not connected) then restores.""" - try: - win.multipointController.mosaic_tile_update.disconnect(mosaic.updateTile) - except TypeError: - return False - win.multipointController.mosaic_tile_update.connect(mosaic.updateTile) - return True - - assert mosaic_feed_connected(), "mosaic feed should be connected in normal mode" - - # Enter performance mode via the toggle-button path. - win.performanceModeToggle.setChecked(True) - win.togglePerformanceMode() - assert win.performance_mode is True - - # Key change: the mosaic's data feed stays CONNECTED in performance mode (so its - # canvas still builds during acquisition; rendering is what gets deferred). - assert mosaic_feed_connected(), "mosaic feed must remain connected in performance mode" - - # Start of run: the mosaic tab is hidden/disabled so rendering defers (no per-tile GL). - win.toggleAcquisitionStart(True) - assert win.imageDisplayTabs.isTabEnabled(mosaic_idx) is False - - # Completion: the mosaic tab is re-enabled and made current so its showEvent - # flushes the single deferred render. - win.toggleAcquisitionStart(False) - qtbot.wait(20) - assert win.imageDisplayTabs.isTabEnabled(mosaic_idx) is True - assert win.imageDisplayTabs.currentWidget() is mosaic diff --git a/software/tests/control/widgets_fluidics/test_gui_wiring.py b/software/tests/control/widgets_fluidics/test_gui_wiring.py index 3828812d6..90595e394 100644 --- a/software/tests/control/widgets_fluidics/test_gui_wiring.py +++ b/software/tests/control/widgets_fluidics/test_gui_wiring.py @@ -42,7 +42,7 @@ def test_the_display_and_record_tabs_are_added(): def test_napari_toggle_exempts_the_fluidics_tab(): - assert "self.FLUIDICS_TAB_NAME" in _source("toggleNapariTabs") + assert "self.FLUIDICS_TAB_NAME" in _source("_set_napari_tabs_enabled") def test_signals_are_connected_and_the_imaging_port_feeds_napari(): diff --git a/software/tests/squid/test_stage.py b/software/tests/squid/test_stage.py index 6c1a0c70e..b227073b2 100644 --- a/software/tests/squid/test_stage.py +++ b/software/tests/squid/test_stage.py @@ -1,8 +1,6 @@ import builtins import logging -import os import pytest -import tempfile import unittest.mock import squid.stage.cephla @@ -30,18 +28,14 @@ def test_simulated_cephla_stage_ops(): assert stage.get_pos() == squid.abc.Pos(x_mm=0.0, y_mm=0.0, z_mm=0.0, theta_rad=0.0) -def test_position_caching(): - (temp_fd, temp_cache_path) = tempfile.mkstemp(".cache", "squid_testing_") - # mkstemp hands back an *open* descriptor. cache_position() writes a temp file and - # os.replace()s it over this path, which Windows refuses with PermissionError while - # another handle to the destination is open, so close it before the call. - os.close(temp_fd) +def test_position_caching(tmp_path): + cache_path = str(tmp_path / "squid_testing.cache") # Use 6 figures after the decimal so we test that we can capture nanometers p = squid.abc.Pos(x_mm=11.111111, y_mm=22.222222, z_mm=1.333333, theta_rad=None) - squid.stage.utils.cache_position(pos=p, stage_config=squid.config.get_stage_config(), cache_path=temp_cache_path) + squid.stage.utils.cache_position(pos=p, stage_config=squid.config.get_stage_config(), cache_path=cache_path) - p_read = squid.stage.utils.get_cached_position(cache_path=temp_cache_path) + p_read = squid.stage.utils.get_cached_position(cache_path=cache_path) assert p_read == p From 6c86c911ccacb22985e29e86468f456d3f93ac3a Mon Sep 17 00:00:00 2001 From: You Yan Date: Wed, 14 Oct 2026 11:09:15 -0700 Subject: [PATCH 11/13] fix(pyqt6): read QCheckBox.stateChanged values the same way under both bindings The signal carries an int on PyQt5 and PyQt6, but on PyQt6 Qt.Checked is a plain Enum member, so `state == Qt.Checked` is always False there. That made toggleClickToMove disconnect on every change (TypeError: 'method' object is not connected, 19 CI failures in test_holder_alignment_dialog.py and test_wellplate_format_widget.py under PyQt6) and silently broke the N/A toggles of the wavelength and source-code fields in the channel configurator. One helper, checkbox_state_is_checked(), replaces the three comparisons and the getattr(.value) dance the Preferences dialog already had. Co-Authored-By: Claude Fable 5.1 --- software/control/widgets.py | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/software/control/widgets.py b/software/control/widgets.py index 281809951..9d49653ac 100644 --- a/software/control/widgets.py +++ b/software/control/widgets.py @@ -54,6 +54,16 @@ # set QT_API environment variable os.environ["QT_API"] = "pyqt6" + +def checkbox_state_is_checked(state) -> bool: + """True when a QCheckBox.stateChanged value means Checked, under PyQt5 and PyQt6. + + The signal carries an int on both bindings, but on PyQt6 Qt.Checked is a plain Enum + member, so ``state == Qt.Checked`` is always False there. + """ + return getattr(state, "value", state) == getattr(Qt.Checked, "value", Qt.Checked) + + # qt libraries import qtpy from qtpy.QtCore import * @@ -1924,11 +1934,7 @@ def create_sim_checkbox(config_key): def _toggle_dev_tab_visibility(self, state): """Show or hide the Dev tab based on checkbox state.""" - # Handle both PyQt5 (int) and PyQt6 (CheckState enum) signal types - # PyQt6 enums have .value property, integers don't - use getattr for compatibility - state_value = getattr(state, "value", state) - checked_value = getattr(Qt.Checked, "value", Qt.Checked) - self.tab_widget.setTabVisible(self._dev_tab_index, state_value == checked_value) + self.tab_widget.setTabVisible(self._dev_tab_index, checkbox_state_is_checked(state)) # The four readers below go through control._def.conf_attribute_reader, the function the running software reads # the same ini with. It strips an inline comment ("30.0 # mm/s"), which ConfigParser keeps as part of the value. @@ -12256,7 +12262,7 @@ def moveStage(self, x, y): self.stage.move_y(dy) def toggleClickToMove(self, state): - if state == Qt.Checked: + if checkbox_state_is_checked(state): self.live_viewer.signal_calibration_viewer_click.connect(self.viewerClicked) else: self.live_viewer.signal_calibration_viewer_click.disconnect(self.viewerClicked) @@ -14115,7 +14121,7 @@ def __init__(self, wavelength_nm=None, parent=None): self.na_label.setVisible(True) def _on_checkbox_changed(self, state): - checked = state == Qt.Checked + checked = checkbox_state_is_checked(state) self.spinbox.setVisible(checked) self.na_label.setVisible(not checked) @@ -14751,7 +14757,7 @@ def __init__(self, source_code=None, parent=None): self.na_label.setVisible(True) def _on_checkbox_changed(self, state): - checked = state == Qt.Checked + checked = checkbox_state_is_checked(state) self.spinbox.setVisible(checked) self.na_label.setVisible(not checked) From ecb4bd0f2e57742690777a39e9a20b8900595a99 Mon Sep 17 00:00:00 2001 From: You Yan Date: Wed, 14 Oct 2026 11:16:51 -0700 Subject: [PATCH 12/13] test: shadow napari 0.7's log-record factory fixture, which broke every fluidics run napari 0.7.1's pytest plugin (napari.utils._testsupport) has a session-wide autouse fixture, use_leak_safe_log_records, that installs a LogRecord factory rewriting record.args as tuple(str(a) for a in args). When a call logs a single dict, logging stores the dict itself as args, so iterating it yields the keys and the record no longer formats ("not all arguments converted during string formatting"). pytest's capture handler re-raises that into the caller. The fluidics library logs each sequence as `_logger.debug("Running: %s", sequence)`, so under the PyQt6 / napari 0.7 environment every fluidics run failed at its first sequence: 5 failures in the fluidics protocol and widget tests on CI, plus a deleted-MplCanvas error in the protocol-widget test that followed the aborted runs. Master's napari 0.5.6 plugin has no such fixture, which is why master is green. None of these tests use napari's fixtures, so tests/conftest.py defines an inert fixture of the same name; a conftest fixture shadows a plugin's. Verified in a PyQt6 6.11 / napari 0.7.1 / numpy 2.2.6 venv: the fluidics protocol and widget tests, the holder-alignment and wellplate-format dialogs go from 24 failures to 128 passed. Co-Authored-By: Claude Fable 5.1 --- software/tests/conftest.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/software/tests/conftest.py b/software/tests/conftest.py index 7424900ed..b5a54e5f9 100644 --- a/software/tests/conftest.py +++ b/software/tests/conftest.py @@ -128,3 +128,18 @@ def cleanup_leaked_hardware(monkeypatch): # Last, after every writer above is closed: delete this test's simulated-acquisition output. cleanup_stub_acquisition_dirs() + + +@pytest.fixture(autouse=True, scope="session") +def use_leak_safe_log_records(): + """Shadow napari's session fixture of the same name (napari.utils._testsupport, napari 0.7). + + napari's pytest plugin swaps in a LogRecord factory that rewrites ``record.args`` as + ``tuple(str(a) for a in args)``. When a call logs a single dict, ``logging`` stores the dict + itself as ``args`` (so ``%(key)s`` formats work), and iterating it yields its keys: the + record then raises "not all arguments converted" and pytest's capture handler re-raises + that into the caller. The fluidics library logs each sequence that way, which failed every + fluidics run in the suite. None of these tests use napari's fixtures, so an inert + stand-in is enough. + """ + yield From c38a49f88f11646c6a87d81b4c3de699a69a7b64 Mon Sep 17 00:00:00 2001 From: You Yan Date: Fri, 16 Oct 2026 20:57:10 -0700 Subject: [PATCH 13/13] build: add setup_24.04.sh and setup_cuda_24.04.sh for Ubuntu 24.04 Same package set as the 22.04 scripts (PyQt6 6.11 / napari 0.7.1); the 24.04 variants add only what Python 3.12 needs: PIP_BREAK_SYSTEM_PACKAGES=1 (PEP 668, one export covering every pip call), apt python3-setuptools/python3-wheel listed explicitly for the Daheng `setup.py build/install` (3.12 ships no distutils), and NVIDIA's ubuntu2404 repo in the cuda script. Both READMEs now point to the script for each release. Verified on a uv-managed Python 3.12 venv (this host is 22.04): every pip line resolves from wheels except cobs, which builds from sdist and falls back to pure Python when no compiler is present, so the build-essential/python3-dev additions from the earlier draft are not needed; the Daheng setup.py install works on 3.12 with Ubuntu's setuptools 68.1.2 and with setuptools 84; `pip check` reports no broken requirements. Not yet run end to end on a real 24.04 machine. Co-Authored-By: Claude Fable 5.1 --- README.md | 2 +- software/README.md | 12 ++- software/setup_24.04.sh | 155 +++++++++++++++++++++++++++++++++++ software/setup_cuda_24.04.sh | 16 ++++ 4 files changed, 183 insertions(+), 2 deletions(-) create mode 100755 software/setup_24.04.sh create mode 100755 software/setup_cuda_24.04.sh diff --git a/README.md b/README.md index 1c0284cf1..7fca10907 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ See related work and applications on: www.cephla.com, www.squid-imaging.org ## How to Start ### Software #### Setting up and run Squid software on Linux -Ubuntu 22.04 is the most tested platform. Other Linux systems should work but performance is not guaranteed. +Ubuntu 22.04 is the most tested platform; Ubuntu 24.04 is supported with its own setup script (`software/setup_24.04.sh`). Other Linux systems should work but performance is not guaranteed. See [README](https://github.com/Cephla-Lab/Squid/blob/master/software/README.md) in `/software` directory for instructions. Toupcam and laser auto-focus camera dependencies will be installed automatically when you run the setup script. diff --git a/software/README.md b/software/README.md index 99f7db118..a487f1a83 100644 --- a/software/README.md +++ b/software/README.md @@ -1,11 +1,21 @@ ## Setting up the environments -Run the following script in terminal to clone the repo and set up the environment +Run the setup script for your Ubuntu release in a terminal to clone the repo and set up the environment. + +Ubuntu 22.04: ``` wget https://raw.githubusercontent.com/Cephla-Lab/Squid/master/software/setup_22.04.sh chmod +x setup_22.04.sh ./setup_22.04.sh ``` +Ubuntu 24.04: +``` +wget https://raw.githubusercontent.com/Cephla-Lab/Squid/master/software/setup_24.04.sh +chmod +x setup_24.04.sh +./setup_24.04.sh +``` +The two scripts install the same packages; the 24.04 one only adds what Python 3.12 requires (pip's `--break-system-packages`, apt setuptools). If you pip-install extras by hand on 24.04, add `--break-system-packages` to the pip command. + Reboot the computer to finish the installation. ## Optional or Hardware-specific dependencies diff --git a/software/setup_24.04.sh b/software/setup_24.04.sh new file mode 100755 index 000000000..440eb86ee --- /dev/null +++ b/software/setup_24.04.sh @@ -0,0 +1,155 @@ +#!/bin/bash +set -eo pipefail + +# Ubuntu 24.04 (Python 3.12) variant of setup_22.04.sh. Keep the two scripts in sync; the only +# intended differences are: +# * PEP 668: 24.04 marks the system Python as externally managed and pip refuses to install +# into it without --break-system-packages, so PIP_BREAK_SYSTEM_PACKAGES=1 is exported +# before the first pip call. +# * python3-setuptools and python3-wheel are apt-installed explicitly: Python 3.12 no longer +# ships distutils, and the Daheng camera API below is installed with a plain +# `setup.py build` / `setup.py install`, which needs setuptools. + +if [[ -n "$TRACE" ]]; then + echo "TRACE variable non-empty, turning on script tracing." + set -x +fi + +SQUID_REPO_PATH="$HOME/Desktop/Squid" + +for i in "$@"; do + case $i in + -r=*|--repo_path=*) + SQUID_REPO_PATH="$(cd "${i#*=}" && pwd)" + shift + ;; + -*|--*) + echo "Unknown option $i" + exit 1 + ;; + *) + ;; + esac +done + +echo "Using SQUID_REPO_PATH='${SQUID_REPO_PATH}'" + +readonly SQUID_REPO_HTTP="https://github.com/Cephla-Lab/Squid.git" +readonly SQUID_SOFTWARE_ROOT="${SQUID_REPO_PATH}/software" +readonly SQUID_REPO_PATH_PARENT="$(dirname "${SQUID_REPO_PATH}")" +readonly DAHENG_CAMERA_DRIVER_ROOT="$SQUID_SOFTWARE_ROOT/drivers and libraries/daheng camera/Galaxy_Linux-x86_Gige-U3_32bits-64bits_1.2.1911.9122" +readonly DAHENG_CAMERA_DRIVER_API_ROOT="$SQUID_SOFTWARE_ROOT/drivers and libraries/daheng camera/Galaxy_Linux_Python_1.0.1905.9081/api" +readonly TOUPCAM_UDEV_RULE_PATH="$SQUID_SOFTWARE_ROOT/drivers and libraries/toupcam/linux/udev/99-toupcam.rules" +readonly PI_UDEV_RULE_DIR="$SQUID_SOFTWARE_ROOT/drivers and libraries/pi/udev" +# update +sudo apt update + +# install packages. python3-setuptools/python3-wheel are listed explicitly (python3-pip pulls +# them in anyway): Python 3.12 has no distutils, and the Daheng API install below is a plain +# `setup.py build` / `setup.py install`, which needs setuptools. Ubuntu's setuptools 68 handles +# it; upstream has deprecated `setup.py install`, so do not pip-upgrade setuptools here. +sudo apt install python3-pip python3-setuptools python3-wheel -y +# Do not apt-install python3-pyqtgraph: it pulls in python3-pyqt5 (see the Qt-binding note +# below). pyqtgraph comes from pip. + +sudo apt-get install git -y +## clone the repo if we don't already have it. +# No matter, make sure the repo's parent dir is there +mkdir -p "${SQUID_REPO_PATH_PARENT}" +if [[ ! -d "${SQUID_REPO_PATH}" ]]; then + git clone "$SQUID_REPO_HTTP" "${SQUID_REPO_PATH}" +else + echo "Using existing repo at '${SQUID_REPO_PATH}' at HEAD=$(cd "${SQUID_REPO_PATH}" && git rev-parse HEAD)" +fi + + +cd "$SQUID_SOFTWARE_ROOT" +mkdir -p "$SQUID_SOFTWARE_ROOT/cache" + +# PEP 668: without this every pip call below fails with "externally-managed-environment". +# pip reads it as --break-system-packages; one export covers the installs, the PyQt5 uninstall +# and the editable fluidics install. Packages land in ~/.local, as on 22.04. +export PIP_BREAK_SYSTEM_PACKAGES=1 + +# Upgrade pip before resolving the dependency graph. +python3 -m pip install --upgrade pip + +# Qt binding: the app targets PyQt6 (QT_API=pyqt6). Exactly ONE Qt binding may be installed: +# PyQt5 and PyQt6 side by side break napari/vispy OpenGL rendering (blank canvases, "Cannot +# SIZE object N because it does not exist"), so remove any pre-existing PyQt5 first. +sudo apt remove -y python3-pyqt5 python3-pyqt5.qtsvg 2>/dev/null || true +pip3 uninstall -y PyQt5 PyQt5-Qt5 PyQt5-sip 2>/dev/null || true + +# install libraries. No "numpy<2" pin: napari 0.7 only needs numpy>=1.24, and the rest of +# the current stack (opencv-python 5.x, pyqtgraph 0.14, ...) targets NumPy 2. +# +# aicsimageio and basicpy are deliberately NOT installed: basicpy pins scipy<1.13, which +# drags the environment back to numpy 1.26 and breaks napari and opencv (see the commit +# "stop installing basicpy/aicsimageio"). Their only consumer, control/stitcher.py, is not +# imported by the application; install them into a separate venv if you need it. +pip3 install pyqtgraph qtpy pyserial pandas imageio crc==1.3.0 lxml numpy tifffile scipy pyreadline3 +pip3 install opencv-python-headless opencv-contrib-python-headless +# napari and PyQt6 pinned to the tested combination and resolved in one pip call, so the +# [pyqt6] extra checks the PyQt6 pins against napari's Qt blocklist (napari 0.7.1 blocks +# PyQt6-Qt6 6.11.0 and 6.11.1, napari/napari#9052): bump them together. +# tensorstore is required by control/ndviewer_light and by tests/control/core/test_zarr_writer.py, +# which begins with pytest.importorskip("tensorstore") -- without it ~79 zarr tests silently skip. +pip3 install "napari[pyqt6]==0.7.1" "PyQt6==6.11.0" "PyQt6-Qt6==6.11.2" "PyQt6-sip==13.12.0" scikit-image dask_image ome_zarr tensorstore pytest pytest-qt pytest-xvfb gitpython matplotlib pydantic_xml pyvisa hidapi filelock lxml_html_clean psutil mcp ndv + +# Optional: PI V-308 / C-414 focus stage (USE_PI_FOCUS_STAGE). Safe to skip if unused; +# squid.stage.pi imports it lazily and only needs it to connect to real hardware, so +# `||` keeps an install failure non-fatal under set -e. +pip3 install pipython || echo "WARNING: pipython install failed; continuing (only needed for USE_PI_FOCUS_STAGE)." >&2 + +# Fluidics library (RUN_FLUIDICS), vendored as the fluidics_v2 submodule. It declares PyQt5 as a +# dependency, so install it --no-deps to keep pip from pulling a PyQt5 wheel over apt's; its real +# runtime deps follow. `||` keeps a checkout without submodules working (fluidics simply stays off). +pip3 install cobs platformdirs ruamel.yaml pyyaml "pydantic>=2,<3" +pip3 install --no-deps -e fluidics_v2/software || echo "WARNING: fluidics library install failed; continuing (only needed for RUN_FLUIDICS)." >&2 + +# install camera drivers +cd "$DAHENG_CAMERA_DRIVER_ROOT" +./Galaxy_camera.run +cd "$DAHENG_CAMERA_DRIVER_API_ROOT" +python3 setup.py build +sudo python3 setup.py install +cd "$SQUID_SOFTWARE_ROOT" +sudo cp "$TOUPCAM_UDEV_RULE_PATH" /etc/udev/rules.d + +# PI C-414 focus stage (USE_PI_FOCUS_STAGE): bind the custom-VID FTDI to ftdi_sio so +# /dev/ttyUSB* appears, and lower the latency timer. Reload + trigger so it applies now. +sudo cp "$PI_UDEV_RULE_DIR/98-pi-c414-bind.rules" "$PI_UDEV_RULE_DIR/99-pi-ftdi-latency.rules" /etc/udev/rules.d +# Trigger with --action=add: the bind rule matches ACTION=="add", so a plain `udevadm +# trigger` (which defaults to action=change) would NOT bind an already-connected C-414. +sudo udevadm control --reload-rules && sudo udevadm trigger --action=add --subsystem-match=usb --subsystem-match=usb-serial --subsystem-match=tty + +# enable access to serial ports without sudo +sudo usermod -aG dialout $USER + +sudo apt autoremove -y + +echo "Holding kernel packages to prevent automatic updates..." +if sudo apt-mark hold \ + linux-image-generic linux-headers-generic linux-generic \ + "linux-image-$(uname -r)" "linux-headers-$(uname -r)"; then + echo "Kernel packages held. Run 'sudo apt-mark unhold linux-image-generic linux-headers-generic linux-generic linux-image-$(uname -r) linux-headers-$(uname -r)' to re-enable." +else + echo "Warning: Failed to hold kernel packages; automatic kernel updates remain enabled." >&2 +fi + +# create desktop shortcut +mkdir -p "$HOME/Desktop" +DESKTOP_FILE="$HOME/Desktop/Squid_hcs.desktop" +ICON_PATH="$SQUID_SOFTWARE_ROOT/icon/cephla_logo.svg" +cat > "$DESKTOP_FILE" << EOF +[Desktop Entry] +Name=Squid_hcs +Icon=$ICON_PATH +Exec=gnome-terminal --working-directory="$SQUID_SOFTWARE_ROOT" -e "/usr/bin/env python3 $SQUID_SOFTWARE_ROOT/main_hcs.py" +Type=Application +Terminal=true +EOF +chmod u+rwx "$DESKTOP_FILE" +# mark as trusted on GNOME +gio set "$DESKTOP_FILE" metadata::trusted true 2>/dev/null || true +echo "Desktop shortcut created at: $DESKTOP_FILE" diff --git a/software/setup_cuda_24.04.sh b/software/setup_cuda_24.04.sh new file mode 100755 index 000000000..4e15033b7 --- /dev/null +++ b/software/setup_cuda_24.04.sh @@ -0,0 +1,16 @@ +#!/bin/bash +# Ubuntu 24.04 variant of setup_cuda_22.04.sh: NVIDIA's ubuntu2404 apt repo, and pip needs +# --break-system-packages on 24.04 (PEP 668), set here through pip's environment variable. +export PIP_BREAK_SYSTEM_PACKAGES=1 +sudo apt update +# sudo apt install nvidia-driver-530 +cd ~/Downloads +wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb +sudo dpkg -i cuda-keyring_1.1-1_all.deb +sudo apt-get update +sudo apt-get -y install cuda +pip install cuda-python +pip install cupy-cuda12x +pip3 install torch torchvision torchaudio + +sudo apt autoremove -y