[canismarko]

Haven contains a number of ophyd-async devices that could be useful for other beamlines. I am hoping to start moving some of these into apstools since ophyd-async is getting close to its first stable release. Ophyd-async is especially good for fly-scanning detectors, which seems useful to others.

This PR:

• adds ophyd-async as a dependency
• includes a simple ophyd-async device (aps machine)

I put the ophyd-async devices in a new _devices_async package so existing deployments shouldn't be affected. I also added ophyd-async as an optional dependency (pip install apstools[async]). My thought was that once ophyd-async is stable, we can rename this package to devices_async and make ophyd-async a default dependency to indicate that this support is ready for general use.

The current release of ophyd-async depends on python >= 3.11. How important is maintaining python 3.10 support? It is schedule to go end-of-life in October.

[prjemian]

Py 3.10 is close to end of support. It's safe for you to drop it from the test matrix here.

[prjemian]

@canismarko

Can you push an empty commit to the branch to run the CI again?

Previous failed due to EPICS connections with a motor (not part of your branch). I'm suspecting this might match a problem I encountered when a conflicting epicscorelibs version was installed. Conflicting version install might be a conda vs. pip situation.

[prjemian]

Investigation of the reproducible CI failure:

The failure is caused by the PR's new ophyd-async[ca] dependency, not by the changed device code.

Evidence:

• PR Initial ophyd-async support #1188 changes CI from pip install -e .[dev] to pip install -e .[dev,async].
• The async extra installs ophyd-async[ca], which installs aioca, which installs epicscorelibs.
• The conda environment already installs pyepics, so the resulting environment contains both the conda EPICS/PyEpics stack and the pip-installed epicscorelibs stack.
• Package installation completes successfully.
• The explicit EPICS smoke tests all pass: caget, PyEpics caget, and an ophyd.EpicsSignalRO connection.
• Test collection then fails in unchanged code at apstools/devices/tests/test_dict_device.py:10, in the module-level m1.wait_for_connection() call. All gp:m1 motor signals fail to connect.
• The immediately preceding PR run failed at the same unchanged test before the final lint-only commit, confirming that this is reproducible and associated with the dependency change.
• The target run's Python 3.13 job fails first; the 3.11, 3.12, and 3.14 jobs are then cancelled. The preceding PR run independently failed in Python 3.11.
• The failing environment contains epicscorelibs-7.0.10.99.0.2, ophyd-1.11.2, ophyd-async-0.21.3, and pyepics-3.5.10.

The likely mechanism is an EPICS Channel Access client-library collision. aioca and PyEpics can coexist only with careful import/library handling. Installing aioca and epicscorelibs with pip after conda has installed PyEpics introduces competing EPICS core libraries. The scalar smoke tests happen to work, but the larger EpicsMotor connection sequence fails. The subsequent double free or corruption / corrupted size vs. prev_size messages are also consistent with a native EPICS-library conflict.

Recommended CI fix:

• Keep the existing classic ophyd test matrix on pip install -e .[dev].
• Test the new async code in a separate job/environment, preferably a pip/venv environment without the conda-provided PyEpics/EPICS stack.
• In that separate job, install .[dev,async] and run the async tests under apstools/_devices_async/tests.

The smallest immediate fix is to change the existing matrix installation back to:

pip install -e .[dev]

That should restore the existing classic test environment; a separate async CI job is still needed to exercise the new tests.

Contributed by OpenCode(argo/gpt56luna)

[prjemian]

Running the tests locally in a conda env with these versions:

(test) jemian@otz .../sandbox/apstools $ conda list epics
# packages in environment at /home/beams/JEMIAN/.conda/envs/test:
#
# Name                     Version          Build            Channel
epicscorelibs              7.0.10.99.0.2    pypi_0           pypi
pyepics                    3.5.10           pypi_0           pypi
(test) jemian@otz .../sandbox/apstools $ conda list ophyd
# packages in environment at /home/beams/JEMIAN/.conda/envs/test:
#
# Name                     Version          Build            Channel
ophyd                      1.11.2           pypi_0           pypi
ophyd-async                0.21.3           pypi_0           pypi
ophyd-registry             2.0.0            pypi_0           pypi

Test for the failure

Here's a test for the code that fails in the CI workflow:

(test) jemian@otz .../sandbox/apstools $ pytest  ./apstools/ -k test_dict_device
============================================== test session starts ===============================================
platform linux -- Python 3.14.8, pytest-9.1.1, pluggy-1.7.0
rootdir: /tmp/sandbox/apstools
configfile: pytest.ini (WARNING: ignoring pytest config in pyproject.toml!)
plugins: platformdirs-4.13.5, typeguard-4.6.0, anyio-4.15.1, zarr-3.4.1
collected 1230 items / 1222 deselected / 8 selected                                                              

apstools/devices/tests/test_dict_device.py ........                                                        [100%]

================================================ warnings summary ================================================
apstools/_devices_async/tests/test_aps_machine.py:6
  /tmp/sandbox/apstools/apstools/_devices_async/tests/test_aps_machine.py:6: PytestUnknownMarkWarning: Unknown pytest.mark.asyncio - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.asyncio

../../../home/beams/JEMIAN/.conda/envs/test/lib/python3.14/site-packages/_pytest/python.py:124
  /home/beams/JEMIAN/.conda/envs/test/lib/python3.14/site-packages/_pytest/python.py:124: PytestRemovedIn10Warning: Passing a non-Collection iterable to parametrize is deprecated.
  Test: apstools/callbacks/tests/test_specwriter.py::test_issue_1084_from_catalog, argvalues type: generator
  Please convert to a list or tuple.
  See https://docs.pytest.org/en/stable/deprecations.html#parametrize-iterators
    metafunc.parametrize(*marker.args, **marker.kwargs, _param_mark=marker)

../../../home/beams/JEMIAN/.conda/envs/test/lib/python3.14/site-packages/_pytest/python.py:124
  /home/beams/JEMIAN/.conda/envs/test/lib/python3.14/site-packages/_pytest/python.py:124: PytestRemovedIn10Warning: Passing a non-Collection iterable to parametrize is deprecated.
  Test: apstools/utils/tests/test_statistics.py::test_xy_statistics, argvalues type: generator
  Please convert to a list or tuple.
  See https://docs.pytest.org/en/stable/deprecations.html#parametrize-iterators
    metafunc.parametrize(*marker.args, **marker.kwargs, _param_mark=marker)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================= 8 passed, 1222 deselected, 3 warnings in 9.55s =================================

[prjemian]

Follow-up/correction to my previous analysis:

The package-conflict explanation is plausible but not proven as the direct root cause. The local test reported above passes test_dict_device with the same relevant versions (epicscorelibs 7.0.10.99.0.2, pyepics 3.5.10, ophyd 1.11.2, and ophyd-async 0.21.3). That demonstrates that merely having those packages installed does not deterministically break the motor connection.

The confirmed CI facts are:

• The failure was introduced temporally with the PR's [async] installation change.
• The failure is in unchanged code while connecting gp:m1.
• The explicit CI EPICS checks pass, but they only check gp:UPTIME, gp:gp:float1, and ad:cam1:Acquire_RBV; none checks gp:m1.
• The IOC is reported as running, but the failing test begins almost immediately afterward. The GP IOC reports only about five seconds of uptime during the readiness checks.
• Every gp:m1 signal fails, rather than one or two individual fields.
• The native-library abort (double free or corruption / corrupted size vs. prev_size) is concerning, but it occurs after the connection timeout and does not by itself prove that it caused the timeout.

The next diagnostic should therefore be an explicit motor readiness check immediately before pytest, for example:

caget gp:m1.RBV gp:m1.VAL gp:m1.DMOV
python -c "import ophyd; m = ophyd.EpicsMotor('gp:m1', name='m'); m.wait_for_connection(); print(m.read())"

If that check fails, the problem is IOC/PV availability or Channel Access setup, independent of pytest. If it passes but pytest fails only with [async], then the EPICS client interaction becomes much stronger evidence. It would also be useful to run the same test matrix with [async] removed, and to compare the actual EPICS library paths loaded by PyEpics and aioca.

For now, I would describe the package interaction as a leading hypothesis rather than a confirmed root cause. Separating the async tests into their own environment remains a reasonable CI design, but adding a gp:m1 readiness check is the most direct next experiment.

Contributed by OpenCode(argo/gpt56luna)

[prjemian]

If gp:UPTIME is available, then gp:m1 must be, too. This docker image has been run thousands of times without this fail. While the additional caget can be confirmative, missing PV is most certainly not the problem here.

[prjemian]

Agreed. My previous follow-up gave the missing-PV/readiness possibility too much weight. Given that gp:UPTIME is available, the IOC is running and Channel Access is working; combined with the long history of this image, a missing gp:m1 PV is not a credible primary explanation. The extra caget would only be confirmatory, not a likely fix.

The useful distinction is instead:

• The failure is not caused by the changed async device implementation.
• The failure appears only after the CI environment begins installing/importing the async stack.
• The same package versions can pass the motor test locally, so the exact failure mechanism is still not proven to be a package-version conflict.
• The CI-only failure, the EpicsMotor connection timeout, and the subsequent native allocator abort point toward process-level EPICS client/library interaction or initialization order, rather than an absent PV.

A better next experiment is to compare the process state with and without the async stack in the same CI environment. In particular, run the unchanged motor test after importing the new async module/package and then without that import, while recording the loaded EPICS CA library paths. That should distinguish an aioca/epicscorelibs interaction from another environment difference.

I retract the missing-PV/readiness conjecture as a proposed cause.

Contributed by OpenCode(argo/gpt56luna)

[canismarko]

Some tests to reproduce this locally. Unfortunately, the tests always passed, so we failed to fail successfully.

With Pixi (no conda-forge packages)

$ pixi run -e all pytest apstools/devices/tests/test_dict_device.py

✓ Passes

With Pixi (including conda-forge packages)

$ pixi add pyepics
$ pixi run -e all pytest apstools/devices/tests/test_dict_device.py

✓ Passes

With mamba

$ mamba config list --sources
channels:
  - conda-forge  # '~/.condarc'
  - apsu  # '~/.condarc'
changeps1: false  # '~/.condarc'
$ mamba env create -n apstools-async python=3.13 pyepics aps-dm-api
$ pip install -e ".[all]"
$ pytest 

✓ Passes