ci(release): gate publishes on a companion the built distribution works with, on main

.github/workflows/release-publish.yml

@@ -35,6 +35,19 @@ jobs:
     permissions:
       contents: read
       actions: read
+    services:
+      postgres:
+        image: postgres:16
+        env:
+          POSTGRES_HOST_AUTH_METHOD: trust
+          POSTGRES_DB: adcp_release_gate
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 10
     steps:
       - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0
         with:
@@ -71,12 +84,6 @@ jobs:
       - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
         with:
           python-version: '3.12'
-      - name: Require installable companion distributions for an adcp release
-        if: inputs.package == 'adcp'
-        run: |
-          python -m pip install packaging
-          version="$(python -c 'import tomllib; print(tomllib.load(open("pyproject.toml", "rb"))["project"]["version"])')"
-          python scripts/check_companion_releases.py "$version"
       - name: Build and check distributions
         env:
           PACKAGE: ${{ inputs.package }}
@@ -88,6 +95,29 @@ jobs:
             python -m build --outdir dist/ packages/adcp-reporting-storage
           fi
           python -m twine check dist/*
+      # Every release, on main and on next, needs a published companion it
+      # actually works with, not one whose version range merely admits it
+      # (#1763: adcp 10.0.0b4 installed beside adcp-reporting-storage 0.2.0b0
+      # and failed every reporting turn). The gate installs the wheel just
+      # built with the published companion into a private environment and
+      # runs scripts/companion_smoke.py there: both sides must declare the same
+      # storage contract, and the ledger must create its schema and round-trip
+      # records on the PostgreSQL service above. For adcp the companion is the
+      # newest published adcp-reporting-storage its range admits; for
+      # adcp-reporting-storage it is the oldest and the newest published adcp
+      # its own requirement accepts.
+      - name: Require a companion release the built distribution works with
+        env:
+          PACKAGE: ${{ inputs.package }}
+          ADCP_PG_TEST_URL: postgresql://postgres@localhost:5432/adcp_release_gate
+        run: |
+          python -m pip install packaging
+          if [ "$PACKAGE" = adcp ]; then
+            version="$(python -c 'import tomllib; print(tomllib.load(open("pyproject.toml", "rb"))["project"]["version"])')"
+            python scripts/check_companion_releases.py "$version" --dist dist/adcp-*.whl
+          else
+            python scripts/check_companion_releases.py --package "$PACKAGE" --dist dist/adcp_reporting_storage-*.whl
+          fi
       - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
         with:
           name: release-dist

RELEASING.md

@@ -79,6 +79,20 @@ so that merges in either direction never change the other line's release flow:
   to `main`'s releases, and adopters opt in with `pip install --pre adcp` or
   `adcp==10.0.0a0`.
 
+- The companion gate (`scripts/check_companion_releases.py`, #1605, #1763)
+  runs for every publish of either package, from `main` and from `next`. An
+  `adcp` publish needs a published `adcp-reporting-storage` whose range
+  accepts it **and that works with it**: the workflow installs the wheel it
+  just built with that storage release into a private environment and runs
+  `scripts/companion_smoke.py`, which refuses a pair whose storage contracts
+  differ (`adcp.reporting.ledger.storage_contract` versus
+  `adcp_reporting_storage.contract`) and round-trips ledger records on a
+  PostgreSQL service. A storage publish is gated the same way against the
+  oldest and the newest published `adcp` its own requirement accepts. So when
+  the contract changes, order the releases: publish the storage release that
+  implements the new contract (its floor is the first `adcp` of that
+  contract) before the `adcp` release that declares it.
+
 One-time setup: add `next` to the `release-publish` environment's deployment
 branches (GitHub Settings → Environments). The PyPI Trusted Publisher needs no
 change, because it is bound to the `release-publish.yml` file name and the

docs/releasing.md

@@ -75,8 +75,14 @@ commits with no such PR. Conventional-commit scopes are free text, but
 On a merge, `release-please.yml` queues `release-publish.yml` with
 `package=adcp-reporting-storage`; the workflow builds that package's sdist and
 wheel from `packages/adcp-reporting-storage` and publishes them through the same
-`release-publish` environment approval. The storage wheel pins
-`adcp>=9.0.0bN,<10`; publish the `adcp` release it depends on first.
+`release-publish` environment approval. The storage wheel's `adcp` floor is
+the first `adcp` release of the storage contract it implements
+(`adcp_reporting_storage.contract`); publish that `adcp` release first.
+Both publishes run the companion gate (`scripts/check_companion_releases.py`):
+the wheel just built is installed beside the published companion in a private
+environment and `scripts/companion_smoke.py` must pass there, so a pair whose
+storage contracts differ, or whose ledger does not round-trip on PostgreSQL,
+never reaches PyPI (#1763).
 
 One-time setup, before the first storage release:
 

scripts/check_companion_releases.py

@@ -1,22 +1,42 @@
 #!/usr/bin/env python3
-"""Refuse to publish an ``adcp`` release its companion distributions can't install with.
+"""Refuse to publish a distribution its companion can't actually work with.
 
 9.0.0 moved the PostgreSQL reporting stores into ``adcp-reporting-storage``
 before that package was on PyPI, so the documented
 ``pip install "adcp-reporting-storage[pg]"`` failed and Postgres adopters were
-stuck on the last beta. Before an ``adcp`` release publishes, every companion
-distribution must have a published, non-yanked release whose ``adcp``
-requirement accepts the version being released.
+stuck on the last beta. Then adcp 10.0.0b4 changed the ledger column lists the
+storage package reads against its DDL, and the published storage release
+(``adcp>=9.0.0,<11``) still *installed* beside it and failed every reporting
+turn (adcp-client-python#1763). Range compatibility is not enough.
+
+Before an ``adcp`` release publishes, every companion distribution must have a
+published, non-yanked release whose ``adcp`` requirement accepts the version
+being released (``compatible_release``). With ``--dist``, the gate goes
+further and proves the pair works: it installs the built distribution and the
+published companion into a private environment and runs
+``scripts/companion_smoke.py`` there, which compares the storage contract both
+sides declare and, when ``ADCP_PG_TEST_URL`` is set, creates the ledger schema
+and round-trips records on PostgreSQL. ``--package adcp-reporting-storage``
+gates the companion in the other direction: the built storage distribution is
+smoked with both the oldest and the newest published ``adcp`` its own
+requirement accepts.
 """
 
 from __future__ import annotations
 
 import argparse
 import json
+import os
+import shutil
+import subprocess
 import sys
+import tempfile
 import urllib.error
 import urllib.request
+import zipfile
 from collections.abc import Callable
+from email.parser import HeaderParser
+from pathlib import Path
 from typing import Any
 
 from packaging.requirements import Requirement
@@ -25,8 +45,12 @@
 
 #: Distributions maintained in this repository that adopters install with adcp.
 COMPANIONS = ("adcp-reporting-storage",)
+#: The extra that installs the PostgreSQL driver the smoke needs.
+STORAGE_EXTRA = "pg"
+SMOKE = Path(__file__).with_name("companion_smoke.py")
 
 Fetch = Callable[[str], Any]
+Run = Callable[[list[str]], subprocess.CompletedProcess[str]]
 
 
 def _pypi(url: str) -> Any:
@@ -40,6 +64,13 @@ def _pypi(url: str) -> Any:
         raise
 
 
+def _run(command: list[str]) -> subprocess.CompletedProcess[str]:
+    environment = {key: value for key, value in os.environ.items() if key != "PYTHONPATH"}
+    return subprocess.run(  # noqa: S603
+        command, capture_output=True, text=True, timeout=900, env=environment, check=False
+    )
+
+
 def _adcp_requirement(requires_dist: list[str] | None) -> Requirement | None:
     for line in requires_dist or ():
         requirement = Requirement(line)
@@ -48,20 +79,25 @@ def _adcp_requirement(requires_dist: list[str] | None) -> Requirement | None:
     return None
 
 
-def compatible_release(companion: str, adcp_version: Version, fetch: Fetch = _pypi) -> str | None:
-    """Return the newest published ``companion`` release that installs with ``adcp_version``."""
-    project = fetch(f"https://pypi.org/pypi/{companion}/json")
-    if not project:
-        return None
+def _published_versions(project: Any) -> list[Version]:
+    """Non-yanked releases of a PyPI project, newest first."""
     candidates = []
-    for raw, files in project.get("releases", {}).items():
+    for raw, files in (project or {}).get("releases", {}).items():
         try:
             version = Version(raw)
         except InvalidVersion:
             continue
         if files and not all(file.get("yanked") for file in files):
             candidates.append(version)
-    for version in sorted(candidates, reverse=True):
+    return sorted(candidates, reverse=True)
+
+
+def compatible_release(companion: str, adcp_version: Version, fetch: Fetch = _pypi) -> str | None:
+    """Return the newest published ``companion`` release that installs with ``adcp_version``."""
+    project = fetch(f"https://pypi.org/pypi/{companion}/json")
+    if not project:
+        return None
+    for version in _published_versions(project):
         release = fetch(f"https://pypi.org/pypi/{companion}/{version}/json")
         requirement = _adcp_requirement((release or {}).get("info", {}).get("requires_dist"))
         if requirement is not None and requirement.specifier.contains(
@@ -71,27 +107,145 @@ def compatible_release(companion: str, adcp_version: Version, fetch: Fetch = _py
     return None
 
 
-def main(argv: list[str] | None = None, fetch: Fetch = _pypi) -> int:
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument("adcp_version", help="the adcp version about to be published")
-    args = parser.parse_args(argv)
-    adcp_version = Version(args.adcp_version)
+def accepted_adcp_releases(requirement: Requirement, fetch: Fetch = _pypi) -> list[str]:
+    """Published, non-yanked ``adcp`` releases the requirement accepts, oldest first."""
+    project = fetch("https://pypi.org/pypi/adcp/json")
+    accepted = [
+        version
+        for version in _published_versions(project)
+        if requirement.specifier.contains(version, prereleases=True)
+    ]
+    return [str(version) for version in sorted(accepted)]
+
+
+def wheel_adcp_requirement(distribution: Path) -> Requirement:
+    """The ``adcp`` requirement a built companion wheel declares."""
+    with zipfile.ZipFile(distribution) as wheel:
+        metadata = next(name for name in wheel.namelist() if name.endswith(".dist-info/METADATA"))
+        headers = HeaderParser().parsestr(wheel.read(metadata).decode("utf-8"))
+    requirement = _adcp_requirement(headers.get_all("Requires-Dist"))
+    if requirement is None:
+        raise SystemExit(f"{distribution.name} declares no adcp requirement")
+    return requirement
+
+
+def smoke(requirements: list[str], *, label: str, run: Run = _run) -> bool:
+    """Install ``requirements`` into a private environment and run the companion smoke."""
+    with tempfile.TemporaryDirectory(prefix="adcp-companion-") as root:
+        environment = Path(root) / "env"
+        if shutil.which("uv"):
+            create = ["uv", "venv", "--quiet", "--python", sys.executable, str(environment)]
+            python = environment / "bin" / "python"
+            install = ["uv", "pip", "install", "--quiet", "--python", str(python)]
+        else:
+            create = [sys.executable, "-m", "venv", str(environment)]
+            python = environment / "bin" / "python"
+            install = [str(python), "-m", "pip", "install", "--quiet"]
+        for step, command in (("create", create), ("install", [*install, *requirements])):
+            completed = run(command)
+            if completed.returncode != 0:
+                print(f"{label}: could not {step} the smoke environment", file=sys.stderr)
+                print(completed.stderr[-4000:], file=sys.stderr)
+                return False
+        script = Path(root) / SMOKE.name
+        shutil.copy2(SMOKE, script)
+        completed = run([str(python), "-I", str(script)])
+        sys.stdout.write(completed.stdout)
+        if completed.returncode != 0:
+            print(f"{label}: {completed.stderr.strip()}", file=sys.stderr)
+            return False
+        return True
+
+
+def gate_adcp(
+    adcp_version: Version, distribution: Path | None, fetch: Fetch = _pypi, run: Run = _run
+) -> int:
+    """An adcp release needs a published companion that installs with it and works with it."""
     missing = []
+    failed = []
     for companion in COMPANIONS:
         found = compatible_release(companion, adcp_version, fetch)
         if found is None:
             missing.append(companion)
-        else:
-            print(f"{companion} {found} installs with adcp {adcp_version}")
+            continue
+        print(f"{companion} {found} installs with adcp {adcp_version}")
+        if distribution is not None and not smoke(
+            [str(distribution), f"{companion}[{STORAGE_EXTRA}]=={found}"],
+            label=f"adcp {adcp_version} + {companion} {found}",
+            run=run,
+        ):
+            failed.append(f"{companion} {found}")
     if missing:
         print(
             f"No published release of {', '.join(missing)} accepts adcp {adcp_version}. "
             "Publish a compatible companion release first.",
             file=sys.stderr,
         )
         return 1
+    if failed:
+        print(
+            f"adcp {adcp_version} does not work with the published {', '.join(failed)}. "
+            "Publish a companion release that matches this adcp's storage contract first.",
+            file=sys.stderr,
+        )
+        return 1
+    return 0
+
+
+def gate_companion(distribution: Path, fetch: Fetch = _pypi, run: Run = _run) -> int:
+    """A companion release must work with the oldest and newest adcp its range accepts."""
+    requirement = wheel_adcp_requirement(distribution)
+    accepted = accepted_adcp_releases(requirement, fetch)
+    if not accepted:
+        print(
+            f"No published adcp release satisfies {requirement}; publish adcp first.",
+            file=sys.stderr,
+        )
+        return 1
+    failed = []
+    for adcp_version in sorted({accepted[0], accepted[-1]}, key=Version):
+        print(f"{distribution.name} ({requirement}) accepts the published adcp {adcp_version}")
+        if not smoke(
+            [f"adcp=={adcp_version}", f"{distribution}[{STORAGE_EXTRA}]"],
+            label=f"{distribution.name} + adcp {adcp_version}",
+            run=run,
+        ):
+            failed.append(adcp_version)
+    if failed:
+        print(
+            f"{distribution.name} does not work with the published adcp {', '.join(failed)} its "
+            "requirement accepts. Raise the floor to the first adcp of its storage contract.",
+            file=sys.stderr,
+        )
+        return 1
     return 0
 
 
+def main(argv: list[str] | None = None, fetch: Fetch = _pypi, run: Run = _run) -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "adcp_version", nargs="?", help="the adcp version about to be published (package adcp)"
+    )
+    parser.add_argument(
+        "--package",
+        choices=("adcp", *COMPANIONS),
+        default="adcp",
+        help="the distribution about to be published",
+    )
+    parser.add_argument(
+        "--dist",
+        type=Path,
+        help="the built wheel of that distribution; when given, the pair is installed and smoked",
+    )
+    args = parser.parse_args(argv)
+    if args.package == "adcp":
+        if not args.adcp_version:
+            parser.error("the adcp version is required for --package adcp")
+        return gate_adcp(Version(args.adcp_version), args.dist, fetch, run)
+    if args.dist is None:
+        parser.error(f"--dist is required for --package {args.package}")
+    return gate_companion(args.dist, fetch, run)
+
+
 if __name__ == "__main__":
     raise SystemExit(main())