From 262407ac8ed23d4575ff9afc67b5983492bfa57c Mon Sep 17 00:00:00 2001 From: Alek Petuskey Date: Sun, 2 Aug 2026 13:58:42 -0700 Subject: [PATCH 1/7] Add release operator checklist --- CONTRIBUTING.md | 4 + spec/process/production-readiness.md | 5 + spec/process/release-checklist.md | 256 +++++++++++++++++++++++++++ 3 files changed, 265 insertions(+) create mode 100644 spec/process/release-checklist.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9c12a93d1..7329e2fbd 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,6 +4,10 @@ The full contributor guide — PR checklist, local gate commands, and the chart-type contribution walkthrough — lives at [`spec/process/contributing.md`](spec/process/contributing.md). +Release maintainers should follow the ordered +[`release operator checklist`](spec/process/release-checklist.md) in addition to +the production-readiness gates. + Quick start: ```bash diff --git a/spec/process/production-readiness.md b/spec/process/production-readiness.md index 0a2f00e7b..50bdcb48a 100644 --- a/spec/process/production-readiness.md +++ b/spec/process/production-readiness.md @@ -319,6 +319,11 @@ Node, Chrome, `ruff`, `ty`, or `pytest` produce direct install/skip guidance. ## Release Checklist +For the ordered maintainer procedure—including PyPI immutability checks, tag +recovery, release monitoring, and clean-install smoke tests—use the +[`release operator checklist`](release-checklist.md). The invariants below are +the release bar that procedure must preserve. + The tag *is* the version. `pyproject.toml` declares `dynamic = ["version"]` and uv-dynamic-versioning derives the distribution version from the latest `v*` git tag, so cutting a release is `git tag vX.Y.Z && git push --tags` — there is no diff --git a/spec/process/release-checklist.md b/spec/process/release-checklist.md new file mode 100644 index 000000000..77fcd012f --- /dev/null +++ b/spec/process/release-checklist.md @@ -0,0 +1,256 @@ +# Release Operator Checklist + +Use this checklist to cut the `xy` distribution, including its bundled +`reflex_xy` integration. The production invariants behind these steps live in +[`production-readiness.md`](production-readiness.md); this document is the +ordered procedure for release operators. + +## Safety rules + +- The `vX.Y.Z` Git tag is the package version. There is no version file to + edit. +- PyPI versions and uploaded files are immutable. If the version exists on + PyPI, do not move its tag or rebuild different artifacts under that version. +- A GitHub release can be recreated under the same version only when PyPI + confirms that no file for the version was uploaded. +- A partially published PyPI release must be retried from the same tag and + commit. The publisher skips files PyPI already accepted. + +## 1. Choose the version + +Use a final or canonical PEP 440 prerelease tag: + +```text +vX.Y.Z +vX.Y.ZaN +vX.Y.ZbN +vX.Y.ZrcN +``` + +Confirm the version does not already exist on PyPI: + +```bash +curl --fail https://pypi.org/pypi/xy/X.Y.Z/json +``` + +A `404` means the version is unused. A successful response means the version +cannot be reused. + +Inspect GitHub for an existing release or tag with the same name. Record where +any existing tag points and whether the release has assets before changing it. + +## 2. Prepare the release commit + +- Add a dated `## [X.Y.Z] — YYYY-MM-DD` entry to `CHANGELOG.md`. +- Confirm the changelog describes every user-visible change since the previous + release. +- Refresh benchmark reports, or record why the previous report still applies. +- Run `make check-full`, or confirm the equivalent required CI checks passed. +- Run `make check-ci` after any workflow or release-wiring change. +- Run `make check-sdist` and `make check-wheel` after packaging changes. +- Run `make check-import` after changing lazy imports, backend loading, or the + `reflex_xy` boundary. +- Run the release workflow manually with its default `dry_run: true` before the + first release after changing the platform matrix, cross-compile toolchain, + PyEmscripten ABI, or tag/version scheme. + +The release pull request must be ready for review, have every actionable review +thread resolved, and pass all required checks before merge. + +## 3. Verify both sdist installation contracts + +The same freshly built sdist must prove both supported paths. Disable caches so +an old wheel cannot hide a broken source archive. + +### Rust-backed installation + +The release workflow installs a pinned Rust toolchain and installs the sdist +with `XY_REQUIRE_CARGO=1`. The smoke must: + +- compile the native core with Cargo; +- import `xy`, `reflex_xy`, and `xy.kernels` from the installed distribution; +- confirm package metadata, `xy.__version__`, and `reflex_xy.__version__` + agree; and +- assert `xy.kernels.BACKEND == "native"`. + +Representative local installation: + +```bash +XY_REQUIRE_CARGO=1 uv pip install --no-cache dist/xy-X.Y.Z.tar.gz +``` + +### Coreless installation + +A separate environment installs the same sdist with `XY_SKIP_CARGO=1`. It +must: + +- import `xy` and `reflex_xy` without NumPy, Reflex, or the native core being + loaded eagerly; +- report matching versions; and +- raise the documented native-core `ImportError` only when compute is + requested. + +Representative local installation: + +```bash +XY_SKIP_CARGO=1 uv pip install --no-cache dist/xy-X.Y.Z.tar.gz +``` + +Tests that exercise the optional Reflex integration must not make the base +Python-floor job install Reflex. Gate those integration-only assertions with +`pytest.importorskip("reflex")`. + +## 4. Merge and record the release commit + +After the release PR merges: + +1. Fetch the latest `main`. +2. Confirm the merge commit contains the changelog and release changes. +3. Record the full commit SHA. +4. Confirm no later commit has moved `main` before selecting the release + target. + +Do not tag a pre-merge branch or a commit with incomplete required checks. + +## 5. Publish the GitHub release + +In **GitHub → Releases → Draft a new release**: + +1. Create `vX.Y.Z` from the recorded release commit on `main`. +2. Set the title to `vX.Y.Z`. +3. Generate notes from the previous library release tag. +4. Confirm the notes include both the release PR and any release-fix PRs. +5. Select **Latest** for a production release, or the prerelease label for a + prerelease. +6. Publish the release. + +Publishing creates the tag and triggers `.github/workflows/release.yml`. Check +the release page immediately and confirm its displayed commit is the recorded +SHA. + +## 6. Monitor every release job + +Find and watch the run: + +```bash +gh run list --repo reflex-dev/xy --workflow release.yml --limit 5 +gh run watch RUN_ID --repo reflex-dev/xy --compact --exit-status +``` + +Do not call the release complete until all of these pass: + +- sdist structural verification; +- Rust-backed sdist build and native import; +- coreless sdist lightweight-import and clear-error smoke; +- native macOS wheels for x86-64 and Apple Silicon; +- native Windows wheels for x86, x64, and arm64; +- manylinux and musllinux wheels for every supported architecture; +- PyEmscripten build and real Pyodide runtime load smoke; and +- trusted PyPI publication. + +The GitHub release may show only GitHub's source archives. The distribution +wheels and sdist are published to PyPI by the workflow. + +## 7. Verify PyPI from a clean environment + +Confirm PyPI exposes the version and expected files: + +```bash +curl --fail https://pypi.org/pypi/xy/X.Y.Z/json +``` + +Create a new environment outside the repository so the checkout cannot shadow +the installed package: + +```bash +TMP_DIR=$(mktemp -d /private/tmp/xy-release-smoke.XXXXXX) +UV_CACHE_DIR="$TMP_DIR/uv-cache" uv venv "$TMP_DIR/.venv" --python 3.12 +UV_CACHE_DIR="$TMP_DIR/uv-cache" uv pip install \ + --python "$TMP_DIR/.venv/bin/python" \ + --no-cache \ + 'xy[reflex]==X.Y.Z' \ + numpy +``` + +Verify versions, import locations, and the backend: + +```python +import importlib.metadata as metadata + +import reflex_xy +import xy +import xy.kernels as kernels + +assert metadata.version("xy") == "X.Y.Z" +assert xy.__version__ == "X.Y.Z" +assert reflex_xy.__version__ == "X.Y.Z" +assert kernels.BACKEND == "native" + +print(xy.__file__) +print(reflex_xy.__file__) +``` + +Both paths must point into the clean environment's `site-packages`. + +## 8. Run published-package examples + +Exercise more than imports. From the clean environment: + +- build the README line chart and export non-empty HTML and SVG files; +- build the first-chart scatter and export self-contained HTML; +- render a pyplot sine chart to a valid, non-empty PNG; and +- construct a `reflex_xy.chart(...)` component and confirm it returns an + `XYChart`. + +These checks cover the declarative API, generated client bundle, native static +export, pyplot compatibility layer, and bundled Reflex integration from the +actual PyPI artifact. + +## Recovery procedures + +### The workflow failed before any PyPI upload + +Fix the failure in a PR and merge it. If the version must be reused: + +1. Confirm the PyPI JSON endpoint still returns `404`. +2. Obtain explicit confirmation before deleting the public GitHub release or + tag. +3. Delete the stale GitHub release first; GitHub disables tag deletion while a + release owns it. +4. Delete the stale tag. +5. Recreate the release and tag at the corrected merge commit. + +Never force-move the tag if PyPI accepted any artifact. + +### PyPI accepted only some files + +Keep the tag and commit unchanged. Rerun the same release workflow. Its +publisher uses `skip-existing` so accepted files remain immutable while the +missing artifacts are uploaded. + +### The Python floor fails because Reflex is missing + +The base floor intentionally excludes optional extras. Use +`pytest.importorskip("reflex")` for Reflex-only tests rather than weakening the +base-package dependency boundary. + +### Local socket tests fail with `PermissionError` + +Check whether the test is running in a sandbox that forbids localhost binding. +Rerun the relevant test in an environment where loopback sockets are allowed +before treating it as a product regression. + +## Final sign-off + +- [ ] The version was unused on PyPI before tagging. +- [ ] The dated changelog entry matches the tag. +- [ ] Required CI and review checks passed on the release commit. +- [ ] The tag points at the recorded merge commit. +- [ ] Native and coreless sdist contracts passed from the same source archive. +- [ ] Every native and PyEmscripten wheel passed verification. +- [ ] Trusted publication completed successfully. +- [ ] PyPI exposes the expected version and artifacts. +- [ ] A clean install reports matching `xy` and `reflex_xy` versions. +- [ ] The clean install loads `xy.kernels.BACKEND == "native"`. +- [ ] Declarative, export, pyplot, and Reflex examples passed from + `site-packages`. From 48c934b33c00bdee7b424f16714f446d7d997b64 Mon Sep 17 00:00:00 2001 From: Alek Petuskey Date: Sun, 2 Aug 2026 17:03:35 -0700 Subject: [PATCH 2/7] Add release video checklist --- spec/process/release-checklist.md | 81 +++++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) diff --git a/spec/process/release-checklist.md b/spec/process/release-checklist.md index 77fcd012f..e75e576a2 100644 --- a/spec/process/release-checklist.md +++ b/spec/process/release-checklist.md @@ -206,6 +206,83 @@ These checks cover the declarative API, generated client bundle, native static export, pyplot compatibility layer, and bundled Reflex integration from the actual PyPI artifact. +## 9. Create the release video and thumbnail + +Prepare the release media only after the published package and examples pass +verification. Use the final release URL and output generated by the released +version: + +```text +RELEASE_URL: https://github.com/reflex-dev/xy/releases/tag/vX.Y.Z +BRAND_URL: https://reflex.dev +PRODUCT_NAME: XY +TARGET_PLATFORM: LinkedIn +DURATION: 15–30 seconds +``` + +Build the video with Remotion at 1920×1080 and 30 fps. The final edit must be +specific to this release, communicate clearly with sound muted, and follow the +current `reflex.dev` visual system: a clean near-white canvas, charcoal +typography, faint lavender grid lines, restrained purple and green accents, +thin borders, and subtle shadows. + +Music and subtle sound effects are optional, but every essential point must +appear on screen. + +### Research and content + +- Read the complete release notes and every linked example used in the video. +- Select the three or four most visually compelling release improvements. +- Verify every feature name, statistic, installation command, and performance + claim. Never invent metrics or features. +- Use actual output from the released version for charts, components, and UI + examples whenever it is available. +- Give every highlight a short headline, a visual demonstration where + possible, and one concise supporting detail. + +Charts must look production-quality and remain completely visible. Do not crop +axes, labels, legends, nodes, flows, or chart boundaries. Keep the motion +energetic and technical without making the composition cluttered. Include a +polished animated product logo at both the beginning and the end. + +### Suggested timeline + +1. **0–3 seconds:** show the XY logo, release number, and a strong release + hook. +2. **3–17 seconds:** demonstrate the main visual features with clear labels + and enough screen time to understand each one. +3. **17–22 seconds:** show verified performance, package-size, compatibility, + or developer-experience improvements. +4. **22–26 seconds:** summarize the highlights and finish with the exact + installation command or release URL. + +### Required deliverables + +- an H.264 MP4 suitable for LinkedIn, between 15 and 30 seconds long; +- a separate 1920×1080 LinkedIn thumbnail PNG; +- complete Remotion source; +- a concise LinkedIn description with the release link and relevant hashtags; + and +- a contact sheet or representative full-resolution frames for review. + +The thumbnail must match `reflex.dev`, include **XY X.Y.Z RELEASE**, list all +major highlights in readable labels, and feature the strongest chart or +product visual prominently. Additional charts are welcome when useful, but +every chart must remain fully visible and uncropped. Avoid black frames, +generic video stills, and dark panels on the light background. + +### Quality and approval gate + +- Run the TypeScript checks and complete a final Remotion render. +- Inspect several full-resolution frames from every scene. +- Confirm all text is readable and no content is clipped. +- Confirm every chart accurately represents the released library. +- Confirm the finished duration is between 15 and 30 seconds. +- Show the final video, thumbnail, and representative frames to the release + owner for review. +- Do not upload, publish, or schedule anything until the release owner + explicitly replies **“approve.”** + ## Recovery procedures ### The workflow failed before any PyPI upload @@ -254,3 +331,7 @@ before treating it as a product regression. - [ ] The clean install loads `xy.kernels.BACKEND == "native"`. - [ ] Declarative, export, pyplot, and Reflex examples passed from `site-packages`. +- [ ] Release video, thumbnail, source, LinkedIn copy, and review frames passed + the quality checks. +- [ ] The release owner explicitly approved the media before publication or + scheduling. From 4deed6cb0848c14dbc88d3a08b5df86f28edb496 Mon Sep 17 00:00:00 2001 From: Alek Petuskey Date: Fri, 7 Aug 2026 11:50:55 -0700 Subject: [PATCH 3/7] Address release checklist review feedback --- spec/process/release-checklist.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/spec/process/release-checklist.md b/spec/process/release-checklist.md index e75e576a2..f71085007 100644 --- a/spec/process/release-checklist.md +++ b/spec/process/release-checklist.md @@ -41,7 +41,7 @@ any existing tag points and whether the release has assets before changing it. ## 2. Prepare the release commit -- Add a dated `## [X.Y.Z] — YYYY-MM-DD` entry to `CHANGELOG.md`. +- Add a dated `## [X.Y.Z] - YYYY-MM-DD` entry to `CHANGELOG.md`. - Confirm the changelog describes every user-visible change since the previous release. - Refresh benchmark reports, or record why the previous report still applies. @@ -87,8 +87,9 @@ must: - import `xy` and `reflex_xy` without NumPy, Reflex, or the native core being loaded eagerly; - report matching versions; and -- raise the documented native-core `ImportError` only when compute is - requested. +- raise the documented native-core `ImportError` when `xy.kernels` is imported + (when the compute layer is touched), never at `import xy` or + `import reflex_xy` itself. Representative local installation: @@ -163,7 +164,7 @@ Create a new environment outside the repository so the checkout cannot shadow the installed package: ```bash -TMP_DIR=$(mktemp -d /private/tmp/xy-release-smoke.XXXXXX) +TMP_DIR=$(mktemp -d "${TMPDIR:-/tmp}/xy-release-smoke.XXXXXX") UV_CACHE_DIR="$TMP_DIR/uv-cache" uv venv "$TMP_DIR/.venv" --python 3.12 UV_CACHE_DIR="$TMP_DIR/uv-cache" uv pip install \ --python "$TMP_DIR/.venv/bin/python" \ From f51b511e2a10c8bbe714de7d9c58ea4dde1f1da8 Mon Sep 17 00:00:00 2001 From: Farhan Ali Raza Date: Mon, 26 Oct 2026 16:26:58 +0500 Subject: [PATCH 4/7] Clarify release verification environment --- spec/process/release-checklist.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/spec/process/release-checklist.md b/spec/process/release-checklist.md index acea189cb..870b77cde 100644 --- a/spec/process/release-checklist.md +++ b/spec/process/release-checklist.md @@ -111,7 +111,7 @@ an old wheel cannot hide a broken source archive. ### Rust-backed installation -The release workflow installs a pinned Rust toolchain and installs the sdist +The release workflow installs the stable Rust toolchain and installs the sdist with `XY_REQUIRE_CARGO=1`. The smoke must: - compile the native core with Cargo; @@ -182,9 +182,13 @@ UV_CACHE_DIR="$TMP_DIR/uv-cache" uv pip install \ numpy ``` -Verify versions, import locations, and the backend: +Change to the temporary directory and invoke the environment's interpreter +explicitly so neither the checkout nor another Python installation can satisfy +the check: -```python +```bash +cd "$TMP_DIR" +"$TMP_DIR/.venv/bin/python" - <<'PY' import importlib.metadata as metadata import reflex_xy @@ -198,13 +202,15 @@ assert kernels.BACKEND == "native" print(xy.__file__) print(reflex_xy.__file__) +PY ``` Both paths must point into the clean environment's `site-packages`. ## 6. Run published-package examples -Exercise more than imports. From the clean environment: +Exercise more than imports. Run every example from `$TMP_DIR` with +`"$TMP_DIR/.venv/bin/python"`: - build the README line chart and export non-empty HTML and SVG files; - build the first-chart scatter and export self-contained HTML; From f1c8c24d63c01bc95eecc05ed225f7ec4ad2d9c5 Mon Sep 17 00:00:00 2001 From: Farhan Ali Raza Date: Mon, 26 Oct 2026 16:43:02 +0500 Subject: [PATCH 5/7] Harden release verification and recovery guidance --- spec/process/release-checklist.md | 61 +++++++++++++++++++++++++++---- 1 file changed, 53 insertions(+), 8 deletions(-) diff --git a/spec/process/release-checklist.md b/spec/process/release-checklist.md index 870b77cde..72653754f 100644 --- a/spec/process/release-checklist.md +++ b/spec/process/release-checklist.md @@ -42,9 +42,11 @@ gh workflow run build_release_artifacts.yml \ -f artifact-prefix=dry-run-- ``` -The optional `tag` input can test a specific valid release-version shape. The -manual workflow builds and verifies artifacts only; it has no PyPI upload, tag, -or GitHub release step. +Leaving `tag` empty is the normal dry-run path. To test version derivation, +supply a valid **unused** tag: every matrix checkout fetches existing tags and +then creates the requested tag locally, so an existing release tag fails by +design. The local tag is never pushed. The manual workflow builds and verifies +artifacts only; it has no PyPI upload, tag-push, or GitHub release step. ## 2. Materialize the changelog @@ -163,10 +165,31 @@ Do not call the release complete until all of these pass: manifest; and - the post-release docs deployment dispatch. -Confirm PyPI exposes the version and expected files: +Confirm PyPI exposes the version and at least one file matching every artifact +pattern required by `[tool.reflex-release]`: ```bash -curl --fail https://pypi.org/pypi/xy/X.Y.Z/json +VERSION=X.Y.Z python3 - <<'PY' +import fnmatch +import json +import os +import tomllib +import urllib.request + +version = os.environ["VERSION"] +request = urllib.request.Request( + f"https://pypi.org/pypi/xy/{version}/json", + headers={"User-Agent": "xy-release-checklist"}, +) +with urllib.request.urlopen(request) as response: + filenames = {item["filename"] for item in json.load(response)["urls"]} +with open("pyproject.toml", "rb") as pyproject: + builds = tomllib.load(pyproject)["tool"]["reflex-release"]["custom-build"] +patterns = next(build["expect-artifacts"] for build in builds if "xy" in build["packages"]) +missing = [pattern for pattern in patterns if not any(fnmatch.fnmatch(name, pattern) for name in filenames)] +assert not missing, f"PyPI is missing expected xy artifacts: {missing}" +print("\n".join(sorted(filenames))) +PY ``` Create a new environment outside the repository so the checkout cannot shadow @@ -276,13 +299,35 @@ causes `release_from_changelog.yml` to retry the same untagged version. A manual dispatch of that workflow on the branch can also re-check it. Do not create a tag or release to recover the run. -### PyPI accepted only some files, or tagging failed after upload +### PyPI accepted only some files, or the tag push failed Leave every accepted PyPI file unchanged. Re-run the same release pipeline for the same changelog version and commit. `uv publish --check-url https://pypi.org/simple/` skips files already present and uploads any missing -ones; after a successful upload, `tag-and-release` can safely finish the tag and -GitHub release. Never force-move a resulting tag. +ones. If no tag exists, `tag-and-release` can safely retry the tag push and +GitHub release. Never create or force-move the tag manually. + +### The tag exists but GitHub release creation failed + +The publish is complete and the tag is immutable. Do not rerun the changelog +pipeline: detection may no-op once it sees the tag. Download the failed run's +`publish-xy` artifact, which contains the exact `release_notes.md` and +`SHA256SUMS` produced before approval, then create the missing release from the +existing tag: + +```bash +gh run download RUN_ID --repo reflex-dev/xy --name publish-xy +gh release create vX.Y.Z \ + --repo reflex-dev/xy \ + --verify-tag \ + --title vX.Y.Z \ + --notes-file release_notes.md \ + SHA256SUMS +``` + +Add `--prerelease` for a prerelease. Confirm the release targets the immutable +tag and contains the saved checksum manifest before manually dispatching the +post-release docs workflow if it did not run. ### The Python floor fails because Reflex is missing From ab9d044371804045862e72f98c7985d2fca821f0 Mon Sep 17 00:00:00 2001 From: Farhan Ali Raza Date: Mon, 26 Oct 2026 17:09:05 +0500 Subject: [PATCH 6/7] Clarify release verification prerequisites --- spec/process/release-checklist.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/spec/process/release-checklist.md b/spec/process/release-checklist.md index 72653754f..74fa723a8 100644 --- a/spec/process/release-checklist.md +++ b/spec/process/release-checklist.md @@ -166,10 +166,11 @@ Do not call the release complete until all of these pass: - the post-release docs deployment dispatch. Confirm PyPI exposes the version and at least one file matching every artifact -pattern required by `[tool.reflex-release]`: +pattern required by `[tool.reflex-release]`. Run this check with Python 3.11 or +later (`python3.11` is shown because the script uses `tomllib`): ```bash -VERSION=X.Y.Z python3 - <<'PY' +VERSION=X.Y.Z python3.11 - <<'PY' import fnmatch import json import os @@ -325,8 +326,11 @@ gh release create vX.Y.Z \ SHA256SUMS ``` -Add `--prerelease` for a prerelease. Confirm the release targets the immutable -tag and contains the saved checksum manifest before manually dispatching the +Add `--prerelease` for a prerelease. When the failed workflow's `MARK_LATEST` +value was false, also add `--latest=false` (including for hotfixes and other +non-latest releases); omit that option when `MARK_LATEST` was true. Confirm the +release targets the immutable tag, preserves the intended Latest state, and +contains the saved checksum manifest before manually dispatching the post-release docs workflow if it did not run. ### The Python floor fails because Reflex is missing From 926a60fc7a1417210954b58e87bff2a42b864f4d Mon Sep 17 00:00:00 2001 From: Farhan Ali Raza Date: Mon, 26 Oct 2026 18:58:54 +0500 Subject: [PATCH 7/7] Allow browser benchmarks time to provision --- .github/workflows/ci.yml | 5 ++++- scripts/verify_ci_workflow.py | 2 +- spec/benchmarks/methodology.md | 4 ++++ tests/test_verify_ci_workflow.py | 4 ++-- 4 files changed, 11 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1ff632478..bd120c3d8 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -323,7 +323,10 @@ jobs: benchmark_vs: name: Cross-library benchmark (${{ matrix.name }}) runs-on: ubuntu-latest - timeout-minutes: 10 + # Browser provisioning occasionally takes more than ten minutes on the + # shared runner mirror. Keep enough room for installation plus the bounded + # benchmark so a slow mirror cannot suppress a matrix artifact. + timeout-minutes: 20 continue-on-error: true strategy: fail-fast: false diff --git a/scripts/verify_ci_workflow.py b/scripts/verify_ci_workflow.py index 0b7a9073f..bbd22e11f 100644 --- a/scripts/verify_ci_workflow.py +++ b/scripts/verify_ci_workflow.py @@ -863,7 +863,7 @@ def validate_ci_workflow(path: Path = DEFAULT_CI_WORKFLOW) -> list[str]: "CI", "parallel cross-library benchmark matrix", "continue-on-error: true", - "timeout-minutes: 10", + "timeout-minutes: 20", "fail-fast: false", "matrix:", "xy: false", diff --git a/spec/benchmarks/methodology.md b/spec/benchmarks/methodology.md index 211e2bf90..7baeebcd7 100644 --- a/spec/benchmarks/methodology.md +++ b/spec/benchmarks/methodology.md @@ -396,6 +396,10 @@ unexplained engine slowdown. no `continue-on-error` are `matplotlib_reference`, `test`, `browser_conformance`, `python_floor`, `sdist`, `wheels`, and `install_without_rust`; only `benchmark_vs`, `benchmark_methodology`, and `benchmark` are non-blocking. +Each isolated `benchmark_vs` shard has a 20-minute job envelope so slow shared- +runner browser provisioning cannot consume the whole job before the bounded +benchmark runs and uploads its report; the benchmark's per-measurement budget +remains 45 seconds. `matplotlib_reference` is **not** absent from the Release-Blocking Gates table in `spec/process/production-readiness.md`: it is the "Matplotlib reference" row, whose evidence column names exactly the job's two halves, diff --git a/tests/test_verify_ci_workflow.py b/tests/test_verify_ci_workflow.py index 929371093..502419f26 100644 --- a/tests/test_verify_ci_workflow.py +++ b/tests/test_verify_ci_workflow.py @@ -924,13 +924,13 @@ def test_ci_workflow_rejects_missing_cross_library_job_timeout(tmp_path: Path) - workflow = Path(".github/workflows/ci.yml").read_text(encoding="utf-8") path = tmp_path / "ci.yml" path.write_text( - workflow.replace(" timeout-minutes: 10\n", "", 1), + workflow.replace(" timeout-minutes: 20\n", "", 1), encoding="utf-8", ) errors = verify_ci_workflow.validate_ci_workflow(path) - assert any("benchmark_vs" in error and "timeout-minutes: 10" in error for error in errors) + assert any("benchmark_vs" in error and "timeout-minutes: 20" in error for error in errors) def test_ci_workflow_rejects_unlocked_competitor_install(tmp_path: Path) -> None: