Summary
The last three Release Benchmarks runs reached the 60-minute job timeout while
saving the Criterion baseline. Packaging, the temporary artifact upload, and
release publication were therefore skipped for v0.4.3, v0.4.4, and v0.4.5.
The v0.4.5 run shows that this is a capacity-planning defect rather than an
input-validation failure: benchmark input validation passed, vs_linalg used
nearly the entire remaining hour, and the runner cancelled the exact suite
during exact_d2/det_exact.
Observed runs:
Current State
- The
release-baseline job has timeout-minutes: 60.
- One step runs full
vs_linalg and exact Criterion suites sequentially with
their normal release settings.
- In v0.4.5,
vs_linalg completed around 57 minutes after the step began. The
exact binary started with less than a minute left and was cancelled before
completing its first dimension.
- Partial
target/criterion output is not packaged, which is correct, but no
durable baseline has been produced by the three most recent releases.
Proposed Changes
- Measure and document the per-suite runtime and the expected worst-case
release-runner budget.
- Choose a release-quality execution plan that leaves deliberate headroom on
GitHub-hosted runners. This may split suites into independently bounded jobs,
right-size the outer timeout, or narrow redundant sampling only where the
benchmark methodology supports it.
- Preserve a single complete, validated Criterion archive for the release;
fail closed instead of publishing partial measurements.
- Keep the exact and comparative benchmark coverage needed by the release and
README/report consumers.
- Add early diagnostics or timing summaries so future growth identifies the
suite that exhausted its budget.
- Update workflow tests and benchmarking/release documentation for the chosen
execution model.
Benefits
- Release baselines complete reliably with enough margin for runner variance.
- Runtime fixes preserve the scientific meaning and completeness of the
published performance evidence.
- Future timeout regressions identify the responsible suite before another
release is published without its archive.
Acceptance Criteria
Summary
The last three
Release Benchmarksruns reached the 60-minute job timeout whilesaving the Criterion baseline. Packaging, the temporary artifact upload, and
release publication were therefore skipped for v0.4.3, v0.4.4, and v0.4.5.
The v0.4.5 run shows that this is a capacity-planning defect rather than an
input-validation failure: benchmark input validation passed,
vs_linalgusednearly the entire remaining hour, and the runner cancelled the
exactsuiteduring
exact_d2/det_exact.Observed runs:
Current State
release-baselinejob hastimeout-minutes: 60.vs_linalgandexactCriterion suites sequentially withtheir normal release settings.
vs_linalgcompleted around 57 minutes after the step began. Theexactbinary started with less than a minute left and was cancelled beforecompleting its first dimension.
target/criterionoutput is not packaged, which is correct, but nodurable baseline has been produced by the three most recent releases.
Proposed Changes
release-runner budget.
GitHub-hosted runners. This may split suites into independently bounded jobs,
right-size the outer timeout, or narrow redundant sampling only where the
benchmark methodology supports it.
fail closed instead of publishing partial measurements.
README/report consumers.
suite that exhausted its budget.
execution model.
Benefits
published performance evidence.
release is published without its archive.
Acceptance Criteria
vs_linalgandexactrelease baselines complete on the supportedGitHub runner within a documented budget and with reasonable headroom.
target/criteriondataset.release performance tooling.
release documentation.
upload successfully.
applicable local CI gate pass.