Skip to content

Keep release Criterion baseline generation within the workflow budget #224

Description

@acgetchell

Summary

The last three Release Benchmarks runs reached the 60-minute job timeout while
saving the Criterion baseline. Packaging, the temporary artifact upload, and
release publication were therefore skipped for v0.4.3, v0.4.4, and v0.4.5.

The v0.4.5 run shows that this is a capacity-planning defect rather than an
input-validation failure: benchmark input validation passed, vs_linalg used
nearly the entire remaining hour, and the runner cancelled the exact suite
during exact_d2/det_exact.

Observed runs:

Current State

  • The release-baseline job has timeout-minutes: 60.
  • One step runs full vs_linalg and exact Criterion suites sequentially with
    their normal release settings.
  • In v0.4.5, vs_linalg completed around 57 minutes after the step began. The
    exact binary started with less than a minute left and was cancelled before
    completing its first dimension.
  • Partial target/criterion output is not packaged, which is correct, but no
    durable baseline has been produced by the three most recent releases.

Proposed Changes

  1. Measure and document the per-suite runtime and the expected worst-case
    release-runner budget.
  2. Choose a release-quality execution plan that leaves deliberate headroom on
    GitHub-hosted runners. This may split suites into independently bounded jobs,
    right-size the outer timeout, or narrow redundant sampling only where the
    benchmark methodology supports it.
  3. Preserve a single complete, validated Criterion archive for the release;
    fail closed instead of publishing partial measurements.
  4. Keep the exact and comparative benchmark coverage needed by the release and
    README/report consumers.
  5. Add early diagnostics or timing summaries so future growth identifies the
    suite that exhausted its budget.
  6. Update workflow tests and benchmarking/release documentation for the chosen
    execution model.

Benefits

  • Release baselines complete reliably with enough margin for runner variance.
  • Runtime fixes preserve the scientific meaning and completeness of the
    published performance evidence.
  • Future timeout regressions identify the responsible suite before another
    release is published without its archive.

Acceptance Criteria

  • Both vs_linalg and exact release baselines complete on the supported
    GitHub runner within a documented budget and with reasonable headroom.
  • The workflow packages and uploads only a complete, validated
    target/criterion dataset.
  • The resulting archive contains every benchmark group consumed by the
    release performance tooling.
  • The runtime change does not silently weaken sampling or the claims made by
    release documentation.
  • A representative workflow run reaches packaging and temporary artifact
    upload successfully.
  • GitHub Actions syntax, security checks, benchmark tooling tests, and the
    applicable local CI gate pass.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingperformancePerformance related issuesreleaseRelease preparation and packagingreproducibilityReproducibility and artifact packagingtestingIssues that come up in testing

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions