Skip to content

feat: enable cycle estimation and allocation exclusion by default#461

Open
not-matthias wants to merge 1 commit into
mainfrom
cod-2950-enable-cycle-estimationexclude-allocators-in-runner-for
Open

feat: enable cycle estimation and allocation exclusion by default#461
not-matthias wants to merge 1 commit into
mainfrom
cod-2950-enable-cycle-estimationexclude-allocators-in-runner-for

Conversation

@not-matthias

Copy link
Copy Markdown
Member

Summary

Graduate --cycle-estimation and --exclude-allocations out of the experimental flag set so both simulation behaviors are enabled for everyone by default.

  • Both are now normal (non-experimental) options with default_value_t = true.
  • They remain toggleable escape hatches: --cycle-estimation=false / --exclude-allocations=false (or env CODSPEED_CYCLE_ESTIMATION, CODSPEED_EXCLUDE_ALLOCATIONS) disable them.
  • Dropped from the Experimental help heading and no longer trigger the experimental-flags warning.
  • --experimental-fair-sched is intentionally left untouched (still experimental, default off).

Changes

File Change
src/cli/shared.rs Add cycle_estimation / exclude_allocations toggleable bools (default true, require_equals) to ExecAndRunSharedArgs
src/cli/experimental.rs Remove both from ExperimentalArgs; only experimental_fair_sched remains
src/cli/run/mod.rs, src/cli/exec/mod.rs Read the new shared fields
src/executor/config.rs OrchestratorConfig::test() defaults both true

Behavior note

Runner.exclude_allocations in upload metadata now serializes true on every simulation run (previously omitted via skip_serializing_if). This is the intended signal, just always present now.

Verification

  • cargo build, cargo fmt --check, cargo clippy --all-targets clean.
  • Verified via the built binary: --cycle-estimation, --cycle-estimation=false, and --exclude-allocations=false all parse; help lists both as normal options.
  • Pre-existing Unsupported system valgrind-executor test failures are environment-gated (unsupported valgrind host) and unrelated to this change.

@greptile-apps

greptile-apps Bot commented Jul 20, 2026

Copy link
Copy Markdown

Greptile Summary

This PR graduates cycle estimation and allocation exclusion into default-on runner options. The main changes are:

  • New shared run and exec flags for cycle estimation and allocation exclusion.
  • Hidden deprecated experimental flags with warning output.
  • Updated orchestrator defaults for run, exec, and test helpers.
  • Updated upload metadata comments for allocation-exclusion serialization.

Confidence Score: 4/5

This is close, but the metadata hash change should be handled before merging.

  • Default runs now serialize excludeAllocations: true.
  • The upload hash uses the serialized metadata, so normal runs can get a new hash for the same benchmark invocation.
  • The metadata version was not changed with that default-path shape change.

Files Needing Attention: src/cli/shared.rs and src/upload/interfaces.rs

Important Files Changed

Filename Overview
src/cli/shared.rs Adds default-on shared options for cycle estimation and allocation exclusion.
src/cli/experimental.rs Keeps the old experimental options hidden and warning-only after graduation.
src/cli/run/mod.rs Passes the new shared options into the run orchestrator config.
src/cli/exec/mod.rs Passes the new shared options into the exec orchestrator config.
src/executor/config.rs Updates test orchestrator defaults to match the new default-on behavior.
src/upload/interfaces.rs Keeps allocation exclusion skipped when false and updates the serialization comment.

Fix All in Claude Code Fix All in Codex

Prompt To Fix All With AI
Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
src/cli/shared.rs:130
**Upload Hash Changes**

Defaulting `exclude_allocations` to `true` still changes upload metadata for normal `run` and `exec` invocations. The value flows into `Runner.exclude_allocations`, which is only skipped when it is `false`, so the default path now serializes `excludeAllocations: true`. Because the upload hash is computed from the serialized metadata and the metadata version is unchanged, the same benchmark invocation can get a different run hash after this release instead of going through an explicit metadata migration. The `--exclude-allocations=false` opt-out keeps the legacy shape, but that does not cover the new default path.

Reviews (5): Last reviewed commit: "feat: enable cycle estimation and alloca..." | Re-trigger Greptile

Comment thread src/cli/shared.rs
Comment thread src/cli/shared.rs
Comment thread src/cli/shared.rs
@codspeed-hq

codspeed-hq Bot commented Jul 20, 2026

Copy link
Copy Markdown

Performance comparison unavailable

A heads-up: this makes performance comparison unavailable when comparing benchmarks before and after this change:

  • Cycle estimation changes how benchmark performance is calculated. It was enabled on only one side of the comparison (base off → head on), so the values are not directly comparable. View runner releases
  • Allocator-time exclusion changes benchmark values by removing allocator time from the measurement. It was enabled on only one side of the comparison (base off → head on), so the values are not directly comparable. View runner releases

Comparing cod-2950-enable-cycle-estimationexclude-allocators-in-runner-for (d4b65cd) with main (8611106)

Open in CodSpeed

@not-matthias
not-matthias marked this pull request as ready for review July 21, 2026 18:27

@GuillaumeLagrange GuillaumeLagrange left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

olgtm, make sure to not merge before we minor release

Comment thread src/cli/experimental.rs
Comment thread src/cli/shared.rs
@not-matthias
not-matthias force-pushed the cod-2950-enable-cycle-estimationexclude-allocators-in-runner-for branch from 92f25c7 to 0f82e27 Compare July 27, 2026 09:26
Comment thread src/cli/experimental.rs
@not-matthias
not-matthias force-pushed the cod-2950-enable-cycle-estimationexclude-allocators-in-runner-for branch from 0f82e27 to 13499ef Compare July 27, 2026 09:32
Comment thread src/cli/experimental.rs
@not-matthias
not-matthias force-pushed the cod-2950-enable-cycle-estimationexclude-allocators-in-runner-for branch from 13499ef to a50fe11 Compare July 27, 2026 09:45
Graduate --cycle-estimation and --exclude-allocations out of the
experimental flag set. They are now normal, default-on options that
can be disabled with --cycle-estimation=false / --exclude-allocations=false
(env: CODSPEED_CYCLE_ESTIMATION, CODSPEED_EXCLUDE_ALLOCATIONS).

--experimental-fair-sched remains experimental.
@not-matthias
not-matthias force-pushed the cod-2950-enable-cycle-estimationexclude-allocators-in-runner-for branch from a50fe11 to d4b65cd Compare July 27, 2026 10:02
Comment thread src/cli/shared.rs
long,
env = "CODSPEED_EXCLUDE_ALLOCATIONS",
default_value_t = true,
action = clap::ArgAction::Set

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Upload Hash Changes

Defaulting exclude_allocations to true still changes upload metadata for normal run and exec invocations. The value flows into Runner.exclude_allocations, which is only skipped when it is false, so the default path now serializes excludeAllocations: true. Because the upload hash is computed from the serialized metadata and the metadata version is unchanged, the same benchmark invocation can get a different run hash after this release instead of going through an explicit metadata migration. The --exclude-allocations=false opt-out keeps the legacy shape, but that does not cover the new default path.

Knowledge Base Used: Upload flow

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/cli/shared.rs
Line: 130

Comment:
**Upload Hash Changes**

Defaulting `exclude_allocations` to `true` still changes upload metadata for normal `run` and `exec` invocations. The value flows into `Runner.exclude_allocations`, which is only skipped when it is `false`, so the default path now serializes `excludeAllocations: true`. Because the upload hash is computed from the serialized metadata and the metadata version is unchanged, the same benchmark invocation can get a different run hash after this release instead of going through an explicit metadata migration. The `--exclude-allocations=false` opt-out keeps the legacy shape, but that does not cover the new default path.

**Knowledge Base Used:** [Upload flow](https://app.greptile.com/codspeed/-/custom-context/knowledge-base/codspeedhq/codspeed/-/docs/upload.md)

How can I resolve this? If you propose a fix, please make it concise.

Fix in Claude Code Fix in Codex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants