Skip to content

Run each benchmark build from its own checkout - #33

Merged
philliphoff merged 1 commit into
mainfrom
claude/benchmark-base-checkout
Oct 11, 2026
Merged

philliphoff merged 1 commit into
mainfrom
claude/benchmark-base-checkout

Conversation

@philliphoff

Copy link
Copy Markdown
Owner

Summary

Fixes the Benchmarks workflow from #28. Its "base" runs measured the head's code, so every base-vs-head comparison so far has compared the head with itself.

Cause. BenchmarkDotNet's default toolchain rebuilds a generated project that references the benchmark .csproj, and it finds that project by searching from the current working directory. The workflow ran both builds from the head checkout, so the base run rebuilt and measured the head's library code.

Evidence.

How the base build ran S57DocumentReader.Read allocated
From the head checkout (current workflow) 7.62 MB, the head's figure
From its own checkout (this PR) 27.9 MB, the base's figure; it matches the old memory budget

Fix. run() now cds into the tree it measures: $RUNNER_TEMP/base for base and $GITHUB_WORKSPACE for head. Artifact paths were already absolute.

The comparisons on #28 and #31 were affected too. Neither changed library code, so their reports were right by accident. #32 merged before this fix reached it, so its CI comparison didn't show its gain. The memory budgets, which are the actual gate, were unaffected: they run in dotnet test, not through BenchmarkDotNet.

This PR changes no library code, so its own Benchmarks run will again compare identical code. It should now report equal allocations between base and head because they really are equal. The first meaningful comparison will be the next PR that touches the libraries.

Standards alignment

  • ISO/IEC 8211 (iso8211)
  • S-57 records, topology, updates or exchange sets (s57-enc)
  • Legacy chart viewer, maintenance only (chart-viewer)
  • N/A: build, CI, docs or tooling only

Standard section references cited in code or docs: none.

Tests

  • Added or updated xUnit tests under tests/
  • Test input is built in memory; no real ENC cells are committed
  • dotnet test --configuration Release passes locally. Not applicable: workflow change only. The workflow YAML parses, and the base/head behaviour was reproduced and fixed locally as shown above.

Documentation

  • Updated the affected package's src/<project>/README.md
  • Updated guides under docs/ if user-facing behavior changed
  • New public APIs have XML doc comments
  • docfx docfx.json --warningsAsErrors passes, if docs changed

Dependencies

  • No new NuGet dependencies, OR versions added to
    Directory.Packages.props (not in the .csproj)
  • gh-advisory-database security check run for any new dependency

Breaking changes

None.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YagvELyJKBANk61DyFGUNJ


Generated by Claude Code

BenchmarkDotNet rebuilds the benchmark project it finds from the working
directory, so the base build, run from the head checkout, measured the
head's code: base and head reported identical allocations. Locally, the
base build run from the head checkout reported S57DocumentReader at
7.62 MB on the harbour cell, the head's figure; run from its own tree it
reports 27.9 MB, the base's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YagvELyJKBANk61DyFGUNJ
@philliphoff
philliphoff merged commit f5663f6 into main Oct 11, 2026
12 checks passed
@philliphoff
philliphoff deleted the claude/benchmark-base-checkout branch October 11, 2026 23:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants