Skip to content

Pick benchmark cells by rule and halve the Benchmarks job's run time - #31

Merged
philliphoff merged 2 commits into
mainfrom
claude/benchmark-cell-selection
Oct 11, 2026
Merged

philliphoff merged 2 commits into
mainfrom
claude/benchmark-cell-selection

Conversation

@philliphoff

@philliphoff philliphoff commented Oct 11, 2026 •

Copy link
Copy Markdown
Owner

Summary

Two changes to the Benchmarks workflow added in #28.

1. Pick cells from the live NOAA catalog by rule

The first Benchmarks run (#28) measured only one real cell. Two of the three hard-coded names have left NOAA's catalog:

warning: US5WA50M is not in the NOAA catalog; skipping it.
warning: US3WA01M is not in the NOAA catalog; skipping it.
US4MD81M edition 36 update 1 (323,000 bytes, sha256 F266B95ED49BD1B2)

NOAA withdraws and renames cells as it re-schemes its coverage, so scripts/DownloadBenchmarkCells.cs now picks cells by rule instead of by name:

  • For each usage band (harbour 5, approach 4, coastal 3, from the cell file name per S-57 Appendix B.1, 5.6.4), it takes the active cell with update files at the 90th percentile of download size. That cell is substantial but typical, and the choice is deterministic for a given catalog.
  • Explicit cell names on the command line still override the rule. The workflow's cache key includes the script's hash, so the cached cells refresh.
  • On this PR's first run the rule picked US5BOSBE, US4LA1FM and US3AK1HP, and all three were benchmarked with the synthetic Large cell.

The current US4MD81M (edition 36) is about 320 KB, against the 3.5 MB edition the synthetic Approach profile was calibrated on. The profile docs now say they model earlier editions and stand for the larger cells an application may still meet. No budgets change; the synthetic bytes are identical.

2. Shorter benchmark iterations

With three real cells the job took 26 minutes. Nearly all of it was BenchmarkDotNet's measurement schedule: the default 500 ms iterations make each case take 10–15 s however fast the operation is. Measured locally on the Core set (4 inputs, 28 cases, one pass):

Settings Time Median per-iteration spread Median standard error
10 × 500 ms (previous) 420 s 6.5% 2.06%
10 × 500 ms, --inProcess 391 s — —
10 × 100 ms 138 s 11.9% 3.75%
15 × 200 ms (this PR) 214 s 6.8% 1.85%

15 × 200 ms halves the time while the median, which is what the comparison uses, stays as precise. --inProcess saved only 7%, so the job keeps a separate process per case.

The workflow is still not a required check: the main CI finishes in about 4 minutes, and you can merge while Benchmarks is still running.

Standards alignment

  • ISO/IEC 8211 (iso8211)
  • S-57 records, topology, updates or exchange sets (s57-enc)
  • Legacy chart viewer, maintenance only (chart-viewer)
  • N/A: build, CI, docs or tooling only

Standard section references cited in code or docs: S-57 Appendix B.1, 5.6.4 (cell file naming).

Tests

  • Added or updated xUnit tests under tests/
  • Test input is built in memory; no real ENC cells are committed
  • dotnet test --configuration Release passes locally (performance tests unchanged, 12/12)

Documentation

  • Updated the affected package's src/<project>/README.md
  • Updated guides under docs/ if user-facing behavior changed
  • New public APIs have XML doc comments
  • docfx docfx.json --warningsAsErrors passes, if docs changed

Dependencies

  • No new NuGet dependencies, OR versions added to
    Directory.Packages.props (not in the .csproj)
  • gh-advisory-database security check run for any new dependency

Breaking changes

None.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YagvELyJKBANk61DyFGUNJ

Two of the three hard-coded cells (US5WA50M, US3WA01M) are no longer in
NOAA's catalog, so the first Benchmarks run measured only US4MD81M. NOAA
withdraws and renames cells as it re-schemes its coverage, so choose
cells by rule instead: for each usage band (harbour, approach, coastal),
the active cell with update files at the 90th percentile of download
size. Explicit cell names still override the rule.

The current US4MD81M (edition 36) is about 320 KB, against the 3.5 MB
edition the synthetic Approach profile was calibrated on. Say so in the
profile docs: the larger profiles stand for the bigger cells an
application may still meet, not a typical current cell.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YagvELyJKBANk61DyFGUNJ
Each case took 10-15 s under BenchmarkDotNet's default 500 ms iterations,
whatever the operation's cost, so the Benchmarks job took 26 minutes.
Locally, 15 iterations of 200 ms halve a pass (420 s to 214 s) with the
median's standard error unchanged (2.06% to 1.85%). Running in-process
saved only 7%, so it stays out-of-process.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YagvELyJKBANk61DyFGUNJ
@philliphoff philliphoff changed the title Pick benchmark cells from the live NOAA catalog by rule Pick benchmark cells by rule and halve the Benchmarks job's run time Oct 11, 2026
@philliphoff
philliphoff merged commit 07446bf into main Oct 11, 2026
12 checks passed
@philliphoff
philliphoff deleted the claude/benchmark-cell-selection branch October 11, 2026 21:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants