Skip to content

Add memory budget tests and base-vs-head benchmark comparison - #28

Merged
philliphoff merged 1 commit into
mainfrom
claude/performance-baseline
Oct 11, 2026
Merged

philliphoff merged 1 commit into
mainfrom
claude/performance-baseline

Conversation

@philliphoff

Copy link
Copy Markdown
Owner

Summary

This is the first step of the performance review. It adds checks that run on every change, so the reader/writer redesign that follows can be measured against today's baseline.

Elapsed time on shared GitHub runners varies 10–20% between identical runs, which is why S100's timing gates have to be loose and get ignored. This PR gates only on figures that are exact, and keeps timings informational:

What it measures Inputs Gate?
tests/EncDotNet.PerformanceTests (new) Bytes allocated by each reader, writer and model builder; bytes retained by the parsed models; linear growth with input size Synthetic harbour cell Yes, through dotnet test in CI
Benchmarks workflow (new) BenchmarkDotNet timings for base vs head Real NOAA cells downloaded in CI, plus the synthetic Large cell No; report in the job summary

Synthetic cells (tests/Shared)

  • Deterministic generator built on the library's own Iso8211FieldBuilder and writer. It uses its own random number generator, so the bytes are identical on every runtime.
  • Profiles are calibrated against NOAA US5WA50M (Harbour) and US4MD81M (Approach). Record counts, edge-length distribution, attributes per feature and pointers per feature all match.
  • Checked against the real cells: parse allocations and retained memory are within about 3% (S57DocumentReader allocates 210.8 MB vs 203.7 MB; retained Iso8211Document is 19.27 MB vs 19.25 MB).
  • Large is four times Approach, with a 15,000-point edge whose record exceeds the 99,999-byte leader limit. Real cells this size are rare, so this case stays synthetic.
  • Update files (.001 and so on) exercise ApplyChanges: attribute changes, deletions, insertions and SGCC coordinate splices.

Budget tests

  • GC.GetAllocatedBytesForCurrentThread around each operation, after warm-up.
  • The project sets TieredCompilation=false. On .NET 10, tier-1 code with dynamic PGO and escape analysis stack-allocates some objects, so with tiering on, the same parse's allocation count moved between 21.9 MB and 29.3 MB within one process. With tiering off, counts are identical across runs.
  • Budgets fail in both directions: over +1% is a regression; under −5% means lower the budget, so a later regression can't hide in the headroom. The failure message gives the new figure.
  • Checked: one extra object per field in Iso8211DocumentReader fails Iso8211DocumentReader.Read at +7.5%.

Benchmarks

  • No longer requires ENCDOTNET_ENC_ROOT; synthetic inputs are always available, and any real cells under it are added.
  • New: writer, DDR reader, SG2D field decoding, S57Chart.FromDocument, read-to-chart and ApplyChanges. A Core category holds the 7 the PR job runs.
  • The workflow builds the head's benchmark harness against the base library code, so only the library differs.
  • It runs base and head in 2 rounds, base-then-head and then head-then-base, on one runner against the same bytes. It flags only changes over 10% in every round.
  • In a local check comparing identical code, single inputs showed −12% and −14%, and neither was flagged.
  • A full run is available through workflow_dispatch with scope: all.

Scripts

  • scripts/DownloadBenchmarkCells.cs downloads US5WA50M, US4MD81M and US3WA01M using the NOAA catalog client. It records edition, update number and SHA-256 in cells.txt, and the report lists them. Cells are cached per ISO week; if the download fails, the run uses synthetic cells only.
  • scripts/CompareBenchmarks.cs produces the Markdown report.

Standards alignment

  • ISO/IEC 8211 (iso8211)
  • S-57 records, topology, updates or exchange sets (s57-enc)
  • Legacy chart viewer, maintenance only (chart-viewer)
  • N/A: build, CI, docs or tooling only

Standard section references cited in code or docs: S-57 Part 3, Annex A (DDR field definitions in the generator) and Annex A.3 (records over 99,999 bytes); S-57 Part 3, 8.4.3 (SGCC update control).

Tests

  • Added or updated xUnit tests under tests/
  • Test input is built in memory; no real ENC cells are committed. Downloaded cells go to .benchmark-cells/, which is git-ignored.
  • dotnet test --configuration Release passes locally: 492 unit tests and 12 performance tests on linux-x64. Windows and Linux arm64 are covered only by CI; budgets should match there because object layout is the same on 64-bit platforms.

Also run locally: dotnet format (whitespace and the CI diagnostics) is clean, and the workflow's base/head build and comparison steps were simulated with a dry job. The NOAA download could not run here because the sandbox's network policy blocks charts.noaa.gov; it only compiles. Its first real run will be in this PR's Benchmarks job.

Documentation

  • Updated the affected package's src/<project>/README.md
  • Updated guides under docs/ if user-facing behavior changed
  • New public APIs have XML doc comments
  • docfx docfx.json --warningsAsErrors passes, if docs changed

CONTRIBUTING.md gains a Performance section covering budgets, updating them, and running locally. .github/copilot-instructions.md lists the new projects.

Dependencies

  • No new NuGet dependencies, OR versions added to
    Directory.Packages.props (not in the .csproj)
  • gh-advisory-database security check run for any new dependency

Breaking changes

None. No library code changes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YagvELyJKBANk61DyFGUNJ


Generated by Claude Code

Elapsed time on shared GitHub runners varies by 10-20% between identical
runs, so timing gates either flake or get ignored. Gate on figures that
are exact instead, and keep timings informational.

- tests/Shared: a deterministic synthetic S-57 cell generator. Profiles
  are calibrated against NOAA US5WA50M and US4MD81M; they reproduce the
  real cells' parse allocations and retained memory within about 3%.
  The Large profile adds scale and a record over 99,999 bytes.
- tests/EncDotNet.PerformanceTests: allocation and retained-memory
  budgets for each reader, writer and model builder, plus a linear
  growth check. Runs in CI with dotnet test. Tiered compilation is off
  because tier-1 escape analysis changes allocation counts mid-run.
- tests/EncDotNet.Benchmarks: per-input benchmarks over synthetic cells
  and any real cells under ENCDOTNET_ENC_ROOT, no longer requiring it;
  adds writer, DDR, field-reader, chart and update benchmarks.
- Benchmarks workflow: runs the Core benchmarks for base and head on one
  runner, interleaved, against the same bytes, and reports changes that
  hold in every round to the job summary. Never fails the build.
- scripts/DownloadBenchmarkCells.cs and scripts/CompareBenchmarks.cs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YagvELyJKBANk61DyFGUNJ
@philliphoff
philliphoff merged commit 8fad40c into main Oct 11, 2026
12 checks passed
@philliphoff
philliphoff deleted the claude/performance-baseline branch October 11, 2026 18:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants