testing.md

Testing

tools/test_plan.py decides what a change must test. Cargo's graph supplies every Rust package, every dependency edge and the split between the fast and the captured-fixture tier. testing/suites.toml holds only what Cargo cannot see. obc suites check validates the test plan and runs on every pull request.

Commands

obc test -p obc-app                       # one package on nextest, then its doctests
obc test -p obc-app --lib                 # the library target alone
obc test fixtures -p obc-route            # the captured-fixture tier, after a fixture sync
obc test affected --base origin/develop   # the selection CI runs; --dry-run prints it
obc test full                             # cross-cutting changes only
obc check fmt clippy device docs          # named CI gates; obc check full runs them all
obc suites check                          # plan drift and command resolution
obc suites select --base REF [--release]  # the plan as text or json

obc test always needs a scope; it never expands to the workspace on its own. affected reads the working tree, prints every selected unit with a reason, and stops at the first failure. A suite whose platforms exclude the host is reported as skipped, never as passed. Every Cargo command needs the CI-pinned runner: cargo install cargo-nextest --version 0.9.143 --locked. The Python suites need pip install -r tools/requirements-test.txt --group planner-snow --group planner-sun (pip 25.1 or later).

Routes

route in testing/suites.toml says when a unit runs:

Route	Meaning
ordinary	the change selects it (the default for every Rust package)
required	it runs whenever one of its CI jobs starts
manual	its own command only: generators, probes, captured-source checks, iOS application tests
live	it contacts a live service; its own command only

Selection is per suite, never per test function. A binary that mixes ordinary work with captured-fixture, live or manual work is split into separate units. Captured fixtures are ordinary work gated on required-features = ["external-fixtures"]; a missing package fails with the exact obc fixtures sync command. Physical procedures have no route; they live in their issue. Each Monday, test-weekly.yml runs manual iOS application tests and ordinary storage tests with default features.

The plan documents

A [[package]] entry adds what Cargo cannot say about a package:

Field	Meaning
name	the Cargo package
route	ordinary by default
triggers	paths whose edge no build graph carries
fixtures	true when a change under fixtures/ reaches the package
command	the local command for a standalone Cargo root
platforms	supported platforms, when restricted

A [[suite]] entry is verification no Cargo package owns:

Field	Meaning
id	stable identifier used by commands and reports
route	one of the four routes
command	one repository-root command, for local and CI use
jobs	the CI jobs that execute it (ordinary and required need at least one)
triggers	paths that select it, including its own test sources
fixtures	captured or production-shaped inputs it needs
platforms, foundation, ci_only	platform restriction; selected by manifest or toolchain changes; not reproducible locally
scoped	follows its own inputs instead of broad policy or unowned-deletion selection
rust_packages	linked libraries; Cargo supplies their non-development dependencies
rust_excludes	source paths outside the linked feature set
package, targets	Cargo test targets this suite owns, which then belong to no tier

Neither entry lists dependencies, test counts, durations or source files; Cargo and the result artifacts supply those. The CI job table is in tools/test_plan.py; obc suites validate-filters checks that it and .github/workflows/ci.yml describe the same jobs.

Selection fails closed: an unowned source path or a selected suite without a CI route is an error. An error publishes no plan, so the ci aggregate fails. Foundation and policy changes select unscoped suites across the relevant graph. An unowned deletion also selects that graph.

The iOS suites are scoped. They follow their source, tests, build inputs and linked Rust libraries. Rust tests, examples, binaries and Markdown do not select a linked library build. A changed suite entry selects that suite; an unrelated entry does not select iOS. Changes to the selector, aggregate or CI workflow select scoped suites too. Full release verification includes every routed suite. Within ios-unit, only selected camera, OBCKit and PMTiles suites execute.

Real time in tests

Do not retry a flaky test; prefer observed state, a controllable clock or a protocol signal to a sleep. A suite that must wait on real time says why in a comment beside it.

CI artifacts

Every job uploads its native results after success or failure; a skipped step uploads nothing, and a missing expected file fails the upload. Download with gh run download RUN_ID --pattern '<prefix>*' --dir test-results.

Artifact	Holds
rust-test-ATTEMPT, rust-fixtures-ATTEMPT	nextest JUnit XML for the fast and fixture tiers; the fast artifact also holds doctests.log and formats-default.log
python-repository-tools-ATTEMPT, python-firmware-tools-ATTEMPT, python-builder-ATTEMPT	unittest and pytest XML
web-builder-ATTEMPT	Vitest JUnit XML
web-builder-browser-ATTEMPT, web-demo-browser-ATTEMPT	the two Chromium journeys, with a screenshot and trace on failure
ios-tests-coverage-ATTEMPT, ios-screenshots-ATTEMPT, ios-application-ATTEMPT	.xcresult bundles; restore the suffix and open in Xcode
desktop-tests-PLATFORM-ATTEMPT	desktop nextest results

The exit status and the CI log stay authoritative; an artifact does not prove a passing run.

Browser journeys

web.builder-browser runs the map builder in Chromium against a digest-pinned loopback catalog, assembles a map through the real WebAssembly bridge and requires the bytes to equal expected/map.obcm. web.demo-browser drives the landing page's save, view, reset and reload controls. For simultaneous runs in separate worktrees, set OBC_BROWSER_PORT to a free port; the defaults are 4180, 4178 and 4182.

live.builder-memory is the same builder against the published catalogue, and it is the one live suite with a browser in it. It assembles the largest published region and holds the wasm linear memory the worker reports at the end of the run against the peak the estimate that admitted the run projected. It is the measurement the projection cannot make: an assembler that buffers a whole map passes every arithmetic test of the estimator. It downloads about 890 MB, needs about 3 GB of free disk and takes minutes, so run it by hand before a release and after a change to the assembler or the estimator.