| semantic-links |
|
|---|
This guide helps contributors select the lowest-cost test layer that can prove an observable behavior. It complements, rather than replaces, the detailed procedures and conventions linked below.
- More unit tests are better. They are the primary target for coverage growth because they are fast, deterministic, and low-maintenance.
- Test as close to the code as possible. Put behavior in a package-level test when it can be proved there; do not promote it to the root application or E2E layer without a boundary that requires it.
- Use root-level integration tests only for orchestration. These tests are for behavior involving multiple services assembled by the tracker application container, rather than an individual package.
- Use E2E as the outermost safety net. It validates the packaged artifact and real-client interoperability, but is slower and less precise than lower layers.
When behavior remains untested, record the reason as required by the unit-test skill.
The project had no automated tests roughly three years ago. E2E tests were introduced first because they could be added without refactoring the existing application. The subsequent extraction and refactoring of workspace packages made unit and package-level integration tests practical.
The E2E suite is therefore proportionally large, but it is not the default model for new coverage. The current direction is toward maintainable unit and package-level tests, as supported by EPIC #1347.
| Layer | Use it when | It proves | It does not prove | Representative example | Authoritative guidance |
|---|---|---|---|---|---|
| Unit and documentation tests | A package type, function, or module has behavior that can run without I/O or a deployed service. | The focused behavior is correct in isolation. | Cross-package wiring, process behavior, or a packaged runtime. | Driver tests |
Unit-test skill; package testing guidance |
| Package-level in-process integration | A package boundary needs real collaborators, such as a server and its handler, without the complete application. | The package's components work together through its public boundary. | Application-wide service coordination or the compiled tracker executable. | HTTP server contract tests | Package testing guidance; test refactoring patterns |
| Root application-level in-process integration | An observable behavior needs the complete application container and multiple coordinated services. | Application startup, cross-service coordination, aggregate metrics, and job or shutdown orchestration. | OS process boundaries, signals sent to the tracker executable, or container-image behavior. | Port-zero metrics suite | Root integration-test guidance |
| Executable-boundary integration | The behavior requires starting the compiled tracker as a child process, such as OS-signal handling. | The native executable starts and reacts correctly at its process boundary. | Container-image behavior or interoperability with an external BitTorrent client. | Native tracker fixture | Child-process configuration isolation |
| Container E2E | The tracker must be exercised as the built container artifact with project-controlled clients. | The image builds and tracker behavior works through its network boundary. | Interoperability with a production BitTorrent client or every database backend. | e2e_tests_runner |
E2E tools usage; container workflow |
| Container plus qBittorrent E2E | Compatibility must be demonstrated against a real BitTorrent client and configured database backend. | The containerized tracker interoperates with qBittorrent for the selected backend. | Isolated package behavior or exhaustive coverage of all failure paths. | qbittorrent_e2e_runner |
E2E tools usage; container workflow |
| Database compatibility | A persistence change affects MySQL or PostgreSQL driver behavior or supported-version compatibility. | The selected tracker-core database-driver scenarios work against the workflow's version matrix. | Complete tracker container behavior or SQLite behavior not covered by the scenario. | Database compatibility workflow | Database compatibility workflow; package testing guidance |
| Manual verification | Automated evidence cannot fully demonstrate a user- or environment-facing outcome. | The recorded scenario worked in the reviewed environment. | Repeatable coverage across inputs, platforms, and future changes. | Manual HTTP completion E2E procedure | Issue-specification workflow; the applicable testing skill |
Choose focused checks while developing, then use the repository gates for their respective responsibilities:
| Owner | Responsibility | Detailed procedure |
|---|---|---|
| Developer-focused checks | Give fast feedback for the changed package, test, documentation page, or workflow. They do not replace repository gates. | Package testing guidance |
| Pre-commit | Performs the fast local gate: dependency checks, all linters, Containerfile linting, and documentation tests. | Pre-commit checks |
| Pre-push | Performs nightly checks and the full stable test suite; it intentionally excludes E2E tests. | Pre-push checks |
| CI | Is the merge authority. It runs workflow-selected validation, including container and qBittorrent E2E coverage where applicable. | Testing workflow; container workflow |
| Manual verification | Complements automated evidence with scenario status and recorded evidence in the relevant issue specification. | Issue-specification workflow |
Use three distinct verification activities; they produce different evidence and must not substitute for one another.
| Activity | Purpose | Required evidence |
|---|---|---|
| Automatic tests | Maintained, repeatable claims about product behavior at the lowest suitable test layer. | Rust test code executed through the repository test toolchain. |
| Manual verification | Real human-oriented use of a finished feature or reproduction of a fixed bug, especially to reveal integration and usability gaps. | Actual steps, commands, program output, relevant tracker logs, and conclusions in the issue-local manual-verification-evidence.md. |
| Disposable verification script | Temporary issue-local automation that efficiently drives or captures a concrete verification scenario. | The script, its automatic-test rationale, and its removal/retention owner in the issue specification. Python additionally requires a recorded reason Rust is unsuitable. |
Automatic tests should absorb durable product-behavior checks from disposable scripts when practical. Manual verification remains necessary after automated tests pass because it verifies how a person actually uses the finished artifact. It routinely exposes integration and usability gaps that tests written by the implementer do not, such as "how am I supposed to use this?" moments, unclear error messages, or missing documentation.
Examples of human-oriented manual verification:
- Bug fix: reproduce the original bug on the fixed build using the same steps the reporter would follow, and confirm it no longer happens.
- New CLI option: build the release binary, read only the
--helpoutput and the documentation, start the tracker the way an operator would, and check the logs show the expected effect. - New configuration behavior: write a real configuration file, start the
tracker, and exercise the affected service with a real client (for example
tracker_clientorcurl) instead of asserting on internal state. - Error path: trigger the failure as a user would (wrong path, bad file, missing permission) and judge whether the message tells the user what to fix.
Each example produces evidence that is a recording of what really happened: the exact commands, their output, and the relevant tracker log lines.
For the operational requirements, see root integration-test guidance and the issue template's manual verification and disposable script sections.
The External Link Check workflow runs Lychee online every Monday at 06:00 UTC and can be started manually from GitHub Actions. It is advisory: it never runs on pushes or pull requests and is not a merge requirement, but Lychee failures fail the workflow. It retains a Markdown report for 14 days regardless of the workflow outcome.
When it fails, download the report and rerun the workflow once to distinguish a transient network
or rate-limit failure from a persistent broken link. Fix persistent URLs. An intentional exception
must be a narrowly scoped, documented exclusion in .github/lychee-online.toml; do not weaken
the offline local-link policy in lychee.toml.
Tracked repository test code is Rust. The operational policy and its narrowly defined Python exception are in the root integration-test guidance.
The unit-test skill is the source of truth for Test Desiderata, behavior-focused naming, visible Arrange-Act-Assert structure, deterministic clocks, isolation, and lifecycle fixture design.
When a correct test does not clearly communicate the behavior it protects, consult the test refactoring-pattern catalog. The catalog contains reviewed repository-native patterns for improving test maintainability, readability, and expressiveness without changing the behavior under test. Use a pattern only where its stated constraints apply; keep the production Act and observable assertions visible.
Use test helpers for shared mock servers and
test data. For root integration tests, follow the port isolation, fixture shutdown,
and scenario constraints in tests/AGENTS.md.
Tests establish correctness claims. Benchmarks measure performance under defined workloads, while profiling identifies where a workload spends time or memory. These tools can reveal regressions and guide optimization, but they do not replace correctness tests or the repository quality gates.