Skip to content

test(api): cut per-test overhead and fixed waits in unit tests - #45388

Merged
hyoban merged 4 commits into
mainfrom
perf/unit-test-fixture-overhead
Nov 3, 2026
Merged

hyoban merged 4 commits into
mainfrom
perf/unit-test-fixture-overhead

Conversation

@hyoban

@hyoban hyoban commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Summary

Follow-up to #45381. In CI even the fastest 10% of API unit tests take ≥18ms, and about two thirds of unit test time is spent in tests under 0.1s. Most of that is autouse setup that every test pays before it runs:

Autouse fixture Local cost per test What it did
_sqlite_engine ~0.8ms Copied the 3.5MB full-schema SQLite template for every test, connected or not
reset_redis_mock ~1.0ms Recursively reset ~370 child mocks of the shared redis_mock

Changes:

  • Lazy SQLite copy: _sqlite_engine copies the template on the engine's first connection (do_connect), guarded by a lock so the copy happens at most once. sqlite_engine still copies up front, because several tests hand sqlite_engine.url to Flask-SQLAlchemy, which opens the file with its own engine.
  • Skip the empty Redis reset: redis_mock.reset_mock() runs only when redis_mock.mock_calls is non-empty. mock_calls records calls on every attached child and return value, so an empty list means the recursive reset has nothing to clear. The explicit per-method resets and return_value assignments still run on every test.
  • No fixed 0.5s waits in teardown: seven local test HTTP servers ran serve_forever() with the default 0.5s poll interval, so each shutdown() waited up to half a second. They now poll every 10ms, as tests/helpers/unit/model_provider_fixtures.py already did. In CI, 82 tests sat at ~0.5s for ~44s in total.
  • One source scan in test_enums: two tests each walked the whole api/ tree (including .venv) and parsed every production module. The walk now prunes tests and .venv, and both tests share one cached parse.
  • One app per controller test module: five human input and data source controller test modules (test_contact, test_human_input_form_compatibility, test_human_input_v2_form_read, test_data_source, test_human_input_channel_wire) built a fresh app per test and registered the whole console blueprint, console namespace, or service API namespace on it. That took a few hundred milliseconds per test, and each fresh ExternalApi also made the error contract check rebuild its OpenAPI document. Each module now builds its app once; per-test state (login user, scenario, SQLite engine binding, application services) stays in function fixtures via monkeypatch.setitem, so it is undone after each test. No test in these modules mutates the app itself.
  • Freeze collection-time objects out of GC: collection imports the whole application, so every automatic full collection during the run rescanned those long-lived objects and stalled whichever test triggered it (a cluster of slack tests sat at exactly 0.25s). tests/conftest.py now runs gc.collect() and gc.freeze() in pytest_collection_finish, which runs in every xdist worker. Objects created by tests are still collected normally.
  • Shared swagger exports: 11 read-only tests in test_generate_swagger_specs.py share one module-scoped generate_specs output instead of each exporting every surface (~4s each in CI). Tests of generation itself (idempotency, live payload match, subprocess exports) still generate on their own.

Local measurements (-n 8 unless noted, no coverage):

  • 2000 no-op tests (-p no:xdist): 11.2s → 5.5s, about 4.3ms → 1.4ms of overhead per test.
  • tests/unit_tests/services (10.4k tests): 80–89s → 69–78s.
  • The six files with local HTTP servers (-p no:xdist): 55.0s → 11.7s.
  • tests/unit_tests/models/test_enums.py: 6.6s → 4.1s.
  • The five controller modules (-p no:xdist, 288 tests): 50.1s → 10.3s.
  • Controller suite (9071 tests, -n 8): 59–60s → 50s.
  • gc.freeze: human_input_v2 unit tests (-p no:xdist) 59s → 54–55s; tests/unit_tests/services (-n 8) 75–78s → 69–74s.

CI results

Baseline: the five merge-queue runs on main right after #45381 (median job time). Each commit of this PR ran once. Job times vary by about ±30s between runners, so each change is also measured by the junit time of the tests it touches (summed across workers), which is far less noisy.

API Controller job API Unit jobs (shard 1 / 2)
main baseline (median) 405s 448s
+ per-test fixtures, shared swagger exports 367s 419s / 397s
+ no fixed waits, single enum scan 383s 427s / 429s
+ one app per controller module 356s 426s / 394s
+ gc.freeze (this PR) 318s (−21%) 398s / 404s (−10%)

Per change, junit time of the affected tests:

Change Tests measured Baseline After
One app per controller module the five controller modules 152–166s 33–38s
No fixed 0.5s waits modules with local HTTP servers 61–63s 12–13s
Shared swagger exports test_generate_swagger_specs 78–79s 62–66s
gc.freeze whole controller suite 469s (previous commit) 420s
gc.freeze whole unit suite 1049s (previous commit) 1027s
Lazy SQLite copy, skipped Redis reset unit tests outside the groups above 1037–1044s 926–1035s (within runner variance)
Single enum source scan test_enums 11–12s 8–11s (negligible)

Collection (~55–60s per unit shard, ~25s for controllers) is unchanged; it is dominated by importing the application in every xdist worker.

Checklist

  • I've verified the change and added or updated tests where meaningful regression risk justifies coverage.
  • I ran make lint && make type-check (backend) and vp staged (frontend) to appease the lint gods

From Claude Code

Every unit test paid ~3ms of autouse setup before running: a copy of the
3.5MB full-schema SQLite file and a recursive reset of ~370 Redis child
mocks, even when it touched neither.

- Copy the SQLite template on the engine's first connection. sqlite_engine
  still copies up front for tests that hand its URL to another engine.
- Skip the recursive Redis mock reset when mock_calls is empty; it records
  calls on every attached child and return value.
- Share one generated spec set across the read-only swagger spec tests
  instead of exporting every surface eleven times.
@github-actions github-actions Bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Pyrefly Type Coverage

Metric Base PR Delta
Type coverage 83.07% 83.08% +0.00%
Strict coverage 82.73% 82.73% +0.00%
Typed symbols 85,682 85,697 +15
Untyped symbols 17,529 17,528 -1
Modules 4373 4373 0

@codecov

codecov Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.79%. Comparing base (24b4275) to head (b481a38).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main   #45388   +/-   ##
=======================================
  Coverage   89.79%   89.79%           
=======================================
  Files        6239     6239           
  Lines      390966   390966           
  Branches    79305    79305           
=======================================
+ Hits       351075   351086   +11     
+ Misses      33791    33782    -9     
+ Partials     6100     6098    -2     
Flag Coverage Δ
api 89.63% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

- Local test HTTP servers ran serve_forever with the default 0.5s poll
  interval, so every shutdown() in teardown waited up to half a second.
  Poll every 10ms, as model_provider_fixtures already does.
- The two EndUserType source scans each walked the whole api tree,
  including .venv, and parsed every production module. Prune tests and
  .venv during the walk and share one cached parse.
@hyoban hyoban changed the title test(api): cut per-test fixture overhead in unit tests test(api): cut per-test overhead and fixed waits in unit tests Nov 3, 2026
Five human input and data source controller test modules built a fresh
app for every test and registered the whole console blueprint, console
namespace, or service API namespace on it. Registration took a few
hundred milliseconds per test, and each fresh ExternalApi also made the
error contract check rebuild its OpenAPI document.

Build the app once per module and keep per-test state (login user,
scenario, SQLite engine binding, application services) in function
fixtures through monkeypatch.setitem, so it is undone after each test.
Collection imports the whole application, so every automatic full GC
during the run rescanned those long-lived objects and stalled whichever
test triggered it (a cluster of slack tests sat at exactly 0.25s).
Collect once and gc.freeze() after collection; objects created by tests
are still collected normally.
@hyoban
hyoban marked this pull request as ready for review November 3, 2026 06:59
@hyoban
hyoban added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit 193fdfa Nov 3, 2026
60 checks passed
@hyoban
hyoban deleted the perf/unit-test-fixture-overhead branch November 3, 2026 07:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants