Repository navigation
test(api): cut per-test overhead and fixed waits in unit tests - #45388
Merged
Merged
Conversation
Every unit test paid ~3ms of autouse setup before running: a copy of the 3.5MB full-schema SQLite file and a recursive reset of ~370 Redis child mocks, even when it touched neither. - Copy the SQLite template on the engine's first connection. sqlite_engine still copies up front for tests that hand its URL to another engine. - Skip the recursive Redis mock reset when mock_calls is empty; it records calls on every attached child and return value. - Share one generated spec set across the read-only swagger spec tests instead of exporting every surface eleven times.
Contributor
Pyrefly Type Coverage
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #45388 +/- ##
=======================================
Coverage 89.79% 89.79%
=======================================
Files 6239 6239
Lines 390966 390966
Branches 79305 79305
=======================================
+ Hits 351075 351086 +11
+ Misses 33791 33782 -9
+ Partials 6100 6098 -2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
- Local test HTTP servers ran serve_forever with the default 0.5s poll interval, so every shutdown() in teardown waited up to half a second. Poll every 10ms, as model_provider_fixtures already does. - The two EndUserType source scans each walked the whole api tree, including .venv, and parsed every production module. Prune tests and .venv during the walk and share one cached parse.
Five human input and data source controller test modules built a fresh app for every test and registered the whole console blueprint, console namespace, or service API namespace on it. Registration took a few hundred milliseconds per test, and each fresh ExternalApi also made the error contract check rebuild its OpenAPI document. Build the app once per module and keep per-test state (login user, scenario, SQLite engine binding, application services) in function fixtures through monkeypatch.setitem, so it is undone after each test.
Collection imports the whole application, so every automatic full GC during the run rescanned those long-lived objects and stalled whichever test triggered it (a cluster of slack tests sat at exactly 0.25s). Collect once and gc.freeze() after collection; objects created by tests are still collected normally.
hyoban
marked this pull request as ready for review
November 3, 2026 06:59
lyzno1
approved these changes
Nov 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #45381. In CI even the fastest 10% of API unit tests take ≥18ms, and about two thirds of unit test time is spent in tests under 0.1s. Most of that is autouse setup that every test pays before it runs:
_sqlite_enginereset_redis_mockredis_mockChanges:
_sqlite_enginecopies the template on the engine's first connection (do_connect), guarded by a lock so the copy happens at most once.sqlite_enginestill copies up front, because several tests handsqlite_engine.urlto Flask-SQLAlchemy, which opens the file with its own engine.redis_mock.reset_mock()runs only whenredis_mock.mock_callsis non-empty.mock_callsrecords calls on every attached child and return value, so an empty list means the recursive reset has nothing to clear. The explicit per-method resets andreturn_valueassignments still run on every test.serve_forever()with the default 0.5s poll interval, so eachshutdown()waited up to half a second. They now poll every 10ms, astests/helpers/unit/model_provider_fixtures.pyalready did. In CI, 82 tests sat at ~0.5s for ~44s in total.test_enums: two tests each walked the wholeapi/tree (including.venv) and parsed every production module. The walk now prunestestsand.venv, and both tests share one cached parse.test_contact,test_human_input_form_compatibility,test_human_input_v2_form_read,test_data_source,test_human_input_channel_wire) built a fresh app per test and registered the whole console blueprint, console namespace, or service API namespace on it. That took a few hundred milliseconds per test, and each freshExternalApialso made the error contract check rebuild its OpenAPI document. Each module now builds its app once; per-test state (login user, scenario, SQLite engine binding, application services) stays in function fixtures viamonkeypatch.setitem, so it is undone after each test. No test in these modules mutates the app itself.tests/conftest.pynow runsgc.collect()andgc.freeze()inpytest_collection_finish, which runs in every xdist worker. Objects created by tests are still collected normally.test_generate_swagger_specs.pyshare one module-scopedgenerate_specsoutput instead of each exporting every surface (~4s each in CI). Tests of generation itself (idempotency, live payload match, subprocess exports) still generate on their own.Local measurements (
-n 8unless noted, no coverage):-p no:xdist): 11.2s → 5.5s, about 4.3ms → 1.4ms of overhead per test.tests/unit_tests/services(10.4k tests): 80–89s → 69–78s.-p no:xdist): 55.0s → 11.7s.tests/unit_tests/models/test_enums.py: 6.6s → 4.1s.-p no:xdist, 288 tests): 50.1s → 10.3s.-n 8): 59–60s → 50s.gc.freeze: human_input_v2 unit tests (-p no:xdist) 59s → 54–55s;tests/unit_tests/services(-n 8) 75–78s → 69–74s.CI results
Baseline: the five merge-queue runs on
mainright after #45381 (median job time). Each commit of this PR ran once. Job times vary by about ±30s between runners, so each change is also measured by the junit time of the tests it touches (summed across workers), which is far less noisy.mainbaseline (median)gc.freeze(this PR)Per change, junit time of the affected tests:
test_generate_swagger_specsgc.freezegc.freezetest_enumsCollection (~55–60s per unit shard, ~25s for controllers) is unchanged; it is dominated by importing the application in every xdist worker.
Checklist
make lint && make type-check(backend) andvp staged(frontend) to appease the lint godsFrom Claude Code