Skip to content

perf(api): speed up API tests by lazy IM imports and smaller contract docs - #45381

Merged
hyoban merged 2 commits into
mainfrom
perf/lazy-im-adapter-imports
Nov 3, 2026
Merged

hyoban merged 2 commits into
mainfrom
perf/lazy-im-adapter-imports

Conversation

@hyoban

@hyoban hyoban commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Summary

API unit and controller test jobs have gotten much slower since early October (merge-queue medians: unit shard ~4m → ~8m, controllers ~3m → ~10m50). This PR fixes the two largest causes that aren't just "more tests".

1. IM provider SDKs imported by create_app() (from #44864)

create_app() reaches core/human_input_v2/im_integration/adapters/factory.py, which imported every IM provider adapter and its vendor SDK at module load. lark_oapi alone costs ~3.5s, so each xdist worker and every subprocess-based test paid it, and existing tests that happened to run first in a worker jumped from ~0.05s to 3–6s.

Each adapter import now lives in its match branch, matching the pattern already used by _card_decoder in services/human_input_v2/im_webhook_runtime.py.

  • create_app() no longer loads lark_oapi, slack_sdk, botbuilder, botframework, alibabacloud_dingtalk, or wechatpy.
  • python -c "from app_factory import create_app; create_app()": ~11.6s → ~7.9s.

2. Error contract check copies every surface model into each test API (from #44001)

The autouse error_response_contract fixture builds an OpenAPI document for each ExternalApi that serves an error response. Test-built APIs borrowed models by copying all ~2200 surface models into api.models, so a one-path document carried every schema and took ~90ms in finalize_openapi_payload. About 1200 test APIs per run made that ~107s of CPU, about 18% of the controller suite locally.

Models are now borrowed through a fallback mapping. Name lookups (the only way Swagger resolves models) fall back to the surfaces, but iteration (the way document finalization merges models) yields only the API's own models. The swap is scoped to the build, so api.models itself is no longer mutated.

  • For all 1200 test APIs in the controller suite, the new documents have identical paths, every $ref resolves, and every referenced component is identical to the old document.
  • Document building time: 108s → 13s of CPU.
  • Controller suite locally (-n 8, two runs each): 74s / 98s → 62s / 79s.
  • Added test_documents_only_the_surface_models_a_test_api_references; it fails on the old helper.

Checklist

  • I've verified the change and added or updated tests where meaningful regression risk justifies coverage.
  • I ran make lint && make type-check (backend) and vp staged (frontend) to appease the lint gods

From Claude Code

create_app() reaches the adapter factory, which imported every provider
adapter and its vendor SDK at module load. lark_oapi alone takes ~3.5s,
so every app start (each xdist worker and subprocess test) paid ~3.7s.
@github-actions github-actions Bot added the size:S This PR changes 10-29 lines, ignoring generated files. label Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Pyrefly Type Coverage

Metric Base PR Delta
Type coverage 83.01% 83.01% -0.00%
Strict coverage 82.66% 82.66% -0.00%
Typed symbols 85,604 85,605 +1
Untyped symbols 17,597 17,598 +1
Modules 4373 4373 0

@codecov

codecov Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.80%. Comparing base (ac0f594) to head (c7c404d).
⚠️ Report is 8 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff             @@
##             main   #45381       +/-   ##
===========================================
+ Coverage   68.81%   89.80%   +20.98%     
===========================================
  Files        6103     6239      +136     
  Lines      368940   390949    +22009     
  Branches    77825    79291     +1466     
===========================================
+ Hits       253897   351075    +97178     
+ Misses     112233    33774    -78459     
- Partials     2810     6100     +3290     
Flag Coverage Δ
api 89.64% <100.00%> (+37.30%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

The controller error contract check builds an OpenAPI document for each
ExternalApi that serves an error. Test APIs borrowed models by copying all
~2200 surface models into api.models, so a one-path document carried every
schema and took ~90ms; with ~1200 test APIs per run that was ~18% of the
controller suite.

Borrow through a fallback mapping that resolves names on lookup but
iterates only the API's own models, so a document carries just the
borrowed models its operations reference. Documents for all 1200 test APIs
keep identical paths and referenced components.
@github-actions github-actions Bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Nov 3, 2026
@hyoban hyoban changed the title perf(api): import IM provider adapters lazily in the factory perf(api): speed up API tests by lazy IM imports and smaller contract docs Nov 3, 2026
@hyoban
hyoban marked this pull request as ready for review November 3, 2026 04:20
@hyoban
hyoban requested a review from QuantumGhost as a code owner November 3, 2026 04:20
@hyoban
hyoban added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit 24b4275 Nov 3, 2026
58 checks passed
@hyoban
hyoban deleted the perf/lazy-im-adapter-imports branch November 3, 2026 04:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants