[JunyiXu-nv]

Stacked on #21447 ([TRTLLM-17262][refactor] Shared middleware package for the aggregated and disaggregated servers) and #21448 ([TRTLLM-17261][refactor] Add the trtllm-serve router skeleton and move /health, /version and /v1/models into ops.system): the first commits below belong to those two PRs (plus one merge commit that joins them, since they are independent of each other) and disappear from this diff when they merge; this PR's own change is the last 2 commits.

@coderabbitai summary

Description

Both trtllm-serve servers build their FastAPI app and run uvicorn with near-identical code that differs in a handful of deliberate details. This PR extracts that assembly into two shared functions in tensorrt_llm/serve/_base/app_builder.py instead of a shared base class, and makes both servers use them:

• build_http_app(*, lifespan, routers, dialect_error_renderers, middleware, route_class=None) assembles an app in one fixed order: FastAPI(lifespan=...), the route class (before any route exists), the exception handlers, each router callable in order, and the built-in middleware last so it stays outermost (add_middleware inserts at the front of the stack).
• serve_uvicorn(app, *, host, port, sockets=None, log_level, timeout_keep_alive, after_start=None) runs it through the telemetry-aware uvicorn server (serve/_telemetry.py); after_start, when given, runs as a task next to the server and receives it.

OpenAIServer.__init__ and OpenAIDisaggServer.__init__ now call build_http_app; their __call__ methods call serve_uvicorn. The validation handlers keep their bodies and are registered with app.add_exception_handler(RequestValidationError, handler) instead of the decorator. The aggregated server's role dispatch moves unchanged into _attach_role_routes(app); its post-start closure becomes the method _register_after_serving(server) with the same body and the same startup-timer names. The disaggregated server gets _attach_routes(app), which calls register_routes(). Both middleware installers are callables that read the server's perf-metrics attributes when they run, after the routes, as the aggregated server did before.

Part of the trtllm-serve refactor (epic TRTLLM-17056, task TRTLLM-17269: cross-cutting pieces and the Responses package). Builds on #21447 (the shared middleware registration both servers call) and #21448 (serve/_base/, attach_routers, the system endpoints). The per-dialect error renderers land next in the dialect_error_renderers slot, and the eventual composition root calls the same two functions.

Behavior

No behavior change.

Every difference between the two apps stays an explicit argument at the call site:

Concern	Aggregated server (openai_server.py)	Disaggregated frontend (openai_disagg_server.py)
route class	route_class=_MsgspecRoute (every add_api_route route decodes msgpack)	none: plain APIRoute
middleware arguments	register_middlewares(..., adjusted_clock=...) for both middlewares	register_middlewares(...) without a clock
middleware relative to routes	after the routes (after the Prometheus instrumentator's middleware)	after the routes (was before; nothing in the disaggregated route registration adds middleware, so the stack is identical, see the middleware-order snapshot)
Prometheus	mount_metrics() inside register_routes, unchanged	app.mount("/prometheus/metrics", ...) inside register_routes, unchanged
validation handler	aggregated body, unchanged	disaggregated body, unchanged
uvicorn log_level	"info"	logger.level, passed through unchanged
keep-alive	TIMEOUT_KEEP_ALIVE (5 s)	config.server_keep_alive_timeout
post-start task	after_start=self._register_after_serving (worker registration with the startup timers)	none
user --middleware	applied by the CLI after construction (unchanged, outermost)	not applied (unchanged)

Snapshots, unchanged for every configuration (FastAPI 0.136.3 and 0.139.2):

• route snapshot (tests/unittest/api_stability/test_serve_routes.py, trtllm_serve_routes.yaml): llm, llm_context, llm_generation, llm_gpt_oss, llm_gpt_oss_harmony_disabled, llm_governor, llm_rl_control, llm_runtime_control, llm_rl_and_runtime_control, llm_perf_metrics, llm_perf_metrics_enable_metrics_env, llm_perf_metrics_output_dir_only, llm_render_endpoints, llm_multi_frontend_attached, mm_encoder, mm_encoder_rl_control, embedding, embedding_runtime_control, disagg_local, disagg_local_perf_metrics, disagg_local_http_cluster_storage, disagg_fleet, disagg_fleet_perf_metrics, disagg_fleet_http_cluster_storage, coordinator, coordinator_http_cluster_storage, visual_gen;
• middleware-order snapshot (tests/unittest/serve/test_middleware_order.py): llm, mm_encoder, embedding, llm_perf_metrics, llm_perf_metrics_output_dir_only, llm_perf_metrics_enable_metrics_env (with the Prometheus instrumentator middleware), disagg_local, disagg_fleet, disagg_local_perf_metrics, disagg_fleet_perf_metrics, coordinator.

Beyond the snapshots, a fingerprint of every assembled app across those 27 configurations (route class, endpoint identity, response class, middleware classes and keyword identities against the server's own attributes, exception-handler identities, lifespan identity, app.state keys, router route class) is identical before and after this change, and so is everything each __call__ hands to uvicorn.

The surface that NeMo-RL's subclass of OpenAIDisaggServer relies on is unchanged (constructor shape, self.app with every route registered when __init__ returns, __call__(host, port, sockets=None), _wrap_entry_point, _handle_exception, _perf_metrics_collector): the subclassing contract test of #21442, tests/unittest/disaggregated/test_disagg_server_contract.py, passes against this branch (6/6) exactly as at the base commit. OpenAIServer's constructor and .app are unchanged.

Not included

• The coordinator app (serve/coordinator_server.py) and the standalone render app (serve/render/): the coordinator's dual-listener mode and the renderer's blocking uvicorn.run do not share the aggregated/disaggregated shape, the coordinator file later moves unchanged with the other disaggregated-serving modules, and neither has middleware or validation handlers to share. Both are follow-up candidates.
• The contents of the error renderers (next PR), moving route registration into routers, and the --middleware CLI handling.
• The unreferenced asyncio.create_task of the post-start task is kept as it was.

Coordination with other work

• tests/unittest/disaggregated/test_openai_disagg_server.py patched uvicorn.Config/uvicorn.Server through tensorrt_llm.serve.openai_disagg_server.uvicorn. That module no longer calls uvicorn, so its now-unused import uvicorn goes (autoflake would strip it anyway) and the test patches tensorrt_llm.serve._base.app_builder.uvicorn, where serve_uvicorn looks the names up. It also now asserts the disaggregated log level.
• [None][fix] serve: make multi-frontend metrics, profiling and liveness launcher-owned #19394 (multi-frontend metrics, profiling and liveness launcher-owned) merged before this branch: its lifespan changes are in the closure passed to build_http_app unchanged. [None][fix] serve: answer a request that fails validation with a readable error #20865 (readable validation errors) is open: the handler stays a closure here and is passed through dialect_error_renderers; a handler that becomes a method is passed the same way. [None][feat] Support disagg serving for post training #19890 (disaggregated serving for post-training, draft) edits openai_disagg_server.py too.
• On 2026-11-03, 72 open PRs touch openai_server.py, openai_disagg_server.py or _telemetry.py; the construction block of both __init__ methods and both __call__ methods change shape here, so whichever lands second rebases its edits onto the new shape (handler registration via dialect_error_renderers, routes via _attach_role_routes / _attach_routes, uvicorn arguments via serve_uvicorn).

CI bookkeeping

• Multi-GPU table: none (tensorrt_llm/serve/_base/ is listed by [TRTLLM-17261][refactor] Add the trtllm-serve router skeleton and move /health, /version and /v1/models into ops.system #21448; both server files are already listed).
• CODEOWNERS: none.
• legacy-files.txt: none (the new module is new typed code, formatted by ruff).
• mypy: tensorrt_llm/serve/_base/app_builder.py added to [tool.mypy] files in pyproject.toml.
• Test lists: none; tests/unittest/serve/_base/test_app_builder.py and tests/unittest/serve/test_server_uvicorn_settings.py are collected by the existing unittest/serve entry of l0_cpu.yml (CPU-Generic stages).

For reviewers: git blame -C -C tensorrt_llm/serve/openai_server.py attributes the relocated handler and post-start bodies to their original commits.

Test Coverage

• New tests/unittest/serve/_base/test_app_builder.py -> 8 passed: route class applies to routes added by routers and only when given; assembly order; built-in middleware outermost after router-added middleware; lifespan and exception-handler pass-through; serve_uvicorn passes host/port/log level/keep-alive/sockets through and schedules after_start with the server; integer log level unchanged; no task when after_start is None; the module imports only FastAPI, Starlette, uvicorn, the standard library and serve/_telemetry.py.
• New tests/unittest/serve/test_server_uvicorn_settings.py -> 2 passed: both servers built through the test factory; the aggregated server hands uvicorn log_level="info", timeout_keep_alive=TIMEOUT_KEEP_ALIVE, the sockets, and schedules _register_after_serving; the disaggregated frontend hands it logger.level, the configured keep-alive, and schedules nothing.
• pytest -m cpu_only tests/unittest/serve tests/unittest/api_stability/test_serve_routes.py -> 164 passed (route snapshot, middleware-order snapshot, layering contracts, ratchet) on FastAPI 0.139.2; the route snapshot, middleware-order, uvicorn-settings, routing, system-endpoint, layering and disaggregated-server tests -> 131 passed on both FastAPI 0.136.3 and 0.139.2.
• pytest -m cpu_only tests/unittest/disaggregated -> 1411 passed, 1 skipped, 2 xfailed.
• NeMo-RL subclassing contract test from [TRTLLM-17262][refactor] Split the disaggregated server's entry-point wrapper into prepare, run, finish and serialize steps #21442 (tests/unittest/disaggregated/test_disagg_server_contract.py), run against this branch and against the base commit -> 6 passed each.
• All serve CPU tests, one pytest process per l0_cpu.yml entry (LLM_MODELS_ROOT=... PYTHONPATH=tests python3 -m test_common.serve.cpu_tests): 66 entries, exit 0, no failures on this branch (unittest/serve: 128 passed, 8 of them new; this run predates the review fixes, which added test_server_uvicorn_settings.py, one assertion, one docstring word and the lambda form of the aggregated middleware installer, all covered by the suites above). The same run at the base commit (before this change) had 11 load-induced timeouts while three other test runs shared the machine (8 in test_cluster_storage.py::TestHttpClusterStorage setup, 2 in test_frontend_lifecycle.py, 1 in test_metrics.py::test_prometheus_contract[multiprocess]); every other entry's count is identical between the two runs. Re-run alone and sequentially: on the branch all three pass (test_cluster_storage.py 53 passed, test_frontend_lifecycle.py 14 passed, test_metrics.py 7 passed); at the base commit the first two still failed the same way, with the cause visible in the output: TestHttpClusterStorage binds the fixed port 18000 and got [Errno 98] address already in use from another concurrent test run on the shared machine, and test_frontend_group_cleanup is a 90 s gRPC-readiness wait that ran at a load average above 12. Neither test touches the code this PR changes.
• pytest tests/unittest/api_stability -> 98 passed.
• bash scripts/run_mypy.sh (strict, repository config) -> Success: no issues found in 21 source files.
• pre-commit run --files <changed files> twice -> clean on the second run; python scripts/legacy_utils.py check-configs -> up to date.
• git grep -n "uvicorn.Config\|FastAPI(lifespan" -- tensorrt_llm/serve/openai_server.py tensorrt_llm/serve/openai_disagg_server.py -> nothing.
• Property check (local, not in CI): over generated assemblies (0-4 routers each adding 0-3 routes and 0-3 middlewares, 0-3 built-in middlewares, either route class) and generated uvicorn arguments, the assembly order, route class, middleware stack, lifespan, exception handler and every uvicorn argument hold on 200 generated inputs each; 13 planted bugs (dropped lifespan, ignored or wrong route class, reversed or re-apped routers, middleware before routers, wiped stack, dropped sockets, hard-coded keep-alive or log level, never or inverted after_start, server not from create_uvicorn_server) were each detected.
• GPU, left to CI: the single-GPU disaggregated end-to-end block in l0_a10.yml (msgpack requests through the route class) and test_e2e.py::test_openai_misc_example[pytorch].

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

🤖 Generated with Claude Code