[JunyiXu-nv]

Stacked on #21448 ([TRTLLM-17261][refactor] Add the trtllm-serve router skeleton and move /health, /version and /v1/models into ops.system), which is itself stacked on #21406 ([TRTLLM-17261][test] Add route, middleware-order and shim-identity snapshots and a CPU test seam for trtllm-serve): the first commits below belong to those PRs and disappear from this diff when they merge; this PR's own change is the last 3 commits.

@coderabbitai summary

Description

tensorrt_llm/serve/responses_utils.py is the Responses API implementation, but four disaggregated-serving modules imported it only for three small service types, a latency logger and a re-exported clock function, and so loaded the tool parsers, web_search and the harmony adapter although the disaggregated frontend does not serve /v1/responses. This PR moves ResponseHooks, done_generator and UCompletionResponseOrGenerator into tensorrt_llm/serve/_base/service_types.py and PeriodicLatencyLogger into tensorrt_llm/serve/_base/latency_logger.py, generated as a verbatim move (check below), rewrites every importer in the repository, and makes openai_client.py and openai_disagg_server.py import get_steady_clock_now_in_seconds from tensorrt_llm._utils, where it is defined (the re-export in responses_utils.py stays). The disaggregated server also imports ResponseHooks from the module that defines it instead of through openai_disagg_service.

Part of the trtllm-serve refactor (epic TRTLLM-17056, task TRTLLM-17262: the small splits that the Responses, request-trace and disaggregated-entry-point changes from the large-scale-serving branches land into when they go upstream). Builds on #21448 (tensorrt_llm/serve/_base/) and #21406 (the layering-contract test). It must merge before the shared Responses stream code moves to tensorrt_llm/serve/_base/responses_stream.py and before the Responses endpoint package move, so that those land next to these types instead of into responses_utils.py.

The old module keeps no re-export: no code outside the repository imports these four names from responses_utils (GitHub code search; every hit is a copy of this repository), and every importer in the repository now imports the new modules: openai_client, openai_disagg_service, openai_service, the function-level imports in router and disagg_coordinator, and the two tests under tests/unittest/disaggregated/. An open PR that imports one of them from responses_utils gets an ImportError after this merges and re-applies its import on the new path; a function-level import does so when the function first runs.

Two guards pin the result (tests/unittest/serve/): a layering contract forbids tensorrt_llm.serve.responses_utils as an import of the disaggregated client, service, coordinator and router, at module level and inside functions; and test_disagg_imports.py imports each of them in a fresh interpreter and checks that the Responses implementation is not loaded at all, with a positive control. Both carry one allowance for responses_done_generator, which openai_disagg_service still takes from responses_utils since #19883 (Responses on disaggregated serving); it leaves with that helper when the shared Responses stream code moves to serve/_base/responses_stream.py, and the allowance only shrinks (the fresh-interpreter case is a strict expected failure until then).

Static import graph (module-level imports; "direct" names what is still imported from responses_utils):

Module	Before	After this PR	After this PR and #21447 (Shared middleware package)
serve.openai_disagg_server	direct	direct: ServerArrivalTimeMiddleware, guard_responses_stream, stream_error_event	direct: guard_responses_stream, stream_error_event (from #19883; they leave with the Responses stream helpers)
serve.openai_client	direct	no	no
serve.openai_service	direct	no	no
serve.openai_disagg_service	direct	direct: responses_done_generator (from #19883)	same
serve.disagg_coordinator	through openai_client	no	no
serve.router	no (function-level import only)	no	no
serve._base.service_types, serve._base.latency_logger	—	no	no

Coordination with open PRs: this PR moves ResponseHooks, done_generator and UCompletionResponseOrGenerator out of tensorrt_llm/serve/responses_utils.py into tensorrt_llm.serve._base.service_types, and PeriodicLatencyLogger into tensorrt_llm.serve._base.latency_logger, with no re-export at the old path. Open PRs that import any of them from responses_utils must retarget their import to the new module: #21442 (Split the disaggregated server's entry-point wrapper; adds UCompletionResponseOrGenerator to the responses_utils import of openai_disagg_server.py), #21447 (Shared middleware package; edits the same import block of openai_disagg_server.py, tests/unittest/disaggregated/test_disagg_openai_client.py, tests/unittest/serve/test_layering.py and legacy-files.txt) and #21377 (Muse Spark Open modeling_v2 TEP target; adds a function-level PeriodicLatencyLogger import from responses_utils in router.py, which would fail when that code first runs). Whichever of each pair lands second re-applies its import edits on the new path. With #21447 the disaggregated server also stops importing ServerArrivalTimeMiddleware from responses_utils; whichever of this PR and #21447 lands second adds openai_disagg_server to the two guards once its last responses_utils import is gone. The same applies to #21060, #21388 and #21452, which rewrite that import block, and to the draft #16402, which adds ResponseHooks and UCompletionResponseOrGenerator imports.

Behavior

No behavior change. Every moved definition is byte-identical (check below); only import statements change. The function-level imports of PeriodicLatencyLogger in router.py and disagg_coordinator.py stay inside their functions, so nothing runs at a different time. openai_client still binds get_steady_clock_now_in_seconds as a module-level name, so the tests that patch tensorrt_llm.serve.openai_client.get_steady_clock_now_in_seconds keep intercepting it (verified with the real clock temporarily raising: the three patching tests pass, 111 unpatched callers fail).

Moves

Pure-move check

## Move spec

```yaml
moves:
- from: tensorrt_llm/serve/responses_utils.py
  to: tensorrt_llm/serve/_base/service_types.py
  group: B
  symbols:
  - ResponseHooks
  - done_generator
  - UCompletionResponseOrGenerator
- from: tensorrt_llm/serve/responses_utils.py
  to: tensorrt_llm/serve/_base/latency_logger.py
  group: B
  symbols:
  - PeriodicLatencyLogger
rewrite_internal_imports: true
```

Pure-move check: base 50752e8947db, head a026344effad, tools ddda1b048441

| Symbol | From | To | Result |
|---|---|---|---|
| `PeriodicLatencyLogger` | tensorrt_llm/serve/responses_utils.py | tensorrt_llm/serve/_base/latency_logger.py | verbatim |
| `ResponseHooks` | tensorrt_llm/serve/responses_utils.py | tensorrt_llm/serve/_base/service_types.py | verbatim |
| `UCompletionResponseOrGenerator` | tensorrt_llm/serve/responses_utils.py | tensorrt_llm/serve/_base/service_types.py | verbatim |
| `done_generator` | tensorrt_llm/serve/responses_utils.py | tensorrt_llm/serve/_base/service_types.py | verbatim |

4 definitions moved verbatim; 0 differences; shim exports 0 names

Moves without a shim, "old path → new path" for the release notes (no code outside the repository imports these names from the old module):

tensorrt_llm.serve.responses_utils.ResponseHooks → tensorrt_llm.serve._base.service_types
tensorrt_llm.serve.responses_utils.done_generator → tensorrt_llm.serve._base.service_types
tensorrt_llm.serve.responses_utils.UCompletionResponseOrGenerator → tensorrt_llm.serve._base.service_types
tensorrt_llm.serve.responses_utils.PeriodicLatencyLogger → tensorrt_llm.serve._base.latency_logger

No kept shim and no shim identity rows. The get_steady_clock_now_in_seconds re-export in responses_utils.py is unchanged (not a moved name). History: git log -C --follow tensorrt_llm/serve/_base/service_types.py and git blame -C -C tensorrt_llm/serve/_base/latency_logger.py.

CI bookkeeping

• Multi-GPU table: none (the existing tensorrt_llm/serve/_base/ entry from [TRTLLM-17261][refactor] Add the trtllm-serve router skeleton and move /health, /version and /v1/models into ops.system #21448 covers both new modules; the edited importers are already listed)
• CODEOWNERS: none (/tensorrt_llm/serve covers the new files; /tests/unittest covers the new test until the serve test-directory rules land)
• legacy-files.txt: tensorrt_llm/serve/_base/latency_logger.py and tensorrt_llm/serve/_base/service_types.py added (both Group B: their code comes from the Group B file responses_utils.py); derived configs regenerated with scripts/legacy_utils.py gen-configs (ruff-legacy.toml, the managed blocks of pyproject.toml and .pre-commit-config.yaml); ruff-legacy-baseline.json: none (the moved code carries no recorded violation at the new paths; responses_utils.py only improves). The stale entries gen-configs warns about predate this PR and are left alone.
• Test lists: none needed; tests/unittest/serve/test_disagg_imports.py is collected through the existing unittest/serve entry of tests/integration/test_lists/test-db/l0_cpu.yml and carries pytest.mark.cpu_only.

Test Coverage

CPU checks run from the branch (no GPU on the development host; GPU tests are left to CI):

• Pure-move check on the head commit (output above): 4 definitions moved verbatim; 0 differences; shim exports 0 names; the same result with the merge base with main as the base commit.
• No file imports a moved name from the old path: an AST scan of every tracked *.py (module-level and function-level from tensorrt_llm.serve.responses_utils import ...) prints nothing; git grep for responses_utils.<moved name> in tests prints nothing; import tensorrt_llm.serve.responses_utils binds none of the four names (no re-export).
• Static import graph (module-level imports) on the head: openai_client, openai_service, disagg_coordinator, router, _base.service_types, _base.latency_logger do not import responses_utils; openai_disagg_server (ServerArrivalTimeMiddleware, guard_responses_stream, stream_error_event) and openai_disagg_service (responses_done_generator) still do, as described above.
• pytest -m cpu_only tests/unittest/serve/test_layering.py -q: 9 passed (the new contract and its one allowance included; the stale-allowance check passes).
• pytest -m cpu_only tests/unittest/serve/test_disagg_imports.py -q -rxX: 5 passed, 1 xfailed (openai_disagg_service, strict) in 2:43; the positive control passes.
• pytest -m cpu_only tests/unittest/serve tests/unittest/disaggregated -q: 1528 passed, 1 skipped, 3 xfailed, 144 deselected, 3 subtests passed in 7:03.
• Patch interception: with tensorrt_llm._utils.get_steady_clock_now_in_seconds temporarily raising, the three tests that patch tensorrt_llm.serve.openai_client.get_steady_clock_now_in_seconds (test_request_metrics_normalize_frontend_clock_domain, test_streaming_metrics_normalize_frontend_clock_domain, test_refused_frontend_port_falls_back_to_shared_port) still pass and 111 unpatched callers in the same file fail; the temporary edit is not committed.
• LLM_MODELS_ROOT=... PYTHONPATH=<abs>/tests:<abs> python3 -m test_common.serve.cpu_tests (one pytest process per CPU test-list entry, as the CPU stage runs them): 66 pytest processes (every kept entry ran; the runner reports the first non-zero code): 4199 passed, 1 xfailed (the declared openai_disagg_service case), 30 deselected, 2 failed, both unrelated to this change and not reproducible as regressions: grpc/openengine/test_frontend_lifecycle.py::test_frontend_group_cleanup[shutdown] timed out (90 s lifecycle wait) on a loaded host and passes in isolation on this branch (1 passed in 49.65s; the OpenEngine frontend imports none of the moved names), and llmapi/apps/test_render_standalone.py::TestCpuOnlyStartup::test_the_launcher_loads_a_cuda_stub_only_when_the_driver_is_missing fails with trtllm-render is not installed identically on the dependency branch (the console script is absent in this source-overlay environment).
• pytest tests/unittest/api_stability -q: 98 passed in 8.16s (no file under tests/unittest/api_stability/references*/ changes).
• pre-commit run --files <every changed file> twice: the first pass applied isort/autoflake/ruff import ordering and removed the four now-unused imports from responses_utils.py; the second pass changed nothing; python scripts/legacy_utils.py check-configs exits 0 (its stale-entry hints predate this PR).
• Registration: tests/integration/test_lists/test-db/l0_cpu.yml lists unittest/serve (directory entry; scripts/test_to_stage_mapping.py --tests "unittest/serve" names the CPU-Generic stages); grep -c "pytest.mark.cpu_only" tests/unittest/serve/test_disagg_imports.py is 1.
• Not applicable here: the two-FastAPI-version run (no route is added or moved), the harmony-test skip check (none of the try-imported symbols changes; postprocess_handlers.py imports only ResponsesStreamingProcessor and create_response_non_store from responses_utils).

GPU coverage (CI): disaggregated/test_disaggregated.py::test_disaggregated_single_gpu and ::test_disaggregated_perf_metrics (l0_a10.yml) exercise ResponseHooks through the perf-metrics hooks; the multi-GPU stages are added because openai_client.py, openai_disagg_server.py, openai_disagg_service.py, responses_utils.py and router.py are in the multi-GPU file list, and they need the ci: full pre-merge approved label.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

🤖 Generated with Claude Code