Repository navigation
[TRTLLM-17269][refactor] Add per-dialect error renderers in serve/exception_handling - #21566
Draft
JunyiXu-nv wants to merge 28 commits into
Draft
JunyiXu-nv wants to merge 28 commits into
JunyiXu-nv wants to merge 28 commits into
Conversation
…apshots and a CPU test seam for trtllm-serve Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
… baselines at the merge base The route reference is generated from the live apps; the object.__new__ baseline and the gRPC import allowance record what exists on main today and only shrink. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…n the snapshot matrix An attached frontend of a multi-frontend server registers ten launcher forwarding routes before the local table; seven share a method and path with local handlers and three use a path parameter that matches every batch id, so registration order decides which handler serves them. The new configuration records all of it, a live invariant pins the forwarding-before-local order independently of the reference, and the route-class failure message now says why the msgpack route class matters. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
- AGENTS.md: the CPU test command sets LLM_MODELS_ROOT, which the runner requires, and the stage-mapping note says the mapper matches list-entry text, so a file listed through its directory maps by the directory. - check_contracts matches each from-import as <module>.<name>, so 'from pkg import sub' and 'from .. import sub' count as imports of the submodule; new test for both forms and for the allowance entry. - The VisualGen /v1/models test asserts the 200 status. - The AGENTS.md command check also recognizes commands that start with LLM_MODELS_ROOT=. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…on the current main The disaggregated frontend now registers its routes with KvHintsRoute (NVIDIA#20269), which strips kv_hints from request bodies unless the server trusts its ingress. Regenerate the route reference: the six disaggregated configurations record KvHintsRoute where they recorded APIRoute; their route sets and order are unchanged. The route-class invariant, which a regenerated reference cannot hide, now requires KvHintsRoute on every API route of the disaggregated frontend and APIRoute on the coordinator app. Dropping the KV-hint policy from the frontend fails it even after a regeneration, and both apps still reject the aggregated server's msgpack route class. The object.__new__ baseline records the sites on main at the merge base: test_kv_hints.py builds one server that way (NVIDIA#20269), and test_harmony_parsing.py no longer builds any (NVIDIA#21176). The harmony parsing tests are now registered as a whole file (NVIDIA#21411), so the node-id entry that the CPU-selection test used as its example is gone. The node-id check now runs select_entries on a one-entry list of its own, so it no longer depends on which node-id entries l0_cpu.yml holds. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ail on silent regressions The route snapshot now requires every configuration other than the three invalid ones to build, so a configuration that starts failing at construction cannot pass through a regenerated reference. The object.__new__ ratchet resolves 'from ... import <server class> as <name>' aliases and records each baseline site as its enclosing function or class plus the server class, so swapping one site for another in the same file no longer passes. The baseline keeps the same 52 sites in the same 26 files. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
… and disaggregated servers Create tensorrt_llm/serve/middleware/ and move ServerArrivalTimeMiddleware (from serve/responses_utils.py), PerfMetricsMiddleware (from serve/perf_metrics.py) and _apply_fastapi_middlewares (from commands/serve.py) into it, verbatim. Both servers now install their built-in middleware through serve/middleware/register.py::register_middlewares, which reproduces each app's stack exactly: the aggregated server passes adjusted_clock to both middlewares, the disaggregated server to neither. User --middleware entries are still applied after construction and stay outermost. The old import paths are removed with no compatibility shim; no code outside the repository imports them. commands/serve.py keeps an import of _apply_fastapi_middlewares for its own two launcher calls. CI bookkeeping in the same change: the multi-GPU trigger table gains tensorrt_llm/serve/middleware/; legacy-files.txt registers middleware/arrival_time.py and middleware/user_middleware.py with the configs gen-configs regenerates; ruff-legacy-baseline.json carries the moved docstring's one D212; mypy checks the new register.py. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…g request state
Replace the assignment scope["state"] = {} with scope.setdefault("state", {})
in ServerArrivalTimeMiddleware.__call__. State already present when a request
reaches this middleware - written by a user --middleware entry, or copied in by
the ASGI server from a lifespan that yields a dict - is now kept, and
server_arrival_time is added to it. Before, that state was replaced by an empty
dict. Neither trtllm-serve app yields lifespan state today and the built-in
middleware order is unchanged, so no built-in path changes behavior.
Two tests pin the new behavior (state from an outer middleware and from a
lifespan survive); both fail on the previous commit.
Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…lm/serve/_base/msgspec_route.py Generated move: _msgpack_decoder, _MsgspecRequest and _MsgspecRoute leave tensorrt_llm/serve/openai_server.py byte for byte; the module imports _MsgspecRoute back for its app router and keeps no re-export (the three names are private). The two disaggregated test importers now import the new module. OpenAIServer.health, .version and .get_model are removed here and replaced by the system router added in the following commits. tensorrt_llm/serve/_base/msgspec_route.py joins legacy-files.txt (its code comes from a legacy-formatted file), and the generated lint configs are regenerated with scripts/legacy_utils.py gen-configs. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ve the system endpoints through it New structure: tensorrt_llm/serve/_base/ (shared serving infrastructure) and tensorrt_llm/serve/api/ (endpoint packages, one router plus one serving class each; the package __init__ imports nothing so routers load lazily, by role). serve/_base/routing.py provides make_router(), which always uses the msgpack-decoding route class, and attach_routers(), which imports and attaches only the routers whose RouterSpec matches the server role. The first endpoint package, serve/api/ops/system/, serves /health, /version and /v1/models for every role. OpenAIServer builds a SystemServing with the injected _health_probe (today's probe body, SIGINT side effect included) and attaches the routers before the role dispatch; the 12 add_api_route lines for the three paths leave the four role tables. Same routes, methods, route classes, bodies and conflict-relevant order: the recorded route reference is unchanged. The layering tests gain the new packages' contracts, the route-snapshot failure message points new endpoints at the router pattern, and the serve agent docs describe where a new endpoint goes. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…w serve packages tensorrt_llm/serve/_base/ and tensorrt_llm/serve/api/ops/ join the multi-GPU trigger list: the msgpack route class decodes the disaggregated frontend's worker requests, and /health is the disaggregated readiness probe, so changes under these directories keep running the multi-GPU stages that exercise that path. The three new modules join the strict mypy file set, and the type-check pre-commit hook now also triggers on changes under tensorrt_llm/serve/ and tensorrt_llm/commands/serve_cli/, so edits to typed serving modules get local type checking; the hook still checks only the declared file set. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
… tests that build routes by hand Two multi-frontend tests build an OpenAIServer without its constructor and call register_routes() themselves. Since /health, /version and /v1/models are attached through the system router by the constructor, those tests no longer saw the three routes: one asserted that /version stays local on an attached frontend (404), the other reported the three per-frontend-safe entries as removed. Both helpers now set the serving object and attach the routers before the hand registration, in the constructor's order, and the route enumeration walks included routers, which FastAPI 0.139 keeps as objects without a path in app.routes. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…n the OpenEngine metrics test The HTTP-enablement case builds a real OpenAIServer through its constructor and looked for /prometheus/metrics by reading .path off every entry of app.routes. Since the constructor attaches the system router, FastAPI 0.139 keeps an included-router object there that has no .path, and the child process failed with AttributeError. The test now matches a GET /prometheus/metrics request against the routes with the public route-matching API, which works on both ends of the pinned FastAPI range. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…d name The check that endpoint packages under tensorrt_llm/serve/api do not import each other read the package off the from-module alone, so `from tensorrt_llm.serve import api` inside an endpoint package passed and `from tensorrt_llm.serve.api.ops import system` inside ops/system failed. It now reads the package off <module>.<name> per imported name, as the layering contracts do, and a synthetic-edge test pins both forms. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…efore splitting it Three tests run OpenAIServer.openai_chat through a server built with the real constructor and the fake generator, and pin what it answers today: a request the handler rejects keeps its 400 body byte for byte, an engine failure keeps HTTP 200 with body null (and /health_generate keeps turning that into a 500), a response that cannot be JSON-encoded keeps its 400, and a stream keeps its frames and media type. They pass on the unsplit handler and must still pass once it is split into a typed core and an HTTP wrapper. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…serve/_base/sse.py Generated move: the chat_stream_generator closure of OpenAIServer.openai_chat is lifted to module level in tensorrt_llm/serve/_base/sse.py unchanged, apart from its two free variables, which become the keyword-only parameters `server` (was `self`) and `raw_request`. openai_chat imports it back and passes server=self, raw_request=raw_request at its one call. The generator keeps its `except Exception:` clause and its two `_postprocess_result` reads. tensorrt_llm/serve/_base/sse.py joins legacy-files.txt (its code comes from a legacy-formatted file) and the generated lint configs are regenerated with scripts/legacy_utils.py gen-configs; ruff-legacy-baseline.json gains the module's one recorded F821 (the `anext` builtin ruff does not know), which openai_server.py keeps counting until its baseline is regenerated. The Anthropic stream tests patched the logger of openai_server to check what the generator logs; they now patch the logger of serve/_base/sse.py, where the generator lives. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ompletion and an HTTP wrapper OpenAIServer.create_chat_completion is the former body of openai_chat with a typed contract: it returns the ChatCompletionResponse for a non-streaming request and the SSE stream as strings for a streaming one, and it raises ServingError (the HTTP status apart from the OpenAI error body) where it used to return an error response, or EngineFatalError after the three side effects of an engine failure. Nothing else in the body changes: the six early rejections raise where they returned, the ValueError and the generic handler raise what they rendered, and the ServingError handler comes first so an early rejection is neither logged nor re-wrapped. openai_chat keeps its name, signature and route and becomes the wrapper that renders the typed result: JSONResponse for the response, a streaming response for the stream, the error body with its status for ServingError, and HTTP 200 with body null after an engine failure, as before. Its own ValueError branch keeps the 400 for a response that cannot be JSON-encoded, which used to fail inside the core. Every caller (the route, /health_generate, the Anthropic messages and batch paths) is unchanged. tensorrt_llm/serve/_base/errors.py holds error_body, ServingError, serving_error and EngineFatalError; create_error_response builds its body with error_body, so a ServingError body and a rendered error can never differ (the over-length-prompt code lives in one place). Tests: the typed core and its errors through a server built with the real constructor; the error builders against create_error_response; the lifted SSE generator with fakes. The three answer-pinning tests added before the split pass unchanged. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
The new error module is written from scratch, so it joins the mypy file list in pyproject.toml; scripts/run_mypy.sh passes with it (the strict global settings apply). tensorrt_llm/serve/_base/sse.py stays out: it is moved legacy code and joins type checking with its package later. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…se in the multimodal prompt test The chat handler now serializes a ChatCompletionResponse and streams any other result of its core, so a stand-in for _create_chat_response that only has model_dump() is no longer rendered as JSON. The test stubs a real ChatCompletionResponse instead; what it checks (the rendered prompt, the decoded image and chat_template_applied reaching the engine) is unchanged. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…d 400 without a traceback The HTTP wrapper answers a response that cannot be JSON-encoded from its own ValueError branch, which logs nothing; the generic handler would render the same 400 but log a traceback. The answer-pinning test now also asserts that no ERROR record is emitted, so dropping that branch fails the test. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…eware package branch as the base for the shared app assembly Base for the shared app assembly: both open PRs it builds on, NVIDIA#21447 (shared middleware package for the aggregated and disaggregated servers) and NVIDIA#21448 (router skeleton and the ops.system endpoints), merged into one branch. Conflicts resolved by keeping both sides: the mypy file list in pyproject.toml and the import-direction contracts in tests/unittest/serve/test_layering.py. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ared assembly of the trtllm-serve apps tensorrt_llm/serve/_base/app_builder.py holds the two functions both servers build on instead of a shared base class. build_http_app() assembles a FastAPI app in one fixed order: the lifespan, the optional route class before any route exists, the exception handlers, the routers in order, and the built-in middleware last so it stays outermost. serve_uvicorn() runs the app through the telemetry-aware uvicorn server, with an optional task that runs next to the server and receives it. The module is typed (added to the mypy file list) and imports only FastAPI, Starlette, uvicorn, the standard library and serve/_telemetry.py. Its tests pin the assembly order, the route class, the middleware stack order, the lifespan and exception-handler pass-through, every uvicorn argument, and the import boundary. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…m-serve apps with build_http_app and serve_uvicorn OpenAIServer and OpenAIDisaggServer construct their FastAPI app through build_http_app() and run uvicorn through serve_uvicorn(). Every difference between the two apps stays an explicit argument at the call site: the msgpack-decoding route class on the aggregated app and the KV-hint policy route class (KvHintsRoute) on the disaggregated frontend, both of which strip kv_hints from untrusted request bodies; the adjusted clock for the aggregated middleware and none for the disaggregated one; "info" and the 5 s keep-alive for the aggregated server, logger.level and the configured keep-alive for the disaggregated one; the worker-registration task after start only on the aggregated server. Both middleware installers read the server's perf-metrics attributes when they run, after the routes, as the aggregated server did before. Both servers keep reading TRTLLM_TRUST_KV_HINTS into app.state.trust_kv_hints, now right after the build; nothing reads the flag before the first request. No behavior change. The validation handlers keep their bodies and are registered with app.add_exception_handler instead of the decorator; the aggregated server's role dispatch goes unchanged into _attach_role_routes(app), and its post-start closure becomes the method _register_after_serving(server). On the disaggregated app the two built-in middlewares are now added after the routes instead of before; nothing in its route registration adds middleware, so the resulting stack is the same and the middleware-order snapshot is unchanged. Tests: tests/unittest/serve/test_server_uvicorn_settings.py builds both servers through the test factory and pins what each hands to uvicorn (log level, keep-alive, sockets, the post-start task only on the aggregated server) and what build_http_app must carry for each app: its route class on every API route and the KV-hint trust flag. The unit test that patched uvicorn through openai_disagg_server now patches it where serve_uvicorn looks it up and also checks the disaggregated log level. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…d app assembly branch as the base for the per-dialect error renderers Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…serve apps before rendering them per dialect Posts invalid bodies to servers built through the real constructors and checks status, body and side effects: the aggregated server's OpenAI shape with its 4096-character cap, the Anthropic envelope, the VisualGen 422 envelope and the metrics call; the disaggregated frontend's uncapped OpenAI shape, Anthropic envelope, counter and throttled log with escaped control characters. The validation text is read from the exception FastAPI raised, so the test does not re-implement the rendering. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…error boundary and the per-dialect error registry tensorrt_llm/serve/_base/errors.py gains the dialect-neutral error types a chat backend raises for any wire dialect to render (DialectNeutralError with InvalidRequestError, ContextLengthExceededError, UpstreamError, UnavailableError, InternalServingError; neutral_error picks the type for an HTTP status), ErrorRule and serving_error_boundary, the error ladder of an HTTP handler as a declaration, with CHAT_LADDER for the chat completion wrapper. The OpenAI body keeps its single builder, error_body. tensorrt_llm/serve/exception_handling/register.py is the per-app registry of error renderers keyed by (dialect, exception type): route_dialect marks an endpoint's dialect, dialect_of reads it from the matched endpoint, default_error_renderers gives the OpenAI and Anthropic renderers for validation errors and the typed errors, and install_exception_handlers puts the registry on app.state and installs the app-level validation handler with a per-server observer for side effects. The module imports no server or VisualGen code; the layering test holds that. Typed (mypy strict), with unit tests for the registry, the boundary and the neutral errors, including that the OpenAI rendering of a neutral error equals OpenAIServer.create_error_response and that the chat completion wrapper answers as before through the boundary. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…re dialect
Both servers replace their validation handler's path test
(request.url.path.startswith("/v1/messages")) and the aggregated server's
VisualGen role test with the per-dialect registry: install_exception_handlers
resolves the dialect of the matched endpoint (the eight aggregated and two
disaggregated Anthropic endpoints carry @route_dialect(ANTHROPIC)), falls back
to the server's default dialect (VisualGen on the VisualGen role, OpenAI
otherwise), calls the server's _observe_validation_error for its side effects
and renders through the registry. Side effects stay as they were: the
aggregated server counts a rejection on its metrics collector and caps the
OpenAI-shaped text at 4096 characters; the disaggregated frontend counts on
its Prometheus counter and logs the first and every 1000th OpenAI-shaped
rejection. The VisualGen 422 renderer is registered with the VisualGen routes,
for that role only, and a VisualGen server without it fails at startup.
The chat completion wrapper's four except branches become
@serving_error_boundary(OPENAI, ladder=CHAT_LADDER, on_engine_fatal=lambda:
None) with the same answers. The Anthropic route tests install the servers'
own handler instead of a copy of it. The multi-GPU trigger list gains
tensorrt_llm/serve/exception_handling/.
Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…a layering contract test_registry_modules_import_no_visual_gen scanned every import statement of serve/exception_handling/register.py and serve/_base/errors.py, at any scope including if TYPE_CHECKING:, for three VisualGen modules. The layering contract for serve/exception_handling/ already forbids those modules for register.py at run time, so the scan's own part was serve/_base/errors.py and the type-checking imports of both files. A layering contract over serve/exception_handling/ and serve/_base/errors.py, with the same three modules and type-checking imports included, replaces the test and its helpers. The contract also resolves relative imports and from-imports of a submodule through its parent package (from tensorrt_llm.serve import openai_video_routes), which the scan, matching module names only, did not see. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
JunyiXu-nv
force-pushed
the
dev-junyix-feat-serve-exception-handling
branch
from
November 4, 2026 12:51
0183e3a to
0bbc9ee
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #21556 (Extract the shared HTTP app assembly into serve/_base/app_builder.py) and #21515 (Split the chat handler into a typed core and lift the SSE generator into serve/_base): the first commits below belong to those two PRs and disappear from this diff when they merge; this PR's own change is the last 4 commits. Draft until both have merged; it is then rebased onto
main, re-verified and marked ready.@coderabbitai summary
Description
Both trtllm-serve apps chose the shape of a request-validation error with the same hard-coded path test,
request.url.path.startswith("/v1/messages"), plus (aggregated server) a role test for VisualGen. This PR replaces thatifwith error renderers registered per wire dialect (OpenAI, Anthropic, VisualGen) intensorrt_llm/serve/exception_handling/register.py. An endpoint declares its dialect with@route_dialect(...); the app-level validation handler resolves the dialect of the endpoint that matched the request (Starlette records it in the ASGI scope), falls back to the app's default dialect (VisualGen on the VisualGen role, OpenAI otherwise), calls the server's observer for its side effects, and renders through the app'sErrorRenderersregistry onapp.state.error_renderers. Both servers install that handler set throughbuild_http_app'sdialect_error_renderersslot (functools.partial(install_exception_handlers, ...)), next to their route class (_MsgspecRouteon the aggregated server,KvHintsRouteon the disaggregated frontend), and setapp.state.trust_kv_hintsright after the build, ahead of the VisualGen startup check described next. The VisualGen 422 renderer is registered only on the VisualGen role, from the VisualGen route registration, and a VisualGen server whose registration is missing fails at startup instead of silently answering with the OpenAI shape.tensorrt_llm/serve/_base/errors.pygains the dialect-neutral error types that chat backends raise for any dialect to render (DialectNeutralErrorwithInvalidRequestError,ContextLengthExceededError,UpstreamError,UnavailableError,InternalServingError;neutral_error(status, message)picks the type), andserving_error_boundary, the error ladder of an HTTP handler written as a declaration (ErrorRule,CHAT_LADDER). The boundary renders the typed errors through the dialect's registered renderers and maps a handler's other exceptions with its ladder; it reads the registry from the request's app and fails with a message naminginstall_exception_handlerswhen an app has none. It is applied to one handler, the chat completion HTTP wrapper of the typed chat core, where it replaces the wrapper's fourexceptbranches with the same answers. One later PR, after the handlers have moved into serving classes, applies it to every route together with the library's exception contract (TRTLLM-16117). The chat core and the chat backends keep raisingServingErrorfor now, because the Anthropic serving class (#21551, Converge the three Anthropic orchestrations into one serving class) is not onmain; switching them to the neutral types is that follow-up's job once it is.error_body, the single builder of the OpenAI error body that #21515 adds intensorrt_llm/serve/_base/errors.py, now also takes a plainintstatus besides anHTTPStatus(code = int(status_code)), so a status with noHTTPStatusmember, such as a worker's 499 carried byUpstreamError, renders a body instead of raising; every existing caller passes anHTTPStatusand gets byte-identical bodies. This extends the module that #21515 introduces, so that PR's author and the rebase at ready-flip carry it knowingly.Part of the trtllm-serve refactor: cross-cutting pieces and Responses package (epic TRTLLM-17056, task TRTLLM-17269). Builds on the shared HTTP app assembly (#21556) and the typed chat core (#21515).
Behavior
No behavior change. Each server keeps its own side effects and bodies, pinned by
tests/unittest/serve/exception_handling/test_server_validation_errors.py, which passes unchanged on the base commit and after the change:/v1/messages*{"type": "error", "error": {"type": "invalid_request_error", "message": str(exc)}}metrics_collector.log_request_error(http_code=400)when a collector is configured{"object": "error", "message": "<loc>: <msg>; ...", "type": "BadRequestError", "param": null, "code": 422}{"error": str(exc)}, text cut at 4096 characters with... [truncated N bytes]log_request_error(http_code=400)when a collector is configured/v1/messages,/v1/messages/count_tokensvalidation_exceptionscounter{"error": str(exc)}, whole textvalidation_exceptionscounter; warning on the 1st and every 1000th rejection with the compact error list, control characters escapedKept on purpose (candidates for a separate behavior-change PR, not made here): the aggregated server counts rejections on its metrics collector while the disaggregated frontend counts and logs; only the aggregated server caps the OpenAI-shaped text; the VisualGen rule stays keyed on the server role (default dialect), not on a path.
Two notes on the validation text. First, FastAPI's
str(RequestValidationError)ends with an endpoint line that names the route's method and path and the endpoint function's file and line; the chat completion route's endpoint is now the boundary wrapper (functools.wrapskeeps the handler's signature, name and docstring, not its code location), so that line namestensorrt_llm/serve/_base/errors.pyinstead oftensorrt_llm/serve/openai_server.py, and on the aggregated server an over-long body's[truncated N bytes]count moves with it. This is not a dialect, status, metrics or log divergence, the line number already changes with any edit of the file, and the pin test compares each body withstr()of the exception FastAPI raised for that request, so it is immune to it; the follow-up that applies the boundary to every route carries the same note. Second, #20865 (serve: answer a request that fails validation with a readable error) is open and changes both servers' validation bodies: this PR keeps today's bodies; if #20865 merges first, this PR is rebased onto it and its OpenAI and Anthropic validation renderers inregister.pyand the disaggregated observer take #20865's bodies, with #20865's request-validation test staying green; if this PR merges first, #20865 changes those two renderer functions and the observer instead of the servers' handlers.How a route's dialect is resolved: the
__trtllm_dialect__marker on the matched endpoint (scope["endpoint"], elsescope["route"].endpoint), else the app's default. The eight aggregated Anthropic endpoints and the two disaggregated ones carry@route_dialect(ANTHROPIC);test_every_messages_route_resolves_to_anthropicwalks every route of the aggregated LLM app and of the disaggregated app and fails, naming the marker, if a/v1/messagesroute resolves to another dialect or any other route to a non-default one. New Anthropic endpoints (the Anthropic serving class of #21551 and the handler moves) must carry the marker.The chat wrapper through the boundary gives the same answers as its four former branches: the core's
ServingErroris rendered with its own status, anEngineFatalErroris answered as HTTP 200 with bodynull, an unencodable response (ValueError) is a 400 without a log, any other failure a 400 with its traceback logged (CHAT_LADDER).test_chat_wrapper_answers_as_beforechecks each on a server built through the real constructor.Moves
None. The disaggregated frontend's throttled validation log moves from a closure in
__init__into the method_observe_validation_errorof the same file, byte-identical after dedent.CI bookkeeping
getMultiGpuFileChangedinjenkins/L0_MergeRequest.groovy): addedtensorrt_llm/serve/exception_handling/, because both apps install the renderers and the disaggregated frontend runs them.tensorrt_llm/serve/exception_handling/register.pyadded to[tool.mypy] files(strict).tests/unittest/serve/are collected by the existingunittest/serveentry oftests/integration/test_lists/test-db/l0_cpu.ymland carrypytest.mark.cpu_only.Open PRs on the same files
OpenAIServer._attach_role_routescalls its renderer function instead of_create_visual_gen_validation_error_response; the body is the same.anthropic_error_responsefrom the adapter's new path.errors.pyimportsErrorResponsefromopenai_protocol, which stays importable.Test Coverage
New tests (all
cpu_only, collected throughunittest/serveinl0_cpu.yml):tests/unittest/serve/exception_handling/test_server_validation_errors.py: both servers' answers (status, body, metrics call, counter, throttled log with the log counter pinned across an Anthropic rejection, escaped control characters), the VisualGen startup check, and the dialect completeness of every route.tests/unittest/serve/exception_handling/test_dialect_error_renderers.py: marker resolution (functions, bound methods, included routers, the route fallback), the per-dialect bodies, the text cap, observer order, the OpenAI fallback and the closest-class rule, the registry read fromapp.stateon each request (a registry swapped in after the install is the one used), streaming renderers.tests/unittest/serve/_base/test_serving_errors.py: the neutral types andneutral_error, both renderings of every typed error including a 499, equality withcreate_error_response, request discovery by annotation, the registry lookup (explicit, from the request's app, the defaults, and the error for an app without one), preserved signature/docstring/OpenAPI, engine-fatal handling, the ladder, and the chat wrapper on a real server.tests/unittest/llmapi/apps/test_anthropic_routes.py: its copy of the validation handler is replaced by the servers' own installer.tests/unittest/llmapi/apps/test_serving_extensions.py: the bare app that mountsopenai_chatinstalls the servers' handler set; a request-schema rejection in that app therefore answers the servers' 400, not FastAPI's default 422.tests/unittest/serve/test_layering.py: two import contracts:serve/exception_handling/imports no server, VisualGen or Responses module;serve/exception_handling/andserve/_base/errors.pyimport no VisualGen module,if TYPE_CHECKING:imports included (replaces the direct-import scan the dialect test file carried).Local runs (CPU; no GPU on the box):
pytest -m cpu_only tests/unittest/serve -q(FastAPI 0.139.2): 200 passed.tests/unittest/serve/exception_handling tests/unittest/serve/_base tests/unittest/api_stability/test_serve_routes.py tests/unittest/serve/api/visual_gen/test_routes_visual_gen.py tests/unittest/llmapi/apps/test_anthropic_routes.py tests/unittest/llmapi/apps/test_serving_extensions.py: 287 passed.pytest -m cpu_only tests/unittest/api_stability/test_serve_routes.py tests/unittest/disaggregated tests/unittest/llmapi/apps/test_disagg_serving_perf_metrics.py tests/unittest/llmapi/apps/test_request_metrics.py tests/unittest/_torch/visual_gen/test_trtllm_serve_endpoints.py: 1623 passed, 1 skipped, 2 xfailed.python3 -m test_common.serve.cpu_tests, one pytest process per test-db entry): 46 sessions green.test_server_validation_errors.pyon the base commit, before the change: 15 passed (the two registry tests were added with the change).[truncated N bytes]count on an over-long body, which follows the endpoint line described above.neutral_errorand both renderings for every HTTP status, the boundary against the former four-branch wrapper, and request discovery over generated signatures: no counter-example; 16 planted bugs in the two modules and 3 in the servers' observers and handler were each caught by the tests or the properties.tensorrt_llm/serve/exception_handling/register.pyandtensorrt_llm/serve/_base/errors.py: no issues;python scripts/legacy_utils.py check-configs: up to date.test_e2e.py::test_anthropic_messages,test_malformed_json_request, the single-GPU disaggregated block.PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.🤖 Generated with Claude Code