Repository navigation
[TRTLLM-17263][refactor] Converge the three Anthropic orchestrations into one serving class - #21551
Draft
JunyiXu-nv wants to merge 26 commits into
Draft
JunyiXu-nv wants to merge 26 commits into
JunyiXu-nv wants to merge 26 commits into
Conversation
…apshots and a CPU test seam for trtllm-serve Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
… baselines at the merge base The route reference is generated from the live apps; the object.__new__ baseline and the gRPC import allowance record what exists on main today and only shrink. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…n the snapshot matrix An attached frontend of a multi-frontend server registers ten launcher forwarding routes before the local table; seven share a method and path with local handlers and three use a path parameter that matches every batch id, so registration order decides which handler serves them. The new configuration records all of it, a live invariant pins the forwarding-before-local order independently of the reference, and the route-class failure message now says why the msgpack route class matters. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
- AGENTS.md: the CPU test command sets LLM_MODELS_ROOT, which the runner requires, and the stage-mapping note says the mapper matches list-entry text, so a file listed through its directory maps by the directory. - check_contracts matches each from-import as <module>.<name>, so 'from pkg import sub' and 'from .. import sub' count as imports of the submodule; new test for both forms and for the allowance entry. - The VisualGen /v1/models test asserts the 200 status. - The AGENTS.md command check also recognizes commands that start with LLM_MODELS_ROOT=. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…s outside-subclass contract Every OpenAI request on the disaggregated frontend, and every Anthropic /v1/messages request, goes through OpenAIDisaggServer._wrap_entry_point. Before reorganizing that closure, pin what it does today: - tests/unittest/disaggregated/test_disagg_entry_point.py pins the per-request counters, the chat-template 400, the conversation identity from the headers, the serialization of streaming and non-streaming responses (including the guarded streamed Responses relay and model_dump(by_alias=True)), the error mapping through _handle_exception, and the single disaggregated_request trace span. - tests/unittest/disaggregated/test_disagg_server_contract.py pins the surface NeMo-RL's OpenAIDisaggServerAdaptor subclass relies on: the constructor (config, req_timeout_secs=..., coordinator_url=None), self.app with every route registered when __init__ returns, __call__(host, port, sockets=None), _wrap_entry_point, _handle_exception and _perf_metrics_collector.http_exceptions. - The OpenAIDisaggServer class docstring and tensorrt_llm/serve/AGENTS.md name that surface, so a change to it is agreed with the NeMo-RL owners first. Both files are CPU-only and collected by the unittest/disaggregated entry of tests/integration/test_lists/test-db/l0_cpu.yml. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…into prepare, run, finish and serialize steps No behavior change. The body of the closure OpenAIDisaggServer builds in _wrap_entry_point becomes a composition of five methods, so that later changes (disconnect watching, in-stream errors, request trace) land in the step they belong to instead of growing one closure: - _prepare_request: the request counters, the chat-template permission (HTTPException 400), the conversation identity, the per-request hooks - _run_entry_point: the call into the disaggregated service - _finish_request: success accounting (total_responses) - _finish_failed_request: every failure, through _handle_exception while the exception is being handled (its traceback logging needs that) - _serialize_response: StreamingResponse with the Responses stream guard, or JSONResponse(content=...model_dump(by_alias=True)), as before The trace span stays on the wrapper that FastAPI calls; the steps open no spans of their own. The statements keep their order inside the same try block: the counters run before the template check, total_responses before serialization, and a serialization failure still reaches _handle_exception. Some step parameters (raw_req of _run_entry_point, req and hooks of the finish steps) are not read yet; they are the inputs the known upcoming changes to each step need, so those changes will not have to alter the signatures again. The characterization tests and the subclass contract tests of the previous commit pass unchanged on both sides of the split. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…lm/serve/_base/msgspec_route.py Generated move: _msgpack_decoder, _MsgspecRequest and _MsgspecRoute leave tensorrt_llm/serve/openai_server.py byte for byte; the module imports _MsgspecRoute back for its app router and keeps no re-export (the three names are private). The two disaggregated test importers now import the new module. OpenAIServer.health, .version and .get_model are removed here and replaced by the system router added in the following commits. tensorrt_llm/serve/_base/msgspec_route.py joins legacy-files.txt (its code comes from a legacy-formatted file), and the generated lint configs are regenerated with scripts/legacy_utils.py gen-configs. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ve the system endpoints through it New structure: tensorrt_llm/serve/_base/ (shared serving infrastructure) and tensorrt_llm/serve/api/ (endpoint packages, one router plus one serving class each; the package __init__ imports nothing so routers load lazily, by role). serve/_base/routing.py provides make_router(), which always uses the msgpack-decoding route class, and attach_routers(), which imports and attaches only the routers whose RouterSpec matches the server role. The first endpoint package, serve/api/ops/system/, serves /health, /version and /v1/models for every role. OpenAIServer builds a SystemServing with the injected _health_probe (today's probe body, SIGINT side effect included) and attaches the routers before the role dispatch; the 12 add_api_route lines for the three paths leave the four role tables. Same routes, methods, route classes, bodies and conflict-relevant order: the recorded route reference is unchanged. The layering tests gain the new packages' contracts, the route-snapshot failure message points new endpoints at the router pattern, and the serve agent docs describe where a new endpoint goes. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…w serve packages tensorrt_llm/serve/_base/ and tensorrt_llm/serve/api/ops/ join the multi-GPU trigger list: the msgpack route class decodes the disaggregated frontend's worker requests, and /health is the disaggregated readiness probe, so changes under these directories keep running the multi-GPU stages that exercise that path. The three new modules join the strict mypy file set, and the type-check pre-commit hook now also triggers on changes under tensorrt_llm/serve/ and tensorrt_llm/commands/serve_cli/, so edits to typed serving modules get local type checking; the hook still checks only the declared file set. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
… tests that build routes by hand Two multi-frontend tests build an OpenAIServer without its constructor and call register_routes() themselves. Since /health, /version and /v1/models are attached through the system router by the constructor, those tests no longer saw the three routes: one asserted that /version stays local on an attached frontend (404), the other reported the three per-frontend-safe entries as removed. Both helpers now set the serving object and attach the routers before the hand registration, in the constructor's order, and the route enumeration walks included routers, which FastAPI 0.139 keeps as objects without a path in app.routes. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…n the OpenEngine metrics test The HTTP-enablement case builds a real OpenAIServer through its constructor and looked for /prometheus/metrics by reading .path off every entry of app.routes. Since the constructor attaches the system router, FastAPI 0.139 keeps an included-router object there that has no .path, and the child process failed with AttributeError. The test now matches a GET /prometheus/metrics request against the routes with the public route-matching API, which works on both ends of the pinned FastAPI range. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…d name The check that endpoint packages under tensorrt_llm/serve/api do not import each other read the package off the from-module alone, so `from tensorrt_llm.serve import api` inside an endpoint package passed and `from tensorrt_llm.serve.api.ops import system` inside ops/system failed. It now reads the package off <module>.<name> per imported name, as the layering contracts do, and a synthetic-edge test pins both forms. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ve/api/anthropic Move tensorrt_llm/serve/anthropic_adapter.py, anthropic_protocol.py and anthropic_batches.py whole into tensorrt_llm/serve/api/anthropic/ as adapter.py, protocol.py and batches.py, and rewrite every import of them in the repository. The old paths are deleted without a compatibility shim: no code outside the repository imports them. Generated change: the three files are copied byte for byte apart from their import lines; the importers change only in their import lines. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…efore splitting it Three tests run OpenAIServer.openai_chat through a server built with the real constructor and the fake generator, and pin what it answers today: a request the handler rejects keeps its 400 body byte for byte, an engine failure keeps HTTP 200 with body null (and /health_generate keeps turning that into a 500), a response that cannot be JSON-encoded keeps its 400, and a stream keeps its frames and media type. They pass on the unsplit handler and must still pass once it is split into a typed core and an HTTP wrapper. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…serve/_base/sse.py Generated move: the chat_stream_generator closure of OpenAIServer.openai_chat is lifted to module level in tensorrt_llm/serve/_base/sse.py unchanged, apart from its two free variables, which become the keyword-only parameters `server` (was `self`) and `raw_request`. openai_chat imports it back and passes server=self, raw_request=raw_request at its one call. The generator keeps its `except Exception:` clause and its two `_postprocess_result` reads. tensorrt_llm/serve/_base/sse.py joins legacy-files.txt (its code comes from a legacy-formatted file) and the generated lint configs are regenerated with scripts/legacy_utils.py gen-configs; ruff-legacy-baseline.json gains the module's one recorded F821 (the `anext` builtin ruff does not know), which openai_server.py keeps counting until its baseline is regenerated. The Anthropic stream tests patched the logger of openai_server to check what the generator logs; they now patch the logger of serve/_base/sse.py, where the generator lives. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
… mark the new package Add tensorrt_llm/serve/api/anthropic/__init__.py, retarget the two module imports of the adapter in test_anthropic_adapter.py to the new path, and add tests/unittest/serve/api/anthropic/test_messages_behavior.py: characterization tests of what the aggregated and the disaggregated server answer on the Anthropic routes today (success, streaming, a stream without chunks, rejected requests, context length, upstream errors, unexpected errors, engine failures, an unconvertible chat response, and the aggregated Message Batches), with the served model and the requested model named differently so that each server's own answer is pinned. The follow-up PR that gives both servers one Anthropic Messages serving class must keep this file unchanged and green. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ompletion and an HTTP wrapper OpenAIServer.create_chat_completion is the former body of openai_chat with a typed contract: it returns the ChatCompletionResponse for a non-streaming request and the SSE stream as strings for a streaming one, and it raises ServingError (the HTTP status apart from the OpenAI error body) where it used to return an error response, or EngineFatalError after the three side effects of an engine failure. Nothing else in the body changes: the six early rejections raise where they returned, the ValueError and the generic handler raise what they rendered, and the ServingError handler comes first so an early rejection is neither logged nor re-wrapped. openai_chat keeps its name, signature and route and becomes the wrapper that renders the typed result: JSONResponse for the response, a streaming response for the stream, the error body with its status for ServingError, and HTTP 200 with body null after an engine failure, as before. Its own ValueError branch keeps the 400 for a response that cannot be JSON-encoded, which used to fail inside the core. Every caller (the route, /health_generate, the Anthropic messages and batch paths) is unchanged. tensorrt_llm/serve/_base/errors.py holds error_body, ServingError, serving_error and EngineFatalError; create_error_response builds its body with error_body, so a ServingError body and a rendered error can never differ (the over-length-prompt code lives in one place). Tests: the typed core and its errors through a server built with the real constructor; the error builders against create_error_response; the lifted SSE generator with fakes. The three answer-pinning tests added before the split pass unchanged. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
The new error module is written from scratch, so it joins the mypy file list in pyproject.toml; scripts/run_mypy.sh passes with it (the strict global settings apply). tensorrt_llm/serve/_base/sse.py stays out: it is moved legacy code and joins type checking with its package later. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…se in the multimodal prompt test The chat handler now serializes a ChatCompletionResponse and streams any other result of its core, so a stand-in for _create_chat_response that only has model_dump() is no longer rendered as JSON. The test stubs a real ChatCompletionResponse instead; what it checks (the rendered prompt, the decoded image and chat_template_applied reaching the engine) is unchanged. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…d 400 without a traceback The HTTP wrapper answers a response that cannot be JSON-encoded from its own ValueError branch, which logs nothing; the generic handler would render the same 400 but log a traceback. The answer-pinning test now also asserts that no ERROR record is emitted, so dropping that branch fails the test. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…feat-serve-anthropic-serving-class (integration base, not part of the reviewable change) Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ix-feat-serve-anthropic-serving-class (integration base, not part of the reviewable change) Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…erving classes The two classes receive the handlers that leave OpenAIServer in the next commit. Their constructors take what those handlers read from the server (the render-resources factory, the batch item runner), so missing wiring fails at construction instead of falling back to a default. The module is registered as a legacy-formatted file before any moved code lands in it, so the formatters keep the moved methods as they are; ruff-legacy.toml and the managed blocks of pyproject.toml and .pre-commit-config.yaml are regenerated from legacy-files.txt. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ndlers out of OpenAIServer Generated move: anthropic_count_tokens goes to AnthropicCountTokensServing and _batch_store, the six Message Batches handlers and _no_such_batch go to AnthropicBatchesServing in tensorrt_llm/serve/api/anthropic/serving.py, verbatim except for the declared rewrite in count_tokens (RenderResources.from_server(self) becomes self._render_resources(), the injected factory). anthropic_messages, _run_batched_message and _synthetic_request are deleted here and replaced by the serving class and the request context added in the next commit. The import cleanup is the formatters' (autoflake, isort, yapf). Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
… aggregated and disaggregated servers The Anthropic Messages orchestration existed three times (aggregated messages, the aggregated Message Batches runner, disaggregated messages), each serializing the chat result, parsing it back and sniffing the error body for a message; the batch runner entered the chat pipeline through a synthetic ASGI request. One AnthropicMessagesServing now runs it for any ChatBackend: the aggregated backend is the typed chat core create_chat_completion, the disaggregated backend is the entry-point wrapper's prepare, run and finish steps without the serialization step (the new _call_entry_point, in the same span). A RequestContext carries what the chat core reads from a request (headers, state, raw JSON, disconnect polling); the batch path builds one directly. The eight aggregated Anthropic routes come from the new router in tensorrt_llm/serve/api/anthropic/api_router.py, attached through the router table and gated by the new harmony_model capability as the route registration was gated. The launcher forwarding routes of an attached frontend register before the routers, so that they keep winning over the local Message Batches handlers. Each server keeps its own answers: the streamed model name and the error-message extraction come from its backend (stream_model, error_message); an engine failure stays a 500 on both. No behavior change. The new modules join the mypy file list; the two serving-class modules join the multi-GPU trigger table because they carry the disaggregated messages path. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ntext and the router on the real apps New tests: tests/unittest/serve/api/anthropic/test_serving.py (the serving class over a fake chat backend: the stream is reframed from the backend's own iterator, the backend's model and message reach the answer, each error kind keeps its status, the batch path hands the backend a non-streaming request and an internal context whose JSON is the converted request); tests/unittest/serve/_base/test_request_context.py (delegation to the HTTP request including its msgpack decoding, the internal context, the disconnect watcher returning at once for it); tests/unittest/serve/api/anthropic/test_api_router.py (the eight routes come from the router with the msgpack route class on the LLM roles only, none on a harmony model or another role, count_tokens decodes a msgpack body, attaching needs the serving objects). All three carry the cpu_only marker and run through the unittest/serve entry of the CPU list. Rewritten on the fixture factory: test_anthropic_routes.py posts to the real apps and their own validation handler instead of a mirror handler on bare servers, drives the batch and count-tokens routes through the serving objects on app.state, and hands the synthetic-request test to test_request_context.py; TestCountTokens in test_render_callers.py builds the count-tokens serving over a namespace. The chat-core and SSE-generator tests follow the renamed parameter; the multi-frontend tests set the Anthropic serving objects before attaching the routers. The object.__new__ ratchet drops to 2 and 1 sites for the two rewritten files; a layering contract keeps the Anthropic package from importing a server. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
1 task done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@coderabbitai summary
Description
Stacked on #21518 (Move the Anthropic serving modules into serve/api/anthropic), #21515 (Split the chat handler into a typed core and lift the SSE generator into serve/_base) and #21442 (Split the disaggregated server's entry-point wrapper into prepare, run, finish and serialize steps); the first commits below belong to those PRs, merged into an integration base, and disappear from this diff when they merge; this PR's own change is the last 4 commits.
The Anthropic Messages orchestration existed three times: the aggregated
/v1/messageshandler, the aggregated Message Batches runner and the disaggregated/v1/messageshandler. Each one serialized the chat result to JSON, parsed it back and sniffed the error body for a message, and the batch runner built a fake ASGI request to enter the chat pipeline. This PR replaces the three with oneAnthropicMessagesServingover an injected chat backend: the aggregated backend is the typed chat corecreate_chat_completion, the disaggregated backend is the entry-point wrapper's prepare, run and finish steps without the serialization step. ARequestContextcarries what the chat core reads from an HTTP request (headers, state, raw JSON, disconnect polling), so the batch path builds one directly instead of a synthetic request. The eight aggregated Anthropic routes (messages, count_tokens, six Message Batches routes) move fromOpenAIServerinto a router undertensorrt_llm/serve/api/anthropic/, attached through the router table and gated as before (not on a harmony/gpt-oss model). Each side keeps its observable behavior; what differs between them (the model a stream without chunks announces, how an error message is extracted) comes from its backend.Part of the trtllm-serve refactor (epic TRTLLM-17056, task TRTLLM-17263: typed return contract and Anthropic convergence). Builds on #21406 (route, middleware-order and shim-identity snapshots and the CPU test seam), #21448 (router skeleton), #21515 (typed chat core), #21518 (Anthropic package) and #21442 (disaggregated entry-point steps).
Behavior
No behavior change. The two servers answered some cases differently before this PR and keep doing so; each row is pinned by
tests/unittest/serve/api/anthropic/test_messages_behavior.py, unchanged from #21518:modelwhen no chunk names onestream_model)message, else the whole body as JSONHTTPException: its detail; returned envelope:message/detail/error/ the whole bodyerror_message)api_error"Internal server error" after SIGINTEngineFatalError, rendered onceharmony_model; the disaggregated frontend keeps its own route)Two theoretical differences of the removed JSON round trip, both unreachable: a chat response that cannot be encoded as JSON (a non-finite float) failed in the chat route's serialization (aggregated 400, disaggregated 500) and would now reach the Anthropic conversion instead; unreachable because the Anthropic request never asks for logprobs, the one remaining float field is a ratio of token counts, and a worker cannot send a non-finite float. A disaggregated success without a body was a 500 from
None.model_dump()and would now read as an engine failure; unreachable because the disaggregated client raises when no body arrives.Route order on an attached frontend of a multi-frontend server (#19394's forwarding): the launcher-owned routes, the five Message Batches routes among them, are forwarded to the launcher and must match before the local handlers. The refactor plan assumed the Anthropic routes conflict with no other route, which is wrong for that configuration: attaching the Anthropic router at the shared
attach_routerscall, which ran before_register_launcher_forwarding_routes(), would have let an attached frontend serve batches from its own store. The forwarding routes now register before the routers, inside the LLM-role block (the same condition as the old branch that registered them), so they keep winning; an A/B of every route's resolution against the base shows 0 differences and the route snapshot is unchanged.RequestContext.from_http(None)raisesValueError(whichopenai_chat'sValueErrorbranch would answer as a 400). Before this PRopenai_chat(request, None)was a supported metrics-skipping call, and the chat core still carries theNoneguards that served it (None if ctx is None else ctx.headers, the early return ofawait_disconnected). No caller passesNonetoday, and the wrapper always builds a context, so those guards are dead code now; they go when the chat core moves into its own serving class (follow-up), not here, so that the core's text stays a pure rename for that move.tensorrt_llm/serve/api/anthropic/protocol.pychanges by one comment line: "(see anthropic_batches)" becomes "(see batches)", the stale reference left by #21518, which could not touch it inside its whole-file move; this PR is the first one that edits the file after it. Nothing else in that module changes.The error contract: both chat backends raise the typed chat core's
ServingError(HTTP status apart from the body) orEngineFatalError; a later PR adds named dialect-neutral error types with per-dialect renderers and switches the backends to them./openapi.json(not a contract): the eight Anthropic paths now appear where the router attaches them (before the role's own table), and the descriptions of/v1/messagesand/v1/messages/count_tokensare the router functions' one-line docstrings instead of the moved methods' docstrings.Moves
Pure-move check
Pure-move check: base 1c031dc, head 3f0a7e0, tools 2e8bb639de1e
ROUTERSServerCapabilitieschat_stream_generatorOpenAIDisaggServer.__init__OpenAIDisaggServer.anthropic_messagesOpenAIServer._synthetic_requestOpenAIServer._batch_storeOpenAIServer._no_such_batchOpenAIServer._run_batched_messageOpenAIServer.anthropic_batch_resultsOpenAIServer.anthropic_cancel_batchOpenAIServer.anthropic_count_tokensOpenAIServer.anthropic_create_batchOpenAIServer.anthropic_delete_batchOpenAIServer.anthropic_get_batchOpenAIServer.anthropic_list_batchesOpenAIServer.anthropic_messagesOpenAIServer.__init__OpenAIServer.create_chat_completionOpenAIServer.openai_chatOpenAIServer.register_routes_anthropic_routes_enabledAnthropicBatchesServing.__init__AnthropicBatchesServingAnthropicCountTokensServing.__init__AnthropicCountTokensServingAnthropicMessagesServing.__init__AnthropicMessagesServing.create_messageAnthropicMessagesServing.run_batched_messageAnthropicMessagesServing_error_typeOpenAIDisaggServer._call_entry_point_DisaggChatBackend.__init___DisaggChatBackend.create_chat_completion_DisaggChatBackend.error_message_DisaggChatBackend.stream_model_DisaggChatBackend_disagg_serving_error_envelope_message_AggregatedChatBackend.__init___AggregatedChatBackend.create_chat_completion_AggregatedChatBackend.error_message_AggregatedChatBackend.stream_model_AggregatedChatBackendReplaced
OpenAIServer.anthropic_messages->tensorrt_llm/serve/api/anthropic/serving.py::AnthropicMessagesServing: one serving class over a ChatBackend, shared by the aggregated and disaggregated servers; route function anthropic_messages in api_router.pyOpenAIServer._run_batched_message->tensorrt_llm/serve/api/anthropic/serving.py::AnthropicMessagesServing: same orchestration as messages, as run_batched_message; no synthetic requestOpenAIServer._synthetic_request->tensorrt_llm/serve/_base/request_context.py::RequestContext: RequestContext.internal replaces the synthetic ASGI scopeAdded
_error_type: status to Anthropic error type, today's ruleAnthropicMessagesServing: new serving class, used by both serversAnthropicMessagesServing.__init__: takes the ChatBackendAnthropicMessagesServing.create_message: the messages orchestration, onceAnthropicMessagesServing.run_batched_message: the batch orchestration, onceAnthropicCountTokensServing: hand-written skeleton class that receives anthropic_count_tokensAnthropicCountTokensServing.__init__: explicit dependencies instead of server attributes, so missing wiring fails at constructionAnthropicBatchesServing: hand-written skeleton class that receives the eight batch methodsAnthropicBatchesServing.__init__: explicit dependencies instead of server attributes, so missing wiring fails at construction_AggregatedChatBackend: ChatBackend over the typed chat core; transitional until the chat handlers move into their own serving class_AggregatedChatBackend.__init__: same_AggregatedChatBackend.create_chat_completion: same_AggregatedChatBackend.stream_model: same_AggregatedChatBackend.error_message: same_envelope_message: today's message extraction for a returned error envelope_disagg_serving_error: a disaggregated failure as a ServingError_DisaggChatBackend: ChatBackend over prepare, run, finish_DisaggChatBackend.__init__: same_DisaggChatBackend.create_chat_completion: same_DisaggChatBackend.stream_model: same_DisaggChatBackend.error_message: sameOpenAIDisaggServer._call_entry_point: prepare, run, finish without serialize, in the disaggregated_request span_anthropic_routes_enabled: the gate of the Anthropic router: not on a harmony (gpt-oss) model, as the route registration was gatedRewritten
OpenAIServer.anthropic_count_tokens(tensorrt_llm/serve/api/anthropic/serving.py):RenderResources.from_server(self)->self._render_resources()Modified (declared)
OpenAIServer.__init__: Anthropic serving objects on app.state; harmony_model capability; the launcher forwarding routes register before the routers so that they keep winning over the local batch routes--- base
+++ head
@@ -164,7 +164,13 @@
detail = detail[:4096] + ' ... [truncated %d bytes]' % (len(detail) - 4096)
return JSONResponse(status_code=400, content={'error': detail})
self.app.state.ops_serving_system = SystemServing(model=self.model, health_probe=self._health_probe)
if self.server_role is ServerRole.VISUAL_GEN:
assert self._is_visual_gen, 'generator must be a VisualGen for VISUAL_GEN server'
self.register_visual_gen_routes()
@@ -176,7 +182,6 @@
self._init_embedding_batcher()
self.register_embedding_routes()
else:
self.app.add_middleware(PerfMetricsMiddleware, expose_headers=self._expose_perf_metrics, writer=self._perf_metrics_writer, adjusted_clock=self._adjusted_steady_clock)
OpenAIServer.register_routes: the Anthropic routes come from serve/api/anthropic/api_router.py--- base
+++ head
@@ -16,15 +16,6 @@
self.resource_governor = None
self.app.add_api_route('/v1/completions', self.openai_completion, methods=['POST'])
self.app.add_api_route('/v1/chat/completions', self.openai_chat if not self.use_harmony else self.chat_harmony, methods=['POST'])
self.app.add_api_route('/v1/responses/{response_id}', self.openai_responses_get_response, methods=['GET'])
self.app.add_api_route('/v1/responses/{response_id}', self.openai_responses_delete_response, methods=['DELETE'])
OpenAIServer.openai_chat: passes RequestContext.from_http(raw_request) to the core--- base
+++ head
@@ -1,6 +1,6 @@
async def openai_chat(self, request: ChatCompletionRequest, raw_request: Request) -> Response:
try:
OpenAIServer.create_chat_completion: parameter raw_request renamed to ctx (a RequestContext); no other change--- base
+++ head
@@ -1,4 +1,4 @@
-async def create_chat_completion(self, request: ChatCompletionRequest, raw_request: Request) -> Union[ChatCompletionResponse, AsyncIterator[str]]:
+async def create_chat_completion(self, request: ChatCompletionRequest, ctx: RequestContext) -> Union[ChatCompletionResponse, AsyncIterator[str]]:
"""Run one chat completion and return a typed result.
@@ -74,13 +74,13 @@
postproc_args.num_prompt_tokens_offset = excluded_tokens
if dynamic_tool_params:
postproc_args.tools = (postproc_args.tools or []) + dynamic_tool_params
@@ -134,24 +134,24 @@
if conversation and conversation[-1].get('content') and (conversation[-1].get('role') == get_role()):
postproc_args.last_message_content = conversation[-1]['content']
postproc_params = PostprocParams(post_processor=chat_stream_post_processor if request.stream else chat_response_post_processor, postproc_args=postproc_args)
OpenAIDisaggServer.__init__: builds the AnthropicMessagesServing--- base
+++ head
@@ -21,6 +21,7 @@
self._ctx_router = self._coordinator.ctx_router
self._gen_router = self._coordinator.gen_router
self._service = OpenAIDisaggregatedService(self._config, self._coordinator, self._create_client, req_timeout_secs=self._req_timeout_secs)
try:
otlp_cfg = config.otlp_config
if otlp_cfg and otlp_cfg.otlp_traces_endpoint:
OpenAIDisaggServer.anthropic_messages: delegates to AnthropicMessagesServing--- base
+++ head
@@ -1,29 +1,3 @@
async def anthropic_messages(self, request: AnthropicMessagesRequest, raw_request: Request) -> Response:
"""Serve Anthropic Messages through the disaggregated chat pipeline."""
ServerCapabilities: harmony_model field, read by the Anthropic router's gate--- base
+++ head
@@ -3,3 +3,4 @@
"""What the composition root decides once at startup; routers are attached from it."""
role: Optional[ServerRole]
role_group: RoleGroup
ROUTERS: the Anthropic RouterSpec joins ROUTERS after ops.system--- base
+++ head
@@ -1 +1 @@
-ROUTERS: tuple[RouterSpec, ...] = (RouterSpec('tensorrt_llm.serve.api.ops.system.api_router', roles=ALL_ROLE_GROUPS),)
+ROUTERS: tuple[RouterSpec, ...] = (RouterSpec('tensorrt_llm.serve.api.ops.system.api_router', roles=ALL_ROLE_GROUPS), RouterSpec('tensorrt_llm.serve.api.anthropic.api_router', roles=frozenset({RoleGroup.LLM}), when=_anthropic_routes_enabled))
chat_stream_generator: keyword parameter raw_request renamed to ctx--- base
+++ head
@@ -1,9 +1,9 @@
-async def chat_stream_generator(promise: RequestOutput, postproc_params: PostprocParams, *, server, raw_request) -> AsyncGenerator[str, None]:
+async def chat_stream_generator(promise: RequestOutput, postproc_params: PostprocParams, *, server, ctx) -> AsyncGenerator[str, None]:
try:
if not server.postproc_worker_enabled:
post_processor, args = (postproc_params.post_processor, postproc_params.postproc_args)
first_response = await anext(promise)
@@ -13,7 +13,7 @@
for pp_res in pp_results:
yield pp_res
yield 'data: [DONE]\n\n'
logger.error(traceback.format_exc())
8 definitions moved verbatim; 0 differences; 1 rewritten (declared); 3 replaced; 23 added; 9 modified (declared); shim exports 0 names