From 59bbeaedcf2cb310defcd050530d372c471ab003 Mon Sep 17 00:00:00 2001 From: GitHub Action Date: Mon, 26 Oct 2026 12:10:29 +0000 Subject: [PATCH 1/3] [Automated] Draft docs (agentgateway): llm: account for Anthropic usage iterations (+2 related) Signed-off-by: GitHub Action --- assets/agw-docs/pages/agentgateway/llm/providers/anthropic.md | 2 ++ assets/agw-docs/standalone/cost-catalog.md | 2 ++ .../standalone/main/documentation/llm/api-types/messages.md | 1 + content/docs/standalone/main/release-notes/release-notes.md | 1 + 4 files changed, 6 insertions(+) diff --git a/assets/agw-docs/pages/agentgateway/llm/providers/anthropic.md b/assets/agw-docs/pages/agentgateway/llm/providers/anthropic.md index bc56ec668..2bf865f75 100644 --- a/assets/agw-docs/pages/agentgateway/llm/providers/anthropic.md +++ b/assets/agw-docs/pages/agentgateway/llm/providers/anthropic.md @@ -5,6 +5,8 @@ Configure [Anthropic (Claude)](https://claude.ai/login) as an LLM provider in {{ > Anthropic excludes cached tokens from the input count that it reports. The CEL field `llm.inputTokens` adds them back, so telemetry, metrics, and token-based limits count a cache-heavy request higher than the number that Anthropic reports. To read the Anthropic number itself, use `llm.providerInputTokens`. For more information, see [Token usage fields]({{< link-hextra path="/documentation/llm/observability/#token-usage-fields" >}}). > > {{< version exclude-if="1.0.x,1.1.x,1.2.x,1.3.x,1.4.x,1.5.x,2.2.x" >}}Client response bodies use the usage convention for the API format that the client called. A `/v1/messages` response uses Anthropic usage fields. In that response, `usage.input_tokens` excludes prompt-cache tokens and cache counts appear separately. A `/v1/chat/completions` response uses OpenAI usage fields, and `usage.prompt_tokens` includes prompt-cache tokens.{{< /version >}} +> +> {{< version exclude-if="1.0.x,1.1.x,1.2.x,1.3.x,1.4.x,1.5.x,1.6.x,2.2.x" >}}When Anthropic reports `usage.iterations`, the `/v1/messages` response keeps that array. Telemetry, cost calculations, and converted Chat Completions responses use the sum of the iteration entries.{{< /version >}} {{< /version >}} {{< version exclude-if="1.0.x,1.1.x,1.2.x,1.3.x,1.4.x,1.5.x,1.6.x,2.2.x" >}} diff --git a/assets/agw-docs/standalone/cost-catalog.md b/assets/agw-docs/standalone/cost-catalog.md index 61e020978..3f101a245 100644 --- a/assets/agw-docs/standalone/cost-catalog.md +++ b/assets/agw-docs/standalone/cost-catalog.md @@ -187,6 +187,8 @@ When a request matches an entry in the catalog, agentgateway populates these CEL {{< version exclude-if="1.0.x,1.1.x,1.2.x,1.3.x,1.4.x,1.5.x,1.6.x,2.2.x" >}}For Anthropic prompt-cache writes with a 1-hour time to live (TTL), `llm.cost.cacheWrite1h` reports the 1-hour cache-write cost. `llm.cost.cacheWrite` reports the cost for cache writes that do not use a 1-hour TTL, such as 5-minute cache writes. When the catalog includes `cacheWrite1h`, `llm.costRates.cacheWrite1h` reports the effective rate.{{< /version >}} +{{< version exclude-if="1.0.x,1.1.x,1.2.x,1.3.x,1.4.x,1.5.x,1.6.x,2.2.x" >}}When an Anthropic Messages response reports `usage.iterations`, the request cost uses a separate calculation for each iteration. If an iteration names a `model` with an exact catalog match, the tokens for that iteration use that model's catalog entry. Otherwise, the iteration uses the response model. The request cost is the sum of those iteration costs.{{< /version >}} + {{< version exclude-if="1.5.x" >}}For page-billed document models, `llm.cost.pages` reports the page-cost component, and `llm.costRates.perPage` reports the USD-per-page rate that was applied.{{< /version >}} The request access log includes `agw.ai.usage.cost.total` only for priced LLM requests. When the model cannot be priced, the access log leaves out the field. diff --git a/content/docs/standalone/main/documentation/llm/api-types/messages.md b/content/docs/standalone/main/documentation/llm/api-types/messages.md index 3e1c301b7..97514c28f 100644 --- a/content/docs/standalone/main/documentation/llm/api-types/messages.md +++ b/content/docs/standalone/main/documentation/llm/api-types/messages.md @@ -88,6 +88,7 @@ These behaviors apply to both the Responses and the Chat Completions conversion. - A function call with empty arguments becomes a `tool_use` block whose `input` is `{}`. When the reply reaches the output token limit partway through the arguments, `stop_reason` is `max_tokens`, and a buffered reply returns the partial arguments in `input` as a string instead of an object. Check `stop_reason` before you parse `input`. - When the provider rejects a prompt that is longer than the model's context window, the converted error message starts with `capability_rejected: prompt_too_long`, followed by the original message. Clients such as Claude Code use this marker to compact the prompt and retry. The marker is added only to an HTTP 400 error that has the `context_length_exceeded` code, or that has no code and a message that says the prompt exceeds the context window. An error that already contains `capability_rejected:` keeps its message. A Gemini or Vertex AI provider returns errors in the Google format, which does not get the marker. - Response usage follows Anthropic Messages conventions. The `usage.input_tokens` field excludes prompt-cache tokens. When the provider returns them, `usage.cache_read_input_tokens` and `usage.cache_creation_input_tokens` report cache tokens separately. When a Gemini OpenAI-compatible response reports reasoning tokens separately from `usage.completion_tokens`, the converted `usage.output_tokens` value includes visible output and reasoning tokens. +- When an upstream Anthropic Messages response contains `usage.iterations`, the client response keeps the per-iteration array. Token usage for telemetry, costs, and Chat Completions conversions uses the sum of the iteration entries, including cache tokens. If an iteration names another `model` with an exact catalog match, that iteration uses the catalog entry for that model. Otherwise, the iteration uses the response model. ### Converting to the Responses format diff --git a/content/docs/standalone/main/release-notes/release-notes.md b/content/docs/standalone/main/release-notes/release-notes.md index 4924aaaa1..196545273 100644 --- a/content/docs/standalone/main/release-notes/release-notes.md +++ b/content/docs/standalone/main/release-notes/release-notes.md @@ -107,6 +107,7 @@ For more information, see [Session key]({{< link-hextra path="/integrations/mcp/ **LLM** +- Anthropic usage accounting includes every `usage.iterations` entry, including compaction and cache-read tokens, when calculating telemetry, costs, and converted Chat Completions usage, and Messages requests always prefer Responses when a provider supports both Responses and Chat Completions because the deprecated `AGENTGATEWAY_MESSAGES_PREFER_COMPLETIONS` override is removed. - OpenAI-compatible responses, such as from Gemini, count reasoning tokens as output usage when agentgateway converts the response to Messages or Responses and when agentgateway records normalized usage. - Messages requests that ask for reasoning no longer send reasoning parameters to older OpenAI chat models whose IDs start with `gpt-3.5`, `gpt-4`, or `chatgpt-4o`. - Bedrock conversion keeps prompt-cache markers when unsupported Messages blocks or blank Chat Completions text are dropped, and Anthropic cache TTLs reach Bedrock cache points. From 42ae9205ea14ce9942c342edcad33b755bbf606f Mon Sep 17 00:00:00 2001 From: Art Berger Date: Mon, 26 Oct 2026 10:22:51 -0400 Subject: [PATCH 2/3] move release notes to #1351 Signed-off-by: Art Berger --- content/docs/standalone/main/release-notes/release-notes.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/docs/standalone/main/release-notes/release-notes.md b/content/docs/standalone/main/release-notes/release-notes.md index 196545273..4924aaaa1 100644 --- a/content/docs/standalone/main/release-notes/release-notes.md +++ b/content/docs/standalone/main/release-notes/release-notes.md @@ -107,7 +107,6 @@ For more information, see [Session key]({{< link-hextra path="/integrations/mcp/ **LLM** -- Anthropic usage accounting includes every `usage.iterations` entry, including compaction and cache-read tokens, when calculating telemetry, costs, and converted Chat Completions usage, and Messages requests always prefer Responses when a provider supports both Responses and Chat Completions because the deprecated `AGENTGATEWAY_MESSAGES_PREFER_COMPLETIONS` override is removed. - OpenAI-compatible responses, such as from Gemini, count reasoning tokens as output usage when agentgateway converts the response to Messages or Responses and when agentgateway records normalized usage. - Messages requests that ask for reasoning no longer send reasoning parameters to older OpenAI chat models whose IDs start with `gpt-3.5`, `gpt-4`, or `chatgpt-4o`. - Bedrock conversion keeps prompt-cache markers when unsupported Messages blocks or blank Chat Completions text are dropped, and Anthropic cache TTLs reach Bedrock cache points. From a7ca0a5e7d0f459cf486fd976985944875d84dac Mon Sep 17 00:00:00 2001 From: Art Berger Date: Mon, 26 Oct 2026 12:33:37 -0400 Subject: [PATCH 3/3] docs: move usage.iterations pricing to the K8s cost page - Remove the messages.md bullet. That list covers Messages requests converted to OpenAI formats, and iterations come from an Anthropic upstream. - Add per-iteration pricing to the K8s costs page (catalog/mod.rs prices each usage line at its own model on an Exact match). - Gate the new asset text with upstream version tokens only. Signed-off-by: Art Berger --- assets/agw-docs/pages/agentgateway/llm/providers/anthropic.md | 2 +- assets/agw-docs/standalone/cost-catalog.md | 2 +- .../kubernetes/main/documentation/llm/cost-controls/costs.md | 2 ++ .../standalone/main/documentation/llm/api-types/messages.md | 1 - 4 files changed, 4 insertions(+), 3 deletions(-) diff --git a/assets/agw-docs/pages/agentgateway/llm/providers/anthropic.md b/assets/agw-docs/pages/agentgateway/llm/providers/anthropic.md index 2bf865f75..dc79db5e0 100644 --- a/assets/agw-docs/pages/agentgateway/llm/providers/anthropic.md +++ b/assets/agw-docs/pages/agentgateway/llm/providers/anthropic.md @@ -6,7 +6,7 @@ Configure [Anthropic (Claude)](https://claude.ai/login) as an LLM provider in {{ > > {{< version exclude-if="1.0.x,1.1.x,1.2.x,1.3.x,1.4.x,1.5.x,2.2.x" >}}Client response bodies use the usage convention for the API format that the client called. A `/v1/messages` response uses Anthropic usage fields. In that response, `usage.input_tokens` excludes prompt-cache tokens and cache counts appear separately. A `/v1/chat/completions` response uses OpenAI usage fields, and `usage.prompt_tokens` includes prompt-cache tokens.{{< /version >}} > -> {{< version exclude-if="1.0.x,1.1.x,1.2.x,1.3.x,1.4.x,1.5.x,1.6.x,2.2.x" >}}When Anthropic reports `usage.iterations`, the `/v1/messages` response keeps that array. Telemetry, cost calculations, and converted Chat Completions responses use the sum of the iteration entries.{{< /version >}} +> {{< version exclude-if="1.6.x,1.5.x,1.4.x,1.3.x,1.2.x,1.1.x,1.0.x" >}}When Anthropic reports `usage.iterations`, the `/v1/messages` response keeps that array. Telemetry, cost calculations, and converted Chat Completions responses use the sum of the iteration entries.{{< /version >}} {{< /version >}} {{< version exclude-if="1.0.x,1.1.x,1.2.x,1.3.x,1.4.x,1.5.x,1.6.x,2.2.x" >}} diff --git a/assets/agw-docs/standalone/cost-catalog.md b/assets/agw-docs/standalone/cost-catalog.md index 3f101a245..68286e8e1 100644 --- a/assets/agw-docs/standalone/cost-catalog.md +++ b/assets/agw-docs/standalone/cost-catalog.md @@ -187,7 +187,7 @@ When a request matches an entry in the catalog, agentgateway populates these CEL {{< version exclude-if="1.0.x,1.1.x,1.2.x,1.3.x,1.4.x,1.5.x,1.6.x,2.2.x" >}}For Anthropic prompt-cache writes with a 1-hour time to live (TTL), `llm.cost.cacheWrite1h` reports the 1-hour cache-write cost. `llm.cost.cacheWrite` reports the cost for cache writes that do not use a 1-hour TTL, such as 5-minute cache writes. When the catalog includes `cacheWrite1h`, `llm.costRates.cacheWrite1h` reports the effective rate.{{< /version >}} -{{< version exclude-if="1.0.x,1.1.x,1.2.x,1.3.x,1.4.x,1.5.x,1.6.x,2.2.x" >}}When an Anthropic Messages response reports `usage.iterations`, the request cost uses a separate calculation for each iteration. If an iteration names a `model` with an exact catalog match, the tokens for that iteration use that model's catalog entry. Otherwise, the iteration uses the response model. The request cost is the sum of those iteration costs.{{< /version >}} +{{< version exclude-if="1.6.x,1.5.x,1.4.x,1.3.x,1.2.x,1.1.x,1.0.x" >}}When an Anthropic Messages response reports `usage.iterations`, the request cost uses a separate calculation for each iteration. If an iteration names a `model` with an exact catalog match, the tokens for that iteration use that model's catalog entry. Otherwise, the iteration uses the response model. The request cost is the sum of those iteration costs.{{< /version >}} {{< version exclude-if="1.5.x" >}}For page-billed document models, `llm.cost.pages` reports the page-cost component, and `llm.costRates.perPage` reports the USD-per-page rate that was applied.{{< /version >}} diff --git a/content/docs/kubernetes/main/documentation/llm/cost-controls/costs.md b/content/docs/kubernetes/main/documentation/llm/cost-controls/costs.md index 63959256b..35c960fbd 100644 --- a/content/docs/kubernetes/main/documentation/llm/cost-controls/costs.md +++ b/content/docs/kubernetes/main/documentation/llm/cost-controls/costs.md @@ -196,6 +196,8 @@ When a request matches an entry in the catalog, {{< reuse "agw-docs/snippets/age For Anthropic prompt-cache writes with a 1-hour time to live (TTL), `llm.cost.cacheWrite1h` reports the 1-hour cache-write cost, and `llm.cost.cacheWrite` reports the cost of the other cache writes, such as 5-minute cache writes. When the catalog sets `cacheWrite1h`, `llm.costRates.cacheWrite1h` reports the effective rate. Traces record the 1-hour component as `agw.ai.usage.cost.cache_write_1h`. +When an Anthropic Messages response reports `usage.iterations`, {{< reuse "agw-docs/snippets/agentgateway.md" >}} prices each iteration separately and adds the results to get the request cost. If an iteration names a `model` with an exact catalog match, the tokens for that iteration use that model's catalog entry. Otherwise, the iteration uses the response model. + The request access log includes `agw.ai.usage.cost.total` only for priced LLM requests. When the model cannot be priced, which is a lookup status of `Unpriced` or `Missing` as described in the next step, the access log leaves out the field. For how to view logs and add cost fields, see [Metrics and logs]({{< link-hextra path="/documentation/llm/observability/" >}}). ### Service tiers {#service-tiers} diff --git a/content/docs/standalone/main/documentation/llm/api-types/messages.md b/content/docs/standalone/main/documentation/llm/api-types/messages.md index 97514c28f..3e1c301b7 100644 --- a/content/docs/standalone/main/documentation/llm/api-types/messages.md +++ b/content/docs/standalone/main/documentation/llm/api-types/messages.md @@ -88,7 +88,6 @@ These behaviors apply to both the Responses and the Chat Completions conversion. - A function call with empty arguments becomes a `tool_use` block whose `input` is `{}`. When the reply reaches the output token limit partway through the arguments, `stop_reason` is `max_tokens`, and a buffered reply returns the partial arguments in `input` as a string instead of an object. Check `stop_reason` before you parse `input`. - When the provider rejects a prompt that is longer than the model's context window, the converted error message starts with `capability_rejected: prompt_too_long`, followed by the original message. Clients such as Claude Code use this marker to compact the prompt and retry. The marker is added only to an HTTP 400 error that has the `context_length_exceeded` code, or that has no code and a message that says the prompt exceeds the context window. An error that already contains `capability_rejected:` keeps its message. A Gemini or Vertex AI provider returns errors in the Google format, which does not get the marker. - Response usage follows Anthropic Messages conventions. The `usage.input_tokens` field excludes prompt-cache tokens. When the provider returns them, `usage.cache_read_input_tokens` and `usage.cache_creation_input_tokens` report cache tokens separately. When a Gemini OpenAI-compatible response reports reasoning tokens separately from `usage.completion_tokens`, the converted `usage.output_tokens` value includes visible output and reasoning tokens. -- When an upstream Anthropic Messages response contains `usage.iterations`, the client response keeps the per-iteration array. Token usage for telemetry, costs, and Chat Completions conversions uses the sum of the iteration entries, including cache tokens. If an iteration names another `model` with an exact catalog match, that iteration uses the catalog entry for that model. Otherwise, the iteration uses the response model. ### Converting to the Responses format