Skip to content

assertUsageDisjointness covers 6 of 16 AI providers — xAI and llamacpp-server map more usage fields than Ollama, which is covered #942

Description

@sroussey

What

assertUsageDisjointness (packages/test/src/contract/ai-provider/assertions/usageNormalization.ts:91) is the assertion the 2026-08-17 review credited as making Usage.input disjoint "with a generic contract test … so a new provider inherits the contract by construction". It asserts input + cached + cacheWrite + output === total for every case of every provider block.

It is invoked through usageNormalizationBlock, which packages/test/src/test/ai-provider/ProviderUsageNormalization.test.ts calls seven times, for six providers:

$ grep -n "provider:" packages/test/src/test/ai-provider/ProviderUsageNormalization.test.ts
39:  provider: "OpenAI (chat-completions shape)",
86:  provider: "OpenAI (Responses shape)",
115: provider: "Anthropic",
163: provider: "Gemini",
192: provider: "Ollama",
220: provider: "DeepSeek",
249: provider: "OpenRouter",

Not covered, with how much usage mapping each actually carries in src:

$ for p in providers/*/; do echo "$(basename $p) $(grep -ric usage $p/src | awk -F: '{s+=$2} END{print s+0}')"; done
xai 61                      ← not covered
llamacpp-server 58          ← not covered
huggingface-inference 44    ← not covered
node-llama-cpp 36           ← not covered
huggingface-transformers 19 ← not covered
chrome-ai 2                 ← not covered
cactus 0 · tf-mediapipe 0 · stable-diffusion-server 0 · mlx 0
--- covered: openrouter 79 · anthropic 73 · deepseek 72 · google-gemini 66 · ollama 61 · openai 54

xai (61) and llamacpp-server (58) each touch more usage code than ollama (61, tied) — and ollama got a block.

Why it matters

  • xAI is OpenAI-compatible in shape, which is exactly the assumption the disjointness contract exists to stop people making silently: OpenAI's prompt_tokens includes cached and Anthropic's input_tokens excludes it, and the whole point of the axis is that the same-looking field means different things per vendor. Xai_Usage.ts is unasserted on that question.
  • llamacpp-server, node-llama-cpp and huggingface-transformers report locally-computed usage with no vendor to check against, which is the case most likely to double-count. createDecodeUsageReporter and createEstimatedOutputUsageReporter both feed cumulative snapshots; nothing asserts the snapshot is disjoint.
  • Four providers report no usage at all (cactus, tf-mediapipe, stable-diffusion-server, mlx), which is its own finding: stable-diffusion-server has had grep -rc usage src → 0 for six consecutive review cycles behind a 125-line README that does not mention it.

Proposed fix

  1. Add blocks for xai and llamacpp-server first — highest mapping surface, lowest effort, and the fixtures are the same shape as the OpenAI and Ollama blocks already in the file.
  2. Then node-llama-cpp and huggingface-transformers, whose numbers are computed rather than received.
  3. Make the omission visible rather than silent: drive ProviderUsageNormalization.test.ts off a record keyed by provider name with an explicit usageMapping: "none" entry for the four that report nothing, so a provider that starts reporting usage without a block is a type error. Same shape as PROVIDER_CASES / PRICED_CASES in inferAdvertisesRegistered.test.ts, which already pins the analogous gap for capabilities and pricing.
  4. For the four with zero usage, either wire it or say so in the README — stable-diffusion-server in particular.

Found during the 2026-09-14 review of providers/. Verified against origin/main @ 2d36880.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions