Skip to content

About

Field notes on running LLMs behind an API gateway: cost per task vs price per token, and why agents should not do date math

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

LLM agent field notes

Short write-ups of things we measured while running language models behind an API gateway, with the code to reproduce the arithmetic.

Note What it shows
The model with the lower price per token cost me twice as much per task Two models, same score on 93 checks, five times the output tokens. Includes the 272K-token price step where the result flips.
I gave an agent a 30-day refund policy. It refunded a 46-day-old order three times. A tool-calling agent read the policy and still got the date arithmetic wrong. The fix is validation in the tool, not a better model.

Code:

  • cost_calculator.py — list-price cost per request for the two models, including the long-prompt tier.
  • refund_tool.py — a refund tool that checks the return window and the amount in code.

Both scripts use only the Python standard library.

Disclosure

These notes come from apimodels.app, a multi-model API gateway (one key and one OpenAI-compatible endpoint for image, video, audio and language models). The tests ran on that gateway. Token counts and list prices are reproducible against any endpoint that serves the same models.

The notes were drafted with AI assistance from our own test data.

Limits

Small tests: seven everyday tasks, one run each, except the refund task, which ran three times per model. Nothing here speaks to long, difficult jobs.

License

MIT for the code. The text is CC BY 4.0.

About

Field notes on running LLMs behind an API gateway: cost per task vs price per token, and why agents should not do date math

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages