LLM Billing Proxy

HTTP proxy that sits between end users and LLM servers and bills them per token.

Requirements

• Python >=3.11
• Python packages from pyproject.toml (dependencies)

Quick start

python3 -m venv venv
source venv/bin/activate

python3 -m llmproxy

The SQLite schema is created automatically on first run (and by llmproxyctl). You can pre-create it with sqlite3 db.sqlite < llmproxy/schema.sql, but that step is optional; the schema script is idempotent.

Configuration

See llmproxy/config.toml for an example configuration file. The program will update the list of configured backends, the rate limits and the [provenance] settings (see AI Act provenance marking) from the config file on SIGHUP. SIGHUP also flushes the authentication cache (see Authentication).

Each backend must define type, which decides the endpoints it is served on:

type	Endpoints
chat	/v1/chat/completions, /v1/completions, /v1/messages, /v1/responses
embedding	/v1/embeddings
transcription	/v1/audio/transcriptions
conversion	/v1/files/convert

A request naming a model on an endpoint of another type (e.g. chat completions with an embedding model) is rejected with 404 and code: model_not_supported before it reaches the backend: it is not billed and does not count against rate limits. An unknown model gets 404 with code: model_not_found. On /v1/messages both use the Anthropic error shape (not_found_error). The type is exposed through /v1/models.

Upgrading: the proxy refuses to start while any backend lacks a valid type (a SIGHUP reload keeps the previous backends; llmproxyctl, which loads the same config, refuses to run as well). Versions without this check ignore the key, so add type to every backend in the deployed config before rolling out the new version.

Each backend may define max_model_len, the real context limit of the deployment in tokens (prompt plus completion). This value is exposed through /v1/models so clients can avoid sending requests that exceed the backend limit. It should match the backend deployment setting (for example vLLM --max-model-len), not just the public model card.

A backend may also define model, the real model id (repo name, e.g. meta-llama/Meta-Llama-3-8B-Instruct) the backend expects in place of the client's model field. This is exposed through /v1/models as model_repo -- (null when unset) -- so clients can inspect which actual model is behind each served model name.

pkill -f -HUP 'python3? .*llmproxy'

# or in Docker
docker kill -s=SIGHUP CONTAINER

Monitoring (Prometheus metrics)

The proxy exposes Prometheus-compatible metrics at /metrics by default. This endpoint can be scraped by Prometheus to monitor request volume, error rates, backend latency, and token usage — enabling alerts when traffic spikes or error rates increase.

Configuration

Metrics are controlled by the [metrics] section in config.toml:

[metrics]
enabled = true        # set to false to disable the /metrics endpoint
path = "/metrics"     # path under which metrics are served

Available metrics

Metric	Type	Labels	Description
llmproxy_requests_total	Counter	method, path, status	Total HTTP requests received
llmproxy_request_duration_seconds	Histogram	method, path, status	End-to-end request latency
llmproxy_active_requests	Gauge	—	In-flight requests
llmproxy_backend_requests_total	Counter	model, status	Requests forwarded to backends
llmproxy_backend_duration_seconds	Histogram	model	Backend response latency
llmproxy_backend_errors_total	Counter	model, error_type	Backend errors (timeout, connection, client_error)
llmproxy_tokens_total	Counter	model, type	Tokens processed (prompt, completion, embedding)
llmproxy_audio_seconds_total	Counter	model	Seconds of audio transcribed
llmproxy_file_conversions_total	Counter	model	PDF pages converted to Markdown
llmproxy_user_requests_total	Counter	user, model	Admitted requests per user (api-key comment, else key id) and model
llmproxy_user_active_requests	Gauge	user, model	In-flight requests per user and model — non-zero series are who is connected right now

Quick start (Kubernetes + Prometheus Operator)

If you run Prometheus Operator (kube-prometheus-stack), the full path from deploy to Grafana is:

1. Deploy the new llmproxy version — the /metrics endpoint is active on port 8080 by default.

2. Create a ServiceMonitor so Prometheus Operator automatically starts scraping:

   # Check your Prometheus selector first:
   kubectl get prometheus -A -o jsonpath='{.items[*].spec.serviceMonitorSelector}'
   
   # Apply the ServiceMonitor (adjust the release label to match):
   kubectl apply -f k8s/servicemonitor.yaml

   If llmproxy has no Kubernetes Service, use the PodMonitor instead:

   kubectl apply -f k8s/podmonitor.yaml

3. Verify Prometheus sees the target:

   # Target should appear as "up":
   kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090
   # Open http://localhost:9090/targets → look for "llmproxy"

4. Import the Grafana dashboard:

   • Grafana → Dashboards → New → Import → Upload JSON file
   • Select grafana/dashboard.json
   • Choose your Prometheus datasource
   • The dashboard has 13 panels: request rate, error rate, status codes, active requests, latency percentiles, backend latency/errors, token usage, plus per-user visibility — connected right now (user × model, in-flight), request rate by user, and user × model activity over the last 15 minutes

Prometheus scrape config (standalone, non-Operator)

If you run Prometheus without the Operator, add a scrape job to your prometheus.yml instead:

scrape_configs:
  - job_name: "llmproxy"
    metrics_path: /metrics
    static_configs:
      - targets: ["llm-billing-proxy:8080"]

See examples/prometheus-scrape.yml for a complete example.

Alert rules

Example alert rules for high request volume and high error rates are provided in examples/prometheus-alerts.yml. Key alerts:

• LLMProxyHighRequestRate — warning when request rate exceeds 100 req/s
• LLMProxyVeryHighRequestRate — critical when request rate exceeds 500 req/s
• LLMProxyHighErrorRate — warning when 5xx error rate exceeds 5%
• LLMProxyCriticalErrorRate — critical when 5xx error rate exceeds 20%
• LLMProxyBackendErrors — per-model backend errors
• LLMProxyHighBackendLatency — p95 backend latency above 30s

Grafana dashboard

A pre-built Grafana dashboard is available at grafana/dashboard.json. Import it via Dashboards → New → Import → Upload JSON file.

Testing

python3 -m unittest

Building the image

To build the Docker image, run the following command:

docker build -t ghcr.io/comtegra/llmproxy:master .

Deployment

The following instructions assume that you already have a Docker compose repository and file.

SQLite

1. Create an SQLite database according to llmproxy/schema.sql. This is optional: the proxy creates the schema automatically on first run. To pre-create it (the script is idempotent):

sqlite3 db.sqlite < llmproxy/schema.sql

2. Copy llmproxy/config.toml to your repository. A good relative path would be secrets/llm-billing-proxy-config.toml.
3. Add appropriate entries to your compose file's services and secrets sections. There's a template compose.yml in this repository.
4. Bring the new service up and verify it started correctly (e.g docker compose up llm-billing-proxy).

MongoDB

1. Create a MongoDB user with the following privileges (see section Database below for JS snippets):

• {resource: {db: "cgc", collection: "api_keys"}, actions: ["find"]}
• {resource: {db: "billing", collection: "events_oneoff"}, actions: ["insert"]}

2. Copy llmproxy/config.toml to your repository. A good relative path would be secrets/llm-billing-proxy-config.toml.
3. In the config edit uri in section db so that it includes credentials for the Mongo user (e.g. mongodb://myuser:mypass@host:27017/?authSource=cgc).
4. Add appropriate entries to your compose file's services and secrets sections. There's a template compose.yml in this repository.
5. Bring the new service up and verify it started correctly (e.g docker compose up llm-billing-proxy).

Testing

Sample data can be added to an SQLite database by using sample.sql. Ditto for MongoDB in sample.js.

After loading sample data you may run a query like this:

curl -v -H'Content-Type: application/json' -H'Authorization: Bearer token2' \
    -d'{"messages": [{"role": "system", "content": "You are an assistant."}, {"role": "user", "content": "Write a limerick about python exceptions"}], "model": "llama31-70b", "stream": true}' \
    'http://localhost:8080/v1/chat/completions'

Database

If using SQLite see llmproxy/schema.sql. Otherwise read on.

This program uses MongoDB for authentication and completion logging. The schema is compatible with Comtergra GPU Core. The MongoDB user needs the following privileges:

db.getSiblingDB("cgc").createRole({
  role: "apiKeysReader",
  privileges: [
    {
      resource: {db: "cgc", collection: "api_keys"},
      actions: ["find"],
    },
  ],
  roles: [],
});
db.getSiblingDB("billing").createRole({
  role: "completionBillingWriter",
  privileges: [
    {
      resource: {db: "billing", collection: "events_oneoff"},
      actions: ["insert"],
    },
  ],
  roles: [],
});

db.getSiblingDB("cgc").createUser({
  user: "llm-billing-proxy",
  pwd: "mypass",
  roles: [
    { role: "completionBillingWriter", db: "cgc" },
    { role: "apiKeysReader", db: "billing" },
  ],
});

Authentication

Users authenticate via bearer tokens. Tokens are stored in the database as SHA256 hashes. They can be generated by the following command:

python3 -c 'import hashlib, secrets; print("Token:", t:=secrets.token_urlsafe(64)); print("Hash:", hashlib.sha256(t.encode()).hexdigest())'

If using SQLite see llmproxy/schema.sql. Otherwise read on.

When a user attempts to authenticate, the following query is performed to the cgc.api_keys collection.

{
    "access_level": "LLM",
    "secret": TOKEN-HASH,
    "$or": [
        {"date_expiry": {"$gt": new Date()}},
        {"date_expiry": null},
    ],
}

TOKEN-HASH is replaced with the SHA256 hash of the bearer token.

The documents are required to have an additional field: user_id.

Authentication cache

Successful key lookups are cached in memory for auth_cache_ttl seconds (default 5, set 0 to disable) so that repeated requests — including ones the proxy is about to reject — do not each cost a database round trip.

Key expiry is not affected: the expiry timestamp is re-checked on every request, so an expired key stops working immediately. What the cache does delay is revocation — deleting or disabling a key can take up to auth_cache_ttl seconds to take effect. Send SIGHUP to flush the cache and make a revocation apply at once.

The llmproxy_auth_cache_hits_total and llmproxy_auth_cache_misses_total counters show whether the cache is doing anything.

Completion logging

If using SQLite see llmproxy/schema.sql. Otherwise read on.

When a user performs a completion, two events (prompt token and completion token counts) are inserted into the collection billing.events_oneoff. These documents have the following fields:

• date_created -- date and time when the request finished processing
• user_id -- user_id of the API key
• api_key_id -- id of the API key
• product -- a string in the following format: MODEL/DEVICE/TYPE, where
  • MODEL is the name of the backend
  • DEVICE is the name of the GPU where the model runs
  • TYPE is prompt, completion, embedding, transcription, or conversion (PDF pages for /v1/files/convert)
• quantity -- token count
• request_id -- request ID to correlate prompt/completion counts

EU AI Act provenance marking (Art. 50(2))

Regulation (EU) 2024/1689 Art. 50(2) requires providers of generative AI systems to mark outputs as artificially generated in a machine-readable format, applicable from 2026-08-02. (The AI Omnibus package, provisionally agreed in May 2026, extends the marking deadline to 2026-12-02 for generative systems already on the market before 2026-08-02 -- verify the final enacted text before relying on the extension.) The proxy marks its generative responses at the point of generation:

• Every generative response (streaming and non-streaming) carries the X-AI-Generated: true header.
• Non-streaming responses of /v1/chat/completions, /v1/completions, /v1/messages and /v1/responses additionally carry a machine-readable top-level provenance object:

"provenance": {
    "ai_generated": true,
    "digital_source_type": "http://cv.iptc.org/newscodes/digitalsourcetype/trainedAlgorithmicMedia",
    "legal_basis": "Regulation (EU) 2024/1689, Article 50(2)",
    "generator": "comtegra-llmproxy",
    "generator_version": "1.4.0",
    "model_id": "llama3-8b",
    "request_id": "same UUID as the X-Request-ID header",
    "timestamp": "2026-08-03T12:00:00Z"
}

digital_source_type is the IPTC Digital Source Type vocabulary term for content fully generated by a trained model (the same term C2PA, OpenAI and Meta use), model_id is the public backend alias the client selected and request_id correlates the content with the billing events and logs. Downstream deployers can rely on the field for their own Art. 50(4) disclosure duties and audits; the body field (not the header) is the durable channel -- it flows through SDK response objects and survives persistence. Cross-origin browser clients can read the header too: every response carries Access-Control-Expose-Headers: X-Request-ID, X-AI-Generated.

Endpoint coverage:

• /v1/chat/completions, /v1/completions, /v1/messages, /v1/responses -- body field + header. SSE streams get the HEADER ONLY; stream bodies are forwarded byte-for-byte, because injecting synthetic events breaks strict SDK stream parsers.
• /v1/audio/transcriptions -- header only. A transcript's semantics come from the input audio (the Art. 50(2) carve-out for output that does not substantially alter the input), and the verbose_json body is forwarded byte-for-byte.
• /v1/files/convert -- header only. Converted Markdown is derived from the uploaded PDF; Marker's JSON body (including any provenance object) is forwarded unchanged.
• /v1/embeddings, /v1/models, error responses -- never marked: vectors, metadata and errors are not synthetic content.

Non-streaming marked bodies are re-serialized (values unchanged; numbers are bit-identical for any double-precision JSON consumer). With logprobs enabled this re-serialization touches every float, which costs some CPU at high request rates.

Official OpenAI/Anthropic SDKs in all languages tolerate and preserve the extra field (e.g. completion.model_extra["provenance"] in openai-python). Should a client with a strict, unknown-field-rejecting parser break, disable the marking without a restart -- the emergency kill switch is:

[provenance]
enabled = false  # then send SIGHUP

The generator string is configurable in the same table. Marking is enabled by default when the [provenance] table is absent.