Skip to content

fix: book proxy savings only for accepted generations and label them as estimates - #32

Open
1aifanatic wants to merge 1 commit into
scaledown-team:mainfrom
1aifanatic:fix/savings-accounting
Open

1aifanatic wants to merge 1 commit into
scaledown-team:mainfrom
1aifanatic:fix/savings-accounting

Conversation

@1aifanatic

Copy link
Copy Markdown

Refs #18. This covers the proxy accounting and labelling parts. What's left is listed below.

Problem

  • server.ts sent both /v1/messages and /v1/messages/count_tokens through addSaving/addRequest, and it did so before the upstream call. Count-only preflights and failed or rejected requests therefore raised the "saved" total.
  • stats.addSaving dropped non-positive values and ignored the transform's netSavingsUsd, so a fold that cost more than it saved never reduced the total.
  • The status line showed ↓ X saved, which reads like measured billing.

Changes

  • Booking happens after upstream. The proxy books savings and request counts only for /v1/messages, and only when the upstream response is 2xx. count_tokens is never booked. The stats write sits in its own try so it can't break a request.
  • Signed net estimate. addSaving(sessionId, tokens, netUsd?) keeps the estimated token reduction (totalSaved, positive only) separate from a new estimatedNetUsd, which accumulates with its sign.
  • Status line reads ↓ ~X tokens saved (est.) and, when present, est. net $… (negative values included).
  • README has a new "How savings are estimated" section. It covers the ~4 chars/token estimate, the fixed $0.30/MTok cache-read and $0.05/MTok ScaleDown rates, that it isn't model-specific, the booking rules, and a pointer to Anthropic usage for real spend.

Tests

  • src/proxy/server-accounting.test.ts (stats module mocked):

    • a successful generation is booked with a positive token count and net dollars;
    • three count_tokens calls book nothing;
    • an upstream 529 books nothing, and the 529 is still passed through to the client;
    • a fold whose ScaleDown cost exceeds its benefit books 0 tokens and a negative net dollar figure.

    All four fail against the current server.ts and pass with this change.

  • src/stats.test.ts: the signed accumulation stays separate from the token total, and non-finite or empty inputs are ignored.

node --experimental-vm-modules node_modules/jest/bin/jest.js src/proxy/server-accounting.test.ts src/stats.test.ts
Tests: 6 passed, 6 total

Both files mock os.homedir(). The existing proxy tests set process.env.HOME instead, which jest's ESM sandbox doesn't pass to os.homedir(), so those tests touch the real ~/.scaledown.

Not in this PR (remaining for #18)

  • Model-specific pricing. Rates are still fixed constants. They are now stated in the README and labelled as estimates, but not looked up per model.
  • Hook-side savings. These are still booked before confirming that the harness applied the replacement. That depends on the hook output contract, which is Align Claude Code hooks with the documented input/output contract #15.
  • Compaction triggered by count_tokens. A count_tokens call can still trigger a ScaleDown fold. It is no longer booked as a saving.
  • Overlap with Fix/proxy fold #23. That PR also edits server.ts (session ids). The conflict is small and confined to the addSaving/addRequest lines.

🤖 Generated with Claude Code

…as estimates

- Book savings/requests only for /v1/messages, never count_tokens, and only
  after the upstream returned 2xx (a rejected or failed request realised no
  saving). Stats writes are wrapped so they can't fail a request.
- addSaving() takes the signed net-dollar estimate and accumulates it as
  estimatedNetUsd, so folds that cost more than they saved now offset
  favourable ones instead of being dropped.
- Status line reads "↓ ~X tokens saved (est.)" plus "est. net $…", and the
  README documents the estimation assumptions (char-based tokens, fixed
  cache-read and ScaleDown rates).

Refs scaledown-team#18

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant