feat(aio): price ai generations by their served service tier by bernatixer · Pull Request #94200 · PostHog/posthog

bernatixer · 2026-09-03T13:30:25Z

Problem

A generation served on OpenAI's flex tier is billed at half the standard token rates, and one served on priority at double — but ingestion prices both at standard. Flex dashboards overstate cost 2×, priority ones understate it 2×.
The price book already carries the real tier prices: the twice-daily OpenRouter sync lands per-model openai-flex (0.5×) and openai-fast (2×) rows in llm-costs.json, as first-party OpenAI routes with OpenRouter discounts stripped. Nothing selects them — resolution only ever reads the plain openai key.

Changes

Events now price from the served tier's own synced row: resolution probes the provider-literal tier key first (openai-flex, google-ai-studio-priority), then -fast (OpenAI's and Anthropic's name for priority), falling back to the standard key when the row has no tier variant. A customer's or internal team's $ai_total_cost_usd matches what the provider actually bills, and future tier price changes arrive through the normal sync PRs, never a code change.
The served tier is read only from the explicit $ai_service_tier property, whose writers (the ai-gateway in PostHog/ai-gateway#451, SDK versions from fix(ai): only record a service tier the provider actually served posthog-js#4818 and feat(ai): emit the served service tier as the explicit $ai_service_tier property posthog-python#932 on) assert response-derived values — on success and error events alike, so a flex stream that dies mid-way still prices its partial tokens at the flex rate it was billed. $ai_model_parameters.service_tier never prices: released SDKs wrote the requested tier there on successful events, and a request can be refused.
Eligibility comes from the book, not an allowlist: a model without a flex row (or a junk tier value on any model) prices at its standard row unchanged.
The tier lookup runs as a strict key check ahead of the resolver's fallback cascade, which ends at the default key — a key that can carry promotional pricing (gpt-5.6-sol today), so a missing tier row must never reach it.
Visible in event properties: $ai_cost_model_provider now reads openai-flex / openai-fast on tiered events, so which row priced a call is queryable.
Known gaps, accepted: gpt-5, o3, and o4-mini have no synced flex row yet, so their flex calls price at standard (overstated, self-corrects when the sync lists them); Python-SDK internal traffic does not record the served tier yet, so it keeps pricing at standard until that emitter lands (tracked separately).

How did you test this code?

New tests against the mocked catalog, one per regression: a served flex tier picks the flex row (including its undiscounted per-search fee, which a multiplier could not express); a served priority tier picks the fast row (the 2× underreport); a missing tier key falls back to the standard row and never the promo-capable default; non-OpenAI providers and auto/default/absent tiers are untouched; request-side tier properties are ignored.
Not run: an end-to-end ingestion of a flex event through capture.
Found while verifying, out of scope here: dated snapshot names mismatch in the model matcher (gpt-5-2025-08-07 resolves to gpt-5.2) — pre-existing, fix incoming as its own PR.

Automatic notifications

Publish to changelog?

Docs update

None: no documented behavior names the ingestion tier pricing yet; the $ai_service_tier taxonomy entry ships alongside the emitters (gateway and SDK PRs linked above).

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Claude Code session directed by the assignee. The design followed the review: a 0.5× multiplier shipped first (the book lacked flex rows for most models at the time); reviewer feedback (resolve by key, don't halve web_search, price the served tier, priority underreports) plus the sync having since filled the gpt-5.x line converged on tier-key resolution.
Skills invoked: /writing-pr-descriptions, /writing-tests, /writing-code-comments.

Events carrying $ai_service_tier=flex resolve to <model>:flex catalog rows, added manually from OpenAI's flex pricing page. Models without a flex row fall back to the standard row via the existing containment match, so unknown tiers can never break pricing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

trunk-io · 2026-09-03T13:30:29Z

😎 Merged successfully - details.

github-actions · 2026-09-03T13:33:23Z

🤖 CI report

✅ Trunk lane — non-backend lane (node:ingestion)

This PR is assigned to the non-backend lane (node:ingestion). It does not run backend Python tests and may merge in parallel with PRs in other lanes.

🚨 Comment density — 19% of added code lines are comments (23 of 118)

This section warns when comments are more than 3% of the code lines a PR adds, and alerts above 6%. Before agent-assisted PRs, the typical share was about 2%. Only full-line comments count. Docstrings, generated files, snapshots, migrations, and workflow files are left out.

Comments that restate the code, record how the change came about, or narrate the next line add noise for the next reader. Keep the comments that explain a reason the code cannot show, and remove the rest. See .agents/skills/writing-code-comments/SKILL.md for the house rules.

Files with the most added comment lines:

File	Comment lines	Added lines
`nodejs/src/ingestion/pipelines/ai/costs/cost-model-matching.test.ts`	12	76
`nodejs/src/ingestion/pipelines/ai/costs/provider-matching.ts`	7	26
`nodejs/src/ingestion/pipelines/ai/costs/cost-model-matching.ts`	4	16

This check does not block merging. It updates on every push and clears when the share drops.

ℹ️ Bundle size — no base branch to compare

Uncompressed size of every built .js bundle, compared against the base branch.

Total: 68.40 MiB (no base branch measurement to compare against yet)

_{Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report}

✅ Eager graph — within budget

How much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy import() / React.lazy chunks are not counted.

Root	Eager (shipped)	Δ vs base	Budget
entry (logged-out pages, app bootstrap) `src/index.tsx`	1.45 MiB · 22 files	(no base measurement)	`███░░░░░░░` 32.3% of 4.51 MiB
authenticated shell (every logged-in page) `src/scenes/AuthenticatedShell.tsx`	8.89 MiB · 3,265 files	(no base measurement)	`█████████░` 91.5% of 9.71 MiB

🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx

Largest files eagerly shipped from src/index.tsx

Size	File
126.8 KiB	`../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js`
24.6 KiB	`../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js`
6.3 KiB	`../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js`
4.5 KiB	`../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js`
3.9 KiB	`../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js`
1.4 KiB	`../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js`
1.3 KiB	`src/RootErrorBoundary.tsx`
912 B	`../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js`
854 B	`src/scenes/ChunkLoadErrorBoundary.tsx`
762 B	`src/index.tsx`

Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx

Size	File
307.3 KiB	`../node_modules/.pnpm/posthog-js@1.425.1_@types+react@18.3.27_react@18.3.1/node_modules/posthog-js/dist/rrweb.js`
272.4 KiB	`../node_modules/.pnpm/posthog-js@1.425.1_@types+react@18.3.27_react@18.3.1/node_modules/posthog-js/dist/module.js`
267.7 KiB	`../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js`
255.5 KiB	`src/taxonomy/core-filter-definitions-by-group.json`
154.2 KiB	`../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js`
126.8 KiB	`../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js`
104.5 KiB	`src/lib/api.ts`
95.8 KiB	`../packages/quill/packages/quill/dist/index.js`
93.3 KiB	`../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js`
90.6 KiB	`../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js`

_{Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479}

✅ Toolbar bundle — eager 2.28 MiB within budget

What the toolbar ships to customer pages, measured from the esbuild output (minified, post-tree-shake). The eager set is the entry plus everything statically imported from it — fetched before any feature runs; deferred chunks load lazily. The eager guardrail is 5.72 MiB. Each output file must also stay below 10 MB, where CloudFront stops compressing it. The module boundary is enforced separately by check-toolbar-graph.

Metric	Size	Δ vs base	Budget
Eager (shipped) entry + static imports	2.28 MiB · 18 files	(no base measurement)	`████░░░░░░` 39.8% of 5.72 MiB
Deferred (lazy)	2.11 MiB · 45 files	(no base measurement)	n/a — loads on demand
Loader `dist/toolbar.js`	1.1 KiB	(no base measurement)	`█░░░░░░░░░` 5.8% of 19.5 KiB

Largest eagerly-shipped chunks

Size	File
754.0 KiB	`dist/toolbar/toolbar-app-A5TLLDAI.css`
599.3 KiB	`dist/toolbar/chunk-chunk-QJTB64BW.js`
484.7 KiB	`dist/toolbar/chunk-chunk-RFL5H2T4.js`
135.6 KiB	`dist/toolbar/chunk-chunk-2CB4DIAM.js`
131.8 KiB	`dist/toolbar/chunk-chunk-FDH2IBXT.js`
71.3 KiB	`dist/toolbar/toolbar-app-NL3DAYKQ.js`
69.0 KiB	`dist/toolbar/chunk-chunk-TSAL54PB.js`
35.6 KiB	`dist/toolbar/chunk-chunk-QYMJML4V.js`
21.0 KiB	`dist/toolbar/chunk-chunk-QRJT6XOH.js`
6.8 KiB	`dist/toolbar/chunk-chunk-DV7IWQNF.js`

_{Posted automatically by check-toolbar-size · sizes are toolbar output bytes (shipped, post-tree-shake) from the esbuild metafile}

ℹ️ Dist folder size — no base branch to compare

Total size of the built frontend/dist folder (all assets), compared against the base branch.

Total: 1460.23 MiB (no base branch measurement to compare against yet)

The OpenRouter endpoints sync already lands flex rates as <provider>-flex cost keys on the base model rows, so the manual :flex rows and model-name remap go: flex events resolve the openai-flex key and OpenRouter tracks price changes for us. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

trunk-io · 2026-09-03T14:10:39Z

_{View Full Report ↗︎ ⋅ Docs}

…ariant keys Applying OpenAI's uniform flex rule ourselves keeps pricing independent of how OpenRouter models service tiers, per team discussion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@posthog/ai records the provider's served service_tier inside $ai_model_parameters, so the flex pricing applies to existing SDK traffic without waiting for a dedicated property emitter. An explicit $ai_service_tier still wins when both are present. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The tier discount does not depend on which path matched the model, so the matching code returns to master's shape and one function halves the token rates on the final result. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The pipeline already promotes model parameters to top-level properties; service_tier joins them, so flex pricing reads one flat $ai_service_tier and the tier becomes a stored, queryable property. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Reverts the service_tier promotion into extractCoreModelParams: no pipeline reorder, no new stored property, no taxonomy or frontend changes. The pricing function reads the fields the event already carries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

OpenAI's flex table is 50% across the board, so the per-field list goes; the only deviation is the undiscounted web-search tool fee, accepted at half a cent per search. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Resolution maps a served flex/priority tier to the book's openai-flex / openai-fast row, so tier prices come from the twice-daily OpenRouter sync instead of a hardcoded multiplier. A model without the tier key prices at its standard row, which also gates eligibility. Request-side tier properties are no longer read. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Provider-key selection lives in one function: the served tier resolves by its own key ahead of the existing cascade. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

richardsolomou · 2026-09-07T07:39:38Z


-    const provider: string | undefined = providerProperty ? String(providerProperty).toLowerCase() : undefined
+    return modelParameters && typeof modelParameters === 'object'
+        ? (modelParameters as Record<string, unknown>)['service_tier']


can we only use a tier with served-side provenance? the @posthog/ai error paths omit the accumulated response tier, so getModelParams keeps the requested tier alongside any partial usage. ingestion can then price those tokens at the wrong tier.

Added a guard on ingestion for this case and also opened a PR on the JS SDK about it.

i still dont think we can treat this field as served-side without an explicit signal. released LangChain clients copy the requested tier into run.modelParams on successful events and never merge the response, so a refused flex request has no $ai_is_error and still gets priced as flex.

Makes sense, changed the code to use $ai_service_tier, also updated SDKs on it, though I will wait to merge this PR first before merging SDKs.

SDK error paths capture the requested tier with partial usage; a tier that was never confirmed served must not discount those tokens. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

richardsolomou · 2026-09-08T11:23:22Z


-    const provider: string | undefined = providerProperty ? String(providerProperty).toLowerCase() : undefined
+    return modelParameters && typeof modelParameters === 'object'
+        ? (modelParameters as Record<string, unknown>)['service_tier']


i still dont think we can treat this field as served-side without an explicit signal. released LangChain clients copy the requested tier into run.modelParams on successful events and never merge the response, so a refused flex request has no $ai_is_error and still gets priced as flex.

richardsolomou · 2026-09-08T11:24:01Z

+// OpenRouter names OpenAI's priority-tier endpoints "fast".
+const SERVICE_TIER_KEY_SUFFIX: Record<string, string> = {
+    flex: '-flex',
+    priority: '-fast',


the suffix needs to depend on the provider rather than only the tier. google/gemini-3-flash-preview has google-ai-studio-priority at 9e-7, but this probes google-ai-studio-fast and falls back to the 5e-7 standard row

I added both priority and fast to check. We could either create a map per provider and suffix, couldn't find it in our current codebase.

…named tier keys $ai_model_parameters.service_tier never prices: released SDKs wrote the requested tier there on successful events, and a request can be refused. $ai_service_tier's writers assert served values on success and error events alike, so the error guard goes. Tier keys are named per provider (google-ai-studio-priority, openai-fast), so resolution probes the literal name before the fast alias. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

veria-ai · 2026-09-08T14:24:20Z

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

A customer-controlled $ai_service_tier of __proto__ or constructor returned an inherited non-array and threw per event. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

deployment-status-posthog · 2026-09-10T08:43:09Z

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-09-10 08:43 UTC	Run
prod-us	✅ Deployed	2026-09-10 08:54 UTC	Run
prod-eu	✅ Deployed	2026-09-10 09:03 UTC	Run

bernatixer self-assigned this Sep 3, 2026

bernatixer and others added 7 commits September 3, 2026 17:44

refactor(aio): halve openai token rates for flex instead of reading v…

56fa6d4

…ariant keys Applying OpenAI's uniform flex rule ourselves keeps pricing independent of how OpenRouter models service tiers, per team discussion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

refactor(aio): halve every rate uniformly on flex

b652dc4

OpenAI's flex table is 50% across the board, so the per-field list goes; the only deviation is the undiscounted web-search tool fee, accepted at half a cent per search. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

refactor(aio): put the flex branch first in applyServiceTierPricing

d614a19

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

bernatixer marked this pull request as ready for review September 3, 2026 16:35

pr-assigner-resolver-posthog Bot requested review from a team September 3, 2026 16:37

brandonleung requested changes Sep 3, 2026

View reviewed changes

Comment thread nodejs/src/ingestion/pipelines/ai/costs/cost-model-matching.ts Outdated

Comment thread nodejs/src/ingestion/pipelines/ai/costs/cost-model-matching.ts Outdated

Comment thread nodejs/src/ingestion/pipelines/ai/costs/cost-model-matching.ts Outdated

richardsolomou requested changes Sep 4, 2026

View reviewed changes

Comment thread nodejs/src/ingestion/pipelines/ai/costs/cost-model-matching.ts Outdated

bernatixer mentioned this pull request Sep 4, 2026

feat(aio): run cluster labeling on the flex service tier #92166

Merged

1 task

bernatixer changed the title ~~feat(aio): price OpenAI flex-tier generations at flex rates~~ feat(aio): price ai generations by their served service tier Sep 4, 2026

bernatixer and others added 3 commits September 4, 2026 13:54

test(aio): exercise tier pricing on a genuinely flex-capable model

e2f14f2

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

refactor(aio): get-prefix the property accessors

49e1118

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

refactor(aio): fold the tier lookup into resolveModelCostForProvider

ba9af86

Provider-key selection lives in one function: the served tier resolves by its own key ahead of the existing cascade. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

bernatixer requested review from Radu-Raicea, brandonleung and richardsolomou September 4, 2026 12:15

bernatixer mentioned this pull request Sep 4, 2026

feat(ai): capture the served service tier into model parameters PostHog/posthog-python#920

Merged

5 tasks

Radu-Raicea approved these changes Sep 4, 2026

View reviewed changes

richardsolomou reviewed Sep 7, 2026

View reviewed changes

bernatixer mentioned this pull request Sep 7, 2026

fix(ai): only record a service tier the provider actually served PostHog/posthog-js#4818

Merged

5 tasks

fix(aio): never tier-price error events

cde222d

SDK error paths capture the requested tier with partial usage; a tier that was never confirmed served must not discount those tokens. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

bernatixer requested a review from richardsolomou September 7, 2026 09:46

richardsolomou requested changes Sep 8, 2026

View reviewed changes

bernatixer mentioned this pull request Sep 8, 2026

feat(ai): emit the served service tier as the explicit $ai_service_tier property PostHog/posthog-python#932

Merged

5 tasks

refactor(aio): table the tier-key suffixes with probe order

ca497b7

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

veria-ai Bot reviewed Sep 8, 2026

View reviewed changes

Comment thread nodejs/src/ingestion/pipelines/ai/costs/provider-matching.ts Outdated

fix(aio): guard the tier table against prototype-chain keys

2d70cf5

A customer-controlled $ai_service_tier of __proto__ or constructor returned an inherited non-array and threw per event. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

bernatixer requested a review from richardsolomou September 8, 2026 14:28

Merge branch 'master' into feat/aio-flex-tier-cost-pricing

f91fe21

brandonleung approved these changes Sep 9, 2026

View reviewed changes

Merge branch 'master' into feat/aio-flex-tier-cost-pricing

961d7be

richardsolomou approved these changes Sep 10, 2026

View reviewed changes

trunk-io Bot merged commit 476104d into master Sep 10, 2026
204 checks passed

trunk-io Bot deleted the feat/aio-flex-tier-cost-pricing branch September 10, 2026 08:18

Uh oh!

Conversation

bernatixer commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Changes

How did you test this code?

Automatic notifications

Docs update

🤖 Agent context

Uh oh!

trunk-io Bot commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 CI report

Uh oh!

trunk-io Bot commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

richardsolomou Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

bernatixer Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

richardsolomou Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

bernatixer Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

richardsolomou Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

richardsolomou Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

bernatixer Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

veria-ai Bot commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR overview

Security review

Uh oh!

Uh oh!

deployment-status-posthog Bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploy status

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bernatixer commented Sep 3, 2026 •

edited

Loading

trunk-io Bot commented Sep 3, 2026 •

edited

Loading

github-actions Bot commented Sep 3, 2026 •

edited

Loading

trunk-io Bot commented Sep 3, 2026 •

edited

Loading

veria-ai Bot commented Sep 8, 2026 •

edited

Loading

deployment-status-posthog Bot commented Sep 10, 2026 •

edited

Loading