Skip to content

feat: Workers usage and billing alert MCP Apps - #421

Open
geoquant wants to merge 27 commits into
mainfrom
geoquant/mcp-app-workers-usage-alert
Open

geoquant wants to merge 27 commits into
mainfrom
geoquant/mcp-app-workers-usage-alert

Conversation

@geoquant

@geoquant geoquant commented Nov 20, 2026 •

Copy link
Copy Markdown

Adds MCP Apps to the server, and the first two: a Workers usage alert and a Workers billing usage alert. In ChatGPT or Claude, "ping me if our Workers traffic gets way out of hand" opens a card with the account's last 89 days of requests, a threshold to drag and an email field. Create alert creates the notification policy.

Same prompt in ChatGPT (MCP App) and a terminal client (text only)

Same prompt, same tool: ChatGPT renders the App; a client without MCP Apps gets the text result and asks for the limit and email. Account details redacted.

What's in it

  • src/mcp-apps/: what every App shares. defineMcpApp() (a tool plus its static ui:// HTML resource, in both tool modes), apiReader() (bounded, read-only API and GraphQL reads with the direct tools' refusal explanations and step-up), MCP_APPS, and the browser shell (host connection, theme, locale). npm run app:build builds each App into app-html.generated.ts; npm run check fails if it's stale.
  • src/threshold-alert/: what any "history of one number, a line, who to tell" App shares: the collector, the model per cadence (rolling 7 days or calendar month), presets and the policy writes.
  • workers_usage_alert: weekly chart, previews an hourly custom_analytics monitor on a rolling 7 days.
  • workers_billing_alert: monthly chart, a billing_usage_alert on worker_requests. Same view.

Writes

The tools only read. The card's Create alert, Save alert and Undo call the existing generated alerting_policies_create, _update and _delete through the host's callServerTool: the click is the confirmation. No tool is added. Code Mode lists those three for the App only (_meta.ui.visibility: ['app']), served as the direct tools serve them, with the session's token and the usual refusal handling. Nothing writes through execute, which OpenAI's directory guidelines refuse for writes.

Routing

Each tool's description names its own example prompt and points to the other. Both say they only email: they never block traffic and aren't dollars. tests/fixtures/threshold-alert-prompts.ts lists the prompts each should and shouldn't reach.

Testing

  • npm run check: app build fresh, format, lint, both type checks, 947 tests. New coverage: both tool modes end to end, the GraphQL windows (including partial data and a missing window), policy recognition, refusals, an OAuth connection without Notifications Read, missing and short history, locale, the app-only write tools, and a create sent once on a network error or answered with a 5xx.
  • The view in a local MCP Apps host at 352 and 560 px, light and dark, with no overflow.
  • ChatGPT desktop against a real account, with natural prompts in new chats and no tool names: spike and "way out of hand" prompts reached workers_usage_alert, bill prompts reached workers_billing_alert, and "shut off our Workers over 1M" got an explanation and no card.

Council review

Rounds: 5

Delegate Round 1 Round 2 Round 3 Round 4 Round 5
GPT Sol (gpt-6.2-sol) 7/10 8/10 8/10 8/10 9/10
Grok (grok-4.7) 7/10 8/10 7.5/10 8.5/10 9/10
Claude Opus (claude-opus-5-5) 7/10 8.5/10 9/10 8.5/10 9/10

All three recommended keeping two tools rather than one with a basis argument. Follow-ups deliberately left out of this PR: listing the app-only write tools only to clients that declare MCP Apps support, and limiting them to the two preset policy bodies.

defineMcpApp() serves a tool and its static ui:// HTML resource in both tool
modes. apiReader() makes bounded, read-only Cloudflare API and GraphQL reads
with the direct tools' refusal explanations and step-up. MCP_APPS is the one
list an App is added to, and app-shell/ is the shared browser-side host
connection and Kumo base. scripts/build-mcp-apps.ts builds every src/<app>/app.
Read-only workers_usage_alert reads this and last month's daily Workers
requests and the account's notification policies, and lets the user pick a
billing_usage_alert threshold on a chart. It shows the policy it would create
but creates nothing; the write will be a separate confirmed tool.
Three presets and a Custom 5-100% slider over October's Workers requests,
an email row prefilled with the user's address for user tokens, and a
no-write hand-off: Create alert sends the policy to the chat with
sendMessage and updateModelContext, or shows the request it would send.
…monitor

Read 89 days of workersInvocationsAdaptive in one GraphQL query, as three
aliased ~30-day windows (the dataset caps one field's span), and preview a
custom_analytics monitor on the rolling 7-day request count instead of a
monthly billing alert. The App ports the fuzzy-input mockup: a draggable
limit over the 89-day chart, the typical week, and an email replay.
The ticks and the threshold tag move to a 60px right gutter, so the logo,
plot and To field share one left edge and the tag, ticks and button one
right edge. Bars sit on those edges, the plot loses its extra top margin,
the line-end dot goes and Edit's label sits on the content edge. The tool
description and limitations say 89 complete days, all Workers analytics
keep, and the doc describes the 89-day read and the proposed monitor.
In a terminal client the model said 'you should see a weekly chart',
though nothing renders there. The description now asks it to give the
suggested threshold and ask in chat for the limit and the email.
…d turn

Create alert handed the policy to the chat, so the model took a second
turn, with its own inference, to write it. The click is the user's
confirmation, so the App now creates exactly the policy it shows with a
tool this server already lists, through callServerTool: the generated
alerting_policies_create in direct mode, execute with one fixed POST in
Code Mode. No tool is added. It shows "Alert on", or Cloudflare's reason
and Try again, and tells the model with updateModelContext so it
doesn't create it again. Hosts without serverTools still hand off to
the chat.

Proved on a test account: one click created the policy in each mode,
Cloudflare stored the 168h window as sent, and both were deleted.
The Create alert button reached the write through a chain of tools:
alerting_policies_create, then execute in Code Mode. OpenAI's directory
guidelines refuse writes through a generic executor, and the chain
could send any body.

An App now names the generated tools its view writes with (appTools).
The direct tools list them already; Code Mode lists only those, with
the Cloudflare policy and _meta.ui.visibility ['app'], and serves them
as the direct tools do: planned and checked by the forge tools service,
sent with the session's token, refusals explained. No tool is added.
The main Worker imports the one generated group, about 55 KB.

Create alert calls alerting_policies_create once in both modes, and
tells the model it's done, so it neither confirms nor repeats it. The
tool description says the same.
The suggestion was a typical week plus a quarter, which sat inside the
history: on a spiky account it would already have emailed, and the line
started low on the chart. It is now a tenth above the busiest 7 days,
rounded up, so no past week would have emailed and the axis grows to
show the line above every bar. The upper limit allows ten times the
busiest 7 days, so a lone spike can be cleared.

The description also tells the model, where the App shows, to point to
the card instead of asking in chat for the limit and email.
With the guidance only in the tool description, ChatGPT still asked in
chat for the threshold and email although the card collects both and
creates the alert on click. The preview now carries nextStep, which the
model reads in the result itself: end without a question where the App
shows, ask where it doesn't. ChatGPT now ends with "click Create alert
on the card"; Claude in a terminal still asks for both.
Empty history and a failed read share Stratus's NotFoundPage drawing on the
plot's floor, copy left; the failed read offers Try again, which calls the tool
again through the host. Loading shows a skeleton in the card's own boxes.
Field messages (Invalid email, Didn't save) sit inside the field, so the
footer stays one row; screen readers hear the full sentence. The created row
reads Past <line> in 7 days · <email>.

mountMcpApp gains optional renderPending and renderError(text, retry).
…created

Undo deletes the alert the click just made with the generated
alerting_policies_delete, by the id the create returned; the form returns
with "Alert deleted" in the field. Edit on that alert turns the button into
Save alert, which updates it with alerting_policies_update instead of creating
a second one. Both are listed app-only in Code Mode like the create, and the
App tells the model after each write.
…ling_alert

Move what every "history of one number + a line + who to tell" App shares
into src/threshold-alert (collector, model per cadence, presets, policy
writes), and add workers_billing_alert as the billing_usage_alert preset on
the same view. The view follows the host locale, and its icon placement and
styles match the playground version.
…tions in

In ChatGPT, "just shut off our Workers if they go over 1M requests" opened
the billing card, and "is there a way to get told when a bot hammers our
Workers?" went to a web search. Both descriptions now say not to call them to
shut off, cap or block traffic, and workers_usage_alert claims questions about
how to be told of a surge. Re-run in ChatGPT: the shut-off prompt opens no
card, the surge question opens the weekly card, and the billing and "way out
of hand" prompts still reach their tools.
Remove the unregistered account-budget design and its open questions, the
extraction notes and the dated ChatGPT run log from docs/threshold-alerts.md,
and references to the internal mockup, Stratus and the local playground from
the docs and comments. No behavior change.
@geoquant
geoquant requested review from a team as code owners November 20, 2026 07:03
Drop docs/mcp-apps.md, docs/threshold-alerts.md and docs/workers-usage-alert.md
and the pointers to them. AGENTS.md keeps its short sections.
- Send a policy create once on a network error: it may have reached Cloudflare
- Read the created policy's id from result.id, not a 32-hex regex
- Fail the history read on GraphQL partial data or a missing window
- Optional alerting reads never step up, so a refusal can't replace the preview
- Keep a named billing limit below the busiest month, and say it would email
- Show the destination account on the card; hand off by account id
- Pending and error cards use the running tool's copy
- Remove the workers-usage-alert create/model re-export shims
- CI checks the MCP App HTML is fresh
- A create with no answer from Cloudflare is unknown, not refused: the card
  says it may have been created and the model is told to check first
- Say when a named limit is outside the line's range instead of moving it silently
- Label the billing card's replay and crossing as estimates
- Pending and error views follow a tool name that arrives late, and Retry
  reads the current host context
- Document that direct-tool POST/PATCH aren't retried on network errors
- A 5xx answer to a create is unknown: a gateway can send it after the save
- An unconfirmed create says so visibly and offers Create anyway; Enter
  doesn't resend it
- A failed Save alert shows its real reason
- Billing card says No crossings charted rather than No past emails
@Vejlineren

Copy link
Copy Markdown

This looks great @geoquant!

One thing I'd like to sort out before merge: src/server.ts:102-103 registers the App tools and the three alerting_policies_* write tools for every Code Mode connection. Since visibility: ['app'] is only a hint, clients without MCP Apps support will still list all five, which adds about 37 KB to each tools/list and works against the "two tools, ~1k tokens" design. Would it make sense to register them only for clients that declare MCP Apps support in this PR, rather than as a follow-up?

A 2026-07-28 request whose _meta envelope declares io.modelcontextprotocol/ui
with text/html;profile=mcp-app gets the Apps as today. Any other client,
including every 2025-era request, gets the App tools as plain tools (no
_meta.ui), no app-only alerting_policies_* writes in Code Mode, and no ui://
pages in resources/list. Code Mode tools/list drops from ~46 KB to ~21 KB
for those clients.

The server always declares the extension, still reads ui:// pages for anyone,
and logs 2026-07-28 tools/list requests that don't declare Apps, by client.
- Sample the Apps log at 1%, record declared=true|false for tools/list and
  tools/call (with the tool), and quote and cut client-supplied names.
- Compare the App MIME type ignoring whitespace and case.
- Hand the parsed body to isLegacyRequest instead of reading it again.
- Docs: accepted 2025-era loss, one size figure.
@geoquant

Copy link
Copy Markdown
Author

Note

🤖 Claude Opus 5.5 responding on behalf of @geoquant

Done in a94ff20 and df001a3. A 2026-07-28 request that declares io.modelcontextprotocol/ui with text/html;profile=mcp-app in its _meta envelope gets the Apps. Every other client gets the two alert tools as plain text tools, without the three alerting_policies_* writes or the ui:// pages in resources/list. That takes Code Mode's tools/list from ~46 KB to ~21 KB.

2025-era requests never count as declaring Apps, since this stateless server only sees their capabilities in initialize. In our logs nearly all tools/list traffic, including Claude's and ChatGPT's, is 2026-07-28. A sampled log line records, per client, whether 2026-07-28 tools/list and tools/call requests declare Apps, so we can check that Apps hosts send the declaration.

…the form's state

accountName fed only the removed Account line. The form's six loose flags
(failure, unconfirmed, live, undoing, undoFailed, notice) become fields of
the Phase they belong to, so impossible combinations can't be represented.
…text, list compact schemas

- The threshold alert collector sends the history and both alerting reads
  together: 368 ms against 545 ms for the three in turn (median of 12, read-only
  token against api.cloudflare.com). A failed history stops the other two, and
  limitations keep their order whichever read answers first.
- A client that renders the card gets the 89 daily counts only in
  structuredContent; the text the model reads names the range instead (~2 KB
  less per call, re-read on every later turn). A client without Apps still gets
  the whole result as text, since that is all it has.
- App tools list without zod's $schema, date regexes and safe-integer bounds:
  each listing goes from 5.2 KB to 4.1 KB, identical in Code Mode and the direct
  tools. zod still validates every argument and result.
The Workers usage alert view goes from 813 KB to 623 KB of HTML, 222 KB to
179 KB gzipped, with every screenshot pixel-identical (both presets, light and
dark, 352 and 560 px, hover, drag, keyboard, created, edit, invalid email,
loading and error).

- Tailwind scanned every Kumo module for classes, so the stylesheet held every
  component's. app:build now builds each App twice and scans only the
  JavaScript the first build shipped: 175 KB of CSS becomes 67 KB.
- Kumo's Button imports its Tooltip for a title prop no App passes, and that
  brings floating-ui and Base UI's popup stores. The build swaps it for a stub
  that throws if a Button ever gets a title: 638 KB of JavaScript becomes
  556 KB.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants