Record every tool call in an audit log - #1554
Open
midego1 wants to merge 1 commit into
Open
Conversation
Executor kept no record of tool usage. A run that called GitHub or Search Console left one HTTP line (POST /mcp 200) and nothing about which integration, which tool, or what came back — and the analytics catalog is anonymous by construction, so it deliberately drops exactly those fields. Two questions were therefore unanswerable after the fact: what did this agent touch, and what did my policies actually stop. `execute` is the one place every call passes through, whatever the plugin kind and whatever the host, so the row is written there, from an `onExit` wrapper that sees every way a call can end: - ok / fail — reached the upstream. `fail` is a tool's own error result, which rides the SUCCESS channel by design and would otherwise be recorded as a healthy call. - blocked / declined — never left the gateway. A policy stopped it, or a human refused the approval. These leave no other trace anywhere: they end before any request is made. - error — the tool or connection did not exist, the plugin failed to load, the transport broke. Arguments and results are never stored: an argument can be a credential. The row keeps the top-level argument NAMES, which is what an audit needs without the table becoming a place secrets accumulate. Writing a row can never change the outcome of the call it describes — a failed write is logged and swallowed, because an audit trail that can take the gateway down with it is worse than one with a gap in it. Readable three ways: `executor.toolCalls.list()`, `GET /api/tool-calls` (filter by integration, connection, outcome, time), and an Activity page in the console. Read-only by construction — a log a caller can edit is not evidence, so there is no write or delete endpoint. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap
Executor keeps no record of tool usage. A run that calls GitHub or Search Console leaves one HTTP line —
— and nothing about which integration, which tool, or what came back. There is no
executions/audittable in the schema, andpackages/core/analyticsis anonymous by construction: its own header forbids tool addresses, connection names and arguments, which are exactly the fields an audit needs. Theexecutor.tool.executespan carries the right data, but only where an OTel exporter is configured, and it can't answer "which connection did this agent use last week".Two questions were therefore unanswerable after the fact: what did this agent touch, and what did my policies actually stop.
What this adds
executeinpackages/core/sdk/src/executor.tsis the one place every call passes through — every plugin kind, every host, MCP and REST alike — so the row is written there, from anEffect.onExitwrapper that sees every way a call can end:okfailblockedblockpolicy stopped itdeclinederrorblockedanddeclinedare the rows that make this worth having: both end before any request is made, so nothing upstream ever saw them and no HTTP-level observation can.Each row carries the address as called, its integration/connection/tool, the governing policy (action + pattern), the duration, and the top-level argument names.
Readable three ways:
executor.toolCalls.list({ integration, connection, outcome, since, limit })GET /api/tool-callswith the same filtersRead-only by construction: a log a caller can edit is not evidence, so there is no write or delete endpoint.
executor.toolCalls.prune({ before })exists for retention, and nothing schedules it — an audit log that silently deletes itself on a default nobody chose seemed worse than one that grows.What it deliberately does not store
Arguments, results, and any text that came from outside:
code; plugins deriveerror.messagefrom upstream response bodies (the OpenAPI plugin lifts it straight out), which routinely echo the request back — token included.ToolError.codeis typed as any string, so a plugin can forward a body into it. Anything not matching an identifier shape is dropped; theoutcomecolumn already says what happened.executetakesunknownargs, so{ "ghp_realtoken": null }is a reachable shape; names are length-bounded, identifier-shaped, and credential-shaped ones are dropped.Failure behaviour
Writing a row can never change the outcome of the call it describes. The write is wrapped in
catchCause, so an insert failure — or a defect — is logged and swallowed. It is awaited on purpose (a forked write would be interrupted when a per-request host tears the executor down, and a silently missing row is the one thing an audit log may not do), but awaited under a 2s cap, so a sick database costs a tool call that timeout rather than the driver's own.Scope and follow-ups
ensureDrizzleRuntimeSchemaFromTablescreates the table at boot fromcoreTables. Cloud gets0016_nosy_expediter.sql.apps/local/src/db/executor-schema.tsis left alone — it already predatesartifact/subject, and it only drives a generate-time baseline.(tenant, created_at)index and the schema layer has no non-unique index API yet. Worth adding before this table gets large; I left a note at the table definition rather than adding a unique index whose leading columns wouldn't serve the query.Verification
packages/core/sdk/src/tool-call-log.test.ts— 18 tests: outcome classification per ending (including a decline raised inside a handler, which arrives wrapped inToolInvocationError), the redaction rules with real-looking secrets, and executor-level assertions that a blocked call and a declined approval each leave a row.format:check,lint,typecheck(44/44) clean;packages/core/sdk613,packages/core/api93,packages/react320 tests pass.src/oauth-flow.test.tsflakes on its local OAuth test server under the parallel run (a different test each time, 3/3 green in isolation) — unrelated to this change.codex reviewover two rounds; every finding from both is addressed in the branch.🤖 Generated with Claude Code