Skip to content

fix(agent): Keep spent plan approvals spent - #181

Merged
yahyafakhroji merged 7 commits into
mainfrom
fix/approval-replay-isolation-and-ci-tests
Nov 18, 2026
Merged

yahyafakhroji merged 7 commits into
mainfrom
fix/approval-replay-isolation-and-ci-tests

Conversation

@yahyafakhroji

@yahyafakhroji yahyafakhroji commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem
Without a database the assistant kept spent plan approvals in memory, so a restart or a second replica accepted them again, and text approvals on surfaces without a message id were refused for only 7.5 to 15 minutes. One plan could also be approved three ways, by card, typed code and typed token, each spent separately.

Solution
resources_apply is offered only with a durable approval store. Each approval is recorded under the plan's approval code inside the project's ledger, and the server derives that code from the verified plan, so no conversation or message id from the client makes it fresh again. The gate also refuses a plan made for a project other than the turn's. Plan approvals live in their own assistant_plan_approvals table, pruned after 30 days, which replicas on the previous release never touch. An approval is released only when the first write never reached the platform.

Creates carry the cost. Creating, deleting and recreating a resource from identical manifests, or retrying a create that may have reached the platform but did not land, is refused until the row ages out, and the refusal tells the model that a plan with changed manifests gets a new code. Old replicas never read the new table, so during a rollout they accept a typed approval again under each new message id until they drain. Approvals spent under the #177 key formats are not in the new table either; production has not shipped #177.

This builds on #182 for the Postgres test helper, so merge that first.

Closes #178

@yahyafakhroji yahyafakhroji added bug Something isn't working security labels Nov 17, 2026
@yahyafakhroji
yahyafakhroji marked this pull request as draft November 17, 2026 02:20
@yahyafakhroji
yahyafakhroji force-pushed the fix/approval-replay-isolation-and-ci-tests branch from 363ed7d to 51bd199 Compare November 17, 2026 02:21
@yahyafakhroji
yahyafakhroji marked this pull request as ready for review November 17, 2026 02:21
@yahyafakhroji
yahyafakhroji marked this pull request as draft November 17, 2026 02:27
@yahyafakhroji
yahyafakhroji force-pushed the fix/approval-replay-isolation-and-ci-tests branch from 51bd199 to c14a6a0 Compare November 17, 2026 02:27
@yahyafakhroji yahyafakhroji changed the title fix: Keep approvals spent, isolate tasks by project, test with Postgres fix(agent): Keep spent plan approvals spent Nov 17, 2026
@yahyafakhroji
yahyafakhroji changed the base branch from main to fix/project-isolation-and-ci-tests November 17, 2026 02:27
@yahyafakhroji
yahyafakhroji marked this pull request as ready for review November 17, 2026 02:27
@ecv ecv mentioned this pull request Nov 17, 2026
2 tasks

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes.

A spent approval can be used again from another conversation. The single-use key is planApprovalKey(contextID, approvalCode), and contextID comes from the client: on /chat/conversations/{contextId}/sendmessage it is a path parameter. Main keyed the card path on the plan token alone, so this is a regression on the path #178 set out to close.

  • A scratch test replayed one card approval (token tok-1, code plan-abc). On conv-1 it applied, a second try on conv-1 was refused as already used, and the same approval on conv-2 applied again (writes=2).
  • The 15-minute plan token expiry and resource-version binding on updates limit what a replay can do, but the approval gate itself does not stop it. A create plan replayed within the window runs twice as far as the gate is concerned.
  • Key the ledger on something the client cannot choose: the plan token or the approval code bound to the project, not the conversation. Add a test that replays one approval across two context ids and expects a refusal.

What holds at c14a6a0: with no durable store, resources_apply is never offered and an injected call gets unknown tool. Concurrent consumes are atomic via INSERT ... ON CONFLICT DO NOTHING. Retained rows make the refusal hold at any age within a conversation.

Follow-ups:

  • During a rolling deploy, replicas on the old binary use the old key format and ignore retain, and their prune deletes every row older than a day. Ship this where no old replica runs alongside, or make the old prune harmless first.
  • Retained rows are never pruned, so the ledger grows without bound. Age them out well beyond a plan's useful life.
  • No test races two Approve calls through gateApplyTool, or a Release against a Consume. Only the store primitive is raced.

Without a database the assistant kept spent approvals in memory, so a
restart or a second replica accepted them again, and text approvals on
surfaces with no message id were refused for only 7.5 to 15 minutes.
resources_apply is now offered only with a durable approval store, and
every approval of a plan is keyed on one value with no time limit, kept
out of the daily prune and released only when the first write never
reached the platform. The prune now uses a partial index and runs at
most once per interval.
Plan approvals shared assistant_exec_approvals behind a retain column.
Replicas still on the previous release prune that table after a day and
ignore the column, and the new prune skipped retained rows forever, so
the ledger grew without bound.

Plan approvals now go to assistant_plan_approvals, created on boot next
to the exec table and pruned after 30 days on the same throttle. The
retain column and its partial index never shipped, so they are gone.
The ledger key held the A2A context id, which the client chooses, so
the same approval sent again under a new context id counted as fresh
and applied a second time. The key is now the plan's approval code,
which the server derives from the verified plan binding, inside the
project's ledger.

A resource created, deleted and created again from identical manifests
carries the same code, so the second create is refused until the row
ages out after 30 days.
A failed prune returned an error from Consume, so a passing database
hiccup in the cleanup refused the approval being recorded. Pruning is
best effort: log the failure, retry on the next Consume, and record the
approval anyway.
The refusal for a spent plan told the model that re-planning gives a
new approval code. That is false for a create that never landed: the
same manifests give the same code for 30 days. The refusal now says
so, and that a plan which changes the manifests gets a new code the
person can approve.
@yahyafakhroji
yahyafakhroji marked this pull request as draft November 18, 2026 00:30
@yahyafakhroji
yahyafakhroji force-pushed the fix/approval-replay-isolation-and-ci-tests branch from c14a6a0 to 970e8c8 Compare November 18, 2026 00:30
@yahyafakhroji

Copy link
Copy Markdown
Contributor Author

Thanks, fixed in d1b86e6..970e8c8.

The ledger keys plan approvals on the project and the plan's approval code, which the server derives from the plan, so the context id is out of the key. A new test replays one approval under other context ids. Plan approvals moved to their own table, assistant_plan_approvals, which keeps rows for 30 days, and main's daily prune never touches it. I added race tests for two approvals at once and for Release against Consume.

One trade-off: the assistant refuses a repeat of an identical create that never landed for 30 days, and the refusal tells the model to change the manifests.

@yahyafakhroji
yahyafakhroji marked this pull request as ready for review November 18, 2026 00:30
@yahyafakhroji
yahyafakhroji requested a review from ecv November 18, 2026 00:30
Base automatically changed from fix/project-isolation-and-ci-tests to main November 18, 2026 00:36
@yahyafakhroji
yahyafakhroji requested a review from a team as a code owner November 18, 2026 00:36
ecv
ecv previously approved these changes Nov 18, 2026

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve.

The hold is fixed: the ledger key no longer contains the conversation id, so a spent approval cannot be replayed from another conversation, and the new tests catch a regression.

  • Replayed at 970e8c8 with the card, typed-code and typed-token paths. TestApprovalReplayUnderAnotherConversationIsRefused applies once (writes=1), then refuses on conv-a, conv-b and an empty context id.
  • Tamper tests: with the !fresh refusal disabled, 6 agent tests fail. Putting the context id back into the key fails TestApprovalReplayUnderAnotherConversationIsRefused, TestConcurrentApprovalsOfOnePlanApplyOnce and TestTextApprovalReplayIsRefusedAtAnyAge. The context-id regression is caught.
  • Cross-project: the key is (project, sha256("resources_apply\0plan\0" + approvalCode)), and the code is an HMAC over a binding that includes the project, the manifests and the replaced resource versions. One project's approval cannot be consumed by, or replay into, another. The test applies the same code in project other and expects a second write.
  • Follow-up 2 is fixed: plan rows move to assistant_plan_approvals with a 30-day prune, rate-limited to one per 10 minutes and tested.
  • Follow-up 3 is fixed: TestConcurrentApprovalsOfOnePlanApplyOnce races 8 turns through gateApplyTool on memory and Postgres, and TestPostgresStoreReleaseRacesConsume covers Release against Consume.
  • Follow-up 1 is mostly fixed. Old replicas prune only assistant_exec_approvals and never touch the new table, so they cannot delete new rows.
  • The closing comment in plan_approval.go overstates the rollout bound. Old replicas key typed approvals on context and message id, or a 7.5-minute window, so during a rollout an old replica accepts the same typed approval again under any new message id, and again after each re-plan. The bound is the token life per replay, not one replay. Exposure is small, because updates fail on the moved resource version and creates hit AlreadyExists, and the old binary already had this. Reword the comment, or drain old replicas before the new guarantee is relied on.
  • CI: test (run 44208407852, head 970e8c8) passed with TEST_DATABASE_REQUIRED=true and a started Postgres 17, so the Postgres store tests ran, not skipped. I could not run Postgres locally, so I rely on this run.

Follow-ups:

  • The key is the approval code alone, so an identical create that never landed, or a delete followed by an identical re-create, is refused for 30 days until a manifest field changes. The refusal says how to escape, but consider a shorter window for creates, or releasing on a failure that may not have landed.
  • The ledger project comes from params.ProjectName and the code from the tool's project binding. Add a test that asserts these are the same value, since the key's tenant isolation depends on it.

The ledger records an approval under the turn's project, and the
approval code under the project the plan was bound to. Both come from
the same request today, but the key's tenant isolation rested on that
wiring. resources_apply now passes the plan's project to the approval,
and the gate refuses a plan made for any other project before it
touches the ledger.
The comment said a replay passes at most once during a rollout. Old
replicas key a typed approval on the message id or a 7.5-minute window,
so they accept it again under each new message id and after each
re-plan, for as long as they run. The comment now says so, and that
the guarantee holds once old replicas have drained.
@yahyafakhroji

Copy link
Copy Markdown
Contributor Author

@ecv thanks. 0acfeff passes the plan's project into the approval, and the gate refuses a plan made for another project before it touches the ledger, with a test for it. 9425ed7 fixes the rollout comment: old replicas accept a typed approval again under each new message id until they drain. I kept the 30-day window for identical creates for now.

@yahyafakhroji
yahyafakhroji requested a review from ecv November 18, 2026 01:42

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve.

Both points from my last review are addressed. The gate now refuses a plan bound to any project other than the turn's before it touches the ledger, and TestApprovalOfAPlanForAnotherProjectIsRefused checks that nothing is written and neither project's approval is spent. The rollout comment now states the typed-approval replay bound correctly.

@yahyafakhroji
yahyafakhroji merged commit 9d08e59 into main Nov 18, 2026
7 checks passed
@yahyafakhroji
yahyafakhroji deleted the fix/approval-replay-isolation-and-ci-tests branch November 18, 2026 02:48
@ecv ecv mentioned this pull request Nov 18, 2026
3 of 4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working security

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Spent plan approvals can be replayed

2 participants