Repository navigation
fix(agent): Keep spent plan approvals spent - #181
Conversation
363ed7d to
51bd199
Compare
51bd199 to
c14a6a0
Compare
ecv
left a comment
There was a problem hiding this comment.
Request changes.
A spent approval can be used again from another conversation. The single-use key is planApprovalKey(contextID, approvalCode), and contextID comes from the client: on /chat/conversations/{contextId}/sendmessage it is a path parameter. Main keyed the card path on the plan token alone, so this is a regression on the path #178 set out to close.
- A scratch test replayed one card approval (token
tok-1, codeplan-abc). Onconv-1it applied, a second try onconv-1was refused as already used, and the same approval onconv-2applied again (writes=2). - The 15-minute plan token expiry and resource-version binding on updates limit what a replay can do, but the approval gate itself does not stop it. A create plan replayed within the window runs twice as far as the gate is concerned.
- Key the ledger on something the client cannot choose: the plan token or the approval code bound to the project, not the conversation. Add a test that replays one approval across two context ids and expects a refusal.
What holds at c14a6a0: with no durable store, resources_apply is never offered and an injected call gets unknown tool. Concurrent consumes are atomic via INSERT ... ON CONFLICT DO NOTHING. Retained rows make the refusal hold at any age within a conversation.
Follow-ups:
- During a rolling deploy, replicas on the old binary use the old key format and ignore
retain, and their prune deletes every row older than a day. Ship this where no old replica runs alongside, or make the old prune harmless first. - Retained rows are never pruned, so the ledger grows without bound. Age them out well beyond a plan's useful life.
- No test races two
Approvecalls throughgateApplyTool, or aReleaseagainst aConsume. Only the store primitive is raced.
Without a database the assistant kept spent approvals in memory, so a restart or a second replica accepted them again, and text approvals on surfaces with no message id were refused for only 7.5 to 15 minutes. resources_apply is now offered only with a durable approval store, and every approval of a plan is keyed on one value with no time limit, kept out of the daily prune and released only when the first write never reached the platform. The prune now uses a partial index and runs at most once per interval.
Plan approvals shared assistant_exec_approvals behind a retain column. Replicas still on the previous release prune that table after a day and ignore the column, and the new prune skipped retained rows forever, so the ledger grew without bound. Plan approvals now go to assistant_plan_approvals, created on boot next to the exec table and pruned after 30 days on the same throttle. The retain column and its partial index never shipped, so they are gone.
The ledger key held the A2A context id, which the client chooses, so the same approval sent again under a new context id counted as fresh and applied a second time. The key is now the plan's approval code, which the server derives from the verified plan binding, inside the project's ledger. A resource created, deleted and created again from identical manifests carries the same code, so the second create is refused until the row ages out after 30 days.
A failed prune returned an error from Consume, so a passing database hiccup in the cleanup refused the approval being recorded. Pruning is best effort: log the failure, retry on the next Consume, and record the approval anyway.
The refusal for a spent plan told the model that re-planning gives a new approval code. That is false for a create that never landed: the same manifests give the same code for 30 days. The refusal now says so, and that a plan which changes the manifests gets a new code the person can approve.
c14a6a0 to
970e8c8
Compare
|
Thanks, fixed in d1b86e6..970e8c8. The ledger keys plan approvals on the project and the plan's approval code, which the server derives from the plan, so the context id is out of the key. A new test replays one approval under other context ids. Plan approvals moved to their own table, One trade-off: the assistant refuses a repeat of an identical create that never landed for 30 days, and the refusal tells the model to change the manifests. |
ecv
left a comment
There was a problem hiding this comment.
Approve.
The hold is fixed: the ledger key no longer contains the conversation id, so a spent approval cannot be replayed from another conversation, and the new tests catch a regression.
- Replayed at 970e8c8 with the card, typed-code and typed-token paths.
TestApprovalReplayUnderAnotherConversationIsRefusedapplies once (writes=1), then refuses onconv-a,conv-band an empty context id. - Tamper tests: with the
!freshrefusal disabled, 6 agent tests fail. Putting the context id back into the key failsTestApprovalReplayUnderAnotherConversationIsRefused,TestConcurrentApprovalsOfOnePlanApplyOnceandTestTextApprovalReplayIsRefusedAtAnyAge. The context-id regression is caught. - Cross-project: the key is
(project, sha256("resources_apply\0plan\0" + approvalCode)), and the code is an HMAC over a binding that includes the project, the manifests and the replaced resource versions. One project's approval cannot be consumed by, or replay into, another. The test applies the same code in projectotherand expects a second write. - Follow-up 2 is fixed: plan rows move to
assistant_plan_approvalswith a 30-day prune, rate-limited to one per 10 minutes and tested. - Follow-up 3 is fixed:
TestConcurrentApprovalsOfOnePlanApplyOnceraces 8 turns throughgateApplyToolon memory and Postgres, andTestPostgresStoreReleaseRacesConsumecovers Release against Consume. - Follow-up 1 is mostly fixed. Old replicas prune only
assistant_exec_approvalsand never touch the new table, so they cannot delete new rows. - The closing comment in
plan_approval.gooverstates the rollout bound. Old replicas key typed approvals on context and message id, or a 7.5-minute window, so during a rollout an old replica accepts the same typed approval again under any new message id, and again after each re-plan. The bound is the token life per replay, not one replay. Exposure is small, because updates fail on the moved resource version and creates hit AlreadyExists, and the old binary already had this. Reword the comment, or drain old replicas before the new guarantee is relied on. - CI:
test(run 44208407852, head 970e8c8) passed withTEST_DATABASE_REQUIRED=trueand a started Postgres 17, so the Postgres store tests ran, not skipped. I could not run Postgres locally, so I rely on this run.
Follow-ups:
- The key is the approval code alone, so an identical create that never landed, or a delete followed by an identical re-create, is refused for 30 days until a manifest field changes. The refusal says how to escape, but consider a shorter window for creates, or releasing on a failure that may not have landed.
- The ledger project comes from
params.ProjectNameand the code from the tool's project binding. Add a test that asserts these are the same value, since the key's tenant isolation depends on it.
The ledger records an approval under the turn's project, and the approval code under the project the plan was bound to. Both come from the same request today, but the key's tenant isolation rested on that wiring. resources_apply now passes the plan's project to the approval, and the gate refuses a plan made for any other project before it touches the ledger.
The comment said a replay passes at most once during a rollout. Old replicas key a typed approval on the message id or a 7.5-minute window, so they accept it again under each new message id and after each re-plan, for as long as they run. The comment now says so, and that the guarantee holds once old replicas have drained.
|
@ecv thanks. 0acfeff passes the plan's project into the approval, and the gate refuses a plan made for another project before it touches the ledger, with a test for it. 9425ed7 fixes the rollout comment: old replicas accept a typed approval again under each new message id until they drain. I kept the 30-day window for identical creates for now. |
ecv
left a comment
There was a problem hiding this comment.
Approve.
Both points from my last review are addressed. The gate now refuses a plan bound to any project other than the turn's before it touches the ledger, and TestApprovalOfAPlanForAnotherProjectIsRefused checks that nothing is written and neither project's approval is spent. The rollout comment now states the typed-approval replay bound correctly.
Problem
Without a database the assistant kept spent plan approvals in memory, so a restart or a second replica accepted them again, and text approvals on surfaces without a message id were refused for only 7.5 to 15 minutes. One plan could also be approved three ways, by card, typed code and typed token, each spent separately.
Solution
resources_applyis offered only with a durable approval store. Each approval is recorded under the plan's approval code inside the project's ledger, and the server derives that code from the verified plan, so no conversation or message id from the client makes it fresh again. The gate also refuses a plan made for a project other than the turn's. Plan approvals live in their ownassistant_plan_approvalstable, pruned after 30 days, which replicas on the previous release never touch. An approval is released only when the first write never reached the platform.Creates carry the cost. Creating, deleting and recreating a resource from identical manifests, or retrying a create that may have reached the platform but did not land, is refused until the row ages out, and the refusal tells the model that a plan with changed manifests gets a new code. Old replicas never read the new table, so during a rollout they accept a typed approval again under each new message id until they drain. Approvals spent under the #177 key formats are not in the new table either; production has not shipped #177.
This builds on #182 for the Postgres test helper, so merge that first.
Closes #178