fix(executor): validate writes before disk, wire rollback, emit validation_failed - #402
fix(executor): validate writes before disk, wire rollback, emit validation_failed#402ceilf6 wants to merge 10 commits into
Conversation
…ation_failed 三个缺陷各自的修法: 1. 坏内容先落盘再校验(#387 的直接后果) 在 call_tool 之前新增 validate_content 阶段:resolveWriteContent 解析 「写盘前即可确定的完整内容」(create_file 有 content;apply_patch 只给 补丁片段,故返回 null 走原后置路径),能解析时先跑 guard.validateCode, 不过则直接返回、根本不调用写工具。validateAfterExecution 新增 contentValidatedBeforeWrite 参数,前置已校验过的内容不再重复跑 guard。 2. needsRollback 恒为 false + 回滚从不执行(#387) needsRollback 原表达式为 `!pass && step.validation.some(v => v.required)`, 而 LLM 生成的计划里 validation 常为空数组,该条件几乎恒假;改为 `!postValidation.pass`。新增 rollbackFailedWrite:写后校验失败时按 toolResult.snapshotId 调 rollback 撤销落盘。 3. validation_failed 事件无发射点(#388) ExecutorConfig 新增 emitEvent 出口,agent.ts 接到 this.emit; 校验拦截(前置/写前/写后)与回滚各自上报事件。 注:SecurityManager 把 rollback 归类为「需审批」,非交互运行会拒绝, 自动回滚需 permissions.allow: ['rollback']——见 PR follow-up。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
代码评审报告: fix(executor): validate writes before disk, wire rollback, emit validation_failed
风险等级: 中
处理建议: 请求修改
决策摘要: 方向正确且执行路径上确实生效,但两条链路未达关联 issue 的验收口径:apply_patch(生产上几乎全部走 LLM 全文件替换)仍然先落盘后校验,#387 的首选修法只覆盖了 create_file;同时 validation_failed 现在也会在普通工具失败时触发,直接污染 #388 想要建立的「拦截可计数」指标。两处修复都很小,建议本轮改完再合。
级联分析
- 变更符号:
Executor.executeStep、Executor.validateAfterExecution(私有,新增带默认值参数)、新增私有resolveWriteContent/rollbackFailedWrite、ExecutorConfig.emitEvent(可选,additive)、ExecutorTraceStage.name新增'validate_content'(additive)、FrontAgent构造函数中的 executor 配置。 - 受影响流程: 步骤执行(pre-write 校验 → call_tool → post-write 校验 → 回滚)、agent 事件流、执行 trace 采集、进度强制引擎的中止判定。
- 变更集外调用方 (text search):
packages/core/src/executor/progress-enforcement.ts:42—needsRollback唯一消费者,语义被放宽后行为改变。packages/core/src/executor/phase-runner.ts— 真实 agent 路径(agent.ts:813→executeStepsWithErrorFeedback)根本不读needsRollback,所以放宽只影响Executor.executeSteps的使用者(如benchmarks/frontagent-flow-benchmark.mjs:206)。apps/desktop/src/state/executionReducer.ts:279/286/289— 三个事件的 handler 已存在,形状匹配,无需改动。benchmarks/eval/run-eval.mjs:97+benchmarks/eval/report.mjs:75— 按event.type计数并直接印在评测报告里,受下面第 1 条影响。packages/mcp-file/src/tools/create-file.ts:88/apply-patch.ts:136确实返回snapshotId,agent.ts:243已注册rollback工具映射,回滚链路可达。
- 置信度: medium(无代码图谱输出,结论来自 Read/Grep 全仓核查;私有方法签名变更与 additive 联合类型不构成外部破坏)
问题发现
-
[高]
validation_failed现在也覆盖普通工具失败,#388 想要的「拦截计数」失真- 证据:
executor.ts:228-231对任意!postValidation.pass发事件;而validateAfterExecution(executor.ts:614-623)在result.success === false时就返回pass:false, results:[], blockedBy:[工具报错]。于是run_command、read_file、create_file撞到「文件已存在」等普通失败都会发validation_failed。benchmarks/eval/run-eval.mjs:97按 type 计数、report.mjs:75把这个数字当作「校验是否起作用」的证据;桌面端executionReducer.ts:283会把工具报错渲染成「校验失败: ...」。 - 受影响调用方/流程: 评测报告的 guard 有效性口径、桌面 UI 日志、后续重跑 benchmark 的结论。
- 最小可行修复: 只在失败确由 guard 产生时发事件(例如判
postValidation.results.length > 0,或让validateAfterExecution区分「工具失败」与「guard 拦截」两种返回并只对后者上报)。issue #388 还要求带 check type/path,当前 payload 只有ValidationResult,可顺带确认是否足够。
- 证据:
-
[高]
apply_patch仍是先落盘后校验,#387 的首选修法只覆盖 create 路径- 证据:
resolveWriteContent(executor.ts:564-584)只认toolParams.content,而apply_patch的内容在patches[0].content。生产上这条路几乎全部走 LLM 生成:planner.ts:465固定发patches: [](注释「补丁由 Executor 生成」),提示词plan-generation.ts:423要求 apply_patch 用changeDescription + needsCodeGeneration,executor-skills.ts:278-288随后生成{operation:'replace', startLine:1, endLine:原文件行数, content: 全文件}——这正是「写盘前已完全可知的完整文件内容」,也正是 markdown 围栏幻觉出现的地方。结果:#387 证据里的.tsx围栏若来自修改类任务,仍会先写进磁盘,只能靠回滚兜底;而回滚在无approvalHandler的非交互 CLI 会被tool-call-handler.ts:187-196直接拒绝(评测侧因run-eval.mjs:94自动批准不受影响,PR 描述中「eval harness 会被安全层挡住」这点与仓库现状不符)。 - 受影响调用方/流程: 所有 modify/bugfix/refactor 任务的写盘路径;#387 的验收项「create/patch 步骤的非法内容不得留在磁盘上」未对 patch 成立。
- 最小可行修复: 在
resolveWriteContent中补一条窄分支——apply_patch且patches为单个整文件replace(startLine === 1且endLine >= 原文件行数,原文件可从collectedContext.files取)时返回该 content。附带收益:这条补上后,validateAfterExecution里的 post-write 内容校验分支与contentValidatedBeforeWrite这个 mode flag 都可以整体删掉,少一条并行校验路径。
- 证据:
-
[中]
needsRollback放宽的范围比 PR 描述更大,且无测试- 证据:
executor.ts:244现在对任何!postValidation.pass置 true,其中包含纯工具失败(见 finding 1 的同一分支),不只是「post-write 校验失败」。唯一消费者progress-enforcement.ts:42会把剩余步骤全标skipped并 break,因此在Executor.executeSteps路径上,一次read_file/run_command失败就会中止整个剩余计划。真实 agent 路径走PhaseRunner(不读该标志),所以爆炸半径限于executeSteps的直接使用者,但这与 PR 里「whenever a step fails post-write validation」的描述不一致。 - 受影响调用方/流程:
benchmarks/frontagent-flow-benchmark.mjs:206及任何以executeSteps为入口的嵌入方。 - 最小可行修复: 把标志收窄到校验类失败(与 finding 1 用同一判据),或保留现语义但在 PR 描述与测试中明确覆盖「普通工具失败也会中止后续步骤」。
- 证据:
-
[中] 回滚失败是静默的,UI 会停在「回滚开始」
- 证据:
executor.ts:599-605只在成功时发rollback_completed,失败仅debugWarn(debug关闭时完全无输出)。安全层拒绝时返回的是{success:false, error},而这里读的是result.message,日志会打成undefined。桌面端已记了rollback_started(executionReducer.ts:287)却永远等不到终止事件。 - 受影响调用方/流程: 非交互运行的可观测性、桌面日志一致性——恰恰是本 PR 想解决的那类盲区。
- 最小可行修复: 失败时也上报一个终止事件(新增
rollback_failed,或复用现有告警事件),并同时读取message ?? error。
- 证据:
行级发现
- [packages/core/src/executor/executor.ts:229] 这里对任何
postValidation失败都发validation_failed,包含validateAfterExecution对纯工具失败返回的pass:false;建议加判据(如postValidation.results.length > 0)只上报 guard 拦截,否则评测计数与 UI 文案都会把工具报错算成校验拦截。 - [packages/core/src/executor/executor.ts:244]
needsRollback: !postValidation.pass把普通工具失败也纳入,progress-enforcement.ts:42会因此中止剩余全部步骤;建议与上一条用同一判据收窄到校验类失败。 - [packages/core/src/executor/executor.ts:577] 只从
toolParams.content取内容,导致apply_patch恒返回 null;而executor-skills.ts:278-288生成的是覆盖整文件的单个replacepatch,内容在写盘前已完全可知,建议补一条整文件 replace 分支,让 #387 的前置校验也覆盖修改路径。 - [packages/core/src/executor/executor.ts:604] 回滚失败只走
debugWarn,且安全层拒绝返回的是{success:false,error}(message为 undefined);建议补一个终止事件并读取message ?? error,避免 UI 停在「回滚开始」。 - [packages/core/src/executor/executor.test.ts:794] 回滚用例全程 mock
callTool,只断言调用参数,没有验证磁盘上的文件被恢复;#387 的验收项要求的是「文件不留在磁盘上」,建议改成真实SnapshotManager+ 临时目录并断言文件内容已还原。
Karpathy 评审
- 假设: 隐含假设「
apply_patch的完整内容写盘前不可知」——在 codegen 路径上不成立(planner.ts:465+executor-skills.ts:288)。另一处隐含假设是postValidation失败等价于 guard 拦截,实际混入了工具失败。PR 描述中关于 eval harness 无审批通道的说法与run-eval.mjs:94不符。 - 简洁性:
WRITE_ACTIONS常量、resolveWriteContent、rollbackFailedWrite拆分合理,命名与既有风格一致,没有多余抽象。contentValidatedBeforeWrite是一个为兼容旧路径而存在的 mode flag;若 finding 2 落地,这个 flag 与 post-write 内容校验分支可整段删除,属于能真正减少概念数的简化。 - 结构质量: 无错误分层、无重复 helper、无文件规模跨界(
executor.ts847 行);trace 阶段与配置字段均为 additive,对下游无破坏。 - 变更范围: 5 个文件都服务于三个既定缺陷,无无关重构或格式噪声。范围本身克制,问题在于覆盖不足而非过宽。
- 验证: 新增用例针对性强(pre-write 用例使用真实 guard + 真实失败样本 + 断言磁盘无文件,这点很好)。但回滚用例完全 mock,且测试自身注释(
executor.test.ts:781)承认刻意绕开了 LLM codegen 分支——而那正是生产上 apply_patch 的主路径。
缺失覆盖
apply_patch走 codegen 全文件替换、内容非法时的端到端用例:断言磁盘上不留下(或已还原)非法文件,这是 #387 明确要求的回归测试且当前 patch 路径未覆盖。- 「普通工具失败不应发
validation_failed」的负向断言(配合 finding 1 的修复)。 Executor.executeSteps路径下,一次非校验类工具失败对剩余步骤的影响(finding 3 的行为变更目前无任何测试锁定)。- 非交互配置(无
approvalHandler、无permissions.allow: ['rollback'])下回滚被安全层拒绝时的事件与日志行为。
| ); | ||
|
|
||
| if (!postValidation.pass) { | ||
| this.config.emitEvent?.({ type: 'validation_failed', result: postValidation }); |
There was a problem hiding this comment.
这里对任何 postValidation 失败都发 validation_failed,包含 validateAfterExecution 对纯工具失败返回的 pass:false;建议加判据(如 postValidation.results.length > 0)只上报 guard 拦截,否则评测计数与 UI 文案都会把工具报错算成校验拦截。
| stepResult, | ||
| validation: postValidation, | ||
| needsRollback: !postValidation.pass && step.validation.some((v) => v.required), | ||
| needsRollback: !postValidation.pass, |
There was a problem hiding this comment.
needsRollback: !postValidation.pass 把普通工具失败也纳入,progress-enforcement.ts:42 会因此中止剩余全部步骤;建议与上一条用同一判据收窄到校验类失败。
| } | ||
|
|
||
| const path = (toolParams.path ?? step.params.path) as string | undefined; | ||
| const content = (toolParams.content ?? step.params.content) as string | undefined; |
There was a problem hiding this comment.
只从 toolParams.content 取内容,导致 apply_patch 恒返回 null;而 executor-skills.ts:278-288 生成的是覆盖整文件的单个 replace patch,内容在写盘前已完全可知,建议补一条整文件 replace 分支,让 #387 的前置校验也覆盖修改路径。
| if (result.success) { | ||
| this.config.emitEvent?.({ type: 'rollback_completed', snapshotId }); | ||
| } else { | ||
| this.debugWarn(`[Executor] Rollback failed for snapshot ${snapshotId}: ${result.message}`); |
There was a problem hiding this comment.
回滚失败只走 debugWarn,且安全层拒绝返回的是 {success:false,error}(message 为 undefined);建议补一个终止事件并读取 message ?? error,避免 UI 停在「回滚开始」。
| }), | ||
| ); | ||
|
|
||
| expect(callTool).toHaveBeenCalledWith('rollback', { snapshotId: 'snap-1' }); |
There was a problem hiding this comment.
回滚用例全程 mock callTool,只断言调用参数,没有验证磁盘上的文件被恢复;#387 的验收项要求的是「文件不留在磁盘上」,建议改成真实 SnapshotManager + 临时目录并断言文件内容已还原。
|
Follow-ups from this PR are now filed as issues:
Both are the same defect class this PR addresses for Still open from the PR description and not filed as an issue, because it is a policy decision rather than a defect: |
|
Correction to my earlier follow-up comment: #405 was a duplicate of #400 (which predates it and already covers Revised follow-up list for this PR:
The #406 finding matters for how this PR should be read: the executor plumbing here blocks writes that the guard flags, and it demonstrably does so (see the before/after probe in the benchmark report PR). But when the checker itself passes the content, the write still lands. Interception rate is now gated on checker strength, not on the executor. |
validation_failed 此前复合了三个来源:执行前结构性拦截、写盘前内容拦截, 以及写盘后校验失败——而最后一类包含纯工具失败,因为 validateAfterExecution 在 `resultObj.success === false` 时返回的是 results 为空的失败结果。 事件名为「校验失败」却把工具失败计入,任何人拿它当拦截数都会被误导。 改为只在「至少一项真实检查判失败」(results 中存在 pass:false 项)时 emit, 指标按构造即为纯拦截信号,不必事后按 results[].type 过滤。 回滚仍只看 !postValidation.pass:工具失败但已产生快照时同样要撤销落盘。 补测试:工具失败(EACCES)不产生 validation_failed。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Pushed The first version of this PR had a metric defect I want to name explicitly, because it would mislead anyone reading the number. The event was emitted from three places, and the third was not an interception at all:
So an Fix: emit only when at least one real check reports New test: Re-verified end to end after the change, through the real |
There was a problem hiding this comment.
代码评审报告: fix(executor): validate writes before disk, wire rollback, emit validation_failed
风险等级: 高
处理建议: 请求修改
决策摘要: 方向正确且证据扎实,但前置校验把 guard 的 block 判定升级成了"写盘否决权"——其中 import_validity 对"后续步骤才创建的相对导入"必然判 block,会让本来能落盘的文件彻底不写;同时 apply_patch(真实计划里全部 modify 步骤)没有拿到任何前置保护,#387 在默认无审批配置下未闭环。建议收窄前置门禁的检查范围并补上 patch 路径后再合。
级联分析
- 变更符号:
Executor.executeStep、Executor.validateAfterExecution(签名 +1 参数)、新增resolveWriteContent/emitValidationFailed/rollbackFailedWrite、ExecutorConfig.emitEvent、ExecutorTraceStage.name联合类型、FrontAgent构造。 - 受影响流程:
executeStep → validate_content → call_tool;Executor.executeSteps → progress-enforcement;phase-runner的失败/恢复回路;agent 事件流 →apps/desktop/src/state/executionReducer.ts;benchmarks/eval/report.mjs对validation_failed的计数口径。 - 变更集外调用方:
packages/core/src/executor/progress-enforcement.ts:42(needsRollback唯一消费者,语义被动过但文件未改)、apps/desktop/src/state/executionReducer.ts:279-290(validation_failed/rollback_started/rollback_completed三个 case 现在首次会真的收到事件)、packages/core/src/skills/executor-skills.ts:278-288(apply_patch 代码生成路径)、packages/core/src/llm/schemas.ts:19-37(决定toolParams实际能带哪些字段)。 - 置信度: medium(本地只读代码 + 文本检索确认调用方,未自行跑 GitNexus;PR 提供的图谱结果与我读到的调用关系一致)
问题发现
-
[高] 前置门禁把
import_validity的 block 判定变成了"不写盘",会吞掉合法文件- 证据:
executor.ts:181-205调用的是完整guard.validateCode,而validateCode会跑checkAllImports(packages/hallucination-guard/src/guard.ts:196-203);相对导入解析不到时返回severity: 'block'(packages/hallucination-guard/src/checks/import-validity.ts:84-90),非.//开头的路径别名(@/components/x)会走checkPackageExists判成"未安装的包"(同文件 36-53、96-126)。改动前这些判定发生在写盘后:step 标记失败,但文件仍在磁盘上,等后续步骤把被导入模块建出来后仍可自洽;改动后该文件根本不存在。 - 受影响调用方/流程: 多文件计划中"先建页面、后建组件"的
create_file步骤;随后phase-runner.ts:271-275记录 phaseErrors →runPhaseRecovery。恢复步骤若是apply_patch,会命中Cannot apply patch: file does not exist,而该错误在executor-skills.ts:291-298被判为可跳过 → 记为成功;onPhaseComplete的tsc --noEmit(agent/phase-checks.ts:99-105)也看不到一个"根本不存在的文件"的导入错误,于是phase-runner.ts:369-376会把原本 failed 的步骤翻成 completed。净效果:文件没写、run 报成功。 - 最小可行修复: 前置阶段只用写盘前可靠的检查(
syntax_validity)做否决,import_validity结果保留为非阻塞或仍留在写盘后判定;若要保留 import 门禁,用已有的config.getCreatedModules?.()(executor/types.ts:23)把"计划内待建模块"排除出 block 集合。并补一条回归测试:内容引用同计划后续步骤才创建的相对模块时,文件必须照常写入。
- 证据:
-
[高]
apply_patch全程没有前置保护,#387 的 patch 侧验收未闭环- 证据: PR 描述称"apply_patch 只带补丁片段所以返回 null",但
packages/core/src/llm/schemas.ts:19-37的STEP_PARAMS_SCHEMA既没有content也没有patches字段,因此 LLM 计划里的 apply_patch 步骤必然走executor-skills.ts:226-289的代码生成分支,产出的是覆盖全文件的单个 replace 补丁(startLine: 1, endLine: originalLines, content: modifiedCode)——写盘后的完整内容在写之前就是已知的。而resolveWriteContent(executor.ts:573-584)只认toolParams.content,对该路径恒返回null。 - 受影响调用方/流程: 所有 modify 类任务。它们只能落到写盘后校验 +
rollbackFailedWrite,而rollback在security.ts:283-292被判为ask/high,非交互运行直接拒绝(PR follow-up 1 已自述)。issue #387 明确要求"a create/patch step whose content is syntactically invalid must not leave the file on disk",patch 侧在默认配置下仍会留下坏文件——正是评测所处的 headless 配置。 - 最小可行修复: 在
resolveWriteContent里识别"单个 replace 且覆盖collectedContext.files.get(path)全部行"的补丁,直接用patch.content作为前置校验内容;其余补丁形态继续返回 null。
- 证据: PR 描述称"apply_patch 只带补丁片段所以返回 null",但
-
[中]
needsRollback现在等价于"步骤失败",语义与字段名脱节- 证据:
validateAfterExecution(executor.ts:626-635)对任何result.success === false都返回pass:false,因此executor.ts:245的!postValidation.pass对EACCES、命令失败等纯工具错误同样为 true——与!stepResult.success完全同义;同时写盘失败时rollbackFailedWrite已经在executor.ts:228-232尝试过回滚,标志位却仍报"需要回滚"。PR 描述把影响表述为"post-write validation 失败时",实际范围是所有失败。 - 受影响调用方/流程:
progress-enforcement.ts:42(Executor.executeSteps这一公开 API:任何一步失败即把余下步骤全部置skipped)。主链路走executeStepsWithErrorFeedback/phase-runner,不读该标志,所以生产影响有限,但 SDK 消费者的 run shape 变了,且变化面比 PR 描述更宽。 - 最小可行修复: 让该标志表达"有内容已落盘且未被成功撤销"(结合
snapshotId与rollbackFailedWrite的返回值),或明确删除它并让消费者直接读stepResult.success;同时在 PR 描述里更正影响范围。
- 证据:
-
[中] 回滚失败没有任何终态信号,恰好在最需要观测的 headless 场景失声
- 证据:
executor.ts:611-617先发rollback_started,成功才发rollback_completed,失败只走debugWarn。AgentEvent联合(packages/core/src/types.ts:793-795)没有失败态;apps/desktop/src/state/executionReducer.ts:286-290会留下一条永远等不到收尾的"回滚开始"日志。默认非交互配置下rollback被安全层拒绝,正是这条静默分支。 - 受影响调用方/流程: 桌面端执行日志;
benchmarks/eval/report.mjs:74-75依赖事件计数评估 guard 有效性——"拦截到了但没能撤销"与"撤销成功"不可区分。 - 最小可行修复: 补一个终态(扩
AgentEvent加rollback_failed,或至少把回滚失败原因追加进stepResult.error),使"坏文件是否仍在磁盘上"可判定。
- 证据:
-
[中] 纯工具失败也会触发回滚,交互式运行会多出审批弹窗
- 证据:
executor.ts:228-232在postValidation.pass === false时无条件调用rollbackFailedWrite,而该分支包含工具自身报错(executor.ts:626-635)。只要工具返回了snapshotId,就会发起一次rollback,在注册了approvalHandler的桌面/CLI 交互模式下触发snapshot_rollback_requires_approval审批(security.ts:283-292)。行内注释说明这是有意的,但这属于用户可见行为变化,PR 描述未覆盖。 - 受影响调用方/流程: 桌面端与交互式 CLI 的失败路径。
- 最小可行修复: 要么把执行器发起的补偿性回滚单独归类(PR 已把它列为 maintainer 决策,可先只在校验驱动的失败上触发),要么在描述与文档里显式说明新增审批点。
- 证据:
行级发现
- [packages/core/src/executor/executor.ts:183] 这里用完整
validateCode做写盘否决,import_validity的 block 判定(含"后续步骤才创建的相对导入"和路径别名)会导致文件根本不落盘;前置门禁建议只依据syntax_validity,import 检查保留在写盘后或用getCreatedModules()排除计划内模块。 - [packages/core/src/executor/executor.ts:573]
WRITE_ACTIONS含apply_patch,但计划 schema 不产出content,apply_patch 恒返回 null;应识别"覆盖全文件的单个 replace 补丁"并取其content,否则所有 modify 步骤没有前置保护。 - [packages/core/src/executor/executor.ts:245]
!postValidation.pass对纯工具错误同样为 true,使该字段与!stepResult.success同义且与"已在上方尝试过回滚"矛盾;建议收窄为"有落盘且未成功撤销"。 - [packages/core/src/executor/executor.ts:616] 回滚失败只
debugWarn,rollback_started没有终态;在安全层拒绝回滚的 headless 运行里这正是最需要上报的分支,建议补失败事件或把原因写进stepResult.error。 - [packages/core/src/executor/executor.ts:231] 工具自身失败也会发起
rollback,交互模式下会新增一次审批弹窗;请确认该行为并在描述中说明,或只在校验驱动的失败上触发。
Karpathy 评审
- 假设: "apply_patch 只携带补丁片段"这一核心假设与实际实现不符(skill 生成的是全文件 replace),据此得出的"无法前置校验"结论不成立;另一处隐含假设是"guard 的 block 判定足够可靠到可以否决写入",但 import 检查存在已知误报类别。
- 简洁性: 三个新私有方法职责清晰、体量合适;
validate_content阶段对所有 step(含 read_file)都会记一条空 stage,属可接受的噪声。emitValidationFailed用results.some(r => !r.pass)区分"校验拦截"与"工具失败"是隐式契约,但注释已写明,可接受。 - 结构质量: 无明显退化,没有跨层泄漏或薄 wrapper;
emitEvent作为可选出口与既有onSecurityDecision模式一致。 - 变更范围: 聚焦,
agent.ts仅 3 行接线,无无关重构。 - 验证: 3 个新执行器测试 + 1 个 agent 契约测试方向正确,但只覆盖真阳性;两个关键测试 mock 了 guard,回滚测试还额外授予
permissions.allow: ['rollback'],因此验证的是接线而非默认配置下的结果。新门禁的假阳性代价(合法文件不落盘)完全没有测试。
缺失覆盖
create_file内容引用"同计划后续步骤才创建"的相对模块时,文件必须照常写入(针对发现 1 的回归测试)。- 路径别名导入(如
@/components/Card)不应触发写盘否决。 apply_patch步骤内容语法无效时,磁盘上不应留下坏文件——按 issue #387 的验收口径断言文件状态,而不只断言callTool('rollback', ...)被调用。- 默认(未授予
permissions.allow: ['rollback'])非交互配置下回滚被拒的路径:断言失败可观测,且不会出现只有rollback_started的悬空事件序列。 Executor.executeSteps在纯工具失败(非校验失败)时是否应中止余下步骤——progress-enforcement的新语义目前无测试。
| const writeContent = this.resolveWriteContent(step, toolParams); | ||
| const contentValidation = await trace.withStage('validate_content', () => | ||
| writeContent | ||
| ? this.config.hallucinationGuard.validateCode( |
There was a problem hiding this comment.
这里用完整 validateCode 做写盘否决,import_validity 的 block 判定(含"后续步骤才创建的相对导入"和路径别名)会导致文件根本不落盘;前置门禁建议只依据 syntax_validity,import 检查保留在写盘后或用 getCreatedModules() 排除计划内模块。
| content: string; | ||
| language: 'typescript' | 'javascript' | 'json' | 'yaml'; | ||
| } | null { | ||
| if (!WRITE_ACTIONS.includes(step.action)) { |
There was a problem hiding this comment.
WRITE_ACTIONS 含 apply_patch,但计划 schema 不产出 content,apply_patch 恒返回 null;应识别"覆盖全文件的单个 replace 补丁"并取其 content,否则所有 modify 步骤没有前置保护。
| stepResult, | ||
| validation: postValidation, | ||
| needsRollback: !postValidation.pass && step.validation.some((v) => v.required), | ||
| needsRollback: !postValidation.pass, |
There was a problem hiding this comment.
!postValidation.pass 对纯工具错误同样为 true,使该字段与 !stepResult.success 同义且与"已在上方尝试过回滚"矛盾;建议收窄为"有落盘且未成功撤销"。
| if (result.success) { | ||
| this.config.emitEvent?.({ type: 'rollback_completed', snapshotId }); | ||
| } else { | ||
| this.debugWarn(`[Executor] Rollback failed for snapshot ${snapshotId}: ${result.message}`); |
There was a problem hiding this comment.
回滚失败只 debugWarn,rollback_started 没有终态;在安全层拒绝回滚的 headless 运行里这正是最需要上报的分支,建议补失败事件或把原因写进 stepResult.error。
| if (!postValidation.pass) { | ||
| this.emitValidationFailed(postValidation); | ||
| // 回滚不看事件口径:工具失败但已产生快照时同样要把落盘撤销 | ||
| await this.rollbackFailedWrite(toolResult); |
There was a problem hiding this comment.
工具自身失败也会发起 rollback,交互模式下会新增一次审批弹窗;请确认该行为并在描述中说明,或只在校验驱动的失败上触发。
…lback failure observable Addresses both repo-guard review rounds on #402. **The pre-write gate had veto power it should not have had.** It ran the full `validateCode`, so `import_validity` could refuse to write a file — and that check blocks on relative imports of modules a *later* step in the same plan creates, and on path aliases like `@/components/x`. Before this PR those verdicts landed after the write, leaving the file on disk to be reconciled by subsequent steps; as a write veto they would have made multi-file plans unable to produce their first file. Only `syntax_validity` now vetoes a write (`PRE_WRITE_VETO_CHECKS`); every other verdict keeps its old post-write meaning. **apply_patch now gets the same protection as create_file.** The claim that a patch only carries a fragment does not hold on the production path: the plan schema emits no `patches`, so every modify step goes through codegen, which produces a single whole-file `replace` (`startLine: 1, endLine: <original>`). That content is fully known before the write. `resolveWriteContent` now recognises exactly that shape (using `collectedContext.files` for the original line count); partial-line patches still fall through to post-write validation. Since the pre-write result is computed on the same content, it is now reused as the post-write verdict instead of re-running the guard, which removes the `contentValidatedBeforeWrite` mode flag entirely. **`needsRollback` now means what its name says.** It was `!postValidation.pass` — synonymous with "the step failed", including plain tool errors, which made `progress-enforcement` abort every remaining step on one failed `read_file`. It now means "content landed on disk and was not successfully undone". **Rollback failure has a terminal state.** `rollback_failed` joins the event union (and the desktop reducer), the reason is read as `message ?? error`, and it is appended to `stepResult.error`. Previously a denied rollback — the default non-interactive case, i.e. the eval harness — left a dangling `rollback_started` and no way to tell whether the bad file was still on disk. Tests: the five coverage gaps the reviewer named — forward-reference import, path-alias import, invalid whole-file patch blocked before disk, partial patch deferred to post-write, and denied rollback observability. Two existing assertions changed because the behaviour they encoded was the defect: the whole-file invalid patch no longer reaches disk at all, and a successful rollback no longer reports `needsRollback`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR fixes executor validation/rollback ordering and makes validation outcomes observable by emitting validation_failed (plus rollback lifecycle events) into the agent event stream, with regression tests to ensure invalid content is blocked before disk when possible and rolled back otherwise.
Changes:
- Add a pre-write
validate_contenttrace stage that validates fully-resolvable write content before invoking file-write tools. - Wire rollback on post-write validation failure (using tool snapshots) and emit
validation_failed/rollback_started/rollback_completed/rollback_failed. - Add executor + agent contract tests and surface rollback failure status in the desktop UI reducer.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/core/src/types.ts | Extends AgentEvent with rollback_failed for rollback terminal-state observability. |
| packages/core/src/executor/types.ts | Adds ExecutorConfig.emitEvent outlet and new validate_content trace stage. |
| packages/core/src/executor/executor.ts | Implements pre-write content validation, rollback-on-failed-validation, and event emission. |
| packages/core/src/executor/executor.test.ts | Adds regression tests for pre-write blocking, single-pass validation, rollback behavior, and event emission semantics. |
| packages/core/src/agent/agent.ts | Wires executor emitEvent into the agent’s event stream via this.emit(...). |
| packages/core/src/agent/agent.test.ts | Adds contract test ensuring executor validation_failed surfaces through agent events. |
| apps/desktop/src/state/executionReducer.ts | Logs rollback_failed so rollback has a terminal UI state (not just rollback_started). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| /** 会把内容写到磁盘的动作——校验必须发生在调用它们之前 */ | ||
| const WRITE_ACTIONS = ['apply_patch', 'create_file']; |
| let rollbackOutcome: { leftOnDisk: boolean; error?: string } = { leftOnDisk: false }; | ||
| if (!postValidation.pass) { | ||
| this.emitValidationFailed(postValidation); | ||
| // 回滚不看事件口径:工具失败但已产生快照时同样要把落盘撤销 | ||
| rollbackOutcome = await this.rollbackFailedWrite(toolResult); | ||
| } |
| // 字段名说的是「需要回滚」,语义就该是「有内容落了盘且没被成功撤销」, | ||
| // 而不是「这一步失败了」——后者与 !stepResult.success 同义,且会让 | ||
| // progress-enforcement 因为一次 read_file/run_command 失败就中止整个剩余计划。 | ||
| needsRollback: rollbackOutcome.leftOnDisk, |
| const types = events.map((event) => event.type); | ||
| expect(types).toContain('rollback_started'); | ||
| // 关键:不能出现只有 started 没有终态的悬空序列 | ||
| expect(types).toContain('rollback_failed'); | ||
| expect(result.stepResult.error).toContain('still on disk'); | ||
| // 坏文件仍在磁盘上 → 后续步骤不能在这个状态上继续 |
There was a problem hiding this comment.
代码评审报告: fix(executor): validate writes before disk, wire rollback, emit validation_failed
风险等级: 高
处理建议: 请求修改
决策摘要: 前置门禁(#387 主体)与事件出口(#388)本身是对的,但写盘后判定复用了未收窄的完整校验结果去触发回滚——import_validity 对「后续步骤才创建的模块」和路径别名必然判 block(packages/hallucination-guard/src/checks/import-validity.ts:42,87),一旦按 PR 自己推荐的 permissions.allow: ['rollback'] 打开自动回滚,刚写好的合法文件会被 create 快照回滚直接 unlink 删除(packages/mcp-file/src/snapshot.ts:113-123);这与本 PR 收窄前置否决权的理由自相矛盾,且现有测试因 mock 未返回 snapshotId 而观察不到。修掉回滚触发口径后可再评。
级联分析
- 变更符号:
Executor.executeStep、Executor.validateAfterExecution(新增第 4 参)、新增resolveWriteContent/emitValidationFailed/rollbackFailedWrite/narrowToPreWriteVeto/resolveFullFileReplaceContent、ExecutorConfig.emitEvent、ExecutorTraceStage.name新成员validate_content、AgentEvent新成员rollback_failed、FrontAgent构造期事件接线、consoleReducer。 - 受影响流程:
executeStep → validate_content → call_tool → validate_after → rollbackFailedWrite → callTool('rollback') → SecurityManager.evaluate(packages/core/src/security.ts:283判ask);emitEvent → FrontAgent.emit →桌面 reducer / CLI bridge / VS Code state;Executor.rollback由零调用方变为执行路径上的活代码。 - 变更集外调用方:
packages/core/src/executor/progress-enforcement.ts:42——needsRollback唯一消费者,语义变更直接改变它的中止条件(text search + 直接读取确认)。packages/core/src/executor/phase-runner.ts/step-feedback-runner.ts—— 不读该字段;而 agent 实际走的是executeStepsWithErrorFeedback(packages/core/src/agent/agent.ts:813→executor.ts:865),所以needsRollback的改动在 agent 主路径上不生效,PR 描述中「更多运行会中止剩余步骤」的影响面被高估。AgentEvent新增成员:apps/cli/src/ui/bridge.ts:100、apps/vscode/src/state.ts:413都有default分支,联合类型扩展不破坏这两处;桌面 reducer 已同步补 case。- 事件消费者
benchmarks/eval/report.mjs:74依赖validation_failed计数,本 PR 会首次让它非零。
- 置信度: medium(无代码图谱工具输出,结论来自仓库内直接读取 + 文本检索;
validate_content/needsRollback/rollback的调用方已逐个打开确认,未确认的是各 app 里非 switch 形式的事件订阅者)
问题发现
-
[致命] 写盘后回滚触发口径未收窄,合法文件会被删除
- 证据:
executor.ts:292把未收窄的rawContentValidation传给validateAfterExecution,executor.ts:737-739原样返回,于是postValidation含import_validity的 block 项;executor.ts:297-301只看!postValidation.pass就调用rollbackFailedWrite。而import-validity.ts:42把路径别名@/components/Card判为「包未安装 / block」,import-validity.ts:87把尚未创建的相对模块判为 block——这正是executor.ts:249-251与测试executor.test.ts:860-862明确认定「合法、应当落盘」的两类内容。create_file成功时必返回snapshotId(packages/mcp-file/src/tools/create-file.ts:88),snapshot.ts:113-123对create快照的回滚是unlinkSync。 - 受影响调用方/流程: 所有走
executeStep的引擎(phase-runner、progress-enforcement、langgraph runner),即 agent 主路径也在内。默认 headless 下SecurityManager(security.ts:283)拒绝 rollback,表现为每个含别名/前向 import 的写入都刷rollback_started + rollback_failed且needsRollback=true;一旦按本 PR follow-up #1 推荐配置permissions.allow: ['rollback'],同样的场景变成刚创建的合法文件被删除,后续步骤在文件不存在的状态上继续。 - 最小可行修复: 让回滚触发口径与否决口径一致——用
narrowToPreWriteVeto(postValidation).pass === false(或同一PRE_WRITE_VETO_CHECKS过滤)作为rollbackFailedWrite的前置条件,import 类失败仍照旧只让步骤失败、不撤销落盘。
- 证据:
-
[高]
apply_patch步骤上的content参数会顶掉真实补丁内容,反而使写入内容完全不被校验- 证据:
executor.ts:660-662对两个写动作一律优先取toolParams.content ?? step.params.content,只有在取不到时才回落resolveFullFileReplaceContent。LLM 计划的 params 是原样透传的(packages/core/src/planner.ts:323),apply_patch技能又整体 spread 原 params(packages/core/src/skills/executor-skills.ts:286-289),因此{path, content, patches:[生成的补丁]}是可达状态。此时前置校验校的是那份不会被写入的content,通过后executor.ts:737-739又把这份结果当作写盘后结论复用——真正落盘的patches[0].content一次都不会过 guard。 - 受影响调用方/流程: 所有
apply_patch步骤;效果是 #387 想堵的漏洞在这条分支上被静默放大(原实现至少还会在写盘后校验result.content)。 - 最小可行修复: 按动作分派——
create_file用content,apply_patch只用resolveFullFileReplaceContent,不接受content兜底。
- 证据:
-
[中]
needsRollback的语义改动建立在一个与仓库不符的前提上- 证据: PR 描述称「LLM 计划普遍
validation: [],所以旧的some(v => v.required)几乎恒假,这就是 #387 的机制」。但packages/core/src/planner.ts:325对 LLM 计划的每一步都用getDefaultValidation(action)覆写 validation,planner.ts:367-374对create_file/apply_patch固定返回两条required: true——写步骤上旧的第二个合取项恒真。validation: []只出现在step-callbacks.ts生成的修复/恢复步骤上,而这些步骤走 phase-runner,根本不读needsRollback。 - 受影响调用方/流程:
progress-enforcement.ts:42是唯一消费者。按新语义,「写盘后校验失败 + 回滚成功」从「中止剩余步骤」变成「继续执行」,而此时create_file的回滚意味着文件已被删除,后续依赖步骤会在缺文件状态下继续(executor-skills.ts:291-298还会把file not found in context判为可跳过,步骤反而记成功)。 - 最小可行修复: 要么保留「写盘后校验失败即中止」的中止语义、另用独立字段表达 leftOnDisk,要么在描述中更正前提并说明为何「回滚成功后继续」对
create_file也安全;无论哪种,PR 描述里已不存在的needsRollback = !postValidation.pass段落需要改写。
- 证据: PR 描述称「LLM 计划普遍
-
[中]
Executor.rollback的返回类型是错的,新调用方用 cast 绕过而不是修掉- 证据:
executor.ts:947声明返回{ success: boolean; message: string },但安全层拒绝时tool-call-handler.ts:62-69返回的是{ success:false, error },没有message。executor.ts:710因此写成result.message ?? (result as { error?: string }).error ?? 'unknown error'。类型契约与实际形状不符,下一个调用方还会踩同一个坑。 - 受影响调用方/流程:
rollbackFailedWrite,以及未来任何Executor.rollback的消费者(该方法此前零调用方,本 PR 让它上线)。 - 最小可行修复: 在
rollback()内把工具返回归一化为{ success, message }(把error并入message),或把签名改成{ success: boolean; message?: string; error?: string },让调用点不再 cast。
- 证据:
行级发现
- [packages/core/src/executor/executor.ts:300] 回滚以未收窄的
postValidation为触发条件,import_validity(前向引用 / 路径别名,必判 block)会导致刚落盘的合法文件被回滚删除;应改用与PRE_WRITE_VETO_CHECKS一致的收窄结果作为回滚触发条件。 - [packages/core/src/executor/executor.ts:661]
content兜底对apply_patch同样生效,会校验一份不会被写入的内容并把该结果复用为写盘后结论,导致真实补丁内容零校验;按动作分派:create_file用content,apply_patch只用resolveFullFileReplaceContent。 - [packages/core/src/executor/executor.ts:322] 新语义的依据(旧
step.validation恒空)与planner.ts:325/367-374不符,写步骤旧条件恒真;此改动实际把「校验失败即中止」放宽为「回滚成功即继续」,而 create 回滚会删文件,需重新论证或改用独立字段。 - [packages/core/src/executor/executor.ts:710] 这里 cast 出
error是在绕开Executor.rollback声明错误的返回类型(executor.ts:947承诺message: string,安全层拒绝时只有error);应在rollback()内归一化返回形状而不是在调用点断言。 - [packages/core/src/executor/executor.test.ts:891] 该用例的
callToolmock 返回{ success: true }且无snapshotId,恰好绕开rollbackFailedWrite,所以观察不到「合法前向引用文件被回滚」这一真实行为;mock 应返回snapshotId并断言未发生rollback调用。
Karpathy 评审
- 假设: 两处关键假设未成立——「LLM 计划 validation 恒空」被
planner.ts:325/367-374证伪;「写盘后判定时文件还在磁盘、后续步骤补齐即可自洽」(executor.ts:249-251)被同一 PR 新接的回滚破坏。另外描述中的参数名contentValidatedBeforeWrite、表达式needsRollback = !postValidation.pass与最终代码不一致,需同步。 - 简洁性:
narrowToPreWriteVeto/resolveFullFileReplaceContent是纯函数、边界清楚,是合理的抽象;问题不在层数,而在收窄逻辑只用在写盘前、没用在回滚触发点,形成了同一份结果的两套口径。 - 结构质量: 新增逻辑集中在 executor 内、复用了既有
rollback()/事件出口,未见薄 wrapper 或文件规模跨界(executor.ts 约 960 行);扣分项是executor.ts:710的 cast 掩盖了rollback()的错误契约,以及resolveWriteContent用「动作无关」的取值顺序处理两个语义不同的写动作。 - 变更范围: 与 #387/#388 目标相符,无无关重构;
rollback_failed事件 + 桌面 reducer 是补齐终态的必要配套,不算范围蔓延。 - 验证: 事件序列、单次校验、前置拦截都有直接断言,pre-write 用例还用了真实 guard 和评测样本,质量高;但两条「不否决」用例(891、927)只断言了写工具被调用和错误文案,未断言步骤结果与是否发生回滚,恰好放过了本报告的第 1 项。
缺失覆盖
- 合法前向引用 / 路径别名 + 写工具返回
snapshotId+permissions.allow: ['rollback']:断言文件仍在磁盘上、未调用rollback、needsRollback为 false。 apply_patch步骤 params 同时带content与生成的patches:断言被校验的是补丁内容而非content。progress-enforcement层面的端到端用例:写步骤校验失败且回滚成功后,剩余步骤是继续还是跳过——当前只有 executor 单步断言,中止语义变更没有任何调用方级覆盖。Executor.rollback在安全层拒绝时的返回形状用例(锁住message/error归一化后的契约)。
| if (!postValidation.pass) { | ||
| this.emitValidationFailed(postValidation); | ||
| // 回滚不看事件口径:工具失败但已产生快照时同样要把落盘撤销 | ||
| rollbackOutcome = await this.rollbackFailedWrite(toolResult); |
There was a problem hiding this comment.
回滚以未收窄的 postValidation 为触发条件,import_validity(前向引用 / 路径别名,必判 block)会导致刚落盘的合法文件被回滚删除;应改用与 PRE_WRITE_VETO_CHECKS 一致的收窄结果作为回滚触发条件。
| } | ||
|
|
||
| const content = | ||
| ((toolParams.content ?? step.params.content) as string | undefined) ?? |
There was a problem hiding this comment.
content 兜底对 apply_patch 同样生效,会校验一份不会被写入的内容并把该结果复用为写盘后结论,导致真实补丁内容零校验;按动作分派:create_file 用 content,apply_patch 只用 resolveFullFileReplaceContent。
| // 字段名说的是「需要回滚」,语义就该是「有内容落了盘且没被成功撤销」, | ||
| // 而不是「这一步失败了」——后者与 !stepResult.success 同义,且会让 | ||
| // progress-enforcement 因为一次 read_file/run_command 失败就中止整个剩余计划。 | ||
| needsRollback: rollbackOutcome.leftOnDisk, |
There was a problem hiding this comment.
新语义的依据(旧 step.validation 恒空)与 planner.ts:325/367-374 不符,写步骤旧条件恒真;此改动实际把「校验失败即中止」放宽为「回滚成功即继续」,而 create 回滚会删文件,需重新论证或改用独立字段。
| // 默认非交互配置下安全层会拒绝 rollback,返回的是 { success:false, error }—— | ||
| // 只读 message 会打成 undefined。这条分支恰恰是 headless 运行里最需要上报的: | ||
| // 没有终态事件,调用方会停在「回滚开始」,无法判定坏文件是否还在磁盘上。 | ||
| const reason = result.message ?? (result as { error?: string }).error ?? 'unknown error'; |
There was a problem hiding this comment.
这里 cast 出 error 是在绕开 Executor.rollback 声明错误的返回类型(executor.ts:947 承诺 message: string,安全层拒绝时只有 error);应在 rollback() 内归一化返回形状而不是在调用点断言。
| makeExecutionContext(), | ||
| ); | ||
|
|
||
| expect(callTool).toHaveBeenCalledTimes(1); |
There was a problem hiding this comment.
该用例的 callTool mock 返回 { success: true } 且无 snapshotId,恰好绕开 rollbackFailedWrite,所以观察不到「合法前向引用文件被回滚」这一真实行为;mock 应返回 snapshotId 并断言未发生 rollback 调用。
…op content from shadowing patches Third repo-guard round on #402. Two of these were defects I introduced. **[fatal] The rollback trigger was never narrowed.** The pre-write veto was limited to `syntax_validity` precisely because `import_validity` blocks on forward references and path aliases — content this PR deliberately lets reach disk. But the *post-write* path passed the un-narrowed result straight into `rollbackFailedWrite`, and a `create` snapshot rollback is `unlinkSync`. With the `permissions.allow: ['rollback']` this PR's own follow-up recommends, a legitimate file importing a module a later step creates would have been written and then **deleted**. The rollback trigger now uses the same `PRE_WRITE_VETO_CHECKS` narrowing as the veto. The two tests covering those cases could not have caught it: their `callTool` mock returned no `snapshotId`, which short-circuits `rollbackFailedWrite`. They now return one, grant the rollback permission, and assert no rollback happens. **[high] `content` shadowed the real patch.** `resolveWriteContent` preferred `toolParams.content` for *both* write actions, but plan params are passed through verbatim and the apply_patch skill spreads them, so `{path, content, patches}` is reachable. The guard would then validate a string that never gets written while the actual patch content went unchecked — worse than before the PR. Resolution is now dispatched by action. **`needsRollback`'s premise was wrong.** The description claimed LLM plans leave `validation` empty; `planner.ts:325` overwrites it via `getDefaultValidation`, which returns two `required: true` rules for both write actions — the old second conjunct was always true on write steps. The abort semantics are therefore restored (a real check failing aborts remaining steps), minus plain tool failures, which was round one's finding. "A write is still on disk" moved to a separate `writeLeftOnDisk` field; conflating a disk-state fact with a scheduling signal is what made the field ambiguous in the first place. **`Executor.rollback` now normalises its return shape.** It promised `{ success, message }` while the security layer's denial returns `{ success: false, error }`, so the new caller had to cast. Normalised inside `rollback()` instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
代码评审报告: fix(executor): validate writes before disk, wire rollback, emit validation_failed
风险等级: 中
处理建议: 请求修改
决策摘要: 前置写盘门禁(#387 的首选方案)实现正确且用真实 guard 验证过,可以合并;但"回滚"这半边在仓库内两个真实 file client 上不可达,其两条测试只因 mock 返回了真实工具从不返回的 content 字段才通过,且 PR 描述中 needsRollback 的表达式与合入代码不一致——建议先补齐这两点再合。
级联分析
- 变更符号:
Executor.executeStep、Executor.validateAfterExecution(私有,1 调用方)、新增resolveWriteContent/rollbackFailedWrite/emitValidationFailed/narrowToPreWriteVeto/resolveFullFileReplaceContent、Executor.rollback(返回形状归一化)、ExecutorConfig.emitEvent、ExecutorOutput.writeLeftOnDisk、AgentEvent联合新增rollback_failed、ExecutorTraceStage新增validate_content。 - 受影响流程:
Agent.execute → executeStepsWithErrorFeedback → PhaseRunner(生产主路径,读stepResult.success,不读needsRollback);Executor.executeSteps → progress-enforcement(benchmark 路径,唯一needsRollback消费者,progress-enforcement.ts:42);FileMCPClient.callTool(packages/runtime-node/src/mcp-clients.ts:33、apps/cli/src/mcp-client.ts:42);desktopconsoleReducer。 - 变更集外调用方:
progress-enforcement.ts:42(needsRollback,语义已变,未同步更新,见发现 2);apps/desktop/src/state/executionReducer.ts已处理新事件;未发现其它AgentEvent穷举 switch 消费者(text search)。Executor.rollback返回形状归一化与SnapshotManager.rollback(packages/mcp-file/src/snapshot.ts:106,返回{success,message})及安全层拒绝形状{success:false,error}(tool-call-handler.ts:63)均对得上——已核实无回归。 - 置信度: medium(无代码图谱可用,以上均为 Read/Grep 文本核查;被改符号的调用方数量少且已逐一确认)
问题发现
-
[高] 写盘后校验 + 回滚这条链路在仓库内两个真实 file client 上不可达,两条"证明回滚生效"的测试建立在虚构的工具返回形状上
- 证据:
validateAfterExecution的写盘后内容来源是result.content ?? toolParams.content ?? step.params.content(executor.ts:752-756)。而createFile返回{success, path, snapshotId}(packages/mcp-file/src/tools/create-file.ts:85),applyPatch返回{success, diff, validation, snapshotId}(apply-patch.ts:132),都不返回content;两个真实 client 均直接透传这两个对象(packages/runtime-node/src/mcp-clients.ts:37-47、apps/cli/src/mcp-client.ts:42-54)。局部行补丁的toolParams.content/step.params.content也不存在,于是写盘后validateCode根本不会被调用,postValidation恒为{pass:true}。反过来,能解析出前置内容的场景(create_file、整文件 replace)语法失败已在写盘前 return,写盘后只可能剩import_validity失败,而executor.ts:303的收窄门禁又刻意不为它触发回滚。合起来:rollbackFailedWrite在这两个 client 下没有可达触发点。executor.test.ts:772-776与1130附近的 mock 通过返回content: 'export const a = {'人为制造了这条路径。 - 受影响调用方/流程:
Agent.execute全部写步骤;#387 中"补丁写入的非法内容不得留在磁盘"这一条对局部行补丁仍不成立;PR follow-up #1 把"回滚不生效"归因于安全层,实际即使授予permissions.allow:['rollback']也不会触发。 - 最小可行修复: 在
validateAfterExecution的写步骤分支里,当没有前置内容且工具未返回content时从磁盘读回已写文件再校验(路径已知、快照已存在,改动约 3 行);并把两条回滚测试的 mock 换成真实工具的返回形状({success, snapshotId}+ 磁盘上确有坏文件),否则它们证明不了任何生产行为。若决定不做,请把 rollback 部分明确降级为"面向外部 MCP client 的兜底"并在 PR 描述里更正 follow-up #1 的归因。
- 证据:
-
[中]
needsRollback的合入语义与 PR 描述不一致,且顺带改变了非写步骤的中止语义- 证据: 描述称新表达式为
!postValidation.pass(更宽),合入代码是postValidation.results.some((r) => !r.pass)(executor.ts:328,更窄)。工具自身失败时validateAfterExecution返回results: [](executor.ts:740-744),因此纯工具失败一律不再中止。planner 给read_file的默认校验是[{type:'file_exists', required:true}](planner.ts:376),旧条件下这类步骤的非 skippable 工具失败会中止剩余计划(progress-enforcement.ts:42),现在不会。描述里"Behavioural change worth reviewer attention"一节讲的是相反方向的放宽。同样过时的还有"apply_patch … returns null and keeps the old post-write path"(实际已支持整文件 replace 前置校验)和contentValidatedBeforeWrite布尔参数(实际是preWriteContentValidation: ValidationResult)。 - 受影响调用方/流程:
progress-enforcement.ts:42(benchmark 与Executor.executeSteps的外部调用方);主 agent 路径不读该字段,影响有限。 - 最小可行修复: 更新 PR 描述为合入表达式,并显式写明"纯工具失败不再中止剩余步骤"这一方向的变化;
read_file工具失败后续步骤继续执行的新行为建议补一条executeSteps级别的测试(目前只在单步层面断言needsRollback === false)。
- 证据: 描述称新表达式为
-
[中]
ExecutorOutput.writeLeftOnDisk是新增公共契约字段,但没有任何生产消费者- 证据: 全仓仅
types.ts:741(定义)、executor.ts:329(赋值)与测试引用(Grep 全仓)。唯一读调度信号的progress-enforcement.ts:42只看needsRollback,主路径的PhaseRunner两个字段都不读。"坏文件仍在磁盘"这一事实对调用方只能通过stepResult.error的子串(executor.ts:314-316)获得。 - 受影响调用方/流程: 任何依赖
ExecutorOutput的下游都要多理解一个永不被读的字段;与 #386 "dead config" 属同一类问题。 - 最小可行修复: 二选一——让
progress-enforcement/PhaseRunner在writeLeftOnDisk为真时中止(这恰是最该中止的一种失败),或删除该字段,保留 error 文案与rollback_failed事件。
- 证据: 全仓仅
-
[中]
validation_failed无法区分"写盘前拦截"与"写盘后失败但文件仍在",而这正是 #388 要统计的量- 证据: 三个发射点(
executor.ts:206、255、298)都发同一形状{type:'validation_failed', result}(emitValidationFailed,executor.ts:688-692)。#388 要求带 check type、path 与 blocking reason;type/reason 在results[].type/blockedBy里可取,path 只存在于 message 字符串与details中,且没有 stage/stepId。于是benchmarks/eval/report.mjs:75的sumEvent(..., 'validation_failed')统计出的数字会把"被前置门禁挡下的真拦截"和"import 前向引用导致的步骤失败(有意放行落盘)"混在一起。 - 受影响调用方/流程: 评测报告脚本、desktop 日志、任何以该事件度量 guard 有效性的消费者。
- 最小可行修复: 事件加一个判别字段(如
stage: 'pre_execution' | 'pre_write' | 'post_write')与path/stepId,desktop reducer 与 report 脚本按 stage 分开计数。
- 证据: 三个发射点(
-
[低] agent 侧"契约测试"绕过公共 API,测不到它声称的契约
- 证据:
agent.test.ts:240用(agent as unknown as { executor: Executor }).executor取私有字段后直接调executeStep。若将来emitEvent的接线从构造期移走、或 executor 字段改名,该 cast 仍能编译,测试仍会绿。 - 受影响调用方/流程: #388 的核心验收("执行器事件能否浮到 agent 事件流")没有真正的回归保护。
- 最小可行修复: 通过
agent.addEventListener+ 公开执行入口驱动一次写步骤;若成本过高,至少断言 executor config 上的emitEvent与agent.emit同源。
- 证据:
-
[低]
executor.test.ts由 664 行增至 1151 行,跨过可扫描边界- 证据: 新增三个内聚的 describe(
write validation/pre-write veto scope/rollback observability),共约 480 行。 - 最小可行修复: 抽到
executor.validation.test.ts,与既有makeConfig/makeStephelper 共享同一 fixture 模块。
- 证据: 新增三个内聚的 describe(
行级发现
- [packages/core/src/executor/executor.ts:752] 这条写盘后分支依赖
result.content/toolParams.content,但真实create_file/apply_patch都不返回content、局部行补丁也没有content参数,因此局部补丁写盘后实际不做任何内容校验;建议在此处按已知 path 从磁盘读回内容再validateCode。 - [packages/core/src/executor/executor.ts:303] 收窄后的回滚门禁只认
syntax_validity,而语法失败已在写盘前 return,所以这个分支在仓库内两个 file client 下没有可达触发点;修复发现 1 的写盘后内容来源后此门禁才有意义,否则请注明它只服务于外部 MCP client。 - [packages/core/src/executor/executor.ts:328] 合入表达式
results.some(r => !r.pass)比 PR 描述的!postValidation.pass窄:纯工具失败(results: [])不再中止剩余计划,而 planner 给read_file的file_exists是required:true,旧条件下会中止;请更新描述并补executeSteps级别回归测试。 - [packages/core/src/executor/executor.ts:690] 三个发射点共用同一事件形状,写盘前拦截与写盘后失败不可区分;建议加
stage与path字段,否则benchmarks/eval/report.mjs统计出的拦截数会混入有意放行的 import 失败。 - [packages/core/src/types.ts:741] 新增的
writeLeftOnDisk没有任何生产读者(progress-enforcement.ts:42只读needsRollback,PhaseRunner两者都不读);要么让调度层消费它,要么删除并保留 error 文案与rollback_failed事件。 - [packages/core/src/executor/executor.test.ts:774] mock 让写工具返回
content,真实applyPatch/createFile从不返回该字段,因此这条"回滚生效"的用例证明不了生产行为;请改用{success, snapshotId}并让坏文件真实落盘。 - [packages/core/src/agent/agent.test.ts:240] 通过 cast 取私有
executor字段绕过了公共 API,接线方式变更时该测试不会失败;建议改由公开执行入口 +addEventListener驱动。
Karpathy 评审
- 假设: 前置门禁默认"整文件 replace 是 modify 步骤的唯一形态",这一点对 codegen 路径成立(
executor-skills.ts:278-288),但resolveFullFileReplaceContent依赖collectedContext.files中的原文件行数与工具端一致——两者同源,已核实。未成立的假设是"局部补丁会落回写盘后判定"(见发现 1)。 - 简洁性:
narrowToPreWriteVeto单成员白名单 + 两处复用是合理的收敛点,注释解释了为何只给syntax_validity否决权,可接受;resolveWriteContent按动作分派、不做content跨动作兜底是正确取舍。回滚那一支则是当前不可达的额外机制 + 一个新事件 + 一个无消费者的输出字段,属于净增复杂度。 - 结构质量:
executor.ts从 ~750 增至 973 行,未越界;两个模块级纯函数抽取得当。退化点集中在ExecutorOutput多了一个无人读的字段,以及事件语义未分层。 - 变更范围: 大体聚焦;
needsRollback对非写步骤中止语义的改动超出 #387/#388 的目标,且未在描述中如实说明。 - 验证: 前置门禁用真实
HallucinationGuard+ 评测集真实失败样本验证,质量高(executor.test.ts的 fence/.tsx、别名 import、前向引用三例都切中要害)。回滚相关两例建立在虚构返回形状上,置信度低。
缺失覆盖
- 局部行补丁写入语法非法内容后的真实端到端用例:用真实
FileMCPClient执行,断言磁盘上的文件内容(当前无任何测试覆盖"补丁写入的坏内容被发现或被撤销")。 Executor.executeSteps层面的中止语义回归:read_file工具失败后,后续步骤是否应继续(新行为)——目前只有单步needsRollback断言。create_file覆写既有文件(overwrite: true,快照类型为modify)时前置门禁拦截后原文件保持不变的断言。- desktop
consoleReducer对rollback_failed分支的 reducer 测试(新增 case 无测试覆盖)。
| return preWriteContentValidation; | ||
| } | ||
|
|
||
| if (WRITE_ACTIONS.includes(step.action)) { |
There was a problem hiding this comment.
这条写盘后分支依赖 result.content/toolParams.content,但真实 create_file/apply_patch 都不返回 content、局部行补丁也没有 content 参数,因此局部补丁写盘后实际不做任何内容校验;建议在此处按已知 path 从磁盘读回内容再 validateCode。
| // 后续步骤才创建的相对模块」和路径别名必然判 block——那正是上面刻意 | ||
| // 放行落盘的两类内容。若按未收窄的结果回滚,`create` 快照的回滚是 | ||
| // unlinkSync,等于把刚写好的合法文件删掉,与放行它的理由直接矛盾。 | ||
| if (!narrowToPreWriteVeto(postValidation).pass) { |
There was a problem hiding this comment.
收窄后的回滚门禁只认 syntax_validity,而语法失败已在写盘前 return,所以这个分支在仓库内两个 file client 下没有可达触发点;修复发现 1 的写盘后内容来源后此门禁才有意义,否则请注明它只服务于外部 MCP client。
| // 旧条件在写步骤上恒真),但排除纯工具失败:`validateAfterExecution` 对 | ||
| // 工具报错返回 results 为空的失败结果,把它算进来会让一次 read_file 失败 | ||
| // 就中止整个剩余计划。 | ||
| needsRollback: postValidation.results.some((result) => !result.pass), |
There was a problem hiding this comment.
合入表达式 results.some(r => !r.pass) 比 PR 描述的 !postValidation.pass 窄:纯工具失败(results: [])不再中止剩余计划,而 planner 给 read_file 的 file_exists 是 required:true,旧条件下会中止;请更新描述并补 executeSteps 级别回归测试。
| */ | ||
| private emitValidationFailed(validation: ValidationResult): void { | ||
| if (validation.results.some((result) => !result.pass)) { | ||
| this.config.emitEvent?.({ type: 'validation_failed', result: validation }); |
There was a problem hiding this comment.
三个发射点共用同一事件形状,写盘前拦截与写盘后失败不可区分;建议加 stage 与 path 字段,否则 benchmarks/eval/report.mjs 统计出的拦截数会混入有意放行的 import 失败。
| * 与 needsRollback 分开表达:后者是「要不要中止后续步骤」的调度信号, | ||
| * 这个是「磁盘处于什么状态」的事实,两者在回滚成功时会分叉。 | ||
| */ | ||
| writeLeftOnDisk?: boolean; |
There was a problem hiding this comment.
新增的 writeLeftOnDisk 没有任何生产读者(progress-enforcement.ts:42 只读 needsRollback,PhaseRunner 两者都不读);要么让调度层消费它,要么删除并保留 error 文案与 rollback_failed 事件。
| } | ||
| return Promise.resolve({ | ||
| success: true, | ||
| content: 'export const a = {', |
There was a problem hiding this comment.
mock 让写工具返回 content,真实 applyPatch/createFile 从不返回该字段,因此这条"回滚生效"的用例证明不了生产行为;请改用 {success, snapshotId} 并让坏文件真实落盘。
|
|
||
| // 执行器的校验事件必须经 emitEvent 汇入 agent 的事件流, | ||
| // 否则「校验是否拦截」在遥测层不可观测(issue #388) | ||
| const executor = (agent as unknown as { executor: Executor }).executor; |
There was a problem hiding this comment.
通过 cast 取私有 executor 字段绕过了公共 API,接线方式变更时该测试不会失败;建议改由公开执行入口 + addEventListener 驱动。
…s reachable Fourth repo-guard round on #402. The reviewer found the rollback half of this PR was unreachable in production, and that my own tests hid it. **Post-write content validation never ran on real clients.** It sourced content from `result.content ?? toolParams.content ?? step.params.content`, but the real `createFile` returns `{success, path, snapshotId}` and `applyPatch` returns `{success, diff, validation, snapshotId}` — neither carries `content`, and a partial-line patch has no `content` param either. So for the one case that genuinely needs post-write checking (partial patches, whose result is unknown before the write), the guard was never invoked and `rollbackFailedWrite` had no reachable trigger. The two rollback tests passed only because their mock returned a `content` field no real tool produces. The file is now read back from disk by path. The new test uses the real tools' return shape, writes to a real temp dir, and asserts the file's contents are restored — so it exercises the production path rather than a fiction. **`validation_failed` now carries `stage` (+ `path`, `stepId`).** The three emit sites mean different things — a pre-write interception is a bad file that never landed, a post-write failure is one that did — and `report.mjs` was summing them into a single "interception count". #388 asked for check type, path and reason; type and reason were already reachable through `results`/`blockedBy`, path was only inside a message string. The desktop reducer now labels the stage too. **`writeLeftOnDisk` has a consumer.** It was a field nothing read — the same dead-config shape this PR series keeps fixing. `progress-enforcement` now aborts on it, which is the state that most warrants aborting: a known-bad file is sitting in the workspace and later steps would build on it. Also: `executor.test.ts` was up to 1151 lines, so the validation suites moved to `executor.validation.test.ts`; the agent contract test now asserts the `emitEvent` wiring exists before reaching through to the private executor, so a broken wire fails before the cast can hide it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
代码评审报告: fix(executor): validate writes before disk, wire rollback, emit validation_failed
风险等级: 高
处理建议: 请求修改
决策摘要: ** 事件接线、回滚终态、needsRollback 拆分都是对的,但把 syntax_validity 升级为写盘否决权 + 回滚触发权时没有校准该检查的可靠性——它是逐行引号奇偶 + 括号栈的启发式(packages/hallucination-guard/src/checks/syntax-validity.ts:190-216),对 "it's"、多行模板字符串、JSX 文本必然误判 block,合并后这类合法文件将根本写不出来,且已落盘的合法内容会被回滚删除/还原。
级联分析
- 变更符号: 原始模型未提供结构化级联字段。
- 受影响流程: 原始模型未提供结构化级联字段。
- 变更集外调用方: unknown
- 置信度: degraded
问题发现
- [高] 写盘否决权授予了启发式语法检查,会阻断合法文件落盘
- 证据:
PRE_WRITE_VETO_CHECKS只放行syntax_validity(executor.ts:32),而该检查的实现是逐行引号奇偶判定 + 朴素括号栈:syntax-validity.ts:194-216对任意单行内奇数个'/"/`直接产出severity: 'block'。const msg = "it's fine";、<p>Don't</p>、跨行模板字符串首行都会命中;checkCommonPatterns自己的 message 都写成 "Possible unclosed …"。HallucinationGuard默认syntaxValidity: true(guard.ts:69),agent 生产路径直接使用(agent/agent.ts:93-97)。 - 受影响调用方/流程: 每一个
create_file与 codegen 整文件replace。改动前这类误判发生在写盘后(文件仍在磁盘、后续步骤/tsc 可裁决),改动后文件根本不产生;LLM 重生成同样内容会再次命中同一启发式,构成确定性失败环。FrontAgent 的主要产物是 React/TSX 组件,命中面很宽。 - 最小可行修复: 把否决资格从「check type」细化到「错误类别」——只对确定性证据放行否决(
checkBrackets的未匹配/未闭合括号、JSON.parse 失败、以及 #387 举证的 markdown 围栏 TS1127 类),排除checkCommonPatterns产出的 "Possible unclosed …" 启发式项;details.errors[].message已带足够信息可判别。
- 证据:
- [高] 同一误判类别现在获得回滚(删除/还原)权
- 证据:
executor.ts:303-304用同一个narrowToPreWriteVeto作为回滚触发口径,配合新增的写盘后读回(executor.ts:788),局部行补丁的最终内容首次进入validateCode。一次引号奇偶误判即触发rollbackFailedWrite,create快照回滚是unlinkSync、modify快照回滚是覆盖还原(packages/mcp-file/src/snapshot.ts:106)。 - 受影响调用方/流程: 交互模式(CLI/desktop)与评测。注意
benchmarks/eval/run-eval.mjs:94传入onApprovalRequest,packages/runtime-node/src/run.ts:182据此把interactive置为 true,因此评测里 rollback 会被自动批准并真正执行——PR 描述中「默认非交互、即评测环境会拒绝回滚」的前提对这个 harness 不成立,评测将实际发生文件删除/还原。 - 最小可行修复: 与 finding 1 共用同一份收窄后的「确定性错误类别」;在此之前不要让回滚由该检查单独驱动。
- 证据:
- [高] 前置否决后不再中止计划,且后续步骤会被记为「跳过成功」
- 证据: 否决路径固定返回
needsRollback: false(executor.ts:266),与本 PR 自述的「真实检查失败 → 中止」不一致。改动前同一失败在写盘后判定,step.validation对写步骤恒为required: true,progress-enforcement.ts:45会中止剩余步骤;改动后计划继续,而目标文件并不存在。后续针对该文件的apply_patch会拿到工具的Cannot apply patch: file does not exist,被skills/executor-skills.ts:291-299判为可跳过,进而由buildSkippedStepOutput记成success: true(executor.ts:355-357)。 - 受影响调用方/流程:
Executor.executeSteps的直接嵌入方——一次运行可能在「零文件产出」的情况下整体呈现为成功。 - 最小可行修复: 否决路径返回
needsRollback: true(磁盘无残留,writeLeftOnDisk保持 false),中止语义与其余真实检查失败一致。
- 证据: 否决路径固定返回
- [中] 复用前置结果作为写盘后判定,会跳过对「误判为整文件」补丁的实际落盘内容校验
- 证据:
executor.ts:773-775一旦存在前置结果就直接返回,读回逻辑(executor.ts:788)被短路。整文件判定依赖collectedContext.files.get(path)的行数(executor.ts:674、resolveFullFileReplaceContent),而step-callbacks.ts:86-106只在read_file/create_file后刷新该 Map——apply_patch成功后不刷新。同一计划内二次修改同一文件时,若真实文件行数多于上下文快照,mcp-file会按startLine:1..staleLines做局部替换并保留尾部(apply-patch.ts:97,114),最终落盘内容 ≠ 被校验的patch.content,而写盘后校验已被复用结果跳过。 - 受影响调用方/流程: 多次修改同一文件的计划;#387 中「非法补丁内容不得留在磁盘」在该分支不成立。
- 最小可行修复: 复用前先比对
readWrittenFile(path)与已校验内容,不一致时按读回内容重新校验(读回逻辑本 PR 已具备,成本仅一次同步读)。
- 证据:
- [低]
validation_failed.path与实际校验对象可能不一致- 证据:
executor.ts:698取step.params.path,而校验路径解析用的是toolParams.path ?? step.params.path(executor.ts:662)。技能改写过 path 的步骤,事件里的 path 会缺失或指向旧值。 - 受影响调用方/流程: #388 的拦截遥测与 desktop 日志中的
[path]前缀。 - 最小可行修复: 把已解析的
writeContent.path传入emitValidationFailed。
- 证据:
行级发现
- [packages/core/src/executor/executor.ts:32] syntax_validity
整类获得写盘否决权,但该检查含逐行引号奇偶启发式("Possible unclosed …"),"it's"`/多行模板/JSX 文本会被判 block;请把否决资格细化到确定性错误类别(括号未闭合、JSON 解析失败、围栏类),排除 "Possible" 启发式项。 - [packages/core/src/executor/executor.ts:266] 前置否决返回
needsRollback: false,使最严重的失败类别不再中止剩余步骤,后续apply_patch会被判为「跳过成功」;应返回true(磁盘无残留由writeLeftOnDisk: false表达)。 - [packages/core/src/executor/executor.ts:304] 回滚触发沿用同一收窄集合,因此同一启发式误判会触发
unlinkSync/覆盖还原;请与否决口径一起收窄到确定性错误类别后再赋予回滚权。 - [packages/core/src/executor/executor.ts:773] 复用前置结果会跳过读回校验;当
collectedContext相对磁盘过期时,被判为「整文件 replace」的补丁实际是局部替换,落盘内容从未被校验——复用前应与readWrittenFile(path)比对,不一致则改用读回内容重校。 - [packages/core/src/executor/executor.ts:698] 事件用
step.params.path,与校验实际使用的toolParams.path ?? step.params.path不同源;传入已解析的writeContent.path更准确。
Karpathy 评审
- 假设: 模型输出需要归一化为固定 Markdown 契约。
- 简洁性: 已提取 summary、finding、evidence 与 fix;原始 prose 不再附在评论中,避免占用下游解析与代理上下文。
- 变更范围: 原始模型未提供结构化范围字段。
- 验证: 需要查看 CI、测试或人工 CR 证据补强合并信心。
缺失覆盖
- 输出未命中 Repo Guard Markdown 契约;建议补充真实模型质量评估覆盖。
| * 而 import_validity 对「同一计划里后续步骤才创建的相对模块」与路径别名会误判 block, | ||
| * 把它纳入否决权会让本来能自洽的多文件计划根本写不出第一个文件。 | ||
| */ | ||
| const PRE_WRITE_VETO_CHECKS = new Set(['syntax_validity']); |
There was a problem hiding this comment.
syntax_validity 整类获得写盘否决权,但该检查含逐行引号奇偶启发式("Possible unclosed …"),"it's"`/多行模板/JSX 文本会被判 block;请把否决资格细化到确定性错误类别(括号未闭合、JSON 解析失败、围栏类),排除 "Possible" 启发式项。
| duration: Date.now() - startTime, | ||
| }, | ||
| validation: contentValidation, | ||
| needsRollback: false, |
There was a problem hiding this comment.
前置否决返回 needsRollback: false,使最严重的失败类别不再中止剩余步骤,后续 apply_patch 会被判为「跳过成功」;应返回 true(磁盘无残留由 writeLeftOnDisk: false 表达)。
| // 放行落盘的两类内容。若按未收窄的结果回滚,`create` 快照的回滚是 | ||
| // unlinkSync,等于把刚写好的合法文件删掉,与放行它的理由直接矛盾。 | ||
| if (!narrowToPreWriteVeto(postValidation).pass) { | ||
| rollbackOutcome = await this.rollbackFailedWrite(toolResult); |
There was a problem hiding this comment.
回滚触发沿用同一收窄集合,因此同一启发式误判会触发 unlinkSync/覆盖还原;请与否决口径一起收窄到确定性错误类别后再赋予回滚权。
| } | ||
|
|
||
| if (['apply_patch', 'create_file'].includes(step.action)) { | ||
| if (preWriteContentValidation) { |
There was a problem hiding this comment.
复用前置结果会跳过读回校验;当 collectedContext 相对磁盘过期时,被判为「整文件 replace」的补丁实际是局部替换,落盘内容从未被校验——复用前应与 readWrittenFile(path) 比对,不一致则改用读回内容重校。
| type: 'validation_failed', | ||
| stage, | ||
| result: validation, | ||
| path: step.params.path as string | undefined, |
There was a problem hiding this comment.
事件用 step.params.path,与校验实际使用的 toolParams.path ?? step.params.path 不同源;传入已解析的 writeContent.path 更准确。
…esults repo-guard's review on #412: the deliverable is a measuring instrument, and it had three ways to emit results that look like findings but are artifacts. **Provenance is now recorded and checked.** Records carry `fixture` and `taskSet`, output files are `<arm>-deep.jsonl` for the deep fixture, and `report-filesense.mjs` reads both and refuses to emit a report if the two arms disagree. Previously the filename carried only the arm, resume-dedup keyed only on `taskId`, and deep task ids do not collide with flat ones — so running both fixtures appended into one file and the report averaged across them while claiming, in its own methodology section, to have used `tasks.json`. **Missing dependencies fail fast.** Without `node_modules`, `setupWorkspace` built a dangling symlink and every `typecheck`/`cmd` check returned non-zero; the only abort heuristic is "zero successful LLM calls", which does not fire when the model is fine. An entire arm of FAIL records would have been recorded as data. A lockfile is committed to match the flat fixture. **The unwired events are labelled.** `validation_failed` and `rollback_failed` have no emit site until #402 lands, so those arrays are necessarily empty on develop — an empty array means "not wired", not "zero interceptions", which is the exact misreading this PR exists to prevent. Rollback collection widened to `rollback_started`/`completed`/`failed` so "undone" and "intercepted but still on disk" are distinguishable. **`options.filesense` now merges instead of replacing.** Whole-object replacement dropped every env-resolved sibling field when a caller overrode one field — the same loss the omitted-case test already guarded against, just via a different door. Regression test included. **The deep fixture's SDD config was a verbatim copy of the flat one**, so its `directory_structure` and `module_boundaries` named `src/components|hooks|utils` — none of which exist here. The deep round would have been nominally "SDD on" with SDD constraining nothing, while three of its tasks test exactly the layering convention that config is supposed to encode. Rewritten for the feature-sliced layout with one-way boundaries. Also: `deep-query-format-convention`'s only assertion echoed its own prompt and would have passed in both arms; the generator now cleans its own subtree (while preserving the committed anchors) instead of leaving orphans that drift the entry count; README documents the arm, the fixture flag, and the generator prerequisite, with a contract test; counts corrected to 242 files / 188 dirs / 430 entries after the planted-bug anchors were added. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… legitimate code Fifth repo-guard round on #402, and the reviewer is right in a way that invalidates the previous approach rather than refining it. `syntax_validity` decides "unclosed string" by counting quotes **per line**. I measured it against the built guard: | input | verdict | |---|---| | `export const msg = "it's fine";` | **block** | | a multi-line template literal | **block** | | `export const P = () => <p>Don't panic</p>;` | **block** | | `export const a = {` (genuinely broken) | block | Granting that check veto power over writes — which is what the last few rounds were carefully narrowing *toward* — would have meant **no file containing an apostrophe inside a string could be written**, and, via the rollback trigger, a legitimate file already on disk could be deleted by a `create` snapshot rollback. That is far worse than the defect #387 describes. The veto predicate is now a markdown fence in a code file. It is high-precision (a line starting with ``` is never valid TS/JS) and it is the failure actually observed in the 2026-07-12 eval — `TS1127: Invalid character`, model prose written into `.tsx`. Everything else returns to post-write handling, which is the pre-PR behaviour. The rollback trigger uses the same predicate against the content that actually landed. Filed #413 for the checker itself; fixing it needs `typescript` as a runtime dependency of the guard, which is a packaging decision that does not belong in this PR. Four regression tests now pin that legitimate code is written — apostrophe in a string, multi-line template, JSX contraction, regex with unbalanced-looking brackets — plus one that the fence is still blocked. Also from this round: - The veto path now returns `needsRollback: true`. It returned false, so the plan continued past a blocked write; a later `apply_patch` on the missing file hits "file does not exist", which `executor-skills` treats as skippable and records as **success**. A run could report success having produced no files. - Reusing the pre-write result as the post-write verdict now first compares against what actually landed. The whole-file判定 depends on a line count from `collectedContext.files`, which `apply_patch` does not refresh — so on a second edit to the same file the tool may replace only the leading lines and keep the tail, making the landed content differ from what was validated. - `validation_failed.path` uses the resolved write path rather than `step.params.path`, which a skill may have rewritten. - The two rollback tests were rebuilt against a real temp directory with the tool actually writing to disk, replacing mocks that returned a `content` field no real tool produces. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
代码评审报告: fix(executor): validate writes before disk, wire rollback, emit validation_failed
风险等级: 高
处理建议: 请求修改
决策摘要: ** 主体方向正确且测试扎实,但 apply_patch 写盘后校验仍会被残留 content 参数遮蔽,导致本 PR 为 #387 声称的补丁路径覆盖在该状态下失效;再加上新公开字段 writeLeftOnDisk 的语义与命名/注释不符,建议这两处修好再合。
级联分析
- 变更符号: 原始模型未提供结构化级联字段。
- 受影响流程: 原始模型未提供结构化级联字段。
- 变更集外调用方: unknown
- 置信度: degraded
问题发现
- [高]
apply_patch的写盘后校验仍被从未落盘的content遮蔽- 证据:
executor.ts:811-815的取值链是result.content ?? toolParams.content ?? step.params.content ?? readWrittenFile(path)。resolveWriteContent(executor.ts:685-692)刻意按 action 分派以避免这个陷阱,注释写明「校验的就是一份不会被写入的内容……比修复前更糟」,但同一份危害在写盘后路径原样保留。planner.ts:323的params: llmStep.params as Record<string, unknown>是自由形状,executor-skills.ts:286-289的 apply_patch 分支返回{...params, patches},所以{path, content, patches}可达——这正是本 PR 自己论证并写了测试的状态。 - 受影响调用方/流程: 所有走
validateAfterExecution的写步骤;直接削弱 #387 在补丁路径上的验收口径。 - 最小可行修复: 写盘后同样按 action 分派——
apply_patch直接用readWrittenFile(path),只对create_file保留toolParams.content/step.params.content。
- 证据:
- [中]
writeLeftOnDisk表达的是「回滚尝试失败」,不是「写入仍在磁盘」- 证据: 该字段只在
rollbackFailedWrite返回leftOnDisk: true时为真,而回滚仅在落盘内容含围栏时才触发(executor.ts:320-323)。写盘后校验因其他原因判失败(未解析的相对 import、guard 语法启发式)时,文件原样留在工作区,字段却是false。types.ts:736-741的注释「已落盘的写入未被成功撤销——坏文件仍在工作区里」和progress-enforcement.ts:42-44的「写入卡在磁盘上没被撤销」都描述了这个更宽的事实,实现只覆盖了子集。 - 受影响调用方/流程: 当前无功能性 bug(
progress-enforcement.ts:45与needsRollback做了 OR),但这是ExecutorOutput上的新公开字段,下一个消费者(例如评测侧统计「有多少坏文件留在盘上」)会按注释理解并读到系统性偏低的值。 - 最小可行修复: 二选一——改名为
rollbackFailed使名实相符;或在「写工具产生了 snapshotId 且写盘后判失败但未回滚」时也置true。
- 证据: 该字段只在
- [中] PR 描述与最终实现不符,且
Fixes #387会关闭尚未满足验收标准的 issue- 证据: 描述称「Only
syntax_validitycan veto a write (PRE_WRITE_VETO_CHECKS)」,但仓内不存在PRE_WRITE_VETO_CHECKS;实际否决判据是detectMarkdownFence(executor.ts:43-65),只拦 markdown 围栏。收窄本身有充分依据——checks/syntax-validity.ts:190-216逐行数引号奇偶,"it's fine"会被判severity: 'block',把它作为写盘否决权确实比缺陷本身更糟,最新提交 c1c6d79 正是为此回退。问题在于评审者据描述作出的合并判断与实际代码不是同一个变更。 - 受影响调用方/流程: #387 的验收标准是「a create/patch step whose content is syntactically invalid must not leave the file on disk」;当前只有围栏子集被拦,其余非法内容照旧落盘且不回滚。
Fixes #387合并即自动关闭。 - 最小可行修复: 更新描述以匹配围栏判据,把关联关系降级为
Relates to #387(或明确保留 issue 并链接 #413 追踪启发式误判),并在描述里显式写出剩余缺口。
- 证据: 描述称「Only
- [低] 否决路径上白跑一次完整 guard
- 证据:
executor.ts:254-262先await validateCode(含会命中文件系统的 import 解析),executor.ts:263-266才算围栏否决;否决触发时rawContentValidation被直接丢弃。 - 受影响调用方/流程: unknown
- 最小可行修复: 先算
buildFenceVeto并提前返回,再对未被否决的内容跑validateCode。
- 证据:
- [低]
validation_failed计数口径未随事件语义更新,#388 的「拦截数」会被前向引用 import 淹没- 证据:
benchmarks/eval/report.mjs:29的sumEvent按事件类型直接累加,不区分stage。而emitValidationFailed('post_write', ...)(executor.ts:316)会在复用的rawContentValidation上触发,checks/import-validity.ts:84-90对「同一计划里后续步骤才创建的相对模块」返回severity: 'block'——这是多文件计划的常态。PR 描述称「the count is usable as an interception count」,但按当前口径统计出的数会由这类良性判定主导。 - 受影响调用方/流程: unknown
- 最小可行修复: 事件已带
stage,在 report 侧按stage === 'pre_write'计拦截数,或在 PR 描述中写明该计数需要按 stage 拆分后才可用。
- 证据:
- [低]
executor.test.ts的全部改动是两个未使用 import- 证据:
existsSync(第 1 行)与AgentEvent(第 8 行)在该文件内再无引用(全文搜索仅命中 import 行),用例应是被搬到了executor.validation.test.ts。biome.json:24-27把noUnusedImports设为warn,所以quality:precommit退出 0 与此并不矛盾。 - 受影响调用方/流程: unknown
- 最小可行修复: 整体还原
executor.test.ts,让该文件不出现在本次 diff 中。
- 证据:
行级发现
- [packages/core/src/executor/executor.ts:815] 新增的读回兜底位于
toolParams.content/step.params.content之后,apply_patch 步骤只要携带残留content就永远走不到这里,落盘补丁内容不受校验;按 action 分派,apply_patch 直接用readWrittenFile(path)。 - [packages/core/src/types.ts:741] 字段实际只表示「回滚尝试且失败」,非围栏类写盘后失败时文件仍在盘上却为
false;改名为rollbackFailed,或在「有 snapshotId、判失败、未回滚」时同样置true。 - [packages/core/src/executor/executor.ts:254] 完整
validateCode(含文件系统 import 解析)在围栏否决判定之前无条件执行,否决路径上结果被丢弃;先算buildFenceVeto并提前返回。 - [packages/core/src/executor/executor.test.ts:1] existsSync
与第 8 行的AgentEvent` 在本文件内均未使用,本文件的唯一改动是这两个残留 import,建议整体还原。
Karpathy 评审
- 假设: 模型输出需要归一化为固定 Markdown 契约。
- 简洁性: 已提取 summary、finding、evidence 与 fix;原始 prose 不再附在评论中,避免占用下游解析与代理上下文。
- 变更范围: 原始模型未提供结构化范围字段。
- 验证: 需要查看 CI、测试或人工 CR 证据补强合并信心。
缺失覆盖
- 输出未命中 Repo Guard Markdown 契约;建议补充真实模型质量评估覆盖。
| (step.params.content as string | undefined); | ||
| const path = step.params.path as string; | ||
| (step.params.content as string | undefined) ?? | ||
| this.readWrittenFile(path); |
There was a problem hiding this comment.
新增的读回兜底位于 toolParams.content/step.params.content 之后,apply_patch 步骤只要携带残留 content 就永远走不到这里,落盘补丁内容不受校验;按 action 分派,apply_patch 直接用 readWrittenFile(path)。
| * 与 needsRollback 分开表达:后者是「要不要中止后续步骤」的调度信号, | ||
| * 这个是「磁盘处于什么状态」的事实,两者在回滚成功时会分叉。 | ||
| */ | ||
| writeLeftOnDisk?: boolean; |
There was a problem hiding this comment.
字段实际只表示「回滚尝试且失败」,非围栏类写盘后失败时文件仍在盘上却为 false;改名为 rollbackFailed,或在「有 snapshotId、判失败、未回滚」时同样置 true。
| ); | ||
|
|
||
| const writeContent = this.resolveWriteContent(step, toolParams, context); | ||
| const rawContentValidation = await trace.withStage('validate_content', () => |
There was a problem hiding this comment.
完整 validateCode(含文件系统 import 解析)在围栏否决判定之前无条件执行,否决路径上结果被丢弃;先算 buildFenceVeto 并提前返回。
| @@ -1,10 +1,11 @@ | |||
| import { mkdirSync, mkdtempSync, rmSync } from 'node:fs'; | |||
| import { existsSync, mkdirSync, mkdtempSync, rmSync } from 'node:fs'; | |||
There was a problem hiding this comment.
existsSync与第 8 行的AgentEvent` 在本文件内均未使用,本文件的唯一改动是这两个残留 import,建议整体还原。
… the rollback field honestly Sixth repo-guard round on #402. **The `content` shadowing was only fixed on one side.** `resolveWriteContent` dispatches by action so a leftover `content` param cannot shadow the real patch — and the post-write path kept the exact chain that trap describes, `result.content ?? toolParams.content ?? step.params.content ?? readWrittenFile`. For an `apply_patch` step carrying a stale `content`, the guard validated a string that was never written while the landed patch went unchecked. Post-write now reads from disk for `apply_patch` and keeps the param chain only for `create_file`. **`writeLeftOnDisk` overclaimed and is now `rollbackFailed`.** It was only true when a rollback was attempted and failed, but rollback only fires on a fence in the landed content — so a post-write failure for any other reason left the file in the workspace with the field reading `false`. The old name and its comment described the wider fact; the implementation covered a subset, and the next consumer counting "how many bad files are on disk" would have read a systematically low number. Also from this round: - The veto predicate now runs *before* the full guard rather than after. The fence check is a string scan; `validateCode` does filesystem resolution for imports. There is no reason to pay for that on content that is about to be refused. - `report.mjs` carries a caveat that a bare `validation_failed` count is no longer an interception count: the event now has a `stage`, and `post_write` includes forward-reference imports that are *deliberately* allowed to land. Only `pre_write` is a real interception. - `executor.test.ts` is reverted — after the suites moved out, its entire diff was two now-unused imports. `Fixes #387` is downgraded in the PR description: with the veto narrowed to markdown fences, the issue's acceptance ("content that is syntactically invalid must not leave the file on disk") is only partly met, and merging should not auto-close it. The remainder is blocked on #413. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
代码评审报告: fix(executor): validate writes before disk, wire rollback, emit validation_failed
风险等级: 中
处理建议: 请求修改
决策摘要: ** 主体设计是对的(前置校验 + 收窄否决口径 + 回滚终态),级联面也确实很小;但新增的写盘否决绕过了 hallucinationGuard.enabledChecks(本仓已有测试与消融基准明确依赖 guard 可关),且对 .yaml 存在真实误伤路径,合并前需要给否决加语言与配置门控。
级联分析
- 变更符号: 原始模型未提供结构化级联字段。
- 受影响流程: 原始模型未提供结构化级联字段。
- 变更集外调用方: unknown
- 置信度: degraded
问题发现
-
[中] 写盘否决绕过 guard 的
enabledChecks,使 guard 不再可关闭- 证据:
executor.ts:258的buildFenceVeto是模块级纯函数,不查this.config.hallucinationGuard的任何开关;packages/hallucination-guard/src/guard.ts:62的enabledChecks为 private 且只有setCheckEnabled无读取入口。仓内executor.test.ts:413正是「honors a fully disabled hallucination guard on the executor validation path」这一契约的守卫测试——它只是因为用例内容不含围栏才继续通过。benchmarks/eval/report.mjs:67-69记录的头号缺陷就是「guard 无法通过公开配置关闭,导致消融臂失效」。 - 受影响调用方/流程: 任何以
enabledChecks: { syntaxValidity: false }甚至全关配置构造 guard 的调用方;消融基准的 ablation 臂从此再也拿不到干净的「guard 关」对照。 - 最小可行修复: 给 guard 加只读访问器(如
isCheckEnabled('syntaxValidity')),或在ExecutorConfig上加显式preWriteVeto?: boolean(默认 true),在executor.ts:257-259处门控;并补一条「全关 guard + 围栏内容照常写入」的用例,把这条契约真正锁住。
- 证据:
-
[中] 否决判据被应用到
yaml,块标量里的围栏是合法内容- 证据:
executor.ts:702只要detectLanguage非 null 就返回 writeContent,而phase-ordering.ts:144-159的返回值包含json与yaml;detectMarkdownFence的正则/^\s*```/对 YAML 块标量(如 issue template 的value: |内嵌 markdown 代码块)会命中。而executor.ts:35-38的文档注释自己写的适用范围是「.ts/.tsx/.js」——代码比注释宽。json不受影响(裸 ``` 无法出现在行首)。 - 受影响调用方/流程: 写
.yaml/.yml的 create_file 步骤会被否决且步骤判失败;在progress-enforcement路径下还会连带中止剩余计划。叠加问题 1,用户没有任何配置手段绕开。 - 最小可行修复: 在
resolveWriteContent或否决调用点把 veto 限定为language === 'typescript' || language === 'javascript',让 yaml/json 保持原有的写盘后判定语义。
- 证据:
-
[中] #388 的验收「interceptions are countable」只完成了一半
- 证据: 事件已带
stage(types.ts:809-814),但唯一的计数消费者benchmarks/eval/run-eval.mjs:97是events[e.type] = (events[e.type] ?? 0) + 1,只按 type 聚合;report.mjs:77-79新增的说明自己承认「必须按 stage 分开计数」,却没有任何代码这么做,而report.mjs:75的裸计数照旧输出。结果是:这一轮之后该数字第一次非零,但按报告自己的口径它不能当拦截数用。 - 受影响调用方/流程:
benchmarks/eval/report.mjs生成的效果报告;#388 想回答的「guard 到底拦下了什么」。 - 最小可行修复:
run-eval.mjs:97改为按e.stage ? \${e.type}:${e.stage}` : e.type聚合,并把report.mjs的那行统计拆成pre_write`(真拦截)与其余两类分别输出,替换当前的散文提醒。
- 证据: 事件已带
-
[中]
needsRollback新口径把「写步骤工具失败」也一并排除出中止条件- 证据:
executor.ts:350改为postValidation.results.some(r => !r.pass),而validateAfterExecution:789-793对工具失败返回results: []。PR 描述给出的理由是 read_file(planner.ts:376确认 read_file 默认file_exists: required,旧条件确实会中止全盘),但同一改动同样覆盖 create_file/apply_patch:写工具报 EACCES 之后,progress-enforcement.ts:45不再中止,计划继续跑。后续若恰好是针对该文件的 apply_patch,会在executor-skills.ts:242抛错兜住;但若后续步骤只是「引用该模块的另一个文件」,则整轮会以「缺模块但步骤全绿」的形式收尾。 - 受影响调用方/流程:
Executor.executeSteps的直接嵌入方(progress-enforcement)。主 agent 路径不读该字段,不受影响。 - 最小可行修复: 把「排除纯工具失败」限定在只读动作上,例如工具失败时对
WRITE_ACTIONS仍返回needsRollback: true,其余动作返回 false;并补一条 progress-enforcement 层的用例锁住这条分支。
- 证据:
-
[低] 回滚触发在读回失败时静默降级为「什么都没发生」
- 证据:
executor.ts:324-327以readWrittenFile(step.params.path)的结果为门;readWrittenFile:765-774在路径越界、文件不可读、或写工具的根目录与config.projectRoot不一致时一律返回undefined,于是既不尝试回滚,rollbackFailed也保持 false,validateAfterExecution:802-805还会复用写盘前结论——三处都表现为「无异常」。 - 受影响调用方/流程: headless 运行下「坏文件是否还在磁盘上」的可判定性,正是本 PR 第 4 节要解决的问题。
- 最小可行修复: 当
landed === undefined且toolResult带snapshotId且动作属WRITE_ACTIONS时,至少debugWarn一条并在stepResult.error注明「未能读回写入内容,回滚未尝试」。
- 证据:
-
[低] PR 描述与实现漂移
- 证据: 描述第 5 节与 GitNexus 小结均称新增
ExecutorOutput.writeLeftOnDisk,实际落地字段是rollbackFailed(types.ts:744);Impact Scope 列了executor.test.ts为改动测试,但变更文件清单里没有它(新增的是executor.validation.test.ts)。 - 受影响调用方/流程: 评审者与后续读 changelog 的人。
- 最小可行修复: 更新描述,或在合并说明里指出字段最终改名的提交(
644a10b)。
- 证据: 描述第 5 节与 GitNexus 小结均称新增
行级发现
- [packages/core/src/executor/executor.ts:258] 否决判据不查 guard 的
enabledChecks,syntaxValidity: false乃至全关配置也照样否决写盘;给 guard 加只读访问器或在ExecutorConfig上加显式开关来门控这一行。 - [packages/core/src/executor/executor.ts:702] 这里会把
yaml/json也交给围栏否决,而 YAML 块标量里的 ``` 是合法内容;把 veto 限定为typescript/`javascript`,其余语言维持写盘后判定。 - [packages/core/src/executor/executor.ts:59] 前置否决路径下文件并未落盘,错误文案却写「written into」,该串会随
stepResult.error进入 recovery 反馈;按是否已落盘区分措辞(如 blocked before write / written into)。 - [packages/core/src/executor/executor.ts:325] 读回失败时这一门会静默跳过回滚且
rollbackFailed仍为 false;补一条WRITE_ACTIONS+ 有 snapshotId 却读不回时的告警与 error 注记。 - [packages/core/src/executor/executor.ts:350] 该口径同时排除了写步骤的工具失败,
progress-enforcement不再中止;建议只对只读动作排除工具失败。 - [benchmarks/eval/report.mjs:77] 这里用散文说明「必须按 stage 分开计数」,但
run-eval.mjs:97仍只按e.type聚合;把计数键改为type:stage并拆分本节统计,否则 #388 的指标依旧不可用。
Karpathy 评审
- 假设: 模型输出需要归一化为固定 Markdown 契约。
- 简洁性: 已提取 summary、finding、evidence 与 fix;原始 prose 不再附在评论中,避免占用下游解析与代理上下文。
- 变更范围: 原始模型未提供结构化范围字段。
- 验证: 需要查看 CI、测试或人工 CR 证据补强合并信心。
缺失覆盖
- 输出未命中 Repo Guard Markdown 契约;建议补充真实模型质量评估覆盖。
| // (import 检查)。围栏一旦命中就直接 return,没必要为一份不会落盘的内容 | ||
| // 白跑一次完整 guard。 | ||
| const contentValidation = writeContent | ||
| ? buildFenceVeto(writeContent.content, writeContent.path) |
There was a problem hiding this comment.
否决判据不查 guard 的 enabledChecks,syntaxValidity: false 乃至全关配置也照样否决写盘;给 guard 加只读访问器或在 ExecutorConfig 上加显式开关来门控这一行。
| } | ||
|
|
||
| const language = detectLanguage(path); | ||
| return language ? { path, content, language } : null; |
There was a problem hiding this comment.
这里会把 yaml/json 也交给围栏否决,而 YAML 块标量里的 ``` 是合法内容;把 veto 限定为 typescript/`javascript`,其余语言维持写盘后判定。
| if (!fence) { | ||
| return { pass: true, results: [] }; | ||
| } | ||
| const message = `Markdown code fence written into ${path} at line ${fence.line}: ${fence.text}`; |
There was a problem hiding this comment.
前置否决路径下文件并未落盘,错误文案却写「written into」,该串会随 stepResult.error 进入 recovery 反馈;按是否已落盘区分措辞(如 blocked before write / written into)。
| // `create` 快照的回滚是 unlinkSync,误判一次就是删掉一个合法文件。 | ||
| // 只有确定性的围栏入码才触发撤销。 | ||
| const landed = this.readWrittenFile(step.params.path as string | undefined); | ||
| if (landed !== undefined && !buildFenceVeto(landed, String(step.params.path)).pass) { |
There was a problem hiding this comment.
读回失败时这一门会静默跳过回滚且 rollbackFailed 仍为 false;补一条 WRITE_ACTIONS + 有 snapshotId 却读不回时的告警与 error 注记。
| // 旧条件在写步骤上恒真),但排除纯工具失败:`validateAfterExecution` 对 | ||
| // 工具报错返回 results 为空的失败结果,把它算进来会让一次 read_file 失败 | ||
| // 就中止整个剩余计划。 | ||
| needsRollback: postValidation.results.some((result) => !result.pass), |
There was a problem hiding this comment.
该口径同时排除了写步骤的工具失败,progress-enforcement 不再中止;建议只对只读动作排除工具失败。
| 3. **\`validation_failed\` 事件从未触发** | ||
| 两臂合计 ${sumEvent(arms.full, 'validation_failed') + sumEvent(arms.ablation, 'validation_failed')} 次。该事件挂在执行器不走的那条校验路径上,导致「校验是否起作用」在遥测层面不可观测。 | ||
|
|
||
| > **口径提醒(本轮之后)**:该事件现已带 \`stage\` 判别字段(\`pre_execution\` / \`pre_write\` / \`post_write\`)。 |
There was a problem hiding this comment.
这里用散文说明「必须按 stage 分开计数」,但 run-eval.mjs:97 仍只按 e.type 聚合;把计数键改为 type:stage 并拆分本节统计,否则 #388 的指标依旧不可用。
…yaml Seventh repo-guard round on #402. The first finding is the same defect this repo has now hit four times, committed by me while fixing an instance of it. **The veto ignored `enabledChecks`.** The executor implements its own pre-write gate, and it consulted no configuration — so `syntaxValidity: false` no longer turned it off. That is exactly the mechanism by which #386 voided the guard arm of the July ablation, and #392 had just fixed it. `HallucinationGuard` gains a public `isCheckEnabled()` (any caller reimplementing a check's semantics outside the guard has to be able to ask), and both the veto and the rollback trigger are gated on it. A regression test pins that the guard stays ablatable. **The veto applied to `.yaml`.** A markdown fence inside a YAML block scalar is legitimate content; `detectLanguage` includes yaml, so those writes would have been refused. The predicate is now restricted to TypeScript/JavaScript, where a line starting with ``` is never valid. Regression test included. **`validation_failed`'s `stage` had no counter.** The event carried it but `run-eval.mjs` aggregated by `type` alone, and `report.mjs` printed a bare count under a prose note saying that count must not be used. The harness now also counts `type:stage`, and the report prints the breakdown — `pre_write` is the only real interception; `post_write` includes the forward-reference imports this PR deliberately lets land. **A write tool's failure aborts again.** Excluding plain tool failures from `needsRollback` was meant to stop one failed `read_file` from killing a plan, but it covered writes too: after an `EACCES` on a write the plan continued, and if the following steps merely *referenced* the missing module they would all pass — the run ending green with the module absent. Read-only actions keep the exclusion; writes do not. **The unreadable-after-write branch is no longer silent.** When the file cannot be read back, no rollback is attempted and nothing is recorded — three ways of looking like "nothing happened". It now warns and annotates `stepResult.error`. Deliberately *not* setting `rollbackFailed`: that drives abort semantics and is a stronger claim than "could not determine". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
代码评审报告: fix(executor): validate writes before disk, wire rollback, emit validation_failed
风险等级: 高
处理建议: 请求修改
决策摘要: #388 的接线(stage 判别、拦截计数、rollback 终态)实现扎实且可合并,但 apply_patch 首次被纳入内容校验,直接把 #413 里已实测会误判的 checkSyntaxValidity(severity block)接到「步骤失败 / 计划中止」路径上,且未在 Known behaviour changes 里披露;另有 228 个与本变更无关、仓库内零引用的 fixture-deep 夹具文件和一个解析不到目标的 node_modules 链接。
级联分析
- 变更符号:
Executor.executeStep(新增 pre-write 阶段与回滚判定)、Executor.validateAfterExecution、Executor.rollback(返回形状归一)、resolveWriteContent/buildFenceVeto/readWrittenFile(新增)、ExecutorOutput.rollbackFailed、AgentEvent(validation_failed增stage/path/stepId,新增rollback_failed)、HallucinationGuard.isCheckEnabled(新公开 API)、ExecutorConfig.emitEvent。 - 受影响流程: create_file / apply_patch 写盘路径;写盘后校验与回滚触发;progress-enforcement 的中止判定;agent 事件流与三个前端消费者;消融评测两臂计数。
- 变更集外调用方:
apps/cli/src/ui/bridge.ts:100与apps/vscode/src/state.ts:413的switch(event.type)均有default分支,事件 union 扩展非破坏性(text search 已确认)。Executor.rollback仅有executor.ts:785一个调用方,返回形状归一无外部影响。needsRollback仅progress-enforcement.ts:45消费;agent 主路径走executeStepsWithErrorFeedback(agent.ts:813),不读该字段——但主路径仍会看到步骤success:false并进入 recovery。enabledChecks经agent.ts:96由config.hallucinationGuard.checks传入,run-eval.mjs:33的 ablation 臂能真正关掉写盘否决,可消融性成立(graph 不可用,以上为 Read/Grep 直接查证)。 - 置信度: medium
问题发现
-
[高]
apply_patch首次被已知会误判的语法检查判定成败,且未披露- 证据: 改动前
validateAfterExecution取result.content ?? toolParams.content ?? step.params.content;codegen 的 apply_patch 分支只产出patches(skills/executor-skills.ts:286-289,不设content),因此补丁步骤此前从不进入validateCode,恒为pass:true。本 PR 后两条路径都会校验:整文件 replace 走executor.ts:294的 pre-writevalidateCode并复用为写盘后结论,局部补丁走executor.ts:862读回磁盘再校验。validateCode(guard.ts:201-214)包含checkSyntaxValidity,其checkCommonPatterns(syntax-validity.ts:186-217)逐行数引号奇偶,失败时severity:'block'(同文件 45-53 行)——即 issue #413 已实测的"it's fine"、多行模板字符串、JSX 撇号全部判 block。 - 受影响调用方/流程: 所有 modify 步骤。文件不会被删(回滚仍只由围栏触发),但步骤被判
success:false;executor.ts:386-388使needsRollback:true,progress-enforcement.ts:45会跳过剩余全部步骤;agent 主路径则消耗 recovery 次数。同时validation_failed:post_write计数被误报污染,削弱 #388 想要的可观测性。PR 自己论证「这条检查不可靠到不能否决写盘」,却让它决定步骤成败与调度,两处口径不一致;#413「Interim state」也要求调用方不要依赖它。 - 最小可行修复: 让 apply_patch 的写盘后结论与否决口径一致——把补丁路径的内容判定同样收敛到
buildFenceVeto(或从补丁路径的results中剔除syntax_validity条目)直到 #413 落地;并补一条断言:内容含"it's fine"的整文件 replace 补丁stepResult.success === true。若确认是有意扩大校验面,至少写进 Known behaviour changes 并说明 recovery/中止的代价。
- 证据: 改动前
-
[中] 228 个与本变更无关的
fixture-deep文件,且node_modules条目在检出中解析不到- 证据: 全仓库 grep
fixture-deep零命中,run-eval.mjs:12仍指向join(HERE, 'fixture');benchmarks/eval/fixture-deep/*下没有任何顶层文件(无package.json/tsconfig.json/sdd.yaml,与benchmarks/eval/fixture/的结构不同),因此它现在也无法作为评测工作区使用;benchmarks/eval/fixture-deep/node_modules在本检出中读取报「不存在」(悬空链接)。 - 受影响调用方/流程: 无运行时影响,但占本 PR 241 个文件中的 228 个,diff 无法聚焦;仓库多出一份无人引用、无
.gitignore、依赖入口悬空的夹具树,后续容易被误当作可用夹具。 - 最小可行修复: 从本 PR 移出
benchmarks/eval/fixture-deep/**,随真正使用它的评测改动一起提交,并在那时补上package.json/.gitignore,不要把node_modules链接提交进仓库。
- 证据: 全仓库 grep
-
[中] PR 描述与实现不符:
writeLeftOnDisk并不存在- 证据: 描述第 5 节、Impact Scope 和 GitNexus 影响摘要都称
ExecutorOutput.writeLeftOnDisk;types.ts:736-744实际新增的是rollbackFailed,其注释明确写着它不等于「写入仍在磁盘」(无快照时不会尝试回滚,文件照样留着而该字段为 false)。 - 受影响调用方/流程: 按描述核对契约的评审者与后续接入方会找不到该字段,或误以为存在一个「文件是否残留」的直接信号。
- 最小可行修复: 更新 PR 描述为
rollbackFailed,并同步其语义说明。
- 证据: 描述第 5 节、Impact Scope 和 GitNexus 影响摘要都称
行级发现
- [packages/core/src/executor/executor.ts:862] 这条读回分支让每个补丁步骤首次进入
validateCode,把syntax_validity的 block 级误判(#413)变成步骤失败与计划中止信号;建议此处改用与写盘否决相同的高精度判据,或剔除syntax_validity结果,直到 #413 修复。 - [packages/core/src/executor/executor.ts:340] 写盘前用
toolParams.path ?? step.params.path解析路径,这里的回滚判定只读step.params.path;一旦技能改写 path(emitValidationFailed的注释正是这么假设的),读回会落空、回滚静默不触发。建议把已解析路径从resolveWriteContent传下来复用。 - [packages/core/src/executor/executor.validation.test.ts:677] 该用例只断言「未被写盘前置校验拦截」,未断言
stepResult.success;按syntax-validity.ts:194-216的逐行引号计数,"it's fine"/ 多行模板字符串 / JSX 撇号三例在写盘后仍会被判 block,步骤实际为失败。补一条expect(result.stepResult.success).toBe(true)才能锁住这个 describe 声称的契约。
Karpathy 评审
- 假设: 「围栏是高精度判据、其余校验维持旧的写盘后语义」这一前提对 create_file 成立,对 apply_patch 不成立——补丁路径此前根本没有写盘后内容语义,本 PR 是新建而非保留。这个隐含假设是问题 1 的根源。
- 简洁性: 否决判据小而独立,
isCheckEnabled让执行器侧门禁受同一份配置管辖(避免重演 #386),是正确的所有权归位。rollback()返回归一消除了调用方各自 cast,方向正确。 - 结构质量:
resolveWriteContent按 action 分派、拒绝跨动作content兜底,是合理的边界选择;但同一份「按 action 取内容」的逻辑在validateAfterExecution:856-862又写了一遍,两处口径靠注释而非代码保持同步,后续容易漂移,可考虑合并为单一解析函数。executor.ts已达约 1100 行,本 PR 新增 ~340 行集中在executeStep与校验辅助上,写盘校验这一簇(围栏判据、补丁解析、读回、回滚编排)具备自然的模块边界,建议后续拆分。 - 变更范围: 核心 diff 聚焦;
benchmarks/eval/fixture-deep/**的 228 个文件属于范围蔓延(问题 2)。 - 验证: 前置拦截、别名/前向引用放行、补丁内容优先于残留
content、回滚被拒的终态事件——这几条都用真实HallucinationGuard与真实磁盘断言,质量高。缺的是「合法代码在写盘后不应被判失败」这条正向断言(见行级发现 3),而它恰好是问题 1 会暴露的地方。
缺失覆盖
- 整文件 replace 补丁,内容含
"it's fine"或多行模板字符串:断言stepResult.success === true且不产生validation_failed(当前无覆盖,是问题 1 的回归门)。 - 局部行补丁写入合法代码后读回:断言步骤成功、无
validation_failed、无 rollback。 executeStepsWithProgressEnforcement层面的用例:一个内容合法的 modify 步骤失败时不应跳过后续步骤(验证新needsRollback口径不会因误判连锁中止整轮计划)。
| (toolParams?.content as string | undefined) ?? | ||
| (step.params.content as string | undefined) ?? | ||
| this.readWrittenFile(path)) | ||
| : this.readWrittenFile(path); |
| // 回滚的触发口径必须和写盘否决口径一致,且同样不能建立在会误判的启发式上: | ||
| // `create` 快照的回滚是 unlinkSync,误判一次就是删掉一个合法文件。 | ||
| // 只有确定性的围栏入码才触发撤销。 | ||
| const landedPath = step.params.path as string | undefined; |
There was a problem hiding this comment.
写盘前用 toolParams.path ?? step.params.path 解析路径,这里的回滚判定只读 step.params.path;一旦技能改写 path(emitValidationFailed 的注释正是这么假设的),读回会落空、回滚静默不触发。建议把已解析路径从 resolveWriteContent 传下来复用。
| ); | ||
|
|
||
| expect(callTool).toHaveBeenCalledTimes(1); | ||
| expect(result.stepResult.error).not.toContain('Pre-write validation failed'); |
There was a problem hiding this comment.
该用例只断言「未被写盘前置校验拦截」,未断言 stepResult.success;按 syntax-validity.ts:194-216 的逐行引号计数,"it's fine" / 多行模板字符串 / JSX 撇号三例在写盘后仍会被判 block,步骤实际为失败。补一条 expect(result.stepResult.success).toBe(true) 才能锁住这个 describe 声称的契约。
They belong to PR #412. Swept in by `git add` while sitting untracked in a shared worktree; this is the third time, so the generated tree is now kept out of the worktree entirely rather than re-untracked after the fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tcomes Eighth repo-guard round on #402, and the finding is a consequence of my own previous fix. Before this PR, `apply_patch` never reached `validateCode` — codegen emits only `patches`, and the content chain had no other source, so patch steps were always `pass: true`. Adding the disk read-back gave that path a content source for the first time, which means the **known-unreliable** syntax heuristic started deciding whether modify steps succeed. Per #413 it blocks on `"it's fine"`, on multi-line template literals, and on JSX contractions. The file would not have been deleted (rollback fires only on a fence), but the step would fail, `needsRollback` would be true, `progress-enforcement` would skip every remaining step, the agent path would burn recovery attempts, and the `validation_failed:post_write` count — the thing #388 exists to make usable — would be polluted with false positives. This PR argues that check is too unreliable to veto a write, then let it decide step outcomes. One predicate cannot have two credibility ratings. `syntax_validity` verdicts are now dropped from post-write results (`dropUnreliableSyntaxVerdicts`), on both the read-back path and the reuse path — credibility does not change because a result is reused. The function is marked for deletion once #413 lands a real parser. That left a gap: a fence written into a `.ts` no longer failed anything. So the fence check on landed content is now an **independent** failure source rather than a filter applied inside `if (!postValidation.pass)`. Step failure and rollback now key off the same deterministic predicate, which also removes the possible mismatch where one fired and the other did not. Test added, as the reviewer asked: a whole-file replace patch containing `"it's fine"` must succeed, must not be rolled back, and must stay on disk. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
代码评审报告: fix(executor): validate writes before disk, wire rollback, emit validation_failed
风险等级: 中
处理建议: 请求修改
决策摘要: 核心方向正确(写盘前否决判据高精度、事件接线到位、回滚终态补齐),但写盘后的围栏判定作用在「整份落盘文件」而非本步骤写入的内容上,会误判并回滚对既有含围栏文件的合法补丁;同时 dropUnreliableSyntaxVerdicts 删除了全部写盘后 syntax_validity 判定,这与 PR 描述「其余判定保持原有写盘后语义」相矛盾且未列入行为变更;executor.ts 由约 770 行跨到约 1140 行,新增逻辑有天然模块边界。
级联分析
- 变更符号:
Executor.executeStep/validateAfterExecution/rollback;新增resolveWriteContent、emitValidationFailed、rollbackFailedWrite、readWrittenFile、buildFenceVeto、dropUnreliableSyntaxVerdicts;AgentEvent(+rollback_failed、validation_failed.stage);ExecutorOutput.rollbackFailed;ExecutorConfig.emitEvent;HallucinationGuard.isCheckEnabled。 - 受影响流程: 写盘前后校验、首次真正触发的自动回滚、
progress-enforcement中止判定、desktop 日志、eval 事件计数。 - 变更集外调用方:
needsRollback唯一消费点确为packages/core/src/executor/progress-enforcement.ts:45(text search,全仓仅此一处读取)。AgentEvent新成员安全:apps/cli/src/ui/bridge.ts:100、apps/vscode/src/state.ts:413、apps/desktop/src/state/executionReducer.ts:342均有default分支。isCheckEnabled在executeStep中对每个 step 无条件调用;ExecutorConfig.hallucinationGuard是类类型,生产侧(agent.ts:107、runtime-node/src/run.ts:173)传真实实例,仅executor.test.ts:341的鸭子类型 mock 未补该方法——该用例走 pre-validation 早返回,未触达调用点,属潜在脆弱点而非当前破坏。- 前置判据的高精度已核实:
/^\s*```/对全仓.ts/.tsx/.js无命中,'```'形式的字符串常量(如packages/sdd/src/workflow/phases/tasks.ts:42)不匹配;resolveFullFileReplaceContent的形状与packages/core/src/skills/executor-skills.ts:278-289实际产出一致;planner.ts:325+getDefaultValidation确实给两个写动作发required: true;安全层拒绝形状确为{success:false,error}(tool-call-handler.ts:62-69),rollback()归一化必要且正确。
- 置信度: medium — 工作区检出落后于 PR 头(缺
dropUnreliableSyntaxVerdicts与effectivePostValidation),这两处只能依据 diff 评审;我未运行测试,pnpm quality:precommit结果按作者陈述采信。
问题发现
-
[高] 写盘后的围栏判定作用于整份落盘文件,且不区分工具是否成功
- 证据:
executor.ts新增 366-377 行——landedContent在callTool之后无条件读回(不看toolResult.success),effectivePostValidation = landedFenceVeto.pass ? postValidation : landedFenceVeto整体覆盖原判定;395 行据此触发rollbackFailedWrite。 - 受影响调用方/流程: 局部行
apply_patch(resolveFullFileReplaceContent只覆盖startLine === 1的整文件替换,其余全部落到这条路径)。三个后果:① 目标文件在本步骤之前就含围栏时(正是 #387 证据里那批已落盘的.tsx),任何合法局部补丁都会被判失败并回滚到同样含围栏的快照——修复这些文件的路径被堵死;② 工具报错(如 EACCES)时若磁盘上的旧文件含围栏,stepResult.error会被替换成 "Markdown code fence written into …",真实工具错误被掩盖,且仍会尝试回滚(交互模式下多一次审批弹窗);③ 事件里的path/line归因到一个并未写过这些行的步骤。 - 最小可行修复: 仅在
toolResult成功时启用landedFenceVeto;对局部补丁,用collectedContext.files.get(path)的原始内容先做一次detectMarkdownFence,原本就有围栏则不归因本步骤(保留postValidation与工具错误)。
- 证据:
-
[中]
dropUnreliableSyntaxVerdicts删除全部写盘后syntax_validity判定,属未申报的范围扩张- 证据:
executor.ts新增 66-76 行,validateAfterExecution的两个出口(复用分支与读回分支)都套用它。guard.validateCode里syntax_validity是唯一的语法信号(guard.ts:201-204)。 - 受影响调用方/流程: 真正的语法破损(#413 表中判定正确的
export const a = {)从此不再让步骤失败,也不再产生validation_failed:post_write。PR 描述第 1 节称「Every other verdict keeps its old post-write meaning」,与此实现直接冲突;「Known behaviour changes」也未列。此外本 PR 同时在给消融基准加分阶段计数,而syntaxValidity开/关两臂的差异被这次删除进一步抹平,会影响正在建立的拦截率指标解释。 - 最小可行修复: 保留结果条目但把
severity降级为warn(pass计算不变,遥测与错误反馈仍可见),或维持删除但在 PR「Known behaviour changes」与report.mjs的阶段说明里写明「post_write 不再包含语法判定」。
- 证据:
-
[中] executor.ts 跨过 1000 行,新增逻辑有天然模块边界
- 证据: 基线约 770 行(检出为 1091 行、含本 PR 大部分改动;diff stat +385/−14),PR 头约 1140 行。新增部分是一个内聚单元:
WRITE_ACTIONS、detectMarkdownFence、buildFenceVeto、FENCE_VETO_LANGUAGES、dropUnreliableSyntaxVerdicts、resolveFullFileReplaceContent、resolveWriteContent、readWrittenFile、rollbackFailedWrite。同时executeStep主体膨胀到约 180 行,控制流被 20 余行论证性注释切断(286-292、311-313、360-364、419-425)。 - 受影响调用方/流程: 后续改动写盘校验策略(#413 落地要换 parser、删掉
dropUnreliableSyntaxVerdicts)都要在这个文件里做,回归面无法隔离测试。 - 最小可行修复: 抽出
packages/core/src/executor/write-validation.ts(纯函数:围栏判据 + 判定过滤 + 补丁内容解析),executor 只保留编排;readWrittenFile/rollbackFailedWrite可留在类内。
- 证据: 基线约 770 行(检出为 1091 行、含本 PR 大部分改动;diff stat +385/−14),PR 头约 1140 行。新增部分是一个内聚单元:
-
[中] 新测试文件 848 行,脚手架成片复制,且含一对功能等价用例
- 证据:
executor.validation.test.ts:154-212与:543-601是同一场景(同样的apply_patch startLine:2,endLine:2、同样落盘```ts、同样断言回滚并恢复原文),只是断言侧重不同;另有 6 处约 40 行的mkdtempSync + new Executor + registerMCPClient + registerToolMapping逐字复制。 - 受影响调用方/流程: 判据一旦调整(第 1、2 条修复都会动),需要同步修改 6 处近似副本;重复用例还会让「哪条锁的是哪个契约」难以判断。
- 最小可行修复: 提取
makeGuardedExecutor({ projectRoot, events, allowRollback })辅助函数,合并那对重复用例,把两条断言并入一个用例。
- 证据:
-
[低] PR 描述与实现的字段名不一致
- 证据: 描述第 5 节与 GitNexus 摘要多处写
ExecutorOutput.writeLeftOnDisk,实现是rollbackFailed(types.ts:744),其文档还明确说明「刻意不叫 writeLeftOnDisk」。 - 受影响调用方/流程: 依据描述评估爆炸半径的维护者会去找一个不存在的字段,也会误以为「写入是否仍在磁盘」已被单一字段表达(实际需要
rollbackFailed与needsRollback并集)。 - 最小可行修复: 更新 PR 描述第 5 节与 GitNexus 摘要中的字段名。
- 证据: 描述第 5 节与 GitNexus 摘要多处写
行级发现
- [packages/core/src/executor/executor.ts:377]
effectivePostValidation用整份落盘文件的围栏判定覆盖postValidation,且上游landedContent不校验toolResult.success;应仅在工具成功时启用,并对局部补丁与collectedContext里的原始内容比对,避免把既有围栏归因给本步骤、掩盖真实工具错误。 - [packages/core/src/executor/executor.ts:66]
dropUnreliableSyntaxVerdicts整类删除syntax_validity,真实语法破损从此不再让步骤失败;建议降级为warn保留结果条目,或在行为变更清单中明写。 - [packages/core/src/executor/executor.ts:384] 该分支的错误文案「could not read back …; rollback was not attempted」会附加到任何读不回文件的写盘后失败上,但按新口径这类失败本就不触发回滚;应改为「无法读回落盘内容,未能核验」之类的表述,避免暗示漏掉了一次回滚。
- [packages/core/src/executor/executor.ts:53] 这行 doc comment 描述的是
buildFenceVeto,却挂在dropUnreliableSyntaxVerdicts的文档块之前,形成悬空注释;移到buildFenceVeto定义处或删除。 - [packages/core/src/executor/executor.ts:365]
landedPath取step.params.path,而写盘前resolveWriteContent取toolParams.path ?? step.params.path,emitValidationFailed的注释还写着「技能可能改写过 step.params.path」;当前技能只做...params透传所以两者相等,但同一路径应统一取值口径,否则未来任何改写 path 的技能都会让读回静默失效。 - [packages/core/src/executor/executor.validation.test.ts:543] 该用例与 154 行用例是同一场景同一断言目标,合并为一条并抽出共用的 executor 脚手架。
Karpathy 评审
- 假设: 「codegen 的 apply_patch 必为
startLine:1整文件 replace」已在executor-skills.ts:278-289核实;但resolveFullFileReplaceContent依赖collectedContext.files的行数快照,而该 Map 只在create_file成功后刷新(step-callbacks.ts:98-106),同一计划内二次修改同一文件时前置校验会静默降级到写盘后——PR 在复用分支里处理了不一致,但没有测试锁住这条降级路径。另一未言明的假设是rollback工具已注册;未注册时callTool抛错被rollback()吞成rollbackFailed: true,从而中止整个剩余计划。 - 简洁性: 围栏判据本身是正确的「窄而确定」取舍,值得保留。但写盘后一段同时存在三条内容来源(复用的 pre-write 结果、读回内容、
landedFenceVeto)和三次同文件readFileSync(366、复用分支、create_file兜底),概念数偏多;抽成一个「解析本次落盘内容 + 判定」的单一函数可以让分支自然消失。 - 结构质量: 见问题 3、4——executor.ts 跨过 1000 行、测试文件 848 行且成片复制,是本轮最明确的退化项;判据逻辑本身是可独立测试的纯函数,拆分成本很低。
- 变更范围: 与既定目标基本吻合;
dropUnreliableSyntaxVerdicts是唯一超出 #388/#387 的行为变更(问题 2)。report.mjs/run-eval.mjs的阶段计数属于 #388 的合理配套。 - 验证: 新增用例质量高——用真实
HallucinationGuard、真实落盘、并专门覆盖了「不该否决」的四类合法代码与 yaml 块标量,ablatable用例锁住了enabledChecks管辖关系,这些是本 PR 最扎实的部分。缺口见下。
缺失覆盖
- 局部补丁作用在本来就含围栏的既有文件上:应断言合法补丁不被判失败、不触发回滚(对应问题 1 的核心风险,也是修复 #387 遗留脏文件的实际路径)。
- 写工具失败(
success:false)且目标文件已含围栏:应断言stepResult.error保留工具错误、不发起回滚。 dropUnreliableSyntaxVerdicts的真阳性方向:export const a = {这类真实破损写盘后现在会通过,没有任何用例记录这一既定行为。- 同一计划内二次修改同一文件(
collectedContext行数已过期):应断言复用分支检测到落盘内容不一致后按读回内容重判。 rollback工具未注册时的路径:当前会rollbackFailed: true并中止剩余计划,无用例说明这是预期。
一点说明:本次评审对 dropUnreliableSyntaxVerdicts 与 effectivePostValidation 只能依据 PR diff——工作区检出落后于 PR 头,不含这两处;其余结论均已在仓库中核对到具体行。
| detectLanguage(String(landedPath)) ?? '', | ||
| ) | ||
| : { pass: true, results: [] }; | ||
| const effectivePostValidation = landedFenceVeto.pass ? postValidation : landedFenceVeto; |
There was a problem hiding this comment.
effectivePostValidation 用整份落盘文件的围栏判定覆盖 postValidation,且上游 landedContent 不校验 toolResult.success;应仅在工具成功时启用,并对局部补丁与 collectedContext 里的原始内容比对,避免把既有围栏归因给本步骤、掩盖真实工具错误。
| * 写盘前的围栏否决不受影响——那条判据是确定性的。 | ||
| * #413 落地(换真 parser)后这个函数应当删除。 | ||
| */ | ||
| function dropUnreliableSyntaxVerdicts(validation: ValidationResult): ValidationResult { |
There was a problem hiding this comment.
dropUnreliableSyntaxVerdicts 整类删除 syntax_validity,真实语法破损从此不再让步骤失败;建议降级为 warn 保留结果条目,或在行为变更清单中明写。
| // 回滚只由确定性的围栏判据触发:`create` 快照的回滚是 unlinkSync, | ||
| // 误判一次就是删掉一个合法文件。 | ||
| const landed = landedContent; | ||
| if (landed === undefined && (toolResult as { snapshotId?: string })?.snapshotId) { |
There was a problem hiding this comment.
该分支的错误文案「could not read back …; rollback was not attempted」会附加到任何读不回文件的写盘后失败上,但按新口径这类失败本就不触发回滚;应改为「无法读回落盘内容,未能核验」之类的表述,避免暗示漏掉了一次回滚。
| return undefined; | ||
| } | ||
|
|
||
| /** 把围栏检出结果表达成 ValidationResult,好让事件与错误文案与其余校验同形。 */ |
There was a problem hiding this comment.
这行 doc comment 描述的是 buildFenceVeto,却挂在 dropUnreliableSyntaxVerdicts 的文档块之前,形成悬空注释;移到 buildFenceVeto 定义处或删除。
| // 所以一份写进 .ts 的围栏不会再让 postValidation 失败——但它确实是坏内容, | ||
| // 必须自己让步骤失败并触发撤销。这也让「判失败」与「触发回滚」用的是同一条 | ||
| // 确定性判据,不会出现一个判失败、另一个不撤销的错位。 | ||
| const landedPath = step.params.path as string | undefined; |
There was a problem hiding this comment.
landedPath 取 step.params.path,而写盘前 resolveWriteContent 取 toolParams.path ?? step.params.path,emitValidationFailed 的注释还写着「技能可能改写过 step.params.path」;当前技能只做 ...params 透传所以两者相等,但同一路径应统一取值口径,否则未来任何改写 path 的技能都会让读回静默失效。
| // 真实 create_file / apply_patch 都不返回 `content`,局部行补丁也没有 | ||
| // `content` 参数。若只认这几个来源,局部补丁写盘后根本不做内容校验, | ||
| // #387 的「非法补丁内容不得留在磁盘」对补丁路径就不成立。 | ||
| it('validates a partial patch by reading the written file, and rolls it back', async () => { |
There was a problem hiding this comment.
该用例与 154 行用例是同一场景同一断言目标,合并为一条并抽出共用的 executor 脚手架。
There was a problem hiding this comment.
代码评审报告: fix(executor): validate writes before disk, wire rollback, emit validation_failed
风险等级: 高
处理建议: 请求修改
决策摘要: 前置围栏否决 + validation_failed 接线这两块是对的且有真实证据支撑,但同一改动顺带把整套 guard(含 PR 自己论证为高误报的 import_validity)第一次接到了 apply_patch 的成败判定上,而 needsRollback 会因此中止整个剩余计划——这条新增的阻塞面没有任何测试覆盖,合并前需要先定性。
级联分析
- 变更符号:
Executor.executeStep/validateAfterExecution/resolveWriteContent(新)/rollbackFailedWrite(新)/readWrittenFile(新)/Executor.rollback;ExecutorOutput.needsRollback+rollbackFailed;AgentEvent.validation_failed(新增必填stage)与rollback_failed(新成员);HallucinationGuard.isCheckEnabled(新)。 - 受影响流程: 所有
create_file/apply_patch步骤的写盘判定;executeStepsWithProgressEnforcement的中止调度;desktop 控制台日志;benchmarks/eval的事件计数。 - 变更集外调用方:
packages/core/src/executor/progress-enforcement.ts:45是needsRollback的唯一消费者(已随本 PR 更新);apps/cli/src/ui/bridge.ts、apps/cli/src/run-logger.ts:111(原样透传,无 switch 遗漏)、apps/vscode/src/state.ts:303均无validation_failed分支,事件联合新增成员不会破坏它们;packages/runtime-node/src/run.ts:173,443传入的是HallucinationGuard实例,新增方法对它们是加法。ExecutorConfig.hallucinationGuard是具体类而非接口,所有 mock 均在包内,已同步。(证据来源:仓库内 Read/Grep,非代码图谱) - 置信度: medium
问题发现
-
[高]
apply_patch首次继承import_validity的 block 判定,会让合法的修改步骤失败并中止整个计划- 证据: 改动前
validateAfterExecution对apply_patch取result.content ?? toolParams.content ?? step.params.content;planner 对 modify 步骤只发params: { path, patches: [] }(packages/core/src/planner.ts:465),codegen 技能返回{...params, patches:[patch]}(skills/executor-skills.ts:286-289),三个来源全为 undefined ⇒ 补丁路径实际上从不做内容校验。本 PR 后:整文件 replace 走resolveWriteContent→rawContentValidation→executor.ts:884复用为写盘后结论;局部补丁走executor.ts:904读回磁盘再跑validateCode。两条路都会带上import_validity。而checkImportValidity(packages/hallucination-guard/src/checks/import-validity.ts:36-53,84-90)对@/components/x判「包未安装 → block」,对后续步骤才创建的相对模块判「无法解析 → block」——正是 PR 描述里点名不敢给写盘否决权的那两类误报。 - 受影响调用方/流程: 每个 modify 步骤。失败后
executor.ts:426-428的results.some(r => !r.pass)为真 ⇒progress-enforcement.ts:45把剩余步骤全部置skipped;agent 主路径则改为消耗 recovery 次数去「修」一个本来正确的 import。 - 最小可行修复: 对写盘后判定采用与写盘否决相同的收窄口径——只让确定性判据(围栏)决定
apply_patch的成败,import_validity的 block 在写动作上降为 warn(保留在results里供遥测),或至少显式声明「modify 步骤现在也受 import 检查阻塞」并补一条整文件 replace + 别名 import 的用例把预期钉死。现有的does not veto a write over a path-alias import只断言了没有Pre-write validation failed,该步骤实际仍以失败收尾(走executor.ts:384的 unreadable 分支),测试名与真实行为不符。
- 证据: 改动前
-
[中]
dropUnreliableSyntaxVerdicts整条剔除syntax_validity,真语法错误从此不再让任何步骤失败,且从事件载荷里消失- 证据:
executor.ts:67用filter(r => r.type !== 'syntax_validity')丢弃结果本身,而非只把它排除出blockedBy。export const a = {这类 #413 表格里判定正确的样本(block、真错),改动前会让 create_file 步骤失败,改动后pass为 true、步骤成功、文件留在磁盘上,围栏判据也覆盖不到它。 - 受影响调用方/流程: 所有写步骤的写盘后判定;
validation_failed:post_write的载荷(validation.results已被裁剪);消融基准中 guard 臂的实际检出能力。#413 的「Interim state」只要求「不要给它写盘否决权」,本 PR 额外取消了它的写盘后判定,这一点没有出现在「Known behaviour changes」里。 - 最小可行修复: 保留
results条目、只把syntax_validity排除出pass/blockedBy计算(等价于就地降级为warn),并在 PR 的 Known behaviour changes 里补一句「真语法错误不再让步骤失败」。
- 证据:
-
[中]
executor.ts越过 1000 行(761 → 1132),executeStep单函数约 235 行,同目录已有可复用的拆分惯例- 证据:
executor.ts现为 1132 行;executeStep(executor.ts:212-447)里串起参数校验、写前否决、guard 调用、工具调用、读回、二次围栏判定、回滚编排、错误串拼装(executor.ts:399-414的三层嵌套三元)八件事。同目录已有progress-enforcement.ts、phase-ordering.ts、tool-call-handler.ts、step-trace-recorder.ts这类按关注点拆出的兄弟模块。 - 受影响调用方/流程: 后续任何改动写盘判定的人都要在这个函数里重新推演一遍「哪份内容、读了几次、谁触发回滚」。
- 最小可行修复: 把
detectMarkdownFence/buildFenceVeto/dropUnreliableSyntaxVerdicts/resolveFullFileReplaceContent/readWrittenFile/rollbackFailedWrite移到同目录新建的write-validation.ts,executeStep只保留编排。
- 证据:
-
[中] 同一次写盘最多读回同一文件三次,且写前/写后的 path 解析口径不一致
- 证据:
executor.ts:880(复用一致性比对)、executor.ts:366(landed 围栏判定)、executor.ts:903/904(内容取值)各读一次。写前用toolParams.path ?? step.params.path(executor.ts:762),写后固定用step.params.path(executor.ts:365,889);emitValidationFailed的注释(executor.ts:792)恰恰说明作者预期技能可能改写路径,两处口径就不该分叉。(现有技能都不改写 path,所以今天不会出错,属于隐性假设而非当前缺陷。) - 受影响调用方/流程: 写盘后判定路径;将来任何会重写 path 的技能会让读回落到错误文件、静默退化为「读不回」分支。
- 最小可行修复: 在
executeStep里读一次 landed 内容与一份resolvedPath,作为参数传给validateAfterExecution,删掉后两处独立读取。
- 证据:
-
[低] PR 描述与 GitNexus 影响摘要引用了不存在的字段
writeLeftOnDisk- 证据: 描述第 5 节、Impact Scope、GitNexus 摘要均写
ExecutorOutput.writeLeftOnDisk;代码里落地的是rollbackFailed(packages/core/src/types.ts:744),语义也不同(后者明确「不等于文件还在磁盘上」)。 - 受影响调用方/流程: 依赖影响摘要做合入判断的评审者。
- 最小可行修复: 更新描述;顺带考虑把
needsRollback改名为它现在真正的含义(是否中止剩余步骤)——回滚已由执行器内部完成,该字段与回滚无关。
- 证据: 描述第 5 节、Impact Scope、GitNexus 摘要均写
-
[低]
report.mjs新增的分阶段表格放在一个断言「事件从未触发」的小节标题下- 证据:
benchmarks/eval/report.mjs:74的标题仍是「validation_failed事件从未触发」,而本 PR 让第 82-84 行的计数首次可能非零,同一小节会自相矛盾。 - 受影响调用方/流程: 后续评测报告的可读性。
- 最小可行修复: 把标题改为中性表述(如「
validation_failed拦截计数」),或按 sum 是否为 0 分支输出结论句。
- 证据:
行级发现
- [packages/core/src/executor/executor.ts:774] 这里让整文件 replace 的补丁内容进入
validateCode,于是apply_patch第一次带上import_validity的 block 判定(别名@/x、后续步骤才创建的相对模块),步骤失败并经needsRollback中止整个计划;建议对写盘后判定沿用与写盘否决相同的收窄口径(import 判定降为 warn),或补测试钉死预期行为。局部补丁路径在:904有同样问题。 - [packages/core/src/executor/executor.ts:67]
filter把syntax_validity条目整条删掉,真语法错误(export const a = {)从此既不失败也不出现在validation.results里;改为保留条目、只将其排除出pass/blockedBy计算,保住遥测可见性。 - [packages/core/src/executor/executor.ts:365]
landedPath只取step.params.path,与写前的toolParams.path ?? step.params.path(:762)口径不一致;按同一解析结果取一次路径并向下传递。 - [packages/core/src/executor/executor.ts:880] 这是同一文件在一次步骤内的第二次读回(第三次在
:903);把 landed 内容读一次后作为参数传入validateAfterExecution。 - [benchmarks/eval/report.mjs:77] 新表格挂在「事件从未触发」这一标题下,本 PR 后计数会非零;把标题改为中性表述或按 sum 分支输出。
Karpathy 评审
- 假设: 「计划 schema 不产出 patches,所有 modify 都走整文件 replace」经
planner.ts:465+executor-skills.ts:286核实成立;但resolveFullFileReplaceContent额外依赖collectedContext.files里有原文件——缺失时前置校验静默跳过,#387 对补丁路径的保护是有条件的,这一点没写进描述。另一处隐性假设是「技能不改写 path」(见发现 4)。 - 简洁性: 围栏判据本身是恰当的最小手术;但
dropUnreliableSyntaxVerdicts是一个为绕开 #413 而生的临时层,它与「写前否决」「写后再判围栏」「复用写前结果」共同构成了四条彼此牵制的规则,读者必须同时持有四条才能推断一次写盘的结局。#413 落地后至少要删掉两条。 - 结构质量:
executor.ts越过 1000 行、executeStep约 235 行、错误文案三层嵌套三元、同一文件读三次——同目录已有按关注点拆分的先例,应抽出write-validation.ts(见发现 3、4)。needsRollback保留旧名但语义已变为调度信号(types.ts 注释自己承认),属于名实不符。 - 变更范围: 主体聚焦。
benchmarks/eval/*的分阶段计数与guard.isCheckEnabled是必要配套。Executor.rollback的返回归一化虽属顺带,但与rollback_failed直接相关,可接受。 - 验证: 新增用例扎实且用了真实 guard 与评测里的真实失败样本,
pnpm quality:precommit已跑。但覆盖的是「不否决写盘」,没有覆盖「步骤最终成败」——两个 import 相关用例都只断言了Pre-write validation failed不出现,掩盖了发现 1 的行为。
缺失覆盖
- 整文件 replace 的
apply_patch,内容含@/x别名或前向引用相对 import:断言步骤成败与needsRollback,把发现 1 的预期行为钉死。 create_file写入真语法错误(如export const a = {,非围栏):断言剔除syntax_validity后的预期结局,防止发现 2 的能力退化被静默改回或永久遗忘。- 集成层用例:写前否决返回
needsRollback: true后,executeStepsWithProgressEnforcement确实把剩余步骤置为skipped(当前只在单步层断言了该字段)。 apply_patch且collectedContext.files缺少目标文件时前置校验被跳过的路径,确认降级到写盘后判定符合 #387 的验收口径。
| const content = | ||
| step.action === 'create_file' | ||
| ? ((toolParams.content ?? step.params.content) as string | undefined) | ||
| : resolveFullFileReplaceContent(toolParams, context?.collectedContext.files.get(path)); |
There was a problem hiding this comment.
这里让整文件 replace 的补丁内容进入 validateCode,于是 apply_patch 第一次带上 import_validity 的 block 判定(别名 @/x、后续步骤才创建的相对模块),步骤失败并经 needsRollback 中止整个计划;建议对写盘后判定沿用与写盘否决相同的收窄口径(import 判定降为 warn),或补测试钉死预期行为。局部补丁路径在 :904 有同样问题。
| * #413 落地(换真 parser)后这个函数应当删除。 | ||
| */ | ||
| function dropUnreliableSyntaxVerdicts(validation: ValidationResult): ValidationResult { | ||
| const kept = validation.results.filter((result) => result.type !== 'syntax_validity'); |
There was a problem hiding this comment.
filter 把 syntax_validity 条目整条删掉,真语法错误(export const a = {)从此既不失败也不出现在 validation.results 里;改为保留条目、只将其排除出 pass/blockedBy 计算,保住遥测可见性。
| // 所以一份写进 .ts 的围栏不会再让 postValidation 失败——但它确实是坏内容, | ||
| // 必须自己让步骤失败并触发撤销。这也让「判失败」与「触发回滚」用的是同一条 | ||
| // 确定性判据,不会出现一个判失败、另一个不撤销的错位。 | ||
| const landedPath = step.params.path as string | undefined; |
There was a problem hiding this comment.
landedPath 只取 step.params.path,与写前的 toolParams.path ?? step.params.path(:762)口径不一致;按同一解析结果取一次路径并向下传递。
| // collectedContext.files 的行数快照,而 apply_patch 成功后该 Map 不刷新—— | ||
| // 同一计划内二次改同一文件时,工具可能只替换了前 N 行并保留尾部, | ||
| // 落盘内容 ≠ 被校验的 patch.content。不一致就按读回内容重新判。 | ||
| const landed = this.readWrittenFile(step.params.path as string | undefined); |
There was a problem hiding this comment.
这是同一文件在一次步骤内的第二次读回(第三次在 :903);把 landed 内容读一次后作为参数传入 validateAfterExecution。
| 3. **\`validation_failed\` 事件从未触发** | ||
| 两臂合计 ${sumEvent(arms.full, 'validation_failed') + sumEvent(arms.ablation, 'validation_failed')} 次。该事件挂在执行器不走的那条校验路径上,导致「校验是否起作用」在遥测层面不可观测。 | ||
|
|
||
| **按阶段分解**(只有 \`pre_write\` 是真拦截;\`post_write\` 含「前向引用 import」这类 |
There was a problem hiding this comment.
新表格挂在「事件从未触发」这一标题下,本 PR 后计数会非零;把标题改为中性表述或按 sum 分支输出。
Linked Issue Or Context
validation_failednever fires).Fixes, see below.Why #387 is only partly closed
#387's acceptance is "a create/patch step whose content is syntactically invalid must not leave the file on disk". This PR blocks the subset that can be detected reliably: a markdown fence in a code file — which is the failure the 2026-07-12 eval actually produced (
TS1127: Invalid character).The general case needs a trustworthy syntax check, and the existing one is not. Measured against the built guard,
checkSyntaxValiditycounts quotes per line and returnsseverity: 'block'forexport const msg = "it's fine";, for multi-line template literals, and for JSX text with a contraction. An earlier revision of this PR gave that check write-veto and rollback-trigger power; it would have made any file containing an apostrophe unwritable, and could have deleted a legitimate file via acreatesnapshot rollback (unlinkSync).Tracked as #413. Fixing it means parsing rather than counting, which means
typescriptas a runtime dependency of the guard — a packaging decision that does not belong in this PR.Summary
Content that can be known before the write is now validated before the write, the interception event has an emit site, and rollback has a terminal state.
1. Pre-write validation, narrowly scoped
create_filecontent, and the whole-filereplacepatch that codegen produces for everyapply_patchstep, are validated beforecallTool. A failure means the tool is never invoked — the bad file does not reach disk at all, so rollback is not the mechanism it depends on.Only a markdown fence can veto a write (
detectMarkdownFence). Every other verdict keeps its old post-write meaning — includingimport_validity, which blocks on relative imports of modules a later step in the same plan creates and on path aliases like@/components/x; as a write veto that alone would have made multi-file plans unable to produce their first file. See "Why #387 is only partly closed" above for why the syntax check is not used here either.The same narrowing gates rollback. That symmetry is load-bearing: a
createsnapshot rollback isunlinkSync, so triggering it on an import verdict would delete the very file the veto scoping deliberately let through.2.
apply_patchis covered, andcontentcannot shadow the patchThe plan schema emits no
patches, so every modify step goes through codegen, which produces a single whole-filereplace(startLine: 1, endLine: <original>). That content is fully known before the write.resolveWriteContentrecognises exactly that shape, usingcollectedContext.filesfor the original line count.Resolution is dispatched by action. Plan params are passed through verbatim and the apply_patch skill spreads them, so
{path, content, patches}is reachable; a cross-actioncontentfallback would have validated a string that never gets written while the real patch content went unchecked. Partial-line patches still fall through to post-write validation.Since the pre-write result is computed on the same content, it is reused as the post-write verdict rather than re-running the guard.
3.
validation_faileddistinguishes interceptions from tool errorsvalidateAfterExecutionreturnsresults: []on a tool failure. The event is emitted only when at least one real check failed, so the count is usable as an interception count — which is what #388 is for.4. Rollback has a terminal state
rollback_failedjoins the event union and the desktop reducer, and the reason is appended tostepResult.error. Previously a denied rollback — the default non-interactive case, i.e. the eval harness — left a danglingrollback_startedand no way to tell whether the bad file was still on disk.Executor.rollbackalso normalises its return shape: it promised{ success, message }while the security layer's denial returns{ success: false, error }.5.
needsRollback/writeLeftOnDiskneedsRollbackkeeps its abort semantics (a real check failing aborts remaining steps inprogress-enforcement) but excludes plain tool failures, which previously made one failedread_fileabort an entire plan. "A write is still on disk" moved to a separatewriteLeftOnDiskfield — conflating a disk-state fact with a scheduling signal is what made the field ambiguous.An earlier draft of this description claimed LLM plans leave
validationempty and that this was #387's mechanism. That is wrong:planner.ts:325overwrites it viagetDefaultValidation, which returns tworequired: truerules for both write actions.Impact Scope
packages/core/src/executor/executor.ts— pre-write stage, veto narrowing, patch resolution, rollback outcome,rollback()normalisation.packages/core/src/types.ts—rollback_failedevent,writeLeftOnDisk.packages/core/src/agent/agent.ts— 3 lines wiringemitEvent.apps/desktop/src/state/executionReducer.ts— therollback_failedcase.executor.test.ts,agent.test.ts.Known behaviour changes
benchmarks/eval/report.mjscountsvalidation_failed; this PR makes that count non-zero for the first time.GitNexus Impact Summary
packages/core/src/executor/, with tests in the same package.needsRollback's only consumer isprogress-enforcement.ts:42; the agent's main path goes throughexecuteStepsWithErrorFeedback/phase-runner, which do not read it, so the blast radius isExecutor.executeSteps' direct embedders. TheAgentEventunion gains one member;apps/cli/src/ui/bridge.tsandapps/vscode/src/state.tsboth havedefaultbranches, and the desktop reducer has a matching case.ExecutorOutput.writeLeftOnDiskis optional and additive.pnpm quality:precommitexit 0.Verification
pnpm quality:precommit— exit 0.HallucinationGuardand the actual failing artifact from the eval run (a markdown fence in a.tsx), and asserts the file is absent from disk.snapshotIdand rollback permitted, asserting no rollback occurs.apply_patchstep carrying bothcontentandpatchesasserts the patch content is what reaches the guard.rollback_failedis emitted and the error says the file is still on disk.Checklist
pnpm quality:precommit, or explained why it could not run.pnpm quality:localfor critical skeleton changes, or explained why it could not run.🤖 Generated with Claude Code