[feat][evaluation]: support declaring custom output field schemas for CustomAgent - #627
Merged
Conversation
M-7358884601
长连接 Agent（CustomAgent）非消息额外 KV 数据透出需求的 coze-loop 侧改动
（方案 B：声明入口在 fornax 应用注册页，经 ListApplicationRPC 注入，IDL 零改动）。
1. entity.CustomAgent 新增 CustomFieldSchemas []*CustomFieldSchema
（复用既有 entity.CustomFieldSchema，写法照抄 CustomRPCServer）
2. 新增 ValidateCustomFieldSchemas 创建时前置校验：
- 字段名 ^[a-zA-Z][a-zA-Z0-9_]*$ + 上限 50 字符，与前端既有
columnNameRuleValidator / maxLength={50} 严格一致（不允许 _ 开头），
避免「前端拦得住、OpenAPI 绕得过」的脏数据死锁
- 保留字拦截（引用 consts 常量，不写字面量）
- 同批重名拦截
- 类型白名单：4 类标量 + 多模态；显式拒绝 Object/Array/Trajectory
（Trajectory 会被 buildOutputSchema 静默丢弃）
3. 单测覆盖 R1-R8b 全部规则与合法用例边界
P0：长连接 Agent 声明输出字段后创建评测对象必失败，报「缺少字段类型」。 根因：ValidateCustomFieldSchemas 把 SchemaKey 当成标量类型的必填载体， 但既有范式（前端 custom-field-schema-convert.ts:35-43 + field-convert.ts convertDataTypeToSchema，后端 buildOutputSchema）里 SchemaKey 只用来标记 Trajectory 一种特例： - 标量 content_type=text, schema_key=undefined, text_schema=JSON Schema - 多模态 content_type=multi_part, schema_key=undefined - 轨迹 content_type=text, schema_key=trajectory 前端提交的 5 种类型 schema_key 全是 undefined，于是全部被判「缺少字段类型」。 改动： - 类型判定改为对齐既有范式：ContentType==MultiPart 即多模态； 显式带 SchemaKey 时查白名单（拦 Trajectory 等）；其余以 text_schema 顶层 type 为准（白名单 string/integer/number/boolean）。 SchemaKey 不再必填。 - Object/Array 的拒绝从「无对应 SchemaKey」迁到「text_schema 顶层 type 为 object/array」，保持原意图在新范式下的等价落点。 - 其他既有校验（名字正则/长度 50/保留字/同批重名）逐字未动。 - 单测改用前端真实 wire 值（schema_key=nil + text_schema）， 并新增 TestValidateCustomFieldSchemas_FrontendWireFormat 防回归。 爆炸半径：ValidateCustomFieldSchemas 仅由 CustomAgent 的 BuildBySource 调用，CustomPsm / SandboxAgent / A2A / WebAgent 均不经过此函数。 Co-Authored-By: Claude <noreply@anthropic.com>
…M-7358884601 三处修正（CR 发现）： 1. 删除保留字校验（原 R6）。该规则本次新增、main 上不存在同类校验，而对照的 既有类型 custom_psm 完全不经过任何校验，一直允许声明 actual_output。 这条规则反而是本需求 P0-2 的根因：它导致前端无法像 custom_psm 那样预置 actual_output 只读锁定行。改为与 custom_psm 一致地接受，前端稳定传、 后端 buildCustomAgentOutputSchema 侧去重。 2. 堵住 text_schema 两条绕过路径。原「多模态」与「显式带白名单 SchemaKey」 两个分支都直接 return nil 不看 TextSchema，而消费侧对 TextSchema 非空是 无条件覆盖 JsonSchema、与走哪条分支无关 —— object/array 因此可借这两条路 夹带进 OutputSchema。新增 validateCustomFieldTextSchemaIfPresent： 为空放行、非空仍查白名单。 3. 新增声明数量上限 R1b = 20，对齐前端表单 maxColumn（custom-field-schema-config .tsx），与既有 maxLength=50 同一条铁律：后端不设限则 OpenAPI 可写入远超上限 的声明，用户下次在页面编辑该 Agent 时因前端上限无法保存（脏数据死锁）。 测试：删除 4 条已失效的保留字必拦用例，新增 actual_output / trajectory 字段名 放行用例、20/21 边界用例、4 条绕过必拦 + 2 条合法对照。均为纯函数直调，无 mockey。
|
|
Asichurter
requested review from
jamesonics and
kidkidkid
and removed request for
jamesonics and
kidkidkid
August 24, 2026 09:13
Codecov Report✅ All modified and coverable lines are covered by tests. @@ Coverage Diff @@
## main #627 +/- ##
==========================================
+ Coverage 78.36% 78.41% +0.04%
==========================================
Files 693 694 +1
Lines 85184 85353 +169
==========================================
+ Hits 66753 66928 +175
+ Misses 14503 14500 -3
+ Partials 3928 3925 -3
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 13 files with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
…d messages in English Translate comments, test names and error messages to English and remove non-public context, so the file is consistent with the repository's open-source conventions. No logic or test data changes. Co-Authored-By: Claude <noreply@anthropic.com>
Asichurter
changed the title
lizwang11
approved these changes
Aug 24, 2026
xueyizheng
approved these changes
Aug 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
feat
Check the PR title
(Optional) Translate the PR title into Chinese
(Optional) More detailed description for this PR(en: English/zh: Chinese)
en:
Allow a CustomAgent evaluation target to declare custom output field schemas, so
downstream evaluation can consume extra key-value output beyond the message body.
Changes:
entity.CustomAgentgains aCustomFieldSchemas []*CustomFieldSchemafield(reusing the existing
entity.CustomFieldSchematype).ValidateCustomFieldSchemas, a create-time validation that rejectsinvalid declarations rather than falling back silently:
^[a-zA-Z][a-zA-Z0-9_]*$, and be at most 50 characters;float / boolean / multipart); object, array and trajectory are rejected.
(Optional) Which issue(s) this PR fixes