Skip to content

feat(eval): support all OSWorld 2.1 tasks with shared environment - #4

Merged
two-tiger merged 2 commits into
AI45Lab:mainfrom
xxxyxun:feature/osworld21-full-environment
Oct 24, 2026
Merged

two-tiger merged 2 commits into
AI45Lab:mainfrom
xxxyxun:feature/osworld21-full-environment

Conversation

@xxxyxun

@xxxyxun xxxyxun commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator

改动

  • 将 OSWorld 2.1 任务清单从 56 题扩展为官方 001–108,沿用已有单题、批量及分片入口。
  • 更新配置模板,使用共享通用 VM 和 prepared runtime。
  • worker 接入共享网站地址解析及 no_proxy 配置。
  • 支持从统一配置传递 GitLab、judge、用户模拟器凭据;私有 job spec 使用 600 权限,结果中的凭据保持脱敏。
  • 记录替代 judge/用户模拟器的模型身份,避免与原生评测配置混淆。
  • 更新现有配置文档、结果契约说明及对应测试。

验证

  • Ruff 检查通过。
  • 17 项相关测试通过。
  • worker 和批量脚本的 Bash 语法检查通过。
  • 真实环境链路已确认 108/108;
  • 环境测试使用固定 5 步真实操作,不代表 agent 解题成绩。

回归记录:
/mnt/shared-storage-gpfs2/evobox-share-gpfs2/xuyuanxun/tandem_osworld21_shared_20261023/rjob_logs/osw21-pr-env108-2100/

范围

不修改核心 harness、Planner、JEV、Verify 或官方评分逻辑。

@two-tiger two-tiger left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@two-tiger
two-tiger merged commit 17bd5ff into AI45Lab:main Oct 24, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants