Repository navigation
feat(eval): add OSWorld 2.1 subset evaluation - #2
Merged
Merged
Conversation
two-tiger
pushed a commit
to two-tiger/Tandem
that referenced
this pull request
Oct 22, 2026
PR AI45Lab#2 added the 56-task OSWorld 2.1 subset on the old flat config (kimi / jev_endpoint / osw21_root at the top level). That layout is gone - the config is now harness / eval / log, with the benchmarks under eval.benchmarks.<name>. This commit moves the OSW2.1 support onto the new schema and fixes what the review of PR AI45Lab#2 turned up. Config: eval.benchmarks.osworld21.{root,vm,asset_root,shared_env,proxy, volume_size,inner_image} eval.mounts gains the canonical view gpfs://gpfs2/evobox-share-gpfs2 -> /mnt/shared-storage-gpfs2/evobox-share-gpfs2 the prepared/runtime symlink layer only resolves there; the dev machine has no canonical mount, so those paths are worker-only. The inner image must be release-pinned (<repo>@sha256:...). The schema rejects a "latest" tag, and build_job_spec refuses to build an osworld21 run without a pinned image, so a wrong image cannot be picked up silently. Fixes from the review: - the osworld21 paths are part of the required-key validation now, so an outdated config fails with "missing keys" instead of a bare KeyError; - the 56-task subset records where it comes from (the environment's validation/coverage.json, passed tasks) and a new test checks that every id in the list still has a task file in the runtime, so the snapshot cannot rot unnoticed (skipped when the runtime is not mounted); - worker.sh reads the spec through one spec_get helper instead of three inline python one-liners; - the anchor comments PR AI45Lab#2 dropped from osworld.py are back: the DEVIATIONS references, the "task_001 answered 404 because the asset URL was baked in at import time" diagnosis, the setup-cache seeding rationale and the receipt discipline; - the batch script documents BENCHMARK / TASKS / ROUTE_POLICY next to the model switches. osworld.py is back to 311 lines: the comments are worth more than the line budget, and the module split is left to the discipline rework. Verification: ruff check src tests demo, pytest (235 passed) and tandem check are green; a mock run through infra/worker.sh still works, and the osworld21 job spec builds with the expected env section.
two-tiger
pushed a commit
to two-tiger/Tandem
that referenced
this pull request
Oct 22, 2026
The osworld21 run died before it did anything: the job spec carried an
empty secrets section, so worker.sh had no registry auth to write to
~/.docker/config.json and exited at the pull step ("缺少 registry 凭据").
Root cause was mine: the rebase conflict resolution took rjob_spec.py
wholesale from the config-refactor side, which dropped the "osworld21"
that PR AI45Lab#2 had added to the credentials whitelist. The env section and
the inner-image guard were ported back, this whitelist was not.
A regression test now checks that every real benchmark produces a spec
with credentials (slow_model, fast_model, registry), so a benchmark
missing from a whitelist fails in the test suite instead of on the
cluster.
worker.sh also stops swallowing the reason: the registry failure path
ran `cat` on an entry log that does not exist yet, and errexit killed
the shell before it could print the message - the platform log only
showed the python error and exit status 1 instead of 3. The docker pull
and dependency check failures now echo the tail of their log files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
改动
接入 OSWorld 2.1 的 56 题子集，复用现有测评链路。
未新增仓库文件，未修改 src/tandem/ 主体和协议。
环境依赖及 setup 兼容由共享 runtime 负责，不在 harness 中增加单题修补。
不包含通用 rjob 生命周期修复。
验证
VM 启动、截图及 A11y 获取、Kimi 调用、JEV 看图选动作、
GUI 执行回执和官方评分均已验证。
score=0、stop_reason=max_steps，仅验证启动链路。
运行记录：osw21-current-startup-20261022-1802-s1。