feat(tasks): add Ag-LiveCodeBench-X — multi-language LiveCodeBench

[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

• Adds Ag-LiveCodeBench-X (dataset, scripts, paper — Agnostics, ICLR 2026): LiveCodeBench v5's 499 stdin/stdout problems, made language-agnostic, so the target language is a task argument rather than a property of the data. Ships status="stable" on a reproduced paper cell — see "Alignment" below.
• language is required with no default: the benchmark measures a different thing per language, so a silent default would let two runs of one task name mean different things. report.json carries language / container_lang / verifier_image, because the task name cannot hold them and a column that doesn't say which language it ran isn't comparable with anything.
• The prompt is a frozen dspy==3.0.0b2 ChatAdapter rendering of upstream's ChainOfThought(SolveProblem), captured by running it: two properties are invisible in the signature (an 8-space instruction indent inside the system message; a trailing space on the first field-list line and not the last). Frozen rather than depending on DSPy at runtime, so a patch release can't re-word the fleet's prompts; pinned by sha256 in tests.
• Response parsing and code extraction are reimplemented, not vendored — upstream ships no license (no Hub license field, no LICENSE) — including an upstream offset bug where an indented [[ ## field ## ]] header leaks ]] into the field body. Reproduced deliberately; "fixing" it would move scores.
• vendor/code-evaluator gains source="agnostics", forwarding a submission to a per-language verifier container over the Agnostics protocol (one JSON line each way). That's what makes Lua / R / Julia / OCaml / Fortran reachable without installing five toolchains in the evaluator image. Purely additive — no existing source changes behaviour.
• The verifier is pinned by digest, not by upstream's mutable tag, and a language with no pinned digest is refused (infra:unpinned-lang) rather than floated. Rationale and the re-pinning procedure are in vendor/code-evaluator/VENDORED.md.

Since the first review pass (rebased onto main, 19 commits): two defects found by the validation run are fixed, and the report was adapted to the interval API that landed on main in the meantime.

• verifier_command enforced the digest-pin refusal only on the default podman path — the override branch substituted an empty string for {image} instead of raising infra:unpinned-lang. podman is the one runtime needing no override, so every other deployment (upstream's own apptainer included) ran the branch where the documented guarantee was missing.
• The module and a test name attributed the rendering's 8-space instruction indent to "the source docstring's 8-space indentation". Upstream's docstring is indented 4, and Signature.instructions dedents it to zero — the 8 is ChatAdapter's own constant. Bytes were always right; the stated reason would mislead whoever re-captures the prompt on the next DSPy bump.

Enumerated divergence with a consequence: upstream's ChatAdapter.__call__ retries a failed parse through JSONAdapter with a differently-shaped prompt; this port makes one call. A reply missing either marker scores 0 here, so the number is a lower bound whose gap is exactly n_unextracted.

Alignment (updated — this section previously said neither had been done): the container hop is now exercised against the real ghcr.io/nuprl/agnostics lua verifier at the digest this branch pins, and one paper cell is reproduced.

	pass@1
Paper Table 1, Qwen 3 8B × Lua	11
This port, 499 problems	10.42 (52/499)

0.58 pp, or 0.30 combined binomial σ (σ_D = 1.96 pp) — inside noise, which was the promotion criterion this PR set for itself. Upstream's README settings for Qwen 3: temperature 0.2, top_p 0.95, max_tokens 2048, language "Lua /nothink", one completion. Verdicts: 74.5% fail:error, 11.7% fail:wrong-output, 10.7% success, 3.1% infra:timeout — the error-dominated mix the paper's own Figure 6 describes for base models on low-resource languages.

Three caveats, all carried in the shipped docstring and reference_impl.notes rather than only here, none of which moves the number out of noise:

• the paper served qwen3_8b_awq (AWQ 4-bit); OpenRouter's only Qwen3-8B provider reports quantization unknown. Same checkpoint, possibly different precision — recorded, not controlled;
• 13 of 499 rollouts never generated (12 provider rate-limits, one content filter) and count as failures, as upstream's pass1 counts a generation that produced no solution. They bias the number down: at the observed rate it would be 10.70;
• the no-JSONAdapter-retry divergence above cost exactly one row (n_unextracted = 1).

The other four languages and every other model in the table are unmeasured, and the docstring says so.

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check)
• Type check clean (ty check) — mypy errors on the new files are exactly the 8+1 repo-wide no-untyped-def stage-method pattern the sibling livecodebench_code_generation_0shot_gen already has, no more
• Unit tests pass — 7296 unit tests after rebasing onto current main, 48 of them new; check_preflight.py 25/25 PASS (incl. check_links --level deep, check_meta_index_sync, check_reference_kind, check_report_declarations, check_dep_coverage)

Manual

• Prompt bytes are byte-identical to the captured DSPy rendering — sha256 match, and mutation-tested: deleting the single trailing space that looks like a typo fails 2 tests.
• Parser reproduces ChatAdapter.parse — all 10 parametrised expectations were produced by running dspy==3.0.0b2's own parser on the same inputs, the indented-header offset bug included.
• End-to-end against the real verifier container — real uvicorn + ghcr.io/nuprl/agnostics at its pinned digest, run under udocker (this box has no podman): success / fail:wrong-output / fail:error come back verbatim from the container, infra:timeout / infra:bad-lang / infra:no-test all fire, and data.verifier_image carries the digest that ran. Control: a hand-written correct Lua solution scores success on a real decoded LiveCodeBench suite while a deliberately wrong variant scores fail:wrong-output — which is what rules out the base64→zlib→pickle→JSON decode and the whitespace handling as an explanation for any score.
• Frozen prompt verified against a real dspy==3.0.0b2 — not just against its own hash. Installing the pinned library and rendering upstream's signature reproduces both the system and the user message byte-identically. (It also caught a mis-attributed comment: the 8-space indent is ChatAdapter's own constant, not the source docstring's 4-space indent, which Signature.instructions dedents away. Fixed.)
• Digest pinning, both paths — lua / jl / ml / f90 resolve to their digests and report them; julia and rust are refused as infra:unpinned-lang; a {lang}-templated override reports no image (naming a digest that didn't run would be worse than naming none).
• Model run and score alignment — 499 problems, Qwen 3 8B × Lua, 10.42 vs the paper's 11 (0.30 σ). See "Alignment" above.
• All eight pinned digests re-checked against the registry (2026-09-11) — every tag still resolves to the digest the table pins, so the pin is currently exact rather than merely historical.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring — on all three sieval/ + tests/ files. Deliberately absent on vendor/code-evaluator/app/exec_agnostics.py: no file under vendor/ carries it, and that patch is written to be landed in scitix/code-evaluator upstream.
• No new upper-layer dependencies added to core/ — core/ is untouched
• Deleted code verified — nothing deleted

If: New or Modified Benchmark

• Reference paper/repo linked in Summary
• Score comparison table — in "Alignment" above: Qwen 3 8B × Lua, published 11 vs measured 10.42, 0.30 σ. Anchored on the paper's Table 1; the remaining cells (Qwen-3 4B, DeepSeek-Coder-6.7B-Instruct, Phi-4-Mini; Julia / R / OCaml / Fortran) are unmeasured — of the other three models, none currently has a serving provider reachable from here.
• Dataset loading — the pinned revision 15bef6bf was downloaded and loaded: 5 parquet shards, 499 rows in test, exactly the loader's three string columns. (Loaded via the pinned revision's snapshot rather than staged with sieval dataset download, to avoid a second 3.9 GB copy on a box at 96% disk.)
• Task registered — auto-discovered; check_tasks sees 66 exports, __init__.pyi and meta/index.json regenerated

If: New Dependency

• None. The loader needs only datasets and the task only httpx, both core; check_dep_coverage passes with no new group.