Add-models: re-judge or retry a model a run already has

[nojibe]

Summary

Two follow-ups to the Apertus add-models job. Until now, the job skipped any blueprint whose latest run already had the model, so a run with bad data for that model couldn't be repaired.

• Re-judge (rejudge). A judging outage near the end of the job (OpenRouter 402s) left Apertus's scores on 3 benchmarks mostly failed: 16 of 40, 107 of 166, and 24 of 25 points. That put it last on each. With rejudge, the job keeps the model's saved responses, drops its saved scores and judges only that model again.
• Retry failed answers (retryFailed). The Current AI router rejected some Apertus requests, up to 30–40% on a few benchmarks. With retryFailed, the job asks the model again only on prompts where its answer failed, and judges just those new answers. Every other answer and score is kept, and a blueprint with no failed answers is skipped.

Each run that changes publishes a new run, the same way adding a model does. Other models' responses and scores are copied unchanged.

Changes

• addModelsToLatestRun(configId, models, logger, { rejudge?, retryFailed? }): both options apply only to listed models the latest run already contains. Asking a model is now one shared helper, used for both added models and retries.
• POST /api/internal/add-models-to-runs accepts rejudge and retryFailed booleans.
• The Add Models To Runs workflow gets rejudge and retry_failed checkboxes, both off by default.

Test plan

• Service tests (8): re-judge keeps responses and drops only that model's scores; retry asks only the failed pair in the run's own variant, keeps the good answer's score, and leaves the retried pair to be judged; retry skips a blueprint with no failures.
• Route tests (9): both flags are passed through.
• Full suite: 1175 passed, 2 skipped. tsc --noEmit is clean.
• After merge:
  • run with rejudge on visual__web_design_101, yka-set and yka__disability-rights-accommodation;
  • then run with retry_failed and rebuild_summaries on the full Apertus list.

Risks

Both flags are off by default, so existing workflow runs behave exactly as before. Each repaired blueprint gets one new published run, and older runs are untouched.

🤖 Generated with Claude Code

https://claude.ai/code/session_015roAwvpcszBvjX1Ce5NFuj

---

Generated by Claude Code

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

_{Tip: disable this comment in your organization's Code Review settings.}

[railway-app]

🚅 Deployed to the app-pr-98 environment in weval

Service	Status	Web	Updated
weval-app	🕒 Building (View Logs)	Web	Oct 1, 2026 at 3:07 pm UTC