ADF judge: bounded decision-model agreement experiment (current judge vs Jev vs Levanto Sage vs local) #965

Goal

A bounded experiment on one ADF judge decision: does a typed decision model agree with the current reasoned verdict, and what does each option cost per completed task (not per call)?

Arms

1. The current ADF judge verdict (baseline).
2. TypeSafe Jev Choice (already reachable through the llm-proxy typesafe:: / jev:: adapter).
3. Levanto Sage choice / yesno, on its free tier (100 decisions a month), with reasoning: "off" and "on" as separate conditions.
4. A local open-weights backend (openjev-sglang or AnyJev), if one is available on the same /v1/systemone shape.

Method

Build the eval with the claude-api build-eval discipline:

• Cases sampled from real judge inputs; hard cases chosen by a human, not by current failures.
• Permute the option order for every decision-model arm. A permutation swing larger than the arm-to-arm difference invalidates that arm's result.
• Run each grader twice to measure its flip rate.
• State the noise floor before comparing arms. Report agreement with Wilson CIs, not point estimates.

Out of scope

Replacing the reasoned judge. Neither vendor returns a justification: Sage's meta.reasoning is telemetry only. So a decision-model verdict cannot satisfy the requirement that verdicts be auditable. This experiment measures agreement and cost only.

Acceptance

• Agreement and cost per task for each arm, with CIs and the permutation swing.
• A written recommendation: keep, trial, or drop.

Related

• Lane-policy decision for metered classifiers (separate issue).

Mirror of Gitea terraphim/terraphim-ai#3418 (source of truth).