[shekharprateek]

Ports upstream d8c07b6 ("Stop omp from running a different model than the one requested", #218), which landed after this repo's last substantive sync and never came across. Unlike the earlier gaps this is a whole missing commit, not a half-finished port — and upstream shipped tests with it, so this is close to a straight port.

The bug

omp's --model is a fuzzy match. Hand it a Bedrock id that isn't in the catalog omp currently has loaded and it resolves to the nearest id it knows instead of erroring. Upstream recorded the incident: omp 18.1.15 sent every us.anthropic.claude-haiku-5-5 request to Haiku 4.5.

A caching detail is what exposes it:

Command	Catalog it sees
omp -p in a fresh agent dir	only the snapshot bundled in the binary
omp models <provider> --json	fetches the full catalog, caches it in that dir

The harness gives every task its own PI_CODING_AGENT_DIR for isolation — so every task starts from the stale bundled snapshot, and any model newer than the installed binary is silently downgraded.

Why this one is worse than it looks

The run succeeds. Six artifacts, a populated metrics.json, real judge scores, a cost figure — all filed under the folder for the model you asked for, with nothing recording that a different model did the work. For a repo whose output is per-model cost/quality comparisons, that's a silently corrupted data point whose only symptom is a score that looks slightly off.

It matters most for omp specifically: it's the default harness in the README's /benchmark example and the one behind the published frontier, so its numbers are what reach vend/swe-router/models.json and drive every routing decision the skill makes.

Two independent layers

Neither is sufficient alone:

1. Prove and warm, before each Bedrock task. List the catalog in that task's agent dir and assert the exact wire id is present. This both verifies the id and populates the dir's cache, so the following omp -p resolves correctly rather than falling back to the bundled snapshot. The warming is why this is a correctness fix, not just a check. On failure it names the remedies: omp update, omp models find <id>, and the id as aws bedrock list-inference-profiles reports it.
2. Verify after the fact. Every assistant message in omp's stream names the model that answered. _omp_models_used collects them; _flag_omp_model_mismatch fails the result if any differs from what was requested — so a mismatch is caught from omp's own reporting rather than trusted.

Supporting pieces:

• Fail-fast in main() so a bad id dies once up front, not once per task across a long batch.
• models_used in metrics.json, so the model that actually answered stays visible in the data.
• error_model_mismatch marked non-retryable — a retry resolves the same id the same way, so it would only spend money to fail again.

Verification

All 10 of upstream's tests error against main and pass with the fix:

reverted to main   ERROR: test_catalog_parses_model_ids ...
with fix           OK (10 tests)

The mismatch test reproduces the original incident verbatim — haiku-5-5 requested, haiku-4-5 answered.

Behavioural checks by hand:

--agent omp --provider bedrock   → exit 1, "omp CLI not found on PATH; see docs/omp-setup.md"
                                   (fires before any repo is cloned)
--agent claude --dry-run         → 0 mentions of omp; no other agent affected

652 tests pass, 10 new. ruff, ruff format, mypy, bandit (0 issues at every severity) clean. The one failure, test_judge_common.test_lowercase_preferred_on_collision, fails on main too — it assumes a case-sensitive filesystem.

One trade-off worth naming

The per-task catalog call costs a subprocess and up to a 60s timeout per task — real overhead on a 21-task dataset, for a problem that only bites when your omp binary predates your model. I kept upstream's trade-off rather than optimising it, on the grounds that a silently wrong data point is worse than a slow run, and because layer 1 is load-bearing for correctness (the cache warming) and not purely a safety tax.

Audit status

This was the last behavioural defect on the list. Remaining: versioning.