Skip to content

Centralize the LLM model names and record the vision default - #155

Merged
hechth merged 5 commits into
mainfrom
hechth/issue145
Sep 25, 2026
Merged

hechth merged 5 commits into
mainfrom
hechth/issue145

Conversation

@hechth

@hechth hechth commented Sep 24, 2026

Copy link
Copy Markdown
Collaborator

Refs #145 — deliberately not Closes, see "Unresolved" below.

Validation of the issue against main

The issue is stale in its main claim. Checked on main (d5f61cc):

Issue claim Actual state on main
"main still contains the retired model name" False — git grep llama-4-scout origin/main -- src tests examples returns nothing. PR #149 merged 2026-09-24, bringing the swap.
"tests/test_pymupdf_parser.py:147 still uses redhatai-scout" False — that site is qwen3.5-122b. The only redhatai-scout left in the repo is a test fixture string in tests/test_service_availability.py:27, where it is deliberate sample content for the failure classifier, not a model being requested.
"examples/find_chemicals_relationships.py:57,68 duplicate the class default" True, still there.
"examples/parse_pdfs.py:14 uses qwen3.5" True, still there.
"No decision is recorded anywhere about which model is the supported vision model" True.

So of the four remaining items, only 3 (make the model configurable) and 4 (align example model names) plus the recording part of 2 were still actionable. Item 1 (confirm the model is currently available) could not be verified — see below.

Changes

  • DEFAULT_MODEL = "gpt-oss-120b" and DEFAULT_VISION_MODEL = "qwen3.5-122b" defined once in src/aoptk/text_generation_api.py, in the style of the existing module-level topics constant. __init__'s default now refers to DEFAULT_MODEL instead of repeating the literal.
  • The four vision call sites (tests/test_text_generation.py ×3, tests/test_pymupdf_parser.py ×1) use DEFAULT_VISION_MODEL. examples/parse_pdfs.py — which does figure→text vision work and passed a bare qwen3.5 — now uses the same constant.
  • examples/find_chemicals_relationships.py passed model="gpt-oss-120b", i.e. the class default, in two places; it now passes no model. The issue itself flags this duplication as a problem, and dropping it rather than introducing a DEFAULT_MODEL import keeps this PR clear of Retry LLM answers the transport accepted but we cannot use #154, which rewrites the same loop.
  • DEFAULT_VISION_MODEL carries the rationale the issue says was recorded nowhere.
  • New tests/test_default_models.py: greps src/aoptk for retired model names, and pins that the constructor default is the shared constant rather than a second copy of its value.

After this, git grep 'model="' over tracked source, tests and examples returns nothing.

Why a static test for retired names

While working on this I found the reason the issue's own verification step ("grep returns nothing") deserves a test rather than a checklist item. tests/service_availability.py classifies "invalid model name" as a service problem, so when the endpoint reports a retired model, the vision tests xfail instead of failing. A regression to a retired model therefore silences the four live vision tests rather than breaking them — the suite structurally cannot catch it. tests/test_default_models.py covers the part the live tests cannot, and is fast, offline, and non-fragile (it names retired models, not the current one, so a routine model bump does not break it).

Unresolved — needs a key to settle, and I did not guess

I could not confirm step 1: whether qwen3.5-122b is still available. GET /v1/models returns 401 without a key, and I have no CERIT_API_KEY here. I kept the value at qwen3.5-122b rather than inventing a replacement, and the comment does not claim it is verified.

Worth flagging: CI evidence suggests qwen3.5-122b may itself now be retired.

  • Run 28241321216 (June, on qwen3.5-122b): the three vision tests PASSED.
  • Runs 35978921687 and 35981367222 (this month, same model string): all four vision tests XFAIL, while every live test using the default model passes in the same runs.

The only variable between passing and xfailing is the vision model name. Given the masking behaviour above, that pattern is consistent with qwen3.5-122b having been retired — but xfail truncates the reason in -v output and I could not extract it from the logs, so this stays a hypothesis, not a conclusion. This needs someone with a key to run pytest -m openai -rs and read the reason; if it is a retired model, the constant is the one-line fix, which is the point of centralizing it.

pytest -m openai is not run on SonarCloud, only on the build matrix's 3.12/ubuntu job.

Verification

  • ruff check, ruff format --check, mypy — clean.
  • pytest -m "not openai" on main + this change: 125 passed, 2 skipped, 25 xfailed on the offline subset; full suite run on the pre-change tree gave 157 passed / 3 skipped / 25 xfailed / 53 xpassed, no regressions.
  • pytest -m openai --collect-only collects all 19 live tests.
  • Constants resolve; TextGenerationAPI().model == DEFAULT_MODEL.

Deliberately not touched

  • tests/test_service_availability.py:27 — redhatai-scout there is sample error text for the failure classifier. Renaming it would weaken that test to satisfy a grep. The new test scopes its scan to src/aoptk for this reason.
  • Making the vision model settable from outside the process (env var / config). The issue offers "a class-level constant or a constructor argument"; the constructor already takes model=, so the constant is the missing half. An env override is a separate design decision.

🤖 Generated with Claude Code

hechth and others added 5 commits September 24, 2026 15:37
The retired model itself is already gone from `main` (PR #149 replaced the
three vision sites), but every model name was still a string literal in a
test or an example, so the next e-infra retirement again means a grep
across the repo. Issue #145 asks for the model to be configurable rather
than hardcoded; this does that.

`DEFAULT_MODEL` and `DEFAULT_VISION_MODEL` are now defined once in
`text_generation_api.py`, in the style of the module-level `topics`
constant, and `__init__`'s default refers to the first. The four vision
call sites in the tests and the PDF-parsing example use the vision
constant; `examples/parse_pdfs.py` used a bare `qwen3.5`, which is now the
same value the tests use. `examples/find_chemicals_relationships.py`
passed the class default explicitly and no longer passes a model at all.

`DEFAULT_VISION_MODEL` also records the decision the issue says was
recorded nowhere. A comment explains the trap: the endpoint answers a
retired model with a bad-request error that the test suite classifies as
an unavailable service and xfails, so pointing at a retired model
silences the vision tests instead of failing them.

`tests/test_default_models.py` covers exactly that blind spot - it greps
`src/aoptk` for retired model names, which the live tests cannot do, and
pins that the constructor default is the shared constant rather than a
second copy of its value.

The issue's first step - confirming the replacement model is still
available - could not be verified from a checkout: it needs a
`CERIT_API_KEY`. See the PR for the CI evidence bearing on it.

Refs #145

Co-Authored-By: Claude Code <noreply@anthropic.com>
Whether a model is still available was only discoverable by calling the
endpoint. Issue #145 is the visible cost: `Qwen3.5-122b` had been archived
by the provider on 2026-08-26, and because the endpoint reports a retired
model as a bad-request error that the test harness classifies as an
unavailable service, the four vision tests xfailed rather than failing. A
month could pass that way unnoticed.

`https://llm.ai.e-infra.cz/status/api/v1/models` answers without a
credential (unlike `/v1/models`, which is why this was not noticed
sooner), so the offer can be checked from a bare checkout.

`models/llm_models.json` pins the 25 models as observed, and
`scripts/update_llm_models.sh` rewrites it. Two properties the weekly job
depends on:

- Observed fields are rewritten, but a curated `note` is carried over. The
  API reports no capability information, so a refresh that erased what we
  know about a model would quietly destroy the only useful content.
- Exit status distinguishes "no change" (0) from "the offer changed" (1)
  from "could not tell" (2). Conflating the last two is how a job reports
  an all-clear on a day the service was unreachable. The comparison ignores
  throughput and `last_seen`, which the provider advances every minute for
  every online model; comparing the raw payload would open a pull request
  weekly over noise.

`--summary BEFORE AFTER` describes a difference in the same terms
`--check` uses, for the weekly job's pull request body.

`tests/test_llm_model_catalog.py` reports what the catalog implies about
our own defaults - retired, absent, an unrecognised status, a stale file,
models added since the last refresh. Those are warnings, not failures: a
provider retiring a model is not a defect in the code under test, and
failing every open pull request over it would only train people to ignore
the suite. Only the catalog's own well-formedness fails, since a malformed
file makes every other check here vacuous. Names are matched
case-insensitively; the provider writes `Gpt-oss-120b` and the constant is
`gpt-oss-120b`.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Asks the status API every Monday whether models/llm_models.json still
matches what the endpoint offers, and opens a pull request with the
refresh when it does not. `workflow_dispatch` is also allowed, so a
maintainer can re-check after the provider announces a change instead of
waiting for Monday.

The job acts on `--check`'s exit status rather than on a `git diff`,
because a write always rewrites the volatile fields and would therefore
produce a pull request every week regardless of what the provider did. An
exit of 2 fails the job instead of being treated as "nothing new".

Two consequences are recorded in the workflow rather than left to be
discovered:

- The pull request is opened with the default `GITHUB_TOKEN`, which does
  not trigger `build.yml`, so its checks need a manual re-run once opened.
  A personal access token would avoid the click at the cost of a credential
  to create and rotate; while these refreshes are rare the click is
  cheaper.
- No `labels:` input. This repository has no label for automated pull
  requests, and the action creates a missing label in the repository
  settings rather than failing, so a typo would silently edit settings.

The job cannot decide the two things that matter about a change, so the
body asks a human to: whether one of our defaults was retired, and whether
a new model serves aoptk better than the current choice. The status API
reports no capabilities, so the second is a judgement call.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The status API reports `Qwen3.5-122b` as archived, last seen 2026-08-26.
It was `DEFAULT_VISION_MODEL`, which settles the symptom in #145 without
leaving anything to infer: the endpoint answers a retired model with a
bad-request error, the harness classifies that as an unavailable service,
and the four vision tests xfail. A green suite had been reporting four
silently skipped image and scanned-PDF tests for a month.

`qwen3.5-int4` is the closest online model of the same family, listed by
the provider since 2026-07-28 and online in 48/48 uptime samples. That is
an availability argument only. The status API exposes no capability data,
and `CERIT_API_KEY` is not available here, so no vision test has been
observed passing against it; the constant's comment says as much rather
than implying a verification that did not happen. Changing it again is a
one-line edit.

`qwen3.5-122b` is added to the retired names that
`tests/test_default_models.py` greps `src` for, so reintroducing it fails
statically instead of xfailing. That grep now ignores case: the provider
writes `Qwen3.5-122b` and the constants do not, and the endpoint accepts
either, so a case-sensitive search would miss the spelling most likely to
be pasted in.

Refs #145

Co-Authored-By: Claude Code <noreply@anthropic.com>
@hechth
hechth merged commit e1d4fdb into main Sep 25, 2026
17 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant