Centralize the LLM model names and record the vision default - #155
Merged
Merged
Conversation
The retired model itself is already gone from `main` (PR #149 replaced the three vision sites), but every model name was still a string literal in a test or an example, so the next e-infra retirement again means a grep across the repo. Issue #145 asks for the model to be configurable rather than hardcoded; this does that. `DEFAULT_MODEL` and `DEFAULT_VISION_MODEL` are now defined once in `text_generation_api.py`, in the style of the module-level `topics` constant, and `__init__`'s default refers to the first. The four vision call sites in the tests and the PDF-parsing example use the vision constant; `examples/parse_pdfs.py` used a bare `qwen3.5`, which is now the same value the tests use. `examples/find_chemicals_relationships.py` passed the class default explicitly and no longer passes a model at all. `DEFAULT_VISION_MODEL` also records the decision the issue says was recorded nowhere. A comment explains the trap: the endpoint answers a retired model with a bad-request error that the test suite classifies as an unavailable service and xfails, so pointing at a retired model silences the vision tests instead of failing them. `tests/test_default_models.py` covers exactly that blind spot - it greps `src/aoptk` for retired model names, which the live tests cannot do, and pins that the constructor default is the shared constant rather than a second copy of its value. The issue's first step - confirming the replacement model is still available - could not be verified from a checkout: it needs a `CERIT_API_KEY`. See the PR for the CI evidence bearing on it. Refs #145 Co-Authored-By: Claude Code <noreply@anthropic.com>
Whether a model is still available was only discoverable by calling the endpoint. Issue #145 is the visible cost: `Qwen3.5-122b` had been archived by the provider on 2026-08-26, and because the endpoint reports a retired model as a bad-request error that the test harness classifies as an unavailable service, the four vision tests xfailed rather than failing. A month could pass that way unnoticed. `https://llm.ai.e-infra.cz/status/api/v1/models` answers without a credential (unlike `/v1/models`, which is why this was not noticed sooner), so the offer can be checked from a bare checkout. `models/llm_models.json` pins the 25 models as observed, and `scripts/update_llm_models.sh` rewrites it. Two properties the weekly job depends on: - Observed fields are rewritten, but a curated `note` is carried over. The API reports no capability information, so a refresh that erased what we know about a model would quietly destroy the only useful content. - Exit status distinguishes "no change" (0) from "the offer changed" (1) from "could not tell" (2). Conflating the last two is how a job reports an all-clear on a day the service was unreachable. The comparison ignores throughput and `last_seen`, which the provider advances every minute for every online model; comparing the raw payload would open a pull request weekly over noise. `--summary BEFORE AFTER` describes a difference in the same terms `--check` uses, for the weekly job's pull request body. `tests/test_llm_model_catalog.py` reports what the catalog implies about our own defaults - retired, absent, an unrecognised status, a stale file, models added since the last refresh. Those are warnings, not failures: a provider retiring a model is not a defect in the code under test, and failing every open pull request over it would only train people to ignore the suite. Only the catalog's own well-formedness fails, since a malformed file makes every other check here vacuous. Names are matched case-insensitively; the provider writes `Gpt-oss-120b` and the constant is `gpt-oss-120b`. Co-Authored-By: Claude Code <noreply@anthropic.com>
Asks the status API every Monday whether models/llm_models.json still matches what the endpoint offers, and opens a pull request with the refresh when it does not. `workflow_dispatch` is also allowed, so a maintainer can re-check after the provider announces a change instead of waiting for Monday. The job acts on `--check`'s exit status rather than on a `git diff`, because a write always rewrites the volatile fields and would therefore produce a pull request every week regardless of what the provider did. An exit of 2 fails the job instead of being treated as "nothing new". Two consequences are recorded in the workflow rather than left to be discovered: - The pull request is opened with the default `GITHUB_TOKEN`, which does not trigger `build.yml`, so its checks need a manual re-run once opened. A personal access token would avoid the click at the cost of a credential to create and rotate; while these refreshes are rare the click is cheaper. - No `labels:` input. This repository has no label for automated pull requests, and the action creates a missing label in the repository settings rather than failing, so a typo would silently edit settings. The job cannot decide the two things that matter about a change, so the body asks a human to: whether one of our defaults was retired, and whether a new model serves aoptk better than the current choice. The status API reports no capabilities, so the second is a judgement call. Co-Authored-By: Claude Code <noreply@anthropic.com>
The status API reports `Qwen3.5-122b` as archived, last seen 2026-08-26. It was `DEFAULT_VISION_MODEL`, which settles the symptom in #145 without leaving anything to infer: the endpoint answers a retired model with a bad-request error, the harness classifies that as an unavailable service, and the four vision tests xfail. A green suite had been reporting four silently skipped image and scanned-PDF tests for a month. `qwen3.5-int4` is the closest online model of the same family, listed by the provider since 2026-07-28 and online in 48/48 uptime samples. That is an availability argument only. The status API exposes no capability data, and `CERIT_API_KEY` is not available here, so no vision test has been observed passing against it; the constant's comment says as much rather than implying a verification that did not happen. Changing it again is a one-line edit. `qwen3.5-122b` is added to the retired names that `tests/test_default_models.py` greps `src` for, so reintroducing it fails statically instead of xfailing. That grep now ignores case: the provider writes `Qwen3.5-122b` and the constants do not, and the endpoint accepts either, so a case-sensitive search would miss the spelling most likely to be pasted in. Refs #145 Co-Authored-By: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #145 — deliberately not
Closes, see "Unresolved" below.Validation of the issue against
mainThe issue is stale in its main claim. Checked on
main(d5f61cc):mainmainstill contains the retired model name"git grep llama-4-scout origin/main -- src tests examplesreturns nothing. PR #149 merged 2026-09-24, bringing the swap.tests/test_pymupdf_parser.py:147still usesredhatai-scout"qwen3.5-122b. The onlyredhatai-scoutleft in the repo is a test fixture string intests/test_service_availability.py:27, where it is deliberate sample content for the failure classifier, not a model being requested.examples/find_chemicals_relationships.py:57,68duplicate the class default"examples/parse_pdfs.py:14usesqwen3.5"So of the four remaining items, only 3 (make the model configurable) and 4 (align example model names) plus the recording part of 2 were still actionable. Item 1 (confirm the model is currently available) could not be verified — see below.
Changes
DEFAULT_MODEL = "gpt-oss-120b"andDEFAULT_VISION_MODEL = "qwen3.5-122b"defined once insrc/aoptk/text_generation_api.py, in the style of the existing module-leveltopicsconstant.__init__'s default now refers toDEFAULT_MODELinstead of repeating the literal.tests/test_text_generation.py×3,tests/test_pymupdf_parser.py×1) useDEFAULT_VISION_MODEL.examples/parse_pdfs.py— which does figure→text vision work and passed a bareqwen3.5— now uses the same constant.examples/find_chemicals_relationships.pypassedmodel="gpt-oss-120b", i.e. the class default, in two places; it now passes no model. The issue itself flags this duplication as a problem, and dropping it rather than introducing aDEFAULT_MODELimport keeps this PR clear of Retry LLM answers the transport accepted but we cannot use #154, which rewrites the same loop.DEFAULT_VISION_MODELcarries the rationale the issue says was recorded nowhere.tests/test_default_models.py: grepssrc/aoptkfor retired model names, and pins that the constructor default is the shared constant rather than a second copy of its value.After this,
git grep 'model="'over tracked source, tests and examples returns nothing.Why a static test for retired names
While working on this I found the reason the issue's own verification step ("grep returns nothing") deserves a test rather than a checklist item.
tests/service_availability.pyclassifies"invalid model name"as a service problem, so when the endpoint reports a retired model, the vision tests xfail instead of failing. A regression to a retired model therefore silences the four live vision tests rather than breaking them — the suite structurally cannot catch it.tests/test_default_models.pycovers the part the live tests cannot, and is fast, offline, and non-fragile (it names retired models, not the current one, so a routine model bump does not break it).Unresolved — needs a key to settle, and I did not guess
I could not confirm step 1: whether
qwen3.5-122bis still available.GET /v1/modelsreturns401without a key, and I have noCERIT_API_KEYhere. I kept the value atqwen3.5-122brather than inventing a replacement, and the comment does not claim it is verified.Worth flagging: CI evidence suggests
qwen3.5-122bmay itself now be retired.28241321216(June, onqwen3.5-122b): the three vision tests PASSED.35978921687and35981367222(this month, same model string): all four vision tests XFAIL, while every live test using the default model passes in the same runs.The only variable between passing and xfailing is the vision model name. Given the masking behaviour above, that pattern is consistent with
qwen3.5-122bhaving been retired — but xfail truncates the reason in-voutput and I could not extract it from the logs, so this stays a hypothesis, not a conclusion. This needs someone with a key to runpytest -m openai -rsand read the reason; if it is a retired model, the constant is the one-line fix, which is the point of centralizing it.pytest -m openaiis not run on SonarCloud, only on the build matrix's 3.12/ubuntu job.Verification
ruff check,ruff format --check,mypy— clean.pytest -m "not openai"onmain+ this change: 125 passed, 2 skipped, 25 xfailed on the offline subset; full suite run on the pre-change tree gave 157 passed / 3 skipped / 25 xfailed / 53 xpassed, no regressions.pytest -m openai --collect-onlycollects all 19 live tests.TextGenerationAPI().model == DEFAULT_MODEL.Deliberately not touched
tests/test_service_availability.py:27—redhatai-scoutthere is sample error text for the failure classifier. Renaming it would weaken that test to satisfy a grep. The new test scopes its scan tosrc/aoptkfor this reason.model=, so the constant is the missing half. An env override is a separate design decision.🤖 Generated with Claude Code