fix(llm): --provider openai sends a model litellm does not know as OpenAI's to the OpenAI endpoint

dokimi/changelog.d/20261027_120000_vlad-perevezentsev_openai-compatible-model-route.md

@@ -0,0 +1,30 @@
+### Fixed
+
+- **`--provider openai` reaches an OpenAI-compatible endpoint with a model litellm does not
+  know as OpenAI's** — such a model (`glm-5.3` behind `OPENAI_API_BASE`, an `org/name` id
+  such as `meta-llama/Llama-3.1-8B`, a model newer than the installed litellm) was sent bare,
+  and litellm refused it ("LLM Provider NOT provided") before any request. Such names now go
+  out as `openai/<model>`, on every path that takes the openai provider: answers, the judge
+  and panel, `eval cross-judge --judges openai/<model>` (#136), whose arms read NOT MEASURED
+  before, and question and persona generation. A name litellm takes for OpenAI's
+  (`gpt-6.1-sol`, `o3`, a `ft:gpt-4o-mini:…` fine-tune), or one with a litellm provider prefix
+  (`azure/…`, `ollama/…`), keeps its route. Every name keeps its response-cache entries: on
+  provider openai the key holds the model name as given, not the route, so it does not move
+  with this fix or with litellm's price map (the remote one or the installed backup, which
+  can disagree on a new model). The double-prefix workaround (`openai/openai/<model>`) still
+  works.
+- **`--provider openai` no longer hands a bare name to another cloud with the OpenAI key** —
+  a bare name that litellm assigns to another provider (about 450 in litellm 1.106: `claude-*`
+  to Anthropic, `amazon.nova-pro-v1:0` and the other Bedrock ids, `gemini-*` to Vertex, Cohere
+  and AI21 names) went to that provider with `OPENAI_API_KEY` as its key; Anthropic and
+  Bedrock refuse it, so such a run worked only while `OPENAI_API_KEY` was unset. It now goes
+  to the OpenAI endpoint like any other name litellm does not know as OpenAI's. To reach those
+  clouds, use `--provider anthropic`, `amazon-bedrock` or `google-vertex`.
+
+### Documentation
+
+- **`docs/reviewing.md`: running against an OpenAI-compatible server** — provider `openai` with
+  `OPENAI_API_BASE` / `OPENAI_BASE_URL`, which names reach the server and which keep a
+  litellm provider prefix's route (write `openai/<name>` for those), a placeholder
+  `OPENAI_API_KEY` for a server that ignores it, and why cost is unknown and
+  `novelty --max-usd` stops after one call.

dokimi/docs/reviewing.md

@@ -27,9 +27,21 @@ Credentials are read from the environment: `OPENAI_API_KEY`,
 `ANTHROPIC_API_KEY`, `OPENROUTER_API_KEY`, `AWS_REGION` with
 `AWS_BEARER_TOKEN_BEDROCK` for Bedrock, and Google application-default
 credentials (`GOOGLE_APPLICATION_CREDENTIALS`, with `VERTEXAI_PROJECT` and
-`VERTEXAI_LOCATION`) for Vertex. The judge must be a different model from the
-one under measurement, because a model that grades its own answers inflates
-the scores. `novelty` refuses a self-judge; `arms run` does not check, so name
+`VERTEXAI_LOCATION`) for Vertex. For an OpenAI-compatible server (vLLM, a
+gateway), use provider `openai` with `OPENAI_API_BASE` (or `OPENAI_BASE_URL`)
+set to its `/v1` URL. A model name that litellm does not take for OpenAI's own
+goes to that server, such as `glm-5.3` or `meta-llama/Llama-3.1-8B`
+(`--judges openai/glm-5.3` in `eval cross-judge`); so does a bare `claude-*`,
+Bedrock or Vertex id, which needs its cloud's provider to reach that cloud. A
+name that starts with a litellm provider prefix (`deepseek/`, `groq/`,
+`ollama/`) keeps that provider's route: write it as `openai/<name>` to send it
+to the server. `OPENAI_API_KEY` must be set even when the server ignores it.
+litellm has no price for such a model, so cost is reported as unknown and
+`novelty --max-usd` stops the run after its first call.
+
+The judge must be a different model from the one under measurement, because a
+model that grades its own answers inflates the scores. `novelty` refuses a
+self-judge; `arms run` does not check, so name
 `--judge --judge-model <other> --judge-provider <judge-provider>` yourself.
 
 ## A skill

dokimi/dokimi/llm.py

@@ -39,13 +39,13 @@
 - **Retry.** Transient errors (:func:`is_retryable`) are retried
   ``max_retries`` times with doubling backoff; anything else is raised on the
   first attempt.
-- **Response cache** (:mod:`dokimi.caching`, off unless
-  ``DOKIMI_RESPONSE_CACHE`` is set). ``complete`` is cached, ``chat`` never is:
-  replaying a completion would skip the tool calls an agent loop measures. The
-  key holds the litellm model, the messages, ``stream`` and ``max_tokens``, and
-  ``temperature``/``seed`` when set; ``draw=n`` is an independent sample with its
-  own entry (draw 0 adds nothing). ``cache_if(text)`` decides whether a reply may
-  be stored *or replayed*. A replay returns the stored usage record.
+- **Response cache** (:mod:`dokimi.caching`, off unless ``DOKIMI_RESPONSE_CACHE`` is set).
+  ``complete`` is cached, ``chat`` never is: replaying a completion would skip the tool calls an
+  agent loop measures. The key holds the litellm model (on openai the caller's name, as the route
+  there moves with litellm's price map), the messages, ``stream`` and ``max_tokens``, and
+  ``temperature``/``seed`` when set; ``draw=n`` is an independent sample with its own entry
+  (draw 0 adds nothing). ``cache_if(text)`` decides whether a reply may be stored *or
+  replayed*. A replay returns the stored usage record.
 - **Usage.** Every reply carries a :class:`UsageRecord`; ``cost_usd`` is ``None``
   when litellm has no price for the model, never a fake ``0.0``.
 - **Truncation.** A live ``complete``/``ask`` reply cut at the output ceiling
@@ -59,10 +59,6 @@
 production; :class:`FakeAdapter` answers from a script and records every
 request, for tests (the ``fake_llm`` fixture). :func:`use_adapter` swaps the
 process default; ``LLM(..., adapter=...)`` overrides it for one object.
-
-``llm_call``, ``llm_call_with_usage``, ``chat_completion``, the
-``ChatOpenAI``/``ChatAnthropic`` shims and ``utils.get_llm`` are delegates to
-:class:`LLM` until their callers move.
 """
 
 from __future__ import annotations
@@ -135,17 +131,22 @@ def resolve_api_key(provider: str, api_key: Optional[str] = None) -> Optional[st
 def resolve_model(model: str, provider: str) -> str:
     """The litellm model string for *model* at *provider*.
 
-    A name that already carries a litellm route (contains ``/``) passes through,
-    except on OpenRouter and openai-codex, which add their own prefix.
+    A name with a ``/`` passes through, except on OpenRouter and openai-codex, which
+    add their own prefix, and on openai, where only a litellm provider prefix counts.
     """
     name = _canonical(provider)
     if name == "openrouter":
         return model if model.startswith("openrouter/") else f"openrouter/{model}"
     if name == "openai-codex":
         return f"openai/{model}"
-    if "/" in model:
-        return model
     if name == "openai":
+        # A name litellm does not take for OpenAI's (glm-5.3, meta-llama/x, claude-*) gets openai/.
+        import litellm
+        known = litellm.open_ai_chat_completion_models | litellm.open_ai_text_completion_models
+        ours = model in known or "ft:gpt-3.5-turbo" in model or "ft:gpt-4" in model  # as litellm
+        routed = (p := model.split("/", 1)[0]) in litellm.provider_list or litellm.JSONProviderRegistry.exists(p) if "/" in model else ours
+        return model if routed else f"openai/{model}"
+    if "/" in model:
         return model
     if name == "google-vertex":
         return f"vertex_ai/{model}"
@@ -706,8 +707,8 @@ def complete(
             model=self.model, provider=self.provider,
         )
         key = caching.response_cache_key(
-            litellm_model=self.litellm_model, messages=messages,
-            extra=caching.draw_extra(self._cache_extra(stream, max_tokens), draw),
+            litellm_model=self.model if _canonical(self.provider) == "openai" else self.litellm_model,
+            messages=messages, extra=caching.draw_extra(self._cache_extra(stream, max_tokens), draw),
         )
         hit = caching.load_response(key)
         if hit is not None and caching.keeps(cache_if, cast(str, hit[0])):
@@ -881,7 +882,6 @@ def parses(raw: str) -> bool:
 # Delegates for the callers that have not moved to :class:`LLM` yet; each is the
 # module's behaviour under the old call shape. They go once nothing imports them.
 
-# Backward-compatibility shims for existing tests/modules.
 LANGCHAIN_AVAILABLE = True
 
 _build_litellm_model = resolve_model

dokimi/tests/test_llm.py

@@ -56,7 +56,18 @@ def _no_ambient_bedrock_env(monkeypatch):
 
 
 @pytest.mark.parametrize("provider, model, expected", [
-    ("openai", "gpt-6.1-sol", "gpt-6.1-sol"),
+    ("openai", "gpt-6.1-sol", "gpt-6.1-sol"),  # on litellm's lists: bare, so its cache key holds
+    ("openai", "gpt-3.5-turbo-instruct", "gpt-3.5-turbo-instruct"),  # litellm's text-completion route
+    ("openai", "ft:gpt-4o-mini:org::id", "ft:gpt-4o-mini:org::id"),  # a fine-tune litellm takes as OpenAI's
+    ("openai", "ft:gpt-3.5-turbo:org::id", "ft:gpt-3.5-turbo:org::id"),  # the other fine-tune family litellm takes as OpenAI's
+    ("openai", "glm-5.3", "openai/glm-5.3"),  # off the lists: an OPENAI_API_BASE server's model
+    ("openai", "claude-sonnet-5-5", "openai/claude-sonnet-5-5"),  # bare, litellm would pick anthropic
+    ("openai", "amazon.nova-pro-v1:0", "openai/amazon.nova-pro-v1:0"),  # bare, litellm would pick bedrock
+    ("openai", "meta-llama/Llama-3.1-8B", "openai/meta-llama/Llama-3.1-8B"),  # an org, not a route
+    ("openai", "openai/glm-5.3", "openai/glm-5.3"),
+    ("openai", "azure/gpt-4o", "azure/gpt-4o"),  # an explicit litellm route passes through
+    ("openai", "ollama/llama3", "ollama/llama3"),  # any litellm provider prefix passes through
+    ("openai", "crusoe/x", "crusoe/x"),  # a provider litellm registers in JSON, not provider_list
     ("openai-codex", "gpt-5", "openai/gpt-5"),
     ("anthropic", "claude-sonnet-5-5", "anthropic/claude-sonnet-5-5"),
     ("openrouter", "openai/gpt-oss-120b", "openrouter/openai/gpt-oss-120b"),
@@ -259,12 +270,15 @@ def broken(model):
     ("22510fd74bf01210757177f302f9b6728b81e9e3b31572bd6245f0c88af78dfd",
      lambda: get_llm("openrouter", "openai/gpt-oss-120b", "k", temperature=0.0).invoke(
          "ask", draw=1)),
+    # Off the installed price map, so routed as openai/gpt-6.1-terra here; main sent it bare.
+    ("c6dc142638019230bb9f5876e709302f8eafd63ea9a22a5990bc086fd7d07e3e",
+     lambda: LLM("gpt-6.1-terra", "openai", api_key="k").complete("ask")),
 ]
 
 
 @pytest.mark.parametrize("key, call", MAIN_WRITTEN,
                          ids=["plain", "system+prefix", "pinned+max_tokens+draw", "stream",
-                              "get_llm-bedrock", "get_llm-openrouter"])
+                              "get_llm-bedrock", "get_llm-openrouter", "openai-rerouted"])
 def test_an_entry_main_wrote_is_hit(key, call, cache_dir, fake_llm):
     usage = UsageRecord(total_tokens=7, cost_usd=0.002, cost_known_calls=1)
     (cache_dir / f"{key}.json").write_text(

dokimi/tests/test_openai_compatible_route.py

@@ -0,0 +1,88 @@
+# Copyright (C) 2026 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+"""A model on an OpenAI-compatible endpoint (``--provider openai``, ``OPENAI_API_BASE``) is
+called, not refused by litellm's provider lookup (#136).
+
+The production adapter runs here with litellm's own routing: ``litellm.completion`` gets
+``mock_response``, so it still resolves the provider from the model string, and refuses a name
+it cannot route ("LLM Provider NOT provided"), but sends nothing.
+"""
+
+import json
+
+import pytest
+
+from dokimi.llm import LLM
+
+JUDGE_REPLY = json.dumps({"correctness": 7, "completeness": 7, "specificity": 7, "code_quality": 7,
+                          "actionability": 7, "aggregate": 0})
+
+
+@pytest.fixture
+def litellm_routes(monkeypatch):
+    """``litellm.completion`` with its real provider lookup and a canned reply; the routed names land in the list."""
+    import litellm
+
+    from dokimi import caching
+
+    real, routed = litellm.completion, []
+
+    def completion(**kwargs):
+        reply = real(**kwargs, mock_response=JUDGE_REPLY)  # raises where litellm cannot route the name
+        routed.append(kwargs["model"])
+        return reply
+
+    monkeypatch.setattr(litellm, "completion", completion)
+    monkeypatch.delenv(caching.RESPONSE_CACHE_ENV, raising=False)
+    monkeypatch.setenv("OPENAI_API_KEY", "sk-test")
+    return routed
+
+
+@pytest.mark.parametrize("model", ["glm-5.3", "meta-llama/Llama-3.1-8B", "claude-sonnet-5-5",
+                                   "ft:gpt-5-mini:org::id"])  # a fine-tune litellm does not place
+def test_a_model_off_litellms_openai_lists_is_sent_to_the_openai_endpoint(litellm_routes, model):
+    assert LLM(model, "openai").ask("Reply with OK.") == JUDGE_REPLY
+    assert litellm_routes == [f"openai/{model}"]
+
+
+@pytest.mark.parametrize("model", ["gpt-4o", "gpt-3.5-turbo-instruct", "ft:gpt-4o-mini:org::id"])
+def test_a_name_litellm_takes_for_openais_goes_out_bare_and_still_routes(litellm_routes, model):
+    assert LLM(model, "openai").ask("Reply with OK.") == JUDGE_REPLY
+    assert litellm_routes == [model]
+
+
+def test_a_reply_cached_under_one_litellm_price_map_replays_under_the_other(litellm_routes, monkeypatch, tmp_path):
+    """A new model is on litellm's remote price map but not on its installed backup
+    (``LITELLM_LOCAL_MODEL_COST_MAP``): the route is bare under one and ``openai/`` under the
+    other, and the response-cache key must not follow it."""
+    import litellm
+
+    from dokimi import caching
+
+    monkeypatch.setenv(caching.RESPONSE_CACHE_ENV, str(tmp_path))
+    listed = set(litellm.open_ai_chat_completion_models) - {"gpt-6.1-terra"}
+    monkeypatch.setattr(litellm, "open_ai_chat_completion_models", listed | {"gpt-6.1-terra"})
+    assert LLM("gpt-6.1-terra", "openai").ask("Reply with OK.") == JUDGE_REPLY
+    monkeypatch.setattr(litellm, "open_ai_chat_completion_models", listed)
+    off_map = LLM("gpt-6.1-terra", "openai")
+    assert off_map.litellm_model == "openai/gpt-6.1-terra"
+    assert off_map.ask("Reply with OK.") == JUDGE_REPLY
+    assert litellm_routes == ["gpt-6.1-terra"]  # the second ask replayed, nothing sent
+
+
+def test_cross_judge_scores_with_an_openai_compatible_judge(litellm_routes, tmp_path):
+    """The reported command: before the fix every score failed and the arm read NOT MEASURED."""
+    from dokimi.cli import build_parser
+
+    answers = tmp_path / "run.json"
+    answers.write_text(json.dumps({"library_name": "L", "baseline_arm": "baseline", "answers": [
+        {"question_id": f"Q{i}", "question_text": "How do I use X?",
+         "arms": {"baseline": {"answer": f"b{i}"}, "skill": {"answer": f"s{i}"}}} for i in range(3)]}),
+        encoding="utf-8")
+    out = tmp_path / "cj.json"
+    args = build_parser().parse_args(["eval", "cross-judge", "--answers", str(answers), "--no-stored",
+                                      "--judges", "openai/glm-5.3", "--output", str(out)])
+    args.func(args)
+    assert litellm_routes == ["openai/glm-5.3"] * 6  # three questions, two arms
+    assert json.loads(out.read_text(encoding="utf-8"))["summary"]["arms"]["skill"]["label"] != "NOT MEASURED"