diff --git a/dokimi/changelog.d/20261027_120000_vlad-perevezentsev_openai-compatible-model-route.md b/dokimi/changelog.d/20261027_120000_vlad-perevezentsev_openai-compatible-model-route.md new file mode 100644 index 000000000..63ebbba20 --- /dev/null +++ b/dokimi/changelog.d/20261027_120000_vlad-perevezentsev_openai-compatible-model-route.md @@ -0,0 +1,30 @@ +### Fixed + +- **`--provider openai` reaches an OpenAI-compatible endpoint with a model litellm does not + know as OpenAI's** — such a model (`glm-5.3` behind `OPENAI_API_BASE`, an `org/name` id + such as `meta-llama/Llama-3.1-8B`, a model newer than the installed litellm) was sent bare, + and litellm refused it ("LLM Provider NOT provided") before any request. Such names now go + out as `openai/`, on every path that takes the openai provider: answers, the judge + and panel, `eval cross-judge --judges openai/` (#136), whose arms read NOT MEASURED + before, and question and persona generation. A name litellm takes for OpenAI's + (`gpt-6.1-sol`, `o3`, a `ft:gpt-4o-mini:…` fine-tune), or one with a litellm provider prefix + (`azure/…`, `ollama/…`), keeps its route. Every name keeps its response-cache entries: on + provider openai the key holds the model name as given, not the route, so it does not move + with this fix or with litellm's price map (the remote one or the installed backup, which + can disagree on a new model). The double-prefix workaround (`openai/openai/`) still + works. +- **`--provider openai` no longer hands a bare name to another cloud with the OpenAI key** — + a bare name that litellm assigns to another provider (about 450 in litellm 1.106: `claude-*` + to Anthropic, `amazon.nova-pro-v1:0` and the other Bedrock ids, `gemini-*` to Vertex, Cohere + and AI21 names) went to that provider with `OPENAI_API_KEY` as its key; Anthropic and + Bedrock refuse it, so such a run worked only while `OPENAI_API_KEY` was unset. It now goes + to the OpenAI endpoint like any other name litellm does not know as OpenAI's. To reach those + clouds, use `--provider anthropic`, `amazon-bedrock` or `google-vertex`. + +### Documentation + +- **`docs/reviewing.md`: running against an OpenAI-compatible server** — provider `openai` with + `OPENAI_API_BASE` / `OPENAI_BASE_URL`, which names reach the server and which keep a + litellm provider prefix's route (write `openai/` for those), a placeholder + `OPENAI_API_KEY` for a server that ignores it, and why cost is unknown and + `novelty --max-usd` stops after one call. diff --git a/dokimi/docs/reviewing.md b/dokimi/docs/reviewing.md index 8a0f8982b..b40065add 100644 --- a/dokimi/docs/reviewing.md +++ b/dokimi/docs/reviewing.md @@ -27,9 +27,21 @@ Credentials are read from the environment: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `OPENROUTER_API_KEY`, `AWS_REGION` with `AWS_BEARER_TOKEN_BEDROCK` for Bedrock, and Google application-default credentials (`GOOGLE_APPLICATION_CREDENTIALS`, with `VERTEXAI_PROJECT` and -`VERTEXAI_LOCATION`) for Vertex. The judge must be a different model from the -one under measurement, because a model that grades its own answers inflates -the scores. `novelty` refuses a self-judge; `arms run` does not check, so name +`VERTEXAI_LOCATION`) for Vertex. For an OpenAI-compatible server (vLLM, a +gateway), use provider `openai` with `OPENAI_API_BASE` (or `OPENAI_BASE_URL`) +set to its `/v1` URL. A model name that litellm does not take for OpenAI's own +goes to that server, such as `glm-5.3` or `meta-llama/Llama-3.1-8B` +(`--judges openai/glm-5.3` in `eval cross-judge`); so does a bare `claude-*`, +Bedrock or Vertex id, which needs its cloud's provider to reach that cloud. A +name that starts with a litellm provider prefix (`deepseek/`, `groq/`, +`ollama/`) keeps that provider's route: write it as `openai/` to send it +to the server. `OPENAI_API_KEY` must be set even when the server ignores it. +litellm has no price for such a model, so cost is reported as unknown and +`novelty --max-usd` stops the run after its first call. + +The judge must be a different model from the one under measurement, because a +model that grades its own answers inflates the scores. `novelty` refuses a +self-judge; `arms run` does not check, so name `--judge --judge-model --judge-provider ` yourself. ## A skill diff --git a/dokimi/dokimi/llm.py b/dokimi/dokimi/llm.py index 0967a2011..b5afaca10 100644 --- a/dokimi/dokimi/llm.py +++ b/dokimi/dokimi/llm.py @@ -39,13 +39,13 @@ - **Retry.** Transient errors (:func:`is_retryable`) are retried ``max_retries`` times with doubling backoff; anything else is raised on the first attempt. -- **Response cache** (:mod:`dokimi.caching`, off unless - ``DOKIMI_RESPONSE_CACHE`` is set). ``complete`` is cached, ``chat`` never is: - replaying a completion would skip the tool calls an agent loop measures. The - key holds the litellm model, the messages, ``stream`` and ``max_tokens``, and - ``temperature``/``seed`` when set; ``draw=n`` is an independent sample with its - own entry (draw 0 adds nothing). ``cache_if(text)`` decides whether a reply may - be stored *or replayed*. A replay returns the stored usage record. +- **Response cache** (:mod:`dokimi.caching`, off unless ``DOKIMI_RESPONSE_CACHE`` is set). + ``complete`` is cached, ``chat`` never is: replaying a completion would skip the tool calls an + agent loop measures. The key holds the litellm model (on openai the caller's name, as the route + there moves with litellm's price map), the messages, ``stream`` and ``max_tokens``, and + ``temperature``/``seed`` when set; ``draw=n`` is an independent sample with its own entry + (draw 0 adds nothing). ``cache_if(text)`` decides whether a reply may be stored *or + replayed*. A replay returns the stored usage record. - **Usage.** Every reply carries a :class:`UsageRecord`; ``cost_usd`` is ``None`` when litellm has no price for the model, never a fake ``0.0``. - **Truncation.** A live ``complete``/``ask`` reply cut at the output ceiling @@ -59,10 +59,6 @@ production; :class:`FakeAdapter` answers from a script and records every request, for tests (the ``fake_llm`` fixture). :func:`use_adapter` swaps the process default; ``LLM(..., adapter=...)`` overrides it for one object. - -``llm_call``, ``llm_call_with_usage``, ``chat_completion``, the -``ChatOpenAI``/``ChatAnthropic`` shims and ``utils.get_llm`` are delegates to -:class:`LLM` until their callers move. """ from __future__ import annotations @@ -135,17 +131,22 @@ def resolve_api_key(provider: str, api_key: Optional[str] = None) -> Optional[st def resolve_model(model: str, provider: str) -> str: """The litellm model string for *model* at *provider*. - A name that already carries a litellm route (contains ``/``) passes through, - except on OpenRouter and openai-codex, which add their own prefix. + A name with a ``/`` passes through, except on OpenRouter and openai-codex, which + add their own prefix, and on openai, where only a litellm provider prefix counts. """ name = _canonical(provider) if name == "openrouter": return model if model.startswith("openrouter/") else f"openrouter/{model}" if name == "openai-codex": return f"openai/{model}" - if "/" in model: - return model if name == "openai": + # A name litellm does not take for OpenAI's (glm-5.3, meta-llama/x, claude-*) gets openai/. + import litellm + known = litellm.open_ai_chat_completion_models | litellm.open_ai_text_completion_models + ours = model in known or "ft:gpt-3.5-turbo" in model or "ft:gpt-4" in model # as litellm + routed = (p := model.split("/", 1)[0]) in litellm.provider_list or litellm.JSONProviderRegistry.exists(p) if "/" in model else ours + return model if routed else f"openai/{model}" + if "/" in model: return model if name == "google-vertex": return f"vertex_ai/{model}" @@ -706,8 +707,8 @@ def complete( model=self.model, provider=self.provider, ) key = caching.response_cache_key( - litellm_model=self.litellm_model, messages=messages, - extra=caching.draw_extra(self._cache_extra(stream, max_tokens), draw), + litellm_model=self.model if _canonical(self.provider) == "openai" else self.litellm_model, + messages=messages, extra=caching.draw_extra(self._cache_extra(stream, max_tokens), draw), ) hit = caching.load_response(key) if hit is not None and caching.keeps(cache_if, cast(str, hit[0])): @@ -881,7 +882,6 @@ def parses(raw: str) -> bool: # Delegates for the callers that have not moved to :class:`LLM` yet; each is the # module's behaviour under the old call shape. They go once nothing imports them. -# Backward-compatibility shims for existing tests/modules. LANGCHAIN_AVAILABLE = True _build_litellm_model = resolve_model diff --git a/dokimi/tests/test_llm.py b/dokimi/tests/test_llm.py index 2710b450a..675b3a3f4 100644 --- a/dokimi/tests/test_llm.py +++ b/dokimi/tests/test_llm.py @@ -56,7 +56,18 @@ def _no_ambient_bedrock_env(monkeypatch): @pytest.mark.parametrize("provider, model, expected", [ - ("openai", "gpt-6.1-sol", "gpt-6.1-sol"), + ("openai", "gpt-6.1-sol", "gpt-6.1-sol"), # on litellm's lists: bare, so its cache key holds + ("openai", "gpt-3.5-turbo-instruct", "gpt-3.5-turbo-instruct"), # litellm's text-completion route + ("openai", "ft:gpt-4o-mini:org::id", "ft:gpt-4o-mini:org::id"), # a fine-tune litellm takes as OpenAI's + ("openai", "ft:gpt-3.5-turbo:org::id", "ft:gpt-3.5-turbo:org::id"), # the other fine-tune family litellm takes as OpenAI's + ("openai", "glm-5.3", "openai/glm-5.3"), # off the lists: an OPENAI_API_BASE server's model + ("openai", "claude-sonnet-5-5", "openai/claude-sonnet-5-5"), # bare, litellm would pick anthropic + ("openai", "amazon.nova-pro-v1:0", "openai/amazon.nova-pro-v1:0"), # bare, litellm would pick bedrock + ("openai", "meta-llama/Llama-3.1-8B", "openai/meta-llama/Llama-3.1-8B"), # an org, not a route + ("openai", "openai/glm-5.3", "openai/glm-5.3"), + ("openai", "azure/gpt-4o", "azure/gpt-4o"), # an explicit litellm route passes through + ("openai", "ollama/llama3", "ollama/llama3"), # any litellm provider prefix passes through + ("openai", "crusoe/x", "crusoe/x"), # a provider litellm registers in JSON, not provider_list ("openai-codex", "gpt-5", "openai/gpt-5"), ("anthropic", "claude-sonnet-5-5", "anthropic/claude-sonnet-5-5"), ("openrouter", "openai/gpt-oss-120b", "openrouter/openai/gpt-oss-120b"), @@ -259,12 +270,15 @@ def broken(model): ("22510fd74bf01210757177f302f9b6728b81e9e3b31572bd6245f0c88af78dfd", lambda: get_llm("openrouter", "openai/gpt-oss-120b", "k", temperature=0.0).invoke( "ask", draw=1)), + # Off the installed price map, so routed as openai/gpt-6.1-terra here; main sent it bare. + ("c6dc142638019230bb9f5876e709302f8eafd63ea9a22a5990bc086fd7d07e3e", + lambda: LLM("gpt-6.1-terra", "openai", api_key="k").complete("ask")), ] @pytest.mark.parametrize("key, call", MAIN_WRITTEN, ids=["plain", "system+prefix", "pinned+max_tokens+draw", "stream", - "get_llm-bedrock", "get_llm-openrouter"]) + "get_llm-bedrock", "get_llm-openrouter", "openai-rerouted"]) def test_an_entry_main_wrote_is_hit(key, call, cache_dir, fake_llm): usage = UsageRecord(total_tokens=7, cost_usd=0.002, cost_known_calls=1) (cache_dir / f"{key}.json").write_text( diff --git a/dokimi/tests/test_openai_compatible_route.py b/dokimi/tests/test_openai_compatible_route.py new file mode 100644 index 000000000..a0d283e33 --- /dev/null +++ b/dokimi/tests/test_openai_compatible_route.py @@ -0,0 +1,88 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""A model on an OpenAI-compatible endpoint (``--provider openai``, ``OPENAI_API_BASE``) is +called, not refused by litellm's provider lookup (#136). + +The production adapter runs here with litellm's own routing: ``litellm.completion`` gets +``mock_response``, so it still resolves the provider from the model string, and refuses a name +it cannot route ("LLM Provider NOT provided"), but sends nothing. +""" + +import json + +import pytest + +from dokimi.llm import LLM + +JUDGE_REPLY = json.dumps({"correctness": 7, "completeness": 7, "specificity": 7, "code_quality": 7, + "actionability": 7, "aggregate": 0}) + + +@pytest.fixture +def litellm_routes(monkeypatch): + """``litellm.completion`` with its real provider lookup and a canned reply; the routed names land in the list.""" + import litellm + + from dokimi import caching + + real, routed = litellm.completion, [] + + def completion(**kwargs): + reply = real(**kwargs, mock_response=JUDGE_REPLY) # raises where litellm cannot route the name + routed.append(kwargs["model"]) + return reply + + monkeypatch.setattr(litellm, "completion", completion) + monkeypatch.delenv(caching.RESPONSE_CACHE_ENV, raising=False) + monkeypatch.setenv("OPENAI_API_KEY", "sk-test") + return routed + + +@pytest.mark.parametrize("model", ["glm-5.3", "meta-llama/Llama-3.1-8B", "claude-sonnet-5-5", + "ft:gpt-5-mini:org::id"]) # a fine-tune litellm does not place +def test_a_model_off_litellms_openai_lists_is_sent_to_the_openai_endpoint(litellm_routes, model): + assert LLM(model, "openai").ask("Reply with OK.") == JUDGE_REPLY + assert litellm_routes == [f"openai/{model}"] + + +@pytest.mark.parametrize("model", ["gpt-4o", "gpt-3.5-turbo-instruct", "ft:gpt-4o-mini:org::id"]) +def test_a_name_litellm_takes_for_openais_goes_out_bare_and_still_routes(litellm_routes, model): + assert LLM(model, "openai").ask("Reply with OK.") == JUDGE_REPLY + assert litellm_routes == [model] + + +def test_a_reply_cached_under_one_litellm_price_map_replays_under_the_other(litellm_routes, monkeypatch, tmp_path): + """A new model is on litellm's remote price map but not on its installed backup + (``LITELLM_LOCAL_MODEL_COST_MAP``): the route is bare under one and ``openai/`` under the + other, and the response-cache key must not follow it.""" + import litellm + + from dokimi import caching + + monkeypatch.setenv(caching.RESPONSE_CACHE_ENV, str(tmp_path)) + listed = set(litellm.open_ai_chat_completion_models) - {"gpt-6.1-terra"} + monkeypatch.setattr(litellm, "open_ai_chat_completion_models", listed | {"gpt-6.1-terra"}) + assert LLM("gpt-6.1-terra", "openai").ask("Reply with OK.") == JUDGE_REPLY + monkeypatch.setattr(litellm, "open_ai_chat_completion_models", listed) + off_map = LLM("gpt-6.1-terra", "openai") + assert off_map.litellm_model == "openai/gpt-6.1-terra" + assert off_map.ask("Reply with OK.") == JUDGE_REPLY + assert litellm_routes == ["gpt-6.1-terra"] # the second ask replayed, nothing sent + + +def test_cross_judge_scores_with_an_openai_compatible_judge(litellm_routes, tmp_path): + """The reported command: before the fix every score failed and the arm read NOT MEASURED.""" + from dokimi.cli import build_parser + + answers = tmp_path / "run.json" + answers.write_text(json.dumps({"library_name": "L", "baseline_arm": "baseline", "answers": [ + {"question_id": f"Q{i}", "question_text": "How do I use X?", + "arms": {"baseline": {"answer": f"b{i}"}, "skill": {"answer": f"s{i}"}}} for i in range(3)]}), + encoding="utf-8") + out = tmp_path / "cj.json" + args = build_parser().parse_args(["eval", "cross-judge", "--answers", str(answers), "--no-stored", + "--judges", "openai/glm-5.3", "--output", str(out)]) + args.func(args) + assert litellm_routes == ["openai/glm-5.3"] * 6 # three questions, two arms + assert json.loads(out.read_text(encoding="utf-8"))["summary"]["arms"]["skill"]["label"] != "NOT MEASURED"