diff --git a/README.md b/README.md index 54f7318..b83ebad 100644 --- a/README.md +++ b/README.md @@ -198,9 +198,11 @@ Notes baked into the setup (hard-won): ## omp (oh-my-pi) `omp-setup.sh` installs [omp](https://omp.sh) if it isn't already present, points -it at the Brev-hosted `webster` model proxy, defaults it to **GLM 5.2**, and turns -on **nerd mode** — Nerd Font symbols plus the `nerd` status line preset -(tok/sec spark, TTFT, context %, cost, cache reads, elapsed time). +it at the Brev-hosted `webster` model proxy, discovers the models accessible to +the supplied key, and turns on **nerd mode** — Nerd Font symbols plus the `nerd` +status line preset (tok/sec spark, TTFT, context %, cost, cache reads, elapsed +time). OMP's model picker refreshes from the proxy, so newly granted models show +up without editing this repository or installer. One line on a new machine — note the key goes on the **right** of the pipe, so the `bash` running the script sees it: @@ -221,6 +223,12 @@ From a local clone: OMP_WEBSTER_API_KEY=sk-... ~/.setup/omp-setup.sh ``` +The installer preserves the current `webster/...` default when that model is +still accessible. On a fresh install it prefers `glm-5.2` when available, then +falls back to the first accessible model. Set `OMP_MODEL=` to +require a particular accessible default (reasoning suffixes such as `:max` are +preserved); the install fails clearly if the key cannot use its base model. + Or without the env var — run from a terminal and it prompts for the key (hidden input). Piping to `bash` consumes stdin, so a piped run cannot prompt and exits non-zero instead of writing an empty key: @@ -240,7 +248,7 @@ What it writes: | Where | What | |---|---| | `~/.omp/agent/models.yml` | `webster` provider: base URL, API key, LiteLLM discovery (mode `0600`) | -| `~/.omp/agent/config.yml` | `symbolPreset: nerd`, `statusLine.preset: nerd`, `modelRoles.default: webster/glm-5.2` | +| `~/.omp/agent/config.yml` | `symbolPreset: nerd`, `statusLine.preset: nerd`, and the access-aware `modelRoles.default` | It is idempotent and merge-safe: re-running reuses the key already on disk, preserves other providers and other model roles (`advisor`, `smol`, `tiny`, …), @@ -251,11 +259,11 @@ and backs up `models.yml` to `models.yml.bak` before any change. Verification runs on every install (`assert_omp_config` / `assert_omp_endpoint` / `assert_omp_smoke`, per [STYLE_GUIDE.md](STYLE_GUIDE.md)): it reads the settings -back out of `omp`, lists models from the endpoint to confirm the URL and key are -good and that `glm-5.2` is served, then sends a real one-shot prompt through the -*configured default* model and asserts the reply. Failure paths — dead endpoint, -endpoint missing the model, no API key, stale duplicate provider entry — are -covered in `test.sh`. +back out of `omp`, authenticates to the endpoint and confirms the configured +default is in that key's non-empty live model catalog, then sends a real one-shot +prompt through the *configured default* model and asserts the reply. Failure +paths — dead endpoint, empty or invalid catalog, inaccessible explicit/default +model, no API key, stale duplicate provider entry — are covered in `test.sh`. It is not wired into `install.sh`, because it needs a secret the bootstrap does not have. Run it separately after the bootstrap. diff --git a/omp-setup.sh b/omp-setup.sh index 72741f4..6ea9a27 100755 --- a/omp-setup.sh +++ b/omp-setup.sh @@ -5,7 +5,8 @@ # # * installs omp if it is not already on PATH # * registers the webster provider (OpenAI-compatible LiteLLM proxy) -# * defaults the agent to glm-5.2 +# * discovers every model accessible to the supplied key +# * preserves an accessible Webster default, otherwise prefers glm-5.2 # * turns on "nerd mode": Nerd Font symbols + the nerd status line preset # (tok/sec spark, TTFT, context %, cost, cache reads, elapsed time) # @@ -15,6 +16,9 @@ # ./omp-setup.sh --check # verify an existing install, change nothing # ./omp-setup.sh --no-smoke # skip the live model round-trip # +# Env overrides: OMP_WEBSTER_API_KEY / WEBSTER_API_KEY, OMP_BASE_URL, +# OMP_MODEL, OMP_SKIP_PROFILE +# # Every run asserts the resulting on-disk state and the live endpoint, per # STYLE_GUIDE.md. Negative paths are covered in test.sh, which sources this # file with SETUP_SKIP_MAIN=1. @@ -26,8 +30,14 @@ set -euo pipefail BASE_URL="${OMP_BASE_URL:-https://webster-models-extnode-3gdrajbr0hiykknxzitck9yaiwo.apps.run.brev.nvidia.com/v1}" PROVIDER_ID="webster" -MODEL_ID="glm-5.2" -DEFAULT_MODEL="${PROVIDER_ID}/${MODEL_ID}" +PREFERRED_MODEL_ID="glm-5.2" +REQUESTED_MODEL="${OMP_MODEL:-}" +MODEL_ID="" +MODEL_EXPLICIT=0 +[ -n "$REQUESTED_MODEL" ] && MODEL_EXPLICIT=1 +DEFAULT_MODEL="" +DISCOVERED_MODEL_IDS="" +DISCOVERY_STATUS="" NPM_PKG="@oh-my-pi/pi-coding-agent" MARK_BEGIN=" # >>> theFong/setup: ${PROVIDER_ID} provider (managed) >>>" @@ -129,6 +139,164 @@ resolve_key() { [ -n "$API_KEY" ] || die "empty API key" } +# ------------------------------------------------------------- model discovery + +# normalize_model_ids RESPONSE_FILE — print one sorted model id per line from +# either the OpenAI `.data` shape or LiteLLM's `.models` shape. jq is preferred, +# with python3 as a portability fallback for standalone one-line installs. +normalize_model_ids() { + local response_file="$1" + if have jq; then + jq -r ' + (.data // .models // []) as $models + | if ($models | type) != "array" then error("model list is not an array") else $models end + | [ + .[] + | if type == "string" then . + elif type == "object" then (.id // .slug // .model // empty) + else empty + end + | select(type == "string" and length > 0) + ] + | unique + | sort + | .[] + ' "$response_file" + return + fi + if have python3; then + python3 - "$response_file" <<'PY' +import json +import sys + +with open(sys.argv[1], encoding="utf-8") as handle: + payload = json.load(handle) +models = payload.get("data", payload.get("models", [])) +if not isinstance(models, list): + raise SystemExit("model list is not an array") +ids = set() +for item in models: + if isinstance(item, str): + model_id = item + elif isinstance(item, dict): + model_id = item.get("id") or item.get("slug") or item.get("model") + else: + continue + if isinstance(model_id, str) and model_id: + ids.add(model_id) +print("\n".join(sorted(ids))) +PY + return + fi + warn "need jq or python3 to parse the endpoint's model catalog" + return 1 +} + +# fetch_omp_models URL KEY — authenticate to /models and update the discovered +# catalog. The key is supplied to curl over stdin rather than exposed in argv. +fetch_omp_models() { + local url="$1" key="$2" response_file http_status normalized + have curl || { DISCOVERY_STATUS="no-curl"; return 1; } + if ! have jq && ! have python3; then + DISCOVERY_STATUS="no-parser" + return 1 + fi + response_file="$(mktemp)" + DISCOVERY_STATUS="" + if ! http_status=$(printf 'Authorization: Bearer %s\n' "$key" \ + | curl -sS -o "$response_file" -w '%{http_code}' --max-time 30 \ + --header @- "$url/models" 2>/dev/null); then + DISCOVERY_STATUS="unreachable" + rm -f "$response_file" + return 1 + fi + + case "$http_status" in + 2*) + if normalized="$(normalize_model_ids "$response_file" 2>/dev/null)"; then + if [ -n "$normalized" ]; then + DISCOVERED_MODEL_IDS="$normalized" + DISCOVERY_STATUS="ok" + rm -f "$response_file" + return 0 + fi + DISCOVERY_STATUS="empty" + else + DISCOVERY_STATUS="invalid" + fi + ;; + 401|403) DISCOVERY_STATUS="auth-$http_status" ;; + *) DISCOVERY_STATUS="http-$http_status" ;; + esac + rm -f "$response_file" + return 1 +} + +model_is_discovered() { + [ -n "$1" ] && printf '%s\n' "$DISCOVERED_MODEL_IDS" | grep -Fqx -- "$1" +} + +discover_models() { + local count + if fetch_omp_models "$BASE_URL" "$API_KEY"; then + count=$(printf '%s\n' "$DISCOVERED_MODEL_IDS" | awk 'NF { n++ } END { print n + 0 }') + log "discovered $count Webster model(s) accessible to this key" + return 0 + fi + case "$DISCOVERY_STATUS" in + auth-*) warn "endpoint rejected the configured key (HTTP ${DISCOVERY_STATUS#auth-})" ;; + empty) warn "endpoint accepted the key but advertised no accessible models" ;; + invalid) warn "endpoint returned an invalid model catalog" ;; + no-curl) warn "curl not found; cannot discover accessible models" ;; + no-parser) warn "need jq or python3 to parse the endpoint's model catalog" ;; + http-*) warn "endpoint returned HTTP ${DISCOVERY_STATUS#http-} while listing models" ;; + *) warn "endpoint unreachable while listing models" ;; + esac + return 1 +} + +select_default_model() { + local installed_default installed_selector installed_id + if [ "$MODEL_EXPLICIT" = 1 ]; then + if model_is_discovered "$REQUESTED_MODEL"; then + MODEL_ID="$REQUESTED_MODEL" + else + MODEL_ID="${REQUESTED_MODEL%:*}" + if [ "$MODEL_ID" = "$REQUESTED_MODEL" ] || ! model_is_discovered "$MODEL_ID"; then + warn "OMP_MODEL=$REQUESTED_MODEL is not accessible to this key" + return 1 + fi + fi + DEFAULT_MODEL="${PROVIDER_ID}/${REQUESTED_MODEL}" + else + installed_default="$(role_default)" + case "$installed_default" in + "$PROVIDER_ID"/*) installed_selector="${installed_default#"$PROVIDER_ID"/}" ;; + *) installed_selector="" ;; + esac + if [ -n "$installed_selector" ] && model_is_discovered "$installed_selector"; then + MODEL_ID="$installed_selector" + DEFAULT_MODEL="$installed_default" + else + installed_id="${installed_selector%:*}" + fi + if [ -z "$MODEL_ID" ]; then + if [ -n "$installed_id" ] && [ "$installed_id" != "$installed_selector" ] \ + && model_is_discovered "$installed_id"; then + MODEL_ID="$installed_id" + DEFAULT_MODEL="$installed_default" + elif model_is_discovered "$PREFERRED_MODEL_ID"; then + MODEL_ID="$PREFERRED_MODEL_ID" + else + MODEL_ID="$(printf '%s\n' "$DISCOVERED_MODEL_IDS" | sed -n '1p')" + fi + fi + fi + [ -n "$MODEL_ID" ] || { warn "could not select a default Webster model"; return 1; } + [ -n "$DEFAULT_MODEL" ] || DEFAULT_MODEL="${PROVIDER_ID}/${MODEL_ID}" + log "selected default model $DEFAULT_MODEL" +} + # ------------------------------------------------------------------ models.yml # merge_models_yml FILE KEY — splice the managed provider block into FILE, @@ -141,8 +309,8 @@ merge_models_yml() { cat >"$block" <<-YAML $MARK_BEGIN - # Brev-hosted LiteLLM proxy (OpenAI-compatible). Models are discovered from - # the proxy and selectable as ${PROVIDER_ID}/, e.g. ${DEFAULT_MODEL}. + # Brev-hosted LiteLLM proxy (OpenAI-compatible). OMP discovers every model + # accessible to this key and exposes it as ${PROVIDER_ID}/. ${PROVIDER_ID}: baseUrl: ${BASE_URL} apiKey: ${key} @@ -150,9 +318,6 @@ merge_models_yml() { authHeader: true discovery: type: litellm - modelOverrides: - ${MODEL_ID}: - contextWindow: 350000 $MARK_END YAML @@ -263,22 +428,27 @@ assert_omp_config() { grep -q "^ baseUrl: $BASE_URL$" "$AGENT_DIR/models.yml" 2>/dev/null \ || { warn "baseUrl is not set to $BASE_URL in models.yml"; return 1; } ok "baseUrl = $BASE_URL" + grep -A8 "^ ${PROVIDER_ID}:$" "$AGENT_DIR/models.yml" 2>/dev/null \ + | grep -q '^ type: litellm$' \ + || { warn "LiteLLM discovery is not enabled for $PROVIDER_ID in models.yml"; return 1; } + ok "LiteLLM model discovery is enabled" } # assert_omp_endpoint URL KEY — the base URL and key must actually serve the # model we just made the default. A config pointing at a dead or wrong endpoint # looks fine on disk and fails on first use. assert_omp_endpoint() { - local url="$1" key="$2" out - have curl || { warn "curl not found; cannot verify the endpoint"; return 1; } - if ! out="$(curl -fsS --max-time 30 -H "Authorization: Bearer $key" "$url/models" 2>&1)"; then - warn "endpoint unreachable or key rejected: $out" + local url="$1" key="$2" count + if ! fetch_omp_models "$url" "$key"; then + warn "could not refresh endpoint model catalog ($DISCOVERY_STATUS)" return 1 fi - case "$out" in - *"\"$MODEL_ID\""*) ok "endpoint reachable, $MODEL_ID is served"; return 0 ;; - *) warn "endpoint reachable but does not serve '$MODEL_ID'"; return 1 ;; - esac + model_is_discovered "$MODEL_ID" || { + warn "endpoint is reachable but no longer serves the configured default '$MODEL_ID'" + return 1 + } + count=$(printf '%s\n' "$DISCOVERED_MODEL_IDS" | awk 'NF { n++ } END { print n + 0 }') + ok "endpoint serves $count accessible model(s), including $MODEL_ID" } # assert_omp_smoke — one real round-trip. No --model flag: this exercises the @@ -309,14 +479,15 @@ usage() { fi cat <&2 exit 1 fi +if ! grep -A8 '^ webster:$' "$scratch/omp/models.yml" | grep -q '^ type: litellm$'; then + echo "FAIL: merge_models_yml did not enable LiteLLM model discovery" >&2 + exit 1 +fi +if grep -A10 '^ webster:$' "$scratch/omp/models.yml" | grep -q 'modelOverrides:'; then + echo "FAIL: merge_models_yml still hardcodes a Webster model override" >&2 + exit 1 +fi # GNU stat first, then BSD/macOS. The order matters and both must be quieted: # GNU reads -f as --file-system and prints a block of filesystem info for the # file while failing on the format string, so trying BSD first captures that @@ -579,43 +587,173 @@ fi if ( export SETUP_SKIP_MAIN=1 source ./omp-setup.sh + MODEL_ID=glm-5.2 assert_omp_endpoint "http://127.0.0.1:1/v1" sk-irrelevant ) >/dev/null 2>&1; then echo "FAIL: assert_omp_endpoint accepted an unreachable endpoint" >&2 exit 1 fi -# ...and must also fail on an endpoint that answers but does not serve the -# model being made the default — otherwise reachability alone would pass. +# Model discovery must support a key that cannot access the preferred model, +# choose a deterministic fallback, and reject explicit inaccessible defaults. if command -v python3 >/dev/null 2>&1; then - omp_port=39997 mkdir -p "$scratch/ompsrv/v1" - printf '{"data":[{"id":"some-other-model"}]}' > "$scratch/ompsrv/v1/models" - python3 -m http.server "$omp_port" --bind 127.0.0.1 --directory "$scratch/ompsrv" >/dev/null 2>&1 & + printf '{"data":[{"id":"zeta-model"},{"id":"some-other-model"}]}' > "$scratch/ompsrv/v1/models" + omp_port_file="$scratch/omp-port" + python3 - "$scratch/ompsrv" "$omp_port_file" >"$scratch/omp-server.log" 2>&1 <<'PYEOF' & +import functools +import http.server +import socketserver +import sys + +handler = functools.partial(http.server.SimpleHTTPRequestHandler, directory=sys.argv[1]) +with socketserver.TCPServer(("127.0.0.1", 0), handler) as server: + with open(sys.argv[2], "w") as handle: + handle.write(str(server.server_address[1])) + server.serve_forever() +PYEOF omp_srv=$! + for _ in $(seq 1 50); do + [ -s "$omp_port_file" ] && break + sleep 0.1 + done + if [ ! -s "$omp_port_file" ]; then + kill "$omp_srv" 2>/dev/null || true + echo "FAIL: OMP loopback model server did not start: $(cat "$scratch/omp-server.log")" >&2 + exit 1 + fi + omp_port=$(cat "$omp_port_file") for _ in 1 2 3 4 5 6 7 8 9 10; do curl -s -o /dev/null -m 1 "http://127.0.0.1:$omp_port/" && break sleep 0.3 done + + omp_dynamic_default=$( + export SETUP_SKIP_MAIN=1 OMP_BASE_URL="http://127.0.0.1:$omp_port/v1" + source ./omp-setup.sh + API_KEY=sk-irrelevant + role_default() { :; } + discover_models + select_default_model + printf '%s\n' "$DEFAULT_MODEL" + ) + omp_dynamic_default=$(printf '%s\n' "$omp_dynamic_default" | tail -n 1) + if [ "$omp_dynamic_default" != "webster/some-other-model" ]; then + kill "$omp_srv" 2>/dev/null || true + echo "FAIL: OMP discovery did not select the first accessible model (got '$omp_dynamic_default')" >&2 + exit 1 + fi + + # Endpoint verification still checks that the selected default remains in + # the key's current catalog, rather than accepting reachability alone. if ( export SETUP_SKIP_MAIN=1 source ./omp-setup.sh + MODEL_ID=glm-5.2 assert_omp_endpoint "http://127.0.0.1:$omp_port/v1" sk-irrelevant ) >/dev/null 2>&1; then kill "$omp_srv" 2>/dev/null || true echo "FAIL: assert_omp_endpoint accepted an endpoint not serving the default model" >&2 exit 1 fi - # ...and must pass when the model IS served, so the two checks above cannot be - # satisfied by an assertion that simply always fails. - printf '{"data":[{"id":"glm-5.2"}]}' > "$scratch/ompsrv/v1/models" + # ...and pass for the access-aware fallback. if ! ( export SETUP_SKIP_MAIN=1 source ./omp-setup.sh + MODEL_ID=some-other-model assert_omp_endpoint "http://127.0.0.1:$omp_port/v1" sk-irrelevant ) >/dev/null 2>&1; then kill "$omp_srv" 2>/dev/null || true - echo "FAIL: assert_omp_endpoint rejected an endpoint serving the default model" >&2 + echo "FAIL: assert_omp_endpoint rejected an accessible dynamic default" >&2 + exit 1 + fi + + if ( + export SETUP_SKIP_MAIN=1 OMP_BASE_URL="http://127.0.0.1:$omp_port/v1" OMP_MODEL=missing-model + source ./omp-setup.sh + API_KEY=sk-irrelevant + role_default() { :; } + discover_models + select_default_model + ) >/dev/null 2>&1; then + kill "$omp_srv" 2>/dev/null || true + echo "FAIL: OMP accepted an explicit model inaccessible to the key" >&2 + exit 1 + fi + + # An accessible installed Webster default wins over both the preference and + # alphabetical fallback, so a re-run does not unexpectedly switch models. + omp_preserved_default=$( + export SETUP_SKIP_MAIN=1 OMP_BASE_URL="http://127.0.0.1:$omp_port/v1" + source ./omp-setup.sh + API_KEY=sk-irrelevant + role_default() { printf 'webster/zeta-model:max\n'; } + discover_models + select_default_model + printf '%s\n' "$DEFAULT_MODEL" + ) + omp_preserved_default=$(printf '%s\n' "$omp_preserved_default" | tail -n 1) + if [ "$omp_preserved_default" != "webster/zeta-model:max" ]; then + kill "$omp_srv" 2>/dev/null || true + echo "FAIL: OMP did not preserve an accessible installed default and reasoning preset" >&2 + exit 1 + fi + + omp_explicit_default=$( + export SETUP_SKIP_MAIN=1 OMP_BASE_URL="http://127.0.0.1:$omp_port/v1" OMP_MODEL=some-other-model:high + source ./omp-setup.sh + API_KEY=sk-irrelevant + role_default() { :; } + discover_models + select_default_model + printf '%s\n' "$DEFAULT_MODEL" + ) + omp_explicit_default=$(printf '%s\n' "$omp_explicit_default" | tail -n 1) + if [ "$omp_explicit_default" != "webster/some-other-model:high" ]; then + kill "$omp_srv" 2>/dev/null || true + echo "FAIL: OMP did not preserve an accessible explicit reasoning preset" >&2 + exit 1 + fi + + printf '{"models":["zeta-model",{"id":"glm-5.2"}]}' > "$scratch/ompsrv/v1/models" + omp_preferred_default=$( + export SETUP_SKIP_MAIN=1 OMP_BASE_URL="http://127.0.0.1:$omp_port/v1" + source ./omp-setup.sh + API_KEY=sk-irrelevant + role_default() { :; } + discover_models + select_default_model + printf '%s\n' "$DEFAULT_MODEL" + ) + omp_preferred_default=$(printf '%s\n' "$omp_preferred_default" | tail -n 1) + if [ "$omp_preferred_default" != "webster/glm-5.2" ]; then + kill "$omp_srv" 2>/dev/null || true + echo "FAIL: OMP did not prefer glm-5.2 when accessible (got '$omp_preferred_default')" >&2 + exit 1 + fi + + # A valid but empty catalog means the key has no usable access and must fail. + printf '{"data":[]}' > "$scratch/ompsrv/v1/models" + if ( + export SETUP_SKIP_MAIN=1 OMP_BASE_URL="http://127.0.0.1:$omp_port/v1" + source ./omp-setup.sh + API_KEY=sk-irrelevant + discover_models + ) >/dev/null 2>&1; then + kill "$omp_srv" 2>/dev/null || true + echo "FAIL: OMP accepted an empty model catalog" >&2 + exit 1 + fi + + printf '{not-json' > "$scratch/ompsrv/v1/models" + if ( + export SETUP_SKIP_MAIN=1 OMP_BASE_URL="http://127.0.0.1:$omp_port/v1" + source ./omp-setup.sh + API_KEY=sk-irrelevant + discover_models + ) >/dev/null 2>&1; then + kill "$omp_srv" 2>/dev/null || true + echo "FAIL: OMP accepted an invalid model catalog" >&2 exit 1 fi kill "$omp_srv" 2>/dev/null || true