Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions docs/architecture.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -364,6 +364,25 @@ without sending the request to an engine. It does not broaden the candidate list
or refresh inventory synchronously; a later discovery update makes a newly
advertised owner eligible.

The Ollama proxy also supports an opt-in exact-artifact capability gate. A
model-bearing inference request with one valid
`X-MrNiceAI-Expected-Artifact-Sha256` value triggers concurrent, bounded
`/api/tags` reads through each already eligible candidate's normal transport.
Only candidates with one unambiguous model record carrying that exact digest
remain eligible. This request-local intersection preserves selection and
scheduler order, happens before reservations or forwarding, and fails closed
when no exact owner can be proven. The expected value is consumed at the proxy.
Inventory redirects are not followed and process-wide concurrency is capped.
At the response commit point, the proxy checks the candidate's `/api/ps`
scheduler inventory and requires the requested model's loaded digest to equal
the preselected digest. A missing, ambiguous, unavailable, or changed live
record fails over before any response byte is released, or fails closed if the
candidate list is exhausted. Any engine-supplied header or trailer named
`X-MrNiceAI-Served-Artifact-Sha256` is removed and replaced with the immutable
verified digest for the candidate that actually served the response, including
after failover. Reserved request trailers are rejected or stripped. Requests
without the expected header retain ordinary behavior.

A request whose model cannot be parsed keeps the ordinary non-model ordering.

Model listings are not routed at all. A `GET` of `/v1/models` or `/api/tags` is
Expand Down
3 changes: 3 additions & 0 deletions services/ollama-proxy/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,13 @@ The proxy listens on `--port` (default 11435) and forwards incoming HTTP request

A response *forwarded from an engine* is different: if the engine set its own `Access-Control-Allow-Origin` (Ollama with `OLLAMA_ORIGINS`), that header is passed through untouched rather than replaced with the wildcard, so a deliberately narrow engine policy is never silently widened and a credentialed response is not broken. An engine that sends no CORS header has expressed no policy to keep, so the proxy supplies its own — and drops any `Access-Control-Allow-Credentials` that arrived without an origin, because a browser rejects that header alongside a wildcard origin and would discard the response the fallback exists to make readable.

**Exact-artifact requests.** A model-bearing inference request may carry exactly one ordinary `X-MrNiceAI-Expected-Artifact-Sha256` header containing 64 lowercase hexadecimal characters; the same field is rejected as a trailer. The proxy consumes the header, reads `/api/tags` from the already model-eligible candidates through their normal local or mTLS transports, and retains only owners whose single matching model record has that digest. Redirects are not followed. A fixed worker pool plus a process-wide gate bound the inventory work; selection, scheduler priority, reservations, and failover keep their existing order within the retained set. Invalid bindings return `400`, an available but non-matching inventory returns `412`, and complete inventory unavailability returns `503`, all before inference. Before committing a candidate response, the proxy reads Ollama's scheduler-owned `/api/ps` inventory and requires the live loaded-model digest to match the selected digest. A missing, ambiguous, unavailable, or changed loaded record fails over before any response byte is released, or fails closed when candidates are exhausted. The proxy removes every engine-supplied `X-MrNiceAI-Served-Artifact-Sha256` header and trailer and writes exactly one header from the verified candidate inventory. The expected header never reaches an engine, a manual pin cannot bypass the digest gate, and requests without the expected header retain ordinary routing behavior while still having reserved engine evidence stripped. This binding currently applies to the Ollama proxy; other engine proxies must implement the same contract independently before they can serve an exact-artifact request.

One limit is outside the proxy's control: current Chromium-based browsers gate a request from a public origin to a local or loopback address behind the user's [Local Network Access](https://chromestatus.com/feature/5152728072060928) permission, which replaced the old server-side opt-in header. No header the proxy sends can grant that. A hosted page needs the permission plus a `fetch(url, { targetAddressSpace: 'loopback' })` annotation; a page served from the local machine is unaffected.

Node selection:
- **Eligibility**: Before routing model-bearing inference, the proxy keeps only nodes whose current Ollama inventory advertises the requested model. Ollama's implicit `:latest` tag is normalized. An empty or non-matching inventory is excluded until a later discovery update; if no advertised owner is routable, the proxy returns a local `502`.
- **Exact artifact**: When the optional expected-artifact header is present, a bounded request-local `/api/tags` read restricts eligibility to the exact digest and a pre-commit `/api/ps` read verifies the loaded runner. No matching installed and loaded owner means no response is committed.
- **Auto**: When no eligible node is explicitly selected, the proxy follows `node/set-priority` (see below), then discovered nodes in stable ID order.
- **Priority (scheduler-driven)**: The Job Scheduler ranks the cluster least-loaded-first by pending workload plus smoothed GPU pressure and, via `nvpair-ui-broker`, pushes the ordered node list with those per-node counts to this proxy with `node/set-priority`. Auto routing sends the request to the listed node carrying the least estimated load. See [`nvpair-job-scheduler`](../nvpair-job-scheduler/README.md).
- **Manual**: Use the `node/select` JSON-RPC method to pin traffic to a specific node. A manual pin **overrides the priority list only when that node is eligible** for the requested model.
Expand Down
Loading