The inference layer for Intel® AI for Enterprise Solutions. Serve any Hugging Face model on Intel® Xeon® CPUs with one command.
Provides the Ansible roles that stand up the model-serving stack — KServe, runtimes, AI gateway, LiteLLM, Langfuse — and
model-manager, a CLI that takes a model from Hugging Face to a live OpenAI-compatible endpoint with NUMA-aware CPU pinning.
Important
This repository is not used standalone. It is a component of the
ai-solutions
platform and is cloned into it at ext/enterprise.ai-inference/. Install the core platform
first — it provisions the Kubernetes cluster and the platform services (cert-manager, Istio,
MetalLB, Envoy Gateway, PostgreSQL, Keycloak, MinIO, observability) that this layer depends on.
Every command below runs from the solutions repo root, not from here.
Serving a model on Kubernetes normally means hand-writing manifests, sizing CPU and memory, pinning cores to the right NUMA node, downloading weights into shared storage, and wiring a route through a gateway — for every model.
This repository replaces that with a catalog and a CLI. You describe a model once in models.yaml, or point --id at a Hugging Face repo, and model-manager resolves the configuration, checks the requested CPU fits the node, downloads the weights, creates the serving pod, and registers the model with the gateway.
The Ansible roles here deploy the serving infrastructure itself: KServe for the model CRDs, an in-cluster AI gateway for model-aware routing, and — in litellm auth mode — LiteLLM for virtual API keys and Langfuse for request tracing.
The result is a single OpenAI-compatible endpoint. The model field in the request body selects which model answers, so any OpenAI SDK client works unchanged.
Want the full picture? See Architecture and Inference Request Flow.
An inference request enters through the LLM gateway, which authenticates and authorizes it, then routes it to a serving engine — vLLM or OpenVINO™ Model Server — and returns an OpenAI-compatible response. The gateway also provides model endpoints, user and key management, token telemetry, and monitoring. It all runs on a Kubernetes-orchestrated, Helm-packaged stack over Intel® Xeon® infrastructure.
Under the hood there are two gateways: the edge gateway terminates TLS and authenticates, while the in-cluster AI gateway routes to the model the request body names. See the Architecture deep-dive and Inference Request Flow for the full component list and execution flow.
Stands up the inference stack on a freshly prepared machine, then serves an LLM. In three steps you'll install the inference layer, deploy a model, and see it answer.
Note
Prerequisite: this layer is not standalone. es_auto_installer.sh is the installer
from the Intel® AI for Enterprise Solutions
repo, which clones this repository into ext/enterprise.ai-inference/. Clone that repo, prep the
machine, and create an environment, then run every command below from its root:
git clone https://github.com/intel/enterprise-ai-solutions.git
cd enterprise-ai-solutions
./es_auto_installer.sh configure && ./es_auto_installer.sh init localThis layer needs the cluster, gateway, auth, and storage its dependency layers provide — the
install inference step below auto-pulls them. Full list →
solutions Prerequisites.
Deploys KServe, the ServingRuntimes, the AI gateway, and — in litellm mode — LiteLLM and Langfuse. No models are deployed yet.
./es_auto_installer.sh install inference
export KUBECONFIG=$(pwd)/env/local/kubeconfig.yaml
kubectl get pods -n kserve # controller should be RunningSettings for this layer live in config.yaml at this repo's root, loaded by the installer as extra vars. See Configuration for every option, including the envoy / agentgateway choice and the cluster CPU-pinning policy.
model-manager resolves the configuration, verifies the requested CPU fits the node, downloads the weights, creates the serving pod, and registers the model with the gateway.
./model-manager deploy qwen3-0-6b --waitOr skip the catalog and deploy any Hugging Face model directly:
./model-manager deploy --id Qwen/Qwen3-0.6B --cpu 8 --memory 16Gi --waitImportant
Gated models (Llama, Mistral, Gemma) require a free Hugging Face token.
Export it as HF_TOKEN=hf_... before deploying.
When the command returns it prints the inference endpoint and a ready-to-run curl example. Full flag reference → Deploy a Model.
With the default auth_provider: keycloak, the helper script auto-discovers the gateway IP and fetches a JWT from the cluster:
source ./ext/enterprise.ai-inference/model_manager/scripts/get-keycloak-token.sh
# Exports: TOKEN, GATEWAY_IP, GATEWAY_DOMAINCall the model — the model field selects which one answers:
curl -sk --noproxy '*' \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
--resolve "inference.solutions.ai:443:$GATEWAY_IP" \
-d '{"model":"qwen3-0-6b","messages":[{"role":"user","content":"Hello!"}],"max_tokens":64}' \
https://inference.solutions.ai/v1/chat/completionsSee what is serving:
kubectl get llminferenceservices,inferenceservices -n llm-inferenceNote
Using auth_provider: litellm instead? LiteLLM becomes the auth boundary and the API is
served on litellm.<your-domain> — not inference.<your-domain> — with a LiteLLM master or
virtual key instead of a JWT. Full walkthrough →
Accessing Models.
Everything speaks the OpenAI-compatible API, so any tool that works with OpenAI works against your stack.
The Quick Start deploys one catalog model with defaults. From here you can change the engine and its version, add models, tune CPU and memory, switch the AI gateway or the CPU-pinning policy, and control what deploys automatically at install time.
| Goal | Guide |
|---|---|
| Deploy, update, and remove models | Deploy a Model |
| Endpoints and authentication modes | Accessing Models |
Add models — the models.yaml catalog |
Model Catalog |
| Engines, versions, and per-workload defaults | Runtimes |
| All configuration options | Configuration |
| Components, roles, and how this repo plugs in | Architecture |
| How a request reaches a model | Inference Request Flow |
| NUMA-aware CPU pinning | CPU Pinning & NUMA |
| Something isn't working | Troubleshooting |
Licensed under the Apache License, Version 2.0.
- Documentation Index
- GitHub Repository
- Intel® AI for Enterprise Solutions (core platform)
- Architecture
- Inference Request Flow
Intel® and Intel® Xeon® are registered trademarks of Intel Corporation or its subsidiaries. Licensed under the Apache License, Version 2.0.