FastAPI Model Inference & Embeddings API

A high-performance asynchronous FastAPI service that serves real-time machine learning predictions and text embeddings with a sub-200 ms latency target. The model is loaded once at start-up, inference runs on a dedicated thread pool so the event loop is never blocked, and every request is validated, timed and tracked.

---

Table of Contents

1. Features
2. Architecture
3. Project Structure
4. Tech Stack
5. Installation
6. Running the API
7. API Reference
8. Configuration
9. How Low Latency Is Achieved
10. Testing
11. Benchmark & Load Testing
12. Training Your Own Classifier
13. Docker & Deployment
14. Troubleshooting
15. Future Improvements
16. Author

---

Features

• Async FastAPI endpoints - /embed, /predict, /similarity, /health, /stats
• Real-time predictions - zero-shot text classification with custom labels per request, or a trained scikit-learn classifier
• Sentence embeddings - 384-dimensional L2-normalised vectors (all-MiniLM-L6-v2)
• Sub-200 ms target - model loaded once, warm-up at start-up, thread-pool inference, LRU embedding cache
• Built-in observability - X-Process-Time-Ms / X-Request-ID headers, p50/p95/p99 latency in /stats, slow-request warnings
• Strict validation - Pydantic v2 schemas with batch-size and text-length limits
• Safe by design - inference timeout (HTTP 504), consistent JSON error responses
• Tested - pytest suite that runs without downloading the model
• Load-test tooling - Locust file and a benchmark script that checks the 200 ms budget
• Container-ready - Dockerfile (CPU PyTorch, model baked into the image, non-root user, health check) and docker-compose

Architecture

flowchart LR
    C[Client] -->|HTTP JSON| U[Uvicorn]
    U --> M[Timing middleware<br/>request id + latency]
    M --> V[Pydantic validation]
    V --> R[Routers<br/>/embed /predict /similarity]
    R -->|run_in_executor| T[Inference thread pool]
    T --> K{LRU cache}
    K -->|miss| E[SentenceTransformer<br/>loaded once at start-up]
    K -->|hit| T
    T --> H[Classifier head<br/>zero-shot or trained]
    H --> R
    R --> C

Loading

Request flow: the async endpoint validates the payload, then hands the blocking model call to a thread pool. While the model computes, the event loop keeps accepting other requests. Repeated texts are answered from the cache without touching the model.

Project Structure

FastAPI-Model-Inference-Embeddings-API/
├── app/
│   ├── main.py                     # app factory, lifespan (model loading), error handlers
│   ├── config.py                   # settings (environment variables, APP_ prefix)
│   ├── schemas.py                  # Pydantic request / response models
│   ├── middleware.py               # timing headers + latency tracking
│   ├── metrics.py                  # rolling p50 / p95 / p99 tracker
│   ├── api/
│   │   ├── deps.py                 # dependency injection helpers
│   │   ├── health.py               # /, /health, /stats
│   │   └── inference.py            # /embed, /predict, /similarity
│   └── services/
│       ├── embedding_service.py    # model wrapper, thread pool, cache lookup
│       ├── classifier_service.py   # zero-shot / trained classifier head
│       └── cache.py                # thread-safe LRU cache
├── tests/                          # pytest suite (fake embedder, no downloads)
├── load_test/locustfile.py         # Locust load test
├── scripts/
│   ├── benchmark.py                # latency benchmark with pass / fail on the budget
│   └── train_classifier.py         # train a classifier on top of embeddings
├── data/sample_training_data.csv   # tiny demo dataset for the training script
├── models/                         # trained classifiers are saved here
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
├── requirements-dev.txt
├── .env.example
├── README.md
└── REPORT.md

Tech Stack

Layer	Technology
Web framework	FastAPI, Uvicorn
Validation & settings	Pydantic v2, pydantic-settings
Model inference	PyTorch, sentence-transformers (all-MiniLM-L6-v2)
Classifier	NumPy zero-shot head, optional scikit-learn
Testing	pytest, httpx, Locust
Deployment	Docker, docker-compose

Installation

Requirements: Python 3.10 or newer (3.11 recommended) and an internet connection for the first run (the model is downloaded once, about 90 MB).

# 1. Create and activate a virtual environment
python -m venv venv
venv\Scripts\activate            # Windows
# source venv/bin/activate       # Linux / macOS

# 2. Upgrade pip
python -m pip install --upgrade pip

# 3. Install PyTorch first (large download - use retries on slow connections)
pip install torch --default-timeout=300 --retries 10

# 4. Install the remaining dependencies
pip install -r requirements.txt --default-timeout=120 --retries 10

# 5. (Optional) development tools: pytest, httpx, locust
pip install -r requirements-dev.txt

CPU-only PyTorch (smaller download):

pip install torch --index-url https://download.pytorch.org/whl/cpu --default-timeout=300 --retries 10

Running the API

uvicorn app.main:app --host 0.0.0.0 --port 8000

Development mode with auto-reload:

uvicorn app.main:app --reload

Production (several worker processes):

uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 2

Once started, open the interactive documentation:

• Swagger UI: http://127.0.0.1:8000/docs
• ReDoc: http://127.0.0.1:8000/redoc

The first start downloads the model and takes longer. Later starts load it from the local cache.

API Reference

GET /health

{
  "status": "ok",
  "model_loaded": true,
  "model": "sentence-transformers/all-MiniLM-L6-v2",
  "classifier": "zero-shot",
  "version": "1.0.0",
  "uptime_seconds": 42.7
}

POST /embed

Convert one or more texts into embedding vectors.

curl -X POST http://127.0.0.1:8000/embed \
  -H "Content-Type: application/json" \
  -d '{"texts": ["FastAPI is fast.", "Embeddings capture meaning."]}'

{
  "embeddings": [[0.0123, -0.0456, "... 384 values"], [0.0345, 0.0211, "..."]],
  "dimension": 384,
  "count": 2,
  "cached": 0,
  "model": "sentence-transformers/all-MiniLM-L6-v2",
  "latency_ms": 18.4
}

POST /predict

Classify texts in real time. Labels are optional; the default labels are positive, negative and neutral.

curl -X POST http://127.0.0.1:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"texts": ["I absolutely love this product!"]}'

{
  "predictions": [
    {
      "label": "positive",
      "confidence": 0.93,
      "scores": {"positive": 0.93, "negative": 0.02, "neutral": 0.05}
    }
  ],
  "classifier": "zero-shot",
  "model": "sentence-transformers/all-MiniLM-L6-v2",
  "latency_ms": 12.1
}

Custom labels (zero-shot):

curl -X POST http://127.0.0.1:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"texts": ["The team won the final 2-1"], "labels": ["sports", "politics", "technology"]}'

(The numbers above are illustrative examples of the response format.)

POST /similarity

curl -X POST http://127.0.0.1:8000/similarity \
  -H "Content-Type: application/json" \
  -d '{"text_a": "A cat sits on the mat.", "text_b": "A kitten rests on a rug."}'

Returns {"similarity": <cosine value between -1 and 1>, "model": "...", "latency_ms": ...}.

GET /stats

Rolling latency percentiles (p50 / p95 / p99), the share of requests inside the 200 ms budget, and cache hit rate.

Response headers

Header	Meaning
X-Process-Time-Ms	Total server-side time for the request
X-Request-ID	Request identifier (echoes the incoming one if provided)

Error codes

Code	Reason
422	Validation error (empty list, text too long, batch too large, fewer than two labels)
504	Inference exceeded APP_INFERENCE_TIMEOUT_SECONDS
500	Unexpected server error

Configuration

Settings are read from environment variables (prefix APP_) or from a .env file. Copy .env.example to .env to start.

Variable	Default	Description
APP_EMBEDDING_MODEL_NAME	sentence-transformers/all-MiniLM-L6-v2	Embedding model
APP_DEVICE	cpu	cpu, cuda or mps
APP_MAX_SEQUENCE_LENGTH	128	Max tokens per text (shorter = faster)
APP_INFERENCE_WORKERS	2	Threads that run the model
APP_TORCH_THREADS	0	PyTorch intra-op threads (0 = default)
APP_CACHE_SIZE	2048	LRU cache entries (0 disables)
APP_LATENCY_BUDGET_MS	200	Budget used by /stats and slow-request warnings
APP_MAX_BATCH_SIZE	64	Max texts per request
APP_MAX_TEXT_LENGTH	2000	Max characters per text
APP_INFERENCE_TIMEOUT_SECONDS	5	Timeout before HTTP 504
APP_CLASSIFIER_PATH	(empty)	Path to a trained .joblib classifier

How Low Latency Is Achieved

1. Model loaded once at application start-up, never per request.
2. Warm-up inferences at start-up, so the first real request is not slow.
3. Thread-pool inference - blocking model calls do not freeze the async event loop.
4. torch.inference_mode() - no gradient tracking, less memory and faster execution.
5. Small, fast model - all-MiniLM-L6-v2 (6 layers, 384 dimensions) with a 128-token limit.
6. LRU cache - repeated texts skip the model completely.
7. Batch de-duplication - identical texts inside one request are encoded once.
8. Normalised embeddings - cosine similarity becomes a single dot product.
9. Multiple Uvicorn workers - use more CPU cores for concurrent traffic.
10. Strict limits and a timeout - bounded batch size, text length and inference time.

Tuning tip: APP_INFERENCE_WORKERS x APP_TORCH_THREADS should not exceed your CPU core count, otherwise threads compete with each other and latency gets worse.

Testing

pytest

The tests use a fake embedding backend, so they run in about a second and need no model download. They cover validation, caching, prediction output, similarity, timing headers and statistics.

Benchmark & Load Testing

Start the API first, then in another terminal:

Quick benchmark (checks the 200 ms budget on p95):

python scripts/benchmark.py --endpoint predict --requests 300 --concurrency 10
python scripts/benchmark.py --endpoint embed   --requests 300 --concurrency 10

The script adds random suffixes to the texts so the cache cannot hide the real inference time. Use --allow-cache to measure cached performance.

Locust load test:

locust -f load_test/locustfile.py --host http://127.0.0.1:8000

Open http://localhost:8089, or run headless:

locust -f load_test/locustfile.py --host http://127.0.0.1:8000 --users 20 --spawn-rate 5 --run-time 60s --headless

Latency depends on your hardware. Measure on the machine you deploy to and record the numbers in REPORT.md.

Training Your Own Classifier

The default zero-shot classifier needs no training data. For higher accuracy on your own task, train a classifier on top of the embeddings:

python scripts/train_classifier.py --data data/sample_training_data.csv

Your CSV needs the columns text,label. The bundled sample file is a tiny demonstration dataset only. Then enable the model:

set APP_CLASSIFIER_PATH=models/classifier.joblib        # Windows (cmd)
# export APP_CLASSIFIER_PATH=models/classifier.joblib   # Linux / macOS
uvicorn app.main:app

/health will then report "classifier": "trained-classifier".

Docker & Deployment

Build and run:

docker build -t fastapi-inference-embeddings-api .
docker run -p 8000:8000 fastapi-inference-embeddings-api

With docker-compose:

docker compose up --build

The image uses CPU-only PyTorch, bakes the model into the image (fast, offline start-up), runs as a non-root user and includes a health check.

Deploying to a platform (Render, Railway, AWS, etc.):

• Deploy the Docker image or connect the GitHub repository and use the Dockerfile.
• Expose port 8000 and use /health as the health-check path.
• Choose an instance with at least 1 GB RAM (2 GB recommended).
• Set WORKERS to the number of CPU cores (keep APP_INFERENCE_WORKERS low).
• Avoid free tiers that sleep: a cold start reloads the model and the first request will be slow.

Troubleshooting

Problem	Solution
No matching distribution found for torch	Upgrade pip, check the internet connection, then use pip install torch --default-timeout=300 --retries 10 (or the CPU index URL above)
Download stops or socket errors	Re-run the same command with --retries 10 --default-timeout=120; pip skips packages that are already installed
First request is very slow	The model is still downloading, or warm-up has not finished; wait for "Service is ready" in the log
Latency above 200 ms	Lower APP_MAX_SEQUENCE_LENGTH, reduce batch sizes, check APP_INFERENCE_WORKERS x APP_TORCH_THREADS versus CPU cores
ModuleNotFoundError: app	Run commands from the project root folder

Future Improvements

• ONNX Runtime / quantised model for faster CPU inference
• Redis cache shared between workers
• API-key authentication and rate limiting
• Prometheus metrics and Grafana dashboard
• Dynamic request batching for higher throughput on GPU
• CI pipeline (GitHub Actions) running tests and a smoke benchmark

Author

Muhammad Faizan Farooq GitHub: faizanjutt473