Skip to content

REPORT.md

Latest commit

 

History

History
159 lines (118 loc) · 9.17 KB

REPORT.md

File metadata and controls

159 lines (118 loc) · 9.17 KB
Raw

Project Report

FastAPI Model Inference & Embeddings API
Author Muhammad Faizan Farooq
Date October 2026
Repository https://github.com/faizanjutt473
Status Implementation complete, benchmark to be recorded on the target machine

1. Executive Summary

This project delivers an asynchronous REST API, built with FastAPI, that serves machine learning predictions and text embeddings in real time. The service is designed around one hard requirement: a response time below 200 ms per request. To meet it, the model is loaded once at start-up, warmed up, and executed on a dedicated thread pool so the asynchronous event loop never blocks. A thread-safe LRU cache removes repeated work, and built-in latency tracking makes the target measurable at any time.

The result is a production-style service with input validation, error handling, monitoring endpoints, automated tests, load-testing tools and a Docker image ready for deployment.

2. Objectives and Requirement Mapping

Requirement Implementation
Deploy an asynchronous FastAPI endpoint async def endpoints in app/api/inference.py; blocking model calls moved to a thread pool
Serve real-time model predictions POST /predict - zero-shot or trained classifier
Serve embeddings POST /embed and POST /similarity
Sub-200 ms latency Model loaded once, warm-up, inference_mode, small model, cache, limits; verified with scripts/benchmark.py and Locust
Deployment Dockerfile, docker-compose, health check, deployment guide in the README

3. System Architecture

Client -> Uvicorn -> Timing middleware -> Pydantic validation -> Router
                                                                   |
                                                    run_in_executor (thread pool)
                                                                   |
                                          LRU cache -> SentenceTransformer (loaded once)
                                                                   |
                                                 Classifier head (zero-shot / trained)

Components

  • main.py - app factory and lifespan. The model is loaded and warmed up before the server accepts traffic.
  • EmbeddingService - wraps the SentenceTransformer model, owns the inference thread pool and the cache lookup logic.
  • ClassifierService - turns embeddings into predictions. Default mode is zero-shot; a trained scikit-learn classifier can be plugged in through a setting.
  • TimingMiddleware / LatencyTracker - add X-Process-Time-Ms and X-Request-ID headers and keep rolling p50 / p95 / p99 statistics exposed at /stats.
  • schemas.py - strict request and response contracts.
  • config.py - all tunable values as environment variables.

4. Technology Choices

Decision Reason
FastAPI Native async support, automatic OpenAPI docs, Pydantic validation
all-MiniLM-L6-v2 Only 6 transformer layers and 384 dimensions: small (about 90 MB), fast on CPU, good general-purpose sentence quality
Thread pool for inference Model computation is CPU/GPU bound. Running it directly inside async def would block the event loop; PyTorch releases the GIL during heavy computation, so threads give real concurrency
L2-normalised embeddings Cosine similarity reduces to a dot product
In-process LRU cache Zero network overhead, simple, effective for repeated texts
Zero-shot head as default The service works immediately with custom labels and needs no training data; a trained classifier is supported for higher accuracy
pydantic-settings Configuration without code changes, ready for containers

5. API Design

Endpoint Method Purpose
/embed POST Embeddings for 1-64 texts
/predict POST Classification of 1-64 texts, optional custom labels
/similarity POST Cosine similarity of two texts
/health GET Liveness and readiness information
/stats GET Latency percentiles and cache statistics

Input limits: at most 64 texts per request, 2000 characters per text, 2-20 labels. Violations return HTTP 422. An inference that exceeds the configured timeout returns HTTP 504, and unexpected failures return a consistent JSON 500 response.

6. Latency Strategy

Technique Effect
Single model load at start-up Removes multi-second loading time from every request
Warm-up inferences The first real request does not pay initialisation costs
Thread-pool execution Event loop stays free; concurrent requests are accepted while the model computes
torch.inference_mode() No autograd bookkeeping, lower memory use
128-token sequence limit Bounds the cost of long inputs
LRU cache + batch de-duplication Repeated texts cost almost nothing
Bounded batch size and text length Prevents a single request from exceeding the budget
Multiple workers Scales across CPU cores
Per-request timing and /stats Makes any budget violation visible immediately

Known trade-offs. Latency grows with batch size and text length; the 200 ms target applies to typical requests (a few short texts). On a CPU with few cores, APP_INFERENCE_WORKERS x APP_TORCH_THREADS must be kept at or below the core count, otherwise threads compete and latency increases.

7. Testing

Automated tests (pytest) - 10 tests, all passing, run against a deterministic fake embedding backend so no model download is required:

  • root and health endpoints
  • embedding shape, dimension and response headers
  • cache behaviour (first call miss, second call hit, identical vectors)
  • validation errors (empty list, blank text, oversized batch, oversized text, missing field)
  • prediction output (probabilities sum to 1, labels are valid)
  • custom labels and the two-label minimum
  • similarity of identical texts
  • statistics endpoint

Performance tools

  • scripts/benchmark.py - concurrent client that reports mean, p50, p95, p99, max and throughput, and exits with PASS or FAIL against the 200 ms budget. Random suffixes prevent the cache from hiding real inference time.
  • load_test/locustfile.py - mixed workload (predict, embed, similarity, health) with a web UI or headless mode.

8. Benchmark Results

Latency depends entirely on the hardware, the number of CPU cores and the request size, so the figures below must be measured on the machine where the service is deployed. They are intentionally not estimated here.

How to measure

uvicorn app.main:app --host 0.0.0.0 --port 8000
python scripts/benchmark.py --endpoint predict --requests 300 --concurrency 10
python scripts/benchmark.py --endpoint embed   --requests 300 --concurrency 10

Results table (fill in after running)

Endpoint Concurrency Requests Throughput (req/s) p50 (ms) p95 (ms) p99 (ms) Within 200 ms
/predict 10 300
/embed (2 texts) 10 300
/similarity 10 300

Test environment (fill in): CPU model and cores: ______ RAM: ______ OS: ______ Python: ______ Workers: ______

Acceptance criterion: p95 latency at or below 200 ms with zero failed requests.

9. Deployment

  • Docker image based on python:3.11-slim, CPU-only PyTorch, model baked into the image, non-root user, built-in health check.
  • docker-compose for one-command start-up.
  • Platforms such as Render, Railway or AWS can run the same image. Recommended: at least 1 GB RAM (2 GB preferred), /health as the health-check path, and an instance that does not sleep, because a cold start reloads the model.

10. Limitations

  • The default classifier is zero-shot: it needs no training data, but its accuracy is lower than a classifier trained on task-specific labelled data. For production use, train one with scripts/train_classifier.py.
  • The cache is per process; with several workers each keeps its own cache.
  • No authentication or rate limiting is included.
  • The bundled training CSV is a tiny demonstration dataset and is not suitable for real evaluation.
  • Embedding quality is tuned for English; multilingual use requires a different model.

11. Future Work

  1. ONNX Runtime and quantisation for faster CPU inference.
  2. Shared Redis cache across workers.
  3. API-key authentication and rate limiting.
  4. Prometheus metrics with a Grafana dashboard.
  5. Dynamic request batching for GPU deployments.
  6. Continuous integration with automated tests and a smoke benchmark.
  7. Model versioning and A/B comparison of classifiers.

12. Conclusion

The project provides a complete, well-structured inference service that satisfies the functional requirements (asynchronous endpoints, real-time predictions, embeddings, deployment) and is engineered for the sub-200 ms target through start-up loading, warm-up, thread-pool execution, caching and strict limits. Built-in monitoring, automated tests, load-testing tools and Docker packaging make the target verifiable and the service ready to deploy.