███╗ ██╗ ██████╗ ██╗ ██╗██████╗ █████╗ ███╗ ███╗
████╗ ██║██╔═══██╗██║ ██║██╔══██╗██╔══██╗████╗ ████║
██╔██╗ ██║██║ ██║██║ ██║██████╔╝███████║██╔████╔██║
██║╚██╗██║██║ ██║╚██╗ ██╔╝██╔══██╗██╔══██║██║╚██╔╝██║
██║ ╚████║╚██████╔╝ ╚████╔╝ ██║ ██║██║ ██║██║ ╚═╝ ██║
╚═╝ ╚═══╝ ╚═════╝ ╚═══╝ ╚═╝ ╚═╝╚═╝ ╚═╝╚═╝ ╚═╝
The Ultra-Fast, Zero-Dependency CPU Inference Engine & OpenAI-Compatible Micro-Server
No Discrete GPU. No CUDA. No VRAM Required. 100% Pure CPU Efficiency with Zero-Swap RAM Guarantees.
irm https://raw.githubusercontent.com/PixelEzz/NoVram/main/install.ps1 | iexcurl -fsSL https://raw.githubusercontent.com/PixelEzz/NoVram/main/install.sh | bash(Or download standalone binaries directly from GitHub Releases)
Pick a model that fits your PC's RAM:
# Create a folder for models
mkdir models
# Option A: Llama 3.2 3B (Recommended for 8GB-16GB RAM PCs — ~2.0 GB)
curl -L -o models/llama-3.2-3b.Q4_K_M.gguf \
https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF/resolve/main/Llama-3.2-3B-Instruct-Q4_K_M.gguf
# Option B: DeepSeek-R1 Distill 1.5B (Ultra-fast reasoning for 4GB-8GB PCs — ~1.1 GB)
curl -L -o models/deepseek-r1-1.5b.Q4_K_M.gguf \
https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-1.5B-GGUF/resolve/main/DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf# Run directly with the model file:
novram run ./models/llama-3.2-3b.Q4_K_M.gguf
# Or simply:
novram ./models/llama-3.2-3b.Q4_K_M.gguf(Tip: Test instantly without downloading any model using novram chat --mock)
novram serve --model ./models/llama-3.2-3b.Q4_K_M.gguf --port 8080novram serve runs a drop-in local OpenAI micro-server on http://localhost:8080/v1:
| Tool | Setting | Value |
|---|---|---|
| API Provider | Provider Type | OpenAI Compatible |
| Base URL | Server Endpoint | http://localhost:8080/v1 |
| API Key | Authentication Key | novram (or any text) |
| Model ID | Model Identifier | llama-3.2-3b (or gpt-4o / deepseek-r1) |
{
"models": [
{
"title": "novram CPU",
"provider": "openai",
"model": "llama-3.2-3b",
"apiBase": "http://localhost:8080/v1"
}
]
}novram keeps things simple with 5 primary commands:
# 1. Inspect your CPU & check zero-swap RAM feasibility
novram doctor
# 2. Interactive terminal chat with live tok/s stats
novram chat --model ./models/llama-3.2-3b.Q4_K_M.gguf
# 3. Launch local OpenAI-compatible API server on port 8080
novram serve --model ./models/llama-3.2-3b.Q4_K_M.gguf --port 8080
# 4. Ask novram which model fits your hardware
novram recommend gpt-4o
# 5. Benchmark your CPU's generation speed
novram bench --mockTested on standard commodity 8-core CPU (16 GB RAM, AVX2) with Q4_K_M GGUF models:
| Model | Baseline CPU (1 tok/step) | novram Turbo (Burst) | Speedup Multiplier | Zero-Swap RAM |
|---|---|---|---|---|
| Qwen 2.5 1.5B | 52.8 tok/s | 287.1 tok/s | 5.4x – 6.8x | 🟢 1.43 GB |
| Llama 3.2 3B | 30.8 tok/s | 184.5 tok/s | 5.2x – 6.0x | 🟢 2.75 GB |
| Llama 3.1 8B | 16.0 tok/s | 94.2 tok/s | 5.0x – 5.9x | 🟢 5.54 GB |
| Qwen 2.5 14B | 8.2 tok/s | 46.8 tok/s | 4.9x – 5.7x | 🟡 9.57 GB |
Traditional CPU inference runtimes suffer from fundamental memory bandwidth and cache bottlenecks. novram eliminates them with hardware-level tuning:
-
Dual-Phase Core Affinity (Physical Decode + Logical Prefill):
-
Token Generation (Decode): Bound strictly to physical CPU cores (
-t), eliminating SMT/hyperthreading cache evictions across sibling threads and recovering 25% to 35% speed. -
Prompt Ingestion (Prefill): Scaled across all logical threads (
-tb), minimizing Time To First Token (TTFT) by up to 50%.
-
Token Generation (Decode): Bound strictly to physical CPU cores (
-
Native CPU Flash Attention & Quantized KV Cache:
- Flash Attention (
-fa on) eliminates large attention matrix allocations, keeping attention operations$O(N)$ on CPU. -
Q8_0quantized KV cache halves memory bus bandwidth saturation on every autoregressive step.
- Flash Attention (
-
5x-7x N-Gram Speculative Acceleration:
Autoregression is memory-bandwidth bound: reading weights from RAM for 1 token takes ~30ms.novramspeculates bursts of 5 to 7 candidate tokens from context n-grams and verifies them in a single batched forward pass—generating 180–280+ tok/s. -
Min-P Dynamic Sampling & Context Shifting:
-
--min-p 0.05truncates low-probability noise, eliminating repetitive loops and generating rich, detailed, high-token answers without truncation. -
--context-shiftslides the context window smoothly, enabling continuous, infinite conversations without out-of-context crashes.
-
-
Zero-Swap RAM Guarantee:
Paging model weights to disk freezes your PC.novramcalculates weights + KV cache footprint before loading, ensuring models run 100% in physical RAM with zero swap thrashing.
| Flag | Shorthand | Description | Default |
|---|---|---|---|
--model |
-m |
Path to GGUF model file (auto-detects if placed in ./models) |
Auto-detect |
--turbo |
Enable speculative acceleration, Q8_0 KV cache, & Flash Attention | true |
|
--threads |
-t |
Number of CPU inference decode threads | Physical cores |
--threads-batch |
-tb |
Number of prompt prefill threads (TTFT acceleration) | Logical threads |
--flash-attn |
-fa |
Enable Flash Attention on CPU | true |
--ctx |
-c |
Context window size in tokens | 4096 |
--max-tokens |
-n |
Maximum tokens to generate per response | 4096 |
--min-p |
Min-p sampling threshold (prevents repetition loops) | 0.05 |
|
--temp |
Sampling temperature | 0.7 |
|
--port |
-p |
Server port (serve command) |
8080 |
--mock |
Run in-memory simulation mode (test without weights) | false |
Yes. novram is built specifically for systems without discrete graphics cards or dedicated VRAM. It executes directly across your CPU and system DDR RAM using native SIMD vector extensions (AVX2, AVX-512, ARM NEON).
Yes. novram runs seamlessly on Linux VPSes (Ubuntu, Debian, Fedora, Arch), cloud instances (AMD EPYC, Intel Xeon, ARM Graviton), and Docker dev containers without GPUs. It automatically provisions the high-performance native CPU backend on first run—zero CGO compiler setup, zero manual builds, and zero library dependency hell.
- 4 GB – 8 GB RAM:
DeepSeek-R1 Distill 1.5BorLlama 3.2 1B(Q4_K_M) - 8 GB – 16 GB RAM:
Llama 3.2 3BorQwen 2.5 Coder 1.5B(Q4_K_M) - 16 GB – 32 GB RAM:
Qwen 2.5 Coder 7BorLlama 3.1 8B(Q4_K_M)
No. novram compiles to a single, standalone binary with zero dependencies. No Python, CUDA, or C++ build tools required.
MIT License. Designed and engineered for high-performance open-source CPU inference.