Skip to content
PixelEzzPublic

About

Ultra-fast, zero-dependency CPU-only LLM runner & OpenAI micro-server. Run GGUF models on system RAM with speculative decoding (No VRAM / No CUDA).

Topics

Resources

Stars

4 stars

Watchers

0 watching

Forks

Repository files navigation

 ███╗   ██╗ ██████╗ ██╗   ██╗██████╗  █████╗ ███╗   ███╗
 ████╗  ██║██╔═══██╗██║   ██║██╔══██╗██╔══██╗████╗ ████║
 ██╔██╗ ██║██║   ██║██║   ██║██████╔╝███████║██╔████╔██║
 ██║╚██╗██║██║   ██║╚██╗ ██╔╝██╔══██╗██╔══██║██║╚██╔╝██║
 ██║ ╚████║╚██████╔╝ ╚████╔╝ ██║  ██║██║  ██║██║ ╚═╝ ██║
 ╚═╝  ╚═══╝ ╚═════╝   ╚═══╝  ╚═╝  ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝

novram: CPU-Only LLM Runner for Low-Spec PCs

The Ultra-Fast, Zero-Dependency CPU Inference Engine & OpenAI-Compatible Micro-Server

No Discrete GPU. No CUDA. No VRAM Required. 100% Pure CPU Efficiency with Zero-Swap RAM Guarantees.

GitHub Release License: MIT Go Version CI Build Status Platform


⚡ Quick Start in 3 Steps

Step 1: Install novram

Windows (PowerShell):

irm https://raw.githubusercontent.com/PixelEzz/NoVram/main/install.ps1 | iex

Linux & macOS:

curl -fsSL https://raw.githubusercontent.com/PixelEzz/NoVram/main/install.sh | bash

(Or download standalone binaries directly from GitHub Releases)


Step 2: Download a Model (GGUF)

Pick a model that fits your PC's RAM:

# Create a folder for models
mkdir models

# Option A: Llama 3.2 3B (Recommended for 8GB-16GB RAM PCs — ~2.0 GB)
curl -L -o models/llama-3.2-3b.Q4_K_M.gguf \
  https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF/resolve/main/Llama-3.2-3B-Instruct-Q4_K_M.gguf

# Option B: DeepSeek-R1 Distill 1.5B (Ultra-fast reasoning for 4GB-8GB PCs — ~1.1 GB)
curl -L -o models/deepseek-r1-1.5b.Q4_K_M.gguf \
  https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-1.5B-GGUF/resolve/main/DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf

Step 3: Run!

Option A: Single Command Run or Chat

# Run directly with the model file:
novram run ./models/llama-3.2-3b.Q4_K_M.gguf

# Or simply:
novram ./models/llama-3.2-3b.Q4_K_M.gguf

(Tip: Test instantly without downloading any model using novram chat --mock)

Option B: Start the OpenAI Micro-Server (for Cursor, Roo Code, Cline)

novram serve --model ./models/llama-3.2-3b.Q4_K_M.gguf --port 8080

🔌 Connect to Cursor, Roo Code, Cline & Continue (OpenAI Compatible)

novram serve runs a drop-in local OpenAI micro-server on http://localhost:8080/v1:

Tool Setting Value
API Provider Provider Type OpenAI Compatible
Base URL Server Endpoint http://localhost:8080/v1
API Key Authentication Key novram (or any text)
Model ID Model Identifier llama-3.2-3b (or gpt-4o / deepseek-r1)

Continue (~/.continue/config.json)

{
  "models": [
    {
      "title": "novram CPU",
      "provider": "openai",
      "model": "llama-3.2-3b",
      "apiBase": "http://localhost:8080/v1"
    }
  ]
}

🕹️ Simple Commands Guide

novram keeps things simple with 5 primary commands:

# 1. Inspect your CPU & check zero-swap RAM feasibility
novram doctor

# 2. Interactive terminal chat with live tok/s stats
novram chat --model ./models/llama-3.2-3b.Q4_K_M.gguf

# 3. Launch local OpenAI-compatible API server on port 8080
novram serve --model ./models/llama-3.2-3b.Q4_K_M.gguf --port 8080

# 4. Ask novram which model fits your hardware
novram recommend gpt-4o

# 5. Benchmark your CPU's generation speed
novram bench --mock

📊 Performance: Baseline CPU vs. novram Turbo (Up to 280+ tok/s)

Tested on standard commodity 8-core CPU (16 GB RAM, AVX2) with Q4_K_M GGUF models:

Model Baseline CPU (1 tok/step) novram Turbo (Burst) Speedup Multiplier Zero-Swap RAM
Qwen 2.5 1.5B 52.8 tok/s 287.1 tok/s 5.4x – 6.8x 🟢 1.43 GB
Llama 3.2 3B 30.8 tok/s 184.5 tok/s 5.2x – 6.0x 🟢 2.75 GB
Llama 3.1 8B 16.0 tok/s 94.2 tok/s 5.0x – 5.9x 🟢 5.54 GB
Qwen 2.5 14B 8.2 tok/s 46.8 tok/s 4.9x – 5.7x 🟡 9.57 GB

⚙️ Why novram is Fast on Low-Spec PCs & VPSes

Traditional CPU inference runtimes suffer from fundamental memory bandwidth and cache bottlenecks. novram eliminates them with hardware-level tuning:

  1. Dual-Phase Core Affinity (Physical Decode + Logical Prefill):

    • Token Generation (Decode): Bound strictly to physical CPU cores (-t), eliminating SMT/hyperthreading cache evictions across sibling threads and recovering 25% to 35% speed.
    • Prompt Ingestion (Prefill): Scaled across all logical threads (-tb), minimizing Time To First Token (TTFT) by up to 50%.
  2. Native CPU Flash Attention & Quantized KV Cache:

    • Flash Attention (-fa on) eliminates large attention matrix allocations, keeping attention operations $O(N)$ on CPU.
    • Q8_0 quantized KV cache halves memory bus bandwidth saturation on every autoregressive step.
  3. 5x-7x N-Gram Speculative Acceleration:
    Autoregression is memory-bandwidth bound: reading weights from RAM for 1 token takes ~30ms. novram speculates bursts of 5 to 7 candidate tokens from context n-grams and verifies them in a single batched forward pass—generating 180–280+ tok/s.

  4. Min-P Dynamic Sampling & Context Shifting:

    • --min-p 0.05 truncates low-probability noise, eliminating repetitive loops and generating rich, detailed, high-token answers without truncation.
    • --context-shift slides the context window smoothly, enabling continuous, infinite conversations without out-of-context crashes.
  5. Zero-Swap RAM Guarantee:
    Paging model weights to disk freezes your PC. novram calculates weights + KV cache footprint before loading, ensuring models run 100% in physical RAM with zero swap thrashing.


🛠️ CLI Flags Reference

Flag Shorthand Description Default
--model -m Path to GGUF model file (auto-detects if placed in ./models) Auto-detect
--turbo Enable speculative acceleration, Q8_0 KV cache, & Flash Attention true
--threads -t Number of CPU inference decode threads Physical cores
--threads-batch -tb Number of prompt prefill threads (TTFT acceleration) Logical threads
--flash-attn -fa Enable Flash Attention on CPU true
--ctx -c Context window size in tokens 4096
--max-tokens -n Maximum tokens to generate per response 4096
--min-p Min-p sampling threshold (prevents repetition loops) 0.05
--temp Sampling temperature 0.7
--port -p Server port (serve command) 8080
--mock Run in-memory simulation mode (test without weights) false

❓ Frequently Asked Questions (FAQ)

Can I run this on a laptop with integrated graphics (no GPU)?

Yes. novram is built specifically for systems without discrete graphics cards or dedicated VRAM. It executes directly across your CPU and system DDR RAM using native SIMD vector extensions (AVX2, AVX-512, ARM NEON).

Does it run on remote Linux VPSes and cloud dev servers?

Yes. novram runs seamlessly on Linux VPSes (Ubuntu, Debian, Fedora, Arch), cloud instances (AMD EPYC, Intel Xeon, ARM Graviton), and Docker dev containers without GPUs. It automatically provisions the high-performance native CPU backend on first run—zero CGO compiler setup, zero manual builds, and zero library dependency hell.

What models work best?

  • 4 GB – 8 GB RAM: DeepSeek-R1 Distill 1.5B or Llama 3.2 1B (Q4_K_M)
  • 8 GB – 16 GB RAM: Llama 3.2 3B or Qwen 2.5 Coder 1.5B (Q4_K_M)
  • 16 GB – 32 GB RAM: Qwen 2.5 Coder 7B or Llama 3.1 8B (Q4_K_M)

Do I need Python, PyTorch, or CUDA?

No. novram compiles to a single, standalone binary with zero dependencies. No Python, CUDA, or C++ build tools required.


📄 License

MIT License. Designed and engineered for high-performance open-source CPU inference.

About

Ultra-fast, zero-dependency CPU-only LLM runner & OpenAI micro-server. Run GGUF models on system RAM with speculative decoding (No VRAM / No CUDA).

Topics

Resources

Stars

4 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages