feat: FIFO admission proxy for local LLM servers

.gitignore

@@ -216,3 +216,6 @@ __marimo__/
 
 # Streamlit
 .streamlit/secrets.toml
+
+# Hermit local state
+.hermit/

CONTRIBUTING.md

@@ -0,0 +1,32 @@
+# Contributing
+
+Runtime is stdlib-only. Pin Python with Hermit (`./bin/python3`, currently 3.13).
+
+```bash
+./bin/hermit install python3@3.13
+./bin/python3 -m pip install -e ".[dev]"
+./bin/python3 -m pytest
+./bin/python3 -m ruff check src tests
+```
+
+## Layout
+
+| Path | Role |
+|---|---|
+| `src/llm_queue_proxy/admission.py` | FIFO slot gate |
+| `src/llm_queue_proxy/paths.py` | Which POSTs take a slot |
+| `src/llm_queue_proxy/server.py` | Reverse proxy + `/__queue/health` |
+| `src/llm_queue_proxy/cli.py` | Flags / `LLM_QUEUE_*` env |
+| `tests/` | Dummy upstream on `127.0.0.1:0` |
+
+New admitted path → `DEFAULT_ADMIT_PREFIXES` + `tests/test_paths.py` + README. New CLI flag → env + README + `tests/test_cli.py`.
+
+## Git
+
+- Default branch `master`
+- Feature branches `YYYY-MM-DD.eizus.description`
+- Author `eizus <hello@cdr.xyz>`
+- One worktree per PR; reviewer `@adrw`
+- `gh pr create --body-file`
+
+Do not bind `0.0.0.0`. Do not log `Authorization`. This proxy does not evict engine KV; it only serializes HTTP generations.

README.md

@@ -1,2 +1,111 @@
 # llm-queue-proxy
-FIFO admission proxy for local LLM servers: one in-flight generation, queue the rest.
+
+FIFO admission proxy for local LLM HTTP servers. One in-flight generation at a time; extra requests wait in arrival order.
+
+Built by [Cedar Labs](https://cdr.xyz). Contributing: [CONTRIBUTING.md](CONTRIBUTING.md).
+
+**This does not shrink model weights or KV.** It only stops two clients from prefilling the same local engine at once.
+
+## Why
+
+Local engines such as [MTPLX](https://mtplx.com) already serialize decode, but each concurrent `/v1/chat/completions` still holds a KV bank through prefill. Two Hermes threads against Flash-Next on a full unified-memory Mac abort with:
+
+> insufficient memory: the request exceeded available GPU memory (sustained critical memory pressure during prefill)
+
+Point Hermes at this proxy (`:8001`) and leave the engine on `:8000`. Generation POSTs take one slot. Health and `/v1/models` stay concurrent.
+
+What it **will** do:
+
+- Queue a second Hermes/WebUI thread behind the first instead of overlapping prefills
+- Return `429` when the wait queue is full, `503` when a waiter times out
+- Stream SSE through without buffering the whole completion
+
+What it **will not** do:
+
+- Evict the other session's KV from the engine
+- Fix a single 60k–100k context that already sits at zero memory headroom
+- Replace `--context-window` caps, closing other apps, or swapping to 27B
+
+## Install
+
+Python 3.11+ (this repo pins **3.13** via [Hermit](https://cashapp.github.io/hermit/)).
+
+```bash
+git clone https://github.com/cdrxyz/llm-queue-proxy.git
+cd llm-queue-proxy
+./bin/hermit install python3@3.13   # first time; downloads Hermit + CPython
+./bin/python3 -m pip install -e .
+```
+
+Or any 3.11+ interpreter:
+
+```bash
+pip install -e .
+```
+
+## Run
+
+```bash
+llm-queue-proxy \
+  --listen 127.0.0.1:8001 \
+  --upstream http://127.0.0.1:8000 \
+  --slots 1 \
+  --max-queue 32 \
+  --queue-timeout 600
+```
+
+Env equivalents: `LLM_QUEUE_LISTEN`, `LLM_QUEUE_UPSTREAM`, `LLM_QUEUE_SLOTS`, `LLM_QUEUE_MAX_QUEUE`, `LLM_QUEUE_TIMEOUT`.
+
+Binds **localhost only**. `0.0.0.0` / `::` are refused.
+
+### Hermes + MTPLX
+
+Keep MTPLX on `127.0.0.1:8000`. Point the Hermes provider at the proxy:
+
+```yaml
+providers:
+  mtplx:
+    api: http://127.0.0.1:8001/v1
+    name: MTPLX
+    api_key: mtplx
+    default_model: qwen3.8-flash-next-mtplx
+```
+
+Same change on `custom_providers[].base_url`.
+
+### Admitted paths (POST only)
+
+`/v1/chat/completions`, `/v1/completions`, `/v1/messages`, `/v1/responses`, and the unprefixed `/chat/completions`, `/completions`, `/messages`, `/responses`.
+
+Override with repeated `--admit /your/path`.
+
+### Proxy health
+
+`GET /__queue/health` → `{ ok, in_flight, queued, slots, max_queue, upstream }` (not forwarded).
+
+## Development
+
+```bash
+. ./bin/activate-hermit          # optional; or call ./bin/python3 directly
+./bin/python3 -m pip install -e ".[dev]"
+./bin/python3 -m pytest
+./bin/python3 -m ruff check src tests
+```
+
+## Architecture
+
+```
+Hermes / other clients
+        |
+        v
+ llm-queue-proxy :8001
+   GET /health, /v1/models  ---- pass through ----.
+   POST /v1/chat/completions -- FIFO slot (1) ----+--> MTPLX :8000
+   extra POSTs wait (or 429 / 503)
+```
+
+Stdlib only at runtime (`http.server` + `http.client`). No third-party proxy framework.
+
+## License
+
+Apache-2.0. See [LICENSE](LICENSE).

bin/.python3@3.13.pkg

@@ -0,0 +1 @@
+hermit

bin/README.hermit.md

@@ -0,0 +1,7 @@
+# Hermit environment
+
+This is a [Hermit](https://github.com/cashapp/hermit) bin directory.
+
+The symlinks in this directory are managed by Hermit and will automatically
+download and install Hermit itself as well as packages. These packages are
+local to this environment.

bin/activate-hermit

@@ -0,0 +1,21 @@
+#!/bin/bash
+# This file must be used with "source bin/activate-hermit" from bash or zsh.
+# You cannot run it directly
+#
+# THIS FILE IS GENERATED; DO NOT MODIFY
+
+if [ "${BASH_SOURCE-}" = "$0" ]; then
+  echo "You must source this script: \$ source $0" >&2
+  exit 33
+fi
+
+BIN_DIR="$(dirname "${BASH_SOURCE[0]:-${(%):-%x}}")"
+if "${BIN_DIR}/hermit" noop > /dev/null; then
+  eval "$("${BIN_DIR}/hermit" activate "${BIN_DIR}/..")"
+
+  if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ]; then
+      hash -r 2>/dev/null
+    fi
+
+    echo "Hermit environment $("${HERMIT_ENV}"/bin/hermit env HERMIT_ENV) activated"
+fi

bin/activate-hermit.fish

@@ -0,0 +1,24 @@
+#!/usr/bin/env fish
+
+# This file must be sourced with "source bin/activate-hermit.fish" from Fish shell.
+# You cannot run it directly.
+#
+# THIS FILE IS GENERATED; DO NOT MODIFY
+
+if status is-interactive
+    set BIN_DIR (dirname (status --current-filename))
+
+    if "$BIN_DIR/hermit" noop > /dev/null
+        # Source the activation script generated by Hermit
+        "$BIN_DIR/hermit" activate "$BIN_DIR/.." | source
+
+        # Clear the command cache if applicable
+        functions -c > /dev/null 2>&1
+
+        # Display activation message
+        echo "Hermit environment $($HERMIT_ENV/bin/hermit env HERMIT_ENV) activated"
+    end
+else
+    echo "You must source this script: source $argv[0]" >&2
+    exit 33
+end

bin/hermit

@@ -0,0 +1,43 @@
+#!/bin/bash
+#
+# THIS FILE IS GENERATED; DO NOT MODIFY
+
+set -eo pipefail
+
+export HERMIT_USER_HOME=~
+
+if [ -z "${HERMIT_STATE_DIR}" ]; then
+  case "$(uname -s)" in
+  Darwin)
+    export HERMIT_STATE_DIR="${HERMIT_USER_HOME}/Library/Caches/hermit"
+    ;;
+  Linux)
+    export HERMIT_STATE_DIR="${XDG_CACHE_HOME:-${HERMIT_USER_HOME}/.cache}/hermit"
+    ;;
+  esac
+fi
+
+export HERMIT_DIST_URL="${HERMIT_DIST_URL:-https://github.com/cashapp/hermit/releases/download/stable}"
+HERMIT_CHANNEL="$(basename "${HERMIT_DIST_URL}")"
+export HERMIT_CHANNEL
+export HERMIT_EXE=${HERMIT_EXE:-${HERMIT_STATE_DIR}/pkg/hermit@${HERMIT_CHANNEL}/hermit}
+
+if [ ! -x "${HERMIT_EXE}" ]; then
+  echo "Bootstrapping ${HERMIT_EXE} from ${HERMIT_DIST_URL}" 1>&2
+  INSTALL_SCRIPT="$(mktemp)"
+  # This value must match that of the install script
+  INSTALL_SCRIPT_SHA256="09ed936378857886fd4a7a4878c0f0c7e3d839883f39ca8b4f2f242e3126e1c6"
+  if [ "${INSTALL_SCRIPT_SHA256}" = "BYPASS" ]; then
+    curl -fsSL "${HERMIT_DIST_URL}/install.sh" -o "${INSTALL_SCRIPT}"
+  else
+    # Install script is versioned by its sha256sum value
+    curl -fsSL "${HERMIT_DIST_URL}/install-${INSTALL_SCRIPT_SHA256}.sh" -o "${INSTALL_SCRIPT}"
+    # Verify install script's sha256sum
+    openssl dgst -sha256 "${INSTALL_SCRIPT}" | \
+      awk -v EXPECTED="$INSTALL_SCRIPT_SHA256" \
+      '$2!=EXPECTED {print "Install script sha256 " $2 " does not match " EXPECTED; exit 1}'
+  fi
+  /bin/bash "${INSTALL_SCRIPT}" 1>&2
+fi
+
+exec "${HERMIT_EXE}" --level=fatal exec "$0" -- "$@"

bin/hermit.hcl

@@ -0,0 +1,2 @@
+github-token-auth {
+}

bin/pip

@@ -0,0 +1 @@
+.python3@3.13.pkg

bin/pip3

@@ -0,0 +1 @@
+.python3@3.13.pkg

bin/pip3.13

@@ -0,0 +1 @@
+.python3@3.13.pkg

bin/pydoc3

@@ -0,0 +1 @@
+.python3@3.13.pkg

bin/pydoc3.13

@@ -0,0 +1 @@
+.python3@3.13.pkg

bin/python

@@ -0,0 +1 @@
+.python3@3.13.pkg

bin/python3

@@ -0,0 +1 @@
+.python3@3.13.pkg

bin/python3-config

@@ -0,0 +1 @@
+.python3@3.13.pkg

bin/python3.13

@@ -0,0 +1 @@
+.python3@3.13.pkg

bin/python3.13-config

@@ -0,0 +1 @@
+.python3@3.13.pkg

pyproject.toml

@@ -0,0 +1,49 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "llm-queue-proxy"
+version = "0.1.0"
+description = "FIFO admission proxy for local LLM servers: one in-flight generation, queue the rest."
+readme = "README.md"
+license = { file = "LICENSE" }
+requires-python = ">=3.11"
+authors = [{ name = "Cedar Labs", email = "hello@cdr.xyz" }]
+keywords = ["llm", "proxy", "openai", "mtplx", "mlx", "queue"]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "License :: OSI Approved :: Apache Software License",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Topic :: Internet :: Proxy Servers",
+]
+dependencies = []
+
+[project.urls]
+Homepage = "https://github.com/cdrxyz/llm-queue-proxy"
+Issues = "https://github.com/cdrxyz/llm-queue-proxy/issues"
+Source = "https://github.com/cdrxyz/llm-queue-proxy"
+
+[project.scripts]
+llm-queue-proxy = "llm_queue_proxy.cli:main"
+
+[project.optional-dependencies]
+dev = ["pytest>=8.0", "ruff>=0.6"]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+pythonpath = ["src"]
+testpaths = ["tests"]
+addopts = "-q"
+
+[tool.ruff]
+target-version = "py311"
+line-length = 100
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B"]

src/llm_queue_proxy/__init__.py

@@ -0,0 +1,3 @@
+"""FIFO admission proxy for local LLM HTTP servers."""
+
+__version__ = "0.1.0"

src/llm_queue_proxy/__main__.py

@@ -0,0 +1,4 @@
+from llm_queue_proxy.cli import main
+
+if __name__ == "__main__":
+    raise SystemExit(main())