Environment
- FreeToken v0.1.2 (pip), daemon + engine (
POST /engine/start, engine on :1919)
- WSL2 Ubuntu on Windows 11, RTX 5060 Ti 16 GB, driver 610.62, CUDA 13.3 toolkit
- Model:
qwen3.6-35b-a3b-nvfp4 (35B MoE, NVFP4), --memory-ratio 0.7
- Observed 2026-08-29; the fatal code path below is still present on
main today
What happened
After a machine reboot, one specific client's requests crashed the engine 100% reproducibly, while byte-similar probe requests from curl survived. It looked exactly like "application X kills the engine" and cost hours of client-side bisecting (streaming vs not, gateway vs direct, system prompt, max_tokens, disconnects — everything survived except the one exact request shape).
The actual cause was on disk: the reboot had cut a Triton JIT cache write mid-file, leaving a truncated JSON metadata file in ~/.triton/cache. Any request whose prompt/batch shape mapped to that kernel-cache key died loading the kernel; every other shape never touched the poisoned key.
Crash (from the serve log)
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
... in triton/runtime/cache.py, get_group (during topk / MoE-router kernel load)
Backend supervisor: backend worker freetoken-TP0-scheduler exited
Backend worker is gone and cannot be restarted; stopping the API server
Why this class is nastier than a normal crash
- Auto-restart can never heal it. The poison lives on disk, so the daemon's
--auto-restart reloads the engine (~4–5 min for a 22 GB model) and it dies again on the next matching request — an infinite crash-loop with a cold load per cycle.
- Each cycle re-maps ~22 GB of weights through WSL RAM, so the loop also thrashes the host while it spins.
- It is invisible to health checks and to every other client — the engine looks healthy until the one poisoned shape arrives.
Repro
- Serve a MoE model; send a few requests so Triton JIT kernels compile and cache.
- Truncate one of the JSON metadata files under
~/.triton/cache/<key>/ to zero bytes (equivalent to a power cut mid-write).
- Send a request whose shape maps to that kernel key → worker exits as above;
--auto-restart reload → same request kills it again.
Verified workaround
rm -rf ~/.triton/cache — after the clear, the previously-fatal byte-identical request streamed 241 chunks to completion. (Downstream we now automate this: our launcher clears the cache after an unclean shutdown, and a log-watching breaker clears it once when it sees ≥3 worker deaths inside 15 min — a threshold a healthy engine can't reach, since one cold load takes longer.)
Suggested fixes
- Treat unparseable cache metadata as a cache miss. The exception originates in Triton's
runtime/cache.py get_group, so the root fix may belong upstream in Triton — but FreeToken could defensively catch cache/JSON errors during kernel load and recompile (or clear the offending entry and retry once) instead of letting the scheduler worker die.
- Don't let a kernel-load failure take down the API server.
python/freetoken/server/api_server.py (_exit_after_backend_death, ~line 87 on main) turns a dead worker into SIGTERM for the whole server. A kernel-load failure is request-shape-scoped — surfacing it as a request-level error (500 for that request, engine stays up for every other shape) would turn an unhealable outage into a visible, debuggable error.
Happy to provide more detail from our logs if useful.
Environment
POST /engine/start, engine on :1919)qwen3.6-35b-a3b-nvfp4(35B MoE, NVFP4),--memory-ratio 0.7maintodayWhat happened
After a machine reboot, one specific client's requests crashed the engine 100% reproducibly, while byte-similar probe requests from curl survived. It looked exactly like "application X kills the engine" and cost hours of client-side bisecting (streaming vs not, gateway vs direct, system prompt, max_tokens, disconnects — everything survived except the one exact request shape).
The actual cause was on disk: the reboot had cut a Triton JIT cache write mid-file, leaving a truncated JSON metadata file in
~/.triton/cache. Any request whose prompt/batch shape mapped to that kernel-cache key died loading the kernel; every other shape never touched the poisoned key.Crash (from the serve log)
Why this class is nastier than a normal crash
--auto-restartreloads the engine (~4–5 min for a 22 GB model) and it dies again on the next matching request — an infinite crash-loop with a cold load per cycle.Repro
~/.triton/cache/<key>/to zero bytes (equivalent to a power cut mid-write).--auto-restartreload → same request kills it again.Verified workaround
rm -rf ~/.triton/cache— after the clear, the previously-fatal byte-identical request streamed 241 chunks to completion. (Downstream we now automate this: our launcher clears the cache after an unclean shutdown, and a log-watching breaker clears it once when it sees ≥3 worker deaths inside 15 min — a threshold a healthy engine can't reach, since one cold load takes longer.)Suggested fixes
runtime/cache.py get_group, so the root fix may belong upstream in Triton — but FreeToken could defensively catch cache/JSON errors during kernel load and recompile (or clear the offending entry and retry once) instead of letting the scheduler worker die.python/freetoken/server/api_server.py(_exit_after_backend_death, ~line 87 onmain) turns a dead worker into SIGTERM for the whole server. A kernel-load failure is request-shape-scoped — surfacing it as a request-level error (500 for that request, engine stays up for every other shape) would turn an unhealable outage into a visible, debuggable error.Happy to provide more detail from our logs if useful.