Repository navigation
--spec-type ngram-* silently no-ops on Ternary-Bonsai-2 (hybrid GDN): flag accepted, never activates, zero logging #203
Description
Summary
--spec-type ngram-* is silently ignored when serving prism-ml/Ternary-Bonsai-2-27B-PQ2_0.gguf (hybrid linear-attention/GDN target): the flag is accepted without any error or warning, but speculation never activates — no log lines at all, timings.draft_n is null in every response, and decode speed is identical to --spec-type none.
Environment
- Apple M4 Max (64 GB), macOS 15 (darwin 25.6.0, arm64)
- This fork, build
9a9394a(version: 0.2.0-dev (build 1)), Metal backend,-ngl 999 -fa on - Model:
Ternary-Bonsai-2-27B-PQ2_0.gguf
Repro
llama-server -m Ternary-Bonsai-2-27B-PQ2_0.gguf \
-c 49152 -np 1 -ngl 999 -fa on --jinja \
--spec-type ngram-mod --spec-ngram-mod-n-match 32 \
--spec-ngram-mod-n-min 16 --spec-ngram-mod-n-max 64Observed on every request: timings.draft_n == null, decode ≈ 31.7 tok/s (identical to no-spec baseline), and grep -iE "spec|ngram|draft" over the server log returns nothing.
Control experiment
On the same build and same target model, --spec-type draft-simple -md Qwen3.5-4B-Q4_K_M.gguf does activate (e.g. draft_n=274, draft_n_accepted=25 on a Chinese prompt), so the speculative pipeline itself works for the hybrid target when driven by a draft model — only the n-gram implementations silently no-op.
Ask
- Please emit a startup warning/error (or at least one log line) when a requested
--spec-typecannot be used for the loaded model — silent no-op cost us hours to diagnose. - Are n-gram speculative implementations planned for hybrid (linear-attention/GDN) targets, e.g. via per-draft state snapshot/rollback?
- Recommended speculative setup for Ternary-Bonsai-2 on Metal? With a generic same-family 4B drafter we measure ~9% (Chinese) / ~52% (code) acceptance and a net slowdown (5.7–12.2 tok/s vs 31.7 baseline), so a first-party or n-gram option would be very valuable on Apple Silicon.
Thanks!
Activity
lyb618888-bojian commentedon Sep 19, 2026
Follow-up data for question 3 (recommended speculative setup on Metal), measured on the same M4 Max / build 9a9394a /
Ternary-Bonsai-2-27B-PQ2_0.gguf(Chinese narrative + code-rewrite probes, ~96-120 completion tokens each):ngram-moddraft-simple+ Qwen3.5-4B-Q4_K_M drafterdraft-mtp+ community grafted MTP file (decent-jawfishBonsai-2-27B-PQ2_0-MTP.gguf), n-max 3Notes:
Hadamard-latent table 'token_embd.weight' is read without the inverse transform; the community's 14-line fix (apply the Hadamard inverse after the MTP embedding lookup insrc/models/qwen35.cpp) was hand-applied and makes it load — that fix looks like it belongs upstream.zhaoyilun commentedon Sep 19, 2026
Independent corroboration on a different backend — same symptom, CUDA instead of Metal.
ProCreations/Ternary-Bonsai-2-27B-MTP(PQ2_0 + one MTP layer in the same file).--spec-type ngram-simple(12-prompt harness, temp 0, top_k 1, seed 7, n_predict 128,-c 262144 -ctk q4_0 -ctv q4_0): 72.2 / 71.8 / 72.7 t/s on a highly repetitive prompt across three consecutive runs, versus 72.7 t/s with--spec-type none. I had briefly read a one-off +11% as a win; three repeats on the identical config showed it was GPU-state drift, not speculation.timingsnever containeddraft_n/draft_n_acceptedin the ngram arm — no accounting, no logging, no activation.What makes this look like an ngram-specific gap rather than a broken accounting path: with
--spec-type draft-mtpon the same model/build,timings.draft_n/draft_n_acceptedare populated normally and decode goes to ~90 t/s median (1.34x, aggregate acceptance 68.1% over 803/1180 drafted tokens). So the counters work on this target; it is specifically thengram-*types that never populate.Harness + raw JSON: https://github.com/zhaoyilun/bonsai2-27b-mtp-repro (MTP A/B in
results/, the ngram arm was run withscripts/mtp_bench.py).bri-prism commentedon Sep 24, 2026
Thanks for the report and the follow-up measurements, and to @zhaoyilun for confirming it on CUDA. We haven't pinned down why the n-gram types never draft on this model yet, but you're right that the server should say so instead of accepting the flag silently. We'll look at both.
On what works today: the MTP embedding fix you hand-applied is now merged as #205 and is in the new release prism-b10735-842b188, so MTP files load without patching. Your numbers matter here too: on Metal even 75% acceptance doesn't beat plain decode for this model yet, so plain decode is the fastest option on Apple Silicon for now.
bri-prism commentedon Sep 25, 2026
Follow-up with a diagnosis. PR #272 makes this visible.
The n-gram speculators do work on the hybrid GDN target. I reproduced with a qwen35 2B (same hybrid GDN architecture as Bonsai 2 27B), both on current
prismand on9a9394a, the build in the original report.ngram-simpledrafts right away (e.g.draft_n 96 / accepted 32on a repetitive 200-token reply), and so doesngram-modwith default settings.What happened with
--spec-ngram-mod-n-match 32 --spec-ngram-mod-n-min 16 --spec-ngram-mod-n-max 64:ngram-modneeds a 32-token span to repeat before it can propose anything. On ~100–300-token replies that never occurs, so nothing is drafted. The same flags givedraft_n 361 / accepted 239(66 %) once the reply reaches 600 tokens. It behaves the same on a dense model, so it isn't specific to GDN.It looked disabled for two reasons, which #272 fixes:
grep specfound nothing. It's now an info line:speculative decoding enabled: ngram-mod.timings.draft_nwas omitted when zero. It's now reported as0whenever speculation is on.For short chat replies,
ngram-simpleorngram-modwith a smalln-match(the default) are the settings that can actually draft. @zhaoyilun, yourngram-simpleresult on b10683 with the MTP-grafted file I couldn't reproduce here (no CUDA box or that file). With #272 you'd seedraft_n: 0plus the startup line, which would tell "enabled, no matches" apart from "never enabled".server: make speculative decoding visible when it is enabled but not …
bri-prism commentedon Sep 26, 2026
#272 is in release prism-b10743-adfffbe. With
--spec-type ngram-*, the server now logsspeculative decoding enabled: <types>at startup and always reportstimings.draft_n/draft_n_accepted, as0when nothing was drafted. That makes "enabled but no n-gram matches" distinguishable from "not enabled". If you rerun your setups on this build,draft_nwill show which case you're in. @zhaoyilun, that would settle thengram-simple/ b10683 case I couldn't reproduce.