comando

Fine-tuning a small open model to parse Italian smart-home commands into structured device actions — and measuring, on a frozen eval set, whether it can beat frontier-model prompting on accuracy, latency, and cost.

Status: G4 — done. All four gates (schema, frozen eval, baselines, LoRA) are green. See PLAN.md for the gated roadmap and PROGRESS.md for the session-by-session log.

The task

"abbassa le luci in soggiorno al 30% e poi metti il riscaldamento a 21 gradi"
→ {"actions": [
     {"intent": "light_dim",   "room": "soggiorno", "value": 30, "unit": "percent"},
     {"intent": "climate_set", "value": 21, "unit": "celsius"}
   ]}

12 intents (lights, climate, shutters, scenes, alarm, state queries, and out_of_scope for utterances that must be refused), multi-action utterances, strict slot validation — a schema-constrained NLU task small enough to fine-tune on a laptop and real enough to be worth measuring. I run a smart-home company (Dymora), so the task is native, but this repo shares no code or data with it.

Why

Production LLM systems often use a frontier model for a task a 4B model could own at a fraction of latency and cost — if you can prove parity with evals you trust. This repo is that proof workflow, end to end: synthetic data generation with label-preserving paraphrase, a hand-verified frozen eval set, prompted frontier baselines, LoRA fine-tuning on Apple Silicon (MLX), and a published comparison — including the ablation that matters (how little data is enough?).

Results

All numbers on data/eval_v0.jsonl — 400 examples, 100% hand-verified, frozen, touched exactly once per model (see EVAL_FREEZE.md). Identical system prompt template and strict shared parser for every model; no provider-specific structured-output modes — the comparison measures models, not tooling.

Model	exact-match	action F1	invalid rate	cost / 1k commands
gpt-5-mini (prompted, 8-shot)	90.25%	91.72%	0.0%	$0.204
claude-haiku-4-5 (prompted, 8-shot)	88.0%	89.81%	0.0%	$0.861
Qwen3-4B-Instruct-2507-4bit (prompted, 8-shot, local)	68.75%	73.80%	5.5%	$0
Qwen3-4B-Instruct-2507-4bit + LoRA (this repo)	98.5%	98.73%	0.0%	$0

The fine-tuned model beats the strongest prompted baseline (gpt-5-mini) outright — not just "within noise of," 8.25pp better — while running free and local. Latency: LoRA model p50 1.31s / p95 1.88s on an M3 Pro (no batching, no speculative decoding, no serving optimization — this is mlx_lm.generate in a for loop).

Ablation: how much data does it take?

LoRA on Qwen3-4B, rank 8, batch 4, lr 1e-4, 1 epoch, completion-only loss (--mask-prompt), evaluated on data/dev.jsonl (186 examples, held out, never frozen — see below). The three sizes are nested prefixes of one shuffle (500 ⊂ 1500 ⊂ 5000), not independent samples, so this is a clean "does more data help" curve, not a noisier "does more diversity help" curve:

Train size	exact-match	action F1	invalid rate
500	73.12%	76.61%	1.61%
1500	95.70%	96.76%	0.54%
5000	94.62%	95.61%	0.0%

1500 examples beats 5000, despite 5000's training set being a strict superset. This wasn't the expected result. Both runs used the same fixed epoch count (1 pass over the training set), so 5000 saw ~3.3× more gradient steps but each pattern repeated proportionally less. My best untested hypothesis: at a fixed 1-epoch budget, the smaller set's higher per-example repetition rate matters more than 5000's higher surface diversity for a task this structured (closed intent/slot vocabulary). A matched-epochs-vs-matched-iters ablation would confirm this, and I haven't run it — flagged here rather than hidden. 1500 was picked (best dev score) for the one production eval_v0 run — the eval_v0 headline number above is the 1500-example adapter, evaluated exactly once, per the freeze policy.

Failure taxonomy

What each model actually gets wrong, not just the aggregate score — the more useful number for deciding whether to trust a fine-tune.

gpt-5-mini and claude-haiku-4-5 fail on the same thing, and it isn't reasoning. 33 of gpt-5-mini's 39 misses (85%), and 44 of claude-haiku-4-5's 48 misses (92%), are the same root cause: Italian has multiple synonyms for a room ("soggiorno" / "salotto" / "sala" all mean living room) that the dataset canonicalizes to one slug (soggiorno). Both frontier models, given only the system prompt's one-line instruction and 8 few-shot examples, echo the surface synonym back literally instead of the canonical slug:

"accendi la luce in sala" → gold: {"room": "soggiorno"} → gpt-5-mini: {"room": "sala"}

This is not a reasoning failure — it's a coverage failure. Eight few-shot examples can't teach an idiosyncratic normalization table; thousands of fine-tuning examples can, implicitly, without it ever being stated. This is the actual mechanism behind the LoRA win, not "the small model got smarter than the frontier model."

The unfine-tuned local Qwen3-4B mostly fails on schema shape, not semantics. 22 of its 400 predictions (5.5%) are invalid JSON — but they're not garbage, they're {"intent": "out_of_scope"} instead of the required {"actions": [{"intent": "out_of_scope"}]}. The model identifies the right answer and gets the wrapper wrong. Fine-tuning fixes this completely (0% invalid rate) — plausibly the easiest part of the whole gap to close.

The fine-tuned model's 6 residual misses on eval_v0 split into two categories, both genuinely interesting:

1. Room-carryover in multi-action commands (4/6). "accendi la luce e chiudi le serrande del soggiorno" — gold keeps light_on roomless (only "le serrande" is modified by "del soggiorno") and puts room: "soggiorno" only on shutter_close. The model puts soggiorno on both actions. I'm not fully convinced the gold label is the more useful answer here, even though it's the more grammatically precise one — a real user probably does mean both actions to apply to the living room. This is a dataset-design convention (each action's room comes from independent grammar slot-filling, not cross-clause coreference) surfacing as a "model error," and it's worth being honest that the convention itself is debatable.
2. Terse scene invocation for a scene the model saw during training, but never terse (2/6). "modalità lettura" (bare) and "metti la modalità lettura" both get predicted as out_of_scope instead of scene_activate(scene="lettura") — despite the 1500-example training subset containing 29 "lettura" examples. Checking those 29: every single one is elaborated ("Attiva modalità lettura in salotto per 30 min", "Metti il soggiorno in modalità lettura: luci al 30% e temperatura 22°C..."). None are the bare 2-4 word form. The model learned "lettura" in context, not the terse invocation pattern — a real generalization gap, not a labeling dispute.

Method (summary)

1. Grammar (src/comando/grammar.py) — seeded templates emit clean (utterance, gold parse) pairs; reproducible from (n, seed).
2. Paraphrase (src/comando/paraphrase.py) — an LLM rewrites surface text only (colloquialisms, ellipsis, typos, dialect flavor); gold labels are never touched and every pair is re-validated after rewrite.
3. Frozen eval (src/comando/review.py, EVAL_FREEZE.md) — 639 candidates generated, 100% hand-reviewed (k/d, single annotator), 586 kept, first 400 frozen to data/eval_v0.jsonl, remaining 186 split to data/dev.jsonl for iteration. Frozen file never edited after freeze; corrections would go to a separate audit file.
4. Baselines (src/comando/baselines.py, src/comando/metrics.py) — gpt-5-mini and claude-haiku-4-5 with strict shared parsing (no provider structured-output modes); Qwen3-4B-Instruct prompted, 8-shot, local via mlx-lm.
5. LoRA (src/comando/lora_data.py + mlx_lm.lora) — Qwen3-4B on an M3 Pro (18GB unified memory); success criterion fixed before training (PLAN.md G3: beat Baseline A on exact-match, or land within 2pp at ≥10× lower cost). Fine-tuned model gets the same system prompt as the baselines minus the few-shot block — the adapter is trained on exactly that zero-shot prompt, so train/inference distributions match exactly.

Limitations (honest, not buried)

• Single annotator. All 586 kept eval/dev labels were reviewed by one person (me), once. No second pass, no inter-annotator agreement number.
• out_of_scope is thin. ~5% of eval_v0. Enough to establish the near-domotic traps (TV, oven, locks, gates) are handled, not enough to bound the false-refusal rate tightly.
• The 5000 < 1500 ablation result is unexplained. Flagged above, not resolved — a fixed-iters-vs-fixed-epochs follow-up would clarify it.
• The room-carryover convention in multi-action gold labels is debatable — see failure taxonomy above. It's applied consistently (so the eval is internally fair), but it may not be the label a real product would want.
• dev.jsonl is 186, not the ~200 originally targeted — leftover pool after the 400-example freeze cut, not a fresh independent draw.
• Recurring paraphrase artifacts were dropped by hand, not fixed at the source. A bare "di" + article contraction bug ("di le" instead of "delle", etc.) hit repeatedly during review; worth fixing in paraphrase.py/grammar.py before any eval_v1.
• No serving optimization. Latency numbers are mlx_lm.generate in a loop — no batching, no continuous batching server, no speculative decoding. A real deployment would be faster than these numbers, on both sides of the comparison.
• Qwen3's thinking mode was never explicitly disabled. Every local prediction is wrapped in an empty <think>\n\n</think>\n\n block — harmless (the parser strips it) but a few free tokens of latency left on the table; enable_thinking=False in the chat template is untested.
• This is a v0. Non-goals (device-level targeting, time/scheduling slots, actual serving) are listed in PLAN.md.

Layout

src/comando/schema.py      strict Pydantic target schema (invalid = invalid, no repair)
src/comando/grammar.py     seeded template generator
src/comando/paraphrase.py  label-preserving LLM surface rewriting
src/comando/dataset.py     train/eval-candidate set construction, dedup, seeds
src/comando/review.py      resumable k/d hand-verification CLI (G1 freeze gate)
src/comando/baselines.py   prompted baselines + adapter eval, one runner for every model
src/comando/metrics.py     exact-match / action-F1 / invalid-rate, strict shared parser
src/comando/lora_data.py   train/dev → mlx-lm chat-format LoRA training data
tests/                     schema rules, generator invariants, metrics, baselines, review
PLAN.md                    gated roadmap, non-goals, success criteria (fixed before results)
docs/PROGRESS.md           session-by-session status log
EVAL_FREEZE.md             frozen eval set policy, provenance, checksum
reports/                   per-model JSON scorecards + raw prediction dumps

Reproduce

uv sync --extra gen-openai --extra eval-anthropic --extra gen-mlx   # all extras together, or uv drops the others
uv run pytest -q

# regenerate data (optional — data/generated/ is gitignored, eval_v0.jsonl and dev.jsonl are not)
uv run python -m comando.dataset --out data/generated --backend openai
uv run python -m comando.review data/generated/eval_candidates.jsonl   # hand-verify, k/d

# baselines (needs OPENAI_API_KEY / ANTHROPIC_API_KEY in .env — personal keys, gitignored)
set -a && source .env && set +a
uv run python -m comando.baselines --model gpt-5-mini --eval data/eval_v0.jsonl
uv run python -m comando.baselines --model claude-haiku-4-5 --eval data/eval_v0.jsonl
uv run python -m comando.baselines --model mlx-community/Qwen3-4B-Instruct-2507-4bit --eval data/eval_v0.jsonl

# LoRA
uv run python -m comando.lora_data --sizes 500,1500,5000
uv run mlx_lm.lora --model mlx-community/Qwen3-4B-Instruct-2507-4bit \
  --train --data runs/lora_data/1500 --fine-tune-type lora --mask-prompt \
  --batch-size 4 --iters 375 --learning-rate 1e-4 --adapter-path adapters/1500 --seed 0
uv run python -m comando.baselines --model mlx-community/Qwen3-4B-Instruct-2507-4bit \
  --adapter adapters/1500 --eval data/eval_v0.jsonl