Skip to content

PROGRESS.md

Latest commit

 

History

History
86 lines (62 loc) · 7.5 KB

PROGRESS.md

File metadata and controls

86 lines (62 loc) · 7.5 KB
Raw

PROGRESS — status log

Updated 2026-08-11. This file is the session handoff: current gate status, exact next commands, open points. See PLAN.md for gates and success criteria.

Gate status

Gate Status Notes
G0 — schema + grammar ✅ GREEN 12 intents incl. out_of_scope, multi-action, seeded grammar
G1 — dataset + frozen eval ✅ GREEN 639 verified (586 kept), eval_v0.jsonl (400) frozen + dev.jsonl (186) split, see EVAL_FREEZE.md
G2 — baselines ✅ GREEN 3 models run on eval_v0.jsonl, see table below and reports/
G3 — LoRA fine-tune ✅ GREEN 98.5% exact-match on eval_v0 — beats Baseline A outright, success criterion met
G4 — writeup + publish ✅ GREEN README rewritten as the report; pushed; repo public at https://github.com/comprido96/comando

Tests: 40/40 green (uv run pytest -q).

G2 baseline results (data/eval_v0.jsonl, n=400)

Model exact_match action_f1 invalid_rate cost
gpt-5-mini 90.25% 91.72% 0.0% $0.08
claude-haiku-4-5 88.0% 89.81% 0.0% $0.34
Qwen3-4B-Instruct-2507-4bit (local, prompted) 68.75% 73.80% 5.5% $0

Local Qwen well behind both API models, with a real invalid-JSON rate the API models never hit — the gap G3's LoRA fine-tune targets.

G3 LoRA results

Ablation on data/dev.jsonl (n=186) — sizes are nested prefixes of one shuffle (500 ⊂ 1500 ⊂ 5000), not independent samples:

Train size exact_match action_f1 invalid_rate
500 73.12% 76.61% 1.61%
1500 95.70% 96.76% 0.54%
5000 94.62% 95.61% 0.0%

5000 scoring below 1500 despite being a strict superset is the interesting non-monotonic result for the writeup — not sampling noise, since it's a nested subset relationship. Candidate hypothesis: fixed-epoch-count training means 5000 sees the same 1 epoch as 1500 but with more surface diversity per gradient step, vs 1500 getting more repetition of a smaller pattern set — needs a real ablation (e.g. matched-iters vs matched-epochs) to confirm, flagged for G4.

Picked 1500 (best dev exact-match) for the one-shot eval_v0.jsonl run, per the "touched once" policy:

Model exact_match action_f1 invalid_rate cost
gpt-5-mini (baseline A) 90.25% 91.72% 0.0% $0.08
Qwen3-4B + LoRA (1500) 98.5% 98.73% 0.0% $0

Success criterion (PLAN.md G3, fixed before training) met outright: beats Baseline A on exact-match, at $0 marginal cost.

Training config (uv run mlx_lm.lora): rank 8 (default), 16 layers, batch 4, lr 1e-4, 1 epoch (iters = size/4), --mask-prompt (completion-only loss on the JSON target), seed 0. Peak mem 9.5 GB — safely under the 18 GB M3 Pro budget (batch 16 was tried first and hit 26 GB / thrashed; reverted). dev.jsonl served as mlx-lm's internal validation set during training (loss monitoring only — no early stopping, no selection bias) and as the ablation/model-selection set after.

Fine-tuned model's system prompt: same style as the prompted baselines, minus the few-shot block (comando.lora_data.SYSTEM_PROMPT = build_system_prompt([])) — the adapter is trained on this exact zero-shot prompt, and comando.baselines --adapter ... automatically drops few-shot too, so train/inference distributions match exactly. Resolves the open point from G1/G2.

Dataset facts (current artifacts in data/generated/, gitignored)

  • train.jsonl: 7,149 examples (grammar + gpt-5-mini paraphrases, label-guard filtered, deduped)
  • eval_candidates.jsonl: 639 examples, zero overlap with train (tested invariant)
  • out_of_scope: 34 eval / 309 train — includes near-domotic traps (TV, oven, locks, gates)
  • Seeds: TRAIN_SEED=1000, EVAL_SEED=2000, FEW_SHOT_SEED=3000 — never reuse across sets
  • Paraphrase backend: gpt-5-mini with reasoning_effort="minimal" (2.6× faster, 3× cheaper than default — measured). Total API spend so far ≈ $3

Next steps, in order

  1. Hand-verify eval candidates done — 586/639 kept.
  2. Freeze eval_v0 done — data/eval_v0.jsonl (400, committed) + EVAL_FREEZE.md. data/dev.jsonl (186) split from leftover verified pool, short of ~200 target (see open points).
  3. G2 baselines done — all 3 models run, see results table above. uv sync needs all three extras together (--extra gen-openai --extra eval-anthropic --extra gen-mlx) or it drops the others.
  4. G3 LoRA done — see results table above. Adapters in adapters/{500,1500,5000}/ (gitignored, reproducible via comando.lora_data + the training config documented above).
  5. G4 writeup done — README rewritten as the report: results table, ablation curve, failure taxonomy (dominant frontier-baseline error is room-synonym canonicalization, not reasoning — 85%/92% of gpt-5-mini/claude-haiku misses; LoRA's 6 residual eval_v0 misses split into a dataset-convention tension and a real terse-phrasing generalization gap), honest limitations.
  6. Publish done — pushed, repo flipped to public by Federico. All 4 gates green.
  7. Link from profile README done — added a bullet + Highlights entry in comprido96/comprido96.

Decisions made (don't relitigate)

  • No provider strict-output modes — identical prompt + strict shared parser for all models; comparison measures models, not tooling
  • Paraphraser may only change surface text — labels copied, never re-derived; conservative lexical guard; numbers stay digits
  • Wrong eval labels are dropped, not edited (single-annotator edit risk)
  • Grammar prepositions are crude by design ("del studio") — paraphrase smooths, hand-verification gates the frozen set
  • Employer-clean is absolute: no TomatoAI data, no TomatoAI API keys (personal keys only, in .env, gitignored)
  • Repo was private until G4 results were real — now public at https://github.com/comprido96/comando, linked from the profile README

Open points

  • out_of_scope share in eval (~5%) is thin — acceptable for v0, note in G4 limitations
  • Pricing constants in baselines.py re-verification done, 2026-08-11: gpt-5-mini $0.25/$2.00 per MTok confirmed via developers.openai.com/api/docs/pricing; claude-haiku-4-5 $1/$5 per MTok confirmed via platform.claude.com official pricing docs. Both match the existing constants exactly — no code change needed
  • data/dev.jsonl is 186, short of ~200 target (leftover pool after 400-cut eval_v0) — top up from fresh generation before G3 if needed, not a G2 blocker
  • Found several recurring paraphrase bugs during hand-verification, worth fixing at the source (paraphrase.py/grammar.py) before any future eval_v1 cut: bare "di" + article not contracted ("di le" / "di il" / "di l'" instead of "delle"/"del"/"dell'"); some declarative statements ("le luci sono accese") mislabeled query_state instead of the real imperative intent when a command follows ("...spegnile"); explicit numeric values in text sometimes dropped from label instead of triggering the _set/_dim value intent
  • The 5000 < 1500 ablation non-monotonicity is unexplained — candidate hypothesis (fixed-epoch vs fixed-iters training budget) not yet tested. Worth a follow-up run before G4 if there's time, otherwise publish as an honest open question
  • G2's Baseline C only ran few-shot prompted (per PLAN.md checklist item, zero-shot variant was skipped as not needed for the story) — note if anyone re-derives numbers from PLAN.md expecting both