Developer experience: bundled recipes, safe defaults, Python training API, server batching

CONTRIBUTING.md

@@ -34,6 +34,13 @@ class MyHead(Head):
 
 Unknown or missing constructor arguments are reported when the config loads.
 
+## Adding a recipe
+
+Recipes ship with the package in `src/decisionformer/recipes/`, so `decisionformer train NAME` works after a plain
+`pip install`. The first comment line is the description `decisionformer recipes` shows. Put only what differs from
+the defaults in a recipe, and add a recipe only after it has learned in a real run. The unit tests load every
+recipe.
+
 ## Development loop
 
 ```bash

README.md

@@ -10,7 +10,7 @@ generation. Three lines to decide, one command to train, one command to serve.
 
 Python 3.10 or newer, with PyTorch 2.6+ and transformers 5.20+.
 
-**With pip**
+**With pip** (from the first public release; until then, install from GitHub)
 
 ```bash
 pip install decisionformer
@@ -114,8 +114,10 @@ curl localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
 ```
 
 The server takes the same `state` and `questions` and returns the same JSON as `decide`, with `id`, `object` and
-`created` added. It is the System One API used by Jev, vLLM and the Decision Index. Over-long requests are
-refused with HTTP 422, never truncated, and `--api-key-env MY_KEY_VAR` requires `Authorization: Bearer <key>`.
+`created` added. It is the System One API used by Jev, vLLM and the Decision Index; the request and response
+schemas are at `/docs`. Over-long requests are refused with HTTP 422, never truncated, and
+`--api-key-env MY_KEY_VAR` requires `Authorization: Bearer <key>`. A request that arrives while the model is idle
+runs at once; requests that arrive while it is busy are answered together, up to `--max-batch` (16) at a time.
 
 For code-readout checkpoints on Qwen3.5 (the Decision 3.0 layout), vLLM can run the model behind the same API.
 Other checkpoints fail with a clear error that points to the default backend:
@@ -141,22 +143,26 @@ Backbones are the architectures built into Hugging Face transformers, loaded wit
 `save_pretrained`; decisionformer has no model code of its own. Hub models that need their own modeling code
 (`trust_remote_code`) are not supported. Decoders read the answer as a code token (`A`, `B`, ...) at the end of
 the prompt, which works from step 0; encoders score each option. Encoder-decoder models without a standalone text
-encoder in transformers (BART) are refused with a clear error.
+encoder in transformers (BART) are refused with a clear error. Fine-tuning Qwen3.5 with the answer-code readout
+has not learned reliably in our runs yet (`docs/benchmarks.md`); loading and serving Qwen3.5 checkpoints such as
+`vllm-sr/d3-edge` works.
 
 ## Fine-tune
 
 ```bash
 decisionformer data split hf:FrontiersMind/deciformer-15k:train --out data/deciformer-15k
-decisionformer train configs/recipes/deciformer15k_qwen35.yaml
-torchrun --nproc-per-node 8 -m decisionformer train configs/recipes/deciformer15k_qwen35.yaml   # multi-GPU
+decisionformer train deciformer15k_qwen3                  # Qwen3-0.6B + LoRA, one epoch
+torchrun --nproc-per-node 8 -m decisionformer train deciformer15k_qwen3                     # multi-GPU
 ```
 
-Every setting lives in a YAML config, and any value can be overridden on the command line with dotted keys
-(`train.lr=1e-4`). `decisionformer config FILE` prints the resolved configuration. A minimal config:
+`deciformer15k_qwen3` is a bundled recipe: `decisionformer recipes` lists them all, and
+`decisionformer init deciformer15k_qwen3` copies one to a YAML file you can edit. Every setting lives in that YAML,
+and any value can be overridden on the command line with dotted keys (`train.lr=1e-4`).
+`decisionformer config FILE` prints the resolved configuration. A minimal config:
 
 ```yaml
 model:
-  backbone: Qwen/Qwen3.5-0.8B        # any Hub id or local path
+  backbone: Qwen/Qwen3-0.6B          # any Hub id or local path
   attention: causal                  # causal | bidirectional | prefix_bidirectional
   head: {type: code_readout}
 finetune:
@@ -168,13 +174,24 @@ data:
   eval: {dev: data/dev.jsonl}
 train:
   output_dir: runs/my-decider
-  lr: 2.0e-4
   push_to_hub: my-org/my-decider     # optional: uploads best/ when training ends (private by default)
 ```
 
-A run writes `runs/<name>/best` and `last`: finished checkpoints in the standard Hugging Face layout, ready for
-`from_pretrained`, `serve` and `push`. It also writes `metrics.jsonl`, per-example predictions at every
-evaluation, resumable state and `summary.json`.
+Everything else has a default. The learning rate follows the fine-tuning mode (2e-5 full, 2e-4 LoRA and QLoRA,
+1e-3 head only). Training evaluates and saves resumable state every 500 steps, and rerunning the same command
+resumes where it stopped; the resumable state is removed once the run finishes. A run writes `runs/<name>/best` and `last`: finished checkpoints in the standard Hugging
+Face layout, ready for `from_pretrained`, `serve` and `push`. It also writes `metrics.jsonl`, per-example
+predictions at every evaluation and `summary.json`. Training warns when the training data shares ids, families or
+identical states with the evaluation or calibration data.
+
+The same from Python, with a recipe name, a YAML file or a dict:
+
+```python
+import decisionformer
+
+summary = decisionformer.train("deciformer15k_qwen3", {"train.max_steps": 200})
+model = decisionformer.AutoDecisionModel.from_pretrained("runs/deciformer15k-qwen3-0.6b/best")
+```
 
 ### On a laptop (Mac or Linux)
 
@@ -184,16 +201,16 @@ steps:
 ```bash
 pip install "decisionformer[data]"
 decisionformer data split hf:FrontiersMind/deciformer-15k:train --out data/deciformer-15k
-decisionformer train configs/recipes/local_smollm2.yaml
+decisionformer train local_smollm2
 decisionformer serve runs/local-smollm2-135m/best        # try it on /v1/systemone
 ```
 
 The device is picked automatically: a CUDA GPU, then Apple silicon (MPS), then the CPU. Set
 `DECISIONFORMER_DEVICE=cpu` to force the CPU. On MPS and CPU, training runs in float32. The recipe keeps memory
 low with micro-batches of 4, at most 4,096 tokens per micro-batch and gradient checkpointing: on a 6-core CPU it
 peaked at 3.7 GB of memory and took about 15 s per step. If memory is still tight, lower them:
-`train.micro_batch_size=2 train.token_budget=2048`. A checkpoint is saved every 100 steps, and
-rerunning the same command resumes from it.
+`train.micro_batch_size=2 train.token_budget=2048`. Resumable state is saved every 100 steps, and rerunning the
+same command resumes from it.
 
 ### Training data
 
@@ -354,7 +371,9 @@ python -m decision_index pipeline --engine decisionformer.integrations.decision_
 
 ## Measured results
 
-On one 8 GB RTX 2060 SUPER in FP16, on MNLI: Qwen3-0.6B with LoRA and `code_readout` went from 0.36 zero-shot to
+On one 8 GB RTX 2060 SUPER in FP16, the starting recipe `deciformer15k_qwen3` (Qwen3-0.6B with LoRA, one epoch of
+deciformer-15k, 30 minutes) reaches **0.824** dev accuracy across its 9 tasks, from 0.75 (VitaminC) to 0.90 (Civil
+Comments), with calibrated ECE 0.023. On MNLI: Qwen3-0.6B with LoRA and `code_readout` went from 0.36 zero-shot to
 **0.83** dev accuracy in 250 steps (calibrated ECE 0.042), and Qwen3.5-0.8B with QLoRA and bidirectional attention
 reached **0.84** in 150 steps. The vLLM backend matched the transformers backend to within 0.001 per probability, with identical choices,
 at a median of 30 ms per 3-question request. Configs and full tables are in [`docs/benchmarks.md`](docs/benchmarks.md).
@@ -369,7 +388,7 @@ ruff check src tests
 ```
 
 Experimental: `decisionformer adapt` runs masked next-token prediction (as in LLM2Vec) before bidirectional
-training (`configs/recipes/adapt_mntp.yaml`). A short stage did not help in our tests.
+training (the `adapt_mntp` recipe). A short stage did not help in our tests.
 
 ## Roadmap
 

configs/benchmarks/mnli_8gb/_base.yaml

@@ -1,5 +1,4 @@
 # 8 GB Turing GPU (RTX 2060 SUPER): fp16 autocast, no bf16 / flash-attention.
-base: ../../base.yaml
 data:
   train: data/mnli/train.jsonl
   calibration: data/mnli/calibration_small.jsonl

docs/benchmarks.md

@@ -13,6 +13,28 @@ records each (`dev_small.jsonl`, `calibration_small.jsonl`, separated by whole f
 ¹ Before v0.2's final pooler default: code-readout heads now read the last token in every attention mode (the
 Decision 3.0 layout).
 
+**deciformer-15k, the starting recipe.** `decisionformer train deciformer15k_qwen3` (Qwen3-0.6B, LoRA r16, causal,
+one epoch of 11,982 records = 375 updates, fp16 on the 2060 SUPER, 30 minutes with evaluations) on the family-level
+split from `decisionformer data split` (1,509 dev records):
+
+| step | dev accuracy (calibrated) | ECE | Brier | NLL |
+|---|---|---|---|---|
+| 125 | 0.792 | 0.039 | 0.314 | 0.585 |
+| 250 | 0.805 | 0.030 | 0.281 | 0.533 |
+| 375 | **0.824** | 0.023 | 0.252 | 0.467 |
+
+Per task at step 375: Civil Comments 0.904, PAWS 0.875, MASSIVE en-US 0.864, SQuAD 2.0 0.860, MASSIVE de-DE 0.823,
+Aegis 2.0 0.790, MultiNLI 0.774, BoolQ 0.772, VitaminC 0.750.
+
+**Known issue: Qwen3.5 with causal attention.** Qwen3.5-0.8B fine-tuned with causal attention stayed at chance on
+MNLI (0.34-0.37) in four runs: LoRA with an fp16 base (150 steps, twice), QLoRA (60 steps), and LoRA on an fp32 base
+(60 steps). Each run's loss rose near step 25 and then settled at the class prior. Before training, the code
+readout's logits equal the language model's own next-token logits for the answer codes exactly, so the prompt,
+answer position and readout rows are correct. Qwen3-0.6B learns with causal attention, and Qwen3.5 learned with
+bidirectional attention and a mean pooler (0.84 above). With bidirectional attention and the last-token readout
+(the Decision 3.0 layout, the default since v0.2), a 40-step QLoRA check on deciformer-15k showed the same loss spike
+near step 20 (1.55 → 2.04 → 1.23) and 0.41 dev accuracy on 128 records, so no Qwen3.5 recipe ships yet.
+
 **Parity with the Decision 3.0 reference runtime.** `AutoDecisionModel.from_pretrained("vllm-sr/d3-edge")` against
 the checkpoint's own `trust_remote_code` runtime, BF16, four requests (choice, noul and score questions, an empty
 state, a JSON state, a 32-option choice): one question per forward pass, the probabilities are identical (maximum

src/decisionformer/__init__.py

@@ -4,17 +4,21 @@
     model = AutoDecisionModel.from_pretrained("org/my-decider")
     model.decide("My card was charged twice.", {"team": {"type": "choice", "criteria": {"billing": None,
                                                                                          "shipping": None}}})
+
+    import decisionformer
+    decisionformer.train("deciformer15k_qwen3")          # fine-tune from a bundled recipe, a YAML file or a dict
 """
 
 __version__ = "0.2.0.dev0"
 
 from decisionformer.api import AutoDecisionModel, Decider, UnsupportedInput  # noqa: E402
-from decisionformer.config import RunConfig, load_config  # noqa: E402
+from decisionformer.config import RunConfig, list_recipes, load_config  # noqa: E402
 from decisionformer.formats import PromptFormat, get_format, register_format  # noqa: E402
 from decisionformer.prompting import ContextLengthError  # noqa: E402
 from decisionformer.schema import DecisionInputError  # noqa: E402
+from decisionformer.trainer import train  # noqa: E402
 
 __all__ = [
     "AutoDecisionModel", "ContextLengthError", "Decider", "DecisionInputError", "PromptFormat", "RunConfig",
-    "UnsupportedInput", "__version__", "get_format", "load_config", "register_format",
+    "UnsupportedInput", "__version__", "get_format", "list_recipes", "load_config", "register_format", "train",
 ]

src/decisionformer/adapt.py

@@ -33,6 +33,7 @@
 from decisionformer.inference import autocast
 from decisionformer.optim import lr_lambda
 from decisionformer.schema import read_records
+from decisionformer.trainer import _setup_logging
 
 logger = logging.getLogger("decisionformer.adapt")
 
@@ -68,8 +69,7 @@ def adapt(config: RunConfig) -> Path:
     if config.finetune.mode not in ("full", "lora"):
         raise ValueError("adapt supports finetune.mode 'full' or 'lora'")
     state = dist_utils.setup(config.distributed.backend, config.distributed.timeout_minutes)
-    logging.basicConfig(level=logging.INFO if state.is_main else logging.WARNING,
-                        format="%(asctime)s %(levelname)s %(name)s: %(message)s", force=True)
+    _setup_logging(state.is_main)
     torch.manual_seed(config.train.seed)
     model_cfg = config.model
     tokenizer = load_tokenizer(model_cfg.backbone, model_cfg.revision, model_cfg.trust_remote_code)

src/decisionformer/api.py

@@ -43,6 +43,13 @@ def _local_dir(name_or_path: str | Path, revision: str | None, token: str | None
     return Path(snapshot_download(str(name_or_path), revision=revision, token=token))
 
 
+def display_name(name_or_path: str | Path, root: Path) -> str:
+    """How a model names itself in responses: its Hub id, or its run name for a local `runs/<name>/best`."""
+    if not Path(str(name_or_path)).expanduser().is_dir():
+        return str(name_or_path)
+    return root.resolve().parent.name if root.name in ("best", "last") else root.name
+
+
 def _prompt_name(config: dict[str, Any]) -> str:
     name = config.get("prompt") or config.get("prompt_format")
     if name:
@@ -132,7 +139,7 @@ def from_pretrained(cls, name_or_path: str | Path, *, revision: str | None = Non
         builder = PromptBuilder(tokenizer, style, kind, max_length=limit, overflow="error", codes=codes,
                                 code_token_ids=config.get("token_ids") or None,
                                 chat_template_kwargs=ext.get("chat_template_kwargs"))
-        name = ext.get("name") or (str(name_or_path) if not Path(str(name_or_path)).is_dir() else root.name)
+        name = ext.get("name") or display_name(name_or_path, root)
         return cls(model, builder, tokenizer, config, device=device, name=name, max_input_tokens=limit,
                    batch_size=batch_size, path=root)