Skip to content

Pre-release fixes from the developer review (includes #5) - #6

Merged
akanyaani merged 1 commit into
mainfrom
pre-release
Oct 23, 2026
Merged

akanyaani merged 1 commit into
mainfrom
pre-release

Conversation

@akanyaani

Copy link
Copy Markdown
Contributor

Fixes for the review findings agreed before open-sourcing. This branch includes #5 (citation and bidirectional results), so merging this PR covers both.

  • Missing data fails first. train checks its data files before loading any model and prints the exact decisionformer data split … command when recipe data is missing. Previously it downloaded the model, then failed with a traceback and left an orphan output folder.
  • One-line config errors. For example, error: unknown setting train.learning_rate (did you mean train.lr?). An override without = is rejected instead of silently ignored.
  • Serving limit. Exports record the length they were trained on in decision_config.json (max_length). Loaded and served models refuse longer requests (HTTP 422, never truncated) unless max_length= / --max-length raises the limit. Previously a 5,088-token request to a model trained on 1,024 tokens was answered silently.
  • Stricter requests. Every question needs a type (previously it silently defaulted to choice), and the server requires state.
  • README:
    • Defines confidence (1 − normalized entropy, not the top probability).
    • Lists the GitHub install first; PyPI comes once v0.2.0 is published.
    • Says the backbone tests use tiny models, and lists the real checkpoints actually trained.
    • Notes CPU latency and that the Decision Index's 1,000 ms latency rule needs a GPU.
  • deciformer15k_qwen3 saves resumable state every 125 steps; with 375 steps and the 500-step default, it never saved before.
  • decisionformer --version.

Not included: the fp32 vs bf16 gap between the training metrics and eval on the export, which needs investigation.

Testing: the full suite passes on the GPU box with the GPU hidden. A smoke test of decisionformer train local_smollm2 in an empty folder fails instantly with the data split command and leaves nothing behind.

- train checks its data files before loading any model and names the
  exact data split command when recipe data is missing; no output folder
  is left behind.
- CLI config errors are one line ("unknown setting train.learning_rate
  (did you mean train.lr?)"); an override without = is rejected.
- Exports record the length they were trained on (decision_config
  max_length); loaded and served models refuse longer requests unless
  max_length / --max-length allows them.
- Every question needs a type, and the server needs a state.
- README: confidence defined; install from GitHub first, PyPI once
  published; backbone tests use tiny models, with the real checkpoints
  listed; CPU latency and the Decision Index latency rule.
- deciformer15k_qwen3 saves resumable state every 125 steps.
- decisionformer --version.
@akanyaani
akanyaani merged commit 9a90e39 into main Oct 23, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant