Skip to content

feat!: make every step a set of questions - #37

Merged
osolmaz merged 1 commit into
mainfrom
feat/questions-as-steps
Oct 29, 2026
Merged

osolmaz merged 1 commit into
mainfrom
feat/questions-as-steps

Conversation

@osolmaz

@osolmaz osolmaz commented Oct 29, 2026

Copy link
Copy Markdown
Owner

Summary

switch asked the model about a state in two ways: a pick at each step, and the final questions after the steps.
Now every step is one set of typed questions, and the domain's transition function gets the answers.
A pick is one choice question that a helper builds, so the final questions, the answer stages and run --state-from are gone.
The plan, with every decision, is in docs/2026-10-29-questions-as-steps.md.

What Changed

The core loop has one path now. A domain gives initial, questions(state) and transition(state, answers), and the run ends when questions gives nothing.

  • Core: new StateMachine type; picking, pickQuestion and pickedChoice in src/core/pick.ts (was choices.ts). switch adds no stop choice and no longer refuses a repeated pick. The limits are windowShare and an optional steps. A run ends as done, step limit or failed. The view function loses its purpose argument. Large question sets are split into batches at every step.
  • Trace and replay: the entries are start, state, questions, answers, ended, note and thrown. A replay checks that each recorded answer fits the questions asked now (ids, types, and for a pick, the choice's key). A fork repeats the first n steps.
  • CLI: no --state-from; result.json loses answers, run.json loses stateFrom.
  • Viewer: frames show each step's questions and answers, with the pick's choices and probabilities.
  • Chess: "Resign the game" is a choice; a resignation is a status of the state.
  • WikiRace: "Give up and lose the race" is a choice; giving up is a field of the state.
  • rubric-judge: gathers evidence by picks, then asks every item's meets and misses questions in one step. The outcome holds each item's verdict, which score and compare read.
  • Docs: README "How it works" and its domain example, docs/MATH.md, AGENTS.md and the example READMEs.

Testing

All tests use the fake decider; no model was called.

  • pnpm check at the root passes: format, lint, typecheck, tests with coverage, DRY, the viewer and the three examples, and slophammer.
  • Tests: root 71, viewer 37, chess 75, rubric-judge 48, WikiRace 19.
  • The README's domain example was typechecked and run with the fake decider outside the repository.

Risks

This is a breaking change for every domain and for old run folders.

  • Old traces and run folders are not read.
  • Re-judging a whole set with another judge file needs one replay --fork per case; there is no set command for it yet.
  • The judge files' texts did not change, so their hashes stay the same, but the judge's version is now 0.6.0.

Follow-ups

  • A reviewer that did not write the change looks for leftovers of the old interface after the merge.

A domain now gives an initial state, a question function and a transition
function from the state and the answers. The run ends when the question
function asks nothing. A pick is one choice question, which the new
`picking` helper builds from a choice function; switch adds no stop choice.

- Remove the final questions, the answer stages and `run --state-from`.
  `replay --fork` asks a case's last questions again.
- The step limit is an optional safety net. A run ends as done, step limit
  or failed.
- A replay checks that each recorded answer fits the questions asked now,
  and finds a recorded pick by its choice's key.
- The trace has questions and answers entries; result.json loses answers,
  and run.json loses stateFrom.
- The view function loses its purpose argument.
- Chess resigns by a choice, WikiRace gives up by a choice, and
  rubric-judge asks its checks as its last step, with the verdicts in the
  outcome.
- Update the README, docs/MATH.md, AGENTS.md, the example READMEs and the
  viewer, and add the plan in docs/2026-10-29-questions-as-steps.md.
@osolmaz osolmaz self-assigned this Oct 29, 2026
@osolmaz
osolmaz merged commit cac1936 into main Oct 29, 2026
1 check passed
@osolmaz
osolmaz deleted the feat/questions-as-steps branch October 29, 2026 03:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant