Skip to content

fix: fix the leftovers that the review of questions as steps found - #38

Merged
osolmaz merged 1 commit into
mainfrom
fix/questions-review
Oct 29, 2026
Merged

osolmaz merged 1 commit into
mainfrom
fix/questions-review

Conversation

@osolmaz

@osolmaz osolmaz commented Oct 29, 2026

Copy link
Copy Markdown
Owner

Summary

A review of #37, by a reviewer that did not write it, found bugs and leftovers of the old interface.
This PR fixes them.
The main bug: the viewer showed "The model is answering…" for good on a step whose request failed.

What Changed

Most fixes are small. Two change behavior in the examples.

  • Viewer:
    • A step without answers at a failed run or a stopped case is marked failed, with its reason (Frame.failed).
    • A pick is found by the facts that picking writes. The new core helper pickKeys does this, and the replay check uses it too. A choice question that is no pick, such as the README's team, is no longer charted as a pick. The unused FrameChoice.key is removed.
  • WikiRace: a dead end (no link or section left) ends the race without a request that offers only "give up". The outcome is unfinished with the reason dead end. Before feat!: make every step a set of questions #37, such a race also ended without a request.
  • rubric-judge:
    • When the gathering steps reach the judge's limit, the state now records that the gathering ended. So a fork before the checks, under a judge with a higher limit, asks only the checks.
    • When only stop is left to pick, the checks come at once.
    • The outcome loses steps, which had a second meaning next to result.steps. The README says to fork at steps - 1.
  • Docs and leftovers: the trace and result comments in src/cli/layout.ts, the package description, P(path) in docs/MATH.md and the plan (choice answers only, because a bool answer names no label), the viewer README, the README's list of the three examples, and a test that named the removed --state-from flag.

Testing

All tests use the fake decider; no model was called.

  • pnpm check at the root passes. Tests: root 71, viewer 39, chess 75, rubric-judge 49, WikiRace 20.
  • New tests: pickKeys; the failed-step panel; a thrown case's step; the WikiRace dead end; the judge's fork after its limit; the judge with only stop left.

Risks

The risk is low. These changes follow the design of #37, and the full check passes.

- The viewer marks a step whose request failed, or whose case stopped,
  as failed, with the reason; before, it showed "The model is
  answering…" for good.
- The viewer finds a pick by the facts that `picking` writes, with the
  new `pickKeys` that the replay check also uses; a choice question that
  is no pick is no longer charted as one. `FrameChoice.key` is gone.
- WikiRace ends at a dead end without a request to give up, as an
  unfinished race with the reason "dead end".
- rubric-judge ends the gathering in its state at the judge's limit, so
  a fork before the checks asks only the checks, and goes to the checks
  at once when only stop is left. Its outcome loses `steps`.
- Fix the trace and result comments, the package description, the path
  probability in docs/MATH.md and the plan, the viewer README, and the
  README's list of examples; remove the test of the removed flag.
@osolmaz osolmaz self-assigned this Oct 29, 2026
@osolmaz
osolmaz merged commit 3c14b1c into main Oct 29, 2026
1 check passed
@osolmaz
osolmaz deleted the fix/questions-review branch October 29, 2026 03:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant