Repository navigation
refactor!: rename legal-judge to rubric-judge, with a general default judge - #35
Merged
Merged
Conversation
… judge The example grades an answer against a rubric for any kind of task: only its judge texts and two search patterns were about legal tasks. The domain and package are now `rubric-judge`, and its names follow (`JudgeState`, `JudgeCase`, `judgeMachine`, `rubricJudge`, `rubricJudgeSetup`). - `judges/general.json`, the default, names no field of work. - `judges/legal.json` keeps the old default's texts, so its scores compare with the earlier runs', and `judges/legal-qualifiers.json` keeps the qualifiers variant. - The search patterns stay: section, clause and article references occur in many kinds of documents. - The README, AGENTS.md, CI and the Slophammer boundaries follow.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The legal-judge example grades a saved answer against a task's rubric, and almost nothing in it is about legal work.
Only the judge file's texts said "a legal task", and two search patterns looked for clause references and a list of currencies.
This change renames the example to
rubric-judgeand gives it a general default judge, while the old legal judge stays as a judge file of its own.Requests made with
judges/legal.jsonare the same as the old default's, so legal scores stay comparable with the earlier runs.What Changed
The package moves to
examples/rubric-judge, and its domain, package and code names follow.The judge texts are the only part that the model reads differently, and only with the new default judge.
judges/general.jsonis the new default: the old texts without "legal", and with a general example in the rules.judges/legal.jsonandjudges/legal-qualifiers.jsonare the old two files, with the new domain name.rubric-judge, and the package@switch-examples/rubric-judge.LegalState,LegalCase,legalMachineandlegalJudgebecomeJudgeState,JudgeCase,judgeMachineandrubricJudge.AGENTS.md, the CI job, the Slophammer boundaries and the lockfile follow the new path.Testing
I ran every check of switch, the viewer and both examples.
pnpm check: format, lint, typecheck, coverage, DRY and Slophammer with no findings. 68 tests in switch, 35 in the viewer, 74 in chess and 46 in rubric-judge.generaland names no legal word, and that both legal judges parse, with the same questions. The judge schema test also refuses a choices limit of 1.I made no model calls, so the general judge has no score yet.
Risks
The general judge has not been measured yet, and all labelled cases so far are legal tasks.
--judge examples/rubric-judge/judges/legal.json.legal-judgeor its judge files need the new names.