Repository navigation

feat(questions): run Inspect, OpenAI evals, skill-creator and promptfoo sets on the answers track - #214

Merged

napetrov merged 32 commits into

intel:mainfrom

napetrov:feat/run-answer-formats

Oct 27, 2026

Merged

feat(questions): run Inspect, OpenAI evals, skill-creator and promptfoo sets on the answers track#214
napetrov merged 32 commits into
intel:mainfrom
napetrov:feat/run-answer-formats

Commits on Oct 26, 2026
- feat(questions): run Inspect, OpenAI evals, skill-creator and promptfoo sets on the answers track
  napetrov
  committed
  4385911
- docs(answer-formats): license header and depends_on
  napetrov
  committed
  08e6af5
- feat(questions): skill-creator imports carry expected_skill
  napetrov
  committed
  aeabce0
- fix(eval): stamp the reference scorer on an arms file graded with --no-judge
  
  xaleryb
  authored and
  napetrov
  committed
  42e373b
- fix(native-import): keep an errored Inspect sample's error on its answer
  
  xaleryb
  authored and
  napetrov
  committed
  3727899
- fix(questions): read Inspect samples the way Inspect does
  
  xaleryb
  authored and
  napetrov
  committed
  c1a10ba
- fix(questions): split a promptfoo .txt prompt file on --- lines
  
  xaleryb
  authored and
  napetrov
  committed
  761d704
- fix(questions): import model-graded OpenAI eval rows that have no ideal
  
  xaleryb
  authored and
  napetrov
  committed
  7b37836
- fix(questions): refuse to import a file whose format is ambiguous
  
  xaleryb
  authored and
  napetrov
  committed
  a2d14c6
- fix(eval): leave a no-retrieval placeholder answer unscored in eval reference
  
  xaleryb
  authored and
  napetrov
  committed
  6326dab
- fix(eval): validate every grading field eval reference would fail on
  
  xaleryb
  authored and
  napetrov
  committed
  45b3428
- fix(questions): an OpenAI json_match row with no JSON ideal is not graded as any JSON
  
  xaleryb
  authored and
  napetrov
  committed
  d24d699
- fix(questions): do not grade an Inspect pattern() scorer as a bare regex search
  
  xaleryb
  authored and
  napetrov
  committed
  7f12fa3
- fix(questions): ask a multiple_correct Inspect question for several letters
  
  xaleryb
  authored and
  napetrov
  committed
  571fa36
- fix(questions): turn malformed import input into an import error
  
  xaleryb
  authored and
  napetrov
  committed
  e68e6a2
- fix(arms): send a prompt_style: verbatim question to arms run unwrapped
  
  xaleryb
  authored and
  napetrov
  committed
  2b16652
- test(questions): pin the grader and importer mappings mutants slipped through
  
  xaleryb
  authored and
  napetrov
  committed
  6bfcfe7
- fix(questions): promptfoo CSV __expected cells follow assertionFromString
  napetrov
  committed
  f5dbac0
- fix(eval): count a scored grade that has an unjudged judge check
  napetrov
  committed
  7569273
- docs(answer-formats): describe judge_unscored in the reference summary
  napetrov
  committed
  f65e2b0
- fix(questions): a promptfoo contains with an empty value is unsupported, not a pass
  napetrov
  committed
  10fbe95
- fix(eval): run regex checks with a timeout; a check that times out is unscored
  
  xaleryb
  authored and
  napetrov
  committed
  7561845
- fix(questions): parse a promptfoo CSV __expected cell as promptfoo does and map it like a YAML assert
  
  xaleryb
  authored and
  napetrov
  committed
  af4fa00
- fix(questions): a promptfoo text assert with no value is unsupported, not graded
  
  xaleryb
  authored and
  napetrov
  committed
  8310771
- Loosen the regex timeout timing assert to 5x the deadline
  
  antonwolfy
  authored and
  napetrov
  committed
  93edda0
- Point the checklist link at benchmarking-and-comparison
  
  antonwolfy
  authored and
  napetrov
  committed
  026f547
- Say contains on json finds the first object or array
  
  antonwolfy
  authored and
  napetrov
  committed
  da882b8
- Address the #214 review: blank values, thresholds, regex, weights
  napetrov
  committed
  804c83b
- fix(eval): treat a match value that normalises to empty as blank
  xaleryb
  committed
  6432007
- fix(import): refuse a negative or non-finite promptfoo assert weight
  xaleryb
  committed
  51442c9
- test(eval): pin the all-mode regex stop and finite weights and thresholds
  xaleryb
  committed
  1a9d30a
- Merge intel/main into feat/run-answer-formats
  xaleryb
  committed
  f814c85

Commits

Commits on Oct 26, 2026