feat(eval): add eval redteam to probe grading checks with planted answers by napetrov · Pull Request #281 · intel/dokimi

feat(eval): red-team grading blocks against wrong, empty, hedged and stuffed answers

napetrov

committed

28a3609

fix(eval): redteam skips the attacks a grading block passes by design

napetrov

committed

71cc8e6

fix(redteam): inconclusive status, single-literal stuffing, hedge collisions

napetrov

committed

068abc8

fix(redteam): negated checks no longer silence wrong, empty and hedge

napetrov

committed

04f2142

fix(redteam): clear wrong answer of check values, stuff negated checks

napetrov

committed

6e16c6b

fix(redteam): skip the stuffed probe for a negated mode all match

napetrov

committed

e881ab5

Merge branch 'main' of https://github.com/intel/dokimi into pr281-work

napetrov

committed

12f653b

fix(redteam): choose the planted wrong answer and the hedge probe by the check itself

napetrov

committed

d96f87d

fix(redteam): grade wrong, empty and hedge against the whole block, threshold included

napetrov

committed

6b6aa2f

fix(redteam): a set with a question that was not measured is inconclusive, not pass

napetrov

committed

19f30ef

fix(redteam): bury aliases between decoys instead of listing them for the stuffed probe

napetrov

committed

3b4c701

fix(redteam): grade each question with its own block when an id repeats

napetrov

committed

1b64508

feat(redteam): record the judge model and provider in the redteam.v1 artifact

napetrov

committed

4d7cef6

refactor(redteam): read check values and build the judge ask through the grader's own helpers

napetrov

committed

70e66bd

fix(redteam): name -o in the cut-violations line when no artifact is written

napetrov

committed

ef1ed29

test(redteam): pin the canary of a whole-answer match and of a string json value

napetrov

committed

f108f53

docs(redteam): list the negated mode all exclusion and say where negated_only appears

napetrov

committed

89641b7

test(redteam): run judge checks through run(ask=) and through the --judge path

napetrov

committed

0918eea

test(redteam): run an ungradable block through the command and pin the empty inputs

napetrov

committed

27828e4

test(redteam): check the artifact of a blocked and an inconclusive command run

napetrov

committed

1946f48

test(redteam): pin the bare stuffed answer of a whole-answer check with aliases

napetrov

committed

c1596ef

docs(redteam): changelog fragment says what an inconclusive run is

napetrov

committed

a296b5a

Address review: answer the review summaries and bot comments in their threads

napetrov

committed

2a259de

docs(redteam): refresh two stale comments on the wrong and stuffed attacks

napetrov

committed

c68a1f1

Navigation Menu

Repository navigation

feat(eval): add eval redteam to probe grading checks with planted answers - #281

feat(eval): add eval redteam to probe grading checks with planted answers#281
napetrov wants to merge 30 commits into
intel:mainfrom
napetrov:feat/verifier-redteam

Commits on Oct 29, 2026

feat(eval): red-team grading blocks against wrong, empty, hedged and stuffed answers

fix(eval): redteam skips the attacks a grading block passes by design

fix(redteam): inconclusive status, single-literal stuffing, hedge collisions

fix(redteam): negated checks no longer silence wrong, empty and hedge

fix(redteam): clear wrong answer of check values, stuff negated checks

fix(redteam): skip the stuffed probe for a negated mode all match

Merge branch 'main' of https://github.com/intel/dokimi into pr281-work

fix(redteam): choose the planted wrong answer and the hedge probe by the check itself

fix(redteam): grade wrong, empty and hedge against the whole block, threshold included

fix(redteam): a set with a question that was not measured is inconclusive, not pass

fix(redteam): bury aliases between decoys instead of listing them for the stuffed probe

fix(redteam): grade each question with its own block when an id repeats

feat(redteam): record the judge model and provider in the redteam.v1 artifact

refactor(redteam): read check values and build the judge ask through the grader's own helpers

fix(redteam): name -o in the cut-violations line when no artifact is written

test(redteam): pin the canary of a whole-answer match and of a string json value

docs(redteam): list the negated mode all exclusion and say where negated_only appears

test(redteam): run judge checks through run(ask=) and through the --judge path

test(redteam): run an ungradable block through the command and pin the empty inputs

test(redteam): check the artifact of a blocked and an inconclusive command run

test(redteam): pin the bare stuffed answer of a whole-answer check with aliases

docs(redteam): changelog fragment says what an inconclusive run is

Address review: answer the review summaries and bot comments in their threads

docs(redteam): refresh two stale comments on the wrong and stuffed attacks

Commits on Oct 30, 2026

fix(redteam): a question set nested too deep to parse exits 2, not with a traceback

fix(redteam): a block with an unrun judge or regex is unprobed, not a pass or a block

Merge remote-tracking branch 'intel/main' into pr281-1029

test(redteam): a regex past its timeout leaves the probe unscored, with and without a threshold

docs: keep the comment blocks this PR adds to two lines

fix(redteam): a json contains check's stuffed answer goes after a decoy