[napetrov]

Summary

Part of #220.

eval redteam QUESTIONS.json checks that each question's grading block rejects the answers it should. It grades five synthetic answers per question:

• canary: the right answer, built from one literal check (match, fuzzy, exact, choice, json). It must pass.
• wrong: a planted wrong answer that carries no positive check value. It must fail.
• empty: the empty answer. It must fail.
• hedge: a non-committal sentence with no check value. It must fail.
• stuffed: the values of one literal check listed with no sentence around them. It must fail. Built for two or more values of a positive check (mode any), or for one or more values of a negated check that is neither mode: all nor a whole-answer match (exact, json, location: exact).

A negate check does not switch the other attacks off: wrong, empty and hedge grade the block's non-negated checks. A block of only negated checks gets none of the three and is reported as negated_only.

Statuses: pass, blocked (an attack was accepted or a canary was rejected), inconclusive (no question was probed). Exit 0 on pass, 1 on blocked or inconclusive, 2 on an unreadable question set or an unwritable -o. -o writes redteam.v1. Judge checks are unscored unless --judge is given, so the default run makes no model call. No score changes.

Evidence

• Red then green: each new test failed before its fix and passes after. The regressions cover: a planted wrong answer that contains a check value (match 31, match 1, fuzzy on) no longer blocks a correct grader; negated checks no longer silence the attacks on a positive check; a single negated value is stuffed; a negated whole-answer check is not stuffed; a check value in the hedge skips the hedge probe; the canary for match at the end with ignore_punctuation: false has no full stop; a run that probed nothing is inconclusive; an unwritable -o exits 2; the summary counts unprobed questions.
• Fresh-context review (FRESH-REVIEW-A2, diff e2147b1...04f2142; its quoted line numbers match 04f2142, not 71cc8e6) found three blockers, all fixed in 6e16c6b: a wrong answer that contained a check value blocked correct graders; stuffing a negated exact, json or location: exact check blocked correct graders; a block with one negated value was never probed. In-session review found the negated-check silencing (fixed in 04f2142) and the hedge clash (fixed in 068abc8).
• Local CI on 6e16c6b (make ci-local --offline): ruff, mypy (273 files), deptry, bandit, schema and registry checks, mkdocs and the CLI-surface job (which reports the added eval redteam) pass. Pytest: 9735 passed, 5 skipped, 1 failed: test_container_agent.py::test_probe_reports_a_writable_docker_socket (AF_UNIX path too long). That test depends on the path length of the worktree: it failed in these /localdisk/napetrov/scratch worktrees and on a clean intel/main checkout (earlier run), and passed in the shorter base worktree.
• Negated mode: all fix on e881ab5: a negated match with mode: all and the default location: end got the stuffed answer Paris Lyon. That answer ends with one value only, so the all-match is false and the negation passes it: the real grader accepted it and a correct grader was blocked. The stuffed probe is now skipped for negated mode: all, as the positive rule already skips mode: all. Red then green: the two new cases failed on 6e16c6b (2 failed, 41 passed) and pass on e881ab5 (43 passed). ruff check and mypy pass. A fresh-context claude -p review of 6e16c6bb..e881ab57 plus stuffed_for found no blocker; it confirmed the grader rejects every stuffed answer that is still built, and its two minor points (the skip is wider than the one failing case; the docstrings overstated) are addressed.
• The redteam tests: 43 cases at e881ab5 (41 at 6e16c6b, which passed in the full run).

Merge Danger

• New command eval redteam and new persisted format redteam.v1 (schema, fixture, registry entry). Both are new in this PR.
• Exit 1 on a blocked or inconclusive run. A CI job that runs eval redteam now fails when a grader accepts a wrong answer, and when a question set has nothing gradeable (for example, judge checks only, without --judge).
• Known limits, each confirmed by a probe on the final code:
  • Threshold blocks: wrong, empty and hedge are graded on the non-negated checks only, so a block with threshold: 0.5 and [match Paris, match negate Lyon] accepts the planted wrong answer, and the red team reports pass. The docs state this.
  • A negated check that the planted wrong answer trips is a false block: [regex ".*", match negate <planted text>] is blocked, but the real grader rejects the planted answer.
  • A regex that matches the hedge text is not skipped; only literal values are checked for a clash with the hedge.
  • A negated mode: all check gets no stuffed probe at all, so a grader bug that accepts a stuffed answer for such a check is not caught. The skip is wider than the one case that false-blocks.
  • A choice check with several letters ("AB"): the canary ANSWER: AB is rejected by the grader, so the run blocks. This is a grader issue in validate_grading, not in the red team.
  • Exit 0 when any one question is probed, even if others are unprobed. The table and the summary show the unprobed count; the exit code does not.
  • With --judge, judge checks call the model. The wrong answer is a fixed sentinel string, not one written for each question.

🤖 Generated with Claude Code

[github-actions]

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

+ added command `dokimi eval redteam`
    + argument `questions`
    + option `--help/-h`
    + option `--judge`
    + option `--judge-model`
    + option `--judge-provider`
    + option `--output/-o`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 99.01961% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.56%. Comparing base (3cfe49c) to head (04c4541).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
dokimi/dokimi/commands/answer_formats.py	0.00%	2 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #281      +/-   ##
==========================================
+ Coverage   93.50%   93.56%   +0.05%     
==========================================
  Files         273      275       +2     
  Lines       35569    35766     +197     
  Branches     6040     6069      +29     
==========================================
+ Hits        33260    33465     +205     
+ Misses       1492     1488       -4     
+ Partials      817      813       -4     

Files with missing lines	Coverage Δ
dokimi/dokimi/artifacts.py	95.12% <ø> (ø)
dokimi/dokimi/commands/evaluate.py	89.93% <100.00%> (+0.06%)	⬆️
dokimi/dokimi/commands/redteam.py	100.00% <100.00%> (ø)
dokimi/dokimi/eval/judge.py	94.48% <100.00%> (+0.06%)	⬆️
dokimi/dokimi/eval/redteam.py	100.00% <100.00%> (ø)
dokimi/dokimi/eval/reference_checks.py	91.66% <100.00%> (+1.81%)	⬆️
dokimi/dokimi/commands/answer_formats.py	66.66% <0.00%> (+0.75%)	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[vlad-perevezentsev]

Review at e881ab5. The idea is useful and the plumbing is right, but the verdict is not yet trustworthy both ways: the stuffed probe blocks every match or fuzzy check with several aliases on a correct grader, and a threshold block whose negated check carries a wrong answer over the line still gets pass. Details are in the line comments.

What I ran, at e881ab5:

• make ci-local --offline: 9,686 passed, 0 failed, 57 skipped; every CI row passed or was skipped for the network.
• run() on small grading blocks: an alias pair under match (any and end) and fuzzy is blocked on stuffed; {threshold: 0.5, Paris + negated London} is pass while the real grade() accepts the planted wrong and the empty answer; a negated-only block is pass; five judge questions without --judge plus one match question is pass with 5 of 6 unprobed.

Checked, no issues

• Wiring: eval redteam is registered under eval, added to the judge-provider coverage test, and makes no model call without --judge; exit codes 1 (blocked, inconclusive) and 2 (bad input, unwritable -o) are tested.
• Schema and changelog: redteam.v1 is new, listed in SCHEMA_VERSIONS, with a fixture; the fragment is under Added and says what the command does. No existing score, default or stored key changes.
• New files: all carry the SPDX header.

[antonwolfy]

Review at e881ab5. The wiring, exit codes and the redteam.v1 schema hold up, but the verdict is not yet trustworthy: it blocks correct graders and passes weak ones. Two further problems are inline.

I agree with @vlad-perevezentsev's four points on the verdict (alias stuffing, threshold with a negated check, negated-only and unprobed runs reporting pass, wrong_for and numeric: true); I reproduced each. New evidence: {match, values: ["Paris", "Lyon"]} is blocked on stuffed, and {match, values: ["-7731.0"], numeric: true} is blocked on wrong.

What I ran, on a Linux review host at e881ab5:

• make ci-local: 27 passed, 1 failed, 7 skipped; 9,739 tests passed, 4 skipped; changed-line coverage passed. The failure is pip-audit, which reports advisories on filelock in the lock files; this PR does not touch them.
• Reproductions for the code findings, with the LLM transport mocked.
• A 107-mutant run on tests/test_redteam.py: 37 mutants pass the full suite; exit codes and the status verdict are pinned, the canary text and the clash logic are not.
• Commits: all six carry Signed-off-by.

Checked, no issues

• Wiring: eval redteam is registered, a default run makes no model call, exit 2 on a missing, unreadable or non-JSON question file and on an unwritable -o.
• Schema: -o output validates against redteam.v1; the kind is in SCHEMA_VERSIONS; the fixture and the changelog fragment are present; mkdocs build --strict passes.

[vchamarthi]

🔍 OpenCodeReview found 2 issue(s) in this PR.

By severity: 1 high, 1 low

Severity	Category	Location	Finding
high	bug	redteam.py:197-198	must-fix: a probe is recorded as accepted even when part of the block was never scored.
low	bug	redteam.py:34-36	nit: a deeply nested questions file makes json.loads in load_questions raise RecursionError.

Review details

• Commit reviewed: c68a1f1 against merge base f0f95a2
• Coverage: 12 files in 3 groups
• Engine: OpenCodeReview v1.13.3, us.anthropic.claude-sonnet-5-5
• Usage: 730k tokens (556k from cache), 1m18s
• Tool calls: 34 (19 code_search, 14 file_read, 1 code_comment)
• Redteam feature core: command, eval logic, schema, artifacts, tests, changelog: 6 files
• Evaluate/judge integration with redteam and related test: 3 files
• Answer formats docs and related command/reference check changes: 3 files

[napetrov]

The findings in this review are all confirmed on e881ab5 and fixed, each answered in its thread: aliases c1596ef, threshold 6b6aa2f, negated-only a296b5a, unprobed a296b5a, numeric wrong d96f87d, hedge d96f87d, judge fields 4d7cef6, shared helpers 70e66bd, -o line ef1ed29.

[napetrov]

The verdict findings it agrees with are fixed (see the replies in the vlad-perevezentsev threads) and its two own threads are answered: repeated ids 1b64508, canary tests f108f53. The pip-audit failure it reports is on filelock in the lock files; this PR changes no lock file (git diff --name-only against main lists none).

[napetrov]

The review has no body of its own; its five inline findings are answered in their threads: doc rows 89641b7 and 89641b7, command and empty-input tests 27828e4, judge tests 0918eea, artifact on blocked runs 1946f48.

[napetrov]

Nothing to change here: the interface diff lists exactly what this PR adds, the command eval redteam with questions, --judge, --judge-model, --judge-provider and --output/-o. The judge fields of this round add no option.

[napetrov]

Covered by the tests of 27828e4, 0918eea and 1946f48: the report is for e881ab5. Locally, test_redteam.py with test_judge_provider_default.py now runs 100% of the lines of dokimi/eval/redteam.py and dokimi/commands/redteam.py (--cov, term-missing); the --judge path, the exit-2 path and the artifact of a blocked run were not run before. The Codecov report will update on this push.

[napetrov]

The 5 inline findings it counts are answered in their threads (see 89641b7, 89641b7, 27828e4, 0918eea, 1946f48).

[vchamarthi]

nit: a deeply nested questions file makes json.loads in load_questions raise RecursionError. It is not caught here, so the command exits with a traceback and code 1, which CI reads as a blocked run. Other commands (arms, catalog) catch RecursionError alongside OSError and ValueError for this reason.

Suggestion:

Suggested change

	except (OSError, ValueError) as e:
	print(f"Error: {e}", file=sys.stderr)
	sys.exit(2)
	except (OSError, ValueError, RecursionError) as e:

[napetrov]

Fixed in 98ccb96. eval redteam now catches RecursionError next to OSError and ValueError when it reads the question set, so a set nested too deep to parse exits 2 with Error: ... on stderr instead of a traceback and exit 1. arms and catalog already catch it the same way.

Reproduced on the PR head with a 100,000-deep [[...]] file: the command died in json.loads with a RecursionError traceback and exit code 1. The new test test_command_exits_2_on_a_question_set_nested_too_deep_to_read fails on the old code and passes with the change.

[vchamarthi]

must-fix: a probe is recorded as accepted even when part of the block was never scored. Grade.passed ignores checks with no score (counted drops them). Take a block of a judge check plus a negated literal check, for example [{judge: X}, {match: Lyon, negate: true}], in the default run with no --judge. The wrong, empty and hedge answers are graded on the negated check alone, which they pass by design. passed comes back True, so each is reported as a violation: the run is blocked and exits 1 for a block that is not gameable. The same happens when a positive regex times out. Mark the probe unscored when any check was unjudged or unrun and the grade says pass.

Suggestion:

Suggested change

	g = grade(grading, question, answer, ask)
	out[name] = Probe(name, g.passed, _detail(g))
	g = grade(grading, question, answer, ask)
	# A pass built from only part of the block (judge checks without a judge, a timed-out regex) is not a measurement.
	passed = None if g.passed and (g.unjudged or g.unrun) else g.passed
	out[name] = Probe(name, passed, _detail(g))

[vlad-perevezentsev]

Blocking: Confirmed, and it has a second side that the suggested fix does not cover. I missed this in my first round.

The false block: [{judge}, {match: Lyon, negate: true}] without --judge is blocked on wrong, empty and hedge, because only the negated check is counted. This came in with 6b6aa2f, which now grades the whole block; before it the judge-only part gave None.

The false pass: {threshold: 0.5, checks: [{judge: names Paris}, {match: Paris}]} without --judge returns pass, exit 0, with the question counted as probed: the judge is dropped from the score, so wrong, empty and hedge score 0 and fail. With a judge that says yes to everything, the real grade() passes the planted wrong answer (0.5 >= 0.5), and run(ask=...) is blocked on all three. So the default run reads an unmeasured judge as a rejection.

Without a threshold a failed counted check is final (every check must pass), so only a pass is unsafe there. With a threshold neither result holds while a check has no score.

Suggestion:

g = grade(grading, question, answer, ask)
partial = g.unjudged or g.unrun
passed = None if partial and (g.passed or g.threshold is not None) else g.passed

plus tests for both blocks above (the first should be unscored, the second unprobed and so inconclusive).

[vlad-perevezentsev]

Review at c68a1f1. Every point of my first round is fixed and I re-ran each repro. One problem is left, and it goes both ways: when part of a block has no score (a judge check without --judge, a timed-out regex), the probe is still read as a measurement. I added the false-pass side to @vchamarthi's thread.

What I ran, at c68a1f1:

• make ci-local --offline: 10,553 passed, 0 failed, 57 skipped; every CI row passed or was skipped for the network.
• The round-one repros: alias pairs under match (any, begin, end) and fuzzy, and numeric -7731.0, are now pass. {threshold: 0.5, Paris + negated London} is now blocked on wrong, empty and hedge. A negated-only block and the "5 judge + 1 match" set are now inconclusive. A value one keeps its hedge probe.
• [judge, negated Lyon] without --judge is blocked (a false block). {threshold: 0.5, [judge, match Paris]} without --judge is pass, but blocked with a judge that says yes to everything.

Checked, no issues

• Stuffed answers: decoys are picked with the check's own grade_check, so the decoys never hold an accepted value. Checks that read a value anywhere in the answer get no probe.
• Repeated ids: each row is graded with its own block. grading_by_id still validates every row before any is graded.
• Shared helpers: check_values is a pure rename. judge_ask builds the same Ask that eval reference used. Nothing else imported _values, and the open PR #280 that edits reference_checks.py adds no new call.
• Artifact, docs and changelog: judge_model and judge_provider are required and nullable in redteam.v1, which no release has written yet. The docs table and the fragment now describe inconclusive and the stuffed rules as coded.

[exai-bot]

✅ OpenCodeReview: Review complete: 0 finding(s) across 13 selected item(s).

Review details

• Commit reviewed: 04c4541 against merge base 3cfe49c
• Coverage: 13 files in 5 groups
• Engine: OpenCodeReview v1.13.3, us.anthropic.claude-sonnet-5-5
• Usage: 716k tokens (508k from cache), 35s
• Tool calls: 28 (18 code_search, 9 file_read, 1 file_read_diff)
• Redteam feature core: command, eval logic, schema, artifact registration: 5 files
• Redteam tests and fixtures: 2 files
• Answer formats docs and command changes: 3 files
• Judge changes and related test: 2 files
• Changelog entry: 1 file

[exai-bot]

This code returns inconclusive (exit 1) when any graded question is unmeasured. The PR description says the opposite in two places: inconclusive (no question was probed) and the known limit "Exit 0 when any one question is probed, even if others are unprobed". The module docstring, schema and changelog match the code. Update the PR description, since the repo guide asks that exit codes be described accurately. A mixed set, such as one negated-only block among normal ones, fails CI.