Repository navigation

Autonomous maintenance: benign control split, dataset-integrity tests, eval harness & leaderboard - #1

Merged

ppradyoth merged 20 commits into

mainfrom

auto/oss-improvements

Aug 8, 2026

Merged

Autonomous maintenance: benign control split, dataset-integrity tests, eval harness & leaderboard#1
ppradyoth merged 20 commits into
mainfrom
auto/oss-improvements

Commits on Jun 26, 2026
- test: add dataset-integrity pytest suite + CI
  
  ppradyoth
  and
  claude
  committed
  02a1e7e
Commits on Jun 27, 2026
- feat: evaluation harness + baseline leaderboard
  claude
  committed
  6a949f4
Commits on Jun 28, 2026
- fix: make package installable via hatchling + entry-point regression tests
  claude
  committed
  0932868
Commits on Jun 29, 2026
- feat: add matched-benign control split + precision/FPR/balanced-accuracy metrics
  claude
  committed
  31997a3
Commits on Jul 1, 2026
- feat: add `tool_shadowing` attack category (MCP tool poisoning / rug pull)
  claude
  committed
  5d40907
Commits on Jul 2, 2026
- feat: add tool_definition_scanner detector (catches MCP tool-poisoning)
  claude
  committed
  6dd1086
Commits on Jul 3, 2026
- feat: severity-weighted detection score + per-severity leaderboard breakdown
  claude
  committed
  0175cd6
Commits on Jul 4, 2026
- feat: agentic_directive_scanner detector (catches goal-hijack & priv-esc)
  claude
  committed
  9fb7c71
Commits on Jul 5, 2026
- feat: per-injection-surface detection breakdown (scorer + leaderboard)
  claude
  committed
  7c4fd2e
Commits on Jul 6, 2026
- feat: Wilson 95% CIs on detection/FPR/balanced-accuracy + leaderboard significance note
  claude
  committed
  16172eb
Commits on Jul 7, 2026
- feat: Matthews correlation coefficient (MCC) metric + leaderboard column
  claude
  committed
  2730055
Commits on Jul 8, 2026
- feat: residual hard set — attacks that evade every discriminating detector
  claude
  committed
  2031b70
Commits on Jul 9, 2026
- feat: ensemble coverage — union ceiling + greedy minimal detector set
  claude
  committed
  b6af4d2
Commits on Jul 10, 2026
- feat: McNemar paired significance test for detector comparison
  claude
  committed
  3ec54f3
Commits on Jul 11, 2026
- feat(eval): bootstrap 95% CI for MCC — the one headline metric without one
  claude
  committed
  290897a
Commits on Jul 12, 2026
- feat(eval): control_channel_scanner — detect tool/MCP output that spoofs the platform's control channel
  claude
  committed
  82e096f
Commits on Jul 13, 2026
- feat(data): +14 frontier samples targeting the residual + fix stale result counts
  claude
  committed
  14995f6
Commits on Jul 14, 2026
- fix(lint): clear F541 in generation/stats.py + add a CI lint job covering generation/
  claude
  committed
  c67218a
Commits on Jul 15, 2026
- fix(schema): require string 'content' on every conversation turn
  claude
  committed
  82af232
Commits on Jul 21, 2026
- fix(detectors): scan tool-parameter descriptions, not just the top-level tool description
  claude
  committed
  3cadb3c

Commits

Commits on Jun 26, 2026

Commits on Jun 27, 2026

Commits on Jun 28, 2026

Commits on Jun 29, 2026

Commits on Jul 1, 2026

Commits on Jul 2, 2026

Commits on Jul 3, 2026

Commits on Jul 4, 2026

Commits on Jul 5, 2026

Commits on Jul 6, 2026

Commits on Jul 7, 2026

Commits on Jul 8, 2026

Commits on Jul 9, 2026

Commits on Jul 10, 2026

Commits on Jul 11, 2026

Commits on Jul 12, 2026

Commits on Jul 13, 2026

Commits on Jul 14, 2026

Commits on Jul 15, 2026

Commits on Jul 21, 2026