Skip to content

agent-fleet 1.7.0: routing hook counts call sites; verifier reads claims files - #68

Merged
4nrry merged 1 commit into
mainfrom
agent-fleet-audit-fixes
Oct 27, 2026
Merged

4nrry merged 1 commit into
mainfrom
agent-fleet-audit-fixes

Conversation

@4nrry

@4nrry 4nrry commented Oct 27, 2026

Copy link
Copy Markdown
Owner

Fixes from an audit of agent-fleet with extension-audit (linter and checklist), plugin-dev's plugin-validator and skill-reviewer. Every finding below was checked in the code before it was changed.

Changes

  • Routing hook (hooks/check-routing.sh): one model: anywhere in a script let it all through. A single routed agent() plus an unrouted fan-out passed, and so did model: inside a comment. The hook now strips // comments and denies once when there are fewer overrides than agent() calls.
  • Workflow (workflows/deep-research.js):
    • Search and fetch now run at effort: "low". Before, they inherited the session's effort, which broke the skill's own rule.
    • Verify keeps the session effort on purpose: no run has measured contesting at a lower level.
    • The final return gains confirmed (claim, source, quote), so the report's quotes can be checked.
    • The PROVENANCE header lists these additions. The built-in body sha still matches Claude Code 2.1.311.
  • Verifier (verify_citations.py):
    • --claims reads fleet_worker's extract output or the workflow's confirmed, and scores each quote against its own URL, in any language. Before this, neither of the plugin's own producers gave it input it could read.
    • Only http and https are fetched. Loopback, private and link-local IP hosts are skipped. Before, file:// was fetched in normal runs.
    • n_skipped and skipped count the quotes that were never scored.
    • The self-test uses a stub fetcher and a TemporaryDirectory, so just check no longer leaves temp folders behind.
  • Worker (fleet_worker.py): a quote under 30 characters now fails, the verifier's own minimum. It was 20, so 20–29 character quotes were never checked.
  • Skill text (SKILL.md):
    • The ratio rule said the opposite of the verifier's docstring. Fixed.
    • The "search X only when…" rule moves from the hook header into the skill, so tools without the hook see it.
    • bench/ links now point at GitHub, since installed copies have no bench/.
    • Child effort is worded without the Workflow-only agent() syntax.
  • Small fixes:
    • Removed the stale "crew's grok and hermes tiers are xhigh"; they are medium since crew 0.15.0.
    • inject.sh anchors papers, so "newspapers" no longer fires the hook.
    • The README names crew as a dependency of the CLI workers.

Left out

  • Description rewrite: not done. Its trigger rate is measured, and a rewrite should come with a new trigger eval.
  • Validator's script + name bypass: not a bug. The Workflow tool gives scriptPath and script precedence over name, so a script always runs instead of the bare built-in.

Testing

  • just check: 108 checks, 0 failed, 0 warnings, run twice.
  • claude plugin validate --strict plugins/agent-fleet: passed.
  • verify_citations.py --self-test: passed. It covers six cases and needs no network.
    • A mutation run broke each new guard in turn (private hosts, file://, the 30-character minimum), and the self-test failed every time.
  • fleet_worker.py --self-test: passed.
  • Routing hook, fed sample JSON:
    • denied: partial routing, a comment-only model:, an unrouted script;
    • passed: a fully routed script with multi-line options, a script with no calls, {} and garbage input;
    • the fork via scriptPath is denied once, for its inherited scope and synthesis steps.
  • Trigger sets (check_hook.py from the 2026-10-26-pesquisa-trigger bench record): dev 14/15 and 0/13, blind-1 12/12 and 0/12, blind-2 12/15 and 0/15, aug 4/10 and 0/10. All unchanged from the record.
  • --claims against docs.python.org:
    • a verbatim sentence was found and an invented one came back absent;
    • a 57-character quote containing inline markup scored only 0.4, which is the existing matcher's behaviour and is not changed here.

🤖 Generated with Claude Code

…ims files

From an audit with extension-audit, plugin-dev's plugin-validator and
skill-reviewer, each finding checked in the code.

- check-routing.sh: one model: anywhere let a whole script through, so one
  routed agent() plus an unrouted fan-out passed, and so did "model:" in a
  comment. It now strips // comments and denies once when overrides are fewer
  than agent() calls.
- deep-research.js: search and fetch run at effort "low"; unset, they
  inherited the session's, against the skill's own rule. Verify keeps the
  session effort, on purpose: no run has measured contesting lower. The final
  return also carries `confirmed` (claim, source, quote).
- verify_citations.py: --claims reads fleet_worker's extract output or the
  workflow's `confirmed` and scores each quote against its own URL, in any
  language; it used to accept only a markdown report. Only http(s) is fetched,
  and loopback, private and link-local IP hosts are skipped (file:// was
  fetched in normal runs). n_skipped counts quotes that were never scored. The
  self-test uses a stub fetcher and a temporary directory it removes.
- fleet_worker.py: a quote under 30 characters fails, the verifier's own
  minimum (it was 20, so 20-29 were never checked).
- SKILL.md: the ratio rule said the opposite of the verifier's docstring; the
  X rule moves from the hook text into the skill; bench links point at GitHub.
- Stale "crew's grok and hermes tiers are xhigh" removed (medium since crew
  0.15.0); inject.sh anchors "papers" so "newspapers" no longer fires.

The trigger sets in bench/.../2026-10-26-pesquisa-trigger score as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@4nrry
4nrry merged commit 68cd264 into main Oct 27, 2026
2 checks passed
@4nrry
4nrry deleted the agent-fleet-audit-fixes branch October 27, 2026 20:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant