Skip to content

Stop wasting AI-review turns on denied shell commands - #36

Merged
gobetti merged 1 commit into
mainfrom
adopt-full-shell-review
Jul 22, 2026
Merged

Stop wasting AI-review turns on denied shell commands#36
gobetti merged 1 commit into
mainfrom
adopt-full-shell-review

Conversation

@gobetti

@gobetti gobetti commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

What

Adopts into the reusable review workflow the permission model we've been piloting repo-locally on ht-json and ht-android. Every repo consumes this workflow, so this is the org-wide rollout of that design.

See it running

It's been live on both pilot repos — you can watch real runs and, at the end of each review job, the Post Harden runner egress step reports the egress audit trail (every host the runner contacted):

Why

On large or repetitive PRs the reviewer often ran out of turns and posted no review at all. The main avoidable cause: under the exclusive command allow-list, every shell construct it reached for that wasn't enumerated (find, sed, awk, python, echo, loops) was denied, and in headless CI each denial costs a turn plus a retry — that denial tax, on top of per-file re-analysis, pushed runs past max_turns into error_max_turns. This removes the tax, not the budget: a genuinely undersized max_turns can still exhaust a very large PR, but reviews stop burning turns on denials.

What changed

A full read shell, with the allow-list's other two jobs moved to fitter mechanisms:

  • Turn budget: --allowedTools 'Read,Glob,Grep,LS,Bash'find/sed/awk/python3/pipes/loops just work, so repeated per-file checks batch into one command.
  • Scope (no builds/tests): a narrow --disallowedTools deny-list of package managers, build/test runners, and network tools.
  • Safety: contents: read (was write); denies on the git add/commit/rm/push commands claude-code-action injects by default (deny beats allow from every source); a step-security/harden-runner egress step in audit mode (egress audit trail + known-bad-endpoint blocking); the ephemeral runner.
  • Prompt: environment-constraints section rewritten to match and to encourage batching.

Preserved unchanged: the workflow_call inputs (model, max_turns, timeout_minutes, extra_prompt), the ${{ inputs.extra_prompt }} append point, the rubric, and the instructions. actions/checkout stays at its current pin (@v5); version bumps are a separate per-repo concern.

Safety model

The deny-list and egress audit are defense-in-depth, not the boundary — a deny-list can't fence bash /dev/tcp or python sockets, and audit mode logs novel-host egress rather than blocking it. The real boundary is the trigger model, documented as a hard adoption precondition in the workflow header: consume this only on pull_request, never pull_request_target, and never send secrets to fork PRs — so fork/untrusted PRs get no secrets and the review can't even authenticate. Block-mode egress and OIDC federation (to drop the static ANTHROPIC_API_KEY) are optional future hardening, not part of this change.

Rollout

After this merges, each consuming repo adopts by restoring its claude.yml pull_request trigger and deleting the repo-local pilot claude-review.yml (currently parked on ht-json and ht-android). Repos that already call @main pick it up on their next PR.

Evidence

In a local A/B repro of ht-json's worst-case review (~9,500-line survey PR), the old allow-list completed 0 of 1 runs with 6 denials; this design completed 2 of 3 with 0 denials and identical findings. Since merging on ht-json, real production reviews run clean — success, under the turn cap, 0 permission denials, comment posted under contents: read (26/34/45 turns across three PRs).

🤖 Generated with Claude Code


View with Codesmith Autofix with Codesmith
Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.


📝 Auto-filled

On large or repetitive PRs the AI reviewer often ran out of turns
and posted no review at all. The main avoidable cause: under an
exclusive command allow-list every shell construct it reached for
that wasn't enumerated (find, sed, awk, python, echo, loops) was
denied, and in headless CI each denial cost a turn plus a retry.
That denial tax, on top of per-file re-analysis, pushed runs past
max_turns into error_max_turns. This removes the tax, not the budget
— a genuinely undersized max_turns can still exhaust a large PR.

Give the reviewer a full read shell (Read/Glob/Grep/LS + Bash) and
move the allow-list's other two jobs to fitter mechanisms:
- scope: a narrow deny-list of package managers, build/test runners,
  and network tools, instead of enumerating every safe command
- safety: contents dropped to read; denies on the git
  add/commit/rm/push commands the action injects by default (deny
  beats allow from every source); a harden-runner egress step in
  audit mode (egress audit trail + known-bad-endpoint blocking); the
  ephemeral runner. The real boundary is the trigger model,
  documented as an adoption precondition callers must honor
  (pull_request, never pull_request_target).

The prompt's environment-constraints section is rewritten to match
and to encourage batching repeated per-file checks. workflow_call
inputs (model, max_turns, timeout_minutes, extra_prompt) are
unchanged; callers adopt by restoring their claude.yml pull_request
trigger and deleting the repo-local pilot workflow.

Validated in repo-local pilots on ht-json and ht-android: reviews
that used to hit the cap now complete with zero permission denials.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@gobetti
gobetti merged commit 16fa62f into main Jul 22, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants