Skip to content

Add rigorous review and notification policy harness - #1

Draft
samrathchadha wants to merge 2 commits into
mainfrom
agent/rigorous-review-harness-20260715
Draft

Add rigorous review and notification policy harness#1
samrathchadha wants to merge 2 commits into
mainfrom
agent/rigorous-review-harness-20260715

Conversation

@samrathchadha

@samrathchadha samrathchadha commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

What changed

Adds organization-wide contribution instructions, a default PR template, three read-only review agents, and a reusable deterministic PR evidence gate. It also stages a least-privilege GitHub-to-Slack design: event/channel matrix, read-only GitHub App and one-scope Slack manifests, exact authorization runbook, strict normalized-event schema, and a transport-free Slack renderer.

Why

Recent PRs frequently state validation or empirical conclusions without a consistent claim/evidence boundary. A reviewer should be able to see what was observed, what was inferred, and how to reproduce a result without reading agent transcripts. Engineering notifications need the same boundary so Slack does not become a noisy transcript feed or leak PR bodies, logs, prompts, or credentials.

Evidence

Claim Status Evidence
The policy checker catches incomplete result-bearing PR descriptions OBSERVED Nine unit tests, including missing reproducibility, hidden template text, and secret-shaped input
The Slack contract rejects raw body/transcript fields, secret-shaped text, external evidence links, arbitrary channel selection, and Slack markup injection OBSERVED Nine contract tests in tests/test_slack_notification_contract.py
Organization-level agent files use GitHub's documented /agents location OBSERVED GitHub custom-agent documentation linked in the review notes
The proposed notifier uses only documented read permissions and Slack chat:write OBSERVED Primary GitHub, Slack, and Claude documentation linked in docs/integrations/slack/README.md
The approach will reduce unsupported claims or notification noise HYPOTHESIZED Requires measured pilot after human approval; this PR does not claim a behavioral result

Validation

  • Exact commands: python3 -m py_compile scripts/pr_evidence_gate.py scripts/slack_notification_contract.py; python3 -m unittest discover -s tests -v; python3 -m json.tool docs/integrations/slack/github-app-manifest.template.json; python3 -m json.tool docs/integrations/slack/normalized-event.schema.json; python3 -m json.tool docs/integrations/slack/normalized-event.example.json; ruby -e 'require "yaml"; YAML.safe_load(File.read("docs/integrations/slack/slack-app-manifest.yaml"), aliases: false)'; git diff --check
  • Outcome: PASS — 18/18 unit tests passed locally; the JSON and YAML parse; the staged diff has no whitespace errors; GitHub's Policy self-test check passed on head 27b8e560cb36ffe1f428100aad710d4d77cee17b.
  • Baseline comparison: no prior organization-wide policy repository or sanitized Slack notification contract existed.

Risks and limitations

The deterministic gates check required structure and obvious unsafe shapes, not truth. Keyword-based research detection can produce false positives, and PR authors can mislabel work as not result-bearing. Agent reviews are non-deterministic and remain advisory. A complete message contract can still contain a false human-authored statement. GitHub Projects webhooks are public preview. The Slack/GitHub manifests are review templates, not proof that an undeployed broker is secure. The shared workflow must be pinned by full SHA from caller repositories.

Non-goals

This PR does not enable paid AI services, install Slack or GitHub Apps, grant OAuth, create tokens or webhooks, store secrets, send Slack messages, deploy the broker, alter repository protections, run product tests, post review comments, or merge any existing PR.

Reproducibility

Result-bearing: no — this is policy and tooling work, not an empirical result.

AI assistance

Codex drafted and locally validated the policy/tooling. A human must review the policy, app permissions, authorization screens, message/data boundary, false-positive profile, and measured rollout before merge or installation.

@samrathchadha samrathchadha changed the title Add rigorous organization-wide PR review harness Add rigorous review and notification policy harness Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant