Skip to content

Add research communication and reproducibility standard - #1

Draft
samrathchadha wants to merge 3 commits into
mainfrom
agent/research-communication-standard-20260715
Draft

Add research communication and reproducibility standard#1
samrathchadha wants to merge 3 commits into
mainfrom
agent/research-communication-standard-20260715

Conversation

@samrathchadha

@samrathchadha samrathchadha commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

What changed

Adds durable research communication rules to AGENTS.md and CLAUDE.md, three read-only Claude Code review agents, and a caller for the shared deterministic PR evidence gate.

The plain-language rule is: say what happened, show where the evidence lives, and do not let the agent that wrote a result be the only agent that checks it.

Why

Recent code and benchmark PRs use inconsistent evidence, validation, and limitation formats. A concrete failure case is a polished result summary that states a percentage without preserving the exact run, sample construction, negative runs, or command needed to check it.

Evidence

Claim Status Evidence
Codex will load the repository review guidance OBSERVED Root AGENTS.md with an explicit Review guidelines section
Claude Code can discover the three project subagents OBSERVED .claude/agents/*.md using documented project-agent frontmatter
The shared PR gate is immutable for this branch revision OBSERVED Caller pins efficientsystemsinc/.github at full commit 73bde24d57f8e49be28b2c3b80735758da16ef18
These prompts will reduce unsupported claims HYPOTHESIZED Requires post-adoption measurement; prompt presence is not behavioral proof

Validation

  • Exact commands: git diff --check; go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.7 .github/workflows/pr-evidence-gate.yml; wc -c AGENTS.md CLAUDE.md
  • Outcome: PASS — no whitespace errors; caller workflow passes actionlint; AGENTS.md is 19,617 bytes, below Codex's default 32 KiB project-instruction limit.
  • Product tests: NOT RUN — policy, prompt, and workflow-only change with no product source or runtime configuration modified.
  • Baseline comparison: the prior draft had claim/reproducibility prose but no Feynman rule, ARS routing, independent empirical subagent, specialized code/writing reviewers, or deterministic caller workflow.

Risks and limitations

Prompt rules are non-deterministic and can be ignored or misapplied. The keyword gate can produce false positives and checks structure rather than truth. The empirical reviewer is a fresh procedural pass, not an independent scientific replication. The shared workflow currently pins an open draft-policy commit and must be repinned to the final merged policy SHA before this PR is merged.

Non-goals

This PR does not enable Claude, Copilot, or Codex billing; install a GitHub App; set API keys; scan source diffs for secrets; post review comments; change branch protection; run benchmarks; or merge code.

Reproducibility

Result-bearing: no — this changes review policy and agent prompts but introduces no empirical result.

AI assistance

Codex drafted and locally validated the policy changes. A human must review the prompt boundaries, GitHub permissions, false-positive profile, and final shared-workflow SHA before merge. No private model transcript or chain-of-thought is included.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant