Add research communication and reproducibility standard - #1
Draft
samrathchadha wants to merge 3 commits into
Draft
Add research communication and reproducibility standard#1samrathchadha wants to merge 3 commits into
samrathchadha wants to merge 3 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Adds durable research communication rules to
AGENTS.mdandCLAUDE.md, three read-only Claude Code review agents, and a caller for the shared deterministic PR evidence gate.The plain-language rule is: say what happened, show where the evidence lives, and do not let the agent that wrote a result be the only agent that checks it.
Why
Recent code and benchmark PRs use inconsistent evidence, validation, and limitation formats. A concrete failure case is a polished result summary that states a percentage without preserving the exact run, sample construction, negative runs, or command needed to check it.
Evidence
AGENTS.mdwith an explicitReview guidelinessection.claude/agents/*.mdusing documented project-agent frontmatterefficientsystemsinc/.githubat full commit73bde24d57f8e49be28b2c3b80735758da16ef18Validation
git diff --check;go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.7 .github/workflows/pr-evidence-gate.yml;wc -c AGENTS.md CLAUDE.mdAGENTS.mdis 19,617 bytes, below Codex's default 32 KiB project-instruction limit.Risks and limitations
Prompt rules are non-deterministic and can be ignored or misapplied. The keyword gate can produce false positives and checks structure rather than truth. The empirical reviewer is a fresh procedural pass, not an independent scientific replication. The shared workflow currently pins an open draft-policy commit and must be repinned to the final merged policy SHA before this PR is merged.
Non-goals
This PR does not enable Claude, Copilot, or Codex billing; install a GitHub App; set API keys; scan source diffs for secrets; post review comments; change branch protection; run benchmarks; or merge code.
Reproducibility
Result-bearing: no — this changes review policy and agent prompts but introduces no empirical result.
AI assistance
Codex drafted and locally validated the policy changes. A human must review the prompt boundaries, GitHub permissions, false-positive profile, and final shared-workflow SHA before merge. No private model transcript or chain-of-thought is included.