Skip to content

Add AutoR agent preset - #11

Open
tangxiangru wants to merge 1 commit into
InternScience:mainfrom
tangxiangru:add-autor-agent
Open

Add AutoR agent preset#11
tangxiangru wants to merge 1 commit into
InternScience:mainfrom
tangxiangru:add-autor-agent

Conversation

@tangxiangru

Copy link
Copy Markdown

Adds a built-in preset for AutoR, following the Add Your Own Agent instructions.

What AutoR is

A stage-structured research harness that drives Claude Code or Codex CLI through an eight-stage pipeline (literature survey → hypothesis → study design → implementation → experimentation → analysis → writing → dissemination). Each stage boundary is normally a human approval gate; for benchmark runs a reviewer agent takes that role, so a run needs no human at any point.

Changes

  • evaluation/agents.json — the autor preset.
  • evaluation/static/logos/autor.svg — 64×64, matching the existing rh.svg house style.
  • README.md — the installation table, the supported-agents table, and the prose list of built-in agents.
"autor": {
  "label": "AutoR",
  "icon": "A",
  "logo": "/static/logos/autor.svg",
  "cmd": "python3 /abs/path/to/AutoR/rcb_agent.py --workspace <WORKSPACE> --prompt <PROMPT>"
}

AutoR is a repository checkout rather than an installed CLI, so the preset uses the /abs/path/to/ placeholder convention already used by the ResearchHarness entry in the same file.

Testing

Driven through TaskRunner with the preset exactly as written (with only the /abs/path/to/ prefix substituted for a real checkout):

preset loaded: AutoR | /static/logos/autor.svg
workspace: workspaces/Chemistry_000_20260806_035047
STATUS: completed | exit_code: 0
report.md: 3185 bytes

The run produced report/report.md at the benchmark path along with report/images/, code/ and outputs/, left data/ and related_work/ untouched, and streamed JSON lines to stdout so _agent_output.jsonl and the run browser's model detection both work. _meta.json came back completed from the harness as expected.

Notes

  • AutoR never reads stdin in this mode. An interactive prompt reachable during a benchmark run raises rather than blocking, which matters here because the harness passes the launching terminal's stdin through to the agent.
  • Setup for the AutoR side is documented at docs/researchclawbench.md.
  • This PR adds the preset only; no leaderboard results are included.

Happy to adjust the label, icon, logo, or command shape to whatever you prefer.

AutoR (https://github.com/tangxiangru/AutoR) is a stage-structured research
harness that drives Claude Code or Codex CLI through an eight-stage pipeline,
with a reviewer agent in place of the human approval gate so a run needs no
human at any point.

Follows the ResearchHarness convention of an `/abs/path/to/` placeholder,
since AutoR is a repository checkout rather than an installed CLI.

Verified locally through `TaskRunner` with the preset as written: workspace
created, agent launched, `status: completed`, exit code 0, and a
`report/report.md` written at the benchmark path along with `report/images/`,
`code/` and `outputs/`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants