Skip to content

docs: publish BU Bench benchmark report - #436

Merged
beubax merged 13 commits into
mainfrom
codex/benchmark-report
Aug 25, 2026
Merged

docs: publish BU Bench benchmark report#436
beubax merged 13 commits into
mainfrom
codex/benchmark-report

Conversation

@beubax

@beubax beubax commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Description

Publishes the 100-task BU Bench results in the main README and adds a detailed, research-style benchmark report.

  • Adds a compact comparison table to the main README, placed above Learn More
  • Adds separate SVG charts for accuracy, total tokens, API-equivalent cost, and agent turns
  • Explains the code-based browser executor, snapshot pruning, and task-aware diff policy
  • Documents category results, the evaluation setup, judge rubric, limitations, and reproduction steps
  • Replaces the previous combined benchmark chart

Related issue: N/A

Type of Change

  • 🐛 Bug fix
  • ✨ New feature
  • 🌐 New site adapter
  • 📝 Documentation
  • ♻️ Refactor
  • 🔧 CI / build / tooling

Checklist

  • I ran the checks relevant to this PR
  • I updated tests or docs if needed
  • I included output or screenshots when useful
  • If I edited skill-src/, I ran make build and committed skills/

Checks

  • git diff --check origin/main...HEAD
  • xmllint --noout benchmarks/charts/*.svg
  • README metric, placement, and local-link assertions
  • uv run --project benchmarks python benchmarks/scripts/run_eval.py --help

The application test suite was also attempted after rebasing. All assertions passed except one unrelated upstream programmatic-runner isolation test that timed out locally; this PR changes documentation and SVG assets only.

Screenshots / Output

The benchmark report embeds the four SVG charts directly beneath their corresponding metrics.

@github-actions

github-actions Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

🟢 No documentation gap found — medium confidence

The automated review found no documentation gap in the supplied changes.

This review is advisory and does not block merging.

@beubax
beubax merged commit 1acd6a8 into main Aug 25, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant