Skip to content

Add yylo-benchmark registry card - #13

Open
InsightFactoryAPP wants to merge 2 commits into
Ariestar:mainfrom
InsightFactoryAPP:add-yylo-benchmark
Open

InsightFactoryAPP wants to merge 2 commits into
Ariestar:mainfrom
InsightFactoryAPP:add-yylo-benchmark

Conversation

@InsightFactoryAPP

Copy link
Copy Markdown
Contributor

Summary

YYLO Benchmark is a thin trusted-host experiment runner for agent tasks: independent attempts over historical Ledger tasks, supplied coding prompts, and workflows. Retained outputs can be evaluated later with different checks or judges; no combined score, winner, or capability inference is made. Categorized test + agent following the inspect-ai / swe-agent / deepeval entries. Sibling of the yylo CLI card (#9, merged); one tool per PR.

Contribution Rules & Checklist

  • No manual README edits: only data/tools/yylo-benchmark.yaml added; README regenerated locally for validation only and not included in the diff
  • Exact file naming: data/tools/yylo-benchmark.yaml, name: yylo-benchmark
  • Allowed effects: execute_code, network_access, read_files, write_files, secret_exposure — all from the whitelist
  • Guardrails: risk.level: high with two concrete guardrails
  • Local validation passed:
    python scripts/lint_registry.py   → Validated 461 tool cards.
    python scripts/generate_readme.py → diff contains only the new yylo-benchmark row + count badges
    

Disclosure

Self-submission by the YYLO team (affiliated), prepared with AI-agent assistance. Every summary / use_when / avoid_when phrase is quotable from the live README. Happy to adjust category, risk framing, or wording to fit the registry's conventions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants