Skip to content

fix(benchmarks): plug three eq_bench_public data-integrity gaps #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
TechNickAI wants to merge 1 commit into from
Closed

fix(benchmarks): plug three eq_bench_public data-integrity gaps #22

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 18 additions & 1 deletion model-benchmarks/scripts/fetch-model.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -452,6 +452,13 @@ def apply_public_eq(data: dict, only_ids: list[str] | None = None) -> tuple[int,
block = eqbench_public.public_eq_block(m["id"], leaderboard, traits)
if block is None:
missing += 1
# Clear stale public fields so outdated data isn't published after a
# mapped row disappears or is renamed upstream.
eq = m.get("benchmarks", {}).get("eq_bench", {})
if any(eq.get(f) is not None for f in eqbench_public.PUBLIC_EQ_FIELDS):
for f in eqbench_public.PUBLIC_EQ_FIELDS:
eq.pop(f, None)
m.get("sources", {}).pop("eq_bench_public", None)
continue
eq = m.setdefault("benchmarks", {}).setdefault("eq_bench", {})
before = {k: eq.get(k) for k in eqbench_public.PUBLIC_EQ_FIELDS}
Expand Down Expand Up @@ -529,7 +536,7 @@ def merge_model(existing_data: dict, new_model: dict) -> dict:
if not new_model.get("speed") and m.get("speed"):
new_model["speed"] = m["speed"]
# Preserve source flags for non-openrouter sources
for src_key in ("artificial_analysis", "eq_bench"):
for src_key in ("artificial_analysis", "eq_bench", "eq_bench_public"):
if m.get("sources", {}).get(src_key) and not new_model["sources"].get(
src_key
):
Expand Down Expand Up @@ -746,6 +753,7 @@ def main():

data = load_model_data()

discovered = []
if args.discover:
discovered = discover_new_models(data, all_models)
if discovered:
Expand Down Expand Up @@ -822,6 +830,15 @@ def main():
f" Public EQ applied to {updated} model(s); "
f"{missing} have no hand-verified public entry (left empty)."
)
elif discovered:
# --discover gates on EQ-Bench map entries, so every discovered model
# already has a verified mapping — fetch their public scores automatically.
print(f"Fetching public EQ-Bench scores for {len(discovered)} discovered model(s)...")
updated, missing = apply_public_eq(data, only_ids=set(discovered))

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Include dry-run discoveries before applying public EQ

When --discover is combined with --dry-run, the fetch loop only prints each transformed model and never merges it into data, so this call filters over a dataset containing none of the IDs in discovered. It therefore performs both EQ-Bench requests but always reports zero updates and shows no public fields for the new models, preventing the dry run from validating the enrichment this branch adds. Merge the temporary models into an in-memory copy for this step or skip/clearly report the enrichment during dry runs.

Useful? React with 👍 / 👎.

print(
f" Public EQ applied to {updated} model(s); "
f"{missing} have no hand-verified public entry (left empty)."
)

if not args.dry_run:
save_model_data(data)
Expand Down
Loading