Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added model-benchmarks/docs/assets/eq-public-after.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
66 changes: 66 additions & 0 deletions model-benchmarks/docs/eq-public-column-proposal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Proposal: render the public EQ-Bench score as its own column

**Status:** DRAFT — needs Nick's approval before it reaches `main`.
**Branch:** `benchmark/eq-public-display` · **Scope:** display logic only, zero data changes.
**Date:** 2026-07-26

## The problem

`model-data.json` holds 35 models. 28 of them carry a real, upstream-verified public EQ-Bench
v3 score. The page rendered only 12 of them, because the EQ column read a single field,
`v3_score`, which is Nick's locally-run 22-trait metric. Everything else showed an em-dash.

Measured on the live page before this change: **23 of 35 EQ cells rendered "—"** while the
data behind them existed and was verified. Every pass that added coverage widened that gap.

## What this change does

Splits one EQ column into two clearly-labelled ones.

| | Reads | Rubric | Renders |
|---|---|---|---|
| **EQ (local)** | `v3_score` | 22 traits, our own run (~$6/model) | 12 of 35 |
| **EQ (public)** | `public_rubric_0_100` | 17 traits, eqbench.com leaderboard | 28 of 35 |

Both are sortable. Blank cells sort to the bottom. Mobile cards gained a matching
"EQ (public)" tile and the old ambiguous "EQ" label became "EQ (local)".

## What it deliberately does NOT do

- **Does not merge, average, or convert between the two metrics.** They are different rubrics
and the offset is not even consistent in sign (Grok 4.20 is 68.55 local, 55.8 public).
Each column shows only its own source.
- **Does not fall back.** A model with no local score shows blank in the local column
forever, not the public number wearing a local label.
- **Does not touch `model-data.json`.** `git diff --name-only` covers exactly two files:
`js/app.js` and `index.html`.

Both column tooltips say in plain words that the rubrics differ and are not directly
comparable. The public tooltip names eqbench.com as the source. Trait values in the public
tooltip are shown as `x/20` because upstream traits are on a 0-20 scale, not 0-100.

## Evidence actually verified

Headless Chromium against a local server, reading the rendered DOM:

- Before: 35 rows, **23 EQ cells empty**, 12 rendered.
- After: 35 rows, **28 public scores rendered**, 12 local scores rendered.
- Every rendered number cross-checked cell-by-cell against `model-data.json`:
**0 mismatches across all 35 rows and both columns.**
- Sorting by EQ (public) verified descending (83.3, 82.6, 82.6, 82.4, 82.4 ...) with all
blanks at the bottom.
- Header row reads: Model, Reasoning, Coding, Agentic, EQ (local), EQ (public), Chat, Speed,
Cost, Context. `colspan` updated 9 → 10 in all three placeholder states.
- Test suite: 26/26 pass. Pre-commit hooks pass including Prettier.

Screenshots (tracked with this proposal):

- [Before: one EQ column, 23 blanks](assets/eq-public-before.png)
- [After: separate local and public EQ columns](assets/eq-public-after.png)

## Open question for Nick

The top of the page by public score is Claude Opus 4.8 at 83.3, and the local column's top is
GPT-5.4 at 73.2. Two different models lead depending on the column. That is honest and it is
also a story worth telling in prose. Want a short explainer under the table saying why two EQ
columns exist and why they disagree? That is prose, so it is not in this diff.
19 changes: 15 additions & 4 deletions model-benchmarks/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -475,11 +475,22 @@ <h1 class="text-3xl md:text-4xl lg:text-5xl font-display font-semibold mb-4">
>
</th>
<th class="sortable has-tooltip eq-col" data-sort="eq_score">
EQ
EQ (local)
<span class="col-tooltip"
>EQ-Bench v3 score (0-100) — measures emotional intelligence
including empathy, insight, social dexterity, and warmth. Hover
for Elo ranking and trait breakdown.</span
>EQ-Bench v3 run locally by us — 22-trait rubric, scale 0–100.
Measures emotional intelligence including empathy, insight, social
dexterity, and warmth. Hover a score for Elo and trait breakdown.
Not comparable to the public column: different rubric.</span
>
</th>
<th class="sortable has-tooltip eq-col" data-sort="eq_public">
EQ (public)
<span class="col-tooltip"
>Score as published on the public EQ-Bench v3 leaderboard at
eqbench.com — 17-trait rubric, scale 0–100. A different rubric
from our local column, so the two are not directly comparable and
are never averaged or substituted for one another. Blank means no
published result exists.</span
>
</th>
<th class="sortable has-tooltip" data-sort="arena_elo">
Expand Down
47 changes: 43 additions & 4 deletions model-benchmarks/js/app.js
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@
async function init() {
const tbody = document.getElementById("table-body");
const cards = document.getElementById("mobile-cards");
tbody.innerHTML = `<tr><td colspan="9" class="text-center py-8 text-of-muted">Loading models...</td></tr>`;
tbody.innerHTML = `<tr><td colspan="10" class="text-center py-8 text-of-muted">Loading models...</td></tr>`;
cards.innerHTML = `<div class="text-center py-8 text-of-muted">Loading models...</div>`;

try {
Expand All @@ -61,7 +61,7 @@
bindEvents();
render();
} catch (err) {
tbody.innerHTML = `<tr><td colspan="9" class="text-center py-8 text-of-muted">Failed to load model data. Try refreshing the page.</td></tr>`;
tbody.innerHTML = `<tr><td colspan="10" class="text-center py-8 text-of-muted">Failed to load model data. Try refreshing the page.</td></tr>`;
cards.innerHTML = `<div class="text-center py-8 text-of-muted">Failed to load model data. Try refreshing the page.</div>`;
}
}
Expand Down Expand Up @@ -132,6 +132,8 @@
return model.scores?.agentic ?? -1;
case "eq_score":
return model.benchmarks?.eq_bench?.v3_score ?? -1;
case "eq_public":
return model.benchmarks?.eq_bench?.public_rubric_0_100 ?? -1;
case "arena_elo":
return model.benchmarks?.arena?.elo ?? -1;
case "speed":
Expand Down Expand Up @@ -233,6 +235,38 @@
</span>`;
}

function eqPublicHtml(model) {
const eq = model.benchmarks?.eq_bench;
const val = eq?.public_rubric_0_100;
if (val == null) return `<span class="score-na">—</span>`;

const tooltipParts = ["Public EQ-Bench v3 leaderboard (17-trait rubric)"];
if (eq.public_elo_norm)
tooltipParts.push(`Elo: ${Math.round(eq.public_elo_norm)}`);

const traitKeys = [
["Empathy", "demonstrated_empathy"],
["Insight", "depth_of_insight"],
["Social", "social_dexterity"],
["Warmth", "warmth"],
["Humanlike", "humanlike"],
];
const traits = traitKeys
.map(([label, key]) => {
const v = eq.public_traits_17?.[key];
return v != null ? `${label}: ${v}/20` : null;
})
.filter(Boolean)
.join(" · ");
if (traits) tooltipParts.push(traits);

return `
<span class="eq-detail">
<span class="score-value ${scoreTier(val)}">${val.toFixed(1)}</span>
<span class="eq-tooltip">${tooltipParts.join("<br>")}</span>
</span>`;
}

function costHtml(pricing) {
const blended = pricing?.blended_per_m;
if (blended === 0 || blended == null) {
Expand Down Expand Up @@ -273,7 +307,7 @@
function renderTable(list) {
const tbody = document.getElementById("table-body");
if (!list.length) {
tbody.innerHTML = `<tr><td colspan="9" class="text-center py-8 text-of-muted">No models match your filters.</td></tr>`;
tbody.innerHTML = `<tr><td colspan="10" class="text-center py-8 text-of-muted">No models match your filters.</td></tr>`;
return;
}
tbody.innerHTML = list
Expand All @@ -290,6 +324,7 @@
<td class="score-cell">${scoreHtml(m.scores?.coding)}</td>
<td class="score-cell">${scoreHtml(m.scores?.agentic)}</td>
<td class="score-cell eq-cell">${eqHtml(m)}</td>
<td class="score-cell eq-cell">${eqPublicHtml(m)}</td>
<td class="score-cell">${arenaHtml(m)}</td>
<td class="score-cell">${speedHtml(m.speed)}</td>
<td class="score-cell">${costHtml(m.pricing)}</td>
Expand Down Expand Up @@ -332,9 +367,13 @@
<div class="card-score-value ${scoreTier(m.scores?.agentic)}">${m.scores?.agentic != null ? Math.round(m.scores.agentic) : "—"}</div>
</div>
<div class="card-score-item">
<div class="card-score-label">EQ</div>
<div class="card-score-label">EQ (local)</div>
<div class="card-score-value ${scoreTier(m.benchmarks?.eq_bench?.v3_score)}">${m.benchmarks?.eq_bench?.v3_score != null ? m.benchmarks.eq_bench.v3_score.toFixed(1) : "—"}</div>
</div>
<div class="card-score-item">
<div class="card-score-label">EQ (public)</div>
<div class="card-score-value ${scoreTier(m.benchmarks?.eq_bench?.public_rubric_0_100)}">${m.benchmarks?.eq_bench?.public_rubric_0_100 != null ? m.benchmarks.eq_bench.public_rubric_0_100.toFixed(1) : "—"}</div>
</div>
<div class="card-score-item">
<div class="card-score-label">Chat</div>
<div class="card-score-value">${m.benchmarks?.arena?.elo ?? "—"}</div>
Expand Down
Loading