Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,11 @@ repos:
args: [--allow-multiple-documents]
- id: trailing-whitespace
name: Trim whitespace
# llms.txt is generated by scripts/fetch-model.py from upstream
# OpenRouter descriptions, which carry trailing whitespace and smart
# quotes. Every pipeline run reintroduces them, so a formatting hook
# here fails permanently rather than catching anything.
exclude: ^model-benchmarks/llms\.txt$

- repo: https://github.com/Lucas-C/pre-commit-hooks
rev: v1.5.5
Expand All @@ -34,6 +39,9 @@ repos:
hooks:
- id: fix-smartquotes
name: Fix smart quotes
# Same reason as trailing-whitespace above: llms.txt is regenerated
# from upstream model descriptions that contain smart quotes.
exclude: ^model-benchmarks/llms\.txt$
- id: forbid-bidi-controls
name: Forbid bidirectional text (security)

Expand Down
19 changes: 19 additions & 0 deletions .prettierignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Prettier ignore
#
# Script-generated JSON data files. These are written by
# `model-benchmarks/scripts/*.py` via `json.dump(..., indent=2)`, which expands
# every array to one element per line. Prettier collapses short arrays onto a
# single line, so the two formatters disagree permanently: every pipeline run
# rewrites the file, Prettier reformats it, and the pre-commit hook fails.
#
# That conflict left `main` red on six consecutive pre-commit runs (2026-07-25
# through 2026-07-28) and the failures were misattributed to whichever PR
# happened to be open, which is how PR #23 sat blocked for five review passes on
# a check that had nothing to do with it.
#
# These files are machine-written and machine-read; their formatting is decided
# by the generator, not by a human editing them. Prettier has no useful opinion
# here.
model-benchmarks/data/model-data.json
model-benchmarks/data/eqbench-upstream-keys.json

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep JSON validation for the ignored snapshot

When eqbench-upstream-keys.json becomes malformed—for example after manual conflict resolution or an interrupted write—this exclusion removes the only configured hook that parses the file as JSON. The benchmark test hook only selects changes under model-benchmarks/scripts or model-benchmarks/tests, and its snapshot tests use temporary files rather than this checked-in snapshot, so a data-only commit can pass pre-commit and then make the weekly watcher fail in load_snapshot(). Add a non-formatting JSON syntax check while excluding these files from Prettier.

Useful? React with 👍 / 👎.

model-benchmarks/data/eqbench-raw.json
Loading