Live site: https://clipbench.abhishekbr.com
Which speech-to-text should your video app use? clipbench is the independent benchmark for captions on real short-form video: accuracy, word timing, hallucination over music, and cost, in six languages.
Caption, clipping and dubbing apps pick a speech-to-text vendor today from numbers measured on phone calls, meetings and read speech, mostly in English. Short-form video is different: music under the voice, sound effects, jump cuts, fast stylised delivery, slang, several speakers, and people switching languages mid sentence. clipbench scores each system on what a caption product actually ships, on clips a person checked by ear, with every comparison backed by a paired statistical test and code anyone can rerun.
Status: results are in progress. No language is published yet: rankings appear only once a language has 100 verified clips scored by at least 6 providers (see below). The pipeline has run end to end on creator captions, which are reported as an unverified reference, never as a ranking.
The full pipeline ran live on 150 Japanese Shorts and clips from 41 channels: 11 systems (Whisper large-v3 and turbo on a laptop, Workers AI Whisper and Nova-3, Groq, Deepgram, AssemblyAI, ElevenLabs Scribe, Gemini Flash and YouTube's own auto captions), about 1,600 transcriptions including stability reruns plus 125 YouTube auto caption tracks, for under 2 USD of API spend. Gemini Flash and Workers AI Nova-3 hit their account quota after 11 and 28 clips, and 25 clips have no YouTube auto track, so those three cover fewer clips. Scored against creator captions, which are unverified, so this is a pipeline check rather than a ranking. What it already surfaced:
- Latency is easy to measure wrong. With ten providers uploading at once, one provider measured 38 s for a clip it returns in 4.6 s alone. clipbench now runs one provider at a time and records the mode.
- Creator captions favour YouTube. YouTube's auto captions scored closest to creator captions, most likely because many creators start from them. Only references corrected by ear are fair ground truth.
- Hallucination is real. One Whisper deployment without a language hint had a 20% hallucination rate (output characters beyond 1.5 times the reference length, per reference character), mostly repetition loops.
- References break. Ten clips failed on every system; at least one is Korean speech under a Japanese caption track. They go back to the review queue automatically.
- Data pipeline: clip lists from any public ids, caption tracks classified from YouTube metadata, stratified sampling (at most 5% per channel, 6 content categories), gentle audio fetching.
- Review tool: a local web app to correct captions by ear and tag music, speakers, names, formatting spans and code-switching, with keyboard shortcuts.
- Providers: 14 batch and 5 streaming adapters behind one interface, with retries, quota detection, pacing, a result cache, a hard budget cap and a verification gate that refuses unvetted paid providers.
- Scoring: language-aware normalisation (kana readings for Japanese, mixed-script Hindi), caption timing against forced alignment, hallucination over music, caption readiness, code-switch accuracy, multiple valid spellings, diarization, language ID, caption file readability, and an LLM judge that only publishes once it agrees with human labels.
- Statistics: bootstrap intervals validated by simulation, paired tests with Benjamini-Hochberg correction, channel-clustered sensitivity checks, and a publication gate on coverage and diversity.
- Site and API: a static site and a Cloudflare Worker that serve only what passed the gate, with a public JSON API and embeddable badges; a privacy check blocks any audio or caption text from being published.
| metric | why a caption app cares |
|---|---|
| CER or WER, per content category and with or without music | the words on screen |
| Caption timing error (median, p90, share of words within 200 ms) | whether a word lights up when it is said |
| Hallucination over non-speech (words per minute of music or silence) | invented lines over the beat |
| Caption readiness (punctuation F1, casing accuracy) | captions you can ship without an edit pass |
| Code-switch accuracy (English inside non-English clips) | real creator speech |
| Cost per audio hour, latency, real-time factor | the bill and the wait |
Languages: English (en), Japanese (ja), Korean (ko), Spanish (es), Hindi (hi) and French (fr). Hinglish is
not a separate language: it is Hindi speech mixed with English, scored inside hi (mixed Devanagari and
Latin script) and as code-switching. A language is published only with 100 or more verified clips, 6 or
more providers that each scored at least 90% of them, and a verified set that passes the diversity rules.
Until then the site shows progress, not a ranking.
How every number is made: METHODOLOGY.md. Every judgement call: DECISIONS.md.
make setup # uv sync with local Whisper, aligner and AWS extras
uv run clipbench data discover --lang en --channels datasets/en/channels.csv # find clips with creator captions
uv run clipbench data build && uv run clipbench data sample --lang en && uv run clipbench data fetch-audio
uv run clipbench review # verify references by ear at http://127.0.0.1:8765
make dry-run run report # cost estimate, run, score, report, site dataPaid providers need keys in .env (see .env.example); a provider without a key is skipped with a
message. The budget cap is budget_usd in clipbench.toml, and a run that would cross it does not start.
| command | does |
|---|---|
clipbench data import-ids <csv> --lang L |
add any list of public clip ids (Shorts, Reels, TikToks, windows) |
clipbench data discover --lang L --channels <csv> |
find Shorts with creator-uploaded captions from a channel list |
clipbench data fetch-info / build / sample / fetch-audio |
metadata, manifest, stratified sample, audio (one at a time, 10 s apart) |
clipbench data check --lang L |
diversity rules: 30+ channels, no channel above 5%, 6+ categories |
clipbench review |
review UI: correct the text, tag category, music spans, code-switch spans |
clipbench align |
forced-align verified text to the audio for word timing references |
clipbench run --providers all --tier verified [--lang L] [--dry-run] |
transcribe with every system, cached and budget guarded (one language per run; --lang is required when the sample holds several) |
clipbench score / errors / report / export <run_id|latest> |
scores, error analysis, report, web data |
clipbench publish --check |
fails if anything publishable holds audio or caption text |
Whisper large-v3 and turbo (local), Groq Whisper, Deepgram, AssemblyAI, ElevenLabs Scribe, Gemini Flash, Amazon Transcribe, YouTube automatic captions, and Wayin where its output exists. Models, prices and data policies as checked: docs/providers.md.
- Check the provider's current docs: model, endpoint, language parameter, word timestamps, batch price per minute, minimum billing, data use and opt-out flags.
- Add a section to
clipbench/providers/pricing.tomlwithmodel_id,usd_per_minute,min_billed_seconds,checked_url,checked_on,data_policy,opt_out. - Write
clipbench/providers/<name>.py: subclassHttpProvider(orProviderfor SDKs and local models), setidandenv_keys, implementtranscribe()returning word timings when the API has them, and set every opt-out flag the API offers. - Register it in
clipbench/providers/registry.pyand add its key to.env.example. - Test it with
httpx.MockTransport: the request, the parsed result, a 429 then success, and no key.
The clip lists are public: datasets/<lang>/clips.csv. Audio and captions are not, because they belong
to the creators. To rebuild a language:
uv run clipbench data import-ids datasets/en/clips.csv --lang en
uv run clipbench data fetch-info && uv run clipbench data build && uv run clipbench data fetch-audioClips that were deleted or made private since will be missing; every run records how many clips it scored.
web/ is the site: landing page, per-language progress and findings, leaderboard, compare, provider
picker for builders, clip browser and methodology, built only from the JSON that clipbench export
writes. It deploys to Cloudflare (static assets plus a small Worker for the public data API). See
docs/WEB.md and docs/V2.md.
Audio, caption text and references live in data/, which is gitignored (a test checks it). Only code,
clip ids, scores, aggregates, charts and error snippets of a few words are published. Audio is fetched
slowly, one file at a time. Providers that would train on API data are only used with their opt-out set.
make test # pytest, no keys needed
make lint # ruff
cd web && bun run test && bun run buildThe Japanese set was seeded from caption tracks collected by Vako.
License: MIT.