Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 12 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -380,21 +380,21 @@ R@1k tok = answer found within the first 1,000 tokens of retrieved context (chun
**RAG cost — 1,000 PDF files vs 1,000 JDF files.** Same pipeline, only the input format differs: chunks → embeddings → vector store → top-5 context → LLM. Tokens counted from the chunks each pipeline produces, dollars from published prices (`bench/prices.json`); the benchmark never calls a paid API.

<!-- bench:cost:start -->
| Per 1,000 documents | JDF · jdf chunk (section) | PDF · pdfplumber extract_text() · fixed 1000/200 |
| Per 10,000 documents | JDF · jdf chunk (section) | PDF · pdfplumber extract_text() · fixed 1000/200 |
|---|---:|---:|
| Accuracy · answer in first 1,000 tokens (BAAI/bge-small-en-v1.5) | **96.9%** | 75.0% |
| Accuracy · top-1 hit | **46.9%** | 34.9% |
| Chunks | **8,000** | 6,249 |
| Embedding tokens, initial index | **1,305,820** | 1,385,673 |
| Embedding cost · OpenAI text-embedding-3-small | **$0.0261** | $0.0277 |
| Local embedding time · bge-small-en-v1.5 (measured throughput) | **17.5 s** | 6.4 s |
| Vector-store payload | **5.8 MB** | 5.6 MB |
| Re-embed tokens when one paragraph changes in every document | **91,000** | 1,402,000 |
| Re-index cost · OpenAI text-embedding-3-small | **$0.0018** | $0.0280 |
| LLM input tokens per 1,000,000 queries (top-5 context) | **716,317,708** | 1,210,218,750 |
| LLM input cost · Claude Sonnet 5 input | **$1,432.64** | $2,420.44 |

1,000 files per format (24-document corpus cycled); tokens counted from each pipeline's chunks, embedding time measured on x86_64 + NVIDIA L40S, 2026-09-28. Other volumes (10,000 documents, 10M queries) are linear estimates, not measurements. Prices: [`bench/prices.json`](bench/prices.json). Method: [`bench/README.md`](bench/README.md).
| Chunks | **80,000** | 62,499 |
| Embedding tokens, initial index | **13,057,945** | 13,857,048 |
| Embedding cost · OpenAI text-embedding-3-small | **$0.2612** | $0.2771 |
| Local embedding time · bge-small-en-v1.5 (measured throughput) | **8.4 min** | 6.4 min |
| Vector-store payload | **57.5 MB** | 55.9 MB |
| Re-embed tokens when one paragraph changes in every document | **910,000** | 14,020,000 |
| Re-index cost · OpenAI text-embedding-3-small | **$0.0182** | $0.2804 |
| LLM input tokens per 10,000,000 queries (top-5 context) | **7,163,177,083** | 12,102,187,500 |
| LLM input cost · Claude Sonnet 5 input | **$14,326.35** | $24,204.38 |

10,000 files per format (24-document corpus cycled); tokens counted from each pipeline's chunks, embedding time measured on Apple M5, 2026-10-14. Other volumes are linear estimates, not measurements. Prices: [`bench/prices.json`](bench/prices.json). Method: [`bench/README.md`](bench/README.md).
<!-- bench:cost:end -->

Reproduce it — Python only, no Node toolchain:
Expand Down
2 changes: 1 addition & 1 deletion bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ Nothing from the corpus leaves the machine. The cost benchmark never calls a pai

**JDF-only properties.** Every committed chunk's `hash` is checked against `sha256(text)`; one paragraph of one document is edited and re-chunked ([`corpus/jdf-chunks.d01-edited.jsonl`](corpus/jdf-chunks.d01-edited.jsonl)) — the number of chunks whose hash changed is what `jdf embed --incremental` re-embeds. A PDF pipeline has no stable chunk identity, so any edit means re-extract, re-chunk, re-embed.

**RAG cost** (`cost_bench.py`). The corpus is cycled to N files per format (default 1,000). Both sides run the same pipeline — chunks → embeddings → vector store → top-5 context → LLM — and only the input format differs; the PDF column is the PDF pipeline that scored best in the accuracy run. Counted: chunks and embedding tokens for the initial index (ceil(chars/4), both sides), vector-store payload, re-index tokens when one paragraph changes in every document (JDF: only hash-changed chunks; PDF: the whole document), LLM input tokens for 1M queries with top-5 context. Measured: local embedding throughput on a real sample of each pipeline's chunks. Dollar rows use `prices.json`. Accuracy from the same run is repeated in the table so cost and quality sit side by side.
**RAG cost** (`cost_bench.py`). The corpus is cycled to N files per format (default 1,000; the published run uses `--files 10000`). Both sides run the same pipeline — chunks → embeddings → vector store → top-5 context → LLM — and only the input format differs; the PDF column is the PDF pipeline that scored best in the accuracy run. Counted: chunks and embedding tokens for the initial index (ceil(chars/4), both sides), vector-store payload, re-index tokens when one paragraph changes in every document (JDF: only hash-changed chunks; PDF: the whole document), LLM input tokens for the query volume (`--queries`, default 1M; the published run uses 10M) with top-5 context. Measured: local embedding throughput on a real sample of each pipeline's chunks. Dollar rows use `prices.json`. Accuracy from the same run is repeated in the table so cost and quality sit side by side.

## Real documents — bring your own corpus (`byoc.py`)

Expand Down
90 changes: 45 additions & 45 deletions bench/results/cost-latest.json
Original file line number Diff line number Diff line change
@@ -1,26 +1,26 @@
{
"date": "2026-09-28",
"date": "2026-10-14",
"machine": {
"cpu": "x86_64",
"platform": "Linux-7.0.0-1013-aws-x86_64-with-glibc2.43",
"python": "3.14.4"
"cpu": "Apple M5",
"platform": "macOS-27.0.1-arm64-arm-64bit-Mach-O",
"python": "3.14.6"
},
"files": 1000,
"files": 10000,
"corpusDocuments": 24,
"pdfPipeline": "pdf-pdfplumber-1000",
"pdfChunking": {
"chars": 1000,
"overlap": 200
},
"queries": 1000000,
"queries": 10000000,
"embedding": {
"provider": "st",
"model": "BAAI/bge-small-en-v1.5",
"version": "sentence-transformers 5.7.0 \u00b7 torch 2.14.0+cu130 \u00b7 cuda:0 (NVIDIA L40S)",
"version": "sentence-transformers 6.0.1 \u00b7 torch 2.14.0 \u00b7 mps:0 (mps:0)",
"measuredOn": "first 2000 unique chunks per pipeline",
"tokensPerSecond": {
"jdf": 74737.4806939267,
"pdfplumber": 217005.35584377687
"jdf": 25818.729679688102,
"pdfplumber": 36230.27857004535
}
},
"prices": {
Expand Down Expand Up @@ -77,7 +77,7 @@
"id": "jdf",
"label": "JDF \u00b7 jdf chunk (section)",
"version": "0.2.2",
"files": 1000,
"files": 10000,
"perDoc": {
"d01": {
"chunks": 8,
Expand Down Expand Up @@ -207,42 +207,42 @@
"mrr10": 0.6455915178571429,
"retriever": "dense:BAAI/bge-small-en-v1.5"
},
"chunks": 8000,
"embedTokens": 1305820,
"chunkStoreBytes": 5752468,
"chunks": 80000,
"embedTokens": 13057945,
"chunkStoreBytes": 57523843,
"embedUsd": {
"openai-text-embedding-3-small": 0.0261164,
"openai-text-embedding-3-large": 0.1697566,
"cohere-embed-v4": 0.1566984
"openai-text-embedding-3-small": 0.2611589,
"openai-text-embedding-3-large": 1.69753285,
"cohere-embed-v4": 1.5669534
},
"localEmbedSeconds": 17.472090146411816,
"localEmbedSeconds": 505.7547432425709,
"reindex": {
"chunksPerEditedDoc": 1,
"tokensPerEditedDoc": 91,
"tokensAllDocsEdited": 91000,
"tokensAllDocsEdited": 910000,
"usd": {
"openai-text-embedding-3-small": 0.00182,
"openai-text-embedding-3-large": 0.01183,
"cohere-embed-v4": 0.01092
"openai-text-embedding-3-small": 0.0182,
"openai-text-embedding-3-large": 0.1183,
"cohere-embed-v4": 0.1092
}
},
"query": {
"ctxTokensPerQuery": 716.3177083333334,
"queries": 1000000,
"inputTokens": 716317708.3333334,
"queries": 10000000,
"inputTokens": 7163177083.333334,
"usd": {
"claude-sonnet-5": 1432.6354166666667,
"claude-haiku-4-5": 716.3177083333334,
"claude-opus-5": 3581.588541666667,
"openai-gpt-4o-mini": 107.44765625000001
"claude-sonnet-5": 14326.354166666668,
"claude-haiku-4-5": 7163.177083333334,
"claude-opus-5": 35815.88541666667,
"openai-gpt-4o-mini": 1074.4765625
}
}
},
{
"id": "pdfplumber",
"label": "PDF \u00b7 pdfplumber extract_text() \u00b7 fixed 1000/200",
"version": "0.11.10",
"files": 1000,
"files": 10000,
"perDoc": {
"d01": {
"chunks": 6,
Expand Down Expand Up @@ -372,34 +372,34 @@
"mrr10": 0.5338748346560847,
"retriever": "dense:BAAI/bge-small-en-v1.5"
},
"chunks": 6249,
"embedTokens": 1385673,
"chunkStoreBytes": 5588311,
"chunks": 62499,
"embedTokens": 13857048,
"chunkStoreBytes": 55884436,
"embedUsd": {
"openai-text-embedding-3-small": 0.02771346,
"openai-text-embedding-3-large": 0.18013749,
"cohere-embed-v4": 0.16628075999999997
"openai-text-embedding-3-small": 0.27714096000000005,
"openai-text-embedding-3-large": 1.8014162400000002,
"cohere-embed-v4": 1.66284576
},
"localEmbedSeconds": 6.385432260932547,
"localEmbedSeconds": 382.47147267194345,
"reindex": {
"chunksPerEditedDoc": 6,
"tokensPerEditedDoc": 1402,
"tokensAllDocsEdited": 1402000,
"tokensAllDocsEdited": 14020000,
"usd": {
"openai-text-embedding-3-small": 0.02804,
"openai-text-embedding-3-large": 0.18226,
"cohere-embed-v4": 0.16823999999999997
"openai-text-embedding-3-small": 0.2804,
"openai-text-embedding-3-large": 1.8226,
"cohere-embed-v4": 1.6824
}
},
"query": {
"ctxTokensPerQuery": 1210.21875,
"queries": 1000000,
"inputTokens": 1210218750.0,
"queries": 10000000,
"inputTokens": 12102187500.0,
"usd": {
"claude-sonnet-5": 2420.4375,
"claude-haiku-4-5": 1210.21875,
"claude-opus-5": 6051.09375,
"openai-gpt-4o-mini": 181.5328125
"claude-sonnet-5": 24204.375,
"claude-haiku-4-5": 12102.1875,
"claude-opus-5": 60510.9375,
"openai-gpt-4o-mini": 1815.328125
}
}
}
Expand Down
6 changes: 3 additions & 3 deletions bench/src/render.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -123,13 +123,13 @@ const costTableHtml = cost ? ` <div class="bench-table-wrap"><table class
${costRows().map(([name, f]) => ` <tr><td>${esc(name)}</td>${costSides.map((s) => `<td${s.id === "jdf" ? ' class="is-jdf"' : ""}>${f(s)}</td>`).join("")}</tr>`).join("\n")}
</tbody>
</table></div>
<p class="rag-bench-note">${int(cost.files)} files per format = the ${cost.corpusDocuments}-document corpus cycled. Tokens are counted from the chunks each pipeline produces (ceil(chars/4), both sides); embedding time is measured throughput on ${esc(machineLabel(cost))} (${cost.date}) applied to the totals. Prices from <code>bench/prices.json</code> (as of ${esc(Object.values(cost.prices.embedding)[0].as_of)}); edit it for your provider — the benchmark never calls a paid API. The PDF column is the PDF pipeline that scored best in the accuracy run. <strong>Scaling:</strong> figures are measured at ${int(cost.files)} documents and ${int(cost.queries)} queries; anything at other volumes (10,000 documents, 10M queries) is a linear estimate, not a measurement — index and re-index costs scale with documents, query cost scales with questions asked. Re-index: JDF re-embeds only chunks whose content hash changed (<code>jdf embed --incremental</code>); a PDF has no chunk identity, so an edit means re-chunking and re-embedding the whole document.</p>` : "";
<p class="rag-bench-note">${int(cost.files)} files per format = the ${cost.corpusDocuments}-document corpus cycled. Tokens are counted from the chunks each pipeline produces (ceil(chars/4), both sides); embedding time is measured throughput on ${esc(machineLabel(cost))} (${cost.date}) applied to the totals. Prices from <code>bench/prices.json</code> (as of ${esc(Object.values(cost.prices.embedding)[0].as_of)}); edit it for your provider — the benchmark never calls a paid API. The PDF column is the PDF pipeline that scored best in the accuracy run. <strong>Scaling:</strong> figures are measured at ${int(cost.files)} documents and ${int(cost.queries)} queries; anything at other volumes is a linear estimate, not a measurement — index and re-index costs scale with documents, query cost scales with questions asked. Re-index: JDF re-embeds only chunks whose content hash changed (<code>jdf embed --incremental</code>); a PDF has no chunk identity, so an edit means re-chunking and re-embedding the whole document.</p>` : "";

// Cost summary for the hero card (JDF vs best-accuracy PDF parser = first PDF side).
const pdfCost = costSides.find((s) => s.id !== "jdf");
const jdfCost = costSides.find((s) => s.id === "jdf");
const costData = cost && pdfCost ? {
files: cost.files, ek: Object.keys(cost.prices.embedding)[0], lk: Object.keys(cost.prices.llm_input)[0],
files: cost.files, queries: cost.queries, ek: Object.keys(cost.prices.embedding)[0], lk: Object.keys(cost.prices.llm_input)[0],
embedLabel: cost.prices.embedding[Object.keys(cost.prices.embedding)[0]].label, llmLabel: cost.prices.llm_input[Object.keys(cost.prices.llm_input)[0]].label,
jdf: { acc: jdfCost.accuracy?.recallAt1000Tok, tokens: jdfCost.embedTokens, usd: jdfCost.embedUsd[Object.keys(cost.prices.embedding)[0]], reindexUsd: jdfCost.reindex.usd[Object.keys(cost.prices.embedding)[0]], queryUsd: jdfCost.query?.usd[Object.keys(cost.prices.llm_input)[0]] },
pdf: { label: pdfCost.label, acc: pdfCost.accuracy?.recallAt1000Tok, tokens: pdfCost.embedTokens, usd: pdfCost.embedUsd[Object.keys(cost.prices.embedding)[0]], reindexUsd: pdfCost.reindex.usd[Object.keys(cost.prices.embedding)[0]], queryUsd: pdfCost.query?.usd[Object.keys(cost.prices.llm_input)[0]] },
Expand Down Expand Up @@ -179,7 +179,7 @@ if (cost) {
`|---|${costSides.map(() => "---:").join("|")}|`,
...costRows().map(([name, f]) => `| ${name} | ${costSides.map((s) => (s.id === "jdf" ? `**${f(s)}**` : f(s))).join(" | ")} |`),
"",
`${int(cost.files)} files per format (${cost.corpusDocuments}-document corpus cycled); tokens counted from each pipeline's chunks, embedding time measured on ${machineLabel(cost)}, ${cost.date}. Other volumes (10,000 documents, 10M queries) are linear estimates, not measurements. Prices: [\`bench/prices.json\`](bench/prices.json). Method: [\`bench/README.md\`](bench/README.md).`,
`${int(cost.files)} files per format (${cost.corpusDocuments}-document corpus cycled); tokens counted from each pipeline's chunks, embedding time measured on ${machineLabel(cost)}, ${cost.date}. Other volumes are linear estimates, not measurements. Prices: [\`bench/prices.json\`](bench/prices.json). Method: [\`bench/README.md\`](bench/README.md).`,
].join("\n");
md = replaceBlock(md, "cost", mdCost);
}
Expand Down
Loading
Loading