From 47d4d5e642e956159a588721ff4031d9c112c5f2 Mon Sep 17 00:00:00 2001 From: uurtech Date: Wed, 14 Oct 2026 15:21:36 +0400 Subject: [PATCH] bench: publish RAG cost at 10,000 files / 10M queries Re-run cost_bench.py --files 10000 --queries 10000000 (verify green), re-render site/README. Hero labels now follow the run's query volume instead of a hard-coded 1M; static copy says 10,000 files. Co-Authored-By: Claude Opus 5.5 --- README.md | 24 ++++----- bench/README.md | 2 +- bench/results/cost-latest.json | 90 +++++++++++++++++----------------- bench/src/render.mjs | 6 +-- docs/bench.json | 90 +++++++++++++++++----------------- docs/docs/benchmark.html | 26 +++++----- docs/index.html | 36 +++++++------- 7 files changed, 138 insertions(+), 136 deletions(-) diff --git a/README.md b/README.md index 589a8ee..6917e89 100644 --- a/README.md +++ b/README.md @@ -380,21 +380,21 @@ R@1k tok = answer found within the first 1,000 tokens of retrieved context (chun **RAG cost — 1,000 PDF files vs 1,000 JDF files.** Same pipeline, only the input format differs: chunks → embeddings → vector store → top-5 context → LLM. Tokens counted from the chunks each pipeline produces, dollars from published prices (`bench/prices.json`); the benchmark never calls a paid API. -| Per 1,000 documents | JDF · jdf chunk (section) | PDF · pdfplumber extract_text() · fixed 1000/200 | +| Per 10,000 documents | JDF · jdf chunk (section) | PDF · pdfplumber extract_text() · fixed 1000/200 | |---|---:|---:| | Accuracy · answer in first 1,000 tokens (BAAI/bge-small-en-v1.5) | **96.9%** | 75.0% | | Accuracy · top-1 hit | **46.9%** | 34.9% | -| Chunks | **8,000** | 6,249 | -| Embedding tokens, initial index | **1,305,820** | 1,385,673 | -| Embedding cost · OpenAI text-embedding-3-small | **$0.0261** | $0.0277 | -| Local embedding time · bge-small-en-v1.5 (measured throughput) | **17.5 s** | 6.4 s | -| Vector-store payload | **5.8 MB** | 5.6 MB | -| Re-embed tokens when one paragraph changes in every document | **91,000** | 1,402,000 | -| Re-index cost · OpenAI text-embedding-3-small | **$0.0018** | $0.0280 | -| LLM input tokens per 1,000,000 queries (top-5 context) | **716,317,708** | 1,210,218,750 | -| LLM input cost · Claude Sonnet 5 input | **$1,432.64** | $2,420.44 | - -1,000 files per format (24-document corpus cycled); tokens counted from each pipeline's chunks, embedding time measured on x86_64 + NVIDIA L40S, 2026-09-28. Other volumes (10,000 documents, 10M queries) are linear estimates, not measurements. Prices: [`bench/prices.json`](bench/prices.json). Method: [`bench/README.md`](bench/README.md). +| Chunks | **80,000** | 62,499 | +| Embedding tokens, initial index | **13,057,945** | 13,857,048 | +| Embedding cost · OpenAI text-embedding-3-small | **$0.2612** | $0.2771 | +| Local embedding time · bge-small-en-v1.5 (measured throughput) | **8.4 min** | 6.4 min | +| Vector-store payload | **57.5 MB** | 55.9 MB | +| Re-embed tokens when one paragraph changes in every document | **910,000** | 14,020,000 | +| Re-index cost · OpenAI text-embedding-3-small | **$0.0182** | $0.2804 | +| LLM input tokens per 10,000,000 queries (top-5 context) | **7,163,177,083** | 12,102,187,500 | +| LLM input cost · Claude Sonnet 5 input | **$14,326.35** | $24,204.38 | + +10,000 files per format (24-document corpus cycled); tokens counted from each pipeline's chunks, embedding time measured on Apple M5, 2026-10-14. Other volumes are linear estimates, not measurements. Prices: [`bench/prices.json`](bench/prices.json). Method: [`bench/README.md`](bench/README.md). Reproduce it — Python only, no Node toolchain: diff --git a/bench/README.md b/bench/README.md index d143e33..57c5569 100644 --- a/bench/README.md +++ b/bench/README.md @@ -57,7 +57,7 @@ Nothing from the corpus leaves the machine. The cost benchmark never calls a pai **JDF-only properties.** Every committed chunk's `hash` is checked against `sha256(text)`; one paragraph of one document is edited and re-chunked ([`corpus/jdf-chunks.d01-edited.jsonl`](corpus/jdf-chunks.d01-edited.jsonl)) — the number of chunks whose hash changed is what `jdf embed --incremental` re-embeds. A PDF pipeline has no stable chunk identity, so any edit means re-extract, re-chunk, re-embed. -**RAG cost** (`cost_bench.py`). The corpus is cycled to N files per format (default 1,000). Both sides run the same pipeline — chunks → embeddings → vector store → top-5 context → LLM — and only the input format differs; the PDF column is the PDF pipeline that scored best in the accuracy run. Counted: chunks and embedding tokens for the initial index (ceil(chars/4), both sides), vector-store payload, re-index tokens when one paragraph changes in every document (JDF: only hash-changed chunks; PDF: the whole document), LLM input tokens for 1M queries with top-5 context. Measured: local embedding throughput on a real sample of each pipeline's chunks. Dollar rows use `prices.json`. Accuracy from the same run is repeated in the table so cost and quality sit side by side. +**RAG cost** (`cost_bench.py`). The corpus is cycled to N files per format (default 1,000; the published run uses `--files 10000`). Both sides run the same pipeline — chunks → embeddings → vector store → top-5 context → LLM — and only the input format differs; the PDF column is the PDF pipeline that scored best in the accuracy run. Counted: chunks and embedding tokens for the initial index (ceil(chars/4), both sides), vector-store payload, re-index tokens when one paragraph changes in every document (JDF: only hash-changed chunks; PDF: the whole document), LLM input tokens for the query volume (`--queries`, default 1M; the published run uses 10M) with top-5 context. Measured: local embedding throughput on a real sample of each pipeline's chunks. Dollar rows use `prices.json`. Accuracy from the same run is repeated in the table so cost and quality sit side by side. ## Real documents — bring your own corpus (`byoc.py`) diff --git a/bench/results/cost-latest.json b/bench/results/cost-latest.json index fa205cf..7709794 100644 --- a/bench/results/cost-latest.json +++ b/bench/results/cost-latest.json @@ -1,26 +1,26 @@ { - "date": "2026-09-28", + "date": "2026-10-14", "machine": { - "cpu": "x86_64", - "platform": "Linux-7.0.0-1013-aws-x86_64-with-glibc2.43", - "python": "3.14.4" + "cpu": "Apple M5", + "platform": "macOS-27.0.1-arm64-arm-64bit-Mach-O", + "python": "3.14.6" }, - "files": 1000, + "files": 10000, "corpusDocuments": 24, "pdfPipeline": "pdf-pdfplumber-1000", "pdfChunking": { "chars": 1000, "overlap": 200 }, - "queries": 1000000, + "queries": 10000000, "embedding": { "provider": "st", "model": "BAAI/bge-small-en-v1.5", - "version": "sentence-transformers 5.7.0 \u00b7 torch 2.14.0+cu130 \u00b7 cuda:0 (NVIDIA L40S)", + "version": "sentence-transformers 6.0.1 \u00b7 torch 2.14.0 \u00b7 mps:0 (mps:0)", "measuredOn": "first 2000 unique chunks per pipeline", "tokensPerSecond": { - "jdf": 74737.4806939267, - "pdfplumber": 217005.35584377687 + "jdf": 25818.729679688102, + "pdfplumber": 36230.27857004535 } }, "prices": { @@ -77,7 +77,7 @@ "id": "jdf", "label": "JDF \u00b7 jdf chunk (section)", "version": "0.2.2", - "files": 1000, + "files": 10000, "perDoc": { "d01": { "chunks": 8, @@ -207,34 +207,34 @@ "mrr10": 0.6455915178571429, "retriever": "dense:BAAI/bge-small-en-v1.5" }, - "chunks": 8000, - "embedTokens": 1305820, - "chunkStoreBytes": 5752468, + "chunks": 80000, + "embedTokens": 13057945, + "chunkStoreBytes": 57523843, "embedUsd": { - "openai-text-embedding-3-small": 0.0261164, - "openai-text-embedding-3-large": 0.1697566, - "cohere-embed-v4": 0.1566984 + "openai-text-embedding-3-small": 0.2611589, + "openai-text-embedding-3-large": 1.69753285, + "cohere-embed-v4": 1.5669534 }, - "localEmbedSeconds": 17.472090146411816, + "localEmbedSeconds": 505.7547432425709, "reindex": { "chunksPerEditedDoc": 1, "tokensPerEditedDoc": 91, - "tokensAllDocsEdited": 91000, + "tokensAllDocsEdited": 910000, "usd": { - "openai-text-embedding-3-small": 0.00182, - "openai-text-embedding-3-large": 0.01183, - "cohere-embed-v4": 0.01092 + "openai-text-embedding-3-small": 0.0182, + "openai-text-embedding-3-large": 0.1183, + "cohere-embed-v4": 0.1092 } }, "query": { "ctxTokensPerQuery": 716.3177083333334, - "queries": 1000000, - "inputTokens": 716317708.3333334, + "queries": 10000000, + "inputTokens": 7163177083.333334, "usd": { - "claude-sonnet-5": 1432.6354166666667, - "claude-haiku-4-5": 716.3177083333334, - "claude-opus-5": 3581.588541666667, - "openai-gpt-4o-mini": 107.44765625000001 + "claude-sonnet-5": 14326.354166666668, + "claude-haiku-4-5": 7163.177083333334, + "claude-opus-5": 35815.88541666667, + "openai-gpt-4o-mini": 1074.4765625 } } }, @@ -242,7 +242,7 @@ "id": "pdfplumber", "label": "PDF \u00b7 pdfplumber extract_text() \u00b7 fixed 1000/200", "version": "0.11.10", - "files": 1000, + "files": 10000, "perDoc": { "d01": { "chunks": 6, @@ -372,34 +372,34 @@ "mrr10": 0.5338748346560847, "retriever": "dense:BAAI/bge-small-en-v1.5" }, - "chunks": 6249, - "embedTokens": 1385673, - "chunkStoreBytes": 5588311, + "chunks": 62499, + "embedTokens": 13857048, + "chunkStoreBytes": 55884436, "embedUsd": { - "openai-text-embedding-3-small": 0.02771346, - "openai-text-embedding-3-large": 0.18013749, - "cohere-embed-v4": 0.16628075999999997 + "openai-text-embedding-3-small": 0.27714096000000005, + "openai-text-embedding-3-large": 1.8014162400000002, + "cohere-embed-v4": 1.66284576 }, - "localEmbedSeconds": 6.385432260932547, + "localEmbedSeconds": 382.47147267194345, "reindex": { "chunksPerEditedDoc": 6, "tokensPerEditedDoc": 1402, - "tokensAllDocsEdited": 1402000, + "tokensAllDocsEdited": 14020000, "usd": { - "openai-text-embedding-3-small": 0.02804, - "openai-text-embedding-3-large": 0.18226, - "cohere-embed-v4": 0.16823999999999997 + "openai-text-embedding-3-small": 0.2804, + "openai-text-embedding-3-large": 1.8226, + "cohere-embed-v4": 1.6824 } }, "query": { "ctxTokensPerQuery": 1210.21875, - "queries": 1000000, - "inputTokens": 1210218750.0, + "queries": 10000000, + "inputTokens": 12102187500.0, "usd": { - "claude-sonnet-5": 2420.4375, - "claude-haiku-4-5": 1210.21875, - "claude-opus-5": 6051.09375, - "openai-gpt-4o-mini": 181.5328125 + "claude-sonnet-5": 24204.375, + "claude-haiku-4-5": 12102.1875, + "claude-opus-5": 60510.9375, + "openai-gpt-4o-mini": 1815.328125 } } } diff --git a/bench/src/render.mjs b/bench/src/render.mjs index 43ac4d1..3d5ee67 100644 --- a/bench/src/render.mjs +++ b/bench/src/render.mjs @@ -123,13 +123,13 @@ const costTableHtml = cost ? `
` ${costSides.map((s) => `${f(s)}`).join("")}`).join("\n")}
${esc(name)}
-

${int(cost.files)} files per format = the ${cost.corpusDocuments}-document corpus cycled. Tokens are counted from the chunks each pipeline produces (ceil(chars/4), both sides); embedding time is measured throughput on ${esc(machineLabel(cost))} (${cost.date}) applied to the totals. Prices from bench/prices.json (as of ${esc(Object.values(cost.prices.embedding)[0].as_of)}); edit it for your provider — the benchmark never calls a paid API. The PDF column is the PDF pipeline that scored best in the accuracy run. Scaling: figures are measured at ${int(cost.files)} documents and ${int(cost.queries)} queries; anything at other volumes (10,000 documents, 10M queries) is a linear estimate, not a measurement — index and re-index costs scale with documents, query cost scales with questions asked. Re-index: JDF re-embeds only chunks whose content hash changed (jdf embed --incremental); a PDF has no chunk identity, so an edit means re-chunking and re-embedding the whole document.

` : ""; +

${int(cost.files)} files per format = the ${cost.corpusDocuments}-document corpus cycled. Tokens are counted from the chunks each pipeline produces (ceil(chars/4), both sides); embedding time is measured throughput on ${esc(machineLabel(cost))} (${cost.date}) applied to the totals. Prices from bench/prices.json (as of ${esc(Object.values(cost.prices.embedding)[0].as_of)}); edit it for your provider — the benchmark never calls a paid API. The PDF column is the PDF pipeline that scored best in the accuracy run. Scaling: figures are measured at ${int(cost.files)} documents and ${int(cost.queries)} queries; anything at other volumes is a linear estimate, not a measurement — index and re-index costs scale with documents, query cost scales with questions asked. Re-index: JDF re-embeds only chunks whose content hash changed (jdf embed --incremental); a PDF has no chunk identity, so an edit means re-chunking and re-embedding the whole document.

` : ""; // Cost summary for the hero card (JDF vs best-accuracy PDF parser = first PDF side). const pdfCost = costSides.find((s) => s.id !== "jdf"); const jdfCost = costSides.find((s) => s.id === "jdf"); const costData = cost && pdfCost ? { - files: cost.files, ek: Object.keys(cost.prices.embedding)[0], lk: Object.keys(cost.prices.llm_input)[0], + files: cost.files, queries: cost.queries, ek: Object.keys(cost.prices.embedding)[0], lk: Object.keys(cost.prices.llm_input)[0], embedLabel: cost.prices.embedding[Object.keys(cost.prices.embedding)[0]].label, llmLabel: cost.prices.llm_input[Object.keys(cost.prices.llm_input)[0]].label, jdf: { acc: jdfCost.accuracy?.recallAt1000Tok, tokens: jdfCost.embedTokens, usd: jdfCost.embedUsd[Object.keys(cost.prices.embedding)[0]], reindexUsd: jdfCost.reindex.usd[Object.keys(cost.prices.embedding)[0]], queryUsd: jdfCost.query?.usd[Object.keys(cost.prices.llm_input)[0]] }, pdf: { label: pdfCost.label, acc: pdfCost.accuracy?.recallAt1000Tok, tokens: pdfCost.embedTokens, usd: pdfCost.embedUsd[Object.keys(cost.prices.embedding)[0]], reindexUsd: pdfCost.reindex.usd[Object.keys(cost.prices.embedding)[0]], queryUsd: pdfCost.query?.usd[Object.keys(cost.prices.llm_input)[0]] }, @@ -179,7 +179,7 @@ if (cost) { `|---|${costSides.map(() => "---:").join("|")}|`, ...costRows().map(([name, f]) => `| ${name} | ${costSides.map((s) => (s.id === "jdf" ? `**${f(s)}**` : f(s))).join(" | ")} |`), "", - `${int(cost.files)} files per format (${cost.corpusDocuments}-document corpus cycled); tokens counted from each pipeline's chunks, embedding time measured on ${machineLabel(cost)}, ${cost.date}. Other volumes (10,000 documents, 10M queries) are linear estimates, not measurements. Prices: [\`bench/prices.json\`](bench/prices.json). Method: [\`bench/README.md\`](bench/README.md).`, + `${int(cost.files)} files per format (${cost.corpusDocuments}-document corpus cycled); tokens counted from each pipeline's chunks, embedding time measured on ${machineLabel(cost)}, ${cost.date}. Other volumes are linear estimates, not measurements. Prices: [\`bench/prices.json\`](bench/prices.json). Method: [\`bench/README.md\`](bench/README.md).`, ].join("\n"); md = replaceBlock(md, "cost", mdCost); } diff --git a/docs/bench.json b/docs/bench.json index 238085e..ceb2126 100644 --- a/docs/bench.json +++ b/docs/bench.json @@ -20289,28 +20289,28 @@ ] }, "cost": { - "date": "2026-09-28", + "date": "2026-10-14", "machine": { - "cpu": "x86_64", - "platform": "Linux-7.0.0-1013-aws-x86_64-with-glibc2.43", - "python": "3.14.4" + "cpu": "Apple M5", + "platform": "macOS-27.0.1-arm64-arm-64bit-Mach-O", + "python": "3.14.6" }, - "files": 1000, + "files": 10000, "corpusDocuments": 24, "pdfPipeline": "pdf-pdfplumber-1000", "pdfChunking": { "chars": 1000, "overlap": 200 }, - "queries": 1000000, + "queries": 10000000, "embedding": { "provider": "st", "model": "BAAI/bge-small-en-v1.5", - "version": "sentence-transformers 5.7.0 · torch 2.14.0+cu130 · cuda:0 (NVIDIA L40S)", + "version": "sentence-transformers 6.0.1 · torch 2.14.0 · mps:0 (mps:0)", "measuredOn": "first 2000 unique chunks per pipeline", "tokensPerSecond": { - "jdf": 74737.4806939267, - "pdfplumber": 217005.35584377687 + "jdf": 25818.729679688102, + "pdfplumber": 36230.27857004535 } }, "prices": { @@ -20367,7 +20367,7 @@ "id": "jdf", "label": "JDF · jdf chunk (section)", "version": "0.2.2", - "files": 1000, + "files": 10000, "perDoc": { "d01": { "chunks": 8, @@ -20497,34 +20497,34 @@ "mrr10": 0.6455915178571429, "retriever": "dense:BAAI/bge-small-en-v1.5" }, - "chunks": 8000, - "embedTokens": 1305820, - "chunkStoreBytes": 5752468, + "chunks": 80000, + "embedTokens": 13057945, + "chunkStoreBytes": 57523843, "embedUsd": { - "openai-text-embedding-3-small": 0.0261164, - "openai-text-embedding-3-large": 0.1697566, - "cohere-embed-v4": 0.1566984 + "openai-text-embedding-3-small": 0.2611589, + "openai-text-embedding-3-large": 1.69753285, + "cohere-embed-v4": 1.5669534 }, - "localEmbedSeconds": 17.472090146411816, + "localEmbedSeconds": 505.7547432425709, "reindex": { "chunksPerEditedDoc": 1, "tokensPerEditedDoc": 91, - "tokensAllDocsEdited": 91000, + "tokensAllDocsEdited": 910000, "usd": { - "openai-text-embedding-3-small": 0.00182, - "openai-text-embedding-3-large": 0.01183, - "cohere-embed-v4": 0.01092 + "openai-text-embedding-3-small": 0.0182, + "openai-text-embedding-3-large": 0.1183, + "cohere-embed-v4": 0.1092 } }, "query": { "ctxTokensPerQuery": 716.3177083333334, - "queries": 1000000, - "inputTokens": 716317708.3333334, + "queries": 10000000, + "inputTokens": 7163177083.333334, "usd": { - "claude-sonnet-5": 1432.6354166666667, - "claude-haiku-4-5": 716.3177083333334, - "claude-opus-5": 3581.588541666667, - "openai-gpt-4o-mini": 107.44765625000001 + "claude-sonnet-5": 14326.354166666668, + "claude-haiku-4-5": 7163.177083333334, + "claude-opus-5": 35815.88541666667, + "openai-gpt-4o-mini": 1074.4765625 } } }, @@ -20532,7 +20532,7 @@ "id": "pdfplumber", "label": "PDF · pdfplumber extract_text() · fixed 1000/200", "version": "0.11.10", - "files": 1000, + "files": 10000, "perDoc": { "d01": { "chunks": 6, @@ -20662,34 +20662,34 @@ "mrr10": 0.5338748346560847, "retriever": "dense:BAAI/bge-small-en-v1.5" }, - "chunks": 6249, - "embedTokens": 1385673, - "chunkStoreBytes": 5588311, + "chunks": 62499, + "embedTokens": 13857048, + "chunkStoreBytes": 55884436, "embedUsd": { - "openai-text-embedding-3-small": 0.02771346, - "openai-text-embedding-3-large": 0.18013749, - "cohere-embed-v4": 0.16628075999999997 + "openai-text-embedding-3-small": 0.27714096000000005, + "openai-text-embedding-3-large": 1.8014162400000002, + "cohere-embed-v4": 1.66284576 }, - "localEmbedSeconds": 6.385432260932547, + "localEmbedSeconds": 382.47147267194345, "reindex": { "chunksPerEditedDoc": 6, "tokensPerEditedDoc": 1402, - "tokensAllDocsEdited": 1402000, + "tokensAllDocsEdited": 14020000, "usd": { - "openai-text-embedding-3-small": 0.02804, - "openai-text-embedding-3-large": 0.18226, - "cohere-embed-v4": 0.16823999999999997 + "openai-text-embedding-3-small": 0.2804, + "openai-text-embedding-3-large": 1.8226, + "cohere-embed-v4": 1.6824 } }, "query": { "ctxTokensPerQuery": 1210.21875, - "queries": 1000000, - "inputTokens": 1210218750, + "queries": 10000000, + "inputTokens": 12102187500, "usd": { - "claude-sonnet-5": 2420.4375, - "claude-haiku-4-5": 1210.21875, - "claude-opus-5": 6051.09375, - "openai-gpt-4o-mini": 181.5328125 + "claude-sonnet-5": 24204.375, + "claude-haiku-4-5": 12102.1875, + "claude-opus-5": 60510.9375, + "openai-gpt-4o-mini": 1815.328125 } } } diff --git a/docs/docs/benchmark.html b/docs/docs/benchmark.html index 5c581de..4f2f0f1 100644 --- a/docs/docs/benchmark.html +++ b/docs/docs/benchmark.html @@ -4,7 +4,7 @@ RAG benchmark — JDF vs PDF — JDF - + @@ -127,25 +127,25 @@

Embeddings — nomic-ai/nomic-embed-text-v1.5

RAG cost — 1,000 PDF files vs 1,000 JDF files

-

Same pipeline on both sides, only the input format differs: chunks → embeddings → vector store → top-5 context → LLM. The 24-document corpus is cycled to 1,000 files per format; embedding tokens are counted from the chunks each pipeline produces; embedding time is measured throughput; dollar figures are tokens × the published prices in bench/prices.json. The benchmark never calls a paid API.

+

Same pipeline on both sides, only the input format differs: chunks → embeddings → vector store → top-5 context → LLM. The 24-document corpus is cycled to 10,000 files per format; embedding tokens are counted from the chunks each pipeline produces; embedding time is measured throughput; dollar figures are tokens × the published prices in bench/prices.json. The benchmark never calls a paid API.

- + - - - - - - - - - + + + + + + + + +
Per 1,000 documentsJDF · jdf chunk (section)PDF · pdfplumber extract_text() · fixed 1000/200
Per 10,000 documentsJDF · jdf chunk (section)PDF · pdfplumber extract_text() · fixed 1000/200
Accuracy · answer in first 1,000 tokens (BAAI/bge-small-en-v1.5)96.9%75.0%
Accuracy · top-1 hit46.9%34.9%
Chunks8,0006,249
Embedding tokens, initial index1,305,8201,385,673
Embedding cost · OpenAI text-embedding-3-small$0.0261$0.0277
Local embedding time · bge-small-en-v1.5 (measured throughput)17.5 s6.4 s
Vector-store payload5.8 MB5.6 MB
Re-embed tokens when one paragraph changes in every document91,0001,402,000
Re-index cost · OpenAI text-embedding-3-small$0.0018$0.0280
LLM input tokens per 1,000,000 queries (top-5 context)716,317,7081,210,218,750
LLM input cost · Claude Sonnet 5 input$1,432.64$2,420.44
Chunks80,00062,499
Embedding tokens, initial index13,057,94513,857,048
Embedding cost · OpenAI text-embedding-3-small$0.2612$0.2771
Local embedding time · bge-small-en-v1.5 (measured throughput)8.4 min6.4 min
Vector-store payload57.5 MB55.9 MB
Re-embed tokens when one paragraph changes in every document910,00014,020,000
Re-index cost · OpenAI text-embedding-3-small$0.0182$0.2804
LLM input tokens per 10,000,000 queries (top-5 context)7,163,177,08312,102,187,500
LLM input cost · Claude Sonnet 5 input$14,326.35$24,204.38
-

1,000 files per format = the 24-document corpus cycled. Tokens are counted from the chunks each pipeline produces (ceil(chars/4), both sides); embedding time is measured throughput on x86_64 + NVIDIA L40S (2026-09-28) applied to the totals. Prices from bench/prices.json (as of 2026-09-12); edit it for your provider — the benchmark never calls a paid API. The PDF column is the PDF pipeline that scored best in the accuracy run. Scaling: figures are measured at 1,000 documents and 1,000,000 queries; anything at other volumes (10,000 documents, 10M queries) is a linear estimate, not a measurement — index and re-index costs scale with documents, query cost scales with questions asked. Re-index: JDF re-embeds only chunks whose content hash changed (jdf embed --incremental); a PDF has no chunk identity, so an edit means re-chunking and re-embedding the whole document.

+

10,000 files per format = the 24-document corpus cycled. Tokens are counted from the chunks each pipeline produces (ceil(chars/4), both sides); embedding time is measured throughput on Apple M5 (2026-10-14) applied to the totals. Prices from bench/prices.json (as of 2026-09-12); edit it for your provider — the benchmark never calls a paid API. The PDF column is the PDF pipeline that scored best in the accuracy run. Scaling: figures are measured at 10,000 documents and 10,000,000 queries; anything at other volumes is a linear estimate, not a measurement — index and re-index costs scale with documents, query cost scales with questions asked. Re-index: JDF re-embeds only chunks whose content hash changed (jdf embed --incremental); a PDF has no chunk identity, so an edit means re-chunking and re-embedding the whole document.

Method

diff --git a/docs/index.html b/docs/index.html index 8d1cc5b..4434ff4 100644 --- a/docs/index.html +++ b/docs/index.html @@ -107,7 +107,7 @@

JDF
JSON document standard

RAG retrieval — same reports as PDF vs JDF
-
1,000 documents · same RAG pipeline · JDF vs the best PDF pipeline
+
10,000 documents · same RAG pipeline · JDF vs the best PDF pipeline
@@ -149,12 +149,12 @@

JDF
JSON document standard

- Full tables, method, RAG cost at 1,000 files → + Full tables, method, RAG cost at 10,000 files → pip install -r requirements.txt && python rag_bench.py reproduces it · --verify checks it
@@ -374,22 +374,22 @@

RAG cost — 1,000 PDF files vs 1,000 JDF files

Same pipeline, only the input format differs: chunks → embeddings → vector store → top-5 context → LLM. Tokens counted from the chunks each pipeline produces, dollars from published prices; the benchmark never calls a paid API.

- + - - - - - - - - - + + + + + + + + +
Per 1,000 documentsJDF · jdf chunk (section)PDF · pdfplumber extract_text() · fixed 1000/200
Per 10,000 documentsJDF · jdf chunk (section)PDF · pdfplumber extract_text() · fixed 1000/200
Accuracy · answer in first 1,000 tokens (BAAI/bge-small-en-v1.5)96.9%75.0%
Accuracy · top-1 hit46.9%34.9%
Chunks8,0006,249
Embedding tokens, initial index1,305,8201,385,673
Embedding cost · OpenAI text-embedding-3-small$0.0261$0.0277
Local embedding time · bge-small-en-v1.5 (measured throughput)17.5 s6.4 s
Vector-store payload5.8 MB5.6 MB
Re-embed tokens when one paragraph changes in every document91,0001,402,000
Re-index cost · OpenAI text-embedding-3-small$0.0018$0.0280
LLM input tokens per 1,000,000 queries (top-5 context)716,317,7081,210,218,750
LLM input cost · Claude Sonnet 5 input$1,432.64$2,420.44
Chunks80,00062,499
Embedding tokens, initial index13,057,94513,857,048
Embedding cost · OpenAI text-embedding-3-small$0.2612$0.2771
Local embedding time · bge-small-en-v1.5 (measured throughput)8.4 min6.4 min
Vector-store payload57.5 MB55.9 MB
Re-embed tokens when one paragraph changes in every document910,00014,020,000
Re-index cost · OpenAI text-embedding-3-small$0.0182$0.2804
LLM input tokens per 10,000,000 queries (top-5 context)7,163,177,08312,102,187,500
LLM input cost · Claude Sonnet 5 input$14,326.35$24,204.38
-

1,000 files per format = the 24-document corpus cycled. Tokens are counted from the chunks each pipeline produces (ceil(chars/4), both sides); embedding time is measured throughput on x86_64 + NVIDIA L40S (2026-09-28) applied to the totals. Prices from bench/prices.json (as of 2026-09-12); edit it for your provider — the benchmark never calls a paid API. The PDF column is the PDF pipeline that scored best in the accuracy run. Scaling: figures are measured at 1,000 documents and 1,000,000 queries; anything at other volumes (10,000 documents, 10M queries) is a linear estimate, not a measurement — index and re-index costs scale with documents, query cost scales with questions asked. Re-index: JDF re-embeds only chunks whose content hash changed (jdf embed --incremental); a PDF has no chunk identity, so an edit means re-chunking and re-embedding the whole document.

+

10,000 files per format = the 24-document corpus cycled. Tokens are counted from the chunks each pipeline produces (ceil(chars/4), both sides); embedding time is measured throughput on Apple M5 (2026-10-14) applied to the totals. Prices from bench/prices.json (as of 2026-09-12); edit it for your provider — the benchmark never calls a paid API. The PDF column is the PDF pipeline that scored best in the accuracy run. Scaling: figures are measured at 10,000 documents and 10,000,000 queries; anything at other volumes is a linear estimate, not a measurement — index and re-index costs scale with documents, query cost scales with questions asked. Re-index: JDF re-embeds only chunks whose content hash changed (jdf embed --incremental); a PDF has no chunk identity, so an edit means re-chunking and re-embedding the whole document.

@@ -823,11 +823,13 @@

Author

if (metricSel) metricSel.addEventListener("change", () => { metric = metricSel.value; render(); }); const costEl = card.querySelector("[data-bench-cost]"); - const SAVINGS_SUB = "1,000 documents · same RAG pipeline · JDF vs the best PDF pipeline"; + const SAVINGS_SUB = `${(data.cost?.files ?? 1000).toLocaleString("en-US")} documents · same RAG pipeline · JDF vs the best PDF pipeline`; if (costEl && data.cost) { const c = data.cost; const usd = (x) => "$" + Math.round(x).toLocaleString("en-US"); const pct = (x) => Math.round(x * 100); + const q = c.queries ?? 1e6; + const qLabel = (n) => n >= 1e9 ? `${(n / 1e9).toLocaleString("en-US")}B` : n >= 1e6 ? `${(n / 1e6).toLocaleString("en-US")}M` : n.toLocaleString("en-US"); const saved = c.pdf.queryUsd - c.jdf.queryUsd; const cut = 1 - c.jdf.queryUsd / c.pdf.queryUsd; const reindexX = c.pdf.reindexUsd / c.jdf.reindexUsd; @@ -839,8 +841,8 @@

Author

costEl.innerHTML = `
${usd(saved)} saved
-
per 1M questions on ${c.llmLabel.replace(" input", "")} — ${usd(c.jdf.queryUsd)} instead of ${usd(c.pdf.queryUsd)}, same pipeline, only the input format differs
-
Measured at 1M questions (${c.files.toLocaleString("en-US")}-document corpus). The saving scales with questions asked, not with document count. Linear estimate, not measured: 10M questions ≈ ${usd(saved * 10)} · 100M ≈ ${usd(saved * 100)}. Your prices in bench/prices.json.
+
per ${qLabel(q)} questions on ${c.llmLabel.replace(" input", "")} — ${usd(c.jdf.queryUsd)} instead of ${usd(c.pdf.queryUsd)}, same pipeline, only the input format differs
+
Measured at ${qLabel(q)} questions over a ${c.files.toLocaleString("en-US")}-document corpus. The saving scales with questions asked, not with document count. Linear estimate, not measured: ${qLabel(q * 10)} questions ≈ ${usd(saved * 10)} · ${qLabel(q * 100)} ≈ ${usd(saved * 100)}. Your prices in bench/prices.json.
−${pct(cut)}%LLM tokens per answer${tokJ != null && tokP != null ? `${Math.round(tokJ).toLocaleString("en-US")} vs ${Math.round(tokP).toLocaleString("en-US")} tokens of context` : "top-5 context"}