feat: generic per-language text corpora (data/corpora) - #63
Merged
Conversation
New language asset alongside freq/audio/alphabets: data/corpora/<lang>.txt — natural text, one sentence per line, ~400KB per language — with SOURCES.json provenance per corpus (generator, mode, pipeline, license note, verify flag). Synthetic mode today: Zipf-weighted sampling from the ranked word lists in data/freq/top1000 (193 languages on disk), which preserves the character- and word-shape statistics of the source corpus. Sentence sources take precedence once wired: CommonVoice transcripts (CC-0) and Tatoeba (CC-BY 2.0 FR) are the priority inputs. Licensing: every corpus is verify:true until its per-language source chain is confirmed clean — the freq pipeline mixes Leipzig (research-use terms) with clean sources. Consumers must respect the flag; DasherCore's importer (dasher-project/DasherCore#73) refuses verify:true corpora by default. ruff + mypy clean. Signed-off-by: will wade <willwade@gmail.com>
Full run of scripts/build_text_corpora.py: one corpus per language with freq data (400KB synthetic text each, 74MB total — comparable to the existing data/audio asset). All carry verify:true in SOURCES.json until their per-language source chain is reviewed; consumers (e.g. dasher-project/DasherCore#73) refuse verify:true corpora by default. Signed-off-by: will wade <willwade@gmail.com>
…nifest) Same access pattern as the existing freq/alphabets data, across the package's entry points: - index.js (CJS/Node): loadCorpus(code) reads data/corpora/<code>.txt from disk; listCorpora() reads SOURCES.json (always shipped in the npm tarball) — lang, mode, verify flag, license note - index.mjs (ESM/browser): listCorpora() from a generated dist/browser-corpora.mjs MANIFEST bundle (the 74MB of text is deliberately not embedded — same reasoning as audio); loadCorpus() works under Node ESM via dynamic fs import and throws descriptive guidance in the browser - index.d.ts: CorpusInfo / Corpus types - package.json files: corpus .txt excluded from the npm tarball (!data/corpora/*.txt) with SOURCES.json shipped — listCorpora() works from npm; the text lives in the repo / vendored checkouts - jest: 3 tests (manifest listing, repo-checkout load, descriptive error) — full suite 54 passed The Python package (src/worldalphabets) gets its accessors alongside the lint baseline merge (follow-up commit). Signed-off-by: will wade <willwade@gmail.com>
Collaborator
Author
|
Now includes the consumer API (npm/Node CJS, Node ESM, browser manifest, TS types) — CI note: Test Python on this branch will keep failing on the pre-existing ruff violations until #64 (the pinned-ruff baseline) merges — merge #64 first, then this rebases/merges green. |
The first cut built corpora by sampling words from the frequency lists
with Zipf weights — plausible character statistics, but the sentences
were nonsense ('A a múlt oh a gyernek a ez a benned a!'). For a
text-entry corpus that's the wrong trade: word-order and inter-word
context matter, and the label 'natural text' was simply false.
build_text_corpora.py now downloads the Tatoeba per-language exports
(cached under .cache/tatoeba/):
- CC0 subset used where a language has substantive coverage (>=500
sentences), otherwise the full export — CC-BY 2.0 FR with
attribution (https://tatoeba.org) recorded in every SOURCES.json
entry and in the README
- verify:false for Tatoeba-derived corpora: licensing-clean WITH
attribution, so downstream importers (DasherCore #73) can consume
them without the review gate
- languages without Tatoeba coverage are skipped and reported —
a missing corpus is honest; a synthetic one is not. The synthetic
mode is removed, deliberately
- 167 languages covered (was 193 synthetic); 27MB (was 74MB)
- dist/browser-corpora.mjs manifest regenerated
README: new 'Text corpora' section documenting access from every
interface (Python get_corpus/list_corpora incl. WA_CORPORA_DIR and
path= resolution; JS loadCorpus/listCorpora incl. browser manifest
behaviour) and the licensing/attribution requirements.
Re-verified in DasherCore: Hungarian with the real-sentence corpus
gives 29 letter boxes / 5.3x spread (synthetic: 24 / 2.3x; maintained
English reference: 25 / 2.4x) — real sentences train a genuinely
better model, as expected.
Signed-off-by: will wade <willwade@gmail.com>
Completes the consumer wiring for data/corpora (JS/TS landed in the previous commit; earlier Python work was lost to a stash mishap and is re-applied here): Python (pip install worldalphabets): - worldalphabets.list_corpora() — manifest from the packaged SOURCES.json (always shipped; 167 Tatoeba entries) - worldalphabets.get_corpus(lang, path=None) — resolves the text via path=, WA_CORPORA_DIR, then package data; descriptive error listing available languages when the files aren't present (27MB is deliberately not packaged) - sync_package_data.py syncs the manifest ONLY (stale .txt swept); MANIFEST.in belt-and-braces excludes corpus text from the wheel C (c library): - wa_corpus_info struct + wa_corpus_count / wa_list_corpora / wa_get_corpus_info — a static MANIFEST table generated from SOURCES.json (wa_data_corpora.c). The text is never embedded; the header documents why and carries the CC-BY attribution requirement - generator extended; stale generated artifacts (morphology-branch leftovers referencing undeclared types) regenerated clean Verified end to end: - C: full cmake build + smoke suite pass; corpus check binary prints 167 entries, hu -> tatoeba-sentences verify=0, missing -> NULL - Python: list_corpora 167; get_corpus via path 368KB of real Hungarian - JS: jest 54/54 (corpus tests from previous commit) - ruff 0.16.5 clean; mypy clean; pytest 4/4 Signed-off-by: will wade <willwade@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New language asset alongside freq/audio/alphabets — nothing consumer-specific (this replaces the closed #62, which wrongly put a Dasher-format exporter here; the Dasher-side consumer now lives in dasher-project/DasherCore#73).
What
scripts/build_text_corpora.pybuildsdata/corpora/<lang>.txt— natural text, one sentence per line, ~400KB per language — for every language with freq data (193 on disk).freq/top1000lists — preserves character- and word-shape statisticsSOURCES.jsonper corpus — generator, pipeline, license note, andverify: trueuntil the per-language source chain is confirmed clean (freq pipeline mixes Leipzig research-use terms with clean sources). Consumers respect the flag.Checks
ruff check(0.16.5) andmypyclean on the new script — run locally before pushing this time.DCO signed.