Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion chart/values-prod.yaml.example
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ media:

# --- Ingestion ----------------------------------------------------------------
# A deployment loads every sentence the filters admit, not the development
# sample. For the Thai 25.0 release that is 21,429 exercises and as many clips,
# sample. For the Thai 27.0 release that is 21,442 exercises and as many clips,
# about 400 MB on the media volume, and a Job that runs for minutes. The Job
# still only runs when rendered with ingest.enabled=true; see docs/deployment.md.
ingest:
Expand Down
2 changes: 1 addition & 1 deletion chart/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,7 @@ ingest:
enabled: false
# How much of the corpus to load: a positive integer selects that many of the
# best candidates, `all` selects every sentence the filters admit (about 21k
# for the Thai 25.0 release). Small by default, because a development cluster
# for the Thai 27.0 release). Small by default, because a development cluster
# needs a catalog, not the catalog; a deployment asks for `all`.
count: 100
# The release directory as the pod sees it. In development kind mounts the
Expand Down
4 changes: 2 additions & 2 deletions docs/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,14 +106,14 @@ onto the media volume. The corpus has to be on a node, mounted from
`ingest.count` decides how much of it is loaded. A positive integer selects that
many of the best candidates, and `all` selects every sentence the quality and
beginner filters admit. The chart default is `100`, a development-sized sample;
`values-prod.yaml.example` sets `all`. For the Thai 25.0 release, `all` is 21,429
`values-prod.yaml.example` sets `all`. For the Thai 27.0 release, `all` is 21,442
exercises and about 400 MB of clips, and the Job runs for minutes, logging its
progress every thousand exercises. Any other value fails at render time.

```bash
kubectl delete job ingest -n typelearn --ignore-not-found
helm template typelearn ./chart --namespace typelearn --values my-values.yaml \
--set ingest.enabled=true --set ingest.corpusHostPath=/srv/cv-corpus-25.0-2026-03-09 \
--set ingest.enabled=true --set ingest.corpusHostPath=/srv/cv-corpus-27.0-2026-09-11 \
--show-only templates/ingest-job.yaml | kubectl apply -n typelearn -f -
kubectl logs -n typelearn -f job/ingest
```
Expand Down
4 changes: 2 additions & 2 deletions docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,11 +90,11 @@ bootstrap.
### 3. Load exercises

Ingestion needs a local Common Voice release directory
(`cv-corpus-25.0-2026-03-09` for Thai). It stays outside the repository and is
(`cv-corpus-27.0-2026-09-11` for Thai). It stays outside the repository and is
never committed — point the cluster at it before creating the cluster:

```bash
export TYPELEARN_CORPUS_DIR=/path/to/cv-corpus-25.0-2026-03-09
export TYPELEARN_CORPUS_DIR=/path/to/cv-corpus-27.0-2026-09-11
```

Then press **Ingest the corpus** in the Tilt UI. It selects 100 exercises from
Expand Down
4 changes: 2 additions & 2 deletions openspec/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@ context: |
configuration at all. Keep it that way.

Corpus:
- Mozilla Common Voice Thai 25.0, on disk in the release directory
cv-corpus-25.0-2026-03-09/th — outside the repository, read in place,
- Mozilla Common Voice Thai 27.0, on disk in the release directory
cv-corpus-27.0-2026-09-11/th — outside the repository, read in place,
never committed. Its path is a parameter, never hardcoded.
- Select exercises from validated.tsv (has path + up_votes). Never from
validated_sentences.tsv, which has neither.
Expand Down
2 changes: 1 addition & 1 deletion openspec/specs/corpus-ingestion/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ later rows for an already-selected sentence are skipped. A count selects a prefi
of the "all" selection.

#### Scenario: Selecting the MVP subset
- **WHEN** the ingestion script runs against the Thai corpus at the supplied corpus root (release directory `cv-corpus-25.0-2026-03-09`) with no selection size given
- **WHEN** the ingestion script runs against the Thai corpus at the supplied corpus root (release directory `cv-corpus-27.0-2026-09-11`) with no selection size given
- **THEN** it emits exactly 100 exercises, each with a unique `sentence`, drawn from the candidate pool matching the filters above

#### Scenario: Selecting an explicit count
Expand Down
8 changes: 4 additions & 4 deletions specs/tech-stack.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,9 +100,9 @@ Tailwind 3 and its `init -p` command was removed in v4.

| Layer | Choice | Notes |
|---|---|---|
| Corpus source | Mozilla Common Voice | CC-0 license, 150+ languages, Thai release 25.0 |
| Corpus location | Release directory `cv-corpus-25.0-2026-03-09/th`, path supplied to the ingestion script | 9.6 GB, **outside the repository**, read in place and never committed |
| Selection source | `validated.tsv` | 148,765 rows; the only manifest carrying both a clip `path` and vote counts |
| Corpus source | Mozilla Common Voice | CC-0 license, 150+ languages, Thai release 27.0 |
| Corpus location | Release directory `cv-corpus-27.0-2026-09-11/th`, path supplied to the ingestion script | 9.7 GB, **outside the repository**, read in place and never committed |
| Selection source | `validated.tsv` | 148,823 rows; the only manifest carrying both a clip `path` and vote counts |
| Clip durations | `clip_durations.tsv` | 366,508 rows mapping clip filename → duration in ms |
| Audio | `clips/*.mp3` | 366,508 files; only the ~100 selected clips are copied into `MEDIA_ROOT` |
| MVP dataset | JSON file | `data/processed/mvp_dataset.json`, produced by the ingestion script |
Expand Down Expand Up @@ -206,7 +206,7 @@ TypeLearn/
```

The Common Voice corpus is **not** part of this tree. It lives outside the
repository in the release directory `cv-corpus-25.0-2026-03-09/`, is
repository in the release directory `cv-corpus-27.0-2026-09-11/`, is
read in place by the ingestion script, and only the selected clips are copied into
`data/media/`. The corpus path is a parameter of the ingestion script, not a
hardcoded constant.
Expand Down
Loading
Loading