Add translation

README.md

@@ -1,7 +1,6 @@
 # TypeLearn
 
 [![CI](https://github.com/Akay7/TypeLearn/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/Akay7/TypeLearn/actions/workflows/ci.yml?query=branch%3Amain)
-[![Translation status](https://hosted.weblate.org/widget/typelearn/svg-badge.svg)](https://hosted.weblate.org/engage/typelearn/)
 [![License](https://img.shields.io/github/license/Akay7/TypeLearn)](LICENSE)
 [![Top language](https://img.shields.io/github/languages/top/Akay7/TypeLearn)](https://github.com/Akay7/TypeLearn)
 
@@ -37,11 +36,8 @@ Development mode, tests, debugging and running several worktrees are covered in
 
 ## Translating
 
-Help translate TypeLearn on [Hosted Weblate](https://hosted.weblate.org/engage/typelearn/).
-You don't need to open a pull request: Weblate commits your translations and opens
-it for you. Any text not translated yet shows in English.
-
-To change the English wording or add a language, see
+To fix a translation or add a language, open a pull request that edits the JSON files
+in `src/frontend/src/locales/`. Any text not translated yet shows in English. See
 [docs/translation.md](docs/translation.md).
 
 ## Audio

chart/templates/ingest-job.yaml

@@ -1,7 +1,22 @@
 {{- if .Values.ingest.enabled }}
 {{- $corpus := .Values.env.CORPUS_PATH | default "/corpus" }}
+{{- /* How much to select: a positive integer, or `all`. Checked on its string
+     form so 100 and "100" are the same value, and refused here rather than in
+     a Job that starts, runs for a while, and fails. */}}
+{{- $count := toString .Values.ingest.count }}
+{{- $size := "" }}
+{{- if eq $count "all" }}
+{{- $size = "--all" }}
+{{- else if regexMatch "^[1-9][0-9]*$" $count }}
+{{- $size = printf "--count %s" $count }}
+{{- else }}
+{{- fail (printf "ingest.count must be a positive integer or \"all\", got %q" $count) }}
+{{- end }}
 # Ingestion: read a Common Voice release in place and copy the selected clips
-# into the media volume.
+# into the media volume, then load the machine translations committed with the
+# backend (exercises/fixtures/translations/) for them — so a catalog rebuilt
+# from scratch comes back translated without anything being translated again.
+# With no fixture yet, the second step says so and does nothing.
 #
 # A plain Job, deliberately not a Helm hook. A hook would run on every upgrade,
 # and this takes minutes and writes into storage every worktree reads — an
@@ -40,7 +55,7 @@ spec:
           command:
             - sh
             - -c
-            - 'python manage.py load_corpus "${CORPUS_PATH:-/corpus}"'
+            - 'python manage.py load_corpus "${CORPUS_PATH:-/corpus}" {{ $size }} && python manage.py load_translations'
           envFrom:
             {{- include "typelearn.envFrom" . | nindent 12 }}
           env:

chart/values-prod.yaml.example

@@ -96,6 +96,14 @@ media:
   accessModes:
     - ReadWriteMany
 
+# --- Ingestion ----------------------------------------------------------------
+# A deployment loads every sentence the filters admit, not the development
+# sample. For the Thai 25.0 release that is 21,429 exercises and as many clips,
+# about 400 MB on the media volume, and a Job that runs for minutes. The Job
+# still only runs when rendered with ingest.enabled=true; see docs/deployment.md.
+ingest:
+  count: all
+
 # --- Workloads ----------------------------------------------------------------
 backend:
   replicas: 2

chart/values.yaml

@@ -116,6 +116,12 @@ env:
   # Where ingested clips live in the container; the media volume mounts here.
   MEDIA_ROOT: /app/media
   CORPUS_PATH: /corpus
+  # Every request reaches Django through the Gateway, whose Envoy appends the
+  # client's address to X-Forwarded-For — so the last entry is the one to
+  # trust, and the rate limits on ratings and suggestions count by it. Raise it
+  # by one for each further proxy that appends (a cloud load balancer in front
+  # of the Gateway); 0 would put every learner behind one shared limit.
+  TRUSTED_PROXY_COUNT: "1"
 
 # Secret values, rendered into a Secret and never into the ConfigMap. Anything
 # put here is readable by anyone who can read the release, so prefer
@@ -209,6 +215,11 @@ worktree:
 # starts one.
 ingest:
   enabled: false
+  # How much of the corpus to load: a positive integer selects that many of the
+  # best candidates, `all` selects every sentence the filters admit (about 21k
+  # for the Thai 25.0 release). Small by default, because a development cluster
+  # needs a catalog, not the catalog; a deployment asks for `all`.
+  count: 100
   # The release directory as the pod sees it. In development kind mounts the
   # host's corpus here; set corpusHostPath to mount it from the node.
   corpusHostPath: ""

docs/deployment.md

@@ -96,6 +96,72 @@ one `kubectl delete`, and recovering from a deleted database is not.
 `externalDatabase` instead, so a managed Postgres is a values change rather than
 a fork of the chart.
 
+### Loading the exercises
+
+Ingestion is a Job the chart renders only when asked, never on install or
+upgrade: it reads a Common Voice release in place and copies the selected clips
+onto the media volume. The corpus has to be on a node, mounted from
+`ingest.corpusHostPath`.
+
+`ingest.count` decides how much of it is loaded. A positive integer selects that
+many of the best candidates, and `all` selects every sentence the quality and
+beginner filters admit. The chart default is `100`, a development-sized sample;
+`values-prod.yaml.example` sets `all`. For the Thai 25.0 release, `all` is 21,429
+exercises and about 400 MB of clips, and the Job runs for minutes, logging its
+progress every thousand exercises. Any other value fails at render time.
+
+```bash
+kubectl delete job ingest -n typelearn --ignore-not-found
+helm template typelearn ./chart --namespace typelearn --values my-values.yaml \
+  --set ingest.enabled=true --set ingest.corpusHostPath=/srv/cv-corpus-25.0-2026-03-09 \
+  --show-only templates/ingest-job.yaml | kubectl apply -n typelearn -f -
+kubectl logs -n typelearn -f job/ingest
+```
+
+A Job's spec cannot change once created, which is why the old one is deleted
+first. Re-running is safe: exercises already loaded are updated in place, clips
+already on the volume are not copied again, and a larger count only adds what
+the smaller one left out. Nothing is ever deleted.
+
+The frontend never downloads the whole catalog, however large it is: each page
+load asks for a random deck of 200 (`deck(size: 200)`), and a new deck is drawn
+when the learner finishes one. The server caps a deck at
+`EXERCISE_DECK_MAX_SIZE` (default 500), settable under `env:` like any other
+backend variable.
+
+### Translations
+
+A learner can turn on a translation of each phrase into the interface language,
+rate it, and suggest a better one. Nothing about this needs setting up for the
+application to run: until translations exist, the setting shows "No translation
+yet" and offers the suggestion form.
+
+**Where machine translations come from.** They are committed with the backend,
+in `src/backend/exercises/fixtures/translations/th.yaml`, so they ship in the
+image. The ingest Job loads them after the corpus, which means a new
+deployment, a rebuilt database, or a re-ingest gets them back without
+translating anything. A deployment needs no provider key and runs no
+translation. The file is made once on a developer's machine (see
+[development.md](development.md#4-translations-optional)), and a correction is an
+edit to it plus a release. To load a new version without re-ingesting, run
+`python manage.py load_translations` in a backend pod.
+
+**Suggestions and ratings.** Learners are anonymous, so both are identified by a
+random id the browser keeps, plus an HMAC of the client address keyed by
+`DJANGO_SECRET_KEY` — the address itself is never stored. Both are rate-limited
+per hour (`TRANSLATION_PROPOSALS_PER_HOUR`, default 20; `TRANSLATION_RATINGS_PER_HOUR`,
+default 300). The address comes from `X-Forwarded-For`, trusting the last
+`TRUSTED_PROXY_COUNT` entries; the chart sets 1, for the Gateway. With another
+proxy in front of the Gateway that also appends to the header, set it to 2 — too
+low, and every learner shares one limit; too high, and a client can choose its own
+address.
+
+A suggestion is never shown until someone approves it. In the admin, under
+**Translations**, the list opens on the pending ones, each beside its sentence and
+the translation learners see now. **Approve** publishes the selection and
+**Reject** hides it for good. Of the published translations of a phrase, learners
+see the best rated, and a newly approved one wins a tie.
+
 ### Rendering before installing
 
 The chart renders without a cluster, which is what makes it reviewable:

docs/development.md

@@ -102,11 +102,77 @@ Then press **Ingest the corpus** in the Tilt UI. It selects 100 exercises from
 clips into the shared media volume. The selection is deterministic — the same
 corpus always yields the same 100 exercises — and it is safe to re-run.
 
+100 is the chart's default for `ingest.count`, and a deployment sets `all` (see
+[deployment.md](deployment.md#loading-the-exercises)). To try the full catalog
+locally — about 21k exercises, 400 MB of clips, a few minutes — render the same
+Job the button does with the size overridden (only the Job mounts the corpus, so
+the backend pod cannot run it). The button reads the backend image from Tilt;
+here it is whatever the running backend uses:
+
+```bash
+NS=<namespace>
+IMAGE=$(kubectl get deploy backend -n $NS -o jsonpath='{.spec.template.spec.containers[0].image}')
+kubectl delete job ingest -n $NS --ignore-not-found
+helm template typelearn chart --namespace $NS --values .tilt/values.yaml \
+  --set ingest.enabled=true --set ingest.corpusHostPath=/corpus --set ingest.count=all \
+  --set-string image.backend.repository="${IMAGE%:*}" --set-string image.backend.tag="${IMAGE##*:}" \
+  --show-only templates/ingest-job.yaml | kubectl apply -n $NS -f -
+kubectl logs -n $NS -f job/ingest
+```
+
+The clips land in the shared `data/media`, so every worktree sees them; the rows
+land only in that worktree's database. Running with a count afterwards does not
+shrink it again — ingestion adds and updates, it never deletes.
+
 The clips live in `data/media` on the host, shared by every worktree and outliving
 the cluster, so this only has to happen once even if you delete and recreate the
 cluster. They are written by the container and owned by a mapped user id; the
 Tiltfile keeps the directory writable so you can still remove them yourself.
 
+### 4. Translations (optional)
+
+The **Show translation** setting needs translations to show. They are committed:
+`src/backend/exercises/fixtures/translations/th.yaml` holds the machine
+translation of every sentence ingestion can select, and **Ingest the corpus**
+loads it with `load_translations` after the corpus. Nothing needs a key or a
+network.
+
+The file is made once, on a developer's machine, with the Anthropic API. The key
+lives only in that shell. It is not in `.env`, the Secret, or the chart. The
+command reads the Common Voice release directly, the same selection ingestion
+makes, so it needs neither the cluster nor a database. Start with a trial run:
+
+```bash
+cd src/backend
+export ANTHROPIC_API_KEY=sk-ant-...
+uv run python manage.py translate_catalog "$TYPELEARN_CORPUS_DIR" --limit 20
+```
+
+Read what came back, then run it without `--limit`. For the whole catalog (about
+21k sentences into 5 languages) that is about 4,300 requests, a one-time spend.
+A re-run only requests the languages an entry is missing, and an interrupted run
+keeps everything that was saved: the file is written every 30 seconds and on
+exit. `--model` and `--effort` trade quality for cost, and `--languages en,ru`
+narrows the targets. Review the diff and commit the file.
+
+To use a local OpenAI-compatible server instead, such as Lemonade, start it and
+pass the model name it lists. No key is needed:
+
+```bash
+uv run python manage.py translate_catalog "$TYPELEARN_CORPUS_DIR" \
+  --provider openai-compatible --model <model name> --batch-size 10 --limit 20
+```
+
+`--base-url` defaults to Lemonade's `http://localhost:8000/api/v1`. Local models
+are less reliable than Claude, so keep the batches small and read the trial run.
+
+To fix a bad translation, edit its line and commit. Loading replaces the text
+and clears that translation's ratings, because they were cast on different
+words. To try the command out without a key, use `--provider offline --dir
+<scratch dir>`, which writes placeholders (`[fr] สวัสดี`). It refuses to write
+them over the committed file. Suggestions made in the app wait under
+**Translations** in the admin (`/admin/`) until they are approved there.
+
 ### Running several worktrees at once
 
 Each git worktree gets its own namespace, its own database, its own application

docs/translation.md

@@ -9,27 +9,24 @@ and any string a language has not translated yet shows in English.
 
 Keys are nested objects grouped by component (`settings` → `button` → `label`)
 and looked up by their dotted path, `t('settings.button.label')`. Don't write a
-flat `"settings.button.label"` key: Weblate writes the keys it adds nested, so a
-file would end up mixing both shapes. `keys.test.js` fails on dotted keys.
+flat `"settings.button.label"` key: a file would end up mixing both shapes. `keys.test.js` fails on dotted keys.
 
-Translations are done on [Hosted Weblate](https://hosted.weblate.org/engage/typelearn/). You don't
-need to open a pull request to translate. Weblate commits the changes and opens
-the pull request for you. To change the English wording or add a string, edit
-`en.json` in a normal pull request. Weblate picks the change up once it merges.
+To change a translation, edit that language's file in a normal pull request. To add a
+string, add it to `en.json` and to every other language file in the same pull request.
 
 A string that depends on a number gets one key per plural form, named the i18next
 v4 way: `symbolCount_one` and `symbolCount_other` inside `sentence` in
 `en.json`. Render it with `tPlural(key, count)` from `src/frontend/src/i18n.js`,
-not `t`. The helper picks the form the active language needs, so on Weblate each
-language gets its own plural forms, such as `_few` and `_many` for Russian.
+not `t`. The helper picks the form the active language needs, so each
+language has its own plural forms, such as `_few` and `_many` for Russian.
 
 ## Adding a language
 
 A new JSON file alone does nothing. The app only loads the languages it lists. To
 add one, open a pull request that, under `src/frontend/src/`:
 
 1. adds `locales/<code>.json` as an empty `{}`, unless a translator already
-   started the language on Weblate and its file exists,
+   started the language and its file exists,
 2. in `i18n.js`, imports that file, adds it to the catalogs passed to
    `createI18n`, and adds the code to `SUPPORTED_LANGUAGES`,
 3. adds the language's own name for itself (e.g. "Deutsch") to `LANGUAGES` in
@@ -38,5 +35,3 @@ add one, open a pull request that, under `src/frontend/src/`:
    `components/__tests__/SettingsMenu.test.js`,
 4. imports the file in `locales/__tests__/keys.test.js` and adds it to
    `CATALOGS`, so its keys and placeholders are checked too.
-
-Once it merges, Weblate lists the language for translators.

openspec/changes/add-phrase-translations/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-09-26