Ask whether text is plausibly Danish, instead of what language it is

[mircealungu]

Replaces langdetect with lingua in language_check — and with it, the framing the module was built on.

The wrong question

The check asked a detector to name a winner and then argued with the answer. That can't separate wrong language from close neighbour: short Danish often wins as Norwegian, so the module carried a hand-maintained list of confusable families ruling that Norwegian was near enough to Danish to allow. The list forgave real defects along with the misreads — a Norwegian learner served a Danish script passed it.

lingua scores a named language, so the question can be the one we actually have: is this plausibly Danish?

• correct Danish → 0.6–0.99 plausible Danish; the neighbours never come up
• English in a Danish lesson → 0.01
• given a paragraph, Danish is 4e-09 plausible Norwegian, so the wrong Scandinavian language is now caught rather than excused

Measured on all 6532 stored lessons before committing to it

	flagged	of which sound text
langdetect	12	7 — Dutch and Danish read as Afrikaans
lingua	8	3

All three known-bad dialogues (235, 236, 238) stay caught. lingua's three remaining flags are one coherent class — Sauron, week-end, Il rooftop — lessons whose target word is foreign, so the lines genuinely contain foreign text.

Cost: +19 MB resident, 0.2 ms per line.

Two levels, named for where they apply

• SENTENCE_LEVEL_FLOOR (0.2) — is this piece of text plausible?
• LESSON_LEVEL_THRESHOLD (0.45) — how much of a lesson may fail before the lesson does?

The second exists because a four-word line is unreliable even here: correct Danish Er det i kantinen? scores 0.12. Aggregation is what turns an unreliable per-line signal into a trustworthy verdict — so individual failures don't need to be right, only the share.

Each level sets its own minimum for what it will judge at all: 3 words for a line inside a lesson, 60 characters for a field checked alone, which has nothing to average against.

What this deletes

_CONFUSABLE_FAMILIES, _accepted_codes, _chunk, passage_mismatch, PASSAGE_CHUNK_CHARS, MAX_WRONG_SHARE, _dialogue_lines, MIN_CHARS_FOR_DETECTION, LANGDETECT_SUPPORTED. Every one existed to make a winner-take-all signal usable in aggregate. Net −24 lines and two fewer tuned numbers, with better results.

Adaptive simplification also now checks each part of a level separately: glued, a level's Danish summary masked its English body — 0.97 plausible as Danish for a field whose content alone scored 0.00.

Scope

langdetect stays in the other eleven modules that use it. They ask an open question — what language is this article? — where naming a winner is exactly the job. Only language_check asks a closed one.

Two integration details worth knowing: lingua splits Norwegian into Bokmål and Nynorsk, so Zeeguu's no maps to NB (without it every Norwegian lesson would be unjudgeable), and Kurdish has no model there either, so it stays "can't judge" exactly as before.

Testing

zeeguu/core/test — 259 passed. The tests changed shape in one telling place: test_scandinavian_neighbours_are_not_flagged_against_each_other asserted that a Danish script passes as a Norwegian lesson. That encoded a langdetect limitation as a requirement, and it's now inverted — a Danish script in a Norwegian lesson is a defect, and we can finally tell.

🤖 Generated with Claude Code

[github-actions]

ArchLens detected architectural changes in the following views: