ci: catch a short translation before it costs two hours on main - #841
Merged
Conversation
Run 33160218909 spent 2h04m translating, failed, and left main red for the
fifteenth day running. Every failure in it came from the same place, and none
of it was visible until the translation had already run.
What the run showed, from its own logs and the artifact it uploaded:
- All three documents doom split into chunks failed; ten of the eleven it
translated in one request passed. The 130KB page came back at 58% of its
size with the first of its three chunks mostly missing -- "Chapter 2",
"Chapter 3" and "Generate signing key" appear in the English source and
nowhere in the translation.
- Both damaged documents ended with a lone ``` -- the closing half of the
wrapper fence the prompt tells the model not to emit. For one of them that
stray line was the only thing wrong with the document.
- Retrying made things worse. One document was two code blocks and one
heading short on attempt 1 and at 47% of its length on attempt 3, and
attempt 3 is what stayed on disk, because each attempt overwrites the file.
So, four changes, in the order they now run:
- Warn on any English source over doom's 60KB chunking limit. This is the
property that predicted failure, it costs a second, and it is reported on
pull requests -- where the translate step never runs, which is why none of
this could be seen before merging. A warning rather than a failure only
because six documents are already over the line.
- Report an unclosed code fence by name, and remove it when it opens at the
very end with nothing after it. A block that opens and contains nothing is
not something an author wrote, and deleting the line cannot destroy text.
An unclosed fence with content after it is reported and left alone.
- Fail a translation that comes back at a fraction of its size. Across the
404 pairs the run produced, the lowest healthy ratio is 0.869 and the
damaged one sits at 0.576; the floor is 0.7. It is a floor, not a
substitute for the counts -- the other damaged document kept its volume.
- Score every document and keep the best attempt instead of the last, so
what the loop leaves for a human is the closest the translator ever got.
All of it is tested without an API key: 51 assertions against the checker and
10 against the retry loop, the latter driving the loop through TRANSLATE_CMD
with a scripted translator. Both suites run under yarn test:translation, which
has no event condition and so runs on pull requests. Each new check was
verified by reverting it and watching its assertions go red: 47 -> 43 without
the fence repair, 47 -> 45 without the volume floor, 10 -> 8 without best-of-N.
Testing the opt-out path turned up a false positive in the size floor added a commit ago. Frontmatter is not translated -- only title and description are, and doom deletes the i18n block from the target -- so an English page that carries `i18n.disableAutoTranslation` or a multi-line `i18n.additionalPrompts` is longer as a file than its faithful translation while being the same length as a document. A hand-written translation of a short opted-out page failed at 67% of its original's size with not a word missing. So the ratio is taken over bodies, and pages under 1KB are left to the structural counts: below that a sentence either way swings the ratio, and a page that small is never chunked, which is where this failure mode comes from. Re-measured over the 403 body pairs from run 33160218909, the separation is unchanged -- lowest healthy 0.849, the damaged document 0.575, floor 0.7. Both are covered, and both were verified by reverting them: comparing whole files again fails the additionalPrompts case, and dropping the size guard fails the short-page case. 56 assertions against the checker, 10 against the loop.
Both are far over doom's 60KB chunking limit -- 967KB and 127KB -- and both failed every attempt in run 33160218909. The 127KB page came back at 58% of its size with its first chunk mostly missing; the 967KB one lost 88 table rows and 22 code blocks. Neither is a translation problem a retry can solve: the model is asked for a faithful rendering of half a megabyte and does not give one. i18n.disableAutoTranslation takes them out of doom's hands entirely. doom skips any source carrying it and never writes the target, so a hand-written Chinese page cannot be overwritten -- unlike sourceSHA, which only holds until the English page is edited, and which is how the earlier damage survived a revert. The size warning now stays quiet for opted-out pages: how doom would have chunked a page it never translates is not something anyone can act on. The existing Chinese page for the supply chain article is left in place and still fails the structural check, since it is the damaged machine translation from before. It stays visible until it is replaced by hand.
The two documents it would have helped are the two now translated by hand, and on the evidence there is no third: the other document that failed every attempt in run 33160218909 failed identically each time, so keeping the best of three would have picked the same file. That leaves the checker's --scores output with no caller and translate-verified .mjs untouched by this branch. The remaining risk is real but speculative -- four sources are still over the chunking limit and still machine-translated, so a future document could still degrade across retries. If one does, the change is small and the evidence for it will be in that run. The loop's tests stay. They cover behaviour that predates this branch and was never covered: that it retries only the documents that failed, that it stops as soon as an attempt verifies, and that it fails when none do.
The machine translation of this page has been damaged since the run that produced it: 43 of its 123 code blocks, 77 of its 129 headings and all 35 of its table rows were missing, and it sat at 58% of the original's size. Three retries in run 33160218909 each made it worse rather than better, which is why the page now opts out of machine translation. This replaces it with a translation written for the page rather than sampled from it. Code blocks, their comments and their example output are reproduced byte for byte, link destinations are untouched, and Kubernetes, Tekton and Kyverno resource kinds, field names and CLI flags stay in English. Verified by the structural check rather than by reading: 129 headings against 129, 123 code blocks against 123, 35 table rows against 35, and 94.8% of the original's size. `yarn build` renders it with no dead links. Both languages also gain explicit heading ids for the seventeen anchors the page links to internally. Without them a translated heading gets a translated slug and every `](#english-slug)` in the Chinese page silently stops resolving. Each id is the one the English heading already generates, so the English page is unchanged in behaviour and external links to those anchors keep working. The ids are assigned by first occurrence, as markdown assigns them: nine heading texts appear more than once here, and taking the wrong one of a pair would have retargeted a link without any sign of it. One pre-existing defect is left alone because it is not mine to decide: the link labelled "Chapter 1: Wait for the pipeline to be signed" points at #step-4-wait-for-the-pipeline-to-be-signed, and Chapter 1's heading reads "Wait for the PipelineRun to be signed", so the anchor has always landed in Chapter 3. The explicit ids preserve exactly that behaviour.
The link says Chapter 1 and has always landed in Chapter 3: Chapter 1's heading reads "Wait for the PipelineRun to be signed", so the slug the link used never matched it. Both places that reference it -- Chapter 2's re-run instructions and Chapter 7's -- meant Chapter 1, which is where the pipeline that produces the image is defined. Chapter 1's heading gets the id the links now use; Chapter 3's loses its explicit one, which existed only to be a link target.
This is the second of the two documents that opted out of machine translation. At 990KB the English source is more than sixteen times the 60KB chunk limit, so doom would have had to split it into seventeen chunks -- the failure mode that produced the truncated Chinese page in the first place. The document was split into 161 prose segments and 160 code blocks. Only the prose was translated; every code block is carried over byte for byte, so the YAML, the kubectl invocations, and the JMESPath expressions cannot drift from the English page. Reassembly is checked against the source: identical heading sequence (108), fence count (320), code block count (160), table rows (591), and an identical set of 89 explicit heading anchors, with all 79 internal link targets resolving. Verified with scripts/check-translation-integrity.mjs (PASS, 1 link) and yarn build (exit 0, no dead links).
Install_Multi-Primary_Service_Mesh_on_Different_Networks failed every one of
the three retranslation attempts in run 33160218909 with "table rows 43 vs 36
-- 7 invented". Nothing was invented. Its English source writes two of its
tables without leading pipes:
Priority | Locality | Details
-------- | -------- | -------
0 | `region1` | Current Cluster, client and server Region match.
which is valid GFM and renders as a table. The row counter matched on a
leading pipe, so it scored those ten lines as zero on the English side, while
the translation -- which the model normalised to the piped form -- scored them
in full. The document had lost nothing and could not be made to pass, so the
retry loop burned two hours and left main red.
Rows are now anchored on the delimiter row instead: a line of dashes and pipes
directly under a line containing a pipe starts a table, and every contiguous
line after it that still carries a pipe is one of its rows. Both spellings
count the same, and a pipe that merely appears in a sentence still counts as
nothing.
Three tests cover it: the two forms compared against each other pass, rows
genuinely dropped from a pipeless table are still counted exactly, and pipes
in prose are not rows. Reverting the counter turns the first and third red.
The English page is 82KB, well over doom's 60KB chunk limit, so every main run splits it and reassembles a Chinese page that fails verification -- three times in run 33160218909 alone, and it will fail again on the next run for the same reason. The Chinese page committed here, translated before the source grew past the limit, is complete: same heading outline, same 54 code blocks, same tables, 98% of the source's size. So the source now opts out of machine translation and keeps the good page, rather than having it overwritten by a chunked one that cannot pass. The page no longer follows English edits automatically; splitting the source into several pages under the limit would restore that, and is the better fix if anyone is willing to restructure the guide. Also drops a stray closing fence that sat alone on the last line of the Chinese page. The check repairs that with --fix, but repairing it on every run is not the same as it not being there.
That page is not ours to change. Opting an English source out of machine translation freezes its Chinese page for everyone, and that is a decision for the people who own the document, not for the pipeline that translates it. Only the two Tekton/Kyverno guides carry the opt-out, because they are hand translated on purpose. The chunking problem the opt-out was working around is real and unchanged: the source is 82KB, doom cuts it at 60KB, and every attempt reassembles a page that fails verification. The next commit handles it in the pipeline instead -- a document that will not converge keeps the translation already committed, and the run carries on -- which leaves every English page untouched. The stray closing fence removed from the Chinese page stays removed: that is a machine-translation artefact on a generated file, not an edit to anyone's document.
…nverge
Two holes, both found by reviewing what run 33160218909 actually did.
**A pull request never checked a document.** "Translate and verify" is gated on
`push` to main, so a branch only ran the check's unit tests -- which prove the
check works without looking at a single page. A hand-written translation could
be added, or an English page edited out from under an existing translation, and
nothing compared the two until main. The check's default scope reads the
working tree, which is right after translate rewrites it and selects nothing on
a clean branch checkout, so `--since <ref>` compares against a ref instead and
judges the pairs the branch touches, in either language. A new English page
with no translation yet is not a branch's problem -- translate writes it on
main -- unless the source opts out of machine translation, in which case
nothing downstream will ever write it and the hand translation has to arrive
with the source.
**One document that cannot converge threw away everyone else's work.** A source
over doom's 60KB limit is cut into chunks, a chunk comes back short, and the
next attempt cuts it the same way; the run then failed before the commit, so
every other document's good translation was discarded too. That is how one 82KB
page kept four others stale in the repository for weeks. Now a document that is
still incomplete after every attempt falls back to the translation already in
git, and the run carries on -- but only if that committed version is itself
complete, verified through the new `--only` scope. With nothing good to fall
back on the run still fails, because keeping a damaged page would publish the
very thing this script exists to stop. The fallback is loud: it names the
document and says the page no longer follows its English source.
Two smaller fixes alongside: TRANSLATE_ATTEMPTS is validated as a whole number,
since `attempt <= Infinity` never ends and a typo would retranslate until the
runner times out; and the opt-out is recognised in its inline YAML form
(`i18n: { disableAutoTranslation: true }`), which doom honours and the check
was reporting as a missing translation.
Covered by nine new assertions across the two test files -- the branch scope,
the new-page and opted-out-page cases, `--only`, both fallback outcomes, the
attempt-count guard, and the inline opt-out.
All four came out of re-reviewing the previous commit rather than from a new symptom, and one of them would have crashed the first pull request that hit it. `branchScope` was read by the missing-translation audit and declared below it, so the moment `--since` met an opted-out source with no translation the run died on the temporal dead zone instead of reporting the gap. It survived the earlier testing only because no such page exists in this repository right now. The declaration moves up to where the rest of the scope lives, and the case it crashes on is now one of the tests. `--since` also judged that gap for every opted-out page in the tree, not just the ones the branch touched -- so an old gap in someone else's page would fail an unrelated pull request, which is how a check becomes something to route around. The scope now carries its source side too, and only pages the branch actually changed are held against it. `--only` answered "nothing wrong" to a path that was not there: a typo in the caller's list scanned nothing and exited 0, which is the one answer a question about specific files must never get. It now exits 2, and refuses a path outside the target tree for the same reason. And the fallback verification reused the failures file the earlier attempts had written. It is cleared first, so a run that dies before writing cannot be read as blaming documents it never judged. Verified end to end against a clone of this repository with a stand-in translator that reproduces the chunked failure on the 82KB source: three attempts fail, the committed translation is restored and verified through --only, the run exits 0, and `git status` shows the page back to its committed content. With the fallback removed from the same simulation the run exits 1 and leaves the truncated page on disk.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Run 33160218909 translated for 2h04m, failed, and left
mainred for the fifteenth day running. Everything below comes out of that run's own logs and thegenerated-docs-33160218909artifact it uploaded — and none of it was observable until the translation had already run, because the translate step is gated ongithub.event_name == 'push' && github.ref == 'refs/heads/main'and never executes on a pull request.That gap is the point of this PR. Every check it adds runs under
yarn test:translation, which has no event condition, so it runs here.What the run showed
Chunking is the property that predicts failure. doom splits any source over 60KB (
maxChunkSizein@alauda/doomlib/cli/translate.js) and translates each piece in a separate request. Of the fourteen documents this run translated, three were chunked and all three failed; ten of the remaining eleven passed.The 130KB page is the clearest case. It came back at 58% of its size, and its first chunk is mostly absent:
docs/endocs/zhUnderstanding major risksSupply chains levels for software artifactsGenerate signing keyChapter 2Chapter 3chains.demo-1The translation begins partway through the English source, in English, inside a code fence. Chunks 2 and 3 are fine. doom joins the three with
translatedChunks.join('\n')and writes the result, because nothing between the model and the file asks whether a chunk came back whole —grep -rn "finish_reason\|max_tokens" node_modules/@alauda/doom/lib/returns nothing.Both damaged documents ended with a lone
```— the closing half of the wrapper fencedoom.config.tsexplicitly tells the model not to emit. ForHow_to_Migrate_MySQL_57_to_80.mdthat stray line was the only thing wrong:fenced code blocks 55 vs 54 -- 1 invented, three attempts running.Retrying made things worse. Each attempt is an independent sample at a fixed temperature, and each one overwrites the file:
Software_Supply_Chain_Security_…Attempt 1 was within two code blocks of passing. Attempt 3 is what stayed on disk.
What this changes
In the order the checks now run:
mainrun. A warning rather than a failure only because six documents are already over the line —warningscan start counting towardsfailonce they are split.How_to_Install_and_use_Evidently.md) and the damaged document sits at 0.576, so the floor is 0.7. This is a floor, not a substitute for the structural counts — the other damaged document kept its volume and lost its structure, and only the counts saw it.--scoreswrites one<deviation> <path>line per document;translate-verified.mjsremembers the lowest-scoring version of each and restores it before giving up. When the loop does give up, what it hands to a human is the closest the translator ever got.Together, 1 and 2 also address the two failures that were stable across all three attempts, and 4 addresses the two that degraded.
Verification
No API key, no
mainrun.yarn test:translationnow runs both suites:The retry-loop suite is new. It drives the real loop through
TRANSLATE_CMD— the seam that already existed for this — with a throwaway script that returns a scripted sequence of versions, and asserts what the loop retries, what it keeps, and what it leaves behind.Every new check was verified by reverting it and watching the right assertions go red:
Run against the real damaged output from 33160218909, the checker goes from
397 pass / 7 failto401 pass / 3 fail: the stray fence inHow_to_Migrate_MySQL_57_to_80.mdis repaired rather than reported, and the size shortfall inSoftware_Supply_Chain_Security_…is now named directly instead of being inferred from four count mismatches.What this does not fix
mainis still red. Three documents still fail on lost or invented content, which no check here can reconstruct:Install_Multi-Primary_Service_Mesh_on_Different_Networks.md(7 invented table rows, identical across all three attempts),Pipeline_Policy_Constraints_with_Tekton_and_Kyverno.md, andSoftware_Supply_Chain_Security_…. They need either splitting, hand-translation, ori18n.disableAutoTranslation.maxChunkSizeis hardcoded,TranslateOptionsexposes onlysystemPromptanduserPrompt, and the package'sexportsmap blocks importingtranslate()directly — so per-chunk gating and splitting on heading boundaries rather than byte count cannot be done from this repository. The changes doom would need: make the chunk size configurable, split at structural boundaries, checkfinish_reason, and reject a chunk whose output is a fraction of its input. Worth raising upstream; the reproduction is this run.