expose complex-scripts feature (#24094) - #25674
Open
unclepomedev wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Objective
Fixes #24094
(Rebase of #24683, closed due to merge conflicts on
main. Re-verified the fix still works on currentmain. That PR had received a community review approval.)Text containing complex scripts that lack explicit word boundaries (CJK, Thai, Khmer, Lao, Myanmar) is segmented by parley's lightweight segmenter, which emits a data error per layout:
parley added a
complex-scriptsCargo feature that switches to dictionary-based segmentation, but it was not reachable from Bevy.Solution
complex_script_segmentationfeature tobevy_textthat forwards toparley/complex-scripts, propagated throughbevy_internalandbevy. It is off by default because the bundled segmentation dictionary increases binary size (see below).This supplies the segmentation data and enables dictionary-based word segmentation; it does not change CJK line-break positions, which ICU keeps permissive by design.
Testing
cargo build -p bevy_text -p bevy_ui -p bevy_ui_widgets, with default features and with--features complex_script_segmentation.cargo tree -e features -i parleyconfirms parley'scomplex-scriptsfeature is only enabled whencomplex_script_segmentationis requested (not pulled in by any default/collection feature).Text2dexample with a Japanese string in a width-bounded box (TextBounds,LineBreak::WordBoundary) underRUST_LOG=warn:No segmentation model for language: jaerror is logged.--features complex_script_segmentation: the error is gone.cargo run -p ci(lints + compile + tests). One unrelated local failure:bevy_ecs ... filtered_backtrace_test, which also fails on main on my machine (an environment-dependent backtrace-symbolization test, unrelated to this PR).Binary size impact
Measured with a stripped release build of the
text_debugexample:complex_script_segmentation: 57,018,696 bytescomplex_script_segmentation: 60,832,984 bytesIt seems the feature should be an opt-in.