Skip to content

fix: preserve short words during Arabic affix removal - #44

Merged
AbdelrahmanBayoumi merged 2 commits into
mainfrom
fix/affixes-short-result-guard
Nov 3, 2026
Merged

AbdelrahmanBayoumi merged 2 commits into
mainfrom
fix/affixes-short-result-guard

Conversation

@AbdelrahmanBayoumi

Copy link
Copy Markdown
Member

Summary

Prevent removeArabicAffixes from corrupting short words such as نور → ور and بنت → نت. Keep the original word when the complete stripped result contains fewer than three Arabic letters, excluding diacritics, tatweel, digits, and other scripts.

  • Process words independently across whitespace, punctuation, and symbols while preserving interior separators.
  • Preserve existing banned-word matching, including exact matches such as فلسطين.
  • Keep the internal stripping helper out of public exports and preserve tree shaking.
  • Update English/Arabic documentation and Unreleased notes without changing published changelog entries.

Scope and limitations

This remains heuristic affix stripping, not morphological analysis. Legitimate two-letter stems remain unchanged (اليد → اليد), and يحبون remains unchanged. Existing incorrect stripping in longer words is outside this fix.

removeArabicAffixes requires Unicode property escape support; unrelated APIs can still load without it.

Validation

  • npm test: 134 tests passed, including fresh builds and artifact checks.
  • Lint, formatting, TypeScript checks, and git diff --check passed.
  • 828 seeded comparisons confirmed unchanged banned-word obfuscation outputs across CJS and ESM.
  • Public exports, tree shaking, and published changelog history verified.

@AbdelrahmanBayoumi
AbdelrahmanBayoumi merged commit ff32f86 into main Nov 3, 2026
4 checks passed
@AbdelrahmanBayoumi

Copy link
Copy Markdown
Member Author

closes #26

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant