From 61385a519c169135c5eb51ea652f199dd24c91e5 Mon Sep 17 00:00:00 2001 From: Abdelrahman Bayoumi Date: Tue, 27 Oct 2026 19:08:34 +0300 Subject: [PATCH 1/2] fix: guard short affix results and preserve banned-word matching --- CHANGELOG.md | 2 + README.md | 2 +- README_AR.md | 2 +- src/scripts/affixes.ts | 41 +++++++++--------- src/scripts/index.ts | 2 +- src/scripts/tashfeer.ts | 4 +- tests/affixes.test.ts | 88 ++++++++++++++++++++++++++++++++++++++ tests/artifacts.test.ts | 28 ++++++++++++ tests/edge-cases.test.ts | 4 +- tests/scripts.test.ts | 18 ++++---- tests/tree-shaking.test.ts | 20 ++++++++- 11 files changed, 172 insertions(+), 39 deletions(-) create mode 100644 tests/affixes.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 0d33f1b..2c279f2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,6 +26,8 @@ - Release publishing now validates the tag/version and runs formatting, lint, and tests before publishing the verified build. ### Fixed +- Retain each original word in `removeArabicAffixes` when the complete stripped result has fewer than three Arabic letters, excluding marks, tatweel, digits, and other scripts from the count. This changes outputs such as `نور`, `بنت`, and `اليد`; banned-word recognition retains its existing stripping behavior. +- Process affix removal independently across whitespace, punctuation, and symbol boundaries while preserving interior separators. Defer Unicode property regular expressions until this function is called, so unsupported engines can still load and use unrelated APIs. - Recognize exact banned-word matches before stripping affixes, so listed words such as `فلسطين` are obfuscated while retaining existing affix-based similarity matching. - Calculate `similarityScore` lengths and edit distance after lowercasing so Unicode case expansion cannot produce negative scores or incorrect similarity ratios. - Prevent invalid levels from hanging `tashfeer`, `tashfeerBannedWords`, and `toOldArabicAndTashfeerBannedWords`: non-number values throw `TypeError`; negative, fractional, and non-finite numbers throw `RangeError`. Random-index generation handles empty input and caps the requested count at the word length ([#35](https://github.com/Seen-Arabic/Arabic-Services-JavaScript/issues/35), [PR #36](https://github.com/Seen-Arabic/Arabic-Services-JavaScript/pull/36)). diff --git a/README.md b/README.md index f82dac4..d72e1a4 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,7 @@ Can be used in Node.js and the browser. - **Convert To Old Arabic And Tashfeer Banned Words**: Transform Arabic text into old script and replace Banned Arabic text with visually similar characters for encoding purposes. (Banned words are words that considered as hate speech in social media) - **Tashfeer**: Replaces Arabic text with visually similar characters for encoding purposes. - **Tashfeer Banned Words**: Replaces Banned Arabic text with visually similar characters for encoding purposes. (Banned words are words that considered as hate speech in social media) -- **Remove Arabic Affixes**: Removes predefined affixes (prefixes and suffixes) from an Arabic word if it starts or ends with those affixes. +- **Remove Arabic Affixes**: Removes listed prefixes and suffixes from letter/mark/number sequences, preserving interior whitespace, punctuation, and symbols. Retains the original word when the complete result has fewer than three Arabic letters. This heuristic can retain legitimate two-letter stems (e.g. `اليد` stays `اليد`) and still strip original letters in longer words; it is not morphological analysis. This function requires Unicode property escape support; other APIs do not acquire that requirement on import. Banned-word matching retains its previous stripping rules. - **Word To Letters**: Convert Arabic word to its pronounced letters. - ... and more! diff --git a/README_AR.md b/README_AR.md index 9696cd5..bbe7ae0 100644 --- a/README_AR.md +++ b/README_AR.md @@ -39,7 +39,7 @@ - **التحويل إلى العربية القديمة وتشفير الكلمات المحظورة:** تحويل النص العربي إلى الخط العربي القديم واستبدال النصوص العربية المحظورة بأحرف مشابهة بصريًا لأغراض الترميز. (الكلمات المحظورة هي الكلمات التي تعتبر كلمات مسيئة في وسائل التواصل الاجتماعي) - **التشفير:** استبدال النص العربي بأحرف مشابهة بصريًا لأغراض الترميز. - **تشفير الكلمات المحظورة:** استبدال النصوص العربية المحظورة بأحرف مشابهة بصريًا لأغراض الترميز. (الكلمات المحظورة هي الكلمات التي تعتبر كلمات مسيئة في وسائل التواصل الاجتماعي) -- **إزالة اللواحق العربية:** إزالة اللواحق المحددة (البادئات واللاحقات) من كلمة عربية إذا بدأت أو انتهت بهذه اللواحق. +- **إزالة اللواحق العربية:** إزالة السوابق واللواحق المحددة من تتابعات الحروف والعلامات والأرقام، مع الحفاظ على الفواصل الداخلية والأسطر وعلامات الترقيم والرموز. تُبقي الكلمة الأصلية إذا احتوت نتيجة القص الكاملة على أقل من ثلاثة أحرف عربية. هذه قواعد تقريبية؛ فقد تُبقي كلمات ذات أصل ثنائي مثل `اليد` كما هي، وقد تحذف حروفًا أصلية من كلمات أطول، وليست تحليلًا صرفيًا. تتطلب هذه الدالة دعم Unicode property escapes، دون فرض هذا المتطلب عند تحميل بقية الخدمات. تحتفظ مطابقة الكلمات المدرجة بقواعد القص السابقة. - **الكلمة إلى حروفها:** تحويل الكلمة العربية إلى حروفها المنطوقة. - ...والمزيد! diff --git a/src/scripts/affixes.ts b/src/scripts/affixes.ts index 0e729ab..e02cc78 100644 --- a/src/scripts/affixes.ts +++ b/src/scripts/affixes.ts @@ -1,18 +1,10 @@ -import { ARABIC_PREFIXES, ARABIC_SUFFIXES } from '../constants/index.js'; -import { setCharAt } from '../utils/index.js'; +import { ARABIC_PREFIXES, ARABIC_SUFFIXES } from '../constants/arabic-letters.js'; +import { setCharAt } from '../utils/string.util.js'; -/** - * Removes predefined affixes (prefixes and suffixes) from an Arabic word if it starts or ends with those affixes. - * This function is designed specifically for processing Arabic text, where certain affixes might need to be removed - * for linguistic, stylistic, or morphological reasons. - * - * @param {string} word - The Arabic word from which the affixes are to be removed. - * @returns {string} The word after removing any matching affixes. Returns the original word trimmed if no affix matches are found. - */ -function removeArabicAffixesFromWord(word: string): string { +/** Legacy word stripping, retained independently for banned-word recognition. */ +export function removeArabicAffixesFromWord(word: string): string { word = word.trim(); if (ARABIC_PREFIXES.includes(word.substring(0, 2))) { - // For: ALEF & LAM word = setCharAt(word, 0, ''); word = setCharAt(word, 0, ''); } else if (ARABIC_PREFIXES.includes(word.substring(0, 1))) { @@ -30,18 +22,25 @@ function removeArabicAffixesFromWord(word: string): string { } /** - * Removes predefined affixes (prefixes and suffixes) from an Arabic text if words start or end with those affixes. - * This function is designed specifically for processing Arabic text, where certain affixes might need to be removed - * for linguistic, stylistic, or morphological reasons. + * Removes listed affixes from each word, retaining its original form if the complete + * stripped result has fewer than three Arabic-script letters. Diacritics, tatweel, + * digits and other scripts do not count toward that minimum. This is a heuristic, + * not morphological analysis; legitimate two-letter stems are also retained in full. + * Words are letter/mark/number sequences; interior whitespace, punctuation and + * symbols are preserved. Unicode property escapes are required when this is called. * * @param {string} text - The Arabic text from which the affixes are to be removed. * @returns {string} The text after removing any matching affixes from each word. Returns the original text trimmed if no affix matches are found. */ export function removeArabicAffixes(text: string): string { - let new_sentence = ''; - text = text.trim(); - for (const word of text.split(' ')) { - new_sentence += removeArabicAffixesFromWord(word) + ' '; - } - return new_sentence.trim(); + const wordPattern = new RegExp('[\\p{L}\\p{M}\\p{N}]+', 'gu'); + const arabicLetterPattern = new RegExp('(?=\\p{Script=Arabic})\\p{Letter}', 'gu'); + return text.trim().replace(wordPattern, (word) => { + const stripped = removeArabicAffixesFromWord(word); + arabicLetterPattern.lastIndex = 0; + for (let count = 0; count < 3; count++) { + if (!arabicLetterPattern.exec(stripped)) return word; + } + return stripped; + }); } diff --git a/src/scripts/index.ts b/src/scripts/index.ts index cd68b17..19ae41a 100644 --- a/src/scripts/index.ts +++ b/src/scripts/index.ts @@ -1,4 +1,4 @@ -export * from './affixes.js'; +export { removeArabicAffixes } from './affixes.js'; export * from './letters.js'; export * from './old-arabic.js'; export * from './tashfeer.js'; diff --git a/src/scripts/tashfeer.ts b/src/scripts/tashfeer.ts index 6decf34..c155d40 100644 --- a/src/scripts/tashfeer.ts +++ b/src/scripts/tashfeer.ts @@ -8,7 +8,7 @@ import { YAA, } from '../constants/arabic-letters.js'; import { similarityScore } from '../utils/index.js'; -import { removeArabicAffixes } from './affixes.js'; +import { removeArabicAffixesFromWord } from './affixes.js'; import { toOldArabic } from './old-arabic.js'; /** @@ -192,7 +192,7 @@ function bannedSimilarityRatio(string: string): number { */ function checkIfBannedWord(string: string): boolean { const std_ratio = 70; - return BANNED_WORDS.includes(string) || bannedSimilarityRatio(removeArabicAffixes(string)) >= std_ratio; + return BANNED_WORDS.includes(string) || bannedSimilarityRatio(removeArabicAffixesFromWord(string)) >= std_ratio; } /** diff --git a/tests/affixes.test.ts b/tests/affixes.test.ts new file mode 100644 index 0000000..b6ca9c5 --- /dev/null +++ b/tests/affixes.test.ts @@ -0,0 +1,88 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { + removeArabicAffixes, + tashfeer, + tashfeerBannedWords, + toOldArabic, + toOldArabicAndTashfeerBannedWords, +} from '../src'; + +describe('Short affix-removal results', () => { + it.each([ + 'نور', + 'بنت', + 'اليد', + 'الدم', + 'بيده', + 'الري', + 'الكي', + 'غزة', + 'حرية', + 'عزة', + 'بِنْت', + 'نـور', + 'نور🙂', + 'نabc', + 'ال12a', + 'ال🙂a', + 'ال𞸀ت', + ])('should retain the original %s when the complete result has fewer than three Arabic letters', (word) => { + expect(removeArabicAffixes(word)).toBe(word); + }); + + it.each([ + ['الكتاب', 'كتاب'], + ['كتابهم', 'كتاب'], + ['مدرسة', 'مدرس'], + ['ال𞸀بت', '𞸀بت'], + ['تفاح', 'فاح'], + ['فلسطين', 'فلسط'], + ['كالسيوم', 'كالسيوم'], + ['والدان', 'والدان'], + ])('should retain the existing result of %s when it has at least three Arabic letters', (word, expected) => { + expect(removeArabicAffixes(word)).toBe(expected); + }); + + it('should guard each word without changing existing trimming or repeated spaces', () => { + expect(removeArabicAffixes(' نور الكتاب بنت ')).toBe('نور كتاب بنت'); + }); + + it.each([ + ['نور\nبنت', 'نور\nبنت'], + ['نور\tبنت', 'نور\tبنت'], + ['نور،بنت', 'نور،بنت'], + ['نور—بنت', 'نور—بنت'], + ['الكتاب،طلابهم', 'كتاب،طلاب'], + ['(الكتاب)', '(كتاب)'], + ['نور🙂كتابهم', 'نور🙂كتاب'], + [' نور \t الكتاب\r\nبنت ', 'نور \t كتاب\r\nبنت'], + ])('should process words independently and preserve separators in %j', (text, expected) => { + expect(removeArabicAffixes(text)).toBe(expected); + }); +}); + +describe('Banned-word recognition compatibility', () => { + afterEach(() => vi.restoreAllMocks()); + + it.each([ + ['فلسطين', true], + ['غزة', true], + ['حرية', true], + ['عزة', true], + ['كتابهم', true], + ['نور', false], + ['بنت', false], + ['جميلة', false], + ])('should preserve recognition of %s in both public obfuscation functions', (word, recognized) => { + let seed = 1; + vi.spyOn(Math, 'random').mockImplementation(() => { + seed = (Math.imul(seed, 1664525) + 1013904223) >>> 0; + return seed / 4294967296; + }); + const obfuscated = tashfeer(word); + seed = 1; + expect(tashfeerBannedWords(word, 1)).toBe(recognized ? obfuscated : word); + seed = 1; + expect(toOldArabicAndTashfeerBannedWords(word, 1)).toBe(recognized ? obfuscated : toOldArabic(word)); + }); +}); diff --git a/tests/artifacts.test.ts b/tests/artifacts.test.ts index c6d7d0d..9d28f07 100644 --- a/tests/artifacts.test.ts +++ b/tests/artifacts.test.ts @@ -101,6 +101,32 @@ describe('Package Artifacts & Module Integrity', () => { expect(arabicServices.removeTashkeel('الخَيْلُ وَاللّيْلُ')).toBe('الخيل والليل'); expect(arabicServices.removeTatweel('كتــــاب')).toBe('كتاب'); + expect(arabicServices.removeArabicAffixes('نور بنت الكتاب')).toBe('نور بنت كتاب'); + expect(arabicServices).not.toHaveProperty('removeArabicAffixesFromWord'); + }); + + it('should load unrelated APIs without Unicode property escapes and defer that requirement to affix removal', () => { + class LimitedRegExp extends RegExp { + constructor(pattern: string | RegExp, flags?: string) { + if (typeof pattern === 'string' && pattern.includes('\\p{')) { + throw new SyntaxError('Unicode property escapes are unavailable'); + } + super(pattern, flags); + } + } + const sandbox: { + window: Record; + RegExp: typeof LimitedRegExp; + ArabicServices?: Record string>; + } = { window: {}, RegExp: LimitedRegExp }; + vm.runInNewContext(fs.readFileSync(path.join(rootDir, packageJson.unpkg), 'utf8'), sandbox); + const api = (sandbox.window.ArabicServices || sandbox.ArabicServices) as Record< + string, + (text: string) => string + >; + expect(api.removeTashkeel('مَرْحَبًا')).toBe('مرحبا'); + expect(api.tashfeerBannedWords('نور')).toBe('نور'); + expect(() => api.removeArabicAffixes('نور')).toThrow('Unicode property escapes are unavailable'); }); it('should support in-process CommonJS loading via Node.js createRequire()', () => { @@ -113,6 +139,8 @@ describe('Package Artifacts & Module Integrity', () => { expect(typeof cjsDist.ArabicServices).toBe('object'); expect(cjsDist.removeTashkeel('مَرْحَبًا')).toBe('مرحبا'); expect(cjsDist.ArabicServices.removeTatweel('رائـــــع')).toBe('رائع'); + expect(cjsDist.removeArabicAffixes('نور بنت الكتاب')).toBe('نور بنت كتاب'); + expect(cjsDist).not.toHaveProperty('removeArabicAffixesFromWord'); }); it('should execute purely as Isomorphic/Universal functions without DOM dependencies', async () => { diff --git a/tests/edge-cases.test.ts b/tests/edge-cases.test.ts index 06064f2..e436ecb 100644 --- a/tests/edge-cases.test.ts +++ b/tests/edge-cases.test.ts @@ -34,8 +34,8 @@ describe('Arabic NLP Edge Cases & Robustness', () => { it('should handle words that are shorter than standard affix length', () => { // 'و' is not in ARABIC_PREFIXES so it stays unchanged expect(removeArabicAffixes('و')).toBe('و'); - // 'ال' is an exact 2-letter prefix and is removed completely - expect(removeArabicAffixes('ال')).toBe(''); + // Keep the original when removing the prefix would leave no Arabic letters. + expect(removeArabicAffixes('ال')).toBe('ال'); // single letter prefix 'ي' expect(removeArabicAffixes('يقرأ')).toBe('قرأ'); }); diff --git a/tests/scripts.test.ts b/tests/scripts.test.ts index fdb012a..1d6fa0d 100644 --- a/tests/scripts.test.ts +++ b/tests/scripts.test.ts @@ -182,10 +182,10 @@ describe('#wordToLetters', () => { }); describe('#removeArabicAffixes', () => { - it('should remove "أ" prefix from a word', () => { + it('should preserve a word when removing "أ" leaves fewer than three Arabic letters', () => { const word = 'أمل'; const result = ArabicServices.removeArabicAffixes(word); - expect(result).toEqual('مل'); + expect(result).toEqual('أمل'); }); it('should remove "ا" prefix & suffix "ة" from a word', () => { @@ -206,10 +206,10 @@ describe('#removeArabicAffixes', () => { expect(result).toEqual('كتاب'); }); - it('should remove "ي" prefix from a word', () => { + it('should preserve a word when removing "ي" leaves fewer than three Arabic letters', () => { const word = 'يوم'; const result = ArabicServices.removeArabicAffixes(word); - expect(result).toEqual('وم'); + expect(result).toEqual('يوم'); }); it('should remove "ت" prefix from a word', () => { @@ -218,16 +218,16 @@ describe('#removeArabicAffixes', () => { expect(result).toEqual('فاح'); }); - it('should remove "ن" prefix from a word', () => { + it('should preserve a word when removing "ن" leaves fewer than three Arabic letters', () => { const word = 'نجم'; const result = ArabicServices.removeArabicAffixes(word); - expect(result).toEqual('جم'); + expect(result).toEqual('نجم'); }); - it('should remove "ب" prefix from a word', () => { + it('should preserve a word when removing "ب" leaves fewer than three Arabic letters', () => { const word = 'بيت'; const result = ArabicServices.removeArabicAffixes(word); - expect(result).toEqual('يت'); + expect(result).toEqual('بيت'); }); it('should remove "ة" suffix from a word', () => { @@ -281,7 +281,7 @@ describe('#removeArabicAffixes', () => { it('should remove prefix & suffix from a sentence', () => { const word = 'المدرسون يحبون طلابهم'; const result = ArabicServices.removeArabicAffixes(word); - expect(result).toEqual('مدرس حب طلاب'); + expect(result).toEqual('مدرس يحبون طلاب'); }); }); diff --git a/tests/tree-shaking.test.ts b/tests/tree-shaking.test.ts index b6bbe67..2adcb86 100644 --- a/tests/tree-shaking.test.ts +++ b/tests/tree-shaking.test.ts @@ -61,11 +61,19 @@ describe('Tree Shaking & Named Exports', () => { path.join(tempDir, 'full.mjs'), `import { ArabicServices } from ${entryPoint}; globalThis.result = ArabicServices;`, ); + fs.writeFileSync( + path.join(tempDir, 'affixes.mjs'), + `import { removeArabicAffixes } from ${entryPoint}; globalThis.result = removeArabicAffixes('نور بنت الكتاب');`, + ); await new Promise((resolve, reject) => { webpack( { mode: 'production', - entry: { single: path.join(tempDir, 'single.mjs'), full: path.join(tempDir, 'full.mjs') }, + entry: { + single: path.join(tempDir, 'single.mjs'), + full: path.join(tempDir, 'full.mjs'), + affixes: path.join(tempDir, 'affixes.mjs'), + }, output: { path: path.join(tempDir, 'output'), filename: '[name].js' }, devtool: false, }, @@ -79,16 +87,24 @@ describe('Tree Shaking & Named Exports', () => { }); const singleBundle = fs.readFileSync(path.join(tempDir, 'output/single.js'), 'utf8'); const fullBundle = fs.readFileSync(path.join(tempDir, 'output/full.js'), 'utf8'); + const affixesBundle = fs.readFileSync(path.join(tempDir, 'output/affixes.js'), 'utf8'); const singleContext = { result: undefined }; const fullContext = { result: undefined }; + const affixesContext = { result: undefined }; vm.runInNewContext(singleBundle, singleContext); vm.runInNewContext(fullBundle, fullContext); + vm.runInNewContext(affixesBundle, affixesContext); expect(singleContext.result).toBe('مرحبا'); expect(fullContext.result).toHaveProperty('tashfeerBannedWords', expect.any(Function)); expect(singleBundle).not.toContain('tashfeerBannedWords'); + expect(affixesContext.result).toBe('نور بنت كتاب'); + expect(affixesBundle).not.toContain('tashfeerBannedWords'); + expect(affixesBundle).not.toContain('فلسطين'); + expect(fullContext.result).not.toHaveProperty('removeArabicAffixesFromWord'); expect(Buffer.byteLength(singleBundle)).toBeLessThan(Buffer.byteLength(fullBundle) / 2); + expect(Buffer.byteLength(affixesBundle)).toBeLessThan(Buffer.byteLength(fullBundle) / 2); console.info( - `Minified consumer bundles: single=${Buffer.byteLength(singleBundle)} B, full=${Buffer.byteLength(fullBundle)} B`, + `Minified consumer bundles: single=${Buffer.byteLength(singleBundle)} B, affixes=${Buffer.byteLength(affixesBundle)} B, full=${Buffer.byteLength(fullBundle)} B`, ); } finally { fs.rmSync(tempDir, { recursive: true, force: true }); From 3b74ef31ad7ef82d7d46e05894ef3c4be2ba4b46 Mon Sep 17 00:00:00 2001 From: Abdelrahman Bayoumi Date: Mon, 2 Nov 2026 17:08:18 +0200 Subject: [PATCH 2/2] docs: explain affix removal and banned-word matching --- CHANGELOG.md | 1 + README.md | 2 + README_AR.md | 2 + docs/README.md | 14 ++++ docs/algorithms/arabic-affixes.md | 69 ++++++++++++++++++ docs/algorithms/banned-word-matching.md | 93 +++++++++++++++++++++++++ 6 files changed, 181 insertions(+) create mode 100644 docs/README.md create mode 100644 docs/algorithms/arabic-affixes.md create mode 100644 docs/algorithms/banned-word-matching.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 2c279f2..979569d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## [Unreleased] ### Added +- Technical articles explaining Arabic affix removal and banned-word matching, including design decisions, examples, and known limitations. - Integration tests for CJS, ESM, and browser UMD artifacts. - Arabic NLP boundary and ReDoS resilience test suites. - Direct CDN links (jsdelivr & unpkg) and GitHub release bundle guide in documentation. diff --git a/README.md b/README.md index d72e1a4..4b3c710 100644 --- a/README.md +++ b/README.md @@ -50,6 +50,8 @@ Can be used in Node.js and the browser. ## 🚀 Getting Started +For implementation details, examples, and known limitations, see the [technical documentation](docs/README.md). + Arabic-Services was designed to work in both the browser and NodeJS applications. ### Browser diff --git a/README_AR.md b/README_AR.md index bbe7ae0..c7dcadd 100644 --- a/README_AR.md +++ b/README_AR.md @@ -47,6 +47,8 @@ ## 🚀 التثبيت +لشرح الخوارزميات وأسباب القرارات والأمثلة والحدود المعروفة، راجع [التوثيق التقني باللغة الإنجليزية](docs/README.md). + تم تصميم خدمات العربية لتعمل في كل من المتصفح وتطبيقات NodeJS. ### المتصفح (Browser) diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..e69314d --- /dev/null +++ b/docs/README.md @@ -0,0 +1,14 @@ +# Technical documentation + +These articles explain how the current source works, why its rules exist, and where its results can be misleading. They describe the repository revision you are reading, including behavior recorded under [Unreleased](../CHANGELOG.md); an installed npm release may behave differently. + +| Article | What it explains | +| ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | +| [Arabic affix removal](algorithms/arabic-affixes.md) | Listed prefix/suffix removal, the short-result guard, word boundaries, and linguistic limitations. | +| [Banned-word matching](algorithms/banned-word-matching.md) | Exact matching, similarity matching, and why matching keeps its own stripping policy. | + +For installation and public API usage, start with the [English README](../README.md) or [Arabic README](../README_AR.md). + +## Keeping articles accurate + +When changing documented behavior, update its article and regression examples together. Explain the decision and its trade-offs; link to the implementation and tests rather than copying their full contents. Include known failures alongside successful examples. Treat experiment results as evidence for a particular input set, not as a general accuracy guarantee. diff --git a/docs/algorithms/arabic-affixes.md b/docs/algorithms/arabic-affixes.md new file mode 100644 index 0000000..24530b6 --- /dev/null +++ b/docs/algorithms/arabic-affixes.md @@ -0,0 +1,69 @@ +# Arabic affix removal: a short-result guard + +`removeArabicAffixes(text: string): string` removes predefined prefixes and suffixes. It can be useful when a caller deliberately wants a lossy text transformation. It does not identify grammatical affixes, recover roots, or guarantee that different forms of a word produce the same output. + +This article describes the current source, including the guard and boundary handling recorded under [Unreleased](../../CHANGELOG.md). + +## Why the guard exists + +The original rules remove letters by their position and membership in a list. They cannot tell whether the initial `ن` in `نور` is an affix or part of the word. Applying those rules without a guard turns `نور` into `ور` and `بنت` into `نت`. + +The current policy keeps the original word if the **complete stripped result** contains fewer than three Arabic-script letters. Three is a conservative cutoff for this transformation, not a linguistic claim that Arabic stems must have three letters. + +## How a word is processed + +1. Trim whitespace from the beginning and end of the whole input. +2. Find each contiguous sequence of Unicode letters, combining marks, and numbers. Adjacent Arabic, Latin, and numeric characters belong to the same sequence; this is not language-aware tokenization. +3. Try one listed prefix, checking two characters before one character. +4. On the remaining text, try one listed suffix, again checking two characters before one character. Neither step repeats until no affix remains. +5. Count letters that have both Unicode `Script=Arabic` and category `Letter` in the candidate. Keep the candidate only if at least three are found; otherwise keep the original sequence. + +The current lists live in [arabic-letters.ts](../../src/constants/arabic-letters.ts): + +- Prefixes: `أ`, `ا`, `إ`, `ال`, `ي`, `ت`, `ن`, `ب`. +- Suffixes: `ة`, `ه`, `ي`, `ى`, `ية`, `ين`, `ون`, `هم`. + +Diacritics, tatweel, digits, punctuation, and other scripts do not satisfy the minimum. Interior whitespace, punctuation, and symbols are preserved. The function does not remove diacritics or normalize spelling before trying the lists. + +## Examples and counterexamples + +| Input | Candidate after listed stripping | Returned text | Reason | +| -------- | -------------------------------- | ------------- | ------------------------------------------------------------------------------------------------------ | +| `نور` | `ور` | `نور` | Two Arabic letters remain. | +| `بنت` | `نت` | `بنت` | Two Arabic letters remain. | +| `الكتاب` | `كتاب` | `كتاب` | Four Arabic letters remain. | +| `كتابهم` | `كتاب` | `كتاب` | The listed suffix is removed. | +| `اليد` | `يد` | `اليد` | A legitimate two-letter stem is also rejected. | +| `يحبون` | `حب` | `يحبون` | The complete candidate is too short; the function does not retry suffix-only removal to produce `يحب`. | +| `تفاح` | `فاح` | `فاح` | The guard cannot detect removal of an original letter from a longer word. | +| `فلسطين` | `فلسط` | `فلسط` | A listed ending can be part of the original word. | + +```javascript +import { removeArabicAffixes } from 'arabic-services'; + +removeArabicAffixes(' نور،الكتاب\nبنت '); // 'نور،كتاب\nبنت' +removeArabicAffixes('المدرسون يحبون طلابهم'); // 'مدرس يحبون طلاب' +``` + +The fallback is atomic: it rejects the entire stripped candidate rather than approving each deletion separately. Otherwise a partial deletion could survive even when a later deletion should make us retain the original word. + +## Why this is not a stemmer replacement + +Expanding the prefix list would change which words are cut; it would not resolve whether those letters are original. Replacing this function with a search stemmer or morphological analyzer would also change its output contract and potentially require dictionaries or dependencies. The chosen fix addresses short destructive results while retaining the existing lists and keeping zero runtime dependencies. + +This trade-off leaves real limitations: short stems may not be reduced, longer words may still be damaged, and stripping an initial letter before a combining mark can leave that mark without its original base. Use a separate morphological solution when grammatical interpretation is required. Do not use this transformation as a prerequisite for assuming that a word is correctly recognized. + +## Runtime and module boundaries + +Calling this function requires Unicode property escape support. Its regular expressions are constructed inside the function, so loading the package or calling unrelated APIs does not introduce that requirement. An unsupported engine will throw a `SyntaxError` when the function is called; use a runtime with that support if affix removal is needed. + +The underlying stripping routine is shared internally with [banned-word matching](banned-word-matching.md), which deliberately does not apply this guard. That routine is not exported from the package root. Named ESM consumers can import `removeArabicAffixes` without retaining the banned-word matching code and list in their bundle. + +## Implementation and verification + +- [Affix implementation](../../src/scripts/affixes.ts) and [public script exports](../../src/scripts/index.ts). +- [Guard, Unicode, separator, and matching regression tests](../../tests/affixes.test.ts). +- [Sentence examples](../../tests/scripts.test.ts) and [short-input boundaries](../../tests/edge-cases.test.ts). +- [Built-module and runtime-isolation checks](../../tests/artifacts.test.ts) and [Webpack tree-shaking checks](../../tests/tree-shaking.test.ts). + +Run `npm run test:unit` for source behavior. Run `npm test` to also rebuild and validate package artifacts. These checks establish the tested contract, not general linguistic correctness. diff --git a/docs/algorithms/banned-word-matching.md b/docs/algorithms/banned-word-matching.md new file mode 100644 index 0000000..bec63ae --- /dev/null +++ b/docs/algorithms/banned-word-matching.md @@ -0,0 +1,93 @@ +# Banned-word matching: exact lookup before similarity + +`tashfeerBannedWords` and `toOldArabicAndTashfeerBannedWords` select tokens for visual obfuscation using the package's predefined `BANNED_WORDS` list. The name describes that list's role in the API; it is not a claim that a word is harmful or that an external platform bans it. The implementation does not consult platform policies or understand sentence meaning. + +This article describes the current source, including exact-match handling recorded under [Unreleased](../../CHANGELOG.md). + +## The matching pipeline + +Both functions trim the input and split it on the literal space character (`' '`). For each resulting token: + +1. Check whether the original token is an exact entry in `BANNED_WORDS`. This comparison is case-sensitive and does not normalize the token. +2. If not, apply the legacy listed prefix/suffix stripping routine described in [Arabic affix removal](arabic-affixes.md), **without its three-letter guard**. +3. Compare that stripped token with every entry in the list using `similarityScore`. List entries themselves are not stripped in this step. +4. Select the token if its highest score is at least `0.7` (70%). + +There is no additional diacritic removal, spelling normalization, or punctuation separation in this matching pipeline. In particular, it does not use the public affix function's newer word-boundary handling. + +## Why exact matching comes first + +`فلسطين` is an explicit list entry. Stripping first produces `فلسط`, which can miss the 70% threshold against the original entry. The exact lookup ensures an explicitly listed token is recognized before a lossy transformation can change it. + +Matching still uses similarity after an exact miss, so a token does not have to appear literally in the list to be selected. + +| Token | Exact entry? | Legacy candidate, if needed | Outcome | +| -------- | ------------ | --------------------------- | ----------------------------------------------------------- | +| `فلسطين` | Yes | Not needed | Selected by exact lookup. | +| `غزة` | No | `غز` | Selected: the candidate is an exact match for a list entry. | +| `حرية` | No | `حر` | Selected: the candidate is an exact match for a list entry. | +| `عزة` | No | `عز` | Selected: the candidate is an exact match for a list entry. | +| `كتابهم` | No | `كتاب` | Selected: the candidate is an exact match for a list entry. | +| `نور` | No | `ور` | Not selected by the current list and threshold. | + +## What the similarity score measures + +`similarityScore` lowercases both strings, then calculates Levenshtein edit distance over UTF-16 code units. It returns: + +```text +1 - distance / max(lowercasedFirst.length, lowercasedSecond.length) +``` + +For two empty strings it returns `1`. The score measures character edits, not meaning, confidence, or probability. A 70% threshold is an existing matching policy, not a measured 70% linguistic accuracy rate. + +```javascript +import { similarityScore, tashfeerBannedWords } from 'arabic-services'; + +similarityScore('كتاب', 'كتاب'); // 1 +similarityScore('كتاب', 'كتا'); // 0.75 +tashfeerBannedWords('نور'); // 'نور' +tashfeerBannedWords('فلسطين'); // Randomized visual obfuscation; no fixed output string. +``` + +## Why matching keeps the legacy stripping policy + +The public affix function now retains `غزة`, `حرية`, and `عزة` because their stripped candidates have fewer than three Arabic letters. Matching historically relies on those short candidates (`غز`, `حر`, `عز`), which are stored in the list. + +Reusing the guarded public function would change recognition. Instead, both paths share the existing stripping routine while applying different acceptance rules. The routine is imported directly between source modules and omitted from package-root exports; it does not add a public matching API or a new configuration flag. + +## What happens after a match + +Both public functions accept a `levelOfTashfeer` argument with default `2`. They apply obfuscation to the original selected token, not its stripped candidate. The output varies because replacement positions and characters use randomness. + +For tokens that are not selected: + +- `tashfeerBannedWords` keeps the token unchanged. +- `toOldArabicAndTashfeerBannedWords` converts it using `toOldArabic`. + +The level must be a non-negative finite integer. Negative, fractional, or non-finite values throw `RangeError`; non-number values throw `TypeError`. + +```javascript +import { tashfeerBannedWords } from 'arabic-services'; + +try { + tashfeerBannedWords('فلسطين', -1); +} catch (error) { + if (!(error instanceof RangeError)) throw error; + // Choose a non-negative integer level before retrying. +} +``` + +## Known limitations + +Literal-space splitting means newlines, tabs, and attached punctuation are not independent matching boundaries. They can change the token being compared, so recognition of a word alone does not guarantee recognition inside every surrounding text format. + +Fuzzy matching can select unrelated words or miss intended variants. The list and threshold encode application-specific choices, not a general moderation classifier. Visual substitution also does not provide cryptographic confidentiality or guarantee any external platform's treatment of the resulting text. + +## Implementation and verification + +- [Matching and obfuscation pipeline](../../src/scripts/tashfeer.ts). +- [Matching list](../../src/constants/banned-words.ts) and [similarity calculation](../../src/utils/string.util.ts). +- [Recognition compatibility tests](../../tests/affixes.test.ts), which seed randomness and check both public functions. +- [General behavior tests](../../tests/scripts.test.ts) and [boundary tests](../../tests/edge-cases.test.ts). + +Run `npm run test:unit` for source behavior. Run `npm test` for fresh builds and artifact checks as well. Passing these tests does not establish a general detection accuracy rate.