Skip to content

Ar TN/TN - #454

Open
ealbasiri wants to merge 6 commits into
NVIDIA:staging/ar_tn_v2from
ealbasiri:ar-tn-date
Open

Ar TN/TN#454
ealbasiri wants to merge 6 commits into
NVIDIA:staging/ar_tn_v2from
ealbasiri:ar-tn-date

Conversation

@ealbasiri

@ealbasiri ealbasiri commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

What does this PR do ?

Adds the Arabic (ar) grammar contributions: expanded cardinals, time, and date — each in both text normalization (TN) and inverse text normalization (ITN).

Scope (3 semiotic areas)

Cardinal (TN + ITN)

  • Expanded to millions via place-value 3-digit period decomposition.
  • Regenerated cardinal/fraction/decimal/money/measure test data to the new canonical spelling.

Time (TN + ITN)

  • HH:MM and HH:MM:SS, 24-hour, midnight handling.
  • Formal verbalization with correct feminine polarity for units and nominative tens in minutes/seconds; suffix (صباحًا/مساءً) and time-zone support.
  • ITN inverts the TN time verbalizer back to digit form.

Date (TN + ITN)

  • day–month–year, year–month–day, month–year, and bare-year forms.
  • Gregorian and Hijri month names (هـ / هجري).
  • numeric separators -, /, \, . (dotted requires full d.m.y so decimals stay decimals) and spelled-out months.
  • Ordinal day, idiomatic genitive year; standalone numbers stay cardinal.
  • ITN emits dd/mm[/yyyy] (or month name + year), Hijri suffix preserved.

Testing

  • pytest --cpu full ar suite: 399 passed, 0 failed.

Before your PR is "Ready for review"

Pre checks:

  • Have you signed your commits? (git commit -s)
  • Do all unittests finish successfully? pytest --cpu — 399 passed. (Sparrowhawk: not run.)
  • Added pytest test cases (cardinal/time/date + regenerated dependent data). (Sparrowhawk cases: not added.)
  • Added __init__.py for new data folders (months, ordinal, time).
  • Followed codeQL — no unused vars/imports in new files.
  • Correct license header on all new Python files.
  • en/graph_utils.py copy header — N/A.
  • Removed import guards — N/A.
  • Updated NeMo documentation — N/A (ar already documented).
  • Added language support to tools/text_processing_deployment/pynini_export.py — ar already registered.

PR Type:

  • New Feature

mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment thread nemo_text_processing/inverse_text_normalization/ar/taggers/date.py Outdated
mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment thread nemo_text_processing/inverse_text_normalization/ar/taggers/decimal.py
mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment thread tests/nemo_text_processing/ar/test_date.py Outdated
mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment thread nemo_text_processing/inverse_text_normalization/ar/taggers/decimal.py
)
self.final_graph_wo_negative = optional_graph_negative + self.final_graph_wo_sign

final_graph = self.add_tokens(self.final_graph_wo_negative)

@mgrafu mgrafu Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to remove coverage for negatives?

@ealbasiri ealbasiri Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why remove?

@github-actions

github-actions Bot commented Jul 24, 2026

Copy link
Copy Markdown

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

@github-actions github-actions Bot added the Stale label Jul 24, 2026
ealbasiri added 6 commits July 27, 2026 21:50
@ealbasiri
feat(ar): expand cardinal to millions, fix orthography, agreement, an…
ee6d1a8 
…d fraction homograph

Cardinal (TN/ITN):
- Expand to millions with place-value 3-digit period decomposition
- Orthography: مئة (drop silent alif), مئتين, separated compound hundreds
  (ثلاث مئة), مئتي construct-state before counted nouns
- Use genitive tens (عشرين) and اثنين by default
- Fix counted-noun (تمييز) agreement: trailing 3-10 -> plural
  (110000 -> مئة وعشرة آلاف), otherwise singular; applied to آلاف and ملايين
Fraction (TN/ITN):
- Resolve homograph collision between dual fractions and cardinals by using
  construct-state dual denominators without final nun (سبعي=2/7 vs سبعين=70);
  both directions now round-trip losslessly
- Verbalizer accepts اثنين numerator
Tests:
- Fix pre-existing NameError (RUN_AUDIO_BASED_TESTS) in ar/test_measure.py
- Regenerate cardinal/fraction/decimal/money/measure test data (TN+ITN) to the
  new canonical spelling; decimal/money/measure grammars unchanged
- Full ar suite: 306 passed

Signed-off-by: Enas Albasiri <ealbasiri@gradcenter.cuny.edu>
@ealbasiri
feat(ar): add time normalization and inverse normalization (TN + ITN)
560f73a 
TN (text_normalization/ar):
- Add time tagger + verbalizer for HH:MM, HH:MM:SS, and H <suffix> inputs
- Formal/regular style: hour as feminine ordinal (24h; midnight 0 -> الثانية عشرة),
  minutes/seconds as <number> دقيقة/دقائق
- Counted-noun agreement:
  * teens via 13_19 data (11 -> إحدى عشرة, 15 -> خمس عشرة)
  * 3-10 plural (خمس دقائق), 1 -> دقيقة واحدة, 2 -> دقيقتان
  * 21-59 ones digit feminized by polarity (خمس/ثمان/إحدى/اثنتان ...)
  * minutes/seconds number is nominative (عشرون/ثلاثون/أربعون/خمسون)
- Constrain suffix verbalization to suffix.tsv values (required for a clean inverse)
- Fix suffix.tsv tab separators
- Wire TimeFst into tokenize_and_classify.py and verbalize.py
- Add data: time/{suffix,time_zone}.tsv, ordinal/ordinals.tsv, number/{3_10mas,13_19}.tsv

ITN (inverse_text_normalization/ar):
- Add time tagger (inverts the TN time verbalizer) + verbalizer (-> HH:MM[:SS])
- Wire into tokenize_and_classify.py and verbalize.py

Tests:
- test_time.py: 30 TN (test_norm) + 19 ITN (test_denorm) hand-written cases
- Full ar suite: 355 passed

Signed-off-by: Enas Albasiri <ealbasiri@gradcenter.cuny.edu>
@ealbasiri
feat(ar): add date normalization and inverse normalization (TN + ITN)…
62f91f8 
…. tests 399 passed.

Signed-off-by: Enas Albasiri <ealbasiri@gradcenter.cuny.edu>
@ealbasiri
chore(ar): add __init__.py to time/date data folders (time, ordinal, …
40a4e05 
…months)

Signed-off-by: Enas Albasiri <ealbasiri@gradcenter.cuny.edu>
@ealbasiri
address review comments (copyright year, sparrowhawk)
2a30ec2 
Signed-off-by: Enas Albasiri <ealbasiri@gradcenter.cuny.edu>
@ealbasiri
style(ar): apply pre-commit (black/isort) formatting
1381f82 
Signed-off-by: Enas Albasiri <ealbasiri@gradcenter.cuny.edu>
@github-actions github-actions Bot removed the Stale label Jul 28, 2026
folivoramanh
folivoramanh requested changes Jul 30, 2026
View reviewed changes

@folivoramanh folivoramanh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • remove unuse import
  • make sure pytest and sparrowhawk test pass
  • update jenkins ar cache date

Comment thread tests/nemo_text_processing/ar/data_inverse_text_normalization/test_cases_time.txt

@folivoramanh folivoramanh Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FAILED tests/nemo_text_processing/ar/test_time.py::TestTime::test_denorm_07__ - AssertionError: input: الثانية عشرة
assert '0' == '12'

  • 12
  • 0

make sure pytest and sparrowhawk test pass

Comment thread tests/nemo_text_processing/ar/data_text_normalization/test_cases_measure.txt
ستة وثمانية آلاف وتسع مئة وأربعة وأربعين من عشرة آلاف في المائة~%6,8944
ستة آلاف وسبع مئة وثمانية وتسعين وتسعة من عشرة في المائة~%6798,9
ست مئة وثمانية وخمسين وثمان مئة وثمانية وثمانين من ألف في المائة~%658,888
اثنين وثمانية من عشرة في المائة~2.8%

@folivoramanh folivoramanh Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this case doensn't have %, why it can normalize and have %?

Comment thread tests/nemo_text_processing/ar/data_text_normalization/test_cases_measure.txt
@@ -2,13 +2,13 @@
تسعة فهرنهايت~f9
اثنا عشر في المائة~12%
تسعة في المائة~09%

@folivoramanh folivoramanh Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the input is only 09, how to know if it is cardinal or measure?

Comment thread tests/nemo_text_processing/ar/data_text_normalization/test_cases_money.txt

@folivoramanh folivoramanh Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

data money got more values than $ €, can you add more test for other money measure?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mgrafu mgrafu mgrafu left review comments
@folivoramanh folivoramanh folivoramanh requested changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ealbasiri @mgrafu @folivoramanh