icu.Normalizer2

icu.Normalizer2

The icu.Normalizer2 API provides normalisation support. The primary functions are to produce a normalised string and to check if a string is normalised.

The Python unicodedata module, and its drop in replacements, offer a unicodedata.normalize method which can normalise a string to NFC, NFKC, NFD or NFKD. Likewise, the icu.Normalizer2 API supports the four common normalisation forms documented in Unicode Standard Annex #15 and NFKC_Casefold:

• getNFCInstance: create a icu.Normalizer2 instance for Unicode NFC normalisation.
• getNFDInstance: create a icu.Normalizer2 instance for Unicode NFD normalisation.
• getNFKCCasefoldInstance: create a icu.Normalizer2 instance for Unicode NFKC_CF normalisation.
• getNFKCInstance: create a icu.Normalizer2 instance for Unicode NFKC normalisation.
• getNFKDInstance: create a icu.Normalizer2 instance for Unicode NFKCD normalisation.

The icu.Normalizer2 API provides a more generic, and powerful, getInstance which can be used to create a normalizer instance using NFC, NFKC, NFD, NFKD, NFKC_CF, but can be customised to an applications needs.

icu.Normalizer2.getInstance(packageName, name, mode)

Normalization Form	packageName	name	mode
NFC	None	nfc	UNormalizationMode2.COMPOSE
NFKC	None	nfkc	UNormalizationMode2.COMPOSE
NFD	None	nfc	UNormalizationMode2.DECOMPOSE
NFKD	None	nfkc	UNormalizationMode2.DECOMPOSE
NFKC_CF	None	nfkc_cf	UNormalizationMode2.COMPOSE

packageName:

The normal usage scenario for the Normalizer2 API, is to use the standard Unicode Normalisation data. In the instance, packageName is given the value None.

It is also possible to set packageName to a binary data file containing custom application specific data file. For instance when normalising data based on IDNA mappings.

name:

For normalisation using canonical equivalence (NFC or NFD), use the value nfc.

For normalisation using the looser compatibility equivalence (NFKC or NFKD) use the value nfkd.

For NFKC_Casefold, use the value nfkc_cf.

Mode:

There are four possible modes available:

1. COMPOSE: Decomposition followed by composition. NFC or NFKC (depending on value of name, see table above).
2. DECOMPOSE: Map, and reorder canonically. NFD or NFKD (depending on value of name, see table above).
3. FCD: "Fast C or D" form.
4. COMPOSE_CONTIGUOUS: Compose only contiguously. Also known as Fast C Contiguous (FCC), useful for internal processing.

See also:

• Canonical Equivalence in Applications (UTN #5)
• ICU documentation: Normalization
• ICU4C API
• PyICU Normalizer2 test
• Unicode Normalization Forms (USA #15)

Functions

• append: Appends the second string to the first string (merging them at the boundary) and returns the first string. Result is normalised if both strings were normalised.
• getDynamicClassID: returns a UClassID for the actual class.
• hasBoundaryAfter: Tests if the character always has a normalisation boundary after it, regardless of context. Used for iterative normalisation.
• hasBoundaryBefore: Tests if the character always has a normalisation boundary before it, regardless of context. Used for iterative normalisation.
• isInert: Tests if the character is normalisation-inert. Used for iterative normalisation.
• isNormalized: Tests if the string is normalised.
• normalize(str): Returns the normalised form of the source string
• normalizeSecondAndAppend: Appends the normalised form of the second string to the first string (merging them at the boundary) and returns the first string.
• quickCheck: Alternative method to Test if the string is normalised
• spanQuickCheckYes: Returns the end of the normalised substring of the input string. Used with normalizeSecondAndAppend().

Filtering

ICU4C provides the FilteredNormalizer2 class which is used when you want to apply normalisation to specific sets of Unicode characters. It uses a Normalizer2 instance with a UnicodeSet to limit which characters normalisation is applied to.

Refer to the notes on FilteredNormalizer2 and UnicodeSet.

Usage

The function normalize will be the common way to normalise a string. Other functions are used ti implement either iterative normalisation or a quick normalisation step.

Using icu.Normalizer2.getInstance:

from icu import Normalizer2, UNormalizationMode2
nfc_normalizer = Normalizer2.getInstance(None, "nfc", UNormalizationMode2.COMPOSE)
s = "officé"
print(' '.join(f"U+{ord(char):04X}" for char in s))
# U+006F U+FB03 U+0063 U+00E9
s_nfc = nfc_normalizer.normalize(s)
print(' '.join(f"U+{ord(char):04X}" for char in s_nfc))
# U+006F U+FB03 U+0063 U+00E9
nfkc_normalizer = Normalizer2.getInstance(None, "nfkc", UNormalizationMode2.COMPOSE)
s_nfkc = nfkc_normalizer.normalize(s)
print(' '.join(f"U+{ord(char):04X}" for char in s_nfkc))
#U+006F U+0066 U+0066 U+0069 U+0063 U+00E9
s_alt = "Dɛ̈tëicëkäŋ akɔ̈ɔ̈n"
print(' '.join(f"U+{ord(char):04X}" for char in s_alt))
# U+0044 U+025B U+0308 U+0074 U+00EB U+0069 U+0063 U+00EB U+006B U+00E4 U+014B U+0020 U+0061 U+006B U+0254 U+0308 U+0254 U+0308 U+006E
nfd_normalizer = Normalizer2.getInstance(None, "nfc", UNormalizationMode2.DECOMPOSE)
s_alt_nfd = nfd_normalizer.normalize(s_alt)
print(' '.join(f"U+{ord(char):04X}" for char in s_alt_nfd))
# U+0044 U+025B U+0308 U+0074 U+0065 U+0308 U+0069 U+0063 U+0065 U+0308 U+006B U+0061 U+0308 U+014B U+0020 U+0061 U+006B U+0254 U+0308 U+0254 U+0308 U+006E

Using the specific purpose initiators:

din_s = "käthɛɛr ke Paguot Thudän"
print(' '.join(f"U+{ord(char):04X}" for char in din_s))
# U+006B U+00E4 U+0074 U+0068 U+025B U+025B U+0072 U+0020 U+006B U+0065 U+0020 U+0050 U+0061 U+0067 U+0075 U+006F U+0074 U+0020 U+0054 U+0068 U+0075 U+0064 U+00E4 U+006E
normalizer = Normalizer2.getNFDInstance()
din_s_norm = normalizer.normalize(din_s)
print(' '.join(f"U+{ord(char):04X}" for char in din_s_norm))
# U+006B U+0061 U+0308 U+0074 U+0068 U+025B U+025B U+0072 U+0020 U+006B U+0065 U+0020 U+0050 U+0061 U+0067 U+0075 U+006F U+0074 U+0020 U+0054 U+0068 U+0075 U+0064 U+0061 U+0308 U+006E

The unicodedata function isnormalized takes a normalisation form and a string as parameters. The PyICU equivalent, instead is applied to the normalisation instance being used, tests a string against that instance.

Testing the two strings above din_s and din_s_norm, we use the Normalizer2 instance we named normalizer:

print(normalizer.isNormalized(din_s))
# False
print(normalizer.isNormalized(din_s_norm))
# True