Unicode.Normalize

Unicode normalization.

There are multiple ways to normalize a string, referred to as Normalization Forms. Which one you want depends on what you want to do with the string, but the most common is NFC, which preserves the exact visual appearance of the string. If you don't care about the details, just use normalize NFC.

import Unicode.Normalize exposing (Form(..), normalize)


normalize NFC "é"
--> "é"


normalize NFC (String.fromList [ 'ᄉ', 'ᅥ', 'ᆼ' ])
--> "성"

Normalization Forms

type Form

= NFC

| NFD

| NFKC

| NFKD

The different ways to normalize a string.

NFC: Canonical decomposition followed by canonical composition.
NFD: Canonical decomposition without recomposition.
NFKC: Compatibility decomposition followed by canonical composition.
NFKD: Compatibility decomposition without recomposition.

Canonical decomposition splits single characters into sequences of equivalent combining characters. Compatibility decomposition extends canonical decomposition by also replacing variants of the same logical character (such as replacing ¼ with 1/4).

Canonical composition undoes the process of canonical decomposition, combining sequences into single characters wherever possible.

For example, the character "ñ" can be represented as a single code point (U+00F1) or as two code points ("n" followed by a combining tilde, U+006E U+0303). The normalization forms handle this differently:

-- NFC composes into a single character
normalize NFC "ñ"  -- (n + combining tilde)
--> "ñ"            -- (single ñ)


-- NFD decomposes into base + combining character
normalize NFD "ñ"  -- (single ñ)
--> "ñ"            -- (n + combining tilde)

The compatibility forms also replace visual variants:

-- NFKC replaces ﬁ ligature with "fi" and composes
normalize NFKC "ﬁñ"
--> "fiñ"


-- NFKD replaces ﬁ ligature with "fi" and decomposes
normalize NFKD "ﬁñ"
--> "fiñ"

Normalize a String

normalize : Form -> String -> String

Normalize a string to a particular Normalization Form.

-- Two equivalent representations of "é" become identical after NFC
normalize NFC "é" == normalize NFC "é"
--> True


-- ASCII strings are unaffected
normalize NFC "hello"
--> "hello"


-- Compatibility normalization replaces special forms
normalize NFKC "①②③"
--> "123"


normalize NFKC "ﬃ"
--> "ffi"

Modules

Unicode.Normalize

algebraicsystems/elm-unicode-normalize/1.0.2

Unicode.Normalize

Normalization Forms

Normalize a String

Modules