WeTextProcessing Documentation

The Text Normalization (TN) pipeline consists of three logical components designed to allow exact input/output span mapping through normalize_with_mapping():

1. Classification and raw-field tagging: Identifies segments and tags them. The tagger must preserve the original written form in its fields.
2. Non-standard word (NSW) verbalization: Converts non-standard forms (like numbers, dates, or symbols) into their spoken/verbalized equivalents.
3. Output post-processing: Handles final transformations such as character-width conversion, symbol mapping, and blacklist removal.

Before modifying rules, refer to the Python rule architecture and contribution guide.

The Text Normalization (TN) and Inverse Text Normalization (ITN) pipeline is divided into two semantic stages to ensure explainability and exact input/output span mapping:

1. Tagger: Classifies the input and serializes the original field values. A tagger must not perform irreversible semantic rewrites. For example, if the input is 12, the tagger should tag it with value: "12". It should not convert it to 十二.
2. Verbalizer: Converts the raw fields serialized by the tagger into their normalized form (e.g., converting "12" to 十二).

Output cleanup tasks like character-width conversion, traditional-to-simplified conversion, and punctuation removal are also considered part of the verbalizer/post-tagging stage. This separation allows normalize_with_mapping() to maintain a valid WFST path from the raw input span to the normalized output span.

The ITN pipeline consists of three logical components:

1. Optional input canonicalization: Pre-processing steps like blacklist removal.
2. Classification and raw-field tagging: Identifying non-standard words (NSW) and tagging them while preserving the original spoken input.
3. Non-standard word verbalization and output post-processing: Converting the tagged fields into their written forms.

Note for developers: The tagger is responsible for preserving spoken input in its fields. The actual conversion to written forms is handled by the verbalizer. If you intend to modify rules, refer to the Python rule architecture and contribution guide before proceeding.

You can install the toolkit via pip to use the command-line interfaces (wetn and weitn) or the Python APIs.

pip install WeTextProcessing

You can perform Inverse Text Normalization (converting spoken text to written forms) using either the Python module directly or the installed weitn command. You must specify the --language and the --text to be processed.

$ python -m itn --language zh --text "二点五平方电线"
# Or, after installation:
$ weitn --language zh --text "二点五平方电线"

You can perform Chinese Text Normalization (TN) using either the tn module via Python or the installed wetn command. Both require specifying the --language (e.g., zh) and the --text to be normalized.

Using Python module

python -m tn --language zh --text "2.5平方电线"

Using installed CLI

wetn --language zh --text "2.5平方电线"

To build the Android application, you must manually generate and place four FST (Finite State Transducer) models into the runtime/android/app/src/main/assets directory. These models are required for Text Normalization (TN) and Inverse Text Normalization (ITN) at runtime.

Prerequisites:

• pynini must be installed: pip install pynini importlib_resources.

Generation Steps: From the repository root, run the following commands to generate the models directly into the assets folder:

assets=runtime/android/app/src/main/assets

# TN: produces zh_tn_tagger.fst and zh_tn_verbalizer.fst
python -m tn --language zh --overwrite_cache --cache_dir "$assets"

# ITN: produces zh_itn_tagger.fst and zh_itn_verbalizer.fst
python -m itn --language zh --overwrite_cache --cache_dir "$assets"

To ensure consistency across the pipeline, every top-level language/direction pipeline must own a single ordered RuleSpec inventory. This inventory is used to build both the tagger and verbalizer unions, preventing rules from being added to only one side.

• RuleSpec(rule): Creates a rule that is used for both tagging and verbalization.
• RuleSpec(rule, weight, verbalize=False): Creates a rule that is used by the classifier but has no top-level verbalizer.

Example of registering rules:

rules = (
    RuleSpec(date, 1.02),
    RuleSpec(cardinal, 1.06),
    RuleSpec(char, 100),
)
tagger = self.tagger_union(rules)
verbalizer = self.verbalizer_union(rules)

Note: The token field-order schema is owned by the pipeline and should be passed through Processor(..., token_orders=TOKEN_ORDERS).

To use the WeTextProcessing runtime, you must first build the project using CMake. The build process differs depending on your operating system.

# Linux/macOS build
$ cmake -B build -DCMAKE_BUILD_TYPE=Release
$ cmake --build build

# Windows build (Visual Studio 17 2022)
$ cmake -DCMAKE_BUILD_TYPE=Release -B build -G "Visual Studio 17 2022" -DBUILD_SHARED_LIBS=0 -DCMAKE_CXX_FLAGS="/ZI"
$ cmake --build build