Overview of the semantica.normalize module
mainThe semantica.normalize module provides tools to standardize raw data before extraction and graph construction. This prevents data fragmentation (e.g., multiple nodes for the same company or different formats for the same date) by collapsing variants into canonical forms.
Key capabilities include:
- Text cleaning: Unicode normalization (NFC/NFKC), whitespace collapsing, and smart-quote/dash normalization.
- Entity canonicalization: Resolving aliases and disambiguating entities using configurable maps.
- Date normalization: Converting various formats (including relative dates) to ISO 8601.
- Number conversion: Parsing strings like
"$1.2B"into floats like1200000000.0. - Language and Encoding: Detecting languages and repairing inconsistent byte encodings.