DOCSEARCHbyAlgolia
Submit Docs

libpostal

repository·master·Indexed 26 days ago

https://github.com/openvenues/libpostal

A C library for parsing and normalizing international street addresses using statistical NLP. It converts free-form, human-readable addresses into normalized forms for machine comparison and full-text indexing, serving as a preprocessing step for geocoding. It includes support for 60+ languages and provides official bindings for Python, Ruby, Go, Java, PHP, and NodeJS.

Tokens
5.7K
Snippets
14
Records
38
Agent score
89%

What's inside libpostal

  1. Overview of libpostal

    master
    libpostal is a C library designed for parsing and normalizing international street addresses using statistical Natural Language Processing (NLP) and open data. It converts free-form, human-readable addresses into clean, normalized forms suitable for machine comparison and full-text indexing. While not a full geocoder, it serves as an effective preprocessing step for geocoding applications to improve consistency across different languages and local conventions.

  2. Core Features of libpostal

    master

    libpostal provides several NLP capabilities for international address processing:

    • Abbreviation expansion: Expands abbreviations (e.g., "rd" to "road") across > 50 languages, including support for ideographic and Germanic languages.
    • International address parsing: Uses Conditional Random Fields (CRF) to parse address strings into components like house_number, road, city, and state. It is trained on over 1 billion addresses and supports nearly every inhabited country.
    • Language classification: Uses multinomial logistic regression to identify the language of an address, including support for regional languages (e.g., Catalan in Spain).
    • Numeric expression parsing: Converts written numbers (e.g., "twenty first" or "quatre-vingt-douze") into digits (e.g., "21st" or "92"). Supports > 30 languages and Roman numeral normalization.
    • Fast tokenization/lexing: High-speed UTF8 word segmentation (> 1M tokens/sec) following the TR-29 spec.
    • UTF8 normalization: Supports NFD decomposition, accent stripping, and Latin-ASCII transliteration.
    • Transliteration: Converts text between scripts (e.g., Cyrillic to Latin) using CLDR transforms.
    • Script detection: Identifies which Unicode scripts (e.g., Han, Latin) are present in a string.

  3. Install libpostal on Mac/Linux

    master

    Prerequisites

    Ubuntu/Debian:

    sudo apt-get install -y curl build-essential autoconf automake libtool pkg-config

    CentOS/RHEL:

    sudo yum install curl autoconf automake libtool pkgconfig

    macOS (Package Managers): Using MacPorts:

    port install libpostal

    Using Homebrew:

    brew install libpostal
    sudo apt-get install -y curl build-essential autoconf automake libtool pkg-config

  4. Improve the address parser accuracy

    master

    If the address parser is not performing well for specific countries, languages, or address styles, you can contribute to its improvement through the following methods:

    1. Contribute to OpenStreetMap (OSM): Adding data with an addr:housenumber tag to OSM will allow that data to be automatically incorporated into the parser during the next training cycle.
    2. Update address formats: For certain issues, updating the formats at https://github.com/OpenCageData/address-formatting may help.
    3. Report patterns: If you identify a consistent pattern of incorrect parses, report it as a GitHub issue to help refine the training data creation process.

  5. Compile libpostal from source (Mac/Linux)

    master

    To compile the C library from source, follow these steps. Note that the --datadir flag requires a directory with several GB of space. The build script automatically appends libpostal to the provided path.

    General Build Process

    git clone https://github.com/openvenues/libpostal
cd libpostal
make distclean
./bootstrap.sh
./configure --datadir=[path_with_space]
make -j4
sudo make install
sudo ldconfig

    Architecture Specific Configurations

    Intel/AMD processors (Default model):

    ./configure --datadir=[path_with_space]

    Apple Silicon / ARM CPUs (Default model): Use --disable-sse2 to ensure the build succeeds, though performance may be lower.

    ./configure --datadir=[path_with_space] --disable-sse2

    Improved Senzing Model: To use the Senzing model (optimized for US, UK, and Singapore addresses), set MODEL=senzing.

    ./configure --datadir=[path_with_space] MODEL=senzing
make -j8
sudo make install
    ./configure --datadir=[...some dir with a few GB of space...] MODEL=senzing

  6. Download libpostal data files

    master

    libpostal requires data files (expansion data, trained parser models, and language classifier models) which are hosted on S3. While make downloads these automatically during build, you can manually trigger a download or update using the libpostal_data command.

    Replace $YOUR_DATA_DIR with the directory path used during your installation configuration:

    libpostal_data download all $YOUR_DATA_DIR/libpostal

  7. Address Deduping Workflow with libpostal

    master

    libpostal is designed to facilitate fuzzy address matching and deduplication by creating normalized geographic strings. Instead of using complex synonym files in search engines, follow this pattern:

    1. Run existing database addresses through libpostal's expand_address function.
    2. Store the resulting normalized strings in your database, search engine, or hash table.
    3. Run new user queries or fresh imports through libpostal and perform searches against the stored normalized strings.

    This approach allows for fuzzy matching in constant time relative to the dataset size.

    1. Run the addresses in your database through libpostal's expand_address
2. Store the normalized string(s) in your favorite search engine, DB, hashtable, etc.
3. Run your user queries or fresh imports through libpostal and search the existing database using those strings

  9. Normalize addresses with expand_address

    master

    The expand_address API converts messy real-world addresses into normalized equivalents suitable for search indexing or hashing. It uses an OSM-trained language classifier to detect the language and apply appropriate normalizations across 60+ languages.

    Python Usage

    from postal.expand import expand_address
expansions = expand_address('Quatre-vingt-douze Ave des Champs-Élysées')

assert '92 avenue des champs-elysees' in set(expansions)

    C Usage

    Note: You must call libpostal_setup() and libpostal_setup_language_classifier() at the start of your program, and libpostal_teardown() and libpostal_teardown_language_classifier() at the end.

  10. Install libpostal on Windows (MSys2/MinGW)

    master

    Windows installation requires MSys2 and MinGW.

    1. Update MSys2:
    pacman -Syu
    1. Install prerequisites:
    pacman -S autoconf automake curl git make libtool gcc mingw-w64-x86_64-gcc
    1. Build the library:
    git clone https://github.com/openvenues/libpostal
cd libpostal
cp -rf windows/* ./
./bootstrap.sh
./configure --datadir=/c
make -j4
make install

    Notes:

    • When setting --datadir, use /c for the C: drive. The script will result in C:\libpostal\.
    • The compiled DLL is located at src/.libs/libpostal-1.dll.
    • To generate a .lib import library for Visual Studio, use lib.exe with the libpostal.def file.
    cp -rf windows/* ./
./bootstrap.sh
./configure --datadir=/c
make -j4
make install

  12. Add new languages or synonyms to libpostal dictionaries

    master

    You can extend libpostal's address parsing and normalization capabilities by adding new languages or modifying existing synonym files. Dictionaries are organized by ISO 639-1 language codes. To add a new language, simply create a new directory named after the language code within the resources/dictionaries folder; no additional configuration is required.

    Dictionary files use a format similar to Lucene/Elasticsearch synonyms. The leftmost string is the canonical/normalized version, and synonyms are appended to the right, delimited by a pipe (|) character.

    drive|dr
street|st|str
road|rd