pycld3

⚠️ Warning: This project is archived as of 2026. It will receive no further releases, bug fixes, or security updates.

Key Limitations:

• Python Version: The last released version (0.22) does not build on Python 3.10 or newer without patches (due to changes in longintrepr.h in CPython 3.11).
• Protobuf Dependency: The library links against libprotobuf. Upgrading protobuf on your system can break import cld3 due to ABI instability.
• Windows: From-source installs on Windows are known to fail due to protobuf include path issues.

Recommended Alternatives:

• gcld3: Google's official Python bindings using pybind11 (closest drop-in replacement).
• lingua-language-detector: Pure-Python, highly accurate for short text.
• fasttext: High-performance, widely used in production.
• pycld2: Bindings to the older CLD2 (no protobuf dependency).

CLD3 is a neural network model for language identification. It works by extracting character ngrams from input text and computing their frequency (fraction of appearance).

Key steps in the inference process:

1. Ngram Extraction: Character ngrams are extracted (e.g., for "banana", the trigram "ana" appears with a fraction of 2/4).
2. Hashing & Embedding: Ngrams are hashed to IDs, which are then represented by dense embedding vectors.
3. Embedding Layer: The model averages these embeddings based on their frequency and concatenates them.
4. Neural Network Pass: The averaged embeddings pass through a hidden (Rectified Linear) layer and a softmax layer to produce a language prediction via a forward pass.

To use pycld3, import the cld3 module and use get_language() to identify the language of a single string, or get_frequent_languages() to find multiple languages in a text. The library returns LanguagePrediction objects containing the BCP-47 language code, the probability, reliability, and the proportion of the text that language occupies.

import cld3

# Detect a single language
prediction = cld3.get_language("This is a test")
print(prediction)
# LanguagePrediction(language='en', probability=0.9999980926513672, is_reliable=True, proportion=1.0)

# Detect multiple frequent languages
for lang in cld3.get_frequent_languages("This piece of text is in English. Този текст е на Български.", num_langs=3):
    print(lang)

CLD3 does not automatically clean input text (e.g., removing URLs or email addresses), which can lead to incorrect language predictions. To improve accuracy, preprocess your text to remove noise before passing it to cld3.get_language().

Note that language detection algorithms generally perform poorly with very short inputs; avoid relying on detection for extremely brief strings.

import re
import cld3

# Example: Removing URLs to improve detection
s = "Je veux que: https://site.english.com/this/is/a/url/path/component#fragment"

# cld3 may incorrectly detect the language due to the URL
print(cld3.get_language(s))

# Preprocess: remove URLs using regex
url_re = r"\b(?:https?://|www\.)[a-z0-9-]+(\\.[a-z0-9-]+)+(?:[/?].*)?"
new_s = re.sub(url_re, "", s)

# Detection is now more accurate
print(cld3.get_language(new_s))

On MacOS, use Homebrew to install the protobuf dependency before installing pycld3 via pip.

brew update
brew upgrade protobuf || brew install -v protobuf
python -m pip install -U pycld3

To install from source on Debian or Ubuntu, you must first install the C++ compiler and Protobuf development headers/compiler.

sudo apt-get update -y
sudo apt-get install -y --no-install-recommends \
    g++ \
    protobuf-compiler \
    libprotobuf-dev
python -m pip install -U pycld3

If you are using CPython 3.6 through 3.9 on MacOS or Linux (including manylinux1), you can install pycld3 using pre-built wheels. These wheels include the necessary protobuf dependencies, so you do not need to install protoc or libprotobuf manually.

python -m pip install -U pycld3

CLD3 relies on Chromium code. To build and run the language_identifier_main executable, follow these steps:

1. Check out the Chromium repository.
2. Copy the CLD3 code to //third_party/cld_3 within the Chromium tree.
3. In src/BUILD.gn, uncomment the language_identifier_main executable.
4. Use gn and ninja to build and execute the model.

gn gen out/Default
ninja -C out/Default third_party/cld_3/src/src:language_identifier_main
out/Default/language_identifier_main

On CentOS 8, you must install from source as root/UID 0. This involves installing build tools and manually compiling Protobuf (version 3.11.4 is used in this example) before installing pycld3.

sudo su -
set -ex
pushd /opt
PROTOBUF_VERSION='3.11.4'
yum update -y
yum install -y autoconf automake gcc-c++ glibc-headers gzip libtool make python3-devel zlib-devel
curl -Lo /opt/protobuf.tar.gz \
    "https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOBUF_VERSION}/protobuf-cpp-${PROTOBUF_VERSION}.tar.gz"
tar -xzvf protobuf.tar.gz
rm -f protobuf.tar.gz
pushd "protobuf-${PROTOBUF_VERSION}"
./configure --with-zlib --disable-debug && make && make install && ldconfig --verbose
popd && rm -rf "protobuf-${PROTOBUF_VERSION}" && popd && set +ex

python -m pip install -U pycld3

Alpine Linux does not support PyPI wheels. You must install the required build dependencies before installing pycld3 from source.

apk --update add g++ protobuf protobuf-dev
python -m pip install -U pycld3

If you encounter an ImportError stating that a protobuf shared object (e.g., libprotobuf.so.22) cannot be opened, Python is likely unable to locate the library in your system path.

You can resolve this by adding the directory containing libprotoc.so to your LD_LIBRARY_PATH environment variable.

# 1. Find the location of libprotoc.so
$ find /usr -name 'libprotoc.so' \( -type l -o -type f \)

# 2. Add that directory to LD_LIBRARY_PATH
export LD_LIBRARY_PATH="$(dirname $(find /usr -name 'libprotoc.so' \( -type l -o -type f \))):$LD_LIBRARY_PATH"

# 3. Verify the fix
$ python -c 'import cld3; print(cld3.get_language("影響包含對氣候的變化以及自然資源的枯竭程度"))'