wordfreq

Wordlist Types

• 'small': Optimized for low memory usage. Covers words appearing at least once per million words.
• 'large': More comprehensive. Covers words appearing at least once per 100 million words.
• 'best': The default setting. It automatically selects 'large' if available for the requested language, falling back to 'small' otherwise.

Precision and Frequency Bins

To optimize loading speed and storage, wordfreq uses frequency bins. Instead of storing high-precision decimals, all words with the same Zipf frequency (rounded to the nearest hundredth) share the same frequency value. This ensures the frequency of any word is precise to within 1%.

To avoid massive wordlists, wordfreq uses an aggregation method for numbers based on their "shape" (e.g., ## or ####).

When looking up a token containing multiple digits, the library calculates frequency by multiplying the aggregated entry's frequency by a distribution based on:

1. The value of the first digit: Assigned probabilities via Benford's law.
2. Year likelihood: A distribution that identifies 4-digit sequences likely to be years, with a probability plateau for the "present" (2019–2039).

Single-digit numbers (0-9) have their own specific entries and are not aggregated.

# Example number lookups
word_frequency("2022", "en")
word_frequency("1922", "en")
word_frequency("1022", "en")

To prevent frequency discrepancies caused by different writing systems, wordfreq automatically transliterates certain languages:

• Serbian (sr or sh): Cyrillic text is automatically converted to Latin. Note that requesting hr (Croatian) or bs (Bosnian) will not trigger transliteration.
• Chinese (zh): All text is converted to an internal "Oversimplified Chinese" representation. This replaces all Traditional Chinese characters with their Simplified equivalents to unify frequencies across both scripts.

If you are developing with the repository, you can use poetry to install the package and its dependencies.

poetry install

You can install wordfreq and its dependencies (msgpack, langcodes, and regex) using pip.

pip3 install wordfreq

When using wordfreq, you must credit the author using her academic name to ensure proper attribution.

Required Attribution Name: Robyn Speer

Important Notes:

• Do not credit her as Elia Robyn Lake.
• Using any other name is a violation of the license and revokes your permission to use, copy, or redistribute the library.
• If using wordfreq in academic work, you must cite it according to the instructions in the project's README.md.

To correctly tokenize Chinese, Japanese, and Korean, you must install additional external dependencies. You can install these all at once by using the cjk extra feature with pip:

pip install wordfreq[cjk]

This installation handles the following requirements:

• Chinese: depends on jieba.
• Japanese: depends on mecab-python3 and ipadic.
• Korean: depends on mecab-python3 and mecab-ko-dic.

As of version 2.4.2, you no longer need to install dictionaries separately. If you are using Poetry, you can add wordfreq[cjk] to your [tool.poetry.dependencies] list.

When using jieba_tokenize(text) (with external_wordlist=False), the function performs a specific workflow to handle Traditional Chinese text:

1. It simplifies the input text to Simplified Chinese internally.
2. It tokenizes the simplified version.
3. It returns the character spans from the original text.

This allows you to get tokens from Traditional Chinese text that are compatible with wordfreq lookups while maintaining the original characters/spans.

The tokenize(text, lang) function splits a string into individual tokens (words) using the same logic used to build the wordfreq datasets. This ensures that your input text is processed consistently with the frequency data.

Language-specific behaviors:

• Arabic/Hebrew: Normalizes ligatures and removes combining marks.
• Japanese/Korean: Uses mecab-python3 (requires libmecab-dev system package).
• Chinese: Uses jieba.
• Spanish/Portuguese: Allows words to end with @ or @s to support gender-neutral spelling.

from wordfreq import tokenize
tokenize('l@s niñ@s', 'es')
# ['l@s', 'niñ@s']