DOCSEARCHbyAlgolia
Submit Docs

neologdn

repository·master·Indexed 43 days ago

https://github.com/ikegami-yukino/neologdn

A Japanese text normalizer designed for use with mecab-neologd. It provides the neologdn.normalize() function to standardize text by converting half-width kana to full-width, normalizing symbols, handling tildes, and shortening contiguous repeating characters.

Tokens
1.4K
Snippets
4
Records
5
Agent score
5%

What's inside neologdn

  1. Install neologdn via pip

    master

    Install the neologdn package using pip. Note that installing this module requires a C++11 compiler.

    If you encounter issues with setuptools, ensure it is installed first. If you encounter a version satisfaction error for setuptools, you can try installing wheel and using the --no-build-isolation flag.

    # Standard installation
pip install neologdn

# If setuptools is missing
pip install setuptools

# Troubleshooting installation errors
pip install wheel
pip install --no-build-isolation neologdn

  2. Use neologdn.normalize() for Japanese text normalization

    master

    The primary way to use this library is via the neologdn.normalize() function. It normalizes Japanese text based on mecab-neologd rules, including converting half-width kana to full-width, normalizing symbols, and handling various hyphen/tilde types.

    Key Features:

    • Kana Normalization: Converts half-width kana (e.g., ﾊﾝｶｸｶﾅ) to full-width (ハンカクカナ).
    • Symbol Normalization: Standardizes full-width symbols (e.g., ！？＠＃ to !?@#).
    • Repetition Shortening: Using the repeat parameter, you can shorten contiguous repeating characters (e.g., ウェーーーーイ to ウェーイ).
    • Tilde Handling: Use the tilde parameter to control how tildes are treated.
    import neologdn

# Basic normalization
neologdn.normalize("ﾊﾝｶｸｶﾅ")  # => 'ハンカクカナ'
neologdn.normalize("全角記号！？＠＃")  # => '全角記号!?@#'

# Shortening repeated characters
# repeat=1 keeps only 1 instance of the repeated character sequence
neologdn.normalize("かわいいいいいいいいい", repeat=6)  # => 'かわいいいいいい'
neologdn.normalize("無駄無駄無駄無駄ァ", repeat=1)  # => '無駄ァ'

# Tilde options
# tilde="normalize": converts to standard tilde (~)
neologdn.normalize("1995〜2001年", tilde="normalize")  # => '1995~2001年'

# tilde="normalize_zenkaku": converts to full-width tilde (〜)
neologdn.normalize("1995~2001年", tilde="normalize_zenkaku")  # => '1995〜2001年'

# tilde="ignore": does not convert the tilde
neologdn.normalize("1995〜2001年", tilde="ignore")  # => '1995〜2001年'

# tilde="remove": removes the tilde entirely
neologdn.normalize("1995〜2001年", tilde="remove")  # => '19952001年'

# Default behavior (removes tilde)
neologdn.normalize("1995〜2001年")  # => '19952001年'

  3. Configure character repetition shortening

    master
    The neologdn.normalize() function accepts a repeat argument to control the shortening of contiguous repeating substrings. This is useful for cleaning up

  4. Configure tilde normalization behavior

    master

    The neologdn.normalize() function accepts a tilde argument to control how tilde-like characters are processed. This is useful when you want to preserve full-width tildes or remove them entirely for cleaner numeric ranges.

    ValueEffect
    "normalize"Converts to standard tilde (~)
    "normalize_zenkaku"Converts to full-width tilde (〜)
    "ignore"Leaves the tilde as is
    "remove"Removes the tilde character entirely (Default)
    # Example of different tilde modes
text = "1995〜2001年"

neologdn.normalize(text, tilde="normalize")        # '1995~2001年'
neologdn.normalize(text, tilde="normalize_zenkaku")  # '1995〜2001年'
neologdn.normalize(text, tilde="ignore")            # '1995〜2001年'
neologdn.normalize(text, tilde="remove")            # '19952001年'

  5. Benchmark neologdn performance in Jupyter Notebook

    master

    To benchmark the performance of neologdn against other normalization functions, you can use the %timeit magic command in a Jupyter Notebook. The process involves:

    1. Preparing a large text dataset (e.g., downloading and extracting a corpus like ldcc-20140209).
    2. Creating a helper function to iterate through the text file and apply a normalization function to each line.
    3. Using %timeit to measure the execution time of the normalization function.
    import neologdn

# Assuming 'normalize' is a helper function that iterates over a text file
def normalize(func):
    with open('/tmp/ld.txt') as fd:
        for line in fd:
            func(line)

# Benchmark neologdn.normalize
%timeit normalize(neologdn.normalize)