python-wordsegment

If you are experiencing slow module import times due to the time taken to construct the word/count dictionaries, you can implement a faster loading strategy by separating the keys (words) and values (counts) into different files.

Standard text-based loading is relatively slow because it requires splitting strings by tabs and converting string counts to floats. A faster approach involves:

1. Storing words in a plain text file (one per line).
2. Storing counts in a binary file using Python's array module (double-precision floats).
3. Using izip (or zip in Python 3) to combine them into a dictionary.

This method can reduce loading time by approximately 60% compared to parsing a single tab-separated text file.

from itertools import izip as zip
from array import array

# Fast loading implementation
with open('words.txt', 'rb') as lines, open('counts.bin', 'rb') as counts:
    words = lines.read().split('\n')
    values = array('d')
    # Note: Replace 333333 with your actual count of items
    values.fromfile(counts, 333333)
    result_dict = dict(zip(words, values))

If you are experiencing slow import times due to the module loading large unigram/bigram text files, you can optimize the loading process by separating the keys (words) and values (counts) into two distinct files: a text file for words and a binary file for counts.

1. Convert text files to optimized formats

Convert the standard tab-separated text files into a newline-separated ASCII file for words and a binary file for double-precision floating-point counts using the array module.

2. Fast loading implementation

Use str.split for the words and array.fromfile for the binary counts to achieve significantly faster loading compared to standard line-by-line parsing.

Note: This approach changes the data format, making the files harder to inspect with standard tools like grep.

from itertools import izip as zip
from array import array

# Optimized loading pattern
with open('words.txt', 'rb') as lines, open('counts.bin', 'rb') as counts:
    words = lines.read().split('\n')
    values = array('d')
    values.fromfile(counts, 333333)  # Replace 333333 with actual count
    result_dict = dict(zip(words, values))

To use a custom corpus instead of the default one, you must manually update three components of the wordsegment module: unigram_counts, bigram_counts, and the clean function.

1. Update Unigram Counts: Replace wordsegment.unigram_counts with a collections.Counter object mapping individual words (unigrams) to their frequency in your corpus.
2. Update Bigram Counts: Replace wordsegment.bigram_counts with a collections.Counter object mapping word pairs (bigrams) to their frequency.
3. Update the clean function: By default, wordsegment.clean lowercases input and removes punctuation. If your corpus relies on specific casing or formatting, you may need to replace wordsegment.clean with a custom function (e.g., an identity function) to prevent the library from sanitizing your input.
4. Update TOTAL (Optional): If segmentation quality is poor after updating counts, update wordsegment.TOTAL to be the sum of all values in your new unigram_counts dictionary.

import wordsegment
from collections import Counter
import re

# 1. Prepare your text and tokenizer
text = "Your custom corpus text here"
def tokenize(text):
    pattern = re.compile('[a-zA-Z]+')
    return (match.group(0) for match in pattern.finditer(text))

# 2. Update unigram counts
wordsegment.unigram_counts = Counter(tokenize(text))

# 3. Update bigram counts
def pairs(iterable):
    iterator = iter(iterable)
    values = [next(iterator)]
    for value in iterator:
        values.append(value)
        yield ' '.join(values)
        del values[0]

wordsegment.bigram_counts = Counter(pairs(tokenize(text)))

# 4. Update clean function if needed (e.g., to preserve case)
wordsegment.clean = lambda value: value

# 5. Update TOTAL if necessary
wordsegment.TOTAL = float(sum(wordsegment.unigram_counts.values()))

# Use the new segmentation
print(wordsegment.segment('yourinput'))