fuzzyset 0.1.2

The fuzzyset search process follows these steps:

1. Normalization: The input string is normalized by removing non-word characters (except spaces and commas) and converting to lowercase.
2. N-gram Generation: The string is broken into n-grams based on the gram_size_lower and gram_size_upper parameters.
3. Reverse Index Lookup: The engine performs a dictionary lookup on the n-grams to find candidate elements in the set.
4. Scoring:
   • It first attempts to match using the largest n-grams (gram_size_upper).
   • For candidates, it computes cosine similarity.
   • If use_levenshtein is True, it truncates the search space to the top 50 candidates and computes a score based on Levenshtein distance to better handle transpositions.
   • If no matches are found with the upper gram size, it repeats the process using the lower gram size (e.g., bigrams).

The FuzzySet data structure allows you to add strings to a set and retrieve them using approximate matches. When searching, it returns a list of (score, matched_value) tuples, where the score is between 0 and 1 (1 being a perfect match).

To optimize performance, you can attempt to import cFuzzySet from cfuzzyset (a Cython-implemented version) and fall back to the standard fuzzyset.FuzzySet if it is not available.

try:
    from cfuzzyset import cFuzzySet as FuzzySet
except ImportError:
    from fuzzyset import FuzzySet

a = FuzzySet()
a.add("michael axiak")
results = a.get("micael asiak")
# Output: [(0.8461538461538461, 'michael axiak')]

Install the fuzzyset package using pip to enable fuzzy string matching and approximate string searching in your Python projects.

pip install fuzzyset

When creating a FuzzySet instance, you can provide several arguments to control the indexing and scoring behavior:

• iterable: An iterable that yields strings to initialize the data structure with.
• gram_size_lower: The lower bound of gram sizes to use, inclusive. Default is 2.
• gram_size_upper: The upper bound of gram sizes to use, inclusive. Default is 3.
• use_levenshtein: A boolean indicating whether to use Levenshtein distance to determine match scoring. This helps handle transpositions. Default is True.

# Example of custom initialization
a = FuzzySet(iterable=['apple', 'banana'], gram_size_lower=2, gram_size_upper=3, use_levenshtein=True)

The FuzzySet supports standard Python container protocols:

• len(fs): Returns the number of unique items in the set.
• bool(fs): Returns True if the set contains any items, False otherwise.

fs = FuzzySet(['a', 'b'])
print(len(fs))  # 2
if fs:
    print("Set is not empty")

You can retrieve matches using square bracket notation fs[key] or the .get(key, default) method.

fs[key]

• Returns a list of tuples: [(score, original_value), ...] sorted by score in descending order.
• If an exact match is found and rel_sim_cutoff >= 1, it returns [(1, exact_match)].
• If no matches are found, it raises a KeyError.

fs.get(key, default=None)

• Returns the list of matches if found, or the default value (defaulting to None) if a KeyError occurs.

fs = FuzzySet(['apple', 'banana'])

# Using square brackets (raises KeyError if no match)
try:
    matches = fs['aple']
except KeyError:
    matches = []

# Using .get() (safer)
matches = fs.get('aple', [])

Use the add(value) method to add a new string to the FuzzySet. The value is converted to lowercase for matching. If the lowercase version of the value already exists in the set, the method returns False and does nothing. Otherwise, it returns None (implicitly True in boolean contexts) and adds the value.

fs = FuzzySet()
fs.add("New Item")