ahocorasick_rs

repository·main·Indexed 42 days ago

https://github.com/g-research/ahocorasick_rs

A high-performance Python wrapper around the Rust aho-corasick library for fast multi-pattern substring searching in strings and bytes. Version 1.0.3 provides the AhoCorasick and BytesAhoCorasick classes, allowing users to configure search behavior via MatchKind and optimize performance using Implementation types such as NoncontiguousNFA, ContiguousNFA, and DFA.

Tokens
2.7K
Snippets
10
Records
11
Agent score
4%

What's inside ahocorasick_rs

  1. Configure MatchKind for overlapping patterns

    main

    When patterns overlap, you can control which match is returned using the matchkind argument in the constructor. This is supported by both AhoCorasick and BytesAhoCorasick.

    Available MatchKind options:

    • MatchKind.Standard (Default): Returns the pattern that matches first semantically (the one that ends earliest in the haystack).
    • MatchKind.LeftmostFirst: Returns the leftmost-in-the-haystack pattern that appears first in the provided list of patterns.
    • MatchKind.LeftmostLongest: Returns the leftmost-in-the-haystack pattern that is the longest.

    To get all overlapping matches, use MatchKind.Standard and set overlapping=True in the search method.

    from ahocorasick_rs import AhoCorasick, MatchKind
    
    # LeftmostFirst example
    ac = AhoCorasick(["disco", "disc"], matchkind=MatchKind.LeftmostFirst)
    print(ac.find_matches_as_strings("discontent")) # ['disco']
    
    # Overlapping matches example
    patterns = ["winter", "onte", "disco", "discontent"]
    ac = AhoCorasick(patterns)
    print(ac.find_matches_as_strings("discontent", overlapping=True))
    # ['disco', 'onte', 'discontent']
  2. Configure pattern storage for AhoCorasick

    main

    When using AhoCorasick.find_matches_as_strings(), you can control how strings are constructed using the store_patterns argument. This affects the memory/speed tradeoff.

    • store_patterns=None (Default): Uses a heuristic (based on total pattern length) to decide whether to store patterns.
    • store_patterns=True: Keeps references to patterns. This can speed up find_matches_as_strings() but increases memory usage.
    • store_patterns=False: Does not keep references. Saves memory but may slow down find_matches_as_strings() for small patterns/haystacks.
    import ahocorasick_rs
    # Keep references to speed up string matching
    ac = ahocorasick_rs.AhoCorasick(patterns, store_patterns=True)
  3. Optimize performance using Implementation types

    main

    You can trade construction speed, memory usage, and search performance by specifying the implementation argument in the constructor.

    Available Implementation options:

    • None (Default): Uses a heuristic to balance construction, memory, and speed.
    • Implementation.NoncontiguousNFA: Fastest to build, moderate memory, slowest search.
    • Implementation.ContiguousNFA: Slower to build than NoncontiguousNFA, excellent memory usage, slightly slower than DFA search.
    • Implementation.DFA: Very slow to build, extremely high memory usage, fastest search execution.

    If you need maximum matching speed and have a manageable number of patterns, try Implementation.DFA.

    from ahocorasick_rs import AhoCorasick, Implementation
    ac = AhoCorasick(["disco", "disc"], implementation=Implementation.DFA)
  4. Search for multiple strings using AhoCorasick

    main

    Use the AhoCorasick class to search for multiple string patterns within a haystack. You can initialize it with any iterable of strings.

    AhoCorasick.find_matches_as_indexes(haystack) returns a list of tuples: (pattern_index, start_index, end_index).

    AhoCorasick.find_matches_as_strings(haystack) returns a list of the actual matched strings.

    import ahocorasick_rs
    patterns = ["hello", "world", "fish"]
    haystack = "this is my first hello world. hello!"
    ac = ahocorasick_rs.AhoCorasick(patterns)
    
    # Returns [(0, 17, 22), (1, 23, 28), (0, 30, 35)]
    matches = ac.find_matches_as_indexes(haystack)
    
    # Returns ['hello', 'world', 'hello']
    strings = ac.find_matches_as_strings(haystack)
  5. Search bytes and buffer objects using BytesAhoCorasick

    main

    For searching bytes, bytearray, memoryview, or other objects supporting the Python buffer API, use BytesAhoCorasick.

    Warning: If searching a mutable buffer, do not mutate it in another thread while find_matches_as_indexes() is running. Patterns must not be mutated during construction.

    Note: find_matches_as_strings() is not supported by BytesAhoCorasick.

    import ahocorasick_rs
    patterns = [b"hello", b"world"]
    ac = ahocorasick_rs.BytesAhoCorasick(patterns)
    haystack = b"hello world"
    
    # Returns [(0, 0, 5), (1, 6, 11)]
    matches = ac.find_matches_as_indexes(haystack)
  6. Find matches as indexes in AhoCorasick

    main

    The find_matches_as_indexes method returns the locations of matches within the haystack. This is useful when you need the exact positions of the patterns.

    Signature: find_matches_as_indexes(haystack, overlapping=False)

    Returns: A list of tuples: (index_into_patterns, start_index_in_haystack, end_index_in_haystack).

    Important Details:

    • For string-based AhoCorasick, the indices returned are Unicode code point indices (character indices), not byte indices, making it compatible with Python string slicing.
    • If overlapping=True, the method will include overlapping results (e.g., searching for 'ana' in 'banana' would return both matches if overlapping is enabled).
    • For BytesAhoCorasick, the indices are byte indices.
    from ahocorasick_rs import AhoCorasick
    
    ac = AhoCorasick(["abc", "bcd"])
    haystack = "abcd"
    
    # Returns [(0, 0, 3), (1, 1, 4)]
    matches = ac.find_matches_as_indexes(haystack, overlapping=True)
  7. Initialize AhoCorasick for string searching

    main

    Use the AhoCorasick class to search for multiple pattern strings within a single haystack string.

    Constructor Signature: AhoCorasick(patterns, matchkind=MatchKind.Standard, store_patterns=None, implementation=None)

    Arguments:

    • patterns: An iterable of strings. Empty strings are not supported and will raise a ValueError.
    • matchkind: Determines how matches are selected. Defaults to MatchKind.Standard.
    • store_patterns: If True, the engine keeps references to the patterns, which speeds up find_matches_as_strings() at the cost of higher memory usage. If None, a heuristic is used: patterns totaling up to 4KB are stored; otherwise, they are not.
    • implementation: The underlying automaton type (Implementation.NoncontiguousNFA, Implementation.ContiguousNFA, or Implementation.DFA).

    Note: If you pass mutable buffers as patterns, do not mutate them in another thread during construction.

    from ahocorasick_rs import AhoCorasick, MatchKind, Implementation
    
    patterns = ["apple", "banana", "cherry"]
    # Using DFA implementation for faster searching
    ac = AhoCorasick(patterns, matchkind=MatchKind.Standard, implementation=Implementation.DFA)
  8. Find matches as strings in AhoCorasick

    main

    The find_matches_as_strings method returns the actual substrings that matched the patterns.

    Signature: find_matches_as_strings(haystack, overlapping=False)

    Returns: A list of strings representing the matched patterns found in the haystack.

    Arguments:

    • haystack: The string to search within.
    • overlapping: If True, includes overlapping matches. Defaults to False.
    from ahocorasick_rs import AhoCorasick
    
    ac = AhoCorasick(["apple", "pie"])
    haystack = "applepie"
    
    # Returns ["apple", "pie"]
    matches = ac.find_matches_as_strings(haystack)
  9. Initialize BytesAhoCorasick for byte searching

    main

    Use BytesAhoCorasick to search for multiple byte patterns within a bytes-like object (e.g., bytes, bytearray, or memoryview).

    Constructor Signature: BytesAhoCorasick(patterns, matchkind=MatchKind.Standard, implementation=None)

    Arguments:

    • patterns: An iterable of bytes-like objects. Empty patterns raise ValueError.
    • matchkind: Determines match selection logic.
    • implementation: The underlying automaton type (Implementation.NoncontiguousNFA, Implementation.ContiguousNFA, or Implementation.DFA).

    Important Safety Warning: If you pass in mutable buffers (like bytearray) as patterns, you MUST NOT mutate them in another thread while the object is being constructed. Once construction is complete, the patterns are no longer referenced, and the buffers can be safely mutated.

    Note: Unlike the string-based version, find_matches_as_indexes for BytesAhoCorasick returns byte indices.

    from ahocorasick_rs import BytesAhoCorasick, MatchKind
    
    patterns = [b"abc", b"def"]
    haystack = b"abcdef"
    
    bac = BytesAhoCorasick(patterns)
    matches = bac.find_matches_as_indexes(haystack)
  10. Configure MatchKind and Implementation types

    main

    The following enums are used to configure the behavior and performance characteristics of the Aho-Corasick automaton.

    MatchKind

    Controls how matches are selected when multiple patterns could match at the same position.

    • MatchKind.Standard: The default matching behavior.
    • MatchKind.LeftmostFirst: Matches the pattern that starts earliest.
    • MatchKind.LeftmostLongest: Matches the longest pattern starting at the earliest position.

    Implementation

    Controls the underlying automaton structure, allowing you to trade construction speed, memory, and search performance.

    • Implementation.NoncontiguousNFA: Non-contiguous NFA.
    • Implementation.ContiguousNFA: Contiguous NFA.
    • Implementation.DFA: Deterministic Finite Automaton (typically fastest for searching, but uses more memory and takes longer to build).