frizbee

repository·main·Indexed 20 days ago

https://github.com/saghen/frizbee

A high-performance, SIMD-accelerated, typo-resistant fuzzy string matcher written in Rust. It utilizes the Smith-Waterman algorithm with affine gaps for accurate and fast matching, particularly for Unicode text. Frizbee supports multi-pattern queries with special syntax for prefix, suffix, substring, and negated matching, as well as parallel execution and a specialized radix sort for match results.

Tokens
9.3K
Snippets
27
Records
41
Agent score
67%

What's inside frizbee

  1. Benchmark Results Overview

    main

    Frizbee's performance is evaluated across several datasets and scenarios, comparing it against Nucleo, FZF, and Iter. The benchmarks cover various character encodings (UTF-8 throughput for 2-byte and 3-byte characters), match scenarios (Partial Match, All Match, No Match, etc.), and sorting performance.

    Key performance characteristics observed:

    • High Throughput: Significant speedups in UTF-8 processing for Arabic (2-bytes) and Korean (3-bytes) datasets.
    • Scenario Efficiency: High performance in 'No Match' and 'Partial Match' scenarios due to effective prefiltering.
    • Parallelism: Significant speedups when using Parallel (x8) execution compared to Sequential execution in many datasets.
  2. How prefiltering works in Frizbee

    main

    Frizbee uses a SIMD-accelerated prefiltering step to quickly discard haystacks that cannot match the needle.

    Unlike other implementations (like Nucleo) that perform multiple sequential loads for each needle character, Frizbee loads the haystack chunk-by-chunk and uses bitmasks to skip already-matched prefixes. This significantly reduces the number of memory loads. Frizbee also identifies and skips prefixes/suffixes of the haystack that contain characters not present in the needle.

  3. How the Smith-Waterman algorithm is implemented

    main

    Frizbee implements the Smith-Waterman algorithm with affine gaps and row-wise parallelism via SIMD.

    Key characteristics:

    • Local Alignment: It finds the best local sequence alignment.
    • Complexity: While the theoretical complexity is $O(nm)$, Frizbee optimizes this for small matrices (typically $m < 128$) using SIMD.
    • Scoring Optimization: When the maximum score is $< 256$ (relative to needle length), Frizbee uses u8 scoring instead of the default u16, effectively doubling the SIMD width.
    • Data Dependency: It uses a sequential layout where horizontal data dependencies are applied immediately to support affine gaps.
  4. Understand Frizbee benchmark configurations

    main

    When analyzing or running benchmarks, Frizbee uses several distinct configuration modes that affect performance and matching behavior:

    • Frizbee (Default): Uses Config::default(). This performs the fastest prefiltering because no typos are allowed.
    • Iter: Uses the match_iter API to lazily match haystacks one at a time, similar to the default configuration.
    • All Scores: Configured with max_typos: None. This retrieves scores for all items without any filtering.
    • 1/2 Typos: Configured with max_typos: Some(1) or max_typos: Some(2). This uses a slower but effective prefilter that allows a small number of typos.
    • 3 Typos: Configured with max_typos: Some(3). This skips prefiltering entirely because, in non-synthetic data, the prefilter would require up to 4 passes over the data without providing significant filtering benefits.
  5. Configure Unicode matching behavior

    main

    Frizbee matches against UTF-8 bytes directly for performance. By default, Config::default() uses UnicodeMatching::Smart, which only uses the slower Unicode path when the needle contains non-ASCII characters.

    Important Considerations:

    • Normalization: Frizbee does not perform unicode normalization. You must apply normalization yourself if required.
    • ASCII vs Multi-byte: With UnicodeMatching::Smart, an ASCII needle matching against a haystack with multi-byte UTF-8 codepoints (like emojis) will receive a lower score because the gap penalty is applied to the number of bytes, not the number of characters.
    • Forcing Unicode: If you need to ensure the Unicode path is always taken (even for ASCII needles), use UnicodeMatching::Always.
    • Case Insensitivity: For case-insensitive matching, the case-flipped version is skipped if it results in a different byte length or multiple codepoints (e.g., German ß to SS).
  6. Run Frizbee benchmarks locally

    main

    You can test the performance of Frizbee by running the included benchmarks. The benchmarks compare Frizbee's performance against different configurations and other tools like Nucleo and FZF. Note that the benchmark results provided in the documentation were generated on a Ryzen 9950x3D running NixOS 26.11.0.

    # Use cargo to run the included benchmarks
    cargo bench
  7. Configure fuzzy matching typos with max_typos

    main
    Frizbee uses a prefiltering step by default to remove haystacks that do not contain all characters in the needle. While this is highly efficient, you can allow for typos by controlling the maximum number of allowed typos using the max_typos property. This allows the algorithm to consider haystacks that might be missing some characters from the needle.
  8. Use the fuzzy_match iterator API

    main

    For a simpler, albeit slightly slower, interface, you can use the FuzzyMatchExt trait to call .fuzzy_match() directly on an iterator of strings. To sort the resulting matches, use the radix_sort_matches function.

    use frizbee::{iter::FuzzyMatchExt, Config, radix_sort_matches};
    
    let haystacks = ["fooBar", "foo_bar", "prelude", "println!"];
    let mut matches: Vec<_> = haystacks
        .iter()
        .fuzzy_match("fBr", &Config::default())
        .collect();
    
    radix_sort_matches(&mut matches);
  9. Use multi-pattern queries with Matcher::from_query

    main

    Frizbee supports multi-pattern matching using whitespace-separated queries. You can control the matching mode for each pattern using specific prefixes:

    PrefixModeDescription
    (none)fuzzyStandard fuzzy matching
    'substringSubstring matching
    ^prefixPrefix matching
    $suffixSuffix matching
    ^...$exactExact matching
    !negatedNegates the following mode (can be combined)

    Example query: "foo !^bar" matches strings containing foo but not starting with bar.

    use frizbee::{Config, Matcher};
    
    let needle = "foo !^bar";
    let haystacks = ["fooBar", "foo_bar", "barfoo", "prelude", "println!"];
    
    let mut matcher = Matcher::from_query(needle, &Config::default());
    let matches = matcher.match_list(&haystacks);
  10. Configure matching parameters per pattern

    main

    If you need different matching behaviors for different needles (for example, scaling allowed typos based on the length of the needle), use Pattern::parse_query to generate a collection of patterns, modify them, and then initialize a Matcher using Matcher::from_patterns.

    use frizbee::{Config, Matcher, Pattern};
    
    let haystacks = ["fooBar", "foo_bar", "barfoo", "prelude", "println!"];
    
    let patterns = Pattern::parse_query("foo !^bar")
        .into_iter()
        .map(|pattern| {
            let max_typos = (pattern.needle.len() / 4) as u16;
            pattern.max_typos(Some(max_typos))
        })
        .collect::<Vec<_>>();
    
    let mut matcher = Matcher::from_patterns(&patterns, &Config::default());
    let matches = matcher.match_list(&haystacks);
  11. Perform basic fuzzy string matching with Matcher

    main

    To perform fuzzy matching, create a Matcher instance using a needle (the search string) and a Config. You can then match against a list of strings using match_list or use match_list_parallel to distribute the workload across multiple threads.

    Note: Frizbee will panic if the needle length exceeds config.scoring.max_needle_len(). The default limit is 10,922 characters.

    use frizbee::{Config, Matcher, Pattern};
    
    let needle = "fBr";
    let haystacks = ["fooBar", "foo_bar", "barfoo", "prelude", "println!"];
    
    let mut matcher = Matcher::new(needle, &Config::default());
    let matches = matcher.match_list(&haystacks);
    // or in parallel (8 threads)
    let matches = matcher.match_list_parallel(&haystacks, 8);
  12. Configure Frizbee matching behavior

    main

    The Config struct controls the global behavior of the matcher. You can customize it using a builder-like pattern.

    Available Configuration Options:

    • max_typos(Option<u16>): The maximum number of characters missing from the needle before an item is filtered out.
    • casing(CaseMatching): Controls case sensitivity.
    • unicode(UnicodeMatching): Controls how Unicode characters are handled.
    • matching(Matching): Selects the core algorithm (e.g., Fuzzy, Exact, Prefix, Suffix, Substring).
    • sort(SortStrategy): Determines the order of results.
    • scoring(Scoring): Fine-tunes the Smith-Waterman scoring parameters.

    Warning: Frizbee will panic if the needle length exceeds the limit calculated from your Scoring configuration. With default settings, the limit is 10,922 characters.

    use frizbee::{Config, Matcher, Matching};
    
    let mut config = Config::default();
    config = config.matching(Matching::Prefix).max_typos(Some(2));
    
    let mut matcher = Matcher::new("foo", &config);