fuzzball.js

repository·master·Indexed 20 days ago

https://github.com/nol13/fuzzball.js

A JavaScript library for fuzzy string matching, ported from the Python TheFuzz/RapidFuzz libraries. It provides various scoring algorithms including ratio, partial_ratio, token_sort_ratio, and token_set_ratio to calculate string similarity. The library supports synchronous and asynchronous extraction of best matches from lists, duplicate removal via fuzz.dedupe(), and advanced configurations for Unicode, astral symbols, and custom pre-processing.

Tokens
19.3K
Snippets
57
Records
71
Agent score
64%

What's inside fuzzball.js

  1. Choose between Full, Lite, and Ultra Lite versions

    master

    Fuzzball provides three bundles to balance features and file size:

    VersionSize (Compressed)Key Differences
    Full~15.1kBComplete feature set, including partial ratios, wildcards, and collation.
    Lite~6.3kBNo partial ratio functions; limited wildcard support.
    Ultra Lite~4.0kBNo partial ratios, no wildcards, no collation, no astral symbol handling. Alphanumeric check strips all non-ASCII characters.

    Note: The full version is optimized to only pull in necessary parts from difflib. The ultra_lite version's extract functions are less optimized for large datasets.

  2. Score across multiple fields or custom objects

    master

    You can extend the matching logic beyond simple strings by using a custom scorer and a processor.

    • Combining Fields: Use a processor to concatenate multiple fields from an object into a single string for scoring.
    • Custom Logic: Provide a scorer function that accepts (query, choice, options). This allows you to perform logic like checking if specific properties (e.g., gender) match exactly before applying fuzzy matching to others.

    When using a custom scorer, the query and choice can be any data type (objects, numbers, etc.) as long as your scorer handles them.

    // Example: Scoring based on multiple fields via processor
    const processor = choice => choice.field1 + " " + choice.field2;
    
    // Example: Custom scorer for complex object matching
    const query = {name: "tiger", gender: "female"};
    const choices = [
      {name: "tigger", gender: "male"},
      {name: "lulu", gender: "female"}
    ];
    
    function myCustomScorer(query, choice, options) {
      if (query.gender !== choice.gender) return 0;
      return fuzz.ratio(query.name, choice.name, options);
    }
    
    const results = fuzz.extract(query, choices, { scorer: myCustomScorer });
  3. Basic Usage of fuzzball

    master

    The core functionality involves calculating similarity ratios between strings or extracting the best matches from a list of choices.

    By default, fuzz.extract returns an array of tuples: [choice, score, index/key]. To receive an array of objects instead, set options.returnObjects = true to get [{choice, score, key}, ...].

    For asynchronous operations, use fuzz.extractAsPromised. This supports cancellation via an AbortController passed in the options.abortController field.

    fuzz = require('fuzzball');
    
    // Simple ratio
    fuzz.ratio("hello world", "hiyyo wyrld"); // 64
    
    // Token set ratio
    fuzz.token_set_ratio("fuzzy was a bear", "a fuzzy bear fuzzy was"); // 100
    
    // Extracting best matches from choices
    const options = {scorer: fuzz.token_set_ratio};
    const choices = ["Hood, Harry", "Mr. Minor", "Mr. Henry Hood"];
    const results = fuzz.extract("mr. harry hood", choices, options);
    // Results: [ [ 'Hood, Harry', 100, 0 ], [ 'Mr. Henry Hood', 85, 2 ], [ 'Mr. Minor', 40, 1 ] ]
    
    // Async extraction with cancellation
    const abortController = new AbortController();
    const asyncOptions = { ...options, abortController };
    
    fuzz.extractAsPromised("gonna get canceled", choices, asyncOptions)
      .then(res => {/* do stuff */})
      .catch((e) => {
        if (e.message === 'aborted') console.log('Search was aborted!');
      });
    
    abortController.abort();
  4. Optimize performance for large datasets

    master

    If you are searching a large list of terms repeatedly, you can pre-calculate expensive processing steps to boost performance.

    For token_sort_ratio (using extract)

    Pre-calculate the sorted tokens and attach them to each choice object as proc_sorted. This prevents process_and_sort() from running during every search.

    // Pre-process choices
    for (let choice of choices) {
      choice.proc_sorted = fuzz.process_and_sort(fuzz.full_process(choice.model));
    }
    // Use in extract
    const options = { scorer: fuzz.token_sort_ratio, full_process: false };
    fuzz.extract(query, choices, options);

    For token_set_ratio (using extract)

    Pre-calculate unique tokens and attach them to each choice object as tokens. This prevents unique_tokens() from running during every search.

    // Pre-process choices
    for (let choice of choices) {
      choice.tokens = fuzz.unique_tokens(fuzz.full_process(choice.model));
    }
    // Use in extract
    const options = { scorer: fuzz.token_set_ratio, full_process: false };
    fuzz.extract(query, choices, options);

    For standalone functions

    If using scorers as standalone functions (not via extract), pass the pre-calculated values via the options object.

    • For token_sort_ratio: Set options.proc_sorted = true and pass both strings already processed by process_and_sort and full_process.
    • For token_set_ratio: Pass an array of two token sets to options.tokens (e.g., [query_tokens, choice_tokens]).
  5. Install fuzzball

    master

    You can install fuzzball via NPM for Node.js environments, or include it in the browser using a UMD bundle or as an ES module.

    Note: If you need to support Internet Explorer or Node.js versions earlier than v14, use version v2.1.6 or earlier. For the smallest possible file size, see the Lite Versions section.

    npm install fuzzball
  6. Use fuzzball in the browser

    master

    To use fuzzball in a web page, you can use a <script> tag with a UMD bundle or import it as an ES module. Ensure your script is served with UTF-8 encoding.

    UMD Bundle:

    <script charset="UTF-8" src="dist/fuzzball.umd.min.js"></script>
    <script>
      fuzzball.ratio("fuzz", "fuzzy");
    </script>

    ES Module:

    <script charset="UTF-8" type="module">
      import {ratio} from './dist/esm/fuzzball.esm.min.js';
      console.log(ratio('fuzz', 'fuzzy'));
    </script>
  7. Manage the fuzzball_demo application scripts

    master

    The fuzzball_demo project is a React application bootstrapped with Create React App. You can manage the development, testing, and production lifecycle using the following npm scripts in the project directory:

    # Run the app in development mode at http://localhost:3000
    npm start
    
    # Launch the test runner in interactive watch mode
    npm test
    
    # Build the app for production in the `build` folder
    npm run build
    
    # Permanently remove the build dependency to expose configuration files
    npm run eject
  8. Use the fuzzball.js Lite API

    master

    The Lite version of fuzzball.js provides a subset of the library's capabilities, optimized for smaller bundle sizes. It exports core string similarity algorithms, extraction utilities, and deduplication logic.

    Key exported functions include:

    • distance: Calculates the edit distance between strings.
    • ratio: Calculates the similarity ratio (default scorer).
    • token_set_ratio: Calculates similarity based on token sets.
    • token_sort_ratio: Calculates similarity based on sorted tokens.
    • extract: Synchronously extracts the best matches from a set of choices.
    • extractAsync: Asynchronously extracts matches to avoid blocking the main thread.
    • extractAsPromised: A Promise-based wrapper for extractAsync.
    • dedupe: Removes duplicate items from an array or object based on similarity thresholds.
  9. Use fuzzball.js ultra-lite for string similarity and extraction

    master

    The ultra_lite version of fuzzball.js provides a compact set of tools for calculating string similarity (Levenshtein distance and various ratios) and performing fuzzy extraction from a set of choices. It is suitable for environments where bundle size is a priority.

    Core Capabilities

    • Similarity Metrics: Calculate distance, ratio, token_set_ratio, and token_sort_ratio between strings.
    • Fuzzy Extraction: Find the best matches for a query within an array of strings or an object of choices.
    • Asynchronous Extraction: Use extractAsync or extractAsPromised to perform searches without blocking the main thread, supporting cancellation via AbortController or cancelToken.
    • Pre-processing: Customize how strings are processed using a processor function or built-in full_process logic.
    import { extract, distance, ratio } from './fuzzball_ultra_lite.esm.min.js';
    
    // Basic similarity
    const dist = distance('kitten', 'sitting');
    const rat = ratio('kitten', 'sitting');
    
    // Fuzzy extraction
    const choices = ['apple', 'banana', 'cherry'];
    const matches = extract('aple', choices);
    // Returns array of [choice, score, index]
  10. How async extraction and cancellation work

    master

    The extractAsync function is designed for non-blocking execution on large arrays or objects. It uses an internal searchLoop that yields control back to the event loop every asyncLoopOffset iterations using setImmediate.

    To stop an ongoing asynchronous search, you can provide either an abortController or a cancelToken in the options object:

    1. AbortController: The function checks options.abortController.signal.aborted. If true, it calls the callback with new Error("aborted").
    2. CancelToken: The function checks options.cancelToken.canceled. If true, it calls the callback with new Error("canceled").

    This mechanism allows developers to cancel heavy fuzzy matching operations in response to user input changes or component unmounting.

  11. Use custom processors and scorers in extract()

    master

    You can extend extract() by providing a processor to transform choice data before scoring, and a scorer to define how similarity is calculated.

    const choices = [{ name: 'Apple' }, { name: 'Banana' }];
    
    extract('apl', choices, {
      // processor extracts the string to be compared from the object
      processor: (obj) => obj.name,
      // scorer can be any function following the (str1, str2, options) signature
      scorer: QRatio,
      returnObjects: true
    });
  12. Use wildcards in fuzzy matching

    master

    You can define a set of characters to act as wildcards during edit distance calculation by setting options.wildcards to a string of characters.

    • Wildcards are case-insensitive (unless options.full_process is false).
    • Limitation: Wildcards are not supported when options.astral is set to true.
    • In fuzzball_lite, token set operations are not wildcard-aware.
    // '*' and 'x' are both treated as wildcards
    const options = { wildcards: "*x" };
    fuzz.ratio('fuzzba*l', 'fuXxball', options); // 100