DOCSEARCHbyAlgolia
Submit Docs

FlexSearch

repository·master·Indexed 11 days ago

https://github.com/nextapps-de/flexsearch

A high-performance, next-generation full-text search library for Browser and Node.js (v0.8.215). It supports multi-field document search, phonetic matching, result highlighting, and persistent indexing across databases including SQLite, Redis, MongoDB, Postgres, and Clickhouse. The library utilizes Workers for parallel index updates and queries and supports a wide range of charsets including Latin, CJK, Arabic, and Cyrillic.

Tokens
49.8K
Snippets
176
Records
211
Agent score
93%

What's inside FlexSearch

  1. Overview of FlexSearch capabilities

    master

    FlexSearch is a high-performance, next-generation full-text search library designed for both Browser and Node.js environments. It is optimized for speed, performing queries significantly faster than many alternative libraries.

    Key Features:

    • Multi-field search: Also known as document search.
    • Advanced Matching: Supports phonetic transformations, partial matching, and tag-search.
    • Scalability: Uses Workers to perform index updates and queries in parallel via dedicated threads.
    • Persistence: v0.8 introduces Persistent Indexes, allowing large datasets to be stored in various databases.
    • Result Enhancement: Supports result highlighting and suggestions.

    Supported Platforms:

    • Browser
    • Node.js

    Supported Databases for Persistent Indexes:

    • InMemory (Default)
    • IndexedDB (Browser)
    • Redis
    • SQLite
    • Postgres
    • MongoDB
    • Clickhouse

    Supported Charsets:

    • Latin
    • Chinese, Korean, Japanese (CJK)
    • Hindi
    • Arabic
    • Cyrillic
    • Greek and Coptic
    • Hebrew

  2. API Overview: Constructors and Core Classes

    master

    FlexSearch provides several specialized classes for different search requirements:

    • Index: The primary class for basic text indexing and searching.
    • Document: Used for searching within structured objects (documents) rather than plain strings.
    • Worker: Enables off-main-thread searching using Web Workers.
    • Encoder: Handles text encoding, including support for custom filters, stemmers, and mappers.
    • Resolver: Manages complex query logic like boolean operations (and, or, xor, not) and result manipulation (boost, limit, offset).
    • IndexedDB: Provides persistent storage using the browser's IndexedDB API.

  3. Implement Fuzzy-Search in FlexSearch

    master

    FlexSearch provides several ways to make queries more tolerant of typos or variations. You can achieve fuzziness through:

    1. Tokenizers: Use tolerant, forward, reverse, or full.
    2. Encoders: Choose a built-in encoder preset. They are sorted by increasing fuzziness: normalize > balance > advanced > extra > soundex.
    3. Language Presets: Use language-specific files (e.g., /lang/en.js) for specific content.
    4. Suggestions: Enable suggestions by passing suggest: true in the search options.

    You can also extend this behavior by applying custom Mapper, Replacer, Stemmer, or Filter instances, or by assigning custom functions to normalize(str), prepare(str), or finalize(arr) on the Encoder.

  4. Understand FlexSearch build naming conventions

    master

    When selecting a build from the /dist/ folder or a CDN, use these abbreviations to identify the capabilities:

    • bundle: All features included; available on window.FlexSearch in browsers.
    • light: Only basic features included; available on window.FlexSearch in browsers.
    • es5: Includes support for EcmaScript5; available on window.FlexSearch.
    • module: A Javascript module (ESM). Members can be imported via named imports (e.g., import { Index, Document } from ...) or as a default export.
    • min: The bundle is minified.
    • debug: Enables debug mode with extra console information and advice. Do not use in production as it reduces performance.

  5. Understand the Encoder in FlexSearch

    master
    The Encoder is a core component of FlexSearch responsible for language processing and "fuzziness" (such as Phonetic or Fuzzy search). It determines how text is normalized, tokenized, and transformed before being indexed. Search capabilities are highly dependent on how the Encoder is configured.

  6. Understand SQLite Table Structure and Naming

    master

    FlexSearch creates separate database files for each index name (e.g., "my-store"). This prevents naming collisions when using multiple stores for different indexes simultaneously.

    For DocumentIndex, WorkerIndex, or Index instances, field names are mapped to specific table names within the database. The internal structure follows this pattern:

    • map:[field] (FlexSearch Data)
    • ctx:[field] (FlexSearch Data)
    • tag:[field] (FlexSearch Data)
    • cfg:[field] (FlexSearch Data)
    • reg (FlexSearch Data)

  7. Understand PostgreSQL table structure and schemas

    master

    FlexSearch uses PostgreSQL SCHEMAs to isolate different indexes. By default, providing a name to the Database constructor (e.g., new Database("my-store", ...) ) creates a schema named after that prefix. This prevents naming collisions when using multiple stores.

    Each index (whether Index, DocumentIndex, or WorkerIndex) requires its own Database instance with its own schema.

    Default Schema Structure:

    DATABASE_NAME
  |__ SCHEMA
        |__TABLE map:field (FlexSearch Data) 
        |__TABLE ctx:field (FlexSearch Data) 
        |__TABLE tag:field (FlexSearch Data) 
        |__TABLE cfg:field (FlexSearch Data) 
        |__TABLE reg       (FlexSearch Data)

    You can force the use of a specific schema (like public) by passing the schema option without providing a name prefix.

    const db = new Database({
    schema: "public"
});

  8. Use Context Search for relevance scoring

    master

    Context Search limits relevance by using a bidirectional moving window of two pointers (terms). The distance between these terms is controlled by the depth option. This allows the engine to score results based on how close related terms are to each other within a document, rather than just their distance from the document root.

    Requirements:

    • Only the strict tokenizer is supported with context search.
    • Increasing depth increases memory consumption.

    Configuration Options:

    • resolution (Number, default: 3): Sets the scoring resolution.
    • depth (Number, default: 1): Enables/disables context index and sets the maximum initial distance of related terms.
    • bidirectional (Boolean, default: true): If true, the context chain can move in both directions. Disable this only if you require more exact matches with fewer results.
    // Basic context search
var index = new FlexSearch({
    tokenize: "strict",
    context: true
});

// Custom context configuration
var index = new FlexSearch({
    tokenize: "strict",
    context: { 
        resolution: 5,
        depth: 3,
        bidirectional: true
    }
});

  9. Understand the IndexedDB table structure for FlexSearch

    master

    When using the IndexedDB adapter, FlexSearch creates a database entry for each unique namespace provided to the IndexedDB constructor (e.g., "my-store"). This prevents naming collisions between different indexes.

    Inside the database, FlexSearch maps its internal components to specific OBJECTSTORE names based on the field names. The standard structure includes:

    • map:field: FlexSearch Data
    • ctx:field: FlexSearch Data
    • tag:field: FlexSearch Data
    • cfg:field: FlexSearch Data
    • reg: FlexSearch Data

  10. Understand Resolution and scoring

    master

    Resolution refers to the maximum count of scoring slots the content is divided into.

    • General Rule: A well-balanced value is $2 * floor(\sqrt{content.length})$, where content is the largest value added via index.add().
    • Minimums: A resolution of 1 disables scoring (unless context is enabled). A suggested minimum is 3 because the first and last slots are reserved.
    • Context Resolution: When context is enabled, the minimum resolution is 1. You can adjust context resolution independently, but it should generally not exceed 50% of the default resolution.

  11. Optimize performance by sharing Encoders

    master

    Assigning an Encoder instance to the top level of a configuration shares it across all fields. This improves encoding efficiency and memory allocation.

    Best Practices:

    • Group similar content types: Share an encoder among fields that contain similar data (e.g., all text fields share one encoder, all numeric IDs share another).
    • Avoid over-sharing: Do not use a single encoder for fields with different content types (e.g., mixing text terms and numeric IDs), as this can negatively impact performance.
    • Cross-index sharing: You can share the same encoder instance across different index types or even multiple different Document instances.
    // Define specialized encoders
const encoder_terms = Encoder(
    Charset.LatinAdvanced,
    { include: { letter: true } }
);
const encoder_numeric = new Encoder(Charset.Default);

// Share encoder_terms across multiple fields in different documents
const orders = Document({
   document: {
       id: "id",
       index: [
           { field: "product_title", encoder: encoder_terms },
           { field: "product_details", encoder: encoder_terms },
           { field: "order_date", encoder: encoder_numeric },
           { field: "customer_id", encoder: encoder_numeric }
       ]
   }
});

const billings = Document({
   document: {
       id: "id",
       index: [
           { field: "product_title", encoder: encoder_terms },
           { field: "product_content", encoder: encoder_terms },
           { field: "billing_date", encoder: encoder_numeric },
           { field: "customer_id", encoder: encoder_numeric }
       ]
   }
});

  12. Understand the core components of FlexSearch

    master

    To achieve optimal search results with FlexSearch, you should understand three fundamental concepts:

    1. Tokenizer: Controls how text is split into searchable terms (e.g., partial matches).
    2. Encoder: Handles how terms are transformed into searchable representations (see doc/encoder.md).
    3. Suggestions: Provides related terms or corrections for queries.