Infidex Search Engine

repository·master·Indexed 42 days ago

A high-performance, zero-dependency .NET search engine featuring typo-tolerant matching via pattern recognition and multi-dimensional hyperspheres. It includes Infiscript, a SQL-like filtering language for complex queries, and supports faceted search, document boosting, and binary index serialization. The engine utilizes a lexicographic ranking model combining BM25+ candidate generation with a three-stage pipeline for precise match quality and positional primacy.

Tokens: 4.4K
Snippets: 12
Records: 17
Agent score: 4%

What's inside Infidex

Thread safety and concurrency in Infidex
master
Infidex is designed for high-concurrency environments:
- Search Operations: SearchEngine.Search is thread-safe and can be called concurrently from multiple threads.
- Indexing/Persistence: Operations like IndexDocuments, IndexDocumentsAsync, IndexDocument, CalculateWeights, Save, and Load acquire an exclusive writer lock. These operations will block other operations while they run.
- Best Practice: Share a single SearchEngine instance per application or service and allow multiple threads to use it concurrently for searching.
https://github.com/lofcz/infidex/blob/master/README.md

Understand the Precedence Hierarchy

master

The Precedence component (the high byte of the 16-bit score) determines the primary ranking tiers. Higher bits always dominate lower bits. The hierarchy is as follows:

Bit	Value	Condition	Rationale
7	128	All query terms found in document	Completeness is paramount
6	64	All terms fully matched (whole word OR exact prefix)	Clean matches beat partial
5	32	Single-term: Strict whole-word match<br>Multi-term: Lexically perfect doc (every doc token explained by query)	Exact intent signal
4	16	Single-term: Perfect doc<br>Multi-term: Strict whole-word match	Secondary exactness
3	8	First match at position 0 (title starts with query)	Positional primacy
2	4	Precise prefix match (query is prefix of token)	Partial but anchored

How Infidex ranking works
master
Infidex uses a lexicographic ranking model that prioritizes structural and positional properties over pure semantic similarity. The ranking is determined by a triple $(\text{Precedence}, \text{Semantic}, \tau)$, where $\tau$ is an 8-bit tiebreaker. This ensures that completeness (finding all terms) and match quality (whole words vs. prefixes) always dominate the final order.
The Three-Stage Pipeline
1. BM25+ Candidate Generation: Uses character n-grams (2-grams and 3-grams) and an inverted index to find potential document candidates.
2. Lexical Coverage Analysis: Analyzes the top candidates for per-term coverage using exact matching, fuzzy matching (Damerau–Levenshtein), joined/split word detection, prefix/suffix matching, and LCS fallback.
3. Lexicographic Score Fusion: Combines the results into a final score where high-order bits represent match quality tiers (Precedence) and low-order bits represent continuous similarity (Semantic).
https://github.com/lofcz/infidex/blob/master/README.md

Perform Faceted Search and Aggregations

master

To enable faceted search (dynamic filters and aggregations), set EnableFacets = true on your Query object. The resulting search results will contain a Facets dictionary where you can iterate through field names and their corresponding value counts.

var query = new Query("science fiction", maxResults: 50)
{
    EnableFacets = true
};

var results = engine.Search(query);

// Get facet counts
if (results.Facets != null)
{
    foreach (var (fieldName, values) in results.Facets)
    {
        Console.WriteLine($"\n{fieldName}:");
        foreach (var (value, count) in values)
        {
            Console.WriteLine($"  {value}: {count} documents");
        }
    }
}

Compile and Serialize Infiscript Filters

master

For high performance and portability, Infiscript filters can be compiled into bytecode. This allows you to parse a filter once, save the resulting bytes to disk, and reload them later without re-parsing the string.

// Compile once, use many times
var filter = Filter.Parse("genre = 'Sci-Fi' AND year >= 2000");
var bytecode = filter.CompileToBytes();

// Save to disk
File.WriteAllBytes("filter.bin", bytecode);

// Load and use later
var loaded = Filter.FromBytecode(File.ReadAllBytes("filter.bin"));
var query = new Query("space") { CompiledFilterBytecode = bytecode };

Persist and load a search index
master
Infidex supports efficient binary serialization to save and load indexes to disk. The binary format includes all documents, the inverted index, TF-IDF weights, FST term index, and word matcher data. The resulting file is typically much smaller than the source data due to byte-quantization and compression.
Synchronous Save/Load
Use Save and Load for standard operations.
Asynchronous Save/Load
Use SaveAsync and LoadAsync for non-blocking I/O operations.
```
// Saving an Index
var engine = SearchEngine.CreateDefault();
engine.IndexDocuments(documents);
engine.Save("my-index.bin");

// Loading an Index
var engine = SearchEngine.Load(
    filePath: "my-index.bin",
    indexSizes: new[] { 3 },
    startPadSize: 2,
    stopPadSize: 0,
    enableCoverage: true
);

// Async versions
await engine.SaveAsync("my-index.bin");
var engine = await SearchEngine.LoadAsync(
    filePath: "my-index.bin",
    indexSizes: new[] { 3 },
    startPadSize: 2,
    stopPadSize: 0
);
```
https://github.com/lofcz/infidex/blob/master/README.md

Use Infiscript for Complex Filtering

master

Infiscript is a SQL-like filtering language used to create complex query filters. You can parse strings into Filter objects using Filter.Parse(). These filters can be applied to a Query object to restrict search results.

using Infidex.Api;

// Example: Boolean logic and range checks
var filter = Filter.Parse("genre = 'Sci-Fi' AND year >= 2000");

// Example: Using filters in a query
var query = new Query("matrix", maxResults: 20)
{
    Filter = Filter.Parse("year >= 2000 AND rating > 7.0")
};

var results = engine.Search(query);

Sort Search Results

master

You can sort query results by any field using the SortBy and SortAscending properties on a Query object. Use fields.GetField("fieldName") to specify the target field.

// Sort by year (descending)
var query = new Query("thriller", maxResults: 20)
{
    SortBy = fields.GetField("year"),
    SortAscending = false
};

// Sort by rating (ascending)
var query = new Query("comedy", maxResults: 20)
{
    SortBy = fields.GetField("rating"),
    SortAscending = true
};

Run Infidex tests
master
To run the comprehensive test suite (covering multilingual relevancy, concurrency, persistence, and performance), use the standard .NET test command.
```
dotnet test
```
https://github.com/lofcz/infidex/blob/master/README.md

Apply Document Boosting

master

You can increase the relevance scores of specific documents using the Boost class. A Boost consists of an Infiscript Filter and a BoostStrength. To use boosts, set EnableBoost = true on your Query and provide an array of Boost objects.

```csharp
// Boost recent movies
var recentBoost = new Boost(
    Filter.Parse("year >= 2020"),
    BoostStrength.Large  // +20 to score
);

// Boost highly-rated content
var ratingBoost = new Boost(
    Filter.Parse("rating >= 8.0"),
    BoostStrength.Medium  // +10 to score
);

var query = new Query("action movie", maxResults: 20)
{
    EnableBoost = true,
    Boosts = new[] { recentBoost, ratingBoost }
};

var results = engine.Search(query);

Boost Strengths:

Small: +5
Medium: +10
Large: +20
Extreme: +40

Perform a Basic Search

master

You can initialize a search engine using SearchEngine.CreateDefault(), index documents using the Document class, and perform searches that are resilient to typos. The Search method returns a result object containing Records with DocumentId and Score properties.

using Infidex;
using Infidex.Core;

// Create search engine
var engine = SearchEngine.CreateDefault();

// Index documents
var documents = new[]
{
    new Document(1L, "The quick brown fox jumps over the lazy dog"),
    new Document(2L, "A journey of a thousand miles begins with a single step"),
    new Document(3L, "To be or not to be that is the question")
};

engine.IndexDocuments(documents);

// Search with typos - still finds matches!
var results = engine.Search("quik fox", maxResults: 10);

foreach (var result in results.Records)
{
    Console.WriteLine($"Doc {result.DocumentId}: Score {result.Score}");
}

Configure the SearchEngine

master

You can initialize the SearchEngine using three different approaches depending on your requirements for speed versus accuracy:

Default Configuration: Recommended for most use cases. It uses multi-size n-grams ([2, 3]) and enables full coverage (exact, LD1 fuzzy, prefix/suffix, join/split, and LCS) to balance speed and accuracy.
Minimal Configuration: Optimized for maximum speed. It uses a single n-gram size ([3]) and disables coverage.
Custom Configuration: Allows full control over n-gram sizes, padding, coverage settings, text normalization, tokenization, and word matching parameters.

When using custom configuration, you can also define fieldWeights to adjust the importance of different fields during search.

// Default Configuration (Recommended)
var engine = SearchEngine.CreateDefault();

// Minimal Configuration (Speed Priority)
var engine = SearchEngine.CreateMinimal();

// Custom Configuration
var engine = new SearchEngine(
    indexSizes: new[] { 2, 3 },           // Character n-gram sizes
    startPadSize: 2,                       // Pad start of words
    stopPadSize: 0,                        // Pad end of words
    enableCoverage: true,                  // Enable lexical matching
    textNormalizer: TextNormalizer.CreateDefault(),
    tokenizerSetup: TokenizerSetup.CreateDefault(),
    coverageSetup: CoverageSetup.CreateDefault(),
    wordMatcherSetup: new WordMatcherSetup
    {
        SupportLD1 = true,                 // Enable fuzzy matching
        SupportAffix = true,               // Enable prefix/suffix
        MinimumWordSizeExact = 2,
        MaximumWordSizeExact = 50,
        MinimumWordSizeLD1 = 4,
        MaximumWordSizeLD1 = 20
    },
    fieldWeights: new[] { 1.0f, 2.0f }   // Per-field weights
);

Infidex Search Engine

What's inside Infidex

The Three-Stage Pipeline

Synchronous Save/Load

Asynchronous Save/Load