blaze

Blaze is a lightweight, hackable full-text search engine written in Go. It is designed for keyword-based search using an inverted index, rather than semantic vector search. It is ideal for developers who want to understand or implement features like BM25 ranking, boolean queries, and proximity search without the complexity of a hyperscale engine.

Key Capabilities:

• Search: Term search, phrase search, boolean queries (AND, OR, NOT), and proximity ranking.
• Ranking: BM25 relevance scoring (industry standard) and proximity-based scoring.
• Text Processing: Tokenization, Snowball (Porter2) stemming for English, stopword filtering, and case normalization.
• Data Structures: Uses Skip Lists for $O(\log n)$ operations and Roaring Bitmaps for fast boolean query execution.

Note on Semantic Search: Blaze is focused on keyword-based search. For vector embeddings and semantic retrieval, use Comet. For production-grade, battle-tested full-text search, consider Bleve.

Blaze provides two primary ranking algorithms. Choosing the right one depends on your search goals:

• Use RankBM25 when: You need industry-standard relevance. It weights terms based on frequency and rarity (TF-IDF style) and performs automatic length normalization. It is recommended for most general use cases like broad topic searches or e-commerce product discovery.
• Use RankProximity when: You need to find terms that appear close together. This is ideal for finding specific phrases, co-occurrences, or when you need position data for snippet generation.

Best Practice: Use both. Use RankBM25 to cast a wide net for general relevance, and RankProximity to refine results for precise phrase matching.

// E-commerce: General product search
bm25Results := idx.RankBM25("wireless bluetooth headphones", 20)

// E-commerce: Exact product name
proxResults := idx.RankProximity("Sony WH-1000XM4", 20)

// Document search: Research papers
// BM25 for broad topic search
papers := idx.RankBM25("neural networks deep learning", 50)

// Document search: Finding specific phrase mentions
mentions := idx.RankProximity("neural networks", 50)

// Best practice: Use both for different purposes!
generalResults := idx.RankBM25(query, 100)    // Cast wide net
preciseResults := idx.RankProximity(query, 20) // Refine results

The QueryBuilder uses an operand stack and an operator stack to evaluate logical expressions. It supports grouping via functional closures to handle nested boolean logic.

Execution Flow Example: For a query like NewQueryBuilder(idx).Group(func(q) { q.Term("machine").Or().Term("deep") }).And().Term("learning").Execute():

1. Group Execution: The sub-builder executes machine OR deep. It looks up the bitmaps for both terms, performs a Union operation, and pushes the resulting bitmap onto the parent stack.
2. Logical Continuation: The builder then processes the And() operator and the learning term.
3. Final Execution: The Execute() method pops the bitmaps from the stack, performs an Intersection (AND), and returns the final filtered bitmap.

// Conceptual usage of the QueryBuilder pattern
query := NewQueryBuilder(idx).
    Group(func(q *SubQueryBuilder) {
        q.Term("machine").Or().Term("deep")
    }).
    And().
    Term("learning").
    Execute()

Blaze processes raw text into searchable tokens through a five-stage pipeline:

1. Tokenization: Splits text on non-alphanumeric characters.
2. Lowercasing: Normalizes case (e.g., "Quick" $\rightarrow$ "quick").
3. Stopword Filtering: Removes common words (e.g., "the", "a", "is").
4. Length Filtering: Removes tokens shorter than a specified threshold (default is 2 chars).
5. Stemming: Uses Snowball/Porter2 algorithms to reduce words to their root (e.g., "running" $\rightarrow$ "run").

You can customize this behavior using blaze.AnalyzerConfig.

// Use default configuration
tokens := blaze.Analyze("The quick brown fox")

// Custom configuration
config := blaze.AnalyzerConfig{
    MinTokenLength:  3,      // Only keep tokens >= 3 chars
    EnableStemming:  false,  // Disable stemming
    EnableStopwords: true,   // Keep stopword filtering
}
tokens := blaze.AnalyzeWithConfig("The quick brown fox", config)

To get started, create a new inverted index using blaze.NewInvertedIndex(), index your documents using the Index method, and perform searches using methods like RankProximity. Each document is assigned an ID (e.g., 1, 2, 3) which is used to map terms to positions.

package main

import (
    "fmt"
    "github.com/wizenheimer/blaze"
)

func main() {
    // Create a new inverted index
    idx := blaze.NewInvertedIndex()

    // Index some documents
    idx.Index(1, "The quick brown fox jumps over the lazy dog")
    idx.Index(2, "A quick brown dog runs fast")
    idx.Index(3, "The lazy cat sleeps all day")

    // Search for documents containing "quick" and "brown"
    matches := idx.RankProximity("quick brown", 10)

    // Print results
    for _, match := range matches {
        fmt.Printf("Document %d (score: %.2f)\n",
            int(match.Offsets[0].DocumentID),
            match.Score)
    }
}