SeekStorm Documentation

The server organizes data in a specific hierarchy. To perform manual backups or restores, you should shut down the server first, then copy/move the directories.

Hierarchy Levels:

1. API Keys: Top level.
2. Indices: One directory per API key.
3. Shards: An index is divided into shards.
4. Levels: Each shard is divided into levels (one level = 64k documents). Once a level reaches 64k documents, it is committed to disk and becomes immutable.

Directory Example:

seekstorm_index/
├─ 0/ (API Key)
│  ├─ 0 (Index)
│  │  ├─ 0 (Shard)
│  │  │  ├─ 0 (Level)
│  │  │  ├─ 1
│  │  │  └─ 2
│  │  └─ 1
│  └─ 1
└─ apikey.json (Contains API key hash and quotas)

seekstorm_index/  
├─ 0/  
│  ├─ 0  
│  ├─ 1  
│  ├─ 2  
├─ 1/  
│  ├─ 0  
│  ├─ 1 ─ shards ─ ├─ 0  
│  │               ├─ 1  
│  │               ├─ 2

To ensure N-gram indexing provides the same top-k results as single-term indexing, SeekStorm implements measures to maintain BM25 score accuracy.

BM25 Scoring Modes:

• LexicalSimilarity::Bm25f: Provides scores almost identical to single-term indexing. It achieves this by storing the document frequency (DF) and term frequency (TF) of each partial term within the N-gram. Note that small differences may occur due to lossy logarithmic compression used for N-gram posting counts.
• LexicalSimilarity::Bm25fProximity: An alternative score that uses the DF and TF of the N-gram itself rather than its partial terms. This honors the proximity of terms within the N-gram for scoring purposes, but it cannot independently score the posting list length and position count of the individual N-gram terms.

The lexical engine uses an inverted index that can be accessed in two modes. The index file format is identical for both, allowing you to switch modes for an existing index at any time:

• RAM mode: The entire index is preloaded into RAM. This provides minimal latency (no disk access during search), even after a cold start, but results in higher RAM consumption and longer initial load times.
• Mmap mode: The index is accessed via memory-mapped files. This provides minimal RAM consumption and minimal load times, as the OS handles caching. It is highly scalable and the cache is persistent between program starts until a reboot.

SeekStorm uses a schema-based approach for documents. Every document can contain arbitrary fields of different types, which can be searched or filtered individually or globally.

• schema.json: Defines the fields, their types, and whether they are stored or indexed.
• Document Store: Documents are stored in JSON format and compressed using Zstandard. Only fields marked as stored in the schema.json will be included in the document store (docstore.bin) and can be returned in search results.

SeekStorm uses a dual-engine architecture to provide hybrid search capabilities. Instead of a single engine attempting to handle all query types, it runs two native, first-class engines in parallel:

1. Lexical Engine: An inverted index optimized for keyword relevance using BM25 or Boolean logic.
2. Vector Engine: A native ANN (Approximate Nearest Neighbor) index using a Leveled-IVF architecture optimized for vector similarity.

A Query Planner sits between the user and these engines. It determines the search intent and strategy (automatically or manually) and coordinates how to combine results from both engines using Result Fusion (such as Reciprocal Rank Fusion - RRF) to produce a single, final ranked result list. This allows for pure lexical, pure vector, or true hybrid search.

String facets allow you to cluster documents by distinct string values (e.g., Language, Brand).

• Supported Types: String16, String32, StringSet16, and StringSet32 (for multiple values per document).
• Counting: Returns distinct values and their occurrence counts.
• Filtering: Use FacetFilter::String16 to restrict results to documents containing specific values.
• Sorting: Results can be sorted by the string facet field in ascending or descending order.
• Limits: String16 supports up to 65,535 unique values; String32 supports up to 4,294,967,295. String values are limited to 100 characters.

To use the SeekStorm Rust client in your project, add the seekstorm_client crate using cargo add.

cargo add seekstorm_client

To use the SeekStorm Rust client, add the following crates to your Cargo.toml file:

cargo add seekstorm_client_rs
cargo add tokio
cargo add serde_json

SeekStorm can automatically convert PDF files to text and index them.

Features:

• Extracts title from metatags, the first line of text, or the filename.
• Extracts creation date from metatags or file creation date (Unix timestamp).
• Copies ingested PDFs to a files subdirectory within the index.

Required Schema: To index PDFs, your index schema must include the following fields:

• title (Text, stored, index_lexical: true, boost: 10)
• body (Text, stored, index_lexical: true)
• url (Text, stored, index_lexical: false)
• date (Timestamp, stored, index_lexical: false, facet: true)

Note for Windows users: Use curl.exe and escape double quotes in the JSON payload.

The SeekStorm client requires an asynchronous runtime. It is recommended to use tokio. You can wrap your SeekStorm logic within a #[tokio::main] async function.

use std::error::Error;

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error + Send + Sync>> {
    // your SeekStorm code here
    Ok(())
}