Infinity AI-Native Database Documentation

The FST library is a C++ reimplementation of the BurntSushi/fst library. It is designed for efficiently storing and searching very large ordered sets or maps (up to billions of entries) where keys are byte strings.

Key characteristics:

• Binary Compatibility: The FST file format is designed to be binary compatible with the original BurntSushi/fst library.
• Memory Efficiency: Optimized for extremely large datasets to ensure all operations remain memory efficient.
• Compression: Uses a finite state machine to compress common prefixes and suffixes in keys.
• Query Capabilities: Supports efficient querying via automata (e.g., regular expressions, Levenshtein distance for fuzzy queries) and lexicographic ranges.

cURLpp is a C++ wrapper for libcURL that provides an object-oriented (OO) layer over the standard C library. While you can use libcURL directly in C++ programs, cURLpp offers several specific advantages:

• Type Safety: Uses C++ type systems to prevent errors.
• Exception Safety: Integrates with C++ exception handling.
• Standard Library Integration: Uses C++ standard library structures instead of custom/homemade ones.
• Option Querying: Allows querying handles for option values (available in version 0.5.x and newer).

PLAID (Product Lookups for AI-Dense retrieval) is designed for high compression and efficient memory management.

Key Features:

• Residual Quantization: Uses 2-bit or 4-bit quantization for compact storage.
• Mmap Support: Supports memory-mapped index files for efficient memory management.
• Batch Processing: Full support for efficient batch search capabilities.
• Incremental Updates: Supports incremental index building.

Performance Characteristics:

• Memory Usage: Reduces footprint to approximately 1/8 to 1/16 of the original size using 2-bit or 4-bit quantization.
• Indexing Complexity: $O(N \times K \times D)$ where $N$ is embeddings, $K$ is centroids, and $D$ is dimension.
• Search Complexity: $O(Q \times K + C \times D)$ where $Q$ is query tokens and $C$ is candidates.

To remove zsv from your system, use the following command:

sudo make uninstall

To build and install the full zsv package (including the library and CLI) from source, you need a basic Unix toolchain with sh and make/gmake.

Run the following commands:

./configure && sudo ./install.sh

Alternatively, you can use:

./configure && sudo make install

To build independent executables into a local build folder instead of a system-wide installation, use make install instead of make all.

The base64 library provides a simple, header-only approach for converting strings to and from Base64 encoding. It requires a C++17 compatible compiler.

To encode a string, use the to_base64() function. To decode a Base64 string back to its original form, use the from_base64() function.

#include <iostream>
#include "base64.hpp"

int main() {
  // Encode a string to base64
  auto base64 = to_base64("Hello, World!");
  std::cout << base64 << std::endl; // SGVsbG8sIFdvcmxkIQ==

  // Decode a base64 string
  auto s = from_base64("SGVsbG8sIFdvcmxkIQ==");
  std::cout << s << std::endl; // Hello, World!
}

To benchmark vector search performance using the provided scripts, follow these steps:

1. Clone the Infinity source code to your local directory.
2. Prepare the dataset: Copy or create a symbolic link of the sift_1m or gist_1m dataset into the test/data/benchmark/sift_1m (or gist_1m) directory within the Infinity source path.
3. Import data and create index: Start the Infinity server, then execute remote_benchmark_knn_import.py to import the dataset and build the index.
4. Execute search benchmark: Run remote_benchmark_knn.py to perform the searching benchmark.