faiss-rs

By default, faiss-rs expects Faiss to be installed on your system and links to it dynamically. You must build Faiss manually using CMake with specific flags enabled.

Steps to build Faiss manually:

1. Build Faiss using CMake with FAISS_ENABLE_C_API=ON and BUILD_SHARED_LIBS=ON. It is recommended to use the c_api_head branch of the Faiss repository for compatibility.
2. Ensure the resulting dynamic library faiss_c is in your system's library path:
   • Linux: Add the path to LD_LIBRARY_PATH (e.g., /usr/lib).
   • macOS: Place libfaiss_c.dylib in /usr/local/lib/.
3. If you require GPU support, enable FAISS_ENABLE_GPU during the Faiss build process.

Note: faiss_c might link to the main faiss shared object, so you may need to install libfaiss.so as well.

cmake -B build -DFAISS_ENABLE_C_API=ON -DBUILD_SHARED_LIBS=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build

The VectorTransform trait provides a common interface for all vector transformation implementations in Faiss. It allows you to manage dimensions, train the transform on a dataset, and apply or reverse the transformation on vectors.

Key methods:

• is_trained(): Returns true if the transform has been trained.
• d_in(): Returns the input dimension.
• d_out(): Returns the output dimension.
• train(&mut self, n: usize, x: &[f32]): Trains the transform using n vectors from the slice x.
• apply_noalloc(&self, x: &[f32]): Applies the transformation to the input slice x and returns a new Vec<f32>.
• reverse_transform(&self, xt: &[f32]): Attempts to reverse the transformation on xt (may be approximate or unimplemented).

// Example of using a VectorTransform (via RandomRotationMatrix)
let mut rrt = RandomRotationMatrix::new(512, 256)?;
rrt.train(100, &data)?;
let transformed = rrt.apply_noalloc(&data);
let original = rrt.reverse_transform(&transformed);

To use GPU-accelerated indexes in faiss-rs, you must provide a GPU resource descriptor to the index's .into_gpu() or .to_gpu() methods.

Key constraints:

• Thread Safety: GPU implementations are not thread-safe. You cannot share GPU resources across thread boundaries (e.g., using Arc<StandardGpuResources> to move it into a thread::spawn will fail to compile because StandardGpuResources does not implement Sync).
• Resource Sharing: While you cannot move resources between threads, multiple indexes can share the same StandardGpuResources instance as long as they all reside within the same thread.

Use StandardGpuResources to manage the underlying CUDA memory and device context required for GPU operations.

use faiss::{StandardGpuResources, MetricType};
use faiss::index::flat::FlatIndex;

let gpu = StandardGpuResources::new()?;
let index = FlatIndex::new(64, MetricType::L2)?;
// Move the CPU index to the GPU using the resource descriptor
let gpu_index = index.into_gpu(&gpu, 0)?;

By default, most Faiss index implementations bind a sequential ID to each vector. If you need to bind vectors to specific, arbitrary IDs, you must use the IdMap<I> wrapper. IdMap<I> wraps an existing index I and provides the necessary mapping functionality while preserving the original index's properties.

Note on GPU usage: IdMap works with GPU-backed indexes, but the ID mapping itself resides in CPU memory. Once an index is wrapped in an IdMap, you cannot move it between the CPU and GPU.

use faiss::{IdMap, Idx, Index, FlatIndex};

let mut index = FlatIndex::new_l2(4)?; 
// FlatIndex does not support arbitrary IDs directly
assert!(index.add_with_ids(&[0., 1., 0., 1.], &[Idx::new(5)]).is_err());

// Wrap it in IdMap to enable arbitrary ID support
let mut index = IdMap::new(index)?;
index.add_with_ids(&[0., 1., 0., 1.], &[Idx::new(5)])?;
assert_eq!(index.ntotal(), 1);

In faiss-rs, GPU indexes are typically created by transferring an existing CPU index to the GPU. There are two primary ways to perform this transfer:

1. to_gpu / to_gpu_multiple: Creates a new GPU index while leaving the original CPU index intact. The two indexes are independent; changes to one will not reflect in the other.
2. into_gpu / into_gpu_multiple: Creates a new GPU index and consumes (discards) the original CPU index. This is more efficient if you no longer need the CPU version.

Similarly, you can move an index back to the CPU using to_cpu (leaving the GPU index intact) or into_cpu (consuming the GPU index).

Note on Lifetimes: GPU indexes are bound by a 'gpu lifetime, ensuring that the provided GpuResources remain in scope for as long as the GPU index exists.

// Example of transferring an index to GPU and back
let mut gpu_index = index_cpu.into_gpu(&res, 0).unwrap();
// ... perform GPU accelerated searches ...
let index_cpu = gpu_index.into_cpu().unwrap();

The Index<Data, Radius> trait is the primary interface for interacting with Faiss indexes. Most users will interact with indexes through this trait.

Core Methods:

• add(&mut self, x: &[Data]): Adds vectors from a C-contiguous slice. The number of vectors is inferred as x.len() / d.
• add_with_ids(&mut self, x: &[Data], xids: &[Idx]): Adds vectors with specific identifiers. Note that not all index types support this.
• train(&mut self, x: &[Data]): Trains the index using the provided data (required for many non-flat indexes).
• search(&mut self, q: &[Data], k: usize): Finds the k closest vectors to the query vectors q. Returns a SearchResult<Radius> containing distances and labels.
• assign(&mut self, q: &[Data], k: usize): Similar to search, but only returns the labels (AssignSearchResult).
• range_search(&mut self, q: &[Data], radius: Radius): Performs a search for vectors within a specific radius.
• reconstruct(&self, key: Idx, output: &mut [Data]): Retrieves a stored vector by its ID.
• reset(&mut self): Clears the entire index.
• ntotal(&self): Returns the total number of vectors currently in the index.
• d(&self): Returns the dimensionality of the vectors.

// Example usage of Index methods
let mut index = index_factory(64, "Flat", MetricType::L2).unwrap();
let data = vec![0.0_f32; 64];
index.add(&data).unwrap();

let query = vec![0.0_f32; 64];
let result = index.search(&query, 5).unwrap();