SimilaritySearchKit Documentation

To perform semantic search, follow these steps:

1. Import SimilaritySearchKit.
2. Initialize a SimilarityIndex with an embedding model (conforming to EmbeddingProtocol) and a distance metric (conforming to DistanceMetricProtocol).
3. Use addItem(id:text:metadata:) to index your content.
4. Use search(_:) to query the index. It returns an array of SearchResult objects containing the item id, a similarity score, and the original metadata.

import SimilaritySearchKit

// 1. Initialize
let similarityIndex = await SimilarityIndex(
    model: NativeEmbeddings(),
    metric: CosineSimilarity()
)

// 2. Add items
await similarityIndex.addItem(
    id: "id1", 
    text: "Metal was released in June 2014.", 
    metadata: ["source": "example.pdf"]
)

// 3. Search
let results = await similarityIndex.search("When was metal released?")
print(results)
// Output: [SearchResult(id: "id1", score: 0.86216, metadata: ["source": "example.pdf"])]

You can override the core components of SimilarityIndex by providing your own implementations of the following protocols:

• EmbeddingsProtocol: Convert a string into a [Float] embedding.
  • func encode(sentence: String) async -> [Float]?
• DistanceMetricProtocol: Calculate nearest neighbors.
  • func findNearest(for queryEmbedding: [Float], in neighborEmbeddings: [[Float]], resultsCount: Int) -> [(Float, Int)]
• TextSplitterProtocol: Chunk long documents.
  • func split(text: String, chunkSize: Int, overlapSize: Int) -> ([String], [[String]]?)
• TokenizerProtocol: Handle custom tokenization.
  • func tokenize(text: String) -> [String]
  • func detokenize(tokens: [String]) -> String
• VectorStoreProtocol: Manage persistence (default is JSON).
  • func saveIndex(items: [IndexItem], to url: URL, as name: String) throws -> URL
  • func loadIndex(from url: URL) throws -> [IndexItem]
  • func listIndexes(at url: URL) -> [URL]