DOCSEARCHbyAlgolia
Submit Docs

KDB.AI Samples and Course

repository·main·Indexed 41 days ago

https://github.com/kxsystems/kdbai-samples

A collection of practical examples and notebooks for KDB.AI, a vector database with time-series capabilities. Includes implementations for Retrieval Augmented Generation (RAG), multimodal search, and recommendation systems, with integrations for LlamaIndex, LangChain, LlamaParse, and Hugging Face. Features documentation on hybrid search, Temporal Similarity Search (TSS), and metadata fuzzy filtering using various distance metrics.

Tokens
18.2K
Snippets
63
Records
75
Agent score
50%

What's inside KDB.AI

  1. Overview of KDB.AI features and capabilities

    main

    KDB.AI is a vector database with time-series capabilities designed for building scalable, real-time Generative AI applications, specifically supporting Retrieval Augmented Generation (RAG) workflows. It allows for hybrid use-cases by combining unstructured vector embedding data with structured time-series datasets.

    Supported Features

    • Index Types: Flat, qFlat, IVF, IVFPQ, HNSW, and qHnsw.
    • Distance Metrics: Euclidean, Inner-Product, and Cosine.
    • Retrieval: Supports Top-N retrieval and metadata-filtered retrieval.
    • Interfaces: Provides both Python and REST interfaces.

  2. KDB.AI Course Overview and Curriculum

    main

    The KDB.AI course is a structured learning path that combines custom instructional content with existing examples from the kdbai-samples repository. The curriculum is divided into three main modules:

    1. Introduction to KDB.AI: Covers fundamental concepts including making queries and managing tables.
    2. Advanced Search Techniques: Focuses on specialized search methods such as Hybrid Search and Temporal Similarity Search (both Non-Transformed and Transformed variants).
    3. Retrieval Augmented Generation (RAG): Provides practical examples of implementing RAG workflows.

    Note that some course notebooks are symlinked or referenced from other parts of the repository to ensure users always use the most up-to-date versions of the examples.

  3. What is Non-Transformed Temporal Similarity Search (TSS)?

    main

    Non-Transformed TSS is a feature in KDB.AI that enables near real-time analysis directly on time series windows.

    Key Advantages:

    • No Embedding Overhead: Unlike traditional vector search, it does not require extracting and embedding vectors into a fixed-dimension space using a machine learning model.
    • No Indexing Overhead: It allows for analysis directly upon time series windows without the need to build a traditional search index.
    • Dynamic Window Sizes: It supports searching with query vectors of different sizes (e.g., searching for a 100-point pattern and a 1000-point pattern simultaneously) without model dependency.
    • Optimized for Fast Data: It is designed for optimized processing of fast-flowing time series datasets, making it ideal for identifying patterns and outliers in real-time.

  4. How KDB.AI Hybrid Search works

    main

    KDB.AI hybrid search increases retrieval relevancy by combining two distinct search methods:

    1. Sparse Vector Search: Uses the BM25 algorithm to find exact keyword matches. This is highly effective for finding specific terms (e.g., "12-month basis").
    2. Dense Vector Search: Uses semantic embeddings to find matches based on meaning and context, even if the exact keywords are different.

    Users can run these independently or combine them. In a hybrid search, the results from both methods are re-ranked based on user-defined weights. Higher weights for a specific index increase its influence on the final ranking of the results.

  5. Set up KDB.AI samples locally

    main

    To run the KDB.AI samples on your local machine, follow these steps:

    Prerequisites

    • A Unix-like terminal.
    • Python version 3.8 or higher.
    • pip installed.

    Installation

    1. Install dependencies: The specific requirements for each sample are listed at the beginning of their respective notebooks. To install all requirements from the repository, run:
      pip install -r requirements.txt
    2. (Optional) Optimize disk space: If you are low on disk space, you can install the CPU-only version of PyTorch to save approximately 1.5GB (as this repository does not require a GPU):
      pip install torch --index-url https://download.pytorch.org/whl/cpu

    Execution

    To view and interact with the samples, start a Jupyter notebook session:

    jupyter notebook --no-browser

    Copy the URL displayed in the terminal and paste it into your web browser.

    pip install -r requirements.txt

  6. Explore KDB.AI sample use-cases

    main

    The repository contains various notebooks demonstrating different implementation patterns:

    Getting Started

    • Python Quickstart: A basic introduction to KDB.AI APIs using Python.

    Advanced RAG & Framework Integrations

    • LlamaIndex: Includes advanced RAG, hybrid search, multimodal RAG, and multi-query retriever samples.
    • LangChain: Demonstrates RAG implementation and RAG evaluation using LangChain.
    • LlamaParse: Uses LlamaParse to extract elements from PDFs for RAG pipelines.
    • ChatGPT: Demonstrates a ChatGPT retrieval plugin integration.

    Specialized Search & Retrieval

    • Temporal Similarity Search (TSS): Includes both TSS_non_transformed and TSS_transformed for time-series search.
    • Multimodal RAG: Samples for unified text/image RAG, Voyage AI embeddings, and video RAG (using TwelveLabs or Voyage AI).
    • Hybrid & Semantic Search: Includes Hybrid Search (dense + sparse), Document Search (semantic search on PDFs), and Multi-Index Search for multimodal retrieval.
    • Filtering: Demonstrates Metadata Filtering and Fuzzy Filtering on metadata columns.

    Industry Specific & Data Patterns

    • Recommendation Systems: Music recommendation using Spotify data.
    • Pattern Matching: Sensor data pattern matching.
    • Computer Vision: Image search on Brain MRI scans.
    • Sentiment Analysis: Analysis of Disneyland Resort reviews.

  8. Install dependencies for LlamaIndex and KDB.AI Hybrid Search

    main

    To use the hybrid search example with LlamaIndex and KDB.AI, install the following Python packages using %pip:

    • llama-index and its specialized integrations: llama-index-embeddings-huggingface, llama-index-llms-openai, llama-index-readers-file, and llama-index-vector-stores-kdbai.
    • Core utilities: kdbai_client, langchain-text-splitters, and pandas.
    %pip install llama-index llama-index-embeddings-huggingface llama-index-llms-openai llama-index-readers-file llama-index-vector-stores-kdbai
%pip install kdbai_client langchain-text-splitters pandas

  9. What is Transformed Temporal Similarity Search (TSS)?

    main
    Transformed Temporal Similarity Search (TSS) is a compression model designed to dimensionally reduce time series windows by more than 99%. It compresses time series windows containing thousands of data points into significantly smaller dimensions (e.g., reducing 1000 points to 8 dimensions) while maintaining the integrity of the original data's shape. These compressed windows are stored as vector embeddings in KDB.AI, enabling efficient vector searches on large time series datasets. This allows for searching over data with different sample rates, provided the windows being searched for and the windows being searched over represent the same time frame.

  10. What is the qFlat index in KDB.AI?

    main

    The qFlat index is a vector index choice in KDB.AI designed for efficient semantic search. Unlike standard in-memory indexes, qFlat is stored on-disk and memory-mapped as required.

    Key Characteristics:

    • Low Footprint: Data inserts have negligible memory and CPU impact.
    • Scalability: The index can grow as long as disk space is available, supporting datasets with up to 1,000,000 vectors.
    • Use Case: Ideal for memory-constrained environments such as edge devices.
    • Compatibility: It supports the same API options as the existing Flat index.

  11. Use fuzzy filtering on metadata columns

    main

    KDB.AI supports fuzzy filtering to handle typos or spelling variations in metadata. This is implemented using a special filter syntax: ['fuzzy', 'column_name', [[query_term, max_distance, metric]]].

    • query_term: The term to match against.
    • max_distance: The maximum allowed distance (integer).
    • metric (optional): The distance algorithm to use. Defaults to Levenshtein.

    Supported distance metrics:

    • Levenshtein (default)
    • Damerau-Levenshtein
    • Hamming
    • Indel
    • Jaro
    • JaroWinkler
    • Longest Common Subsequence
    • Optimal String Alignment (OSA)
    • Prefix
    • Postfix
    # Default Levenshtein fuzzy filter
# Matches 'Director' even with typos, max distance of 2
print(table.search(vectors=query_vector, n=3, filter=[['fuzzy','Director',[['Goerge Lucas', 2]]]]))

# Fuzzy filter with a specific metric (OSA)
print(table.search(vectors=query_vector, n=3, filter=[['fuzzy','Genre',[['sceince fictoin', 2, 'osa']]]]))

# Fuzzy filter combined with other filters (within + damerau_levenshtein)
print(table.search(
    vectors=query_vector, 
    n=3, 
    filter=[("within", "ReleaseYear", [2000, 2010]), ['fuzzy', 'Director', [['Peter Jacksen', 1, 'damerau_levenshtein']]]]
))

  12. How Transformed Temporal Similarity Search (TSS) works

    main

    Transformed Temporal Similarity Search (TSS) is a highly optimized method for time series analytics in KDB.AI. It allows users to store high-dimensional time series windows (e.g., 100 data points) and automatically compresses them into a much smaller number of dimensions (e.g., 8 dimensions) for the backend search index. This compression significantly reduces the memory footprint and accelerates similarity searches while maintaining satisfactory pattern matching accuracy.

    To use it, you must set the column type to tsc in the embedding_configurations when creating your table and specify the target compressed dimensions in the index params.

    For more details, see the official documentation: