Deep Lake Documentation

Deep Lake is a database optimized for AI, designed for storing and searching data plus vectors for LLM applications, and managing datasets for deep learning model training.

Key features include:

• Multi-Cloud Support: Upload, download, and stream datasets to/from S3, Azure, GCP, Activeloop cloud, local storage, or in-memory storage (including S3-compatible storage like MinIO).
• Native Compression & Lazy Indexing: Store media (images, audio, video) in native compression with NumPy-like lazy indexing for efficient memory usage.
• Deep Learning Dataloaders: Built-in dataloaders for PyTorch and TensorFlow with automatic dataset shuffling.
• Integrations: Native support for LangChain, LlamaIndex, Weights & Biases, MMDetection, and MMSegmentation.
• Instant Visualization: Support for visualizing datasets with bounding boxes, masks, and annotations via the Deep Lake App.

Deep Lake uses different column classes depending on the required access level (read-write vs. read-only) and whether you are interacting with the data itself or the schema definition.

Deep Lake provides three dataset classes depending on your required access level:

1. Dataset: Full read-write access. Use this for creating, modifying, appending, or deleting data and managing branches/tags.
2. ReadOnlyDataset: Read-only access. Use this to prevent accidental modifications while still accessing all data, metadata, and version history. It is returned by deeplake.open_read_only().
3. DatasetView: A lightweight, read-only view of query results. It is optimized for ML framework integration (PyTorch/TensorFlow) and returned by .query() or tag operations.

The Deep Lake Python API is organized into several core components for managing multi-modal AI data:

• Dataset Classes: Manage data storage and access using Dataset, ReadOnlyDataset, and DatasetView.
• Column Classes: Interact with data columns using Column and ColumnView.
• Types: Define data using basic numeric types or ML-optimized types.
• Schemas: Use pre-built schema templates for common data structures.
• Query Language: Use TQL (Deep Lake Query Language) for data operations.
• Metadata: Manage metadata at both the dataset and column levels.
• Version Control: Utilize versioning, history, branches, and tags for dataset management.

To use Deep Lake with MMDetection, you must install specific versions of PyTorch and MMCV, and then install MMDetection from the dev-2.x branch, as the Deep Lake integration currently supports MMDetection 2.x.

python -m pip install torch==1.12.0+cu116 torchvision==0.13.0+cu116 -f https://download.pytorch.org/whl/torch_stable.html
python -m pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu116/torch1.12.0/index.html

git clone  -b dev-2.x https://github.com/open-mmlab/mmdetection.git
cd mmdetection
python3 -m pip install -e .

You can create a new dataset or open an existing one using the following methods:

• deeplake.create(path): Creates a new dataset at the specified path.
• deeplake.open(path): Opens an existing dataset with full read-write access.
• deeplake.open_read_only(path): Opens an existing dataset with read-only access.
• deeplake.exists(path): Checks if a dataset exists at the given path.

To perform similarity searches, you can use proprietary models (like OpenAI's text-embedding-3-large) or open-source models from the MTEB leaderboard. You must generate numerical embeddings for your text data and store them in a Deep Lake column with an Embedding type. This allows for cosine similarity comparisons during queries.

import openai
import os

# Set your API key
os.environ["OPENAI_API_KEY"] = "your_api_key"

def embedding_function(texts, model="text-embedding-3-large"):
    if isinstance(texts, str):
        texts = [texts]

    texts = [t.replace("\n", " ") for t in texts]
    return [data.embedding for data in openai.embeddings.create(input = texts, model=model).data]

To use ColPali for end-to-end neural multi-modal search, install the required dependencies via pip:

!pip install colpali-engine accelerate

Use specialized Deep Lake types instead of generic arrays to enable efficient compression, storage, and searching:

• deeplake.types.Image(): Enables efficient image compression/decompression.
• deeplake.types.Video(): Supports H264 compression.
• deeplake.types.Mesh(): Supports STL and PLY formats.
• deeplake.types.Text(): Enables efficient text search and indexing.
• deeplake.types.Embedding(): Optimized for vector similarity search.
• deeplake.types.Link(): Use this for images already in cloud storage to avoid data duplication by passing the URL instead of the raw bytes.