LanceDB Multimodal AI Lakehouse

LanceDB is a multimodal AI lakehouse built on the Lance columnar format, designed for fast and scalable vector search. Key capabilities include:

• Fast Vector Search: Millisecond latency for searching billions of vectors using state-of-the-art indexing.
• Comprehensive Search: Supports vector similarity search, full-text search, and SQL queries.
• Multimodal Support: Ability to store and query vectors, metadata, and multimodal data such as text, images, videos, and point clouds.
• Advanced Data Management: Features include zero-copy reads, automatic versioning (managing data versions without extra infrastructure), and GPU support for building vector indexes.

The @lancedb/lancedb JavaScript SDK provides a comprehensive API for interacting with LanceDB. The API is organized into several categories:

• Namespaces: Specialized modules like embedding and rerankers.
• Classes: Core objects for managing the database, such as Connection, Table, Index, and various Query types (e.g., VectorQuery, BooleanQuery).
• Interfaces: Configuration and result objects used across the SDK, including ClientConfig, CreateTableOptions, IndexOptions, and AddDataOptions.
• Functions: Entry points and utility functions like connect, connectNamespace, and makeArrowTable.
• Type Aliases: Data shape definitions like DataLike, SchemaLike, and RecordBatchLike.
• Enumerations: Fixed sets of values for query types and operators, such as FullTextQueryType and Operator.

LanceDB provides client SDKs for several languages. You can find the specific API references for each supported language below:

• Python: Detailed API documentation for the Python client.
• JavaScript/TypeScript: API documentation for JS/TS environments.
• Java: API documentation for the Java client.
• Rust: Official documentation hosted on docs.rs.

For full conceptual documentation, guides, and tutorials, visit docs.lancedb.com.

The Scannable class is a data source abstraction that allows data to be scanned as a stream of Arrow RecordBatches. It is used by consumers like Table.add, createTable, or mergeInsert to pull data without materializing the entire dataset in JavaScript memory.

Scannable wraps the following properties:

• schema: The Arrow schema of the data.
• numRows: The total number of rows (can be null).
• rescannable: A boolean flag indicating if the data can be scanned multiple times.

Data is transferred across the JS↔Rust boundary as Arrow IPC Stream messages. Only one batch is in flight at a time to maintain efficiency.

The Query class in @lancedb/lancedb acts as a builder for constructing complex queries against a LanceDB table. You typically obtain a Query instance by calling .query() or .search() on a Table object. Once a query is constructed, you can chain methods to refine your search criteria before executing it.

// Example of how a Query is typically initiated (based on Table documentation references)
const query = table.query().where("id > 10");

When performing full-text searches in @lancedb/lancedb, you can use the Operator enumeration to define how search terms are combined. This allows you to control whether all terms must be present or if any single term is sufficient for a match.

Available operators:

• And: Requires all specified terms to match.
• Or: Requires at least one of the specified terms to match.

In the @lancedb/lancedb Node.js SDK, the TableLike type represents data structures that can be treated as a table for operations such as ingestion or querying. It is a union type that accepts either an ArrowTable or a generic object.

type TableLike = ArrowTable | object;

The readConsistencyInterval property (number, in seconds) controls how often the client checks for updates to a table from other processes. This applies only to read operations; write operations are always consistent.

• Strong Consistency: Set readConsistencyInterval to 0. Every read will check for updates, which increases per-read latency and cost due to object storage checks.
• Eventual Consistency: Set a non-zero value (e.g., 60). The table will only be checked for updates if the specified interval has passed since the last check.
• No Consistency Check (Default): If set to None (or omitted), consistency is not checked, providing the best performance.

The general workflow for using LanceDB in Python follows these three steps:

1. Connect: Use lancedb.connect() (synchronous) or lancedb.connect_async() (asynchronous) to establish a connection to a database.
2. Manage Tables: Use the returned lancedb.DBConnection or lancedb.AsyncConnection object to create or open tables.
3. Query/Modify: Use the returned lancedb.table.Table or lancedb.AsyncTable object to perform queries or modify the data within those tables.

import lancedb

# 1. Connect
db = lancedb.connect("data/sample-lancedb")

# 2. Create/Open Table
table = db.create_table("my_table", data=[...])

# 3. Query/Modify
results = table.search("query").to_pandas()

When using an IVF PQ index, you can tune the trade-off between latency and recall using the following methods:

• nprobes(nprobes): Sets the number of partitions (clusters) to search. Higher values increase recall but increase latency. Default is 20.
• minimumNprobes(n)`` / maximumNprobes(n)`: Provides fine-grained control for queries with narrow filters.
• refineFactor(refineFactor): A multiplier used during the refine step. LanceDB performs an ANN search for limit * refineFactor results, then fetches full uncompressed values to re-rank them. This improves recall and ensures accurate distance values in the _distance column. If not called, distances are approximate based on quantized values.