Milvus Vector Database

Milvus is a high-performance, distributed vector database designed for scaling AI applications. It manages unstructured data (text, images, multi-modal) by organizing it into vectors and allows for efficient similarity searches.

Key architectural features include:

• Distributed & K8s-native: Separates compute and storage for horizontal scaling.
• Hardware Acceleration: Supports CPU/GPU acceleration for high-performance vector search.
• Deployment Modes:
  • Standalone: Single machine deployment.
  • Milvus Lite: A lightweight version for local Python development.
  • Zilliz Cloud: A fully managed service (Serverless, Dedicated, or BYOC options).
• Search Capabilities: Supports dense vector semantic search, sparse vector full-text search (BM25, SPLADE, BGE-M3), and hybrid search.

Milvus is an open-source vector database designed for AI applications and vector similarity search, accelerating unstructured data retrieval.

Key characteristics:

• Cloud-Native Architecture: Milvus 2.0 uses a storage-and-compute separation design where components are stateless, enhancing elasticity and flexibility.
• High Performance: Capable of millisecond-level searches across trillion-scale vector datasets.
• Hybrid Querying: Supports multi-vector search (up to 10 vector fields per Collection) with reranking strategies like Reciprocal Rank Fusion (RRF) and Weighted Scoring.
• Lambda Architecture: Supports both stream and batch processing for data storage, balancing real-time efficiency with batch processing throughput.
• Scalability: Provides component-level scalability for precise resource expansion.

Milvus integrates with a wide range of AI development tools and data pipelines:

• AI Frameworks: LangChain, LlamaIndex, OpenAI, and HuggingFace.
• Embedding Utilities: Use pymilvus[model] to transform unstructured data into vector embeddings and leverage reranking models.
• Administration & Debugging:
  • Attu: GUI-based administration.
  • Birdwatcher: System debugging.
  • Prometheus/Grafana: Monitoring.
• Data Management:
  • Milvus CDC: Data synchronization.
  • VTS: Data migration.
• Data Connectors: Spark, Kafka, Fivetran, and Airbyte.

As of the current design specification, client-side telemetry is implemented in the Go SDK only.

While protobuf stubs may exist in other language implementations, the following SDKs do not currently implement the client-side telemetry logic:

• pymilvus (Python)
• Java SDK
• Node.js SDK
• Rust SDK

When evaluating client coverage for telemetry, only Go SDK clients should be considered supported.

Milvus snapshots allow you to create point-in-time copies of collections. This feature is used for data backup, versioning, and restoration. A snapshot captures the complete state of a collection at a specific timestamp, including:

• Vector data (binlogs)
• Metadata (schema, partition information, and collection properties)
• Indexes (index metadata and file paths)
• Deletion logs (deltalogs)

An External Table (also known as an External Collection) is a specialized data collection in Milvus that allows you to query data residing in external storage systems like S3, Iceberg, or Delta Lake without importing or copying the data into Milvus local storage.

It acts as a query layer over your existing data lake, providing a unified interface for vector similarity search, scalar filtering, and hybrid search while maintaining compatibility with standard Milvus query APIs.

Channel Exclusive Mode is a load balancing strategy introduced in Milvus 2.6.0 for the QueryCoord service. Unlike traditional balancing that distributes channels and segments across QueryNodes without strict isolation, Channel Exclusive Mode assigns each channel exclusively to a dedicated set of Read-Write (RW) nodes.

Key Benefits:

• Predictable Performance: Reduces resource contention by preventing multiple channels from competing for the same QueryNode resources.
• Improved Isolation: Ensures channels run on separate node groups to prevent interference.
• Dynamic Adaptation: The system automatically initializes exclusive metadata when conditions are met and falls back to traditional score-based balancing if exclusivity constraints (like insufficient nodes) cannot be satisfied.

The QueryView versioning tracks how data is distributed across nodes at the ShardOnReplica level. The version is a tuple (D, Q) ordered lexicographically:

• D (Data Dimension): Increases when the underlying DataVersion changes (storage-level changes like new flushes).
• Q (Query Dimension): Increases when data is redistributed without changing the underlying data (load-level changes like Balancing or Recovery).

Lifecycle Behavior:

• When D increases (new data arrives), Q is reset to 1 because the new data must be redistributed.
• An increase in Q represents pure redistribution (e.g., moving a segment from Node1 to Node2 during a balance operation) and does not change the DataVersion.
• Rollbacks are not allowed; the version number can only increase non-strictly.