repository·main·Indexed 25 days ago
https://github.com/qdrant/fastembedA lightweight, high-performance Python library for generating text, image, and multimodal embeddings using the ONNX Runtime. FastEmbed supports dense, sparse, and late interaction (ColBERT) embeddings, as well as document reranking via TextCrossEncoder. It is designed for efficiency in serverless environments and provides optional GPU acceleration via the fastembed-gpu package, including support for multi-GPU setups and integration with the qdrant-client.
To use GPU acceleration, install fastembed-gpu and specify the CUDAExecutionProvider in the providers argument when initializing a model.
pip install fastembed-gpufrom fastembed import TextEmbedding
embedding_model = TextEmbedding(
model_name="BAAI/bge-small-en-v1.5",
providers=["CUDAExecutionProvider"]
)To use FastEmbed directly within the Qdrant client for seamless embedding generation during ingestion and querying, install the qdrant-client with the [fastembed] extra.
Note: If you are using zsh, you may need to wrap the installation command in quotes.
pip install qdrant-client[fastembed]
# or for zsh users:
pip install 'qdrant-client[fastembed]'Install the lightweight FastEmbed library using pip. You can choose between the standard CPU version or the GPU-supported version.
CPU version:
pip install fastembedGPU version:
pip install fastembed-gpuYou can integrate FastEmbed directly into the qdrant-client workflow. First, install the integration: pip install qdrant-client[fastembed] (or [fastembed-gpu]). This allows you to pass a model_name directly into models.Document and use client.upload_collection with documents to automate embedding generation.
from qdrant_client import QdrantClient, models
client = QdrantClient("localhost", port=6333)
model_name = "sentence-transformers/all-MiniLM-L6-v2"
payload = [
{"document": "Qdrant has Langchain integrations", "source": "Langchain-docs"},
{"document": "Qdrant also has Llama Index integrations", "source": "LlamaIndex-docs"},
]
docs = [models.Document(text=data["document"], model=model_name) for data in payload]
ids = [42, 2]
client.create_collection(
"demo_collection",
vectors_config=models.VectorParams(
size=client.get_embedding_size(model_name),
distance=models.Distance.COSINE
)
)
client.upload_collection(
collection_name="demo_collection",
vectors=docs,
ids=ids,
payload=payload,
)
search_result = client.query_points(
collection_name="demo_collection",
query=models.Document(text="This is a query document", model=model_name)
).pointsBinary Quantization can significantly improve retrieval latency (up to 20x), but may impact accuracy. To mitigate this, use the following strategy:
limit using the binary index. For example, if limit=10 and oversampling=5.0, Qdrant retrieves 50 candidates.rescore=True in QuantizationSearchParams. This tells Qdrant to take the candidates found via binary search and re-rank them using their original high-precision vector values. This combination typically yields higher accuracy than binary search alone.Use the TextEmbedding class to embed a list of strings. The .embed() method returns a generator of vectors, which is memory-efficient for large datasets. You can convert this generator to a list or a NumPy array.
import numpy as np
from fastembed import TextEmbedding
documents: list[str] = [
"This is built to be faster and lighter than other embedding libraries e.g. Transformers, Sentence-Transformers, etc.",
"fastembed is supported by and maintained by Qdrant.",
]
# This triggers model download and initialization
embedding_model = TextEmbedding()
# Returns a generator
embeddings_generator = embedding_model.embed(documents)
# Convert to list
embeddings_list = list(embeddings_generator)
# Convert to numpy array
embeddings_array = np.array(list(embedding_model.embed(documents)))To build a Retrieval-Augmented Generation (RAG) system using FastEmbed, install the necessary packages including fastembed, qdrant-client, datasets, and the Hugging Face ecosystem for LLM support.
!pip install -U fastembed datasets qdrant-client peft transformers accelerate bitsandbytes -qqWhen using models for retrieval tasks (like BGE), input texts should follow specific prefixing conventions to ensure optimal performance.
query: or passage: .query: prefix.You can export a Hugging Face model to ONNX and apply optimizations using optimum. This process involves exporting the model, applying an optimization configuration (e.g., O4), and saving the optimized model and tokenizer to a directory.
Steps:
ORTModelForFeatureExtraction.from_pretrained(model_id, export=True).ORTOptimizer with an AutoOptimizationConfig (like O4()) to optimize the model.from optimum.onnxruntime import AutoOptimizationConfig, ORTModelForFeatureExtraction, ORTOptimizer
from transformers import AutoTokenizer
from pathlib import Path
model_id = "BAAI/bge-small-en-v1.5"
save_dir = Path(f"fast-{model_id.split('/')[1]}")
save_dir.mkdir(parents=True, exist_ok=True)
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = ORTModelForFeatureExtraction.from_pretrained(model_id, export=True)
# Optimization
optimization_config = AutoOptimizationConfig.O4()
optimizer = ORTOptimizer.from_pretrained(model)
optimizer.optimize(
save_dir=save_dir,
optimization_config=optimization_config,
use_external_data_format=True
)
# Save results
model = ORTModelForFeatureExtraction.from_pretrained(save_dir)
tokenizer.save_pretrained(save_dir)When performing retrieval tasks (queries vs. passages) using the default model, you should prepend specific labels to your strings to optimize performance:
query: to the beginning of the string.passage: to the beginning of the string.To use GPU acceleration, install the fastembed-gpu package.
CRITICAL: fastembed-gpu and fastembed cannot coexist in the same environment. Similarly, onnxruntime-gpu and onnxruntime cannot coexist. If you have the standard version installed, you must uninstall it before installing the GPU version.
!pip install fastembed-gpu