DOCSEARCHbyAlgolia
Submit Docs

Sentence Transformers

repository·main·Indexed 12 days ago

https://github.com/huggingface/sentence-transformers

A framework for computing embeddings for semantic search, retrieval, and reranking. Version 6.0.0.dev0 supports dense, sparse, and multi-vector models, providing tools for both Bi-Encoders for efficient large-scale retrieval and Cross-Encoders for high-accuracy sentence pair scoring and re-ranking. It includes support for multimodal data (text, image, audio, video) and various training loss functions such as LambdaLoss and BinaryCrossEntropyLoss.

Tokens
220K
Snippets
454
Records
786
Agent score
96%

What's inside Sentence Transformers

  1. Overview of Cross-Encoder Loss Functions

    main
    The sentence_transformers.cross_encoder.losses module provides various loss functions designed for fine-tuning cross-encoder models. The selection of a loss function is critical and depends heavily on your specific downstream task and the structure of your available training data. There is no universal loss function; you should choose one that aligns with your evaluation metrics and data format (e.g., ranking, classification, or regression).

  2. Overview of Cross-Encoder training

    main

    Training Cross-Encoder models involves several key components: initializing the Model, preparing the Dataset, choosing a Loss Function, configuring Training Arguments, setting up an Evaluator, and using a Trainer to execute the process.

    Cross-Encoders are typically used for:

    1. Reranking: Acting as a second-stage component in a search stack to rerank top candidates from a retriever. These models always have 1 output label.
    2. Pair Classification: Classifying pairs of text into multiple categories (e.g., NLI tasks like 'contradiction', 'entailment', 'neutral'). These models have multiple output labels.

  3. Overview of Unsupervised Learning for Sentence Embeddings

    main

    Unsupervised learning methods allow you to learn semantically meaningful sentence embeddings using only raw text, without the need for labeled training data (pairs of sentences or similarity scores).

    Note on Performance: Unsupervised approaches are an active research area. While they achieve acceptable performance for general sentence embedding tasks, they often perform poorly for specific semantic search tasks (finding relevant passages given a query). For optimal results on a specific corpus, it is recommended to use Domain Adaptation, which combines unsupervised learning on your target domain with existing labeled data.

  4. Common application use-cases

    main

    Sentence Transformers can be applied to various NLP and multimodal tasks. Key application areas include:

    • Computing Embeddings: Dense and Sparse embeddings.
    • Semantic Textual Similarity (STS): Measuring similarity between text pairs using Dense or Sparse methods.
    • Semantic Search: Finding relevant documents using Dense or Sparse search.
    • Retrieve & Re-Rank: Implementing two-stage pipelines (Dense only, or Sparse/Dense/Hybrid retrieval).
    • Clustering: Grouping similar embeddings.
    • Mining: Paraphrase mining and Translated Sentence mining.
    • Multimodal Search: Multilingual image search, clustering, and duplicate detection.

    Detailed examples can be found in the examples/sentence_transformer/applications directory.

  5. Evaluate models during training with sentence_transformers.sentence_transformer.evaluation

    main
    The sentence_transformers.sentence_transformer.evaluation module provides various evaluator classes designed to be used during the training process. These evaluators allow you to monitor model performance on specific tasks such as similarity, retrieval, or classification as the model weights update.

  6. Build SparseEncoder networks with sparse_encoder.modules

    main
    The sentence_transformers.sparse_encoder.modules module provides the fundamental building blocks required to construct SparseEncoder networks from scratch. These modules allow for the implementation of various sparse encoding architectures, such as SPLADE-based pooling or autoencoder-based approaches. For guidance on how to combine these modules into a complete training pipeline, refer to the Training Overview.

  7. Explore SentenceTransformers usage examples

    main

    The examples/ directory provides practical implementations of SentenceTransformers for various machine learning workflows. Use these examples to understand how to apply the library to specific tasks:

    • Applications: Implement tasks like clustering or semantic search.
    • Evaluation: Evaluate SentenceTransformer models for common downstream tasks.
    • Training: Fine-tune transformer models (e.g., BERT, RoBERTa, XLM-RoBERTa) to generate sentence embeddings.
    • Unsupervised Learning: Train sentence embedding models when labeled data is unavailable.

  8. Use SentenceTransformers for various applications

    main

    SentenceTransformers supports a wide range of NLP and multimodal tasks. Common use cases include:

    • Computing Embeddings: Generating vector representations for sentences.
    • Clustering: Grouping similar sentences together.
    • Cross-Encoders: Performing high-accuracy similarity scoring or classification by processing sentence pairs simultaneously.
    • Parallel Sentence Mining: Finding translated sentence pairs across different language corpora.
    • Paraphrase Mining: Identifying duplicate or paraphrased sentences within large datasets.
    • Semantic Search: Finding semantically similar sentences to a given query within a large collection.
    • Retrieve & Rerank: Combining a fast bi-encoder for initial retrieval with a high-precision Cross-Encoder for re-ranking results.
    • Image Search: Using image-text models to map images and text into the same vector space for cross-modal retrieval.
    • Text Summarization: Performing extractive summarization by selecting the $k$ most representative sentences from a document.

  9. Evaluate multi-vector (late-interaction) models

    main

    The sentence_transformers.multi_vector_encoder.evaluation module provides specialized evaluators designed for multi-vector (late-interaction) models. These evaluators are built to handle ragged per-token embeddings and use the MaxSim scoring mechanism to assess model performance.

    Available evaluators include:

    • MultiVectorInformationRetrievalEvaluator: For standard information retrieval tasks.
    • MultiVectorNanoBEIREvaluator: Specifically for Nano-BEIR benchmarks.
    • MultiVectorTripletEvaluator: For evaluating models based on triplet loss objectives (anchor, positive, negative).
    • MultiVectorRerankingEvaluator: For assessing reranking capabilities.
    • MultiVectorDistillationEvaluator: For evaluating distillation processes.

  10. Use Main Modules to build SentenceTransformer models

    main

    The sentence_transformers.sentence_transformer.modules package provides core building blocks for constructing custom SentenceTransformer models from scratch.

    Key modules include:

    • Pooling: Used to aggregate token-level embeddings into a single sentence-level embedding.
    • Normalize: Used to normalize embeddings (e.g., to unit length) during the model forward pass.
    • StaticEmbedding: Provides static embeddings, with specialized constructors from_model2vec and from_distillation for specific initialization patterns.

  11. Choose the right Multi-Vector-Encoder Evaluator

    main

    Multi-Vector-Encoder Evaluators are located in sentence_transformers.multi_vector_encoder.evaluation. They mirror bi-encoder evaluators but use MaxSim scoring end-to-end. Use the following guide to select an evaluator based on your task:

    TaskEvaluator
    Retrieval on common English IR benchmarksMultiVectorNanoBEIREvaluator
    Custom retrieval corpus (your own docs/queries/qrels)MultiVectorInformationRetrievalEvaluator
    Distillation from a cross-encoder teacherMultiVectorDistillationEvaluator
    Reranking a fixed candidate list per queryMultiVectorRerankingEvaluator
    Triplet accuracy (anchor score positive > negative?)MultiVectorTripletEvaluator

    Default recommendation for training: Use MultiVectorNanoBEIREvaluator on a subset of NanoBEIR datasets (e.g., ["msmarco", "nq", "fiqa2018"]) during training, and the full suite at the end of the run. It is cheap, well-calibrated, and has a stable metric key format.

  12. Evaluate SparseEncoder models during training

    main
    The sentence_transformers.sparse_encoder.evaluation module provides several evaluator classes designed to assess the performance of a SparseEncoder model during the training process. These evaluators allow you to monitor how well the sparse embeddings perform across different tasks such as retrieval, similarity, classification, and reranking.