FlagEmbedding (BGE)

BGE-VL is a series of multimodal retrieval models trained using the MegaPairs data synthesis method. They are designed for universal multimodal retrieval and achieve state-of-the-art performance on zero-shot composed image retrieval (CIR) and the Massive Multimodal Embedding Benchmark (MMEB).

Model Families:

• BGE-VL-CLIP: Includes BGE-VL-base and BGE-VL-large. These are efficient models based on the CLIP architecture.
• BGE-VL-MLLM: Includes BGE-VL-MLLM-S1 and BGE-VL-MLLM-S2. These leverage Multimodal Large Language Models for enhanced retrieval performance, particularly in composed image retrieval tasks.

FlagEmbedding (BGE) is a one-stop retrieval toolkit designed for search and Retrieval-Augmented Generation (RAG) workflows. It provides tools for three primary stages of the retrieval pipeline:

1. Inference: Using embedding models and rerankers for retrieval tasks.
2. Evaluation: Assessing the performance of retrieval models.
3. Fine-tuning: Training and optimizing embedding models and rerankers for specific datasets.

BGE (BAAI General Embedding) is a toolkit for Retrieval-Augmented Generation (RAG) and search. The project is organized into several functional areas:

• Inference: Includes Embedder and Reranker modules.
• Fine-tuning: Includes Embedder and Reranker fine-tuning workflows.
• Evaluation: Tools for assessing model performance.
• Datasets & Research: Specialized datasets and research implementations (e.g., BGE-M3, Visualized-BGE).

The repository contains research for extending the context window of Large Language Models (LLMs):

• LongLLM QLoRA: Extends the context length of models like Llama-3-8B-Instruct (e.g., from 8K to 80K) using efficient QLoRA fine-tuning. This method preserves short-context capabilities while enabling long-context understanding.
• Activation Beacon: A method to condense LLM raw activations into compact forms. This allows the model to perceive much longer contexts within a limited context window, providing an effective and low-cost way to extend context length.

The FlagEmbedding tutorial curriculum covers the full lifecycle of retrieval-augmented generation and embedding tasks. The roadmap includes:

• Embedding: Using BGE and Sentence Transformers, including BGE-M3.
• Metrics: Similarity functions and evaluation metrics for searching.
• Indexing: Practical indexing using libraries like Faiss (including GPU usage, index types, and quantizers) and Milvus.
• Evaluation: Full pipelines for evaluating embedding models using benchmarks like MTEB, C-MTEB, MSMARCO, BEIR, MIRACL, and MLDR.
• Reranking: Using accurate models to rerank candidates retrieved by efficient retrievers to balance accuracy and efficiency.
• RAG (Retrieval-Augmented Generation): Implementing RAG from scratch or using popular frameworks like LangChain and LlamaIndex.

A BGE-Reranker (or cross-encoder) takes a question and a document as input and directly outputs a similarity score, rather than generating embeddings.

Because cross-encoders are more computationally expensive than embedding models, the recommended pattern is to use a two-stage retrieval process:

1. Retrieval Stage: Use a BGE embedding model to retrieve the top-k (e.g., top 100) relevant documents.
2. Reranking Stage: Use a BGE-Reranker to re-rank those top-k documents to identify the most accurate final results (e.g., top 3).

The MVRB (Massive Visualized IR Benchmark) is designed to evaluate retrieval tasks centered on screenshots. It consists of four meta-tasks:

1. Screenshot Retrieval (SR)
2. Composed Screenshot Retrieval (CSR)
3. Screenshot QA (SQA)
4. Open-Vocabulary Classification (OVC)

UniSE-MLLM has demonstrated state-of-the-art performance on this benchmark compared to OCR+Text Retrievers, General Multimodal Retrievers, and Screenshot Document Retrievers.

Vis-IR (Visualized Information Retrieval) is an emerging IR paradigm where multimodal information (text, images, tables, charts) is jointly represented by a unified visual format called Screenshots.

To support this paradigm, the project provides:

• VIRA (Vis-IR Aggregation): A large-scale dataset of captioned and question-answer screenshots.
• UniSE (Universal Screenshot Embeddings): A family of retrieval models that enable screenshots to query or be queried across arbitrary data modalities.
• MVRB (Massive Visualized IR Benchmark): A benchmark covering Screenshot Retrieval (SR), Composed Screenshot Retrieval (CSR), Screenshot QA (SQA), and Open-Vocabulary Classification (OVC).

BGE-Reasoner is an end-to-end reasoning-intensive information retrieval (IR) framework designed for complex tasks. It covers the entire retrieval pipeline through three core components:

1. BGE-Reasoner-Rewriter: Performs query rewriting to improve retrieval quality.
2. BGE-Reasoner-Embed: Handles the retrieval stage (embedding-based).
3. BGE-Reasoner-Reranker: Performs reranking to refine the retrieved results.

The framework is optimized for reasoning-intensive benchmarks like BRIGHT, achieving state-of-the-art performance by combining query rewriting, hybrid retrieval (Embedding + BM25), and multi-size reranking.

LLM Embedder is an embedding model fine-tuned based on feedback from LLMs. It is specifically designed to support Retrieval-Augmented Generation (RAG) needs, such as:

• Knowledge retrieval
• Memory retrieval
• Example retrieval
• Tool retrieval

It has been fine-tuned across six specific tasks: Question Answering, Conversational Search, Long Conversation, Long-Range Language Modeling, In-Context Learning, and Tool Learning.