DOCSEARCHbyAlgolia
Submit Docs

BCEmbedding Documentation

repository·master·Indexed 23 days ago

https://github.com/netease-youdao/bcembedding

A specialized embedding and reranking library by NetEase Youdao designed for RAG pipelines. It features EmbeddingModel for bilingual/crosslingual dual-encoder retrieval and RerankerModel for cross-encoder refinement supporting Chinese, English, Japanese, and Korean. The library includes integrations for LangChain and LlamaIndex, MTEB evaluation tools, and support for AMD Ryzen AI NPU acceleration via ONNX export and EP Context Cache.

Tokens
10.3K
Snippets
13
Records
40
Agent score
84%

What's inside BCEmbedding

  1. Overview of BCEmbedding

    master

    BCEmbedding (Bilingual and Crosslingual Embedding) is a two-stage retrieval model repository developed by NetEase Youdao, designed specifically for Retrieval Augmented Generation (RAG) applications. It consists of two primary components:

    1. EmbeddingModel: A dual-encoder model used for the first stage of retrieval. It generates semantic vectors for bilingual (Chinese and English) and crosslingual tasks. It is 'instruction-free', meaning it does not require specific task instructions to function.
    2. RerankerModel: A cross-encoder model used for the second stage of retrieval to refine results. It supports Chinese (ch), English (en), Japanese (ja), and Korean (ko). It is capable of reranking long passages (between 512 and 32,000 tokens) and provides meaningful relevance scores to filter low-quality results.

    BCEmbedding is optimized for RAG tasks such as translation, summarization, and question answering, and is used in production systems like QAnything.

  2. Compare BCEmbedding model performance metrics

    master

    BCEmbedding provides benchmark results for both Embedding and Reranker models based on MTEB (Massive Text Embedding Benchmark) and LlamaIndex RAG evaluation scenarios.

    Embedding Models

    Evaluated across 119 datasets covering Retrieval, STS, PairClassification, Classification, Reranking, and Clustering in ["en", "zh", "en-zh", "zh-en"] languages.

    • bce-embedding-base_v1: Outperforms other base-sized models and is competitive with the best large-sized models.

    Reranker Models

    Evaluated across 12 datasets in ["en", "zh", "en-zh", "zh-en"] languages.

    • bce-reranker-base_v1: Outperforms other base and large reranker models.

    RAG Performance

    In LlamaIndex-based RAG evaluation:

    • Using bce-embedding-base_v1 alone (WithoutReranker) outperforms other open-source and closed-source embedding models.
    • Using bce-reranker-base_v1 with a fixed embedding model outperforms other reranker models.
    • The combination of bce-embedding-base_v1 and bce-reranker-base_v1 achieves State-of-the-Art (SOTA) performance.

  3. Compare BCEmbedding model performance

    master

    BCEmbedding provides high-performance embedding and reranker models optimized for bilingual (en, zh) and crosslingual (en-zh, zh-en) tasks.

    Embedding Models

    bce-embedding-base_v1 is a state-of-the-art open-source model that outperforms other models of comparable size in Retrieval tasks. It uses a cls pooler and does not require specific instructions.

    Reranker Models

    bce-reranker-base_v1 outperforms other open-source reranker models in Reranking tasks.

    RAG Performance

    For Retrieval-Augmented Generation (RAG) workflows, the combination of bce-embedding-base_v1 and bce-reranker-base_v1 is considered State-of-the-Art (SOTA).

  4. BCE Reranker Performance in RAG Evaluations

    master

    This document provides a reproduction of RAG (Retrieval-Augmented Generation) evaluations originally published by LlamaIndex. It compares various embedding models against different reranker models using metrics such as hit_rate and mrr (Mean Reciprocal Rank).

    Key findings from the evaluation:

    • bce-reranker-base_v1 consistently outperforms other rerankers (Cohere, BGE) across almost all tested embedding models, including OpenAI, BGE, Cohere, JinaAI, GTE, and E5.
    • bce-embedding-base_v1 shows superior performance when used without a reranker compared to most other embedding models in the list.
    • For high-performance RAG pipelines, combining bce-embedding-base_v1 with bce-reranker-base_v1 yields the highest recorded scores in this benchmark (e.g., 96.36 hit_rate / 82.20 mrr).

  5. Evaluate Embedding Models using MTEB

    master

    You can evaluate the semantic representation of embedding models using the MTEB (Massive Text Embedding Benchmark) framework. The evaluation covers 114 datasets across tasks like Retrieval, STS, PairClassification, Classification, Reranking, and Clustering in bilingual and crosslingual settings.

    Prerequisites: Install the mteb package:

    pip install mteb==1.1.1

    Usage: Run the eval_embedding_mteb.py script. You must specify the --model_name_or_path and the recommended --pooler method.

    Pooling Methods:

    • Use --pooler mean for: jina-embeddings-v2-base-en, m3e-base, m3e-large, e5-large-v2, multilingual-e5-base, multilingual-e5-large, and gte-large.
    • Use --pooler cls for other models.
    • For jinaai/jina-embeddings-v2-base-en, you must also include the --trust_remote_code flag.

  6. Evaluate RAG performance using LlamaIndex

    master

    To evaluate how BCEmbedding performs in a Retrieval-Augmented Generation (RAG) pipeline, you can reproduce the LlamaIndex evaluation workflow.

    Prerequisites:

    1. Install required packages:
      pip install transformers==4.36.0
pip install llama-index==0.9.22
    2. Set required environment variables for OpenAI and Cohere:
   ```bash
export OPENAI_BASE_URL={openai_base_url}
export OPENAI_API_KEY={your_openai_api_key}
export COHERE_APPKEY={your_cohere_api_key}

    Execution: Note: At least two GPUs are required.

    1. Reproduce LlamaIndex Blog Benchmark:

      CUDA_VISIBLE_DEVICES=0,1 python BCEmbedding/tools/eval_rag/eval_llamaindex_reproduce.py

    2. Run Multi-Domain/Cross-lingual Benchmark: This uses a more robust dataset (CrosslingualMultiDomainsDataset) covering multiple domains like Computer Science, Physics, etc.

      CUDA_VISIBLE_DEVICES=0,1 python BCEmbedding/tools/eval_rag/eval_llamaindex_multiple_domains.py

    3. Summarize RAG Results:

      python BCEmbedding/tools/eval_rag/summarize_eval_results.py --results_dir BCEmbedding/results/rag_reproduce_results

  7. Install BCEmbedding

    master

    You can install BCEmbedding using pip or by installing from the source code. It is recommended to create a conda environment first and manually install a torch version compatible with your local CUDA to avoid compatibility issues with automatically installed versions.

    Using Conda (Recommended Setup)

    conda create --name bce python=3.10 -y
conda activate bce

    Option 1: Minimal Pip Installation

    pip install BCEmbedding==0.1.5

    Option 2: Install from Source (Recommended)

    git clone git@github.com:netease-youdao/BCEmbedding.git
cd BCEmbedding
pip install -v -e .
    conda create --name bce python=3.10 -y
conda activate bce
pip install BCEmbedding==0.1.5

  11. Evaluate Reranker models using MTEB

    master

    Evaluate reranker models using the MTEB framework. The evaluation covers 12 datasets specifically for the "Reranking" task in bilingual and cross-lingual modes.

    python BCEmbedding/tools/eval_mteb/eval_reranker_mteb.py --model_name_or_path maidalun1020/bce-reranker-base_v1

  12. Use BCEmbedding EmbeddingModel and RerankerModel

    master

    The BCEmbedding package provides high-level wrappers for embedding and reranking tasks.

    EmbeddingModel

    Use EmbeddingModel to convert a list of sentences into embeddings. By default, it uses the cls pooler.

    RerankerModel

    Use RerankerModel to calculate relevance scores or rerank passages. It provides two main methods:

    1. compute_score(sentence_pairs): Calculates scores for a list of [query, passage] pairs.
    2. rerank(query, passages): Performs reranking. This method includes advanced preprocessing for handling very long passages, which is suitable for production environments.