Mixture-of-Agents (MoA)

AlpacaEval uses various annotator configurations (evaluators) to score model outputs. These configurations define the API provider, model, parameters, parsing functions, and prompts used during the evaluation process.

Key metrics used to compare annotators include:

• Human agreement: Correlation with human judgments.
• Price: Cost per 1000 examples.
• Time: Seconds per 1000 examples.
• Spearman/Pearson corr.: Statistical correlation with human labels.
• Bias: Tendencies like preferring longer responses or lists.

Commonly used annotator identifiers include alpaca_eval_gpt4, alpaca_eval_gpt4_fn (using OpenAI function calls), and claude_ranking.

Length-controlled AlpacaEval (LCAE) is an advancement designed to mitigate length bias in automatic evaluators. It addresses the issue where models can artificially increase win-rates by producing longer outputs.

Key benefits of LCAE include:

• Higher Correlation: Increases correlation with Chat Arena to 0.98 (up from 0.94 in AlpacaEval 2.0).
• Reduced Gameability: Decreases length gameability to ~6% (compared to ~21% in standard AlpacaEval), making it 3x harder to manipulate via prompt length.
• Predictive Modeling: Uses a Generalized Linear Model (GLM) to predict win-rates for different baselines, satisfying properties like win_rate(m,b) = 1 - win_rate(b,m) and win_rate(m,m) = 0.5.

The following datasets are released by LMSYS and are available on Hugging Face for research and evaluation purposes:

• LMSYS-Chat-1M: A large-scale real-world LLM conversation dataset.
• Chatbot Arena Conversation Dataset: Conversations from the Chatbot Arena platform.
• MT-bench Human Annotation Dataset: Human judgments used for MT-bench evaluation.

https://huggingface.co/datasets/lmsys/lmsys-chat-1m
https://huggingface.co/datasets/lmsys/chatbot_arena_conversations
https://huggingface.co/datasets/lmsys/mt_bench_human_judgments

To integrate Mixture-of-Agents (MoA) into your own applications, you can use the minimal implementation provided in moa.py. This example uses 2 layers and 4 LLMs.

Prerequisites:

1. Install the Together Python library: pip install together
2. Obtain a Together API Key.
3. Export your API key to the environment variable TOGETHER_API_KEY.

Execution: Run the script using:

python moa.py

pip install together
export TOGETHER_API_KEY=<your_key>
python moa.py

The test_embedding playground provides scripts to demonstrate common machine learning tasks using embeddings:

• Text Similarity: Evaluate how similar two pieces of text are using test_sentence_similarity.py.
• Classification: Build and test custom classifiers using test_classification.py.
• Semantic Search: Perform searches based on relative text meaning using test_semantic_search.py.

Prerequisites:

• Download the required dataset from Kaggle (Amazon Fine Food Reviews).
• An OpenAI API key is required for comparison tasks.

You can install the stable release or the nightly version of AlpacaEval using pip.

Stable release:

pip install alpaca-eval

Nightly version:

pip install git+https://github.com/tatsu-lab/alpaca_eval

To use LightLLM as the worker implementation, replace the standard fastchat.serve.model_worker with fastchat.serve.lightllm_worker. Other FastChat components like the controller, Gradio web server, and OpenAI API server remain unchanged.

When launching, you must provide the --max_total_token_num argument. Refer to the LightLLM documentation to calculate the appropriate value for this argument.

python3 -m fastchat.serve.lightllm_worker --model-path lmsys/vicuna-7b-v1.5 --tokenizer_mode "auto" --max_total_token_num 154000

Navigate to the monitor directory within FastChat and run the clean_battle_data.py script to clean battle data.

cd ~/FastChat/fastchat/serve/monitor
python3 clean_battle_data.py

To redirect LangChain from the official OpenAI API to your local FastChat server, set the OPENAI_API_BASE and OPENAI_API_KEY environment variables. Since FastChat is local, the API key can be set to EMPTY.

export OPENAI_API_BASE=http://localhost:8000/v1
export OPENAI_API_KEY=EMPTY

To install FastChat, update your system packages, install tmux and htop, set up an Anaconda environment with Python 3.9, and install the FastChat package in editable mode from the cloned repository.

sudo apt update
sudo apt install tmux htop

wget https://repo.anaconda.com/archive/Anaconda3-2022.10-Linux-x86_64.sh
bash Anaconda3-2022.10-Linux-x86_64.sh

conda create -n fastchat python=3.9
conda activate fastchat

git clone https://github.com/lm-sys/FastChat.git
cd FastChat
pip3 install -e .

When creating evaluation sets, you should consider data volume and data quality. You can use the following notebooks to perform statistical analysis (such as paired t-tests) to determine how many samples are needed to distinguish between models and which datasets provide the most statistical power.

Available Notebooks:

• Analyzing an eval set: Google Colab Notebook

Key Findings from Analysis:

• Sample Size: Most models can be distinguished with as few as 50-150 samples.
• Data Quality: Certain datasets, like self-instruct, may yield less statistical power and could potentially be removed from evaluation sets.