AlignBench

AlignBench uses an LLM-as-Judge approach, specifically utilizing gpt-4-0613 to analyze and score model responses. The evaluation process incorporates several key mechanisms to ensure reliability and interpretability:

• Single-point Scoring: Assigns a final score from 1 to 10 for each response.
• Chain-of-Thought (CoT): The judge model generates multi-dimensional analysis and reasoning before providing the final score.
• Rule Calibration: Uses detailed scoring rules (mapping score ranges to quality levels) provided in the prompt to compare model answers against the reference answer.
• Multi-dimensional Analysis: Instead of a one-size-fits-all approach, different question types use different evaluation dimensions. These dimensions and their definitions are configured in the config directory.

AlignBench is a benchmark for evaluating Chinese LLM alignment, containing 683 high-quality samples across categories like Mathematics, Logical Reasoning, and Writing Ability. The dataset is stored in data/data_release.jsonl in JSONL format. Each line represents a sample with the following schema:

• question_id (integer): Unique identifier.
• category (string): Primary category (e.g., 数学计算).
• subcategory (string): Secondary classification.
• question (string): The user query.
• reference (string): A high-quality reference answer.

Example of a mathematics category sample:

{
    "question_id": 1,
    "category": "数学计算",
    "subcategory": "初等数学",
    "question": "有一串彩珠，按“2红3绿4黄”的顺序依次排列。第600颗是什么颜色?",
    "reference": "一组"2红3绿4黄"共有9颗珠子。600除以9的商是66，余数是6。因此，第600颗珠子是在第67组的第6颗，即"2红3绿4黄"中的第6颗，也就是黄色。所以，第600颗珠子是黄色。"
}

AlignBench uses a multi-dimensional, rule-calibrated LLM-as-Judge approach (currently using GPT-4-0613) to ensure reliability and interpretability. The pipeline employs several key techniques:

• Point-wise Grading: Assigns a final rating from 1 to 10 for each model answer.
• Chain-of-Thought (CoT): The evaluator generates multi-dimensional explanations before providing a final rating to improve reasoning.
• Rule-calibrated Referencing: Uses the provided high-quality reference answer and specific grading rules (included in the prompt) to guide the score generation.
• Multi-dimensional Analysis: Tailors evaluation criteria to the specific task type. Different dimensions and their definitions are configured in the config directory.

If you use AlignBench in your research, please cite the original paper using the BibTeX entry below.

@misc{liu2023alignbench,
      title={AlignBench: Benchmarking Chinese Alignment of Large Language Models},
      author={Xiao Liu and Xuanyu Lei and Shengyuan Wang and Yue Huang and Zhuoer Feng and Bosi Wen and Jiale Cheng and Pei Ke and Yifan Xu and Weng Lam Tam and Xiaohan Zhang and Lichao Sun and Hongning Wang and Jing Zhang and Minlie Huang and Yuxiao Dong and Jie Tang},
      year={2023},
      eprint={2311.18743},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

The AlignBench leaderboard reports evaluation results for 17 Chinese-supported Large Language Models (LLMs). Results are provided using two different judging methodologies: gpt-4-0613 and CritiqueLLM.

Scores are broken down into two primary dimensions:

1. Reasoning (中文推理): Includes sub-metrics for Average, Math (数学), and Logic (逻辑).
2. Language (中文语言): Includes sub-metrics for Average, Fundamental Tasks (基本任务), Chinese Understanding (中文理解), Comprehensive Q&A (综合问答), Text Writing (文本写作), Role Playing (角色扮演), and Professional Ability (专业能力).

You can refer to the leaderboard tables in the documentation to compare model performance across these specific dimensions.