LLM Comparator

LLM Comparator is an interactive visualization tool and a Python library designed for qualitative, side-by-side analysis of Large Language Model (LLM) evaluation results. It allows users to compare responses from two different models (Model A and Model B) at both the individual example level and across broader data slices (e.g., by prompt category).

Key analytical capabilities include:

• Score Distribution: Visualizing win rates and quality differences across categories.
• Rationale Summary: Diving into the reasons (rationales) provided by LLM judges for specific score differences.
• Custom Functions: Discovering specific behavioral differences, such as structural patterns (e.g., use of bulleted lists) or stylistic traits (e.g., verbosity).

The library coordinates comparative evaluations through three main phases managed by the llm_comparator.comparison.run() function:

1. Judging: An LLMJudgeRunner uses a generation model to compare two model responses for every prompt and generate a verdict and explanation.
2. Bulletizing: A RationaleBulletGenerator condenses the judge's results into concise bullet points.
3. Clustering: A RationaleClusterGenerator takes the bullets, embeds them, groups them into clusters based on similarity, and generates labels for those clusters.

This workflow produces a JSON file compatible with the LLM Comparator app.

You can interact with the visualization tool directly in your browser at https://pair-code.github.io/llm-comparator/.

To analyze your own data:

1. Prepare a JSON file following the LLM Comparator schema.
2. Upload your JSON file to the web interface to begin interactive analysis.

The llm-comparator package on PyPI is a Python library used to create the JSON files required by the visualization tool.

Capabilities include:

• Generating a complete JSON file from scratch, including side-by-side LLM-based evaluations and rationale clusters, given input prompts and models.
• Performing only the rationale clustering steps if you already possess prompts and model outputs.

For a practical implementation guide, refer to the basic_demo.ipynb notebook in the repository.

You can install the library via PyPI (recommended) or by installing from source for development.

Via PyPI:

pip install llm_comparator

From Source (for contributors):

git clone https://github.com/PAIR-code/llm-comparator.git
cd llm-comparator/python
pip install -e .

It is recommended to use a Python virtual environment.

# [OPTIONAL] We recommend using a Python virtual environemnt.
python3 -m venv ~/.venv/llm_comparator
source ~/.venv/llm_comparator/bin/activate

# Install via PyPI
pip install llm_comparator

To run the LLM Comparator visualization locally, follow these steps:

1. Clone the repository:

   git clone https://github.com/PAIR-code/llm-comparator.git
   cd llm-comparator

2. Install dependencies:

   npm install

3. Build the project:

   npm run build

4. Start the local server:

   npm run serve

LLM Comparator comparisons follow a structured three-step execution model to transform raw model outputs into clustered rationales:

1. Judgement Phase: An LLMJudgeRunner evaluates the inputs to determine which model performed better and why.
2. Bulletization Phase: A RationaleBulletGenerator takes those judgements and extracts specific, granular rationale bullets.
3. Clustering Phase: A RationaleClusterGenerator takes the bullets and groups them into clusters, calculating similarity scores between them.

This pipeline allows the web application to show not just which model won, but the thematic reasons (clusters) behind the wins and losses.

To ensure robust evaluation, LLMJudgeRunner uses a technique called 'flipping' to mitigate position bias (where a judge might favor the first response it sees).

When you specify num_repeats, the runner:

1. Creates math.ceil(num_repeats * 0.5) runs where response_a is the original A and response_b is the original B (is_flipped=False).
2. Creates math.floor(num_repeats * 0.5) runs where response_a is the original B and response_b is the original A (is_flipped=True).

Example: If num_repeats=3:

• 2 runs with is_flipped=False (A vs B)
• 1 run with is_flipped=True (B vs A)

During parsing, if a run was flipped, the score is multiplied by -1.0 to normalize the direction of the comparison.

To support multimodal comparisons (e.g., text and images), the input_text, output_text_a, and output_text_b fields in an Example can accept an array of SequenceChunk objects instead of a simple string.

Each chunk must specify its type using the SequenceChunkType enum:

• text: For standard string data.
• image_byte: For raw image data (base64 encoded JPEG).

export enum SequenceChunkType {
  TEXT = 'text',
  IMAGE_BYTE = 'image_byte',
}

export interface SequenceChunk {
  data: string;
  type: SequenceChunkType;
}