lighteval

LightEval organizes metrics into different levels of granularity depending on whether the score is calculated per individual sample or across the entire dataset (corpus).

• SampleLevelMetric: Metrics that compute a score for each individual sample in the evaluation set (e.g., ExactMatches, ROUGE, F1_score).
• CorpusLevelMetric: Metrics that aggregate results across the entire corpus to produce a single global score (e.g., CorpusLevelF1Score, CorpusLevelPerplexityMetric).
• MetricGrouping: A way to group multiple metrics together. There are specific groupings for corpus-level (CorpusLevelMetricGrouping) and sample-level (SampleLevelMetricGrouping) metrics.

The main JSON results file is organized into several sections:

General Configuration & Summary

• config_general: Metadata including lighteval_sha, model_name, model_sha, model_size, and timing information (start_time, end_time, total_evaluation_time_secondes).
• summary_general: General statistics for the entire run (e.g., total padded/non-padded samples).

Task-Specific Information

• config_tasks: Detailed configuration for each task, including prompt_function, metric definitions (name, higher_is_better, etc.), and few_shots_select settings.
• summary_tasks: Task-specific metadata and hashes (e.g., hash_examples, hash_full_prompts).
• versions: Version information for tasks and datasets.

Results

• results: The actual evaluation metrics and scores for each task (e.g., em, maj@8).

SGLang supports distributing models across multiple GPUs using two methods:

1. Tensor Parallelism (tp_size): Splits the model itself across multiple GPUs. Use this when a single model is too large for one GPU.
2. Data Parallelism (dp_size): Runs multiple copies of the model on different GPUs to process different data batches in parallel. Use this when the model fits on a single GPU and you want to speed up evaluation.

Set these by adding tp_size=N or dp_size=N to your model_args.

The detailed evaluation files (Parquet) contain three primary columns for each sample:

• __doc__: The document used for evaluation, including the gold reference, few-shot examples, and other task hyperparameters.
• __model_response__: Contains the model's generations, log probabilities, and the original input sent to the model.
• __metric__: The calculated metric value for that specific sample.

When creating multilingual tasks, you must choose a formulation that defines how the prompt is presented and how the answer is expected. There are three primary types:

1. Multiple Choice Formulation (MCF): Used for standard multiple choice questions where the model selects from lettered options (e.g., A, B, C, D).
2. Classification Formulation (CF): Used for tasks where the model generates the answer text directly without choosing from a list.
3. Hybrid Formulation: Used for tasks that present choices but expect the model to provide the full text of the correct answer.

# Multiple Choice Formulation (MCF)
MCFFormulation()

# Classification Formulation (CF)
CFFormulation()

# Hybrid Formulation
HybridFormulation()

LightEval provides several specialized loggers to capture different aspects of the evaluation process. Depending on your needs, you can use:

• GeneralConfigLogger: Logs general configuration settings.
• DetailsLogger: Logs detailed information about the evaluation.
• MetricsLogger: Logs evaluation metrics.
• VersionsLogger: Logs version information.
• TaskConfigLogger: Logs task-specific configurations.

Lighteval uses a caching system to speed up evaluations by storing and reusing model predictions (generations, logits, and probabilities) on disk. This prevents redundant computations when running the same evaluation multiple times or comparing different metrics on the same model outputs.

Cache Invalidation (Recreation)

A new cache is automatically generated whenever any of the following change:

• Model configuration: parameters, quantization settings, etc.
• Model weights: different revisions or checkpoints.
• Generation parameters: temperature, max_tokens, etc.

This mechanism ensures that cached results remain consistent with the specific model setup used during generation.

The Lighteval Python API is built around several core abstractions:

• EvaluationTracker: Manages logging and saving of results. It supports local saving and pushing results to the Hugging Face Hub.
• PipelineParameters: Configures the execution environment, including parallelism settings (via launcher_type) and task-specific configurations like custom_tasks_directory or max_samples.
• Model Configuration: Defines the model identity and runtime parameters (e.g., model_name, dtype). Configuration classes vary by backend (e.g., VLLMModelConfig for VLLM, or Transformers-based configs).
• Pipeline: The central orchestrator that accepts tasks, model configuration, pipeline parameters, and an evaluation tracker to execute the evaluation lifecycle.

Cached data is stored using HuggingFace datasets in a specific hierarchical structure on disk. This structure uses the model name, a hash of the model configuration, and the task name to organize files:

.cache/
└── huggingface/
    └── lighteval/
        └── predictions/
            └── {model_name}/
                └── {model_hash}/
                    └── {task_name}.parquet

• {model_name}: The model name (Hub path or local path).
• {model_hash}: A hash of the model configuration used to trigger cache invalidation if parameters change.
• {task_name}: The name of the specific evaluation task.

In LightEval, model configurations define the model and its specific parameters. You can specify these parameters in two ways:

1. model-args: Pass parameters directly via command-line arguments or configuration strings.
2. YAML files: Define parameters in a dedicated model YAML file.

For a concrete implementation example, refer to the vllm_model_config.yaml structure in the repository examples.