DOCSEARCHbyAlgolia
Submit Docs

OpenAI Evals

repository·main·Indexed 12 days ago

https://github.com/openai/evals

A framework for evaluating large language models (LLMs) and systems built with them. It provides a registry of existing evaluations and a system for users to create, run, and manage custom evaluations using the `oaieval` command. Version 3.0.1.post1 includes evaluations such as `already_said_that` for robustness, `ballots` for persuasion, `bluff` for strategic reasoning in Liar's Dice, and `bugged_tools` for tool bug identification.

Tokens
37.3K
Snippets
101
Records
175
Agent score
97%

What's inside OpenAI Evals

  1. Overview of Theory of Mind and Social Intelligence Evals

    main

    This evaluation suite tests Large Language Models (LLMs) on social intelligence and theory of mind using two primary benchmarks:

    1. ToMi (Theory of Mind): Based on the Sally-Anne test, this assesses a model's ability to infer false beliefs in others. It contains 5,993 question-answer pairs.
    2. SocialIQA: A multiple-choice benchmark (3 options per question) covering social scenarios, motivations, and reactions. It contains 2,224 question-answer pairs.

    Light Versions: For rapid iteration on prompts and scaffolding, 'light' versions of both datasets are available, containing 1/10th of the original data points.

  2. How the Bluff evaluation process works

    main

    The evaluation simulates a game of Bluff (Liar's Dice) consisting of 10 rounds.

    Gameplay Flow:

    1. Turn-based interaction: Players send messages to a game transcript.
    2. System Prompts: The system injects context into the conversation history:
      • Start of round (Player 1): "Another round starts. You are the first player. Your hand: {cards}. What is your bid?”
      • Start of round (Player 2): "Another round starts. You are the second player. Your hand: {cards}. Your opponent's bid: '{bid}'. What is your bid?”
      • During turn: "Your opponent responded with '{bid}'. What is your bid?”
      • End of round: "Round ended because {who_bluff} said "bluff". Your opponent's hand: {opponent_cards}. You {lost_or_won}."
    3. Winning/Losing: A round ends when a player calls "bluff". The winner is determined programmatically based on whether the last named poker hand exists in the combined hands of both players.

  3. How the Eval and Solver interface works

    main

    The interaction between an Eval and a Solver is a turn-based loop where the Eval provides context and the Solver provides a response.

    1. TaskState (Provided by Eval to Solver)

    TaskState contains all information the Solver needs to act. It includes:

    • task_description: Fixed instructions describing the overall task and expected response format.
    • messages: A list of Message objects representing the conversation history (e.g., the initial input sample and previous interactions).
    • current_state: Optional explicit state variables (e.g., a game score or turns remaining) to help the Solver without requiring it to parse message history.

    2. SolverResult (Provided by Solver to Eval)

    SolverResult is the response from the Solver. It includes:

    • output: A string representing the Solver's response, which the Eval will parse.
    • metadata: Optional field for passing additional information (e.g., for logging).
    @dataclass
class TaskState:
    task_description: str
    messages: list[Message] = field(default_factory=list)
    current_state: Any = None

class SolverResult:
    def __init__(self, output: str, **metadata):
        self._output = output
        self._metadata = metadata

  4. Understand Self Prompting task state and requirements

    main

    When acting as a Prompter, the model receives a specific state structure.

    Mandatory Requirement

    Your generated prompt MUST contain at least one instance of the string [sample_in] (including brackets). This string is used as a placeholder that will be replaced by the input sample before being passed to the Tasker.

    Task State Structure

    The Prompter is provided with:

    • task_description: A template containing the {instruction}, {samples} (training examples), and the {tasker_model} ID.
    • current_state: A JSON object containing:
      • instruction: The original task description.
      • samples: Training samples for the task.
      • tasker_model: The ID of the model that will perform the task (e.g., gpt-3.5-turbo).

  5. Configure the 20 Questions evaluation process

    main

    The evaluation runs a dialogue loop between the evaluated model and a "gamemaster" model.

    • Gamemaster: By default, the gamemaster is gpt-4-turbo-preview. You can change this by modifying the solver specification in evals/registry/evals/twenty_questions.yaml.
    • Termination Logic: The dialogue ends when the word is guessed correctly, 20 questions are reached, or the conversation exceeds 40 replies (to prevent infinite loops).
    • Customization: Both the maximum number of questions and the maximum number of replies can be controlled via the evaluation's YAML configuration file.

  6. What are Postprocessors and when to use them

    main

    Postprocessors are output-tidying steps used by solvers to clean up generated text before evaluation. They are particularly useful for generative language model solvers that might produce correct answers in formats that don't match the expected evaluation criteria (e.g., adding extra quotes, periods, or whitespace).

    Common use cases include:

    • Removing surrounding whitespace.
    • Stripping quotation marks.
    • Removing trailing punctuation like periods.
    • Converting formats to ensure exact match criteria do not produce false negatives.

  7. How the Bugged Tools evaluation process works

    main

    The evaluation follows a multi-turn conversation pattern:

    1. Initialization: The solver receives a task description and a list of available tools.
    2. Tool Interaction: The solver interacts with tools by generating the flag (@NAME: INPUT).
    3. Turn Limit: By default, the solver has ten turns to complete the task. Each turn consists of one text generation from the solver followed by a response from the eval (either tool output or a reminder of the task).
    4. Completion: The solver completes the task by outputting (@Answer: OUTPUT).
    5. Bug Detection: If the solver suspects a tool is bugged, it must output (@Bugged: NAME).
    6. Scoring: The system parses the conversation to compare the solver's (@Bugged: NAME) prediction against the ground truth label.

  8. What are completion functions

    main
    Completion Functions are generalizations of model completions. Instead of testing a raw model directly, a completion function allows you to wrap the model with additional logic or operators (such as giving the model access to a browser or search tools) before returning a response. This abstraction allows you to run any evaluation against a complex system rather than just a single LLM call.

  9. Understand the Schelling Point evaluation process and metrics

    main

    The evaluation measures coordination by comparing two prompting variants:

    1. Baseline setting (direct prompt): The model is asked to select a word from text without being told it needs to coordinate.
    2. Coordination setting (contextual information prompt): The model is explicitly told that other copies of itself are seeing the same text in a different order and asks it to select a word that its copies will also select.

    Metrics

    • runtime_error_rate: Percentage of samples that failed due to an error.
    • no_ci_convergence_rate: Convergence rate in the baseline (direct prompt) setting.
    • ci_convergence_rate: Convergence rate in the contextual information setting.
    • ci_delta (Main Metric): The difference between ci_convergence_rate and no_ci_convergence_rate. This isolates the capability of deliberate coordination.

  10. Format and usage of Registry Data JSONL files

    main

    Registry data in OpenAI Evals is stored in .jsonl files. These files must be pulled via git-lfs or downloaded manually to be viewed. Each line in the JSONL represents a single test case containing an input (typically a list of message roles) and an ideal value (the ground truth). The structure of the ideal key and how it is processed depends on the evaluation class used in the associated .yaml template.

    Common patterns include:

    • Match class: Checks if the model's completion starts with the value provided in the ideal key.
    • FuzzyMatch class: Checks if the completion includes a normalized version of the ideal key (which can be a string or a list of strings) or vice-versa.
    • ModelBasedClassify class: Uses a model-graded YAML (e.g., fact.yaml) to compare the factual content of the completion against the ideal ground truth.
    // Example for Match class
{"input": [{"role": "system", "content": "Complete the phrase."}, {"role": "user", "content": "Once upon a "}], "ideal": "time"}

// Example for FuzzyMatch class
{"input": [{"role": "user", "content": "Who plays eleven in stranger things?"}], "ideal": ["Millie Bobby Brown"]}

// Example for ModelBasedClassify class
{"input": [{"role": "system", "content": "Solve this puzzle..."}], "ideal": "The answer is X"}

  11. What are Solvers and how do they differ from Completion Functions?

    main

    A Solver is an abstraction representing the entire system used to attempt an evaluation, including the model, prompting strategies, and tools (scaffolding).

    This differs from the older Completion Function abstraction in the following ways:

    • Separation of Concerns: Solvers allow for 'Solver-agnostic' evals. Instead of the eval baking in specific prompt formats (which might favor ChatCompletion models), the eval defines the task, and the Solver handles the model-specific or strategy-specific implementation.
    • Scope: While a Completion Function is typically just a function that takes a prompt and returns a completion, a Solver can be a complex system with state and tools.

    Note: The Solvers framework is currently in Beta. For new dataset submissions relying on existing eval templates, it is recommended to continue using the original Eval classes with CompletionFn rather than SolverEval with Solvers.

  12. How the Self Prompting evaluation process works

    main

    The evaluation follows a two-stage process involving a Prompter and a Tasker:

    1. Prompt Generation (Prompter): The Prompter is provided with a task_description and a current_state (containing the task instruction, training samples, and the target Tasker model ID). The Prompter's goal is to generate a new prompt that includes the mandatory placeholder string [sample_in]. This placeholder is where the actual task input will be inserted.
    2. Task Execution (Tasker): The generated prompt is concatenated with each input sample from the test dataset (replacing [sample_in] with the sample input). The Tasker model then processes this combined prompt.
    3. Scoring: The Tasker's output is compared against the ground-truth label using an exact match criterion to determine accuracy.