UltraFeedback Documentation

The UltraFeedback dataset uses a JSON structure where each entry contains a single instruction and multiple model completions. This format is designed to support preference learning and multi-dimensional evaluation (e.g., helpfulness, honesty, truthfulness).

Key components of the schema:

• source: The origin dataset (e.g., sharegpt).
• instruction: The user prompt.
• models: A list of model names that generated completions for this instruction.
• completions: An array of objects, one for each model, containing:
  • model: The name of the model.
  • principle: The alignment principle used (e.g., helpfulness).
  • custom_system_prompt: The system prompt used to align the model behavior.
  • response: The actual text generated by the model.
  • annotations: A dictionary of multi-dimensional ratings (e.g., instruction_following, honesty, truthfulness, helpfulness). Each annotation includes a Rating and a Rationale. For certain metrics like truthfulness, it may also include a Type field.

{
    "source": "sharegpt",
    "instruction": "I am going to cairo in June of this year...",
    "models": [
        "falcon-40b-instruct",
        "gpt-4",
        "starchat",
        "wizardlm-7b"
    ],
    "correct_answers": ["None"],
    "incorrect_answers": ["None"],
    "completions": [
        {
            "model": "falcon-40b-instruct",
            "principle": "helpfulness",
            "custom_system_prompt": "As an AI assistant...",
            "response": "Cairo is a city that has something for everyone...",
            "annotations": {
                "instruction_following": {
                    "Rating": "2",
                    "Rationale": "The text only partially addresses the task goal..."
                },
                "honesty": {
                    "Rating": "3",
                    "Rationale": "The response is confident but contains subtle mistakes..."
                },
                "truthfulness": {
                    "Type": ["1", "2"],
                    "Rationale": "The text suggests whitewater rafting on the Nile...",
                    "Rating": "3",
                    "Rationale For Rating": "The text provides some truthful information..."
                },
                "helpfulness": {
                    "Type": ["1", "2"],
                    "Rationale": "The response is clear and relevant...",
                    "Rating": "3",
                    "Rationale For Rating": "The text is correct and provides useful information..."
                }
            }
        }
    ]
}

The process(responses, aspect) function parses the raw string output from the LLM evaluator into structured JSON-like dictionaries. It uses regex patterns to extract ratings and rationales based on the specific aspect being evaluated.

Supported Aspect Formats:

1. instruction_following or honesty Expects a pattern: Rating: <value>\nRationale: <text> Returns: [{'Rating': '...', 'Rationale': '...'}]

2. truthfulness or helpfulness Expects a pattern: Type: <value>\nRationale: <text>\nRating: <value>\nRationale For Rating: <text> Returns: [{'Type': '...', 'Rationale': '...', 'Rating': '...', 'Rationale For Rating': '...'}]

Error Handling: If the LLM output does not strictly follow the expected regex pattern, the function raises ValueError or AttributeError.

# Example of what the process function expects as input 'responses'
# For 'instruction_following':
responses = """
Rating: 5
Rationale: The model followed all constraints perfectly.
"""

# For 'truthfulness':
responses = """
Type: 1
Rationale: The model provided incorrect info.
Rating: 2
Rationale For Rating: The response was partially true but misleading.
"""

The annotate function processes a dataset entry (an example dictionary) to generate constructive feedback and an overall quality score for each completion within that entry. It uses an LLM (typically GPT-4) to act as a teacher, evaluating the response based on helpfulness, truthfulness, honesty, and instruction following.

Input Format Requirements:

• The example must be a dictionary containing an instruction key.
• The example must contain a completions list, where each item is a dictionary containing:
  • response: The text generated by the model.
  • principle: A string used to determine how to handle the custom_system_prompt (specifically checking for verbalized_calibration).
  • custom_system_prompt: A string providing additional context for the instruction.

Output Format:

• The function modifies the example in-place, adding two keys to each completion dictionary:
  • critique: A string containing the qualitative feedback.
  • overall_score: A numeric score (derived from the LLM's Overall Score: [1-10] output).

Prompting Logic: The feedback is generated using a template that expects the following structure in the LLM response:

### Feedback
[Your feedback]
Overall Score: [1-10]

# Example structure of an input 'example' dictionary
example = {
    "instruction": "Explain quantum physics.",
    "completions": [
        {
            "response": "Quantum physics is...",
            "principle": "standard",
            "custom_system_prompt": "Be concise."
        }
    ]
}

# After calling annotate(example):
# example["completions"][0]["critique"] = "..."
# example["completions"][0]["overall_score"] = 8.0

The Conversation class is used to manage prompt templates and maintain conversation history for various LLM architectures. It handles the formatting of system prompts, roles, and messages into a single string suitable for model generation based on a specific SeparatorStyle.

Key attributes:

• name: The name of the template.
• system: The system prompt.
• roles: A list of strings representing the roles (e.g., ['USER', 'ASSISTANT']).
• messages: A list of [role, message] pairs.
• sep_style: A SeparatorStyle enum determining how messages are delimited.
• sep and sep2: Primary and secondary separators.
• stop_str: A string used to stop generation.
• stop_token_ids: A list of integer token IDs used to stop generation.

Common methods:

• get_prompt(): Returns the formatted prompt string.
• append_message(role, message): Adds a new message to the history.
• update_last_message(message): Updates the last message (typically used to fill in the assistant's response after generation).
• to_openai_api_messages(): Converts the history to the OpenAI chat completion format.
• to_gradio_chatbot(): Converts the history to Gradio's chatbot format.

from src.comparison_data_generation.fastchat import conv_vicuna_v1_1

conv = conv_vicuna_v1_1.copy()
conv.append_message(conv.roles[0], "Hello!")
conv.append_message(conv.roles[1], "Hi!")
conv.append_message(conv.roles[0], "How are you?")

# The last message is often None before generation to act as a placeholder
conv.append_message(conv.roles[1], None)

print(conv.get_prompt())