Ragas: Evaluation Framework for RAG and LLM Applications

Ragas provides a diverse set of evaluation metrics categorized by application type and task. Metrics can be LLM-based (using one or more LLM calls to derive a score) or custom-written. Use the following categories to select the appropriate metrics for your use case:

Retrieval Augmented Generation (RAG)

Focuses on the quality of retrieved context and the faithfulness of the generated response.

• Context Precision
• Context Recall
• Context Entities Recall
• Noise Sensitivity
• Response Relevancy
• Faithfulness
• Multimodal Faithfulness
• Multimodal Relevance

Nvidia Metrics

• Answer Accuracy
• Context Relevance
• Response Groundedness

Agents or Tool Use Cases

Designed for evaluating agentic workflows and tool interactions.

• Topic adherence
• Tool call Accuracy
• Tool Call F1
• Agent Goal Accuracy

Natural Language Comparison

Used for comparing text outputs using semantic or traditional string-based methods.

• Factual Correctness
• Semantic Similarity
• Non LLM String Similarity (includes BLEU Score, CHRF Score, ROUGE Score, String Presence, and Exact Match)

SQL

• Execution based Datacompy Score
• SQL query Equivalence

General Purpose

Flexible scoring methods for various criteria.

• Aspect critic
• Simple Criteria Scoring
• Rubrics based scoring
• Instance specific rubrics scoring

Other Tasks

• Summarization

Aspect Critique is a binary evaluation method used to assess submissions against specific predefined aspects (e.g., harmlessness, correctness, coherence). It uses LLM-based evaluation to determine if a submission aligns with a defined aspect, returning a discrete value from a set of allowed responses.

To implement this, use the DiscreteMetric class. You must provide a name, a list of allowed_values, a prompt containing the evaluation criteria, and an llm instance.

Key Configuration:

• allowed_values: A list of strings that the LLM is permitted to return (e.g., ["safe", "unsafe"]).
• prompt: A template string that includes the {response} placeholder. The prompt should explicitly instruct the LLM to only use the values provided in allowed_values.
• strictness: A parameter used for self-consistency checks. An ideal range for maintaining consistent predictions is typically between 2 and 4.

from ragas.metrics import DiscreteMetric

# Example initialization
metric = DiscreteMetric(
    name="aspect_name",
    allowed_values=["value1", "value2"],
    prompt="Evaluate based on criteria. Response: {response}. Answer with only 'value1' or 'value2'.",
    llm=llm
)

Ragas distinguishes between metrics based on the interaction depth they evaluate:

1. Single-turn metrics: Evaluate a single interaction between a user and the AI. These inherit from SingleTurnMetric and are scored using the single_turn_ascore method. They require a SingleTurnSample object.
2. Multi-turn metrics: Evaluate performance across multiple turns of interaction. These inherit from MultiTurnMetric and are scored using the multi_turn_ascore method. They require a MultiTurnSample object.

Single-turn usage example:

from ragas.metrics import FactualCorrectness

scorer = FactualCorrectness()
await scorer.single_turn_ascore(sample) # sample is a SingleTurnSample

Multi-turn usage example:

from ragas.metrics import AgentGoalAccuracy
from ragas import MultiTurnSample

scorer = AgentGoalAccuracy()
await scorer.multi_turn_ascore(sample) # sample is a MultiTurnSample

When evaluating autonomous agents, it is recommended to use metrics that focus on binary decisions or discrete classification scores rather than ambiguous continuous scales. Ragas provides two primary ways to implement this:

1. AspectCritic: Evaluates whether a submission follows specific user-defined criteria (e.g., brand tone or request completeness) using LLM judgments to return a binary outcome (1 or 0).
2. RubricsScore: Assesses responses against a detailed, user-defined rubric to assign scores based on specific qualitative descriptions.

Both metrics require an llm (wrapped via LangchainLLMWrapper) to perform the evaluation.

from ragas.metrics import AspectCritic, RubricsScore
from ragas.dataset_schema import SingleTurnSample, MultiTurnSample, EvaluationDataset
from ragas import evaluate

# Example: Using RubricsScore for recommendation quality
rubrics = {
    "score-1_description": (
        "The item requested by the customer is not present in the menu and no recommendations were made."
    ),
    "score0_description": (
        "Either the item requested by the customer is present in the menu, or the conversation does not include any food or menu inquiry (e.g., booking, cancellation). This score applies regardless of whether any recommendation was provided."
    ),
    "score1_description": (
        "The item requested by the customer is not present in the menu and a recommendation was provided."
    ),
}

recommendations = RubricsScore(rubrics=rubrics, llm=evaluator_llm, name="Recommendations")

# Example: Using AspectCritic for binary compliance (e.g., Request Completeness)
request_completeness = AspectCritic(
    name="Request Completeness",
    llm=evaluator_llm,
    definition=(
        "Return 1 The agent completely fulfills all the user requests with no omissions. "
        "otherwise, return 0."
    ),
)

# Example: Using AspectCritic for Brand Voice
brand_tone = AspectCritic(
    name="Brand Voice Metric",
    llm=evaluator_llm,
    definition="Return 1 if the AI's communication is friendly, approachable, helpful, clear, and concise; otherwise, return 0.",
)