autoevals

LLM-as-a-judge scorers use language models to evaluate outputs based on semantic understanding. They typically return a score between 0 and 1. Available scorers include:

• Factuality: Checks if the output is factually consistent with the expected ground truth given an input.
• Battle: Compares output against expected to see which is better.
• ClosedQA: Evaluates answers to closed-ended questions.
• Humor: Evaluates if the output is humorous.
• Security: Checks for security vulnerabilities or unsafe content.
• Moderation: Uses OpenAI's moderation API to check for policy violations (sexual content, hate speech, etc.).
• Sql: Evaluates SQL query correctness.
• Summary: Evaluates the quality of text summaries.
• Translation: Evaluates translation quality.

import { Factuality } from "autoevals";

const result = await Factuality({
  input: "What is the capital of France?",
  output: "Paris",
  expected: "The capital of France is Paris",
});
// Score: 1.0

RAG scorers evaluate RAG systems by assessing both context retrieval and answer generation quality. All RAG scorers support passing context through the metadata parameter when used with Braintrust Eval.

Key RAG scorers include:

• ContextRelevancy: How relevant the retrieved context is to the question.
• ContextRecall: How well the context supports the expected answer.
• ContextPrecision: Whether relevant context appears before irrelevant context.
• ContextEntityRecall: Whether the context contains entities from the expected answer.
• Faithfulness: Whether the answer's claims are supported by the context.
• AnswerRelevancy: How relevant the answer is to the question.
• AnswerSimilarity: Semantic similarity between answer and expected answer.
• AnswerCorrectness: A combination of factuality and semantic similarity.

from autoevals.ragas import ContextRelevancy

scorer = ContextRelevancy()
result = scorer.eval(
    input="What is the capital of France?",
    output="Paris",
    context=[
        "Paris is the capital of France.",
        "Berlin is the capital of Germany."
    ]
)

The autoevals npm and PyPI packages are published together at a shared version via a single GitHub Actions workflow (.github/workflows/publish.yaml). A single approval gate controls both lanes; if one fails, neither is released.

Release Types

• stable (default): Publishes npm autoevals@<v> (with provenance) and PyPI autoevals==<v> (with attestation). Creates tags js-<v> and py-<v> and two GitHub Releases.
• prerelease: Publishes the committed version to the npm rc dist-tag. No GitHub release is created.

Release Workflow

1. Sync Versions: Bump the version in both package.json and py/autoevals/version.py simultaneously.
2. Merge: Merge the version bump into the main branch.
3. Trigger Workflow: Run the publish workflow with release_type=stable and the full 40-character sha of the merged commit.
4. Approve: Approve the publish environment when the workflow reaches the approval gate.

# Example workflow invocation parameters
release_type: stable
sha: <40-char-commit-sha>

Autoevals provides model-graded evaluators like Factuality. By default, these use the OPENAI_API_KEY environment variable. You can run evaluations synchronously by calling the evaluator instance directly, or asynchronously using the .eval_async() method.

Evaluators return a result object containing a score (typically in the range [0, 1]) and metadata (which may include a rationale).

from autoevals.llm import *
import asyncio

# Create a new LLM-based evaluator
evaluator = Factuality()

input = "Which country has the highest population?"
output = "People's Republic of China"
expected = "China"

# Using the synchronous API
result = evaluator(output, expected, input=input)
print(f"Factuality score (sync): {result.score}")
print(f"Factuality metadata (sync): {result.metadata['rationale']}")

# Using the asynchronous API
async def main():
    result = await evaluator.eval_async(output, expected, input=input)
    print(f"Factuality score (async): {result.score}")
    print(f"Factuality metadata (async): {result.metadata['rationale']}")

asyncio.run(main())