agentevals

For graph-based agents (like LangGraph), agentevals uses a graph trajectory format instead of simple message lists. This format represents trajectories in terms of nodes visited (steps) rather than just messages, making it easier to evaluate complex agent behaviors like tool calls and interrupts.

A GraphTrajectory consists of:

• inputs: A list of inputs representing the start of new invocations in a thread.
• results: The final output from each turn in the thread.
• steps: A list of lists representing the internal nodes/steps taken for each turn (e.g., ['__start__', 'agent', 'tools', '__interrupt__']).

# Python structure
class GraphTrajectory(TypedDict):
    inputs: Optional[list[dict]]
    results: list[dict]
    steps: list[list[str]]

// TypeScript structure
export type GraphTrajectory = {
  inputs?: (Record<string, unknown> | null)[];
  results: Record<string, unknown>[];
  steps: string[][];
};

The trajectory_match_mode (Python) or trajectoryMatchMode (TypeScript) defines the strategy used to compare the actual trajectory against the reference.

Available modes:

• strict: Ensures trajectories contain the same messages in the same order with the same tool calls. It allows for differences in message content (text). Useful for enforcing specific tool call sequences.
• unordered: Ensures trajectories contain the same tool calls, but the order does not matter. Useful when the specific sequence of information retrieval is flexible.
• subset: Ensures the actual trajectory contains a subset of the tool calls found in the reference trajectory. Useful for ensuring an agent does not call tools beyond the expected set.
• superset: Ensures the actual trajectory contains a superset of the tool calls found in the reference trajectory. Useful for ensuring key tools are called, even if the agent calls extra ones.

By default, tool calls are only considered equal if they use the same tool and have identical arguments. You can customize this using tool_args_match_mode (Python) or toolArgsMatchMode (TypeScript) and tool_args_match_overrides (Python) or toolArgsMatchOverrides (TypeScript).

Match Modes:

• exact: (Default) Arguments must match exactly.
• ignore: Any two tool calls for the same tool are considered equivalent regardless of arguments.
• subset: A tool call is equivalent if its arguments are a subset/superset of the reference tool call's arguments.
• superset: A tool call is equivalent if its arguments are a superset/subset of the reference tool call's arguments.

Overrides: tool_args_match_overrides takes precedence over the global match mode. It accepts a dictionary where keys are tool names and values are:

• A ToolArgsMatchMode ("exact", "ignore", "subset", or "superset").
• A list of field names within the tool call that must match exactly.
• A comparator function (arg1, arg2) -> bool that returns whether the arguments are equal.

The LLM-as-judge trajectory evaluator uses a language model to assess an agent's trajectory. Unlike trajectory match evaluators, it does not require a reference trajectory to function, making it useful for evaluating general reasoning or correctness.

If you have a reference trajectory, you can provide it to the evaluator by using a prompt that supports a reference variable (such as TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE) and passing the reference_outputs argument during evaluation.

# Python Example: Basic usage without reference
import json
from agentevals.trajectory.llm import create_trajectory_llm_as_judge, TRAJECTORY_ACCURACY_PROMPT

evaluator = create_trajectory_llm_as_judge(
  prompt=TRAJECTORY_ACCURACY_PROMPT,
  model="openai:o3-mini"
)

outputs = [
    {"role": "user", "content": "What is the weather in SF?"},
    {
        "role": "assistant",
        "content": "",
        "tool_calls": [
            {
                "function": {
                    "name": "get_weather",
                    "arguments": json.dumps({"city": "SF"}),
                }
            }
        ],
    },
    {"role": "tool", "content": "It's 80 degrees and sunny in SF."},
    {"role": "assistant", "content": "The weather in SF is 80 degrees and sunny."},
]

eval_result = evaluator(outputs=outputs)
print(eval_result)

For TypeScript users, you can log evaluations to LangSmith using vitest or jest integrations.

Setup:

1. Set the following environment variables:
   • LANGSMITH_API_KEY: Your LangSmith API key.
   • LANGSMITH_TRACING: Set to "true".
2. Use langsmith/vitest (or langsmith/jest) to define tests with ls.describe and ls.test.
3. Use ls.logOutputs to record the agent's behavior.
4. Run the tests using your preferred runner (e.g., vitest run).

import * as ls from "langsmith/vitest";
import { createTrajectoryLLMAsJudge } from "agentevals";

const trajectoryEvaluator = createTrajectoryLLMAsJudge({
  model: "openai:o3-mini",
});

ls.describe("trajectory accuracy", () => {
  ls.test("accurate trajectory", {
    inputs: { messages: [{ role: "user", content: "What is the weather in SF?" }] },
    referenceOutputs: { messages: [...] },
  }, async ({ inputs, referenceOutputs }) => {
    const outputs = [...];
    ls.logOutputs({ messages: outputs });

    await trajectoryEvaluator({
      inputs,
      outputs,
      referenceOutputs,
    });
  });
});