deepagentsjs

The @deepagents/evals package is a generic evaluation harness designed for deepagents. It provides a standardized runner interface, a runner registry, trajectory parsing, and custom Vitest matchers that integrate with LangSmith feedback for experiment tracking.

Package Exports

The tool-usage-relational evaluation suite tests an agent's ability to perform multi-step tool chaining over relational datasets. It specifically focuses on scenarios where data entities (such as users, locations, and foods) are connected via IDs.

Key testing capabilities include:

• Sequential Lookups: Verifying the agent can navigate relational hops (from 1 to 4 hops deep) to find connected information.
• Parallel Fan-out: Testing if the agent can correctly identify and execute multiple independent tool calls simultaneously in a single step when the tasks do not depend on each other.

The subagents evaluation suite is designed to test subagent delegation capabilities. Specifically, it evaluates how well an agent performs task tool routing to two types of subagents:

1. Named subagents: Specific subagents with defined identities or roles.
2. General-purpose subagents: Subagents designed for broad, non-specific tasks.

Use these evals to ensure your agent correctly identifies when a task requires delegation and routes the tool call to the appropriate subagent entity.

The tool-selection evaluation suite is designed to assess an agent's ability to perform three specific capabilities:

1. Direct tool selection: Choosing the correct tool for a single, straightforward task.
2. Indirect tool selection: Selecting tools based on complex or nuanced reasoning where the tool choice is not immediately obvious.
3. Multi-step tool chaining: Successfully sequencing multiple tool calls to complete a complex, multi-stage objective.

The basic evaluation suite is designed to measure core agent behaviors. It focuses on fundamental capabilities that every agent should possess, specifically:

• System prompt adherence: Ensuring the agent follows the instructions and constraints provided in its system prompt.
• Simple reasoning: Testing the agent's ability to perform logical steps to reach a conclusion.
• Avoiding unnecessary tool calls: Verifying that the agent only invokes tools when required by the task, preventing redundant or incorrect tool usage.

The all package provides an aggregate eval runner that allows you to execute all available evaluation suites through a single top-level ls.describe block in eval.test.ts. This approach ensures that the entire execution run is captured under a single LangSmith experiment target, making it easier to monitor and compare results across different suites.

Each individual suite is implemented as a function following the pattern <suiteName>Suite(runner), exported from its own index.ts file. The aggregate runner in evals/all/eval.test.ts imports these functions and invokes them directly to orchestrate the full evaluation process.

The external-benchmarks package runs a curated, high-difficulty evaluation suite consisting of 15 cases. These cases are sampled from the following benchmark sources:

• FRAMES
• Nexus
• BFCL v3 (Berkeley Function Calling Leaderboard v3)

Use this package to test agent performance against standardized, hard-set benchmark samples.

The files evaluation suite is designed to test an agent's ability to perform standard file system operations. Use these evals to verify that an agent can correctly execute the following tasks:

• Read: Accessing file contents.
• Write: Creating or overwriting files.
• Edit: Modifying existing files.
• ls: Listing directory contents.
• grep: Searching for patterns within files.
• glob: Using pattern matching to find files.
• Parallel I/O: Handling multiple I/O operations concurrently.

The ACP server provides several core capabilities for agentic workflows:

Core Features

• Full Filesystem Access: Read, write, and edit files within the workspace.
• Code Search: Support for grep and glob patterns.
• Task Management: Tracking todo lists for complex tasks.
• Subagent Delegation: Spawning specialized subagents.
• Session Persistence: Maintaining context across interactions.
• Multiple Modes: Switching between Agent, Plan, and Ask modes.

Customization

You can customize the server behavior by editing examples/acp-server/server.ts. Key configurable areas include:

• Model selection
• System prompts
• Skills and memory paths
• Custom tools
• Middleware configuration

The summarization evaluation suite is used to assess how middleware handles summarization tasks. Specifically, it evaluates:

• Summarization middleware behavior: How the system processes and generates summaries.
• History offloading: The effectiveness of moving older conversation history into summaries.
• Continued task performance after compaction: How well the agent performs its tasks after the conversation history has been compacted via summarization.