AutoVoiceEvals

The repository is organized as follows:

• main.py: The primary entry point.
• config.yaml: The main configuration file for users to edit.
• .env.example: A template for API keys; copy this to .env.
• examples/: Contains provider-specific configuration templates (vapi.config.yaml, smallest.config.yaml, elevenlabs.config.yaml, livekit.config.yaml).
• autovoiceevals/: The core package containing the CLI, config loaders, provider clients (Vapi, Smallest, ElevenLabs, LiveKit), the Claude LLM client, the evaluator, and the research/pipeline logic.

Each evaluation scenario produces a composite score calculated as follows:

composite = 0.50 * should_score + 0.35 * should_not_score + 0.15 * latency_score

• should_score: The fraction of "agent should do X" criteria passed.
• should_not_score: The fraction of "agent should NOT do X" criteria passed.
• latency_score: 1.0 if the response is under 3 seconds, otherwise 0.5.

Weights and thresholds for these components are configurable in config.yaml under the scoring: key.

Note on Simplicity: If a prompt change results in the same score but a shorter prompt length, the system treats it as a KEEP to favor efficiency and lower costs.

The description field is critical because Claude uses it to generate adversarial attacks. Instead of writing attack vectors manually, you provide context so the system can figure out how to break your agent.

To maximize attack quality, include:

• Core Functionality: What the agent does (e.g., booking, support).
• Offerings: Services, menu items, or products with specific prices.
• Contextual Details: Staff names, roles, business hours, and locations.
• Policies: Cancellation rules, refund policies, or delivery zones.
• Constraints: Explicitly state what the agent cannot do (e.g., 'cannot give medical advice', 'cannot override pricing').

Create a .env file by copying the example and populate it with your API keys. You must provide an ANTHROPIC_API_KEY (used by Claude to generate scenarios and judge conversations) and the key for your specific voice platform.

Required Keys:

• ANTHROPIC_API_KEY (Always required)

Platform-Specific Keys:

• Vapi: VAPI_API_KEY
• Smallest AI: SMALLEST_API_KEY
• ElevenLabs: ELEVENLABS_API_KEY
• LiveKit: LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET

cp .env.example .env

To install autovoiceevals, clone the repository and install the required Python dependencies using pip.

git clone https://github.com/ArchishmanSengupta/autovoiceevals.git
cd autovoiceevals
pip install -r requirements.txt

The project provides two primary execution modes via the CLI:

• Research Mode (python main.py research): An autoresearch loop designed for iterative prompt optimization. It proposes one change at a time, keeps changes that improve the score, and reverts those that do not. It runs until max_experiments is reached.
• Pipeline Mode (python main.py pipeline): A single-pass audit. It generates adversarial attacks, performs a one-shot prompt improvement, and then verifies the result. This is best for a quick assessment of agent weaknesses.

# For iterative prompt optimization
python main.py research

# For a quick single-pass audit
python main.py pipeline

After a run, detailed data is stored in the results/ directory:

| File | Description | |---|---|results.tsv| One row per experiment containing score, CSAT, and pass rate.| | results/autoresearch.json| Full dataset including transcripts, eval criteria, proposals, and reasoning.| | results/best_prompt.txt| The highest-scoring prompt found during the run, ready for deployment.|

You can also view a human-readable summary via the CLI: python main.py results.

Use the main.py CLI to execute different optimization and auditing workflows.

Commands:

• python main.py research: Starts the iterative optimization loop. It runs until interrupted (Ctrl+C) or until autoresearch.max_experiments is reached in config.yaml. It proposes changes, tests them, and keeps improvements.
• python main.py research --resume: Resumes a previously interrupted research run.
• python main.py pipeline: Performs a single-pass audit (Attack $\rightarrow$ Improve $\rightarrow$ Verify) and then stops.
• python main.py results: Displays a summary of the completed run, including score progression, experiments, and the best prompt found.

# Iterative optimization
python main.py research

# Single-pass audit
python main.py pipeline

# View results
python main.py results