Waza

The examples/ directory contains several specialized suites to demonstrate different Waza capabilities:

For skills that require follow-up questions, you can configure a responder. The responder is an LLM that simulates a user to interact with the skill. This is mutually exclusive with follow_up_prompts.

During a run, the responder can:

• reply: Sends an answer back to the agent to continue the conversation.
• stop: Signals the agent is finished.
• abstain: Fails the run with an abstained outcome, indicating the prompt was too vague.

If the agent exceeds max_followups, the loop stops with the cap_exhausted outcome.

Fields (under inputs.responder):

• instructions (required): The target configuration the responder represents and the rules for when it should abstain.
• max_followups (required): Maximum number of responder replies before the loop stops (must be >= 1).
• model (optional): The model used for the responder LLM. Defaults to config.model.

# task.yaml
inputs:
  prompt: "Add a new agent to my application"
  responder:
    model: gpt-4o          # optional; defaults to config.model
    instructions: |
      The agent you want is "research-agent" with system instructions
      "Search the web and summarise findings", tools web_search + url_fetch,
      and no handoffs. Answer the skill's questions consistently with this.
      If you genuinely can't infer an answer, abstain.
    max_followups: 8

Waza uses YAML-based evaluation specifications to define how skills and tasks are tested. An eval spec includes metadata (name, skill, schemaVersion), configuration for execution (trials, timeouts, models), input variables, lifecycle hooks, MCP mocks, graders, and task definitions.

Key configuration options include:

• config.executor: Set to mock or copilot-sdk.
• config.max_attempts: Number of retries for failed graders (default: 1).
• config.instruction_files: List of .md files to append to the agent's system message.
• config.group_by: Dimension to organize results (e.g., model).
• tasks: Glob patterns (e.g., tasks/*.yaml) or a tasks_from CSV source.

name: my-eval
skill: my-skill
schemaVersion: "1.2"
version: "1.0"

config:
  trials_per_task: 3
  max_attempts: 3
  timeout_seconds: 300
  parallel: false
  executor: mock
  model: claude-sonnet-4-20250514
  group_by: model
  instruction_files:
    - .github/instructions/project.instructions.md

inputs:
  api_version: v2

tasks:
  - "tasks/*.yaml"

WGP/1 is a standardized JSON protocol used by registry-distributed graders to ensure interoperability regardless of the runtime. Whether a grader is running as a WASM module or an external program, it communicates using this schema over stdin/stdout (for programs) or a thin host ABI (for WASM).

Request Format (sent by Waza): Contains the task details, agent output (including tool calls and files written), effective configuration, and execution context.

Response Format (returned by Grader): Contains the score, a pass/fail boolean, a rationale string, evidence (spans), and custom metrics.

This protocol allows a prototype (e.g., written in Python as a runtime: program) to be converted into a sandboxed WASM artifact later without changing the specification.

### Request Example
```json
{
  "schema": "wgp/1",
  "task": { "id": "task-42", "input": "...", "expected": "..." },
  "agent": {
    "output": "...",
    "tool_calls": [ { "name": "...", "arguments": { } } ],
    "files_written": [ { "path": "...", "sha256": "..." } ]
  },
  "config": { /* effective deep-merged config */ },
  "context": { "workspace_dir": "/tmp/waza-xxx", "trial": 1 }
}

Response Example

{
  "schema": "wgp/1",
  "score": 0.83,
  "passed": true,
  "rationale": "Found 3 of 4 expected facts; missed citation for claim #2.",
  "evidence": [ { "kind": "span", "ref": "agent.output[120..180]" } ],
  "metrics": { "fact_recall": 0.75, "fact_precision": 1.0 }
}

You can materialize a clean copy of a local git repository into a task's workspace using the worktree strategy. This is useful for testing skills against a specific local codebase without manual staging. Waza uses git worktree add --detach to create an isolated, cheap checkout that shares the same .git object store as the source.

To use this, define a repos list in your task.yaml under inputs. You can also set an optional workdir (relative to the workspace) to specify where the agent should start its execution, typically matching the dest path.

# task.yaml
id: my-task
name: Repo-aware task
inputs:
  prompt: "Explain the layout of this repository"
  workdir: my-repo          # optional: where the agent starts (relative to workspace)
  repos:
    - type: worktree        # required; only "worktree" is currently supported
      source: /path/to/local/clone   # required; local git repo to source from
      commit: main          # optional: commit SHA, branch, or tag (defaults to HEAD)
      dest: my-repo         # optional: subdir under workspace (omit to use workspace root)