Monocle Observability and Testing Framework

Monocle Test Tools is a framework for validating AI agent tracing. Key capabilities include:

• Trace-based Testing: Automatically generate test code from trace files and perform offline testing against pre-recorded trace JSON files.
• Agent & Tool Validation: Verify agent invocations, task delegation, and ensure tools are called with correct inputs and produce expected outputs.
• Response & Inference Testing: Validate agentic responses and model inference against expected content, including integration with Okahu or custom evaluation tools.
• Performance & Quality Metrics: Verify token usage, duration limits, error states, and warnings.
• Developer Experience: Use a fluent API for chained assertions, mock tools to avoid external dependencies, and a declarative API (@monocle_testcase) to describe runs as data.

Monocle is a GenAI-specific observability layer built on OpenTelemetry. It consists of three core components:

1. Metamodel: Defines standardized entities (agents, prompts, responses, tools, vector operations) and maps them to specific span attributes.
2. Instrumentation adapters: Automatically create spans for key operations (agent runs, tool calls, LLM invocations, retrieval queries) for supported GenAI frameworks.
3. OTLP-compatible traces: Emits standard OpenTelemetry traces that can be consumed by any existing OTLP backend (file, console, cloud storage, or APM).

The linter enforces several categories of rules to ensure observability best practices:

1. Required Fields: For all inference spans, entity.2.name (model name) and entity.2.type (model type) must be present.
2. Token Count: For supported providers (OpenAI, Anthropic, Gemini, AWS Bedrock, Azure OpenAI), prompt_tokens and completion_tokens must be present in the metadata event. If the provider is not in this list, missing tokens trigger a warning instead of an error.
3. Tool Metadata: For agentic tool invocation spans (span.type=agentic.tool.invocation), tool.status and tool.return_value are required.
4. Naming Convention: All span names must follow snake_case (allowing dots and underscores). Patterns like CamelCase or kebab-case will trigger warnings.

A Scope is a tag-value pair used to group different traces together.

• Scopes can be explicitly specified by the developer.
• Scopes can be auto-generated as a GUID by Monocle.
• Multiple scopes can be active simultaneously.
• Every span emitted while a scope is active will automatically record all currently active scopes.

Since GitHub Copilot does not include token counts in its hook payload, monocle-apptrace copilot-setup enables Copilot's native OpenTelemetry file exporter.

Monocle correlates gen_ai.usage.* attributes from the local OTel file to the active turn at Stop time using the OTel trace ID. This allows token usage data to be attached to the emitted spans without capturing any conversation content.

All telemetry is written to ~/.monocle/.copilot_otel/copilot.jsonl.

When using check_eval(), adhere to the following constraints to avoid ValueError exceptions:

1. Mutual Exclusivity: You must provide exactly one of eval_name, template_path, or template. Providing more than one is an error.
2. Required Assertions: You must provide at least one of expected or not_expected. Omitting both is an error.
3. No Overlap: The values in expected and not_expected must not overlap.
4. Fact Name Compatibility: Each built-in metric supports specific fact_name types. Using an unsupported combination (e.g., a metric that only works on traces with agentic_turns) will raise an error.
5. Time-Window Parameters: min_facts, fail_threshold, and max_facts can only be used if you are performing a time-window filtered evaluation.

Monocle uses a hierarchical tracing model to observe code execution:

• Span: An observation of a specific code or method execution. A span records start/end times and metadata (arguments, return values, etc.). Spans can have a parent span (child relationship) or be a root span if they have no parent.
• Trace: A collection of spans sharing a common traceID. A trace begins when the first active span is created and ends when the top-level (root) span completes. This allows grouping all related operations across different methods and processes into a single logical execution flow.

For a more structured approach, use the @MonocleValidator().monocle_testcase(test_cases) decorator. This allows you to define expected trace spans (including span_type and entities) in a JSON-like structure and validate them against the actual execution. This is particularly useful for complex multi-agent hierarchies where you want to ensure specific agents and tools are called in a specific sequence or with specific attributes.

@MonocleValidator().monocle_testcase(agent_test_cases)
async def test_crewai_travel_agent(my_test_case: TestCase):
    result = await execute_crewai_travel_request(my_test_case.test_input[0])
    return result

Monocle's sentiment evaluation distinguishes between three primary states:

• Positive: Requires explicit emotional content expressing positivity.
• Neutral: Factual, business-oriented, or successful interactions that lack emotional tone (e.g., a successful flight booking conversation is often classified as neutral).
• Negative: Expresses frustration, complaints, or negative emotional content.

Monocle works by intercepting agent session events via hooks. The process follows this lifecycle:

1. Event Capture: The agent hook (e.g., claude-hook) reads event JSON from stdin as agent session events fire.
2. Logging: An Event Handler records each event into a per-session JSONL log.
3. Replay: Upon receiving a Stop event, a Replay Session reconstructs the conversation turns.
4. Emission: The system emits OpenTelemetry (OTel) spans to configured Exporters (Okahu, file, or console).

An anchor span is created by a top-level method to provide an observation window for high-level SDK or service calls (e.g., a LangChain invoke() call).

Anchor spans allow you to compare the time taken by a high-level GenAI service invocation against the local logic surrounding it. They act as a parent to the more granular inference or retrieval spans triggered by the underlying SDK implementation.