Latitude LLM

This document catalogs the various uses of AI.generate within the system. Each usage is associated with specific telemetry tags and metadata structures. Note that AI.embed and AI.rerank calls are not wrapped in capture and are therefore not exported to Latitude.

Key Resources:

• Tag constants: Located in packages/domain/ai/src/ai-generate-telemetry.ts.
• Evaluation capture helpers: Located in packages/domain/evaluations/src/runtime/ai-telemetry.ts.

Flaggers are automatic issue detectors authored by Latitude. They inspect settled sessions (after conversation-intelligence analysis) to identify defects. When a defect is found, they write SYSTEM-authored annotation scores that integrate into the same signal-discovery pipeline as human annotations.

Detection occurs in two distinct passes:

1. Screening: A deterministic, free pass that runs on 100% of sessions. It gathers session hints, runs deterministic detection strategies, writes hard matches immediately, and routes remaining candidates to the LLM pass.
2. Classification: An LLM-based pass that performs classification, adversarial review, and drafting. This pass is only reached via 'hinted' paths or 'sampled' gates.

Agent dispatch allows you to automatically trigger a hosted coding agent when a Latitude signal requires engineering follow-up. Latitude acts as the trigger and context provider by sending the agent a prompt containing the signalId, a deep link, and sample trace IDs. The agent then uses the Latitude MCP (Model Context Protocol) connection to investigate the signal, root-cause the issue, and implement a fix.

Workflow:

1. Trigger: A dispatch trigger fires (e.g., new signal discovered, escalation, regression, or incident opening).
2. Context Assembly: Latitude assembles a prompt with the signal details and fires the configured target.
3. Investigation: The agent (connected via MCP) investigates, implements a fix, runs checks, and reports changes.

Note: Latitude does not run the fix or manage the agent's completion loop; it only provides the trigger and data.

The @tools/ai-benchmarks package is an offline benchmark harness for Latitude's AI features (including flaggers, annotators, and evaluation runtimes).

For each benchmark target, the harness:

1. Fetches data: Uses a mapper to create fixtures/<target>.jsonl.
2. Executes tests: Runs fixtures through the actual AI logic.
3. Aggregates metrics: Calculates precision, recall, F1, and per-tactic breakdowns.
4. Compares against baselines: Diffs results against a baseline committed at baselines/<target>.json to surface prompt changes as a flip list.

The Observability platform migration tool is a self-serve feature designed to backfill historical observability data into Latitude. It allows users to import traces, spans, session identity, user identity, tags, metadata, messages, usage, and cost from external platforms like Langfuse, LangSmith, or Braintrust.

Key characteristics:

• Idempotent and Resumable: Imports can be re-run without creating duplicates due to deterministic ID mapping.
• Non-Billable: Imported spans are not billable by default and do not trigger live evaluation fan-outs.
• Provenance: All imported spans are tagged with metadata indicating their source (e.g., import.source, import.job_id).

The Latitude MCP is a remote, OAuth-authenticated, streamable HTTP Model Context Protocol server located at https://api.latitude.so/v1/mcp. It allows AI agents (such as Claude, Cursor, Codex, Gemini, Zed, and OpenCode) to read and manage your Latitude workspace. The server dynamically generates tools from the Latitude API, enabling agents to manage projects, members, keys, traces, annotations, scores, searches, signals, and datasets.

To manage or revoke access for connected agents, navigate to your organization's Settings → Keys and locate the OAuth Keys section.

Simulations serve as the CI and regression-testing layer for the Latitude reliability system. They allow you to execute agent behavior locally or in CI, evaluate it using Latitude issue monitors or custom code, and optionally upload traces and scores back to the Latitude platform for hosted reporting.

Key characteristics:

• Local-first: The CLI and SDK are designed to work as standalone runners without requiring a hosted Latitude workspace.
• Optional Instrumentation: Simulations can run without instrumenting your LLM application by using score-only uploads.
• Language Agnostic: The CLI can run simulations for any language by executing a user-configured command that targets *.sim.* entrypoints.

In the Latitude system, a trace is considered a member of a signal if it carries that signal's signal_id.

Membership vs. Verdict:

• Membership: Determined by the presence of a signal_id on a score. Occurrence reads (e.g., counting how many times a signal happened) use signal_id and do not filter by the verdict.
• Verdict (passed): Indicates the result of an evaluation or annotation.
  • For evaluations (detectors): passed = true means the behavior was present (the detector matched).
  • For annotations (human sentiment): passed = true means a thumbs-up, while passed = false means a thumbs-down.

Writing Scores:

• The system stamps the signal_id on a score only when the verdict is present (passed = true).
• If a run is absent (e.g., an evaluation that didn't match), the score is still written, but the signal_id is null.

Flaggers are Latitude's built-in automatic annotators that check completed traces for common failure categories. When a match is found, Latitude adds an annotation to the trace. These annotations can be used for:

• Signal discovery
• Evaluation alignment
• Score analytics
• Trace-level review