Pydantic Logfire Documentation

Pydantic Logfire is an AI-native observability platform designed to provide full-stack visibility into AI applications. Unlike AI-only tools, Logfire correlates LLM calls with the surrounding application context, such as HTTP requests, database queries, and business logic.

Key capabilities include:

• Full-stack AI observability: Contextualized traces including LLM calls, APIs, and databases.
• OpenTelemetry-based: Compatible with any language or framework that supports OpenTelemetry.
• LLM-specific features: Token tracking, cost monitoring, conversation inspection, and tool call visibility.
• SQL-based querying: Query observability data using PostgreSQL-compatible syntax via the Apache DataFusion engine.

Enterprise Dedicated is a fully managed, single-tenant deployment of Logfire. It provides a fully isolated environment for each tenant, including a dedicated virtual network, managed Kubernetes cluster, managed database, and object storage. This ensures complete data separation.

Key features:

• Configurable Region: Choose deployment regions based on latency, data residency, or compliance needs.
• Managed Infrastructure: Pydantic handles all provisioning, upgrades, and operations.
• Cloud Support: Currently supports GCP (Google Cloud Platform). Additional providers are planned.
• Access: Customers access their dedicated instance via a unique hostname.

Logfire dashboards allow you to visualize observability data using SQL-powered charts and tables. There are two types of dashboards available:

• Standard dashboards: Pre-configured and maintained by the Logfire team. They provide continuous updates and improvements. You can enable or disable them for your project, but they cannot be modified directly.
• Custom dashboards: Fully editable and customizable dashboards created by you. You can define your own queries, layouts, chart types, and variables.

It is recommended to start with standard dashboards and then use them as templates for custom dashboards if they do not meet your specific needs.

Pydantic Logfire provides three enterprise-tier deployment models depending on your organization's compliance, infrastructure, and management needs:

• Enterprise Cloud: A fully managed, SLA-backed SaaS service. Best for organizations needing custom retention, custom SSO, and industry-specific compliance (like HIPAA BAAs).
• Enterprise Dedicated: A fully managed, single-tenant deployment on isolated infrastructure.
• Enterprise Self-Hosted: An on-premises deployment via Kubernetes. Best for organizations with strict data sovereignty or highly sensitive data requirements.

To inquire about these plans, contact the sales team at sales@pydantic.dev.

When monitoring AI agents and LLM calls, Logfire uses two primary concepts to organize data:

• Trace: The full, complete record of a single request or agent run.
• Span: An individual step or unit of work contained within a trace (e.g., a single model call or a tool execution).

Logfire Prompt Management allows you to author, version, test, and ship LLM prompts directly from the Logfire UI. Instead of hardcoding prompts in your application code, you treat them as first-class configuration objects that your application fetches via the Logfire SDK.

The Production Workflow:

1. Author a prompt template in the Logfire UI.
2. Save versions to freeze the template text as you iterate.
3. Test using scenarios (saved test cases), datasets (sweeping scenarios over many cases), and runs (execution records).
4. Promote a version by pointing a label (e.g., production) at a specific version on the Managed Variables page.
5. Consume the prompt in your application using the Logfire SDK, which fetches the version associated with your specified label or rollout targeting.

Logfire uses a five-part scaffold to build offline evaluations. Understanding these components is essential for setting up repeatable tests for your AI system:

1. Dataset: A collection of test cases (inputs and optional expected outputs).
2. Task: The AI system under test (e.g., a function, an LLM call, or a Pydantic AI agent).
3. Scorers (evaluators): Judges attached to the task that produce a score. Scorers can be code-based (exact match), LLM-as-a-judge, or human-based.
4. Score: The individual rating produced by a scorer for a specific output.
5. Experiment: A single run of a Task over an entire Dataset, resulting in a table of scores used for version comparison.

Logfire uses four types of telemetry to provide observability:

• Span: A single unit of work (an operation) with a name, a start time, and a duration. Spans can be nested.
• Trace: A tree of spans representing the complete path of a single request (e.g., an API call or an LLM run) through your system.
• Metric: A value measured and aggregated over time (e.g., latency, CPU load) to reveal trends and trigger alerts.
• Log: A timestamped record of a single event without a duration.