DOCSEARCHbyAlgolia
Submit Docs

Opik: Open-Source AI Observability, Evaluation, and Optimization Platform

repository·main·Indexed Apr 15, 2026

https://github.com/comet-ml/opik

Opik is an open-source platform for LLM application observability, evaluation, and optimization. It features a Java backend using Redis-compatible RQ structures and Python workers, supporting self-hosted and cloud modes. The platform offers SDKs for Python, TypeScript, and Ruby (via OpenTelemetry) to log traces, annotate spans, and integrate with frameworks like Google ADK, Autogen, and Flowise AI. Key capabilities include deep LLM call tracking, automated evaluation with Datasets and Experiments, LLM-as-a-Judge metrics, CI/CD integration via PyTest, and production monitoring with dashboards supporting 40M+ traces/day. It includes Opik Agent Optimizer and Opik Guardrails for continuous improvement and security.

Tokens
42.2K
Snippets
127
Records
307
Agent score
98%

What's inside Opik

  1. Overview of Opik Prompt Management

    main

    Opik provides a centralized prompt library designed to help you version, reuse, and manage prompts across different projects. Instead of hardcoding strings throughout your application, you can use the library to maintain a single source of truth.

    Key Features:

    • Versioning: Track changes to your prompts over time.
    • Reusability: Use the same prompts across multiple projects.
    • Centralization: Manage all prompts in one location.

    Important Scoping Note: In Opik 2.0, prompts are project-scoped. When creating prompts, you must specify a project_name to ensure they are associated with the correct project.

  2. Overview

    main
    The system uses a reactive, event-driven architecture to handle high-volume trace ingestion efficiently. It processes batches of 1-1000 traces, deduplicates them, binds them to projects, stores them in ClickHouse, and triggers asynchronous event handlers for thread management, scoring, and analytics.

  3. What is Opik?

    main
    Opik is an open-source platform designed to streamline the entire lifecycle of LLM applications. It empowers developers to evaluate, test, monitor, and optimize their models and agentic systems.

  4. Overview of Opik REST API client modules

    main

    The Opik Python SDK provides several categories of REST API clients to manage different aspects of the platform:

    Core Resource Clients

    Used for primary observability and organization tasks:

    • traces: Observability of execution flows.
    • spans: Detailed execution tracking.
    • datasets: Management of evaluation data.
    • experiments: Testing and comparison.
    • projects: Organizational grouping.

    Feedback & Evaluation Clients

    Used for managing human and automated evaluation:

    • annotation_queues: Human review workflows.
    • feedback_definitions: Defining feedback score types.
    • automation_rule_evaluators: Automated evaluation rules.
    • optimizations: Running optimization experiments.

    Content & Asset Clients

    Used for managing templates and files:

    • prompts: Prompt templates and versions.
    • attachments: File attachments for traces and spans.

    System & Configuration Clients

    Used for platform management:

    • check: Health checks.
    • workspaces: Workspace management.
    • llm_provider_key: API key configuration.
    • service_toggles: Feature toggles.
    • system_usage: Usage monitoring.

    Integration Clients

    Used for external system connectivity:

    • chat_completions: Chat completion APIs.
    • open_telemetry_ingestion: OpenTelemetry data ingestion.
    • guardrails: Content validation.
    • redirect: URL redirection.

  5. Overview of Cost Intelligence

    main

    Cost Intelligence is an enterprise feature in Opik designed to provide granular visibility into LLM spending, specifically for coding agents. While standard provider bills show total token counts, Cost Intelligence attributes token usage to specific 'cost buckets' to identify framework overhead versus actual user input.

    Key Capabilities

    • Exact Spend Attribution: Uses the provider's own usage payload for token totals (no estimation or re-tokenization).
    • Token-level Attribution: Splits every call across approximately 20 cost buckets, including system prompt, tools, MCP servers, memory, skills, agents, thinking, tool I/O, and user input.
    • Visibility & Analytics: Provides per-user leaderboards, session-level drilldowns, and per-repository context.
    • Actionable Savings: Offers priced recommendations to reduce spend (typically by 15% to 30%) by identifying unused MCP servers, dead skills, or runaway tool output.

    Important Notes

    • Access: This is an enterprise feature and requires workspace admin access. Contact your Comet account team if it is not visible in your workspace.
    • Privacy: Only counts and metadata are captured; prompt and response content are never sent. It is designed to be invisible to developers with no added latency.

    Supported Agents

    • Claude Code: Supported
    • Codex: Supported
    • Cursor: Supported

    Deployments using a gateway (e.g., corporate LLM gateway, LiteLLM, AWS Bedrock access gateway) are supported by pointing the proxy's upstream to that gateway.

  6. Overview of Opik Python SDK Testing Categories

    main

    The Opik Python SDK uses a multi-tiered testing strategy to balance speed, isolation, and realism. Developers can leverage these different categories depending on whether they are testing internal logic, third-party integrations, or full system workflows:

    1. Unit Tests: Fast, isolated tests using a fake backend. No network calls are made.
    2. Library Integration Tests: Tests integrations (like OpenAI or LangChain) using a fake Opik backend. Requires API keys for the external libraries but not for Opik.
    3. E2E Tests: End-to-end tests that validate core functionality against a real Opik backend.
    4. E2E Library Integration Tests: The most comprehensive tests, combining real library calls (e.g., LiteLLM) with a real Opik backend.
    5. Smoke Tests: Minimal sanity checks to ensure imports and basic operations function.

  7. Overview of Opik Agent Optimizer SDK

    main

    The Opik Agent Optimizer SDK provides tools for optimizing LLM prompts and agents. It uses a standardized API across different optimization algorithms, ensuring consistency and interoperability.

    Key capabilities include:

    • Standardized API: All optimizers implement the same optimize_prompt() interface.
    • Multiple Algorithms: Support for evolutionary, few-shot, meta-prompt, and GEPA strategies.
    • MCP Support: Built-in support for Model Context Protocol tool calling.
    • Standardized Output: All optimizers return OptimizationResult objects.
    • Monitoring: Built-in LLM and tool call counters for usage tracking.

  8. What is Opik?

    main

    Opik is an open-source observability and evaluation platform designed for AI agents and LLM applications. It covers the entire LLM application lifecycle, including:

    • AI Agent Tracing & Observability: Deep tracing of LLM calls, conversation logs, and agent activities, providing full trace trees for multi-step agents and tool calls.
    • LLM Evaluation: Tools for datasets, experiments, and 'LLM-as-a-judge' metrics (e.g., hallucination detection, content moderation, and RAG evaluation like answer relevance and context precision).
    • Prompt & Agent Optimization: Includes the Opik Agent Optimizer SDK to improve prompts and agent behavior.
    • Production Monitoring: Scalable dashboards and online evaluation rules to monitor feedback scores, trace volume, and token usage.
    • Opik Guardrails: Features to implement safe and responsible AI practices.
    • CI/CD Evaluation: PyTest integration to test LLM pipelines on every commit.

    Opik is licensed under Apache-2.0, allowing for full self-hosting of the entire platform (backend, web app, etc.) without needing enterprise licenses.

  10. Overview of Opik Observability Concepts: Metrics, Optimization, and Evaluation

    main

    Opik provides three additional pillars for LLM observability beyond basic tracing:

    1. Metrics

    Metrics provide quantitative, objective assessments of model performance. They allow for trend analysis and comparative analysis between different models or prompts.

    • Common Types: Accuracy, Quality (coherence, relevance), Efficiency (latency, throughput), Cost (token usage), and Custom metrics.

    2. Optimization

    Optimization is the systematic process of refining LLM prompts and configurations. This involves:

    • Prompt Engineering: Refining instructions.
    • Parameter Tuning: Adjusting temperature, top-p, etc.
    • Few-shot Learning: Optimizing in-context examples.
    • Tool Integration: Improving tool/function interaction.

    3. Evaluation

    Evaluation is a framework for testing prompts and models against datasets. It consists of:

    • Datasets: Collections of test cases (inputs/expected outputs).
    • Experiments: Individual evaluation runs.
    • Metrics: Quantitative measures used during the run.
    • Comparative Analysis: Side-by-side comparison of different approaches.

  11. Overview of Opik Agent Optimizer

    main

    Opik Agent Optimizer is an open-source SDK designed to automate the tuning of LLM prompts, tools, and agent workflows. Instead of manual prompt engineering, it uses datasets, metrics, and traces already logged to Opik to iteratively improve performance using various optimization algorithms.

    Key Capabilities

    • Agent-aware optimization: Goes beyond system prompts to optimize MCP tool signatures, function-calling schemas, and multi-step orchestration logic.
    • Framework agnostic: Works with various LLM frameworks and supports both proprietary and open-source algorithms (like GEPA).
    • Deep observability: Every optimization trial logs prompts, tool calls, traces, and metric reasoning to the Opik dashboard.
    • Flexible deployment: Can be run locally, in Docker, or in Kubernetes to keep data within your network.
    • Optimization Studio: A no-code UI for configuring and reviewing optimization workflows directly from the Opik dashboard.

  12. Overview of Opik capabilities

    main

    Opik is a comprehensive platform designed to support the entire LLM lifecycle, from development to production. It provides tools for observability, evaluation, and monitoring to help developers build and maintain reliable LLM applications.

    Core capabilities include:

    • Tracing: Centralized tracking of all LLM application interactions to debug complex chains and identify issues.
    • Evaluation: Automated, consistent, and reproducible evaluations using standardized datasets and metrics to replace manual checks.
    • Production Monitoring: Real-time performance tracking and testing tools to ensure confidence during deployment.
    • Integrations: A rich ecosystem that works with popular frameworks such as OpenAI and Azure.
    • Deployment Options: Available as an Open Source project or an Enterprise solution for enhanced security and scalability.