DOCSEARCHbyAlgolia
Submit Docs

AgentOps Documentation

repository·main·Indexed 26 days ago

https://github.com/agentops-ai/agentops

An observability and developer tool platform for AI agents, providing tools for building, evaluating, and monitoring agents via session replays and LLM cost management. It features OpenTelemetry-based instrumentation for providers like OpenAI, Anthropic, Google GenAI, and IBM WatsonX AI, as well as frameworks such as CrewAI, AG2/AutoGen, smolagents, and LangChain. The SDK includes the V3Client for API requests and specialized tools for tracking token usage, latency, and agent execution lifecycles.

Tokens
127.6K
Snippets
363
Records
628
Agent score
90%

What's inside AgentOps

  2. Overview of the AgentOps Exporter

    main

    The Exporter is a service designed to bridge the gap between the legacy AgentOps data schema and the new OpenTelemetry (OTEL) backend. It performs two primary functions:

    1. Parallel Processing: For data arriving via the legacy API endpoint, the exporter simultaneously writes data to the existing Supabase (Postgres) schema and reformats it into OTEL format for the ClickHouse backend.
    2. One-Time Migration: It provides a script to migrate all historical data accumulated in the Postgres schema into the ClickHouse backend.

  3. Overview of Jockey Kubernetes Deployment Package

    main
    Jockey is a Python package designed for Kubernetes-based deployment orchestration specifically for AI agents. It provides a model-style interface for managing Kubernetes resources (Deployments, Services, ConfigMaps, Secrets, etc.) and orchestrating deployment workflows including Docker image building, registry operations, and real-time status monitoring.

  4. Overview of OpenAI Agents SDK Instrumentation

    main
    This module provides automatic instrumentation for the OpenAI Agents SDK. It intercepts the SDK's trace processor interface and monkey-patches the Runner class to capture the full execution lifecycle, including streaming operations. The instrumentation converts captured data into OpenTelemetry spans and metrics that follow Generative AI semantic conventions.

  5. Overview of AgentOps

    main
    AgentOps is an observability platform designed for AI agents and applications. It provides real-time monitoring, distributed tracing for multi-step AI workflows, cost analytics for model providers, error tracking, and team collaboration features via a multi-user dashboard.

  7. Review the Evaluations Roadmap

    main
    AgentOps provides an evolving suite of evaluation tools. Currently, the Python SDK supports custom evaluation metrics, multi-session/cross-session metrics, and custom event tag tracking. Future developments include an Evaluation builder API, Agent scorecards, and an evaluation playground with leaderboards. A Javascript/Typescript SDK is currently in Alpha.

  8. Core Capabilities of AgentOps

    main

    AgentOps provides the following observability and monitoring features for AI agents:

    • Comprehensive Observability: Track agent performance, user interactions, and API usage.
    • Real-Time Monitoring: Access session replays, metrics, and live monitoring tools.
    • Cost Control: Monitor and manage spending on LLM and API calls.
    • Failure Detection: Identify agent failures and multi-agent interaction issues.
    • Tool Usage Statistics: Detailed analytics on how agents utilize external tools.
    • Session-Wide Metrics: Holistic view of agent sessions with comprehensive statistics.

  9. Understand AgentOps Dashboard project structure

    main

    The dashboard follows a Next.js App Router structure:

    • app/: Contains layouts, pages, and route handlers.
      • (with-layout)/: Routes sharing the main application layout (header, sidebar).
    • components/: Reusable UI components.
      • ui/: Low-level components (e.g., Button, Card) based on shadcn-ui.
    • lib/: Utility functions, type definitions (types_db.ts), and service integrations (e.g., lib/supabase/).
    • hooks/: Custom React hooks for data fetching (e.g., useMetrics, useTraces).
    • public/: Static assets.
    • tests/: Unit and integration tests.

  10. Track LangGraph workflows with AgentOps

    main

    AgentOps automatically captures the following telemetry for LangGraph workflows:

    • Graph Structure: Nodes, edges, and entry points.
    • Execution Flow: The specific path taken through the graph.
    • Node Executions: Inputs and outputs for each node.
    • LLM Calls: All interactions with language models within nodes.
    • Tool Usage: Details of any tools called within the graph.
    • State Changes: Evolution of the graph state.
    • Timing Information: Duration of individual nodes and total execution time.

  11. Understand Spans in AgentOps v0.4

    main
    In AgentOps v0.4, 'Spans' are the primary unit used to track operations in agent workflows, replacing the previous 'Events' concept. Spans are created using decorators and are automatically nested to form a hierarchical trace of your agent's execution. This hierarchy allows you to visualize the flow and relationship between different operations.

  12. Review the Debugging Roadmap

    main

    AgentOps is building out capabilities across several debugging domains:

    • Performance testing: Event latency analysis and agent workflow execution pricing (Success validators and agent controller tests are in development).
    • Environments: Honeypot and prompt injection detection (via PromptArmor) and multi-agent framework visualization are available. Non-stationary environment testing and execution containers are planned.
    • LLM Testing: API bill tracking is available. Token limit overflow flags and LLM non-deterministic function detection are in development.
    • Reasoning and execution testing: Infinite loops and recursive thought detection are currently under development.