DOCSEARCHbyAlgolia
Submit Docs

Langfuse Documentation

repository·main·Indexed 42 days ago

https://github.com/langfuse/langfuse-docs

Documentation for Langfuse, featuring a site built with Fumadocs and Next.js. Includes guides on the Langfuse Docs MCP server, raw Markdown URL endpoints for LLM consumption, and technical setups for the QA chatbot experiments and LiveKit Voice Agent integration.

Tokens
237.7K
Snippets
523
Records
814
Agent score
51%

What's inside Langfuse

  1. What is Langfuse?

    main

    Langfuse is an open-source AI engineering platform designed to help teams build, monitor, and improve LLM applications. It provides a unified environment for the full development lifecycle, including:

    • Tracing: Debug LLM applications with detailed traces and agent graphs.
    • Prompt Management: Manage and version prompts collaboratively.
    • Evaluations: Run automated evaluations using methods like LLM-as-a-judge or code evaluators.
    • Analytics: Track costs, latency, and performance via dashboards.

    You can use Langfuse as a managed cloud service or self-host it on your own infrastructure.

  2. Overview of Use-case Page Analytics

    main

    This analytics implementation tracks traffic, next-actions, CTA exposure, content interest, and completed conversions for the three Langfuse use-case landing pages (chat_agents, coding_agents, and workflow_automation).

    Tracking is managed via PostHog and relies on the route registry as the source of truth for which pages are tracked.

    Key Dashboards (Project 7132):

    • Traffic and next steps: Monitors visitors, pageviews, first next destinations, and specific interactions like benefits opened or gateway/hooks selections.
    • CTA visibility and clicks: Tracks exposed visitors, exposed clickers, placement CTR, and all clicks (including quick activations).
    • Completed conversions: Tracks page visits and CTA placements leading to completed signups or accepted sales inquiries.

    Important Deployment Notes:

    • All source-page queries are filtered to $host = langfuse.com. Preview and localhost traffic are excluded.
    • New use_case:* and sales:inquiry_completed data requires a website deployment to be active.
    • SQL tables use a fixed rolling 30-day window; date ranges are controlled within the SQL, not by the dashboard date picker.

  3. Overview of Langfuse Platform Features

    main

    Langfuse is an open-source AI engineering platform designed for agent evaluation and observability. It provides a continuous loop for the AI engineering lifecycle by connecting tracing, monitoring, datasets, experiments, and evaluation.

    Key capabilities include:

    • Observability: Hierarchical traces for LLM calls, tool invocations, and retrieval steps. Supports filtering by user, session, cost, latency, and custom metadata.
    • Evaluation: Supports LLM-as-a-judge, heuristic functions, and human review for production data or experiments.
    • Prompt Management: Decouples prompts from code with deployment and rollback capabilities.
    • Experiments: Tools to define test cases and compare model/prompt results side-by-side.
    • Human Annotation: Workflows for human-in-the-loop review to create golden datasets.
    • Cost and Latency Tracking: Monitoring via dashboards and automated alerts.

  4. Overview of Langfuse SDKs

    main

    Langfuse provides specialized SDKs for observability, allowing you to create custom observations and traces, and utilize prompt management and evaluation features. The SDKs are built on OpenTelemetry (OTEL), ensuring compatibility with existing OTEL-based instrumentation libraries.

    Supported SDKs:

    • Python SDK v4
    • JS/TS SDK v5
    • Other Languages: Supported via standard OpenTelemetry integration.

    Key Characteristics:

    • Low Latency: Uses fully asynchronous requests for background processing.
    • Reliability: SDK errors are caught and logged internally to prevent breaking your application.
    • Compatibility: Works with both Langfuse Cloud and self-hosted instances. You only need to change your credentials and base_url to switch environments.
    • Accuracy: Provides accurate latency tracking via synchronous timestamps and unique IDs for downstream use.

  5. Compare Langfuse and Galileo AI features

    main

    Langfuse and Galileo AI offer different approaches to the LLM engineering lifecycle across three pillars: Observability, Evaluation, and Annotation.

    Observability

    Langfuse provides fine-grained token and cost tracking (including reasoning and cached tokens), full-text search across inputs/outputs/metadata, and support for distributed tracing via OpenTelemetry. It also features dedicated UI for tool calls and agent graphs, the ability to replay traces in the Langfuse Playground, and full trace exports as JSON. You can also build custom dashboard widgets to monitor specific metrics.

    Evaluation

    Both tools support datasets and versioning. Langfuse allows triggering experiments via the UI or SDKs and enables users to correct model outputs directly in the trace detail view for future fine-tuning. Langfuse also features a native comments system for traces and observations. For evaluation methods, Langfuse supports LLM-as-a-Judge (using any configured model) and allows writing Python or TypeScript code evaluators directly in the UI.

    Annotation

    Langfuse provides integrated annotation queues for domain experts to review filtered traces. It supports annotating traces and observations via the Trace detail view and allows adding human annotation scores in the Dataset Experiment compare view. Langfuse also includes a native comment feature for text annotations.

  6. Langfuse Core Capabilities

    main

    Langfuse is an open-source observability platform designed for engineering teams to treat LLM traces like standard application logs. Key capabilities include:

    • Observability: Captures the full lifecycle of LLM interactions, including prompts, model responses, tool calls, and latency.
    • Engineering-First Workflow: Supports complex debugging (such as tracing chain-of-thought), prompt management, and diverse evaluation methods (human feedback, SDK methods, or LLM-as-a-Judge).
    • Scalability: High-performance ClickHouse-based architecture designed to handle millions of events with minimal latency in production environments.

  7. Alternatives to the Public API for data export

    main

    If you need to export or access Langfuse data, you can use several alternatives to direct Public API calls:

    • Query via SDKs: Use typed Python or JS/TS wrappers for the same endpoints to simplify development.
    • Langfuse UI: Perform manual batch-exports directly from the user interface.
    • Blob Storage: Set up scheduled, automated exports to cloud storage for long-term archival or external processing.

  8. Common production metrics to track in Langfuse

    main

    When performing online evaluation (monitoring agents in production), track these key metrics:

    1. Costs: Monitor token usage per model (e.g., gpt-4o) to identify and optimize expensive steps.
    2. Latency: Observe the time taken for individual steps or the entire run to identify bottlenecks.
    3. User Feedback: Capture direct user signals (thumbs up/down) to refine agent behavior.
    4. LLM-as-a-Judge: Use automated LLM evaluations for real-time quality checks (toxicity, correctness, etc.).

  9. What is Autoresearch

    main
    Autoresearch is an automated experimentation loop designed to optimize code or AI skills. It uses an outer "optimizer" agent to read a target file, generate hypotheses for improvement, and apply changes. An inner loop then executes the modified file against an evaluator that scores the result. If the score improves, the change is kept; otherwise, it is discarded. This allows for high-speed, hands-off iteration (e.g., ~12 experiments per hour).

  10. What is the Langfuse v4 observations-first data model?

    main

    Langfuse v4 introduces an observations-first data model built on a new, wide, and mostly immutable ClickHouse table.

    Key Benefits

    • Performance: Eliminates joins and deduplication at read time, optimizing for ClickHouse access patterns. This results in 10x+ faster dashboard loads for large projects and reduces initial table loads from seconds to milliseconds.
    • New Features: The new model enables features such as:
      • Full-text search across inputs, outputs, and metadata.
      • Filter search bar.
      • Alerts.
      • Faster Observations API v2 and Metrics API v2.

  11. Run experiments using the Experiment runner SDK

    main

    The Experiment runner SDK (available in Python and JS/TS) provides a high-level abstraction for running experiments on both local data and datasets hosted on Langfuse. It automatically handles concurrent execution, automatic tracing for observability, flexible evaluation (item-level and run-level), error isolation, and dataset integration.

    Key Benefits

    • Concurrent execution: Configurable limits on parallel tasks.
    • Automatic tracing: Every execution is traced for debugging.
    • Flexible evaluation: Supports both item-level and aggregate run-level evaluators.
    • Error isolation: Individual task failures do not stop the entire experiment.

  12. Handle null and empty values in `observations_v2/` JSON/JSONL

    main

    In the enriched observations_v2/ export (JSON/JSONL), unset values are mapped to specific defaults. This is important for data processing pipelines to distinguish between a literal null and an empty string or zero.

    Value in observations_v2/Fields
    "" (empty string)input, model_id, model_parameters, output, parent_observation_id, prompt_id, prompt_name, provided_model_name, status_message, version
    nullcompletion_start_time, end_time, input_price, latency, output_price, prompt_version, time_to_first_token, total_price, usage_pricing_tier_name
    0total_cost (Note: 0 cannot be distinguished from a genuine zero cost as it reads from cost_details['total'])