DOCSEARCHbyAlgolia
Submit Docs

LangSmith Documentation

repository·main·Indexed 41 days ago

https://github.com/langchain-ai/langsmith-docs

A platform for debugging, testing, and monitoring LLM applications. Documentation covers resource hierarchy (Organizations and Workspaces), authentication via Personal Access Tokens (PATs) and Service Keys, Enterprise RBAC, resource tagging, data retention tiers, usage limits, and rate limit handling.

Tokens
244.6K
Snippets
511
Records
897
Agent score
51%

What's inside LangSmith

  1. Overview of LangSmith Evaluation How-To Guides

    main

    LangSmith provides a comprehensive suite of evaluation capabilities divided into two main workflows: Offline Evaluation and Online Evaluation.

    Offline Evaluation

    Used to evaluate and improve applications before deployment. This includes:

    • Running evaluations via the SDK (Python/JS) or the UI.
    • Defining custom evaluators (LLM-as-a-judge, custom code, pairwise, or summary).
    • Configuring evaluation data (splits, filtered views, or specific dataset versions).
    • Managing evaluation jobs (repetitions, rate limiting, and local execution).
    • Binding default evaluators to datasets to automate scoring for all future experiments.

    Online Evaluation

    Used to monitor live performance on production data using:

    • LLM-as-judge evaluators.
    • Custom code evaluators.

    Feedback and Annotation

    LangSmith allows for human-in-the-loop improvements through:

    • User Feedback: Logging feedback directly from your application.
    • Annotation Queues: Using expert feedback to improve datasets.
    • Manual Annotation: Annotating traces inline within the LangSmith UI.

  2. Overview of Self-Hosting LangSmith

    main

    Self-hosting LangSmith allows you to deploy the platform on your own infrastructure. The documentation provides comprehensive guides for the entire lifecycle of a self-hosted instance, including:

    • Architecture: Understanding the services and storage requirements (ClickHouse, Postgres, Redis, etc.).
    • Installation: Deployment via Kubernetes or Docker.
    • Configuration: Setting up SSO (OAuth2.0/OIDC), external databases (Postgres, ClickHouse, Redis), blob storage, and authentication methods.
    • Operations: Upgrading instances, managing TTLs (Time To Live), and monitoring via observability stacks.
    • Observability: Exporting telemetry (logs, metrics, traces) using OpenTelemetry (OTel) collectors.

  3. Authentication methods for Self-Hosted LangSmith

    main

    Self-hosted installations offer granular control over authentication. Depending on your environment and security requirements, you can choose from the following methods:

    • SSO with OAuth 2.0 and OIDC: Recommended for production. This allows integration with external identity providers like Auth0 or Okta. LangSmith supports any OIDC-compliant provider.
    • Email/Password (Basic Auth): Requires minimal configuration and does not need an external identity provider. This is best suited for self-hosted trials.
    • None: Disables all authentication. This mode is restricted to a single organization and workspace and should only be used for infrastructure-level installation verification. Note: This mode is scheduled for removal after the launch of Basic Auth.

  4. Manage prompts in the LangSmith Prompt Hub

    main

    The LangSmith Prompt Hub allows you to organize, manage, and version prompts to streamline LLM development. You can perform the following tasks:

    • Create and Update: Manually create or modify prompts within the UI.
    • Programmatic Management: Use the LangSmith API to manage prompts via code.
    • Versioning and Tagging: Use commit tags to track changes and manage prompt versions.
    • Automation: Trigger webhooks automatically whenever a prompt is committed.
    • Public Sharing: Use the LangChain Hub to share prompts publicly.

  5. Organization Management how-to guides

    main

    LangSmith provides several administrative guides for managing your organization, including account setup, workspace configuration, billing, and access control. Key administrative tasks include:

    Account & Organization Setup

    • Create an account and API key: Initial setup for LangSmith access.
    • Set up an organization: Includes creating an organization, managing/navigating workspaces, managing users, and managing the organization via the API.
    • Set up a workspace: Includes creating workspaces, managing users within a workspace, configuring workspace settings, and deleting workspaces.

    Billing & Business Information

    • Set up billing: Configure payment methods and plans.
    • Update business information: Update invoice email, tax ID, and other business details.

    Access Control & Governance

    • Set up access control (Enterprise only): Create custom roles and assign roles to users.
    • Set up resource tags: Create, assign, delete, and filter resources using tags.
    • Data purging for compliance: Manage data retention and purging for regulatory compliance.

  6. Get started with LangSmith

    main

    LangSmith is a platform for building production-grade LLM applications, providing tools for monitoring, evaluation, and prompt engineering. It is framework-agnostic and can be used with or without LangChain or LangGraph.

    Core capabilities include:

    • Observability: Analyze traces, configure metrics, dashboards, and alerts.
    • Evals: Evaluate application performance over production traffic using automated scoring or human feedback.
    • Prompt Engineering: Iterate on prompts with automatic version control and collaboration features.

  7. Bulk Exporting Trace Data in LangSmith

    main

    LangSmith allows bulk exporting traces to external destinations like BigQuery, Snowflake, RedShift, or Jupyter Notebooks for offline analysis.

    Key Details:

    • Plan Requirement: Only available for LangSmith Plus or Enterprise tiers.
    • Format: Data is exported in Parquet columnar format.
    • Scope: You can target specific LangSmith projects and date ranges.
    • Constraints: Exports have a runtime timeout of 24 hours and are subject to concurrency limits.

  8. Iterate on prompts using the LangSmith Playground

    main

    The LangSmith Playground is a web-based environment for rapid prompt and model iteration. Key capabilities include:

    • Evaluation: Run evaluations directly within the playground UI.
    • Multimodal Support: Include multimodal content (like images) in your prompts.
    • Configuration: Manage model settings and configurations.
    • Tool Use: Test how models interact with tools.
    • Prompt Canvas: Use the Prompt Canvas for advanced prompt iteration.
    • Custom Endpoints: Connect the playground to a custom LangServe model server or any OpenAI-compliant model provider/proxy.
    • Multi-turn Conversations: Use multiple messages to simulate chat interactions.

  9. Calculate token-based costs for traces

    main

    LangSmith tracks token-based costs for LLM runs, rolling them up to the trace and project levels. There are two primary methods for tracking costs:

    1. Derived from token counts and model prices: You provide token counts in the run data, and LangSmith calculates the cost using a configured model pricing map. This is the recommended approach for most linear pricing models.
    2. Directly specified: You provide the exact cost as part of the run data. This is recommended for models with non-linear pricing (e.g., price changes after a certain token threshold) or non-token-based pricing schemes.

    If token counts are not explicitly provided, LangSmith approximates them using tiktoken.

  10. Access telemetry data for self-hosted LangSmith

    main

    For self-hosted LangSmith deployments, you can access telemetry data (logs, metrics, and traces) through several methods:

    1. Exporting Telemetry: Export logs, metrics, and traces to a custom collector or backend of your choice.
    2. Configuring an OTel Collector: Use OpenTelemetry (OTel) collector configurations to gather LangSmith telemetry data.
    3. Deploying a Managed Observability Stack: Use LangSmith to deploy a basic observability stack that allows you to view logs, metrics, and traces for your deployment directly.

  11. Authentication methods in LangSmith Cloud

    main

    LangSmith Cloud provides several ways for users to access the platform:

    • Email/Password: Standard sign-up and login using an email address and password.
    • Social Providers: Authentication via GitHub or Google credentials.
    • SAML SSO: Enterprise customers can configure SAML SSO and SCIM for centralized identity management and user provisioning.

  12. What is backtesting and how does it work

    main

    Backtesting is the process of assessing new versions of an application (enhanced prompts, models, tools, or architectures) by comparing their outputs against historical production data. This provides a clearer indication of improvement compared to using pre-production datasets.

    The Backtesting Workflow:

    1. Select sample runs from your production tracing project.
    2. Transform runs into a dataset and record the original outputs as a baseline experiment.
    3. Execute the new system on that same dataset and compare the results using evaluators.