DOCSEARCHbyAlgolia
Submit Docs

LangSmith Cookbook

repository·main·Indexed 21 days ago

https://github.com/langchain-ai/langsmith-cookbook

A collection of practical recipes and real-world use cases for mastering LangSmith. It provides guidance on tracing, testing, evaluating, and optimizing LLM applications using Python and TypeScript/JavaScript, featuring examples for Next.js and Streamlit, dataset enrichment with Lilac, and OpenAI fine-tuning workflows.

Tokens
60.5K
Snippets
189
Records
229
Agent score
76%

What's inside LangSmith Cookbook

  1. Explore LangSmith feedback implementation examples

    main

    The LangSmith cookbook provides several patterns for harnessing user feedback, AI-assisted feedback, and other signals to improve and monitor applications. You can explore these patterns through the following implementations:

    Python / Streamlit Examples

    • Minimal Chat App: A Streamlit application that captures user feedback and shares traces. It includes both a vanilla_chain.py (using LLMChain) and an expression_chain.py (using LangChain Expression Language).
    • Real-time RAG Chat Bot Evaluation: A Streamlit walkthrough demonstrating how to automatically check for hallucinations in RAG responses against retrieved documents.
    • LangChain Agents: An implementation showing how to instrument a web-search agent with tracing and human feedback.

    TypeScript / Next.js Examples

    • Next.js Chat App: A simple TypeScript chat application demonstrating tracing and feedback capture.

    Automated & Algorithmic Feedback

    • Algorithmic Feedback Pipeline: An automated approach to feedback metrics for monitoring and performance tuning.
    • Real-time Automated Feedback: Uses an async callback to automatically generate feedback metrics for every run, allowing for real-time evaluation of production runs.

  2. JavaScript Testing & Evaluation Examples

    main

    This section of the cookbook provides examples for incorporating LangSmith into TypeScript/JavaScript testing and evaluation workflows. Currently, it features vision-based evaluation techniques.

    Vision-based Evals in JavaScript

    You can use GPT-4V to evaluate AI-generated UIs by following the vision-evals guide.

    External JS Evaluation Guides

    Since the cookbook is expanding its JS examples, you can refer to the following official LangChain JS and LangSmith documentation for comprehensive quickstarts:

  3. Use LangChain Hub to manage LLM components

    main

    The LangChain Hub (https://smith.langchain.com/hub) allows you to efficiently manage, version, and deploy LLM components like prompts. You can use the hub to:

    • Integrate prompts into RAG pipelines: Pull pre-configured prompts (e.g., for RetrievalQA) directly into your application.
    • Implement Prompt Versioning: Instead of pulling the 'latest' version, select specific prompt versions to ensure deployment stability and prevent breaking changes when prompts are updated.
    • Streamline Prompt Development: Save prompts directly from the LangSmith playground to the hub and integrate them into your code as RunnablePromptTemplate objects.

  4. Backtest production applications

    main

    To benchmark new versions of a production application, follow the backtesting pattern:

    1. Convert existing production runs into a test dataset.
    2. Run your new system against this dataset.
    3. Compare the performance of the new system against the baseline established by the production runs.

  5. Automate feedback with algorithmic pipelines

    main

    LangSmith supports both manual user feedback and automated 'algorithmic' feedback. You can:

    • Batch Processing: Build an algorithmic feedback pipeline to evaluate production runs as a batch job.
    • Real-time Feedback: Use an async callback to automatically generate feedback metrics for every run in real-time.

  6. Trace nested tool calls within a single trace

    main
    To ensure that all subcalls made by a tool are included within the parent trace rather than appearing as separate, disconnected runs, use run_manager.get_child() to create a child run and pass its callbacks to the tool's execution.

  7. Implement custom evaluators with RunEvaluator

    main

    To perform specialized monitoring, you can implement custom evaluators by subclassing RunEvaluator. This allows you to extract specific data from the Run object (like inputs, outputs, or child runs) and pass them to a scoring engine.

    Relevance Evaluator

    Used to grade a response based on the user's question and chat history, without considering retrieved documents. This helps detect if the bot is being overly influenced by retrieved content (prompt injection).

    Faithfulness Evaluator

    Used to penalize contradictory or off-topic information. This evaluator can traverse the trace to find specific child runs (e.g., a retriever named RetrieveDocs) to extract the reference documents used for the response.

    Each evaluator's evaluate_run method should return an EvaluationResult containing the score and a comment (reasoning).

    class RelevanceEvaluator(RunEvaluator):
    def __init__(self):
        self.evaluator = load_evaluator(
            "score_string", criteria="relevance", normalize_by=10
        )

    def evaluate_run(
        self, run: Run, example: Optional[Example] = None
    ) -> EvaluationResult:
        try:
            text_input = (
                get_buffer_string(run.inputs["chat_history"])
                + f"\nhuman: {run.inputs['query']}"
            )
            result = self.evaluator.evaluate_strings(
                input=text_input, prediction=run.outputs["output"]
            )
            return EvaluationResult(
                **{"key": "relevance", "comment": result.get("reasoning"), **result}
            )
        except Exception as e:
            return EvaluationResult(key="relevance", score=None, comment=repr(e))

  8. Capture Run IDs using collect_runs

    main

    To associate user feedback with a specific LLM execution, you need the run_id. In the Streamlit example, the collect_runs() context manager is used to capture the trace of a streaming chain in memory, allowing you to retrieve the id of the traced run.

    with collect_runs() as cb:
    for chunk in chain.stream(input_dict, config={"tags": ["Streamlit Chat"]}):
        # ... process chunks ...
    run_id = cb.traced_runs[0].id

  9. Evaluate Chat Bots using Simulated Users or Single-turn Evals

    main

    There are two primary patterns for evaluating chatbots in this cookbook:

    1. Simulated Users: Evaluate a chatbot by giving a simulated user a specific task. The user scores the assistant based on task completion and adherence to instructions.
    2. Single-turn Evals: Evaluate multi-turn conversations by treating each data point as an individual dialogue turn. This involves setting up a multi-turn conversation dataset and evaluating the bot's response at each step.

  10. Evaluate Agent tool use and intermediate steps

    main

    To evaluate agents, you can use two specific strategies:

    1. Intermediate Steps: Compare the sequence of actions taken by an agent against an expected trajectory to grade the effectiveness of tool use.
    2. Tool Selection: Evaluate the precision of the tools selected by the agent. You can include an automated prompt writer to iteratively improve tool descriptions based on identified failure cases.

  11. Manage and version prompts with LangChain Hub

    main

    The LangChain Hub allows you to store, version, and retrieve LLM prompts. Key patterns include:

    • RetrievalQA Chain: Pulling prompts directly from the hub to use in RAG pipelines.
    • Prompt Versioning: Instead of using the 'latest' tag, select specific prompt versions to ensure deployment stability and prevent breaking changes when prompts are updated.
    • Runnable PromptTemplate: Saving prompts directly from the LangChain playground to the hub and integrating them into runnable chains.