TruLens Documentation

The Snowflake AI Stack example provides a template for building production-ready Retrieval Augmented Generation (RAG) systems. Key features include:

• Flexible Knowledge Base: Supports various data sources via interchangeable loaders.
• Performant Retrieval: Utilizes Arctic Embed for high-performance similarity search.
• Hallucination Prevention: Employs TruLens LLM Judge to filter retrieved context before the generation step.
• Observability: Integrates TruLens for tracing and evaluating application runs, with results viewable in an interface.
• Interactive UI: A Streamlit-based interface supporting multi-turn chat and streaming generation.

TruLens apps are built upon AppDefinition and App interfaces. They are categorized into Core Apps (included in the main installation) and Optional Apps (requiring additional packages).

Core Apps

• TruBasicApp: A basic app implementation.
• TruApp: A standard app implementation.
• TruVirtual: A virtual app implementation.

Optional Apps

• TruChain: For LangChain applications (requires trulens-apps-langchain).
• TruLlama: For LlamaIndex applications (requires trulens-apps-llamaindex).
• TruRails: For NVIDIA NeMo applications (requires trulens-apps-nemo).

TruLens constructs feedback functions by combining feedback providers (general models) with feedback implementations (specific prompts and logic). Feedback providers are categorized into three main types:

1. Classification-based Providers: Used for tasks like moderation or specific classification, often using models that aren't full LLMs.
2. Generation-based Providers: Use Large Language Models (LLMs) to evaluate feedback.
3. Embedding-based Providers: Use embeddings for similarity or semantic evaluation.

The TruLens examples repository is organized into four distinct categories based on stability and purpose:

• Quickstarts: Actively maintained examples for critical workflows (building, evaluating, and tracking LLM apps). These are the recommended starting point and are featured in the TruLens documentation under 'Getting Started'.
• Expositions: Verified examples organized by component (e.g., /models, /frameworks, /vector-dbs) or /use_cases. These are part of the 'TruLens cookbook' and are verified against specific dependency sets updated at major releases.
• Experimental: Examples that may break between releases.
• Dev: Examples used for developing or testing new releases.

TruLens constructs feedback functions using two primary components: a feedback provider (an instance of the trulens.core.feedback.Provider class) and a feedback implementation.

Feedback functions are composed of specific prompts and custom logic designed for particular evaluation tasks. There are two main types of implementations:

1. Generation-based feedback implementations: These use a generative model (LLM) to perform evaluations based on instructions (system messages), templates (user messages), and custom preprocessing/postprocessing logic.
2. Classification-based Providers: These rely on specific classification models (often tailor-made for evaluation) rather than general-purpose LLMs, involving preprocessing and postprocessing logic around the model call.

This project supports two distinct modes of operation for evaluating and monitoring the agent:

1. Batch Evaluation (run_eval.py): Used for running evaluations on a pre-collected dataset with metric computation.
2. Production Monitoring (server.py): A FastAPI backend used for real-time monitoring of the agent in production using @trace_with_run.

The RAG Triad is a framework used by TruLens to evaluate Retrieval-Augmented Generation (RAG) applications for hallucinations. It evaluates three specific edges of the RAG architecture to ensure correctness:

1. Context Relevance: Verifies that the retrieved context chunks are relevant to the input query. This ensures the LLM is provided with useful information rather than noise.
2. Groundedness: Verifies that the LLM's response is strictly supported by the retrieved context. This is done by breaking the response into claims and searching for evidence within the context to prevent the LLM from exaggerating or straying from facts.
3. Answer Relevance: Verifies that the final response actually addresses the user's original query.

Achieving satisfactory scores across all three metrics provides confidence that the application is hallucination-free relative to its knowledge base.

TruLens implements the 'Honest, Harmless, and Helpful' framework (adapted from Anthropic) as a suite of feedback functions to evaluate LLM application performance and alignment.

• Honest: Focuses on accuracy and the reliable retrieval and use of information to prevent hallucinations.
• Harmless: Focuses on preventing offensive, discriminatory, or biased content, ensuring the AI refuses dangerous requests, and acting with appropriate modesty regarding sensitive advice.
• Helpful: Focuses on the AI's ability to perform tasks concisely and efficiently, responding in the correct language and a helpful tone.

TruLens provides two primary mechanisms for performing evaluations during the execution of an LLM application to influence or control the application's behavior:

1. In-line Evaluations: These are executed during the agent's execution flow. The results are passed back to the agent, allowing it to use the evaluation feedback to assist in orchestration and decision-making.
2. Guardrails: These are used to monitor and potentially block inputs, outputs, or intermediate results produced by applications (such as RAG systems or agents) to ensure safety and quality.

If you require trulens-apps-nemo, you must use a configuration that uses langchain < 1.0, as nemoguardrails is currently incompatible with LangChain 1.x. Use the following command to install the compatible version:

pip install trulens[langchain,nemo]