DOCSEARCHbyAlgolia
Submit Docs

NVIDIA Agentic RAG Example

repository·main·Indexed 41 days ago

https://github.com/nvidia/workbench-example-agentic-rag

An example project demonstrating how to enhance Retrieval-Augmented Generation with an agentic reasoning layer. The system can route queries to live web searches via Tavily, evaluate responses for accuracy to prevent hallucinations, and iterate on answers. It supports deployment via NVIDIA AI Workbench with options for cloud endpoints, self-hosted Ollama containers, or NVIDIA NIM.

Tokens
7.1K
Snippets
19
Records
26
Agent score
51%

What's inside Agentic RAG

  1. How the Agentic RAG layer works

    main

    The Agentic RAG system enhances standard Retrieval-Augmented Generation by adding an agentic reasoning layer that performs three main functions:

    1. Route: Evaluates the RAG context for relevance to the user query. If the context is insufficient (thin), it triggers a live web search.
    2. Evaluate: Inspects generated responses for relevance and accuracy, specifically flagging potential hallucinations.
    3. Iterate: Executes multiple cycles of generation and evaluation to refine the final answer.

    This layer allows the system to bridge gaps in local document context using real-time web data while maintaining accuracy through self-correction.

  2. Quickstart Agentic RAG in Easy Mode

    main

    Use this mode to run the application immediately using free cloud endpoints without modifying code.

    Prerequisites

    • NVIDIA AI Workbench installed.
    • Internet Connection (required for document embedding via NVIDIA endpoints).

    Setup Steps

    1. Obtain API Keys:
    2. Clone Project: Use NVIDIA AI Workbench to clone the repository: https://github.com/NVIDIA/workbench-example-agentic-rag.
    3. Configure Environment Variables: When prompted by Workbench, provide your NVIDIA_API_KEY and TAVILY_API_KEY. If the prompt doesn't appear, navigate to Project Container > Variables > Configure in the Workbench Desktop App.
    4. Run the App:
      • Click Open Chat.
      • Navigate to the Document tab in the web app.
      • Click Add to Context to ingest documents/URLs.
      • Type your question and press Enter.

  3. Modify Vector Database clearing behavior

    main

    By default, clearing the vector database performs a full clearing (deleting the collection and all associated files). To preserve previous searches and files in the persist directory, modify the _clear() function in code/chatui/utils/database.py.

    Change the delete_all parameter from True to False:

    def _clear(
    persist_directory: str = "/project/data",
    collection_name: str = "rag-chroma",
    delete_all: bool = False  # Changed from True to False
):

    Note: Hidden files (starting with '.') are always preserved. The current collection will still be cleared even if delete_all is False.

    def _clear(
    persist_directory: str = "/project/data",
    collection_name: str = "rag-chroma",
    delete_all: bool = False
):

  4. Self-host inference using NVIDIA NIM (Advanced)

    main

    NVIDIA NIM (NVIDIA Inference Microservices) provides optimized, production-ready inference. It offers better performance than Ollama but requires more complex setup and typically higher GPU memory (24GB+ vRAM).

    Setup Workflow

    1. Deploy NIM Container: Pull and run the NIM container on your remote GPU machine using an NVIDIA NGC API key.
    2. Configure NIM Container: The container automatically downloads and configures the model on startup.
    3. Add as Endpoint: In AI Workbench, select Models > Self-Hosted Endpoint and provide the remote IP, port, and model name.
    # 1. Login to NVIDIA NGC
docker login nvcr.io
# Use $oauthtoken as username and your NGC key as password

# 2. Pull and Run NIM
docker pull nvcr.io/nim/deepseek-r1-distill-llama-8b:latest

docker run -d --name nim \
  --gpus all --restart unless-stopped \
  -p <remote-port>:8000 \
  -e NGC_API_KEY=<your-ngc-api-key> \
  nvcr.io/nim/deepseek-r1-distill-llama-8b:latest

  5. Deploy and verify NVIDIA NIM container

    main

    NIM deployment requires an NVIDIA NGC API key passed as an environment variable (NGC_API_KEY). The container maps the remote port to internal port 8000. NIM uses an OpenAI-compatible API format.

    # Login to NVIDIA NGC
docker login nvcr.io

# Pull and run the NIM container
docker pull nvcr.io/nim/deepseek-r1-distill-llama-8b:latest

docker run -d --name nim \
  --gpus all --restart unless-stopped \
  -p <remote-port>:8000 \
  -e NGC_API_KEY=<your-ngc-api-key> \
  nvcr.io/nim/deepseek-r1-distill-llama-8b:latest

# Verify running
docker ps | grep nim

# Test the OpenAI-compatible endpoint
curl -X POST http://localhost:<remote-port>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "Hello, NIM!"}
    ]
  }'

  6. Enable higher resolution error messaging in the Monitor tab

    main

    By default, the Monitor tab only displays stdout messages. To capture stderr (error messages) or all output for debugging, you must modify the Logger class in code/chatui/utils/logger.py and update the stream assignments in code/chatui/pages/converse.py.

    Option 1: Optional Error Capture

    Modify the Logger class to include a capture_errors flag. This allows you to log stderr only when explicitly requested.

    Option 2: Capture All Output

    Modify the Logger class to include a capture_all flag. This removes filtering and logs everything (including debug messages and internal processing) to the log file.

    Caveats

    • Capturing all output or errors can create significant noise in the Monitor tab.
    • Some error messages may be expected and not represent actual issues.
    • Log files will grow larger when capturing more output.
    • It is recommended to use these settings temporarily for debugging rather than as a permanent configuration.
    # Example: Enabling error capture in code/chatui/pages/converse.py
sys.stdout = logger.Logger("/project/code/output.log", 'stdout')
sys.stderr = logger.Logger("/project/code/output.log", 'stderr', capture_errors=True)

# Example: Enabling capture of all output in code/chatui/pages/converse.py
sys.stdout = logger.Logger("/project/code/output.log", 'stdout', capture_all=True)
sys.stderr = logger.Logger("/project/code/output.log", 'stderr', capture_all=True)

  7. Modify Agentic RAG in Intermediate Mode

    main

    Use this mode to customize the application logic, prompts, or UI. This is intended for developers who want to prototype improvements.

    Workflow

    1. Fork the repository to your GitHub account.
    2. Clone the fork into NVIDIA AI Workbench.
    3. Add VS Code to the project within Workbench.
    4. Create an experiment branch to protect the main branch.
    5. Edit the code via VS Code to perform tasks such as:
      • Adjusting recursion limits or the number of web sites returned by Tavily.
      • Adding new endpoints from build.nvidia.com.
      • Modifying the agent's behavior or prompts.
      • Customizing the Gradio UI look and feel.

  8. Add a new NVIDIA endpoint to the Model dropdown

    main

    To add a new NVIDIA model to the Gradio interface's Models tab, you must update code/chatui/pages/converse.py. This makes the model available across all pipeline components.

    1. Find the model path: Get the provider-model path from build.nvidia.com (e.g., nvidia/llama-3_1-nemotron-ultra-253b-v1).
    2. Define the identifier: In code/chatui/pages/converse.py, under the Model identifiers with prefix section, define a new variable:
      NEMO = "nvidia/llama-3_1-nemotron-ultra-253b-v1"
    3. Update the dropdown list: In the build_page() function, add your new variable to the model_list:
      model_list = [LLAMA, MISTRAL, NEMO]
    4. (NVIDIA Internal Only): If using INTERNAL_API, update the logic in the # Modify model identifiers section to prepend the internal endpoint prefix:
      if INTERNAL_API != '':
    NEMO = f'{INTERNAL_API}/nvidia/llama-3_1-nemotron-ultra-253b-v1'
    NEMO = "nvidia/llama-3_1-nemotron-ultra-253b-v1"
model_list = [LLAMA, MISTRAL, NEMO]

  9. Modify Tavily Search Settings

    main

    The TAVILY_K parameter controls the number of search results returned from Tavily and the breadth of information available to the agent. This is configured in code/chatui/utils/graph.py.

    Option 1: Environment Variable (Recommended)

    Set the TAVILY_K variable in a terminal attached to your container. The app must be restarted for this to take effect.

    export TAVILY_K=5

    Option 2: Direct Code Modification

    In graph.py, update the DEFAULT_TAVILY_K value:

    DEFAULT_TAVILY_K = 3
TAVILY_K = int(os.getenv("TAVILY_K", DEFAULT_TAVILY_K))

  10. Modify the Agent's Recursion Limit

    main

    The recursion limit controls how many times the agent can re-route questions, re-try generation, or iterate through retrieval/grading. You can modify this in code/chatui/pages/converse.py.

    Option 1: Environment Variable (Recommended)

    Set the RECURSION_LIMIT variable in a terminal attached to your container. The app must be restarted for this to take effect.

    export RECURSION_LIMIT=20

    Option 2: Direct Code Modification

    In converse.py, update the DEFAULT_RECURSION_LIMIT value:

    DEFAULT_RECURSION_LIMIT = 10
RECURSION_LIMIT = int(os.getenv("RECURSION_LIMIT", DEFAULT_RECURSION_LIMIT))

  11. Self-host inference using Ollama (Intermediate)

    main

    Ollama is an easier-to-manage option for running containerized inference on a remote GPU. It is suitable for experimentation and development, requiring as little as 8 GB of vRAM depending on the model.

    Setup Workflow

    1. Deploy Ollama Container: Pull and run the container on your remote GPU machine.
    2. Pull Model: Execute a command inside the container to load your desired model (e.g., llama2:7b).
    3. Add as Endpoint: Configure the Agentic RAG app in AI Workbench to point to your remote IP, port, and model name using the Self-Hosted Endpoint option.
    # 1. Deploy Ollama Container on the remote
docker pull ollama/ollama:latest

docker run -d --name ollama \
  --gpus all --restart unless-stopped \
  -p <remote-port>:11434 \
  -v ollama_data:/root/.ollama \
  ollama/ollama:latest

# 2. Pull a model into the container
docker exec ollama ollama pull llama2:7b

  12. Deploy and verify Ollama container

    main

    To deploy Ollama on a remote machine, use Docker to map a local <remote-port> to the container's internal port 11434. Ensure you use the --gpus all flag to enable GPU acceleration.

    # Pull the container
docker pull ollama/ollama:latest

# Run the container
docker run -d --name ollama \
  --gpus all --restart unless-stopped \
  -p <remote-port>:11434 \
  -v ollama_data:/root/.ollama \
  ollama/ollama:latest

# Verify it is running
curl http://localhost:11434/api/tags