NVIDIA Workbench Example Hybrid RAG

This guide demonstrates how to run RAG using NVIDIA Inference Endpoints hosted on the NVIDIA API Catalog.

Prerequisites

• An NGC account to generate an NVIDIA API Key.
• A valid NVIDIA_API_KEY obtained from an NVIDIA API catalog model card.

Setup Steps

1. Install and Open: Install NVIDIA AI Workbench and open it.
2. Clone Project: In AI Workbench, click Clone Project and enter the repository URL. Wait for the environment build to complete.
3. Configure Secrets: On the Project Dashboard, resolve the NVIDIA_API_KEY warning by inputting your key.
4. Launch App: Select Open Chat in the top right to open the Gradio app in your browser.

Using the Chat App

1. Initialize Backend: Click Set up RAG Backend in the Gradio UI.
2. Select Mode: Choose the Cloud option, select a model family and name, and submit a query.
3. Upload Documents:
   • Click Upload Documents Here in the right-hand panel.
   • Select Click to Upload and choose your text files.
   • Once uploaded, the Toggle to Use Vector Database next to the text input will activate.
4. Query: Submit questions to query your uploaded documents.

Use this mode to run models directly on your local hardware. This requires a GPU with sufficient VRAM depending on the model size and quantization level.

Hardware Requirements

Configuration for Gated Models

If you want to use gated models (e.g., Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.1), you must provide a Hugging Face API token:

1. Stop the project container in the Project Dashboard.
2. Navigate to Environment > Project Container > Variables > Add.
3. Add a new variable:
   • Name: HUGGING_FACE_HUB_TOKEN
   • Value: Your Hugging Face API Key
   • Mark as: Sensitive
4. Restart the Chat application.

Setup Steps

1. Open the Chat UI via the Open Chat button.
2. Click Set up RAG Backend (one-time initialization).
3. Under Inference Settings > Inference Mode, select Local System.
4. Select a model from the dropdown or input a custom Hugging Face model path.
5. Select a quantization level (supported: full, 8-bit, and 4-bit bitsandbytes).
6. Click Load Model to download/cache the weights.
7. Click Start Server to begin inference.

Using RAG

1. In the Chat UI, select Upload Documents Here and upload text files.
2. Once uploaded, the Toggle to Use Vector Database will turn on by default.
3. Query your documents.

Variable Name: HUGGING_FACE_HUB_TOKEN
Variable Value: (Your HF API Key)

If you encounter issues while using this Hybrid RAG example in AI Workbench, follow these steps:

Cloning the Repository

1. Open AI Workbench.
2. Select a location (e.g., Local).
3. Click Clone Existing Project (if first project) or Clone Project (top right).
4. Enter the repository URL and click Clone.

Monitoring Container Build

After cloning, the container build process may take several minutes.

• Check the Build Status widget at the very bottom of the Workbench window.
• Expand the widget to see real-time build output.
• Wait until the widget displays Build Ready before proceeding.

Running the Chat Application

Once the build is complete (Build Ready), click the green Open Chat button located at the top right of the interface.

Debugging the Chat Application

If the Chat application fails or behaves unexpectedly:

1. Locate the Output widget at the bottom left of the AI Workbench window.
2. Click to expand it.
3. Use the dropdown to navigate to Applications > Chat to view real-time debug messages.

Customizing the Project

To modify the code or configuration:

• Ensure the container is built.
• Click the dropdown next to the Open Chat button at the top right.
• Select JupyterLab to start editing. You can also configure VSCode support via the AI Workbench documentation.

The compose.yaml file defines the deployment configuration for running a local NVIDIA NIM (NVIDIA Inference Microservice) container. To use this configuration, ensure you have the NVIDIA Container Toolkit installed and an NVIDIA_API_KEY set in your environment.

Key configuration details:

• Image: Uses nvcr.io/nim/meta/llama-3.1-8b-instruct:latest.
• Runtime: Must be set to nvidia to enable GPU acceleration.
• GPU Reservation: Requires a reservation for 1 device with gpu capabilities.
• Ports: Maps host port 8000 to container port 8000.
• Volumes: Binds /tmp on the host to /opt/nim/.cache/ in the container for caching.
• Environment Variables: Requires NGC_API_KEY, which is sourced from the host's NVIDIA_API_KEY environment variable.

services:
  local-nim:
    image: nvcr.io/nim/meta/llama-3.1-8b-instruct:latest
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [ gpu ]
    ports:
      - "8000:8000"
    volumes:
      - type: bind
        source: /tmp
        target: /opt/nim/.cache/
    environment:
      - NGC_API_KEY=${NVIDIA_API_KEY:?Error NVIDIA_API_KEY not set}

networks:
  default:
    name: hybrid-rag