DOCSEARCHbyAlgolia
Submit Docs

NVIDIA RAG Blueprint

repository·main·Indexed 20 days ago

https://github.com/nvidia-ai-blueprints/rag

A modular, enterprise-ready reference architecture for building Retrieval-Augmented Generation (RAG) pipelines. It utilizes NVIDIA NIM microservices for high-performance inference, retrieval, and extraction. The blueprint supports multiple deployment modes, including Docker, Kubernetes with Helm, a Python library mode, and a containerless Lite mode. It features customizable components for LLM/embedding models, data ingestion, vector databases (such as Milvus and Elasticsearch), and integration with NeMo Guardrails and the NeMo Agent Toolkit for ReAct agent workflows.

Tokens
229.6K
Snippets
566
Records
897
Agent score
71%

What's inside nvidia-ai-blueprints-rag

  1. Overview of NVIDIA RAG Blueprint Notebooks

    main

    The repository provides several categories of Jupyter notebooks for different development stages:

    Beginner Notebooks

    • ingestion_api_usage.ipynb: Interacting with the ingestion service to upload/process documents.
    • retriever_api_usage.ipynb: Using the retriever service with various query techniques.
    • image_input.ipynb: Multimodal query support (text + images) using VLM embeddings and visual queries.

    Intermediate Notebooks

    • summarization.ipynb: Customizing document summarization (page filtering, extraction strategies) using Library or Docker modes.
    • evaluation_01_ragas.ipynb: Evaluating RAG systems using the Ragas library.
    • evaluation_02_recall.ipynb: Measuring retrieval performance via recall metrics at various top-k thresholds.
    • nb_metadata.ipynb: Implementing metadata ingestion, filtering, and extraction for enhanced retrieval.
    • rag_library_usage.ipynb: End-to-end usage of the NVIDIA RAG Python client (ingestion, collection management, querying).
    • rag_library_lite_usage.ipynb: Containerless deployment using Milvus Lite and NeMo Retriever Library subprocess mode. Note: Does not support image/table/chart citations or summarization.
    • langchain_nvidia_retriever.ipynb: Integration with LangChain using NVIDIARAGRetriever for sync/async retrieval and ChatNVIDIA chaining.

    Advanced Notebooks

    • building_rag_vdb_operator.ipynb: Extending the system by building custom vector database (VDB) operators (e.g., OpenSearch) using the VDBRag base class.
    • mcp_server_usage.ipynb: Using the NVIDIA RAG Model Context Protocol (MCP) server via SSE, streamable-http, or stdio to call Ingestor and RAG tools.
    • nat_mcp_integration.ipynb: Integrating NeMo Agent Toolkit (NAT) with the MCP server to build intelligent agents via YAML-configured workflows.

    Deployment Notebooks

    • launchable.ipynb: A deployment-ready notebook designed for Brev environments.

  2. Overview of Filesystem RAG Benchmarks Skill

    main

    The Filesystem RAG benchmarks skill allows developers and engineers to run RAGAS quality benchmarks against NVIDIA RAG Blueprint deployments. It evaluates both retrieval and generation quality using a filesystem-based approach.

    Key Components

    • corpus/: Directory containing the reference data.
    • train.json: The training dataset.
    • evaluate_rag.py: The execution script used for RAGAS quality evaluation.

    Use Case

    Use this skill to perform dataset preparation, evaluation execution, and result analysis to measure the performance of your RAG implementation.

  4. What is the NVIDIA RAG Blueprint?

    main

    The NVIDIA RAG Blueprint is a reference solution and foundational framework for building Retrieval-Augmented Generation (RAG) pipelines using NVIDIA NIM microservices. It is designed to ground AI responses in enterprise knowledge to reduce hallucinations and ensure accuracy, compliance, and freshness.

    Key capabilities include:

    • Agentic RAG: A LangGraph-based plan-and-execute pipeline for complex, multi-hop, or ambiguous queries.
    • Multimodal Ingestion: Extraction of text, tables, charts, infographics, and audio from documents.
    • Hybrid Search: Support for dense and sparse search with GPU-accelerated indexing (via NVIDIA cuVS).
    • Enterprise Features: Reranking, query decomposition, programmable guardrails, and evaluation via the RAGAS framework.

  5. Overview of the RAG Performance Benchmarking Skill

    main

    The RAG Performance Benchmarking skill is used to profile and load test a deployed NVIDIA RAG Blueprint server. It combines a profiling pass with an aiperf load test, all driven by a single YAML configuration file.

    Developers and engineers use this skill to identify latency, throughput, and bottleneck characteristics of their RAG deployment under specific, configurable load patterns.

  6. Benchmark NVIDIA RAG Blueprint performance with `rag-perf`

    main

    Use the rag-perf CLI to measure the performance of your deployed NVIDIA RAG Blueprint system. It provides metrics on latency, throughput, and per-stage timing (retrieval, reranking, and LLM TTFT).

    rag-perf performs two distinct passes for every benchmark point:

    1. Profiling pass: Uses direct async httpx requests to capture server-side per-stage timing and identify bottlenecks (retrieval, reranking, or llm).
    2. Load-test pass: Uses aiperf to drive concurrent traffic, capturing TTFT (mean/p50/p90/p99), end-to-end latency, output-token throughput, request throughput, and error rates.

    Note: rag-perf measures speed and concurrency, whereas evaluate_rag.py (based on RAGAS) measures answer quality. They are complementary tools.

    uv run --project scripts/rag-perf rag-perf -c <path_to_config.yaml>

  7. Understand the NVIDIA RAG Blueprint source code structure

    main

    The core implementation of the NVIDIA RAG Blueprint is located in the src/nvidia_rag directory. The project is organized into four primary functional areas:

    1. ingestor_server/: Manages document ingestion, task submission, and NVIDIA-specific ingestion logic via a FastAPI server.
    2. rag_server/: The core RAG engine. It handles response generation, reflection (for relevance and groundedness), validation, and Vision Language Model (VLM) integration using a FastAPI server.
    3. utils/: A collection of shared utilities for LLM interaction, embedding management, vector storage, reranking, and object storage (S3-compatible or filesystem).
    4. observability/: Tools for monitoring the system, including OpenTelemetry instrumentation for Langchain and custom metrics collection.

  8. What is Query Decomposition and how does it work?

    main

    Query decomposition is an advanced Retrieval-Augmented Generation (RAG) technique used to boost accuracy for multi-hop reasoning or context-rich queries. It works by breaking down a complex, multi-faceted query into simpler, focused subqueries. Each subquery is processed independently to gather context, which is then synthesized into a final response.

    Core Algorithm

    1. Subquery Generation: Breaks the complex query into focused questions.
    2. Iterative Processing: For each recursion depth:
      • Processes each subquery independently.
      • Rewrites queries using accumulated context.
      • Retrieves and ranks relevant documents.
      • Generates focused answers.
      • Collects contexts for synthesis.
    3. Follow-up Generation: Creates follow-up questions for information missing from the original query.
    4. Termination Check: Stops if no follow-up is needed or the MAX_RECURSION_DEPTH is reached.
    5. Final Synthesis: Generates a comprehensive response from all collected contexts and the conversation.

    Limitations and Constraints

    • Knowledge Base Requirement: Query decomposition is not available for direct LLM calls (when use_kb=false). It requires knowledge base integration to process subqueries and retrieve documents.
    • Single Collection Only: It is currently limited to single collection operations. Multi-collection queries are not supported when enabled and will not function as expected.

  9. What is Agentic RAG and how does it work?

    main

    Agentic RAG is an advanced retrieval pattern that treats a query as a reasoning task rather than a single retrieval step. While standard RAG performs a one-pass (embed $\rightarrow$ retrieve $\rightarrow$ generate), Agentic RAG uses an LLM-driven agent to plan, execute, and verify answers.

    Core Workflow

    The pipeline is implemented as a LangGraph state machine with five stages:

    1. Initial Retrieval: Performs standard vector DB search and reranking to inform the planner about what is in the corpus.
    2. Planner (Two-phase): An LLM selects a plan shape:
      • Scope discovery: Probes the corpus for ambiguous queries to refine the plan.
      • Answer plan: Creates specific retrieval tasks.
      • Empty plan: A low-cost path for simple queries where initial retrieval is sufficient.
    3. Task Execution: Each task acts as a mini-agent that retrieves, answers, and optionally uses a Seed-query generator to reformulate queries if the initial answer is partial.
    4. Synthesis: Merges sub-answers and context into a final response.
    5. Verification (Optional): A quality gate that checks for coverage gaps or vague claims. If it fails, the system triggers a targeted re-plan/re-retrieval.

    When to use Agentic RAG

    • Multi-hop questions: Queries requiring multiple pieces of information.
    • Ambiguous queries: Questions that need scope discovery to align with the corpus.
    • Cross-document queries: Spanning multiple sources.
    • Numeric pulls: Extracting data from complex tables or charts.

  10. Derive corpus filenames from URLs using the Stem Rule

    main

    To ensure downstream citation matching works correctly, derive the filename for your corpus/ files and the contexts[].filename field in train.json using the following rule based on the source URL:

    1. Identify the path_last_segment: The last /-separated component of the URL path.
    2. Identify the fragment: The URL fragment (if any).
    3. Apply the Stem Rule:
      • If the URL has a fragment: stem = path_last_segment + "#" + fragment
      • If the URL has no fragment: stem = path_last_segment

    Critical Constraints

    • Do NOT unquote: Do not call urllib.parse.unquote() on the segment. Keep percent-encoding (e.g., %20) exactly as it appears in the URL.
    • Do NOT sanitize: Do not use slugification or sanitization functions that strip characters like %, ', ., #, or non-ASCII bytes. Any transformation breaks the alignment between the corpus file, the train.json reference, and the ingestor's document_name.
    • API Exception: If an external API requires a decoded title to fetch the content, decode it only for that specific API call. The resulting file on disk must remain encoded.
    stem = path_last_segment + "#" + fragment   (if URL has a fragment)
stem = path_last_segment                     (if no fragment)

  11. How service-specific API keys override global keys

    main

    The RAG Blueprint allows for fine-grained control by using service-specific API keys. These keys provide the ability to use different billing accounts, rate limits, or security segregation for different components.

    Service-specific keys take precedence over global keys. The system follows this fallback order:

    1. Service-specific key (e.g., APP_LLM_APIKEY)
    2. NVIDIA_API_KEY
    3. NGC_API_KEY
    4. None

    Note: For security, API keys must be configured via the NvidiaRAGConfig object or environment variables before initialization. They cannot be passed as runtime parameters in API requests.

  12. Customize VLM prompts and reasoning modes

    main

    VLM behavior is controlled via the vlm_template section in src/nvidia_rag/rag_server/prompt.yaml.

    Reasoning Modes

    You can toggle between reasoning and non-reasoning modes using the APP_VLM_ENABLE_THINKING environment variable:

    • Reasoning mode (default): APP_VLM_ENABLE_THINKING=true. The model produces a chain-of-thought trace. The trace is streamed in the reasoning_content field, while the final answer streams through content.
    • Non-reasoning mode: APP_VLM_ENABLE_THINKING=false. The model skips the reasoning trace and returns only the final answer in the content field.

    VLM Parameters

    Use these environment variables to tune the model:

    • APP_VLM_TEMPERATURE (Default: 0.6)
    • APP_VLM_TOP_P (Default: 0.95)
    • APP_VLM_MAX_TOKENS (Default: 32768)
    • APP_VLM_THINKING_TOKEN_BUDGET (Default: 16384)