DOCSEARCHbyAlgolia
Submit Docs

NVIDIA NIM Anywhere

repository·main·Indexed 42 days ago

https://github.com/nvidia/nim-anywhere

A toolkit for building and scaling Retrieval-Augmented Generation (RAG) applications using NVIDIA NIM micro-services. It facilitates the integration of LLMs, Embedding NIMs, and Reranker NIMs into production-ready RAG architectures, featuring a FastAPI server with LangServe integration and support for Milvus Vector Databases.

Tokens
15.9K
Snippets
41
Records
82
Agent score
54%

What's inside NVIDIA NIM Anywhere

  2. Overview of NVIDIA NIM Anywhere

    main

    NVIDIA NIM Anywhere provides the tooling required to integrate NVIDIA NIM micro-services into Retrieval-Augmented Generation (RAG) pipelines. It is designed to scale from small-scale labs to full production environments, allowing developers to build RAG architectures that can easily incorporate NIMs as needed.

    NIM Anywhere is particularly useful for enterprises that need to leverage internal, confidential data for generative AI without sending that data to external providers like OpenAI. By using RAG, you can dynamically retrieve relevant information from your local databases and feed it to the model during inference, enhancing accuracy and reliability without modifying the model itself.

  3. Overview of application architecture in NIM Anywhere

    main

    NIM Anywhere applications and integrations are orchestrated by NVIDIA AI Workbench. The project structure is organized into demo services and external service integrations to demonstrate how to build production-ready AI applications using NIMs.

    Demo Services

    Located in the code folder, these services include:

    • Chain Server: A sample application that uses NIMs with LangChain. It supports experimentation both with and without Retrieval-Augmented Generation (RAG).
    • Chat Frontend: An interactive UI server designed to interface with the Chain Server.
    • Evaluation: A directory containing sample notebooks for demonstrating retrieval scoring and validation.
    • Notebooks: Interactive notebooks located at the root of the code folder for technical deep dives and advanced usage.

    Integrations

    The project demonstrates how to integrate NIMs with common infrastructure components:

    • Redis: Used for managing conversation history.
    • Milvus: Used as a vector database.
    • LLM NIM: Provides optimized Large Language Models.

  4. Best practices for Live Labs Lib variables and imports

    main

    When authoring labs with Live Labs Lib, follow these two technical constraints:

    1. Non-persistent variables: To ensure a variable is not persistent, append the suffix _derived to its name.
    2. Streamlit imports: Never import a Streamlit page, including the current page being authored.

  5. Use header and summary files for README branding

    main

    The documentation build process uses specific files for the top of the README:

    • Header File (docs/_HEADER.md): This content is written verbatim to the very beginning of the README.
    • Summary File: This content is placed immediately after the header and before the Table of Contents. It is processed by Pandoc to allow for embedded images and project descriptions.

  6. Choose between Cloning and Forking for NIM Anywhere

    main

    Depending on your goal, choose one of two methods to obtain the project:

    • Cloning (Recommended for starting): Use this for the fastest setup and easiest way to pull updates. Note that cloning does not allow for local modifications to be saved back to the repository.
    • Forking (Recommended for development): Use this if you need to make and save local changes. To keep a fork up to date, you must manually pull changes from the upstream repository. Follow standard GitHub workflows for working with forks.

  7. Communicate between pages using postMessage

    main

    For advanced use cases where multiple pages within a single view need to communicate, the application uses the Javascript postMessage framework. Messages posted to the application shell (window.top) are forwarded to all iframes.

    Posting a message

    To send a message from a Gradio page to the app shell, execute Javascript (via a Gradio event) that targets window.top. For example, to send a dictionary containing a boolean flag:

    window.top.postMessage({"use_kb": true}, '*');

    Consuming a message

    To receive a message on another page, add an event listener for the message event. If the message is trusted, you can use the data to update Gradio components by matching their elem_id.

    // Example: Sending a message
window.top.postMessage({"use_kb": true}, '*');

// Example: Receiving and applying a message to a Gradio component
window.addEventListener(
  "message",
  (event) => {
      if (event.isTrusted) {
          use_kb = gradio_config.components.find((element) => element.props.elem_id == "use_kb");
          use_kb.props.value = event.data["use_kb"];
      };
  },
  false);

  8. Configure the documentation structure via _TOC.md

    main
    The docs/_TOC.md file is the central control for the documentation build. It defines the specific order in which files are concatenated to create the final README. Only files explicitly listed in docs/_TOC.md will be included in the generated documentation.

  9. How the README is generated and updated

    main

    The project's README.md is automatically generated by combining several individual files located in the docs folder. Do not edit README.md directly, as your changes will be overwritten during the next build.

    To modify the documentation, you must edit the constituent files in the docs directory. The build process uses Pandoc to convert GitHub Flavored Markdown into the final README. The generation order is controlled by the Table of Contents file. Documentation can be previewed using the Workbench file browser window.

  10. Generate an NGC Personal Key

    main

    To allow AI Workbench to access NVIDIA's cloud resources, you must generate an NGC Personal Key (which begins with nvapi-).

    1. Navigate to the NGC Personal Key Manager.
    2. Select Generate Personal Key.
    3. Enter a Key name, set an expiration (e.g., 12 months), and select all services.
    4. Click Generate Personal Key.
    5. Important: Save the key immediately. It cannot be retrieved later, and losing it requires generating a new one. Treat this key like a password.

  11. Add a Remote Location to local AI Workbench

    main

    After completing the remote installation, you must link the remote machine to your local AI Workbench application. Open the AI Workbench application and click Add Remote Location. Fill in the following fields:

    FieldDescription
    Location NameAny short name for this new location
    DescriptionAny brief metadata for this location
    Hostname or IP AddressThe hostname or address used to remotely SSH (e.g., REMOTE-MACHINE)
    SSH PortUsually left blank; enter if using a nonstandard port
    SSH UsernameThe username used for making an SSH connection (e.g., REMOTE_USER)
    SSH Key FileThe path to the private key (e.g., /home/USER/.ssh/id_rsa)
    Workbench DirectoryUsually left blank; this is where Workbench remotely saves state

  12. Audit Python environment for vulnerabilities using audit.sh

    main

    To check the installed versions of all Python packages (including indirect dependencies) for security vulnerabilities, run the audit.sh script. The script generates a report identifying:

    • Warning state: Packages with potential issues.
    • Error state: Packages with active CVEs and known vulnerabilities.

    Note: Any package in an 'error state' must be resolved immediately to mitigate security risks.

    /project/code/tools/audit.sh