NanoLLM Documentation

NanoLLM is a lightweight, high-performance library designed for building responsive, low-latency interactive agents on Jetson hardware. It utilizes optimized inferencing APIs to support:

• Quantized LLMs: Efficient execution of large language models.
• Multimodality: Support for Vision-Language Models (VLM).
• Speech Services: Integration with ASR (Automatic Speech Recognition) and TTS (Text-to-Speech).
• RAG: Vector databases for Retrieval-Augmented Generation.
• Web Frontends: Interfaces for interacting with the models.

NanoLLM provides optimized local inference for Large Language Models (LLMs) on edge devices. It offers HuggingFace-like APIs and supports a wide range of capabilities including:

• Quantization: Efficiently running models with reduced precision.
• Vision/Language & Multimodal Models: Support for models that process both text and images.
• Multimodal Agents: Agents capable of interacting with multiple data modalities.
• Speech: Integration for speech-related tasks.
• Vector DB & RAG: Support for Vector Databases and Retrieval-Augmented Generation (RAG) workflows.

NanoLLM supports several major model architectures, including fine-tuned derivatives that share the same base architecture:

• Llama
• Llava
• StableLM
• Phi-2
• Gemma
• Mistral
• GPT-Neox

Other model types may be supported via the various quantization APIs (MLC, AWQ, HF).

You can expose Python functions that the model can invoke via code generation. By using the @bot_function decorator, NanoLLM automatically wraps your function to:

1. Perform regex matching on the model's output to detect function calls.
2. Execute the function using eval() if called.
3. Append the function's return text to the chat history so the model can use the result in its subsequent reply.

To enable these functions during generation, pass an instance of BotFunctions() to the functions parameter of NanoLLM.generate().

from nano_llm import bot_function
from datetime import datetime

@bot_function
def DATE():
    """ Returns the current date. """
    return datetime.now().strftime("%A, %B %-d %Y")

@bot_function
def TIME():
    """ Returns the current time. """
    return datetime.now().strftime("%-I:%M %p")

NanoLLM is optimized for Jetson Orin running JetPack 5/6. The recommended way to deploy is via containers built by jetson-containers.

Images are available on DockerHub under the dustynv/nano_llm repository. For specific instructions on running containers and samples, refer to the Installation Guide and Release Notes in the documentation.

docker pull dustynv/nano_llm:latest

To add NanoLLM dependencies (including CUDA, PyTorch, and LLM inference APIs) to your own existing container, use the --base argument with jetson-containers/build.sh. Your container should be based on the same version of Ubuntu as your JetPack installation.

jetson-containers/build.sh --base my_container:latest --name my_container:llm nano_llm

To create a new plugin type, you must derive from the nano_llm.Plugin interface and implement the process() function. The process() function is responsible for handling incoming data and returning the outgoing data.

Alternatively, for simpler use cases, you can use callback functions instead of defining a full class. For example, you can use chat_plugin.add(my_function) to receive chat output via a callback.

If you want to build your own container on top of NanoLLM without running the full build process, use NanoLLM as the base image in your Dockerfile using a FROM statement.

FROM dustynv/nano_llm:r36.2.0
# Your custom setup here

You can run an interactive chat session via the command line. If using a multimodal model (e.g., liuhaotian/llava-v1.6-vicuna-7b), you can provide image filenames or URLs and queries to chat about the images. Use /reset to clear the chat history.

Example command:

python3 -m nano_llm.chat --api mlc \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --quantization q4f16_ft

Due to complex dependencies, the recommended installation method is using the Docker container images built by jetson-containers.

1. Clone and install the jetson-containers repository:

   git clone https://github.com/dusty-nv/jetson-containers
   bash jetson-containers/install.sh

2. Run the NanoLLM container using autotag to automatically select the image compatible with your JetPack-L4T version:

   jetson-containers run $(autotag nano_llm)