Nano-vLLM

Nano-vLLM provides an API that mirrors vLLM's interface. You can perform offline inference by initializing the LLM class and using the generate method. Note that the LLM.generate method signature may have minor differences compared to the original vLLM.

from nanovllm import LLM, SamplingParams

# Initialize the LLM engine
# enforce_eager: boolean to control eager mode
# tensor_parallel_size: number of GPUs for tensor parallelism
llm = LLM("/YOUR/MODEL/PATH", enforce_eager=True, tensor_parallel_size=1)

# Configure sampling parameters
sampling_params = SamplingParams(temperature=0.6, max_tokens=256)

# Define prompts and generate outputs
prompts = ["Hello, Nano-vLLM."]
outputs = llm.generate(prompts, sampling_params)

# Access the generated text
print(outputs[0]["text"])

Install Nano-vLLM directly from the GitHub repository using pip:

pip install git+https://github.com/GeeeekExplorer/nano-vllm.git

If you need to download model weights manually before using Nano-vLLM, use the huggingface-cli. For example, to download Qwen/Qwen3-0.6B to a specific local directory:

huggingface-cli download --resume-download Qwen/Qwen3-0.6B \
  --local-dir ~/huggingface/Qwen3-0.6B/ \
  --local-dir-use-symlinks False

The SamplingParams class is used to control the generation process.

Parameters:

• temperature (float): Controls randomness.
• max_tokens (int): The maximum number of tokens to generate.

The LLM class is the primary entry point for inference.

Initialization Parameters:

• model_path (str): Path to the model weights.
• enforce_eager (bool): Whether to use eager mode.
• tensor_parallel_size (int): The number of GPUs to use for tensor parallelism.

Methods:

• generate(prompts, sampling_params): Takes a list of prompt strings and a SamplingParams object. Returns a list of outputs where each output is a dictionary containing the generated "text".