runpod-workers/worker-vllm

repository·main·Indexed 19 days ago

https://github.com/runpod-workers/worker-vllm

An OpenAI-compatible serverless worker for Runpod that uses the vLLM inference engine to serve Large Language Models. It supports deployment via pre-built Docker images or custom images with baked-in weights, configuration through environment variables or config.yaml, and integration with the OpenAI Python SDK and HTTP requests.

Tokens: 20.3K
Snippets: 32
Records: 42
Agent score: 66%

What's inside worker-vllm

Understand the request flow and engine architecture
main
The worker processes requests through a structured pipeline:
Request Flow: RunPod Request → handler.py → JobInput → Engine Selection → vLLM Generation → Streaming Response
Engine Modes:
- OpenAI-compatible: Uses OpenAIvLLMEngine to provide a drop-in replacement for OpenAI APIs (e.g., /openai/v1/chat/completions). Routing is determined by the openai_route boolean in the JobInput.
- Native vLLM: Uses vLLMEngine for standard vLLM inference patterns.
Key Features:
- Streaming: Token-level streaming is supported by default.
- Dynamic Batching: Uses adaptive batch sizes that grow from a minimum to a maximum value using a growth_factor for efficiency.
https://github.com/runpod-workers/worker-vllm/blob/main/docs/conventions.md
Initialize the OpenAI Client for Runpod
main
To use the vLLM Serverless Endpoint Worker with the OpenAI SDK, initialize the OpenAI client using your Runpod API Key and the specific endpoint URL. The base URL must follow the pattern https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1.
Note: Use the /openai/ prefix to ensure requests are served directly rather than through the standard RunPod job queue.
```
from openai import OpenAI
import os

# Initialize the OpenAI Client with your Runpod API Key and Endpoint URL
client = OpenAI(
    api_key=os.environ.get("RUNPOD_API_KEY"),
    base_url="https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1",
)
```
https://github.com/runpod-workers/worker-vllm/blob/main/README.md
Pass vLLM Engine Arguments via Environment Variables
main
The worker allows you to configure any vLLM AsyncEngineArgs field by setting an environment variable using the UPPERCASED field name. The worker automatically discovers and applies these fields.
Format: <FIELD_NAME_UPPERCASED>=<value>
Rules:
- Only valid AsyncEngineArgs fields are applied; unknown keys are silently ignored.
- Values are automatically cast to the correct type (int, float, bool, str, or JSON for dict/list/tuple).
- For a full list of available arguments, refer to the vLLM AsyncEngineArgs documentation.
Common Examples:
- MAX_MODEL_LEN=4096 (maps to max_model_len)
- ENFORCE_EAGER=true (maps to enforce_eager)
- ENABLE_CHUNKED_PREFILL=true (maps to enable_chunked_prefill)
Backward-compatibility Aliases:
- MODEL_NAME $\rightarrow$ model
- TOKENIZER_NAME $\rightarrow$ tokenizer
- MAX_CONTEXT_LEN_TO_CAPTURE $\rightarrow$ max_seq_len_to_capture
- MODEL_REVISION $\rightarrow$ revision
```
| Environment Variable     | vLLM Engine Arg          | Value Example |
| ------------------------ | ------------------------ | ------------- |
| `MAX_MODEL_LEN`          | `max_model_len`          | `4096`        |
| `ENFORCE_EAGER`          | `enforce_eager`          | `true`        |
| `ENABLE_CHUNKED_PREFILL` | `enable_chunked_prefill` | `true`        |
| `NUM_SCHEDULER_STEPS`   | `num_scheduler_steps`   | `8`           |
| `TOKENIZER_POOL_SIZE`    | `tokenizer_pool_size`    | `4`           |
```
https://github.com/runpod-workers/worker-vllm/blob/main/docs/configuration.md
Use HTTP requests to call RunPod vLLM Worker
main
You can interact with the vLLM worker via standard HTTP requests (e.g., curl) by targeting the RunPod OpenAI-compatible endpoint.
1. Endpoint URL: Use https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1/chat/completions (or the relevant path for the specific method).
2. Authorization: Use your RunPod API Key in the Authorization: Bearer <YOUR RUNPOD API KEY> header.
3. Payload: Ensure the model field in your JSON body matches your deployed model's name.
```
curl https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <YOUR RUNPOD API KEY>" \
-d '{
"model": "<YOUR DEPLOYED MODEL REPO/NAME>",
"messages": [
  {
    "role": "user",
    "content": "Why is Runpod the best platform?"
  }
],
"temperature": 0,
"max_tokens": 100
}'
```
https://github.com/runpod-workers/worker-vllm/blob/main/README.md

Build a Docker image with the model baked in

main

If you want to avoid downloading models at runtime, you can build a custom Docker image with the model weights included. This requires passing specific --build-arg flags during the docker build process.

Build Arguments

Argument	Description
`MODEL_NAME`	Required. The model to bake in
`MODEL_REVISION`	Model revision to load (default: `main`)
`BASE_PATH`	Storage directory for HF cache and model (default: `/runpod-volume`). Set to something like `/models` to ensure weights are inside the image
`QUANTIZATION`	Quantization method
`WORKER_CUDA_VERSION`	CUDA version (recommended: `12.1.0`)
`TOKENIZER_NAME`	Tokenizer repository if different from model
`TOKENIZER_REVISION`	Tokenizer revision (default: `main`)
`VLLM_NIGHTLY`	Set to `true` to use the latest nightly vLLM and `transformers` from source

Examples

Build with OpenChat-3.5:

docker build -t username/image:tag --build-arg MODEL_NAME="openchat/openchat_3.5" --build-arg BASE_PATH="/models" .

Build with vLLM Nightly:

docker build -t username/image:tag --build-arg VLLM_NIGHTLY=true --build-arg MODEL_NAME="meta-llama/Llama-3.1-8B-Instruct" --build-arg BASE_PATH="/models" .

Enable DeepGEMM for MoE and MQA Logits
main
DeepGEMM is required for MQA logits computation on supported hardware (e.g., DeepSeek V4 models).
Important Requirements:
- Must set VLLM_USE_DEEP_GEMM to exactly "1" (to enable) or "0" (to disable). Do not use true or false.
- Requires CUDA 13.0+ and SM90+ (H100/H200).
Usage Notes:
- Setting VLLM_USE_DEEP_GEMM=0 disables the MoE part and falls back to flashinfer/cutlass FP8 kernels. This can improve performance on H20 GPUs and reduces cold-start time by skipping the DeepGEMM warmup phase.
```
# To enable DeepGEMM (requires SM90+ and CUDA 13.0+)
export VLLM_USE_DEEP_GEMM="1"

# To disable DeepGEMM (useful for H20 GPUs or faster cold starts)
export VLLM_USE_DEEP_GEMM="0"
```
https://github.com/runpod-workers/worker-vllm/blob/main/docs/configuration.md
Deploy vLLM using pre-built Docker images
main
The recommended way to deploy a vLLM worker on Runpod Serverless is to use the pre-built Docker image. This allows you to deploy any model supported by vLLM by configuring environment variables.
Docker Image: runpod/worker-v1-vllm:<version>
Requirements:
- CUDA >= 13.0
- For gated or private models, you must provide a Hugging Face token.
Deployment Guide: Follow the Runpod step-by-step guide to deploy via the Runpod Console.
```
runpod/worker-v1-vllm:<version>
```
https://github.com/runpod-workers/worker-vllm/blob/main/README.md
Security and Best Practices for Worker vLLM
main
When deploying or developing with Worker vLLM, adhere to the following security and resource management patterns:
Secret Management
- Build Secrets: Use Docker secrets when handling Hugging Face (HF) tokens during the image build process.
- Runtime Secrets: Use environment variable injection for sensitive credentials at runtime.
- Token Handling: Implement secure authentication patterns for all token-based access.
Resource Limits
- Memory Bounds: Use configurable GPU memory limits to prevent OOM (Out of Memory) errors.
- Request Limits: Implement concurrency and timeout controls to manage workload pressure.
- Model Safety: Use the trust_remote_code flags appropriately when loading models.
Logging Security
- Sanitization: Ensure no secrets or sensitive credentials are included in logs.
- Request Logging: Use configurable request/response logging settings.
- Performance Monitoring: Use safe metrics collection methods that do not leak sensitive data.
https://github.com/runpod-workers/worker-vllm/blob/main/docs/conventions.md
Include Hugging Face tokens securely during Docker build
main
To deploy private or gated models when building an image, use Docker BuildKit secrets to prevent your HF_TOKEN from being exposed in the image layers.
1. Enable BuildKit: export DOCKER_BUILDKIT=1
2. Export your token: export HF_TOKEN="your_token_here"
3. Build using the --secret flag:
```
docker build -t username/image:tag --secret id=HF_TOKEN --build-arg MODEL_NAME="openchat/openchat_3.5" .
```
```
export DOCKER_BUILDKIT=1
export HF_TOKEN="your_token_here"
docker build -t username/image:tag --secret id=HF_TOKEN --build-arg MODEL_NAME="openchat/openchat_3.5" .
```
https://github.com/runpod-workers/worker-vllm/blob/main/README.md
Modify Python OpenAI Client to use RunPod vLLM Worker
main
To use your deployed vLLM worker with the OpenAI Python SDK, you must update the client initialization and the model parameter.
1. Initialize the Client: Set the api_key to your RunPod API Key and the base_url to https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1. Replace <YOUR ENDPOINT ID> with your actual deployed endpoint ID.
2. Update Model Name: Change the model parameter in your completion calls to match your deployed model's repository or name.
```
from openai import OpenAI
import os

# Initialize client with RunPod credentials
client = OpenAI(
    api_key=os.environ.get("RUNPOD_API_KEY"),
    base_url="https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1",
)

# Use your deployed model name
response = client.chat.completions.create(
    model="<YOUR DEPLOYED MODEL REPO/NAME>",
    messages=[{"role": "user", "content": "Why is Runpod the best platform?"}],
    temperature=0,
    max_tokens=100,
)
```
https://github.com/runpod-workers/worker-vllm/blob/main/README.md
Choose a deployment model for worker-vllm
main
There are two primary ways to deploy this worker on RunPod:
1. Pre-built Images (Recommended):
  Image: runpod/worker-v1-vllm:<version>
  Mechanism: Downloads the model from Hugging Face at runtime.
  Configuration: Managed entirely via environment variables.
  Best for: Quick deployment and rapid model experimentation.
2. Baked Model Images:
  Mechanism: The model is downloaded during the Docker build process and embedded in the image.
  Configuration: Settings are stored in /local_model_args.json inside the container.
  Best for: Production environments where minimizing cold start times is critical.
```
# Example Docker Build Arguments
# MODEL_NAME: Primary model identifier
# BASE_PATH: Storage location strategy
# QUANTIZATION: Optimization settings
# WORKER_CUDA_VERSION: CUDA compatibility
```
https://github.com/runpod-workers/worker-vllm/blob/main/docs/conventions.md

Configure vLLM worker via environment variables

main

You can configure the worker using environment variables. The worker automatically maps any environment variable that matches an uppercase version of a vLLM AsyncEngineArgs field to the engine.

Core Configuration Variables

Variable	Description	Default
`MODEL_NAME`	Path of the model weights (Local folder or HF repo ID)	`facebook/opt-125m`
`HF_TOKEN`	HuggingFace access token for gated/private models
`MAX_MODEL_LEN`	Model's maximum context length
`QUANTIZATION`	Quantization method (`awq`, `gptq`, `squeezellm`, `bitsandbytes`)
`TENSOR_PARALLEL_SIZE`	Number of GPUs	`1`
`GPU_MEMORY_UTILIZATION`	Fraction of GPU memory to use (0.0 to 1.0)	`0.95`
`MAX_NUM_SEQS`	Maximum number of sequences per iteration	`256`
`CUSTOM_CHAT_TEMPLATE`	Custom chat template override (Jinja2 string)
`ENABLE_AUTO_TOOL_CHOICE`	Enable automatic tool selection	`false`
`TOOL_CALL_PARSER`	Parser for tool calls (`mistral`, `hermes`, `llama3_json`, etc.)
`OPENAI_SERVED_MODEL_NAME_OVERRIDE`	Override served model name in API
`MAX_CONCURRENCY`	Maximum concurrent requests	`30`

Passing custom vLLM engine arguments

To pass any other AsyncEngineArgs field, use the UPPERCASED field name.

Example mappings:

ENFORCE_EAGER $\rightarrow$ enforce_eager
ENABLE_CHUNKED_PREFILL $\rightarrow$ enable_chunked_prefill

runpod-workers/worker-vllm

What's inside worker-vllm

Build Arguments

Examples

Secret Management

Resource Limits

Logging Security

Core Configuration Variables

Passing custom vLLM engine arguments