Runpod Flash

Runpod scales workers based on the REQUEST_COUNT scaler. When active requests exceed a defined threshold, new workers are provisioned; they scale down after an idle_timeout when requests drop.

Concurrency Patterns:

• Asynchronous functions (async def): Can handle multiple requests concurrently within a single worker thread. This is highly efficient for I/O-bound tasks.
• Synchronous functions (def): Block the worker thread and are processed one at a time per worker.

@api.post("/api/process")
async def process_data(x: int):
    import asyncio
    await asyncio.sleep(10)  # simulate work
    return {"result": x}

# 5 concurrent requests:
# requests 1-3: concurrent on worker 1 (async)
# requests 4-5: concurrent on worker 2 (new worker)
# all 5 complete in ~10s

Load-Balanced (LB) workers expose standard HTTP endpoints (GET, POST, etc.) behind a load balancer. These are designed for low-latency API endpoints that require horizontal scaling.

To define an LB worker, instantiate an Endpoint object with a name, cpu type, and a workers tuple defining the scaling range (e.g., workers=(1, 3)). You then use this instance as a decorator for your route handlers.

Example LB Worker:

from runpod_flash import Endpoint

api = Endpoint(name="lb_worker", cpu="cpu3c-1-2", workers=(1, 3))

@api.post("/process")
async def process(input_data: dict) -> dict:
    return {"status": "success", "echo": input_data}

@api.get("/health")
async def health() -> dict:
    return {"status": "healthy"}

Flash provides several key capabilities for cloud-native AI development:

• Remote Execution: @Endpoint functions run on Runpod Serverless GPUs/CPUs.
• Implicit Endpoint Resolution: When you run python script.py, Flash automatically routes calls to your deployed endpoints based on name and environment.
• Auto-scaling: Workers scale from 0 to N based on demand.
• Dependency Management: Specified dependencies are automatically installed on remote workers.
• Concurrency Control: Use max_concurrency to allow a single worker to process multiple jobs simultaneously.
• Two Patterns:
  • Queue-based: Use @Endpoint for batch/asynchronous work.
  • Load-balanced: Use Endpoint() + routes for REST APIs.

CPU and GPU endpoints in Flash differ in their base Docker images and default disk allocations:

• CPU Endpoints: Use python:*-slim base images and default to 20GB of container disk.
• GPU Endpoints: Use pytorch/pytorch:* base images and default to 10GB of container disk.

When using PodTemplate, Flash performs a field-level merge: explicit fields in your template override the defaults, while unset fields inherit the resource class defaults.

Flash handles cross-platform builds automatically to ensure compatibility with Runpod's Linux x86_64 infrastructure:

• Platform Targeting: Dependencies are always installed for Linux x86_64, regardless of whether you build on macOS, Windows, or Linux.
• Python Version: Targets Python 3.12 for wheel ABI selection regardless of your local interpreter.
• Binary Wheels: Only pre-built binary wheels are used to prevent platform-specific compilation issues.

Optimizing Package Size

Runpod Serverless has a 1.5GB limit on deployment archives. To stay under this limit:

1. Use --no-deps to only install direct dependencies specified in Endpoint definitions.
2. Use --exclude to skip packages already present in your base image (e.g., torch, torchvision, torchaudio for GPU images).

When your application contains functions on multiple endpoints (e.g., a CPU preprocessor and a GPU inference engine), the build process generates a manifest that enables automatic service discovery and routing. This allows functions to call each other directly by name.

# CPU endpoint function
@Endpoint(name="preprocessor", cpu="cpu3c-4-8")
def preprocess(data):
    return clean_data

# GPU endpoint function
@Endpoint(name="inference", gpu=GpuGroup.AMPERE_80)
async def inference(data):
    # calls CPU endpoint function via the generated manifest
    clean = preprocess(data)
    return results

from runpod_flash import Endpoint, GpuGroup

@Endpoint(name="preprocessor", cpu="cpu3c-4-8")
def preprocess(data):
    return clean_data

@Endpoint(name="inference", gpu=GpuGroup.AMPERE_80)
async def inference(data):
    clean = preprocess(data)
    return results

The Flash CLI automatically detects the app name based on your current working directory or local configuration.

If you need to target a specific app that is not the one detected by the current directory, you can override this behavior using the --app flag in your commands.

# In project directory, app is auto-detected
flash deploy

# Overriding the detected app
flash deploy --app other-project
flash env list --app other-project

CPU LoadBalancers use a specialized hashing mechanism that only considers CPU-relevant fields. This prevents false drift detection when GPU-specific fields are set to their defaults.

Included fields for CPU LB hash:

• datacenter
• flashboot
• imageName
• networkVolume
• instanceIds
• workersMin
• workersMax
• scalerType
• scalerValue
• type
• idleTimeout
• executionTimeoutMs
• locations

Excluded fields:

• GPU-specific fields such as gpuIds, gpuCount, allowedCudaVersions, and minCudaVersion are explicitly ignored to avoid unnecessary updates.