Runpod Python SDK

Mirror Structure

Files are stored at {volume_path}/.cache/{namespace} using a size-bucketed approach:

• Small files (< 256 KiB): Packed into a single small.tar archive to reduce network metadata overhead.
• Large files: Stored unpacked in a big/ subdirectory, preserving their original relative path.
• Manifest: A manifest.json file acts as the commit marker. It records file metadata (size, mtime) and is written last to ensure a complete mirror.

Sync Logic

• Large files: Incremental transfers. Files are diffed by size/mtime against the manifest, and unchanged files are skipped.
• Small files: The small.tar archive is re-packed in its entirety whenever any small file changes.
• Idempotency: Re-running hydrate or sync when no changes are detected results in zero file copies.
• Safety: The system prevents directory traversal attacks by validating that every archive member and big-file destination resolves inside the configured dirs before writing.

The Runpod Serverless architecture is designed for high-performance I/O and reliability through several key design decisions:

• Asyncio Event Loop: A single event loop manages all asynchronous operations and HTTP requests via aiohttp.
• File-Based State Persistence: Job state is persisted to a .runpod_jobs.pkl file on disk. This allows the worker to survive restarts, though it introduces a small I/O overhead (5-15ms) per job.
• Separate Heartbeat Process: A dedicated multiprocessing.Process handles heartbeats. This ensures that even if a user handler blocks the main event loop, the worker remains visible to the Runpod API.
• Progress Updates: Updates are handled via ephemeral threads to ensure they are non-blocking for the user handler.
• Lazy Loading: Heavy dependencies (like boto3 or fastapi) are imported only when needed, reducing cold start times by 32-42%.

The Runpod serverless module is a framework designed to transform a container into a worker pod for the Runpod serverless platform. It manages the lifecycle of user-defined handler functions by polling the Runpod API for jobs, executing them, and transmitting results back to the platform.

Core Capabilities

• Job Lifecycle Management: Handles everything from job acquisition to completion.
• Concurrency & Scaling: Manages concurrent job processing with configurable scaling.
• Flexible Handlers: Supports synchronous handlers, asynchronous handlers, and generator functions for streaming results.
• Local Development: Includes a FastAPI server to provide a local testing environment.

Core Responsibilities

1. Job Acquisition: Polling the Runpod API for new jobs.
2. Job Processing: Executing the user-provided handler with the job input.
3. Result Transmission: Sending outputs (including streamed results) back to the Runpod API.
4. Heartbeat Management: Signaling worker availability to the platform.
5. State Persistence: Tracking in-progress jobs to handle restarts.
6. Error Handling: Capturing and reporting exceptions from the user handler.

graph TB
    subgraph "Entry Point"
        START[runpod.serverless.start]
    end

    subgraph "Worker Orchestration"
        WORKER[worker.py::main]
        FITNESS[run_fitness_checks]
        SCALER[JobScaler]
        HEARTBEAT[Heartbeat Process]
    end

    subgraph "Job Management"
        JOBFETCH[get_jobs]
        JOBRUN[run_jobs]
        JOBHANDLE[handle_job]
        QUEUE[asyncio.Queue]
    end

    subgraph "State & Logging"
        STATE[JobsProgress Singleton]
        LOGGER[RunPodLogger]
    end

    subgraph "Communication"
        HTTP[HTTP Client]
        RESULT[send_result]
        STREAM[stream_result]
        PROGRESS[progress_update]
    end

    subgraph "User Code"
        HANDLER[User Handler Function]
    end

    subgraph "External Services"
        RUNPOD[Runpod API]
    end

    START --> WORKER
    WORKER --> FITNESS
    FITNESS --> HEARTBEAT
    FITNESS --> SCALER
    SCALER --> JOBFETCH
    SCALER --> JOBRUN
    JOBFETCH --> QUEUE
    JOBRUN --> QUEUE
    JOBRUN --> JOBHANDLE
    JOBHANDLE --> HANDLER
    JOBHANDLE --> RESULT
    JOBHANDLE --> STREAM
    HANDLER --> PROGRESS
    PROGRESS --> HTTP
    RESULT --> HTTP
    STREAM --> HTTP
    HTTP --> RUNPOD
    JOBFETCH --> HTTP
    HEARTBEAT --> HTTP
    SCALER --> STATE
    JOBHANDLE --> STATE
    SCALER --> LOGGER

    style START fill:#1976d2,stroke:#0d47a1,stroke-width:3px,color:#fff
    style FITNESS fill:#1976d2,stroke:#0d47a1,stroke-width:3px,color:#fff
    style SCALER fill:#d32f2f,stroke:#b71c1c,stroke-width:3px,color:#fff
    style HANDLER fill:#388e3c,stroke:#1b5e20,stroke-width:3px,color:#fff
    style STATE fill:#f57c00,stroke:#e65100,stroke-width:3px,color:#fff
    style RUNPOD fill:#7b1fa2,stroke:#4a148c,stroke-width:3px,color:#fff

The execution flow of a Runpod serverless worker follows a structured sequence from startup to shutdown:

1. Initialization: The runpod.serverless.start() function parses arguments and configures the worker environment.
2. Fitness Checks: In production environments, the system runs run_fitness_checks to validate worker health before accepting jobs.
3. Mode Selection: The system determines the execution mode, routing the worker to either a local testing environment (FastAPI), a realtime API, or a production worker loop.
4. Worker Loop: The JobScaler component takes control, managing the acquisition of jobs from the API and orchestrating their processing.
5. Concurrent Execution: The system processes multiple jobs simultaneously based on the configured scaling parameters.
6. Graceful Shutdown: Signal handlers are utilized to ensure that the worker terminates cleanly, allowing for state persistence or job cleanup.

Fitness checks are validation routines that run only once at worker startup, before the first job is processed. They are designed to ensure the environment is correctly configured (e.g., GPU availability, model files, disk space) before the worker begins accepting jobs.

Key Behaviors

• Execution Timing: They run only on the actual Runpod serverless platform. Local development and testing modes skip them automatically.
• Execution Order: Checks execute in the order they were registered (top to bottom in your code).
• Failure Behavior: If any check fails, the worker logs an error, exits immediately with code 1, and the container is marked as unhealthy.
• Success Behavior: If all checks pass, the worker continues startup, starts the heartbeat process, and begins accepting jobs.

Best Practices

• Keep Checks Fast: Minimize startup time. Avoid time-consuming operations like model training or large data processing.
• Use Descriptive Errors: Raise RuntimeError with specific details (e.g., which environment variable is missing or the exact status code received) to aid debugging.
• Handle Transient Failures: For network-dependent checks, implement retry logic using asyncio to prevent unnecessary worker exits due to temporary glitches.

import runpod

@runpod.serverless.register_fitness_check
def check_first():
    print("This runs first")

@runpod.serverless.register_fitness_check
def check_second():
    print("This runs second")

For complex operations (like downloading large files or making system changes), you can trigger a worker refresh to ensure a clean state. There are two ways to do this:

1. Global Refresh: Enable it during startup. This refreshes the worker after every job, regardless of success or failure.

   runpod.serverless.start({"handler": handler, "refresh_worker": True})

2. Selective Refresh: Return a refresh_worker key in your handler's return dictionary. This allows you to refresh only when specific job conditions are met.

   def handler(job):
       if job["input"].get("refresh", False):
           return {"output": "Success", "refresh_worker": True}
       return {"output": "Success"}

from runpod.serverless import start

def handler(job):
    return {"output": "Job completed successfully"}

start({"handler": handler, "refresh_worker": True})