DOCSEARCHbyAlgolia
Submit Docs

smolvm Documentation

repository·main·Indexed 26 days ago

https://github.com/smol-machines/smolvm

An OCI-native microVM runtime and CLI tool for managing isolated Linux virtual machines with sub-second cold starts. smolvm enables sandboxing untrusted code with network restrictions, packing workloads into portable .smolmachine files, and running OCI-compliant images with hardware-level isolation. It includes support for GPU acceleration via Vulkan, CUDA API remoting, and a Node.js embedded SDK (smolvm-embedded) for programmatic VM management.

Tokens
26.4K
Snippets
48
Records
181
Agent score
88%

What's inside smolvm

  1. UDP Relay and Protocol Support

    main

    The virtio-net backend includes a userspace UDP relay that supports protocols like QUIC, HTTP/3, and NTP.

    • Egress Policy: UDP egress follows the same rules as TCP (static CIDRs and DNS-learned IPs). Denied UDP destinations result in a silent black hole.
    • NAT Behavior: The relay uses destination-keyed sockets and handles NAT-style idle expiry (flows expire after 60s, destination sockets after 120s).

  2. Achieve sub-second VM elasticity with smolvm forks

    main
    Unlike traditional container-style replicas that require a full cold-load (re-booting, re-initializing the engine, and re-loading weights), smolvm uses a 'warm fork' approach. New replicas inherit the loaded model state from a golden VM in sub-second time.

  4. Ensure atomic deployment of smolvm and drvlib

    main

    When deploying smolvm, you must ensure that the smolvm binary and all files in drvlib/ (such as libcudart.so.12 and libcuda.so.1) are restaged from the same build atom atomically. The system uses a wire-hash guard that will refuse connections if there is a mismatch between the daemon and the shim.

    If you encounter a cp: Text file busy error when attempting to update the smolvm binary, it indicates that a daemon is still running the file. You must kill the daemon before replacing the binary.

  5. Use weight sharing to reduce VRAM usage during forks

    main

    By default, smolvm uses COPY mode for forks, which can significantly increase VRAM usage because it stages all golden memory for the clone.

    To reduce the VRAM footprint and potentially avoid OOM issues during forks, use the --share-weights flag or set the SMOLVM_CUDA_FORK_SHARE_WEIGHTS environment variable to 1. This enables weight sharing between the golden and its clones.

  6. Reproduce smolvm GPU benchmarks

    main

    To reproduce the benchmark environments and experiments, follow these steps:

    1. Create the Guest: Use the baked toolchain image.
      machine create --from smolvm-gpu-testbed.smolmachine
    2. Host Setup: Use a Lambda persistent FS smolvm-testbed (us-east-1) which contains the engine, pinned venv, and model cache. Boot the testbed using:
      ./testbed_boot.sh
    3. Run Harnesses: Use the provided scripts for specific comparisons:
      • vllm_compare.sh for vLLM scaling.
      • l8_vllm_scale.sh for scaling tests.
      • real-sweep.sh for learning rate sweeps.
      • unique-demo.sh for unique artifact generation.

  7. Install and develop smolvm-embedded

    main

    To set up the local development environment for the smolvm-embedded Node.js SDK, navigate to the sdks/node directory and run the following commands to install dependencies, build the project, and run tests.

    cd sdks/node
npm install
npm run build
npm test
npm run --workspace smolvm-embedded test:integration
npm run smoke

  8. Set up the smolvm GPU testbed

    main

    The smolvm-gpu-testbed.smolmachine artifact (133 MB) is a packed Debian Bookworm machine containing gcc, ca-certificates, and experiment scripts located at /opt/experiments/.

    Host Prerequisites

    Before running the testbed, ensure the host meets these requirements:

    1. smolvm version: Must be >= v1.6.12 (specifically including PR #675 fixes for vLLM/H100) with its shim pack (drvlib/ containing libcudart.so.12, libcuda.so.1, libcublas/Lt, and libnvidia-ml). The PROTO_HASH must match the binary.
    2. Guest Virtual Environment: A directory containing the following exact versions:
      • torch 2.6.0+cu124
      • unsloth 2026.7.2
      • trl 0.18.2
      • datasets 4.3.0
      • vllm 0.8.5 (Note: Newer torch wheels may demand cudart symbols not exported by the shim.)
    3. Hugging Face Model Cache: Models (e.g., Qwen2.5-0.5B/7B-Instruct and bnb-4bit variants) must be pre-downloaded on the host. Do not rely on virtiofs for HF cache, as it breaks cache locking across VMs.

    Available Experiment Scripts

    Scripts are located at /opt/experiments/ inside the machine:

    • heartbeat.py: Minimal fork-correctness probe (plain torch).
    • density_train3.py: 3-way QLoRA fine-tune fork sweep (Unsloth, alpaca).
    • realsweep.py: Full LR sweep with held-out eval + saved adapters.
    • vllm_compare.py: vLLM serving replica (cold-vs-fork comparisons).

  9. Understand the behavior of forked 'Golden' instances

    main

    When using the smolvm fork CLI, the original instance (the 'Golden') is designed to stay frozen as a template. This is intended behavior, not a bug.

    • The Golden instance remains in a frozen state to serve as the base for all subsequent forks.
    • Functional workloads and serving requests should be directed to the clones created from the Golden, rather than the Golden itself.
    • If you observe the Golden instance failing to respond or showing degraded performance after a fork, verify if you are attempting to use the frozen template instead of the active clones.

  10. Sandbox untrusted code with network restrictions

    main

    Networking is disabled by default. To allow specific outbound hosts while keeping others blocked, use the --allow-host flag. This is useful for sandboxing untrusted code while permitting access to specific registries or APIs.

    # network is off by default — untrusted code can't phone home
smolvm machine run --image alpine -- nslookup example.com

# lock down egress — only allow specific hosts
smolvm machine run --net --image alpine --allow-host registry.npmjs.org -- wget -q -O /dev/null https://registry.npmjs.org

# fails — not in allow list
smolvm machine run --net --image alpine --allow-host registry.npmjs.org -- wget -q -O /dev/null https://google.com

  11. Build and run SmolVM using cargo-make

    main

    SmolVM uses cargo-make to orchestrate development tasks.

    To build and codesign the binary (macOS only), use cargo make dev. The resulting binary will be located at ./target/release/smolvm.

    To run smolvm with environment variables (DYLD_LIBRARY_PATH and SMOLVM_AGENT_ROOTFS) configured automatically, prefix your commands with cargo make smolvm.

    # Install cargo-make
cargo install cargo-make

# Build and codesign (macOS)
cargo make dev

# Run smolvm with automatic environment setup
cargo make smolvm --version
cargo make smolvm machine run --net --image alpine:latest -- echo hello
cargo make smolvm machine ls

  12. Pack workloads into portable .smolmachine files

    main

    Convert a workload and its dependencies into a single, self-contained binary using smolvm pack create. This produces a .smolmachine file that boots in <200ms on any supported platform with the same architecture.

    smolvm pack create --image python:3.12-alpine -o ./python312
./python312 run -- python3 --version