DOCSEARCHbyAlgolia
Submit Docs

Bhatti microVM Orchestrator

repository·main·Indexed 42 days ago

https://github.com/sahil-shubham/bhatti

An open-source microVM orchestrator optimized for AI coding agents, providing high-performance, near-instant sandbox creation and resumption using the krucible VMM (v2). It features a tiered rootfs system (minimal, browser, docker, computer), a comprehensive CLI for sandbox and volume management, and a debugging-first systems curriculum for low-level OS and virtualization learning.

Tokens
49.9K
Snippets
136
Records
272
Agent score
54%

What's inside Bhatti

  1. Key features of Bhatti sandboxes

    main

    Bhatti offers several advanced features for sandbox management and execution:

    • Preview URLs: Automatically generate public URLs for development. For example, bhatti publish dev -p 3000 generates a URL like https://dev-k3m9x2.bhatti.sh which supports auto-wake from sleep.
    • Session-aware exec: TTY sessions are persistent; they survive disconnects and scrollback is replayed upon reattachment.
    • OCI image support: You can pull OCI images to use as sandbox bases using bhatti image pull <image_name>.
    • Persistent volumes: Volumes that survive sandbox destruction and can be mounted across multiple sandboxes.
    • Streaming exec: Supports real-time NDJSON output via the Accept: application/x-ndjson header.
    • Guest hardening: Sandboxes execute as uid 1000, the config drive is unmounted after boot, and connection/session limits are enforced.
    bhatti publish dev -p 3000
bhatti image pull python:3.12

  3. Handle SQLite Concurrency and Transaction Risks

    main

    The bhatti store uses SQLite, but users should be cautious of concurrency issues in specific operations:

    1. DEFERRED Transactions: Operations like AttachPersistentVolume, DeletePersistentVolume, and UpdateSandboxLabels use DEFERRED transactions. This involves a SELECT followed by an UPDATE in Go code. If two callers perform this simultaneously, one update may be silently lost because the second UPDATE is a legal write against a snapshot.
    2. Busy Timeouts: While busy_timeout(5000) is configured, it may not prevent SQLITE_BUSY_SNAPSHOT errors during read-then-write cycles in DEFERRED transactions.
    3. Lack of Cancellation: Most store methods do not accept a context.Context, meaning requests cannot be cancelled or subject to deadlines.

  4. Understand Bhatti flag precedence

    main

    Bhatti follows the 12-factor app pattern for configuration precedence. When multiple configuration sources are provided, the order of precedence is:

    1. CLI Flags (highest priority)
    2. Environment Variables
    3. Configuration Files (lowest priority)

    Note: This is a behavior change from previous versions. If both an environment variable and a configuration file setting are present, the environment variable will take precedence.

  5. Use the lohar systemd-like init system

    main

    The lohar component includes a partial reimplementation of systemd to manage services via dependency-ordered supervision.

    Key Features:

    • Unit Files: Services are defined via units that declare dependencies using Wants, Requires, and After.
    • Dependency Management: The system uses a topological sort (depgraph.go) to determine the correct startup order and handles dependency cycles.
    • Cgroup Placement: To prevent race conditions where a forking daemon escapes its intended control group, lohar uses a specialized spawn helper (lohar spawn) instead of a standard exec.Command.

  6. Configure Bootstrapping via Config Drive

    main

    Because the guest agent (lohar) is not available immediately at boot, Bhatti uses a 'Config Drive' to bootstrap the VM.

    Mechanism: A tiny, read-only ext4 filesystem is attached as a drive. During early boot, the kernel/init system reads this drive to obtain essential configuration before the agent is active.

    Config Drive Contents:

    • Hostname
    • Authentication tokens
    • Environment variables
    • Secrets
    • Init scripts

  7. Understand the responsibilities and constraints of PID 1

    main

    PID 1 is the first userspace process started by the kernel. It has unique behaviors and critical responsibilities:

    • Ancestor of all processes: When a process's parent dies, the orphan is reparented to PID 1.
    • Kernel Panic on Exit: If PID 1 exits, the kernel panics ("Attempted to kill init!"). Therefore, a PID 1 implementation must never return (e.g., by blocking forever with select{}).
    • Early System Setup: It is responsible for mounting essential filesystems like /proc, /sys, /dev, /tmp, and bringing up the loopback interface.
    • Zombie Reaping: It must reap orphaned processes (zombies) by calling wait(). If it fails to do this, zombies will accumulate indefinitely.
    • Signal Handling: The kernel does not apply default fatal actions to PID 1 for signals it hasn't explicitly handled, preventing accidental termination via kill.

    Safety Warning: Running code designed for PID 1 (like lohar's runAgent) on a real host as root is extremely dangerous and can trigger system reboots or shutdowns.

  8. Verify mastery with the five "done" tests

    main

    Use these five criteria to determine if you have truly mastered the curriculum. Check each one when you can confidently perform the task:

    1. Trace bhatti exec dev -- echo hello keystroke→byte→back, naming every layer, without looking.
    2. Read a bug report and know which file to open in under a minute.
    3. Explain the rory incident (W1) from memory, including why "always Full".
    4. Follow a specific procedure (not just a 'vibe') when given a dmesg / strace / hung process.
    5. Explain the design rationale ("here's why it's built this way") rather than attributing logic to AI.

  9. Implement multi-tenant security and rate limiting

    main

    Security in bhatti is achieved through defense-in-depth across multiple layers:

    • API Scoping: Per-user API keys (SHA-256).
    • Network Isolation: Per-user L2 networking via subnetFromIndex.
    • Resource Caps: Limits on CPU/Memory per user.
    • Rate Limiting: Token-bucket algorithm implementation (e.g., limiting to 30 creates/min).

  10. Understand the bhatti topology: Daemon vs. CLI-direct

    main

    The bhatti project uses a dual-mode topology depending on whether you are building a persistent platform or running ephemeral tasks.

    1. Platform Mode (Daemon-based)

    Used for persistent, multi-sandbox, multi-tenant environments. The bhatti CLI communicates via HTTP to a resident bhatti daemon (server), which manages the registry, thermal management, and public proxies. The daemon spawns a bhatti vmm helper for each sandbox.

    2. Local/CI Mode (CLI-direct)

    Used for local, one-shot, or ephemeral sandboxes. This mode is daemonless and avoids HTTP overhead. The CLI spawns a bhatti vmm helper directly, executes the command, and tears down.

    Topology Diagram:

                ┌─ bhatti CLI ──HTTP──► bhatti daemon (server) ──► registry, thermal, public proxy, auth
            │                              │ spawns
  one binary ┤                              └─► bhatti vmm (helper, dlopen libkrun)  ── per sandbox
            │
            └─ bhatti run (CLI-direct) ───────► bhatti vmm (helper)   ── local, daemonless, ephemeral

  11. Understand process signaling and exit codes

    main

    When working with processes in bhatti, keep these signaling and exit code conventions in mind:

    Exit Codes and Signals

    Shells use a convention where an exit code of 128 + N indicates the process was terminated by signal N:

    • 137: 128 + 9 (Terminated by SIGKILL)
    • 139: 128 + 11 (Terminated by SIGSEGV)

    Signaling Process Groups

    • kill 1234: Signals only the specific process with PID 1234.
    • kill -1234: Signals every process within the process group 1234.
    • In lohar, setting Setpgid: true in SysProcAttr allows the system to manage a command and all its children as a single group, which can then be killed simultaneously using a negative PID.

  12. Avoid diff-snapshot corruption by using Full snapshots

    main

    A critical reliability issue in bhatti was identified where 'diff snapshots' (using dirty-page tracking via KVM_GET_DIRTY_LOG) could lead to silent corruption. If a write to guest RAM bypasses dirty tracking, the resulting diff snapshot misses pages, causing the restored guest kernel to be internally inconsistent and crash shortly after restore.

    Recommendation: To ensure reliability, disable diff snapshots and use Full snapshots exclusively. This trades off disk space and snapshot speed for guaranteed consistency.

    Configuration change: Set track_dirty_pages: false to ensure all snapshots are Full snapshots.

    track_dirty_pages: false