exo Distributed Inference Engine

exo is a platform designed to run frontier AI models locally by connecting multiple devices into a single AI cluster. It enables running models that are too large for a single device's memory by distributing them across a network of devices.

Key Capabilities:

• Automatic Device Discovery: Devices running exo automatically find each other on the network without manual configuration.
• RDMA over Thunderbolt: Supports RDMA over Thunderbolt 5, which can reduce latency between devices by up to 99%.
• Topology-Aware Auto Parallel: Automatically determines the optimal model sharding strategy based on real-time device topology, resource availability, and network bandwidth/latency.
• Tensor Parallelism: Supports model sharding to improve inference speed (e.g., up to 3.2x speedup on 4 devices).
• MLX Backend: Uses MLX as the inference engine and MLX distributed for communication.
• API Compatibility: Provides compatibility with several major AI interfaces, allowing you to use existing tools:
  • OpenAI Chat Completions API
  • Claude Messages API
  • OpenAI Responses API
  • Ollama API
• Custom Models: Supports loading custom models directly from the HuggingFace hub.

exo provides multiple API-compatible interfaces, including OpenAI Chat Completions, Claude Messages, OpenAI Responses, and Ollama.

Workflow for deploying a model instance:

1. Preview Placements: Use /instance/previews?model_id=<ID> to see valid sharding and placement options.
2. Create Instance: POST a placement object to /instance. This is an asynchronous operation.
3. Await Readiness: Use the /instance/await?model_id=<ID> endpoint. It returns an SSE stream that emits a ready message when the instance is active.
4. Inference: Send requests to the compatible endpoint (e.g., /v1/chat/completions for OpenAI-style or /v1/messages for Claude-style).
5. Cleanup: Delete the instance via DELETE /instance/YOUR_INSTANCE_ID.

# 1. Preview placements
curl "http://localhost:52415/instance/previews?model_id=llama-3.2-1b"

# 2. Create instance (using a placement from step 1)
curl -X POST http://localhost:52415/instance \
  -H 'Content-Type: application/json' \
  -d '{"instance": {...}}'

# 3. Wait for readiness (SSE stream)
curl -N "http://localhost:52415/instance/await?model_id=mlx-community/Llama-3.2-1B-Instruct-4bit"

# 4. Send a chat completion (OpenAI format)
curl -N -X POST http://localhost:52415/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "mlx-community/Llama-3.2-1B-Instruct-4bit",
    "messages": [{"role": "user", "content": "What is Llama 3.2 1B?"}],
    "stream": true
  }'

# 5. Delete the instance
curl -X DELETE http://localhost:52415/instance/YOUR_INSTANCE_ID

RDMA (Remote Direct Memory Access) is available on macOS 26.2+ for Macs with Thunderbolt 5 (e.g., M4 Pro Mac Mini, M4 Max Mac Studio, M4 Max MacBook Pro, M3 Ultra Mac Studio).

Activation Steps:

1. Shut down your Mac.
2. Hold the power button for 10 seconds to enter the Boot Menu.
3. Select Options to enter Recovery mode.
4. Open Terminal from the Utilities menu.
5. Run: rdma_ctl enable
6. Reboot.

Important Requirements & Caveats:

• Connectivity: Devices in an RDMA cluster must be connected to all other devices in the cluster via TB5 cables.
• Port Restriction: On Mac Studio, do not use the Thunderbolt 5 port located next to the Ethernet port.
• Source Configuration: If running from source, use the script tmp/set_rdma_network_config.sh to disable Thunderbolt Bridge and set DHCP on each RDMA port.
• OS Matching: Ensure all devices in the cluster run the exact same macOS version (including beta versions) to ensure discovery.

exo provides a macOS app that runs in the background. It requires macOS Tahoe 26.2 or later.

Installation:

• Download the latest build: EXO-latest.dmg
• Or via Homebrew: brew install --cask exo

Cluster Isolation (Custom Namespace): You can isolate your exo cluster from others on the same network using the EXO_LIBP2P_NAMESPACE setting. This can be configured in the app's Advanced settings or by setting the EXO_LIBP2P_NAMESPACE environment variable when running from source.

Uninstallation:

• Recommended: Use the app menu: menu bar icon → Advanced → Uninstall.
• Manual (if app is deleted): Run the standalone uninstaller script:

  sudo ./app/EXO/uninstall-exo.sh

  Note: You must manually remove EXO from 'Login Items' in System Settings after using the script.

Note: The app will request permission to modify system settings and install a new Network profile.

To run exo from source on macOS, you can use Nix for a streamlined installation, or manually install prerequisites using Homebrew.

Using Nix: If Nix is installed, run:

nix run .#exo

Note: To accept the Cachix binary cache and avoid the Xcode Metal ToolChain, configure /etc/nix/nix.conf with trusted-users and experimental-features = nix-command flakes, then restart the Nix daemon.

Manual Installation (Prerequisites):

• Xcode: Required for the Metal ToolChain (MLX compilation).
• Homebrew: For package management.
• uv: For Python dependency management.
• node: For building the dashboard.
• rust: Nightly version required for Rust bindings.
• macmon: For hardware monitoring (use the specific pinned fork to avoid crashes on Apple M5).

Full Setup Steps:

1. Clone the repository.
2. Build the dashboard.
3. Run exo using uv.

This starts the exo dashboard and API at http://localhost:52415/.

# Clone exo
git clone https://github.com/exo-explore/exo

# Build dashboard
cd exo/dashboard && npm install && npm run build && cd ..

# Run exo
uv run exo

The exo-bench tool measures model prefill and token generation speed across different placement configurations.

Prerequisites:

• Nodes must be running uv run exo before starting the benchmark.
• The tool communicates via the /bench/chat/completions endpoint.

Usage: Run bench/exo_bench.py with the following parameters:

• --model: Model ID (short or HuggingFace ID).
• --pp: Prompt size hints (comma-separated integers, e.g., 128,256).
• --tg: Generation lengths (comma-separated integers, e.g., 128,256).
• --max-nodes: Limit placements to N nodes (default: 4).
• --instance-meta: Filter by ring, jaccl, or both (default: both).
• --sharding: Filter by pipeline, tensor, or both (default: both).
• --repeat: Number of repetitions per configuration (default: 1).
• --warmup: Warmup runs per placement (default: 0).
• --json-out: Output file for results (default: bench/results.json).

# Basic benchmark
uv run bench/exo_bench.py \
  --model Llama-3.2-1B-Instruct-4bit \
  --pp 128,256,512 \
  --tg 128,256

# Advanced benchmark with filters
uv run bench/exo_bench.py \
  --model Llama-3.2-1B-Instruct-4bit \
  --pp 128,512 \
  --tg 128 \
  --max-nodes 2 \
  --sharding tensor \
  --repeat 3 \
  --json-out my-results.json

To run exo from source on Linux, ensure you have uv, node (v18+), and rust (nightly) installed.

Installation via System Package Manager (Ubuntu/Debian):

sudo apt update
sudo apt install nodejs npm
curl -LsSf https://astral.sh/uv/install.sh | sh
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup toolchain install nightly

Installation via Homebrew on Linux:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
brew install uv node
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup toolchain install nightly

Running exo:

1. Clone the repository.
2. Build the dashboard.
3. Run exo using uv.

Note: On Linux, exo currently runs on CPU. GPU support is under development.

This starts the exo dashboard and API at http://localhost:52415/.

# Clone exo
git clone https://github.com/exo-explore/exo

# Build dashboard
cd exo/dashboard && npm install && npm run build && cd ..

# Run exo
uv run exo