LitServe Documentation

repository·main·Indexed 26 days ago

https://github.com/lightning-ai/litserve

LitServe is a lightweight Python-based framework for building custom inference servers for models, agents, RAG pipelines, and multi-model workflows. It provides a structured way to define inference logic via the `ls.LitAPI` class and deploy servers using `ls.LitServer`. Key features include batching, streaming, GPU autoscaling, and support for PyTorch, JAX, and TensorFlow. It offers a 2x speedup over FastAPI and integrates with the Lightning CLI for cloud or self-hosted deployment.

Tokens: 8.5K
Snippets: 20
Records: 47
Agent score: 87%

What's inside LitServe

Compare LitServe deployment options
main
LitServe can be used in two ways:
1. Self-Managed (DIY): You manage the deployment yourself (e.g., using Docker). This is free and provides full control over your infrastructure, including the ability to use any engine (vLLM, etc.) and your own VPC.
2. Fully Managed on Lightning: A managed service that provides one-click deployment, built-in load balancing, GPU autoscaling, scale-to-zero (serverless), and enterprise features like SOC2/HIPAA compliance, versioning, and observability. It supports various engines like vLLM and Ollama.
https://github.com/lightning-ai/litserve/blob/main/README.md
Explore LitServe inference pipeline examples
main
LitServe supports a wide variety of model types and use cases. You can find community-built templates and official examples for:
- LLMs: Llama 3.2, LLM Proxy servers, and Agents with tool use.
- RAG: vLLM RAG (Llama 3.2) and RAG APIs using LlamaIndex.
- NLP: Hugging Face models, BERT, and Text embedding APIs.
- Multimodal: OpenAI CLIP, MiniCPM, Phi-3.5 Vision, Qwen2-VL, and Pixtral.
- Audio/Speech: Whisper, AudioCraft, StableAudio, DeepFilterNet, XTTS V2, and Parler-TTS.
- Vision: Stable Diffusion 2, AuraFlow, Flux, Aura SR (Super Resolution), Background Removal, and ControlNet.
- Classical ML: Random Forest and XGBoost.
- Miscellaneous: Media conversion (ffmpeg) and hybrid PyTorch + TensorFlow APIs.
You can browse over 100 community-built templates on the Lightning AI Studios platform.
https://github.com/lightning-ai/litserve/blob/main/README.md
Deploy LitServe servers
main
You can deploy your LitServe application using the lightning CLI.
- To Lightning Cloud (with autoscaling and monitoring): lightning deploy server.py --cloud
- To self-host anywhere: lightning deploy server.py
- Locally for development: Run the python file directly (e.g., python server.py).
```
# Deploy for free with autoscaling, monitoring, etc...
lightning deploy server.py --cloud

# Or run locally (self host anywhere)
lightning deploy server.py

# python server.py
```
https://github.com/lightning-ai/litserve/blob/main/README.md
Install LitServe
main
Install the LitServe package using pip to begin building custom inference servers in Python.
```
pip install litserve
```
https://github.com/lightning-ai/litserve/blob/main/README.md
Deploy to the cloud with Lightning AI
main
You can deploy your LitServe application to the cloud using the Lightning CLI. This allows for managed hosting with features like autoscaling, security, and high uptime. Use the --cloud flag to trigger a cloud deployment of your server script.
```
lightning deploy server.py --cloud
```
https://github.com/lightning-ai/litserve/blob/main/README.md

Create an inference engine with LitAPI

main

To build a custom inference server, subclass ls.LitAPI and implement the setup and predict methods.

setup(self, device): Use this method to initialize models, databases, or clients (e.g., OpenAI). The device argument specifies the hardware to use.
predict(self, request): This method contains your core inference logic. It receives the request data and returns the processed output.

Once defined, wrap the API instance in ls.LitServer and call .run() to start the server.

import litserve as ls

# define the api to include any number of models, dbs, etc...
class InferenceEngine(ls.LitAPI):
    def setup(self, device):
        self.text_model = lambda x: x**2
        self.vision_model = lambda x: x**3

    def predict(self, request):
        x = request["input"]
        # perform calculations using both models
        a = self.text_model(x)
        b = self.vision_model(x)
        c = a + b
        return {"output": c}

if __name__ == "__main__":
    # 12+ features like batching, streaming, etc...
    server = ls.LitServer(InferenceEngine(max_batch_size=1), accelerator="auto")
    server.run(port=8000)

Build and run the generated Docker container
main
After running dockerize, a Dockerfile is created in your current directory. Use the following commands to build and run your container:
Build the container
```
docker build -t litserve-model .
```
Run the container (CPU)
```
docker run -p 8000:8000 litserve-model:latest
```
Run the container (GPU)
If you used gpu=True during dockerization, run with:
```
docker run --gpus all -p 8000:8000 litserve-model:latest
```
Push to a registry
```
docker push litserve-model
```
https://github.com/lightning-ai/litserve/blob/main/src/litserve/docker_builder.py

Enable Model Context Protocol (MCP) integration

main

You can enable Model Context Protocol (MCP) support in your LitServe APIs to make your models accessible as tools within MCP-compatible AI systems (like Claude Desktop). This allows AI agents to discover and call your LitServe endpoints as tools.

Prerequisites You must have the fastmcp package installed:

pip install fastmcp

Quick Start Pass an instance of MCP to your LitAPI class, then pass that same instance to your LitServer.

from pydantic import BaseModel
from litserve.mcp import MCP
import litserve as ls

class PowerRequest(BaseModel):
    input: float

class MyLitAPI(ls.LitAPI):
    def decode_request(self, request: PowerRequest) -> int:
        return request.input

if __name__ == "__main__":
    # Initialize MCP with a description for the AI agent
    mcp = MCP(description="Returns the power of a number.")
    # Pass mcp to your API
    api = MyLitAPI(mcp=mcp)
    server = ls.LitServer(api)
    server.run()

Notes

MCP integration is optional and does not affect standard non-MCP clients.
Tool names are automatically sanitized (e.g., / becomes _).
Original API endpoints remain unchanged and fully functional.

from pydantic import BaseModel
from litserve.mcp import MCP
import litserve as ls

class PowerRequest(BaseModel):
    input: float

class MyLitAPI(ls.LitAPI):
    def decode_request(self, request: PowerRequest) -> int:
        return request.input

if __name__ == "__main__":
    mcp=MCP(description="Returns the power of a number.")
    api = MyLitAPI(mcp=mcp)
    server = ls.LitServer(api)
    server.run()

Migrate deprecated LitServer arguments to LitAPI
main
As of version v0.3.0, several parameters previously passed to LitServer are being moved to the LitAPI class. If you use them in LitServer, you will see a DeprecationWarning.
Deprecated arguments to move to LitAPI:
- max_batch_size
- batch_timeout
- stream
- api_path
- loop
- spec
Migration Example:
```
# Old way (deprecated)
server = ls.LitServer(api, max_batch_size=8, stream=True)

# New way (recommended)
api = MyAPI(max_batch_size=8, stream=True)
server = ls.LitServer(api)
```
https://github.com/lightning-ai/litserve/blob/main/src/litserve/server.py
Use OpenAIEmbeddingSpec to host OpenAI-compatible embedding services
main
The OpenAIEmbeddingSpec allows you to deploy an embedding model that follows the OpenAI API schema. This enables compatibility with OpenAI SDKs and other clients expecting the /v1/embeddings endpoint.
Key Requirements:
- No Streaming: The spec does not support streaming (using yield in predict or encode_response) because embedding generation is not a sequential operation.
- Response Format: Your predict method should return a dictionary containing an embeddings key. For example: {"embeddings": [[0.1, 0.2, ...], [0.3, 0.4, ...]]}.
- Batching Note: If you use client-side batching (sending a list of inputs), you cannot use LitServe's dynamic batching. You must set max_batch_size=1 in your LitAPI or send single inputs from the client.
```
import numpy as np
from typing import List
from litserve.specs import OpenAIEmbeddingSpec, EmbeddingRequest
import litserve as ls

class TestAPI(ls.LitAPI):
    def setup(self, device):
        self.model = None

def predict(self, inputs) -> list[list[float]]:
        # inputs is a string
        return np.random.rand(1, 768).tolist()

if __name__ == "__main__":
    server = ls.LitServer(TestAPI(), spec=OpenAIEmbeddingSpec())
    server.run()
```
https://github.com/lightning-ai/litserve/blob/main/src/litserve/specs/openai_embedding.py
Initialize and run a LitServer
main
Use LitServer to transform your LitAPI models into production-ready APIs. You can serve a single model by passing one LitAPI instance, or multiple models by passing a list of LitAPI instances.
To run the server, call the .run() method. You can specify the port and the number of API servers (for scaling).
```
import litserve as ls

class MyAPI(ls.LitAPI):
    def setup(self, device):
        self.model = load_model()  # model loading logic

    def predict(self, x):
        return self.model(x)

# Create and run server
server = ls.LitServer(MyAPI())
server.run(port=8000)
```
https://github.com/lightning-ai/litserve/blob/main/src/litserve/server.py

Integrate LitServe with Claude Desktop via MCP

main

To use your LitServe MCP server with Claude Desktop, follow these steps:

Install mcp-remote globally:

npm install -g mcp-remote

Add the following configuration to your Claude Desktop settings, replacing the URL with your server's URL plus /mcp/:

{
  "mcpServers": {
    "litserve": {
      "command": "npx",
      "args": [ "mcp-remote", "https://8000-YOUR_HOST_NAME.cloudspaces.litng.ai/mcp/" ]
    }
  }
}

{
  "mcpServers": {
    "litserve": {
      "command": "npx",
      "args": [ "mcp-remote", "https://8000-YOUR_HOST_NAME.cloudspaces.litng.ai/mcp/" ]
    }
  }
}

LitServe Documentation

What's inside LitServe

Build the container

Run the container (CPU)

Run the container (GPU)

Push to a registry

Key Requirements: