Xorbits Inference (Xinference) Documentation

Xorbits Inference (Xinference) is a powerful and comprehensive distributed inference framework designed for deploying various models, including Large Language Models (LLM), speech recognition models, and multi-modal models. It allows users to deploy their own models or use built-in open-source models with a single command.

Key capabilities include:

• Simplified Deployment: One-command deployment for LLMs, speech, and multi-modal models.
• Heterogeneous Hardware Support: Utilizes engines like ggml to leverage both GPU and CPU for inference, reducing latency and increasing throughput.
• Flexible Interfaces: Supports OpenAI-compatible RESTful APIs (including Function Calling), RPC, Command Line, and a Web UI.
• Distributed Computing: Supports distributed deployment with a built-in resource scheduler to manage models across a cluster of machines.
• Open Ecosystem: Seamless integration with popular libraries such as LangChain, LlamaIndex, Dify, and Chatbox.

Xinference is structured into several key modules that handle different aspects of model serving:

• API: Manages the RESTful interface and authentication (via oauth2).
• Client: Provides interfaces for interaction, including the oscar Actor Client for cluster interaction and a restful client for service interaction.
• Core: The engine of the system, containing:
  • worker.py & supervisor.py: Logic for worker actors (computation) and supervisor actors (lifecycle management and scheduling).
  • model.py: Defines the Model Actor, which handles inference requests and data streams.
  • status_guard.py: Monitors model statuses (creating, updating, terminating) via UIDs.
  • cache_tracker.py: Manages model version cache locations and information.
  • metrics.py & resource.py: Collects throughput, latency, CPU/GPU, and memory usage.
  • event.py: Gathers runtime events like warnings and errors.
• Deploy: Provides the Command Line Interface (CLI).
• Model: Handles model descriptions, creation, and caching.
• Frontend: The Next.js-based Web UI.

Xinference is built on the Xoscar actor framework. Each actor is a basic unit for model inference and can utilize specific CPU cores or GPU devices. Actors are hosted in asynchronous, non-blocking actor pools.

Key concepts for developers extending the system:

• Actor Actions: Define behaviors like launch_model, list_models, or terminate_model within a class inheriting from xoscar.Actor.
• Lifecycle Methods: Use __post_create__ for initialization after an actor is created and __pre_destroy__ for cleanup before destruction.
• Asynchronous Invocation: When calling methods via an actor reference (e.g., worker_ref.launch_model()), even synchronous methods are treated as asynchronous.

import xoscar as xo

class WorkerActor(xo.Actor):
    def __init__(self, *args, **kwargs):
        ...
    async def launch_model(self, model_id, n_gpu, ...):
        # launch an inference engine, use specific model class to load model checkpoints
        ...
    async def list_models(self):
        # list models on this actor
        ...
    async def terminate_model(self, model_id):
        # terminate the model
        ...
    async def __post_create__(self):
        # called after the actor instance is created
        ...
    async def __pre_destroy__(self):
        # called before the actor instance is destroyed
        ...