DOCSEARCHbyAlgolia
Submit Docs

LLMOS Documentation

website·Indexed 31 days ago

https://llmos.1block.ai/docs

Documentation for the LLMOS platform, featuring guides on machine learning cluster creation, model and dataset management, GPU device monitoring, and ModelService deployment. Includes technical instructions for installation, upgrading (e.g., v0.2.0 to v0.3.1), storage configuration with Ceph, and user authentication via API keys and role templates.

Tokens
17K
Snippets
51
Records
127
Agent score
52%

What's inside LLMOS

  1. Overview of LLMOS Private Registries

    LLMOS registries are private repositories used to store and manage machine learning models and datasets. They provide a secure, centralized way to make assets accessible to model services and other applications within an LLMOS cluster. Currently, registries only support object storage backends via HTTPS.

  2. Overview of LLMOS Managed Addons

    Managed Addons are components used to extend the functionality of an LLMOS cluster. LLMOS includes several essential system addons that are enabled by default and cannot be disabled:

    • LLMOS GPU Stack: Provides virtual GPUs (vGPU) and multi-accelerator support.
    • LLMOS Monitoring: Implements monitoring and alerting via the Prometheus Operator.
    • LLMOS Redis: Provides a managed Redis service for chat storage and system services.
    • Rook Ceph Operator: Adds high-availability, fault-tolerant system storage.
    • KubeRay Operator: Manages Ray clusters and ML resources.
    • System Update Controller: Handles node upgrades via customizable plans.

  3. Understand LLMOS Global and Namespace Roles

    LLMOS uses role templates to define predefined permissions for users. There are two primary types of roles:

    1. Global Roles: Grant permissions at the system level. Users with global roles have access to the assigned resources across all namespaces they can access.
    2. Namespace Roles: Grant permissions specifically within a designated namespace. These are used to restrict or provide access to resources on a per-namespace basis.

  4. Understand 1Block.AI Service offerings

    1Block.AI provides Infrastructure as a Service (IaaS), AI-powered tools, and machine learning services. Key features include infrastructure management, model hosting, and data analysis.

  5. Understand LLMOS user types and permissions

    LLMOS defines two primary user types:

    • Admin: Has full access to the cluster, can create new users, assign roles, and manage role permissions via Role Templates. Admins have access to resources across all namespaces.
    • Standard User: Has basic permissions to access the LLMOS dashboard and manage resources only within assigned namespaces. By default, they have access to the llmos-public namespace.

  6. Understand LLMOS node roles and architecture

    LLMOS clusters consist of nodes that can be either virtual or physical machines. There are three primary node roles:

    • cluster-init: The first node in the cluster is designated as the cluster-init node by default.
    • server nodes: Host system components including the API server, controller-manager, and etcd, in addition to user workloads.
    • worker nodes: Primarily dedicated to running user workloads, while handling essential system services like DaemonSet services and tunnel proxy.

  7. Access LLMOS Monitoring Tools

    When monitoring is enabled, you can manage Prometheus rules, alerts, and dashboards via LLMOS Management > Monitoring. The available tools include:

    • Config: View and update monitoring settings.
    • Prometheus Graph: Access metrics and graphs.
    • Grafana: Explore dashboards.
    • Alertmanager: Manage alert configurations.

  8. Compare LLMOS storage options

    LLMOS provides two built-in storage options depending on the environment needs:

    • Local Path Storage: Saves data on the host machine. Best for testing or single-node setups; does not support high availability (HA) or fault tolerance.
    • Ceph Cluster Storage: A distributed storage service providing high availability and data fault tolerance. Supports block and filesystem storage, suitable for medium to large production environments.

  9. Understand LLMOS GPU Stack components

    The LLMOS GPU Stack is a core part of the platform and is enabled by default. When active, the following components are deployed in the llmos-gpu-stack-system namespace:

    • Nvidia GPU Operator: Manages Nvidia GPU resources.
    • GPU Device Plugin: A DaemonSet that exposes and manages GPU devices.
    • GPU Device Manager: A Deployment that manages LLMOS GPU custom resource definitions (CRDs).
    • Volcano Scheduler: A plugin used for batch and gang scheduling of GPU workloads.

  10. Understand LLMOS namespace isolation and scoping

    Namespaces in LLMOS isolate groups of resources within a single cluster. Resource names must be unique within a specific namespace but can be duplicated across different namespaces.

    Scoping Rules:

    • Namespaced Objects: Scoping applies to objects like Notebooks and Model Services.
    • Cluster-wide Objects: Scoping does not impact objects such as Nodes, GPU Devices, or Storage Classes.

  11. Understand LLMOS System Architecture

    LLMOS is an open-source AI infrastructure management software designed to simplify the deployment and scaling of Large Language Models (LLMs) on public clouds or private GPU servers. The architecture consists of the following core components:

    • Server Node: A cloud-based or on-premises machine hosting the LLMOS platform and LLMOS-optimized Kubernetes. Server nodes can also act as worker nodes, though system components are prioritized.
    • Worker Node: Responsible for executing user workloads.
    • LLMOS-Operator: Manages the lifecycle of system components, including the LLMOS API-server, controllers, and system addons.
    • LLMOS-Controller: Manages the lifecycle of resources such as LLM models, notebooks, machine learning clusters, and jobs.
    • Redis: A key-value store used for API chats and fault-tolerant configurations.
    • Workloads: Computational tasks utilizing requested CPU, GPU, memory, and storage volumes.

  12. Configure and create Local Models for faster deployment

    Local models are cached copies of registry-stored models pre-downloaded to local storage. This eliminates download time during model service startup, reducing network traffic and improving reliability.

    Prerequisites: Local model functionality requires Ceph Storage to be enabled and configured.

    Creation Steps:

    1. Navigate to the model details page of a registry model.
    2. Click Create Local Model.
    3. Configure the following:
      • New Version: Specify the version for the local model.
      • Registry: The source registry (auto-filled).
      • Namespace: Choose the target namespace.
    4. Click Create to start the caching process.

    Local models support version control, allowing you to maintain multiple versions for rollbacks or comparisons. When creating a Model Service, you can select these local models as the model source.