DOCSEARCHbyAlgolia
Submit Docs

OpenSCOW

repository·master·Indexed 43 days ago

https://github.com/pkuhpc/openscow

A web-based management platform for heterogeneous computing resources that bridges HPC and AI computing power through a unified interface and standardized management layer. It includes services for AI, authentication, and resource management, with support for Slurm cluster deployment via scow-vagrant.

Tokens
177.4K
Snippets
580
Records
888
Agent score
55%

What's inside OpenSCOW

  1. Overview of OpenSCOW capabilities

    master

    OpenSCOW is a dual-purpose platform designed for both supercomputing users and system administrators:

    Supercomputing Portal (User-facing)

    Enables users to utilize supercomputing resources via a modern web browser without installing local software. Key features include:

    • Terminal (Shell): Web-based command line access.
    • Task Management: Submitting and managing computational tasks.
    • File Management: Managing files within the supercomputing environment.
    • Remote Desktop: Accessing GUI applications via remote desktop.

    Management System (Admin-facing)

    Assists operations personnel in establishing management mechanisms. Key features include:

    • User Account Management: Managing personnel and access.
    • Resource Billing: Managing fees and billing related to computational resources (jobs).
    • Quotas: Managing account and user limits.

    OpenSCOW supports multi-cluster environments and is built with a modern technology stack for easy deployment, maintenance, and extension.

  2. Overview of scow Web (@scow/mis-web)

    master
    The scow Web package (part of the @scow/mis-web service) is a web-based interface designed to interact with backend services primarily through gRPC. It serves as a frontend component within the OpenSCOW ecosystem for managing supercomputing resources on the web.

  3. Overview of OpenSCOW

    master

    OpenSCOW (Super Computing On Web) is a computing platform system designed to manage heterogeneous computing resources, including HPC (High-Performance Computing) and AI computing power. It aims to lower the barrier for building, managing, and using cluster platforms by providing a unified management layer for different hardware vendors and software stacks.

    Key features include:

    • Graphical user interface (GUI) for ease of use.
    • Support for computing resource fusion (integrating various computing networks).
    • Integration of AI and HPC workloads.
    • Rapid deployment and out-of-the-box usability.

  4. Overview of the OpenSCOW Portal System

    master
    The OpenSCOW Portal System serves as the primary web-based entry point for users of High-Performance Computing (HPC) clusters. It provides a centralized interface to interact with cluster resources through a web browser, eliminating the need for local terminal configuration for basic tasks.

  5. Overview of the @scow/ai service

    master

    The @scow/ai package provides AI services within the OpenSCOW ecosystem. It is built using a modern web stack consisting of Next.js, tRPC, MikroORM, and OpenAPI.

    Developers can interact with the AI service using two different communication protocols:

    1. tRPC: For type-safe, end-to-end TypeScript integration.
    2. HTTP: For standard RESTful API calls, supported via OpenAPI specifications.

  6. Overview of OpenSCOW

    master

    OpenSCOW is a dual-purpose platform designed for High-Performance Computing (HPC) environments, serving both end-users and system administrators through two primary components:

    1. HPC Portal: A web-based interface that allows users to access HPC resources from any device with a modern browser. It provides:

      • Shell: Web-based terminal access.
      • Job Management: Interface for submitting and monitoring jobs.
      • File Management: Web-based file operations.
      • Interactive Apps, Desktops, and GUIs: Access to graphical applications and desktop environments.

    2. HPC MIS (Management Information System): A toolset for administrators to manage operations, including:

      • User and Account Management
      • Billing
      • Quotas

  7. Overview of the OpenSCOW AI System (Beta)

    master

    The OpenSCOW AI System (currently in Beta) provides a Web interface for managing AI training workflows. It allows users to manage datasets, algorithms, models, and training environments (via images) to facilitate and automate AI training tasks.

    Key capabilities include:

    • Datasets: Version management for personal datasets and sharing/usage of public datasets.
    • Images: Generation of training environment images and sharing/usage of public images.
    • Algorithms: Version management for personal algorithms and sharing/usage of public algorithms.
    • Jobs: Rapid job execution using datasets, algorithms, and images, along with comprehensive job management.
    • Models: Version management for personal models and sharing/usage of public models.

    Since this is a Beta feature, users are encouraged to provide feedback via GitHub Issues.

  8. Integrate with OpenSCOW

    master

    OpenSCOW provides several integration paths depending on your requirements:

    • Custom Authentication Systems: Replace or extend the default authentication logic to use your own identity provider.
    • OpenSCOW API: Programmatically invoke OpenSCOW functionalities via its API.
    • OpenSCOW Hook: Listen to system events emitted by OpenSCOW to trigger external workflows.
    • UI Extension: Embed your own user interface components directly into the OpenSCOW web interface.

  10. What is the OpenSCOW Scheduler Adapter

    master
    The Scheduler Adapter (scheduler-adapter) is an abstraction layer that decouples OpenSCOW from specific job scheduler implementations (like Slurm). Instead of OpenSCOW interacting directly with scheduler internals, it communicates with the adapter via a standardized interface. This allows OpenSCOW to support different schedulers by simply providing a new gRPC server that implements the required interface.

  11. Overview of OpenSCOW system architecture

    master

    OpenSCOW is a modular system composed of several containerized modules that facilitate supercomputing capabilities on the web. The architecture is divided into several functional layers:

    • Core Infrastructure Modules: Basic services including gateway, auth (authentication), and audit (auditing).
    • Portal System: The user-facing interface consisting of portal-web, portal-server, and novnc-client (for remote desktop/terminal access).
    • Management Information System (MIS): The administrative layer consisting of mis-web and mis-server.
    • Cluster Interaction Layer: Modules that bridge OpenSCOW with computing clusters, specifically the App Proxy and Adapter modules.
    • Deployment Tools: The cli module used for managing and deploying the system.

    For specific deployment instructions, refer to the individual documentation pages for each component.

  12. How custom authentication systems work in OpenSCOW

    master

    OpenSCOW allows you to implement a custom authentication service if the built-in system does not meet your requirements.

    Core Concepts

    • Identity: Users are identified by an identityId, which also serves as their login name on the cluster.
    • Token-based Authentication: After login, the authentication system must issue a token for the current session. OpenSCOW will call your service's validation API with this token for every authenticated request.
    • Implementation: A custom authentication service is essentially an HTTP server implementing a specific set of APIs.

    API Requirements

    • Public APIs: Any endpoint starting with /public is directly accessible by users.
    • Data Format: All responses and parameters in the body must be in JSON format.