DOCSEARCHbyAlgolia
Submit Docs

CSGHub Documentation

repository·main·Indexed 26 days ago

https://github.com/opencsgs/csghub

An open-source, on-premise management platform for Large Language Model (LLM) assets, serving as a private alternative to Huggingface for managing models, datasets, and spaces. It provides tools for deployment via Docker Compose, Helm Charts, or an all-in-one script, as well as a SaaS version for model hosting, dedicated inference endpoints, and fine-tuning instances.

Tokens
12K
Snippets
24
Records
90
Agent score
87%

What's inside CSGHub

  1. Overview of CSGHub

    main

    CSGHub is an open-source, trustworthy large model asset management platform designed to govern the lifecycle of LLM and LLM applications. It manages assets such as datasets, model files, and application code.

    Key Capabilities:

    • Unified Management: A single hub for model files, datasets, and large-scale model application codes.
    • Development Compatibility: Supports HTTPS and SSH protocols for Git commands and web interface operations.
    • Large Model Native Features: Supports version management, model format conversion, automatic data processing, and dataset preview (using Apache Arrow and DuckDB for Parquet files).
    • Security & Permissions: Supports integration with corporate user systems, asset visibility settings, and zero-trust authentication.
    • Private Deployment: Designed for on-premise, fully offline operation, independent of cloud vendors.
    • Extensibility: Provides microservice submodules and standardized OpenAPIs for integration with existing systems.

  2. Overview of CSGHub Core Features

    main

    CSGHub is an open-source, reliable large-scale model asset management platform designed for the lifecycle of LLM assets (datasets, model files, code, etc.). It is designed for native large-scale model management and can be deployed completely offline on-premises.

    Key Capabilities:

    • Unified Asset Management: A one-stop Hub for model files, datasets, and LLM application code.
    • Ecosystem Compatibility: Supports both Git commands and Web interfaces via HTTPS and SSH protocols.
    • Native LLM Design: Supports version control, model format conversion, automatic data processing, and dataset previews (using Apache Arrow/DuckDB for Parquet).
    • Security & Permissions: Supports integration with enterprise systems, asset visibility settings, and zero-trust authentication interfaces.
    • Deployment: Supports one-click private deployment and integration via standardized OpenAPI and microservices.

  3. Overview of CSGHub Features

    main

    CSGHub is an open-source platform for managing the full lifecycle of Large Language Model (LLM) assets, including models, datasets, and code.

    Key capabilities include:

    • Unified Asset Management: Version control, asset tracking, and support for large file operations.
    • Extensible Development: Integration with SDKs like Gradio and Streamlit, support for HTTPS/SSH, and one-click inference/fine-tuning.
    • Data Management: Built-in format conversion (CSV, JSON, Parquet), web-based data preview, and E2E data processing pipelines with parallel computing.
    • Deployment & Security: Enterprise-ready on-premises deployment, integration with enterprise user systems, asset visibility settings, and license tracking.
    • High Availability: Support for load balancing, resource scheduling, and disaster recovery.

  4. CSGHub Architecture: Portal vs Server

    main

    CSGHub is composed of two main service parts:

    1. CSGHub Portal: The web interface and management layer (this repository).
    2. CSGHub Server: A high-performance backend implemented in Go.

    Developers looking to integrate CSGHub into external systems or perform secondary development should use the CSGHub Server API. The server project is located at: https://github.com/OpenCSGs/csghub-server.

  5. CSGHub Architecture Overview

    main

    CSGHub is composed of two primary components:

    1. CSGHub Portal: The frontend management interface (this repository).
    2. CSGHub Server: A high-performance backend implemented in Golang.

    Users can interact with assets via a Web interface, Git command line (HTTPS/SSH), or a natural language Chatbot. The platform utilizes Git Servers, Git LFS (Large File Storage), and Object Storage Service (OSS) for its data layer.

  7. CSGHub Technical Design and Deployment

    main

    CSGHub is built on a service-oriented architecture consisting of a backend server and a web interface.

    Core Technologies:

    • Storage: Integrates a Git server with Git LFS protocol and Object Storage Service for reliable data management.
    • Data Processing: Leverages Apache Arrow and DuckDB for efficient dataset management and Parquet file previews.
    • Deployment Methods: Services can be deployed using Docker Compose or Kubernetes Helm Charts.

    Architecture Components:

    • CSGHub Portal: The web interface (contained in this repository).
    • CSGHub Server: A high-performance backend developed in Golang (maintained in a separate repository).

  8. Configure Go Path and Environment Variables

    main

    Set up GOPATH

    Add the following to your shell configuration file (e.g., .bashrc or .zshrc):

    export GOPATH=$HOME/go
export PATH=$PATH:$GOPATH/bin

    Then reload the config using source ~/.bashrc or source ~/.zshrc.

    Manage Environment Variables with direnv (Recommended)

    1. Install direnv following its official documentation.
    2. Add the hook to your shell config:
      • Bash: eval "$(direnv hook bash)"
      • Zsh: eval "$(direnv hook zsh)"
    3. In the project root, create your environment file:
      cp .envrc.example .envrc
    4. Edit .envrc to set required variables (e.g., CSGHUB_PORTAL_S3_ACCESS_KEY_ID).
    5. Allow the environment to load:
      direnv allow .
    export GOPATH=$HOME/go
export PATH=$PATH:$GOPATH/bin

# direnv setup
cp .envrc.example .envrc
direnv allow .

  9. Deploy CSGHub using the All-in-One script

    main

    Use the all_in_one deployment script to launch a complete CSGHub instance including csghub-portal, csghub-server, nginx, postgresql, git-server, minio, casdoor, and other core services.

    Important Limitation: This deployment does not include space, model fine-tuning, or inference functions because those require a Kubernetes environment which is not part of this script.

    Prerequisites

    Hardware

    • Minimum: 2c CPU, 6GB RAM, 50GB Hard Disk
    • Recommended: 4c CPU, 16GB RAM, 500GB Hard Disk

    Software

    • Any Linux OS with x86_64 architecture
    • Docker Engine (>=5:20.10.24)

    Deployment Steps

    1. Navigate to the all_in_one directory.
    2. Edit the .env file and set SERVER_DOMAIN to the current host's IP address or domain name. DO NOT use 127.0.0.1 or localhost.
    3. (Optional) If you do not have a Kubernetes cluster, you can ignore the space and registry related configurations in .env.
    4. Run the startup.sh script.
    5. Once services are started, access the service at http://[SERVER_DOMAIN].

  11. Configure CSGHub Portal environment files

    main

    CSGHub Portal provides two template files for environment configuration:

    • .env.example: Use this for docker-compose or by loading it via the source command.
    • .envrc.example: Use this with direnv to automatically load variables when entering the project directory.

    To use them, copy one of the templates to .env or .envrc and modify the values according to your deployment needs.

  12. Integrate with CSGHub Server

    main

    The CSGHub Portal (this repository) is the web interface, while the core logic resides in the CSGHub Server, a separate high-performance Golang project.

    To integrate your own frontend system with the backend or to understand the server-side implementation, you must refer to the dedicated CSGHub Server repository.