DOCSEARCHbyAlgolia
Submit Docs

RL Swarm Documentation

repository·main·Indexed 23 days ago

https://github.com/gensyn-ai/rl-swarm

A peer-to-peer, open-source system for decentralized reinforcement learning. RL Swarm allows users to contribute hardware to collaborative model training runs, currently focusing on the CodeZero coding environment on the Gensyn Testnet using a multi-agent system of Solver, Proposer, and Evaluator nodes.

Tokens
8.3K
Snippets
11
Records
51
Agent score
82%

What's inside RL Swarm

  1. How CodeZero Roles Work

    main

    The CodeZero environment uses a multi-agent system where different nodes perform distinct roles to enable collaborative learning on programming challenges.

    Note: Most users will run Solver nodes.

    RoleDescription
    SolverLearns locally on code tasks using GRPO; shares rollouts with peers
    ProposerGenerates coding problems and adjusts difficulty heuristically
    EvaluatorFrozen model predicting correctness and assigning rewards

    CodeZero uses the following models:

    • Solver: Qwen2.5-Coder-0.5B-Instruct
    • Evaluator: Qwen2.5-Coder-1.5B-Instruct (frozen)

  2. Understand RL Swarm identity management

    main

    RL Swarm uses an on-chain identity system managed via an Alchemy modal sign-in screen. Users sign in with an email or Google account, which creates an EOA (Externally Owned Account) managed by Alchemy.

    Key components:

    • userApiKey: Local session keys (distinct from EOA keys).
    • swarm.pem: A local file that maintains your peer identity. This file is registered on-chain and linked to your email/EOA.

    Identity Scenarios:

    • Linking multiple nodes to one EOA: Sign up each node using the same email address. Each node gets a unique peer ID, but they all link to the same EOA.
    • Running a node with an existing identity: Use the original swarm.pem and log in with the original email address. Note: registration might throw a log error, but transactions will still work.
    • Lost swarm.pem or running multiple nodes: Run from scratch with the same email address to generate a new swarm.pem.
    • Changing identities: If you try to link an existing swarm.pem to a different email address, it will not work.

  4. Configure Huggingface and AI Prediction Market during Setup

    main

    During the initial swarm setup, you will encounter several interactive prompts:

    Huggingface Integration

    If you wish to upload your trained models to Hugging Face, you will be prompted for a Hugging Face access token. You can generate this in your Hugging Face account settings under Access Tokens.

    AI Prediction Market

    By default, you will be entered into the AI Prediction Market experiment (answering Y or pressing ENTER).

    • Mechanism: Models place bets on which answer to a reasoning problem is correct.
    • Rewards: Correct bets placed earlier in the process pay out more than later bets.
    • Opt-out: To opt out, answer N when prompted during setup.

  5. Set up environment variables for the UI components

    main

    Before running the development server, you must create a .env file in the root directory of the project and provide the following environment variables:

    • NEXT_PUBLIC_ALCHEMY_API_KEY: Your Alchemy API key.
    • NEXT_PUBLIC_PAYMASTER_POLICY_ID: Your Paymaster Policy ID.

    These variables are required for the ui-components-qs-nextjs package to function correctly.

    NEXT_PUBLIC_ALCHEMY_API_KEY= ...
NEXT_PUBLIC_PAYMASTER_POLICY_ID= ...

  6. Install and Run RL Swarm via Docker

    main

    The recommended way to run RL Swarm is using Docker to ensure a consistent setup.

    Prerequisites

    • Install Docker and ensure the daemon is running.
    • For Docker Desktop users, increase the Memory Limit in Settings > Resources > Advanced to the maximum possible value to prevent crashes.
    • Ensure you have Python >=3.10, <=3.13 installed on your host if needed for local management.

    Setup Steps

    1. Clone the repository:
      git clone https://github.com/gensyn-ai/rl-swarm
cd rl-swarm
    2. Start the Swarm: Run the command corresponding to your hardware from the repository root.

    For CPU-only support (e.g., Mac or non-GPU machines):

    docker-compose run --rm --build -Pit swarm-cpu

    For GPU support (NVIDIA GPUs):

    docker-compose run --rm --build -Pit swarm-gpu

    Note: If docker-compose fails (common on Ubuntu), try using docker compose (without the hyphen).

    git clone https://github.com/gensyn-ai/rl-swarm
# For CPU
docker-compose run --rm --build -Pit swarm-cpu
# For GPU
docker-compose run --rm --build -Pit swarm-gpu

  7. Install and Run RL Swarm via Shell Script (Experimental Mode)

    main

    If you want to experiment with the GenRL library or customize parameters in rgym_exp/config/rg-swarm.yaml, run the swarm using a Python virtual environment and the provided shell script.

    Setup Steps

    1. Create and activate a virtual environment:
      python3 -m venv .venv
source .venv/bin/activate
    2. Run the swarm script:
      ./run_rl_swarm.sh
    python3 -m venv .venv
source .venv/bin/activate
./run_rl_swarm.sh

  8. Run the RL Swarm Web server with Docker Compose

    main

    To spin up the webserver and OpenTelemetry (OTEL) containers for displaying gossip messages and training metrics, use docker-compose from the rl_swarm directory. This configuration automatically sets the INITIAL_PEERS environment variable, allowing the server to connect to the seed node and emit metrics to the OTEL container for local testing.

    docker-compose build --no-cache
docker-compose up

  9. Update RL Swarm to the CodeZero Environment

    main

    To continue earning participation points and transition from the old reasoning-gym environment to the new CodeZero environment, you must update your installation. All previous activity and leaderboard points are preserved.

    If using Docker

    1. Pull the latest changes:
      git pull
    2. Rebuild and restart your containers:
      docker-compose build

    If using the run_rl_swarm.sh script

    You must recreate your Python virtual environment to ensure compatibility:

    1. Pull the latest changes:
      git pull
    2. Remove and recreate the .venv:
      rm -rf .venv
python -m venv .venv
source .venv/bin/activate
    3. Restart your swarm.
    # For Docker users
git pull
docker-compose build

# For script users
git pull
rm -rf .venv
python -m venv .venv
source .venv/bin/activate

  10. Build the webserver image using Dockerfile.webserver

    main

    If you only want to run the webserver without the full docker-compose stack, you can build the image using the specific Dockerfile.webserver located in the root directory.

    docker build -t swarmui -f Dockerfile.webserver .

  12. How ProposerService interacts with the DHT

    main

    The ProposerService uses a ProposerClientDHT to communicate with the swarm via a HivemindBackend.

    • Inserting Proposals: Proposals are stored under the proposer sub-key in the DHT. Each proposal object contains proposer_model, proposal_question, proposal_tests, and proposal_raw.
    • Requesting Training Data: The service retrieves data from the solver sub-key. It filters for samples where sample['dataset'] == 'proposer' and returns a list of samples, optionally sampled to match the requested train_batch_size.