DOCSEARCHbyAlgolia
Submit Docs

softlearning

repository·master·Indexed 23 days ago

https://github.com/rail-berkeley/softlearning

A deep reinforcement learning toolbox for training maximum entropy policies in continuous domains. It utilizes TensorFlow/Keras for models and Ray for large-scale experiment orchestration, supporting universes such as gym, robosuite, and dm_control.

Tokens
1.1K
Snippets
4
Records
5
Agent score
31%

What's inside softlearning

  1. Install Softlearning via Conda

    master

    To install Softlearning locally using Conda, follow these steps:

    1. Install MuJoCo: Download and install MuJoCo 1.50 and 2.00. Ensure files are in ~/.mujoco/mjpro150 and ~/.mujoco/mujoco200_{platform}. You may need to symlink the 2.00 installation so both gym and dm_control can find it: ln -s ~/.mujoco/mujoco200_{platform} ~/.mujoco/mujoco200
    2. Setup License: Copy your mjkey.txt to ~/.mujoco/mjkey.txt.
    3. Clone and Install:
      • Clone the repository.
      • Create the environment from environment.yml.
      • Install the package in editable mode to enable the CLI.

    To remove the environment, use conda deactivate followed by conda remove --name softlearning --all.

    git clone https://github.com/rail-berkeley/softlearning.git ${SOFTLEARNING_PATH}

cd ${SOFTLEARNING_PATH}
conda env create -f environment.yml
conda activate softlearning
pip install -e ${SOFTLEARNING_PATH}

  2. Install Softlearning via Docker Compose

    master

    To run Softlearning in a containerized environment using Docker Compose:

    1. Build and Run: Use docker-compose with the development CPU configuration. You must pass your MuJoCo license key via the MJKEY environment variable.
    2. Access Container: Use docker exec to enter the running container's bash shell.
    3. Cleanup: Use the down command with --rmi all and --volumes to remove the container, images, and volumes.
    # Build and run
export MJKEY="$(cat ~/.mujoco/mjkey.txt)" \
    && docker-compose \
        -f ./docker/docker-compose.dev.cpu.yml \
        up \
        -d \
        --force-recreate

# Access the container
docker exec -it softlearning bash

# Cleanup
docker-compose \
    -f ./docker/docker-compose.dev.cpu.yml \
    down \
    --rmi all \
    --volumes

  3. Train an agent using the Softlearning CLI

    master

    Use the softlearning run_example_local command to train an agent. You must specify the algorithm, universe (e.g., gym, robosuite, dm_control), domain, and task. Use --checkpoint-frequency to save checkpoints for later use or resuming.

    softlearning run_example_local examples.development \
    --algorithm SAC \
    --universe gym \
    --domain HalfCheetah \
    --task v3 \
    --exp-name my-sac-experiment-1 \
    --checkpoint-frequency 1000

  4. Simulate a trained policy

    master

    To simulate a policy, use the simulate_policy module from examples.development. You must provide the absolute path to the saved checkpoint.

    Common arguments:

    • --max-path-length: Maximum length of the simulation path.
    • --num-rollouts: Number of rollouts to perform.
    • --render-kwargs: A JSON string for rendering options (e.g., '{"mode": "human"}').
    python -m examples.development.simulate_policy \
    ${SAC_CHECKPOINT_DIR} \
    --max-path-length 1000 \
    --num-rollouts 1 \
    --render-kwargs '{"mode": "human"}'

  5. Reference: `softlearning run_example_local` arguments

    master

    The following arguments are available for the training command (passed to Ray Tune/Ray Init):

    ArgumentDescription
    --universe{robosuite,dm_control,gym}
    --domainThe specific domain name
    --taskThe specific task name
    --algorithmThe RL algorithm (e.g., SAC)
    --exp-nameName for the experiment
    --checkpoint-frequencyIterations between checkpoints (0 to disable)
    --restorePath to checkpoint (used for resuming)
    --cpus / --gpusResources to allocate to the Ray process
    --trial-cpus / --trial-gpusResources to allocate for each trial
    --num-samplesNumber of times to repeat each trial
    --local-dirDestination folder for results
    --upload-dirURI to sync results (e.g. s3://...)