OSRS PvP Reinforcement Learning Framework

The project is divided into two primary functional components:

• PvP ML (pvp-ml): A Python-based core machine learning system. It orchestrates the training workflow, manages connections to the simulation, and provides a socket-based API for real-time model predictions.
• Simulation RSPS (simulation-rsps): A platform for conducting simulated PvP fights to train agents efficiently. It is built upon the Elvarg RSPS private server.

The training system manages the simulation during training sessions, though the simulation can be operated independently.

The ReinforcementLearningPlugin extends the game logic to support ML training through several key components:

• RemoteEnvironmentServer: A socket server that exposes routes for API interactions, including login/logout and step and reset requests (following the OpenAI Gym interface pattern).
• RemoteEnvironmentPlayerBot: Enables control over training agents via the API.
• AgentBotLoader: Supports independent ML-driven agents.
• EnableAgentCommand: Allows for player-controlled agents to be used in conjunction with the system.

The NH Environment is specialized for 1v1 Nightsight (NH) fights.

• Action Space: Uses a MultiDiscrete action space with 11 action heads.
• Observation Space: Extensive observation space.
• Contract: For the full technical specification of the observation and action spaces, refer to the environment contract at contracts/environments/NhEnv.json.

Environment contracts define the interface between the ML system and the game simulation. They are defined using JSON files located in contracts/environments. These files describe the available actions and observations for a specific environment.

Existing contracts include:

• NhEnv.json: The primary environment for the project.
• DharokEnv.json: A partially implemented test environment for Dharok edge-style combat.

To begin using the project, clone the repository and set up the two main subprojects individually. The project consists of a machine learning system and a game simulation.

1. Clone the repository:

   git clone https://github.com/Naton1/osrs-pvp-reinforcement-learning

2. Follow the specific setup instructions found in the README of each subproject:
   • pvp-ml/README.md for the machine learning system.
   • simulation-rsps/README.md for the game simulation.

Once set up, you can train custom models, serve existing models via an API, or evaluate models within the simulation.

To test an agent's performance, you can run it against a simulated server.

1. Run the eval command providing the path to your model.
2. Connect to the simulated server using a client to play against the agent.

eval --model-path <model-path-here>

To serve pre-trained models for real-time predictions using the socket-based API:

1. Set up the pvp-ml project following its internal documentation.
2. Run the pvp-ml API using the serve models via API command.

The project supports distributed rollouts using Ray for cluster or multi-core CPU training.

• Distributed Training: Use the --distribute flag. If you provide a number, it specifies the parallel rollout count. If omitted, it uses all available CPU cores.
• AWS Cluster Management:
  • Scale up: ray up cluster.yml
  • Scale down: ray down cluster.yml
  • View Dashboard: ray attach cluster.yml --port-forward=8265

# Train with a specific number of parallel rollouts
train --preset <preset> --distribute <parallel-rollout-count>

# Train using all available CPU cores
train --preset <preset>

# AWS Cluster Management
ray up cluster.yml
ray down cluster.yml
ray attach cluster.yml --port-forward=8265

To play against a trained agent within the simulated environment:

1. Set up the pvp-ml project.
2. Set up the simulation-rsps project and start the server.
3. Start a RSPS client and connect to the server.
4. Start an agent evaluation session for the specific model you wish to test.

You can train customized models on the simulation for any defined environment by using YAML configuration files.

1. Define your experiment parameters in a YAML file (refer to pvp-ml/config for existing configurations).
2. For detailed execution steps, refer to the start a training job guide in the pvp-ml directory.

To train on a new environment, you must first add it to the system (see 'Add New Environment' below).

The ElvargServer provides the simulated game environment for reinforcement learning. By default, it runs the ReinforcementLearningPlugin, which launches a remote environment server for API interactions (connection details are located in the pvp-ml module).

Requirements:

• Java 17

Steps:

1. Navigate to the ElvargServer directory.
2. Run the server using Gradle: ./gradlew run.

cd ElvargServer
./gradlew run

The PvP ML core logic requires conda to manage dependencies.

1. Create the environment: Use the provided environment.yml file.
2. Activate the environment: Activate the local ./env directory.

Note on GPU/CPU: By default, training uses a GPU if available. For CPU-only training, you must manually uncomment the cpuonly line in environment.yml before running the creation command.

# Create the environment
conda env create -p ./env -f environment.yml

# Activate the environment
conda activate ./env