Atropos Documentation

The Pay-to-Play environment is a reinforcement learning (RL) environment designed for training AI agents under economic constraints. It implements a 'Mixture of Judges' approach where an agent must strategically select and pay for specialized 'agent cards' to evaluate its responses.

Key features include:

• Economic Constraints: Uses real or simulated USDC payments (on Base blockchain).
• Strategic Selection: Agents choose from multiple specialized cards (e.g., Technical Expert, Creative Thinker) with varying prices and expertise.
• Budget Management: Agents must balance the cost of high-quality evaluations against a finite budget across training iterations.
• Performance Tracking: Historical evaluation data informs future selection decisions.

The environment is structured around several key files and datasets that define the interaction logic and the scope of possible behaviors:

• cat_server.py: The main implementation containing the cat-caretaker interaction logic.
• catbot_arena.py: An alternative arena-style environment (currently contains GSM8k-based placeholders).
• cat_behaviors.json: A database of 35 authentic cat behaviors and their meanings (includes communication, body language, physical actions, and behavioral indicators).
• cat_scenarios.json: A collection of 61 scenarios covering categories such as Nutrition, Health, Comfort, Safety, and Enrichment.

Atropos environments are designed for training and evaluating language models on specific tasks. Each environment provides a specialized implementation of a task, including its own input format, reward function, and evaluation metrics. The environments are categorized into two main types:

1. Main Environments: Focused on RL (Reinforcement Learning) training and include comprehensive datasets.
2. Evaluation Environments: Located in the eval_environments/ directory, these are benchmark-focused and primarily designed for model evaluation.

The mcp_tool_calling environment (referred to in documentation as NousWhiteHouse) is a reinforcement learning (RL) project designed to improve an agent's ability to use the Model Context Protocol (MCP). The primary goal is to enable LLMs to dynamically discover and invoke tools more effectively through context-aware decision-making, rather than relying on manual API matching.

This environment is particularly useful for RL research aimed at achieving faster and more accurate tool calling and seamless tool integration for LLMs.

DeepSacrifice is a design prototype for a lightweight reinforcement learning (RL) loop in a chess environment. It is designed to train an agent to play aggressive, sacrificial, and attacking chess.

Key components of the loop include:

• Human-in-the-loop RL: The user acts as the environment, providing direct interaction.
• LLM-based reward model: A language model acts as the reward function, providing dense feedback on aggression, brilliance, and sacrifice justification instead of simple win/loss outcomes.
• Policy improvement: The agent's move-selection strategy (policy) is updated based on the rewards received at the end of each game (episode).

The GRPO Trainer is composed of several specialized modules:

• grpo.py: Main CLI entry point for dispatching training modes.
• run.py: Unified launcher for shared_vllm mode (starts vLLM + trainer).
• cli.py: Source of truth for all CLI arguments.
• config.py: TrainingConfig Pydantic model for hyperparameters.
• api.py: Communication with Atropos API.
• data.py: Batch preprocessing, padding, and logprob extraction.
• model.py: Model loading and CUDA IPC attachment.
• training.py: GRPO loss computation (importance sampling and clipping).
• trainers.py: Implementation of the 4 training modes.
• vllm_api_server.py: Custom vLLM server with /generate endpoint and LoRA support.
• vllm_manager.py: vLLM process lifecycle management.
• checkpointing.py: Save/load checkpoints and adapters.

This environment is designed for Reinforcement Learning (RL) tasks involving de novo protein binder design. Unlike classic RL, which relies on fast simulations, this environment uses Large Language Models (LLMs) to navigate a multi-step pipeline of scientific tools. The LLM learns to use these tools as function calls to transform a target protein sequence into a functional binder.

The Design Pipeline:

1. Target Sequence → Structure (AlphaFold): Converts a sequence into a 3D .pdb file.
2. Target Structure → Binder Backbone (RFdiffusion): Generates a binder backbone .pdb from the target structure.
3. Backbone → Full Binder Sequence (ProteinMPNN): Generates a .fasta file with side chains from the backbone.
4. Evaluate Binding (AlphaFold-Multimer): Predicts the complex structure of the target and binder to evaluate binding success.

The Padres Spatial RL Environment is a 3D spatial reasoning environment using PyBullet physics simulation. It challenges LLMs to understand and manipulate objects (cubes and spheres) in a simulated 3D world.

Core Features

• 3D Physics Simulation: Uses PyBullet for gravity and collision detection.
• Object Manipulation: Control over position and orientation of objects.
• Real-time Visualization: A Three.js-based web interface for live scene viewing via WebSocket communication.
• Spatial Reasoning Tasks: Includes conditional positioning, distance constraints, and multi-objective scoring.