OpenEnv

OpenEnv is a unified framework designed for building, deploying, and interacting with isolated execution environments for agentic reinforcement learning. It is optimized for tasks like code generation, web browsing, and game playing.

Key features include:

• Gymnasium-Style APIs: Uses familiar step(), reset(), and state() interfaces for integration with RL frameworks.
• Container-First Design: Environments are packaged as containers for reproducible deployment.
• HTTP-Native: Environments can be deployed as HTTP services for distributed training and remote execution.
• Secure Isolation: Provides sandboxed environments to run untrusted agent code safely.
• Rich Environment Library: Includes pre-built environments for various domains (games, coding, web, etc.).
• CLI Tools: Provides a command-line interface for managing and deploying environments.

The Grid World environment is a deterministic 5x5 square grid designed for Reinforcement Learning (RL) prototyping.

Environment Specifications:

• Grid Size: 5x5
• Agent Start: (0,0) (top-left)
• Goal Position: (4,4) (bottom-right)
• Actions: UP, DOWN, LEFT, RIGHT
• Dynamics: Deterministic. Actions move the agent one step in the chosen direction. If an action would move the agent off the grid, the agent remains in its current cell.
• Reward Function (Sparse):
  • -0.1 per step (living cost).
  • +1.0 for reaching the goal at (4,4).
• Termination: The episode ends when the agent reaches the goal.

The OpenApp environment is a web application simulation ecosystem built on top of BrowserGym and the OpenApps framework. It allows agents to interact with a suite of simulated web applications using browser-based actions like clicking, filling forms, navigating, and scrolling.

Key Features:

• Multiple Apps: Includes calendar, todo list, messenger, and map applications.
• Browser-Based Actions: Supports standard UI interactions.
• Task-Based Evaluation: Supports optional task goals with automatic reward calculation.
• Configurable: Allows customization of app configurations and behaviors.

Ideal Use Cases:

• Training and evaluating UI agents.
• Testing web automation strategies.
• Researching human-computer interaction.
• Developing multimodal agents.

The Pelican SVG Environment is a reproducible, executable target for testing and training models on blind vector drawing. The model is tasked with generating an SVG of an animal riding a vehicle without seeing the result, requiring it to maintain spatial arrangements in its latent space.

It is designed for:

• Evaluation: Scoring frontier models via HF Inference Providers using examples/pelican_svg_eval.py.
• Training: Running reinforcement learning (e.g., GRPO) against the environment using TRL via examples/pelican_svg_grpo.py.

While inspired by Simon Willison's 'pelican riding a bicycle' prompt, this environment expands the task catalogue to include 30 different combinations of animals and vehicles to ensure the scorer measures drawing ability rather than memorized patterns.

OpenEnv supports various community-maintained environments designed for specific agent training and evaluation tasks. Two notable examples include:

• RLVE Gym: A suite of 400 environments that procedurally generate reasoning problems for Language Model (LM) training. It features configurable difficulty levels.
• Reasoning Core: Provides formally verifiable symbolic reasoning tasks covering domains such as logic, mathematics, planning, and syntax.

You can find detailed documentation and hosted versions for these environments on Hugging Face Spaces.

The repl_env provides a Python REPL environment designed for Recursive Language Model (RLM) execution. It allows a model to inspect context, execute Python code with persistent state across turns, and call LLM/RLM queries directly from within the REPL.

Key capabilities include:

• Executing Python code.
• Calling llm_query(...) or rlm_query(...) for recursive child runs.
• Terminating with FINAL(...), FINAL_VAR(...), or by setting answer = {"content": ..., "ready": True}.

import asyncio
from repl_env import REPLEnv

async def main():
    async with REPLEnv(base_url="http://127.0.0.1:8000") as env:
        await env.reset(context="alpha beta gamma", task_prompt="Count the words")
        await env.execute("count = len(context.split())")
        await env.execute("print(FINAL(count))")

asyncio.run(main())

The environment supports 6 games across two categories:

Single-Player Games (No Opponent)

• catch: Move horizontally to catch a falling ball.
• cliff_walking: Navigate grid without falling off cliff.
• 2048: Classic tile-merging puzzle.
• blackjack: Simplified blackjack (HIT/STAND only).

Multi-Player Games (with Bot Opponent)

• tic_tac_toe: Classic 3x3 game.
• kuhn_poker: 2-player simplified poker (game theory benchmark).