Cradle Framework

The Cradle repository is organized to support both core framework logic and specific environment implementations (games/software). Key directories include:

• conf/: Configuration files for environments, LLM models (e.g., openai_config.json, env_config_rdr2_main_storyline.json).
• res/[game or software]/: Environment-specific resources, including prompts/templates/ (e.g., action_planning.prompt), skills/ (auto-generated), and icons/.
• cradle/environment/[game or software]/: The core implementation for a specific environment. Contains atomic_skills/, composite_skills/, skill_registry.py, and ui_control.py.
• cradle/runner/: Contains the logical flow of execution for each game/software.
• cradle/provider/: Contains execution providers like llm, object_detect, and process (pre/post-processing).
• runner.py: The main entry point for the framework.

To set up the base Cradle environment, create a new Conda environment with Python 3.10 and install the required dependencies from requirements.txt.

conda create --name cradle-dev python=3.10
conda activate cradle-dev
pip3 install -r requirements.txt

The runner defines the main execution logic, including self-reflection and information gathering processes.

To implement a new runner, copy an existing runner (e.g., cradle/runner/skylines_runner.py), rename it (e.g., newgame_runner.py), and modify the logic to suit your game. Note that future versions of Cradle aim to unify all runners into a single implementation, which may eventually remove the need for manual runner implementation.

Follow these steps to execute the framework agent on the Dealer's Life 2 environment:

1. Verify the game display settings (1920x1080, 16:9, Fullscreen) are applied.
2. Launch the Dealer's Life 2 game.
3. Initialize a new game and complete the character setup.
4. Complete the tutorial and navigate to the main game screen.
5. Before engaging in haggling with the first customer, run the framework agent using the runner.py script with the specific environment configuration file.

python runner.py --envConfig "./conf/envs/dealers.json"

To extend Cradle to support a new software application, follow these steps:

1. Update Repository: Always pull the latest /main branch to your work branch.
2. Create Configuration: Add a new configuration file following the pattern conf/env_config_[software_name].json.
3. Setup Environment Folder:
   • Copy the directory cradle/environment/chrome to cradle/environment/[software_name].
   • Replace all instances of chrome within that folder with your software's name.
4. Setup Resources:
   • Copy the res/chrome folder to res/[software_name].
   • Replace all instances of chrome within that folder with your software's name.
   • Modify prompts and template-matching icon images for UI elements that SAM2SOM cannot recognize.

Depending on the task type, follow these initialization steps:

Main Storyline

1. Launch RDR2.
2. Pass all introductory videos to start from Chapter #1.
3. Pause the game with esc before running the script.

Open-Ended Task

1. Use the provided game save located in res/rdr2/saves.
2. Copy the save file to your local game save directory: C:\Users<Your Name>\appdata\Roaming\Goldberg SocialClub Emu Saves\RDR2
3. Load the save and pause the game with esc.

Cradle requires specific OS-level display settings to function correctly:

1. Display Scale: Set your main display scale to 125%.
2. Resolution: Ensure your main display resolution is greater than 1920x1080.
3. File Explorer View: Set the folder that the agent will open to display in Large icons or Extra large icons format.

To start the agent with RDR2:

1. Launch RDR2.
2. Pass all introductory videos.
3. Pause the game.
4. Run the prototype runner from the terminal.

python prototype_runner.py

The videosubfinder tool is required for subtitle processing.

1. Download videosubfinder from SourceForge.
2. Extract the files into the res/tool/subfinder folder. The expected structure is:

   ├── res
   │   ├── tool
   │   │   ├── subfinder
   │   │   │   ├── VideoSubFinderWXW.exe
   │   │   │   ├── test.srt
   │   │   │   └── ...

3. Tuning: To optimize extraction, you can overwrite res/tool/subfinder/settings/general.cfg using the file res/tool/general.clg. Only modify these settings if necessary, as they are pre-tuned for this environment.

To ensure compatibility with Cradle, the Dealer's Life 2 environment must be configured with specific display settings. This environment is only available on Windows.

Required settings:

• Monitor Resolution: 1920x1080
• Aspect Ratio: 16:9
• Resolution: 1920x1080
• Window Mode: Fullscreen

To ensure the agent can control the game effectively, apply the following settings in RDR2:

1. Mouse Mode: Set to DirectInput.
2. Controls:
   • Set both Tap and Hold Speed Control to on (allows running with 'w' without holding shift).
   • Set Aiming Mode to Hold To Aim.
3. Graphics/Screen:
   • Resolution: 1920x1080 (or any 16:9 aspect ratio).
   • Screen Type: Windowed Borderless.
4. Display/Mini-map:
   • Radar Blip Size = Large.
   • Map Blip Size = Large.
   • Radar = Expanded (or press Alt + X).
5. Subtitles: Enable showing the speaker's name.