AndroidWorld Environment Project

Follow these steps to set up the AndroidWorld environment:

1. Set up Android Emulator:

   • Download Android Studio.
   • Create an Android Virtual Device (AVD) with these specific settings:
     • Hardware: Pixel 6
     • System Image: Tiramisu, API Level 33
     • AVD Name: AndroidWorldAvd

2. Launch Emulator via CLI: Launch the emulator using the -grpc 8554 flag (required for accessibility forwarding). Do not use the Android Studio UI.

   # Path varies by OS
   EMULATOR_NAME=AndroidWorldAvd
   ~/Library/Android/sdk/emulator/emulator -avd $EMULATOR_NAME -no-snapshot -grpc 8554

3. Python Environment Setup (Recommended using conda):

   conda create -n android_world python=3.11.8
   conda activate android_world

4. Install AndroidWorld Package:

   git clone https://github.com/google-research/android_world.git
   cd ./android_world
   pip install -r requirements.txt
   python setup.py install

5. Configure API Keys: Set your model provider keys as environment variables (e.g., in .bashrc):

   export OPENAI_API_KEY=your-key
   export GCP_API_KEY=your-key

6. Install ffmpeg:

   • Linux (Ubuntu/Debian): sudo apt update && sudo apt install ffmpeg
   • macOS: brew install ffmpeg

# Example launch command
EMULATOR_NAME=AndroidWorldAvd
~/Library/Android/sdk/emulator/emulator -avd $EMULATOR_NAME -no-snapshot -grpc 8554

For apps using file system storage, extend task_eval.TaskEval and use file_validators for verification.

1. Define the Task Class: Inherit from task_eval.TaskEval.
2. Configure Metadata: Set app_names, complexity, schema (e.g., file_validators.CreateFile.schema), and a template string.
3. Initialize Validators: In __init__, instantiate a validator (e.g., file_validators.CreateFile) passing the params and the app's data directory (e.g., device_constants.MARKOR_DATA).
4. Lifecycle Methods: Implement initialize_task, is_successful, and tear_down by delegating to the validator's corresponding methods.
5. Parameter Generation: Implement generate_random_params to provide the necessary data for the task template.

class MarkorCreateNote(task_eval.TaskEval):
  app_names = ("markor",)
  complexity = 2
  schema = file_validators.CreateFile.schema
  template = "Create a new note in Markor named {file_name} with the following text: {text}"

  def __init__(self, params: dict[str, Any]):
    super().__init__(params)
    self.create_file_task = file_validators.CreateFile(
        params, device_constants.MARKOR_DATA
    )

  def initialize_task(self, env: interface.AsyncEnv) -> None:
    super().initialize_task(env)
    self.create_file_task.initialize_task(env)

  def is_successful(self, env: interface.AsyncEnv) -> float:
    super().is_successful(env)
    return self.create_file_task.is_successful(env)

  @classmethod
  def generate_random_params(cls) -> dict[str, str | int]:
    return {"file_name": _generate_random_file_name(), "text": _generate_random_file_text()}

  def tear_down(self, env: interface.AsyncEnv) -> None:
    super().tear_down(env)
    self.create_file_task.tear_down(env)

Use run.py to execute the benchmark suite.

Important: The first time you run this, you must include --perform_emulator_setup to install necessary apps and set permissions. This is a one-time setup.

Key Arguments:

• --suite_family: Set to android_world for standard tasks or miniwob for web-based tasks.
• --agent_name: The name of the agent to evaluate (e.g., t3a_gpt4).
• --tasks: (Optional) A comma-separated list of specific tasks to run (e.g., ContactsAddContact,ClockStopWatchRunning). If omitted, the entire suite runs.
• --perform_emulator_setup: Required for the initial app/permission setup.
• --checkpoint_dir: Use this to resume a failed run by pointing to the original output directory.

Running MiniWoB++: Set --suite_family=miniwob. These tasks render common web elements as native Android UI widgets (like time-pickers) to test agent generalization.

python run.py \
  --suite_family=android_world \
  --agent_name=t3a_gpt4 \
  --perform_emulator_setup \
  --tasks=ContactsAddContact,ClockStopWatchRunning

To extend AndroidWorld with SQLite-based tasks, follow these steps to leverage the sqlite_validators abstractions.

1. Define a Data Class: Create a @dataclasses.dataclass(frozen=True) that mirrors the SQLite table structure.
2. Create a Base Task Class: Inherit from sqlite_validators.SQLiteApp. Specify app_name_with_db, app_names, db_key, db_path, table_name, and row_type (your data class).
3. Implement Task Logic: Create a class that extends both your base class and sqlite_validators.AddMultipleRows. Define:
   • complexity: Integer value.
   • template: A string with placeholders for task parameters.
   • _get_random_target_row(): Returns a target data instance.
   • validate_addition_integrity(): Uses sqlite_validators.validate_rows_addition_integrity to compare before, after, and reference_rows using specific compare_fields.
   • generate_random_params(): Returns a dictionary containing task parameters and special keys like sqlite_validators.ROW_OBJECTS and sqlite_validators.NOISE_ROW_OBJECTS.

Note: AndroidWorld automatically handles state via initialize_state and tear_down.

@dataclasses.dataclass(frozen=True)
class CalendarEvent:
  start_ts: int
  end_ts: int
  title: str
  location: str = ''
  description: str = ''
  repeat_interval: int = 0
  repeat_rule: int = 0

class _SimpleCalendar(sqlite_validators.SQLiteApp):
  app_name_with_db = "simple calendar pro"
  app_names = ("simple calendar pro",)
  db_key = "id"
  db_path = "data/data/com.simplemobiletools.calendar.pro/databases/events.db"
  table_name = "events"
  row_type = CalendarEvent

class SimpleCalendarAddOneEvent(sqlite_validators.AddMultipleRows, _SimpleCalendar):
  complexity = 2
  template = "In Simple Calendar Pro, create a calendar event on {year}-{month}-{day}..."

  def validate_addition_integrity(self, before, after, reference_rows) -> bool:
     return sqlite_validators.validate_rows_addition_integrity(
          before, after, reference_rows,
          compare_fields=['start_ts', 'end_ts', 'title', 'location', 'description']
      )