AI2-THOR Documentation

AI2-THOR provides three primary interaction frameworks:

• iTHOR: A high-level interaction framework for research in embodied common sense reasoning.
• ManipulaTHOR: A mid-level interaction framework for visual manipulation of objects using a robotic arm.
• RoboTHOR: A framework for Sim2Real research using simulated scene counterparts of the physical world.

AI2-THOR 1.0 replaced the abstract inventory system with a physical Agent Hand.

• World Space Manipulation: When an object is picked up, it moves to the Hand in world space. This means holding large objects (like a baseball bat) can physically restrict the Agent's ability to move through narrow spaces.
• Visibility Obstruction: Held objects realistically block the Agent's view. The size of the object affects visibility; for example, holding a basketball may block the view of smaller objects on a desk.

In AI2-THOR 1.0, visibility and interactability are decoupled to allow for more realistic scenarios (e.g., seeing through glass but being unable to touch what is behind it).

• Visibility: An object is considered visible if the Agent's camera can draw a line to any part of the object's form (previously, it required a line to the center of the object).
• Interactability: An object is interactable only if it is both visible and reachable.
• Transparency: Some objects (e.g., glass shower doors, glass cups) have a Transparent property. The Agent can see through these objects, but they still obstruct physical interaction with objects behind them.

Debug Mode Colors:

• Magenta: Visible and Interactable.
• Yellow: Visible, but NOT Interactable (e.g., blocked by a transparent object or out of reach).

In AI2-THOR, objects are categorized by how they interact with physics and agent actions:

• Pickupable objects: Small objects that can be picked up by the agent's hand. They possess an objectOrientedBoundingBox metadata value that remains constant regardless of rotation.
• Moveable objects: Larger objects (e.g., Chair, Table, Sofa, Microwave, Toaster) that are not explicitly attached to the scene structure. They cannot be picked up by the agent, but they can be manipulated using physics-based actions like PushObject and PullObject.

All Sim Objects possess an axisAlignedBoundingBox metadata value, which represents a world-axis-oriented box encompassing the object. Note that this box's volume may change if the object rotates because it aligns with world axes rather than local object axes.

ManipulaTHOR is a specialized environment within AI2-THOR designed for visual manipulation using a robotic arm. It introduces several actions specifically for arm-based interaction:

• MoveArm: Uses inverse kinematics to move the position of the grasper.
• MoveArmBase: Adjusts the height of the arm by moving the base up and down.
• PickupObject: Uses the grasper to pick up SimObjects.
• ReleaseObject: Drops objects currently held by the grasper.
• SetHandSphereRadius: Adjusts the radius of the arm's grasper.
• MoveContinuous: Enables continuous movement that detects collisions while moving through the scene.
• RotateContinuous: Enables continuous rotation that detects collisions while rotating within the scene.

In AI2-THOR 2.0, certain state changes occur automatically based on the environment and object interactions without requiring explicit agent actions. These are known as Contextual Interactions.

Examples include:

• Breaking: Breakable objects may break if dropped with sufficient force.
• Cleaning: Dirty dishwater becomes clean when moved under running water.
• Cooking: Potatoes are cooked when moved over an active stove burner.
• Filling: Fillable objects are filled with water when moved under a running water source.
• Extinguishing: Lit candles are put out if placed in water.

Receptacles in AI2-THOR 1.0 have improved fidelity and physics:

• Placement Fidelity: Objects can be placed anywhere across the entire surface of a receptacle with unrestricted rotation (previously limited to specific points/rotations).
• Moveable Receptacles: If a receptacle can be picked up (e.g., a bowl), any objects contained within it move collectively with the receptacle (e.g., picking up a bowl with an egg inside).
• Animated Open/Close: Openable receptacles (like cabinets) now feature animated opening and closing. These animations are physics-aware; if an object (like a chair) blocks the opening path, the animation will attempt to push the obstructing object out of the way.

Each scene supports randomizing object locations within valid receptacles.

• Seeding: Randomization can be seeded to ensure reproducible scene states.
• Constraints: Objects are placed based on their receptacle restrictions (e.g., a tomato can spawn in a Microwave or Fridge, but not in a Drawer).
• Limitations: Large objects like Fridges or Sofas currently cannot be randomized.

AI2-THOR includes WebGL templates (located in unity/Assets/WebGLTemplates) that can be used as a starting point for custom interfaces or crowdsourcing tasks.

To run the HideNSeek demo template:

1. Open the AI2-THOR project in Unity.
2. Select the WebGL Template named HideNSeek (unity/Assets/WebGLTemplates/HideNSeek).
3. In the Unity Build menu, switch the target platform to WebGL/HTML5 and build.
4. Once the local server launches, ensure your URL includes the required GET parameters for objects: http://localhost:<port>?object=Tomato&variation=0

Supported object values (case-sensitive):

• Tomato
• Bread
• Plunger
• Cup
• Knife

Note: Always leave variation=0.

You can customize the keyboard shortcut used to toggle Isolate mode by accessing the Isolationist preferences.

1. Open Edit > Preferences... > Isolationist.
2. Select your desired base key and modifiers.

AI2-THOR provides specialized documentation for its different environments and development paths:

• iTHOR Documentation: For the standard iTHOR environment.
• ManipulaTHOR Documentation: For the ManipulaTHOR environment focused on object manipulation.
• RoboTHOR Documentation: For the RoboTHOR environment.
• AI2-THOR Colab: A repository to run AI2-THOR in the cloud via Google Colab.
• AllenAct: An Embodied AI Framework that provides first-class support for AI2-THOR.
• Unity Development: Notes for developers editing the AI2-THOR backend in Unity.
• WebGL Development: Documentation for packaging AI2-THOR for the web (useful for annotation tasks).

You can also interact with a live demo in your browser at the official AI2-THOR demo site.