Flightmare Quadrotor Simulator

Flightmare is a modular quadrotor simulator consisting of two decoupled components: a configurable rendering engine (built on Unity) and a flexible physics engine for dynamics simulation.

Key features include:

• A large multi-modal sensor suite (including 3D point-cloud extraction).
• An API for reinforcement learning capable of simulating hundreds of quadrotors in parallel.
• Virtual-reality (VR) headset integration for environmental interaction.
• Support for various applications like path-planning, visual-inertial odometry, and deep learning.

Flightmare is a modular quadrotor simulator designed for applications such as reinforcement learning, path-planning, and visual-inertial odometry. It consists of two decoupled components that can run independently:

1. Rendering Engine: A configurable engine built on Unity.
2. Physics Engine: A flexible engine for dynamics simulation.

Key features include a multi-modal sensor suite (with 3D point-cloud extraction), an API for parallel reinforcement learning (simulating hundreds of quadrotors), and VR headset integration.

The Flightmare binaries are released with the following environments:

• Industrial (Included in the git repository version)
• Warehouse
• Garage
• NaturalForest

Note: The git repository version only includes the Industrial environment.

Flightmare is organized into several specialized software components:

• flightlib: The core Flightmare Library. It handles quadrotor dynamics, sensor simulation, the Unity bridge, and provides a Python wrapper.
• flightrender: The Flightmare Rendering Engine. It provides photo-realistic 3D environments and outputs RGB images, depth, and segmentation data.
• flightrl: Reinforcement Learning module. Contains deep RL algorithms (e.g., PPO) and examples like quadrotor control.
• flightros: ROS Wrapper. Interfaces the Flightmare Library with ROS, including examples for PID controller quadrotor control and RGB camera simulation.

Sensors are actors that must be attached to a parent quadrotor. Once attached, they follow the quadrotor and gather data from the surroundings after each rendering step.

To use a sensor, you must provide a function that defines how to handle the retrieved data.

Available Sensors:

• Cameras:
  • RGB
  • Depth
  • Semantic segmentation
• Collision detector

Note: IMU, Lidar raycast, Optical flow, and Event-based cameras are planned for future updates.

The calibration matrix is not extracted from Unity; it must be calculated on the client-side (e.g., in ROS or C++). Because image dimensions (width x height) and the Field of View (FOV) are user-defined, you must compute the focal length manually.

1. Calculate focal length (f): f = (image.height / 2.0) / tan((M_PI * FOV / 180.0) / 2.0)

2. Construct the intrinsic matrix: Since fx = fy, the matrix is:

   [ [fx, 0, image.width/2], 
     [0, fy, image.height/2], 
     [0, 0, 1] ]

An actor is any entity that plays a role in the simulation and is loaded dynamically into the environment.

Common actors include:

• Quadrotors
• Sensors
• Gates

Actor prefabs are located in Assets/Resources/ on the server-side and can be added to the simulation by the client via the unity-bridge.

Flightmare operates using a client-server architecture to separate simulation logic from rendering:

• Client: The module run by the user to model dynamics and control the simulation. It communicates with the server via a specific IP and port.
• Server: The rendering engine that represents the simulation. It provides the main API methods to spawn quadrotors, modify the environment, and retrieve the current state of the simulation.

This separation allows the user to run complex control logic in a client while the server handles the heavy lifting of physics and rendering.

When editing the Unity project, load the Top_Level_Scene. It contains three essential GameObjects required for the Flightmare environment:

• HD_Camera: The main camera responsible for rendering images.
• Camera_Controller: The central controller used for loading different scenes, moving the camera, retrieving camera poses, and publishing images.
• splash_window: The UI interface.

When running the application in the Unity Editor (clicking Play), the application opens two TCP sockets: one for publishing images and one for subscribing to camera poses.