DOCSEARCHbyAlgolia
Submit Docs

UnrealCV Documentation

repository·5.2·Indexed 24 days ago

https://github.com/unrealcv/unrealcv

A toolset for computer vision researchers to build and interact with virtual worlds using Unreal Engine. It provides a bridge between Unreal Engine and machine learning frameworks like PyTorch and TensorFlow via a Python client and server plugin. Features include a multi-modal sensor system for capturing RGB images, depth maps, surface normals, and optical flow, as well as high-level APIs for object manipulation, scene annotation, and Blueprint integration. Supports Unreal Engine 4 (4.12, 4.13, 4.14) and Unreal Engine 5.6.

Tokens
12.3K
Snippets
31
Records
85
Agent score
75%

What's inside UnrealCV

  1. Overview of UnrealCV

    5.2

    UnrealCV is a tool designed for computer vision researchers to build virtual worlds using Unreal Engine 4 (UE4). It functions by extending UE4 with a plugin that provides:

    1. UnrealCV commands: A set of commands to interact with the virtual world.
    2. Communication bridge: Enables communication between UE4 and external programs like Caffe or TensorFlow.

    Supported Platforms:

    • UE4 Versions: 4.12, 4.13, 4.14.
    • Operating Systems: Windows, Linux, and Mac.

    Supported Clients:

    • Python
    • MATLAB (experimental)

  2. Overview of UnrealCV usage modes

    5.2

    UnrealCV provides two primary ways to interact with virtual worlds:

    1. Compiled Game Binary: Use a pre-built game binary that has UnrealCV embedded. This method requires no knowledge of Unreal Engine; you simply run the game.
    2. Unreal Engine Plugin: Install the UnrealCV plugin directly into Unreal Engine and use the Unreal Editor to build and customize your own virtual worlds.

    UnrealCV facilitates communication between the Unreal Engine environment and external programs like PyTorch or TensorFlow.

  3. Understand the UnrealCV sensor system

    5.2

    UnrealCV uses a multi-modal sensor system to capture data from virtual scenes. Sensors are implemented as components, and the primary wrapper for these components is the AFusionCameraActor.

    Camera Sensors

    Camera sensors capture 2D data by inheriting from UBaseCameraSensor (which itself inherits from USceneCaptureComponent2D).

    Key sensor types include:

    • UFusionCamSensor: A combined sensor that includes multiple functions. This is the recommended sensor to use.
    • UPawnCamSensor: A special version of UFusionCamSensor that automatically tracks the viewpoint of the Pawn. This corresponds to camera 0 in the Python API.
    • ULitCamSensor: Captures lit RGB data.
    • UDepthCamSensor: Captures depth data.
    • UNormalCamSensor: Captures surface normals.
    • UAnnotationCamSensor: Extracts RGB annotation colors.

    Non-Camera (Legacy) Sensors

    These sensors extract 3D or structural information:

    • FSkeletonSensor: Extracts data from a SkeletalMesh.
    • FVertexSensor: Extracts vertex data from a mesh.
    • FBoneSensor: Extracts bone-related data.

  4. Capture depth data in UnrealCV

    5.2

    UnrealCV supports two distinct methods for calculating depth data:

    1. Distance to the camera center: Measures the radial distance from the camera's optical center to the object.
    2. Distance to the camera plane: Measures the perpendicular distance from the camera's plane to the object.

  5. How UnrealCV architecture works

    5.2

    UnrealCV facilitates Inter-Process Communication (IPC) between an Unreal Engine 4 game and computer vision algorithms. The system is split into two primary components:

    1. unrealcv server: An embedded module loaded into an Unreal Engine 4 game. Upon launching the game, the server starts a TCP server and waits for incoming commands.
    2. unrealcv client: An external program (using Python, MATLAB, or other languages) that sends plain text commands to the server via TCP to control the scene or retrieve information.

    Commands are parsed using regular expressions, and the modular design allows for easy extension of the command system. This architecture ensures the system is cross-platform and language-agnostic.

  6. Understand the UnrealCV project layout

    5.2

    The repository is organized into the following structure:

    • client/: Contains client-side implementation and usage examples.
      • examples/: Task-oriented examples using client code.
      • matlab/: MATLAB client implementation.
      • python/: Python client implementation.
      • scripts/: Build and packaging scripts.
    • Content/: Plugin data files.
    • docs/: Project documentation.
    • Resources/: Plugin resource files.
    • Source/: C++ source code for the UnrealCV plugin.
    • test/: Test suites.
    • UnrealCV.uplugin: The descriptor file required to install the plugin into Unreal Engine 4.
    client/            # Client code for Python and MATLAB
    examples/      # Examples showing how to use client code to do tasks
    matlab/        # MATLAB client
    python/        # Python client
    scripts/       # Scripts for tasks, such as building and packaging
Content/           # Plugin data
docs/              # Documentation of UnrealCV
Resources/         # Plugin resource
Source/            # Plugin C++ source code
test/              # Test code
UnrealCV.uplugin   # Descriptor file for an UE4 plugin
README.md

  8. Compile UnrealCV plugin from source code

    5.2

    To use experimental features or versions not available on GitHub, you can compile the plugin from source. This requires the unrealcv Python package for the unrealcv.automation module.

    Compilation Steps

    1. Install the required Python package:
      pip install -U unrealcv
    2. Run the build script. The script attempts to automatically locate your Unreal Engine installation on Windows and Mac:
      python build.py
    3. If the script fails to find your engine, manually specify the path using the --UE flag:
      python build.py --UE {UE_PATH}

    Verification and Deployment

    • A successful build will display: Automation.Execute: BUILD SUCCESSFUL.
    • The resulting binaries will be located in the Plugins/UnrealCV folder.
    • Copy these binaries to your project's Plugins folder or the Engine's Engine/Plugins folder to complete the installation.
    # Install the latest version of unrealcv
pip install -U unrealcv

# Build using automatic engine detection
python build.py

# Build by manually specifying the Unreal Engine path
python build.py --UE "/Users/Shared/Epic Games/UE_5.2"

  10. Compile the C++ project on Windows

    5.2
    1. Install Visual Studio.
    2. Right-click playground.uproject and select Generate Visual Studio project files.
    3. Open the resulting .sln solution file in Visual Studio and build the solution. The UnrealCV plugin will be compiled together with the game project.

  11. Prepare UE4 for UnrealCV packaged binaries

    5.2

    To ensure UnrealCV commands function correctly in a packaged game binary, you must modify the Unreal Engine configuration. Add the following line to the end of your engine config file located at Engine\Config\ConsoleVariables.ini:

    r.ForceDebugViewModes = 1

    Without this setting, the packaged game may not allow the view mode changes required by UnrealCV.

    r.ForceDebugViewModes = 1

  12. Find 3D resources for UE4 and UnrealCV

    5.2

    To create high-quality synthetic datasets, you can source 3D models and scenes from the following providers:

    • Unreal Marketplace: Official assets, including specialized architectural visualization and feature examples.
    • UE4Arch: High-quality architectural visualization.
    • Evermotion: Professional 3D assets.
    • TurboSquid: Industry-standard high-quality 3D models.
    • ShapeNet: Free 3D models provided by the research community.