The ego4d repository provides tools for working with the Ego4D and Ego-Exo4D datasets. Key components include:

• Downloaders:
  • ego4d CLI (command: ego4d) for the Ego4D dataset.
  • egoexo CLI (command: egoexo) for the Ego-Exo4D dataset.
• Feature Extraction: An API for extracting features, including wrappers for models like Omnivore and SlowFast.
• Research Tools: Includes dataloaders and research code (e.g., clep for Contrastive Language Ego-centric video Pre-training).
• Video Reading: A simple API abstracting libraries like TorchAudio and PyAV.
• Visualization: A visualization engine located in the viz directory.

The Human Pose Estimation pipeline follows these logical stages:

1. Preprocess egoexo data: Includes downloading files, Aria frame extraction, Exo camera frame extraction, and exporting paths/camera calibrations to JSON.
2. Obtain human bounding boxes: Utilizes Aria trajectory, FasterRCNN, and heuristics.
3. 2D Pose Detection
4. Triangulation
5. Smoothing

The review interface is a React application built using a create-react-app template with the mephisto-review template.

If you need to update the interface to integrate upstream changes from the template, you can re-run the create-react-app command and manually migrate the following files/folders:

• review/src/custom/
• review/src/index.js

npx create-react-app review --template mephisto-review

For a specific commentary, you can access the proficiency rating (a 1-10 score and text reason) and the drawing/path data.

1. Locate the data.json file for a specific commentary. The path is typically constructed using the commentary ID: os.path.join(annotation_root, "expert_commentary/", commentary_id, "data.json").
2. The proficiency key in this JSON contains the performance rating.
3. The annotations list within this JSON contains events which represent the drawing paths.

import json

# Assuming 'ann' is an annotation object from the dataset
data_path = os.path.join(egoexo_annotation_root, "expert_commentary/", ann["commentary"], "data.json")
data = json.load(open(data_path))

# Access proficiency
print(data["proficiency"])

# Access drawing events
for comm in data["annotations"]:
    if len(comm["events"]) > 0:
        print("Found drawing events")
        break

To download and view narrations in the Mephisto review interface, run the following pipeline from the recipes/ folder. This command gathers the first 5 IDs, downloads the corresponding videos, prepares the input, and launches the review server:

./1_gather_ids.sh 5 | ./2_dl_videos.sh | ./3_prepare_input.sh | ./4_review.sh

Prerequisites:

• Ego4D CLI
• Mephisto CLI
• jq (Optional: required only if creating custom input files from the recipes/ folder)

$ ./1_gather_ids.sh 5 | ./2_dl_videos.sh | ./3_prepare_input.sh | ./4_review.sh

Use the ego4d command to download specific datasets to a local directory.

Example: Download full scale videos and annotations for version 2 to ~/ego4d_data:

ego4d --output_directory="~/ego4d_data" --datasets full_scale annotations --metadata

If the ego4d command is not in your PATH, you can use the python module syntax:

python -m ego4d.cli.cli --output_directory="~/ego4d_data" --datasets full_scale annotations --metadata --version v2

The easiest way to launch the review interface is using the run_viz.sh script located in the repository root. This script installs the ego4d viz dataset and launches the interface.

Prerequisites:

1. Install Mephisto: pip install mephisto
2. Install the Ego4D CLI.

Configuration: If your videos or the viz dataset are not located in the default directory (~ ano4d_data), you must update the VID_ROOT configuration. You can modify configuration options by passing flags to the script or by examining the script's contents.

./run_viz.sh

You can install the ego4d module using either PyPi or by cloning the repository. Ensure you are using Python 3.10 or higher. It is highly recommended to use a conda or pyenv environment.

### Option 1: From the PyPi package

```bash
pip install ego4d --upgrade

Option 2: Clone/Download the Code

Step 1: Create or Use an Environment

conda create -n ego4d python=3.11 -y
conda activate ego4d

Step 2: Install from source

pip install .