Moment-DETR

This directory provides a simplified model checkpoint, model_best.ckpt, designed to streamline the feature extraction pipeline.

Key Characteristics:

• Inputs: Uses only CLIP image and text features.
• Training: Trained from scratch without ASR (Automatic Speech Recognition) pre-training.
• Performance Note: Because it lacks ASR pre-training, performance may be lower than the results reported in the original paper.

Included Files:

• model_best.ckpt: The model checkpoint.
• Training/evaluation log files.
• Training configurations.
• Prediction files for the QVHighlights validation split.

To use Moment-DETR, you need to clone the repository, prepare the feature files, and set up a Conda environment with specific dependencies.

1. Clone the repository:

git clone https://github.com/jayleicn/moment_detr.git
cd moment_detr

2. Prepare feature files: Download moment_detr_features.tar.gz (8GB) and extract it into the project root directory:

tar -xf path/to/moment_detr_features.tar.gz

3. Create Conda environment and install dependencies: This project requires Python 3.7 and PyTorch (tested with version 1.9.0).

# create conda env
conda create --name moment_detr python=3.7
# activate env
conda activate moment_detr
# install pytorch with CUDA 11.0
conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch
# install other python packages
pip install tqdm ipython easydict tensorboard tabulate scikit-learn pandas

To run inference using a saved checkpoint, use the inference.sh script. You must provide the path to the checkpoint and the name of the split you are evaluating (e.g., val or test).

bash moment_detr/scripts/inference.sh CHECKPOINT_PATH SPLIT_NAME

Moment-DETR supports weakly supervised pretraining using ASR captions and subsequent finetuning.

1. Pretraining: Run the pretraining script to train for 100 epochs on ASR captions. Checkpoints are saved in results.

bash moment_detr/scripts/pretrain.sh

2. Finetuning: To finetune, use the standard training script but pass the path to your pretrained checkpoint using the --resume flag.

bash moment_detr/scripts/train.sh --resume ${PRETRAIN_CHECKPOINT_PATH}

# Pretraining
bash moment_detr/scripts/pretrain.sh

# Finetuning
bash moment_detr/scripts/train.sh --resume ${PRETRAIN_CHECKPOINT_PATH}

To evaluate your model's predictions using the provided sample files, run the evaluation script from the project root. This uses eval.py to process sample_val_preds.jsonl and outputs the results to sample_val_preds_metrics.json.

bash standalone_eval/eval_sample.sh

To submit to the Codalab evaluation server (if available), you must create a single .zip file.

Important: The .zip file must not contain an enclosing folder; the files must be at the root of the archive.

The archive must contain exactly two files:

1. hl_val_submission.jsonl
2. hl_test_submission.jsonl

Both files must follow the JSONL format specified in the prediction format documentation.

To use your own dataset, you must:

1. Prepare annotations in the same format as QVHighlights (see the data/ directory).
2. Extract video features using HERO_Video_Feature_Extractor.
3. Copy moment_detr/scripts/train.sh and modify the dataset-specific parameters, such as annotation paths and feature paths.

You can train Moment-DETR on the QVHighlights train split using the provided training script. By default, it trains for 200 epochs using SlowFast and Open AI CLIP features. Results and checkpoints are saved in the results directory.

To train without the saliency loss, you can set the --lw_saliency flag to 0.

# Standard training
bash moment_detr/scripts/train.sh 

# Training without saliency loss
bash moment_detr/scripts/train.sh --lw_saliency 0

All raw video data for the QVHighlights dataset can be downloaded from the following link:

https://nlp.cs.unc.edu/data/jielei/qvh/qvhilights_videos.tar.gz

To run predictions on your own raw videos and text queries, follow these steps:

1. Install additional dependencies:

pip install ffmpeg-python ftfy regex

2. Execute the prediction script: Set the PYTHONPATH to the current directory and run run_on_video/run.py. This script loads a default checkpoint and runs an example prediction.

For custom implementations, refer to the run_example function inside run_on_video/run.py.

pip install ffmpeg-python ftfy regex
PYTHONPATH=$PYTHONPATH:. python run_on_video/run.py