repository·master·Indexed 21 days ago
https://github.com/stanfordasl/trajectron-plus-plusA framework for dynamically-feasible trajectory forecasting using heterogeneous data. It supports training and evaluation on datasets including ETH, UCY, and nuScenes, incorporating dynamics integration and map information. The framework includes tools for data processing, model training (GPU/CPU), offline evaluation, and online execution for streaming data.
Preprocessed data splits for ETH and UCY datasets are located in experiments/pedestrians/raw. To convert them into the format required by the model, run the process_data.py script from the experiments/pedestrians directory. This process typically takes 10-15 minutes.
cd experiments/pedestrians
python process_data.pyRun the training script from the trajectron/ directory. You must specify the training and evaluation data dictionaries, a log directory, and a model configuration file.
Example: Training on ETH - University without dynamics integration:
python train.py --eval_every 10 --vis_every 1 --train_data_dict eth_train.pkl --eval_data_dict eth_val.pkl --offline_scene_graph yes --preprocess_workers 5 --log_dir ../experiments/pedestrians/models --log_tag _eth_vel_ar3 --train_epochs 100 --augment --conf ../experiments/pedestrians/models/eth_vel/config.jsonKey Arguments:
--train_data_dict / --eval_data_dict: Path to the .pkl data files.--offline_scene_graph yes: Preprocesses the scene graph to speed up training.--log_dir: Directory where models and Tensorboard logs are saved.--log_tag: A suffix added to the log directory name.--conf: Path to the JSON model configuration file.--augment: Enables dataset rotation augmentation.--eval_every: Frequency of evaluation in epochs.--vis_every: Frequency of Tensorboard visualization in epochs.To use the nuScenes dataset, you must first download it from the nuScenes website and place the contents in experiments/nuScenes.
Setup Steps:
experiments/nuScenes.maps folder contents into experiments/nuScenes/v1.0-mini/maps.Processing Commands:
Run process_data.py from the experiments/nuScenes directory using one of the following commands depending on your dataset version:
For mini nuScenes:
python process_data.py --data=./v1.0-mini --version="v1.0-mini" --output_path=../processed
For full nuScenes:
python process_data.py --data=./v1.0 --version="v1.0-trainval" --output_path=../processed
Note: To generate a validation set (by default only training and test sets are produced), modify line 406 in process_data.py to: val_scene_names = val_scenes.
cd experiments/nuScenes
# Mini nuScenes
python process_data.py --data=./v1.0-mini --version="v1.0-mini" --output_path=../processed
# Full nuScenes
python process_data.py --data=./v1.0 --version="v1.0-trainval" --output_path=../processedTrajectron++ supports an "online" execution mode for streaming data. Use the test_online.py script to run predictions iteratively. During execution, predictions are visualized and saved as figures in a pred_figs/ folder located within the specified model directory.
Usage Notes:
test_online.py at line 110.test_online.py.CLI Arguments:
--log_dir <path>: Directory containing model logs.--data_dir <path>: Directory containing processed data.--conf <path_to_json>: Path to the configuration JSON.--eval_data_dict <path_to_pkl>: The specific evaluation data file.--map_encoding (Optional): Required for models using map information.--incl_robot_node (Optional): Required for models using robot future information.# Example: Online execution with Dynamics Integration and Maps
python test_online.py --log_dir=../experiments/nuScenes/models --data_dir=../experiments/processed --conf=config.json --eval_data_dict=nuScenes_test_mini_full.pkl --map_encodingWhen cloning the repository, you must include submodules to ensure all dependencies and data are correctly loaded. You can do this in one step using the --recurse-submodules flag, or by initializing and updating them manually after a standard clone.
Note: If you require the version used for ECCV 2020, check out the eccv2020 branch. The master branch contains newer updates that may not be interoperable with the ECCV 2020 version.
# Option 1: Clone everything at once
git clone --recurse-submodules <repository cloning URL>
# Option 2: Clone normally, then load submodules
git submodule init
git submodule updateTo install the necessary dependencies, create a new Conda environment with Python 3.6 and install the requirements via pip. Additionally, register the environment as an IPython kernel to use it within Jupyter notebooks.
# Create and activate the environment
conda create --name trajectron++ python=3.6 -y
source activate trajectron++
# Install dependencies
pip install -r requirements.txt
# Register as an IPython kernel
python -m ipykernel install --user --name trajectronpp --display-name "Python 3.6 (Trajectron++)"conda create --name trajectron++ python=3.6 -y
source activate trajectron++
pip install -r requirements.txt
python -m ipykernel install --user --name trajectronpp --display-name "Python 3.6 (Trajectron++)"Training commands for nuScenes are executed from the trajectron/ directory. Different model variants are achieved by changing the --conf file and specific flags.
| Model Variant | Command Flags |
|---|---|
| Base | --conf .../vel_ee/config.json |
| +Dynamics Integration | --conf .../int_ee/config.json |
| +Dynamics, Maps | --conf .../int_ee/config.json --map_encoding |
| +Dynamics, Maps, Robot Future | --conf .../robot/config.json --incl_robot_node --map_encoding |
To train without the ego-vehicle:
Use the command for '+Dynamics Integration, Maps' but modify line 132 of train.py to return return_robot=False.
# Example: Base Model Training
python train.py --eval_every 1 --vis_every 1 --conf ../experiments/nuScenes/models/vel_ee/config.json --train_data_dict nuScenes_train_full.pkl --eval_data_dict nuScenes_val_full.pkl --offline_scene_graph yes --preprocess_workers 10 --batch_size 256 --log_dir ../experiments/nuScenes/models --train_epochs 20 --node_freq_mult_train --log_tag _vel_ee --augmentThe 'Best of 20' evaluation measures the error of the best trajectory among 20 sampled trajectories. This is useful for assessing the quality of the model's generative capabilities.
results/{dataset}_12*fde_best_of.csv.results/{dataset}_12*ade_best_of.csv.These can be applied to both position and velocity (using the _vel_ prefix in the filename).
# Example: Loading Best-of-20 FDE results
for dataset in dataset_names:
for f in glob.glob(f"results/{dataset}_12*fde_best_of.csv"):
dataset_df = pd.read_csv(f)
dataset_df['method'] = "Ours"
# ... process dataKDE NLL analysis evaluates the model's probability density estimation.
results/{dataset}_12*kde_full.csv.results/{dataset}_vel_12*kde_full.csv.Calculate the mean of the value column for each dataset to compare performance.
# Example: Calculating mean KDE NLL for position
for dataset in dataset_names:
files = glob.glob(f"results/{dataset}_12*kde_full.csv")
for f in files:
dataset_df = pd.read_csv(f)
print(f"{alg_name} KDE NLL for {dataset}: {dataset_df['value'].mean()}")To perform displacement error analysis, you need to load and combine CSV result files for different methods (e.g., 'Ours', 'Trajectron', 'sgan').
results/{dataset}*attention_radius_3*fde_most_likely.csv (for FDE) or results/{dataset}*attention_radius_3*ade_most_likely.csv (for ADE). Ensure you rename the metric column to error_type and value to error_value.csv/old/curr_*_errors.csv. Note that in these files, you may need to map fse to fde and mse to ade to maintain consistency.seaborn.boxplot to compare error distributions across datasets like eth, hotel, univ, zara1, and zara2.# Example: Loading 'Ours' FDE results
import pandas as pd
import glob
perf_df = pd.DataFrame()
for dataset in ['eth', 'hotel', 'univ', 'zara1', 'zara2', 'Average']:
for f in glob.glob(f"results/{dataset}*attention_radius_3*fde_most_likely.csv"):
dataset_df = pd.read_csv(f)
dataset_df['dataset'] = dataset
dataset_df['method'] = "Ours"
perf_df = perf_df.append(dataset_df, ignore_index=True, sort=False)
perf_df = perf_df.rename(columns={"metric": "error_type", "value": "error_value"})To perform qualitative analysis with nuScenes data, you must initialize the NuScenesMap from the nuScenes devkit. This requires the path to your nuScenes dataset and the devkit path. You can specify the map name (e.g., 'boston-seaport') to load the corresponding spatial data.
nuScenes_data_path = # Data Path to nuScenes data set
nuScenes_devkit_path = './devkit/python-sdk/'
sys.path.append(nuScenes_devkit_path)
from nuscenes.map_expansion.map_api import NuScenesMap
nusc_map = NuScenesMap(dataroot=nuScenes_data_path, map_name='boston-seaport')To visualize trajectories within a specific geographic area, use nusc_map.render_map_patch to create a background map and then overlay trajectories using plot_vehicle_nice or plot_vehicle_mm.
Steps:
x_min, y_min, x_max, and y_max.layers to render (e.g., 'drivable_area', 'lane', 'ped_crossing').nusc_map.render_map_patch.plot_vehicle_nice (for standard samples) or plot_vehicle_mm (for mode-specific samples) to draw on the axes.# 1. Define ROI and layers
x_min, x_max, y_min, y_max = 773.0, 1100.0, 1231.0, 1510.0
layers = ['drivable_area', 'road_segment', 'lane', 'ped_crossing', 'walkway', 'stop_line', 'road_divider', 'lane_divider']
# 2. Render map patch
my_patch = (x_min, y_min, x_max, y_max)
fig, ax = nusc_map.render_map_patch(my_patch, layers, figsize=(10, 10), alpha=0.1, render_egoposes_range=False)
# 3. Plot trajectories
plot_vehicle_nice(ax, predictions, scene.dt, max_hl=10, ph=ph, map=None, x_min=x_min, y_min=y_min)
plot_vehicle_mm(ax, predictions_mm, scene.dt, max_hl=10, ph=ph, map=None, x_min=x_min, y_min=y_min)