emotion2vec

To evaluate emotion2vec on the IEMOCAP dataset, you can generate the necessary features and labels from the raw dataset.

1. Download the IEMOCAP_full_release from the official website.
2. Define your local paths for the dataset root, manifest, checkpoints, features, and model directories.
3. Run the manifest generation script.
4. Run the feature extraction script using emotion2vec.
5. Copy the generated train.emo file to your feature path.

This process produces three files in your feature path: train.npy, train.lengths, and train.emo.

# Set your own paths
IEMOCAP_ROOT=/path/to/IEMOCAP_full_release
manifest_path=/path/to/manifest
checkpoint_path=/path/to/emotion2vec_ckpt
feat_path=/path/to/feats
fairseq_root=/path/to/fairseq_root
model_path=../upstream

# generate manifest and labels from raw dataset
bash scripts/iemocap_manifest_and_labels.sh $IEMOCAP_ROOT $manifest_path

# generate features with emotion2vec
bash scripts/emotion2vec_extract_features.sh $fairseq_root $manifest_path $model_path $checkpoint_path $feat_path

cp $manifest_path/train.emo $feat_path

The emotion2vec+ models are foundation models for Speech Emotion Recognition (SER). They can classify audio into 9 specific emotion classes.

When using AutoModel.generate() with an emotion2vec+ model:

• Set extract_embedding=False to get the 9-class emotion labels and scores.
• Set extract_embedding=True to get both the emotion labels/scores and the underlying features.

9-class emotion mapping: 0: angry, 1: disgusted, 2: fearful, 3: happy, 4: neutral, 5: other, 6: sad, 7: surprised, 8: unknown

Model IDs:

• iic/emotion2vec_plus_seed
• iic/emotion2vec_plus_base
• iic/emotion2vec_plus_large
• iic/emotion2vec_base_finetuned (Jan. 2024 release)

from funasr import AutoModel

# Use an emotion2vec+ model for 9-class emotion recognition
model_id = "iic/emotion2vec_plus_large"

model = AutoModel(
    model=model_id,
    hub="ms",  # "ms" or "modelscope" for China mainland users; "hf" or "huggingface" for other overseas users
)

wav_file = f"{model.model_path}/example/test.wav"
# extract_embedding=False returns {'feats', 'labels', 'scores'}
rec_result = model.generate(wav_file, output_dir="./outputs", granularity="utterance", extract_embedding=False)
print(rec_result)

To perform emotion recognition using a fine-tuned BaseModel, you need to load a checkpoint into the model instance and pass feature tensors along with a padding mask.

1. Initialize the model: Create a BaseModel instance specifying the input_dim (e.g., 768 for emotion2vec features) and the output_dim corresponding to your number of emotion labels.
2. Load weights: Use torch.load to load your checkpoint and model.load_state_dict() to apply them.
3. Prepare inputs:
   • feat: A tensor of shape (batch_size, sequence_length, input_dim).
   • padding_mask: A boolean tensor of shape (batch_size, sequence_length) where True indicates padding (though in the example, torch.zeros(...).bool() is used to indicate no padding).
4. Extract prediction: The model returns logits. Use torch.max(outputs.data, dim=1) to find the index of the highest probability emotion.
5. Map to label: Use a dictionary to map the predicted index back to a human-readable label (e.g., 'ang', 'hap', 'neu', 'sad').

import torch
from model import BaseModel

# 1. Setup label mapping
label_dict={'ang': 0, 'hap': 1, 'neu': 2, 'sad': 3}
idx2label = {v: k for k, v in label_dict.items()}

# 2. Initialize and load model
model = BaseModel(input_dim=768, output_dim=len(label_dict))
ckpt = torch.load('outputs/2024-01-14/22-57-42/model_1.pth')
model.load_state_dict(ckpt)

# 3. Prepare dummy input (batch_size=1, seq_len=100, input_dim=768)
feat = torch.randn(1, 100, 768)
padding_mask = torch.zeros(1, 100).bool()

# 4. Inference
outputs = model(feat, padding_mask)

# 5. Get prediction
_, predict = torch.max(outputs.data, dim=1)
print(idx2label[predict.item()])

The train_iemocap function serves as the main entrypoint for training downstream emotion recognition models using the IEMOCAP dataset. It uses hydra for configuration management, loading settings from config/default.yaml.

Key behaviors:

• Cross-Validation: It performs a 5-fold cross-validation by treating each of the 5 IEMOCAP sessions as a test set sequentially.
• Data Loading: Uses load_ssl_features to load pre-extracted SSL features and train_valid_test_iemocap_dataloader to create training, validation, and testing loaders.
• Model Architecture: Initializes a BaseModel with an input dimension of 768 (standard for many SSL models) and an output dimension corresponding to the number of emotion labels.
• Optimization: Uses RMSprop with a CyclicLR scheduler.
• Checkpointing: Saves the model state dictionary (.pth) for the best performing epoch based on Validation Weighted Accuracy (WA).
• Evaluation: Reports Weighted Accuracy (WA), Unweighted Accuracy (UA), and F1 score for both validation and testing phases.

@hydra.main(config_path='config', config_name='default.yaml')
def train_iemocap(cfg: DictConfig):
    # Implementation details for training on IEMOCAP
    ...

if __name__ == '__main__':
    train_iemocap()