DOCSEARCHbyAlgolia
Submit Docs

CMU-Multimodal SDK (mmsdk)

repository·main·Indexed 18 days ago

https://github.com/cmu-multicomp-lab/cmu-multimodalsdk

A toolkit for loading multimodal datasets and constructing neural multimodal deep learning models. It consists of two primary modules: mmdatasdk for downloading and processing datasets using computational sequences (.csd files), and mmmodelsdk for building models with a library of fusion models, including Tensor Fusion Networks (TFN). The SDK supports benchmark datasets such as CMU-MOSEI, CMU-MOSI, Social-IQ 1.0, and POM.

Tokens
3.5K
Snippets
7
Records
15
Agent score
63%

What's inside cmu-multimodalsdk

  1. Overview of the CMU-MOSI dataset

    main

    The CMU-Multimodal Opinion Sentiment Intensity (MOSI) dataset is used for sentiment analysis tasks. Key characteristics include:

    • Content: 93 videos featuring 89 distinct speakers, all discussing movie reviews.
    • Structure: Each video contains exactly one speaker.
    • Balance: The dataset is approximately gender-balanced.
    • Sentiment Scoring: Scores range from -3 (extremely negative) to 3 (extremely positive).
    • Sentiment Binarization:
      • Traditional approach: Negative is defined as score < 0; non-negative is score >= 0.
      • Modern approach (post-2019): Negative is score < 0; positive is score > 0. Values equal to zero are treated separately. This newer method typically yields higher accuracy.
    • 7-class Sentiment: Discretized range [-3, 3] is achieved using the round function.

  2. Overview of the CMU-MOSEI dataset

    main

    CMU-MOSEI (CMU Multimodal Opinion Sentiment and Emotion Intensity) is a large-scale dataset for sentence-level sentiment analysis and emotion recognition using YouTube monologue videos. It contains over 65 hours of annotated video from 1000+ speakers across 250 topics.

    Label Structure: Labels are provided in an "All Labels" computational sequence with the following order: [sentiment, happy, sad, anger, surprise, disgust, fear]

    Value Ranges:

    • Sentiment: [-3, 3]
    • Emotions: [0, 3] (Multiple emotions can be present simultaneously)

  3. Use Tensor Fusion Networks for Multimodal Sentiment Analysis

    main

    The tensor_fusion module implements the Tensor Fusion Network (TFN) for multimodal sentiment analysis, as described in the EMNLP 2017 paper Tensor Fusion Network for Multimodal Sentiment Analysis.

    This implementation is designed to capture interactions between multimodal dimensions by allowing all information to interact efficiently, which can help in modeling sparse interactions and potentially reducing overfitting compared to early fusion methods.

  4. How computational sequences work in mmdatasdk

    main

    In mmdatasdk, a multimodal dataset is treated as a collection of computational sequences. Each sequence represents one modality in a hierarchical format and is self-contained, meaning it can be used to train models in isolation.

    Data Structure

    Computational sequences are stored as .csd (computational sequential data) files using the HDF5 format. Each sequence contains:

    • data: A hierarchy of features categorized by a unique multimodal source identifier (e.g., a video ID). Each source contains two 2D numpy arrays:
      • features: The computational descriptors.
      • intervals: The associated timestamps.
    • metadata: Information regarding the sequence, including versioning and integrity checks.

    Integrity and Sharing

    Because each sequence is self-contained and includes integrity metadata, they can be shared and downloaded individually. The SDK uses these integrity checks to ensure that users download the exact computational sequences intended by the provider.

  5. Understand the CMU-Multimodal SDK architecture

    main

    The SDK is composed of two primary modules designed for different stages of the multimodal machine learning workflow:

    1. mmdatasdk: A module for downloading and processing multimodal datasets. It uses computational sequences to manage data.
    2. mmmodelsdk: A module providing tools to utilize complex neural models and layers for building new multimodal models. It includes implementations of fusion models from prior research.

    This separation allows users to first handle data acquisition and alignment via mmdatasdk before moving to model construction with mmmodelsdk.

  6. Binarization rules for CMU-MOSEI sentiment and emotions

    main

    When processing CMU-MOSEI data, you may need to binarize or discretize the continuous scores.

    7-Class Sentiment Binarization

    The following logic is used to map continuous sentiment scores to 7 discrete classes:

    • < -2 $\rightarrow$ -3
    • [-2, -1) $\rightarrow$ -2
    • [-1, 0) $\rightarrow$ -1
    • 0 $\rightarrow$ 0
    • (0, 1] $\rightarrow$ 1
    • (1, 2] $\rightarrow$ 2
    • > 2 $\rightarrow$ 3

    Binary Sentiment Metrics

    There are two ways to define binary sentiment depending on the research context:

    1. negative/non-negative: < 0 or \ge 0 (Common in papers prior to 2019).
    2. negative/positive: < 0 or > 0 (Common in papers after 2019).

    Binary Emotion Binarization

    Emotions are binarized based on presence:

    • emotion = 0: Emotion is not present.
    • emotion > 0: Emotion is present.
    def cmumosei_round(a):
        if a < -2:
                res = -3
        if -2 <= a and a < -1:
                res = -2
        if -1 <= a and a < 0:
                res = -1
        if 0 <= a and a <= 0:
                res = 0
        if 0 < a and a <= 1:
                res = 1
        if 1 < a and a <= 2:
                res = 2
        if a > 2:
                res = 3

  7. Understand the difference between TFN and Bilinear Compact Pooling

    main
    While related, Tensor Fusion (TFN) and Bilinear Compact Pooling are distinct algorithms. Bilinear compact pooling is considered a special case of tensor fusion that only includes a trimodal component. In contrast, TFN uses a different tensor operation to maintain the eigenvectors of each modality (unimodal) while also modeling bimodal interactions.

  8. Install the CMU-Multimodal SDK

    main

    To use the SDK, clone the repository and install it via pip. You can perform a standard installation or an editable installation for development purposes.

    Standard Installation

    git clone git@github.com:A2Zadeh/CMU-MultimodalSDK.git
cd CMU-MultimodalSDK
pip install .

    Development Installation

    If you intend to modify the SDK, use the -e flag to install in editable mode:

    git clone git@github.com:A2Zadeh/CMU-MultimodalSDK.git
cd CMU-MultimodalSDK
pip install -e .

  9. Align computational sequences in mmdatasdk

    main

    Since different modalities often have different frequencies, you must align them to a common reference (e.g., labels or a specific modality).

    1. Add sequences: Use add_computational_sequences(dataset_dict, local_path) to fetch additional sequences (like labels) into your existing mmdataset object.
    2. Align: Use align(key) where key is the identifier for the reference sequence (e.g., 'Opinion Segment Labels').

    After alignment, data is indexed by video and segment, for example v0[2] refers to the third segment of video v0.

    from mmsdk import mmdatasdk

# 1. Initialize dataset
cmumosi_highlevel = mmdatasdk.mmdataset(mmdatasdk.cmu_mosi.highlevel, 'cmumosi/')

# 2. Add label sequences
cmumosi_highlevel.add_computational_sequences(mmdatasdk.cmu_mosi.labels, 'cmumosi/')

# 3. Align all sequences to the 'Opinion Segment Labels'
cmumosi_highlevel.align('Opinion Segment Labels')

  10. Access the CMU MOSEAS Western Europe Dataset

    main

    The CMU MOSEAS Western Europe Edition covers French, Spanish, Portuguese, and German. There are two primary methods for accessing this data:

    1. Using the SDK (Recommended): Use the mmsdk to process the data. This is the preferred method because multimodal processing can be computationally intensive. Note that computational sequences for this specific edition are being finalized and will be released soon.
    2. Downloading Raw Files: If you require the raw data, you must contact Amir Zadeh at abagherz@andrew.cmu.edu to request access.

    Warning: The raw dataset is approximately 640GB in size. Ensure you have sufficient storage capacity before attempting to download.

  11. Perform word-level alignment with collapse_functions

    main

    To perform word-level alignment, you can align modalities to a specific sequence (like glove_vectors) while simultaneously summarizing (collapsing) other modalities using custom functions.

    When using align(key, collapse_functions=[...]):

    • Each function in collapse_functions must accept two arguments: intervals (a $m \times 2$ matrix) and features (a $m \times n$ matrix).
    • The function must return a $1 \times n$ vector.
    • Multiple functions are applied sequentially, and their outputs are concatenated into the final result.
    import numpy
from mmsdk import mmdatasdk

# Define a collapse function (e.g., simple average)
def myavg(intervals, features):
    return numpy.average(features, axis=0)

# Initialize
cmumosi_highlevel = mmdatasdk.mmdataset(mmdatasdk.cmu_mosi.highlevel, 'cmumosi/')

# Align everything to glove_vectors, collapsing other modalities using myavg
cmumosi_highlevel.align('glove_vectors', collapse_functions=[myavg])

# Add labels and align to them
cmumosi_highlevel.add_computational_sequences(mmdatasdk.cmu_mosi.labels, 'cmumosi/')
cmumosi_highlevel.align('Opinion Segment Labels')

  12. Fetch and manage datasets with mmdatasdk

    main

    To work with datasets, import mmdatasdk from mmsdk. You can instantiate an mmdataset object by providing a dataset dictionary (e.g., mmdatasdk.cmu_mosi.highlevel) and a local directory for storage. This process automatically downloads the required computational sequences into the specified folder.

    Key Concepts:

    • Computational Sequences: These host information from a specific modality or view (e.g., word vectors, facial expressions, or acoustic features).
    • Standard Datasets: Located in mmdatasdk/dataset/standard_datasets.
    from mmsdk import mmdatasdk
# Fetch highlevel features for CMU-MOSI and store in 'cmumosi/' folder
cmumosi_highlevel = mmdatasdk.mmdataset(mmdatasdk.cmu_mosi.highlevel, 'cmumosi/')