EmotiEffLib

The primary method for training EmotiEffNet models (such as EmotiEffNet-B2) and classifying emotional features for the AFEW and VGAF datasets is via the affectnet/train_emotions-pytorch.ipynb notebook.

Prerequisite: Before running the main training notebook, you must run the preprocessing notebooks for the specific datasets you are using:

• affectnet/train_emotions.ipynb
• AFEW_train.ipynb
• VGAF_train.ipynb

EmotiEffLib provides specific tutorials for both Python and C++ interfaces. You can find step-by-step guides in the following directories:

• Python tutorials: tutorials/python
• C++ tutorials: tutorials/cpp

To run the Python tutorials and examples provided in the repository, you must install the necessary dependencies using pip from the requirements.txt file located in the examples directory.

pip install -r requirements.txt

If you are working with the local repository, you can install the library directly from the source code using pip install . with optional dependency extras:

• Standard (ONNX support): pip install .
• With Torch support: pip install .[torch]
• With Engagement prediction (requires TensorFlow): pip install .[engagement]
• Full installation: pip install .[all] or pip install .[torch,engagement]

# Install EmotiEffLib with ONNX and Torch support from local source
pip install .[torch]

To create personalized models for engagement detection, use the Jupyter notebooks provided in the training_and_examples/personalized_models/ directory. The process follows a pipeline of feature extraction, universal model training, custom dataset processing, and user adaptation:

1. Extract Visual Features: Use 01 - Features-extractor.ipynb to extract visual features from video files.
2. Train Universal Model: Use 02 - Engagement user-independent model training.ipynb to train a universal engagement recognition model that is independent of specific users.
3. Process Custom Datasets: Use 03 - Custom dataset features extractor.ipynb to extract visual features specifically from your personalized dataset.
4. Adapt to Users: Use 04 - Engagement user-adaptation.ipynb to create models that are specifically adapted to individual users based on the extracted features.

To run the EmotiEffLib mobile application, you must convert the existing models to mobile-friendly formats. Navigate to the mobile_app directory and execute the conversion scripts for TFLite and PyTorch Lite.

Note: Ensure you have the necessary model files available before running these scripts.

cd mobile_app
python to_tflite.py
python to_pytorchlite.py

If you are using the EmotiEffCppLib (C++ interface), you must first prepare the models for inference using the provided Python script:

python models/prepare_models_for_emotieffcpplib.py

To use EmotiEffCppLib as a dependency in your own C++ project, add the repository as a submodule and configure your CMakeLists.txt to include the library and its headers. You must specify the paths to your chosen inference engine (Libtorch or ONNX Runtime).

# At least one of these variables should be specified
set(WITH_TORCH "/path/to/libtorch")
set(WITH_ONNX "/path/to/ONNXRuntime")

add_subdirectory("${PROJECT_SOURCE_DIR}/path/to/emotiefflib/emotieffcpplib")

include_directories(
    "${PROJECT_SOURCE_DIR}/path/to/emotiefflib/emotieffcpplib/include"
)

target_link_libraries(your_project PRIVATE emotiefflib)