Dolphin Video Interaction Platform

To extend Dolphin with a new video foundation model, follow these two steps:

1. Implement the model in modules/

Create a new Python package under the modules directory. Implement a class in the package's __init__.py file that includes an __init__ method and an inference function. For example, if adding face2video, create modules/face2video/__init__.py and define the FaceText2Video class.

2. Register the model in configs/backend.yaml

Add the model to the model_zoos section and define its corresponding tool in the tools section.

Example configuration for FaceText2Video:

model_zoos:
  FaceText2Video:  # foundation model class
    target: modules.face2video.FaceText2Video # path of the class in project
    params: # params passed to the class
      device: cuda:0
tools:
  - name: Generate Video Condition On Face Video
    desc: "useful when you want to generate a new video from both the user description and a facial keypoints video..."
    instance: FaceText2Video
    func: inference

model_zoos:
  FaceText2Video:
    target: modules.face2video.FaceText2Video
    params:
      device: cuda:0
tools:
  - name: Generate Video Condition On Face Video
    desc: "..."
    instance: FaceText2Video
    func: inference

To create a new model implementation in the face3d framework, you must subclass BaseModel and implement five core methods. This abstraction handles network initialization, data preprocessing, the forward pass, parameter optimization, and command-line option management.

Required methods to implement:

1. __init__(self, opt): Initialize the class. You must call BaseModel.__init__(self, opt) first. During initialization, you must define four lists:
   • self.loss_names (list of str): Training losses to plot/save.
   • self.model_names (list of str): Network names to display/save.
   • self.visual_names (list of str): Networks used for training visualization.
   • self.optimizers (list): Initialized optimizers (one per network, or grouped using itertools.chain).
2. set_input(self, input): Unpack data from the dataloader and apply preprocessing. input is a dictionary containing data and metadata.
3. forward(self): The core logic to produce intermediate results. This is called by both optimize_parameters and test.
4. optimize_parameters(self): Logic to calculate losses, compute gradients, and update network weights. This is called every training iteration.
5. modify_commandline_options(self, parser, is_train) (Optional): Add model-specific options to the provided parser. is_train (bool) indicates if the current phase is training or testing.

from face3d.models.base_model import BaseModel

class MyCustomModel(BaseModel):
    def __init__(self, opt):
        super().__init__(opt)  # Must call BaseModel.__init__
        self.loss_names = ['loss_a']
        self.model_names = ['net_G']
        self.visual_names = ['net_G']
        self.optimizers = [torch.optim.Adam(self.net_G.parameters())]

    def set_input(self, input):
        # Unpack data
        self.real_A = input['A']

    def forward(self):
        # Forward pass logic
        self.fake_B = self.net_G(self.real_A)
        self.loss_a = self.criterion(self.fake_B, self.real_B)

    def optimize_parameters(self):
        self.forward()
        self.loss_a.backward()
        self.optimizers[0].step()

    def modify_commandline_options(self, parser, is_train):
        parser.add_argument('--my_param', type=float, default=0.5)
        return parser

To add a custom model to the face3d.models package, follow these steps:

1. Create a new file named <model_name>_model.py in the modules/sadtalker/src/face3d/models/ directory.
2. Define a class named <ModelName>Model (e.g., DummyModel) that inherits from BaseModel.
3. Implement the following five required methods:
   • __init__(self, opt): Initialize the class. You must call BaseModel.__init__(self, opt) first. During initialization, you must define four specific attributes:
     • self.loss_names (list of str): Training losses to plot and save.
     • self.model_names (list of str): Names of the networks used in training.
     • self.visual_names (list of str): Images to display and save.
     • self.optimizers (list of optimizers): Initialized optimizers. You can use itertools.chain to group multiple networks into a single optimizer.
   • set_input(self): Unpack data from the dataset and apply preprocessing.
   • forward(self): Produce intermediate results.
   • optimize_parameters(self): Calculate loss, compute gradients, and update network weights.
   • modify_commandline_options(self, opt): (Optional) Add model-specific options and set default values.

Once implemented, you can use your model by passing the --model <model_name> flag to the training or testing scripts.