DOCSEARCHbyAlgolia
Submit Docs

DeepGaze PyTorch Implementations

repository·main·Indexed 42 days ago

https://github.com/matthias-k/deepgaze

A collection of PyTorch implementations for saliency and scanpath models, including DeepGaze I, II, IIE, III, and the Multi-Scale Dataset Bias (MSDB) model. It provides tools for spatial saliency prediction, scanpath-aware predictions using fixation history, and a specialized adaptation workflow for MSDB to tune models to new datasets using CLIP and DINOv2 features.

Tokens
6.1K
Snippets
17
Records
19
Agent score
53%

What's inside DeepGaze

  1. Use DeepGaze MSDB (Spatial Saliency Model)

    main

    DeepGaze MSDB (Multi-Scale Dataset Bias) is a saliency model that combines CLIP and DINOv2 features with learned dataset-specific parameters. It can use dataset-specific parameters for known datasets or averaged parameters for generalization to new datasets.

    Important: You must provide the pixel_per_dva (pixels per degree of visual angle) for your specific display setup (viewing distance and screen resolution). For example, the MIT1003 dataset was collected at 35 pixels per degree.

    To use the model, you need to provide an image tensor and a center-bias log density tensor. You can use a precomputed center-bias (e.g., from MIT1003) or a uniform center-bias using np.zeros.

    import torch
import deepgaze_pytorch
from deepgaze_pytorch import MSDBDataset

DEVICE = 'cuda'
model = deepgaze_pytorch.DeepGazeMSDB(pretrained=True).to(DEVICE)

# ... prepare image_tensor and centerbias_tensor ...

# For a known dataset (e.g., MIT1003):
log_density_prediction = model(image_tensor, centerbias_tensor, pixel_per_dva=35.0, dataset=MSDBDataset.MIT1003)

# For a new/unknown dataset (generalization mode):
log_density_prediction = model(image_tensor, centerbias_tensor, pixel_per_dva=35.0, dataset=None)

  2. Adapt DeepGaze MSDB to a new dataset

    main

    You can adapt DeepGaze MSDB to a new dataset by training 13 dataset-specific scalar parameters while keeping the CLIP+DINOv2 backbone and the saliency network frozen. The new dataset slot is initialized from the average of the existing datasets, allowing for quick specialization with minimal data.

    Workflow:

    1. Load stimuli and fixations (using load_fixations_csv or your own pysaliency objects).
    2. Fit a center bias over your fixations using fit_centerbias.
    3. Call adapt_dataset_parameters to train the new slot.
    4. Use the returned new_dataset_index to perform predictions on your dataset.

    Saving/Loading Weights: To save only the adapted head weights, use model.head_state_dict(). To reload them, you must first build a DeepGazeMSDB instance, call model.add_dataset() to ensure parameter shapes match, and then use model.load_state_dict(..., strict=False).

    from deepgaze_pytorch import DeepGazeMSDB
from deepgaze_pytorch.custom_data import load_fixations_csv, fit_centerbias
from deepgaze_pytorch.msdb_adaptation import adapt_dataset_parameters

# 1. Load data
stimuli, fixations = load_fixations_csv('images/', 'fixations.csv')

# 2. Fit center bias
centerbias = fit_centerbias(stimuli, fixations)

# 3. Adapt
model = DeepGazeMSDB(pretrained=True)
model, new_dataset_index = adapt_dataset_parameters(
    model, stimuli, fixations, centerbias,
    pixel_per_dva=21.75,
    train_directory='adaptation_run',
)

# 4. Use
log_density = model(image_tensor, centerbias_tensor, pixel_per_dva=21.75, dataset=new_dataset_index)

# Save head-only weights
import torch
torch.save(model.head_state_dict(), 'deepgazemsdb_mydataset.pth')

  3. How DeepGaze MSDB adaptation works

    main

    DeepGaze MSDB uses a frozen CLIP+DINOv2 backbone and a shared saliency network, but maintains a small set of per-dataset parameters (13 scalars) including multi-scale weights, a gaussian blur sigma, a center-bias weight, and a priority scaling.

    To adapt the model to a new dataset, you add a new parameter slot and train only these 13 scalars. The new slot is initialized from the average of the model's original datasets (the dataset=None state), meaning it starts from a generalized behavior and requires minimal data to specialize. Adapting to a new dataset does not modify the predictions for the original built-in datasets.

  4. Save and reload adapted DeepGaze MSDB weights

    main

    To save only the adapted head (the 13 parameters) rather than the full model, use model.head_state_dict().

    To reload these weights into a new model instance:

    1. Initialize a DeepGazeMSDB model.
    2. Call model.add_dataset() to create a slot for the new data.
    3. Load the weights using model.load_state_dict() with strict=False.
    # Saving
torch.save(model.head_state_dict(), 'adapted.pth')

# Reloading
model = DeepGazeMSDB(pretrained=True)
model.add_dataset()
model.load_state_dict(torch.load('adapted.pth'), strict=False)

  5. Workflow: Training DeepGaze III

    main

    Training DeepGaze III follows a multi-stage curriculum to achieve high performance:

    1. Pretraining (Spatial): Train the saliency_network on a large dataset like SALICON. The scanpath_network is None during this stage.
    2. Finetuning (Spatial): Finetune the spatial model on a specific target dataset (e.g., MIT1003) using optimized center-bias parameters.
    3. Scanpath Training (Partially Frozen):
      • Initialize the scanpath_network and fixation_selection_network.
      • Freeze early layers of the saliency_network (e.g., saliency_network.layernorm0, saliency_network.conv0, etc.) to stabilize training.
      • Train the temporal components.
    4. Full Finetuning: Unfreeze all parameters and perform a final training pass with a very low learning rate (e.g., 1e-5) to fine-tune the entire system.

  6. Use DeepGaze III (Scanpath Model)

    main

    DeepGaze III is a scanpath model, meaning its predictions depend on the history of previous fixations. When calling the model, you must provide the history of previous fixation coordinates as tensors. The model uses an internal attribute included_fixations to determine which parts of the history to use.

    import torch
import deepgaze_pytorch

DEVICE = 'cuda'
model = deepgaze_pytorch.DeepGazeIII(pretrained=True).to(DEVICE)

# ... prepare image_tensor and centerbias_tensor ...

# Fixation history in pixel coordinates
fixation_history_x = np.array([1024//2, 300, 500, 200, 200, 700])
fixation_history_y = np.array([768//2, 300, 100, 300, 100, 500])

x_hist_tensor = torch.tensor([fixation_history_x[model.included_fixations]]).to(DEVICE)
y_hist_tensor = torch.tensor([fixation_history_y[model.included_fixations]]).to(DEVICE)

log_density_prediction = model(image_tensor, centerbias_tensor, x_hist_tensor, y_hist_tensor)

  7. Use DeepGaze IIE (Spatial Saliency Model)

    main

    DeepGaze IIE is a spatial saliency model. It requires an image tensor and a center-bias log density tensor. Unlike MSDB, it does not take a dataset or pixel_per_dva argument in its call.

    import torch
import deepgaze_pytorch

DEVICE = 'cuda'
model = deepgaze_pytorch.DeepGazeIIE(pretrained=True).to(DEVICE)

# ... prepare image_tensor and centerbias_tensor ...

log_density_prediction = model(image_tensor, centerbias_tensor)

  8. Reference: MSDBDataset available datasets

    main

    The MSDBDataset enum provides identifiers for the built-in datasets supported by DeepGaze MSDB for specialized parameter usage.

    MSDBDataset.MIT1003
MSDBDataset.CAT2000
MSDBDataset.COCO_FREEVIEW
MSDBDataset.DAEMONS
MSDBDataset.FIGRIM

  9. Use DeepGaze IIE for saliency prediction

    main

    To perform saliency prediction with DeepGaze IIE, initialize the model using deepgaze_pytorch.DeepGazeIIE(pretrained=True). The model requires an image tensor and a center-bias log density tensor.

    Center-bias preparation:

    1. Load a precomputed center-bias (e.g., from MIT1003) or use a uniform template via np.zeros((1024, 1024)).
    2. Rescale the center-bias to match the input image dimensions using scipy.ndimage.zoom.
    3. Renormalize the log density by subtracting scipy.special.logsumexp(centerbias).
    4. Convert both the image and the center-bias to PyTorch tensors with shape [Batch, Channels, Height, Width].
    import numpy as np
import torch
from scipy.ndimage import zoom
from scipy.special import logsumexp
import deepgaze_pytorch

DEVICE = 'cuda'
model = deepgaze_pytorch.DeepGazeIIE(pretrained=True).to(DEVICE)
image = ... # Load your image

# Prepare centerbias
centerbias_template = np.load('centerbias_mit1003.npy')
centerbias = zoom(centerbias_template, (image.shape[0]/centerbias_template.shape[0], image.shape[1]/centerbias_template.shape[1]), order=0, mode='nearest')
centerbias -= logsumexp(centerbias)

# Prepare tensors
image_tensor = torch.tensor([image.transpose(2, 0, 1)]).to(DEVICE)
centerbias_tensor = torch.tensor([centerbias]).to(DEVICE)

# Predict
log_density_prediction = model(image_tensor, centerbias_tensor)

  10. Use DeepGaze III for scanpath-aware saliency prediction

    main

    DeepGaze III extends saliency prediction by incorporating previous fixation history. When calling the model, you must provide the current image tensor, the center-bias tensor, and tensors representing the history of previous fixations in $x$ and $y$ coordinates.

    Handling Fixation History:

    • The model uses an attribute model.included_fixations to determine which indices from your provided history should be used in the prediction.
    • Fixation coordinates should be provided as tensors of shape [Batch, History_Length].
    • To generate a scanpath, you can iteratively sample the next fixation from the predicted log density using pysaliency.models.sample_from_logdensity and append it to your history.
    import torch
import deepgaze_pytorch

DEVICE = 'cuda'
model = deepgaze_pytorch.DeepGazeIII(pretrained=True).to(DEVICE)

# Fixation history (pixel coordinates)
fixation_history_x = np.array([512, 300, 500])
fixation_history_y = np.array([384, 300, 100])

# Prepare tensors
image_tensor = torch.tensor([image.transpose(2, 0, 1)]).to(DEVICE)
centerbias_tensor = torch.tensor([centerbias]).to(DEVICE)

# Use model.included_fixations to filter the history
x_hist_tensor = torch.tensor([fixation_history_x[model.included_fixations]]).to(DEVICE)
y_hist_tensor = torch.tensor([fixation_history_y[model.included_fixations]]).to(DEVICE)

# Predict
log_density_prediction = model(image_tensor, centerbias_tensor, x_hist_tensor, y_hist_tensor)

  11. Sample fixations from predicted log density

    main

    To simulate a scanpath, you can sample new fixation coordinates from the model's output log density.

    1. Obtain the log density prediction from the model.
    2. Convert the prediction to a NumPy array: log_density_prediction.detach().cpu().numpy()[0, 0].
    3. Use pysaliency.models.sample_from_logdensity(logD, rst=random_state) to get the next (x, y) coordinates.
    4. Append these coordinates to your history and repeat the prediction loop.
    from pysaliency.models import sample_from_logdensity
import numpy as np

# ... inside a loop ...
# logD is the numpy array from model output
# rst is a numpy.random.RandomState instance
next_x, next_y = sample_from_logdensity(logD, rst=rst)

# Add to history for next iteration
fixations_x.append(next_x)
fixations_y.append(next_y)

  12. Prepare scanpath training datasets

    main

    Use prepare_scanpath_dataset to create a PyTorch DataLoader for training the temporal scanpath model. This uses FixationDataset.

    Key arguments for FixationDataset:

    • included_fixations: Specifies which previous fixations to include for conditioning (e.g., [-1, -2, -3, -4]).
    • allow_missing_fixations: Boolean to allow sequences with fewer than the requested history.
    • transform: Uses FixationMaskTransform(sparse=False) for mask generation.
    • lmdb_path: Path for caching data in LMDB format.
    def prepare_scanpath_dataset(stimuli, fixations, centerbias, batch_size, path=None):
    if path is not None:
        path.mkdir(parents=True, exist_ok=True)
        lmdb_path = str(path)
    else:
        lmdb_path = None

    dataset = FixationDataset(
        stimuli=stimuli,
        fixations=fixations,
        centerbias_model=centerbias,
        included_fixations=[-1, -2, -3, -4],
        allow_missing_fixations=True,
        transform=FixationMaskTransform(sparse=False),
        average='image',
        lmdb_path=lmdb_path,
    )

    loader = torch.utils.data.DataLoader(
        dataset,
        batch_sampler=ImageDatasetSampler(dataset, batch_size=batch_size),
        pin_memory=False,
        num_workers=0,
    )

    return loader