TorchScale

You can install TorchScale via pip:

pip install torchscale

To develop locally, clone the repository and install in editable mode:

git clone https://github.com/microsoft/torchscale.git
cd torchscale
pip install -e .

To prepare the dataset for LongViT fine-tuning on TCGA survival prediction, follow these steps:

1. Organize WSIs: Download TCGA diagnostic whole slides from the NIH Genomic Data Commons Data Portal and organize them in a directory structure.
2. Download Annotations: Obtain the dataset annotation CSV and cross-validation splits from the MCAT repository.
3. Generate Index: Create index JSON files for each split using create_tcga_survival_index.py.
4. Resize Images: Resize whole slide images (WSIs) to your target size using convert_wsi_to_images.py.
5. Split Large Images (Optional): For very large images (e.g., 32,768x32,768), use split_to_small_images.py to parallelize training across multiple GPUs by splitting the sequence of patches along the sequence dimension.

To set up the environment for LongViT, install the required dependencies, a specific version of fairseq, and xformers using the following commands:

pip install -r requirements.txt
pip install git+https://github.com/shumingma/fairseq.git@moe
pip install -v -U git+https://github.com/facebookresearch/xformers.git@v0.0.20#egg=xformers

BERT pretraining requires a sharded data format managed by a streaming dataloader.

Directory Structure:

Data/
├── json/
│   ├── train.json
│   └── valid.json
├── shard/
│   ├── train/
│   │   ├── 00000.txt
│   │   └── ...
│   └── valid/
│       ├── 00000.txt
│       └── ...
├── dict.txt
└── sentencepiece.bpe.model

Shard File Format: Each shard should contain no more than 10K lines. One sentence per line, with an empty line separating documents.

JSON Metadata Format:

[
    {
        "source": [
            "shard/train/00000.txt",
            "shard/train/00001.txt"
        ],
        "source_lang": "en",
        "weight": 1.0
    }
]

Extracting dict.txt from a SentencePiece model:

spm_export_vocab --model=sentencepiece.bpe.model | sed 's/\t/ /g' | tail -n +4 > dict.txt

To use TorchScale with FairSeq, install the repository as an editable package and install the required dependencies including a specific version of FairSeq and Infinibatch.

# Install the repo as a package:
git clone https://github.com/microsoft/torchscale.git
cd torchscale
pip install -e .
pip install git+https://github.com/shumingma/fairseq.git@moe
pip install git+https://github.com/shumingma/infinibatch.git
pip install iopath
pip install numpy==1.23.0

Self-supervised pretraining can be performed on TCGA diagnostic slides using the DINO objective. For detailed instructions, refer to the get_started_for_tcga_pretraining.md guide.

Pretrained Model Weights: You can download the pretrained LongViT model (trained on TCGA diagnostic slides) here:

• LongViT

Model Configuration:

• #layer=12
• hidden=384
• FFN factor=4x
• #head=16
• patch=32x32

To prepare data for LongViT pretraining, you can generate 1,024x1,024 regions from Whole Slide Images (WSIs). Use the data_preprocessing/generate_1024_crops.py script, specifying the source WSI directory, the destination crops directory, and the number of regions to generate per image.

# we randomly generate 100 small regions for each whole slide image
python data_preprocessing/generate_1024_crops.py /path/to/your_WSIs /path/to/your_crops 100

To perform LongViT pretraining within the DINO framework, follow these steps:

1. Replace the standard vision_transformer.py in the DINO repository with the LongViT version located at ../pretraining/vision_transformer.py.
2. Modify the configuration to set the global crop size to 1024 and the local crop size to 512.

For faster training on Turing, Ampere, Ada, or Hopper GPUs, install Flash Attention:

pip install flash-attn

Alternatively, install xFormers based on your CUDA version:

CUDA 11.8:

pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu118

CUDA 12.1:

pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu121