DOCSEARCHbyAlgolia
Submit Docs

basenji

repository·master·Indexed 19 days ago

https://github.com/calico/basenji

A deep learning framework for predicting genomic features from DNA sequences. It includes implementations and tools for Akita (3D genome folding/Hi-C contact maps), Basset-style peak prediction, Saluki (mRNA degradation rates), and a modified version of Enformer components. The repository provides scripts for data preparation (akita_data.py, akita_data_read.py), model training (basenji_train.py, akita_train.py), and utilities to download pre-trained TensorFlow models and TFRecord datasets.

Tokens
51.2K
Snippets
133
Records
186
Agent score
64%

What's inside basenji

  1. Access Enformer components in Basenji

    master

    While the official Enformer implementation is maintained by Google DeepMind, this repository includes a modified version of the Enformer components. The implementation in this repository uses a parameters JSON file that approximates the original Enformer architecture but includes several specific modifications:

    • Positional Encodings: Uses central mask only.
    • Pooling: Attention pooling is replaced with max pooling.
    • Optimization: Settings are tuned for smaller batch sizes.
    • Regularization: L2 weight decay is applied throughout.
    • Architecture: Filter counts may differ from the original implementation.

  2. Interpret Saluki *in silico* mutagenesis (ISM) scores

    master
    ISM scores in this project are mean-centered as a normalization for the four nucleotides at each position, ensuring they sum to zero. To determine the importance of a specific genomic position, use the normalized ISM score of the reference nucleotide.

  5. Download the Akita manuscript model and data

    master

    To replicate the results from the 'Sequential regulatory activity predictions with deep convolutional neural networks' (Genome Research 3/2018) paper, use the provided shell scripts to download the necessary assets:

    • Use get_model.sh to download the saved TensorFlow model.
    • Use get_data.sh to download the training/testing dataset. Note: The dataset is approximately 125 Gb.
    ./get_model.sh
./get_data.sh

  6. Access Scikit-learn SNP classifiers

    master

    Scikit-learn random forest SNP classifiers for Mendelian disease and GWAS complex traits are available at https://console.cloud.google.com/storage/browser/basenji_barnyard/sad/classifiers/.

    To use these models in Python, restore them using joblib.load.

    import joblib
model = joblib.load('path_to_classifier.joblib')

  8. Train new Akita models

    master

    Training new Akita models follows the standard Basenji syntax but utilizes specific scripts: akita_data.py for data preparation and akita_train.py for training.

    Dependencies

    Beyond the standard Basenji dependencies, you must install the following via pip:

    • astropy
    • cooler
    • cooltools
    • intervaltree

    A full workflow for preparing training data and training a model is provided in tutorial.ipynb.

  9. Download Akita manuscript model and data

    master

    To use the pre-trained Akita model (trained on human Hi-C and Micro-C data at 2048bp resolution mapped to hg38), use the following scripts provided in the repository:

    • get_model.sh: Downloads the saved TensorFlow model.
    • get_data.sh: Downloads the training, validation, and test TFRecords (~10 Gb) used to train and assess the model.
    ./get_model.sh
./get_data.sh

  10. Manage SLURM job execution with basenji_sat_bed_multi.py

    master

    The basenji_sat_bed_multi.py tool is optimized for SLURM environments. It manages job submission, concurrency, and environment activation.

    Key Execution Controls

    • Environment Management: Use -e <conda_env> to specify which Anaconda environment the worker processes should activate (e.g., -e tf2.6).
    • Concurrency: Use -p <int> to define the total number of worker processes and --max_proc <int> to limit how many run concurrently.
    • Queue Selection: Use -q <queue> to specify the SLURM partition/queue (e.g., -q geforce).
    • Job Naming: Use -n <name> to set a prefix for the SLURM job names.
    • Resuming Work: If a run was interrupted, use the -r or --restart flag. The script will check for existing scores.h5 files in the job subdirectories to determine which processes need to be re-run.

  11. Access Saluki models and datasets

    master

    The models, training, validation, and test TFRecords files associated with the Saluki research (The genetic and biochemical determinants of mRNA degradation rates in mammals) are available via Zenodo. You can find these files in the datasets/deeplearning/train_gru subdirectory of the repository or via the DOI link provided in the manuscript documentation.

    https://doi.org/10.5281/zenodo.6326409

  12. Train a model with basenji_train.py

    master

    Use the basenji_train.py script to train a convolutional neural network for sequential predictions. The script requires a configuration parameters file and an HDF5 data file containing training and validation data.

    Arguments:

    • params_file (Text table): Model configuration parameters.
    • data_file (HDF5): Input training and validation data.

    Hardware Recommendation: It is highly recommended to train on a GPU for reasonable performance. TensorFlow will prioritize gpu:0 over cpu:0 if a supported GPU is available.

    python basenji_train.py <params_file> <data_file>