DOCSEARCHbyAlgolia
Submit Docs

DeepVariant Documentation

repository·r1.10·Indexed 26 days ago

https://github.com/google/deepvariant

DeepVariant is a deep learning-based variant caller that uses convolutional neural networks to classify pileup image tensors from aligned reads (BAM/CRAM). It supports germline variant calling in diploid organisms across Illumina, PacBio, and Oxford Nanopore technologies. The documentation covers installation via Docker, source build, or prebuilt binaries, as well as the DeepTrio extension for calling variants in trios or duos. It includes guides for model training, gVCF support, and deployment on Google Cloud Platform.

Tokens
47.5K
Snippets
87
Records
166
Agent score
87%

What's inside DeepVariant

  1. Overview of DeepTrio

    r1.10

    DeepTrio is a variant caller built on top of DeepVariant specifically designed for calling variants in trios or duos. It uses a neural network to consider genetic inheritance and generates variant candidates from all samples simultaneously to ensure consistent genotype calls across the trio.

    DeepTrio follows a three-stage pipeline:

    1. make_examples (specialized for DeepTrio)
    2. call_variants (shared with DeepVariant)
    3. postprocess_variants (shared with DeepVariant)

    Available models include:

    • Illumina whole genome data (WGS)
    • Illumina whole exome data (WES)
    • PacBio HiFi whole genome data (PacBio WGS)

  2. DeepTrio Performance Benchmarks (WGS, WES, PacBio)

    r1.10

    The following benchmarks represent the average runtime and accuracy for DeepTrio models.

    Average Total Runtime

    • WGS: ~4h 37m
    • PacBio: ~6h 14m
    • WES: ~14m

    Accuracy (on chr20)

    Accuracy is measured using Recall, Precision, and F1 Score for both SNPs and INDELs across WGS, WES, and PacBio models. Detailed metrics for specific samples (HG002, HG003, HG004) can be found in the official case study reports.

  3. Evaluate DeepVariant results using `hap.py`

    r1.10

    After running DeepVariant, use hap.py to benchmark the resulting VCF against a truth set (e.g., Genome in a Bottle benchmarks). This process compares the predicted variants to known truth variants in specified regions.

    sudo docker run \
  -v "${PWD}/benchmark":"/benchmark" \
  -v "${PWD}/input":"/input" \
  -v "${PWD}/output":"/output" \
  -v "${PWD}/reference":"/reference" \
  -v "${PWD}/happy:/happy" \
  jmcdani20/hap.py:v0.3.12 /opt/hap.py/bin/hap.py \
    /benchmark/HG004_GRCh38_1_22_v4.2.1_benchmark.vcf.gz \
    /output/HG004.output.vcf.gz \
    -f /benchmark/HG004_GRCh38_1_22_v4.2.1_benchmark_noinconsistent.bed \
    -r /reference/GRCh38_no_alt_analysis_set.fasta \
    -o /happy/happy.output \
    --engine=vcfeval \
    --pass-only \
    -l chr20 \
    --target-regions=/input/HG004.giab_na24143.hifi_reads.lima.0--0.lima.IsoSeqX_bc01_5p--IsoSeqX_3p.refined.grch38.mm2.splitN.fc.depth.10x.exons.bed \
    --threads=$(nproc)

  4. Download reference, input, and benchmark data for the G400 case study

    r1.10

    To replicate the Complete Genomics G400 case study, you must download the GRCh38 reference genome, the HG002 chr20 BAM files, Genome in a Bottle (GIAB) benchmarks, and the specific Complete Genomics G400 model weights.

    1. Download Reference (GRCh38)

    mkdir -p reference

FTPDIR=ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/001/405/GCA_000001405.15_GRCh38/seqs_for_alignment_pipelines.ucsc_ids

curl ${FTPDIR}/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.gz | gunzip > reference/GRCh38_no_alt_analysis_set.fasta
curl ${FTPDIR}/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.fai > reference/GRCh38_no_alt_analysis_set.fasta.fai

    2. Download HG002 chr20 BAM

    mkdir -p input

HTTPDIR=https://storage.googleapis.com/deepvariant/complete-case-study-testdata

curl ${HTTPDIR}/HG002.complete_g400.V350151728.grch38.chr20.bam > input/HG002.complete_g400.V350151728.grch38.chr20.bam

curl ${HTTPDIR}/HG002.complete_g400.V350151728.grch38.chr20.bam.bai > input/HG002.complete_g400.V350151728.grch38.chr20.bam.bai

    3. Download GIAB Benchmarks for HG002

    mkdir -p benchmark

FTPDIR=ftp://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/AshkenazimTrio/HG002_NA24385_son/NISTv4.2.1/GRCh38

curl ${FTPDIR}/HG002_GRCh38_1_22_v4.2.1_benchmark_noinconsistent.bed > benchmark/HG002_GRCh38_1_22_v4.2.1_benchmark_noinconsistent.bed
curl ${FTPDIR}/HG002_GRCh38_1_22_v4.2.1_benchmark.vcf.gz > benchmark/HG002_GRCh38_1_22_v4.2.1_benchmark.vcf.gz
curl ${FTPDIR}/HG002_GRCh38_1_22_v4.2.1_benchmark.vcf.gz.tbi > benchmark/HG002_GRCh38_1_22_v4.2.1_benchmark.vcf.gz.tbi

    4. Download Complete Genomics G400 model

    HTTPDIR=https://storage.googleapis.com/deepvariant/complete-case-study-testdata

curl ${HTTPDIR}/complete-g400/weights-60-0.993753.ckpt.data-00000-of-00001 > input/weights-60-0.993753.ckpt.data-00000-of-00001

curl ${HTTPDIR}/complete-g400/weights-60-0.993753.ckpt.index > input/weights-60-0.993753.ckpt.index

curl ${HTTPDIR}/complete-g400/example_info.json > input/example_info.json

curl ${HTTPDIR}/complete-g400/model.example_info.json > input/model.example_info.json

  5. Benchmark DeepVariant VCFs using hap.py

    r1.10

    After generating variant calls, you can benchmark them against Genome in a Bottle (GIAB) benchmarks using hap.py via Docker. This process evaluates the accuracy of your calls (Precision, Recall, F1 Score, etc.) against a truth set.

    Required inputs for hap.py:

    • Truth VCF: The benchmark VCF file.
    • Query VCF: The VCF file produced by DeepVariant.
    • -f: A BED file defining the confident regions in the truth set.
    • -T: The capture target BED file used during DeepVariant execution.
    • -r: The reference FASTA file.
    • --engine=vcfeval: Specifies the evaluation engine.
    • --pass-only: Filters for variants that passed quality filters.
    mkdir -p happy

sudo docker pull jmcdani20/hap.py:v0.3.12

sudo docker run \
  -v "${PWD}/benchmark":"/benchmark" \
  -v "${PWD}/input":"/input" \
  -v "${PWD}/output":"/output" \
  -v "${PWD}/reference":"/reference" \
  -v "${PWD}/happy:/happy" \
  jmcdani20/hap.py:v0.3.12 /opt/hap.py/bin/hap.py \
  /benchmark/HG003_GRCh38_1_22_v4.2.1_benchmark.vcf.gz \
  /output/HG003.output.vcf.gz \
  -f /benchmark/HG003_GRCh38_1_22_v4.2.1_benchmark_noinconsistent.bed \
  -T /input/idt_capture_novogene.grch38.bed \
  -r /reference/GRCh38_no_alt_analysis_set.fasta \
  -o /happy/happy.output \
  --engine=vcfeval \
  --pass-only

  6. Enable runtime profiling with make_examples

    r1.10

    If you are running make_examples directly instead of using the run_deepvariant wrapper, use the --runtime_by_region flag followed by a filename (ending in .tsv) to specify where the profiling data should be saved.

    Note on Sharding: If you are using sharded examples (e.g., examples@64), you must provide a sharded runtime profile file with the same number of shards (e.g., runtimes@64.tsv).

  7. Perform analysis with hap.py against truth sets

    r1.10

    Use hap.py to benchmark your VCF output against a truth set (e.g., Genome in a Bottle benchmarks). This process uses the vcfeval engine to compare variants.

    Key arguments:

    • --engine=vcfeval: Specifies the comparison engine.
    • --pass-only: Filters for pass-only records.
    • -l <chr>: Limits the analysis to a specific chromosome (e.g., chr20).
    • -f <bed>: Provides a BED file for filtering/regions.
    • -r <fasta>: The reference genome FASTA file.
    sudo docker run \
  -v "${PWD}/benchmark":"/benchmark" \
  -v "${PWD}/input":"/input" \
  -v "${PWD}/output":"/output" \
  -v "${PWD}/reference":"/reference" \
  -v "${PWD}/happy:/happy" \
  jmcdani20/hap.py:v0.3.12 /opt/hap.py/bin/hap.py \
  /benchmark/HG002_GRCh38_1_22_v4.2.1_benchmark.vcf.gz \
  /output/HG002.output.vcf.gz \
  -f /benchmark/HG002_GRCh38_1_22_v4.2.1_benchmark_noinconsistent.bed \
  -r /reference/GRCh38_no_alt_analysis_set.fasta \
  -o /happy/HG002.output \
  --engine=vcfeval \
  --pass-only \
  -l chr20

  9. Benchmark DeepVariant outputs using hap.py

    r1.10

    You can evaluate the quality of DeepVariant variant calls against a truth set (such as Genome in a Bottle benchmarks) using hap.py. This case study uses the vcfeval engine to compare a query VCF against a truth VCF.

    Required inputs for hap.py:

    • Truth VCF file.
    • Query VCF file (produced by DeepVariant).
    • Truth BED file (to filter regions).
    • Reference genome FASTA.
    • Output directory.

    Common flags:

    • --engine=vcfeval: Specifies the evaluation engine.
    • --pass-only: Filters for pass-only variants.
    • -l: Limits benchmarking to a specific region.
    sudo docker run \
  -v "${INPUT_DIR}":"${INPUT_DIR}" \
  -v "${OUTPUT_DIR}":"${OUTPUT_DIR}" \
  -v "${PWD}/happy:/happy" \
  jmcdani20/hap.py:v0.3.12 /opt/hap.py/bin/hap.py \
  "${INPUT_DIR}/${TRUTH_VCF}" \
  "${OUTPUT_DIR}/${OUTPUT_VCF}" \
  -f "${INPUT_DIR}/${TRUTH_BED}" \
  -r "${INPUT_DIR}/${REF}" \
  -o "${OUTPUT_DIR}/hg003.ul.r104.ont.chr20.happy.output" \
  --engine=vcfeval \
  --pass-only \
  -l "${REGION}"

  10. Run DeepVariant with GPU acceleration

    r1.10

    To use GPU acceleration, pull the -gpu tagged image and use the --gpus 1 flag in Docker. Note that only the call_variants step utilizes the GPU; make_examples and postprocess_variants run on the CPU.

    sudo docker run --gpus 1 \
  -v "${INPUT_DIR}":"/input" \
  -v "${OUTPUT_DIR}:/output" \
  google/deepvariant:"${BIN_VERSION}-gpu" \
  /opt/deepvariant/bin/run_deepvariant \
  ...

  11. Run DeepVariant for Whole Exome Sequencing (WES) on a CPU-only machine

    r1.10

    To process Whole Exome Sequencing (WES) data using a single machine with a CPU, use the google/deepvariant Docker image with the --model_type WES flag. This flag selects a model optimized for Illumina WES data.

    Key flags:

    • --model_type WES: Specifies the WES-optimized model.
    • --ref: Path to the reference FASTA file.
    • --reads: Path to the input BAM file.
    • --regions: Path to the capture target BED file (e.g., exome capture targets).
    • --output_vcf: Path for the resulting VCF file.
    • --output_gvcf: Path for the resulting gVCF file.
    • --num_shards: Number of shards to use (typically set to the number of available CPUs using $(nproc)).
    • --intermediate_results_dir: (Optional) Directory to store intermediate outputs like make_examples.tfrecord and call_variants.tfrecord.
    • --dry_run=true: (Optional) Use this to see the required flags for individual steps without executing them.
    mkdir -p output
mkdir -p output/intermediate_results_dir

BIN_VERSION="1.10.0"

sudo docker run \
  -v "${PWD}/input":"/input" \
  -v "${PWD}/output":"/output" \
  -v "${PWD}/reference":"/reference" \
  google/deepvariant:"${BIN_VERSION}" \
  /opt/deepvariant/bin/run_deepvariant \
  --model_type WES \
  --ref /reference/GRCh38_no_alt_analysis_set.fasta \
  --reads /input/HG003.novaseq.wes_idt.100x.dedup.bam \
  --regions /input/idt_capture_novogene.grch38.bed \
  --output_vcf /output/HG003.output.vcf.gz \
  --output_gvcf /output/HG003.output.g.vcf.gz \
  --num_shards $(nproc) \
  --intermediate_results_dir /output/intermediate_results_dir

  12. Evaluate DeepTrio VCF results using hap.py

    r1.10

    To evaluate the VCF files produced by DeepTrio, use the hap.py program via Docker. This process compares the DeepTrio output VCF for each sample against the corresponding truth set (e.g., from GIAB).

    To run the evaluation, you must mount your input directory (containing truth VCFs, BED files, and FASTA references) and an output directory to the Docker container. The command uses the vcfeval engine and can be restricted to specific genomic regions using the -l flag.

    # Pull docker image locally
sudo docker pull jmcdani20/hap.py:v0.3.12

# Evaluate HG002
sudo docker run -it \
  -v "${INPUT_DIR}":"/input" \
  -v "${OUTPUT_DIR}:/output" \
  jmcdani20/hap.py:v0.3.12 /opt/hap.py/bin/hap.py \
  /input/HG002_GRCh38_1_22_v4.2.1_benchmark.vcf.gz \
  /output/HG002.output.vcf.gz \
  -f "/input/HG002_GRCh38_1_22_v4.2.1_benchmark_noinconsistent.bed" \
  -r "/input/GRCh38_no_alt_analysis_set.fasta" \
  -o "/output/HG002.happy" \
  --engine=vcfeval \
  --pass-only \
  -l chr20:10000000-10010000