AlphaFold

The AlphaFold DB metadata is available in BigQuery, allowing for complex SQL queries over 214M proteins. This is useful for creating custom subsets of the data based on criteria like species, confidence (pLDDT), or sequence information.

Metadata Table: bigquery-public-data.deepmind_alphafold.metadata

Warning: While Google Cloud offers a free tier (BigQuery Sandbox), repeated or large queries may incur costs if you have an upgraded billing account. Monitor your usage in the Google Cloud console.

AlphaFold Server job requests are defined in JSON files. A single JSON file can contain a list of multiple job descriptions (dictionaries). Each job dictionary must include a name, modelSeeds, sequences, dialect, and version.

To automate repetitive tasks, you can generate these files by running a job via the AlphaFold Server GUI and downloading the resulting zip file. The file <job_name>_job_request.json inside the zip serves as a perfect template.

Note: JSON comments are not supported.

{
  "name": "Test Fold Job Number One",
  "modelSeeds": [],
  "sequences": [...],
  "dialect": "alphafoldserver",
  "version": 1
}

To download all proteins for a specific species, you can download sharded tar files from Google Cloud Storage. Each shard contains at most 10,000 proteins.

1. Find the NCBI taxonomy ID ([TAX_ID]) for your target species.
2. Use gcloud storage cp to download the shards.
3. Un-tar the downloaded files and un-gzip the individual files within.

gcloud storage cp gs://public-datasets-deepmind-alphafold-v4/proteomes/proteome-tax_id-[TAX_ID]-*_v4.tar .

If you have a list of specific files (a manifest), you can download them using gcloud storage cp with the --read-paths-from-stdin flag. Note that this method is slower than downloading sharded tar files due to per-file overhead.

cat [manifest file] | gcloud storage cp --read-paths-from-stdin .

If you need to move the AlphaFold dataset between Google Cloud buckets or to another cloud service, you can use the Storage Transfer Service.

Note: Using this service may incur costs. Check the Google Cloud pricing page before initiating large transfers.

The full dataset is approximately 23 TiB. For large-scale processing (e.g., in an HPC environment), the recommended method is to download the sharded proteome tar files rather than individual files. This minimizes per-file latency.

Recommended Download Command

Use the gcloud storage cp command to recursively download the proteomes directory from the public bucket:

gcloud storage cp --recursive gs://public-datasets-deepmind-alphafold-v4/proteomes/ .

Post-Download Processing

1. Un-tar: You will need to un-tar the proteome files.
2. Un-gzip: Un-gzip the individual files within the tarballs.

Warning: After un-taring, the dataset expands to approximately 644 million files. Ensure your filesystem is configured to handle this high number of inodes/files.

The final relaxation step (Amber relaxation) can be time-consuming. Use the --models_to_relax flag to control which models are processed:

• all: All predicted models are relaxed.
• best: Only the most confident model (based on ranking confidence) is relaxed.
• none: No relaxation is performed. This may result in predictions with stereochemical violations but is useful for troubleshooting or saving time.

Use the run_alphafold.py script to perform protein structure prediction. The script processes FASTA files to generate protein models, optionally performing template searches and Amber relaxation.

Note on Multimers: If a FASTA file contains multiple sequences, it will be folded as a multimer. Ensure that model_preset is set to multimer when working with multi-sequence files.

# Example usage (conceptual)
python run_alphafold.py \
  --fasta_paths=target.fasta \
  --output_dir=./results \
  --data_dir=/path/to/params \
  --uniref90_database_path=/path/to/uniref90 \
  --mgnify_database_path=/path/to/mgnify \
  --template_mmcif_dir=/path/to/templates \
  --max_template_date=2023-01-01 \
  --obsolete_pdbs_path=/path/to/obsolete_pdbs.txt \
  --use_gpu_relax=True

Use this SQL query to calculate the mean of prediction confidence fractions (pLDDT) per species:

SELECT
 organismScientificName AS name,
 SUM(fractionPlddtVeryLow) / COUNT(fractionPlddtVeryLow) AS mean_plddt_very_low,
 SUM(fractionPlddtLow) / COUNT(fractionPlddtLow) AS mean_plddt_low,
 SUM(fractionPlddtConfident) / COUNT(fractionPlddtConfident) AS mean_plddt_confident,
 SUM(fractionPlddtVeryHigh) / COUNT(fractionPlddtVeryHigh) AS mean_plddt_very_high,
 COUNT(organismScientificName) AS num_predictions
FROM bigquery-public-data.deepmind_alphafold.metadata
GROUP by name
ORDER BY num_predictions DESC;

To download a custom subset of proteins (e.g., high-confidence Homo sapiens proteins) without downloading the entire dataset, use BigQuery to generate a list of Cloud Storage URIs. You can then pipe this list into gcloud storage cp.

with file_rows AS (
  with file_cols AS (
    SELECT
      CONCAT(entryID, '-model_v4.cif') as m,
      CONCAT(entryID, '-predicted_aligned_error_v4.json') as p
    FROM bigquery-public-data.deepmind_alphafold.metadata
    WHERE organismScientificName = "Homo sapiens"
      AND (fractionPlddtVeryHigh + fractionPlddtConfident) > 0.5
  )
  SELECT * FROM file_cols UNPIVOT (files for filetype in (m, p))
)
SELECT CONCAT('gs://public-datasets-deepmind-alphafold-v4/', files) as files
from file_rows

Use the dnaSequence entity type. Note that this refers to single stranded DNA. To model double stranded DNA, include a second dnaSequence entry containing the reverse complement strand.

Key Fields:

• sequence (string): DNA sequence (only A, T, G, C allowed).
• count (integer): Number of copies.
• modifications (optional list): DNA chemical modifications. Requires modificationType (CCD code) and basePosition (integer).
  • Allowed modificationType codes: CCD_5CM, CCD_C34, CCD_5HC, CCD_6OG, CCD_6MA, CCD_1CC, CCD_8OG, CCD_5FC, CCD_3DR.

{
  "dnaSequence": {
    "sequence": "GATTACA",
    "modifications": [
      {
        "modificationType": "CCD_6OG",
        "basePosition": 1
      }
    ],
    "count": 1
  }
}