Process raw data for training
mainIf you want to use your own data or replicate the pipeline, follow these steps in the scripts/process directory.
Prerequisites
- Install requirements:
pip install -r requirements.txt. - Install external dependencies:
mmseqsandredis.
Step-by-Step Pipeline
- Preprocess CCD Dictionary: Use
ccd.pywith acomponents.ciffile to createccd.pkl. - Create Sequence Clusters: Use
cluster.pywith your sequences andmmseqsto assign proteins to similarity clusters. - Generate MSAs: Ensure MSA files are named using the SHA256 hash of the query sequence. For ColabFold-style MSAs, ensure Uniref sequences start with
>UniRef100_UNIREFID. - Process MSAs:
- Start a
redis-serverusing the providedtaxonomy.rdb. - Run
msa.pyto annotate sequences with taxonomy IDs.
- Start a
- Process Structures:
- Start a
redis-serverusingccd.rdb. - Run
rcsb.py(formmcifformat) using your clustering JSON to generate processed structural data.
- Start a
# 1. Setup environment
cd scripts/process
pip install -r requirements.txt
# 2. Preprocess CCD
python ccd.py --components components.cif --outdir ./ccd
# 3. Clustering
python cluster.py --ccd ccd.pkl --sequences pdb_seqres.txt --mmseqs PATH_TO_MMSEQS_EXECUTABLE --outdir ./clustering
# 4. MSA Processing
redis-server --dbfilename taxonomy.rdb --port 7777
python msa.py --msadir YOUR_MSA_DIR --outdir YOUR_OUTPUT_DIR --redis-port 7777
# 5. Structure Processing
redis-server --dbfilename ccd.rdb --port 7777
python rcsb.py --datadir PATH_TO_MMCIF_DIR --cluster clustering/clustering.json --outdir YOUR_OUTPUT_DIR --use-assembly --max-file-size 7000000 --redis-port 7777