website·Indexed 29 days ago
https://parsomics.orgDocumentation for Parsomics, a platform for processing protein annotation and prokaryote genome data. Includes guides on using the parsomics-cli, database management via SQL and REST API, and a comprehensive developer guide for creating and publishing plugins to support various formats like GTDB-Tk, dRep, Prokka, and InterProScan.
parsomics project provides a suite of data management tools specifically for metagenomics research. Its primary function is data integration: it processes output files from various metagenomics tools and converts them into a standardized relational database representation.parsomics project consists of a set of core modules and specialized plugin modules. The core functionality is provided by parsomics-core, parsomics-api-server, and parsomics-cli. Extendability is achieved through plugin modules, which include parsomics-plugin-clean, parsomics-plugin-dbcan, parsomics-plugin-interpro, and parsomics-plugin-proteinfer.overview.txt format from run_dbCAN. It transforms multiple annotation columns (HMMER, dbCAN_sub, DIAMOND) into individual rows. Additionally, it extracts start and stop coordinates from descriptions (e.g., "CE4(38-162)") into separate coord_start and coord_stop fields.NOT NULL and UNIQUE. This schema acts as the blueprint for how data is stored and related within the system.drepentry table stores information about genomes and their clusters as assigned by dRep, including whether a genome is the representative 'winner' for its cluster. It maintains a relationship with the drepdirectory table.CREATE TABLE drepentry (
genome_name VARCHAR NOT NULL,
genome_cluster_name VARCHAR NOT NULL,
is_winner BOOLEAN NOT NULL,
directory_key INTEGER NOT NULL,
key SERIAL NOT NULL,
PRIMARY KEY (key),
UNIQUE (genome_name, directory_key),
FOREIGN KEY(directory_key) REFERENCES drepdirectory (key)
)The parsomics project is organized into multiple repositories called 'modules'. Key modules include:
parsomics-core: Library for inserting data into the database.parsomics-api-server: A REST API for database interaction.parsomics-cli: Command-line interface for managing analyzes, databases, and plugins.parsomics-plugin-interpro: A plugin for processing InterproScan annotations.Data ingestion in parsomics follows a hierarchical structure:
overview.txt files must be TSV files including a header row. The columns must appear in the exact order listed below. The Gene ID column must contain the protein name used to identify the protein in the Parsomics database.| Column name | Obligatoriness | Data type | Nullability |
|---|---|---|---|
| Gene ID | Mandatory | String | Not nullable |
| EC# | Optional | String | Nullable |
| HMMER | Optional | String | Nullable |
| dbCAN_sub | Optional | String | Nullable |
| DIAMOND | Optional | String | Nullable |
| eCAMI | Optional | String | Nullable |
| Signalp | Optional (ignored) | String | N/A |
| #ofTools | Optional (ignored) | Integer | N/A |
To define which files your plugin should process and how to extract the genome name, create a subclass of ValidatedFileWithGenome.
_VALID_FILE_TERMINATIONS: A ClassVar[list[str]] containing the file extensions or suffixes that identify valid files. These must be specific enough to distinguish actual data files from logs or other noise in the output directory.genome_name property: This method should return the genome name as a string, typically by removing the defined terminations from the filename.from pathlib import Path
from typing import ClassVar
from parsomics_core.entities.files.validated_file import ValidatedFileWithGenome
class InterproTsvValidatedFile(ValidatedFileWithGenome):
# Define suffixes that identify valid files
_VALID_FILE_TERMINATIONS: ClassVar[list[str]] = [
"_interpro_out.tsv",
"_interpro.tsv",
]
@property
def genome_name(self) -> str:
file_name: str = Path(self.path).name
# Remove valid file terminations to extract genome name
for termination in InterproTsvValidatedFile._VALID_FILE_TERMINATIONS:
file_name = file_name.removesuffix(termination)
genome_name = file_name
if genome_name is None:
raise Exception(
f"Failed at extracting genome name from interpro tsv file: {self.path}"
)
return genome_nameparsomics-core library. These utilities are available in the parsomics_core.plugin_utils module. For example, search_protein_by_name can be used to find proteins by their names.from parsomics_core.plugin_utils import search_protein_by_name