DOCSEARCHbyAlgolia
Submit Docs

FPSim2

repository·master·Indexed 41 days ago

https://github.com/chembl/fpsim2

A high-performance Python/C++ package for fast molecular similarity searching, optimized for high thresholds (≥0.7). It utilizes CPU POPCNT instructions and supports both multicore CPU and GPU-accelerated Tanimoto searches. The library features a compressed file format based on PyTables and BLOSC2, supports various RDKit fingerprint types, and includes the iSIM contributor module for advanced molecular dataset statistics.

Tokens
10.7K
Snippets
40
Records
56
Agent score
49%

What's inside FPSim2

  1. Overview of FPSim2

    master

    FPSim2 is a NumPy-centric Python/C++ package designed for high-performance compound similarity searches. It is optimized for high search thresholds (typically $\ge 0.7$) and is used in the ChEMBL and SureChEMBL interfaces.

    Key technical features include:

    • Fast Bit Counting: Utilizes the CPU POPCNT instruction.
    • Sublinear Speedups: Implements bounds for optimized search performance.
    • Optimized Storage: Uses a compressed file format based on PyTables and BLOSC2 for fast read speeds.
    • Hardware Acceleration: Supports both multicore CPU and GPU-based similarity searches.
    • Flexible Search Modes: Supports both in-memory and on-disk search modes, as well as distance matrix calculation.

  2. Overview of FPSim2 capabilities

    master

    FPSim2 is a NumPy-centric Python/C++ package based on RDKit designed for high-speed molecular similarity searches. It is optimized for high search thresholds (>=0.7) and features:

    • High Performance: Utilizes CPU POPCNT instructions for fast bit counting and implements sublinear speedup bounds.
    • Optimized Storage: Uses a compressed file format optimized for read speeds via PyTables and BLOSC2.
    • Flexible Search Modes: Supports fast multicore CPU and GPU similarity searches, with both in-memory and on-disk search modes.
    • Advanced Math: Supports distance matrix calculation.

  3. Configure Tversky weighting parameters `a` and `b`

    master

    The a and b parameters in Tversky search functions control the asymmetric weighting of the query and reference fingerprints:

    • a: Controls the weighting of the query fingerprint.
    • b: Controls the weighting of the reference fingerprint.

    Adjusting these values allows you to fine-tune the similarity measure. For example, setting a to a higher value than b will give more weight to the query fingerprint.

  4. Use the SQLAlchemy backend for SQL databases

    master

    While PyTables h5 is the primary backend for FPSim2, you can use a SQLAlchemy backend to store fingerprints in MySQL, PostgreSQL, or Oracle databases. This is useful for incremental builds, integration with existing database infrastructure, or when you prefer database management over file-based storage.

    Note on performance: The SQLAlchemy backend has noticeably slower fingerprint loading times compared to PyTables h5 files. However, once loaded, search performance is identical to the PyTables backend.

  5. Getting started with FPSim2

    master

    To begin using FPSim2, you must first install the package and generate a fingerprint database (db) file. The standard workflow involves:

    1. Installation: Setting up the Python environment and dependencies.
    2. Database Creation: Generating the fingerprint database file from molecular data.
    3. Sanitization: Ensuring molecular data is in the correct format for fingerprinting.
    4. Backend Selection: Optionally using an alternative SQLAlchemy backend if required.

  6. Perform an incremental load into an existing SQL table

    master

    You can append new molecules to an existing table using create_db_table. If your source data is in the same SQL instance, you can pass a SQLAlchemy CursorResult object directly to create_db_table to facilitate the incremental load.

    from sqlalchemy import create_engine, text
from FPSim2.io import create_db_table

fp_type = "Morgan"
fp_params = {"radius": 2, "fpSize": 256}
db_url = "postgresql://user:password@hostname:5432/fpsim2"
table_name = "fpsim2_fp_table"
mol_format = "smiles"

sql_query = text(f"""
    SELECT
        smiles,
        mol_id
    FROM
        structure
    WHERE
        mol_id > COALESCE((
            SELECT
                MAX(mol_id)
            FROM
                {table_name}
        ), 0)
    ORDER BY
        mol_id
""")

engine = create_engine(db_url)
with engine.connect() as conn:
    cursor = conn.execute(sql_query)
    create_db_table(cursor, db_url, table_name, mol_format, fp_type, fp_params)

  7. Retrieve the top K most similar hits

    master

    If you need a specific number of hits rather than all hits above a threshold, use the top_k methods. Similar to symmetric searches, you can choose between in-memory and on-disk execution.

    In-memory Top K

    Use FPSim2Engine.top_k with the k parameter to specify the number of hits to return. Use n_workers to speed up the search on large datasets.

    On-disk Top K

    Use FPSim2Engine.on_disk_top_k for datasets larger than RAM. Ensure the engine is initialized with in_memory_fps=False.

    from FPSim2 import FPSim2Engine

# In-memory Top K
fp_filename = 'chembl_35_v0.6.0.h5'
fpe = FPSim2Engine(fp_filename)
query = 'CC(=O)Oc1ccccc1C(=O)O'
results = fpe.top_k(query, k=100, threshold=0.7, metric='tanimoto', n_workers=1)

# On-disk Top K
fpe_disk = FPSim2Engine(fp_filename, in_memory_fps=False)
results_disk = fpe_disk.on_disk_top_k(query, k=100, threshold=0.7, metric='tanimoto')

  8. Run Tversky similarity searches in memory

    master

    Use the FPSim2Engine.tversky method to perform a Tversky similarity search using fingerprints loaded into RAM. The Tversky coefficient is a generalization of the Tanimoto coefficient that allows for asymmetric weighting between the query and reference fingerprints.

    To optimize performance, use the n_workers parameter to split the search across multiple threads, which is particularly effective for large datasets.

    from FPSim2 import FPSim2Engine

fp_filename = 'chembl_35_v0.6.0.h5'
fpe = FPSim2Engine(fp_filename)

query = 'CC(=O)Oc1ccccc1C(=O)O'
results = fpe.tversky(query, threshold=0.7, a=0.7, b=0.3, n_workers=1)

  9. Run Tversky similarity searches on disk

    master

    If your fingerprint dataset is too large to fit in available RAM, use the FPSim2Engine.on_disk_tversky method.

    To use this method, you must initialize the FPSim2Engine with in_memory_fps=False. Note that on-disk searches are significantly slower than in-memory searches and should only be used when necessary.

    from FPSim2 import FPSim2Engine

fp_filename = 'chembl_35_v0.6.0.h5'
fpe = FPSim2Engine(fp_filename, in_memory_fps=False)

query = 'CC(=O)Oc1ccccc1C(=O)O'
results = fpe.on_disk_tversky(query, threshold=0.7, a=0.7, b=0.3)

  10. Use iSIM contributor module for molecular comparisons

    master

    The iSIM module (developed by the Miranda-Quintana Group) allows for performing comparisons of multiple molecules simultaneously. It yields values equivalent to the average pairwise comparisons of molecules represented by binary fingerprints.

    To use these functions, you must first initialize an FPSim2Engine with a fingerprint file (e.g., an .h5 file).

    from FPSim2.contrib.isim.isim_comp import (
    get_sim_dict,
    calculate_medoid,
    calculate_outlier,
    calculate_comp_sim,
)
from FPSim2 import FPSim2Engine

fp_filename = "chembl_35_v0.6.0.h5"
fpe = FPSim2Engine(fp_filename)

# get average similarity of the dataset
sim_dict = get_sim_dict(fpe)

# calculate medoid of the dataset
medoid = calculate_medoid(fpe)

# calculate outlier of the dataset
outlier = calculate_outlier(fpe)

# calculate complementary similarity
comp_sim = calculate_comp_sim(fpe)

  11. Run on-disk substructure screenouts with FPSim2Engine.on_disk_substructure

    master

    If your dataset is larger than your available RAM, you must use the FPSim2Engine.on_disk_substructure method.

    Note: This method is significantly slower than the in-memory version and should ONLY be used if the dataset does not fit in memory. To prepare the engine for on-disk operations, you must initialize FPSim2Engine with in_memory_fps=False.

    Like the in-memory method, you can use n_workers to parallelize the search across multiple threads.

    from FPSim2 import FPSim2Engine

fp_filename = 'chembl_35_v0.6.0.h5'
fpe = FPSim2Engine(fp_filename, in_memory_fps=False)

query = 'CC(=O)Oc1ccccc1C(=O)O'
results = fpe.on_disk_substructure(query, n_workers=1)