DOCSEARCHbyAlgolia
Submit Docs

PecanPy: Parallelized node2vec and node2vec+ Implementation

repository·master·Indexed 41 days ago

https://github.com/krishnanlab/pecanpy

A high-performance Python implementation of node2vec and node2vec+ algorithms optimized for generating node embeddings in large biological networks. It features a CLI for embedding generation and graph format conversion, supporting .edg and .npz files. The library provides specialized execution modes (PreComp, SparseOTF, and DenseOTF) based on network size and density to optimize memory and runtime, along with an experimental Node2vec++ implementation for continuous bias-factor parameterization.

Tokens
6.6K
Snippets
22
Records
34
Agent score
50%

What's inside PecanPy

  1. Choose the appropriate PecanPy mode

    master

    PecanPy uses three primary modes optimized for different network sizes and densities. Choosing the correct mode is critical for performance.

    ModeNetwork size/densityOptimization
    PreComp<10k nodes, <0.1% edgesPrecompute second order transition probabilities, using CSR graph
    SparseOTF (default)(≥10k nodes, ≥0.1% and <20% of edges) or (<10k nodes, ≥0.1% edges)Transition probabilities computed on-the-fly, using CSR graph
    DenseOTF>20% of edgesTransition probabilities computed on-the-fly, using dense matrix

    Recommendations:

    • Use PreComp if the graph is small and sparse.
    • Use SparseOTF if the graph is sparse but not necessarily small.
    • Use DenseOTF if the graph is extremely dense.

  2. Run node2vec with the PecanPy CLI

    master

    PecanPy provides a command-line utility pecanpy to generate node embeddings. You must specify an --input file, an --output file, and a --mode.

    Basic node2vec usage:

    pecanpy --input demo/karate.edg --output demo/karate.emb --mode SparseOTF

    Using node2vec+ (for weighted graphs): To enable the node2vec+ extension, which handles weighted graphs more effectively, add the --extend flag. Note that for unweighted graphs, node2vec+ is equivalent to standard node2vec.

    pecanpy --input demo/karate.edge --output demo/karate_n2vplus.emb --mode SparseOTF --extend

  3. Install PecanPy

    master

    You can install the latest stable release of PecanPy via pip. For development purposes, you can install it in editable mode from the GitHub repository.

    Stable Release:

    $ pip install pecanpy

    Development Mode:

    $ git clone https://github.com/krishnanlab/pecanpy.git
$ cd pecanpy
$ pip install -e .

  4. How to handle implicit node IDs in PecanPy graphs

    master

    When loading graphs from files (like CSR .npz files) that do not contain an explicit IDs field, PecanPy can assign canonical integer IDs (0, 1, 2...) to nodes. This behavior is controlled by the implicit_ids parameter in read_npz methods.

    • If implicit_ids=False (default): If the IDs field is missing, PecanPy will automatically set node IDs to the canonical ordering (0 to $N-1$) and issue a warning.
    • If implicit_ids=True: The canonical ordering is used, but the warning message is suppressed.

    This is particularly relevant when loading graphs prepared by other libraries (like scipy.sparse) that might not include node metadata.

  5. Choose a PecanPy implementation strategy

    master

    PecanPy provides several classes for generating walks, depending on whether you want to precompute transition probabilities or calculate them on-the-fly (OTF).

    ClassStrategyDescription
    FirstOrderUnweightedDirect SamplingDirectly samples edges for first-order random walks.
    PreCompFirstOrderPrecomputed 1st OrderPrecomputes and stores 1st order transition probabilities using alias tables.
    PreCompPrecomputed 2nd OrderPrecomputes and stores 2nd order transition probabilities. Requires calling preprocess_transition_probs() before generating walks.
    SparseOTFOn-the-fly SparseCalculates transition probabilities on-the-fly for sparse graphs.
    DenseOTFOn-the-fly DenseCalculates transition probabilities on-the-fly for dense graphs.

  6. How to choose the best PecanPy execution mode

    master

    PecanPy provides several execution modes to optimize for memory usage and runtime based on the characteristics of your graph. The CLI will issue warnings if a more efficient mode is recommended.

    Mode Selection Logic

    Graph CharacteristicRecommended Mode
    High Density (density $\ge 0.2$)DenseOTF
    Low Density (density $< 0.001$) AND Small Size (nodes $< 10,000$)PreComp
    Medium Density ($0.001 \le$ density $< 0.2$)SparseOTF
    Low Density (density $< 0.001$) AND Large Size (nodes $\ge 10,000$)SparseOTF

    Special Constraints

    • FirstOrderUnweighted: Requires weighted=False and p=1, q=1.
    • PreCompFirstOrder: Requires p=1, q=1.
    • Node2vec+ (--extend):
      • Currently not implemented for directed graphs.
      • For unweighted graphs, node2vec+ is equivalent to standard node2vec.

  7. Use the PecanPy CLI to generate node2vec embeddings

    master

    The pecanpy command line utility allows you to perform parallelized node2vec and node2vec+ embedding generation on graphs provided in .edg or .npz formats.

    Basic Usage

    To run PecanPy in PreComp mode on a karate network:

    pecanpy --input demo/karate.edg --output demo/karate.emb --mode PreComp

    Input and Output

    • --input: Path to the input graph file. Supports .edg or .npz formats.
    • --output: Path to the output embeddings file.
      • If the path ends in .npz, the output is saved as a NumPy compressed file containing node IDs and vectors.
      • Otherwise, it is saved using the gensim save_word2vec_format (text format).

    Execution Modes

    Choosing the right --mode is critical for performance based on graph density and size:

    • DenseOTF: Recommended for high-density networks (density $\ge 0.2$).
    • SparseOTF: Recommended for sparse networks (density $\ge 0.001$ and density $< 0.2$, or large sparse networks).
    • PreComp: Recommended for small, very sparse networks (density $< 0.001$ and nodes $< 10,000$).
    • FirstOrderUnweighted: A specialized mode that only works when weighted=False and p=q=1.
    • PreCompFirstOrder: A specialized mode that only works when p=q=1.

    Common Parameters

    • --dimensions: Number of embedding dimensions (default: 128).
    • --walk-length: Length of random walk per source (default: 80).
    • --num-walks: Number of walks per source (default: 10).
    • --window-size: Context size for optimization (default: 10).
    • --epochs: Number of epochs in SGD for Word2Vec training (default: 1).
    • --workers: Number of parallel workers (default: 0, which uses all available threads).
    • --p / --q: Hyperparameters for node2vec (default: 1.0).
    • --weighted: Specify if the graph is weighted.
    • --directed: Specify if the graph is directed.
    • --extend: Enable the node2vec+ extension.
    • --gamma: Noisy edge threshold parameter for node2vec+.
    • --random_state: Random seed for reproducibility.

  10. Configure PecanPy node2vec parameters

    master

    When initializing a PecanPy object (such as PreComp, SparseOTF, or DenseOTF), you can configure the following parameters to control the node2vec algorithm behavior:

    • p (float): Return parameter. Values < 1 encourage returning to the previous vertex; values > 1 discourage it (default: 1).
    • q (float): In-out parameter. Values < 1 encourage walking "outward"; values > 1 encourage walking within a localized neighborhood (default: 1).
    • workers (int): Number of threads to spawn for running node2vec (default: 1).
    • verbose (bool): Whether to show a progress bar for walk generation (default: False).
    • extend (bool): If True, uses the node2vec+ extension (default: False).
    • gamma (float): Multiplication factor for the std term of edge weights added to average edge weights as the noisy edge threshold. Only used when extend=True (default: 0).
    • random_state (int, optional): Random seed for reproducibility. To ensure full reproducibility, use workers=1 and potentially set the PYTHONHASHSEED environment variable to match this value.
    from pecanpy import pecanpy as node2vec

# Example configuration
g = node2vec.PreComp(p=0.5, q=1, workers=4, verbose=True, extend=True, gamma=0.1)

  11. Configure input file formats

    master

    PecanPy supports two main input formats:

    1. Edgelist (.edg): The default format. Each line should contain node IDs and an optional weight: node1_id node2_id <weight_float, optional> The default delimiter is a tab (\t), but you can change it using the --delimiter option.

    2. Numpy Array (.npz): Supported only when using DenseOTF mode. You can convert an .edg file to a .npz file using the --task todense command:

      pecanpy --input $input_edgelist --output $output_npz --task todense
    pecanpy --input $input_edgelist --output $output_npz --task todense

  12. Understand the PecanPy output format

    master

    The output file contains $n+1$ lines for a graph with $n$ vertices:

    1. Header line: num_of_nodes dim_of_representation
    2. Representation lines: Each subsequent line contains the node ID followed by its $d$-dimensional representation: node_id dim_1 dim_2 ... dim_d