DOCSEARCHbyAlgolia
Submit Docs

MAGE (Memgraph Advanced Graph Extensions)

repository·main·Indexed 43 days ago

https://github.com/memgraph/mage

An open-source collection of user-defined graph analytics modules and procedures that extend the Cypher query language. MAGE allows developers to run complex graph algorithms written in C++, Python, CUDA, and Rust directly inside Memgraph, including GPU-accelerated algorithms via cuGraph and GNN-based modules.

Tokens
9.7K
Snippets
16
Records
22
Agent score
54%

What's inside MAGE

  1. Use the correct data structures for Memgraph query modules

    main

    When implementing query modules, distinguish between internal logic structures and those used for Memgraph communication:

    • Logic Structures: Use Edge, List, Map, Memgraph, Path, Property, Value, and Vertex for your procedure logic.
    • Return Structures: Use structures prefixed with Mgp (e.g., Mgp...) to bridge data between your Rust code and Memgraph. These are specifically designed to return data to the Memgraph client.
    • Avoid Raw Pointers: While underlying pointers prefixed with mgp_ are exposed, you should not manipulate them directly; the library is designed to abstract this complexity.

  2. Handle strings using CStr for Memgraph compatibility

    main
    The Memgraph query module API provides C strings. Because Rust String types can contain null bytes (which are invalid in C strings), you must use &CStr or the c_str library when interacting with the rsmgp API to ensure compatibility and prevent translation errors.

  3. What are MAGE query modules?

    main

    MAGE provides query modules, which are user-defined procedures that extend the Cypher query language with advanced graph analytics and algorithms.

    These modules are implemented in multiple programming languages and are grouped into modules that can be loaded into Memgraph. When Memgraph starts, it automatically attempts to load query modules from all *.so (shared object) and *.py (Python) files found in the directory specified by the --query-modules-directory flag.

  4. Run E2E Migration Tests

    main

    The E2E Migration Testing Framework validates data migration from MySQL and PostgreSQL to Memgraph using the migrate module. It tests data type conversions, error handling, and uses a per-data-type approach for granular coverage. The framework automatically manages database containers and test data setup.

    To run the full suite of tests, use the provided shell script. You can also filter tests by keyword (e.g., mysql) or run them directly via pytest.

    # Run all tests (recommended)
./test_e2e_migration.sh

# Run specific tests (e.g., MySQL only)
./test_e2e_migration.sh -k mysql

# Run using pytest directly
python3 -m pytest e2e_migration/ -v

  5. Install MAGE on Linux from source

    main

    This method is intended for local development.

    Prerequisites

    Ensure you have a Linux-based Memgraph package installed (Ubuntu, Debian, or CentOS). You also need the following dependencies installed on your system:

    • libcurl4, libpython3.12, libssl-dev, libboost-all-dev, openssl, build-essential, make, cmake, curl, g++, python3, python3-pip, python3-setuptools, python3-dev, clang, unixodbc, uuid-dev.

    Installation Steps

    1. Clone the repository with submodules:
    git clone --recurse-submodules https://github.com/memgraph/mage.git && cd mage
    1. Install Rust and Python dependencies:
    curl https://sh.rustup.rs -sSf | sh -s -- -y \
&& export PATH="/root/.cargo/bin:${PATH}" \
&& python3 -m pip install -r /mage/python/requirements.txt \
&& python3 -m pip install -r /mage/python/tests/requirements.txt \
&& python3 -m pip install torch-sparse torch-cluster torch-spline-conv torch-geometric torch-scatter -f https://data.pyg.org/whl/torch-1.12.0+cu102.html
    1. Compile and install modules: Run the setup script to compile the query modules and copy them to the Memgraph query modules directory (defaulting to /usr/lib/memgraph/query_modules):
    python3 setup build -p /usr/lib/memgraph/query_modules

    Loading Modules

    Query modules are loaded by Memgraph on startup. If Memgraph is already running, you must manually load them using the following query in your preferred querying platform:

    CALL mg.load_all();

  6. Add rsmgp-sys to your Rust project

    main

    To develop Memgraph query modules in Rust, add rsmgp-sys and c_str_macro to your Cargo.toml. You must also configure the library type as cdylib so it can be loaded as a dynamic library by Memgraph.

    [dependencies]
c_str_macro = "1.0.2"
rsmgp-sys = "^0.1.0"

[lib]
name = "query_module_name"
crate-type = ["cdylib"]

  7. Create a Memgraph query module in Rust

    main

    To develop a Memgraph query module using rsmgp-sys, create a new standard Rust project within the rust/ directory of the repository. The project must be configured as a dynamic library by specifying cdylib in the [lib] section of your Cargo.toml file. This allows the resulting binary to be loaded as a module by Memgraph.

    [lib]
name = "query_module_name"
crate-type = ["cdylib"]

  9. Build a custom MAGE Docker image from source

    main

    If you need to include new algorithms or custom changes, you can build your own MAGE Docker image.

    1. Clone the repository with submodules:
    git clone --recurse-submodules https://github.com/memgraph/mage.git && cd mage
    1. Download the Memgraph package into the repository. Replace ${MEMGRAPH_VERSION} with the latest release and ${ARCHITECTURE} with amd64 or arm64.

    For amd64:

    curl -L "https://download.memgraph.com/memgraph/v${MEMGRAPH_VERSION}/ubuntu-24.04/memgraph_${MEMGRAPH_VERSION}-1_${ARCHITECTURE}.deb" > memgraph-${ARCHITECTURE}.deb

    For arm64:

    curl -L "https://download.memgraph.com/memgraph/v${MEMGRAPH_VERSION}/ubuntu-24.04-aarch64/memgraph_${MEMGRAPH_VERSION}-1_arm64.deb" > memgraph-arm64.deb
    1. Build the image using docker buildx. Set ${architecture} to amd64 or arm64:
    DOCKER_BUILDKIT=1 docker buildx build \
--tag memgraph-mage:prod \
--target prod \
--platform linux/${architecture} \
--file Dockerfile.release \
--load .
    1. Run the container:
    docker run --rm -p 7687:7687 -p 7444:7444 --name mage memgraph-mage

    Note: If you make changes to the MAGE code while a container is running, you must stop it and rebuild the image.

  11. Install MAGE using Docker Hub

    main

    The fastest way to use MAGE is to pull the pre-built image from Docker Hub. This image contains Memgraph preloaded with all MAGE modules. Run the following command to download and start the container, mapping the default Memgraph ports (7687 for Bolt and 7444 for HTTP).

    docker run -p 7687:7687 -p 7444:7444 memgraph/memgraph-mage

  12. Install dependencies for cuGraph module

    main

    To set up the environment for the cuGraph module on Ubuntu 20.04, follow these steps to install CMake, CUDA, and system libraries.

    1. Install CMake (Version 3.22+)

    cuGraph requires CMake version 20.0+ (the example uses 3.22).

    2. Install CUDA (Example: CUDA 11.6)

    Install CUDA on your device using the NVIDIA repository.

    3. Configure Environment Variables

    Ensure CUDA_HOME, LD_LIBRARY_PATH, and PATH are correctly set to point to your CUDA installation.

    4. Install System Dependencies

    Install required development libraries for BLAS, LAPACK, and Boost.

    5. Install NCCL

    NVIDIA Collective Communications Library (NCCL) is required. You can install it via the NVIDIA repository.

    # 1. Install CMake 3.22
sudo apt remove --purge cmake
sudo apt install build-essential libssl-dev
wget https://github.com/Kitware/CMake/releases/download/v3.22.2/cmake-3.22.2.tar.gz
tar -zxvf cmake-3.22.2.tar.gz
cd cmake-3.22.2
./bootstrap
make
sudo make install

# 2. Install CUDA 11.6
sudo apt-get install linux-headers-$(uname -r)
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/11.6.1/local_installers/cuda-repo-ubuntu2004-11-6-local_11.6.1-510.47.03-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2004-11-6-local_11.6.1-510.47.03-1_amd64.deb
sudo apt-key add /var/cuda-repo-ubuntu2004-11-6-local/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda

# 3. Set Environment Variables
export CUDA_HOME=/usr/local/cuda
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
export PATH=$PATH:$CUDA_HOME/bin

# 4. Install System Dependencies
sudo apt install libblas-dev liblapack-dev libboost-all-dev

# 5. Install NCCL
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub
sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /"
sudo apt-get update
sudo apt install libnccl2=2.11.4-1+cuda11.6 libnccl-dev=2.11.4-1+cuda11.6