DOCSEARCHbyAlgolia
Submit Docs

AgensGraph Documentation

repository·main·Indexed 23 days ago

https://github.com/skaiworldwide-oss/agensgraph

A multi-model graph database built on PostgreSQL that supports simultaneous use of relational (SQL) and graph (openCypher/GQL) data models. Features include an ACID-compliant platform, connectivity-aware scan pruning via auto_gather_graphmeta, and an Asynchronous and Direct IO (AIO) subsystem for improved throughput. Supports JDBC, Python, Node.js, and Go drivers, and integrates with AGViewer for graph data visualization.

Tokens
59.3K
Snippets
34
Records
286
Agent score
81%

What's inside AgensGraph

  2. Enable Connectivity-Aware Scan Pruning with `auto_gather_graphmeta`

    main

    By default, unlabelled Cypher elements (e.g., (b) in MATCH (a)-[:KNOWS]->(b)) trigger scans across all vertex/edge labels. Enabling auto_gather_graphmeta allows the planner to use the ag_graphmeta catalog to prune these scans by only looking at labels that are actually connected in the graph.

    How to enable

    SET auto_gather_graphmeta = on;

    Key Behaviors

    • Transitive Pruning: A labelled edge constrains its endpoints, and these constraints propagate through the entire pattern.
    • Safety: Pruning is a plan-time optimization and never changes query results. It is safe to use with prepared statements and connection poolers.
    • Regathering: If you enable this via postgresql.conf, it will not automatically regather existing data. To build a baseline for an existing graph, run SET auto_gather_graphmeta = on; in a session or call regather_graphmeta() after toggling it on.
    • Bulk Loading: Maintaining the catalog during bulk loads is efficient (uses in-memory counters).
    SET auto_gather_graphmeta = on;   -- enables maintenance AND planner pruning

  3. How the AIO subsystem prevents deadlocks and starvation

    main

    In a multi-process environment like Postgres, naive AIO can cause deadlocks if a backend performing readahead blocks before its IO completions are processed.

    AgensGraph's AIO implementation prevents this by ensuring IO completions can be processed by any backend in the system. This is achieved through two primary mechanisms:

    1. Direct Processing: Using technologies like io_uring that allow completions to be handled by any backend.
    2. Worker Mode: Offloading completion processing to dedicated AIO workers, ensuring that even if the original issuing backend is blocked, the IO completion is still processed.

  4. Use AIO handles to manage IO operations

    main

    AIO handles are the central abstraction for performing IO. The lifecycle of an IO operation typically follows these steps:

    1. Acquire: Obtain a handle using pgaio_io_acquire(). Note that pgaio_io_acquire() must always succeed; to avoid self-deadlock, do not acquire a handle without eventually defining it (calling a pgaio_io_start_*() function).
    2. Define: Associate the operation with the handle using low-level functions like pgaio_io_start_*().
    3. Release (Optional): If a handle is acquired but no longer needed (e.g., before holding a contended lock), release it without defining it using pgaio_io_release().

    Handles are limited in number and must be reused as soon as they have completed. To track completion, use AIO callbacks or AIO wait references.

  5. Handle IO completion with AIO callbacks

    main

    Because multiple layers of the system (e.g., bufmgr.c and md.c) often need to react to the same IO completion, the AIO subsystem allows associating multiple completion callbacks with a single handle.

    Key details:

    • Identification: Since shared memory cannot contain function pointers, callbacks are identified by IDs of type PgAioHandleCallbackID (currently a single byte).
    • Staging: Callbacks are also used to "stage" an IO, such as increasing buffer reference counts to ensure the buffer remains valid while the IO is in progress.
    • Safety: Callbacks are executed in critical sections and may be executed by any backend, so they must be safe to execute in these contexts.

  6. Use the AIO subsystem for asynchronous and direct IO

    main

    AgensGraph provides an Asynchronous and Direct IO (AIO) subsystem to improve throughput and reduce latency, particularly for WAL writes and large buffer reads. Using Direct IO avoids the CPU overhead of copying data between the kernel's page cache and the postgres buffer pool by using DMA, and avoids double buffering.

    To use AIO, you typically interact with PgAioHandle and PgAioWaitRef objects. A common pattern involves:

    1. Acquiring an AIO handle via pgaio_io_acquire.
    2. Registering completion callbacks (e.g., PGAIO_HCB_SHARED_BUFFER_READV) so that buffer descriptors are updated automatically when the IO completes.
    3. Associating buffer data with the handle using pgaio_io_set_handle_data_32.
    4. Passing the handle to a storage manager function like smgrstartreadv.
    5. Performing other work to hide latency.
    6. Waiting for completion using pgaio_wref_wait.
    7. Checking the result status via PgAioReturn and reporting errors with pgaio_result_report.
    /* Example of reading a buffer into shared buffers using AIO */
PgAioReturn ioret;
PgAioHandle *ioh = pgaio_io_acquire(CurrentResourceOwner, &ioret);

PgAioWaitRef iow;
pgaio_io_get_wref(ioh, &iow);

pgaio_io_register_callbacks(ioh, PGAIO_HCB_SHARED_BUFFER_READV, 0);

pgaio_io_set_handle_data_32(ioh, (uint32 *) buffer, 1);

smgrstartreadv(ioh, operation->smgr, forknum, blkno, BufferGetBlock(buffer), 1);

perform_other_work();

pgaio_wref_wait(&iow);

if (ioret.result.status == PGAIO_RS_ERROR)
    pgaio_result_report(ioret.result, &ioret.target_data, ERROR);

if (ioret.result.status != PGAIO_RS_OK)
    pgaio_result_report(ioret.result, &ioret.target_data, ERROR);

  7. Quick Start with Docker

    main

    You can quickly run AgensGraph using Docker. This process involves pulling the image, running a container with specific environment variables for credentials, and then connecting via the AgensGraph client.

    1. Pull the image: docker pull skaiworldwide/agensgraph (defaults to latest tag).

    2. Run the container: Map port 5455 to the internal 5432 and set your credentials using POSTGRES_USER, POSTGRES_PASSWORD, and POSTGRES_DB environment variables.

    3. Connect to the client: Use docker exec to access the agens CLI tool inside the container.

    # 1. Pull the image
docker pull skaiworldwide/agensgraph

# 2. Create and run the container
docker run \
    --name agensgraph \
    -p 5455:5432 \
    -e POSTGRES_USER=postgres \
    -e POSTGRES_PASSWORD=agens \
    -e POSTGRES_DB=agens \
    -d \
    skaiworldwide/agensgraph

# 3. Connect to AgensGraph client
docker exec -it agensgraph agens -d agens -U postgres

  9. Tune AgensGraph performance for graph workloads

    main

    AgensGraph is built on PostgreSQL, but default settings are optimized for relational queries. For graph analytics (e.g., multi-hop traversals, neighborhood aggregation), you should increase memory for sorts/hashes and enable parallelism.

    Settings can be applied via ALTER SYSTEM SET ...; followed by SELECT pg_reload_conf();, or by editing postgresql.conf.

    Recommended Configuration

    • Memory: Increase work_mem to avoid disk spills during heavy GROUP BY or DISTINCT operations. Increase maintenance_work_mem for faster index builds on large edge labels.
    • Planner: Set effective_cache_size to ~70% of RAM and random_page_cost to 1.1 if using SSDs.
    • Parallelism: Increase max_parallel_workers_per_gather and max_parallel_workers to speed up full-edge-scan analytics.
    # Memory: avoid disk spills in graph aggregation/sort
work_mem = 256MB                # per sort/hash node; raise for heavy GROUP BY / DISTINCT
maintenance_work_mem = 1GB      # faster index builds and VACUUM on large edge labels

# Planner: trust indexes and account for the cache
effective_cache_size = <~70% of RAM>
random_page_cost = 1.1          # for SSDs, where index scans are nearly as cheap as sequential

# Parallelism: speeds up full-edge-scan analytics
max_parallel_workers_per_gather = 4
max_parallel_workers = <number of CPU cores>
max_worker_processes = <number of CPU cores>

  10. Overview of Snowball-Based Stemming in AgensGraph

    main
    AgensGraph utilizes the Snowball stemming project for word stemming operations. The implementation includes derived C files from the Snowball project, allowing stemming capabilities to work without requiring the Snowball-to-C compiler to be present on the target installation. The module is licensed under a BSD-style license.

  11. Overview of GiST Indexing in AgensGraph

    main

    GiST (Generalized Search Tree) is a flexible indexing implementation for Postgres that supports various data types and search patterns. It is designed to handle variable-length keys and provides a robust framework for complex indexing tasks.

    Key Features:

    • Variable length keys: Supports composite (multi-key) and variable-length data.
    • Ordered search: Enables nearest-neighbor search.
    • Concurrency: Implements high-concurrency access methods.
    • Reliability: Provides recovery support via WAL (Write-Ahead Logging) and a NULL-safe interface to the GiST core.
    • Efficient Building: Supports both a buffering build algorithm (for bulk loads) and a sorted build method (for sorted input).

  12. Use libpq-oauth for OAuth Device Authorization flow

    main

    The libpq-oauth module implements the OAuth Device Authorization flow (RFC 8628) for libpq clients. It is an optional shared library that allows libpq to handle OAuth authentication requests from the server.

    If a connection string permits OAuth and the server requests it, libpq will attempt to delay-load this module using dlopen(). If the module is not found or fails to load, the connection attempt will fail.

    Note that this module has a dependency on libcurl. Users who wish to avoid the libcurl dependency can choose not to install this module.