Gaffer Graph Database Framework

repository·develop·Indexed 21 days ago

https://github.com/gchq/gaffer

A scalable graph database framework designed for high-rate data ingestion and rapid querying of large-scale graphs with rich properties. Gaffer supports multiple storage backends, including Accumulo and in-memory Java Maps, and includes libraries for Spark, Flink, and bitmap operations, as well as a TinkerPop framework implementation called GafferPop.

Tokens
5.1K
Snippets
13
Records
34
Agent score
80%

What's inside Gaffer

  1. Overview of the Accumulo Store

    develop

    The Accumulo Store is a component of Gaffer designed to work with Apache Accumulo. For detailed administration and implementation guidance, refer to the official Gaffer documentation.

  2. What is Gaffer?

    develop

    Gaffer is a graph database framework designed for storing and querying very large graphs containing rich properties on nodes and edges. It is built for flexibility and scalability, allowing for rapid prototyping that can transition to production systems.

    Key features include:

    • High-performance querying: Rapid traversal across large numbers of nodes and edges.
    • High-rate ingestion: Supports continual data ingestion and batch bulk ingest via MapReduce or Spark.
    • Rich Data Model: Storage of arbitrary Java objects on nodes and edges.
    • In-database Aggregation: Automatic, user-configurable aggregation of statistical properties (e.g., counts, histograms, sketches).
    • Data Transformation: Versatile query-time summarisation, filtering, and transformation.
    • Security & Compliance: Fine-grained data access controls and hooks for policy/compliance rules.
    • Data Lifecycle Management: Automated, rule-based removal of data (age-off).
    • Analysis Integration: Retrieval of graph data into Apache Spark.
    • REST API: A fully-featured REST interface for interacting with the graph.
  3. Explore related Gaffer repositories

    develop

    Depending on your workflow, you may find these related projects useful:

    • gafferpy: A Python shell for executing Gaffer operations.
    • gaffer-docker: Code for running Gaffer using Docker or Kubernetes.
    • koryphe: An extensible functions library for filtering, aggregating, and transforming data based on the Java Function API (a dependency of Gaffer).
  4. Understand the Road Traffic graph model

    develop

    The Road Traffic example models UK road use data as a Gaffer graph to demonstrate data exploration and summarization. The graph consists of several edge types representing different relationships:

    • Region to Location: e.g., South West $\rightarrow$ Bristol
    • Location to Roads: e.g., Bristol $\rightarrow$ M32
    • Roads and their Junctions: e.g., M32 $\rightarrow$ M32:1
    • Junctions and their locations: e.g., M32:2 $\rightarrow$ 361150,175250
    • Traffic counts between junctions: e.g., M32:1 $\rightarrow$ M32:2 (representing hourly traffic flow)

    Traffic Count Edges (RoadUse): Multiple edges exist between the same two junctions, one for each hour of observation. These edges contain properties including:

    • Start of the hour
    • End of the hour
    • Total vehicle count for the hour
    • A map of vehicle type to count for the hour
  5. Deploy the Federated Demo

    develop

    To build and run the Federated Demo, ensure you have Java 8, Maven, and Git installed. The process involves cloning the Gaffer repository and running a provided start script which handles Maven dependencies like Tomcat.

    Once deployed, the REST API will be available at localhost:8080/rest.

    # Clone the Gaffer repository (master branch, depth 1)
    git clone --depth 1 --branch master https://github.com/gchq/Gaffer.git
    cd Gaffer
    
    # Run the start script to download dependencies and start the demo
    ./example/federated-demo/scripts/start.sh
  6. Run the Federated Demo - Road Use scripts

    develop

    The road-use directory contains examples of operations and scripts for generating graphs. You can use the provided JSON files to manually test operations via Swagger or curl. To automatically generate three graphs and populate them with data, run the createGraphs.sh script from within the road-use directory.

    ./scripts/createGraphs.sh
  7. Add Gaffer to a Maven project

    develop

    Gaffer is hosted on Maven Central. To use the Java API, you must include the core Gaffer graph module and a specific store module corresponding to your chosen database technology.

    For example, to use the Accumulo store, add the following dependencies to your pom.xml:

    <dependency>
        <groupId>uk.gov.gchq.gaffer</groupId>
        <artifactId>graph</artifactId>
        <version>${gaffer.version}</version>
    </dependency>
    <dependency>
        <groupId>uk.gov.gchq.gaffer</groupId>
        <artifactId>accumulo-store</artifactId>
        <version>${gaffer.version}</version>
    </dependency>