Amazon Neptune Tools

DEPRECATION NOTICE

glue-neptune is deprecated. Users should migrate to neptune-python-utils.

Purpose

glue-neptune is a Python library designed for AWS Glue jobs to facilitate writing data to Amazon Neptune. It automates several steps in the data ingestion pipeline for property graphs.

Key Features

• Catalog Integration: Retrieve Neptune connection information directly from the Glue Data Catalog.
• Schema Formatting: Automatically create label, node ID, and edge ID columns in Glue DynamicFrames. These columns are named to match the requirements of the Neptune CSV bulk load format for property graphs.
• Direct Writing: Enables writing data from Glue DynamicFrames directly to Amazon Neptune.

neptune-python-utils is a Python 3 library designed to simplify using gremlinpython to connect to Amazon Neptune. It provides specialized support for IAM DB Authentication, sessioned Gremlin interactions, and data ingestion from AWS Glue jobs.

Key capabilities include:

• Connecting via IAM DB Authentication.
• Triggering and monitoring Neptune bulk load operations.
• Creating sessioned clients for implicit transactions across multiple requests.
• Retrieving Neptune connection info from the Glue Data Catalog.
• Preparing AWS Glue DynamicFrames for Neptune CSV bulk load format (including label, node, and edge ID columns).
• Writing data from DynamicFrames directly to Neptune.
• Performing batch insert and upsert operations, including support for updating single cardinality properties, updating all properties, or replacing all properties during upserts.

This solution allows you to index existing data from an Amazon Neptune database into Amazon ElasticSearch (or OpenSearch) before enabling Neptune's native full-text search integration. This is useful for populating an index with historical data.

Workflow:

1. An AWS Lambda function triggers the process.
2. AWS Batch executes neptune-export, which pulls data from Neptune and publishes it to an Amazon Kinesis Data Stream in the Neptune Streams format.
3. A second Lambda function polls the Kinesis Stream and publishes the records to your ElasticSearch/OpenSearch cluster.

Once the existing data is indexed, you can remove this solution from your account.

This utility converts source CSV files into the Amazon Neptune Gremlin load data format required for bulk ingestion via the Neptune loader endpoint. It uses a configuration-driven approach where a JSON file defines how to map CSV rows to graph nodes, edges, and properties.

Key features include:

• Support for multiple configuration files.
• Ability to select nodes/edges from multiple source files.
• Support for both local file systems and Amazon S3 as source/destination.
• Automatic duplicate detection based on a uniqueKey.
• Support for multiple encodings.

Since Amazon Neptune does not have native partitioning, neptune-gremlin implements a partition strategy using Gremlin's partition strategy concept. By calling connection.setPartition(partitionName), the library automatically adds a _partition property to every node and edge created or queried using that connection. This allows you to segment your graph into logical sub-graphs.

connection.setPartition("test_partition")

There are three primary patterns for migrating data from Neo4j to Amazon Neptune using the neo4j-to-neptune tools:

1. Manual Multi-Step: Export CSV from Neo4j using APOC $\rightarrow$ Convert CSV using convert-csv $\rightarrow$ Manually trigger Neptune Bulk Load API.
2. Local Convert then Bulk Load: Export CSV from Neo4j using APOC $\rightarrow$ Use convert-csv with bulk load parameters to automatically convert, upload to S3, and initiate the Neptune load.
3. Automatically Stream and Convert then Bulk Load: Stream data directly from Neo4j using convert-csv (via URI/credentials) $\rightarrow$ Automatically convert, upload to S3, and initiate the Neptune load.

The conversion utility follows a specific sequence to ensure mappings and transformations are applied correctly:

1. Label Mapping: Applies vertexLabels and edgeLabels mappings.
2. Filtering: Applies skipVertices and skipEdges rules.
3. ID Transformation: Transforms the remaining IDs using the provided templates.
   • Pass 1: Processes vertices and builds the ID mapping.
   • Pass 2: Processes edges using the vertex ID mapping established in Pass 1.

The tool expects specific header columns based on the data type:

Vertex CSV Files

• ~id: Required. If missing, an error is generated.
• ~label: Expected. If missing, the tool defaults to the label 'vertex'.
• Repeating IDs: If the same ~id appears on multiple rows, the tool creates Gremlin property steps with Set cardinality. This allows building up properties or adding new ones across multiple rows.

Edge CSV Files

• ~id: Required.
• ~label: Required.
• ~from: Required.
• ~to: Required.
• Note: If only one of ~from or ~to is present, the tool will generate an error. Unlike vertices, if an ID appears more than once in an edge file, the tool will generate an error because Set cardinality is not supported for edges.

You can remap Neo4j vertex and edge labels to new names in the Neptune output.

Vertex Labels

• Neo4j vertices can have multiple labels (e.g., Person:Employee).
• Each label is mapped independently if a mapping exists.
• Unmapped labels are kept as-is.
• The output uses semicolons (;) as separators for multiple labels.

Edge Labels

• Neo4j relationships have a single type/label.
• The entire label is mapped if a mapping exists.
• Unmapped labels are kept as-is.

Note: Label mappings are applied before ID transformations. Matching is case-sensitive.

vertexLabels:
  Person: Individual
  Company: Organization

edgeLabels:
  WORKS_FOR: EMPLOYED_BY
  KNOWS: CONNECTED_TO