aws-glue-libs

The aws-glue-libs repository provides Python libraries for the local development of AWS Glue PySpark batch jobs. It includes the awsglue library, which extends Apache Spark with additional data types and operations specifically for Glue ETL workflows.

Note: This repository is for batch jobs. For Glue streaming development, use the aws-glue-streaming-libs repository.

AWS Glue scripts typically follow a pattern of initializing a GlueContext and using it to create DynamicFrame objects.

• GlueContext: Extends PySpark's SQLContext to provide Glue-specific operations. It serves as the entry point for Glue programs.
• DynamicFrame: The core distributed data structure in Glue. Unlike Spark DataFrames, DynamicFrames handle inconsistent schemas more flexibly by representing records in a self-describing way, removing the need for upfront schema definitions or costly inference steps.

You can convert a DynamicFrame to a standard Spark DataFrame using the toDF method to leverage existing Spark SQL operations.

# Conceptual workflow
glueContext = GlueContext(...)
dynamic_frame = glueContext.create_dynamic_frame.from_options(...)
data_frame = dynamic_frame.toDF()

The awsglue package is designed to run on the AWS Glue service and is not executable independently. However, you can install it locally to enable auto-completion and linting in your IDE. To do this, you must first install pyspark via pip:

pip install pyspark

After extracting your Spark distribution, set the SPARK_HOME environment variable. Replace /home/$USER/ with your actual installation path.

# Glue version 2.0
export SPARK_HOME=/home/$USER/spark-2.4.3-bin-hadoop2.8

# Glue version 3.0
export SPARK_HOME=/home/$USER/spark-3.1.1-amzn-0-bin-3.2.1-amzn-3

# Glue version 4.0
export SPARK_HOME=/home/$USER/spark-3.3.0-amzn-1-bin-3.3.3-amzn-0

# Glue version 5.0
export SPARK_HOME=/home/$USER/spark-3.5.4-bin-hadoop3

# Glue version 5.1
export SPARK_HOME=/home/$USER/spark-3.5.6-bin-hadoop3

The awsglue library provides various operations in the transforms module to manipulate DynamicFrames. These range from simple operations like DropFields to complex transformations like Relationalize (which flattens nested datasets into relational tables).

Once a transform is imported, it is invoked using the .apply() method.

# Syntax pattern for transforms
TransformClass.apply(args...)

When setting up your local environment, ensure you use the correct Python version and the corresponding repository branch for your target AWS Glue version:

Once your environment is configured, use the executables in the bin directory to interact with Glue:

• Glue PySpark Shell: Start an interactive PySpark shell for Glue.
• Glue Spark Submit: Submit a Glue Spark application.
• Glue Pytest: Run tests using pytest. (Note: This requires the pytest module to be installed and available in your PATH).

# Glue shell
./bin/gluepyspark

# Glue submit
./bin/gluesparksubmit

# pytest
./bin/gluepytest

The gluepyspark executable launches an interactive PySpark shell that has been pre-configured with AWS Glue extensions. It works by sourcing the Glue environment setup (glue-setup.sh) before executing the standard PySpark binary. You can pass any standard PySpark arguments or flags directly to this command.

To use it, ensure your environment is set up and call the script from your terminal.

./bin/gluepyspark

# You can also pass standard PySpark arguments
./bin/gluepyspark --version

The gluepytest CLI tool is a wrapper around pytest designed to run tests for Spark code that requires the AWS Glue environment. It automatically sources the necessary Glue environment setup via glue-setup.sh before executing pytest with any arguments you provide.

# Usage: pass any standard pytest arguments to the command
./bin/gluepytest tests/my_glue_test.py --verbose

The gluesparksubmit command is a wrapper around spark-submit designed to run AWS Glue Spark jobs in a local environment. It automatically sources the necessary Glue environment configurations via glue-setup.sh and includes required Glue Python libraries in the Spark classpath using the --py-files flag.

To use it, ensure you have already performed the local setup (e.g., via glue-setup.sh) so that the SPARK_HOME and GLUE_PY_FILES environment variables are correctly populated. You can then pass any standard spark-submit arguments directly to this command.

# Usage pattern:
./bin/gluesparksubmit [spark-submit-arguments] <your-python-script.py> [script-arguments]

# Example:
./bin/gluesparksubmit --master local[*] my_glue_job.py --arg1 value1