DOCSEARCHbyAlgolia
Submit Docs

AWS SDK for pandas (awswrangler)

repository·main·Indexed 26 days ago

https://github.com/aws/aws-sdk-pandas

The AWS SDK for pandas, also known as awswrangler, provides integration between pandas and AWS services including S3, Athena, Glue, Redshift, and Timestream. It simplifies data lake operations and AWS data workflows, offering support for distributed computing via Modin and Ray, and providing utilities for converting between PyArrow, Athena, and Pandas data types.

Tokens
55.3K
Snippets
207
Records
327
Agent score
87%

What's inside aws-sdk-pandas

  1. Enable distributed mode via lazy Ray initialization

    main

    AWS SDK for pandas (awswrangler) supports distributed mode using the Ray engine. Instead of initializing Ray automatically at import, the engine is now initialized lazily upon the first distributed API call.

    To control whether distributed mode is used, you can:

    1. Switch the engine or memory format using environment variables.
    2. Use wr.engine.set to explicitly configure the engine before making your first distributed API call.

    Note: Because initialization is lazy, running tests across multiple threads may result in each thread initializing its own Ray runtime.

  2. Access AWS SDK for pandas tutorial notebooks

    main

    Comprehensive tutorial notebooks for AWS SDK for pandas (awswrangler) are available on the official GitHub repository. These notebooks provide practical examples and guided walkthroughs for various data engineering tasks.

    https://github.com/aws/aws-sdk-pandas/tree/main/tutorials

  3. Use TypedDict for grouped parameters in AWS SDK for pandas

    main

    To reduce function signature complexity and improve clarity, AWS SDK for pandas groups related parameters into TypedDict objects. Instead of passing dozens of individual arguments, you can pass a single dictionary or a typed object to a parameter.

    These grouped parameters are available in the wrangler.typing module. Common examples include:

    • GlueCatalogParameters
    • AthenaCacheSettings
    • AthenaUNLOADSettings
    • AthenaCTASSettings
    • RaySettings

    You can provide these settings in two ways: as a standard Python dictionary with string keys, or by instantiating the specific TypedDict class from wrangler.typing.

    # Option 1: Using a standard dictionary
wr.athena.read_sql_query(
    "SELECT * FROM ...",
    ctas_approach=True,
    athena_cache_settings={"max_cache_seconds": 900},
)

# Option 2: Using the TypedDict class from wrangler.typing
wr.athena.read_sql_query(
    "SELECT * FROM ...",
    ctas_approach=True,
    athena_cache_settings=wr.typing.AthenaCacheSettings(
        max_cache_seconds=900,
    ),
)

  4. Optimize CSV and JSON I/O performance using PyArrow

    main

    AWS SDK for pandas automatically switches between PyArrow and Pandas engines for CSV and JSON I/O to balance performance and feature parity.

    When using distributed modes like Ray or Modin, the library will automatically use PyArrow if the parameters you provide are supported by PyArrow. This results in significantly faster performance (e.g., ~30s vs ~120s for 5 GiB of CSV data).

    If you provide parameters that PyArrow does not support (such as comment in CSV reading), the library will fallback to the slower Pandas engine to ensure all requested features work correctly.

    This logic applies to:

    • wr.s3.read_csv
    • wr.s3.read_json
    • wr.s3.to_json
    • wr.s3.to_csv
    # This will be loaded by PyArrow (faster), as `doublequote` is supported
wr.s3.read_csv(
    path="s3://my-bucket/my-path/",
    dataset=True,
    doublequote=False,
)

# This will be loaded using the Pandas I/O functions (slower), 
# as `comment` is not supported by PyArrow
wr.s3.read_csv(
    path="s3://my-bucket/my-path/",
    dataset=True,
    comment="#",
)

  5. Install AWS SDK for pandas with distributed mode support

    main

    To enable distributed processing using Ray and Modin, install the library with the following optional dependencies:

    pip install "awswrangler[ray,modin]"

    Once installed, awswrangler will automatically detect ray and modin at import and enable distributed mode.

    pip install "awswrangler[ray,modin]"

  6. Use AWS Lambda Managed Layers for AWS SDK for pandas

    main

    AWS SDK for pandas provides managed Lambda layers to simplify deployment in AWS Lambda functions. These layers include the necessary dependencies for the library to run efficiently in a serverless environment.

    To use them, select the ARN that matches your specific requirements for:

    1. Region: The AWS region where your Lambda function resides (e.g., us-east-1, eu-central-1).
    2. Python Version: The runtime version of your Lambda function (e.g., 3.10, 3.11, 3.12, 3.13, 3.14).
    3. Architecture: The CPU architecture of your Lambda function (x86_64 or arm64).

    Refer to the table below for available ARNs in specific regions.

  7. Configure AWS SDK for pandas in SageMaker Notebook Lifecycle

    main

    To automatically install awswrangler in all compatible SageMaker conda environments upon instance start, use the following bash script in your Lifecycle Configuration:

    #!/bin/bash

set -e

sudo -u ec2-user -i <<'EOF'

# PARAMETERS
PACKAGE=awswrangler

# Note that "base" is special environment name, include it there as well.
for env in base /home/ec2-user/anaconda3/envs/*; do
    source /home/ec2-user/anaconda3/bin/activate $(basename "$env")
    if [ "$env" = 'JupyterSystemEnv' ]; then
        continue
    fi
    nohup pip install --upgrade "$PACKAGE" &
    source /home/ec2-user/anaconda3/bin/deactivate
done
EOF
    #!/bin/bash

set -e

sudo -u ec2-user -i <<'EOF'

# PARAMETERS
PACKAGE=awswrangler

# Note that "base" is special environment name, include it there as well.
for env in base /home/ec2-user/anaconda3/envs/*; do
    source /home/ec2-user/anaconda3/bin/activate $(basename "$env")
    if [ "$env" = 'JupyterSystemEnv' ]; then
        continue
    fi
    nohup pip install --upgrade "$PACKAGE" &
    source /home/ec2-user/anaconda3/bin/deactivate
done
EOF

  8. Use Modin for distributed data processing

    main

    In distributed mode, awswrangler APIs return and accept Modin data frames instead of standard Pandas data frames. To ensure all operations leverage the entire cluster/machine resources instead of running on a single thread, you must replace your pandas import with modin:

    import modin.pandas as pd  # instead of import pandas as pd
    import modin.pandas as pd
import awswrangler as wr

# Example: Reading and processing large datasets
df = wr.s3.read_parquet(path="s3://ursa-labs-taxi-data/2017/")
df.drop("vendor_id", axis=1, inplace=True)
df1 = df[df["trip_distance"] > 1]

  9. Use AWS SDK for pandas as a Managed Lambda Layer

    main

    AWS SDK for pandas is available as a Managed Layer in all AWS commercial regions. You can access it via the AWS Lambda console or by using its ARN.

    ARN Format: arn:aws:lambda:<region>:336392948345:layer:AWSSDKPandas-Python<python-version>:<layer-version>

    Example ARN (Python 3.8): arn:aws:lambda:us-east-1:336392948345:layer:AWSSDKPandas-Python38:1

    Note:

    • There is a minimum one-week delay between a version release and its availability in the Lambda console.
    • Lambda functions with less than 512MB of memory may be insufficient for some workloads.

    Retrieve ARNs via CLI:

    aws ssm describe-parameters --parameter-filters "Key=Name, Option=BeginsWith, Values=/aws/service/aws-sdk-pandas/3.4.0/"

  10. Configure the execution engine and memory format

    main

    AWS SDK for pandas (awswrangler) allows you to switch between different execution engines and memory formats to support distributed computing.

    By default, the engine is set to python and the memory_format is set to pandas. The engine is automatically determined at import based on installed dependencies (e.g., ray may be selected if available).

    You can manually override these settings using the wr.engine.set() and wr.memory_format.set() methods.

  12. Quick Start with AWS SDK for pandas

    main

    This guide demonstrates common tasks including storing data to an S3 Data Lake, reading from S3 and Athena, interacting with Redshift Spectrum, and writing/querying Amazon Timestream.

    import awswrangler as wr
import pandas as pd
from datetime import datetime

df = pd.DataFrame({"id": [1, 2], "value": ["foo", "boo"]})

# Storing data on Data Lake
wr.s3.to_parquet(
    df=df,
    path="s3://bucket/dataset/",
    dataset=True,
    database="my_db",
    table="my_table"
)

# Retrieving the data directly from Amazon S3
df = wr.s3.read_parquet("s3://bucket/dataset/", dataset=True)

# Retrieving the data from Amazon Athena
df = wr.athena.read_sql_query("SELECT * FROM my_table", database="my_db")

# Get a Redshift connection from Glue Catalog and retrieving data from Redshift Spectrum
con = wr.redshift.connect("my-glue-connection")
df = wr.redshift.read_sql_query("SELECT * FROM external_schema.my_table", con=con)
con.close()

# Amazon Timestream Write
df = pd.DataFrame({
    "time": [datetime.now(), datetime.now()],   
    "my_dimension": ["foo", "boo"],
    "measure": [1.0, 1.1],
})
rejected_records = wr.timestream.write(df,
    database="sampleDB",
    table="sampleTable",
    time_col="time",
    measure_col="measure",
    dimensions_cols=["my_dimension"],
)

# Amazon Timestream Query
wr.timestream.query("""
SELECT time, measure_value::double, my_dimension
FROM "sampleDB"."sampleTable" ORDER BY time DESC LIMIT 3
""")