DOCSEARCHbyAlgolia
Submit Docs

AWS Embedded Metrics Python

repository·master·Indexed 42 days ago

https://github.com/awslabs/aws-embedded-metrics-python

A Python library for generating CloudWatch Metrics embedded within structured log events using the Embedded Metric Format (EMF). It allows developers to monitor real-time metrics while preserving high-cardinality context for analysis in CloudWatch Logs Insights. Features include the @metric_scope decorator, MetricsLogger for publishing metrics, and support for various deployment environments including AWS Lambda, EC2, ECS, Fargate, and Docker.

Tokens
3.5K
Snippets
13
Records
17
Agent score
52%

What's inside AWS Embedded Metrics Python

  1. Configure generator flushing behavior with metric_scope

    master

    The @metric_scope decorator supports both sync and async generators. You can control when metrics are flushed during generator execution using the flush_on_yield parameter:

    • v3.x behavior: Metrics are flushed after each yield by default.
    • v4.0+ behavior: Metrics are flushed only at completion by default (for better performance).

    To explicitly set this behavior, use @metric_scope(flush_on_yield=True|False).

    @metric_scope(flush_on_yield=False)
def my_generator(metrics):
    yield 1
    metrics.put_metric("Count", 1, "Count")
    yield 2

  5. Deploy the Lambda Function Example

    master

    To deploy the provided Lambda function example, follow these steps:

    1. Customize the function.py file in the example directory to suit your requirements.
    2. Create a Lambda function in your AWS account.
    3. Execute the deploy.sh script, passing the Lambda function's ARN and the target AWS region as arguments.

    The script expects the format: ./deploy.sh <LAMBDA_FUNCTION_ARN> <REGION>.

    ./deploy.sh arn:aws:lambda:us-east-1:123456789012:function:MyFunction us-east-1

  6. Use metric_scope to log metrics and properties

    master

    The easiest way to use the library is to decorate your handler function with @metric_scope. This decorator injects a metrics object (an instance of MetricsLogger) as the first argument to your function.

    You can use this object to:

    • Add dimensions using put_dimensions(dict).
    • Record metrics with put_metric(name, value, unit, resolution).
    • Attach high-cardinality context/metadata using set_property(key, value).

    This approach allows you to monitor aggregated values in CloudWatch while retaining detailed event context in CloudWatch Logs Insights.

    from aws_embedded_metrics import metric_scope
from aws_embedded_metrics.storage_resolution import StorageResolution

@metric_scope
def my_handler(metrics):
    metrics.put_dimensions({"Foo": "Bar"})
    metrics.put_metric("ProcessingLatency", 100, "Milliseconds", StorageResolution.STANDARD)
    metrics.put_metric("Memory.HeapUsed", 1600424.0, "Bytes", StorageResolution.HIGH)
    metrics.set_property("AccountId", "123456789012")
    metrics.set_property("RequestId", "422b1569-16f6-4a03")
    metrics.set_property("DeviceId", "61270781-c6ac-46f1")

    return {"message": "Hello!"}

  8. Run the EC2 emulation example locally

    master

    To emulate a remote EC2 deployment and test the library's auto-configuration, you can run the provided EC2 example. This process requires Docker and valid AWS credentials. The script performs the following actions:

    1. Starts a local EC2 metadata endpoint on 3000/tcp (used by the library for auto-configuration).
    2. Starts the CloudWatch Agent.
    3. Sets up a virtual environment, installs the library, and executes the app.py script.

    Ensure you have Docker installed and your AWS credentials configured in your environment before running the script.

    export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export AWS_REGION=

./run-example.sh

  10. Configure aws-embedded-metrics in-process

    master

    You can configure the library programmatically using the get_config() function from aws_embedded_metrics.config.

    from aws_embedded_metrics.config import get_config
Config = get_config()

Config.service_name = "MyApp"
Config.service_type = "NodeJSWebApp"
Config.log_group_name = "LogGroupName"
Config.log_stream_name = "LogStreamName"
Config.namespace = "MyApplication"
Config.disable_metric_extraction = True

  11. Configure aws-embedded-metrics via Environment Variables

    master

    All configuration can be set using environment variables with the AWS_EMF_ prefix. This is recommended to be done as close to application startup as possible.

    Environment VariableDescriptionRequirements
    AWS_EMF_SERVICE_NAMEOverrides the service name.1-255 ASCII chars
    AWS_EMF_SERVICE_TYPEOverrides the service type.1-255 ASCII chars
    AWS_EMF_LOG_GROUP_NAMEDestination Log Group (for agent-based platforms).1-512 chars, pattern [\.\-_/#A-Za-z0-9]+
    AWS_EMF_LOG_STREAM_NAMEDestination Log Stream (for agent-based platforms).1-512 chars, cannot contain : or *
    AWS_EMF_NAMESPACEOverrides the CloudWatch namespace.1-512 ASCII chars
    AWS_EMF_DISABLE_METRIC_EXTRACTIONSet to true to omit EMF metadata from logs (disables CloudWatch extraction).boolean

  12. Configure Namespace and Timestamp

    master

    Control where metrics are published and when they occurred.

    • set_namespace(value: str): Sets the CloudWatch namespace. Defaults to aws-embedded-metrics if not specified. Name must be 1-255 ASCII characters.
    • set_timestamp(timestamp: datetime): Sets the metric timestamp. If not set, the current client time is used. Invalid timestamps will trigger an InvalidTimestampError.
    set_namespace("MyApplication")
set_timestamp(datetime.datetime.now())