DOCSEARCHbyAlgolia
Submit Docs

Elastic APM Python Agent

repository·main·Indexed 19 days ago

https://github.com/elastic/apm-agent-python

The official Python implementation for Elastic APM, designed to monitor and instrument Python applications. It provides automatic instrumentation for popular web frameworks, including aiohttp, and supports manual tracking of transactions, spans, and exceptions via the Client API. Features include ECS logging formatting, distributed tracing with TraceParent, and support for Python 3.6+.

Tokens
33.1K
Snippets
109
Records
199
Agent score
65%

What's inside elastic-apm-agent-python

  1. Overview of Elastic APM Python Agent

    main
    The elastic-apm module is the official Python agent for Elastic APM. It provides automatic instrumentation and out-of-the-box support for popular web frameworks like Django and Flask. For other WSGI-compatible web applications, it can be extended using custom integrations. Additionally, the agent can be used in any non-web Python application to monitor performance and errors.

  2. Advanced topics in Elastic APM Python Agent

    main

    The Elastic APM Python Agent provides several advanced capabilities for fine-grained control over observability. Key advanced topics include:

    • Instrumenting custom code: Manually creating spans and transactions to track specific parts of your application logic.
    • Sanitizing data: Ensuring sensitive information (like passwords or PII) is removed from captured data before it is sent to the APM Server.
    • Understanding Agent internals: Deep dives into how the agent hooks into your application and manages data.
    • Local testing: Instructions for running the agent's test suite in a local environment.

  3. What is the OpenTelemetry API Bridge?

    main

    The Elastic APM OpenTelemetry bridge allows you to create Elastic APM Transactions and Spans using the OpenTelemetry API. This enables you to use Elastic APM's automatic instrumentations while keeping your custom instrumentations vendor-neutral.

    Mapping behavior:

    • If a span is created while no transaction is active, the bridge automatically creates an Elastic APM Transaction.
    • Inner spans are mapped to Elastic APM Span objects.

  4. Migrate from tags to labels (v5.0.0 breaking changes)

    main

    In version 5.0.0, the term tags was renamed to labels to align with other Elastic language agents.

    • The tags API was replaced by the labels API.
    • For backward compatibility, the old tags API remains supported until version 6.0 of the agent.

  5. Implement distributed tracing with TraceParent

    main

    To link transactions across multiple services into a single distributed trace, you must propagate the TraceParent. The Python Agent automatically adds TraceParent to outgoing HTTP request headers.

    To manually continue a trace in a new service, use the following utilities:

    1. elasticapm.get_trace_parent_header(): Retrieves the current TraceParent as a string (requires an active transaction).
    2. elasticapm.trace_parent_from_string(traceparent_string): Creates a TraceParent object from a string.
    3. elasticapm.trace_parent_from_headers(headers_dict): Creates a TraceParent object from a dictionary of headers (e.g., headers received from an incoming HTTP request).

    Pass the resulting TraceParent object to client.begin_transaction(..., trace_parent=parent).

    import elasticapm

client = elasticapm.Client(service_name="foo", server_url="https://example.com:8200")

# Retrieve the current TraceParent as a string, requires active transaction
traceparent_string = elasticapm.get_trace_parent_header()

# Create a TraceParent object from a string and use it for a new transaction
parent = elasticapm.trace_parent_from_string(traceparent_string)
client.begin_transaction(transaction_type="script", trace_parent=parent)
# Do some work
client.end_transaction(name=__name__, result="success")

# Create a TraceParent object from a dictionary of headers, provided
# automatically by the sending service if it is using an Elastic APM Agent.
parent = elasticapm.trace_parent_from_headers(headers_dict)
client.begin_transaction(transaction_type="script", trace_parent=parent)
# Do some work
client.end_transaction(name=__name__, result="success")

  6. How to sanitize data using processors

    main

    To remove sensitive data from events sent to Elastic APM, you can implement a custom processor. A processor is a function that accepts a client instance and an event (which can be an ERROR, TRANSACTION, SPAN, or METRICSET) and returns the modified event.

    Key behaviors:

    • Modify an event: Return the modified event object.
    • Drop an event: Return False (or any falsy value) to prevent the event from being sent.
    • Error handling: If a processor raises an exception, the event is dropped and a WARNING log is issued.

    Use the @for_events decorator to restrict your processor to specific event types. Available constants are ERROR, TRANSACTION, SPAN, and METRICSET from elasticapm.conf.constants.

    Example of a processor that removes exception stacktraces:

    from elasticapm.conf.constants import ERROR
from elasticapm.processors import for_events

@for_events(ERROR)
def my_processor(client, event):
    if 'exception' in event and 'stacktrace' in event['exception']:
        event['exception'].pop('stacktrace')
    return event
    from elasticapm.conf.constants import ERROR
from elasticapm.processors import for_events

@for_events(ERROR)
def my_processor(client, event):
    if 'exception' in event and 'stacktrace' in event['exception']:
        event['exception'].pop('stacktrace')
    return event

  9. Understand the Elastic APM architecture components

    main

    The Elastic APM Python agent does not operate in isolation. To function correctly, it works as part of a stack involving:

    • APM Server: Receives the data sent by the agent.
    • Elasticsearch: Stores the collected APM data.
    • Kibana: Provides the interface for visualizing and analyzing the data.

    Always check the Agent and Server compatibility matrix to ensure your versions are compatible.

  10. How the Elastic APM Python Agent collects data

    main

    The Elastic APM Python agent gathers APM events (transactions and spans), errors, and metrics using three primary mechanisms:

    1. Framework Integration: Uses framework-specific hooks and signals (e.g., Django or Flask signals) to track incoming requests and background tasks. This typically requires minor configuration changes (like adding an app to INSTALLED_APPS).
    2. Instrumentation: Automatically wraps functions in supported libraries (like database drivers or HTTP clients) using the wrapt library to capture metadata such as query strings, URLs, and execution time. This requires no code changes.
    3. Background Collection: A background thread collects system and application metrics at regular intervals.

    All collected data is sent to the APM Server, which then forwards it to Elasticsearch for visualization in Kibana.

  11. Understand the agent's background thread model

    main

    When the Python agent is instantiated, it starts three background threads per process. It is important to note that in multi-process environments (like gunicorn or uwsgi workers), each worker process will spawn its own set of these three threads.

    The three threads are:

    1. Metrics Collection Thread: Regularly collects system and application metrics.
    2. Configuration Thread: Regularly fetches remote configuration from the APM Server.
    3. Data Processing Thread: Processes collected data and sends it to the APM Server via HTTP.

  12. How log correlation works in the Elastic Python APM Agent

    main

    Log correlation allows you to navigate between logs, traces, and services. The agent injects correlation IDs into your logs so that for a specific log, you can see the transaction context and user parameters, and vice-versa.

    Important Note: The Elastic Python APM Agent does not send logs to Elasticsearch. It only injects correlation IDs and reformats logs. You must use an ingestion strategy like Filebeat to move logs to your stack.