DOCSEARCHbyAlgolia
Submit Docs

Eliot Python Logging Library

repository·master·Indexed 22 days ago

https://github.com/itamarst/eliot

A Python logging library focused on causal tracing. Instead of isolated events, Eliot records nested actions and tasks to create a tree of events that explain the 'why' behind application behavior and errors. It features automatic context propagation for asyncio and Trio, a @log_call decorator for function logging, and the ability to route standard Python logging via EliotHandler.

Tokens
16K
Snippets
59
Records
77
Agent score
77%

What's inside Eliot

  1. Supported environments and use cases for Eliot

    master

    Eliot is versatile and supports several advanced use cases and runtimes:

    • Distributed Systems: Causal tracing across multiple services.
    • Single Process: Standard logging within a single application.
    • Scientific Computing: Built-in support for NumPy and Dask.
    • Asynchronous Programming: Support for Asyncio and Trio coroutines.
    • Networking Frameworks: Support for Twisted.

    Compatibility:

    • Python 3.9, 3.8, 3.7, 3.6, and PyPy3.
    • Python 2.7 (Legacy support mode).

  2. What is Eliot and how does it work?

    master

    Eliot is a Python logging system designed to capture causal chains of actions rather than just a stream of disconnected facts. Unlike standard logging, Eliot tracks how actions spawn other actions and whether they eventually succeed or fail. This allows you to reconstruct the story of your software's execution to understand what happened and, crucially, what caused it.

    Key capabilities include:

    • Pinpointing performance bottlenecks.
    • Understanding code path selection.
    • Tracing the cause of errors.
    • Causal tracing across distributed systems.
    • Support for asynchronous frameworks like asyncio, Trio, and Twisted.
    • Support for scientific computing with NumPy and Dask.

  3. Use eliot.twisted.inline_callbacks for action context preservation

    master

    When using Twisted's inlineCallbacks with Eliot actions, the standard twisted.internet.defer.inlineCallbacks decorator will cause the action context to be lost during yield points. This results in logs following a yield being recorded without their parent action.

    To fix this, substitute twisted.internet.defer.inlineCallbacks with eliot.twisted.inline_callbacks. This version preserves the generator's action context, ensuring that code resumed after a yield is still considered a child of the active action.

    from eliot import start_action
from eliot.twisted import inline_callbacks

@inline_callbacks  # Use the eliot version, NOT twisted.internet.defer.inlineCallbacks
def go():
    with start_action(action_type=u"yourapp:subsystem:frob") as action:
        d = some_deferred_api()
        x = yield d
        # This log will correctly be a child of the action
        action.log(message_type=u"some-report", x=x)

  4. How Eliot integrates with journald

    master

    When using the journald output, Eliot maps its structured data to native systemd journal fields as follows:

    • MESSAGE: Stores the Eliot message as a JSON object.
    • ELIOT_TASK: Stores the task UUID.
    • ELIOT_TYPE: Stores the message or action type (if available).
    • SYSLOG_IDENTIFIER: Stores the value of sys.argv[0].
    • Priorities:
      • Failed actions are assigned priority 3 (err).
      • Tracebacks are assigned priority 2 (crit).

  5. How actions and tasks work in Eliot

    master

    Eliot uses the concept of actions to provide a narrative of what happened in your code. An action represents a logical operation that has a start and a finish (either success or failure).

    Key Concepts:

    • Actions: Higher-level constructs than individual log messages. They can be nested, creating a tree structure where an action's parent is deduced from the Python call stack or explicit context managers.
    • Tasks: A top-level action with no parent. Tasks are the roots of the action forest. Every log message is part of a task.
    • Nesting: When you create a new action inside a with start_action(...) block, the new action automatically becomes a child of the current action.

    By tracing these actions, you can see a causal narrative of your application's execution.

    from eliot import start_action

with start_action(action_type="store_data"):
    x = get_data()
    store_data(x)

  6. Ensure message uniqueness for serialized task identifiers

    master

    When using serialized task identifiers for cross-process tracing, follow these best practices to avoid causality errors:

    • Use once: A serialized task identifier should be used at most once. If an operation is retried, call serialize_task_id() again to generate a fresh identifier. Using the same ID for multiple attempts can result in duplicate task_uuid and task_level values, making the log tree difficult to interpret.
    • Alternative (New Task): If you cannot guarantee uniqueness, start a completely new Eliot task upon receiving a remote request, but explicitly log the original remote task identifier as a field. This allows for manual or automated reconstruction of the relationship.
    • Alternative (Process Identity): If the same identifier must be sent to multiple processes, include a unique process or thread identifier in every log message to distinguish them.

  7. Understand Eliot log message structure (task_uuid and task_level)

    master

    Eliot output is typically a list of dictionaries (e.g., JSON). To reconstruct the tree of actions from a flat list, look for these two special fields:

    • task_uuid: The unique identifier for the top-level task. All messages belonging to the same task share this ID.
    • task_level: A list of integers representing the position in the tree. For example, [3, 2, 4] means the message is the 4th child of the 2nd child of the 3rd child of the task.

    Sorting messages by task_level allows you to visualize the hierarchical relationship between actions and messages.

  8. Identify performance bottlenecks using Eliot traces

    master

    Unlike standard profilers that only show which functions are slow, Eliot logs the inputs to functions. This allows you to correlate execution time with specific input values (e.g., finding that a function is only slow when a specific parameter is passed). You can use eliot-tree combined with grep to inspect the duration of specific calls.

    # Example: finding slow 'double' calls in a log
$ eliot-tree out.log | grep -A1 "double.*started"

  9. Choosing meaningful log levels

    master

    Since Eliot allows arbitrary fields for log levels, avoid generic levels like INFO or WARN if they don't provide enough context. Instead, choose levels that reflect the operational urgency or environment.

    Examples of meaningful levels include:

    • for test environment
    • for production environment
    • investigate tomorrow
    • wake me in the middle of the night

    When implementing services, consider choosing levels that are meaningful at an organizational level to improve the utility of your logs.

  11. Understand the structure of Eliot message fields

    master

    Eliot messages are typically serialized to JSON objects. When creating custom fields, ensure they adhere to these constraints:

    • Field Names: Must be of type str.
    • Field Values: Must be JSON-compatible. Supported types are int, float, None, str, dict, or list. Nested dictionaries and lists must only contain these supported types.

  12. Process Eliot logs using Logstash and Elasticsearch

    master

    You can use the ELK stack (Elasticsearch, Logstash, and Kibana) to store, process, and visualize Eliot logs.

    1. Logstash acts as the processing tool to load Eliot log files into Elasticsearch.
    2. Elasticsearch serves as the search and analytics engine for storing the logs.
    3. Kibana provides a web UI for humans to browse the logs.

    This workflow assumes Eliot messages are written as JSON objects, one per line (the default behavior for eliot.to_file() and eliot.logwriter.ThreadedFileWriter).