DOCSEARCHbyAlgolia
Submit Docs

Telemere Structured Logging and Telemetry

repository·master·Indexed 43 days ago

https://github.com/taoensso/telemere

A high-performance, structured logging and telemetry library for Clojure and ClojureScript. It provides a unified API for traditional logging, structured data logging, tracing, and performance monitoring. Telemere supports advanced filtering, sampling, and rate limiting, with built-in handlers for Console, OpenTelemetry, Slack, and more. Key macros include `log!`, `trace!`, and `signal!` for generating telemetry signals.

Tokens
11K
Snippets
24
Records
56
Agent score
55%

What's inside Telemere

  1. Overview of Telemere

    master

    Telemere is a next-generation structured logging and telemetry library for Clojure and ClojureScript. It provides a single unified API to handle:

    • Traditional logging: String-based messages.
    • Structured logging: Rich Clojure data types and nested structures.
    • Tracing: Nested flow tracking with optional data (interoperable with OpenTelemetry).
    • Performance monitoring: Tracking runtimes of nested forms.

    Telemere is designed to be fast, flexible, and data-oriented, allowing you to pass rich data through the entire pipeline (filters, middleware, and handlers) without immediate serialization.

    (tel/log! {:level :info, :id ::login, :data {:user-id 1234}, :msg "User logged in!"})

  2. Explore community handlers and tools

    master

    Telemere is designed to have a stable core while allowing the community to extend it via handlers, transforms, and utilities.

    Currently available community extensions include:

    • Axiom.co handler: A handler for sending telemetry to Axiom.

    Developers are encouraged to contribute additional handlers, transforms, or analysis tools via Pull Requests.

  3. Map Timbre Appenders to Telemere Handlers

    master
    In Telemere, the concept of an 'appender' is replaced by a 'handler'. To migrate, check if your existing Timbre appenders are available as included Telemere handlers or community handlers. If not, you can write a custom handler; since signals are plain Clojure/Script maps, a handler is simply a function that processes these maps.

  4. Configure handler dispatch and specific options

    master

    Handler configuration in Telemere is split into two categories:

    1. Dispatch opts: Common to all handlers. These control how signals are routed to handlers, including dispatch priority, filtering, transformations (modifying signal maps before they reach handlers), async queue semantics, and back-pressure.
    2. Handler-specific opts: Unique to each handler constructor (e.g., handler:console). These are used to customize the behavior of a specific output target.

    It is common practice to customize handlers by passing Clojure/Script functions into their constructor calls.

  5. How signal filtering and transformation works

    master

    Telemere uses a multi-stage filtering and transformation pipeline to decide if a signal is created and if a handler processes it. A signal is only passed to a handler if it satisfies ALL of the following conditions:

    1. Call filters pass: These decide if a signal can be created. They can be evaluated at compile-time or runtime based on sample rate, kind, ns, id, level, when form, and rate limit.
    2. Handler filters pass: These decide if a specific handler is allowed to handle a signal that has already been created. They are evaluated at runtime based on sample rate, kind, ns, id, level, when fn, and rate limit.
    3. Call transform returns non-nil: A transformation function (fn [signal]) => ?modified-signal applied at the call site must return a non-nil value.
    4. Handler transform returns non-nil: A transformation function (fn [signal]) => ?modified-signal applied at the handler level must return a non-nil value.

    Key Concepts:

    • Additive Filtering: Call filters and handler filters are additive. Handlers can be more restrictive than call filters, but never less restrictive. Call filters control signal creation; handler filters control signal distribution.
    • Transformations as Filters: Transform functions provide a way to both modify signal data or filter signals by returning nil to skip handling.
    • Performance: Filtering is $O(1)$, except for rate limits which are $O(n_{windows})$.

  6. Compare Telemere and Tufte

    master

    Telemere and Tufte are complementary libraries. While Telemere provides general observability signals, Tufte provides deep performance profiling.

    FeatureTelemereTufte
    Primary FocusGeneral observability, structured logs, and tracing.Detailed performance monitoring and profiling.
    Profiling DepthBasic profiling of a single form to :runtime-nsecs.Dynamic and thread-local profiling of arbitrary nested forms with mergeable stats.
    Usage PatternConveying aggregate performance info as part of system signals.Detailed measurement of specific code paths.

    Note: Upcoming Tufte v3 will share the same core as Telemere and offer an identical API for managing filters and handlers.

  7. Telemere vs Timbre

    master

    Telemere is a superset of Timbre. While Timbre is a pure Clojure/Script logging library, Telemere offers many improvements in speed, robustness, and flexibility.

    • Migration: There is no requirement to migrate, as Timbre will continue to be maintained. However, Telemere is designed to make migration easy.
    • Relationship: Telemere's core is being developed as a shared library that will eventually provide unified capabilities to Timbre, Telemere, and Tufte.

  8. Understand the argument order of `event!`

    master

    Unlike most other signal creators in Telemere which take an options map as the first argument, event! takes the event ID as the second argument (the options map comes first).

    Example pattern: (event! {:data <long-or-multiline-data>} ::event-id)

    This design choice prioritizes ergonomics for the common case:

    1. The primary argument (the event ID) is typically a very short keyword.
    2. The event ID does not depend on :let bindings.
    3. The options (like :data) are often large or multi-line.

    By putting the ID second, the most important piece of information (what happened) is identified immediately, followed by the heavy data payload. If you prefer a single-map approach, you can use signal! directly, which takes a single map argument.

  9. Use signal `kind` for advanced taxonomy

    master
    Every signal has a kind key (e.g., log! produces signals with a :log kind). While often left as default, you can use kind to implement a custom taxonomy of signals separate from IDs and namespaces. This allows for advanced filtering and setting minimum levels based on the signal category.

  10. Configure signal levels and error handling

    master

    Telemere provides flexibility in how signal severity and errors are defined:

    • Arbitrary Levels: Signal levels are represented as integers. You can use standard keyword aliases (like :info, :warn, etc.) or define your own arbitrary integer levels.
    • Decoupled Errors: The presence of an :error value in a signal is not strictly tied to its level. A signal can have an :error value without having an :error level, and vice versa. Use the error-signal? utility to check for error states based on your specific requirements.

  11. How signal filtering works in Telemere

    master

    A signal is only passed to a handler if it satisfies four sequential stages of filtering. These filters are additive: call filters decide if a signal is created, and handler filters decide if a specific handler is allowed to process that created signal. Handlers can be more restrictive than call filters, but not less.

    For a signal to reach a handler, the following must all be true:

    1. Signal call filters pass (checked at both compile-time and runtime). Parameters include sample rate, kind, ns, id, level, when form, and rate limit.
    2. Signal handler filters pass (checked at runtime). Parameters include sample rate, kind, ns, id, level, when fn, and rate limit.
    3. Call transform (fn [signal]) => ?modified-signal returns a non-nil value.
    4. Handler transform (fn [signal]) => ?modified-signal returns a non-nil value.

    Transform functions allow you to modify or filter signals based on arbitrary data. Returning nil from a transform function effectively skips handling for that signal.

  12. How signal creators and handlers work

    master

    Telemere operates using two primary components:

    1. Signal creators: Macros used within your code to conditionally create signal maps (e.g., log!, event!, trace!).
    2. Signal handlers: Components that process created signals to perform actions like analyzing data or writing it to a console, file, queue, or database.

    This architecture allows you to separate the act of capturing telemetry from the logic of how that telemetry is processed or stored.