LoggerJSON

repository·master·Indexed 43 days ago

https://github.com/nebo15/logger_json

A collection of Elixir formatters for producing structured JSON logs compatible with cloud observability platforms such as Google Cloud, Datadog, and ElasticSearch (ECS). It includes specialized formatters like LoggerJSON.Formatters.Basic, GoogleCloud, Datadog, and Elastic, as well as utilities for Ecto telemetry logging and Plug connection metadata extraction.

Tokens
5.6K
Snippets
19
Records
27
Agent score
45%

What's inside LoggerJSON

  1. Supported formatters overview

    master

    LoggerJSON provides several specialized formatters for different platforms:

    • LoggerJSON.Formatters.Basic: A generic structured JSON format suitable for any system.
    • LoggerJSON.Formatters.GoogleCloud: Formats logs for Google Cloud Logger and Google Cloud Error Reporter.
    • LoggerJSON.Formatters.Datadog: Formats logs following Datadog's standard attribute naming conventions.
    • LoggerJSON.Formatters.Elastic: Formats logs conforming to the Elastic Common Schema (ECS) for use with ElasticSearch, LogStash, etc.
  2. Elastic Common Schema (ECS) formatting details

    master

    The LoggerJSON.Formatters.Elastic formatter ensures logs conform to ECS.

    • Errors: When an exception is thrown, the message field is populated with the error message, and error.message, error.stack_trace, and error.type fields are automatically set.
    • Custom Exceptions: If you define a custom exception type with id and/or code fields, these will be mapped to error.id and error.code respectively.
    • Custom Metadata: Any custom metadata added via Logger.metadata/1 or passed directly to the log call is added to the root of the JSON object, allowing you to populate additional ECS fields.
  3. Enable LoggerJSON formatters

    master

    You can enable a formatter in three different ways depending on your configuration needs:

    1. In runtime.exs: Best for environment-specific configuration.
    2. In your application code (e.g., application.ex): Best for dynamic runtime updates using :logger.update_handler_config/3.
    3. In config.exs: Best for static compile-time configuration. Note that you must use a tuple format here because new/1 is not available during compilation.
    # 1. In runtime.exs
    config :logger, :default_handler,
      formatter: LoggerJSON.Formatters.Basic.new(metadata: [:request_id])
    
    # 2. In application.ex (Runtime update)
    formatter = LoggerJSON.Formatters.Basic.new(metadata: :all)
    :logger.update_handler_config(:default, :formatter, formatter)
    
    # 3. In config.exs (Compile-time tuple format)
    config :logger, :default_handler,
      formatter: {LoggerJSON.Formatters.Basic, metadata: [:request_id]}
  4. Configure log level from environment variable

    master

    To make your logging level configurable via the LOG_LEVEL environment variable, call LoggerJSON.configure_log_level_from_env!() within your application's startup logic (e.g., in application.ex).

    # Inside application.ex
    LoggerJSON.configure_log_level_from_env!()
  5. Supported LoggerJSON formatters

    master

    LoggerJSON provides several formatters tailored for different logging ecosystems:

    • LoggerJSON.Formatters.Basic: A standard JSON formatter for any structured logging system (e.g., ElasticSearch, Logstash).
    • LoggerJSON.Formatters.GoogleCloud: Formatter for Google Cloud Logger and Google Cloud Error Reporter.
    • LoggerJSON.Formatters.Datadog: Formatter for Datadog.
    • LoggerJSON.Formatters.Elastic: Formatter conforming to the Elastic Common Schema (ECS) for ElasticSearch, LogStash, FileBeat, and Kibana.
  6. Configure dynamic log levels in LoggerJSON.Plug

    master

    If you need to decide whether to log a request based on its content (e.g., logging only errors or specific paths), you can provide a dynamic level handler.

    Pass a tuple {Module, function, arguments} as the third argument to LoggerJSON.Plug.attach/3. The handler will call Module.function(conn, arguments_list) where conn is automatically prepended to the arguments.

    Return a valid Logger.level() (like :info, :warn, :error) to log, or false to skip logging.

    defmodule MyLoggerLevel do
      def should_log(conn, "/health_check")
        false # Do not log health checks
      end
    
      def should_log(_conn, _)
        :info # Log everything else at :info level
      end
    end
    
    # Attaching with the dynamic handler
    LoggerJSON.Plug.attach("dynamic-logger", [:myapp, :plug, :stop], {MyLoggerLevel, :should_log, []})
  7. Virtual metadata keys in Basic formatter

    master

    The Basic formatter can automatically compute "virtual" metadata keys from the :mfa (Module, Function, Arity) entry in your log metadata. This mirrors the behavior of Elixir's built-in Logger.Formatter.

    Supported virtual keys:

    • :module: Extracted from the module in the :mfa tuple.
    • :function: Extracted as a string in the format "function_name/arity".
    • :initial_call: Extracted using Exception.format_mfa/3.

    These keys are automatically included if they are not already present in the metadata and if the :metadata configuration allows for them.

  8. Configure JSON encoder

    master

    By default, LoggerJSON uses Jason as the JSON encoder. If you are using Elixir 1.18 or later, you can switch to the built-in JSON module by setting the :encoder option in your config.exs. This must be done at compile-time.

    config :logger_json, encoder: JSON
  9. Datadog formatter output structure

    master

    The Datadog formatter outputs a JSON object containing several top-level keys designed for Datadog Log Management. Key sections include:

    • syslog: Contains hostname, severity, timestamp, and optionally env.
    • logger: Contains execution context like file_name, line, method_name, and thread_name.
    • message: The actual log message (can be a string or a structured map/keyword list).
    • error: If the log level is within :reported_levels, an error object is included containing kind, message, and stack.
    • dd.span_id and dd.trace_id: Automatically mapped from OpenTelemetry (otel_span_id, otel_trace_id) or native Datadog fields to enable log-trace correlation.
    • http: If a Plug.Conn is present in the metadata, detailed HTTP request information (URL, status, method, etc.) is included.
    {
      "domain" => ["elixir"],
      "logger" => {
        "file_name" => "/path/to/file.exs",
        "line" => 44,
        "method_name" => "Module.function/arity",
        "thread_name" => "#PID<0.234.0>"
      },
      "message" => "Hello",
      "syslog" => {
        "hostname" => "MacBook-Pro",
        "severity" => "notice",
        "timestamp" => "2024-04-11T23:03:39.726Z"
      }
    }
  10. Log-trace correlation with OpenTelemetry

    master
    The Datadog formatter automatically handles the conversion of OpenTelemetry trace and span IDs to the format expected by Datadog. It looks for otel_span_id and otel_trace_id in the log metadata and maps them to dd.span_id and dd.trace_id respectively. This ensures that logs are correctly correlated with traces in the Datadog UI.
  11. Automatic HTTP request logging with Plug.Conn

    master

    If Plug.Conn is loaded in your application, the Basic formatter automatically detects if a :conn key (containing a %Plug.Conn{} struct) is present in the log metadata. If found, it extracts and adds a request key to the JSON output containing:

    • connection: protocol, method, path, and status.
    • client: user_agent and ip.