loki-logback-appender Documentation

repository·main·Indexed 19 days ago

https://github.com/loki4j/loki-logback-appender

A high-performance, lightweight Logback appender designed to ship logs directly to Grafana Loki. It features dynamic label generation from Logback patterns, MDC, KVP, or SLF4J markers, support for structured metadata, and compatibility with both JSON and Protobuf Loki API flavors. The appender is zero-dependency and includes performance monitoring via Micrometer.

Tokens
12.9K
Snippets
38
Records
48
Agent score
63%

What's inside loki-logback-appender

  1. Overview of loki-logback-appender

    main

    loki-logback-appender is a high-performance, lightweight Logback appender designed for Grafana Loki. It is an unofficial, community-driven project that focuses on speed and minimal overhead.

    Key capabilities include:

    • Dynamic Label Generation: Automatically creates Loki labels and metadata from Logback patterns, MDC (Mapped Diagnostic Context), KVP (Key-Value Pairs), or SLF4J markers.
    • Structured Metadata: Supports structured metadata for enhanced log context.
    • Fast JSON Layout: Optimized formatting for log messages.
    • API Support: Compatible with both JSON and Protobuf Loki API flavors, including support for Grafana Cloud.
    • Zero-dependency: Designed to be lightweight without pulling in unnecessary transitive dependencies.
    • Performance Monitoring: Includes performance metrics to track appender health.
  2. Key Features of Loki4j

    main

    Loki4j provides several advanced capabilities for log management:

    • Dynamic Label and Metadata Generation: Generate Loki labels and structured metadata from Logback patterns, MDC, KVP, or SLF4J markers. This allows precise control over the label set for each log record.
    • Structured Metadata Support: Unlike labels, structured metadata is not indexed in Loki but improves search efficiency by avoiding full message body scans.
    • Fast JSON Layout: Switch from plain text to a fast JSON layout for Logstash-compatible log message formatting.
    • API Flavor Support: Supports both JSON and Protobuf Loki API flavors (JSON is the default).
    • Grafana Cloud Compatibility: Supports HTTP basic authentication for hosted services like Grafana Cloud.
    • Zero-dependency: The library has a minimal footprint and introduces no new transitive dependencies if logback-classic is already present.
    • Performance Monitoring: Supports instrumentation via Micrometer to monitor metrics like encode/send duration and batch counts.
  3. Distinguish between Loki labels and structured metadata

    main

    When configuring Loki4jAppender, you must choose between Labels and Structured metadata based on the cardinality of your data:

    • Labels: Used by Loki for indexing to improve search performance. They must be low-cardinality (a small, relatively static set of values). Both keys and values must be plain strings.
    • Structured metadata: Introduced in Loki v2.9.0, this allows attaching high-cardinality metadata (like thread names, class names, or unique IDs) without indexing them. This avoids the performance penalties of high-cardinality labels while still allowing efficient searching.

    Best Practice: Use labels for static, low-cardinality values (e.g., app, host) and structured metadata for everything else (e.g., level, thread, class).

    <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
        <labels>
            app = my-app
            host = ${HOSTNAME}
        </labels>
        <structuredMetadata>
            level = %level
            thread = %thread
            class = %logger
        </structuredMetadata>
        <batch>
            <staticLabels>true</staticLabels>
        </batch>
    </appender>
  4. Serialize arbitrary objects in JSON layout

    main

    Loki4j uses a high-performance JSON serialization algorithm that avoids runtime reflection. Because of this, arbitrary objects attached via Logback Key-Value Pairs (KVP) are not automatically serialized into complex JSON structures by default.

    To handle complex objects, choose one of these three strategies:

    Implement com.github.loki4j.logback.json.JsonFieldSerializer<Object> and register it via the <fieldSerializer> setting in the kvp section. This allows you to use JsonEventWriter to manually build the JSON structure for specific types.

    2. Use RawJsonString

    If you use a reflection-based library (like Jackson) to pre-serialize your object into a JSON string, wrap that string in com.github.loki4j.logback.json.RawJsonString. The writer will then embed the string directly without escaping. Warning: You are responsible for ensuring the string contains valid JSON.

    3. Default writeObjectField() behavior

    If you do nothing, JsonEventWriter.writeObjectField() handles types as follows:

    • String: JSON string
    • Integer/Long: JSON number
    • Boolean: JSON boolean
    • Iterable: JSON array
    • RawJsonString: Raw JSON (no escaping)
    • Other types: The result of .toString() is rendered as a JSON string (e.g., "obj":"MyObject@12345").
    // Example: Implementing a custom field serializer
    public class TestFieldSerializer implements JsonFieldSerializer<Object> {
        @Override
        public void writeField(JsonEventWriter writer, String fieldName, Object fieldValue) {
            if (fieldValue instanceof TestJsonKvData) {
                writer.writeCustomField(fieldName, w -> {
                        var td = (TestJsonKvData)fieldValue;
                        w.writeBeginObject();
                        w.writeObjectField("userId", td.userId);
                        w.writeFieldSeparator();
                        w.writeObjectField("userName", td.userName);
                        w.writeFieldSeparator();
                        w.writeObjectField("sessionId", td.sessionId);
                        w.writeEndObject();
                    }
                );
            } else {
                writer.writeObjectField(fieldName, fieldValue);
            }
        }
    }
    <message class="com.github.loki4j.logback.JsonLayout">
        <kvp>
            <fieldSerializer class="io.my.TestFieldSerializer" />
        </kvp>
    </message>
  5. Select a compatible Protobuf version

    main

    If your project already uses a specific version of Protobuf, you must select a version of loki-protobuf that matches. The versioning follows the pattern 1.0.0_pbX.Y.0, where X.Y.0 corresponds to the Protobuf version.

    To find compatible versions:

    1. Check the PB-VERSION file in the repository for a list of supported versions.
    2. If your required version is missing, you can either contribute a PR to add it to the PB-VERSION file or manually generate Java files from Loki-specific .proto files within your own project (advanced).
  6. Add custom React pages

    main

    Custom pages are built using React components.

    1. Save your component as a .js file in website/pages/en.
    2. To make the page accessible via the top navigation bar, add it to the headerLinks in website/siteConfig.js using the page key.
    // website/siteConfig.js
    {
      headerLinks: [
        ...
        { page: 'my-new-custom-page', label: 'My New Custom Page' },
        ...
      ],
      ...
    }
  7. Add a new blog post

    main

    To add a blog post:

    1. Ensure the blog is enabled in the headerLinks field of website/siteConfig.js by including { blog: true, label: 'Blog' }.
    2. Create a new Markdown file in website/blog/ using the naming convention YYYY-MM-DD-My-Blog-Post-Title.md.
    3. Include the required YAML front matter (e.g., author, authorURL, title).
    // website/siteConfig.js
    headerLinks: [
        ...
        { blog: true, label: 'Blog' },
        ...
    ]
    // website/blog/2018-05-21-New-Blog-Post.md
    ---
    author: Frank Li
    authorURL: https://twitter.com/foobarbaz
    authorFBID: 503283835
    title: New Blog Post
    ---
    
    Lorem Ipsum...
  8. Enable tracing mode via system properties

    main

    To enable tracing mode, set the Java system property loki4j.trace to AsyncBufferPipeline. You can pass this via the command line when starting your application.

    To prevent the massive amount of trace information from cluttering your console, it is recommended to redirect stderr to a file.

    # Enable tracing mode
    java -Dloki4j.trace=AsyncBufferPipeline -jar my-app.jar
    
    # Enable tracing mode and redirect stderr to a log file
    java -Dloki4j.trace=AsyncBufferPipeline -jar my-app.jar 2>loki4j-trace.log
  9. Add dynamic metadata using SLF4J Markers

    main

    You can use SLF4J Markers to attach specific key-value pairs to individual log records as either labels or structured metadata.

    1. Enable Marker Reading

    First, you must enable marker scanning in your Loki4jAppender configuration by setting <readMarkers> to true:

    <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
        <readMarkers>true</readMarkers>
    </appender>

    2. Use Markers in Code

    Use StructuredMetadataMarker to add high-cardinality data to a specific log event:

    import com.github.loki4j.slf4j.marker.StructuredMetadataMarker;
    
    // ...
    var marker = StructuredMetadataMarker.of("exceptionClass", () -> ex.getClass().getSimpleName());
    log.error(marker, "Unexpected error", ex);

    For Labels (Use Sparingly)

    Use LabelMarker to add dynamic labels. Note that adding high-cardinality values as labels is not recommended as it can degrade Loki performance:

    import com.github.loki4j.slf4j.marker.LabelMarker;
    
    // ...
    var marker = LabelMarker.of("exceptionClass", () -> ex.getClass().getSimpleName());
    log.error(marker, "Unexpected error", ex);
    import com.github.loki4j.slf4j.marker.StructuredMetadataMarker;
    
    void handleException(Exception ex) {
        var marker = StructuredMetadataMarker.of("exceptionClass", () -> ex.getClass().getSimpleName());
        log.error(marker, "Unexpected error", ex);
    }
  10. Quick Start with Loki4j

    main

    Loki4j allows you to push logs from your Java application directly to Loki using a Logback appender.

    Prerequisites

    • Java: 17 or higher
    • Logback: v1.6.x (for the current stable version)

    Installation

    Add the dependency to your build tool of choice:

    Maven

    <dependency>
        <groupId>com.github.loki4j</groupId>
        <artifactId>loki-logback-appender</artifactId>
        <version>%version%</version>
    </dependency>

    Gradle

    implementation 'com.github.loki4j:loki-logback-appender:%version%'

    Configuration

    Add the Loki4jAppender to your logback.xml and point it to your Loki HTTP push endpoint:

    <contextName>my-app</contextName>
    
    <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
        <http>
            <url>http://localhost:3100/loki/api/v1/push</url>
        </http>
    </appender>
    
    <root level="DEBUG">
        <appender-ref ref="LOKI" />
    </root>