Logstash Logback Encoder

The LogstashTcpSocketAppender supports multiple connection strategies to determine the order in which destination connections are attempted and when to reestablish connections. Logs are sent to only one destination at a time.

Available Strategies:

1. preferPrimary (Default): Prefers the first configured destination. If it is down, it attempts the next. If a connection succeeds but closes before minConnectionTimeBeforePrimary (default 10s) has elapsed, it tries the next destination to avoid 'connection storms'. If it stays open longer than that threshold, it will eventually attempt to return to the primary destination.
2. roundRobin: Attempts connections in a circular order. If a connection fails, it moves to the next.
3. random: Attempts connections in a random order.

You can implement your own strategy by implementing the DestinationConnectionStrategy interface.

<appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
    <destination>destination1.domain.com:4560</destination>
    <destination>destination2.domain.com:4560</destination>
    <destination>destination3.domain.com:4560</destination>
    <connectionStrategy>
        <preferPrimary>
            <secondaryConnectionTTL>5 minutes</secondaryConnectionTTL>
        </preferPrimary>
    </connectionStrategy>
</appender>

For greater flexibility in the JSON format and data included in LoggingEvents and AccessEvents, use LoggingEventCompositeJsonEncoder or AccessEventCompositeJsonEncoder.

These encoders are composed of one or more JSON providers that contribute to the output. No providers are configured by default; you must explicitly add the ones you want within a <providers> block.

To optimize performance and reduce garbage collection pressure, you can configure the minBufferSize property. The default is 1024 bytes. It is strongly advised to set this to at least the average size of your encoded events.

<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
    <providers>
        <mdc/>
        <pattern>
            <pattern>
                {
                  "timestamp": "%date{ISO8601}",
                  "myCustomField": "fieldValue",
                  "relative": "#asLong{%relative}"
                }
            </pattern>
        </pattern>
        <stackTrace>
            <throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
                <maxDepthPerThrowable>30</maxDepthPerThrowable>
                <maxLength>2048</maxLength>
                <shortenedClassNameLength>20</shortenedClassNameLength>
                <exclude>^sun\.reflect\..*\.invoke</exclude>
                <exclude>^net\.sf\.cglib\.proxy\.MethodProxy\.invoke</exclude>
                <evaluator class="myorg.MyCustomEvaluator"/>
                <rootCauseFirst>true</rootCauseFirst>
            </throwableConverter>
        </stackTrace>
    </providers>
</encoder>

You can mask specific values using regex or literal strings. When using regex, all matches within a string field will be replaced. To match the entire field value, use ^ and $ anchors.

Configuration Options:

• <defaultMask>: Sets the mask string (defaults to ****).
• <value>: Adds a single value/regex using the default mask.
• <values>: A comma-separated list of values/regexes using the default mask.
• <valueMask>: Allows specifying a custom mask for specific values. The mask string can reference regex capturing groups (e.g., $1).
• <valueMaskSupplier>: Class name for dynamic value masking via MaskingJsonGeneratorDecorator.ValueMaskSupplier.
• <valueMasker>: Class name for advanced masking via net.logstash.logback.mask.ValueMasker.

Behavior: Values are passed through every configured masker sequentially. The output of one masker becomes the input to the next.

Paths follow a format similar to JSON Pointer. Use these rules to target specific fields:

• Absolute paths: Start with / (e.g., /@timestamp).
• Partial paths: Do not start with /; they match the sequence anywhere in the output.
• Single token: A path with no / matches all occurrences of that field name.
• Wildcards: Use * to match any token at that position.
• Escaping: Use ~1 to escape / and ~0 to escape ~ within a token.

Configuration Options:

• <defaultMask>: Sets the mask string (defaults to ****).
• <path>: Adds a single path using the default mask.
• <paths>: A comma-separated list of paths using the default mask.
• <pathMask>: Allows specifying a custom mask for specific paths.
• <pathMaskSupplier>: Class name for dynamic path masking via MaskingJsonGeneratorDecorator.PathMaskSupplier.
• <fieldMasker>: Class name for advanced masking via net.logstash.logback.mask.FieldMasker.

The pattern provider allows you to define a JSON template using Logback's PatternLayout conversion specifiers. The encoder will populate the values within the template.

Type Conversion Operations

Because patterns are processed as strings, you can use special operations to ensure correct JSON types:

• #asLong{...}: Converts resulting string to a Long (or null).
• #asDouble{...}: Converts resulting string to a Double (or null).
• #asBoolean{...}: Converts resulting string to a Boolean. Supports true, yes, y, 1 as true.
• asNullIfEmpty{...}: Converts empty string to null.
• #asJson{...}: Converts resulting string to a JSON object (or null).
• #tryJson{...}: Converts to JSON, or keeps as string if conversion fails.
• #nullNA{...}: (AccessEvents only) Replaces a dash (-) with null.

Omitting Empty Fields

Set <omitEmptyFields>true</omitEmptyFields> to remove fields from the output if they are null, empty strings, empty arrays, or empty objects.

<pattern>
    <pattern>
        {
            "line_str": "%line",
            "line_long": "#asLong{%line}",
            "has_message": "#asBoolean{%mdc{hasMessage}}",
            "json_message": "#asJson{%message}"
        }
    </pattern>
</pattern>

The encoders and layouts use ThreadLocal internally for performance. In environments with application reloading (like web containers) or high thread counts (like virtual threads), you should follow these two practices to prevent memory leaks:

1. Use an asynchronous appender: Use LogstashTcpSocketAppender, LoggingEventAsyncDisruptorAppender, or Logback's AsyncAppender. This limits the number of threads using the encoder/layout, thereby limiting the number of ThreadLocal instances and reducing resource contention.
2. Cleanly shut down Logback: Ensure you call the Logback shutdown process when the application stops or reloads. This stops the asynchronous threads and makes their ThreadLocal values eligible for garbage collection.

Common Error Symptoms:

• Tomcat errors indicating a ThreadLocal (specifically net.logstash.logback.util.ThreadLocalHolder.Holder) was not removed during web application stop.
• Tomcat errors indicating an asynchronous appender thread (e.g., logback-appender-ASYNC-2) failed to stop.