logback-kafka-appender Documentation

Kafka uses partitions to manage scalability and ordering. logback-kafka-appender allows you to control how log messages are distributed across partitions using different keying strategies.

Key considerations:

• Ordering: Kafka only guarantees read order within a single partition. To ensure messages from a specific source (like a host or thread) stay in order, use a strategy that uses that source as the message key.
• Distribution: Using a key (like HostNameKeyingStrategy) can lead to uneven distribution across partitions if there are fewer unique keys than partitions.
• Fixed Partitioning: You can bypass keying strategies entirely by providing a fixed partition using the partition property.

Decide whether to prioritize log delivery or application performance by choosing one of the following strategies:

To implement a custom strategy, extend com.github.danielwegener.logback.kafka.delivery.DeliveryStrategy.

Add logback-kafka-appender and logback-classic as runtime dependencies to your project.

Important: Due to breaking changes in the Logback Encoder API, you must use at least Logback version 1.2.

<!-- Maven pom.xml -->
<dependency>
    <groupId>com.github.danielwegener</groupId>
    <artifactId>logback-kafka-appender</artifactId>
    <version>0.2.0</version>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-classic</artifactId>
    <version>1.2.3</version>
    <scope>runtime</scope>
</dependency>

<!-- build.sbt -->
libraryDependencies += "com.github.danielwegener" % "logback-kafka-appender" % "0.2.0"
libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.2.3"

Even with AsynchronousDeliveryStrategy, the appender can block during Kafka metadata exchanges if brokers are unreachable. To ensure the appender never blocks your application, wrap the KafkaAppender with Logback's native AsyncAppender and set <neverBlock>true</neverBlock>.

<configuration>
    <appender name="kafkaAppender" class="com.github.danielwegener.logback.kafka.KafkaAppender">
        <!-- Kafka Appender configuration -->
    </appender>

    <appender name="ASYNC" class="ch.qos.logback.classic.AsyncAppender">
        <!-- If true, discards messages when the internal queue is full -->
        <neverBlock>true</neverBlock>  
        <appender-ref ref="kafkaAppender" />
    </appender>

    <root level="info">
        <appender-ref ref="ASYNC" />
    </root>
</configuration>

If the built-in strategies do not meet your requirements (for example, if you want to use Kafka's log compaction facility), you can implement the KeyingStrategy<T> interface. The createKey method should return a byte[] representing the key.

package foo;
import com.github.danielwegener.logback.kafka.keying.KeyingStrategy;
import ch.qos.logback.classic.spi.ILoggingEvent;
import java.nio.ByteBuffer;

/* This is a valid example but does not really make much sense */
public class LevelKeyingStrategy implements KeyingStrategy<ILoggingEvent> {
    @Override
    public byte[] createKey(ILoggingEvent e) {
        return ByteBuffer.allocate(4).putInt(e.getLevel()).array();
    }
}

The appender supports any ch.qos.logback.core.encoder.Encoder. You can use standard encoders like PatternLayoutEncoder or third-party ones like LogstashEncoder.

To implement custom serialization (e.g., for BSON or Avro), extend ch.qos.logback.core.encoder.Encoder<ILoggingEvent>. Note that the appender will not call headerBytes() or footerBytes() methods.

public class MyEncoder extends ch.qos.logback.core.encoder.Encoder<ILoggingEvent> {
    // Implementation details...
}