Plumelog Documentation

Plumelog is a simple and easy-to-use Java distributed logging component. It is designed to collect logs without code intrusion by leveraging existing logging frameworks like log4j, log4j2, or logback. It uses Elasticsearch as the query engine and supports setting trace IDs to facilitate correlated log queries across distributed systems.

Key features include:

• Zero Code Intrusion: Works transparently with existing projects (supports Dubbo and Spring Cloud).
• High Throughput: Optimized for high-performance log collection and querying.
• No Local Disk Usage: Logs are pushed to queues, preventing application local disk space consumption.
• Flexible Deployment: Supports multiple modes depending on scale and complexity.

Plumelog is a simple and easy-to-use distributed logging component for Java. It provides a non-intrusive way to collect logs and trace request chains across distributed systems.

Key features include:

• Non-intrusive collection: Supports log4j, com.plumelog.lite.log4j2, and logback to collect logs and set trace IDs for easy correlation.
• Elasticsearch-based: Uses Elasticsearch as the primary query engine for high throughput and efficient searching.
• Zero local disk footprint: Logs are sent directly to the backend, meaning no local disk space is consumed on the application server, making it maintenance-free.
• Transparent integration: Does not affect the running application and requires no modifications to existing projects. It supports Dubbo and Spring Cloud environments.

Plumelog supports metadata resolution within Markdown files. You can define metadata by adding a block of text between triple dashes (---) at the beginning of the file. The metadata must be written in key:value format.

When processed, this metadata is resolved into md_json/blog.json. The system automatically preserves the filename and the __html representation of the content.

---
key1: hello
key2: world
---

# blog1

Content goes here...

Depending on your project scale, choose one of the following deployment architectures:

• Single Redis Small Cluster: Suitable for most small to medium-sized projects.
• Multi-Redis Large Cluster: Recommended for large projects or projects where specific services have extremely high log volumes.
• Kafka Cluster: For very high-volume projects.
• Hybrid Mode (Redis + Kafka): For extremely high-volume projects where certain services have even higher requirements than others.

Plumelog supports metadata resolution via YAML front matter. You can define metadata by writing text between triple dashes (---) at the beginning of a Markdown file using the key:value format.

When processed, this metadata is resolved into md_json/blog.json. The system automatically preserves the filename and the __html content alongside your custom metadata keys.

---
key1: hello
key2: world
---

# blog3

filling text...

Plumelog's architecture is divided into several specialized modules that handle the lifecycle of a log from collection to storage and visualization:

• plumelog-core: The core component containing the log collection side. It is responsible for collecting logs and pushing them to queues like Kafka or Redis.
• plumelog-server: Responsible for asynchronously consuming logs from the queues and writing them into Elasticsearch.
• plumelog-demo: A Spring Boot-based usage example.
• plumelog-lite: An embedded integration version that requires no external deployment, suitable for small, independent projects.

You can implement global tracing by extending AbstractAspect and defining a pointcut.

Important Constraints:

1. Manual vs Global: Manual @Trace annotations will be ignored if a global aspect is configured.
2. Performance: Global tracing generates a large volume of logs. Avoid using global tracing on high-concurrency modules or applying it globally across the entire application without careful consideration.
3. Troubleshooting: If no trace data appears, first check if a traceID is being generated, then verify if your Spring AOP configuration is active.

@Aspect
@Component
public class AspectConfig extends AbstractAspect {

    @Around("within(com.xxxx..*)") // Replace with your package path
    public Object around(JoinPoint joinPoint) {
        return aroundExecute(joinPoint);
    }
}

Plumelog offers several deployment modes depending on your scale and infrastructure requirements:

• lite mode: A standalone single-machine version that does not depend on external middleware. It is easy to deploy but has limited performance (suitable for < 10GB of logs per day). Best used with SSDs for small management systems.
• redis or kafka mode: Distributed cluster modes suitable for large-scale internet companies. You choose between Redis or Kafka based on your existing infrastructure.
• plumelog-lite: An embedded version that you can include directly in your project via Maven/POM. It includes a built-in query interface and is ideal for small, independent projects or outsourced software.

Note: You should have a basic understanding of logback and log4j configurations before proceeding.

Plumelog supports metadata resolution within Markdown files. You can define custom metadata by placing text between triple dashes (---) at the beginning of the file using the key:value format.

When processed, this metadata is resolved into md_json/blog.json. The system automatically preserves the filename and __html fields alongside your custom keys.

---
key1: hello
key2: world
---

# blog5

filling text...

Plumelog Lite is a lightweight mode designed for small-scale projects. It can be used as an embedded dependency (plumelog-lite) within your project and does not require separate deployment of Redis or Elasticsearch.

Limitations of Lite Mode:

• No support for extended fields.
• No error statistics.
• No error alarms.

Requirements:

• If you use Lite mode, you must upgrade your client to version 3.5 or higher.

You can quickly deploy Plumelog using the official Docker image. Use the docker run command to start a container with a specific log model (e.g., redis).

To run a container named plumelog on port 8891 using redis as the queue model, execute:

docker run -d -p 8891:8891 -e plumelog.model=redis --name=plumelog ylyue/plumelog

Alternatively, you can use the docker-compose.yml file located in the current directory for deployment.

To prevent trace logs from cluttering your console or local files, you can use a filter in your Logback configuration.

For version 3.4.1 and later, use the built-in com.plumelog.logback.util.FilterSyncLogger class in your logback.xml.

<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
    <!-- This filter suppresses all trace logs -->
    <filter class="com.plumelog.logback.util.FilterSyncLogger">
        <level>info</level>
        <filterPackage>com.plumelog.trace.aspect.AbstractAspect</filterPackage>
    </filter>
    <encoder>
        <Pattern>${CONSOLE_LOG_PATTERN}</Pattern>
        <charset>UTF-8</charset>
    </encoder>
</appender>