Logstash

The Logstash Integration Tests, also known as RATS, are full integration test sets. They are designed to validate Logstash by:

• Starting Logstash from a binary.
• Running configurations using the -e flag.
• Utilizing external services such as Kafka, Elasticsearch, and Beats.

The framework is a hybrid system consisting of:

• Bash scripts: Primarily used for service setup.
• Ruby service files: Manage service lifecycles.
• RSpec: Used for all test assertions.

Logstash is an open source data collection engine featuring real-time pipelining capabilities. It is designed to dynamically unify data from disparate sources and normalize it for various destinations.

Logstash uses a pipeline architecture consisting of:

• Input plugins: To ingest various types of events.
• Filter plugins: To enrich and transform data.
• Output plugins: To send processed data to destinations.
• Codecs: To simplify the ingestion and processing of data formats.

The logstash-integration-failure_injector plugin is intended exclusively for Logstash pipeline testing purposes. It is not a production plugin.

If you are modifying the plugin's source code, you must rebuild the gem file to apply your changes.

gem build logstash-integration-failure_injector.gemspec

Logstash provides a wide ecosystem of over 200 plugins to process and transform data. Depending on your specific data processing needs, you can categorize your requirements into the following functional areas:

• Core Operations: Basic data manipulation and processing.
• Deserializing Data: Converting raw data formats (like JSON or strings) into structured Logstash events.
• Extracting Fields and Wrangling Data: Pulling specific values out of fields or reshaping existing data.
• Enriching Data with Lookups: Adding context to events by performing lookups against external or internal data sources.

For a complete list of all available processing tools, refer to the Filter plugins and Codec plugins documentation.

Logstash supports authentication and encryption over HTTPS for its Elasticsearch output, input, and filter plugins, as well as for monitoring and centralized pipeline management.

Starting with Elasticsearch 8.0, clusters are secured by default. To establish communication, you must:

1. Configure authentication credentials for Logstash.
2. Grant authorized users permission to access the Logstash indices.
3. Enable TLS/SSL in the Elasticsearch output section of your Logstash configuration to allow communication with a secured cluster.

A Dead Letter Queue (DLQ) is a mechanism to temporarily store events that Logstash cannot process, preventing them from blocking the pipeline or being lost.

When it is triggered:

• Elasticsearch Output: When the Elasticsearch output receives a response code of 400 or 404 (indicating an event that cannot be retried, such as a mapping error). Note that if the HTTP request itself fails (e.g., Elasticsearch is unreachable), Logstash will retry indefinitely and the DLQ will not be used.
• Conditional Statements: When an error occurs during the evaluation of a conditional statement (e.g., comparing incompatible types like a string and an integer).

Each event in the DLQ includes the original event, metadata describing the failure reason, information about the plugin that wrote the event, and a timestamp.

Logstash supports native Java plugins through a system composed of three main parts:

1. Java execution engine extensions: Allows Java plugins to run within Logstash pipelines.
2. Java APIs: Found in the co.elastic.logstash.api package. Important: When developing, only reference classes or interfaces within the co.elastic.logstash.api package. Referencing concrete implementations outside of this package may cause your plugin to break if Logstash internals change.
3. Packaging and deployment tooling: Automates the process of preparing your plugin for use in Logstash.

To create a plugin, follow this general workflow:

1. Choose a plugin type: input, codec, filter, or output.
2. Set up your development environment.
3. Code the plugin logic.
4. Package and deploy the plugin.
5. Run Logstash with the new plugin loaded.

Logstash shuts down output plugins both asynchronously and cooperatively. Your plugin must implement two methods to handle this:

1. stop(): This method is called to signal that the plugin should stop sending events. A common pattern is to set a volatile boolean stopped = true; flag.
2. awaitStop(): This method is called to block the shutdown process until the plugin has finished its cleanup. A java.util.concurrent.CountDownLatch is a recommended way to implement this.

Note: awaitStop() should not be used to signal the stop; it is only for waiting for the stop to complete.

private final CountDownLatch done = new CountDownLatch(1);
private volatile boolean stopped = false;

@Override
public void stop() {
    stopped = true;
    done.countDown();
}

@Override
public void awaitStop() throws InterruptedException {
    done.await();
}