Fluentd

To contribute to Fluentd, you must fork the upstream repository to your GitHub account and then clone your fork locally. To keep your local environment synchronized with the official Fluentd repository, you should add an upstream remote pointing to https://github.com/fluentd/fluentd.git.

# Clone your fork
$ git clone https://github.com/$your_github_account/fluentd.git

# Add the official upstream remote
$ cd fluentd
$ git remote add upstream https://github.com/fluentd/fluentd.git
$ git remote -v

To get started with Fluentd, install the gem, run the daemon with a configuration file, and use fluent-cat to test data ingestion.

1. Install Fluentd: Use gem install fluentd.
2. Run Fluentd:
   • To run with a configuration directory: fluentd -s conf
   • To run with a specific configuration file in the background: fluentd -c conf/fluent.conf &
3. Test Ingestion: Pipe JSON data into fluent-cat to verify your setup: echo '{"json":"message"}' | fluent-cat debug.test.

$ gem install fluentd
$ fluentd -s conf
$ fluentd -c conf/fluent.conf &
$ echo '{"json":"message"}' | fluent-cat debug.test

Always create a new branch for every feature or bug fix to keep your work isolated from the master branch.

$ git checkout master
$ git pull upstream master
$ git checkout -b new-feature

When committing changes, you must include a sign-off using the -s flag. After committing, push your branch to your personal fork (origin) using the -u flag to set up tracking.

# Stage changes
$ git add <file>

# Commit with sign-off
$ git commit -s

# Push to your fork
$ git push -u origin new-feature

To develop on Fluentd, ensure you have the following prerequisites installed:

• Ruby: version 3.3 or later
• git: must be in your PATH.

Follow these steps to prepare the local environment:

1. Install Bundler: gem install bundler
2. Install Dependencies: Run bundle install --path vendor/bundle to install the required gems into the project directory.
3. Run Tests: Use bundle exec rake test to execute the test suite.

$ gem install bundler
$ bundle install --path vendor/bundle
$ bundle exec rake test

Before starting new work, ensure your local master branch is up to date with the official upstream repository by pulling from your fork and rebasing against the upstream master.

$ git checkout master
$ git pull origin master
$ git fetch upstream
$ git rebase upstream/master

The Formatter helper automatically manages the lifecycle of all registered formatters. When the parent plugin undergoes lifecycle changes, the helper propagates these calls to all configured formatters using formatter_operate.

Supported lifecycle methods that are propagated to formatters include:

• start
• stop
• before_shutdown
• shutdown
• after_shutdown
• close
• terminate

If a formatter encounters an error during these operations, the error is caught and logged with the specific usage identifier and the formatter instance to prevent the entire plugin from crashing.

You can choose how fluent-cat connects to the Fluentd daemon:

• TCP (Default): Uses the --host and --port options to connect via a network socket.
• Unix Socket: Use the --unix flag and specify the location of the socket file using the --socket PATH option.

The fluent-cat tool supports three primary input formats for data read from stdin:

1. json (default): Expects each line to be a valid JSON object. The object is parsed and sent as a record.
2. msgpack: Expects a MessagePack stream from stdin. This is more efficient for binary data.
3. none: Treats each line from stdin as a plain string. You must specify a --message-key (defaults to message) which will wrap the string into a JSON-like record: {"message_key": "line content"}.

To add formatter support to a custom Fluentd plugin, include Fluent::PluginHelper::Formatter in your plugin class. This helper provides a standardized way to manage multiple formatters via a <format> configuration section.

Configuration Schema

When including the helper, your plugin will automatically support a format configuration section. The following parameters are available within that section:

• usage: (String) A unique identifier for the formatter instance. Used to distinguish between different formatters in the same plugin.
• @type: (String) The type of the formatter plugin to instantiate.

Because the helper uses Fluent::Configurable, you can define multiple <format> blocks in your configuration file, provided multi: true is set in the plugin's internal config_section definition.