Clicktail Ingestion Agent

Before ingesting data, you must ensure the target ClickHouse database and tables exist. The project provides SQL schema files in the schema/ directory.

1. Create the database using the provided db.sql.
2. Create the specific table for your parser (e.g., mysql.sql) using the clickhouse-client.

# Create Database
cat schema/db.sql | clickhouse-client --multiline

# Create Table for MySQL slow logs
cat schema/mysql.sql | clickhouse-client --multiline

You can install clicktail using three different methods: via the Apt package repository, by downloading an individual .deb package, or by building from source.

Using Apt

First, add the Altinity ClickHouse repository to your system, then install the package.

Using Source

Use go get to fetch the repository. You can specify a custom GOPATH to control the installation path.

# Install via Apt
curl -s https://packagecloud.io/install/repositories/Altinity/clickhouse/script.deb.sh | bash
apt-get install clicktail

# Install from source
go get github.com/Altinity/clicktail

# Install to a specific path
GOPATH=/usr/local go get github.com/Altinity/clicktail
GOPATH=/usr/local go build
GOPATH=/usr/local go install

Dynamic sampling allows you to adjust the sampling rate based on the content of the logs. This is useful for capturing more detail during low-traffic periods.

Configuration:

• --dynsampling <field>: Enable dynamic sampling using the specified field (can be used multiple times; fields are concatenated to form the key).
• --dynsample_window <seconds>: Measurement window size (default: 30).
• --dynsample_minimum <rate>: If traffic falls below this rate, the dynsampler stops sampling (default: 1).

Note: When dynamic sampling is enabled, clicktail automatically disables tail-sampling (sampling while tailing) because sampling decisions must be made after the content is parsed.

While clicktail supports command-line options, you can use a clicktail.conf file to manage settings. Note that the configuration file is not picked up by default; you must explicitly specify it using the -c flag.

The configuration file uses INI-style sections. You must define the [Required Options] section first.

Required Sections

• [Application Options]: General settings like APIHost.
• [Required Options]: Essential settings including ParserName, LogFiles, and Dataset.
• [Parser Name Options]: Parser-specific settings (e.g., [MySQL Parser Options]).

[Application Options]
APIHost = http://localhost:8123/

[Required Options]
ParserName = mysql
LogFiles = /var/log/mysql/mariadb-slow.log
Dataset = clicktail.mysql_slow_log

[MySQL Parser Options]
Host = localhost:3306
User = username
Pass = userpass

If you need to ingest old log data into ClickHouse, use the --backfill flag. This is a convenience flag that automatically configures several other options to ensure all historical data is processed once.

When --backfill is set, it overrides the following:

• --backoff: Set to true (retries sending failed events when rate limited).
• --tail.read_from: Set to beginning.
• --tail.stop: Set to true (stops after reaching the end of the file).

clicktail -p mysql -f /var/log/mysql/audit.log -d mysql_audit --backfill

For clicktail to communicate with ClickHouse, the server must be accessible. If clicktail is running on a different host, you must enable external connections in ClickHouse.

1. Edit /etc/clickhouse-server/config.xml.
2. Locate the <listen_host> tag.
3. Add <listen_host>0.0.0.0</listen_host> immediately after the existing <listen_host>127.0.0.1</listen_host> line.
4. Ensure the ClickHouse port (default 8123) is open in your firewall.

<!-- In /etc/clickhouse-server/config.xml -->
<listen_host>127.0.0.1</listen_host>
<listen_host>0.0.0.0</listen_host>

The regex parser uses the RE2 syntax (compatible with Go's regexp package) to extract structured data from log files. You can use it via the honeytail CLI by specifying the -p regex flag and providing regex patterns and time field configurations via --regex.* flags.

To extract fields, use named capture groups in your line_regex. For example, (?P<name>pattern) creates a field named name.

honeytail -p regex -k $HONEYTAIL_WRITEKEY \
  -f some/path/system.log \
  --dataset 'MY_TEST_DATASET' \
  --backfill \
  --regex.line_regex="(?P<time>\d{2}:\d{2}:\d{2}) (?P<field1>\w+)" \
  --regex.line_regex="(?P<foo>\w+)" \
  --regex.timefield="time" \
  --regex.time_format="%H:%M:%S"

When using the regex parser, the following configuration flags are available via the CLI:

• --regex.line_regex: A regular expression string containing named capture groups used to extract fields from a line. Multiple --regex.line_regex flags can be provided.
• --regex.timefield: The name of the field (extracted via line_regex) that should be treated as the event timestamp.
• --regex.time_format: The Go-style time format string used to parse the value in the designated timefield.

As of the current version, clicktail supports the following log parsers:

• ArangoDB
• MongoDB
• MySQL (including mysqlaudit)
• PostgreSQL
• nginx
• regex (custom parsing)

clicktail can read data from STDIN or from files on disk.

To see which parser modules are supported by your current build of clicktail, run the CLI with the --list (or -l) flag.

Supported Parsers:

• arangodb
• json
• keyval
• mongo
• mysql
• mysqlaudit
• nginx
• postgresql
• regex

clicktail --list