elastdocker

In Filebeat 9, the container input type is deprecated. ElastDocker has migrated to the filestream input with a container parser, which is the recommended approach for ES 9+.

Key changes in filebeat/filebeat.docker.logs.yml and filebeat/filebeat.monitoring.yml:

• type: container is replaced by type: filestream with unique IDs.
• Added parsers.container configuration for Docker log parsing.
• Added prospector.scanner.symlinks: true for Docker log paths.

To send traces, logs, and metrics from your application to the apm-server container, you must authenticate using the ELASTIC_APM_SECRET_TOKEN defined in your .env file.

Requirements:

• The apm-server container must be running.
• Port 8200 must be reachable.
• Use https for the serverUrl as TLS is enabled by default.

Node.js Example: Initialize the agent at the very top of your application entry point (before express or http):

const apm = require('elastic-apm-node').start({
  serviceName: 'foobar',
  secretToken: process.env.ELASTIC_APM_SECRET_TOKEN,
  serverUrl: 'https://localhost:8200',
})

const apm = require('elastic-apm-node').start({
  serviceName: 'foobar',
  secretToken: process.env.ELASTIC_APM_SECRET_TOKEN,

  // https is enabled by default as per elastdocker configuration
  serverUrl: 'https://localhost:8200',
})

To deploy the preconfigured Elastic Stack (ELK) using Elastdocker, follow these steps:

1. Clone the repository:

   git clone https://github.com/sherifabdlnaby/elastdocker.git

2. Configure Linux Virtual Memory (Linux hosts only): Elasticsearch requires a higher vm.max_map_count. Run this as root:

   sysctl -w vm.max_map_count=262144

3. Initialize Keystore and Certificates: Generate the Elasticsearch Keystore and TLS self-signed certificates:

   mise run stack:setup

4. Start the stack:

   mise run up
   # OR
   docker compose up -d

5. Access Kibana: Visit https://localhost:5601.

   • Note: HTTPS is enabled by default.
   • Default Credentials: Username elastic, Password changeme (can be modified in .env).

Upgrading to Elasticsearch 9 involves breaking changes in configuration and monitoring architecture.

Required Upgrade Path

You must upgrade to Elasticsearch 8.19.x before moving to 9.x: 8.17.0 $\rightarrow$ 8.19.x (run Upgrade Assistant) $\rightarrow$ 9.x

Clean Installation for ES 9

If you are starting fresh with Elasticsearch 9:

1. Set ELK_VERSION=9.4.2 in your .env file.
2. Run mise run stack:setup to prepare the stack.
3. Run mise run up (or mise run all to include monitoring components).

This project uses mise to manage tool versions (linters, formatters) and expose tasks.

1. Install mise

curl https://mise.run | sh          # or: brew install mise
echo 'eval "$(mise activate zsh)"' >> ~/.zshrc   # for zsh
mise doctor                         # verify installation

2. Initialize the repository

Run these commands once inside the project directory to trust the configuration and install pinned tools:

mise trust      # allow the repo's mise config to load
mise run setup  # install pinned tools and git hooks

For Elasticsearch 9+, Stack Monitoring is performed using Metricbeat. When you execute mise run monitoring, Metricbeat collects metrics from all stack components and sends them to Elasticsearch. You can then view these metrics in the Stack Monitoring tab within Kibana.

Note on Architecture:

• ES 8.x and earlier: Used internal xpack.monitoring for self-monitoring.
• ES 9.x: Uses external Metricbeat collection for better scalability and reliability.

Production Best Practice: In production environments, cluster metrics should be shipped to a dedicated, separate monitoring cluster.

You can automatically discover, parse, and ship logs from all Docker containers running on your host to Elasticsearch using Filebeat. Run the following command:

mise run collect-docker-logs

If you need to add custom credentials (e.g., for an S3 Snapshot Repository), you can extend the keystore generation by modifying the ./setup/keystore.sh script. After modifying the script, re-run the keystore generation command:

mise run keystore

The stack behavior is primarily controlled through the .env file. Key parameters include:

• ELASTIC_PASSWORD: Sets the password for the elastic superuser. Important: If you change this, you must run mise run stack:setup and restart the stack.
• ELK_VERSION: Specifies the Elastic Stack version (default: 9.4.2). Any version >= 9.0.0 is compatible.
• ELASTICSEARCH_HEAP: Memory allocation for Elasticsearch (default: 1GB).
• LOGSTASH_HEAP: Memory allocation for Logstash.
• ELASTIC_APM_SECRET_TOKEN: The secret token used for APM authentication.

The kibana service uses the following environment variables:

• ELASTIC_USERNAME: Username for Elasticsearch authentication.
• ELASTIC_PASSWORD: Password for Elasticsearch authentication.
• ELASTICSEARCH_HOST_PORT: The full HTTPS URL for the Elasticsearch host.
• KIBANA_PORT: The port on which Kibana listens.
• KIBANA_ENCRYPTION_KEY: Encryption key for Kibana.
• KIBANA_SAVED_OBJECTS_ENCRYPTION_KEY: Encryption key for Kibana saved objects.
• KIBANA_REPORTING_ENCRYPTION_KEY: Encryption key for Kibana reporting.

Additionally, Kibana loads an environment file from ./secrets/.env.kibana.token.

The apm-server service uses the following environment variables:

• ELASTIC_USERNAME: Username for Elasticsearch authentication.
• ELASTIC_PASSWORD: Password for Elasticsearch authentication.
• ELASTICSEARCH_HOST_PORT: The full HTTPS URL for the Elasticsearch host.
• ELASTIC_APM_SECRET_TOKEN: The secret token for APM authentication.
• APMSERVER_PORT: The port on which the APM server listens.