repository·master·Indexed 21 days ago
https://github.com/flant/loghouseA log management solution for Kubernetes that uses Fluentd for processing and ClickHouse for efficient storage of large log volumes. It includes a web UI for querying and monitoring, a structured query language for filtering logs, and support for both single-node and distributed ClickHouse cluster schemas. Deployment is managed via Helm, requiring Kubernetes >=1.9 and cert-manager.
StatefulSet. This schema defines the underlying data structures used by Loghouse for log storage and processing within ClickHouse.Loghouse uses a structured query syntax based on expressions and logical operators.
Basic Structure:
EXPRESSION [QUERY_OPERATOR QUERY]
Logical Operators (QUERY_OPERATOR):
ANDORExpression Structure (EXPRESSION):
KEY EXPRESSION_OPERATOR VALUE
Key Selection (KEY):
[a-zA-Z] and containing [a-zA-Z0-9_\-\.].* matches any key.~ is a shortcut for the label. prefix (e.g., ~component is equivalent to label.component).Loghouse supports two typical ClickHouse schema configurations depending on your deployment architecture:
Refer to the specific documentation for original/ or cluster/ to understand the differences in table structures and requirements.
Loghouse uses a distributed architecture to collect and store logs:
DaemonSet. It watches log directories from all hosts. It uses the Kubernetes_metadata filter to enrich logs with pod/container metadata (names, namespaces, labels, etc.) via the Kubernetes API.record_modifier filter.clickhouse-client CLI.string_fields, number_fields, boolean_fields, null_fields, or labels (for container labels).string_fields table.StatefulSet. It uses a hostPath volume or a PersistentVolumeClaim (PVC).Loghouse is deployed to Kubernetes using Helm. It requires Kubernetes version >=1.9 and cert-manager to be already installed in your cluster.
helm repo add loghouse https://flant.github.io/loghouse/charts/You can install using one of two methods:
This method allows you to inspect and modify the configuration before deployment.
helm fetch loghouse/loghouse --untar
# Edit the configuration
# vim loghouse/values.yaml
# Deploy to the 'loghouse' namespace
helm install --namespace loghouse -n loghouse loghouseNote: If image pulling is slow, use the --timeout 1200 flag.
For quick installations with specific overrides:
helm install -n loghouse loghouse/loghouse --set 'param=value' ...# Add repo
helm repo add loghouse https://flant.github.io/loghouse/charts/
# Install via local values
helm fetch loghouse/loghouse --untar
# (edit loghouse/values.yaml)
helm install --namespace loghouse -n loghouse loghouseLoghouse provides a default Prometheus dashboard for Grafana to monitor the health and performance of the stack.
Key metrics available via this dashboard include:
The ClickHouse portion of the dashboard is based on the f1yegor clickhouse dashboard.
You can customize the dashboard for your specific environment by modifying the variables in the Grafana dashboard settings.
Loghouse 0.3 uses database version 3. While newer versions of Loghouse are compatible with older schemas, performance may be degraded. To ensure optimal performance, you should update your schema using the provided Helm chart or by running the manual migration task via Rake.
To run the manual table creation task, use the following command:
DO_DB_DEPLOY=true rake create_logs_tablesFor cluster deployments requiring horizontal scaling, use the alternative ClickHouse schema. This setup allows you to add additional shards to enlarge available storage space or increase data processing speed.
Key configuration details:
config.xml.logs is configured within the remote_servers section.Critical Requirement: Every node in the cluster must have unique macros configured:
{shard}: Must be unique for every shard.{replica}: Must be unique for every node.The Web UI (loghouse-dashboard) is accessible via the address specified in your values.yaml configuration under the loghouse_host key.
Authentication:
You will be prompted for basic authorization. Credentials are generated via htpasswd and configured using the auth parameter in your values.yaml file.
To use a ClickHouse instance that is running outside of the Loghouse deployment, enable the external flag and provide the list of endpoints in your values.yaml file.
clickhouse:
external: true
externalEndpoints:
- 10.0.0.1
- 10.0.0.2
- 10.0.0.3When using a ClickHouse cluster where the database schema is managed externally or via a different process, set doDbDeploy to false in your values.yaml to prevent Loghouse from attempting to deploy the database itself.
doDbDeploy: falseUpgrading to v0.3.0 or later requires manual cleanup because the Helm chart was rewritten with new API versions and Helm hook policies.
Before upgrading, remove existing jobs and ingress resources in the loghouse namespace:
kubectl -n loghouse delete jobs,ing --allWarning: The database schema changed in v0.3.0. You must prepare a backup of your ClickHouse data before proceeding. A migration task will start automatically at the end of the upgrade process.