DOCSEARCHbyAlgolia
Submit Docs

Open-Monitor Documentation

repository·master·Indexed 18 days ago

https://github.com/webankpartners/open-monitor

An enterprise-grade monitoring plugin that wraps Prometheus to provide enhanced alert management, advanced visualization, and automated data archiving. It includes monitor-server, a Go-based backend for managing metric views, agents, and alerts, as well as monitor-agent components such as node_exporter for hardware/OS metrics and ping_exporter for network connectivity checks.

Tokens
26.5K
Snippets
76
Records
113
Agent score
62%

What's inside Open-Monitor

  1. Overview of the PING monitoring program

    master

    The ping_exporter is a monitoring program designed to perform PING checks on a set of IP addresses. It supports two methods for defining the target IP sources:

    1. API-based: IP addresses can be fetched dynamically from an API.
    2. Local File-based: IP addresses can be provided via a local ip.txt list.

    This component is used to monitor network connectivity to specific endpoints.

  2. Overview of Monitor-server

    master

    Monitor-server is a backend monitoring service developed in Go. It encapsulates Prometheus data querying and manages monitoring resources.

    Key capabilities include:

    • View Management: Provides basic metric views for Host, MySQL, and Redis, as well as support for custom views.
    • Agent Management: Handles registration, deregistration, and start/stop operations for monitoring objects.
    • Alert Management: Supports threshold configuration, alert group management, alert suppression (masking), and alert dispatching.

  3. Overview of Open-Monitor

    master

    Open-Monitor is a monitoring and alarm management plugin built on top of Prometheus. It encapsulates Prometheus functionality to provide non-intrusive monitoring, enhanced alarm management, and graphical dashboards. It is designed to monitor resources and applications through plugins.

    Core Components:

    • Prometheus: The underlying time series database and monitoring system.
    • Alert Manager: Handles alert routing and management.
    • Monitor: The upper layer that encapsulates configuration management and chart display (Backend: Go + Gin + Xorm; Frontend: Vue + ECharts).
    • Agent_manager: Manages monitoring agents.
    • Ping_exporter: Provides ping-based metrics.
    • Archive_mysql_tool: Handles data archiving.

    Key Capabilities:

    • Endpoint Management: Register, de-register, start, and stop endpoints. Supports synchronization from CMDB and group management.
    • Data Management: Configuration for data collection and data query functions.
    • Alarm Management: Threshold configuration, log monitoring, alarm triggering, and persistence of Prometheus alert rules. Supports un-recovered alarm panels and historical alarms.
    • Dashboard Management: Graphical configuration and custom dashboards supporting mainstream types (Host, MySQL, Redis, Tomcat, etc.) and native PromQL queries.

  4. Manage monitoring objects and groups

    master

    Open-Monitor uses a hierarchy of monitoring units to organize data.

    Core Concepts:

    • Monitoring Object: The most basic unit (e.g., a single host or an application process). To add one, you must provide the Agent IP and Agent Port.
    • Hierarchical Object (Business Level): A logical grouping that contains multiple technical monitoring objects. This allows you to view the monitoring status of an entire business service at once.
    • Object Group: A collection of monitoring objects. Configuring thresholds or log alarms at the group level automatically applies those policies to all member objects within the group.

  5. Key Features of Open-Monitor

    master

    Open-Monitor provides several high-level capabilities for infrastructure and application monitoring:

    1. Monitoring Object Management

    • Registration/Deregistration: Manually manage objects or sync automatically from a CMDB.
    • Group Management: Organize objects into groups for collective alert configuration.
    • Hierarchical Structure: Organize monitoring by business units or organizational architecture.
    • Specialized Monitoring: Supports process monitoring and business log monitoring.

    2. Visualization and Views

    • Pre-defined Views: Default metrics views for Host, MySQL, Redis, Java, and System metrics.
    • PromQL Support: Native Prometheus Query Language support with the ability to save query configurations.
    • Custom Views: Highly configurable custom dashboards that can be set as the homepage.

    3. Alert Management

    • Persistence: Persists and distributes Prometheus alerting rules.
    • Alert Panels: Displays unrecovered alerts and historical alert data.
    • Notification Management: Manages alert recipients and includes integrated email notification support.

    4. Agent and Tooling

    • Agent Management: Uses agent_manager to quickly deploy monitoring for MySQL, Redis, JMX, and Nginx.
    • Liveness Detection: Built-in tools for ping, telnet, and http checks.

    5. Data Archiving

    • Automated Archiving: Automatically archives monitoring data to MySQL, managing sharding and table partitioning to support long-term data retrieval.

  6. How business metric collection works via content matching

    master

    Open-Monitor collects business metrics by scanning business system logs line by line. It uses regular expressions to identify and extract JSON strings from log lines. Once a match is found, the system parses the JSON to extract key-value pairs for metric calculation.

    Matching Process:

    1. The system applies a configured regular expression to each log line.
    2. It extracts the substring captured by the first set of parentheses ().
    3. This substring is parsed as a JSON object.

    Example:

    • Regex: \[.*\]\[.*\]\[.*\]\[.*\]\[.*\]\[.*\]\[.*\]\[(.*)\]\[.*\]
    • Log Line: [INFO][2020-01-01 10:00:00 169][ConsumeMessageThread_9][][20111201A][][][{"costTime":52,"method":"GET","resCode":"200"}][]
    • Extracted JSON: {"costTime":52,"method":"GET","resCode":"200"}

    Note: You can configure multiple JSON segments, but avoid using duplicate keys across segments as they will overwrite each other.

    Regex: \[.*\]\[.*\]\[.*\]\[.*\]\[.*\]\[.*\]\[.*\]\[(.*)\]\[.*\]
Log Line: [INFO][2020-01-01 10:00:00 169][ConsumeMessageThread_9][][20111201A][][][{"costTime":52,"method":"GET","resCode":"200"}][]

  7. Understand `node_ntp_sanity` and causality violation

    master

    The node_ntp_sanity metric provides an aggregate health score for NTPD. It evaluates several factors:

    1. Stratum and Leap Flag status.
    2. Sane Freshness: Based on node_ntp_reference_timestamp_seconds.
    3. Root Distance: Must be less than collector.ntp.max-distance.
    4. Causality Violation: Must be less than collector.ntp.local-offset-tolerance.

    Causality Violation Calculation: It is a lower bound estimate of clock error calculated using SNTP as the positive portion of: abs(node_ntp_offset) - node_ntp_rtt / 2

  8. Estimate storage capacity requirements

    master

    To estimate the storage capacity needed for Open-Monitor, use the following logic based on the performance report:

    • 24h Capacity: Calculated using the official formula: samples per second * retention time * (1[min] or 2[max]).
    • 30d Capacity: Based on the configured local storage time limit.

    Refer to the performance table for specific estimates based on the number of node_exporter instances and total metric count.

  9. Configure Node Exporter collectors

    master

    Node Exporter uses pluggable metric collectors to expose hardware and OS metrics. You can control which collectors are active using command-line flags:

    • Enable a collector: Use the --collector.<name> flag.
    • Disable a default collector: Use the --no-collector.<name> flag.

    Collectors are categorized by whether they are enabled by default or require explicit activation.

    # Example: Enabling a specific collector and disabling a default one
./node_exporter --collector.some_collector --no-collector.default_collector

  11. Build base image v1.2

    master

    The v1.2 build updates Prometheus to version 3.4.2. It builds upon the v1.1 image and includes tzdata.

    # 1. Run base container (from v1.1)
docker run --name open-monitor-base -itd --privileged=true ccr.ccs.tencentyun.com/webankpartners/wecube-prometheus:v1.1 /bin/sh

# 2. Install tzdata
docker exec -it open-monitor-base /bin/sh
# Inside container:
apk add -U tzdata
exit

# 3. Copy new Prometheus
docker cp prometheus-3.4.2.linux-amd64.tar.gz open-monitor-base:/app/monitor/

# 4. Update Prometheus
docker exec -it open-monitor-base /bin/sh
# Inside container:
cd /app/monitor
tar zxf prometheus-3.4.2.linux-amd64.tar.gz
rm -rf prometheus
mv prometheus-3.4.2.linux-amd64 prometheus
rm -f *.gz
exit

# 5. Export, Import, and Push
docker export -o prometheus-3.4.2.tar open-monitor-base
docker import prometheus-3.4.2.tar prometheus-base:v1.2
docker login --username=XXXX ccr.ccs.tencentyun.com
docker tag prometheus-base:v1.2 ccr.ccs.tencentyun.com/webankpartners/wecube-prometheus:v1.2
docker push ccr.ccs.tencentyun.com/webankpartners/wecube-prometheus:v1.2

  12. Use recording rules to maintain metric compatibility

    master

    To avoid breaking existing alerts or dashboards when upgrading node_exporter to 0.16.0+, you can use Prometheus recording rules to create duplicate metrics.

    • To translate old metrics to new ones: Use example-16-compatibility-rules.yml.
    • To translate new metrics back to the old format: Use example-16-compatibility-rules-new-to-old.yml.

    Note: Using recording rules creates extra data volume and re-aligns timestamps.