Prometheus Node Exporter

The node_ntp_sanity metric provides an aggregate health score for the NTP daemon. It evaluates several factors:

1. Stratum and Leap Flag: Ensures the clock is synchronized.
2. Freshness: Ensures the last adjustment was recent.
3. Root Distance: Must be less than the collector.ntp.max-distance threshold.
4. Causality Violation: Must be less than the collector.ntp.local-offset-tolerance threshold.

Causality Violation Calculation: Causality violation is a lower bound estimate of clock error calculated using SNTP. It is defined as the positive portion of: abs(node_ntp_offset) - node_ntp_rtt / 2

The timex collector exports the state of the kernel time synchronization flag. This flag is maintained by a time-keeping daemon and is raised by the Linux kernel if the daemon fails to update it regularly.

Usage Notes:

• Chrony: Modern versions work correctly, but all versions require the rtcsync option in the configuration to maintain this flag. Older versions (e.g., chrony-1.30 on Debian/jessie) may clear the STA_UNSYNC flag during initialization and fail to indicate status correctly.
• OpenNTPD: Does not support this flag until version 5.9p1.
• systemd-timesyncd: To monitor if systemd-timesyncd is functioning correctly, use a combination of the sync_status and offset metrics exported by this collector.

To build the node_exporter binary from the source repository, ensure you have the Go compiler installed. If you are on RHEL/CentOS, you must also install the glibc-static package. Use make build to compile the binary, then execute the resulting ./node_exporter with any desired configuration flags.

git clone https://github.com/prometheus/node_exporter.git
cd node_exporter
make build
./node_exporter <flags>

When running node_exporter in a container to monitor the host system, you must ensure the exporter has access to the host's namespaces and filesystem.

Key requirements:

1. Use --net="host" and --pid="host" to access host network and process namespaces.
2. Bind-mount the host root filesystem into the container (e.g., -v "/:/host:ro,rslave").
3. Use the --path.rootfs flag to tell node_exporter where the host filesystem is mounted inside the container. This flag acts as a prefix for filesystem access.
4. Any non-root mount points you wish to monitor must also be explicitly bind-mounted into the container.
5. If using the timex collector, you may need to add the --cap-add=SYS_TIME capability to the container.

docker run -d \
  --net="host" \
  --pid="host" \
  -v "/:/host:ro,rslave" \
  quay.io/prometheus/node-exporter:latest \
  --path.rootfs=/host

Example scripts for the Node Exporter textfile collector have been moved to a dedicated community repository. You can find them at:

https://github.com/prometheus-community/node-exporter-textfile-collector-scripts

Starting with version 0.16.0, node_exporter renamed many metrics to comply with Prometheus naming best practices. This breaking change affects existing dashboards and queries. To manage the transition, you can use one of the following three strategies:

1. Update Dashboards: Modify Grafana dashboards to use multiple queries so they can display both old and new metric formats simultaneously during the transition.
2. Use Recording Rules: Use provided recording rule sets to translate metrics between formats. This creates duplicate metrics, allowing old queries to continue working, but increases data volume and may re-align timestamps.
3. Run Dual Versions: Run the old and new versions of the exporter on different ports and add an additional scrape job to Prometheus. It is recommended to only enable collectors with name changes that are relevant to your monitoring.

Node Exporter uses collectors to gather different types of system metrics. You can control which collectors are active using command-line flags:

• Enable a collector: Use --collector.<name>.
• Disable a default collector: Use --no-collector.<name>.
• Enable only specific collectors: Disable all defaults first, then enable only what you need using --collector.disable-defaults --collector.<name> ....

Note on performance: When enabling collectors that are disabled by default, monitor scrape_duration_seconds to ensure collection doesn't time out, and scrape_samples_post_metric_relabeling to monitor changes in cardinality.

# Example: Enable only cpu and meminfo
./node_exporter --collector.disable-defaults --collector.cpu --collector.meminfo

You can optionally pass CLI arguments to the node_exporter LaunchDaemon by creating a file at /usr/local/etc/node_exporter.args. The daemon will read these arguments upon startup.

echo -- '--web.listen-address=:9101' | sudo tee /usr/local/etc/node_exporter.args

To run node_exporter as a system-wide daemon on macOS, you must place the binary in /usr/local/bin/ and the configuration .plist file in /Library/LaunchDaemons/. After copying the files, use launchctl bootstrap to register the service with the system.

Note: If you installed via a package manager, these manual steps are likely unnecessary.

sudo cp -n node_exporter /usr/local/bin/
sudo cp -n examples/launchctl/io.prometheus.node_exporter.plist /Library/LaunchDaemons/
sudo launchctl bootstrap system/ /Library/LaunchDaemons/io.prometheus.node_exporter.plist

The perf collector exposes performance metrics. On Linux, it may require adjusting sysctl settings to allow access to kernel/user measurements.

Setup requirements: Set kernel.perf_event_paranoid to a value that allows the necessary access (typically 0 for the most complete set):

sysctl -w kernel.perf_event_paranoid=0

Configuration options:

• Specify CPUs: By default, it collects for CPUs running node_exporter. Use --collector.perf.cpus to specify a range or stride (e.g., 2-6 or 1-10:5).
• Tracepoints: Use --collector.perf.tracepoint="<name>" to collect specific tracepoint counts (e.g., --collector.perf.tracepoint="sched:sched_process_exec").

# Example: Collect perf metrics on CPUs 2 through 6
./node_exporter --collector.perf --collector.perf.cpus=2-6