cAdvisor (Container Advisor)

repository·master·Indexed 12 days ago

https://github.com/google/cadvisor

A running daemon that collects, aggregates, processes, and exports resource usage and performance characteristics for running containers. It provides insights into resource isolation parameters, historical usage, and network statistics at both the container and machine level. Features include a built-in web UI, a versioned REST API with a Go client, and support for multiple storage drivers including Prometheus, BigQuery, ElasticSearch, InfluxDB, Kafka, Redis, and StatsD.

Tokens
33.9K
Snippets
107
Records
148
Agent score
95%

What's inside cAdvisor

  1. Understand the cAdvisor transition to OpenTelemetry (Otel) Collector

    master

    The cAdvisor project is moving toward a maintenance mode with a planned transition of its standalone functionality to the OpenTelemetry (Otel) Collector.

    Key Changes in the Transition:

    • Standalone Mode: Instead of using the cAdvisor standalone binary, users are encouraged to move to Otel Collector using new receivers that collect similar container information.
    • Metrics Export: Otel Collector can be configured to export metrics to a Prometheus endpoint or any other supported exporter.
    • Metric Naming: Metric names may be renamed to align with Otel Semantic Convention (e.g., preferring dots over underscores).
    • Endpoints: While cAdvisor supports various endpoints, the Otel Collector approach will primarily expose a single Prometheus endpoint for all containers and will not support container-specific endpoints.
    • Events: The transition involves moving from a pull model to a push model, as Otel Collector collects events as Otel events rather than providing a dedicated events endpoint.
    • Runtimes: The goal is for Otel Collector to support the same environments and runtimes as cAdvisor (including non-Kubernetes environments) by implementing specialized receivers.
  2. Configure Housekeeping intervals

    master

    Housekeeping refers to the periodic actions cAdvisor takes to gather container stats. You can control the frequency and behavior of these actions.

    Dynamic Housekeeping:

    • --allow_dynamic_housekeeping=true: Allows cAdvisor to vary the housekeeping interval based on container activity. Disabling this provides predictable intervals but increases resource usage.

    Interval Settings:

    • --global_housekeeping_interval=1m0s: Interval for global housekeeping (used for detecting new containers).
    • --housekeeping_interval=1s: Interval for per-container housekeeping (used for gathering container stats).
    • --max_housekeeping_interval=1m0s: The maximum allowed interval between container housekeepings.
  3. Access the cAdvisor Web UI

    master

    cAdvisor provides a built-in web interface to visualize container resource usage. The UI is accessible at the hostname and port where cAdvisor is running:

    http://<hostname>:<port>/

    For more details on the interface, see docs/web.md.

  4. Deploy cAdvisor to Kubernetes using Kustomize

    master

    cAdvisor uses Kustomize to manage Kubernetes manifests. You can deploy cAdvisor to your cluster using a remote build by pointing Kustomize to the official repository with a specific version tag.

    Ensure you have Kustomize installed before proceeding.

    VERSION=v0.42.0
    kustomize build "https://github.com/google/cadvisor/deploy/kubernetes/base?ref=${VERSION}" | kubectl apply -f -
  5. Run the built cAdvisor binary

    master

    After building, you can run the cAdvisor binary using sudo.

    If you built with Perf Support, you must provide a performance events configuration file using the -perf_events_config flag to enable performance monitoring.

    # Standard run
    $ sudo ./cadvisor
    
    # Run with performance monitoring configuration
    $ sudo ./cadvisor -perf_events_config=perf/testing/perf-non-hardware.json
  6. Build the cAdvisor Docker container

    master

    To build the cAdvisor Docker container from source, run the provided build script. This script performs a static build of the cAdvisor binary and then packages it into a minimal Docker image named google/cadvisor:beta. Note that the beta image is highly stripped down, containing only the cAdvisor binary.

    $ ./deploy/build.sh
  7. Run cAdvisor in a Docker container

    master

    To quickly monitor your machine's containers using Docker, run the cAdvisor image with the necessary volume mounts to allow it to observe the host's filesystem, Docker state, and system statistics.

    Note: If you are using CentOS, Fedora, RHEL, or LXC, refer to the specific running instructions in docs/running.md.

    VERSION=0.55.1 # use the latest release version from https://github.com/google/cadvisor/releases
    sudo docker run \
      --volume=/:/rootfs:ro \
      --volume=/var/run:/var/run:ro \
      --volume=/sys:/sys:ro \
      --volume=/var/lib/docker/:/var/lib/docker:ro \
      --volume=/dev/disk/:/dev/disk:ro \
      --publish=8080:8080 \
      --detach=true \
      --name=cadvisor \
      --privileged \
      --device=/dev/kmsg \
      ghcr.io/google/cadvisor:$VERSION # for versions prior to v0.53.0, use gcr.io/cadvisor/cadvisor instead
  8. Deploy cAdvisor with perf support on Kubernetes

    master

    If you require cAdvisor with performance monitoring support, use the overlays/examples_perf overlay. This includes necessary modifications to the DaemonSet and the ConfigMap for perf events configuration.

    To generate and apply the daemonset with perf support patches:

    kubectl kustomize deploy/kubernetes/overlays/examples_perf | kubectl apply -f -
    kubectl kustomize deploy/kubernetes/overlays/examples_perf | kubectl apply -f -
  9. Run cAdvisor as a Standalone Binary

    master

    cAdvisor is distributed as a static Go binary with no external dependencies. You can run it directly on your host system. Note that running without root privileges may cause cAdvisor to gracefully degrade by omitting certain data sources it cannot access.

    $ sudo cadvisor
  10. Run cAdvisor on CentOS, Fedora, and RHEL

    master

    RHEL and CentOS environments often require additional permissions and specific volume mounts due to tightened security and cgroup hierarchy locations.

    General RHEL/CentOS Requirements

    • Use --privileged=true to allow cAdvisor to access the Docker daemon socket.
    • If cgroup hierarchies are mounted in /cgroup, add --volume=/cgroup:/cgroup:ro.

    RedHat 7 Specific Configuration

    If you are running on a RedHat 7 host, the standard Docker command may throw OCI errors. Use the following specific command instead:

    docker run \
    --volume=/:/rootfs:ro \
    --volume=/var/run:/var/run:rw \
    --volume=/sys/fs/cgroup/cpu,cpuacct:/sys/fs/cgroup/cpuacct,cpu \
    --volume=/var/lib/docker/:/var/lib/docker:ro \
    --publish=8080:8080 \
    --detach=true \
    --name=cadvisor \
    --privileged=true \
    google/cadvisor:latest
  11. Build cAdvisor from source

    master

    Once you have the source code and Go installed, navigate to the cAdvisor source directory and use make to build the binary or run tests.

    Note: Ensure you are in the $GOPATH/src/github.com/google/cadvisor directory.

    # Build the cAdvisor binary
    $ make build
    
    # Run only unit tests
    $ make test
  12. Configure Perf Events in cAdvisor

    master

    cAdvisor can measure CPU performance counters (hardware events like cache misses or retired instructions) using the kernel perf subsystem. This is configured via a JSON file passed to the --perf_events_config flag.

    Core Configuration Options

    • --perf_events_config="<path_to_json>": Path to the JSON configuration file. An empty value disables perf events.
    • --disable_metrics="percpu": Aggregates core perf events to reduce data volume.
    • --disable_metrics="": Exposes core perf events per CPU. Note that exposing many events per CPU may cause "too many opened files" errors; if this occurs, increase the system limit using ulimit -n <value>.

    JSON Configuration Schema

    The configuration file supports core and uncore (events outside the CPU core) sections. You can specify events by name (using libpfm4 supported names) or by providing specific hardware type and config values obtained from perf tools.

    To find specific hardware values, use perf stat -I 5000 -vvv -e EVENT_NAME and look for the perf_event_attr section.

    # Example command line invocation
    ./cadvisor --perf_events_config="/path/to/config.json" --disable_metrics="percpu"