DOCSEARCHbyAlgolia
Submit Docs

NetApp Harvest

repository·main·Indexed 41 days ago

https://github.com/netapp/harvest

An open-source observability tool that collects, transforms, and exports performance, capacity, and hardware metrics from NetApp and Cisco storage and networking devices to time-series databases such as Prometheus.

Tokens
553.7K
Snippets
985
Records
1.5K
Agent score
49%

What's inside NetApp Harvest

  1. Overview of the Harvest Model Context Protocol (MCP) Server

    main

    The Harvest MCP server is a lightweight service that enables LLM-based MCP clients (such as GitHub Copilot or Claude Desktop) to interact with your infrastructure monitoring data. It acts as a bridge between natural language queries and your time-series database (Prometheus or VictoriaMetrics) where Harvest stores metrics from ONTAP, StorageGRID, E-Series, and Cisco Nexus Switches.

    Core Functionality

    • Natural Language Queries: Allows users to ask questions about infrastructure health, capacity, and performance.
    • Data Access: Connects directly to your existing Prometheus or VictoriaMetrics instance.
    • Standardized Interface: Provides a standardized MCP interface for intelligent data analysis and insights.

  2. What is NetApp Harvest?

    main

    NetApp Harvest is an open-metrics endpoint designed for ONTAP, StorageGRID, E-Series, and Cisco Nexus Switches. It functions by collecting performance, capacity, and hardware metrics, transforming them, and routing them to a time-series database of your choice.

    Key features include:

    • Observability: Provides datacenter insights via included Grafana dashboards.
    • Extensibility: Allows for the collection of new metrics by editing included template files.
    • Flexibility: Offers customizable collection, augmentation, and export workflows.

  3. Overview of NetApp Harvest

    main

    NetApp Harvest is an open-metrics endpoint designed to bring observability to NetApp and Cisco infrastructure, including:

    • ONTAP
    • StorageGRID
    • E-Series
    • Cisco Nexus Switches

    Harvest functions by collecting performance, capacity, and hardware metrics, transforming them, and routing them to a time-series database of your choice. It provides a complete observability pipeline including pollers, collectors, templates, exporters, and an optional auto-discover daemon. It is designed to work with separately installed instances of Prometheus/InfluxDB and Grafana, providing out-of-the-box dashboards for the supported platforms.

  5. Overview of the StatPerf Collector

    main

    The StatPerf collector gathers performance metrics from ONTAP by executing the statistics show -raw -object $object command via the private Rest CLI. It is intended for environments where ZapiPerf, RestPerf, or KeyPerf collectors cannot be utilized.

    Requirements:

    • Permissions: Requires specific ONTAP permissions. See the StatPerf Collector Permissions documentation.
    • Multi-Admin Verification (MAV): If MAV is enabled in your cluster, you must allow diagnostic mode queries in the MAV rules to permit the collector to run.

  6. Overview of the Prometheus Exporter

    main

    The Prometheus exporter in Harvest performs two primary functions:

    1. Metric Formatting: It converts Harvest metrics into the Prometheus line protocol.
    2. Web Endpoint Hosting: It creates a web endpoint at http://<ADDR>:<PORT>/metrics (or https:// if TLS is enabled) for Prometheus to scrape.

    Additionally, the exporter serves an overview of all available metrics and collectors at the root address scheme://<ADDR>:<PORT>/.

    Note: Because Prometheus uses a polling model, you must configure Prometheus to scrape each individual poller's endpoint.

  7. Overview of the EMS collector

    main

    The EMS collector collects ONTAP Event Management System (EMS) events via the ONTAP REST API. It uses YAML template files to define which events to collect, which properties to export as metrics, and which labels to attach.

    Key Concepts:

    • Metrics: For every received EMS event, the collector creates metrics prefixed with ems_events.
    • Normal EMS events: Single-shot events that occur when a problem is detected. ONTAP does not send a separate event when the issue is resolved.
    • Bookend EMS events: Events that occur in matching pairs (e.g., an 'issue detected' event and a 'resolved' event). Harvest can use these pairs to track the lifecycle of an event.
    • Template Customization: You can collect new events or add labels by editing the default template or by extending existing templates.

    Note: Events that occurred while the EMS collector was not running will not be captured.

  8. What is the Matrix package and how does it work?

    main

    The matrix package provides the matrix.Matrix data structure, which serves as the architectural backbone of Harvest. It is used for the storage, manipulation, and transmission of both numeric and non-numeric (string) data between collectors, plugins, and exporters.

    Conceptually, a Matrix is a 2-dimensional array of metrics and instances:

    • Metrics (matrix.Metric): These represent the rows. Metrics are typed (e.g., int64, float64) and contain the actual numeric data. They can also hold labels (useful for histograms).
    • Instances (matrix.Instance): These represent the columns. Instances act as pointers to the data and store non-numeric metadata via labels (*dict.Dict).

    Because the Matrix uses hash tables for access, every metric and instance added must have a unique key.

  9. Understand the Matrix and Tree data structures

    main

    Harvest relies on two primary data structures for handling information:

    Matrix

    The Matrix is the backbone of Harvest. It provides storage for numerical and label data. All collectors and exporters interact via the Matrix.

    • Usage: Collectors write metrics to it; Exporters read from it.
    • Constraints: Metrics must be typed, histograms must be converted to 'flat' metrics, and instance/metric names must be unique.

    Tree

    The Tree (type *node.Node in pkg/tree/node/node.go) is used for unstructured and untyped data.

    • Usage: It is primarily used for API calls and template configuration.
    • Workflow: Collectors often parse raw data (like XML) into a Tree, then extract specific information from the Tree to write into a Matrix.

  10. Use endpoints to enhance data collection

    main

    The endpoints section allows you to perform additional REST queries to augment the data collected by the primary query. This is useful for collecting metrics not available in the main endpoint or for querying private CLI endpoints.

    Key Concepts

    • Joining Data: Use the ^^ notation in the counters section of both the main query and the endpoint to link them via a common key (e.g., ^^uid).
    • instance_add: If set to true, the endpoint will create new records in the matrix rather than modifying existing ones. This is required when collecting related objects that cannot be retrieved in a single call (e.g., collecting both volumes and their flexgroup constituents).

    Example: Joining Disk and Private CLI data

    query: api/storage/disks
object: disk

# Main query provides the base matrix
counters:
  - ^^uid => uuid
  - ^name => disk

# Endpoint provides additional metrics linked by uuid
endpoints:
  - query: api/private/cli/disk
    counters:
      - ^^uid => uuid
      - disk_io_kbps_total => stats_io_kbps
    endpoints:
  - query: api/private/cli/disk
    counters:
      - ^^uid                   => uuid
      - ^type
      - disk_io_kbps_total      => stats_io_kbps
      - sectors_read            => stats_sectors_read
      - sectors_written         => stats_sectors_written

  11. Understand Harvest's release and support lifecycle

    main

    NetApp releases Harvest on a quarterly schedule (February, May, August, and November).

    Key support policies:

    • Release Support Duration: Each Harvest release is intended to be supported for 12 months. For example, a release in YY.MM is supported until YY+1.MM is released.
    • ONTAP Compatibility: Each release supports the most recently released version of ONTAP. Support for older ONTAP versions is attempted but not guaranteed; breaking changes regarding ONTAP compatibility will be noted in the changelog.
    • Upgrade Requirement: To receive support from NetApp, you must upgrade if your current Harvest version is more than 12 months old. It is recommended to run the latest version to benefit from frequent feature updates and bug fixes.

  12. Understand CiscoRest parameter hierarchy and overrides

    main

    CiscoRest parameters are distributed across three levels of configuration. Parameters defined at a lower level override those defined at a higher level, allowing for global defaults with object-specific overrides.

    1. Harvest configuration file (e.g., harvest.yml): Defines the poller, target address, credentials, and datacenter.
    2. CiscoRest configuration file (default: conf/ciscorest/default.yaml): Defines global defaults for all objects and maps object names to their template files.
    3. Object configuration file (located in conf/ciscorest/nxos/$version/): Defines specific queries and parameters for an individual object.

    Note: The addr and datacenter parameters cannot be overridden in the lower-level CiscoRest or Object configuration files; they must be defined in the Harvest configuration file.