SuzieQ Documentation

repository·develop·Indexed 21 days ago

https://github.com/netenglabs/suzieq

An open-source network observability framework and application (version 0.24.0) designed to collect, normalize, and analyze multi-vendor network data. SuzieQ supports platforms including Cumulus, EOS, IOS, IOSXE, IOSXR, JunOS, Palo Alto Panos, NXOS, SONIC, and Linux servers. It provides a CLI, GUI, REST API, and Python library to perform path analysis, network validation via asserts, and state summaries across routing tables, BGP, OSPF, and EVPN.

Tokens
42.6K
Snippets
152
Records
207
Agent score
74%

What's inside SuzieQ

  1. What is SuzieQ and how does it work?

    develop

    SuzieQ is an open-source, multi-vendor network observability platform designed for network engineers and designers. It functions as both a framework and an application to improve network understanding through observability.

    Core Workflow:

    1. Data Gathering: Uses an agentless model via SSH or REST API to collect data from routers, bridges, and Linux servers.
    2. Normalization: Converts vendor-specific data into a vendor-agnostic format.
    3. Storage: Stores normalized data in Parquet files (a big data format).
    4. Analysis: Provides access to the data through a CLI, GUI, REST API, or Python library.

    Supported Platforms:

    • Arista: EOS
    • Cisco: IOS, IOS-XE, IOS-XR, NXOS
    • Cumulus: Linux
    • Juniper: Junos (QFX, EX, MX, SRX, and Evolved OS)
    • Palo Alto: Panos (version 8.0 or higher)
    • Other: SoNIC devices and Linux servers.
  2. SuzieQ Core Applications Overview

    develop

    Once installed, SuzieQ provides four primary applications for network observability:

    ApplicationPurpose
    sq-pollerGathers data from routers, bridges, and Linux servers.
    suzieq-guiThe GUI front end for viewing, querying, and analyzing data.
    suzieq-cliThe CLI front end for viewing, querying, and analyzing data.
    sq-rest-serverThe REST API server for programmatic access to SuzieQ data.
  3. Navigate the Path page layout

    develop

    The Path interface is divided into two main areas:

    1. Sidebar: Used for user input and configuring the path simulation.
    2. Main Window: Displays the resulting path data, which includes:
      • Path Summary: An overview of the simulated path.
      • Graphical View: A visual graph representing the path from source to destination.
      • Failure Tables: Detailed tables for failed resources, categorized by devices, interfaces, or protocols.
      • Tabular Path: A structured, text-based representation of the path.
  4. Handle sensitive data in the inventory

    develop

    To avoid storing secrets in plain-text within your inventory file, SuzieQ supports three methods for usernames, passwords, passphrases, and tokens:

    • plain:<password>: Stores the sensitive information as is.
    • env:<ENV_VARIABLE>: Retrieves the value from a specified environment variable.
    • ask: Prompts the user for the information via stdin at runtime.
  5. Configure Service Polling and State Change Detection

    develop

    When defining a service, you can control how SuzieQ observes state changes:

    • period: An integer defining the polling frequency in seconds. Defaults to 15 if omitted.
    • type: Set to counters if the command output is expected to change every time it is run (e.g., incrementing counters). This tells SuzieQ to skip change detection to optimize performance.
    • ignored-fields: A list of field names to exclude when determining if a state change has occurred. This is useful for fields that change constantly but don't represent a meaningful state transition (e.g., lastUpTime or relative timestamps).
    • keys: A list of fields that uniquely identify a record (e.g., interface for interface counters). If a device produces multiple records for a service, keys is mandatory. SuzieQ automatically adds namespace and hostname to the keys.
  6. Implement Data Extraction via `normalize` or `textfsm`

    develop

    The apply section determines how raw command output is converted into structured data. You must choose one of two methods:

    1. Structured JSON Output (normalize): If the command returns JSON, use the normalize key. It accepts a list of xpath-like fields to map and transform the nested JSON data into a flat structure.
    2. Unstructured Text Output (textfsm): If the command returns plain text, use the textfsm key. This specifies the location of a TextFSM template used to parse the text into a structured format. If the path is not fully specified, it is assumed to be in the config/textfsm directory.

    Device Matching Logic

    When SuzieQ looks for an apply configuration for a device, it follows this priority:

    1. Matches a specific hostname provided in the apply dictionary.
    2. Matches a device type (e.g., eos, nxos, linux) identified during startup.
    3. If no match is found, the service extraction is skipped for that device.
  7. How SuzieQ handles timestamps and data capture

    develop

    Understanding how SuzieQ records time is critical for accurate network observability:

    • Timestamp Origin: Timestamps are recorded from the perspective of the poller. The time recorded is when the result of a command execution is received by the poller, not necessarily when the command was sent. All timestamps use the UTC timezone.
    • Change-based Storage: To save space, SuzieQ only saves a record if the data has changed compared to the previous poll. If the data is identical, no new record is stored.
    • Handling Volatile Fields: To prevent constant re-saving of data due to minor changes, SuzieQ applies specific logic:
      • Uptimes: Converted into a timestamp so they don't trigger a 'change' every poll.
      • Protocol States: Normalized (e.g., BGP states are often simplified to up or down) to avoid noise from transient states.
      • Reboots: If a node reboots and its uptime changes significantly, the poller clears in-memory records and re-saves all data for that node.
    • Counters: Services marked as counter in the schema (like interface counters) are saved every poll interval regardless of whether a change is detected.
  8. Testing the poller code

    develop

    The poller is primarily tested indirectly through the automatic generation of test data (using run-once=gather).

    Limitations:

    • This method does not catch issues related to the poller's behavior over time, such as how it handles device reboots or long-running state changes.
    • It focuses on the data generation phase rather than the continuous polling lifecycle.
  9. Normalize Field Names and Values

    develop

    Normalization allows you to unify data from different device vendors into a consistent schema.

    Field Name Normalization

    Map different vendor-specific names to a single common name. For example, mapping bytesReceived, rxBytes, and inBytes all to a single rxBytes field.

    Value Normalization

    Use simple arithmetic transformations to unify values:

    • Unit Conversion: Convert time fields (e.g., milliseconds to seconds) or counter units.
    • Field Unification: Combine multiple fields into one. For example, if one device provides totalLinkTransitions and another provides totalLinkUps and totalLinkDowns, you can sum the latter two to match the former.

    Note: To ensure consistency (especially when fields are missing), always specify a default value for non-string fields.

  10. Understand the SuzieQ CLI command structure

    develop

    The SuzieQ CLI follows a consistent command pattern used in both the interactive modal CLI and the REST API. Commands are structured as:

    **<table> <verb> <filters>**

    CLI Modes

    1. Modal CLI: Enter the interactive shell by running suzieq-cli. This mode provides auto-completion as you type.
    2. Linux CLI: Execute commands directly from your terminal using the standard -- option method.

    Command Components

    • Table: The data category you want to act upon (e.g., bgp, interface, route).
    • Verb: The action to perform on that table (e.g., show, summarize, top).
    • Filters: Arguments used to narrow down the results (e.g., hostname, namespace, vrf).

    Note: Most verbs accept all available filters, with the exception of summarize, which does not support the columns filter.

    suzieq> bgp show hostname=router1
  11. Use the `show` verb to retrieve basic data

    develop

    The show verb is the most fundamental command in SuzieQ. It returns the most basic data associated with a specific service or command. Many commands have a direct mapping to a database table, and the show verb will display data from that table.

    Key features:

    • Filtering: Like most SuzieQ commands, show supports filtering (e.g., by namespace, hostname, or type).
    • Column Selection: You can use the columns= filter to specify exactly which fields you want to see. Use columns='*' to display all available columns for that table.
    suzieq> device show namespace=nxos
    
    suzieq> bgp show namespace=ospf-ibgp columns='hostname vrf peer peerHostname state asn bfdStatus updateSource'
  12. Future Support: Cross-Table Augmented Columns

    develop

    While not supported in the current Phase 1 implementation, the schema model is designed to support augmented columns that pull data from other tables using a table:field syntax in the depends key.

    Example of intended future syntax:

    {
        "name": "os",
        "type": "string",
        "description": "network os that created this entry",
        "depends": "device:os"
    }

    In this example, the os column would be derived from the os field located in the device table.