PuppetDB

repository·main·Indexed 43 days ago

https://github.com/puppetlabs/puppetdb

A fast, scalable, and reliable data warehouse for Puppet that caches Puppet-generated data and provides an API for advanced querying and data retrieval. It features PostgreSQL inheritance-based partitioning for timeseries data and includes tools for load simulation and performance testing.

Tokens
104.3K
Snippets
309
Records
470
Agent score
51%

What's inside PuppetDB

  1. Understand the PuppetDB Query API structure

    main

    The PuppetDB query API is implemented as HTTP GET requests. A typical request URL follows this pattern:

    https://<SERVER>:<PORT>/pdb/query/<API_VERSION>/<ENDPOINT>?<PARAMETER>=<VALUE>

    Components

    • API Version: Currently uses v4 (e.g., /pdb/query/v4).
    • Endpoints: Represent PuppetDB entities (e.g., resources, nodes). Each endpoint may have sub-endpoints that act as shortcuts for common queries.
    • URL Parameters: Used to configure behavior. Parameters are separated by & and must be URL-encoded.

    Querying Methods

    There are two primary ways to structure a request:

    1. Endpoint-specific: https://puppetdb:8081/pdb/query/v4/<ENDPOINT>?query=<QUERY_STRING>
    2. Entity context (Root): https://puppetdb:8081/pdb/query/v4?query=<QUERY_STRING> (Note: Only the root endpoint supports PQL).

    Response Format

    All queries return data with the application/json content type.

    https://puppetdb:8081/pdb/query/v4/resources?limit=50&offset=50
  2. What is Session Logging in PuppetDB?

    main

    By default, PuppetDB only logs successfully negotiated HTTP or HTTPS connections. Connections that fail before reaching the application layer (e.g., SSL handshake failures) are closed without a log entry.

    Enabling Session Logging allows you to see and inspect these failed connection attempts. This is useful for troubleshooting unexpected traffic or connections that fail to work as expected.

    Warning: Session logging is extremely noisy and can impact the availability of the PuppetDB node. It should only be enabled during active troubleshooting and disabled immediately afterward.

  3. Overview of PuppetDB capabilities and data

    main

    PuppetDB is a data store that collects and indexes data generated by Puppet®. It provides a searchable database of every resource being managed across your infrastructure.

    Key capabilities:

    • Advanced Puppet Features: Enables features like exported resources, allowing nodes to manage resources on other nodes.
    • Inventory & Metadata: Provides a massive inventory of node metadata.
    • Queryable API: While Puppet uses a subset of the data for exported resources, the full dataset is accessible via PuppetDB's query APIs.

    Data stored in PuppetDB:

    • The most recent facts from every node.
    • The most recent catalog for every node.
    • Event reports for every node (retention is configurable, defaulting to 14 days).
  4. Understand the Catalog wire format version 6

    main

    PuppetDB receives catalogs from Puppet Servers using version 6 of the catalog interchange format. This format is a JSON-serialized representation of a Puppet catalog, specifically converted by the PuppetDB catalog terminus to ensure compatibility.

    Key constraints:

    • Encoding: Must be strict UTF-8.
    • Null values: null is generally not allowed anywhere in the catalog, with the specific exception of the transaction_uuid field, which may be null.
  5. Understand the `/pdb/query/v4/event-counts` response format

    main

    The response is a JSON array of maps. Each map represents an object and its associated event counts. The structure of the subject map depends on the summarize_by value:

    • summarize_by=certname: subject contains {"title": "<certname>"}.
    • summarize_by=resource: subject contains {"type": "<type>", "title": "<title>"}.
    • summarize_by=containing_class: subject contains {"title": "<class_name>"}.

    Puppet Enterprise (PE) Specifics

    In Puppet Enterprise, successes and noops are subdivided into intentional and corrective changes based on the corrective_change flag. The following fields are available:

    • intentional_successes
    • corrective_successes
    • intentional_noops
    • corrective_noops

    These PE-specific fields can also be used within the counts_filter parameter.

    // Example response for summarize_by=resource
    [
      {
        "subject_type": "resource",
        "subject": { "type": "Notify", "title": "Foo happened" },
        "failures": 0,
        "successes": 1,
        "noops": 0,
        "skips": 0
      }
    ]
  6. Use dot notation to query structured JSON data

    main

    PuppetDB supports dot notation to query nested JSON structures (hashes and arrays) within certain endpoints like inventory and resources.

    Rules for dot notation:

    • Hash descendence: Represented by a period-separated sequence of key names (e.g., facts.system_uptime.days).
    • Array indexing: Supported on the inventory endpoint using brackets (e.g., facts.processors.models[0]).
    • Quoting: If a field component contains a dot, it must be double-quoted (e.g., facts."x.y".z). In AST queries, these quotes must be JSON-escaped.

    Note: Dot notation for hash descendence is currently under development and may be expanded to other endpoints.

    ["=", "facts.system_uptime.days", 0]
    ["~", "facts.processors.models[0]", "Intel.*"]
  7. Perform subqueries using select_<ENTITY> operators

    main

    PuppetDB allows you to perform subqueries to combine data from different endpoints (resources, facts, nodes, and fact-contents) in a single query. This is useful when the result of one query (e.g., finding all nodes with a specific class) needs to be used as a filter for another query (e.g., finding the IP addresses of those specific nodes).

    Available subquery operators:

    • select_resources: Takes a resource query and returns the resulting resources.
    • select_facts: Performs a subquery against the facts endpoint.
    • select_nodes: Performs a subquery against the nodes endpoint.
    • select_fact_contents: Performs a subquery against the fact-contents endpoint.

    Subqueries can be nested and used from any queryable endpoint.

    ["and",
      ["=", "name", "ipaddress"],
      ["in", "certname",
        ["extract", "certname", ["select_resources",
                                   ["and",
                                     ["=", "type", "Class"],
                                     ["=", "title", "Apache"]
                                   ]]]
      ]
    ]
  8. Querying resources with PQL

    main

    The resources query type allows you to find Puppet resources.

    Basic Syntax

    • All resources: resources {}
    • Filtered resources: resources { type = "User" and title = "nick" }
    • Logical Operators: Use and, or, and ! (NOT). Example to exclude a specific definition: resources { type = "User" and title = "nick" and !(file = "/path/to/file" and line = 111) }.
    • Regular Expressions: Use the ~ operator for pattern matching. Example: resources { certname ~ 'www\d+.example.com' }.

    Projection (Choosing fields)

    To limit the returned fields, use the syntax resources[field1, field2, ...]. Example: resources[certname, type, title] { type = "Class" and title = "Apache" }.

    Resource Result Object

    Each resource in the result array contains:

    • parameters: An object containing all resource parameters and values.
    • line: The line number where the resource was declared.
    • file: The file where the resource was specified.
    • exported: Boolean indicating if the resource was exported.
    • tags: An array of tags on the resource.
    • title: The resource title.
    • type: The resource type.
    • resource: The internal PuppetDB identifier.
    • certname: The node that owns the resource.
    # Find a specific user
    resources { type = "User" and title = "nick" }
    
    # Find all 'www' servers using regex
    resources { certname ~ 'www\d+.example.com' }
    
    # Return only specific fields
    resources[certname, type, title] { type = "Class" and title = "Apache" }