Intel PerfSpect

PerfSpect is a performance analysis tool for Linux systems designed to collect system configuration data and hardware performance metrics. It supports both local execution and remote execution via SSH. The architecture is divided into several layers:

• CLI Layer: Provides commands like report, benchmark, telemetry, flamegraph, lock, metrics, config, update, and extract.
• ReportingCommand Framework: Orchestrates common workflows for most reporting commands, handling target setup, parallel data collection, and report generation.
• Target Layer: Provides an abstraction for interacting with systems, allowing the same collection code to run on LocalTarget (local machine) or RemoteTarget (via SSH).
• Script Engine: Manages the execution of embedded collection scripts and their dependencies on targets using a generated controller.sh.
• Report Engine: Processes collected data into various formats including txt, json, html, and xlsx.

The metrics command uses a specialized signalManager to coordinate shutdowns. Unlike reporting commands that attempt a graceful shutdown, the metrics command prioritizes immediate termination of child processes because it streams perf stat output in real-time, meaning data is already captured incrementally.

Signal Flow

When a signal (like SIGINT from Ctrl-C or SIGTERM) is received, or an internal error occurs:

1. The signalManager cancels the internal context.Context.
2. It immediately sends SIGKILL to all child processes via util.SignalChildren(syscall.SIGKILL).
3. The collection loop (collectOnTarget) detects shouldStop() == true and exits.
4. The processing pipeline (processPerfOutput) detects the context cancellation, drains any remaining data, and the command outputs the metrics collected so far before exiting.

Remote Target Behavior

When running against a remote target, the metrics command kills the local SSH process. This causes the remote perf stat process to become orphaned. This is considered acceptable because the metrics are streamed incrementally, and the remote process will eventually be cleaned up by the OS or the target's temporary directory cleanup on the next run.

                     perfspect (Go)
                     ┌─────────────────────────────────────────────┐
                     │  signalManager                              │
                     │  - ctx/cancel (context.Context)             │
                     │  - sigChannel (SIGINT, SIGTERM)             │
                     │  - handleSignals() goroutine                 │
                     └──────────┬──────────────────────────────────┘
                                │
            ┌───────────────────┼───────────────────┐
            ▼                   ▼                   ▼
     Target A              Target B              Target N
     ┌──────────┐          ┌──────────┐          ┌──────────┐
     │ perf stat│          │ perf stat│          │ perf stat│
     │(streamed)│          │(streamed)│          │(streamed)│
     └──────────┘          └──────────┘          └──────────┘

Tables are used to define exactly what data should be collected and how to extract it from script outputs. A TableDefinition maps specific scripts to specific data fields.

Key Fields in TableDefinition:

• Name: The name of the table.
• ScriptNames: A list of scripts that provide the necessary data.
• Fields: A list of FieldDefinition objects describing the data points to extract.
• Architectures, Vendors, MicroArchitectures: Optional filters to limit data collection to specific hardware profiles.

Data Extraction: Field values are retrieved using a ValuesFunc, which supports regex extraction, JSON parsing, or custom logic to transform raw script output into structured data.

type TableDefinition struct {
    Name              string
    ScriptNames       []string           // Scripts that provide data for this table
    Fields            []FieldDefinition  // Fields to extract from script outputs
    Architectures     []string           // Optional: limit to specific architectures
    Vendors           []string           // Optional: limit to specific vendors
    MicroArchitectures []string          // Optional: limit to specific microarchitectures
}

PerfSpect utilizes goroutines to manage parallel operations across three main areas:

1. Multi-target collection: Each target is assigned its own goroutine for independent operation.
2. Script execution: Within a single target, scripts are managed by controller.sh. Scripts marked as concurrent run in parallel, while sequential scripts are executed one-by-one.
3. Signal handling: A dedicated goroutine monitors for SIGINT or SIGTERM to coordinate a graceful shutdown across all active targets.

Graceful Shutdown Flow: When a signal like SIGINT is received, the signal handler:

1. Activates the signal handler goroutine.
2. Sends SIGINT to the controller.sh PID for every active target.
3. Waits for the controllers to exit (subject to a timeout).
4. If the timeout is reached, it sends SIGKILL to ensure processes are terminated.
5. Prints any partial results collected before the shutdown.

The Target interface abstracts the difference between local and remote systems. This allows all data collection logic to remain agnostic of whether it is running on the local machine or a remote host via SSH.

Key capabilities provided by the Target interface include:

• Command execution (RunCommand, RunCommandEx)
• File transfers (PushFile, PullFile)
• Remote environment management (CreateTempDirectory)
• Connectivity validation (CanConnect)
• Privilege elevation and architecture detection.

type Target interface {
    CanConnect() bool
    RunCommand(cmd *exec.Cmd) (stdout, stderr string, exitCode int, err error)
    RunCommandEx(cmd *exec.Cmd, timeout int, newProcessGroup bool, reuseSSH bool) (...)
    PushFile(srcPath, dstPath string) error
    PullFile(srcPath, dstDir string) error
    CreateTempDirectory(rootDir string) (string, error)
    // ... additional methods for privilege elevation, architecture detection, etc.
}

Most high-level commands (such as report, telemetry, flamegraph, and lock) utilize the ReportingCommand framework to ensure a consistent execution lifecycle.

A ReportingCommand encapsulates:

• Tables: A list of table.TableDefinition objects specifying what data to collect.
• ScriptParams: Parameters passed to the collection scripts.
• SummaryFunc/InsightsFunc: Optional functions to build summaries or generate recommendations from the data.
• AdhocFunc: Optional post-collection actions.

The Workflow Lifecycle:

1. Initialization: Parse flags and validate inputs.
2. Target Setup: Initialize targets (local or remote via --target/--targets flags).
3. Parallel Collection: For each target, copy scripts/dependencies, run the collection controller, and retrieve outputs in parallel.
4. Report Generation: Orchestrate the creation of reports in requested formats (txt, json, html, xlsx).
5. Cleanup: Run optional adhoc actions.

type ReportingCommand struct {
    Cmd          *cobra.Command
    Tables       []table.TableDefinition  // What data to collect
    ScriptParams map[string]string        // Parameters for scripts
    SummaryFunc  SummaryFunc              // Optional: build summary from collected data
    InsightsFunc InsightsFunc             // Optional: generate insights/recommendations
    AdhocFunc    AdhocFunc                // Optional: post-collection actions
}