Intel® Performance Counter Monitor (Intel® PCM)

The pcm-iio tool is used to monitor PCIe transactions. It provides a breakdown of metrics per PCIe bus (IIO stack) and/or per PCIe device.

Metrics are categorized into three groups:

Inbound (PCIe device DMA into system)

• IB write: Bytes per second requested by the PCIe device to write to main memory via DMA.
• IB read: Bytes per second requested by the PCIe device to read from main memory via DMA.

Outbound (CPU MMIO to the PCIe device)

• OB read: Bytes per second requested by the CPU to read from the PCIe device via MMIO.
• OB write: Bytes per second requested by the CPU to write to the PCIe device via MMIO.

IOMMU metrics

• IOTLB Lookup: IOTLB lookups per second.
• IOTLB Miss: IOTLB misses per second.
• Ctxt Cache Hit: Context cache hits per second.
• 256T Cache Hit: Second Level Page Walk Cache Hits to a 256T page per second.
• 512G Cache Hit: Second Level Page Walk Cache Hits to a 512G page per second.
• 1G Cache Hit: Second Level Page Walk Cache Hits to a 1G page per second.
• 2M Cache Hit: Second Level Page Walk Cache Hits to a 2M page per second.
• IOMMU Mem Access: IOMMU memory accesses per second.

To use the pcm.exe utility on Windows, you must first compile and install the Windows MSR driver (msr.sys). For systems with more than 64 logical cores, ensure you compile all binaries in x64 platform mode.

Prerequisites

1. Compile the MSR driver: Follow the driver compilation guide.
2. Sign the driver: For Windows 7 and later, you must sign msr.sys. To allow test-signed drivers, run bcdedit /set testsigning on in an administrator command prompt and reboot.

Installation Steps

1. Build pcm.exe:

   cmake -B build
   cmake --build build --config Release --parallel

   The executable will be in build\bin\Release.
2. Setup Directories:
   • Create a directory in C:\Program Files (x86)\PCM\ (as Administrator).
   • Copy msr.sys to c:\windows\system32.
   • Copy pcm.exe to your PCM directory.
3. Execution: Run pcm.exe from the PCM directory as an Administrator.

Troubleshooting

If you encounter the error Starting MSR service failed with error 3 The system cannot find the path specified., run:

pcm --uninstallDriver

and optionally reboot.

The Grafana front-end can be installed on any host system with connectivity to the target system. Ensure curl and docker are installed on the host.

Using Telegraf and InfluxDB

Navigate to scripts/grafana in the PCM source directory and run start.sh providing the target system's address and port:

sudo bash start.sh http://target_system_address:9738

Using Prometheus

Alternatively, use start-prometheus.sh to start Prometheus and Grafana containers:

sudo bash start-prometheus.sh target_system_address:9738

Monitoring Multiple Hosts

To monitor multiple hosts, provide a text file (e.g., targets.txt) containing the IP addresses and ports in the following format:

host1_ipaddress:pcmport
host2_ipaddress:pcmport

Then run:

sudo bash start.sh targets.txt
# OR
sudo bash start-prometheus.sh targets.txt

Note: If the host and target are the same machine, do not use localhost. Use the external IP address or hostname instead, as localhost inside the container resolves to the container's own private IP.

If pcm-raw is compiled with the simdjson submodule, it can automatically translate event names into raw encodings using JSON event lists from the Intel perfmon repository.

Setup Requirements: To enable this, you must ensure simdjson is present in the src/ directory. You can do this by cloning the repository recursively or manually:

cd src/
git clone https://github.com/simdjson/simdjson.git
# Then re-compile PCM

Usage:

• Use -tr to enable translation.
• Use -e <EVENT_NAME> for individual events.
• Use -el <FILE> to load event groups from a file (supports multiplexing via semicolon-separated groups).

Example (Individual events):

pcm-raw -tr -e INST_RETIRED.ANY -e CPU_CLK_UNHALTED.THREAD

Example (Event file):

pcm-raw -tr -el event_file.txt

pcm-raw -tr -e INST_RETIRED.ANY -e CPU_CLK_UNHALTED.THREAD -e CPU_CLK_UNHALTED.REF_TSC -e LD_BLOCKS.STORE_FORWARD -e UNC_CHA_CLOCKTICKS -e UNC_M_CAS_COUNT.RD

The pcm-sensor-server exposes PCM metrics via HTTP in JSON or Prometheus formats for Grafana integration.

Building

1. Ensure msr.sys is compiled and signed.
2. Build the server:

   cmake -B build
   cmake --build build --config Release --target pcm-sensor-server

   The executable is located at build\bin\Release\pcm-sensor-server.exe. Note: For HTTPS support, ensure OpenSSL is installed and available to CMake.

Running

1. Deployment:
   • Copy pcm-sensor-server.exe to a protected directory (e.g., C:\Program Files\PCM\).
   • Copy msr.sys to c:\windows\system32.
   • If using WinPmem for memory bandwidth, copy winpmem_x64.sys (or winpmem_x86.sys) to c:\windows\system32.
   • Security Warning: Do not place binaries in user-writable directories (Downloads, Desktop, etc.).
2. Execution: Run as Administrator:

   pcm-sensor-server.exe

CLI Options

• -p <port>: Run on a specific port (default: 9738).
• -D <level>: Set debug verbosity level (0 = no debug).
• -s: Enable HTTPS (requires OpenSSL and certificate files).
• -h or --help: Show help.

Accessing Metrics

• JSON: http://localhost:9738/ (requires Accept: application/json header).
• Prometheus: http://localhost:9738/metrics.

Windows Limitations

• No Daemon Mode: The -d option is not supported; the server runs in the foreground.
• No Real-time Priority: The -R option is not supported.
• Signal Handling: Use Ctrl+C to stop the server (SIGTERM/SIGINT are not supported).

cmake -B build
cmake --build build --config Release --target pcm-sensor-server

# Run as Administrator
pcm-sensor-server.exe -p 9738

The Perfmon front end allows viewing PCM counters via Windows Performance Monitor.

Installation Steps

1. Driver Setup: Compile and sign msr.sys, then copy it to c:\windows\system32.
2. Build Components: Build pcm-lib.dll and PCM-Service.exe using CMake or Visual Studio.
3. Deployment:
   • Copy PCM-Service.exe, PCM-Service.exe.config, and pcm-lib.dll into a PCM sub-directory in C:\Program Files.
   • Note: The .config file is required to prevent System.NotSupportedException related to CAS policy.
4. Service Setup:
   • Open an Administrator terminal in the PCM directory.
   • Run: "PCM-Service.exe" -Install
   • Run: net start pcmservice
5. Usage: Open Windows Performance Monitor (perfmon) and look for new PCM* counters.

Troubleshooting

• Service won't start: Check the Windows Event Viewer under Windows Logs > Application for error messages.
• 'custom counter file view is out of memory': This occurs if another application initialized performance counters before PCM.
  • Fix 1: Stop/disable the competing application and reboot.
  • Fix 2: Edit your machine.config file to add:

    <system.diagnostics>
      <performanceCounters filemappingsize="2097152" />
    </system.diagnostics>