Scrutiny

repository·master·Indexed 27 days ago

https://github.com/analogj/scrutiny

A SMART data collection and monitoring system featuring a Hub (Web interface and InfluxDB) and Spokes (collectors). It includes the scrutiny-collector-metrics and scrutiny-collector-selftest CLI tools for gathering hardware metrics via smartctl, as well as automated data downsampling and retention for long-term trend analysis.

Tokens
20.8K
Snippets
50
Records
106
Agent score
92%

What's inside scrutiny

  1. Associate devices with a host using host-id

    master

    In a Hub & Spoke deployment, use a host-id to associate devices with a friendly hostname. This ID is passed from the collector to the web-api during data upload. You can set the host-id using one of three methods:

    1. Collector Config File: Define it in your collector.yaml.
    2. CLI Argument: Use the --host-id flag when running the collector.
    3. Environment Variable: Set the COLLECTOR_HOST_ID variable.
  2. Schedule Scrutiny Collector with pfSense Cron

    master

    To ensure data is collected periodically, add a cron job via the pfSense webGUI at Services/Cron. Use the following configuration to run the collector every 15 minutes as root and suppress confirmation emails:

    • Minute: */15
    • Hour: *
    • Day of the Month: *
    • Month of the Year: *
    • Day of the Week: *
    • User: root
    • Command: /opt/scrutiny/bin/scrutiny-collector-metrics-freebsd-amd64 run --api-endpoint "http://localhost:8080" >/dev/null 2>&1
    /opt/scrutiny/bin/scrutiny-collector-metrics-freebsd-amd64 run --api-endpoint "http://localhost:8080" >/dev/null 2>&1
  3. Install the 'Community Applications' Plugin on UnRAID

    master

    To install Scrutiny on UnRAID, you first need the Community Applications plugin.

    1. Navigate to your UnRAID plugins tab at <UnRaid_IP_Address>/Plugins.
    2. Select the 'Install Plugin' tab.
    3. Enter the following URL into the input field to install the plugin:
    https://raw.githubusercontent.com/Squidly271/community.applications/master/plugins/community.applications.plg
  4. Configure custom script notifications

    master

    Scrutiny supports a "script" based notification system that executes a custom script whenever a notification is triggered. When the script is executed, Scrutiny provides notification data via the following environment variables:

    SCRUTINY_SUBJECT - eg. "Scrutiny SMART error (%s) detected on device: %s"
    SCRUTINY_DATE 
    SCRUTINY_FAILURE_TYPE - EmailTest, SmartFail, ScrutinyFail
    SCRUTINY_DEVICE_NAME - eg. /dev/sda
    SCRUTINY_DEVICE_TYPE - ATA/SCSI/NVMe
    SCRUTINY_DEVICE_SERIAL - eg. WDDJ324KSO
    SCRUTINY_MESSAGE - eg. "Scrutiny SMART error notification for device: %s\nFailure Type: %s\nDevice Name: %s\nDevice Serial: %s\nDevice Type: %s\nDate: %s"
    SCRUTINY_HOST_ID - (optional) eg. "my-custom-host-id"
  5. Update smartd database for Scrutiny

    master

    To improve device detection, download the drivedb.h file and run smartctl once to load it into the application.

    1. Download the database:
      cd /volume1/@Entware/scrutiny/bin/
      wget https://raw.githubusercontent.com/smartmontools/smartmontools/master/smartmontools/drivedb.h
    2. Run smartctl with the --drivedb flag for a specific device:
      smartctl -d sat --all /dev/sda --drivedb=/volume1/@Entware/scrutiny/bin/drivedb.h
  6. Set up the Scrutiny Hub using Docker Compose

    master

    The Hub consists of the Scrutiny Web interface and an InfluxDB instance for storing SMART data. You can deploy the Hub using Docker Compose. Use environment variables to configure InfluxDB initialization and Scrutiny's connection to InfluxDB.

    Key configuration variables for Scrutiny include:

    • SCRUTINY_WEB_INFLUXDB_HOST: The hostname of the InfluxDB service.
    • SCRUTINY_WEB_INFLUXDB_PORT: The port for InfluxDB (default 8086).
    • SCRUTINY_WEB_INFLUXDB_TOKEN: The InfluxDB admin token.
    • SCRUTINY_WEB_INFLUXDB_ORG: The InfluxDB organization name.
    • SCRUTINY_WEB_INFLUXDB_BUCKET: The InfluxDB bucket name.
    • SCRUTINY_NOTIFY_URLS: A space-separated list of Shoutrrr URIs for notifications (e.g., Gotify or ntfy).
    networks:
      monitoring: # A common network for all monitoring services to communicate into
      notifications: # To Gotify or another Notification service
    
    services:
      influxdb:
        restart: unless-stopped
        container_name: influxdb
        image: influxdb:2.8
        ports:
          - 8086:8086
        volumes:
          - ${DIR_CONFIG}/influxdb2/db:/var/lib/influxdb2
          - ${DIR_CONFIG}/influxdb2/config:/etc/influxdb2
        environment:
          - DOCKER_INFLUXDB_INIT_MODE=setup
          - DOCKER_INFLUXDB_INIT_USERNAME=Admin
          - DOCKER_INFLUXDB_INIT_PASSWORD=${PASSWORD}
          - DOCKER_INFLUXDB_INIT_ORG=homelab
          - DOCKER_INFLUXDB_INIT_BUCKET=scrutiny
          - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=SUPER-SECRET-TOKEN
          - TZ=Europe/Stockholm
        networks:
          - monitoring
    
      scrutiny:
        restart: unless-stopped
        container_name: scrutiny
        image: ghcr.io/analogj/scrutiny:latest-web
        ports:
          - 8080:8080
        volumes:
          - ${DIR_CONFIG}/config:/opt/scrutiny/config
        environment:
          - SCRUTINY_WEB_INFLUXDB_HOST=influxdb
          - SCRUTINY_WEB_INFLUXDB_PORT=8086
          - SCRUTINY_WEB_INFLUXDB_TOKEN=SUPER-SECRET-TOKEN
          - SCRUTINY_WEB_INFLUXDB_ORG=homelab
          - SCRUTINY_WEB_INFLUXDB_BUCKET=scrutiny
          - SCRUTINY_NOTIFY_URLS=http://gotify:80/message?token=a-gotify-token ntfy://username:password@host:port/topic
          - TZ=Europe/Stockholm
        depends_on:
          influxdb:
            condition: service_healthy
        networks:
          - notifications
          - monitoring
  7. Install Scrutiny Collector as a binary (without Docker)

    master

    To run a Spoke on a machine without Docker, install the Scrutiny Collector binary manually.

    1. Install smartmontools via your package manager (e.g., apt install smartmontools -y).
    2. Create a directory for the binary (e.g., /opt/scrutiny/bin).
    3. Download the appropriate collector binary from the Scrutiny Releases.
    4. Make the binary executable.

    To run the collector once and send data to your Hub, use the run command with the --api-endpoint flag pointing to your Hub's URL.

    # Installing dependencies
    apt install smartmontools -y 
    
    # 1. Create directory for the binary
    # 2. Download the binary into that directory
    # 3. Make it executable
    # 4. List the contents of the library for confirmation
    mkdir -p /opt/scrutiny/bin && \
    curl -L https://github.com/AnalogJ/scrutiny/releases/download/v0.8.1/scrutiny-collector-metrics-linux-amd64 > /opt/scrutiny/bin/scrutiny-collector-metrics-linux-amd64 && \
    chmod +x /opt/scrutiny/bin/scrutiny-collector-metrics-linux-amd64 && \
    ls -lha /opt/scrutiny/bin
    
    # Run the collector
    /opt/scrutiny/bin/scrutiny-collector-metrics-linux-amd64 run --api-endpoint "http://192.168.0.100:8080"
  8. Configure Service Account for Rootless Podman

    master

    To run Scrutiny rootless, you must first create a restricted service account (refer to INSTALL_MANUAL.md for account creation). After creating the account (e.g., scrutiny-svc), you must assign sub-UIDs/GIDs and enable lingering so the services continue running when the user is not logged in.

    # Add sub-uids and sub-gids (starting at 500000 to avoid conflicts)
    sudo usermod --add-subuids 500000-565535 scrutiny-svc
    sudo usermod --add-subgids 500000-565535 scrutiny-svc
    
    # Enable lingering so quadlets stay running without an active session
    sudo loginctl enable-linger scrutiny-svc
  9. Install InfluxDB and Webapp/API via Docker on Windows (WSL)

    master

    For users on Windows using WSL with Docker, you can run the database and web interface using Docker Compose.

    1. Copy the example.hubspoke.docker-compose.yml file.
    2. Remove the collector section from the bottom of the file.
    3. Run docker-compose up -d to start the services.

    Once completed, the webapp is accessible at http://localhost:8080 (default), though the dashboard will be empty until the collector is configured.

    docker-compose up -d
  10. Enable Collector debug mode and logging

    master

    You can enable debug logging and specify a log file for the collector using environment variables (for Docker) or CLI arguments (for direct execution).

    Using Docker (Environment Variables):

    DEBUG=true
    COLLECTOR_LOG_FILE=/tmp/collector.log

    Using CLI (Direct Execution):

    scrutiny-collector-metrics run --debug --log-file /tmp/collector.log
    # Docker environment variables
    DEBUG=true
    COLLECTOR_LOG_FILE=/tmp/collector.log
    
    # CLI arguments
    scrutiny-collector-metrics run --debug --log-file /tmp/collector.log