Vigil Microservices Status Page

Vigil uses three different methods to monitor infrastructure, depending on the type of service and network accessibility:

• HTTP / TCP / SSH / ICMP services: Vigil performs frequent probes against these targets to check for reachability.
• Application services: You can integrate the Vigil Reporter library directly into your application (e.g., a NodeJS app) to receive reports when the application fails or when the host server's system is overloaded.
• Local services: For services located on a different LAN or unreachable by the Vigil master server, you can install a Vigil Local daemon to act as a slave and report status back to the master.

To ensure rapid notification and minimize downtime, it is recommended to configure these probes and reports to run frequently.

Vigil uses specific status variants to represent the health of your services:

• healthy: No issues ongoing.
• sick: Services are under high load.
• dead: An outage is occurring.
• partial: A special status reported when a node has part of its replica dead, but the service is still considered partially degraded rather than in a full outage. This requires the probe.service.node.min_replicas_available option to be configured on the node.

Vigil Local is an optional slave daemon used to monitor hosts that are behind a firewall or bound to a local loop/LAN network that the main Vigil master cannot reach.

Vigil Local monitors local poll and script replicas and periodically reports their status to the main Vigil status page master server.

To actively submit detailed health information (like CPU and RAM load) from your applications to Vigil, use a Vigil Reporter library. This allows Vigil to detect if a host system is under high load.

Available libraries include:

• NodeJS: node-vigil-reporter
• TypeScript: ts-vigil-reporter
• Python: py-vigil-reporter
• Golang: go-vigil-reporter
• Rust: rs-vigil-reporter
• Dart: dart-vigil-reporter
• C#: cs-vigil-reporter

If a library for your language is not listed, you can manually report metrics using the Vigil Reporter HTTP API.

You can define custom probes using shell scripts. These are passed in the Vigil configuration as a list of scripts to be executed for a specific node.

Scripts must return specific exit codes to communicate health status:

• rc=0: healthy
• rc=1: sick
• rc=2 or higher: dead

Since scripts are often multi-line, use triple single quotes (''') to define them in the configuration.

Security Warning: Scripts run in a system shell via a Vigil-owned sub-process. Always run Vigil as a non-root UNIX user with limited privileges to prevent scripts from performing unauthorized root-level actions.

scripts = [
  '''
  # Do some work...
  exit 1
  '''
]

You can run Vigil using a pre-built Docker image.

1. Pull the image:

docker pull valeriansaliou/vigil:v1.29.0

2. Run the container with a local configuration file:

docker run -p 8080:8080 -v /path/to/your/vigil/config.cfg:/etc/vigil.cfg valeriansaliou/vigil:v1.29.0

Important Configuration for Docker: In your config.cfg, you must set:

• server.inet to 0.0.0.0:8080 (to allow external access to the container).
• assets.path to ./res/assets/ (to use the assets bundled inside the container).

For Debian-based systems, you can use the official Vigil APT repository. Note that pre-built packages currently target 64-bit Debian 11 (bullseye) and 12 (bookworm), but they are compatible with other Debian versions and Ubuntu.

1. Add the repository:

echo "deb [signed-by=/usr/share/keyrings/valeriansaliou_vigil.gpg] https://packagecloud.io/valeriansaliou/vigil/debian/ bookworm main" > /etc/apt/sources.list.d/valeriansaliou_vigil.list

2. Import the GPG key:

curl -fsSL https://packagecloud.io/valeriansaliou/vigil/gpgkey | gpg --dearmor -o /usr/share/keyrings/valeriansaliou_vigil.gpg

3. Update and install:

apt-get update
apt-get install vigil

4. Configure and restart: Edit /etc/vigil/vigil.cfg and then run service vigil restart.

To ensure reliable monitoring and avoid false positives, follow these deployment recommendations:

• Isolation: Host Vigil on a safe, separate server. This server should reside on a different physical machine and network than your monitored infrastructure.
• Whitelisting: If your monitored HTTP services use bot protection (e.g., Cloudflare or Distil Networks) that challenges unknown IPs, you must whitelist the Vigil server's public IP addresses (both IPv4 and IPv6). Failure to do so will cause Vigil to report services as dead when they encounter a bot challenge.

If you prefer using Rust's package manager, you can install the server directly via Cargo:

cargo install vigil-server

Ensure your $PATH is configured to include Cargo binaries. To build from source manually, use:

cargo build --release

Dependencies for source compilation:

• libssl-dev (OpenSSL headers): Required for HTTPS probes and email notifications.
• libstrophe-dev (XMPP library headers): Required only if you need the XMPP notifier.

When Vigil sends a notification to Gotify, it constructs a message with the following characteristics:

• Title: Uses the branding.page_title configured in Vigil.
• Message Body: A multi-line string containing:
  • A prefix if it is a startup alert (`This is a startup alert.

) or a reminder (This is a reminder.

). - The status of the service (e.g., STATUS: OK`). - A list of nodes/replicas. - The timestamp of the notification.

• Priority: If the notification is marked as escalated_for(0), the priority parameter is set to 10 in the Gotify request.