NATS Surveyor

NATS Surveyor is a Prometheus exporter that polls NATS servers for Statz messages to generate monitoring data. It allows a single exporter to connect to any NATS server and provide a complete picture of a NATS deployment without requiring sidecars or additional monitoring components.

Requirement: System accounts must be enabled in your NATS configuration to use Surveyor.

Surveyor outputs metrics in the format nats_core_NNNN_metric, where NNNN represents the component type: server, route, or gateway.

To facilitate filtering in Prometheus, metrics include the following labels:

• server_cluster
• server_name
• server_id
• server_route_name (for routes)
• server_gateway_name (for gateways)

Connectivity Monitoring: The nats_up metric is used to monitor the health of the connection. It returns 1 when connected to NATS. If it returns 0 and no other NATS metrics are present, it indicates a connectivity issue between the exporter and the NATS system, helping you distinguish between an exporter failure and a NATS connectivity problem.

To monitor JetStream, create .json files in the jetstream directory. Files are watched via fsnotify.

Requirement: Ensure the provided user has access to the $JS.EVENT.> subject.

Only one authentication method is required per file.

{
  "name":       "my account",
  "jwt":        "jwt portion of creds, must include seed also",
  "seed":       "seed portion of creds, must include jwt also",
  "credential": "/path/to/file.creds",
  "nkey":       "nkey seed",
  "token":       "token",
  "username":    "username, must include password also",
  "password":    "password, must include user also",
  "tls_ca":      "/path/to/ca.pem, defaults to surveyor's ca if one exists",
  "tls_cert":    "/path/to/cert.pem, defaults to surveyor's cert if one exists",
  "tls_key":     "/path/to/key.pem, defaults to surveyor's key if one exists"
}

Surveyor uses Viper to read configuration files, supporting JSON, TOML, YAML, HCL, envfile, and Java properties.

To use a specific file, pass the --config flag. If not specified, Surveyor looks for files at /etc/nats-surveyor/nats-surveyor[.ext] or ./nats-surveyor[.ext].

The configuration format mirrors the CLI flags. For example, a nats-surveyor.yaml file might look like this:

servers: nats://127.0.0.1:4222
accounts: true
log-level: debug

Use the following commands to manage your deployment:

• Start (pulling latest images): docker compose up --pull always (run from the docker-compose directory).
• Stop (keep containers): docker compose stop.
• Restart (after stopping): docker compose up.
• Cleanup (remove containers): docker compose down.

# Start
docker compose up --pull always

# Stop but keep containers
docker compose stop

# Restart
docker compose up

# Remove everything
docker compose down

Once the stack is running, access the Grafana dashboard via your browser:

1. Navigate to http://127.0.0.1:3000 (or the hostname of the remote host).
2. Log in with default credentials:
   • User: admin
   • Password: admin
3. Navigate to "Manage dashboards" and select the NATS Surveyor dashboard to monitor your deployment.

Surveyor supports environment variables prefixed with NATS_SURVEYOR_. To map a flag to an environment variable, convert the flag name to uppercase and replace dashes with underscores.

Examples:

• --servers $\rightarrow$ NATS_SURVEYOR_SERVERS
• --accounts $\rightarrow$ NATS_SURVEYOR_ACCOUNTS
• --log-level $\rightarrow$ NATS_SURVEYOR_LOG_LEVEL
• --user $\rightarrow$ NATS_SURVEYOR_USER
• --password $\rightarrow$ NATS_SURVEYOR_PASSWORD

NATS_SURVEYOR_SERVERS=nats://127.0.0.1:4222
NATS_SURVEYOR_ACCOUNTS=true
NATS_SURVEYOR_LOG_LEVEL=debug
NATS_SURVEYOR_USER=myuser
NATS_SURVEYOR_PASSWORD=mypassword

To access the necessary Statz messages, Surveyor must authenticate using a NATS system account. You can provide credentials using one of the following four methods:

1. Chained Credentials File: Use the --creds flag to provide a file containing both the JWT and the NKey seed.
2. Direct JWT and Seed: Use the --jwt and --seed flags to provide the user JWT and NKey seed directly.
3. Token File: Use the --token-file flag to point to a file containing a bearer token, combined with the --user flag.
4. Username and Password: Use the --user and --password flags for standard authentication.

# Method 1: Credentials file
./nats-surveyor --creds ./test/SYS.creds

# Method 2: Direct JWT and Seed
./nats-surveyor --jwt $NATS_USER_JWT --seed $NATS_NKEY_SEED

# Method 3: Token file
./nats-surveyor --user sys --token-file /var/run/secrets/tokens/<token>

All CLI flags can be set using environment variables. The prefix for these variables is NATS_SURVEYOR_. Hyphens in flag names are replaced with underscores.

For example, the --servers flag can be set via NATS_SURVEYOR_SERVERS.