fly-log-shipper

The fly-log-shipper utilizes Vector as its underlying engine. The nats source component in Vector receives logs and passes them through downstream transforms and sinks.

Configuration is managed via a shell wrapper script that evaluates environment variables to dynamically generate the final Vector configuration file. This allows you to enable or disable specific Vector sinks (providers) by toggling the corresponding environment variables.

While the log shipper handles connections automatically, you can manually connect your own Fly apps to the NATS server to publish or subscribe to logs.

Connection Details:

• Server: nats://[fdaa::3]:4223 (IPv6)
• User: Your Fly organization slug (get via fly orgs list)
• Password: Your Fly personal access token (get via fly tokens create readonly personal)

Subject Schema: The subject follows the pattern logs.<app_name>.<region>.<instance_id>. You can use standard NATS wildcards to subscribe to specific subsets of logs.

To run multiple instances of fly-log-shipper for high availability and ensure that logs are only processed by one instance, use NATS subscription queues. You can configure a specific queue name by setting the QUEUE environment variable (or secret). This ensures that messages are distributed across subscribers of the named queue rather than being sent to all of them.

# Example: Setting the QUEUE secret/env var
export QUEUE="my-log-shipper-queue"

To deploy a new log shipper application on Fly.io that automatically reads your organization's logs and sends them to an external provider, follow these steps:

1. Launch the app: Use the official Docker image. Ensure you use --no-public-ips for security.

   fly launch --image flyio/log-shipper:latest --no-public-ips

2. Configure NATS source: Set the required NATS secrets (see NATS source configuration).
3. Configure Provider: Set the secrets for your chosen log destination (see Provider configuration).

Important: After generating your fly.toml, you must update the internal_port in the [[services]] section to match the Vector internal port defined in vector-configs/vector.toml (typically 8686). Failure to do this will cause health checks to fail during deployment.

To ship logs to SigNoz, set the following secrets:

Self-hosted SigNoz: Set SIGNOZ_INGESTION_URL to your own endpoint. SIGNOZ_INGESTION_KEY is not required for self-hosted deployments.

To ship logs to AWS CloudWatch, set the following secrets:

To ship logs to Datadog, set the following secrets:

You can use the HTTP sink to send log alerts to Slack by setting these secrets:

Example for setting keywords:

fly secrets set SLACK_ALERT_KEYWORDS="[r'SIGTERM', r'reboot']"

To ship logs to an AWS S3 bucket (or S3-compatible stores like Cloudflare R2 or Wasabi), set the following secrets:

The log shipper uses NATS to subscribe to Fly.io logs. Configure the following secrets in your Fly app to define the source:

To test the NATS stream, you can launch a temporary NATS client on Fly.io, SSH into it, and publish a message.

1. Launch client:

   fly launch --image="synadia/nats-server:nightly" --name="nats-client"

2. SSH into app:

   fly -a nats-client ssh console

3. Add NATS context:

   nats context add nats --server [fdaa::3]:4223 --description "NATS Demo" --select \
     --user <YOUR FLY ORG SLUG> \
     --password <YOUR PAT>

4. Publish a test message:

   nats pub "logs.test" "hello world"