Upright Documentation

repository·main·Indexed 21 days ago

https://github.com/basecamp/upright

A self-hosted synthetic monitoring system designed as a Rails engine. Upright runs health check probes (HTTP, SMTP, Playwright, and Traceroute) from multiple geographic locations and reports metrics via Prometheus and OpenTelemetry. It features subdomain-based routing, support for OpenID Connect authentication, and deployment via Kamal.

Tokens
9.8K
Snippets
37
Records
51
Agent score
73%

What's inside Upright

  1. Create a custom probe type

    main

    To extend Upright with a new probe type (e.g., ping), follow these four steps:

    1. Register the type: Add the name and icon to the initializer.
    2. Create the probe class: Create a file in probes/ extending FrozenRecord::Base and including Upright::Probeable and Upright::ProbeYamlSource. Implement check (returns truthy/falsy), probe_type, and probe_target.
    3. Define probes in YAML: Create a file named probes/{type}_probes.yml (e.g., probes/ping_probes.yml) containing the probe configurations.
    4. Schedule it: Add a recurring job in config/recurring.yml using Solid Queue.
    # 1. Register in config/initializers/upright.rb
    Upright.configure do |config|
      config.probe_types.register :ping, name: "Ping", icon: "📶"
    end
    
    # 2. Create probes/ping_probe.rb
    class PingProbe < FrozenRecord::Base
      include Upright::Probeable
      include Upright::ProbeYamlSource
    
      stagger_by_site 3.seconds
    
      def check
        @ping_output, status = Open3.capture2e("ping", "-c", "1", "-W", "5", host)
        status.success?
      end
    
      def on_check_recorded(probe_result)
        if @ping_output.present?
          Upright::Artifact.new(name: "ping.log", content: @ping_output).attach_to(probe_result)
        end
      end
    
      def probe_type = "ping"
      def probe_target = host
    end
    
    # 3. Define probes/ping_probes.yml
    - name: "Cloudflare DNS"
      host: "1.1.1.1"
    
    # 4. Schedule in config/recurring.yml
    production:
      ping_probes:
        command: "PingProbe.check_and_record_all_later"
        schedule: every 30 seconds
  2. Deploy Upright with Kamal

    main

    Upright can be deployed using Kamal. A typical config/deploy.yml configuration defines the service, image, and server roles (web and jobs). It also supports setting up accessories like Playwright for browser-based probes, Prometheus for metrics, and Alertmanager for alerting. Environment variables can be tagged to specific hosts to set site-specific subdomains (e.g., SITE_SUBDOMAIN).

    service: upright
    image: your-org/upright
    
    servers:
      web:
        hosts:
          - ams.upright.example.com: [amsterdam]
          - nyc.upright.example.com: [new_york]
          - sfo.upright.example.com: [san_francisco]
      jobs:
        hosts:
          - ams.upright.example.com: [amsterdam]
          - nyc.upright.example.com: [new_york]
          - sfo.upright.example.com: [san_francisco]
        cmd: bin/jobs
    
    proxy:
      app_port: 3000
      ssl: true
      hosts:
        - "*.upright.example.com"
    
    env:
      secret:
        - RAILS_MASTER_KEY
      tags:
        amsterdam:
          SITE_SUBDOMAIN: ams
        new_york:
          SITE_SUBDOMAIN: nyc
        san_francisco:
          SITE_SUBDOMAIN: sfo
    
    accessories:
      playwright:
        image: jacoblincool/playwright:chromium-server-1.55.0
        port: "127.0.0.1:53333:53333"
        roles:
          - jobs
    
      prometheus:
        image: prom/prometheus:v3.2.1
        hosts:
          - ams.upright.example.com
        cmd: >-
          --config.file=/etc/prometheus/prometheus.yml
          --storage.tsdb.path=/prometheus
          --storage.tsdb.retention.time=30d
          --web.enable-otlp-receiver
        files:
          - config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
          - config/prometheus/rules/upright.rules.yml:/etc/prometheus/rules/upright.rules.yml
    
      alertmanager:
        image: prom/alertmanager:v0.28.1
        hosts:
          - ams.upright.example.com
        cmd: --config.file=/etc/alertmanager/alertmanager.yml
        files:
          - config/alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml
  3. Install Upright in a new Rails project

    main

    Upright is designed to be run as a Rails engine within its own application and deployed using Kamal. To set up a new project, create a Rails app with SQLite3, add the upright gem, and run the installation generator.

    Note that Upright uses subdomain-based routing. The app subdomain is used for the admin interface, while site-specific subdomains (e.g., nyc, lon) are used to view probe results for specific geographic locations.

    rails new my-upright --database=sqlite3 --skip-test
    cd my-upright
    bundle add upright
    bin/rails generate upright:install
    bin/rails db:migrate
    
    # Start the server
    bin/dev
  4. Configure Prometheus Observability

    main

    Upright exposes metrics via a Puma plugin at http://0.0.0.0:9394/metrics. To scrape these metrics, configure your Prometheus scrape_configs to target localhost:9394.

    Available Metrics:

    • upright_probe_duration_seconds: Probe execution duration
    • upright_probe_up: Probe status (1 = up, 0 = down)
    • upright_http_response_status: HTTP response status code

    Labels: Metrics include the following labels: type, name, site_code, site_city, and site_country.

    scrape_configs:
      - job_name: upright
        static_configs:
          - targets: ['localhost:9394']
  5. Define HTTP, SMTP, and Playwright probes

    main

    Probes are defined in YAML files within the probes/ directory.

    HTTP Probes (probes/http_probes.yml)

    Supports url, expected_status, alert_severity (medium, high, critical), and basic_auth_credentials (referencing Rails credentials).

    SMTP Probes (probes/smtp_probes.yml)

    Requires a host field. Note: Most cloud providers block outbound port 25 by default; you may need to request unblocking.

    Playwright Probes

    Generate a new browser-based probe class using: bin/rails generate upright:playwright_probe MyServiceAuth.

    For probes requiring authentication, implement a Playwright::Authenticator class in probes/authenticators/.

    # probes/http_probes.yml
    - name: Main Website
      url: https://example.com
      expected_status: 200
    
    - name: API Health
      url: https://api.example.com/health
      expected_status: 200
      alert_severity: critical
  6. Local Development Setup

    main

    To set up a new local development environment, run bin/setup. This command installs dependencies, prepares the database, and starts the development server.

    To run the server, use:

    bin/dev

    Access the application at http://app.upright.localhost:3000.

    Default Credentials:

    • Username: admin
    • Password: upright (or the value of the ADMIN_PASSWORD environment variable)
    bin/setup
  7. Upgrade Playwright Versions

    main

    Playwright versions are managed via Upright::PLAYWRIGHT_VERSION in lib/upright/version.rb. This single version constant drives the Ruby gem, the npm package, and the trace viewer versions.

    To perform an upgrade:

    1. Update PLAYWRIGHT_VERSION in lib/upright/version.rb.
    2. Update the version in package.json.
    3. Run bin/setup (or manually: npm install && npx playwright install chromium && rake playwright:sync).
    4. Run bin/rails test to verify compatibility.
    5. Commit the updated public/trace-viewer/ files, package.json, and package-lock.json.
  8. Schedule probes with Solid Queue

    main

    Use config/recurring.yml to schedule probes. Each entry specifies a command (the class method to trigger the check) and a schedule using every syntax.

    production:
      http_probes:
        command: "Upright::Probes::HTTPProbe.check_and_record_all_later"
        schedule: every 30 seconds
    
      my_service_auth:
        command: "Probes::Playwright::MyServiceAuthProbe.check_and_record_later"
        schedule: every 15 minutes
  9. Configure Playwright Probes and Authenticators Paths

    main

    Upright automatically loads custom probes and authenticators from paths defined in Upright.configuration.

    • Probes: Files matching *_probe.rb in the probes_path are required. They are loaded into the ::Probes::Playwright namespace.
    • Authenticators: Files in the authenticators_path are required. They are loaded into the ::Playwright::Authenticator namespace.
  10. Understand the StaticCredentials authentication flow

    main

    The StaticCredentials strategy implements a standard OmniAuth flow:

    1. Request Phase: The strategy renders a simple HTML form with a username text field and a password password field. The form title is determined by the title option.
    2. Callback Phase: Upon form submission, the strategy retrieves the username and password from the request parameters. It validates them against the provided credentials hash using ActiveSupport::SecurityUtils.secure_compare to prevent timing attacks.
    3. Success: If credentials match, the uid is set to the username, and the info hash is populated with { name: username, email: "#{username}@localhost" }.
    4. Failure: If credentials do not match or are blank, the strategy calls fail!(:invalid_credentials).
  11. Configure probe result retention

    main

    Upright automatically cleans up old probe results. You can tune the retention thresholds in config/initializers/upright.rb to manage database size and data availability.

    Upright.configure do |config|
      config.stale_success_threshold = 24.hours     # Default: 24 hours
      config.stale_failure_threshold = 30.days       # Default: 30 days
      config.failure_retention_limit = 20_000        # Default: 20,000
    end
  12. Configure Upright hostname and authentication

    main

    You can configure the base hostname and authentication methods via the Upright.configure block in config/initializers/upright.rb.

    Hostname

    Set your production hostname to enable correct subdomain routing.

    Authentication

    • Static Credentials: By default, use admin/upright. Warning: Set the ADMIN_PASSWORD environment variable before deploying to production.
    • OpenID Connect: Supports providers like Logto, Keycloak, Duo, and Okta.
    # config/initializers/upright.rb
    Upright.configure do |config|
      config.hostname = "upright.com"
      
      # For OpenID Connect
      config.auth_provider = :openid_connect
      config.auth_options = {
        issuer: "https://your-tenant.logto.app/oidc",
        client_id: ENV["OIDC_CLIENT_ID"],
        client_secret: ENV["OIDC_CLIENT_SECRET"]
      }
    end