GoAlert Documentation

repository·master·Indexed 25 days ago

https://github.com/target/goalert

GoAlert is an on-call scheduling and automated escalation system that uses SMS and voice calls to notify personnel during incidents. The documentation covers installation via binaries or Docker, integration testing with Grafana 7 and 8, and detailed technical guides on database switchover (SWO) mode using the swosync and swogrp packages. It also includes developer guides for the SendIt proxy tool and the web application's form validation system.

Tokens
13.9K
Snippets
34
Records
85
Agent score
83%

What's inside GoAlert

  1. Overview of GoAlert

    master
    GoAlert is an on-call scheduling and automated escalation platform. It provides automated notifications via SMS or voice calls to ensure the correct personnel are engaged during incidents at the appropriate time.
  2. Understand Logical Sync with swosync

    master

    The swosync package manages logical replication from a source database to a destination database during a switchover process.

    Instead of replicating point-in-time differences (which requires sequential processing of every intermediate state), swosync uses a 'final state' strategy. It tracks changes (INSERT, UPDATE, DELETE) in a change_log table as table, row_id pairs. During synchronization, it fetches the current state of those rows from the source and applies them to the destination. This ensures each row is replicated at most once per sync cycle, regardless of how many updates occurred between sync points.

  3. Understand SWO Group cluster state transitions

    master

    The swogrp package orchestrates the state and transitions of the SWO process. The cluster state is determined by the sequence of messages in the message log, which serves as the source of truth.

    Typical state transitions follow this lifecycle:

    1. Unknown: Initial state.
    2. Resetting: Triggered by an admin sending cancel and reset commands to the API.
    3. Idle: Reached after the engine node becomes leader and logs reset-end.
    4. Syncing: Triggered by an admin sending execute. The engine performs InitialSync and multiple LogicalSync operations.
    5. Pausing: The engine and API log paused messages.
    6. Executing: The engine performs remaining LogicalSync operations and a FinalSync.
    7. Done: The engine logs done to signal completion.
  4. Run codemods using jscodeshift

    master

    GoAlert provides a collection of codemod scripts in the web/src/codemods directory to perform codebase cleanup and updates. To run these scripts, you must use jscodeshift.

    1. Install jscodeshift globally:
      yarn global add jscodeshift
    
    2. Execute a codemod by using the `-t` flag to reference the specific script, followed by the target directory or file paths you wish to update.
    
    Example: Applying the `relpath.js` codemod to all files under the `./app` directory:
    ```bash
    jscodeshift -t ./codemods/relpath.js ./app
    jscodeshift -t ./codemods/relpath.js ./app
  5. Manage and Rotate API Keys

    master

    API key rotation in GoAlert is a manual process. To simplify management, the following features are available:

    • Duplicate Function: In the UI, you can use the 'Duplicate' function to copy the parameters of an existing key into a new one. This pre-fills information and helps facilitate the rotation process.
    • Usage Auditing: The system tracks API key usage (up to once per minute), recording the key ID, IP address, and user agent for monitoring and auditing purposes.
  6. The swosync Synchronization Strategy

    master

    The synchronization process follows these steps to move data from the old database to the new database:

    1. Read Changes: Identify all changed table and row_id pairs from the change_log.
    2. Fetch Data: Retrieve the current row data for every changed row from the source.
    3. Insert Missing Rows: Insert rows present in the old DB but missing in the new DB, following foreign key (fkey) dependency order.
    4. Update Existing Rows: Update rows present in both databases, following fkey dependency order.
    5. Delete Extra Rows: Delete rows present in the new DB that no longer exist in the old DB, following reverse fkey dependency order.
    6. Cleanup: Delete the processed entries from the change_log table.
    7. Iterate: Repeat steps 1-6 until the databases are closely synchronized.
    8. Final Sync:
      • Obtain a 'stop-the-world' lock (pausing all transactions).
      • Perform one final synchronization.
      • Update the use_next_db pointer.
    9. Complete: Release the lock. The new database is now the primary for all future transactions.
  7. Configure Twilio for Voice and SMS

    master

    Twilio is the supported provider for voice and SMS notifications.

    Setup Steps:

    1. In the GoAlert Admin page, enter the following from your Twilio Dashboard in the Twilio section:
      • From Number: Your Twilio TRIAL NUMBER.
      • Account SID: Your Twilio ACCOUNT SID.
      • Auth Token: Your Twilio AUTH TOKEN.
    2. Enable Twilio using the toggle.
    3. Configure Webhook: In the Twilio Dashboard, navigate to Phone Numbers -> [Your Number] -> Messaging. Update the webhook URL for A MESSAGE COMES IN to <GOALERT_PUBLIC_URL>/api/v2/twilio/message.
  8. Configure GitHub Authentication

    master

    GoAlert supports GitHub OAuth for authentication. You can optionally restrict access to specific users, organizations, or teams.

    Setup Steps:

    1. Create an OAuth App in GitHub.
    2. Use the following configuration values:
      • Application name: GoAlert
      • Homepage URL: <GOALERT_PUBLIC_URL>
      • Authorization callback URL: <GOALERT_PUBLIC_URL>/api/v2/identity/providers/github/callback
    3. In the GoAlert Admin page, enter the Client ID and Client Secret.
    4. Enable GitHub authentication and New Users using the toggles.
    5. (Optional) Fill out Allowed Users or Allowed Orgs to restrict access. If limiting to an org/team, users must manually click "Grant" access on their first login.
  9. Start GoAlert in Switchover (SWO) mode

    master

    To initiate a database switchover, start the GoAlert instance by providing both the current database URL via --db-url and the target database URL via --db-url-next. This allows the system to transition from the "old" database to the "new" database with minimal user impact.

    For local development, you can start a development instance specifically in switchover mode using make start-swo.

    make start-swo
  10. Run GoAlert via Binary or Docker

    master

    To start GoAlert, you must provide the --db-url, --public-url, and --data-encryption-key flags.

    Binary Execution

    goalert --db-url postgres://goalert@localhost/goalert --data-encryption-key super-awesome-secret-key --public-url https://goalert.example.com

    Docker Execution

    docker run -p 8081:8081 -e GOALERT_DB_URL=postgres://goalert@localhost/goalert -e GOALERT_DATA_ENCRYPTION_KEY=super-awesome-secret-key -e GOALERT_PUBLIC_URL=https://goalert.example.com goalert/goalert
  11. Generate CA files for certificate verification

    master

    To generate the Certificate Authority (CA) files used for creating and verifying future certificates, run the gen-cert ca command. This produces four files: system.ca.pem, system.ca.key, plugin.ca.pem, and plugin.ca.key.

    Security Warning: Keep these files secure, especially the .key files, as they are used to generate the deployment certificates.

    goalert gen-cert ca