GoAlert Documentation
repository·master·Indexed 25 days ago
https://github.com/target/goalertGoAlert is an on-call scheduling and automated escalation system that uses SMS and voice calls to notify personnel during incidents. The documentation covers installation via binaries or Docker, integration testing with Grafana 7 and 8, and detailed technical guides on database switchover (SWO) mode using the swosync and swogrp packages. It also includes developer guides for the SendIt proxy tool and the web application's form validation system.
What's inside GoAlert
- GoAlert is an on-call scheduling and automated escalation platform. It provides automated notifications via SMS or voice calls to ensure the correct personnel are engaged during incidents at the appropriate time.
Understand Logical Sync with swosync
masterThe
swosyncpackage manages logical replication from a source database to a destination database during a switchover process.Instead of replicating point-in-time differences (which requires sequential processing of every intermediate state),
swosyncuses a 'final state' strategy. It tracks changes (INSERT, UPDATE, DELETE) in achange_logtable astable, row_idpairs. During synchronization, it fetches the current state of those rows from the source and applies them to the destination. This ensures each row is replicated at most once per sync cycle, regardless of how many updates occurred between sync points.Understand SWO Group cluster state transitions
masterThe
swogrppackage orchestrates the state and transitions of the SWO process. The cluster state is determined by the sequence of messages in the message log, which serves as the source of truth.Typical state transitions follow this lifecycle:
- Unknown: Initial state.
- Resetting: Triggered by an
adminsendingcancelandresetcommands to the API. - Idle: Reached after the
enginenode becomes leader and logsreset-end. - Syncing: Triggered by an
adminsendingexecute. The engine performsInitialSyncand multipleLogicalSyncoperations. - Pausing: The engine and API log
pausedmessages. - Executing: The engine performs remaining
LogicalSyncoperations and aFinalSync. - Done: The engine logs
doneto signal completion.
Run codemods using jscodeshift
masterGoAlert provides a collection of codemod scripts in the
web/src/codemodsdirectory to perform codebase cleanup and updates. To run these scripts, you must usejscodeshift.- Install
jscodeshiftglobally:yarn global add jscodeshift
2. Execute a codemod by using the `-t` flag to reference the specific script, followed by the target directory or file paths you wish to update. Example: Applying the `relpath.js` codemod to all files under the `./app` directory: ```bash jscodeshift -t ./codemods/relpath.js ./appjscodeshift -t ./codemods/relpath.js ./app- Install
Manage and Rotate API Keys
masterAPI key rotation in GoAlert is a manual process. To simplify management, the following features are available:
- Duplicate Function: In the UI, you can use the 'Duplicate' function to copy the parameters of an existing key into a new one. This pre-fills information and helps facilitate the rotation process.
- Usage Auditing: The system tracks API key usage (up to once per minute), recording the
key ID,IP address, anduser agentfor monitoring and auditing purposes.
The swosync Synchronization Strategy
masterThe synchronization process follows these steps to move data from the old database to the new database:
- Read Changes: Identify all changed
tableandrow_idpairs from thechange_log. - Fetch Data: Retrieve the current row data for every changed row from the source.
- Insert Missing Rows: Insert rows present in the old DB but missing in the new DB, following foreign key (fkey) dependency order.
- Update Existing Rows: Update rows present in both databases, following fkey dependency order.
- Delete Extra Rows: Delete rows present in the new DB that no longer exist in the old DB, following reverse fkey dependency order.
- Cleanup: Delete the processed entries from the
change_logtable. - Iterate: Repeat steps 1-6 until the databases are closely synchronized.
- Final Sync:
- Obtain a 'stop-the-world' lock (pausing all transactions).
- Perform one final synchronization.
- Update the
use_next_dbpointer.
- Complete: Release the lock. The new database is now the primary for all future transactions.
- Read Changes: Identify all changed
Configure Twilio for Voice and SMS
masterTwilio is the supported provider for voice and SMS notifications.
Setup Steps:
- In the GoAlert Admin page, enter the following from your Twilio Dashboard in the Twilio section:
- From Number: Your Twilio
TRIAL NUMBER. - Account SID: Your Twilio
ACCOUNT SID. - Auth Token: Your Twilio
AUTH TOKEN.
- From Number: Your Twilio
- Enable Twilio using the toggle.
- Configure Webhook: In the Twilio Dashboard, navigate to Phone Numbers -> [Your Number] -> Messaging. Update the webhook URL for A MESSAGE COMES IN to
<GOALERT_PUBLIC_URL>/api/v2/twilio/message.
- In the GoAlert Admin page, enter the following from your Twilio Dashboard in the Twilio section:
Configure GitHub Authentication
masterGoAlert supports GitHub OAuth for authentication. You can optionally restrict access to specific users, organizations, or teams.
Setup Steps:
- Create an OAuth App in GitHub.
- Use the following configuration values:
- Application name:
GoAlert - Homepage URL:
<GOALERT_PUBLIC_URL> - Authorization callback URL:
<GOALERT_PUBLIC_URL>/api/v2/identity/providers/github/callback
- Application name:
- In the GoAlert Admin page, enter the Client ID and Client Secret.
- Enable GitHub authentication and New Users using the toggles.
- (Optional) Fill out Allowed Users or Allowed Orgs to restrict access. If limiting to an org/team, users must manually click "Grant" access on their first login.
Start GoAlert in Switchover (SWO) mode
masterTo initiate a database switchover, start the GoAlert instance by providing both the current database URL via
--db-urland the target database URL via--db-url-next. This allows the system to transition from the "old" database to the "new" database with minimal user impact.For local development, you can start a development instance specifically in switchover mode using
make start-swo.make start-swoRun GoAlert via Binary or Docker
masterTo start GoAlert, you must provide the
--db-url,--public-url, and--data-encryption-keyflags.Binary Execution
goalert --db-url postgres://goalert@localhost/goalert --data-encryption-key super-awesome-secret-key --public-url https://goalert.example.comDocker Execution
docker run -p 8081:8081 -e GOALERT_DB_URL=postgres://goalert@localhost/goalert -e GOALERT_DATA_ENCRYPTION_KEY=super-awesome-secret-key -e GOALERT_PUBLIC_URL=https://goalert.example.com goalert/goalertConfigure and use GoAlert Webhooks
masterGoAlert sends webhooks as
POSTrequests with aapplication/jsoncontent type to specified endpoints.Important Constraints:
- Webhook calls must complete within 3 seconds to avoid timeouts.
Generate CA files for certificate verification
masterTo generate the Certificate Authority (CA) files used for creating and verifying future certificates, run the
gen-cert cacommand. This produces four files:system.ca.pem,system.ca.key,plugin.ca.pem, andplugin.ca.key.Security Warning: Keep these files secure, especially the
.keyfiles, as they are used to generate the deployment certificates.goalert gen-cert ca