Merge pull request #3246 from target/switchover-guide

swo: add guide and in-app documentation

docs/getting-started.md

@@ -22,6 +22,14 @@ in your setup.
 
 Note: If you are using default install of Postgres on Debian (maybe others) you may run into an issue where the OOM (out of memory) killer terminates the supervisor process. More information along with steps to resolve can be found [here](https://www.postgresql.org/docs/current/kernel-resources.html#LINUX-MEMORY-OVERCOMMIT).
 
+### Switchover
+
+GoAlert natively supports database switchover functionality, allowing you to switch between databases with minimal disruption. This can be especially useful for maintenance, migration, or certain disaster recovery scenarios (it does require both old and new DB to be usable).
+
+To perform a database switchover, GoAlert offers an easy-to-follow, step-by-step guide that you can find in the [Switchover Guide](./switchover.md). We strongly recommend reviewing this guide to understand the best practices and the sequence of steps involved.
+
+Further information about the theory of operation and implementation is available in the [`swo` package README](../swo/README.md).
+
 ### Encryption of Sensitive Data
 
 It is also recommended to set the `--data-encryption-key` which is used to encrypt sensitive information (like API keys) before transmitting to the database.
@@ -181,11 +189,11 @@ You'll need to create an MX-type DNS record for your GoAlert server.
 
 Note: This SMTP server only handles incoming messages/generating alerts; it will not send outgoing mail. For notifying users via email, you can configure an external SMTP server integration in the admin UI.
 
-To enable the built-in SMTP ingress, pass the `--smtp-listen` and/or `--smtp-listen-tls` flag with the address to listen on, e.g. `--smtp-listen=0.0.0.0:9025`. You may use both flags with different ports.  You must also provide a domain name with `--email-integration-domain`, to be used for generating the alert-creating email addresses.
+To enable the built-in SMTP ingress, pass the `--smtp-listen` and/or `--smtp-listen-tls` flag with the address to listen on, e.g. `--smtp-listen=0.0.0.0:9025`. You may use both flags with different ports. You must also provide a domain name with `--email-integration-domain`, to be used for generating the alert-creating email addresses.
 
 If you use the TLS variant, you must also pass the TLS cert and key. To do so, use either `--smtp-tls-cert-file` and `--smtp-tls-key-file` (with paths to the cert and key files) or `--smtp-tls-cert-data` and `--smtp-tls-key-data` to pass the cert and key data directly as strings. The cert and key must be PEM-encoded.
 
-By default, only the `--email-integration-domain` will be allowed for the TO address on incoming emails.  To allow other domains, you can pass a comma-separated list of domains to `--smtp-additional-domains`, e.g. `--smtp-allowed-domains="example.com,foo.io"`. Messages addressed to domains not matching the given list will be rejected.
+By default, only the `--email-integration-domain` will be allowed for the TO address on incoming emails. To allow other domains, you can pass a comma-separated list of domains to `--smtp-additional-domains`, e.g. `--smtp-allowed-domains="example.com,foo.io"`. Messages addressed to domains not matching the given list will be rejected.
 
 ### Slack
 

docs/switchover.md

@@ -0,0 +1,78 @@
+# Switchover Guide for GoAlert
+
+Switchover (SWO) is a feature that allows a live system to switch from one database to another safely and with little to no user impact.
+
+This guide provides a quick and easy-to-follow set of steps for performing a database switchover in GoAlert.
+
+## Steps to Perform Switchover
+
+1. **Preparation:** Configure all GoAlert instances with the `--db-url-next` flag (or using `GOALERT_DB_URL_NEXT` environment variable). Ensure that every instance is started or restarted with this configuration.
+
+   ```
+   goalert --db-url=<old-db-url> --db-url-next=<new-db-url>
+   ```
+
+   OR using environment variables:
+
+   ```
+   export GOALERT_DB_URL=<old-db-url>
+   export GOALERT_DB_URL_NEXT=<new-db-url>
+   ```
+
+2. **Check Configuration in UI:**
+
+   - Navigate to the `Admin` section in the left sidebar of the UI.
+   - Click on the `Switchover` page.
+   - Ensure that all instances of GoAlert (displayed as Nodes) have a green checkmark next to `Config Valid?`.
+
+3. **Initialize Switchover:**
+
+   - While still on the `Switchover` page, click the `RESET` button. This should initiate configuration and other checks.
+   - Make sure everything looks good and is validated.
+
+4. **Execute Switchover:**
+
+   - Click the `EXECUTE` button to perform the database switch.
+   - If the operation fails (it will fail safely), click `RESET` and then `EXECUTE` again.
+
+5. **Post-Switchover Configuration:**
+
+   - Once the switchover is successful, reconfigure all instances to use only `--db-url`, now pointing to the **NEW** database URL.
+   - Remove the `--db-url-next` flag or unset `GOALERT_DB_URL_NEXT`.
+
+   ```
+   goalert --db-url=<new-db-url>
+   ```
+
+   OR using environment variables:
+
+   ```
+   export GOALERT_DB_URL=<new-db-url>
+   unset GOALERT_DB_URL_NEXT
+   ```
+
+## Rollback Procedures
+
+In case you encounter issues during the switchover or decide to cancel the operation, you can do so safely by following these rollback steps:
+
+1. **Cancel Switchover in UI:**
+
+   - Navigate to the `Switchover` page under the `Admin` section in the UI.
+   - Click the `RESET` button. This will cancel the ongoing switchover process and restore the original database configuration.
+
+2. **Reconfigure Instances:**
+
+   - Redeploy or restart your GoAlert instances with the original `--db-url` flag, while removing the `--db-url-next` flag or the corresponding environment variable `GOALERT_DB_URL_NEXT`.
+
+   ```
+   goalert --db-url=<old-db-url>
+   ```
+
+   OR using environment variables:
+
+   ```
+   export GOALERT_DB_URL=<old-db-url>
+   unset GOALERT_DB_URL_NEXT
+   ```
+
+**Note:** After a successful switchover, the old database will be marked as obsolete. GoAlert instances will refuse to start if configured to use this old database. Therefore, rollback after a successful switchover is not possible without administrative intervention.

web/src/app/admin/switchover/AdminSwitchover.tsx

@@ -12,6 +12,9 @@ import { AdminSWODone } from './AdminSWODone'
 import { AdminSWOWrongMode } from './AdminSWOWrongMode'
 import { AdminSWODBVersionCard } from './AdminSWODBVersionCard'
 import { AdminSWOStatusCard } from './AdminSWOStatusCard'
+import { Button } from '@mui/material'
+import AppLink from '../../util/AppLink'
+import OpenInNewIcon from '@mui/icons-material/OpenInNew'
 
 const query = gql`
   query {
@@ -45,7 +48,7 @@ const mutation = gql`
   }
 `
 
-export default function AdminSwitchover(): JSX.Element {
+export function AdminSwitchoverInterface(): JSX.Element {
   const [{ fetching, error, data: _data }, refetch] = useQuery({
     query,
   })
@@ -154,3 +157,20 @@ export default function AdminSwitchover(): JSX.Element {
     </Grid>
   )
 }
+
+export default function AdminSwitchover(): JSX.Element {
+  return (
+    <React.Fragment>
+      <Button
+        variant='contained'
+        endIcon={<OpenInNewIcon />}
+        component={AppLink}
+        to='/admin/switchover/guide'
+        newTab
+      >
+        Switchover Guide
+      </Button>
+      <AdminSwitchoverInterface />
+    </React.Fragment>
+  )
+}

web/src/app/admin/switchover/AdminSwitchoverGuide.tsx

@@ -0,0 +1,8 @@
+import React from 'react'
+import swoGuide from '../../../../../docs/switchover.md'
+
+import Markdown from '../../util/Markdown'
+
+export default function AdminSwitchoverGuide(): JSX.Element {
+  return <Markdown value={swoGuide} />
+}

web/src/app/main/AppRoutes.tsx

@@ -39,6 +39,7 @@ import UserSessionList from '../users/UserSessionList'
 import { useSessionInfo } from '../util/RequireConfig'
 import WizardRouter from '../wizard/WizardRouter'
 import LocalDev from '../localdev/LocalDev'
+import AdminSwitchoverGuide from '../admin/switchover/AdminSwitchoverGuide'
 
 // ParamRoute will pass route parameters as props to the route's child.
 function ParamRoute(props: RouteProps): JSX.Element {
@@ -116,6 +117,7 @@ export const routes: Record<string, JSXElementConstructor<any>> = {
   '/admin/message-logs': AdminMessageLogsLayout,
   '/admin/alert-counts': AdminAlertCounts,
   '/admin/switchover': AdminSwitchover,
+  '/admin/switchover/guide': AdminSwitchoverGuide,
 
   '/wizard': WizardRouter,
   '/docs': Documentation,

web/src/app/util/AppLink.tsx

@@ -54,7 +54,7 @@ const AppLink: ForwardRefRenderFunction<HTMLAnchorElement, AppLinkProps> =
         ref={ref}
         to={to}
         href={to}
-        component={external ? 'a' : WrapLink}
+        component={external || newTab ? 'a' : WrapLink}
         {...other}
       />
     )