Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Staging to production #537

Merged
merged 15 commits into from
Oct 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion compose-dev.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
services:
lucee:
image: ${LUCEE_IMAGE}
image: ${LUCEE_IMAGE}:${LUCEE_IMAGE_VERSION}
ports:
- "${LUCEE_PORT}:80"
restart: always
Expand Down
2 changes: 1 addition & 1 deletion compose.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
services:
lucee:
image: ${LUCEE_IMAGE}
image: ${LUCEE_IMAGE}:${LUCEE_IMAGE_VERSION}
ports:
- "${LUCEE_PORT}:80"
restart: always
Expand Down
48 changes: 48 additions & 0 deletions config/backup/backup.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
#!/bin/bash

# Enable automatic export of variables
set -a

# Load the .env file from the project root
source "$(dirname "$0")/../../.env"

# Dynamically generate volume names based on the project
DB_VOLUME="${COMPOSE_PROJECT_NAME}_db_volume"
USERDATA_VOLUME="${COMPOSE_PROJECT_NAME}_userdata_volume"

# Date format for versioning (e.g., 20241022_2300)
TIMESTAMP=$(date +"%Y%m%d_%H%M")

# Check if the /backup folder exists and create it if necessary
if [ ! -d "/backup" ]; then
mkdir -p /backup
fi

# Create remote directories if they don't exist
ssh -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP} "mkdir -p ${REMOTE_BACKUP_PATH}/db ${REMOTE_BACKUP_PATH}/userdata ${REMOTE_BACKUP_PATH}/lucee"

# Backup database volume and store in the remote db directory
docker run --rm -v ${DB_VOLUME}:/volume -v /backup:/backup alpine sh -c "tar -czf /backup/database_${TIMESTAMP}.tar.gz -C /volume ."
scp -i ${SSH_KEY_PATH} /backup/database_${TIMESTAMP}.tar.gz ${SERVER_USER}@${SERVER_IP}:${REMOTE_BACKUP_PATH}/db/

# Backup userdata volume and store in the remote userdata directory
docker run --rm -v ${USERDATA_VOLUME}:/volume -v /backup:/backup alpine sh -c "tar -czf /backup/userdata_${TIMESTAMP}.tar.gz -C /volume ."
scp -i ${SSH_KEY_PATH} /backup/userdata_${TIMESTAMP}.tar.gz ${SERVER_USER}@${SERVER_IP}:${REMOTE_BACKUP_PATH}/userdata/

# Backup Lucee image and store in the remote lucee directory
docker save -o /backup/image_${LUCEE_IMAGE}_${LUCEE_IMAGE_VERSION}_${TIMESTAMP}.tar ${LUCEE_IMAGE}:${LUCEE_IMAGE_VERSION}
scp -i ${SSH_KEY_PATH} /backup/image_${LUCEE_IMAGE}_${LUCEE_IMAGE_VERSION}_${TIMESTAMP}.tar ${SERVER_USER}@${SERVER_IP}:${REMOTE_BACKUP_PATH}/lucee/

# Rotate backups: Keep only the latest 30 backups per type

# For database backups
ssh -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP} "cd ${REMOTE_BACKUP_PATH}/db && ls -tp | grep -v '/$' | tail -n +31 | xargs -I {} rm -- {}"

# For userdata backups
ssh -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP} "cd ${REMOTE_BACKUP_PATH}/userdata && ls -tp | grep -v '/$' | tail -n +31 | xargs -I {} rm -- {}"

# For Lucee image backups
ssh -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP} "cd ${REMOTE_BACKUP_PATH}/lucee && ls -tp | grep -v '/$' | tail -n +31 | xargs -I {} rm -- {}"

# Disable automatic export of variables
set +a
74 changes: 74 additions & 0 deletions config/backup/readme.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@

# Backup for the Production and Staging Environments

**Purpose**

This directory contains the necessary configurations and scripts for backing up and restoring the **database**, **user data**, and the **Lucee image** in the **production** and **staging** environments. The backup and restore processes are automated using Docker and shell scripts to ensure consistency, reliability, and minimal manual intervention.

Note: These scripts are not intended for use in the **development** environment.


## **Structure**

- **backup.sh**: A shell script that automates the process of creating backups for the **database volume**, **user data volume**, and **Lucee image**. Each backup is timestamped to ensure that multiple versions can be maintained. The script also uses `scp` to transfer the backups securely to a remote server.

- **restore.sh**: A shell script that automates the process of restoring backups from the remote server. It retrieves the backups for the **database**, **user data**, and **Lucee image**, and restores them to the appropriate Docker volumes.

- **.env**: Contains environment variables required for the backup and restore processes, such as volume names, SSH key, server IP, and remote paths. The backup does not have its own `.env` file; instead, it uses the `.env` file from the main project.

## **Usage**

### **Backup**

To create a backup of the **database**, **user data**, and **Lucee image**, follow these steps:

1. Ensure that the `.env` file is correctly configured with your **production** or **staging** environment settings (e.g., volume names, remote server path, SSH keys, and server IP).

2. Navigate to the `config/backup/` directory:
`cd config/backup/`

3. Run the backup script:
`bash backup.sh`

This will:

- Backup the **database volume**.
- Backup the **user data volume**.
- Backup the **Lucee image**.
- Securely transfer all backups to the remote backup server.

Each backup will be **timestamped** in the format `YYYYMMDD_HHMM`, ensuring you can differentiate between multiple backup versions.

### **Restore**

To restore from a backup, first navigate to the `config/backup/` directory:
`cd config/backup/`

Then, if you run the restore script without any parameters:
`bash restore.sh`

It will display a list of available options, such as:
Usage: restore.sh [--db [TIMESTAMP]] [--userdata [TIMESTAMP]] [--lucee-image [TIMESTAMP]] [--list]

To perform a restore, you need to specify which backup you want to restore by using one of the following options:
- To restore the **latest** backup for the **database**:
`bash restore.sh --db`

- To restore a **specific** database backup by **timestamp**:
`bash restore.sh --db 20241019_2300`

- To list all available backups on the remote server:
`bash restore.sh --list`


## **Automating Backups**

To automate the backup process, you can set up a **cron job** to run the backup script at regular intervals (e.g., daily). For example, to run the backup every night at midnight, add the following entry to your crontab:
`0 0 * * * /path/to/your/project/config/backup/backup.sh`


## **Notes**

- These backups are intended for the **production** and **staging** environments. Ensure that the environment variables in the `.env` file are correctly configured before running any backups or restores.
- The backup script automatically **rotates** backups, keeping only the **latest 30 backups** per backup type (database, user data, Lucee image) by removing older backups on the remote server.
- Always ensure that your **SSH keys** and **server information** are secure, as they are used for transferring backups between the production or staging environment and the remote server.
93 changes: 93 additions & 0 deletions config/backup/restore.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
#!/bin/bash

# Enable automatic export of variables
set -a

# Load the .env file from the project root
source "$(dirname "$0")/../../.env"

# Dynamically generate volume names based on the project
DB_VOLUME="${COMPOSE_PROJECT_NAME}_db_volume"
USERDATA_VOLUME="${COMPOSE_PROJECT_NAME}_userdata_volume"

# Check if the /restore folder exists and create it if necessary
if [ ! -d "/restore" ]; then
mkdir -p /restore
fi

# Functions for the various restore processes

restore_db() {
if [ -z "$1" ]; then
echo "Restoring the latest database backup..."
# Get the latest database backup
LATEST_DB_BACKUP=$(ssh -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP} "ls -t ${REMOTE_BACKUP_PATH}/db | head -n 1")
else
echo "Restoring database backup from $1..."
LATEST_DB_BACKUP="database_$1.tar.gz"
fi
scp -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP}:${REMOTE_BACKUP_PATH}/db/${LATEST_DB_BACKUP} /restore/database.tar.gz
docker run --rm -v ${DB_VOLUME}:/volume -v /restore:/restore alpine sh -c "tar -xzf /restore/database.tar.gz -C /volume"
docker restart ${MYSQL_CONTAINER_NAME}
}

restore_userdata() {
if [ -z "$1" ]; then
echo "Restoring the latest userdata backup..."
# Get the latest userdata backup
LATEST_USERDATA_BACKUP=$(ssh -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP} "ls -t ${REMOTE_BACKUP_PATH}/userdata | head -n 1")
else
echo "Restoring userdata backup from $1..."
LATEST_USERDATA_BACKUP="userdata_$1.tar.gz"
fi
scp -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP}:${REMOTE_BACKUP_PATH}/userdata/${LATEST_USERDATA_BACKUP} /restore/userdata.tar.gz
docker run --rm -v ${USERDATA_VOLUME}:/volume -v /restore:/restore alpine sh -c "tar -xzf /restore/userdata.tar.gz -C /volume"
docker restart ${LUCEE_CONTAINER_NAME}
}

restore_lucee_image() {
if [ -z "$1" ]; then
echo "Restoring the latest Lucee image backup..."
# Get the latest Lucee image backup
LATEST_LUCEE_BACKUP=$(ssh -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP} "ls -t ${REMOTE_BACKUP_PATH}/lucee | head -n 1")
else
echo "Restoring Lucee image backup from $1..."
LATEST_LUCEE_BACKUP="image_${LUCEE_IMAGE}_${LUCEE_IMAGE_VERSION}_$1.tar"
fi
scp -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP}:${REMOTE_BACKUP_PATH}/lucee/${LATEST_LUCEE_BACKUP} /restore/image_${LUCEE_IMAGE}_${LUCEE_IMAGE_VERSION}.tar
docker load -i /restore/image_${LUCEE_IMAGE}_${LUCEE_IMAGE_VERSION}.tar
}

list_backups() {
echo "Available backups on remote server:"
echo "Database backups:"
ssh -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP} "ls ${REMOTE_BACKUP_PATH}/db"
echo ""
echo "Userdata backups:"
ssh -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP} "ls ${REMOTE_BACKUP_PATH}/userdata"
echo ""
echo "Lucee image backups:"
ssh -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP} "ls ${REMOTE_BACKUP_PATH}/lucee"
echo ""
}

# Show help if no options have been specified
if [ $# -eq 0 ]; then
echo "Usage: $0 [--db [TIMESTAMP]] [--userdata [TIMESTAMP]] [--lucee-image [TIMESTAMP]] [--list]"
exit 1
fi

# Process the specified options
while [[ "$#" -gt 0 ]]; do
case $1 in
--db) restore_db "$2"; shift ;;
--userdata) restore_userdata "$2"; shift ;;
--lucee-image) restore_lucee_image "$2"; shift ;;
--list) list_backups; exit 0 ;;
*) echo "Unknown option: $1"; exit 1 ;;
esac
shift
done

# Disable automatic export of variables
set +a
22 changes: 18 additions & 4 deletions config/example.env
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
## Project name
COMPOSE_PROJECT_NAME=saaster # Must be unique on docker host.

## Lucee settings (default: lucee/lucee:6.0-nginx)
LUCEE_IMAGE=lucee/lucee:6.0-nginx # For the initial setup, please leave it as is, then use your own image name afterward.
## Lucee settings
# Keep the Lucee image name for setup, then replace with your own later.
# Default: lucee/lucee:6.0-nginx
LUCEE_IMAGE=lucee/lucee
LUCEE_IMAGE_VERSION=6.0-nginx
LUCEE_CONTAINER_NAME=saaster_lucee # Must be unique on docker host.
LUCEE_PORT=8080 # Must be unique on docker host.
LUCEE_ADMIN_PASSWORD=defaultpass
Expand All @@ -22,8 +25,19 @@ FLYWAY_DB_FOLDER=core # core or myapp
FLYWAY_MIGRATION_TYPE=migrate # migrate or repair
FLYWAY_CONTAINER_NAME=saaster_flyway # Must be unique on docker host.

## Inbucket settings (dev)
## Inbucket settings (development environment)
INBUCKET_CONTAINER_NAME=saaster_inbucket # Will be used as the SMTP server in Lucee.
INBUCKET_SMTP_PORT=2500 # Must be unique on docker host. In Lucee Admin please set to 2500.
INBUCKET_WEB_PORT=9000 # Must be unique on docker host.
INBUCKET_POP3_PORT=1100 # Must be unique on docker host.
INBUCKET_POP3_PORT=1100 # Must be unique on docker host.


## Backup settings

# Backup path on the remote server
REMOTE_BACKUP_PATH=/backups

# Server information for SCP Transfer
SERVER_USER=root
SERVER_IP=xxx.xxx.xxx.xxx
SSH_KEY_PATH=~/.ssh/id_rsa