From 45b3e22610bfcca3a1d563998fc1fd02705399a7 Mon Sep 17 00:00:00 2001 From: Aline Abler Date: Mon, 19 Aug 2024 11:25:00 +0200 Subject: [PATCH 1/2] Document how to restore from k8up backup --- .../pages/how-tos/restore-from-backup.adoc | 117 ++++++++++++++++++ .../ROOT/pages/references/parameters.adoc | 6 + docs/modules/ROOT/partials/nav.adoc | 1 + docs/modules/ROOT/partials/vault-warning.adoc | 8 ++ 4 files changed, 132 insertions(+) create mode 100644 docs/modules/ROOT/pages/how-tos/restore-from-backup.adoc create mode 100644 docs/modules/ROOT/partials/vault-warning.adoc diff --git a/docs/modules/ROOT/pages/how-tos/restore-from-backup.adoc b/docs/modules/ROOT/pages/how-tos/restore-from-backup.adoc new file mode 100644 index 0000000..270f685 --- /dev/null +++ b/docs/modules/ROOT/pages/how-tos/restore-from-backup.adoc @@ -0,0 +1,117 @@ += Restore from Backup + +When configuring component-vault with `backup.enabled = true`, the component sets up regular backups using k8up. +This how-to explains how Vault can be restored from such a backup. + +== Information on the Vault Unseal Key and Root Token + +include::partial$vault-warning.adoc[] + +== Prerequisites + +* `restic` - command line tool to access backups made by k8up +* `vault` - command line tool to interact with Vault +* `kubectl` +* Write access to the cluster's tenant repository +* Read access to the restic repository in which k8up stored the Vault backups +* The Vault instance's *unseal key* and *root token* - these must be backed up manually; they are not part of the automated k8up backup. + +== Procedure + +=== 1. Set up new Vault instance + +* Add the `vault` application to your cluster configuration +* Initially disable backups by setting `.backups.enabled` to `false` +* Compile and push the cluster config and wait for Vault to start. + +=== 2. Retrieve the Vault snapshot + +* Set up the restic credentials (values correspond to the component parameters `backup.bucket` and `backup.password`) ++ +[source,shell] +---- +export AWS_ACCESS_KEY_ID="S3_KEY backup.bucket.accesskey" +export AWS_SECRET_ACCESS_KEY="S3_SECRET backup.bucket.secretkey" +export RESTIC_REPOSITORY="s3:https://path.to.my/bucket" +export RESTIC_PASSWORD="RESTIC_REPO_KEY backup.password" +---- +* Retrieve the latest Vault snapshot to your local disk ++ +[source,shell] +---- +mkdir restore +restic restore --target restore/ latest +---- +* Verify the snapshot file ++ +[source,shell] +---- +ls restore +# This should show a file named "[instance name]-backup.snapshot" +---- + +=== 3. Restore the snapshot + +* Expose the Vault pod ++ +[source,shell] +---- +kubectl port-forward -n $VAULT_INSTANCE_NAME ${VAULT_INSTANCE_NAME}-0 8200 +---- +* In a separate terminal, prepare the environment to access Vault ++ +[source,shell] +---- +# Get root token to log in +export VAULT_TOKEN="$(kubectl get secret -n $VAULT_INSTANCE_NAME ${VAULT_INSTANCE_NAME}-seal -ojsonpath='{.data.vault-root}' | base64 -d)" +export VAULT_ADDR="http://127.0.0.1:8200" +---- +* Restore the backup ++ +[source,shell] +---- +vault operator raft snapshot restore -force restore/${VAULT_INSTANCE_NAME}-backup.snapshot +---- + +=== 4. Unseal Vault + +If you were logged into the Vault UI, you should have gotten logged out now. +This is expected. + +* Open your browser at http://localhost:8200 +* Use the *Vault Unseal Key* of the Vault instance you've just restored to unseal Vault +* Use the *Vault root token* of the Vault instance you've just restored to log in with the `Token` method +* Verify that the restore worked, and secrets are now restored in Vault. + +[IMPORTANT] +==== +The unseal key and root token of the Vault instance you're restoring need to have been stored separately. +Without them, the restore procedure cannot be completed. +==== + +=== 5. Update the Vault Secret + +NOTE: Without this step, your Vault instance will not be able to auto-unseal. + +* Encode the Vault credentials ++ +[source,shell] +---- +export VAULT_UNSEAL_KEY="OLD_UNSEAL_KEY" +export VAULT_ROOT_TOKEN="OLD_ROOT_TOKEN" + +echo -n "$VAULT_UNSEAL_KEY" | base64 -w0 +echo -n "$VAULT_ROOT_TOKEN" | base64 -w0 +---- +* Update the Vault secret ++ +[source,shell] +---- +kubectl edit secret -n ${VAULT_INSTANCE_NAME} ${VAULT_INSTANCE_NAME}-seal +---- +* Update the `vault-root` and `vault-unseal-0` keys to reflect the values you have just encoded +* Save the secret + +=== 6. Cleanup + +* Reenable `backups.enabled` in the component configuration diff --git a/docs/modules/ROOT/pages/references/parameters.adoc b/docs/modules/ROOT/pages/references/parameters.adoc index f1d3dc8..54b5b6d 100644 --- a/docs/modules/ROOT/pages/references/parameters.adoc +++ b/docs/modules/ROOT/pages/references/parameters.adoc @@ -252,6 +252,12 @@ default:: `true` Whether to do backups using k8up. +[WARNING] +.Manual Setup Required +==== +include::partial$vault-warning.adoc[] +==== + == `backup.schedule` [horizontal] diff --git a/docs/modules/ROOT/partials/nav.adoc b/docs/modules/ROOT/partials/nav.adoc index 62bae9d..471a668 100644 --- a/docs/modules/ROOT/partials/nav.adoc +++ b/docs/modules/ROOT/partials/nav.adoc @@ -1,4 +1,5 @@ * xref:index.adoc[Home] * xref:tutorials/install.adoc[Installation] * xref:how-tos/lieutenant.adoc[Lieutenant Integration] +* xref:how-tos/restore-from-backup.adoc[Restore Vault from Backup] * xref:references/parameters.adoc[Parameters] diff --git a/docs/modules/ROOT/partials/vault-warning.adoc b/docs/modules/ROOT/partials/vault-warning.adoc new file mode 100644 index 0000000..1e2e18f --- /dev/null +++ b/docs/modules/ROOT/partials/vault-warning.adoc @@ -0,0 +1,8 @@ +`component-vault` leverages k8up's application specific backups to create Vault snapshots. +Restoring such a snapshot is only possible by providing the Vault unseal key and root token. +As these are sensitive secrets, they are not backed up as part of the component's automated backup process. + +When setting up Vault, you must store the Vault unseal key and root token in a safe location in order to be able to restore backups. + +In a running Vault instance, both of these secrets can be found in the `[instance name]-seal` secret in the Vault instance's namespace. +In order to have a reliable Vault backup, this secret must be backed up separately to a safe location. From 89b529b10fe65d604dbbd1600b6363fc7952eb50 Mon Sep 17 00:00:00 2001 From: Aline Abler Date: Mon, 26 Aug 2024 12:50:36 +0200 Subject: [PATCH 2/2] Add changes to address code review --- .../pages/how-tos/restore-from-backup.adoc | 58 ++++++++++++------- docs/modules/ROOT/partials/vault-warning.adoc | 2 +- 2 files changed, 38 insertions(+), 22 deletions(-) diff --git a/docs/modules/ROOT/pages/how-tos/restore-from-backup.adoc b/docs/modules/ROOT/pages/how-tos/restore-from-backup.adoc index 270f685..de79bcb 100644 --- a/docs/modules/ROOT/pages/how-tos/restore-from-backup.adoc +++ b/docs/modules/ROOT/pages/how-tos/restore-from-backup.adoc @@ -20,29 +20,30 @@ include::partial$vault-warning.adoc[] === 1. Set up new Vault instance -* Add the `vault` application to your cluster configuration -* Initially disable backups by setting `.backups.enabled` to `false` -* Compile and push the cluster config and wait for Vault to start. +. Add the `vault` application to your cluster configuration. +.. If your old instance of vault is still running on the cluster, you can use component instantiation to create a second instance by adding `vault as new-vault` to your application list, and configuring it under `new_vault`. +. Initially disable backups by setting `.backups.enabled` to `false` +. Compile and push the cluster config and wait for Vault to start. === 2. Retrieve the Vault snapshot -* Set up the restic credentials (values correspond to the component parameters `backup.bucket` and `backup.password`) +. Set up the restic credentials (values correspond to the component parameters `backup.bucket` and `backup.password`) + [source,shell] ---- -export AWS_ACCESS_KEY_ID="S3_KEY backup.bucket.accesskey" -export AWS_SECRET_ACCESS_KEY="S3_SECRET backup.bucket.secretkey" +export AWS_ACCESS_KEY_ID="S3_KEY" # from component configuration: backup.bucket.accesskey +export AWS_SECRET_ACCESS_KEY="S3_SECRET" # from component confiugration: backup.bucket.secretkey export RESTIC_REPOSITORY="s3:https://path.to.my/bucket" -export RESTIC_PASSWORD="RESTIC_REPO_KEY backup.password" +export RESTIC_PASSWORD="RESTIC_REPO_KEY" # from component configuration: backup.password ---- -* Retrieve the latest Vault snapshot to your local disk +. Retrieve the latest Vault snapshot to your local disk + [source,shell] ---- mkdir restore restic restore --target restore/ latest ---- -* Verify the snapshot file +. Verify the snapshot file + [source,shell] ---- @@ -52,13 +53,13 @@ ls restore === 3. Restore the snapshot -* Expose the Vault pod +. Expose the Vault pod + [source,shell] ---- kubectl port-forward -n $VAULT_INSTANCE_NAME ${VAULT_INSTANCE_NAME}-0 8200 ---- -* In a separate terminal, prepare the environment to access Vault +. In a separate terminal, prepare the environment to access Vault + [source,shell] ---- @@ -66,7 +67,7 @@ kubectl port-forward -n $VAULT_INSTANCE_NAME ${VAULT_INSTANCE_NAME}-0 8200 export VAULT_TOKEN="$(kubectl get secret -n $VAULT_INSTANCE_NAME ${VAULT_INSTANCE_NAME}-seal -ojsonpath='{.data.vault-root}' | base64 -d)" export VAULT_ADDR="http://127.0.0.1:8200" ---- -* Restore the backup +. Restore the backup + [source,shell] ---- @@ -78,10 +79,10 @@ vault operator raft snapshot restore -force restore/${VAULT_INSTANCE_NAME}-backu If you were logged into the Vault UI, you should have gotten logged out now. This is expected. -* Open your browser at http://localhost:8200 -* Use the *Vault Unseal Key* of the Vault instance you've just restored to unseal Vault -* Use the *Vault root token* of the Vault instance you've just restored to log in with the `Token` method -* Verify that the restore worked, and secrets are now restored in Vault. +. Open your browser at http://localhost:8200 +. Use the *Vault Unseal Key* of the Vault instance you've just restored to unseal Vault +. Use the *Vault root token* of the Vault instance you've just restored to log in with the `Token` method +. Verify that the restore worked, and secrets are now restored in Vault. [IMPORTANT] ==== @@ -93,7 +94,7 @@ Without them, the restore procedure cannot be completed. NOTE: Without this step, your Vault instance will not be able to auto-unseal. -* Encode the Vault credentials +. Encode the Vault credentials + [source,shell] ---- @@ -103,15 +104,30 @@ export VAULT_ROOT_TOKEN="OLD_ROOT_TOKEN" echo -n "$VAULT_UNSEAL_KEY" | base64 -w0 echo -n "$VAULT_ROOT_TOKEN" | base64 -w0 ---- -* Update the Vault secret +. Update the Vault secret + [source,shell] ---- kubectl edit secret -n ${VAULT_INSTANCE_NAME} ${VAULT_INSTANCE_NAME}-seal ---- -* Update the `vault-root` and `vault-unseal-0` keys to reflect the values you have just encoded -* Save the secret +. Update the `vault-root` and `vault-unseal-0` keys to reflect the values you have just encoded +. Save the secret +. Verify that auto-unseal works: +.. Restart all vault pods simultaneously: ++ +[source,shell] +---- +kubectl delete pod -n $VAULT_INSTANCE_NAME ${VAULT_INSTANCE_NAME}-{0..2} +---- +.. Expose the Vault UI ++ +[source,shell] +---- +kubectl port-forward -n $VAULT_INSTANCE_NAME ${VAULT_INSTANCE_NAME}-0 8200 +---- +.. Verify that http://localhost:8200[the Vault UI] does not prompt you for the unseal key + === 6. Cleanup -* Reenable `backups.enabled` in the component configuration +. Reenable `backups.enabled` in the component configuration diff --git a/docs/modules/ROOT/partials/vault-warning.adoc b/docs/modules/ROOT/partials/vault-warning.adoc index 1e2e18f..8b453b5 100644 --- a/docs/modules/ROOT/partials/vault-warning.adoc +++ b/docs/modules/ROOT/partials/vault-warning.adoc @@ -1,5 +1,5 @@ `component-vault` leverages k8up's application specific backups to create Vault snapshots. -Restoring such a snapshot is only possible by providing the Vault unseal key and root token. +Restoring such a snapshot is only possible by providing the Vault https://developer.hashicorp.com/vault/docs/concepts/seal[unseal key] and https://developer.hashicorp.com/vault/docs/concepts/tokens#root-tokens[root token]. As these are sensitive secrets, they are not backed up as part of the component's automated backup process. When setting up Vault, you must store the Vault unseal key and root token in a safe location in order to be able to restore backups.