diff --git a/docs/kubernetes/backup.md b/docs/kubernetes/backup.md new file mode 100644 index 000000000..0b992417e --- /dev/null +++ b/docs/kubernetes/backup.md @@ -0,0 +1,131 @@ +# Filestore Backups User Guide (Beta) + +>**Attention:** Filestore Backup relies on CSI VolumeSnapshot which is a Beta feature in k8s enabled by default in +Kubernetes 1.17+. CSI VolumeSnapshot should not be confused with Filestore Backups. Filestore CSI driver leverages CSI VolumeSnapshot capability to support Filestore Backups by specifying a `type` parameter in the VolumeSnapshotClass object. + +>**Attention:** VolumeSnapshot is only available in the driver version "master". + +>**Prerequisites:**: Volume Snapshot CRDs and snapshot-controller needs to be installed for the backup example to work. Please refer to [this](https://kubernetes-csi.github.io/docs/snapshot-controller.html#deployment) for additional details. GKE clusters 1.17+ come pre-installed with the above mentioned CRDs and snapshot controller, see [here](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/volume-snapshots). + +### Backup Example +A Filestore backup is a copy of a file share that includes all file data and metadata of the file share from the point in time when the backup is created. It works for Basic HDD and Basic SSD tier instances. Once a backup of a file share is created, the original file share can be modified or deleted without affecting the backup. A file share can be completely restored from a backup as a new Filestore instance or onto an existing file share. For more details refer to this [documentation](https://cloud.google.com/filestore/docs/backups) + +The [CSI Snapshot](https://github.com/container-storage-interface/spec/blob/master/spec.md#createsnapshot) feature is leveraged to create Filestore Backups. By specifying a `type: backup` field in the VolumeSnapshotClass parameters, filestore CSI driver understands how to initiate a backup for a Filestore instance backed by the Persistent Volume. In future release when Filestore snapshots will be supported, an appropriate `type` parameter will be set in the VolumeSnapshotClass to indicate Filestore snapshots. + +1. Create `StorageClass` + + If you haven't created a `StorageClass` yet, create one first: + + ```console + $ kubectl apply -f ./examples/kubernetes/backups-restore/sc.yaml + ``` + + If a non-default network is used for the filestore instance, provide a network paramter to the storage class. + + ```yaml + apiVersion: storage.k8s.io/v1 + kind: StorageClass + metadata: + name: csi-filestore + provisioner: filestore.csi.storage.gke.io + parameters: + network: # Change this network as per the deployment + volumeBindingMode: WaitForFirstConsumer + allowVolumeExpansion: true + ``` + +2. Create default `VolumeSnapshotClass` + + ```console + $ kubectl create -f ./examples/kubernetes/backups-restore/backup-volumesnapshotclass.yaml + ``` + +3. Create source PVC and Pod + + ```console + kubectl create -f ./examples/kubernetes/backups-restore/source-pod-pvc.yaml + ``` +4. Wait for PVC to reach 'Bound' status. + ```console + $ kubectl get pvc source-pvc + NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE + source-pvc Bound pvc-62e3d593-8844-483b-8884-bb315678b3b7 1Ti RWX csi-filestore 27m + ``` + +5. Generate sample data + + The source PVC is mounted into `/demo/data` directory of this pod. This pod will create a file `sample-file.txt` in `/demo/data` directory. Check if the file has been created successfully: + + ```console + $ kubectl exec source-pod -- ls /demo/data/ + ``` + + The output should be: + + ``` + lost+found + sample-file.txt + ``` + +6. Create a `VolumeSnapshot` of the source PVC (This internally generates Filestore instance backup) + + ```console + $ kubectl create -f ./examples/kubernetes/backups-restore/backup.yaml + ``` + +7. Verify that `VolumeSnapshot` has been created and it is ready to use: + + ```console + $ kubectl get volumesnapshot backup-source-pvc -o yaml + ``` + + The output is similar to this: + + ```yaml + apiVersion: snapshot.storage.k8s.io/v1beta1 + kind: VolumeSnapshot + metadata: + creationTimestamp: "2020-11-13T03:04:03Z" + finalizers: + - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection + - snapshot.storage.kubernetes.io/volumesnapshot-bound-protection + ... + spec: + source: + persistentVolumeClaimName: source-pvc + volumeSnapshotClassName: csi-gcp-filestore-backup-snap-class + status: + boundVolumeSnapshotContentName: snapcontent-191fde18-5eb3-4bb1-9f64-0356765c3f9f + creationTime: "2020-11-13T03:04:39Z" + readyToUse: true + restoreSize: 1Ti + ``` + +8. Restore the `VolumeSnapshot` into a new PVC and create a pod to use the PVC: + + Create a new PVC. Specify `spec.dataSource` section to restore from VolumeSnapshot `backup-source-pvc`. + + ```console + $ kubectl create -f ./examples/kubernetes/backups-restore/restored-pod-pvc.yaml + ``` +9. Wait for PVC to reach 'Bound' status. + ```console + $ kubectl get pvc restored-pvc + NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE + restored-pvc Bound pvc-53ced778-6a28-4960-aeb7-82b7bb093981 1Ti RWX csi-filestore 4m39s + ``` + +10. Verify sample data has been restored: + + Check data has been restored in `/demo/data` directory: + + ```console + $ kubectl exec restored-pod -- ls /demo/data/ + ``` + + Verify that the output is: + + ``` + lost+found + sample-file.txt + ``` diff --git a/docs/kubernetes/pre-provisioned-pv.md b/docs/kubernetes/pre-provisioned-pv.md new file mode 100644 index 000000000..d3ac86376 --- /dev/null +++ b/docs/kubernetes/pre-provisioned-pv.md @@ -0,0 +1,62 @@ +# Kubernetes Pre-Provisioned Filesore instance User Guide + +This guide gives a simple example on how to use this driver with filestore instances that have +been pre-provisioned by an administrator. + +## Pre-Provision a Filestore instance + +If you have not already pre-provisioned a filestore instance on GCP you can do that now. + +1. Create a filestore instance following the instructions [here](https://cloud.google.com/filestore/docs/creating-instances) + + +## Create Persistent Volume for Filestore instance + +1. Create example Storage Class + +```bash +kubectl apply -f ./examples/kubernetes/sc-latebind.yaml +``` + +2. Create example Persistent Volume + +**Note:** The `volumeHandle` should be updated +based on the zone, Filestore instance name, and share name created. `storage` value +should be generated based on the size of the underlying instance. VolumeAttributes `ip` must +point to the filestore instance IP, and `volume` must point to the [fileshare](https://cloud.google.com/filestore/docs/reference/rest/v1beta1/projects.locations.instances#FileShareConfig) name. + +```bash +kubectl apply -f ./examples/kubernetes/pre-provision/preprov-pv.yaml +``` + +## Use Persistent Volume In Pod + +1. Create example PVC and Pod + +```bash +$ kubectl apply -f ./examples/kubernetes/pre-provision/preprov-pod-demo.yaml +``` + +2. Verify PV is created and bound to PVC + +```bash +$ kubectl get pvc +NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE +preprov-pvc Bound my-pre-pv 1Ti RWX csi-filestore 76s +``` + +3. Verify pod is created and in `RUNNING` state (it may take a few minutes to + get to running state) + +```bash +$ kubectl get pods +NAME READY STATUS RESTARTS AGE +web-server-1 1/1 Running 0 119s +``` + +4. Verify contents of the mounted volume + +```bash +kubectl exec web-server-1 -- ls /usr/share/nginx/html +lost+found +``` \ No newline at end of file diff --git a/docs/kubernetes/resize.md b/docs/kubernetes/resize.md new file mode 100644 index 000000000..65e381bf1 --- /dev/null +++ b/docs/kubernetes/resize.md @@ -0,0 +1,190 @@ +# Kubernetes Resize User Guide + +>**Attention:** Volume Resize is a Kubernetes Beta feature enabled by default in 1.16+. +>**Attention:** Volume Resize is only available in the driver version master + +### Resize Example + +This example dynamically provisions a filestore instance and performs online resize of the instance (i.e while the volume is mounted on a Pod). For more details about CSI VolumeExpansion capability see [here](https://kubernetes-csi.github.io/docs/volume-expansion.html) + +1. Ensure resize field `allowVolumeExpansion`is set to True, in the example Zonal Storage Class + ``` + apiVersion: storage.k8s.io/v1 + kind: StorageClass + metadata: + name: csi-filestore + provisioner: filestore.csi.storage.gke.io + volumeBindingMode: WaitForFirstConsumer + allowVolumeExpansion: true + ``` + +2. Create example Zonal Storage Class with resize enabled + ``` + $ kubectl apply -f ./examples/kubernetes/sc-latebind.yaml + ``` + +3. Create example PVC and Pod + ``` + $ kubectl apply -f ./examples/kubernetes/demo-pod.yaml + ``` + +4. Verify PV is created and bound to PVC + ``` + $ kubectl get pvc test-pvc + NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE + test-pvc Bound pvc-def880a7-b6a6-4b12-b8ad-4fa6d928c142 1Ti RWX csi-filestore 9m36s + ``` + +5. Verify pod is created and in `RUNNING` state (it may take a few minutes to get to running state) + ``` + $ kubectl get pods + NAME READY STATUS RESTARTS AGE + web-server 1/1 Running 0 1m + ``` + +6. Check current filesystem size on the running pod + ``` + $ kubectl exec web-server -- df -h /usr/share/nginx/html + Filesystem Size Used Avail Use% Mounted on + :/vol1 1007G 76M 956G 1% /usr/share/nginx/html + ``` + +7. Get the zone information from the `volumeHandle` of PV spec + ```console + $ kubectl get pv pvc-def880a7-b6a6-4b12-b8ad-4fa6d928c142 -o yaml + ``` + ```yaml + apiVersion: v1 + kind: PersistentVolume + metadata: + annotations: + pv.kubernetes.io/provisioned-by: filestore.csi.storage.gke.io + creationTimestamp: "2020-11-13T05:15:21Z" + finalizers: + - kubernetes.io/pv-protection + ... + spec: + accessModes: + - ReadWriteMany + capacity: + storage: 1Ti + claimRef: + apiVersion: v1 + kind: PersistentVolumeClaim + name: test-pvc + namespace: default + resourceVersion: "18588" + uid: def880a7-b6a6-4b12-b8ad-4fa6d928c142 + csi: + driver: filestore.csi.storage.gke.io + fsType: ext4 + volumeAttributes: + ip: + storage.kubernetes.io/csiProvisionerIdentity: 1605236375597-8081-filestore.csi.storage.gke.io + volume: vol1 + volumeHandle: modeInstance/us-central1-b/pvc-def880a7-b6a6-4b12-b8ad-4fa6d928c142/vol1 + persistentVolumeReclaimPolicy: Delete + storageClassName: csi-filestore + volumeMode: Filesystem + status: + phase: Bound + ``` +7. Verify the filestore instance properties: + + ```console + $ gcloud beta filestore instances describe pvc-def880a7-b6a6-4b12-b8ad-4fa6d928c142 --zone us-central1-b + ``` + + ```yaml + createTime: '2020-11-13T05:13:10.929454142Z' + fileShares: + - capacityGb: '1024' + name: vol1 + nfsExportOptions: + - accessMode: READ_WRITE + ipRanges: + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + squashMode: NO_ROOT_SQUASH + labels: + kubernetes_io_created-for_pv_name: pvc-def880a7-b6a6-4b12-b8ad-4fa6d928c142 + kubernetes_io_created-for_pvc_name: test-pvc + kubernetes_io_created-for_pvc_namespace: default + storage_gke_io_created-by: filestore_csi_storage_gke_io + name: projects//locations/us-central1-b/instances/pvc-def880a7-b6a6-4b12-b8ad-4fa6d928c142 + networks: + - ipAddresses: + - + modes: + - MODE_IPV4 + network: default + reservedIpRange: + state: READY + tier: STANDARD + ``` + +7. Resize volume by modifying the field `spec -> resources -> requests -> storage` + ``` + $ kubectl edit pvc test-pvc + apiVersion: v1 + kind: PersistentVolumeClaim + ... + spec: + resources: + requests: + storage: 2Ti + ... + ... + ``` +8. Verify the PVC and PV reflect the new changes + ``` + $ kubectl get pvc test-pvc + NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE + test-pvc Bound pvc-86fb8466-68ee-48a8-bca8-d6c02538962f 2Ti RWX csi-filestore 11m + + $ kubectl get pv pvc-86fb8466-68ee-48a8-bca8-d6c02538962f + NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE + pvc-86fb8466-68ee-48a8-bca8-d6c02538962f 2Ti RWX Delete Bound default/test-pvc csi-filestore 9m46s + ``` + +9. Verify filesystem resized on the running pod + ``` + $ kubectl exec web-server -- df -h /usr/share/nginx/html + Filesystem Size Used Avail Use% Mounted on + :/vol1 2.0T 70M 1.9T 1% /usr/share/nginx/html + ``` +10. Verify the filestore instance properties: + + ```console + $ gcloud beta filestore instances describe pvc-def880a7-b6a6-4b12-b8ad-4fa6d928c142 --zone us-central1-b + ``` + + ```yaml + createTime: '2020-11-13T05:13:10.929454142Z' + fileShares: + - capacityGb: '2048' # Size doubled + name: vol1 + nfsExportOptions: + - accessMode: READ_WRITE + ipRanges: + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + squashMode: NO_ROOT_SQUASH + labels: + kubernetes_io_created-for_pv_name: pvc-def880a7-b6a6-4b12-b8ad-4fa6d928c142 + kubernetes_io_created-for_pvc_name: test-pvc + kubernetes_io_created-for_pvc_namespace: default + storage_gke_io_created-by: filestore_csi_storage_gke_io + name: projects//locations/us-central1-b/instances/pvc-def880a7-b6a6-4b12-b8ad-4fa6d928c142 + networks: + - ipAddresses: + - + modes: + - MODE_IPV4 + network: default + reservedIpRange: + state: READY + tier: STANDARD + ``` \ No newline at end of file diff --git a/docs/kubernetes/topology.md b/docs/kubernetes/topology.md new file mode 100644 index 000000000..f1a2fbe55 --- /dev/null +++ b/docs/kubernetes/topology.md @@ -0,0 +1,127 @@ +# Kubernetes Topology User Guide + +>**Attention:** Topology is a Kubernetes feature enabled by default in 1.14-1.16(Beta) and 1.17+(GA). +>**Attention:** Topology is only available in the driver master version + +To access Filestore instances, the Compute engine VM instances (or the kubernetes cluster composed of those instances) must be in the same Google Cloud project and VPC network as the Filestore instance, unless the Filestore instance is on a shared VPC network. Once an instance is created, its authorized network cannot be changed. Filestore instances are zonal resources available to all instances in a given VPC network. CSI topology feature can be leveraged to hand pick a zone (or a candidate set of zones) where an instance can be deployed dynamically. For more details into the CSI topology feature see [here](https://kubernetes-csi.github.io/docs/topology.html) + +### CSI Topology with Immediate binding mode Example + +This example dynamically provisions a filestore instance and uses storage class `allowedTopologies` parameter to pick the zone where a filestore instance is deployed. + +1. Create `StorageClass` + + ```console + $ kubectl apply -f ./examples/kubernetes/topology/immediate-binding/sc-immediate-allowedtopo.yaml + ``` + If the filestore instance is going to use a non-default network, setup the `network` + + ```yaml + apiVersion: storage.k8s.io/v1 + kind: StorageClass + metadata: + name: csi-filestore-immediate-binding-allowedtopo + provisioner: filestore.csi.storage.gke.io + volumeBindingMode: Immediate + allowVolumeExpansion: true + parameters: + network: # Change this network as per the deployment + allowedTopologies: + - matchLabelExpressions: + - key: topology.gke.io/zone + values: + # Change this to the intended zone (or set of zones). + - us-central1-a + - us-central1-b + ``` + +2. Wait for PVC to reach 'Bound' status. + ```console + $ kubectl get pvc test-pvc-fs-immediate-binding-allowedtopo + NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE + test-pvc-fs-immediate-binding-allowedtopo Bound pvc-64e6ce36-523d-4172-b3b3-3c1080ab0b9e 1Ti RWX csi-filestore-immediate-binding-allowedtopo 5m7s + ``` +3. Verify that the `volumeHandle` captured in the PersistentVolume object specifies the intended zone. + ```yaml + kubectl get pv pvc-64e6ce36-523d-4172-b3b3-3c1080ab0b9e -o yaml + apiVersion: v1 + kind: PersistentVolume + metadata: + annotations: + pv.kubernetes.io/provisioned-by: filestore.csi.storage.gke.io + creationTimestamp: "2020-11-13T03:48:48Z" + finalizers: + - kubernetes.io/pv-protection + ... + spec: + accessModes: + - ReadWriteMany + capacity: + storage: 1Ti + claimRef: + apiVersion: v1 + kind: PersistentVolumeClaim + name: test-pvc-fs-immediate-binding-allowedtopo + namespace: default + resourceVersion: "8013" + uid: 64e6ce36-523d-4172-b3b3-3c1080ab0b9e + csi: + driver: filestore.csi.storage.gke.io + fsType: ext4 + volumeAttributes: + ip: + storage.kubernetes.io/csiProvisionerIdentity: 1605236375597-8081-filestore.csi.storage.gke.io + volume: vol1 + volumeHandle: modeInstance/us-central1-a/pvc-64e6ce36-523d-4172-b3b3-3c1080ab0b9e/vol1 + persistentVolumeReclaimPolicy: Delete + storageClassName: csi-filestore-immediate-binding-allowedtopo + volumeMode: Filesystem + status: + phase: Bound + ``` +4. Verify the filestore instance properties + + ```console + $ gcloud beta filestore instances describe pvc-64e6ce36-523d-4172-b3b3-3c1080ab0b9e --zone us-central1-a + ``` + ```yaml + createTime: '2020-11-13T03:46:18.870400740Z' + fileShares: + - capacityGb: '1024' + name: vol1 + nfsExportOptions: + - accessMode: READ_WRITE + ipRanges: + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + squashMode: NO_ROOT_SQUASH + labels: + kubernetes_io_created-for_pv_name: pvc-64e6ce36-523d-4172-b3b3-3c1080ab0b9e + kubernetes_io_created-for_pvc_name: test-pvc-fs-immediate-binding-allowedtopo + kubernetes_io_created-for_pvc_namespace: default + storage_gke_io_created-by: filestore_csi_storage_gke_io + name: projects//locations/us-central1-a/instances/pvc-64e6ce36-523d-4172-b3b3-3c1080ab0b9e + networks: + - ipAddresses: + - + modes: + - MODE_IPV4 + network: default + reservedIpRange: + state: READY + tier: STANDARD + ``` + +5. Ensure that the deployment is up and running. + ```console + $ kubectl get deployment + NAME READY UP-TO-DATE AVAILABLE AGE + web-server-immediate-allowedtopo 5/5 5 5 6m10s + ``` + +### CSI Topology with WaitForFirstCustomer binding mode Example + +The steps are same as Immediate mode binding. Use the following yamls `./examples/kubernetes/topology/delayed-binding/sc-delayed-allowedtopo.yaml` and `./examples/kubernetes/topology/delayed-binding/demo-deployment-delayed-allowedtopo.yaml`. +If the topology of the node selected by the scheduler is not in `allowedTopology` parameter of StorageClass, provisioning fails +and the scheduler will continue with a different node. \ No newline at end of file diff --git a/examples/kubernetes/pre-provision/preprov-pod-demo.yaml b/examples/kubernetes/pre-provision/preprov-pod-demo.yaml index 0928dc23b..6ba31dbae 100644 --- a/examples/kubernetes/pre-provision/preprov-pod-demo.yaml +++ b/examples/kubernetes/pre-provision/preprov-pod-demo.yaml @@ -13,3 +13,15 @@ spec: - name: mypvc persistentVolumeClaim: claimName: preprov-pvc +--- +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: preprov-pvc +spec: + accessModes: + - ReadWriteMany + storageClassName: csi-filestore + resources: + requests: + storage: 1Ti \ No newline at end of file diff --git a/examples/kubernetes/pre-provision/preprov-pv.yaml b/examples/kubernetes/pre-provision/preprov-pv.yaml index 21fa2d7b7..2bbc10b3f 100644 --- a/examples/kubernetes/pre-provision/preprov-pv.yaml +++ b/examples/kubernetes/pre-provision/preprov-pv.yaml @@ -14,9 +14,8 @@ spec: volumeMode: "Filesystem" csi: driver: "filestore.csi.storage.gke.io" + # Modify this to use the one, filestore instance and share name volumeHandle: "modeInstance/us-central1-c/test-preprov-fs/vol1" - fsType: "nfs" volumeAttributes: - ip: 10.175.240.186 # MODIFY THIS TO Pre-provisioned FS INSTANCE IP - volume: vol1 - + ip: # Modify this to Pre-provisioned Filestore instance IP + volume: vol1 # Modify this to Pre-provisioned Filestore instance share name diff --git a/examples/kubernetes/pre-provision/preprov-pvc.yaml b/examples/kubernetes/pre-provision/preprov-pvc.yaml deleted file mode 100644 index b898216aa..000000000 --- a/examples/kubernetes/pre-provision/preprov-pvc.yaml +++ /dev/null @@ -1,11 +0,0 @@ -kind: PersistentVolumeClaim -apiVersion: v1 -metadata: - name: preprov-pvc -spec: - accessModes: - - ReadWriteMany - storageClassName: csi-filestore - resources: - requests: - storage: 1Ti diff --git a/examples/kubernetes/topology/immediate-binding/demo-deployment-immediate-allowedtopo.yaml b/examples/kubernetes/topology/immediate-binding/demo-deployment-immediate-allowedtopo.yaml index 200297418..83d7d9bd7 100644 --- a/examples/kubernetes/topology/immediate-binding/demo-deployment-immediate-allowedtopo.yaml +++ b/examples/kubernetes/topology/immediate-binding/demo-deployment-immediate-allowedtopo.yaml @@ -21,7 +21,7 @@ spec: - mountPath: /usr/share/nginx/html name: mypvc nodeSelector: - topology.gke.io/zone: us-central1-a + topology.gke.io/zone: us-central1-b # For a zonal cluster change the zone where the cluster is deployed. volumes: - name: mypvc persistentVolumeClaim: diff --git a/examples/kubernetes/topology/immediate-binding/sc-immediate-allowedtopo.yaml b/examples/kubernetes/topology/immediate-binding/sc-immediate-allowedtopo.yaml index e1b38def0..4f64aa453 100644 --- a/examples/kubernetes/topology/immediate-binding/sc-immediate-allowedtopo.yaml +++ b/examples/kubernetes/topology/immediate-binding/sc-immediate-allowedtopo.yaml @@ -10,3 +10,4 @@ allowedTopologies: - key: topology.gke.io/zone values: - us-central1-a + - us-central1-b