[docs] CPLB virtualServers and keepalived

Signed-off-by: Juan-Luis de Sousa-Valadas Castaño <[email protected]>
k0sproject · Apr 30, 2024 · 7c5ef55 · 7c5ef55
1 parent 49dc2bc
commit 7c5ef55
Show file tree

Hide file tree

Showing 2 changed files with 76 additions and 26 deletions.
diff --git a/docs/configuration.md b/docs/configuration.md
@@ -304,14 +304,24 @@ node-local load balancing.
 
 Configuration options related to k0s's [control plane load balancing] feature
 
-| Element         | Description                                                                                                 |
-| --------------- | ----------------------------------------------------------------------------------------------------------- |
-| `enabled`       | Indicates if control plane load balancing should be enabled. Default: `false`.                              |
-| `vrrpInstances` | Configuration options related to the VRRP. This is an array which allows to configure multiple virtual IPs. |
+| Element      | Description                                                                                 |
+| ------------ | ------------------------------------------------------------------------------------------- |
+| `enabled`    | Indicates if control plane load balancing should be enabled. Default: `false`.              |
+| `type`       | Indicates the backend for CPLB. If this isn't defined to `Keepalived`, CPLB will not start. |
+| `keepalived` | Contains the keepalived configuration.                                                      |
 
 [control plane load balancing]: cplb.md
 
-##### `spec.network.controlPlaneLoadBalancing.VRRPInstances`
+##### `spec.network.controlPlaneLoadBalancing.Keepalived`
+
+Configuration options related to keepalived in [control plane load balancing]
+
+| Element          | Description                                                                                                 |
+| ---------------- | ----------------------------------------------------------------------------------------------------------- |
+| `vrrpInstances`  | Configuration options related to the VRRP. This is an array which allows to configure multiple virtual IPs. |
+| `virtualServers` | Configuration options related LoadBalancing. This is an array which allows to configure multiple LBs.       |
+
+##### `spec.network.controlPlaneLoadBalancing.keepalived.vrrpInstances`
 
 Configuration options required for using VRRP to configure VIPs in control plane load balancing.
 
@@ -324,6 +334,18 @@ Configuration options required for using VRRP to configure VIPs in control plane
 | `advertInterval`  | Advertisement interval in seconds. Default: `1`.                                                                  |
 | `authPass`        | The password used for accessing vrrpd. This field is mandatory and must be under 8 characters long                |
 
+##### `spec.network.controlPlaneLoadBalancing.keepalived.virtualServers`
+
+Configuration options required for using VRRP to configure VIPs in control plane load balancing.
+
+| Element           | Description                                                                                                                      |
+| ----------------- | -------------------------------------------------------------------------------------------------------------------------------- |
+| `ipAddress`          | The load balancer's listen address.                                                                                           |
+| `delayLoop`          | Delay timer for health check polling in seconds. Default: `0`.                                                                |
+| `lbAlgo`             | Algorithm used by keepalived. Supported algorithms: `rr`, `wrr`, `lc`, `wlc`, `lblc`, `dh`, `sh`, `sed`, `nq`. Default: `rr`. |
+| `lbKind`             | Kind of ipvs load balancer. Supported values: `NAT`, `DR`, `TUN`  Default: `DR`.                                              |
+| `persistenceTimeout` | Timeout value for persistent connections in seconds. Default: `360` (6 minutes).                                              |
+
 ### `spec.controllerManager`
 
 | Element     | Description                                                                                                             |

diff --git a/docs/cplb.md b/docs/cplb.md
@@ -3,17 +3,23 @@
 For clusters that don't have an [externally managed load balancer](high-availability.md#load-balancer) for the k0s
 control plane, there is another option to get a highly available control plane called control plane load balancing (CPLB).
 
-CPLB allows automatic assigned of predefined IP addresses using VRRP across masters.
+CPLB has two features that often will be combined, but normally will be used together: VRRP Instances, which allows
+automatic assignation of predefined IP addresses using VRRP across control plane nodes. VirtualServers allows to
+do Load Balancing to the other control plane nodes.
+
+This feature is intended to be used for external traffic. This feature is fully compatible with
+[node-local load balancing (NLLB)](nllb.md) which means CPLB can be used for external traffic and NLLB for
+internal traffic at the same time.
 
 ## Technical functionality
 
-The k0s control plane load balancer provides k0s with virtual IPs on each
-controller node. This allows the control plane to be highly available using
-VRRP (Virtual Router Redundancy Protocol) as long as the network
-infrastructure allows multicast and GARP.
+The k0s control plane load balancer provides k0s with virtual IPs and TCP
+load Balancing on each controller node. This allows the control plane to
+be highly available using VRRP (Virtual Router Redundancy Protocol) and
+IPVS long as the network infrastructure allows multicast and GARP.
 
 [Keepalived](https://www.keepalived.org/) is the only load balancer that is
-supported so far and currently there are no plans to support other alternatives.
+supported so far. Currently there are no plans to support other alternatives.
 
 ## VRRP Instances
 
@@ -46,19 +52,23 @@ following:
   These do not provide any sort of security against ill-intentioned attacks, they are
   safety features to prevent accidental conflicts between VRRP instances in the same
   network segment.
+* If `VirtualServers` are used, the cluster configuration doesn't specify a non-empty
+  [`spec.api.externalAddress`][specapi]. `VRRPInstances` are compatible.
 
 Add the following to the cluster configuration (`k0s.yaml`):
 
 ```yaml
 spec:
-  api:
-    externalAddress: <External address> # This isn't a requirement, but it's a common use case.
   network:
     controlPlaneLoadBalancing:
       enabled: true
-      vrrpInstances:
-      - virtualIPs: ["<External address IP>/<external address IP netmask"]
-        authPass: <password>
+      type: Keepalived
+      keepalived:
+        vrrpInstances:
+        - virtualIPs: ["<External address IP>/<external address IP netmask"]
+          authPass: <password>
+        virtualServers:
+        - ipAddress: "ipAddress"
 ```
 
 Or alternatively, if using [`k0sctl`](k0sctl-install.md), add the following to
@@ -69,14 +79,16 @@ spec:
   k0s:
     config:
       spec:
-        api:
-          externalAddress: <External address> # This isn't a requirement, but it's a common use case.
         network:
           controlPlaneLoadBalancing:
             enabled: true
-            vrrpInstances:
-            - virtualIPs: ["<External address IP>/<external address IP netmask>"]
-              authPass: <password>
+            type: Keepalived
+            keepalived:
+              vrrpInstances:
+              - virtualIPs: ["<External address IP>/<external address IP netmask>"]
+                authPass: <password>
+              virtualServers:
+              - ipAddress: "<External ip address>"
 ```
 
 Because this is a feature intended to configure the apiserver, CPLB noes not
@@ -86,7 +98,9 @@ the k0s controllers to make changes.
 ## Full example using `k0sctl`
 
 The following example shows a full `k0sctl` configuration file featuring three
-controllers and three workers with control plane load balancing enabled:
+controllers and three workers with control plane load balancing enabled.
+Additionally it defines [spec.api.sans](configuration.md#specapi) so that the
+kube-apiserver certificate is valid for the virtual IP:
 
 ```yaml
 apiVersion: k0sctl.k0sproject.io/v1beta1
@@ -142,13 +156,18 @@ spec:
     config:
       spec:
         api:
-          externalAddress: 192.168.122.200
+          sans:
+          - 192.168.122.200
         network:
           controlPlaneLoadBalancing:
             enabled: true
-            vrrpInstances:
-            - virtualIPs: ["192.168.122.200/24"]
-              authPass: Example
+            type: Keepalived:
+            keepalived:
+              vrrpInstances:
+              - virtualIPs: ["192.168.122.200/24"]
+                authPass: Example
+              virtualServers:
+              - ipAddress: "<External ip address>"
 ```
 
 Save the above configuration into a file called `k0sctl.yaml` and apply it in
@@ -319,6 +338,15 @@ controller-1
 controller-2
 2: eth0    inet 192.168.122.87/24 brd 192.168.122.255 scope global dynamic noprefixroute eth0\       valid_lft 2182sec preferred_lft 2182sec
 3: dummyvip0    inet 192.168.122.200/32 scope global dummyvip0\       valid_lft forever preferred_lft forever
+
+$ for i in controller-{0..2} ; do echo $i ; ipvsadm --save -n; done
+IP Virtual Server version 1.2.1 (size=4096)
+Prot LocalAddress:Port Scheduler Flags
+  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
+TCP  192.168.122.200:6443 rr persistent 360
+  -> 192.168.122.185:6443              Route   1      0          0
+  -> 192.168.122.87:6443               Route   1      0          0
+  -> 192.168.122.122:6443              Route   1      0          0
 ````
 
 And the cluster will be working normally: