From 7c5ef552fdd105cf26d6997b421718d8d17b8f6e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Juan-Luis=20de=20Sousa-Valadas=20Casta=C3=B1o?= Date: Tue, 30 Apr 2024 15:17:49 +0200 Subject: [PATCH] [docs] CPLB virtualServers and keepalived MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Juan-Luis de Sousa-Valadas Castaño --- docs/configuration.md | 32 ++++++++++++++++---- docs/cplb.md | 70 ++++++++++++++++++++++++++++++------------- 2 files changed, 76 insertions(+), 26 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index c694750d79dd..8b821238c2ff 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -304,14 +304,24 @@ node-local load balancing. Configuration options related to k0s's [control plane load balancing] feature -| Element | Description | -| --------------- | ----------------------------------------------------------------------------------------------------------- | -| `enabled` | Indicates if control plane load balancing should be enabled. Default: `false`. | -| `vrrpInstances` | Configuration options related to the VRRP. This is an array which allows to configure multiple virtual IPs. | +| Element | Description | +| ------------ | ------------------------------------------------------------------------------------------- | +| `enabled` | Indicates if control plane load balancing should be enabled. Default: `false`. | +| `type` | Indicates the backend for CPLB. If this isn't defined to `Keepalived`, CPLB will not start. | +| `keepalived` | Contains the keepalived configuration. | [control plane load balancing]: cplb.md -##### `spec.network.controlPlaneLoadBalancing.VRRPInstances` +##### `spec.network.controlPlaneLoadBalancing.Keepalived` + +Configuration options related to keepalived in [control plane load balancing] + +| Element | Description | +| ---------------- | ----------------------------------------------------------------------------------------------------------- | +| `vrrpInstances` | Configuration options related to the VRRP. This is an array which allows to configure multiple virtual IPs. | +| `virtualServers` | Configuration options related LoadBalancing. This is an array which allows to configure multiple LBs. | + +##### `spec.network.controlPlaneLoadBalancing.keepalived.vrrpInstances` Configuration options required for using VRRP to configure VIPs in control plane load balancing. @@ -324,6 +334,18 @@ Configuration options required for using VRRP to configure VIPs in control plane | `advertInterval` | Advertisement interval in seconds. Default: `1`. | | `authPass` | The password used for accessing vrrpd. This field is mandatory and must be under 8 characters long | +##### `spec.network.controlPlaneLoadBalancing.keepalived.virtualServers` + +Configuration options required for using VRRP to configure VIPs in control plane load balancing. + +| Element | Description | +| ----------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| `ipAddress` | The load balancer's listen address. | +| `delayLoop` | Delay timer for health check polling in seconds. Default: `0`. | +| `lbAlgo` | Algorithm used by keepalived. Supported algorithms: `rr`, `wrr`, `lc`, `wlc`, `lblc`, `dh`, `sh`, `sed`, `nq`. Default: `rr`. | +| `lbKind` | Kind of ipvs load balancer. Supported values: `NAT`, `DR`, `TUN` Default: `DR`. | +| `persistenceTimeout` | Timeout value for persistent connections in seconds. Default: `360` (6 minutes). | + ### `spec.controllerManager` | Element | Description | diff --git a/docs/cplb.md b/docs/cplb.md index fc8614eea230..ab836187b182 100644 --- a/docs/cplb.md +++ b/docs/cplb.md @@ -3,17 +3,23 @@ For clusters that don't have an [externally managed load balancer](high-availability.md#load-balancer) for the k0s control plane, there is another option to get a highly available control plane called control plane load balancing (CPLB). -CPLB allows automatic assigned of predefined IP addresses using VRRP across masters. +CPLB has two features that often will be combined, but normally will be used together: VRRP Instances, which allows +automatic assignation of predefined IP addresses using VRRP across control plane nodes. VirtualServers allows to +do Load Balancing to the other control plane nodes. + +This feature is intended to be used for external traffic. This feature is fully compatible with +[node-local load balancing (NLLB)](nllb.md) which means CPLB can be used for external traffic and NLLB for +internal traffic at the same time. ## Technical functionality -The k0s control plane load balancer provides k0s with virtual IPs on each -controller node. This allows the control plane to be highly available using -VRRP (Virtual Router Redundancy Protocol) as long as the network -infrastructure allows multicast and GARP. +The k0s control plane load balancer provides k0s with virtual IPs and TCP +load Balancing on each controller node. This allows the control plane to +be highly available using VRRP (Virtual Router Redundancy Protocol) and +IPVS long as the network infrastructure allows multicast and GARP. [Keepalived](https://www.keepalived.org/) is the only load balancer that is -supported so far and currently there are no plans to support other alternatives. +supported so far. Currently there are no plans to support other alternatives. ## VRRP Instances @@ -46,19 +52,23 @@ following: These do not provide any sort of security against ill-intentioned attacks, they are safety features to prevent accidental conflicts between VRRP instances in the same network segment. +* If `VirtualServers` are used, the cluster configuration doesn't specify a non-empty + [`spec.api.externalAddress`][specapi]. `VRRPInstances` are compatible. Add the following to the cluster configuration (`k0s.yaml`): ```yaml spec: - api: - externalAddress: # This isn't a requirement, but it's a common use case. network: controlPlaneLoadBalancing: enabled: true - vrrpInstances: - - virtualIPs: ["/ + type: Keepalived + keepalived: + vrrpInstances: + - virtualIPs: ["/ + virtualServers: + - ipAddress: "ipAddress" ``` Or alternatively, if using [`k0sctl`](k0sctl-install.md), add the following to @@ -69,14 +79,16 @@ spec: k0s: config: spec: - api: - externalAddress: # This isn't a requirement, but it's a common use case. network: controlPlaneLoadBalancing: enabled: true - vrrpInstances: - - virtualIPs: ["/"] - authPass: + type: Keepalived + keepalived: + vrrpInstances: + - virtualIPs: ["/"] + authPass: + virtualServers: + - ipAddress: "" ``` Because this is a feature intended to configure the apiserver, CPLB noes not @@ -86,7 +98,9 @@ the k0s controllers to make changes. ## Full example using `k0sctl` The following example shows a full `k0sctl` configuration file featuring three -controllers and three workers with control plane load balancing enabled: +controllers and three workers with control plane load balancing enabled. +Additionally it defines [spec.api.sans](configuration.md#specapi) so that the +kube-apiserver certificate is valid for the virtual IP: ```yaml apiVersion: k0sctl.k0sproject.io/v1beta1 @@ -142,13 +156,18 @@ spec: config: spec: api: - externalAddress: 192.168.122.200 + sans: + - 192.168.122.200 network: controlPlaneLoadBalancing: enabled: true - vrrpInstances: - - virtualIPs: ["192.168.122.200/24"] - authPass: Example + type: Keepalived: + keepalived: + vrrpInstances: + - virtualIPs: ["192.168.122.200/24"] + authPass: Example + virtualServers: + - ipAddress: "" ``` Save the above configuration into a file called `k0sctl.yaml` and apply it in @@ -319,6 +338,15 @@ controller-1 controller-2 2: eth0 inet 192.168.122.87/24 brd 192.168.122.255 scope global dynamic noprefixroute eth0\ valid_lft 2182sec preferred_lft 2182sec 3: dummyvip0 inet 192.168.122.200/32 scope global dummyvip0\ valid_lft forever preferred_lft forever + +$ for i in controller-{0..2} ; do echo $i ; ipvsadm --save -n; done +IP Virtual Server version 1.2.1 (size=4096) +Prot LocalAddress:Port Scheduler Flags + -> RemoteAddress:Port Forward Weight ActiveConn InActConn +TCP 192.168.122.200:6443 rr persistent 360 + -> 192.168.122.185:6443 Route 1 0 0 + -> 192.168.122.87:6443 Route 1 0 0 + -> 192.168.122.122:6443 Route 1 0 0 ```` And the cluster will be working normally: