Skip to content

Commit

Permalink
[docs] CPLB virtualServers and keepalived
Browse files Browse the repository at this point in the history 
Signed-off-by: Juan-Luis de Sousa-Valadas Castaño <[email protected]>
  • Loading branch information
@juanluisvaladas
juanluisvaladas committed May 2, 2024
1 parent 5b73e3c commit 6c6c346
Show file tree
Hide file tree
Showing 2 changed files with 78 additions and 26 deletions.
32 changes: 27 additions & 5 deletions docs/configuration.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -304,14 +304,24 @@ node-local load balancing.
Configuration options related to k0s's [control plane load balancing] feature

| Element | Description |
| --------------- | ----------------------------------------------------------------------------------------------------------- |
| `enabled` | Indicates if control plane load balancing should be enabled. Default: `false`. |
| `vrrpInstances` | Configuration options related to the VRRP. This is an array which allows to configure multiple virtual IPs. |
| Element | Description |
| ------------ | ------------------------------------------------------------------------------------------- |
| `enabled` | Indicates if control plane load balancing should be enabled. Default: `false`. |
| `type` | Indicates the backend for CPLB. If this isn't defined to `Keepalived`, CPLB will not start. |
| `keepalived` | Contains the keepalived configuration. |
[control plane load balancing]: cplb.md
##### `spec.network.controlPlaneLoadBalancing.VRRPInstances`
##### `spec.network.controlPlaneLoadBalancing.Keepalived`
Configuration options related to keepalived in [control plane load balancing]
| Element | Description |
| ---------------- | ----------------------------------------------------------------------------------------------------------- |
| `vrrpInstances` | Configuration options related to the VRRP. This is an array which allows to configure multiple virtual IPs. |
| `virtualServers` | Configuration options related LoadBalancing. This is an array which allows to configure multiple LBs. |
##### `spec.network.controlPlaneLoadBalancing.keepalived.vrrpInstances`
Configuration options required for using VRRP to configure VIPs in control plane load balancing.
Expand All @@ -324,6 +334,18 @@ Configuration options required for using VRRP to configure VIPs in control plane
| `advertInterval` | Advertisement interval in seconds. Default: `1`. |
| `authPass` | The password used for accessing vrrpd. This field is mandatory and must be under 8 characters long |
##### `spec.network.controlPlaneLoadBalancing.keepalived.virtualServers`
Configuration options required for using VRRP to configure VIPs in control plane load balancing.
| Element | Description |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `ipAddress` | The load balancer's listen address. |
| `delayLoop` | Delay timer for health check polling in seconds. Default: `0`. |
| `lbAlgo` | Algorithm used by keepalived. Supported algorithms: `rr`, `wrr`, `lc`, `wlc`, `lblc`, `dh`, `sh`, `sed`, `nq`. Default: `rr`. |
| `lbKind` | Kind of ipvs load balancer. Supported values: `NAT`, `DR`, `TUN` Default: `DR`. |
| `persistenceTimeout` | Timeout value for persistent connections in seconds. Default: `360` (6 minutes). |

### `spec.controllerManager`

| Element | Description |
Expand Down
72 changes: 51 additions & 21 deletions docs/cplb.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,23 @@
For clusters that don't have an [externally managed load balancer](high-availability.md#load-balancer) for the k0s
control plane, there is another option to get a highly available control plane called control plane load balancing (CPLB).

CPLB allows automatic assigned of predefined IP addresses using VRRP across masters.
CPLB has two features that often will be combined, but normally will be used together: VRRP Instances, which allows
automatic assignation of predefined IP addresses using VRRP across control plane nodes. VirtualServers allows to
do Load Balancing to the other control plane nodes.

This feature is intended to be used for external traffic. This feature is fully compatible with
[node-local load balancing (NLLB)](nllb.md) which means CPLB can be used for external traffic and NLLB for
internal traffic at the same time.

## Technical functionality

The k0s control plane load balancer provides k0s with virtual IPs on each
controller node. This allows the control plane to be highly available using
VRRP (Virtual Router Redundancy Protocol) as long as the network
infrastructure allows multicast and GARP.
The k0s control plane load balancer provides k0s with virtual IPs and TCP
load Balancing on each controller node. This allows the control plane to
be highly available using VRRP (Virtual Router Redundancy Protocol) and
IPVS long as the network infrastructure allows multicast and GARP.

[Keepalived](https://www.keepalived.org/) is the only load balancer that is
supported so far and currently there are no plans to support other alternatives.
supported so far. Currently there are no plans to support other alternatives.

## VRRP Instances

Expand Down Expand Up @@ -46,19 +52,23 @@ following:
These do not provide any sort of security against ill-intentioned attacks, they are
safety features to prevent accidental conflicts between VRRP instances in the same
network segment.
* If `VirtualServers` are used, the cluster configuration doesn't specify a non-empty
[`spec.api.externalAddress`][specapi]. `VRRPInstances` are compatible.

Add the following to the cluster configuration (`k0s.yaml`):

```yaml
spec:
api:
externalAddress: <External address> # This isn't a requirement, but it's a common use case.
network:
controlPlaneLoadBalancing:
enabled: true
vrrpInstances:
- virtualIPs: ["<External address IP>/<external address IP netmask"]
authPass: <password>
type: Keepalived
keepalived:
vrrpInstances:
- virtualIPs: ["<External address IP>/<external address IP netmask"]
authPass: <password>
virtualServers:
- ipAddress: "ipAddress"
```
Or alternatively, if using [`k0sctl`](k0sctl-install.md), add the following to
Expand All @@ -69,24 +79,30 @@ spec:
k0s:
config:
spec:
api:
externalAddress: <External address> # This isn't a requirement, but it's a common use case.
network:
controlPlaneLoadBalancing:
enabled: true
vrrpInstances:
- virtualIPs: ["<External address IP>/<external address IP netmask>"]
authPass: <password>
type: Keepalived
keepalived:
vrrpInstances:
- virtualIPs: ["<External address IP>/<external address IP netmask>"]
authPass: <password>
virtualServers:
- ipAddress: "<External ip address>"
```

Because this is a feature intended to configure the apiserver, CPLB noes not
support dynamic configuration and in order to make changes you need to restart
the k0s controllers to make changes.

[specapi]: configuration.md#specapi

## Full example using `k0sctl`

The following example shows a full `k0sctl` configuration file featuring three
controllers and three workers with control plane load balancing enabled:
controllers and three workers with control plane load balancing enabled.
Additionally it defines [spec.api.sans](configuration.md#specapi) so that the
kube-apiserver certificate is valid for the virtual IP:

```yaml
apiVersion: k0sctl.k0sproject.io/v1beta1
Expand Down Expand Up @@ -142,13 +158,18 @@ spec:
config:
spec:
api:
externalAddress: 192.168.122.200
sans:
- 192.168.122.200
network:
controlPlaneLoadBalancing:
enabled: true
vrrpInstances:
- virtualIPs: ["192.168.122.200/24"]
authPass: Example
type: Keepalived:
keepalived:
vrrpInstances:
- virtualIPs: ["192.168.122.200/24"]
authPass: Example
virtualServers:
- ipAddress: "<External ip address>"
```

Save the above configuration into a file called `k0sctl.yaml` and apply it in
Expand Down Expand Up @@ -319,6 +340,15 @@ controller-1
controller-2
2: eth0 inet 192.168.122.87/24 brd 192.168.122.255 scope global dynamic noprefixroute eth0\ valid_lft 2182sec preferred_lft 2182sec
3: dummyvip0 inet 192.168.122.200/32 scope global dummyvip0\ valid_lft forever preferred_lft forever
$ for i in controller-{0..2} ; do echo $i ; ipvsadm --save -n; done
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.122.200:6443 rr persistent 360
-> 192.168.122.185:6443 Route 1 0 0
-> 192.168.122.87:6443 Route 1 0 0
-> 192.168.122.122:6443 Route 1 0 0
````
And the cluster will be working normally:
Expand Down

0 comments on commit 6c6c346

Please sign in to comment.