In dual-stack mode --node-ip to kubelet only have one address

[uablrek]

Before creating an issue, make sure you've checked the following:

• You are running the latest released version of k0s
• Make sure you've searched for existing issues, both open and closed
• Make sure you've searched for PRs too, a fix might've been merged already
• You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

vm-001 ~ # uname -srvmo; cat /etc/os-release || lsb_release -a
Linux 6.9.3 k0sproject/k0s#6 SMP PREEMPT_DYNAMIC Fri Jun 14 10:33:26 CEST 2024 x86_64 GNU/Linux
NAME="Xcluster"
VERSION="2024.06.28"
ID=xcluster
PRETTY_NAME="Xcluster"
VERSION_ID="2024.06.28"

Version

vm-001 ~ # k0s version v1.30.2+k0s.0

Sysinfo

`k0s sysinfo`

vm-001 ~ # k0s sysinfo
Total memory: 982.2 MiB (warning: 1.0 GiB recommended)
Disk space available for /var/lib/k0s: 4.8 GiB (pass)
Name resolution: localhost: [127.0.0.1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 6.9.3 (pass)
  Max. file descriptors per process: current: 4096 / max: 4096 (warning: < 65536)
  AppArmor: unavailable (pass)
  Executable in PATH: modprobe: /sbin/modprobe (pass)
  Executable in PATH: mount: /bin/mount (pass)
  Executable in PATH: umount: /bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": available (device filters attachable) (pass)
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": available (is a listed root controller) (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass)
    CONFIG_CPUSETS: Cpuset support: built-in (pass)
    CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass)
    CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: built-in (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: built-in (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: built-in (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: built-in (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: built-in (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: built-in (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
      CONFIG_NETFILTER_NETLINK: built-in (pass)
      CONFIG_NF_NAT: built-in (pass)
      CONFIG_IP_SET: IP set support: module (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
      CONFIG_IP_VS: IP virtual server support: module (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
        CONFIG_IP_VS_SH: Source hashing scheduling: module (pass)
        CONFIG_IP_VS_RR: Round-robin scheduling: module (pass)
        CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: built-in (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: module (pass)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: module (pass)
        CONFIG_IP_NF_MANGLE: Packet mangling: module (pass)
      CONFIG_NF_DEFRAG_IPV4: built-in (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: built-in (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: built-in (pass)
        CONFIG_IP6_NF_MANGLE: Packet mangling: built-in (pass)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass)
      CONFIG_NF_DEFRAG_IPV6: built-in (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: built-in (pass)
      CONFIG_LLC: built-in (pass)
      CONFIG_STP: built-in (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: built-in (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)

What happened?

In dual-stack mode only the ipv4 address is specified in the "--node-ip" parameter to kubelet. This causes the node object to have only an ipv4 address.

Steps to reproduce

1. Start k0s in dual-stack mode
2. On a worker node do: ps ... | grep kubelet and check the "--node-ip" parameter
3. Or, do k0s kubectl get node $(hostname) -o json | jq .status.addresses. Only ipv4 is present

Expected behavior

Both ipv4 and ipv6 addresses should be specified in "--node-ip", example:

--node-ip=192.168.1.2,fd00::c0a8:102

Then, when the worker joins, the node object will have both addresses:

# kubectl get node $(hostname) -o json | jq .status.addresses
[
  {
    "address": "192.168.1.2",
    "type": "InternalIP"
  },
  {
    "address": "fd00::c0a8:102",
    "type": "InternalIP"
  },
  {
    "address": "vm-002",
    "type": "Hostname"
  }
]

Actual behavior

Only ipv4 is specified in "--node-ip", and the node object only gets an ipv4 address:

vm-002 ~ # k0s kubectl get node $(hostname) -o json | jq .status.addresses
[
  {
    "address": "192.168.1.2",
    "type": "InternalIP"
  },
  {
    "address": "vm-002",
    "type": "Hostname"
  }
]

Screenshots and logs

No response

Additional context

I install with k0sctl.

k0scat.yaml

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
spec:
  k0s:
    version: $__k0sver
    config:
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: Cluster
      metadata:
        name: k0s
      spec:
        api:
          k0sApiPort: 9443
          port: 6443
        installConfig:
          users:
            etcdUser: etcd
            kineUser: kube-apiserver
            konnectivityUser: konnectivity-server
            kubeAPIserverUser: kube-apiserver
            kubeSchedulerUser: kube-scheduler
        konnectivity:
          adminPort: 8133
          agentPort: 8132
        network:
          kubeProxy:
            disabled: false
            mode: iptables
          provider: calico
          calico:
            mode: "bird"
            envVars:
              IP_AUTODETECTION_METHOD: "interface=eth1"
              IP6_AUTODETECTION_METHOD: "interface=eth1"
          kuberouter:
            autoMTU: true
            mtu: 0
            peerRouterASNs: ""
            peerRouterIPs: ""
          podCIDR: 10.244.0.0/16
          serviceCIDR: 10.96.0.0/16
          dualStack:
            enabled: true
            IPv6podCIDR: "$PREFIX:0af4:0:0/108"
            IPv6serviceCIDR: "$PREFIX:0a60:0:0/108"
        podSecurityPolicy:
          defaultPolicy: 00-k0s-privileged
        storage:
          type: etcd
        telemetry:
          enabled: false
  hosts:
  - ssh:
      address: 192.168.1.1
      user: root
      port: 22
      keyPath: /root/.ssh/id_dropbear_ssh
    role: controller+worker
    installFlags:
    - --disable-components=konnectivity-server
    noTaints: true
    os: alpine
    privateInterface: eth1
    privateAddress: 192.168.1.1
    k0sDownloadURL: file:///root/www/k0s-$__k0sver-amd64
  - ssh:
      address: 192.168.1.2
      user: root
      port: 22
      keyPath: /root/.ssh/id_dropbear_ssh
    role: worker
    os: alpine
    privateInterface: eth1
    privateAddress: 192.168.1.2
    k0sDownloadURL: file:///root/www/k0s-$__k0sver-amd64
  - ssh:
      address: 192.168.1.3
      user: root
      port: 22
      keyPath: /root/.ssh/id_dropbear_ssh
    role: worker
    os: alpine
    privateInterface: eth1
    privateAddress: 192.168.1.3
    k0sDownloadURL: file:///root/www/k0s-$__k0sver-amd64
  - ssh:
      address: 192.168.1.4
      user: root
      port: 22
      keyPath: /root/.ssh/id_dropbear_ssh
    role: worker
    os: alpine
    privateInterface: eth1
    privateAddress: 192.168.1.4
    k0sDownloadURL: file:///root/www/k0s-$__k0sver-amd64

[twz123]

I don't think k0s passes any --node-ip flags to kubelet at all. According to the CLI flag's description, it defaults to IPv4. That's what you're seeing in your cluster, I suppose.

  --node-ip string                                           IP address (or comma-separated dual-stack IP addresses) of the node. If unset, kubelet will use the node's default IPv4 address, if any, or its default IPv6 address if it has no IPv4 addresses. You can pass '::' to make it prefer the default IPv6 address rather than the default IPv4 address.

You could pass your own values via --kubelet-extra-args=--node-ip=... in spec.hosts[*].installFlags for the time being.

/cc @ncopa WDYT about this?

[uablrek]

I don't think k0s passes any --node-ip flags to kubelet at all.

@twz123 You are right. I specified privateAddress:. If I remove that, no "--node-ip" is passed to kubelet.

vm-002 ~ # ps www | grep kubelet
  464 root     1267m S    /var/lib/k0s/bin/kubelet --root-dir=/var/lib/k0s/kubelet --config=/var/lib/k0s/kubelet-config.yaml --kubeconfig=/var/lib/k0s/kubelet.conf --v=1 --cert-dir=/var/lib/k0s/kubelet/pki --containerd=/run/k0s/containerd.sock --runtime-cgroups=/system.slice/containerd.service

But for dual-stack it seems that you must specify "comma-separated dual-stack IP addresses" to get it right:

vm-002 ~ # k0s kubectl get node $(hostname) -o json | jq .status.addresses
[
  {
    "address": "192.168.1.2",
    "type": "InternalIP"
  },
  {
    "address": "vm-002",
    "type": "Hostname"
  }
]

(ipv6 is missing)

[twz123]

If it's connected to privateAddress, then it's probably something in k0sctl, not in k0s itself ... Indeed, it is: https://github.com/k0sproject/k0sctl/blob/v0.18.1/pkg/apis/k0sctl.k0sproject.io/v1beta1/cluster/host.go#L320-L327

Not sure what to do here, since the IPv6 is not part of the k0sctl config at all. /cc @kke How would k0s or k0sctl know which IP to use if it isn't specified? Sure, there could be some magic auto-detection, but less magic is usually preferable. Would it make sense to make privateAddress a list and accept multiple IPs, or is that a bad idea? 🤔

[uablrek]

You could pass your own values via --kubelet-extra-args=--node-ip=... in spec.hosts[*].installFlags for the time being.

Yes, that works 😃

vm-002 ~ # k0s kubectl get node $(hostname) -o json | jq .status.addresses
[
  {
    "address": "192.168.1.2",
    "type": "InternalIP"
  },
  {
    "address": "fd00::c0a8:102",
    "type": "InternalIP"
  },
  {
    "address": "vm-002",
    "type": "Hostname"
  }
]

I removed privateAddress: and instead specified:

    installFlags:
    - "--kubelet-extra-args=--node-ip=192.168.1.2,fd00::192.168.1.2"

If privateAddress: is only used for passing an address to kubelet via --node-ip, it should take comma-separated dual-stack IP addresses, same as kubelet.

For a better user experience, k0sctl may specify the first IPv4 and the first global IPv6 for "privateInterface:" as --node-ip if dual-stack is enabled.

This problem should be documented in any case.

I am happy with this work-around, and you may close this issue if you like, or keep it for reference.

[ncopa]

see also k0sproject/k0s#3954