kube-router failed to start when installation with default settings

[bizhao]

Before creating an issue, make sure you've checked the following:

• You are running the latest released version of k0s
• Make sure you've searched for existing issues, both open and closed
• Make sure you've searched for PRs too, a fix might've been merged already
• You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

Linux 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Version

v1.30.0+k0s.0

Sysinfo

`k0s sysinfo`

Total memory: 15.6 GiB (pass)
Disk space available for /var/lib/k0s: 74.7 GiB (pass)
Name resolution: localhost: [127.0.0.1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 5.15.0-106-generic (pass)
  Max. file descriptors per process: current: 1048576 / max: 1048576 (pass)
  AppArmor: active (pass)
  Executable in PATH: modprobe: /usr/sbin/modprobe (pass)
  Executable in PATH: mount: /usr/bin/mount (pass)
  Executable in PATH: umount: /usr/bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": available (device filters attachable) (pass)
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": available (is a listed root controller) (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass)
    CONFIG_CPUSETS: Cpuset support: built-in (pass)
    CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass)
    CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
      CONFIG_NETFILTER_NETLINK: module (pass)
      CONFIG_NF_NAT: module (pass)
      CONFIG_IP_SET: IP set support: module (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
      CONFIG_IP_VS: IP virtual server support: module (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
        CONFIG_IP_VS_SH: Source hashing scheduling: module (pass)
        CONFIG_IP_VS_RR: Round-robin scheduling: module (pass)
        CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: module (pass)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: module (pass)
        CONFIG_IP_NF_MANGLE: Packet mangling: module (pass)
      CONFIG_NF_DEFRAG_IPV4: module (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: module (pass)
        CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass)
      CONFIG_NF_DEFRAG_IPV6: module (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass)
      CONFIG_LLC: module (pass)
      CONFIG_STP: module (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: built-in (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)

What happened?

Install a single node cluster following the guide.
bizhao@ubuntu22:$ curl -sSLf https://get.k0s.sh | sudo sh
Downloading k0s from URL: https://github.com/k0sproject/k0s/releases/download/v1.30.0+k0s.0/k0s-v1.30.0+k0s.0-amd64
k0s is now executable in /usr/local/bin
bizhao@ubuntu22:$ sudo k0s install controller --single
bizhao@ubuntu22:$ sudo k0s start
bizhao@ubuntu22:$ sudo k0s status
Version: v1.30.0+k0s.0
Process ID: 1651
Role: controller
Workloads: true
SingleNode: true
Kube-api probing successful: true
Kube-api probing last error:
bizhao@ubuntu22:$ sudo k0s kubectl get nodes
NAME STATUS ROLES AGE VERSION
ubuntu22 Ready control-plane 47s v1.30.0+k0s
bizhao@ubuntu22:$ sudo k0s kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kube-proxy-pb696 1/1 Running 0 52s
kube-system metrics-server-5cd4986bbc-mf9f4 0/1 ContainerCreating 0 53s
kube-system coredns-85c69f454c-tcnc6 0/1 ContainerCreating 0 53s
kube-system kube-router-wnmtw 0/1 CrashLoopBackOff 2 (22s ago) 52s

sudo k0s kubectl logs kube-router-wnmtw -n kube-system
Defaulted container "kube-router" out of: kube-router, install-cni-bins (init), install-cniconf (init)
I0513 06:28:12.439110 2898 version.go:66] Running /usr/local/bin/kube-router version v2.1.0, built on 2024-03-02T15:45:54-0600, go1.21.9
failed to run kube-router: failed to listen on :8080 for metrics: unable to open :8080: listen tcp :8080: bind: address already in use

bizhao@ubuntu22:~$ sudo lsof -i -P -n | grep 8080
kine 1660 kube-apiserver 10u IPv4 25963 0t0 TCP *:8080 (LISTEN)

Steps to reproduce

1. curl -sSLf https://get.k0s.sh | sudo sh
2. sudo k0s install controller --single
3. sudo k0s kubectl get pods -A

Expected behavior

All pods should start successfully

Actual behavior

kube-router cannot be started due to port conflicts.

Screenshots and logs

No response

Additional context

I changed the metricsPort from 8080 to 8090 and re-install k0s.
All pods are successfully started.

Looks like both kine and kube-router is trying to use 8080.

[ncopa]

I ran into this as well. Seems like our integration test is broken and did not catch this.

[twz123]

I ran into this as well. Seems like our integration test is broken and did not catch this.

#4420 fixes the kube-router DaemonSet so that it doesn't report readiness too quickly. The check-singlenode test is now reliably failing with that tweak. Needs to be rebased after #4421 has been merged.

[twz123]

The actual fix in #4421 will not be backported beyond 1.30. Versions prior to 1.30 will be operational, even with the metrics port clash, as kube-router 1.x will simply log an error about not being able to bind the metrics port, and continue to work. Kube-router metrics can't be retrieved in those clusters unless kube-routers metrics port gets changed to something else in the k0s config.

The test changes from #4419 will be backported, though. They will prove that k0s will be operational with the default --single configuration, and will catch regressions early.