Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Manual upgrade failing because of helm timeout type #4556

Closed
3 of 4 tasks
coutug opened this issue Jun 5, 2024 · 3 comments
Closed
3 of 4 tasks

Manual upgrade failing because of helm timeout type #4556

coutug opened this issue Jun 5, 2024 · 3 comments
Labels
bug Something isn't working

Comments

@coutug
Copy link

coutug commented Jun 5, 2024 •
edited
Loading

Before creating an issue, make sure you've checked the following:

  • You are running the latest released version of k0s
  • Make sure you've searched for existing issues, both open and closed
  • Make sure you've searched for PRs too, a fix might've been merged already
  • You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

Linux 6.1.0-20-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.85-1 (2024-04-11) x86_64 GNU/Linux
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Version

v1.29.5+k0s.0

Sysinfo

`k0s sysinfo` 
Machine ID: "ba9e6982836321b38e631faf28531fce18b41a13c8010cddc8be9802f057360b" (from machine) (pass)
Total memory: 251.8 GiB (pass)
Disk space available for /var/lib/k0s: 398.2 GiB (pass)
Name resolution: localhost: [127.0.0.1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 6.1.0-20-amd64 (pass)
  Max. file descriptors per process: current: 1048576 / max: 1048576 (pass)
  AppArmor: active (pass)
  Executable in PATH: modprobe: /usr/sbin/modprobe (pass)
  Executable in PATH: mount: /usr/bin/mount (pass)
  Executable in PATH: umount: /usr/bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": available (device filters attachable) (pass)
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": available (is a listed root controller) (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass)
    CONFIG_CPUSETS: Cpuset support: built-in (pass)
    CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass)
    CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
      CONFIG_NETFILTER_NETLINK: module (pass)
      CONFIG_NF_NAT: module (pass)
      CONFIG_IP_SET: IP set support: module (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
      CONFIG_IP_VS: IP virtual server support: module (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
        CONFIG_IP_VS_SH: Source hashing scheduling: module (pass)
        CONFIG_IP_VS_RR: Round-robin scheduling: module (pass)
        CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: module (pass)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: module (pass)
        CONFIG_IP_NF_MANGLE: Packet mangling: module (pass)
      CONFIG_NF_DEFRAG_IPV4: module (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: module (pass)
        CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass)
      CONFIG_NF_DEFRAG_IPV6: module (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass)
      CONFIG_LLC: module (pass)
      CONFIG_STP: module (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: module (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)

What happened?

I tried to upgrade locally from v1.29.5+k0s.0 to v1.30.1+k0s.0 but it kept failing. I previously upgraded from v1.28.4+k0s.0 to v1.29.5+k0s.0 the same way flawlessly. I'm almost certain my problem is related to #4172, since creating a new cluster using v1.30.1+k0s.0 works. I would like to know how to make the upgrade works for my currently working cluster.
When trying to change the value for the timeout in the ClusterConfig to 10m, the change is refused since the value type accepted are integer and not string. However, the proper type to use for latest version is string instead of integer.

# * spec.extensions.helm.charts[0].timeout: Invalid value: "string": spec.extensions.helm.charts[0].timeout in body must be of type integer: "string"

Steps to reproduce

  1. Install k0s on with HA configuration
  2. Upgrade manually as describe here: https://docs.k0sproject.io/v1.30.1+k0s.0/upgrade/#upgrade-a-k0s-node-locally

Expected behavior

A working upgraded node.

Actual behavior

The k0s systemd service fails to run.

Screenshots and logs

k0sctl config 
apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
spec:
  hosts:
  - ssh:
      address: 172.22.126.51
      user: k0s
      port: 22
      keyPath: ~/.ssh/k0s_rsa
    role: controller
    installFlags:
      - --debug
  - ssh:
      address: 172.22.126.52
      user: k0s
      port: 22
      keyPath: ~/.ssh/k0s_rsa
    role: controller
    installFlags:
      - --debug
  - ssh:
      address: 172.22.126.53
      user: k0s
      port: 22
      keyPath: ~/.ssh/k0s_rsa
    role: controller
    installFlags:
      - --debug
  - ssh:
      address: 172.22.1.84
      user: k0s
      port: 22
      keyPath: ~/.ssh/k0s_rsa
    role: worker
    installFlags:
    - --labels="node.k0sproject.io/role=worker"
  - ssh:
      address: 172.22.1.85
      user: k0s
      port: 22
      keyPath: ~/.ssh/k0s_rsa
    role: worker
    installFlags:
    - --labels="node.k0sproject.io/role=worker"
  - ssh:
      address: 172.22.1.86
      user: k0s
      port: 22
      keyPath: ~/.ssh/k0s_rsa
    role: worker
    installFlags:
    - --labels="node.k0sproject.io/role=worker"
  k0s:
    version: v1.29.5+k0s.0
    dynamicConfig: true
    config:
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: Cluster
      metadata:
        name: k0s
      spec:
        api:
          externalAddress: 172.22.126.11
          sans:
            - 172.22.126.11
            - 172.22.126.51
            - 172.22.126.52
            - 172.22.126.53
          extraArgs:
            service-node-port-range: 30000-32767
        telemetry:
          enabled: true
        network:
          podCIDR: 10.96.0.0/16
          serviceCIDR: 10.10.0.0/16
          dualStack:
            enabled: true
            IPv6podCIDR: "fd00::/108"
            IPv6serviceCIDR: "fd01::/108"
          provider: custom
          kubeProxy:
            disabled: true
        extensions:
          helm:
            concurrencyLevel: 5
            repositories:
            - name: cilium
              url: https://helm.cilium.io
            charts:
            - name: cilium
              chartname: cilium/cilium
              version: "1.14.7"
              values: |
                hubble:
                  enabled: true
                  listenAddress: ":4244"
                  metrics:
                    enabled:
                      - dns
                      - drop
                      - tcp
                      - flow
                      - port-distribution
                      - icmp
                      - http
                  relay:
                    enabled: true
                  ui:
                    enabled: true
                  ipam:
                    mode: "cluster-pool"
                    operator:
                      clusterPoolIPv4PodCIDRList: ["10.96.0.0/16"]
                      clusterPoolIPv6PodCIDRList: ["fd00::/108"]
                ipv6:
                  enabled: true
                envoy:
                  enabled: true
                kubeProxyReplacement: "true"
                k8sServiceHost: 172.22.126.11
                k8sServicePort: 6443
                rollOutCiliumPods: true
              namespace: kube-system
ClusterConfig 
apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  creationTimestamp: "2024-03-08T20:29:55Z"
  generation: 1
  name: k0s
  namespace: kube-system
  resourceVersion: "197"
  uid: 49a3d446-dd3c-40f2-8729-bc7b0295310b
spec:
  extensions:
    helm:
      charts:
      - chartname: cilium/cilium
        name: cilium
        namespace: kube-system
        order: 0
        timeout: 0
        values: |
          hubble:
            enabled: true
            listenAddress: ":4244"
            metrics:
              enabled:
                - dns
                - drop
                - tcp
                - flow
                - port-distribution
                - icmp
                - http
            relay:
              enabled: true
            ui:
              enabled: true
            ipam:
              mode: "cluster-pool"
              operator:
                clusterPoolIPv4PodCIDRList: ["10.96.0.0/16"]
                clusterPoolIPv6PodCIDRList: ["fd00::/108"]
          ipv6:
            enabled: true
          envoy:
            enabled: true
          kubeProxyReplacement: "true"
          k8sServiceHost: 172.22.126.12
          k8sServicePort: 6443
          rollOutCiliumPods: true
        version: 1.14.7
      concurrencyLevel: 5
      repositories:
      - caFile: ""
        certFile: ""
        insecure: false
        keyfile: ""
        name: cilium
        password: ""
        url: https://helm.cilium.io
        username: ""
    storage:
      create_default_storage_class: false
      type: external_storage
  network:
    dualStack:
      IPv6podCIDR: fd00::/108
      IPv6serviceCIDR: fd01::/108
      enabled: true
    kubeProxy:
      disabled: true
      iptables:
        minSyncPeriod: 0s
        syncPeriod: 0s
      ipvs:
        minSyncPeriod: 0s
        syncPeriod: 0s
        tcpFinTimeout: 0s
        tcpTimeout: 0s
        udpTimeout: 0s
      metricsBindAddress: 0.0.0.0:10249
      mode: iptables
    kuberouter:
      autoMTU: true
      hairpin: Enabled
      ipMasq: false
      metricsPort: 8080
      mtu: 0
      peerRouterASNs: ""
      peerRouterIPs: ""
    nodeLocalLoadBalancing:
      envoyProxy:
        apiServerBindPort: 7443
        image:
          image: quay.io/k0sproject/envoy-distroless
          version: v1.24.12
        konnectivityServerBindPort: 7132
      type: EnvoyProxy
    podCIDR: 10.96.0.0/16
    provider: custom
Journalctl logs 
Jun 05 21:14:39 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:39" level=info msg="I0605 21:14:39.019336  233273 handler.go:286] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager" component=kube-apiserver stream=stderr
Jun 05 21:14:39 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:39" level=info msg="E0605 21:14:39.023352  233273 available_controller.go:460] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io \"v1beta1.metrics.k8s.io\": the object has been modified; please apply your changes to the latest version and try again" component=kube-apiserver stream=stderr
Jun 05 21:14:39 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:39" level=debug msg="Failed to ensure the existence of the cluster configuration" component=clusterConfig-reconciler error="json: cannot unmarshal number into Go struct field Chart.spec.extensions.helm.charts.timeout of type string"
Jun 05 21:14:40 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:40" level=debug msg="Failed to ensure the existence of the cluster configuration" component=clusterConfig-reconciler error="json: cannot unmarshal number into Go struct field Chart.spec.extensions.helm.charts.timeout of type string"
Jun 05 21:14:41 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:41" level=debug msg="Failed to ensure the existence of the cluster configuration" component=clusterConfig-reconciler error="json: cannot unmarshal number into Go struct field Chart.spec.extensions.helm.charts.timeout of type string"
Jun 05 21:14:42 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:42" level=debug msg="Failed to ensure the existence of the cluster configuration" component=clusterConfig-reconciler error="json: cannot unmarshal number into Go struct field Chart.spec.extensions.helm.charts.timeout of type string"
Jun 05 21:14:42 k-test1-ctl13 k0s[233185]: I0605 21:14:42.799008  233185 leaderelection.go:260] successfully acquired lease kube-node-lease/k0s-ctrl-k-test1-ctl13
Jun 05 21:14:42 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:42" level=info msg="Acquired leader lease" component=controllerlease
Jun 05 21:14:42 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:42" level=info msg="acquired leader lease" component=controllerlease
Jun 05 21:14:43 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:43" level=debug msg="Probing components" component=prober
Jun 05 21:14:43 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:43" level=debug msg="Failed to ensure the existence of the cluster configuration" component=clusterConfig-reconciler error="json: cannot unmarshal number into Go struct field Chart.spec.extensions.helm.charts.timeout of type string"
Jun 05 21:14:44 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:44" level=debug msg="Failed to ensure the existence of the cluster configuration" component=clusterConfig-reconciler error="json: cannot unmarshal number into Go struct field Chart.spec.extensions.helm.charts.timeout of type string"
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=debug msg="counting controller lease holders" component=controllerlease
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=debug msg="notifying subscribers (1) about controller count: 3" component=controllerlease
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=debug msg="Sending stop message" component=konnectivity
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=debug msg="Waiting for stopping is done" component=konnectivity
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="Shutting down pid 233272" component=konnectivity
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=debug msg="Got event" component=Konnectivity event="{2024-06-05 21:14:45.275211949 +0000 UTC m=+13.336666017 restarting konnectivity server due to server count change map[serverCount:3]}"
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.275351  233272 server.go:167] Shutting down server." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="E0605 21:14:45.275437  233272 server.go:465] \"health server could not listen\" err=\"http: Server closed\"" component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.275454  233272 server.go:467] Health server stopped listening" component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="E0605 21:14:45.275482  233272 server.go:423] \"admin server could not listen\" err=\"http: Server closed\"" component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.275487  233272 server.go:425] Admin server stopped listening" component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="Starting to supervise" component=konnectivity
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="Started successfully, go nuts pid 233455" component=konnectivity
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=debug msg="waiting for server count change" component=konnectivity
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=debug msg="Got event" component=Konnectivity event="{2024-06-05 21:14:45.280654816 +0000 UTC m=+13.342108874 started konnectivity server map[serverCount:3]}"
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.294889  233455 options.go:148] ServerCert set to \"\"." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.294978  233455 options.go:149] ServerKey set to \"\"." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.294981  233455 options.go:150] ServerCACert set to \"\"." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.294984  233455 options.go:151] ClusterCert set to \"/var/lib/k0s/pki/server.crt\"." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.294990  233455 options.go:152] ClusterKey set to \"/var/lib/k0s/pki/server.key\"." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.294993  233455 options.go:153] ClusterCACert set to \"\"." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.294997  233455 options.go:154] Mode set to \"grpc\"." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295001  233455 options.go:155] UDSName set to \"/run/k0s/konnectivity-server/konnectivity-server.sock\"." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295005  233455 options.go:156] DeleteUDSFile set to true." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295009  233455 options.go:157] Server port set to 0." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295014  233455 options.go:158] Server bind address set to \"\"." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295018  233455 options.go:159] Agent port set to 8132." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295022  233455 options.go:160] Agent bind address set to \"\"." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295027  233455 options.go:161] Admin port set to 8133." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295032  233455 options.go:162] Admin bind address set to \"127.0.0.1\"." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295036  233455 options.go:163] Health port set to 8092." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295041  233455 options.go:164] Health bind address set to \"\"." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295046  233455 options.go:165] Keepalive time set to 1h0m0s." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295077  233455 options.go:166] Frontend keepalive time set to 1h0m0s." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295086  233455 options.go:167] EnableProfiling set to false." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295094  233455 options.go:168] EnableContentionProfiling set to false." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295103  233455 options.go:169] ServerID set to 35c14efde7c4c084741960546145d033." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295112  233455 options.go:170] ServerCount set to 3." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295122  233455 options.go:171] AgentNamespace set to \"kube-system\"." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295131  233455 options.go:172] AgentServiceAccount set to \"konnectivity-agent\"." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295145  233455 options.go:173] AuthenticationAudience set to \"system:konnectivity-server\"." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295155  233455 options.go:174] KubeconfigPath set to \"/var/lib/k0s/pki/konnectivity.conf\"." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295167  233455 options.go:175] KubeconfigQPS set to 0.000000." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295180  233455 options.go:176] KubeconfigBurst set to 0." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295193  233455 options.go:177] ProxyStrategies set to \"destHost,default\"." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.295207  233455 options.go:178] CipherSuites set to [\"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256\" \"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384\" \"TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256\" \"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256\" \"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384\" \"TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256\"]." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.296735  233455 server.go:130] Starting frontend server for client connections." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.297019  233455 server.go:145] Starting agent server for tunnel connections." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.297302  233455 server.go:152] Starting admin server for debug connections." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=info msg="I0605 21:14:45.297389  233455 server.go:159] Starting health server for healthchecks." component=konnectivity stream=stderr
Jun 05 21:14:45 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:45" level=debug msg="Failed to ensure the existence of the cluster configuration" component=clusterConfig-reconciler error="json: cannot unmarshal number into Go struct field Chart.spec.extensions.helm.charts.timeout of type string"
Jun 05 21:14:46 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:46" level=debug msg="Failed to ensure the existence of the cluster configuration" component=clusterConfig-reconciler error="json: cannot unmarshal number into Go struct field Chart.spec.extensions.helm.charts.timeout of type string"
Jun 05 21:14:47 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:47" level=debug msg="Failed to ensure the existence of the cluster configuration" component=clusterConfig-reconciler error="json: cannot unmarshal number into Go struct field Chart.spec.extensions.helm.charts.timeout of type string"
Jun 05 21:14:47 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:47" level=debug msg="not the leader, can't approve certificates" component=csrapprover
Jun 05 21:14:48 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:48" level=debug msg="Failed to ensure the existence of the cluster configuration" component=clusterConfig-reconciler error="json: cannot unmarshal number into Go struct field Chart.spec.extensions.helm.charts.timeout of type string"
Jun 05 21:14:49 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:49" level=debug msg="Failed to ensure the existence of the cluster configuration" component=clusterConfig-reconciler error="json: cannot unmarshal number into Go struct field Chart.spec.extensions.helm.charts.timeout of type string"
Jun 05 21:14:50 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:50" level=debug msg="Failed to ensure the existence of the cluster configuration" component=clusterConfig-reconciler error="json: cannot unmarshal number into Go struct field Chart.spec.extensions.helm.charts.timeout of type string"
Jun 05 21:14:51 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:51" level=debug msg="Failed to ensure the existence of the cluster configuration" component=clusterConfig-reconciler error="json: cannot unmarshal number into Go struct field Chart.spec.extensions.helm.charts.timeout of type string"
Jun 05 21:14:52 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:52" level=debug msg="Failed to ensure the existence of the cluster configuration" component=clusterConfig-reconciler error="json: cannot unmarshal number into Go struct field Chart.spec.extensions.helm.charts.timeout of type string"
Jun 05 21:14:53 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:53" level=debug msg="Probing components" component=prober
Jun 05 21:14:53 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:53" level=debug msg="Failed to ensure the existence of the cluster configuration" component=clusterConfig-reconciler error="json: cannot unmarshal number into Go struct field Chart.spec.extensions.helm.charts.timeout of type string"
Jun 05 21:14:54 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:54" level=debug msg="Failed to ensure the existence of the cluster configuration" component=clusterConfig-reconciler error="json: cannot unmarshal number into Go struct field Chart.spec.extensions.helm.charts.timeout of type string"
Jun 05 21:14:55 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:55" level=debug msg="counting controller lease holders" component=controllerlease
Jun 05 21:14:55 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:55" level=debug msg="notifying subscribers (1) about controller count: 3" component=controllerlease
Jun 05 21:14:55 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:55" level=debug msg="waiting for server count change" component=konnectivity
Jun 05 21:14:55 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:55" level=debug msg="Failed to ensure the existence of the cluster configuration" component=clusterConfig-reconciler error="json: cannot unmarshal number into Go struct field Chart.spec.extensions.helm.charts.timeout of type string"
Jun 05 21:14:56 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:56" level=debug msg="Failed to ensure the existence of the cluster configuration" component=clusterConfig-reconciler error="json: cannot unmarshal number into Go struct field Chart.spec.extensions.helm.charts.timeout of type string"
Jun 05 21:14:57 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:57" level=debug msg="Failed to ensure the existence of the cluster configuration" component=clusterConfig-reconciler error="json: cannot unmarshal number into Go struct field Chart.spec.extensions.helm.charts.timeout of type string"
Jun 05 21:14:57 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:57" level=debug msg="not the leader, can't approve certificates" component=csrapprover
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=debug msg="Failed to ensure the existence of the cluster configuration" component=clusterConfig-reconciler error="json: cannot unmarshal number into Go struct field Chart.spec.extensions.helm.charts.timeout of type string"
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=debug msg="Failed to ensure the existence of the cluster configuration" component=clusterConfig-reconciler error="json: cannot unmarshal number into Go struct field Chart.spec.extensions.helm.charts.timeout of type string"
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="stopped component CRD"
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="stopped component CRD"
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="stopped component EtcdMemberReconciler"
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="stopped component Status"
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="stopped component CSRApprover"
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=debug msg="Sending stop message" component=k0s-control-api
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=debug msg="Waiting for stopping is done" component=k0s-control-api
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="CSR Approver context done" component=csrapprover
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="Shutting down pid 233415" component=k0s-control-api
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="stopped component K0SControlAPI"
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="stopped component Manager"
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="stopped component LeasePool"
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=debug msg="Sending stop message" component=kube-apiserver
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=debug msg="Waiting for stopping is done" component=kube-apiserver
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="Shutting down pid 233273" component=kube-apiserver
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="Lost leader lease" component=poolleaderelector
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="lost leader lease" component=poolleaderelector
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="I0605 21:14:58.595594  233273 controller.go:128] Shutting down kubernetes service endpoint reconciler" component=kube-apiserver stream=stderr
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="I0605 21:14:58.595628  233273 genericapiserver.go:670] \"[graceful-termination] pre-shutdown hooks completed\" name=\"PreShutdownHooksStopped\"" component=kube-apiserver stream=stderr
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="I0605 21:14:58.595597  233273 genericapiserver.go:527] \"[graceful-termination] shutdown event\" name=\"ShutdownInitiated\"" component=kube-apiserver stream=stderr
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="I0605 21:14:58.595640  233273 genericapiserver.go:530] \"[graceful-termination] shutdown event\" name=\"AfterShutdownDelayDuration\"" component=kube-apiserver stream=stderr
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="I0605 21:14:58.595675  233273 genericapiserver.go:594] \"[graceful-termination] shutdown event\" name=\"NotAcceptingNewRequest\"" component=kube-apiserver stream=stderr
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="I0605 21:14:58.595681  233273 genericapiserver.go:628] \"[graceful-termination] not going to wait for active watch request(s) to drain\"" component=kube-apiserver stream=stderr
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="I0605 21:14:58.595692  233273 genericapiserver.go:619] [graceful-termination] in-flight non long-running request(s) have drained" component=kube-apiserver stream=stderr
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="I0605 21:14:58.595725  233273 genericapiserver.go:660] \"[graceful-termination] shutdown event\" name=\"InFlightRequestsDrained\"" component=kube-apiserver stream=stderr
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="I0605 21:14:58.595740  233273 dynamic_cafile_content.go:171] \"Shutting down controller\" name=\"request-header::/var/lib/k0s/pki/front-proxy-ca.crt\"" component=kube-apiserver stream=stderr
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="I0605 21:14:58.595736  233273 controller.go:157] Shutting down quota evaluator" component=kube-apiserver stream=stderr
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="I0605 21:14:58.595759  233273 controller.go:176] quota evaluator worker shutdown" component=kube-apiserver stream=stderr
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="I0605 21:14:58.595789  233273 controller.go:176] quota evaluator worker shutdown" component=kube-apiserver stream=stderr
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="I0605 21:14:58.595801  233273 controller.go:176] quota evaluator worker shutdown" component=kube-apiserver stream=stderr
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="I0605 21:14:58.595808  233273 controller.go:176] quota evaluator worker shutdown" component=kube-apiserver stream=stderr
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="I0605 21:14:58.595815  233273 controller.go:176] quota evaluator worker shutdown" component=kube-apiserver stream=stderr
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="I0605 21:14:58.595824  233273 system_namespaces_controller.go:77] Shutting down system namespaces controller" component=kube-apiserver stream=stderr
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="I0605 21:14:58.595847  233273 controller.go:84] Shutting down OpenAPI AggregationController" component=kube-apiserver stream=stderr
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="I0605 21:14:58.595848  233273 establishing_controller.go:87] Shutting down EstablishingController" component=kube-apiserver stream=stderr
Jun 05 21:14:58 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:14:58" level=info msg="I0605 21:14:58.595870  233273 dynamic_serving_content.go:146] \"Shutting down controller\" name=\"aggregator-proxy-cert::/var/lib/k0s/pki/front-proxy-client.crt::/var/lib/k0s/pki/front-proxy-client.key\"" component=kube-apiserver stream=stderr
...
Jun 05 21:15:04 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:15:04" level=info msg="stopped component Etcd"
Jun 05 21:15:04 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:15:04" level=info msg="All node components stopped"
Jun 05 21:15:04 k-test1-ctl13 k0s[233185]: time="2024-06-05 21:15:04" level=info msg="stopping konnectivity server reconfig loop"
Jun 05 21:15:04 k-test1-ctl13 k0s[233185]: Error: failed to start cluster components: failed to ensure the existence of the cluster configuration: timed out waiting for the condition
Jun 05 21:15:04 k-test1-ctl13 systemd[1]: k0scontroller.service: Main process exited, code=exited, status=1/FAILURE
Jun 05 21:15:04 k-test1-ctl13 systemd[1]: k0scontroller.service: Failed with result 'exit-code'.
Jun 05 21:15:04 k-test1-ctl13 systemd[1]: k0scontroller.service: Consumed 34.079s CPU time.

Additional context

The challenge comes from the fact that I can't seem to be able to change the timeout value to a correct one for the new version and the upgrade fails without a correct value. Also, I noticed that the default value mentionned in the docs does not seem to apply since mine is set to 0 by default.
https://docs.k0sproject.io/v1.30.1+k0s.0/helm-charts/#chart-configuration
@coutug coutug added the bug Something isn't working label Jun 5, 2024
@twz123
Copy link
Member

twz123 commented Jun 6, 2024

Right. You're hitting #4172 (comment). Removing the timeout from all chart resources before upgrading doesn't work for you?

@coutug
Copy link
Author

coutug commented Jun 6, 2024

I didnt thought about simply removing it, it now works. Thank you!

@coutug coutug closed this as completed Jun 6, 2024
@twz123
Copy link
Member

twz123 commented Jun 6, 2024

Also, I noticed that the default value mentionned in the docs does not seem to apply since mine is set to 0 by default.

Thanks for pointing out. -> #4566

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@twz123 @coutug