Helm extension can't add helm chart repository

[Aldans]

Before creating an issue, make sure you've checked the following:

• You are running the latest released version of k0s
• Make sure you've searched for existing issues, both open and closed
• Make sure you've searched for PRs too, a fix might've been merged already
• You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

$ uname -srvmo; cat /etc/os-release || lsb_release -a
Linux 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 GNU/Linux
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Version

v1.30.1+k0s.0 and v1.29.2

Sysinfo

`k0s sysinfo`

Total memory: 3.8 GiB (pass)
Disk space available for /var/lib/k0s: 16.7 GiB (pass)
Name resolution: localhost: [127.0.0.1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 5.4.0-163-generic (pass)
  Max. file descriptors per process: current: 1048576 / max: 1048576 (pass)
  AppArmor: active (pass)
  Executable in PATH: modprobe: /usr/sbin/modprobe (pass)
  Executable in PATH: mount: /usr/bin/mount (pass)
  Executable in PATH: umount: /usr/bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 1 (pass)
    cgroup controller "cpu": available (pass)
    cgroup controller "cpuacct": available (pass)
    cgroup controller "cpuset": available (pass)
    cgroup controller "memory": available (pass)
    cgroup controller "devices": available (pass)
    cgroup controller "freezer": available (pass)
    cgroup controller "pids": available (pass)
    cgroup controller "hugetlb": available (pass)
    cgroup controller "blkio": available (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass)
    CONFIG_CPUSETS: Cpuset support: built-in (pass)
    CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass)
    CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
      CONFIG_NETFILTER_NETLINK: module (pass)
      CONFIG_NF_NAT: module (pass)
      CONFIG_IP_SET: IP set support: module (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
      CONFIG_IP_VS: IP virtual server support: module (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
        CONFIG_IP_VS_SH: Source hashing scheduling: module (pass)
        CONFIG_IP_VS_RR: Round-robin scheduling: module (pass)
        CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: module (pass)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: module (pass)
        CONFIG_IP_NF_MANGLE: Packet mangling: module (pass)
      CONFIG_NF_DEFRAG_IPV4: module (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: module (pass)
        CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass)
      CONFIG_NF_DEFRAG_IPV6: module (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass)
      CONFIG_LLC: module (pass)
      CONFIG_STP: module (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: built-in (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)

What happened?

Tried to create single cluster it was created, but helm chart not installed, looks like can't add helm repo

sudo k0s install controller --single -c k0s.yaml && sudo k0s start

config k0s.yaml look like this

---
apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  creationTimestamp: null
  name: k0s
spec:
  api:
    address: 192.168.0.8
    k0sApiPort: 9443
    port: 6443
    sans:
    - 192.168.0.8
    - 10.11.12.172
  controllerManager: {}
  extensions:
    helm:
      repositories:
      - name: bitnami
        url: https://charts.bitnami.com/bitnami
      charts:
      - name: argo-workflows
        chartname: bitnami/argo-workflows
        version: "6.5.0"
        namespace: argo
        values: |
          server:
            auth:
              mode: server
              secure: false

$ sudo k0s status

Version: v1.30.1+k0s.0
Process ID: 76682
Role: controller
Workloads: true
SingleNode: true
Kube-api probing successful: true
Kube-api probing last error:

if install helm and add repo manual, it is working well

$ helm repo add bitnami https://charts.bitnami.com/bitnami
"bitnami" has been added to your repositories

$ helm repo list
NAME   	URL
bitnami	https://charts.bitnami.com/bitnami

Steps to reproduce

1. install k0s
2. create k0s.yaml config file like example above
3. create single cluster sudo k0s install controller --single -c k0s.yaml && sudo k0s star
4. check config status sudo k0s config status

Expected behavior

No response

Actual behavior

Check config status:

$ sudo k0s config status
LAST SEEN   TYPE      REASON              OBJECT              MESSAGE
2m19s       Warning   FailedReconciling   clusterconfig/k0s   can't reconcile helm based extensions: can't init repository "https://charts.bitnami.com/bitnami": can't add repository: "repo" is not a valid chart repository or cannot be reached: Get "https://repo.broadcom.com/bitnami-files/index.yaml": remote error: tls: handshake failure

Screenshots and logs

No response

Additional context

On last Monday 2024.06.10 it was working well

[makhov]

Hello, the detective agency "Mandelbug" is here!

This is an interesting case, where everyone is a murderer.

Let's restore the sequence of events:

• Helm maintainers for an unknown reason add a check for the server name to the TLSConfig in case the InsecureSkipVerify flag is set. Probably, they thought that it helps with security (it's not)
• We, introducing the helm feature, for some reason (probably for testing purposes) added the InsecureSkipVerify: true and never changed that.
• Bitnami creates a very popular helm chart repository
• VMware buys Bitnami
• Broadcom buys VMware
• And finally last week Broadcom decided to move the Bitnami repository to the repo.broadcom.com domain using a redirect.

Obviously, a redirect with a pinned Server Name won't work properly, and our poor k0s helm deployer dies fails to add the repo.

As usual, casualties could have been avoided, but the fix has been open for 3 years and has never been merged.

We'll think about how we can fix the issue on the k0s side and will try to push the helm upstream to merge the fix on their side.

As a workaround, you can use https://repo.broadcom.com/bitnami-files as a Bitnami repo URL.

[twz123]

This is triggered by --insecure-skip-tls-verify, which k0s sets by default, unfortunately.

Reproducer using the Helm CLI:

$ helm repo add bitnami https://charts.bitnami.com/bitnami --force-update --insecure-skip-tls-verify
Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: Get "https://repo.broadcom.com/bitnami-files/index.yaml": remote error: tls: handshake failure

The upstream Helm issue:

• Helm pull fails if pulling from a repository that redirects to another domain helm/helm#9317

and the Pull Request addressing this:

• Fix issue with helm pull failing if pulling from a repository that redirects to another domain helm/helm#9318

[twz123]

When #4660 gets released, you should be able to use the following to circumvent the problem:

apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  name: k0s
spec:
  # [...]
  extensions:
    helm:
      repositories:
      - name: bitnami
        url: https://charts.bitnami.com/bitnami
        insecure: false
  # [...]

[github-actions]

The issue is marked as stale since no activity has been recorded in 30 days