Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

v0.17.5 can't connect to a single host config #674

Open
nwneisen opened this issue Apr 5, 2024 · 6 comments
Open

v0.17.5 can't connect to a single host config #674

nwneisen opened this issue Apr 5, 2024 · 6 comments

Comments

@nwneisen
Copy link

nwneisen commented Apr 5, 2024 •
edited
Loading

Summary

v0.17.5 can no longer connect to a single host node. v0.17.4 works fine.

Details

I just installed v0.17.5 and created a new config using k0sctl init and modified the hosts section to be a single host that pointed to a VM on my local machine. The apply command kept hanging up when I tried to apply the config.

nick:~/code/homelab: k0sctl apply -c homelab.yaml -d
DEBU[0000] Loaded configuration:
apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: homelab-cluster
spec:
  hosts:
  - ssh:
      address: 10.252.198.229
      user: ubuntu
      port: 22
      keyPath: id_rsa
    role: single
  k0s:
    version: v1.28.7+k0s.0
    versionChannel: stable
    dynamicConfig: false
    config: {} 

⠀⣿⣿⡇⠀⠀⢀⣴⣾⣿⠟⠁⢸⣿⣿⣿⣿⣿⣿⣿⡿⠛⠁⠀⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀█████████ █████████ ███
⠀⣿⣿⡇⣠⣶⣿⡿⠋⠀⠀⠀⢸⣿⡇⠀⠀⠀⣠⠀⠀⢀⣠⡆⢸⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀███          ███    ███
⠀⣿⣿⣿⣿⣟⠋⠀⠀⠀⠀⠀⢸⣿⡇⠀⢰⣾⣿⠀⠀⣿⣿⡇⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀███          ███    ███
⠀⣿⣿⡏⠻⣿⣷⣤⡀⠀⠀⠀⠸⠛⠁⠀⠸⠋⠁⠀⠀⣿⣿⡇⠈⠉⠉⠉⠉⠉⠉⠉⠉⢹⣿⣿⠀███          ███    ███
⠀⣿⣿⡇⠀⠀⠙⢿⣿⣦⣀⠀⠀⠀⣠⣶⣶⣶⣶⣶⣶⣿⣿⡇⢰⣶⣶⣶⣶⣶⣶⣶⣶⣾⣿⣿⠀█████████    ███    ██████████
k0sctl v0.17.5 Copyright 2023, k0sctl authors.
Anonymized telemetry of usage will be sent to the authors.
By continuing to use k0sctl you agree to these terms:
https://k0sproject.io/licenses/eula
DEBU[0000] Preparing phase 'Set k0s version'            
DEBU[0000] Preparing phase 'Connect to hosts'           
INFO[0000] ==> Running phase: Connect to hosts 
DEBU[0000] [ssh] 10.252.198.229:22: HashKnownHosts is set to "yes", will hash newly added keys 
DEBU[0000] using SSH_AUTH_SOCK=/run/user/1000/keyring/ssh 
DEBU[0000] [ssh] 10.252.198.229:22: using all keys (2) from ssh agent because a keypath was not explicitly given 
DEBU[0000] [ssh] 10.252.198.229:22: using an unencrypted private key from id_rsa 
DEBU[0000] [SSH] 10.252.198.229: failed to connect: ssh dial: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain 
DEBU[0005] retrying, attempt 1 - last error: not connected: client connect: ssh dial: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain 
DEBU[0005] [ssh] 10.252.198.229:22: HashKnownHosts is set to "yes", will hash newly added keys 
DEBU[0005] using SSH_AUTH_SOCK=/run/user/1000/keyring/ssh 
DEBU[0005] [ssh] 10.252.198.229:22: using all keys (2) from ssh agent because a keypath was not explicitly given 
DEBU[0005] [SSH] 10.252.198.229: failed to connect: ssh dial: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

I reverted my k0sctl version to v0.17.4 and it worked fine.

nick:~/code/homelab: k0sctl apply -c homelab.yaml 

⠀⣿⣿⡇⠀⠀⢀⣴⣾⣿⠟⠁⢸⣿⣿⣿⣿⣿⣿⣿⡿⠛⠁⠀⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀█████████ █████████ ███
⠀⣿⣿⡇⣠⣶⣿⡿⠋⠀⠀⠀⢸⣿⡇⠀⠀⠀⣠⠀⠀⢀⣠⡆⢸⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀███          ███    ███
⠀⣿⣿⣿⣿⣟⠋⠀⠀⠀⠀⠀⢸⣿⡇⠀⢰⣾⣿⠀⠀⣿⣿⡇⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀███          ███    ███
⠀⣿⣿⡏⠻⣿⣷⣤⡀⠀⠀⠀⠸⠛⠁⠀⠸⠋⠁⠀⠀⣿⣿⡇⠈⠉⠉⠉⠉⠉⠉⠉⠉⢹⣿⣿⠀███          ███    ███
⠀⣿⣿⡇⠀⠀⠙⢿⣿⣦⣀⠀⠀⠀⣠⣶⣶⣶⣶⣶⣶⣿⣿⡇⢰⣶⣶⣶⣶⣶⣶⣶⣶⣾⣿⣿⠀█████████    ███    ██████████
k0sctl v0.17.4 Copyright 2023, k0sctl authors.
Anonymized telemetry of usage will be sent to the authors.
By continuing to use k0sctl you agree to these terms:
https://k0sproject.io/licenses/eula
INFO ==> Running phase: Connect to hosts 
INFO [ssh] 10.252.198.229:22: connected           
INFO ==> Running phase: Detect host operating systems 
INFO [ssh] 10.252.198.229:22: is running Ubuntu 22.04.4 LTS 
INFO ==> Running phase: Acquire exclusive host lock 
INFO ==> Running phase: Prepare hosts    
INFO ==> Running phase: Gather host facts 
INFO [ssh] 10.252.198.229:22: using homelab as hostname 
INFO [ssh] 10.252.198.229:22: discovered ens3 as private interface 
INFO ==> Running phase: Validate hosts   
INFO ==> Running phase: Gather k0s facts 
INFO [ssh] 10.252.198.229:22: found existing configuration 
INFO [ssh] 10.252.198.229:22: is running k0s single version v1.28.7+k0s.0 
INFO ==> Running phase: Validate facts   
INFO [ssh] 10.252.198.229:22: validating configuration 
INFO ==> Running phase: Release exclusive host lock 
INFO ==> Running phase: Disconnect from hosts 
INFO ==> Finished in 2s                  
INFO k0s cluster version v1.28.7+k0s.0 is now installed 
INFO Tip: To access the cluster you can now fetch the admin kubeconfig using: 
INFO      k0sctl kubeconfig              
A new version v0.17.5 of k0sctl is available: https://github.com/k0sproject/k0sctl/releases/tag/v0.17.5

Environment

Host: Pop OS
VM manager: multipass
VM distro: Ubuntu 22.04
@twz123
Copy link
Member

twz123 commented Apr 5, 2024

Just to make sure, can you try to invoke k0sctl v0.17.5 via env -u SSH_AUTH_SOCK k0sctl ... and see if it makes a difference?

@kke
Copy link
Contributor

kke commented Apr 5, 2024

Probably has something to do with k0sproject/rig#162

@nwneisen
Copy link
Author

nwneisen commented Apr 5, 2024

Just to make sure, can you try to invoke k0sctl v0.17.5 via env -u SSH_AUTH_SOCK k0sctl ... and see if it makes a difference?

This is also with things switched over to a lima VM that works with v0.17.4

nick@amd-t14:~/Downloads: env -u SSH_AUTH_SOCK ./k0sctl-linux-x64 apply -c ~/code/homelab/
charts/       .git/         homelab.yaml  values.yaml   
Chart.yaml    .helmignore   Makefile      
nick@amd-t14:~/Downloads: env -u SSH_AUTH_SOCK ./k0sctl-linux-x64 apply -c ~/code/homelab/homelab.yaml --debug
DEBU[0000] Loaded configuration:
apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: homelab-cluster
spec:
  hosts:
  # - ssh:
  #     address: 192.168.1.60
  #     user: nick
  #     port: 22
  #     keyPath: /home/nick/.ssh/id_rsa
  #   role: single
  - ssh:
      address: 127.0.0.1
      user: nick
      port: 60022
      keyPath: /home/nick/.lima/_config/user
    role: single
  k0s:
    version: v1.28.7+k0s.0
    versionChannel: stable
    dynamicConfig: false
    config: {} 

⠀⣿⣿⡇⠀⠀⢀⣴⣾⣿⠟⠁⢸⣿⣿⣿⣿⣿⣿⣿⡿⠛⠁⠀⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀█████████ █████████ ███
⠀⣿⣿⡇⣠⣶⣿⡿⠋⠀⠀⠀⢸⣿⡇⠀⠀⠀⣠⠀⠀⢀⣠⡆⢸⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀███          ███    ███
⠀⣿⣿⣿⣿⣟⠋⠀⠀⠀⠀⠀⢸⣿⡇⠀⢰⣾⣿⠀⠀⣿⣿⡇⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀███          ███    ███
⠀⣿⣿⡏⠻⣿⣷⣤⡀⠀⠀⠀⠸⠛⠁⠀⠸⠋⠁⠀⠀⣿⣿⡇⠈⠉⠉⠉⠉⠉⠉⠉⠉⢹⣿⣿⠀███          ███    ███
⠀⣿⣿⡇⠀⠀⠙⢿⣿⣦⣀⠀⠀⠀⣠⣶⣶⣶⣶⣶⣶⣿⣿⡇⢰⣶⣶⣶⣶⣶⣶⣶⣶⣾⣿⣿⠀█████████    ███    ██████████
k0sctl v0.17.4 Copyright 2023, k0sctl authors.
Anonymized telemetry of usage will be sent to the authors.
By continuing to use k0sctl you agree to these terms:
https://k0sproject.io/licenses/eula
DEBU[0000] Preparing phase 'Set k0s version'            
DEBU[0000] Preparing phase 'Connect to hosts'           
INFO[0000] ==> Running phase: Connect to hosts 
DEBU[0000] [ssh] 127.0.0.1:60022: HashKnownHosts is set to "yes", will hash newly added keys 
DEBU[0000] [ssh] 127.0.0.1:60022: using an unencrypted private key from /home/nick/.lima/_config/user 
DEBU[0000] [SSH] 127.0.0.1: failed to connect: ssh dial: dial tcp 127.0.0.1:60022: connect: connection refused 
DEBU[0005] retrying, attempt 1 - last error: not connected: client connect: ssh dial: dial tcp 127.0.0.1:60022: connect: connection refused 
DEBU[0005] [ssh] 127.0.0.1:60022: HashKnownHosts is set to "yes", will hash newly added keys 
DEBU[0005] [SSH] 127.0.0.1: failed to connect: ssh dial: dial tcp 127.0.0.1:60022: connect: connection refused 
DEBU[0010] retrying, attempt 2 - last error: not connected: client connect: ssh dial: dial tcp 127.0.0.1:60022: connect: connection refused 
DEBU[0010] [ssh] 127.0.0.1:60022: HashKnownHosts is set to "yes", will hash newly added keys 
DEBU[0010] [SSH] 127.0.0.1: failed to connect: ssh dial: dial tcp 127.0.0.1:60022: connect: connection refused 
DEBU[0015] retrying, attempt 3 - last error: not connected: client connect: ssh dial: dial tcp 127.0.0.1:60022: connect: connection refused 
DEBU[0015] [ssh] 127.0.0.1:60022: HashKnownHosts is set to "yes", will hash newly added keys 
DEBU[0015] [SSH] 127.0.0.1: failed to connect: ssh dial: dial tcp 127.0.0.1:60022: connect: connection refused

@twz123
Copy link
Member

twz123 commented Apr 5, 2024

Your last log snippets show network errors ("connection refused"), not SSH handshake failures ("ssh: handshake failed: ssh: unable to authenticate"), btw.

@kke kke mentioned this issue Apr 10, 2024
Closed
@jadolg
Copy link

jadolg commented Apr 13, 2024

What is the disposition of your keys? I see from the logs there are 2 keys loaded in the ssh agent and one key got loaded from the default configuration:

DEBU[0000] [ssh] 10.252.198.229:22: using all keys (2) from ssh agent because a keypath was not explicitly given 
DEBU[0000] [ssh] 10.252.198.229:22: using an unencrypted private key from id_rsa

Is this correct?
If so, which of these should have access to the machine?
I want to try to reproduce it on a test.

@nwneisen
Copy link
Author

Just to make sure, can you try to invoke k0sctl v0.17.5 via env -u SSH_AUTH_SOCK k0sctl ... and see if it makes a difference?

Running with env -u SSH_AUTH_SOCK is successful

nick@amd-t14:~/code/homelab (master): env -u SSH_AUTH_SOCK ~/Downloads/k0sctl-linux-x64 apply -c ~/code/homelab/homelab.yaml  -d
DEBU[0000] Loaded configuration:
apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: homelab-cluster
spec:
  hosts:
  # - ssh:
  #     address: 192.168.1.2
  #     user: root
  #     port: 22
  #     keyPath: ~/.ssh/id_ed25519
  #   role: single
  - ssh:
      address: 10.252.198.12
      user: ubuntu
      port: 22
      keyPath: ~/id_rsa
    role: single
  # - ssh:
  #     address: 127.0.0.1
  #     user: nick
  #     port: 60022
  #     keyPath: /home/nick/.ssh/id_ed25519
  #   role: single
  k0s:
    version: v1.28.7+k0s.0
    versionChannel: stable
    dynamicConfig: false
    config: {} 

⠀⣿⣿⡇⠀⠀⢀⣴⣾⣿⠟⠁⢸⣿⣿⣿⣿⣿⣿⣿⡿⠛⠁⠀⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀█████████ █████████ ███
⠀⣿⣿⡇⣠⣶⣿⡿⠋⠀⠀⠀⢸⣿⡇⠀⠀⠀⣠⠀⠀⢀⣠⡆⢸⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀███          ███    ███
⠀⣿⣿⣿⣿⣟⠋⠀⠀⠀⠀⠀⢸⣿⡇⠀⢰⣾⣿⠀⠀⣿⣿⡇⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀███          ███    ███
⠀⣿⣿⡏⠻⣿⣷⣤⡀⠀⠀⠀⠸⠛⠁⠀⠸⠋⠁⠀⠀⣿⣿⡇⠈⠉⠉⠉⠉⠉⠉⠉⠉⢹⣿⣿⠀███          ███    ███
⠀⣿⣿⡇⠀⠀⠙⢿⣿⣦⣀⠀⠀⠀⣠⣶⣶⣶⣶⣶⣶⣿⣿⡇⢰⣶⣶⣶⣶⣶⣶⣶⣶⣾⣿⣿⠀█████████    ███    ██████████
k0sctl v0.17.5 Copyright 2023, k0sctl authors.
Anonymized telemetry of usage will be sent to the authors.
By continuing to use k0sctl you agree to these terms:
https://k0sproject.io/licenses/eula
DEBU[0000] Preparing phase 'Set k0s version'            
DEBU[0000] Preparing phase 'Connect to hosts'           
INFO[0000] ==> Running phase: Connect to hosts 
DEBU[0000] [ssh] 10.252.198.12:22: HashKnownHosts is set to "yes", will hash newly added keys 
DEBU[0000] [ssh] 10.252.198.12:22: using an unencrypted private key from /home/nick/id_rsa 
DEBU[0000] [ssh] 10.252.198.12:22: executing `uname | grep -q Linux` 
DEBU[0000] [ssh] 10.252.198.12:22: executing `cat /etc/os-release || cat /usr/lib/os-release` 
DEBU[0000] [ssh] 10.252.198.12:22: PRETTY_NAME="Ubuntu 22.04.4 LTS" 
DEBU[0000] [ssh] 10.252.198.12:22: NAME="Ubuntu"        
DEBU[0000] [ssh] 10.252.198.12:22: VERSION_ID="22.04"   
DEBU[0000] [ssh] 10.252.198.12:22: VERSION="22.04.4 LTS (Jammy Jellyfish)" 
DEBU[0000] [ssh] 10.252.198.12:22: VERSION_CODENAME=jammy 
DEBU[0000] [ssh] 10.252.198.12:22: ID=ubuntu            
DEBU[0000] [ssh] 10.252.198.12:22: ID_LIKE=debian       
DEBU[0000] [ssh] 10.252.198.12:22: HOME_URL="https://www.ubuntu.com/" 
DEBU[0000] [ssh] 10.252.198.12:22: SUPPORT_URL="https://help.ubuntu.com/" 
DEBU[0000] [ssh] 10.252.198.12:22: BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 
DEBU[0000] [ssh] 10.252.198.12:22: PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 
DEBU[0000] [ssh] 10.252.198.12:22: UBUNTU_CODENAME=jammy 
DEBU[0000] [ssh] 10.252.198.12:22: executing `[ "$(id -u)" = 0 ]` 
DEBU[0000] [ssh] 10.252.198.12:22: executing `sudo -n true` 
INFO[0000] [ssh] 10.252.198.12:22: connected            
DEBU[0000] Preparing phase 'Detect host operating systems' 
INFO[0000] ==> Running phase: Detect host operating systems 
INFO[0000] [ssh] 10.252.198.12:22: is running Ubuntu 22.04.4 LTS 
DEBU[0000] Preparing phase 'Acquire exclusive host lock' 
INFO[0000] ==> Running phase: Acquire exclusive host lock 
DEBU[0000] [ssh] 10.252.198.12:22: executing `sudo -s -- test -d /run/lock` 
DEBU[0000] [ssh] 10.252.198.12:22: executing `mktemp`   
DEBU[0000] [ssh] 10.252.198.12:22: /tmp/tmp.xpThjmAuvI  
DEBU[0000] [ssh] 10.252.198.12:22: executing `sudo -s -- cat > "/tmp/tmp.xpThjmAuvI"` 
DEBU[0000] [ssh] 10.252.198.12:22: writing 70 bytes to command stdin: 331f512cab4ba61b7d1883acdcf9606bc8a833b996544906657a073a67b19257-43696 
DEBU[0000] [ssh] 10.252.198.12:22: executing `sudo -s -- mv -n "/tmp/tmp.xpThjmAuvI" "/run/lock/k0sctl"` 
DEBU[0000] [ssh] 10.252.198.12:22: executing `test -f "/tmp/tmp.xpThjmAuvI"` 
DEBU[0000] [ssh] 10.252.198.12:22: executing `sudo -s -- rm -f "/tmp/tmp.xpThjmAuvI"` 
DEBU[0000] [ssh] 10.252.198.12:22: executing `sudo -s -- test -d /run/lock` 
DEBU[0000] [ssh] 10.252.198.12:22: started periodic update of lock file /run/lock/k0sctl timestamp 
DEBU[0000] Preparing phase 'Prepare hosts'              
INFO[0000] ==> Running phase: Prepare hosts    
DEBU[0000] [ssh] 10.252.198.12:22: executing `sudo -s -- command -v -- "curl" 2> /dev/null` 
DEBU[0000] [ssh] 10.252.198.12:22: /usr/bin/curl        
DEBU[0001] [ssh] 10.252.198.12:22: executing `sudo -s -- command -v -- "hostname" 2> /dev/null` 
DEBU[0001] [ssh] 10.252.198.12:22: /usr/bin/hostname    
DEBU[0001] [ssh] 10.252.198.12:22: executing `grep 'container=docker' /proc/1/environ 2> /dev/null` 
DEBU[0001] Preparing phase 'Gather host facts'          
INFO[0001] ==> Running phase: Gather host facts 
DEBU[0001] [ssh] 10.252.198.12:22: executing `uname -m` 
DEBU[0001] [ssh] 10.252.198.12:22: x86_64               
DEBU[0001] [ssh] 10.252.198.12:22: executing `cat /etc/machine-id || cat /var/lib/dbus/machine-id` 
DEBU[0001] [ssh] 10.252.198.12:22: cc23a39e4ffb40329de3dcb699bd92bd 
DEBU[0001] [ssh] 10.252.198.12:22: executing `hostname 2> /dev/null` 
DEBU[0001] [ssh] 10.252.198.12:22: homelab              
INFO[0001] [ssh] 10.252.198.12:22: using homelab as hostname 
DEBU[0001] [ssh] 10.252.198.12:22: executing `PATH=/usr/local/sbin:/usr/sbin:/sbin:$PATH; (ip route list scope global | grep -E "\b(172|10|192\.168)\.") || (ip route list | grep -m1 default)` 
DEBU[0001] [ssh] 10.252.198.12:22: default via 10.252.198.1 dev ens3 proto dhcp src 10.252.198.12 metric 100 
INFO[0001] [ssh] 10.252.198.12:22: discovered ens3 as private interface 
DEBU[0001] [ssh] 10.252.198.12:22: executing `PATH=/usr/local/sbin:/usr/sbin:/sbin:$PATH ip -o addr show dev ens3 scope global` 
DEBU[0001] [ssh] 10.252.198.12:22: 2: ens3    inet 10.252.198.12/24 metric 100 brd 10.252.198.255 scope global ens3\       valid_lft forever preferred_lft forever 
DEBU[0001] Preparing phase 'Validate hosts'             
INFO[0001] ==> Running phase: Validate hosts   
DEBU[0001] [ssh] 10.252.198.12:22: executing `sudo -s -- true` 
DEBU[0001] Preparing phase 'Gather k0s facts'           
DEBU[0001] [ssh] 10.252.198.12:22: executing `sudo -s -- command -v -- "k0s" 2> /dev/null` 
DEBU[0001] [ssh] 10.252.198.12:22: /usr/local/bin/k0s   
INFO[0001] ==> Running phase: Gather k0s facts 
DEBU[0001] [ssh] 10.252.198.12:22: executing `sudo -s -- /usr/local/bin/k0s version` 
DEBU[0001] [ssh] 10.252.198.12:22: v1.28.7+k0s.0        
DEBU[0001] [ssh] 10.252.198.12:22: has k0s binary version v1.28.7+k0s.0 
DEBU[0001] [ssh] 10.252.198.12:22: executing `sudo -s -- test -e /etc/k0s/k0s.yaml 2> /dev/null` 
DEBU[0001] [ssh] 10.252.198.12:22: executing `sudo -s -- cat -- /etc/k0s/k0s.yaml 2> /dev/null` 
INFO[0001] [ssh] 10.252.198.12:22: found existing configuration 
DEBU[0001] [ssh] 10.252.198.12:22: executing `sudo -s -- stat /run/systemd/system` 
DEBU[0001] [ssh] 10.252.198.12:22: File: /run/systemd/system 
DEBU[0001] [ssh] 10.252.198.12:22: Size: 80             Blocks: 0          IO Block: 4096   directory 
DEBU[0001] [ssh] 10.252.198.12:22: Device: 19h/25d      Inode: 284         Links: 3 
DEBU[0001] [ssh] 10.252.198.12:22: Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root) 
DEBU[0001] [ssh] 10.252.198.12:22: Access: 2024-04-14 17:48:33.102862615 -0600 
DEBU[0001] [ssh] 10.252.198.12:22: Modify: 2024-04-14 17:48:33.078862666 -0600 
DEBU[0001] [ssh] 10.252.198.12:22: Change: 2024-04-14 17:48:33.078862666 -0600 
DEBU[0001] [ssh] 10.252.198.12:22: Birth: -             
DEBU[0001] [ssh] 10.252.198.12:22: executing `sudo -s -- systemctl show -p FragmentPath k0scontroller.service 2> /dev/null | cut -d"=" -f2` 
DEBU[0001] [ssh] 10.252.198.12:22: /etc/systemd/system/k0scontroller.service 
DEBU[0001] [ssh] 10.252.198.12:22: executing `sudo -s -- /usr/local/bin/k0s status -o json` 
DEBU[0001] [ssh] 10.252.198.12:22: {                    
DEBU[0001] [ssh] 10.252.198.12:22: "Version": "v1.28.7+k0s.0", 
DEBU[0001] [ssh] 10.252.198.12:22: "Pid": 3836,         
DEBU[0001] [ssh] 10.252.198.12:22: "PPid": 0,           
DEBU[0001] [ssh] 10.252.198.12:22: "Role": "controller", 
DEBU[0001] [ssh] 10.252.198.12:22: "SysInit": "",       
DEBU[0001] [ssh] 10.252.198.12:22: "StubFile": "",      
DEBU[0001] [ssh] 10.252.198.12:22: "Output": "",        
DEBU[0001] [ssh] 10.252.198.12:22: "Workloads": true,   
DEBU[0001] [ssh] 10.252.198.12:22: "SingleNode": true,  
DEBU[0001] [ssh] 10.252.198.12:22: "Args": [            
DEBU[0001] [ssh] 10.252.198.12:22: "/usr/local/bin/k0s", 
DEBU[0001] [ssh] 10.252.198.12:22: "controller",        
DEBU[0001] [ssh] 10.252.198.12:22: "--config=/etc/k0s/k0s.yaml", 
DEBU[0001] [ssh] 10.252.198.12:22: "--data-dir=/var/lib/k0s", 
DEBU[0001] [ssh] 10.252.198.12:22: "--single=true"      
DEBU[0001] [ssh] 10.252.198.12:22: ],                   
DEBU[0001] [ssh] 10.252.198.12:22: "WorkerToAPIConnectionStatus": { 
DEBU[0001] [ssh] 10.252.198.12:22: "Message": "",       
DEBU[0001] [ssh] 10.252.198.12:22: "Success": true      
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "ClusterConfig": {   
DEBU[0001] [ssh] 10.252.198.12:22: "metadata": {        
DEBU[0001] [ssh] 10.252.198.12:22: "name": "k0s",       
DEBU[0001] [ssh] 10.252.198.12:22: "creationTimestamp": null 
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "kind": "ClusterConfig", 
DEBU[0001] [ssh] 10.252.198.12:22: "apiVersion": "k0s.k0sproject.io/v1beta1", 
DEBU[0001] [ssh] 10.252.198.12:22: "spec": {            
DEBU[0001] [ssh] 10.252.198.12:22: "api": {             
DEBU[0001] [ssh] 10.252.198.12:22: "address": "10.252.198.12", 
DEBU[0001] [ssh] 10.252.198.12:22: "k0sApiPort": 9443,  
DEBU[0001] [ssh] 10.252.198.12:22: "port": 6443,        
DEBU[0001] [ssh] 10.252.198.12:22: "sans": [            
DEBU[0001] [ssh] 10.252.198.12:22: "10.252.198.12",     
DEBU[0001] [ssh] 10.252.198.12:22: "fe80::5054:ff:fe83:7af3", 
DEBU[0001] [ssh] 10.252.198.12:22: "127.0.0.1"          
DEBU[0001] [ssh] 10.252.198.12:22: ]                    
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "controllerManager": {}, 
DEBU[0001] [ssh] 10.252.198.12:22: "scheduler": {},     
DEBU[0001] [ssh] 10.252.198.12:22: "storage": {         
DEBU[0001] [ssh] 10.252.198.12:22: "etcd": {            
DEBU[0001] [ssh] 10.252.198.12:22: "externalCluster": null, 
DEBU[0001] [ssh] 10.252.198.12:22: "peerAddress": "10.252.198.12" 
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "kine": {            
DEBU[0001] [ssh] 10.252.198.12:22: "dataSource": "sqlite:/var/lib/k0s/db/state.db?mode=rwc\u0026_journal=WAL\u0026cache=shared" 
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "type": "etcd"       
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "network": {         
DEBU[0001] [ssh] 10.252.198.12:22: "calico": null,      
DEBU[0001] [ssh] 10.252.198.12:22: "dualStack": {},     
DEBU[0001] [ssh] 10.252.198.12:22: "kubeProxy": {       
DEBU[0001] [ssh] 10.252.198.12:22: "mode": "iptables",  
DEBU[0001] [ssh] 10.252.198.12:22: "metricsBindAddress": "0.0.0.0:10249", 
DEBU[0001] [ssh] 10.252.198.12:22: "iptables": {        
DEBU[0001] [ssh] 10.252.198.12:22: "syncPeriod": "0s",  
DEBU[0001] [ssh] 10.252.198.12:22: "minSyncPeriod": "0s" 
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "ipvs": {            
DEBU[0001] [ssh] 10.252.198.12:22: "syncPeriod": "0s",  
DEBU[0001] [ssh] 10.252.198.12:22: "minSyncPeriod": "0s", 
DEBU[0001] [ssh] 10.252.198.12:22: "tcpTimeout": "0s",  
DEBU[0001] [ssh] 10.252.198.12:22: "tcpFinTimeout": "0s", 
DEBU[0001] [ssh] 10.252.198.12:22: "udpTimeout": "0s"   
DEBU[0001] [ssh] 10.252.198.12:22: }                    
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "kuberouter": {      
DEBU[0001] [ssh] 10.252.198.12:22: "autoMTU": true,     
DEBU[0001] [ssh] 10.252.198.12:22: "mtu": 0,            
DEBU[0001] [ssh] 10.252.198.12:22: "metricsPort": 8080, 
DEBU[0001] [ssh] 10.252.198.12:22: "hairpin": "Enabled", 
DEBU[0001] [ssh] 10.252.198.12:22: "ipMasq": false,     
DEBU[0001] [ssh] 10.252.198.12:22: "peerRouterASNs": "", 
DEBU[0001] [ssh] 10.252.198.12:22: "peerRouterIPs": ""  
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "nodeLocalLoadBalancing": { 
DEBU[0001] [ssh] 10.252.198.12:22: "type": "EnvoyProxy", 
DEBU[0001] [ssh] 10.252.198.12:22: "envoyProxy": {      
DEBU[0001] [ssh] 10.252.198.12:22: "image": {           
DEBU[0001] [ssh] 10.252.198.12:22: "image": "quay.io/k0sproject/envoy-distroless", 
DEBU[0001] [ssh] 10.252.198.12:22: "version": "v1.24.12" 
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "apiServerBindPort": 7443, 
DEBU[0001] [ssh] 10.252.198.12:22: "konnectivityServerBindPort": 7132 
DEBU[0001] [ssh] 10.252.198.12:22: }                    
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "podCIDR": "10.244.0.0/16", 
DEBU[0001] [ssh] 10.252.198.12:22: "provider": "kuberouter", 
DEBU[0001] [ssh] 10.252.198.12:22: "serviceCIDR": "10.96.0.0/12", 
DEBU[0001] [ssh] 10.252.198.12:22: "clusterDomain": "cluster.local" 
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "telemetry": {       
DEBU[0001] [ssh] 10.252.198.12:22: "enabled": true      
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "installConfig": {   
DEBU[0001] [ssh] 10.252.198.12:22: "users": {           
DEBU[0001] [ssh] 10.252.198.12:22: "etcdUser": "etcd",  
DEBU[0001] [ssh] 10.252.198.12:22: "kineUser": "kube-apiserver", 
DEBU[0001] [ssh] 10.252.198.12:22: "konnectivityUser": "konnectivity-server", 
DEBU[0001] [ssh] 10.252.198.12:22: "kubeAPIserverUser": "kube-apiserver", 
DEBU[0001] [ssh] 10.252.198.12:22: "kubeSchedulerUser": "kube-scheduler" 
DEBU[0001] [ssh] 10.252.198.12:22: }                    
DEBU[0001] [ssh] 10.252.198.12:22: },                   
DEBU[0001] [ssh] 10.252.198.12:22: "images": {          
DEBU[0001] [ssh] 10.252.198.12:22: "konnectivity": {    
DEBU[0002] [ssh] 10.252.198.12:22: "image": "quay.io/k0sproject/apiserver-network-proxy-agent", 
DEBU[0002] [ssh] 10.252.198.12:22: "version": "v0.1.4"  
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "pushgateway": {     
DEBU[0002] [ssh] 10.252.198.12:22: "image": "quay.io/k0sproject/pushgateway-ttl", 
DEBU[0002] [ssh] 10.252.198.12:22: "version": "1.4.0-k0s.0" 
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "metricsserver": {   
DEBU[0002] [ssh] 10.252.198.12:22: "image": "registry.k8s.io/metrics-server/metrics-server", 
DEBU[0002] [ssh] 10.252.198.12:22: "version": "v0.6.4"  
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "kubeproxy": {       
DEBU[0002] [ssh] 10.252.198.12:22: "image": "quay.io/k0sproject/kube-proxy", 
DEBU[0002] [ssh] 10.252.198.12:22: "version": "v1.28.7" 
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "coredns": {         
DEBU[0002] [ssh] 10.252.198.12:22: "image": "quay.io/k0sproject/coredns", 
DEBU[0002] [ssh] 10.252.198.12:22: "version": "1.11.1"  
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "pause": {           
DEBU[0002] [ssh] 10.252.198.12:22: "image": "registry.k8s.io/pause", 
DEBU[0002] [ssh] 10.252.198.12:22: "version": "3.8"     
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "calico": {          
DEBU[0002] [ssh] 10.252.198.12:22: "cni": {             
DEBU[0002] [ssh] 10.252.198.12:22: "image": "quay.io/k0sproject/calico-cni", 
DEBU[0002] [ssh] 10.252.198.12:22: "version": "v3.26.1-1" 
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "node": {            
DEBU[0002] [ssh] 10.252.198.12:22: "image": "quay.io/k0sproject/calico-node", 
DEBU[0002] [ssh] 10.252.198.12:22: "version": "v3.26.1-1" 
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "kubecontrollers": { 
DEBU[0002] [ssh] 10.252.198.12:22: "image": "quay.io/k0sproject/calico-kube-controllers", 
DEBU[0002] [ssh] 10.252.198.12:22: "version": "v3.26.1-1" 
DEBU[0002] [ssh] 10.252.198.12:22: }                    
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "kuberouter": {      
DEBU[0002] [ssh] 10.252.198.12:22: "cni": {             
DEBU[0002] [ssh] 10.252.198.12:22: "image": "quay.io/k0sproject/kube-router", 
DEBU[0002] [ssh] 10.252.198.12:22: "version": "v1.6.0-iptables1.8.9-1" 
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "cniInstaller": {    
DEBU[0002] [ssh] 10.252.198.12:22: "image": "quay.io/k0sproject/cni-node", 
DEBU[0002] [ssh] 10.252.198.12:22: "version": "1.1.1-k0s.1" 
DEBU[0002] [ssh] 10.252.198.12:22: }                    
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "default_pull_policy": "IfNotPresent" 
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "extensions": {      
DEBU[0002] [ssh] 10.252.198.12:22: "storage": {         
DEBU[0002] [ssh] 10.252.198.12:22: "type": "external_storage", 
DEBU[0002] [ssh] 10.252.198.12:22: "create_default_storage_class": false 
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "helm": {            
DEBU[0002] [ssh] 10.252.198.12:22: "concurrencyLevel": 5, 
DEBU[0002] [ssh] 10.252.198.12:22: "repositories": null, 
DEBU[0002] [ssh] 10.252.198.12:22: "charts": null       
DEBU[0002] [ssh] 10.252.198.12:22: }                    
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "konnectivity": {    
DEBU[0002] [ssh] 10.252.198.12:22: "adminPort": 8133,   
DEBU[0002] [ssh] 10.252.198.12:22: "agentPort": 8132    
DEBU[0002] [ssh] 10.252.198.12:22: }                    
DEBU[0002] [ssh] 10.252.198.12:22: }                    
DEBU[0002] [ssh] 10.252.198.12:22: },                   
DEBU[0002] [ssh] 10.252.198.12:22: "K0sVars": {         
DEBU[0002] [ssh] 10.252.198.12:22: "AdminKubeConfigPath": "/var/lib/k0s/pki/admin.conf", 
DEBU[0002] [ssh] 10.252.198.12:22: "BinDir": "/var/lib/k0s/bin", 
DEBU[0002] [ssh] 10.252.198.12:22: "CertRootDir": "/var/lib/k0s/pki", 
DEBU[0002] [ssh] 10.252.198.12:22: "DataDir": "/var/lib/k0s", 
DEBU[0002] [ssh] 10.252.198.12:22: "EtcdCertDir": "/var/lib/k0s/pki/etcd", 
DEBU[0002] [ssh] 10.252.198.12:22: "EtcdDataDir": "/var/lib/k0s/etcd", 
DEBU[0002] [ssh] 10.252.198.12:22: "KineSocketPath": "/run/k0s/kine/kine.sock:2379", 
DEBU[0002] [ssh] 10.252.198.12:22: "KonnectivitySocketDir": "/run/k0s/konnectivity-server", 
DEBU[0002] [ssh] 10.252.198.12:22: "KubeletAuthConfigPath": "/var/lib/k0s/kubelet.conf", 
DEBU[0002] [ssh] 10.252.198.12:22: "KubeletVolumePluginDir": "/usr/libexec/k0s/kubelet-plugins/volume/exec", 
DEBU[0002] [ssh] 10.252.198.12:22: "ManifestsDir": "/var/lib/k0s/manifests", 
DEBU[0002] [ssh] 10.252.198.12:22: "RunDir": "/run/k0s", 
DEBU[0002] [ssh] 10.252.198.12:22: "KonnectivityKubeConfigPath": "/var/lib/k0s/pki/konnectivity.conf", 
DEBU[0002] [ssh] 10.252.198.12:22: "OCIBundleDir": "/var/lib/k0s/images", 
DEBU[0002] [ssh] 10.252.198.12:22: "DefaultStorageType": "kine", 
DEBU[0002] [ssh] 10.252.198.12:22: "RuntimeConfigPath": "/run/k0s/k0s.yaml", 
DEBU[0002] [ssh] 10.252.198.12:22: "StatusSocketPath": "/run/k0s/status.sock", 
DEBU[0002] [ssh] 10.252.198.12:22: "StartupConfigPath": "/etc/k0s/k0s.yaml", 
DEBU[0002] [ssh] 10.252.198.12:22: "EnableDynamicConfig": false, 
DEBU[0002] [ssh] 10.252.198.12:22: "HelmHome": "/var/lib/k0s/helmhome", 
DEBU[0002] [ssh] 10.252.198.12:22: "HelmRepositoryCache": "/var/lib/k0s/helmhome/cache", 
DEBU[0002] [ssh] 10.252.198.12:22: "HelmRepositoryConfig": "/var/lib/k0s/helmhome/repositories.yaml" 
DEBU[0002] [ssh] 10.252.198.12:22: }                    
DEBU[0002] [ssh] 10.252.198.12:22: }                    
INFO[0002] [ssh] 10.252.198.12:22: is running k0s single version v1.28.7+k0s.0 
DEBU[0002] [ssh] 10.252.198.12:22: executing `sudo -s -- test -e /var/lib/k0s/pki/admin.conf 2> /dev/null` 
DEBU[0002] [ssh] 10.252.198.12:22: executing `sudo -s -- env "KUBECONFIG=/var/lib/k0s/pki/admin.conf" /usr/local/bin/k0s kubectl get -n kube-system namespace kube-system -o template={{.metadata.uid}}` 
DEBU[0002] [ssh] 10.252.198.12:22: f6d0f3ba-9a40-4731-aeea-179b84dc8e4b 
DEBU[0002] Preparing phase 'Validate facts'             
INFO[0002] ==> Running phase: Validate facts   
DEBU[0002] Preparing phase 'Download k0s binaries to local host' 
DEBU[0002] Preparing phase 'Upload k0s binaries to hosts' 
DEBU[0002] Preparing phase 'Download k0s on hosts'      
DEBU[0002] [ssh] 10.252.198.12:22: k0s version on target host is already v1.28.7+k0s.0 
DEBU[0002] Preparing phase 'Upload files to hosts'      
DEBU[0002] Preparing phase 'Install k0s binaries on hosts' 
DEBU[0002] Preparing phase 'Prepare ARM nodes'          
DEBU[0002] Preparing phase 'Configure k0s'              
DEBU[0002] using existing k0s config                    
DEBU[0002] [ssh] 10.252.198.12:22: executing `mktemp`   
DEBU[0002] [ssh] 10.252.198.12:22: /tmp/tmp.WAjV6pMRJA  
DEBU[0002] [ssh] 10.252.198.12:22: executing `mktemp 2> /dev/null` 
DEBU[0002] [ssh] 10.252.198.12:22: /tmp/tmp.TTSimMYnR4  
DEBU[0002] [ssh] 10.252.198.12:22: executing `cat > /tmp/tmp.TTSimMYnR4` 
DEBU[0002] [ssh] 10.252.198.12:22: writing 1654 bytes to command stdin 
DEBU[0002] [ssh] 10.252.198.12:22: executing `sudo -s -- install -D -m 0600 -- /tmp/tmp.TTSimMYnR4 /tmp/tmp.WAjV6pMRJA` 
INFO[0002] [ssh] 10.252.198.12:22: validating configuration 
DEBU[0002] [ssh] 10.252.198.12:22: comparing k0s version v1.28.7+k0s.0 with >= v1.23.1+k0s.0 
DEBU[0002] [ssh] 10.252.198.12:22: comparison result true 
DEBU[0002] [ssh] 10.252.198.12:22: executing `sudo -s -- /usr/local/bin/k0s config validate --config "/tmp/tmp.WAjV6pMRJA"` 
DEBU[0002] [ssh] 10.252.198.12:22: configuration will not change 
DEBU[0002] [ssh] 10.252.198.12:22: executing `sudo -s -- rm -f -- /tmp/tmp.WAjV6pMRJA 2> /dev/null` 
DEBU[0002] Preparing phase 'Restore cluster state'      
DEBU[0002] Preparing phase 'Run Before Apply Hooks'     
DEBU[0002] Preparing phase 'Initialize the k0s cluster' 
DEBU[0002] Preparing phase 'Install controllers'        
DEBU[0002] Preparing phase 'Install workers'            
DEBU[0002] Preparing phase 'Upgrade controllers'        
DEBU[0002] UpgradeControllers phase prep starting       
DEBU[0002] 1 controllers in total                       
DEBU[0002] UpgradeControllers phase prepared, 0 controllers needs upgrade 
DEBU[0002] Preparing phase 'Upgrade workers'            
DEBU[0002] 0 workers in total                           
DEBU[0002] UpgradeWorkers phase prepared, 0 workers needs upgrade 
DEBU[0002] Preparing phase 'Reset workers'              
DEBU[0002] 0 workers in total                           
DEBU[0002] ResetWorkers phase prepared, 0 workers will be reset 
DEBU[0002] Preparing phase 'Reset controllers'          
DEBU[0002] 1 controllers in total                       
DEBU[0002] ResetControllers phase prepared, 0 controllers will be reset 
DEBU[0002] Preparing phase 'Run After Apply Hooks'      
DEBU[0002] Preparing phase 'Release exclusive host lock' 
INFO[0002] ==> Running phase: Release exclusive host lock 
DEBU[0002] [ssh] 10.252.198.12:22: stopped lock cycle, removing file 
DEBU[0002] [ssh] 10.252.198.12:22: executing `sudo -s -- rm -f -- /run/lock/k0sctl 2> /dev/null` 
DEBU[0002] Preparing phase 'Disconnect from hosts'      
INFO[0002] ==> Running phase: Disconnect from hosts 
INFO[0003] ==> Finished in 2s                  
INFO[0003] k0s cluster version v1.28.7+k0s.0 is now installed 
INFO[0003] Tip: To access the cluster you can now fetch the admin kubeconfig using: 
INFO[0003]      k0sctl kubeconfig

What is the disposition of your keys?

I copy the multipass key from /var/snap/multipass/common/data/multipassd/ssh-keys/id_rsa to my home directory and change the ownership to be my logged in user. This is the same setup I have used for previous versions of k0sctl. I am explicitly setting the path in my config file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@kke @twz123 @nwneisen @jadolg