Set ETCD_UNSUPPORTED_ARCH on ARM controller nodes #184

erdii · 2021-08-09T16:54:57Z

This PR explores a way to enable cluster creation on arm nodes.

Alternative to this implementation: Add systemd drop-in file via hosts[].files but this is hardcoding node architectures in the cluster configuration

# arm64/override.conf
[Service]
Environment=ETCD_UNSUPPORTED_ARCH=arm64

# arm/override.conf
[Service]
Environment=ETCD_UNSUPPORTED_ARCH=arm

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
spec:
  hosts:
  - ssh:
      address: 192.168.35.12
      user: root
    files:
    # on arm64: use arm64/override.conf
    # on arm: use arm/override.conf
    - src: arm/override.conf
      dstDir: /etc/systemd/system/k0scontroller.service.d
      perm: 0644

kke · 2021-08-10T07:22:25Z

phase/inject_arm_fixes.go

+func (p *InjectARMFixes) Prepare(config *config.Cluster) error {
+	p.Config = config
+
+	hosts := p.Config.Spec.Hosts.Filter(func(h *cluster.Host) bool {


I suppose this is only needed for controllers

Suggested change

hosts := p.Config.Spec.Hosts.Filter(func(h *cluster.Host) bool {

hosts := p.Config.Spec.Hosts.Filter(func(h *cluster.Host) bool {

if h.Role == "worker" {

return false

}

kke · 2021-08-10T07:25:04Z

phase/inject_arm_fixes.go

+`, h.Metadata.Arch)
+
+	name, err := func() (string, error) {
+		d, err := ioutil.TempDir("", "k0scontroller.service.d.")


ioutil package is being phased out, os.MkdirTemp replaces ioutil.TempDir.

🤔 Maybe just host.Configurer.WriteFile(....) it instead of wrestling with tempfiles and host.Files.

Signed-off-by: erdii <[email protected]>

erdii · 2021-08-10T08:39:54Z

Heyo @kke 👋
Good catches! I've (again! 😀) incorporated all of your suggestions. :)

Disclaimer: I don't have time to test h.Configurer.WriteFile right now.

kke · 2021-08-11T10:59:27Z

Another problem, only works for systemd.

kke · 2021-08-11T11:56:00Z

Ok I broke it. Fixing it for openrc and relatives seems to require some more changes to rig.

kke · 2021-08-12T07:16:27Z

phase/prepare_arm.go

+
+	p.hosts = p.Config.Spec.Hosts.Filter(func(h *cluster.Host) bool {
+		arch := h.Metadata.Arch
+		return h.Role != "worker" && (arch == "arm" || arch == "arm64")


Note to self: I think this needs aarch64 as well.

Possibly armv7l and some others... maybe just strings.HasPrefix with arm and aarch, then figure out 32/64bit separately to choose if etcd_unsupported_arch should be set to arm or arm64.

https://etcd.io/docs/v3.5/op-guide/supported-platform/ here's the list of supported platforms and which ones need the env.

I think it needs to print a warning too.

kke · 2021-08-12T08:05:32Z

I think it needs a ServiceEnvironment(s string, map[string]string) in rig.

For systemd it can create such override.conf thing as done here and for openrc it can create a /etc/conf.d/<service-name> file (syntax just simple FOO="bar") which openrc will eval before starting a service.

kke · 2021-08-12T08:06:17Z

I think this also needs to be considered when doing a k0sctl reset.

…m-support

kke · 2021-08-16T08:27:13Z

phase/prepare_arm.go

@@ -0,0 +1,58 @@
+package phase


I just do not understand what is wrong with this file now.

Vim gives me:

go vet gives me:

# github.com/k0sproject/k0sctl/cmd cmd/apply.go:79:5: undefined: phase.PrepareArm # github.com/k0sproject/k0sctl/cmd vet: cmd/apply.go:79:11: PrepareArm not declared by package phase

any ideas, @erdii @jnummelin @jasmingacic ?

hmmmmmmmmmmm, is it the _arm in filename? maybe.

Yes it is.

Files with os and architecture specific suffixes automatically follow those same constraints, e.g. name_linux.go will only build on linux, name_amd64.go will only build on amd64. This is the same as having a //+build amd64 line at the top of the file

kke · 2021-08-16T09:29:54Z

Maybe it works now. @erdii ptal

erdii · 2021-08-17T09:18:41Z

Wow thanks! I'll take a look in the next couple of days. Please ping me if I forget to do so! 😅

erdii

Just tried this on my local setup with 3 odroid-hc4 and 3 raspberry pi 4 boards and it worked like a breeze! :)

Thank you very much @kke !

Edit: all my machines are on archlinuxarm so i can't speak for non-systemd distributions

erdii · 2021-08-18T14:51:53Z

phase/arm_prepare.go

+func (p *PrepareArm) etcdUnsupportedArch(h *cluster.Host) error {
+	var arch string
+	switch h.Metadata.Arch {
+	case "aarch32", "arm32", "armv7l", "armhfp", "arm-32":


Where is that list from? Maybe it makes sense to link the source here in a comment.

Several sources, I googled around, there seems to be a huge load of different ones, it's possible this is still incomplete. I think the most important ones here are aarch32 and armv7l which I think are commonly seen in raspberries.

erdii · 2021-08-18T14:55:52Z

phase/arm_prepare.go

+	}
+
+	log.Warnf("%s: enabling ETCD_UNSUPPORTED_ARCH=%s override - you may encounter problems with etcd", h, arch)
+	h.Environment["ETCD_UNSUPPORTED_ARCH"] = arch


What did h.Environment previously do? Will this also set /etc/environment?

It goes both ways, this way also all the HTTP_PROXY stuff etc that user might have configured will get into the env override file, there's a possibility it will make them end up in k0s process more consistently.

erdii · 2021-08-18T15:06:15Z

Just had these questions - other than that LGTM!

erdii · 2021-08-18T15:06:38Z

Maybe h.Environment needs a documentation update to clarify what exactly it does after this change? 🤔

erdii · 2021-08-19T13:56:15Z

Actually now that i think of it I'm afraid that the service override file could have stayed in place during my run of k0sctl reset - I'll have to test another time while ensuring that the file is actually gone.

erdii · 2021-08-19T15:10:35Z

I did just let it run in the background of a meeting and there are two issues:

Due to the use of backtick-delimited strings the ServiceEnvironmentContent functions in rig return strings with verbatim \n in them. -> This one can be fixed by replacing the backticks with double-quotes which I verified on my local system.
A systemctl daemon-reload is needed when systemd already knew about the k0scontroller service from earlier. I'm not quite sure where to add it. I'd tend to add this logic inside the initsystem package. 🤔

What do you think?

erdii · 2021-08-19T15:26:28Z

UpdateServiceEnvironment could call DaemonReload as the final step

erdii · 2021-08-19T15:38:51Z

Proposed fix in rig: k0sproject/rig#40

kke · 2021-08-23T07:20:19Z

The rig fixes are available as v0.4.4-beta.2 currently, I'm in the middle of fixing the sudo stuff in #192 so it will bring in some other changes, should have the v0.4.4 out any time now.

erdii · 2021-08-23T08:37:01Z

Observation: when the controller init phase fails, a subsequent reset it not cleaning up service environment

erdii · 2021-08-23T11:29:47Z

Another observation: the empty folder /etc/systemd/system/k0scontroller.service.d stays in place after k0sctl reset

kke · 2021-08-23T11:47:47Z

Clean-up improved

kke · 2021-08-23T12:14:55Z

The empty dir can be fixed separately k0sproject/rig#42

kke · 2021-08-23T12:15:23Z

Merging to roll out a beta

erdii · 2021-08-23T12:30:31Z

Thank you! I will test it asap :)

erdii · 2021-08-23T14:58:39Z

works 👍 thank you very much @kke

dwarf-king-hreidmar · 2021-09-11T05:42:41Z

Is there a binary for linux arm or should i build it myself? Attempting to try on RPis

kke · 2021-09-13T07:16:15Z

Is there a binary for linux arm or should i build it myself? Attempting to try on RPis

I thought there was, but looks like there isn't. I'll add it to the build scripts.

kke · 2021-09-13T13:17:59Z

@dwarf-king-hreidmar Linux-arm64 binaries now in https://github.com/k0sproject/k0sctl/releases/tag/v0.10.2

erdii marked this pull request as ready for review August 9, 2021 17:02

erdii mentioned this pull request Aug 9, 2021

Add ArchLinux support #183

Merged

kke reviewed Aug 10, 2021

View reviewed changes

set ETCD_UNSUPPORTED_ARCH on arm nodes

1bab18f

Signed-off-by: erdii <[email protected]>

erdii force-pushed the add-arm-support branch from f654960 to 1bab18f Compare August 10, 2021 08:16

use h.Configurer.WriteFile to upload arm override file

824a291

Signed-off-by: erdii <[email protected]>

Finetuning

e5409ea

kke reviewed Aug 12, 2021

View reviewed changes

kke mentioned this pull request Aug 16, 2021

Functions for service environment override k0sproject/rig#34

Merged

kke added 5 commits August 16, 2021 11:10

Use rig 0.3.24 for service env management

1230745

Merge branch 'main' into add-arm-support

bcf9e64

Whoops, left out the arm

063fd2e

Merge branch 'add-arm-support' of github.com:erdii/k0sctl into add-ar…

b1b765c

…m-support

Missing return

ea62bda

kke reviewed Aug 16, 2021

View reviewed changes

kke changed the title ~~enable ETCD_UNSUPPORTED_ARCH on arm nodes~~ Set ETCD_UNSUPPORTED_ARCH on arm nodes Aug 16, 2021

kke added 4 commits August 16, 2021 11:57

Update go.mod

652a925

Try trickery

2d477d3

Ok the file cant be called _arm

194671e

ineffassign

6fc18d2

kke changed the title ~~Set ETCD_UNSUPPORTED_ARCH on arm nodes~~ Set ETCD_UNSUPPORTED_ARCH on ARM controller nodes Aug 16, 2021

kke added the enhancement New feature or request label Aug 16, 2021

kke mentioned this pull request Aug 16, 2021

Allow setting environment in "k0s install" k0sproject/k0s#1054

Closed

Merge branch 'main' into add-arm-support

7870739

erdii commented Aug 18, 2021

View reviewed changes

kke added 2 commits August 23, 2021 14:20

Merge branch 'main' into add-arm-support

bc4fa62

Clean up service environment during reset even if k0s not running

40a9ecc

More cleaning up

f76f476

Ok, after hooks were interntionally even if it errored

7a95c7d

kke merged commit d97cc1f into k0sproject:main Aug 23, 2021

erdii deleted the add-arm-support branch August 23, 2021 12:30

kke mentioned this pull request Sep 13, 2021

Add linux arm64 build target #209

Merged

Set ETCD_UNSUPPORTED_ARCH on ARM controller nodes #184

Set ETCD_UNSUPPORTED_ARCH on ARM controller nodes #184

Conversation

erdii commented Aug 9, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erdii commented Aug 10, 2021

kke commented Aug 11, 2021

kke commented Aug 11, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kke commented Aug 12, 2021

kke commented Aug 12, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kke commented Aug 16, 2021

erdii commented Aug 17, 2021

erdii left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erdii commented Aug 18, 2021

erdii commented Aug 18, 2021

erdii commented Aug 19, 2021

erdii commented Aug 19, 2021 • edited Loading

erdii commented Aug 19, 2021

erdii commented Aug 19, 2021

kke commented Aug 23, 2021

erdii commented Aug 23, 2021

erdii commented Aug 23, 2021

kke commented Aug 23, 2021

kke commented Aug 23, 2021

kke commented Aug 23, 2021

erdii commented Aug 23, 2021

erdii commented Aug 23, 2021

dwarf-king-hreidmar commented Sep 11, 2021

kke commented Sep 13, 2021

kke commented Sep 13, 2021 • edited Loading

erdii commented Aug 9, 2021 •

edited

Loading

erdii left a comment •

edited

Loading

erdii commented Aug 19, 2021 •

edited

Loading

kke commented Sep 13, 2021 •

edited

Loading