Fixes for http_proxy, find_os.sh and Salman's changes

[jimdowling]

No description provided.

[jimdowling]

I am going to merge this, if that's ok. This branch has been our released karamel for the last 6 weeks.

[SirOibaf]

Additionally, I see the PR in setup-chef which already starts with gem server. Why having it in both places?

[jimdowling]

All changes for gem-server support are now here in karamel. I have tested community and enterprise with a gem server and it works. This branch is currently our 0.6 release.

[SirOibaf]

No, it doesn't work. I'm honestly trying to help here, all the points I listed above are valid and you didn't address those. The fact that you made a release of this branch it's irrelevant.

Again. The 2.5.0 directory doesn't exists. You don't see it because the gem server doesn't fail:

root@hopsworks0:/home/vagrant# /opt/chefdk/embedded/bin/gem server --port 12335 --dir /this/directory/does/not/exists
Server started at http://0.0.0.0:12335
Server started at http://[::]:12335

Starting the server on make_solo, means that all the machines run a gem server.

Moreover, not all the gems are available and the recipes fall back transparently to the upstream repository. This means that when you actually need the gem server ( i.e. in air-gapped installations) karamel will be broken.

Recipe: kzookeeper::default
  * kzookeeper[3.4.7] action install
  Recipe: <Dynamically Defined Resource>
    * chef_gem[zookeeper] action install[2020-08-30T10:50:52+00:00] WARN: failed to find gem zookeeper (>= 0) from [http://127.0.0.1:12335]

      - install version  of package zookeeper
    * chef_gem[json] action install

You can listen to me or not, at this point, as I said above, I'm done with the complete rewrite. Also for internal development and enterprise customers we use a separate build of Karamel, so we won't be affected by this PR.

[jimdowling]

This branch installs chefdk-3.7.23-1.el7.x86_64.rpm, which is version 2.5.0
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/

You're right about the gemserver on every host being a bad idea. I can change it to only start on the head host. In hopsworks-installer.sh, karamel is always run on the head server, so karamel itself could start the gemserver process (with sudo) as a child process. The advantage here is that when karamel exits, the gemserver also exits.

I didn't know about the fallback behaviour of the gem server. However, the only fix required is to add any gems to the directory (on the head node). This could again be done in Karamel with sudo command.
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/
I can look through the cookbooks, i really don't think there are many others apart from ZK. We have never added any gems, afaik, to cookbooks, so it is not something that should break unless we upgrade a cookbook that upgrades its gems.

[jimdowling]

I fixed the problems identified in the last comment.
I used make_solo_rb.sh to start the gem server, but it only starts on the same host as Karamel. It does this by checking if the karamel dir is found on the local host. There are only 2 gems used by our installation - both were stored in the cache during installation. The zookeeper one and a json one. They are now distributed with karamel and copied into the gemserver cache path. I think i have covered the issues you raised.