Allowing relocation of repo downloaded binaries

[johnccfm]

Now that it's technically possible due to the Jinja templating, could we allow relocation of the downloaded binaries without having to fork the repo and maintain patches?

For example, lines like:

installer: 'salt://win/repo-ng/cpu-z/cpu-z_1.71.1-setup-en.exe'

Could become:

installer: 'salt://{{ salt["pillar.get"]("repo_binary_root", "win/repo-ng") }}/cpu-z/cpu-z_1.71.1-setup-en.exe'

I don't think there's a way to fetch the actual configuration value without enabling the option that makes the master config visible on minions.

If people agree this is a reasonable idea I'd be happy to send a PR.

[TheBigBear]

@johnccfm not sure I immediately get what this would mean in practice, and what the consequences would be for the majority of the repo-ng consumers?

Could you please elaborate a bit more?

[johnccfm]

The intention is that if you do nothing, nothing will change - the {{ }} part should default to /win/repo-ng which is the default path.

If like me you use a slightly different directory structure, or you want to put all your binary files somewhere else, you currently have to create your own fork (not a bad idea) and edit every .SLS file with a salt:// path in it, and maintain those changes going forward.

With the above style used every time a salt:// path occurs in a repo SLS file, I could just create a pillar item, e.g.

repo_binary_root: win/repo-ng/binaries

And my desired path would be used.

[twangboy]

I assume this is to save disk space on your master? What do you think @UtahDave?

[TheBigBear]

@johnccfm is it really to do with saving disk space on the master? I thought/understood it was to have your own way (or your own directory actually) on the master where you saved your sls installers, basically a different place from where the project assumes they would be?

[johnccfm]

It's not about saving space, for me at least. It's just about flexibility. I have my master set up like this:-

# Location of the repo on the master:
winrepo_dir_ng: '/srv/salt/states/base/win/repo-ng'

#
# List of git repositories to include with the local repo:
winrepo_remotes_ng:                                                                                                                                                                                                                            
  - 'https://github.com/fastmarkets/salt-winrepo-ng.git'
  - 'https://github.com/fastmarkets/fm-winrepo-ng.git'

So to take JRE 8 as an example, the most likely path I'm going to have for the binary would not be:-

installer: 'salt://win/repo-ng/jre8_x86/jre-8u60-windows-i586.exe'

but

installer: 'salt://win/repo-ng/salt-winrepo-ng/jre8_x86/jre-8u60-windows-i586.exe'

That would mean I'll need to edit the jre8.sls file, and maintain the changes.

Also, a downside of putting the binaries in subdirectories of the git repo is that they'll show up as changes (because there's no .gitignore). Again, I could add a .gitignore, but if I keep the binaries in a separate tree that avoids the whole problem.

Maybe there's a better way to solve this that I'm missing?

[TheBigBear]

@johnccfm ok, the whole differentiation between 'x86' an 'x64' in sls files is mostly a pre 2015.8 release issue that never really actually worked pre Berryllium anyway.

Would it help you (all?) if we simply worked on rationalising the 'salt://' paths in the 'sls' files to

installer: 'salt://win/repo-ng/salt-winrepo-ng/jre8/jre-8u60-windows-i586.exe'

So that the binaries for both the 'x86' and the 'x64' 'sls' file would also live in the 'sls' file repo, simply in the same directory as the corresponding 'sls' file itself?

Or would that create issues for git operations, or the salt+git that is built-in?

I presume saltstack don't actually want to distribute binary installers from their git repo? As actually commuting the installer binaries into the GitHub git repo would not be legal anyway in a number of cases.

Or should we have a somewhat standard directory structure in parallel for binaries, if mixing both content and binaries is a bad idea?

Just thinking out loud, here.

But does this all still hold up when pygit2 is used and the repo directory names are nothing but hashes? (or was that only temporary anyway and then changed for the final release of 2015.8.0?)

[ghost]

@TheBigBear, @johnccfm,
I think this would be a very nice idea to separate the installer binaries from the installation definitions. This would allow e.g. to have the installer directory be visible to me (as admin) as a file share to easily drop the files from my Windows machine into it without messing around with the salt states directory.
And logic (aka sls files) and content (aka binaries) is separate.

• update the sls definitions via git workflow
• update the manual-download installer files via file share

The solution demonstrated in @johnccfm 's first post seems to be ok.
Changing the default directory structure would be more stable but requires all existing users to move all installer dirs from repo-ng to the newly defined binary installer directory.
Directory tree under salt://win/ could be something like:

• repo
• repo-ng
• installers

gqgunhed

[johnccfm]

Any chance of some movement on this? I've just rebuilt my salt masters, using a lot more standard setup. However, because I have two repos (the salt-ng one, and my own), the windows repo is in salt://win/repo-ng/salt-winrepo-ng even without me changing any of the locations. This means that just by having two git repos, the locations are all wrong.

I also still think that having the binaries in the git controlled directories is distasteful, but that's possibly more personal than objective fact!

I'm happy to submit a pull request based on the above, if everyone is happy with the idea. It seems to avoid breaking things for anyone while allowing easy changing of the location for those that want to.

[UtahDave]

@johnccfm can you put your binary installer files directly on the master's file system instead of putting them in your git repo?