reproducible build [Documentation]

[krux02]

This is just a Documentation request, because I am not 100% sure how to get this goal. My project here has problems with reproducibility, and I don't know how to ensure that the project can always be build by other people in the same way as it can be build on my system. Things that make it hard for me are that I constantly use the latest development version of Nim, and I submit patches to the libraries I use. This means the latest version with a version tag on nimble will often not be enough to build the master branch of my project.

I would be nice, if the documentation page would link to a chapter about the build system and how to create a reproducible project/library. Also on how to ensure that a project is reproducible and not only working on the current system but not on other system.

[Araq]

Looking for my 'nawabs' project? :-)

[andreaferretti]

This is of paramount importance to me as well. There are two things that can be improved by documentation/culture alone:

• many libraries suggest that they are available using nimble install. In my opinion, nimble install should be used only to get executables - dependencies of a project should be specified in a .nimble file, together with the version required. This will diminish reliance on globally installed libraries
• adding a dependency on a particular version of a library is not of much use, unless nimble is actually able to find that version. This means that every time one changes version in the nimble file, a git tag with the same version should be added. For instance, I am not able to depend on opengl-sandbox 0.1.0, because it is not tagged.
• it goes without saying that if one introduces changes in a library, this should be accompanied by a version bump often enough that one can actually refer to a version where the change is available (many project just never bump version, denying the whole utility of having a version to refer to)

I think that every documentation (official nim documentation, the nimble github page, tutorials...) should strive to emphasize the use of nimble files (as opposed to global installs). It takes a few second to bump version because it is just a git tag and a line in the nimble file - let us take advantage of this!

@Araq I looked at nawabs and it seems interesting, but it is not yet clear to me what the workflow is like. Is there a minimal example project? In particular, it is not clear to me how and where to specify dependencies for a project. Say I am writing a web app with jester - how would I make the jester dependency appear in the workspace? nawabs build jester? there is not anything to build, I just need to have it available in my project. Maybe it is trivial, but I cannot figure it out from the README

[zielmicha]

For my projects, I use nimenv, which produces shell scripts like this. The script downloads, builds Nim and clones dependencies (tagged by specific commit hashes). For development, it relies on manually managed repositories (nimenv localsetup). It's much simpler than newabs.

[andreaferretti]

Does it handle recursive dependencies? There are essential when the ecosystem grows.

Many package managers handle recursive dependencies, then generate a lock file that lists the resulting dependencies at a particular commit, so that rebuilding will give the same result

[dom96]

This is something that Nimble needs to gain support for via lock files (relevant issue nim-lang/nimble#127).

In my opinion, nimble install should be used only to get executables - dependencies of a project should be specified in a .nimble file, together with the version required. This will diminish reliance on globally installed libraries

I disagree.

adding a dependency on a particular version of a library is not of much use, unless nimble is actually able to find that version. This means that every time one changes version in the nimble file, a git tag with the same version should be added. For instance, I am not able to depend on opengl-sandbox 0.1.0, because it is not tagged.

Yes, a version bump should always accompany a git tag. You can also depend on commit hashes, but that has the potential to open its own can of worms.

[andreaferretti]

I disagree

Can you explain why? The issue I see with using - say - nimble install jester is that now there is a single version of jester, and projects without a nimble file will use that particular version. This makes for an easy start, but has all the problems of global things.

Namely, I start writing some web applications using jester. A few months from now, jester gets a new feature. I update to use the new feature, and everything looks ok. Until I try to build some old project, that maybe is incompatible with that version of jester. Now what? In fact, I don't even know anymore what was the correct jester version to use. Now, for a single dependency which tries hard not to break backward compatibility, this could be easily fixed, but with more dependencies, things can go easily wrong.

Also, consider that I may know of this problem and avoid updating jester for this. But then, I install some other library which depends on a newer version of jester, and my efforts are vanified (not to pick on jester, it is just an example).

Also, without a single place to list dependencies, it will be harder to rebuild it when I change my laptop (what Nim libraries did I have installed globally?)

For all these reasons, I find relying on globally installed libraries a bad practice (especially since there is a simple alternative). And I would avoid even mentioning this way to beginners, lest they learn an impractical way of working with dependencies which will give rise to a brittle ecosystem.

Since you disagree, I would be interested in knowing what advantages do you see in using nimble install with respect to using a nimble file

[dom96]

The issues you describe should disappear once lock files are implemented for Nimble.

You seem to imply that it is possible to avoid nimble install, but it is not. When you nimble install an executable then its dependencies will also be installed globally. So I don't think advising to never install dependencies via nimble install is wise. This global installation of packages has been discussed thoroughly in this issue.

Certainly when developing a package or application you should be using nimble build and/or nimble c to test that the dependencies you have specified in your .nimble file are correct.

[krux02]

Yes I looked up the documentation on nimble:

> nimble --help | grep build
  build                           Builds a package.

Then I found some information in "How I start":

We could compile it as we did before for hello.nim, but since we already set our package up to include the brainfuck binary, let’s make  nimble do the work:

That's all documentation I found. And the issue is that this is not enoguh. Nimble needs it's own chapter here: https://nim-lang.org/documentation.html

[dom96]

@krux02 have you read this? https://github.com/nim-lang/nimble#nimble-

[Araq]

I looked at nawabs and it seems interesting, but it is not yet clear to me what the workflow is like. Is there a minimal example project? In particular, it is not clear to me how and where to specify dependencies for a project. Say I am writing a web app with jester - how would I make the jester dependency appear in the workspace? nawabs build jester? there is not anything to build, I just need to have it available in my project. Maybe it is trivial, but I cannot figure it out from the README

It processes .nimble file to determine dependencies.

[andreaferretti]

@Araq thank you, that was not clear!

@dom96 I think that one issue is that nimble install is one of the first things mentioned, while creating a package is mentioned much later. And it is not clear that this applies to projects that are not necessarily meant to be published.

About the lock file and so on: I agree that having a lock file is useful, but not relying on global libs is a first step at least. I agree that installing executable may involve installing library dependencies globally, I am just saying that relying on said global libraries should not be encouraged for development.

For instance, out of the latest 10 libraries published on nimble (one of which is my own), the following 4 suggest a global nimble install:

• http://fragworks.io/
• https://github.com/euantorano/tempdir.nim
• https://github.com/tulayang/nimnode
• https://github.com/euantorano/sysrandom.nim

[krux02]

can you add a link to both nawbs and nimble documentation to the documentation root page here:
https://nim-lang.org/documentation.html

[dom96]

@andreaferretti ahh, I see what you mean and I agree. Package readmes/websites should suggest adding the package name to your project's .nimble file as a dependency and not to install it.

@krux02 can you? :)

[krux02]

What should I edit?

https://github.com/nim-lang/website/blob/master/jekyll/documentation.html
https://github.com/nim-lang/Nim/blob/devel/web/documentation.rst

[dom96]

The first. And IMO nawabs shouldn't be mentioned.

[Araq]

This is not a Nim feature request.

[andreaferretti]

Agreed, this is not a Nim feature request, but here we are, more than two years later, and no clear sign of progress.

Is nawabs still the way you recommend?

Lock files are still not on nimble, Status has issues preventing them to use nimble, we have a competing proposal, and just today I tried using a project (still in development, I didn't expect to make it work smoothly) where I was unable to track dependencies.

I think we should do something, anything, before the ecosystem becomes too brittle

[Araq]

Is nawabs still the way you recommend?

No, it hasn't kept up with Nimble's development. I'm considering to revive it. ;-)

I think we should do something, anything, before the ecosystem becomes too brittle

Agreed, here is what I do: I make my project.nim.cfg contain

--noNimblePath
--path:depA/src
--path:depB
--path:depC/src

But that's only a part of the solution, of course.

[dom96]

Let's implement lock files. It honestly shouldn't be that difficult to implement a basic version of them.

All these other proposals are just wasting more time. We've somehow gone from local project dependencies to "let's have a command that appends --path to the nim.cfg file", and yet @Araq apparently still wants both.

It just boggles my mind.

[Araq]

Nimble and Atlas both support lockfiles now.