Support for nimble dependencies

[endragor]

Building complex projects often requires libraries that may not be found in Nim's stdlib. Currently there is no good way (known to me) to use external libraries from nakefile - a workaround is to have a separate tool built and launched from nakefile. But it would make build process simpler if the intermediary is removed.

A simple way to implement this is for nake to look for nakefile.nimble and, if it is present, run nimble c nakefile.nim instead of nim c nakefile.nim.

[yglukhov]

I'm not sure I get your point. Why not just list the needed libs in your nimble file, and import them to the nakefile? E.g. I'm using jester in my nakefile.

[endragor]

Which nimble file do you mean? Say, you are building your project for the first time - then there are no packages in ~/.nimble/pkgs/, so nake build is going to fail, because it compiles nakefile.nim without caring about nimble dependencies.

I don't think it makes sense to depend on presence of certain packages in the local repository if they are not explicitly specified as dependencies of the nakefile. Frankly, I think the current behaviour of nim(ble) with considering the local packages to be in path of every compiled project doesn't make sense - a package should only be added to path if there is an explicit dependency on it. That helps to have reproducible builds and ensure that if something compiles on one machine then it will also compile on any other machine.

[yglukhov]

IMO, you're mixing several issues here.

• nakefile should be allowed to import third party packages. That is done by:

  1. List your nakefile dependencies in the nimble file along with your project dependencies
  2. Install all the dependencies: nimble install -dy
  3. Run nake as you normally would, at this point all the needed dependencies are in ~/.nimble/pkgs/.

• More proper dependency management by nimble in terms of dependency tree reproducibility. Honestly I don't know what to say here, except it really needs to be addressed somehow. Relevant discussions: Add NIMBLE_DIR environment variable nim-lang/nimble#340, Locking of dependencies nim-lang/nimble#127, reproducible build [Documentation] nim-lang/Nim#5776, Project-local package dirs with precedence nim-lang/nimble#131, and more =)

[endragor]

Yes, I know I also mixed the nimble dependency management topic into this, but the original post was about the first bullet in your comment. I find what you describe to be a workaround rather than final solution:

• One command is not enough to build a project, CI and developers have to do both nimble install -dy and nake <something> (assuming they already have nake).

• It makes sense to clearly separate build dependencies from project dependencies (if nimble managed dependencies properly, this would likely even be mandatory). Build process may require libraries not required by the project itself and in the worst case they may require different versions of the same library. Generally speaking, build code is a completely independent thing from the project it builds, so their dependencies should be managed separately. And it works that way in ecosystems of other languages for similar reasons.

What do you think?

[yglukhov]

One command is not enough to build a project

Thats not an issue to me, but I agree that making everything needed in one single command would be fancier, if that doesn't come at the cost of build times. I guess nimble will want to have some command like validateDeps or smth, that should in most cases check paths against its lockfile, and be blazing fast, or download/install those packages that failed validation.

It makes sense to clearly separate build dependencies from project dependencies

Some while ago we discussed with @dom96 that there should be "develop" dependencies that would be installed for your root package, but not its dependencies. Cargo does this similarly. That looks like should cover your case.

However I'm not sure about the scenario when your build tools dependencies would conflict with project dependencies. All of those are managed within the same repo anyway, so the maintainer should likely have no problems with keeping those versions coherent.

[endragor]

Cargo has a notion of build scripts and you can specify [build-dependencies] in the manifest. And you end up using one command to build your project. It is similar in other mature ecosystems.

What I'm talking about is a way to specify build dependencies when using nake. Nimble by itself cannot replace Cargo here, because it doesn't really have a notion of build scripts (yes, the nimble file is NimScript, but it's too restrictive comparing to Nim and cannot replace it when it comes to complex build logic). I think whatever is implemented on nimble side, it cannot help with nakefile's dependencies, because nimble is not aware of nake.

The idea I expressed in the original post is to make nake aware of nimble. The change is quite straightforward and simple - try to locate nimble file near the nakefile and, if it exists, use nimble to compile the nakefile. Do you think there could be problems with such approach? It looks conceptually sound to me.

[yglukhov]

I was talking about Cargo development dependencies, please have a look at that.

I'm not sure that your suggestion (namely a separate nimble file) is conceptually correct/consistent because (a) i think there should be one nimble file per package (arguable) and (b) i have my nakefiles next to my nimble files, and thus I can not introduce a new nimble file because nimble allows only one nimble file per dir, also it is likely that different tools may rely on searching of nimble file by recursively going upwards, and thus they can hit your nakefile nimble file which you may not want. Does that make sense?

[endragor]

I agree that there should be one nimble file per package. But nimble c nakefile.nim will work both for that (standard) case and my case, in which nakefile orchestrates a collection of tools and client/server apps, each having its own nimble file. So the suggestion I made will work for you too, if we make nake look for any .nimble file, not just nakefile.nimble. If later nimble adds support for dev dependencies, then nake will use a flag that tells nimble to use dev dependencies.
Thoughts?

[yglukhov]

But nimble c nakefile.nim will work

Hrm, I was not aware of that. If that's true I see nothing wrong. To summarize: compile nakefile.nim with nimble c nakefile.nim if a nimble file is found next to the nakefile. Otherwise use the current logic, that is nim c nakefile.nim. Is that precise? If so, you can count me in =)

[endragor]

Yes, that's precise, thank you. Should we also consider looking for .nimble file upwards to cover the case you described?

I've noticed nimble c uses --noNimblePath and specifies precise --path arguments for each dependency (makes sense), which means it won't find nake unless it's a dependency. So nake path should first be retrieved from nimble path nake and then passed to nimble c via --path argument.

[yglukhov]

Should we also consider looking for .nimble file upwards to cover the case you described?

I don't have a strong opinion regarding that one. Should we? =) Probably that would be more intuitively consistent behavior?

So nake path should first be retrieved ...

Should not nake be listed in the nimble file as a dependency because it actually is? =)

[endragor]

Yeah I guess you are right, it makes sense to explicitly specify nake as a dependency.

As about going upwards - I think it makes sense to follow this logic: "if nimble c works, then use it". Which means nimble file should be searched for the way nimble searches for it.

[yglukhov]

I don't remember how nimble behaves here, but yeah, I agree that consistency with nimble is a good reference.

[dom96]

Frankly, I think the current behaviour of nim(ble) with considering the local packages to be in path of every compiled project doesn't make sense - a package should only be added to path if there is an explicit dependency on it. That helps to have reproducible builds and ensure that if something compiles on one machine then it will also compile on any other machine.

This is how Nimble works. It's the very difference between compiling with Nim vs. Nimble.

It makes sense to clearly separate build dependencies from project dependencies (if nimble managed dependencies properly, this would likely even be mandatory).

This is planned.

---

Actually now that I've looked into this: all dependencies in a Nimble file are build dependencies, no? So I don't see the need to change anything, just specify requires "nake" and run nimble c nakefile.

[dom96]

Hah. I see you guys have reached the same conclusions, so can't we close this issue?

[yglukhov]

We can't until nake runs nimble c nakefile ;)

[endragor]

@dom96 It's not resolved yet, so no (i.e. it requires implementation in nake). And no, not all dependencies are build dependencies. There are dependencies needed to build the project (required by nakefile or another build tool) and dependencies that are needed for the project itself - that's the distinguishment I was making and it is also made by Cargo as I noted above.

[dom96]

oh right, cause nake is a binary that performs some magic.

Why not just create a nake task in Nimble? :)

task "nake", "runs a nakefile":
  exec "nimble c nakefile.nim"

There are dependencies needed to build the project (required by nakefile or another build tool) and dependencies that are needed for the project itself

That's a distinction that Nimble won't make for Nimble packages. Because all dependencies are build dependencies as far as Nim is concerned.

[dom96]

There is a concept of "foreign dependencies" in Nimble: https://github.com/nim-lang/nimble#external-dependencies. And these are actually runtime dependencies.