neurocontainers management

[marcelzwiers]

Hi all,

I recently submitted two small PRs to add some functionality to containers.sh. I then prepared a third PR (which I didn't submit) to add another feature, but I stopped because I got the feeling that it is better to entirely rewrite the function. Before doing so, I thought it would be better to first discuss it here :-)

The way I see it from a user perspective is that what is needed is a feature-rich container management tool, comparable to what apt is for debian, what conda is for anaconda, etc. In that way you could do e.g.:

• containers install fmriprep/23.2.0
• containers uninstall fmriprep
• containers list
• containers config
• containers config -h
• containers info fsl
• containers search bidscoin
• containers test mrtrix
• etc

Not everything needs to be implemented overnight, but before start adding functions I think that a good foundation is needed. If I would start this myself, I would do so in Python. What are your thoughts?

[stebo85]

Dear @marcelzwiers,

we also considered reimplementing transparent-singularity and neurocommand and then push it upstream, but we found out that there is a project already, which is more mature and has a very active community: https://github.com/singularityhub/singularity-hpc and by now it has almost all features we would need.

We are planning to switch over to shpc in the near future, so we can profit from a larger ecosystem and developer community.

Have you played with shpc and would it work for you? Would you be keen to help us transitioning?

[marcelzwiers]

I hadn't heard of transparent singularity before, and now I hear from shpc from you :-). I'll discuss it with our HPC admin, and see what he thinks about it

[stebo85]

This videos gives a good overview :) https://youtu.be/xvi-Om8jj2E?si=rS7WA6UTvTUnfAuW very keen to work on this - let me know if you are keen and then we can have a quick chat to see how we can resource this best

[marcelzwiers]

Sorry for the slow response, I was ill/busy. I looked more at singularity-hpc, and it seems very interesting. I myself am, however, more of an application developer and do not have the background to work on this. I still need to talk to our HPC admin to see how he thinks about this. In the meantime, I was wondering, how do you deal with point releases? Do you want to build and add a new container for every version of the software. The amount of neurocontainers can become quite large then, no? If not for disk space, then at least for the neurodesk GUI / menus?

[stebo85]

ok, no problem!

we currently have no guideline on which releases of a software should be in Neurodesk. We leave this up to the tool developers. Disk space is no problem, but I agree that space in the menu system is limited. If there are too many releases of a certain software package we can remove older versions from the menu system - the releases will still be available for reproducibility, just not visible in the menu.