Moving from Docker in SCALE to Docker in a jail (... in SCALE)

[arpel]

Hi, I have a "old" system on the first versions of Scale (20.xx) that I now want to upgrade to the latest versions ... loosing the ability to directly run Docker within Scale.

Before upgrading the system, I wanted to test how it could actually replace my current (very simple portainer-based) setup : I went through all the steps but I can't start any container within the jail because :

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: mkdir /sys/fs/cgroup/cpuset/docker/37c265e12a2ab4f068e377bc895b789298932a082fe9a20bd2cf4e69c8b781fe: read-only file system: unknown.

As said I still have docker installed and running on the underlying system and the read-only error seems to come from that, but wouldn't the /sys directory inside the jail be coming from the jail "local" rootfs directory and not from the base system one ?

Is there any mean to actually make my test and run docker at the same time on the base system and within a jail ?

Thanks for your insights !

[Jip-Hop]

I never tested this combination. You haven't yet upgraded to the latest version of SCALE? Perhaps you can stop and disable the docker services on the host (temporarily) without fully uninstalling and then see if docker works in the jail.

But if I were you I wouldn't bother with trying to get both to work simultaneously and just upgrade to the latest version of SCALE and focus on the solution to make docker work there instead. You can always boot back into an older version of SCALE after the upgrade and roll back.

[arpel]

Thanks for your very quick answer !

I stopped all docker services (not sure about "disable") on the host and the cgroup/cpuset docker directory is not removed (yet not updated anymore) and still read only for the jail.

[Jip-Hop]

I'd try in a VM with the latest version of SCALE to confirm for yourself that jailmaker works with Docker and portainer. You don't have to migrate anything one by one. You can just install docker inside the jail and mount your existing portainer folder in the jail and have all your services back up and running.

[arpel]

Thanks!
For the "migration" you're right it should work this way, the 'only' modifications being the bind mounts (for host's folders used as volumes). Nevertheless I wanted to switch from zfs storage driver to overlay2 ... which will imply recreating all volumes.

[Jip-Hop]

Ah you're using docker volumes instead of bind mounts... That complicates things.

[rexzhang]

systemd-nspawn may be used to run a command or OS in a light-weight namespace container

systemd-nspawn limits access to various kernel interfaces in the container to read-only, such as /sys/, /proc/sys/ or /sys/fs/selinux/. The host's network interfaces and the system clock may not be changed from within the container. Device nodes may not be created. The host system cannot be rebooted and kernel modules may not be loaded from within the container.

[rexzhang]

systemd-nspawn is NOT a FULL container solution

[rexzhang]

I've migrated 30+ containers into the Jail Maker program, including NextCloud/Jellyfin/Syncthing/Posgres....

[arpel]

I've no doubt it should work, I'm just wondering how to execute a "smooth" transition : moving one service at a time rather than shutting everything off and building back everything from scratch in one shot.

[rexzhang]

It took me about an hour; but before that, almost all of my in-container directories were mapped to the filesystem instead of the docker volume.

I don't know if TrueNAS20 supports KVM. Maybe you can:container in docker -> container in docker in KVM(share with NFS?) -> upgrade to 23.x -> move into jail

[Jip-Hop]

Please post your jailmaker config file and describe in detail the steps you took to install docker inside the jail. Which OS and version did you choose for the jail rootfs? Which exact truenas version are you running? What did you do exactly to enable docker on the host? Which version of jailmaker are you using?

[arpel]

Here you are :

• Jailmaker version : 1.01, followed the exact Github front-page steps to install, no additional config flags (intel-cpu passthrough & docker-compatible)

• Jailmaker rootfs : the default Debian Bookworm

• Docker inside jail : apt install curl && cd /tmp && curl -fsSL https://get.docker.com -o get-docker.sh && sudo sh get-docker.sh && cd ~ && docker

• Truenas version : TrueNAS-SCALE-22.02.0

• Docker on the host (already from you ;) ) : https://gist.github.com/Jip-Hop/af3b7a770dd483b07ac093c3b205323f/revisions, but on earlier revisions (daemon.json + start and stop directly via systemctl

        "exec-opts": ["native.cgroupdriver=cgroupfs"],
        "storage-driver": "zfs",
        "iptables": true,
        "bridge": "",
        "dns": ["1.1.1.1"]``` 

[arpel]

Jail configuration (no manual edits) :

startup=0
docker_compatible=1
gpu_passthrough_intel=1
gpu_passthrough_nvidia=0
systemd_nspawn_user_args=
# You generally will not need to change the options below
systemd_run_default_args=--property=KillMode=mixed --property=Type=notify --property=RestartForceExitStatus=133 --property=SuccessExitStatus=133 --property=Delegate=yes --property=TasksMax=infinity --collec>
systemd_nspawn_default_args=--keep-unit --quiet --boot

[arpel]

Still in the the spirit of having both docker running at the same time, I switched the docker cgroup driver to systemd on the jail (via native.cgroupdriver=systemd) so that it does not try to write in the same cpuset/docker directory : it's not working either and failing with the following error :
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: mkdir /sys/fs/cgroup/cpuset/system.slice: read-only file system: unknown.
This directory does not exist on the host system :

root@DockerJail:/etc/docker# ll /sys/fs/cgroup/cpuset/
total 0
dr-xr-xr-x  3 root root   0 Mar  6  2022 .
drwxr-xr-x 14 root root 360 Dec 29 18:46 ..
-rw-r--r--  1 root root   0 Dec 30 13:09 cgroup.clone_children
-rw-r--r--  1 root root   0 Dec 30 13:09 cgroup.procs
-r--r--r--  1 root root   0 Dec 30 13:09 cgroup.sane_behavior
-rw-r--r--  1 root root   0 Dec 30 13:09 cpuset.cpu_exclusive
-rw-r--r--  1 root root   0 Dec 30 13:09 cpuset.cpus
-r--r--r--  1 root root   0 Dec 30 13:09 cpuset.effective_cpus
-r--r--r--  1 root root   0 Dec 30 13:09 cpuset.effective_mems
-rw-r--r--  1 root root   0 Dec 30 13:09 cpuset.mem_exclusive
-rw-r--r--  1 root root   0 Dec 30 13:09 cpuset.mem_hardwall
-rw-r--r--  1 root root   0 Dec 30 13:09 cpuset.memory_migrate
-r--r--r--  1 root root   0 Dec 30 13:09 cpuset.memory_pressure
-rw-r--r--  1 root root   0 Dec 30 13:09 cpuset.memory_pressure_enabled
-rw-r--r--  1 root root   0 Dec 30 13:09 cpuset.memory_spread_page
-rw-r--r--  1 root root   0 Dec 30 13:09 cpuset.memory_spread_slab
-rw-r--r--  1 root root   0 Dec 30 13:09 cpuset.mems
-rw-r--r--  1 root root   0 Dec 30 13:09 cpuset.sched_load_balance
-rw-r--r--  1 root root   0 Dec 30 13:09 cpuset.sched_relax_domain_level
drwxr-xr-x 14 root root   0 Mar  6  2022 docker
-rw-r--r--  1 root root   0 Dec 30 13:09 notify_on_release
-rw-r--r--  1 root root   0 Dec 30 13:09 release_agent
-rw-r--r--  1 root root   0 Dec 30 13:09 tasks

I'm now wondering how this could even work with no docker installed on the host system ?

[Jip-Hop]

Please also post your config file.

[arpel]

I added the jail config to my post above, if you were talking about Docker config, basically nothing (no daemon.json) just added one to try a different cgroup driver.

[Jip-Hop]

Thanks the jailmaker config file looks OK, you have docker_compatible=1 which is crucial in this case.

If you still want to try to have both working side by side I recommend upgrading to the latest version of SCALE which still had docker included (latest release before 23.10 which I think is version 22.12.4.2). You're running such an old version, I never tested jailmaker on it.

Additionally I recommend to choose debian bullseye as the rootfs for the jail in which you install docker. This is the same version of debian which the TrueNAS host uses. I don't know if it is required, but it's the combination I remember I used. The way you install docker in it looks fine to me.

Lastly you're going to have to migrate your docker volumes (which are now created using the zfs storage driver) to bind mounts. Using bind mounts you can easily run your compose files or portainer stacks regardless of where docker is running (directly on the host, inside jailmaker or even inside the docker compose TrueCharts app). You just have to make the directory containing your date available inside jailmaker or the TrueCharts app.

Good luck!

[arpel]

Thanks for your replies, I actually went to the exact steps above (upgraded to 22.12.4.2) and everything went all fine, including running docker at the same time on the host and within the jail : I'm now having 2 Portainers web-interfaces running on the same machine (still one on the host and the other in the Jail) !

Moving the volumes is actually quite easy with https://github.com/BretFisher/docker-vackup

Thank you very much for your help !