Prune and clean clash with backups

[psyciknz]

Is there a way, or feature request that if repository locked at time of prune/clean that it can retry?

Especially helpful for multiple systems to single repository, it's not always easy to find a gap.

[djmaze]

There is no special feature for that. Do you do backups more than once a day? Otherwise you should just schedule your prune job so there is no overlap. And if there are too many hosts using a single repository, it might make sense to split those up into using multiple different repositories.

That said, I would be open to have a retry mechanism as a feature. But I probably won't implement that myself. So open for PRs :)

[psyciknz]

Yeah I do run multiples.
i did at one point have split repositories, but I then read about deduplication, and it was recommended to use one repo, so swapped it back

[psyciknz]

As a follow on. I have managed to fix by only setting one machine to do the prunes. But what I have just observed (whiile trying to see why my POST_COMMANDS_SUCCESS and POST_COMMANDS_FAILURE where not running, was that after the backup a forget is running. And this seems to be crashing the container:

Files:          70 new,    49 changed, 1302179 unmodified
Dirs:           30 new,   114 changed, 203486 unmodified
Data Blobs:    797 new
Tree Blobs:    135 new
Added to the repository: 456.822 MiB (456.884 MiB stored)

processed 1302298 files, 48.280 GiB in 10:07
snapshot 32815790 saved
Backup successful
Forget about old snapshots based on RESTIC_FORGET_ARGS = --keep-last 10 --keep-daily 2 --keep-weekly 1 --keep-monthly 1
unable to create lock in backend: repository is already locked by PID 7601 on Tosto by root (UID 0, GID 0)
lock was created at 2022-09-15 08:02:18 (7m45.635617444s ago)
storage ID a2c32eed
the `unlock` command can be used to remove stale locks

This was a docker logs, which crash, and therefore restarted the container. Which I had to remove the run backup on start.

I think my multiple machines to one repo is causing this problem, but should a failure of the restic forget cause the container to drop?

[psyciknz]

hmm further to this, I think it was being killed due to memory restrictions on the containers. Depending how you view the logs you don't see the killed which if the container being restarted.

[djmaze]

Oh right. Memory restrictions. That's possible. With docker inspect <CONTAINER ID> you should hopefully be able to see the exit reason.

[psyciknz]

Ahh confirmed that yes, the memory was hitting the restrictions as part of a forget. And in doing so container was killed, which killed off any post commands from running.