Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use renameat2(RENAME_EXCHANGE) for truly atomic renames with --temp-dir #655

Open
nh2 opened this issue Oct 25, 2024 · 1 comment
Open

Comments

@nh2
Copy link

nh2 commented Oct 25, 2024

So far, rsync had not way to atomically rename directories into place, because one could not replace one non-empty directory with another.

Thus, rsync had to move the old dir "out of the way", or delete it recursively, and the new dir into its place.

This means rsync fails when 2 rsyncs write the same files to the same destination directory, because one would delete files that the other is in the middle of writing.

This would manifest as errors such as:

rsync: [receiver] rename "/my-tempdir/myfile.ZWWXzu" -> "myfile": Permission denied (13)

or

rsync: [generator] failed to set times on "/my-sourcedir/mydir": No such file or directory (2)
rsync: [generator] recv_generator: mkdir "/my-sourcedir/mydir/myfile" failed: No such file or directory (2)
** Skipping any contents from this failed directory ***

Edit: I think the above errors happen only in specific situations, e.g. when the target directory contains some subdirectories which are not writable or so. I haven't quite figured the situation out yet. I suspect that in such cases, rsync needs to delete the target directory first, causing the errors. If write permissions are available in the target directory, ``--temp-dirseems to already be atomic, because inside it only plain files are created, and not parallel directory hierarchies -- at least if--delay-updates` is not used.

Now it's possible

renameat2(dirfd, "alpha", dirfd, "bravo", RENAME_EXCHANGE);

I feature-request that rsync use this to make --temp-dir moves actually atomic.

Then 2 concurrent rsyncs to the same target directory should work without errors, even if subdirectories are involved.

@nh2 nh2 changed the title Use renameat2(RENAME_EXCHANGE) for truly atomic renames with --partial-dir Use renameat2(RENAME_EXCHANGE) for truly atomic renames with --temp-dir Oct 25, 2024
@nh2
Copy link
Author

nh2 commented Oct 26, 2024

Probably worth saying, for RENAME_EXCHANGE:

Both pathnames must exist

This means rsync would first have to try a normal rename, and if that fails with EEXIST, fall back to renameat2 with RENAME_EXCHANGE.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant