tar: .: file changed as we read it with fuse-overlayfs
#384
Comments
|
Great news: I spent a lot of time, but did manage to reduce that down to a small binary that reproduces the problem! Here's the new steps to reproduce: λ cat test.c
#define _GNU_SOURCE
#include <dirent.h>
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <errno.h>
#include <string.h>
void exit_on_system_error(int fd_to_check, const char* msg) {
if (fd_to_check != -1)
return;
fprintf(stderr, "%s:%d: %s: %s\n", __FILE__, __LINE__, msg, strerror(errno));
_exit(1);
}
int main() {
exit_on_system_error(creat("/tmp/myfile", 0666),
"creat");
int fd_root_dir = openat(AT_FDCWD, "/", O_RDONLY);
exit_on_system_error(fd_root_dir, "openat");
char buf[4096];
exit_on_system_error(getdents64(fd_root_dir, buf, sizeof(buf)),
"getdents64");
struct stat stat_arg;
exit_on_system_error(fstatat(AT_FDCWD, "/", &stat_arg, 0),
"newfstatat");
}
λ gcc test.c -o a
λ podman run --rm -v "/tmp:/mnt" ubuntu:22.04 sh -c "stat -c %y / && /mnt/a && stat -c %y /"
2022-12-21 15:14:31.187780859 +0000
2022-12-21 15:14:31.365780865 +0000Basically, it happens when Removing any of the 4 calls will make the problem no longer reproduce. |
|
could you strace the fuse-overlayfs process while you run that program? |
Sure! Here it is: strace_fuse-overlayfs.txt To do the strace I used this script: |
Worth noting though, apparently |
it forks itself. You'd need to use the |
Thanks! Adding a Removing the |
|
Anything else I can provide here? |
|
fuse-overlayfs is mostly in maintenance mode at this point since there is a native file system in the kernel that replaces it, so it is unlikely I am going to spend much time troubleshooting fuse-overlayfs unless it is a major issue. So the easiest would be to provide a patch. Said so, I can give some guidance if needed. Could you please attach the output you get from strace when you add |
Sure, here goes: strace_fuse-overlayfs-w-f.txt |
|
@giuseppe hello, so, any news, is there anything interesting in the logs? |
FTR, it doesn't really. I would love to migrate to the native overlay, however due to containers/podman#16541 it is not currently feasible, and from my understanding of the discussion on that bugreport, there will be no solution any time soon. So it seems like fixing |
I've no time to look into this issue at the moment, if anyone is willing to troubleshoot and possibly open a PR, I'll help to get it merged |
|
Okay, so, I'm trying to debug the /usr/bin/fuse-overlayfs -o lowerdir=/home/constantine/.local/share/containers/storage/overlay/l/S4FZLHMYW6WDUORNLKFHZIXOZG,upperdir=/home/constantine/.local/share/containers/storage/overlay/369531141b1ac0c2fe3fc26a5732df66ac56221200c25cb4d456545e530a8b26/diff,workdir=/home/constantine/.local/share/containers/storage/overlay/369531141b1ac0c2fe3fc26a5732df66ac56221200c25cb4d456545e530a8b26/work,,volatile /home/constantine/.local/share/containers/storage/overlay/369531141b1ac0c2fe3fc26a5732df66ac56221200c25cb4d456545e530a8b26/mergedHowever, executing same command in terminal results in |
Apparently yes, it is. one of them I see mentioned on other podman issues. Unclear what causes them since e.g. |
/kind bug
Description
I've recently migrated CI to use
fuse-overlayfsdue to an issue with with native overlay, and after some time it started sporadically failing with error in title. I am not completely sure fuse-overlayfs is the cause, because of an unrelated podman bug which I'll report separately, but that was the only change, so it likely is.Problem comes down to this:
To dispel any doubts the
tarcommand is valid: you can test it on a host system, it will succeed. Thetaris being asked to enter/dir with-C /, then to save everything from the current dir that is not--excludeed into/tmp/flash.tar. However, for the purposes of a "minimal testcase" the--excludecommand excludes everything, so/tmp/flash.tarwill only contain an empty./dir.The
tarerror comes down to mtime of a/being changed while it's running. Why does that happen, I haven't yet found. I did find though that various odd combinations make that not reproduce. Some examples in the workarounds section below.The error happens on Ubuntu and Arch and with different containers. I'm reporting Arch as it's my working system and is easily accessible.
Known Workarounds
Here you can see the
tarcommand is being called twice, but the error only happens the first time.No error. The only thing changed is we call a
ls /, which alone fixes the problem.No error. There are 2 changes:
tardestination dir belongs to host through-v …touch /command before callingtarDoing any of them alone will result in the error, that only works as a combination.
Steps to reproduce the issue (in terms of terminal commands):
Describe the results you received:
An error:
tar: .: file changed as we read it
Describe the results you expected:
No output and zero exit code.
Output of
podman version:Output of
podman info:Details
Package info:
Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?
Yes
The text was updated successfully, but these errors were encountered: