fatal error:(The read operation timed out)

[huanyuxinchen]

Hi, I tried downloading the train dataset with AWS S3, but every time when I had downloaded almost 140K images, the downloading process would be interrupted, and the error happened: "fatal error:(The read operation timed out)", could you please help me?

[DamonLiuTHU]

may be this problem is caused by the file system, check the limit of the numbers of files per dir.

[huanyuxinchen]

@DamonLiuTHU Thank you for your reply，but I have checked other dir， and I found some dir has more than 400K images，so I think this problem is not caused by this reason.

[rabienrose]

I can only download about 400k images from train folder, than it gave me the error of "fatal error:(The read operation timed out)". it is like there are only about 400k images in the s3 train folder.

[huanyuxinchen]

@rabienrose Me to, I have download 448341 images from train folder, then when I try AWS s3 sync again, the error happened: "fatal error:(The read operation timed out)".

[rabienrose]

@huanyuxinchen I found it recovered this morning, it started downloading the further more files now.

[superzrx]

@rabienrose I have the same problem after downloaded 251k images . The aws just exit by itself, could you please share how it started downloading more files?

[huanyuxinchen]

@rabienrose I still cannot download the further more files.

[rabienrose]

I wrote a script to start aws s3 sync again after it exit with error. So it keep trying until the transferring recover.

[superzrx]

@rabienrose It works! Thank you.

[huanyuxinchen]

@rabienrose I still cannot download, can you share your scripts? Thank you!

[rabienrose]

@huanyuxinchen
while :
do
aws s3 --no-sign-request sync s3://open-images-dataset/train ./train
done

[huanyuxinchen]

@rabienrose Thank you, I will try!