buffer_size must be greater than zero error when use custom dataset

[darouwan]

When I use my custom dataset to train model from scratch, it output the error:

WARNING:tensorflow:tf.keras.callbacks.experimental.BackupAndRestore endpoint is deprecated and will be removed in a future release. Please use tf.keras.callbacks.BackupAndRestore.
W0827 10:55:05.103817 16528 callbacks.py:1888] tf.keras.callbacks.experimental.BackupAndRestore endpoint is deprecated and will be removed in a future release. Please use tf.keras.callbacks.BackupAndRestore.
Traceback (most recent call last):
File "F:\python_project\automl\efficientnetv2\main_tf2.py", line 312, in
app.run(main)
File "F:\python_project\automl\venv\lib\site-packages\absl\app.py", line 308, in run
_run_main(main, args)
File "F:\python_project\automl\venv\lib\site-packages\absl\app.py", line 254, in _run_main
sys.exit(main(argv))
File "F:\python_project\automl\efficientnetv2\main_tf2.py", line 278, in main
get_dataset(training=True, image_size=image_size, config=config),
File "F:\python_project\automl\efficientnetv2\main_tf2.py", line 225, in get_dataset
return ds_strategy.distribute_datasets_from_function(
File "F:\python_project\automl\venv\lib\site-packages\tensorflow\python\distribute\distribute_lib.py", line 1189, in distribute_datasets_from_function
return self._extended._distribute_datasets_from_function( # pylint: disable=protected-access
File "F:\python_project\automl\venv\lib\site-packages\tensorflow\python\distribute\mirrored_strategy.py", line 593, in _distribute_datasets_from_function
return input_util.get_distributed_datasets_from_function(
File "F:\python_project\automl\venv\lib\site-packages\tensorflow\python\distribute\input_util.py", line 132, in get_distributed_datasets_from_function
return input_lib.DistributedDatasetsFromFunction(
File "F:\python_project\automl\venv\lib\site-packages\tensorflow\python\distribute\input_lib.py", line 1372, in init
self.build()
File "F:\python_project\automl\venv\lib\site-packages\tensorflow\python\distribute\input_lib.py", line 1393, in build
_create_datasets_from_function_with_input_context(
File "F:\python_project\automl\venv\lib\site-packages\tensorflow\python\distribute\input_lib.py", line 1875, in _create_datasets_from_function_with_input_context
dataset = dataset_fn(ctx)
File "F:\python_project\automl\efficientnetv2\datasets.py", line 444, in dataset_fn
return self._input_fn(
File "F:\python_project\automl\efficientnetv2\datasets.py", line 403, in _input_fn
dataset = self.make_source_dataset(current_host, num_hosts)
File "F:\python_project\automl\efficientnetv2\datasets.py", line 357, in make_source_dataset
dataset = dataset.shuffle(num_files_per_shard, seed=self.shuffle_seed)
File "F:\python_project\automl\venv\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 3961, in shuffle
super(DatasetV1, self).shuffle(
File "F:\python_project\automl\venv\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 1531, in shuffle
return ShuffleDataset(
File "F:\python_project\automl\venv\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 5016, in init
variant_tensor = gen_dataset_ops.shuffle_dataset_v3(
File "F:\python_project\automl\venv\lib\site-packages\tensorflow\python\ops\gen_dataset_ops.py", line 7344, in shuffle_dataset_v3
_ops.raise_from_not_ok_status(e, name)
File "F:\python_project\automl\venv\lib\site-packages\tensorflow\python\framework\ops.py", line 7209, in raise_from_not_ok_status
raise core._status_to_exception(e) from None # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node _wrapped__ShuffleDatasetV3_device/job:localhost/replica:0/task:0/device:CPU:0}} buffer_size must be greater than zero. [Op:ShuffleDatasetV3]

The command I am using is:
python main_tf2.py --mode=train --model_name=efficientnetv2-s --dataset_cfg=imagenet --model_dir=models --use_tpu=False --data_dir=F:\python_project\datasets\molding_class\train

But if I use embedded dataset, it works well:
python main_tf2.py --mode=train --model_name=efficientnetv2-s --dataset_cfg=imagenet --model_dir=models --use_tpu=False

My dataset strusture is like :

.
├── GOOD
│   ├── KA12U9314_B9_DOWN.jpg
│   ├── KA12UA115_D7_DOWN.jpg
│   └── KA12UA188_B11_DOWN.jpg
├── MD07
│   ├── S361M5099_A16_DOWN.jpg
│   └── S36QM2288_E3_DOWN.jpg
└── MD20
├── S361M4502_E16_DOWN.jpg
└── S369M190A_E1_DOWN.jpg

My environment is:
OS: Windows 11
GPU: 3060 12G
Tensorflow 2.10
CUDA 11.8

How to solve it? Thanks!

[darouwan]

It my case, it should be related to

automl/efficientnetv2/datasets.py

Lines 342 to 343 in c7392f2

	filenames = tf.io.gfile.glob(
	os.path.join(self.data_dir, self.split_info['files']))

When I debug, the filenames here is empty