Quick question

[JoshK68]

Hi,

I'm trying to prefect my data and it's giving me an error:
train_data = train_data.prefetch.batch(32).prefetch(tf.data.AUTOTUNE)
test_data = test_data.prefetch.batch(32).prefetch(tf.data.AUTOTUNE)

train_data

AttributeError: 'DirectoryIterator' object has no attribute 'prefetch'

Here's the link for my code: https://colab.research.google.com/drive/1iXMloev_OJ2GF_ot2oYosw7HWGGfuU_v?usp=sharing
The error is occuring around the bottom, near line 122
Can you please advise what I'm doing wrong and how to fix this?

P.S. I know I don't need to prefetch this tiny dataset...and am just practicing with the concept of prefetching

Thank you very much!

[gauravreddy08]

Hey @JoshK68,

Use the image_dataset_from_directory function instead, it does the same job !!

So, replace your code :

train_datagen = ImageDataGenerator(rescale = 1./255)
valid_datagen = ImageDataGenerator(rescale=1./255)


train_data = train_datagen.flow_from_directory(train_dir,
                                               batch_size=32,
                                               target_size=(224,224),
                                               class_mode='binary',
                                               seed=42)

valid_data = valid_datagen.flow_from_directory(test_dir,
                                               batch_size=32,
                                               target_size=(224,224),
                                               class_mode='binary',
                                               seed=42)

with,

train_data = tf.keras.preprocessing.image_dataset_from_directory(train_dir,
                                                                 batch_size=32,
                                                                 label_mode='binary',
                                                                 image_size=(224, 224),
                                                                 seed=42)

test_data = tf.keras.preprocessing.image_dataset_from_directory(train_dir,
                                                                batch_size=32,
                                                                label_mode='binary',
                                                                image_size=(224, 224),
                                                                seed=42)

And also, you don't need to batch your data again here, as you've already mentioned the batch_size above,

train_data = train_data.prefetch.batch(32).prefetch(tf.data.AUTOTUNE)

And if your trying to unbatch the dataset imported by the above piece of code I suggested, use train_data.unbatch() instead of from_tensor_slices(train_data)

[JoshK68]

Hi Gaurxvreddy,

Thank you very much!
Two quick followups:

1. flow_from_directory vs image_dataset_from_directory... Do we use image_dataset_from_directory here (and everytime) that we're working with images and we'd use the 'flow_from_directory' only when working with other data such as with Skimlit?

2. Also, that seems to work well...but when I .fit on that with:

Fit on cloned model (cloned on model 4)

history_6 = model_6.fit(train_data,
epochs=5,
steps_per_epoch=int(0.1 * len(train_data)), #len(train_data) ----> int(0.1 * len(train_dataset))
validation_data=test_data,
validation_steps=int(0.1 * len(test_data)))

then it gives me this error: Expect x to be a non-empty array or dataset.

https://colab.research.google.com/drive/1iXMloev_OJ2GF_ot2oYosw7HWGGfuU_v#scrollTo=sBmcnV4Tk12s

I checked len(train_data) and it doesn't seem to be empty. Any advice?

Thank you much!

[JoshK68]

@gaurxvreddy,

One more quick question (I'm going through all my old notebooks and making sure reviewing all the mistakes, so sorry for so many questions these couple days!)...

I know Daniel did a section on 'cross validation', but I looked everywhere multiple times.
I'm trying to review cross validation... where in the notebook or his beautiful online book, might it be?

Thank you much!
Josh

[gauravreddy08]

Hey @JoshK68,

"so sorry for so many questions these couple days!"

It's totally fine !! We all learn by asking, or at least that's how I learned.

And about your question,

It's strange, I didn't find it too. I guess in this course, Daniel had only explained the idea behind cross-validation, not the code part. I may be wrong, don't take my word 😂. However I'll let you know if I find it anywhere !!

[JoshK68]

Hi @gaurxvreddy,

Thank you. Unfortunately that doesn't fix that prefetch error.
I used this code as you suggested: train_data = train_data.prefetch(tf.data.AUTOTUNE) (I took out the .batch so not to be repetitive).
It gives:
<PrefetchDataset shapes: ((None, None, 224, 224, 3), (None, None, 1)), types: (tf.float32, tf.float32)>

When I fit it a few lines below (around line 100), then it gives this error:
ValueError: Expect x to be a non-empty array or dataset.

I checked Google and SO and still don't know how to fix this.

Can you please advise?

Thank you much!
Josh

[JoshK68]

Here's the link: https://colab.research.google.com/drive/1iXMloev_OJ2GF_ot2oYosw7HWGGfuU_v?usp=sharing
Can you figure out what's messing it up? Got to be somewhere around line 100

[gauravreddy08]

(None, None, 224, 224, 3) what the two None's mean is that you have batched the data twice. Unbatch it and it should work!

[JoshK68]

@gaurxvreddy,

This is excellent. Thank you!

Totally makes sense, when I created a batch it created a None.
Unbatching it is like deleting the Nones, right?

Very cool and Thank you again!

Josh