Confusion on terms "Transfer Learning" and "Feature Extraction"

[ammarsaf]

Hi @mrdbourke . Regarding to your lesson, it happened to me a confusion between the terms of transfer learning and feature extraction. Are those terms is actually similar or interchangeable, or it has a slight different on definition? Appreciate for your time.

[mrdbourke]

Hi @Ammar-Azman ,

Good question!

Yes the terms are quite similar.

I'd say feature extraction is a form of transfer learning.

If transfer learning is taking what one model knows about a certain problem and applying it to your own, feature extraction is one way to do it.

Another form of transfer learning is fine-tuning which is a more involved technique.

For example:

• Features extraction = keep base model largely frozen (do not change it) and take the outputs for our own problem
• Fine-tuning = update the base model to our own kind of problem with our own data

You can see an example of each of these in Notebook 05: https://github.com/mrdbourke/tensorflow-deep-learning/blob/main/05_transfer_learning_in_tensorflow_part_2_fine_tuning.ipynb

--

PS I noticed this is closed, I re-opened it but let me know if you'd like to keep it closed!

[ammarsaf]

Thanks for reopening it. Just to confirmed my understanding, so

1. Feature extraction is using the transfer learning model without froze any of the backbones, train using our own dataset, and now it learned the context of our own dataset.

2. Fine-tuning is using the transfer learning and unfroze certain part of the backbone, train using our own dataset, and same it has learned our own dataset. What I understand is, the fine tuning approach is used whenever the feature extraction approach that we used at first shown overfitting / underfitting results, hence we unfroze certain part of the backbone to increase the model complexity and reduce the overfit part.

Correct me if I'm wrong @mrdbourke

[mrdbourke]

Yes your definitions are correct.

Feature extraction is usually great to get quick and good results (if your data is similar to the trained model's data).

Fine-tuning usually works best when you've got a larger amount of data and can "tune" the pretrained model to your own data.

In general, start with feature extraction to see how it goes, then move to fine-tuning.

[ammarsaf]

Thank you @mrdbourke for the clarification!