MAE Backbone Reconstruction visualization on custom dataset (No pretraining done on custom dataset)

[Aakash3101]

Hi, I have used the pretrained weights of the ViT backbone given for the encoder for the MAERec model, I have trained the MAERec model with my custom dataset and the Union14M-L dataset. I want to confirm the visualizations given by the encoder (ViT backbone) is giving accurate reconstructions for my custom dataset. Is there any way to check the reconstructions of my custom dataset?

One way would be to pretrain the ViT backbone with my custom dataset. But I don't want to waste time when I have already trained the MAERec model. So I am thinking if I can used the Encoder weights (of MAERec model) and pass them to the MAE model so that I can get reconstructions on my custom dataset.

[Mountchicken]

Hi @Aakash3101
It is no longer possible to do a reconstruction using the ViT in MAERec. The reason is that at this point the ViT has been fine-tuned on the recognition task and can no longer be used for reconstruction

[Aakash3101]

Thanks @Mountchicken , but is there any way to visualize the interim steps of the MAERec model so that I can find where my model is performing wrong, something like Attention maps?? It would be really helpful if you could guide me regarding it.

[Mountchicken]

Well, It may not be easy to visualize some middle results. You can try to build a test set to see if the final model performs well on it.