FullGrad for Vision Transformers

[imanuelroz]

Hi I wanted to know if there is a version of FullGrad which could be applied on Vision Transformers like ViT or the Swin Transformer, or if there are some small changes that could be done in the code in order to do it. Thank you in advance.

[suraj-srinivas]

Hi! Sorry for the late reply. So technically Fullgrad is proposed for convolution or fully connected neural networks, and as a result the completeness conditions may not be satisfied for transformers.

However you are free to use Simple / Smooth FullGrad in this case, which don't have completeness associated. I haven't tested it for transformers myself, but you'd need to change this line:

fullgrad-saliency/saliency/tensor_extractor.py

Line 25 in 2121d21

if isinstance(m, nn.Conv2d) or isinstance(m, nn.Linear) or isinstance(m, nn.BatchNorm2d):

to include self-attention layers and exclude fully connected maybe.

If you do happen to use it, I'd be happy to learn about your experience or the issues you faced!