Unexpected Prediction Result on Zero-Shot VQA Task

[zsun5]

Hello,

I am doing a zero-shot evaluation for the VQA task but got unexpected results. The results in _predict.json look like this:

{"question_id": "10", "answer": "no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no"}, {"question_id": "12", "answer": "no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no"}, {"question_id": "13", "answer": "yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes"}, {"question_id": "19", "answer": "<code_2640><code_5423><code_5423><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_279><code_4021><code_4021><code_4021><code_4021><code_4021><code_4021><code_4021><code_4021><code_4021><code_4021><code_4021><code_5151><code_5151><code_5151><code_5151><code_5151><code_5151><code_5151><code_5151><code_5151><code_5151><code_5151><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026><code_3026>"}

When I made zero-shot predictions on my own dataset, the final results were all in the format of <code_xxxx>.
Do you have any idea how I caused the issue?

Thank you very much.

[taokz]

This is a common phenomenon in zero-shot settings due to the lack of diverse VQA (instruction-following data) during pretraining, which limits the model’s ability to understand human intent. The <code_xxxx> is the image code used for masked image infilling (a pretraining task), which means the model is mistakenly interpreting the question as a prompt for image infilling.

To address this issue, one option is to use the instruction-tuned checkpoints provided in this repository. Alternatively, I recommend fine-tuning the model for better performance.