reproduce ance results

[zhanghan9797]

How to get train query embedding?

[zhanghan9797]

How to get msmarco train query embedding?

[MXueguang]

Hi @zhanghan9797,

Sorry for the late reply, I just get back from holidays.
msmarco train topics is not self-contained in anserini because the file is big.
You can get the train query embedding with ANCE checkpoint by:

1. prepare a train topics file: a tsv file where each line is <query_id>\t<query_text>. The train topics can be found on official website of msmarco dataset.
2. Then run the query encoding with ANCE checkpoint using this script https://github.com/castorini/pyserini/blob/master/pyserini/encode/query.py

python -m pyserini.encode.query --topics <path to tsv file> --encode castorini/ance-msmarco-passage --output <output path>

it uses cpu by default, specify --device cuda:0 to use gpu