List of encoder supported

[dayuyang1999]

Hi Team,

Can I use any embedding model for creating dense index?

If not, is there a list of supported models?

Best,
Dayu

[lintool]

Yes, start here: https://github.com/castorini/pyserini/blob/master/docs/usage-index.md#building-a-dense-vector-index

I would also suggest that you go through our onboarding guide: https://github.com/castorini/onboarding/blob/master/README.md#-foundations-of-retrieval

[dayuyang1999]

Hi,

Thanks for prompt reply.

The link you provided gave the command to generate run on TREC 2019 queries:

python -m pyserini.search.faiss \
  --threads 16 --batch-size 512 \
  --encoder-class auto \
  --encoder BAAI/bge-base-en-v1.5 --l2-norm --query-prefix "Represent this sentence for searching relevant passages:" \
  --index msmarco-v1-passage.bge-base-en-v1.5 \
  --topics dl19-passage \
  --output run.msmarco-v1-passage.bge-base-en-v1.5-pytorch.dl19.txt \
  --hits 1000

However, I found that, for creating dense index, there is no --query-prefix argument

https://github.com/castorini/pyserini/blob/master/pyserini/encode/__main__.py

[lintool]

I believe @MXueguang did the encoding using Tevatron [1] - but if you want to add the feature to pyserini.encode, pull requests welcome!

[1] http://tevatron.ai/

[dayuyang1999]

Thank you so much! One more question:

According to the training script in tevatron, does that mean the model trained using default similarity measure used in tevatron is optimized on dot product instead of cosine similarity?

So using a model trained using tevatron to compute cosine similarity is wrong?

https://github.com/texttron/tevatron/blob/main/src/tevatron/retriever/modeling/encoder.py

[dayuyang1999]

Oh seems not.

if I need a model to produce cosine similarity, I can do that by passing --normalize = True