How to fix jnius.JavaException from "query" being null

[Yuv-sue1005]

Hello,

I'm trying to use pyserini.search.lucene to do a bm25 retrieval run on an index created using pyserini.index.lucene. Here is the exact command I'm inputting:

python -m pyserini.search.lucene
--index tot/indexes/TREC_ToT_jsonl
--topics tot/topics-and-qrels/train-2024/queries_reformated.jsonl
--output tot/runs/bm25/bm25_train.run
--output-format trec
--hits 1000
--bm25 --k1 1.0 --b 1.0

My queries file is a jsonl file with each line being in the {"id": "1234", "text": "Insert query text here"} format. Upon running the above command, I get the following error:

Traceback (most recent call last):
File "/usr/local/anaconda3/envs/TREC-ToT/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/anaconda3/envs/TREC-ToT/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/usr/local/anaconda3/envs/TREC-ToT/lib/python3.10/site-packages/pyserini/search/lucene/main.py", line 296, in
hits = searcher.search(text, args.hits, query_generator=query_generator, fields=fields)
File "/usr/local/anaconda3/envs/TREC-ToT/lib/python3.10/site-packages/pyserini/search/lucene/_searcher.py", line 152, in search
hits = self.object.search(q, k)
File "jnius/jnius_export_class.pxi", line 1163, in jnius.JavaMultipleMethod.call
File "jnius/jnius_export_class.pxi", line 877, in jnius.JavaMethod.call
File "jnius/jnius_export_class.pxi", line 964, in jnius.JavaMethod.call_method
File "jnius/jnius_utils.pxi", line 79, in jnius.check_exception
jnius.JavaException: JVM exception occurred: Cannot invoke "org.apache.lucene.search.Query.rewrite(org.apache.lucene.search.IndexSearcher)" because "query" is null java.lang.NullPointerException

Just because this information might be useful, here are my versions used:

pyserini - 0.35.0
python - 3.10.14
java - 21.0.3

Anyone know what's going wrong? Let me know if you need more information, and thanks for your help!

[lintool]

What's the format of tot/topics-and-qrels/train-2024/queries_reformated.jsonl? Most likely it's in a format Pyserini can't read...
Reformat into a TSV?

[Yuv-sue1005]

I would tell you the format of queries_reformated.jsonl, but converting to a TSV fixed it! I tried converting it into a TSV file before, but I missed the '\t' while reformatting, which messed with pyserini. Thanks so much for your help!

For people in the future with the same problem, here's the python code to convert a jsonl file into a tsv file (just to save you some time):

import json

with open('Insert-tsv-file-here.tsv', 'w') as out:
  with open('Insert-jsonl-file-here.jsonl', 'r') as i:
    for line in i:
      l = json.loads(line)
      out.write(l['query_id_key'] + '\t' + l['query_text_key'] + '\n')
      # If your query text contains '\n', use the below two lines instead of the out.write line above
      ##p = l['query_text_key'].find('\n')
      ##out.write(l['query_id_key'] + '\t' + l['query_text_key'][:p-1] + l['query_text_key'][p+1:] + '\n')