memory insufficiency while running ANCE

[noem-1998]

Hi, I am experimenting with pyserini for the first time. And while running ANCE on google colab, I get an error due to large allocation. Is there any way to run it on a 12GB RAM environment?

Here's the error i got:

Using pre-defined topic order for msmarco-passage-dev-subset
Attempting to initialize pre-encoded queries ance-msmarco-passage-dev-subset.
Downloading index at https://github.com/castorini/pyserini-data/raw/main/encoded-queries/query-embedding-ance-msmarco-passage-dev-subset-20210419-9323ec.tar.gz...
query-embedding-ance-msmarco-passage-dev-subset-20210419-9323ec.tar.gz: 19.0MB [00:00, 22.8MB/s]
Extracting /root/.cache/pyserini/queries/query-embedding-ance-msmarco-passage-dev-subset-20210419-9323ec.tar.gz into /root/.cache/pyserini/queries/query-embedding-ance-msmarco-passage-dev-subset-20210419-9323ec.adad81bb1495eff2f0463e809ecc01b8...
Initializing ance-msmarco-passage-dev-subset...
Attempting to initialize pre-built index msmarco-passage-ance-bf.
Downloading index at https://rgw.cs.uwaterloo.ca/JIMMYLIN-bucket0/pyserini-indexes/dindex-msmarco-passage-ance-bf-20210224-060cef.tar.gz...
dindex-msmarco-passage-ance-bf-20210224-060cef.tar.gz: 23.4GB [05:31, 75.6MB/s]
Extracting /root/.cache/pyserini/indexes/dindex-msmarco-passage-ance-bf-20210224-060cef.tar.gz into /root/.cache/pyserini/indexes/dindex-msmarco-passage-ance-bf-20210224-060cef.f6332edb8f06ba796850388cf975b414...
Initializing msmarco-passage-ance-bf...
tcmalloc: large alloc 27162083328 bytes == 0xdb04000 @ 0x7f935cb87887 0x7f93222fa981 0x7f9322305763 0x7f93220dff81 0x5d814d 0x56305c 0x55cd91 0x5d8941 0x4997c7 0x5d8868 0x4997a2 0x55cd91 0x5d8941 0x5da092 0x586de6 0x5d8cdf 0x55dc1e 0x4fd2db 0x4997c7 0x55cd91 0x55d743 0x627376 0x5aaeb9 0x4990ca 0x55cd91 0x5d8941 0x4990ca 0x55cd91 0x5d8941 0x5d8506 0x677b11

[lintool]

Hi @noem-1998 thanks for your interest in Pyserini. Unfortunately, there is no solution other than to find a machine with more memory...