Improve performance for VCFAnnotator

[korikuzma]

Parallelize? If we want people to use the tool, we have to improve the performance

[wesleygoar]

I do think we should parallelize. I think that would be an easy win.

[wesleygoar]

I think we should def investigate ways to improve single threaded performance as well.

[korikuzma]

I can take a stab at it since people really want this.

[korikuzma]

Goal: 10k/s

[github-actions]

This issue was marked stale due to inactivity.

[quinnwai]

Using v2.0.0a3 on a Mac Book Pro (13-inch 2017, 2.3 GHz Dual-Core Intel Core i5, 8 GB memory)...

• 13.09 s ± 1.76 s for ~10,000 (9,889) variants so ~755 variants/s. Default params so getting ref and alt IDs and timing averaged over 10 rounds.

[wesleygoar]

Running the vcf annotator script on a MacBook Pro (16-inch 2021, M1 Pro, 32 GB Memory) using default params on main vrs-python branch. (commit 64fee4c)
File: vcf including all chr1 variants from the GIAB benchmarking file

• 1,703,455 Variants
• Average vars/sec over 5 runs = 6,062.07 variants/sec
• This means we are annotating over 12,000 alleles/sec

[korikuzma]

@quinnwai @wesleygoar Thanks. I think it would be beneficial to add the commit y'all used

[wesleygoar]

@quinnwai @wesleygoar Thanks. I think it would be beneficial to add the commit y'all used as well as the command / params you used to test

@korikuzma I have updated my post to include the commit I was on during testing.