Use Futures for Images

[stepanyanhayk]

No description provided.

[akshay-krishnan]

which datasets has this been tested on?

[stepanyanhayk]

which datasets has this been tested on?

I tested this on gerrard-hall-100 and skydio-32 datasets. We already use futures for corresponder generator images. This PR also makes it more consistent to use futures in scene-optimizer.

[ayushbaid]

@stepanyanhayk is there any performance implication from this?

[stepanyanhayk]

@stepanyanhayk is there any performance implication from this?

When running on cluster, I was seeing data reading errors (happening nondeterministically), using futures in all places for data reading seems to fix the issue.

[akshay-krishnan]

which datasets has this been tested on?

I tested this on gerrard-hall-100 and skydio-32 datasets. We already use futures for corresponder generator images. This PR also makes it more consistent to use futures in scene-optimizer.

could you test on skydio-500 or whatever the largest dataset that we have? I am wondering if we will run out of CPU memory for large datasets.

[akshay-krishnan]

It would also be great if you looked at the dashboard and verified how the memory is being managed as the images are being loaded. Good to understand whether all images are being loaded at once? is there a peak memory usage there?