POC for ssd cache

[imotov]

Description

Proof of concept for adding SSD cache to make sure this architecture is feasible. It is still in the very early stages, not all features are implemented.

To setup add the following node configuration is needed:

storage:
  cache:
    storage_uri: file:///tmp/storage
    max_cache_storage_disk_usage: "5G"

On the index side the following setting should be added to the config:

    "index_uri": "cache://file:///tmp/path_to/index/index_id'",

Fixes #3443

How was this PR tested?

Still need to add tests.

Outstanding issues

This is still WIP. The following tasks are still need to be finished for MVP:

• Clean up cache storage configuration (needs some reasonable defaults)
• Comprehensive unit testing of subcomponents
• Rehydration of the current cache split state on the node startup
• Limiting the maximum space used by downloaded and in-flight splits
• Postponing split rebalancing until the cluster reached the desired number of nodes or a certain time passed after the number of nodes threshold has been breached.
• Throttling the number of simulations split downloads and network load on search nodes.
• Make storage copy_to_storage more robust
• Optimize reading metadata with a large rate of change
• Modify URI format to cache:file:///tmp

Beyond MVP:

• Refactor Storage Configuration

[imotov]

Superseded by #3786