Object Store Selection requires a Distributed Object Store - should it?

[jmchilton]

I don't recall exactly why I implemented it this way but I think it was somewhat intentional. I guess the semantics of the hierarchical object store make stronger claims about where data is going to end up than the distributed object store does. Does this make sense?

Does anyone use the hierarchical and does anyone use it that wants users to be able to bypass its selection?

The state affairs with #18127 is a bit more interesting. It is kind of a bug in the PR that you can add object stores even if the configuration won't respect them and that is an easy fix I will try to make very soon to the PR. But once that change is made... when should user object stores be available? If an admin is just running a single object store should we disable those options or should we have them and somehow implicitly create... something like a distributed object store wrapper around the disk object store internally to support that configuration. Or alternatively, do we just assume all big instances are using distributed object store and update the docs to really call out loudly they are required in order to use the functionality.

[nuwang]

Requiring a distributed object store seems reasonable to me, especially if it can simplify the code. Aren't these other object store types without a uuid considered "legacy" for anything but the most trivial cases?