Support XRootD URLs with multiple servers.

[chrisburr]

From the TXNetFile docs:

The "url" argument must be of the form

root://server1:port1[,server2:port2,...,serverN:portN]/pathfile,

Note that this means that multiple servers (>= 1) can be specified in the url. The connection will try to connect to the first server:port and if that does not succeed, it will try the second one, and so on until it finds a server that will respond.

I didn't realise this was an pre-existing feature in ROOT but I'd been asking for it for a while. Being able to do this better matches with how data is stored on the grid and makes it easier to when sites are down.

The implementation of this shouldn't only check that the server is alive, it should also fallback if errors are returned when trying to access the data (i.e. only querying the config isn't enough to know if the site is down).

[jpivarski]

Are the fallbacks accessible in the XRootD.client.URL object?

Recovering from a nonexistent URL takes quite a while. The timeout has to be an integer number of seconds and passing "1" means as much as 5‒10 seconds in practice. (Eventually, I'll be turning off unit test #1, or at least making it part of a "long tests" suite.)

[jpivarski]

@lobis, @nsmith-, is this something that can be done with fsspec-xrootd now?

It could be cool to finish off an issue with a low number like #5!

[nsmith-]

I think this ties very closely with CoffeaTeam/fsspec-xrootd#36 The implementations would share a lot of the same logic w.r.t. keeping open connections to multiple servers and ensuring we use the best server for a file at any given time.