Implement hot/popular file replication on the pools

[DmitryLitvintsev]

Current hot pool replication is triggered by PoolManager if thresholds for hot pool replication are met and it is enabled. Any request to read a file whose replica is on that pools would trigger a p2p request. This solution does not work well due to certain "inertia" - PoolManager determines that pool is too hot when it is too late and because shuffling all files via p2p creates even more load on the hot pool.

Proposal is to have hot/popular file replica replication on the pool. And make it pro-active. I.e. when pool detects N transfers (active and queued) for a particular file it will issue migration copy -replicas=COPIES to spread that file across multiple pool before receiving new request to read this file.

If COPIES > number of available pools the migration job goes into sleeping mode once it has places replica to all pools. This might be OK, or behavior needs to be modified - complete with warning into log file.

So the pool config would get 3 parameters:

1. one parameter enables the feature
2. two parameters replication threshold N and COPIES to control the module

For possible future MAPE based implementation: we can envisage a policy driven "central" service that would just send necessary parameters to the pools to execute the copy (that is N and COPIES will be changed dynamically per message from such a central service). For now we want to put a foundation for this work on the pools.

[greenc-FNAL]

@DmitryLitvintsev Please explain, "MAPE?"

[DmitryLitvintsev]

https://en.wikipedia.org/wiki/Autonomic_computing