Calculation of Rucio RSE quota

This wiki provides a quick overview of how a Rucio RSE quota information is calculated and considered or not for input data placement. Note that this logic depends on the limits created for a given Rucio account, instead of using whatever RSE (global) quota is available for the RSEs.

Rucio python APIs

This process depends on a pre-discussion with the CMS Rucio admins, who need to define the proper Rucio account limits in the database and for every single RSE that is planned to be used. Once those limits have been defined, we can simply consume that information through the standard Rucio python client APIs.

First, we start by collecting the RSE limits for the Rucio account that we are concerned with, for that we need to use the get_local_account_limits API, which provides a dictionary with the RSE name and the storage account limit, in bytes. An example of the output and use of this API is:

> client.get_local_account_limits("wmcore_transferor")
{'T1_US_FNAL_Disk': 9911633611278710, 
 'T2_DE_DESY': 611695766948279,
 ...}

Secondly, we can now call the get_local_account_usage API to collect the Rucio account usage in each RSE. This API provides a list of dictionaries with information further information for each RSE and account, where:

• bytes: is the total amount of bytes being used by the current account at a given RSE
• bytes_limit: is the Rucio account limit at a given RSE (actually it's the same data retrieved from get_local_account_limits)
• bytes_remaining: is the difference between bytes_limit and bytes, so the actual remaining quota for a Rucio account at a given RSE.

An example of the output and use of this API is:

> pprint(list(client.get_local_account_usage("wmcore_transferor")))
[{'bytes': 512930764594725,
  'bytes_limit': 611695766948279,
  'bytes_remaining': 98765002353554,
  'files': 99718,
  'rse': 'T2_DE_DESY',
  'rse_id': '0ad9b7eb296849219cb3af243b59c334'},
 {'bytes': 8460303335843503,
  'bytes_limit': 9911633611278710,
  'bytes_remaining': 1451330275435207,
  'files': 1737414,
  'rse': 'T1_US_FNAL_Disk',
  'rse_id': '087ee3383b9d45f6b31814af07b2c56d'},
...]

RSE quota logic

Based on the WMCore implementation for constructing the RSE quotas for a given Rucio account, in this example considered for the wmcore_transferor account, we can describe the following algorithm currently in use by MSTransferor:

1. Initialize a RSEQuotas object with the rucio account, a fraction of the account quota that we want to use (e.g. 0.9, i.e. 90%), and a minimum threshold to consider for available space (e.g. 1TB).

2. fetchStorageQuota: calls getAccountLimits wrapper API for the wmcore_transferor account (which uses the get_local_account_limits Rucio API)

   a. discard any Tape and Export RSEs

   b. the limit returned is considered as: quota, bytes_limit and bytes_remaining

3. fetchStorageUsage: calls getAccountUsage wrapper API for the wmcore_transferor account (which uses the get_local_account_usage Rucio API)

   a. the bytes field reports the RSE+account usage, in bytes.

   b. calculate bytes_remaining from the quota (from the previous step) minus bytes usage field

4. evaluateQuotaExceeded: use the quota fraction configuration to calculate the "usable" quota (rse['quota'] * self.quotaFraction)

   a. set usable quota as the minimum of usable quota or bytes remaining in the rse (rse['quota_avail'] = min(quotaAvail, rse['bytes_remaining'])). Note that bytes_remaining can be negative, in case the RSE is oversubscribed.

   b. if usable quota (quota_avail) is smaller than the bare minimum threshold (e.g. 1TB), then mark the RSE as out-of-space, otherwise consider it as available for rule creation.

Miscellaneous

The RSEQuotas WMCore module was created back in the days that we had to support both PhEDEx and Rucio. While reviewing this implementation, we can apparently simplify this logic to:

1. Fetch the wmcore_transferor quota/limit, usage and remaining bytes for each RSE through the get_local_account_usage Rucio API. Where:

• bytes_limit: total amount of bytes that wmcore_transferor can use (the actual limit in a given RSE).
• bytes: amount of bytes currently in use by wmcore_transferor;
• bytes_remaining: amount of bytes still available for wmcore_transferor (bytes_limit - bytes).

2. Provided a specific quota fraction that we want to use, calculate the usable quota as:

bytes_limit_usable = bytes_limit * quota_fraction

3. Calculate the remaining bytes as:

bytes_remaining_usable = bytes_limit_usable - bytes

4. If we want to ensure that RSEs will only be used if they have a bare minimum amount of bytes available (called minimum_threshold), we calculate it as:

if bytes_remaining_usable > minimum_threshold:
  RSE good for use
else:
  RSE does not have enough quota available

In short, there is no need to use the get_local_account_limits Rucio API.

Further simplification of this, without considering any quota fraction, can rely solely on the bytes_remaining field.

Miscellaneous

Rucio usage and quotas within WM have been recently (April/2023) discussed in a Data Management meeting, you can find a slide deck in the indico page.