Installing WMCore and dependencies via conda

Overview

The following instructions allow you to install WMCore packages and dependencies on an external machine. This is useful for debugging e.g.: MicroServices, as you can run and modify central services components on the fly to test your code, without requiring additional resources like a VM or container to work with.

Conda installation

First, follow Anaconda's documentation to install it as a user:

https://docs.anaconda.com/anaconda/install/index.html

Then, create a WMCore environment with the dependencies needed for it:

# Installation
conda create -c conda-forge -n wmcore3 python=3.8 mysql gcc python-gfal2 openssl pycurl
conda activate wmcore3
git clone https://github.com/dmwm/WMCore
cd WMCore
# Select a tag, e.g.: 2.1.6
git checkout 2.1.6
# Now, let's remove gfal2 from requirements, so pip does not try to install it, as we will
# provide it via conda
cat requirements.txt | grep -v gfal2 > req.txt
pip install -r req.txt .
pip install reqmgr2ms-rulecleaner
cd ..

The Hack to the requirements.txt file above are due to the fact gfal2 will be installed via conda, but it is not recognized by pip3, so we need to remove the requirement from the corresponding file. This is the same trick that is applied when building the containers used in the K8s infrastructure.

It is advised that you export your proxy certificates for authentication:

voms-proxy-init -voms cms -valid 192:00
export X509_USER_CERT=/tmp/x509up_u$(id -u)
export X509_USER_KEY=/tmp/x509up_u$(id -u)
export X509_USER_PROXY=/tmp/x509up_u$(id -u)

You can configure the RUCIO client as well, like this:

mkdir etc
cat << EOF > etc/rucio.cfg

[common]
[client]
rucio_host = http://cms-rucio-int.cern.ch
auth_host = https://cms-rucio-auth-int.cern.ch
auth_type = x509
ca_cert = /etc/grid-security/certificates/
client_cert = $X509_USER_CERT
client_key = $X509_USER_KEY
client_x509_proxy = $X509_USER_PROXY

EOF

export RUCIO_HOME=$PWD/etc

After that, you should be able to use import WMCore packages and use them, provided that you have all configuration files relevant to the component you want to test. E.g.:

$ conda activate wmcore3
(wmcore3)$ python3
Python 3.8.15 | packaged by conda-forge | (default, Nov 22 2022, 08:49:35)
[GCC 10.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> MSRuleCleaner.
MSRuleCleaner.alertDeletablePU(           MSRuleCleaner.getGlobalLocks(             MSRuleCleaner.setArchivalDelayExpired(
MSRuleCleaner.alertStatusAdvanceExpired(  MSRuleCleaner.getMSOutputTransferInfo(    MSRuleCleaner.setClean(
MSRuleCleaner.archive(                    MSRuleCleaner.getRequestRecords(          MSRuleCleaner.setLogDBClean(
MSRuleCleaner.change(                     MSRuleCleaner.getRucioRules(              MSRuleCleaner.setParentDatasets(
MSRuleCleaner.cleanRucioRules(            MSRuleCleaner.mro(                        MSRuleCleaner.setPlineMarker(
MSRuleCleaner.execute(                    MSRuleCleaner.resetCounters(              MSRuleCleaner.unifiedConfig(
MSRuleCleaner.findTargetStatus(           MSRuleCleaner.sendAlert(                  MSRuleCleaner.updateReportDict(

Example

For example, for MSRuleCleaner, if you have the RUCIO client configured and provide the ReqMgr2Secrets.py and config-ruleCleaner.py (this can be gotten from an already deployed service, for example. This won't be discussed in this guide), you can execute the following script:

https://github.com/dmwm/WMCore/blob/master/bin/adhoc-scripts/updateTotalStats.py

and run the component like this:

(wmcore3) python3 -i -- updateTotalStats.py -c $workPath/config-ruleCleaner.py
# Inside python command line
msRuleCleaner.execute(reqStatus="Completed")