setup wmcore virtual environment

Brief summary for python versions:

While writing the current document we are in the process of migrating to python3 and the information regarding python2.7 may very easily go stale really soon, but I will try to cover both setups as much as I can.

Brief summary on eventual deployment target points:

The so described method bellow is to have your WMCore/WMAgent development virtual environment setup and running, where in addition to the major dependency packages you can also setup all the utilities, IDEs and tools that you find useful in making your development process as comfortable and productive as you wish. Giving that this is a virtual environment one may set it up wherever one finds it convenient, but the two most probable places would be either the remote CERN infrastructure or the developer's personal PC. There are pros and cons for both approaches:

• Remote deployment:
  • Con: Because there are still dependencies that exist in WMCore to packages from the OS itself, which may or may not be resolved on the lxplus machines, there is a possibility one to end up using a VM instance from his personal openstack project or a Docker container, so that he has the rights to install the missing packages himself.
  • Con: May require wasting some resources from your personal openstack project at CERN
  • Pro: Your deployment environment lives behind the CERN firewall, which makes a lot of things simpler
  • Pro: You may reach some integration services' REST APIs directly from python and make some single line tests/queries on the go, which speeds things a lot.
  • Pro: You may try to run some of your services directly from the source path where you are doing your development without the need for a full service deployment, just by changing your python source path
  • Pro: You have the ability to patch an already deployed service as it already runs in your VM/Container directly from your development virtual environment (supposed the service is running on the same VM/Container you are deploying your virtual environment at).
• Local deployment:
  • Pro: You have the comfort of being able to work locally and fully offline
  • Pro: You do not need to use VMs or Containers for the setup
  • Con: Some of the CERN services are out of (direct) reach for you since they live behind of the CERN firewall and sometimes you may need to create a tunnel or a socks proxy in order to do so.
  • Con: you cannot patch a live service directly from your development environment. You will have to either make a PR and redeploy or patch the running service using the PR as a patch or in case you want to avoid an early PR you must create a git --diff patch and scp it to the final destination.

Install the python virtual environment:

• python2:

$ git clone https://github.com/dmwm/WMCore.git
$ pip2 install -U virtualenv
$ mkdir WMCore.venv2
$ /usr/bin/python2 -m venv WMCore.venv2

• python3:

$ git clone https://github.com/dmwm/WMCore.git
$ mkdir WMCore.venv3
$ /usr/bin/python3 -m venv WMCore.venv3

And just for keeping track on how the size of the virtual environment grows in the process:

# Installing all WMCore dependencies:
*NOTE:*  The following instruction is a repetition of several trail and error attempts. 
So there is a possibility I have some OS package dependencies resolved in some of the previous tests 
and being well forgotten by the time I write this wiki. So any one who finds such, is 
very welcome to edit the document here directly. Thank you in advance.

### Installing the OS packages dependencies:
Some of the python packages from the WMCore dependency list even though installed 
thorough `pip` are still relying on some OS tools and services, that's why those 
need to be resolved in advance before one starts the python deployment. 
A good example is `pycurl` which uses the system curl library and gcc to recompile 
things linking the shared libraries we have it configured with (e.g. openSSL).

$ sudo yum install gcc python-devel mariadb mariadb-devel openssl openssl-devel

*NOTE: * One needs to find the equivalent package names for the hosting OS he is about to use for this deployment.

(e.g. for Debian 10.7):

$ sudo apt-get update $ sudo apt-get install gcc libpython2-dev python2-dev libpython3-dev python3-dev libmysqlclient-dev libmariadb-dev libcurl4-openssl-dev libssl-dev

### Installing the python packages dependencies:
* python2:

$ cd WMCore $ source ../WMCore.venv2/bin/activate (WMCore.venv2)$ pip install -r requirements.txt

   * You will most probably face the following error:

ERROR: Command errored out with exit status 1:
 command: /home/tivanov/Projects/WMCoreDev.d/WMCore.venv2/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-CyTllw/cx-oracle/setup.py'"'"'; __file__='"'"'/tmp/pip-install-CyTllw/cx-oracle/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-HZLyn6
     cwd: /tmp/pip-install-CyTllw/cx-oracle/
Complete output (5 lines):
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-install-CyTllw/cx-oracle/setup.py", line 170, in <module>
    raise DistutilsSetupError("cannot locate an Oracle software " \
distutils.errors.DistutilsSetupError: cannot locate an Oracle software installation
----------------------------------------

ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

This is because Oracle for WMAgent is a centrally provided (by CERN) service and 
since in private deployments or at FNAL machines we would neve use it but we would 
mariadb as backend instead, the suggested solution would be:

(WMCore.venv2)$ cp requirements.txt requirementsNoOracle.txt (WMCore.venv2)$ sed -e 's/cx-Oracle.*//g' requirementsNoOracle.txt (WMCore.venv2)$ pip cache purge (WMCore.venv2)$ pip install -r requirementsNoOracle.txt

   * In case you run into the following error (happens mostly for Debian due to packages versions mismatch:

creating build/temp.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Dversion_info=(1,2,5,'final',1) -D__version__=1.2.5 -I/usr/include/mariadb -I/usr/include/mariadb/mysql -I/usr/include/python2.7 -c _mysql.c -o build/temp.linux-x86_64-2.7/_mysql.o
In file included from _mysql.c:44:
/usr/include/mariadb/my_config.h:3:2: warning: #warning This file should not be included by clients, include only <mysql.h> [-Wcpp]
 #warning This file should not be included by clients, include only <mysql.h>
  ^~~~~~~
_mysql.c: In function ‘_mysql_ConnectionObject_ping’:
_mysql.c:2005:41: error: ‘MYSQL’ {aka ‘struct st_mysql’} has no member named ‘reconnect’
  if ( reconnect != -1 ) self->connection.reconnect = reconnect;
                                         ^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------

ERROR: Command errored out with exit status 1: /home/tivanov/Projects/WMCoreDev.d/WMCore.venv2/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-F1BUsK/mysql-python/setup.py'"'"'; file='"'"'/tmp/pip-install-F1BUsK/mysql-python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-pWwZYT/install-record.txt --single-version-externally-managed --compile --install-headers /home/tivanov/Projects/WMCoreDev.d/WMCore.venv2/include/site/python2.7/MySQL-python Check the logs for full command output.

The solution should be:

(WMCore.venv2)$ sed '/st_mysql_options options;/a unsigned int reconnect;' /usr/include/mysql/mysql.h -i.bkp

   * The virtual environment size at the end of this step:

(WMCore.venv2)$ du -hs ../WMCore.venv2/ 213M ../WMCore.venv2/

* python3:

$ cd WMCore $ source ../WMCore.venv3/bin/activate (WMCore.venv3)$ pip install -r requirements_py3.txt

   * In case you run into the following error or anything related to missing `wheel` package:

---

Failed building wheel for pycurl Running setup.py clean for pycurl Running setup.py bdist_wheel for rucio-clients ... error Complete output from command /home/tivanov/Projects/WMCoreDev.d/WMCore.venv3/bin/python3 -u -c "import setuptools, tokenize;file='/tmp/pip-install-45k4i4hp/rucio-clients/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d /tmp/pip-wheel-beei9j2l --python-tag cp37: usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: -c --help [cmd1 cmd2 ...] or: -c --help-commands or: -c cmd --help

error: invalid command 'bdist_wheel'

The solution should be:

(WMCore.venv3)$ pip install wheel

   * In case one runs into this error:

rust 0.1.1 has requirement mock<=1.0.1, but you'll have mock 4.0.3 which is incompatible. ... =============================DEBUG ASSISTANCE============================= If you are seeing a compilation error please try the following steps to successfully install cryptography: 1) Upgrade to the latest pip and try again. This will fix errors for most users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip 2) Read https://cryptography.io/en/latest/installation.html for specific instructions for your platform. 3) Check our frequently asked questions for more information: https://cryptography.io/en/latest/faq.html 4) Ensure you have a recent Rust toolchain installed: https://cryptography.io/en/latest/installation.html#rust 5) If you are experiencing issues with Rust for this release only you may set the environment variable CRYPTOGRAPHY_DONT_BUILD_RUST=1. =============================DEBUG ASSISTANCE=============================

error: can't find Rust compiler

If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.

To update pip, run:

    pip install --upgrade pip

and then retry package installation.

If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.

This package requires Rust >=1.41.0.

----------------------------------------

This package requires Rust >=1.41.0.

---

Failed building wheel for cryptography

The solution should be:

(WMCore.venv3)$ pip install --upgrade pip

   * Any of the equivalent errors as in the Python2 environment should have the 
identical solutions too.

   * The virtual environment size at the end of this step:

(WMCore.venv3)$ du -hs ../WMCore.venv3/ 375M ../WMCore.venv3/

# Notes about pycrul and SSL backends:
Since we still need to use SSL for authentication purposes we have to have `pycrl` 
compiled with the same authentication backend library as `libcurl` (openssl in most of the cases),
so it is a must at the end of the deployment procedure to check if this is the case with the mainstream package 
coming with `pip install pycrl`. The error that one should expect at run time should be similar to:

ImportError: pycurl: libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (none/other)

And the exact reason how it happens is highlighted here: 
http://pycurl.io/docs/latest/install.html#ssl


In case one runs into such a `libcurl` - `pycurl` backend mismatch, there are two 
possible situation to this (independent of the python version):

   * Use pycrl with openssl:

(WMCore.venv2)$ sudo yum install python-devel openssl openssl-devel (WMCore.venv2)$ export PYCURL_SSL_LIBRARY=openssl (WMCore.venv2)$ export CPLUS_INCLUDE_PATH=/usr/include/python2.7/ (WMCore.venv2)$ export C_INCLUDE_PATH=/usr/include/python2.7/ (WMCore.venv2)$ export LDFLAGS=-L/usr/local/opt/openssl/lib (WMCore.venv2)$ export CPPFLAGS=-I/usr/include/openssl/ (WMCore.venv2)$ unset LDFLAGS

   * Or in case you need to use pycurl with nss:

(WMCore.venv2)$ pip uninstall pycurl (WMCore.venv2)$ export PYCURL_SSL_LIBRARY=nss (WMCore.venv2)$ pip install --compile --install-option="--with-nss" --no-cache-dir pycurl

# Adding Ipython into your virtual environment directly:
In case you have `ipython` in the OS/VM/Container where you are setting up your development environment, 
you may run into the absurd situation to have a python3 environment but once you try to start `ipython` you see 
the interpreter you start is actually a python2. In this case, and actually in general, it is 
strongly recommended to have the `ipython` package setup inside the virtual environment so that 
any interpreter, packages or library mismatches with version from outside the virtual 
environment may happen. 

Here is the actual mistake visualised:

* python2:

(WMCore.venv2)$ which ipython /home/user/.local/bin//ipython

* python3:

(WMCore.venv3)$ which ipython /home/user/.local/bin//ipython

(WMCore.venv3)$ ipython /home/tivanov/.local/lib/python2.7/site-packages/IPython/core/interactiveshell.py:726: UserWarning: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv. warn("Attempting to work in a virtualenv. If you encounter problems, please " Python 2.7.16 (default, Oct 10 2019, 22:02:15) Type "copyright", "credits" or "license" for more information.

IPython 5.10.0 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details.

In [1]:

The solution in both cases python{2,3} should be:

(WMCore.venv3)$ pip install ipython (WMCore.venv3)$ deactivate

$ source ../WMCore.venv3/bin/activate

(WMCore.venv3)$ which ipython /home/tivanov/Projects/WMCoreDev.d/WMCore.venv3/bin/ipython

(WMCore.venv3)$ ipython Python 3.7.3 (default, Jul 25 2020, 13:03:44) Type 'copyright', 'credits' or 'license' for more information IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]:

   * The virtual environment size at the end of this step:

(WMCore.venv3)$ du -hs ../WMCore.venv3/ 397M ../WMCore.venv3/

# Rucio and Wmcore virtual environment:
In the next step we are about to run a WMCore service from inside the virtual environment. 
We presume the development environment have been deployed on a VM. 
We already have the rucio package installed there from pip and also the Rucio Client
configuration file. Now  we need need to change the configuration at the proper place:

(WMCore.venv3)$ cat << EOF >../WMCore.venv3/etc/rucio.cfg [common] [client] rucio_host = http://cmsrucio-int.cern.ch auth_host = https://cmsrucio-auth-int.cern.ch auth_type = x509 ca_cert = /etc/grid-security/certificates/ client_cert = $X509_USER_CERT client_key = $X509_USER_KEY client_x509_proxy = $X509_USER_PROXY request_retries = 3 EOF

Test the result:

(WMCore.venv3)$ export X509_USER_CERT=/data/auth/dmwm-service-cert.pem (WMCore.venv3)$ export X509_USER_KEY=/data/auth/dmwm-service-key.pem (WMCore.venv3)$ rucio -a wmcore_transferor whoami status : ACTIVE account : wmcore_transferor account_type : SERVICE created_at : 2020-08-18T16:04:03 updated_at : 2020-08-18T16:04:03 suspended_at : None deleted_at : None email : *****@cern.ch

In case you would prefer to use your credentials instead of the wmcore ones then 
in the rucio cfg just use:

auth_type = x509_proxy

And generate your proxy yourself as usual with `voms-proxy-init --voms cms


# Running services locally directly from your source code local repository:
In case you have chosen to setup this environment remotely behind the CERN firewall,
in something like a VM or a Docker Container, you may try to run a WMCore service directly from 
your local code repository. This in theory should be achievable by just changing 
your python path to point to the desired place where you have synced WMCore core 
from github. In practice this is tested and proven to work very well only for Micros Services. 

Here is what you need to do in order to do that:

(WMCore.venv3)$ cat << EOF |tee -a ../WMCore.venv3/bin/activate export PYTHONPATH=</FIXME/PATH/TO/LOCAL/REPO>/WMCore/src/python/:$PYTHONPATH export export RUCIO_HOME=</FIXME/PATH/TO/LOCAL/VENVHOME>/WMCore.venv3/ export X509_USER_CERT=/data/auth/dmwm-service-cert.pem export X509_USER_KEY=/data/auth/dmwm-service-key.pem unset X509_USER_PROXY EOF

Since we are about to run without the central services deployment environment and 
without the configurations coming from there we need to construct an `init.py` 
script so that a minimal set of `msconfig` parameters could be passed to the service. 

(WMCore.venv3)$ mkdir -p ../test/ms-rulecleaner-standalone (WMCore.venv3)$ cat << EOF > ../test/rulecleaner-standalone/init.py import sys import os import pdb import logging

from pprint import pformat

from WMCore.MicroService.Unified.MSRuleCleaner import MSRuleCleaner

FORMAT = "%(asctime)s:%(levelname)s:%(module)s:%(funcName)s(): %(message)s" logging.basicConfig(stream=sys.stdout, format=FORMAT, level=logging.DEBUG) logger = logging.getLogger()

Service config

msConfig = {"enableRealMode": False, "verbose": True, "interval": 1 *60, "services": ['ruleCleaner'], "rucioWmaAcct": "wma_test", "rucioMStrAccount": "wmcore_transferor", "useRucio": True, "rucioAccount": "wma_test", "wmstatsUrl": "https://tivanov-unit02.cern.ch/wmstatsserver", "logDBUrl": "https://tivanov-unit02.cern.ch/couchdb/wmstats_logdb", 'logDBReporter': 'reqmgr2ms_ruleCleaner', 'archiveDelayHours': 8, 'reqmgr2Url': 'https://tivanov-unit02.cern.ch/reqmgr2', 'msOutputUrl': 'https://tivanov-unit02.cern.ch/ms-output', 'reqmgrCacheUrl': 'https://tivanov-unit02.cern.ch/couchdb/reqmgr_workload_cache', 'phedexUrl': 'https://tivanov-unit02.cern.ch/phedex/datasvc/json/prod', 'dbsUrl': 'https://cmsweb-testbed.cern.ch/dbs/int/global/DBSReader', 'couchDBUrl': 'https://cmsweb-testbed.cern.ch/couchdb', 'rucioUrl': 'http://cmsrucio-int.cern.ch', 'rucioAuthUrl': 'https://cmsrucio-auth-int.cern.ch'}

reqStatus = ['announced', 'rejected', 'aborted-completed']

msRuleCleaner = MSRuleCleaner(msConfig) msRuleCleaner.resetCounters()

result = msRuleCleaner._execute(reqRecords)

result = msRuleCleaner.execute(reqStatus) logger.info('Execute result: %s', pformat(result))

EOF

And then we have two options, either run it outside  and just observe/debug the 
results or explore within `ipython`:

(WMCore.venv3)$ ipython test/ms-rulecleaner-standalone/init.py

or:  

(WMCore.venv3) tivanov@tot-t590:WMCore$ ipython Python 3.7.3 (default, Jul 25 2020, 13:03:44) Type 'copyright', 'credits' or 'license' for more information IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: %load_ext autoreload

In [2]: %autoreload 2

In [3]: %run ../test/ms-rulecleaner-standalone/init.py

p.s. We need the `aoutoreload` so that we get assured every time a change in the
source code happens the relevant module will be reloaded at runtime while re-executing
with ipython.  

From here on the Code/TestRun cycles repeats on your pace.

# Patching a live WMCore service direclty from the local working tree:
In case you chose to deploy your virtual environment and develop directly on a 
machine or container which is having the WMCore central services running you can 
patch a running service directly from your working area and just restart it:

(WMCore.venv3)$ git diff --no-color src/python/WMCore/Services/pycurl_manager.py | sudo bin/patchComponent.sh reqmgr2ms Patching component: wmagent Hmm... Looks like a unified diff to me... The text leading up to this was:

|diff --git a/src/python/WMCore/Services/pycurl_manager.py b/src/python/WMCore/Services/pycurl_manager.py |index 75bb01d..d544bae 100644 |--- a/src/python/WMCore/Services/pycurl_manager.py |+++ b/src/python/WMCore/Services/pycurl_manager.py

patching file WMCore/Services/pycurl_manager.py Using Plan A... Hunk #1 succeeded at 99. done

(WMCore.venv3)$ (A=/data/cfg/admin; cd /data; $A/InstallDev -s start:reqmgr2ms)

or in case you have a commit ready to be added to a PR but not yet uploaded upstream.

(WMCore.venv3)$ git show e6bfaf41f3f370201fb1fe656b0c5b82409e04a2 --no-color | sudo bin/patchComponent.sh reqmgr2ms Patching component: wmagent Hmm... Looks like a unified diff to me... The text leading up to this was:

|commit e6bfaf41f3f370201fb1fe656b0c5b82409e04a2 |Author: Todor Ivanov [email protected] |Date: Sat Mar 13 04:38:21 2021 +0100 | | Fix difference between str/bytes in python2 and python3. | |diff --git a/src/python/WMCore/Services/pycurl_manager.py b/src/python/WMCore/Services/pycurl_manager.py |index d544bae..35607be 100644 |--- a/src/python/WMCore/Services/pycurl_manager.py |+++ b/src/python/WMCore/Services/pycurl_manager.py

patching file WMCore/Services/pycurl_manager.py Using Plan A... Hunk #1 succeeded at 60. Hunk #2 succeeded at 68. Hunk #3 succeeded at 82. Hunk #4 succeeded at 100. done

(WMCore.venv3)$ (A=/data/cfg/admin; cd /data; $A/InstallDev -s start:reqmgr2ms)

# Running WMCore with python debugger:

# Including pylint and pep coding style guiding tools to the virtual environment: