List of PSet tweaks applied by WMAgent during job runtime

Overview

Pset tweaking is tightly coupled in WMCore, WMControl and CMSSW, and it brings complications that affect the support of multiple python versions (in CMSSW and WMCore) and job configuration (Pset) tweaks applied during runtime.

This topic is meant to be an investigation of what exactly is tweaked by WMAgent - during the job runtime; what the source of the information is (things from the workflow description? things from the WMAgent job description? things based on the site and/or storage? things related to the data to be accessed? etc)

PSet tweaking:

The following scripts are the main ones used in the PSet tweaking:

https://github.com/dmwm/WMCore/blob/master/src/python/PSetTweaks/PSetTweak.py https://github.com/dmwm/WMCore/blob/master/src/python/PSetTweaks/WMTweak.py

The standalone script in development to replace the tweaking part can be seen below: https://gist.github.com/davidlange6/3b0cb365aac669a714d9f288b0bf420d

Job runtime details

WMAgent executes the following script per job: https://github.com/dmwm/WMCore/blob/master/etc/submit.sh

There, Job object is unpacked via Unpacker.py runtime script This creates job directory, unpacks the sandbox and setups environment so job can be called using the Startup runtime script Sandboxes are built via TaskMaker, which calls SandBoxCreator SandBoxCreator invokes the CMSSWFetcher plugin, which will fetch the config files and Pset tweaks for the step. PSetTweaks are gotten from configCache as a JSON This line reads a json dictionary file and creates a PSetTweak object, which is saved as another JSON file. Startup.py bootstraps the job and executes it The execute method will look and use the proper executor to run here (e.g.: the CMSSW executor) For a CMSSW executor, prescripts are added here and looked here, being SetupCMSSWPset the only one added by this method. This is where the Pset tweaking is called.