Skip to content

List of PSet tweaks applied by WMAgent during job runtime

Jump to bottom
Kenyi Hurtado edited this page Aug 21, 2020 · 15 revisions

Overview

Pset tweaking is tightly coupled in WMCore, WMControl and CMSSW, and it brings complications that affect the support of multiple python versions (in CMSSW and WMCore) and job configuration (Pset) tweaks applied during runtime.

This topic is meant to be an investigation of what exactly is tweaked by WMAgent - during the job runtime; what the source of the information is (things from the workflow description? things from the WMAgent job description? things based on the site and/or storage? things related to the data to be accessed? etc)

PSet tweaking:

The following scripts are the main ones used in the PSet tweaking:

https://github.com/dmwm/WMCore/blob/master/src/python/PSetTweaks/PSetTweak.py https://github.com/dmwm/WMCore/blob/master/src/python/PSetTweaks/WMTweak.py

The standalone script in development to replace the tweaking part can be seen below: https://gist.github.com/davidlange6/3b0cb365aac669a714d9f288b0bf420d

Job runtime details

  1. WMAgent executes the following script per job: https://github.com/dmwm/WMCore/blob/master/etc/submit.sh

  2. There, Job object is unpacked via Unpacker.py runtime script

  • This creates job directory, unpacks the sandbox and setups environment so job can be called using the Startup runtime script
  • Sandboxes are built via Taskmaker, which calls SandBoxCreator
  1. SandBoxCreator invokes the CMSSWFetcher plugin, which will fetch the config files and Pset tweaks for the step.
  • PSetTweaks are gotten from configCache as a JSON
  • This line reads a json dictionary file and creates a PSetTweak object, which is saved as another JSON file.
  1. Startup.py bootstraps the job and executes it
  1. For a CMSSW executor, prescripts are added here and looked here, being SetupCMSSWPset the only one added by this method. This is where the Pset tweaking is called.

SetupCMSSWPset prescript

  • Each CMSSW step (cmsRun1, cmsRun2, etc) will call SetupCMSSWPset once The command looks like this (note SetupCMSSWPset is called via ScriptInvoke)
2020-07-15 00:03:37,065:INFO:Scram:    Invoking command: /cvmfs/cms.cern.ch/slc6_amd64_gcc491/cms/cmssw/CMSSW_7_4_0/external/slc6_amd64_gcc491/bin/python2.7 -m WMCore.WMRuntime.ScriptInvoke WMTaskSpace.cmsRun1 SetupCMSSWPset
  • Note the command above calls SetupCMSSWPset via ScriptInvoke, which will load the job step object. This what we would need to change in WMCore

The WMTweak.applyTweak needs the following input parameters:

  • Self.process, which is the stepSection object
  • A PSetTweak object
  • A fixupDict dictionary, to make sure Psets and configuration values exist.

Where the pset tweaks are defined at 4 different stages:

  1. Task pset tweaks Input: StepSection of FwjReport object

    • Parameters modified: 
      - process.GlobalTag.globaltag
- process.GlobalTag.DBParameters.transactionId

  2. Job Pset tweaks

    • Input: WMBS Job object
    • Parameters modified: 
      process.source.firstLuminosityBlock
process.source.fileNames
process.source.secondaryFileNames
process.source.firstEvent
process.maxEvents.input
process.source.skipEvents
process.source.lumisToProcess

  3. Output module Pset tweaks

    • Input parameters:
      • Module output, gotten from here
      • WMBS Job object
    • Parameters modified:
    process.<outMod>.fileName
process.<outMod>.logicalFileName
    • Where outMod is the output module, e.g.: .input.outputModule = 'AODSIMoutput'

  4. Random generator tweak

    • Input parameters: None (but uses getBaggage() method from WMBS job object)
    • Parameters modified:
    process.RandomNumberGeneratorService.<randService>.initialSeed
    • Where randService comes from: RandomNumberGeneratorService.x_internal_name

Generation of Pset tweaks

Refer to the following presentation for details about the generation of these tweaks and how they are fetched in WMConfigCache:

https://indico.cern.ch/event/946436/contributions/3977022/attachments/2087948/3507926/Upload%20to%20ReqMgr2%202020%2008%2014.pdf

PSet tweak examples and job environment to reproduce the whole pset tweak stages procedure

The following repository:

https://gitlab.cern.ch/khurtado/psettweakmigration/

provides the following:

  1. A set of PSet tweaks and input files (inside the test directory) for the following stages:
  • Task level, Job level, Output module level

It also provides the output files as gotten by the current Pset tweaks methods. They can be used with a standalone script replacement, as long as python includes the CMSSW release modules (details in the README file in the repository)

  1. A job sandbox environment

This allows the user to run a production job, generate and apply the Pset tweaks interactively. It can be used to modify SetupCMSSWPset with a different applyTweak method, for example.

Please, read the README file for more details on how to use these files.

Clone this wiki locally