Skip to content

Injecting, assigning and validating test requests

Jump to bottom
Alan Malta Rodrigues edited this page Mar 1, 2023 · 9 revisions

Injecting and assigning requests

This is a quick tutorial aimed for the creation and assignment of test requests into cmsweb(-testbed) or in your own private machine. In order to create these requests, you need to:

  1. have a valid proxy in your environment (X509_USER_PROXY)
  2. download this script: https://github.com/dmwm/WMCore/blob/master/test/data/ReqMgr/inject-test-wfs.py

Injection and assigning workflows

In order to inject and/or assign test workflows using this WMCore script, you need to first setup your X509 certificate or proxy variable, e.g.:

export X509_USER_CERT=/data/srv/current/auth/reqmgr2/dmwm-service-cert.pem
export X509_USER_KEY=/data/srv/current/auth/reqmgr2/dmwm-service-key.pem

or use your own proxy for this:

export X509_USER_PROXY=/data/cert/myproxy.pem

Now you need to download the inject-test-wfs.py script from WMCore:

curl https://raw.githubusercontent.com/dmwm/WMCore/master/test/data/ReqMgr/inject-test-wfs.py > inject-test-wfs.py

and finally you can create a workflow in cmsweb-testbed with the following command:

python3 inject-test-wfs.py -u "https://cmsweb-testbed.cern.ch" -m Integration -f TaskChainPyquenZeemumuJets_PU.json -c Agent105_Validation -r Agent105_CDB16_forceComplete -t testbed-dev -a Integ_Test -p forceComplete_Alan_TEST_v13

In summary, inject-test-wfs.py will perform the following actions:

  • it will clone WMCore repository (or it can use a local WMCore copy)
  • it will use the json templates available under "WMCore/test/data/ReqMgr/requests/", either from DMWM or Integration directories.
  • in general, DMWM templates should create less and smaller jobs, so you get faster results
  • the Integration templates are meant for full WMAgent or CMSWEB validations, usually longer jobs.
  • it will create AND assign requests for you (against ReqMgr2, by default), according to the parameters given in the command line
  • you can also only inject workflows, skipping the assignment part.

Some useful examples (don't forget to adapt parameters according to your needs):

1. Injecting and assigning ALL templates under DMWM directory, against cmsweb-testbed

python3 inject-test-wfs.py -m DMWM -c DMWM_Test -r Test_bugFix_4scriptInject -t testbed-cmssrv95 -a DMWM_Test -p bugFix_TEST_injectScript_v4

where:

  • -m: mode, either DMWM or Integration directories
  • -c: campaign name (so you can look for this campaign in WMStats)
  • -r: request string (how your workflow will be named, excluding the usual username and timestamp)
  • -t: team name (the agent you want to assign this workflow), the team name can be found by visiting wmstats->agent info tab in wmstats
  • -a: acquisition era (for TaskChain workflows, it will use the CMSSWVersion from the template - due to DQM harvesting rules...)
  • -p: processing string (for TaskChain workflows, it will append your string provided to the TaskName)

1. Injecting and assigning a specific template against my private services:

python3 inject-test-wfs.py -u "https://alan-cloud1.cern.ch" -m Integration -f TaskChainPyquenZeemumuJets_PU.json -c Agent105_Validation -r Agent105_CDB16_forceComplete -t alan-devvm -a Integ_Test -p forceComplete_Alan_TEST_v13

1. Only injecting a specific workflow but do not assigning it:

python3 inject-test-wfs.py -m DMWM -f MonteCarlo_eff.json -c Agent105_Validation -r Agent105_CDB16_forceComplete --injectOnly

Validating requests

When these workflows move to 'completed' status, then we can use a secondary script to do some request validation against couch/dbs/phedex. It would check the input x output lumi section, events, number of files and etc.

Here is how you can use this script to validate a workflow (you need to have the env variable set):

> curl https://raw.githubusercontent.com/dmwm/WMCore/master/test/data/ReqMgr/validate-test-wfs.py > validate-test-wfs.py
> python3 validate-test-wfs.py -r alan-cloud1.cern.ch -w WORKFLOW_NAME -v

or, you can pass a text file with the workflow names you want to validate, like
> python3 validate-test-wfs.py -i FILE_NAME -v

where:

  • -w: a workflow name
  • -i: a input text file
  • -c: Url for DBS/PhEDEx (it defaults to cmsweb-testbed)
  • -r: Url for the reqmgr instance (it defaults to cmsweb-testbed)
  • -v: enables verbose mode
Clone this wiki locally