Home

Welcome to the WMCore wiki, powered by GitHub.

Introduction

This wiki is meant to provide a description as complete as possible of the different applications of the Workflow Management system (ReqMgr2, WMStats, GlobalWorkQueue, WMAgent) on what they do and how they do it. (WMCore library is also used for other applications, DBS,CRABServer CRABClient and WmAgentScript

If you are a developer, this should help you get on track to start making changes to the code but you must also checkout the developers tutorials for DMWM. Finally, there is no better documentation than the code itself if you want to work with the project, you'll likely spend many hours just going through it even after reading this.

If you are a non-developer, this will give you basic overview on WMAgent System.

General information for both users (data operators) and developers

Workflow Management System Overview

The following pages contain detailed information about the different projects and how they work.

WM Overview

WMAgent

ReqMgr

WorkQueue

WMStats

Some important logics and concept on the system.

This section contains special pages which describe a particular phenomenon in the system, usually the information here can be inferred after reading the project pages completely.

• Workflow creation and assignment definition

• Request Types and Arguments

• TaskChain vs StepChain

• StepChain Parentage

• How is a request divided to jobs and submitted?

• How is the location of data resolved?

• Deterministic Pileup

• TimePerEvent convention

• Job priority handling in Agent system

Installation (Deployment):

Deploy agent (assume condor is already installed and connected to global pool. For Oracle backend, oracle account should already set up.

• WMAgent deployment script
• How to patch production machine
• DMWM deploy document

Usage and APIs

• Injecting, assigning and validating test requests
• Store result requests
• ReqMgr2 APIs
• WMStats APIs
• ReqMgr2 MicroService APIs (not yet in production)

Operation related documents

• various operation documents
• workqueue couch db replication procedure
• WMAgent monitoring documentation
• Global WorkQueue monitoring documentation
• ReqMgr2 monitoring documentation

Trouble shooting

• WMAgent/ReqMgr/WorkQueue trouble shooting

Information for Developers

Development Tutorial

• New WMCore member setup
• Setting up WMCore developing environment using github
• Useful git commands

Test setup

• How to set up unittest env (linux)
• How to set up reqmgr, workqueue, wmstats in cern vm
• All about Jenkins and our automated tests
• How to patch production/test machine (apply patches to WMAgent)
• Injecting, assigning and validating test requests
• Debugging, log locations
• Give access to other users to your vm

Issue tracking and release procedure

• Tagging and Releasing
• Creating issues
• release document

CMS Development Tools and Support

• DMWM tutorial
• DMWM build document
• DMWM deploy document

Some Implementation details

• WMBS (db backend for WMAgent)
• CouchDB Diagram
• Workqueue replication procedure
• JobSplitting Argorithms

On going developments

DMWM deployment transition (packaging with PyPi and using docker)

• Improving WMAgent deployment (temporary transitional plan)
• DWWM packaging with PyPi
• WMCore with Docker

Python3 Transition

• list of things need to be done manually
• List of futurize stage2 fixes

PhEDEx to Rucio Transition

• PhEDEx to Rucio Transition
• Rucio data management plan

WMAgent Refactoring (dividing jobs in work units)

• Overview
• Implementation

Unified Porting

• Plan

Go transition

• WorkQueue GO implementation

Miscellaneous

• Building a new python spec file with dependency
• Profiling the memory usage of a function
• WMCore dependencies on other projects (need to be updated)

Deprecated and/or outdated documents

Just keeping them for the information and reference purposed or need to be updated and put in a proper category.

• Request creation requirement
• How to set up all-in-one test
• BigCouch setup
• Accessing the log files from cmsweb
• Draining Steps
• Deployment legacy instructions