Standard workflow

The Standard workflow refers to the default workflow used on grid sites. Behind the scenes, it relies on internal python queues to keep track of the job objects, and consists of multiple steps executed in parallel using threads. Each thread polls a queue until it gets a job object to process; after processing, the result is put in another queue for further processing and the thread starts polling its input queue again. The job object itself, is an entity that contains all necessary information about running the payload such as software release version, parameters for payload setup, transfer type of input files, etc.

Specifically, in this workflow the Pilot performs payload download, setup, stage-in, execution and stage-out, along with various verifications, monitoring and server job updates at selectable intervals.

The figure above shows the internal generic flow of job objects. A downloaded job is inserted into the jobs queue and ends up in either the finished_jobs or failed_jobs queue. The completed_jobs queue contains a copy of a job object that has completed running (finished or failed) and is used for internal bookkeeping.