Production System

• Authors: L.Arrabito, J. Bregeon, AT

Following the discussion at the 2017 DUW, we propose the development of a new System, called 'Production System' (PS). The goal is to develop a system with similar objectives of the LHCb production system, but keeping it as general and as simple as possible to be adopted by several communities. The idea is to build a system on the top of the Transformation System, to further automatise the management of large 'productions', where a production is a set of transformations 'linked' together by input/output data. In order to define a Production, we also propose some enhancement in the transformation definition.

1. Transformation System enhancement

The current transformation definition should be enhanced with new attributes:

• InMQ optional (Input Metaquery). NEW attribute of type: MetaQuery. It would replace the current 'fileMask'. It's optional since a transformation could have no input data, as for MC simulation.
• OutMQ optional (Output Metaquery). NEW attribute of type: MetaQuery. The jobs created by the transformation will produce some data with some associated metadata. This attribute represents a set of metadata with their expected ranges of values.
• OutputMetaData optional. NEW attribute of type: dict. The jobs created by the transformation will produce some data with some associated metadata. This attribute represents the set of metadata key-value pairs (the ones that are known in advance).

• Note that these are regular metadata (as in DFC), it's just that one has the opportunity to define these at the transformation level
• Note that there is a kind of redundancy with "OutMQ". It should be discussed whether it’s necessary to keep these 2 distinct attributes or only the OutMQ.

In order to keep backward compatibility we could start by adding the new attributes as optional.

2. Production System

• Production definition ** A set of 'linked' transformations: {T0, T1, T2, ...} *** 2 transformations Ti, Tj are 'linked' if some of the output data of Ti is an input of Tj, and this is handled by the production system via the input and ouput meta query of each transformation (there must be an intersection between OutMQ,j and InMQ,i or between OutMQ,i and InMQ,j). ** Main production attributes : ProductionID, ProductionName, Status
• Production Execution : start/monitor/stop
• Production Validator ** Invoked during Production definition or modificaton (i.e. when adding a new transformation to the production) ** Checks intersections between MetaQueries of 2 supposed ‘linked’ transformations, e.g.: OutMQ,i and InMQ,j ** Allows to establish if 2 transformations are 'linked'

2.1 Architecture

In order to give a more precise idea of the PS desired functionalities and of the proposed architecture, we have done a very preliminary implementation of the PS and the of the TS enhancement, based on v6r19:

https://github.com/arrabito/DIRAC/tree/ProdSys_v6r19

Here below a short description of the currently implemented components.

ProductionDB :

• 1 table containing production definitions (Productions table)
• 1 table containing the associations between transformations and productions (ProductionTransformations table) ProductionManager Service :
• Exposes methods to create/delete/monitor productions (manipulating the Productions table)
• Exposes methods to associate transformations to productions (manipulating the ProductionTransformations table)

Production Client Exposes all the methods to manage productions A transformation can be added to a production with or without a ‘parent’ transformation. A transformation added to a production with a parent transformation is a transformation whose InMQ interesects with the OutMQ of the parent transformation. For the transformations belonging to a production, we call 2 transformations ‘linked’, if one is the parent of the other or vice-versa.

Utilities

• ProdTransManager : uses the TS and the PS clients. It manages the transformations associated to the productions
• ProdValidator : uses the TS client and the FileCatalog. It validates the production definitions.

Some limitations of the current implementation The production validation done by the ProdValidator Utility is extremely simplistic. • It just checks if InMQ is a subset of OutMQ of 2 supposed linked transfotmations or vice-versa. This logic should be improved. • Only '=' and 'in' operators are accepted for the moment in the In/OutMQ of the transformations belonging to a production • For the production validation, we don’t use for the moment the OutMetaData attribute of the transformations, but only the OutMQ (and the InMQ).

The progress of a production is not yet defined.

3. Tests and use cases

The integration test below :

https://github.com/arrabito/DIRAC/blob/ProdSys_v6r19/tests/Integration/ProductionSystem/Test_Client_TS_Prod.py

illustrates how the user would use the ProductionClient (toghether with the TransformationClient) to create a new production. In this test, we also show the role of the Production Validator, which checks if the production definition is valid. Note that a transformation can have several parent transformations. In this case the ProdValidator checks that all the ‘links’ are valid. On the other hand, several transformations can have the same parent transformation.

Here below we give an example of how the user would interact with the PS to define a simple production made of 2 transformations (t1 : MC Simulation -> t2 : DataProcessing), using the TS and the PS clients. In this example, the user should :

1. Create 2 transformations using the TS client, also defining the corresponding In/OutQueries.
2. Create an ‘empty’ production with a given name, using the Production Client.
3. Add the first transformation (t1) to the production, without specifying any parent transformation.
4. Add the second transformation (t2) to the production, specifying that it has a parent transformation (t1). At this point the ProdValidator checks that the 2 transformations are ‘linked’. If not, the production is not valid and an error is returned.

Later on the user can start/stop/delete a production and inspect the transformations is composed of (with their parent relations).

The monitoring of the progress of a production should also be included.