-
Notifications
You must be signed in to change notification settings - Fork 107
ReqMgr2 MicroService APIs
This wiki will list the RESTP APIs for all WMCore Microservices. They return data in json, so it requires the header "Accept: application/json" to be set.
- MSTransferor APIs
- MSMonitoring APIs
- MSOutput APIs
- MSUnmerged APIs
- MSPileup APIs
- MSPileup tasks run as independent service (daemon) which provides data placement to MSPileup workflows. This daemon does not have external APIs exposed to end-users.
This microservice is responsible for the workflow input data placement. It has CouchDB as a backend database, where it persists the workflow transfers summary and keeps updating the transfers completion in specified intervals. Once all the data placement requests have been made, the workflow is moved to the next status staging.
This returns very basic information of the service, like its status and version.
One can also provide the detail=true query string in order to retrieve a few more details of the service. This extra information contains data regarding its last execution cycle.
GET /ms-transferor/data/status HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
An example of a curl command to read the agent configuration is:
curl --cert $X509_USER_CERT --key $X509_USER_KEY -X GET https://cmsweb.cern.ch/ms-transferor/data/status?detail=true
This API can be used to retrieve information for a given workflow. The information returned corresponds to the transfer document created when handling the input data to be transferred. It includes the dataset names, transfer request IDs, transfer completion, etc. Note that those documents are only available while the workflow is active, basically between staging and running-* statuses.
GET /ms-transferor/data/info?request=WORKFLOW_NAME HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
An example of a curl command to read the agent configuration is:
curl --cert $X509_USER_CERT --key $X509_USER_KEY -X GET https://cmsweb.cern.ch/ms-transferor/data/info?request=pdmvserv_task_B2G-RunIISummer16NanoAODv6-02873__v1_T_200331_212615_94
This microservice is responsible for monitoring the data transfer requests created by MSTransferor. It keeps updating the local couch document with the transfers completion in a periodic cycle (X hours), once transfers have completed, the workflow is moved to the next status staged).
This returns very basic information of the service, like its status and version.
One can also provide the detail=true query string in order to retrieve a few more details of the service. This extra information contains data regarding its last execution cycle.
GET /ms-monitor/data/status HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
An example of a curl command to read the agent configuration is:
curl --cert $X509_USER_CERT --key $X509_USER_KEY -X GET https://cmsweb.cern.ch/ms-monitor/data/status?detail=true
This API is equivalent to the ms-transferor info API.
Thus, it can be used to retrieve information for a given workflow. The information returned corresponds to the transfer document created when handling the input data to be transferred. It includes the dataset names, transfer request IDs, transfer completion, etc.
Note that those documents are only available while the workflow is active, basically between staging and running-* statuses.
GET /ms-monitor/data/info?request=WORKFLOW_NAME HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
An example of a curl command to read the agent configuration is:
curl --cert $X509_USER_CERT --key $X509_USER_KEY -X GET https://cmsweb.cern.ch/ms-monitor/data/info?request=pdmvserv_task_B2G-RunIISummer16NanoAODv6-02873__v1_T_200331_212615_94
This microservice is responsible for the workflow output data placement. It has MongoDB as a backend database, where it persists a workflow record for the output data placement. This microservice fetches workflows in the status closed-out or announced.
This returns very basic information of the service, like its status and version.
One can also provide the detail=true query string in order to retrieve a few more details of the service. This extra information contains data regarding its last execution cycle, like the number of requests processed/consumed in the service's last cycle.
GET /ms-output/data/status HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
An example of a curl command to read the agent configuration is:
curl --cert $X509_USER_CERT --key $X509_USER_KEY -X GET https://cmsweb.cern.ch/ms-output/data/status?detail=true
This API can be used to retrieve information for a given workflow. The information returned contains information regarding the output data produced, amount of Disk copies made, destinations, replication rule IDs, others.
Once all the output data transfers have been successfully done, the workflow record is marked as TransferStatus=done (instead of pending) status.
Note that such workflow records should only be created for workflows that went through either the closed-out or the announced status.
GET /ms-output/data/info?request=WORKFLOW_NAME HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
An example of a curl command to read the agent configuration is:
curl --cert $X509_USER_CERT --key $X509_USER_KEY -X GET https://cmsweb.cern.ch/ms-output/data/info?request=pdmvserv_RVCMSSW_12_0_0_pre5BuMixing_14_210730_160440_5107
This microservice is responsible for the unmerged area cleanup for the sites storage elements.
This service is executed with 3 different classes - corresponding to 3 different services/replicas in the Kubernetes realm - in order to contact each of those replicas, we have initially implemented a specific query string, where:
-
rse_type=t1: will fetch information from the MSUnmerged replica responsible for all the T1 storage elements; -
rse_type=t2t3us: will fetch information from the MSUnmerged replica responsible for all the T2 and T3 storage elements, in US (United States); -
rse_type=t2t3: will fetch information from the MSUnmerged replica responsible for all the remaining T2 and T3 storage elements.
Currently the rse_type option is used only int the status interface, as shown bellow.
This returns very basic information of the service, like its status and version.
One can also provide the detail=true query string in order to retrieve a few more details of the service. This extra information contains data regarding the service itself and RSE cleanup, collected at the last execution cycle of the service.
GET /ms-unmerged/data/status?rse_type=t2t3 HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
An example of a curl command to read the agent configuration is:
curl --cert $X509_USER_CERT --key $X509_USER_KEY -X GET https://cmsweb.cern.ch/ms-unmerged/data/status?&rse_type=t2t3
This API can be used to retrieve information for a given storage element (RSE). For directory detailed information, one needs to provide detail=true as a query string, in addition to the RSE name. At its current state.
GET /ms-unmerged/data/info?&rse=RSE_NAME HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
An example of a curl command to read the agent configuration is:
curl --cert $X509_USER_CERT --key $X509_USER_KEY -X GET https://cmsweb.cern.ch/ms-unmerged/data/info?rse=T2_IT_Rome&detail=true
This micro-service is responsible for handling Premix pileup data. It has the following REST APIs:
These set of APIs are used to fetch pileup data from the micro-service. The main endpoint is /ms-pileip/data/access, e.g.
curl -v "http://localhost:8249/ms-pileup/data/access?pileupName=pileup"
curl -v "http://localhost:8249/ms-pileup/data/access?campaign=Campaign"
and, we can use filters to extract certain fields from the doc, e.g.
curl -v "http://localhost:8249/ms-pileup/data/access?pileupName=pileup&filters=["campaigns", "pileupType"]"
The POST APIs have several roles:
- create document in MSPileup service
curl -v -X POST -H "Content-type: application/json" -d '{"bla":1}' \
http://localhost:8249/ms-pileup/data/access
- fetch data via provided JSON query (spec)
curl -v -X POST -H "Content-type: application/json" \
-d '{"query":{"pileupName":"bla"}}' \
http://localhost:8249/ms-pileup/data/access
- fetch data by using query and filters
curl -v -X POST -H "Content-type: application/json" \
-d '{"query":{"pileupName": "bla"}, "filters":["campaigns"]}' \
http://localhost:8249/ms-pileup/data/access
The PUT HTTP API are used to update specific data in MSPileup service, e.g.
curl -v -X PUT -H "Content-type: application/json" -d '{"bla":1}' \
http://localhost:8249/ms-pileup/data/access
The DELETE APIs are used to delete specific data in MSPileup service, e.g.
curl -v -X DELETE -H "Content-type: application/json" -d '{"bla":1}' \
http://localhost:8249/ms-pileup/data/access
curl -v -X DELETE -H "Content-type: application/json" -d '{"pileupName":"pileup"}' \
http://localhost:8249/ms-pileup/data/access