Skip to content

trouble shooting

Jump to bottom
Alan Malta Rodrigues edited this page Feb 10, 2016 · 36 revisions

WMAgent trouble shoot

Workqueue is not acquiring request: acquired status but not moving to running open status.

  1. Check WorkQueueManager Component Log.
  2. If the error message is showing json parse error, wq view is corrupted. 
    ERROR:WorkQueueManagerWMBSFileFeeder:Error in wmbs inject loop: unterminated array starting at position 0:
    1. shutdown the agent, and shutdown couch server
    2. remove the view from /data1/database/.workqueue_design
    3. start the couch server and rebuild the view by triggering one of the view 
      curl http://localhost:5984/workqueue/_design/WorkQueue/_view/availableByPriority
    4. If the view rebuild is finished, start the agent.

Note, however, in case this issue is happening to all the agents, then it probably is an issue with central workqueue.

Datasets were produced with None either for AcquisitionEra or ProcessingString.

  1. In these cases these blocks and files will bug DBS3Upload all the time since it does not pass the Lexicon validation in DBS. So, what we need to do is basically to close those blocks and mark them (and its files) as injected in dbsbuffer tables. 1. Shutdown PhEDExInjector and DBS3Upload 2. Gather a list of workflow names and their bad output dataset names: SELECT DISTINCT dbsbuffer_workflow.name, dbsbuffer_dataset.path FROM dbsbuffer_dataset INNER JOIN dbsbuffer_dataset_subscription ON dbsbuffer_dataset.id = dbsbuffer_dataset_subscription.dataset_id INNER JOIN dbsbuffer_block ON dbsbuffer_block.dataset_id = dbsbuffer_dataset_subscription.dataset_id INNER JOIN dbsbuffer_file ON dbsbuffer_file.block_id = dbsbuffer_block.id INNER JOIN dbsbuffer_workflow ON dbsbuffer_workflow.id = dbsbuffer_file.workflow WHERE dbsbuffer_block.blockname LIKE '/%/None-%' AND dbsbuffer_block.status != 'Closed'; 3. Find the blocks and files that needs manual intervention (just to keep record) SELECT * FROM dbsbuffer_block WHERE blockname LIKE '/%/None-%' AND status!='Closed'; SELECT * FROM dbsbuffer_file WHERE lfn LIKE '/store/%/None/%' AND status!='InDBS'; 4. Close and mark them as injected UPDATE dbsbuffer_block SET status='Closed' WHERE blockname LIKE '/%/None-%' AND status!='Closed'; UPDATE dbsbuffer_file SET status='InDBS', in_phedex='1' WHERE lfn LIKE '/store/%/None/%' AND status!='InDBS'; 5. elog them in the workflow team.

  2. If there are children of these None samples (usually correctly named), they will fail DBS injection because their parent information is missing as well. Those will also have to be marked as injected, we better perform the following procedure for them: 1. Find the exact block name in the DBS3Upload logs 2. Then we mark all its files as injected in DBS and PhEDEx (replace BLOCKNAME in the query below): UPDATE dbsbuffer_file SET status='InDBS', in_phedex='1' WHERE block_id=(SELECT id FROM dbsbuffer_block WHERE blockname='BLOCKNAME' AND status!='Closed') AND status!='InDBS'; 3. Finally, we mark the same block as Closed (replace BLOCKNAME in the query below): UPDATE dbsbuffer_block SET status='Closed' WHERE blockname='BLOCKNAME' AND status!='Closed';

Clone this wiki locally