Checking Ingest Daemon Works

The Ingest Daemon runs as the np04data user. Currently on protodune-ingestd-np04.cern.ch and protodune-ingestd-np02.cern.ch Despite the names of these machines, each one can handle data from both np02 and np04 if necessary.

All data management people have rights to log into this VM as root and as np04data.

Ingest daemon is installed in the /home/np04data/FTS-light/product_3.8.1 directory. It does not auto-start on reboot right now

np04data 32191     1 12 Jun22 ?        2-02:13:03 python3 Mover.py -c mover.cfg -l /tmp/ingest-daemon.log

If it needs a restart

cd /home/np04data/FTS-light/bin
./start.sh

If you need to stop it, just kill the Mover.py process

Can verify the GUI is running by

• CERN and Fermilab: http://protodune-gw.cern.ch:8080/ingest-daemon-np04
• CERN and Fermilab: http://protodune-gw.cern.ch:8080/ingest-daemon-np02

Elsewhere have to tunnel as described here Making an ssh tunnel to CERN (and the NGINX server)

• Everywhere else: http://127.0.0.1:8080/ingest-daemon-np04
• Everywhere else: http://127.0.0.1:8080/ingest-daemon-np02

Look at the Charts section to see recent transfers. note that the graphs show your local timezone by default, whereas the logs show everything in UTC.

Logs are in the same working directory plus also in /tmp/ingest-daemon.log

If you look at the chart you will see it is not uncommon to have a bunch of red failed transfers at 21:00 each night, this has been happening when the fts delegation expires.

Syntax of how to do the delegation can be found in

/etc/cron.d/get_ticket_np04data

Can monitor FTS3 transfers generated by the ingest daemon at https://fts3-public.cern.ch:8449

protoDUNE data pipeline monitoring: https://dune.monitoring.edi.scotgrid.ac.uk/app/dashboards#/view/21a10bf0-0226-11ef-b9ca-6fc4e3f16d60

IngestD: https://dune.monitoring.edi.scotgrid.ac.uk/app/dashboards#/view/28002070-2f54-11ef-b9ca-6fc4e3f16d60

DeclaD: https://dune.monitoring.edi.scotgrid.ac.uk/app/dashboards#/view/c97984b0-2f53-11ef-b9ca-6fc4e3f16d60