Skip to content

List of useful sites for monitoring DUNE data

Heidi Schellman edited this page Jul 11, 2024 · 29 revisions

General DUNE connection issues

https://github.com/DUNE/FAQ/projects/1

issue trackers

https://github.com/DUNE/dune-prod-utils/issues

https://github.com/DUNE/data-mgmt-ops/issues

SPS status

https://op-webtools.web.cern.ch/vistar/?usr=SPS1

Raw Data movement

SPS schedule https://ps-sps-coordination.web.cern.ch/ps-sps-coordination/

Master page for np02/np04 daq https://twiki.cern.ch/twiki/bin/view/CENF/DUNEDAQProto

(Note: see below for how to set up proxies that allow external access to CERN sites.)

NP04 ELISA logbook https://pdsp-elog.app.cern.ch/elisa/display (requires CERN login and membership in an E-group.)

NP02-coldbox logbook https://np-vd-coldbox-elog.app.cern.ch/elisa/display?logbook=np-vd-coldbox-elog (requires CERN login)

Overview of the CERN FTS3 instance:

https://fts3-public.cern.ch:8449/fts3/ftsmon/#/
Grafana for the CERN FTS3 instance: (needs CERN SSO)
https://monit-grafana.cern.ch/d/veRQSWBGz/fts-servers-dashboard
Grafana for just DUNE on CERN FTS3: (needs CERN SSO) https://monit-grafana.cern.ch/d/veRQSWBGz/fts-servers-dashboard?orgId=25&var-group_by=vo&var-vo=dune&var-src_country=All&var-dst_country=All&var-src_site=All&var-source_se=All&var-dst_site=All&var-dest_se=All&var-fts_server=All&var-staging=All&var-bin=1h&var-dest_hostname=All&var-source_hostname=All

IngestD and DeclaD status monitoring offsite

IngestD: https://dune.monitoring.edi.scotgrid.ac.uk/app/dashboards#/view/28002070-2f54-11ef-b9ca-6fc4e3f16d60

DeclaD: https://dune.monitoring.edi.scotgrid.ac.uk/app/dashboards#/view/c97984b0-2f53-11ef-b9ca-6fc4e3f16d60

Metacat

MetaCat demo:

DUNE MetaCat GUI: https://metacat.fnal.gov:9443/dune_meta_demo/app/gui/index METACAT_AUTH_SERVER_URL=https://metacat.fnal.gov:8143/auth/dune_dev METACAT_SERVER_URL=https://metacat.fnal.gov:9443/dune_meta_prod/app

MetaCat production:

DUNE MetaCat GUI: https://metacat.fnal.gov:9443/dune_meta_prod/app/gui/index METACAT_SERVER_URL=https://metacat.fnal.gov:9443/dune_meta_prod/app METACAT_AUTH_SERVER_URL=https://metacat.fnal.gov:8143/auth/dune
DUNE MetaCat GUI: https://metacat.fnal.gov:9443/dune_meta_prod/app/gui/datasets

MetaCat history:

https://dbdata0vm.fnal.gov:8043/service_charts/dune-meta-prod?window=week (SSO)

Rucio

https://lsvip.fnal.gov/monitor/d/000000002/rucio-operational-overview?orgId=1&from=1709889091063&to=1710254544922

Relevant databases

IFBEAM https://dbweb8.fnal.gov:8443/ifbeam/bmon/pdunemon/Display

Run config: NEED TO UPDATE WITH CURRENT WEBSITE

uConDB test

https://dbdata0vm.fnal.gov:9443/dune_con_prod/app/get?table=pdunesp.lifetime_purmon&type=data&tag=v1.0&t0=1541524343&t1=1541697143&columns=center,low,high

Slow controls

Overview talk from database meeting: https://indico.fnal.gov/event/54126/contributions/239184/attachments/154057/200060/slow-cotrols-data_lino_gerlach.pdf

Limited access: detector experts access that data via a website https://np04-slow-control.web.cern.ch/

DCS-DB is accessed via read-only rest API https://gitlab.cern.ch/ep-dt-di/dcsdb-rest

DB itself can only be accessed from within CERN network and CERN strongly discourages the use of vpns for this purpose

THIS LINK IS DEAD AND THAT REPO DOES NOT EXIST NOW https://gitlab.cern.ch/ep-dt-di/dcsdb-rest gave me a 404 error after CERN authentication. -> only a few members have been granted access to that repo -> I contacted one of the maintainers Maybe I need to join a group? We may not need this but having some insight is useful.

General monitoring

FNAL systems status: https://fermi.servicenowservices.com/wp?id=evg-service-status (requires SSO)

Fifemon https://fifemon.fnal.gov/monitor/d/000000004/experiment-overview?from=now-24h&to=now-5m&var-experiment=dune&orgId=1

DUNE CRIC - https://dune-cric.cern.ch/core/experimentsite/list/

EGI https://accounting.egi.eu/wlcg/countries/

ETF https://etf-dune-preprod.cern.ch/etf/check_mk/index.py?start_url=%2Fetf%2Fcheck_mk%2Fview.py%3Fview_name%3Dallhosts

THIS LINK IS BUSTED OSU global map http://fermicloud584.fnal.gov:3000/new under construction

Perfsonar: https://psmad.opensciencegrid.org/maddash-webui/index.cgi?dashboard=DUNE%20Mesh%20Config

Offline Data transfer and storage monitoring

Rucio monitor https://dune.monitoring.edi.scotgrid.ac.uk/app/dashboards#/view/7eb1cea0-ca5e-11ea-b9a5-15b75a959b33?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1d,to:now))

SAM project monitoring https://samweb.fnal.gov:8483/station_monitor/dune/stations/dune/projects

Data Dispatcher project monitoring https://metacat.fnal.gov:9443/dune/dd/gui/P/projects

Enstore monitoring https://www-stken.fnal.gov/enstore/encp_enstore_system.html

Dcache network https://mrtg.fnal.gov/weathermap/wx-protodune.html (requires proxy or VPN)

ESNET plot https://my.es.net/oscars/view/RR94 (changed from the previous /P6FG)

DUNE FTS monitoring: https://dune.monitoring.edi.scotgrid.ac.uk/app/dashboards#/view/c9268890-749c-11ee-a635-c11ccc9f77dd

protoDUNE data pipeline monitoring: https://dune.monitoring.edi.scotgrid.ac.uk/app/dashboards#/view/21a10bf0-0226-11ef-b9ca-6fc4e3f16d60

DAQ machines storage usage: https://dune.monitoring.edi.scotgrid.ac.uk/app/dashboards#/view/6a95ab70-04da-11ef-b9ca-6fc4e3f16d60

Job monitoring

(Heidi’s old link https://wiki.dunescience.org/wiki/Useful_Production_Links)

JustIn https://justin.dune.hep.ac.uk/docs/

JustIn monitoring from JustIn db: https://dune.monitoring.edi.scotgrid.ac.uk/app/dashboards#/view/4717b1e0-175c-11ef-b9ca-6fc4e3f16d60

POMS https://pomsgpvm01.fnal.gov/poms/show_campaigns

Fifebatch - can check out batch sites https://fifemon.fnal.gov/kibana/goto/ee9456df25ce77417eaee9ffa56c4def

offsite running jobs for last 24 hours: https://lsvip.fnal.gov/monitor/d/000000053/experiment-batch-details?orgId=1&var-experiment=dune&viewPanel=10

jobs efficiency for last 24 hours: https://fifemon.fnal.gov/monitor/d/000000069/user-efficiency-details?orgId=1&from=now-7d&to=now-5m&var-user=dunepro

jobs distribution among countries: https://fifemon.fnal.gov/kibana/goto/79b9c2e5e06828809d3a7ea410045bff

Issues/Complaints

Slack channel https://dunescience.slack.com/archives/C6GFAPG4U

Service Now https://fermi.servicenowservices.com/wp

Service now service catalog: (restricted) https://fermi.servicenowservices.com/nav_to.do?uri=%2Flist_service_areas.do%3Fdivision%3Dscientific_computing

github area for issues? DUNE Data Challenge 4

https://github.com/DUNE/dist-comp/labels/Operations

GGUS http://cern.ch/go/p6s8

Reporting

FCRSG Plots: https://fifemon.fnal.gov/monitor/d/r6UDhH-iz/sppm-sc-pmt-prep?orgId=1&var-expName=DUNE&var-graphiteQuery=%7Bdune,DUNE,marslbne,lbne%7D&var-esQuery=(dune%20OR%20DUNE%20OR%20marslbne%20OR%20lbne)&var-NASQuery=(dune*%20OR%20DUNE*%20OR%20marslbne*%20OR%20lbne*)&var-dCacheQuery=(dune%7CDUNE%7Cmarslbne%7Clbne)&var-enstoreQuery=(dune%7CDUNE%7Cmarslbne%7Clbne)

General Documentation

FNAL db map https://dbweb0.fnal.gov/redirector/index.html

Workflow https://dune.github.io/dune-wfs/

Metacat documentation https://dbdata0vm.fnal.gov:9443/dune_meta_demo/app/static/docs/html/index.html

DUNE Computing training https://dune.github.io/computing-training-basics/index.html

DUNE wiki

Current wiki https://wiki.dunescience.org/wiki/DUNE_Computing (edit this one)

Development wiki https://wiki.dunescience.org/wiki/DUNE_Computing_Wiki_Dev (do not edit, just comment) Accessing websites behind VPN’s

How to do proxies

Here is an example of syntax for accessing particular sites at CERN and FNAL

ssh [email protected] -L 8009:p3s-content.cern.ch:80 -L 8008:p3s-web.cern.ch:80 -L 8080:protodune-ftslight7.cern.ch:8080 -L 8123:np04-srv-001.cern.ch:8123

Simplified without p3s-web (which is now defunct)

Ssh [email protected] -L 8080:protodune-ftslight7.cern.ch:8080

ssh -l $FNALUSER dunegpvm01.fnal.gov -L 8787:dunesamgpvm02.fnal.gov:80 -L 8788:mrtg.fnal.gov:8788

OUR LOGBOOK

THIS SHOULD EITHER BE NUKED OR START TO GET USED Logbook for data challenge https://dbweb8.fnal.gov:8443/ECL/dunepro/E/index?category_id=16 ask Heidi to become a member if you cannot create entries.

OLD pages

Old: https://wiki.dunescience.org/wiki/Setting_up_monitoring_pages which pointed to many of the pages.

Need experts to suggest a list of the sites and suggestions on how to get proxies to access. Old page was https://wiki.dunescience.org/wiki/Data_Movement_Web_Pages_-_detailed_documentation but most links are now busted.