DataManagement

5. Data Management Basic

5.1 Getting information

5.1.1 SE availability

• The first thing is to know which Storage Elements are available for the VO, run the command:

  dirac-dms-show-se-status
  

  For example:

  bash-4.1$ dirac-dms-show-se-status
  ProductionSandboxSE                Active          Active
  IN2P3-disk                         Active          Active
  DIRAC-USER                         Active          Active
  KEK-disk                           Active          Active
  KEK2-disk                          Active          Active
  CPPM-disk                          Active          Active
  

  SE names are defined in the DIRAC Configuration. These are the names that you will use with various data management commands.

5.2 Uploading a file to the Grid

• The next step is to upload a file to a Storage Element and register it into the DIRAC File Catalog. Execute the command:

  dirac-dms-add-file <LFN> <FILE> <SE>
  

  Output must look like this:

  $ dirac-dms-add-file /vo.france-asia.org/user/v/vhamar/TestPekin-2.txt test-121113 CPPM-disk
  Uploading /vo.france-asia.org/user/v/vhamar/TestPekin-2.txt
  Successfully uploaded file to CPPM-disk
  

  Note: The output of this command must be successful before continuing with other exercises.

5.3 Obtaining information about the data

5.3.1 Metadata

• After a file is registered into DIRAC File Catalog the metadata could be consulted any time with:

  dirac-dms-catalog-metadata <LFN>
  

  For example, the metadata for TestPekin-2.txt file is:

  $ dirac-dms-catalog-metadata  /vo.france-asia.org/user/v/vhamar/TestPekin-2.txt
  

5.3.2 File Metadata

• More detailed file metadata can be obtained with the following command:

  dirac-dms-lfn-metadata <LFN>
  

  For example:

  {'Failed': {},
  'Successful': {'/vo.france-asia.org/user/v/vhamar/TestPekin-2.txt': {'Checksum': 'b525b3bb',
                                          'ChecksumType': 'Adler32',
                                          'CreationDate': datetime.datetime(2013, 11, 12, 13, 34, 27),
                                          'FileID': 62L,
                                          'GID': 2,
                                          'GUID': '69F783D6-59AB-B753-A698-E662F8B89850',
                                          'Mode': 509,
                                          'ModificationDate': datetime.datetime(2013, 11, 12, 13, 34, 27),
                                          'Owner': 'vhamar',
                                          'OwnerGroup': 'dirac_user',
                                          'Size': 1128L,
                                          'Status': 1,
                                          'UID': 2}}}
  

5.4 Downloading a file

• Retrieve the file previously uploaded to the Grid using the command:

  dirac-dms-get-file <LFN>
  

  Output must be like shown below:

  $ dirac-dms-get-file /vo.france-asia.org/user/v/vhamar/TestPekin-2.txt
  {'Failed': {},
  'Successful': {'/vo.france-asia.org/user/v/vhamar/TestPekin-2.txt': '/afs/in2p3.fr/home/h/hamar/DIRAC-v6r10/TestPekin-2.txt'}}
  

5.5 Data Replication

5.5.1 Replicating a file

• The command used to create another replica of a given file:

  dirac-dms-replicate-lfn <LFN> <SE>
  

  For example:

  $ dirac-dms-replicate-lfn /vo.france-asia.org/user/v/vhamar/TestPekin-2.txt DIRAC-USER
  {'Failed': {},
  'Successful': {'/vo.france-asia.org/user/v/vhamar/TestPekin-2.txt': {'register': 1.1856651306152344,
                                                                      'replicate': 13.616289138793945}}}
  

5.5.2 Replica information

• The following command allows to obtain the replica information for the given file:

  dirac-dms-lfn-replicas <LFN>
  

  An example ouput is shown below:

  $ dirac-dms-lfn-replicas  /vo.france-asia.org/user/v/vhamar/TestPekin-2.txt
  {'Failed': {},
   'Successful': {'/vo.france-asia.org/user/v/vhamar/TestPekin-2.txt': {
   'CPPM-disk': 'srm://marsedpm.in2p3.fr:8446/srm/managerv2?SFN=/dpm/in2p3.fr/home/vo.france-asia.org/user/v/vhamar/TestPekin-2.txt',
   'DIRAC-USER': 'dips://dirac.ihep.ac.cn:9148/DataManagement/StorageElement/vo.france-asia.org/user/v/vhamar/TestPekin-2.txt'}}}
  

5.5.3 Removing a replica

• To remove replicas use the command:

  dirac-dms-remove-replicas <LFN> <SE>
  

  For example::

  $ dirac-dms-remove-replicas /vo.formation.idgrilles.fr/user/v/vhamar/Test-Lyon.txt IBCP-disk Successfully removed DIRAC-USER replica of /vo.formation.idgrilles.fr/user/v/vhamar/Test-Lyon.txt

5.6 Removing Files

• Please remove all the files created during the T.P, using this command:

  dirac-dms-remove-files <LFN>
  

  For example:

  $ dirac-dms-remove-files  /vo.formation.idgrilles.fr/user/v/vhamar/Test-Lyon.txt
  $
  

5.7 Getting the list of user files

• To create a list of all the files stored by the user:

  dirac-dms-user-lfns
  

  After running the command a file with all the LFNs will be created, the file name is associated with the user's VO:

  $ dirac-dms-user-lfns
   /vo.formation.idgrilles.fr/user/v/vhamar: 0 files, 1 sub-directories
   /vo.formation.idgrilles.fr/user/v/vhamar/2: 0 files, 3 sub-directories
   /vo.formation.idgrilles.fr/user/v/vhamar/2/2389: 1 files, 0 sub-directories
   /vo.formation.idgrilles.fr/user/v/vhamar/2/2390: 1 files, 0 sub-directories
   /vo.formation.idgrilles.fr/user/v/vhamar/2/2391: 1 files, 0 sub-directories
   3 matched files have been put in vo.formation.idgrilles.fr-user-v-vhamar.lfns