Merge pull request #217 from Mu2e/v06-dev

Merge in v06-dev branch

.gitignore

@@ -1 +1,9 @@
 *~
+*.art
+*.root
+*.log
+*.so
+*.d
+*.pcm
+*__pycache__
+backing

.muse

@@ -2,4 +2,5 @@
 ROOT_LIBRARY_PATH
 
 PYTHONPATH utils
+PYTHONPATH utils/helper
 PATH bin

README.md

@@ -1,18 +1,32 @@
-# TrkAna
+# EventNtuple
 
 ## Introduction
 
-TrkAna is an event-based ntuple for Mu2e analyses. Each entry in the ntuple corresponds to a single Mu2e event. 
+The EventNtuple is a simplified data format that can be used for Mu2e analyses.
 
+## Structure
+The EventNtuple structure is complex.
 
-## Useful Links
+The help understand what all the branches and leaves mean, we have an [```ntuplehelper```](doc/ntuplehelper.md) tool
 
-* [Mu2eWiki page](https://mu2ewiki.fnal.gov/wiki/TrkAna)
-* [utility to get explanations of branch and leaf names](tutorial/pages/helper.md)
-* [version history of TrkAna](https://mu2ewiki.fnal.gov/wiki/TrkAna#Version_History_.2F_TrkAna_Musings)
-* [tutorial](tutorial/README.md) (some but not all updated for v5)
-* [instructions for developers](https://mu2ewiki.fnal.gov/wiki/TrkAna#For_Developers)
-* [instructions for validation](validation/README.md)
+## Utilities
+To help with analyzing the EventNtuple given its complex structure, we have two sets of utilities:
+* [RooUtil](utils/rooutil/README.md) for ROOT-based analyses, and
+* [PyUtil](utils/pyutil/README.md) for python-based analyses.
 
-## Becoming obsolete
-* [spreadsheet explaining branch and leaf names](https://docs.google.com/spreadsheets/d/1IMZVgj74vxGTWqAFRtFEEcYChujJyc38VcopZh52jhc/edit)
+Both of these are currently undergoing rapid development. Feel free to use these and let us know your thoughts.
+
+## Validation
+Validation scripts and instructions are [here](validation/README.md)
+
+## Previous Versions
+The version history of EventNtuple is [here](https://mu2ewiki.fnal.gov/wiki/EventNtuple).
+
+### Upgrading from v5 to v6
+* For analyzers: a list of branch/leaf/name changes is [here](doc/v5-to-v6.md)
+* For developers: instructions to update your development area are [here](doc/v5-to-v6_developers.md)
+
+## Other Useful Links
+
+* [Mu2eWiki page](https://mu2ewiki.fnal.gov/wiki/EventNtuple)
+* [tutorial](tutorial/README.md)

bin/ntuplehelper

@@ -1,20 +1,20 @@
 #!/bin/python3
 
 import argparse
-from ntuplehelper import nthelp
+from ntuplehelper import nthelper
 
 # A main function so that this can be run on the command line
 def main():
     parser = argparse.ArgumentParser(
         prog='ntuplehelper',
-        description='A python utility class to get information about branches and leaves',
+        description='A python utility class to get information about branches and leaves in the Mu2e EventNtuple',
         epilog='For help, post to the #trkana Slack channel')
 
-    parser.add_argument('--branches', required=True, nargs='*', help="Use format \"branch.leaf\". Multiple items are allowed. Using a \"*\" for the leaf will print description of all leaves for the branch")
+    parser.add_argument('branches', nargs='*', help="Use format \"branch.leaf\". Multiple items are allowed. Using a \"*\" for the leaf will print description of all leaves for the branch")
 
     args = parser.parse_args()
 
-    nth = nthelp()
+    nth = nthelper()
     nth.whatis(args.branches)
 
 

tutorial/pages/helper.md → doc/ntuplehelper.md

@@ -1,31 +1,31 @@
 # How to get help -- ```ntuplehelper```
 
-On this page, you will learn how to use the [```ntuplehelper```](../../utils/ntuplehelper.py) utility to get more information about what the different branches and leaves are. You will need to be in a TrkAna environment e.g.
+On this page, you will learn how to use the [```ntuplehelper```](../../utils/ntuplehelper.py) utility to get more information about what the different branches and leaves are. You will need to be in a EventNtuple environment e.g.
 
 ```
 mu2einit
-muse setup TrkAna
+muse setup EventNtuple
 ```
 
 You can use this utility:
 
 * on the command line:
 
 ```
-ntuplehelper --branches dem.nhits demmcsim.*
+ntuplehelper dem.nhits demmcsim.*
 ```
 
 * on the ROOT command lineL
 ```
-root[0] .! ntuplehelper --branches dem.nhits demmcsim.*
+root[0] .! ntuplehelper dem.nhits demmcsim.*
 ```
 
 * from within python:
 
 ```
 >>> import ntuplehelper
->>> nthelp = ntuplehelper.nthelp()
->>> nthelp.whatis(['dem.hits', 'demmcsim.*'])
+>>> nthelper = ntuplehelper.nthelper()
+>>> nthelper.whatis(['dem.hits', 'demmcsim.*'])
 ```
 
 The above produces this output:

doc/v5-to-v6.md

@@ -0,0 +1,76 @@
+# Major Changes from v5 to v6
+
+There was a major update when we moved from v5 to v6. Here are tables of changes for you to consult. Note that the same change may appear in multiple tables since it may be important in one area but defined in another. For example, the folder name is defined in fcl but is also important to know in the Tree / Branch / Leaf Changes.
+
+Instructions for developers to upgrade their development areas are [here](v5-to-v6_developers.md)
+
+## Tree / Branch / Leaf Changes
+|   | v5 | v6 | Notes |
+|---|---|---|---|
+| tree name | ```trkana``` | ```ntuple``` | defined in code |
+| folder name | ```TrkAna``` | ```EventNtuple``` | defined in fcl |
+| folder name | ```TrkAnaExt``` | ```EventNtuple``` | defined in fcl |
+| branch | ```dem```, ```uem```, etc. | ```trk``` | all track types are now in a single branch |
+| branch | ```demfit```, ```uemfit```, etc. | ```trksegs``` | track segments |
+| branch | ```demtrkqual``` | ```trkqual``` | otherwise it would have been ```trktrkqual```|
+| branch | ```demtch``` | ```trkcalohit``` | |
+| branch | ```demtchmc``` | ```trkcalohitmc``` | |
+| branch | ```hcnt``` | ```hitcount``` | also now filled |
+| branch | ```demtsh``` | ```trkhits``` | |
+| branch | ```demtshmc``` | ```trkhitsmc``` | |
+| branch | ```demtsm``` | ```trkmats``` | mats = materials |
+| branch | ```helixinfo``` | ```helices``` | also now a vector branch |
+| branch | ```demlh``` | ```trksegpars_lh``` | pars = parameters, lh = LoopHelix |
+| branch | ```demch``` | ```trksegpars_ch``` | pars = parameters, ch = CentralHelix |
+| branch | ```demkl``` | ```trksegpars_kl``` | pars = parameters, kl = KinematicLine |
+| branch | ```demmcssi``` | ```trksegsmc``` | information of the surface steps at same places as trksegs|
+
+## Fcl File Changes
+Many fcl files were renamed, updated, or deleted. New fcl files introduced in v6 and all fcl files are documented on the new [fcl README page](../fcl/README.md)
+
+| fcl file | v5 --> v6 | notes |
+|----|---|----|
+| TrkAnaReco.fcl | from_mcs-mockdata.fcl | for running on mock datasets, ```genCountLogger``` also removed  |
+| TrkAnaExtracted.fcl | from_mcs-extracted.fcl | for running on mcs "extracted position" datasets, now #includes from_mcs-mockdata.fcl as a base |
+| TrkAnaReco_ceSimReco.fcl | from_mcs-ceSimReco.fcl | for running on the output of ceSimReco.fcl |
+| TrkAnaRecoEnsemble-Data.fcl | from_mcs-mockdata_noMC.fcl | for running on mock data with no MC information |
+| TrkAnaDeM.fcl | only #includes updated |  |
+| TrkAnaDeMFromDigis.fcl | only #includes updated |  |
+| TrkAnaLineFromDigis.fcl | only #includes updated |  |
+| TrkAnaReflections.fcl | only #includes updated |  |
+| TrkAnaTrigger.fcl | only #includes updated |  |
+| TrkAnaTriggerFromDigis.fcl | only #includes updated |  |
+| TrkAnaRecoEnsemble-MC.fcl | deleted | use from_mcs-mockdata.fcl instead|
+| TrkAnaReco_mergedKalSeeds.fcl | deleted | all track types in one branch is now the default |
+| CrvExpert.fcl | deleted | |
+| TrkAnaReco_wTrkQualFilter.fcl | deleted | |
+| TrkAnaReco_addMCUpstream.fcl | deleted | |
+| TrkAnaReco_addStopTgtVDs.fcl | deleted | |
+| TrkAnaReco_multipleBestCrv.fcl | deleted | |
+| TrkAnaReco_multipleBestCrv_differentThresholds.fcl | deleted | |
+| TrkAnaReco_MultipleTrkQual.fcl | deleted | |
+
+## Fcl Parameter Changes 
+|  | v5 | v6 | Notes |
+|---|--|--|---|
+| folder name | ```TrkAna``` | ```EventNtuple``` | |
+| default output file name | ```nts.owner.trkana-reco.version.sequencer.root``` | ```nts.owner.description.version.sequencer.root``` | |
+| prolog.fcl table name | ```TrkAnaTreeMaker``` | ```EventNtupleMaker``` | the table that contains the default module configuration |
+| prolog.fcl physics block | ```TrkAnaReco``` | ```EventNtuple``` | the block that contains ```producers```, ```analyzers```, etc. |
+| prolog.fcl path names | ```Sequence``` | ```Path``` | for consistency with previous stages of processing|
+| EventNtupleMaker module parameter | ```ProcessEmptyEvents``` | removed | not used anywhere |
+| EventNtupleMaker module parameter | ```required``` | removed | not used anywhere |
+| ```FillHits``` default value | ```false``` | ```true``` | changed in prolog.fcl |
+
+
+## Other Code Changes
+
+|  | v5 | v6 | Notes |
+|---|--|--|---|
+| src file name | ```TrkAnaTreeMaker_module.cc``` | ```EventNtupleMaker_module.cc``` | |
+| art module name | ```TrkAnaTreeMaker``` | ```EventNtupleMaker``` |  |
+| tree name | ```trkana``` | ```ntuple``` | defined in code |
+| art module function | ```fillAllInfos``` | ```fillTrackBranches``` | more descriptive |
+| art module code | ```ntuple->Branch("evtinfo.")``` | ```ntuple->Branch("evtinfo")``` | period removed for better access in RooUtil |
+| art module code | ```ntuple->Branch("evtinfomc.")``` | ```ntuple->Branch("evtinfomc")``` | period removed for better access in RooUtil |
+| info struct | ```TrkFitInfo``` | ```TrkSegInfo``` | to match new branch name, also other changes in module (e.g. ```_allTFIs``` to ```_allTSIs```)|

doc/v5-to-v6_developers.md

@@ -0,0 +1,18 @@
+# Updating your TrkAna v5 area to an EventNtuple v6 area
+
+Rename your own fork on GitHub. Go to settings and change the repository name under "General"
+
+From your working directory before doing ```muse setup```:
+
+```
+mv TrkAna/ EventNtuple/
+cd EventNtuple/
+git remote set-url origin [email protected]:YourUsername/EventNtuple
+git remote set-url mu2e https://github.com/Mu2e/EventNtuple
+cd ../
+rm -rf build/XXX/ # if you like
+mu2einit
+muse setup
+muse build -j4 --mu2eCompactPrint
+```
+

fcl/README.md

@@ -0,0 +1,22 @@
+# Fcl Files
+## Naming Convention
+The naming convention for fcl files is:
+
+```
+from_tier-type_extra.fcl
+```
+
+where ```tier``` is the data tier of the input dataset, ```type``` is the type of dataset (e.g. primary-only, extracted position), and ```extra``` gives some extra information (optional)
+
+## Table of Fcl Files
+
+| fcl file | runs on | additional info |
+|----------|-----|-----|
+| from_mcs-mockdata.fcl | mock datasets | |
+| from_mcs-mockdata_noMC.fcl | mock datasets | doesn't include MC in output |
+| from_mcs-extracted.fcl | reconstructed extracted position datasets | |
+| from_mcs-primary.fcl | reconstructed primary (i.e. no background hits) datasets | |
+| from_mcs-mixed.fcl | reconstructed mixed (i.e. primary+background hits) datasets | |
+| from_mcs-ceSimReco.fcl | output of ceSimReco.fcl | |
+| from_mcs-mockdata_separateTrkBranches.fcl | mock datasets | example on how to separate the tracks into separate branches again|
+| from_mcs-mockdata_selectorExample.fcl | mock datasets | example on how to use a selector to select certain types of tracks |

fcl/TrkAnaDeM.fcl

@@ -1,7 +1,7 @@
 # Minimal job to run TrkAna on the 'best' downstream electrons. Runs on mcs files
 #include "Offline/fcl/minimalMessageService.fcl"
 #include "Offline/fcl/standardServices.fcl"
-#include "TrkAna/fcl/prolog.fcl"
+#include "EventNtuple/fcl/prolog.fcl"
 
 process_name: TADeM
 source : { module_type : RootInput }
@@ -30,7 +30,7 @@ physics :
   TriggerPath : [ MergeKKDeM, MakeSS ]
   analyzers : {
     TrkAnaDeM : {
-      @table::TrkAnaTreeMaker
+      @table::EventNtupleMaker
       FitType : LoopHelix
       diagLevel : 2
       FillMCInfo : true

fcl/TrkAnaDeMFromDigis.fcl

@@ -1,6 +1,6 @@
 # Minimal job to reconstructed downstream electrons from digis and analyze the output with TrkAna
 # #include this script in a subscript which specifies the database (ie
-##include "TrkAna/fcl/TrkAnaDeMFromDigis.fcl"
+##include "EventNtuple/fcl/TrkAnaDeMFromDigis.fcl"
 #services.DbService.purpose: MDC2020_best
 #services.DbService.version: v1_3
 #services.DbService.verbose : 2
@@ -9,7 +9,7 @@
 #include "Offline/fcl/standardServices.fcl"
 #include "Production/JobConfig/reco/prolog.fcl"
 #include "Offline/Mu2eKinKal/fcl/prolog.fcl"
-#include "TrkAna/fcl/prolog.fcl"
+#include "EventNtuple/fcl/prolog.fcl"
 
 process_name: TADeMDigis
 source : { module_type : RootInput }
@@ -63,7 +63,7 @@ physics :
   ]
   analyzers : {
     TrkAnaDeM : {
-      @table::TrkAnaTreeMaker
+      @table::EventNtupleMaker
       FitType : LoopHelix
       diagLevel : 2
       FillMCInfo : true

fcl/TrkAnaLineFromDigis.fcl

@@ -4,7 +4,7 @@
 #include "Offline/fcl/standardServices.fcl"
 #include "Production/JobConfig/reco/prolog.fcl"
 #include "Offline/Mu2eKinKal/fcl/prolog.fcl"
-#include "TrkAna/fcl/prolog.fcl"
+#include "EventNtuple/fcl/prolog.fcl"
 
 process_name: TALineDigis
 source : { module_type : RootInput }
@@ -35,7 +35,7 @@ physics :
   RecoPath : [ @sequence::Reconstruction.LineRecoMCPath ]
   analyzers : {
     TrkAnaLine : {
-      @table::TrkAnaTreeMaker
+      @table::EventNtupleMaker
       FitType : KinematicLine
       diagLevel : 2
       FillMCInfo : true