Further issues with k4run

[Zehvogel]

See discussion in: #134

Two further issues popped up:

• list comprehensions in option files stopped working, fixed by: k4run: fix more bugs #140
• Some FCC option file causes another error that looks like a propName exists more than once, to be investigated further in this issue

[giovannimarchiori]

Hi @Zehvogel
this is how I setup the environment

export FCCBASEDIR=$PWD
source /cvmfs/sw-nightlies.hsf.org/key4hep/setup.sh                                                                                                                                                            

export K4RECCALORIMETER=$PWD/k4RecCalorimeter/install/share/k4RecCalorimeter
export K4SIMGEANT4=$PWD/k4SimGeant4/install/share/k4SimGeant4
export FCCDETECTORS=$PWD/FCCDetectors/

# with K4WCORE                                                                                                                                                                                                  
export K4FWCORE=$PWD/k4FWCore/install/share/k4FWCore
export PATH=$PWD/k4RecCalorimeter/install/bin/:$PWD/k4FWCore/install/bin/:$PWD/k4SimGeant4/install/bin:$PATH
export CMAKE_PREFIX_PATH=$PWD/k4RecCalorimeter/install:$PWD/k4FWCore/install/:$PWD/k4SimGeant4/install/:$PWD/FCCDetectors/install/:$CMAKE_PREFIX_PATH
export LD_LIBRARY_PATH=$PWD/k4RecCalorimeter/install/lib:$PWD/k4RecCalorimeter/install/lib64:$PWD/k4FWCore/install/lib:$PWD/k4SimGeant4/install/lib:$PWD/k4SimGeant4/install/lib64:$PWD/FCCDetectors/install/lib:$PWD/FCCDetectors/install/lib64:$LD_LIBRARY_PATH
export PYTHONPATH=$PWD/k4RecCalorimeter/install/python:$PWD/k4FWCore/install/python:$PWD/k4SimGeant4/install/python:$PYTHONPATH

then make a fresh build and run with

cd LAr_scripts/FCCSW_ecal/
fccrun runTopoAndSlidingWindowAndCaloSim.py 

Output:

/home/gmarchio/work/fcc-main/FCCDetectors/
Traceback (most recent call last):
  File "/home/gmarchio/work/fcc-main/k4FWCore/install/bin/k4run", line 131, in <module>
    add_arguments(parser, ApplicationMgr())
  File "/home/gmarchio/work/fcc-main/k4FWCore/install/bin/k4run", line 65, in add_arguments
    parser.add_argument( f"--{propName}", f"--{propNameReversed}", type=proptype, help=props[prop][1],
  File "/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-06-24/x86_64-almalinux9-gcc11.3.1-opt/python/3.10.10-4lh2y5/lib/python3.10/argparse.py", line 1441, in add_argument
    return self._add_action(action)
  File "/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-06-24/x86_64-almalinux9-gcc11.3.1-opt/python/3.10.10-4lh2y5/lib/python3.10/argparse.py", line 1807, in _add_action
    self._optionals._add_action(action)
  File "/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-06-24/x86_64-almalinux9-gcc11.3.1-opt/python/3.10.10-4lh2y5/lib/python3.10/argparse.py", line 1643, in _add_action
    action = super(_ArgumentGroup, self)._add_action(action)
  File "/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-06-24/x86_64-almalinux9-gcc11.3.1-opt/python/3.10.10-4lh2y5/lib/python3.10/argparse.py", line 1455, in _add_action
    self._check_conflict(action)
  File "/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-06-24/x86_64-almalinux9-gcc11.3.1-opt/python/3.10.10-4lh2y5/lib/python3.10/argparse.py", line 1592, in _check_conflict
    conflict_handler(action, confl_optionals)
  File "/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-06-24/x86_64-almalinux9-gcc11.3.1-opt/python/3.10.10-4lh2y5/lib/python3.10/argparse.py", line 1601, in _handle_conflict_error
    raise ArgumentError(action, message % conflict_string)
argparse.ArgumentError: argument --SimG4Svc.SimG4FullSimActions.OutputLevel/--OutputLevel.SimG4Svc.SimG4FullSimActions: conflicting option strings: --SimG4Svc.SimG4FullSimActions.OutputLevel, --OutputLevel.SimG4Svc.SimG4FullSimActions

[Zehvogel]

Looks like SimG4FullSimActions('SimG4Svc.SimG4FullSimActions') exists more than once in ApplicationMgr().allConfigurables.values()

(FCCDataSvc('EventDataSvc'), GenAlg('GenAlg'), MomentumRangeParticleGun('ToolSvc.ParticleGun_Electron'), MomentumRangeParticleGun('GenAlg.ParticleGun_Electron'), HepMCToEDMConverter('HepMCToEDMConverter'), GeoSvc('GeoSvc'), SimG4FullSimActions('SimG4Svc.SimG4FullSimActions'), SimG4FullSimActions('SimG4Svc.SimG4FullSimActions'), SimG4Svc('SimG4Svc'), SimG4ConstantMagneticFieldTool('ToolSvc.SimG4ConstantMagneticFieldTool'), SimG4SaveCalHits('ToolSvc.saveECalBarrelHits'), SimG4SaveCalHits('ToolSvc.saveECalEndcapHits'), SimG4SaveCalHits('ToolSvc.saveHCalBarrelHits'), SimG4PrimariesFromEdmTool('ToolSvc.EdmConverter'), SimG4PrimariesFromEdmTool('SimG4Alg.EdmConverter'), SimG4Alg('SimG4Alg'), CalibrateInLayersTool('ToolSvc.CalibrateECalBarrel'), CalibrateCaloHitsTool('ToolSvc.CalibrateHCal'), CalibrateCaloHitsTool('ToolSvc.CalibrateECalEndcap'), CalibrateCaloHitsTool('ToolSvc.CalibrateHCalEndcap'), CalibrateInLayersTool('CreateECalBarrelCellsStep1.CalibrateECalBarrel'), CreateCaloCells('CreateECalBarrelCellsStep1'), RedoSegmentation('ReSegmentationEcal'), CreateCaloCells('CreateECalBarrelCells'), CellPositionsECalBarrelTool('ToolSvc.CellPositionsECalBarrel'), CreateCaloCellPositionsFCCee('ECalBarrelPositionedCells'), CalibrateCaloHitsTool('CreateHCaloCells.CalibrateHCal'), CreateCaloCells('CreateHCaloCells'), CalibrateCaloHitsTool('CreateEcalEndcapCaloCells.CalibrateECalEndcap'), CreateCaloCells('CreateEcalEndcapCaloCells'), CreateEmptyCaloCellsCollection('CreateEmptyCaloCells'), CaloTowerTool('ToolSvc.towers'), CreateCaloClustersSlidingWindow('CreateClusters'), CreateCaloCellPositionsFCCee('ECalBarrelPositionedCaloClusterCells'), CorrectCaloClusters('correctCaloClusters'), CaloTopoClusterInputTool('ToolSvc.CreateTopoInput'), TopoCaloNeighbours('ToolSvc.ReadNeighboursMap'), TopoCaloNoisyCells('ToolSvc.ReadNoisyCellsMap'), CaloTopoClusterInputTool('CreateTopoClusters.CreateTopoInput'), TopoCaloNeighbours('CreateTopoClusters.ReadNeighboursMap'), TopoCaloNoisyCells('CreateTopoClusters.ReadNoisyCellsMap'), CellPositionsECalBarrelTool('CreateTopoClusters.CellPositionsECalBarrel'), CaloTopoClusterFCCee('CreateTopoClusters'), CreateCaloCellPositionsFCCee('ECalBarrelPositionedCaloTopoClusterCells'), CorrectCaloClusters('correctCaloTopoClusters'), PodioOutput('out'), ChronoAuditor('ChronoAuditor'), AuditorSvc('AuditorSvc'), EventCounter('event_counter'), ApplicationMgr('ApplicationMgr'))

I updated #140 which should restore the old behaviour for now.

However, I think we should investigate if the duplication itself is not a bug. Any thoughts on that @BrieucF, @kjvbrt?

[BrieucF]

Hi, not sure whether it is a bug or not but the duplication seems to be due to the fact that all tools declared in the steering file (even if not used by any algorithm) are picked up by the ApplicationMgr. When passed to e.g. SimG4Svc in this case, it is somehow picked up a second time. An additional piece of information is that this seems to happen only because SimG4Svc is passed as an ExtSvc of ApplicationMgr (tools passed to algorithm belonging to TopAlg appear only once in app_mgr.allConfigurables.values()).

[Zehvogel]

@giovannimarchiori can you confirm if #140 solves your issues now?

[giovannimarchiori]

Hi @Zehvogel
yes, #140 solves the issues
Thanks,
Giovanni