Interpolation From Plot File

[mikaelm123]

Hello,

I attempted to start from a plt file and a chk file while changing the grid resolution by doubling it or halving it. Each time I started from a different resolution from either file it throws a Segfault error. However, starting from those files without changing the resolution works just fine. Are there any additional steps to take besides the ones listed above to use the interpolation feature?

Thanks in advance.

I did look through the other posts concerning this and followed those steps which is what led me to the above errors. However, since the post was answered I thought it might get lost. Sorry if this double post is unnecessary.

[baperry2]

What case are you running? Does it have EB?

PeleC utilizes the PltFileManager utility in PelePhysics to restart from plt files. This utility does not support restarting with a coarser base grid than the plt file from which you are restarting. However, it should support restarting with a finer grid, and it appears there may be a bug in the code for that, which is causing your segfault for this case. Note also that PeleC does not support changing the base grid when restarting from chk files.

I did some quick tests and I am able to restart from a plt file in the EB-ConvergingNozzle case, but not in the PMF case. @marchdf and @jrood-nrel I think this has to do with when the EB ifdefs were removed from PeleC. Commenting out the EB ifdef here in AMReX InterpFromCoarseLevel appears to solve the issue.

[marchdf]

That's interesting... but I am not sure I follow the solution. We always compile with AMREX_USE_EB=TRUE. But we are careful about passing the factory around and maybe that's not happening here.... what's the segfault? Can you share a backtrace?

[baperry2]

To be clear, the problem is fixed for non-EB cases by commenting out lines 1010-1015 of the file linked above.

The full backtrace for the Segfault is:

Host Name: el1
=== If no file names and line numbers are shown below, one can run
addr2line -Cpfie my_exefile my_line_address
to convert my_line_address (e.g., 0x4a6b) into file name and line number.
Or one can use amrex/Tools/Backtrace/parse_bt.py.

=== Please note that the line number reported by addr2line may not be accurate.
One can use
readelf -wl my_exefile | grep my_line_address'
to find out the offset for that line.

0: ./PeleC3d.gnu.ex() [0x648ad6]
amrex::BLBackTrace::print_backtrace_info(_IO_FILE*)
../../../Submodules/AMReX/Src/Base/AMReX_BLBackTrace.cpp:196:36

1: ./PeleC3d.gnu.ex() [0x64a386]
amrex::BLBackTrace::handler(int)
../../../Submodules/AMReX/Src/Base/AMReX_BLBackTrace.cpp:96:15

2: /lib64/libc.so.6(+0x36400) [0x7f1d7ef86400]

3: ./PeleC3d.gnu.ex() [0x780c6b]
amrex::FabArrayamrex::EBCellFlagFab::ParallelCopy_nowait(amrex::FabArrayamrex::EBCellFlagFab const&, int, int, int, amrex::IntVect const&, amrex::IntVect const&, amrex::Periodicity const&, amrex::FabArrayBase::CpOp, amrex::FabArrayBase::CPC const*, bool) inlined at ../../../Submodules/AMReX/Src/Base/AMReX_FabArrayCommI.H:264:5 in amrex::EB2::Level::fillEBCellFlag(amrex::FabArrayamrex::EBCellFlagFab&, amrex::Geometry const&) const
../../../Submodules/AMReX/Src/Base/AMReX_FabArrayCommI.H:325:17
amrex::FabArrayamrex::EBCellFlagFab::ParallelCopy(amrex::FabArrayamrex::EBCellFlagFab const&, int, int, int, amrex::IntVect const&, amrex::IntVect const&, amrex::Periodicity const&, amrex::FabArrayBase::CpOp, amrex::FabArrayBase::CPC const*)
../../../Submodules/AMReX/Src/Base/AMReX_FabArrayCommI.H:264:5
amrex::FabArrayamrex::EBCellFlagFab::ParallelCopy(amrex::FabArrayamrex::EBCellFlagFab const&, int, int, int, int, int, amrex::Periodicity const&, amrex::FabArrayBase::CpOp)
../../../Submodules/AMReX/Src/Base/AMReX_FabArray.H:831:10
amrex::EB2::Level::fillEBCellFlag(amrex::FabArrayamrex::EBCellFlagFab&, amrex::Geometry const&) const
../../../Submodules/AMReX/Src/EB/AMReX_EB2_Level.cpp:440:26

4: ./PeleC3d.gnu.ex() [0x71d1c8]
amrex::EBDataCollection::EBDataCollection(amrex::EB2::Level const&, amrex::Geometry const&, amrex::BoxArray const&, amrex::DistributionMapping const&, amrex::Vector<int, std::allocator >, amrex::EBSupport)
../../../Submodules/AMReX/Src/EB/AMReX_EBDataCollection.cpp:28:49

5: ./PeleC3d.gnu.ex() [0x7008d9]
std::vector<int, std::allocator >::~vector() inlined at ../../../Submodules/AMReX/Src/Base/AMReX_Vector.H:25:7 in amrex::makeEBFabFactory(amrex::Geometry const&, amrex::BoxArray const&, amrex::DistributionMapping const&, amrex::Vector<int, std::allocator > const&, amrex::EBSupport)
/nopt/nrel/apps/base/2020-05-12/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.4.0-xrmnckz436rnkcwooooi2r7n35pojkov/include/c++/8.4.0/bits/stl_vector.h:567:15
amrex::Vector<int, std::allocator >::~Vector()
../../../Submodules/AMReX/Src/Base/AMReX_Vector.H:25:7
void __gnu_cxx::new_allocatoramrex::EBDataCollection::construct<amrex::EBDataCollection, amrex::EB2::Level const&, amrex::Geometry const&, amrex::BoxArray const&, amrex::DistributionMapping const&, amrex::Vector<int, std::allocator > const&, amrex::EBSupport&>(amrex::EBDataCollection*, amrex::EB2::Level const&, amrex::Geometry const&, amrex::BoxArray const&, amrex::DistributionMapping const&, amrex::Vector<int, std::allocator > const&, amrex::EBSupport&)
/nopt/nrel/apps/base/2020-05-12/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.4.0-xrmnckz436rnkcwooooi2r7n35pojkov/include/c++/8.4.0/ext/new_allocator.h:136:4
void std::allocator_traits<std::allocatoramrex::EBDataCollection >::construct<amrex::EBDataCollection, amrex::EB2::Level const&, amrex::Geometry const&, amrex::BoxArray const&, amrex::DistributionMapping const&, amrex::Vector<int, std::allocator > const&, amrex::EBSupport&>(std::allocatoramrex::EBDataCollection&, amrex::EBDataCollection*, amrex::EB2::Level const&, amrex::Geometry const&, amrex::BoxArray const&, amrex::DistributionMapping const&, amrex::Vector<int, std::allocator > const&, amrex::EBSupport&)
/nopt/nrel/apps/base/2020-05-12/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.4.0-xrmnckz436rnkcwooooi2r7n35pojkov/include/c++/8.4.0/bits/alloc_traits.h:475:4
std::_Sp_counted_ptr_inplace<amrex::EBDataCollection, std::allocatoramrex::EBDataCollection, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<amrex::EB2::Level const&, amrex::Geometry const&, amrex::BoxArray const&, amrex::DistributionMapping const&, amrex::Vector<int, std::allocator > const&, amrex::EBSupport&>(std::allocatoramrex::EBDataCollection, amrex::EB2::Level const&, amrex::Geometry const&, amrex::BoxArray const&, amrex::DistributionMapping const&, amrex::Vector<int, std::allocator > const&, amrex::EBSupport&)
/nopt/nrel/apps/base/2020-05-12/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.4.0-xrmnckz436rnkcwooooi2r7n35pojkov/include/c++/8.4.0/bits/shared_ptr_base.h:545:39
std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<amrex::EBDataCollection, std::allocatoramrex::EBDataCollection, amrex::EB2::Level const&, amrex::Geometry const&, amrex::BoxArray const&, amrex::DistributionMapping const&, amrex::Vector<int, std::allocator > const&, amrex::EBSupport&>(amrex::EBDataCollection*&, std::_Sp_alloc_shared_tag<std::allocatoramrex::EBDataCollection >, amrex::EB2::Level const&, amrex::Geometry const&, amrex::BoxArray const&, amrex::DistributionMapping const&, amrex::Vector<int, std::allocator > const&, amrex::EBSupport&)
/nopt/nrel/apps/base/2020-05-12/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.4.0-xrmnckz436rnkcwooooi2r7n35pojkov/include/c++/8.4.0/bits/shared_ptr_base.h:677:16
std::__shared_ptr<amrex::EBDataCollection, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocatoramrex::EBDataCollection, amrex::EB2::Level const&, amrex::Geometry const&, amrex::BoxArray const&, amrex::DistributionMapping const&, amrex::Vector<int, std::allocator > const&, amrex::EBSupport&>(std::_Sp_alloc_shared_tag<std::allocatoramrex::EBDataCollection >, amrex::EB2::Level const&, amrex::Geometry const&, amrex::BoxArray const&, amrex::DistributionMapping const&, amrex::Vector<int, std::allocator > const&, amrex::EBSupport&)
/nopt/nrel/apps/base/2020-05-12/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.4.0-xrmnckz436rnkcwooooi2r7n35pojkov/include/c++/8.4.0/bits/shared_ptr_base.h:1342:71
std::shared_ptramrex::EBDataCollection::shared_ptr<std::allocatoramrex::EBDataCollection, amrex::EB2::Level const&, amrex::Geometry const&, amrex::BoxArray const&, amrex::DistributionMapping const&, amrex::Vector<int, std::allocator > const&, amrex::EBSupport&>(std::_Sp_alloc_shared_tag<std::allocatoramrex::EBDataCollection >, amrex::EB2::Level const&, amrex::Geometry const&, amrex::BoxArray const&, amrex::DistributionMapping const&, amrex::Vector<int, std::allocator > const&, amrex::EBSupport&)
/nopt/nrel/apps/base/2020-05-12/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.4.0-xrmnckz436rnkcwooooi2r7n35pojkov/include/c++/8.4.0/bits/shared_ptr.h:359:59
std::shared_ptramrex::EBDataCollection std::allocate_shared<amrex::EBDataCollection, std::allocatoramrex::EBDataCollection, amrex::EB2::Level const&, amrex::Geometry const&, amrex::BoxArray const&, amrex::DistributionMapping const&, amrex::Vector<int, std::allocator > const&, amrex::EBSupport&>(std::allocatoramrex::EBDataCollection const&, amrex::EB2::Level const&, amrex::Geometry const&, amrex::BoxArray const&, amrex::DistributionMapping const&, amrex::Vector<int, std::allocator > const&, amrex::EBSupport&)
/nopt/nrel/apps/base/2020-05-12/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.4.0-xrmnckz436rnkcwooooi2r7n35pojkov/include/c++/8.4.0/bits/shared_ptr.h:706:39
std::shared_ptramrex::EBDataCollection std::make_shared<amrex::EBDataCollection, amrex::EB2::Level const&, amrex::Geometry const&, amrex::BoxArray const&, amrex::DistributionMapping const&, amrex::Vector<int, std::allocator > const&, amrex::EBSupport&>(amrex::EB2::Level const&, amrex::Geometry const&, amrex::BoxArray const&, amrex::DistributionMapping const&, amrex::Vector<int, std::allocator > const&, amrex::EBSupport&)
/nopt/nrel/apps/base/2020-05-12/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.4.0-xrmnckz436rnkcwooooi2r7n35pojkov/include/c++/8.4.0/bits/shared_ptr.h:722:42
amrex::EBFArrayBoxFactory::EBFArrayBoxFactory(amrex::EB2::Level const&, amrex::Geometry const&, amrex::BoxArray const&, amrex::DistributionMapping const&, amrex::Vector<int, std::allocator > const&, amrex::EBSupport)
../../../Submodules/AMReX/Src/EB/AMReX_EBFabFactory.cpp:22:24
std::_MakeUniqamrex::EBFArrayBoxFactory::__single_object std::make_unique<amrex::EBFArrayBoxFactory, amrex::EB2::Level const&, amrex::Geometry const&, amrex::BoxArray const&, amrex::DistributionMapping const&, amrex::Vector<int, std::allocator > const&, amrex::EBSupport&>(amrex::EB2::Level const&, amrex::Geometry const&, amrex::BoxArray const&, amrex::DistributionMapping const&, amrex::Vector<int, std::allocator > const&, amrex::EBSupport&)
/nopt/nrel/apps/base/2020-05-12/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.4.0-xrmnckz436rnkcwooooi2r7n35pojkov/include/c++/8.4.0/bits/unique_ptr.h:835:30
amrex::makeEBFabFactory(amrex::Geometry const&, amrex::BoxArray const&, amrex::DistributionMapping const&, amrex::Vector<int, std::allocator > const&, amrex::EBSupport)
../../../Submodules/AMReX/Src/EB/AMReX_EBFabFactory.cpp:120:97

6: ./PeleC3d.gnu.ex() [0x44298c]
std::vector<int, std::allocator >::~vector() inlined at ../../../Submodules/AMReX/Src/Base/AMReX_Vector.H:25:7 in std::enable_if<amrex::IsFabArray<amrex::MultiFab, void>::value, void>::type amrex::InterpFromCoarseLevel<amrex::MultiFab, amrex::PhysBCFunct<amrex::GpuBndryFuncFabpele::physics::pltfilemanager::FillExtDirDummy >, amrex::InterpBase, amrex::NullInterpHookamrex::FArrayBox, amrex::NullInterpHookamrex::FArrayBox >(amrex::MultiFab&, amrex::IntVect const&, double, amrex::MultiFab const&, int, int, int, amrex::Geometry const&, amrex::Geometry const&, amrex::PhysBCFunct<amrex::GpuBndryFuncFabpele::physics::pltfilemanager::FillExtDirDummy >&, int, amrex::PhysBCFunct<amrex::GpuBndryFuncFabpele::physics::pltfilemanager::FillExtDirDummy >&, int, amrex::IntVect const&, amrex::InterpBase*, amrex::Vector<amrex::BCRec, std::allocatoramrex::BCRec > const&, int, amrex::NullInterpHookamrex::FArrayBox const&, amrex::NullInterpHookamrex::FArrayBox const&) [clone .constprop.325]
/nopt/nrel/apps/base/2020-05-12/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.4.0-xrmnckz436rnkcwooooi2r7n35pojkov/include/c++/8.4.0/bits/stl_vector.h:567:15
amrex::Vector<int, std::allocator >::~Vector()
../../../Submodules/AMReX/Src/Base/AMReX_Vector.H:25:7
std::enable_if<amrex::IsFabArray<amrex::MultiFab, void>::value, void>::type amrex::InterpFromCoarseLevel<amrex::MultiFab, amrex::PhysBCFunct<amrex::GpuBndryFuncFabpele::physics::pltfilemanager::FillExtDirDummy >, amrex::InterpBase, amrex::NullInterpHookamrex::FArrayBox, amrex::NullInterpHookamrex::FArrayBox >(amrex::MultiFab&, amrex::IntVect const&, double, amrex::MultiFab const&, int, int, int, amrex::Geometry const&, amrex::Geometry const&, amrex::PhysBCFunct<amrex::GpuBndryFuncFabpele::physics::pltfilemanager::FillExtDirDummy >&, int, amrex::PhysBCFunct<amrex::GpuBndryFuncFabpele::physics::pltfilemanager::FillExtDirDummy >&, int, amrex::IntVect const&, amrex::InterpBase*, amrex::Vector<amrex::BCRec, std::allocatoramrex::BCRec > const&, int, amrex::NullInterpHookamrex::FArrayBox const&, amrex::NullInterpHookamrex::FArrayBox const&)
../../../Submodules/AMReX/Src/Base/AMReX_Vector.H:25:7

7: ./PeleC3d.gnu.ex() [0x444f46]
pele::physics::pltfilemanager::PltFileManager::fillPatchFromPlt(int, amrex::Geometry const&, int, int, int, amrex::MultiFab&)
../../../Submodules/PelePhysics/Utility/PltFileManager/PltFileManager.cpp:238:30

8: ./PeleC3d.gnu.ex() [0x86c93f]
std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_is_local() const inlined at /nopt/nrel/apps/base/2020-05-12/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.4.0-xrmnckz436rnkcwooooi2r7n35pojkov/include/c++/8.4.0/bits/basic_string.h:220:6 in PeleC::initLevelDataFromPlt(int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, amrex::MultiFab&)
/nopt/nrel/apps/base/2020-05-12/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.4.0-xrmnckz436rnkcwooooi2r7n35pojkov/include/c++/8.4.0/bits/basic_string.h:211:26
std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_dispose()
/nopt/nrel/apps/base/2020-05-12/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.4.0-xrmnckz436rnkcwooooi2r7n35pojkov/include/c++/8.4.0/bits/basic_string.h:220:6
std::__cxx11::basic_string<char, std::char_traits, std::allocator >::~basic_string()
/nopt/nrel/apps/base/2020-05-12/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.4.0-xrmnckz436rnkcwooooi2r7n35pojkov/include/c++/8.4.0/bits/basic_string.h:657:9
PeleC::initLevelDataFromPlt(int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, amrex::MultiFab&)
../../../Source/IO.cpp:771:53

9: ./PeleC3d.gnu.ex() [0x7df52b]
PeleC::initData()
../../../Source/PeleC.cpp:684:23

10: ./PeleC3d.gnu.ex() [0x664149]
amrex::Amr::defBaseLevel(double, amrex::BoxArray const*, amrex::Vector<int, std::allocator > const*)
../../../Submodules/AMReX/Src/Amr/AMReX_Amr.cpp:2476:14

11: ./PeleC3d.gnu.ex() [0x66c0e1]
amrex::Amr::initialInit(double, double, amrex::BoxArray const*, amrex::Vector<int, std::allocator > const*) inlined at ../../../Submodules/AMReX/Src/Amr/AMReX_Amr.cpp:1112:20 in amrex::Amr::init(double, double)
../../../Submodules/AMReX/Src/Amr/AMReX_Amr.cpp:1244:19
amrex::Amr::init(double, double)
../../../Submodules/AMReX/Src/Amr/AMReX_Amr.cpp:1112:20

12: ./PeleC3d.gnu.ex() [0x42992e]
main
../../../Source/main.cpp:145:28

13: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f1d7ef72555]

14: ./PeleC3d.gnu.ex() [0x42aaf2]
_start at ??:?

[mikaelm123]

What case are you running? Does it have EB?

PeleC utilizes the PltFileManager utility in PelePhysics to restart from plt files. This utility does not support restarting with a coarser base grid than the plt file from which you are restarting. However, it should support restarting with a finer grid, and it appears there may be a bug in the code for that, which is causing your segfault for this case. Note also that PeleC does not support changing the base grid when restarting from chk files.

I did some quick tests and I am able to restart from a plt file in the EB-ConvergingNozzle case, but not in the PMF case. @marchdf and @jrood-nrel I think this has to do with when the EB ifdefs were removed from PeleC. Commenting out the EB ifdef here in AMReX InterpFromCoarseLevel appears to solve the issue.

I was running a non-EB case and this fixed the issue. Thank you very much for the assistance!

[marchdf]

@baperry2 this feels like a factory thing isn't getting initialized right... Are you able to look into this?

[baperry2]

The fix for this has now been merged into the development branch (#645). Now when interpolating from a coarer pltfile, use pelec.init_pltfile_coarse_levels in the input file to specify how many coarser levels the pltfile data has than the new simulation.