Enable Adaptive Mesh Refinement

[nuclearkevin]

This PR enables adaptive mesh refinement for the OpenMC mesh mirror. This allows for the use of AMR (as implemented in MOOSE) within Cardinal.

The main changes include looping over active elements instead of all elements when generating the cell -> element map and creating a copy of the mesh containing only active elements when using a MeshTally. The latter is required for two reasons: to avoid errors which are thrown by MOOSE because of the equation system created in the openmc::LibMesh class, and to prevent erroneous tallying on non-active elements in the refinement/coarsening hierarchy. Creating a copy of the mesh without the refinement hierarchy is a severe limitation of the current approach and should be remedied in the future.

[moosebuild]

Job Documentation, step Sync to remote on b598421 wanted to post the following:

View the site here

This comment will be updated on new commits.

[nuclearkevin]

For some reason the AMR MeshTally test crashes in CIVET, but doesn't crash on Pinchot or my local machine. I've ran the test through gdb and Valgrind (thorough memtest) and it executes successfully, so I'm at a bit of a loss at the moment.

Since I use the MOOSE Conda environment on both my local machine and Pinchot (but CIVET builds libMesh from a specific commit hash), I'm starting to wonder if there is an incongruence between the version of libMesh in the Conda distribution and the one required for Cardinal's MOOSE submodule. I'm building from source to investigate that at the moment, hopefully it yields a simple fix so I can get the tests passing.

This wasn't the cause. Valgrind and GDB both report no errors when testing this branch compiled without the MOOSE Conda environment.

[moosebuild]

Job Precheck, step Clang format on 80b23d2 wanted to post the following:

Your code requires style changes.

A patch was auto generated and copied here
You can directly apply the patch by running, in the top level of your repository:

curl -s https://mooseframework.inl.gov/cardinal/docs/PRs/964/clang_format/style.patch | git apply -v

Alternatively, with your repository up to date and in the top level of your repository:

git clang-format 018b5f2f2c76d194be67bbc150f8b29c8649e284

[moosebuild]

Job Coverage, step Generate coverage on b598421 wanted to post the following:

Coverage

	84bf8d	#964 b59842
	Total	Total	+/-	New
Rate	92.96%	92.92%	-0.05%	98.18%
Hits	7703	7715	+12	54
Misses	583	588	+5	1

Diff coverage report

Full coverage report

This comment will be updated on new commits.

[nuclearkevin]

@aprilnovak I managed to track down and patch the bug that's been plaguing this PR; it should be good for review now.

[aprilnovak]

Can you explain where this went?

[nuclearkevin]

This #else statement was the cause of the segmentation faults on CIVET, and more generally it resulted in memory leaks when Cardinal is built without DAGMC and using a moving mesh.

When the mesh is changing OpenMCCellAverageProblem needs to reset tallies on every timestep (after the first step), and so it calls resetTallies() at the beginning of the step. This is immediately followed by a call to setupProblem(). Afterwards, if the mesh is moving setupProblem() is called a second time in that #else statement resulting in an initialization of OpenMC tallies twice for each time they're reset. This causes a memory leak as we don't manage the extra OpenMC tally created on each step.

The reason why this caused a segmentation fault is a little more complicated and involves the copy of the mesh created in MeshTally containing only active elements. This mesh copy is deleted both when resetTally() and initializeTally() are called (it's a unique pointer). Since we've been creating two OpenMC tallies per timestep (by accident) the first of those tallies still thinks the mesh copy is valid, and so it tries to tally on datastructures that have had their memory freed. This behaviour is undefined and results in a segmentation fault.

This bug only occurs when Cardinal is compiled without DAGMC as resetTallies() is called again when skinning the geometry (the previous usecase for the #ifdef/#else statement), so nothing breaks.

[aprilnovak]

What does this test show exactly? Can we see the outlines of a CSG cell become more refined as the mesh changes?

[nuclearkevin]

This test ensures that problems can run mesh adaptivity when using cell tallies, so it's measuring the increasing number of elements. It's really just a surrogate for a RunApp executioner with some extra information attached.

[aprilnovak]

Would you please also add a news markdown file to announce this capability? I think it warrants it!

[nuclearkevin]

@aprilnovak I can definitely do that! Should I also add an entry for the filter system (since it's a pretty big addition to Cardinal as well)?