Skip to content

Commit

Permalink
Speed up nonsparse AD initial setup significantly
Browse files Browse the repository at this point in the history
We have been setting `ADReal::do_derivatives=true` by default and only
toggling to `false` during residual computation. This can be very bad
for non-sparse calculations. For instance in a navier-stokes input file
supplied by Gavin Ridley, I saw 10 seconds spent in initial condition
computation and 9 seconds in `ComputeMaterialsObjectThread` for a total
of 19 seconds in `FEProblemBase::initialSetup`. This was 27% of the
total computation time! With the changes here
`FEProblemBase::initialSetup` no longer even appears in the graph, which
is how it should be.

I recall that I was forced to do derivative calculations by default for
the reasons stated in the comment I'm deleting. My memory says some
phase field object was initializing its material properties in its
constructor, or more likely during `initialSetup`, so we had to enable
derivative calculations then or else its results would be garbage.
Hopefully, however, if there are objects still doing things like that we
can do a little more fine-grained control to make their objects work
without significantly sabotaging everyone's performance.

Refs #14701
  • Loading branch information
lindsayad committed Nov 5, 2020
1 parent 7d42999 commit db42526
Show file tree
Hide file tree
Showing 2 changed files with 6 additions and 10 deletions.
2 changes: 2 additions & 0 deletions framework/src/base/MooseApp.C
Original file line number Diff line number Diff line change
Expand Up @@ -352,6 +352,8 @@ MooseApp::MooseApp(InputParameters parameters)
}
#endif
ADReal::do_derivatives = false;
Registry::addKnownLabel(_type);
Moose::registerAll(_factory, _action_factory, _syntax);
Expand Down
14 changes: 4 additions & 10 deletions framework/src/problems/FEProblemBase.C
Original file line number Diff line number Diff line change
Expand Up @@ -349,11 +349,6 @@ FEProblemBase::FEProblemBase(const InputParameters & parameters)
_num_grid_steps(0),
_displaced_neighbor_ref_pts("invert_elem_phys use_undisplaced_ref unset", "unset")
{
// Initialize static do_derivatives member. We initialize this to true so that all the default AD
// things that we setup early in the simulation actually get their derivative vectors initalized.
// We will toggle this to false when doing residual evaluations
ADReal::do_derivatives = true;

_time = 0.0;
_time_old = 0.0;
_t_step = 0;
Expand Down Expand Up @@ -5156,12 +5151,7 @@ FEProblemBase::computeResidualSys(NonlinearImplicitSystem & /*sys*/,
NumericVector<Number> & residual)
{
TIME_SECTION(_compute_residual_sys_timer);

ADReal::do_derivatives = false;

computeResidual(soln, residual);

ADReal::do_derivatives = true;
}

void
Expand Down Expand Up @@ -5449,6 +5439,8 @@ FEProblemBase::computeJacobianInternal(const NumericVector<Number> & soln,
void
FEProblemBase::computeJacobianTags(const std::set<TagID> & tags)
{
ADReal::do_derivatives = true;

if (!_has_jacobian || !_const_jacobian)
{
TIME_SECTION(_compute_jacobian_tags_timer);
Expand Down Expand Up @@ -5538,6 +5530,8 @@ FEProblemBase::computeJacobianTags(const std::set<TagID> & tags)
_displaced_problem->setCurrentlyComputingJacobian(false);
_safe_access_tagged_matrices = true;
}

ADReal::do_derivatives = false;
}

void
Expand Down

0 comments on commit db42526

Please sign in to comment.