The BaseRestartWorkChain runs handlers only for the first iteration

[yakutovicha]

• AiiDA Troubleshooting Documentation
• AiiDA Discourse Forum

Describe the bug

About a year ago I noticed some strange behaviour in the Cp2kBaseWorkChain, which is based on the BaseRestartWorkChain. In essence, the work chain's handler would only be applied after the first iteration. After the second iteration, the same handler would not be applied:

Report: [21071|Cp2kBaseWorkChain|run_process]: launching Cp2kCalculation<21072> iteration #1
Warning: output parser returned exit code<400>: The calculation stopped prematurely because it ran out of walltime.
Report: [21071|Cp2kBaseWorkChain|restart_incomplete_calculation]: The CP2K calculation wasn't completed. The restart of the calculation might be able to fix the problem.
Report: [21071|Cp2kBaseWorkChain|inspect_process]: Cp2kCalculation<21072> failed but a handler dealt with the problem, restarting

⏫  The handler was nicely applied for the first iteration

Report: [21071|Cp2kBaseWorkChain|run_process]: launching Cp2kCalculation<21083> iteration #2
Warning: output parser returned exit code<400>: The calculation stopped prematurely because it ran out of walltime.
Report: [21071|Cp2kBaseWorkChain|inspect_process]: Cp2kCalculation<21083> failed and error was not handled, restarting once more 

⏫  The handler was not applied for the second iteration

Report: [21071|Cp2kBaseWorkChain|run_process]: launching Cp2kCalculation<21088> iteration #3
Warning: output parser returned exit code<400>: The calculation stopped prematurely because it ran out of walltime.
Report: [21071|Cp2kBaseWorkChain|inspect_process]: Cp2kCalculation<21088> failed and error was not handled for the second consecutive time, aborting

I didn't escalate the issue, because the same functionality seemed to work fine on the various AiiDAlab machines. I still don't know why 🤷 .

However, the issue popped up again in the recent aiida-cp2k PR.

I guess the bug was introduced in #5546

Steps to reproduce

Steps to reproduce the behaviour:

It is enough to submit a new example example_base_md_reftraj_restart.py from aiidateam/aiida-cp2k#207

Expected behaviour

The handler should be called as many times as the simulation restarts.

Your environment

• Operating system: Ubuntu 22.04.1 LTS (Docker container)
• Python version: 3.9.13, 3.10.13
• aiida-core version: 2.1.2, 2.4.3, 2.5.0