JP-2943:Level_3 error: cannot reshape array in image3 pipeline

[stscirij]

Added code to try and solve intermittent problem of corrupted files in outlier_detection step of image3 pipeline

Resolves JP-2943

Closes #7271

This PR addresses problems noted in operations when running the image3 pipeline. During the outlier_rejection step,
data read from files written during the step appear corrupted. This is probably because the I/O has not fully completed when the file is accessed later in processing. To try and fix this, the file is read back in after a delay and the shape of the array in the data read back is compared to the shape of the data in the model that was written. If they are the same, processing continues. If they aren't, the output file is deleted, the delay doubled, and the readback/compare loop repeated. If the wait time exceeds 40s, processing continues with a warning that the file may be corrupted.
This change occurs in the resample step, and only if in_memory processing is not selected.

Checklist for maintainers

• added entry in CHANGES.rst within the relevant release section
• updated or added relevant tests
• updated relevant documentation
• added relevant milestone
• added relevant label(s)
• ran regression tests, post a link to the Jenkins job below.
  How to run regression tests on a PR
• Make sure the JIRA ticket is resolved properly

[codecov]

Codecov Report

Attention: Patch coverage is 68.96552% with 9 lines in your changes are missing coverage. Please review.

Project coverage is 75.30%. Comparing base (4cc0ac1) to head (2d8454a).
Report is 31 commits behind head on master.

❗ Current head 2d8454a differs from pull request most recent head adb6075. Consider uploading reports for the commit adb6075 to get more accurate results

Files	Patch %	Lines
jwst/resample/resample.py	68.96%	9 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8305      +/-   ##
==========================================
+ Coverage   75.15%   75.30%   +0.14%     
==========================================
  Files         470      474       +4     
  Lines       38604    38992     +388     
==========================================
+ Hits        29014    29363     +349     
- Misses       9590     9629      +39     

Flag	Coverage Δ		*Carryforward flag
nightly	77.33% <ø> (-0.06%)	⬇️	Carriedforward from 0066901

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[stscirij]

Full regression run on PR branch here:
https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1229/

[hbushouse]

@stscirij Latest updates look good. Please do another full regtest run to make sure none of the recent changes have messed up anything.

[stscirij]

Developer regtest here:

https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1324/

[hbushouse]

Looks good now. Regtest results are clean.