JP-3153: post-commissioning AMI3 updates

[rcooper295]

Resolves JP-3153 and the following sub-tasks and linked tickets:

• JP-1713
• JP-1714
• JP-2904
• JP-3346
• JP-3348

This PR addresses all changes to the NIRISS AMI mode analysis software since its last delivery in 2020. The main change is a complete overhaul of output formats. Two new datamodels were created: AmiOIModel (spacetelescope/stdatamodels#174) and AmiLgFitModel (spacetelescope/stdatamodels#199).

Step changes:
ami_analyze:

• Additional input arguments and defaults: affine2d, src, bandpass, usebp, firstfew, chooseholes, run_bpfix.
• Outputs are now two types of OIFITS files: averaged across integrations and per-integration, and the AmiLgFitModel which stores the cropped/cleaned data, model, and residuals. The OIFITS format is a standard in the field for storing interferometric observables; the AmiLgFitModel product is mostly intended as a diagnostic tool.
• The filter throughput reference file used previously was too coarsely sampled across the bandpass; now WebbPSF filter definitions are used and combined with a source spectrum from the Phoenix library (A0V by default) using synphot and stsynphot.
• A median filter is used to identify integrations with bright CR hits not identified by the jump detection step (e.g. when NGROUPS <=3) and DO_NOT_USE flags added to the DQ array.
• A fourier-based bad pixel correction is applied by default to fix pixels flagged DO_NOT_USE and/or JUMP_DET.

ami_average:

• This step will be turned off in the default AMI3 pipeline flow until the effects of averaging together multiple reference star observations are better understood. The code is left as-is except "skip = True" has been added to the spec block. This step will not work if run manually on the AmiOIModel products.

ami_normalize:

• Rewritten to use the averaged AmiOIModel datamodels.

Other changes include processing 3D instead of 2D images by default for AMI exposures (rateints and calints files)
The calwebb_ami3 pipeline has been updated to run only the ami_analyze and ami_normalize steps. Association generator needs to be updated

Checklist for maintainers

• added entry in CHANGES.rst within the relevant release section
• updated or added relevant tests
• updated relevant documentation
• added relevant milestone
• added relevant label(s)
• ran regression tests, post a link to the Jenkins job below.
  How to run regression tests on a PR
• Make sure the JIRA ticket is resolved properly

[codecov]

Codecov Report

Attention: Patch coverage is 80.35160% with 190 lines in your changes are missing coverage. Please review.

Project coverage is 55.01%. Comparing base (2fb073e) to head (30f38fc).
Report is 11 commits behind head on master.

❗ Current head 30f38fc differs from pull request most recent head f49415b. Consider uploading reports for the commit f49415b to get more accurate results

Files	Patch %	Lines
jwst/ami/oifits.py	80.41%	56 Missing ⚠️
jwst/ami/utils.py	72.72%	39 Missing ⚠️
jwst/ami/instrument_data.py	79.83%	25 Missing ⚠️
jwst/pipeline/calwebb_ami3.py	0.00%	19 Missing ⚠️
jwst/ami/lg_model.py	65.78%	13 Missing ⚠️
jwst/ami/ami_analyze.py	71.42%	12 Missing ⚠️
jwst/ami/bp_fix.py	94.48%	7 Missing ⚠️
jwst/ami/leastsqnrm.py	91.13%	7 Missing ⚠️
jwst/ami/ami_analyze_step.py	86.20%	4 Missing ⚠️
jwst/ami/ami_normalize_step.py	0.00%	3 Missing ⚠️
... and 2 more

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #7862       +/-   ##
===========================================
- Coverage   75.31%   55.01%   -20.30%     
===========================================
  Files         474      390       -84     
  Lines       38965    38895       -70     
===========================================
- Hits        29345    21400     -7945     
- Misses       9620    17495     +7875     

Flag	Coverage Δ
nightly	?

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.