JP-2928: Update data for MIRI dark pipeline tests

[melanieclarke]

Partially addresses JP-2928

Update MIRI test data for the dark pipeline to flight data examples. Data is taken from PID 4482 obs 14, chosen because the number of groups is small, which makes for faster testing. EXP_TYPEs tested are MIR_DARKIMG and MIR_DARKMRS (with detector MIRIFULONG and MIRIFUSHORT).

Tasks

• request a review from someone specific, to avoid making the maintainers review every PR
• add a build milestone, i.e. Build 11.3 (use the latest build if not sure)
• Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types)
  • update or add relevant tests
  • update relevant docstrings and / or docs/ page
  • start a regression test and include a link to the running job (click here for instructions)
    • Do truth files need to be updated ("okified")?
      • after the reviewer has approved these changes, run okify_regtests to update the truth files
• if a JIRA ticket exists, make sure it is resolved properly

news fragment change types...

• changes/<PR#>.general.rst: infrastructure or miscellaneous change
• changes/<PR#>.docs.rst
• changes/<PR#>.stpipe.rst
• changes/<PR#>.datamodels.rst
• changes/<PR#>.scripts.rst
• changes/<PR#>.fits_generator.rst
• changes/<PR#>.set_telescope_pointing.rst
• changes/<PR#>.pipeline.rst

stage 1

• changes/<PR#>.group_scale.rst
• changes/<PR#>.dq_init.rst
• changes/<PR#>.emicorr.rst
• changes/<PR#>.saturation.rst
• changes/<PR#>.ipc.rst
• changes/<PR#>.firstframe.rst
• changes/<PR#>.lastframe.rst
• changes/<PR#>.reset.rst
• changes/<PR#>.superbias.rst
• changes/<PR#>.refpix.rst
• changes/<PR#>.linearity.rst
• changes/<PR#>.rscd.rst
• changes/<PR#>.persistence.rst
• changes/<PR#>.dark_current.rst
• changes/<PR#>.charge_migration.rst
• changes/<PR#>.jump.rst
• changes/<PR#>.clean_flicker_noise.rst
• changes/<PR#>.ramp_fitting.rst
• changes/<PR#>.gain_scale.rst

stage 2

• changes/<PR#>.assign_wcs.rst
• changes/<PR#>.badpix_selfcal.rst
• changes/<PR#>.msaflagopen.rst
• changes/<PR#>.nsclean.rst
• changes/<PR#>.imprint.rst
• changes/<PR#>.background.rst
• changes/<PR#>.extract_2d.rst
• changes/<PR#>.master_background.rst
• changes/<PR#>.wavecorr.rst
• changes/<PR#>.srctype.rst
• changes/<PR#>.straylight.rst
• changes/<PR#>.wfss_contam.rst
• changes/<PR#>.flatfield.rst
• changes/<PR#>.fringe.rst
• changes/<PR#>.pathloss.rst
• changes/<PR#>.barshadow.rst
• changes/<PR#>.photom.rst
• changes/<PR#>.pixel_replace.rst
• changes/<PR#>.resample_spec.rst
• changes/<PR#>.residual_fringe.rst
• changes/<PR#>.cube_build.rst
• changes/<PR#>.extract_1d.rst
• changes/<PR#>.resample.rst

stage 3

• changes/<PR#>.assign_mtwcs.rst
• changes/<PR#>.mrs_imatch.rst
• changes/<PR#>.tweakreg.rst
• changes/<PR#>.skymatch.rst
• changes/<PR#>.exp_to_source.rst
• changes/<PR#>.outlier_detection.rst
• changes/<PR#>.tso_photometry.rst
• changes/<PR#>.stack_refs.rst
• changes/<PR#>.align_refs.rst
• changes/<PR#>.klip.rst
• changes/<PR#>.spectral_leak.rst
• changes/<PR#>.source_catalog.rst
• changes/<PR#>.combine_1d.rst
• changes/<PR#>.ami.rst

other

• changes/<PR#>.wfs_combine.rst
• changes/<PR#>.white_light.rst
• changes/<PR#>.cube_skymatch.rst
• changes/<PR#>.engdb_tools.rst
• changes/<PR#>.guider_cds.rst

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 63.68%. Comparing base (1f87cb2) to head (205a2a1).
Report is 23 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #8925   +/-   ##
=======================================
  Coverage   63.68%   63.68%           
=======================================
  Files         375      375           
  Lines       38689    38691    +2     
=======================================
+ Hits        24638    24640    +2     
  Misses      14051    14051           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[melanieclarke]

Regression tests with locally generated truth files:
https://github.com/spacetelescope/RegressionTests/actions/runs/11524214245

No failures.

[jemorrison]

Melanie as someone who made the MIRI darks for commissioning and then passed the code to the MIRI team at STScI to make darks - we never used the jwst dark pipeline. Does any team use the dark pipeline ? I do not think NIRCam does - does NirSpec ? MIRI has some issues (we have a drifting zero point) that force us to use custom software. I would ask the MIRI team if they ever use jwst.pipeline.darkpipeline.
In fact I would ask any instrument team if they use the dark pipeline. I am not sure any team does.
Looking at the calwebb_dark.py for MIRI - it skips the IPC correction but does not skip the refpix correction nor the reset correction. Both need to be skipped for the creation of a dark. In addition it should not skip the first frame correction because the dark does try to use the data to figure out a correction. But like I said this is not being used and it might just be confusing to have it in the regression tests. But that is your call.

[melanieclarke]

Does any team use the dark pipeline ? I do not think NIRCam does - does NirSpec ?

NIRSpec does not. I'm pretty sure none of the JWST instrument teams use it for real calibration purposes, so the current function of the DarkPipeline is most likely just quicklook for dark products, so that something is available from MAST. We can make changes to the dark pipeline if MIRI wants, or remove it entirely if it's not useful to anyone, but that should be a separate ticket. This PR is just to validate current behavior with an updated data set.

[jemorrison]

When I try and look at the regression test results I get a 404 error.

[melanieclarke]

When I try and look at the regression test results I get a 404 error.

That's odd. I can see them, at this link?

[jemorrison]

The new data is SLOWR1 which should not matter for what the test is doing. So that seems fine.

[jemorrison]

I am logging on via can and I still get a 404 error. Maybe I have not set up something for the GitHub actions. Did I need to do something to see the results. I have not used the GitHub actions yet.

[jemorrison]

Ok the link does not work - but if I go to "actions option" on GitHub I can see the run. Where do I go to look at the results from GitHub actions ?

[melanieclarke]

The regression tests are in a separate private repo: https://github.com/spacetelescope/RegressionTests

Maybe you don't have permissions for it? @zacharyburnett - can you please check for Jane?

[zacharyburnett]

The regression tests are in a separate private repo: https://github.com/spacetelescope/RegressionTests

Maybe you don't have permissions for it? @zacharyburnett - can you please check for Jane?

unfortunately it looks like you have to be a member of spacetelescope first

this is because ITSD requires that the RegressionTest repository be private

[jemorrison]

The updated MIRI dark data looks fine. It would be nice to find out if any team is using the dark pipeline. But that is a different issue.