CR showers code excessive flagging in MRS bright sources

[stscijgbot-jp]

Issue JP-3677 was created on JIRA by David Law:

Filing a ticket to track this as additional examples have arisen in the community.

The MIRI CR showers routine can sometime flag genuine source pixels as showers for very bright sources.  Previously this was noted primarily for observations of Jupiter and Saturn, which have ultra-bright continua, and complex spectra that cause saturation in multiple places.  This has now been seen in galaxy data though, flagging large sections of continuum emission around bright spectral lines.

As an example, see jw01701012001_02105_00001_mirifulong_uncal

The figure attached shows the region around a bright Ch3 emission line processed with the default B11.0 pipeline on the left, and with the showers routine enabled on the right.  Perhaps it's worth capping how bright showers can be?  Unlike snowballs than can range in magnitude, showers are all low-level blotches that are never (as far as I'm aware) in excess of a few DN/s.

[m-samland]

Could you elaborated @drlaw1558? I think I might have a dataset in which the top of emission might be cut off because of such an issue. Is this happening due to parameters set in the jump detection step or one of the other correction steps (pixel replace / spec3 outlier rejection)?

[drlaw1558]

@m-samland : In the case listed here the issue is with the shower-correction routine in the jump step. When some pixels start saturating the shower flagging can expand the size of the flagged regions too much. Note that the shower-correction is turned off by default in the pipeline at the moment for exactly this reason, so unless you're turning on the shower corrections in your own reprocessing this shouldn't currently affect you.

I'd suggest filing a helpdesk ticket for your dataset (if you haven't already) and we can try to see what's going on there.

[stscijgbot-jp]

Comment by David Law on JIRA:

Adding a note for future reference; an example of an extremely bright REAL shower can be seen in jw04278017001_03105_00002_mirifushort

[stscijgbot-jp]

Comment by David Law on JIRA:

Adding an idea that may help with the accidental bright-source flagging:

We could potentially run ramp fitting both before and after the shower-flagging, and look for any pixels in which the returned slope in DN/s changed by more than X DN/s, or if a slope that used to be finite with value above Y DN/s changed to NaN.  Since the shower flagging routine should only ever change things at the roughly 1 DN/s level and below, it should be easy to identify failure cases where the change is too big to be due to a real shower and 'undo' the shower flag for that pixel.

• This would require some new juggling of JUMP flags set by the shower routine to make a 'tentative' flagging before committing that flagging to the main DQ array.  Shouldn't require more than one copy of the DQ array though.
• In practice we wouldn't want to actually run the ramp fitting step, fast though the C routine is.  Since we're looking for very big differences though, we may be able to get away with a simple nanmedian of group differences though, which should be much faster and not require calling a different pipeline step.

Thoughts Michael Regan ?

[stscijgbot-jp]

Comment by David Law on JIRA:

Ok, in testing out this idea I ended up coding it in stcal.  See PR spacetelescope/stcal#306 and #8890

In brief, this does what I suggested above, doing a quick and rough rate estimate based on the median group differences, and NOT using the shower-flagged jumps in cases where doing so either turns pixels NaN that used to be finite or changes the rates by more than the maximum shower amplitude (set to 2 DN/s by default).  As such, it performs equally well to the previous version for cases like jw01232002001_02101_00002_mirifushort_uncal and jw04278017001_03105_00002_mirifushort_uncal with normal showers, and does not accidentally flag bright sources for examples like jw01701012001_02105_00001_mirifulong_uncal

This adds about 30 sec to the total runtime in a 3-int, 94-group test case.  Might be better ways to optimize.

[stscijgbot-jp]

Comment by David Law on JIRA:

Ok, now better optimized.  Using nanmean instead of nanmedian both improves the quality of what this is trying to do AND drops the runtime from 30 sec to just 2 sec.