Target groups treat all other than first target as background

[stscijgbot-jp]

Issue JP-3795 was created on JIRA by Peter Zeidler:

I recently looked at the GO data of JWST-3855, for which I am a Co-I. FOr this program we observed a series of O-star in the SMC young star cluster NGC 346 using the NIRSpec FS. Given the relatively small distance between the individual targets (less than the visit splitting distance) and the excellent astrometry, based of NIRCam imaging we created target groups, for which we did WATA on the first target and then bind offsets between the individual sources. This technique was very successful (we hit all targets in the center of the slit) and saved a lot of overheads.

When looking at the Mast data and when I tried to reduce the data myself I noticed that by default the first source in the target groups is marked as the science target, while all the remaining targets are classified as background, which obviously is not correct in such a case. I believe that in most cases target groups are rather used as individual targets (like in JWST-3855) and not as it is currently setup in the pipeline thus it may make sense to change this default setting in the pipeline. As it is currently setup none of the step 2 and 3 association files are correct and of course a quick look using the MAST data is impossible. I only tried it for this program but I assume that this affects all instruments in the same way.

I am happy to provide more information.

 

[stscijgbot-jp]

Comment by David Law on JIRA:

Hm, I'm puzzled by this, but I'm also not super familiar with fixed-slit interaction with target groups.

Looking at the program, there are two observations.  Obs 1 is a group of 7 stars, and Obs 2 is a group of 12.  No targets have linked dedicated background observations.  Each uses a 6-point dither.

On MAST, the spec2 ASN files indeed have many EXPTYPE=background entries. It looks like each dither position 1 is using as background targets dither positions 4/5/6 of all other exposures.  Likewise, dither position 4 is using positions 1/2/3 of the other exposures.  I think that's largely as expected- select out the dithers where the source is far enough away that the data can be differenced.  What's interesting though is that it's selecting these dither positions on all targets in the target group.

I'm inclined to agree that that's not necessarily what I'd expect scientifically; the 'background' that you're trying to subtract could be quite different between different targets if, say, they're embedded in some diffuse emission.  Note that this is a different use case than if there were a dedicated background, which I would expect to apply to all targets in the target group.

Assigning to Christian Hayes to clarify intended performance for NIRSpec FS background subtraction in this case.

[stscijgbot-jp]

Comment by Christian Hayes on JIRA:

I can confirm that the inclusion/exclusion of dither positions in the spec2 ASN files is correct for these observations.  Dither positions 1/2/3 should have 4, 5, and 6 as background members, and vice-versa.  This is because 1/2/3 are at one primary dither position and 4/5/6 are at another primary dither position and spectra at the same primary dither position shouldn't be used for backgrounds.

I'm not familiar with all of the ways that target groups are used and how they get mapped to association pools, but at least for this case because the additional targets were not listed as dedicated backgrounds I don't think they should be matched in as background members for a different target.

The other issue is that only one spec3 association is created (just for the first target), and there should be multiple spec3 associations (one for each target).

[stscijgbot-jp]

Comment by David Law on JIRA:

Thanks Christian Hayes ; passing over to Tyler Pauly to continue the conversation about what we're currently doing.

A snag with the spec3 though is that the 'expected' behavior may well be different for different instrument modes.  E.g., you could use target groups for the IFU to cover multiple regions within a galaxy (in which case you'd want them all in the same spec3 data cube).  Trying to game out all possible use cases is probably impractical in an automated sense, but agreed we should have some kind of behavior that is reasonable most of the time.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

It looks like the spec2 problem could be fairly easily fixed for this program by excluding potential background members with mismatched target ID.  That field is already available in the pool (TARGETID).  Is there any reason that wouldn't generalize well?

For the spec3 missing associations, running asn_generate for obs 1 locally on pool file jw03855_20241103t042455_pool.csv, I see the 7 spec3 associations I would expect to see.  I'm not sure why only the seventh one appears in a MAST search (jw03855-o001_20241103t042455_spec3_00007_asn.json, for
the first target, act 02).  I'll ask around.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

Mystery immediately solved by running the spec3 files locally: there's a naming collision.  All 7 spec3 files for obs1 produce identically named output, so only the last one is saved.

Including the target ID in the spec3 name for NIRSpec FS would fix it.

[stscijgbot-jp]

Comment by David Law on JIRA:

Melanie Clarke Nice catch; those two fixes sound reasonable to me.  One thing to note though is that changing the spec3 names would trigger the usual DMS stale file and new-file problems, but just to let DMS know about.

Would also need to be sure that any target ID checking doesn't cause issues for the case of dedicated backgrounds, which would almost always have a different target ID.  (I.e., we're only trying to change the logic for auto-determined nod backgrounds based on dither position).

Would probably be worth checking more broadly if Target Groups for other modes is going to trigger this issue with overwritten spec3 asns as well (presumably so) and that target ID would fix those too.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

Here's the draft fix I was playing with:

#8943

As written, for spec2, it only affects background candidates identified from other science members - dedicated backgrounds would be unaffected.  The spec3 fix so far is specifically for NIRSpec FS, so if we think this is the right approach, we might consider whether we should add target ID to any of the other source-based output names that don't currently include it.