Merge branch 'master' into emicorrfix

CHANGES.rst

@@ -8,11 +8,19 @@ associations
   sub-pixel dithers, so that only exposures from other nod positions
   are used as background members in "spec2" associations. [#8184]
 
+-  JP-3290 Isolate candidate processing into their own pools [#8227]
+
 cube_build
 ----------
 
 - Add a warning message to log if no valid data is found on the detector. [#8220]
 
+datamodels
+----------
+
+- Fixed a bug in the ``ModelContainer`` data model, due to which the ``models_grouped``
+  property would return opened data models instead of file names. [#8191]
+
 documentation
 -------------
 
@@ -28,6 +36,9 @@ documentation
 
 - Change docs theme to ``sphinx-rtd-theme`` [#8224]
 
+- Reorganized ``jump`` and ``ramp_fitting`` step docs content that's split between
+  the jwst and stcal repos. [#8253]
+
 emicorr
 -------
 
@@ -44,15 +55,39 @@ extract_1d
 - Fixed a bug in the calling of optional MIRI MRS 1d residual fringe
   correction that could cause defringing to fail in some cases. [#8180]
 
-- Added a hook to bypass the ``extract_1d`` step for NIRISS SOSS data in 
+- Added a hook to bypass the ``extract_1d`` step for NIRISS SOSS data in
   the FULL subarray with warning. [#8225]
+
+- Fixed a bug in the ATOCA matrix solve for NIRISS SOSS that would cause failures on
+  good input data in some cases. [#8273]
 
+- Added a trap in the NIRISS SOSS ATOCA algorithm for cases where nearly all
+  pixels in the 2nd-order spectrum are flagged and would cause the step
+  to fail. [#8265]
+
+extract_2d
+----------
+
+- Fixed crash when user provides an integer value for the `slit_name` argument,
+  by converting to a string. This change had been done in #8108, but it got undone
+  by another PR. [#8272]
+
+general
+-------
+
+- Update minimum required photutils version to 1.5.0 [#8211]
+
 outlier_detection
 -----------------
 
 - Removed ``grow`` from the ``outlier_detection`` step parameters,
   because it's no longer used in the algorithms. [#8190]
 
+- Fixed bug in removing intermediate files, so that the search for intermediate
+  files does not rely on the input files having a "cal" suffix, which was causing
+  original input files to accidentally get deleted instead of just the intermediate
+  files. [#8263]
+
 photom
 ------
 
@@ -69,10 +104,16 @@ photom
 - Get the values of PIXAR_A2 and PIXAR_SR from AREA reference file
   instead of PHOTOM reference file to avoid missmatching values. [#8187]
 
-- Added a hook to bypass the ``photom`` step when the ``extract_1d`` step 
+- Added a hook to bypass the ``photom`` step when the ``extract_1d`` step
   was bypassed and came before the ``photom`` step, e.g. for NIRISS SOSS
   data in the FULL subarray. [#8225]
 
+pipeline
+--------
+
+- Updated the ``calwebb_spec2`` pipeline to include NRS_BRIGHTOBJ in
+  the list of modes for running the ``nsclean`` step. [#8256]
+
 refpix
 ------
 
@@ -84,6 +125,15 @@ refpix
   to avoid failures for moderate-brightness sources due to extremely low
   throughput at the long wavelength end of MRS band 4C. [#8199]
 
+resample
+--------
+
+- Use the same ``iscale`` value for resampling science data and variance arrays. [#8159]
+
+- Changed to use the high-level APE 14 API (``pixel_to_world_values`` and
+  ``world_to_pixel_values``) for reproject, which also fixed a bug, and
+  removed support for astropy model [#8172]
+
 residual_fringe
 ---------------
 
@@ -98,17 +148,6 @@ tweakreg
 - Added option to choose IRAFStarFinder and segmentation.SourceFinder
   instead of DAOStarFinder and exposed star finder parameters. [#8203]
 
-general
--------
-
-- Update minimum required photutils version to 1.5.0 [#8211]
-
-pipeline
---------
-
-- Updated the ``calwebb_spec2`` pipeline to include NRS_BRIGHTOBJ to
-  the list of modes to run nsclean when skip is set to False. [#8256]
-
 
 1.13.4 (2024-01-25)
 ===================
@@ -198,7 +237,7 @@ documentation
   reported typos in ``tweakreg`` documentation. [#8084]
 
 emicorr
-----------
+-------
 
 - Added new step for removing EMI from all MIRI data. [#7857]
 
@@ -218,9 +257,6 @@ extract_2d
   be passed as a string, e.g. slit_name='67'. Included this
   in the corresponding documentation. [#8081]
 
-- Fixed potential future crash if MSA slitlet name is not an
-  integer. [#8108]
-
 general
 -------
 

docs/conf.py

@@ -94,6 +94,7 @@ def check_sphinx_version(expected_version):
     'matplotlib': ('http://matplotlib.org/', None),
     'gwcs': ('https://gwcs.readthedocs.io/en/stable/', None),
     'stdatamodels': ('https://stdatamodels.readthedocs.io/en/latest/', None),
+    'stcal': ('https://stcal.readthedocs.io/en/latest/', None),
 }
 
 if sys.version_info[0] == 2:

docs/jwst/jump/arguments.rst

@@ -1,7 +1,10 @@
+.. _jump_arguments:
+
 Arguments
 =========
 
-The ``jump`` step has 30 optional arguments that can be set by the user:
+The ``jump`` step has many optional arguments that can be set by the user.
+The details for each are listed below.
 
 **Parameters for Baseline Cosmic Ray Jump Detection**
 

docs/jwst/jump/description.rst

@@ -4,101 +4,34 @@ Description
 :Class: `jwst.jump.JumpStep`
 :Alias: jump
 
+This step finds and flags outliers (usually caused by cosmic-ray hits) in
+each pixel of an "up the ramp" IR exposure.
+
 Assumptions
 -----------
-We assume that the ``saturation`` step has already been applied to the input
-science exposure, so that saturated values are appropriately flagged in the
-input GROUPDQ array. We also assume that steps such as the reference pixel
-correction (``refpix``) and non-linearity correction (``linearity``) have been applied, so
-that the input data ramps do not have any non-linearities or noise above the modeled Poisson
+We assume that the :ref:`saturation <saturation_step>` step has already been applied to
+the input exposure, so that saturated ramp groups are appropriately flagged in the
+input GROUPDQ array. We also assume that steps such as
+:ref:`reference pixel correction <refpix_step>` and
+:ref:`non-linearity correction <linearity_step>` have been applied,
+so that the input data ramps do not have any non-linearities or noise above the modeled Poisson
 and read noise due to instrumental effects. The absence of any of these preceding corrections
-or residual non-linearities or noise can lead to the false detection of jumps in the ramps,
-due to departure from linearity.
+or the presence of residual non-linearities and noise can lead to false detection of jumps
+in the ramps, due to departure from linearity.
 
 The ``jump`` step will automatically skip execution if the input data contain fewer
-than 3 groups per integration, because the baseline algorithm requires two first
-differences to work.
-
-Algorithm
----------
-The algorithm for this step is called from the external package ``stcal``, an STScI
-effort to unify common calibration processing algorithms for use by multiple observatories.
-
-This routine detects jumps in an exposure by looking for outliers
-in the up-the-ramp signal for each pixel in each integration within
-an input exposure. On output, the GROUPDQ array is updated with the DQ flag
-"JUMP_DET" to indicate the location of each jump that was found.
-In addition, any pixels that have non-positive or NaN values in the gain
-reference file will have DQ flags "NO_GAIN_VALUE" and "DO_NOT_USE" set in the
-output PIXELDQ array.
-The SCI and ERR arrays of the input data are not modified.
-
-The current implementation uses the two-point difference method described
-in `Anderson & Gordon (2011) <https://ui.adsabs.harvard.edu/abs/2011PASP..123.1237A>`_.
-
-Two-Point Difference Method
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
-The two-point difference method is applied to each integration as follows:
-
-#. Compute the first differences for each pixel (the difference between
-   adjacent groups)
-#. Compute the clipped (dropping the largest difference) median of the first differences for each pixel.
-#. Use the median to estimate the Poisson noise for each group and combine it
-   with the read noise to arrive at an estimate of the total expected noise for
-   each difference.
-#. Compute the "difference ratio" as the difference between the first differences
-   of each group and the median, divided by the expected noise.
-#. If the largest "difference ratio" is greater than the rejection threshold,
-   flag the group corresponding to that ratio as having a jump.
-#. If a jump is found in a given pixel, iterate the above steps with the
-   jump-impacted group excluded, looking for additional lower-level jumps
-   that still exceed the rejection threshold.
-#. Stop iterating on a given pixel when no new jumps are found or only one
-   difference remains.
-#. If the there are only three differences (four groups), the standard median
-   is used rather than the clipped median.
-#. If there are only two differences (three groups), the smallest one is compared to the larger
-   one and if the larger one is above a threshold, it is flagged as a jump.
+than 3 groups per integration, because the baseline algorithm requires at least
+two first differences to work.
 
-#. If flagging of the 4 neighbors is requested, then the 4 adjacent pixels will
-   have ramp jumps flagged in the same group as the central pixel as long as it has
-   a jump between the min and max requested levels for this option.
+Note that the core algorithms for this step are called from the external package
+``stcal``, an STScI effort to unify common calibration processing algorithms
+for use by multiple observatories.
 
-#. If flagging of groups after a ramp jump is requested, then the groups in the
-   requested time since a detected ramp jump will be flagged as ramp jumps if
-   the ramp jump is above the requested threshold.  Two thresholds and times are
-   possible for this option.
-
-Note that any ramp values flagged as SATURATED in the input GROUPDQ array
-are not used in any of the above calculations and hence will never be
-marked as containing a jump.
-
-Multiprocessing
-===============
-This step has the option of running in multiprocessing mode. In that mode it will
-split the input data cube into a number of row slices based on the number of available
-cores on the host computer and the value of the max_cores input parameter. By
-default the step runs on a single processor. At the other extreme if max_cores is
-set to 'all', it will use all available cores (real and virtual). Testing has shown
-a reduction in the elapsed time for the step proportional to the number of real
-cores used. Using the virtual cores also reduces the elapsed time but at a slightly
-lower rate than the real cores.
-
-If multiprocessing is requested the input cube will be divided into a number of
-slices in the row dimension (with the last slice being slightly larger, if needed).
-The slices are then sent to twopoint_difference.py by detect_jumps. After all the
-slices have finished processing, detect_jumps assembles the output group_dq cube
-from the slices.
-
-Subarrays
-=========
-The use of the reference files is flexible. Full-frame reference
-files can be used for all science exposures, in which case subarrays will be
-extracted from the reference file data to match the science exposure, or
-subarray-specific reference files may be used.
+:ref:`Algorithm <stcal:jump_algorithm>`
+---------------------------------------
 
 Large Events (Snowballs and Showers)
-====================================
+------------------------------------
 All the detectors on JWST are affected by large cosmic ray
 events. While these events, in general, affect a large number of
 pixels, the more distinguishing characteristic is that they are
@@ -107,12 +40,40 @@ counts. These excess counts are, in general, below the detection
 threshold of normal cosmic rays.
 
 To constrain the effect of this halo, the jump step will fit ellipses or circles that
-enclose the large events and expand the ellipses and circles by the input expansion_factor
-and mark them as jump.
+enclose the large events and expand the ellipses and circles by the input
+`expand_factor` and mark them as jump (see :ref:`jump step arguments <jump_arguments>`
+for details).
 
-The two types of detectors respond differently. The large events in the near-infrared
+The two different types of JWST detectors respond differently. The large events in the near-infrared
 detectors are almost always circles with a central region that is saturated.
 The saturated core allows the search for smaller events without false positives.
-The MIRI detectors do not, in general, have a saturated center and are only rarely circular.
+The mid-IR (MIRI) detectors do not, in general, have a saturated center and are only rarely circular.
 Thus, we fit the minimum enclosing ellipse and do not require that there are saturated pixels
 within the ellipse.
+
+Multiprocessing
+---------------
+This step has the option of running in multiprocessing mode. In that mode it will
+split the input data cube into a number of row slices based on the number of available
+cores on the host computer and the value of the ``max_cores`` input parameter. By
+default the step runs on a single processor. At the other extreme, if ``max_cores`` is
+set to "all", it will use all available cores (real and virtual). Testing has shown
+a reduction in the elapsed time for the step proportional to the number of real
+cores used. Using the virtual cores also reduces the elapsed time, but at a slightly
+lower rate than the real cores.
+
+If multiprocessing is requested, the input cube will be divided into a number of
+slices in the row dimension (with the last slice being slightly larger, if needed),
+and sent for processing in parallel.
+In the event the number of cores (and hence slices) selected exceeds the number of
+available image rows, the number of slices will be reduced to match the number of rows.
+After all the slices have finished processing, the output GROUPDQ cube - containing
+the DQ flags for detected jumps - is reassembled from the slices.
+
+Subarrays
+---------
+Full-frame reference files can be used for all science exposures even if the
+science exposure was taken in a subarray mode. If so, subarrays will be
+extracted from the reference file data to match the science exposure.
+Alternatively, subarray-specific reference files, which match the science
+exposure, may be used.

docs/jwst/ramp_fitting/appendix.rst

@@ -1,7 +1,12 @@
 Appendix
 ========
 
-The derivation of the segment-specific readnoise variance (:math:`{ var^R_{s}  }`) is shown here. This pertains to both the 'conventional' and 'weighted` readnoise variances - the only difference being the number of groups in the segment.  This derivation follows the standard procedure to fitting data to a straight line, such as in chapter 15 of Numerical Recipes.  The segment-specific variance from read noise corresponds to :math:`{\sigma_b^2}` in section 15.2. 
+The derivation of the segment-specific readnoise variance (:math:`{ var^R_{s}  }`)
+is shown here. This pertains to both the 'conventional' and 'weighted' readnoise
+variances - the only difference being the number of groups in the segment.
+This derivation follows the standard procedure for fitting data to a straight line,
+such as in chapter 15 of Numerical Recipes.  The segment-specific variance from
+read noise corresponds to :math:`{\sigma_b^2}` in section 15.2. 
 
 For read noise R, weight w = :math:`{1 / R^2}`, which is a constant. 
 
@@ -28,7 +33,7 @@ Summations needed:
 :math:`{\sum_{k=1}^n k^2= n \cdot (n+1) \cdot (2 \cdot n+1) / 6 = n^3/3 + n^2/2 +n/6 }`
 
 
-The variance from read noise 
+The variance due to read noise 
 = :math:`{var^R_{s} = S_1 / D = S_1 / (S_1 \cdot S_{xx} - S_x^2)}` 
 
 
@@ -43,4 +48,5 @@ The variance from read noise
 
 = :math:`{ \dfrac{12 \cdot R^2}  {(n^3 - n) \cdot t^2}}` 
 
-This is the equation in the code and in the segment-specific computations section of the Description.
+This is the equation used in the step code and listed in the segment-specific computations
+section of the step Description.

docs/jwst/ramp_fitting/arguments.rst

@@ -1,6 +1,6 @@
 Arguments
 =========
-The ramp fitting step has three optional arguments that can be set by the user:
+The ramp fitting step has the following optional arguments that can be set by the user:
 
 * ``--save_opt``: A True/False value that specifies whether to write
   the optional output product. Default is False.
@@ -17,13 +17,13 @@ The ramp fitting step has three optional arguments that can be set by the user:
   saturated.
 
 * ``--maximum_cores``: The number of available cores that will be
-  used for multi-processing in this step. The default value is '1' which does not use
-  multi-processing. The other options are either an integer, 'quarter', 'half', and 'all'.
+  used for multi-processing in this step. The default value is '1', which results in no
+  multi-processing. Other options are either an integer, 'quarter', 'half', and 'all'.
   Note that these fractions refer to the total available cores and on most CPUs these include
   physical and virtual cores. The clock time for the step is reduced almost linearly by the
   number of physical cores used on all machines. For example, on an Intel CPU with
   six real cores and six virtual cores, setting maximum_cores to 'half' results in a
   decrease of a factor of six in the clock time for the step to run. Depending on the system,
-  the clock time can also decrease even more with maximum_cores is set to 'all'.
+  the clock time can also decrease even more with maximum_cores set to 'all'.
   Setting the number of cores to an integer can be useful when running on machines with a
   large number of cores where the user is limited in how many cores they can use.