NXmx general writer

[phyy-nx]

This adds a general NXmx writer to dxtbx

Implemented along the lines of the existing cbf_writer that is used by the image averager, it has these important features:

• Multi panel, hierarchical detector support. This combines homogenous coordinate transformations implemented for the cbf_writer with the NeXus writing code in the SwissFEL writer to read general dxtbx detector models and write them in NeXus format.
• Handles the general spectra case, including 1D and 2D spectra, single shot and multishot spectra, and testing for variants, where the wavelength is not the same as the spectrum weighted wavelength. Tested using SwissFEL data with spectra.
• Handles appending data to existing file by using special h5py syntax for noting a dataset has no maximum size.
• Handles using external untrusted pixels masks from DIALS and adding them to NeXus

What is missing:

• Scans
• Gonios

The plan is to add these in an upcoming DIALS code camp :)

Here's a couple of examples of running it. First, this one eats 10 cbfs files and makes a single NeXus file:

libtbx.python `libtbx.find_in_repositories dxtbx`/src/dxtbx/format/nxmx_writer.py ../thaumatin/data/th_8_2_000*.cbf output_file=ten_thau.h5 instrument_name="dummy" source_name="dummy" start_time="dummy" end_time_estimated="dummy" sample_name="dummy"

Note that there are some required NeXus keys that can't be read from the dxtbx object models. Those have to be provided somehow so I've made the API require them.

Second example. In this one I am trying to be more careful about the required NeXus metadata. This example is on a SwissFEL master with spectra, so the result combines the master and its subfiles into one file.

libtbx.python `libtbx.find_in_repositories dxtbx`/src/dxtbx/format/nxmx_writer.py ../run_000795.JF07T32V01_raw_master_new.h5 output_file=run_795.h5 instrument_name="SwissFEL ARAMIS BEAMLINE ESB" instrument_short_name="ESB" source_name="SwissFEL ARAMIS" source_short_name="SwissFEL ARAMIS" start_time="2022-09-01T00:00:00.000" end_time_estimated="2022-09-01T00:05:00.000Z" sample_name="Sample1"

I think there's some room for improvement:

• I think the bitdepth of the output is too high (int32 or float64). I think it should be able to autodetect the useful bit depth, then downshift as needed.
• I also think we can compress the data.

[codecov]

Codecov Report

Merging #615 (4f1501e) into main (eda7169) will decrease coverage by 0.98%.
The diff coverage is 0.24%.

❗ Current head 4f1501e differs from pull request most recent head 273b01a. Consider uploading reports for the commit 273b01a to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #615      +/-   ##
==========================================
- Coverage   39.08%   38.11%   -0.98%     
==========================================
  Files         181      182       +1     
  Lines       15864    16268     +404     
  Branches     3066     3160      +94     
==========================================
  Hits         6201     6201              
- Misses       9077     9481     +404     
  Partials      586      586              

[phyy-nx]

Ok, this is ready for others to look at. Notes:

1. This PR includes NXmx: handle multidimensional arrays #612 so that should probably go in first
2. There are changes here to imageset.py, and nexus/__init.py that I found while working on the writer
3. I think the fixed rotation rotation matrix in goniometers is not supported here, which is worthy of review
4. I think the s0 vector isn't supported either

I'd like to take a look at 3 and 4 before merging, but they aren't stoppers I think.

[phyy-nx]

Ok the tests should pass now. This ready for re-review! Again, depends on #612.

[phyy-nx]

@ndevenish looks ready? :)

[ndevenish]

Yeah, I think so. The release is currently running through CI, which means the branch-off is chosen, so it'll be "official" next release, barring build failures.

[phyy-nx]

Yay!