New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP DXTBX compatibility with numpy 2.0 #751

Draft

ndevenish wants to merge 3 commits into main from numpy_2

Draft

WIP DXTBX compatibility with numpy 2.0 #751

Unrestrict numpy in dxtbx CI

Azure Pipelines / cctbx.dxtbx failed Aug 16, 2024 in 14m 50s

Build #20240816.15 had test failures

8 errors / 20 warnings

Details

Tests

Failed: 32 (0.69%)
Passed: 2,356 (51.00%)
Other: 2,232 (48.31%)
Total: 4,620

Code coverage

1965 of 5731 branches covered (34.29%)
7475 of 16890 lines covered (44.26%)

Annotations

Check failure on line 1576 in Build log

azure-pipelines / cctbx.dxtbx

Build log #L1576

5 test(s) failed, 770 test(s) collected.

Check failure on line 794 in Build log

azure-pipelines / cctbx.dxtbx

Build log #L794

6 test(s) failed, 770 test(s) collected.

Check failure on line 1825 in Build log

azure-pipelines / cctbx.dxtbx

Build log #L1825

Cmd.exe exited with code '1'.

Check failure on line 794 in Build log

azure-pipelines / cctbx.dxtbx

Build log #L794

6 test(s) failed, 770 test(s) collected.

Check failure on line 1 in tests/command_line/test_show_mask_info.py::test_show_mask_info

azure-pipelines / cctbx.dxtbx

tests/command_line/test_show_mask_info.py::test_show_mask_info

[gw0] linux -- Python 3.9.19 /home/vsts/work/1/conda_base/bin/python3.9

dials_data = &lt;DataFetcher: /home/vsts/work/1/data&gt;
capsys = &lt;_pytest.capture.CaptureFixture object at 0x7f93e2b38960&gt;

    def test_show_mask_info(dials_data, capsys):
        data = dials_data("image_examples", pathlib=True) / "dectris_eiger_master.h5"
    
&gt;       show_mask_info.run([str(data)])

tests/command_line/test_show_mask_info.py:9: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../conda_base/lib/python3.9/site-packages/dxtbx/command_line/show_mask_info.py:18: in run
    el = ExperimentListFactory.from_filenames(options.filenames)
../../conda_base/lib/python3.9/site-packages/dxtbx/model/experiment_list.py:678: in from_filenames
    imageset = format_class.get_imageset(
../../conda_base/lib/python3.9/site-packages/dxtbx/format/FormatMultiImage.py:161: in get_imageset
    format_instance = cls.get_instance(filenames[0], **format_kwargs)
../../conda_base/lib/python3.9/site-packages/dxtbx/format/Format.py:274: in get_instance
    Class._current_instance_ = Class(filename, **kwargs)
../../conda_base/lib/python3.9/site-packages/dxtbx/format/FormatHDF5.py:16: in __init__
    Format.__init__(self, image_file, **kwargs)
../../conda_base/lib/python3.9/site-packages/dxtbx/format/Format.py:168: in __init__
    self.setup()
../../conda_base/lib/python3.9/site-packages/dxtbx/format/Format.py:178: in setup
    self._start()
../../conda_base/lib/python3.9/site-packages/dxtbx/format/FormatHDF5EigerNearlyNexus.py:358: in _start
    self._detector_model = DetectorFactory(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = &lt;dxtbx.format.nexus.DetectorFactory object at 0x7f93e2b0b820&gt;
obj = &lt;dxtbx.format.nexus.NXdetector object at 0x7f93e294ffa0&gt;
beam = &lt;dxtbx_model_ext.Beam object at 0x7f93e57dc830&gt;, shape = (1, 3269, 3110)

    def __init__(self, obj, beam, shape=None):
        # Get the handles
        nx_file = obj.handle.file
        nx_detector = obj.handle
        nx_module = obj.modules[0].handle
    
        # Get the detector name and type
        if "type" in nx_detector:
            detector_type = str(nx_detector["type"][()])
        else:
            detector_type = "unknown"
        detector_name = str(nx_detector.name)
    
        try:
            # underload_value: The lowest value at which pixels for this detector would be reasonably be measured.
            # https://manual.nexusformat.org/classes/applications/NXmx.html#nxmx-entry-instrument-detector-underload-value-field
            min_trusted_value = float(nx_detector["underload_value"][()])
        except KeyError:
            min_trusted_value = -0x7FFFFFFF
    
        try:
            # saturation_value: The value at which the detector goes into saturation.
            # Data above this value is known to be invalid.
            # https://manual.nexusformat.org/classes/applications/NXmx.html#nxmx-entry-instrument-detector-saturation-value-field
            max_trusted_value = float(nx_detector["saturation_value"][()])
        except KeyError:
            max_trusted_value = 0x7FFFFFFF
    
        trusted_range = min_trusted_value, max_trusted_value
    
        # Get the detector thickness
        thickness = nx_detector["sensor_thickness"]
        thickness_value = float(thickness[()])
        thickness_units = thickness.attrs["units"]
        thickness_value = float(convert_units(thickness_value, thickness_units, "mm"))
    
        # Get the detector material
        material = {
            "Si": "Si",
            "Silicon": "Si",
            "Sillicon": "Si",
            "CdTe": "CdTe",
            "GaAs": "GaAs",
        }.get(h5str(nx_detector["sensor_material"][()]))
        if not material:
            raise RuntimeError(
                "Unknown material: %s" % nx_detector["sensor_material"][()]
            )
    
        try:
            x_pixel = nx_detector["x_pixel_siz

Raw output


            /home/vsts/work/1/conda_base/lib/python3.9/site-packages/dxtbx/format/nexus.py:1112: Boost.Python.ArgumentError: Python argument types in

Check failure on line 1 in tests/model/test_detector.py::test_get_panel_projection_2d_from_axes

azure-pipelines / cctbx.dxtbx

tests/model/test_detector.py::test_get_panel_projection_2d_from_axes

[gw1] linux -- Python 3.9.19 /home/vsts/work/1/conda_base/bin/python3.9

dials_data = &lt;DataFetcher: /home/vsts/work/1/data&gt;

    def test_get_panel_projection_2d_from_axes(dials_data):
        # Get test data
        pytest.importorskip("h5py")
        filename = dials_data("image_examples", pathlib=True) / "dectris_eiger_master.h5"
&gt;       experiment = ExperimentListFactory.from_filenames([filename])[0]

tests/model/test_detector.py:331: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../conda_base/lib/python3.9/site-packages/dxtbx/model/experiment_list.py:678: in from_filenames
    imageset = format_class.get_imageset(
../../conda_base/lib/python3.9/site-packages/dxtbx/format/FormatMultiImage.py:161: in get_imageset
    format_instance = cls.get_instance(filenames[0], **format_kwargs)
../../conda_base/lib/python3.9/site-packages/dxtbx/format/Format.py:274: in get_instance
    Class._current_instance_ = Class(filename, **kwargs)
../../conda_base/lib/python3.9/site-packages/dxtbx/format/FormatHDF5.py:16: in __init__
    Format.__init__(self, image_file, **kwargs)
../../conda_base/lib/python3.9/site-packages/dxtbx/format/Format.py:168: in __init__
    self.setup()
../../conda_base/lib/python3.9/site-packages/dxtbx/format/Format.py:178: in setup
    self._start()
../../conda_base/lib/python3.9/site-packages/dxtbx/format/FormatHDF5EigerNearlyNexus.py:358: in _start
    self._detector_model = DetectorFactory(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = &lt;dxtbx.format.nexus.DetectorFactory object at 0x7fadb48e9730&gt;
obj = &lt;dxtbx.format.nexus.NXdetector object at 0x7fadb48a5550&gt;
beam = &lt;dxtbx_model_ext.Beam object at 0x7fadb47d9cb0&gt;, shape = (1, 3269, 3110)

    def __init__(self, obj, beam, shape=None):
        # Get the handles
        nx_file = obj.handle.file
        nx_detector = obj.handle
        nx_module = obj.modules[0].handle
    
        # Get the detector name and type
        if "type" in nx_detector:
            detector_type = str(nx_detector["type"][()])
        else:
            detector_type = "unknown"
        detector_name = str(nx_detector.name)
    
        try:
            # underload_value: The lowest value at which pixels for this detector would be reasonably be measured.
            # https://manual.nexusformat.org/classes/applications/NXmx.html#nxmx-entry-instrument-detector-underload-value-field
            min_trusted_value = float(nx_detector["underload_value"][()])
        except KeyError:
            min_trusted_value = -0x7FFFFFFF
    
        try:
            # saturation_value: The value at which the detector goes into saturation.
            # Data above this value is known to be invalid.
            # https://manual.nexusformat.org/classes/applications/NXmx.html#nxmx-entry-instrument-detector-saturation-value-field
            max_trusted_value = float(nx_detector["saturation_value"][()])
        except KeyError:
            max_trusted_value = 0x7FFFFFFF
    
        trusted_range = min_trusted_value, max_trusted_value
    
        # Get the detector thickness
        thickness = nx_detector["sensor_thickness"]
        thickness_value = float(thickness[()])
        thickness_units = thickness.attrs["units"]
        thickness_value = float(convert_units(thickness_value, thickness_units, "mm"))
    
        # Get the detector material
        material = {
            "Si": "Si",
            "Silicon": "Si",
            "Sillicon": "Si",
            "CdTe": "CdTe",
            "GaAs": "GaAs",
        }.get(h5str(nx_detector["sensor_material"][()]))
        if not material:
            raise RuntimeError(
                "Unknown material: %s" % nx_detector["sensor_material"][()]
            )
    
        try:
            x_pixel = nx_detector["x_pixel_size"][()] * 1000.0
            y_pixel = nx_detector["y_pixel_size"][()] * 1000.0
    
            legacy_beam_x = float(x_pixel * nx_detecto

Raw output


            /home/vsts/work/1/conda_base/lib/python3.9/site-packages/dxtbx/format/nexus.py:1112: Boost.Python.ArgumentError: Python argument types in

Check failure on line 1 in tests/format/test_FormatHDF5EigerNearlyNexus.py::test_semi_synthetic_dectris_eiger_nearly_nexus

azure-pipelines / cctbx.dxtbx

tests/format/test_FormatHDF5EigerNearlyNexus.py::test_semi_synthetic_dectris_eiger_nearly_nexus

[gw0] linux -- Python 3.9.19 /home/vsts/work/1/conda_base/bin/python3.9

dials_data = &lt;DataFetcher: /home/vsts/work/1/data&gt;
tmpdir = local('/home/vsts/work/1/tests/popen-gw0/test_semi_synthetic_dectris_ei0')

    def test_semi_synthetic_dectris_eiger_nearly_nexus(dials_data, tmpdir):
        master_h5 = dials_data("image_examples", pathlib=True) / "dectris_eiger_master.h5"
    
        if not os.access(master_h5, os.R_OK):
            pytest.skip("Test images not available")
    
        assert FormatHDF5EigerNearlyNexus.understand(master_h5)
    
&gt;       expts = ExperimentListFactory.from_filenames([master_h5])

tests/format/test_FormatHDF5EigerNearlyNexus.py:43: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../conda_base/lib/python3.9/site-packages/dxtbx/model/experiment_list.py:678: in from_filenames
    imageset = format_class.get_imageset(
../../conda_base/lib/python3.9/site-packages/dxtbx/format/FormatMultiImage.py:161: in get_imageset
    format_instance = cls.get_instance(filenames[0], **format_kwargs)
../../conda_base/lib/python3.9/site-packages/dxtbx/format/Format.py:274: in get_instance
    Class._current_instance_ = Class(filename, **kwargs)
../../conda_base/lib/python3.9/site-packages/dxtbx/format/FormatHDF5.py:16: in __init__
    Format.__init__(self, image_file, **kwargs)
../../conda_base/lib/python3.9/site-packages/dxtbx/format/Format.py:168: in __init__
    self.setup()
../../conda_base/lib/python3.9/site-packages/dxtbx/format/Format.py:178: in setup
    self._start()
../../conda_base/lib/python3.9/site-packages/dxtbx/format/FormatHDF5EigerNearlyNexus.py:358: in _start
    self._detector_model = DetectorFactory(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = &lt;dxtbx.format.nexus.DetectorFactory object at 0x7f93e2aeb690&gt;
obj = &lt;dxtbx.format.nexus.NXdetector object at 0x7f93e2aeb1e0&gt;
beam = &lt;dxtbx_model_ext.Beam object at 0x7f93e246f0b0&gt;, shape = (1, 3269, 3110)

    def __init__(self, obj, beam, shape=None):
        # Get the handles
        nx_file = obj.handle.file
        nx_detector = obj.handle
        nx_module = obj.modules[0].handle
    
        # Get the detector name and type
        if "type" in nx_detector:
            detector_type = str(nx_detector["type"][()])
        else:
            detector_type = "unknown"
        detector_name = str(nx_detector.name)
    
        try:
            # underload_value: The lowest value at which pixels for this detector would be reasonably be measured.
            # https://manual.nexusformat.org/classes/applications/NXmx.html#nxmx-entry-instrument-detector-underload-value-field
            min_trusted_value = float(nx_detector["underload_value"][()])
        except KeyError:
            min_trusted_value = -0x7FFFFFFF
    
        try:
            # saturation_value: The value at which the detector goes into saturation.
            # Data above this value is known to be invalid.
            # https://manual.nexusformat.org/classes/applications/NXmx.html#nxmx-entry-instrument-detector-saturation-value-field
            max_trusted_value = float(nx_detector["saturation_value"][()])
        except KeyError:
            max_trusted_value = 0x7FFFFFFF
    
        trusted_range = min_trusted_value, max_trusted_value
    
        # Get the detector thickness
        thickness = nx_detector["sensor_thickness"]
        thickness_value = float(thickness[()])
        thickness_units = thickness.attrs["units"]
        thickness_value = float(convert_units(thickness_value, thickness_units, "mm"))
    
        # Get the detector material
        material = {
            "Si": "Si",
            "Silicon": "Si",
            "Sillicon": "Si",
            "CdTe": "CdTe",
            "GaAs": "GaAs",
        }.get(h5str(nx_detector["sensor_material"][()]))
        if not material:
            raise RuntimeError(
                "Unknown material: %s" % nx_detector["sensor_ma

Raw output


            /home/vsts/work/1/conda_base/lib/python3.9/site-packages/dxtbx/format/nexus.py:1112: Boost.Python.ArgumentError: Python argument types in

Check failure on line 1 in tests/nexus/test_jf16M.py::test_jf16M_matches_expected_hierarchy

azure-pipelines / cctbx.dxtbx

tests/nexus/test_jf16M.py::test_jf16M_matches_expected_hierarchy

[gw1] linux -- Python 3.9.19 /home/vsts/work/1/conda_base/bin/python3.9

dials_data = &lt;DataFetcher: /home/vsts/work/1/data&gt;

    def test_jf16M_matches_expected_hierarchy(dials_data):
        try:
            h5path = (
                dials_data("lysozyme_JF16M_4img", pathlib=True)
                / "lyso009a_0087.JF07T32V01_master_4img.h5"
            )
        except Exception as e:
            print(type(e), str(e))
            raise
&gt;       img = dxtbx.load(h5path)

tests/nexus/test_jf16M.py:16: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../conda_base/lib/python3.9/site-packages/dxtbx/__init__.py:69: in load
    return format_instance(os.fspath(filename))
../../conda_base/lib/python3.9/site-packages/dxtbx/format/FormatNXmx.py:59: in __init__
    super().__init__(image_file, **kwargs)
../../conda_base/lib/python3.9/site-packages/dxtbx/format/FormatHDF5.py:16: in __init__
    Format.__init__(self, image_file, **kwargs)
../../conda_base/lib/python3.9/site-packages/dxtbx/format/Format.py:168: in __init__
    self.setup()
../../conda_base/lib/python3.9/site-packages/dxtbx/format/Format.py:178: in setup
    self._start()
../../conda_base/lib/python3.9/site-packages/dxtbx/format/FormatNXmx.py:74: in _start
    self._detector_model = dxtbx.nexus.get_dxtbx_detector(nxdetector, wavelength)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

nxdetector = &lt;nxmx.NXdetector object at 0x7fadb451c9b0&gt;, wavelength = 1.368479

    def get_dxtbx_detector(
        nxdetector: nxmx.NXdetector,
        wavelength: float,
    ) -&gt; dxtbx.model.Detector:
        """Generate a dxtbx detector model from an NXdetector and NXbeam.
    
        If the NXdetector contains multiple NXdetector_modules, then a hierarchical detector
        will be returned, else a "flat" detector model with a single panel will be returned
        where there is only a single NXdetector_module.
        """
    
        detector = dxtbx.model.Detector()
    
        root: dxtbx.model.Detector | dxtbx.model.Panel
        if len(nxdetector.modules) &gt; 1:
            root = detector.hierarchy()
        else:
            root = detector
    
        for module in nxdetector.modules:
            if len(nxdetector.modules) &gt; 1:
                # Set up the detector hierarchy
                if module.fast_pixel_direction.depends_on is not None:
                    reversed_dependency_chain = list(
                        reversed(
                            nxmx.get_dependency_chain(
                                module.fast_pixel_direction.depends_on
                            )
                        )
                    )
                    pg: dxtbx.model.Detector | dxtbx.model.Panel | None = None
    
                    # Verify that equipment_components in the dependency chain are
                    # 1) contiguous and 2) unique
                    found = []
                    for dependency in reversed_dependency_chain:
                        if dependency.equipment_component:
                            if dependency.equipment_component in found:
                                assert dependency.equipment_component == found[-1]
                            found.append(dependency.equipment_component)
    
                    # Group any transformations together that share the same equipment_component
                    # to reduce the number of hierarchy levels
    
                    # Keep transformations without equipment_component set separate by using
                    # a different key
                    counter = 0
    
                    def equipment_component_key(dependency):
                        if dependency.equipment_component:
                            return dependency.equipment_component  # always a string
                        else:
                            nonlocal counter
                            counter += 1

Raw output


            /home/vsts/work/1/conda_base/lib/python3.9/site-packages/dxtbx/nexus/__init__.py:446: Boost.Python.ArgumentError: Python argument types in

View more details on Azure Pipelines

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP DXTBX compatibility with numpy 2.0 #751

WIP DXTBX compatibility with numpy 2.0 #751

Build #20240816.15 had test failures

Details

Annotations

azure-pipelines / cctbx.dxtbx

azure-pipelines / cctbx.dxtbx

azure-pipelines / cctbx.dxtbx

azure-pipelines / cctbx.dxtbx

azure-pipelines / cctbx.dxtbx

azure-pipelines / cctbx.dxtbx

azure-pipelines / cctbx.dxtbx

azure-pipelines / cctbx.dxtbx

Re-running checks...

WIP DXTBX compatibility with numpy 2.0 #751

Are you sure you want to change the base?

Unrestrict numpy in dxtbx CI

WIP DXTBX compatibility with numpy 2.0 #751

Build #20240816.15 had test failures

Details

Annotations

azure-pipelines / cctbx.dxtbx

azure-pipelines / cctbx.dxtbx

azure-pipelines / cctbx.dxtbx

azure-pipelines / cctbx.dxtbx

azure-pipelines / cctbx.dxtbx

azure-pipelines / cctbx.dxtbx

azure-pipelines / cctbx.dxtbx

azure-pipelines / cctbx.dxtbx

Re-running checks...