Tiff files with WxH > signed int 32 don't work under macos

[ap--]

I'm investigating why some tiffs under macos don't work.
At the beginning I assumed it's files size, but it turns out that the ones in my local test set that work were all made with libvips-8.9.1. under windows. And the ones that don't work were made from a conda env on linux (libvips-8.8.3).

I'll update here when I have a test case with synthetic data.

[ap--]

🎉

I can generate synthetic images that fail identically compared to our test images 🥇
I am currently making them in a conda environement:

conda create -n bughunt -c conda-forge pyvips numpy

make_fake_image.py

import math
import numpy
import pyvips


def make_image(N, M):
    arr = numpy.zeros(N * M * 3, dtype=numpy.uint8)
    image = pyvips.Image.new_from_memory(arr.data, N, M, 3, 'uchar')
    image.write_to_file(
        f"test_image_{N}x{M}.tif",
        pyramid=True,
        tile=True,
        compression="jpeg",
        properties=True,
    )


if __name__ == "__main__":
    import sys
    try:
        make_image(int(sys.argv[1]), int(sys.argv[2]))
    except (TypeError, ValueError):
        print("call with integers for image width and height")

create an image failing with the identical error as ours:

user@computer:~$ python make_test_image.py 37848 81194

click to see error

(paquo) ➜  PathDrive ipython                      
Python 3.8.5 | packaged by conda-forge | (default, Jul 22 2020, 17:24:51) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from paquo.projects import QuPathProject                                                                                                                                                                       

In [2]: qp = QuPathProject('./test_projects/test_libvips_08')                                                                                                                                                          

In [3]: qp.add_image('./test_image_37848x81194.tif')                                                                                                                                                                   
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.esotericsoftware.kryo.serializers.FieldSerializer (file:/Users/poehlmann/Environments/envs/paquo/bin/QuPath-0.2.2.app/Contents/app/kryo-2.24.0.jar) to field java.util.regex.Pattern.pattern
WARNING: Please consider reporting this to the maintainers of com.esotericsoftware.kryo.serializers.FieldSerializer
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING:paquo.projects:No matching downsample for thumbnail! This might take a long time...
WARNING:QUPATH:q.l.i.s.b.BioFormatsImageServer: Temp memoization directory created at /var/folders/6w/qw4h_znn53zgxml9dzfk2kwc0000gn/T/qupath-memo-6799496160351941871
WARNING:QUPATH:q.l.i.s.b.BioFormatsImageServer: If you want to avoid this warning, either disable Bio-Formats memoization in the preferences or specify a directory to use
WARNING:QUPATH:q.l.i.s.o.OpenslideImageServer: Openslide: Property 'openslide.mpp-x' not available, will return default value NaN
WARNING:QUPATH:q.l.i.s.o.OpenslideImageServer: Openslide: Property 'openslide.mpp-y' not available, will return default value NaN
WARNING:QUPATH:q.l.i.s.o.OpenslideImageServer: Openslide: Property 'openslide.objective-power' not available, will return default value NaN
ERROR:QUPATH:q.l.i.s.o.OpenslideImageServer: Unable to read thumbnail using OpenSlide: Huffman table 0x00 was not defined
WARNING:QUPATH:q.l.i.s.o.OpenslideServerBuilder: Unable to open file:/Users/poehlmann/PathDrive/test_image_37848x81194.tif with OpenSlide: Huffman table 0x00 was not defined
WARNING:QUPATH:q.l.i.servers.ImageServerProvider: Unable to open UriImageSupport (class qupath.lib.images.servers.openslide.OpenslideServerBuilder) support=3.5, builders=1
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
~/Environments/envs/paquo/lib/python3.8/site-packages/_jpype.cpython-38-darwin.so in qupath.lib.gui.commands.ProjectImportImagesCommand.getThumbnailRGB()

~/Environments/envs/paquo/lib/python3.8/site-packages/_jpype.cpython-38-darwin.so in qupath.lib.images.servers.AbstractImageServer.getDefaultThumbnail()

~/Environments/envs/paquo/lib/python3.8/site-packages/_jpype.cpython-38-darwin.so in qupath.lib.images.servers.AbstractTileableImageServer.readBufferedImage()

~/Environments/envs/paquo/lib/python3.8/site-packages/_jpype.cpython-38-darwin.so in qupath.lib.images.servers.AbstractTileableImageServer.readBufferedImage()

~/Environments/envs/paquo/lib/python3.8/site-packages/_jpype.cpython-38-darwin.so in qupath.lib.images.servers.AbstractTileableImageServer.createDefaultRGBImage()

~/Environments/envs/paquo/lib/python3.8/site-packages/_jpype.cpython-38-darwin.so in java.awt.image.BufferedImage.<init>()

~/Environments/envs/paquo/lib/python3.8/site-packages/_jpype.cpython-38-darwin.so in java.awt.image.DirectColorModel.createCompatibleWritableRaster()

~/Environments/envs/paquo/lib/python3.8/site-packages/_jpype.cpython-38-darwin.so in java.awt.image.Raster.createPackedRaster()

~/Environments/envs/paquo/lib/python3.8/site-packages/_jpype.cpython-38-darwin.so in java.awt.image.DataBufferInt.<init>()

Exception: Java Exception

The above exception was the direct cause of the following exception:

java.lang.NegativeArraySizeException      Traceback (most recent call last)
~/Development/paquo/paquo/projects.py in add_image(self, filename, image_type, allow_duplicates)
    216             try:
--> 217                 thumbnail = ProjectImportImagesCommand.getThumbnailRGB(server, None)
    218             except NegativeArraySizeException:

java.lang.NegativeArraySizeException: java.lang.NegativeArraySizeException: -1221936784

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-3-f6ccd20e61be> in <module>
----> 1 qp.add_image('./test_image_37848x81194.tif')

~/Environments/envs/paquo/lib/python3.8/contextlib.py in inner(*args, **kwds)
     73         def inner(*args, **kwds):
     74             with self._recreate_cm():
---> 75                 return func(*args, **kwds)
     76         return inner
     77 

~/Development/paquo/paquo/projects.py in add_image(self, filename, image_type, allow_duplicates)
    217                 thumbnail = ProjectImportImagesCommand.getThumbnailRGB(server, None)
    218             except NegativeArraySizeException:
--> 219                 raise RuntimeError(
    220                     "Thumbnailing FAILED. Image might be too large and has no embedded thumbnail."
    221                 )

RuntimeError: Thumbnailing FAILED. Image might be too large and has no embedded thumbnail.

Dear friend integer overflow:

The java error made me suspicious:

java.lang.NegativeArraySizeException: java.lang.NegativeArraySizeException: -1221936784

see:

>>> import ctypes
>>> ctypes.c_int32(37848 * 81194).value
-1221936784

[sdvillal]

Is it then that we are not creating bigtiffs?
http://bigtiff.org/

libvips has a switch which obviously we need to use

[sdvillal]

I would simply stop converting files this way, if possible. We have plenty alternatives.

[ap--]

This is my preferred method of handling this issue 😅

[ap--]

But so, these open fine under Windows and Linux QuPath and they crash the import process on MacOS QuPath.
So I guess it's worth reporting this to qupath upstream, because it's a bug in the macos version of qupath.