fix(heic): Don't auto-transform camera-rotated images (#4142)

It seems that unlike most image formats where we merely note the camera
rotation but don't rotate the pixels, the heic reader automatically
rotated.

The trick is that using the C++ API wrapper of libheif, there is no way
to pass the decoder options struct that is where you need to say not to
auto-transform. Needed to drop down to the lower level C API for this
one spot.

We want this format to behave like the others -- the orientation is
advisory, and it's up to the app how to deal with it. But we have back
compatibility to deal with.

If a rotated image is aecountered, OIIO 2.5 and earlier will by default
still auto-rotate to preserve compatibility, but OIIO 2.6+ will not
auto-rotate.

When not rotating, the Orientation metadata will reflect the desired
display orientation of the image, according to what was in the file.

When auto-rotating, Orientation will be 1 (canonical display
orientation), and a new attribute "heif:Orientation" will reflect what
was originally in the file.

When opening a file for input, the special configuration metadata hint
"heif:reorient" can express a preference for auto-rotation, overriding
the default. Thus, setting this hint to 0 for OIIO 2.5 will turn off
auto-rotation, and setting it to 1 for OIIO 2.6+ will make it
auto-rotate like it used to do.

I also added some additional tests for heif files.

Along the way I fixed a rather embarrassing documentation problem as
well, where all the orientation codes were "off by 1", oh boy.


Fixes #4123

---------

Signed-off-by: Larry Gritz <[email protected]>

src/cmake/testing.cmake

@@ -260,8 +260,10 @@ macro (oiio_add_all_tests)
                     IMAGEDIR openexr-images
                     URL http://github.com/AcademySoftwareFoundation/openexr-images)
     oiio_add_tests (heif
-                    FOUNDVAR Libheif_FOUND ENABLEVAR ENABLE_Libheif
-                    URL https://github.com/nokiatech/heif/tree/gh-pages/content)
+                    FOUNDVAR Libheif_FOUND
+                    ENABLEVAR ENABLE_Libheif
+                    IMAGEDIR oiio-images/heif
+                    URL http://github.com/AcademySoftwareFoundation/openexr-images)
     oiio_add_tests (ico
                     ENABLEVAR ENABLE_ICO
                     IMAGEDIR oiio-images URL "Recent checkout of oiio-images")

src/doc/builtinplugins.rst

@@ -685,6 +685,11 @@ preferred except when legacy file access is required.
    * - ``oiio:Gamma``
      - float
      - the gamma correction specified in the RGBE header (if it's gamma corrected).
+   * - ``heif:Orientation``
+     - int
+     - If the configuration option ``heif:reorient`` is nonzero and
+       reorientation was performed, this will be set to the original
+       orientation in the file.
 
 
 **Configuration settings for HDR input**
@@ -704,6 +709,14 @@ options are supported:
      - ptr
      - Pointer to a ``Filesystem::IOProxy`` that will handle the I/O, for
        example by reading from memory rather than the file system.
+   * - ``oiio:reorient``
+     - int
+     - The default of 1 means to let libheif auto-reorient the image to
+       undo the camera's orientation (this will set a "heif:Orientation"
+       metadata to the Exif orientation code indicating the original
+       orientation of the image). If this hint is set to 0, the pixels will be
+       left in their orientation as stored in the file, and the "Orientation"
+       metadata will reflect that.
 
 **Configuration settings for HDR output**
 
@@ -750,6 +763,15 @@ currently supported for reading, but not yet writing. All pixel data is
 uint8, though we hope to add support for HDR (more than 8 bits) in the
 future.
 
+The default behavior of the HEIF reader is to reorient the image to the
+orientation indicated by the camera, and to report the "Orientation" metadata
+as 1 (indicating that the image should be displayed as returned) and set the
+"oiio:OriginalOrientation" metadata to what was originally stored in the file.
+If you want to read the image without automatic reorientation, you can set the
+configuration option "oiio:reorient" to 0, in which case the pixels will be
+left in their orientation as stored in the file, and the "Orientation"
+metadata will reflect that.
+
 **Configuration settings for HEIF input**
 
 When opening an HEIF ImageInput with a *configuration* (see
@@ -769,6 +791,12 @@ attributes are supported:
        cause the reader to leave alpha unassociated (versus the default of
        premultiplying color channels by alpha if the alpha channel is
        unassociated).
+   * - ``oiio:reorient``
+     - int
+     - If nonzero, asks libheif to reorient any images (and report them as
+       having Orientation 1). If zero, then libheif will not reorient the
+       image and the Orientation metadata will be set to reflect the camera
+       orientation.
 
 **Configuration settings for HEIF output**
 

src/doc/imageinput.rst

@@ -923,6 +923,10 @@ hints are supported by each reader) are:
        cause the reader to leave alpha unassociated (versus the default of
        premultiplying color channels by alpha if the alpha channel is
        unassociated).
+   * - ``oiio:reorient``
+     - int
+     - If zero, disables any automatic reorientation that the reader may
+       ordinarily do to present te pixels in the preferred display orientation.
 
 Examples:
 

src/doc/stdmetadata.rst

@@ -77,21 +77,21 @@ Display hints
 
 .. option:: "Orientation" : int
 
-    y default, image pixels are ordered from the top of the display to the
-    ottom, and within each scanline, from left to right (i.e., the same
-    rdering as English text and scan progression on a CRT).  But the
-    "Orientation"` field can suggest that it should be displayed with
-    different orientation, according to the TIFF/EXIF conventions:
+    By default, image pixels are ordered from the top of the display to the
+    bottom, and within each scanline, from left to right (i.e., the same
+    ordering as English text and scan progression on a CRT).  But the
+    `"Orientation"` field can suggest that it should be displayed with
+    a different orientation, according to the TIFF/EXIF conventions:
 
     ===  ==========================================================================
-     0   normal (top to bottom, left to right)
-     1   flipped horizontally (top to botom, right to left)
-     2   rotated :math:`180^\circ` (bottom to top, right to left)
-     3   flipped vertically (bottom to top, left to right)
-     4   transposed (left to right, top to bottom)
-     5   rotated :math:`90^\circ` clockwise (right to left, top to bottom)
-     6   transverse (right to left, bottom to top)
-     7   rotated :math:`90^\circ` counter-clockwise (left to right, bottom to top)
+     1   normal (top to bottom, left to right)
+     2   flipped horizontally (top to bottom, right to left)
+     3   rotated :math:`180^\circ` (bottom to top, right to left)
+     4   flipped vertically (bottom to top, left to right)
+     5   transposed (left to right, top to bottom)
+     6   rotated :math:`90^\circ` clockwise (right to left, top to bottom)
+     7   transverse (right to left, bottom to top)
+     8   rotated :math:`90^\circ` counter-clockwise (left to right, bottom to top)
     ===  ==========================================================================
 
 .. option:: "PixelAspectRatio" : float

src/heif.imageio/heifinput.cpp

@@ -8,6 +8,14 @@
 
 #include <libheif/heif_cxx.h>
 
+#define MAKE_LIBHEIF_VERSION(a, b, c, d) \
+    (((a) << 24) | ((b) << 16) | ((c) << 8) | (d))
+
+#if LIBHEIF_NUMERIC_VERSION >= MAKE_LIBHEIF_VERSION(1, 17, 0, 0)
+#    include <libheif/heif_properties.h>
+#endif
+
+
 
 // This plugin utilises libheif:
 //   https://github.com/strukturag/libheif
@@ -18,6 +26,7 @@
 //     https://github.com/nokiatech/heif/tree/gh-pages/content
 
 
+
 OIIO_PLUGIN_NAMESPACE_BEGIN
 
 class HeifInput final : public ImageInput {
@@ -50,6 +59,7 @@ class HeifInput final : public ImageInput {
     bool m_associated_alpha        = true;
     bool m_keep_unassociated_alpha = false;
     bool m_do_associate            = false;
+    bool m_reorient                = true;
     std::unique_ptr<heif::Context> m_ctx;
     heif_item_id m_primary_id;             // id of primary image
     std::vector<heif_item_id> m_item_ids;  // ids of all other images
@@ -137,6 +147,7 @@ HeifInput::open(const std::string& name, ImageSpec& newspec,
 
     m_keep_unassociated_alpha
         = (config.get_int_attribute("oiio:UnassociatedAlpha") != 0);
+    m_reorient = config.get_int_attribute("oiio:reorient", 1);
 
     try {
         m_ctx->read_from_file(name);
@@ -200,14 +211,14 @@ HeifInput::seek_subimage(int subimage, int miplevel)
         return false;
     }
 
+    auto id     = (subimage == 0) ? m_primary_id : m_item_ids[subimage - 1];
+    m_ihandle   = m_ctx->get_image_handle(id);
+    m_has_alpha = m_ihandle.has_alpha_channel();
+    auto chroma = m_has_alpha ? heif_chroma_interleaved_RGBA
+                              : heif_chroma_interleaved_RGB;
+#if 0
     try {
-        auto id     = (subimage == 0) ? m_primary_id : m_item_ids[subimage - 1];
-        m_ihandle   = m_ctx->get_image_handle(id);
-        m_has_alpha = m_ihandle.has_alpha_channel();
-        auto chroma = m_has_alpha ? heif_chroma_interleaved_RGBA
-                                  : heif_chroma_interleaved_RGB;
-        m_himage    = m_ihandle.decode_image(heif_colorspace_RGB, chroma);
-
+        m_himage = m_ihandle.decode_image(heif_colorspace_RGB, chroma);
     } catch (const heif::Error& err) {
         std::string e = err.get_message();
         errorf("%s", e.empty() ? "unknown exception" : e.c_str());
@@ -217,6 +228,23 @@ HeifInput::seek_subimage(int subimage, int miplevel)
         errorf("%s", e.empty() ? "unknown exception" : e.c_str());
         return false;
     }
+#else
+    std::unique_ptr<heif_decoding_options, void (*)(heif_decoding_options*)>
+        options(heif_decoding_options_alloc(), heif_decoding_options_free);
+    options->ignore_transformations = !m_reorient;
+    // print("Got decoding options version {}\n", options->version);
+    struct heif_image* img_tmp = nullptr;
+    struct heif_error herr = heif_decode_image(m_ihandle.get_raw_image_handle(),
+                                               &img_tmp, heif_colorspace_RGB,
+                                               chroma, options.get());
+    if (img_tmp)
+        m_himage = heif::Image(img_tmp);
+    if (herr.code != heif_error_Ok || !img_tmp) {
+        errorfmt("Could not decode image ({})", herr.message);
+        m_ctx.reset();
+        return false;
+    }
+#endif
 
     int bits = m_himage.get_bits_per_pixel(heif_channel_interleaved);
     m_spec = ImageSpec(m_ihandle.get_width(), m_ihandle.get_height(), bits / 8,
@@ -270,21 +298,79 @@ HeifInput::seek_subimage(int subimage, int miplevel)
             decode_xmp(metacontents, m_spec);
         } else {
 #ifdef DEBUG
-            std::cout << "Don't know how to decode meta " << m
-                      << " type=" << m_ihandle.get_metadata_type(m)
-                      << " contenttype='"
-                      << m_ihandle.get_metadata_content_type(m) << "'\n";
-            std::cout << "---\n"
-                      << string_view((const char*)&metacontents[0],
-                                     metacontents.size())
-                      << "\n---\n";
+            print(
+                "Don't know how to decode meta {} type='{}' contenttype='{}'\n",
+                m, m_ihandle.get_metadata_type(m),
+                m_ihandle.get_metadata_content_type(m));
+            print("---\n{}\n---\n",
+                  string_view((const char*)metacontents.data(),
+                              metacontents.size()));
 #endif
         }
     }
 
+    // Try to discover the orientation. The Exif is unreliable. We have to go
+    // through the transformation properties ourselves. A tricky bit is that
+    // the C++ API doesn't give us a direct way to get the context ptr, we
+    // need to resort to some casting trickery, with knowledge that the C++
+    // heif::Context class consists solely of a std::shared_ptr to a
+    // heif_context.
+    // NO int orientation = m_spec.get_int_attribute("Orientation", 1);
+    int orientation = 1;
+    const heif_context* raw_ctx
+        = reinterpret_cast<std::shared_ptr<heif_context>*>(m_ctx.get())->get();
+    int xpcount = heif_item_get_transformation_properties(raw_ctx, id, nullptr,
+                                                          100);
+    orientation = 1;
+    xpcount     = std::min(xpcount, 100);  // clamp to some reasonable limit
+    std::vector<heif_property_id> xprops(xpcount);
+    heif_item_get_transformation_properties(raw_ctx, id, xprops.data(),
+                                            xpcount);
+    for (int i = 0; i < xpcount; ++i) {
+        auto type = heif_item_get_property_type(raw_ctx, id, xprops[i]);
+        if (type == heif_item_property_type_transform_rotation) {
+            int rot = heif_item_get_property_transform_rotation_ccw(raw_ctx, id,
+                                                                    xprops[i]);
+            // cw[] maps to one additional clockwise 90 degree turn
+            static const int cw[] = { 0, 6, 7, 8, 5, 2, 3, 4, 1 };
+            for (int i = 0; i < rot / 90; ++i)
+                orientation = cw[orientation];
+        } else if (type == heif_item_property_type_transform_mirror) {
+            int mirror = heif_item_get_property_transform_mirror(raw_ctx, id,
+                                                                 xprops[i]);
+            //                                1  2  3  4  5  6  7  8
+            static const int mirrorh[] = { 0, 2, 1, 4, 3, 6, 5, 8, 7 };
+            static const int mirrorv[] = { 0, 4, 3, 2, 1, 8, 7, 6, 5 };
+            if (mirror == heif_transform_mirror_direction_vertical) {
+                orientation = mirrorv[orientation];
+            } else if (mirror == heif_transform_mirror_direction_horizontal) {
+                orientation = mirrorh[orientation];
+            }
+        }
+    }
+
     // Erase the orientation metadata because libheif appears to be doing
     // the rotation-to-canonical-direction for us.
-    m_spec.erase_attribute("Orientation");
+    if (orientation != 1) {
+        if (m_reorient) {
+            // If libheif auto-reoriented, record the original orientation in
+            // "oiio:OriginalOrientation" and set the "Orientation" attribute
+            // to 1 since we're presenting the image to the caller in the
+            // usual orientation.
+            m_spec.attribute("oiio:OriginalOrientation", orientation);
+            m_spec.attribute("Orientation", 1);
+        } else {
+            // libheif supplies oriented width & height, so if we are NOT
+            // auto-reorienting and it's one of the orientations that swaps
+            // width and height, we need to do that swap ourselves.
+            // Note: all the orientations that swap width and height are 5-8,
+            // whereas 1-4 preserve aspect ratio.
+            if (orientation >= 5) {
+                std::swap(m_spec.width, m_spec.height);
+                std::swap(m_spec.full_width, m_spec.full_height);
+            }
+        }
+    }
 
     m_subimage = subimage;
     return true;