feat: Python wheels workflow and build backend #4428

zachlewis · 2024-09-16T14:45:45Z

Description

Summary

This PR is the spiritual successor to #3249. It implements a scikit-build-core-based python build-backend, making it possible to use pip install . to build from source; and it adds a Github workflow for building with cibuildwheel and publishing to pypi.org binary distributions (bdists) of the Python module / extensions / CLI tools for cpython 3.8-3.12, across major operating systems and architectures.

When you pip install OpenImageIO, pip attempts to retrieve an OpenImageIO bdist from pypi.org for the host's platform / architecture / Python interpreter. If it can't find something appropriate, pip will attempt to build locally from the OpenImageIO source distribution (sdist), downloading and temporarily installing cmake and ninja if necessary.

PEP-Compliant Packaging: `pyproject.toml`

The pyproject.toml file is organized in three parts:

Package metadata: standard attributes identifying and describing the Python package, its run-time and build-time requirements, entry-points to executable scripts, and so forth.
scikit-build-core options: governs how pip install ... interacts with cmake.
cibuildwheel options: additional steps and considerations for building, packaging, "repairing", and testing relocatable wheel build artifacts in isolated environments.

Additions to ` init .py`

Previously, we were using a custom OpenImageIO/__ init__.py file to help Python-3.8+ on Windows load the shared libraries linked by the Python module (i.e., the .dll files that live alongside oiiotool.exe under $PATH).

This PR adds an additional method for loading the DLL path, necessitated by differences between pip-based and traditional CMake-based installs.

It also adds a mechanism for invoking binary executables found in the .../site-packages/OpenImageIO/bin directory. This provides a means for exposing Python script "shims" for each CLI tool, installed to platform-specific locations under $PATH, while keeping the actual binaries in a static location relative to the dynamic libraries. Upshot is, in pyproject.toml,
each item under [project.scripts] is turned into a Python script upon installation that behaves indistinguishably to the end user to the CLI binary executable of the same name.

Relocatable Binary Distributions with `cibuildwheel` + `repairwheel`

cibuildwheel is a widely-used tool for drastically streamlining the process of building, repairing, and testing Python wheels across platforms, architectures, interpreters, and interpreter versions.

Importantly, cibuildwheel allows for an arbitrary "repair-wheel-command" step, intended as a means for collecting and bundling any external shared libraries linked by the module (or the CLI tools) living outside the package, and subsequently patching RPATH entries to point to relative paths to these libraries inside the wheel. This ensures that compiled extensions have no runtime dependencies outside the pip toolchain.

In order to keep wheel file sizes to an absolute minimum, our repair-wheel-command is a multi-step process, implemented in a new tasks.py file living in the repo's root, invoked with invoke:

Remove everything except for the python module and the binary executables under /bin.
Use repairwheel to collect and bundle only and exactly the shared libraries needed -- i.e., missing dependencies built by OIIO's build system, as well as any other needed libraries found in the environment.

Additionally, the cibuildwheel-based builds set CMAKE_BUILD_TYPE to "MinSizeRel" to optimize for size (instead of speed) -- this seems to shave ~1.5MB off each .whl's size.

Note: the steps taken in tasks.py remove everything except that which is absolutely necessary for end users to import the Python modules and run the CLI tools, partly in order to determine a baseline minimum file size for the wheels. However, part of the motivation for setting things up the way I have is to make it easier to add things back into the wheels in the future; e.g., should we decide to package the headers and/or the fonts, generate or modify pkgconfig .pcs and/or CMake modules and configs, etc. Also worth noting, there's pyproject.toml metadata we can add to help downstream scikit-build-core-based builds find OpenImageIO.

"Wheels" Github workflow

I straight-up copied .github/workflows/wheel.yml from OpenColorIO and made a few OIIO-specific modifications. When pushing a commit tagged v2.6* or v3*, the workflow will invoke a platform-agnostic "build sdist" (source distribution) task, followed by a series of tasks for building OIIO wheels for cpython-3.8-3.12 on Windows, Linux (x86_64 + aarch64, new libstdc++), and MacOS (x86_64 + arm64, min deployment 10.15) and persisting build artifacts; followed finally by a task for publishing the build artifacts to pypi.org

Note: For the sake of simplicity and troubleshooting, I've made as few changes to OpenColorIO's wheel.yml as I could get away with; but in the future, we can also build wheels for the PyPy interpreter, and possibly pyodide.

Note: THE PUBLISH TASK WILL FAIL until a "trusted publisher" is set up on pypi.org. See https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/

Additional Dependency Recipes

In my testing, I found that I needed to add a handful of src/cmake/build_xxxx.cmake scripts to successfully produce wheels with cibuildwheel:

libjpeg-turbo: Mandatory OIIO dependency that doesn't otherwise exist on the manylinux Docker images that cibuildwheel uses by default.
zlib: Mandatory OIIO dependency not found on vanilla Windows runners
yaml-cpp: Mandatory OCIO dependency that seems to cause trouble on Windows for OIIO, when OCIO is left to build and install the missing dependency itself. The exact nature of the problem warrants further investigation; but having OIIO provide yaml-cpp for the OCIO build suffices for the purpose of generating bdists.

I've submitted separate PRs for these, since they fall outside the intended scope of this PR.

Note: One of the aims for this PR is to leverage OIIO's new dependency-self-building mechanisms as much as possible, so that users who pip install from source are guaranteed a nominally working installation. However, we can also use platform-specific package managers for the wheels uploaded to pypi.org (e.g., added as cibuildwheel "before-build" commands for each platform).

Other Changes

I made some minor adjustments to pythonutils.cmake and fancy_add_executable.cmake that only affect scikit-build-core-based installs -- namely, on Linux and macOS, I'm setting the INSTALL_RPATH property to point to the relative path to the dynamic libraries, for the Python module and CLI tools, respectively. This helps ensure that pip-based builds and installs from source (as opposed to installs from repaired, pre-built wheels) yield relocatable, importable packages, without needing to mess with $LD_LIBRARY_PATH etc.

Tests

The cibuildwheel "test-command" is oiiotool --help. If that command elicits code zero, it means the "oiiotool" Python script installed by the wheel is able to import OpenImageIO; that the actual binary executable oiiotool is properly packaged and exists in the expected location (e.g., at .../site-packages/OpenImageIO/bin); and that all runtime dependencies are found.

The build system has been updated to specifically detect the Python3 Development.Module meta component, as opposed to the entire Development component. This allows for better compatibility with python distributions that do not provide the Development.Embed component, which is only required for projects that ship embedded Python interpreters. The changes have been made in CMakeLists.txt and pythonutils.cmake files. Signed-off-by: Zach Lewis <[email protected]>

@aclark4life

Scikit-build-core is used for collecting CMake and Ninja as needed, and for invoking the build. When invoked via cibuildwheels, `repairwheel` is used after each build to re-bundle and relink the shared library dependencies into properly redistributable whl archives. The command-line tools are exposed under the [project.scripts] section. This commit incorporates or is otherwise inspired by similar efforts by @aclark4life and @JeanChristopheMorinPerso, as well as @remia's work on the OpenColorIO wheels. Signed-off-by: Zach Lewis <[email protected]>

Use Apache-2.0 license identifier instead of BSD-3-Clause. Add "OpenImageIO Contributors" / "[email protected]" as author. I could not bring myself to remove Larry as an author and maintainer. Signed-off-by: Zach Lewis <[email protected]>

…tic libdeflate It's now possible to disable building of shared `libdeflate` libs. Also, we're checking for and aliasing `libdeflate` in `externalpackages.cmake`, just before checking for TIFF, as opposed to only doing so within `build_TIFF.cmake`. This change is necessary for certain build systems and pipelines that utilize cached dependency builds. Specifically, when building wheels for multiple versions of cpython, `cibuildwheel` would complete the first build, and then throw an exception on the *second* build re: not being able to find `Deflate::Deflate`. Moving the aliasing above the check for TIFF ensures that the expected aliasing always takes place, whether or not TIFF needs to be built; whereas before, we were only creating the alias when initially building TIFF. Signed-off-by: Zach Lewis <[email protected]>

Build and link missing libjpeg-turbo shared + static libs Signed-off-by: Zach Lewis <[email protected]>

Instead of creating a separate OpenImageIO.OpenImageIO.command_line module for the CLI shims, move the CLI shim logic up to a "_command_line()" method in OpenImageIO.__init__.py. (Maybe this method should still be called "main()" though?) This also means the module is technically importable from OpenImageIO.OpenImageIO, but that's an improvement over OpenImageIO.OpenImageIO.OpenImageIO...! Signed-off-by: Zach Lewis <[email protected]>

…oftwareFoundation#4358) As suggested by Moritz Moeller Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…twareFoundation#4359) * Get rid of some obsolete cmake code. * Movement (but no change) to some parts of CMakeLists.txt, primarily to make it closer to the corresponding file in OSL to make it easy for me to diff them and port innovations back and forth between them. * Some typo/etc fixes * Remove unused OIIO_UNUSED_OK macro that's been deprecated since 2.0, and OIIO_CONSTEXPR and OIIO_CONSTEXPR14, neither of which have been needed for years. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

This seems to break builds under certain circumstances. Better to handle the problem with cached rebuilds another way, either in a FindLibdeflate.cmake, or by always locally-building libdeflate and TIFF. Signed-off-by: Zach Lewis <[email protected]>

…directory, under module root Added conditional logic to set relative RPATHs when building with scikit-build. This change ensures that the Python module and compiled cli tools correctly find all built dynamic libraries relative to a shared root, and keeps distributions self-contained and relocatable (i.e., without requiring a `repairwheel` step). Signed-off-by: Zach Lewis <[email protected]>

* The build-directory is no longer hard-coded to a local "build_wheels" path. The 'repairwheel' tool needs to know where it can find any dynamic libs compiled by the build system; and, frustratingly, doesn't seem to consider libraries already bundled in the whl. We can either point repairwheel to directory within an unzipped whl; or, we can point repairwheel back to where the compiled dependencies live inside the build-directory, under .../dist/deps/lib. There isn't a straightforward way of passing information from skbuild to repairwheel directly; but here, we're using cibw to set an environment variable dictating to where scikit-build-core builds, which we can also reference in the repair-wheel step. * Always (re)build the TIFF dependency when building local wheels. This is a workaround for an issue where "Deflate::Deflate" either can't be found, or can't be redeclared, under certain circumstances. A more robust solution might be to instead write a FindLibdeflate.cmake module that adds an alias for Deflate::Deflate as needed. * Add "wheelhouse" directory created by cibuildwheel to .gitignore. Signed-off-by: Zach Lewis <[email protected]>

We don't want to locally build TIFF when it already exists; but if we build static TIFF libs locally once, we have to rebuild every time (i.e., for subsequent builds), or else "Deflate::Deflate" is forgotten. This commit forces TIFF to be rebuilt every time, but only for cibuildwheel unix builds. Under normal circumstances, only missing dependencies will be locally built. Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

Update pyproject.toml configuration The project's TOML file has been updated to reflect changes in the build system, dependencies, and licensing. The scikit-build-core version requirement has been bumped up, and new tools have been added for wheel repair and invocation. The license text has also been simplified to only include Apache-2.0. Signed-off-by: Zach Lewis <[email protected]>

The wheel repair command in the pyproject.toml file has been refactored to use an invoke task. he before-build step now also installs invoke. A new tasks.py file has been added with a 'wheel_repair' task that slims down and repairs the wheel file. Step 1: Remove `lib`, `include`, `share` directories from wheel Step 2: Let `repairwheel` fix the wheel with freshly-built libraries found in {build_dir}/lib and {build_dir}/deps/dist/lib. Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

Also, tiny bit of tidying. Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

It's an OCIO dependency. Signed-off-by: Zach Lewis <[email protected]>

Apparently MSVC is having trouble linking Python3::Python otherwise... Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

- Lowered the required Python version from 3.9 to 3.7 - Added numpy as a new dependency - Refined DLL loading for Windows in Python 3.8+ - Adjusted build verbosity settings - Moved CMAKE_INSTALL_LIBDIR setting into platform-specific overrides for Linux and macOS only - On Windows, do not make adjustments to the INSTALL_RPATH. - Reorganized variables in cibuildwheel configuration for better readability - Refactored variable names in __init__._call_program function to follow PEP8 guidelines Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

Stolen nearly line-for-line from OpenColorIO's wheel workflow. Signed-off-by: Zach Lewis <[email protected]>

…cademySoftwareFoundation#4365) Fixes a simple copy/paste error in a copy constructor where the y coordinate gets initialised twice instead of y and z. Signed-off-by: Anton Dukhovnikov <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…emySoftwareFoundation#4348) Additional stack manipulation commands: * `--popbottom` discards the bottom-of-stack image * `--stackreverse` reverses the order of the whole stack * `--stackclear` fully empties the stack * `--stackextract <index>` moves the indexed item from the stack (index 0 means the top) to the top. Make `--for` work correctly in both directions: * Correct behavior if `--for` has a negative step value. * If the end value is less than the begin value and no step is supplied, assume -1 (analogous to how we usually assueme step=1 under ordinary circumstances). * Error if step is 0 (presume it will make an infinite loop). Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…TORY (AcademySoftwareFoundation#4368) It's not just in oiiotool. This seems clearer and adheres to the env variable naming convention we chose. Reminder: This controls whether command line history gets written to output image metadata by default by oiiotool and maketx. We historically did it, but recently stopped because of security concerns. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

JPEG output configuration hint "jpeg:iptc" (default: 1), if set to 0, will suppress IPTC block output to the file. In the process, we changed the return type of utility function encode_iptc_iim() to return true if anything was successfully encoded, false otherwise. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…ion#4347) Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

*This* seems to be the key we were looking for... Also, it looks like libpng 1.6.44 (released less than a week ago) generates a proper CMake config, so prefer that if it's available.... Signed-off-by: Zach Lewis <[email protected]>

Turns out the macos-12 runners are deprecated. Also, the publish step should only occur when pushing a tag that starts with v2.6 or v3. Signed-off-by: Zach Lewis <[email protected]>

I'm not sure if this actually affects anything, since OCIO is a required dependency of OIIO; but if OCIO doesn't need to be built, then neither does yaml-cpp. Signed-off-by: Zach Lewis <[email protected]>

- Add support for Python 3.13 - Update minimum required Python version from 3.7 to 3.8 (Need to further investigate CIBW troubles with python 3.7) - Upgrade scikit-build-core dependency to >=0.10.6,<1 (up from 0.10) - Add oldest-supported-numpy as a new build dependency. I don't think numpy is actually needed at build time, but this seems to help certain builds not have to compile numpy from source. - Skip 32-bit builds in cibuildwheel configuration - Enable free-threaded support for Python 3.13 - Adjust manylinux and musllinux images versions in cibuildwheel configuration for local testing -- these are currently overridden by environment variables set in the Wheels github workflow. Specifically, use the manylinux2014 (i.e., CentOS-7) image for python 3.7 and 3.8; and use manylinux_2_28 (Rocky) for the rest. Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

it was screwing everything up. Signed-off-by: Zach Lewis <[email protected]>

...to see if this fixes a problem with building on Windows. Signed-off-by: Zach Lewis <[email protected]>

The aarch64 wheels are currently failing , and they take forever to build. Moving the aarch64 wheels builds to the MacOS M1 runners will hopefully fix both problems at once, provided these mac runners have a docker engine installed... Signed-off-by: Zach Lewis <[email protected]>

Create a separate job just for building aarch64 wheels. Another attempt to get the wheels cross-compiled on the apple silicon runners. Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

We have separate PRs in for dealing with PNG and yaml-cpp deps. Removing the recipes from this PR to simplify things, and to check my previous assertion that are needed for the wheels github workflow. Signed-off-by: Zach Lewis <[email protected]>

* Create a dedicated job for linux-arm builds. They take substantially longer than the x86_64 builds, because the aarch64 architecture is being emulated. Ideally, we want to build these wheels on ARM runners, which should be widely available in a few months. * Disable tests for Linux ARM wheels. Apparently, the tests are failing, I think due to trouble CIBW has with testing ARM wheels from x86_64 runners. I've checked the builds myself on my local machine (Rocky and Alma aarch64 docker containers running on a M1 Mac), and they seem totally fine. We'll have to re-enable when we switch to ARM runners. Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: zachlewis <[email protected]>

zachlewis · 2024-09-23T16:27:54Z

An update:
After much trial and error, I've determined two critically important things re: the wheel workflow runs:

The linux x86_64 runners aren't able to perform the wheels tests for the emulated aarch64 builds.
I'm disabling the manylinux_aarch64 tests for now, and added a #TODO about re-enabling the tests when linux-arm runners are available.
(I've tested the build artifacts locally, and they behave as expected).
The macOS ARM runners are having trouble with the system-installed PNG-1.4.
(This is what initially prompted deps: Add build recipe for PNG #4423, which apparently has its own troubles).

For posterity, here are the relevant parts of the logs:

...
    -- Could NOT find PNG (missing: PNG_DIR)
    -- PNG was not found
    -- PNG  is outside the required range 1.6.0...10000.0.0 
    -- PNG library not found 
    --     Try setting PNG_ROOT ?
    --     Maybe this will help:  src/build-scripts/build_libpng.bash

...

 --
    -- =========================================================================
    -- = Dependency report                                                     =
    -- =========================================================================
    --
    -- The following dependencies found externally:
    --     BZip2 1.0.8
    --     Freetype 2.13.3
    --     libjpeg-turbo 3.0.4
    --     pybind11 2.13.6
    --     Python3 3.13.0
    --     ZLIB 1.2.12
    -- The following dependencies were found but were too old:
    --     GIF (found 4, needed >= 5.0)
    --     PNG (found 1.4.12, needed >= 1.6.0)
    -- The following dependencies were not found:
    --     DCMTK
    --     FFmpeg
    --     fmt 7.0...<10.99  (10.2.1 BUILT LOCALLY)
    --     Imath 3.1  (3.1.10 BUILT LOCALLY)
    --     JXL
    --     libdeflate 1.18  (1.20 BUILT LOCALLY)
    --     Libheif
    --     LibRaw
    --     Nuke
    --     OpenColorIO 2.2...<2.9  (2.3.2 BUILT LOCALLY)
    --     OpenCV
    --     OpenEXR 3.1  (3.2.4 BUILT LOCALLY)
    --     OpenJPEG
    --     Ptex
    --     Qt5
    --     Qt6
    --     Robinmap 1.2.0  (1.3.0 BUILT LOCALLY)
    --     TBB
    --     TIFF 4.0  (4.6.0 BUILT LOCALLY)
    --     WebP
    --
    -- =========================================================================
    --
    -- Configuring done (105.4s)
    -- Generating done (0.1s)
    -- Build files have been written to: /tmp/build_oiio_wheels
    *** Building project with Ninja...
    Change Dir: '/tmp/build_oiio_wheels'
  
    Run Build Command(s): /private/var/folders/0g/hj_q_pzx65bbjnslxz9n0src0000gn/T/pip-build-env-fd58mh83/normal/lib/python3.13/site-packages/ninja/data/bin/ninja -v
    
...


    [88/151] /Applications/Xcode_15.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -DEMBED_PLUGINS=1 -DOIIO_FREETYPE_VERSION=\"2.13.3\" -DOIIO_INTERNAL=1 -DOIIO_OPENEXR_CORE_DEFAULT=0 -DOIIO_PYTHON_VERSION=\"3.13.0\" -DOIIO_QT_VERSION=\"\" -DOIIO_TBB_VERSION=\"\" -DOIIO_USE_EXR_C_API=1 -DOpenImageIO_EXPORTS -DUSE_FREETYPE=1 -DUSE_JPEG_TURBO=1 -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS -I/tmp/build_oiio_wheels/include/OpenImageIO -I/tmp/build_oiio_wheels/include -I/tmp/build_oiio_wheels/src/include -I/Users/runner/work/OpenImageIO/OpenImageIO/src/include -I/tmp/build_oiio_wheels/deps/include -I/OpenEXR -isystem /tmp/build_oiio_wheels/deps/dist/include -isystem /tmp/build_oiio_wheels/deps/dist/include/Imath -isystem /tmp/build_oiio_wheels/deps/dist/include/OpenEXR -isystem /Library/Frameworks/Mono.framework/Headers -isystem /opt/homebrew/include -isystem /opt/homebrew/include/freetype2 -Os -DNDEBUG -std=c++17 -arch arm64 -isysroot /Applications/Xcode_15.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.5.sdk -mmacosx-version-min=11.0 -fPIC -fvisibility=hidden -Wall -Werror -Wno-unused-function -Wno-overloaded-virtual -Wno-unneeded-internal-declaration -Wno-unused-private-field -Wno-tautological-compare -Qunused-arguments -Wunknown-warning-option -Wno-unused-local-typedefs -fno-math-errno -MD -MT src/libOpenImageIO/CMakeFiles/OpenImageIO.dir/__/ico.imageio/icoinput.cpp.o -MF src/libOpenImageIO/CMakeFiles/OpenImageIO.dir/__/ico.imageio/icoinput.cpp.o.d -o src/libOpenImageIO/CMakeFiles/OpenImageIO.dir/__/ico.imageio/icoinput.cpp.o -c /Users/runner/work/OpenImageIO/OpenImageIO/src/ico.imageio/icoinput.cpp
    FAILED: src/libOpenImageIO/CMakeFiles/OpenImageIO.dir/__/ico.imageio/icoinput.cpp.o
    /Applications/Xcode_15.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -DEMBED_PLUGINS=1 -DOIIO_FREETYPE_VERSION=\"2.13.3\" -DOIIO_INTERNAL=1 -DOIIO_OPENEXR_CORE_DEFAULT=0 -DOIIO_PYTHON_VERSION=\"3.13.0\" -DOIIO_QT_VERSION=\"\" -DOIIO_TBB_VERSION=\"\" -DOIIO_USE_EXR_C_API=1 -DOpenImageIO_EXPORTS -DUSE_FREETYPE=1 -DUSE_JPEG_TURBO=1 -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS -I/tmp/build_oiio_wheels/include/OpenImageIO -I/tmp/build_oiio_wheels/include -I/tmp/build_oiio_wheels/src/include -I/Users/runner/work/OpenImageIO/OpenImageIO/src/include -I/tmp/build_oiio_wheels/deps/include -I/OpenEXR -isystem /tmp/build_oiio_wheels/deps/dist/include -isystem /tmp/build_oiio_wheels/deps/dist/include/Imath -isystem /tmp/build_oiio_wheels/deps/dist/include/OpenEXR -isystem /Library/Frameworks/Mono.framework/Headers -isystem /opt/homebrew/include -isystem /opt/homebrew/include/freetype2 -Os -DNDEBUG -std=c++17 -arch arm64 -isysroot /Applications/Xcode_15.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.5.sdk -mmacosx-version-min=11.0 -fPIC -fvisibility=hidden -Wall -Werror -Wno-unused-function -Wno-overloaded-virtual -Wno-unneeded-internal-declaration -Wno-unused-private-field -Wno-tautological-compare -Qunused-arguments -Wunknown-warning-option -Wno-unused-local-typedefs -fno-math-errno -MD -MT src/libOpenImageIO/CMakeFiles/OpenImageIO.dir/__/ico.imageio/icoinput.cpp.o -MF src/libOpenImageIO/CMakeFiles/OpenImageIO.dir/__/ico.imageio/icoinput.cpp.o.d -o src/libOpenImageIO/CMakeFiles/OpenImageIO.dir/__/ico.imageio/icoinput.cpp.o -c /Users/runner/work/OpenImageIO/OpenImageIO/src/ico.imageio/icoinput.cpp
    In file included from /Users/runner/work/OpenImageIO/OpenImageIO/src/ico.imageio/icoinput.cpp:9:
    /Users/runner/work/OpenImageIO/OpenImageIO/src/ico.imageio/../png.imageio/png_pvt.h:251:9: error: no matching function for call to 'png_get_iCCP'
            png_get_iCCP(sp, ip, &profile_name, &compression_type, &profile_data,
            ^~~~~~~~~~~~
    /Library/Frameworks/Mono.framework/Headers/png.h:2398:24: note: candidate function not viable: no known conversion from 'png_bytep *' (aka 'unsigned char **') to 'png_charpp' (aka 'char **') for 5th argument
    PNG_EXPORT(png_uint_32,png_get_iCCP) PNGARG((png_const_structp png_ptr,
                           ^
    /Library/Frameworks/Mono.framework/Headers/pngconf.h:1361:58: note: expanded from macro 'PNG_EXPORT'
    #  define PNG_EXPORT(type,symbol) PNG_IMPEXP type PNGAPI symbol
                                                             ^
    In file included from /Users/runner/work/OpenImageIO/OpenImageIO/src/ico.imageio/icoinput.cpp:9:
    /Users/runner/work/OpenImageIO/OpenImageIO/src/ico.imageio/../png.imageio/png_pvt.h:641:13: error: no matching function for call to 'png_set_iCCP'
                png_set_iCCP(sp, ip, "Embedded Profile", 0, icc_profile, length);
                ^~~~~~~~~~~~
    /Library/Frameworks/Mono.framework/Headers/png.h:2405:17: note: candidate function not viable: no known conversion from 'unsigned char *' to 'png_charp' (aka 'char *') for 5th argument
    PNG_EXPORT(void,png_set_iCCP) PNGARG((png_structp png_ptr,
                    ^
    /Library/Frameworks/Mono.framework/Headers/pngconf.h:1361:58: note: expanded from macro 'PNG_EXPORT'
    #  define PNG_EXPORT(type,symbol) PNG_IMPEXP type PNGAPI symbol
                                                             ^
    2 errors generated.
    [89/151] /Applications/Xcode_15.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -DEMBED_PLUGINS=1 -DOIIO_FREETYPE_VERSION=\"2.13.3\" -DOIIO_INTERNAL=1 -DOIIO_OPENEXR_CORE_DEFAULT=0 -DOIIO_PYTHON_VERSION=\"3.13.0\" -DOIIO_QT_VERSION=\"\" -DOIIO_TBB_VERSION=\"\" -DOIIO_USE_EXR_C_API=1 -DOpenImageIO_EXPORTS -DUSE_FREETYPE=1 -DUSE_JPEG_TURBO=1 -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS -I/tmp/build_oiio_wheels/include/OpenImageIO -I/tmp/build_oiio_wheels/include -I/tmp/build_oiio_wheels/src/include -I/Users/runner/work/OpenImageIO/OpenImageIO/src/include -I/tmp/build_oiio_wheels/deps/include -I/OpenEXR -isystem /tmp/build_oiio_wheels/deps/dist/include -isystem /tmp/build_oiio_wheels/deps/dist/include/Imath -isystem /tmp/build_oiio_wheels/deps/dist/include/OpenEXR -isystem /Library/Frameworks/Mono.framework/Headers -isystem /opt/homebrew/include -isystem /opt/homebrew/include/freetype2 -Os -DNDEBUG -std=c++17 -arch arm64 -isysroot /Applications/Xcode_15.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.5.sdk -mmacosx-version-min=11.0 -fPIC -fvisibility=hidden -Wall -Werror -Wno-unused-function -Wno-overloaded-virtual -Wno-unneeded-internal-declaration -Wno-unused-private-field -Wno-tautological-compare -Qunused-arguments -Wunknown-warning-option -Wno-unused-local-typedefs -fno-math-errno -MD -MT src/libOpenImageIO/CMakeFiles/OpenImageIO.dir/__/ico.imageio/icooutput.cpp.o -MF src/libOpenImageIO/CMakeFiles/OpenImageIO.dir/__/ico.imageio/icooutput.cpp.o.d -o src/libOpenImageIO/CMakeFiles/OpenImageIO.dir/__/ico.imageio/icooutput.cpp.o -c /Users/runner/work/OpenImageIO/OpenImageIO/src/ico.imageio/icooutput.cpp
    FAILED: src/libOpenImageIO/CMakeFiles/OpenImageIO.dir/__/ico.imageio/icooutput.cpp.o
    /Applications/Xcode_15.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -DEMBED_PLUGINS=1 -DOIIO_FREETYPE_VERSION=\"2.13.3\" -DOIIO_INTERNAL=1 -DOIIO_OPENEXR_CORE_DEFAULT=0 -DOIIO_PYTHON_VERSION=\"3.13.0\" -DOIIO_QT_VERSION=\"\" -DOIIO_TBB_VERSION=\"\" -DOIIO_USE_EXR_C_API=1 -DOpenImageIO_EXPORTS -DUSE_FREETYPE=1 -DUSE_JPEG_TURBO=1 -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS -I/tmp/build_oiio_wheels/include/OpenImageIO -I/tmp/build_oiio_wheels/include -I/tmp/build_oiio_wheels/src/include -I/Users/runner/work/OpenImageIO/OpenImageIO/src/include -I/tmp/build_oiio_wheels/deps/include -I/OpenEXR -isystem /tmp/build_oiio_wheels/deps/dist/include -isystem /tmp/build_oiio_wheels/deps/dist/include/Imath -isystem /tmp/build_oiio_wheels/deps/dist/include/OpenEXR -isystem /Library/Frameworks/Mono.framework/Headers -isystem /opt/homebrew/include -isystem /opt/homebrew/include/freetype2 -Os -DNDEBUG -std=c++17 -arch arm64 -isysroot /Applications/Xcode_15.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.5.sdk -mmacosx-version-min=11.0 -fPIC -fvisibility=hidden -Wall -Werror -Wno-unused-function -Wno-overloaded-virtual -Wno-unneeded-internal-declaration -Wno-unused-private-field -Wno-tautological-compare -Qunused-arguments -Wunknown-warning-option -Wno-unused-local-typedefs -fno-math-errno -MD -MT src/libOpenImageIO/CMakeFiles/OpenImageIO.dir/__/ico.imageio/icooutput.cpp.o -MF src/libOpenImageIO/CMakeFiles/OpenImageIO.dir/__/ico.imageio/icooutput.cpp.o.d -o src/libOpenImageIO/CMakeFiles/OpenImageIO.dir/__/ico.imageio/icooutput.cpp.o -c /Users/runner/work/OpenImageIO/OpenImageIO/src/ico.imageio/icooutput.cpp
    In file included from /Users/runner/work/OpenImageIO/OpenImageIO/src/ico.imageio/icooutput.cpp:10:
    /Users/runner/work/OpenImageIO/OpenImageIO/src/ico.imageio/../png.imageio/png_pvt.h:251:9: error: no matching function for call to 'png_get_iCCP'
            png_get_iCCP(sp, ip, &profile_name, &compression_type, &profile_data,
            ^~~~~~~~~~~~
    /Library/Frameworks/Mono.framework/Headers/png.h:2398:24: note: candidate function not viable: no known conversion from 'png_bytep *' (aka 'unsigned char **') to 'png_charpp' (aka 'char **') for 5th argument
    PNG_EXPORT(png_uint_32,png_get_iCCP) PNGARG((png_const_structp png_ptr,
                           ^
    /Library/Frameworks/Mono.framework/Headers/pngconf.h:1361:58: note: expanded from macro 'PNG_EXPORT'
    #  define PNG_EXPORT(type,symbol) PNG_IMPEXP type PNGAPI symbol
                                                             ^
    In file included from /Users/runner/work/OpenImageIO/OpenImageIO/src/ico.imageio/icooutput.cpp:10:
    /Users/runner/work/OpenImageIO/OpenImageIO/src/ico.imageio/../png.imageio/png_pvt.h:641:13: error: no matching function for call to 'png_set_iCCP'
                png_set_iCCP(sp, ip, "Embedded Profile", 0, icc_profile, length);
                ^~~~~~~~~~~~
    /Library/Frameworks/Mono.framework/Headers/png.h:2405:17: note: candidate function not viable: no known conversion from 'unsigned char *' to 'png_charp' (aka 'char *') for 5th argument
    PNG_EXPORT(void,png_set_iCCP) PNGARG((png_structp png_ptr,
                    ^
    /Library/Frameworks/Mono.framework/Headers/pngconf.h:1361:58: note: expanded from macro 'PNG_EXPORT'
    #  define PNG_EXPORT(type,symbol) PNG_IMPEXP type PNGAPI symbol
                                                             ^
    2 errors generated.

As you can see, the build correctly finds and identifies the system PNG as "too old", but seems to continue trying to build with /Library/Frameworks/Mono.framework/Headers/png.h.

Two things worth noting:

This isn't causing problems during the CI / macos-14 runs because a dynamic libpng-1.6 is provided by the homebrew steps
The build_PNG.cmake from deps: Add build recipe for PNG #4423 does seem to fix this problem; but I've temporarily removed the build recipe from this PR so I could better identify and isolate the various problems causing check failures.

(I've also removed the yaml-cpp build recipe from this PR, because it's apparently not as required for building Windows wheels as I'd previously thought.)

There are some CI workflow failures that I can't make heads nor tails of -- they don't seem related to anything this PR is affecting. It's possible the failures reflect something I screwed up when merging the main branch into this branch. This PR has gotten kind of hairy, and it might be worth my time to create a new, cleaner PR, with all changes rebased on top of the current main branch.

Signed-off-by: Zach Lewis <[email protected]>

zachlewis and others added 30 commits September 16, 2024 10:55

int(python): Adjust module metadata

5e86b2d

Use Apache-2.0 license identifier instead of BSD-3-Clause. Add "OpenImageIO Contributors" / "[email protected]" as author. I could not bring myself to remove Larry as an author and maintainer. Signed-off-by: Zach Lewis <[email protected]>

build(deps): Add libjpeg-turbo recipe

5c1a1b8

Build and link missing libjpeg-turbo shared + static libs Signed-off-by: Zach Lewis <[email protected]>

docs: clarify that IBA::rotate params are pixel coordinates (AcademyS…

bd271a3

…oftwareFoundation#4358) As suggested by Moritz Moeller Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

cleanup: fix comment, whitespace

50f0b4e

Signed-off-by: Zach Lewis <[email protected]>

build: add rye artifacts to gitignore

1997724

Signed-off-by: Zach Lewis <[email protected]>

build: simplify wheel_edit task

80acc8c

Signed-off-by: Zach Lewis <[email protected]>

cleanup: add copyright and license information to tasks.py

2ee72cd

Also, tiny bit of tidying. Signed-off-by: Zach Lewis <[email protected]>

build(deps): add zlib build recipe

0f5e1a3

Signed-off-by: Zach Lewis <[email protected]>

build(deps): add yaml-cpp build recipe

13f60ce

It's an OCIO dependency. Signed-off-by: Zach Lewis <[email protected]>

build(python): On windows, include Development.Embed Python3 component

b96525e

Apparently MSVC is having trouble linking Python3::Python otherwise... Signed-off-by: Zach Lewis <[email protected]>

build(python): Increase verbosity of scikit-build-core build

42caf75

Signed-off-by: Zach Lewis <[email protected]>

cleanup(python): formatting for PEP compliance

151333f

Signed-off-by: Zach Lewis <[email protected]>

ci: add wheel workflow

0810544

Stolen nearly line-for-line from OpenColorIO's wheel workflow. Signed-off-by: Zach Lewis <[email protected]>

fix(png): round dpi resolution to nearest 0.1 (AcademySoftwareFoundat…

6eddaf5

…ion#4347) Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

zachlewis added 5 commits September 16, 2024 14:37

deps(PNG): adjust libpng library install dir

0160971

Signed-off-by: Zach Lewis <[email protected]>

deps(PNG): disable Framework build for libpng

35210f1

*This* seems to be the key we were looking for... Also, it looks like libpng 1.6.44 (released less than a week ago) generates a proper CMake config, so prefer that if it's available.... Signed-off-by: Zach Lewis <[email protected]>

ci(wheels): Use newer macos x86_64 runners, and refine publish condition

889e46d

Turns out the macos-12 runners are deprecated. Also, the publish step should only occur when pushing a tag that starts with v2.6 or v3. Signed-off-by: Zach Lewis <[email protected]>

deps(yaml-cpp): specify that yaml-cpp is a dependency of OCIO

f5e118d

I'm not sure if this actually affects anything, since OCIO is a required dependency of OIIO; but if OCIO doesn't need to be built, then neither does yaml-cpp. Signed-off-by: Zach Lewis <[email protected]>

zachlewis force-pushed the python_cibuildwheels branch from 28c5cda to 16d634e Compare September 17, 2024 17:26

zachlewis added 3 commits September 17, 2024 13:28

fix(wheels): disable free-threaded support for now

ce5db05

Signed-off-by: Zach Lewis <[email protected]>

ci(wheel): revert CIBW environment override

ab7e348

it was screwing everything up. Signed-off-by: Zach Lewis <[email protected]>

deps(PNG): Always disable shared libs

ffa522e

...to see if this fixes a problem with building on Windows. Signed-off-by: Zach Lewis <[email protected]>

zachlewis force-pushed the python_cibuildwheels branch 3 times, most recently from 74ca1b7 to 334ac4d Compare September 18, 2024 20:09

zachlewis force-pushed the python_cibuildwheels branch from 334ac4d to 1258db1 Compare September 18, 2024 20:11

ci(wheels): refine process for building Linux aarch64 wheels

5dc9ceb

Create a separate job just for building aarch64 wheels. Another attempt to get the wheels cross-compiled on the apple silicon runners. Signed-off-by: Zach Lewis <[email protected]>

zachlewis force-pushed the python_cibuildwheels branch from 60a2108 to 5dc9ceb Compare September 18, 2024 20:23

ci(wheels): Install Docker to macos-14 runner for cross compile

2a5e713

Signed-off-by: Zach Lewis <[email protected]>

zachlewis force-pushed the python_cibuildwheels branch 2 times, most recently from a04c0c3 to 0a811d1 Compare September 18, 2024 20:40

ci(wheels): Use podman instead of Docker for MacOS ARM64 Linux cross

59787c2

Signed-off-by: Zach Lewis <[email protected]>

zachlewis force-pushed the python_cibuildwheels branch from 0a811d1 to 59787c2 Compare September 18, 2024 20:48

ci(wheels): Revert to using a linux runner for building aarch64 wheels

72402d2

Signed-off-by: Zach Lewis <[email protected]>

zachlewis force-pushed the python_cibuildwheels branch from 7ecfa3b to 72402d2 Compare September 19, 2024 00:21

zachlewis and others added 4 commits September 19, 2024 17:18

Merge branch 'main' into python_cibuildwheels

c313ad3

deps: Remove yaml-cpp and PNG recipes

36a4d29

We have separate PRs in for dealing with PNG and yaml-cpp deps. Removing the recipes from this PR to simplify things, and to check my previous assertion that are needed for the wheels github workflow. Signed-off-by: Zach Lewis <[email protected]>

Merge branch 'main' into python_cibuildwheels

18d192f

Signed-off-by: zachlewis <[email protected]>

zachlewis force-pushed the python_cibuildwheels branch from 0879030 to 18d192f Compare September 23, 2024 20:15

cleanup: reconcile minor differences with main branch

fb677f4

Signed-off-by: Zach Lewis <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Python wheels workflow and build backend #4428

feat: Python wheels workflow and build backend #4428

zachlewis commented Sep 16, 2024

zachlewis commented Sep 23, 2024

feat: Python wheels workflow and build backend #4428

Are you sure you want to change the base?

feat: Python wheels workflow and build backend #4428

Conversation

zachlewis commented Sep 16, 2024

Description

Summary

PEP-Compliant Packaging: pyproject.toml

Additions to __ init __.py

Relocatable Binary Distributions with cibuildwheel + repairwheel

"Wheels" Github workflow

Additional Dependency Recipes

Other Changes

Tests

zachlewis commented Sep 23, 2024

PEP-Compliant Packaging: `pyproject.toml`

Additions to ` init .py`

Relocatable Binary Distributions with `cibuildwheel` + `repairwheel`