[mobile] Move to net8 MAUI and add MacCatalyst support #406

rmarinho · 2024-03-28T00:30:25Z

This pull request primarily aims to update the MauiVisionSample project in the mobile/examples/Maui directory to support the MacCatalyst platform, along with some other improvements and updates. The most significant changes include updating the target frameworks in the project file, adding MacCatalyst-specific files, updating package references, and adding debug logging.

This pull request primarily focuses on updating the MauiVisionSample application to support the MacCatalyst platform and updating the target frameworks to .NET 8.0. It also includes changes to improve debugging, and package updates.

The most important changes are:

This pull request includes several changes to enhance compatibility with .NET 8.0, add support for MacCatalyst, and update dependencies in the MauiVisionSample and MauiSuperResolution projects.

Compatibility and Platform Support:

MainPage.xaml.cs: Added support for MacCatalyst in the execution provider options. (mobile/examples/Maui/MauiVisionSample/MauiVisionSample/MainPage.xaml.cs)
MauiVisionSample.csproj: Updated target frameworks to .NET 8.0 and added support for MacCatalyst. (mobile/examples/Maui/MauiVisionSample/MauiVisionSample/MauiVisionSample.csproj) [1] [2] [3]
MauiSuperResolution.csproj: Updated target frameworks to .NET 8.0 and added support for MacCatalyst. (mobile/examples/super_resolution/MAUI/MauiSuperResolution/MauiSuperResolution.csproj) [1] [2]

Dependency Updates:

MauiVisionSample.csproj: Updated various package references, including Microsoft.ML.OnnxRuntime, SkiaSharp, and added new references for Microsoft.Maui.Controls and Microsoft.Extensions.Logging.Debug. (mobile/examples/Maui/MauiVisionSample/MauiVisionSample/MauiVisionSample.csproj)
MauiSuperResolution.csproj: Updated package references for Microsoft.ML.OnnxRuntime and its extensions. (mobile/examples/super_resolution/MAUI/MauiSuperResolution/MauiSuperResolution.csproj)

Logging and Debugging:

MauiProgram.cs: Added conditional logging for the debug configuration. (mobile/examples/Maui/MauiVisionSample/MauiVisionSample/MauiProgram.cs)

MacCatalyst Specific Files:

MacCatalyst Platform Files: Added necessary files for MacCatalyst support, including AppDelegate.cs, Entitlements.plist, Info.plist, and Program.cs for both projects.
- MauiVisionSample: (mobile/examples/Maui/MauiVisionSample/MauiVisionSample/Platforms/MacCatalyst/AppDelegate.cs) [1] (mobile/examples/Maui/MauiVisionSample/MauiVisionSample/Platforms/MacCatalyst/Entitlements.plist) [2] (mobile/examples/Maui/MauiVisionSample/MauiVisionSample/Platforms/MacCatalyst/Info.plist) [3] (mobile/examples/Maui/MauiVisionSample/MauiVisionSample/Platforms/MacCatalyst/Program.cs) [4]
- MauiSuperResolution: (mobile/examples/super_resolution/MAUI/MauiSuperResolution/Platforms/MacCatalyst/AppDelegate.cs) [1] (mobile/examples/super_resolution/MAUI/MauiSuperResolution/Platforms/MacCatalyst/Entitlements.plist) [2] (mobile/examples/super_resolution/MAUI/MauiSuperResolution/Platforms/MacCatalyst/Info.plist) [3] (mobile/examples/super_resolution/MAUI/MauiSuperResolution/Platforms/MacCatalyst/Program.cs) [4]

Solution Files:

Solution Files: Added solution files for both projects to support Visual Studio.
- MauiVisionSample: (mobile/examples/Maui/MauiVisionSample/MauiVisionSample/MauiVisionSample.sln)
- MauiSuperResolution: (mobile/examples/super_resolution/MAUI/MauiSuperResolution/MauiSuperResolution.sln)

rmarinho · 2024-03-28T00:33:46Z

Right now we are hitting this on Catalyst

/usr/local/share/dotnet/packs/Microsoft.MacCatalyst.Sdk/17.2.8043/targets/Xamarin.Shared.Sdk.targets(3,3): Error: No matching framework found inside '/Users/ruimarinho/.nuget/packages/microsoft.ml.onnxruntime.extensions.dummy/0.10.0/runtimes/ios/native/onnxruntime_extensions.xcframework'. SupportedPlatform: '/Users/ruimarinho/.nuget/packages/microsoft.ml.onnxruntime.extensions.dummy/0.10.0/runtimes/ios/native/onnxruntime_extensions.xcframework', SupportedPlatformVariant: 'ios', SupportedArchitectures: 'maccatalyst'. (MauiVisionSample)

I have talked with @YUNQIUGUO and we identified the issue , we need to update the Microsoft.ML.Onnxruntime.Extensions.Dummy package to support MacCatalyst too for x86 and arm

YUNQIUGUO · 2024-04-10T17:03:16Z

Update: We identified the issue that for this to work, the onnxruntime_extensions.xcframework requires catalyst build support to be added as well. And with that supported, @rmarinho helped verify with a Nuget ORT package contains catalyst support and Nuget ORT Extension package with catalyst support (manually replaced and tested for now), this .NET MAUI app works in MacCatalyst platform.

So next step is when the official/dev version dummy extensions nuget package is released, we can update the package dependency here in this project and should be good to go.

rmarinho · 2024-04-10T18:22:57Z

Yap sounds good . will wait for dev package

rmarinho · 2024-06-11T23:43:05Z

Hello, I just tested the sample with the latest version of the onxx runtime and seems to be working fine, can we merge this?

Thanks

rmarinho · 2024-07-12T14:02:58Z

@YUNQIUGUO do you think this can be merged?

lutzapps · 2024-08-08T22:45:51Z

@rmarinho & @YUNQIUGUO
I think there is still a problem with the "net8.0-maccatalyst" (x64) target support for the "Microsoft.ML.OnnxRuntime.Extensions", and I mean with the full Runtime Extensions and not the "Microsoft.ML.OnnxRuntime.Extensions.Dummy" nugget (which has no real implementation, and is only needed to fullfill iOS build requirement).

With the latest prerelease version "0.11.0-dev-20240524-2333-8d8670f7" of "Microsoft.ML.OnnxRuntime.Extensions", I was able to run the "MAUISuperResolution" perfectly fine for the Android target and also all fine in the iOS-Simulator. But when running the same code with MacCatalyst x64 target, I get an runtime error at inference time:

[ErrorCode:Fail] Fatal error: com.microsoft.extensions:DecodeImage(-1) is not a registered function/op

In contrast to the "MauiVisionSample" which does NOT use a pre-/postprocessing pipeline in the ONNX model, the "MauiSuperResolution" sample DOES USE and need the Extensions support for custom pre-/post processing like DecodeImage() and EncodeImage() Vision Operators:

const std::vector<const OrtCustomOp*>& VisionLoader() {
  static OrtOpLoader op_loader(CustomCpuStruct("EncodeImage", ort_extensions::KernelEncodeImage),
                               CustomCpuStruct("DecodeImage", ort_extensions::KernelDecodeImage),
                               CustomCpuStruct("DrawBoundingBoxes", ort_extensions::DrawBoundingBoxes));
  return op_loader.GetCustomOps();
}

FxLoadCustomOpFactory LoadCustomOpClasses_Vision = VisionLoader;

But even as the code in "MauiSuperResolution" registers for the Runtime Extensions support with following code in "OrtInferenceSession.cs":

            // enable pre/post processing custom operators from onnxruntime-extensions
L37:       _sessionOptions.RegisterOrtExtensions();

that ORT extensions are only found at runtime for Android and iOS, but NOT for MacCatalyst-x64.

I checked the "bin/debug/net8-maccatalyst/maccatalyst-x64" directory, and there is a "onyxruntime_extensions" Unix Executable File, but somehow that is not found at runtime or it is the wrong version for maccatalyst-x64.

So I think it is not sufficient to test ORT Extensions support or .NET8 support with the "MauiVisionSample", which only needs the Dummy Extensions. The test should be done with "MauiSuperResolution", which DOES use (and need) the ONNX Runtime Extensions. The latest "0.11.0-dev-20240524-2333-8d8670f7" nugget package of "Microsoft.ML.OnnxRuntime.Extensions" now allows to compile without error against the "net8.0-maccatalyst" target, but it can not find the standard ONNX Runtime Extensions functions/ops at runtime at inference time.

Verified with:
macOS Sonoma 14.6 on MacBookPro (Intel x64)
OS: Darwin x64 23.6.0
Xcode-15.4.0
VSCode Version: 1.92.1 (Universal)
.NET SDK 8.0.303
.NET MAUI workload:: Installation Source: SDK 8.0.300, Manifest Version: 8.0.61/8.0.100

MauiSuperResolution.csproj::

<TargetFrameworks>net8.0-android;net8.0-ios;net8.0-maccatalyst</TargetFrameworks>

<SupportedOSPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'ios'">14.2</SupportedOSPlatformVersion>
<SupportedOSPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'maccatalyst'">14.0</SupportedOSPlatformVersion>
<SupportedOSPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'android'">21.0</SupportedOSPlatformVersion>

<PackageReference Include="Microsoft.Extensions.Logging.Debug" Version="8.0.0" />
<PackageReference Include="Microsoft.Maui.Controls" Version="$(MauiVersion)" />
<PackageReference Include="Microsoft.Maui.Controls.Compatibility" Version="$(MauiVersion)" />
<PackageReference Include="Microsoft.ML.OnnxRuntime" Version="1.18.1" />
<PackageReference Include="Microsoft.ML.OnnxRuntime.Extensions" Version="0.11.0-dev-20240524-2333-8d8670f7" />
<PackageReference Include="Microsoft.ML.OnnxRuntime.Managed" Version="1.18.1" />

skottmckay · 2024-09-06T04:56:07Z

The latest release of the extensions package should now have a mac-catalyst build.

https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime.Extensions/0.12.0

mobile/examples/Maui/MauiVisionSample/MauiVisionSample/MauiVisionSample.csproj

rmarinho · 2024-09-08T01:00:21Z

yap I updated all packages and I also hit that error on MacCatalyst on MBP M3 using the MAUISuperResolution" wi

I do think we can merge this pr as it fixes and updates most for net8

 Documents dotnet --info
.NET SDK:
 Version:           9.0.100-rc.1.24452.12
 Commit:            81a714c6d3
 Workload version:  9.0.100-rc.1.24453.3
 MSBuild version:   17.12.0-preview-24422-09+d17ec720d

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  14.6
 OS Platform: Darwin
 RID:         osx-arm64
 Base Path:   /usr/local/share/dotnet/sdk/9.0.100-rc.1.24452.12/

.NET workloads installed:
Configured to use workload sets when installing new manifests.
 [maui]
   Installation Source: SDK 9.0.100-rc.1
   Manifest Version:    9.0.0-rc.1.24453.9/9.0.100-rc.1
   Manifest Path:       /usr/local/share/dotnet/sdk-manifests/9.0.100-rc.1/microsoft.net.sdk.maui/9.0.0-rc.1.24453.9/WorkloadManifest.json
   Install Type:        FileBased


Host:
  Version:      9.0.0-rc.1.24431.7
  Architecture: arm64
  Commit:       static

.NET SDKs installed:
  7.0.317 [/usr/local/share/dotnet/sdk]
  8.0.401 [/usr/local/share/dotnet/sdk]
  9.0.100-rc.1.24452.12 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 7.0.20 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.8 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 9.0.0-rc.1.24452.1 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 7.0.20 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.8 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.0-rc.1.24431.7 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

lutzapps · 2024-09-08T01:02:15Z

@skottmckay and @rmarinho
The problem I have is NOT that I not get a valid build for Mac Catalyst, but that an App which actually really then USES the ORT-Extensions errors out at runtime, because of something is still missing.

Now with version 0.12 the error changed to "One or more errors occurred. (The type initializer for 'Microsoft.ML.OnnxRuntime.NativeMethods' threw an exception.)".

With v 0.11-dev it was another error I send here before/above. So with your example of the "Dummy" package it may be possible now to built for MacCatalyst, but when you actually use ORT-Extension v0.12 it does NOT work on MacCatalyst.

Update onxx runtime Update maui super resolution Add default sln Add Catalyst platform to super resolution Update versions

skottmckay · 2024-09-09T08:16:42Z

Unfortunately the build config for mac-catalyst is excluding the image decoding/encoding operators due to an issue with opencv. This PR should hopefully address that.

rmarinho · 2024-09-23T16:57:08Z

Any else I need to do to get this merged? Thanks

lutzapps · 2024-09-23T21:45:49Z

@skottmckay so basically your mentioned PR only is a different wording for a clearer error message like "NSImage is not supported on Mac Catalyst", but the image decoding/encoding operators will NOT be included for Mac Catalyst? Is there no hope that this issue with opencv can be fixed for Mac Catalyst?

rmarinho force-pushed the update-maui-sample branch 2 times, most recently from dc1c61c to da455a2 Compare June 11, 2024 23:38

rmarinho marked this pull request as ready for review June 11, 2024 23:42

skottmckay reviewed Sep 6, 2024

View reviewed changes

mobile/examples/Maui/MauiVisionSample/MauiVisionSample/MauiVisionSample.csproj Outdated Show resolved Hide resolved

rmarinho force-pushed the update-maui-sample branch from 5308275 to 74cd88f Compare September 8, 2024 00:20

[mobile]Move to net8 MAUI and add MacCatalyst support

d49ffd7

Update onxx runtime Update maui super resolution Add default sln Add Catalyst platform to super resolution Update versions

rmarinho force-pushed the update-maui-sample branch from 87a2bbd to d49ffd7 Compare September 8, 2024 01:09

rmarinho requested a review from skottmckay September 8, 2024 01:10

Remove extra line

e302142

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[mobile] Move to net8 MAUI and add MacCatalyst support #406

[mobile] Move to net8 MAUI and add MacCatalyst support #406

rmarinho commented Mar 28, 2024 •

edited

Loading

rmarinho commented Mar 28, 2024

YUNQIUGUO commented Apr 10, 2024 •

edited

Loading

rmarinho commented Apr 10, 2024

rmarinho commented Jun 11, 2024

rmarinho commented Jul 12, 2024

lutzapps commented Aug 8, 2024 •

edited

Loading

skottmckay commented Sep 6, 2024

rmarinho commented Sep 8, 2024

lutzapps commented Sep 8, 2024 •

edited

Loading

skottmckay commented Sep 9, 2024

rmarinho commented Sep 23, 2024

lutzapps commented Sep 23, 2024

[mobile] Move to net8 MAUI and add MacCatalyst support #406

Are you sure you want to change the base?

[mobile] Move to net8 MAUI and add MacCatalyst support #406

Conversation

rmarinho commented Mar 28, 2024 • edited Loading

Compatibility and Platform Support:

Dependency Updates:

Logging and Debugging:

MacCatalyst Specific Files:

Solution Files:

rmarinho commented Mar 28, 2024

YUNQIUGUO commented Apr 10, 2024 • edited Loading

rmarinho commented Apr 10, 2024

rmarinho commented Jun 11, 2024

rmarinho commented Jul 12, 2024

lutzapps commented Aug 8, 2024 • edited Loading

skottmckay commented Sep 6, 2024

rmarinho commented Sep 8, 2024

lutzapps commented Sep 8, 2024 • edited Loading

skottmckay commented Sep 9, 2024

rmarinho commented Sep 23, 2024

lutzapps commented Sep 23, 2024

rmarinho commented Mar 28, 2024 •

edited

Loading

YUNQIUGUO commented Apr 10, 2024 •

edited

Loading

lutzapps commented Aug 8, 2024 •

edited

Loading

lutzapps commented Sep 8, 2024 •

edited

Loading