Support for custom types (complex objects, JSI HostObjects)

[mrousavy]

Hi all!
I'm opening this discussion to talk about supporting custom types in TurboModules.

Context

TurboModules (and CodeGen) allow you to write native functions that will be exposed to JavaScript. Those native functions can accept parameters and return values, which can be of types Double (number), Boolean (boolean), String (string), or an Array ([]) of said types, and even a nested Map ({}) of said types.

Using this architecture, one might be able to create an Image editor library where the user can pass in an Image, do modifications on the Image, and get the new Image as a return value. This might look a little like this:

// imageUrl is string to Image file
const imageUrl = await openImagePicker()

await makeImageGreyscale(imageUrl)
await blurImage(imageUrl, 0.8)
await cropImage(imageUrl, 2000, 1000)

As you might notice, the native functions receive the imageUrl as a parameter, which means they need to read the Image from disk, do the modification (e.g. greyscale, blur or crop), and then write it back to disk.

With the power of JSI we could also create HostObjects, aka native objects that are exposed to JS. In this case, the Image type (UIImage or Bitmap) could be exposed to JS and passed around directly:

// image is `Image` type (`ImageHostObject` with `UIImage` on iOS, `Bitmap` on Android)
const image = await openImagePicker()

await makeImageGreyscale(image)
await blurImage(image, 0.8)
await cropImage(image, 2000, 1000)

This greatly improves performance since we skip 6x file I/O.

Additionally, simple things like:

const width = getImageWidth(imageUrl)

Could be replaced by:

const width = image.width

Which is easier to read and understand since we follow the object-oriented paradigm (instead of C-style programming).

Implementation

I've worked on a "proof of concept" implementation of this Image type here: https://github.com/mrousavy/react-native-jsi-image

To integrate this in the TurboModule and CodeGen architecture a few changes are required, none of which I consider breaking.

[mrousavy]

cc @kelset @raedle

[kelset]

cc @michbil

[andrestone]

I think the idea is great. As I understand you mentioned Image as an example but this would work for other media types as well.

So what would the implementation look like? Is it simply adding a TypedArrayKind type and let library authors handle higher level abstractions?

[intergalacticspacehighway]

would be awesome if HostObject can be passed as a prop in NativeComponent. e.g. an <Image /> component can accept an Image HostObject.

[mrousavy]

Yep, that's the idea! <Image /> then doesn't need to read from disk, it can directly use the already existing in-memory Image object!

[cipolleschi]

Hi @mrousavy, thanks for this proposal. I think it is very interesting. Do you mind creating a PR on the react-native repo with the changes to JSI, Codegen and TurboModule so that we can evaluate them?

[kelset]

cc @ZihanChen-MSFT