Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specify coercion of non-wasm types #62

Open
geekbeast opened this issue Nov 1, 2023 · 1 comment
Open

Specify coercion of non-wasm types #62

geekbeast opened this issue Nov 1, 2023 · 1 comment

Comments

@geekbeast
Copy link
Contributor

geekbeast commented Nov 1, 2023

The current tensor types are not all supported in core wasm.

    enum tensor-type {
        FP16,
        FP32,
        FP64,
        BF16,
        U8,
        I32,
        I64
    }

WASM supports the following core types:

i32 : 32-bit integer.
i64 : 64-bit integer.
f32 : 32-bit float.
f64 : 64-bit float.

In addition, there is some utility types such as u8, u16, u32, i32, etc but these do not exist for floating point.

The complication is that models can take non-standard wasm types and output non-started wasm types (packed into bytes). Even if tensors are resources, at some point they must be created and at some point they must be interpreted for use, usually in the WASM guest.

One possible behavior is that we take everything as one of the core wasm types and then coerce on the host side to the correct tensor type. Another mutually exclusive alternative is that we explicitly say we are not doing any coercion, so there must be some ability to emulate the type and it must be implicitly castable in-place by the bindings.

The latter would seem like less work for implementers and the former is likely to be more usable by consumers of the API.

We can also of course do nothing and hope that Google adds support for fp16 into core wasm, but there's a lot of other exotic floating point types like fp8 and fp4 that are becoming relevant to generative ai (see https://arxiv.org/abs/2310.16836v1). This may be a dead in the hype cycle, but I suspect the techniques will be forever relevant for larger model sizes.

@geekbeast
Copy link
Contributor Author

@squillace Thoughts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant