-
Notifications
You must be signed in to change notification settings - Fork 455
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ONNX export] Musicgen for text-to-audio #1297
Comments
Hi, |
It would be great if this feature is done. |
hi @xenova, May i know if have a plan or schedule to support Optimum for convert it to ONNX model? |
any update? |
Hi @kanger45 @MaiZhiHao @zeke-john #1779 is merged, which exports Musicgen in several parts to generate audio samples conditioned on a text prompt (Reference: https://huggingface.co/docs/transformers/model_doc/musicgen#text-conditional-generation). This uses the decoder KV cache. The following subcomponents are exported:
This is usable e.g. in transformers.js, there is no implementation in Optimum for the runtime for now. |
@zeke-john yes, it should work as long as the checkpoint (& model repo) follows Transformers style (e.g. https://huggingface.co/facebook/musicgen-small/tree/main). .bin & .safetensors are supported, not sure about .pt |
Are there any supported ways to finetune musicgen besides the way i did it, so it stays a transformers model? Or can you convert a .pt model into a transformers model format? |
@zeke-john You should try to use https://github.com/huggingface/transformers/blob/main/src/transformers/models/musicgen/convert_musicgen_transformers.py which should allow you to do the conversion (audiocraft format to transformers format). |
@fxmarty after we export several onnx model, how can we run these onnx model locally? |
@fxmarty Would it be possible to add support for the stereo model? It seems to error out with finding the right index of the number of heads (I think). Since the codebook of the stereo model is larger. It could have something to do with the way the config file is read and the model is loaded, not fully sure.
|
Feature request
Musicgen was recently added to 🤗 Transformers (model doc) and it would be great to be able to export those models to ONNX with Optimum.
Motivation
This will allow me to support music generation models in Transformers.js
Your contribution
I will integrate into transformers.js once available in optimum.
The text was updated successfully, but these errors were encountered: