Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add multilingual translation serving #395

Merged
merged 1 commit into from
Aug 22, 2024
Merged

Add multilingual translation serving #395

merged 1 commit into from
Aug 22, 2024

Conversation

jonatanklosko
Copy link
Member

Adds a translation serving. It builds on top of text generation serving, but the serving input expects a source and target language to be specified alongside the text. The language is configured as part of input, so that the same serving can be used for different language pairs.

repository_id = "facebook/nllb-200-distilled-600M"

{:ok, model_info} = Bumblebee.load_model({:hf, repository_id})
{:ok, tokenizer} = Bumblebee.load_tokenizer({:hf, repository_id})
{:ok, generation_config} = Bumblebee.load_generation_config({:hf, repository_id})

serving = Bumblebee.Text.translation(model_info, tokenizer, generation_config)

text = "The bank of the river is beautiful in spring"

Nx.Serving.run(serving, %{
  text: text,
  source_language_token: "eng_Latn",
  target_language_token: "pol_Latn"
})
#=> %{
#=>   results: [
#=>     %{
#=>       text: "W wiosnę brzeg rzeki jest piękny",
#=>       token_summary: %{input: 11, output: 13, padding: 0}
#=>     }
#=>   ]
#=> }

@jonatanklosko jonatanklosko merged commit 9421eca into main Aug 22, 2024
2 checks passed
@jonatanklosko jonatanklosko deleted the jk-translation branch August 22, 2024 10:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant