Support VLM in chat completion (+some specs updates)

[Wauplin]

Follow-up PR after huggingface/huggingface.js#915.

Updates chat completion input parameters (tools and stream_options) + add supports for VLMs => sending image as text message.

Also adds TextToSpeech spec definition but not used at the moment (same specs as TextToAudio).

Also added an example in docs:

from huggingface_hub import InferenceClient

# works with remote url or base64 encoded url
image_url ="https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"

client = InferenceClient("mistralai/Mistral-Nemo-Instruct-2407")
output = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {"url": image_url},
                },
                {
                    "type": "text",
                    "text": "Describe this image in one sentence.",
                },
            ],
        },
    ],
)

print(output.choices[0].message.content)
#A determine figure of Lady Liberty stands tall, holding a torch aloft, atop a pedestal on an island.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[MoritzLaurer]

Cool! One great addition would be to also explicitly document how people can use a local, non-public image in PIL and convert it to a base64 string which the API also understands. People often get confused about this. I use code similar to this:

import base64
from io import BytesIO
from PIL import Image

# Open an image using PIL
image = Image.open('path_to_your_image.png')

def image_to_data_url(image):
    # Use the image's format or default to 'PNG' if format is None
    format = image.format or 'PNG'
    with BytesIO() as buffer:
        image.save(buffer, format=format)
        img_str = base64.b64encode(buffer.getvalue()).decode('utf-8')
    return f"data:image/{format.lower()};base64,{img_str}"

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "some prompt"
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": image_to_data_url(image)
                },
            }
        ]
    }
]

[Wauplin]

I'll see how I can integrate that in the docs 👍 I don't want the snippet to be bloated with 3 different use cases. For now I have:

>>> image_path = "/path/to/image.jpeg"
>>> with open(image_path, "rb") as f:
...     base64_image = base64.b64encode(f.read()).decode("utf-8")
>>> image_url = f"data:image/jpeg;base64,{base64_image}"

I'm thinking that in addition to the method docstring we should maybe add a section in the Inference guide typically to explain better how to deal with system prompt, tools, images, etc.

[hanouticelina]

All good for me! thanks @Wauplin

[LysandreJik]

Thank you @Wauplin

[osanseviero]

Nice!

[osanseviero]

should this be pixtral? nemo instruct is text-only