[Safetensors parser] Support optional file path

[mishig25]

Before

Safetensors parser was in assumption that safetensors files are strictly named model.safetensors, which is not the case always.

This PR

Support optional file names so that other names like llama3.safetensors would still work

[coyotte508]

What happens if the safetensors index file is inside a folder?

Should we use the folder as the "base" path for the other files?

[mishig25]

cc: @apolinario for the upcoming moon support

[julien-c]

lgtm with a few suggestions!

[coyotte508]

Looks good, just this question:

What happens if the safetensors index file is inside a folder?

Should we then use the folder as the "base" path for the other files?

(if relevant)

[julien-c]

yes valid question from @coyotte508 actually (read too fast initially)

IIRC @Narsil the shard index file contains relative filenames (there's no notion of directory), right? we should always assume the single files are in the same folder imo, correct?

(we can fix this in a later PR though i guess – and add a test from a real model)

[mishig25]

(we can fix this in a later PR though i guess – and add a test from a real model)

merging this PR

[osanseviero]

Note that for diffusion, the situation is a bit tricky as we have non-diffusers safetensors at the top directory, while diffusers safetensors in folders

See https://huggingface.co/stabilityai/sdxl-turbo/tree/main

• Files people usually download manually https://huggingface.co/stabilityai/sdxl-turbo/blob/main/sd_xl_turbo_1.0.safetensors (e.g. for Auto1111)
• File usually accessed via diffusers https://huggingface.co/stabilityai/sdxl-turbo/blob/main/text_encoder/model.fp16.safetensors

[julien-c]

@osanseviero yes. So this PR should work, no?

[osanseviero]

Yes, this PR is good, but we might need to make some distinction for diffusers in the UI between these two types of safetensors models.

[mishig25]

Yes, this PR is good, but we might need to make some distinction for diffusers in the UI between these two types of safetensors models.

I think that task is for the moon

[osanseviero]

Yes, this indeed is for moon

[julien-c]

yes valid question from @coyotte508 actually (read too fast initially)

IIRC @Narsil the shard index file contains relative filenames (there's no notion of directory), right? we should always assume the single files are in the same folder imo, correct?

Let's open an issue so we don't forget this, ok?

[julien-c]

@osanseviero for diffusers on the main model page i think we will still only parse a file model.safetensors so we won't display a model size for diffusers (too complex to calculate from different files / not worth it for now IMO)