OpenAI new model: large v3 turbo - 8x faster!

[thewh1teagle]

openai/whisper#2361

How can we convert the .pt file to ggml and quantize it?
I tried to use the script from models folder, converted and quanitized but it failed to load

2024-09-30T23:50:38.038110Z ERROR whisper_rs::whisper_sys_tracing: whisper_model_load: tensor 'encoder.conv1.weight' has wrong size in model file: got 983040, expected 1966080
2024-09-30T23:50:38.038140Z ERROR whisper_rs::whisper_sys_tracing: whisper_init_with_params_no_state: failed to load model

[Runtrons]

+1 this. I think it is now a little different on how it runs, which will complicate things. But a new fast model, with whisper.cpp could be awesome!

[thewh1teagle]

I don't see much improvements comparing to ggml-medium.
It's certainly not 8x faster than medium model. maybe 1.2x faster.
Also much less accurate than whisper-medium. (Hebrew)

Am I missing something?

Also I compared it to the tiny model. it's much slower than that.

[misutoneko]

Yeah the speed seems similar to medium. OTOH I'm using this on old hw in CPU-only mode so idk.
I also noticed that the --suppress_tokens "" trick doesn't work with this one, or maybe it just needs its own set of tokens to suppress.

EDIT: Turbo does hallucinate like a champ. It seems sound descriptions have been mostly removed, so the model can't fall back to them like most of the earlier models. I wonder if this is the reason why it likes to hallucinate so much.

That trick I mentioned is for the original Whisper (by OpenAI) and when I previously tested it with whisper.cpp, it didn't really work. BUT it is possible to manipulate token probabilities directly in code. I've been playing with that a little bit, the results are a bit lackluster for now though. I guess there's only so much that can be accomplished with code only.

[silvacarl2]

if it really is a whisper-large-v3 model version, then it is smoking fast compared to just using regular whisper-large-v3

at a quick glance it seems to be as accurate, possibly more accurate.