Automatic Inference Batching Support

[SinanAkkoyun]

Hey!
the HuggingFace text-generation-inference is an inference server solution that can, if you do concurrent HTTP requests, automatically batch compute generations. (Let's say there is a request in progress and another one on the way, it can automatically adapt the batching in-progress)

I want to build an inference solution based on faster-whisper.
Is manual batching supported? I am not sufficient enough to safely implement it on my own, but I would like to build up on top of that, if possible.

[arnavmehta7]

Faster-whisper performs batching internally for various vad_segments. I would be curious to know if we can get more speedups if we batch multiple audios.

Afaik, huggingface one, has a DELTA during which, if multiple requests are sent then they will be batched and run together, otherwise not.

[guillaumekln]

Faster-whisper performs batching internally for various vad_segments.

No, there is no batching at this time. See #59 which is the main issue for batching support.

[arnavmehta7]

@guillaumekln oh very very sorry for the oversight. I feel this could be easily done by taking this as reference https://github.com/m-bain/whisperX/blob/main/whisperx/asr.py#L210-L230