An attempt to add support for decoding input with ffmpeg in Apple M1

[jingcodeguy]

I have been thinking about adding FFmpeg support during the file reading process without needing to convert the file manually each time while using a Mac M1 Max. Although I can resolve this issue by using a bash script to convert the file first and then pass it to whisper.cpp, I still want to try streamlining the process.

Inspired by @WilliamTambellini's PR for adding support for decoding input with FFmpeg in Linux, I have tried to implement this feature for M1.

However, it is not 100% successful because it seems that Whisper cannot recognize the text normally using the resulting WAV file. Here are the details of my trial and implementation.

At first, I attempted to use the implemented ffmpeg-transcode.cpp but soon realized that it depends on the version of FFmpeg. With the latest versions (after 6.x), there are many changes, and there is not much example or reference to migrate the old methods. It is necessary to read and test based on the documentation, which is not easy to digest, at least for me.

Then I tried another approach by using a subprocess. I conducted isolated tests for a simple implementation of the subprocess using the Boost library and the Poco library. However, I found that there were problems with the Boost library when integrating it into common.cpp. So, I chose Poco, which works properly.

I tried to replicate the implemented FFmpeg logic for Linux by picking up the file and returning a WAV audio in the pipe. I have tried two approaches; one almost worked, and the other did not work. ("Did not work" means Whisper cannot transcribe it properly, "almost worked" means Whisper can transcribe some parts but then gives garbage.)

During the isolation test, I output the stream read to a file to ensure the audio file works.

The implementation is as follows:
The first one uses a custom WAV header:

#include <iostream>
#include <fstream>
#include <vector>
#include <cstdint>
#include <string>
#include <sstream>
#include <cstdio>
#include <Poco/Process.h>
#include <Poco/Pipe.h>
#include <Poco/PipeStream.h>

struct WavHeader {
    char riff[4] = {'R', 'I', 'F', 'F'};
    uint32_t overall_size;
    char wave[4] = {'W', 'A', 'V', 'E'};
    char fmt_chunk_marker[4] = {'f', 'm', 't', ' '};
    uint32_t length_of_fmt = 16;
    uint16_t format_type = 1;
    uint16_t channels = 1;
    uint32_t sample_rate = 16000;
    uint32_t byterate = 32000;
    uint16_t block_align = 2;
    uint16_t bits_per_sample = 16;
    char data_chunk_header[4] = {'d', 'a', 't', 'a'};
    uint32_t data_size;
};

bool ffmpeg_decode_audio(const std::string & ifname, std::vector<uint8_t> & wav_data) {
    std::ostringstream oss;
    oss << "ffmpeg -i \"" << ifname << "\" -f wav -ar 16000 -ac 1 -c:a pcm_s16le -";
    std::string ffmpeg_command = oss.str();

    Poco::Pipe outPipe;
    Poco::ProcessHandle ph = Poco::Process::launch("/bin/sh", {"-c", ffmpeg_command}, nullptr, &outPipe, nullptr);

    Poco::PipeInputStream istr(outPipe);

    std::vector<char> audioData;
    std::vector<char> buffer(4096);
    while (istr) {
        istr.read(buffer.data(), buffer.size());
        std::streamsize count = istr.gcount();
        audioData.insert(audioData.end(), buffer.begin(), buffer.begin() + count);
    }

    WavHeader header;
    header.data_size = audioData.size();
    header.overall_size = header.data_size + sizeof(WavHeader) - 8;

    wav_data.clear();
    wav_data.reserve(sizeof(header) + audioData.size());
    
    wav_data.insert(wav_data.end(), reinterpret_cast<uint8_t*>(&header), reinterpret_cast<uint8_t*>(&header) + sizeof(header));
    wav_data.insert(wav_data.end(), audioData.begin(), audioData.end());

    int result = ph.wait();

    fprintf(stdout, "FFmpeg process finished with result: %d\n", result);
    return result == 0;
}

This method almost does the job, but transcription begins to give garbage after handling 1-2 minutes of the file. It is believed that the format of the data affects the result. Before adding the custom WAV header, although the output audio file could be played, the transcription gave only garbage. When I use MediaInfo to check out the specs of the file, the format is incorrect compared to the one I use FFmpeg manually output in the terminal.

The other approach is merely another attempt to give a correct header or WAV format, but then Whisper cannot transcribe it and gives only garbage.

Because my knowledge of handling WAV or signal processing is very limited, and I am not experienced in writing code to handle memory and file format, any veteran who can share their knowledge would be highly appreciated.