Resampling support? (mainly for compatibility)

[JasonLG1979]

I've been playing around with resampling and I came up with this:

https://gist.github.com/JasonLG1979/8e9f4a9af489b68cc0ab463d6b32b6f6

It's a pure vanilla rust, no external crates resampler that could pretty simply be integrated into librespot. The way it's written now it's hard coded to resample to 48kHz but that would not be difficult to change but it's really not super necessary. The idea being that if a person can help it they really shouldn't resample anyway and this resampler is for compatibility purposes for people who have devices that do not support 44.1kHz and that do not want to use external resamplers for whatever reason. It uses an adaptive resampling approach since it's bufferless and you never really know how big a sample packet is going to be. It uses cubic, quadratic, linear or sample and hold depending on where it's at processing the packet so it never shorts the output samples. 99.99% of the time it uses cubic only at the end of a packet % 4 samples get processed by the other methods depending on how many there are.

I'm using it right now with librespot just to test it. I've hacked it in after the normalizer and hard coded the alsa backend to 48kHz. I thought I'd get some opinions here before I do the work of integrating it correctly.

Cubic isn't exactly the best method as far as quality but it's still very serviceable IMHO and it's super light weight using no perceivable additional CPU on my desktop. I have yet to test it on my Pi Zero v2.

[roderickvd]

Certainly interesting to evaluate.

Upside is that this enables using cpal directly instead of Rodio and be compatible with Windows. I think my own repo should still have a cpal branch that could easily be integrated. Rodio has a lot of unnecessary cruft that we could do without.

On the flip side we could question if we should then not just take a look at rubato as instead of having a bespoke solution.

And of course, maybe first questions to ask do we need the compatibility? Do we want cpal instead of Rodio? And on Linux, this instead of libresample or the likes? Not saying yes or no but just stating the questions.

Let’s hear everyone’s thoughts.

[JasonLG1979]

It was just curiosity really that made me write it. I just wanted to see how exactly resamplers worked and if I could write one. Turns out a basic resampler isn't that complicated.

I'd be interested in adding rubato. Another reason I wrote it in vanilla Rust is so that it would have no additional deps. I also purposely didn't use a more complex and computationally expensive interpolation method because the idea is for it to be a good enough solution not a perfect one. The purpose would be to serve more as a fallback then an actual feature. That's why I'm tempted to say we only support resampling to 48kHz since that's the most common sampling rate besides 44.1kHz.

[JasonLG1979]

@roderickvd I've done sort of a deep dive into resampling and interpolation. This now does Windowed Sinc Interpolation. It will resample from 44.1kHz to 48kHz, 88.2kHz or 96kHz. You can choose from the Hann, Hamming, Nuttall, Blackman, Blackman-Harris, and Blackman-Nuttall window functions, and you can choose Sinc Qualities, Low, Medium, High, and EatMyCpu.

[JasonLG1979]

@roderickvd Update:

It now does Linear or Sinc Windowed Interpolation depending on InterpolationQuality.

The main entry point StereoIntervaledResampler takes:

        sample_rate: Option<SampleRate>,
        quality: Option<InterpolationQuality>,
        window: Option<Window>,

Where SampleRate is:

#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub enum SampleRate {
    #[default]
    Hz44100,
    Hz48000,
    Hz88200,
    Hz96000,
}

InterpolationQuality is:

#[derive(Clone, Copy, Debug, Default)]
pub enum InterpolationQuality {
    #[default]
    Low, // Linear
    Medium, // Sinc with a window length of 129
    High,   // Sinc with a window length of 257
}

and Window is:

#[derive(Clone, Copy, Debug, Default)]
pub enum Window {
    Hann,
    Hamming,
    Nuttall,
    Blackman,
    #[default]
    BlackmanHarris,
    BlackmanNuttall,
}

Passing None for any value is the same as passing it's default.

If the sample_rate == SampleRate::Hz44100 resampling is bypassed.

Testing on my Pi Zero v2 seems to show that it's lighter weight then libsamplerate by a good margin, able to resample to SampleRate::Hz96000, InterpolationQuality::High easily.

Oh, did I mention it's also dual threaded? Anyway here's a taste of what htop tells me about CPU usage on my Pi Zero v2 using SampleRate::Hz96000, InterpolationQuality::High:

[JasonLG1979]

In the long run this is my proposal:

#[derive(Clone, Copy, Debug, Default)]
pub struct AudioPipelineConfig {
    format: AudioFormat,
    sample_rate: SampleRate,
    interpolation_quality: InterpolationQuality,
    interpolation_window_function: InterpolationWindowFunction,
    normalisation_type: NormalisationType,
    volume_type: VolumeType,
    dither_type: DitherType,
    decoder: Decoder,
    backend: Backend, 
}

// The idea is that all player would do is handle message passing back and forth between the
// the audio pipeline and the API bits. Player would for instance hand url's to the audio pipeline
// and instruct it to do things like pause, stop, play, and seek, and whatnot. The pipeline would do
// it's thing and be able to provide information like the current url, spotify id, position, duration,
// and whatever other things that the pipeline should know, and ofc respond to commands.
//
// This makes for a good split in the separation of concerns. The player doesn't know about nor does
// care about the internal state of the pipeline beyond what the pipeline tells it and the pipeline
// doesn't know about the player at all.   

// PipeLine Flow:
//      Decoder -> De-interleave samples (if the decoder can't do that for us?)
//
//      In separate left and right channel threads ->
//          Interpolate if sample rate != input sample rate ->
//          Normalize if normalisation type != Bypass ->
//          Apply volume if volume type != Bypass ->
//          Convert to whatever format the backend needs apply dither if applicable ->
//          Send results back to the main PipeLine thread
//
//      Interleave samples in the main Pipeline thread ->
//      Feed the Backend
//
//  Bubble up all errors.

[JasonLG1979]

I added Resampling support to the Raspotify branch (a Frankenstein'd version of Librespot main) I have not changed all the backends in that branch to use the different sample rates yet since all Raspotify uses is the ALSA and PulseAudio backends, but so far so good.

[JasonLG1979]

This commit moves resampling, normalization and volume out of player into a SamplePipeline.

JasonLG1979@28fe6d1

It would not be hard to port it over to dev if there is any interest?