Fix: improve support for large (1-2h+) audio files [Getting crash While entering foreground from background]

[ReimannSamuel]

While sending the audio message im using the Waveform. Once i send the audio message and move to the background and return back to foreground im getting crash. Package version 13.0.2
Since we are using the minimum deplyoment version of iOS 13 what is the latest version of the package i should use?

[ReimannSamuel]

can u pls support @dmrschmidt ?

[dmrschmidt]

Hey, thanks for reporting this. I'm traveling at the moment with limited access. Will have better access tomorrow afternoon / evening Bali time.

This one looks kinda strange to me... It's crashing in WaveformAnalyzer:178, yeah?

I'm not quite sure how that line could crash, unless backfillPaddingSampleCount16 has an invalid value. Its value is not visible from the debug console screenshot you shared. Could you have a look what the value is?

13.0.2 is the latest version that you can use with this old of an iOS version to support.

[ReimannSamuel]

We are planning for a release today in 2 hrs. Is it possible to look into the issue?

[dmrschmidt]

I can absolutely empathize with the urgency of your situation.

As I mentioned in my last message, I am traveling currently and it is already late in the evening in my timezone. So as much as I’d be of more help, I won‘t be able to assist you resolve the issue within the next 2h.

Once I am back on my laptop with good internet tomorrow, I will look into it. Please remember to add the answer to my sampleCount question above.

My hip shooting best guess right now is that this is an edge case relating to the duration of that audio. If it’s possible to also attach the audio file here that would definitely allow me to look into this more independently.

[ReimannSamuel]

Hi @dmrschmidt , Is there any solution for the above mentioned issue?

[dmrschmidt]

Please update this issue with the information I had asked for last week & attach the audio file causing the problem so I can investigate.

[ReimannSamuel]

Its not about the audio file. Even if i send the recorded audio message and move to the background while the message is still in the sending state and move to the foreground after few seconds the crash occurs as mentioned above

[dmrschmidt]

And what is the value of backfillPaddingSampleCount16 when the crash occurs? From the "stack trace" in the screenshot, it appears that the only reason this crash could happen on that specific line is, when this variable has an invalid value for the array creation. So either negative or overflowing.

[ReimannSamuel]

VideoVideoc044b16d6b594b26a310ef521e60c8b1.mp3 <https://drive.google.com/file/d/1bA045UwYsLLBQs1eJVPpyqM_AlOruYD-/view?usp=drive_web> This is the audio im using Reimann Samuel W Junior iOS Developer, PRODUCT TEAM <https://in.linkedin.com/in/sathish-t> <https://www.facebook.com/vignesh.n> <https://www.instagram.com/3984sathish> <https://twitter.com/vignesh.n> <https://www.pinterest.com/vignesh.n> <http://stackoverflow.com/vignesh.n> <https://www.behance.net/vignesh.n> ***@***.***,80.200345,15z/data=!4m2!3m1!1s0x0:0xb3e84ab20dc3785e> No. B8, 9 &10. 8th floor, Baashyaam willow square, Guindy, Chennai - 600032, India. +91 9865656320 www.contus.com ***@***.*** [image: GPTW_CONTUS TECH Logo (1).png]

…

On Mon, 22 Jan 2024 at 11:23 AM, Dennis Schmidt ***@***.***> wrote: Please update this issue with the information I had asked for last week & attach the audio file causing the problem so I can investigate. — Reply to this email directly, view it on GitHub <#93 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A5XDMCICYCXXVQG7V756HR3YPX5GZAVCNFSM6AAAAABCAF3OAGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBTGMYTEMZQHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[dmrschmidt]

Can't download the file. Requested access via the Google website I land on.

[dmrschmidt]

got it now 👍

[ReimannSamuel]

The value of backfillPaddingSampleCount16 is 1766440960 , missingSampleCount is 883752960 and backfillPaddingSampleCount is 883220480 during the crash

[ReimannSamuel]

[dmrschmidt]

So... That file is enormous. Almost 3 hours, which equates to roughly 1.6GiB uncompressed file size. While the library attempts to be resourceful, this is still an enormous amount of data to process.

Given that it looks like the crash occurs in a line which is only allocating memory, and the values you posted look generally normal, I can only assume that the device is running out of memory.

There's really not much I can do about that in the short term. Debugging this might take a lot of time, especially while I cannot reproduce the issue myself so far. Will try on my device soon, but I only have an iPhone 15 Pro right now. Also other factors within your app may contribute to this as well potentially, especially considering the backgrounding aspect.

It might be that there was some progress happening while it is backgrounding and then the memory state is not correctly handled when coming back into the foreground and resuming the operation.

I will experiment a little bit, but unfortunately all I can say for now is that the library is not optimized to deal with files of that extreme size.

The waveform of a 3h file also becomes fairly useless as the resolution of a phone screen is just obviously way too low. My recommendation as a workaround here is for now probably to use a fake waveform for files of that size. Not great, but all I can suggest at this point.

[dmrschmidt]

So yeah, it's trying to allocate 1766440960 byte, which - if my math isn't totally wrong - is roughly those 1.6GiB. So that would cause an out of memory crash easily.

So what I wrote before stays true. The library is not designed to work with files of this magnitude. It's designed to work well for typical audio files in the 3-10 minute range.

Adapting it to efficiently deal with files of these sizes will take time and I can't promise anything there at the moment.

[ReimannSamuel]

Thanks for the update @dmrschmidt . If i set a fake wave form , while playing the audio how will it work? and In future is there any possibilities to work on this for larger audio files?

[dmrschmidt]

I've put this on my To Do list, so eventually I hope to be able to support this. I kind of thought that it should have already worked to some degree, or at least not that it would ever attempt to allocate gigabytes of data at once. Anyway, I'm pretty busy with other projects atm, so I don't think this will come anytime soon, I'm afraid.

For inspiration on how to indicate progress with a fake waveform, have a look at two different approaches for SwiftUI and UIKit in ProgressWaveformView and ProgressViewController.

Essentially either drawing two shapes over each other (the SwiftUI example) or using a clipping mask for two images (the UIKit example).

I'd recommend using a static image / or svg (for SwiftUI shape support) and hand-rolling it entirely. But of course you could also stick to using the library and simply use a fake mp3 file as a stand-in. That would allow using the above code without any changes. But it requires unnecessary processing and storing of a fake mp3.

[ReimannSamuel]

Thanks for the support @dmrschmidt . If the waveform is supported for larger audio files then it will be very useful.

[Lessica]

The same situation here. But my audio file is only 9.8 MB (compressed in m4a, duration: 46:16).

The app crashes while entering foregound from background (wave analyzing is in progress), I found the peak memory is extremely high (over 4 GB) before the crash:

There must be something wrong.

[Lessica]

DSWaveformImage/Sources/DSWaveformImage/WaveformAnalyzer.swift

Line 166 in 4c56578

guard sampleLength / samplesPerPixel > 0 else { return downSampledData }

This guard does not make any scene to my situation.

[dmrschmidt]

The compressed size is not really relevant here, since the processing happens in memory and needs to uncompress the file for that. Which for 46 minutes is still around 450MB.

The guard there is really just a workaround that hasn’t been improved yet.

That being said, there is definitely issues with the code for large files of course. Reading from the buffer is supposed to happen in chunks. This used to happen inside of an @autoreleasepool to avoid memory spikes. I had removed that at some point because it seemed not necessary anymore with Swift if I remember correctly. That may have been a wrong assumption, so that’s my best guess so far.

[dmrschmidt]

I will try to carve out some time over the next couple days to look into this issue again and hopefully find a fix.

[dmrschmidt]

I did find a bit of time to look into this today and think at least understand part of the underlying issue a bit better. I think it's related to the handling of the buffer, which appears to not get free'd up quickly enough - or something along those lines. With files this long, the downsampling to 1 one single pixel on a typical iPhone screen requires the system to look at - in my current test file case - a whole 145MB of uncompressed audio data. This should generally still be fine, but it appears that the rolling buffer that's in use grows too large in some scenarios. I still have to figure out what that scenario is though. Just haphazardly sprinkling autoreleasepools in different places did not yield any improvements so far.

So I'll keep this updated here when I find out more.