Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmark For Audio Feature Extraction Libraries #22

Open
liufeigit opened this issue Apr 23, 2023 · 2 comments
Open

Benchmark For Audio Feature Extraction Libraries #22

liufeigit opened this issue Apr 23, 2023 · 2 comments
Labels
discussion Open-ended discussion for developers and users functionality New function

Comments

@liufeigit
Copy link
Member

liufeigit commented Apr 23, 2023

Benchmark

Introduce

In the field of deep learning for audio, the mel spectrogram is the most commonly used audio feature. The performance of mel spectrogram features can be benchmarked and compared using audio feature extraction libraries such as the following:

Library Language Version About
audioFlux C/Python 0.1.5 A library for audio and music analysis, feature extraction
torchaudio Python 0.11.0 Data manipulation and transformation for audio signal processing, powered by PyTorch
librosa Python 0.10.0 Python library for audio and music analysis
essentia C++/Python 2.0.1 C++ library for audio and music analysis, description and synthesis, including Python bindings
  • audioFlux: developed in C with a Python wrapper, it has different bridging processes for different platforms, and supports OpenBLAS, MKL, etc.
  • TorchAudio: developed in PyTorch, which is optimized for CPUs and uses MKL as its backend. This evaluation does not include the GPU version of PyTorch.
  • librosa: developed purely in Python, mainly based on NumPy and SciPy, with NumPy using OpenBLAS as its backend.
  • Essentia: developed in C++ with a Python wrapper, it uses Eigen and FFTW as its backend.

There are many factors that can affect the performance evaluation results, including CPU architecture, operating system, compilation system, selection of basic linear algebra libraries, and usage of project APIs, all of which can have a certain impact on the evaluation results.

For the most common mel features in the audio field, the major performance bottlenecks are FFT computation, matrix computation, and multi-threaded parallel processing, while minor bottlenecks include algorithmic business implementation and Python packaging.

  • Regarding FFT computation, librosa uses scipy's fftpack for accelerated FFT computation, which is slower than FFTW3, MKL, and Accelerate.
  • Regarding matrix computation, MKL is faster than OpenBLAS, while OpenBLAS is faster than Eigen.
  • Regarding multi-threaded parallel processing, it depends on whether each project has support for it.

Scripts

If you want to compare and test multiple libraries, you can use:

$ python run_benchmark.py -p audioflux,torchaudio,librosa -r 1000 -er 10  -t 1,5,10,100,500,1000,2000,3000
  • -p: The library name, list
  • -r: The number of sample data, number
  • -er: The number of run_xxx.py calls, number
  • -t: The time of each sample data, list

If you want to test a single library, you can use:

$ python run_audioflux.py -r 1000 -t 1,5,10,100,500,1000,2000,3000

If you want to see more usage instructions, you can execute python run_xxx.py --help

Notice

In the field of audio, libraries related to audio feature extraction have their own functional characteristics and provide different types of features. This evaluation does not aim to test all the performance comparisons of their feature extraction in detail. However, as the mel spectrum is one of the most important and fundamental features, all of these libraries support it.

There are many factors that can affect the performance evaluation results, such as CPU architecture, operating system, compilation system, choice of basic linear algebra library, and the usage of project APIs, which will have a certain impact on the evaluation results. In order to be as fair as possible and to better reflect actual business needs, the following conditions are based on in this evaluation:

  1. macOS/Linux operating system, three types of CPUs: Intel/AMD/M1.
  2. The libraries use the latest official release version or the latest official source code compiled with high performance support, and the fastest one is selected.
  3. In terms of API usage, the official standards are followed, and "warming up" is used for each corresponding method of the libraries (the first execution time is not counted), and the execution time of the initialization is not counted.
  4. In terms of data length, various actual business considerations are taken into account when selecting the test data.

When the data is short, the first execution time of most libraries may be relatively slow. To reflect actual business needs and to be fair, this first execution time is not counted. If the library API design provides initialization functions, they will be created and repeatedly called in actual business scenarios, and the initialization execution time is also not counted.

Warn

⚠️ When using Python scientific computing related libraries such as Conda, PyTorch, TensorFlow, XGBoost, LightGBM, etc., almost all of them use Intel Math Kernel Library (MKL). MKL uses OpenMP for parallel acceleration, but only one instance of OpenMP can exist in the same process. When these libraries are used together, it is best to link all libraries to the same location of libomp, otherwise an error will occur. Modifying the environment variables according to the prompt may result in slower program execution and unreliable results. Relevant tools can be used to rewrite the libomp linking path of the related libraries.

Performance

Base benchmark

Use audioFlux/torchaudio/librosa script, for AMD/Intel/M1 CPUs and Linux/macOS operating system.

The time required to calculate the mel-spectrogram for 1000 sample data according to a TimeStep of 1/5/10/100/500/1000/2000/3000. Where fft_len=2048, slide_len=512, sampling_rate=32000.

Linux - AMD

- OS: Ubuntu 20.04.4 LTS
- CPU: AMD Ryzen Threadripper 3970X 32-Core Processor

TimeStep audioflux torchaudio librosa
1 0.04294s 0.07707s 2.41958s
5 0.14878s 1.05589s 3.52610s
10 0.18374s 0.83975s 3.46499s
100 0.67030s 0.61876s 6.63217s
500 0.94893s 1.29189s 16.45968s
1000 1.43854s 2.23126s 27.78358s
2000 3.08714s 4.10869s 45.12714s
3000 4.90343s 5.86299s 51.62876s

Linux - Intel

- OS: Ubuntu 20.04.4 LTS
- CPU: Intel(R) Core(TM) i7-6850K CPU @ 3.60GHz

TimeStep audioflux torchaudio librosa
1 0.08106s 0.11043s 5.51295s
5 0.11654s 0.16005s 5.77631s
10 0.29173s 0.15352s 6.13656s
100 1.18150s 0.39958s 10.61641s
500 2.23883s 1.58323s 28.99823s
1000 4.42723s 3.98896s 51.97518s
2000 8.73121s 8.28444s 61.13923s
3000 13.07378s 12.14323s 70.06395s

macOS - Intel

- OS: 12.6.1 (21G217)
- CPU: 3.8GHz 8‑core 10th-generation Intel Core i7, Turbo Boost up to 5.0GHz

TimeStep audioflux torchaudio librosa
1 0.07605s 0.06451s 1.70139s
5 0.14946s 0.08464s 1.86964s
10 0.16641s 0.10762s 2.00865s
100 0.46902s 0.83551s 3.28890s
500 1.08860s 5.05824s 8.98265s
1000 2.64029s 9.78269s 18.24391s
2000 5.40025s 15.08991s 33.68184s
3000 7.92596s 24.84823s 47.35941s

macOS - M1

- OS: 12.4 (21F79)
- CPU: Apple M1

TimeStep audioflux torchaudio librosa
1 0.06110s 0.06874s 2.22518s
5 0.23444s 0.07922s 2.55907s
10 0.20691s 0.11090s 2.71813s
100 0.68694s 0.63625s 4.74433s
500 1.47420s 3.37597s 13.83887s
1000 3.00926s 6.76275s 25.24646s
2000 5.99781s 12.69573s 47.84029s
3000 8.76306s 19.03391s 69.40428s

Summarize

In summary, from the performance comparison results of the three libraries, librosa takes the most time, which is also in line with common sense.
On linux/amd processors, audioflux is slightly faster than torchaudio, but slightly slower on linux/intel.
On the macOS system, for large-size sample data, audioflux is faster than torchaudio, and intel is more obvious than m1; for small-size sample data, torchaudio is faster than audioflux.

⚠️ Although the development of benchmark is attempted to be as objective and fair as possible, every benchmarks have their drawbacks, and are limited to particular testing procedures, datasets and platforms. And also, this benchmark does not compare additional features that a library may support, or other APIs, cross-platform, etc. We encourage users to benchmarks with their own data sets and platforms.

@liufeigit liufeigit added the good first issue Good for newcomers label Apr 23, 2023
@liufeigit liufeigit pinned this issue Apr 23, 2023
@liufeigit liufeigit added discussion Open-ended discussion for developers and users functionality New function and removed good first issue Good for newcomers labels Apr 23, 2023
@liufeigit liufeigit changed the title Benchmark For Audio Feature Extraction Library Benchmark For Audio Feature Extraction Libraries Apr 24, 2023
@LorenzoMonni
Copy link

Why Essentia results are not present in the comparison?

@fanjink
Copy link

fanjink commented Jan 15, 2024

@LorenzoMonni I have tested it before. Is this what you need?
(MTG/essentia#1326)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discussion Open-ended discussion for developers and users functionality New function
Projects
None yet
Development

No branches or pull requests

3 participants