Add aggregate_function argument to utils.concat()

[hagenw]

This adds the aggregate_function argument to audformat.utils.concat().
If overwrite=False and aggregate_function is not None it will be used as a callable that combines values for all entries that have more than one data point.

NOTE: in test_utils_concat.py only the tests test_concat_aggregate_function() and test_concat_overwrite_aggregate_function() are new, the other part is moved from test_util.py.

[codecov]

Codecov Report

Merging #401 (2dc437e) into main (d461614) will not change coverage.
The diff coverage is 100.0%.

Files	Coverage Δ
audformat/core/utils.py	100.0% <100.0%> (ø)

[frankenjoe]

index = audformat.filewise_index(['f1', 'f2', 'f3', 'f4'])
df1 = pd.DataFrame(
    {
        'a': [1, 1, 1],
        'b': [1, 1, 1],
    },
    index=index[:3],
)
df2 = pd.DataFrame(
    {
        'a': [2, 2, 2],
        'b': [2, 2, 2],
    },
    index=index[1:],
)

audformat.utils.concat([df1, df2], aggregate_function=np.sum)

         a     b
file            
f1    <NA>  <NA>
f2       3     3
f3       3     3
f4       2     2

But I was expecting:

         a     b
file            
f1       1     1
f2       3     3
f3       3     3
f4       2     2

[hagenw]

Thanks for spotting this, I fixed it (and a related issue) and now we get:

      a  b
file      
f1    1  1
f2    3  3
f3    3  3
f4    2  2

[frankenjoe]

Thanks for spotting this, I fixed it (and a related issue) and now we get:

I can confirm that example is now working with two frames, but when I add a third one I get:

index = audformat.filewise_index(['f1', 'f2', 'f3', 'f4'])
df1 = pd.DataFrame(
    {
        'a': [1, 1, 1],
        'b': [1, 1, 1],
    },
    index=index[:3],
)
df2 = pd.DataFrame(
    {
        'a': [2, 2, 2],
        'b': [2, 2, 2],
    },
    index=index[1:],
)
df3 = pd.DataFrame(
    {
        'a': [3],
        'b': [3],
    },
    index[:1],
)

audformat.utils.concat([df1, df2, df3], aggregate_function=np.sum)

      a  b
file      
f1    1  1
f2    3  3
f3    3  3
f4    2  2

but what I would expect is:

      a  b
file      
f1    4  4
f2    3  3
f3    3  3
f4    2  2

[hagenw]

Thanks for finding another example that didn't worked. There was indeed a bigger error in how the overlapping values were collected as for the very first column we need to collect all values that overlap with any of the other columns. Whereas for all other columns it is just fine to collect values that overlap with the first column. I changed the code accordingly, and added a few more tests (most likely still not enough ;) ).

Now we get:

import audformat
import numpy as np
import pandas as pd

index = audformat.filewise_index(['f1', 'f2', 'f3', 'f4'])
df1 = pd.DataFrame(
    {
        'a': [1, 1, 1],
        'b': [1, 1, 1],
    },
    index=index[:3],
)
df2 = pd.DataFrame(
    {
        'a': [2, 2, 2],
        'b': [2, 2, 2],
    },
    index=index[1:],
)
df3 = pd.DataFrame(
    {
        'a': [3],
        'b': [3],
    },
    index[:1],
)

audformat.utils.concat([df1, df2, df3], aggregate_function=np.sum)

      a  b
file      
f1    4  4
f2    3  3
f3    3  3
f4    2  2

[frankenjoe]

Looks good, can't find another failing example. Nice extension, strange we didn't come up with this aggregate solution earlier.

[hagenw]

I guess there was no urgent need to have it earlier.