Issue #134 merge partitions with inconsistent grids amongst partitions

[JoerivanEngelen]

Fix #134 and #86,

Add support for merging partitions with different grids in each partition, for example 1D grid in one partition and 2D grids in each. Quite some refactoring was required:

• groups grids, data_objects, and other_vars by gridname
• all partitions with a certain grid should have the same variables, this is validated
• connectivity dims are padded to maximum size if not consistent
• add property to AbstractUgrid to get connectivity dimension names

[veenstrajelmer]

Tested with both examples from #134 and #86 and they both work. However, I do notice odd behaviour. When using xu.open_mfdataset the merging takes 0.3 seconds. When using xu.open_dataset the merging takes >260 seconds:

import glob
import xugrid as xu
import datetime as dt

file_nc = r'p:\1230882-emodnet_hrsm\GTSMv5.0\SO_NHrivGTSM\computations\BD014_fix_mapformat4\output\gtsm_model_0*_map.nc'
file_nc_list = glob.glob(file_nc)
partitions = []
for iF, file_nc_one in enumerate(file_nc_list):
    uds_part = xu.open_mfdataset(file_nc_one)
    partitions.append(uds_part)
dtstart = dt.datetime.now()
uds = xu.merge_partitions(partitions)
print(f'merging took: {(dt.datetime.now()-dtstart).total_seconds():.2f} sec')

Using xarray 2023.11.0

[Huite]

I guess you can also try loading the datasets with just xr.open_dataset, then turning them into UgridDatasets, then merging them.
That will give the same result, I'm fairly sure. The xugrid.open_mfdataset just adds the data_vars kwarg then calls xr.open_mfdataset.
From the source, parallel=False by default in the mf_opendataset call and that triggers some dask usage or not -- so that shouldn't make a difference here. But maybe there's something more subtle going on as well.

Is there maybe a difference between the eagerness? If you call .compute() in the timing, are the timings the same?

[veenstrajelmer]

@Huite, thanks for your suggestions. Indeed, the combination of ds=xr.open_dataset() and xu.UgridDataset(ds) behaves the same. And your second suggestion, to call .compute() significantly increases the performance indeed (0.8 sec). This also triggered me to remember the importance of the chunks argument (chunks={"time":1} is the default in dfm_tools), this also worked as a charm (0.5 sec). So I guess we can conclude there was no additional problem introduced with this PR, and the PR also provides what it aims to do.

[veenstrajelmer]

Thanks for implementing this. There seemed to bee some issues, but input from @Huite I think we can conclude this PR works like a charm.

[Huite]

Uh, this is added under fixed and changed? Maybe just mention it only for "fixed"

[JoerivanEngelen]

I was a bit in doubt:

I added it to changed, because it is a change of behavior: Prior to this changeset, we test explicitly if all grids occur in every partitions and throw a error if this is not the case. Therefore this changeset doesn't really "fix" a bug, it alters behavior.

On the other hand, the changeset brings xugrid closer to supporting UGRID conventions, so if the premise of xugrid is to support every UGRID file (at least: those without mistakes), you can argue this is a "fix".

For now, I just kept the text under "changed".

[Huite]

Great job!