Improve geotop functionality #196

dbrakenhoff · 2023-06-21T12:33:38Z

See #193 .

Introduces:

Restructure geotop reading to be more similar to regis (e.g. get_geotop now downloads raw data)
Use dask.delayed to download geotop from URL
Add support in caching for dask.delayed datasets
rewrite memory intensive operations to be more memory efficient

Warning: this PR introduces some breaking changes:

nlmod.read.geotop.get_geotop() now loads raw geotop data (what used to be performed by get_geotop_raw_within_extent(). Using this old method will raise a DeprecationWarning).
to reproduce the old get_geotop(), which converts geotop data to a layer model:

geotop_ds = nlmod.read.geotop.get_geotop(extent)  # load raw data
layer_ds = nlmod.read.geotop.to_model_layers(geotop_ds)  # convert to layer model

- use chunks and dask delayed to keep memory usage efficient - get_geotop now gets raw data - add deprecationwarning to get_geotop_raw_within_extent()

- now takes geotop_ds as input to convert to layers (not extent)

- do not use sortby but check diff of y dimension and flip if necessary

- use .values instead of .data to load data into memory if using dask - undo onno changes for now w missing strat code

- use .values to deal with dask arrays - use get_geotop - update docstring

…top_to_ml_layers

Add

nlmod/dims/layers.py

- replace Exception with more specific types - add LayerError and MissingValueError - fix some typos

dbrakenhoff and others added 23 commits June 16, 2023 11:13

improve get_geotop

2c918b4

- use chunks and dask delayed to keep memory usage efficient - get_geotop now gets raw data - add deprecationwarning to get_geotop_raw_within_extent()

old get_geotop is renamed geotop_to_layermodel:

456b983

- now takes geotop_ds as input to convert to layers (not extent)

clean up old code

84593ce

improve get_regis:

62a9861

- do not use sortby but check diff of y dimension and flip if necessary

add cache support for chunked datasets

56a264e

add log msg to split_layers

68a4c1f

remove nan layer (value -127) from geo_eenheden

4ed3976

use values instead of data to deal with potential dask array

25f4f72

improve geotop:

f9cc9d7

- use .values instead of .data to load data into memory if using dask - undo onno changes for now w missing strat code

if caching with dask, close and reopen ds to ensure ds points to file

34dfdcf

fix DeprecationWarning

de54e3d

fix tests

87e7afe

fix geotop_lithok_in_cross_section

a6a26c5

- use .values to deal with dask arrays - use get_geotop - update docstring

Rename geotop_to_layermodel to to_model_layers and remove convert_geo…

471e724

…top_to_ml_layers

Some minor changes

26db2e8

Fix combination of regis and geotop

a551f55

Improve geotop.to_model_layers and add method insert_layer

fd588c7

Add extra legends for layer models

776a654

Add

Add Formatie van Nieuwkoop, Laagpakket van Nij Beets in geotop

ecbe766

Add test_insert_layer

e016683

Fix tests and codacy issues

25ad000

Set the default of method_gullies to "add_to_layer_below"

c033401

Rename gullies to geulen and gully to geul

c45d9a0

rubencalje marked this pull request as ready for review July 17, 2023 13:39

dbrakenhoff commented Jul 17, 2023

View reviewed changes

nlmod/dims/layers.py Outdated Show resolved Hide resolved

dbrakenhoff commented Jul 17, 2023

View reviewed changes

dbrakenhoff added 2 commits July 17, 2023 17:49

resolve my own comments

b4140db

please codacy

28ee878

- replace Exception with more specific types - add LayerError and MissingValueError - fix some typos

dbrakenhoff merged commit 787e12c into dev Jul 20, 2023

dbrakenhoff deleted the 193-geotop branch July 20, 2023 11:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve geotop functionality #196

Improve geotop functionality #196

dbrakenhoff commented Jun 21, 2023 •

edited

Loading

Improve geotop functionality #196

Improve geotop functionality #196

Conversation

dbrakenhoff commented Jun 21, 2023 • edited Loading

dbrakenhoff commented Jun 21, 2023 •

edited

Loading