Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No reproducibility of steps blended nowcast when using noise_stddev_adj='auto' #346

Closed
mpvginde opened this issue Feb 6, 2024 · 4 comments
Assignees
Labels
bug Something isn't working

Comments

@mpvginde
Copy link
Contributor

mpvginde commented Feb 6, 2024

When running 2 identical blended nowcasts with pysteps.blending.steps.forecast using
a fixed seed
and setting
noise_stddev_adj = 'auto',
the results (sometimes) differ.

I suspect this behavior is coming from the
pysteps.noise.utils.compute_noise_stddev_adjs function.

This function takes the seed argument, but inside the function the seed is always set to None:

randstates = []
seed = None
for k in range(num_iter):
randstates.append(np.random.RandomState(seed=seed))
seed = np.random.randint(0, high=1e9)

Leading to different normalization weights for every run and impacting the final results, preventing reproducibility.

Kind regards,
@mpvginde

@dnerini
Copy link
Member

dnerini commented Feb 6, 2024

Hi @mpvginde thanks for reporting this issue! Do you want to go ahead and submit a pr (looks like a simple fix)? Otherwise I'll try to have a look on it

@mpvginde
Copy link
Contributor Author

mpvginde commented Feb 7, 2024

Sure @dnerini. I'll have a look at it.

@mpvginde mpvginde self-assigned this Feb 7, 2024
@mpvginde mpvginde added the bug Something isn't working label Feb 7, 2024
@mpvginde
Copy link
Contributor Author

mpvginde commented Feb 9, 2024

A small update:
The issue might be more complex as I initially thought.
When I fix passing the seed in the pysteps.noise.utils.compute_noise_stddev_adjs results become reproducable for the pysteps.blending.steps.forecast routine.
However, for pysteps.nowcast.steps.forecast results still differ between identical runs.
Only when I disable the dask parallel workers in the _update helper routine in nowcasts.steps the results become reproducable.
I suspect this might be linked to issue #337, I will investigate further.

@mpvginde
Copy link
Contributor Author

mpvginde commented Feb 9, 2024

See #337 (comment)

Should be fixed with PR #347

mpvginde added a commit to mpvginde/pysteps that referenced this issue Feb 20, 2024
dnerini added a commit that referenced this issue Mar 5, 2024
* Bugfix: fix random placement of ensemble members in numpy array due to dask multi-threading (#337)

* Bugfix: make STEPS (blending) nowcast reproducable when the seed argument is given (#346)

* Bugfix: make STEPS (blending) nowcast reproducable, independent of number of workers (#346)

* Formatting with black

---------

Co-authored-by: ned <[email protected]>
@mpvginde mpvginde closed this as completed Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants