Skip to content
This repository has been archived by the owner on Jun 2, 2023. It is now read-only.

Add random seed replicates to predict step #131

Open
jds485 opened this issue Aug 22, 2022 · 0 comments
Open

Add random seed replicates to predict step #131

jds485 opened this issue Aug 22, 2022 · 0 comments
Assignees

Comments

@jds485
Copy link
Member

jds485 commented Aug 22, 2022

Noted in #125, here:

Random draws of samples into the training and testing sets have resulted in different outcomes for training using all available gages vs. using gages only within the region for which testing will be completed. Adding several replicates and comparing the resulting distribution of RMSEs is one way around this problem.

I think the replicates should use the same selected attributes (from correlation and Boruta screening) and same hyperparameters (to reduce computation times). So, the only difference between replicates would be the dataset used.

With replicate models, we should edit:

  • feature importance plots to display the average value over all replicates (can add error bars)
  • Model RMSE comparison barplots (e.g., p6_compare_RMSE_RF_png) should show average test error and average validation error over all replicates (can add error bars)
  • predicted vs. observed scatterplots to show distributions over all replicates (not sure how is best to show this - would be too crowded to add error bars).
  • residual maps based on average residual over replicates
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants