Mismatching result from Auto model and single model with same set of hyperparameters

[masadshoaib]

Dear concerned,

I am writing to discuss an issue I've encountered while working with Nixtla's platform, specifically regarding hyperparameter tuning and its impact on model output consistency.

Recently, I have been experimenting with Nixtla's DilatedRNN using default configurations, alongside running its auto-hyperparameter tuning function (AutoDilatedRNN) with identical settings (same set of parameter values) over 60 months of data. My objective was to achieve consistent output across both approaches. I looked at Nixtla's github's code for NeuralForecast (dilated_rnn.py) for the default values (which again, may not be optimal) of parameters for DilatedRNN and am specifying just that set in hyperparameter space for AutoDilatedRNN.

However, despite setting the configurations the same for both scenarios, I've observed discrepancies in the error returned by each. Upon closer examination, I've identified that during hyperparameter optimization, the model utilizes a validation set in addition to the training data. This results in the training data being limited to 48 months, with the remaining 12 months allocated to validation.

To align the conditions with the default model, I reduced the training data for the default call to 48 months. Surprisingly, the outputs still do not match, suggesting underlying factors influencing the results beyond the training data duration.

Given these findings, I am reaching out to your team to seek clarification on how one can achieve consistent results when running hyperparameter optimization, ensuring they align with those from the default model setup. Could you please provide guidance or insights into this matter?

I appreciate your prompt attention to this issue and look forward to your valuable input. Please let me know if there are any additional details or data I can provide to facilitate this discussion.

I have attached the pdf of the configuration I am passing for the Auto model to this message FYI: RNNandAutoRNN.pdf

Thank you for your assistance.

Asad

[elephaint]

Hi,

Happy to help - can you create a minimal example demonstrating the discrepancies?

[masadshoaib]

Hi elephaint!

Wanted to follow up on our discussion. Does the shared example suffice? Happy to share a screen recording of the entire thing. Sample data, code, and sample output.

copied from below:
Absolutely! (and thanks so much!)

I am using 60 datapoints as training dataset. Each unit represents a month of sales (Hence, the data spans last 5 years). I only have 3 columns, unique_id, ds, and y. I also have a test data of 12 points.

I train DilatedRNN and AutoDilatedRNN on this with same set of parameter values (as shown in the pdf attached earlier).

The forecasts output by each of the two above and errors (RMSE/MAPE/etc. relative to test dataset) I get are different from each other. (PS: if I dont specify AutoDRNN param values and let it run generally, the errors increase relative to the pre-tuned default performance!)

I believe this might be because the AutoModel also takes in 12 units as a validation set and hence it's training data is 60-12=48 months. So, I limit my dilatedRNN model's training data to 48 too. However, the forecasts produced by it and the Auto model do not match either.

Does this elucidate my point? Happy to share a screen recording with the process and results if that makes this easier!

[masadshoaib]

Absolutely! (and thanks so much!)

I am using 60 datapoints as training dataset. Each unit represents a month of sales (Hence, the data spans last 5 years). I only have 3 columns, unique_id, ds, and y. I also have a test data of 12 points.

I train DilatedRNN and AutoDilatedRNN on this with same set of parameter values (as shown in the pdf attached earlier).

The forecasts output by each of the two above and errors (RMSE/MAPE/etc. relative to test dataset) I get are different from each other. (PS: if I dont specify AutoDRNN param values and let it run generally, the errors increase relative to the pre-tuned default performance!)

I believe this might be because the AutoModel also takes in 12 units as a validation set and hence it's training data is 60-12=48 months. So, I limit my dilatedRNN model's training data to 48 too. However, the forecasts produced by it and the Auto model do not match either.

Does this elucidate my point? Happy to share a screen recording with the process and results if that makes this easier!

[elephaint]

Hi,

Sorry for not coming back earlier - can you share a minimal piece of standalone code that I can run that reproduces the issue?

[masadshoaib]

Not a problem. There is a difference between prediction from each[nf.predict and nf2.predict]. Here you go:

assuming you have the necessary imports.

Running the DilatedRNN

horizon = 12
max_steps = 1000 # based on dilated_rnn.py

models = [
DilatedRNN(h=horizon, max_steps=max_steps),
]
nf = NeuralForecast(models=models, freq='MS')
nf.fit(df=train_data)
nf.predict(horizon) # to be matched with the prediction from below

---

Running the Auto DilatedRNN with a single set of parameters (default values taken by DilatedRNN)

max_steps = 1000
dilatedRNN_config = {
'max_steps': max_steps,
"input_size": -1,
"inference_input_size": -1,
"h": None,
"cell_type": "LSTM",
"encoder_hidden_size": 200,
"dilations": [[1, 2], [4, 8]],
"context_size": 10,
"decoder_hidden_size": 200,
"decoder_layers": 2,
"futr_exog_list": None,
"hist_exog_list": None,
"stat_exog_list": None,
"learning_rate": 0.001,
"batch_size": 32,
"num_lr_decays": 3,
"early_stop_patience_steps": -1,
"val_check_steps": 100,
"batch_size": 32,
"valid_batch_size": 32,
"step_size": 1,
"scaler_type": "robust",
"random_seed": 1,
"num_workers_loader": 0,
"drop_last_loader": False,
"optimizer": None,
"optimizer_kwargs": None,
"lr_scheduler": None,
"lr_scheduler_kwargs": None,
'enable_checkpointing': False, 'val_check_interval': 100, 'check_val_every_n_epoch': None # Trainer kwargs
}

horizon = 12
num_samples = 1

nf2 = NeuralForecast(
models=[
AutoDilatedRNN(h=horizon,config = dilatedRNN_config, loss= MAE(), valid_loss = MAE(), num_samples = num_samples),
],
freq='MS'
)
nf2.fit(train_data, val_size = 12)
nf2.predict(horizon) # to be matched with prediction from above

[elephaint]

Thanks - I can't run this code as there is no dataset defined. Can you make the example standalone? (i.e. I should be able to copy and paste your code and it should run and lead to the issue you describe)

[masadshoaib]

sure let me share that with you in some time.

[masadshoaib]

Here you go. Please note that the predictions given by both in the final_df are different as shown by the diff column.

!pip install nixtla
!pip install NeuralForecast

from neuralforecast import NeuralForecast
from neuralforecast.auto import DilatedRNN, AutoDilatedRNN
from neuralforecast.losses.pytorch import RMSE, MAE
import pandas as pd

from typing import List, Tuple, Dict

from sklearn.metrics import mean_absolute_error, mean_squared_error

# Setting up the dataframes

train_data2 = pd.DataFrame({
    'unique_id': ['test-0'] * 60,  # 60 elements
    'ds': pd.date_range(start='2018-01-01', periods=60, freq='MS'),  # 60 elements
    'y': [63202.0,
 103224.5,
 111117.98,
 114329.0,
 108873.25,
 54203.02,
 86218.1202,
 93635.24,
 128048.0,
 124754.46,
 131659.00999999998,
 121901.49,
 89554.36009999999,
 138511.0,
 170329.0,
 143534.47999999998,
 77849.5001,
 76367.70999999998,
 88739.4099,
 122639.92,
 139260.0,
 158036.0001,
 159479.5,
 53163.895500000006,
 104816.511,
 161734.01,
 229404.3072,
 219401.8639,
 74706.5978,
 120536.01999999996,
 108064.7785,
 107818.415,
 153914.315,
 129599.932,
 117563.8949,
 62912.5835,
 174501.56300000002,
 191323.105,
 226644.8306,
 149521.4569,
 39708.3959,
 82081.96659999999,
 51392.1871,
 107311.5811,
 110006.954,
 121564.97430000002,
 112380.19569999998,
 89952.5831,
 142210.4084,
 172362.2267,
 163108.3762,
 118614.05010000002,
 51305.5352,
 84403.1251,
 110113.65610000002,
 128331.4126,
 123344.89500000002,
 145871.16280000002,
 118498.74099999998,
 81374.17670000001]
})


actual_data = pd.DataFrame({
    'unique_id': ['test-0'] * 12,
    'ds': pd.date_range(start='2023-01-01', periods=12, freq='MS'),
    'Actual_Qty': [177578.9375,
 263270.37490000005,
 130901.2323,
 46545.7235,
 79886.51950000001,
 57278.882,
 81313.90830000001,
 114359.66060000002,
 66363.0761,
 151170.48500000002,
 107519.225,
 114884.1]

})

# Default Dilated RNN without hyperparameter optimization
horizon = 12
max_steps = 1000

models = [
          DilatedRNN(h=horizon, max_steps=max_steps),
          ]
nf = NeuralForecast(models=models, freq='MS')

nf.fit(df=train_data2)

fcst_df = nf.predict()

# AutoDilated RNN with one set of hyperparameters, the same ones which are given to DilatedRNN by default.
max_steps = 1000
dilatedRNN_config = {
    'max_steps': max_steps,
    "input_size": -1,
    "inference_input_size": -1,
    "h": None,
    "cell_type": "LSTM",
    "encoder_hidden_size": 200,
    "dilations": [[1, 2], [4, 8]],
    "context_size": 10,
    "decoder_hidden_size": 200,
    "decoder_layers": 2,
    "futr_exog_list": None,
    "hist_exog_list": None,
    "stat_exog_list": None,
    "learning_rate": 0.001,
    "batch_size": 32,
    # "loss": MAE(), # needs to be defined in the init function below
    # "valid_loss": MAE(), # needs to be defined in the init function below
    "num_lr_decays": 3,
    "early_stop_patience_steps": -1,
    "val_check_steps": 100,
    "batch_size": 32,
    "valid_batch_size": 32,
    "step_size": 1,
    "scaler_type": "robust",
    "random_seed": 1,
    "num_workers_loader": 0,
    "drop_last_loader": False,
    "optimizer": None,
    "optimizer_kwargs": None,
    "lr_scheduler": None,
    "lr_scheduler_kwargs": None,
    'enable_checkpointing': False, 'val_check_interval': 100, 'check_val_every_n_epoch': None # Trainer kwargs
    }

num_samples = 25
n_series = 1 # used for multivariate

nf2 = NeuralForecast(
    models=[
        AutoDilatedRNN(h=horizon,config = dilatedRNN_config, loss= MAE(), valid_loss = MAE(), num_samples = 1),
    ],
    freq='MS'
)

nf2.fit(train_data2, val_size = 12)

fcst_df2 = nf2.predict()

# Merging the two forecasts as well as the actual data
final_df = fcst_df.merge(fcst_df2, on = ['ds', 'unique_id']).reset_index()
final_df['diff'] = final_df['DilatedRNN']-final_df['AutoDilatedRNN'] # to gauge the differences between the two predictions
final_df = final_df.merge(actual_data, on = ['unique_id', 'ds'])

print(mean_absolute_error(final_df['Actual_Qty'], final_df['DilatedRNN']))
print(mean_absolute_error(final_df['Actual_Qty'], final_df['AutoDilatedRNN']))
print(final_df)

[elephaint]

In the 'normal' DilatedRNN, you also have to train with a validation set (as does AutoDilatedRNN):

nf.fit(df=train_data2, val_size=12)

Making that change leads to identical results for both methods on my machine with your code.

Let me know if that solves the issue for you,

[masadshoaib]

It does! thank you so much!

[masadshoaib]

@elephaint another question: When I have train_data for two unique_ids, how to specify that to Auto_DilatedRNN?

My train_data, as shared above, has three columns, unique_id, ds, and y. But now instead of a single unique_id (test-0), I have two (test-0, test-1).

Do I need to specify unique_ids somehow when calling the Auto_model?

If I do hyper-param optimization on the train_data using Auto_DilatedRNN having single unique_id, I get an error lower than the non-hyper-optimized model (i.e., the regular DilatedRNN). And this is expected (and the goal of hyper-param optimization).

However, when I call the Auto_model (i.e., AutoDilatedRNN) on the train dataset having two unique_id, I end up with a higher error than if I just called the non-optimized model (i.e. regular DilatedRNN) for one unique_id and the opposite for the other.

Can you please guide me about this? Many thanks!

Here's the code snippet (should you need to run it):

Imports and Installs

!pip install nixtla
!pip install NeuralForecast

from neuralforecast import NeuralForecast
from neuralforecast.auto import DilatedRNN, AutoDilatedRNN
from neuralforecast.losses.pytorch import RMSE, MAE
import pandas as pd

from typing import List, Tuple, Dict

from sklearn.metrics import mean_absolute_error, mean_squared_error

Setting up the dataframes

train_data2 = pd.DataFrame({
'unique_id': ['test-0'] * 60 + ['test-1'] * 60, # 60 elements
'ds': pd.date_range(start='2018-01-01', periods=60, freq='MS').tolist() + pd.date_range(start='2018-01-01', periods=60, freq='MS').tolist(), # 60 elements each for 'test-0' and 'test-1'
'y': [63202.0, 103224.5, 111117.98, 114329.0, 108873.25, 54203.02, 86218.1202, 93635.24, 128048.0, 124754.46, 131659.00999999998, 121901.49, 89554.36009999999, 138511.0, 170329.0, 143534.47999999998, 77849.5001, 76367.70999999998, 88739.4099, 122639.92, 139260.0, 158036.0001, 159479.5, 53163.895500000006, 104816.511, 161734.01, 229404.3072, 219401.8639, 74706.5978, 120536.01999999996, 108064.7785, 107818.415, 153914.315, 129599.932, 117563.8949, 62912.5835, 174501.56300000002, 191323.105, 226644.8306, 149521.4569, 39708.3959, 82081.96659999999, 51392.1871, 107311.5811, 110006.954, 121564.97430000002, 112380.19569999998, 89952.5831, 142210.4084, 172362.2267, 163108.3762, 118614.05010000002, 51305.5352, 84403.1251, 110113.65610000002, 128331.4126, 123344.89500000002, 145871.16280000002, 118498.74099999998, 81374.17670000001, 48710.0, 50648.33, 62477.0, 63663.0, 53089.0, 32879.0, 29453.0, 27483.0, 57825.0, 71060.0, 59058.0, 55709.0, 41270.79, 66417.0, 81619.0, 108195.0001, 20043.0, 50103.0, 34913.0, 50942.0, 62446.0, 53002.0, 93837.0, 28171.0, 54429.0, 81507.9167, 100153.833, 122369.0, 41400.0, 54521.583, 36117.2503, 45895.25, 78801.25, 60997.25, 56257.5834, 47694.8333, 68566.12520000001, 78821.209, 125403.4159, 54204.4999, 10475.6249, 75008.7077, 41501.1667, 41225.0, 35159.75, 77580.9584, 55342.500100000005, 42327.5417, 88843.9168, 104285.0417, 50964.0001, 42008.3333, 72844.169, 73522.5421, 69892.83350000001, 82048.9583, 106956.3334, 105103.0416, 100294.2511, 60981.7502]
})

actual_data = pd.DataFrame({
'unique_id': ['test-0'] * 12 + ['test-1']*12,
'ds': pd.date_range(start='2023-01-01', periods=12, freq='MS').tolist() + pd.date_range(start='2023-01-01', periods=12, freq='MS').tolist(),
'Actual_Qty': [177578.9375, 263270.37490000005, 130901.2323, 46545.7235, 79886.51950000001, 57278.882, 81313.90830000001, 114359.66060000002, 66363.0761, 151170.48500000002, 107519.225, 114884.1, 214928.6671, 24430.334000000003, 77216.0906, 59655.919, 58646.174900000005, 66150.0943, 56996.2968, 85112.5022, 67122.20880000001, 65644.544, 85356.415, 84726.014]
})

Default Dilated RNN without hyperparameter optimization

horizon = 12
max_steps = 1000

models = [
DilatedRNN(h=horizon, max_steps=max_steps),
]
nf = NeuralForecast(models=models, freq='MS')

nf.fit(df=train_data2, val_size = 12)

fcst_df = nf.predict()
fcst_df = fcst_df.reset_index()

AutoDilated RNN (Hyper-optimization) with Optuna backend

nf4 = NeuralForecast(
models=[
AutoDilatedRNN(h=horizon, num_samples = 25, backend = 'optuna'),
],
freq='MS'
)

nf4.fit(train_data2, val_size = 12)

fcst_df_auto = nf4.predict()
fcst_df_auto = fcst_df_auto.reset_index()

Merging the two forecasts as well as the actual data

final_df3 = fcst_df_auto.merge(fcst_df, on = ['unique_id','ds'])
final_df3 = final_df3.merge(actual_data, on = ['unique_id', 'ds'])

Result

print('DilatedRNN RMSE')
for id in final_df3['unique_id'].unique():
subset = final_df3[final_df3['unique_id'] == id]
print(id, mean_squared_error(subset['Actual_Qty'], subset['DilatedRNN'])**0.5)

print('AutoDilatedRNN RMSE')
for id in final_df3['unique_id'].unique():
subset = final_df3[final_df3['unique_id'] == id]
print(id, mean_squared_error(subset['Actual_Qty'], subset['AutoDilatedRNN'])**0.5)