NBEATSx forecasting horizon above 20?

[kostas-nikolaou]

Hi, sorry if there's something obvious I'm missing out, I'm altering the NBEATSx example code for some custom data, but when the forecast horizon is h>20, I get the following error in predict:

futr_df must have one row per id and ds in the forecasting horizon (21)

Anyone knows what's happening? Here is my code:

import numpy as np
import pandas as pd
import pytorch_lightning as pl
import matplotlib.pyplot as plt
from neuralforecast import NeuralForecast
from neuralforecast.models import NBEATSx
from neuralforecast.losses.pytorch import MQLoss, DistributionLoss
from neuralforecast.tsdataset import TimeSeriesDataset
from datetime import datetime
import yfinance as yf

msft = yf.download('MSFT',start=datetime(2010,1,1))
aapl=   yf.download('AAPL', start=datetime(2010,1,1))
msft=msft.rename(columns={'Adj Close':'MSFT'})
aapl=aapl.rename(columns={'Adj Close':'AAPL'})
total=pd.concat([msft,aapl],axis=1)
total=total.reset_index()
total=total.drop(['Open','High','Low','Close',"Volume"],axis=1)
total['unique_id']=0.0
total=total.rename(columns={'MSFT':'y','Date':'ds'})
total=total[['unique_id','ds','y','AAPL']]
a=21
Y_train_df= total[total.ds<total['ds'].values[-a]]
Y_test_df=  total[total.ds>=total['ds'].values[-a]].reset_index(drop=True)
model = NBEATSx(h=a, input_size=2,
                loss=MQLoss(level=[80, 90]),
                #loss=DistributionLoss(distribution='Normal', level=[80, 90]),
                scaler_type='robust',
                dropout_prob_theta=0.5,
                hist_exog_list=['AAPL'],
                futr_exog_list=['AAPL'],
                max_steps=100,
                val_check_steps=40,
                early_stop_patience_steps=4)

nf = NeuralForecast(
    models=[model],
    freq='B'
)
nf.fit(df=Y_train_df, val_size=25)
Y_hat_df = nf.predict(futr_df=Y_test_df,verbose=True)

Thanks in advance

[jmoralez]

Hey @kostas-nikolaou, thanks for using neuralforecast. Can you provide which version you're using? Your example runs fine for me.

[kostas-nikolaou]

According to pip, neuralforecast-1.6.3 ray-2.7.0

[kostas-nikolaou]

So, when I tried today it worked fine the first time, but when increasing the horizon above 21 I got the same type of error, even when resetting the runtime.

[jmoralez]

It seems that the data doesn't have 2023-09-04, which was a Monday so it's being produced as a business day in neuralforecast, but since it's not in the futr_df you get that error. If you add that date it should be fine, for example you can use the value of Sep 1st (kind of a backfill)
total = pd.concat([total, total[total['ds'] == '2023-09-01'].assign(ds=pd.to_datetime('2023-09-04'))])

[kmizuki]

@jmoralez
Hello. For this problem, it can be resolved by duplicating the data from the previous day, but if there is a desire to avoid training on days with missing values, are there other workarounds, such as changing the business days recognized by neuralforecast? In my data, errors occur at least on the following dates.

2022-12-26
2023-01-02
2023-01-16
2023-02-20
2023-04-07
2023-05-29
2023-06-19
2023-07-04
2023-09-04
2023-11-23

[jmoralez]

Hey. This shouldn't be a problem for training, only for predicting. We recently added a helper method (make_future_dataframe) to produce the expected future dates, you can install from github to use it, that should help with the predict step. The dates are just a convenience, they aren't used for training nor forecasting, so you can just replace your values in the expected dates.