Multivariate - Beyond the Basics

This notebook shows multivariate forecasting procedures with scalecast. It requires 0.18.2. It uses the Avocados dataset.

We will treat this like a demand forecasting problem. We want to know how many total Avocados will be in demand in the next quarter. But since we know demand and price are intricately related, we will use the historical Avocado prices as a predictor of demand.

import pandas as pd
import numpy as np
from scalecast.Forecaster import Forecaster
from scalecast.MVForecaster import MVForecaster
from scalecast.Pipeline import MVPipeline
from scalecast.util import (
    find_optimal_transformation,
    find_optimal_lag_order,
    break_mv_forecaster,
    backtest_metrics,
    backtest_for_resid_matrix,
    get_backtest_resid_matrix,
    overwrite_forecast_intervals,
)
from scalecast import GridGenerator

Read in hyperparameter grids for optimizing models.

GridGenerator.get_example_grids()
GridGenerator.get_mv_grids()

pd.options.display.max_colwidth = 100
pd.options.display.float_format = '{:,.2f}'.format

# arguments to pass to every Forecaster/MVForecaster object we create
Forecaster_kws = dict(
    test_length = 13,
    validation_length = 13,
    metrics = ['rmse','r2'],
)

# model summary columns to export everytime we check a model's performance
export_cols = ['ModelNickname','HyperParams','TestSetR2','TestSetRMSE']

# read data
data = pd.read_csv('avocado.csv',parse_dates=['Date']).sort_values(['Date'])
# sort appropriately (not doing this could cause issues)
data = data.sort_values(['region','type','Date'])

data.head()

	Unnamed: 0	Date	AveragePrice	Total Volume	4046	4225	4770	Total Bags	Small Bags	Large Bags	XLarge Bags	type	year	region
51	51	2015-01-04	1.22	40,873.28	2,819.50	28,287.42	49.90	9,716.46	9,186.93	529.53	0.00	conventional	2015	Albany
50	50	2015-01-11	1.24	41,195.08	1,002.85	31,640.34	127.12	8,424.77	8,036.04	388.73	0.00	conventional	2015	Albany
49	49	2015-01-18	1.17	44,511.28	914.14	31,540.32	135.77	11,921.05	11,651.09	269.96	0.00	conventional	2015	Albany
48	48	2015-01-25	1.06	45,147.50	941.38	33,196.16	164.14	10,845.82	10,103.35	742.47	0.00	conventional	2015	Albany
47	47	2015-02-01	0.99	70,873.60	1,353.90	60,017.20	179.32	9,323.18	9,170.82	152.36	0.00	conventional	2015	Albany

# demand
vol = data.groupby('Date')['Total Volume'].sum()

# price
price = data.groupby('Date')['AveragePrice'].sum()

# one Forecaster object needed for each series we want to predict multivariately
# volume
fvol = Forecaster(
    y = vol,
    current_dates = vol.index,
    future_dates = 13,
    **Forecaster_kws,
)

# price
fprice = Forecaster(
    y = price,
    current_dates = price.index,
    future_dates = 13,
    **Forecaster_kws,
)

# combine Forecaster objects into one MVForecaster object
# all dates will line up and all models will recursively predict values for all series
mvf = MVForecaster(
    fvol,
    fprice,
    names=['volume','price'],
    **Forecaster_kws,
)

mvf

MVForecaster(
    DateStartActuals=2015-01-04T00:00:00.000000000
    DateEndActuals=2018-03-25T00:00:00.000000000
    Freq=W-SUN
    N_actuals=169
    N_series=2
    SeriesNames=['volume', 'price']
    ForecastLength=13
    Xvars=[]
    TestLength=13
    ValidationLength=13
    ValidationMetric=rmse
    ForecastsEvaluated=[]
    CILevel=None
    CurrentEstimator=mlr
    OptimizeOn=mean
    GridsFile=MVGrids
)

1. Transformations

To make the forecasting task easier, we can transform the data in each Forecaster object before feeding them to the MVForecaster object. The below function will search through many transformations, using out-of-sample testing to score each one. We pass four possible seasonalities to the function (monthly, quarterly, bi-annually, annually) and the results are several seasonal adjustments get selected.

transformers = []
reverters = []

for name, f in zip(('volume','price'),(fvol,fprice)):
    print(f'\nFinding best transformation for the {name} series.')
    transformer, reverter = find_optimal_transformation(
        f,
        m = [
            4,
            13,
            26,
            52,
        ],
        test_length = 13,
        num_test_sets = 2,
        space_between_sets = 13,
        return_train_only = True,
        verbose = True,
    )
    transformers.append(transformer)
    reverters.append(reverter)

Finding best transformation for the volume series.
Using mlr model to find the best transformation set on 2 test sets, each 13 in length.
All transformation tries will be evaluated with 4 lags.
Last transformer tried:
[]
Score (rmse): 19481972.64636622
--------------------------------------------------
Last transformer tried:
[('DetrendTransform', {'loess': True})]
Score (rmse): 22085767.144446835
--------------------------------------------------
Last transformer tried:
[('DetrendTransform', {'poly_order': 1})]
Score (rmse): 19630858.294620857
--------------------------------------------------
Last transformer tried:
[('DetrendTransform', {'poly_order': 2})]
Score (rmse): 22320325.279892858
--------------------------------------------------
Last transformer tried:
[('DeseasonTransform', {'m': 4, 'model': 'add'})]
Score (rmse): 18763298.437913556
--------------------------------------------------
Last transformer tried:
[('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'})]
Score (rmse): 18061445.02080934
--------------------------------------------------
Last transformer tried:
[('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'}), ('DeseasonTransform', {'m': 26, 'model': 'add'})]
Score (rmse): 18351627.623842016
--------------------------------------------------
Last transformer tried:
[('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'}), ('DeseasonTransform', {'m': 52, 'model': 'add'})]
Score (rmse): 15388459.611609437
--------------------------------------------------
Last transformer tried:
[('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'}), ('DeseasonTransform', {'m': 52, 'model': 'add'}), ('Transform', <function find_optimal_transformation.<locals>.boxcox_tr at 0x0000022788613D30>, {'lmbda': -0.5})]
Score (rmse): 15776741.170206662
--------------------------------------------------
Last transformer tried:
[('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'}), ('DeseasonTransform', {'m': 52, 'model': 'add'}), ('Transform', <function find_optimal_transformation.<locals>.boxcox_tr at 0x0000022788613D30>, {'lmbda': 0})]
Score (rmse): 15640424.466095788
--------------------------------------------------
Last transformer tried:
[('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'}), ('DeseasonTransform', {'m': 52, 'model': 'add'}), ('Transform', <function find_optimal_transformation.<locals>.boxcox_tr at 0x0000022788613D30>, {'lmbda': 0.5})]
Score (rmse): 15512957.889126703
--------------------------------------------------
Last transformer tried:
[('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'}), ('DeseasonTransform', {'m': 52, 'model': 'add'}), ('DiffTransform', 1)]
Score (rmse): 15929820.9564328
--------------------------------------------------
Last transformer tried:
[('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'}), ('DeseasonTransform', {'m': 52, 'model': 'add'}), ('DiffTransform', 4)]
Score (rmse): 14324958.982509937
--------------------------------------------------
Last transformer tried:
[('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'}), ('DeseasonTransform', {'m': 52, 'model': 'add'}), ('DiffTransform', 4), ('DiffTransform', 13)]
Score (rmse): 18135344.27502767
--------------------------------------------------
Last transformer tried:
[('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'}), ('DeseasonTransform', {'m': 52, 'model': 'add'}), ('DiffTransform', 4), ('DiffTransform', 26)]
Score (rmse): 21861866.629635938
--------------------------------------------------
Last transformer tried:
[('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'}), ('DeseasonTransform', {'m': 52, 'model': 'add'}), ('DiffTransform', 4), ('DiffTransform', 52)]
Score (rmse): 20808840.990807127
--------------------------------------------------
Last transformer tried:
[('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'}), ('DeseasonTransform', {'m': 52, 'model': 'add'}), ('DiffTransform', 4), ('ScaleTransform',)]
Score (rmse): 14324958.982509933
--------------------------------------------------
Last transformer tried:
[('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'}), ('DeseasonTransform', {'m': 52, 'model': 'add'}), ('DiffTransform', 4), ('MinMaxTransform',)]
Score (rmse): 14324958.98250994
--------------------------------------------------
Final Selection:
[('DeseasonTransform', {'m': 4, 'model': 'add', 'train_only': True}), ('DeseasonTransform', {'m': 13, 'model': 'add', 'train_only': True}), ('DeseasonTransform', {'m': 52, 'model': 'add', 'train_only': True}), ('DiffTransform', 4), ('ScaleTransform', {'train_only': True})]

Finding best transformation for the price series.
Using mlr model to find the best transformation set on 2 test sets, each 13 in length.
All transformation tries will be evaluated with 4 lags.
Last transformer tried:
[]
Score (rmse): 22.25551611050048
--------------------------------------------------
Last transformer tried:
[('DetrendTransform', {'loess': True})]
Score (rmse): 25.65997061765327
--------------------------------------------------
Last transformer tried:
[('DetrendTransform', {'poly_order': 1})]
Score (rmse): 22.148856499520484
--------------------------------------------------
Last transformer tried:
[('DetrendTransform', {'poly_order': 2})]
Score (rmse): 32.75467733406476
--------------------------------------------------
Last transformer tried:
[('DetrendTransform', {'poly_order': 1}), ('DeseasonTransform', {'m': 4, 'model': 'add'})]
Score (rmse): 21.72760152488739
--------------------------------------------------
Last transformer tried:
[('DetrendTransform', {'poly_order': 1}), ('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'})]
Score (rmse): 20.055641074156764
--------------------------------------------------
Last transformer tried:
[('DetrendTransform', {'poly_order': 1}), ('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'}), ('DeseasonTransform', {'m': 26, 'model': 'add'})]
Score (rmse): 22.020127438895862
--------------------------------------------------
Last transformer tried:
[('DetrendTransform', {'poly_order': 1}), ('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'}), ('DeseasonTransform', {'m': 52, 'model': 'add'})]
Score (rmse): 14.604251739058533
--------------------------------------------------
Last transformer tried:
[('DetrendTransform', {'poly_order': 1}), ('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'}), ('DeseasonTransform', {'m': 52, 'model': 'add'}), ('DiffTransform', 1)]
Score (rmse): 18.183007629056675
--------------------------------------------------
Last transformer tried:
[('DetrendTransform', {'poly_order': 1}), ('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'}), ('DeseasonTransform', {'m': 52, 'model': 'add'}), ('DiffTransform', 4)]
Score (rmse): 15.96916031713575
--------------------------------------------------
Last transformer tried:
[('DetrendTransform', {'poly_order': 1}), ('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'}), ('DeseasonTransform', {'m': 52, 'model': 'add'}), ('DiffTransform', 13)]
Score (rmse): 18.4021660531495
--------------------------------------------------
Last transformer tried:
[('DetrendTransform', {'poly_order': 1}), ('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'}), ('DeseasonTransform', {'m': 52, 'model': 'add'}), ('DiffTransform', 26)]
Score (rmse): 25.298723431620186
--------------------------------------------------
Last transformer tried:
[('DetrendTransform', {'poly_order': 1}), ('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'}), ('DeseasonTransform', {'m': 52, 'model': 'add'}), ('DiffTransform', 52)]
Score (rmse): 19.452999810002588
--------------------------------------------------
Last transformer tried:
[('DetrendTransform', {'poly_order': 1}), ('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'}), ('DeseasonTransform', {'m': 52, 'model': 'add'}), ('ScaleTransform',)]
Score (rmse): 14.604251739058554
--------------------------------------------------
Last transformer tried:
[('DetrendTransform', {'poly_order': 1}), ('DeseasonTransform', {'m': 4, 'model': 'add'}), ('DeseasonTransform', {'m': 13, 'model': 'add'}), ('DeseasonTransform', {'m': 52, 'model': 'add'}), ('MinMaxTransform',)]
Score (rmse): 14.604251739058554
--------------------------------------------------
Final Selection:
[('DetrendTransform', {'poly_order': 1, 'train_only': True}), ('DeseasonTransform', {'m': 4, 'model': 'add', 'train_only': True}), ('DeseasonTransform', {'m': 13, 'model': 'add', 'train_only': True}), ('DeseasonTransform', {'m': 52, 'model': 'add', 'train_only': True})]

Plot the series after a transformation has been taken.

fvol1 = transformers[0].fit_transform(fvol)
fvol1.plot();

fprice1 = transformers[1].fit_transform(fprice)
fprice1.plot();

Now, combine into an MVForecaster object.

mvf1 = MVForecaster(
    fvol1,
    fprice1,
    names = ['volume','price'],
    **Forecaster_kws,
)

2. Optimal Lag Selection

Method 1: Univariate out-of-sample testing

• The functions below choose the best lags based on what minimizes RMSE on an out-of-sample validation set.

fvol1.auto_Xvar_select(try_trend=False,try_seasonalities=False)
fvol1.get_regressor_names()

['AR1',
 'AR2',
 'AR3',
 'AR4',
 'AR5',
 'AR6',
 'AR7',
 'AR8',
 'AR9',
 'AR10',
 'AR11',
 'AR12',
 'AR13']

fprice1.auto_Xvar_select(try_trend=False,try_seasonalities=False)
fprice1.get_regressor_names()

['AR1', 'AR2', 'AR3', 'AR4', 'AR5', 'AR6', 'AR7', 'AR8', 'AR9', 'AR10', 'AR11']

Method 2: Information Criteria Search with VAR

lag_order_res = find_optimal_lag_order(mvf1,train_only=True)
lag_orders = pd.DataFrame({
    'aic':[lag_order_res.aic],
    'bic':[lag_order_res.bic],
})

lag_orders

	aic	bic
0	12	4

Method 3: Multivariate Cross Validation with MLR

lags = [
    1,
    2,
    3,
    4,
    9,
    10,
    11,
    12,
    13,
    {'volume':13,'price':9},
    [4,9,12,13],
]

grid = dict(
    lags = lags
)

mvf1.set_optimize_on('volume')

mvf1.ingest_grid(grid)
mvf1.cross_validate(k=3,test_length=13,verbose = True,dynamic_tuning=True)

Num hyperparams to try for the mlr model: 11.
Fold 0: Train size: 139 (2015-02-01 00:00:00 - 2017-09-24 00:00:00). Test Size: 13 (2017-10-01 00:00:00 - 2017-12-24 00:00:00).
Fold 1: Train size: 126 (2015-02-01 00:00:00 - 2017-06-25 00:00:00). Test Size: 13 (2017-07-02 00:00:00 - 2017-09-24 00:00:00).
Fold 2: Train size: 113 (2015-02-01 00:00:00 - 2017-03-26 00:00:00). Test Size: 13 (2017-04-02 00:00:00 - 2017-06-25 00:00:00).
Chosen paramaters: {'lags': 10}.

3. Model Optimization with Cross Validation

def forecaster(mvf):
    mvf.tune_test_forecast(
        ['lasso','ridge','xgboost','lightgbm'],
        cross_validate = True,
        k = 3,
        test_length = 13,
        dynamic_tuning=True,
        limit_grid_size=.2,
        min_grid_size=4,
    )

forecaster(mvf1)

mvf1.plot(series='volume');

mvf1.export('model_summaries',series='volume')[['Series'] + export_cols + ['Lags']].style.set_properties(height = 5)

	Series	ModelNickname	HyperParams	TestSetR2	TestSetRMSE	Lags
0	volume	lasso	{'alpha': 0.02}	-0.089158	1.399202	10
1	volume	ridge	{'alpha': 0.04}	-0.050000	1.373819	10
2	volume	xgboost	{'n_estimators': 250, 'scale_pos_weight': 5, 'learning_rate': 0.2, 'gamma': 3, 'subsample': 0.8}	-0.017800	1.352589	10
3	volume	lightgbm	{'n_estimators': 250, 'boosting_type': 'goss', 'max_depth': 2, 'learning_rate': 0.01}	-0.183962	1.458827	[4, 9, 12, 13]

mvf1

MVForecaster(
    DateStartActuals=2015-02-01T00:00:00.000000000
    DateEndActuals=2018-03-25T00:00:00.000000000
    Freq=W-SUN
    N_actuals=165
    N_series=2
    SeriesNames=['volume', 'price']
    ForecastLength=13
    Xvars=[]
    TestLength=13
    ValidationLength=13
    ValidationMetric=rmse
    ForecastsEvaluated=['lasso', 'ridge', 'xgboost', 'lightgbm']
    CILevel=None
    CurrentEstimator=lightgbm
    OptimizeOn=volume
    GridsFile=MVGrids
)

fvol1, fprice1 = break_mv_forecaster(mvf1)

reverter = reverters[0]
fvol1 = reverter.fit_transform(fvol1)

fvol1.plot();

fvol1.export('model_summaries')[export_cols].style.set_properties(height = 5)

	ModelNickname	HyperParams	TestSetR2	TestSetRMSE
0	lasso	{'alpha': 0.02}	0.388886	13185802.986461
1	ridge	{'alpha': 0.04}	0.249991	14607592.004013
2	xgboost	{'n_estimators': 250, 'scale_pos_weight': 5, 'learning_rate': 0.2, 'gamma': 3, 'subsample': 0.8}	0.455020	12451896.764192
3	lightgbm	{'n_estimators': 250, 'boosting_type': 'goss', 'max_depth': 2, 'learning_rate': 0.01}	0.281993	14292550.306560

4. Model Stacking

def model_stack(mvf,train_only=False):
    mvf.add_signals(['lasso','ridge','lightgbm','xgboost'],train_only=train_only)
    mvf.set_estimator('catboost')
    mvf.manual_forecast(
        lags = 13,
        verbose = False,
    )

model_stack(mvf1,train_only=True)

mvf1.plot(series='volume');

mvf1.export('model_summaries',series='volume')[['Series'] + export_cols + ['Lags']].style.set_properties(height = 5)

	Series	ModelNickname	HyperParams	TestSetR2	TestSetRMSE	Lags
0	volume	lasso	{'alpha': 0.02}	-0.089158	1.399202	10
1	volume	ridge	{'alpha': 0.04}	-0.050000	1.373819	10
2	volume	xgboost	{'n_estimators': 250, 'scale_pos_weight': 5, 'learning_rate': 0.2, 'gamma': 3, 'subsample': 0.8}	-0.017800	1.352589	10
3	volume	lightgbm	{'n_estimators': 250, 'boosting_type': 'goss', 'max_depth': 2, 'learning_rate': 0.01}	-0.183962	1.458827	[4, 9, 12, 13]
4	volume	catboost	{'verbose': False}	0.069340	1.293392	13

fvol1, fprice1 = break_mv_forecaster(mvf1)

fvol1 = reverter.fit_transform(fvol1)

fvol1.export('model_summaries',determine_best_by='TestSetRMSE')[export_cols].style.set_properties(height = 5)

	ModelNickname	HyperParams	TestSetR2	TestSetRMSE
0	xgboost	{'n_estimators': 250, 'scale_pos_weight': 5, 'learning_rate': 0.2, 'gamma': 3, 'subsample': 0.8}	0.455020	12451896.764192
1	catboost	{'verbose': False}	0.447398	12538678.460293
2	lasso	{'alpha': 0.02}	0.388886	13185802.986461
3	lightgbm	{'n_estimators': 250, 'boosting_type': 'goss', 'max_depth': 2, 'learning_rate': 0.01}	0.281993	14292550.306560
4	ridge	{'alpha': 0.04}	0.249991	14607592.004013

fvol1.plot_test_set(order_by='TestSetRMSE');

fvol1.plot(order_by='TestSetRMSE');

5. Multivariate Pipelines

def mvforecaster(mvf,train_only=False):
    forecaster(mvf)
    model_stack(mvf,train_only=train_only)

pipeline = MVPipeline(
    steps = [
        ('Transform',transformers),
        ('Forecast',mvforecaster),
        ('Revert',reverters),
    ],
    **Forecaster_kws,
)

fvol1, fprice1 = pipeline.fit_predict(fvol,fprice,train_only=True)

fvol1.plot_test_set(order_by='TestSetRMSE');

fvol1.plot(order_by='TestSetRMSE');

fvol1.export('model_summaries',determine_best_by='TestSetRMSE')[export_cols].style.set_properties(height = 5)

	ModelNickname	HyperParams	TestSetR2	TestSetRMSE
0	lightgbm	{'n_estimators': 150, 'boosting_type': 'dart', 'max_depth': 1, 'learning_rate': 0.1}	0.511729	11786259.617968
1	lasso	{'alpha': 0.53}	0.440399	12617822.749627
2	ridge	{'alpha': 1.0}	0.433704	12693080.386907
3	catboost	{'verbose': False}	0.384513	13232893.075803
4	xgboost	{'n_estimators': 250, 'scale_pos_weight': 10, 'learning_rate': 0.2, 'gamma': 0, 'subsample': 0.8}	0.381760	13262451.219228

6. Backtesting

backtest_results = pipeline.backtest(
    fvol,
    fprice,
    n_iter = 4,
    fcst_length = 13,
    test_length = 0,
    jump_back = 13,
)

backtest_metrics(
    backtest_results[:1], # volume only
    mets=['rmse','mae','r2','bias'],
    #names=['volume','price'],
)

		Iter0	Iter1	Iter2	Iter3	Average
Model	Metric
lasso	rmse	12,676,035.50	19,334,820.60	12,408,199.56	6,865,555.60	12,821,152.81
	mae	10,622,259.32	16,818,913.78	10,196,006.95	4,890,701.26	10,631,970.33
	r2	0.44	-2.98	-0.17	0.34	-0.59
	bias	-103,321,272.45	-216,135,991.11	106,453,214.29	7,242,071.64	-51,440,494.41
ridge	rmse	13,245,785.08	19,757,231.61	12,581,587.51	8,092,421.06	13,419,256.32
	mae	10,864,770.69	17,175,778.82	10,362,478.27	6,239,668.77	11,160,674.14
	r2	0.38	-3.16	-0.20	0.09	-0.72
	bias	-119,334,285.17	-221,927,247.43	109,823,042.50	55,636,823.14	-43,950,416.74
xgboost	rmse	19,261,511.73	15,233,136.06	15,781,395.08	6,583,385.75	14,214,857.16
	mae	15,981,374.97	13,767,893.53	13,479,663.34	5,216,355.57	12,111,321.85
	r2	-0.30	-1.47	-0.89	0.40	-0.57
	bias	-103,418,980.41	-151,259,604.70	155,235,475.90	-16,515,829.46	-28,989,734.67
lightgbm	rmse	11,239,291.40	17,262,898.64	14,840,433.88	7,289,722.74	12,658,086.67
	mae	9,087,987.10	15,146,134.37	12,373,711.83	5,735,222.10	10,585,763.85
	r2	0.56	-2.17	-0.67	0.26	-0.51
	bias	-86,731,196.07	-189,464,392.96	140,926,488.55	43,025,640.10	-23,060,865.10
catboost	rmse	17,455,804.71	14,955,271.67	16,116,336.26	6,315,491.61	13,710,726.06
	mae	14,805,029.65	13,567,739.76	13,603,593.10	5,036,532.77	11,753,223.82
	r2	-0.07	-1.38	-0.97	0.45	-0.50
	bias	-108,026,362.31	-146,926,722.77	162,948,970.47	24,860,226.70	-16,785,971.98

7. Dynamic Intervals

backtest_results = backtest_for_resid_matrix(
    fvol,
    fprice,
    pipeline = pipeline,
    alpha = 0.1, # 90% intervals
)

backtest_resid_matrix = get_backtest_resid_matrix(backtest_results)

overwrite_forecast_intervals(
    fvol1,
    fprice1,
    backtest_resid_matrix=backtest_resid_matrix,
    alpha=0.1,
)

fvol1.plot(models='top_1',order_by='TestSetRMSE',ci=True);

8. LSTM Modeling

fvol1 = transformers[0].fit_transform(fvol1)
fprice1 = transformers[1].fit_transform(fprice1)

fvol1.add_ar_terms(13)

fvol1.set_estimator('rnn')
fvol1.tune()
fvol1.auto_forecast(call_me='lstm_uv')

fvol1.add_series(fprice1.y,called='price')
fvol1.add_lagged_terms('price',lags=13,drop=True)
fvol1

Forecaster(
    DateStartActuals=2015-02-01T00:00:00.000000000
    DateEndActuals=2018-03-25T00:00:00.000000000
    Freq=W-SUN
    N_actuals=165
    ForecastLength=13
    Xvars=['AR1', 'AR2', 'AR3', 'AR4', 'AR5', 'AR6', 'AR7', 'AR8', 'AR9', 'AR10', 'AR11', 'AR12', 'AR13', 'pricelag_1', 'pricelag_2', 'pricelag_3', 'pricelag_4', 'pricelag_5', 'pricelag_6', 'pricelag_7', 'pricelag_8', 'pricelag_9', 'pricelag_10', 'pricelag_11', 'pricelag_12', 'pricelag_13']
    TestLength=13
    ValidationMetric=rmse
    ForecastsEvaluated=['lasso', 'ridge', 'xgboost', 'lightgbm', 'catboost', 'lstm_uv']
    CILevel=None
    CurrentEstimator=rnn
    GridsFile=Grids
)

fvol1.tune()
fvol1.auto_forecast(call_me='lstm_mv')

fvol1.plot_test_set(models=['lstm_uv','lstm_mv']);

fvol1.plot(models=['lstm_uv','lstm_mv']);

fvol1 = reverters[0].fit_transform(fvol1,exclude_models=['lightgbm','lasso','ridge','xgboost','catboost'])
fprice1 = reverters[1].fit_transform(fprice1,exclude_models=['lightgbm','lasso','ridge','xgboost','catboost'])

ms = fvol1.export('model_summaries')
ms = ms[export_cols]
ms.style.set_properties(height = 5)

	ModelNickname	HyperParams	TestSetR2	TestSetRMSE
0	lasso	{'alpha': 0.53}	0.440399	12617822.749627
1	ridge	{'alpha': 1.0}	0.433704	12693080.386907
2	xgboost	{'n_estimators': 250, 'scale_pos_weight': 10, 'learning_rate': 0.2, 'gamma': 0, 'subsample': 0.8}	0.381760	13262451.219228
3	lightgbm	{'n_estimators': 150, 'boosting_type': 'dart', 'max_depth': 1, 'learning_rate': 0.1}	0.511729	11786259.617968
4	catboost	{'verbose': False}	0.384513	13232893.075803
5	lstm_uv	{'layers_struct': [('LSTM', {'units': 50, 'activation': 'tanh', 'dropout': 0.2, 'return_sequences': True}), ('LSTM', {'units': 50, 'activation': 'tanh', 'dropout': 0.2, 'return_sequences': True}), ('LSTM', {'units': 50, 'activation': 'tanh', 'dropout': 0.2, 'return_sequences': False})], 'epochs': 50, 'verbose': 0}	0.449703	12512491.086281
6	lstm_mv	{'layers_struct': [('LSTM', {'units': 50, 'activation': 'tanh', 'return_sequences': True}), ('LSTM', {'units': 50, 'activation': 'tanh', 'return_sequences': True}), ('LSTM', {'units': 50, 'activation': 'tanh', 'return_sequences': False})], 'epochs': 50, 'verbose': 0}	0.514143	11757086.227344

fvol1.plot_test_set(order_by = 'TestSetRMSE');

fvol1.plot(order_by = 'TestSetRMSE');

9. Benchmarking against Naive Model

fvol1 = transformers[0].fit_transform(fvol1)
fvol1.set_estimator('naive')
fvol1.manual_forecast()
fvol1 = reverters[0].fit_transform(fvol1,exclude_models=['lightgbm','lasso','ridge','xgboost','catboost','lstm_uv','lstm_mv'])

ms = fvol1.export('model_summaries',determine_best_by='TestSetRMSE')
ms = ms[export_cols]
ms.style.set_properties(height = 5)

	ModelNickname	HyperParams	TestSetR2	TestSetRMSE
0	lstm_mv	{'layers_struct': [('LSTM', {'units': 50, 'activation': 'tanh', 'return_sequences': True}), ('LSTM', {'units': 50, 'activation': 'tanh', 'return_sequences': True}), ('LSTM', {'units': 50, 'activation': 'tanh', 'return_sequences': False})], 'epochs': 50, 'verbose': 0}	0.514143	11757086.227344
1	lightgbm	{'n_estimators': 150, 'boosting_type': 'dart', 'max_depth': 1, 'learning_rate': 0.1}	0.511729	11786259.617968
2	lstm_uv	{'layers_struct': [('LSTM', {'units': 50, 'activation': 'tanh', 'dropout': 0.2, 'return_sequences': True}), ('LSTM', {'units': 50, 'activation': 'tanh', 'dropout': 0.2, 'return_sequences': True}), ('LSTM', {'units': 50, 'activation': 'tanh', 'dropout': 0.2, 'return_sequences': False})], 'epochs': 50, 'verbose': 0}	0.449703	12512491.086281
3	lasso	{'alpha': 0.53}	0.440399	12617822.749627
4	ridge	{'alpha': 1.0}	0.433704	12693080.386907
5	catboost	{'verbose': False}	0.384513	13232893.075803
6	xgboost	{'n_estimators': 250, 'scale_pos_weight': 10, 'learning_rate': 0.2, 'gamma': 0, 'subsample': 0.8}	0.381760	13262451.219228
7	naive	{}	-0.188041	18384892.054096

fvol1.plot_test_set(order_by = 'TestSetRMSE');

fvol1.plot(order_by = 'TestSetRMSE');