Hyperparameter tuning using adaptive bayesian contextual hyperband for time series forecasting

Abstract

In Machine learning, hyperparameters are all those parameters the user explicitly defines to enhance a model s learning. Although they are defined outside the model, hyperparameters significantly impact the model s behavior and performance. Tuning these hyperparameters plays a significant role in improving the model performance. Any tuning algorithm aims to efficiently navigate the hyperparameter search space to find the optimal configuration in minimal time. Different tuning algorithms proposed in the recent decade have gained popularity due to increased computational power. Random, grid search, algorithms involving Bayesian rule, bandit-based solutions, and evolutionary algorithms are some of the popularly used tuning algorithms. The commonality in these algorithms is finding a good trade-off between exploration versus exploitation of the search space, given the limited budget for tuning. With the same goal, this work proposes a hyperparameter tuning algorithm called Bayesian Contextual Hyperband (BCHB), incorporating Bayesian and multi-armed bandit techniques. The algorithm divides the total budget, the number of iterations or time allotted to each hyperparameter configuration, into different brackets that start with more exploration of the search space and end with the exploitation of the selected best configurations. Each bracket is further divided into iterations that select and evaluate a fixed number of configurations on an objective function. True objective function evaluation is an expensive step, and hence in the proposed algorithm, it is substituted by a surrogate model of hyperparameter configurations and their corresponding validation losses obtained from the history of past evaluations. newline