A New Two-Stage Approach with Boosting and Model Averaging for Interval-Valued Crude Oil Prices Forecasting in Uncertainty Environments

In view of the intrinsic complexity of the oil market, crude oil prices are influenced by numerous factors that make forecasting very difficult. Recognizing this challenge, numerous approaches have been introduced, but little work has been done concerning the interval-valued prices. To capture the underlying characteristics of crude oil price movements, this paper proposes a two-stage forecasting procedure to forecast interval-valued time series, which generalizes point-valued forecasts to incorporate uncertainty and variability. The empirical results show that our proposed approach significantly outperforms all the benchmark models in terms of both forecasting accuracy and robustness analysis. These results can provide references for decision-makers to understand the trends of crude oil prices and improve the efficiency of economic activities.


INTRODUCTION
As one of the most important commodities, crude oil plays a vital role in various fields. In the past decades, crude oil prices have been extremely volatile (see Figure 1). The oil-related industries are highly sensitive to oil price changes (Ebrahim et al., 2014;Taghizadeh-Hesary et al., 2016). Accurate prediction of crude oil prices and the market volatility is valuable for market participants to make risk management plans and investment decisions (Zaabouti et al., 2016;Zhang et al., 2020). The crude oil prices are volatile, and are dependent on many factors such as market trends, sentiments and stock markets. The aforementioned factors make the crude oil prices unstable and makes its prediction complicated and challenging. Thus, we aim to develop a reliable model for crude oil price forecasting.
In recent literatures, most of the existing methods focus on the point-valued crude oil closing prices (Abramson and Finizza, 1995;Zhang et al., 2008;Kilian, 2009;Zhang et al., 2009;Shin et al., 2013;Zhao et al., 2017;Binder et al., 2018;Álvarez-Díaz, 2019). However, the use of closing prices has the disadvantage that it does not take into account the oil price variation information within a given period time, e.g., the midpoint and range of crude oil prices in October 2008 are about $76.61/bbl and $36.31/bbl respectively. While the midpoint and range of crude oil prices in November 2009 are around $77.99/bbl and $5.42/bbl respectively. Such forecasts with point-valued crude oil price data have not been particularly successful when compared with the intervalvalued time series forecasts (see Sun et al., 2019). What is more, recent studies also provide empirical evidence suggesting that ITS models have achieved great success on improving the forecast accuracy in a wide range of fields such as stock price forecasting (Maia and de Carvalho, 2011;Xiong et al., 2017) and forecasting in energy markets, such as electric power demand (García-Ascanio and Maté, 2010;Hu et al., 2015), and crude oil prices (Yang et al., 2016). By accessing more information (e.g., highs, lows, midpoints, and range), an interval-based method is expected to be superior to the point-based method (Sun et al., 2018). Here, highs and lows are points of inflection for prices. The price range is the difference between two boundaries, which gives the interval length. It can be regarded as a measure of volatility to reflect the price fluctuation. For example, instead of traditional point-based method, Yang et al. (2012) introduce interval dummy variables in the autoregressive conditional interval models. Sun et al. (2019) apply a threshold autoregressive interval-valued model. Qiao et al. (2019) develop an intervalvalued factor pricing model. Conclusions from prior studies suggest that interval-valued time series (ITS) models may produce more accurate forecasts. Therefore, the desirable characteristics of the interval modeling make them ideal candidates for the prediction of crude oil prices. In addition, it is well known that a large set of factors are responsible for changes in the crude oil price, including overall economic conditions, demand and supply, monetary policy, as well as speculative trading (Hamilton, 2008;Yoshino and Taghizadeh-Hesary, 2014). Thus, the number of potential predictors can be very large. In such cases, interval-valued variable selection is considered necessary and becomes the critical step in achieving promising forecasting performances in data-rich environments. On the other hand, in practice, when only some of the variables are selected to include as the predictors in a model, model misspecification is unavoidable, which can worsen the model forecast performance of the model. Therefore, model averaging is considered to take a weighted average of possible combinations of selected interval-valued predictors.
For these reasons, this paper proposes a new two-stage procedure for interval valued crude oil price forecasting based on boosting and model averaging. First, we extend the L 2 boosting method by Buhlmann (2006) to achieve variable selection for the interval model. Several penalized methods have been proposed to achieve variable selection. Examples include the class of Bridge estimators (Frank and Friedman, 1993), where the Lasso-type estimators are included a special case (Knight and Fu, 2000), or the smoothly clipped absolute deviation (SCAD) estimator (Fan and Li, 2001). Instead of these regularized (penalized) methods, Donald et al. (2009) apply information criteria for moment selection, Ng and Bai (2009) develop boosting for variable selection, where variable selection and shrinkage are performed simultaneously to increase prediction accuracy. The proposed vector boosting algorithm can achieve significant dimension reduction when a long list of interval-valued variables is available.
Next, we extend the LsoMA method developed by Liao et al. (2019) to average predictions from interval models with intervalvalued exogenous variables to reduce model uncertainty. The idea of model averaging (MA) is first introduced to combine predictions from many forecasting models by Bates and Granger (1969) and has received great interest in econometrics and statistics. Model averaging is an extension of model selection which can substantially reduce the selection bias induced by selecting only one candidate model. Hoeting et al. (1999) provide a comprehensive summary of previous research on Bayesian model averaging (BMA) where models are weighted by the posterior model probabilities. Unlike BMA, frequentist model averaging (FMA) usually select the optimal weighting with the smallest information criteria scores (Buckland et al., 1997;Hjort and Claeskens, 2003;Hjort and Claeskens, 2006;Zhang and Liang, 2011;Zhang et al., 2012;Xu et al., 2014), Mallows model averaging (MMA) by Hansen (2007), jackknife model averaging (JMA) by Hansen and Racine (2012). Liao and Tsay (2016) extend MMA to the situation of the VAR models.
Univariate and bivariate methods are broadly the two main approaches in the interval modeling literature. In the univariate method, models are presented separately for a pair of attributes of interval variables (e.g., midpoint and range). The two attributes are estimated separately (De Carvalho et al., 2004;Maia et al., 2008), thus only information of one attribute is used in estimating model parameters at a time. Unlike the univariate method, the bivariate method estimates the two attributes simultaneously (e.g., Cheung et al., 2009;He et al., 2010;Lima Neto and De Carvalho, 2010;Arroyo et al., 2011;González-Rivera and Lin, 2013), which is more desirable in ITS forecasting. Therefore, in this paper, in order to consider possible interdependence between midpoint and range, the LsoMA methods are constructed following the bivariate modeling approach to efficiently use the contained information.
This paper proposes a two-stage vector boosting model averaging (2SVBMA) forecasting framework: Stage 1 uses vector L 2 Boosting to select interval-valued variables; Stage 2 uses the leave-subject-out cross-validation model averaging method with exogenous interval-valued variables to average interval-valued predictions. Our procedure combines the merits of these two techniques and can be easily adapted to any new situation. We compare our 2SVBMA method with other competing methods including model selection methods by Akaike information criterion (AIC), Bayesian information criterion (BIC), Hannan-Quinn (HQ), and model averaging methods by smoothed AIC, smoothed BIC (Buckland et al., 1997), smoothed HQ, and MMA in interval model. The empirical results indicate that the 2SVBMA method has better forecasting performance than the commonly used model selection and averaging methods.
Our proposed 2SVBMA forecasting procedure has a few appealing features. First, this approach extends the forecasting success of point-valued data models of crude oil price to intervalvalued data models, which is capable of assessing and forecasting the changes in both the trend and volatility of crude oil prices simultaneously due to the informational gain from intervalvalued data. Second, our vector boosting method provides a parsimony and feasible solution to the interval-valued variable selection problem for interval models. Third, the extended interval-valued LsoMA model with interval-valued exogenous variables demonstrates the gains in forecast accuracy through forecast combination. By doing so, our approach improves crude oil price forecasting performances significantly.
The remainder of this paper is organized as follows. Section 2 first proposes 2SVBMA methodology, starts with extended L 2 boosting to interval-valued variable selection and develops the LsoMA with interval-valued model with interval-valued exogenous variables. Section 3 provides the empirical implementations. Section 4 discusses the empirical results. Section 5 concludes.

Model Framework
Let (Ω, F , P) be a probability space, where Ω is the set of elementary events, F is the σ-field of events, and P: F → [0, 1] is the σ-additive probability measure. An interval random variable is defined as a measurable mapping (Arroyo et al., 2011;González-Rivera and Lin, 2013). A stochastic ITS {y t [y L,t , y U,t ]} T t 1 can be represented by its midpoint and range, i.e., y t < y c,t , y r,t >, where y c,t 1 2 (y L,t + y U,t ) and y r,t y U,t − y L,t . Assume that {y t } is stationary and follows a vector autoregressive models with interval-valued exogenous variables: is an interval-valued sequence with mean zero and covariance matrix Eε t ε t ′ ≡ Σ , and α i and β j are the coefficient matrix that satisfies . This data generating process guarantees the natural order of the intervals, i.e., the lower bound is smaller than or equal to the upper bound.
In matrix form, (1) is represented by and where The least squares estimators of Π c and Π r are given bŷ

First Stage: Vector Boosting
We first extend L 2 Boosting regularization method to interval model to select a subset of interval-valued variables. Z k is the k th row in Z. They are the potential interval-valued variables that will be selected by vector boosting. Z k,t is the t th element in Z k and Π k is the corresponding k th interval-valued coefficient of Π, where k 1, . . . , p + q. Let m denote the m th iteration in the vector boosting procedure, andM denote the maximum number of iteration. At each step m, the interval-valued variableΠk m that is most relevant to the "current interval-valued residual" is selected. Denote F m,t as the strong learner and f m,t as the weak learner for k 1, Vector L 2 Boosting performs an interval-valued variable selection for Y using the following procedure: August 2021 | Volume 9 | Article 707937 1. When m 0, the initial weak learner for y t is The interval-valued variables that has the minimum sum of squared residuals is picked up, such that 3) The weak learner is where Z km,t is the interval-valued variable that is selected.
4) The strong learner F m,t is updated as with c m > 0, where c m is a learning rate, which can be seen as a small step size when updating F m,t .
To avoid overfitting, a version of AIC is used to choose the optimal number of iteration M . Define P m Z km (Z km ′ Z km ) −1 Z km ′ to be an T × T matrix. From Equation (10), The strong learner at each step m is AIC is given as Then M arg min m 1,...,M AIC(m).

Second Stage: LsoMA
After selecting these important exogenous interval-valued variables, LsoMA technique is extended to interval candidate models with interval-valued exogenous variables, which is adopted to reduce model uncertainty and increase forecast accuracy.
Consider S candidate models used to approximate the DGP in Eq. (1) with S to be infinite if the sample size is going to infinity. The sth (1 ≤ s ≤ S) candidate model is given by Then the model averaging estimator of conditional mean μ isμ (w) S s 1 w sμ (s) . To obtain the optimal weights, it is common to minimize the following squared loss function: However, this loss is infeasible because of the unknown conditional mean μ. We follow the spirit of Liao et al. (2019) to use the following feasible leave-subject-out cross-validation criterion of choosing weights is the leave-subject-out cross-validation estimator after deleting some observations around S + t, andμ (w) S s 1 w sμ (s) ; see more discussions in Liao et al. (2019). Minimizing this criterion, we haveŵ and thus the model averaging estimator isμ (ŵ) . As Liao et al. (2019) proved, the weight obtained by minimizing the feasible cross-validation criterion LsoMA(w) is asymptotically optimal in the sense of achieving the lowest possible quadratic errors, i.e., This shows that the squared error loss obtained from the selected weight vectorŵ is asymptotically equivalent to the infeasible optimal averaging estimator.

EMPIRICAL IMPLEMENTATIONS
This section applies the proposed 2SVBMA procedure to forecast the real price of crude oil. Data and preliminary analysis are introduced in Section 3.1. Then the selected interval-valued factors are introduced in Section 3.2. Section 3.3 introduces the candidate models. Section 3.4 provides competing methods.

Data and Preliminary Analysis
Following Wang et al. (2017), Chai et al. (2018) and Yu et al. (2019), the daily point-valued WTI crude oil prices are used to construct the interval-valued monthly prices. y U,t and y L,t denote the daily maximum and minimum prices within tth month. y c,t (y U,t + y L,t )/2 and y r,t y U,t − y L,t are the midpoint and range from an interval-valued price observation y t 〈y c,t , y r,t 〉. The data period used in the research is from January 2005 to December 2017. Data on crude oil prices are collected from the US Energy Information Administration (EIA). Figure 2 presents the interval-valued crude oil prices: the range (y r,t , right y-Axis), the maximum (y U,t , left y-Axis), and minimum (y L,t , left y-Axis) prices within 1 month, where we can see that the boundaries and ranges are interlinked, e.g., a strong increase in volatility (y r,t ) is accompanied by a significant decrease in crude oil prices during the second half of 2008. Table 1 presents the summary of statistical characteristics. First, it is shown that the spread of ranges is slightly smaller than the volatility in the boundaries (Dy U,t y U,t − y avg,t−1 and Dy L,t y L,t − y avg,t−1 ), where y avg,t is the monthly prices from EIA. In addition, the skewness and leptokurtic kurtosis are different among y r,t , Dy L,t and Dy U,t . Compared with Dy L,t and Dy U,t , y r,t is with greater skewness and higher leptokurtic. We can see from Table 1 that the interval-valued data can capture more information than the point-valued data.

Interval-Valued Control Variables in the First Stage
The potential choices of monthly interval-valued explanatory variables from various aspects are considered in this section, including the stock market, commodity market, technology factor, search query data, speculation, monetary market and currency market (Pan et al., 2014;Wang et al., 2016;Wang et al., 2017;Chai et al., 2018;Yu et al., 2019); see Table 2 for more discussions. First, the Augmented Dickey-Fuller tests suggest that  the null hypothesis for the original control variables is hardly rejected at the 5% significance level, except for non-commercial net long ratio (NL t ) and the Federal funds rate (FD t ). For stationarity, we use the Hukuhara's difference of intervalvalued exogenous variables. The Hukuhara's difference between a pair of intervals is essentially equal to the regular difference between points in intervals. As Yang et al. (2016) mentioned, the concept of interval with Hukuara's difference is useful and suitable for econometric analysis of interval data. Take S&P 500 index (SP t ) as an example. It is defined as where Δ is the Hukuhara's difference between intervals, and Δ is the regular difference between intervals. This implies that the midpoints and centers of these interval-valued exogenous variables are stationary after Hukuhara's difference. Similarly, we have ΔDJ t , ΔGF t , ΔCF t and ΔRD t ; see specific definitions in Table 2.
Second, Table 3 provides a summary of statistical characteristics. It is shown that no matter whether the time series is transferred by Hukuhara's difference, the midpoints and ranges for interval-valued control variables appear to have different skewness and leptokurtic kurtosis properties. This suggests that using one attribute of ITS contains partial information only. Thus, it is highly desirable to utilize the information contained in interval-valued data.
Third, we use the extended L 2 Boosting regularization method to select interval-valued control variables. Specifically, we set the lag length L 12 for every control variable and thus the number of the potential explanatory interval-valued variables equals 12 × 9 108. For vector boosting, we start with the learning rate c 0.01, iteration 100 times. These parameters are adjusted during training. After using various training sets, ΔSP t+h−1 , ΔGF t+h−1 , ΔGF t+h−2 , ΔGF t+h−3 , WB t+h−1 , and GT t+h−4 are selected with duplicates removed and used to do h-step-ahead out-of-sample forecasts of interval-valued crude oil prices.
Furthermore, these selected interval-valued control variables have important economic interpretation for crude oil prices as follows: ΔSP t+h−1 : It provides information of fundamentals and volatility contained in S&P 500. The movement of S&P 500 Index may closely mirror that of the crude oil prices (e.g.,   Kilian, 2009;Miller and Ratti, 2009;Balcilar et al., 2015;Ding et al., 2016). As discussed in Kilian (2009) and Miller and Ratti (2009), the oil price shocks influence stock prices by affecting expected cash flows and discount rates, since crude oil is an important input in production and its price can influence the costs for the manufacturing and transport sectors. ΔGF t+h−j (j 1,2,3): It is the logarithmic difference between Comex gold future prices at t + h − j and t + h − j − 1, which provides information in Comex gold future market (e.g., Baur and Lucey, 2010;Reboredo, 2013;Souček, 2013;Kang et al., 2017). Gold serves as store of value especially during periods of economic uncertainties. Oil prices can affect levels of inflation (Zhao et al., 2016). Gold investment can be used as a hedge against inflation and currency depreciation. It can also be viewed as a safe haven against the stock market turbulence for investors.
WB t+h−1 : It is WTI-Brent spot price spread, which is the price difference between crude oil and the byproducts refined from it. The crack spread gives the profit margin that a refinery can expect. Thus, a tight spread can be seen as a indicator that refiners may slow production to tighten supply.
GT t+h−4 : It is the search query data collected from Internet, which has been widely applied as indicator when analyzing the crude oil prices and has been demonstrated to be effective in improving forecasts performance (Fantazzini and Fomichev, 2014;Li et al., 2015a;Wu et al., 2021;Yang et al., 2021). The keyword "oil price" is searched in the Google Trend search engine. Search query data is expected to reflect the psychological aspects of investors when they making strategic investment decisions in the crude oil market (Li et al., 2015b).
The AIC criterion for the sth candidate model (1 ≤ s ≤ 15) is AIC (s) ln |Σ (s) | + 2s2 2 /T, whereŝ minimizes AIC (s) and s) ) as the residual covariance matrix from the sth candidate model. Similarly, BIC and HQ are model selection methods, minimizing the corresponding criteria BIC (s) ln |Σ (s) | + (ln T)s2 2 /T, HQ (s) ln |Σ (s) |+ 2(ln ln T)s2 2 /T, respectively. These three selected candidate models ares used as benchmark models. Four model averaging (or forecast combination) methods are considered here. MMA proposed by Liao and Tsay (2016) is an extension of Mallows criterion to vector regression models. Specifically, the multivariate Mallow criterion for model averaging takes the following form:

EMPIRICAL RESULTS
This section compares the forecasting performance of the proposed 2SVBMA approach with various competing methods presented in previous studies by using interval-valued crude oil prices. The whole sample from 2005 January to 2017 December are divided into two parts: one is used for parameter estimation, and the other is used for out-of-sample forecasting. Various subsamples for estimation and forecast are used to test prediction accuracy; see Tables 4, 5. Tables 4, 5 report the MSPEs of h-step-ahead (1,4,8,12) forecasts for the interval-valued crude oil prices using various estimation and forecast samples. First, it is worth noticing that for the horizons of 1, 4, 8 and 12 months, the 2SVBMA method outperforms other competing methods in most cases; out of the 48 cases considered, with respect to RMSFE of midpoints and ranges, it yields the best outcomes 42 times and the second best outcomes 6 times. Intuitively, the proposed 2SVBMA method selects the important factors at the first stage and then give the optimal weights averaging across the 12 nested regression forecasts. Second, 2SVBMA based on LsoMA outperforms various model averaging and model selection methods, including MMA. One possible explanation is that leave-subject-out cross-validation is more suitable for vector autoregressive situations with heteroscedastic and auto-correlated errors. Additionally, as shown in Liao et al. (2019), the approximate unbiasedness of LsoMA and its asymptotic optimality in terms of obtaining the lowest quadratic errors are established. This is why LsoMA outperforms other model averaging methods (i.e., SAIC, SBIC, and SHQ) in the second stage.
Second, the SBIC estimators always produce the second-best forecasts after the 2SVBMA estimator among all model averaging methods, while SAIC achieves higher forecast criteria than other model averaging methods. Similarly, BIC always yields best forecasts among all model selection methods, while the AIC estimator achieves higher MSFE in most cases. This happens because AIC prefers selecting the relatively complicated model, which is inappropriate for out-of-sample forecasting even though it has good in-sample fitting. A simple model may be better for out-of-sample forecasting.
Furthermore, it is shown that at the second stage, model averaging forecasts outperform model selection forecasts in almost 90% of all cases. The significant advantages of model averaging support the argument of Rapach et al. (2010) that Note: "Estimation" denotes the sample during this period used to estimate parameters, and "Forecast" denotes the sample during this period used to do out-of-sample forecasts. The best forecasts are marked by boldface, and the second best forecasts are marked by underline.
Frontiers in Energy Research | www.frontiersin.org August 2021 | Volume 9 | Article 707937 "model uncertainty and instability seriously impair the forecasting ability of individual predictive regression models." Overall, the proposed approach using interval-valued data is capable of assessing and forecasting the changes in both level and volatility. We can see from the results that forecasting with model averaging is generally better than obtaining the predictions from just one model (model selection). Since we may choose a very different model when there are small changes in the original data set, which may lead to a big change in the final conclusions, resulting in non-effective decision-making due to the unstable forecasting process. The proposed method is able to help obtain more stable decision-making when a long list of interval-valued predictors is available in a wide range of fields, for example, the daily trading strategy in the finance field.

CONCLUSION
We propose a novel 2SVBMA forecasting procedure to capture the relevant information available in the interval format and the underlying characteristics of crude oil price movements. Vector L 2 Boosting in the first stage and LsoMA in the second stage are extended to interval models with interval-valued exogenous variables. Empirical results show that our proposed approach outperforms other competing model averaging and model selection methods in terms of MSFE of midpoints and ranges.
There are some limitations and potential extensions of our study. First, more advanced optimization algorithms for interval-valued variable selection can be proposed in future work. Second, the candidate models with different structures in model averaging methods can further be developed to enhance forecasting. It would also be interesting to develop interval-based machine learning methods to improve forecast accuracy. Furthermore, the proposed methodology in this paper can be extended to the vector autoregressive (VAR) model, which can cover more applications in economics and finance.
In general, 2SVBMA provides a methodological framework for interval-valued data forecasting when there are a large number of potential predictors. For example, this methodology can be used to quantify the impact of COVID-19 pandemic on oil and gas industry. 2SVBMA can also provide implications for the post-COVID recovery management. The accurate prediction of crude oil prices will assist policy makers in understanding issues affecting different oil industry segments, and help governments be better prepared for the recovery.

COMPLIANCE WITH ETHICAL STANDARDS
The authors thank a number of the participants at Symposium on Interval Data Modelling: Theory and Applications (SIDM 2019) in Beijing for their valuable comments and Note: "Estimation" denotes the sample during this period used to estimate parameters, and "Forecast" denotes the sample during this period used to do out-of-sample forecasts. The best forecasts are marked by boldface, and the second best forecasts are marked by underline.
Frontiers in Energy Research | www.frontiersin.org August 2021 | Volume 9 | Article 707937 suggestions. This work was partially supported by National Natural Science Foundation of China (Nos. 71973116, 71988101, 72073126, 72091212), and the disciplinary funding of Central University of Finance and Economics. The authors declare no competing interests. This article does not contain any studies with human participants performed by any of the authors.

DATA AVAILABILITY STATEMENT
The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.

AUTHOR CONTRIBUTIONS
All three authors contributed equally to this work and the order of authorship has nothing other than alphabetical significance.

FUNDING
This work was partially supported by National Natural Science Foundation of China (Nos. 71973116, 71988101, 72073126, 72091212), the funding of Forecasting and Monitoring of COVID-19 in countries along "Belt and Road" and Related Economic Impacts (ANSO-SBA-2020-12), and the disciplinary funding of Central University of Finance and Economics.