Neonatal mortality is an important public health problem that reflects the development of a country, as well as the quality of care provided to the newborn. This article presents the development and comparison of classical models and machine learning models for time series forecasting, applied to the forecast of monthly neonatal mortality rates in the metropolitan region of Paraiba River Valley and North Coast - São Paulo State - Brazil. The database used comprised the monthly rates from January 2000 to December 2020. The models compared were Seasonal Autoregressive Integrated Moving Average, random forest, support vector machine (SVM), light gradient boosting machine, categorical boosting (CatBoost), gradient boosting (GB), extreme gradient boosting, and multilayer perceptron. The best parameters and hyperparameters of the models tested were adjusted through an exhaustive computational search. The results showed that the CatBoost, SVM, and GB models presented the lowest values in the error metrics evaluated, and the SVM model presented better precision. The forecasts of the SVM model showed a behavior very close to the actual rates, which was confirmed by the application of the paired t-test. These results corroborate that time series forecasting models can significantly contribute as a decision support tool for public health problems. Neonatal mortality, time series analysis, forecasting, data-driven models, machine learning ORCID: 0000-0002-3829-4118 (E. Rodríguez); 0000-0003-1120-1708 (E. Rodríguez); 0000-0001-9793-750X (L. Nascimento); 0000-00022215-0734 (A. da Silva); 0000-0001-6510-9187 (F. Marins)
Neonatal mortality is associated with the occurrence of newborn death in the first 28 days of life and is considered an important public health problem [1]. In this context, both the infant mortality rate and the neonatal mortality rate can be considered relevant indicators for evaluating the quality of care provided to newborns, public health, and the well-being of the population, as well as the country's development.
Globally, from 1990 to 2015, infant mortality has been decreasing, which has been mainly reflected in post-neonatal mortality, but unfortunately, the number of neonatal deaths has not been showing the same reduction [2], (Figure 1). Corroborating this finding, according to the World Health Organization (WHO), from 1990 to 2019, neonatal deaths decreased from 5.0 to 2.4 million, but only in 2019, 47% of deaths in children under 5 years old occurred in the first 28 days of life [3]. Therefore, reducing the number of neonatal deaths still remains a major challenge [4].
The risk of death of the newborn is greatest during the first hours and days of life [4], with approximately 1 million newborns worldwide dying in the first 24 hours of life and about 75% of deaths
2022 Copyright for this paper by its authors. occurring during the first week of life [3]. In addition, it is important to emphasize that newborns are highly vulnerable, and those newborns with low birth weight, premature delivery, or with some health problems are even more vulnerable and have a higher risk of death in the neonatal period [1].
Reducing infant and neonatal mortality has become a global concern and has been reflected in the approach to the Millennium Development Goals (MDGs) and the Sustainable Development Goals (SDGs). Specifically, the MDGs sought to reduce infant mortality by 2015 [5], and Goal 3 of Health and Well-being of the SDGs aims to reduce neonatal mortality to at least 12 per 1,000 live births by 2030 [6].
In the case of Brazil, from 1996 to 2020, infant mortality rates decreased, reaching in 2015 the established target of the MDGs in reducing infant mortality [7]. However, it appears that the reduction in the number of infant deaths was mainly due to the decrease in post-neonatal mortality (between 28 and 365 days of life) [4,8], as shown in Figure 1.
To improve the conditions for decision-making in public health problems, it is possible to use technological advances and, with the historical data stored, it is possible to extract useful knowledge to predict future events [9-11]. Thus, current and past knowledge can be used to model and make future predictions that can help managers to face the serious problems already mentioned [11], making it possible to identify areas of uncertainty, such as quantitative mortality predictions [12].
In this context, time series forecasting plays an important role and can be a useful tool for planning public health policies and improving the provision of health services and care [12-14].
In this way, several algorithms can be applied to the development of time series models, which can capture complex patterns in the available data [9, 11, 14]. Some of the classic time series methods are the Autoregressive Integrated Moving Average (ARIMA), and the Seasonal Autoregressive Integrated Moving Average (SARIMA) also known as Seasonal ARIMA, among others [9, 11].
On the other hand, in recent years there has been a growing interest in the application of Machine Learning (ML) in several areas of study [15], and the medical field has been no exception, and ML has been used as a decision support tool in several problems [16, 17]. In addition, ML techniques have been used for time series forecasting, which has resulted in the development of efficient models that often outperform classical models [9, 11].
It is important to highlight that in the literature there are still few works on health forecasts with time series [13]. Therefore, the general objective of this article was to present the development of forecasting models in time series, using classical statistical techniques, and ML techniques, using real data on neonatal mortality in the metropolitan region of Paraiba River Valley and North Coast – São Paulo State - Brazil.
The article is organized as follows: Section 2 describes the materials and methods adopted; Section 3 presents the results and their discussion, describing the main findings. Finally, Section 4 presents the conclusions, followed by the bibliographic references.
Exploratory data analysis.
Calculate mortality rate.
NMR = Number of infant deaths under 28 days of age X 1,000
Number of live births
NMR
Splitting the
dataset 3 Model building and model fit
Classical methods Identify stationarity.
ADF and KPSS
test. - Differentiation
ARIMA
SARIMA 4 Performance
Evaluation metrics:
Test
MAE MSE
RMSE MAPE r SMAPE
Structuring of time series.
Train
Test Machine Learning Data normalization.
Original
Normalized deaths by the number of live births per month, expressed per 1,000, as formulated in (1): 28 ℎ × 1,000 (1)
Then, the monthly time series was structured and graphs were built to better visualize the behavior of these data. In addition, the dataset was split in a 90:10 ratio for training and testing, considering rates from January 2000 to November 2018 for training and from December 2018 to December 2020 for testing. Step 3 - Forecasting Models:
A time series is a sequence of chronologically ordered random observations, denoted by = { , +1, +2, . . . , }, where is an observation at instant satisfying 1 ≤ ≤ , where the duration of the entire period considered. As can be seen in the available literature, classical statistical methods, as well as ML techniques, have been successfully used in time series forecasting [9, 11].
In this work, the SARIMA, Random Forest (RF), Support Vector Machine (SVM), Light Gradient Boosting Machine (LGBM), Categorical Boosting (CatBoost), Gradient Boosting (GB), eXtreme Gradient Boosting (XGBoost), and Multilayer Perceptron (MLP) models were tested, and their performances were compared in forecasting monthly neonatal mortality rates. The models were implemented with the Python v.3.9.7 programming language and in the Jupyter-notebook v.6.4.6 development environment. from non-stationary to stationary.
The SARIMA model is an extension of the ARIMA model, being that the ARIMA( , , ) model includes the statistical procedures of autoregression (AR) of order , the integration (I) that indicates the number of differences necessary to guarantee the stationarity of the series, and the moving average (MA) of order . In the case of the SARIMA( , , )( , , )s model, the letter ‘s’ subscript represents the seasonality period of the series, and , , and are respectively the autoregressive, differential, and moving average procedures of the seasonal part of the ARIMA process [9].
A time series is stationary when it satisfies the properties of the mean ( ( )= ), variance ( ( )= ( − )2= 2) and autocovariance ( = ( , + )= [( − )( + − )]) do not change over time. Many of the analyzed data are not stationary series and the common practice is to transform non-stationary data into stationary data through successive differences [9], being that generally, the first difference ( = − −1) the first difference is enough to convert the series is no unit root, that is, the series is stationary [19,20].
The Augmented Dickey–Fuller (ADF) test and the Kwiatkowski–Phillips–Schmidt–Shin (KPSS) test were used to verify stationarity in the monthly neonatal mortality rate data, adopting a significance level of 5%. The null hypothesis ( = 1) of the ADF test is the existence of a unit root, that is, the series is not stationary, and the null hypothesis ( 0: | | < 1) of the KPSS test is that there Thus, the autoregressive (AR) process of order is given in (2):
= 0 + 1 −1 + 2 −2 + ⋯ + − + where is the model parameters and is the random error also known as white noise.
Equation (3) corresponds to the moving averages (MA) model of order :
= 0 − 1 −1 − 2 −2 − ⋯ − − where is the parameters of the MA model.
The ARIMA model is based on the stationarity assumption, applying the combination of the AR model with the MA model in the differentiated series, as presented in (4): where is a differentiated time series of order [9,11,20]. (2) (3) (4) where
and and ′ is the normalized value of [17, 22].
For the development of supervised ML models, all normalized observations of the training series ( ) were considered. Thus, the training dataset was formatted in a subsequence of observations = {( 1, 1), ( 2, 2), … , ( , )}, as presented in (7): ′ =
− are, respectively, the minimum and maximum rates of the training dataset,
In the case of the SARIMA model, equation (5) is used: where and are the non-seasonal parameters of the AR and MA model, seasonal parameters of the AR and MA model, respectively, is the seasonality period, is the nonseasonal differentiation, and is the differentiation of the seasonal part of the model [11, 21].
On the other hand, in the case of ML models, the original time series data from the training set were normalized using the MIN-MAX method. The MIN-MAX method transforms the data in the range [0 – 1] [10, 17, 22], as shown in (6): = 1,1 2,2 3,3 ⋮ 1,2 2,3 3,4 ⋮ ⋯ ⋯ ⋯ ⋯
1, 2, +1 3, +2
⋮ [ , +( − ) , +( −2) ⋯ , +( −1)] ∧ = 3, +3 1, +1 2, +2
⋮ [ , + ] of labels. where is the matrix of input attributes of the temporal pattern of dimension , and is the vector Furthermore, is the size of the sliding window, that is, the number of previous mortality rates considered to predict +1, which indicates the posterior value of 1, . In other words, ML models were trained with a finite sequence of × pairs, where mortality rates from previous months were used to forecast a future observation.
Among the ML techniques that can be used to forecast time series are neural networks such as MLP, SVM and ensemble methods such as RF, GB, LGBM, CatBoost, and XGBoost.
MLP is a neural network that has a layered structure (input layer, one or more hidden layers, and output layer), composed of neurons also known as perceptrons. From the MLP network architecture, it is possible to approximate continuous functions providing a non-linear mapping between inputs and outputs [9, 23]. The dimension of the input attributes is equal to the number of neurons in the input layer of the neural network, as well as the number of neurons in the output layer corresponds to the target variable [9]. Each neuron in the network has an activation function that will determine the perceptron output, in addition to each neuron receiving information from the neuron of the previous layer until the MLP prediction is generated [23].
SVM uses the input data to map it onto a high-dimensional non-linear hyperplane, also known as a feature space [24]. The construction of the SVM model is subject to linear constraints and the estimation of unknown parameters, such as the weight vector and the hyperplane bias [9]. The hyperplane estimation is performed using the training data and the Lagrange multiplier method to define the edges of the ideal separating hyperplane [24].
The ensemble methods were proposed to reduce the variation and improve the accuracy of the developed models [25]. A wide variety of ensemble methods have been proposed, such as the RF, GB, LGBM, CatBoost, and XGBoost methods. The construction of the models of the ensemble method are based on decision trees, but the training process for each model is different. The decision trees of the models and their variations based on gradient boosting are trained iteratively, unlike the RF models that train the decision trees in parallel [25]. (5) (6) (7) Step 4 - Performance Metrics:
According to the literature, there is no single standard evaluation metric [22, 26], so, here, the model accuracies were evaluated and compared using various metrics, such as Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE) [22], Mean Absolute Percentage Error (MAPE), Symmetric
Mean Absolute Error (SMAPE) [27], and Correlation Coefficient (r). The calculations of these metrics can be performed by (8)-(13):
(∑ ̂ )− (∑ )(∑ ̂ ) √[ ∑ 2 − (∑ )2][ ∑ ̂ 2 − (∑ ̂ )2] (8) (9) (10) (11) (12) (13) where is the actual value of the mortality rate for the period = 1,2,3, . . . , , ¯ is the average of , and ̂ is the predicted value of observation .
The model with the best performance was selected by evaluating the lowest values in the MAE, MSE, RMSE,
MAPE, and
SMAPE metrics, in addition to the highest value in the correlation coefficient [22, 27, 26]. In addition, the predicted rates and the actual rates were compared using the paired t-test to verify significant differences between the values of each model ( − < − ), adopting a significance level of 5% ( − > 0.05) [28].
In this section, the main results of the descriptive analysis of the data are presented, besides the predictions of the neonatal mortality rate of the models tested.
In the analyzed period from 2000 to 2020, the metropolitan region of Paraiba River Valley and North Coast – São Paulo State – Brazil had 697,903 live births and 6,631 neonatal deaths, with an average of 2,769 live births and 26 deaths per month. Using descriptive statistics techniques, applied to monthly neonatal mortality rates, the minimum and maximum monthly rates were obtained, which were 3.71 and 16.47, respectively. The mean monthly rate was 9.48, the standard deviation was equal to 2.54, the median was equal to 9.20, and the first and third quartiles were equal to 7.52 and 11.25 respectively. the distribution and behavior of values over time can be observed. In the annual analysis, it was observed that neonatal mortality rates had a decreasing behavior, in addition to showing a slight increase in 2017 and a variable behavior in the next three years. Likewise, in the monthly analysis, it was observed that June presented a decrease and less variation in mortality rates.
Figure 3 (C) shows the time series line graph of the neonatal mortality rate for the 227 months considered for training the models, and it was possible to observe the presence of a decreasing trend and a possible variation of the mean, which may indicate the non-stationarity of the series.
Furthermore, it is known that in the case of the classical SARIMA method, the stationarity of the time series is a prerequisite [9, 11]. The collected data were analyzed and evaluated for the possible presence of non-stationarity through the ADF and KPSS tests, and these results are presented in Table 1. (a) ADF: Augmented Dickey–Fuller; (b) KPSS: Kwiatkowski–Phillips–Schmidt–Shin; (c) CV: critical value. p-value 0.010 0.100 The results of the ADF and KPSS tests for the analysis of the existence of a unit root in the original time series confirmed that the data are non-stationary (Table 1), and therefore, a first-order difference was applied to the series to make it stationary. The stationarity of the differentiated series was confirmed by the ADF and KPSS tests at a significance level of 5%, revealing the rejection of the null hypothesis ( < and − < 0.05) in the ADF test and the acceptance of the null hypothesis ( < and − > 0.05) in the KPSS test.
The estimations of the best parameters of the SARIMA( , , )( , , )s model were performed by applying an exhaustive computational search. Furthermore, the models with the best combinations of parameters were selected using the AIC to select those with the best fit. Table 2 presents the best parameter configuration of the SARIMA model.
Likewise, Table 3 presents the best settings for the hyperparameters of the tested ML models. The search for the best hyperparameters was performed using the grid search method, in which the parameters selected were those that maximized the performance of the model. Window size: 12, hidden_layer_sizes: (50, 25), activation: 'relu', learning_rate: 'constant', learning_rate_init: 0.00045, max_iter: 120, tol: 0.0001, alpha: 0.0013, batch_size: 201, epsilon: 1e-05
The performance of classical and ML statistical models is shown in Figure 4. The metrics MAE, MSE, RMSE, MAPE, SMAPE, and r were calculated by evaluating the difference between the original rates and the rates predicted by the models, with the test dataset.
When comparing these models, the classic SARIMA (MAE = 1.362, MSE = 2.880, RMSE = 1.697, MAPE = 0.067, SMAPE = 0.189, r = 0.087) time series forecast model presented the highest values in the performance metrics, indicating low adequacy in the prediction of monthly neonatal mortality rates, as can be seen in Figure 4. However, when the performance of the SARIMA model was compared with the ML models, it became evident that the ML models tested had a better ability to predict monthly mortality rates.
From the performance of the ML models tested, the CatBoost (MAE = 1.168, MSE = 2.216, RMSE = 1.488, MAPE = 0.058, SMAPE = 0.162, r = 0.499), SVM (MAE = 1.027, MSE = 1.804, RMSE = 1.343, MAPE = 0.051, SMAPE = 0.143, r = 0.660), and GB (MAE = 1.088, MSE = 1.761, RMSE = 1.327, MAPE = 0.054, SMAPE = 0.151, r = 0.629) techniques presented the best evaluation metrics, that is, they presented the lowest values in the error metrics and the highest values in the correlation coefficient.
On the other hand, the LGBM (MAE = 1.232, MSE = 2.270, RMSE = 1.507, MAPE = 0.061, SMAPE = 0.171, r = 0.482), XGBoost (MAE = 1.266, MSE = 2.706, RMSE = 1.645, MAPE = 0.063, SMAPE = 0.176, r = 0.275), MLP (MAE = 1.289, MSE = 2.826, RMSE = 1.681, MAPE = 0.064, SMAPE = 0.179, r = 0.208), and RF (MAE = 1.310, MSE = 2.752, RMSE = 1.659, MAPE = 0.065, SMAPE = 0.179, r = 0.383) models presented the worst values in the evaluated metrics, but even so, they were lower values when compared to the SARIMA model.
Figure 5 illustrates the forecasts of monthly neonatal mortality rates, for the period from December/2018 to December/2020, based on the Catboost, SVM, and GB models, and the original rates.
Through a visual analysis in Figure 5, the SVM model showed better behavior in forecasting monthly rates, showing a high degree of proximity between the original data and the forecasted values. Likewise, when evaluating its performance through error metrics, the SVM model presented the lowest values in error rates and the highest value in r. To confirm this result, a paired t-test was performed to compare the performance of the SVM (mean = 7.359, standard deviation = 1.193), CatBoost (mean = 7.671, standard deviation = 0.886), and GB (mean = 7.495, standard deviation = 1.057), as shown in Table 4. Note that the mean and standard deviation of the real data analyzed were 7,274 and 1,748, respectively.
The results of the paired t-test showed that the mean of the real data and the mean of the selected models were very close values, so the null hypothesis was accepted for the three models. Furthermore, comparing the t-value and t-critical results of the SVM, CatBoost, and GB models (Table 4), it was observed that the t-value of the SVM (0.311) model is lower than that of the GB (0.824), and CatBoost (1.353) models, indicating that the predictions with the SVM model is closer to the real values, unlike the other models. Thus, the prediction of events with time series related to neonatal deaths can significantly help managers in decision-making, such as resource management, strategic planning, development of public policies, improvement of care for newborns and mothers, etc., with the aim of mitigating risks and reducing the number of deaths [10, 12, 13].
Neonatal mortality is an important public health problem because newborns have a higher risk of death in the first 28 days of life. Worldwide, the mortality rate has decreased considerably and this decrease in deaths occurred mainly in the post-neonatal period. In Brazil, the same behavior has been observed when evaluating infant and neonatal mortality rates.
In this sense, the prediction of future events can significantly help in the reduction of deaths, as well as in the development of public policies to mitigate risks with preventive care for newborns and improve the assistance received.
In this work, classical models and ML models were compared to predict monthly neonatal mortality rates in the metropolitan region of Paraiba River Valley and North Coast – São Paulo State - Brazil, for the period from January 2000 to December 2020. The classical statistical model evaluated in this work was SARIMA, in addition to the supervised models of RF, SVM, LGBM, CatBoost, GB, XGBoost, and MLP.
The SARIMA model presented the highest values in the error metrics evaluated, compared to the ML models. The SVM model presented better precision, demonstrating that it was possible to capture the complex patterns of the data for the prediction of monthly neonatal mortality rates. The results were reinforced by the paired t-test, confirming that the SVM model predicted the mortality rates with a behavior very close to the actual rates.
Among the limitations of this work is the use of data provided by DATASUS, in addition to the comparison of ML models only with the SARIMA method. In future work, we intend to compare other classical time series models, as well as develop models for predicting mortality rates for the municipalities in the analyzed region.
Finally, it is important to highlight that prediction models are not a solution for neonatal mortality; however, they can significantly contribute to the approach of preventive measures for newborn care.
This work was supported by the Coordination for the Improvement of Higher Education Personnel [grant number CAPES - 001]; and partially by National Council for Scientific and Technological Development [grant numbers CNPq – 304197/2021-1; CNPq 303090/2021-9].
[1] L. Hug, M. Alexander, D. You, L. Alkema. "National, regional, and global levels and trends in neonatal mortality between 1990 and 2017, with scenario-based projections to 2030: a systematic analysis", Lancet Glob Health 7(6): e710–e720, 2019 . doi: 10.1016/S2214-109X(19)30163-9 [2] World Health Organisation. "Global Nutrition Targets 2025: Low birth weight policy brief", 2014.
URL: https://apps.who.int/iris/bitstream/handle/10665/149020/WHO_NMH_NHD_14.5_ eng.pdf?ua=1 [3] World Health Organisation. "Newborns - improving survival and well-being", 2020. URL: https://www.who.int/news-room/fact-sheets/detail/newborns-reducing-mortality [4] J.E. Lawn, H. Blencowe, S. Oza, D. You, A.C. Lee, P. Waiswa, et al. "Every Newborn: progress, priorities, and potential beyond survival", The lancet 384(9938): 189-205, 2014. doi: 10.1016/S0140-6736(14)60496-7 [5] United Nations. "The Millennium Development Goals Report 2015", 2016. URL: https://www.un.org/millenniumgoals/2015_MDG_Report/pdf/MDG%202015%20rev%20(July% 201).pdf [6] United Nations. "Transforming our world: the 2030 Agenda for Sustainable Development", 2015.
URL: https://www.un.org/ga/search/view_doc.asp?symbol=A/%20RES/70/1&Lang=E [7] C.D.S.R. Marinho, T.B.M. Flor, J.M.F. Pinheiro, M.Â.F. Ferreira. "Millennium Development Goals: the impact of healthcare interventions and changes in socioeconomic factors and sanitation on under-five mortality in Brazil", Cadernos de Saúde Pública 36(10):e00191219, 2020. doi: 10.1590/0102-311X00191219 [8] M.D.C. Leal, C.L. Szwarcwald, P.V.B. Almeida, E.M.L. Aquino, M.L. Barreto, et al. "Reproductive, maternal, neonatal and child health in the 30 years since the creation of the Unified Health System (SUS)", Ciência & Saúde Coletiva 23: 1915-1928, 2018. [Article in Portuguese, English] doi: 10.1590/1413-81232018236.03942018 [9] A.R.S. Parmezan, V.M. Souza, G.E. Batista, "Evaluation of statistical and machine learning models for time series prediction: Identifying the state-of-the-art and the best conditions for the use of each model", Information sciences 484: 302-337, 2019. doi: 10.1016/j.ins.2019.01.076 [10] N. Shakhovska, I. Darmoriz, Y. Vyklyuk, Y. Kryvenchuk, P. Pukach, "Visualization of the Epidemics Forecasting Results". The 4th International Conference on Informatics and Data-Driven Medicine (IDDM 2021), 3038(2021): 283-292. [11] F. Petropoulos, D. Apiletti, V. Assimakopoulos, M.Z. Babai, D.K. Barrow, S.B. Taieb, et al. "Forecasting: theory and practice", International Journal of Forecasting 38(3): 705-871, 2022). doi: 10.1016/j.ijforecast.2021.11.001 [12] K.J. Foreman, N. Marquez, A. Dolgert, K. Fukutaki, N. Fullman, M. McGaughey, et al. "Forecasting life expectancy, years of life lost, and all-cause and cause-specific mortality for 250 causes of death: reference and alternative scenarios for 2016-40 for 195 countries and territories", Lancet 392(10159):2052-2090, 2018. doi: 10.1016/S0140-6736(18)31694-5 [13] I.N. Soyiri, D.D Reidpath. "An overview of health forecasting", Environ Health Prev Med 18(1):19, 2013. doi: 10.1007/s12199-012-0294-6 [14] A.B.S. Silva, A.C.M. Araújo, P.G. Frias, M.B.R Vilela, C.V.B. "Auto-Regressive Integrated Moving Average Model (ARIMA): conceptual and methodological aspects and applicability in infant mortality", Rev Bras Saúde Mater Infant 21(2):657-666, 2021. doi: 10.1590/180693042021000200016 [15] E.Y.A. Rodríguez, E.C.A. Rodríguez, A.F Silva, P.M.R. Rizol, R.C. Miranda, F.A.S. Marins (in press). "Analysis of Machine Learning integration into SupplyChain Management", Int J Logist Syst Manag, 2021. [16] D.S. Char, N.H. Shah, D. Magnus. "Implementing Machine Learning in Health Care - Addressing Ethical Challenges", New England Journal of Medicine 378(11): 981-983, 2018. doi:10.1056/NEJMp1714229. [17] E. Rodríguez, E. Rodríguez, L. Nascimento, A. Silva, F. Marins, "Machine learning techniques to predict overweight or obesity". The 4th International Conference on Informatics and Data-Driven Medicine (IDDM 2021), 3038(2021): 190-204. [18] Information Technology Department of the Brazilian Unified Health System. "Ministry of Health:
Health Information", 2022. [in Portuguese] URL: https://datasus.saude.gov.br/ [19] D. Kwiatkowski, P.C.B. Phillips, P. Schmidt, Y.Shin. "Testing the null hypothesis of stationarity against the alternative of a unit root: How sure are we that economic time series have a unit root?", Journal of Econometrics 54(1-3):159-178, 1992. doi: 10.1016/0304-4076(92)90104-Y [20] W.A. Fuller, Introduction to Statistical Time Series, 1996. Canada: John Wiley & Sons, Inc. [21] M. Cools, E. Moons, G. Wets, "Investigating the variability in daily traffic counts through use of ARIMAX and SARIMAX models: assessing the effect of holidays on two site locations", I.
Transportation research record 2136(1):57-66, 2009. [22] E.Y.A. Rodríguez, A.A.R. Gamboa, E.C.A. Rodríguez, A.F. da Silva, P.M.S.R. Rizol, F.A.S.
Marins. "Comparison of adaptive neuro-fuzzy inference system (ANFIS) and machine learning algorithms for electricity production forecasting", IEEE Latin America Transactions 20(10):2288– 2294, 2022. [23] M. Tim, K. Ekrem, T. Burak, M. Leandro, P. Fayola. "Sharing Data and Models in Software
Engineering", 2014. [24] R. Collobert, S. Bengio. "SVMTorch: Support vector machines for large-scale regression problems", Journal of machine learning research, 1(Feb), 143-160, 2001.