Organizations collect data from various sources, and these datasets may have characteristics that are unknown. Selecting the appropriate statistical and machine learning algorithm for data analytical purposes benefits from understanding these characteristics, such as if it contains temporal attributes or not. This paper presents a theoretical basis for automatically determining the presence of temporal data in a dataset given no prior knowledge about its attributes. We use a method to classify an attribute as temporal, non-temporal, or hidden temporal. A hidden (grouping) temporal attribute can only be treated as temporal if its values are categorized in groups. Our method uses a Ljung-Box test for autocorrelation as well as a set of metrics we proposed based on the classification statistics. Our approach detects all temporal and hidden temporal attributes in 15 datasets from various domains.
Datasets can be temporal or non-temporal. A dataset is temporal if one or more attributes is a time sequence [1]. An example of a temporal dataset is a stock market dataset, in which each value of an attribute corresponds to the daily stock price. Time series normally present a time-dependency, meaning that a value is dependent on its past values. Time-series analysis has applications ranging from stock market prediction to digital signal metrics [2], and in communications [3].
Techniques for non-temporal data, such as Support Vector Machine (SVM) [4] and Isolation Forest (IF) [5] only discover associations among attributes of individual data records and cannot be used for analyzing time-series data because associations may exist among multiple records in a time series [6]. Other approaches, such as Autoregressive Moving Average (ARIMA) [7] and Long Short-Term
dependencies in a dataset in advance in order to choose the best analysis approach. Existing analysis approaches rely on domain experts to identify the type of data and to choose appropriate techniques to model the data. How(S. Ghosh) based on a portmanteau test [1] for autocorrelation to determine the presence of temporal data. To find groupof the series with a delayed copy of itself. It gives critical ing attributes that yield temporal sequences we use a information on whether a value in the series can be used brute force approach by testing each unique value as to infer information about another value. A common way a possible grouping attribute. Finally, we propose metto analyze temporal data is to create a model that fits the rics that help determine whether the result of the portdata, and the most widespread technique is regression manteau test should be accepted or rejected based on analysis, which uses autocorrelation [12]. Therefore, auan integrated perspective of the dataset. We evaluated tocorrelation is going to be the most important metric the proposed method on fiteen datasets, where each to determine if a dataset has or does not have temporal attribute was given a priori classification by domain exdependence. perts. We demonstrated that our approach was able to discover all the temporal attributes.
The paper is organized as follows. Section 2 presents the theoretical background that forms the basis of our work. Section 3 discusses our proposed approach to de- about evaluating the fitness of an autoregressive model, tect temporal dependencies in datasets. Sections 4 and 5 describe our experiments and results using 15 diferent datasets. Section 7 concludes the paper.
A time series is a sequence of observations equally spaced and ordered by time [11]. Normally, these observations are not independent from each other because their relative order is important. This non-independence means that there is a temporal dependence implying that future values are influenced by past values. The classical approach for analyzing temporal series is to consider them as a combination of four components. This combination can be additive or multiplicative: where is a temporal series.
A secular trend (Trend in Eqs. 1 and 2) describes the consistent tendency of the data over a long period. A seasonal variation (Seasonal in Eqs. 1 and 2) describes the periodic fluctuation within cycles. The cyclical component (referred to as Cyclical in Eqs. 1 and 2) describes to longer periodic fluctuations. The irregular component (Irregular in Eqs. 1 and 2) describes small changes that are unpredictable.
Finally, autocorrelation is a measure of the similarity of the observations at certain lag, that is, the correlation = + + + = ⋅ ⋅ ⋅ (1) (2)
which is done by analyzing the autocorrelation of the model’s residuals. However, because we do not have prior knowledge about the data we are unable to apply these methods which require certain assumptions [13]. The most popular methods are Ljung-Box [14], Box
Peña [17] and Monte-Carlo [18] which overcomes some of the limitations of the first two [ 14, 15] but are more focused on time-series model’s residuals. Both Ljung-Box and Box-Pierce methods are portmanteau tests which allows testing the autocorrelation of a time series at multiple lags at the same time. The null hypothesis of the test is that the data is independently distributed while the alternative hypothesis is that the data exhibits serial correlation up to any lag. The distribution of the tests approximates asymptotically to a 2 and the rejection of the null hypothesis will indicate to us that there is autocorrelation in our data.
The method that we used is Ljung-Box, which is a modification of Box-Pierce and it approximates better to a 2 [14]. The formula is: () = ( + 2) ⋅ (3) ∑ =1 − 2 where is the number of samples, is the maximum lag to test for autocorrelation, and is the autocorrelation.
The degree of freedom of the 2, when there is no other information about the data, should be equal to the number of lags up to where the autocorrelation is being tested.
no information about the data is known. The higher the lag the lower the performance of the test. Also, the lag should be a fraction of the sequence length.
rule of m=min(n/2,40), while Box et al. [20] suggest m=20, and Tsay [21] suggests m=ln(n) warning that when seasonal behavior is expected, this behavior needs to be taken into consideration and lag values at multiples of the seasonality are more important.
In stage A, we do nested iterations over all the numeric attributes and all their unique values. We group the dataset by those values and classify all other attributes as time-dependent or not. As an example, using dataset from Figure 1, while we are at the iteration of the attribute ‘county’, we group by ‘county’, and for each group, we classify the other attributes (‘date’, ‘cases’, and ‘deaths’) as temporal or not. The following pseudo code describes the process, which computes a set of metrics we analyze in stage B using a decision tree to determine the temporal attributes.
for each attribute A do for each unique value x of A do smallDB = SELECT * WHERE A = x; classification(smallDB);
Based on possible temporal characteristics, we categorized datasets into three types.
• No temporal dependence: Given a dataset with no temporal information, no autocorrelation is expected. end • A continuous evenly-sampled time-ordered end dataset: Given a dataset that corresponds to a single time window, we can detect the temporal The classification part of stage A is diagrammed in dependence by computing the autocorrelation of Figure 3. It consists of analyzing a single attribute and each attribute over the entire dataset. determining if it has autocorrelation. We do a Ljung-Box • Temporal dependence within a grouping at- test to detect statistically significant autocorrelation. In tribute: There is no observable temporal depen- parallel, we apply a threshold (0.5 in our examples) to dence when the dataset is considered as a whole, determine if the autocorrelation is also quantitatively but the temporal dependence becomes appar- significant for the specific posterior use of the dataset. If ent when grouped by some attribute. In such both tests pass, we consider the sequence to have tempoa case, we can detect the temporal dependence by ral dependency. computing the autocorrelation of each attribute The metrics outputted on stage A consists of a table within each group. Finding the proper grouping showing statistics of all the classification when grouping attribute is the main challenge in this case. the dataset by each attribute. The rows are the attributes of the dataset and the columns are the metrics described
Figure 1 exemplifies the third case, where a dataset in Table 1. To address the first and second types of dataset may have hidden temporal dependencies that are uncov- described at the beginning of this section, we add a row ered once the proper attribute is used to form groups. On consisting of no-grouping-by-any-attribute, where we the left, the entire dataset does not exhibit any autocorre- show the classification of the attributes if no grouping is lation for any of the attributes. On the other hand, on the done. As an example of how the metrics are computed, right, after grouping by attribute ‘county’ the attributes let us consider the dataset from Figure 1. First, we group ‘deaths’ and ‘cases’ correspond to temporal series. by ‘date’ and classify each attribute as temporal or not. Our Algorithm In this case, in none of the groups the attributes were classified as temporal. Next, we group by ‘county’ and
resulting table to determine if grouping by attributes gen- We conduct diferent experiments to show how the meterates temporal sequences. We designed a decision tree, rics we defined can help determine if there is temporal shown in Figure 5 to guide the analysis of the table. The dependence in the dataset. We run the algorithm against tree first discards attributes with small percentage of data
which consists of reported votes by county in the governor race in the US elections 2020 (Figure 8). It has 1025 entries, 2 non-numeric attributes, and 3 numerical attributes, none of which has temporal dependence.
Figure 9 shows that no autocorrelation was found, as expected.
One of the limitations of using autocorrelation, as we will discuss in Section 6.1, is that other types of relationships can also produce correlation. To illustrate this, we used the ‘biomechanical’ dataset (Figure 10), for which there are two false positives based on the result table of Figure 7. The dataset consist of six biomechanical attributes derived from the shape and orientation of the pelvis and lumbar spine of 310 patients. Despite the lack of temporal dependence in the data, the results, as shown in Figure 11, indicates the presence of autocorrelation
as expected are both the number of cases and deaths, as shown in Figure 14.
whether or not they contain temporally dependent data.
The core of our algorithm is based on the autocorrelation as the method to determine if there is a temporal dependence in a section of the data. Our algorithm relies on a set of proposed metrics to integrally classify the dataset. • Cross-sectional data: when there are dependence Among these metrics, the percentage of data used in other than temporal between attribute values. the analysis, the number of groups, and the average of Even though autocorrelation is a necessary condiautocorrelated sequences found were the three metrics tion to exploit temporal data information, it is not that provided the most relevant information for making a suficient condition to determine if the data is a decision. The other metrics were not used in any of temporal. For example, our method will fail when the examples but we believe that they could come handy a dataset has correlations that are not temporal in larger datasets. For example, the standard deviation but spatial [26]. should not be too large as it would mean that there is a • Non-stationarity: when time-series statistical particular grouping attribute value with more temporal properties vary over time. In such cases, the autosequences than the rest, which is probably as a result correlation cannot be calculated using the mean of an outlier, and should be handled carefully to avoid a and the variance but needs to be estimated. Simifalse positive. Both the average autocorrelation and the lar methods could be used as when dealing with maximum autocorrelation are used as tiebreakers when missing values [27]. the other metrics have same values.
Our approach could identify temporal sequences, The decision tree to analyze the metrics is currently when the sequence corresponds to the entire dataset, not automated as we require a higher volume of use cases and also when grouping by attributes was needed. Typi- to generalize the rules. Similarly, for tuning the hyperpacal datasets fall in both cases, meaning that an attribute rameters, such as the autocorrelation threshold we used can present autocorrelation as a whole sequence and as to determine if an autocorrelation was significant, we multiple grouped subsequences. The latter case is impor- need more extensive analysis and cases. tant because it allows to improve the data analysis. For example, if we have an outlier detection algorithm for 6.2. Future work temporal data, we may apply that to a single sequence as well as to diferent subsequences constructed from the same data, to increase the chance of detecting more outliers. Another use case is when the algorithm has high time complexity. In such a case, it may be better to only explore the outliers in the smaller subsequences than in the entire sequence.
Statistical exploration and optimization We will investigate whether diferent types of correlations, such as Pearson, Kendall, Spearman, and estimation from the power spectral density can be used within the LjungBox or Box-Pierce test [28]. We will conduct a deep analysis on which autocorrelation function to use when no prior information on the data is known. Currently, the algorithm goes over all numeric attributes searching for autocorrelation. This is time consuming and should be, if possible, improved.
Working with categorical attributes Datasets may • Small sample size: when the number of samples consist of categorical attributes, such as boolean labels, is small, no statistical test will have enough sig- names, IDs, and dates. These attributes may be temporal nificance. as well. For example, a positive value for a patient with a • Unevenly-sampled data: when there is no con- non-curable disease is unlikely to become negative in the stant time-spacing between samples. If the un- future. Thus, finding a way to process such attributes is even sampling is due to missing data points and important. We will use one-hot encoding to pre-process the sample size is large enough, the approach the categorical attributes. should converge to the same values as if all data points were present. However, if there is no 7. Conclusions pattern in the sampling rate, diferent methods should be used to calculate the autocorrelation In this paper, we have presented a technique that uses indirectly, such as estimating the autocorrelation autocorrelation to determine the presence of temporal using the statistical approaches [23]. data within its attributes without any prior knowledge • Missing values: when there are null values in the about the database. The algorithm was tested for diferent data. there are many methods [24] to overcome databases, including those with and without temporal missing values in time-series data and specifically dependence data, and specifically focused on databases for the Ljung-Box test[25]. However, under the containing hidden temporal groups. For these cases, we assumption that we do not have prior information proposed metrics to find the grouping attributes that on the data, none of these methods can be used. unveil such hidden groups. The results show that we [12] A. Dotis-Georgiou, Autocorrelation in Time-series were able to successfully classified attributes as temporal Data, 2019. URL: https://www.influxdata.com/blog/ or not, and also to find grouping attributes that form autocorrelation-in-time-series-data/, influx Data temporal groups. Finally, we discussed the limitations of article, accessed 25th July 2021. the approach and potential improvement paths. [13] G. Maddala, Introduction to Econometrics, Wiley, 2001. [14] G. Ljung, G. Box, On a Measure of Lack of Fit in Acknowledgments Time Series Models, Biometrika 65 (1978). [15] G. Box, D. Pierce, Distribution of Residual AutoThis work was supported in part by funding from NSF correlations in Autoregressive-Integrated Moving under Award Numbers CNS 1822118, IIS 2027750, OAC Average Time Series Models, Journal of the Ameri1931363, Statnett, ARL, AMI, Cyber Risk Research, and can Statistical Association 72 (1970) 397–402. NIST. [16] D. Scott, Applied Econometrics with R by Christian Kleiber, Achim Zeileis, International Statistical References Review 77 (2009) 164–164. [17] D. Peña, J. Rodríguez, A Powerful Portmanteau [1] P. J. Brockwell, R. A. Davis, Introduction to Time Test of Lack of Fit for Time Series, Journal of the
Series and Forecasting, Springer, 2008. American Statistical Association 97 (2002) 601–610. [2] H. Luetkepohl, M. Krätzig, In Applied Time Series [18] J.-M. Dufour, L. Khalaf, Monte Carlo Test Methods Econometrics, Applied Time Series Econometrics in Econometrics, 2007.
(2004). [19] S. Documentation, wntestq Portmanteau (Q) Test [3] M. Allen, The SAGE Encyclopedia of Communica- Description, 2019. URL: http://www.stata.com/ tion Research Methods, SAGE Publications, 2017. manuals13/tswntestq.pdf, accessed on 24th July [4] Y. Chen, W. Wu, Application of One-class Sup- 2021.
port Vector Machine to Quickly Identify Multivari- [20] E. Ziegel, G. Box, G. Jenkins, G. Reinsel, Time series ate Anomalies from Geochemical Exploration Data, analysis, forecasting, and control, Technometrics Geochemistry: Exploration, Environment, Analysis 37 (1995) 238.
17 (2017) 231–238. [21] R. Tsay, Analysis of Financial Time Series. Financial [5] Z. Cheng, C. Zou, J. Dong, Outlier Detection Using Econometrics, 2002.
Isolation Forest and Local Outlier Factor, in: Con- [22] J. C. Escanciano, I. N. Lobato, An automatic Portference on Research in Adaptive and Convergent manteau Test for Serial Correlation, Journal of Systems, Association for Computing Machinery, Econometrics 151 (2009) 140–149. 2019, p. 161–168. [23] K. Rehfeld, N. Marwan, J. Heitzig, J. Kurths, Com[6] H. Lu, Y. Liu, Z. Fei, C. Guan, An Outlier Detec- parison of Correlation Analysis Techniques for Irtion Algorithm based on Cross-Correlation Analy- regularly Sampled Time Series, Nonlinear Processes sis for Time Series Dataset, IEEE Access 6 (2018) in Geophysics 18 (2011) 389–404.
53593–53610. [24] I. Pratama, A. E. Permanasari, I. Ardiyanto, R. In[7] P. M. Maçaira, A. M. T. Thomé, F. L. C. Oliveira, drayani, A Review of Missing Values Handling A. L. C. Ferrer, Time Series Analysis with Explana- Methods on Time-series Data, in: 2016 Internatory Variables: A Systematic Literature Review, En- tional Conference on Information Technology Sysvironmental Modelling & Software 107 (2018) 199 – tems and Innovation (ICITSI), 2016, pp. 1–6. 209. [25] D. Stofer, C. Toloi, A Note on the Ljung– [8] Y. Yu, X. Si, C. Hu, J. Zhang, A Review of Recurrent Box–Pierce Portmanteau Statistic with Missing Neural Networks: LSTM Cells and Network Archi- Data, Statistics & Probability Letters 13 (1992) tectures, Neural Computation 31 (2019) 1235–1270. 391–396. [9] W. Lin, M. Orgun, G. Williams, An overview of [26] A. Zuur, Spatial Correlation, 2019. URL: temporal data mining (2019). http://userwww.sfsu.edu/efc/classes/biol710/ [10] H. Homayouni, S. Ghosh, I. Ray, An Approach spatial/spat-auto.htm, san Fransisco State Univerfor Testing the Extract-Transform-Load Process in sity article, accessed on 24th June 2021. Data Warehouse Systems, in: Proceedings of the [27] G. P. Nason, R. Von Sachs, G. Kroisandt, Wavelet 22nd International Database Engineering and Ap- processes and adaptive estimation of the evolutionplications Symposium, IDEAS 2018, Association for ary wavelet spectrum, Journal of the Royal StatistiComputing Machinery, 2018, p. 236–245. cal Society Series B 62 (2000) 271–292. [11] W. Wei, Time Series Analysis: Univariate and Mul- [28] C. Chatfield, The Analysis of Time Series: An Intivariate Methods, volume 33, 1989. troduction, Fourth Edition, Chapman & Hall/CRC
DB elections incomes countries biomechanical crime covid1 energy yahoo india exchange covid2 wage market avocado suicides Description General information about reporting votes to governor race by county.
American citizens incomes from 2015 broken into male and female statistics.
Information on population, region, area size, infant mortality and more.
Patient data of six biomechanical attributes derived from the shape and orientation of the pelvis and lumbar spine.
FBI crime statistics for 2012 on population less than 250,000.
Covid cases and death statistics for USA. Daily energy delivery by Fort Collins power facility. Real and synthetic time-series. The synthetic dataset consists of time-series with varying trend, noise and seasonality. The real dataset consists of time-series representing the metrics of various Yahoo services.
Population of india by year.
Exchange currencies by year.
Covid cases and death by County in the USA. Indian markets quantity and price per year.