When accidents, disasters, or other large-scale events occur, they significantly disrupt trafic, leading to congestion and reduced mobility. To efectively address these issues, it is crucial to precisely detect the underlying causes of these disruptions through the analysis of human mobility data. A common approach in anomaly detection is to employ dimensionality reduction techniques to compute reconstruction errors. However, the reconstruction errors generated by traditional methods are influenced by the correlations among features, which may obscure the true causes of anomalies. To overcome this limitation, we introduce an approach that calculates the SHAP (SHapley Additive exPlanations) values associated with the reconstruction errors resulting from dimensionality reduction. We conducted experiments using a dataset of human mobility patterns to evaluate the efectiveness of this method. The results demonstrate that our approach provides a more accurate explanation of anomalies compared to conventional methods.
Anomaly detection in spatial-temporal population data has gained significant attention in recent years due to its potential applications in urban planning, disaster management, and public safety. Accurate identification of unusual patterns in human mobility can help authorities respond more efectively to disruptive events such as accidents and disasters. However traditional anomaly detection methods often face challenges in capturing the complex spatial and temporal dependencies in high dimensional population data.
Previous studies have explored various approaches to
anomaly detection in spatial temporal data. Ochiai et al.[
On the other hand, Takeishi[
Building upon the insights from Ochiai et al. and Takeishi, we propose a novel anomaly detection framework that combines the strengths of both approaches. Our method integrates SHAP (SHapley Additive exPlanations) values with dimensionality reduction to identify and explain anomalies in spatial-temporal population data. By leveraging the explanatory power of SHAP values, we aim to improve the accuracy and interpretability of anomaly detection results, while also extending the applicability of Takeishi’s approach to high-dimensional data.
The main contributions of this paper are as follows: • We introduce a novel anomaly detection method that integrates dimensionality reduction with SHAP values to identify anomalies in spatialtemporal population data. • We evaluate the efectiveness of our approach using a real-world dataset of human mobility patterns in a major urban area, demonstrating its superiority compared to traditional reconstruction error-based methods. • We extend the applicability of Takeishi’s Shapley value-based approach to high-dimensional spatialtemporal data, enhancing its utility for real-world scenarios.
The remainder of this paper is organized as follows.
Section 2 provides an overview of related work in anomaly
detection and spatial-temporal data analysis. Section 3
describes our proposed methodology in detail. Section 4
presents the experimental setup. Section 5 discusses the
experimental results. Finally, Section 6 concludes the paper
and discusses future research directions.
The detection of anomalies in urban population flows has
been extensively explored using diverse data sources,
including road sensors[
This research leverages population data extracted from
communication logs between mobile devices and base
stations to enhance anomaly detection capabilities. In contrast
to road sensors and surveillance cameras, mobile device data
captures a wide array of individual behaviors throughout the
entire city, thus ofering a more comprehensive solution for
detecting anomalies. For instance, Yabe et al.[
This section details our proposed methodology, SHAP Values of Reconstruction Error, for anomaly detection, incorporating SHAP values derived from reconstruction errors. We begin by describing Principal Component Analysis (PCA) as the foundation for dimensionality reduction. Subsequently, we outline the traditional method based on reconstruction errors. Then, we introduce our enhanced approach that integrates SHAP values to improve the accuracy and effectiveness of anomaly detection. Finally, we explain the calculation of SHAP values for multi-dimensional objective variables, extending the methodology to more complex scenarios.
PCA is a widely-used technique for reducing the dimensionality of high-dimensional data while preserving the most significant features. By projecting the data onto a lowerdimensional space, PCA identifies the principal components that capture the maximum variance in the data.
Given a dataset X ∈ R× with samples and features, the PCA process involves the following steps: 1. Standardize the Data: Subtract the mean of each feature from the dataset to center the data around the origin. 2. Compute the Covariance Matrix: Calculate the covariance matrix C = 1 X X. 3. Perform Eigenvalue Decomposition: Decompose the covariance matrix into eigenvalues and eigenvectors: C = VΛV , where Λ is a diagonal matrix containing the eigenvalues, and V is a matrix whose columns are the corresponding eigenvectors. 4. Select Principal Components: Choose the top eigenvectors corresponding to the largest eigenvalues to form the principal components. These components maximize the variance retained in the lower-dimensional space.
In this study, we set the threshold for the variance to be retained at 90%. This means that we select the number of principal components such that the cumulative variance explained by these components is at least 90%. By retaining the principal components that explain the majority of the variance, PCA ensures that the most important features of the data are preserved, allowing for efective dimensionality reduction and subsequent analysis.
Using the principal components obtained from PCA, we can perform dimensionality reduction and reconstruction. Consider a test data vector y ∈ R. The reduced representation y ∈ R is obtained using the mapping function : R → R, and the reconstructed vector yˆ∈Risc o:mRpu→tedRusainsgfotlhloewrse:construction function yˆ = ((y))
The reconstruction error y, a measure of fidelity of reconstruction, is defined as the squared Euclidean distance between the original and reconstructed vectors: y = ‖y − yˆ‖22 = ∑︁( − ˆ)2 =1 This error metric helps identify significant deviations from normal patterns, which are potential indicators of anomalies.
Additionally, the reconstruction error for each feature is calculated as:
= | − ˆ| This feature-specific error is employed to identify which specific features are exhibiting anomalies.
Typically, the SHAP value () for each feature is calculated using all features of the instance as explanatory variables and the prediction ˆ as the objective variable as follows:
() = ℎ_(; ˆ, )
This formulation allows for measuring the impact of feature on the prediction ˆ, providing a concrete metric for understanding the significance of each feature in the model.
Similarly, the SHAP value of reconstruction error () is calculated using all features of the instance as explanatory variables and the reconstruction error as the objective variable using the following function:
() = ℎ_(; , )
This formulation measures the impact of feature on the reconstruction error , providing a concrete metric for evaluating the significance of each feature in anomaly detection.
for Multidimensional Obejective
The SHAP value of Reconstructtion Error () for feature ,
when the objective variable is represented in one dimension,
is calculated using the following equation [
∑︁ ⊆{ 1,...,}∖ ( − | | − 1)!||! [ (∪) − ()] ! (1) (2) (3) (4) (5) (6)
In this equation, (∪) denotes the model’s predicted value when the feature set includes feature , and ( ) represents the predicted value when the set is used without feature . || denotes the number of elements in the feature subset , and is the total number of features.
Subsequently, the SHAP Values of Reconstruction Error ()() for feature impacting the -th dimension of the objective variable is calculated as the average diference between the model predictions with and without feature , across all combinations of features, thus extending equation 6 into the multidimensional context of SHAP Values of Reconstruction Error as follows: ()() = 1 ∑︁ ⎡ ∑︁
⎣ =1 ⊆{ 1,...,}∖ [ ()(∪) − ()( )]]︁ ( − | | − 1)!||! ! (7)
This formulation allows for measuring the impact of feature across diferent combinations of features on each dimension of the recostruction error. By doing so, it provides a concrete metric for elucidating the significance of feature in anomaly detection, ofering detailed insights into the causes of anomalies in specific dimensions.
This study utilizes Mobile Spatial Statistics (MSS) [
The goal of this study is to assess the accuracy of anomaly detection in spatial-temporal population data. We compare two methodologies: a traditional approach using reconstruction errors, and a novel approach using SHAP Values of Reconstruction Errors.
During the test phase, artificial anomalies are introduced by altering population figures within selected grids. For each timestamp , a grid is chosen randomly, and , is modified to the maximum or minimum value seen during the training period, defined as: , = {︃max(,′ : ′ ∈ train), if max anomaly min(,′ : ′ ∈ train), if min anomaly
The eficacy of each detection method is quantified using the Hits@ metric, which determines if the true anomalous grid is among the top ranks based on anomaly scores. The scores are calculated using the following equations: = | − ˆ| (as defined in Equation 3) (8) ()() = 1 ∑︁ ⎡ ∑︁
⎣ =1 ⊆{ 1,...,}∖ [ ()(∪) − ()( )]]︁ ( − | | − 1)!||!
! (as defined in Equation 7)
Rankings for each grid are obtained by comparing these scores against all others in the dataset.
Hits@k Calculations For both methods, Hits@k is deifned and calculated separately for each method to assess the eficacy in identifying the true anomalies within the top ranks of predicted anomalies. The total number of test instances, denoted as , is used to normalize the calculations, ensuring that the results are proportional to the size of the test dataset. The calculations are as follows: where 1(· ) is the indicator function, and represents the total number of test instances. These metrics facilitate a direct comparison of the methods’ efectiveness in accurately detecting anomalies.
This study utilizes Mobile Spatial Statistics data generated
from communication records between NTT DOCOMO’s
base stations and mobile devices. This data is divided into
mesh units across Japan in accordance with the Regional
Mesh standards provided by the Ministry of Internal
Affairs and Communications Statistics Bureau[
The training data consists of population records with a 10minute resolution from October 17 and October 24, 2022, totaling 288 instances (6× 24× 2), were prepared. A PCA model was trained with these data, setting the dimensionality reduction to retain 90% of the variance.
The test data consists of 144 population records (6× 24) from October 31, 2022, matching the same month and day of the week as the training data. Anomalies were inserted using the method described in Section 4.2.1. For each instance, one grid was randomly selected, and its population count was replaced with either the maximum or minimum population observed for that grid. A total of 288 tests were conducted to determine if the grid with the altered population could be accurately identified.
This section presents the results of anomaly detection experiments conducted using both the traditional reconstruction error method and the proposed SHAP value method. The performance of each method is illustrated through selected examples at various timestamps, as detailed in Table 1.
The analysis shows varying levels of detection accuracy for each method, with lower ranking values indicating higher precision in anomaly detection. Specifically, at 19:10 on October 31, 2022, both methods accurately detected the anomaly in grid 5339-3588-4, achieving the lowest possible rank of 1. This instance demonstrates the efectiveness of both approaches in scenarios where there is a substantial change in population, from 1.378 to -1.257.
Conversely, at 10:50 on the same day, the anomaly in grid 5339-3574-1 was detected with lower accuracy, resulting in ranks of 9 and 8 for the reconstruction error and SHAP methods, respectively. This indicates a reduced efectiveness of both methods in detecting anomalies associated with smaller changes in population, from 1.429 to 2.187.
Moreover, at 9:10, the SHAP method outperformed the reconstruction error method by more accurately identifying the anomaly in grid 5339-3584-1, with a rank of 2 compared to 6. This demonstrates the SHAP method’s enhanced ability to detect subtle yet significant changes in population, from 1.536 to 1.682.
A comprehensive evaluation using the Hits@k metric, which assesses performance under scenarios of maximum
MAX = 1 0.382 0.417 and minimum population changes, is summarized in Table 2. Hits@k values range from 0 to 1, with higher values indicating more efective anomaly detection.
For maximum population changes, the SHAP method demonstrates superior performance with Hits@k scores of 0.417 at = 1 and 0.472 at = 3, exceeding the scores of the reconstruction error method, which are 0.382 at = 1 and 0.458 at = 3. Similarly, in scenarios of minimum population changes, the SHAP method achieves better scores of 0.375 at = 1 and 0.438 at = 3, outperforming the reconstruction error method’s scores of 0.340 at = 1 and 0.410 at = 3. These findings confirm the efectiveness of the SHAP method in consistently identifying anomalies under varied conditions, highlighting its superiority over the traditional method.
This paper evaluated the performance of established reconstruction error techniques and the SHAP value method for anomaly detection in spatio-temporal population datasets. The study highlighted the SHAP method’s enhanced capability for precise anomaly identification, which is crucial for high-accuracy applications such as urban planning and emergency management. The experimental datasets were synthetically modified to include anomalies, ofering a controlled setting to examine and contrast the performance of these methods. Future research aims to extend the application of these techniques to real-world data, particularly in scenarios impacted by events like accidents, disasters, or significant public gatherings.