Anomaly detection uncovers any abnormal action in the given data set for a different activity. It is an activity to finding patterns that do not conform to expected behavior. Several activities, such as the hourly temperature of a place or the number of users who visit a website per minute, have a regular pattern or waveform that depicts normal behavior. Any deviation from this normal or common pattern means that there is something unusual happening. Several methods have been used to detect anomalies/abnormal behavior, such as clustering through finding the underlying patterns or structures within a collection of data to form different clusters. This article describes an anomaly as a particular wave shape that has never been seen before. A library of normal waveforms is created and later used to reconstruct a waveform that is being tested. The "normal" waveforms are determined using the K-means clustering algorithm to group/cluster the similar waveforms. The experiments resulted in a reconstruction error of 23.8%, which indicated there was something abnormal.
Anomaly detection is a method used to recognize abnormal patterns that do not match expected behavior, called outliers. Nowadays, a massive amount of data and software available on social platforms are available to ensure service. This will be required to systematically monitor the data to detect an abnormal event such as Detection of abnormal health conditions, intrusion detection, and fraud detection.
Numerous types of algorithms have been developed for anomaly detection for twin academic research, commercial interest, and the use of all this to protect such a specific application or device.
In most kinds of records such as temperature, visits on a website, etc. there is usually a normal accepted behavior indicating that everything is working fine. It might be hard to determine an anomaly based on an instantaneous value of a waveform in some cases. Instead, it's better to consider the shape of the waveform as discussed in this paper.
A library of different waveforms is created, and with the use of k-means clustering, we can develop what constitutes the normal shape of a waveform. The dataset used is plotted to form a wave of a particular shape. The waveform using the waveform library is reconstructed. If there is any poor reconstruction, then the wave is certainly containing some abnormalities.
Jianliang, Haikun, and Ling [
They perform data processing for more extensive features in two ways; first, they calculate the
mean absolute deviation [
Xinlong and Weishi [
Clustering is “the process that groups objects into subclasses so that those subclasses are entirely
related or have some similarities and the different clusters are altogether dissimilar from each other”
[
3.1.
Partitioning clustering [
A hierarchical clustering [
Density-based algorithm [
Grid-based clustering [
Partitioning is done to divide a group of data into k-cluster [
Considering the d-dimension data set { | ∈ , = 1,2, … } , the method to follow easy steps to classify given dataset, number of clusters , , … . to define K centroid = , , … . , one for each group, = ∑ ∈
Select all points closest to a considering dataset and relate them to the nearest centroid. When a
topic does not stand out, the first steps compete [
A new obligate done between the current dataset and closest new centroid, an iterative loop has been generated due to this loop centroid change their location slowly until no change is done .lastly algorithm minimizing an objective function.
The objective function = ∑ ∑ , where ,
and cluster center . 3.6.
Step1: (initialize) randomly choose K occurrence , , … .
Step2: (allotment) provide each occurrence to the nearest center: , < ( , ) i.e., = 1,2 … . . & = 1,2 … … ≠ , = 1,2 … . . and then allocate Step3: (updating) Recalculate the centroid of the cluster ∗, ∗, … … . . ∗ ; Step4: (loop) if = {1 … . . } ∗ = then stop the algorithm and initial ∗, ∗, … … . . ∗ represent the end group, or else allocate = ∗ & perform step 2 and 3 until there isn't any more modification 3.7.
K-mean algorithm is essential for a considerable large dataset [
With the k-mean algorithm, the cluster number is required first, yet the number of clusters is
usually determined later. It is sensitive to outliers [
It is not easy to predict k-value, the k-mean algorithm with global cluster does not work well, and it does not perform well with a different group and size
In our approach, we define an anomaly as a shape of the waveform that has not been experienced or seen last. The algorithm will build a normal shape library and reconstruct the waveform and determine its shape. The waveform is said to have an abnormal behavior if the reconstruction of the waveform by the library is inadequate. 4.1.
In this paper, the data used for the experiments is the EKG dataset publicly available at PhysioNet. This version of the data provides a regular waveform which provides a reasonable basis for exploring the proposed algorithm.
To fully ascertain what constitutes a waveform's normal shape, we use k-means clustering to form different clusters to determine a normal cluster group. The clusters can be in separate n-dimensional space that’s defined by n-features. In our case, the clusters are determined programmatically using the clustering algorithm. 4.3.
We shall consider a 32-dimensional waveform space to perform anomaly detection such that for every point within the space, it potentially represents a waveform segment. We cluster the segments that are similar to each other. “The middle of each cluster (the centroid) will measure the prototypical waveform pattern that all those segments are specific instances of”. Still, where necessary, from all the waveforms, the average of them within a cluster will be considered as the centroid.
Since the center of the cluster is also a point that is part of the waveform space, this makes it also “waveform”. This therefore means that, the centroids of the clusters will constitute a set of “normal waveform segments”.
Now using these normal segments, we try to reconstruct the dataset being tested. The reconstruction will be considered as good if the data is similar to the original. However, suppose there is some abnormal shape in the data. In that case, it won't be possible to reconstruct the waveform using the “normal” waveform library, which then results into a reconstruction error. This therefore indicates that there is some anomaly.
The algorithm is divided into two parts; 4.4.
Divide the waveform data into segments of say n samples.
Create a space of n dimensions with each of the segments considered as one point. Cluster the segment point and also determine the centroids of the clusters. Cluster centroids provide a library of normal waveform shapes. 4.4.1. Testing
Using the cluster centroids in the training phase, we try to reconstruct the waveform data.
An abnormal shape is obtained if there is any reconstruction error on any segment. 4.5.
Let FN be the filename of the data set. Let DS be the variable that holds the data after reading it using the ekg_data primitive function.
Using matplotlib.pyplot function and sample size of 300, plot the data. Label the x and y axes, then show the plotted graph.
Let seg_len be the segment length of value 32, slide_len of value two, and an array named segments to store them. Loop through the DS dataset to split it into waveform segments.
Perform clustering of the segments in a 32-dimensional space using the k-means algorithm provided in python's library scikit-learn.
The different clusters created at this point then constitutes the normal waveform shapes of the library
Try to reconstruct the data or the waveform, which will be tested using the learned library. The reconstruction process consists of four sub-steps which include;
First, the data is split into segments that are overlapping.
Secondly, we determine the segment's cluster centroid by using the samples' average in a cluster.
Using the centroid from the above step, we reconstruct the segment.
Finally, we join all the segments to form the waveform.
We decide whether there is an anomaly or not based on the reconstruction error obtained. If there is a very high reconstruction error, then we conclude that there is an anomaly.
Using the EKG dataset, we performed clustering on the data and also try to reconstruct to find the error between the original plotted data and the newly revamped data.
The new shape formed indicates an anomaly detected after reconstruction of the data since there is a substantial visible error. The reconstruction error obtained was 23.8%.
The figure below shows the original data's waveform, the reconstructed information, and the error after reconstruction.
Anomaly detection is essential in everyday lives, and therefore it's vital to expand on the research continuously.
Every different kind of data consists of a regular pattern that depicts normal behavior. When the data are plotted to form a waveform, it can also be hard to determine an anomaly of a waveform based on its instantaneous value but by its shape.
This paper sees how K-means clustering performs a key role in identifying normal waveform shapes in time series data to form trained library waveforms. Based on the data's reconstruction, we determine the shape of the waveform after reconstruction tested by the library created. The error obtained signifies that there is an anomaly since the shape of the waveform has not been seen before.
Special Appreciation to the Department of Software Engineering, Delhi Technological University, for supporting this work.