This paper describes analysis results of system log Anomaly Detection literature from time period of 2018 to 2023. The literature was found using keywords "log anomaly", "machine learning", "neural network". A total of 80 different scientific papers have been analyzed. It has been determined that most popular neural networks are LSTM/BiLSTM; most common datasets are HDFS, BGL and Thunderbird; Most popular evaluation metrics include F1, precision and accuracy. Most of research sought to address issues of improving model detection accuracy, lowering system resource use and making model more suitable real time detection.
As time goes by complexity and scale of software systems is rapidly increasing, which forces the necessity for new anomaly detection methods to be developed [1]. As the amount of data needed to be analyzed increases, the need to fully automate detection process increases [2]. As treats to system security become more sophisticated, the amount of needed to be analyzed data points keeps increasing as well, which at the same time makes it harder to use supervised training approach and properly interpret received data [3]. Another major issue is prevalence of 0 day exploits which are usually impossible to predict in advance [4]. All of the aforementioned issues are normally addressed through use of anomaly detection methods.
To limit the scope of research it was chosen to focus on the specific keywords: "Log", "Anomaly detection", "machine learning", "neural networks". arxiv.org[5] and sciencedirect.com [6] databases have been used for research paper collection. A total of 117 different research papers have been analyzed. All papers have been written during 2000-2023 time period with 78 of the papers being from 2018-2023 time period. Only research and conference papers have been analyzed.
For the purposes of this paper the following has been chosen to analyze:
1. Which neural network and machine learning approaches are being used? 2. What metrics have been used to evaluate suggested approaches and how do different approaches compare to each other? 3. Which data sets are being used to train models? 4. What problems in anomaly detection have been identified? 5. What findings/conclusions have been made?
1) Anomaly detection: It is an approach seeking to identify unusual events based on comparisons to standard situation. The anomalous event is something which cannot be fully anticipated in advance and as result cannot be detected via traditional pattern based detection methods. To declare an anomaly an outlier needs to be found. This outlier could appear through various contexts like statistical outlier, situation/sequence outlier, timing outlier and so on… It is usually assumed that the amount of anomalous data is much less numerous than normal data. Most popular approach to solving anomaly detection problems is use of semi supervised training, where models are trained exclusively on normal data [3].
2) Log data: this is information gathered in sequential order and presented in lines. Each log entry contains all the necessary information to identify various system states at given time moments. Data is usually saved in either string or numerical values and is saved in easily readable text files. By following log entries it should be possible to reconstruct how system continuously functioned in the past, so if system deviates from expected behavior, log analysis should identify the moment of system malfunction.
Log data can be used to determine in advance if there are any risks for system failure and also can be used to detect possible intrusions. In order to achieve this, multiple data entries need to be analyzed at once in order to identify any abnormal patterns [3].
3) Neural Networks are subset of Artificial Intelligence (AI) research. They are algorithms based on neuroscience seeking to replicate function of human brain. These networks consist of many input units, which are arranged in sets of layers. Initially preproccessed data is fed to initial layer and after performing initial data transformations, layer results are passed to subsequent layers7. Over time Neural Network discovers patterns within its data and then can use it to classify data into various categories. 4) Machine Learning (ML) is a subset of AI research, seeking to imitate human intellect through self learning algorithms. Firstly it is provided with preprocessed data, then a chosen model is applied to discover any meaningful patterns within given data [8]. The given data can either be labeled to enhance model accuracy, which is called "Supervised Learning". In case of Unsupervised training provided data is unlabeled and patterns need to be discovered using statistical methods.
When compared to neural networks, classical, or "non-deep", machine learning is more dependent on human intervention to learn. Human experts determine the set of features to understand the differences between data inputs, usually requiring more structured data to learn [9]. Traditional machine learning methods include Isolation Forest, SVM, kNN, Naive Bayes, Polynomial/Linear Regression, PCA and other methods.
Table 1 showcases the amount of publications released during recent years. Publication amount is the exact number of research papers released during that year. Any papers which also include research into neural network use are counted as well. It can be said that during recent 5 years the anomaly detection field has received an increased amount of attention from the research community. During last 3 year period majority of written literature covers Neural Network methods and standard machine learning methods (like Knn, decision trees, SVM…) are becoming less popular. Table 3 provides the list of all commonly used machine learning methods. Any method which only has been used once within researched literature has been included in "other" category. It has been determined that SVM is the most frequently used machine learning method. Its primary advantage over Neural Networks is its significantly faster computational speed, which is important when it's necessary to detect system anomalies as soon as possible. Some other notable advantages include ability to handle high dimensional data and low risk of overfitting [11]. HDFS is a key component of Hadoop, offering reliable storage through data replication, integrates with big data frameworks and supports batch processing [12]. Within reviewed literature it appeared the most frequently and often was simultaneously used with BGL and Thundebird [13], both of which are popular supercomputer log datasets. Table 5 showcases all frequently used evaluation metrics. Any research metric which has only been used once within all research papers is included in "other" category. It has been determined that F1 Score (Formula 1) was the most commonly used evaluation metric. This metric is calculated using Recall (Formula 2) and Precision (Formula 3) metrics, so in most of research papers all 3 metrics have been used simultaneously. Within these formulas True Positive stands for all correctly identified elements, False Negative stands for all elements which have been incorrectly labeled as false, False Positive are all elements incorrectly labeled as true.
Precision is a good way of determining reliability of individual results which helps to minimize the risk of spending unnecessary resources on managing false alarms. Recall is useful for determining how much of an impact false negatives might have which is very important as all it takes is one missed anomaly to cause massive system damage. As both Precision and Recall are important, F1 ensures that both of them can be represented using a single metric [14].
True Postive True Positive+ False Positive (3) Table 6 lists most common anomaly detection problems described within research papers. Any problem which has only been mentioned once has been assigned to "other" category. The largest concern specified by research literature is that due to increase in data amount, the extent to which log data analysis should be automated should increase as well [15][16] [17]. Another major issue being brought up is that by itself log data does not include a sufficient amount of data to effectively determine new treats [18]. Often while relying on log data, only time context is established and additional data context is ignored [19]. Further issues could also be introduced while parsing log data, which could further degrade anomaly detection accuracy [20]. One of the main requirements for successful anomaly detection is timely discovery of new treats. In order to comply with it and provide near real time detection, some necessary compromises need to be done. For example often this means only relying on most simple log data analysis and ignoring additional system analysis tools [21][22]. Furthermore state of the art anomaly detection methods with highest detection accuracy are usually unfit for time sensitive issue detection [23]. Another concern is that due to amount of information needed to be processed, cloud computing becomes necessary, which introduces issues of data transfer speeds [24][25]. To add on top of that due to software updates, models designed for previous software versions might severely degrade in accuracy [26].
Some additional issues being brought up in literature included having difficulty to perform simultaneous parallel analysis when each input is part of time series and requires proper understanding of its context [27]; not all problems might be reflected within logs and the issues of software program itself might be overlooked [28]; anomaly detection methods do not get sufficiently compared to each other [29];
traditional machine learning methods such as SVM are unable to perform sufficiently accurate analysis of temporal information of discrete log messages [30]; Certain anomaly detection models have not been sufficiently tested in real life application [31]; models based on statistical methods might be insensitive to importance of log entry order sequences [32].
The following were the main findings of analyzed literature:
1.Embedding multi-core point-by-point convolution and global average pooling achieves significant advantages in terms of arithmetic power, memory and high availability, while ensuring detection accuracy [23].
2.Gumbel Noise Score Matching model demonstrated the capability of score matching for anomaly detection on categorical types in both tabular and image datasets. It also provided a unified framework for modeling mixed data types via score matching [33].
3.In transformer based models adapter-based tuning consistently outperforms training and fine-tuning models [16].
4.Dividing log events into dependent and independent types is an effective way to boost model accuracy [17].
5.Taking a character-based approach to process log events (lines) contributes to higher performance as the model may take advantage of characters deleted in word-based approaches, such as numbers and punctuation. Merging the parser, vectorizer, and classifier components into one deep neural network, allows model to learn log data at the language level [34].
6.Models trained on multi-project datasets are not only more accurate in standard tests but also more robust to sequence evolutions and more accurate in ahead of time anomaly predictions [34].
7.Though the presence of critical logs often indicates problems, their absence does not necessarily imply a healthy system status. An important reason is that sometimes determining where and how to place an informative log statement is difficult. In some cases, faults do not affect metrics, while in other cases, metrics exhibit unusual patterns (e.g., jitters) even if the system is experiencing minor performance fluctuations instead of faults. Hence, simply identifying anomalous metric patterns is insufficient [1].
8.Faults can cause unexpected behaviors involving either logs or metrics, or both of them. So the two data sources should be analyzed comprehensively to reveal the actual anomalies [1].
9. Intrinsic structure of host-based logs, as captured by persistence images and the spectrum of graph and hypergraph Laplacians, contains discriminative information about whether or not the logs are anomalous [35].
10. Data augmentation can simulate deviations in log data that occur from service updates over time which contribute to successful anomaly detection [25].
11. Multimodal approach can improve the scores for anomaly detection for multiple modalities in comparison to the single modalities of logs and traces [36].
12. Filtering out common log entries can noticeably improve anomaly detection accuracy [37].
During this survey it has been determined that over recent years the popularity of this topic has been increasing. The problems identified within research papers still need to be addressed and no universal solution has been discovered which would allow anomaly detection methods to keep up with ever increasing amount of generated log data and general increasing complexity of system software. It has also been determined that neural networks are continuously increasing in popularity, while traditional machine learning methods are becoming less popular. It has been determined that the most popular neural network model is LSTM/BiLSTM, most commonly used dataset is HDFS and most frequently used evaluation metric is F1 score.