Anomaly Detection for System Logs Literature Overview*

Abstract

This paper describes analysis results of system log Anomaly Detection literature from time period of 2018 to 2023. The literature was found using keywords “log anomaly”, “machine learning”, “neural network”. A total of 80 different scientific papers have been analyzed. It has been determined that most popular neural networks are LSTM/BiLSTM; most common datasets are HDFS, BGL and Thunderbird; Most popular evaluation metrics include F1, precision and accuracy. Most of research sought to address issues of improving model detection accuracy, lowering system resource use and making model more suitable real time detection.

Keywords

• eol>Deep learning
• Neural networks
• Machine Learning
• Log messages
• Literature Review
• Anomaly Detection
• Cyber Security
• Classification

1. Introduction

2. Key definitions 1) Anomaly detection: It is an approach seeking to identify unusual events based on comparisons to standard situation. The anomalous event is something which cannot be fully anticipated in advance and as result cannot be detected via traditional pattern based detection methods. To declare an anomaly an outlier needs to be found. This outlier could appear through various contexts like statistical outlier, situation/sequence outlier, timing outlier and so on… It is usually assumed that the amount of anomalous data is much less numerous than normal data. Most popular approach to solving anomaly detection problems is use of semi supervised training, where models are trained exclusively on normal data[ 3 ]. 2) Log data: this is information gathered in sequential order and presented in lines. Each log entry contains all the necessary information to identify various system states at given time moments. Data is usually saved in either string or numerical values and is saved in easily readable text files. By following log entries it should be possible to reconstruct how system continuously functioned in the past, so if system deviates from expected behavior, log analysis should identify the moment of system malfunction. Log data can be used to determine in advance if there are any risks for system failure and also can be used to detect possible intrusions. In order to achieve this, multiple data entries need to be analyzed at once in order to identify any abnormal patterns[ 3 ]. 3) Neural Networks are subset of Artificial Intelligence (AI) research. They are algorithms based on neuroscience seeking to replicate function of human brain. These networks consist of many input units, which are arranged in sets of layers. Initially preproccessed data is fed to initial layer and after performing initial data transformations, layer results are passed to subsequent layers7. Over time Neural Network discovers patterns within its data and then can use it to classify data into various categories. 4) Machine Learning (ML) is a subset of AI research, seeking to imitate human intellect through self learning algorithms. Firstly it is provided with preprocessed data, then a chosen model is applied to discover any meaningful patterns within given data[ 8 ]. The given data can either be labeled to enhance model accuracy, which is called “Supervised Learning”. In case of Unsupervised training provided data is unlabeled and patterns need to be discovered using statistical methods.

When compared to neural networks, classical, or "non-deep", machine learning is more dependent on human intervention to learn. Human experts determine the set of features to understand the differences between data inputs, usually requiring more structured data to learn[ 9 ]. Traditional machine learning methods include Isolation Forest, SVM, kNN, Naive Bayes, Polynomial/Linear Regression, PCA and other methods.

3. Survey Results

It can be said that during recent 5 years the anomaly detection field has received an increased amount of attention from the research community. During last 3 year period majority of written literature covers Neural Network methods and standard machine learning methods (like Knn, decision trees, SVM…) are becoming less popular. Table 4 contains amounts of all most commonly used datasets. Industrial category refers to unnamed datasets which used specific industrial process log data. Private category includes all datasets, which cannot be disclosed due to a non disclosure agreement. Generated category includes all synthetic datasets which were generated specifically for the research study. Any dataset which didn't fall into previous 3 categories and was only mentioned once within all research papers, has been included in "other" category. HDFS is a key component of Hadoop, offering reliable storage through data replication, integrates with big data frameworks and supports batch processing[ 12 ]. Within reviewed literature it appeared the most frequently and often was simultaneously used with BGL and Thundebird[ 13 ], both of which are popular supercomputer log datasets.

Precision is a good way of determining reliability of individual results which helps to minimize the risk of spending unnecessary resources on managing false alarms. Recall is useful for determining how much of an impact false negatives might have which is very important as all it takes is one missed anomaly to cause massive system damage. As both Precision and Recall are important, F1 ensures that both of them can be represented using a single metric[ 14 ].

2⋅ Precision⋅ Recall F 1=

Precision+ Recall Recall= One of the main requirements for successful anomaly detection is timely discovery of new treats. In order to comply with it and provide near real time detection, some necessary compromises need to be done. For example often this means only relying on most simple log data analysis and ignoring additional system analysis tools[ 21 ][ 22 ]. Furthermore state of the art anomaly detection methods with highest detection accuracy are usually unfit for time sensitive issue detection[23]. Another concern is that due to amount of information needed to be processed, cloud computing becomes necessary, which introduces issues of data transfer speeds[24][25]. To add on top of that due to software updates, models designed for previous software versions might severely degrade in accuracy[26].

Some additional issues being brought up in literature included having difficulty to perform simultaneous parallel analysis when each input is part of time series and requires proper understanding of its context[27]; not all problems might be reflected within logs and the issues of software program itself might be overlooked[28]; anomaly detection methods do not get sufficiently compared to each other[29]; traditional machine learning methods such as SVM are unable to perform sufficiently accurate analysis of temporal information of discrete log messages[30]; Certain anomaly detection models have not been sufficiently tested in real life application[31]; models based on statistical methods might be insensitive to importance of log entry order sequences[32].

3.1. Primary Findings

The following were the main findings of analyzed literature: 1.Embedding multi-core point-by-point convolution and global average pooling achieves significant advantages in terms of arithmetic power, memory and high availability, while ensuring detection accuracy [23]. 2.Gumbel Noise Score Matching model demonstrated the capability of score matching for anomaly detection on categorical types in both tabular and image datasets. It also provided a unified framework for modeling mixed data types via score matching [33]. 3.In transformer based models adapter-based tuning consistently outperforms training and fine-tuning models[ 16 ]. 4.Dividing log events into dependent and independent types is an effective way to boost model accuracy [ 17 ]. 5.Taking a character-based approach to process log events (lines) contributes to higher performance as the model may take advantage of characters deleted in word-based approaches, such as numbers and punctuation. Merging the parser, vectorizer, and classifier components into one deep neural network, allows model to learn log data at the language level [34]. 6.Models trained on multi-project datasets are not only more accurate in standard tests but also more robust to sequence evolutions and more accurate in ahead of time anomaly predictions [34]. 7.Though the presence of critical logs often indicates problems, their absence does not necessarily imply a healthy system status. An important reason is that sometimes determining where and how to place an informative log statement is difficult. In some cases, faults do not affect metrics, while in other cases, metrics exhibit unusual patterns (e.g., jitters) even if the system is experiencing minor performance fluctuations instead of faults. Hence, simply identifying anomalous metric patterns is insufficient [ 1 ]. 8.Faults can cause unexpected behaviors involving either logs or metrics, or both of them. So the two data sources should be analyzed comprehensively to reveal the actual anomalies [ 1 ]. 9. Intrinsic structure of host-based logs, as captured by persistence images and the spectrum of graph and hypergraph Laplacians, contains discriminative information about whether or not the logs are anomalous[35]. 10. Data augmentation can simulate deviations in log data that occur from service updates over time which contribute to successful anomaly detection[25]. 11. Multimodal approach can improve the scores for anomaly detection for multiple modalities in comparison to the single modalities of logs and traces [36]. 12. Filtering out common log entries can noticeably improve anomaly detection accuracy [37].

4. Conclusions

During this survey it has been determined that over recent years the popularity of this topic has been increasing. The problems identified within research papers still need to be addressed and no universal solution has been discovered which would allow anomaly detection methods to keep up with ever increasing amount of generated log data and general increasing complexity of system software. It has also been determined that neural networks are continuously increasing in popularity, while traditional machine learning methods are becoming less popular. It has been determined that the most popular neural network model is LSTM/BiLSTM, most commonly used dataset is HDFS and most frequently used evaluation metric is F1 score. [23] Zumin Wang, Jiyu Tian, Hui Fang, Liming Chen, Jing Qin, LightLog: A lightweight temporal convolutional network for log anomaly detection on the edge, 2022. URL: https://www.sciencedirect.com/science/article/abs/pii/S1389128621005119 [24] Bruno Wassermann, David Ohana, Ronen Schaffer, Robert Shahla, Elliot K. Kolodner, Eran Raichstein, Michal Malka, DeCorus: Hierarchical Multivariate Anomaly Detection atCloud-Scale, 2022. URL: https://arxiv.org/pdf/2202.06892.pdf [25] Thorsten Wittkopp, Alexander Acker, Sasho Nedelkoski, Jasmin Bogatinovski, Dominik Scheinert, Wu Fan, Odej Kao, A2Log: Attentive Augmented Log Anomaly Detection, 2021. URL: https://arxiv.org/pdf/2109.09537.pdf [26] Harold Ott, Jasmin Bogatinovski, Alexander Acker, Sasho Nedelkoski, Odej Kao, Robust and Transferable Anomaly Detection in LogData using Pre-Trained Language Models, 2021. URL: https://arxiv.org/pdf/2102.11570.pdf [27] Prateek Chanda, Malay Bhattacharya, Distributed Anomaly Detection in Edge Streams usingFrequency based Sketch Datastructures, 2021. URL: https://arxiv.org/pdf/2111.13949.pdf [28] Yukyung Lee, Jina Kim, Pilsung Kang , LAnoBERT : System Log Anomaly Detectionbased on BERT Masked Language Model, 2023. URL: https://arxiv.org/pdf/2111.09564.pdf [29] Zhuangbin Chen, Jinyang Liu, Wenwei Gu, Yuxin Su, Jieming Zhu, Yongqiang Yang, Michael R. Lyu, Experience Report: Deep Learning-based System Log Analysisfor Anomaly Detection, 2022. URL: https://arxiv.org/pdf/2107.05908.pdf [30] Haixuan Guo, Shuhan Yuan, Xintao Wu, LogBERT: Log Anomaly Detection via BERT, 2021. URL: https://arxiv.org/pdf/2103.04475.pdf [31] Jonghyeon Ko, Marco Comuzz, Online anomaly detection using statisticalleverage for streaming business process events, 2021. URL: https://arxiv.org/pdf/2103.00831.pdf [32] Yicheng Guo, Yujin Wen, Congwei Jian, Yixin Lian, Yi Wan, Detecting Log Anomalies with MultiHead Attention (LAMA), 2021. URL: https://arxiv.org/pdf/2101.02392.pdf [33] Ahsan Mahmood, Junier Oliva, Martin Styner, Anomaly Detection via Gumbel Noise Score Matching, 2023. URL: https://arxiv.org/pdf/2304.03220.pdf [34] Shayan Hashemi, Mika Mäntylä, OneLog: Towards End-to-End Training in Software Log Anomaly Detection , 2021. URL: https://arxiv.org/pdf/2104.07324v1.pdf [35] Thomas Davies, Topological Data Analysis for Anomaly Detection in Host-Based Logs , 2022. URL: https://arxiv.org/pdf/2204.12919.pdf [36] Jasmin Bogatinovski, Sasho Nedelkoski, Multi-Source Anomaly Detection in Distributed IT Systems, 2021. URL: https://arxiv.org/pdf/2101.04977.pdf [37] Siavash Ghiasvand, Florina M. Ciorba, Anomaly Detection in High Performance Computers: A Vicinity Perspective, 2019. URL: https://arxiv.org/pdf/1906.04550.pdf

References

[1] Cheryl Lee , Tianyi Yang , Zhuangbin Chen, Yuxin Su, Yongqiang Yang and Michael R. Lyu , Heterogeneous Anomaly Detection for SoftwareSystems via Semi-supervised Cross-modal Attention , 2023 . URL: https://arxiv.org/pdf/2302.06914.pdf

[2] Thorsten Wittkopp , Dominik Scheinert, Philipp Wiesner, Alexander Acker, and Odej Kao, PULL: Reactive Log Anomaly DetectionBased On Iterative PU Learning , 2023 . URL: https://arxiv.org/pdf/2301.10681.pdf

[3] Max Landauer , Sebastian Onder , Florian Skopik, and Markus Wurzenberger, Deep Learning for Anomaly Detection in Log Data:A Survey , 2023 . URL: https://arxiv.org/pdf/2207.03820.pdf

[4] Rasheed Ahmad , Izzat Alsmadi, Wasim Alhamdani, Lo'ai Tawalbeh , Zero-day attack detection: a systematic literature review , 2023 . URL: https://link.springer.com/article/10.1007/s10462-023-10437-z

[5] , About arXiv, 2024 . URL: https://info.arxiv.org/about/index.html

[6] , About Science Direct, 2024 . URL: https://www.elsevier.com/products/sciencedirect

[7] Chris Woodford , Neural networks, 2023 . URL: https://www.explainthatstuff.com/introduction-toneural-networks.html

[8] Sara Brown, Machine learning , explained , 2021 . URL: https://mitsloan.mit.edu/ideas-made -tomatter/machine-learning-explained

[9] IBM , What is machine learning?, 2024 . URL: https://www.ibm.com/topics/machine-learning

[10] Amir Farzada ,, T. Aaron Gullivera, Log Message Anomaly Detection and Classification UsingAutoB/LSTM and Auto-GRU , 2021 . URL: https://arxiv.org/pdf/ 1911 .08744.pdf

[11] Suthar Mudra Bhavikkmuar , Advantages of Support Vector Machines (SVM) , 2023 . URL: https://iq.opengenus.org/advantages-of-svm/

[12] Donal Tobin , The Ultimate Guide to HDFS for Big Data Processing , 2023 . URL: https://www.integrate.io/blog/guide-to -hdfs-for-big-data-processing/

[13] Adam Oliner , Jon Stearley, What Supercomputers Say: A Study of Five System Logs , 2007 . URL: https://ieeexplore.ieee.org/document/4273008/

[14] Nikolaj Buhl , F1 Score in Machine Learning , 2023 . URL: https://encord.com/blog/f1-score -inmachine-learning/

[15] Jasmin Bogatinovski , Gjorgji Madjarov, Sasho Nedelkoski, Jorge Cardoso, Odej Kao, Leveraging Log Instructions in Log-based AnomalyDetection , 2022 . URL: https://arxiv.org/pdf/2207.03206.pdf

[16] Hongcheng Guo , Xingyu Lin, Jian Yang , Yi Zhuang, Jiaqi Bai, TieqiaoZheng, Liangfan Zheng, Weichao Hou, Bo Zhang, Zhoujun Li, TRANSLOG: A Unified Transformer-based Framework forLog Anomaly Detection , 2022 . URL: https://arxiv.org/pdf/2201.00016.pdf

[17] Yongzheng Xie , Hongyu Zhang, Bo Zhang, Muhammad Ali Babar, Sha Lu, LogDP: Combining Dependency and Proximityfor Log-based Anomaly Detection , 2021 . URL: https://arxiv.org/pdf/2110. 01927 .pdf

[18] He Cheng , Depeng Xu, Shuhan Yuan, Xintao Wu, Fine-grained Anomaly Detection in Sequential Datavia Counterfactual Explanations , 2022 . URL: https://arxiv.org/pdf/2210.04145.pdf

[19] Saswati Ray , Sana Lakdawala, Mononito Goswami, Chufan Gao , Learning Probabalistic Graph Neural Networks forMultivariate Time Series Anomaly Detection , 2021 . URL: https://arxiv.org/pdf/2111.08082v1.pdf

[20] Van-Hoang Le , Hongyu Zhang, Log-based Anomaly Detection Without Log Parsing , 2021 . URL: https://arxiv.org/pdf/2108. 01955 .pdf

[21] Davide Sanvito , Giuseppe Siracusano, Sharan Santhanam, Roberto Gonzalez, Roberto Bifulco, syslrn: Learning What to Monitor for EfficientAnomaly Detection, 2022 . URL: https://arxiv.org/pdf/2203.15324.pdf

[22] Yipeng Ji , Jingyi Wang, Shaoning Li , Yangyang Li , Shenwen Lin , Xiong Li , An Anomaly Event Detection Method Based on GNN Algorithmfor Multi-data Sources , 2021 . URL: https://arxiv.org/pdf/2104.08761.pdf