Malware detection and classification is a critical issue in cybersecurity. Systems acting through signatures sufer the problem of not being able to detect attacks via zero-day malware. Among the approaches that can detect unknown attacks are the possibilities ofered by analyzing the sequence of API calls performed by the executable. Such information can be extracted through static and dynamic analysis methods in a sandbox environment. This work proposes an analysis of diferent techniques to detect malware and subsequently classify them by identifying the family of belonging. Machine Learning algorithms based on trees are compared with Deep Learning algorithms based on Recurrent Neural Networks. The results obtained lead to choosing an algorithm based on RNNs for malware detection and an algorithm based on trees for malware classification.
The constant growth of the development and difusion of new technologies expose systems to potential risks. Among the main risks is malware, malicious software created to steal, spy, or, more in general, damage infected systems. In order to mitigate this risk, it is necessary to adopt tools that can detect, classify and block these threats on time. The literature analysis shows that most malware detection systems are based on static analysis techniques such as signature verification, which are very good for all known malicious software, but at the same time, inefective for new malware since the signatures of these applications are not yet available. An alternative approach is using sandbox information (such as CAPEv2), which provides dynamic application analysis. The sandbox software encapsulates the information extracted in reports for each file analyzed. The reports contain the ordered sequence of API calls called from the analyzed file, which approximates the behavior of the software. The analysis of API calls sequences through machine learning models to detect both known malware and as-yetunknown (zero-day) malware is of great interest to researchers. Once the threat is detected, using machine learning models, it is possible to determine the malware’s type based on the information extracted from the dynamic analysis. Labeling the detected malware with the appropriate family allows experts in the field to optimize the analysis time and thus minimize the response time for each identified threat. This work is proposed as an analysis of several known techniques in order to detect and classify malware.
The goals of this paper are: • Compare the performance of tree-based Machine Learning algorithms and Recurrent Neural Networks (RNN)-based Deep Learning algorithms for the analysis of API calls sequences; • Evaluate and identify which basic models perform well for malware detection based on
API call sequence analysis; • Compare the performance of API calls-based models for malware classification in their category.
This paper is organized as follows. In section 2 the most interesting approaches related to malware detection and classification models based on the analysis of API calls sequences are reported. Section 3 describes the algorithms that are used in this work. Section 4 details the performed experiments and their notable results. Conclusions and possible future developments of the proposed work are presented in section 5.
Malware analysis systems can leverage machine learning algorithms to extract meaningful patterns from data to detect the presence of malware and recognize the category to which the malware belongs. An innovative approach involves using the sequence of API calls invoked by the malware itself as a "trace" of its behavior within the system. The following section analyzes the works that represent state of the art for Malware Detection and Malware Classification systems based on API Call sequences analysis.
Ki et al. [
In a previous work [
Several supervised Machine Learning algorithms have been used in the literature to detect and
classify malware [
In [
Catak et al. [
To overcome these issues, the authors present an innovative model of Malware Classification called SIMPLE. Based on the input API calls sequences, they are pre-treated through a pretrained word embedding technique and finally modeled by a Long Short Term Memory network to preserve the sequence information. Unlike other models of the same type, the output is represented through multiple prototypes generated by clustering. The presented model achieves the best accuracy concerning the reference models.
In [12] the authors propose a methodology that uses feature extraction and feature selection techniques to process sequences for analysis by classification algorithms.
Recurrent Neural Networks and LSTM have been extensively used in Pattern Recognition to exploit temporal sequence as in [13, 14]. Li et al. [15] propose to use Recurrent Neural Networks to analyze ordered sequences of API calls and classify malware families. Since each malware can invoke sequences of API calls of diferent lengths, the authors compare the use of two popular types of RNNs, which are widely used to analyze time series. In the above work, Long Short-Term Memory and Gated Recurrent Unit (GRU) algorithms are used to clarify the best RNN architecture for malware classification by analyzing long sequences of API calls. In the described experiments, a reference dataset was used to evaluate and compare the performance of the classifiers. The results showed that the LSTM model and the GRU model achieve very similar performance, and both are efective at classifying malware through the API call sequence analysis.
A classifier called "Random Transformer Forest" is proposed in [ 16], which uses a Transformer forest for analyzing API call sequences and categorizing malware categories. Unlike traditional machine learning and deep learning models, Transformer-based models process the sequences as a whole and learn the relationships between API calls through Multi-Head Attention mechanisms and positional embeddings. The reported experiments show that the proposed model outperforms the performance obtained using an LSTM architecture. Moreover, it is shown that BERT or CANINE, the pre-trained Transformer models, reach better performance when classifying highly unbalanced malware families.
Consider the goal of implementing a malware detection system. Such a system should also be able to classify the detected malware. It is desired that the system performs the operations by evaluating information extracted from the dynamic analysis of files. Therefore, it is necessary to obtain a workflow that can perform the operations mentioned earlier, which are, in order: dynamic analysis of the executable, identification of its type (malware/goodware), and, if it is a malware, classification in its specific family. An isolated and secure environment is used for the dynamic analysis, i.e., CAPEv2, which examines the file and returns a report containing the sequence of API calls invoked by the analyzed file. Thus, it is desired to find which Machine Learning and Deep Learning algorithms achieve the best performance when used to analyze API call sequences.
The proposed system uses a two-step approach to increase the detection rate of malicious files and improve the accuracy in classifying malware families. In detail, the system uses in the ifrst step a Malware Detection algorithm to detect malware, which, regardless of the family of belonging, must be blocked and analyzed by experts in the field. During the second step, a Malware Classification algorithm is used to support the analysis of malicious files; it inputs the sequence of API calls captured for the malware in question and outputs the corresponding family.
Therefore, according to the Malware Detection model, it is necessary to build a binary classifier, which allows for detecting as many malware as possible; instead, in the second step, it is necessary to build a multi-class classifier, which accurately assigns to each malware the appropriate class.
The classification algorithms analyzed in this paper are now reported. All the algorithms examined can be used both to build binary classification models, i.e., to build the Malware Detection model; and for multiple classification problems, i.e., to build the Malware Classification model. From the review of the literature conducted in the previous section, it can be seen that some of the most interesting approaches are tree-based classifiers and deep learning algorithms generally used to classify textual sequences; based on this study, a choice of algorithms to be analyzed was made.
The algorithms examined in the proposed work are: • Random Forest [17]: classifier obtained from an aggregation, through bagging technique, of decision trees; • CatBoost [18]: machine learning method based on Gradient Boosting applied to decision trees; • XGBoost [19]: algorithm for creating prediction models from an ensemble of smaller prediction models, represented by decision trees; • ExtraTrees [20]: algorithm that aggregates the results of multiple unrelated decision trees into a forest that produces the classification output. The idea behind it is most similar to the Random Forest algorithm and difers from it only in the way the decision trees are constructed in the forest; • TabNet [21]: Neural network for tabular data processing; • Bidirectional Long Short-Term Memory (Bi-LSTM): evolution of the model Long Short Term Memory [22], belongs to the category of RNN. It is defined bidirectional since the flow of information propagates in both directions; • Bidirectional Gated Recurrent Unit (Bi-GRU): evolution of the Gated Recurrent Unit [23], belongs to the category of RNNs. It uses a bidirectional approach to analyze sequences in both directions as well.
This section describes the experiments performed in the proposed work. All experiments were executed on a machine with 16 cores, 38 GB of RAM, and Ubuntu 20.04.3 operating system. Section 4.1 describes the datasets used to evaluate the algorithms considered in this work. In sections 4.2 and 4.3, the results obtained in the experiments performed for the construction of the Malware Detection and Malware Classification model, respectively, are presented. The obtained results and the achievement of the proposed goals are discussed in section 4.4.
The malware-analysis-datasets-api-call-sequences and APIMDS datasets were examined to
evaluate malware detection algorithms. The malware-analysis-datasets-api-call-sequences [24] dataset
contains 42,797 malware-related API call sequences and 1079 goodware API call sequences.
Each API call sequence is made of the first 100 consecutive non-repeating calls associated with
the parent process, extracted from the "calls" elements of the reports obtained from Cuckoo
Sandbox. For malware-analysis-datasets-api-call-sequences dataset, there are no labels for the
malware categories. The APIMDS [
Since the APIMDS dataset contains both malware family labels and records labeled as goodware, it is used in the present work for both the malware detection task and the malware classification task. For the evaluation of the malware classification algorithms, the Catak and APIMDS datasets were considered.
To build the Catak dataset [
Starting from the state of the art study on algorithms carried out in [
Tables 1 and 2 shows the results obtained from the Malware Detection experiment on the malware-analysis-datasets-api-call-sequences (Table 1) and APIMDS (Table 2) datasets. Specifically, the results are computed through stratified cross-validation with equal to 10 and averaged across them. Accuracy, Recall, and F1-Score metrics are calculated with macro averaging. The tables also contain the training and prediction times measured by the Malware Detection experiment. The training time represents the average training times measured to build the Malware Detection model in the 10 k-Fold experiments. In the same way, the prediction time represents the average of the prediction times measured to identify the malware contained in the test dataset in the 10 experiments of the k-Fold.
This section describes the experiments performed to compare classification algorithms for malware family identification. Specifically, the performance obtained using the Machine Learning and Deep Learning algorithms described in section 3 were compared. The experiments were performed using the Catak and APIMDS datasets. Tests were not repeated on the malwareanalysis-datasets-api-call-sequences dataset since it does not contain malware family labels. To build the Malware Classification models, all sub-sequences with consecutively repeated API calls were removed, leaving only one call for each sub-sequence. There is a high concentration of null values since not all records have the same amount of calls. In order to mitigate this problem, the columns in the dataset in which there is a presence of null values greater than 20% have been removed.
Tables 3 and 4 presents the calculated results for each algorithm derived from the stratified cross-validation experiment with equal to 10 and averaged with each other. The Accuracy, Recall, F1-Score metrics are calculated with macro averaging, instead the AUC ROC value was calculated according to the "one vs rest" approach. The reference dataset for table 3 is APIMDS, while table 4 presents the results obtained on the dataset Catak.
The tables also include the training and prediction times measured by the Malware Classification experiment. As in previous experiments, the training time is the average of the training times measured to build the Malware Classification model in the 10 k-Fold experiments. Similarly, the prediction time represents the average of the prediction times measured to classify the malware family contained in the test dataset for the 10 k-Fold experiments.
The above section results show that the considered algorithms constitute the state of the art for
malware detection and classification systems. In the related experiment for Malware Detection,
it can be observed that Deep Learning algorithms do not always achieve better performance
than Machine Learning algorithms and vice versa. It is notable that, for both datasets analyzed
for Malware Detection, Bi-LSTM and Bi-GRU algorithms (which are generally used to classify
textual sequences) always achieve higher performances than tree-based algorithms. According
to the results reported in Tables 1 and 2, the Bi-LSTM and Bi-GRU algorithms outperform
the CatBoost algorithm, which was identified as the most suitable algorithm for building the
Malware Detection system in previous work [
For this reason, when choosing the Malware Classification model, it is necessary to consider the F1-score index, which is a harmonic average of the Accuracy and Recall scores. From the results shown in tables 3 and 4, it can be observed that the algorithm that overall gets the best results is ExtraTrees. For the APIMDS dataset, the ExtraTrees algorithm scores the highest values for the F1-score and Accuracy metrics and one of the highest AUC ROC index scores. For the Catak dataset, all metrics computed in the experiments using the ExtraTrees algorithm are higher than the metrics obtained using the other surveyed algorithms.
For completeness, observing the temporal performances, it emerges how the best techniques allow recognition in relatively short times, even if not suitable for a context with large volumes of data. The training phase is the heaviest; however, this can be carried out cyclically, such as during the hours in which there is less trafic to analyze.
Aiming at realizing a workflow for malware analysis, experimental analysis was carried out between the performances of malware detection and classification algorithms based on the analysis of API call sequences. From the obtained results, it is possible to state that defining the best algorithm for the addressed tasks is not always possible. In the experiments carried out for Malware Detection, the Bi-GRU algorithm has obtained the best performances in terms of Recall, AUCROC, F1-Score, and Accuracy in both analyzed datasets. On the other hand, the analysis of the experiments performed for Malware Classification shows the best results by the ExtraTrees decision tree-based algorithm.
This study shows that the algorithms based on the RNN architecture achieve great performance in malware detection. RNNs can work on sequences of an arbitrary length, overcoming the limitations imposed by other tree-based ones such as CatBoost. This feature allows Malware Detection models to generalize over new records and identify sub-sequences of API calls that can be traced back to malware. The results reported in section 4.2 show that RNN-based algorithms detect more malicious files than tree-based algorithms, as they achieve a significantly higher Recall score at the cost of comparable accuracy with the models analyzed in previous work. Unlike the detection model, higher classification accuracy is needed in malware classification cases. Since many types of malware invoke similar API sequences, tree-based algorithms can better discriminate the particular malware type.
Future developments of this work will focus on studying pre-processing techniques that can remove the noise in the API calls sequences and optimize the discrimination ability of the classification models. Moreover, it is of great interest to investigate the use of ensemble learning techniques based on innovative RNN architectures. Future analysis will also evaluate the applicability of such solutions in contexts with large volumes of data to be analyzed, i.e., in systems where time is critical.
This work was partially supported by the Fondo Europeo di Sviluppo Regionale Puglia Programma Operativo Regionale (POR) Puglia 2014–2020 Axis I–Specific Objective 1a–Action 1.1 (Research and Development) – Project Title: CyberSecurity and Security Operation Center (SOC) Product Suite by BV TECH S.p.A., under Grant CUP/CIG B93G18000040007. family recognition with multi-prototype modeling, Computers & Security 106 (2021) 102273. [12] U. Nandagopal, S. Thirumalaivelu, Classification of malware with mist and n-gram features using machine learning, International Journal of Intelligent Engineering & Systems (2021). [13] V. Dentamaro, P. Giglio, D. Impedovo, L. Moretti, G. Pirlo, Auco resnet: an end-to-end network for covid-19 pre-screening from cough and breath, Pattern Recognition 127 (2022) 108656. [14] G. Cicirelli, D. Impedovo, V. Dentamaro, R. Marani, G. Pirlo, T. D’Orazio, Human gait analysis in neurodegenerative diseases: a review, IEEE Journal of Biomedical and Health Informatics (2021). [15] C. Li, J. Zheng, Api call-based malware classification using recurrent neural networks,
Journal of Cyber Security and Mobility (2021) 617–640. [16] F. Demirkıran, A. Çayır, U. Ünal, H. Dağ, An ensemble of pre-trained transformer models for imbalanced multiclass malware classification, arXiv preprint arXiv:2112.13236 (2021). [17] L. Breiman, Random forests, Machine learning 45 (2001) 5–32. [18] J. T. Hancock, T. M. Khoshgoftaar, Catboost for big data: an interdisciplinary review,
Journal of big data 7 (2020) 1–45. [19] C. Bentéjac, A. Csörgő, G. Martínez-Muñoz, A comparative analysis of gradient boosting algorithms, Artificial Intelligence Review 54 (2021) 1937–1967. [20] P. Geurts, D. Ernst, L. Wehenkel, Extremely randomized trees, Machine learning 63 (2006) 3–42. [21] S. O. Arık, T. Pfister, Tabnet: Attentive interpretable tabular learning, in: AAAI, volume 35, 2021, pp. 6679–6687. [22] M. Schuster, K. K. Paliwal, Bidirectional recurrent neural networks, IEEE transactions on
Signal Processing 45 (1997) 2673–2681. [23] K. Cho, B. Van Merriënboer, D. Bahdanau, Y. Bengio, On the properties of neural machine translation: Encoder-decoder approaches, arXiv preprint arXiv:1409.1259 (2014). [24] A. Oliveira, R. Sassi, Behavioral malware detection using deep graph convolutional neural networks, TechRxiv (2019).