In 2020, according to the detection and dissemination of malicious programs by CNCERT / CC (National Internet Emergency Center), more than 42.98 million samples of malicious programs were found in the whole year. According to the above report, malicious code has great harm. Based on the above situation, it is of great significance to study malicious code. At this stage, there have been many researches on malicious code analysis and detection methods. In 2011, L.Nataraj et al. proposed a malicious code visualization method and classification method, which converts binary data files of malicious code into texture images, and classifies malicious software through KNN model and Euclidean distance [2]. Through experimental demonstration, this method can effectively improve the detection speed of malicious code, and also ensure the accuracy level of traditional static detection methods. On this basis, many researchers have begun to try to use the detection and classification method based on malicious code image to train the appropriate malicious code classifier. For example, Zhang Jinglian et al. proposed a malicious code classification technology based on feature fusion, which extracts and fuses the features by extracting the opcode instructions and grayscale image texture of malicious code, and uses Random Forest (RF) to classify the malicious code families [3]. The above methods all visualize the malicious code, convert the malicious code into a grayscale image for further classification, and achieve good results.
With the application of Convolutional Neural Network (CNN) in object detection, more and more researchers have put forward a series of achievements. For example, Ross B. Girshick proposed the Faster R-CNN network [5]. For target detection in grayscale images of malicious code, we need to detect the position of the section (.text) where the core feature opcodes are located in the grayscale image. The loss of the traditional Faster R-CNN model is composed of two parts: classification loss and regression loss. During the training process, many hyperparameters will be generated, which requires human and material resources to adjust the parameters. At the same time, the imbalance of data distribution will also affect the detection effect of the model. Therefore, this paper optimizes the loss function for the above problems, and introduces Rank & Sort Loss to achieve better detection results.
Static disassembly of malicious code to achieve the visualization of malicious code. IDA Pro is used for static disassembly, by importing malicious software into IDA Pro to get malicious code binary file (.bytes file) and assembly file (.asm file).On this basis, the obtained binary executable file is used as input data, and regard it as the original Bytes binary stream. A hexadecimal number can be considered as a combination of binary number, and four binary numbers can be converted to a hexadecimal number. Because the range of hexadecimal number is only between 0 and 16, which corresponds to two hexadecimal numbers of 256 pixel value of grayscale image, this method can convert the original data into a simple gray image. The sequence of binary streams corresponding to the gray level of each 8-bit pixel is segmented and then arranged into a sequence to form the corresponding grayscale image. Figure 1 shows the schematic diagram of malicious code visualization.
Rank & Sort (RS) Loss [8] was proposed by K. Oksuz et al in 2021. Rank & Sort Loss is composed of Rank loss and Sort loss. Rank loss enables all positive samples to be sorted before negative samples, and only the negative samples with higher scores are selected for calculation. Sort loss uses IoU as the classification label, so that the positive samples in the prediction box can be sorted by continuous values as the label. Besides, Rank & Sort Loss does not require multi-task weight or coefficient adjustment. The definition of the loss function is shown in formula (1):
Where P is the collection of positive sample, 𝑙 𝑅𝑆 (𝑖) is the sum of rank error and sort error at this time, 𝑙 𝑅𝑆 * (𝑖) is the sum of the target rank error and the target sort error, using equations ( 2) and ( 3) respectively express:
Where 𝑖 and 𝑗 are the sample numbers, 𝑦 is the score label, and 𝑟𝑎𝑛𝑘(𝑖) is the number of all positive samples and negative samples that are greater than or equal to the positive sample's classification score; 𝑟𝑎𝑛𝑘 − (𝑖) is the number of all negative samples that is greater than or equal to the positive sample's classification score; 𝑟𝑎𝑛𝑘 + (𝑖) is the number of all positive samples that is greater than or equal to the positive sample's classification score, 𝑥 𝑖𝑗 is the classification score, 𝐻(𝑥) is the unit step function, 𝑙 𝑅 * (𝑖) is the target rank error.
The training process for the whole network mainly includes the following steps: convolution channel number is 18, which is used to predict whether each prediction box contains an object, and the other convolution channel number is 36, which is used to adjust the prior box, get a suggestion box. Then, the classification and regression are carried out through the ROIPooling layer. The improved method is to replace the cross entropy loss in the classification loss with the RS loss, and the regression loss adopts the GIoU loss. The weighted parameter of the regression loss of the improved model is the RS loss divided by the regression loss.
This experiment selects malicious sample data from the Kaggle platform of the Microsoft Malware Security Defense Center. There are 1839 samples from 6 malicious code sample families, as shown in Table 1.
In the experiment, the data were randomly divided into training set and test set according to 8:2. The accuracy rate and recall rate are selected as the evaluation criteria. Table 3 shows the accuracy rate and recall rate of each class of the traditional machine learning classifier, the traditional Faster R-CNN network framework and the improved Faster R-CNN network framework. The experimental results show that the Faster R-CNN network model is better than the traditional machine learning classification model, and the improved Faster R-CNN network model is further improved on the basis of the traditional Faster R-CNN network model. After the introduction of RS Loss, the accuracy of the model is increased by 1.9 percentage points compared with the original model. At the same time, the complexity of the model is measured by Floating point Operations (FLOPs) and Parameters. The results show that the amount of Parameters of the model is reduced by 15.37% compared with the original model, and the amount of FLOPs is reduced by 23.58%.It shows that the introduction of RS loss can greatly reduce the amount of parameters and effectively reduce the complexity of the model.
In order to further improve the detection and classification effect of malicious code, this paper proposes a malicious code classification model based on improved Faster R-CNN. The method of using Rank & Sort loss function effectively reduces the number of hyperparameters. In the process of model training, there is no need to repeatedly adjust the hyperparameters. We only need to adjust the learning rate to improve the model performance, avoiding the complex parameter adjustment process and one loss dominant situation. The experimental results show that the model is more feasible and effective. In the following work, the detection ability of the model can be further improved by adding Attention Mechanism and Data Augmentation.