Recent work has shown that deep-learning algorithms for malware detection are also susceptible to adversarial examples, i.e., carefullycrafted perturbations to input malware that enable misleading classi cation. Although this has questioned their suitability for this task, it is not yet clear why such algorithms are easily fooled also in this particular application domain. In this work, we take a rst step to tackle this issue by leveraging explainable machine-learning algorithms developed to interpret the black-box decisions of deep neural networks. In particular, we use an explainable technique known as feature attribution to identify the most in uential input features contributing to each decision, and adapt it to provide meaningful explanations to the classi cation of malware binaries. In this case, we nd that a recently-proposed convolutional neural network does not learn any meaningful characteristic for malware detection from the data and text sections of executable les, but rather tends to learn to discriminate between benign and malware samples based on the characteristics found in the le header. Based on this nding, we propose a novel attack algorithm that generates adversarial malware binaries by only changing few tens of bytes in the le header. With respect to the other state-of-the-art attack algorithms, our attack does not require injecting any padding bytes at the end of the le, and it is much more e cient, as it requires manipulating much fewer bytes.
Despite their impressive performance in many di erent tasks, deep-learning algorithms have been shown to be easily fooled by adversarial examples, i.e., carefully-crafted perturbations of the input data that cause misclassi cations [1{ 6]. The application of deep learning to the cybersecurity domain does not constitute an exception to that. Recent classi ers proposed for malware detection, including the case of PDF, Android and malware binaries, have been indeed shown to be easily fooled by well-crafted adversarial manipulations [1, 7{11]. Despite the sheer number of new malware specimen unleashed on the Internet (more than 8 millions in 2017 according to GData4) demands for the application of e ective automated techniques, the problem of adversarial examples has signi cantly questioned the suitability of deep-learning algorithms for these tasks. Nevertheless, it is not yet clear why such algorithms are easily fooled also in the particular application domain of malware detection.
In this work, we take a rst step towards understanding the behavior of
deeplearning algorithms for malware detection. To this end, we argue that
explainable machine-learning algorithms, originally developed to interpret the black-box
decisions of deep neural networks [12{15], can help unveiling the main
characteristics learned by such algorithms to discriminate between benign and malicious
les. In particular, we rely upon an explainable technique known as feature
attribution [
Our empirical ndings show that MalConv learns to discriminate between benign and malware samples mostly based on the characteristics of the le header, i.e., almost ignoring the data and text sections, which instead are the ones where the malicious content is typically hidden. This means that also depending on the training data, MalConv may learn a spurious correlation between the class labels and the way the le headers are formed for malware and benign les.
To further demonstrate the risks associated to using deep learning \as is"
for malware classi cation, we propose a novel attack algorithm that generates
adversarial malware binaries by only changing few tens of bytes in the le header.
With respect to the other state-of-the-art attack algorithms in [
The structure of this paper is the following: in Section 2 we introduce how to solve the problem of malware detection using machine learning techniques, and we present the architecture of MalConv; (ii) we introduce the integrated gradient technique as an algorithm that extracts which features contributes most ! ∈ ℝ7×9 Convolutional
layer ,... ,. ,. *... ,/
Embedding ! = #(%) Padding bytes (set to 256) '( ... ') ')*( ... '+
Convolutional layer σ
Temporal max pooling
Fully connected for the classi cation problem; (iii) we collect the result of the method mentioned above applied on MalConv, highlighting its weaknesses, and (iv) we show how an adversary may exploit this information to craft an attack against MalConv. 2
Ra et al. [
To tackle all of these issues, Ra et al. develop a deep neural network that takes as input a whole program, whose architecture is represented in Fig. 1. First, the input le size is bounded to d = 221 bytes, i.e., 2 MB. Accordingly, the input le x is padded with the value 256 if its size is smaller than 2 MB; otherwise, it is cropped and only the rst 2 MB are analyzed. Notice that the padding value does not correspond to any valid byte, to properly represent the absence of information. The rst layer of the network is an embedding layer, de ned by a function : f0; 1; : : : ; 256g ! Rd 8 that maps each input byte to an 8dimensional vector in the embedded space. This representation is learned during the training process and allows considering bytes as unordered categorical values. The embedded sample Z is passed through two one-dimensional convolutional layers and then shrunk by a temporal max-pooling layer, before being passed through the nal fully-connected layer. The output f (x) is given by a softmax function applied to the results of the dense layer, and classi es the input x as malware if f (x) 0:5 (and as benign otherwise).
We debate the robustness of this approach, as it is not clear the rationale behind what has been learned by MalConv. Moreover, we observe that the deep neural network focuses on the wrong sequences of bytes of an input binary. 3
We address the problem mentioned at the end of Section 2 by introducing
techniques aimed to explain the decisions of machine-learning models. In particular,
we will focus on techniques that explain the local predictions of such models,
by highlighting the most in uential features that contribute to each decision. In
this respect, linear models are easy to interpret: they rank each feature with a
weight proportional to their relevance. Accordingly, it is trivial to understand
which feature caused the classi er to assign a particular label to an input
sample. Deep-learning algorithms provide highly non-linear decision functions, where
each feature may be correlated at some point with many others, making the
interpretation of the result nearly impossible and leading the developers to naively
trust their output. Explaining predictions of deep-learning algorithms is still an
open issue. Many researchers have proposed di erent techniques to explain what
a model learns and which features mostly contribute to its decisions. Among the
proposed explanation methods for machine learning, we decided to focus on a
technique called integrated gradients [
The rst axiom is called sensitivity : an attribution method satis es sensitivity if, for every input that di er in one feature from the baseline but they are classi ed di erently, then the attribution of the di ering feature should be nonzero.
Moreover, if the learned function does not mathematically depend on a particular feature, the attribution should be zero. On the contrary, if sensitivity is not satis ed, the model is focusing on irrelevant features, as the attribution method fails to weight the contribution of each variable. Authors state that gradients violate the sensitivity axioms, hence using them during the training phase by applying back-propagation implies attributing wrong importance to the wrong features.
Axiom II: Implementation Invariance. The second axiom is called implementation invariance, and it is built on top of the notion of functional equivalence: two networks are functionally equivalent if their outputs are equal on all inputs, despite being implemented in di erent ways. Thus, an attribution method satis es implementation invariance if it produces the same attributions for two functionally equivalent networks. On the contrary, if this axiom is not satis ed, the attribution method is sensitive to the presence of useless aspects of the model. On top of these two axioms, Sundararajan et al. propose the integrated gradient method that satis es both sensitivity and implementation invariance. Hence, this algorithm should highlight the properties of the input model and successfully attributing the correct weights to the feature believed relevant by the model itself.
Integrated Gradients. Given the input model f , a point x and baseline x0, the attribution for the ith feature is computed as follows:
IGi(x) = (xi x0 ) i
x0)) 0 This is the integral of the gradient computed on all points that lie on the line passing through x and x0. If x0 is a good baseline, each point in the line should add a small contribution to the classi cation output. This method satis es also the completeness axiom: the attributions add up to the di erence between the output of the model at the input x and x0. Hence, the features that are important for the classi cation of x should appear by moving on that line.
Since we can only compute discrete quantities, the integral can be approximated using a summation, adding a new degree of freedom to the algorithm, that is the number of points to use in the process: Sundararajan et al. state that the number of steps could be chosen between 20 and 300, as they are enough to approximate the integral within the 5% of accuracy. 4
We applied the integrated gradient technique for trying to grasp the intuition of
what is going on under the hood of MalConv deep network. For our experiments
we used a simpli ed version of MalConv, with an input dimension shrunk to 220
instead of 221, that is 1 MB instead of 2 MB, trained by Anderson et al. [
Windows Portable Executable format. The Windows Portable Executable format6 (PE) describes the structure of a Windows executable. Each program begins with the DOS header, which is left for retro-compatibility issues and for telling the user that the program can not be run in DOS. The only two useful information contained into the DOS header are the DOS magic number MZ and the value contained at o set 0x3c, that is an o set value that point to the real PE header. If the rst one is modi ed, Windows throws an error and the
program is not loaded, while if the second is perturbed, the operating system can not nd the metadata of the program. The PE header consists of a Common Object File Format (COFF) header, containing metadata that describe which Windows Subsystem can run that le, and more. If the le is not a library, but a standalone executable, it is provided with an Optional Header, which contains information that are used by the loader for running the executable. As part of the optional header, there are the section headers. The latter is composed by one entry for each section in the program. Each entry contains meta-data used by the operating system that describes how that particular section needs to be loaded in memory. There are other components speci ed by the Windows PE format, but for this work this information is enough to understand the output of the integrated gradients algorithm applied to MalConv.
Integrated gradients applied to malware programs. The integrated gradients technique works well with image and text les, but we need to evaluate it on binary programs. First of all, we have to set a baseline: since the choice of the latter is crucial for the method to return accurate results, we need to pick a sample that satis es the constraints7 highlighted by the authors of the method: (i) the baseline should be an empty signal with null response from the network; (ii) the entropy of the baseline should be very low. If not, that point could be an adversarial point for the network and not suitable for this method.
Hence, we have two possible choices: (i) the empty le, and (ii) a le lled with zero bytes. For MalConv, an empty le is a vector lled by the special padding number 256, as already mentioned in Section 2. While both these baselines satisfy the second constraint, only the empty le has a null response, as the zero vector is seen as malware with the 20% of con dence. Hence, the empty le is more suitable for being considered the ground truth for the network. The method returns a matrix that contains all the attributions, feature per feature, V 2 Rd 8, where the second dimension is produced by the embedding layer. We compute the mean for each row of V to obtain a signed scalar number for each feature, resulting in a point v0 2 Rd and it may be visualized in a plot. Figure 2 highlights the importance attributed by MalConv to the header of an input malware example using the empty le as baseline, marking with colours each byte's contribution. We can safely state that a sample is malware if it is scored as such by VirusTotal8 which is an online service for scanning suspicious les. The cells colored in red symbolize that MalConv considered them for deciding whether the input sample is malware. On the contrary, cells with blue background are the ones that MalConv retains representative of benign programs. Regarding our analysis, we may notice that MalConv is giving importance to portions of the input program that are not related to any malicious intent: some bytes in the DOS header are colored both in red and blue, and a similar situation is found for the other sections. Notably, the DOS header is not even read by modern operating systems, as the only important values are the magic number and the value contained at o set 0x3c, as said in the previous paragraph. All the other 7 https://github.com/ankurtaly/Integrated-Gradients/blob/master/howto.md 8 https://www.virustotal.com bytes are repeated in every Windows executable, and they are not peculiar neither for goodware nor malware programs. We can thus say that this portion of the program should not be relevant for discriminating benign from malicious programs. Accordingly, one would expected MalConv to give no attribution to these portions of bytes.
We can observe the attribution assigned by integrated gradients to every
byte of the program, aggregated for each component of the binary, in Figure 3.
Each entry of the histogram is the sum of all the contributions given by each
byte in that region. The histogram is normalized using the l1 norm of its
components. The color scheme is the same as the one described for Figure 2. MalConv
puts higher weights at the beginning of the le, and this fact has been already
formalized by Kolosnjaji et al. [
We showed that MalConv bases its classi cations upon unreliable and spurious
features, as also indirectly shown by state-of-the-art attacks against the same
network, which inject bytes in speci c regions of the malware sample without
even altering the embedded malicious code [
Attack Algorithm. Our attack algorithm is given as Algorithm 1. It rst
computes the representation of each byte of the input le x in the embedded space,
as Z (x). The gradient G of the classi cation function f (x) is then
computed w.r.t. the embedding layer. Note that we denote with gi; zi 2 R8 the ith
row of the aforementioned matrices. For each byte indexed in I, i.e., in the
subset of header bytes that can be manipulated, our algorithm follows the strategy
implemented by Kolosnjaji et al. [
The result of the attack applied using an example can be appreciated in Figure 4: the plot shows the probability of being recognized as malware, which is the curve coloured in blue. Each iteration of the algorithm is visualized as black vertical dashed lines, showing that we need to interpret the gradient di erent times to produce an adversarial example. This is only one example of the success of the attack formalized above, as we tested it on 60 di erent malware inputs, taken from both The Zoo9 and Das Malverik.10 Experimental results con rm our doubts: 52 malware programs over 60 evade MalConv by only perturbing the DOS header. Note that a human expert would not be fooled by a similar attack, as the DOS header can be easily stripped and replaced with a normal one without perturbed bytes. Accordingly, MalConv should have learned that these bytes are not relevant to the classi cation task.
Algorithm 1: Evasion algorithm input : x input binary, I indexes of bytes to perturb, f classi er, T max iterations. output: Perturbed code that should
achieve evasion against f . t 0; while f (x) 0:5 ^ t < T do
Z (x) 2 Rd 8; G r f (x) 2 Rd 8; forall i 2 I do gi gi=kgik2; s 0 2 R256; d 0 2 R256; forall b 2 0; :::; 255 do zb (b); gi> (zb zi); kzb (zi + sb gi)k; arg minj:sj>0 dj ; sb db end xi t + 1; end t end
Fig. 4: Evading MalConv by perturbing only few bytes in the DOS header. The black dashed lines represent the iterations of the algorithm. In each iteration, the algorithm manipulates 58 header bytes (excluding those that can not be changed). The blue curve shows the probability of the perturbed sample being classi ed as malware across the di erent iterations. 6
We discuss here some approaches that leverage deep learning for malware
detection, which we believe may exhibit similar problems to those we have highlighted
for MalConv in this work. We continue with a brief overview of the already
proposed attacks that address vulnerabilities in the MalCon architecture. Then, we
discuss other explainable machine-learning techniques that may help us to gain
further insights on what such classi ers e ectively learn, and potentially whether
and to which extent we can make them more reliable, transparent and secure.
Deep Malware Classi ers. Saxe and Berlin [
Evasion attacks against MalConv. Kolosnjaji et al. [
Explainable Machine Learning. Riberio et al. [
We have shown that, despite the success of deep learning in many areas, there are still uncertainties regarding the precision of the output of these techniques. In particular, we use integrated gradients to explain the results achieved by MalConv, highlight the fact that the deep neural network attributes wrong nonzero weights to well known useless features, that in this case are locations inside a Windows binary.
To further explain these weaknesses, we devise an algorithm inspired by recent papers in adversarial machine learning, applied to the malware detection. We show that perturbing few bytes is enough for evading MalConv with high probability, as we applied this technique on a concrete set of samples taken from the internet, achieving evasion on most of all the inputs.
Aware of this situation, we are willing to keep investigating the weaknesses of these deep learning malware classi ers, as they are becoming more and more important in the cybersecurity world. In particular, we want to devise attacks that can not be easily recognized by a human expert: the perturbations in the DOS header are easy to detect, as mentioned in Section 5 and it may be patched without substantial e ort. Instead, we are studying how to hide these modi cations in an unrecognizable way, like lling bytes between functions: compilers often leave space between one function and the other to align the function entry point to multiple of 32 or 64, depending on the underlying architecture. The latter is an example we are currently investigating. Hence, all classi ers that rely on raw byte features may be vulnerable to such attacks, and our research may serve as a proposal for a preliminary penetration testing suite of attacks that developers could use for establishing the security of these new technologies.
In conclusion, we want to emphasize that these intelligent technologies are far from being considered secure, as they can be easily attacked by aware adversaries. We hope that our study may focus the attention on this problem and increase the awareness of developers and researchers about the vulnerability of these models.