In the present paper we describe a new, updated and re ned dataset speci cally tailored to train and evaluate machine learning based malware tra c analysis algorithms. To generate it, we started from the largest databases of network tra c captures available online, deriving a dataset with a set of widely-applicable features and then cleaning and preprocessing it to remove noise, handle missing data and keep its size as small as possible. The resulting dataset is not biased by any speci c application (although speci cally addressed to machine learning algorithms), and the entire process can run automatically to keep it updated.
In the recent years, the growing number of cyber attacks motivated an intense research in the eld of malware detection, and nowadays cyber security professionals can rely on increasingly e ective and e cient detection techniques which, however, need to be constantly updated. Indeed, malware detection itself presents a number of inherent challenges, rst of all the ability of maintaining an up-to-date knowledge base containing distinctive features of all the current classes of malware.
Most of the research is currently focusing on the use of machine learning techniques for malware detection. This approach presents a number of advantages, rst of all its capability to automatically identify the malware characteristics by observing a set of training samples, and generalise these results to detect new variants without having actually seen them before.
Since network-based malware, such as botnets, is currently one of the most common kind of malware, in this paper we focus on malware tra c analysis, which tries to detect malware by analysing its behaviour in terms of data sent and received through a network.
However, the quality of such classi ers is largely determined by the quality of the underlying training dataset, and having a high-quality dataset requires to collect real malware tra c data, keep it updated, and make this data actually usable by a machine learning algorithm. Unfortunately, publicly available malware tra c databases are few, and mostly outdated.
In the present paper we describe a complete process that creates an updated malware tra c dataset suitable to train and evaluate machine learning based malware tra c classi ers. To this aim, we apply a Knowledge Discovery in Databases (KDD) process, starting from the largest databases of network tra c captures available online, deriving a dataset with a set of widely-applicable features and then cleaning and preprocessing it to remove noise, handle missing data and keep its size as small as possible. The resulting dataset is not biased by any speci c application (although speci cally addressed to machine learning algorithms), and the entire process can run automatically to keep it updated.
It is worth noting that this process requires the right use of a number of di erent data manipulation techniques and algorithms such as pro ling, imputation and scaling, which in turn require a good knowledge of many di erent machine learning, data analytics and statistics issues to be e ectively applied. As a part of our contribution, we try to make all these steps completely transparent to the researchers who just want to develop and e ectively test a new malware tra c classi er.
We evaluate the dataset quality by measuring the presence of malware through anomaly detection and present an experiment where the dataset is used to build a simple deep neural network classi er which reaches a very high detection rate.
Of course, both the current dataset and its generation algorithm have been made available to the research community. 2
Malware features extraction and selection is a largely addressed topic in the literature. As an example, Cabau et al. [6] extract malware features by dynamically executing malware in a controlled environment and monitoring three types of actions: lesystem writes, registry operations and network access operations. arkac et al. [43] address the speci c problem of feature extraction for metamorphic malware, whereas Zhao et al. [42] focus on feature extraction and selection from Android applications. Zhang et al. [40] perform feature selection for malware detection introducing the novel Ensemble Feature Selection technique, which aims at reducing the number and size of the selected features in order to obtain datasets easier to handle. Finally, Arora et al. [3] also address the issue of android malware detection, but to this aim they use features extracted from the network tra c generated by malware apps coming from the Android Malware Genome Project, which makes this approach slightly more similar to our malware tra c analysis context.
However, a common characteristic of all such kind of works is that they present feature extraction and selection techniques and examples of features extracted using the proposed methodologies, but never provide the researchers with a full, realistic dataset created through their approach. Moreover, when a malware detection technique is also proposed, it is often evaluated on a speci cally-tailored dataset, making any comparison di cult.
The majority of the publicly-available malware detection datasets, like Android PRAGuard [23], the Android Malware Dataset [38] or EMBER [2] are devoted to malware detection in executable les, in particular Android applications. Indeed, the current literature presents few works concerning the creation of public datasets for malware tra c detection purposes. Therefore, most papers presenting malware tra c analysis algorithms usually carry on their experiments using some classical, although outdated, datasets like CTU13 [10], UGR'16 [21], CCC09 [13] or KDD99 [37]. However, while these datasets can be a suitable benchmark to compare di erent approaches, they cannot give realistic information about the quality of a malware detector, since they do not take into consideration current malware and its novel attack mechanisms. 3
The tra c data used to build our dataset was extracted from two distinct sources: legitimate tra c comes from pcap les marked as Normal in the Malware Capture Facility Project (MCFP) belonging to the Stratosphere project. In particular, the tra c collected so far from the MCFP project is composed of 15 pcap les with a total size greater than 7 GByte. malicious tra c comes from the MTA [9] repository, which o ers a collection of over one thousand four hundred zip les, each containing one or more pcap les. Every binary le in these pcaps has been recognised as malicious by IDS and Antivirus softwares (Suricata [26], VirusTotal [8]). It is worth noting that the pcap les provided by MTA are password protected, so we developed an ad-hoc scraping algorithm that automates their download and decompression. The currently collected MTA tra c is made up of 2112 pcap les, with a total size of than 4.8 GB. These observations cover a time span from June 2013 to August 2019.
The MTA repository receives almost daily new tra c logs. Our framework performs periodic downloads of these data, so our dataset is constantly growing and is constantly updated. However, since the frequency of malware tra c updates is much higher than the legitimate one (from MCFP), balancing the two parts is not trivial. 3.1
The typical approach to build detection models is to run a bot in a controlled environment and monitor its outgoing network connections. Unfortunately, this process is not as easy as it seems: for example, bots often open a large number of additional connections to legitimate sites to create some \noise" so, to identify the real command and control (C&C), it is necessary to observe all the network tra c produced by a bot in a long period of time. The tra c features to be observed must take in account this issue, and try to "summarise" the bot behaviour throughout time with a set of meaningful measures.
Moreover, since network connections (especially for malicious tra c) are nowadays always encrypted, it is not possible to analyse the packet payload. Therefore, features must be extracted from observable characteristics of the tra c like the packet ratio, length or protocol. Such characteristics are typically aggregated in time windows to extract statistical measures that actually represent the features. In this approach, the packets present in each pcap le are split in subsets spanning a xed amount of time (e.g., 15 minutes as used in [19]). However, timebased aggregation requires an arbitrary choice, i.e., the window size, which may have a great impact in the dataset quality, since di erent attack types often require di erent windows sizes to be captured. Moreover, since pcap les are varying in size, time-based aggregation always leads to discarding a certain amount of information at the end of each pcap. Therefore, in our approach, we adopted a di erent strategy, by aggregating packets in each pcap having the same source address (i.e., coming from the same host) in segments, even if they are not sequential in time. This allows us to have a di erent point of view on the tra c, that takes into account the hosts involved in the tra c rather than the packet ow itself. Moreover, common attacks such as DDoS are much more easily identi ed if features like the packet ratio are calculated relative to the hosts, and not to the overall tra c: indeed, in the latter case, the overall background tra c, interleaving the attack packets, may make the attack less evident.
We extracted a total of 50 features, which are reported in Table 2 together with their formulas. Note that, for sake of simplicity, unless otherwise speci ed from now on with packets we refer to the packets present in a speci c segment, i.e., packets in a particular pcap le having all the same source address. Moreover, we summarise in Table 1 some meaningful sets and functions that will be used to simplify the feature formulas.
These selected features have been inspired by several best practices in the eld of malware detection, as well as by the observation of the current malwares. The relevance of each feature is brie y discussed in the following. In particular, each feature in the table has an associated relevance note in the "rel" column. S S S S S S S S S S S S S S S Dp Da U R dom(p) len(p) nchar(s) ndot(s) nhyph(s) ndigit(s) ndnsque(p) ndnsans(p) ndnsadd(p) ndnsaut(p) occur(s) t(p) ttl(p) valid(s) the sent packets sent packets using the DNS protocol sent packets using the TCP protocol sent packets using the UDP protocol sent packets using the HTTP protocol sent packets containing an HTTP request sent packets containing a DNS question record sent packets containing a DNS resource record sent TCP packets with the ACK ag set sent TCP packets with the SYN ag set sent TCP packets with the FIN ag set sent TCP packets with the PSH ag set sent TCP packets with the URG ag set sent TCP packets with the RST ag set sent TCP packets ith both the ACK and SYN ags set sent packets with payload length < 32 (small packets) distinct packet destination ports distinct packet destination addresses distinct user agents in the HTTP sent packets the received packets number of domains referred in DNS packet p payload length of packet p number of characters in s number of dots in s number of hyphens in s number of digits in s number of items in question section of DNS packet p number of items in answer section of DNS packet p number of items in additional section of DNS packet p number of items in authority section of DNS packet p number occurrences of each distinct value in the set s arrival time of packet p TTL reported in DNS packet p true if all the domain names th set s are valid 1. Features fAck,Syn,Fin,Psh,Urg,RstgFlagDist: have been chosen since it has been empirically shown (see, e.g., [1], [17], [39]) that the presence many packets with of certain TCP ags set may indicate malware tra c. 2. Features fTCP,UDP,DNSgOverIP: have been chosen since many attacks exploit speci c characteristics of these protocols. As an example, trojans and other remote access issue a large number of DNS requests to locate their command and control server, so an high DNSOverIP ratio may indicate malicious tra c [25]. 3. Features MaxLen, MinLen, AvgLen, StdDevLen, MaxIAT, MinIAT, AvgIAT, AvgDeltaTime, MaxLenRx, MinLenRx, AvgLenRx, StdDevLenRx, MaxIATRx, MinIATRx, AvgIATRx,StartFlow, EndFlow, DeltaTime, FlowLen, FlowLenRx: have been chosen since packet number, size and inter-arrival times are useful to detect ooding-style attacks [16, 17]. 4. Feature PktIORatio: has been chosen since in DDoS-style attacks the number of sent packets is much higher than the received ones [36]. 5. Feature FirstPktLen: has been chosen since many times the rst sent packet reveals useful characteristics of the tra c (see, e.g., [14], [28]).
formula jST CP j jSf j
jSj jSpj tj(Spjn) t(p0) Pu2jUSjnchar(u) PpDji2stSiDncQtUnAchar(dom(pj ))
Ppj 2SDR ttl(pj ) jSDRj
t(p0) t(pn) jUj jSDNS j P
jSj jSDNS j P
jSj jSDNS j P
jSj jSDNS j P
jSj t(pn) len(p0) P P pj 2S pj 2R H len(pj ) len(pj ) Rg min, max, avg, flen(pj )jpj 2 Sg min, max, avg, flen(pj )jpj 2 Rg
ACKSY N jS j jDaj maxfjDpjg jRj jSj max occur(flen(pj )jpj 2Sg)
jSj jSsmallj
jSj t(p0) jSACjSKSSjY N j jnSTjSCjP j jSURGj j pj 2SDQjvalid(dom(pj ))oj
jSDQj feature f FlagDist pOverIP AvgDeltaTime AvgDistinctUALen AvgDomainChar AvgDomainDigit AvgDomainDot AvgDomainHyph AvgTTL DeltaTime DistinctUA DNSADist DNSQDist DNSRDist DNSSDist EndFlow FirstPktLen FlowLen FlowLenRx HTTPpkts MaxIATRx, MinIAT, MaxIAT, AvgIAT MinIATRx, AvgIATRx MinLen, MaxLen, AvgLen, StdDevLen MinLenRx, MaxLenRx, AvgLenRx, StdDevLenRx NumConnections NumDstAddr NumPorts PktIOratio RepeatedPktLenRatio SmallPktRatio StartFlow SynAcksynRatio UrgFlagDist ValidURLratio notes with f fAck; Syn; F in; P sh; Rstg. available if not TCP with p 2 fT CP; U DP; DN Sg
2
unzero if jSj = 1
NaN if jSHRj = 0
NaN if jSDQj = 0
NaN if jSDQj = 0
NaN if jSDQj = 0
NaN if jSDQj = 0
zero if jSj = 1
unavailable if not TCP
unavailable not TCP or UDP
(
NaN if jSDQj = 0 pj 2SD ndnsans(pj ) pj 2SD ndnsque(pj ) pj 2SD ndnsadd(pj ) pj 2SD ndnsaut(pj ) jS j min of f(t(pj ) t(pj 1))jpj 2 Rg min, max, avg of f(t(pj ) t(pj 1))jpj 2 Sg min, avg of f(t(pj ) t(pj 1))jpj 2 zero if jRj < 2 zero if jSj < 2 zero if jRj < 2 stddev stddev of of
StdDevLen is NaN if jSj < 2
StdDevLenRx is NaN if jRj < 2
6. Features DNSQDist,DNSADist,DNSRDist,DNSSDist: have been chosen since, as already
described, malwares often send DNS requests with speci c characteristics [25].
7. Features RepeatedPktLenRatio, SmallPktRatio: has been chosen since in DDoS-style
rel
(
In this section we describe the automatic process that generates the MTA-KDD'19 dataset, with the features listed in Section 3.1. The overall process is depicted in Figure 1.
In particular, we try to apply a minimal preprocessing on the data, limited to the manipulations needed to make the dataset suitable to be used in machine learning applications and optimise its performances, e.g. by removing unnecessary data. Therefore, we are con dent that the resulting dataset is not biased by any particular application. 4.1
As introduced in the previous section, to build the initial dataset, i.e., to extract the tra c features, for each distinct pcap le, we group the contained packets based on the source address (segmentation), and then compute the features in Table 2 on these groups. The set of features related to a speci c segment become a row of our dataset, that we shall call dataset sample. Stating from the pcap les described in Section 3, we obtain a dataset with the characteristics shown in Table 3. It is worth noting that the dataset is balanced, i.e., the distribution of malware and legitimate samples is very similar.
Sample Type Malware Legitimate Total In the pro ling phase we analyse the dataset features in order to detect and remove certain low-complexity issues that may a ect the data and in uence the next phases. In particular, we perform the following steps.
Duplicates Duplicate samples may in uence the classi ers and cause over tting. Even if the
source dataset showed a very low number of duplicates, we remove them all.
Highly-correlated features During the pro ling phase we also extract useful information
about the feature correlation, which is measured using the Pearson method [29]. Features with
a high correlation, 0:95 in our case, are removed from the dataset since they do not convey
additional information and, if preserved, would only increase the amount of resources needed
to process the dataset in the following phases. Therefore, we randomly select a feature in each
high-correlation pair and remove it. In particular, the removed features are AvgDeltaTime
(correlated to AvgIATRx with = 0.96199), MaxIATRx (correlated to MaxIAT with =
0.99853), DNSADist (correlated to DNSQDist with = 0.99564), DNSRDist (correlated to
DNSADist with = 0.97725), DNSSDist (correlated to DNSRDist with = 0.9981), EndFlow
(correlated to StartFlow with = 1). In table 2 the features removed for high correlation are
marked with the number (
Zeros, Unavailable Values, and Missing Values Some features may have zero values for di erent reasons. "Real" zeroes are those that are useful to classify the tra c, but sometimes a zero may be used to represent an unavailable value, which has a very di erent meaning, i.e., in some sense, it must be ignored by classi ers, since it does not convey any information (e.g., TCP-speci c features are unavailable in UDP tra c). Finally, since tra c data tend to be algorithm standardized scaler minmax scaler maxabsolute scaler robust scaler quantile transformer power transformer incomplete and noisy, as we already noticed in the formulas, some features may have missing values indicated by a Not a Number (NaN) value.
Based in this, we rst remove the URGFlagDist feature that is set to zero in all the samples,
so it seems to not convey any information about the tra c. In table 2 this feature is marked
with the number (
Then, we remove any sample that contains an unavailable feature value. With this lter, we drop 6828 samples, whereas the remaining dataset has still 64554 samples, 53.21% of which represent malware and 46,79% legitimate tra c, so the dataset is now even more balanced.
Finally, features with many NaN ( 50%, so with a small number of real values), are removed
from the dataset since they do not convey enough information. In particular, the removed
features are AvgDomainChar (97.6% missing values), AvgDomainDot (97.6%), AvgDomainHyph
(97.6%), AvgDomainDigit (97.6%), AvgTTL (99.7%), SynAcksynRatio (96.5%),
ValidURLratio (97.6%), DistinctUA (96.6%), AvgDistinctUALen (96.6%). In table 2 the features removed
for high correlation are marked with the number (
In the previous phase we removed the features containing too much missing values. However, all the remaining NaN (still present in the two features StdDevLen and StdDevLenRx) must be replaced with real(istic) values in order to fed the dataset to a classi er. The imputation phase compensates these missing values based on the other, concrete values of the same feature.
To this aim, we selected the Multivariate Imputation by Chained Equation (MICE) algorithm, which has high performance and e ciency. This type of imputation works by lling the missing data multiple times. Multiple imputations are better than a single imputation as they measure the uncertainty of the missing values more precisely. The chained equations approach is also very exible and can handle di erent variables of di erent data types (i,e., continuous or binary) as well as bounds or skip patterns [4]. 4.4
The dataset contains features highly varying in magnitudes, units and range. Since most machine learning algorithms use the Eucledian distance between two data points in their computations, it is necessary to scale the data with appropriate methodologies.
However, feature scaling may heavily in uence the results of some algorithms. Therefore, we rst select three a ne transformers and two nonlinear transformers i.e., MinMaxScaler [31], MaxAbsScaler [30], StandardScaler [35], RobustScaler [34], QuantileTransformer [33] and PowerTransformer [32], and try to scale the features with each of them. Then, to have a raw idea of the impact of such scaling on a machine learning algorithm, we use 70% of the scaled dataset to train three classi ers, namely Gaussian Naive Bayes (GNB), Random Forest (RF), and Logistic Regression (LogReg), and then evaluate the reached classi er quality by computing its weighted average precision (i.e., the ratio between correctly predicted positive observations and total predicted positive observations) on the remaining 30%. The results in Table 4 show that the PowerTransformer makes two of the three classi ers reach the highest precision, so our dataset generation process will adopt this scaling methodology. 5
Once the dataset is correctly setup, we can apply some further techniques to evaluate its quality and suitability for machine learning algorithms. In particular, we rst want to understand if it actually contains an adequate amount of outliers, which can be seen as not-normal tra c samples and can be detected by a ML classi er as anomalies, i.e., possible malware. Then, we will also verify the detection accuracy that a more powerful (w.r.t. the GNB used for scaler evaluation) classi er can reach using this dataset. 5.1
The basic assumptions for anomaly detection in network tra c are that \The majority of the network connections are normal tra c, only a small percentage of tra c are malicious " [27] and that \The attack tra c is statistically di erent from normal tra c" [15].
Given the size of our dataset, we cannot apply such an outlier detection to all the features in all the samples. Therefore, we estimated the feature importance based on the Gini index of each feature on the entire dataset through a random forest (RF) classi er [5], and selected the six features with the highest score (the most informative ones), i.e., DNSQDist, StartFlow, MinIATRx, MaxLen, DeltaTime and AvgIAT. Then, we selected the largest malware pcap from
MTA (258.5 MB) and extracted the dataset samples corresponding to its packets. Finally, we identi ed the outliers following the commonly-used rule that marks a data point as an outlier if it is more than 1:5 IQR above the third quartile or below the rst quartile, where IQR is the Inter Quartile Range. To this aim, we used the WEKA framework [12] on the selected features of this reduced dataset.
The results clearly indicate a large number of outliers, which are also graphically shown in Figure 2 as red dots. To further check if such outliers can be actually associated to malware tra c, we identi ed the IP addresses corresponding to the packets generating an outlier and discovered that all these packets were marked as part of a malware attack in the source les. 5.2
The last quality measure for our nal dataset is derived from a classi cation experiment realised using a multilayer perceptron. In previous papers [20] we showed that this kind of classi er, if correctly set-up and fed with a "good" dataset, can reach a very high accuracy. However, the dataset used in [20] was smaller and not updated with respect to the one we present here.
We used a multilayer perceptron with a quite trivial rectangle-shaped fully connected network, trained for 10 epochs, with two hidden layers of 2f neurons each without dropout among them, where f = 34 is the number of features in our nal dataset. We performed a 5-cross fold validation, by splitting the dataset in ve segments and then performing ve experiments, each of which uses a di erent segment (thus 20% of the dataset) as test set and the rest (80%) as the training set. In all the experiments the network performed in a very similar way: as an example, Figure 3 shows the confusion matrix taken from one of the experiments, with an accuracy of 99.74%. Actually, the average accuracy of all the experiments was 99.69% with a standard deviation of 0.24%. Also the other metrics, i.e., speci city, precision and recall, have very good values. This clearly shows that the dataset o ers a good source of information to detect current malware using a neural network classi er. In this paper we presented new dataset, namely MTA-KDD'19, built to ease testing and comparison of machine learning-based malware tra c analysis algorithms. The data sources used are up-to-date, i.e. they contain the most recent malware tra c, and are continuously updated, making the dataset quite realistic. Moreover, we performed an accurate feature selection and data preprocessing in order to make the dataset as small as possible and e ectively usable in a ML classi er, without introducing any experiment or algorithm-speci c bias. Indeed, some preliminary quality measures on the nal dataset show that it constitutes a good source of information to train any kind of ML classi er. The complete dataset is publicly available [18], and we will soon publish as open source on the same site the algorithm that can be used to build and update it.
As a future work, we plan to apply more complex feature selection strategies in order to further reduce the number of features to the most informative ones. Moreover, we are studying how to better handle unavailable feature values, in order to reduce the dataset samples excluded due to this issues, and evaluating the impact of the imputation phase on the dataset quality. Of course, we will further validate our dataset by evaluating its performances on di erent neural network architectures and other machine learning models.