=Paper=
{{Paper
|id=Vol-3731/paper16
|storemode=property
|title=Hardening Machine Learning based Network Intrusion Detection Systems with synthetic netflows
|pdfUrl=https://ceur-ws.org/Vol-3731/paper16.pdf
|volume=Vol-3731
|authors=Andrea Venturi,Dimitri Gall,Dario Stabili,Mirco Marchetti
|dblpUrl=https://dblp.org/rec/conf/itasec/VenturiGSM24
}}
==Hardening Machine Learning based Network Intrusion Detection Systems with synthetic netflows==
https://ceur-ws.org/Vol-3731/paper16.pdf
Hardening Machine Learning based Network
Intrusion Detection Systems with Synthetic NetFlows
Andrea Venturi1,∗,† , Dimitri Galli1,† , Dario Stabili2 and Mirco Marchetti1
1
University of Modena and Reggio Emilia, Department of Engineering “Enzo Ferrari”, 41125 Modena, Italy
2
University of Bologna, Department of Computer Science and Engineering, 40126 Bologna, Italy
Abstract
Modern Network Intrusion Detection Systems (NIDS) involve Machine Learning (ML) algorithms to
automate the detection process. Although this integration has significantly enhanced their efficiency, ML
models have been found vulnerable to adversarial attacks, which alter the input data to fool the detectors
into producing a misclassification. Among the proposed countermeasures, adversarial training appears
to be the most promising technique; however, it demands a large number of adversarial samples, which
typically have to be manually produced. We overcome this limitation by introducing a novel methodology
that employs a Graph AutoEncoder (GAE) to generate synthetic traffic records automatically. By design,
the generated samples exhibit alterations in the attributes compared to the original netflows, making
them suitable for use as adversarial samples during the adversarial training procedure. By injecting the
generated samples into the training set, we obtain hardened detectors with better resilience to adversarial
attacks. Our experimental campaign based on a public dataset of real enterprise network traffic also
demonstrates that the proposed method even improves the detection rates of the hardened detectors in
non-adversarial settings.
Keywords
ML-based NIDS, Graph Neural Network, Data Augmentation, Adversarial Training
1. Introduction
Modern Network Intrusion Detection Systems (NIDS) are increasingly relying on some Machine
Learning (ML) mechanisms to automate their tasks [1]. Nevertheless, despite the proven
efficiency of ML algorithms in detecting malicious activities [2], there is also a large research
trend evidencing how ML-based NIDS are also affected by severe vulnerabilities. Among these,
the most prominent threat is posed by adversarial attacks, which consist of small perturbations
applied to the input samples to fool the detector into producing an incorrect classification [3].
As the cost of even a single misclassification is extremely high in security-related contexts,
it is essential to devise valid countermeasures against adversarial attacks. Yet, no existing
solution completely addresses all types of attacks and is suitable for all the possible models [4].
In this paper, we propose a novel methodology to improve the robustness of ML-based NIDS
ITASEC 2024: The Italian Conference on CyberSecurity, April 09–11, 2024, Salerno, Italy
∗
Corresponding author.
†
These authors contributed equally.
Envelope-Open andrea.venturi@unimore.it (A. Venturi); dimitri.galli@unimore.it (D. Galli); dario.stabili@unibo.it (D. Stabili);
mirco.marchetti@unimore.it (M. Marchetti)
Orcid 0000-0003-3822-968X (A. Venturi); 0009-0006-0280-2498 (D. Galli); 0000-0001-6850-334X (D. Stabili);
0000-0002-7408-6906 (M. Marchetti)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
against adversarial attacks. Our approach leverages the concept of adversarial training, in which
adversarial records are incorporated into the training set of vulnerable ML-based NIDS. By
re-training them on the resulting augmented dataset, the obtained hardened ML-based NIDS
can detect even the perturbed samples.
Despite its efficacy, even adversarial training is affected by several limitations as (i) it requires
a large number of adversarial samples typically produced manually; (ii) the adversarial records
injected into the training set must be representative of all the possible perturbations that
an attacker could apply; (iii) the hardened classifiers are often subject to a decrease in the
performance in non-adversarial contexts. We overcome these limitations by proposing an
original approach for generating synthetic network flows automatically, leveraging a novel
architecture that extends the Adversarially Regularized Graph AutoEncoder (ARGA) model [5].
We hypothesize that the samples generated through our procedure can be treated as candidate
adversarial attacks suitable for the adversarial training procedure, thus avoiding the expensive
and error-prone manual production of a sufficient number of representative adversarial samples.
We apply our methodology to an experimental case study, considering a state-of-the-art
ML-based NIDS trained on a publicly available dataset. We verify our hypothesis, demonstrating
that the proposed approach leads to hardened detectors that are more resilient when facing
adversarial attacks than the baseline original versions. As an additional positive effect, we
highlight that the hardened detectors also show improved performance in non-adversarial
contexts, overcoming the most important limitation of adversarial training techniques.
The remainder of the paper is organized as follows. Section 2 introduces adversarial attacks
and compares our paper against related work. Section 3 describes the proposed methodology.
Section 4 defines the case study used for performance evaluation. Section 5 presents and
discusses all the experimental results. Section 6 draws the conclusions with some final remarks.
2. Background and Related Work
ML-based NIDS have demonstrated high efficacy in identifying malicious activities with minimal
false positives. These tools usually analyze network traffic in the form of netflows, which are
tabular data structures summarizing the characteristics of the communication between two
hosts in a network with metadata and statistics (e.g., duration of the transmission, number of
exchanged bytes). Network flows are extremely popular in network intrusion detection as they
allow fast analysis of the traffic data without incurring storage or privacy issues [6].
Despite the benefits provided by the introduction of ML algorithms in the detection processes,
they are also shown to be vulnerable to adversarial attacks [7, 8]. In this paper, we focus on
adversarial attacks based on evasive samples, where an attacker applies smart perturbations
to the features of a malicious netflow so that it is incorrectly classified as benign by a trained
ML-based NIDS. To obtain a suitable adversarial attack, an attacker must solve the following
minimization problem [9]:
arg min 𝑓 (𝑥 ′ ) ≠ 𝑓 (𝑥), with 𝑥′ = 𝑥 + 𝜖 (1)
𝜖
where 𝑥 is the original sample, 𝑓 (𝑥) denotes the output of the ML model 𝑓 computed on 𝑥 and 𝜖 is
the perturbation added to 𝑥 to create the adversarial sample 𝑥 ′ , which leads to a misclassification.
Adversarial attacks are particularly critical in security contexts where a single mistake
can have significant consequences for an organization [10]. Previous research on adversarial
attacks against ML-based NIDS leveraging manipulated samples has shown that even minimal
perturbations can drastically decrease the performance of ML detectors [11].
Some countermeasures exist, but the solutions are still at an early stage, especially in network
intrusion detection [4]. Among existing strategies in the literature, adversarial training is
indicated as one of the most effective [12]. The idea is to train ML algorithms over augmented
datasets that include adversarial perturbed records [13]. However, this process requires the
generation of many adversarial samples to be effective. In addition, adversarially manipulated
samples to be injected into the dataset must be representative of a wide range of attacks.
We overcome these limitations by proposing a method based on a novel Graph AutoEncoder
(GAE) architecture that is able to generate automatically a large number of potential adversarial
samples that can be used in the adversarial training procedure. The GAE allows us to take
into account the network topology in the generation process. Since graph representation can
express the complex dependencies among different malicious transmissions, the generative
process leveraging the Graph Neural Network (GNN) is proven to be more effective.
Similar approaches exist in the literature. The papers in [14] and [15] represent our closest
related work. In particular, the authors of [14] adopt an architecture based on a Generative
Adversarial Network (GAN) to automatically generate adversarial examples that, when included
in the training set, lead to good robustness. However, they do not evaluate the performance
of the hardened detector in non-adversarial contexts. This limitation is also present in other
similar papers (e.g., [16, 17]). Instead, we evaluate the hardened classifiers even in normal
scenarios, and we demonstrate that our approach leads to increased detection rates. We remark
that other countermeasures are affected by a drop in performance when no adversarial attack
occurs (e.g., [9, 18, 19]). On the other hand, in [15], a framework based on Deep Reinforcement
Learning (DRL) is employed to perturb samples to obtain an adversarial evasion, and the results
are used as a means for adversarial training. With respect to this solution, we do not generate
evasive samples explicitly, but rather we reconstruct netflows that are similar to the original
ones that might not bypass the classifiers. As we will explain in Section 3, our approach exploits
this similarity to enhance the detection performance of the ML-based NIDS, even in the absence
of adversarial attacks.
3. Methodology
In this section, we present our novel methodology to strengthen the robustness of ML-based
NIDS against adversarial attacks. The general idea is to follow a strategy based on adversarial
training, in which the original ML detectors are re-trained over an augmented training set
containing many adversarial samples. This process leads to the development of hardened
classifiers, which are capable of correctly identifying adversarial attacks with high confidence.
As discussed before, adversarial training is among the most promising defensive techniques,
but it has several limitations. Our approach overcomes them with an automatic generation of
realistic — yet synthetic — netflows that, when injected into the training set of a ML-based NIDS,
can enhance the detector’s resilience to adversarial attacks. In particular, our methodology
0 0
0 EMBEDDING FLOW 0
PROCESSING HARDENING
1 GENERATION RECONSTRUCTION 1
0 0
Original Graph Embeddings Synthetic Detector
Flows Flows
Figure 1: Methodology phases.
leverages an architecture based on ARGA [5], a recently proposed GNN for graph embedding.
Our choice is driven by the ability of GAE models to also take into account the topology
of the network when encoding the records. This allows the production of more accurate
synthetic samples with respect to traditional autoencoders, which consider each data point
individually [20].
The workflow of our methodology consists of four phases, as illustrated in Figure 1. The first
phase, referred to as processing, converts the original netflow data into a graph representation.
Hence, the embedding generation phase leverages the ARGA architecture, which accepts the
graph as input and embeds both features and topological information of its components in
compact vector representations. Then, in the flow reconstruction phase, these embeddings are
used to reconstruct the traffic samples, making sure that they represent valid netflows. Finally,
the hardening phase uses the reconstructed malicious flows as adversarial samples to inject into
the training sets of ML-based NIDS during the adversarial training process for obtaining the
hardened classifiers.
In the following, a more detailed explanation of the considered threat model and of the four
phases of the proposed methodology.
3.1. Threat Model
We consider the same threat model previously proposed in [4]. In this scenario, a medium-sized
enterprise with several internal hosts communicates with the internet through a single border
router. The network is monitored using a ML-based NIDS, which analyzes traffic through
netflow data from the border router. We suppose that some internal network hosts have been
compromised by remote attackers capable of controlling them. Their purpose is to evade
the detection of their malicious communications perturbing a close subset of features of the
corresponding netflows, similarly as in previous proposals [11, 14]. For example, they can
increment the duration of the malicious netflows by retarding the communications. Similarly,
they can increment the number of bytes or packets by injecting junk data. Our proposal aims
to devise a hardened version of the ML-based NIDS that is not susceptible to these kinds of
modifications. We remark that our approach does not insert any novel component to the
monitored network, and instead operates directly on the ML-based NIDS.
3.2. Processing
The first phase involves all the operations required to convert the network traffic into a suitable
graph representation that can be submitted to the ARGA model in the subsequent phase.
A graph is formally defined as G = (V, E), where V denotes the set of vertices and E represents
the set of edges. Assuming that the network traffic is provided in the form of netflows D, which
are the most common data representation in network intrusion detection, it can be naturally
converted into a graph. A common approach associates the endpoints present in the netflows
to the nodes, while the edges correspond to every netflow of the dataset. In other words, an
edge links together the two endpoints appearing in a netflow as the source and destination of
the communication and stores its features. This type of graph is denoted as flow graph GD [21].
With flow graphs, the flow generation task requires a GAE capable of reconstructing edge
features. Nevertheless, it is important to remark that most GNN in the literature are designed
to perform tasks at the node level rather than on edges [22]. This motivates us to consider a
dual graph representation named line graph 𝐿(GD ). Line graphs can be obtained from flow
graphs through a linearization procedure [23]. In particular, the edges in GD are translated to
the nodes of 𝐿(GD ). Then, two nodes in 𝐿(GD ) are connected if the corresponding edges in GD
share a common endpoint. This representation allows us to convert the task of generating new
samples from an edge generation problem in GD to a node generation problem in 𝐿(GD ), with
the advantage of exploiting the most advanced GNN architectures.
3.3. Embedding Generation
After having obtained a suitable graph representation, we proceed to generate synthetic netflow
data through our innovative architecture. We logically divide this core operation into two
separated — yet extremely connected — phases: embedding generation and flow reconstruction.
The former stage aims to produce vector representations of the nodes in the line graph in a
low-dimensional space, which will be then utilized to reconstruct the original records in the
latter stage.
For the first step, we use the Adversarially Regularized Graph Autoencoder (ARGA) model [5],
which is an unsupervised model that has been recently adopted in network intrusion detection
with evident success [24]. In particular, ARGA generates an embedded representation for each
node of 𝐿(GD ). As a thorough description of the architecture is out of the scope of this paper,
we outline here the fundamental principles of its design and operation.
The architecture of ARGA consists of two main modules: a GAE and an adversarial module.
The GAE generates the embeddings Z, and is composed of an encoder and a decoder. The
encoder accepts as input the adjacency matrix A ∈ {0, 1}𝑛×𝑛 of 𝐿(GD ), which is a square matrix
that expresses the connections for each of the 𝑛 nodes, and the content matrix X ∈ ℝ𝑛×𝑓 of 𝐿(GD ),
in which each row is the 𝑓-dimensional feature vector of each node. The encoder produces
the embeddings Z ∈ ℝ𝑛×𝑑 through a Graph Convolutional Network (GCN) [25], encoding both
topological information about the neighborhood of a node (expressed by matrix 𝐴) and its
netflow features (expressed by matrix 𝑋) into a single vector of dimensions 𝑑 << 𝑓. The decoder
takes Z as input and aims to reconstruct the adjacency matrix A, leveraging another GCN. The
training process essentially aims to minimize the reconstruction error between the original
adjacency matrix A and the reconstructed one A′ . In other words, throughout training, the
encoder learns to generate embeddings with increasing quality, while the decoder exploits the
information encoded in the embeddings to reconstruct A with high fidelity.
The adversarial module employs a Multi-Layer Perceptron (MLP) to recognize whether the
0
Y'
0
Classifier
1
z1 x'1 0
z2 x'2 0
Z D'
zn x'n 1
x'1
x'2
Regressor
X' x'n
Figure 2: Architecture for the reconstruction of the original samples.
latent variable belongs to a real prior distribution (e.g., a Gaussian distribution) or is generated
by the generator (i.e., the encoder). By training the GAE and the adversarial module together, it
is possible to force the latent variables to follow a particular probability distribution, enhancing
their robustness even in the case of noisy and sparse input data [5].
3.4. Flow Reconstruction
The ARGA model is capable of reconstructing the A matrix. Nevertheless, we are interested in
producing a set of synthetic netflows to inject into the training set of ML-based NIDS.
To this purpose, we propose an original architecture based on two independent components:
a classifier and a regressor, as illustrated in Figure 2. Both components take the embeddings
generated in the previous phase as input. However, they serve different goals: the regressor aims
to reconstruct the features X′ of the initial netflows, while the classifier is used to reproduce
the respective labels Y′ . We then merge the two parts to obtain the reconstructed dataset
D′ =< X′ , Y′ >.
We remark that even the original paper for ARGA proposes a variant model, named
ARGA_AX, that was able to reconstruct the content matrix [5]. However, two main reasons
guided our choice of a novel architecture based on a separate classifier and regressor. First, the
original methodology does not involve any label reconstruction. On the other hand, for our
purposes, it is crucial to consider also the labels in the reconstruction process, as we aim to
harden supervised ML-based NIDS. Second, we note that there exist numerous papers indicating
that GCN-based decoders are not effective when reconstructing the features of the original
samples (e.g., [26, 27]). We confirm this limitation of GCN architectures in our preliminary
experimental campaign before the production of this paper.
The training workflow for this stage is similar to the one used for the decoder of the GAE with
both the components trying to minimize the reconstruction error between the original records
D and the reproduced samples D′ . In particular, the regressor minimizes the error between X
and X′ , while the classifier minimizes the difference between Y and Y′ . This translates to a
generation of a new set of synthetic netflows that can be related to the original ones using the
following equation:
D′ = D + 𝜖 (2)
where D and D′ refer to the initial set of netflows provided as input to the procedure and the
output of the entire process, respectively, and the 𝜖 represents the error introduced by our
architecture. Our intuition is that the perturbation produced by an attacker when devising
adversarial evasion samples (Equation 1) can be interpreted as the 𝜖 in Equation 2. Hence, the
netflows in D′ represent adversarial attack candidates that could evade a given ML-based NIDS.
In this sense, we hypothesize that we can employ them to perform adversarial training.
We remark, however, that a careless generation procedure can lead to an incorrect set of
synthetic netflows with unfeasible feature values, whose introduction in a training set could
corrupt the learning procedure of the ML-based NIDS. For this reason, we verify that the
generated netflows present features that are feasible in practice. For example, we impose that
the regressor must not produce negative values for numerical features (e.g., duration, number
of exchanged bytes). Similarly, we exclude from D′ the netflows presenting values outside the
corresponding feature domain or with wrong values in the derived features (e.g., bytes per
second). In this way, we make sure that the reconstructed records are representative of realistic
samples and can thus be safely injected into the training set.
3.5. Hardening
The last phase involves all the operations necessary to perform adversarial training on a target
ML-based NIDS. Supposing that the original ML-based NIDS has been trained on the dataset D,
we re-train it over an augmented dataset D∗ that is formed by merging together the original
records D and the generated samples D′ . Our hypothesis is the same as in adversarial training:
because the dataset D′ includes perturbed netflows acting as adversarial samples, the final
hardened ML-based NIDS is capable of detecting even adversarial attacks. However, as the
samples in the set D′ are designed to be similar to the original ones in the set D, our adversarial
training procedure can enhance the detection performance in the absence of adversarial attacks.
4. Case Study
In this section, we describe the experimental case study in which we apply our approach to
enhance the resilience of an ML-based NIDS at the state-of-the-art. We present the considered
dataset and the details of the implementation of the methodology1 .
4.1. Dataset
We base our case study on a widely used dataset for network intrusion detection: ToN-IoT [28].
ToN-IoT is a collection containing telemetry data of IoT/IIoT sensors, operating systems logs,
and network traffic traces. For our purposes, we consider only the reduced network traces,
which include various cyberattacks and normal traffic from an enterprise IoT network.
We apply common preprocessing operations as in [24]. We consider Backdoor, DDoS, DoS,
Injection, Password, Ransomware, Scanning, and XSS attacks in our evaluation, excluding those
with a too small number of netflows that would prevent a valid performance assessment. Hence,
1
Source code available at https://github.com/dimgalli/hardening-ml-nids.git.
we build eight separate collections of data where each contains 200 000 normal records and
10 000 attack samples, to maintain a realistic 20:1 benign:malicious ratio. We refer to each
dataset as D𝛼 , where 𝛼 indicates the type of attack considered. These will be used as input to our
flow generation methodology, and for training and evaluating the ML-based NIDS (Section 4.2).
4.2. Implementation
We now detail the implementation of the introduced methodology, starting from the detector
target of our analysis, and then passing to each proposed phase.
4.2.1. Detector
In order to obtain the ML-based NIDS target of our analysis, we follow the best practices
considering binary classifiers tailored to detect specific attack variants, rather than a multi-class
solution [29].
We use Random Forest (RF) classifiers, as related literature indicates that this algorithm is
among the best-performing models for network intrusion detection tasks [30]. We utilize the
same parameters as in previous work for adversarial attacks against ML-based NIDS [31].
We train two versions of each RF instance. A first baseline version is trained using the original
samples provided in the considered dataset D𝛼 . Instead, a second hardened version is trained
using the augmented dataset D∗𝛼 as described in Section 3.5, and as we will detail in Section 4.2.3.
The training-test split used to form the respective sets for both versions of each RF instance
is 80-20, regardless of the considered dataset (original or augmented). To simulate realistic
scenarios, we maintain a benign:malicious ratio of 20:1 in both training and test sets [32].
4.2.2. Flow Generation
During the processing stage, we build a line graph 𝐿(G𝐷𝛼 ) from the dataset D𝛼 described in
Section 4.1. The purpose is to train an instance of our model that is tailored to produce netflows
from each specific attack in ToN-IoT.
The embedding generation procedure accepts as input 𝐿(G𝐷𝛼 ) obtained in the processing
stage and uses the ARGA model to produce the encoded variables Z. In our case study, we
choose the ARVGA_GD variant [5]. The architecture includes a variational encoder and a
decoder with two consecutive GCN layers. The discriminator is built on a standard MLP with
one hidden layer and one output neuron. We keep the same parameters as in the original paper.
As classifier and regressor for the flow reconstruction stage, we again rely on the RF algorithm.
Both the RF classifier and the RF regressor have a number of trees equal to 10, while the criterion
to measure the quality of splits is the Gini impurity for the classification model and the mean
squared error for the regression model. In the end, we obtain as output the synthetic flows D′𝛼 .
4.2.3. Hardening
In the hardening stage, we use the generated netflows D′𝛼 to create the augmented dataset D∗𝛼 .
In particular, we inject only the synthetic malicious flows of each D′𝛼 into the respective
D𝛼 to obtain D∗𝛼 . This is because attackers typically manipulate only malicious netflows to
evade detection. Hence, we are not interested in perturbations of benign data points. As this
procedure doubles the number of malicious records in D∗𝛼 with respect to the ones present in
D𝛼 , we maintain a 20:1 benign:malicious ratio by inserting the necessary benign samples from
the pool of available ones.
After having obtained the augmented dataset, we proceed to re-train the RF classifiers on the
corresponding D∗𝛼 to create the hardened versions.
5. Evaluation and Results
In this section, we detail the experiments of our case study. We first introduce the evaluation
scenarios, and then we present the results.
For each instance of the RF detectors (i.e., baseline or hardened), we consider two different
evaluation scenarios: standard evaluation (Section 5.1) and adversarial evaluation (Section 5.2).
In the standard evaluation scenario, we aim to validate whether the considered classifiers
exhibit good detection capabilities in the absence of adversarial attacks. Hence, we test each
model using the corresponding original test set obtained from D𝛼 . The goal of this experiment
is twofold. First, it lets us assess the goodness of the baseline detectors. Second, it lets us verify
that the defensive methodology that we propose does not produce hardened models with lower
detection rates in non-adversarial settings with respect to the baseline versions. This is of
crucial importance because otherwise, the defensive efforts to enhance resilience in the presence
of adversarial attacks would be wasted by a lower level of robustness in normal conditions.
In the adversarial evaluation scenario, we assess the detection performance of the baseline
and hardened detectors against adversarial attacks. In particular, we choose an attack strategy
that has been demonstrated to evade RF-based NIDS in [11]. We remark that this strategy alters
only the malicious netflows for each attack by manipulating combinations of different features
with increasing perturbation steps. We choose to report the average performance score of each
model among all the altered attributes and the perturbation steps.
We evaluate the classifiers by considering performance metrics that are largely employed
in network intrusion detection. In particular, we use the F1-score for the standard evaluation,
while we utilize the Detection Rate (DR) for the adversarial evaluation. The scores of these
metrics range between 0 and 1, with higher values indicating better performance.
5.1. Standard Evaluation
We evaluate initially the baseline and the hardened classifiers in non-adversarial contexts. The
results are presented in Table 1a, where each value represents the F1-score obtained by each
instance of the baseline and hardened models against the corresponding test set. For each row,
we use the bold to indicate the best value. The last row summarizes the scores of each instance
with the average value among all models.
The results have a twofold value. First, we observe that the baseline classifiers obtain
performance scores that are similar to those of the state-of-the-art [33, 34], with an average of
0.981. This is an indicator that the considered RF detectors represent a valid benchmark for our
experiments. More notably, we note that, in this scenario, the hardened instances obtain scores
that are even higher than baseline models, with an average of 0.987. We remark that the unique
Table 1
Performance scores of the baseline and hardened detectors in the two evaluation settings.
(a) Scores in standard evaluation. (b) Scores in adversarial evaluation.
F1-score DR
Attack Attack
Baseline Hardened Baseline Hardened
Backdoor 1.000 0.999 Backdoor 0.613 0.747
DDoS 0.981 0.988 DDoS 0.781 0.828
DoS 0.995 0.997 DoS 0.465 0.917
Injection 0.991 0.996 Injection 0.887 0.972
Password 0.983 0.987 Password 0.567 0.624
Ransomware 0.923 0.939 Ransomware 0.269 0.587
Scanning 0.998 0.998 Scanning 0.727 0.995
XSS 0.978 0.987 XSS 0.925 0.945
avg 0.981 0.987 avg 0.654 0.827
case in which the F1-score value for the hardened detector is lower than the baseline model is
the Backdoor attack, but the drop is negligible (just 0.001), and the performance is maintained
at top-level (0.999).
We can conclude that this experiment demonstrates that our approach devises hardened
ML-based NIDS that are effective even in absence of adversarial attacks. This property is crucial
for obtaining models that can be deployed in real-world scenarios.
5.2. Adversarial Evaluation
Then, we pass to assess the performance of the baseline and the hardened classifiers in the
adversarial scenario. For this case, in Table 1b, we report the average performance scores
obtained by each detector against the adversarial attacks produced following the strategy
proposed in [11]. We use the Detection Rate (DR or Recall) as the performance metric.
We immediately notice that the baseline classifiers show insufficient performance in detecting
adversarial samples, with an average DR of just 0.654. This denotes the lack of robustness of
baseline detectors, and it is in line with the expectations from the state-of-the-art. On the
contrary, we observe that hardened models are far more robust with an average DR of 0.827
(an increment of over 26%). Among the classifiers, we highlight those for the DoS, Injection,
Scanning and XSS attacks that are even above a DR equals to 0.9. This positive result further
testifies to the effectiveness of the proposed methodology as a valid countermeasure against
adversarial attacks in network intrusion detection.
6. Conclusions
Adversarial attacks represent a serious menace to ML-based NIDS, as they allow skilled attackers
to evade detection. Some countermeasures exist, but the research is still at an early stage. In
this paper, we propose a novel approach to enhance the robustness of ML-based NIDS against
adversarial attacks. In particular, we present a novel architecture based on a GAE that is able to
generate automatically synthetic netflows that we exploit as a means to adversarial training,
a popular defensive technique in the literature. We apply our methodology to a case study
involving a state-of-the-art ML-based NIDS trained on a public dataset containing real traffic
traces. The results highlight the efficacy of our method in enhancing the robustness of the
detectors when facing adversarial attacks. Moreover, we denote that our approach is capable
of not reducing the performance of the hardened classifiers even in the absence of adversarial
attacks, overcoming the limitations of traditional adversarial training scenarios.
References
[1] A. L. Buczak, E. Guven, A survey of data mining and machine learning methods for
cyber security intrusion detection, IEEE Communications surveys & tutorials 18 (2015)
1153–1176.
[2] R. Sommer, V. Paxson, Outside the closed world: On using machine learning for network
intrusion detection, in: 2010 IEEE symposium on security and privacy, IEEE, 2010, pp.
305–316.
[3] B. Biggio, F. Roli, Wild patterns: Ten years after the rise of adversarial machine learning,
in: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications
Security, 2018, pp. 2154–2156.
[4] G. Apruzzese, M. Andreolini, L. Ferretti, M. Marchetti, M. Colajanni, Modeling realistic
adversarial attacks against network intrusion detection systems, Digital Threats: Research
and Practice (DTRAP) 3 (2022) 1–19.
[5] S. Pan, R. Hu, S.-f. Fung, G. Long, J. Jiang, C. Zhang, Learning graph embedding with
adversarial training methods, IEEE transactions on cybernetics 50 (2019) 2475–2487.
[6] A. Yehezkel, E. Elyashiv, O. Soffer, Network anomaly detection using transfer learning
based on auto-encoders loss normalization, in: Proceedings of the 14th ACM Workshop
on Artificial Intelligence and Security, 2021, pp. 61–71.
[7] B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Šrndić, P. Laskov, G. Giacinto, F. Roli, Evasion
attacks against machine learning at test time, in: Machine Learning and Knowledge
Discovery in Databases: European Conference, ECML PKDD 2013, Prague, Czech Republic,
September 23-27, 2013, Proceedings, Part III 13, Springer, 2013, pp. 387–402.
[8] A. Venturi, C. Zanasi, On the feasibility of adversarial machine learning in malware and
network intrusion detection, in: 2021 IEEE 20th International Symposium on Network
Computing and Applications (NCA), IEEE, 2021, pp. 1–8.
[9] K. Grosse, N. Papernot, P. Manoharan, M. Backes, P. McDaniel, Adversarial examples for
malware detection, in: Computer Security–ESORICS 2017: 22nd European Symposium on
Research in Computer Security, Oslo, Norway, September 11-15, 2017, Proceedings, Part II
22, Springer, 2017, pp. 62–79.
[10] K. Grosse, L. Bieringer, T. R. Besold, B. Biggio, K. Krombholz, Machine learning security in
industry: A quantitative survey, IEEE Transactions on Information Forensics and Security
18 (2023) 1749–1762.
[11] G. Apruzzese, M. Colajanni, Evading botnet detectors based on flows and random forest
with adversarial samples, in: 2018 IEEE 17th International Symposium on Network
Computing and Applications (NCA), IEEE, 2018, pp. 1–8.
[12] P. Maini, E. Wong, Z. Kolter, Adversarial robustness against the union of multiple per-
turbation models, in: International Conference on Machine Learning, PMLR, 2020, pp.
6640–6650.
[13] A. Kantchelian, J. D. Tygar, A. Joseph, Evasion and hardening of tree ensemble classifiers,
in: International conference on machine learning, PMLR, 2016, pp. 2387–2396.
[14] M. Usama, M. Asim, S. Latif, J. Qadir, et al., Generative adversarial networks for launching
and thwarting adversarial attacks on network intrusion detection systems, in: 2019 15th
international wireless communications & mobile computing conference (IWCMC), IEEE,
2019, pp. 78–83.
[15] G. Apruzzese, M. Andreolini, M. Marchetti, A. Venturi, M. Colajanni, Deep reinforcement
adversarial learning against botnet evasion attacks, IEEE Transactions on Network and
Service Management 17 (2020) 1975–1987.
[16] H. S. Anderson, J. Woodbridge, B. Filar, Deepdga: Adversarially-tuned domain generation
and detection, in: Proceedings of the 2016 ACM workshop on artificial intelligence and
security, 2016, pp. 13–21.
[17] Y. Ji, B. Bowman, H. H. Huang, Securing malware cognitive systems against adversarial
attacks, in: 2019 IEEE international conference on cognitive computing (ICCC), IEEE,
2019, pp. 1–9.
[18] A. Al-Dujaili, A. Huang, E. Hemberg, U.-M. O’Reilly, Adversarial deep learning for robust
detection of binary encoded malware, in: 2018 IEEE Security and Privacy Workshops
(SPW), IEEE, 2018, pp. 76–82.
[19] G. Apruzzese, M. Colajanni, L. Ferretti, M. Marchetti, Addressing adversarial attacks against
security systems based on machine learning, in: 2019 11th international conference on
cyber conflict (CyCon), volume 900, IEEE, 2019, pp. 1–18.
[20] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, G. Monfardini, The graph neural
network model, IEEE transactions on neural networks 20 (2008) 61–80.
[21] W. W. Lo, S. Layeghy, M. Sarhan, M. Gallagher, M. Portmann, E-graphsage: A graph neural
network based intrusion detection system for iot, in: NOMS 2022-2022 IEEE/IFIP Network
Operations and Management Symposium, IEEE, 2022, pp. 1–9.
[22] S. Zhang, H. Tong, J. Xu, R. Maciejewski, Graph convolutional networks: a comprehensive
review, Computational Social Networks 6 (2019) 1–23.
[23] F. Harary, R. Z. Norman, Some properties of line digraphs, Rendiconti del circolo matem-
atico di palermo 9 (1960) 161–168.
[24] A. Venturi, M. Ferrari, M. Marchetti, M. Colajanni, Arganids: a novel network intrusion
detection system based on adversarially regularized graph autoencoder, in: Proceedings
of the 38th ACM/SIGAPP Symposium on Applied Computing, 2023, pp. 1540–1548.
[25] T. N. Kipf, M. Welling, Semi-supervised classification with graph convolutional networks,
arXiv preprint arXiv:1609.02907 (2016).
[26] J. Park, M. Lee, H. J. Chang, K. Lee, J. Y. Choi, Symmetric graph convolutional autoen-
coder for unsupervised graph representation learning, in: Proceedings of the IEEE/CVF
international conference on computer vision, 2019, pp. 6519–6528.
[27] M. Ma, S. Na, H. Wang, Aegcn: An autoencoder-constrained graph convolutional network,
Neurocomputing 432 (2021) 21–31.
[28] A. Alsaedi, N. Moustafa, Z. Tari, A. Mahmood, A. Anwar, Ton_iot telemetry dataset: A
new generation dataset of iot and iiot for data-driven intrusion detection systems, IEEE
Access 8 (2020) 165130–165150.
[29] M. Stevanovic, J. M. Pedersen, An analysis of network traffic classification for botnet
detection, in: 2015 International Conference on Cyber Situational Awareness, Data
Analytics and Assessment (CyberSA), IEEE, 2015, pp. 1–8.
[30] B. Abraham, A. Mandya, R. Bapat, F. Alali, D. E. Brown, M. Veeraraghavan, A comparison
of machine learning approaches to detect botnet traffic, in: 2018 International Joint
Conference on Neural Networks (IJCNN), IEEE, 2018, pp. 1–8.
[31] G. Apruzzese, M. Colajanni, L. Ferretti, A. Guido, M. Marchetti, On the effectiveness of
machine and deep learning for cyber security, in: 2018 10th international conference on
cyber Conflict (CyCon), IEEE, 2018, pp. 371–390.
[32] D. Arp, E. Quiring, F. Pendlebury, A. Warnecke, F. Pierazzi, C. Wressnegger, L. Cavallaro,
K. Rieck, Dos and don’ts of machine learning in computer security, in: 31st USENIX
Security Symposium (USENIX Security 22), 2022, pp. 3971–3988.
[33] A. R. Gad, A. A. Nashat, T. M. Barkat, Intrusion detection system using machine learning for
vehicular ad hoc networks based on ton-iot dataset, IEEE Access 9 (2021) 142206–142217.
[34] A. Sharma, H. Babbar, A. Sharma, Ton-iot: Detection of attacks on internet of things in
vehicular networks, in: 2022 6th International Conference on Electronics, Communication
and Aerospace Technology, IEEE, 2022, pp. 539–545.