The most recent proposals of Machine and Deep Learning algorithms for Network Intrusion Detection Systems (NIDS) leverage Graph Neural Networks (GNN). These techniques create a graph representation of network trafic and analyze both network topology and netflow features to produce more accurate predictions. Although prior research shows promising results, they are biased by evaluation methodologies that are incompatible with real-world online intrusion detection. We are the first to identify these issues and to evaluate the performance of a state-of-the-art GNN-NIDS under real-world constraints. The experiments demonstrate that the literature overestimates the detection performance of GNN-based NIDS. Our results analyze and discuss the trade-of between detection delay and detection performance for diferent types of attacks, thus paving the way for the practical deployment of GNN-based NIDS.
In this paper, we identify and discuss two common pitfalls in the evaluation methodology of related works that lead to unreliable results. Firstly, they consider randomly sampled subsets of the original trafic capture to form the train and test sets, respectively. However, such a random division may conceal some contiguous structural patterns exhibited by malicious trafic that are important for correct detection. Secondly, they only consider complete evaluation graphs built on large test sets. There are two issues with this assumption: it is inapplicable in real-world settings in which it is unknown when it is the right time to construct the graph and perform the evaluation; it considers huge graphs built over time spans of several hours, thus introducing unacceptable detection delays.
We propose a novel evaluation methodology that overcomes the identified issues by avoiding a random division between train and test sets and evaluating the GNN-based detectors by varying the time-window on which to build the evaluation graphs. In this way, we can assess whether and how much the completeness of the graph is important for the classification performed by these tools, and we can evaluate the trade-of between detection delay and detection accuracy.
We apply our methodology to a recent GNN-based NIDS at the state-of-the-art and consider two publicly available datasets largely employed in related literature. The results show diferent performance patterns. Most classifiers exhibit an unstable behavior when the time-windows are too short, suggesting a lower bound for the detection delay. On the other hand, a subset of them is robust to these perturbations, suggesting that, in certain situations, the GNN does not rely on structural patterns for the predictions. These findings provide useful insights for the deployment of GNN-based NIDS and pave the way for additional investigations aimed at devising more eficient solutions.
The remainder of this paper is structured as follows. Section 2 provides background on GNN and discusses related work. Section 3 focuses on the pitfalls of previous approaches and presents our novel evaluation methodology. Section 4.3 introduces the considered case study, while Section 5 shows the experimental results. Finally, Section 6 draws the conclusions and provides insights for future research directions.
Modern NIDS increasingly involve ML techniques for their tasks. Traditional approaches are
based on network flows (or netflows ), which consist of tabular structures in which each entry
summarizes, with a predefined set of statistics and metrics (also referred to as features), the
communication between two endpoints in a network over a certain period of time [
Traditional ML models treat netflows one at a time and independently from each other [
An alternative and recent research trend aims to address this issue by proposing the usage of Graph Neural Networks (GNNs) for network intrusion detection [11]. These models belong to a modern family of Deep Learning (DL) techniques that do not operate over the classical Euclidean space but accept a graph as an input to the model. The idea behind this approach is that cyber-attacks present some topological patterns that can be easily detected by looking at the overall network graph structure. In other words, a graph representation can express the complex dependencies among diferent malicious operations so that a GNN-based NIDS can lead to robust detectors that do not rely on single netflows.
Related literature for GNN-based NIDS has demonstrated the value of these tools both
in traditional and adversarial environments [
Despite the good performance of GNN-based NIDS, there is still a significant gap between research and practice. As we will discuss in Section 3, GNN-based NIDS are evaluated on a graph built from a random testing subset of the considered dataset, which may conceal some crucial topology information. Most importantly, this evaluation approach does not consider the temporal constraints required to build the graph itself in real-world settings. In this paper, our aim is to bridge this gap by proposing a more realistic evaluation methodology that should be followed to better assess the detection capabilities of GNN-based NIDS in real-world scenarios.
GNN-based NIDS follow a two-step workflow in which the graph is usually built only before training and evaluating the network. Related works employ an evaluation methodology that mimics that used for standard ML algorithms, in which a given dataset is randomly separated into train and test sets, which are then used to create the training and test graphs, respectively. We identify two main pitfalls with this approach if applied to network intrusion detection. Firstly, the random division between training and test samples is valid only when the data samples are evaluated independently of each other, as in classical ML algorithms [16]. Instead, GNNs link correlated nodes, and hence, a random division can break the underlying topology of the network structure and lead the models to learn an incorrect graph structure. Secondly, a similar evaluation methodology builds a test graph using all the data in the test set. As dataset captures span several hours or even days [17, 18], building a graph on such a large test set would require waiting for a long time to collect all the netflows transmitted during that period. This is unfeasible in a realistic scenario as it would introduce excessive detection delays [19].
Complete DDoS History
Let us suppose that we build a GNN-based NIDS trained to detect DDoS attacks. These attacks typically involve multiple attacking hosts sending trafic to a single victim node , resulting in a graph structure as shown in Figure 1. Here, the attacker’s nodes overflow the victim node 1 with multiple netflows , forming the graph on the right. Without loss of generality, we suppose that the nodes send their trafic in order at times . If considered singularly, the netflows of a DDoS attack can show characteristics resembling normal trafic, and can be misclassified by detectors analyzing them independently (e.g., standard ML-based NIDS) [ 20]. The strength of applying a GNN for network intrusion detection lies in the fact that these models do not rely exclusively on netflow features (even if they still play an important role) and can easily learn to identify such a peculiar structure shown by the DDoS graph, raising alarms for the involved flows. However, the standard evaluation scenario considered by previous works uses the entire history of events to build the testing graph employed to assess the performance of the GNN.
Let us now consider Figure 2. Here, we suppose that the construction of the testing graph occurs periodically, considering only the netflows coming in a time-window of length . This reflects a more realistic evaluation scenario in which it is unknown when it is the right time to build the evaluation graph and perform the prediction. For instance, if we consider the case of Figure 2, where the test set capture starts at time 0, the first evaluation graph will comprise the netflows received between 0 and = 0 + < 2, when the second netflow of the DDoS attack is transmitted. The result would not contain the complete DDoS attack scenario and could complicate the work of the GNN as it has been trained on standard DDoS attacks. Similar conclusions can be drawn for the other time periods shown in the figure.
We can conclude that previous works employing complete graphs limit their evaluation to the best-case settings, which are biased by the presence of the whole history of the attack. To overcome this pitfall, we propose a more reliable evaluation scenario that can be used to assess GNN-based NIDS that need to be deployed in practice. We avoid a random split between training and testing sets to let the model learn the actual graph as reported in the dataset capture. Then, to reflect a more practical deployment scenario, we temporally divide the test set into diferent time-constant periods of length . Hence, we build one evaluation graph for each of the obtained subsets using only the netflows for the corresponding time-period, as shown in Figure 2. In this way, we simulate a GNN-based NIDS that is invoked periodically at a constant time-rate of , and is provided with a more realistic evaluation graph that reflects the practical
Complete Test-Set History t 0
experimental scenarios.
We now present a case study in which we apply our evaluation methodology to assess the
performance of E-GraphSAGE [
For the scope of this paper, it is worth noting that the experimental campaign presented by the authors of E-GraphSAGE is subject to the pitfalls described in the previous section: the train and test sets are randomly separated, and it is only evaluated by considering the complete test graph. Our case study extends the experiments performed in the original work to assess the capabilities of E-GraphSAGE in more realistic evaluation scenarios. We base our tests on the implementation of E-GraphSAGE made available by the same authors in [24]. However, we avoid the standardization phase of the IP addresses and Port numbers intended by the authors, as it would completely disrupt the network topology [13]. Moreover, unlike the provided multi-class version, as the best practices for network intrusion detection suggest [25, 26], we build multiple binary detectors that are specifically tailored to detect each attack in our datasets.
Our experimental campaign is based on two widely used datasets for network intrusion detection: CTU-13 [17] and ToN-IoT [18].
CTU-13 includes thirteen real-world network trafic captures that mix normal trafic with trafic from diferent botnet variants. The data is already labeled in a netflow format, which is suitable for our study. The botnets in the dataset exhibit unique structures, such as DDoS and Port Scans, with each capture lasting from a few hours to entire days, making it a good benchmark for GNN-based NIDS. We preprocess the CTU-13 dataset by removing non-TCP trafic and outliers, as well as explicit IP addresses and port numbers to avoid learning shortcut patterns [16]. Instead, we use categorical features, such as "IP Address Type" and "Port Type," which indicate the network type of the original IP address and the IANA group of the original port number. The final set of features includes duration, source/destination/total packets, source/destination/total bytes, bytes/s, packets/s, ratio of src/dst bytes, protocol, port type, IP address type, flow direction, source/destination state, and type of service. The malicious samples from each botnet variant are separated into dedicated collections, and those with insuficient samples or too short of a capture time are excluded from the experiments as they would prevent a thorough analysis. The final set of botnet variants considered in our experiments is reported in the first column of Table 1a.
ToN-IoT [18] represents another valid benchmark, considered by the same authors of
EGraphSAGE in the original paper [
In this section, we describe the implementation of our evaluation methodology outlined in Section 3 and used in our case study. As mentioned before, we create a separate binary EGraphSAGE model for each attack in our datasets. To do this, we extract a train-test set pair for each attack. However, unlike previous works, we do not randomly divide the samples into training and test sets.
For the CTU-13 dataset, we divide the malicious samples obtained after the preprocessing phase into groups based on their capturing date. We then attach the benign flows belonging to the same time-period to each group. We select the group from the second day of each capture to form the test set, while the remaining groups are combined to form the training set. To mimic the typical data imbalance in real-world environments, we enforce a 20 : 1 benign:malicious ratio in both the training and test sets [16]. The details are shown in Table 1a.
For the ToN-IoT dataset, each malicious capture is divided into two groups, namely training and testing groups, based on the capture date, as most attacks last exactly two days. For attacks that last less than a day (Ransomware, Injection, and XSS), the training group includes the first 75% of the samples, and the last 25% forms the testing group. We then merge the benign flows into the groups. However, the benign samples in this dataset are far fewer than the malicious ones and are spread across several days. Thus, we form the final training set by merging the training group with the benign samples from the day with the highest number of benign flows (175 300 benign flows). Similarly, for the test set, we select the benign samples from the day with the second highest benign trafic ( 65 280 flows). In both sets, we maintain the 20 : 1 benign:malicious ratio, as for CTU-13. In this way, we maximize the number of malicious samples in both training and test sets, while simulating real-world captures for the evaluation of our classifiers. The details are shown in Table 1a.
We then train the E-GraphSAGE instances on each attack. For this phase, we use the entire
training sets obtained before, as we realistically assume that they involve trafic samples that
have already been captured and labeled in the considered network environment. We repeat the
learning phase for 200 epochs for all the attacks, as done in the original E-GraphSAGE [
In this Section, we present our experimental campaign. We evaluate the E-GraphSAGE classifiers in two scenarios. The first scenario, which we refer to as the baseline evaluation, follows the procedure used in related literature and discussed in Section 3 (see Figure 1). Here, we construct a complete graph representation from each entire test set and proceed evaluating each classifier on it. This approach allows us to validate the E-GraphSAGE algorithm in the common evaluation setting. On the other hand, the second scenario, which we refer to as the Reduced Time Window scenario, considers the proposed novel evaluation methodology. Here, we divide the test sets into shorter time periods of varying lengths, as described in the previous section. For each time period length , we extract 100 diferent graphs by varying the starting time . We then average the results across the 100 graphs for each .
We evaluate the classifiers considering the standard metrics used in network intrusion detection, namely F1-Score, Recall (Detection Rate), and Precision. Considering an attack sample as positive, the three metrics can be computed as follows: =
+ =
+
* 1 − = 2 * + (1) in which TP, FP, TN and FN refer to the True Positive, False Positive, True Negative and False Negative predictions returned by the classifier. These metrics range between 0 and 1, with higher values indicating better performance.
We now present the results of the E-GraphSAGE classifiers’ performance in the baseline scenario
for the CTU-13 and ToN-IoT datasets in Tables 2a and 2b, respectively. All metrics show high
scores, indicating the quality of the detectors. In the CTU-13 dataset (Table 2a), all classifiers
achieve F1-scores above 0.95 for all botnets, with an average of 0.972. The precision scores are
almost perfect, with an average of 0.991, evidencing a low number of false positives, while the
recall lies at 0.955, which is a solid indicator of good detection rates. The best performance is
provided by the Murlo classifier, with an average F1-Score of 0.997. Similarly, for the ToN-IoT
dataset, the classifiers achieved an average F1-score of 0.956, which is consistent with the
state-of-the-art results [
Overall, these results demonstrate the eficacy of the E-GraphSAGE classifiers in the standard evaluation scenario where the complete graph is considered during the testing phase. However, as explained in Section 3, this approach may overestimate performance in real-world settings where complete graphs are not available.
We now consider the Reduced Time Window scenario in which we apply the proposed evaluation methodology. The results for the two datasets are provided in Table 3 and 4, respectively. For each attack, we show three average scores in the first three subrows, computed based on 100 evaluation graphs with diferent values indicated in the columns. The cells in each table are colored from red to white, with darker red indicating a larger loss compared to the baseline value. Additionally, we present the average number of malicious and benign samples in the evaluation graphs for each in the last subrow.
After examining Tables 3 and 4, we observe three distinct performance patterns. For several classifiers, we notice a correlation between lower performance scores and shorter values. As discussed in Section 3, this intuitive trend arises as a result of the GNN struggling to detect topological patterns of malicious communication in evaluation graphs created from shorter values. For the CTU-13 dataset (Table 3), this tendency is particularly evident for the Neris detector, in which we register a significant drop of over 30% in the F1-score when is set to 1 second (from 0.985 to 0.671), and a monotone raise when grows. Another example is the Murlo classifier, which maintains low scores for values below 10 minutes. Other meaningful cases for the ToN-IoT dataset (Table 4) are represented by the classifiers for the DDoS, Ransomware, Injection and XSS attacks which show steep declines in F1-scores going from 0.942, 0.844, 0.958 and 0.986 to 0.683, 0.593, 0.500 and 0.844, respectively, when is set to 1 second.
We also observe a subset of classifiers that exhibit minimal sensitivity to the reduction of time intervals used to construct the evaluation graphs. For instance, the Virut botnet detector of the CTU-13 dataset passes from an F1-score of 0.954 to 0.943 when is set to 1 second (loss of just 1%), and it converges to the baseline within the 10 seconds time-window. A similar trend is ofered by the Password-Cracking classifier (F1-score from for the DoS and Backdoor classifiers are not even impacted. This unexpected behavior might imply that the E-GraphSAGE detectors for these attack categories may not rely on malicious topological patterns for detection, but rather on the netflow features. These results also suggest that, for certain attacks, it might be convenient to employ more traditional ML techniques that ofer comparable performance, while being far less complex and better explainable.
While most of the classifiers fall into the first two categories, there are two exceptions worth noting. The classifier for Rbot in the CTU-13 dataset rapidly recovers to scores close to the baseline already from the smallest time-windows. However, it shows a significant performance decrease when considering evaluation graphs built on captures lasting 30 minutes and 1 hour (the baseline F1-score of 0.954 drops to 0.794 and 0.721, respectively). Similarly, the Scan detector for the ToN-IoT dataset resembles the behavior of the second group appearing robust even when is set to 1 second, but it exhibits a severe Precision drop at the 3 hours interval (from 0.944 to 0.777). Although these classifiers perform well for small evaluation graphs, they represent meaningful cases in which the topological features can weaken detection performance for large evaluation graphs.
In summary, these results endorse our novel evaluation methodology, confirming that previous approaches using complete test graphs may lead to upper-bound performance and an overestimated sense of security. Furthermore, they assume great significance for the practical deployment of these tools in real-world settings, ofering valuable insights into the optimal time-window length combinations for achieving the highest detection rates. Finally, they also suggest that for certain cases, employing conventional ML algorithms that do not take into account topological structures may be more appropriate for network intrusion detection. Performance of E-GraphSAGE on the botnets of the CTU dataset. delayed detection. Second, it suggests that in some cases, the detectors ignore the topological patterns for classification, and their inclusion might even result in decreased performance. Therefore, in such scenarios, traditional ML algorithms relying exclusively on netflow features may be a more suitable and less expensive option. Our results show which attack categories can actually benefit from a GNN-based approach and analyze the trade-of between detection delay and detection performance. Future work will extend our case study to other state-of-the-art GNN-based NIDS proposed in the literature and aim to validate our findings using explainability methods and tools.
This work was partially supported by project SERICS (PE00000014) under the MUR National Recovery and Resilience Plan funded by the European Union - NextGenerationEU. adversarial learning against botnet evasion attacks, IEEE Transactions on Network and Service Management 17 (2020) 1975–1987. [11] W. Jiang, Graph-based deep learning for communication networks: A survey, Computer
Communications (2021). [12] D. Pujol-Perich, J. Suarez-Varela, A. Cabellos-Aparicio, P. Barlet-Ros, Unveiling the potential of graph neural networks for robust intrusion detection, ACM SIGMETRICS Performance Evaluation Review 49 (2022) 111–117. [13] H. Zhu, J. Lu, Graph-based intrusion detection system using general behavior learning, in: GLOBECOM 2022-2022 IEEE Global Communications Conference, IEEE, 2022, pp. 2621–2626. [14] J. Zhou, Z. Xu, A. M. Rush, M. Yu, Automating botnet detection with graph neural networks, arXiv preprint arXiv:2003.06344 (2020). [15] Q. Xiao, J. Liu, Q. Wang, Z. Jiang, X. Wang, Y. Yao, Towards network anomaly detection using graph embedding, in: Computational Science–ICCS 2020: 20th International Conference, Amsterdam, The Netherlands, June 3–5, 2020, Proceedings, Part IV 20, Springer, 2020, pp. 156–169. [16] D. Arp, E. Quiring, F. Pendlebury, A. Warnecke, F. Pierazzi, C. Wressnegger, L. Cavallaro, K. Rieck, Dos and don’ts of machine learning in computer security, in: 31st USENIX Security Symposium (USENIX Security 22), 2022, pp. 3971–3988. [17] S. Garcia, M. Grill, J. Stiborek, A. Zunino, An empirical comparison of botnet detection methods, Computers & Security 45 (2014) 100–123. [18] A. Alsaedi, N. Moustafa, Z. Tari, A. Mahmood, A. Anwar, Ton_iot telemetry dataset: A new generation dataset of iot and iiot for data-driven intrusion detection systems, IEEE Access 8 (2020) 165130–165150. [19] A. Venturi, C. Zanasi, On the feasibility of adversarial machine learning in malware and network intrusion detection, in: 2021 IEEE 20th International Symposium on Network Computing and Applications (NCA), IEEE, 2021, pp. 1–8. [20] F. De Gaspari, D. Hitaj, G. Pagnotta, L. De Carli, L. V. Mancini, The naked sun: Malicious cooperation between benign-looking processes, in: Applied Cryptography and Network Security: 18th International Conference, ACNS 2020, Rome, Italy, October 19–22, 2020, Proceedings, Part II 18, Springer, 2020, pp. 254–274. [21] W. Hamilton, Z. Ying, J. Leskovec, Inductive representation learning on large graphs,
Advances in neural information processing systems 30 (2017). [22] J. Lan, J. Z. Lu, G. G. Wan, Y. Y. Wang, C. Y. Huang, S. B. Zhang, Y. Y. Huang, J. N.
Ma, E-minbatch graphsage: An industrial internet attack detection model, Security and Communication Networks 2022 (2022). [23] E. Caville, W. W. Lo, S. Layeghy, M. Portmann, Anomal-e: A self-supervised network intrusion detection system based on graph neural networks, arXiv preprint arXiv:2207.06819 (2022). [24] waimorris, E-graphsage, https://github.com/waimorris/E-GraphSAGE, 2022. [25] D. Wu, B. Fang, J. Wang, Q. Liu, X. Cui, Evading machine learning botnet detection models via deep reinforcement learning, in: ICC 2019-2019 IEEE International Conference on Communications (ICC), IEEE, 2019, pp. 1–6. [26] B. Biggio, I. Corona, Z.-M. He, P. P. Chan, G. Giacinto, D. S. Yeung, F. Roli, One-and-a-halfclass multiple classifier systems for secure learning against evasion attacks at test time, in: Multiple Classifier Systems: 12th International Workshop, MCS 2015, Günzburg, Germany, June 29-July 1, 2015, Proceedings 12, Springer, 2015, pp. 168–180.