With a constantly increasing amount of encrypted network trafic and a new type of attack (“zero-day”), network trafic anomaly detection shows significant benefits over traditionally used signature-based packet inspection methods for cybersecurity attack detection. Using NetFlow or similar protocols is an attractive approach to providing accounting information about network communications due to its simplicity and applicability in a real-life network environment. Even though the basic set of information in flow data is not suficient for eficient machine learning techniques, they are quite suitable for the application of entropy-based anomaly detection techniques. In this paper, we present comprehensive work in research, development and implementation of network trafic anomaly detection solutions based on the entropy of flow data. Starting from the well-known entropy-based approach, we reveal the results of our methodic work in solving the main challenges in designing an eficient anomaly detection solution empowered with the original classification method. Since the proof of concept was achieved in the laboratory environment using ofline datasets, the solution has been implemented relying on the existing NetFlow Analyzer software product NetVizura. Even at the minimum viable product stage, the application confirms high performances and great applicability of the anomaly detection and classification method in real-life network environments.
ous financial threats (ransomware), service provisioning issues, etc.
The modern enterprise network environments face the The European Union Agency for Cybersecurity, ENISA
necessity to respond adequately to the increased multi- annually publishes reports on the cybersecurity
landheterogeneity needs, reflected through a range of difer- scape situation, and for 2022 the prime identified
inent user devices, a variety of existing applications, ser- trusions correspond to ransomware, social engineering
vices, and various data in diferent formats and through- threats, malware, intrusions targeting data and user
priputs, that need solid storage and real-time processing vacy, attacks to the data and service availability (Denial
available. The rise of the global need for bringing ofice of Service and Internet threats), disinformation attacks,
functionality to remote working personnel has addition- encompassing the misinformation and the supply-chain
ally contributed to the need for the development and im- attacks [
The current cyber threat landscape indicates several related to the HTTPS Flood intruders, with 16% of overinterrelated activities starting from the intruder attempts all malicious activity. It is also noticed, that TCP ACK to infiltrate, scanning, and collect data and infrastruc- attacks had a relatively high part in these activities, with ture vulnerability information, intending to proceed with 6.5% of performed attacks. The rest corresponds to the more severe attack routines. It relates to further data/in- TCP SYN, IP Fragmentation, UDP, DNS, L2TP, and SNMP frastructure compromise and negative repercussions to amplification attacks. If analyzing attacks by category, operational abilities, productivity, privacy abuse, seri- the volumetric Direct Flood attacks contributed to 68% of the total attacks, while there was a noticeable increase BISEC’22: 13th International Conference on Business Information of 330% of the application attacks activity, forming 17.5% *SeCcourrrietys,pDonecdeimngbearu0th3,o2r.022, Belgrade, Serbia of all the 2022 cyberattack landscape. The volumetric $ slavko.gajin@rcub.bg.ac.rs (S. Gajin) amplification attacks were also present, while in lower © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License amounts than the other two categories.
CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org)
As reported by Truesec, the total number of cyber at- in this field. The third section presents the main solution,
tacks increased by 160%, with the highest percentage for while section four discusses other important details and
ransom (34%), access harvesting (32%), resource hijack- solutions in the proposed methodology, demonstrating
ing (12%), business email compromise (8%), data theft the experimental results. Finally, the paper is concluded
(7%). 40% of the attacks in the analyzed period originated by summarizing the main achievements and results, and
from publicly accessible and vulnerable systems, with discussing further development.
a noticeable rise of zero-day exploits and ransomware
attacks [
This dynamic emergence of new-generation attacks, 2. Related work and the newly identified vulnerability targets, made perfect conditions for the increase in zero-day attacks, and Being one of the most significant research fields, cyberdiferent forms of amplification and application attacks, security and privacy provisioning have raised high stanwith persistent use of cryptography for the attack trafic dards for the accepted and applied techniques and apgeneration. As a response, a large number of research proaches. Available literature provides access to signifigroups and individuals are developing advanced and orig- cant research work focused on anomaly and cybersecuinal solutions for accurate, eficient, and real-time intru- rity attack detection, the so-called Intrusion Detection sion detection. As there is no exclusive, all-encompassing Systems (IDS). These papers are mostly published in the solution for every incident case, network environment, area of machine learning algorithms and are oriented or user need, the goal is to find specific prevention and towards the techniques for extracting the network trafic detection methods that would fulfil the general range of pattern’s behaviour. modern network security requirements. The most common solutions are usually based on spe
In anomaly detection, a considerable part of the re- cific custom-made algorithms and procedures for
inspectsearch community is now directed toward the analysis ing network trafic and further detection and
identificaof trafic behaviour and its characteristics, both struc- tion of network anomalies [
In this paper, we present the work resulted from the In IDS, the entropy-based approach is easily
impleproject “Network Trafic Anomaly Detection system mented by relying on flow-based data and it has found a
based on NetFlow data analysis – TRADE”, under the large portion of usability in situations when anomalous
European EUREKA programme. The main project goal trafic appears with high intensity and volumes. In that
was to develop a competitive market-ready cybersecurity case, it appears with weighty spikes in data distribution
threat detection solution. To achieve this goal, initial and does not leave these intrusions hidden [
Despite some statements that flow-based analysis re
The rest of the paper is organized as follows: the sec- lies on insuficiently detailed information, asserting the ond section highlights the most relevant scientific work fact that these are only the basic information of network ets per second and flows per second metrics, given in communications collected and exported by routers, the a break-down structure of top contributors, such as the lfow-based detection approach has gained strong enough most intensive hosts, protocols, services etc. Additionally, approval among the research communities [13, 14]. NetFlow Analyzer keeps the logs in the internal database
The main issue with the application of entropy-based allowing manual drill-down analysis and data forensics. technics is some intrinsic deficiencies, being the most Therefore, network trafic anomalies can be eficiently stumbling the fact that these techniques are eligible investigated, but still hardly detected, with an exception only when network trafic structure significantly changes of very intensive volumetric anomalies, such as DDoS through the attack [15, 16]. Another study analyzes the attacks. use of the real-time trafic behaviour profiling solution developed for the needs of anomalies and attack detec- 3.2. Entropy-based approach tion [17]. With a goal to detect particular behaviour patterns, the study is based on the analysis of trafic data, To detect anomalies in network trafic the collected flow previously preprocessed with a set of entropy-based and data must be first processed and transformed into a simdata-mining techniques. This research relies on trafic pler form and metrics that provide a possibility to proaggregation and evaluates two additional attributes, the file normal behaviour. Then, significant changes to norpacket size and packet rate attributes. mal behaviour are treated as an anomaly. Two main
The goal of our research was to propose a solution that approaches to completing this task relate to machine makes the most of entropy-based techniques advantages learning and entropy calculation techniques. and flow data collected in real-time while providing a Supervised machine learning is a very popular aphigh level of detection accuracy and eficiency, demon- proach in the research literature, but the fact that it relies strated through practical implementation. on labelled data gives very limited options for practical implementation in a real-time environment. Unsupervised machine learning for anomaly detection, mostly 3. Proposed solution based on clustering methods, is more feasible for practical usage, but still very complex and demanding in data pro3.1. Flow-based approach cessing. For these reasons, the entropy-based approach The access to network trafic provides a reach set of in- attracts the interest of both the research community and formation for the analysis and anomaly detection, which industry. In the context of anomaly detection techniques includes both packet header and payload. However, this entropy is a single value that can be interpreted as the approach is not scalable and flexible enough, since it measure of the evenness of a data distribution. A signifirequires direct access to network resources. Also, it gen- cant change in a data distribution causes a change in the erates a huge amount of data, which requires high pro- entropy value, which is considered anomalous behaviour cessing power for real-time analysis. of network trafic and the indication of security threats.
A more flexible approach is achieved by the Cisco Net- Data distribution is obtained through the aggregation Flow protocol, where routers keep track of each network process using a selected attribute as an aggregation key communication, counting the transferred bytes and pack- and counting or summarizing other attributes from raw ets, and exporting this information to an external server. flow data. For instance, summarizing the total bytes or A large number of the collected data is treated as real- packets received by each destination IP address during a time logs about the communication activities, identified short period, the so-called epoch, will produce a correby the source and destination IP and port numbers, as sponding data distribution. Typically, some addresses are well as the protocol used. Despite a lack of packet pay- more active than others, resulting in uneven data distribuload, these logs provide an excellent source of data for tion and certain entropy values. In case of a DDoS attack, the analysis of network communication. The success of a targeting destination host receives a huge amount of this approach was confirmed by a great acceptance by trafic which cause a high peak in the data distribution users and adoption on the market. Many other vendors and a significant change of the entropy value. proposed similar protocols, such as Jflow by Juniper Net- The flow data is identified by a flow tuple, which conworks, Cflow by Alcatel-Lucent, NetStream by Huawei, sists of the source and destination IP addresses, protocol Rflow by Ericsson, while IEEE has standardized IPFIX type and source and destination port numbers. These protocol [18]. identification attributes are good candidates for the ag
So-called NetFlow Analyzer software collects the data, gregation keys and we will shortly label them as , , processes it and provides statistics about the trafic struc- , and respectively. ture at the interfaces level, router level or in the whole Since network communications between two pears network. Statistics are produced in bits per second, pack- mostly conduct in both directions, it is useful to pair two corresponding originally unidirectional flow data into a = (1 − )ˆ− 1 + − 1.
ˆ
relatively from the baselined entropy value ˆ, using a multiplication factor , the so-called threshold factor, that makes the range wider:
T = ˆ − ˆ ¯ = ˆ + ˆ
= [︀ T, ¯ ︀]
The entropy values that fall into the margin are
(1) considered regular, while the entropy value out of the
margin triggers an alarm as an indication of an anomaly:
(4)
(5)
(6)
(7)
(8)
single bidirectional flow record [
and port number are associated with the initiator of the a weighting factor, to adjust the influences of the input conversation, while the destinate address and port num- values and make the baseline value more or less smooth. ber relate to the responding side. Consequently, the byte and packet counts relate to the sending side, either source
The next step is to predict and baseline the standard deviation () of the entropy values, also using the EMA or destination. These volumetrics attributes are labelled approach: with , , and respectively. In the above mentioned example, the total bytes sent to a destination host is labelled as []. This metric, the so-called feature, relates to the data distribution and the corresponding entropy value for a certain epoch. During a time, epoch by epoch, the process generates a time data series of entropy values for each feature.
The Shannon entropy [19] is commonly used, while some other authors propose the usage of Tsallis [20] and Rényi [21] parametrized entropy types. For a given feature data distribution with the total number of elements, where mi relates to the value of the element and () is the empirical probability, the Shannon entropy is defined by the following equation: () = ∑︁ =1 () log ()
1 where the empirical probability is given by: () = , = ∑︁ =1 .
(2)
Shannon entropy reaches the maximum value of log , while more unevenness leads to lower, but always positive values. To always get the values between 0 and 1 it is useful to use the scaling factor of 1/log .
The next challenge is to detect the changes in the entropy time-series values for any of the used features. To do so, the trend of the time series entropy data must be predicted based on the recent values, which is also known as the baseline values. Then, the acceptable variation in these values is calculated and set around the baseline and used as thresholds. A simple approach is the windowing mechanism setting the lower and upper thresholds by calculating the minimum and maximum values for the last epochs. To make this accepted margin more tolerant, these thresholds are further increased by some scaling factor, usually by the value of 3.
thresholds to recent variations of the observed values, a more advanced and flexible approach is based on the
, labelled as ˆ, is predicted recursively, considering both entropy and baselined values in the previous epoch: ˆ = (1 − ℎ)ˆ− 1 + ℎ− 1. (3)
The thresholds define a margin for acceptable variations: = ︂{ , , ∈ ∈/ 4.
in the fact that a complex data structure can be transformed into a diferent domain of time series data for the observed features that can be easier analyzed to detect unusual behaviour.
To be eficient for practical usage in real time environment anomaly detection solution requires proper feature selection (the aggregation keys and the calculating attributes) and solving other challenges related to the specific domain of network behaviour analysis based on collected flow data. In this section, we will demonstrate and discuss these challenges and propose proper solutions to them.
window techniques, takes into account previous values.
tropy values, the unusually low entropy values gradually lead to lowering the baselined values too and widening the threshold margin. As a result, an anomaly could be eventually treated as normal behaviour. Consequently,
Many authors in the scientific literature for DDoS attack detection propose entropy-based approaches that rely on the volumetrics attributes only, such as total bytes and packets number [23, 24, 25, 26]. However, our research has shown very limited usability of those approaches, mostly because regular trafic in today’s network usage often assumes large data transfer, such as backup, data download or torrent communications. These occasional trafic loads cause a large variation in the corresponding entropy values and consequently a large threshold to detect by inspecting top contributors in the data distribution, which is already achieved by NetFlow Analyzer software.
For that reason, we propose using additional features that represent the communication structure rather than transferred data volume in bytes and packets. In addition to the flow count attribute for a certain aggregation key, with the second-level aggregation, we propose counting the appearances of all distinct elements of other identification attributes which are not used in the aggregation key.
ports that some source IP address communicates with, (10) labelled as [], indicates a port scanning attack. Since these features represent the behaviour of network communications, we will call them behaviour features.
At the beginning of the baselining calculation, there is no history and previous data items to be used for the prediction. The entropy prediction in the next epoch can get the value of entropy in the current epoch, but more importantly, it is needed to properly estimate the standard deviation. A large initial standard deviation will cause a slow convergence process until it gets stabilized, which could keep some anomalies undetected (Figure 4).
margin. Regular entropy values can easily fall out of the margin triggering a false positive alarm. More importantly, since the baselined values stay unchanged during alarms, this alarm could be locked for a longer period, which is shown in Figure 5.
The solution to this problem, which happens in practi- Figure 7: False positive alarms elimination by anomaly score. cal implementation when the application is restarted, is to persist the last value of standard deviation calculated before the application restarted and reuse it as an initial value.
tance of the normalized entropy value from the thresholds (0 and 1) in epoch , denoted as ∆ , as follows:
z-score, is a commonly used approach to scale entropy
values into a specific confidence interval to remove the
bias and make data comparable [
the anomaly score as a cumulative sum of the metric ∆ in a sliding window during the last ′ epochs or ˜ = ¯ −− TT (11) isnmfluaelnlecre, osifntchee tphreevliaosutsalylatrrmiggdeeraecdtiavlaatrimon. Htoowaveoviedr,tihfe the alarm is not active, a smaller sliding window is taken
After normalization, the range of acceptable values is into account, ′′, depending on how fast we would like
scaled into the interval [
from the more severe ones, illustrated in Figure 6, we define the anomaly score as a measure of deviation from the regular behaviour for each feature separately, taking into account the time period when the alarm is active. Obviously, higher deviation and longer alarm activation periods indicate greater anomalies. The rationale behind this is based on the fact that short spikes in entropy values could present acceptable data variations, due to which they are less important than deviations that last longer, even if they are less intensive. = {︃ ∑︀ ′
=1 ∆ − +1 ∑︀ ′′ =1 ∆ − +1 ∆ < 0 ∆ = 0
Figure 7 demonstrates the benefits of anomaly score usage in eliminating false positive alarms.
Detection performances of entropy-based approaches highly depend on the relative amount of anomalous activities in comparison to regular network behaviour for the observed feature. If the network is heavily loaded with regular trafic, the straightforward detection is limited only to highly intensive anomalies, while less aggressive malicious activities may remain undetected.
To address the above-mentioned issue, we propose the partitioning of network trafic into smaller subgroups and by removing all flows associated with attacks and other background anomalies. The modelled anomalies were gradually increased to check the sensitivities of the features.
Figure 8 shows the entropy drops caused by the anomalies following the “N1-1N” communication pattern which are detected with the flow count feature aggregated by the source port (f[s]).
The results obtained from the thorough analysis of apply a detection technique to each subgroup separately. which features are triggered by which anomaly type are This partitioning can be based on diferent criteria, such summarized in Table 2. Even a brief overview of the table as the protocol type (TCP, UDP, ICMP), service type (DNS, reveals that entropies behave diferently for diferent email, web service, windows services etc.) or sub net- communication patterns, while some of them are not working (user trafic, voice VLANs, data centre, branch afected by a particular anomaly at all (the empty cells in ofices etc.). the table). More importantly, the ways how the entropies are afected by the modelled anomalies follow a very
It can be observed that the entropy drop (labelled ‘X’) Relying on the behaviour features rather than the vol- occurs only when all features in the aggregation key have umetric features provides better capturing of diferent a single occurrence in the model (marked with ‘1’). For communication patterns. Considering the flow identifica- instance, aggregation by the source IP address causes an tion attributes and their cardinality in the communication entropy drop only in the first 8 models, since a single (one or many occurrences), we developed 16 communi- host as a source of the anomaly greatly contributes to cation patterns. For example, the flows that present a the calculated distributions. port scan attack use a single source and destination IP In addition to anomaly detection, analyzing the trigaddresses and many destination port numbers. Labelling gered features can provide valuable information about with the cardinality one (“1”) or many (“N”), in order: the communication pattern of the anomaly, indicating source IP address, source port, destination IP address and the type of potential attack. destination port, namely (“Ss-Dd”), we can describe this port scan attack with the “11-1N” pattern. Consequently, if the source port number is randomly chosen with many 5. Implementation occurrences (“N”), the communication pattern is “1N-1N”.
Using this labelling convention, all 16 communication When the research contributions were proven in the ofpatterns, with the associated attacks are given in Table 1. fline laboratory environment using the commonly used datasets, namely CTU-13 [27] and CIC-IDS2017 [28], the next step was to implement the concept for usage in real4.8. Anomaly modelling and classification life network communications. This section presents a developed architecture and the implementation details.
To analyze how diferent features are afected by different communication patterns, we have modelled the anomalies by injecting synthetically generated flows into 5.1. Architecture a dataset of flows that correspond to normal trafic with no anomalies. We have obtained regular flow data from A high-level architecture of the proposed methodology, the public flow-based dataset CTU-13 [ 27], the trace “51“, illustrated in Figure 9, consists of the following main building blocks:
The proposed architecture is implemented as a new module within the NetVizura solution, working in synergy with the NetVizura NetFlow Analyzer module [29]. Both modules can work on the same server as a monolith application, while in the case of higher trafic load they can be deployed on separate nodes, but still sharing the common user interface on the primary node. The backend application is developed in Java programming language, while the frontend user interface is developed in Javascript React framework combined with the legacy Google Web Toolkit elements.
Elasticsearch is used as a database for the entropy timeseries data, top contributors in the data distributions used to calculate entropies, as well as raw flow data. As a non-SQL database, it is highly optimised and eficient for real-time data inserting at a high rate, as well as fast data retrieving. • Flow Preprocessing – Two unidirectional flows A typical use case is when the user notices a single from both directions between two peers are alarm containing the most relevant information, such paired into a single record, the so-called bidirec- as the anomaly class indicating the attack type. In a tional flow, which gives more information and separate tab, shown in Figure 10, a user can inspect the ensures greater detection eficiency. entropy values in the recent period for a selected feature, • Flow partitioning – Bidirectional flows are fil- while the individual elements that mostly contribute to tered by protocols, services or IP addresses, and the entropy changes are shown in a separate time chart. divided into diferent sub-gropus, which are ana- Click on any of these elements opens a new tab which lyzed separately. shows raw data associated with the selected element. • Aggregation – Flow data during an epoch are Users can further filter and aggregate the flow attributes aggregated based on the identification features, and the resulting data are visualized at the throughput calculating additional behaviour features. The or volume chart (Figure 11). results are data distributions for each aggregation The current version of the anomaly detection module key and feature used. is demonstrated in an operational environment (TRL-7 de• Entropy calculation – Entropy is calculated velopment level) achieving performances of 15 K flows/s over each data distribution in each epoch, gener- on a server with 16 CPU cores and 32 GB RAM and using ating time series entropy values for every feature. the Elasticsearch database on a separate server. Current • Entropy change detection – A significant development is focused on memory optimisation which change in the entropy value indicates a change appears to be the major bottleneck for a higher flow proin network communication behaviour. The chal- cessing rate, expecting to achieve up to 50 K flows/s on a lenge is to accurately recognize changes resulting single server. in anomalies and distinguish them from normal
The work presented in this paper has partially been funded by the Ministry of Education, Science and Technological Development of the Republic of Serbia under the EUREKA project “Network Trafic Anomaly Detection system based on NetFlow data analysis – TRADE”, grant number E!13304 and the contract number 451-0368/2022-14/200103.
for entropy-based network trafic anomaly detection and classification that relies on flow data. The method brings several novelties and improvements important for practical implementation.
Firstly, we developed a generalized concept of behaviour features based on the aggregation of the second degree, which better reflects the structure of network communications using only basic flow attributes. Based on these features, we have modelled 16 trafic patterns associated with anomalies that follow a wide range of security attacks. Secondly, we improved the entropy change detection process and reduced many false positive alarms by introducing the anomaly score metrics.
Thirdly, the comprehensive experiments have shown that diferent anomalies trigger alarms on diferent features, which can be interpreted as a characteristic signature that can be additionally used for anomaly classiifcation. And finally, the applicability of the method is demonstrated through the implementation and real-time usage in real-life network environments.
The current work is oriented to performance optimization and improved usability, while further research is focused to the consolidation of multiple alarms using unsupervised machine learning. DDoS Attacks in Cloud, International Journal of international wireless communications and mobile Computer Applications 62 (2013) 42–47. computing conference, 2010, pp. 479–483. [12] S.-E. Benkabou, K. Benabdeslem, B. Canitia, Unsu- [27] S. Garcia, M. Grill, J. Stiborek, A. Zunino, An empervised outlier detection for time series by entropy pirical comparison of botnet detection methods, and dynamic time warping, Knowledge and Infor- computers & security 45 (2014) 100–123. mation Systems 54 (2018) 463–486. [28] I. Sharafaldin, A. H. Lashkari, A. A. Ghorbani, To[13] A. Sperotto, G. Schafrath, R. Sadre, C. Morariu, ward generating a new intrusion detection dataset A. Pras, B. Stiller, An overview of ip flow-based and intrusion trafic characterization., ICISSp 1 intrusion detection, IEEE communications surveys (2018) 108–116.
& tutorials 12 (2010) 343–356. [29] NetVizura, Network monitoring solutions, 2022. [14] B. Li, J. Springer, G. Bebis, M. H. Gunes, A survey URL: https://www.netvizura.com/. of network flow applications, Journal of Network and Computer Applications 36 (2013) 567–581. [15] İ. Özçelik, R. R. Brooks, Deceiving entropy based dos detection, Computers & Security 48 (2015) 234– 245. [16] A. Lakhina, M. Crovella, C. Diot, Mining anomalies using trafic feature distributions, ACM SIGCOMM computer communication review 35 (2005) 217–228. [17] K. Xu, Z.-L. Zhang, S. Bhattacharyya, Internet trafifc behavior profiling for network security monitoring, IEEE/ACM Transactions On Networking 16 (2008) 1241–1252. [18] B. Claise, P. Trammell, B.and Aitken, Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information, RFC 7011, IETF, 2013. URL: https://www.ietf.org/rfc/rfc7011.
txt. [19] C. E. Shannon, A mathematical theory of communication, The Bell system technical journal 27 (1948) 379–423. [20] C. Tsallis, Possible generalization of boltzmanngibbs statistics, Journal of statistical physics 52 (1988) 479–487. [21] A. Rényi, et al., On measures of entropy and information, in: Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, volume 1, Berkeley, California, USA, 1961, pp.
547–561. [22] A. Lawrance, P. Lewis, An exponential movingaverage sequence and point process (ema1), Journal of Applied Probability 14 (1977) 98–113. [23] A. Lakhina, M. Crovella, C. Diot, Diagnosing network-wide trafic anomalies, ACM SIGCOMM computer communication review 34 (2004) 219–230. [24] P. Bojović, I. Bašičević, S. Ocovaj, M. Popović, A practical approach to detection of distributed denialof-service attacks using a hybrid detection method, Computers & Electrical Engineering 73 (2019) 84– 96. [25] O. Joldzic, Z. Djuric, P. Vuletic, A transparent and scalable anomaly-based dos detection method,
Computer Networks 104 (2016) 27–42. [26] D. Rossi, S. Valenti, Fine-grained trafic classification with netflow data, in: Proceedings of the 6th