The number of users using internet and local networks is increasing every day. As a consequence, there are many threats of trying to have an access to private password, to data or to injure users by other ways. Fortunately, current generation of network devices allows a real-time scraping of structured snapshots of a traffic on the networks. This information is provided by various technologies. Two the mostly used technologies are the NetFlow format introduced by CISCO and the sFlow format. These technologies allow us to observe the individual flows on the network. A flow is an unidirectional component of TCP connection (or UDP/ICMP equivalent), defined as a set of packets with identical source and destination IP addresses, ports and protocol, packed size, MAC addresses, switch ports, flags and more.
An information provided by NetFlow or sFlow can be used to detect a network attack. The most frequent attacks on networks can be divided to three main classes [1]: Breaks privacy rules, compromising the information confidentiality; Alters information, compromising the data integrity; Denial of service attacks (DOS or DDOS attacks), which make a network infrastructure unavailable or unreliable, compromising the availability of a resource.
The protection of networks is, therefore, more than useful, if it is vital for long time. This problem requires the monitoring of real distributed hosts, the various events and exchanges between these hosts. It is necessary to use MAS due to the complexity of this problems.
The aim of this paper is to propose a multi-agent system for network intrusion detection M-AHIDS. The main contribution of the M-AHIDS is the integration of several anomaly detection techniques and machinery of multi-agent temporal logic with hybrid negotiation. Every detection technique is represented by featuring a specific detection autonomous agent and every agent determines the flow trustworthiness from aggregated connection. We took an inspiration for our agents in project CAMNEP [2,3]. All CAMNEP agents are more less separate IDS and the project CAMNEP tries to connect their results to more trustworthy result. But we have decided to use another approach in our IDS. Our agents are as simple as possible. In addition to that, we have a developed new innovative agent -Web agent which is a significant advantage of our system. The Web agent is able to detect a trustworthy host from his activity on the web pages and this is based on our past project [4][5][6] about de-anonymization of an Internet user. This project is still deployed on all web pages of Comenius University and we can detect ordinary users' behaviour from its data.
We have used another new approach for making decisions about intrusion from detection agent's knowledge base. For this propose we have used specifically developed multi-agent temporal logic (MTL). The anomalies are used as an input for machinery of MTL which is represented by a logical agent. The logical agent is one of the system advantages because it has huge capabilities for making a right decision about the intrusions from detected anomalies. MTL allows us to collect knowledge from every detection agent from past to future. All detected intrusions are our past states in MTL and for the future states we will use the prediction methods from past and actual connections collection.
The most important contributions of our research presented in this paper are: Integration of the several anomaly detection techniques in a form of agent; Machinery of the multi-agent temporal logic; Hybrid negotiation with argumentation and immune cell inspiration; New innovative detection agent -Web agent which is able to detect a trustworthy host from his activity on the web pages. M-AHIDS is partially implemented and tested on our Department of Applied Informatics. Obtained results of M-AHIDS are comparable to another IDS.
The organization of the paper is as follows: in section 2 -overview of the existing solutions and approaches which we use; in section 3 -proposal of a detection system architecture; in section 4 -detailed description of all agents in M-AHIDS; in section 5 -overview of case study, tests and results.
Intrusion Detection System or IDS is software, hardware or combination of both used to detect an intruder's activity. The base characteristics of IDS [7] are neutralizing illegal intrusion attempts in real time. For this reason it must be executed constantly in a host or in a network.
There are many IDS. Each of them has some advantage and disadvantage. Their strengths or weaknesses depend mostly on how they recognizes the threats. Two main approaches for detection intrusion are [1]:
Behavior-based intrusion detection approach, which discovers intrusive activity by a comparing a user's or a system's behaviour with a normal behaviour profile; Knowledge-based (signature-based) intrusion detection approach, which detects intrusions upon a comparison between the parameters of the users' session and the known pattern attacks stored in a database.
An advantage of behaviour-based IDS is an ability to detect new form of intrusion, but their disadvantage is a possibility of un-detection of small intrusion or intrusion hidden in normal behaviour. On the another side knowledge-based IDS has an advantage in low false-positive alert for well known intrusion and high success rate for this intrusion. Their disadvantage is a low probability of detection of new intrusion.
One of the best known knowledge-base IDS is Snort [8]. Snort is an open source IDS available to general public. Architecture of Snort is logically divided into multiple components. These components work together to detect particular attacks and to generate output in a required format from the detection system. A Snort-based IDS consists of the following major components: Packet Decoder, Preprocessors, Detection Engine, Logging and Alerting System and Output Modules. Snort uses rules stored in text the files that can be modified by a text editor. Finding signatures and using them in rules is a tricky job, since more rules you use, more processing power is required to process captured data in real time.
There are several behaviour-based IDS. One of the most complex solution is CAMNEP [2,3]. This project is based on trust models of network flows which is built from trustfulness values of individual flows from all agents. CAMNEP uses five type of detection agent. Each of these agent has different methodology of intrusion detection and all these agents are in core separate IDS. Authors of CAMNEP named this agents as: Lakhina Entropy agent [9], Lakhina Volume agent [10], MINDS agent [11], TAPS agent [12] and XU agent [13]. All of these agents use the same NetFlow protocol and all agents have capability to decide if a connection is intrusion or not. These agents are more less separate IDS and project CAMNEP tries to connect their result to more trustworthy result. We have decided to use another approach in our IDS. Our agents are as simple as can be.
One agent covers only one intrusion detection method and every agent separately evaluates every connection. Evaluating of connection means that agent compute score for the connection. Higher score indicates more suspicion behaviour. We have achieved more effective structure with this approach, because we don't have redundant computation. Another positive effect of this approach is that we know exactly how well which agent evaluates every connection.
Different interesting IDS for our research is the Multi-Agents Immune System for Network Intrusions detection (MAISId) [7]. Biological inspiration is very useful for many scientific departments. Inspiration in this case is biological immune cell. Immune cells have membrane receivers, who allow them to recognize specifically an epitope of an antigen [7]. The immune system is mainly founded on three elements: gene database of genes, negative selection and the clonal selection. The gene database makes it possible to generate antibodies. The negative selection makes it possible to remove the inappropriate antibodies, and the clonal selection makes it possible to keep the best antibodies to make cells memories of them. These three processes are independent; they are subjected to no central body to manage them.
MAISId is a system that performs frames analyses by a group of immune agents collaboration. These agents are distributed on the network to achieve simultaneous treatments, and are auto-adaptable to the evolution of the environment and have also the property of communication and coordination in order to ensure a good detection of intrusions in a distributed network.
An advantage of this approach is that MAISId can generate many different patterns to recognise intrusion in network flow. A disadvantage is a possibility that the system throws away a pattern which can be useful in the future.
A biological inspiration from MAISId was useful also in our M-AHIDS. We have used the idea of the biological immune cells in two cases. The first case of application is in the middle between the evaluation score from detection agent and the multi-agent temporal logic in logical agent. The second case of application is during negotiation among agents. The negotiation approaches are described bellow in this section. M-AHIDS has not created new agents for intrusion detection yet, but we are rating successfulness of our agent. This rating influences weights in logical agent, which finally makes decision about the connection.
There are two major inconveniences of the existing IDS [14]. The first one is their difficulties to adapt oneself to the changes of the network architecture and especially how to integrate these modifications in the detection methods. The second one is their high rate of false-positives (false alert).
On the another side the intrusion detection system is effective if it has the following characteristics [15,1]: Distribution -to ensure the monitoring in various nodes of the network the analysis task must be distributed. Autonomyfor a fast analysis, distributed entities must be autonomous at the host level. Delegation -each autonomous entity must be able to carry out its new tasks in a dynamical way. Communication and cooperation -complexity of the coordinated attacks requires a correlation of several analyses carried out in network nodes. Reactivity -intrusion detection major goal is to react quickly to an intrusion. Adaptability -an intrusions detection system must be open to all network architecture changes.
The negotiation is essential in settings where autonomous agents have conflicting interests and a desire to cooperate. For this reason, a mechanisms in which the agents exchange the potential agreements according to the various rules of interaction which have become very popular in recent years as evident, for example, in the auction and mechanism design community [16]. We use negotiation for finally deciding in M-AHIDS which connection is intrusion and which is normal.
There are basically 3 type of negotiation: Heuristic, Game-theoretic and Argumentation.
The heuristic-base approach can be a model for multi-issue negotiation under time constraints in an incomplete information setting. An important property of this model is the existence of a unique equilibrium [17]. Another solution [18] uses approximating the rational choice of negotiation strategies with the use of decision functions. PhD thesis [19] describes lot of heuristic-base approaches and other approaches used for negotiation.
The game-theoretic approach for negotiation can be used in an auction [20], where the seller wants to sell the items and to get the highest possible payments for them while every bidder wants to acquire the items at the lowest possible price. Authors of paper [21] use mathematical model of the network security domain. This concrete method is used for IDS and provides the mathematical formulation for the two persons security game between the defender and the attacker. Another similar approach is trust-based solution for robust self-configuration of distributed intrusion detection systems from [22,23] is defined as a game-theoretical frame-work suitable for the collaboration of multiple heterogeneous IDS systems and it introduces a simple effective game solution concept -FIRE.
The argumentation as negotiation is the most interesting approach for our M-AHIDS. Argumentation works by constructing series of logical steps (arguments) for and against propositions of interest and as such may be seen as an extension of classical logic [24]. In classical logic, an argument is a sequence of inferences leading to a true conclusion. In argumentation system arguments can be not only a proof that propositions are true or false, but also a suggestion that propositions might be true or false. The strength of such suggestion is ascertained by examining the propositions used in the relevant arguments. This form of argumentation may be seen as a formalisation of work on informal logic and argumentation in philosophy, though it should be stressed that it was developed independently.
A formal mental model of the agents based on minimal-structure of possible worlds (time lines) has been developed using modal operators for beliefs, desires, intentions and goals having an appropriate set of properties in [25]. This approach was an inspiration for our argumentation and for a logical machinery implemented in the logical agent. Our solution is describe in the next section 4.3.
Diagram of M-AHIDS is shown in figure 2. M-AHIDS is based on Microsoft .net 4.5 framework and multi-vendor sampling technology sFlow. It originally runs on Microsoft server 2012. However, it can run also on Linux base operation system with mono project. M-AHIDS is implemented as multi-thread application which uses sFlow for receiving sFlow UDP datagrams.
sFlow is a multi-vendor sampling technology embedded within switches and routers. It provides the ability to continuously monitor application level traffic flows at wire speed on all interfaces simultaneously. sFlow monitoring of high-speed, routed and switched networks has the following properties [26]: Accurate, Detailed, Scalable, Low Cost and Timely M-AHIDS save approximately 10 minute window of received sFlow datagrams in SQLlite in-memory database. This technology of in-memory database enables to analyse a lot of received data very quickly. All detection agents work with this database and it is also an input to logical agent.
M-AHIDS network intrusion detection system is made as four layer system.
The first layer contains in our case network 10Gb switch with sFlow agent. Switch can be replaced with another network device with sFlow agent. sFlow agent sends sFlow datagram to our IDS, which is also the sFlow collector.
The second layer contains sFlowTool and pre-processing agent. sFlowTool receives sFlow UDP datagrams. M-AHIDS reads encoded result from sFlowTool and important data saves to in-memory database. Nowadays we use these information from sFlow: 'srcIP', 'dstIP', 'srcMAC', 'dstMAC', 'srcPort', 'dstPort', 'IPProtocol', 'sampledPacketSize', 'UDPBytes', 'TCPFlags', 'inPort', 'outPort' and 'time'.
The third layer contains the detection agents. Every agent is implemented as an autonomy thread. The number of the actually active agents depends on the number of the cores in computer processor.
The forth layer contains logical agent, database with results and front-end for network administrator, which admin can use to correct the results.
As we mentioned in section 2, we have taken an inspiration for our agent in the project CAMNEP [2,3]. However, there are two main differences: We have built the agents differently and we have a logical agent to complete the final decisions.
The first step after IDS receive sFlow datagram is pre-processing as can be seen on figure 2. For covering this function we implement a pre-processing agent. Our IDS is designed for a huge network traffic on 10Gb switch. For this reason, we must do some quick decisions, which connections are interesting (connection has probability of being a intrusion). Like the other mentioned IDS we do this with several rules. The rules define which source, destination, port and protocol or they combination are OK and they are not interesting for the detection agents. Administrator of network can define and edit these rules.
Nowadays we have tested 5 types of intrusion detection agents. Two of these agents have arguments suitable for specification. Using this, we get 11 intruder detection agents. Every detection agent evaluate every connection from preprocessing agent. This evaluation is a integer number. Higher number means more unusual behaviour.
Average agent computes average number of connections with same property (dscIP, srcIP, dscPort, srcPort).
Volume agent counts number of the connections which have a same property and which are connected to the connections which have another same property. Concretely, we map with this method srcIP to dstIP, dstIP to srcIP, srcIP to dstPort and dstIP to srcPort. All of these mappings are provided by separate agents, which are running parallelly.
Cluster agent is the most computationally hard agent. This agent computes normalization distance between each of the connections. Agents use dscIP, srcIP, dscPort, srcPort, dstMac, srcMac for distance computations.
Web agent is one of our new contribution for this area of research. Web agent uses the database of university web page's visitors and it compares IP address of web page visitor and IP address form sFlow. If IP address is in both databases, we can decide if behind connection there is some system or a real user and then we can determine intrusion score for the connection. To determine the connection, the visited pages are analysed. If web pages are systematically visited page by page, then this is done with high probability by some system. If same page is visited more than once in short time, then the visitor was with high probability a real human user. We have database of university web page visitors from our Internet users anonymity research [4][5][6].
Entropy agent captures degree of diffusion or gathering of distribution of connection properties. This detection method is based on equation:
where S = N i=1 n i and X is set of connection properties X = {n 1 , ..., n N }.
Logical agent makes final decision about every connection and if this agent decides that this connection is intrusion, then agent inserts this connection to result disk storage database. Our logical agent is based on Multi-agent Temporal Logic MTL which we mentioned in section 2 and which we describe in subsubsection MTL in M-AHIDS below. This logic is developed especially for needs of M-AHIDS. The past states in MTL are from previous results, which are saved in permanent database. The future states will be computed by time series and Fourier transform. These future states are not implemented yet.
Logical agent has 3 important tasks. The first is to build knowledge base from results of detection agent. In this stage, LA normalizes the results to real numbers from interval 0, 1 . Normalization uses network administrator's corrections and immune inspiration for updating DA trust weights. Trust weights are also real numbers from interval 0, 1 . Higher number means more trust for the agent.
After normalization, LA uses argumentation framework to negotiate final decision -which connections are intrusions. We describe our argumentation framework in subsubsection Argumentation framework below. The last task for LA is to save results to permanently database.
MTL in M-AHIDS is one of the modal logics. Naturally, there are many approaches of how to build logical agents but we have decided for the multiagent temporal logic (MTL). We have chosen this logic, because it allows as to compare every detection agent in time. This property of the MTL we use to decide, which connections are finally the intrusion.
We define simple logic syntax because nowadays we use only small subset of possible power of MTL. There are many reasons for this choice. One of the most significant is real time running of computationally hard problems in IDS. However, it is strength enough for making correct final decisions. Syntax of logic where φ is logic formula and p ∈ prop is:
Connectors F i , G i , P i and H i are temporal connectors for one agent a i ∈ A and F A , G A , P A and H A are connectors for all agents. For every judge connection there is one atomic formula p which acts in M-AHIDS as a connection with normal behaviour. The argumentation framework is one of the approaches for negotiation amongst agents. Nowadays, we use only very tiny framework which is definitely not complete because the intrusion detection is very computationally hard and M-AHIDS must work parallel with network operation. But we are still optimizing it and we will also extend this argumentation framework.
The base of our argumentation is the binary relation −→. φ −→ φ means that φ is stronger than φ . The logical formulas φ and φ belong to −→ iff both contain same the atomic formula p with a opposite value. That means that the two DAs have contradictorily results about trust of same the connection. For solving this contradiction we use this rules: X i ϕ : w −→ X j ϕ : w , H i −→ P j , G i −→ F j and if X A ϕ then ϕ where X ∈ {F, G, P, H} and agent weights w > w .
We have implemented M-AHIDS button up using several iterations, because the most important requirement on IDS is real time detection. After each iteration we did performance test and optimization. Nowadays we have the proposed intrusion detection system M-AHIDS partially implemented . We did not make a long time test, because the M-AHIDS is still in implementing and developing stage. However, we did some tests. During these tests, the system was supervised and it learnt usual network behaviour. After three day of learning we tested system for some attack as DOS, DDOS, Port Scans, BitTorrents (there are usually unwanted in department network) and Malwares.
In the figure 3 The table 2 shows a false positive rate of the agents. We tested M-AHIDS during usual week network operation. Every anomaly was sent 100 times and with these anomalies we sent same number of connections with similar properties as sent anomalies. During these tests we got 3 percent false negative detections.
In this paper we have presented a proposal of a system for detection intrusions in a network. The most important system features of developed and partially implemented M-AHIDS are integration of the several anomaly detection techniques in a form of agent, machinery of a multi-agent temporal logic, hybrid negotiation with argumentation and immune cell inspiration and last but not least new innovative Web agent which is able to detect trustworthy host from his activity on web pages. This agent is based on our previous research which is deployed on all web pages of Comenius University for one and half year.
When we set the system to pass about 3 percent false negatives in the normal connections then we got 36 percent false positives in malicious connections, what is satisfaction result because project CAMNEP [3] has with 1 percent false negatives in the normal connections 40 percent false positives in malicious. M-AHIDS is still in developing state. However, we have implemented the most of the presented features of M-AHIDS. Only one important feature we have not implemented yet -prediction of a normal network behaviour from the collected data.
As a next step we would like to implement the rest of the features to M-AHIDS, to optimize the already implemented features and to provide more and longer tests.