Analysis of the antivirus technologies, showed that they are not able to detect new spyware with high efficiency, which significantly reduces the reliability and efficiency of its identification. Techniques based on heuristic analysis have a high rate of false positives. The paper presents a new technique for the spyware detection method in computer systems that provides a principle of proactivity and is based on mechanisms machine learning with the reinforcementlearning. The suggested method of spyware detection is based on software behavior analysis in computer systems. The suggested method involves the computer systems monitoring concerning the software, operates with the behavior.
Today spyware is one of the most common threats on the Internet to businesses and
individual users, since it can steal sensitive information and harm the network [
Some types of spyware are able to install other additional malware or removing certain programs and modify the parameters of the operating systems. In addition, this kind of malware can redirect browser activity, which entails visiting websites blindly with the risk of virus infection.
Some types of viruses can bring spyware along for the ride as they spread [
The main problem is that there is no clear border between benign and malicious spyware applications when the received information is used to the detriment. 2
Related works
Today solutions to the problems of security are widely presented in the literature. One
of the ways to prevent and defend computer system from the spyware spread is to
construct resilient systems [
The work [
In the paper [
The paper [
The works [
The paper [
The papers [
In the work [
The articles [
In [
Nevertheless, the mentioned above approaches have common drawbacks: they don’t take into account a set of features, that may assign benign and malicious spyware clearly. 3
Spyware Detection Technique based on Reinforcement Learning
A new spyware detection technique based on reinforcement learning is proposed. It is a proactive approach for the malware detection, and allows detecting all types of spyware. The technique is based on mechanisms machine learning and is able to detect new unknown spyware.
The suggested method of the spyware identification uses software behavior analysis in the computer systems.
The main steps of the proposed approach are presented below: 1. Spyware sample construction. 2. Usage of the reinforcement learning algorithm, the rewards evaluation. 3. Computer systems monitoring concerning the software behavior. 4. Features selection that may indicate the presence of spyware in the computer systems. 5. Evaluation of the reward for research object. 6. Comparison of the obtained rewards with the rewards values of the known spyware.
Let us consider the steps of the method in more detail. 3.1
Computer systems monitoring concerning the software behavior In order to conduct the monitoring stage, we are to investigate the spyware functioning in the computer system. Thus, let us present it as a tuple: = 〈 1, 1, 1, 1, 1, 1, 1 1, 1〉, (1) where 1 = – a set of malware’s actions that tells an attacker about its presence on the network; 1 = – a set of actions to penetrate the operation system of the computer system via copying its body into the system directory; 1 = ℎ – a set of actions that register the spyware’s body into the system registry in order to automatically start after the operating system restarting; 1 = – a set of actions for the messages sending, – via e-mail; – via messengers, , 1; 1 – an operating memory of the computer system; 1 = – a set of actions when an attacker begins to tap a specified TCP port; 1 = – a set of actions by which spyware accesses the command line of the victim's computer (only works when receiving the message); 1 = 4=1– a set of spyware lifecycle stages; 1 = – a set of spyware functions, which is determined by spyware lifecycle stages. Let us 11 present the function of the infection, 11 ⇒ → { | ∈ }, where – a set of malicious spyware actions, T - a set of the computer systems in the network, – an infected computer system with the spyware. Let u present the function of copying to the operating system directory and register itself in the system registry 21 ⇒ 1 21 → {ℎ|ℎ ∈ 1} ; a command to re-give the collected data an attacker to the attacker to 31 listen to the specified TCP/IP port, 31 ⇒ 1 → 2′ ; an access function to a command 41 line of the victim computer 41 ⇒ 1 → { | ∈ 1}. 3.2
Usage of the Reinforcement Learning
The proposed technique is based on the usage of the reinforcement learning. It is a
new approach in the machine learning. The main idea of it is learn how a research
object’s states are being changed under the specified action in the situation when there
is a single reward feature. During the learning stage aims the achievement of the
target via usage of a reinforcement factor finding the most optimal object’s action to be
performed in each its state. The action is chosen anyway. And after that, the agent is
receiving a new state or a reward. Iterating this process, the specified agent is being
learned that the best action to be obtained is to the expect maximum value as in a
form of rewards [
Primarily, in each iteration the agent is to be changed its current states s, s∈S and is
to select a specified action a, a∈A. During the state changes the actor also is able to
receive some reward signal with value r, r∈R. In these iterations, in order to get a
useful experience in regards to states, agent, the transition and reward values, the
agent demands best action and the system evaluation corresponding to the learning
procedure [
An effective approach for the reinforcement learning usage is the temporal
difference technique [
This approach operates with the agent which is everting to evaluate the value function on the base of getting the step-by-step reward and the defined reward for the further object’s state which it obtains.
In this point of view, the most useful and simple method is the Q-learning
approach [
It means, that whether the agent is selecting the specified action a being in the specified state S and further is changing it to state S′, it is obtaining the maximum Q value of next state.
The employment of the reinforcement learning enables the spyware detection via
the usage of the feedback of the reward obtaining, which is able to present data in a
form, that enables a high efficiency identification [
Spyware detection technique based on reinforcement learning operates with m specified research object’s states – the set of variants of the API functions, that perform malicious activity. Thus, the set of research object’s states can be presented as S={Si, | i = 1, m}.
In this case, object’s states present the properties of the Markov chains, so it contains the state’s information corresponding to the past and the present, and it is important for the training procedure.
In order to perform spyware detection, the technique has to identify n actions – the changes concerning executable.
In the approach we deal with the different categories API calls used for the spyware execution in the computer system: 1. Registry (registry manipulation for spyware installation, activation, stealth functioning etc). 2. Network management (managing the network related commands). 3. Memory management (managing the system memory for its hiding, temporary presence in the system etc). 4. File i/o (file management with stolen information). 5. Socket (socket related commands). 6. Processes and threads (operations with registers, threads for executing main spyware functioning). 7. Dynamically linked libraries (dll manipulations).
After the agent has chosen the action it had obtained an answer from the environment - reward or punishment. It is a of the environment’s feedback corresponded with the mentioned system action.
Reinforcement learning deals with quality function. It is the presentation of the optimal action’s state value, correlated with other actions’ states. The quality function value dependents on both the state’s and action’s values, e.g. Q(s, a) demonstrates the Q(s1,a2) the optimal action’s state value of action a2 in the state s1.
Based on the quality function it's chosen the optimal action’s state value concerning the current state, which is used for the learning process.
Employing the reinforcement learning as the approach to the spyware detection, let us define the correspondence between its notions (environment, action, reward and agent) and spyware detection items.
Let's assume the environment item as the representation of the system monitor, able to extract spyware’s features, which are to be analyzed.
The result of the analysis is the conclusion about the spyware’s presence or absence in the computer system, and it is used for the reward definition.
The reward approach in the reinforcement learning is able to categorize objects via supervised learning. It answers to agent a question in what way the learning process may move on concerning object’s actions.
The mechanism of rewarding has to present the changes that the environment has executed subsequently the agent implements the action in the current state. Also. It demonstrates the achievement process to the target – maximum value of the detection efficiency.
After that, the pair state/reward are sent to the agent. The agent obtains the set of
features that may indicate the spyware’s presence and the reward and uses them for
the system learning via Q-learning algorithm [
Algorithm 1 operates with the parameters α, which defines the learning speed. In practice, its optimal value is 0.1, is usually recommended for the learning rate, and we have used this value as well, and γ – the discount factor, which is used for the divergence prevention with the value 0.9. The aim off learning is to achieve the optimal action’s quality value Qmax(s′,a′) for the new system’s states.
The situation, when the agent has achieved needed values, that is object is the spyware on benign. While these values haven’t achieved, the algorithm is repeated, otherwise it is terminated.
The Q-learning algorithm as a training tool involves the steps, 1. Initialization of the Q(A, s) value to zero for all states and actions. 2. Obtaining of the system’s current state. 3. Choosing procedure for the object’s action on the base of algorithm. 4. The object action performance and gathering of the information via monitoring and obtaining the rewards concerning this object’s action. 5. The obtaining of the new system state (S') caused by the object’s action performance. 6. The evaluation of the values Q (A, s) using formula:
Q(St, at )=Q(St,at )+α[r(t+1)+γ maxa (Q(S(t+1),a_t )-Q(St,at))],
(3)
where α=[
The scheme of the spyware detection process using the reinforcement learning functioning is presented in Fig.1.
In order to assess the efficiency of the proposed technique a number of
experiments were carried out. For the purpose of the classifier training, the API call
sequence of spyware samples and benign application from dataset [
The training dataset included 16350 malicious samples and 14571 benign samples. All malicious samples were divided into 6 spyware classes: adware, keyloggers, infostealers, red shell spyware, tracking cookies, rootkits.
In order to determine effectiveness of the proposed spyware detection technique
17231 spyware samples from evaluation dataset [
The experimental results were estimated using standard detection metrics [
SN =TP/(TP + FN), SP =TN/ (TN + FP), E=(TP + TN)/(TP + TN + FP + FN) (4)
Distribution of the malicious samples that involved the experiments by spyware classes and results of experiments are presented in the Table 1.
The experimental results show that usage of the reinforcement learning demonstrated ability to detect spyware with the efficiency in the range of 95,63 to 99,05% with the false positives in the range of 0,02 to 0,28%.
Spyware types adware keyloggers infostealers red shell spyware tracking cookies rootkits Total number
Number of spyware samples 2419
The work presents the technique for the spyware detection method in computer systems that provides a principle of proactivity and is based on mechanisms of machine learning with the reinforcement. The suggested method of spyware detection uses software behavior analysis in computer systems. The suggested method involves the computer systems monitoring concerning the software, operates with the behavior.
Experimental research demonstrated a promising result of the detection ability of the proposed approach, while the false positives rate is low.
The further work may be devoted to the development of the techniques that involve other machine learning algorithms and spyware features analysis.