-The article presents variants for building a cognitive early warning system about a computer attack on the information resources of the Russian Federation on the basis of Big Data technologies. The essence of Big Data technologies is considered in the context of its application in information security. Approaches for stream processing of data based on the CEP model, the MapRedutze modification, the model of actors, the combination of the model of actors and the modification of MapRedutze are analyzed. The architecture of the prototype of the software-hardware complex "Warning-2016", a typical scheme of the hardware implementation of the stand and the technical specification of its equipment are presented.
At present, the technology of processing, storage and
analysis of large data, Big Data (hereinafter - the technology of
Big Data) are becoming increasingly important for the state of
critical infrastructure security monitoring of the Russian
Federation (electrical networks, pipelines, communication
systems, and so forth.) and monitoring the corresponding
criteria and indicators of information security and sustainability
of functioning in general [
Big Data technologies are already being used in a number
of cybersecurity applications. For example, in Security
information and Event Management systems (SIEM)
nonrelational database (NoSQL) are used to store logs, messages
and security events. In the near future, a qualitative leap in the
development of SIEM is expected on the basis of models and
methods of forecast analytics. In the known solutions of Red
Lambda, Palantir, etc., Big Data technologies are used to build
user profiles and social groups in order to detect abnormal
behavior [
The relevance of Big Data technologies is confirmed by the
fundamental possibility to conduct "online analysis" of packet
and streaming data, to isolate and process significant simple
and complex cybersecurity events in real (or quasi-real) time
scale, and to generate new useful knowledge for detection and
prevention of security incidents. It is significant that Big Data
is able to provide proactive security and monitor the impending
information security incidents even before they can adversely
affect the sustainability of the critical infrastructure [
Thus, by Big Data technologies in information security we
will understand the technologies of efficient processing of
dynamically growing data volumes (structured and
unstructured) in heterogeneous Internet / Itranet and IIoT / IoT
systems for solving urgent security tasks. The practical
significance of Big Data technologies lies in the ability to
detect primary and secondary signs of preparation and conduct
of computer attacks, the detection of abnormal behavior of
controlled objects and subjects, the classification of previously
unknown mass and group cyber attacks (including new DDOS
and APT), the detection of the traces of computer traces
crimes, etc., that is, in all cases when the use of traditional
means of information protection (SIEM, IDS / IPS, system of
protection from unauthorized access to information,
cryptographic information protection facility, antiviruses, etc.)
is not very effective [
II.
Currently, the following approaches are known for
streaming data processing [
In the first approach, the classic Complex Event Processing (CEP) model is used. The use of CEP allows you to search for "significant" cybersecurity events in a data stream over a certain time interval, perform a correlation analysis of events, and allocate appropriate event patterns that require immediate response.
To automate the process of developing data processing systems based on CEP, a number of tools are proposed (f. e., StreamBase with its own declarative programming languages StreamSQL and EventFlow).
The practice of using CEP has shown that it is optimal for the collection and processing of simple cyber-security events. For example, to extract events from several data streams, aggregate them into complex events, reverse decomposition, etc. However, the implementation of complex logic for handling cybersecurity events is difficult. To solve this problem, we proposed an approach based on the generalization of MapReduce to the processing of streaming data. allocated Storm system (Twitter), and S4 (Yahoo!) and Zont (Moscow Institute of Physics and Technology, MIPT).
In the second approach, the D-Streams model of discrete streams is used, in which streaming computations are presented as sets of non-session deterministic batch calculations on small time series intervals. It is significant that such a representation of calculations allowed not only to implement the complex logic of processing cybersecurity events, but also to offer better methods of restoration than traditional replication and backup copying. The fact is that in practice, in computer networks with a large number of nodes (from hundreds or more), failures and "hangs" (or "slow" nodes) inevitably occur, and here the operative data recovery in case of failure or failure is important enough. Since, even a minimum delay of 10-30 seconds can be critical for making the right decision.
It should be stated that, apparently, for the known systems the streaming data Storm, MapReduce Online et al. resiliency reached threshold values. The systems mentioned are based on the model of "long-lived" session operators, which, upon receiving the message, update the internal state and send a new message further. In this case, the system is restored by replicating to a pre-prepared copy of the node or by backing it up in a data stream, meaning "replay" of messages on each new copy of the "fallen" node.
As a result, the use of the replication mechanism results in a costly two or three times the node reservation, and the use of a backup in the data stream is characterized by significant time delays due to the need to wait for the nodes to "update" when the data is re-run through the operators. In addition, none of these approaches can not cope with "hangs". Replication systems use Flu synchronization protocols to coordinate replicas and hangs slow both replicas. When backing up, any "hang" is considered a failure with subsequent costly recovery.
The D-Stream model offers better recovery methods. For example, the Resilient Distributed Datasets (RDD) recovery method, which allows you to restore data directly from memory without having to replicate for several sub-seconds, or a method of parallel restoration of the state of a "lost node" in which, when the node falls "It initiates the" connection "of the workable nodes of the cluster to the" recalculation of the lost "structure of the RDD. Note that in traditional systems of continuous data processing such restoration is impossible due to complex synchronization protocols.
Note that using the D-Streams model requires splitting an array of input data into streams, which inevitably results in the loss of certain events. In addition, in the case of large flows, the data processing system is no longer flexible and scalable. The time the system responds to events slows down and the system moves further away from the real-time mode. To solve these problems was proposed third approach - based on actor model.
In the third approach, the streaming data processing
systems are based on the actor model. Here, actors are
understood as some primitives of parallel computations. The
main advantage of the actors is the ability to store states,
including those obtained from historical data, which can be
used to highlight significant cyber security events. Among the
known solutions of the streaming data based on actor model
The first two solutions, Storm and S4 [
The third of the named Zont system can work with a large number of actors. This is true when working with a cloud of sensors, when each sensor is assigned its own actor. To develop distributed resilient systems, it is possible to use the Erlang & RIAK Core development environment. Here, the functional language Erlang (Ericsson) allows you to create programs that can work in a distributed computing environment on several nodes (processors, cores of one processor, cluster of machines), and the open library Riak Core (Basho Technologies) allows to create distributed applications according to Amazon's Dynamo architecture.
Thus, all three systems, Storm, S4 and Zont (MIPT) can be used to process a large data stream from several sources. In this case, Zont is optimally suited for working with a cloud of sensors.
In the fourth approach, the combined advantages of the second and third approaches, which allows the system to create the streaming data in real time based on the combination and modification MapReduce actor model.
III.
BIG DATA
Consider the possible options for building a cognitive early warning system about a computer attack on the information resources of the Russian Federation (software and hardware complex, SHC "Warning-2016") on the basis of Big Data technologies.
Variant 1. Implementation of the experimental model of the SHC "Warning-2016" based on HBase.
Here the basis for the proposed solution was the nonrelational distributed database HBase, working on top of the HDFS file system (Hadoop Distributed File System).
This database allows you to perform analytical and predictive operations on terabytes of data to assess the threats to cybersecurity and the stability of the critical infrastructure as a whole. It is also possible to prepare in the automated mode appropriate scenarios for detection, neutralization and warning.
The second hypothesis analysis module is designed to handle large amounts of data, respectively, from it require high performance. The module interacts with standard configuration servers and is implemented in C language (via PECL, PHP extensions repository). Special interactive tools based on JavaScript / CSS / DHTML and libraries such as jQuery have been developed to work with the content of the proper provision of cybersecurity.
As a data store, MySQL is used - Percona Server (version 5.6) with the XtraDB engine. DB servers are integrated into a multi-master cluster using the Galera Cluster. For balancing the database servers, haproxy is used. Redis (version 2.8) is used to implement task queues, as well as for data caching.
As a web server, nginx is used. Involved PHP-FPM with APC enabled. For balancing HTTP requests, DNS (multiple Arecords) is used. To develop special client applications running Apple iOS programming languages are used: Objective C, C ++, and Apple iOS SDK based on Cocoa Touch, CoreData, UIKit. The above applications are compatible with devices running Apple iOS version 9 and above.
To develop applications running Android OS, the native Google SDK is used. The applications are compatible with devices running Android OS version 4.1 and higher. Software development for the web platform is carried out using PHP and JavaScript.The experimental sample SHC "Warning-2016" is deployed in Saint Petersburg Electrotechnical University "LETI" on the platform DigitalOcean and contains in its composition: 3 servers for production stage; 1 server for testing stage.
In addition, CloudFlare is used - to increase the speed of the
service (through the use of CDN) and protection from DoS
attacks. The carried out load testing of the experimental model
of the SHC "Warning 2016" indicates the viability of the
proposed technical solution [
Variant 2. Implementation of the prototype of the SHC "Warning-2016" on the basis of the telematics platform Zont (MIPT)The possible architecture of the experimental layout of the SHC "Warning-2016" based on Zont is presented (see Fig. 1). Here the basis of the proposed solution was the telematics platform Zont, which allows creating resiliency scalable cloud systems for streaming large data. Table 1 describes the modules of the experimental layout of the SHC "Warning2016" based on Zont.
It is significant that Zont has its own specialized storage, built on the basis of Riak Core technology, for storing and retrieving archived data that represent time series. The backend for the repository is LevelDB, which is developed by Google. LevelDB is an embedded KV database specifically designed for use as a backend in the construction of specialized databases, providing operations for writing, searching and sequential data viewing.
The key advantages of the database are high speed of data recording, predictable speed of data search by key and high speed of sequential reading. Also worth noting the following important characteristics of LevelDB for the storage of time series:
a) Use of the LSM tree model, which allows providing high resistance to failures and failures;
At the same time, geo-index support based on geo-hash technology was added to store the input data of mobile sensors. This index is specifically designed to store a large array of information about the spatial position of point objects, in particular, moving sensors.
As a result, it allowed to obtain better performance indicators compared to such well-known universal solutions as Postgres GiST and MongoDB 2dsphere. The hardware implementation of the prototype SHC "Warning-2016" is a cluster of general-purpose servers connected by a network.
The hardware implementation of the prototype SHC "Warning-2016" is a cluster of general-purpose servers connected by a network (Table 2).
A typical scheme of the hardware implementation of the stand is presented (see Fig. 2).
Erlang Virtual Server (Erlang Node Server) to implement the distributed cloud platform module; Erlang Virtual Server (Erlang Node test Server) to implement test module, including sensor network emulator; Auxiliary software server of the Platform, which contains the JBoss application server and PostgreSQL DBMS. The system and application software of the stand included: Linux OS CentOS 6.4; Erlang R160B2 as the execution environment of the distributed machine Erlang; HAProxy - as a proxy server, JBoss - as an application server; PostgreSQL - for storing metadata, etc.
Preliminary tests of the experimental design of the SHC
"Warning of 2016" showed its ability to act [
IV.
The obtained positive experience of using Big Data for solving information security problems testifies to the expediency of choosing solutions based on the actor model and Map Reduce technology over the distributed cloud KV data warehouse. At the same time, solutions based on CEP turned out to be less demanding for memory. They allow storing data in a single "window" of events, but they demanded considerable computing resources when analyzing such "windows". And based on the actor model solutions were less demanding of computer resources, but more demanding of memory due to the need to duplicate data for each event / object. Accordingly, the solutions based on the modification of MapReduce took an intermediate position.
In our opinion, the technology Big Data to radically change the situation in the following areas of information security: Proactive management of cybersecurity incidents; Early detection, prevention and elimination of the consequences of computer attacks; Predictive network monitoring of cybersecurity; Authentication, user authorization and identity management; Preventing computer crime and fraud; Information security risk management; Compliance with regulatory requirements, etc.
In this case, the first results should be expected precisely in
the proactive management of incidents of cybersecurity and
early warning of computer attacks. Note that the known results
of foreign developers of information protection tools confirm
this assumption. For example, in 2015-2016. RSA and IBM
announced plans to create a new generation of security
management centers (Security Operations Center, SOC) [