We present initial work that applies big data and deep models to a cyber security data analysis with a use case approach. We explored new technologies such as BDP (Big Data Platform) as a service on the Amazon AWS system and Lexical Link Analysis (LLA). BDP provides various analytics in near real-time to help decision makers respond to threats and in a timely manner. We also used LLA as an example of deep models and a data-driven unsupervised ML method that can improve cyber decision making.
The cyber data was taken from multiple routers in the Los Alamos National Laboratorys internal network (LANL 2017). The data set contains windows authentication events and processes, domain name lookups, network flow data, and hacking events. The data contains 58 days and total 12 gigabytes of network information and 1.6 billion events. There were known malicious activities (identified as Red Team Actions) conducted within this network during this time period.
Some of the information contained within the dataset was anonymized or deidentified. While this removes significant amounts of information from the data set, there is still valuable information to be gleaned about the behavior of the network due to unity of identification across the five different files (i.e. User 1 or U1 is the same user across all data sets and Computer 1 or C1 is the same computer across all data sets).
Some of the well-known ports (e.g. http port 80, 443, etc.), protocols (e.g. 6 for Transmission Control Protocol), and system users (e.g. SYSTEM or Local Service) were left identified within the datasets. Time was captured in onesecond intervals, starting with a time epoch of (1). In order to illustrate the methodologies studied in this paper, we started with the Domain Name Service (DNS) data set. Figure 1 shows a snapshot of the LANL-DNS data. Time, source computer, and computer resolved are the attributes.
The LANL cyber data set was chosen for a number of
different reasons over other popular open source data sets (e.g.,
DARPA
In order to incrementally test cyber data sets using potential big data and deep models including ML/AI methods, the LANL-DNS data file was initially pre-processed, analyzed, and interpreted to understand the output results shown in this paper before testing on other more complex data sets. The steps for understanding the data: Perform data visualization and exploration: display and visualize data initially and check data quality.
Perform unsupervised machine learning to discover interesting patterns and anomalies.
Apply supervised learning to generate more precise classification or prediction models.
Defense Information Systems Agency (DISA) ’s BDP is on Amazon Web Services (AWS) and a mix of big data standard tools and customization including tools for ingestion, data management, security, data exploration, and data analysis. These functions are supported by open source tools including PostgreSQL, Apache Maven, Apache Spark, Apache Storm (Kronos), Elastic Search, GEM prospector, Hadoop, Map/Reduce, Kafka, Accumulo, Unity, IronHide (Kibana), Zookeeper, Kryolibrary, NodeJS, R-Shiny.
BDP can process large-scale real time data feeds to provide useful visualizations of the data for initial data exploration to discover anomalous events. Ingestion of the LANLDNS data into the BDP cluster included the following steps: Customized and formatted a rapid deployment archive (RDA) for parsing the csv file data Connected a puppet server to upload data to the Kronos server which ingested and parsed the data
For the data visualization and exploration, we used Unity and Kibana/Iron Hide. Unity uses queries to visualize time series, histograms, and pie charts for the initial examinations of the data. Iron Hide creates Data-driven documents (D3) visualizations including heat maps, graphs, and charts which could indicate threats. Figure 2 shows the Unity histogram of the event counts (i.e., each line in the LANL-DNS data is an event associated with a timestamp) for all the computers. Figure 3 shows a Kibana heat map of number of connections made for each computer (y-axis) over time (x-axis). These tools could show big data in a near real-time to provide rapid updates for a focused segment.
Visualization/Exploration Using Gephi and Plotly For the data exploration, we also used a open source network display program Gephi (Gephi 2018) as a way to visualize the LANL cyber data that shows the connections between points in data sets. Gephi uses Source and Target fields to draw the network graphs. Gephi also includes a timeline function to allow a user to view the connections between nodes at specific times or in a range of times.
The LANL flow data was displayed with Gephi. Since the red team created hacking events such as teal colored computer nodes in Figure 5, the hacking or hacked computer nodes resulted from the red teams actions. Each node is a computer. Figure 4 shows the hacking events during a 24hour period. One teal node is hacking, the orange nodes are being hacked, purple nodes are neither hacking nor being hacked. The color of the edges between nodes represents the protocols used for the connections. Purple edges are most likely TCP. Green connections are protocol-1 which may be related to the hacked computers.
The shape of the graph provides clues as to the nature of the nodes. Nodes that are highly connected to other nodes may be name servers or popular web servers. The hacked nodes seem in the area of the nodes with higher numbers of connections (high centralities).
We also explored the Sankey graph with Python Plotly (Sankey 2018). Figure 5 shows a Sankey graph to categorize how different parameters such as protocols, port numbers, and packets connected to each other in the LANL flow data. For example, protocol-6 is mostly associated with port ranges 1025-65536 and then port ranges 0-1024.
In a LLA (Zhao, MacKinnon, and Gallup 2015), describes the characteristics of a complex system using a list of attributes or features with specific vocabularies or lexical terms. Because number of lexical terms can be potentially very large from big data, the model can be viewed as a deep model for big data. For example, we can describe a system using word pairs or bi-grams as lexical terms extracted from text data. LLA automatically discovers word pairs, and displays them as word pair networks. Bi-grams allow LLA to be extended to numerical or categorical data. For example, for structured data such as attributes from databases, we discretize and then categorize attributes and their values to word-like features. The word pair model can further be extended to a context-concept-cluster model (Zhao and Zhou 2014). A context can represent a location, a time point or an object (e.g. file name) shared across data sources. For example, in information assurance, information is the context, assurance is the concept. The timestamp, computer name are the contexts to link different data sources.
Figure 6 shows an example of such a word network discovered from text data. Clean energy, renewable energy are two bi-gram word pairs. For a text document, words are represented as nodes and word pairs as the links between nodes. A word center (e.g., energy in Figure 6) is formed around a word node connected with a list of other words to form more word pairs with the center word energy.
We computed associations and links as pairs of a source computer and a resolve computer from the LANL-DNS data set. The strength of the associations and links are defined as how many time points or events that the two computers are linked via “source” or “resolve”.
The output from LLA for the LANL-DNS data processing identified 15237 unique active devices (computers). There are no identifying features differentiating an end user device such as a personal computer versus a DNS server; all are identified as anonymous devices, such as C123. Figure 7 shows an example of a LLA network discovered from the LANL-DNS data. Each node is a computer. The links represent how likely two computers are linked as a “source” and “resolve” pair in the events (timestamps).A correlation measure is computed using Equation (1). Colored nodes (computers) are grouped into one clusters based on their link patterns using LLA.
rij = (Linked Events Computer i and Computer j) p(Events Computer i)(Events Computer j) (1)
One can filter the nodes based on the strength of the links in LLA as shown in Figure 8.
The detail LLA outputs for the LANL-DNS data set are listed as follows:
Output 1: The list of words representing the computers in the data set and nodes in the network with the following characteristics computed as shown in Figure 2.
Group: what group a node belongs. A node or a word is a computer.
Betweenness: how many connections belong to the different groups.
Degree in: how many connections a computer (word) as resolve. Degree out: how many connections a computer (word) as source.
Output 2: The list of associations of computer associations.
After the initial data exploration, the question of the research is that how to predict hacking and hacked computers from these data sets. We computed additional metrics based on the Output 1 of LLA as follows:
DIV: degree in/degree out if degree out not 0;else 0;
Figure 11 show a gains chart for predicting the hacked and hacking computers. The x-axis shows the computer sequence number ranked by the four metrics. The y-axis shows percentage of hacked or hacking computer nodes. 1.75% out of 15237 total computers are either hacked or hacking as the ground truth, therefore, if there is a perfect prediction algorithm, the top 1.75% of the sorted nodes (based on the perfect scores) should predict 100% of the hacked or hacking computers as shown in the leftmost curve (two straight lines). The two results are interesting:
The best performed prediction metric is Multi (degree in*degree out) where the top 2160 nodes (14%) include 62% of the total hacked or hacking nodes. This is the best gain over other scores: For example, if sorted by the degree in scores, the top 14% contains 56% of the total hacked or hacking nodes. If sorted by the random scores, 14% contains 14% of the total hacked or hacking nodes, which is the worst performing prediction.
The bottom ranked 40% of the nodes (from 9112 to 15237) are normal. This is also significant since we can eliminate the 40% nodes when examining hacked or hacking nodes, which is a big labor saving for cyber security analysts.
The metric “degree in*degree out” indicates highly
active devices are more likely to be hacked. The highly
active devices do not mean they are anomalous, however, a
common behavior seen in malicious actions is increased
activity of devices that may be participating involved in the
unauthorized action. We later computed an activity metric
by counting the number of event (i.e. timestamps) a
computer is associated in the data set. This is a much simpler
metric to compute than the associations in LLA. The
activity metric shows a similar gain to the best LLA metric. We
also appended other node characteristics of in the flows data
such as the number of source ports, number of destination
ports, total duration of a nodes connections, total packets
of a nodes connections, total bytes of a nodes connections
as shown in Figure 12, and then apply supervised machine
learning methods using the tool
We applied big data and deep analytics methods to the LANL cyber data used to detect the hacked or hacking computers in a network where DNS, flows, services and login information are collected. We showed how big data visualization and exploration tools such as BDP, Gephi, and Python Plotly can explore ig data to provide meaningful information to decision makers. Gephi and Plotly are good for prototyping. The BDP has security advantages and shows potential for finding anomalies in near real time through various metrics. LLA computes the associations, statistics and centralities for nodes (computers) and derived metrics are significantly useful to predict hacked or hacking nodes in the gains chart evidently. The best performing metric shows the top 14% of the nodes include 62% hacked or hacking nodes and the bottom 40% of the nodes are 100% normal, therefore can be eliminated from examination.
Authors would like to thank the Naval Research Program at the Naval Postgraduate School and the Naval Research Enterprise Internship Program at the Office of Naval Research for the research support. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied of the U.S. Government.
Zhao, Y., and Zhou, C. 2014. US patent 8,903,756: System and method for knowledge pattern search from networked agents. retrieved from https://www.google.com/patents/us8903756.