Intrusion detection systems generate a large number of streaming alerts. It can be overwhelming for analysts to quickly and effectively understand behavior within a network. Critical alerts occur so infrequently that it can be difficult to determine what surrounding alerts are actually related to them, providing a deep challenge to analysts. What if an analyst could provide a collection of known critical alerts and quickly receive a summary detailing their temporal behaviors within a network as well consistently co-occurring signatures that pre-empt or succeed the critical action? What if this information could be provided in near real time, with no training data, and with the capability to adapt to changing temporal patterns and relationships across signatures? The Concept Learning for Intrusion Event Aggregation in Realtime with Rare co-Occurring Alert signature Discovery (CLEAR-ROAD) answers that question, revealing consistent co-occurrences derived from alerts with similar temporal arrival patterns. Alerts are aggregated, or sequenced, based on their unique and invariant arrival patterns, not external training data. The signature patterns expressed by such temporal activity are then discovered through pattern mining techniques. A constrained databasing approach is used to reduce the number of sequences processed by an average of 90% for individual streams. Case studies are conducted to analyze the co-occurring signatures found across two real world datasets, one from a SOC operation and another from a penetration testing competition. CLEAR-ROAD is able to find consistently cooccurring signatures across streams and datasets quickly and effectively. Differences in temporal behavior are also found to lead to unique co-occurring signatures for some critical alerts. Case studies show the clear and near-immediate benefits provided to analysts by the system.
Network security has never been more important than in recent times. As networks continue to grow in scale so to does the threat of malicious activity. Even with Security operation centers (SOCs) staffed with analysts dedicated to monitoring a network it is easy for them to be overwhelmed. Modern intrusion detection systems (IDSs) generate massive numbers of alerts quickly making it difficult for analysts to quickly or easily understand what is happening.
Imagine the scenario where a network suddenly sees an influx of a certain critical alert such as “ET WEB_SERVER ColdFusion administrator access." While the signature can inform analysts of general intent, they would be driven to find what additional alert signatures are being generated to construct an overall attacker “action". Even using a short time window around the critical signature, a manual query by an analyst will return a large number of unique alert signatures.
This work aims to solve a unique problem motivated through discussions with real world SOC operators. Given a set of critical signatures from a SOC analyst, can they quickly be provided with the timing information of any other alert signatures which co-occur with statistical dependence? Critical signatures are rare occurring very infrequently in a network and it may not be obvious at a glance how they relate to other alerts. While pattern mining of cyber alerts has been briefly explored in recent literature, this is the first to the author’s knowledge which aims to provide such directed and clear insights into alert co-occurrence to cyber analysts.
This paper introduces and details the Concept Learning for Intrusion Event Aggregation
in Realtime with Rare co-Occurring Alert signature Discovery (CLEAR-ROAD) system. Using
data driven statistical processing, IDS alerts can be sequenced in real time based on their
temporal arrival patterns with no external training data. Pattern mining techniques applied to
constrained sequence data bases [
By processing alerts in this way the additional temporal context and relationships across signatures is represented. Of the 113 critical signatures analyzed, 71 were found to have statistically co-occurring signatures. 65 of these had consistent co-occurring signatures across external sources across two IDS datasets exhibiting similar temporal behavior within the same network. In some cases, variation in temporal behavior lead to unique co-occurring signatures for each timing pattern. These results are highlighted and discussed through thorough case studies of the “GPL EXPLOIT CodeRed v2 root.exe access" and “ET WEB_SERVER ColdFusion administrator access" signatures.
The rest of this paper is organized as follows. Section 2 discusses related work in the field of alert aggregation and correlation. Section 3 details the motivation and challenges in finding cooccurring signatures. Section 4 details the CLEAR-SPADE architecture. Section 5 introduces the experimental set up, datasets used and the differences between them while Section 6 details the case studies conducted through the eyes of a SOC analyst. Section 7 concludes the paper. 2 2.1
IDS systems are used to raise alerts to network administrators of suspected anomalous or
malicious activity [
Alert correlation methods aim to process cyber alerts and provide deeper meaning to them [
Statistically driven approaches to alert correlation attempt to determine relationships across
alerts without prior domain knowledge [
Many alert correlation techniques use some type of windowing when processing new alerts [
In [
Previous applications of pattern mining to cyber alert data would treat entire streams of alerts as individual sequences. Pattern mining algorithms are then applied to the group of all sequences within a network. While this allows for frequent patterns to be found across streams, it does not allow for patterns unique to streams to be explored. Creating sequences within streams is difficult as there is no clear or obvious beginning or ending point to a users “actions" found within alerts. 2.5
The Concept Learning for Intrusion Event Aggregation in Realtime (CLEAR) system is able
to dynamically learn the temporal arrival patterns of cyber alert streams in near real-time and
with no training data [
Finding signatures that occur with a analyst specified critical signature presents a new and
unique challenge. Figure 1 shows an example stream of cyber alerts over time. This mimics an
analyst’s view after querying for the critical alerts and those that occurred temporally near to it.
Are these alerts related to one another? Is the occurrence of the critical signature statistically
dependent on those around it? If so, what temporal patterns exist between them? Can this
example be applied to future instances of the critical signature? These are questions raised
by SOC analysts when attempting to process IDS alert data and ones that do not have easy
answers. By removing a reliance on external historic data and training, analysts can be confident
that the results and statistics they are being presented with are representative of their current
network.
Not every malicious entity attacks a network the same way. Further, a single attacker may
deploy different strategies to reach the same goal when met with resistance. If all alerts from a
single external IP are collected into a single sequence, the new co-occurrences brought on by such
changes in strategy could go unnoticed. Pattern mining algorithms do not consider the number
of pattern occurrences within the same sequence, only the number of sequences containing them
[
Aggregates created by CLEAR are strong candidates for use as sequences in this context.
They are created in a data driven and unsupervised way with no external training data;
sequencing temporally similar alert arrivals together. CLEAR aggregates are temporally invariant
and statistically unique from one another [
By sequencing alerts by their temporal arrival patterns it is possible to analyze and understand signature co-occurrences within specific episode types. Concepts generated by CLEAR are statistically unique from one another. Should a specific alert signature appear in multiple concepts, it is safe to assume that the signature was generated by two unique temporal alert “episodes." The temporal structure of concepts helps give context to any co-occurring signatures found within, and can potentially change which signatures are co-occurring for a given critical one. SOC analysts are concerned with the flow between alerts: which signature co-occurs prior to or after a critical one? At what timing? CLEAR aggregates are a series of successive alerts, however aggregation is based solely on the inter arrival times (IATs) of alerts. Additional analysis is necessary to discover any patterns across other alert attributes within individual concepts. Sequential pattern mining algorithms process collections of temporally ordered items in order to find patterns that occur frequently. From these sequences, rules and their statistical confidence and correlation can be derived. This well fits the goals of this research, to find co-occurring alerts and understand their temporal relationship to a critical signature. The application of sequential pattern and rule mining to CLEAR aggregates and concepts will produce co-occurring signatures unique to the temporal patterns exhibited by the episodes containing them. 4
To quickly and effectively find and deliver co-occurring alert signatures and their temporal characteristics to a SOC analyst the Concept Learning for Intrusion Event Aggregation in Realtime with Rare co-Occurring Alert signature Discovery (CLEAR-ROAD) system is presented. Alerts are processed by CLEAR as they are generated by the IDS and maintain concepts and aggregates for each stream with at most a two alert delay. ROAD post-processes these concepts and aggregates with sequential pattern and rule mining. To reduce overhead and increase efficiency, sequence databases (SDBs) are constrained to only process sub-sequences containing the critical signature while still producing statistics accurately in relation to the entire database. ROAD’s processing can be manually run by an analyst at anytime or scheduled to occur at fixed intervals, processing recent historic time windows based on supplied parameters. ROAD finds all co-occurring signatures and collects and presents multiple levels of statistics to the analyst. High level summaries for critical signatures as well as in depth statistics for each co-occurring signature are collected and delivered to analysts quickly.
A flowchart describing the overall process of the presented system can be found in Figure 2. CLEAR runs in an online manner processing IDS alerts and outputting aggregates in nearreal time. Each aggregate is mapped to the temporal concept it represents. These aggregate and concept mappings are then ingested by ROAD, and processed with cSPADE to extract cooccurring alert signatures. To reduce processing overhead, a constrained database is constructed using the analyst supplied critical signatures. Only aggregates containing the critical signatures are processed by cSPADE, with additional post processing done to the results to account for the aggregates not included in the SDB. Sequences and rules are then parsed, processed and tabulated before being presented to the analyst.
While this processed can be manually executed by an analyst at any time, it can also be run periodically to analyze potential changes in co-occurrence. The system can process any number of critical signatures specified by the analyst when it is invoked. 4.1
After being processed by CLEAR, ROAD analyzes the sequenced alerts to extract co-occurring
signatures. The constrained SPADE (cSPADE) algorithm is used [
cSPADE analyzes a sequence data base (SDB) which is made up of a number of individual sequences S. Each sequence Si contains some number of events Si = ei;1; ei;2; :::ei;n with each event containing a number of items im 2 I that occur at a specific time tn. Each successive event in a sequence occurs at a time later than the previous event ei;a; ei;b; ta < tb. If multiple items occur at the same time they are stored in the same event ei;j = Ij ji1; i2; :::im: cSPADE processes all sequences in the SDB looking for any sub-sequences s S which occur at a frequency higher than a user designated minimum. A sub-sequence’s frequency, or support Sup(s); is the count of sequences it is found in divided by the total number of sequences in the SDB N .
Association rules can be mined from frequent sequences to derive potential relationships
between item occurrences [
Support = Sup(A) =
A
N Conf (A ! B) =
Sup(A ! B)
Sup(A) (1) (2)
A number of metrics have been used to analyze the “importance" of individual rules [
As noted in [
SDBs are further constrained by discarding any sequences not containing the critical signature. After being processed by cSPADE, the resulting support statistics are scaled to account for the unconstrained database. Figure 3 shows the boxplot of the overall size reduction of the SDB obtained when analyzing only aggregates containing the critical signature for all critical signatures found in the CPTC dataset. SDBs saw on average a 90.1% reduction in sequences when including only aggregates containing the signature.
Experiments were conducted over two unique sets of Suricata [
The second dataset was collected at the 2018 National Collegiate Penetration Testing
Competition (CPTC) [
Data streams were created based on “adversary" IPs. For RSOC this was done using the external IPs captured by the IDS system. In CPTC this was done using the known IPs given to each of the team members. By parsing the data in this way each stream can be most closely interpreted as a single attacker’s behavior, and is a common approach to parsing intrusion alerts in research.
The CPTC dataset resulted in 50 streams made up of nearly two hundred thousand alerts over the course of the day of the competition. The RSOC environment by comparison saw six hundred and twenty thousand alerts in its first day alone generated by over one thousand external IP addresses. Although the RSOC dataset experienced a larger overall number of alerts, they are spread over a large number of relatively short streams. On average, RSOC streams contain 378.9 alerts while CPTC streams contained 2097.6 alerts. It is very common in a real world scenario for an external IP to connect to a network and generate only a small number of alerts. In many cases these are false alarms raised by non-malicious use of the network. Even when malicious however, it is still not uncommon for an attacker to connect to a network with a unique IP address in order to conduct a short burst of activity. The CPTC competition by contrast provided individuals with unique IP addresses to be used for the ten hour competition. This results in a small number of very long streams, contrasting the RSOC dataset.
IDS signatures contained in the CPTC dataset were manually mapped by SOC experts to corresponding attack stages. For the following results, any alert mapped to the attack stages “arbitrary code execution, brute force credentials, command and control, data exfiltration, and privilege escalation" were treated as critical. 5.1
Table 1 shows the breakdown of critical signatures found in the CPTC dataset by attack stage. Of the 113 signatures present CLEAR-ROAD found at least one co-occurring signature for 71 (62.8%) of them. The final column of the table shows regularly co-occurring signatures, those that co-occur with the critical signature at a rate of at least fifty percent across the entire dataset.
Atk. Stage ARB. CODE EXE BRUTE FORCE CREDS COM. & CON.
DATA EXFIL PRIV ESC
The final two columns of the table list the counts of critical signatures also found in the RSOC database. Of the 30 signatures found in both datasets, a total of 14 had the same cooccurring signatures in both datasets. The command and control and privilege escalation critical signatures made up a majority of those that had similar co-occurring signatures across datasets. 5.2
Table 2 shows high level summary results for individual critical signatures from both datasets. Even at this high level, key insights can be made regarding the occurrence patterns of critical signatures to help determine where to focus. As expected, critical signatures are extremely rare in the set of all generated IDS alerts for both datasets, in most cases a critical alert occurs fewer than one in one hundred alerts. However even with this rarity these signatures appear in aggregates with a high number of unique signatures. Were an analyst to manually query the critical alerts and the signatures surrounding them it would still be quite difficult for them to determine which are truly co-occurring.
By comparison, even just high level results can provide immediate feedback to an analyst. The bottom four rows of the table highlight a group of critical signatures that co-occur with one another. These four signatures are all variations of alerts that notify specific configuration options in a PHP URI. It is clear that the occurrence of these signatures are related given their identical statistics. Most likely they are borne from an attacker sweeping the network for a number of potential PHP vulnerabilities. The signatures also co-occur with extremely high lift, indicating that they occur together in aggregates and can very rarely be found separate from one another.
To better explore the CLEAR-ROAD system and how it can impact an analyst’s ability to understand network activity, results are presented as case studies. The only assumption made in collecting results is that the analyst has a known collection of individual alert signatures that are considered “critical" and that this was provided to the system at run-time. In the presented experiments critical signatures were those categorized by SOC experts as “command and control," “privilege escalation," “arbitrary code execution" and “data exfiltration." Examples chosen best highlight certain findings regarding co-occurrence consistency but are not the only examples contained within the datasets. 6 6.1
The critical signature “CDERED" (GPL EXPLOIT CodeRed v2 root.exe access) was only observed in the CPTC dataset, but was an attack vector leveraged by a number of the teams giving good insight into pattern consistency across users. The CodeRed worm attempts to connect to random hosts in the hope of finding a Microsoft IIS web server. Figure 4 shows a selection of boxplots representing the IATs of alerts in the concepts from 7 teams containing the signature. Each of the two temporal “modes" could potentially see unique or additional correlated signatures possibly accounting for the differences in presentation of the critical signature across streams. Having this initial high-level context helps frame the analyst’s expectations as they delve deeper into the statistics presented regarding individual co-occurring signatures.
Table 3 shows statistics relating individual co-occurring signatures with the critical signature under analysis. The third column represents the count of aggregates containing the critical and co-occurring signature while direction columns indicates the order of occurrence between the two signatures. The lift column shows the average lift for all occurrences of the signature pairs while the IAT column shows the average arrival time between the critical and co-occurring signatures. The IATs and appearances of the co-occurring signatures match with the frequencies and timings of the concepts seen in Figure 4. There is a strong consistency in these co-occurring signatures with the critical signature independent of the source, but dependent on the timing. Thanks to the structure of the CPTC competition each individual attacker was targeting a copy of the same network. With the same configuration, the same action generated the same set of cooccurring signatures, even among any other alerts generated by other actions being taken within the network.
To fully understand the relationships between critical and co-occurring signatures and any potential attack vectors executed, analysis of the individual signatures and their causes are needed. The first group of co-occurring signatures “ROOTA" (GPL WEB_SERVER / root access), “MSAAC" (GPL EXPLOIT /msadc/samples/ access) and “ISAPIA" (GPL EXPLOIT ISAPI .idq access) co-occur with CDERED quickly, with IATs measuring in milliseconds. Signatures ROOTA and MSAAC are triggered by an attempt to access specific directories of an IIS server, the usual target of the CDERED worm. Successful access is what most likely lead to the code red exploit being deployed against the server, generating the critical signature shortly after the initial access. Signatures vary commonly followed the code red alert an indicates a successful buffer overflow on a IIS server. This could indicate the worm was successful in finding and exploiting an IIS server.
The second temporal behavior highlights a completely different attacker action. The signature “JEXBO" (ETPRO WEB_SERVER JexBoss Common URI struct Observed 2 (INBOUND)) relates to a java platform testing tool that has been used in ransomware attacks such as “SamSam." Jexboss is used in conjunction with web servers so it is not unreasonable to assume that it was used as a potential injection vector for the code red worm. About 30 seconds after the Jexboss signature, the code red signature was alerted followed by the “DTLEAK" (ETPRO WEB_SERVER Possible Information Leak Vuln CVE-2015-1648) which alerts of a potential data leak through the opening of a command terminal.
Figure 5 Illustrates the differences in timing between the found correlated signatures and a potential timing flow of the overall attack. Such a plot makes clear the average timings between critical and co-occurring signatures while also highlighting the temporal discrepancy between the two “modes" that CDERED is seen in in the network. Having the knowledge of these cooccurring signatures can help a SOC analyst to adjust their defenses to appropriately react when a ROOTA or JEXBO alert is generated with in the network to prevent future potential exploits The critical alert signature “CFADMN" (ET WEB_SERVER ColdFusion administrator access) occurred in both datasets with each experiencing very different appearance rates for co-occurring signatures. While both datasets saw “CFAPIA" (ET WEB_SERVER ColdFusion adminapi access), “CFUTIL" (ET WEB_SERVER ColdFusion componentutils access) and “DRUPAL" (ET EXPLOIT Possible CVE-2014-3704 Drupal SQLi attempt URLENCODE 1), they did not co-occur with CFADMN in the CPTC dataset with the same frequency as in RSOC. This may seem strange given that most of these co-occurring signatures are directly related to ColdFusion servers, but in fact highlights another strength of CLEAR-ROAD’s data driven processing. Just as different network topologies can cause unique timing characteristics within streams, so too can the maintenance and configuration of the infrastructure. The CFADMN alert is raised when the administrator of an Adobe ColdFusion web server is remotely accessed. This is simply one of a number of alerts that can be raised when a malicious entity is attempting to access a ColdFusion server. CFAPIA, CFUTIL and CFPWDA (ET WEB_SERVER ColdFusion password.properties access) all alert on different methods of gaining access to a ColdFusion server. If not configured properly, it is possible to retrieve the component utils page, the administrator page or the adminapi page of a ColdFusion server through a standard web call. It is possible for sensitive infomation, such as login credentials to be stored in plaintext in the component utils pages.
The discrepancy in co-occurrence numbers across datasets is most likely caused by effective security measures taken within the RSOC environment. ColdFusion servers in RSOC are most likely well updated and defended to avoid vulnerabilities that allow for data leaks through external accesses. Therefor most attackers will attempt to access the administrator, api, and component utils unsuccessfully. The CPTC competition however intentionally builds networks with vulnerabilities for teams to attack. It is likely that a ColdFusion server used in the competition could be accessed in this way, meaning teams did not need to attempt both CFUTIL and CFADMN accesses as frequently throughout the competition. Another indicator that the CPTC ColdFusion servers were poorly maintained is the “CFPWDA" (ET WEB_SERVER ColdFusion password.properties access) which exploits a vulnerability that provides the attacker with hashed administrator passwords. This exploit has been patched as of 2013, but it is not unlikely that such an out of date server was intentionally included in a penetration testing competition. Figure 6 illustrates the co-occurrences and timing for signatures related to CFADMN in both datasets.
Also interesting is the order of the three signatures across datasets. Since all three are independent vectors of attacking a ColdFusion server order does not truly matter, however there is a clear difference between the core approach used by CPTC teams and real world entities. Interestingly both datasets see alerts with the “DRUPAL" (ET EXPLOIT Possible CVE-2014-3704 Drupal SQLi attempt URLENCODE 1) co-occurring before CFADMN. The DRUPAL signature can alert to potential SQL injection attacks against Drupal 7 web servers. Drupal servers are not based on the same coding language as ColdFusion servers. Most likely attackers are targeting web servers using a script, this theory is given more evidence by the cooccurrence of the “NMAPSC" (ET SCAN Nmap Scripting Engine User-Agent Detected (Nmap Scripting Engine)) in CPTC and the “STREX" (ET EXPLOIT Apache Struts 2 REST Plugin XStream RCE (ProcessBuilder)) exploit commonly seen preceding DRUPAL in RSOC. Likely each network is being scanned for web servers with potential vulnerabilities which when found are targeted. The variation in ordering would indicate different scripts are being used in each network. 7
It is infeasible to expect an analyst to be able to manually query and process all instances of a specific critical alert in a timely manner. While some automated approaches to alert correlation exist they require extensive training sets requiring manual labeling and are not focused on providing fast, intuitive feedback to an analyst in real time. SOC analysts are interested in knowing which signatures are actively co-occurring with certain “critical" signatures within their networks. Given the high number of alert signatures and the rate at which alerts are generated, it is nearly impossible to expect an analyst to discover these co-occurring signatures manually.
The CLEAR-ROAD system is able to quickly and effectively provide this key information to analysts defending a network. A real world SOC operation’s alert data was processed and co-occurring signatures were found. Deeper analysis of the signatures produced gave strong supporting evidence towards the co-occurrences truly stemming from an attacker’s action. By first processing alert arrivals to learn the temporal behaviors within a stream, co-occurring signatures and patterns are given an additional dimension of context. As seen in the first case study some critical signatures are used in different ways resulting in unique temporal patterns and co-occurring signatures. CLEAR-ROAD was able to find consistent alert co-occurrences across streams and across datasets with unique alert timings and vastly different stream characteristics.
CLEAR-ROAD’s approach to SDB construction saw on average a 90.1 % reduction in the number of sequences processed with no impact to pattern or rule mining results. This allows for much lower processing times, providing an analyst with insights even quicker. Systems such as this that aim to provide new and unique perspective to SOC analysts are necessary. While automation of IDS systems and models is inevitable, the human element can never be fully removed from cyber defense. Developing and providing tools such as CLEAR-ROAD allow for smarter and more proactive defense. [11] P. Lenca, B. Vaillant, P. Meyer, and S. Lallich. Association rule interestingness measures: Experimental and theoretical studies. In Quality Measures in Data Mining, pages 51–76.
Springer, 2007.