=Paper=
{{Paper
|id=Vol-2009/fmt-proceedings-2017-paper16
|storemode=property
|title=Rule Creation in a Knowledge-assisted Visual Analytics Prototype for Malware Analysis
|pdfUrl=https://ceur-ws.org/Vol-2009/fmt-proceedings-2017-paper16.pdf
|volume=Vol-2009
|authors=Johannes Schick,Markus Wagner,Niklas Thür,Christina Niederer,Gernot Rottermanner,Paul Tavolato,Wolfgang Aigner
|dblpUrl=https://dblp.org/rec/conf/fmt/Schick0TNRTA17
}}
==Rule Creation in a Knowledge-assisted Visual Analytics Prototype for Malware Analysis==
https://ceur-ws.org/Vol-2009/fmt-proceedings-2017-paper16.pdf
Rule Creation in a Knowledge-assisted Visual
Analytics Prototype for Malware Analysis
Johannes Schick1 , Markus Wagner2 , Niklas Thür2 , Christina Niederer2 , Gernot Rottermanner2 ,
Paul Tavolato2 , Wolfgang Aigner2
St. Pölten University of Applied Sciences, Austria
Email: 1 dm171566@fhstp.ac.at, 2 first.last@fhstp.ac.at
Abstract—The increasing number of malicious software (mal- language using formal grammars (syntactic pattern recognition
ware) requires domain experts to shift their analysis process to- [5], [6] or for more details [7]). The task of the analyst is
wards more individualized approaches to acquire more informa- the development of a set of grammar rules incorporating their
tion about unknown malware samples. KAMAS is a knowledge-
assisted visual analytics prototype for behavioral malware analy- knowledge about (malicious) behaviors of malware samples.
sis. It allows IT-security experts to categorize and store potentially In this context, visual analytics (VA) is needed to support
harmful system call sequences (rules) in a knowledge database. the analysts in integrating their knowledge. VA plays an
To meet the increasing demand for individualization of analysis essential role in supporting data analysis, since it combines
processes, analysts should be able to create individual rules. data processing capabilities of computer systems with the
This paper is a visualization design study, which describes the
design and implementation of a Rule Creation Area (RCA) into knowledge and experience of users [8].
KAMAS and its evaluation by domain experts. It became clear According to Keim et al. [9], VA also connects automated
that continuous integration of experts in interaction processes analysis techniques with interactive visualizations in order to
improves the knowledge generation mechanism of KAMAS. combine different types of information and obtain understand-
Additionally, the outcome of the evaluation revealed that there ing from complex data sets. To make reasoning out of this
is a demand for adjustment and re-usage of already stored rules
in the RCA. massive amount of data, it is necessary to include ”implicit”
[10] or ”tacit” [11] knowledge in the analysis process. By
I. I NTRODUCTION externalizing the implicit/tacit knowledge of domain experts,
Nowadays, domain experts have to deal with an ever in- it is possible to provide explicit knowledge in form of data,
creasing number of malicious software (malware) which in which is independent from the current user of the system.
addition is becoming more targeted, persistent and unknown. This extracted knowledge can subsequently be connected
Malwares are able to disturb computer operations and gather through interactive visualization tools [11]. In addition to these
personal information of the system’s owner without raising findings, Lee et al. [12] stated that visualization is necessary
attention [1] [2]. When it comes to analyzing methods, there to analyze potential malware more effectively.
are two approaches for the identification of malware. On the This paper provides a design study [13] dealing with the
one hand, software can be analyzed without actually executing implementation and evaluation of a separate Rule Creation
it, which is called static analysis. Obfuscation techniques Area (RCA) [14] into a Knowledge-Assisted Visual Malware
used by malware developers can render this task virtually Analaysis System (KAMAS) [15]. In order to meet quality
impossible [3]. Dynamic analysis observes actions performed standards, this paper follows a problem-oriented research ap-
by potential malware while it is being executed in a protected proach. In conjunction with this prototype, this means that the
environment. More precisely, analysts observe execution traces visualization and implementation of the system is performed
of programs; for the sake of simplicity malware analysts under consideration of specific real-world problems defined by
often reduce these traces to function calls, neglecting all other domain experts [13]. Thus, the main objectives of this research
simple machine instructions. Therefore, dynamic analysis is are:
also known as behavior-based analysis [3]. • Clarify, why the implementation of a separate area for
In behavior-based analysis malware analysts have to deal rule creation is necessary and how malware analysts can
with large amounts of data, which can lead to a very complex benefit from it.
analysis process: a trace of a malware sample may often • Presenting the design and implementation of the RCA
comprise thousands of system calls and analysts have to find into the KAMAS system with a detailed description of
similar system call patterns within thousands of such traces. all involved components and functionalities.
In order to simplify this process, analysts need automated • Conducting an evaluation of the implemented system in
approaches for finding such patterns and categorizing them as cooperation with malware analysis experts to proof the
potentially harmful or harmless. However, such identification effectiveness of the deployed functionalities.
of patterns relies heavily on the analysts knowledge, which • Reflection of the implemented functionalities under con-
makes it impossible to automate this process completely sideration of their evaluation by real world users and the
[4]. These patterns of behaviors can be defined as a formal resulting future research.
116
Rule Creation in a Knowledge-assisted Visual Analytics Prototype for Malware Analysis
II. R ELATED W ORK AND BACKGROUND analysis system by Mohaisen et al. [22] is an example for
a program, which is capable of fulfilling this task. Just like
Since there were no interactive visualization tools available KAMAS, it tries to tackle shortcomings of existing systems
which cover all requirements for malware analysts, Wagner by combining methodologies of static and behavior-based
et al. [15] developed the KAMAS prototype. With KAMAS, approaches. By running malware samples in a virtualized
analysts are able to categorize function call traces in terms of environment, the system collects data which is subsequently
their potential harmfulness and store them into a knowledge used for automated classifying and clustering of samples into
database (KDB). The KDB assists them in further observation different malware families. However, AMAL does not provide
and simplifies the analyzing process. In order to expand the an interactive user interface, nor does it provide the possibility
range of functionalities and subsequently improve the effec- to integrate externalized expert knowledge into the analysis
tiveness of KAMAS, Wagner et al. [16] suggested an interface process. Another project with similar approach to KAMAS is
design for the RCA [14]. The RCA allows the construction of the visualization tool SEEM [23], which enables analysts to
completely new rules by using single system and API calls in compare large sets of malware and their associated attributes.
the same structure as generated by the sequitur algorithm [17]. As mentioned in Section I, supportive visualization is
These rules can subsequently be stored in the KDB. needed in order to provide a more efficient approach of
Knowledge Generation in VA: Thomas and Cook [18] define analyzing potential malware samples [12]. In their state of
VA as approach to gain knowledge from massive, dynamic, the art report on visualization systems in the field of malware
ambiguous, and often conflicting data. Based on the research analysis, Wagner et al. [24] concluded that future systems
by Sacha et al. [19] following findings can be determined: should provide a compound of classification overviews for
VA uses data to draw conclusions on a specific application comparison and detail views for individual analysis.
field and gain insights into the problem domain. On the one
hand, there is the combination of perceptive skills following III. M ETHOD
the capability of drawing conclusions by humans. On the In general, this paper is a design study following the
other hand, there is the computing and data storage capability design principles/ideas proposed in [13], which is described as
of machines. Both of these aspects can be combined in problem-orientated research approach. This includes a problem
visual representations. Interactions with VA tools provide a definition, the design and implementation of a visualization
possibility for analysts to detect patterns in their data, thus system which solves the problem, the evaluation of the proto-
assisting them in verifying or falsifying their initial hypothesis. type as well as a reflection about lessons learned and possible
By clustering and classifying the found patterns, the outcome improvements [13]. The problem was defined by Wagner et
of their exploration can be visualized. al. [16] in their design study, which addresses the need for
According to Chen et al. [10], the aim of knowledge- the implementation of a separate area for rule creation in the
assisted visualization is to automate reasoning about abstracted KAMAS prototype. All scientific publications directly related
information from a set of data. Additionally, they also men- to KAMAS [4] [16] [15] [24] served as a basis for the general
tioned that the field of knowledge-assisted visualization is still understanding of the prototype and its background.
in development, since the growing amount of data requires Requirements & Features: The functionalities and interface
systems to continuously adapt to these challenges. As stated by design of the RCA were designed according to the rule build-
Wang et al. [11], the nature of tacit knowledge can be defined ing screen prototype ’CallNet’ presented in [16]. Furthermore,
as intimate and specialized. However, by using interactive ’CallNet’ and its desired functionalities were already reviewed
visualization it can be connected with explicit knowledge, by usability experts. In order to ensure compatibility with the
which is processable by computers or can be stored in a sequitur algorithm [17], the extracted knowledge has to be
database [11]. Recent research has focused even more on stored in a rule-based interface and structure. Based on the
the role of humans in this process. In order to deal with task definitions and the outcome of the design study, following
the increasingly ambitious challenges in the field of VA, key requirements (R) for the RCA can be defined:
the philosophy has to shift from a ”human in the loop” R1 Consistency: To ensure an effective usage of the user
philosophy to a ”human is the loop” viewpoint [20]. This new interface, it is necessary to provide consistent interac-
approach focuses on recognizing the workflow of analysts and tion techniques throughout the whole system. In this
consequently adapting interaction processes to the needs of the specific case, the input data for the RCA originates
analyst. from another interface section of the program. There-
Appliance of VA Techniques to Malware Analysis: Accord- fore, the interaction visualization should be related to
ing to Alazab et al. [21], all executable programs have the aim the movement of data, e.g. ’Drag & Drop’ operations.
to perform actions using API calls. The process of malware R2 Creation Support: The amount of data offered by this
analysis involves the observation of system call sequence system is particularly high. Thus, additional support
patterns and the actions they cause. Both Alazab et al. [21] and in the process of rule creation is important. By
Mohaisen et al. [22] emphasize classification and clustering of giving the analyst additional interaction possibilities,
patterns in terms of their maliciousness or benignity as a main e.g. automatically validated suggestions for single
task for malware analysts. AMAL, a behavior-based malware calls, the rule creation process can be accelerated.
117
Rule Creation in a Knowledge-assisted Visual Analytics Prototype for Malware Analysis
Moreover the interface has to provide the possibility access information of already stored rules like the name, the
to switch the highlighting of these calls based on assigned concept and the calls it consists of.
higher or lower frequency to support creating rules RCA in General: The RCA (see Figure 1.2) generally consists
with individual preferences. of three main areas. First, the analyst can drop single calls,
R3 Editing Options: The process of rule creation requires which he previously selected and dragged from the ’Call
the system to allow the editing of rules at any time Exploration’ table into the Rule Creation Table (RCT) (see
and to offer a quick way to restart the process. As Figure 1.2.b). Secondly, above and below the RCT, the inter-
a consequence, the analyst has to be able to reorder face provides suggestions for single calls which occur either
and delete single calls of the dropped call sequence before (see Figure 1.2.a) or after (see Figure 1.2.c) the dropped
and to reset the whole RCA to its default state. system call sequence. At last, on the bottom of the RCA the
R4 Knowledge Extraction and Extension: Finally, the analyst has the possibility to reset the whole RCA to its default
newly created rule should be used to extend the state (see Figure 1.2.d) and to switch the highlighting of the
spectrum of computerized knowledge in the system. call suggestions (see Figure 1.2.e).
Therefore, it is necessary to offer the possibility Rule Creation Table in the RCA: After adding the first call
of moving rules from the RCA to the KDB. By from the ’Call Exploration’ table to the RCT, an additional
implementing this functionality, the knowledge gen- row gets added on the top of the table. This row makes
eration loop (see Figure 2) can be expanded, which it possible to drag the newly created rule (which contains
should subsequently improve the effectiveness of the all single calls inside the RCT) and add it to the KDB.
analysis process. Furthermore, the number in the second column of the RCT
The features of the RCA were implemented according to represents the occurrence of the newly created rule in the
the defined key requirements. In general, the design and loaded file. If there is a need to reorder calls inside the RCT,
implementation followed a user-centered design process [25]. this can be achieved by simply dragging a single call and move
During the development process, continuous exchange with it to the desired position. The original call from the desired
researchers/developers of the KAMAS prototype was per- position then switches position with the dragged call. Also,
formed. Thus, it was possible to adjust requirements and single calls can be deleted from the RCT by right clicking on
discuss alternative solutions. the desired call and using the ’Delete’ pop-up. It must also
Evaluation: In order to evaluate the implemented features be pointed out that every interaction performed in the RCT
of the prototype, two malware analysis experts reviewed the affects the occurrence column and call suggestions, since these
system in the course of a semi-structured, qualitative user test. components depend on the values inside the RCT. Even though
During this test, both experts had to solve different tasks, a rule usually can contain 1 to n calls, the maximum number
which occur in the rule creation process. The results were of calls inside the RCT was limited to eight calls in order to
documented by written notes and afterwards categorized based provide enough space for the other areas in the RCA.
on their importance. Afterwards, the results of the evaluation Call Suggestions in the RCA: Above and below the RCT,
were summarized and rated in a list inspired by Nielsen’s the interface offers suggestions for calls, which can be dragged
severity rating procedure [26]. With these ratings, it was and dropped into the RCT. The suggestions above (see Fig-
possible to provide a clear overview of the most important ure 1.2.a) represent calls from the loaded file which occur
findings as well as potentially negligible aspects. before system call sequences with the same structure as the
one inside the RCT, whereas the calls below (see Figure 1.2.c)
IV. D ESIGN AND I MPLEMENTATION represent calls which occur after the currently dropped system
The design and functionalities of the RCA are based on call sequence. Moreover, the font size of the call suggestions
the ’CallNet’ prototype [16], which allows users to create varies depending on their occurrence. By default, more fre-
rules from scratch with system and API calls. The KAMAS quent single calls are displayed with a bigger font. If a single
prototype and its implemented functionalities developed by call appears in multiple system call sequences of the loaded
Wagner et al. [15] served as a basis to expand the prototype’s file, the font size increases by one for every found similar
spectrum of features. The implementation resulted in the single call. Thus, every call suggestion displayed in the user
realization of the RCA (see Figure 1), which was achieved interface is unique and the analyst gets a better overview of
using the programming language Java. which single calls are more or less frequent.
Call Exploration: The ’Call Exploration’ table (see Fig- Control Buttons in the RCA: Provided that the RCT contains
ure 1.3) provides a list of all system and API calls of the loaded at least one single call, a ’Reset’ button (see Figure 1.2.d) is
file showing their occurrence in the file, the name and the ID available at the bottom of the RCA. This button offers the
of the call. In addition to the already available functionalities possibility to set the whole area back to its default state. The
from [15], the possibility to drag single calls from the table second button (see Figure 1.2.e) is responsible for handling the
to the RCA was implemented. highlighting of the call suggestions and is only visible when
KDB: The KDB (see Figure 1.1) offers the possibility to the currently dropped system call sequence offers suggestions.
save and organize rules in tree structure based concepts With the use of this button, the analyst can switch between
visualized as folder structure. Furthermore, the analyst can highlighting more or less frequent call suggestions.
118
Rule Creation in a Knowledge-assisted Visual Analytics Prototype for Malware Analysis
Fig. 1. User interface of the KAMAS prototype with activated RCA. 1) The KDB where newly created rules can be stored. 2) The RCA including the call
suggestions before (2.a) and after (2.c) the currently dropped call sequence, the rule creation table (2.b), the button to reset the whole RCA (2.d) and the
button to change the call suggestion size according to their occurrence (2.e). 3) The ’Call Exploration’ table with a list of all single calls included in the
currently loaded file.
Filter Pipelines for Call Suggestions: Depending on the process was limited to use either preprocessed rules or their
currently dropped system call sequence, the call suggestions included single calls. Thus, the analyst was not able to change
get validated through one general and two separated filter patterns like e.g. the order of included single calls inside a
action pipelines. The general pipeline loops through all rules rule. Following the implementation of the RCA, the analyst
of the loaded file and eliminates every rule which does not can now switch to the ’Rule Creation’ screen and create own
include the exact same system call sequence as in the RCT. rules from scratch. In the beginning, the analyst can explore
In the next step, the remaining rules serve as input data for and select specific calls from the single call table and drag
the call suggestion validation which is finally displayed above them into the RCA. After the desired calls were added to RCT,
and below the RCT. The first pipeline loops through every the interface offers suggestions for calls which occur before
remaining rule and checks if there is another single call before and after the currently dropped call sequence. These calls can
the first one of the sequence. If so, this call is displayed as also be used in the further creation process by dragging them
call suggestion above the RCT in the user interface. The same into the RCT. Additionally, it is possible to highlight either
applies to the second pipeline, except that it extracts the single more or less frequent call suggestions by increasing their font
call after the last one of the sequence which is subsequently size. This can be achieved by clicking the ’Enlarge less/more
displayed below the RCT. frequent calls’ button. The number next to the calls inside the
Usage Scenario: First, the analyst loads a new file into the RCT represents the occurrence of the currently dropped rule
system and KAMAS automatically provides an overview of all in the analysis file. During the whole process, the analyst can
included single calls in the ’Call Explorer’ (see Figure 1.3) as adjust the created rule by reordering calls inside the RCT via
well as all preprocessed rules in the ’Rule Explorer’. The ’Rule drag and drop or deleting unnecessary calls via right clicking
Explorer’ serves as a graphical summary and exploration area on the desired call and using the ’Delete’ pop-up. Finally,
with colored highlighting of all included rules depending on the rule can be dragged at the top of the RCT and moved
the current knowledge state of the KDB. If the analyst wants to the KDB. Afterwards the RCA can be reset to its default
to store one of these rules in KDB, this can either be achieved state by clicking the ’Reset Rule Creation Area’ button. The
by selecting the full rule or just specific parts of this rule and analyst can now return to the ’Rule Exploration’ screen and
drag and dropping it to the KDB. Further functionalities linked continue the analysis with an updated KDB containing the
to the ’Rule Explorer’ are described by Wagner et al. [15]. In newly created rule.
the initial version of the KAMAS prototype, the rule storing
119
Rule Creation in a Knowledge-assisted Visual Analytics Prototype for Malware Analysis
Fig. 2. Comparison of the Knowledge Generation Loop before and after the implementation of the RCA. Left: Knowledge Generation Loop of the initial
KAMAS prototype visualized by Wagner et al. [15]. Right: Knowledge Generation Loop including rule creation process. The range of possibilities for the
analyst has increased.
V. T HE K NOWLEDGE G ENERATION L OOP VI. E VALUATION
As third step in this design study [13], it was necessary to
Initial Knowledge Generation Loop: Wagner et al. [15]
evaluate the newly implemented functionalities with real world
provided an overview of the knowledge generation process in
users. Therefore, a formative usability evaluation including a
the KAMAS system (see Figure 2 on the left). It highlights the
usability test with malware analysis experts was conducted and
KDB, which contains all known rules and the data exploration
finally compressed to the most important findings.
through the analysts as central elements in the knowledge
generation loop. By extending the KDB with new rules, the A. Method
system automatically revalidates the highlighting in the ’Rule
Exploration’ area depending on the new knowledge state. The TABLE I
rules used for the extension of the KDB are automatically OVERVIEW OF THE DOMAIN EXPERTS WHO PARTICIPATED IN THE USER
generated and displayed in the ’Rule Exploration’ table. TEST. (E = EXPERT )
Extension of Knowledge Generation Loop: With the im- Person Age Gender In field Experience Education
plementation of the RCA, the possibilities for knowledge E1 30-39 male 5 years expert MSc
generation have expanded. The usage scenario now includes E2 60-69 male 6 years advanced PhD
the following process (see Figure 2 on the right): The analyst
can load a new file, which is going to be checked automatically Participants: For this user test, two malware analysis experts
against the stored data in the KDB (see Figure 2.1). After- (see Table I) were invited to test and comment all function-
wards, the system provides a visualization of the loaded rules alities of the RCA. Both experts had more than five years of
in comparison to its current knowledge state in the system experience in this field and were also part of previous KAMAS
(see Figure 2.2). In contrast to the initial KAMAS prototype, case studies. Therefore, both of them were familiar with the
the analyst can now choose between two different options to general appearance and functionality of the user interface.
expand the KDB. The analyst can either use automatically Design and Procedure: In the beginning, the experts received
generated rules from the ’Rule Exploration’ table (see Fig- a brief overview about the main functionalities of the RCA. In
ure 2.3 and Figure 2.7), or create own rules, which can include order to provide a realistic testing environment, an execution
presently unknown sequences based on the system’s current trace sample was provided and loaded into the system. Then,
state (see Figure 2.8 for the creation and Figure 2.9 for the the experts were asked to test each possible feature and to
knowledge generation). As a result, the KAMAS prototype is speak out their thoughts on the user interface.
not limited to rely on automatically generated rules anymore, Apparatus and Materials: Both case studies were performed
but rather offers more flexibility by providing an opportunity in a seminar room. To guarantee the testing of all possible
to create rules from scratch based on individual needs. features, a questionnaire based on the functionalities of the
120
Rule Creation in a Knowledge-assisted Visual Analytics Prototype for Malware Analysis
TABLE II
OVERVIEW OF EVALUATED FEATURE REQUESTS , SEVERITIES AND EFFORT (FR: 1: = NICE TO HAVE , 2: = GOOD FEATURE , 3: = ENHANCES USABILITY;
SE: 0: = NO PROBLEM , 1: = COSMETIC PROBLEM , 2: = MINOR PROBLEM , 3: = MAJOR PROBLEM , 4: = CATASTROPHE ; E FFORT: 1: = MIN , 2: = AVERAGE ,
3: = MAX ).
Description Feature Request (FR) Severity (SE) Effort
Call Exploration: Change selection mode to single selection 2 3 1
Call Exploration: Display only the last part of single call names 2 3 1
RCT: Provide a clear visualization of the drop location 3 3 2
RCT: Add arrow symbols to visualize the possibility of reordering 1 1 1
RCT: Display only the last part of single call names 2 3 1
Enlarge-Button: Change ’Enlarge’ to ’Highlight’ 1 1 1
Call Suggestions: Display current highlighting status in separate label 1 1 1
Call Suggestions: Change ’before/after’ to ’which appear before/after’ 1 1 1
Call Suggestions: Change ’scheme’ to ’sequence’ 1 1 1
KDB: Implement possibility to use known rules in the RCA 3 3 3
RCA: Display name of rule if it is already stored in the KDB 2 3 2
RCA: Implement a separate save button 1 2 2
Connection lines: Use logical elements 1 0 2
system was provided. The comments stated by the experts Deletion of Single Calls and Reset of RCA: Both experts
were documented by notes on the printed questionnaire. were able to delete single calls from the RCT and reset of the
whole RCA to its default state without any uncertainty.
B. Results Adding Created Rule to KDB: Both experts expressed their
wish for a possibility to drag rules from the KDB into the
Moving Single Calls to the RCT: E2 mentioned that it RCA. Additionally, the RCA should display the name in a
was difficult to find the desired calls, since the names of the label when editing an already created rule. In order to provide
displayed single calls were not fully readable due to lack of an alternative for the drag and drop approach, E1 suggested
space in the ’Call Exploration’ table. He suggested to display to implement a separate save button for the storing of rules
only the last part of the names because this part mostly into the KDB.
differs from other single call names. Both of them showed General Exploration: Both experts were pleased with the
uncertainty regarding whether if it is possible to move more general appearance of the user interface. They found the func-
than one call at the same time or not. Consequently, they tionalities to be valuable and the interface easy to understand.
were not sure if their desired calls were correctly dropped Furthermore, the simplicity of the user interface was rated
into the RCT. E2 suggested to change the selection mode of positively.
the ’Call Exploration’ table to single selection when the RCA
is activated in order to avoid misconceptions. C. Rating
Moving Call Suggestions to the RCT: While they tried to Based on the experts’ comments, the exploration results
move the calls to the RCT, both of them were confused by the were combined and rated in a list of the most important issues
drop location inside the RCT. They stated that they were not (see Table II). The rating procedure in this list is inspired
able to recognize a drop scheme, which led to the assumption by Nielsen’s severity ratings [26]. It includes a description of
that the calls get randomly added to the table. the issue, feature requests (FR), severities (SE) as well as the
Reordering of Calls Inside the RCT: E1 did not recognize associated effort for the solution of the issue. The conducted
the possibility of reordering in the first place and suggested to rating is illustrated in Table II.
add arrow symbols near the table cells of the RCT in order to Summary: The conducted evaluation showed that the im-
make it more clear. Furthermore, E2 mentioned that long call plemented functionalities were well received by the domain
names could get cropped off inside the table cells. Thus, the experts, although there are still certain improvements to con-
reordering of multiple calls with the same structure could be sider for the future. By rating found issues and suggested
difficult to recognize. improvements (see Table II), it was possible to determine
Change Highlighting of Call Suggestions: Both experts had major areas for further development of the current prototype.
problems understanding the naming convention of the button.
Since the word ’Enlarge’ is present in both versions of this VII. L IMITATIONS
button, E2 expected the call suggestions displayed above and Following the evaluation by malware analysis experts, cer-
below the RCT to extend to the follow-up call. Furthermore, tain limitations in the scope of functionalities for this prototype
he suggested to change the description linked to the call can be determined:
suggestions to ’Calls which appear before/after dropped call Adjustment of Stored Rules: This prototype does not provide
sequence’ to clarify the meaning. E1 added to show the current a possibility to drag already stored rules from the KDB into
highlighting state in a separate label to provide a more clear the RCA. However, the workflow of malware analysts also
appearance for the user. includes the manual adaption of already found rules [4]. With
121
Rule Creation in a Knowledge-assisted Visual Analytics Prototype for Malware Analysis
the implementation of this feature, it would be possible to the whole process from scratch. A particularly interesting
cover all essential needs of malware analysts and subsequently outcome of the evaluation was that both experts expressed their
improve the analysis process even more. wish for reusing/adjusting already stored rules in the RCA.
Displaying of Rule Names: The process of rule creation can This aspect was not taken into account during the development
also lead to a situation, where analysts are constructing rules, of the current prototype version. After the evaluation, it can
which are already stored in the KDB. However, the RCA is not be considered as highly recommendable to implement this
able to recognize already known rules and consequently does feature.
not provide the rule name in the user interface. By enabling R4 Knowledge Extraction and Extension: The possibility to
the RCA to check the currently constructed rule against the drag the newly created rule and store it in the KDB was also
KDB and subsequently recognize known rules, the workflow well received by the experts. As mentioned in Section IV,
of analysts can be enhanced by e.g. preventing the storage of the implementation of this feature expands the knowledge
duplicates in the KDB. generation loop (see Figure 2). Analysts are now more flexible
Creating Rules with More than Eight Single Calls: The when it comes to the extension of the KDB. By providing
RCA offers the possibility to create rules with up to eight the possibility to create individual rules based on the experts
single calls. Nevertheless, rules can contain much more single current state of knowledge, the prototype shifts towards the
calls in reality. Since the RCA also provides call suggestions in the beginning mentioned ”human is the loop” philosophy
above and below the RCT, the capacity of space in the RCA is [20]. As a result, the overall knowledge generation process is
rather limited. To overcome this, for example Focus+Context getting more individualized and the following analysis process
and/or aggregation techniques could be applied. can draw upon different expertises.
Drop Location Visualization: As mentioned in Section VI, Lessons Learned: In the course of this design study, it became
the RCA does not provide a visual preview of the currently clear that the continuous integration of domain experts in
dragged single call in the RCT. Since both experts were interaction processes enhances the efficiency of the analysis
struggling with this issue, the implementation of a visual procedure. As the number of malware families is growing,
preview of the dragged single call would have enormous higher importance has to be attached to the integration of
potential for improving the quality of the user interface. expert knowledge [24]. At the same time, VA techniques have
to adapt to the need for more human integration in the analysis
VIII. R EFLECTION AND C ONCLUSION process [20]. With the implementation of the RCA based on
In order to complete the methodology of Sedlmair et al. the interface design prototype by Wagner et al. [16], both
[13], this section focuses on the reflection of the combined previously mentioned challenges were tackled. In cooperation
results emerging from the design and implementation of the with malware analysis experts, the implemented prototype was
prototype and its evaluation by real world users. The require- proven to enhance the knowledge generation process and to
ments (R1 - R4) described in Section III were omnipresent handle the need for increasing focus on human interactions
during all steps in this design study and serve as point of in VA. However, the evaluation also revealed that interaction
reference for the following reflection. visualization is a key factor for providing a satisfying solu-
R1 Consistency: In order to stick to the defined requirements, tion. Additionally, it showed that there are still possibilities
drag and drop operations served as the major interaction to improve the knowledge generation process. Subsequently,
technique in this prototype. This involves the addition of single humans could be even more integrated into the previously
calls and call suggestions to the RCT, the reordering of calls mentioned knowledge generation loop. In this system, the
inside the RCT as well as the storing of the created rule in the knowledge is stored based on the same rule structure as they
KDB. Both analysts were comfortable with the handling of the are generated by sequitur [17]. But the storage of knowledge
given interaction possibilities. However, the evaluation showed depends on the structure of the underlying data. Thus, also
that additional visualization is needed to make the outcome of value ranges or process structures can be used.
drag and drop operations fully transparent. Future Work: The usage of already stored rules for rule
R2 Creation Support: As mentioned in the beginning, ana- creation can be seen as the next logical step for further
lysts have to deal with a large amount of data during the explo- development of the presented prototype. Additionally, the
ration process. Therefore, the implemented prototype provides enhancement of interaction visualization should round off
call suggestions to accelerate and simplify the rule creation the overall appearance and usability of the user interface.
process. Based on the currently dropped call sequence, the In general, further exchange with malware analysis experts
previously described filter pipeline (see Section IV) validates should be taken into account in order to stay on track with the
the displayed call suggestions automatically. Additionally, the developments in the scene.
prototype offers a possibility to highlight more or less fre-
quent call suggestions, which assists analysts in their decision ACKNOWLEDGMENTS
making process. This work was supported by the Austrian Science Fund
R3 Editing Options: To ensure editability during the rule (FWF) via the “KAVA-Time” project (P25489-N23). We
creation process, the prototype provides possibilities to delete would also like to thank all focus group members and test
and reorder single calls in the RCT as well as a button to restart participants who have agreed to volunteer in this project.
122
Rule Creation in a Knowledge-assisted Visual Analytics Prototype for Malware Analysis
R EFERENCES [21] M. Alazab, S. Venkataraman, and P. Watters, “Towards Understanding
Malware Behaviour by the Extraction of API Calls,” in 2010 Second
[1] T. Micro, “Addressing big data security challenges: The right tools for Cybercrime and Trustworthy Computing Workshop, 2010, pp. 52–59.
smart protection,” US: Trend Micro, 2012. [22] A. Mohaisen, O. Alrawi, and M. Mohaisen, “AMAL: High-fidelity,
[2] E. Gandotra, D. Bansal, and S. Sofat, “Malware Analysis and Classi- behavior-based automated malware analysis and classification,” Com-
fication: A Survey,” Journal of Information Security, vol. 05, no. 02, puters & Security, vol. 52, pp. 251–266, 2015.
p. 56, 2014. [23] R. Gove, J. Saxe, S. Gold, A. Long, and G. Bergamo, “SEEM: A
[3] M. Egele, T. Scholte, E. Kirda, and C. Kruegel, “A Survey on Automated Scalable Visualization for Comparing Multiple Large Sets of Attributes
Dynamic Malware-analysis Techniques and Tools,” ACM Computing for Malware Analysis,” in Proceedings of the Eleventh Workshop on
Surveys, vol. 44, no. 2, pp. 6:1–6:42, 2008. Visualization for Cyber Security, ser. VizSec ’14. New York, NY,
[4] M. Wagner, W. Aigner, A. Rind, H. Dornhackl, K. Kadletz, R. Luh, USA: ACM, 2014, pp. 72–79.
and P. Tavolato, “Problem Characterization and Abstraction for Visual [24] M. Wagner, F. Fischer, R. Luh, A. Haberson, A. Rind, D. A. Keim, and
Analytics in Behavior-based Malware Pattern Analysis,” in Proceedings W. Aigner, “A survey of visualization systems for malware analysis,” in
of the Eleventh Workshop on Visualization for Cyber Security. ACM, Eurographics Conference on Visualization (EuroVis) - STARs, R. Borgo,
2014, pp. 9–16. F. Ganovelli, and I. Viola, Eds. Cagliari (Sardinia / Italy): The
Eurographics Association, 2015, pp. 105–125.
[5] K. Fu, Syntactic pattern recognition and applications, ser. Prentice-Hall
[25] H. Sharp, Y. Rogers, and J. Preece, Interaction Design: Beyond Human
advanced reference series: Computer science. Prentice-Hall, 1982.
Computer Interaction. John Wiley & Sons, 2007.
[6] R. Gonzalez and M. Thomason, Syntactic pattern recognition: an intro-
[26] J. Nielsen, Usability Engineering. Morgan Kaufmann Publishers Inc.,
duction. Addison-Wesley Publishing Company,Reading, MA, 1978.
1993.
[7] H. Dornhackl, K. Kadletz, R. Luh, and P. Tavolato, “Malicious Behavior
Patterns,” in IEEE International Symposium on Service Oriented System
Engineering, 2014, pp. 384–389.
[8] E. Kandogan and U. Engelke, “Agile Visual Analytics in Data Science
Systems,” in IEEE International Conference on High Performance
Computing and Communications; IEEE International Conference on
Smart City; IEEE International Conference on Data Science and Systems
(HPCC/SmartCity/DSS), 2016, pp. 1512–1519.
[9] D. Keim, J. Kohlhammer, G. Ellis, and F. Mansmann, Mastering the
Information Age Solving Problems with Visual Analytics. Eurographics
Association, 2010.
[10] M. Chen, D. Ebert, H. Hagen, R. S. Laramee, R. v. Liere, K. L. Ma,
W. Ribarsky, G. Scheuermann, and D. Silver, “Data, Information, and
Knowledge in Visualization,” IEEE Computer Graphics and Applica-
tions, vol. 29, no. 1, pp. 12–19, 2009.
[11] X. Wang, D. H. Jeong, W. Dou, S.-W. Lee, W. Ribarsky, and R. Chang,
“Defining and applying knowledge conversion processes to a visual
analytics system,” Computers & Graphics, vol. 33, no. 5, pp. 616–623,
2009.
[12] D. Lee, I. S. Song, K. J. Kim, and J. h. Jeong, “A Study on Malicious
Codes Pattern Analysis Using Visualization,” in International Confer-
ence on Information Science and Applications, 2011, pp. 1–5.
[13] M. Sedlmair, M. Meyer, and T. Munzner, “Design Study Methodology:
Reflections from the Trenches and the Stacks,” IEEE Transactions on
Visualization and Computer Graphics, vol. 18, no. 12, pp. 2431–2440,
2012.
[14] J. Schick, M. Wagner, N. Thür, C. Niederer, G. Rottermanner,
P. Tavolato, and W. Aigner, “Supporting knowledge-assisted rule cre-
ation in a behavior-based malware analysis prototype,” in Poster of the
14th Workshop on Visualization for Cyber Security (VizSec), Phoenix,
Arizona, USA, 2017.
[15] M. Wagner, A. Rind, N. Thür, and W. Aigner, “A knowledge-assisted
visual malware analysis system: Design, validation, and reflection of
kamas,” Computers & Security, vol. 67, pp. 1–15, 2017.
[16] M. Wagner, A. Rind, G. Rottermanner, C. Niederer, and W. Aigner,
“Knowledge-assisted rule building for malware analysis,” in Proceedings
of the 10th Forschungsforum der österreichischen Fachhochschulen, FH
des BFI Wien. Vienna, Austria: FH des BFI Wien, 2016.
[17] R. Luh, G. Schramm, M. Wagner, and S. Schrittwieser, “Sequitur-based
inference and analysis framework for malicious system behavior,” in
Workshop for Formal Methods in Software Engineering (ForSE), 3rd
International Conference on Information Systems Security and Privacy
(ICISSP), SCITEPRESS Digital Library. Porto, Portugal: SCITEPRESS
Digital Library, 2017, pp. 632–643.
[18] J. J. Thomas and K. A. Cook, “A visual analytics agenda,” IEEE
Computer Graphics and Applications, vol. 26, no. 1, pp. 10–13, 2006.
[19] D. Sacha, A. Stoffel, F. Stoffel, B. C. Kwon, G. Ellis, and D. A. Keim,
“Knowledge Generation Model for Visual Analytics,” IEEE Transactions
on Visualization and Computer Graphics, vol. 20, no. 12, pp. 1604–
1613, 2014.
[20] A. Endert, M. S. Hossain, N. Ramakrishnan, C. North, P. Fiaux,
and C. Andrews, “The human is the loop: new directions for visual
analytics,” Journal of Intelligent Information Systems, vol. 43, no. 3,
pp. 411–435, 2014.
123