=Paper=
{{Paper
|id=Vol-2578/BigVis10
|storemode=property
|title=A task-based evaluation methodology for visual representation of dynamic networks
|pdfUrl=https://ceur-ws.org/Vol-2578/BigVis10.pdf
|volume=Vol-2578
|authors=Pablo Camarillo-Ramirez,Francisco Cervantes-Alvarez,Luis Fernando Gutiérrez-Preciado
|dblpUrl=https://dblp.org/rec/conf/edbt/Camarillo-Ramirez20
}}
==A task-based evaluation methodology for visual representation of dynamic networks==
https://ceur-ws.org/Vol-2578/BigVis10.pdf
A task-based evaluation methodology for visual
representation of dynamic networks
Pablo Camarillo-Ramirez Francisco Cervantes-Alvarez Luis F. Gutiérrez-Preciado
Western Institute of Technology and Western Institute of Technology and Western Institute of Technology and
Higher Education Higher Education Higher Education
Tlaquepaque, Jalisco, Mexico Tlaquepaque, Jalisco, Mexico Tlaquepaque, Jalisco, Mexico
ng724453@iteso.mx fcervantes@iteso.mx lgutierrez@iteso.mx
ABSTRACT of EDA tasks might be the waste of computing resources or
Current evaluation approaches for visualization strategies of dy- user time to perform analysis tasks. In this work we present an
namic networks are focused on maintaining the mental map of evaluation methodology focussed on the usability of tools that
the network over the time or keeping a certain shape to make it support EDA tasks with dynamic networks.
easy to navigate, however the available tools for analyzing tempo- The rest of this paper is organized as follows. In the Section 2
ral network have not been evaluated in terms of how easy to use we explore works that inspired the development of the method-
they are to perform exploratory data analysis tasks with dynamic ology proposed in this work. We describe briefly EDA and how a
networks. In this work we present an evaluation methodology connected data structure can be useful to perform this kind of
that guides the usability assessment of software tools used to an- analysis. In the Section 3 we provide details about the methodol-
alyze dynamic networks by using the standard ISO 9241-11. This ogy proposed. In the Section 4 it is shown how the methodology
methodology has been applied successfully with two popular described is used to evaluate the usability of Gephi and Cytoscape
open source tools used to analyze temporal networks. for temporal tasks on dynamic networks. Finally in the Section 5
we conclude about the advantages and improvements needed to
KEYWORDS the methodology proposed based on the results presented.
dynamic networks, graph drawing, usability, drawing evaluation
2 RELATED WORK
1 INTRODUCTION 2.1 Exploratory Data Analysis with dynamic
Due to its impact on business and data analysis, the analysis networks
of networks has become one of the most prominent research The exploratory data analysis consists on finding answers to
areas in recent years. The function of a network is to represent numerous questions about data [2]. To obtain these answers
links between entities, revealing the structure and nature of re- analysts use mainly software tools. In [22] authors conclude
lationships in data. Network visualization is one of the main that EDA is about hypothesis generation rather than hypothesis
means of exploratory graph analysis [24] and it has becomes testing. This definition, however, does not take into account the
relevant for business when network visualization supports the questions that analysts may have and the process to solve them.
decision-making process[5]. For those problems with connected Another well-known work that defines EDA is the Information
data which is represented as network, a good visual representa- Seeking Mantra by Ben Shneiderman [19] that generalizes the
tion is highly required to perform successfully exploratory data EDA process into three steps: (1) Overview first, (2) filter, and
analysis (EDA). To determine whether a network drawing tech- then (3) details-on-demand. In summary, this definition indirectly
nique is good or not, several approaches have been proposed states that EDA is the process to find what items are interesting
such as those approaches focussed on characteristics of network and deserve further examination. According to Andrienko and
layout [7], [8], [10], clusters in graph [17] or network shape [6]. Andrienko [2], visualization systems are frequently employed to
All of these strategies are focussed on visualizing static networks support EDA tasks.
only. Another type of networks that recently are becoming rele-
vant in the EDA field are those that changes over time, known
as dynamic networks. The most common ways to visualize a 2.2 Task taxonomies for temporal EDA tasks
dynamic network includes animations, timeline of changes or A task can be understood as an entity formed by two compo-
a hybrid visualization [4]. For these kind of networks, most of nents: target and constraints. A target refers to the unknown
the existent evaluation strategies are focused on preserving the information to be obtained, and the constraints points out to the
mental map over the time [23],[3]. As a matter of fact, Beck et known conditions that system needs to fulfill; a task therefore
al. [4] conclude that most of these evaluation approaches are not involves finding a target given a set of constraints.
necessarily involving users, hence the motivation of this paper According to [13] task taxonomies play a vital role in the
to propose a user-centred evaluation methodology rather than design and evaluation of visualization systems, because they re-
network structure or aesthetics properties. veal and categorize the application needs. This categorization
The importance of evaluating the usability of a software lies on supports the process to design a system that provides an appro-
managing the potential risks that can arise from inappropriate priate visual representation of a dynamic network to complete
outcomes of interaction. For instance, an undesired outcome exploratory tasks.
© 2020 Copyright for this paper by its author(s). Published in the Workshop Proceed-
There are many works explaining different aspects of an ex-
ings of the EDBT/ICDT 2020 Joint Conference (March 30-April 2, 2020, Copenhagen, ploratory task on a static network. Lee et al. [14] define a graph vi-
Denmark) on CEUR-WS.org. Use permitted under Creative Commons License At- sualization task taxonomy and classified the tasks as: (1) Topology-
tribution 4.0 International (CC BY 4.0)
based (adjacency, accessibility, common connection, connectivity)
EDBT 2020, March 30-April 2, 2020, Copenhagen, Denmark P. Camarillo-Ramirez, F. Cervantes-Alvarez, and L. F. Gutiérrez-Preciado
(2) Attribute-based (On the nodes and On the links), (3) Brows- rows and columns and a colored intersection encodes an edge.
ing (Follow path and Revisit) and (4) Overview, a compound The approaches discussed in this section are concentrated on the
exploratory task to get estimated values quickly. mental map preservation by using node-link diagrams.
Shneiderman and Aris [20] define a task taxonomy of networks One of the most used criterion to determine whether a visual
as a collection of task associated to (1) Basic networks (unlabeled representation algorithm of a network is good or not is if it can
nodes and undirected links), (2) Node/Link labels, (3) Directed preserve the mental map. The intention of the mental map preser-
networks, and (4) Node/Link attributes. vation is to keep the network layout over time in order to offload
Along with these entities, authors propose a list of tasks specif- the cognitive effort required to comprehend the information con-
ically associated to basic networks (count number of nodes, com- tained in the network [3].
pute degree for every node, find betweenness centrality, etc.), but One of the works focussed on maintaining the mental map of a
they conclude there are an unlimited number of tasks that could temporal network is the Hyperbolic temporal layout proposed by
be defined. Cengiz and Balcisoy [23] that represents the evolution of relations
On the other hand, for temporal analysis, Yi et al. [25] propose among network actors and structural patterns of a social network.
a task classification that visualization techniques should support On the other hand, Archambault and Purchase [3] have con-
to perform temporal social network analysis (TSNA): temporal ducted some experiments focussed on the human factors in tem-
changes at the global level, temporal changes at the subgroup poral network drawing rather than algorithmic considerations.
level and temporal associations among nodal and level attributes. They found that preserving the mental map is not always helpful
when performing tasks on dynamic networks.
2.2.1 A task taxonomy for network evolution analysis. For
temporal analysis of networks, analysts are interested in three
different targets: entities, properties, and temporal features. Con-
2.4 Usability evaluation
straints are the (limited) resources such as display size or I/O The term usability can be understood as the software capability
devices used to perform exploratory tasks [1]. Entities include of being used. One of the most important benefits of having a
node/link, group or network. The properties include both struc- software highly usable might be a little time on performing a
tural properties and domain attributes. Finally, temporal features task.
consist of those features that answer the question about the net- We can distinguish two approaches that might help us to
work’s evolution. In fact, Jae-wook et al. [1] take these three di- outline the evaluation methodology proposed in this paper: A
mensions to define a design space 1 to formulate a task taxonomy consolidated model called Quality in Use Integrated Measurement
for temporal networks. This design space and some examples of (QUIM) proposed by Seffah et al. [18] and the standard ISO 9241-
temporal tasks are shown in the Table 1. 11 [11].
2.2.2 A task taxonomy for temporal graph visualization. An- 2.4.1 Quality in Use Integrated Measurement. The model de-
other taxonomy proposed for tasks on a temporal network is scribed in [18] includes 10 usability factors: (1) Efficiency, (2)
presented in [13]. This approach covers not only temporal net- Effectiveness, (3) Productivity, (4) Satisfaction, (5) Learnability,
works, but also static networks, multivariate graphs, and graph (6) Safety, (7) Trustfulness, (8) Accessibility, (9) Universality, and
comparison. The main idea of that work is to extend the An- (10) Usefulness. These factors are decomposed into 26 sub-factors
drienko framework [2]. The Andrienko framework consists in which are further-decomposed into 127 specific usability met-
data model and task framework. The task framework applies the rics. Authors proposal included an editor tool 2 that supports the
task definition previously mentioned (targets and constraints). activities to obtain usability measurement. Unfortunately this
The data model identifies the data items that might participate as editor is not longer available.
target or constraint. However, one of the main limitations of the
2.4.2 ISO 9241-11. The aforementioned model was inspired
Adrienko’s framework is that it does not consider graph data. For
by analyzing several standards, frameworks and models previ-
example, the information of an edge is difficult to model under
ously proposed. One of these standards is the ISO 9142-11 [11].
the data model presented by such framework. The extension
This standard measures the usability of a software (or hardware)
proposed by Kerracher et al. [13] includes the structural tasks
in terms of efficiency, effectiveness and satisfaction in a context
that considers the questions associated to relational tasks for the
of use. The context of use can be understood as the users, tasks
networked data.
equipment (software and materials), and the physical and social
environment in which a product is used.
2.3 Evaluation approaches of dynamic
network visualization 2.5 Software tools and libraries to visualize
In this section we will discuss some of the most popular strate- networks
gies to analyze the quality of the visual representation of the
2.5.1 Cytoscape. Cytoscape3 is an open source software for
dynamic networks. We can distinguish two main approaches
visualizing complex networks. It is a software developed by Cy-
to evaluate visualization systems for dynamic networks: those
toscape Consortium and it is founded by the U.S. National In-
focussed on the importance of maintaining the mental map and
stitute of General Medical Sciences (NIGMS). Its main goal was
those concentrated on profiling the visualization in terms of net-
to offer a tool for biological research, however nowadays it is
work structure or layout. The most common way to visualize
a general tool for complex network analysis and visualization.
a network is by using a node-link diagram to represent entities
The architecture of Cytoscape offers the capability to increase
and their connections. Another way to visualize a network is
functionalities by developing adding plugins. Currently there
by using adjacency matrices where the nodes are represented as
1 A design space is a multidimensional combination and interaction of input variables 2 http://rana.cs.concordia.ca/odusim
3 https://cytoscape.org
and process parameters that have been demonstrated to provide assurance of quality.
A task-based evaluation methodology for dynamic networks EDBT 2020, March 30-April 2, 2020, Copenhagen, Denmark
Table 1: Design space of temporal taxonomy proposed by Jae-wook et al. [1]
Entities
Node or Link Group Network
Individual Single Occurrences Examine Network’s Clustering Coefficient
temporal
Birth or Death Find when the tendency #freebiefriday appears
features
Replacement Find when changes the in-degree of #firdayfeeling tendency
Growth & Contraction Observe the Network’s growth (forward)
Temporal
features Shape of changes
features Observe if the Clustering Coefficient converges at some
Convergence & Divergence
time point
Compare the stability states between starting and end-
Stability
ing point
Observe the repeated relationship between tendencies
Repetition
#fridaymotivation and #fridayfeeling
Observe the Clustering Coefficient peaks or valleys for
Peak or Valley
the entire network
Rate of changes Fast & Slow Observe the speed of tendencies creation
features
Accelerate & Decelerate Identify the acceleration for tendencies creation
are ten available apps in the Cytoscape marketplace under the 2.5.5 ReGraph. Part of the suite provided by Cambridge Intel-
Network dynamics category. ligence, ReGraph 5 is a library of React components and analysis
functions for client-side network visualization.
2.5.2 Gephi. Gephi [16] is another open software tool useful
to explore and understand graphs. It is an interactive visualiza- 3 EVALUATING VISUALIZATION OF
tion and exploration platform for many kinds of networks and DYNAMIC NETWORKS
complex systems, dynamic and hierarchical graphs. The goal is to
help data analysts to form a hypothesis, intuitively discover pat- For EDA with temporal networks, only the experiments described
terns, isolate structure singularities or faults during data sourcing. in [3] take into account the user experience of a visual repre-
Its last version supports visualize dynamic networks by using a sentation of temporal networks. These experiments are focussed
continuous representation of connected data. on the importance of the mental map preservation for dynamic
graph drawing. We propose a new methodology based on ISO
2.5.3 NTDV. The Network Dynamic Temporal Visualization 9241-11 to evaluate the usability of a visualization system in
[21] is a package for language R to visualize dynamic networks. terms of effectiveness, efficiency and satisfaction in a context of
Its last version was released on May 2019 and it provides capabil- use. This methodology can be summarized as follows:
ities to analyze and visualize networks such as birth, death, and • Establish the context of use: (1) obtain or generate the
reincarnation of objects in the network over time. It supports time-evolving network in format required by the software
discrete and continuous representation for time, which allows to tool to be evaluated, (2) Define a subset of EDA tasks that
visualize many kinds of datasets with temporal connected data. the software tool should be capable to perform, (3) select
The NDTV package generates network movies or interactive a group of users or analysts that should complete the EDA
HTML5 animations, timelines and other visualizations ways of tasks, and (4) fix the layout algorithm that will be observed
dynamic networks. every time slice.
• Analyze EDA tasks selected: (1) measure time every user
2.5.4 KeyLines. KeyLines 4 is a SDK developed by Cambridge takes to complete the task (if he/she does), (2) apply a sat-
Intelligence company for building web applications to perform isfaction questionnaire after finishing every one of these
network visualization. One of the main features of this SDK tasks, and (3) compute Effectiveness and Efficiency met-
is the capability that offers to manage dynamic networks with rics.
its time bar. With this time bar, users can filter data by time
and date, observe network evolution and perform any EDA task. 3.1 Effectiveness of dynamic network
Another key feature of KeyLines is the map mode that enables the visualization tools
functionality to visualize networks on maps, and thus perform
spatial analysis. The effectiveness metric can be obtained by using the completion
rate equation 1. In our context, given an EDA task, it is asked to
a set of analysts to complete the task under same conditions. The
4 https://cambridge-intelligence.com/keylines/ 5 https://cambridge-intelligence.com/regraph/
EDBT 2020, March 30-April 2, 2020, Copenhagen, Denmark P. Camarillo-Ramirez, F. Cervantes-Alvarez, and L. F. Gutiérrez-Preciado
more EDA tasks are completed, the higher is the effectiveness time points and thus all tendencies created or connected with a
score for this task. shared timestamp are observed in the same time point. With this
approach it is possible to generate a dynamic network from this
N sample of tweets. In the Figure 1 it is shown the static data model
E f f ectiveness = (1)
T of the network that is being visualized.
Where N represents the number of tasks completed success-
fully and T stands for the total number of tasks undertaken.
3.2 Efficiency of dynamic network Is_Related_To
visualization tools
One of the main motivations to evaluate the usability of the
current software tools that support EDA tasks is to measure :Tendency :Tendency
the time employed to complete an EDA tasks with a dynamic
network. Said that, we compute the efficiency of a software tool
for dynamic network analysis in terms of the time needed to Figure 1: Data model of the dataset
complete a task. In the equation 2 if is shown how the Efficiency
can be calculated.
ÍR Í N n i j 4.1.2 Define a subset of EDA tasks. Based on the task taxon-
j=1 i=1 t i j omy for network evolution analysis [1] we define the next subset
E f f iciency = (2)
NR of tasks as part of the context of use for the usability tests. This
Where N is the number of tasks, R is the number of users, if taxonomy has been selected to perform the case study because
the user successfully completes the i − th task ni j = 1 otherwise of its clear categorization of tasks and the number of examples
ni j = 0 and ti j represents the time spent by j −th user to complete provided by the original authors. The tasks selected are the in-
the i − th task. tersection of those tasks that can be completed by using the two
tools we are evaluating in these study:
3.3 User satisfaction of dynamic network
(1) BD01: Determine the time point when the tendency #free-
visualization tools biefriday appears (Birth\Death)
The strategy suggested to assess the user satisfaction is to apply (2) GrCtr01: Observe the Network’s growth (Growth & Con-
the ASQ questionnaire [15] after completing every EDA task. traction)
This questionnaire surveys the user satisfaction in terms of task (3) GrCtr02: Observe the Network’s contraction (backward)
difficulty, time spent to complete the task and usefulness of the (Growth & Contraction)
documentation provided by the software to complete the task.
What we propose is to change the original 7-point scale to a 4.1.3 Select a group of analysts. The users selected to com-
5-point scale because after expose the original questionnaire to plete the EDA tasks are people that is involved (or interested)
some users, they suggested us to reduce the number of options. in network analysis. Specifically, the population selected is in-
The 5-point scale resultant is as follows: terested on analyzing networks that changes over time. Eleven
users performed the EDA tasks in the given context of use.
(1) Strongly agree
(2) Agree 4.1.4 Fix the layout algorithm. For every tool it was fixed a
(3) Neutral different layout algorithm. For Cytoscape it was fixed the Kamada-
(4) Disagree Kawai [12] algorithm and for Gephi it was fixed the Frunchter-
(5) Strongly disagree man Reingold algorithm [9].
4 CASE OF STUDY WITH CYTOSCAPE AND 4.2 Analyze EDA tasks
GEPHI Once established the context of use, the core of the study is the
In order to show how the methodology proposed can be applied observation of the user experience on performing the aforemen-
we are going to evaluate two open source tools that support the tioned EDA tasks by using two different software tools.
EDA of temporal networks: [16] and Cytoscape. These tools were
4.2.1 Measure time to complete every task. The entire session
selected because both are open source projects and once evalu-
was recorded, from the begin of the tasks until the user was
ated they can be improved by the open source community itself.
notified that heRshe has completed the task. The goal of recording
The methodology proposed is a guide to obtain effectiveness,
every session is not only to measure the time spent but also to
efficiency and satisfaction in a context of use.
observe whether the user could or not complete the given task.
By observing the duration of the recording, it can be obtained the
4.1 Establish the context of use
time employed for every user to complete the task (See Figure 2)
4.1.1 Generate dynamic network. The dynamic network pre-
sented to the users represents the evolution over 99 minutes of a 4.2.2 Apply satisfaction questionnaire. After competing every
sample of 142 posts on Twitter 6 . Every node represent a tendency task, all users were asked to complete the questionnaire men-
or hashtag mentioned in the post: two tendencies are related or tioned in the section 3.
connected if they are mentioned in the same post. In order to
4.2.3 Compute metrics. By using the equations 2 and 1, the
add dynamics to this dataset, the timestamp is used to create the
Efficiency and Effectiveness metrics can be computed respec-
6 https://www.trackmyhashtag.com/historical-twitter-data tively.
A task-based evaluation methodology for dynamic networks EDBT 2020, March 30-April 2, 2020, Copenhagen, Denmark
Average Time Spent (in seconds)
70
60
50
(a) Observing network evolution with Gephi
40
BD01 GrCtr01 GrCtr02
EDA tasks performed
Gephi Cytoscape
Figure 3: Temporal analysis efficiency for Cytoscape and
Gephi
(b) Observing network evolution with Cytoscape Another way to interpret the results obtained is to analyze
the satisfaction results for every software tool independently.
Figure 2: Case of study to measure time spent on EDA For example, for EDA tasks performed by using Gephi, it can be
tasks observed a correlation between satisfaction expressed with the
time spent for every task and the how difficult users found every
task. When users expressed a positive experience (or neutral),
they also agree with the time spent to complete these tasks.
4.3 Results Finally, we can analyze the satisfaction results in terms of infor-
mation provided by the software interface to complete temporal
All users could complete successfully the three tasks analyzed
EDA tasks. If we compare the results obtained from Cytoscape
with both Cytoscape and Gephi. The task efficiency (with the
and Gephi, there is a notorious difference between the user satis-
given context of use) is 100%: all users completed the tasks with
faction between these two tools. For Cytoscape users expressed
a reasonable amount of time.
a neutral or positive experience. However, for Gephi at least 14%
For the task Efficiency, there is a clear difference between
of users expressed they strongly disagree with the information
Cytoscape and Gephi for all tasks . In general, it can be observed
provided to complete the analyzed tasks.
that Cytoscape is less efficient than Gephi: 43.24% for BD01,
33.52% for GrCtr01, and 11.46% for GrCtr02 task. In the Figure
it 3 can be observed the average time spent to complete the 5 CONCLUSIONS AND FUTURE WORK
analyzed tasks. The methodology presented in this work shows an effective way
Analyzing results obtained from the task that involves find- to evaluate software tools that supports EDA with temporal net-
ing when appears a specific tendency in the network’s timeline works based on the user experience. Results obtained shows that
(BD01), we can observe that 57% of users agree with the ease of the evaluation can be performed independently by analyzing
completing this task by using Gephi. For the same task, users correlations between satisfaction and effectiveness data. In addi-
spent in average 43.24% less time to fulfil the task with Gephi. tion to, this methodology can be used to compare two or more
This tendency is consistent with the rest of tasks and their satis- software tools and to guide the improvement process of them.
faction results. The set of tasks proposed in the taxonomies analyzed in Sec-
One interesting finding in the task that involves observing the tion 2 do not consider large graphs and we consider that a new
network contraction over time (GrCtr02). For the analyzed tasks, taxonomy (or extension) should be proposed to cover EDA with
this is the only task where users expressed a better satisfaction of large temporal networks. The future task taxonomy needs take
using Cytoscape. In terms of task difficulty, 57% of users consider into account the navigation capabilities offered by devices used
easy to complete the GrCtr02 task, meanwhile 43% disagree with to fulfil EDA tasks such as touch-screen devices.
the difficulty to complete this task with Gephi, even when the Regarding to the software that supports EDA, it should con-
average time to complete this task by using Gephi was 11% bet- sider that temporal tasks do not depend of a good animation. For
ter than the time spent with Cytoscape. Probably this results is instance, to analyze the shape of changes another visual com-
caused because the user interface of Cytoscape clearly shows the ponents like timeline charts are might be helpful. Actually, for
options to complete this tasks and Gephi requires more inputs to labeled graphs many visual tools are required to navigate, explore
get the same animation. and analyse successfully temporal data.
EDBT 2020, March 30-April 2, 2020, Copenhagen, Denmark P. Camarillo-Ramirez, F. Cervantes-Alvarez, and L. F. Gutiérrez-Preciado
Table 2: Satisfaction results for temporal tasks
BD01 GrCtr01 GrCtr02
Cytoscape Gephi Cytoscape Gephi Cytoscape Gephi
Strongly agree 0% 0% 29% 0% 57% 0%
Agree 14% 57% 29% 71% 29% 57%
Task
Neutral 29% 43% 14% 0% 0% 0%
difficulty
Disagree 43% 0% 29% 29% 14% 43%
Strongly disagree 14% 0% 0% 0% 0% 0%
Strongly agree 14% 14% 29% 29% 43% 14%
Time
Agree 29% 29% 43% 43% 29% 43%
spent to
Neutral 29% 57% 14% 0% 29% 29%
complete
Disagree 14% 0% 14% 29% 0% 14%
the task
Strongly disagree 14% 0% 0% 0% 0% 0%
Strongly agree 0% 0% 14% 0% 14% 0%
Information Agree 29% 29% 14% 71% 29% 43%
provided to Neutral 57% 29% 57% 14% 57% 29%
complete the task Disagree 14% 29% 14% 0% 0% 14%
Strongly disagree 0% 14% 0% 14% 0% 14%
ACKNOWLEDGMENTS of the 2006 AVI Workshop on BEyond Time and Errors: Novel Evaluation Methods
for Information Visualization (BELIV ’06). ACM, New York, NY, US, 1–5. https:
To those users and analysts that help us to complete the usability //doi.org/10.1145/1168149.1168168
tests. To the National Science and Technology Council of Mexico [15] James Lewis. 1991. Psychometric evaluation of an after-scenario questionnaire
for computer usability studies: The ASQ. SIGCHI Bull. 23 (01 1991), 78–81.
for the scholarship granted to the authors to develop this study. https://doi.org/10.1145/122672.122692
[16] Bastian Mathieu, Sebastien Heymann, and Mathieu Jacomy. 2009. Gephi: An
Open Source Software for Exploring and Manipulating Networks. In Proceed-
REFERENCES ings of the Third International ICWSM Conference (2009). The AAAI Press, San
[1] Jae-wook Ahn, Catherine Plaisant, and Ben Shneiderman. 2013. A task taxon- Jose, CA, US, 361–362. http://www.aaai.org/ocs/index.php/ICWSM/09/paper/
omy for network evolution analysis. IEEE transactions on visualization and view/154
computer graphics 20, 3 (2013), 365–376. [17] Amyra Meidiana, Seok-Hee Hong, Peter Eades, and Daniel Keim. 2019. A
[2] Natalia Andrienko and Gennady Andrienko. 2005. Exploratory Analysis of Quality Metric for Visualization of Clusters in Graphs. In Graph Drawing and
Spatial and Temporal Data: A Systematic Approach. Springer-Verlag, Berlin, Network Visualization, Daniel Archambault and Csaba D. Tóth (Eds.). Springer
Heidelberg, DE. International Publishing, Cham, 125–138.
[3] Daniel Archambault and Helen C. Purchase. 2013. The “Map” in the mental [18] Ahmed Seffah, Mohammad Donyaee, Rex Kline, and Harkirat Padda. 2006.
map: Experimental results in dynamic graph drawing. International Journal of Usability measurement and metrics: A consolidated model. Software Quality
Human-Computer Studies 71, 11 (2013), 1044 – 1055. https://doi.org/10.1016/j. Journal 14 (06 2006), 159–178. https://doi.org/10.1007/s11219-006-7600-8
ijhcs.2013.08.004 [19] B. Shneiderman. 1996. The eyes have it: a task by data type taxonomy for
[4] Fabian Beck, Michael Burch, Stephan Diehl, and Daniel Weiskopf. 2017. A information visualizations. In Proceedings 1996 IEEE Symposium on Visual
Taxonomy and Survey of Dynamic Graph Visualization. Comput. Graph. Languages. IEEE, Boulder, CO, US, 336–343. https://doi.org/10.1109/VL.1996.
Forum 36, 1 (Jan. 2017), 133–159. https://doi.org/10.1111/cgf.12791 545307
[5] Richard Brath and David Jonker. 2015. Graph analysis and visualization: [20] Ben Shneiderman and Aleks Aris. 2006. Network Visualization by Semantic
discovering business opportunity in linked data. John Wiley & Sons, Indianapolis, Substrates. IEEE Transactions on Visualization and Computer Graphics 12, 5
IN, US. (Sept. 2006), 733–740. https://doi.org/10.1109/TVCG.2006.166
[6] Peter Eades, Seok-Hee Hong, An Nguyen, and Karsten Klein. 2017. Shape- [21] T. Butts, Carter and Leslie-Cook, Ayn and N. Krivitsky, Pavel and Bender-
Based Quality Metrics for Large Graph Visualization. J. Graph Algorithms deMoll, Skye and Almquist, Zack . 2019. Network Dynamic Temporal Visual-
Appl. 21, 1 (2017), 29–53. ization. http://statnet.org
[7] Y. Frishman and Ayellet Tal. 2004. Dynamic Drawing of Clustered Graphs. In [22] John W. Tukey. 1977. Exploratory Data Analysis. Addison-Wesley, Reading,
IEEE Symposium on Information Visualization. IEEE, Austin, TX, US, 191–198. MA, US.
https://doi.org/10.1109/INFVIS.2004.18 [23] Uraz Cengiz Turker and Selim Balcisoy. 2014. A visualisation technique for
[8] Y. Frishman and A. Tal. 2008. Online Dynamic Graph Drawing. IEEE Trans- large temporal social network datasets in Hyperbolic space. Journal of Visual
actions on Visualization and Computer Graphics 14, 4 (July 2008), 727–740. Languages & Computing 25, 3 (2014), 227 – 242. https://doi.org/10.1016/j.jvlc.
https://doi.org/10.1109/TVCG.2008.11 2013.10.008
[9] Petr Gajdoš, Tomáš Ježowicz, Vojtěch Uher, and Pavel Dohnálek. 2016. A [24] Tatiana Von Landesberger, Arjan Kuijper, Tobias Schreck, Jörn Kohlhammer,
parallel Fruchterman–Reingold algorithm optimized for fast visualization of Jarke J van Wijk, J-D Fekete, and Dieter W Fellner. 2011. Visual analysis
large graphs and swarms of data. Swarm and Evolutionary Computation 26 of large graphs: state-of-the-art and future research challenges. Computer
(2016), 56 – 63. https://doi.org/10.1016/j.swevo.2015.07.006 graphics forum 30, 6 (2011), 1719–1749.
[10] T. E. Gorochowski, M. di Bernardo, and C. S. Grierson. 2012. Using Aging [25] Ji Soo Yi, Niklas Elmqvist, and Seungyoon Lee. 2010. TimeMatrix: An-
to Visually Uncover Evolutionary Processes on Networks. IEEE Transactions alyzing Temporal Social Networks Using Interactive Matrix-Based Vi-
on Visualization and Computer Graphics 18, 8 (Aug 2012), 1343–1352. https: sualizations. International Journal of Human–Computer Interaction 26,
//doi.org/10.1109/TVCG.2011.142 11-12 (2010), 1031–1051. https://doi.org/10.1080/10447318.2010.516722
[11] ISO. 2018. Ergonomics of human-system interaction – Part 11: Usability: Defini- arXiv:https://doi.org/10.1080/10447318.2010.516722
tions and concepts. Standard. International Organization for Standardization,
Geneva, CH.
[12] T. Kamada and S. Kawai. 1989. An Algorithm for Drawing General Undirected
Graphs. Inf. Process. Lett. 31, 1 (April 1989), 7–15. https://doi.org/10.1016/
0020-0190(89)90102-6
[13] N. Kerracher, J. Kennedy, and K. Chalmers. 2015. A Task Taxonomy for Tem-
poral Graph Visualisation. IEEE Transactions on Visualization and Computer
Graphics 21, 10 (Oct 2015), 1160–1172. https://doi.org/10.1109/TVCG.2015.
2424889
[14] Bongshin Lee, Catherine Plaisant, Cynthia Sims Parr, Jean-Daniel Fekete, and
Nathalie Henry. 2006. Task Taxonomy for Graph Visualization. In Proceedings