Analyzing data collected in empirical visualization experiments is a time consuming task. In order to support visualization designers analyzing the data collected during such experiments, we intend to develop an analysis model. This analysis model will automatically choose appropriate visual analysis methods according to an analysis task to evaluate a new visualization technique. We will contribute an idea how this analysis model can look like, and what steps need to be taken to define the model. Furthermore, we investigate which visual analysis methods are available or have to be developed first, and how analysis tasks can be defined.
Information visualization has the goal to represent information data in a
graphical way to uncover concealed inner relationships by offering suitable visual
representations. To evaluate if a new information visualization technique supports a
user to uncover these relationships user experiments can be conducted [
The analysis process evaluating data collected in empirical visualization experiments is often a time consuming task. Furthermore, visualization designers often don’t have the skills to conduct and analyze such experiments. Therefore, we contribute a concept for an analysis model which will support visualization designers to analyze collected data in an experiment. This analysis model will be based on analysis tasks and visual analysis methods.
An analysis task is derived from a research question, to specify how this
research question can be validated. For example, if an empirical visualization
experiment wants to compare two visualization techniques with each other and
find out why one visualization technique can be used to answer a task faster,
a potential analysis task could be to investigate the “overall spatial pattern of
[eye] movements” [
Visual analysis methods are visualization techniques designed or adapted
to represent experimental data, such as for example eye movement data. The
“Time Radar Tree” visualization technique developed by Burch et al. [
Therefore, our work on an analysis model aimt at answering the following research question: What are appropriate visual analysis methods for analysis tasks required to evaluate empirical visualization experiments? To answer this question, this paper will outline how an analysis model could look like and what steps need to be taken to develop such an analysis model. 2
Visualization research can be segmented into scientific visualization and
information visualization. Scientific visualization uses data from domains such as
biology, engineering, or meteorology, and is often inherently spatial. Information
visualization maps abstract data to a spatial domain, such as for example data
from social networks, or business data [
In this paper, we will present an outline of an analysis model to support visualization designers during the evaluation of an empirical visualization experiment. Our analysis model (cf. middle block of Figure 1) uses input from three different data categories: information about the experimental design, the experiment conduction, and the matching model. These three data categories will be described in more detail in the following.
The experimental design is shown in the upper block in Figure 1 and can be classified by an experimental categorization based on the visualization technique evaluated, the experimental design method, the research question, and the data collection methods used. We will only evaluate visualization techniques from information visualization, such as node-link-diagrams, or time radar trees. Data collection methods can be interviews, cases studies, surveys, focus groups, data collections using eye tracking, or else. The research question investigated has to be defined by a user.
Possible data collected during the experiment conduction is shown in the left block in Figure 1. This data will be automatically collected and should be available in a machine readable format. The data types will depend on the data collection method defined in the experimental design and can for example be eye tracking data, participant data, benchmark data such as completion times and accuracy rates, and interaction data.
The most important part of our analysis model is the matching model in the lower block in Figure 1. It contains possible analysis tasks as well as visual analysis methods. In the matching model each analysis task will be matched to one or multiple visual analysis methods.
The analysis model itself furthermore consists off a decision model and of the analysis process. The decision model suggests appropriate analysis tasks, and visual analysis methods based on the information from the experimental design. It will also create a chosen visual analysis methods based on the collected experimental data. The analysis process is the part where user is included and can give feedback. We have defined the following steps for the decision model (DM) and the analysis process (AP). 1. DM: Based on the information from the experimental design the decision model suggests multiple appropriate analysis tasks to evaluate the research question. 2. AP: The user chooses one analysis task he wants to investigate. 3. DM: Based on the decision of the user, several visual analysis methods are suggests appropriate for the analysis task. 4. AP: The user chooses one of the offered visual analysis methods. 5. DM: The chosen visual analysis method is created by the decision model using the collected empirical data. 6. AP: The created visual analysis method is used to interrogate the research question. 7. AP: Go back to step 1 or 3 and repeat until results are enough.
Creating this analysis model requires answering the following subquestions. For each subquestion we propose to take the following steps, which will be discussed further in section 3.
– Which empirical visualization experiments are being used to evaluate visualization techniques and how can they be classified? → Section 3.1: Categorization of empirical visualization experiments. – Are there general analysis tasks for analyzing data collected in an empirical visualization experiment? → Section 3.2: Definition and evaluation of analysis tasks. – Which visual analysis methods are available to analyze collected data in an empirical visualization experiment? → Section 3.3: Definition and evaluation of visual analysis methods. – Which visual analysis methods are appropriate for each analysis task and how can visual analysis methods be mapped to analysis tasks in a general way? → Section 3.4: Creation and validation of matching process for analysis tasks with visual analysis methods – Which new visual analysis methods first have to be developed? → Section 3.5: Creation and evaluation of new visual analysis methods. – Is there a general analysis model to evaluate empirical visualization experiments and how can this look like? → Section 3.6: Creation and validation of the analysis model. 3
In the following, our methods and procedures are described for each step defined in detail in section 2. 3.1
Psychology differentiates between true experiments, quasi experiments, and non experiments. Data collection methods can for example be interviews, case studies, focus groups, or data collection using eye tracking. The first step, to find an analysis model for empirical visualization experiments will be to find categories for empirical visualization experiments, and to classify those depending on the experimental method, the data collection method, the type of visualization technique investigated, as well as the research question investigated. This categorization is necessary, to find appropriate analysis tasks. It will include a review of conducted empirical visualization experiments. 3.2
Based on the categorization of empirical visualization experiments appropriate analysis task will be derived. This is based on related work, as well as on expert reviews with visualization designers conducting experiments. These expert reviews are necessary to understand different types of visualization techniques, their goals and tasks, in order to infer appropriate analysis tasks. Evaluation of analysis tasks will be performed by visualization designers by conducting a user experiment, where participants have to match analysis task to types of visualization techniques. 3.3
As a next step, an investigation of available visual analysis methods will be conducted. Here participants from the field of psychology or human-computer interaction who have conducted user experiments will be interviewed which visual analysis methods exists or which visual analysis methods are missing for analyzing empirical visualization experiments. 3.4
After defining analysis tasks, and collecting appropriate visual analysis methods the matching process will be developed, based on matching guidelines, which will be to be defined first. This matching process will be evaluated in a user experiment where participants have to either match tasks and methods themselves or pairs of tasks and methods will be shown, and participants have to decide on the usefulness of this match. Visualization, psychology, and human-computer interaction experts will be used as participants for the experiment. 3.5
Once the matching process is defined, and experts have been interviewed about the visual analysis methods, missing visual analysis methods will be developed. A new visual analysis method will then be evaluated in a user experiment to investigate its usability, as well as how good tasks and goals intended can be solved. 3.6
The last step is to combine the parts of the experimental methods, the analysis tasks, and the visual analysis methods into an analysis model. This includes the definition of guidelines how the decision model uses the input data, and how it interacts with the user. The analysis model will first be developed based on an exemplary workflow for one type of empirical visualization experiment to show how the model operates. This workflow will be evaluated in a case study with an appropriate visualization domain. After showing that the model works for one visualization domain further domains are added and evaluated. 4
In this section related work of analysis tasks, and visual analysis methods will be presented and discussed. Analysis tasks mainly focus on tasks in combination with the analysis of eye tracking experiments. 4.1
Andrienko et al. [
Relation to display content or structure; Individual search strategy; – Spatial pattern of attention distribution; Relation of attention foci to display content or structure; Repeated visits; – Relation of movements to particular AOIs; Returns to previous points; Places where users have difficulties; – Connections between AOIs; Presence and frequency of repeated moves; – Comparison of trajectories; – Comparison of spatial patterns of movements of different user groups; – Comparison of spatial patterns of attention of different users or user groups; – Comparison of spatial patterns of movements on different displays; – Comparison of attention distribution on different displays; – Evolution of eye movements over time; General search strategy; Types of activities and their temporal order; – Changes of attention distribution over time; – Frequent/typical sequences of attending AOIs; Cyclic scanning behavior. 4.2
Visual analysis method from the eye tracking domain have been developed. The
most prominent are heat map and scanpath visualizations which will be briefly
presented in the following. Other visualizations techniques have been developed
to meet different requirements of different application domains, such as for
example the circular heat map transition diagram by Blascheck et al. [
Scanpath: A scanpath visualizes the fixation data of each participant
individually. This visualization technique shows the complete eye movement path of one
participant, or multiple participants by using different colors. Each fixation is
indicated by a circle where the radius of this circle is based on fixation duration,
saccades are visualized by lines connecting these fixation circles [
To illustrate how our analysis model can be used in an empirical visualization experiment we will describe the evaluation a real eye tracking study from the information visualization domain.
We will use the study described by Burch et al. [
Collected data in this study was the eye tracking data for each participant. The participant data contained information about mathematical backgrounds, video gaming skills, and soccer interests, as well as gender and age. The completion times and answers were also collected during the experiment for each question.
Analyzing this experiment using our analysis model, and using the defined
steps in section 2, the first step is to get the information about the experimental
design. In this experiment the experimental design method is a true
experiment with a between-subject design, as participants were split into two groups
randomly, one for each visualization tool. The data collection method used eye
tracking, and questionnaires to collect eye movement date, completion times,
and accuracy rates. The visualization technique is a radial visualization type
compared to a Cartesian representation. One possible research question in this
experiment could be to find out why “Time Line Trees” can be used to answer
counting questions faster than “Time Radar Trees”. To examine this question
the analysis task “Overall spatial pattern of [eye] movements; relations to display
content or structure” from section 4.1 could be chosen. This analysis task would
require a visual analysis method showing all eye movements of one participant,
which could be a scan path visualization of each participant. Other potential
visual analysis methods might be better suited to investigate this analysis task,
like the parallel scan paths by Raschke et al. [
In this paper we investigated if an analysis model can be created to support visualization designers in evaluating their empirical visualization experiments. We formulated how a potential analysis model could look like by introducing the concept of analysis tasks and visual analysis methods. The appropriate visual analysis methods and analysis tasks will be chosen according to the experimental design, and experiment conduction. In an application example we illustrated how the analysis model can be used to investigate an empirical visualization experiment. We further defined the appropriate methods and procedures that need to be used to realize our analysis model. The implementation of this model will be future work.