Especially in lifelong or professional learning, the picture of a continuous learning analytics process emerges. In this process, heterogeneous and changing data source applications provide data relevant to learning, at the same time as questions of learners to data change. This reality challenges designers of analytics tools, as it requires analytics tools to deal with data and analytics tasks that are unknown at application design time. In this paper, we describe a generic visualization tool that addresses these challenges by enabling the visualization of any activity log data. Furthermore, we evaluate how well participants can answer questions about underlying data given such generic versus custom visualizations. Study participants performed better in 5 out of 10 tasks with the generic visualization tool, worse in 1 out of 10 tasks, and without significant difference when compared to the visualizations within the data-source applications in the remaining 4 of 10 tasks. The experiment clearly showcases that overall, generic, standalone visualization tools have the potential to support analytical tasks sufficiently well.
Reflective learning is invaluable for individuals, teams and institutions to
successfully adapt to the ever changing requirements on them and to continuously
improve. When reflective learning is data-driven, it comprises two stages: data
acquisition and learning analytics. Often, relevant data is data about learner
activities, and equally often, relevant activities leave traces not in a single but
in multiple information systems. For instance, [
A picture of a continuous learning analytics process emerges, in which heterogeneous and ever changing data source applications provide relevant data for learning analytics, at the same time as questions of learners to data also continuously change. To support such a continuous analytics process, we have developed a generic visualization tool for multi-user, multi-application activity log data. In this paper, we describe the tool as well as the results of a task-based comparative evaluation for the use case of reflective workplace learning. The generic visualization tool integrates data from heterogeneous sources in comprehensible visualizations. It includes a set of visualizations which are not designed for specific data source applications, thus the term generic. It can visualize any activity log data published on its cloud storage. The only prior assumptions are that every entry in the data should be: i) time stamped and ii) associated with a user. The tool thus strikes a balance between generality (few prior assumptions) and specificity.
One key concern was whether the developed generic visualizations tools would be as comprehensible as those designed specifically for a given application or dataset. In this paper we describe an experiment comparing the performance of study participants along learning analytics tasks given the generic visualizations and visualizations custom-designed for the respective data. 2
Others before us have pointed out the need to collect and analyze data for
learning across users (multi-user) and applications (multi-application), both in the
community of learning analytics and open learner modeling: Learning analytics
measures relevant characteristics about learning activities and progress with the
goal to improve both the learning process and its outcome [
Visualizations in learning analytics and open learner modeling play the role
of enabling users (most often students, teachers, but also institutions or
policy makers - cf. [
We have developed a generic visualization tool for activity log data that addresses two fundamental challenges shared in many scenarios at the intersection of learning analytics, open learner modeling, and reflective learning on the basis of (activity log) data: Data from multiple applications shall be visualized; and at the time of designing the visualization tool, the concrete data sources and consequently the concrete analytic tasks are unknown. 3.1
We make only two assumptions about data, namely that they are i) time stamped
and ii) every data entry is associated with a user. The second assumption is useful
because in a lot of learning scenarios, learning is social [
The generic visualization tool (Vis-Tool) is implemented as a client-server architecture. It is a web application implemented in HTML5/Javascript and Java. The Vis-Tool itself does not capture any activity log data, but is responsible for the visualization of the data in a sophisticated and meaningful way. Through its server component, it is connected to a could storage that stores application data and manages access to data. The data source applications store their activity log data on the cloud storage in so-called spaces: Private spaces store data of only one user, while shared spaces collect data from multiple users. Single sign-on provides a common authentication mechanism for all data-source applications, the Vis-Tool and the cloud storage. The rationale behind this chosen architecture is to deal with data collection, data analysis and data visualization separately. The Vis-Tool expects data in an XML format described by a publicly available XML schema. In addition, the schema must extend a base schema that contains a unique ID for all objects, a timestamp and a user ID as mandatory fields. 3.3
The Vis-Tool organizes visualizations in form of a dashboard style similar to
[
The Vis-Tool provides four types of charts with different visual channels: geo chart, bar chart, line chart and timeline chart (see Figure 1).
The geo chart is used for data that contains geo positions. Besides the ”latitude” and ”longitude”, the chart consists also of the “popup header” and “popup text” as additional visual channels. Both of them are shown in a popup window when clicking on an entry. The bar chart is available for any structure of data. It contains the “aggregate” channel and the “operator” setting. While the “aggregate” channel defines, which data property should be aggregated, the “operator” defines how the data will be aggregated (count, sum, average, min max) in order to be displayed. The line chart contains “x-axis”, “y-axis”, and “label” (on hover text). It is available for data with numerical data properties. Our timeline chart is similar to the line chart but does not have an “y-axis” channel. All charts have the “group by” channel. It defines how data can be grouped with the help of colors. For example, if we use a user id to group the data belonging to one user, all data captured by this user will be presented with the same color. If several users are added to a group, all data captured by the users belonging to this group, will be presented with the same color. This feature makes it possible to easily spot user patterns across applications. 3.5
Users create charts in the Vis-Tool by selecting data they wish to visualize, selecting a chart type, and then filling the chart’s visual channels. The Vis-Tool, however, presents only those options to the user that are possible for any given data. Technically this is solved with chart matching and channel mapping. 3.5.1 Chart Matching For each chart type, we created a chart description consisting of a list of all channels, mandatory as well as optional, and the data types that the channels can visualize. At runtime, the XML schemas that describe the structure of the user data are parsed and the data properties including their data types are extracted. Based on the extracted data structures and the available channels per chart, chart matching is performed. The matching determines whether a chart is able to visualize a dataset described by a parsed schema.
This is done by checking for each channel of the chart, if the data structure has at least one property whose data type can be visualized by the given channel. For instance, the line chart consists of x-axis, y-axis as mandatory channels and the hover text as optional channel. The x-axis can visualize numeric values and date-time values and the y-axis can handle numeric values. The hover text channel is able to handle numeric values, date-times and strings. The line chart will be available for the parsed data structure, if the structure contains at least one numeric type or date-time for the x-axis and a numeric type for the y-axis. The hover text is an optional channel and therefore not of relevance for the decision, if the line chart is able to present the parsed data structure or not. For a given data structure, chart matching is performed for each chart type. Those chart types that match with the given data structures are added to the list of possible chart types and can be selected. 3.5.2 Channel Mapping Channel mapping takes place if a user selects one of the available charts. An initial channel mapping is automatically provided to the user when adding a chart to the visualization. Users can adapt the mapping of a property to another chart channel via the interface. 4
The Vis-Tool can be used in any use case in which analysis of multi-user and
multi-application activity log data makes sense. A lot of learning analytics and
open learner modeling use cases fall into this category, as argued above. The
task-based comparative evaluation that we subsequently describe and discuss in
this paper assumes a specific use case however. It is one of knowledge workers
who work in a team, carry out a significant amount of their work on desktop
PCs, and spend a significant amount of time traveling. In the sense of reflective
work-integrated learning [
Collaborative Mood Tracking - MoodMap App3 [
Example analysis on MoodMap data for workplace learning are to review and
reflect on the development of individual mood in comparison to team mood, and
in relationship to other events or activities that happen simultaneously.
PC Activity Logging - KnowSelf 4 [
Geo Tagged Notes - CroMAR [
In prior work, we explored the potential benefit of analyzing data from PC
activity logging data together with collaborative mood tracking data in such a
use case [
We performed an evaluation to compare custom visualizations in the data source applications (in-app) with generic visualizations (Vis-Tool). The goal was to establish how the comprehensibility of generic visualizations, designed without specific prior knowledge about (meaning of) data, compares to custom in-app visualizations that were customized for a specific kind of data and task.
5.1
We prepared a test dataset with data about three users, called D, L and S, containing two weeks of data from all applications. To do so, we extracted data from real usage scenarios of the single applications. For MoodMap, we selected two weeks of the three most active users out of a four-week dataset. For KnowSelf, we selected three two-week periods of log data out of a 7-month dataset from a single user. For CroMAR, we used the dataset from a single user who had travelled significantly in a two-seeks period, and manually created two similar datasets to simulate three users. The data were shifted in time so that all datasets for all applications and users had the same start time and end time. 5.2
The evaluation is intended to test the comprehensibility of generic visualizations
for learning analytics. We wanted to investigate how understandable are generic
visualizations compared to the custom visualizations that are specifically
designed for data of one specific application. Our initial hypothesis was that the
generic visualizations could be as meaningful as custom visualizations. As we
wanted to rule out confounding factors from different interaction schemes, we
opted to perform the experiment on paper based mock-ups. These were created
from the datasets by capturing screenshots of the in-app visualizations and the
generic ones generated with the Vis-Tool. We prepared short analytics tasks
(see Table 1) that participants should solve with the use of the given
visualizations. The tasks are plausible in line with the chosen use cases (see Section 4)
above, which were constructed based on use cases of knowledge workers that
were previously evaluated in their working environment [
The evaluation followed a within-participants design. For each tool (MoodMap
App (MM), KnowSelf (KS), CroMAR (CM)) we created a number of tasks
(MM=4, KS=4, CM=2). We created variants of each task with different datasets
for each condition (Vis-Tool, in-app). Thus, there were 20 trials per participant
(10 tasks in 2 conditions for each participant). Tasks and tools were randomized
across participants to avoid favoring either. We grouped the tasks by tool and
randomized the order of groups, the tasks within groups and the order of
condition (in-app visualization / generic visualization). The experimenter measured
the duration (time to completion) and real performance for each task.
Additionally, subjective scores of difficulty were measured through self-assessment using
the NASA-TLX workload measure [
The study followed the format of a structured interview, where the experimenter first explained the goals, the applications and the tasks participants would perform. The participant then proceeded to the first task, which finalized
T# App Task 1 MM On the given day, to whom did belong the worst single energy (arousal) and to whom did belong the single worst feeling(valence)? 2 MM On the given day, to whom did belong the worst average energy (arousal) and to whom did belong the worst average feeling (valence)? 3 MM Find out on which day in the two recorded weeks was entered the best energy(arousal) and best feeling (valence) of the user! 4 MM Find out on which days (dates) the MoodMap App was not used at all! 5 KS On given day, when exactly (at what time) did the given user had the longest break? How long was the break? 6 KS Find out on which day in the two recorded weeks did L work (regardless of breaks) longest? 7 KS Find out which application was most frequently used in the last two weeks by given user! 8 KS Find out which user used MS Word most often on the given day! 9 CM (a) Find out where (in which Countries) in Europe have notes been taken! (b) Find out in which cities in Austria did L and D take notes! 10 CM (a) Find out how many notes have been created at Inffeldgasse, Graz! (b) Find out how many notes have been created in Graz! Table 1: Tools and evaluation tasks. L, D and S are the initials of the users to whom the data belong. with the NASA-TLX. After finishing each group a questionnaire was distributed to directly evaluate the visual design, comprehensibility and user preference of in-app visualizations in comparison to the Vis-Tool visualizations. 5.3
Eight people participated in the experiment, all knowledge workers (researchers and software developers). Three of them were female, 5 male. 3 participants were aged between 18-27 and 5 were aged between 28-37. 6
Overall, our study participants performed significantly better with the generic visualization tool in five (T2, T3, T7, T9, T10) out of ten tasks, worse in only one (T5) task and without significant difference when compared to the in-app visualizations in the remaining four (T1, T4, T6, T8) tasks. To analyze results, the Fisher’s Test was used to check the homogeneity of variances. The tf-test was used to test significance for cases with homogeneous variance. If not, the Walch-Satterthwaite test was used. 6.1
The NASA-TLX includes six metrics, which are considered scales of workload. We used the simplified R-TLX method to compute workload by averaging the scores. Figure 3 (MoodMap vs. Vis-Tool), Figure 4 (KnowSelf vs. Vis-Tool) and Figure 5 (CorMAR vs. Vis-Tool) show the box plots of the significant results for NASA-TLX metric: mental demand (MD), physical demand (PD), temporal demand (TD), measured performance (MP) and frustration (F) as well as the workload (W), computed as the average of all self-evaluation metrics and the measured performance (MP). Task duration (D) for all apps is given in Figure 2. The result of the t-test for T2 indicates that participants experienced significantly less workload when using Vis-Tool than MoodMap, t(9) = 3.17; p < .01. Also, the task duration was significantly lower in the case of Vis-Tool, t(9) = 3.18; p < .01. In fact all individual metrics show significantly better scores in favor of Vis-Tool. For T3, there was a significant less workload and significant less duration when using Vis-Tool, t(9) = 2.13; p < .05 respectively t(9) = 3.44; p < .01. For T5, there was a significantly lower workload when using KnowSelf in comparison to Vis-Tool, t(9) = 2.21; p < .05. Individual metrics show a significant difference in effort and physical demand (see Figure 4). For T7, except for measured performance (MP), significant differences were found in every other metric. Participants experienced significantly lower workload using Vis-Tool, t(9) = 4.60; p < .01. They also spent significantly less time solving the task with Vis-Tool, t(9) = 3.64; p < .01. In the group CroMAR VS Vis-Tool, the results of both tasks show significant differences in favor of the Vis-Tool (see Figure 5). For T9, there was a significant difference in measured performance, t(9) = 3.16; p < .02. Individual metrics show significant difference in mental demand. For T10, there was a significantly less workload when using Vis-Tool, t(9) = 2.36; p < .04. Analysis of individual metrics showed significant differences in mental and physical demand. Duration was also significantly different in favor of Vis-Tool, t(9) = 4.68; p < .01. 6.2
The summarized results of the user preferences regarding the used apps for solving the given tasks are presented in Table 2. For the tasks T1-T4 and T9T10 Vis-Tool was preferred over both MoodMap and CroMAR. For the tasks T5-T8 the results of Vis-Tool vs. KnowSelf were ambiguous. For T5 and T6 participants preferred KnowSelf whereas for the tasks T7 and T8 they go for the Vis-Tool. This correlates with TLX where users performed better using KnowSelf in T5 but much worse in T7.
The results of the question “How did you like the visual design of the
visualisations for the given tasks?” (see Figure 6) showed a clear preference for
the visual design of the Vis-Tool in comparison to the MoodMap (tasks T1-T4)
and CorMAR (tasks T9-T10). In contrast, for the tasks T5-T8 they preferred
the visual design of KnowSelf over that of the Vis-Tool. Regarding the question
“How meaningful were the given visualizations for the given tasks?” the
participants stated that Vis-Tool visualizations where significantly more meaningful
for the given tasks in comparison to the MoodMap and CroMAR (see Figure 6).
Interestingly, there were no significant results regarding Vis-Tool and KnowSelf.
Overall, the performance of study participants was satisfactory with the
VisTool, showing comparable and mostly even better performance when compared
with in-app visualizations. In many cases, study participants had a significantly
lower workload and were significantly quicker to solve the tasks using generic
visualizations: Participants achieved significantly better results with the
VisTool than with the MoodMap App in two out of four tasks in terms of workload
and time to task completion (T2, T3 - see also Figure 2 and 3), better results
with the Vis-Tool than with KnowSelf in one out of four tasks (T7 - see also
Figure 2 and 4) and better results with the Vis-Tool than with CroMAR in two
out of two tasks in terms of task performance (T9) and workload and duration
(T10 - see also Figure 2 and 5). These results are also confirmed by the answers
of the questions regarding the comprehensibility of the visualizations with regard
to the given tasks (see Table 6).
These results are not a statement on the quality of design of the specific apps
per se. All three used activity logging applications have successfully been used to
induce and support learning in the workplace. Rather, the results are a function
of whether the data source applications have been designed to answer the type
of questions about data that study participants were asked to answer in the
evaluation. The focus of CroMAR for instance was in location-related,
augmentedreality-style, visualization of geo-tagged data in order to support situated
reflection on events [
Vis-Tool more supportive in a task that relates to a longer period of time (T7). Another example of generic visualizations adding benefit to in-app visualizations is that the data source applications had different support for multiple users: KnowSelf is a purely single-user application; nonetheless, there is a plausible interest within teams to know how others in the team use their time. CroMAR visualizes data from multiple users but does not visually mark which data comes from which user, and MoodMap App is a real collaborative tracking application. Our study results therefore clearly showcase that and how generic visualizations can add benefit to in-app visualizations when users want to solve analytic tasks beyond those that were known at application design time. 7.2
A limitation of the current implementation of the Vis-Tool is, that it is only able to display given properties, but cannot calculate new values. For instance, in KnowSelf, the data entries contain the start and the end time but not the duration. The visualizations in KnowSelf make use of such derived data properties: As KnowSelf developers know exactly what kind of data were available, they could also easily implement calculations based on given data and use these for visualizations. In the Vis-Tool on the other hand, we have in general too little prior knowledge about data to automatically perform meaningful calculations on data in order to compute “derived data properties”. Technically, it would be possible to extend the Vis-Tool’s user interface such that calculations on given data can be specified, but we assume that ease of use would be rather difficult to achieve. In addition, such functionality would increasingly replicate very generic spreadsheet (e.g., Excel), statistical analysis (e.g., SPSS) or visualization (e.g., ManyEyes) functionality. It might be easier overall to shift the burden “back” to data source applications, in the sense of requiring data source applications to provide derived values that are of interest themselves. 7.3
In this work we have focused on the comprehensibility of visualizations. We did not formally evaluate the user interaction itself, i.e. the process of creating a specific visualization. However, we are aware that the Vis-Tool requires users to become familiar with concepts such as mappings and visual channels. A plausible emerging scenario is to differentiate between two user roles: One role (expert) would be responsible for creating a set of meaningful visualizations. The expert would know concretely which data source applications are available and what kind of analytic tasks users will want to solve. This person does not need to write code, but needs to have some training or experience with the Vis-Tool. The set of meaningful visualizations would be stored and serve as pre-configuration for learners. A second role (learner) would then only need to load a pre-configured set of visualizations and “use” them, similar to the study participants in the task-based evaluation discussed in this paper. Of course, users would have the freedom to explore the mapping interface if interested, and generate new visualizations. Based on this overall scenario, more complex usage scenarios for generic visualization tools like ours could be elaborated that involve for instance sharing and recommending dashboards. 8
We have developed a generic visualisation tool for activity log data that addresses two fundamental challenges shared in many scenarios at the intersection of learning analytics, open learner modelling, and workplace learning on the basis of (activity log) data: Data from multiple applications shall be visualised; and at the time of designing the visualisation tool, the concrete data sources and consequently the concrete analytic tasks are unknown. The Vis-Tool makes only two assumptions about data, namely that they are time-stamped and are associated with users. The comprehensibility of the Vis-Tools visualisations was evaluated in an experiment along data analytics tasks that were designed on the background of workplace learning. This evaluation was carried out within the target user group of knowledge worker, and based on real-world data. It thus constitutes firm ground, also for other researchers, to compare the suitability of other generic visualisations with, or to proceed with the next step in the design process for such a generic visualisation tool, namely the design of the user interaction process.
The project “MIRROR - Reflective learning at work” is funded under the FP7 of the European Commission (project nr. 257617). The Know-Center is funded within the Austrian COMET Program - Competence Centers for Excellent Technologies - under the auspices of the Austrian Federal Ministry of Transport, Innovation and Technology, the Austrian Federal Ministry of Economy, Family and Youth and by the State of Styria. COMET is managed by the Austrian Research Promotion Agency FFG.