=Paper=
{{Paper
|id=Vol-1103/paper5
|storemode=property
|title=Improving awareness and reflection through collaborative, interactive visualizations of badges
|pdfUrl=https://ceur-ws.org/Vol-1103/paper5.pdf
|volume=Vol-1103
|dblpUrl=https://dblp.org/rec/conf/ectel/CharleerKSD13
}}
==Improving awareness and reflection through collaborative, interactive visualizations of badges==
https://ceur-ws.org/Vol-1103/paper5.pdf
Improving awareness and reflection through
collaborative, interactive visualizations of badges
Sven Charleer, Joris Klerkx, Jose Luis Santos and Erik Duval
Dept. of Computer Science, KU Leuven, Celestijnenlaan 200A
3001 Leuven, Belgium
{Sven.Charleer,Joris.Klerkx,JoseLuis.Santos,Erik.Duval}@cs.kuleuven.be
Abstract. This paper introduces novel ways of improving awareness
and reflection through visualizations of badges as an abstraction of learn-
ing analytics data. We report initial findings with both a personal dash-
board approach, Navi Badgeboard, that provides details on student and
class progress, and a collaborative, interactive tabletop visualization,
Navi Surface, to promote group reflection. We evaluate both approaches
to find improvements among students regarding awareness and reflec-
tion on course activities. Our results indicate that Navi Badgeboard
helps with awareness of personal activity while Navi Surface improves
collaboration resulting in better reflection.
Keywords: learning analytics, learning dashboards, collaboration, re-
flection, awareness, visualization, badges
1 Introduction
Feedback and collaborative discourse, between student and teacher, among stu-
dents and even with external parties, leads to significant gains in learning [2].
Traditional tools for this are exams, class discussions, self-assessment and peer
evaluations, but also (micro-)blogging (Twitter, Wordpress, Facebook) can help
students share and reflect on their work, collaborate, discuss and learn from
class mates [10]. These processes leave behind a multitude of learner traces that
reflect activity and progress of students [17]. We strongly believe that visual-
izations of these traces in so-called learning dashboards can assist in creating a
feedback loop of awareness, reflection, sense-making and impact [17] and improve
motivation [13].
In previous work, we have developed visualizations of learning traces through
tools such as StepUp![14], SAM[6] and TinyARM[13]. While these traces pro-
vide a broad insight on student activities, the abundance of information can be
overwhelming for both student and teacher, even when presented through bar
charts, line charts and parallel coordinates.
This paper focusses on our attempts to visualize an abstraction of trace data.
We limit the data to the essential course goal settings. The goal is to assist in
informing the students individually about their progress and to enable discus-
sion in class. This abstraction can be achieved by defining badges for activities.
69
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
We visualize the data through two applications that we have developed in an
attempt to improve awareness and reflection: a personal learning dashboard to
support individual awareness and reflection and an interactive visualization on
a multitouch tabletop to support collaborative awareness and reflection.
In section 2, we will discuss more examples of learning dashboards, collab-
orative visualizations and the use of badges in learning. Section 3 explains the
setting of the course we use to evaluate our tools and explains the badge sys-
tem in more detail. Section 4 elaborates on 2 approaches: the personal dashboard
and the collaborative visualizations. Section 5 details the evaluation of the tools,
followed by some ideas for future development and conclusions in section 6.
2 Related Work
The Quantified Self (QS)1 movement, which focusses on collecting user traces
and using data for self-improvement, is spreading across multiple domains but is
probably best known for its application in personal health [16]. Through mobile
phone apps and an increase in cheap tracking devices (e.g. Fitbit2 , Nike+3 ),
people can become more aware of their health and modify their behavior by
tracking activities such as walking, sleeping, running, etc. A similar change in
behavior can be achieved with students by applying QS to learning, tracking the
traces students leave behind through e.g. blogging, time tracking etc. [4]
These learning traces can help students become more aware of their activities.
By visualizing these traces through interactive dashboards, students and teachers
are provided with better ways of exploring and understanding this abundance of
data [14][6]. Personal dashboards can also be populated with grade and badge
data [8].
Badges, which are essentially another form of abstraction of the tracked data,
bring with them many benefits and uses: The creation process of the badges can
influence the design of the course [7] and hence create clearer goals for both
student and teacher. Badges can be used as feedback and are proven to directly
impact behavior and motivate students in off- and online courses [11][13][7]. Skill
recognition can be brought outside the classroom to support life long learning
by using badges as certifications in e.g. Massive Open Online Courses [7][5].
As collaboration can have a serious impact on learning [2] and the possibilities
of collaborative visualizations are yet to be explored more fully [9], this paper
does not only look at personal learning dashboards but also at ways of com-
bining interactive discourse with learning traces and more specifically badges.
Our research focusses on awareness and reflection through tabletop displays: as
a collaborative tool in a formal setting, they cause students working in small
groups to articulate and reflect on their insights more than while using more
conventional displays [15] or paper [12]. They can also be used in more infor-
mal settings, a public place like a university hall where collaboration between
1
http://quantifiedself.com
2
http://www.fitbit.com
3
http://nikeplus.nike.com
70
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
strangers helps increase awareness and reflection regarding a specific topic [1].
We focus on how to improve awareness and create a better insight of the learning
process.
3 Learning Analytics Data
3.1 Course Setting
We evaluated our tools in a class of 26 engineering students between the ages
of 20 and 25 following a course on Human-Computer Interaction (HCI). This
year, the course focusses on the design, development and evaluation (usability,
usefulness) of a recommendation application. Students work in groups of 3 and
improve their application through iterative development. The course, while it in-
cludes face-to-face studio sessions on the Science Campus of KU Leuven, is open
to everyone. All data including presentations, course material and reports is pub-
licly accessible online, through Slideshare4 , the course wiki5 and the group blogs
which students are required to update regularly. They use Twitter with a course
specific hashtag (#chikul13 6 ). Discourse happens through class discussions and
comments on each other’s blogs and tweets.
3.2 Learning Traces
The student generates data by blogging, commenting and tweeting. These ac-
tivities leave behind traces that can be used by learning dashboards to visualize
activity and progress. Both individual and group activity can be visualized, but
also data on interaction between students, groups and even interaction with ex-
ternal people is available. Visualizations can help students become aware of their
activities and compare their performance to evaluate their progress in class.
Automated trackers hook up to the RSS feeds of the blogs and connect to
the Twitter API. They gather student generated data, store it in a database and
make it accessible to other applications through a REST service. On these ser-
vices, learning dashboards can be developed visualizing the data. This framework
is explained more in detail in [13].
3.3 Badges
In previous attempts, we have focussed on showing all the collected data. While
these dashboards provide an abundance of information, it is interesting to limit
the feedback to the essentials (e.g. in the HCI course: regular blog activity, com-
menting activity). Badges help abstract the data and create a more generalized
overview of the traces. This can provide a better understanding of the goals
settings and the required activity.
A badge is set up as follows:
4
http://slideshare.net
5
http://ariadne.cs.kuleuven.be/wiki/index.php/Chi_2013
6
https://twitter.com/search/realtime?q=chikul13&src=typd
71
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
– a badge icon with an easily identifiable image related to the semantics of the
badge (see Fig. 1).
– a color coding for categorizing the badge by type (positive, negative, neutral).
– a bronze/silver/gold medal for badges indicates different levels of achieve-
ments.
– a textual description on how the badge can be achieved.
To define what badges we use in our HCI course, we looked at the activities
that are important. Blogging and Twitter play a big role in the course as they
are indicators of commitment and collaboration, so we want to award this be-
havior. For example, badges are awarded for a specific number of tweets, posts,
comments. High comment activity on a post is usually triggered by the content
of that post. Therefore badges can also indicate quality of the content of a blog
post which can be derived from the number of comments it receives, by internals
(students, teachers) and externals (visitors). Inactivity is a behavior students
should attempt to avoid and can be detected through lack of digital traces, that
we translate into a negative badge. In total we defined 51 badges of which 42
were awarded repeatedly bi-weekly. The full list of 51 badges can be found at
http://navi-hci.appspot.com.
Badges are automatically assigned. Certain activity in the tracked data will
trigger an event when requirements are met and a mail is sent to the student
with information on the awarded badge. This badge data is also stored and can
be accessed through a REST service, creating an open data framework on which
other visualizations can easily be developed.
We follow the Mozilla OpenBadge Standard7 , so that students can choose to
publish their awards on social networks.
In the next section, we will discuss how these badges are shared among
students, in an individual and collaborative way.
4 Personal and Collaborative Interactive Visualizations
4.1 Two Approaches
Badges can assist in informing students individually about their progress but
can also play the role of a catalyst for discussion. We present two methods that
attempt to increase awareness and reflection in quite different ways.
The first approach relies on personal dashboards. Navi Badgeboard provides
an overview of achievements and progress. Students can also compare progress
with that of the class. This tool is used in a personal way, usually outside of
class on a desktop computer or mobile device. The data is open and public and
therefore not only students can access each others’ dashboard, but also teachers
and externals.
While the first approach gives students the opportunity to check their progress
on their own time, the second approach is more controlled and direct by facing
7
http://openbadges.org
72
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
students with their achievements (or lack thereof) in a public setting. One way
to achieve this is by projecting Navi Badgeboard in class while the teacher mod-
erates a discussion around specific badges. As only the teacher is in control of
this projection, students play a more passive role in this discussion. To create a
more active discussion where all parties have equal power in steering the con-
versation, a more interactive visualization is necessary with support for multiple
users. Therefore we take our second approach to tabletop displays, creating Navi
Surface which allows students and teacher, or groups of students, to create and
moderate a more open and deeper reflection discourse.
4.2 Navi Badgeboard
Through Navi Badgeboard, the user can discover the badges and therefore the
intended course learning outcomes of the HCI course and find more detailed
information on how these badges can be achieved. It also presents the user with
a list of all students participating in the course. From this list, each student’s
Personal Badge Dashboard can be accessed.
Personal Badge Dashboards contain a list of badges per bi-weekly period (see
Fig. 1): greyed-out badges have not yet been acquired by the student, colored
badges have. The user can cycle through the bi-weekly periods.
Fig. 1. Navi Badgeboard - Personal Badge Overview: A student’s badge overview for
a given period. Colored badges have been achieved, greyed-out ones have not. The
number next to the badge (e.g. the number highlighted by the red circle) indicates how
many students in class have been awarded this badge.
The number next to the badge indicates the total times the badge has been
awarded to the class in the selected period. A high number next to a grey icon
indicates the student is one of the few students without the badge. A low number
next to a colored icon indicates the student is one of the few to have earned this
badge. Depending on the badge, either situation can be a good or a bad thing.
From the Personal Badge Overview, the user can drill down to a Class Pro-
gression View, a visualization of the class’s badge progression over time. An
example is shown in figure 2 where the X-axis represents time and the Y-axis
73
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
Fig. 2. Navi Badgeboard - Class Progression: Every line represents a badge. Circles
represent the moment in time the student has been awarded the respective badge. The
user can choose which badges to visualize through the filter options.
the number of students that have been awarded a specific badge. Every colored
line represents the progression of a badge for the class. The circle indicates when
a particular student was awarded this badge. The Class Progression View gives
the user an immediate idea of how early or late the student is at achieving a
specific badge compared to the rest of the class.
Students can also share their badges as a way of skill recognition to exter-
nal parties. Through a ’send to backpack’ button, students with Mozilla Open
Badges8 accounts can send these badges to their Mozilla Open Badges Backpack
and publish them on social network sites.
Navi Badgeboard is developed using HTML5, Javascript, D3.js9 and Java
deployed on the Google App Engine10 . This application can be accessed from
mobile devices and desktop browsers and is publicly available 11 .
4.3 Navi Surface
An interactive visualization can help students get a better understanding of the
course activity data. Adding collaboration to the process creates opportunities
for a more active discourse around the data. To enable such collaboration, we
developed Navi Surface, a first prototype of a multi-user multi-touch tabletop
display application developed using HTML5, Javascript and Paper.js12 .
8
http://openbadges.org
9
http://d3js.org
10
https://appengine.google.com
11
http://navi-hci.appspot.com
12
http://paperjs.org
74
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
Navi Surface presents the users with a list of students and badges available in
the course. The badges are displayed per bi-weekly period and the user can cycle
through these periods. This information is located at the bottom of the display
and is interactive: each student name and badge can be touched and dragged.
The remainder of the screen is called the Playfield. All interactive items can
be dragged onto the Playfield. The badges in the Playfield light up the names
of students that have been awarded these badges. Student names light up the
badges that have been awarded to the respective students. Dropping badges onto
the Playfield also displays their detailed information.
Touching and holding an item will activate the relationship visualization:
lines will connect the item to all its related items on the Playfield e.g. a student
name will be connected to all its awarded badges. As the application supports
multi-touch, multiple items can be moved and touched simultaneously, creat-
ing more interesting visual relationships (see Fig. 4) and enabling collaborative
interaction with the data.
Fig. 3. Navi Surface: The bottom left shows the list of badges of a specific period.
The bottom right contains the students’ names. The items in the Playfield (top) are
touched and held to display the relationships between them.
The tabletop display can be placed in the classroom. As the CHI course
format is a studio session, students can freely get up and walk to the tabletop
to access their own and class mates’ information. The teacher can also invite
students to the tabletop display. In our CHI course, as students work in groups
of 3, the teacher can invite a group to discuss their progress. While the teacher
can guide the process by dragging items onto the Playfield, students can also
interact and steer the conversation. This engagement causes a personal but also
collaborative interaction and evaluation.
75
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
Fig. 4. Navi Surface: Students actively using the tabletop display application during
our evaluation session.
5 Evaluation
5.1 Navi Badgeboard
22 of the 26 students between the ages of 20 and 25 participated in the evaluation
of Navi Badgeboard. First we evaluated the usability of Navi Badgeboard using
the System Usability Scale (SUS)[3]. It scored 65 which means it scored below
average. We also asked students to fill in an online survey that goes more into
detail on the different functionalities and visualizations which revealed additional
interesting information.
Fig. 5. Navi Badgeboard Functionality Importance Questionnaire: A) Overview of
other students’ badges B) Overview of number of badges achieved by the entire class
C) Overview of all badges and descriptions D) Filter options for the Class Progression
graph E) Class Statistic graph per badge F) Add to OpenBadge backpack
Figure 5 shows how important students consider the different functionalities
using a 5-level Likert item (2 - Not at all important, 6 - Extremely important)
with one extra option (1 - Did not know the feature existed). Students mostly
agreed the dashboard gives an accurate representation of the activity of the
76
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
class. The overview of the badges which shows the students what badges are
achievable in class was rated important. The global overview of the class’s pro-
gression however was deemed less important. As students also seemed to have
less preference for class mates’ personal dashboards, we can assume that they
were more interested in gaining awareness of their own goals and tasks than
awareness of others. This is also confirmed by the lack of interest for the Class
Progression View. These views were very valuable to the teacher and teacher
assistant however.
Fig. 6. Navi Badgeboard questionnaire: A) The system increases my motivation B)
The system promotes commenting on blogs of other students. C) The system promotes
reading blogs of other students. D) The system promotes Twitter participation.
Figure 6 shows the results of 4 5-scale (1- Strongly disagree, 5 - Strongly
agree) Likert questions. Students believed motivation was improved through
the Navi Badgeboard. They were under the impression it promotes commenting,
reading blogs and Twitter, activities these badges were designed to impact. While
this was the case for Navi Badgeboard, our dashboards showing raw activity data
through lists, tables and charts had a lesser impact on motivation [13].
The Google Analytics data showed that most activity happened around the
CHI course’s studio sessions. It can be assumed that students check up on their
badges before a session starts. Students are also notified by email when awarded
a badge. Two students mentioned that they would only visit Navi Badgeboard
at such occasions.
While we have included Open Badges support, almost no students were in-
terested in or even aware of this functionality. While we could improve the usage
of this feature by making it more prominent, most badges might have been too
specific to the course process, giving the students no incentive to share them
externally.
We can conclude that Navi Badgeboard does improve awareness of the goals
and tasks required to successfully complete the course through the Badge Overview
page. Students also regularly check up on their own progress, usually before a
77
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
class session begins. We can assume students wish to reflect on their progress
through these regular visits. However, they seem less interested in other students’
achievements.
5.2 Navi Surface
While we will evaluate further prototypes of Navi Surface during course sessions,
this initial evaluation of the tabletop display application took place during a
poster session just before the end of the CHI course. This gave us the opportunity
to not only evaluate the tool with CHI students but also outsiders. 14 students
walked up to the tabletop to test the application using the think-aloud protocol.
Students were left to experiment alone or in group and hints were only given
when the participant(s) got stuck. Student actions were recorded on video and
they were given a questionnaire afterwards. 10 students approached and tested
the tabletop in groups: 2 groups of 2 students and 2 groups of 3. Not all students
were part of the CHI course.
The application received a SUS score of 71 which is just above average.
However, only taking into account the CHI course students, the application
received a score of 77. We assume that due to the abstract nature of the data, in
its current form, Navi Surface does not give enough insight on the course content
which makes it harder to use for outsiders. We will discuss how we can improve
this further in section 6.
The goal of Navi Surface is to provide better understanding of the data and
thus also increase awareness through collaborative interaction. This collaborative
interaction should also ignite further discussions to create a deeper reflection.
Fig. 7. Navi Surface questionnaire: A) The visualization improves my awareness of
the class’s general progress B) The visualization helps me understand the meaning of
the badges C) The visualization helps me see what badges someone earned in a given
period D) The visualization helps me compare badges of students E) I would like to
use this tool together with the teacher to evaluate my progress F) I would like to use
this tool together with other students to compare our progress
78
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
Figure 7 contains box plots of the results of the 5-scale Likert questionnaire (1
- Strongly disagree, 2 - Strongly agree). Students do not believe their awareness of
class progression was in any way improved. This result was expected as the first
prototype only shows badges for the students dragged into the Playfield area.
Only after dragging all names onto the tabletop would the user get a better idea
of class status regarding the awards. As badges are only shown per period, an
overview of the entire course length is also not available.
Navi Surface was built with multi-user collaboration in mind and while a
single user experience is possible, it was not the goal of the application. This
matches our finding in the questionnaire: there was a preference of using the tool
in group. There was also an interest in using the tool together with a teacher.
Observation of the students while using the tool also confirmed that collab-
oration improved the reflection process as students understood the tool and the
data much quicker. After getting a good grasp of what Navi Surface provided,
they spontaneously started discussing their progress based on the badges. They
reflected on why and how certain badges were achieved and others were not.
They also experienced this collaboration to be more fun.
On the other hand, students faced with Navi Surface by themselves were
more hesitant and needed input from the observer to continue using the tool.
While the questionnaires confirm that they prefer to use it in group, it is clear
that without the collaboration and social discourse this interaction enables, the
actions are less spontaneous and much less deeper reflection occurs.
The tool was also tested on students who were not part of the CHI course.
This however proved less successful especially with students faced with the tool
by themselves. As Navi Surface does not provide any details on the actual data
behind the badges (blog posts, comments and tweets), the data is very unclear
to outsiders. This also affected the SUS score (see above). In section 6, we will
discuss how we can provide more detailed information and hereby also make
Navi Surface more interesting to outsiders.
6 Conclusion and Future Work
Learning dashboards provide a means of visualizing the abundance of traces
which learning analytics allows us to collect from students. We look at simplifying
the data by emphasizing the more important student activities and course goals
and visualizing these through badges.
While the personal dashboard has improved perceived awareness with stu-
dents and the overview of class progress was deemed valuable by teachers and
teacher assistants, we believe Navi Surface has more potential in helping aware-
ness and reflection with students through its collaborative nature. In its current
state, Navi Surface already enables students to understand the data quicker and
plays a catalyst in discussions. Navi Surface makes the process fun and students
show interest in using this tool with class mates but also with teachers.
More evaluations will give us deeper insights and we believe that there are
still many unexplored possibilities which makes further development of this tool
79
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
very interesting. By simply adding more course data to the visualizations and
allowing students to drill down on badges to reveal more data, students could
discover why and how certain badges have been awarded (e.g. the specific blog
post or comment that triggered a badge), creating a better insight of the progress
and a deeper reflection on the learning process.
Adding more detailed course data to the visualization does not only benefit
student and teacher, but could help outsiders comprehend the inner workings of
a course better. Open school days can help students choose their future classes
based on the real data provided through Navi Surface. Parents’ evenings can
become more interactive as parents and teacher can utilize the tabletop to dig
deeper into the details of the learning process of son or daughter.
While we mainly work with blog and Twitter data, Navi Surface can easily be
extended to support even richer learning analytics data. Students in a more in-
quiry based learning environment leave behind richer artifacts (e.g. photographs,
geographical coordinates) which could provide even more interesting visualiza-
tions and therefore an even better insight through Navi Surface. Even MOOCs
(Massive Open Online Courses) can benefit from the abstraction to badges and
the visualization through Navi Surface.
With few developments, many new possibilities open up. We will therefore
continue our research into badges and tabletop displays as a way of improving
awareness and reflection in the class room and beyond.
7 Acknowledgement
The research leading to these results has received funding from the European
Community’s Seventh Framework Programme (FP7/2007-2013) under grant agree-
ment No 318499 - weSPOT project.
References
1. Antle, A.N., Bevans, A., Tanenbaum, J., Seaborn, K., Wang, S.: Futura: design for
collaborative learning and game play on a multi-touch digital tabletop. In: Pro-
ceedings of the fifth international conference on Tangible, embedded, and embodied
interaction, ACM (2011) 93–100
2. Black, P., Wiliam, D.: Assessment and classroom learning. Assessment in education
5(1) (1998) 7–74
3. Brooke, J.: Sus: A quick and dirty usability scale, pages 189–194. usability evalu-
ation in industry (1996)
4. Duval, E.: Attention please!: learning analytics for visualization and recommen-
dation. In: Proceedings of the 1st International Conference on Learning Analytics
and Knowledge, ACM (2011) 9–17
5. Goligoski, E.: Motivating the learner: Mozilla?s open badges program. Access to
Knowledge: A Course Journal 4(1) (2012)
6. Govaerts, S., Verbert, K., Duval, E., Pardo, A.: The student activity meter for
awareness and self-reflection. In: Proceedings of the 2012 ACM annual conference
extended abstracts on Human Factors in Computing Systems Extended Abstracts,
ACM (2012) 869–884
80
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
7. Higashi, R., Abramovich, S., Shoop, R., Schunn, C.: The roles of badges in the
computer science student network. In: 2012 Games+Learning+Society Conference.
(June 2012)
8. Holman, C., Aguilar, S., Fishman, B.: Gradecraft: what can we learn from a game-
inspired learning management system? In: Proceedings of the Third International
Conference on Learning Analytics and Knowledge, ACM (2013) 260–264
9. Isenberg, P., Elmqvist, N., Scholtz, J., Cernea, D., Ma, K.L., Hagen, H.: Col-
laborative visualization: Definition, challenges, and research agenda. Information
Visualization 10(4) (2011) 310–326
10. Lin, W.J., Liu, Y.L., Kakusho, K., Yueh, H.P., Murakami, M., Minoh, M.: Blog
as a tool to develop e-learning experience in an international distance course. In:
Advanced Learning Technologies, 2006. Sixth International Conference on, IEEE
(2006) 290–292
11. McDaniel, R., Lindgren, R., Friskics, J.: Using badges for shaping interactions in
online learning environments. In: Professional Communication Conference (IPCC),
2012 IEEE International, IEEE (2012) 1–4
12. Piper, A.M., Hollan, J.D.: Tabletop displays for small group study: affordances of
paper and digital materials. In: Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems, ACM (2009) 1227–1236
13. Santos, J.L., Charleer, S., Parra, G., Klerkx, J., Duval, E., Verbert, K.: Evaluating
the use of open badges in an open learning environment. In: Proceedings of the
European Conference on Technology Enhanced Learning. (accepted)
14. Santos, J.L., Verbert, K., Govaerts, S., Duval, E.: Addressing learner issues with
stepup!: an evaluation. In: Proceedings of the Third International Conference on
Learning Analytics and Knowledge, ACM (2013) 14–22
15. Shaer, O., Strait, M., Valdes, C., Feng, T., Lintz, M., Wang, H.: Enhancing ge-
nomic learning through tabletop interaction. In: Proceedings of the 2011 annual
conference on Human factors in computing systems, ACM (2011) 2817–2826
16. Smarr, L.: Quantifying your body: A how-to guide from a systems biology per-
spective. Biotechnology Journal 7(8) (2012) 980–991
17. Verbert, K., Duval, E., Klerkx, J., Govaerts, S., Santos, J.L.: Learning analytics
dashboard applications. American Behavioral Scientist (0002764213479363) (2013)
81