This paper introduces novel ways of improving awareness and reflection through visualizations of badges as an abstraction of learning analytics data. We report initial findings with both a personal dashboard approach, Navi Badgeboard, that provides details on student and class progress, and a collaborative, interactive tabletop visualization, Navi Surface, to promote group reflection. We evaluate both approaches to find improvements among students regarding awareness and reflection on course activities. Our results indicate that Navi Badgeboard helps with awareness of personal activity while Navi Surface improves collaboration resulting in better reflection.
Feedback and collaborative discourse, between student and teacher, among
students and even with external parties, leads to significant gains in learning [
In previous work, we have developed visualizations of learning traces through
tools such as StepUp![
This paper focusses on our attempts to visualize an abstraction of trace data. We limit the data to the essential course goal settings. The goal is to assist in informing the students individually about their progress and to enable discussion in class. This abstraction can be achieved by defining badges for activities.
We visualize the data through two applications that we have developed in an attempt to improve awareness and reflection: a personal learning dashboard to support individual awareness and reflection and an interactive visualization on a multitouch tabletop to support collaborative awareness and reflection.
In section 2, we will discuss more examples of learning dashboards, collaborative visualizations and the use of badges in learning. Section 3 explains the setting of the course we use to evaluate our tools and explains the badge system in more detail. Section 4 elaborates on 2 approaches: the personal dashboard and the collaborative visualizations. Section 5 details the evaluation of the tools, followed by some ideas for future development and conclusions in section 6. 2
The Quantified Self (QS)1 movement, which focusses on collecting user traces
and using data for self-improvement, is spreading across multiple domains but is
probably best known for its application in personal health [
These learning traces can help students become more aware of their activities.
By visualizing these traces through interactive dashboards, students and teachers
are provided with better ways of exploring and understanding this abundance of
data [
Badges, which are essentially another form of abstraction of the tracked data,
bring with them many benefits and uses: The creation process of the badges can
influence the design of the course [
As collaboration can have a serious impact on learning [
strangers helps increase awareness and reflection regarding a specific topic [
We evaluated our tools in a class of 26 engineering students between the ages of 20 and 25 following a course on Human-Computer Interaction (HCI). This year, the course focusses on the design, development and evaluation (usability, usefulness) of a recommendation application. Students work in groups of 3 and improve their application through iterative development. The course, while it includes face-to-face studio sessions on the Science Campus of KU Leuven, is open to everyone. All data including presentations, course material and reports is publicly accessible online, through Slideshare4, the course wiki5 and the group blogs which students are required to update regularly. They use Twitter with a course specific hashtag (#chikul13 6). Discourse happens through class discussions and comments on each other’s blogs and tweets. 3.2
The student generates data by blogging, commenting and tweeting. These activities leave behind traces that can be used by learning dashboards to visualize activity and progress. Both individual and group activity can be visualized, but also data on interaction between students, groups and even interaction with external people is available. Visualizations can help students become aware of their activities and compare their performance to evaluate their progress in class.
Automated trackers hook up to the RSS feeds of the blogs and connect to
the Twitter API. They gather student generated data, store it in a database and
make it accessible to other applications through a REST service. On these
services, learning dashboards can be developed visualizing the data. This framework
is explained more in detail in [
In previous attempts, we have focussed on showing all the collected data. While these dashboards provide an abundance of information, it is interesting to limit the feedback to the essentials (e.g. in the HCI course: regular blog activity, commenting activity). Badges help abstract the data and create a more generalized overview of the traces. This can provide a better understanding of the goals settings and the required activity.
A badge is set up as follows:
– a badge icon with an easily identifiable image related to the semantics of the badge (see Fig. 1). – a color coding for categorizing the badge by type (positive, negative, neutral). – a bronze/silver/gold medal for badges indicates different levels of achievements. – a textual description on how the badge can be achieved.
To define what badges we use in our HCI course, we looked at the activities that are important. Blogging and Twitter play a big role in the course as they are indicators of commitment and collaboration, so we want to award this behavior. For example, badges are awarded for a specific number of tweets, posts, comments. High comment activity on a post is usually triggered by the content of that post. Therefore badges can also indicate quality of the content of a blog post which can be derived from the number of comments it receives, by internals (students, teachers) and externals (visitors). Inactivity is a behavior students should attempt to avoid and can be detected through lack of digital traces, that we translate into a negative badge. In total we defined 51 badges of which 42 were awarded repeatedly bi-weekly. The full list of 51 badges can be found at http://navi-hci.appspot.com.
Badges are automatically assigned. Certain activity in the tracked data will trigger an event when requirements are met and a mail is sent to the student with information on the awarded badge. This badge data is also stored and can be accessed through a REST service, creating an open data framework on which other visualizations can easily be developed.
We follow the Mozilla OpenBadge Standard7, so that students can choose to publish their awards on social networks.
In the next section, we will discuss how these badges are shared among students, in an individual and collaborative way. 4 4.1
Badges can assist in informing students individually about their progress but can also play the role of a catalyst for discussion. We present two methods that attempt to increase awareness and reflection in quite different ways.
The first approach relies on personal dashboards. Navi Badgeboard provides an overview of achievements and progress. Students can also compare progress with that of the class. This tool is used in a personal way, usually outside of class on a desktop computer or mobile device. The data is open and public and therefore not only students can access each others’ dashboard, but also teachers and externals.
While the first approach gives students the opportunity to check their progress on their own time, the second approach is more controlled and direct by facing
students with their achievements (or lack thereof) in a public setting. One way to achieve this is by projecting Navi Badgeboard in class while the teacher moderates a discussion around specific badges. As only the teacher is in control of this projection, students play a more passive role in this discussion. To create a more active discussion where all parties have equal power in steering the conversation, a more interactive visualization is necessary with support for multiple users. Therefore we take our second approach to tabletop displays, creating Navi Surface which allows students and teacher, or groups of students, to create and moderate a more open and deeper reflection discourse. 4.2
Through Navi Badgeboard, the user can discover the badges and therefore the intended course learning outcomes of the HCI course and find more detailed information on how these badges can be achieved. It also presents the user with a list of all students participating in the course. From this list, each student’s Personal Badge Dashboard can be accessed.
Personal Badge Dashboards contain a list of badges per bi-weekly period (see Fig. 1): greyed-out badges have not yet been acquired by the student, colored badges have. The user can cycle through the bi-weekly periods.
The number next to the badge indicates the total times the badge has been awarded to the class in the selected period. A high number next to a grey icon indicates the student is one of the few students without the badge. A low number next to a colored icon indicates the student is one of the few to have earned this badge. Depending on the badge, either situation can be a good or a bad thing.
From the Personal Badge Overview, the user can drill down to a Class Progression View, a visualization of the class’s badge progression over time. An example is shown in figure 2 where the X-axis represents time and the Y-axis the number of students that have been awarded a specific badge. Every colored line represents the progression of a badge for the class. The circle indicates when a particular student was awarded this badge. The Class Progression View gives the user an immediate idea of how early or late the student is at achieving a specific badge compared to the rest of the class.
Students can also share their badges as a way of skill recognition to external parties. Through a ’send to backpack’ button, students with Mozilla Open Badges8 accounts can send these badges to their Mozilla Open Badges Backpack and publish them on social network sites.
Navi Badgeboard is developed using HTML5, Javascript, D3.js9 and Java deployed on the Google App Engine10. This application can be accessed from mobile devices and desktop browsers and is publicly available 11. An interactive visualization can help students get a better understanding of the course activity data. Adding collaboration to the process creates opportunities for a more active discourse around the data. To enable such collaboration, we developed Navi Surface, a first prototype of a multi-user multi-touch tabletop display application developed using HTML5, Javascript and Paper.js12.
Navi Surface presents the users with a list of students and badges available in the course. The badges are displayed per bi-weekly period and the user can cycle through these periods. This information is located at the bottom of the display and is interactive: each student name and badge can be touched and dragged.
The remainder of the screen is called the Playfield. All interactive items can be dragged onto the Playfield. The badges in the Playfield light up the names of students that have been awarded these badges. Student names light up the badges that have been awarded to the respective students. Dropping badges onto the Playfield also displays their detailed information.
Touching and holding an item will activate the relationship visualization: lines will connect the item to all its related items on the Playfield e.g. a student name will be connected to all its awarded badges. As the application supports multi-touch, multiple items can be moved and touched simultaneously, creating more interesting visual relationships (see Fig. 4) and enabling collaborative interaction with the data.
The tabletop display can be placed in the classroom. As the CHI course format is a studio session, students can freely get up and walk to the tabletop to access their own and class mates’ information. The teacher can also invite students to the tabletop display. In our CHI course, as students work in groups of 3, the teacher can invite a group to discuss their progress. While the teacher can guide the process by dragging items onto the Playfield, students can also interact and steer the conversation. This engagement causes a personal but also collaborative interaction and evaluation.
22 of the 26 students between the ages of 20 and 25 participated in the evaluation
of Navi Badgeboard. First we evaluated the usability of Navi Badgeboard using
the System Usability Scale (SUS)[
Figure 6 shows the results of 4 5-scale (1- Strongly disagree, 5 - Strongly
agree) Likert questions. Students believed motivation was improved through
the Navi Badgeboard. They were under the impression it promotes commenting,
reading blogs and Twitter, activities these badges were designed to impact. While
this was the case for Navi Badgeboard, our dashboards showing raw activity data
through lists, tables and charts had a lesser impact on motivation [
The Google Analytics data showed that most activity happened around the CHI course’s studio sessions. It can be assumed that students check up on their badges before a session starts. Students are also notified by email when awarded a badge. Two students mentioned that they would only visit Navi Badgeboard at such occasions.
While we have included Open Badges support, almost no students were interested in or even aware of this functionality. While we could improve the usage of this feature by making it more prominent, most badges might have been too specific to the course process, giving the students no incentive to share them externally.
We can conclude that Navi Badgeboard does improve awareness of the goals and tasks required to successfully complete the course through the Badge Overview page. Students also regularly check up on their own progress, usually before a class session begins. We can assume students wish to reflect on their progress through these regular visits. However, they seem less interested in other students’ achievements. 5.2
While we will evaluate further prototypes of Navi Surface during course sessions, this initial evaluation of the tabletop display application took place during a poster session just before the end of the CHI course. This gave us the opportunity to not only evaluate the tool with CHI students but also outsiders. 14 students walked up to the tabletop to test the application using the think-aloud protocol. Students were left to experiment alone or in group and hints were only given when the participant(s) got stuck. Student actions were recorded on video and they were given a questionnaire afterwards. 10 students approached and tested the tabletop in groups: 2 groups of 2 students and 2 groups of 3. Not all students were part of the CHI course.
The application received a SUS score of 71 which is just above average. However, only taking into account the CHI course students, the application received a score of 77. We assume that due to the abstract nature of the data, in its current form, Navi Surface does not give enough insight on the course content which makes it harder to use for outsiders. We will discuss how we can improve this further in section 6.
The goal of Navi Surface is to provide better understanding of the data and thus also increase awareness through collaborative interaction. This collaborative interaction should also ignite further discussions to create a deeper reflection.
Figure 7 contains box plots of the results of the 5-scale Likert questionnaire (1 - Strongly disagree, 2 - Strongly agree). Students do not believe their awareness of class progression was in any way improved. This result was expected as the first prototype only shows badges for the students dragged into the Playfield area. Only after dragging all names onto the tabletop would the user get a better idea of class status regarding the awards. As badges are only shown per period, an overview of the entire course length is also not available.
Navi Surface was built with multi-user collaboration in mind and while a single user experience is possible, it was not the goal of the application. This matches our finding in the questionnaire: there was a preference of using the tool in group. There was also an interest in using the tool together with a teacher.
Observation of the students while using the tool also confirmed that collaboration improved the reflection process as students understood the tool and the data much quicker. After getting a good grasp of what Navi Surface provided, they spontaneously started discussing their progress based on the badges. They reflected on why and how certain badges were achieved and others were not. They also experienced this collaboration to be more fun.
On the other hand, students faced with Navi Surface by themselves were more hesitant and needed input from the observer to continue using the tool. While the questionnaires confirm that they prefer to use it in group, it is clear that without the collaboration and social discourse this interaction enables, the actions are less spontaneous and much less deeper reflection occurs.
The tool was also tested on students who were not part of the CHI course. This however proved less successful especially with students faced with the tool by themselves. As Navi Surface does not provide any details on the actual data behind the badges (blog posts, comments and tweets), the data is very unclear to outsiders. This also affected the SUS score (see above). In section 6, we will discuss how we can provide more detailed information and hereby also make Navi Surface more interesting to outsiders. 6
Learning dashboards provide a means of visualizing the abundance of traces which learning analytics allows us to collect from students. We look at simplifying the data by emphasizing the more important student activities and course goals and visualizing these through badges.
While the personal dashboard has improved perceived awareness with students and the overview of class progress was deemed valuable by teachers and teacher assistants, we believe Navi Surface has more potential in helping awareness and reflection with students through its collaborative nature. In its current state, Navi Surface already enables students to understand the data quicker and plays a catalyst in discussions. Navi Surface makes the process fun and students show interest in using this tool with class mates but also with teachers.
More evaluations will give us deeper insights and we believe that there are still many unexplored possibilities which makes further development of this tool very interesting. By simply adding more course data to the visualizations and allowing students to drill down on badges to reveal more data, students could discover why and how certain badges have been awarded (e.g. the specific blog post or comment that triggered a badge), creating a better insight of the progress and a deeper reflection on the learning process.
Adding more detailed course data to the visualization does not only benefit student and teacher, but could help outsiders comprehend the inner workings of a course better. Open school days can help students choose their future classes based on the real data provided through Navi Surface. Parents’ evenings can become more interactive as parents and teacher can utilize the tabletop to dig deeper into the details of the learning process of son or daughter.
While we mainly work with blog and Twitter data, Navi Surface can easily be extended to support even richer learning analytics data. Students in a more inquiry based learning environment leave behind richer artifacts (e.g. photographs, geographical coordinates) which could provide even more interesting visualizations and therefore an even better insight through Navi Surface. Even MOOCs (Massive Open Online Courses) can benefit from the abstraction to badges and the visualization through Navi Surface.
With few developments, many new possibilities open up. We will therefore continue our research into badges and tabletop displays as a way of improving awareness and reflection in the class room and beyond. 7
The research leading to these results has received funding from the European Community’s Seventh Framework Programme (FP7/2007-2013) under grant agreement No 318499 - weSPOT project.