This paper reports on our ongoing research around the use of learning analytics technology for awareness and self-reflection by teachers and learners. We compare two case studies. Both rely on an open learning methodology where learners engage in authentic problems, in dialogue with the outside world. In this context, learners are encouraged to share results of their work, opinions and experiences and to enrich the learning experiences of their peers through comments that promote reflection and awareness on their activity. In order to support this open learning process, we provided the students with StepUp!, a student activity visualization tool. In this paper, we focus on the evaluation by students of this tool, and the comparison of results of two case studies. Results indicate that StepUp! is a useful tool that enriches student experiences by providing transparency to the social interactions. The case studies show also how time spent on predefined high level activities influence strongly the perceived usefulness of our tool.
This paper reports on a comparison of two recent experiments with learning
analytics. In our view, learning analytics focuses on collecting traces that learners
leave behind and using those traces to improve learning [
Our courses follow an open learning approach where engineering students
work individually or in groups of three or four on realistic project assignments in
an open way. Students use twitter (with course hash tags), wikis, blogs and other
web 2.0 tools such as Toggl1 and TiNYARM2., to report and communicate about
their work with each other and the outside world in a community of practice
kind of way [
Students share their reports, problems and solutions, enabling peer students to learn from them and to contribute as well. However, teachers, assistants and students themselves can get overwhelmed and feel lost in the abundance of tweets, blog posts, blog comments, wiki changes, etc. Moreover, most students are not used to such a community based approach and have difficulties in understanding this process. Therefore the reflection on the activity of the community can help users to understand what is going on and what is expected of them.
In this paper, we present two follow-up studies to our earlier work [
In our courses, we encourage students to be responsible of their own learning activities, much in the same way as we expect them to be responsible of their professional activities later on. In order to support them in this process, our studies focus on how learning dashboards can promote reflection and self awareness by students. To this end, we consider different ways to capture traces and to identify which traces are relevant to visualize for the users. Finally, we analyze how visualizing these traces affects the perception and actions of the learner.
These experiments rely on the design, implementation, deployment and evaluation of dashboards with real users in ongoing courses. We evaluated our prototypes in two elaborate case studies: in the first case study, we introduced StepUp! to the students at the beginning of the course, visualizing blog and twitter activity and time reported on the different activities of the course using Toggl. They could access the tool but it was not mandatory. After a period of time, we evaluated the tool with students by using a questionnaire and Google Analytics3 to track the actual use of the tool.
In the second case study, StepUp! visualized student activities from blogs,
twitter and TiNYARM, a tool to track read, skimmed and suggested papers in
a social context [
As time tracking is so prominent in what we visualize, we also discuss the importance of tracking time on high-level definition of activities and the potential differences between automatic and manual tracking of the data.
The remainder of this text is structured as follows: the next section presents our first case study, in a human-computer interaction course. Section 3 describes
the second case study, in a master thesis student group. Results are discussed in Section 4. Section 5 presents conclusions and plans on future work. 2 2.1
One of the main challenges with learning analytics is to collect data that reflect
relevant learner and teacher activities [
Some activities are tracked automatically: this is obviously a more secure and
scalable way to collect traces of learning activities. Much of our work in this area
is inspired by “quantified self” applications [
We rely on software trackers that collect relevant traces from the Web in the form of digital student deliverables: the learners post reports on group blogs, comment on the blogs of other groups and tweet about activities with a course hash tag. Those activities are all tracked automatically: we basically process RSS feeds of the blogs and the blog comments every hour and collect the relevant information (the identity of the person who posted the blog post or comment and the timestamp) into a database with activity traces. Similarly, we use the twitter Application Programming Interface (API) to retrieve the identity and timestamp of every tweet with the hash tag of the course.
Moreover, we track learner activities that may or may not produce a digital outcome with a tool called Toggl: this is basically a time tracking application that can be configured with a specific set of activities. In our HCI course, we make a distinction between the activities reported on in this way, based on the different tasks that the students carry out in the course: 1. evaluation of google plus; 2. brainstorming; 3. scenario development; 4. design and implementation of paper prototype; 5. evaluation of paper prototype; 6. design and implementation of digital prototype; 7. evaluation of digital prototype; 8. mini-lectures; 9. reading and commenting on blogs by other groups; 10. blogging on own group blog.
The first six items above correspond to course topics: the students started with the evaluation of an existing tool (Google Plus6) and then went through one cycle of user-centered design of their own application, from brainstorming over
scenario development to the design, implementation and evaluation of first a
paper and then a series of) digital prototype(s) [
When students use Toggl, they can do so in semi-automatic mode or manually. Semi-automatic mode means that, when they start an activity, they can select it and click on a start button. When they finish the activity, they click on a stop button. Manually means that the students have to specify activity, time, and duration to Toggl. In this way, students can add activities that they forgot to report or edit them manually. Of course, on the one hand, this kind of tracking is tedious and error prone - hence the manual option. On the other hand, requiring students to log time may make them more aware of their time investment and may trigger more conscious decisions about what to focus on or how much time to spend on a specific activity.
The main course objective is to change the perspective of how they look at software applications, from a code-centric view to a more user-centric view. That is an additional reason why self-reflection is important in this context. 2.2
Figure 1 illustrates how the data are made available in their complete detail in our StepUp! tool: this is a “Big Table” overview where each row corresponds with a student. The students are clustered in the groups that they belong to. For instance: rows 1-3 contain the details of the students ‘anneeverars’, ‘ganji ’ and ‘greetrobijns’ (see marker 1 at Figure 1). These three students work together in a group called ‘chigirlpower’, the second column in the table (marker 2). The green cells in that second column indicate that these students made 8, 9 and 13 posts in their group blog respectively (marker 3). Rows 4-6 contain the details of the second group, called ‘chikulua12‘: they made 1, 4 and 18 comments on the blog of the first group (column 2) and 9, 6 and 9 posts in their own blog (column 3) respectively (marker 4). The rightmost columns (marker 5) in the table indicate the total number of posts, the total number of hours spent on the course (Toggl) and the total number of tweets.
The two rightmost columns are sparklines[
As is obvious from Figure 1, this is a somewhat complex tool. Originally, the idea was that this would mainly be useful for the teacher - who can indeed provide very personal feedback to the students, based on the in-depth data provided by the table. However, somewhat to our surprise, and as illustrated by Figure 2 and Figure 3, this overview is used by almost all students once per week, for an average of about 10 minutes.
Nevertheless, in order to provide a more personalized and easy to understand
view that students can consult more frequently, which is important for awareness
support, we have developed a mobile application for these data (see Figure 4)
that we released recently, as discussed in future work section below.
We carried out a rather detailed evaluation six weeks into the course, based on
online surveys. In the evaluation, we used five instruments, in order to obtain a
broad view of all the positive and negative issues that these could bring up:
1. open questions about student opinions of the course;
2. questions related to their awareness of their own activities, those of their
group and those of other groups;
3. opinions about the importance of the social media used in the course;
4. questions about how StepUp! supports awareness of their own activity, that
of their group and of other groups;
5. a System Usability Scale (SUS) evaluation focused on the tool [
Another goal of our evaluations is to gather new requirements to improve the course and the deployed tools. This task becomes complex because sometimes students are not aware about the goals of the course.
Below, we summarize the main outcomes of this evaluation.
Demographics In total, 27 students participated in the evaluation; they are between 20 and 23 years old and include 23 males and 4 females. All the participants are students of the Human Computer Interaction course.
Open Questions For the open questions, the students were asked about positive and negative aspects of the course, and they were asked how they would improve the course.
Overall, the use of the learning analytics seems to be well received, as illustrated by the following quotes: “I like the interactive courses. As professor Duval said himself, it allows him to adjust us faster. We (the students) keep on the right track. Otherwise, we might do a lot of worthless work and thus lose valuable time we could invest better in other ways in this course.” or “The course is different from any courses I taken before as there is class participation, immediate feedback etc.”. Neither the negative aspects mentioned, nor the suggestions to improve the course related to the use of learning analytics. Awareness We asked students questions on whether they think they are aware of how they, their group and the other students in class spend efforts and time in the course, and whether they consider this kind of information important.
Overall, the students think that they are very aware of their own efforts, just a little bit less aware of the efforts of the other members in their group, and less aware of the efforts by members of other groups - Figure 5 (left box plot) provides more details.
StepUp! support As illustrated by Figure 5 (right box plot), students evaluate the support by StepUp! for increased awareness rather positively: the students agree that the tool reinforces transparency, that it helps to understand how peers and other students invest efforts in the course. This is important because these data suggest that the tool does achieve its main goal.
SUS questionnaire Overall, the SUS usability questionnaire rating of StepUp!
is 77 points on a scale of 100. This score rates the dashboard as good [
The second case study ran with 13 master students working on their master thesis. All of them work on HCI topics such as music visualization and augmented reality. In this case study, most students work individually on their thesis topics, except for two students who work together on one topic.
As in the previous case study, they report their progress on blogs, share opinions and communicate with their supervisors and each other on twitter. In addition, they use TiNYARM. The use of this tool is intended to increase the awareness of supervisors and students. They can suggest papers to each other, see what others have read and read papers that are suggested to them.
In our previous experiment [
These two students work together on a thesis topic (augmented reality). The green cells in that second column indicate that these students made 17 and 15 posts in their blog respectively (marker 2). Row 3 contains the details of another student who is working individually on his thesis: he made 2 comments on the blog of the group working on augmented reality (column 2) and 43 posts in his own blog (column 3) (marker 3). The rightmost columns in the table indicate the total number of tweets and read, skimmed, suggested and to read papers (marker 4).
The rightmost column is a sparkline that provides a quick glance of the overall evolution of the twitter, blog and TiNYARM activity for a particular student. They can be activated to reveal more details of student activity (marker 5). 3.3
We carried out the same detailed evaluation as in the previous case study. However, in this case study, students had not accessed the tool before. The idea behind of this evaluation setup was to analyze how the use or not use of the tool before influenced the perceived usefulness of the tool.
Demographics In total, 12 students participated in the evaluation; they are between 21 and 25 years old and include 10 males and 2 females. Open Questions For the open questions, the students were asked about positive and negative aspects of the course, and they were asked how they would improve the course.
Overall, the use of social networks seems to be well received, as illustrated by the following quotes: “The blogs are a good way to get an overview of what everyone is doing. ” or “Having a blog is also a good thing for myself, because now I have most of the information I processed in one place.” Awareness We asked students questions on whether they think they are aware of how they, and the other students in class spend efforts in the course, and whether they consider this kind of information important.
Overall, the students think that they are very aware of their own efforts and less aware of the efforts by other members of the course - Figure 7 (left box plot) provides the details. These results are similar to the previous case study. StepUp! support As illustrated by Figure 7 (right box plot), students evaluate the support by StepUp! different from the previous case study. They consider that StepUp! provides better transparency, but indicate that this tool is less useful to understand how others spend their efforts. As we discuss in the next section, time seems to be a really useful indicator to understand how others are behaving, being this the main difference with the previous use case.
One of the students remarked that he would have liked to realize earlier his low activity on commenting blogs, an all the rest agreed that they should have been more active in the use of social networks.
SUS questionnaire Overall, the SUS usability questionnaire rating of StepUp!
is 84 points on a scale of 100. This score rates the dashboard as almost excellent
[
Although the usability results can be encouraging, results of this case study indicate that StepUp! is less useful to understand the efforts of peer students. As Toggl data was not included in the visualizations of this case study, this may have affected this perceived usefulness. These results indicate that further evaluation studies are required to assess the impact of visualized data to support awareness. 4
The field of learning analytics has known explosive growth and interest recently.
Siemens et al. [
In this paper, we have presented two case studies. The first study focuses on visualizing social network activity and complementarily time reporting on predefined activities in a course that follows an open learning approach. The second case study focuses exclusively on the social network activity.
Time is a commonly used indicator for planning. Based on the European
Qualification Framework of higher education, degrees and courses have been
assigned a number of credits called European Credit Transfer System (ECTS).
Each of these credits have an estimation of time, one credit is approximately 30
hours. Therefore, time spent seems to be a good indicator to take into account
for reflection and to check whether the time spent by the student in the course
is properly distributed. Time is also used in empirical studies[
However, time tracking is not an easy task. Manual tracker systems and
applications such as Trac[
From the evaluations and discussion above is clear that many open research
issues remain. We briefly discuss some of them below.
1. What are relevant learner actions? We track tweets and blog posts and ask
students to track their efforts on specific course topics and activities.
However, we track quantitative data that tells us little or nothing about the
quality of what students do. Obviously, these data provide in some sense
information about necessary conditions: if the students spend no time on
particular topics, then they will probably not learn a lot about them either.
However, they may spend a lot of time on topics and not learn a lot. Or they
may be quite efficient and learn a lot with little investment of time. It is
clear, that we need to be quite careful with the interpretation of these data.
2. How can we capture learner actions? We rely on software trackers for laptop
or desktop interactions, and social media for learner interactions (through
twitter hash tags and blog posts and comments). We could further augment
the scope of the data through physical sensors for mobile devices. However,
capturing all relevant actions in an open environment in a scalable way is
challenging.
3. How can we evaluate the usability, usefulness and learning impact of
dashboards? Whereas usability is relatively easy to evaluate (and we have done
many such evaluations of our tools), usefulness, for instance in the form of
learning impact, is much harder to evaluate, as this requires longer-term and
larger-scale evaluations.
4. How can we enable goal setting and connect it with the visualizations, so as to
close the feedback loop and enable learners and teachers to react to what they
observe and then track the effect of their reactions? We are experimenting
with playful gamification approaches, that present their own challenges [
Our main goal with StepUp! is to provide students with a useful tool and to empower them to become better students. From our point of view, they should work in an open way sharing their knowledge with the world and having some impact in others opinion.
StepUp! supports our open learning approach providing more transparency in the social interaction. It provides students an opportunity to reflect on their activity to take a look to this quantitative data and see how others are performing within the community.
Time tracking seems to be a useful indicator for students to understand how students spend their efforts and to increase awareness on the course activity. Furthermore, usefulness of a tool is not only based on conclusions driven by visualizations. How we collect the traces also influences such a factor. To this end, manual and automatic tracking require more research. Design is also a factor that influences the use of our application. To this end, we are currently experimenting with other approaches. For instance, we have currently deployed a mobile web application (see Figure 5) that provides a quick overview and indicators on their activity. We expect to reduce the cognitive efforts making them more attractive to use these tools.
In conclusion, we believe that a sustained research effort on learning analytics dashboards, with a systematic evaluation of both usability and usefulness, can help to make sure that the current research hype around learning analytics can lead to real progress. As we already mention in section 2, we propose to deploy new versions of StepUp! on different devices to research how devices can influence the reflection process from a Human Computer Interaction perspective, for instance evaluating the profile view (Figure 4) for mobile devices. Furthermore, as explained in section 4, we are interested mainly to figure out the relevant traces for the students, to involve sensors to track external data and to enable goal setting. 6
This work is supported by the STELLAR Network of Excellence (grant agreement no. 231913). Katrien Verbert is a Postdoctoral Fellow of the Research Foundation -Flanders (FWO). The work of Jose Luis Santos has received funding from the EC Seventh Framework Programme (FP7/2007-2013) under grant agreement no 231396 (ROLE).