Learning Analytics Dashboards (LAD) provide a means to leverage data to support learners, teachers, and counselors. This paper reports on an in-depth analysis of how learners interact with a LAD. N=1,406 first-year students in 12 different study programs were invited to use a LAD to support them in their transition from secondary to higher education. The LAD provides actionable feedback about five of the learning skills assessed by the Learning and Study Strategies Inventory (LASSI): concentration, anxiety, motivation, test strategies, and time management. We logged access to and behavior within the LAD and analyzed their relationship with these learning skills. While eight out of ten students accessed the LAD, students with lower time management scores tend to have a lower click-trough rate. Once within the LAD, students with lower scores for specific learning skills are accessing the corresponding information and remediation possibilities more often. Regardless of their scores for any of the other learning skills, learners with higher motivation scores are reading the remediation possibilities for the other four learning skills more often. Gender and study program have an influence on how learners use the LAD. Our findings may help both researchers and practitioners by creating awareness about how LAD use in itself may depend on the context and profile of the learner.
This paper reports on a Learning Analytics Dashboard (LAD) that was offered to N=1,406 first-year students in 12 different STEM (Science, Technology, Engineering, and Mathematics) study programs at the University of Leuven, Belgium. The LAD aimed at supporting students in transition from secondary to higher education with actionable information about five meta-cognitive abilities –further referred to as ‘learning skills’– that contribute to academic achievement: concentration, anxiety, motivation, test strategies, and time management. Students’ learning skills were assessed using the Learning and Study Strategies Inventory (LASSI) at the beginning of the academic year. In total 80.7% of the invited students did click trough to the dashboard, but the response rate was different depending on the profile of the student, as was the level of activity within the LAD.
Previously [
LAD user activity related to other (non-corresponding) learning skills? These questions were inspired by the context of student counseling services in which the project was embedded. Counselors had divergent expectations about if the LAD could reach students with different profiles equally and especially to its appeal to students with lower learning skill scores. In our opinion, understanding how different types of students – in this case, students with different learning skill scores – perceive and interact with LADs provides useful information to improve the design of future student-facing LADs. 2
Learning Analytics (LA) is a relatively young field at the intersection of
theory, design, and data science, and borrows from many related disciplines [
Learning Analytics Dashboards are using data visualization as a technique
to deliver actionable information. An overview of interesting LADs was presented
in [
Learning dispositions are closely related to what we appoint in this paper
as ‘learning skills’. They are described as a concept that “refers to a relatively
enduring tendency to behave in a certain way”, “identified in the action a
person takes in a particular situation” and closely related to ‘competence’, ‘style’,
or ‘capability’ [
To facilitate future comparison and generalization, this section follows the
recommendation [
The dashboard used data about students, their learning skills, and scores that
were already available within the institution, but fragmented across services and
not fed back to the data subjects in a direct way. Fig. 2 describes a high-level
overview of data flows and systems involved. The Learning and Study
Strategies Inventory (LASSI) [
Needs Assessment As in many higher education programs, the first
examination period in the University of Leuven takes place in the middle of the academic
year. Awaiting a first formal assessment, first-year students have only limited
information at their disposal to estimate their own academic abilities. The outcome
of their learning strategies is therefore uncertain. Social-comparison theory
suggests that in absence of objective knowledge about their own position, individuals
turn to comparing themselves to others [
Visual Design The dashboard contained extensive textual information
accompanied by easy to understand charts. Fig. 1 shows the dashboard from the
perspective of a random student in the ‘Engineering Technology’ program. The
content was divided into six tabs (marked by A in the screenshot), the first one
containing an introduction about the objective of the LAD, the origin of the
data, and the connection to the research project. The five subsequent tabs each
went into detail about one specific learning skill. These tabs were offered in
alphabetical order (in Dutch). The upper part of each learning skill tab contained
information about the learning skill definition, the student’s level, and
comparison to peers. The lower part contained similar information about students from
the previous year and linked it to their academic performance measured by the
cumulative study efficiency (CSE), the percentage of obtained credits from total
credits. CSE is a measure for academic progress that is subject to binding (‘30%
rule’) and soft (‘50% rule’) institutional regulations [
Student perceptions Once a user opened a third tab within the dashboard, a yellow message box was shown at the bottom of the screen, asking the the student to answer three simple questions on a 1–5 scale. A minority (14,7%) of students provided complete feedback. Results (see Fig. 3) were generally positive (4–5/5) with respect to the perceived usefulness (89%) and clearness (71%) of the information, and positive but less outspoken (55%) when students were asked if they wanted to receive more of this type of information.
Usability test In-house Human-Computer Interaction researchers and data visualization experts were consulted to improve upon a working prototype of the dashboard. Some, but not all of their suggestions were subsequently implemented, keeping timely delivery in mind. A new iteration of the dashboard is scheduled for the next academic year. This offers an opportunity for additional usability testing and improvement.
Actual effects At the time of design of the dashboard and study, randomized control-trials (RCTs) were found difficult to justify in the context of first-year 0 2 tunoC 04 0 8 0 6 0 2 tunoC 04 0 0 0 1 2 3 4 5 I find this information useful (n= 170 ).
1 2 3 4 5 I find this information clear (n= 169 ).
1I would2like to r3eceive m4ore of 5
this type of information (n= 172 ).
student support. The procedure would require the exclusion of some students
from access to the information which was decided to be ethically questionable.
For an interesting discussion of RCTs and alternatives in the context of distance
learning, see [
Student use In [
The LAD was built on recent work in educational sciences on learning skills, their
role in explaining learning performance in higher education and the specifics
thereof in STEM education. At the same time, the LAD is also an artifact in the
"learning through the act of building" tradition of design science research [
Several student counselors were involved. They shared their expectations about user activity in relation to the learning skills and profile of the student. However, we did not start our analysis from strong a priori theory-driven hypotheses. Rather, we applied a more inductive approach, using exploratory data analysis to spot links, while working toward an integrated model.
We reuse the same type of plot throughout the paper. Figures 4–6, 8 represent
logistic regression information in a compact format [
Once students clicked through, the relationship between learning skill levels and
user activity seemed to be inverted: the top row of plots in Fig. 5 shows a
downward slope for each of the learning skills with regard to reading corresponding
tips: the lower students’ levels, the more likely they were to read the
improvement advice. The difference was found to be significant for each of the learning
skills separately [
A similar picture is sketched for returning a second time (or more) to reread the content related to a weaker learning skill. A one-tailed Man-Whitney test was applied. For concentration (p=1e 4), motivation (p=0.0025), test strategy (p=3e 4) and time management (p=0.0021), the difference in distributions of learning skill scores for students that did return to the corresponding tabs was found to be significant at the 5% level, but not for anxiety (p=0.1546). For some learning skills, an association was found with student’s behavior (reading tips, returning to tabs) for other, non-corresponding learning skills. Reading tips. Figure 6 (top) shows an upward slope for two out of five learning skills, indicating an increased probability of opening at least one of the tips related to the other learnings skills higher for students with higher motivation and test strategy scores. A one-tailed Mann–Whitney test was conducted; only for motivation (p=0.0017) the distributions differed significantly between the group that accessed at least one the non-corresponding tips and the group that did not read any of the non-corresponding tips (p=0.2719 for concentration, p=0.4994 for anxiety, p=0.0975 for test strategy, p=0.4648 for time management). Returning to tabs. Figure 6 (bottom) shows a downward slope for each of the learning skills, suggesting an increased probability of returning to at least one the tabs related to the other learnings skills higher for students with lower scores for any of the learning skills. A one-tailed Mann–Whitney test resulted in support of this at the 5% level for concentration (p=0.0112), anxiety (p=0.0262), motivation (p=0.0369) and time management (p=0.0142), but not for test strategy (p=0.0848). 4.4
Fig. 7 shows that learning skills are not uncorrelated. Some of the outcomes attributed above to one learning skill may be due to underlying effects of an)s 1.0 ( tab 0.8 r e toh 0.6 o t rnu 0.4 t e .rb 0.2 o rP 0.0 other, correlated learning skill. In order to further analyze students’ engagement with the dashboard we constructed several logistic regression models. We added information about study program4 and gender and interrelationships between the learning skills. Our goal was not so much to look for absolute numbers, but rather to check if earlier results remain valid outside of isolation.
The initial predictors for each of the models are the students’ scores for each of the learning skills (CON, ANX, MOT, TST, and TMT), the study program in which they are enrolled (using ‘Bio Engineers’ as reference group) and gender (female=1). In order to find the best fitting predictors, a stepwise regression procedure in both directions based on the Akaike information criterion (AIC) was applied. Model fit was assessed using likelihood ratio chi-square tests. For a summary of results, see Table 1.
The clicked model aimed to predict the probability of students clicking through to the dashboard depending on their profile. Of all students that received an invitation, 80.73% clicked through to the dashboard. Students with higher test strategy scores and students with higher time management scores seemed to have an increased probability to click through, but only for the latter the model yielded a statistically significant result. The study program the student is enrolled in also played a significant role. Gender and the other learning skill scores were not significant. 4 Twelve study programs were grouped into six study program groups: BioEngineering; CBBGG (Chemistry, Biology, Biochemistry-Biotechnology, Geography, Geology), Engineering Science, Engineering Science: Architecture, Engineering Technology, and MIP (Mathematics, Informatics, Physics)
CON
N O C ANX
X N A
The return to tab models aimed to expose which student characteristics play a role in revisiting tabi for learning skill i compared to visiting tabi only once or not. An interesting finding was that students with a lower score for a specific learning skill score i tended to return more often to the corresponding tabi. This finding was present for all return to tab models except for time management. A possible explanation for this could be that most students accessed the tabs from left to right, in the same alphabetical order they were presented. Time management happened to be discussed on the last, outer right tab on the dashboard, thus students did not need to return to the time management tab after a first skimming of the dashboard because they were already on it. Study program was a significant predictor for returning to the anxiety tab, the motivation tab, and the time management tab. Also male students seemed to return more often to a specific tab compared to female students. This was significant at the 5% level for returning to the concentration tab, the motivation tab, and the test strategy tab.
The tips models aimed to predict which students clicked on the ‘Okay, what now?’ button for each tabi, which gave them practical tips to improve the corresponding learning skill. Similar to return to tab , students with a lower scorei were predicted to be more likely to click on the ‘Okay, what now’ button corresponding to the learning skill i. The students’ study program was significant for the anxiety tips, the motivation tips, and the time management tips. Female students seemed to click more often on the tips compared to male students. This was significant at the 5% level for the anxiety and the test strategy tip clicks. Motivation played an remarkable role for all four of the other learning skills: for any given level of concentration, anxiety, test strategy, and time management, an increased level of motivation led to a higher user user interest in the improvement tips. 4.5
Several extended models were tested to include interaction effects of learning skills with gender, interaction of motivation with the other four learnings skills and cross-interaction of all learning skills, whether or not combined with gender interactions. Most of these models provided only limited additional information at the expense of increased complexity. Therefore, we confine ourselves here to the interesting case of interaction between anxiety and gender. Remember that anxiety is measured on an inverted scale: the higher the anxiety score, the lower the level of anxiety.
As shown by Fig. 8 , anxiety exhibits an interaction effect with gender for the probability of returning to the anxiety tab. The slope is less or more straight for male students (left-hand side) and clearly downward for female students. Also the difference in means as shown by the blue vertical lines is negligible for male students, while it is distinct for female students.
To illustrate the impact of the interaction effect, Table 2 provides two simplified logistic regression models, one that tries to predict the return to anxiety tab using only the anxiety level and gender as predictors and a second model that also includes the interaction term. The influence of the anxiety level disappears almost completely for male students in the second model and strongly increases for female students. This paper presented an in-depth analysis of how the learner profile, in particular the learning skill levels of STEM students in transition from secondary to higher education, affected the use of a LAD.
Regarding the question which audience was reached by the LAD, we showed that learning skills may indeed affect the click-through (response) rate for the LAD (RQ-1): the higher the score for a particular learning skill, the more likely students were to access the dashboard. While most learning skills seemed to exhibit such an effect in isolation, only time management did so with statistical significance in a reduced model. The higher the time management skills of the student, the more likely the student was to access the dashboard. One unexplored interpretation may be that students with inadequate time management skills simply missed or forgot our invitation.
As an indicator of the ability of the dashboard to invoke self-reflection, we determined that learning skill levels influenced user activity related to content about those learning skills in our LAD (RQ-2). In most cases, with the exception of returning to the time management tab, which is possibly explained by an order effect, a lower learning skill level tended to lead to increased user activity (revisiting tabs, reading tips) concerning this particular learning skill.
Furthermore, we did show that particular learning skills affected user activity in connection to other learning skills (RQ-3). This was especially the case for motivation. We have reason to believe that motivated students are engaging with the dashboard more intensively because they see it as an opportunity to improve, something we would like to see further explored.
Additionally, we found indications for the influence of study program and gender. Especially the role of gender deserves to be studied more thoroughly. For example, we noted that male students were more likely to reread some of the learning skill tabs, while female students accessed the tips more frequently. Moreover, gender demonstrated an interaction effect that was explicitly outspoken in relation to anxiety and reading anxiety tips.
Our results are pointing into the direction of students using the LAD for self-reflection and to gather actionable insight. However, while our study did target a relatively high number of students (N=1,406) in comparison to most LAD-related work, we did not include a formal assessment of effective impact on learning. Our work suggests that design of LADs may be improved by understanding how students interact with them and how this interaction pattern differs depending on the profile of the student. Future work may include a more fine-grained tracking of how students use the LAD, for instance to learn if students spend more time reading texts or if they are predominantly looking at the data visualization. For this study, only actions within the LAD were tracked. To probe into the actionability of the information provided, data collection could be extended to some of the actions suggested by the LAD: increased LMS activity, electronic scheduling of counseling meetings, registration for a workshop to improve a learning skill. The analysis may also be extended to include study achievement following dashboard usage, for instance to determine if non-use of the LAD is indicative for students at risk of failure.
Acknowledgement. This research is co-funded by the Erasmus+ program of the European Union (562167-EPP-1-2015-1-BE-EPPKA3-PI-FORWARD).