Learning Analytics (LA) consists of using educational data to inform teaching strategies and management decisions, aiming to improve students' learning. The successful implementation of LA in Higher Education Institutions (HEIs) involves technical aspects and infrastructure but also stakeholders' acceptance. The SHEILA framework proposes instruments for diagnosis of HEIs for LA adoption, including stakeholders' views. In this paper, we present the results of the application of SHEILA's surveys to identify the highest and lowest expectations about LA adoption, in the views of students and instructors, and compare their ideal and realistic expectations. Results confirmed the high interest in using LA for improving the learning experience, but with ideal expectations higher than realistic expectations, and point out key challenges and opportunities for Latin American researchers to join eforts towards building solid evidence that can inform educational policy-makers and managers, and support the development of strategies for LA services in the region.
As the amount of educational data increases, and tools for analysis become more available,
Learning Analytics (LA) becomes more popular [
In Latin America (LATAM), LA adoption is still much lower than in North America and
Europe [
However, LA implementation is not straightforward, and is highly dependent on context [9, 10]. Although interest in LA has grown considerably around the world [11, 9], few studies specifically address the key role of contextual factors in implementing LA successfully at the institutional level [8]. The SHEILA framework (Supporting Higher Education to Integrate Learning Analytics) [12] is the main such initiative, providing instruments to build a diagnosis of HEIs in terms of several aspects that impact on the successful adoption of LA. As SHEILA is grounded in empirical research undertaken in the European context [12], the LALA project (Learning Analytics in Latin America) [13] encourages local adaptations of its methods and instruments, aiming at generating a corpus of knowledge and contextual evidence for the region.
The SHEILA framework comprises dimensions that include political context, internal capacity, engagement strategy and learning frameworks [12]. Perhaps most importantly, it recommends the identification of key stakeholders and their needs and desires. As a matter of fact, stakeholder engagement and buy-in is considered a challenge for successfully implementing LA, besides pedagogical grounding, resources, and ethics and privacy [9]. As stakeholders diagnosis is very particular to regional specificities, including for example culture, bureaucracy, and social inequality, existing research based on SHEILA [12, 9] may not account for LATAM HEIs. There is yet few findings about the impact of stakeholders’ opinions and behaviors for LA adoption in Latin America.
In this paper, we address this gap with empirical research in a Brazilian HEI, presenting stakeholders’ opinions and perceptions that can help increase buy-in in the process of implementing LA. Such collective efort in gathering empirical evidence has been pointed out by other LATAM researchers [8]. Previous research performed through focus groups indicate students and instructors’ interest in LA, in particular for improving the learning process, providing and receiving personalized feedback, adapting teaching practices to students’ needs, and making evidence-based pedagogical decisions [14, 15]. The present research complements such qualitative findings with quantitative data from a survey using a questionnaire focused on stakeholders’ ideal and predicted expectations [16, 10]. We aimed to answer the following research questions: RQ1: What are the highest and lowest expectations regarding the adoption of LA, in the views of students and instructors? RQ2:What are the diferences and similarities between students’ and instructors’ ideal and predicted expectations about the adoption of LA?
The instrument used for data collection was based on SHEILA’s survey [16, 10], empirically tested and aiming for a diagnosis of HEIs at scale, by providing a comparison between ideal and predicted (or realistic) expectations from the main stakeholders groups (students and instructors). The instrument itself prompts participants to rate their expectations in two separate 7-point Likert scales: ideal and predicted (explained in the instrument). In this paper, ideal expectations are desired outcomes based on the hope stakeholders have, while predicted expectations are realistic beliefs about what is perceived as viable to be implemented. By analyzing these two kinds of expectations, a deeper understanding of stakeholders’ perspectives can be reached, identifying main areas to focus. Generally speaking, topics that receive the highest ratings in realistic expectations are considered priority in service planning [16]. We translated the questionnaire to Brazilian Portuguese, making small semantic adaptations to fit the context of Brazilian HEIs. We also removed a question about sharing the students data for a third party company as in Brazil it is not possible for public universities to share data with private companies. Questions from the adapted questionnaire for the instructors were maintained.
The questionnaire included a brief introduction to LA and the purpose of the study, asking for informed consent for participation. We also collected demographic information, such as age and gender, and educational data (course, study field, degree, among other information). The themes addressed by the survey were: (i) Data Privacy (4 items for students): Whether the university is allowed to collect, use and analyze the data obtained from the students and for what purpose the institution may use these data. (ii) Academic Progress (6 items for instructors, 2 for students): What kind of information could benefit students and instructors helping to check on students’ progress in the courses. (iii) Feedback (4 items for instructors, 3 for students): How students would like to receive feedback / what are the ways of giving feedback that instructors find the most appropriate. (iv) Decision-making (2 items for instructors, 1 for students): How educational data can help students and instructors take action upon problematic situations identified. (v) Intervention (1 item for instructors, 1 for students): Whether the instructors or the institution should intervene when being notified by the system of a student at risk, and how this should be approached. (vi) Training (3 items for instructors): What kind of training for instructors will be provided for them to be capable of analyzing data efectively.
This study was undertaken in a HEI that ofers face-to-face and online courses, with access to the same LMS (Moodle). While online courses occur fully through this platform, in the face-to-face courses the LMS is used as support to share materials, submit assignments and interact in online discussions.
The questionnaires were created using Google Forms, and sent through the university oficial communication channels, including social networks and emails lists from departments and direct contact with course coordinators. The survey had 241 participants from the HEI (192 students and 49 instructors), from several areas of knowledge and courses (online and face-toface) (Tables 1 and 2). The higher number of participants from Information Technology (IT) courses is due to the authors’ belonging to this area thus having better reach.
The quantitative analysis adopted to answer the first research question focused on the description of the survey results into two boxplots, which include the median rating score of each item for ideal and predicted expectations, and the outliers for each scale.
In order to address RQ2, we compared ideal and realistic expectations from students and instructors. For this analysis, only participants inclined to agreement were considered, i.e. those who answered 5 to 7 in the Likert scale. We performed statistical analysis over this sample and we assessed the percentage of agreement in instructors’ and students’ responses (separately) and the comparison between ideal and realistic expectation. More specifically, we applied the McNemar test [17] that performs a statistical comparison of two related samples. In this analysis, we aimed to reach 95% of reliability.
Instructors’ responses are shown in the boxplot in Figure 1, where the vertical lines mark the lowest, median and highest values; the outer limit of the boxes show the first and third quartiles; and the dots correspond to outliers. So for instance, in Q4, about visualizing students’ progress, almost all instructors rated their ideal expectations as 7 (with 3 outliers only); but regarding realistic expectations the answers ranged mainly from 5 to 7 (being 2 the lowest rating).
Instructors had high ideal expectations about their institution adopting LA, but were less optimistic about the viability (median rating scores between 5 and 6). The items with almost unanimous highest ideal expectations were: access to students’ progress (Q4-I and Q5-I), university support on data analysis (Q7-I), understanding of data (Q11-I), learning profile (Q12-I) and visualization of learning performance (Q16-I). Some of these also had the highest median ratings of perceived feasibility (Q4-I, Q5-I, Q11-I, Q12-I and Q16-I). The item about university support on data analysis (Q7-I) oscillated between agreement and neutrality, with the biggest interval (answers between 3 and 7).
Students had high ideal expectations as well, but lower than instructors’ expectations (Figure 2). The items with highest realistic expectations had the median rating scores between 5 and 7, i.e., higher values than those expressed by the instructors. Students’ highest expectations regarded consent for use of their educational data (Q2-S) and use of data for other purposes (Q5-S); accessing their educational progress (Q3-S) and educational goals (Q7-S). The biggest gap between median ratings (3-7) was found in Q10-S, regarding intervention based on LA indicating that a student is at-risk of failing or dropping out.
Table 3 shows the results of the analysis of instructors’ answers, where "n" refers to the number of participants inclined to agree with the item (having answered 5-7 in the Likert scale) and "%" is the percentage of the total number of participant instructors. There were significant diferences between instructors’ ideal and realistic expectations for the majority of items, but ideal expectations were higher. Q4-I and Q5-I were the only two items with similarity between expectation and reality, with high levels of agreement. These items were about instructors accessing students’ data on courses they are teaching or have taught previously, indicating that they think that this is viable in their present context. It was not necessary to perform Bonferroni adjustment, as all statistical tests had significance values less than or equal to 0.01.
In this section, we discuss the survey results considering research in other countries using the same instrument [8, 10, 16], as well as our previous qualitative results from the same HEI using the SHEILA instruments for focus groups to investigate similar themes [18, 15].
Data analysis showed that instructors and students had positive views about the adoption of LA in their institution, which confirms results from other contexts [ 10, 16], and our previous ifndings [ 18, 15]. The survey results add that these stakeholders have ideal expectations higher than realistic expectations, i.e. they wish for LA to be implemented, but are unsure about its viability in a foreseeable future, considering the context of their institution. Previous research using the same survey instrument in other HEIs [10, 16] also showed ideal expectations scale with a ceiling efect, with ideal expectations higher than the realistic, reinforcing the tendency of stakeholders’ uncertainty about what can be achieved in their present context.
According to the survey, instructors are particularly interested in visualizing students’ progress, learning profiles and performance, consonant with findings from the focus groups [18, 15] previously performed, which indicated instructors’ particular interest in: decreasing students’ dropout; improving students’ learning and their own teaching; and viewing students’ progress. Although in the focus groups, instructors were somewhat reluctant about the access to and use of students’ data (in line with other research findings [ 10]), fearing that this could become intrusive, the survey shows that they consider access to student data viable, even at present (items related to this topic – Q4-I and Q5-I – showed similarity between ideal and realistic expectations). Meanwhile, they were less optimistic about the support they can get from HEIs to help them analyze and understand this data, and act upon it (Q7-I) (also previously identified in the literature as an important challenge [10]).
According to the survey, students were also especially interested in visualizing their progress and keeping track of their learning goals. This is in line with qualitative findings, which indicate that students particularly support the adoption of LA with the purpose of improving their learning experience. The use of such educational data was of little concern for students in the focus groups [18, 15], but the survey indicates very high ideal expectations that the HEIs will keep this data safe (Q2-S) (reinforced by previous similar results [16]). As for the use of personal data, students were more cautious, which was confirmed by the survey results, where asking for consent to use their data (Q1-S) appeared as an important aspect, and one that they considered rather feasible in their present context.
In the focus groups, students were interested in better feedback through the identification of weaknesses in their learning and suggestions to improve it (confirming findings in [ 8]), which is aligned with previous evidence that students need meaningful information about their progress to motivate them to improve and remain engaged [16]. Students were in favor of the system alerting instructors early if they were at-risk of failing a course or could improve, but there were also reflections on their own responsibility for their learning. For their part, instructors in the focus groups mostly agreed with the obligation for teaching staf and/or HEIs to take action when dificulties in students’ learning are identified by LA methods, consonant with [8]. However, in the survey, this same topic (Q14-I) presented a large diference between instructors’ ideal and realistic expectations, and had the lowest ratings of agreement, indicating that instructors were in fact unsure about this obligation, as also identified in [ 10]. Students were also uncertain about the viability of instructors being obliged to take action when they are identified as underperformers or at-risk (Q10-S, lowest percentage of agreement and larger diference between ideal and realistic expectations). These somewhat contradictory findings reflect the hot topic still open to discussion, about the moral obligation instructors would have to act, versus students’ need to be autonomous and responsible for their learning [10, 19, 16].
This study presented the findings of a survey aimed at investigating stakeholders’ expectations on the adoption of LA in a Brazilian HEI, thus adding empirical evidence to the research eforts towards guiding the development of LA services in LATAM [8]. Following qualitative research undertaken previously through focus groups [18, 15], the present study aimed to complement evidence with a quantitative analysis that included a larger number of participants and a comparison between ideal and realistic expectations of key stakeholders.
The main limitation of the research is the small size of the sample, given that in the HEI, the population of the instructors and students is around 1.200 and 17.000, respectively. Additionally, a large part of the responses were provided by students and instructors from IT related courses, who are most likely to accept the use of new technologies in their context.
Our evidences, taken in perspective along with other research within LALA and SHEILA projects, in LATAM [8] and globally [16][10], reinforce the importance of stakeholder engagement for a successful implementation of LA. Together, the empirical evidence collected so far by researchers reveal convergent findings, such as: the need for HEIs to ensure all collected data is safely kept, within a transparent process with stakeholders’ consent; the benefits that LA can bring to the learning process by shedding light on students’ needs and making this visible for them and for the instructors; the wish students have for timely and quality feedback; and the need expressed by instructors for institutional support to help them understand data and take efective action upon them.
Our research and other surveys on the same topic and using the same instrument [10, 16] show ideal expectations above realistic. The reasons for this disparity may vary substantially in diferent contexts, including instructors’ self-eficacy, familiarity with technology and analytics, institutional resources, bureaucracy, and data privacy legislation. Given the particularities of Latin America since colonization, which led to deep socioeconomic inequality, lack of resources and systemic institutional eficiency [ 8], stakeholders’ wishes may be more distant to their actual beliefs than in other regions of Europe and North America. The lack of belief in the country’s institutions, the lack of self-belief, and low levels of familiarity with technology can be barriers to stakeholder buy-in, thus important aspect to be considered and addressed by administrators.
Another key topic, with divergent expectations in the literature, is about the responsibility to act, once data become available. Instructors’ opinions vary about how much they should be expected to take action, for example to contact and help students at-risk. Some researchers and educators argue that the students, on being informed of their progress with rich information, should take responsibility for their learning, with instructors’ support. In other words, who should be the protagonist once data is visualized by all? Instructors’ "obligation to act" is still in debate [19], along with discussions on the risk of discouraging students’ autonomy and creating a culture of passivity. This involves complex pedagogical and political decisions that need to be carefully considered, while maintaining instructors’ and students’ autonomy.
For future work, we intend to broaden the survey and extend the study to managers and
institutional leaders, based on the SHEILA framework. Additionally, we want to establish
partnerships with other Brazilian and Latin American institutions, to run similar studies and
further compare the results. In this way, we hope to help creating evidence that reflects the
identity(ies) of Latin America [