After the CO VID-19 Pandemic: A Study of 7 Spanish Universities

Abstract

The purpose of this article is to present a study that explores the ideal and predicted expectations of academic staf to learning analytics implementation. The study is focused on teaching staf from seven Spanish universities. Results show that teachers valued positively the learning analytics services as an instrument to facilitate feedback to students, and provide access to accurate data that indicates learning progress, and that the institutions will provide the means to guide teachers in accessing student analytics. The aspect that generates the least interest refers to the obligation to act in support of students whose academic results show poor performance. Results also shows that the expectations observed in Spain are somewhat more optimistic than those observed in previous studies in Europe, but clearly more pessimistic than those conducted in Latinoamerica.

Keywords

• Learning analytics adoption
• higher education
• teacher staf perspective
• comparative study

-

CO

VID-19 A CEUR Workshop Proceedings

1. Introduction

Learning Analytics (LA) is a research and practice field that aims at improving learning and teaching drawing on the analysis of traces of data left by the participants in learning experiences [ 1 ]. Despite its significant advances since 2011, adoption of LA solutions by end users, especially by teachers is still scarce [ 2 ]. Some of the most relevant studies regarding LA adoption by students used the SELAQ (Student Expectations of Learning Analytics Questionnaire) [ 3 ], an instrument that was adapted to study teachers’ expectations towards LA in their practice in diferent countries in Europe [ 4 ] and Latin America [ 5, 6 ]. The instrument collects data about ideal expectations (expected results that stem from individual aspirations), and predicted expectations (practical beliefs regarding what teachers consider achievable) considering the implementation of LA through 16 questions organized into 4 categories: 1) Goals of LA, 2) Teachers’ needs for LA services, 3) Teachers’ perceptions about students’ needs for LA services, and 4) Challenges regarding implementation of LA services at the universities. For detailed description of SELAQ items see Appendix A in [ 4 ].

Since cultural values and social/geographical contexts afect the users’ vision about tools and their applicability [ 7 ], it is relevant to study whether there is any diference between the results obtained in other geographical contexts and in Spain. To address this issue, this paper presents an overview and preliminary results of the SELAQ+ study undertaken by the Spanish Network of Learning Analytics (SNOLA)1. The main goal of the study was to analyse the expectations regarding LA of university teachers in Spain, compare them with those of previous studies, and to analyse if diferences in reported expectations could be related to other characteristics of the respondents. A previous work has already analysed teachers’ perspectives in Spain but it only included one university and it was based on a qualitative approach without using SELAQ [ 8 ]. In addition, the work reported in [ 4 ] used the SELAQ questionnaire for teachers’ expectations but it only included the same Spanish institution. The responses of the same SELAQ+ questionnaire were independently collected in each participating university, and were unified in a single dataset for the data analysis. This paper reports the results related to the two following research questions: • RQ1. What are the results of ideal and predicted expectations? • RQ2. How diferent are the results in Spain with respect to those obtained in previous equivalent studies?

The structure of the paper is as follows: in the next section, a description of the SELAQ+ study is given as well as the methodology in use; in section 3 the results of the study are explained and discussed according to the above research questions. Finally, section 4 contains the main conclusions.

2. The SELAQ+ study

The SELAQ+ study, carried out between September and December 2022, used the teacheroriented version of the SELAQ instrument used in studies with teachers [ 4, 9 ]. The study was called SELAQ+ because a specific question was added to the SELAQ questionnaire in order to eventually capture the changes brought about by the efect of the intensive use of ICT during the COVID-19 pandemic. This topic falls outside the scope of this paper and will be further analysed in future publications.

The study was conducted in accordance with the GDPR and approved by the ethics committees of the participating universities. The final questionnaire was distributed by SNOLA members and resulted in 490 responses being collected from the 7 participating universities: Universidad Autónoma de Madrid (70), Universidad Nacional de Educación a Distancia (108), Universitat Oberta de Catalunya (61), Universitat Pompeu Fabra (92), Universidad del País Vasco (49), Universidad de Valladolid (74) and Universidad de Vigo (36). The teachers had an average of 18.90 years of experience (standard deviation 10.94), 209 and 272 were female and male, respectively (9 preferred not to declare their gender). According to the area of knowledge, 211 teachers correspond to the Science, Technology, Engineering and Mathematics (STEM) field, 249 to Humanities, 17 to Health Sciences and 13 to other categories.

To answer RQ1 we firstly report on descriptive statistics, and secondly we assess the diferences in the means of the scales (ideal and predicted) through a Wilcoxon statistical signed-rank test. Particularly, the efect sizes obtained from the Wilcoxon test provide insights into the magnitude of the diferences between the ideal and predicted samples [ 10 ]. The larger the efect size on the SELAQ items, the worse the teachers’ expectation that the institution will provide the means to make LA efective enough. Regarding RQ2, the comparison with previous studies gives us the opportunity to contextualize our results. Results regarding RQ2 are reported using the same format in the baseline studies to facilitate comparability.

3. Results

3.1. RQ1 What are the results of ideal and predicted expectations?

Regarding RQ1, the results presented in Table 1 show that the means of ideal expectations were higher than predicted expectations for all items, and significant statistical diferences were found for every case (p-value < 0.001). The items most valued by the respondents (>6 in the 7-points Likert scale) were Q11 (Feedback format), Q09 (Data accuracy), Q04 (Class progress) and Q01 (Guidance). Item Q14 (Obligation to act) achieved the lowest score as well as the lower efect size (r = 0.33). We consider the Q14 to be the one least desired by teachers, but the one that they feel their institutions would be inclined to provide the means to achieve. It should be noted that all the items have efect sizes greater than 0.3, corresponding to medium and high efects according to Cohen’s criterion [ 11 ]. In terms of predicted expectations, the most valued items were Q04, Q12 (Complete profile), Q01 and Q16 (Understanding learning). Note that Q04 and Q01 also appear among the most highly rated in terms of ideal expectations while the Q14 (Obligation to act) is again the less desired item.

3.2. RQ2 How diferent are the results in Spain with respect to those obtained in previous equivalent studies?

Regarding RQ2, the SELAQ+ results were compared with previous studies conducted with the SELAQ instrument for teachers in: i) Europe (2017), ii) Brazil (2021), and iii) Chile and Ecuador (2018). These studies involved several universities with diferent levels of implementation of learning analytics, and the main findings are described below.

i) In general terms, the comparative analysis shows that in ideal expectations the SELAQ+ item scores are higher than the scores in the Europe study [ 4 ], while the predicted expectations in SELAQ+ were lower. This suggests more positive values for the services that should be provided and greater scepticism about what is actually expected from the institutions. The comparison between the results of SELAQ+ and the cases of the Europe study shows that, in terms of ideal expectations, the scores of the former are lower than those of the SP (Spain), NL (Netherlands), and UK (United Kingdom) cases, afecting a diferent number of items in each case. Compared to the SP case, the SELAQ+ scores are lower in the items Q03 (Shared experience), Q05 (Data access), Q06 (Student agency), Q08 (Regular updates), Q09 (Data accuracy), Q12 (Complete profile), Q13 (LA in feedback), and Q16 (Understanding learning), while compared to predicted expectations, the SELAQ+ scores are lower in all items. Compared to the NL case, the SELAQ+ study shows lower scores for items Q01 (Guidance), Q02 (Professional development), Q04 (Class progress), Q09 (Data accuracy), Q11 (Feedback format), and Q13, and a similar score for item Q06. In the case of the UK, the SELAQ+ scores on ideal expectations are lower for all items. However, SELAQ+ predicted expectation scores are lower in the UK case for items: Q01 (Guidance), Q02 (Professional development), Q03 (Shared experience), Q04 (Class progress), Q05 (Data access), Q07 (Early intervention), Q08 (Regular updates), Q14 (Obligation to act), and Q15 (Skill development). In contrast to the above cases, the case of Estonia (EST) is peculiar in that the ideal expectations scores in the SELAQ+ study are lower for almost all items, except for Q06 and Q14 (Obligation to act). Similarly to the SP and NL cases, the scores for predicted expectations in the SELAQ+ study are lower, except for item Q12.

ii) In the case of the study presented in [ 5 ], we observe that the participants in the Brazilian universities were more enthusiastic: 5.78 in SELAQ+ versus 6.35 in Brazil. For all items, the scores obtained in the SELAQ+ are lower than those obtained in the Brazilian study, both for ideal and predicted expectations. For all items, the ideal expectations scores exceed the 6 points, except Q14 (Obligation to act) while in SELAQ+ only 4 items exceed the 6 point scores, and by coincidence Q14 is also the worst valued. In the Brazilian study the highest efect size corresponds to Q08 (Regular updates) and the lowest corresponds to Q03 (Shared experience), while in SELAQ+ are the Q11 (Feedback format) and Q14 respectively.

iii) Our results have also been compared with those of the LATAM study, carried out in four universities in Ecuador and Chile [ 6 ]. In this case, the benchmark data refers to the percentage of teachers who scored above 6 points in the SELAQ survey. The comparison shows that for all items the percentages in SELAQ+ are lower than in the LATAM study, except for item P11, which has the same percentage in both studies and is also the highest (see figure 1). The largest diference was observed on item P14, which could be interpreted as LATAM participants being more willing to support students whose educational records show low performance in terms of ideal expectations.

4. Discussion and conclusions

The teachers surveyed are generally in favor of the implementation of LA services in their institutions, placing greater interest in those aspects that constitute challenges in the implementation of LA in the institutions. Teachers point out the need to receive information in the right format, accuracy and scope, as well as the need for guidance to efectively access data. In relation to these aspects, the overall expectations observed in the SELAQ+ study are somewhat more optimistic than those observed in the Europe study in ideal expectations terms, but clearly more pessimistic than the LATAM studies [ 4 ]. The aspect that generates the least interest is the one that refers to the obligation to act in support of students whose analytical results show poor performance, which is interpreted as a rejection of the possible increase in workload that this would entail.

The study has a number of limitations. It did not cover some Spanish regions and the sample is limited. Some of its findings need further insight by means of qualitative approaches to understand them better. In spite of these limitations the results of this study contribute with new knowledge to the discussion about adoption of LA in an international and diverse society. The conclusions may be used as a basis for further research regarding this topic, and to foster discussion with institutions about the use of LA in higher education.

Acknowledgments

This research is part of grants RED2018-102725-T and RED2022-134284-T, and of projects PID2019-105951RB-I00, PID2020-112584RB-C31, PID2020-112584RB-C32, PID2020-112584RBC33 and TED2021-131787B-I00 funded by MICIN/AEI/10.13039/501100011033, as well as the grant IT-1437-22 (ADIAN) by the Department of Education, Universities and Research of the Basque Government.

References

[1] LAK '11: Proceedings of the 1st International Conference on Learning Analytics & Knowledge , Association for Computing Machinery, New York, NY, USA, 2011 .

[2] Y.-S. Tsai , D. Gasevic , Learning analytics in higher education - challenges and policies: A review of eight learning analytics policies , in: Proceedings of the Seventh International Learning Analytics & Knowledge Conference , LAK '17, Association for Computing Machinery, New York, NY, USA, 2017 , p. 233 - 242 . URL: https://doi.org/10.1145/3027385.3027400. doi:1 0 . 1 1 4 5 / 3 0 2 7 3 8 5 . 3 0 2 7 4 0 0 .

[3] A. Whitelock-Wainwright , D. Gašević , Y.-S. Tsai , H. Drachsler , M. Schefel , P. J. MuñozMerino , K. Tammets , C. Delgado Kloos, Assessing the validity of a learning analytics expectation instrument: A multinational study , Journal of Computer Assisted Learning 36 ( 2020 ) 209 - 240 .

[4] K. Kollom , K. Tammets , M. Schefel , Y.-S. Tsai , I. Jivet , P. J. Muñoz-Merino , P. M. MorenoMarcos , A. Whitelock-Wainwright , A. R. Calleja , D. Gasevic , et al., A four-country crosscase analysis of academic staf expectations about learning analytics in higher education , The Internet and Higher Education 49 ( 2021 ) 100788 .

[5] T. Pontual Falcão , R. Lins Rodrigues , C. Cechinel , D. Dermeval , E. Harada Teixeira de Oliveira, I. Gasparini , R. D. Araújo , T. Primo , D. Gasevic , R. Ferreira Mello , A penny for your thoughts: students and instructors' expectations about learning analytics in brazil , in: Proceedings of the Seventh International Learning Analytics & Knowledge Conference , LAK '22, Association for Computing Machinery, New York, NY, USA, 2022 , p. 186 - 196 . URL: https://doi.org/10.1145/3506860.3506886. doi:1 0 . 1 1 4 5 / 3 5 0 6 8 6 0 . 3 5 0 6 8 8 6 .

[6] I. Hilliger , M. Ortiz-Rojas , P. Pesántez-Cabrera , E. Scheihing , Y.-S. Tsai , P. J. Muñoz-Merino , T. Broos , A. Whitelock-Wainwright , M. Pérez-Sanagustín , Identifying needs for learning analytics adoption in latin american universities: A mixed-methods approach , The Internet and Higher Education 45 ( 2020 ) 100726 .

[7] O. Viberg , I. Jivet, M. Schefel , Designing Culturally Aware Learning Analytics: A Value Sensitive Perspective , Springer International Publishing, Cham, 2023 , pp. 177 - 192 . URL: https://doi.org/10.1007/978-3- 031 -27646-0_ 10 . doi:1 0 . 1 0 0 7 / 9 7 8 - 3 - 0 3 1 - 2 7 6 4 6 - 0 _ 1 0 .

[8] P. J. Muñoz-Merino , P. M. Moreno-Marcos , A. Rubio-Fernández , Y.-S. Tsai , D. Gašević , C. Delgado Kloos , A systematic analysis of learning analytics using multi-source data in the context of spain , Behaviour & Information Technology ( 2022 ) 1 - 15 .

[9] D. Turnbull , R. Chugh , J. Luck , Transitioning to e-learning during the covid-19 pandemic: How have higher education institutions responded to the challenge? , Education and Information Technologies 26 ( 2021 ) 6401 - 6419 .

[10] M. Tomczak , E. Tomczak, The need to report efect size estimates revisited. an overview of some recommended measures of efect size ., Trends in sport sciences 21 ( 2014 ).

[11] J. Cohen , A power primer , Psychological Bulletin 112 ( 2020 ) 155 - 159 .