Validation of the Learning Self-Regulation Questionnaire: The Peruvian Case

Abstract

This research aims to validate the instrument called Learning Self-Regulation Questionnaire adjusted to the Peruvian context, for which reliability and validity analyses were carried out. Cronbach's alpha, McDonald's Omega, and ordinal alpha were used to test its reliability for construct validity. The instrument was applied to 355 university students enrolled in in-person, blended, and e-learning modalities and comprised 47 Likert scale 5-point questions. The results obtained from Cronbach's alpha, McDonald's Omega, and ordinal alpha were above 0.7 and 0.8 for all constructs, which indicates that the instrument is reliable for obtaining responses on the dimensions mentioned in the questionnaire, while the model presented good overall fit indices. In conclusion, the results presented herein show good validity and reliability, thus making this instrument use feasible.

Keywords

• eol>Learning self-regulation 1
• Planning 2
• Metacognitive3 1

1. Introducction

The pandemic caused by COVID-19 led to school and university closures due to the risk of infection, so it became necessary to adopt new learning spaces, especially in the digital infrastructure, through the remote modality, to conduct synchronous sessions using virtual platforms. The other prevailing modality was distance learning through online activities, video conferences, forums, and evaluations. These technological resources allowed students to continue learning despite the complex circumstances worldwide because of the lockdown [ 1 ]. This whole scenario significantly impacted education due to the massive and untimely implementation of virtuality, which has become a core element at universities over the last two years, establishing itself as an unlimited source of education that poses challenges to all of us.

Ante Given this scenario, reviewing and researching relevant aspects identified to improve the students’ learning is imperative. Thus, it is worth identifying how they have adapted to virtual settings when faced with this new learning environment, and the strategies adopted for them to continue studying at present after their return to the face-to-face modality. This research seeks to delve into the students’ self-regulation when learning, highlighted by [ 2 ], wherein the significance of identifying students’ adjustment to this new setting to achieve academic success was pointed out. Furthermore, as stated by [ 3 ], self-regulation is crucial for virtual learning, as it allows and forces students to manage their time, so a broader and more effective use of these

0000-0002-7856-9876 (Elisa Montoya-Cantoral); 0000-0002-0370-6581 (V. Pinto-Guillergua); 0000-0003-09845846 (P.E. Gómez-Carbonel); 0009-0004-2820-589X (E. Zeña-Raya) 0000-0002-8621-7942 (K. Villalba-Condori) © 2023 Copyright for this paper by its authors.

Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

CEUR Workshop Proceedings (CEUR-WS.org) types of strategies is required compared to non-virtual environments, where students receive inperson support. This is reinforced by [ 4 ] and [ 5 ] as they indicate that self-regulation has played an essential role in learning experiences during lockdown; nonetheless, the actual effect of these new experiences on these processes is still not known.

Hence, it is crucial to first identify that self-regulation is understood as the individuals’ ability to self-direct themselves to successfully perform their activities, while being aware of their cognitive, mental, and socio-affective skills [ 6 ]. In other words, a self-regulated student can build scenarios that favor and help them learn efficiently, hence, they are responsible for making decisions and acting to attain their objectives. Along the same lines, several authors agree that this is neither a linear nor static process but somewhat cyclical, comprised of three phases: planning, execution, and self-reflection [ 7 ] [ 8 ]. Thus, self-regulation in students is constantly faced with constant changes depending on the context.

On the other hand, the number of research conducted in the educational field has significantly increased in recent years. In this context, the translation, adaptation, and implementation of different instruments to gather data have become crucial since the correct adaptation of these instruments will help obtain reliable and valid data to guarantee the quality of the study [ 9 ]. It is worth highlighting the availability of a wide range of instruments that measure self-regulated learning, such as the questionnaire adapted to Spanish by [ 10 ], as these delve into its phases and the effectiveness for its measurement.

Among these is the questionnaire used by [ 11 ], which was designed to be something other than directly applied to university students, although it highlights the adjustment of several writing-related items to the digital context in response to a study applied to higher education students from Macau The questionnaire used, called the English Self-Regulated Learning Questionnaire (ESRLQ), introduces digital writing practices, such as taking notes on digital platforms or reviewing different text genres (such as videos, for example) that reinforce autonomous learning. It is based on the proposal made by [ 12 ], arranged in 48 items intertwined in the following dimensions: (a) Self-evaluation; (b) Goal setting and planning; (c) Organization and transformation; (d) Review and memorization; (e) Search for social assistance; (f) Persistence; (g) Search for opportunities; and (h) Notes taken.

In a complementary way, the work of [ 13 ] is an abbreviated form of the Motivated Strategies for Learning Questionnaire (MSLQ), and the work of [14] is applied to the Colombian context of a university in Santa Marta. The short version of the questionnaire contains 40 items, although not divided into the three dimensions set out by Zimmerman. In line with this latest research, two features characterize the work conducted by [15]: i) it is adjusted to the Argentine reality regarding differentiating sociocultural practices and technology access; and ii) the original MSQF proposal was reduced, from 80 items to 41. The questionnaire is designed so that information is retrieved from students more accurately. Despite not being arranged by Zimmerman’s dimensions, some of them can be recognized: (a) Self-evaluation; (b) Goal setting and planning; and (c) Self-reflection.

This is why creating a self-regulated learning instrument became necessary, considering that the Peruvian higher education level has unique characteristics, such as its education system and its linguistic, social, and cultural diversity. These peculiarities influence the students and their self-regulating strategies to self-direct themselves throughout their professional training. On the other hand, the lack of an instrument adapted to the reality of Peruvian higher education limits professors and researchers in constructing effective educational interventions that help promote students’ self-regulation.

For this reason, this research aims to validate the Learning Self-Regulation Questionnaire for the Peruvian context. This adaptation is based on a meticulous comparative content analysis of the Motivated Strategies for Learning Questionnaire (MSQL) [16] and the Motivated Strategies for Learning Questionnaire - Short Form (MSQL SF) adjusted to the Colombian context [17].

2. Method

2.1 Sample

Reliability and validity analyses were conducted to validate the Learning Self-Regulation Questionnaire adjusted to the Peruvian context. Cronbach's alpha, McDonald's Omega, and ordinal alpha were used to test its reliability and construct validity, conducted through an exploratory factor analysis (EFA) and a confirmatory factor analysis (CFA).

To select participants, neither a strict nor necessarily random sample procedure is required for this validation study, made up of 355 students enrolled in the Faculty of Business Sciences (19.4%), Faculty of Health Sciences (27.4%), Law School (7.9%), Faculty of Human Sciences (20.3%) and Faculty of Engineering and Architecture (25.4%). The surveyed participants studied under distance (36.9%), face-to-face (58, 6%) and blended (4.5) modalities. As for their progress in their careers, 38% are in their first year, 40% in the second year, 12% are currently completing the third year, 6% are in the fourth year, and 3% of them are completing their fifth year, and their ages range from 18 30 years old (69.6%), 31 ++ 40 years old (19.7%), and older than 40 (10.7%), as shown in Table 1.

This instrument is based on the theoretical foundations of the cyclical model of self-regulated learning by Zimmerman (2002), and the contributions made by Pintrich (1991) to understand student motivation and its relationship with self-regulation. Similarly, the instrument proposed by [15] (2021) is founded on the abovementioned authors. On this basis, the instrument comprises 47 questions items, each distributed and adapted to the phase model that Zimmerman proposed. The first, known as Planning, considers the Self-motivating beliefs sub-phase, subdivided into Goal setting, Self-efficacy, and Value of tasks, whereas the second phase (Execution) includes the sub-phase Self-control, divided into Metacognitive self-control (Metacognitive), with its branches: Search for help, Environment, Time and strategies and Motivational self-control (Motivational). Finally, the third phase (Self-reflection) is responsible for monitoring and managing emotions once the results of the task or activity are received. Each item is answered online through a 5-point Likert scale: Always (5), Most of the times (4), Occasionally (3), Rarely (2) and (1) Never.

2.3 Procedure

The instrument's reliability

The following procedures were implemented for questionnaire validation purposes: To test the reliability of the questionnaire, Cronbach's alpha, McDonald's Omega and ordinal alpha, measures used to ensure the instrument’s internal consistency, were calculated. As per [18], the acceptable value is a coefficient equal to or higher than 0.70. The software included SPSS, version 26, Jamovi, and the Factor Analysis program.

Exploratory and confirmatory factor analyses

Exploratory and confirmatory factor analyses were carried out to establish the construct validity. As for the exploratory factor analysis, the principal component method was employed to extract the factors with the maximum data variance for each construct under study. To determine the factor loadings of each of the items that are part of every construct, oblique rotation by the Promax method was considered appropriate, given that the factors (constructs) found must be strongly correlated to form a new construct or second-order factors, so the abovementioned method is chosen [19]. Factor loadings in oblique rotation tend to be lower, but a value above 0.30 is an acceptable minimum [20].

Bartlett’s sphericity test and the Kaiser-Meyer-Olkin (KMO) Measure of Sampling Adequacy will be used to determine whether conducting a Factor Analysis is appropriate. The KMO index must be above 0.75 to consider that carrying out a factor analysis is very adequate, whereas 0.5 is acceptable, but a value lower than 0.5 indicates that a factor analysis is unacceptable. The results were obtained through the program SPSS, version 26.

Conversely, the confirmatory factor analysis was applied, allowing for correcting some flaws resulting from the EFA. These models provide the appropriate statistical framework to evaluate the validity and reliability of each item in constructing a measurement instrument. The software used was Amos, version 26.

3. Results and Discussion

Reliability Analysis

This study evaluated reliability using three internal consistency coefficients: Cronbach's alpha, McDonald's Omega, and Ordinal Alpha. The first is used for quantitative variables or scales with at least 5 categories based on Pearson’s correlations [21], whereas McDonald's Omega is specific for Likert scale variables or others with fewer response options, based on communality [22]. Another more robust alternative for ordinal variables is the Ordinal Alpha coefficient, which relies on polychoric correlations [23].

As shown in Table 2, good McDonald's Omega indices were obtained, above 0.7 and 0.8 for all constructs, which indicates that the instrument is reliable for getting answers on the dimensions mentioned in the questionnaire. Likewise, Cronbach's alpha indices are also above 0.7 and 0.8, except for the motivational construct, which showed an acceptable index, and the data resulting from the analysis are similar, both for Cronbach's alpha and McDonald's Omega, although even better results are obtained through the Ordinal Alpha coefficient. In this regard, Omega and Ordinal Alpha are more accurate indicators for the ordinal measurement level, which allows for confirming that the questionnaire's internal consistency is good. In other words, similar responses may be obtained if this questionnaire is applied again.

Construct Validity

Objectives

Self-efficacy

Task

Strategies

Environment

Support

Cronbach’s

Alpha

McDonald's

Omega

Ordinal Alpha 0.814 0.805 0.726 0.852 0.776 0.701 0.635 0.869 0.817 0.811 0.742 0.854 0.782 0.709 0.707 0.871 0.852 0.864 0.809 0.886 0.827 0.751 0.768 0.906

Once the questionnaire’s reliability was established, construct validity was performed through an exploratory factor analysis and a confirmatory factor analysis, which allowed the researchers to determine the potential existence of concepts not initially made explicit by the researcher in the theoretical-empirical structure supporting the instrument’s design.

Exploratory Factor Analysis (EFA)

The results in Table 3 show the different adequacy measurements of Kayse-Meyer-Olkin (KMO) tests for the constructs Planning, Metacognitive, Motivational and Self-Reflection, equal to 0.888, 0.899, 0.656, and 0.86, respectively. In other words, KMO tests are very good for almost every value, as they exceed 0.75 to apply the factor analysis that ensures that the dimensions proposed in the questionnaire are measured. Nonetheless, an acceptable KMO was obtained for the Motivational construct. In addition, Bartlett’s sphericity test was significantly high (p_value˂0.00) for all constructs.

Based on the previous processes, the factor structures of each of the constructs were analyzed according to the percentage of cumulative explained variance, as shown in Table 3. Almost all are above 60%, which means that it is valid to measure the phases of self-regulation of learning established in the dimensions, obtaining 62.1% in the Planning construct, 52.6% for the Metacognitive construct, 62.6% obtained in the Motivational construct and the highest, Selfreflection (65.8%). Finally, factor loadings are above 0.7 on average, except for a few items (which obtained 0.31 and 0.39), regarded as acceptable for oblique rotation, thus confirming its capacity to measure the self-regulation phases.

Confirmatory Factor Analysis (CFA)

The CFA allows for correcting flaws resulting from the EFA; flow charts represent the graph according to the proposed constructs. Rectangles symbolize the observable items or variables, while ellipses represent non-observable factors, constructs, or variables. Unidirectional arrows express saturations, and bidirectional arrows indicate correlation; these models provide the appropriate statistical framework for assessing the validity and reliability of each item when a measurement instrument is developed. In the CFA chart, it is essential to observe the factor loadings that allow us to know the weight of the regression coefficient between the observed variable and its corresponding construct. The closer to one another, the greater the relationship.

The results indicate that the regression weights, error variances, and correlations are statistically significant. As shown in Figure 1, the standardized regression weights, or factor loadings of the items with the dimension are mostly above 0.7, confirming its validity as an instrument to measure self-regulation.

In addition, Figure 1 shows a second-order CFA chart. First, factor loadings or standardized regression weights between each observed variable and the construct are identified, with values ranging between 0.51 and 0.81. That is to say, the construct Environment is highly related to the five observable variables and is also significant (p_value˂0.00). Similarly, the other constructs, such as Strategies, Support, Motivational, Self-Reflection, Objectives, Self-Efficacy and Task, are highly related to their respective observable variables.

In the second-order, factor loadings are even greater, ranging from 0.66 to 0.92; i.e., the ratio between the first and second-order constructs is high. The Planning construct shows high ratios for its Objectives (0.74), Self-Efficacy (0.76), and Support (0.92). Similarly, the Metacognitive construct is positively related to Strategies (0.91), Environment (0.66), and Support (0.84).

Table 4 shows that the proposed model meets several overall fit criteria, with a value of 1.852 for the normalized chi-square index and an RMSEA (root mean square error of approximation) of 0.046, below 0.08, and even below 0.05. It presents two good criteria for the incremental index of CFI (comparative fit index) and IFI (incremental fit index). Also, their parsimony adjustment indexes in the PRATIO (parsimony ratio), PNFI (Parsimony Normed Fit Index), and PCFI (Parsimony Comparative Fit Index) are good. Therefore, the results presented here show good validity and reliability, making the use of this instrument feasible. It is important to mention that items 16, 20, 31, 35, 36, 39, and 41 were removed as their loading was lower than the rest, coinciding from the theoretical point of view.

These results align with the MSLQ-SF questionnaire developed by [24], with a high degree of reliability α = 0.70 for its application. Likewise, it focuses on coping strategies; the instrument has internal validity and is derived from the MSLQ version, where its internal congruence is observed. Thus, its use to measure motivation and learning strategies among Spanish students is feasible. Furthermore, [15] found that the instrument helps understand the students’ learning dynamics in Colombia. A value of 0.883 was obtained from the Cronbach’s alpha scale, yielding a positive reliability level to the instrument fitted to the 37-item instrument resulting from the full MSQF version. The main conclusion reached is that this instruments internal structure has been found to be functional for educational measurement purposes, although this does not immediately translate into a valid instrument for the Colombian population. This can be interpreted based on the multidimensionality of the Colombian educational system and the country's social diversity.

Conclusion

In conclusion, good indices were obtained for Cronbach’s Alpha, McDonald’s Omega, and Ordinal Alpha, above 0.7 and 0.8 for all constructs; these levels indicate that the instrument is reliable for obtaining responses about the dimensions mentioned in the questionnaire, and the model showed good overall fit indices. Therefore, the results presented in this study show good validity and reliability which indicates the fesiablity of this instrument.

The planning phase shows a high validity and reliability index in its construction, with a KMO index of 0.888, which allows for relying on the formulations for questions used to measure this phase of the self-regulation cycle in university students. It is worth mentioning that, in the planning phase, the value aspect of the task is acceptable, but below the other dimensions; this information allows institutions to develop and implement strategies that can enhance this aspect. The same results were obtained for the rest of the constructs, thus confirming reliability on the instrument’s validity.

The University of Lima has funded this project [14] F. S. Tinoco, E. B. Heras, A. Hern y L. Zapata, “Validación del cuestionario de motivación y estrategias de aprendizaje forma corta-MSLQ SF, en estudiantes universitarios de una institución pública-Santa Marta”, Psicogente, vol. 14, no. 25, 2011. [15] J. Masso, M. Romero, R. Fonseca y A. Gallego, “Cuestionario de motivación y estrategias de aprendizaje forma corta-MSLQ SF en estudiantes universitarios: análisis de la estructura interna”, 2021. [16] B. Zimmerman, A. Kitsantas y M. Campillo, “Evaluación de la Autoeficacia Regulatoria: Una Perspectiva Social Cognitiva”, Revista Evaluar, vol. 5, no. 1, pp. 01-21, 2005. DOI: 10.35670/1667-4545.v5.n1.537 [17] P. Pintrich y E. De Groot, “Motivational and Self-Regulated Learning Components of Classroom Academic Performance”, Journal of Educational Psychology, vol. 82, no. 1, pp. 3340, 1990. DOI: 10.1037/0022-0663.82.1.33 [18] J. C. Nunnally, “Una descripción general de la medición psicológica”, en B. B. Wolman (eds), Diagnóstico clínico de trastornos mentales, Springer, Boston, MA, 1978. DOI: 10.1007/9781-4684-2490-4_4 [19] S. Lloret-Segura, A. Ferreres-Traver, A. Hernández-Baeza e I. Tomás-Marco, “El Análisis Factorial Exploratorio de los Ítems: una guía práctica, revisada y actualizada”, Anales de Psicología, vol. 30, no. 3, pp. 1151-1169, 2014. DOI: 10.6018/analesps.30.3.199361 [20] R. B. Kline, “Book Review: Psychometric theory (3rd ed.)”, Journal of Psychoeducational

Assessment, vol. 17, no. 3, pp. 275-280, 1999. DOI: 10.1177/073428299901700307. [21] P. Elosua Oliden y B. D. Zumbo, «Coeficientes de fiabilidad para escalas de respuesta categórica ordenada», PST, vol. 20, n.º Número 4, pp. 896–901, dic. 2008. [22] A. F. Hayes y J. J. Coutts, “Utilice Omega en lugar del Alfa de Cronbach para estimar la confiabilidad. Pero…”, Métodos y medidas de comunicación, vol. 14, no. 1, pp. 1-24, 2020.

DOI: 10.1080/19312458.2020.1718629 [23] A. Freiberg Hoffmann, J. B. Stover, G. de la Iglesia y M. Fernández Liporace, “CORRELACIONES POLICÓRICAS Y TETRACÓRICAS EN ESTUDIOS FACTORIALES EXPLORATORIOS Y CONFIRMATORIOS”, Ciencias Psicológicas, vol. 7, no. 2, pp. 151-164, 2013. Recuperado en 30 de octubre de 2023, de http://www.scielo.edu.uy/scielo.php?script=sci_arttext&pid=S168842212013000200005&lng=es&tlng=es. [24] F. Zurita-Ortega, A. Martinez-Martinez, R. Chacon-Cuberos y J. L. Ubago Jiménez, “Analysis of the Psychometric Properties of the Motivation and Strategies of Learning Questionnaire— Short Form (MSLQ-SF) in Spanish Higher Education Students”, Social Sciences, vol. 8, no. 5, pp. 1-11, 2019. DOI: 10.3390/socsci8050132

References

[1] A. Boshoff-Knoetze , L. Duminy , and Y. Du Toit , “ Examining the effect of self-regulation failure on academic achievement in emergency remote teaching and learning versus face-to- face , ” Journal of Applied Research in Higher Education , vol. 15 , no. 2 , pp. 342 - 354 , 2023 , doi: 10.1108/JARHE-08-2021-0305.P. S. Abril, R. Plant , The patent holder's dilemma: Buy, sell, or troll? , Communications of the ACM 50 ( 2007 ) 36 - 44 . doi: 10 .1145/1188913.1188915.

[2] R. Cerezo , M. Esteban , G. Vallejo, M. Sanchez-Santillan , and J. C. Nunez , “ Differential Efficacy of an Intelligent Tutoring System for University Students: A Case Study with Learning Disabilities,” Sustainability , vol. 12 , no. 21 , 2020 , doi: 10.3390/su12219184.

[3] P. Uceda , L. Polo , and G. Cruzado, “ Chatbot as a remote learning self-regulation strategy in pandemics times | Chatbot como estrategia de autorregulación del aprendizaje remoto en tiempos de pandemia,” in Proceedings of the LACCEI international Multi-conference for Engineering, Education and Technology, 2021 . doi: 10 .18687/LACCEI2021.1.1.327.

[4] D. Eckley , A. Allen , P. Millear , and K. T. Rune, “ COVID-19's impact on learning processes in Australian university students,” SOCIAL PSYCHOLOGY OF EDUCATION , vol. 26 , no. 1 , pp. 161 - 189 , 2023 , doi: 10.1007/s11218-022-09739-x.

[5] D. Al-Shaikhli , “ The effect of the tracking technology on students' perceptions of their continuing intention to use a learning management system,” Educ Inf Technol (Dordr) , vol. 28 , no. 1 , pp. 343 - 371 , 2023 , doi: 10.1007/s10639-022-11156-8.

[6] B. J. Zimmerman , “ Investigating Self-Regulation and Motivation: Historical Background, Methodological Developments , and Future Prospects”, http://dx.doi.org/10.3102/0002831207312909, vol. 45 , núm . 1, pp. 166 - 183 , mar. 2008 , doi: 10.3102/0002831207312909.

[7] A. Hadwin , S. Järvelä y M. Miller , “ Self-regulation, co-regulation and shared regulation in collaborative learning environments ”, 2017 . DOI: 10 .4324/9781315697048-6.

[8] Pintrich , P. R. , Smith , D. A. F. , García , T. y McKeachie , W. J. , “ Fiabilidad y validez predictiva del cuestionario de estrategias motivadas para el aprendizaje (MLSQ)”, Medición educativa y psicológica , vol. 53 , pp. 801 - 813 , 1993 .

[9] R. Hernández , C. Fernández y M. Baptista , Metodología de la investigación, Sexta. México: Mc Graw Hill , 2014 . Consultado: oct. 06 , 2022 . https://www.uca.ac.cr/wpcontent/uploads/2017/10/Investigacion.pdf

[10] T. Rojas-Ospina y M. Valencia-Serrano , “Adaptación y Validación de un Cuestionario Sobre Estrategias de Autorregulación de la Motivación en Estudiantes Universitarios”, Psykhe (Santiago) , vol. 28 , no. 1 , pp. 1 - 15 , 2019 . DOI: 10 .7764/psykhe.28.1.1128B.

[11] B. J. Zimmerman y A. Kitstantas , “Fases de desarrollo en auto-Regulación: Cambio de objetivos de proceso a objetivos de resultado”, Revista de Psicología Educativa, vol. 89 , no. 1 , pp. 29 - 36 , 1997 .

[12] P. Pintrich y T. García, “ Self-Regulated Learning in College Students: Knowledge, Strategies, and Motivation”, Advances in Motivation and Achievement , vol. 7 , 1991 . DOI: 10 .1016/S0749- 7423 ( 08 ) 60040 - 460040 -4)

[13] C. Wang y B. Bai , “ Validating the instruments to measure ESL/EFL learners' self-efficacy beliefs and self-regulated learning strategies”, TESOL quarterly , vol. 51 , no. 4 , pp. 931 - 947 , 2017 .