Massive Open Online Courses (MOOCs) phenomenon is the new frontier of online learning, where un-limited time and no location restrictions allow users to follow di erent strategies of learning. In the learning analytics literature there are many contributes dealing on how MOOC learners' behaviour a ects their performance and in uences reaching the course achievements. The present paper proposes a statistical model to analyse the relationship among learning, performance and engagement in a MOOC framework. As MOOCs o er di erent forms of learning, it is necessary to consider engagement and learning as multidimensional concepts, measurable using several indicators. The network of relationships is estimated through Partial Least Squares Path Modeling and di erences in learners' behaviour according to age are also explored.
MOOCs phenomenon became popular in recent years as widely used learning
tools in higher education institutes both in traditional and distance universities.
Since MOOCs o er a variety of learning instruments, such as video lectures,
multiple-choice quizzes, discussion forums and documents, the customization of
learning analytics (LA) to the MOOC framework represents an important
challenge [
M. Carannante et al.
directly measurable. The analysis refers to one of the courses of IPSAMOOCs
series o ered by the Platform FedericaX, the EdX MOOCs platform of the
"Federica WebLearning" Center at University of Naples Federico II [
Learning and engagement can be de ned as two important drivers of students'
performance. The present study aims at modeling the e ect of these two
components on the outcome of students attending a MOOC. As either learning
and engagement and performance are complex concepts that cannot be directly
measured by a single indicator, the rst part of the study has been devoted to
their conceptualisation and operativization [
The paper explores the simplest structure relating the previous dimensions and sub-dimensions: the model is shown in Figure 1 where the direction of the arrows goes from the driver to the dependent concept (numbers on the arrows will be explained in Section 3). Further developments will regard the exploration of more complex structures with more connections (e.g. from engagement
Learning and engagement assessment in MOOCs ... to learning and vice-versa). For easiness of interpretation, the polarity of the indicators related to Procrastination has been reversed. The asterisk symbol has been assigned to the indicators with reversed polarity.
In the model, Performance is the outcome dimension, that is it estimates the degree of e ective learning of users. For this block, we consider a unique variable, the rate of correct responses with respect to total quizzes. The full list of the indicators adopted for each dimension and sub-dimension is shown in Table 1 (the last column will be described in Section 3, while numbers form 1 to 5 in the label column refer to the modules in which the course is structured). 3
PLS-PM is a consolidated statistical method able to model complex
multivariate relationships among blocks of observed indicators (also known as manifest
variables, MVs) and unobserved concepts named latent variables (LVs). A
PLSPM is made of a measurement (or outer) model relating each block of MVs to
its corresponding LV and a structural (or inner) model connecting the LVs in
accordance with a network of linear relationships. The PLS-PM algorithm
allows to estimate separately the blocks of the measurement model and the
structural model. In the PLS-PM terminology, Engagement, Learning and
Performance and their related sub-dimensions represents LVs while the indicators
described in Table 1 are the MVs. Figure 1 represents the adopted structural
model. For further methodological details about PLS-PM see [
A preliminary analysis has been carried out to check unidimensionality and internal consistency of LVs taking into account the di erence between the rst and the second eigenvalue (the 1st eigenvalue is expected to be the only one greater than 1 and much higher than the second one), the Cronbach's and the Dillon-Goldstein's (grater than 0.7 in case of unidimensionality). M. Carannante et al.
The estimation of the measurement part of the model allows to measure the importance of each indicator on the corresponding LV. Such measures, named outher weights, are shown in Table 1. All the coe cients are statistically signi cant at the 0.01 level using the classical t-test where standard errors are estimated through a bias-corrected bootstrap approach with 200 samples.
For Frequency-based activity, active days and rate backward give less contribution to the variable explication than the others, this means that learners pay more attention to the number and the kind of activities than to perform these activities as many days as possible, and they do not consider so important coming back on videos. For Time-based activity, the greatest contribution is given by video completion and the lowest by video time. This implies that it is not important how much time learners spend on videos, but if they watch the whole video. Participation to problems (rate problem) has a very high impact on Interaction with respect to rate forum. For Regularity, interval days, ave time1 and rate return give a very few contribution, this implies that learners' regularity is less conditioned by the interval of days between activities, by the activities of the rst subsection and the return on previous subsections. Finally, for No Procrastination the size of the coe cients increases moving from the rst module to the last, meaning that as the course progresses, it is much more important for students to be on time in starting an activity as soon as the learning materials are available.
The estimation of the structural part of the model allows to measure the impact of each sub-dimension on the related LV and of Engagement and Learning on Performance. Such weights, named path coe cients, are the numbers on the
Learning and engagement assessment in MOOCs ... arrows in Figure 1. The model suggests that Performance is mainly a ected by Learning (coe cient equal to 0.75) while Engagement plays a quite irrelevant role (coe cient equal to 0.03) even not signi cant. On one hand to improve Learning it is advisable to act on actions related to Frequency-based activity which has the highest impact on Learning (coe cient equal to 0.71), almost three times greater than Time-based activity and Interaction. On the other hand, Engagement can be improved acting on Regularity (coe cient equal to 0.55) and reducing Procrastination (coe cient equal to 0.49) as well.
The analysis of the contribution (expressed by the path coe cient) of each
determinant (i.e. each explanatory LV in the structural model) to the
Performance can be deepened through an importance-performance map (IPMA) [
A further exploration has been carried out to evaluate if the estimated relationships change in case of an observed heterogeneity, for example related to age. In order to identify di erences by age we consider two groups of students: up to 24 years, that is the age of scholar or academic students, and over 24, that is the age of self-regulated learners. The group of over 24 years students signi cantly di ers from the youngest students showing higher coe cients with respect to Frequency-based activity, Interaction and Engagement. The di erent role played by engagement is particularly interesting as it results a driver of performance just for self-regulated learners. Further explorations of group di erences can be realised using importance preference mapping.
In conclusion, we performed a multivariate model to estimate the concepts of Learning, Engagement and Performance and to measure the relationships among them. Moreover the proposed PLS-PM also allows to explore lower order relationships providing the impact of each sub-dimension and of each indicator.