The relationship between student engagement and learning outcomes has been extensively studied in the context of online learning. However, it has been less investigated in face-toface learning. In this paper, we describe initial findings from a study of student engagement and outcomes at a 'bricks-and-mortar' (BaM) university, where engagement is characterized by a diverse set of systems and agents, spanning both physical and digital spaces. We ask whether the substantial relationship often found in online environments between engagement and outcomes, holds in a BaM setting. We present initial analysis of data traces from various sources, each relating to a different dimension of engagement. Initial results indicate a weak relation between engagement and outcomes, suggesting that this important relationship may be substantively different in face-to-face/BaM and online learning environments. These preliminary findings highlight the need for further research, tackling challenges which are specific to face-toface learning in a bricks-and-mortar university environment.
Student engagement and learning outcomes are amongst the most ill-defined and
broadly interpreted theoretical concepts, for which there are no currently agreed upon
frameworks for operationalization [
Online learning is not analogous to face-to-face learning and each requires
different conceptualization and operationalization frameworks [
The phrase “student engagement” has come to refer to the level of involvement
students appear to have within their classes and their institutions in the context of
learning [
It is enormously challenging to measure the depth of understanding at a
coursespecific learning outcome [17, 19]. Module results usually include assessment tools
that are defined by clarifying specific learning objectives [13]. There are important
differences between face-to-face and online learning, including the pedagogical basis
for assessment. Instructivism, which is common in BaM's face-to-face learning,
maintains that knowledge should be transferred directly from the instructor to the
learner without further interactions [15]. On the other hand, social constructivism is
often implemented in collaborative online learning environments, whereby the teacher
is seen as a facilitator between students, content and platforms, and social interactions
are more central [
Engagement has been shown to correlate with performance in online learning. In this study, we ask how this relation manifests in a BaM university. More specifically, we attempt to determine what types of interactions and student characteristics can predict specific learning outcomes. Working with data from a traditional BaM university in the UK, we have collected data from various university systems for 30,781 undergraduate students across three academic years commencing in Autumn 2013, 2014 and 2015. Tables 1, 2 and 3 below summarize the variables extracted to operationalize engagement, demographic characteristics and learning outcomes respectively. The systems from which the variables are extracted, as well as basic descriptive statistics, are also presented. Our initial unit of analysis was the aggregate of all interactions involving a specific student in a specific year, resulting in a dataset of 52,553 records.
Variable Number of attended career events Number of signed-up career events Proportion of career events signed-up to that were attended Number of logins Number of logins Number of logins Number of library's fines paid Number of logins Number of submitted evaluations Number of logins Number of papers' views Number of all interactions Number of committee interactions Number of enrolled programs Variable Gender Disability (type of disability) National identity Nationality
System Career Events System. Events are optional and cover a wide range of topics.
Career Events System Careers Events System Virtual Learning Environment (VLE) Inter Library Loans (Library ILL).
Online access to borrow books from other UK libraries.
Library. Online access to academic journals and eresources and manage library resource loans.
Library MACE (Module and Course Evaluation) system. An optional quality questionnaires for students.
MACE Exam's archival system Exam's archival system All systems (VLE, Library ILL, Library, MACE, Exam's archival system) Student's guild (buying tickets to guild's events, holding positions on volunteering project committees) Registration system 0 0 0 0 0 0 0 0 0 0 0 0 0 1.99 0.46 79.72 0.00 0.19 0.12 1.17 0.83 3.29 15.35 100.66 0.03 1.07 3.34 0.46 68.83 0.63 0.70 0.54 1.45 1.07 6.64 28.20 87.66 0.18 0.26 1,595 265 256 0
Here we present our two-step analysis. First, we present the significant pairwise relations found between outcome variables, and engagement or demographics variables. Second, we present a multivariate model to try and predict student success based on features found to be significantly correlated with outcome at our first step.
Since none of our outcome variables are normally distributed, we used Spearman's rank correlation test to find significant relationships between them and any numeric engagement variable. A quick exploration of the engagement variables shows that VLE logins, Past Exam views, Library logins, MACE submissions and event attendances follow a typical power law distribution as one might expect, where many students use each individual system sparingly and few students use each system often. When correlating outcome with categorical variables, we have used Mann-Whitney U test and Kruskal-Wallis H test. For space limitations, we only show significant relations with the normalized result outcome variables, we are omitting significant relations which were found to be very weak, in addition to some of the post-hoc results. For our regression model, we used the ‘logged’ version of some of the numeric variables in an attempt to make them more normally distributed, which while helpful has not fully solved the problem of non-normality. As there are some students who have not accessed some of the systems at all, we make the transformation x -> log(x+1) which maps 0 to 0 and prevents the problem of trying to use log(0). We have fitted a model to predict the weighted average results for a year, resulting (F(11, 44425) = 512.7, R2 = 0.1126, Adjusted R2 = 0.1124), p< 2.2e-16, Residual standard error=20.08. The parameter estimates and significances are detailed in Table 5 below. For the categorical variables in the table, the first factor that appears in the data is assumed to have a coefficient of 0 (e.g. “Female” has no effect on our model) and other factors for that category are assigned a coefficient of which the significance is then determined.
Our model appears to struggle from there being a low number of high scores in the dataset. Generally, we find that being male and older is likely to reduce your assessment results, as is living at home and being disabled. It also appears that being ‘more engaged’ is beneficial, except apparently logging onto the VLE too much could be a disadvantage for the overall result.
One of the major challenges of learning analytics in a BaM setting is the need to integrate analytics across different spaces and tools. In this study, we describe initial steps into exploring the relationship between learning outcome and engagement variables, where measures about engagement are integrated from students' interactions with a variety of systems and services, physical and digital. In addition, we have added demographic variables to be able to easily identify finer grained cohorts for further analysis. Following the collection and integration phase, we have shown here a regression model, predicting the aggregative score of all module grades at the end of the year. Our model shows the predictive values of demographics variables such as age, disability and being away from home, along with engagement variables, showing interactions with some of the university's systems and services, partially supporting existing evidence of the relation between engagement and outcome. Interestingly, most of the significant estimates were shown with systems which are not directly related to learning, but rather with a wider framework of interactions held between students and the university facilities, such as career events, committee activities and quality questionnaires. Moreover, interactions with the VLE, the digital system which coordinates most learning activities, were shown to be negatively correlated with module grades. Taking into consideration that the VLE, as well as library resources, are not used equally or standardly across all modules, this requires further investigation.
A problematic aspect for utilizing our findings is the observed range of residuals in the modelling exercise. For example, this could suggest that using a predictive modelling technique to determine failing students is unlikely to be effective, at least when using a student per year timescale and seeing how an individual student's performance changes over time. We could hopefully explain more variance by reducing this to a termly timescale. Module assessments are usually derived directly from different learning objectives and designs [13]. This variation could further cause a differentiation in the dependencies on different systems. The importance of module assessments in the total aggregative score also varies. Thus, a more predictive model could result from analyzing students enrolled to a specific module, course or programme. Age was shown to negatively affect the outcome. In addition to the reported relations above, when exploring secondary relations, among engagement and demographic variables, the age's negative relations with some engagement variables (such as all interactions with digital systems and career events attendance) suggests an explanation to this negative effect, and is subject to further analysis. In addition, some positive correlations among the engagement variables themselves (such as VLE logins, MACE, exams' archive and all digital interactions) supports the notion that "students who do stuff also do more stuff".1 Traditional approaches of the teaching-centered paradigm are usually measured by summative scores. Nonetheless, an educational institution's definition of learning outcomes, as well as the subjective expectation each student adopt, does not necessarily adhere to the taken summative measurements. Some students are after First Class Honors while others are aiming simply to complete the course or find a decent career. Therefore our goal should be to enable a flexible, multi-dimensional, possibly sometimes subjective framework for 'learning outcomes', and to seek to find 1
http://blogs.edweek.org/edweek/edtechresearcher/2014/03/big_data_mooc_research_breakthrough_learning_a ctivities_lead_to_achievement.html relations between various dimensions of engagement, demographics and various dimensions of learning outcome. For example, adding data from surveys (or other sources) could broaden our current limits of the data by complementing it with students' self-perceived interactions, data about their face-to-face interactions with each other or with their instructors, informal interactions (such as interacting over social media), their own perceptions about what is considered to be their 'learning outcomes' and more. In addition, some of our current variables are too coarse. For example, adding finer grained VLE activities, such as posting on a bulletin board and downloading material, separating data about career events' attendance by the event type and more, are crucial and can benefit our overall understanding about students' engagement.
[10] Harris, R., & Nikitenko, G. (2014). Comparing online with brick and mortar course learning outcomes An analysis of quantitative methods curriculum in public administration. Teaching Public Administration, 32(1), 95–107. http://doi.org/10.1177/0144739414523284 [11] Kuzilek, J., Hlosta, M., Herrmannova, D., Zdrahal, Z., & Wolff, A. (2015). OU analyse: Analysing at-risk students at The Open University. Learning Analytics Review, 1-16. [12] McConnell, D. (2000). Implementing computer supported cooperative learning.
In Millar, R. (2013). Improving science education: Why assessment matters. Valuing Assessment in Science Education: Pedagogy, Curriculum, Policy, 55– 68, Psychology Press. [13] Millar, R. (2013). Improving science education: Why assessment matters. Valuing Assessment in Science Education: Pedagogy, Curriculum, Policy, 55–68. [14] Moore, M. G. (1989). Editorial: Three types of interaction. The American
Journal of Distance Education, 3(2), 1–6. [15] Onyesolu, M. O., Nwasor, V. C., Ositanwosu, O. E., & Iwegbuna, O. N. (2013).
Pedagogy: Instructivism to Socio-Constructivism through Virtual Reality. International Journal of Advanced Computer Science and Applications, 4(9), 40–47. Retrieved from http://ijacsa.thesai.org/ [16] Picciano, A. G. (2002). Beyond student perceptions: Issues of interaction, presence, and performance in an online course. Journal of Asynchronous Learning Network, 6(1), 21–40. [17] Prince, M. (2004). Does Active Learning Work ? A Review of the Research.
Journal of Engineering Education, 93(July), 223–231.
http://doi.org/10.1002/j.2168-9830.2004.tb00809.x [18] Reyes, J. A. (2015). The skinny on big data in education: Learning analytics simplified. TechTrends, 59(2), 75–80. http://doi.org/10.1007/s11528-015-08421 [19] Shavelson, R. J., & Huang, L. (2003). Responding Responsibly. Change: The Magazine of Higher Learning, 35(1), 10–19.
http://doi.org/10.1080/00091380309604739 [20] Song, L., & McNary, S. W. (2011). Understanding students’ online interaction: Analysis of discussion board postings. Journal of Interactive Online Learning, 10(1), 1–14. [21] Wei, H., Peng, H., & Chou, C. (2015). Can more interactivity improve learning achievement in an online course? Effects of college students’ perception and actual use of a course-management system on their learning achievement. Computers & Education, 83, 10–21.
http://doi.org/10.1016/j.compedu.2014.12.013 [22] Zhu, E. (2006). Interaction and cognitive engagement: An analysis of four asynchronous online discussions. Instructional Science, 34(6), 451–480. http://doi.org/10.1007/s11251-006-0004-0 This research was supported by the Effective Learning Analytics Project at the University of Exeter. A project which aims to help students reach their full academic potential while studying at Exeter.