Empowering instructors through customizable collection and analyses of actionable information

Abstract

The use of analytics to support learning has been increasing over the last few years. However, there is still a significant disconnect between what algorithms and technology offer and what everyday instructors need to integrate actionable items from these tools into their learning environments. In this paper we present the evolution of the Student Relationship Engagement System, a platform to support instructors to select, collect, and analyze student data. The approach provides instructors the ultimate control over the decision process to deploy various actions. The approach has two objectives: to increase instructor data literacies and competencies, and to provide a low adoption barrier to promote a data-driven pedagogical improvement culture in educational institutions. The system is currently being used in 58 courses and 14 disciplines, and reaches over 20,000 students. • Information systems~Decision support systems • Humancentered computing~Visual analytics • Computing methodologies~Machine learning approaches • Applied computing~Education • Software and its engineering~Software creation and management Learning analytics adoption; scaling up; instructors; curriculum design and delivery; teaching approaches; machine learning.

1. INTRODUCTION

Since the early days of learning analytics (LA), the promise has been that the collection and analysis of large educational datasets could yield “actionable intelligence” [8, p41] to improve the overall student learning experience. At some of the institutions that have adopted LA, this intelligence typically takes the form of algorithms that predict student outcomes and aim to reduce attrition and failure rates [10; 16; 44; 53]. The higher education sector has been one of the first to explore the adoption of these techniques [ 22 ]. Despite these initiatives, recent reviews highlight the lack of widespread adoption of LA in the higher education sector [10; 44]. Various explanations have been suggested for this. At a high level, these include policy and ethical challenges [41; 54], institutional leaders’ misconceptions of LA [ 10 ], and the sector’s general culture of resistance to change [19; 40]. At an operational level, other authors have reported the inflexibility of vendor solutions, and difficulties in accessing data [ 38 ], as well as the accuracy of such data [ 6 ]. To add complexity to this situation, evidence is mounting that the onesize-fits-all approach, typical in LA, may be inadequate in explaining student outcomes [21; 34; 55] and addressing the needs of students in different disciplines [ 43 ].

Notwithstanding, there is increasing interest in the instructor-facing benefits of LA. These include detecting patterns and trends, using data to support decision making, testing assumptions, and understanding the effect of learning designs [ 25 ]. Tools that display and analyze student data can help instructors reflect on their designs and better understand the relationships between variables [15; 51]. Moreover, new tools are being developed that address a long-held appeal to connect LA with the learning sciences [18], by helping instructors understand how learner behaviors correspond with their pedagogical intent [ 11 ]. Recent results in the area of artificial intelligence in education suggest a shift in focus away from fully self-contained decision systems to a paradigm based on human intelligence amplification [ 5 ]. However, low data literacies and competencies pose a significant barrier to address this shift and achieve wider LA acceptance and adoption [6; 24].

Taken together, these suggest that greater impact of LA (e.g. insight into curricular design and delivery versus prediction of retention), may be catalyzed by addressing, and indeed leveraging, identified adoption barriers. In this paper, we take the position that, to be effective, LA must empower instructors with tangible solutions to address pressing needs [15; 37]. For some, this may mean addressing immediate retention issues [ 10 ], that is, “to satisfy a tangible, small-scale problem” [38, p236], while pushing instructors along the adoption pipeline [ 35 ] to more involved insights. This builds on findings from early adoption of computers in teaching, where “use of computers for one purpose may encourage enthusiasm for further computer use” [26, p7]. We present a case study of a bespoke web-based LA solution at the University of Sydney, outline its capabilities and impact, to date, and highlight the flow-on impacts for shifting teaching practices, curricular design and delivery, and growing a culture of LA use. We use Greller and Drachsler’s [ 24 ] generic LA design framework to situate our work in terms of stakeholders, objectives, data, instruments, and limitations.

2. OVERVIEW OF OUR APPROACH

We opted for a bottom-up approach where a basic but high-utility system was developed and improved collaboratively with instructors. From an early stage, this meant that our system addressed pressing objectives of key stakeholders [ 14 ]. Our design philosophy shared common themes with other LA developments, including usability, usefulness, data interoperability, real-time operation, flexibility, and generalizability [8; 15; 23]. However, in contrast to other approaches, our system’s growth was instructorcentered and ‘organic’, initially addressing a small-scale need (originally, tracking class attendance) and iteratively building features into the system (e.g. personalized interventions, data mining to uncover hidden relationships in course design) as instructors’ data literacies and competencies grew. A recent review of LA implementations at Australian institutions suggests that such early small-scale applications can have large impacts on capacity building [ 10 ].

2.1 Data collection

The importance of having the right data in the right place is a central issue for LA [ 28 ]. Most practical LA implementations involve collecting data into a central database available to the instrument [e.g. 3; 15; 38] or building analytics directly into the data source [e.g. 33].

Recognizing that both LMS and student information system (SIS) data have shortcomings [21; 31], and in keeping with our instructor-empowering philosophy, we opted for a hybrid approach where instructors could decide which data were most important for their contexts. For example, our discussions with instructors identified that class engagement and attendance data Figure 1. A smartphonewere important, in keeping with friendly in situ data evidence-based practice for recording and display student outcomes [42; 47]. interface. Unsurprisingly, interim grade and other performance data were also relevant [ 9 ]. Therefore, we started by developing a web-based, and smartphone-friendly, system that was easy and efficient to use and met these contextual needs (Figure 1). Since technology acceptance and adoption are closely linked with usefulness and usability [ 12 ], this was a first step in empowering instructors’ data usage. Due to technical limitations of our institution’s information technology infrastructure and capabilities, our system could not programmatically access LMS or SIS data. Other authors have solved this issue by capitulating to vendor-locked solutions, which offer a level of automatization but at the cost of flexibility, customizability, and possibly even scalability [ 38 ]. We addressed the issue by building in an additional facility to import any studentmatched data required through semi-automated data file uploads. This is a similar design philosophy to Graf et al. [ 23 ] in allowing free choice of data, and addresses realistic instructional situations where course-specific nuances can confound less flexible systems [ 38 ]. Serendipitously, this had the unintended advantage of forcing instructors to consider the data they were entering, in terms of its relevance to their context and pedagogical design. In fact, the criticality of these contextual factors is becoming much clearer [e.g. 15; 21], lending strong support to our approach. In terms of Greller and Drachsler’s [ 24 ] framework, our approach addressed the direct objectives of stakeholders in providing a stable, easy to use instrument that collected immediately relevant data.

2.2 Data extraction and affordances for action

Once the right data are in the right place, the typical progression in LA usually involves visualization via dashboards [ 45 ]. However, there is a danger that these visually appealing interfaces may distract users (such as instructors, students, and management) from a deeper understanding of the underlying data. Greller and Drachsler astutely describe that “enticing visualisations… [and] the simplicity and attractive display of data information may delude the data clients, e.g. teachers, away from the full pedagogic reality” [24, p52]. With this in mind, we decided to minimize visualizations and instead provide instructors with the ability to run large-scale customized queries on their students’ data. This meant that instructors of even very large courses could select, collect, and extract the data they wanted, and also run basic analyses that are of interest to their contexts [ 23 ]. Importantly, we aimed to avoid algorithmic black boxes [ 35 ], which are present in other solutions [e.g. 2], instead giving instructors full control of the process. This level of functionality was built to respond to pressing institutional needs to address issues of student engagement, taking advantage of the data that were already being collected. Using the customizable analysis engine, instructors could specify conditions and efficiently identify particular groups of students (Figure 2). Once identified, instructors could then deliver personalized feedback to students via email or the cellular network. We observed that instructors “relied on their intuition and hunches to know when students are struggling, or to know when to suggest relevant learning resources” [13, p20].

In addition to this approach to extracting information at scale, we also focused on a seldom-raised concern, namely “the focus of LA appears fixed to an institutional scale rather than a human scale” [31, p4]. We therefore wished to promote the power of LA in augmenting human interaction. To this end, our system design allowed instructors to customize the information that could be immediately extracted and displayed to other staff (such as tutors and support staff) as they worked directly with students in face-toface contexts (e.g. Figure 1). In a similar application, Lonn et al. [ 37 ] empowered academic advisors with pertinent student data. While use of our system in this way has been predominantly operational (e.g. redirecting students in class if they have not completed assigned pre-work), we envisage that, as more relevant data are available, this ‘mini human dashboard’ approach will spark deep human conversations supported by the relevant data. In terms of Greller and Drachsler’s [ 24 ] framework, our approach allowed both staff (faculty as well as student support staff) and student stakeholders to take advantage of data through the instrument. In this process, information was prepared and presented to stakeholders, and the transparent analysis engine also forced instructors to develop data interpretation and decision-making competencies [ 24 ]. Moreover, we saw our approach as reflecting the human judgment and instructor empowerment roots of LA [ 52 ].

2.3 Guided semi-automated discovery

The closely related field of educational data mining has a greater focus on automated methods of discovering meaning in educational data than LA [ 4 ], which address one of the key opportunities for LA, namely “to unveil and contextualize so far hidden information out of the educational data” [24, p47]. Data mining techniques in LA [ 4 ] have primarily focused on outcome prediction through regression and classification [e.g. 21], semantic analyses [ 29 ], and social network analysis [e.g. 36]. However, data mining techniques typically require substantial technical understanding and are beyond the capabilities of most instructors [ 56 ]. Additionally, input variables are differentially predictive for each instructional context [ 21 ], necessitating a more nuanced and contextualized approach to information discovery.

To this end, we are in the initial stages of testing an approach that helps instructors uncover hidden relationships in data about their students. We are combining the data they have already collected in our system with the machine learning application programming interfaces (APIs) provided by BigML (https://bigml.com). Our approach involves instructors selecting data to analyze, based on their pedagogical context and intent, using a drag-and-drop graphical user interface where they can also transform and/or combine data (Figure 3) and select a target (dependent) variable (e.g. an interim grade). The system then runs a series of machine learning algorithms (see section 3.2) against these data and returns analysis results for instructors to interpret in their context. This approach is more user-friendly than a similar system designed by Pedraza-Perez et al. [ 46 ], and can also include data beyond the LMS. This process may provide novel insights into curriculum design and delivery, such as visual and statistical identification of factors that impact student outcomes, and identifying patterns in performance across multiple courses with different course designs. Other possible insights are outlined in section 3.2. In terms of Greller and Drachsler’s [ 24 ] framework, this nascent approach adds algorithmic capability to the instrument to provide certain stakeholders with possibly hidden information, beyond that of prediction. However, it requires higher data literacies and competencies, such as critical evaluation skills (internal limitations [ 24 ]). By working through the other steps of the process already outlined (namely data selection, collection, extraction, and basic analyses), our presumption is that instructors will have gained some of these competencies. Together, we see this as a combination of LA and educational data mining, where instructor judgment is empowered through leveraging machine learning [ 52 ].

2.4 Preliminary outcomes

The first version of our system was trialed with four courses in 2012. Since then, it has been adopted in 14 disciplines and 58 courses, covering over 20,000 students. This approach has allowed our system to evolve functionality through collaboration with the instructors who are using it. Although lacking empirical data, anecdotal feedback indicates that uptake is, in part, due to the customizability and afforded actions (i.e. usefulness [ 12 ]) and easeof-use of the system. This contrasts with the issues highlighted by Lonn et al. [ 38 ] around their scaled-up LA system with a vendorlocked approach not being “nimble enough to be responsive to idiosyncratic cases” [38, p238]. The interventions for students, using our system, have contributed to sustained improvements in retention as well as overall performance (Figure 4). Now that instructors have more experience working with their data, we are collaborating with them to expand opportunities afforded by our system to further understand, optimize, and transform their teaching.

HD DI CR PS FA 100% 80% 60% 40% 20% 0% 2011 2012 2013 2014

Attrited

3. UNDERSTANDING, OPTIMIZING, AND

TRANSFORMING TEACHING

3.1 Teaching practices

Too often the student experience at university is one of isolation from instructors, which is especially poignant for students transitioning to higher education where instructors can appear disconnected [ 30 ]. While LA may exacerbate this situation by defocusing the human aspects of learning [31], our approach encourages instructors to break this pattern: hence the name of our system, the Student Relationship Engagement System (SRES). The strength of the SRES lies in the ability for instructors to customize analyses to the needs of their course and students. One of the primary goals of the SRES is to personalize communication with students and engage them in conversations about their learning. This is particularly important when operating at scale with large cohorts, as data-driven personalizations are a key factor in promoting student engagement [ 7 ]. We see this as a blending of Greller and Drachsler’s [ 24 ] objectives of reflection and prediction, where timely data are extracted to aid co-reflection by instructors and students. We find that this approach can also encourage more meaningful student-faculty contact, thus addressing a constant warning in the field that students’ internal conditions must be taken into account [20].

3.2 Instructional and curricular design and delivery

Currently, we are trialing several newer developments in the SRES in our own courses to explore further ways to support decision making [ 24 ] about instructional and curricular design and delivery. Here, we present three proof-of-concept examples that attempt to derive meaning in our contexts by analyzing real course data (Table 1) using machine learning tools. Instructors can select (Figure 3) data that are most relevant in their contexts (for example, mid-term test grade, session length in the LMS, attendance count early in the semester, average grade of online quizzes early in the semester, activity in online forums, etc), and apply these tools to uncover hidden patterns. For example, what relationship is there between class attendance, different aspects of online engagement, and test grades?

3.2.1 Decision trees

Decision tree algorithms generate hierarchical conditions-based predictive models that attempt to explain conditions or patterns in data that lead to a particular outcome [ 49 ]. In our context, the decision tree discovered through machine learning suggested that early quiz performance (which was only worth a low proportion of the final grade) was an important factor in student success (Figure 5). While instructor intuition about their students may predict this, there is value in having data demonstrating various ‘paths to success’. Additionally, when one considers that each of these quizzes are worth only 0.65% of a students’ final grade (again emphasizing the importance of context and design), this dataenabled discovery becomes the grounds for supporting the evidence-based practices of emphasizing time on task and continuous assessment. These analyses are now driving pedagogical changes (e.g. decisions on provision of feedback in these quizzes versus no feedback) to improve student performance. For instructors, this approach not only helps identify struggling students, but also supports decisions about learning activities and assessing course effectiveness [50; 51].

In many cases in LA and educational data mining, decision tree algorithms are used purely as opaque models for prediction of student outcomes [e.g. 27; 32]. However, this does not take full advantage of the fact that decision trees are one of the few machine learning algorithms that can produce easily human-interpretable and -understandable predictive models, in the form of choices and rules [ 49 ]. As in our example, analysis of LMS interaction and completion data with decision trees can reveal behavioral and earlyperformance characteristics of high- and low-performing students, and allows instructors to adapt their courses and interventions based on this information [17; 50].

3.2.2 Association rule mining

Association rule mining reveals typically hidden patterns in data that commonly occur together [4; 51]. These patterns are expressed as rules or relationships of varying strength from antecedent to consequent conditions. Our application leverages a BigML visualization to graphically represent these rules. In our context, association rule mining provided evidence that lower in-class attendance was associated with lower online activity, and that lower online activity was a central node between other disengagement measures (Figure 6, main network). On the other hand, common relationships were also found between strong mid-term test marks, high online quiz marks, and strong pre-class quiz performance (Figure 6, bottom-left network), although interestingly high online activity was not included. While again this might seem obvious, this data-driven finding could trigger curriculum or instructional design changes to better engage students [ 48 ].

The associations discovered could also inform intervention strategies by identifying linked problem areas [ 50 ]. common understanding that higher discussion forum engagement is correlated with higher performance [e.g. 39], and again reemphasizes the importance of considering contextual and pedagogical factors [ 21 ]. In our context, the online forum functioned in a question and answer format, which may help to explain why a cluster of poorer-performing students had higher engagement, i.e. posting of questions. Together, these analyses and their data-driven findings can be powerful for instructors because they help to support or refute a priori assumptions about their students, pedagogical strategies, and curricular design. Clustering may also provide insight into behaviors common to groups of differentially-performing students [ 1 ]. Some have even suggested that clustering students based on observed behaviors may assist formation of congruous student groups [ 50 ].

3.3 Cultural shifts

Our approach leveraged existing instructor needs to introduce them to a data-driven LA system, the SRES. A consequence of doing so has been to force them to think about their contexts and the relevant data. We are currently analyzing these instructor capability outcomes, as others have suggested that “implementing early and to small scale, even if inadequately, will build capacity” [10, p38]. Our approach certainly started small-scale, and was perhaps somewhat inadequate in not providing automatic access to the plethora of data available in LMS logs and the SIS. However, our hope is that by starting small and introducing instructors to datadriven ways of operating, we can introduce them to deeper LA ‘by stealth’ and gradually expand their capabilities, in parallel with expansion of the system’s capabilities.

4. CONCLUSION

The field of learning analytics is under unprecedented pressure to effectively bridge the gap between technological capacity and tangible improvements of the student experience. The shift towards tools that enhance current instructional practice is occurring. In this paper we have presented the evolution of the Student Relationship Engagement System following an organic and instructor-centric approach. The platform provides a high level of control over data collection and processing as well as direct control over the actions derived from the analysis. The current uptake of the tool across disciplines suggests its suitability to promote data literacy skills and a culture of data-supported innovation. As further avenues to explore, we have identified the need to increase the understanding of how instructors are empowered through data-driven analysis of learning designs and delivery.

5. ACKNOWLEDGMENTS

We thank many instructors and student support staff for their input into the process, countless students for their enthusiasm, and the Australasian Society for Computers in Learning in Tertiary Education (ascilite) Learning Analytics Special Interest Group for their support. [16] ECAR-ANALYTICS Working Group (2015) The Predictive Learning Analytics Revolution: Leveraging Learning Data for Student Success. EDUCAUSE Center for Analysis and Research, Louisville. [17] Falakmasir, M. H. and Habibi, J. (2010) Using educational data mining methods to study the impact of virtual classroom in e-learning. In Proceedings of the 3rd International

Conference on Educational Data Mining, Pittsburgh. [18] Ferguson, R. (2012) Learning analytics: drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5-6), 304-317. [19] Ferguson, R., Clow, D., Macfadyen, L., Essa, A., Dawson, S. and Alexander, S. (2014) Setting learning analytics in context: overcoming the barriers to large-scale adoption. In Proceedings of the Fourth International Conference on Learning Analytics And Knowledge. [20] Gašević, D., Dawson, S. and Siemens, G. (2015) Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64-71. http://dx.doi.org/10.1007/s11528-014-0822-x Proceedings of the 6th International Conference on Learning Analytics and Knowledge, Edinburgh.

References

[1] Amershi , S. and Conati , C. ( 2009 ) Combining Unsupervised and Supervised Classification to Build User Models for Exploratory . Journal of Educational Data Mining , 1 ( 1 ), 18 - 71 .

[2] Arnold , K. E. ( 2010 ) Signals: Applying Academic Analytics . Educause Quarterly , 33 ( 1 ), n1 .

[3] Arnold , K. E. and Pistilli , M. D. ( 2012 ) Course signals at Purdue: using learning analytics to increase student success . In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge , Vancouver.

[4] Baker , R. S. and Inventado , P. S. ( 2014 ). Educational data mining and learning analytics Learning Analytics (pp. 61 - 75 ): Springer.

[5] Baker , R. S. ( 2016 ) Stupid Tutoring Systems , Intelligent Humans. International Journal of Artificial Intelligence in Education , 1 - 15 .

[6] Bichsel , J. ( 2012 ) Analytics in higher education: Benefits, barriers, progress, and recommendations . Available: https://net.educause.edu/ir/library/pdf/ERS1207/ers1207.pdf

[7] Bridgeman , A. and Rutledge , P. ( 2010 ) Getting personal - feedback for the masses . Synergy , 30 , 61 - 68 .

[8] Campbell , J. P. , DeBlois, P. B. and Oblinger , D. G. ( 2007 ) Academic analytics: A new tool for a new era . EDUCAUSE review , 42 ( 4 ), 40 - 57 .

[9] Clow , D. ( 2012 ) The learning analytics cycle: closing the loop effectively . In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge , Vancouver.

[10] Colvin , C. , Rogers , T. , Wade , A. , Dawson , S. , Gašević , D. , Buckingham Shum , S. , Nelson , K. J. , Alexander , S. , Lockyer , L. , Kennedy , G. , Corrin , L. and Fisher , J. ( 2015 ) Student retention and learning analytics: a snapshot of Australian practices and a framework for advancement (draft report). Australian Office for Learning and Teaching , Sydney, Australia.

[11] Corrin , L. , Kennedy , G. , Barba , P. D. , Bakharia , A. , Lockyer , L. , Gasevic , D. , Williams , D. , Dawson , S. and Copeland , S. ( 2015 ) Loop: A learning analytics tool to provide teachers with useful data visualisations . In Proceedings of the 32nd Conference of the Australasian Society for Computers in Learning in Tertiary Education , Perth.

[12] Davis , F. D. ( 1993 ) User acceptance of information technology: system characteristics, user perceptions and behavioral impacts . International Journal of Man-Machine Studies , 38 ( 3 ), 475 - 487 .

[13] Dietz-Uhler , B. and Hurn , J. E. ( 2013 ) Using learning analytics to predict (and improve) student success: A faculty perspective . Journal of Interactive Online Learning , 12 ( 1 ), 17 - 26 .

[14] Drachsler , H. and Greller , W. ( 2012 ) Confidence in learning analytics . In Proceedings of the 2nd International Conference on Learning Analytics & Knowledge , Vancouver.

[15] Dyckhoff , A. L. , Zielke , D. , Bültmann , M. , Chatti , M. A. and Schroeder , U. ( 2012 ) Design and implementation of a learning analytics toolkit for teachers . Journal of Educational Technology & Society , 15 ( 3 ), 58 - 76 .

[21] Gašević , D. , Dawson , S. , Rogers , T. and Gasevic , D. ( 2016 ) Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success . The Internet and Higher Education , 28 , 68 - 84 . http://dx.doi.org/10.1016/j.iheduc. 2015 . 10 .002

[22] Goldstein , P. J. and Katz , R. N. ( 2005 ) Academic analytics: The uses of management information and technology in higher education . Available: http://www.educause.edu/library/resources/academicanalytics -uses-management-information-and-technologyhigher-education

[23] Graf , S. , Ives , C. , Rahman , N. and Ferri , A. ( 2011 ) AAT: a tool for accessing and analysing students' behaviour data in learning systems . In Proceedings of the First International Conference on Learning Analytics and Knowledge , Banff.

[24] Greller , W. and Drachsler , H. ( 2012 ) Translating learning into numbers: A generic framework for learning analytics . Journal of Educational Technology & Society , 15 ( 3 ), 42 - 57 .

[25] Gunn , C. , McDonald , J. , Donald , C. , Milne , J. , Nichols , M. and Heinrich , E. ( 2015 ) A practitioner's guide to learning analytics . In Proceedings of the 32nd Conference of the Australasian Society for Computers in Learning in Tertiary Education , Perth.

[26] Jacobsen , D. M. ( 1998 ) Adoption Patterns of Faculty Who Integrate Computer Technology for Teaching and Learning in Higher Education . In Proceedings of the 98th World Conference on Educational Multimedia and Hypermedia , Freiburg.

[27] Jayaprakash , S. M. , Moody, E. W., Lauría , E. J. , Regan , J. R. and Baron , J. D. ( 2014 ) Early alert of academically at-risk students: An open source analytics initiative . Journal of Learning Analytics , 1 ( 1 ), 6 - 47 .

[28] Jones , D. , Beer , C. and Clark , D. ( 2013 ) The IRAC framework: Locating the performance zone for learning analytics . In Proceedings of the 30th Conference of the Australasian Society for Computers in Learning in Tertiary Education , Sydney.

[29] Knight , S. and Littleton , K. ( 2015 ) Discourse-centric learning analytics: Mapping the terrain . Journal of Learning Analytics , 2 ( 1 ), 185 - 209 .

[30] Krause , K. ( 2005 ) Understanding and promoting student engagement in university learning communities. Paper presented as keynote address: Engaged, Inert or Otherwise Occupied , 21 - 22 .

[32] Lauría , E. J. , Moody, E. W., Jayaprakash , S. M. , Jonnalagadda , N. and Baron , J. D. ( 2013 ) Open academic analytics initiative: initial research findings . In Proceedings of the Third International Conference on Learning Analytics and Knowledge , Leuven.

[33] Liu , D. Y. T. , Froissard , J.-C. , Richards , D. and Atif , A. ( 2015 ) An enhanced learning analytics plugin for Moodle: student engagement and personalised intervention . In Proceedings of the 32nd Conference of the Australasian Society for Computers in Learning in Tertiary Education , Perth.

[34] Liu , D. Y. T. , Froissard , J.-C. , Richards , D. and Atif , A. ( 2015 ) Validating the Effectiveness of the Moodle Engagement Analytics Plugin to Predict Student Academic Performance . In Proceedings of the 2015 Americas Conference on Information Systems , Puerto Rico.

[35] Liu , D. Y. T. , Rogers , T. and Pardo , A. ( 2015 ) Learning analytics - are we at risk of missing the point ? In Proceedings of the 32nd Conference of the Australasian Society for Computers in Learning in Tertiary Education , Perth.

[36] Lockyer , L. , Heathcote , E. and Dawson , S. ( 2013 ) Informing pedagogical action: Aligning learning analytics with learning design . American Behavioral Scientist, 57 ( 10 ), 1439 - 1459 .

[37] Lonn , S. , Krumm , A. E. , Waddington , R. J. and Teasley , S. D. ( 2012 ) Bridging the gap from knowledge to action: Putting analytics in the hands of academic advisors . In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge , Vancouver.

[38] Lonn , S. , Aguilar , S. and Teasley , S. D. ( 2013 ) Issues, challenges, and lessons learned when scaling up a learning analytics intervention . In Proceedings of the Third international conference on learning analytics and knowledge , Leuven.

[39] Macfadyen , L. P. and Dawson , S. ( 2010 ) Mining LMS data to develop an “early warning system” for educators: A proof of concept . Computers & Education , 54 ( 2 ), 588 - 599 .

[40] Macfadyen , L. P. and Dawson , S. ( 2012 ) Numbers are not enough. Why e-learning analytics failed to inform an institutional strategic plan . Journal of Educational Technology & Society , 15 ( 3 ), 149 - 163 .

[41] Macfadyen , L. P. , Dawson , S. , Pardo , A. and Gašević , D. ( 2014 ) Embracing big data in complex educational systems: The learning analytics imperative and the policy challenge . Research & Practice in Assessment , 9 ( 2 ), 17 - 28 .

[42] Massingham , P. and Herrington , T. ( 2006 ) Does attendance matter? An examination of student attitudes, participation, performance and attendance . Journal of university teaching & learning practice , 3 ( 2 ), 3 .

[43] McPherson , J. , Tong , H. L. , Fatt , S. J. and Liu , D. Y. T. ( 2016 ) Learning analytics and disciplinarity: Building a typology of disciplinary differences from student voices . In

[44] Newland , B. , Martin , L. and Ringan , N. ( 2015 ) Learning analytics in UK HE 2015: A HeLF survey report: HeLF Heads of e-Learning Forum . Available: http://www.helf.ac.uk/

[45] Pardo , A. ( 2014 ). Designing Learning Analytics Experiences . In J. A. Larusson and B. White (Eds.), Learning Analytics (pp. 15 - 38 ): Springer.

[46] Pedraza-Perez , R. , Romero , C. and Ventura , S. ( 2010 ) A Java desktop tool for mining Moodle data . In Proceedings of the 4th International Conference on Educational Data Mining , Eindhoven.

[47] Rodgers , J. R. ( 2001 ) A panel-data study of the effect of student attendance on university performance . Australian Journal of Education , 45 ( 3 ), 284 - 295 .

[48] Romero , C. , Ventura , S. and De Bra , P. ( 2004 ) Knowledge discovery with genetic programming for providing feedback to courseware authors. User Modeling and User-Adapted Interaction , 14 ( 5 ), 425 - 464 .

[49] Romero , C. , Ventura , S. , Espejo , P. G. and Hervás , C. ( 2008 ) Data Mining Algorithms to Classify Students . In Proceedings of the 1st International Conference on Educational Data Mining , Montreal.

[50] Romero , C. , Ventura , S. and García , E. ( 2008 ) Data mining in course management systems: Moodle case study and tutorial . Computers & Education , 51 ( 1 ), 368 - 384 .

[51] Romero , C. and Ventura , S. ( 2010 ) Educational data mining: a review of the state of the art . Systems, Man, and Cybernetics , Part C: Applications and Reviews, IEEE Transactions on, 40 ( 6 ), 601 - 618 .

[52] Siemens , G. and Baker , R. S. ( 2012 ) Learning analytics and educational data mining: towards communication and collaboration . In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge , Vancouver.

[53] Siemens , G. , Dawson , S. and Lynch , G. ( 2013 ) Improving the quality and productivity of the higher education sector . Society for Learning Analytics Research.

[54] Slade , S. and Prinsloo , P. ( 2013 ) Learning analytics ethical issues and dilemmas . American Behavioral Scientist , 57 ( 10 ), 1510 - 1529 .

[55] Wolff , A. , Zdrahal , Z. , Nikolov , A. and Pantucek , M. ( 2013 ) Improving retention: predicting at-risk students by analysing clicking behaviour in a virtual learning environment . In Proceedings of the 3rd international conference on learning analytics and knowledge , Leuven. http://dx.doi.org/10.1145/2460296.2460324

[56] Zorilla , M. E. , García-Saiz , D. and Balcázar , J. L. ( 2010 ) Towards parameter-free data mining: Mining educational data with yacaree . In Proceedings of the 4th International Conference on Educational Data Mining , Eindhoven.