Instructional perspective using Learning Analytics in Computer Science education

Abstract

Learning Analytics is a complex phenomenon that has to consider the collection and analysis of information about learners together with the need to allow educators to manage and process it. The current work presents an instructional perspective to deal with such analytical complexity in a computer education context enabling a simple and versatile processing of different learning data sources. Tested courses reveal the potential of this perspective using tools to diagnose and visualize different learning analysis scenarios.

Keywords

• Instructional approach
• Learning Analytics
• Computing education

1 Introduction

Learning Analytics (LA) is a complex phenomenon that deals with “the measurement, collection, analysis and reporting of data about learners and their contexts” [ 1 ]. LA cannot be “blind” in the sense it neglects the educational context in which measurement or analysis are developed. In the current work, Computer Science education is addressed as a discipline that has traditionally been object of this kind of analytics processes. For example, student logging and behavior have been analyzed in introductory programming courses [ 2 ], predictions of students’ performance have been based on data collected in their computing courses that can be used by educators [ 3 ], metrics have been proposed to quantify the rate of student errors and detect if he or she is struggling with important programming concepts [ 4 ], control-flow mechanisms have been set up for analysing students’ progress [ 5 ] and analysing the process data of students have provided educators with insights about students' patterns of programming [ 6 ].

Gašević et al [ 7 ] reminded us about the LA focus on learning and how “computational aspects of learning analytics must be well integrated within the existing educational research”. In a similar line, authors in [ 8 ] advocate for taking into account instructional conditions when applying learning analytics. The book “Developing Effective Educational Experiences through Learning Analytics” [ 9 ] describes a practical view about the adoption of data mining and analysis techniques in academic institutions to improve the outcome of student learning. Moreover, the role of learning analytics in future education models [ 10 ] demands that the applied use of student learning data in this context can further assist teachers and help improve practice. Accordingly, instructional methods concerning Computer Science Copyright © 2017 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. education have to be considered in order to configure the specific learning scenarios where LA techniques and tools can be deployed. The current work introduces an instructional perspective that intends to match those educational issues present in different Computing teaching settings with the collected data and analysis processes that can be performed with them.

The presented perspective concerns the multiple learning scenarios and instructional methods that are present in Computer Science education. For example, a strong practice lab component or problem-based approaches are usually addressed in computing curricula [ 11 ]. Moreover, such perspective has to be close to students and lecturers who are the main actors in these educational settings. Therefore, a practical and simple LA approach has to be provided that allow Computing lectures to easily formulate learning analytical questions about the academic outcomes familiar to them and get an understandable answer back. That means a selection of LA tools with no programming requirements, and supporting an intuitive deployment and delivering visual graphical reports. After obtaining a broad view of the stated problem and its solution, a more detailed analysis could be performed.

The remainder of the work is structured as follows. Section 2 describes some related works to the application of practical learning analytics is a Computing education context. Section 3 introduces an instructional perspective that intends to link traditional methods used when teaching computing issues with those activities and assessment mechanisms which enable an LA treatment. A case study dealing with three computing courses is presented in section 4 together with their Results in the section 5. Finally, some Conclusions and further works are drawn.

2 Related works

As mentioned in [ 12 ], the overall LA process is an iterative cycle structured in three major steps: i) data collection and pre-processing, ii) analytics and action, and iii) post-processing. The current work focuses on the first step addressed to gather information that can be relevant for specific instructional methods in a Computing educational context. Baker & Siemens [ 13 ] remarked the importance of using more simple and intuitive tools to make LA accessible to a wide range of educators. They commented that analytical tools in the early 2000s were technically complex but recent versions of programs such as RapidMiner1, SPSS2 or SAP3 are easy to use by individuals with low-level technical knowledge. Even, popular tools like Microsoft Excel or the more recent Tableau Software4 have incorporated visualization and analytical features very powerful but simple to use. There are learning analytics tools addressed to Engineering Education [ 14 ] or specialized to visualize Computer Science Teamwork [ 15 ] but instead, this work is oriented towards exploring the use of generic tools such as Excel and Tableau to take advantage of their functionalities in order to get a fast processing view of spreadsheet datasets. 1 https://rapidminer.com/ 2 https://www.ibm.com/analytics/us/en/technology/spss/ 3 https://www.sap.com/product/analytics 4 https://www.tableau.com

This simplicity is the main advantage of this kind of analytical tools since they allow educators to manage their data in a quite easy way and obtain a first overall view that can be further refined. There are some proposals about the use of log data downloaded in Excel for further manipulation through various formulae and pivot tables [ 16 ] and the deployment of Excel Pivot Tables [ 17 ]. Similarly, Tableau has been used to get a visual perspective of academic analytics at the University of Phoenix [ 18 ] and Friesen [ 19 ] also remarked its use combined with SAP as analytic tool. At the end, the final purpose can be to provide lecturers with tools that allow them to get rapid answers from questions about the learning performance in their specific educational scenarios. Several works have been addressed to build “dashboards” to support such response process enabling the collection of data and presenting it to instructors and students with the aim to “positively influence learning outcomes and retention”. For instance, VizDeck [ 20 ] a sample of web-based tool for exploratory visual analytics is presented. Olivares [ 21 ] also proposed a user-centered, learning dashboard tailored for computing courses extended with the OSBLE+ tool. However, this kind of dashboards is sometimes rather inflexible and it is necessary to trade-off between analytics power and the versatility required to analyze the impact of different instructional methods in specific learning settings.

3 Instructional perspective

A wide range of instructional methods is quite common in the Computer Science educational context. Caspersen & Bennedsen [ 22 ] introduced a learning theoretic approach for instructional design of a programming course. The proposed Guide to Teaching Computer Science [ 11 ] remarks the importance of an activity-based approach in such discipline. This guide describes strategies for promoting problemsolving skills, assessing learning processes or dealing with pupils’ misunderstandings. Such teaching strategies can be used as key issues to explore different ways in which learning analytics could address them. For example, by providing information about lab-based tasks for solving problems, project work assignments, assessment activities or questions asked by students. More recently, Zendler and Klaudt [ 23 ] also reviewed some of the most appropriate methods for computer science teaching. Lab centered instruction [ 24 ], or project works [ 25 ] are some of these instructional methods examples. In all these methods, there is a wide range of learning activities which can be tracked and analyzed. Such activities are complemented with assessment strategies either formative or summative designed to evaluate them. Table 1 shows a list of instructional methods, which display possible activities that can be proposed by lecturers (and potentially done by students) together with assessment mechanisms and learning outcomes that could be tracked. For example, in a Direct Instruction method, lecturers usually impart knowledge or demonstrate a skill by transmitting some ideas or concepts. In this instructional context, learning activities consist of lectures or demonstrations which can be assessed in a formative way (e.g. using multiple-choice questions) or through a final examination. Besides these assessment results, other outcomes can be tracked such as the accesses to didactic resources (e.g. a video recording or a file document) or the classroom attendance. Once teaching actions, assessment results and other potential outcomes have been identified, it should be easy for the lecturer to ask herself or himself what are the learning items to be analyzed. However, this a cumbersome process and, many times, bounded by the available information [ 26 ]. In the current case, LA sources are mainly based on collecting data from e-learning platforms (e.g. a Learning Management System) but also institutional tools that register exam grades or student attendance lists can be used. The key is to have information about learning activities that can be easily managed and processed using simple tools. Spreadsheets either in text or excel formats are able to meet these requirements since they are mostly used to export data from e-learning platforms and visualize these data items. Once detected these data sources, the next step would consist of trying to find potential connections among them and perform an initial analysis using visual LA tools. 3

Case study

The proposed instructional perspective has been tested in a Computing Bachelor degree within three courses dealing with different disciplines and instructional methods. A first-year course called Computer Technology (CT) is addressed as a core subject taught during the spring term to more than 500 enrolled students. During the second year an Operating System (OS) course is taught to about 400 students also as core subject, and finally, a Project Management (PM) course is taught in the third year. These three courses are all compulsory but they address very different topics using several instructional methods and teaching strategies. Lectures are a common instrument for transmitting knowledge (direct instruction) but according with the topic complexity they are complemented with specific mechanisms to tackle such complexity. For example, the use of a mobile app to solve quizzes and to get a quick view about the student level of knowledge in Computer Technology as it is show in Fig1. The OS course deals with programming skills and therefore, hands-on lab tasks are a crucial resource in this context. In the case of the PM course, learning activities are focused on collaborative tasks oriented towards the documentation of the different steps of a software project development.

4 Results

Results from the proposed case study are described in this section that displays some charts obtained using the Tableau software. These charts stem from data collected in the case courses and they show the potential of such tool to analyse them according to specific instructional issues. First, the CT course is addressed to compare learning outcomes in an interactive instruction scenario. Fig. 2 shows a chart displaying the average examination grades for a set of course groups. The red square in the chart reports about grades in two groups that worked with the Socrative mobile app to enhance the classroom interaction.

Meanwhile, Fig. 3 shows a scatter diagram that represents the relationship between average scores in the quizzes carried in the classroom, concerning units 1 and 2 (ClassQuizzes U1U2), and the grades obtained in the official examination for the same units (Exam1P17), which is common for all the groups. These figures apply only to the 47 students belonging to those groups regularly using Socrative (1G1 and 1G2 represented in Fig. 2). In this case, a positive correlation is observed between both variables, although is not very high (the Pearson correlation coefficient r is 0,53 with a p-value = 0,00024). Fig. 4 shows a zoomed image of the red square displayed on the scatter diagram of Fig. 3.

Next, a project-based instructional scenario has been checked in the PM course. Fig. 5 shows a screenshot of the Tableau tool that displays the connection among different data sources such as the accesses to project resources, students’ scores or project group information. Fig. 6 shows part of bar chart that represents the number of accesses to project resources by group using the Tableau functionality of “calculated fields”.

5 Conclusions

The current work has presented an instructional perspective about the use of learning analytics tools that intends to combine the focus on several teaching actions and student outcomes together with the simplicity to process them by using popular tools such as Microsoft Excel or Tableau. The purpose is far from obtaining a fancy or detailed view about collected learning data but rather to get a first visual impression over them. In a future many instructors and educators maybe could become “data scientists” but at the present time, they mostly lack the technical knowledge and expertise to carry out a complex learning analytics process, even in a Computer Science education context like the one addressed in the current paper. The computing courses tested in this work show the potential of this kind of analytical tools to deal with multiple types of instructional methods and settings allowing users to collect and display learning information in an easy and simple way. Further works plan to offer a systematic and rigorous guide to computing educators who need a fast and understandable overview of their learning scenarios and those data sources that feature them.

Acknowledgments Thanks to the support of the Computer Engineering department (DISCA) and the ETSINF (Escola Tècnica Superior d’Enginyeria Informàtica) at the Universitat Politècnica de València.

References

1. Long , P. , Siemens , G. Penetrating the Fog: Analytics in Learning and Education . EDUCAUSE Review , 46 ( 5 ), 2011

2. Watson , C. , Li , F.W.B. and Godwin , J.L. Predicting Performance in an Introductory Programming Course by Logging and Analyzing Student Programming Behavior . Proceedings of the IEEE 13th International Conference on Advanced Learning Technologies ( 2013 ), 319 - 323

3. Carter , A.S , Hundhausen , C.D. , Adesope , O. The Normalized Programming State Model: Predicting Student Performance in Computing Courses Based on Programming Behavior . Proceedings of the Eleventh Annual International Conference on International Computing Education Research . ACM. ( 2015 ) 141 - 150

4. Becker , B. A. A new metric to quantify repeated compiler errors for novice programmers . Proceedings of the 21st Annual Conference on Innovation and Technology in Computer Science Education ACM ( 2016 ) 296 - 301

5. Hovemeyer , D. Hellas , A. , Petersen , A. , Spacco , J. Control-Flow-Only Abstract Syntax Trees for Analyzing Students' Programming Progress . In Proceedings of the 2016 ACM Conference on International Computing Education Research (ICER '16) . ACM, New York, NY, USA ( 2016 ) 63 - 72

6. Carter , A.S , Hundhausen , C.D. Using Programming Process Data to Detect Differences in Students' Patterns of Programming . In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE '17) . ACM, New York, NY, USA ( 2017 ) 105 - 110

7. Gašević , D. , Dawson , S. , Siemens , G. Let's not forget: Learning analytics are about learning TechTrends , 59 ( 1 ) ( 2015 ) 64 - 71

8. Gašević , D. , Dawson , S. , Rogers , T. , Gašević , D.. Learning analytics should not promote one size ts all:The effects of instructional conditions in predicating academic success . The Internet and Higher Education ( 2016 )

9. Anderson , M. and Gavan , C. , (eds.). Developing Effective Educational Experiences through Learning Analytics. Advances in Educational Marketing, Administration and Leadership. IGI Global ( 2016 )

10. Steigler-S. , Pardo , A. , Dawson , S. The role of Learning Analytics in future education models . https://www.telstra.com.au/content/dam/tcom/business-enterprise/industries/pdf/ tele0126_ whitepaper_5_spreads_lr_notrims .pdf

11. Hazzan , O. , T. Lapidot and N. Ragonis ,. Guide to Teaching Computer Science: An Activity-Based Approach . 1st Edn., Springer, New York ( 2011 )

12. Chatti , M. A. , Dyckhoff , A. L. Schroeder , U. , Thüs , H. A reference model for learning analytics . International Journal of Technology Enhanced Learning , Vol. 4 , No. 5 , ( 2012 ) 318 - 331

13. Baker , R.S.J. and Siemens , G. Educational data mining and learning analytics . The Cambridge Handbook of the Learning Sciences . Cambridge University Press ( 2014 ) 253 - 274

14. Conde-González , M. & Hernandez García , A. Using Learning Analytics tools in Engineering Education , Conference: LASI Spain ( 2016 )

15. Hamid Tarmazdi , Rebecca Vivian , Claudia Szabo , Katrina Falkner , Nickolas Falkner, Using Learning Analytics to Visualise Computer Science Teamwork , Proceedings of the ACM Conference on Innovation and Technology in Computer Science Education , ( 2015 )

16. Dierenfeld , H. , Merceron , A. Learning Analytics with Excel Pivot Tables . 1st Moodle Research Conference ( 2012 )

17. Phillips , R. , Maor , D. , Cumming-Potvin , W. , Roberts , P. , Herrington , J. , Preston , G. , Moore , E. and Perry , L. Learning analytics and study behaviour: A pilot study . In: ASCILITE ( 2011 ) 997 - 1007

18. Sharkey , M. Academic analytics landscape at the University of Phoenix, Proceedings of the 1st International Conference on Learning Analytics and Knowledge ( 2011 )

19. Friesen , N. Learning Analytics: Readiness and Rewards/L'analyse de l'apprentissage: etat de preparation et recompenses. Canadian Journal of Learning and Technology/La revue canadienne de l'apprentissage et de la technologie, 39 ( 4 ), ( 2013 ) 1 - 12

20. Key , A , Howe , B. , Perry , D. , Aragon , C. R. VizDeck: Self-organizing Dashboards for Visual Analytics . ACM SIGMOD International Conference on Management of Data , ( 2012 ) 681 - 684

21. Olivares , D. Exploring Learning Analytics for Computing Education . In Proceedings of the eleventh annual International Conference on International Computing Education Research (ICER '15) . ( 2015 ) 271 - 272

22. Caspersen , M.E. , Bennedsen , J. Instructional design of a programming course: A learning theoretic approach . Proceedings of the Third International Workshop on Computing Education Research ACM ( 2007 ). 111 - 122

23. Zendler , A. , Klaudt , D . Instructional Methods to Computer Science Education as Investigated by Computer Science Teachers- Journal of Computer Science 2015 , 11 ( 8 ): 915 - 927

24. Titterton , N., C.M. Lewis and M.J. Clancy . Experiences with lab-centric instruction . Comput. Sci. Educ ., 20 : ( 2010 ) 97 - 102

25. Pucher R. , Lehner M. Project Based Learning in Computer Science - A Review of More than 500 Projects. Procedia - Social and Behavioral Sciences . Volume 29 , ( 2011 ) 1561 - 1566

26. Buendia , F. , Benlloch , J.V. Learning analytics sources: Beyond learning platforms . International Symposium on Computers in Education SIIE ( 2016 ) 1 - 6