when using Learning Analytics for Learners

Abstract

In this position paper we suggest that Learning Analytics for Learners (LAL) requires new digital competences of learners. The use of Open Learner Models, Learning Analytics, and Visual Analytics creates new opportunities for learners, but our own research indicates that digital competence to take advantage of these opportunities needs to be extended to specifically address data literacy and use. We outline how our framework for data literacy and use for teaching can guide research to develop a framework for data literacy and use for learning, and identify a number of research questions. •Applied computer→ Education •Human-centred computing→ Visualisation→ Visualization application domains •General and reference→ •Empirical studies •Design •Information systems→ •Data mining •Human centered computing→ •User models •User studies •Usability testing •HCI theory, concepts and models •Interactive systems and tools •Visual analytics •Student assessment

1. INTRODUCTION

Technology rich classrooms (e.g., [1]) offer new pedagogical possibilities, new ways of learning, and generate new types of data that can be used both for assessment and for improving teaching and learning [2]. While these technology and information-rich classrooms enable 21st Century pedagogy (e.g., [3], [ 4 ]), they also place new demands and require new competence of teachers and learners. That is, these data-rich work environments require new knowledge, skills, and abilities to lever the possibilities in, and beyond these classrooms. Accordingly, teacher capacity development for using ICT and data for their students’ learning and for their own professional development ([5], [ 6 ]) needs to be fostered. Furthermore, there are new demands on students to use and understand data and its visualisation, for learning. Open Learner Models, Learning Analytics, and Visual Analytics can be used to provide evidence (data) to students for self-reflection and this position paper outlines what we mean needs to be explored with respect to data literacy and use for learners when these models and methods are taken into use with learners.

2. BACKGROUND

In our perspective there are three areas of research are relevant for LAL: Open Learner Models; Learning Analytics; and, Visual Analytics. An Open Learner Model (OLM) [ 7 ] is an artefact that is well suited to visualise competence models of student’s learning, and learning analytics can create the evidence to be modelled in the learner model that the OLM visualised, and can also be used to analyse students’ use of the OLM. Each of these areas is briefly presented below in the context of the recently completed EU NEXT-TELL project, our nationally funded project iComPAss, and our focus on data literacy and use for learning in our research Centre for the Science of Learning and Technology (SLATE).

Open Learning Models (OLM): In the Next-Tell project we used an Open Learner Model [ 7 ] to facilitate both self-reflection on the part of learners, and teacher planning and decision- making ([ 8 ], [ 9 ]). An OLM is similar to student progress and performance reports, however, it does more than report progress, it models and externalises competences and skills. Built on a learner model (LM), an OLM traditionally is a representation of a learner’s skill and abilities inferred while the learner interacts with learning material in an intelligent tutoring system (ITS). State-of-the-art research is extending the use of OLMs (beyond ITSs) to situations where there are multiple data sources, both automatic and manual, feeding its learner model ([ 10 ], [ 11 ], [2]), and visualising the learner’s competences in various ways ([2]; [ 8 ]). The OLM thus facilitates reflection, planning, self-assessment and self-directed learning [ 7 ], In our iComPAss project we investigate the use of an OLM in professional competence development for firefighters and in the education of leaders for healthcare management, and in SLATE we continue the work from NEXT-TELL in investigating the use of OLMs with students.

Learning Analytics and Data Mining: Learning Analytics (LA) “is the use of intelligent data, learner-produced data, and analysis models to discover information and social connections, and to predict and advise on learning.” and as such has a predictive nature. LA has been used to provide data for various stakeholders, such as teachers, school leaders, policy makers, and even for learners themselves. Important for LA to be useful for leaners is visualisations. In NEXT-TELL Kickmeir-Rust et al. [ 12 ] used LA to analyse several learning tools and input the evidence in the OLM.

Visual Analytics: Visual analytics is “the science of analytical reasoning facilitated by interactive visual interfaces.” ([ 13 ], [ 14 ] ). Applying visual analytics methods to education and training has the potential to enable decision makers, including learners, to see and explore learner data in order to have evidence for decision making. In iComPAss and SLATE we investigate this potential. There are multitudes of ways in which data can be visualised and our research will focus on how best to visual the competences to support self-reflection in an individual learner.

We identify another important challenge that LAL raises – understanding the new competence demands on learners when using LA for learning. Thus, we are interested in a dialogue at LAL on the competences that learners need in order to understand OLMs, the use of LA, and the visualisations that are presented to them either through dashboards or other means.

In SLATE we will be carrying out research to identify these competences in order to understand better data literacy and use related to using data for learning. We are building on the work we have carried out in NEXT-TELL with respect to data literacy and use for teaching [5]. This is presented briefly after the introduction of the NEXT-TELL vision of the information and technology rich classroom.

3. THE NEXT-TELL CLASSROOM

As indicated in the introduction of this paper, the insurgence of use of technology for learning has increased the amount and types of data available in technology and information rich classrooms, and tools to handle this data are emerging. A NEXT-TELL classroom is such a classroom where a plethora of data is generated through the use of digital tools and services. Figure 1 can be used to explain this vision.

PR A Next-Tell classroom is also a learning environment where tools such as Moodle (LMS), Google Docs, Immersive Environments (such as Second Life (secondlife.com) or OpenSim (opensimulator.org)), formative assessment tools, digital educational games, or electronic tests can be found. As students use these tools various types of data that are generated through the use of the tools are tracked and stored as evidence of use. These might include log files such as Chat logs from OpenSim or traces of who wrote what in a Google doc, a document such as the final text written in Google doc, the item responses (i.e., answers given) to a test, a video saved to an e-portfolio in Moodle, or a grid of responses to a Repertory Grid exercise (e.g., RGFA tool [ 15 ]), or an activity visualisation of one’s participation in OpenSim. The collected evidence can either be used directly or be processed electronically by an assessment analysis tool such as PRONIFA that uses the CbKST algorithm ([ 12 ]) or manually by the teacher (e.g., evaluate the video created by a student) transforming the evidence into information that can be used to update the student’s competence model in the NEXT-TELL Independent Open Learner Model ([ 16 ]). Ideally as much as possible is automated. We can look at this process from a teacher perspective, see figure 2. The teacher uses a planner to specify learning activities and assessments (fully digital if possible). The activities and assessments get delivered automatically to the students, or the teacher manually delivers them to the students (e.g., a test on paper). As the student engages in the learning activity or assessments data is being recorded for further analysis by either another tool or the teacher manually. The analysis of the data is feed into the independent OLM where the student’s digital competence model is updated and visualised ([ 16 ]). The teacher can use this student information to adjust their pedagogical practice to an individual student (view an individual competence model) or to the entire class (view the competence model of the class). Using this teacher perspective and our experience in working with teachers we identified and identify data literacy and use skills that teachers would have to develop for a NEXT-TELL classroom.

4. DATA LITERACY AND USE FOR

TEACHING AND LEARNING

From our experience in using NEXT-TELL tools with high school science teachers and their students over the past two years we suggest that the existing definitions of digital competence need to be extended to specifically address data literacy and use. In NEXT-TELL we explored the knowledge, skills, and abilities required to make effective use of the new kinds of data and information available for teaching, assessment, and diagnosing learning in the technology and information rich classroom. Based on these findings we developed a framework that encompasses the various aspects of data literacy and use required by teachers, and illustrate these using examples from the NEXT-TELL project [5]. In Wasson & Hansen [5] we present a framework for data literacy and use for teaching that encompasses the teacher’s need to have an understanding of (1) how the configuration of technology tools or applications impacts the data generated (conceptually and technically), (2) what and how data is generated by a tool or

application, (3) how the data is analysed (if it is done automatically), (4) how the data/data interpretations can be used in a pedagogical manner (for both teaching and formative assessment), and (5) how the data and data interpretations can be shared. We propose that this framework serve as the basis for developing a framework for data literacy and use for learning. The points in the existing framework will be taken one by one and examined for its relevance for a framework for data literacy and use for learning. For example, with respect to (1) we would need to use the technology tools (e.g., an OLM or an assessment tool) with learners and identify the competence the learner needs in order to understand how various configurations of the tools impacts the data that is generated. For (4) we would have to examine what competence is need to use the available data / data interpretations for learning. In addition, we need to study students using OLMs, LA, and visualisations in their learning and identify the competence needed. Building on the existing framework already raises questions such as:

What do students need to know about data as evidence?

How do the tools the students use for learning generate data that can be used as evidence of their learning? What do students need to know about how learning analytics uses their student data to understand and visualise their learning? What competence is required by learners in order to use the visualisation data for self-reflection, planning, self-directed learning, etc.? What competence is required in order for students to carry out their own learning analytics on their own student data?

What visualisations are best understood by students?

Are students concerned about ethical and privacy issues related to the use of their learning data? Should they be? We imagine that the LAL workshop will raise many more such questions that are important to investigate in order that we can develop a framework for data literacy and use for learning. For example, issues related to privacy, ethics, and data protection are certainly relevant, especially when these are complex issues, which students and teachers may never entirely understand and thus protection must be provided by policy.

5. CONCLUSIONS

As education begins to embrace new trends such as educational data mining, learning analytics, visual analytics, and big data, there will be even more intricate data literacy and use skills that learners (and teachers) will need to develop. Our future research on the role of these approaches in education should be cognisant of the impact on digital competence for learners and can use the framework for data literacy and use for learning to identify how leaners need to be trained on the new approaches and the tools that embrace them. Finally, research is needed to understand the impact on using student data for learners; does it improve learning? We believe that these issues are important for those participating in the LAL workshop to discuss.

6. ACKNOWLEDGMENTS

This research is funded by the Research Council of Norway grant number 246765/H20 and the Centre for the Science of Learning and Technology (SLATE), University of Bergen, Norway.

7. REFERENCES

[1] Groff, J. (2013). Technology Rich Innovative Learning Environments. OECD Report. http://www.oecd.org/edu/ceri/TechnologyRich%20Innovative%20Learning%20Environments%20by% 20Jennifer%20Groff.pdf [2] Reimann, P., Bull, S., Kickmeir-Rust, M., Vatrapu, R. & Wasson, B. (Eds.) (2016). Measuring and visualising competence development in the information‐rich classroom.

London: Routledge. [3] Redecker, Christine, Leis, Miriam , Leendertse, Matthijs , Punie, Yves, Gijsbers, Govert, Kirschner, Paul , Stoyanov, Slavi and Hoogveld, Bert (2011). The Future of Learning: Preparing for Change. JRC IPTS. Seville, Spain. http://www.google.ca/url?sa=t&rct=j&q=&esrc=s&source= web&cd=1&ved=0CB8QFjAA&url=http%3A%2F%2Fftp.jrc.es %2FEURdoc%2FJRC66836.pdf&ei=Rm0hVMKQFYeAywPMgI GgCA&usg=AFQjCNFMqUolTcs_vPPVEbvSFj3mVIcAGA&sig2 =MkAWEbOrh9fVhG3niklN7A&bvm=bv.75775273,d.bGQ

References

[4] Sharples , M. , McAndrew , P. , Weller , M. , Ferguson , R. , FitzGerald, E., Hirst , T. & Gaved , M. ( 2013 ). Innovating Pedagogy 2013 . Open University Innovation Report 2. Milton Keynes: The Open University. ISBN 978-1-78007- 937-0

Wasson , B. , & Hansen , C. ( 2016 ). Data literacy and use for teaching . In P. Reimann, S. Bull , M. Kickmeir-Rust , R.

Vatrapu & B. Wasson (Eds.). Measuring and visualising competence development in the information‐rich classroom , 56 - 73 . London: Routledge.

[6] Luckin , R. , Hansen , C. , Wasson , B. , Avramides , K. , Hunter , J. & Oliver , M. ( 2016 ). Teacher inquiry into students' learning: Research pedagogical innovations . In P. Reimann, S. Bull , M. Kickmeir-Rust , R. Vatrapu & B. Wasson (Eds.).

Measuring and visualising competence development in the information‐rich classroom , 74 - 91 . London: Routledge.

[7] Bull , S. & Kay , J. ( 2007 ). Student Models that Invite the Learner In: The SMILI Open Learner Modelling Framework , International Journal of Artificial Intelligence in Education 17 ( 2 ), 89 - 120 .

[8] Bull , S. , Wasson , B. , Kickmeier-Rust , M. , Johnson , M.D., Moe , E. , Hansen , C. , Meissl-Egghart , G. & Hammermüller , K. ( 2012 ). Assessing English as a Second Language: From Classroom Data to a Competence-Based Open Learner Model , ICCE 2012 , November, Singapore.

[9] Bull , S. and Wasson , B. ( 2016 ). Competence Visualisation: Making Sense of Data from 21st Century Technologies in Language Learning . ReCALL, 28 ( 02 ): 147 - 165 .

[10] Morales , R. , Van Labeke, N. , Brna , P. & Chan , M.E. ( 2009 ). Open Learner Modelling as the Keystone of the Next Generation of Adaptive Learning Environments . In C. Mourlas & P. Germanakos (eds), Intelligent User Interfaces , 288 - 312 , London: ICI.

[11] Mazzola , L. & Mazza , R. ( 2010 ). GVIS: A Facility for Adaptively Mashing Up and Representing Open Learner Models , In M. Wolpers et al (eds), EC-TEL 2010 , Berlin: Springer Verlag, 554 - 559 .

[12] Kickmeier-Rust , M. and Albert , D. ( 2016 ). CompetenceBased Knowledge Space Theory: Options for the 21st Century ClassroomIn P . Reimann, S. Bull , M. KickmeirRust , R. Vatrapu & B. Wasson (Eds.). Measuring and visualising competence development in the information‐rich classroom , 74 - 91 . London: Routledge.

[13] Thomas , J. and Kielman , J. ( 2009 ). Challenges for visual analytics . Information Visualization , 8 ( 4 ), 309 - 314 .

[14] Thomas , J.J. & Cook , K.A . (Eds)( 2005 ). Illuminating the path: The research and development agenda for visual analytics . IEEE Computer Society , 2005 .

[15] Vatrapu , R. Reimann , P. , Hussain , A & Pantazos , K. ( 2016 ). Repertory Grids for Formative Assessment: Concept, Method, Tools, and Findings . In P. Reimann, S. Bull , M. Kickmeir-Rust , R. Vatrapu & B. Wasson (Eds.). Measuring and visualising competence development in the informationrich classroom , 121 - 136 . London: Routledge.

[16] Johnson , M.D. & Bull , S. ( 2016 ). Designing for Visualisation of Formative Information on Learning . In P. Reimann, S. Bull , M. Kickmeir-Rust , R. Vatrapu & B. Wasson (Eds.). Measuring and visualising competence development in the information‐rich classroom , 221 - 234 . London: Routledge.