Seeing Learning Analytics Tools as Orchestration Technologies: Towards Supporting Learning Activities across Physical and Digital Spaces

Abstract

This panel paper proposes to consider the process that learners or educators commonly follow while interacting with learning analytics tools as part of an orchestration loop. This may be particularly valuable to facilitate understanding of the key role that learning analytics may have to provide sustained support to learners and educators. The complexity of learning situations where learning occurs across varied physical spaces and multiple educational tools are involved requires a holistic and practical approach. The proposal is to build on principles of orchestration that can help link technical and theoretical aspects of learning analytics with the practitioner. The panel paper provides: 1) a brief description of the relevance of the notions of orchestration and orchestrable technologies for learning analytics; and 2) the illustration of the orchestration loop as a process followed by learners or educators when they use learning analytics tools.

Keywords

• learning analytics
• classroom orchestration
• Cross-LAK
• physical and digital spaces

Introduction

It has been emphasised that learning analytics research and practice has a holistic and human-centred perspective, primarily aimed at leverage human judgement and understanding (Siemens & Baker, 2012) . Learning analytics is also focused on empowering educators and learners and thus requires to keep the human in the loop. Moreover, the learning analytics community is differentiated from other educational data science perspectives because of its particular view of learning as a whole, complex activity. This includes understanding that students’ activity and their actual learning not only occurs while they interact with single learning tools (e.g. with an intelligent tutoring system or the learning management system only), but with a variety of tools, connected or unconnected with each other, and commonly distributed across different physical s paces (PérezSanagustín et al., 2012 ).

A promising yet underexplored perspective to understand how students and educators may interact with learning analytics tools to gain a more holistic view of activity across physical and digital spaces is the metaphor of orchestration. Orchestration takes account of the variability and complexity of classrooms (and blended learning scenarios) by considering this as a question of “usability in which the classroom is the user” (Dillenbourg et al., 2011) . It also recognises the key role of educators in adapting the available pedagogic and technological resources to help students achieve their intended learning goals (Dillenbourg, et al., 2011) . This perspective emphasises that technology should be practical, minimalist, controllable and flexible to facilitate rather than hinder the learning activities (Dillenbourg, 2013) . An evolved notion of this approach has been embraced by the communities of Technology Enhanced Learning and Computer-Supported Collaborative Learning (Prieto et al., 2015) . One of the reasons for this is that it has shown some potential to help to link research-based results with everyday educational practice.

There has only been a small number of research outputs mentioning orchestration and learning analytics together (e.g. Martinez-Maldonado et al., 2016; Rodríguez Triana et al., 2014; Verbert et al., 2013) . However, there is an implicit overlap in both perspectives, particularly because learning analytics tools commonly support educators and learners by making visible aspects of their learning in order for them to take some action as a consequence.

This panel paper is aimed at generating discussion about the relevance of the notion of orchestration technology for learning analytics; and the notion of orchestration loop as a process followed by learners or educators when they use learning analytics tools.

Orchestration Technology in Learning Analytics

Prieto et al.’s (2015) orchestration framework identifies 4 main orchestration tasks that educators commonly have to perform. These are: 1) Design and planning; 2) Regulation and management; 3) Adaptation, flexibility and intervention; and 4) Awareness and assessment. Orchestration technology may support the management of the orchestration or some part of it, in one or more of these orchestration tasks. This includes, for example, interfaces that help teachers manage the class workflow, enhance their awareness or track students’ progress, or re-design the tasks after looking at the data generated in previous activities. We can easily realise that learning analytics tools are currently mostly used to support awareness and different sorts of assessment. Thus, learning analytics tools can be considered as a special type of orchestration tool just by definition. However, MartinezMaldonado et al. (2016) demonstrated that learning analytics tools can also be used to provide support in the other orchestration tasks (e.g. during the learning design, to regulate class scripts or to perform semi-automated interventions). Mike Sharples (2013) also introduced the notion of shared orchestration, which suggested that these tasks are not just limited to the things that educators have to do, but that can be distributed among other stakeholders to different extents. For example, in self-regulated learning scenarios, the role of the teacher may exist but students have to orchestrate their own learning. This is particularly important for learning analytics for learners (Bull et al., 2016) .

By contrast, an orchestrable technology allows teachers to configure or adapt the use of the technology for different purposes, before the class and/or while the class is being conducted (Tchounikine, 2013). This can help teachers’ target the technology to a range of pedagogical objectives rather than restricting the learning analytics tool to specific educators (or students) usage. Examples of this kind of tools include efforts to create configurable open dashboards that can be customised by educators to accomplish their particular needs – see Open Learning Analytics (Siemens et al., 2011) . There is also a nascent interest in collecting data from multiple data sources and trying to make sense of the learners’ heterogeneous data at a higher level. An example of this is the CLA Toolkit (Kitto et al., 2015) which provides an infrastructure to collect gather information from learner’s activity through multiple social media tools (e.g. facebook, twitter, youtube). The challenge for an educator would be how to coordinate the pedagogical approach to teaching using multiple tools but also how to make sense of the partially collected data as part of the learner’s activity may be tracked across multiple platforms.

In short, taking an approach of orchestration for learning analytics is a dynamic perspective that has the potential to attend authentic issues considering that learning activities can occur in the classroom or in other spaces. Moreover, if multiple tools are used, there is an increase in the orchestration load too ( Prieto et al., 2012 ). For learning analytics, this may generate additional technical and pedagogic challenges to create tools that can support educators or students in making sense of learning data coming from multiple heterogeneous learning systems.

Iterative Orchestration of Learning Analytics Tools

Verbert et al. (2013) proposed that the design of visual learning analytics tools (such as dashboards) can be built and developed following an orchestration idea of “modest computing” approach (Dillenbourg, et al., 2011) . This approach tries to empower people with key tools and/or information to take their own decisions, rather than automating decisions on their behalf. With this perspective in mind, the user has a crucial role in the loop where the educational technology and the learning analytics tool sits. We can understand the notion of iteration in orchestration of learning activities through learning analytics tools from a personal informatics perspective as a starting point. This has been described by Verbert et al. (2013) as the process users follow to: have access to data (i. awareness); ask questions and assess the relevance of the data (ii. reflection); answer questions, getting new insights (iii. sensemaking); to finally induce new meaning or insights (iv. impact).

This four-stage iterative process occurs while users interact with a learning analytics tool in a given phase. This process, from an orchestration perspective, mimics the orchestration loop that includes: the teacher or the student monitors the classroom or learning situation (possibly aided by a learning analytics tool), compares its state to some intended state (assessment), and adapts the scenario accordingly (intervention). This loop highlights two key tasks in the orchestration function that can be aided by learning analytics tools: state awareness (which can be improved by learning analytics tools that make visible aspects of the learning activity that may otherwise be hard to see) and workflow manipulation (which can be improved by enhancing the decision making process of the teacher or the students to self-regulate their learning).

Implications

Learning analytics can have a key role in supporting both face-to-face and blended learning activities. The learning activity is physically and socially situated and thus is strongly shaped by the tools, space and social dynamics where it sits. However, non-online learning activities have been considerably neglected by the learning analytics efforts.

The orchestration metaphor may be relevant for generating learning analytics solutions in authentic learning settings. However, how can we start the conversation between the two very different academic communities? Moreover, how can the different actors (e.g. teachers, students, developers and designers of learning analytics tools, educators and researchers) communicate and gain common understanding of the particular needs and the mutual objectives that each has?

Orchestration may be particularly important for scenarios where learning occurs in different physical and digital spaces because of its holistic perspective towards the different tasks that educators and/or learners need to do that can shape learning (e.g. design, regulation, management, intervention, evaluation, keep awareness etc.). Then, how can the metaphor of orchestration facilitate the understanding of the complexity of learning activities that occur across multiple digital environments and physical locations? A better understanding of the commonalities and particularities of each field is most needed in order to connect technical and theoretical aspects of the learning analytics research with the real-world practitioners. Tchounikine, Pierre. (2013). Clarifying design for orchestration: orchestration and orchestrable technology, scripting and conducting. Computers & Education, 69(1), 500-503.

Verbert, Katrien, Erik Duval, Joris Klerkx, Sten Govaerts, and José Luis Santos. (2013). Learning Analytics Dashboard Applications. American Behavioral Scientist, 57(10), 1500-1509.

References

Bull , Susan, Blandine Ginon, Judy Kay, and Michael Kickmeier-Rust. ( 2016 ). Learning Analytics for Learners . In Proceedings of the International Conference on Learning Analytics and Knowledge , (pp. to appear).

Dillenbourg , Pierre. ( 2013 ). Design for classroom orchestration . Computers & Education , 69 ( 0 ), 485 - 492 .

Dillenbourg , Pierre, Guillaume Zufferey, Hamed Alavi, Patrick Jermann, Son Do-Lenh, Quentin Bonnard, Sébastien Cuendet, and Frédéric Kaplan. ( 2011 ). Classroom orchestration: The third circle of usability . In Proceedings of the International Conference on Computer Supported Collaborative Learning , (pp. 510 - 517 ). Hong Kong , 4 - 8 July 2011 . New York: Springer

Kitto , Kirsty, Sebastian Cross , Zak Waters, and Mandy Lupton. ( 2015 ). Learning analytics beyond the LMS: the connected learning analytics toolkit . In Proceedings of the Proceedings of the Fifth International Conference on Learning Analytics And Knowledge , (pp. 11 - 15 ). Poughkeepsie, New York: ACM.

Martinez-Maldonado , Roberto, Bertrand Schneider, Sven Charleer, Simon Buckingham Shum, Joris Klerkx, and Erik Duval. ( 2016 ). Interactive Surfaces and Learning Analytics: Data, Orchestration Aspects, Pedagogical Uses and Challenges . In Proceedings of the International Conference on Learning Analytics and Knowledge , (pp. to appear).

Pérez-Sanagustín , Mar, Patricia Santos, Davinia Hernández-Leo, and Josep Blat. ( 2012 ). 4SPPIces: A case study of factors in a scripted collaborative-learning blended course across spatial locations . International Journal of Computer-Supported Collaborative Learning , 7 ( 3 ), 443 - 465 .

Prieto , Luis P , Yannis Dimitriadis , Juan I Asensio-Pérez , and Chee-Kit Looi . ( 2015 ). Orchestration in learning technology research: evaluation of a conceptual framework . Research in Learning Technology, 23 .

Prieto , Luis Pablo, Juan Alberto Muñoz-Cristóbal, Juan Ignacio Asensio-Pérez, and Yannis Dimitriadis. ( 2012 ). Making Learning Designs Happen in Distributed Learning Environments with GLUE!-PS . In A. Ravenscroft, S. Lindstaedt , C. Kloos & D. Hernández-Leo (Eds.), 21st Century Learning for 21st Century Skills , (pp. 489 - 494 ). Saarbrücken, Germany: Springer Berlin Heidelberg.

Rodríguez Triana , María Jesús, Alejandra Martínez Monés, Juan I Asensio Pérez , and Yannis Dimitriadis. ( 2014 ). Scripting and monitoring meet each other: Aligning learning analytics and learning design to support teachers in orchestrating CSCL situations . British Journal of Educational Technology , 46 ( 2 ), 330 - 343 .

Sharples , Mike. ( 2013 ). Shared orchestration within and beyond the classroom . Computers & Education , 69 ( 1 ), 504 - 506 .

Siemens , George, and Ryan S. J. d. Baker. ( 2012 ). Learning analytics and educational data mining: towards communication and collaboration . In Proceedings of the International Conference on Learning Analytics and Knowledge , (pp. 252 - 254 ). Vancouver, Canada, April 29 - May 2. New York: ACM.

Siemens , George, Dragan Gasevic, Caroline Haythornthwaite, Shane Dawson, S Buckingham Shum , Rebecca Ferguson, Erik Duval, Katrien Verbert, and RSJD Baker. ( 2011 ). Open Learning Analytics: an integrated & modularized platform. Proposal to design, implement and evaluate an open platform to integrate heterogeneous learning analytics techniques .