Orchestrating 21 st Century Learning Ecosystems using Analytics Michail N. Giannakos Department of Computer and Information Science Norwegian University of Science and Technology (NTNU), michailg@idi.ntnu.no Abstract: The systematic use of learning technologies has become widely employed in the past years, diverse technologies have been applied in a variety of teaching practices; for instance learning tools which allow you to flip the classroom or monitor and enhance other learning practices. However, the developed systems are only a subset of different kinds of learning materials and learning tools that an educator should take into consideration; and most importantly they do not offer an overview of the different learning experiences and dynamics. Information gathered from multiple technologies via learning analytics can allow us to orchestrate the respective technologies and practices, and support better learning. Therefore, there is an emerging need for the learning technology community to develop new knowledge about how analytics allow us to better orchestrate different e-learning tools and learning practises. In this paper, we present indicative examples of how learning analytics from different sources can allow us to make sense of learning phenomena. Our aim is to provide insights of how heterogeneous learning analytics can help us to better understand and further develop teaching approaches enhancing students’ dynamics and needs in a ubiquitous learning era. Keywords: heterogeneous learning analytics, learning ecosystems, learning orchestration, ubiquitous learning. Introduction Many scholars have used "orchestration" to refer to the design and real-time management of multiple classroom activities, various learning processes and numerous teaching actions (Dillenbourg & Jermann, 2010). In 21st century’s learning spaces, instructors have to orchestrate multiple tools in the best possible way. They need a fine-grained control of time and progress. To do so, they need to translate students’ interactions into a sequence of useful information (e.g., learning progress). Contemporary learning practices and scenarios integrate individual activities (e.g. self-reading), team-work (e.g. problem solving) and class-wide activities (e.g. quizzes, lectures), an important element of these integrated activities is the required monitoring and management “orchestration”. Hence, understanding students’ interactions is even more essential in today’s education. Siemens (2003) described learning ecosystem as a mean for orchestrating a variety of learning approaches given by the varied characteristics of learning processes. Learning ecosystem is seen as an environment which is “consistent with (not antagonistic to) how learners learn.” His approach focused on the learning process dimension and takes into account different forms of learning analytics, like learners’ characteristics and interaction with the learning environment. The field of learning analytics is broadly concerned with how the collection, analysis and application of data can be used to improve processes and outcomes related to learning (Siemens et al., 2011). Increasing motivation, autonomy, effectiveness, and efficiency of learners and teachers is an important driver for learning analytics developments (Buckingham Shum, Gašević, & Ferguson, 2012). Learning analytics allow instructors and researchers to discover important learning episodes and phenomena (e.g., moment of learning/misconception), get better understanding of learner characteristics/needs; and understand the features that make the learning material effective. There is therefore a need to leverage learning analytics capabilities to assist instructors in the orchestration of their learning practices and respective technologies. During the last years several technologies to assist students’ learning have been developed. For instance various Learning Management Systems (LMSs), classroom response systems and other ubiquitous learning technologies have proven their ability to improve students’ learning experience. Triangulating analytics, from different sources like video learning analytics and LMSs, has proven its enormous potential on 47 Copyright © 2016 for this paper by its authors. Copying permitted for private and academic purposes. discovering important learning episodes and phenomena as well as portraying better understanding of learners’ experience (Giannakos, Krogstie, & Aalberg, 2016). However, the highly promising potential of combining analytics from many and diverse resources to better orchestrate e-learning tools and learning remains unexplored. Collecting and managing integrated learning analytics from different learning spaces like video lectures, wikis, mobile learning applications, quizzes, LMSs and so forth, will allow us to better understand students’ progress, experience and usage behavior. Exploring important issues like, the dynamics between different e-learning tools, students’ prioritization of e-learning tools, the association of different orchestrations with students’ learning experience and the combination of different learning practices with different set of e- learning tools, will allow us to construct novel principles and technical knowledge in order to increase benefits arising from the efficient orchestration. Thus, there is a need to leverage learning analytics capabilities to formulate a conceptual framework for assisting researchers and instructors in improving the orchestration of e- learning tools and practices as well as harmonizing heterogeneous learning analytics streams. Background and Open Research Question A traditional ecosystem has been described as “the complex of living organisms, their physical environment, and all their interrelationships in a particular unit of space” (Encyclopedia Britannica (2011)). By applying this simple and good working definition to learning; we can describe a learning ecosystem “as the complex of living organisms in a learning environment (e.g. students, educators, resources), and all their interrelationships in a particular unit of space (can be digital or physical)” (Giannakos, Krogstie, & Aalberg, 2016). In a learning ecosystem it is important to consider the interrelationships of the main actors (students and educators) but also the role of the learning space (both digital and physical). The learning space is by analogy the physical environment in a traditional ecosystem, includes (organisms) information and digital resources like slides, lecture recordings, blog entries and forum discussions; but also physical materials like books, notes and handicrafts, to mention few. The space is where teaching or learning is happening and where such processes and interrelationships are conducted. The interrelationships exist (Chang & Guetl, 2007; Shum & Ferguson, 2012; Sharples, 2013) between the main actors (students and educators), the main actors with the resources, and the resources themselves (e.g. recommender systems). Those interrelationships shape the quality and value of students’ learning experience; heterogeneous learning analytics have a significant role to play in the near future, since they can help us to better understand and further develop teaching approaches enhancing students’ dynamics and needs in the emerging ubiquitous learning era of the 21st century. Triangulating learning analytics from different learning spaces will definitely allow us to better understand and improve students’ progress and experiences. In fact we contend that the most compelling effect of learning analytics lies on their integration and synthesis in order to portray students’ learning experience. The thesis of this article is that learning analytics can inform us to better orchestrate different e-learning tools and learning practises. In particular, we pose the following open research questions as a way to guide our future work: RQ1: What kind of learning analytics can help orchestrate a learning ecosystem? RQ2: How can different learning analytics be integrated to improve educators’ decisions? RQ3: How do integrated learning analytics contribute to the creation of more meaningful and efficient set of technologies for learning? and how can different technologies be coupled to help students overcome the difficulties they face while keeping them engaged? In order to cope up with the aformentioned research questions there is a need for empirically-oriented research to develop new knowledge about how analytics allow us to better orchestrate different tools and practises. Evidence based models, tools and recommendations/guidelines drawn from large scale user-oriented studies will allow us to shed light and pave the way for richest learning experiences. The empirically-oriented research needs to be utilized in an iterative process of: design, implementation, analysis, and revision. This will allow us to address educational problems in real-world settings, with two primary goals: to develop knowledge and solutions (McKenny & Reeves, 2012). By iteratively, designing different orchestrations, implementing them and collecting/combining diverse analytics we will be able to portray students’ progress and interaction with the materials. This will allow us to understand 48 how different orchestrations support students’ awareness, experience, participation, and knowledge acquisition differently. Integration of the empirical results and requirements as well as refinement of a framework with practical (e.g., best practices) and technical (e.g., systems’ design guidelines) knowledge (Figure 1), will help us to produce research that contributes towards the orchestration of multiple technologies to support better learning and teaching. Figure 1. Graphical representation of the research approach Early Reflections As aforementioned, in order to be able to cope with these critical research questions there is a need for repetitive large scale empirical studies. However, in order to have some initial reflections on the thesis of this article we attempted to provide some insights of how analytics from different sources can help us to better understand students’ learning. In other words, the goal of this empirical validation is to provide some analytics-based evidence regarding the importance of the proposed research questions and approach. The early results should not be seen as an evaluation of the research questions (since they are definitely not), but as reflections rising from a particular case as well as empirical evidence for further development of the research area. The case study in an introductory computer science course, named web technology. The focus of this course is on the World Wide Web as a platform for interactive applications, content publishing and social services. By the end of the course students are expected to be able to design and develop web-pages and web- applications. Students have to deliver specific assignments, work with a self-selected group project and take written examination; these three components are also the evaluation criteria. The course materials, digital communication as well as the assignments and project-work are orchestrated from a Learning Management System (LMS). This fundamental knowledge in this course was made available beforehand using video lectures, and weekly exercises. Upon students’ completion of the video lecture, instructors were able to access all the video analytics and visualize students’ watching behavior. Such information allowed us to make sense of students’ engagement with the video lectures. In order to recall students’ knowledge we used a gamified classroom response system at the beginning of the class. The instructor prepared a session with questions related to the basic knowledge, supported with different forms of audio visual materials (e.g., videos). The class was equipped with a projector, which was used to project the main screen of the quiz/game, and each student used his/her own mobile phone to give the answer to the respective question (typical setup of clickers). At the end of the each class, the instructor could download all the collected analytics of the quiz/game (e.g., correct answers, response time) and explore students’ understanding. With the visualization of the students’ watching engagement (based on repeated views, skips etc.) and score on the gamified classroom response system, we reach the conclusion that the scores are highly associated with the video engagement. As we can see from Figure 2, the highly watched videos resulted high scores during the quiz. Hence, by triangulating analytics from different resources we were able to understand why students’ scored lower in these particular quizzes. 49 Time Time 9 8 7 Score 6 5 4 Mobile Mobile Mobile Mobile Mobile Mobile Mobile Quiz 1 Quiz 2 Quiz 3 Quiz 4 Quiz 5 Quiz 6 Quiz 7 Time Time Figure 2. Visualization of the students’ video lecture engagement and score on the gamified classroom response system This particular example is indeed very simple, it however allows us to understand why students’ had low/high performance the learning technology A (classroom response system) by looking into the learning analytics collected from the learning technology B (video learning analytics). Hence, by integrating heterogeneous learning analytics streams from different learning spaces will definitely allow us to understand the cause of different learning phenomena as well as improve students’ experience in 21st century learning ecosystems. Conclusions Today there is a huge demand for innovative learning and professional development, with strong impact on both academia and industry. This demand is intertwined with the move towards new modes of new ubiquitous learning technologies. Contemporary learning systems and their analytics are only a subset of different kinds of learning materials and learning tools that an educator should take into consideration; and most importantly they do not offer an overview of the different learning experiences and dynamics. Information gathered from multiple technologies via learning analytics can allow us to orchestrate the respective technologies and practices, and support better learning. Therefore, there is an emerging need for the learning technology community to develop new knowledge about how analytics allow us to better orchestrate different e-learning tools and learning practices. Making sense of heterogeneous learning analytics can bring innovation by encouraging schools, universities and life-long learning initiatives to adopt new learning practices. In this work-in-progress contribution, we explore the notion of learning ecosystem, as well as we present an indicative example of how learning analytics from different sources can allow us to make sense of 50 learning phenomena. Our overreaching objective is to provide insights of how heterogeneous learning analytics can help us to better understand and further develop teaching approaches enhancing students’ dynamics and needs in a ubiquitous learning era. References Buckingham Shum, S., Gaševic, D., & Ferguson, R. (Eds.). (2012). Learning Analytics and Knowledge, ACM. Chang, V. & Guetl, C. (2007). E-learning ecosystem (ELEs)-a holistic approach for the development of more effective learning environment for small-and-medium sized enterprises (SMEs). In: Digital Ecosystems and Technologies Conference, pp. 420–425. New York: IEEE Press Cruz-Benito, J., García-Peñalvo, F. J., & Therón, R. (2014). Defining generic data collectors for Learning Analytics: Facing up the heterogeneous data from heterogeneous environments. In Proc. IEEE In. Conference on Advanced Learning Technologies (ICALT ’14), 365-366. Dillenbourg, P., & Jermann, P. (2010). Technology for classroom orchestration. In New science of learning (pp. 525-552). Springer New York. Encyclopedia Britannica. (2011). Ecosystem. Encyclopedia Britannica, Inc. (Eds.) Last retrieved January 21st, 2016, http://www.britannica.com/EBchecked/topic/178597/ecosystem Giannakos, M. N., Krogstie, J., & Aalberg, T. (2016). Toward a Learning Ecosystem to Support Flipped Classroom: A Conceptual Framework and Early Results. In State-of-the-Art and Future Directions of Smart Learning (pp. 105-114). Springer Singapore. McKenney, S. & Reeves, T. (2012). Conducting Educational Research Design: What, Why and How. NY: Taylor & Francis Shum, S. B., & Ferguson, R. (2012). Social Learning Analytics. Educational Technology & Society, 15(3), 3– 26. Sharples, M. (2013). Shared orchestration within and beyond the classroom. Computers & Education, 69, 504- 506. Siemens, G., Gašević, D., Haythornthwaite, C., Dawson, S., Buckingham Shum, S., Ferguson,R., et al. (2011). Open Learning Analytics: An Integrated and Modularized Platform (Concept Paper): SOLAR. Siemens, G. (2003). Learning Ecology, Communities, and Networks - Extending the classroom, elearnspace, last retrieved Apr. 21st, 2015, http://www.elearnspace.org/Articles/learning_communities.htm Acknowledgments This work was funded by the Norwegian Research Council under the projects FUTURE LEARNINGS (number: 255129/H20) and SE@VBL (number: 248523/H20). 51