=Paper=
{{Paper
|id=Vol-1601/CrossLAK16Paper9
|storemode=property
|title=Orchestrating 21st Century Learning Ecosystems Using Analytics
|pdfUrl=https://ceur-ws.org/Vol-1601/CrossLAK16Paper9.pdf
|volume=Vol-1601
|authors=Michail Giannakos
|dblpUrl=https://dblp.org/rec/conf/lak/Giannakos16
}}
==Orchestrating 21st Century Learning Ecosystems Using Analytics==
https://ceur-ws.org/Vol-1601/CrossLAK16Paper9.pdf
Orchestrating 21 st Century Learning Ecosystems using Analytics
Michail N. Giannakos
Department of Computer and Information Science
Norwegian University of Science and Technology (NTNU),
michailg@idi.ntnu.no
Abstract: The systematic use of learning technologies has become widely employed in the
past years, diverse technologies have been applied in a variety of teaching practices; for
instance learning tools which allow you to flip the classroom or monitor and enhance other
learning practices. However, the developed systems are only a subset of different kinds of
learning materials and learning tools that an educator should take into consideration; and most
importantly they do not offer an overview of the different learning experiences and dynamics.
Information gathered from multiple technologies via learning analytics can allow us to
orchestrate the respective technologies and practices, and support better learning. Therefore,
there is an emerging need for the learning technology community to develop new knowledge
about how analytics allow us to better orchestrate different e-learning tools and learning
practises. In this paper, we present indicative examples of how learning analytics from
different sources can allow us to make sense of learning phenomena. Our aim is to provide
insights of how heterogeneous learning analytics can help us to better understand and further
develop teaching approaches enhancing students’ dynamics and needs in a ubiquitous learning
era.
Keywords: heterogeneous learning analytics, learning ecosystems, learning orchestration,
ubiquitous learning.
Introduction
Many scholars have used "orchestration" to refer to the design and real-time management of multiple classroom
activities, various learning processes and numerous teaching actions (Dillenbourg & Jermann, 2010). In 21st
century’s learning spaces, instructors have to orchestrate multiple tools in the best possible way. They need a
fine-grained control of time and progress. To do so, they need to translate students’ interactions into a sequence
of useful information (e.g., learning progress). Contemporary learning practices and scenarios integrate
individual activities (e.g. self-reading), team-work (e.g. problem solving) and class-wide activities (e.g. quizzes,
lectures), an important element of these integrated activities is the required monitoring and management
“orchestration”. Hence, understanding students’ interactions is even more essential in today’s education.
Siemens (2003) described learning ecosystem as a mean for orchestrating a variety of learning
approaches given by the varied characteristics of learning processes. Learning ecosystem is seen as an
environment which is “consistent with (not antagonistic to) how learners learn.” His approach focused on the
learning process dimension and takes into account different forms of learning analytics, like learners’
characteristics and interaction with the learning environment.
The field of learning analytics is broadly concerned with how the collection, analysis and application of
data can be used to improve processes and outcomes related to learning (Siemens et al., 2011). Increasing
motivation, autonomy, effectiveness, and efficiency of learners and teachers is an important driver for learning
analytics developments (Buckingham Shum, Gašević, & Ferguson, 2012). Learning analytics allow instructors
and researchers to discover important learning episodes and phenomena (e.g., moment of
learning/misconception), get better understanding of learner characteristics/needs; and understand the features
that make the learning material effective. There is therefore a need to leverage learning analytics capabilities to
assist instructors in the orchestration of their learning practices and respective technologies.
During the last years several technologies to assist students’ learning have been developed. For
instance various Learning Management Systems (LMSs), classroom response systems and other ubiquitous
learning technologies have proven their ability to improve students’ learning experience. Triangulating
analytics, from different sources like video learning analytics and LMSs, has proven its enormous potential on
47
Copyright © 2016 for this paper by its authors. Copying permitted for private and academic purposes.
discovering important learning episodes and phenomena as well as portraying better understanding of learners’
experience (Giannakos, Krogstie, & Aalberg, 2016). However, the highly promising potential of combining
analytics from many and diverse resources to better orchestrate e-learning tools and learning remains
unexplored.
Collecting and managing integrated learning analytics from different learning spaces like video
lectures, wikis, mobile learning applications, quizzes, LMSs and so forth, will allow us to better understand
students’ progress, experience and usage behavior. Exploring important issues like, the dynamics between
different e-learning tools, students’ prioritization of e-learning tools, the association of different orchestrations
with students’ learning experience and the combination of different learning practices with different set of e-
learning tools, will allow us to construct novel principles and technical knowledge in order to increase benefits
arising from the efficient orchestration. Thus, there is a need to leverage learning analytics capabilities to
formulate a conceptual framework for assisting researchers and instructors in improving the orchestration of e-
learning tools and practices as well as harmonizing heterogeneous learning analytics streams.
Background and Open Research Question
A traditional ecosystem has been described as “the complex of living organisms, their physical environment,
and all their interrelationships in a particular unit of space” (Encyclopedia Britannica (2011)). By applying this
simple and good working definition to learning; we can describe a learning ecosystem “as the complex of living
organisms in a learning environment (e.g. students, educators, resources), and all their interrelationships in a
particular unit of space (can be digital or physical)” (Giannakos, Krogstie, & Aalberg, 2016). In a learning
ecosystem it is important to consider the interrelationships of the main actors (students and educators) but also
the role of the learning space (both digital and physical). The learning space is by analogy the physical
environment in a traditional ecosystem, includes (organisms) information and digital resources like slides,
lecture recordings, blog entries and forum discussions; but also physical materials like books, notes and
handicrafts, to mention few. The space is where teaching or learning is happening and where such processes and
interrelationships are conducted. The interrelationships exist (Chang & Guetl, 2007; Shum & Ferguson, 2012;
Sharples, 2013) between the main actors (students and educators), the main actors with the resources, and the
resources themselves (e.g. recommender systems). Those interrelationships shape the quality and value of
students’ learning experience; heterogeneous learning analytics have a significant role to play in the near future,
since they can help us to better understand and further develop teaching approaches enhancing students’
dynamics and needs in the emerging ubiquitous learning era of the 21st century.
Triangulating learning analytics from different learning spaces will definitely allow us to better
understand and improve students’ progress and experiences. In fact we contend that the most compelling effect
of learning analytics lies on their integration and synthesis in order to portray students’ learning experience. The
thesis of this article is that learning analytics can inform us to better orchestrate different e-learning tools and
learning practises. In particular, we pose the following open research questions as a way to guide our future
work:
RQ1: What kind of learning analytics can help orchestrate a learning ecosystem?
RQ2: How can different learning analytics be integrated to improve educators’
decisions?
RQ3: How do integrated learning analytics contribute to the creation of more
meaningful and efficient set of technologies for learning? and how can
different technologies be coupled to help students overcome the difficulties
they face while keeping them engaged?
In order to cope up with the aformentioned research questions there is a need for empirically-oriented
research to develop new knowledge about how analytics allow us to better orchestrate different tools and
practises. Evidence based models, tools and recommendations/guidelines drawn from large scale user-oriented
studies will allow us to shed light and pave the way for richest learning experiences.
The empirically-oriented research needs to be utilized in an iterative process of: design,
implementation, analysis, and revision. This will allow us to address educational problems in real-world
settings, with two primary goals: to develop knowledge and solutions (McKenny & Reeves, 2012). By
iteratively, designing different orchestrations, implementing them and collecting/combining diverse analytics we
will be able to portray students’ progress and interaction with the materials. This will allow us to understand
48
how different orchestrations support students’ awareness, experience, participation, and knowledge acquisition
differently. Integration of the empirical results and requirements as well as refinement of a framework with
practical (e.g., best practices) and technical (e.g., systems’ design guidelines) knowledge (Figure 1), will help us
to produce research that contributes towards the orchestration of multiple technologies to support better learning
and teaching.
Figure 1. Graphical representation of the research approach
Early Reflections
As aforementioned, in order to be able to cope with these critical research questions there is a need for repetitive
large scale empirical studies. However, in order to have some initial reflections on the thesis of this article we
attempted to provide some insights of how analytics from different sources can help us to better understand
students’ learning. In other words, the goal of this empirical validation is to provide some analytics-based
evidence regarding the importance of the proposed research questions and approach. The early results should
not be seen as an evaluation of the research questions (since they are definitely not), but as reflections rising
from a particular case as well as empirical evidence for further development of the research area.
The case study in an introductory computer science course, named web technology. The focus of this
course is on the World Wide Web as a platform for interactive applications, content publishing and social
services. By the end of the course students are expected to be able to design and develop web-pages and web-
applications. Students have to deliver specific assignments, work with a self-selected group project and take
written examination; these three components are also the evaluation criteria. The course materials, digital
communication as well as the assignments and project-work are orchestrated from a Learning Management
System (LMS). This fundamental knowledge in this course was made available beforehand using video lectures,
and weekly exercises. Upon students’ completion of the video lecture, instructors were able to access all the
video analytics and visualize students’ watching behavior. Such information allowed us to make sense of
students’ engagement with the video lectures.
In order to recall students’ knowledge we used a gamified classroom response system at the beginning
of the class. The instructor prepared a session with questions related to the basic knowledge, supported with
different forms of audio visual materials (e.g., videos). The class was equipped with a projector, which was used
to project the main screen of the quiz/game, and each student used his/her own mobile phone to give the answer
to the respective question (typical setup of clickers). At the end of the each class, the instructor could download
all the collected analytics of the quiz/game (e.g., correct answers, response time) and explore students’
understanding.
With the visualization of the students’ watching engagement (based on repeated views, skips etc.) and
score on the gamified classroom response system, we reach the conclusion that the scores are highly associated
with the video engagement. As we can see from Figure 2, the highly watched videos resulted high scores during
the quiz. Hence, by triangulating analytics from different resources we were able to understand why students’
scored lower in these particular quizzes.
49
Time
Time
9
8
7
Score
6
5
4
Mobile Mobile Mobile Mobile Mobile Mobile Mobile
Quiz 1 Quiz 2 Quiz 3 Quiz 4 Quiz 5 Quiz 6 Quiz 7
Time
Time
Figure 2. Visualization of the students’ video lecture engagement and score on the gamified classroom response
system
This particular example is indeed very simple, it however allows us to understand why students’ had
low/high performance the learning technology A (classroom response system) by looking into the learning
analytics collected from the learning technology B (video learning analytics). Hence, by integrating
heterogeneous learning analytics streams from different learning spaces will definitely allow us to understand
the cause of different learning phenomena as well as improve students’ experience in 21st century learning
ecosystems.
Conclusions
Today there is a huge demand for innovative learning and professional development, with strong impact on both
academia and industry. This demand is intertwined with the move towards new modes of new ubiquitous
learning technologies. Contemporary learning systems and their analytics are only a subset of different kinds of
learning materials and learning tools that an educator should take into consideration; and most importantly they
do not offer an overview of the different learning experiences and dynamics. Information gathered from
multiple technologies via learning analytics can allow us to orchestrate the respective technologies and
practices, and support better learning. Therefore, there is an emerging need for the learning technology
community to develop new knowledge about how analytics allow us to better orchestrate different e-learning
tools and learning practices. Making sense of heterogeneous learning analytics can bring innovation by
encouraging schools, universities and life-long learning initiatives to adopt new learning practices.
In this work-in-progress contribution, we explore the notion of learning ecosystem, as well as we
present an indicative example of how learning analytics from different sources can allow us to make sense of
50
learning phenomena. Our overreaching objective is to provide insights of how heterogeneous learning analytics
can help us to better understand and further develop teaching approaches enhancing students’ dynamics and
needs in a ubiquitous learning era.
References
Buckingham Shum, S., Gaševic, D., & Ferguson, R. (Eds.). (2012). Learning Analytics and Knowledge, ACM.
Chang, V. & Guetl, C. (2007). E-learning ecosystem (ELEs)-a holistic approach for the development of more
effective learning environment for small-and-medium sized enterprises (SMEs). In: Digital Ecosystems
and Technologies Conference, pp. 420–425. New York: IEEE Press
Cruz-Benito, J., García-Peñalvo, F. J., & Therón, R. (2014). Defining generic data collectors for Learning
Analytics: Facing up the heterogeneous data from heterogeneous environments. In Proc. IEEE In.
Conference on Advanced Learning Technologies (ICALT ’14), 365-366.
Dillenbourg, P., & Jermann, P. (2010). Technology for classroom orchestration. In New science of learning (pp.
525-552). Springer New York.
Encyclopedia Britannica. (2011). Ecosystem. Encyclopedia Britannica, Inc. (Eds.) Last retrieved January 21st,
2016, http://www.britannica.com/EBchecked/topic/178597/ecosystem
Giannakos, M. N., Krogstie, J., & Aalberg, T. (2016). Toward a Learning Ecosystem to Support Flipped
Classroom: A Conceptual Framework and Early Results. In State-of-the-Art and Future Directions of
Smart Learning (pp. 105-114). Springer Singapore.
McKenney, S. & Reeves, T. (2012). Conducting Educational Research Design: What, Why and How. NY:
Taylor & Francis
Shum, S. B., & Ferguson, R. (2012). Social Learning Analytics. Educational Technology & Society, 15(3), 3–
26.
Sharples, M. (2013). Shared orchestration within and beyond the classroom. Computers & Education, 69, 504-
506.
Siemens, G., Gašević, D., Haythornthwaite, C., Dawson, S., Buckingham Shum, S., Ferguson,R., et al. (2011).
Open Learning Analytics: An Integrated and Modularized Platform (Concept Paper): SOLAR.
Siemens, G. (2003). Learning Ecology, Communities, and Networks - Extending the classroom, elearnspace,
last retrieved Apr. 21st, 2015, http://www.elearnspace.org/Articles/learning_communities.htm
Acknowledgments
This work was funded by the Norwegian Research Council under the projects FUTURE LEARNINGS (number:
255129/H20) and SE@VBL (number: 248523/H20).
51