=Paper=
{{Paper
|id=Vol-1601/CrossLAK16Paper14
|storemode=property
|title=Towards Integrated Learning Design with Across-spaces Learning Analytics: A Flipped Classroom Example
|pdfUrl=https://ceur-ws.org/Vol-1601/CrossLAK16Paper14.pdf
|volume=Vol-1601
|authors=Davinia Hernández-Leo,Abelardo Pardo
|dblpUrl=https://dblp.org/rec/conf/lak/Hernandez-LeoP16
}}
==Towards Integrated Learning Design with Across-spaces Learning Analytics: A Flipped Classroom Example==
https://ceur-ws.org/Vol-1601/CrossLAK16Paper14.pdf
Towards integrated learning design with across-spaces learning
analytics: a flipped classroom example
Davinia Hernández-Leo, Serra Húnter Fellow, ICT Department, Universitat Pompeu Fabra, Barcelona,
davinia.hernandez@upf.edu
Abelardo Pardo, Faculty of Engineering & Information Technologies, The University of Sydney, Australia,
abelardo.pardo@sydney.edu
Abstract: In this paper we discuss work in progress regarding the integration of learning
analytics and learning design in the frame of the Integrated Learning Design Environment
(ILDE). ILDE is a community platform where teachers can design learning activities using
multiple authoring tools. Authoring tools can be generic, meaning that designs authored can
be deployed in multiple learning systems, or specific, when designs authored can be deployed
in particular systems (e.g., mobile learning applications). These particular systems may be
devoted to supporting activities in specific virtual or physical spaces. For across-spaces
learning designs involving multiple systems to support activities in diverse spaces, ILDE
enables the selection and articulation of multiple authoring tools in what we call “design
workflows”. This paper argues that this integrated approach to learning design can also benefit
an articulated, meaningful interpretation of learning analytics across-spaces. This calls for an
extension of ILDE incorporating learning analytics. The proposed extension is illustrated with
activities across-spaces in a flipped classroom scenario.
Keywords: learning analytics, learning design, learning flow across spaces, flipped classroom
Introduction
The learning design research field deals with supporting teachers in shaping the best possible activities for their
learners to learn (Laurillard, 2012; Lockyer, Bennett, Agostinho, & Harper, 2009). The activities should provide
learners with the motivation for learning and offer a set of learning tasks, supporting resources and tools (Mor,
Craft, & Hernández-Leo, 2013). Contributions to learning design include representations, conceptualization
templates, authoring tools, design frameworks and methodologies that support teachers in the creation, sharing
and implementation of learning designs (Hernández-Leo, Moreno, Chacón, & Blat, 2014; Laurillard, 2012;
Lockyer et al., 2009; Mor et al., 2013; Mor & Mogilevsky, 2013). Learning design authoring tools are often
specific, meaning that they support the creation of designs deployable in particular technologies for activities in
virtual or physical spaces; see, for instance, QuesTInSitu for the design of learning activities in geo-located
physical places (Santos, Pérez-Sanagustín, Hernández-Leo, & Blat, 2011).
The Integrated Learning Design Environment (ILDE) is a community platform where teachers can
design learning activities using multiple authoring tools (Hernández-Leo, Asensio-Pérez, Derntl, Prieto, &
Chacón, 2014). The design of across-spaces learning situations typically involves the use of diverse authoring
tools. Each authoring tool serves to create activities to be performed in a particular space; e.g., a location-based
activity outside the classroom and activities in a learning management system (Pérez-Sanagustín et al., 2012).
To support an integrated design of these situations, ILDE enables the selection and articulation of multiple
authoring tools in what we call “design workflows”. In this paper, we argue that this integrated approach to
learning design can also benefit an articulated, meaningful interpretation of learning analytics across-spaces.
This calls for an extension of ILDE incorporating learning analytics aligned with learning design.
Alignment of learning design with learning analytics research has been mostly focused on facilitating
students’ self-regulation, nurture teachers’ monitoring and eventually lead to pedagogical interventions
(Rodríguez-Triana, Martínez-Monés, Asensio-Pérez, & Dimitriadis, 2015; Wise, 2014; Jovanovic et al., 2008).
Moreover, there is an emerging encouraging discussion about the role that learning analytics can have to inform
learning design (Lockyer, Heathcote, & Dawson, 2013; Pardo, Ellis, & Calvo, 2015). The results offered by
learning analytics can provide evidence to evaluate pedagogical plans and to advise their eventual reuse and
redesign. The state of the art in this area is still in its early days but there are already preliminary experiences
that show the potential and challenges of applying learning analytics to support learning (re)design (Mor,
Ferguson, & Wasson, 2015).
74
Copyright © 2016 for this paper by its authors. Copying permitted for private and academic purposes.
Next section describes the integrated approach for learning design supported by ILDE. The following
section elaborates the ideas for extending ILDE with learning analytics with an illustrative example based on the
flipped classroom.
ILDE, an integrated environment for learning design
As aforementioned, learning design tools are varied. They can be oriented to the design of learning activities
compliant with particular pedagogical approaches or support diverse stages of the design process
(conceptualization, authoring, implementation).
Learning design conceptualization tools support teachers in reflecting about the context in which
designs will be applied, e.g., Personas, Factors and Concerns (Mor & Mogilevsky, 2013), or in sketching ideas
for the design, e.g., Learning Objectives, Course Features, Course Map, (Cross, S., Galley, R., Brasher, A.,
Weller, 2012; Mor & Mogilevsky, 2013). Authoring is the step between the conceptualization of the learning
design and its implementation with students, in virtual spaces (e.g., Virtual Learning Environments, VLEs) or in
physical spaces with (partial or complete) support of digital devices (of different kind, from mobile phones to
laptops). Learning design authoring tools enable the production of detailed definitions of learning designs that
can be deployed in a specific learning setting. Examples of authoring tools are Web Collage (Villasclaras-
Fernández, Hernández-Leo, Asensio-Pérez, & Dimitriadis, 2013), for the authoring of collaborative learning
activities; QuesTInSitu, for the design of location-based activities supported by mobile devices (Santos et al.,
2011); or OpenGLM (Derntl, Neumann, & Oberhuemer, 2011), as a more general authoring tool whose designs
can be deployed in learning management systems (Prieto et al., 2013).
An integrative approach to articulate learning design tooling can offer a holistic view of the
pedagogical intent reflected in several tools used along a learning design process. ILDE enables such integrative
approach by integrating multiple existing learning design tools for conceptualization, authoring and
implementation in a single environment (Hernández-Leo, Asensio-Pérez, et al., 2014) (see Figure 1). In ILDE, a
holistic view of the pedagogical intent is facilitated by means of a so-called learning design “workflow”.
Teachers can select which learning design tools, out of the possible options integrated in ILDE, they will be
using in the process of creating a learning design. This approach envisages a scenario where teacher-led inquiry
and learning analytics results can be aligned and interpreted in the frame of the whole output resulting from
learning design workflow.
Figure 1. Schema with some of the tools integrated in ILDE (Hernández-Leo, Asensio-Pérez, et al., 2014)
(several installations of ILDE available at http://ilde.upf.edu/about).
Across-spaces learning situations typically require the use of multiple tools to support activities in
diverse spaces (e.g., mobile learning applications to support activities in the physical space, learning
management systems or virtual worlds to support virtual activities). The corresponding learning design
authoring tools for each activity can be articulated in an integrated way in ILDE learning design workflows.
Learning analytics derived from the diverse tools used to support activities in physical and virtual spaces can be
also in turn documented aggregately in this type of integrated environment. The following section illustrates this
idea with an example of a learning design for flipped classroom activities.
75
Exploring extension with learning analytics in a flipped classroom example
The example selected to illustrate proposed approach is based on the flipped classroom methodology, as an
example of an across-spaces learning situation. In particular, the learning design spans a week and consists of a
preparation task that learners are asked to complete online followed by a set of face-to-face activities in plenary,
tutorial and lab sessions (Pardo et al., 2015).
A possible workflow to follow in the design of flipped learning activities for a week, from
conceptualization to authoring, is shown in Figure 2. This workflow suggests the use of two learning design
conceptualization tools to reflect and document the context (Persona Card, Factors and Concerns), two
additional conceptualization tools (Learning Objectives and Heuristic Evaluation) to sketch and document the
targeted learning and design objectives, the Reauthoring tool (https://bitbucket.org/abelardopardo/reauthoring)
to edit the preparation tasks to be completed online with the support of computer systems and additional
authoring tools to specify the activities that will be carried out in the classroom.
Figure 2. Design of a flipped learning classroom in ILDE (http://ilde.upf.edu/sydneyuni). Clicking on the design
of each activity design, for each space: initial preparation (virtual / before class), plenary session (physical), etc.
leads to the specific design and its analytics (see Figure 3).
A particular example of the application of the workflow to design flipped learning activities for a first-
year Computer Systems course at the university level entails a set of material, social and intentional factors
depending on the context that are reflected in ILDE using conceptualization tools. Concerns mostly rely on the
risks around lack of participation, considering the characteristics of the context (e.g., Personas: in this scenario
typically tech savvy but disengaged profile, with good technical skills but plans to complete the course with
minimum effort). If students do not participate actively in the preparation activities (those scheduled before the
lecture), face-to-face sessions will not be effective. Moreover, if the activities in the plenary session are reduced
to the exposition of factual knowledge, students will perceive no value derived from attending the session and
will resort to view the recording. These concerns lead to explicit design objectives around encouraging student
engagement in the preparation activities, and then schedule face-to-face sessions properly aligned with the
objectives.
The learning design conceptualization undertaken sets the basis for the learning design-decision
making, to be reflected in the actual authoring of the learning tasks that will be proposed to students. The
previous conceptualizations identify as critical the preparation tasks to be done online, in a virtual space.
Therefore, the teacher decides that the preparation tasks will contain a set of engaging exercises that will enable
students to get familiar with new terms and concepts. The set of exercises consists of interacting with online
videos, reading course notes, answering self-assessed formative questions, and providing the solution to a
sequence of concept test questions, all supported by a computer system. Interactive actions, beyond passive
watching of videos, is considered critical to foster engagement. Learners are asked to complete the preparation
76
tasks virtually before coming to the actual physical classroom. For the plenary session, the teacher plans active
learning tasks (short exercises, exchange ideas with neighbors, voting, conducting a discussion, etc.) Tutorial
sessions are based in problem-based learning and lab sessions in practical activities that require the use of
electronic equipment.
The system used to support the preparation tasks has its corresponding data collecting mechanism for
each exercise. The data is processed and provided to teachers, which in turn can feed ILDE to document the
impact of this particular task in the context of the whole learning design and contextual characteristics (see how
this could be implemented in ILDE in Figure 3). The top left graph shows the number ratio of incorrect answers
in a sequence of 12 exercises. The teacher may clearly see how question number 5 has the largest rate, aspect to
be considered if a potential redesign if the task is going to be reused, for example, the following academic year.
Similarly, the top right graph shows the number percentage of correct, incorrect answers, and requests to view
the solutions of two multiple choice questions. This analytics of the activity in the virtual space can be used to
quickly detect questions with unusually high number of incorrect responses, or high number of request to see the
solution is used to detect more difficult questions. Finally, the bottom of the figure shows three histograms with
the number of video events recorded for three videos (from top bar, play, pause, loaded, and finished video).
This visualization can be used to estimate the level of difficulty depending on the percentage of pause events.
Figure 3. Tab with learning analytics information about the impact of the initial preparation task.
Those videos with an unusually large number of pause events may suggest a larger intrinsic difficulty
of the described material or, considering the contextual aspects documented in the Persona Cards, special
problems with the English language used in the videos for certain types of students’ profiles. Similarly, learning
analytics from activities in the physical space can also feed ILDE to also provide impact information of the tasks
carried out in plenary, tutorial or lab sessions (Pardo et al., 2015). By navigating through the learning designs
aggregated in a design workflow, teachers can explore - in an integrative way - the learning analytics of the
completed across-spaces situation. Moreover, because conceptualization aspects are also documented in the
workflow, teachers can interpret the analytics considering the characteristics of the context.
Conclusion
Learning analytics across spaces and tools is challenging and conveys risks related to multiple and
heterogeneous data sources and contextual aspects. This paper argues that risks could be minimize if learning
analytics is aligned with learning design using an integrative approach. The work in progress presented in this
workshop proposes an integration driven by a learning design workflow. A learning design workflow relates the
set of conceptualization (documenting context) and authoring tools (enabling implementation of activities in
particular spaces) used to design the across-spaces learning situation. An illustrative example shows that it is
possible to link learning design of tasks that occurs in different spaces with leanring analytics that exploit data
for tasks distributed across spaces. Focus is on providing insights to educators about what happened in the
77
across-learning situation. A number of challenges remain to make feasible the implementation of the proposed
idea and further investigate its implications, including the collection of data in face-to-face classrooms or the
synchronization of data collection in learning systems, or the meaningful cross-analysis of heterogeneous data
for holistic visualizations.
References
Cross, S., Galley, R., Brasher, A., Weller, M. (2012). OULDI-JISC Project Evaluation Report: the impact of
new curriulum design tools and approaches on institutional process and design cultures. Retrieved from
http://oro.open.ac.uk/34140/
Derntl, M., Neumann, S., & Oberhuemer, P. (2011). Propelling Standards-based Sharing and Reuse in
Instructional Modeling Communities: The Open Graphical Learning Modeler (OpenGLM). In Advanced
Learning Technologies (ICALT), 2011 11th IEEE International Conference on (pp. 431–435). IEEE.
Hernández-Leo, D., Asensio-Pérez, J. I., Derntl, M., Prieto, L. P., & Chacón, J. (2014). ILDE: Community
Environment for Conceptualizing, Authoring and Deploying Learning Activities. In Open Learning and
Teaching in Educational Communities (pp. 490–493). Graz, Austria: Springer.
Jovanovic, J., Gasevic, D., Brooks, C., Devedzic, V., Hatala, M., Eap, T., & Richards, G. (2008). LOCO-
Analyst: semantic web technologies in learning content usage analysis. International Journal of
Continuing Engineering Education and Life Long Learning, 18(1), 54–76.
Laurillard, D. (2012). Teaching as a Design Science: Building Pedagogical Patterns for Learning and
Technology. New York: Routledge.
Lockyer, L., Bennett, S., Agostinho, S., & Harper, B. (2009). Handbook of research on learning design and
learning objects: issues, applications, and technologies (2 volumes). IGI Global, Hershey, PA.
Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: Aligning learning analytics
with learning design. American Behavioral Scientist, 0002764213479367.
Mor, Y., Craft, B., & Hernández-Leo, D. (2013). The art and science of learning design: Editoral. Research in
Learning Technology, 21.
Mor, Y., Ferguson, R., & Wasson, B. (2015). Editorial: Learning design, teacher inquiry into student learning
and learning analytics: A call for action. British Journal of Educational Technology, 46(2), 221–229.
Mor, Y., & Mogilevsky, O. (2013). Learning design studio: educational practice as design inquiry of learning. In
Scaling up Learning for Sustained Impact (pp. 233–245). Springer Berlin Heidelberg.
Pardo, A., Ellis, R. A., & Calvo, R. A. (2015). Combining observational and experiential data to inform the
redesign of learning activities. In Proceedings of the Fifth International Conference on Learning Analytics
And Knowledge - LAK ’15 (pp. 305–309). New York, New York, USA: ACM Press. 3
Pérez-Sanagustín, M., Ramirez-Gonzalez, G., Hernández-Leo, D., Muñoz-Organero, M., Santos, P., Blat, J., &
Kloos, C. D. (2012). Discovering the campus together: a mobile and computer-based learning experience.
Journal of Network and Computer Applications, 35(1), 176–188.
Prieto, L. P., Asensio-Perez, J. I., Munoz-Cristobal, J. A., Dimitriadis, Y. A., Jorrin-Abellan, I. M., & Gomez-
Sanchez, E. (2013). Enabling teachers to deploy CSCL designs across distributed learning environments.
Learning Technologies, IEEE Transactions on, 6(4), 324–336.
Rodríguez-Triana, M. J., Martínez-Monés, A., Asensio-Pérez, J. I., & Dimitriadis, Y. (2015). Scripting and
monitoring meet each other: Aligning learning analytics and learning design to support teachers in
orchestrating CSCL situations. British Journal of Educational Technology, 46(2), 330–343.
Santos, P., Pérez-Sanagustín, M., Hernández-Leo, D., & Blat, J. (2011). QuesTInSitu: From tests to routes for
assessment in situ activities. Computers & Education, 57(4), 2517–2534.
Villasclaras-Fernández, E., Hernández-Leo, D., Asensio-Pérez, J. I., & Dimitriadis, Y. (2013). Web Collage: An
implementation of support for assessment design in CSCL macro-scripts. Computers & Education, 67,
79–97.
Wise, A. F. (2014). Designing pedagogical interventions to support student use of learning analytics. In
Proceedins of the Fourth International Conference on Learning Analytics And Knowledge (pp. 203–211).
Acknowledgments
This research is partly funded by RecerCaixa, the Spanish Ministry of Economy and Competitiveness under
RESET (TIN2014-53199-C3-3-R) and the Maria de Maeztu Units of Excellence Programme (MDM-2015-
0502) and a José Castillejo mobility scholarship granted to DHL by the Spanish Government to visit the
University of Sydney.
78