In this paper we discuss work in progress regarding the integration of learning analytics and learning design in the frame of the Integrated Learning Design Environment (ILDE). ILDE is a community platform where teachers can design learning activities using multiple authoring tools. Authoring tools can be generic, meaning that designs authored can be deployed in multiple learning systems, or specific, when designs authored can be deployed in particular systems (e.g., mobile learning applications). These particular systems may be devoted to supporting activities in specific virtual or physical spaces. For across-spaces learning designs involving multiple systems to support activities in diverse spaces, ILDE enables the selection and articulation of multiple authoring tools in what we call “design workflows”. This paper argues that this integrated approach to learning design can also benefit an articulated, meaningful interpretation of learning analytics across-spaces. This calls for an extension of ILDE incorporating learning analytics. The proposed extension is illustrated with activities across-spaces in a flipped classroom scenario.
The learning design research field deals with supporting teachers in shaping the best possible activities for their learners to learn (Laurillard, 2012; Lockyer, Bennett, Agostinho, & Harper, 2009). The activities should provide learners with the motivation for learning and offer a set of learning tasks, supporting resources and tools (Mor, Craft, & Hernández-Leo, 2013). Contributions to learning design include representations, conceptualization templates, authoring tools, design frameworks and methodologies that support teachers in the creation, sharing and implementation of learning designs (Hernández-Leo, Moreno, Chacón, & Blat, 2014; Laurillard, 2012; Lockyer et al., 2009; Mor et al., 2013; Mor & Mogilevsky, 2013). Learning design authoring tools are often specific, meaning that they support the creation of designs deployable in particular technologies for activities in virtual or physical spaces; see, for instance, QuesTInSitu for the design of learning activities in geo-located physical places (Santos, Pérez-Sanagustín, Hernández-Leo, & Blat, 2011).
The Integrated Learning Design Environment (ILDE) is a community platform where teachers can design learning activities using multiple authoring tools (Hernández-Leo, Asensio-Pérez, Derntl, Prieto, & Chacón, 2014). The design of across-spaces learning situations typically involves the use of diverse authoring tools. Each authoring tool serves to create activities to be performed in a particular space; e.g., a location-based activity outside the classroom and activities in a learning management system (Pérez-Sanagustín et al., 2012). To support an integrated design of these situations, ILDE enables the selection and articulation of multiple authoring tools in what we call “design workflows”. In this paper, we argue that this integrated approach to learning design can also benefit an articulated, meaningful interpretation of learning analytics across-spaces. This calls for an extension of ILDE incorporating learning analytics aligned with learning design.
Alignment of learning design with learning analytics research has been mostly focused on facilitating students’ self-regulation, nurture teachers’ monitoring and eventually lead to pedagogical interventions (Rodríguez-Triana, Martínez-Monés, Asensio-Pérez, & Dimitriadis, 2015; Wise, 2014; Jovanovic et al., 2008). Moreover, there is an emerging encouraging discussion about the role that learning analytics can have to inform learning design (Lockyer, Heathcote, & Dawson, 2013; Pardo, Ellis, & Calvo, 2015). The results offered by learning analytics can provide evidence to evaluate pedagogical plans and to advise their eventual reuse and redesign. The state of the art in this area is still in its early days but there are already preliminary experiences that show the potential and challenges of applying learning analytics to support learning (re)design (Mor, Ferguson, & Wasson, 2015).
Next section describes the integrated approach for learning design supported by ILDE. The following section elaborates the ideas for extending ILDE with learning analytics with an illustrative example based on the flipped classroom.
As aforementioned, learning design tools are varied. They can be oriented to the design of learning activities compliant with particular pedagogical approaches or support diverse stages of the design process (conceptualization, authoring, implementation).
Learning design conceptualization tools support teachers in reflecting about the context in which designs will be applied, e.g., Personas, Factors and Concerns (Mor & Mogilevsky, 2013), or in sketching ideas for the design, e.g., Learning Objectives, Course Features, Course Map, (Cross, S., Galley, R., Brasher, A., Weller, 2012; Mor & Mogilevsky, 2013). Authoring is the step between the conceptualization of the learning design and its implementation with students, in virtual spaces (e.g., Virtual Learning Environments, VLEs) or in physical spaces with (partial or complete) support of digital devices (of different kind, from mobile phones to laptops). Learning design authoring tools enable the production of detailed definitions of learning designs that can be deployed in a specific learning setting. Examples of authoring tools are Web Collage (VillasclarasFernández, Hernández-Leo, Asensio-Pérez, & Dimitriadis, 2013), for the authoring of collaborative learning activities; QuesTInSitu, for the design of location-based activities supported by mobile devices (Santos et al., 2011); or OpenGLM (Derntl, Neumann, & Oberhuemer, 2011), as a more general authoring tool whose designs can be deployed in learning management systems (Prieto et al., 2013).
An integrative approach to articulate learning design tooling can offer a holistic view of the pedagogical intent reflected in several tools used along a learning design process. ILDE enables such integrative approach by integrating multiple existing learning design tools for conceptualization, authoring and implementation in a single environment (Hernández-Leo, Asensio-Pérez, et al., 2014) (see Figure 1). In ILDE, a holistic view of the pedagogical intent is facilitated by means of a so-called learning design “workflow”. Teachers can select which learning design tools, out of the possible options integrated in ILDE, they will be using in the process of creating a learning design. This approach envisages a scenario where teacher-led inquiry and learning analytics results can be aligned and interpreted in the frame of the whole output resulting from learning design workflow.
Across-spaces learning situations typically require the use of multiple tools to support activities in diverse spaces (e.g., mobile learning applications to support activities in the physical space, learning management systems or virtual worlds to support virtual activities). The corresponding learning design authoring tools for each activity can be articulated in an integrated way in ILDE learning design workflows. Learning analytics derived from the diverse tools used to support activities in physical and virtual spaces can be also in turn documented aggregately in this type of integrated environment. The following section illustrates this idea with an example of a learning design for flipped classroom activities.
The example selected to illustrate proposed approach is based on the flipped classroom methodology, as an example of an across-spaces learning situation. In particular, the learning design spans a week and consists of a preparation task that learners are asked to complete online followed by a set of face-to-face activities in plenary, tutorial and lab sessions (Pardo et al., 2015).
A possible workflow to follow in the design of flipped learning activities for a week, from conceptualization to authoring, is shown in Figure 2. This workflow suggests the use of two learning design conceptualization tools to reflect and document the context (Persona Card, Factors and Concerns), two additional conceptualization tools (Learning Objectives and Heuristic Evaluation) to sketch and document the targeted learning and design objectives, the Reauthoring tool (https://bitbucket.org/abelardopardo/reauthoring) to edit the preparation tasks to be completed online with the support of computer systems and additional authoring tools to specify the activities that will be carried out in the classroom.
A particular example of the application of the workflow to design flipped learning activities for a firstyear Computer Systems course at the university level entails a set of material, social and intentional factors depending on the context that are reflected in ILDE using conceptualization tools. Concerns mostly rely on the risks around lack of participation, considering the characteristics of the context (e.g., Personas: in this scenario typically tech savvy but disengaged profile, with good technical skills but plans to complete the course with minimum effort). If students do not participate actively in the preparation activities (those scheduled before the lecture), face-to-face sessions will not be effective. Moreover, if the activities in the plenary session are reduced to the exposition of factual knowledge, students will perceive no value derived from attending the session and will resort to view the recording. These concerns lead to explicit design objectives around encouraging student engagement in the preparation activities, and then schedule face-to-face sessions properly aligned with the objectives.
The learning design conceptualization undertaken sets the basis for the learning design-decision making, to be reflected in the actual authoring of the learning tasks that will be proposed to students. The previous conceptualizations identify as critical the preparation tasks to be done online, in a virtual space. Therefore, the teacher decides that the preparation tasks will contain a set of engaging exercises that will enable students to get familiar with new terms and concepts. The set of exercises consists of interacting with online videos, reading course notes, answering self-assessed formative questions, and providing the solution to a sequence of concept test questions, all supported by a computer system. Interactive actions, beyond passive watching of videos, is considered critical to foster engagement. Learners are asked to complete the preparation tasks virtually before coming to the actual physical classroom. For the plenary session, the teacher plans active learning tasks (short exercises, exchange ideas with neighbors, voting, conducting a discussion, etc.) Tutorial sessions are based in problem-based learning and lab sessions in practical activities that require the use of electronic equipment.
The system used to support the preparation tasks has its corresponding data collecting mechanism for each exercise. The data is processed and provided to teachers, which in turn can feed ILDE to document the impact of this particular task in the context of the whole learning design and contextual characteristics (see how this could be implemented in ILDE in Figure 3). The top left graph shows the number ratio of incorrect answers in a sequence of 12 exercises. The teacher may clearly see how question number 5 has the largest rate, aspect to be considered if a potential redesign if the task is going to be reused, for example, the following academic year. Similarly, the top right graph shows the number percentage of correct, incorrect answers, and requests to view the solutions of two multiple choice questions. This analytics of the activity in the virtual space can be used to quickly detect questions with unusually high number of incorrect responses, or high number of request to see the solution is used to detect more difficult questions. Finally, the bottom of the figure shows three histograms with the number of video events recorded for three videos (from top bar, play, pause, loaded, and finished video). This visualization can be used to estimate the level of difficulty depending on the percentage of pause events.
Those videos with an unusually large number of pause events may suggest a larger intrinsic difficulty of the described material or, considering the contextual aspects documented in the Persona Cards, special problems with the English language used in the videos for certain types of students’ profiles. Similarly, learning analytics from activities in the physical space can also feed ILDE to also provide impact information of the tasks carried out in plenary, tutorial or lab sessions (Pardo et al., 2015). By navigating through the learning designs aggregated in a design workflow, teachers can explore - in an integrative way - the learning analytics of the completed across-spaces situation. Moreover, because conceptualization aspects are also documented in the workflow, teachers can interpret the analytics considering the characteristics of the context.
Learning analytics across spaces and tools is challenging and conveys risks related to multiple and heterogeneous data sources and contextual aspects. This paper argues that risks could be minimize if learning analytics is aligned with learning design using an integrative approach. The work in progress presented in this workshop proposes an integration driven by a learning design workflow. A learning design workflow relates the set of conceptualization (documenting context) and authoring tools (enabling implementation of activities in particular spaces) used to design the across-spaces learning situation. An illustrative example shows that it is possible to link learning design of tasks that occurs in different spaces with leanring analytics that exploit data for tasks distributed across spaces. Focus is on providing insights to educators about what happened in the across-learning situation. A number of challenges remain to make feasible the implementation of the proposed idea and further investigate its implications, including the collection of data in face-to-face classrooms or the synchronization of data collection in learning systems, or the meaningful cross-analysis of heterogeneous data for holistic visualizations.