With the increased use of technology in education, educational data science has become more accessible, providing tools and techniques for making sense of educational data [1]-[3]. On the other hand, blended educational practices are complex and combine face to face -f2f- instruction with online activities supported by technology. Thus, nowadays, educators and researchers could benefit from data extracted from blended educational practices for analysing and reflecting about the characteristics of the educational settings they have designed [3]. The data collected can serve different purposes, depending on their nature. Whereas data about students' interactions can be used to understand their learning processes and experiences (learning analytics), metrics of design decisions and related aspects characterizing learning designs (design analytics) can provide awareness and reflection on decisions made during the learning design process as well as inform future design decisions [1][4]. Moreover, the metrics and patterns of design activity within a community of teachers and related stakeholders (community analytics) can provide awareness and reflection about individual and collective design activities and trigger orientation and inspiration about how to improve the design practices [1], [5]. In this paper, we explore the opinions of teachers regarding the use of design analytics, probably the least explored type of data in educational technology together with community analytics. Specifically, we analyse the value that design analytics can offer in authoring experiences using the edCrumble learning design tool [6]. Three workshops were carried out in different contexts to give participants the opportunity of interacting with the tool and evaluating the design analytics provided. The first workshop took place on a teaching innovation conference (April 2018) and participants were 23 teachers, researchers and educational technology-related stakeholders (9 female). Whereas the second and third workshops were placed within a collaboration with two schools in the frame of a research project (May-June 2018). In this case, participants were 14 high school teachers, half of them from each school community. Each workshop (of 90 minutes) consisted of a role-playing game where participants were organized in groups of 2-4 people. Each group of participants represented an imaginary school and each participant of each group represented a teacher in charge of a subject matter (simulating different educational communities). The role-playing game had two main parts (individual and in groups). Each part had
three steps. The individual activity (at “imaginary” teacher-role level) consisted of the
following tasks: (
The education challenge proposed at the group level was asking teachers to use community analytics for balancing the out-of-class workload between the different designs they created which were part of a whole curriculum to be worked with the same cohort of students (within the simulated school community). The objective was first, to analyse how many hours of homework they had given to the students in total, counting all the designs generated by the school (the community analytics allowed the participants to see the aggregated out-of-class hours of all the designs automatically). And second, they were invited to reduce the total hours of work outside the classroom (if there were many) to a certain number that they would consider appropriate, debating what strategy to follow.
At the end of the workshop in one of the schools (n=7), researchers asked participants to fill in a research questionnaire individually (the questionnaire could not be used in the other workshops due to lack of time). The questionnaire (which was anonymous) had three main questions for evaluating the design analytics of edCrumble and, for each of them, participants indicated their level of agreement (using a five-point Likert scale). They were invited to provide an optional comment or open response. Moreover, participants of both schools (n=14) were asked to evaluate design analytics (individually) based on several factors (following a five-point Likert scale of agreement). Finally, the design artefacts resulting from all workshops were analysed, checking whether participants were able to generate the community analytics correctly and whether they successfully solved the educational challenge proposed.
After analysing the results, we can state that design analytics have the potential of supporting teachers during the design process facilitating data-based decision making. The results have shown positive attitudes and opinions of teachers regarding the use of different types of design analytics as they think analytics can help them to improve their designs. Moreover, teachers valued that design analytics could facilitate the reflection during the design process as well as provide them support for achieving a design continuity between the in-class and out-of-class activities. Lastly, aggregated design analytics from multiple designs across educators in a community (community analytics) have also been valued offering design support and awareness at the school community level, with the potential of facilitating the design coordination among teachers of the same students’ cohort.
Acknowledgements. The authors would like to thank all the participants who participated in the workshops as well as Alex Young Pedersen for revising the manuscript. This work has been partially funded “la Caixa Foundation” (CoT project, 100010434) and FEDER, the National Research Agency of the Spanish Ministry of