This paper presents the co-creation process followed during the conceptualization, development and evaluation of edCrumble: a learning design (LD) tool which provides an innovative visual representation of the LDs characterized by data analytics with the aim of facilitating the planning, visualization, understanding and reuse of complex LDs. Researchers used several participants' sources and profiles, different methods (including paper and web-based prototyping, questionnaires, interviews, focus groups, role-play games, sharing activities) and workshop types (isolated vs. long-time). Participatory design workshops and activities are described as well as the challenges encountered during the co-design process with the aim of informing other researchers who are thinking of using co-creation. These challenges include the recruitment and motivation of participants, the management of their expectations, the prioritization of the feedback diversity and a short evaluation of the methods used.
Co-creation refers to any act of collective creativity which can be used at all points along the product development, from the idea generation but also at all key moments of decision throughout the design process [1]. The practices of co-creation in design (co-design or participatory design) date back to the 70s starting with the user-centred design approach. But nowadays, we are moving from simply designing products for users (user-centred) to designing for the future experiences or purposes of people (codesigning) [1]. Therefore, it is necessary to reconsider the role of designers (design developer, facilitator and generator) to achieve user participation in design [2].
Learning Design (LD) aims to support teachers in the process of documenting their teaching practices, making their learning design ideas explicit and sharable [3]. But despite its potentialities regarding teaching and learning innovations, there is a gap on the adoption of LD by the practitioners [4]. Whereas some initiatives of participatory design have been identified in order to include users’ insights on LD solutions [5], more work is needed to explore how the use of co-creation during the conceptualization and development of specific LD tools may contribute on reducing this gap. ILDE2/edCrumble is a LD tool for teachers of any educational level, which provides an innovative visual representation of the LDs characterized by data analytics with the aim of facilitating the planning, visualization, understanding and reuse of complex LDs [6]. Specifically, the decision-making during the LD process is supported by two types of analytics: resulting from the design of the activities sequenced in a timeline (LD analytics); and aggregated meta-data extracted from several grouped LDs created by multiple teachers within a community (community analytics).
In this paper, we present the process followed during the conceptualization, development and evaluation of edCrumble (https://ilde2.upf.edu/edcrumble/) using participatory design workshops, with the aim of reporting our experience of implementing co-creation. Specifically, we describe the activities used in our approach, identifying and discussing the challenges we found in our case study: including the recruitment and motivation of participants, management of their expectations, the prioritization of feedback diversity and a short evaluation of the methods used.
During the co-creation process several workshops were carried out in different
contexts: (
Co-creation was used during the Conceptualization, Development and Evaluation phases of edCrumble. Participatory design workshops were carried out using several research methods and instruments depending on the workshop and its context (Fig.1.). Fig. 1. Co-creation procedure (participatory design workshops’ instruments and methods) during the Conceptualization, Development and Evaluation phases of edCrumble. (Access online figures of the paper here: https://www.upf.edu/web/tide/edcrumble_pictures) The Conceptualization phase consisted of two workshops with the aim of defining the edCrumble’ main objectives and features (see Fig.1. Conceptualization: conceptualization workshops 1 and 2). Both used paper prototyping activities, where participants were working in groups and completed a final individual questionnaire for sharing their reflections with the researchers. The Development phase consisted of several workshops with two school communities, which were part of a research project (see Fig.1. Development). During this phase, participatory design workshops served for advancing on the development of a web-based prototype of edCrumble using participants’ insights and reflections. The same workshops’ structure was followed for each school community despite the context was different: in the first school the workshops were about Problem Based Learning (PBL) and in the second school, they were about Flipped Classroom (FC). During this phase, participants worked with different versions of the online prototype and participated on different activities which included focus groups, sharing and discussing activities, questionnaires and interviews.
The Evaluation phase consisted of several evaluation workshops and an evaluation study (see Fig.1. Evaluation). In the workshops, participants were involved in a roleplay game whereas they were using edCrumble with the aim of evaluating its usability and utility. Apart from the design artefacts resulting from the activities, researchers used a questionnaire for collecting participants’ feedback. In the evaluation study, researchers worked in parallel with students and professors for evaluating edCrumble as well as collecting their insights about blended learning and course design. The study included time for working with edCrumble, questionnaires and interviews. 2.3
Conceptualization workshop 1. The aim of the workshop activity was to challenge each participant to design a blended-learning course using Massive Open Online Courses (MOOCs). The workshop lasted two hours and the 24 participants were divided into seven workgroups. The workflow of the activity was a five-step process described on [7] which used several paper materials: a LD template, three framework sheets and printed LD examples. Specifically, one step of the LD design template was asking participants to represent their blended LDs using a first timeline model (Fig.2. left). This visual representation model was evaluated based on the participants sheets (with the participants insights collected using a questionnaire) contributing in the initial conceptualization of the main element of the edCrumble editor: the timeline.
Conceptualization workshop 2. The main objective of the workshop was exploring with the participants how visually represent blended LDs and how these visualizations can facilitate others’ LDs understanding. Researchers prepared two LDs cases descriptions sheets (both were using MOOCs mixed with face-to-face courses) extracted from the literature. Moreover, the main material used was a paper LD template with a new visual model of the timeline with resources’ layers designed based on the results from the workshop 1 and the literature. Below the timeline, in the LD template sheet, there was an empty space for placing paper activity cards (which were drawn and filled in with stickers by the participants following a provided legend) (Fig.2. middle and right). The 15 participants were working in groups of two/three people using a LD template per workgroup. The two printed cases descriptions were divided equally between the existing groups, in such a way that half of the groups worked with one of the examples and the other half with the other one. Once each group had a case description sheet, they followed the following steps: 1. Read the case provided and represent it using the LD template sheet –placing the activities and the resources described on the case using the timeline and filling in the activity cards (and place them on the sheet) (Fig.2. middle). 2. After completing the LD template with their case (Fig.2. right), they had to exchange the completed LDs templates between workgroups and interpret the LD template produced by another group (only looking on the visual representation and without knowing the LD case description of the template received as the exchange occurred between groups that had different LD cases). 3. Finally, each group could check if they had understood well the LD template received by looking on the corresponding case description. Last, participants were asked to complete a questionnaire providing their insights about the process. Results of this workshop pointed out the main strengths and weaknesses of the visual representation proposed and were useful for discussing whether the timeline and layers provided by the template were valid for designing hybrid courses. The outputs of the workshop helped to improve the visual representation and have a more solid base to start the development of the online version.
Fig. 2. Paper template of the workshop 1 with three templates filled in by the participants (left); Participants of the workshop 2 completing a paper LD template (middle); Scanned paper LD template resulting from the workshop 2 -with the timeline and the activity cards (right). 2.4
Researchers prepared a first online prototype based on the results obtained during the conceptualizing phase (a web-based tool which provides an editor to work with the evolved timeline model on an interactive way). The aim of the participatory design workshops of this phase were prototyping and assessing the preliminary versions of the authoring tool with the participants of two school communities (Fig.3.). The following steps were carried out in each community.
Development workshop 1. In which teachers had to design a LD using the online
prototype of edCrumble, with the help of the researchers (participants were asked to
come to the workshop with a concrete LD idea). It was a 2h workshop with the
following steps: (
Development workshop 1.2. In the case of the School #2, they had another 2h workshop because they needed more time for designing the interventions using the tool and be prepared for implementing the LDs in their classrooms. In this case, researchers took observation notes of teachers’ using the tool for usability improvements.
Class implementations. Teachers implemented their LDs in class. During this step, which took between 4 and 9 weeks, researchers were available online for solving teachers’ doubts regarding the use of technology selected for using in their class.
Development workshop 2. In this workshop, which took 1-2h depending on the
school, teachers followed three steps: (
Interviews. We carried out seven semi-structured face-to-face interviews (three
teachers from School#1 and four from School#2 –due time and resources constraints
we could not interview all 24) of about 45 minutes each. The interviews consisted of a
series of open-ended questions (see details in [4]) that invited participants to share
their perspectives regarding (
Results from this co-creation phase gave rise to a series of design principles (collected in [4]) and facilitated the development of the tool through different prototype versions (see Fig.1. Development phase). Workshops 1 and 1.2 reported about the LD process using the tool. Whereas workshop 2 allowed to study how was the use of edCrumble for redesigning teachers’ own LDs and for understanding others’ LDs.
Evaluation workshops. Each workshop consisted of a role-play game where participants were placed in groups of 2-4 people. Each group of participants represented an imaginary school and each participant of each group represented a teacher of a topic (simulating different educational communities). The role-play game had two main parts (individual and in group) which each of them had three steps.
The individual activity (at “imaginary” teacher-role level) consisted of: (
At the end of the workshop, researchers asked participants to fill in a research
questionnaire for evaluating edCrumble. Last, participants were asked to discuss in groups
about the educational problems which they think edCrumble can solve as well as
those not solved by the tool but can or should be addressed in future versions.
Fig. 4. Printed LDs for each teacher role during the evaluation workshops (left); Participants of
the evaluation study working with edCrumble online version (middle and right).
Evaluation Study. The evaluation study had the following phases (see Fig.1.):
1. PHASE S1 (Registration): students registered voluntarily for the study indicating
3-5 subjects of their bachelor’s degree which they would like to report.
2. PHASE S2 (edCrumble design work): researchers assigned the subjects to the
students registered depending on their preferences. The workshop was about 2h:
a. 10 minutes: students signed the consent form and a document with information
about their bank account (they received 15€ as complementary compensation).
b. 15 minutes: researchers explained the aim and procedure of the study and did a
short demonstration of how to document a course plan in edCrumble.
c. 80 minutes: students worked with edCrumble in their computers to introduce the
course plan on the system (Fig. 4. middle and right). Students were asked in
advance to come sufficiently prepared to be able to report the course’s LD.
d. 15 minutes: students filled out the first research questionnaire which had two
main objectives: (
Being recruiting participants a challenging task, we used several sources to recruit them: two local teaching innovation conferences and the frame of three research projects. The project with the schools was the unique case where we had the opportunity of having the same group of participants during more than one workshop. In the other cases, participants only attended one workshop, being difficult for them to appreciate the complete picture of the whole co-design process and feel that they were part of something beyond the isolated activity in which they participated. Moreover, in each of these workshops, we needed to save workshops’ time for explaining the research context and ask them collaboration (permission for collecting their data). Whereas in the case of the project with the schools, we only needed to do this task at the beginning of their first workshop (saving time in the rest of the workshops). Nevertheless, working with the same teachers during a long-time period (nine months) was also challenging in terms of keeping their motivation with the activities. Specially, because the workshops were during the academic course, after classes. Due to their restricted availability, we adapted ourselves to their schedule when negotiating the dates and times (sometimes shortening the workshops’ time or avoiding weeks where they had more work) despite they were agreed collaboration partners in the framing of the research project (with a complementary compensation to the schools).
Both strategies (isolated and long-time period workshops) had advantages and inconveniences, but we believe that this combination has been the key to be able to carry out the co-creation process during all phases. Since we have been able to schedule the workshops on the fly (higher degree of flexibility) bearing in mind the needs of our research along its whole process (it would have been difficult to elaborate a completed plan from the beginning). Furthermore, having different participants’ sources have allowed us to work with different stakeholders, including a group of experts in TEL during the conceptualization phase which added value to our process. 3.2
Due to our context, the workshops had to contribute something to the participants beyond participating in a co-creation process –in almost all workshops we did, the cocreation was not the unique goal: e.g., how to design blended learning with MOOCs or with data analytics (conference workshops), learning PBL and FC methodologies (schools’ project) etc. This was good for attracting participants, but it was challenging in terms of managing the limited time and expectations. While we were teaching something to the participants, we had to collect data and fitting the corresponding cocreation activity (using edCrumble somehow). The hardest point was managing participants’ expectations, finding a balance between their collaboration in our research and our contribution to them in terms of learning something in the activities (especially because time was always very limited:1-2h). E.g. during the development phase, it was a bit demanding for participants learning a new software and creating a LD. For this reason, in the case of the evaluation workshops we used a role-play game (LDs were already prepared). Therefore, they felt more relax, since they could enjoy the tool without feeling pressured to have their own LD ideas in parallel. 3.3
Method Pros Cons Paper Reducing development time-effort. Time/cost consuming (preparation of the mateproto- High flexibility in the expression of rials, analysis of the resulting paper artefacts). type ideas by the participants. Participants engagement depending on their profile (some people are reluctant to collaborate in activities that require crafts).
Web- High satisfaction of the participants Need of managing frustrations during the early based at the end of the process in feeling phases (early-prototype errors and usability low proto- that they have collaborated in creat- developed, sense of losing time…). type ing something real. Developing time and cost consuming.
Possibility of collecting system’ data for the analysis (e.g. log files).
Ques- Valuable individual time for partic- Finding a balance between the time needed to tionnaire ipants reflection and expression of carry it out (workshop time consuming) and the their ideas and opinions. number of items to get the necessary data. Focus High flexibility in the expression of Qualitative analysis with high time consuming. group ideas by the participants.
High quality data Sharing Participants can discuss their own & dis- cases and exchange experiences cussing (learning from others).
Depending on the num. of participants, high amount of time is needed.
Need of moderate the discussion when short time available (keeping the focus, ask relevant questions, select only representative cases for sharing…).
Qualitative analysis with high time consuming.
Participants not experiencing their own cases.
Roleplay game Interviews
Reducing participants’ required effort on preparing their cases (saving workshop time).
High quality data Participants’ limited availability (in our isolated High flexibility in the expression of workshops: difficult to have the opportunity to ideas by the participants. keep in contact with participants and ask them collaboration; in our project workshops: teachers’ time limitations).
Qualitative analysis with high time consuming.
Despite the feedback collected was very diverse during all process, the most challenging phase regarding its prioritization was during the development of the online tool. We had to be able to analyse the feedback after each workshop and prioritize it to prepare a new version for the next workshop. The prioritization process was always a balance between considering the feasible points to be developed in the time we had until the next workshop, and that a direct proposal from the participants would always be included to motivate them to continue in the process (since during the use of the first versions it was quite frustrating for them to use a system that was not yet very usable). Having new versions of the prototype in each workshop allowed us to advance considering participants’ insights and engaging them in the co-creation process. 4
During the co-creation process of edCrumble, researchers used several participants’ sources, different methods and participatory design workshop types (isolated vs. longtime). Co-creation had a positive impact in the design and decision-making process of our research, but it also presented some challenges. We hope that this experience and the challenges documented can help other researchers who are thinking of using cocreation in the design of teacher tools.
Acknowledgements. Authors want to thank all the participants who collaborated in the study. This work has been partially funded by RecerCaixa (CoT project) and the Spanish Ministry of Economy and Competitiveness under MDM-2015-0502, TIN2014-53199-C3-3-R, TIN2017-85179-C3-3-R.