Co-creation process and challenges in the conceptualization and development of the edCrumble learning design tool Laia Albó [0000-0002-7568-9178] and Davinia Hernández-Leo [0000-0003-0548-7455] ICT Department, Universitat Pompeu Fabra, Barcelona, Spain [laia.albo, davinia.hernandez-leo]@upf.edu Abstract. This paper presents the co-creation process followed during the con- ceptualization, development and evaluation of edCrumble: a learning design (LD) tool which provides an innovative visual representation of the LDs charac- terized by data analytics with the aim of facilitating the planning, visualization, understanding and reuse of complex LDs. Researchers used several partici- pants’ sources and profiles, different methods (including paper and web-based prototyping, questionnaires, interviews, focus groups, role-play games, sharing activities) and workshop types (isolated vs. long-time). Participatory design workshops and activities are described as well as the challenges encountered during the co-design process with the aim of informing other researchers who are thinking of using co-creation. These challenges include the recruitment and motivation of participants, the management of their expectations, the prioritiza- tion of the feedback diversity and a short evaluation of the methods used. Keywords: Co-creation, Learning Design, Authoring tool, edCrumble. 1 Introduction Co-creation refers to any act of collective creativity which can be used at all points along the product development, from the idea generation but also at all key moments of decision throughout the design process [1]. The practices of co-creation in design (co-design or participatory design) date back to the 70s starting with the user-centred design approach. But nowadays, we are moving from simply designing products for users (user-centred) to designing for the future experiences or purposes of people (co- designing) [1]. Therefore, it is necessary to reconsider the role of designers (design developer, facilitator and generator) to achieve user participation in design [2]. Learning Design (LD) aims to support teachers in the process of documenting their teaching practices, making their learning design ideas explicit and sharable [3]. But despite its potentialities regarding teaching and learning innovations, there is a gap on the adoption of LD by the practitioners [4]. Whereas some initiatives of participatory design have been identified in order to include users’ insights on LD solutions [5], more work is needed to explore how the use of co-creation during the conceptualiza- tion and development of specific LD tools may contribute on reducing this gap. 2 ILDE2/edCrumble is a LD tool for teachers of any educational level, which provides an innovative visual representation of the LDs characterized by data analytics with the aim of facilitating the planning, visualization, understanding and reuse of complex LDs [6]. Specifically, the decision-making during the LD process is supported by two types of analytics: resulting from the design of the activities sequenced in a timeline (LD analytics); and aggregated meta-data extracted from several grouped LDs created by multiple teachers within a community (community analytics). In this paper, we present the process followed during the conceptualization, devel- opment and evaluation of edCrumble (https://ilde2.upf.edu/edcrumble/) using partici- patory design workshops, with the aim of reporting our experience of implementing co-creation. Specifically, we describe the activities used in our approach, identifying and discussing the challenges we found in our case study: including the recruitment and motivation of participants, management of their expectations, the prioritization of feedback diversity and a short evaluation of the methods used. 2 Co-creation in edCrumble 2.1 Participants and Sample During the co-creation process several workshops were carried out in different con- texts: (1) two teaching innovation conferences; (2) one research project event; (3) a collaboration with two schools in the frame of a research project; and (4) a learning innovation project in our university. Participants (140, 40% female) had different profiles depending on the workshop –choice based on the opportunity (see Table 1). Table 1. Research contexts descriptions and participants’ profiles. Context Participants Num. ID Dates Description Profile (#female) Workshop May ‘16 Massive Open Online Cours- 24 (9) UCATx conference conference #1 es related Staff & Professors. Workshop June ‘17 Expert researchers on TEL - 15 (5) RESET project project meeting with teaching experience. CoT School #1 High school Teachers. 10 (6) Research Oct.’17- project project with 10 (8) June’18 (Recer- School #2 High school Teachers. schools caixa) Workshop ITWorldEdu Teachers, Researchers and 23 (9) April ‘18 conference #2 conference EdTech related stakeholders. Engineering school - Undergraduate students 32 (10) University March- Universitat Pompeu local project July’18 Professors 26 (9) Fabra Total 140 (56) 3 2.2 Procedure and instrumentation Co-creation was used during the Conceptualization, Development and Evaluation phases of edCrumble. Participatory design workshops were carried out using several research methods and instruments depending on the workshop and its context (Fig.1.). Fig. 1. Co-creation procedure (participatory design workshops’ instruments and methods) dur- ing the Conceptualization, Development and Evaluation phases of edCrumble. (Access online figures of the paper here: https://www.upf.edu/web/tide/edcrumble_pictures) The Conceptualization phase consisted of two workshops with the aim of defining the edCrumble’ main objectives and features (see Fig.1. Conceptualization: conceptual- ization workshops 1 and 2). Both used paper prototyping activities, where participants were working in groups and completed a final individual questionnaire for sharing their reflections with the researchers. 4 The Development phase consisted of several workshops with two school communi- ties, which were part of a research project (see Fig.1. Development). During this phase, participatory design workshops served for advancing on the development of a web-based prototype of edCrumble using participants’ insights and reflections. The same workshops’ structure was followed for each school community despite the con- text was different: in the first school the workshops were about Problem Based Learn- ing (PBL) and in the second school, they were about Flipped Classroom (FC). During this phase, participants worked with different versions of the online prototype and participated on different activities which included focus groups, sharing and discuss- ing activities, questionnaires and interviews. The Evaluation phase consisted of several evaluation workshops and an evaluation study (see Fig.1. Evaluation). In the workshops, participants were involved in a role- play game whereas they were using edCrumble with the aim of evaluating its usability and utility. Apart from the design artefacts resulting from the activities, researchers used a questionnaire for collecting participants’ feedback. In the evaluation study, researchers worked in parallel with students and professors for evaluating edCrumble as well as collecting their insights about blended learning and course design. The study included time for working with edCrumble, questionnaires and interviews. 2.3 Co-creation activities during the conceptualization phase Conceptualization workshop 1. The aim of the workshop activity was to challenge each participant to design a blended-learning course using Massive Open Online Courses (MOOCs). The workshop lasted two hours and the 24 participants were di- vided into seven workgroups. The workflow of the activity was a five-step process described on [7] which used several paper materials: a LD template, three framework sheets and printed LD examples. Specifically, one step of the LD design template was asking participants to represent their blended LDs using a first timeline model (Fig.2. left). This visual representation model was evaluated based on the participants sheets (with the participants insights collected using a questionnaire) contributing in the initial conceptualization of the main element of the edCrumble editor: the timeline. Conceptualization workshop 2. The main objective of the workshop was explor- ing with the participants how visually represent blended LDs and how these visualiza- tions can facilitate others’ LDs understanding. Researchers prepared two LDs cases descriptions sheets (both were using MOOCs mixed with face-to-face courses) ex- tracted from the literature. Moreover, the main material used was a paper LD template with a new visual model of the timeline with resources’ layers designed based on the results from the workshop 1 and the literature. Below the timeline, in the LD template sheet, there was an empty space for placing paper activity cards (which were drawn and filled in with stickers by the participants following a provided legend) (Fig.2. middle and right). The 15 participants were working in groups of two/three people using a LD template per workgroup. The two printed cases descriptions were divided equally between the existing groups, in such a way that half of the groups worked with one of the examples and the other half with the other one. Once each group had a case description sheet, they followed the following steps: 5 1. Read the case provided and represent it using the LD template sheet –placing the activities and the resources described on the case using the timeline and filling in the activity cards (and place them on the sheet) (Fig.2. middle). 2. After completing the LD template with their case (Fig.2. right), they had to ex- change the completed LDs templates between workgroups and interpret the LD template produced by another group (only looking on the visual representation and without knowing the LD case description of the template received as the ex- change occurred between groups that had different LD cases). 3. Finally, each group could check if they had understood well the LD template received by looking on the corresponding case description. Last, participants were asked to complete a questionnaire providing their insights about the process. Results of this workshop pointed out the main strengths and weaknesses of the visual representation proposed and were useful for discussing whether the timeline and lay- ers provided by the template were valid for designing hybrid courses. The outputs of the workshop helped to improve the visual representation and have a more solid base to start the development of the online version. Fig. 2. Paper template of the workshop 1 with three templates filled in by the participants (left); Participants of the workshop 2 completing a paper LD template (middle); Scanned paper LD template resulting from the workshop 2 -with the timeline and the activity cards (right). 2.4 Co-creation activities during the development phase Researchers prepared a first online prototype based on the results obtained during the conceptualizing phase (a web-based tool which provides an editor to work with the evolved timeline model on an interactive way). The aim of the participatory design workshops of this phase were prototyping and assessing the preliminary versions of the authoring tool with the participants of two school communities (Fig.3.). The fol- lowing steps were carried out in each community. Development workshop 1. In which teachers had to design a LD using the online prototype of edCrumble, with the help of the researchers (participants were asked to come to the workshop with a concrete LD idea). It was a 2h workshop with the fol- lowing steps: (1) Introduction to edCrumble; (2) Work with edCrumble designing a LD for being implemented within their classrooms (a PBL or a FC design); (3) Focus group where researchers asked questions about the experience that participants had with the use of the tool, discussing their strengths and weaknesses. (4) Last, partici- pants were asked to answer a research questionnaire individually. Development workshop 1.2. In the case of the School #2, they had another 2h workshop because they needed more time for designing the interventions using the 6 tool and be prepared for implementing the LDs in their classrooms. In this case, re- searchers took observation notes of teachers’ using the tool for usability improve- ments. Class implementations. Teachers implemented their LDs in class. During this step, which took between 4 and 9 weeks, researchers were available online for solving teachers’ doubts regarding the use of technology selected for using in their class. Development workshop 2. In this workshop, which took 1-2h depending on the school, teachers followed three steps: (1) Working with edCrumble for documenting the LDs implemented at class, adding the design changes suffered by the real imple- mentations; (2) Sharing their implementation experiences and a joint reflection about the possible redesign of their original LDs considering the lessons learned; (3) Last, participants were asked to answer a research questionnaire individually. Interviews. We carried out seven semi-structured face-to-face interviews (three teachers from School#1 and four from School#2 –due time and resources constraints we could not interview all 24) of about 45 minutes each. The interviews consisted of a series of open-ended questions (see details in [4]) that invited participants to share their perspectives regarding (1) how they used to design and document their educa- tional practices before knowing our tool and (2) how was the design process they followed during the workshops using edCrumble. Results from this co-creation phase gave rise to a series of design principles (col- lected in [4]) and facilitated the development of the tool through different prototype versions (see Fig.1. Development phase). Workshops 1 and 1.2 reported about the LD process using the tool. Whereas workshop 2 allowed to study how was the use of edCrumble for redesigning teachers’ own LDs and for understanding others’ LDs. Fig. 3. Development workshops’ participants working with the online prototype of edCrumble. 2.5 Co-creation activities during the evaluation phase Evaluation workshops. Each workshop consisted of a role-play game where partici- pants were placed in groups of 2-4 people. Each group of participants represented an imaginary school and each participant of each group represented a teacher of a topic (simulating different educational communities). The role-play game had two main parts (individual and in group) which each of them had three steps. The individual activity (at “imaginary” teacher-role level) consisted of: (1) Design of a short teaching unit with the ILDE2/edCrumble online version –a printed LD was provided by the researchers for each teacher role (see Fig. 4. left); (2) Analyse the data resulting from the elaborated LD; and (3) Sharing the design created within the ILDE2/edCrumble community. Whereas the group activity (at “imaginary” school- 7 role level) implied: (1) Grouping several designs to generate community analytics; (2) Solving an educational challenge; and (3) Discussing results with all participants. At the end of the workshop, researchers asked participants to fill in a research ques- tionnaire for evaluating edCrumble. Last, participants were asked to discuss in groups about the educational problems which they think edCrumble can solve as well as those not solved by the tool but can or should be addressed in future versions. Fig. 4. Printed LDs for each teacher role during the evaluation workshops (left); Participants of the evaluation study working with edCrumble online version (middle and right). Evaluation Study. The evaluation study had the following phases (see Fig.1.): 1. PHASE S1 (Registration): students registered voluntarily for the study indicating 3-5 subjects of their bachelor’s degree which they would like to report. 2. PHASE S2 (edCrumble design work): researchers assigned the subjects to the stu- dents registered depending on their preferences. The workshop was about 2h: a. 10 minutes: students signed the consent form and a document with information about their bank account (they received 15€ as complementary compensation). b. 15 minutes: researchers explained the aim and procedure of the study and did a short demonstration of how to document a course plan in edCrumble. c. 80 minutes: students worked with edCrumble in their computers to introduce the course plan on the system (Fig. 4. middle and right). Students were asked in ad- vance to come sufficiently prepared to be able to report the course’s LD. d. 15 minutes: students filled out the first research questionnaire which had two main objectives: (1) ask students about their opinion about blended learning and course design; and (2) evaluate edCrumble. 3. PHASE S3 (evaluation and design readjustments): based on the subjects intro- duced, researchers prepared a second questionnaire with the aim of crosschecking the different designs introduced on the system, so each subject could be validated by other students. After one week, students received the second research question- naire by email, and based on their responses, researchers readjusted the LDs in the edCrumble system (validating the LDs reported). 4. PHASE P1 (blended learning survey): professors answered a questionnaire about blended learning and course design. 5. PHASE P2 (design interviews): based on the subjects introduced by the students and the responses of the professors’ questionnaire (phase P1), researchers made a list of possible professors of interest on being interviewed. Interviews were carried out with the aim of discussing the resulting visual representation of the LD ob- tained with the edCrumble and if they would introduce some changes based on the information received from the study (using the tool). 8 3 Discussion and lessons learned 3.1 Participants’ recruitment process and motivation Being recruiting participants a challenging task, we used several sources to recruit them: two local teaching innovation conferences and the frame of three research pro- jects. The project with the schools was the unique case where we had the opportunity of having the same group of participants during more than one workshop. In the other cases, participants only attended one workshop, being difficult for them to appreciate the complete picture of the whole co-design process and feel that they were part of something beyond the isolated activity in which they participated. Moreover, in each of these workshops, we needed to save workshops’ time for explaining the research context and ask them collaboration (permission for collecting their data). Whereas in the case of the project with the schools, we only needed to do this task at the begin- ning of their first workshop (saving time in the rest of the workshops). Nevertheless, working with the same teachers during a long-time period (nine months) was also challenging in terms of keeping their motivation with the activities. Specially, because the workshops were during the academic course, after classes. Due to their restricted availability, we adapted ourselves to their schedule when negotiating the dates and times (sometimes shortening the workshops’ time or avoiding weeks where they had more work) despite they were agreed collaboration partners in the framing of the re- search project (with a complementary compensation to the schools). Both strategies (isolated and long-time period workshops) had advantages and in- conveniences, but we believe that this combination has been the key to be able to carry out the co-creation process during all phases. Since we have been able to sched- ule the workshops on the fly (higher degree of flexibility) bearing in mind the needs of our research along its whole process (it would have been difficult to elaborate a completed plan from the beginning). Furthermore, having different participants’ sources have allowed us to work with different stakeholders, including a group of experts in TEL during the conceptualization phase which added value to our process. 3.2 Managing workshops’ time and participants’ expectations Due to our context, the workshops had to contribute something to the participants beyond participating in a co-creation process –in almost all workshops we did, the co- creation was not the unique goal: e.g., how to design blended learning with MOOCs or with data analytics (conference workshops), learning PBL and FC methodologies (schools’ project) etc. This was good for attracting participants, but it was challenging in terms of managing the limited time and expectations. While we were teaching something to the participants, we had to collect data and fitting the corresponding co- creation activity (using edCrumble somehow). The hardest point was managing par- ticipants’ expectations, finding a balance between their collaboration in our research and our contribution to them in terms of learning something in the activities (especial- ly because time was always very limited:1-2h). E.g. during the development phase, it was a bit demanding for participants learning a new software and creating a LD. For 9 this reason, in the case of the evaluation workshops we used a role-play game (LDs were already prepared). Therefore, they felt more relax, since they could enjoy the tool without feeling pressured to have their own LD ideas in parallel. 3.3 Potential and challenges of the co-creation methods used Table 2 shows a summary of the pros and cons of the methods selected with the aim of enriching our lessons learned and serve others thinking on similar scenarios. Table 2. Evaluation of the methods used during the edCrumble co-creation process. Method Pros Cons Paper Reducing development time-effort. Time/cost consuming (preparation of the mate- proto- High flexibility in the expression of rials, analysis of the resulting paper artefacts). type ideas by the participants. Participants engagement depending on their profile (some people are reluctant to collaborate in activities that require crafts). Web- High satisfaction of the participants Need of managing frustrations during the early based at the end of the process in feeling phases (early-prototype errors and usability low proto- that they have collaborated in creat- developed, sense of losing time…). type ing something real. Developing time and cost consuming. Possibility of collecting system’ data for the analysis (e.g. log files). Ques- Valuable individual time for partic- Finding a balance between the time needed to tionnaire ipants reflection and expression of carry it out (workshop time consuming) and the their ideas and opinions. number of items to get the necessary data. Focus High flexibility in the expression of Qualitative analysis with high time consuming. group ideas by the participants. High quality data Sharing Participants can discuss their own Depending on the num. of participants, high & dis- cases and exchange experiences amount of time is needed. cussing (learning from others). Need of moderate the discussion when short time available (keeping the focus, ask relevant questions, select only representative cases for sharing…). Qualitative analysis with high time consuming. Role- Reducing participants’ required Participants not experiencing their own cases. play effort on preparing their cases game (saving workshop time). Inter- High quality data Participants’ limited availability (in our isolated views High flexibility in the expression of workshops: difficult to have the opportunity to ideas by the participants. keep in contact with participants and ask them collaboration; in our project workshops: teach- ers’ time limitations). Qualitative analysis with high time consuming. 10 3.4 Prioritization of feedback diversity Despite the feedback collected was very diverse during all process, the most challeng- ing phase regarding its prioritization was during the development of the online tool. We had to be able to analyse the feedback after each workshop and prioritize it to prepare a new version for the next workshop. The prioritization process was always a balance between considering the feasible points to be developed in the time we had until the next workshop, and that a direct proposal from the participants would always be included to motivate them to continue in the process (since during the use of the first versions it was quite frustrating for them to use a system that was not yet very usable). Having new versions of the prototype in each workshop allowed us to ad- vance considering participants’ insights and engaging them in the co-creation process. 4 Conclusions During the co-creation process of edCrumble, researchers used several participants’ sources, different methods and participatory design workshop types (isolated vs. long- time). Co-creation had a positive impact in the design and decision-making process of our research, but it also presented some challenges. We hope that this experience and the challenges documented can help other researchers who are thinking of using co- creation in the design of teacher tools. Acknowledgements. Authors want to thank all the participants who collaborated in the study. This work has been partially funded by RecerCaixa (CoT project) and the Spanish Ministry of Economy and Competitiveness under MDM-2015-0502, TIN2014-53199-C3-3-R, TIN2017-85179-C3-3-R. References 1. Sanders, E. B. N. and Stappers, P. J.: Co-creation and the new landscapes of design. CoDesign 4(1), 5–18 (2008). 2. Lee, Y.: Design participation tactics: the challenges and new roles for designers in the co- design process. CoDesign 4(1), 31–50 (2008). 3. Conole, G.: Designing for learning in an open world, vol.4. Springer Science & Business Media (2012). 4. Albó, L. and Hernández-leo, D.: Identifying design principles for learning design tools: the case of edCrumble. In Proccedings of the 13th European Conference on Technology Enhanced Learning, EC-TEL 2018. (Accepted). 5. Mor, Y., Warburton, S., and Winters, N.: Participatory pattern workshops: A methodology for open learning design inquiry. Res. Learn. Technol 20(SUPPL), 163–175 (2012). 6. Albó, L. and Hernández-leo, D.: edCrumble: designing for learning with data analytics. In Proccedings of the 13th EC-TEL 2018. (Accepted). 7. Albó, L. and Hernández-leo, D.: Blended learning with MOOCs : towards supporting the learning design process. In The Online, Open and Flexible Higher Education Conference 2016, pp. 578–588 (2016).