=Paper=
{{Paper
|id=Vol-2190/CC-TEL_2018_paper_8
|storemode=property
|title=Co-Creation Process and Challenges in the Conceptualization and Development of the edCrumble Learning Design Tool
|pdfUrl=https://ceur-ws.org/Vol-2190/CC-TEL_2018_paper_8.pdf
|volume=Vol-2190
|authors=Laia Albó,Davinia Hernández-Leo
|dblpUrl=https://dblp.org/rec/conf/ectel/AlboH18b
}}
==Co-Creation Process and Challenges in the Conceptualization and Development of the edCrumble Learning Design Tool==
https://ceur-ws.org/Vol-2190/CC-TEL_2018_paper_8.pdf
Co-creation process and challenges in the
conceptualization and development of the edCrumble
learning design tool
Laia Albó [0000-0002-7568-9178] and Davinia Hernández-Leo [0000-0003-0548-7455]
ICT Department, Universitat Pompeu Fabra, Barcelona, Spain
[laia.albo, davinia.hernandez-leo]@upf.edu
Abstract. This paper presents the co-creation process followed during the con-
ceptualization, development and evaluation of edCrumble: a learning design
(LD) tool which provides an innovative visual representation of the LDs charac-
terized by data analytics with the aim of facilitating the planning, visualization,
understanding and reuse of complex LDs. Researchers used several partici-
pants’ sources and profiles, different methods (including paper and web-based
prototyping, questionnaires, interviews, focus groups, role-play games, sharing
activities) and workshop types (isolated vs. long-time). Participatory design
workshops and activities are described as well as the challenges encountered
during the co-design process with the aim of informing other researchers who
are thinking of using co-creation. These challenges include the recruitment and
motivation of participants, the management of their expectations, the prioritiza-
tion of the feedback diversity and a short evaluation of the methods used.
Keywords: Co-creation, Learning Design, Authoring tool, edCrumble.
1 Introduction
Co-creation refers to any act of collective creativity which can be used at all points
along the product development, from the idea generation but also at all key moments
of decision throughout the design process [1]. The practices of co-creation in design
(co-design or participatory design) date back to the 70s starting with the user-centred
design approach. But nowadays, we are moving from simply designing products for
users (user-centred) to designing for the future experiences or purposes of people (co-
designing) [1]. Therefore, it is necessary to reconsider the role of designers (design
developer, facilitator and generator) to achieve user participation in design [2].
Learning Design (LD) aims to support teachers in the process of documenting their
teaching practices, making their learning design ideas explicit and sharable [3]. But
despite its potentialities regarding teaching and learning innovations, there is a gap on
the adoption of LD by the practitioners [4]. Whereas some initiatives of participatory
design have been identified in order to include users’ insights on LD solutions [5],
more work is needed to explore how the use of co-creation during the conceptualiza-
tion and development of specific LD tools may contribute on reducing this gap.
2
ILDE2/edCrumble is a LD tool for teachers of any educational level, which provides
an innovative visual representation of the LDs characterized by data analytics with the
aim of facilitating the planning, visualization, understanding and reuse of complex
LDs [6]. Specifically, the decision-making during the LD process is supported by two
types of analytics: resulting from the design of the activities sequenced in a timeline
(LD analytics); and aggregated meta-data extracted from several grouped LDs created
by multiple teachers within a community (community analytics).
In this paper, we present the process followed during the conceptualization, devel-
opment and evaluation of edCrumble (https://ilde2.upf.edu/edcrumble/) using partici-
patory design workshops, with the aim of reporting our experience of implementing
co-creation. Specifically, we describe the activities used in our approach, identifying
and discussing the challenges we found in our case study: including the recruitment
and motivation of participants, management of their expectations, the prioritization of
feedback diversity and a short evaluation of the methods used.
2 Co-creation in edCrumble
2.1 Participants and Sample
During the co-creation process several workshops were carried out in different con-
texts: (1) two teaching innovation conferences; (2) one research project event; (3) a
collaboration with two schools in the frame of a research project; and (4) a learning
innovation project in our university. Participants (140, 40% female) had different
profiles depending on the workshop –choice based on the opportunity (see Table 1).
Table 1. Research contexts descriptions and participants’ profiles.
Context Participants
Num.
ID Dates Description Profile
(#female)
Workshop May ‘16 Massive Open Online Cours- 24 (9)
UCATx conference
conference #1 es related Staff & Professors.
Workshop June ‘17 Expert researchers on TEL - 15 (5)
RESET project
project meeting with teaching experience.
CoT School #1 High school Teachers. 10 (6)
Research
Oct.’17- project
project with 10 (8)
June’18 (Recer- School #2 High school Teachers.
schools
caixa)
Workshop ITWorldEdu Teachers, Researchers and 23 (9)
April ‘18
conference #2 conference EdTech related stakeholders.
Engineering school - Undergraduate students 32 (10)
University March-
Universitat Pompeu
local project July’18 Professors 26 (9)
Fabra
Total 140 (56)
3
2.2 Procedure and instrumentation
Co-creation was used during the Conceptualization, Development and Evaluation
phases of edCrumble. Participatory design workshops were carried out using several
research methods and instruments depending on the workshop and its context (Fig.1.).
Fig. 1. Co-creation procedure (participatory design workshops’ instruments and methods) dur-
ing the Conceptualization, Development and Evaluation phases of edCrumble. (Access online
figures of the paper here: https://www.upf.edu/web/tide/edcrumble_pictures)
The Conceptualization phase consisted of two workshops with the aim of defining the
edCrumble’ main objectives and features (see Fig.1. Conceptualization: conceptual-
ization workshops 1 and 2). Both used paper prototyping activities, where participants
were working in groups and completed a final individual questionnaire for sharing
their reflections with the researchers.
4
The Development phase consisted of several workshops with two school communi-
ties, which were part of a research project (see Fig.1. Development). During this
phase, participatory design workshops served for advancing on the development of a
web-based prototype of edCrumble using participants’ insights and reflections. The
same workshops’ structure was followed for each school community despite the con-
text was different: in the first school the workshops were about Problem Based Learn-
ing (PBL) and in the second school, they were about Flipped Classroom (FC). During
this phase, participants worked with different versions of the online prototype and
participated on different activities which included focus groups, sharing and discuss-
ing activities, questionnaires and interviews.
The Evaluation phase consisted of several evaluation workshops and an evaluation
study (see Fig.1. Evaluation). In the workshops, participants were involved in a role-
play game whereas they were using edCrumble with the aim of evaluating its usability
and utility. Apart from the design artefacts resulting from the activities, researchers
used a questionnaire for collecting participants’ feedback. In the evaluation study,
researchers worked in parallel with students and professors for evaluating edCrumble
as well as collecting their insights about blended learning and course design. The
study included time for working with edCrumble, questionnaires and interviews.
2.3 Co-creation activities during the conceptualization phase
Conceptualization workshop 1. The aim of the workshop activity was to challenge
each participant to design a blended-learning course using Massive Open Online
Courses (MOOCs). The workshop lasted two hours and the 24 participants were di-
vided into seven workgroups. The workflow of the activity was a five-step process
described on [7] which used several paper materials: a LD template, three framework
sheets and printed LD examples. Specifically, one step of the LD design template was
asking participants to represent their blended LDs using a first timeline model (Fig.2.
left). This visual representation model was evaluated based on the participants sheets
(with the participants insights collected using a questionnaire) contributing in the
initial conceptualization of the main element of the edCrumble editor: the timeline.
Conceptualization workshop 2. The main objective of the workshop was explor-
ing with the participants how visually represent blended LDs and how these visualiza-
tions can facilitate others’ LDs understanding. Researchers prepared two LDs cases
descriptions sheets (both were using MOOCs mixed with face-to-face courses) ex-
tracted from the literature. Moreover, the main material used was a paper LD template
with a new visual model of the timeline with resources’ layers designed based on the
results from the workshop 1 and the literature. Below the timeline, in the LD template
sheet, there was an empty space for placing paper activity cards (which were drawn
and filled in with stickers by the participants following a provided legend) (Fig.2.
middle and right). The 15 participants were working in groups of two/three people
using a LD template per workgroup. The two printed cases descriptions were divided
equally between the existing groups, in such a way that half of the groups worked
with one of the examples and the other half with the other one. Once each group had a
case description sheet, they followed the following steps:
5
1. Read the case provided and represent it using the LD template sheet –placing the
activities and the resources described on the case using the timeline and filling in
the activity cards (and place them on the sheet) (Fig.2. middle).
2. After completing the LD template with their case (Fig.2. right), they had to ex-
change the completed LDs templates between workgroups and interpret the LD
template produced by another group (only looking on the visual representation
and without knowing the LD case description of the template received as the ex-
change occurred between groups that had different LD cases).
3. Finally, each group could check if they had understood well the LD template
received by looking on the corresponding case description. Last, participants
were asked to complete a questionnaire providing their insights about the process.
Results of this workshop pointed out the main strengths and weaknesses of the visual
representation proposed and were useful for discussing whether the timeline and lay-
ers provided by the template were valid for designing hybrid courses. The outputs of
the workshop helped to improve the visual representation and have a more solid base
to start the development of the online version.
Fig. 2. Paper template of the workshop 1 with three templates filled in by the participants (left);
Participants of the workshop 2 completing a paper LD template (middle); Scanned paper LD
template resulting from the workshop 2 -with the timeline and the activity cards (right).
2.4 Co-creation activities during the development phase
Researchers prepared a first online prototype based on the results obtained during the
conceptualizing phase (a web-based tool which provides an editor to work with the
evolved timeline model on an interactive way). The aim of the participatory design
workshops of this phase were prototyping and assessing the preliminary versions of
the authoring tool with the participants of two school communities (Fig.3.). The fol-
lowing steps were carried out in each community.
Development workshop 1. In which teachers had to design a LD using the online
prototype of edCrumble, with the help of the researchers (participants were asked to
come to the workshop with a concrete LD idea). It was a 2h workshop with the fol-
lowing steps: (1) Introduction to edCrumble; (2) Work with edCrumble designing a
LD for being implemented within their classrooms (a PBL or a FC design); (3) Focus
group where researchers asked questions about the experience that participants had
with the use of the tool, discussing their strengths and weaknesses. (4) Last, partici-
pants were asked to answer a research questionnaire individually.
Development workshop 1.2. In the case of the School #2, they had another 2h
workshop because they needed more time for designing the interventions using the
6
tool and be prepared for implementing the LDs in their classrooms. In this case, re-
searchers took observation notes of teachers’ using the tool for usability improve-
ments.
Class implementations. Teachers implemented their LDs in class. During this
step, which took between 4 and 9 weeks, researchers were available online for solving
teachers’ doubts regarding the use of technology selected for using in their class.
Development workshop 2. In this workshop, which took 1-2h depending on the
school, teachers followed three steps: (1) Working with edCrumble for documenting
the LDs implemented at class, adding the design changes suffered by the real imple-
mentations; (2) Sharing their implementation experiences and a joint reflection about
the possible redesign of their original LDs considering the lessons learned; (3) Last,
participants were asked to answer a research questionnaire individually.
Interviews. We carried out seven semi-structured face-to-face interviews (three
teachers from School#1 and four from School#2 –due time and resources constraints
we could not interview all 24) of about 45 minutes each. The interviews consisted of a
series of open-ended questions (see details in [4]) that invited participants to share
their perspectives regarding (1) how they used to design and document their educa-
tional practices before knowing our tool and (2) how was the design process they
followed during the workshops using edCrumble.
Results from this co-creation phase gave rise to a series of design principles (col-
lected in [4]) and facilitated the development of the tool through different prototype
versions (see Fig.1. Development phase). Workshops 1 and 1.2 reported about the LD
process using the tool. Whereas workshop 2 allowed to study how was the use of
edCrumble for redesigning teachers’ own LDs and for understanding others’ LDs.
Fig. 3. Development workshops’ participants working with the online prototype of edCrumble.
2.5 Co-creation activities during the evaluation phase
Evaluation workshops. Each workshop consisted of a role-play game where partici-
pants were placed in groups of 2-4 people. Each group of participants represented an
imaginary school and each participant of each group represented a teacher of a topic
(simulating different educational communities). The role-play game had two main
parts (individual and in group) which each of them had three steps.
The individual activity (at “imaginary” teacher-role level) consisted of: (1) Design
of a short teaching unit with the ILDE2/edCrumble online version –a printed LD was
provided by the researchers for each teacher role (see Fig. 4. left); (2) Analyse the
data resulting from the elaborated LD; and (3) Sharing the design created within the
ILDE2/edCrumble community. Whereas the group activity (at “imaginary” school-
7
role level) implied: (1) Grouping several designs to generate community analytics; (2)
Solving an educational challenge; and (3) Discussing results with all participants.
At the end of the workshop, researchers asked participants to fill in a research ques-
tionnaire for evaluating edCrumble. Last, participants were asked to discuss in groups
about the educational problems which they think edCrumble can solve as well as
those not solved by the tool but can or should be addressed in future versions.
Fig. 4. Printed LDs for each teacher role during the evaluation workshops (left); Participants of
the evaluation study working with edCrumble online version (middle and right).
Evaluation Study. The evaluation study had the following phases (see Fig.1.):
1. PHASE S1 (Registration): students registered voluntarily for the study indicating
3-5 subjects of their bachelor’s degree which they would like to report.
2. PHASE S2 (edCrumble design work): researchers assigned the subjects to the stu-
dents registered depending on their preferences. The workshop was about 2h:
a. 10 minutes: students signed the consent form and a document with information
about their bank account (they received 15€ as complementary compensation).
b. 15 minutes: researchers explained the aim and procedure of the study and did a
short demonstration of how to document a course plan in edCrumble.
c. 80 minutes: students worked with edCrumble in their computers to introduce the
course plan on the system (Fig. 4. middle and right). Students were asked in ad-
vance to come sufficiently prepared to be able to report the course’s LD.
d. 15 minutes: students filled out the first research questionnaire which had two
main objectives: (1) ask students about their opinion about blended learning and
course design; and (2) evaluate edCrumble.
3. PHASE S3 (evaluation and design readjustments): based on the subjects intro-
duced, researchers prepared a second questionnaire with the aim of crosschecking
the different designs introduced on the system, so each subject could be validated
by other students. After one week, students received the second research question-
naire by email, and based on their responses, researchers readjusted the LDs in the
edCrumble system (validating the LDs reported).
4. PHASE P1 (blended learning survey): professors answered a questionnaire about
blended learning and course design.
5. PHASE P2 (design interviews): based on the subjects introduced by the students
and the responses of the professors’ questionnaire (phase P1), researchers made a
list of possible professors of interest on being interviewed. Interviews were carried
out with the aim of discussing the resulting visual representation of the LD ob-
tained with the edCrumble and if they would introduce some changes based on the
information received from the study (using the tool).
8
3 Discussion and lessons learned
3.1 Participants’ recruitment process and motivation
Being recruiting participants a challenging task, we used several sources to recruit
them: two local teaching innovation conferences and the frame of three research pro-
jects. The project with the schools was the unique case where we had the opportunity
of having the same group of participants during more than one workshop. In the other
cases, participants only attended one workshop, being difficult for them to appreciate
the complete picture of the whole co-design process and feel that they were part of
something beyond the isolated activity in which they participated. Moreover, in each
of these workshops, we needed to save workshops’ time for explaining the research
context and ask them collaboration (permission for collecting their data). Whereas in
the case of the project with the schools, we only needed to do this task at the begin-
ning of their first workshop (saving time in the rest of the workshops). Nevertheless,
working with the same teachers during a long-time period (nine months) was also
challenging in terms of keeping their motivation with the activities. Specially, because
the workshops were during the academic course, after classes. Due to their restricted
availability, we adapted ourselves to their schedule when negotiating the dates and
times (sometimes shortening the workshops’ time or avoiding weeks where they had
more work) despite they were agreed collaboration partners in the framing of the re-
search project (with a complementary compensation to the schools).
Both strategies (isolated and long-time period workshops) had advantages and in-
conveniences, but we believe that this combination has been the key to be able to
carry out the co-creation process during all phases. Since we have been able to sched-
ule the workshops on the fly (higher degree of flexibility) bearing in mind the needs
of our research along its whole process (it would have been difficult to elaborate a
completed plan from the beginning). Furthermore, having different participants’
sources have allowed us to work with different stakeholders, including a group of
experts in TEL during the conceptualization phase which added value to our process.
3.2 Managing workshops’ time and participants’ expectations
Due to our context, the workshops had to contribute something to the participants
beyond participating in a co-creation process –in almost all workshops we did, the co-
creation was not the unique goal: e.g., how to design blended learning with MOOCs
or with data analytics (conference workshops), learning PBL and FC methodologies
(schools’ project) etc. This was good for attracting participants, but it was challenging
in terms of managing the limited time and expectations. While we were teaching
something to the participants, we had to collect data and fitting the corresponding co-
creation activity (using edCrumble somehow). The hardest point was managing par-
ticipants’ expectations, finding a balance between their collaboration in our research
and our contribution to them in terms of learning something in the activities (especial-
ly because time was always very limited:1-2h). E.g. during the development phase, it
was a bit demanding for participants learning a new software and creating a LD. For
9
this reason, in the case of the evaluation workshops we used a role-play game (LDs
were already prepared). Therefore, they felt more relax, since they could enjoy the
tool without feeling pressured to have their own LD ideas in parallel.
3.3 Potential and challenges of the co-creation methods used
Table 2 shows a summary of the pros and cons of the methods selected with the aim
of enriching our lessons learned and serve others thinking on similar scenarios.
Table 2. Evaluation of the methods used during the edCrumble co-creation process.
Method Pros Cons
Paper Reducing development time-effort. Time/cost consuming (preparation of the mate-
proto- High flexibility in the expression of rials, analysis of the resulting paper artefacts).
type ideas by the participants. Participants engagement depending on their
profile (some people are reluctant to collaborate
in activities that require crafts).
Web- High satisfaction of the participants Need of managing frustrations during the early
based at the end of the process in feeling phases (early-prototype errors and usability low
proto- that they have collaborated in creat- developed, sense of losing time…).
type ing something real. Developing time and cost consuming.
Possibility of collecting system’
data for the analysis (e.g. log files).
Ques- Valuable individual time for partic- Finding a balance between the time needed to
tionnaire ipants reflection and expression of carry it out (workshop time consuming) and the
their ideas and opinions. number of items to get the necessary data.
Focus High flexibility in the expression of Qualitative analysis with high time consuming.
group ideas by the participants.
High quality data
Sharing Participants can discuss their own Depending on the num. of participants, high
& dis- cases and exchange experiences amount of time is needed.
cussing (learning from others). Need of moderate the discussion when short
time available (keeping the focus, ask relevant
questions, select only representative cases for
sharing…).
Qualitative analysis with high time consuming.
Role- Reducing participants’ required Participants not experiencing their own cases.
play effort on preparing their cases
game (saving workshop time).
Inter- High quality data Participants’ limited availability (in our isolated
views High flexibility in the expression of workshops: difficult to have the opportunity to
ideas by the participants. keep in contact with participants and ask them
collaboration; in our project workshops: teach-
ers’ time limitations).
Qualitative analysis with high time consuming.
10
3.4 Prioritization of feedback diversity
Despite the feedback collected was very diverse during all process, the most challeng-
ing phase regarding its prioritization was during the development of the online tool.
We had to be able to analyse the feedback after each workshop and prioritize it to
prepare a new version for the next workshop. The prioritization process was always a
balance between considering the feasible points to be developed in the time we had
until the next workshop, and that a direct proposal from the participants would always
be included to motivate them to continue in the process (since during the use of the
first versions it was quite frustrating for them to use a system that was not yet very
usable). Having new versions of the prototype in each workshop allowed us to ad-
vance considering participants’ insights and engaging them in the co-creation process.
4 Conclusions
During the co-creation process of edCrumble, researchers used several participants’
sources, different methods and participatory design workshop types (isolated vs. long-
time). Co-creation had a positive impact in the design and decision-making process of
our research, but it also presented some challenges. We hope that this experience and
the challenges documented can help other researchers who are thinking of using co-
creation in the design of teacher tools.
Acknowledgements. Authors want to thank all the participants who collaborated in
the study. This work has been partially funded by RecerCaixa (CoT project) and the
Spanish Ministry of Economy and Competitiveness under MDM-2015-0502,
TIN2014-53199-C3-3-R, TIN2017-85179-C3-3-R.
References
1. Sanders, E. B. N. and Stappers, P. J.: Co-creation and the new landscapes of design.
CoDesign 4(1), 5–18 (2008).
2. Lee, Y.: Design participation tactics: the challenges and new roles for designers in the co-
design process. CoDesign 4(1), 31–50 (2008).
3. Conole, G.: Designing for learning in an open world, vol.4. Springer Science & Business
Media (2012).
4. Albó, L. and Hernández-leo, D.: Identifying design principles for learning design tools: the
case of edCrumble. In Proccedings of the 13th European Conference on Technology
Enhanced Learning, EC-TEL 2018. (Accepted).
5. Mor, Y., Warburton, S., and Winters, N.: Participatory pattern workshops: A methodology
for open learning design inquiry. Res. Learn. Technol 20(SUPPL), 163–175 (2012).
6. Albó, L. and Hernández-leo, D.: edCrumble: designing for learning with data analytics. In
Proccedings of the 13th EC-TEL 2018. (Accepted).
7. Albó, L. and Hernández-leo, D.: Blended learning with MOOCs : towards supporting the
learning design process. In The Online, Open and Flexible Higher Education Conference
2016, pp. 578–588 (2016).