=Paper=
{{Paper
|id=Vol-2128/industrial3
|storemode=property
|title=EDUCATE: Creating the Golden Triangle for Research-Informed Industrial Collaborations within Educational Technology
|pdfUrl=https://ceur-ws.org/Vol-2128/industrial3.pdf
|volume=Vol-2128
|authors=Mutlu Cukurova,Rose Luckin,Alison Clark-Wilson,Tunde Olatunji,Michael McDonald
}}
==EDUCATE: Creating the Golden Triangle for Research-Informed Industrial Collaborations within Educational Technology==
https://ceur-ws.org/Vol-2128/industrial3.pdf
EDUCATE: Creating the Golden Triangle for research-informed
industrial collaborations within education technology
Mutlu Cukurova1, Rose Luckin1, Alison Clark-Wilson1, Greg Moore2, Tunde Olatunji3, Michael McDonald4
1. UCL Institute of Education, University College London
2. Freeformers, 3. LYRICAL KOMBAT, 4. LiguaPracticaVR
Abstract: EDUCATE is a London-based partnership that supports and promotes the use of
research-based education technology, allowing entrepreneurs and start-ups to develop their
products and services, and grow their companies in an evidence-informed manner. The
EDUCATE process connects businesses with researchers who will mentor, guide and support
them on their research journey. A key aspect of this research journey is the evaluation of the
emerging technology being developed by each business that works with the EDUCATE
programme. However, conducting impact evaluations of technology in education is
challenging, particularly so for emerging technologies, as innovation and change are their
essence. Here, we present a pragmatic approach to impact evaluations of emerging
technologies in education which we use within the EDUCATE project. We illustrate the use
of this process through exemplification in the shape of three case studies of educational
technology businesses who have adopted the EDUCATE process.
Introduction
Innovation within the development of commercial educational technology is increasing rapidly. Large
companies have their own in-house research teams to help them connect to existing research and create their
own research projects and publications. However, there is a growing community of small and medium sized
businesses across the globe, who are also using innovative technology to develop their educational technology
products or services. These businesses find it hard to engage with the research community to access existing
research that is relevant to their business, and to understand how best to generate evidence of their own.
EDUCATE (https://educate.london) is a unique project bringing together entrepreneurs and innovators,
with academics, researchers and educators, to deliver world-class education technology products and services.
We describe this linking of the three communities within the education technology ecosystem: developers,
researchers and users (learners and/or educators) as the Golden Triangle (Luckin, 2016). EDUCATE is
designed to ‘fill the gap’ for smaller businesses that cannot afford their own research labs. EDUCATE provides
a rigorous and comprehensive training programme with a focus on pedagogical research and investigation of not
only ‘what works – but also when, how and why’? EDUCATE provides a physical and virtual working space
for cohorts of EdTech Small and Medium Enterprises (SMEs), with training and support to help them to use
research evidence to inform the design of their products and services. Businesses learn how to devise an
effective evaluation for their product or service appropriate to its stage of development and scale.
Background
The impact of technology on learning and teaching is often at the forefront of demands, particularly from those
who dictate the funding available to pay for technology within education systems. This is not an unreasonable
expectation. However, as has been shown in numerous meta-level investigations (see for instance Cox et al.,
2003) evaluation of the impact of technology on educational outcomes is a challenging task. This challenge is
even greater when evaluating emerging innovative technologies. Today’s emerging technologies include, but are
not limited to, virtual reality implementations (Merchant et al., 2014), augmented reality implementations
(Dunleavy, & Dede, 2014), mobile learning devices (Crompton, Diane, & Gregory, 2017), ‘internet of things’
hardware with sensors (Cukurova et al., 2018), and technologies that allow collaborative learning at a great
scale (Cress, Moskaliuk, & Jeong, 2016). Change is at the core of these technologies both because they evolve
over time, but also arguably their raison d'être is to transform the learners’ experience (Cukurova, & Luckin,
2018).
The increased challenge is at least partially due to the connotation that, in traditional impact evaluations,
evidence regarding the impact of an intervention is considered as a shield against change. The generation of
scientifically robust evidence about the impact of an educational technology can therefore be taken as a message
for the stakeholders of this technology to standardise it and scale it up. However, change is the essence of
emerging technologies. For instance, three years after the original report reviewing emerging technology
innovations in education (Luckin et al., 2012), there was evidence that only 39 of the 150 innovations were still
in active use. Therefore, in the context of emerging technologies, the value is to be found in the careful
consideration of different types of evidence that are appropriate to the current state of the technology as well as
in the use of robust methods to generate them. In this paper, we argue for a pragmatic approach to evaluate the
impact of emerging technologies.
Evidence-informed emerging technologies in education
In the past, due to limitations of technology and its use in education, impact evaluations were often completed
through straightforward questions such as “what are the effects of this physics teaching software on students’
understanding of the concept of gravity?” or “Is there an impact of students’ understanding of chemical bonding
after engaging with this simulation?” The evaluations were also undertaken with traditional methods such as
pre- and post-test evaluations (Cox, 2008). However, in the case of emerging technologies, the aim is often to
transform students’ experience of traditional education. Therefore, the exact nature of the intended experience,
including the expected outcome measures and contextual factors, should be clearly defined as the initial step of
the impact evaluation process. This may be conceptualised as a ‘logic model’ or ‘theory of change’ for the
innovation. If we take the position that a person’s context is the sum of their interactions in ther world (Luckin,
2010), then the context in which the impact is to be measured, should be as transparent as possible. This
transparency of context is required to identify the outcome measure(s) which the impact evaluation will study
(Cukurova, Luckin, & Baines, 2017). The complex design of emerging educational technologies requires much
more understanding of human-computer interactions (Cox, 2005) as well the wider learning context in which
they are being implemented.
In addition, emerging technologies vary enormously and multiple researchers have made clear that the
design and use of an educational technology plays a big role in its impact on educational outcomes (see for
instance Reeves, 2008; Pilkington, 2008). Not all emerging technologies are equal in their potential to afford
efficacy. Any kind of impact evaluation in educational technology research, therefore, requires detailed
knowledge of the nature of the evaluated technology, their different representations and the ways in which they
may contribute to learning (Pilkington, 2008). Here we present one method of bringing transparency to the
evaluation of emerging technologies and their contexts.
A theory of change for emerging technologies
Change is the essense of emerging educational technologies. This involves changes in the emerging technology
itself due to its agile and iterative nature, as well as changes in the experiences of learners. According to the idea
of a theory of change (see for example, Kellogg Foundation, 2004), until a change occurs, a state of equilibrium
exists in which any forces that might drive change are equal to the forces that drive a resistance to change. For
change to happen, the balance in the equilibrium needs to be upset (Fullan, 1983). This imbalance can be
achieved either by strengthening the driving forces behind the change, or by removing the barriers that resist the
change (Fullan, 2001). In the context of emerging technologies, a theory of change can essentially be
represented as a diagram that explains how an emerging technology might have an impact on its users. There are
five main steps in the process if creating a theory of change for a particular intervention, such as an emerging
technology:
•What is the expected impact of the emerging technology?
Step1
•What are the intermediate outcomes of the emerging
Step2 technology?
•What are the planned implementation activites for the
Step3 emerging technology?
•What additional resources or input are needed for change to
Step4 occur?
•How will the users will be prepared for this change?
Step5
Figure 1: Theory of Change Diagram Steps for Emerging Educational Technologies
A pragmatic approach to impact evaluations
Impact evaluations of any emerging technology require the generation of evidence regarding any effects arising
from an educational intervention involving that emerging technology. However, views about what constitutes
“evidence” may vary considerably among and between stakeholders. It is important to note here that the type of
evidence does not necessarily reflect the quality of the evidence, and different types of evidence have different
advantages and disadvantages (Marshall, & Cox, 2008). There are 4 main categories of evidence: anecdotal,
descriptive, correlational, and causal evidence. It is increasingly clear that different types of evidence have
different advantages and disadvantages. Therefore, relying on one study or one type of evidence is unlikely to
provide enough reliable evidence to judge the impact of an emerging educational technology due to the
complexity and diversity of the educational landscape.
Multiple sources of evidence are needed in order to strengthen the argument that a particular
technology intervention will be successful under a variety of conditions. Both quantitative and qualitative
sources of evidence are valuable for the statistical power of their large sample sizes (correlational and causal
evidence) and the explanatory power of more in-depth questioning (anecdotal and declarative evidence). It
would be premature if decisions were taken about whether or not to implement an intervention based on one
type of evidence only. A more holistic approach is needed in order to reach a complete picture regarding the
impact of emerging technologies in education. All evidence types may shed light on why an intervention of an
emerging technology succeeded or not, and in what circumstances. Rather than arguing about the overall
superiority of one particular type of evidence or research approach over others, perhaps a more important
question to ask is what type of evidence is the most appropriate type of evidence for this particular emerging
technology innovation, and how can we design and implement interventions and strategies that might help us
generate this type of evidence? Such questions are particularly important in domains with little prior research
(Cobb et al., 2003) which is very often the case for emerging educational technologies' impact evaluations.
Innovation stages of emerging technologies
Figure 2: Innovation spiral as presented in Using Research Evidence: A Practical Guide (Nesta,2016)
The spiral above was developed by Nesta, one of the partners in the EDUCATE project, to capture the different
stages of the innovation process and it can be used to identify the different innovation stages of an emerging
technology. As argued by Brecton et al., (2016) different stages of innovation would require different types of
evidence. For instance, during the initial stages of exploring opportunities and challenges, as well as generating
ideas, it would be beneficial to focus on literature reviews and design principles, identifying what has worked or
failed in the past in different contexts. This evidence can then be used in the design decisions made for the
emerging technologies. These design principles and lessons can help both developers and users of emerging
educational technologies follow strategies that are more likely to have an impact. During the developing and
testing stage, rapid cycle evaluations that would generate anecdotal and descriptive evidence would be
beneficial, whereas during the making the case stage it would be beneficial to undertake impact evaluations that
would generate some correlational evidence. Once an emerging technology reaches a certain level of maturation
through these stages, during the delivery and implementation stage, causal evidence may be required to prove its
impact. The growing, scaling and spreading stage may require larger scale experimental evaluations. System-
level change can only be provided through multiple big scale evaluations from various contexts with clear
implementation manuals that would ensure impact in multiple places. It is interesting to note here, that by the
time an emerging technology reaches the stage of changing systems, a certain level of technological and system
maturation will have been achieved and it will be valid to question the extent to which the educational
technology remains emerging in nature.
The approach put forward by Nesta considers evidence in a holistic manner and recognises the value of
different types of evidence at different stages in the emerging technology innovation cycle. This method
contrasts with more traditional approaches that consider evidence to be of types organised in a hierarchical
manner, with causal research evidence considered as the ‘gold standard’ and other types of evidence being
undervalued as a consequence. We argue for a synergy of evidence types and research methodologies, which
will generate different types of evidence for the impact measures used for emerging technology evaluation.
Our suggestion is based on the view that kite-marking a certain technology as ‘effective’ based on
‘gold standard’ causal evidence, and in so doing encouraging its scaling might be a futile approach for emerging
technologies. This futility is mainly due to the fact that change is a fundamental feature of emerging
technologies. As mentioned in the introduction to this chapter, emerging technologies are constantly evolving
and being implemented in different contexts with different populations. Therefore the value of previous
experimental evaluations for an emerging technology is limited. In addition to which, meaningful large-scale
positivist evaluations of emerging technologies are expensive and they take a long time to complete. There are
various research methodologies that can produce valuable indicators of the potential impact of an emerging
technology, and their use should be encouraged before engaging in large-scale trials.
Three Case Studies
To illustrate the EDUCATE particants’ research journey three case studies are presented for companies that are
at different stages in the innovation spiral (Figure 2). The first is at an early stage with a first prototype of the
product. The second company is more established, with an existing group of users – giving the company access
to existing research data and enabling them to have a more developed logic model. The third is a company that
already has thousands of users, and is looking to utilise its extensive database to develop new AI algorithms to
offer a more personalised user experience.
Case Study 1: LYRICAL KOMBAT is developing an EdTech product that is being designed to encourage and
reward young people to re-imagine hip hop lyrics and poetry through the format of text battles. In so doing, it
aims to connect a hip hop generation to the work of Shakespeare and beyond. This vision is captured in the
logic model in Figure 3, which was developed and refined during the company’s engagement within the
EDUCATE programme.
Figure 3 LYRICAL KOMBAT: Logic model
The founder of LYRICAL KOMBAT, in discussions with the EDUCATE research mentors, then proposed a
research question that would form part of his research proposal that seeks to evaluate the impact of students’
engagement with his product, “Is rhyme detection skill a predictor of ability for the production of rhyming
text?”. This reserach project is now being developed by the entreprenuer, with the support of their
EDUCATE expert mentors. A participatory design research methodology is evolving through which the
entrepreneur can begin to uncover the impact of his proptotype design, prior to committing to a costly product
development and big scale evaluation studies.
Case Study 2: LinguapracticaVR
LinguapracticaVR offers a virtual reality (VR) immersive English language experience for second-language
English learners. Teachers and students interact with, and contribute to, the learning platform. The VR resources
aim to enable teachers to design more engaging lessons that are in turn motiviating for students to want to
pursue learning English and impact positively on students’ attainment.
LinguapracticaVR devised the following set of research questions, which are now being refined alongside
collaborations with their EDUCATE expert research mentor:
• To what extent does task-based learning within immersive virtual environments (“VR”) increase the
ability of the EFL learners to express actions performed in the past?
• To what extent does task-based learning within VR increase a student’s motivation to learn more
English?
• To what extent does task-based learning within VR increase a teacher’s ability to engage students
better?
LinguapracticaVR has identified its sample of users, and is supporting both teachers and students to become
familiar with the VR enrivonment, which includes the creation of contextualised teaching resources, prior to
beginning a formal pilot study that will generate user data.
Case Study 3: Freeformers has an established blended learning programme that provides face to face and
online training for business employees. The purpose is to grow and upskill the workforce, with a particular
focus on developing mindset. In its engagement with EDUCATE, Freeformers is developing a research-
informed new product that aims to measure changes in participants’ mindset, skillset and behaviours.
Freeformers is seeking to research hypothesis that their underlying training model develops user’s
mindsets and creates resultant changes in behaviour.
For this research question Freeformers has adopted a quasi-experimental research approach, as by examining
the changes in user’s responses to statements, they can test the effects of learning on user’s mindsets.
Their methodology involves surveys of a group of 81 existing users before, during and after a particular
intervention. The survey will also be shared with a control group of representative colleagues who are not
participating in the learning programme of the company. By monitoring resultant changes in behaviour,
particularly frequency of certain actions, Freeformers aim to analyze the efficacy of their pedagogical method
in the workplace.
Conclusions
Emerging technologies can disrupt and bring about unexpected change, the consequences of which must then be
managed. Their evaluation is a key part of the way in which their impacts are effectively integrated into learning
and teaching settings to bring the best benefit to learners and teachers. Through the EDUCATE process,
we introduce entrepreneurs to an approach to evaluating impact with two main steps. First one is the creation
of a clear theory of change to identify outcome measures and assumptions that are behind the expected impact
of the emerging technology intervention. Secondly, the identification of the type of evidence and methods to
generate it that are the most appropriate for the current innovation stage of the emerging technology.
The three case studies presented here provide exemplification of contrasting methodological
approaches adopted by EDUCATE companies as they collaborated with us to develop their theory of
change and outline a proposal for their own research study. As the case studies presented here exemplifies the
value of the pragmatic approach we have taken, rather than focusing on a particular research methodology or
paradigm for all companies engaged in the EDUCATE programme.
References:
Breckon, J. (2014) Using research evidence: A Practical guidance. Nesta:UK.
Cobb, P., Confrey, J., DiSessa, A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational
research. Educational researcher, 32(1), 9-13.
Cox, M. J. (2005). Educational conflict: The problems in institutionalizing new technologies in education. In G.
Kouzelis, M. Pournari, M. Stoeppler, & V. Tselfes (Eds.), Knowledge in the new technology. (pp. 139–
165). Frankfurt: Peter Lang.
Cox, M. J. (2008). Researching IT in education. In International handbook of information technology in primary
and secondary education (pp. 965-981). Springer US.
Cox, M.J., Webb, M., Abbott, C., Blakeley, B., Beauchamp, T., & Rhodes, V. (2003). ICT and pedagogy: A
review of the research literature. ICT in schools research and evaluation series, 18.
Crompton, H., Burke, D., & Gregory, K. H. (2017). The use of mobile learning in PK-12 education: A
systematic review. Computers & Education, 110, 51-63.
Cress, U., Moskaliuk, J., & Jeong, H. (Eds.). (2016). Mass collaboration and education (Vol. 16). Springer.
Cukurova, M., & Luckin, R. (2018). Measuring the impact of emerging technologies in education: A pragmatic
approach. In International handbook of information technology in primary and secondary education
(2nd Edition). Springer, Boston, MA.
Cukurova, M., Luckin, R., & Baines, E. (2018). The significance of context for the emergence and
implementation of research evidence: the case of collaborative problem-solving. Oxford Review of
Education, 1-16.
Cukurova, M., Luckin, R., Millán, E., & Mavrikis, M. (2018). The NISPI framework: Analysing collaborative
problem-solving from students' physical interactions. Computers & Education, 116, 93-109.
Dunleavy, M., & Dede, C. (2014). Augmented reality teaching and learning. In Handbook of research on
educational communications and technology (pp. 735-745). Springer New York.
Fredericks, J., Blumenfeld, P., & Paris, A. (2004). School engagement: Potential of the concept, state of the
evidence. Review of Educational Research, 74, 59–96.
Fullan, M. (2001). Leading in a culture of change. San Francisco, CA: Jossey-Bass.
Fullan, M. (1983). Evaluating program implementation: What can be learned from Follow Through. Curriculum
Inquiry, 13(2), 215-227.
Kellogg Foundation. (2004). W.K. Kellogg Foundation logic model development guide.
http://www.wkkf.org/knowledge-center/resources/2006/02/ WK-Kellogg-Foundation-Logic-Model- ·
Development-Guide.aspx.
Luckin, R. (2010). Re-designing Learning Contexts. Routledge: London.
Luckin, R., Bligh, B., Manches, A., Ainsworth, S., Crook, C., & Noss, R. (2012). Decoding learning: The proof,
promise and potential of digital education. London, UK: Nesta. Retrieved from
http://www.nesta.org.uk/publications/decoding-learning
Luckin, R; (2016) Mainstreaming Innovation in Educational Technology. Advances in SoTL , 3 (1).
Marshall, G., & Cox, M. J. (2008). Research methods: Their design, applicability and reliability. In International
handbook of information technology in primary and secondary education (pp. 983-1002). Springer,
Boston, MA.
McDougall A. Murnane, J., Jones, A. & Reynolds, N. (Eds.), Researching IT in Education: Theory, Practice and
Future Directions. Routledge: Taylor and Francis Group: Abingdon. England
Merchant, Z., Goetz, E. T., Cifuentes, L., Keeney-Kennicutt, W., & Davis, T. J. (2014). Effectiveness of virtual
reality-based instruction on students' learning outcomes in K-12 and higher education: A meta-analysis.
Computers & Education, 70, 29-40.
Pilkington, R. (2008). Measuring the impact of IT on students’ learning. In J. Voogt, & G. Knezek (Eds.),
International handbook of information technology in primary and secondary education. Berlin
Heidelberg New York: Springer.
Reeves, T. C. (2008). Evaluation of the design and development of IT tools in education. In J. Voogt, & G.
Knezek (Eds.), International handbook of information technology in primary and secondary education.
Berlin Heidelberg New York: Springer.