=Paper=
{{Paper
|id=Vol-2676/paper4
|storemode=property
|title=Designing Technology-enhanced Learning research for sustainable impact: The Learning Layers case.
|pdfUrl=https://ceur-ws.org/Vol-2676/paper4.pdf
|volume=Vol-2676
|authors=Tobias Ley,Paul Carder,Rose Dewey,Raymond Elferink,Pekka Kämäräinen,Werner Müller,Gilbert Peffer,Tamsin Treasure-Jones
|dblpUrl=https://dblp.org/rec/conf/ectel/LeyCDEKMPT20
}}
==Designing Technology-enhanced Learning research for sustainable impact: The Learning Layers case.==
https://ceur-ws.org/Vol-2676/paper4.pdf
Designing Technology-enhanced Learning Research for
sustainable impact: The Learning Layers case
Tobias Ley1, Paul Carder2, Rose Dewey2, Raymond Elferink3, Pekka Kämäräinen4,
Werner Müller5, Gilbert Peffer3, Tamsin Treasure-Jones3
1 School of Educational Sciences, Tallinn University, Estonia
2 NHS Bradford Districts Clinical Commissioning Group, Bradford, UK
3 Kubify BV, Utrecht, The Netherlands
4 Institut Technik & Bildung, University of Bremen, Germany
5 Independent Researcher and Lecturer, Germany
tley@tlu.ee
Abstract. Research in technology-enhanced Learning (TEL) is often criticized
for not leading to sufficient and sustainable impact in practice. This is especially
so for workplace learning technology where learning and the use of technology
needs to be embedded in working practices. We report on Learning Layers, a
TEL project on workplace learning that ran between 2012 and 2016. We have
identified three cases where impact on practice is still visible today, as learning
technology originally developed in the project is still in use, impacts learning,
teaching and working practices in healthcare practices, construction training and
in the application of e-posters in academic conferences. We use the cases to iden-
tify factors during the lifetime of the project that have facilitated sustainable take
up of research results, namely a flexible and collaborative project roll out, an
open research process, early focus on end-users and an inclusive process of plan-
ning for impact and exploitation. Taken together, these factors suggest to look at
research projects as providing flexible and networked co-creation platforms for
innovation.
Keywords: Technology-enhanced Learning; Workplace Learning; Research;
Impact; Sustainability; R&D projects; Co-design
1 The challenge of long-term impact of TEL research
Research on Technology-enhanced Learning (TEL) has often been criticized for not
leading to long-term, sustainable impact on practice. While this points to a typical re-
search-practice gap that can be observed in many disciplines, the gap is somehow re-
markable in the case of TEL research, as it is often problem-based and rather applied
which should increase the likelihood of applications in practice.
There are surely a number of notable exceptions that have demonstrated the re-
search-practice gap in TEL can be closed and that research has contributed remarkably
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
2
to innovation. These exceptions can be especially found in the context of formal in-
struction and in schools. For example, several intelligent tutoring systems have found
broad take-up in schools and have also led to commercial success. Some large-scale
learning platforms for learning in schools have been widely successful. An example is
the GOLAB inquiry learning platform that uses virtual labs for science education. Fi-
nally, there are several platforms that promote teacher collaboration. The GOLAB plat-
form is an example, as is the eTwinning platform that connects thousands of teachers
throughout Europe, or less well known cases, such as LeMill [6] with 41,000 registered
teachers has shared 66,000 open educational resources in 86 languages (numbers from
2013). Still, the most likely fate of TEL research is that uptake in practice ceases once
research funding runs out. The situation is even more challenging for projects focusing
on workplace learning where learning and the use of technology needs to be tightly
integrated with working practices.
In FP6 and FP7, the European Commission has funded several large-scale projects
(called Integrated Projects) which were meant to change this situation, and particularly
focused on technology for workplace learning [7]. These projects were targeted at im-
pact, not only in terms of research, but also in terms of commercial technology devel-
opment and application of learning technology in practice. Besides research institu-
tions, these projects therefore included also technology companies to ensure commer-
cial uptake, as well as application partners who were the intended beneficiaries of the
learning technology (e.g. training companies, or companies seeking to apply technol-
ogy for their internal workplace learning purposes).
With this paper, we are intending to shed light on the possible success factors of
these R&D projects. We started from the following research question: what are the
successful practices that have been integrated in these R&D projects to make long-term
impact more likely?
2 Case Study: analyzing sustained impact of the Learning
Layers project
2.1 Choice of the research method
Answering the research question is difficult for at least two reasons. First, answering
the question would require a larger impact analysis of TEL R&D projects that were
completed years ago (an analysis that is much beyond the scope of this paper). Sec-
ondly, the question is difficult to answer because successful cases are actually difficult
to locate. While publishing is very active during the runtime of the project or shortly
after to describe effects of research on practice (e.g. [10,12]), and longer term scientific
reviews are being conducted to understand impact on research [6], locating long-term
impacts on practice is much more difficult.
We decided therefore to conduct a case study, and chose the Learning Layers 1 pro-
ject as our case. Despite having finished already in 2016, several partners are still in
contact and could therefore share experiences of developments since then.
1 http://learning-layers.eu
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
3
For this paper, we have collected incidental evidence, rather than conducting a more
systematic impact analysis. Evidence was collected through the analysis of three cases
which are being undertaken by some of the application and technology partners of the
original project. These activities are still ongoing, and can be clearly linked to some of
the results produced during the project lifetime which have since then been significantly
extended. The impact cases were analyzed to understand which of the practices under-
taken in the project during its lifetime could have had a positive influence on the sus-
tainability of those impact cases.
2.2 The Learning Layers project: scaling workplace learning
The Learning Layers R&D project was conducted between 2012 and 2016 with the
main objective of using learning technology to scale workplace learning particularly in
those sectors that had been less inclined to use technologies for learning. We addressed
learning in two of these sectors: to address the first sector, we considered workplace
learning of healthcare professionals working in general health care practices in the UK.
The second application was undertaken in the German construction industry, where we
especially focused on the learning of construction apprentices and on bridging learning
in a training center and at the workplace. Practitioners from these sectors were active
participants in the research, and the project followed a highly stakeholder-driven co-
design approach [7].
Technologies used in those cases were especially focused on mobile and social tech-
nologies that allowed to contextualize learning in the work setting, and allow for col-
laborative knowledge building. The set of technologies was complemented by several
backend services that allowed quick deployment, as well as data-driven technologies
that allowed tracing of some of the social learning processes [9].
Altogether, the project was implemented by 17 partners from 7 different countries
over the course of 4 years. The project was integrative in the sense that it combined a
number of related activities (including research studies, collaborative design, techno-
logical development, and conducting multiple evaluation, exploitation and outreach ac-
tivities) into an overall approach and with a total effort of over 100 person years. The
project was also set up in a very collaborative way, meaning that most partners were
involved in many of the project activities. There were not many “hand-over” tasks, but
rather tasks were running in parallel and were then synchronized at particular mile-
stones (e.g. for the testing of a set of technologies in a particular setting).
From a scientific point of view, the project can be considered successful from typical
scientific standards: its outcomes have been documented in multiple scientific papers
(a recent review paper summarizes some of these [6]), and in 2018 the team won a
European research award for Vocational Education and Training 2. Also from a practical
point of view, several studies confirmed the impact in practice (examples are given in
[2, 12]). However, as with many other projects, several of the project activities stopped
after the project ended. Most of the deployments in practice were not continued after
2 http://results.learning-layers.eu/vet-2018/
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
4
funding had ended, but most of the research and some of the technological develop-
ments moved to other projects and are being further developed there, partly also com-
mercially (see next section).
3 Three cases of sustained impact on practice
While many of the deployments in practice that were happening within the more con-
fined contexts of the project were discontinued after the end of the project, it is remark-
able that some initiatives that were started through the project are still ongoing, even 4
years after the project has ended. Three of these cases that are described in this section
center around the development of Learning Toolbox (LTB) [2], a technology that al-
lows the creation of stacks of digital content, distributed and accessed through mobile
devices. The initial prototype of this platform was developed and evaluated in the pro-
ject as a response to the needs of the training center Bau-ABC Rostrup in Germany, a
major application partner from the construction sector (see case 1 below). While the
initial design idea referred to digitization of training materials, instruction sheets, and
self-assessment procedures, the further development turned LTB into an interactive
toolset to support training and learning activities more widely. The technology was
subsequently commercialized by the Kubify startup 3 and found application in further
application settings (see case 2 and 3).
3.1 Case 1: LTB for construction training in the Bau-ABC training center
Problem addressed. In the training center, LTB was applied in the context of appren-
tices’ construction projects. The instructions for the apprentices to conduct these pro-
jects had been provided orally with the help of instructive worksheets (for preparing
the project plans). Likewise, the reporting on the projects was done manually. With the
help of the LTB, the typical structure of the projects, as self-organized learning tasks
became transparent: apprentices had to search for resources, draft the plan, report the
implementation and assess the outcome.
Stakeholders involved. The co-design work was carried out as a collaborative pro-
cess by researchers, technical partners and full-time trainers from Bau-ABC. During an
earlier phase of the work the project team provided basic multimedia training for some
voluntary trainers. At a later phase the project team and these trainers provided an in-
tensive training campaign for all trainers of the training center. In the pilot testing of
the Learning Toolbox a core group of trainers introduced the toolset in their training
and results were monitored by the project team. Also, at the final phase of the project,
the use of LTB as support for construction work processes was demonstrated for several
craft trade companies. As a result, follow-up processes (feasibility studies and project
initiatives) were started with some companies.
Learning technology and its use. LTB allows trainers to easily customize contents
and learning processes for different training purposes and according to the pedagogic
3 https://ltb.io/company/
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
5
priorities of the trainers. Thus, LTB made it easier for the trainers to emphasize inde-
pendent searches in a wider range of web resources. This is essential for borehole build-
ers, for example, who are working alone on remote construction sites. Also, it makes it
possible to give learners gradual access to a wider range of resources (and to solutions
of their peers) once they have learned to develop their own solutions to the project tasks.
Moreover, LTB encourages apprentices to document their projects within the platform.
This enables instructors to see progress of their apprentices in real time and provide
more timely feedback. The LTB also encourages the self-organization of the instructors
in terms of streamlining their content and sharing common resources between the dif-
ferent professions. While this approach to collaborative teaching was already there at
Bau-ABC, the LTB offers a further channel to systematize this practice. Altogether, the
co-design process, the piloting phase and the follow-up phase have been characterized
by intensive R&D dialogue underpinned by the accompanying research approach of the
research institute ITB at the University of Bremen.
Impact. After the Learning Layers project had ended, the use of LTB was encour-
aged across all trades in which Bau-ABC Rostrup provides apprentice training. Conse-
quently, most of the apprentices’ projects are now being supported through LTB in
some form. Based on this case, other German training centers in the construction sector
are in the process of trialing LTB both for initial and continuing VET. Due to the clo-
sure of the training centers because of the COVID19 pandemic, the trainers of Bau-
ABC Rostrup have prepared trade-specific stacks with LTB to support independent
learning. The trainers of Bau-ABC have become strong multipliers of innovation both
within their organisation and in their networking with other training centers and partner
companies.
3.2 Case 2: LTB in UK health care practices to counter over-use of antibiotics
Problem addressed. This case addresses the problem of over-use of antibiotics which
leads to a rise in antimicrobial resistance. As part of a nationwide 5-year strategy to
champion responsible use of antibiotics, the Bradford Districts Clinical Commissioning
Group (CCG) which is part of the UK National Health Service (NHS) has run a series
of audit and feedback projects over the past few years, which involve sending out prac-
tice-specific paper audit and feedback reports to all practices in West Yorkshire and
Harrogate (approx. 300 general practices), covering the regional Integrated Care Sys-
tem (ICS). The latest project in this audit and feedback series focuses on antibiotic pre-
scribing. One problem the CCG has faced with these projects is that the reports are
sometimes not well shared to all appropriate staff within the practice, and may remain
with a ‘gatekeeper’.
Stakeholders involved. Main stakeholder is the Bradford Districts CCG, specifi-
cally the West Yorkshire Research and Development team (a NHS R&D team for all
West Yorkshire and Harrogate general practices) and the medicines optimization teams
within the CCG. The program targeted general practices in West Yorkshire and Harro-
gate and the staff members within those practices. The case also draws on the LTB
which was provided supported by Kubify, the start-up that had commercialized the
technology. The University of Leeds has been involved in developing the audit and
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
6
feedback program. Further parties were Public Health England and the local authority
infection control teams.
Learning technology and its use. LTB has been used to create stacks for each prac-
tice and thereby improve the accessibility of the practice reports as well as to enable
the sharing of additional resources which could not be included in the main report due
to space. The app has thus improved the range of information that can be shared, and
links are also shared which allow users to read more in-depth into the topic areas. The
use of LTB has also enabled the spread of information more widely, as the team sug-
gested that the stack poster (a paper-based poster displaying the link to the stack and a
QR code) should be displayed in the practice to allow any interested staff to access the
stack and resources. The use of the stack also allows for all the information to be kept
by interested staff in one central place, so previous reports and resources can be referred
back to at any point. It can also be accessed via a personal mobile device, so gives the
opportunity for users to access the information at the most convenient time for them,
and without the need to have the paper report or to log in to a system.
Impact. Anecdotal evidence has been collected about the use of the stacks and the
benefit that users have found. Usage statistics indicate that some practices have had a
number of accesses to their stack, indicating that multiple members of staff may have
accessed. A systematic impact study of the use of LTB has not yet been conducted, but
is currently planned at a later stage. During the Covid-19 pandemic, it was not possible
to send out paper reports, so it can be anticipated that the stacks will become more
important. We also hope to be able to show a positive impact on antibiotic prescribing
over time, which aims to reduce antimicrobial resistance in the population.
3.3 Case 3: LTB as an ePoster solution for academic conferences
Problem addressed. This use case addresses the broad problem of how conference
attendees can more effectively present, share and develop their knowledge during and
after conferences. However, it starts from a very specific problem that we refer to as
the “paper poster problem” [12]. The paper poster session is a common component of
many academic and scientific conferences, but it has limitations. The format does not
allow for the inclusion of multimedia resources in the presentation, it does not support
interaction with the data, recording of the discussions about the poster or easy sharing
of the knowledge (the poster) within and beyond the conference. Existing ePoster so-
lutions had only partly addressed these issues by moving to a digital format (a PDF on
screen) without taking advantage of the opportunity to enrich the ePoster content or
support informal learning around the content.
Stakeholders involved. This use case was developed by three of the individuals
involved in the Learning Layers project. The individuals developed the use case, piloted
it and built the company (Kubify) after the end of the project. Looking at the use case
more broadly, the other stakeholders are conference delegates, scientific associations,
conference organizing companies and venues. In terms of the business model, it is usu-
ally the scientific association or conference organizing company that is the customer
and the conference delegates who are the main users.
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
7
Learning Technology and its use. This use case involved adapting LTB so that it
could be used as an ePoster platform by conferences. It allows the ePoster author to
easily create an ePoster that is a collection of rich multimedia and interactive resources.
It allows the conference delegates to open and engage with these ePosters on their own
devices (mobile phones, laptops) rather than using hired screens. This interaction can
include exploring and interacting with the ePosters, engaging in a discussion attached
to the ePoster, adding content to the ePoster and sharing the ePoster with others. The
educational aim is to facilitate informal learning by supporting key activities such as
actively engaging with content (rather than passive consumption), discussion of con-
tent, addition of new content (where the author allows this), sharing of content and
making connections with people with similar interests. A formal evaluation has not yet
been undertaken, but reviews of the ePosters created for the conferences show that au-
thors are taking up the opportunities to enhance their content. Typical ePosters now
include videos, links to surveys, data presentations that the viewer can actively explore,
research material that can be reused by others and some even allow viewers to add
‘missing’ content and ideas. Furthermore, the ePosters are still being accessed long af-
ter the conference. The poster sessions themselves are also changing with the introduc-
tion of more structure (as compared to the open drinks reception poster sessions) such
as poster pitches, round table poster sessions, ePoster BarCamps, reciprocal peer review
in the week prior to the conference and poster sessions with CPD credits awarded for
engagement with the poster content [4].
Impact. This case was developed after the Learning Layers project ended and the
first pilot was with a large, international medical education conference in 2017 [3]. In
2018 it was used at 6 conferences across Europe. In 2019 this number grew to 14 and
also included US conferences. The forecast for 2020 is that it will be used by more than
30 conferences with growth in the US being particularly strong. The use of the platform
by conferences also has an overlap with the other cases discussed in this paper, since
we have evidence that users who first experienced the platform at a conference have
subsequently introduced it to their organization as an educational or knowledge sharing
tool. The educational impact (in terms of improved informal learning, knowledge shar-
ing and development) has not been formally evaluated since the small founding team
has been focusing on developing the company. However, the anecdotal feedback from
users, the number of returning customers and the increase in sales suggest that the so-
lution is valued by the stakeholders. Future plans are still focused on company devel-
opment and sales, but ideally will also include a formal evaluation, possibly as a re-
search collaboration.
4 Lessons Learned: R&D practices towards impactful TEL
research
Looking at the cases presented in the last section, we are presenting here some of the
lessons we learned for how to organize and implement R&D projects for impactful TEL
research. The lessons learned focus on those factors within the lifetime of the project,
and not on all the steps that were necessary for the later successful commercialization.
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
8
4.1 Strong focus on co-design and stakeholder engagement
The Learning Layers project was heavily built on contextual inquiry and collaborative
design. We employed phases of contextual inquiry, and the main R&D work especially
in the first years was undertaken by co-design teams composed of researchers, practi-
tioners and technology developers. The co-design methodology followed standard
practices of identifying practical problems, gaining an in-depth understanding of the
context in which practitioners were working and exploring different design solutions.
The process was supported by co-design platforms, open software development plat-
forms and R&D methodologies by which partners worked together closely in an agile
manner. Contact between partners was on weekly, sometimes even daily, timescales
and included rich and challenging discussions to help partners to understand the com-
plex learning contexts and the different perspectives in the different Learning Layers
use cases. The design teams were not predetermined in the project plan, but flexibly
formed in the first year, and also open to changes in their composition later on. The
teams also undertook quite an effort to engage with a wider set of practitioners beyond
those involved in the project.
This working mode helped sustain some of the activities beyond the project lifetime.
In case 1, for example, the co-design process, the piloting phase and the follow-up phase
have been similarly characterized by an intensive collaborative R&D dialogue that led
to further adjustment of the tool development aligned with the pedagogic approach of
the trainers.
While co-design methodologies may be standard for commercial projects, they are
much less common for research projects. Despite the fact that recent calls have been
made to include knowledge co-production methodologies to increase the impact of ac-
ademic research [14], there are still preconceptions towards these methodologies when
it comes to academic publishing. Within the project, a large emphasis was made to
improve existing co-design methodologies to strengthen their rigor for research (e.g.
[1]).
4.2 Revising early commitments and applying technology in previously
unforeseen contexts
The flexibility described in the previous paragraph has a positive side-effect, namely
that certain commitments made early in the design process may be questioned and re-
vised at later stages. This is critical for sustained impact, as application circumstances
and priorities change, and ongoing technological developments might make things pos-
sible that previously were not. A flexible process allows to take up opportunities as they
arise. This might also lead to a situation where certain technologies are applied in con-
texts that were not originally foreseen.
As an example, consider case 2, the application of Learning Toolbox in the Health
Care practices. In early design sessions in those practices, the use of mobile technology
was not prioritized, as certain restrictions on mobile phone use were in place at the time,
the majority of the staff were not using smart devices, and also the contextualization of
learning in terms of physical location (which would speak for the use of mobiles) was
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
9
not seen as a major challenge. So the use of mobile technology was mainly pursued in
the construction domain (as case 1 testifies). However, as circumstances changed (i.e.
mobile phones became more widely used and accepted in healthcare), and new oppor-
tunities arose (i.e. by having a mobile platform successfully tested in another context),
these earlier decisions were revised. As case 3 shows, this extended to contexts that
were not even seen as the original project application contexts at all.
This discussion also points to another important factor of the project, namely the
project size, scale and length. Quite often, large and long-term projects as the current
one are criticized for being overly complex and overburdened with administrative and
coordination tasks. While this may be true, in the current case we also see some benefits
of having a project with the capacity to address a problem from different sides and with
different technological solutions. It also shows that if a project is successful in building
a common understanding and trust, this can later facilitate the uptake of solutions, even
if this involves commercial risks.
4.3 Flexible inclusion of partners during later phases of the project
It follows from the previous two lessons learned that, due to the flexibility, some of the
original project partners may lose interest, change their roles or new ones may enter. If
we look at the cases, then we notice that this is precisely what happened with those
initiatives that have been successful for a longer period of time. The final setup of part-
ners that finally sustained the initiatives were not all part of the research project from
the beginning. Rather, several partners were exchanged half way through the four-year
project, and several other ones joined after the project had already ended. This possi-
bility of including partners later in the process was critical and allowed us to take up
emergent opportunities.
It should be noted that these kinds of partner changes were actually not foreseen as
a regular process in how FP7 projects were set up. Rather they were treated as extraor-
dinary changes that required a massive administrative overhead for changing contracts
and the workplan. While it is clear that projects of this size carry a large responsibility
in terms of financial risks that needs to be planned and justified, it is probably also true
that flexibility could be better built into the funding programs to allow opportunities for
sustainable impact to arise.
4.4 Planning for exploitation and commercialization of solutions
Obviously, a key factor that contributed to the sustainability of all three cases lies in the
fact that commercialization of the technology was finally successful. While it was un-
certain at the end of the project, how the innovations could be sustained and spread, an
approach to exploitation planning had been initiated in the project that put emphasis on
discovering initial viable solutions, building exploitation journeys and building teams
and business cases to realize them [8]. In the construction sector, it was essential that
the developers of LTB and the accompanying researchers from ITB took several initi-
atives to launch follow-up activities with construction companies.
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
10
The commercialization of LTB was finally successful through considerable work,
personal risk and investment undertaken by the 3 co-founders of Kubify after the Learn-
ing Layers project. Their work has led to the establishment of a viable, commercial
company that has gained income and customer traction, and therefore has the capacity
to maintain both the ePoster use case and also other use cases involving LTB.
An interesting aspect of this collaborative partnership is that the three individuals
came from separate Learning Layers partners and different EU countries. It is very un-
likely they would have met had it not been for the Learning Layers project. The three
individuals each bring a slightly different perspective to the work – technical develop-
ment, educational research, finance and project management – which together were
critical for final commercialization.
The highly connected and collaborative nature of the Learning Layers project (as
described under 4.1) may have increased the likelihood that some of those individuals
(from different partners) would have started joint initiatives through the project, even
if not initially foreseen in the project plan. And they would have then continued to stay
in contact and work together beyond the project, having identified shared interests and
seen the benefits of working together, particularly in an interdisciplinary and co-design
manner. Another potential success factor for commercialization was that the commu-
nity, built up during the Learning Layers project, gave the founding team access to
advice and insights from others after the project as well. Yet another key factor was the
fairly liberal licensing and intellectual property agreements that had been agreed at the
outset. Software was licensed under open source licenses that permitted commercial
use.
4.5 R&D projects as networked innovation hubs
If to summarize these four lessons learned, we can see a common pattern emerging. It
seems that R&D projects, especially ones of such scale, should not be understood as
large research studies planned out in detail over four years, in case they are expected to
produce sustained impact in practice. A better way of conceiving these would be in
terms of networked innovation hubs or platforms in which research elements can re-
ceive initial validation, and a team is built and supported by a larger community to start
an innovation process. This requires that trust is built, relationships are formed, and a
common understanding of problems and possible solutions is created. From a techno-
logical point of view, this should be accompanied by an open approach to building the
architecture instead of aiming for a tightly integrated solution. Open and agile methods
increase the chance that innovative solutions are discovered that then have a viable
chance for commercialization.
5 Conclusions and Limitations
In this paper, we have analyzed three cases that have sustained some of the pedagogical
and technological innovation for workplace learning that were initially created and
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
11
tested in the Learning Layers R&D project. From this analysis, we propose that suc-
cessful R&D project with regards to sustainable impact on practices should be con-
ceived as networked and flexible innovation hubs in which initial research ideas and
prototypes are developed and validated in practice. We have suggested a few R&D
practices that are conducive to such approach, such as co-design and a highly collabo-
rative and agile R&D methodology. If through these means sufficient trust, common
understanding and a community has been created within the project, then it becomes
much more likely that commercial opportunities can be sought quickly, individuals are
ready to take resulting risk, and initiatives are sustained beyond the lifetime of the pro-
ject.
Clearly, the analysis we have undertaken here is initial and to some extent specula-
tive for several reasons. First, as we are considering a time frame of several years, a
multitude of intervening factors certainly have played a role in determining which ini-
tiatives were sustained and for which reasons. Second, as our analysis was done by
hindsight, we may have distorted the facts. And thirdly, as we have mainly focused the
analysis of success factors on the period of the project lifetime, the success factors we
have derived here are necessarily limited, and they can not do justice to all the work
that was essential in making these cases into successful examples of TEL adoption.
In follow-up work, we will try to make a more systematic analysis by employing a
more structured analysis approach and a larger set of respondents. In that work, we will
also focus on the period after the project had ended in order to understand the dynamics
that ensued after the successful completion of the project.
Acknowledgments
This research has received funding from the European Union’s FP7 research and inno-
vation program under grant no. 318209 (Learning Layers, http://learning-layers.eu),
and from Horizon 2020 under grant no. 669074 (CEITER, http://ceiter.tlu.ee).
References
1. Dennerlein, S., Tomberg, V., Treasure-Jones, Tamsin Theiler, D., Lindstaedt, S., & Ley, T.
(2020). Co-designing tools for workplace learning: A method for analysing and tracing the
appropriation of affordances in design-based research. Information and Learning Sciences,
121(3/4), 175–205. https://doi.org/10.1108/ILS-09-2019-0093
2. Elferink, R. (2018). Learning Toolbox - Build, organise, and share your own apps for com-
munication, collaboration, and learning. http://results.learning-layers.eu/tools/learning-
toolbox/
3. Kämäräinen, P. (2018). Use of Learning Toolbox by Bau-ABC Trainers and Apprentices,
http://results.learning-layers.eu/impacts/c-11/
4. Kubify (2017). A short case study of the ePosters at AMEE 2017. Report for customer pub-
lished on Kubify’s website, https://ltb.io/wp-content/uploads/2019/06/LTB-for-ePosters-
AMEE-2017-case-study.pdf, last accessed 23rd April 2020.
5. Kubify (2020). How are conferences using Kubify’s LTB ePosters. Article written for Ku-
bify website, https://ltb.io/2020/04/26/how-are-conferences-using-kubifys-ltb-eposters/
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
12
6. Leinonen, T., Purma, J., Põldoja, H., & Toikkanen, T. (2010). Information Architecture and
Design Solutions Scaffolding Authoring of Open Educational Resources. IEEE Transac-
tions on Learning Technologies, 3(2), 116–128. https://doi.org/10.1109/TLT.2010.2
7. Ley, T. (2020). Knowledge structures for integrating working and learning: A reflection on
a decade of learning technology research for workplace learning. British Journal of Educa-
tional Technology, 51(2), 331–346. https://doi.org/10.1111/bjet.12835
8. Ley, T., Cook, J., Dennerlein, S., Kravcik, M., Kunzmann, C., Pata, K., … Trattner, C.
(2014). Scaling informal learning at the workplace: A model and four designs from a large-
scale design-based research effort. British Journal of Educational Technology, 45(6), 1036–
1048. https://doi.org/10.1111/bjet.12197
9. Peffer, G., Elferink, R., Kämäräinen, P. (2018). Exploitation - the Journey From Early Ap-
proaches to Mature Concepts, http://results.learning-layers.eu/methods/exploitation-jour-
neys/
10. Ruiz-Calleja, A., Dennerlein, S., Kowald, D., Theiler, D., Lex, E., & Ley, T. (2019). An
Infrastructure for Workplace Learning Analytics: Tracing Knowledge Creation with the So-
cial Semantic Server. Journal of Learning Analytics, 6(2), 120–139.
https://doi.org/10.18608/jla.2019.62.9
11. Thalmann, S., Ley, T., Maier, R., Treasure-Jones, T., Sargianni, C., & Manhart, M. (2018).
Evaluation at Scale: An Approach to Evaluate Technology for Informal Workplace Learning
Across Contexts. International Journal of Technology Enhanced Learning, 10(4), 289–308.
https://doi.org/10.1504/IJTEL.2018.10013211
12. Treasure-Jones, T., Elferink, R., Martinez, G. & Peffer, G. (2018). Creating a place for learn-
ing – Bridging physical and virtual learning spaces at conferences. In European Conference
on Technology Enhanced Learning (EC-TEL) Practitioner Proceedings 2018, Leeds, UK.
http://ceur-ws.org/Vol-2193/
13. Treasure-Jones, T., Sarigianni, C., Maier, R., Santos, P., & Dewey, R. (2019). Scaffolded
contributions, active meetings and scaled engagement: How technology shapes informal
learning practices in healthcare SME networks. Computers in Human Behavior, in press.
https://doi.org/10.1016/J.CHB.2018.12.039
14. van den Akker, W., & Spaapen, J. (2017). Productive Interactions: societal impact of aca-
demic research in the knowledge society. LERU position paper. Retrieved from
https://www.leru.org/files/Productive-Interactions-Societal-Impact-of-Academic-Re-
search-in-the-Knowledge-Society-Full-paper.pdf
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).