Research in technology-enhanced Learning (TEL) is often criticized for not leading to sufficient and sustainable impact in practice. This is especially so for workplace learning technology where learning and the use of technology needs to be embedded in working practices. We report on Learning Layers, a TEL project on workplace learning that ran between 2012 and 2016. We have identified three cases where impact on practice is still visible today, as learning technology originally developed in the project is still in use, impacts learning, teaching and working practices in healthcare practices, construction training and in the application of e-posters in academic conferences. We use the cases to identify factors during the lifetime of the project that have facilitated sustainable take up of research results, namely a flexible and collaborative project roll out, an open research process, early focus on end-users and an inclusive process of planning for impact and exploitation. Taken together, these factors suggest to look at research projects as providing flexible and networked co-creation platforms for innovation.
to innovation. These exceptions can be especially found in the context of formal
instruction and in schools. For example, several intelligent tutoring systems have found
broad take-up in schools and have also led to commercial success. Some large-scale
learning platforms for learning in schools have been widely successful. An example is
the GOLAB inquiry learning platform that uses virtual labs for science education.
Finally, there are several platforms that promote teacher collaboration. The GOLAB
platform is an example, as is the eTwinning platform that connects thousands of teachers
throughout Europe, or less well known cases, such as LeMill [
In FP6 and FP7, the European Commission has funded several large-scale projects
(called Integrated Projects) which were meant to change this situation, and particularly
focused on technology for workplace learning [
With this paper, we are intending to shed light on the possible success factors of these R&D projects. We started from the following research question: what are the successful practices that have been integrated in these R&D projects to make long-term impact more likely?
Answering the research question is difficult for at least two reasons. First, answering
the question would require a larger impact analysis of TEL R&D projects that were
completed years ago (an analysis that is much beyond the scope of this paper).
Secondly, the question is difficult to answer because successful cases are actually difficult
to locate. While publishing is very active during the runtime of the project or shortly
after to describe effects of research on practice (e.g. [
We decided therefore to conduct a case study, and chose the Learning Layers1 project as our case. Despite having finished already in 2016, several partners are still in contact and could therefore share experiences of developments since then. 1 http://learning-layers.eu
For this paper, we have collected incidental evidence, rather than conducting a more systematic impact analysis. Evidence was collected through the analysis of three cases which are being undertaken by some of the application and technology partners of the original project. These activities are still ongoing, and can be clearly linked to some of the results produced during the project lifetime which have since then been significantly extended. The impact cases were analyzed to understand which of the practices undertaken in the project during its lifetime could have had a positive influence on the sustainability of those impact cases. 2.2
The Learning Layers R&D project was conducted between 2012 and 2016 with the
main objective of using learning technology to scale workplace learning particularly in
those sectors that had been less inclined to use technologies for learning. We addressed
learning in two of these sectors: to address the first sector, we considered workplace
learning of healthcare professionals working in general health care practices in the UK.
The second application was undertaken in the German construction industry, where we
especially focused on the learning of construction apprentices and on bridging learning
in a training center and at the workplace. Practitioners from these sectors were active
participants in the research, and the project followed a highly stakeholder-driven
codesign approach [
Technologies used in those cases were especially focused on mobile and social
technologies that allowed to contextualize learning in the work setting, and allow for
collaborative knowledge building. The set of technologies was complemented by several
backend services that allowed quick deployment, as well as data-driven technologies
that allowed tracing of some of the social learning processes [
Altogether, the project was implemented by 17 partners from 7 different countries over the course of 4 years. The project was integrative in the sense that it combined a number of related activities (including research studies, collaborative design, technological development, and conducting multiple evaluation, exploitation and outreach activities) into an overall approach and with a total effort of over 100 person years. The project was also set up in a very collaborative way, meaning that most partners were involved in many of the project activities. There were not many “hand-over” tasks, but rather tasks were running in parallel and were then synchronized at particular milestones (e.g. for the testing of a set of technologies in a particular setting).
From a scientific point of view, the project can be considered successful from typical
scientific standards: its outcomes have been documented in multiple scientific papers
(a recent review paper summarizes some of these [
While many of the deployments in practice that were happening within the more
confined contexts of the project were discontinued after the end of the project, it is
remarkable that some initiatives that were started through the project are still ongoing, even 4
years after the project has ended. Three of these cases that are described in this section
center around the development of Learning Toolbox (LTB) [
Problem addressed. In the training center, LTB was applied in the context of apprentices’ construction projects. The instructions for the apprentices to conduct these projects had been provided orally with the help of instructive worksheets (for preparing the project plans). Likewise, the reporting on the projects was done manually. With the help of the LTB, the typical structure of the projects, as self-organized learning tasks became transparent: apprentices had to search for resources, draft the plan, report the implementation and assess the outcome.
Stakeholders involved. The co-design work was carried out as a collaborative process by researchers, technical partners and full-time trainers from Bau-ABC. During an earlier phase of the work the project team provided basic multimedia training for some voluntary trainers. At a later phase the project team and these trainers provided an intensive training campaign for all trainers of the training center. In the pilot testing of the Learning Toolbox a core group of trainers introduced the toolset in their training and results were monitored by the project team. Also, at the final phase of the project, the use of LTB as support for construction work processes was demonstrated for several craft trade companies. As a result, follow-up processes (feasibility studies and project initiatives) were started with some companies.
Learning technology and its use. LTB allows trainers to easily customize contents and learning processes for different training purposes and according to the pedagogic 3 https://ltb.io/company/ priorities of the trainers. Thus, LTB made it easier for the trainers to emphasize independent searches in a wider range of web resources. This is essential for borehole builders, for example, who are working alone on remote construction sites. Also, it makes it possible to give learners gradual access to a wider range of resources (and to solutions of their peers) once they have learned to develop their own solutions to the project tasks. Moreover, LTB encourages apprentices to document their projects within the platform. This enables instructors to see progress of their apprentices in real time and provide more timely feedback. The LTB also encourages the self-organization of the instructors in terms of streamlining their content and sharing common resources between the different professions. While this approach to collaborative teaching was already there at Bau-ABC, the LTB offers a further channel to systematize this practice. Altogether, the co-design process, the piloting phase and the follow-up phase have been characterized by intensive R&D dialogue underpinned by the accompanying research approach of the research institute ITB at the University of Bremen.
Impact. After the Learning Layers project had ended, the use of LTB was encouraged across all trades in which Bau-ABC Rostrup provides apprentice training. Consequently, most of the apprentices’ projects are now being supported through LTB in some form. Based on this case, other German training centers in the construction sector are in the process of trialing LTB both for initial and continuing VET. Due to the closure of the training centers because of the COVID19 pandemic, the trainers of BauABC Rostrup have prepared trade-specific stacks with LTB to support independent learning. The trainers of Bau-ABC have become strong multipliers of innovation both within their organisation and in their networking with other training centers and partner companies. 3.2
Problem addressed. This case addresses the problem of over-use of antibiotics which leads to a rise in antimicrobial resistance. As part of a nationwide 5-year strategy to champion responsible use of antibiotics, the Bradford Districts Clinical Commissioning Group (CCG) which is part of the UK National Health Service (NHS) has run a series of audit and feedback projects over the past few years, which involve sending out practice-specific paper audit and feedback reports to all practices in West Yorkshire and Harrogate (approx. 300 general practices), covering the regional Integrated Care System (ICS). The latest project in this audit and feedback series focuses on antibiotic prescribing. One problem the CCG has faced with these projects is that the reports are sometimes not well shared to all appropriate staff within the practice, and may remain with a ‘gatekeeper’.
Stakeholders involved. Main stakeholder is the Bradford Districts CCG, specifically the West Yorkshire Research and Development team (a NHS R&D team for all West Yorkshire and Harrogate general practices) and the medicines optimization teams within the CCG. The program targeted general practices in West Yorkshire and Harrogate and the staff members within those practices. The case also draws on the LTB which was provided supported by Kubify, the start-up that had commercialized the technology. The University of Leeds has been involved in developing the audit and feedback program. Further parties were Public Health England and the local authority infection control teams.
Learning technology and its use. LTB has been used to create stacks for each practice and thereby improve the accessibility of the practice reports as well as to enable the sharing of additional resources which could not be included in the main report due to space. The app has thus improved the range of information that can be shared, and links are also shared which allow users to read more in-depth into the topic areas. The use of LTB has also enabled the spread of information more widely, as the team suggested that the stack poster (a paper-based poster displaying the link to the stack and a QR code) should be displayed in the practice to allow any interested staff to access the stack and resources. The use of the stack also allows for all the information to be kept by interested staff in one central place, so previous reports and resources can be referred back to at any point. It can also be accessed via a personal mobile device, so gives the opportunity for users to access the information at the most convenient time for them, and without the need to have the paper report or to log in to a system.
Impact. Anecdotal evidence has been collected about the use of the stacks and the benefit that users have found. Usage statistics indicate that some practices have had a number of accesses to their stack, indicating that multiple members of staff may have accessed. A systematic impact study of the use of LTB has not yet been conducted, but is currently planned at a later stage. During the Covid-19 pandemic, it was not possible to send out paper reports, so it can be anticipated that the stacks will become more important. We also hope to be able to show a positive impact on antibiotic prescribing over time, which aims to reduce antimicrobial resistance in the population. 3.3
Problem addressed. This use case addresses the broad problem of how conference
attendees can more effectively present, share and develop their knowledge during and
after conferences. However, it starts from a very specific problem that we refer to as
the “paper poster problem” [
Stakeholders involved. This use case was developed by three of the individuals involved in the Learning Layers project. The individuals developed the use case, piloted it and built the company (Kubify) after the end of the project. Looking at the use case more broadly, the other stakeholders are conference delegates, scientific associations, conference organizing companies and venues. In terms of the business model, it is usually the scientific association or conference organizing company that is the customer and the conference delegates who are the main users.
Learning Technology and its use. This use case involved adapting LTB so that it
could be used as an ePoster platform by conferences. It allows the ePoster author to
easily create an ePoster that is a collection of rich multimedia and interactive resources.
It allows the conference delegates to open and engage with these ePosters on their own
devices (mobile phones, laptops) rather than using hired screens. This interaction can
include exploring and interacting with the ePosters, engaging in a discussion attached
to the ePoster, adding content to the ePoster and sharing the ePoster with others. The
educational aim is to facilitate informal learning by supporting key activities such as
actively engaging with content (rather than passive consumption), discussion of
content, addition of new content (where the author allows this), sharing of content and
making connections with people with similar interests. A formal evaluation has not yet
been undertaken, but reviews of the ePosters created for the conferences show that
authors are taking up the opportunities to enhance their content. Typical ePosters now
include videos, links to surveys, data presentations that the viewer can actively explore,
research material that can be reused by others and some even allow viewers to add
‘missing’ content and ideas. Furthermore, the ePosters are still being accessed long
after the conference. The poster sessions themselves are also changing with the
introduction of more structure (as compared to the open drinks reception poster sessions) such
as poster pitches, round table poster sessions, ePoster BarCamps, reciprocal peer review
in the week prior to the conference and poster sessions with CPD credits awarded for
engagement with the poster content [
Impact. This case was developed after the Learning Layers project ended and the
first pilot was with a large, international medical education conference in 2017 [
Looking at the cases presented in the last section, we are presenting here some of the lessons we learned for how to organize and implement R&D projects for impactful TEL research. The lessons learned focus on those factors within the lifetime of the project, and not on all the steps that were necessary for the later successful commercialization. 4.1
The Learning Layers project was heavily built on contextual inquiry and collaborative design. We employed phases of contextual inquiry, and the main R&D work especially in the first years was undertaken by co-design teams composed of researchers, practitioners and technology developers. The co-design methodology followed standard practices of identifying practical problems, gaining an in-depth understanding of the context in which practitioners were working and exploring different design solutions. The process was supported by co-design platforms, open software development platforms and R&D methodologies by which partners worked together closely in an agile manner. Contact between partners was on weekly, sometimes even daily, timescales and included rich and challenging discussions to help partners to understand the complex learning contexts and the different perspectives in the different Learning Layers use cases. The design teams were not predetermined in the project plan, but flexibly formed in the first year, and also open to changes in their composition later on. The teams also undertook quite an effort to engage with a wider set of practitioners beyond those involved in the project.
This working mode helped sustain some of the activities beyond the project lifetime. In case 1, for example, the co-design process, the piloting phase and the follow-up phase have been similarly characterized by an intensive collaborative R&D dialogue that led to further adjustment of the tool development aligned with the pedagogic approach of the trainers.
While co-design methodologies may be standard for commercial projects, they are
much less common for research projects. Despite the fact that recent calls have been
made to include knowledge co-production methodologies to increase the impact of
academic research [
The flexibility described in the previous paragraph has a positive side-effect, namely that certain commitments made early in the design process may be questioned and revised at later stages. This is critical for sustained impact, as application circumstances and priorities change, and ongoing technological developments might make things possible that previously were not. A flexible process allows to take up opportunities as they arise. This might also lead to a situation where certain technologies are applied in contexts that were not originally foreseen.
As an example, consider case 2, the application of Learning Toolbox in the Health Care practices. In early design sessions in those practices, the use of mobile technology was not prioritized, as certain restrictions on mobile phone use were in place at the time, the majority of the staff were not using smart devices, and also the contextualization of learning in terms of physical location (which would speak for the use of mobiles) was not seen as a major challenge. So the use of mobile technology was mainly pursued in the construction domain (as case 1 testifies). However, as circumstances changed (i.e. mobile phones became more widely used and accepted in healthcare), and new opportunities arose (i.e. by having a mobile platform successfully tested in another context), these earlier decisions were revised. As case 3 shows, this extended to contexts that were not even seen as the original project application contexts at all.
This discussion also points to another important factor of the project, namely the project size, scale and length. Quite often, large and long-term projects as the current one are criticized for being overly complex and overburdened with administrative and coordination tasks. While this may be true, in the current case we also see some benefits of having a project with the capacity to address a problem from different sides and with different technological solutions. It also shows that if a project is successful in building a common understanding and trust, this can later facilitate the uptake of solutions, even if this involves commercial risks. 4.3
It follows from the previous two lessons learned that, due to the flexibility, some of the original project partners may lose interest, change their roles or new ones may enter. If we look at the cases, then we notice that this is precisely what happened with those initiatives that have been successful for a longer period of time. The final setup of partners that finally sustained the initiatives were not all part of the research project from the beginning. Rather, several partners were exchanged half way through the four-year project, and several other ones joined after the project had already ended. This possibility of including partners later in the process was critical and allowed us to take up emergent opportunities.
It should be noted that these kinds of partner changes were actually not foreseen as a regular process in how FP7 projects were set up. Rather they were treated as extraordinary changes that required a massive administrative overhead for changing contracts and the workplan. While it is clear that projects of this size carry a large responsibility in terms of financial risks that needs to be planned and justified, it is probably also true that flexibility could be better built into the funding programs to allow opportunities for sustainable impact to arise. 4.4
Obviously, a key factor that contributed to the sustainability of all three cases lies in the
fact that commercialization of the technology was finally successful. While it was
uncertain at the end of the project, how the innovations could be sustained and spread, an
approach to exploitation planning had been initiated in the project that put emphasis on
discovering initial viable solutions, building exploitation journeys and building teams
and business cases to realize them [
The commercialization of LTB was finally successful through considerable work, personal risk and investment undertaken by the 3 co-founders of Kubify after the Learning Layers project. Their work has led to the establishment of a viable, commercial company that has gained income and customer traction, and therefore has the capacity to maintain both the ePoster use case and also other use cases involving LTB.
An interesting aspect of this collaborative partnership is that the three individuals came from separate Learning Layers partners and different EU countries. It is very unlikely they would have met had it not been for the Learning Layers project. The three individuals each bring a slightly different perspective to the work – technical development, educational research, finance and project management – which together were critical for final commercialization.
The highly connected and collaborative nature of the Learning Layers project (as described under 4.1) may have increased the likelihood that some of those individuals (from different partners) would have started joint initiatives through the project, even if not initially foreseen in the project plan. And they would have then continued to stay in contact and work together beyond the project, having identified shared interests and seen the benefits of working together, particularly in an interdisciplinary and co-design manner. Another potential success factor for commercialization was that the community, built up during the Learning Layers project, gave the founding team access to advice and insights from others after the project as well. Yet another key factor was the fairly liberal licensing and intellectual property agreements that had been agreed at the outset. Software was licensed under open source licenses that permitted commercial use. 4.5
If to summarize these four lessons learned, we can see a common pattern emerging. It seems that R&D projects, especially ones of such scale, should not be understood as large research studies planned out in detail over four years, in case they are expected to produce sustained impact in practice. A better way of conceiving these would be in terms of networked innovation hubs or platforms in which research elements can receive initial validation, and a team is built and supported by a larger community to start an innovation process. This requires that trust is built, relationships are formed, and a common understanding of problems and possible solutions is created. From a technological point of view, this should be accompanied by an open approach to building the architecture instead of aiming for a tightly integrated solution. Open and agile methods increase the chance that innovative solutions are discovered that then have a viable chance for commercialization. 5
In this paper, we have analyzed three cases that have sustained some of the pedagogical and technological innovation for workplace learning that were initially created and tested in the Learning Layers R&D project. From this analysis, we propose that successful R&D project with regards to sustainable impact on practices should be conceived as networked and flexible innovation hubs in which initial research ideas and prototypes are developed and validated in practice. We have suggested a few R&D practices that are conducive to such approach, such as co-design and a highly collaborative and agile R&D methodology. If through these means sufficient trust, common understanding and a community has been created within the project, then it becomes much more likely that commercial opportunities can be sought quickly, individuals are ready to take resulting risk, and initiatives are sustained beyond the lifetime of the project.
Clearly, the analysis we have undertaken here is initial and to some extent speculative for several reasons. First, as we are considering a time frame of several years, a multitude of intervening factors certainly have played a role in determining which initiatives were sustained and for which reasons. Second, as our analysis was done by hindsight, we may have distorted the facts. And thirdly, as we have mainly focused the analysis of success factors on the period of the project lifetime, the success factors we have derived here are necessarily limited, and they can not do justice to all the work that was essential in making these cases into successful examples of TEL adoption.
In follow-up work, we will try to make a more systematic analysis by employing a more structured analysis approach and a larger set of respondents. In that work, we will also focus on the period after the project had ended in order to understand the dynamics that ensued after the successful completion of the project.
This research has received funding from the European Union’s FP7 research and innovation program under grant no. 318209 (Learning Layers, http://learning-layers.eu), and from Horizon 2020 under grant no. 669074 (CEITER, http://ceiter.tlu.ee).