=Paper=
{{Paper
|id=Vol-3292/paper04
|storemode=property
|title=Towards a multi-faceted framework for planning and evaluating innovation in Engineering Education
|pdfUrl=https://ceur-ws.org/Vol-3292/DCECTEL2022_paper04.pdf
|volume=Vol-3292
|authors=Erna Engelbrecht,Remon Rooij,Marcus Specht
|dblpUrl=https://dblp.org/rec/conf/ectel/EngelbrechtRS22
}}
==Towards a multi-faceted framework for planning and evaluating innovation in Engineering Education==
https://ceur-ws.org/Vol-3292/DCECTEL2022_paper04.pdf
Towards a multi-faceted framework for planning and evaluating
innovation in Engineering Education
Erna Engelbrecht , Remon M. Rooij and Marcus M. Specht
Delft University of Technology, Mekelweg 5, Delft, 2628 CD, The Netherlands
Abstract
For universities, educational change at institutional level is a slow process [1], [2]. To keep up
with societal and technological advancement, education innovation project leaders at
universities need practical guidelines and procedures in place that will enable sustainable and
scalable innovation that can meet the needs of industry as we transition from Industry 4.0 to
Industry 5.0 [3]. To develop such guidelines and procedures, we need to conduct socially
responsible, evidence-based educational research [4]. This paper is part of a larger study during
which we will conceptualize the planning and evaluation of innovation in engineering education
at the Delft University of Technology (TU Delft). From this conceptualization, a framework for
planning and evaluation of education innovation will emerge. The data collection process will
take place in six phases: (1) Exploration of the problem (2) feasibility studies;
(3) conceptualization and development of the framework; (4) piloting of the framework and its
associated processes; (5) field study; and lastly, (6) evaluation of the framework. This paper
provides an initial overview of the literature, as well as an explanation of the proposed research
methodology.
Keywords 1
Innovation, Higher Education, engineering education, research methodology, concept mapping
1. Introduction work in an unpredictable and unstable world
[8].
There is a need for socially responsible,
The COVID pandemic, conflict with world
evidence-based educational research [4] to
powers, the consequent fast tracking of energy
produce practical guidelines and appropriate
transition, and the exponential advancement of
measurement instruments that can support
technology brings about novel problems that
sustainable innovation in engineering education
need novel solutions. As a consequence,
that meet the needs of future graduates and an
education is in need of transformation [3], [5],
ever-changing society [9]–[11].
[6]. Universities of technology are responsible
In this paper we describe the initial plan for
for the education of engineers who need to be
a research initiative during which we will
equipped with holistic skill sets for dealing with
develop a multifaceted innovation framework
an increasingly unpredictable future.
that can guide the planning and evaluation of
Unfortunately, universities are slow to
innovation initiatives in Higher Engineering
change [1], [2] and innovations are often short-
Education (HEE). This framework will serve
lived [7]. Consequently, time and money is
project teams and individuals at all levels,
spent with little to no impact, while graduates
including educators, educational support staff
may find themselves insufficiently prepared to
and management. It is envisioned that this
Proceedings of the Doctoral Consortium of Seventeenth European
Conference on Technology Enhanced Learning, September 12–16,
2022, Toulouse, France
EMAIL: e.engelbrecht@tudelft.nl (A. 1); r.m.rooij@tudelft.nl (A.
2); m.m.specht@tudelft.nl (A. 3)
ORCID: 0000-0001-8731-347X (A. 1); 0000-0002-2784-1892 (A.
2); 0000-0002-6086-8480 (A. 3)
© 2022 Copyright for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
EC-TEL 2022: Doctoral Consortium (EA-TEL.EU)
framework would help to align, for example, its discussed. We then look at a number of existing
users’ goals, expectations, resource allocation frameworks for innovation and the evaluation
and communication flows. thereof, before positioning the current study.
The purpose of this endeavor is to facilitate
the feasibility, impact and sustainability of 2.1 Why innovate?
innovations in engineering education. To this
end, the following research questions will be
addressed: There are various definitions of innovation
1. How can we define the contextual discussed in detail in the literature [12]–[14].
characteristics that influence For the purpose of this study, however,
innovation in HEE? education innovation will be defined as: Any
2. How can we conceptualize the planning change that significantly increases the impact
and evaluation of innovation in on education processes.
engineering education? This initial definition will be further
3. To what extent can this informed and refined as the research project
conceptualization be applied to ensure develops. Currently, the definition is
feasibility, sustainability and impact of purposefully open to interpretation to allow for
education innovation that aligns HEE flexibility and freedom for exploration until a
with the needs of society and industry? more comprehensive definition emerges.
Each research question will be addressed Why is innovation in engineering education
during the different phases of a larger research needed? The world is changing fast due to
project. The research questions will be refined societal and technological developments, and
after a more in-depth literature review has been HEE needs to keep up the pace. Some authors
conducted. [15]–[17] argue that a new type of engineering
graduate is needed for taking on global
problems in an unpredictable and probably
unstable future [8] as we transition to Industry
2. Theoretical background 5.0 [3], [18], [19]. There are more works
providing a lengthier discussion on this matter
This study is initiated at a time when a [8], [13], [20], however, we will briefly touch
global pandemic, conflict with world leaders, on it here as well. This is not to say that we can
energy transition and data privacy is predict the future to determine with accuracy
dominating Western media. The question is what skills our (future) graduates will need – we
whether or not continuation of our current can only make educated guesses.
education system will suffice in preparing our The literature speculates, for example, on
engineering students for such an unpredictable the significance of automation, the Internet-of-
and insecure future. For example, the COVID Things, Artificial Intelligence, and big data
pandemic led to a shift in how many companies [21]–[23].
do business, and pushed industry and education In addition to technological developments,
towards online and hybrid methods. At the there are also growing concerns of global
same time emergency energy transition plans problems such as data privacy, climate change,
are being developed as a consequence of the pollution, food insecurity and a need for energy
conflict in Eastern Europe. transition. Our ‘educated guessing’ could
What kind of engineering professionals do therefore focus on tasks that cannot (yet) be
we need in such a rapidly changing world? performed by machines, or tasks performed in
What kind of curriculum agility do we need in collaboration with machines that require human
these kinds of circumstances? Does the intervention, for example, critical thinking and
engineering education community need to wait ethical decision-making.
for the next crisis for large scale innovation and Furthermore, our graduates will also need
fundamental changes to take place? durable skills such as digital literacy, analytical
This review of the literature first provides a thinking, resilience and problem-solving [3],
brief introduction to why innovation in [6], [18].
engineering education is needed. Next, the Education innovation not only happens top-
facilitation of innovation and the consequences down (instruction from institutional and faculty
of unguided, unsupported innovation is managers, program leaders, lawmakers and
policy makers), but also takes place bottom-up. with the service provider, and produce
These innovations are often driven by educators guidelines for best practices.
or course teams, student feedback, changes in At the time of writing, there were plans for
the field (and consequent updating of course eventually phasing out many of these
content), funding (or lack thereof) and/or ‘emergency online education’ tools and
increase in student numbers. Such innovations replacing them with policy compliant
tend to be introduced incrementally, which alternatives. In hindsight, what was needed was
might lead to loss of coherence within the a framework for educators and support
program [1]. personnel to evaluate the feasibility and
To keep programs up to date, course content, suitability of the tool; guidance for good
curricula and teaching methods need practices during usage; and eventually making
coordinated renewal strategies. In fact, not only informed decisions by evaluating how it was
do we need renewal, but more fundamental used, its impact, and to determine how to go
transformation is needed to ensure coherence in forward. Addressing this need will be the main
curricula that equips our graduates with the objective of this study.
skills needed to face our (rapidly changing) real The intention here would not be to create an
world problems. additional hurdle, but rather to equip
practitioners with a framework for making
2.2 Facilitation of innovation better decisions that are more sustainable in the
long run in all aspects of the education process.
The framework should open communication
At the start of the pandemic we found lines between various levels of stakeholders to
ourselves in an emergency situation where we ensure feasibility, impact, sustainability, and
were forced to find alternative methods for dissemination of education innovations in the
conducting everyday business. Many educators engineering domain.
hastened to get their courses online, while
others were more reluctant to adapt, hoping that
life would get back to normal soon. During this 2.3 Scoping existing education
time, institutions were forced to adjust and innovation evaluation frameworks
innovate quickly. At TU Delft, pockets of
innovation initiatives became more visible as To position this research initiative in the
practitioners were trying to find alternatives research field, an initial literature search was
and reaching out for help. However, most of done using Google Scholar. This was chosen to
these initiatives were somewhat painful, get a general idea of what is already available
uncoordinated, and sporadic at best, since there on this topic. Once the research project has been
was no emergency plan in place. approved, a more rigorous search will be
Educators who have been teaching using the conducted, as described further on in Research
blended course format seemed to have adapted methodology in section 3.
more quickly to the situation than those who In this section we will provide a brief
were newer to online education [24]. The introduction to five evaluation frameworks.
authors go on to explain that centralized support The overview will identify similarities and
initiatives were emerging, and as the pandemic differences in the elements which the
progressed, an increasing amount of frameworks consist of, as well as any patterns
cooperation and exchange of information was that might emerge.
observed. Unfortunately, communication
thereof did not always seem to reach those who By investigating formative, summative and
needed it [24]. illuminative evaluation goals, a 10-step process
One example of this is the large number of model was proposed [25] which defines the
educators opting to use Zoom for presenting stages in the process of evaluating education
their lectures online, despite it neither having innovations. According to this model, both the
been an approved, nor centrally supported at academic context and the governing policies
TU Delft. In fact, the sheer number of Zoom need to be taken consideration in the first stage,
users was so overwhelming that the university as these can have a ‘significant impact on
was forced to negotiate licensing agreements innovative practices’.
When defining the academic context, the framework under development in that it
author included the curriculum, the teaching addresses the following four dimensions:
processes, and learning. In terms of policy, both anticipation (being in touch with social and
policies at institutional level, as well as policies technological change), reflexivity (adjusting
that govern the tertiary education sector were behavior based on past experiences), inclusion
taken into account. This initial step of defining (involving a wider circle of contributors), and
the context and policy framework is then responsiveness (adapting in response to
followed by defining the goals of the changing circumstances). These dimensions
evaluation; identification of stakeholders; align with the underpinning reasons for the
aspects of the innovation and criteria for need for innovation, discussed earlier in this
evaluation; data collection and analysis; as well review, and according to the authors, have
and dissemination of the findings. emerged from public debate on new
Another process-based framework [2] maps developments in science and technology.
out the process of innovation in Higher [15] developed the Course Innovation
Education, and includes the following: Framework (CIF) with which to analyze
Identifying the current stage of the multiple aspects of course innovation. Aimed at
innovation implementation process and policy makers and educators, this framework
associated challenges. The stages are (1) provides input for analyzing, mapping out and
recognition of need, (2) planning, (3) making decisions on course innovations. Using
initiating, and (4) institutionalization. Curriculum Development Theory [28] as part
Determining the aim, type, nature and of the conceptual foundation, the intended,
measures to institutionalize the innovation. implemented and attained forms of innovation
Identifying the innovation itself, the were taken into consideration. Within the CIF
problem it addresses, and the people framework, different stages of the course
involved in the innovation activity. innovation life cycle, as well as different
Evaluating the learning curve and processes of innovation are considered.
adjusting aims and methods for Furthermore, the framework is both informed
institutionalization. by the literature and policy (top-down), as well
Analyzing potential factors that might as practice and interviews (bottom-up).
affect institutionalization of an innovation
(opportunity, compatibility and agency). From this brief discussion, the following
This framework provides a very useful preliminary conclusions can be drawn:
insight on the complexity and instructiveness of The impact on student learning should
the innovation process itself. By taking these be one of the main aspects of a framework,
elements into account, the framework can as it gives an important indication of the
provide a starting point for identifying elements impact of the innovation;
for consideration to minimize potential pitfalls Stakeholders should be another key
that could hinder dissemination of innovations. element – not only the students, but the
[26] attempted to develop a more educators themselves, and management.
contextualized evaluation methodology. The institutional context and the
Although the framework was developed with policies that apply to it can have
the purpose of evaluating courses, instead of implications for the dissemination process
innovations in education, it is worth looking at of innovations; and
the framework to inform the evaluation Education innovations should serve a
(application) process of the framework under specific purpose. More strongly put, it
development in the current study. The should solve a specific problem. The
framework includes the following aspects: framework should help to conceptualize the
purpose (of the evaluation), content (what to problem and how it can be solved.
evaluate), usage (by whom the analysis will be Based on these points we can already identify
done and how the results will be shared), and important elements that will define ‘innovation’
method (when and how evaluations should be in this study. Besides, of course, it being novel,
done). it should have a (positive) impact on its
[27] developed a framework that serves to stakeholders, be compliant with policy
ensure responsible innovation. It informs the
requirements and be fit for purpose by solving 3. Research methodology
some or other problem.
To address the research questions, the data
2.4 Positioning the framework to be collection for this sequential mixed methods
developed during this study study will be done in six phases. The following
table summarizes the phases that will be
undertaken in the current study:
Although many authors have investigated
innovation evaluation and evaluation
frameworks in the past, each of them was Table 1
conducted within their unique institutional and Research phases, based on [29]
educational contexts. It could be assumed that Phases Description
the discussion on evaluation frameworks for Phase 1:
HEE will continue to evolve organically as the Systematized literature
Exploration of
world changes and education follows suit. The review, PRISMA
the problem
present study aims to contribute to this Analysis of innovation
through
evolution, specifically in the light of global project documentation
secondary
challenges that urgently need to be considered data collection
in the renewal and development processes in
Testing initial framework
engineering education.
design
Both top-down and bottom-up innovation
can flourish when managerial support is in Phase 2: Interview project leaders
place and open communication lines are Feasibility for feedback
maintained. If not, innovation initiatives are study Reflection, and
stifled, making it more difficult (and costly) to implementation of
bring about change. The intended evaluation improvements
framework aims to contribute in that regard: Phase 3:
Group concept mapping
increasing the autonomy and impact of all Primary data
in collaboration with
levels of innovative project leaders, ensuring collection and
project leaders
that their innovations contribute to the shared analysis; and
goals of the degree program and/or institution. Development of initial
development
Therefore, the framework to be developed framework
of intervention
should be comprehensive enough to serve as a Piloting framework
multi-stakeholder instrument that can be Interview/focus group
applied firstly as a forecasting tool to determine discussions with project
education innovations’ potential, feasibility and Phase 4:
leaders/project groups
fit within the institutional context and assist in Prototyping
Reflection and implement
the planning and design phases; secondly to
inform the implementation process; and thirdly improvements after each
for the assessment of those innovations in terms iteration
of impact, sustainability, and dissemination. Apply framework to
In addition to this, this study aims to innovation initiatives – at
contribute to the discussion on fundamental least 1 x before, 1 x during
changes needed in engineering education . In an and 1 x after
attempt to accomplish this, the framework will Phase 5: Field implementation of
be developed in collaboration with various study innovation
engineering education innovation project Reflection and
leaders. This will be done by building on implementation of
existing innovation initiatives of educators, and improvements after each
in turn, support with dissemination of their iteration
work. Ultimately, a consolidated, multi- Phase 6:
stakeholder framework will emerge that can be Feedback and Evaluation of framework
applied widely across the institution, aligning
reflection
innovation practice bilaterally.
During the first phase, the problem itself and During Phases 3 – 5, the research
its context will be explored. participants will consist of the project leaders
This phase aims to address the first research from innovation initiatives at TU Delft. Project
question: leaders can include Educators, Educational
1. How can we define the contextual Advisors and Managers from the eight TU Delft
characteristics that influence innovation in faculties and the department of Teaching and
HEE? Learning Services (TLS) at TU Delft. The
A systematized literature review will be selection of education innovations which the
conducted for an in-depth theoretical participants are involved in will be made to
understanding of the context within which include, but are not limited to, for example,
innovation in engineering education should education technology, teaching methodology,
take place. Considering the advancement of learning environments, and course content.
technology and developments in society at During Phase 1 of the study, a list will be
large, education needs to be updated to be able drafted of participants to include, from which
to meet the demand of skills and knowledge they will be selected. During the selection
needed in the future, as discussed earlier. process, the optimal number of participants will
The systematized method for literature be decided on to get a fair demographic
review will be followed to ensure academic representation of participants, their innovation
rigor similar to a systematic review, while initiatives and the phases they are in.
allowing for some flexibility to complete the Phases 4 – 6 will focus on the third research
review in good time. In fact, a systematized question:
review is recommended for post-graduate 3. To what extent can this
research [30]. conceptualization be applied to ensure
During Phase 2 we will conduct two feasibility, sustainability and impact of
feasibility studies. First, we will test the education innovation that aligns HEE with the
primary data collection process that will take needs of society and industry?
place in Phase 3. After Phase 3 (development of This leads us to Phase 4, where application
the framework) has been completed, another of the evaluation framework will be piloted on
feasibility study will be conducted to test the a small scale on education innovation cases to
implementation process and usability of the test for feasibility, applicability and impact of
framework itself (in effect extending Phase 2 the framework. This will be followed by focus
beyond Phase 3). Improvements will be made groups/interviews involving project leaders and
by reflecting on how the process went, and peers for the purpose of feedback and reflection
based on interviews with participants of the for improvement, before continuing onto the
feasibility studies. next phase. The data will be analyzed, based on
During Phase 3 the primary, mixed method which preliminary conclusions can be drawn.
data will be collected by means of Concept Then, during the fifth phase, the field study
Mapping [31]. Here, project leaders will be will be carried out by applying the framework
guided through a brainstorming session to to education innovation initiatives. Innovations
generate ideas on how the planning and for this study will be chosen based on the
evaluation of innovations should be conducted. phases that they are in – before, during, and
These ideas will then be analyzed by means of after implementation.
a cluster analysis and multidimensional scaling For Phases 4 – 5, at least three iterations will
to sort, rank and map the ideas. Use of this be done, starting with simpler innovations with
technique enables the researcher to fill gaps a small scope, and then scaling up to larger
where knowledge is incomplete or uncertain by innovation initiatives. The size and scope of the
collecting information which a group of experts initiatives will be determined relative to each
have reached consensus on [31]. other and can be as simple as, for example
Based on this conceptualization, a (hypothetically speaking), using a new tool for
framework for education innovation will be a single activity vs. migration to a new learning
developed. Phase 3, therefore, will aim to management system.
address the second research question: Lastly, Phase 6 will follow, where the
2. How can we conceptualize the planning framework will be evaluated by means of
and evaluation of innovation in engineering questionnaires. The questionnaires will be sent
education? to project leaders and other stakeholders to
evaluate the usefulness, impact (internal and Data collection, analysis and
external), and validity of the framework. discussion of results;
Project leaders as well as Comenius and Literature review on innovation
Education Fellows from the 4TU (four frameworks and comparison with own
Universities of Technology in the Netherlands) intervention;
will be included during Phase 6. The evaluation Application of the intervention
process will be done for all three stages of developed, and discussion of feedback
innovation projects – before, during and after received on its application; and
implementation. Evaluation of intervention and
This process will be repeated until the discussion of final results of the study.
framework is sufficiently validated. Furthermore, cross-departmental sessions
Any problems experienced, or points for will be held to share progress and new insights
improvement during iterations, will be dealt with Teaching and Learning Services (TLS) at
with before moving on to the next iteration. TU Delft. Lastly, workshops will be provided
Additional iterations will be added if it is found to other PhD candidates on lessons learned
that three iterations are insufficient to draw during the research process.
strong conclusions, or if an iteration has failed
for some reason or another.
By combining qualitative and quantitative 6. Conclusion
data, a holistic view of the feasibility, impact,
sustainability, and dissemination of innovations This study will attempt to conceptualize the
that are guided by the evaluation framework process and evaluation of innovation needed to
can be captured. As explained, this will be meet the demand of industry and society. This
conducted in iterations, with moments for conceptualization will serve project leaders of
reflection for improvement in-between phases. innovation initiatives both bilaterally and
during the planning and evaluation phases of
their innovation initiatives.
4. Ethical considerations and data By providing the right support, tools and
management processes in place for planning and evaluating
innovation, educators and teaching teams will
The research will not impact on human be more equipped to implement feasible,
subjects and there is no foreseen conflict of sustainable and meaningful educational change
interest or risk involved. A detailed data that will enable us to train holistically educated
management plan will be drawn up in engineers.
consultation with a TU Delft Data Steward. The
data management plan will detail how the data 7. Acknowledgements
will be indexed and made accessible, and
reusable. All data collected during this research
The research initiative is funded by the 4TU
initiative will be stored on a password protected
Centre for Engineering Education (CEE) and
database on the TU Delft server, as well as the
will be conducted in collaboration with Marcus
4TU.ResearchData2 repository for scientific
Specht as Supervisor and Remon Rooij as
research data in the Netherlands.
Promotor.
This Word template was created by
5. Dissemination of research Aleksandr Ometov, TAU, Finland. The
template is made available under a Creative
The research progress and results will be Commons License Attribution-ShareAlike 4.0
shared at conferences, journal publications, International (CC BY-SA 4.0).
poster presentations and workshops. The main
topics intended are as follows: 8. References
Literature review – innovation trends
and contexts, and the way forward [1] C. Bajada, P. Kandlbinder, and R.
Research methodology Trayler, ‘A general framework for cultivating
2
http://researchdata.4tu.nl
innovations in higher education curriculum’, vol. 1, no. 1, pp. 91–93, Jan. 2015, doi:
High. Educ. Res. Dev., vol. 38, no. 3, pp. 465– 10.1080/23735082.2015.1008120.
478, Apr. 2019, doi: [12] C. J. Fraser, G. Duignan, D. Stewart,
10.1080/07294360.2019.1572715. and A. Rodrigues, ‘Overt and Covert:
[2] Y. Cai, ‘From an Analytical Strategies for Building Employability Skills of
Framework for Understanding the Innovation Vocational Education Graduates’, J. Teach.
Process in Higher Education to an Emerging Learn. Grad. Employab., vol. 10, no. 1, pp.
Research Field of Innovations in Higher 157–172, 2019.
Education’, Rev. High. Educ., vol. 40, no. 4, pp. [13] F. E. H. M. Smulders, A. Kamp, and C.
585–616, 2017, doi: 10.1353/rhe.2017.0023. Fortin, ‘The CDIO framework and new
[3] S. Güğerçı̇ N, ‘How Employees Survive perspectives on technological innovation’,
In The Industry 5.0 Era: In-Demand Skills Of presented at the The 14th International CDIO
The Near Future’, Int. J. Discip. Econ. Adm. Conference, 2018. [Online]. Available:
Sci. Stud. IDEAstudies, vol. 7, no. 31, pp. 524– https://repository.tudelft.nl/islandora/object/uu
533, Jan. 2021, doi: 10.26728/ideas.452. id%3A27702436-f7d8-4cc9-ad09-
[4] T. C. Reeves, J. Herrington, and R. ebb3f9020d01
Oliver, ‘Design research: A socially [14] S. G. Vincent-Lancrin, G. Jacotin, J.
responsible approach to instructional Urgel, S. Kar, and C. González-Sancho,
technology research in higher education’, J. Measuring Innovation in Education: A Journey
Comput. High. Educ., vol. 16, no. 2, pp. 96– to the future. Paris: OECD Publishing, 2017.
115, Mar. 2005, doi: 10.1007/BF02961476. [15] V. C. Tassone, H. J. A. Biemans, P. den
[5] L. Guàrdia, D. Clougher, T. Anderson, Brok, and P. Runhaar, ‘Mapping course
and M. Maina, ‘IDEAS for Transforming innovation in higher education: a multi-faceted
Higher Education: An Overview of Ongoing analytical framework’, High. Educ. Res. Dev.,
Trends and Challenges’, Int. Rev. Res. Open pp. 1–15, Oct. 2021, doi:
Distrib. Learn., vol. 22, no. 2, pp. 166–184, Jan. 10.1080/07294360.2021.1985089.
2021, doi: 10.19173/irrodl.v22i2.5206. [16] P. den Brok, Cultivating the growth of
[6] S. Grabowska, S. Saniuk, and B. life-science graduates: on the role of
Gajdzik, ‘Industry 5.0: improving educational ecosystems. 2018. Accessed: Feb.
humanization and sustainability of Industry 18, 2022. [Online]. Available:
4.0’, Scientometrics, vol. 127, no. 6, pp. 3117– https://doi.org/10.18174/458920
3144, Jun. 2022, doi: 10.1007/s11192-022- [17] A. Kamp, Navigating the landscape of
04370-1. higher engineering education: coping with
[7] C. J. Bonk and D. A. Wiley, ‘Preface: decades of accelerating change ahead. 2020.
Reflections on the waves of emerging learning [18] J. Mitchell and D. Guile, ‘Fusion Skills
technologies’, Educ. Technol. Res. Dev., vol. and Industry 5.0: Conceptions and Challenges’,
68, no. 4, pp. 1595–1612, Aug. 2020, doi: in Insights Into Global Engineering Education
10.1007/s11423-020-09809-x. After the Birth of Industry 5.0, M.
[8] A. Kamp, Engineering Education in Bouezzeddine, Ed. IntechOpen, 2022. doi:
the Rapidly Changing World: Rethinking the 10.5772/intechopen.100096.
Vision for Higher engineering Education. TU [19] SOCIETY 5.0: a people-centric super-
Delft, Faculty of Aerospace Engineering, 2016. smart society. Place of publication not
[9] S. Brooks, S. Brown, D. Davis, and J. identified: SPRINGER Verlag, SINGAPOR,
Lebeau, ‘Adoption of Technological 2020.
Innovations: A Case Study of the ASSESS [20] R. Klaassen, M. Dijk, R. Hoope, D.
Website’, Adv. Eng. Educ., vol. 4, no. 1, 2014. Ceulemans, A. Kamp, and M. Van der Sanden,
[10] T. C. Reeves and L. Lin, ‘The research ‘A DESIGN-BASED VISION ON FUTURE
we have is not the research we need’, Educ. ROLES IN ENGINEERING’, 2018.
Technol. Res. Dev., vol. 68, no. 4, pp. 1991– [21] J. Grodotzki, T. R. Ortelt, and A. E.
2001, Aug. 2020, doi: 10.1007/s11423-020- Tekkaya, ‘Remote and Virtual Labs for
09811-3. Engineering Education 4.0’, Procedia Manuf.,
[11] T. C. Reeves and P. M. Reeves, vol. 26, pp. 1349–1360, 2018, doi:
‘Reorienting educational technology research 10.1016/j.promfg.2018.07.126.
from things to problems’, Learn. Res. Pract., [22] M. Hernandez-de-Menendez, C. A.
Escobar Díaz, and R. Morales-Menendez,
‘Engineering education for smart 4.0
technology: a review’, Int. J. Interact. Des.
Manuf. IJIDeM, vol. 14, no. 3, pp. 789–803,
Sep. 2020, doi: 10.1007/s12008-020-00672-x.
[23] R. A. Ramirez-Mendoza, R. Morales-
Menendez, H. Iqbal, and R. Parra-Saldivar,
‘Engineering Education 4.0: — proposal for a
new Curricula’, in 2018 IEEE Global
Engineering Education Conference
(EDUCON), Tenerife, Apr. 2018, pp. 1273–
1282. doi: 10.1109/EDUCON.2018.8363376.
[24] N. L. Wahls, GM. Dijkstra, and G. M.
Ouwehand, ‘Blending Your Education:
Lessons Learned During COVID’, 2022.
[Online]. Available:
https://digitelpro.eadtu.eu/images/Blending_Y
our_Education__Lessons_Learned_During_C
OVID.pdf
[25] C. Jacobs, ‘The Evaluation of
Educational Innovation’, Evaluation, vol. 6, no.
3, pp. 261–280, Jul. 2000, doi:
10.1177/13563890022209280.
[26] C. Nygaard and D. Z. Belluigi, ‘A
proposed methodology for contextualised
evaluation in higher education’, Assess. Eval.
High. Educ., vol. 36, no. 6, pp. 657–671, Oct.
2011, doi: 10.1080/02602931003650037.
[27] J. Stilgoe, R. Owen, and P.
Macnaghten, ‘Developing a framework for
responsible innovation’, Res. Policy, vol. 42,
no. 9, pp. 1568–1580, Nov. 2013, doi:
10.1016/j.respol.2013.05.008.
[28] J. van den Akker, ‘Curriculum
Perspectives: An Introduction’, in Curriculum
Landscapes and Trends, Dordrecht: Springer
Netherlands, 2004, pp. 1–10. doi: 10.1007/978-
94-017-1205-7_1.
[29] A. E. Kelly, R. A. Lesh, and J. Y. Baek,
Eds., Handbook of design research methods in
education: innovations in science, technology,
engineering, and mathematics learning and
teaching. New York : London: Routledge,
2008.
[30] M. J. Grant and A. Booth, ‘A typology
of reviews: an analysis of 14 review types and
associated methodologies: A typology of
reviews, Maria J. Grant & Andrew Booth’,
Health Inf. Libr. J., vol. 26, no. 2, pp. 91–108,
Jun. 2009, doi: 10.1111/j.1471-
1842.2009.00848.x.
[31] W. M. K. Trochim, ‘An introduction to
concept mapping for planning and evaluation’,
Eval. Program Plann., vol. 12, no. 1, pp. 1–16,
Jan. 1989, doi: 10.1016/0149-7189(89)90016-
5.