For universities, educational change at institutional level is a slow process [1], [2]. To keep up with societal and technological advancement, education innovation project leaders at universities need practical guidelines and procedures in place that will enable sustainable and scalable innovation that can meet the needs of industry as we transition from Industry 4.0 to Industry 5.0 [3]. To develop such guidelines and procedures, we need to conduct socially responsible, evidence-based educational research [4]. This paper is part of a larger study during which we will conceptualize the planning and evaluation of innovation in engineering education at the Delft University of Technology (TU Delft). From this conceptualization, a framework for planning and evaluation of education innovation will emerge. The data collection process will take place in six phases: (1) Exploration of the problem (2) feasibility studies; (3) conceptualization and development of the framework; (4) piloting of the framework and its associated processes; (5) field study; and lastly, (6) evaluation of the framework. This paper provides an initial overview of the literature, as well as an explanation of the proposed research methodology.
The COVID pandemic, conflict with world
powers, the consequent fast tracking of energy
transition, and the exponential advancement of
technology brings about novel problems that
need novel solutions. As a consequence,
education is in need of transformation [
Unfortunately, universities are slow to
change [
There is a need for socially responsible,
evidence-based educational research [
In this paper we describe the initial plan for a research initiative during which we will develop a multifaceted innovation framework that can guide the planning and evaluation of innovation initiatives in Higher Engineering Education (HEE). This framework will serve project teams and individuals at all levels, including educators, educational support staff and management. It is envisioned that this framework would help to align, for example, its users’ goals, expectations, resource allocation and communication flows.
The purpose of this endeavor is to facilitate the feasibility, impact and sustainability of innovations in engineering education. To this end, the following research questions will be addressed: 1. How can we define the contextual characteristics that influence innovation in HEE? 2. How can we conceptualize the planning and evaluation of innovation in engineering education? 3. To what extent can this conceptualization be applied to ensure feasibility, sustainability and impact of education innovation that aligns HEE with the needs of society and industry? Each research question will be addressed during the different phases of a larger research project. The research questions will be refined after a more in-depth literature review has been conducted.
This study is initiated at a time when a global pandemic, conflict with world leaders, energy transition and data privacy is dominating Western media. The question is whether or not continuation of our current education system will suffice in preparing our engineering students for such an unpredictable and insecure future. For example, the COVID pandemic led to a shift in how many companies do business, and pushed industry and education towards online and hybrid methods. At the same time emergency energy transition plans are being developed as a consequence of the conflict in Eastern Europe.
What kind of engineering professionals do we need in such a rapidly changing world? What kind of curriculum agility do we need in these kinds of circumstances? Does the engineering education community need to wait for the next crisis for large scale innovation and fundamental changes to take place?
This review of the literature first provides a brief introduction to why innovation in engineering education is needed. Next, the facilitation of innovation and the consequences of unguided, unsupported innovation is discussed. We then look at a number of existing frameworks for innovation and the evaluation thereof, before positioning the current study.
There are various definitions of innovation discussed in detail in the literature [12]–[14]. For the purpose of this study, however, education innovation will be defined as: Any change that significantly increases the impact on education processes.
This initial definition will be further informed and refined as the research project develops. Currently, the definition is purposefully open to interpretation to allow for flexibility and freedom for exploration until a more comprehensive definition emerges.
Why is innovation in engineering education
needed? The world is changing fast due to
societal and technological developments, and
HEE needs to keep up the pace. Some authors
[15]–[17] argue that a new type of engineering
graduate is needed for taking on global
problems in an unpredictable and probably
unstable future [8] as we transition to Industry
5.0 [
The literature speculates, for example, on the significance of automation, the Internet-ofThings, Artificial Intelligence, and big data [21]–[23].
In addition to technological developments, there are also growing concerns of global problems such as data privacy, climate change, pollution, food insecurity and a need for energy transition. Our ‘educated guessing’ could therefore focus on tasks that cannot (yet) be performed by machines, or tasks performed in collaboration with machines that require human intervention, for example, critical thinking and ethical decision-making.
Furthermore, our graduates will also need
durable skills such as digital literacy, analytical
thinking, resilience and problem-solving [
Education innovation not only happens
topdown (instruction from institutional and faculty
managers, program leaders, lawmakers and
policy makers), but also takes place bottom-up.
These innovations are often driven by educators
or course teams, student feedback, changes in
the field (and consequent updating of course
content), funding (or lack thereof) and/or
increase in student numbers. Such innovations
tend to be introduced incrementally, which
might lead to loss of coherence within the
program [
To keep programs up to date, course content, curricula and teaching methods need coordinated renewal strategies. In fact, not only do we need renewal, but more fundamental transformation is needed to ensure coherence in curricula that equips our graduates with the skills needed to face our (rapidly changing) real world problems.
At the start of the pandemic we found ourselves in an emergency situation where we were forced to find alternative methods for conducting everyday business. Many educators hastened to get their courses online, while others were more reluctant to adapt, hoping that life would get back to normal soon. During this time, institutions were forced to adjust and innovate quickly. At TU Delft, pockets of innovation initiatives became more visible as practitioners were trying to find alternatives and reaching out for help. However, most of these initiatives were somewhat painful, uncoordinated, and sporadic at best, since there was no emergency plan in place.
Educators who have been teaching using the blended course format seemed to have adapted more quickly to the situation than those who were newer to online education [24]. The authors go on to explain that centralized support initiatives were emerging, and as the pandemic progressed, an increasing amount of cooperation and exchange of information was observed. Unfortunately, communication thereof did not always seem to reach those who needed it [24].
One example of this is the large number of educators opting to use Zoom for presenting their lectures online, despite it neither having been an approved, nor centrally supported at TU Delft. In fact, the sheer number of Zoom users was so overwhelming that the university was forced to negotiate licensing agreements with the service provider, and produce guidelines for best practices.
At the time of writing, there were plans for eventually phasing out many of these ‘emergency online education’ tools and replacing them with policy compliant alternatives. In hindsight, what was needed was a framework for educators and support personnel to evaluate the feasibility and suitability of the tool; guidance for good practices during usage; and eventually making informed decisions by evaluating how it was used, its impact, and to determine how to go forward. Addressing this need will be the main objective of this study.
The intention here would not be to create an additional hurdle, but rather to equip practitioners with a framework for making better decisions that are more sustainable in the long run in all aspects of the education process. The framework should open communication lines between various levels of stakeholders to ensure feasibility, impact, sustainability, and dissemination of education innovations in the engineering domain.
To position this research initiative in the research field, an initial literature search was done using Google Scholar. This was chosen to get a general idea of what is already available on this topic. Once the research project has been approved, a more rigorous search will be conducted, as described further on in Research methodology in section 3.
In this section we will provide a brief introduction to five evaluation frameworks. The overview will identify similarities and differences in the elements which the frameworks consist of, as well as any patterns that might emerge.
By investigating formative, summative and illuminative evaluation goals, a 10-step process model was proposed [25] which defines the stages in the process of evaluating education innovations. According to this model, both the academic context and the governing policies need to be taken consideration in the first stage, as these can have a ‘significant impact on innovative practices’.
When defining the academic context, the author included the curriculum, the teaching processes, and learning. In terms of policy, both policies at institutional level, as well as policies that govern the tertiary education sector were taken into account. This initial step of defining the context and policy framework is then followed by defining the goals of the evaluation; identification of stakeholders; aspects of the innovation and criteria for evaluation; data collection and analysis; as well and dissemination of the findings.
Another process-based framework [2] maps out the process of innovation in Higher Education, and includes the following: Identifying the current stage of the innovation implementation process and associated challenges. The stages are (1) recognition of need, (2) planning, (3) initiating, and (4) institutionalization. Determining the aim, type, nature and measures to institutionalize the innovation. Identifying the innovation itself, the problem it addresses, and the people involved in the innovation activity. Evaluating the learning curve and adjusting aims and methods for institutionalization. Analyzing potential factors that might affect institutionalization of an innovation (opportunity, compatibility and agency).
This framework provides a very useful insight on the complexity and instructiveness of the innovation process itself. By taking these elements into account, the framework can provide a starting point for identifying elements for consideration to minimize potential pitfalls that could hinder dissemination of innovations.
[
[27] developed a framework that serves to ensure responsible innovation. It informs the framework under development in that it addresses the following four dimensions: anticipation (being in touch with social and technological change), reflexivity (adjusting behavior based on past experiences), inclusion (involving a wider circle of contributors), and responsiveness (adapting in response to changing circumstances). These dimensions align with the underpinning reasons for the need for innovation, discussed earlier in this review, and according to the authors, have emerged from public debate on new developments in science and technology.
[15] developed the Course Innovation Framework (CIF) with which to analyze multiple aspects of course innovation. Aimed at policy makers and educators, this framework provides input for analyzing, mapping out and making decisions on course innovations. Using Curriculum Development Theory [28] as part of the conceptual foundation, the intended, implemented and attained forms of innovation were taken into consideration. Within the CIF framework, different stages of the course innovation life cycle, as well as different processes of innovation are considered. Furthermore, the framework is both informed by the literature and policy (top-down), as well as practice and interviews (bottom-up).
From this brief discussion, the following preliminary conclusions can be drawn: The impact on student learning should be one of the main aspects of a framework, as it gives an important indication of the impact of the innovation; Stakeholders should be another key element – not only the students, but the educators themselves, and management. The institutional context and the policies that apply to it can have implications for the dissemination process of innovations; and Education innovations should serve a specific purpose. More strongly put, it should solve a specific problem. The framework should help to conceptualize the problem and how it can be solved.
Based on these points we can already identify important elements that will define ‘innovation’ in this study. Besides, of course, it being novel, it should have a (positive) impact on its stakeholders, be compliant with policy requirements and be fit for purpose by solving some or other problem.
Although many authors have investigated innovation evaluation and evaluation frameworks in the past, each of them was conducted within their unique institutional and educational contexts. It could be assumed that the discussion on evaluation frameworks for HEE will continue to evolve organically as the world changes and education follows suit. The present study aims to contribute to this evolution, specifically in the light of global challenges that urgently need to be considered in the renewal and development processes in engineering education.
Both top-down and bottom-up innovation can flourish when managerial support is in place and open communication lines are maintained. If not, innovation initiatives are stifled, making it more difficult (and costly) to bring about change. The intended evaluation framework aims to contribute in that regard: increasing the autonomy and impact of all levels of innovative project leaders, ensuring that their innovations contribute to the shared goals of the degree program and/or institution.
Therefore, the framework to be developed should be comprehensive enough to serve as a multi-stakeholder instrument that can be applied firstly as a forecasting tool to determine education innovations’ potential, feasibility and fit within the institutional context and assist in the planning and design phases; secondly to inform the implementation process; and thirdly for the assessment of those innovations in terms of impact, sustainability, and dissemination.
In addition to this, this study aims to contribute to the discussion on fundamental changes needed in engineering education . In an attempt to accomplish this, the framework will be developed in collaboration with various engineering education innovation project leaders. This will be done by building on existing innovation initiatives of educators, and in turn, support with dissemination of their work. Ultimately, a consolidated, multistakeholder framework will emerge that can be applied widely across the institution, aligning innovation practice bilaterally.
To address the research questions, the data collection for this sequential mixed methods study will be done in six phases. The following table summarizes the phases that will be undertaken in the current study:
During the first phase, the problem itself and its context will be explored.
This phase aims to address the first research question:
1. How can we define the contextual characteristics that influence innovation in HEE?
A systematized literature review will be conducted for an in-depth theoretical understanding of the context within which innovation in engineering education should take place. Considering the advancement of technology and developments in society at large, education needs to be updated to be able to meet the demand of skills and knowledge needed in the future, as discussed earlier.
The systematized method for literature review will be followed to ensure academic rigor similar to a systematic review, while allowing for some flexibility to complete the review in good time. In fact, a systematized review is recommended for post-graduate research [30].
During Phase 2 we will conduct two feasibility studies. First, we will test the primary data collection process that will take place in Phase 3. After Phase 3 (development of the framework) has been completed, another feasibility study will be conducted to test the implementation process and usability of the framework itself (in effect extending Phase 2 beyond Phase 3). Improvements will be made by reflecting on how the process went, and based on interviews with participants of the feasibility studies.
During Phase 3 the primary, mixed method data will be collected by means of Concept Mapping [31]. Here, project leaders will be guided through a brainstorming session to generate ideas on how the planning and evaluation of innovations should be conducted. These ideas will then be analyzed by means of a cluster analysis and multidimensional scaling to sort, rank and map the ideas. Use of this technique enables the researcher to fill gaps where knowledge is incomplete or uncertain by collecting information which a group of experts have reached consensus on [31].
Based on this conceptualization, a framework for education innovation will be developed. Phase 3, therefore, will aim to address the second research question:
2. How can we conceptualize the planning and evaluation of innovation in engineering education?
During Phases 3 – 5, the research participants will consist of the project leaders from innovation initiatives at TU Delft. Project leaders can include Educators, Educational Advisors and Managers from the eight TU Delft faculties and the department of Teaching and Learning Services (TLS) at TU Delft. The selection of education innovations which the participants are involved in will be made to include, but are not limited to, for example, education technology, teaching methodology, learning environments, and course content. During Phase 1 of the study, a list will be drafted of participants to include, from which they will be selected. During the selection process, the optimal number of participants will be decided on to get a fair demographic representation of participants, their innovation initiatives and the phases they are in.
Phases 4 – 6 will focus on the third research question:
3. To what extent can this conceptualization be applied to ensure feasibility, sustainability and impact of education innovation that aligns HEE with the needs of society and industry?
This leads us to Phase 4, where application of the evaluation framework will be piloted on a small scale on education innovation cases to test for feasibility, applicability and impact of the framework. This will be followed by focus groups/interviews involving project leaders and peers for the purpose of feedback and reflection for improvement, before continuing onto the next phase. The data will be analyzed, based on which preliminary conclusions can be drawn.
Then, during the fifth phase, the field study will be carried out by applying the framework to education innovation initiatives. Innovations for this study will be chosen based on the phases that they are in – before, during, and after implementation.
For Phases 4 – 5, at least three iterations will be done, starting with simpler innovations with a small scope, and then scaling up to larger innovation initiatives. The size and scope of the initiatives will be determined relative to each other and can be as simple as, for example (hypothetically speaking), using a new tool for a single activity vs. migration to a new learning management system.
Lastly, Phase 6 will follow, where the framework will be evaluated by means of questionnaires. The questionnaires will be sent to project leaders and other stakeholders to evaluate the usefulness, impact (internal and external), and validity of the framework. Project leaders as well as Comenius and Education Fellows from the 4TU (four Universities of Technology in the Netherlands) will be included during Phase 6. The evaluation process will be done for all three stages of innovation projects – before, during and after implementation.
This process will be repeated until the framework is sufficiently validated.
Any problems experienced, or points for improvement during iterations, will be dealt with before moving on to the next iteration. Additional iterations will be added if it is found that three iterations are insufficient to draw strong conclusions, or if an iteration has failed for some reason or another.
By combining qualitative and quantitative data, a holistic view of the feasibility, impact, sustainability, and dissemination of innovations that are guided by the evaluation framework can be captured. As explained, this will be conducted in iterations, with moments for reflection for improvement in-between phases.
The research will not impact on human subjects and there is no foreseen conflict of interest or risk involved. A detailed data management plan will be drawn up in consultation with a TU Delft Data Steward. The data management plan will detail how the data will be indexed and made accessible, and reusable. All data collected during this research initiative will be stored on a password protected database on the TU Delft server, as well as the 4TU.ResearchData2 repository for scientific research data in the Netherlands.
The research progress and results will be shared at conferences, journal publications, poster presentations and workshops. The main topics intended are as follows: Literature review – innovation trends and contexts, and the way forward Research methodology 2 http://researchdata.4tu.nl Data collection, analysis and discussion of results; Literature review on innovation frameworks and comparison with own intervention; Application of the intervention developed, and discussion of feedback received on its application; and Evaluation of intervention and discussion of final results of the study.
Furthermore, cross-departmental sessions will be held to share progress and new insights with Teaching and Learning Services (TLS) at TU Delft. Lastly, workshops will be provided to other PhD candidates on lessons learned during the research process.
This study will attempt to conceptualize the process and evaluation of innovation needed to meet the demand of industry and society. This conceptualization will serve project leaders of innovation initiatives both bilaterally and during the planning and evaluation phases of their innovation initiatives.
By providing the right support, tools and processes in place for planning and evaluating innovation, educators and teaching teams will be more equipped to implement feasible, sustainable and meaningful educational change that will enable us to train holistically educated engineers.
The research initiative is funded by the 4TU Centre for Engineering Education (CEE) and will be conducted in collaboration with Marcus Specht as Supervisor and Remon Rooij as Promotor.
This Word template was created by Aleksandr Ometov, TAU, Finland. The template is made available under a Creative Commons License Attribution-ShareAlike 4.0 International (CC BY-SA 4.0).
[
[23] R. A. Ramirez-Mendoza, R. MoralesMenendez, H. Iqbal, and R. Parra-Saldivar, ‘Engineering Education 4.0: — proposal for a new Curricula’, in 2018 IEEE Global Engineering Education Conference (EDUCON), Tenerife, Apr. 2018, pp. 1273– 1282. doi: 10.1109/EDUCON.2018.8363376.
[24] N. L. Wahls, GM. Dijkstra, and G. M. Ouwehand, ‘Blending Your Education: Lessons Learned During COVID’, 2022. [Online]. Available: https://digitelpro.eadtu.eu/images/Blending_Y our_Education__Lessons_Learned_During_C OVID.pdf
[25] C. Jacobs, ‘The Evaluation of Educational Innovation’, Evaluation, vol. 6, no. 3, pp. 261–280, Jul. 2000, doi: 10.1177/13563890022209280.
[
[27] J. Stilgoe, R. Owen, and P. Macnaghten, ‘Developing a framework for responsible innovation’, Res. Policy, vol. 42, no. 9, pp. 1568–1580, Nov. 2013, doi: 10.1016/j.respol.2013.05.008.
[28] J. van den Akker, ‘Curriculum Perspectives: An Introduction’, in Curriculum Landscapes and Trends, Dordrecht: Springer Netherlands, 2004, pp. 1–10. doi: 10.1007/97894-017-1205-7_1.
[29] A. E. Kelly, R. A. Lesh, and J. Y. Baek, Eds., Handbook of design research methods in education: innovations in science, technology, engineering, and mathematics learning and teaching. New York : London: Routledge, 2008.
[30] M. J. Grant and A. Booth, ‘A typology of reviews: an analysis of 14 review types and associated methodologies: A typology of reviews, Maria J. Grant & Andrew Booth’, Health Inf. Libr. J., vol. 26, no. 2, pp. 91–108, Jun. 2009, doi: 10.1111/j.14711842.2009.00848.x.
[31] W. M. K. Trochim, ‘An introduction to concept mapping for planning and evaluation’, Eval. Program Plann., vol. 12, no. 1, pp. 1–16, Jan. 1989, doi: 10.1016/0149-7189(89)900165.