The majority of the efforts in assessing educators' digital competence over the past decade have been focused on developing evidence-based and scientifically reliable assessment instruments. These instruments are often created ad-hoc by research groups without deeper understanding of the educators' needs and expected benefits for digital competence assessment. That implies that although the instrument might give valid and reliable results for the researchers it disregards all other related stakeholders - educators, school leaders, educational technologist, teacher trainers etc. To understand and guide evidence-informed decision-making when developing, adapting or implementing digital competence assessment instruments it is important to accommodate all stakeholders to provide meaningful assessment results and data. To provide a solution for this problem we have designed a trade-off model which focuses on mapping the digital competence assessment instruments to stakeholder needs and expected benefits. Our research is divided into three main phases. First, we focused on understanding the concept and domain of educators' digital competence. For which we analysed the existing educators' digital competence frameworks, models and similar previous mappings from the literature. Secondly, to explain the alternative digital competence assessment approaches and instruments we mapped the underlying assessment processes and piloted alternative instrument with different educator groups. The third and final phase focused on designing, developing and validating the trade-off model. The following describes all three phases and provides an overview of the initial findings which are accompanied with suggestions for further research in the field of educators' digital competence assessment.
Using technologies in teaching and learning is not considered a novel practice any more but rather presented as a norm for quality education. Innovative and pedagogically reasonable ways to implement technologies on the other hand has presented difficult among teachers and thus the discussion on educators’ digital competence has gained popularity. However, it is evident that not only mapping the needed digital competence of educators is needed but more importantly we need to understand the level of digital competence of educators to support meaningful professional development. Digital competence is considered as a goal oriented, confident and critical use of technologies for work, employability, learning, leisure and inclusive participation in society [1].
Educational assessment has been a central discussion for overall quality assurance in educational settings or trying to understand knowledge development [2]. Harlen & James [3] have stated that there are three general assessment approaches which also related to digital competence assessment - formative, summative and diagnostic assessment. Within these assessment approaches there is a variety of instruments, most notably self-assessment, knowledge-based tests and authentic assessment instruments like e-portfolios of reflective journals. It can be argued that for the past decade the efforts have mainly been towards developing self-assessment instruments which are cost-effective, mostly adaptable and cover variety of educators’ groups (i.e. primary to higher and vocational education). However, research done piloting and implementing these self-assessment instruments proposes a question whether educators assess their digital competence or something else entirely. Benali et al. [4] propose that majority of educators often assess their self-confidence in integrating technologies to their pedagogical practice and fail to give suitable evidence of their current practices. It is also considered that many digital competence assessment instruments which are based on self-assessment do not cover digital competence but rather focus on low-order cognitive skills [5], [6].
Previous research has also revealed that knowledge-based testing and authentic assessment requires higher volume of resources, both financial and human capital and is difficult to monitor [7].
Regardless the form of assessment and type of used instruments it is concluded that there is a sustainability issue which implies that there is a contradiction between the number of digital competence frameworks and models and the number of corresponding instruments.
Another dimension in educators’ digital competence assessment is the understanding of the related stakeholder groups who either require access to the assessment results or data. Adhering to these stakeholder group needs and expectations has proven to be a difficult task [8]. On one hand we lack a clear understanding of these stakeholder profiles but more importantly there is little research which describes the needs.
The doctoral research was done in three
phases implementing design-based research
methodology [
[RQ1] What are the implications and alternative approaches of assessing educators’ digital competence?
[RQ2] What are the stakeholder requirements and needs for educators’ digital competence assessment?
[RQ3] How are the alternative assessment approaches established and sustained? 2.1.
The doctoral research focuses on the
Estonian educational setting and educators.
Based on Lucas et al. [
Estonia operates in a decentralized
educational system which allows competition
between schools but also provides school and
educator autonomy [
The first phase of the research was to understand and delineate the concept and domain of educators’ digital competence and assessment. This phase was guided by the research question:
[RQ1] What are the implications and alternative approaches of assessing educators’ digital competence?
We carried out a systematic literature review
(SLR) [
Based on the analysis the SLR provided four key results which helped to better define the concept of educators’ digital competence. Additionally, the results provided the first insight to the implications related to the alternative assessment approaches and instrument.
First, the SLR confirmed that majority of the educators’ digital competence assessment related research focuses on quantitative studies by implementing self-assessment instruments and there is a clear lack of qualitative research to accompany the results to explain the reliability and validity of the instruments.
Secondly, used self-assessment instruments are created ad-hoc often based on country specific framework and targeted specific group of educators (i.e. in-service teachers, student teachers etc.).
Third and considerably most fundamental result revealed that self-assessment is often one-dimensional, meaning that there is relatively low possibility to understand and explain why and how educators approach digital competence self-assessment. To this end it is important to embed alternative assessment approaches like testing or authentic assessment – including portfolios, reflective journals and observations to understand educators’ perceptions of their competence and make sense of the evidence provided by the educators. Furthermore, alternative and combined competence assessment would potentially further the research if educators assess their digital competence r rather selfefficacy or self-confidence.
The final key result of the SLR presented the need for validated guidelines for the digital competence assessment processes. One of the proposed solutions was a large-scale participatory research which would focus on piloting alternative assessment instruments and approaches.
Based on the SLR results we concluded that
the future research lines included following the
DigCompEdu framework [
The second and most extensive phase of the
study focused on implementing alternative
digital competence assessment instruments
based on the DigCompEdu framework [
RQ1] What are the implications and alternative approaches of assessing educators’ digital competence?
[RQ2] What are the stakeholder requirements and needs for educators’ digital competence assessment?
While the main focus of this phase was to identify the implications of alternative approaches, the research done also gave input to the related stakeholder groups and the respective needs.
During this phase four studies were conducted which included self-assessment instruments, knowledge-based testing and eportfolio based digital competence assessment approaches. The focus of the four studies was the following:
Study 1 – In-service teachers’ perceptions of digital competence during distance learning period.
Study 2 – Comparative multiple-case study of three combined self-assessment and knowledge-based testing digital competence assessment approaches.
Study 3 – SELFIE4Teachers [
Study 4 – Competence based LMS2 focusing on e-portfolio based assessment of digital competence.
Table 1 describes the methodology, research instrument, samples and timeline of these studies.
Table 1 Second phase studies.
Methodology Instrument
Sample Study time
Study
1 Quan
SA 1125 2020
Study Study
2 3 Quan MM SA&KB SA&NGT 2248 18 2019- 2022 2021
Study
4 Qual Auth.
84 2022 SA – Self-assessment.
KB – Knowledge-based test.
NGT – Nominal Group Technique group interview.
Auth. – Authentic assessment using e-portfolio.
Main results of the four studies can be described in the following key ideas. First, when implementing self-assessment instruments, on average, educators assess their digital competence as average technology users. In some cases, this describes the educators’ inability of assessing their own competence and once again presents the question whether they assess digital competence or perceived self-confidence.
Second outcome of the studies revealed that
educators are unable to provide appropriate
evidence to describe their digital competence.
As always there are exceptions, but the main
issue lies in the fact that educators do not
differentiate the different digital competence
dimensions [
The third result describes the educators’ expectations towards the assessment instrument, stating that the used instruments often include hard to understand concepts and definitions. Simultaneously, the educators brought out issues with the instrument length, time spent on completion and the feedback report usability.
The final contribution of the four studies relates to the validity, reliability and sustainability of the used instruments. Based on the research we concluded that although there 2 https://edidaktikum.ee are a lot of efforts in designing and developing these assessment instruments they often lack in reliability. Additionally, as instrument validity is a multifaceted concept (i.e. face validity, construct validity etc.) it boils down to the stakeholder needs. The second phase of the doctoral research also confirmed that there is a continuous issue with digital competence assessment instrument sustainability where focus on re-designing and developing new instruments is considered of higher priority, rather than updating the excising instruments.
The third and final phase of the research focuses on identifying the stakeholder specific trade-offs in educators’ digital competence assessment, developing and validating the trade-off model. This phase followed two research questions:
[RQ2] What are the stakeholder requirements and needs for educators’ digital competence assessment?
[RQ3] How are the alternative assessment approaches established and sustained?
The third phase included two main studies where the first focused on identifying the stakeholder profiles (in-service teacher, student teacher, advanced teacher, teacher trainer, educational technologist, school leader, qualification examination assessment board member) and scenarios and on the stakeholder expectations and needs, resulting in the first version of the trade-off model. The study was a combined quantitative (N=1125) and qualitative (N=4) methodology.
The second and final study of the doctoral research included the validation of the stakeholder profiles and the trade-off model. The study was done following a Nominal Group Technique and included representatives of each stakeholder profile (N=6).
As this phase of the research is still underway the following describes initial outcomes. We consider noteworthy that all stakeholders consider the process of digital competence assessment valuable which helps to understand the professional development needs of educators. Furthermore, the inductive analysis of the differences in stakeholder needs gave us a clear indication that it is nearly impossible to provide a reliable and of high validity universal digital competence assessment instrument. This means that a tradeoff model could provide a solution to adhere to the stakeholder needs. The results also provide deeper understanding on the stakeholder specific scope and dimension of educators’ digital competence assessment expectations.
The doctoral research is currently in the final stages where our efforts are focused on publishing the results of finalized studies and formulating the analytical overview and main scientific contributions.
While digital competence assessment and more specifically educators’ digital competence has been an ongoing discussion and research topic for more than 15 years our research provides a new dimension to understanding the assessment instruments, approaches and processes. This doctoral research can be described a metalevel research which aims to describe and provide solutions for the digital competence assessment through multiple stakeholder lens rather than trying to provide one universal solution to a multifaceted research problem.
The doctoral research has received funding from: 1. The European Union’s Horizon 2020 research and innovation program under grant agreement No. 669074. Activity was supported through and according to Mobilitas Plus MOBEC001 CEITER action plan. 2. The European Union’s Horizon 2020 research and innovation program under grant agreement No. 856954.
[1] Ferrari, A. Digital Competence in Practice: An Analysis of Frameworks. (2013). Joint Research Centre of the European Commission., 91. https://doi.org/10.2791/82116. [2] Broadfoot, P., & Black, P. Redefining assessment? The first ten years of assessment in education. Assessment in Education: Principles, Policy and Practice, 11(1), 7–26. (2004). https://doi.org/10.1080/096959404200020 8976 [3] Harlen, W., & James, M. Assessment and learning: Differences and relationships between formative and summative assessment. International Journal of Phytoremediation, 21(1), 365–379. (1997). https://doi.org/10.1080/096959497004030 4 [4] Benali, M., Kaddouri, M. & Azzimani, T.
Digital competence of Moroccan teachers of English. International Journal of Education and Development using ICT, 14(2). (2018). Open Campus, The University of the West Indies, West Indies. Retrieved May 6, 2022 from https://www.learntechlib.org/p/184691/. [5] Kluzer, S., & Priego, L. P. DigComp into action: get inspired, make it happen. Publications Office of the European Union. JRC Science for Policy Report, EUR 29115 EN. Editors: S. Carretero, Y. Punie, R. Vuorikari, M. Cabrera, & W. O’Keefe. (2018). https://doi.org/10.2760/112945 [6] Siddiq, F., Hatlevik, O. E., Olsen, R. V., Throndsen, I., & Scherer, R. Taking a future perspective by learning from the past - A systematic review of assessment instruments that aim to measure primary and secondary school students’ ICT literacy. Educational Research Review, 19, 58–84. (2016). https://doi.org/10.1016/j.edurev.2016.05.0 02 [7] Zenouzagh, M. Z. The effect of online summative and formative teacher assessment on teacher competences. Asia Pacific Education Review, 20(3), 343– 359. (2019). https://doi.org/10.1007/s12564-018-95661 [8] Struyven, K., Blieck, Y., & De Roeck, V.
The electronic portfolio as a tool to develop and assess pre-service student teaching competences: Challenges for quality. Studies in Educational Evaluation, 43, 40–54. (2014). https://doi.org/10.1016/j.stueduc.2014.06. 001