Psychometric Properties of a 21st Century Digital Skills Scale

Abstract

To prepare students to cope successfully with increasingly complex life and environments based upon the extensive application of information-communication technology (ICT), today's education should focus on cultivating 21st century digital skills. To assess the extent to which these ICT-based skills have been attained, an appropriate scale with good psychometric properties needs to be applied. By using a sample of 667 teachers from about 200 secondary schools across Serbia, this study examined the psychometric properties of a 21st century digital skills scale.

1. Introduction

Today, education needs to prepare students to cope successfully with increasingly complex life and work environments (e.g., [1]). To this end, different aspects of learning need to be fostered, such as communication and collaboration, whose learning benefits have been advocated by many social and constructivist approaches to learning (e.g., [2]). Those aspects of learning may concern 21st century skills that promote successful citizenship in a global society (e.g., [3, 4]).

A recently proposed 4C’s model underlines four 21st century skills: creativity, critical thinking, communication, and collaboration (see Fig. 1). It is stressed that these skills should be considered as

2022 Copyright for this paper by its authors. the foundation for a full model of learning and innovation skills comprising: ( 1 ) creativity and innovation, ( 2 ) critical thinking and problem solving, ( 3 ) communication, and ( 4 ) collaboration [5].

Concerning the extensive application of information-communication technology (ICT) in almost all areas of life and work, it is appropriate to focus on 21st century digital skills. A recent detailed review of the literature identified seven such core ICT-based skills. These skills were: ( 1 ) technical skills, ( 2 ) information management skills, ( 3 ) communication, ( 4 ) collaboration, ( 5 ) creativity, ( 6 ) critical thinking, and ( 7 ) problem solving [6]. Many of these skills can be cultivated in innovative digital learning environments (e.g., [7, 8]). Note that a model related to learning with technology, proposed more than twenty years ago, suggests there should be three Cs related to thinking: critical thinking, creative thinking, and complex thinking, the last of which can be used to denote thinking that occurs in the activities of designing, problem solving, and decision making [9].

It is important to underline that all digital skills require digital competence, which has been recognized as one of the eight key competences for lifelong learning, focusing not only on digital technologies but also on communication, critical thinking, and collaboration, among other things [10]. To measure the development of digital education in Serbia, the Serbian Ministry of Education, Science and of Technological development recently published “Digital Competence Framework” [11], whose application would contribute to the development of students’ digital competences for living and working in a digital society.

Apart from learning a specific subject, e-learning may be used to cultivate 21st century digital skills. To be able to evaluate the extent to which these skills have been promoted, an instrument measuring this promotion needs to be applied. According to the authors’ readings, such an instrument is lacking in the literature at present. To improve this state, we developed a short instrument (a 7-item scale), being concerned with using it to determine potential benefits of online teaching using platform technologies (which was especially relevant during the Covid-19 pandemic).

The reliability (Cronbach’s alpha) of this instrument was over 0.90 and the single-factor model of the applied seven items could explain more than 70% of the total variance among their values [12]. To apply this scale in further research confidently, it needs to have good psychometric properties. Hence, the present study focused on the examination of these properties, which dealt with representativity, reliability, homogeneity, and validity. These properties, respectively, stand for the following issues: the adequacy of the sampling of the applied items from the universe of all items concerning the same construct to be measured; the precision of the measurement of this construct using these applied items; the size of the main component measured by the applied items; and the individual alignments (correlations) of these items with this main component [13].

The applied research question was: Do the developed 7-item scale have good representativity, reliability, homogeneity, and validity? A positive answer to this question could contribute to the improvement mentioned above concerning the missing instrument that assesses the promotion of 21st century digital skills.

The second section describes the methodology used in this empirical research, while the third section summarizes and discusses the main findings. The final, closing section critically examines this empirical research and gives suggestions for further research.

2. Methodology

A convenient sample was used. It comprised of 667 teachers: while 279 were from primary schools, 388 were from secondary. Of those 388 teachers, 181 came from gymnasiums (grammar schools), whereas 207 worked in vocational secondary schools. Teachers came from about 200 schools across Serbia. Among them were 100 primary schools, 40 gymnasiums, and 60 vocational schools.

The attainment of 21st century skills was examined using a 7-item instrument, whose indicators were derived from the above-mentioned study of van Laar and colleagues [6]. These indicators are listed in Table 1. To collect their values (measured on a 0–10 scale ranging from never to almost always), part of an online questionnaire was used. The questionnaire also comprised several questions concerning participants’ background data, including their gender, experience with online teaching, and type of school they teach in.

The collected data were examined by an SPSS macro [13], which determines representativity, reliability, homogeneity, and validity of the instrument in question. Although developed in the end of 1990s, this macro has been used in many empirical studies that assessed these psychometric properties of different instruments (e.g., [14, 15]).

For each psychometric feature, this macro calculates the values of different measures. These values are expressed on a 0–1 scale, considering values close to 1 as a sign of good quality.

An acceptable cut-off for those values may be set around 0.70 or 0.80. For example, the usual reliability cut-off is 0.70 [16], meaning that the reliability above 0.70 is acceptable. Regarding representativity, in particular the so-called Kaiser-Mayer-Olkin measure of sampling adequacy, values between 0.70 and 0.80 are usually considered good [17].

3. Results and Discussion

The representativity, reliability, homogeneity, and validity of the translated instrument are presented in Tables 2–5. The content of these tables clearly evidence that the applied scale had good psychometric features, which answers the applied research question in a positive way. It can be thus said that this scale successfully measures one underlying construct and thus it can confidently be used in further research. Hence, the outcome of this study contributes to developing an instrument that assesses the promotion of 21st century digital skills, which has been a neglected research area so far, to the authors’ readings.

The data summarized in Table 5 evidence good individual psychometric features of the applied items. Hence, statistical analyses may deal with using individual items as well. Their somewhat low reliabilities (especially of items 1, 3, 4, and 7) can be improved when the initial, raw scores are transformed into Guttman’s [18] image scores; this approach have been successfully applied in a number of studies elsewhere (e.g., [19, 20]).

To clarify potential significance of using this 7-item instrument in educational research, it was checked whether there were differences among teachers from different kinds of school regarding the extent to which they promoted 21st century digital skills in their online teaching during the Covid-19 pandemic. By representing the value of the promotion in question by the average value of responses to the applied seven items, it was found that secondary school teachers fostered digital skills more than primary school teachers (the medians were 8.00 vs. 7.43, respectively; recall that a 0–10 scale for teachers’ answers was applied), whereas gymnasium (grammar school) teachers fostered those skills more than vocational school teachers: the medians were 8.43 vs. 7.71, respectively [12]. Because students’ abilities are, in general, more diverse in vocational schools than in gymnasiums (as well as more in primary than secondary schools), this outcome, which should be taken as a sign of external validity of the applied 7-item instrument, does support its application in further research on promoting 21st century digital skills. Measures of Reliability Under Guttman’s Measurement Model

Guttman-Nicewander ITEM I encouraged students to understand the basic functionalities and

modes of work with digital platform and computer programs used I encouraged students to search, select, and organize information in

order to successfully attend lectures I encouraged students to effectively share different types of

information (text, images, videos, etc.) among class participants I encouraged students to work in teams in order to effectively share different types of information, have discussions, and make decisions regarding the work goals set I encouraged students to consider known facts in new ways or to use

new ideas to produce the solution required I encouraged students to evaluate validity and expediency of the

information and approaches used in the class work I encouraged students to understand the tasks given, knowledge and skills needed to solve them, as well as to apply such knowledge and skills to find solutions required REP 0.99 0.99 0.99 0.99 0.99 0.99 0.99

REP – Representativity; REL – Reliability; HOM – homogeneity; H – Validity in Hotelling’s space; B – Validity in Burt’s space psi 1 psi 2 psi 3 lambda 1 lambda 3 lambda 6 beta 3 Rho h 1 h 2 h 5 0.61 0.75 0.59 0.58 0.75 0.71 0.62 REL

HOM 0.76 0.85 0.76 0.75 0.85 0.83 0.78

4. Closing Remarks

By using a large sample of secondary school teachers, this study examined the psychometric properties of a 7-item scale that was used to assess the extent to which participants promoted 21st century digital skills in their online teaching. These properties dealt with representativity, reliability, homogeneity, and validity. The examination showed that this scale was of a good quality for each of these properties.

Despite this outcome, nothing can be said about the use of this scale to survey students’ opinions about the extent to which they acquired 21st century digital skills in their online (or other technology supported) learning. Further research may thus examine the psychometric properties of this scale when used by students whose learning has been supported with technology in some way.

Further research may also focus on developing and testing a larger scale, where each 21st digital skill is represented by several items (indicators). Bearing in mind 4C’s model of 21st century skills mentioned above (whose founding skills are creativity, critical thinking, communication, and collaboration), this larger scale may have a complex factor structure that would include some of those founding skills.

Acknowledgements

The authors wish to thank all teachers who participated in this study. The research done by the first and second authors was funded by the Ministry of Education, Science and Technological Development of the Republic of Serbia (Contract No. 451-03-68/2022-14/200018).

5. References

Ministry of Education, Science and Technogical Development (MESTD) Okvir digitalnih kompetencija. Nastavnik za digitalno doba. [Digital competence framework. Teacher for the digital age], 2019. URL: https://mpn.gov.rs/vesti/usvojen-novi-okvir-digitalnihkompetencija-nastavnika/

Kadijevich, D.M., Gutvajn, N., & Ljubojevic, D. Fostering 21st century digital skills using platform technologies. Manuscript submitted for publication.

Knezevic, G., & Momirovic, K. (1996). RTT9g and RTT10g: Two programs for analysis of the metric characteristic of composite measuring instruments. In Momirovic, K. (Ed.), Measuring in Psychology (Vol. II, pp. 35–56). Beograd: Institut za Kriminoloska i socioloska istrazivanja i Centar za primenjenu psihologiju.

Kadijevich, D. (2000). Representativity, reliability, homogeneity and validity of Selwyn’s computer attitude scale for 16–19 education, Psihologija, 33, 3/4 (2000), 491–498.

Cupać, Đ., Micić, I., Mijatović, N., Pavlović, S., & Stekić, K. (2019). Development and perception of the Multifaceted psychopathy self-assessment inventory [Razvoj i percepcija Multifacetnog inventara za samoprocenu psihopatije]. Psihološka istraživanja, ( 1 ), XXII, 153– 184. URL: https://doi.org/10.5937/PSISTRA22-19141

Nunnally, J.C. (1978). Psycometric theory (2nd ed). McGraw Hill.

Hair, J.F., Black, W.C., Babin, B.J., Anderson, R.E., & Tatham, R.L. (2009). Multivariate Data Analysis (7th ed). Pearson.

Guttman, L. (1953). Image theory for the structure of quantitative variates. Psychometrika, 21( 3 ), 277–296. https://doi.org/10.1007/BF02289264

Kadijevich, D.M., Odovic, G., & Maslikovic, D. (2016). Using ICT and quality of life: Comparing persons with and without disabilities. In Miesenberger, K., Bühler, C., & Penaz, P. (Eds.) ICCHP 2016, Part I. LNCS, vol. 9758, pp. 129–133. https://doi.org/10.1007/978-3-31941264-1_18

Kadijevich, D.M., Ljubojevic, D., & Gutvajn, N. (2022). What kind of e-assessment feedback is important to students? An empirical study. In Passey D., Leahy, D., Williams, L., Holvikivi, J., & Ruohonen, M. (Eds.), Digital transformation of education and learning – Past, present and future. OCCE 2021. IFIP Advances in Information and Communication Technology, 642, 261-273. https://doi.org/10.1007/978-3-030-97986-7_22

References

[1] OECD , OECD Skills Outlook 2019 : Thriving in a Digital World , OECD Publishing, Paris, 2019 . https://doi.org/10.1787/df80bc12-en

[2] Gomez , E.A. , Wu , D. , & Passerini , K. Computer-supported team-based learning: The impact of motivation, enjoyment and team contributions on learning outcomes , Computers & Education , ( 2010 ) 55 ( 1 ), 378 - 390 . https://doi.org/10.1016/j.compedu. 2010 . 02 .003

[3] Carnevale , A.P. , & Smith , N. Workplace basics, The skills employees need and employers want . Human Resource Development International , ( 2013 ) 16 ( 5 ), 491 - 501 . https://doi.org/10.1080/13678868. 2013 .821267

[4] Voogt , J. , & Roblin , N.P. A comparative analysis of international frameworks for 21st century competences: Implications for national curriculum policies , Journal of Curriculum Studies , ( 2012 ) 44 ( 3 ), 299 - 321 . https://doi.org/10.1080/00220272. 2012 .668938

[5] Partnership for 21st Century Learning, Framework for 21st century learning , 2015 . URL: https://www.battelleforkids.org/networks/p21/frameworks-resources

[6] van Laar , E., van Deursen , A.J.A.M. , van Dijk , J.A.G.M. , & de Haan, J, The relation between 21st-century skills and digital skills: A systematic literature review . Computers in Human Behavior , ( 2017 ) 72 , 577 - 588 . https://doi.org/10.1016/j.chb. 2017 . 03 .010

[7] Kadijevich , D.M. , Critical e -learning skills and cultivating them , in: S. Jovanović (Ed.) Proceedings of the eight international conference on e-learning , Belgrade Metropolitan university, Belgrade, Serbia, 2017 , pp. 30 - 33 .

[8] Niess , M.L. , & Gillow-Wiles , H. , Expanding teachers' technological pedagogical reasoning with a systems pedagogical approach Australasian Journal of Educational Technology , ( 2017 ), 33 ( 3 ), 77 - 95 . https://doi.org/10.14742/ajet.3473

[9] Jonassen , D.H. , Computers as mindtools for schools: Engaging critical thinking (2nd ed) , Prentice Hall , 2000 .

[10] European Commission , Directorate-General for Education, Youth, Sport and Culture, Key competences for lifelong learning . Publications Office of the European Union , 2019 . URL: https://doi.org/10.2766/569540 [16] [17] [18]