578 Training elementary school teachers-to-be at Computer Science lessons to evaluate e-tools Nadiia V. Olefirenko1[0000-0002-9086-0359], Ilona I. Kostikova1[0000-0001-5894-4846], Nataliia O. Ponomarova1[0000-0002-0172-8007], Kateryna O. Lebedieva1[0000-0001-7560-3226], Vira M. Andriievska1[0000-0003-1632-4045] and Andrey V. Pikilnyak2[0000-0003-0898-4756] 1 H. S. Skovoroda Kharkiv National Pedagogical University, 29 Alchevskyh Str., Kharkiv, 61002, Ukraine {olefirenkonn, ilonakostikova, ponomna, andvera80}@gmail.com 2 Kryvyi Rih National University, 11 Vitalii Matusevych Str., Kryvyi Rih, 50027, Ukraine pikilnyak@gmail.com Abstract. The study purpose is to develop methodological support for students’ training for evaluation e-tools for young learners and to check its effectiveness experimentally. The module “Expert evaluation of the quality of e-tools for young learners” is offered for teachers-to-be. The determination of the weighting factor of each criterion by expert evaluations was organized. Educational principles, correlation e-tool content with the curriculum, interactivity, multimedia, assistance system, ergonomic requirements are mentioned. On the basis of the criterion rank, the significance of each criterion was calculated. The indicators to determine the level of preliminary expert evaluations of e-tools are proposed. The results are calculated with nonparametric methods of mathematical statistics, in particular, Pearson’s criterion χ2. The conclusion is the expert evaluation has different activity stages, gradually becoming a common phenomenon. Training teachers-to-be for e-tool expert evaluation at Computer Science, Mathematics, English is a complex process. Keywords: e-tools; young learners; elementary school; experimental research; expert evaluation; weighting factor. 1 Introduction Elementary school teachers-to-be are implementing a state policy on reforming education; they should train young learners for life and activities in a digital society, in a world where the process of getting new knowledge is constantly changed, where new skills and life-long learning are needed [17; 18; 35]. To our mind, a teacher of elementary school plays great role in learners’ success to be ready to live in a high-tech society [23]. UNESCO recommendations emphasize that for a modern teacher it is not enough to be knowledgeable in the field of information and communication technologies (ICT) and be able to formulate appropriate technological skills for young learners. A teacher should be able to help children to use modern technologies to cooperate successfully, ___________________ Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 579 to solve problems, to study creatively. In the curriculum one of the key competencies is a digital one, which provides confident and, at the same time, critical application of information and communication technologies, ownership of information and media literacy, understanding ethics when working with information (copyright, intellectual property etc.). At the present stage of information technology development the spectrum of digital tools that became available for use in the elementary school has expanded considerably. For a lesson preparing at education web portals and web pages (Ukrainian forums of education ideas “Lesson” http://osvita.ua/publishing/urok/5934, “Island of Knowledge” http://shkola.ostriv.in.ua), multimedia presentations [14; 31], e-textbooks and manuals [3; 15], e-tools for testing [1; 19], videos of real experimental researches [20; 30], digital schemes and cards [29] and so on are offered. The presented e-tools are developed by the experienced teachers for their own lessons, taking into account the specifics of their own approaches to teaching a particular subject or topic at school. However, every lesson is unique, and every computer using must be justified, a teacher during a lesson preparation should not only use a proper e-tool, but also evaluate it as for the effectiveness in achieving the lesson goals. So, training university students as elementary school teachers-to-be how to evaluate e-tools for young learners is important. 2 Recent work Different aspects of training elementary school teachers-to-be to use the different technologies in young learners’ education are analysed in many scientific studies. Thus, the problems of development students’ information competence, use of information technology in young learners’ education are considered in the writings of Clive L. Dym and co-authors [7], Mandina Shadreck [27], Bernard Atrogor Oko and Louisa Uwatt [21], Gladwell Wambiri Njeri and Mary Nyokabi Ndani [34], Vanessa W. Vongkulluksn, Ananya M. Matewos, Gale M. Sinatra and Julie A. Marsh [33]. General criteria are reflected in some documents [25; 32]. Different problems of evaluating and improving ICT use are analysed in some works [2; 4; 5; 9; 8; 26; 29]. Our previous works highlight the education potential of e-tools for teaching young learners, e-tool creation in various instrumental environments [22], ICT use for young learners at English lessons [12], in students’ English learning [13]. However, some problems of students’ training for evaluation e-tools for young learners to select the appropriate ones have not been covered in previous research studies. The purpose of the article is to develop a methodological support for students’ training to evaluate e-tools for young learners and to check its effectiveness experimentally. 580 3 Material and methods 3.1 Explored materials used in the experiment The choice of e-tools used in the experimental study is connected to the type diversity of e-tools that teachers use at different lesson stages at elementary school (apps, video tools, multimedia presentations, e-manuals, education environments, etc.). To train teachers-to-be for elementary school at Computer Science, Mathematics, English lessons we offered some tools that cover subject or topic learning. To investigate e-tools for young learners we chose the e-courses for 3-4 grades at elementary school: the complex of educational games “Hour-of-code” for teaching Computer Science with young learners (Fig. 1); the e-course GeoGebra for Mathematics lessons “Adding Fractions” (https://www.geogebra.org/m/xm7EHdmG), “Build a Square Workshop” (https://www.geogebra.org/m/w6kbvzmp) (author John Golden) (Fig. 2); the e-course that is a part of the English language course “Fairyland Express Publishing i-eBook” (Fig. 3), and other popular e-courses as e-tools. Fig. 1. The complex of educational games “Hour-of-code” Fig. 2. The e-course GeoGebra for Mathematics lessons 3.2 Methods for investigation To solve article purpose the following research methods were used. 581 Fig. 3. The course “Fairyland Express Publishing i-eBook” for English Theoretical ones: analysis of scientific works, systematization of scientists’ views and results, study of documents (to know the requirements for e-tools, to determine some aspects of training teachers-to-be to evaluate e-tools for young learners). Experimental ones: a pedagogical experiment for checking the effectiveness of the offered methodological support; diagnostic ones as questionnaires, observations, analysis of the students’ test results (for collecting data about students’ evaluation skills); nonparametric methods of mathematical statistics, in particular, Pearson criterion χ2 (for calculating the results of empirical research); the method of “expert evaluation” with the rank definition of each criterion (for calculating concordance coefficient that indicates the consistency degree of all “experts” opinions). 4 Results To our mind, the expert evaluation of different e-tools is based on students’ skills to evaluate an e-tool for adhering to the complex of psychological, pedagogical, ergonomic, technical requirements, the skill to check the effectiveness of every component, the skill to finish the e-tool untimely, the skill to assess the general design of e-tools, the skill to predict young learners’ actions in digital environment, their reactions to learning information and help, the skill to assess the level of the developed e-tools to the lesson aim. Consequently, to train students-to-be the structure of the learning module “Expert evaluation of the quality of e-tools for young learners” was developed. The module is taught in the Computer Science classes within the discipline “Information and communication technologies in education” for teachers-to-be, future masters of the specialty “Primary Education”. In order to take up the learning module “Expert evaluation of the quality of e-tools for young learners” we identified the tasks and expected results (knowledge and skills) for students after studying this module (Fig. 4). The module content was developed, a set of educational and methodological materials was prepared such as demonstration materials for familiarizing students with the requirements to be met by learning the e- tool, the algorithm of expert evaluation, electronic templates for the expert evaluation, 582 the content of practical and laboratory tasks for students was selected, the task for self- learning and further discussion was selected, the set of e-tools for students’ training was selected. Expected results: Tasks:  to highlight the knowledge: content and types of  the demands to learning e- expert evaluation of tools; Module learning e-tools;  the types of expert “Expert  to show the evaluation of learning e- evaluation of the procedure of expert tools; quality of e-tools evaluation of any  the criteria for the quality of for young learning e-tool; learning e-tools; learners”  to acquaint with the principles of skills: professional e-tool  to determine indicators to evaluation. the quality of learning e- tools for expert evaluation;  to use the expert evaluation method to rank certain indicators;  to carry out a preliminary evaluation of learning e- tools according to the determined indicators. Fig. 4. Module content “Expert evaluation of the quality of e-tools for young learners” The topics from “Expert evaluation of the quality of e-tools for young learners” are presented in Table 1. The pedagogical experiment was conducted during 2018-2019 on the basis of the Faculty of Primary Education in H. S. Skovoroda Kharkiv National Pedagogical University, Ukraine. The experiment involves 188 teachers-to-be. The experiment was carried out at several stages initial, developing, final ones. At the initial stage the experimental and control groups were formed. To do this, we conducted a survey on the awareness of the importance of the preliminary expert evaluation of e-tools, available knowledge and skills in this activity. To determine the level of awareness of the skills, the students answered the questions about their attitude towards the use of e-tools in the classroom, the frequency of use (at each lesson or not), readiness to select a specific lesson in Mathematics with e-tools, attitudes toward knowledge and skills for acquisition expert evaluation. In addition, we asked to determine the importance of each requirements for the analysis and evaluation of e-tools on a scale from 0 (not important) to 5 (necessary): scientific presentation of e-tools, problem statement, availability of e-tools, visibility, consistency in learning, 583 interactivity, multimedia, assistance system, adaptability to young learners’ opportunities and needs, game component, visual design in e-tools, ensuring success situations. In addition, we asked the students to identify the statements from the proposed list with which they agreed: ─ I understand that the skill to carry out an expert evaluation of the quality of e-tools is important for my future professional activity. ─ The level of my teaching skill does not depend on the ability to assess e-tools. ─ The skill to select and use high-quality e-tools in primary education enhances my own status, public recognition, allows me to implement various educational, research and other opportunities. Table 1. Topics of the module Topic Main content Specificity of young learners as users of e-tools. Psychological and pedagogical requirements, which apply to all types of learning e-tools such as scientific presentation of e-tools, problem statement, availability of e- tools, visibility, consistency in learning. Psychological and pedagogical Psychological and requirements, which are additionally advanced to e-tools such as pedagogical interactivity, multimedia, assistance system. Concepts and types of demands for e-tools interactivity in software e-tools. Requirements to be met by e-tools designed for teaching young learners (adaptability to young learners’ opportunities and needs, game component, visual design in e-tools, ensuring success situations). Ways of providing psychological and pedagogical requirements in e-tools. Ergonomic concept in the learning digital environment. Ergonomic requirements for learning e-tools (overall visualization of software Ergonomic, environment, colour characteristics, object location on a screen, text technical and health- outlook, numeric and sign information, audio information, user’s saving requirements feedback, hyperlinks and navigation elements; time-limiters in performing for e-tools individual actions). Health-saving requirements. Technical requirements. Ways to ensure the health and technical requirements for e-tools. Educational expertise of e-tools as an activity aimed to develop a reasonable evaluation of the quality of the developed tools and its conformity to lesson aim. Content, methodical, design, ergonomic Educational demands. Standardization of learning e-tools. The concept of ‘electronic expertise of e-tools certification’. Criteria and indicators of learning e-quality. The quality of the implementation learning e-tool in a curriculum as an object of the educational expertise. Quantitative Application of the expert evaluation method when choosing criteria for methods of expert assessing the quality of e-tools. Determination of weighting factors of the evaluation of e-tools criteria to the developed e-tools. To determine the initial level of knowledge and skills in evaluating e-tools, we proposed to determine the content of some requirements such as the scientific presentation of the educational e-tools, system assistance, game component. On the one hand, they are intuitive, and, on the other hand, they demand some additional explanations. In addition, we suggested the students to determine the advantages and 584 disadvantages of e-tools at Computer Science lessons, at English lessons, to evaluate their quality and create the ways to improve them. According to the surveys results, we combined the students as for the level of their motivation, knowledge and skills to evaluate e-tools into four groups: low, average, sufficient, high. On the basis of the obtained data, the contingent of the experimental and control groups was set up – 104 students were included in the control group, 84 – in the experimental group, which was determined by the set of academic groups. The data obtained at this research stage are presented in Table 2. Table 2. Initial stage of expert evaluation skills for e-tools (persons) Low Average Sufficient High Indicator C Е C Е C Е C Е awareness of importance of the preliminary expert evaluation 35 32 32 28 21 13 16 11 of learning e-tools understanding the system of requirements for learning e-tools 32 28 33 28 24 16 15 12 for schoolchildren skill to evaluate the system of requirements for learning e- 34 26 48 42 17 14 5 2 tools for schoolchildren The obtained results were calculated by nonparametric methods of mathematical statistics, in particular, according to the Pearson criteria χ2: at this stage, the difference between students of experimental and control groups was insignificant and obtained the value χ2 from 0.4 to 1.2 at the level of significance of 5%, which is less than read by young learners from a computer screen, taking into account competently the psychological and physiological characteristics of young learners. We offered such tasks. 1. Analyze the slide visualization for:  the compliance of a general tool design with its content;  the emotions that a slide can cause to a child;  the presence of homogeneous or aggressive fields, the feasibility of making changes;  the number of objects that are designed once in a child’s view. 2. Make rules for tool visualization for young learners, taking into account their psychological and physiological characteristics. 3. Make presentation slides “Animals” at English lesson using the elements. Change the object size, amount in one slide, background, color scale, etc., if necessary. Explain the need for the changes made. 4. Using a color wheel, select the colors those that are contrasting, analogous, making a contrast triad (Table 3). 5. Take a look at the psychological and pedagogical requirements, which should correspond to the e-tool. Determine how each requirement in the chosen e-tool is implemented. Fill in table 4. 585 Table 3. Colors: contrasting, analogous, making contrast triad Analogous Making contrast Color Sample Contrasting Color Colors triad Green red blue, light green purple, dark orange Red Purple Light green Table 4. The psychological and pedagogical requirements in the chosen e-tool How it is implemented Requirement Brief requirement content (what elements, which way) For example, “the content should scientific correspond to the current state of presentation science development” problem statement availability visibility consistency in learning interactivity assistance system adaptability game component visual design ensuring success situations 6. Analyze the presentations for young learners. Determine whether different types of fonts are used, and the headset and size are selected. Determine the distance from which the entire presentation content is clearly visible (Table 5). Table 5. Presentation content Presentation Age / Font Letter height at demonstration Headset Font name Grade height through projector During practical classes at University, students learned to identify the criteria and indicators that were essential for analyzing the quality of the author’s e-tools, to analyze 586 the compliance of professional and own developments with the selected criteria. The determination of the weighting factor of each criterion by the method of expert evaluations was organized [10; 16; 36]. For this purpose, in each academic group, students identified a set of criteria for later e-tool evaluation. They minded educational principles; correlation e-tool content with the curriculum; interactivity, multimedia, assistance system; ergonomic requirements. To determine the weighting factor of each criterion, the students in academic group acted as experts and determined individually the rank of each criterion (from 1 to 4). The experimental group received the data presented in Table 6. Table 6. Table of criterion rank for e-tool expert evaluation Criterion / Expert # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 educational principles (x1) 1 1 1 1 1 2 3 1 1 1 3 1 3 3 1 correlation e-tool content with 3 4 2 2 4 1 1 3 2 2 1 4 1 1 2 the curriculum (x2) interactivity, multimedia, 2 2 3 3 2 3 2 2 3 3 2 2 2 2 3 assistance system (x3) ergonomic requirements (x4) 4 3 4 4 3 4 4 4 4 4 4 3 4 4 4 Next, the concordance coefficient was calculated, which indicated the consistency degree of all students’ opinion as “experts”. In the experimental group the value was W = 0.52, indicating the average degree of consistency in expert evaluations. It should be noted that in the control group, after calculating the concordation coefficient, the table of criterion rank needed coordinating and editing. On the basis of the table of criterion rank, the significance of each criterion was calculated. For that we found the values that were inverse to the rank sum for each criterion, and then determined the required weighting factors. According to the experts, the importance of each criterion was: educational principles 0.36; correlation e-tool content with the curriculum 0.26; interactivity, multimedia, assistance system 0.24; ergonomic requirements 0.14. The students chose one e-tool for self-evaluations. Every student evaluated the criterion degree in the e-tool and expressed it in points from 0 to 3. For example, 3 points for high level, 2 points for sufficient level, 1 point for medium level, 0 point for low level. After that, every student calculated the e-tool evaluation, taking into account weighting factor of each criterion (by the formula Ф = ∑ V × P Ф = ∑ V × P , where Vk – weighting factor of each criterion on the basis of expert evaluations, Pk – the demonstration degree of each criterion). Consequently, as a result of the e-tool expert evaluation, every student gave it a general score: 2.51–3.0 for high level, 1.51–2.50 for sufficient level, 0.76 –1.50 for medium level, and 0.0 – 0.75 for low level. According to the results, students did not always come to the same consensus about the e-tool quality. It indicated different experience levels of using such e-tools, subjectivity in expert evaluation. At the same time, such activities allowed teachers-to- be to pay more attention to suggestions for improving e-tools, before giving their own evaluation about the e-tool quality. 587 In the final stage of the experiment, we formulated the indicators to determine the level of preliminary expert evaluations of e-tools: ─ importance of preliminary e-tool expert evaluations; ─ requirements to e-tools for young learners; ─ knowledge of expert evaluation content; ─ checking the data reliability; ─ using expert evaluation to indicator ranks; ─ expert evaluation for e-tool requirements for young learners; ─ level of self-readiness for e-tool expert evaluation. The results of the experiment about the effectiveness of teaching students to e-tool expert evaluation based on the indicators presented in Table 7. In Table 7, the control group is marked with letter C, and the experimental one is marked with letter E. Table 7. Results of the effectiveness of teaching students to e-tool expert evaluation based on the indicators (percent) Level Indicator Group Low Medium Sufficient High C 24.0 28.8 26.9 20.2 importance of preliminary e-tool expert evaluations Е 8.3 10.7 38.1 42.9 C 17.3 37.5 26.9 18.3 requirements to e-tools for young learners Е 7.1 13.1 36.9 42.9 C 30.8 43.3 11.5 14.4 knowledge of expert evaluation content Е 6.0 14.3 33.3 46.4 C 17.3 29.8 30.8 22.1 checking the data reliability Е 4.8 20.2 25.0 50.0 C 47.1 43.3 7.7 1.9 using expert evaluation to indicator ranks Е 2.4 7.1 46.4 44.0 expert evaluation for e-tool requirements for young C 17.3 35.6 37.5 9.6 learners Е 4.8 2.4 44.0 48.8 C 21.2 34.6 30.8 13.5 level of self-readiness for e-tool expert evaluation E 11.9 39.3 41.7 7.1 So, the quantitative data show that there are significant changes in the experimental group as for teaching students for e-tool expert evaluation in comparison with the previous experiment stage: the difference between the control and experimental groups is quite noticeable in almost all indicators. For example, in the control group the high and sufficient levels as for the second indicator ‘requirements to e-tools for young learners’ is 18.3% and 26.9% accordingly, in the experimental group 42.9% and 36.9% accordingly. A significant difference is between the groups according to the fifth and sixth indicators as ‘using expert evaluation to indicator ranks’ and ‘expert evaluation for e-tool requirements for young learners’. The obtained results are calculated with nonparametric methods of mathematical statistics. In particular, according to Pearson’s criterion χ2: the obtained 588 values are significantly higher than the critical value, it indicates the effectiveness of teaching students to e-tool expert evaluation. 5 Discussion General criteria and indicators of the ICT quality in teaching and learning, their evaluating and improving were analyzed in some works [4; 5; 9; 8; 26; 28]. No doubt, that a modern teacher should be trained to work in a new digital society, in the face of high expectations regarding teachers’ competences relating to the development of e-tools that promote effective schooling. As for expert evaluations by students, any teacher, in our opinion, should be able to choose and develop their own evaluation methods that are consistent with lesson aims and content, to use evaluation data to improve teaching, and to motivate children’s learning. The problems of evaluating the teaching and learning quality, e-books, any curriculum, e-tools are the research subject by many scholars. The most scholars conclude that educational evaluating is a complex process. The experimental researches on the problem of educational evaluation are investigated in some works. Ghaida Alayyar, Petra Fisser, Joke Voogt underline “the Technological Pedagogical Content Knowledge (TPACK) framework has been used to prepare pre-service science teachers at the Public Authority of Applied Education and Training in Kuwait for ICT integration in education. Pre-service teachers worked in teams to design an ICT solution for an authentic problem they faced during in-school training” [2]. Most researchers insist on the need to train students to evaluate e-tool quality. As for peculiarities of young learners’ teaching the results of Mandina Shadreck’s pilot studies show that elementary school teachers have a lack in their knowledge and skills to integrate tools into the learning process with schoolchildren [27]. Birgit Pepin and co-authors write “digital curriculum resources (DCR) offer opportunities for change: of understandings concerning the design and use of DCR; of their quality; and of the processes related to teacher / student interactions with DCR – they provide indeed the foundations for change” [24]. Nils Frederik Buchholtz and co-authors underline the importance of educational evaluation: “combining and integrating the two forms of assessment present the possibility of evaluating different aspects of the pre- service teachers’ perceptions of opportunities to learn” [6]. To sum up the researchers’ results we confirm our data that the expert evaluation has different activity stages, gradually becoming a common phenomenon. To our mind, the research in the field of e-tool evaluation is connected probably with the standardization and systematization tendency of e-tool content. 6 Conclusions After the experiment, we came to the conclusion that training students – teachers-to-be for elementary school – for e-tool expert evaluation in Mathematics, Computer Science, English is a complex process. During the experiment, students learned the peculiarities of selecting such e-tools that can be used at the school lessons in different subjects. We 589 have created and developed the methodological support for training students for elementary school to e-tool expert evaluation. The experimental checking passed successfully, as it is confirmed by the methods of mathematical statistics, so we can recommend the offered methodological support for students’ training for evaluation e- tools for young learners to use. References 1. Abdula, A.I., Baluta, H.A., Kozachenko, N.P., Kassim, D.A.: Peculiarities of using of the Moodle test tools in philosophy teaching. In: Kiv, A.E., Shyshkina, M.P. (eds.) Proceedings of the 7th Workshop on Cloud Technologies in Education (CTE 2019), Kryvyi Rih, Ukraine, December 20, 2019, CEUR-WS.org, online (2020, in press) 2. Alayyar, G.M., Fisser, P., Voogt, J.: Developing technological pedagogical content knowledge in pre-service science teachers: support from blended learning. Australasian journal of educational technology 28(8), 1298–1316 (2012). doi:10.14742/ajet.773 3. Babenko, V.O., Yatsenko, R.M., Migunov, P.D., Salem, A.B.M.: MarkHub Cloud Online Editor as a modern web-based book creation tool. In: Kiv, A.E., Shyshkina, M.P. (eds.) Proceedings of the 7th Workshop on Cloud Technologies in Education (CTE 2019), Kryvyi Rih, Ukraine, December 20, 2019, CEUR-WS.org, online (2020, in press) 4. Ball, S.: Evaluating Educational Programs. In: Bennett R., von Davier M. (eds). Advancing Human Assessment. Methodology of Educational Measurement and Assessment, pp. 341– 362. Springer, Cham (2017). doi:10.1007/978-3-319-58689-2_11 5. Bredtmann, J., Crede, C. J., Otten, S.: Methods for evaluating educational programs: does writing center participation affect student achievement? Evaluation and Program Planning 36(1), 115–123 (2013). doi:10.1016/j.evalprogplan.2012.09.003 6. Buchholtz, N.F., Krosanke, N., Orschulik, A.B., Vorhölter, K.: Combining and integrating formative and summative assessment in mathematics teacher education. ZDM Mathematics Education 50(4), 715–728 (2018). doi:10.1007/s11858-018-0948-y 7. Dym, C., Agogino, A., Eris, O., Frey, D., Leifer, L.: Engineering design thinking, teaching and learning. Journal of Engineering Education 94(1), 103–120 (2013). doi:10.1002/j.2168- 9830.2005.tb00832.x 8. Fox, M.A., Hackerman, N.: Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics. National Research Council. The National Academies Press, Washington (2003). doi:10.17226/10024 9. Hativa, N.: Teaching for Effective Learning in Higher Education. Springer, Dordrecht (2000). doi:10.1007/978-94-010-0902-7 10. Hu, Z., Petoukhov, S., Dychka, I., He, M. (eds.): Advances in Computer Science for Engineering and Education II. Advances in Intelligent Systems and Computing, vol. 938. Springer, Cham (2019). doi:10.1007/978-3-030-16621-2 11. Ivanova, H.I., Lavrentieva, O.O., Eivas, L.F., Zenkovych, Iu.O., Uchitel, A.D.: The students’ brainwork intensification via the computer visualization of study materials. In: Kiv, A.E., Shyshkina, M.P. (eds.) Proceedings of the 7th Workshop on Cloud Technologies in Education (CTE 2019), Kryvyi Rih, Ukraine, December 20, 2019, CEUR-WS.org, online (2020, in press) 12. Kostikova, I.I., Gulich, O.O., Holubnycha, L.O., Besarab, T.P.: Interactive whiteboard use at English lessons: from university students to young learners. Espacios 40(12), 10 (2019) 590 13. Kostikova, I.I.: Information and communication technologies in students’ language learning. International Journal of Education and Science 1(1-2), 7–14 (2018). doi:10.26697/ijes.2018.1-2.01 14. Kozlovsky, E.O., Kravtsov, H.M.: Multimedia virtual laboratory for physics in the distance learning. In: Semerikov, S.O., Shyshkina, M.P. (eds.) Proceedings of the 5th Workshop on Cloud Technologies in Education (CTE 2017), Kryvyi Rih, Ukraine, April 28, 2017. CEUR Workshop Proceedings 2168, 42–53. http://ceur-ws.org/Vol-2168/paper7.pdf (2018). Accessed 21 Mar 2019 15. Kravtsov, H., Pulinets, A.: Interactive Augmented Reality Technologies for Model Visualization in the School Textbook. CEUR-WS.org, online (2020, in press) 16. Landeta, J.: Current validity of the Delphi method in social sciences. Technological Forecasting and Social Change 73(5), 467–482 (2006) 17. Leshchenko, M., Hrynko, V., Kosheliev, O.: Methods of Designing Digital Learning Technologies for Developing Primary School Pre-Service Teachers’ 21st Century Skills. CEUR-WS.org, online (2020, in press) 18. Midak, L.Ya., Kravets, I.V., Kuzyshyn, O.V., Pahomov, J.D., Lutsyshyn, V.M., Uchitel, A.D.: Augmented reality technology within studying natural subjects in primary school. In: Kiv, A.E., Shyshkina, M.P. (eds.) Proceedings of the 2nd International Workshop on Augmented Reality in Education (AREdu 2019), Kryvyi Rih, Ukraine, March 22, 2019. CEUR Workshop Proceedings 2547, 251–261. http://ceur-ws.org/Vol-2547/paper18.pdf (2020). Accessed 10 Feb 2020 19. Mintii, I.S., Shokaliuk, S.V., Vakaliuk, T.A., Mintii, M.M., Soloviev, V.N.: Import test questions into Moodle LMS. In: Kiv, A.E., Soloviev, V.N. (eds.) Proceedings of the 6th Workshop on Cloud Technologies in Education (CTE 2018), Kryvyi Rih, Ukraine, December 21, 2018. CEUR Workshop Proceedings 2433, 529–540. http://ceur-ws.org/Vol- 2433/paper36.pdf (2019). Accessed 10 Sep 2019 20. Nechypurenko, P.P., Starova, T.V., Selivanova, T.V., Tomilina, A.O., Uchitel, A.D.: Use of Augmented Reality in Chemistry Education. In: Kiv, A.E., Soloviev, V.N. (eds.) Proceedings of the 1st International Workshop on Augmented Reality in Education (AREdu 2018), Kryvyi Rih, Ukraine, October 2, 2018. CEUR Workshop Proceedings 2257, 15–23. http://ceur-ws.org/Vol-2257/paper02.pdf (2018). Accessed 30 Nov 2018 21. Oko, B.A., Uwatt, L.: ICT and Teachers’ Performance in Terms of Lesson Preparation and Delivery in Primary Schools in Ogoja Education Zone of Cross River State, Nigeria. Global Journal of Educational Research 14(2), 87–92 (2015). doi:10.4314/gjedr.v14i1.2 22. Olefirenko, N. (2012). Use GeoGebra In Primary Pupils Training. GeoGebra International Journal of Romania 2(2), 40 (2013) 23. Olefirenko, N.V., Kostikova, I.I., Ponomarova, N.O., Bilousova, L.I., Pikilnyak, A.V.: E- learning resources for successful math teaching to pupils of primary school. In: Kiv, A.E., Soloviev, V.N. (eds.) Proceedings of the 6th Workshop on Cloud Technologies in Education (CTE 2018), Kryvyi Rih, Ukraine, December 21, 2018. CEUR Workshop Proceedings 2433, 443–458. http://ceur-ws.org/Vol-2433/paper30.pdf (2019). Accessed 10 Sep 2019 24. Pepin, B., Choppin, J., Ruthven, K., Sinclair, N.: Digital curriculum resources in mathematics education: foundations for change. ZDM Mathematics Education 49(5), 645– 661 (2017). doi:10.1007/s11858-017-0879-z 25. Quality Criteria for Digital Learning Resources, Version 1.0. http://eqnet.eun.org/c/document_library/get_file?folderId=11090&name=DLFE-101.pdf (2010). Accessed 28 Nov 2019 591 26. Schilling, K., Applegate, R.: Best methods for evaluating educational impact: a comparison of the efficacy of commonly used measures of library instruction. Journal of the Medical Library Association 100(4), 258–269 (2012). doi:10.3163/1536-5050.100.4.007 27. Shadreck, M.: Integrating ICTs into the environmental science primary school classroom in Chegutu district, Zimbabwe: problems and solutions. European Journal of Science and Mathematics Education 3(1), 90–96 (2015) 28. Shapovalov, V.B., Shapovalov, Ye.B., Bilyk, Zh.I., Megalinska, A.P., Muzyka, I.O.: The Google Lens analyzing quality: an analysis of the possibility to use in the educational process. In: Kiv, A.E., Shyshkina, M.P. (eds.) Proceedings of the 2nd International Workshop on Augmented Reality in Education (AREdu 2019), Kryvyi Rih, Ukraine, March 22, 2019. CEUR Workshop Proceedings 2547, 117–129. http://ceur-ws.org/Vol- 2547/paper09.pdf (2020). Accessed 10 Feb 2020 29. Shyshkina, M.P., Kohut, U.P., Popel, M.V.: The Design and Evaluation of the Cloud-based Learning Components with the Use of the Systems of Computer Mathematics. In: Ermolayev, V., Suárez-Figueroa, M.C., Yakovyna, V., Kharchenko, V., Kobets, V., Kravtsov, H., Peschanenko, V., Prytula, Ya., Nikitchenko, M., Spivakovsky A. (eds.) Proceedings of the 14th International Conference on ICT in Education, Research and Industrial Applications. Integration, Harmonization and Knowledge Transfer (ICTERI, 2018), Kyiv, Ukraine, 14-17 May 2018, vol. II: Workshops. CEUR Workshop Proceedings 2104, 305–317. http://ceur-ws.org/Vol-2104/paper_156.pdf (2018). Accessed 30 Nov 2018 30. Striuk, A.M., Rassovytska, M.V., Shokaliuk, S.V.: Using Blippar Augmented Reality Browser in the Practical Training of Mechanical Engineers. In: Ermolayev, V., Suárez- Figueroa, M.C., Yakovyna, V., Kharchenko, V., Kobets, V., Kravtsov, H., Peschanenko, V., Prytula, Ya., Nikitchenko, M., Spivakovsky A. (eds.) Proceedings of the 14th International Conference on ICT in Education, Research and Industrial Applications. Integration, Harmonization and Knowledge Transfer (ICTERI, 2018), Kyiv, Ukraine, 14- 17 May 2018, vol. II: Workshops. CEUR Workshop Proceedings 2104, 412–419. http://ceur-ws.org/Vol-2104/paper_223.pdf (2018). Accessed 30 Nov 2018 31. Tkachuk, V., Yechkalo, Yu., Semerikov, S., Kislova, M., Khotskina, V.: Exploring Student Uses of Mobile Technologies in University Classrooms: Audience Response Systems and Development of Multimedia. CEUR-WS.org, online (2020, in press) 32. UNESCO ICT Competency Framework for Teachers. UNESCO, Paris (2011) 33. Vongkulluksn, V.W., Matewos, A.M., Sinatra, G.M., Marsh, J.A.: Motivational factors in makerspaces: a mixed methods study of elementary school students’ situational interest, self-efficacy, and achievement emotions. International Journal of STEM Education 5, 43. (2018). doi:10.1186/s40594-018-0129-0 34. Wambiri Njeri, G., Nyokabi Ndani, M.: Kenya primary school teachers’ preparation in ICT teaching: teacher beliefs, attitudes, self-efficacy, computer competence, and age. African Journal of Teacher Education 5(1) (2017). doi:10.21083/ajote.v5i1.3515 35. Yaroshenko, O.G., Samborska, O.D., Kiv, A.E.: An integrated approach to digital training of prospective primary school teachers. In: Kiv, A.E., Shyshkina, M.P. (eds.) Proceedings of the 7th Workshop on Cloud Technologies in Education (CTE 2019), Kryvyi Rih, Ukraine, December 20, 2019, CEUR-WS.org, online (2020, in press) 36. Ziemba, P., Piwowarski, M., Jankowski, J., Wątróbski, J.: Method of Criteria Selection and Weights Calculation in the Process of Web Projects Evaluation. In: Hwang, D., Jung, J.J., Nguyen, NT. (eds.) Computational Collective Intelligence. Technologies and Applications. ICCCI 2014. Lecture Notes in Computer Science, vol. 8733, pp. 684–693. Springer, Cham (2014). doi:10.1007/978-3-319-11289-3_69