The study purpose is to develop methodological support for students' training for evaluation e-tools for young learners and to check its effectiveness experimentally. The module “Expert evaluation of the quality of e-tools for young learners” is offered for teachers-to-be. The determination of the weighting factor of each criterion by expert evaluations was organized. Educational principles, correlation e-tool content with the curriculum, interactivity, multimedia, assistance system, ergonomic requirements are mentioned. On the basis of the criterion rank, the significance of each criterion was calculated. The indicators to determine the level of preliminary expert evaluations of e-tools are proposed. The results are calculated with nonparametric methods of mathematical statistics, in particular, Pearson's criterion χ2. The conclusion is the expert evaluation has different activity stages, gradually becoming a common phenomenon. Training teachers-to-be for e-tool expert evaluation at Computer Science, Mathematics, English is a complex process.
Elementary school teachers-to-be are implementing a state policy on reforming
education; they should train young learners for life and activities in a digital society, in
a world where the process of getting new knowledge is constantly changed, where new
skills and life-long learning are needed [17; 18; 35]. To our mind, a teacher of
elementary school plays great role in learners’ success to be ready to live in a high-tech
society [
UNESCO recommendations emphasize that for a modern teacher it is not enough to be knowledgeable in the field of information and communication technologies (ICT) and be able to formulate appropriate technological skills for young learners. A teacher should be able to help children to use modern technologies to cooperate successfully, ___________________ Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). to solve problems, to study creatively. In the curriculum one of the key competencies is a digital one, which provides confident and, at the same time, critical application of information and communication technologies, ownership of information and media literacy, understanding ethics when working with information (copyright, intellectual property etc.).
At the present stage of information technology development the spectrum of digital
tools that became available for use in the elementary school has expanded considerably.
For a lesson preparing at education web portals and web pages (Ukrainian forums of
education ideas “Lesson” http://osvita.ua/publishing/urok/5934, “Island of
Knowledge” http://shkola.ostriv.in.ua), multimedia presentations [14; 31], e-textbooks
and manuals [3; 15], e-tools for testing [1; 19], videos of real experimental researches
[20; 30], digital schemes and cards [
However, every lesson is unique, and every computer using must be justified, a teacher during a lesson preparation should not only use a proper e-tool, but also evaluate it as for the effectiveness in achieving the lesson goals. So, training university students as elementary school teachers-to-be how to evaluate e-tools for young learners is important. 2
Different aspects of training elementary school teachers-to-be to use the different
technologies in young learners’ education are analysed in many scientific studies. Thus,
the problems of development students’ information competence, use of information
technology in young learners’ education are considered in the writings of Clive L. Dym
and co-authors [
Different problems of evaluating and improving ICT use are analysed in some works [2; 4; 5; 9; 8; 26; 29].
Our previous works highlight the education potential of e-tools for teaching young
learners, e-tool creation in various instrumental environments [
The purpose of the article is to develop a methodological support for students’ training to evaluate e-tools for young learners and to check its effectiveness experimentally.
Explored materials used in the experiment The choice of e-tools used in the experimental study is connected to the type diversity of e-tools that teachers use at different lesson stages at elementary school (apps, video tools, multimedia presentations, e-manuals, education environments, etc.). To train teachers-to-be for elementary school at Computer Science, Mathematics, English lessons we offered some tools that cover subject or topic learning.
To investigate e-tools for young learners we chose the e-courses for 3-4 grades at elementary school: the complex of educational games “Hour-of-code” for teaching Computer Science with young learners (Fig. 1); the e-course GeoGebra for Mathematics lessons “Adding Fractions” (https://www.geogebra.org/m/xm7EHdmG), “Build a Square Workshop” (https://www.geogebra.org/m/w6kbvzmp) (author John Golden) (Fig. 2); the e-course that is a part of the English language course “Fairyland Express Publishing i-eBook” (Fig. 3), and other popular e-courses as e-tools. To solve article purpose the following research methods were used. Theoretical ones: analysis of scientific works, systematization of scientists’ views and results, study of documents (to know the requirements for e-tools, to determine some aspects of training teachers-to-be to evaluate e-tools for young learners).
Experimental ones: a pedagogical experiment for checking the effectiveness of the offered methodological support; diagnostic ones as questionnaires, observations, analysis of the students’ test results (for collecting data about students’ evaluation skills); nonparametric methods of mathematical statistics, in particular, Pearson criterion χ2 (for calculating the results of empirical research); the method of “expert evaluation” with the rank definition of each criterion (for calculating concordance coefficient that indicates the consistency degree of all “experts” opinions). 4
To our mind, the expert evaluation of different e-tools is based on students’ skills to evaluate an e-tool for adhering to the complex of psychological, pedagogical, ergonomic, technical requirements, the skill to check the effectiveness of every component, the skill to finish the e-tool untimely, the skill to assess the general design of e-tools, the skill to predict young learners’ actions in digital environment, their reactions to learning information and help, the skill to assess the level of the developed e-tools to the lesson aim.
Consequently, to train students-to-be the structure of the learning module “Expert evaluation of the quality of e-tools for young learners” was developed. The module is taught in the Computer Science classes within the discipline “Information and communication technologies in education” for teachers-to-be, future masters of the specialty “Primary Education”.
In order to take up the learning module “Expert evaluation of the quality of e-tools for young learners” we identified the tasks and expected results (knowledge and skills) for students after studying this module (Fig. 4). The module content was developed, a set of educational and methodological materials was prepared such as demonstration materials for familiarizing students with the requirements to be met by learning the etool, the algorithm of expert evaluation, electronic templates for the expert evaluation, the content of practical and laboratory tasks for students was selected, the task for selflearning and further discussion was selected, the set of e-tools for students’ training was selected.
Module “Expert evaluation of the quality of e-tools for young learners”
Tasks: to highlight the content and types of expert evaluation of learning e-tools; to show the procedure of expert evaluation of any learning e-tool; to acquaint with the principles of professional e-tool evaluation.
Expected results: knowledge: the demands to learning e
tools; the types of expert evaluation of learning etools; the criteria for the quality of
learning e-tools; skills: to determine indicators to the quality of learning etools for expert evaluation; to use the expert evaluation method to rank certain indicators; to carry out a preliminary evaluation of learning etools according to the determined indicators. The topics from “Expert evaluation of the quality of e-tools for young learners” are presented in Table 1.
The pedagogical experiment was conducted during 2018-2019 on the basis of the Faculty of Primary Education in H. S. Skovoroda Kharkiv National Pedagogical University, Ukraine. The experiment involves 188 teachers-to-be. The experiment was carried out at several stages initial, developing, final ones.
At the initial stage the experimental and control groups were formed. To do this, we conducted a survey on the awareness of the importance of the preliminary expert evaluation of e-tools, available knowledge and skills in this activity.
To determine the level of awareness of the skills, the students answered the questions about their attitude towards the use of e-tools in the classroom, the frequency of use (at each lesson or not), readiness to select a specific lesson in Mathematics with e-tools, attitudes toward knowledge and skills for acquisition expert evaluation. In addition, we asked to determine the importance of each requirements for the analysis and evaluation of e-tools on a scale from 0 (not important) to 5 (necessary): scientific presentation of e-tools, problem statement, availability of e-tools, visibility, consistency in learning, interactivity, multimedia, assistance system, adaptability to young learners’ opportunities and needs, game component, visual design in e-tools, ensuring success situations. In addition, we asked the students to identify the statements from the proposed list with which they agreed: ─ I understand that the skill to carry out an expert evaluation of the quality of e-tools is important for my future professional activity. ─ The level of my teaching skill does not depend on the ability to assess e-tools. ─ The skill to select and use high-quality e-tools in primary education enhances my own status, public recognition, allows me to implement various educational, research and other opportunities.
To determine the initial level of knowledge and skills in evaluating e-tools, we proposed to determine the content of some requirements such as the scientific presentation of the educational e-tools, system assistance, game component. On the one hand, they are intuitive, and, on the other hand, they demand some additional explanations. In addition, we suggested the students to determine the advantages and disadvantages of e-tools at Computer Science lessons, at English lessons, to evaluate their quality and create the ways to improve them.
According to the surveys results, we combined the students as for the level of their motivation, knowledge and skills to evaluate e-tools into four groups: low, average, sufficient, high. On the basis of the obtained data, the contingent of the experimental and control groups was set up – 104 students were included in the control group, 84 – in the experimental group, which was determined by the set of academic groups. The data obtained at this research stage are presented in Table 2. aowflaeraernneisnsgoef-tiomoplsortance of the preliminary expert evaluation 35 32 32 28 understanding the system of requirements for learning e-tools 32 28 33 28 for schoolchildren tsokoillls tfooresvcahluoaotlechtihlderesynstem of requirements for learning e- 34 26 48 42 21 24 17
The obtained results were calculated by nonparametric methods of mathematical statistics, in particular, according to the Pearson criteria χ2: at this stage, the difference between students of experimental and control groups was insignificant and obtained the value χ2 from 0.4 to 1.2 at the level of significance of 5%, which is less than read by young learners from a computer screen, taking into account competently the psychological and physiological characteristics of young learners.
We offered such tasks. 1. Analyze the slide visualization for: the compliance of a general tool design with its content; the emotions that a slide can cause to a child; the presence of homogeneous or aggressive fields, the feasibility of making changes; the number of objects that are designed once in a child’s view. 2. Make rules for tool visualization for young learners, taking into account their psychological and physiological characteristics. 3. Make presentation slides “Animals” at English lesson using the elements. Change the object size, amount in one slide, background, color scale, etc., if necessary.
Explain the need for the changes made. 4. Using a color wheel, select the colors those that are contrasting, analogous, making a contrast triad (Table 3). 5. Take a look at the psychological and pedagogical requirements, which should correspond to the e-tool. Determine how each requirement in the chosen e-tool is implemented. Fill in table 4. For example, “the content should correspond to the current state of
science development” Color Green
Red Purple Light green
scientific presentation problem statement availability visibility consistency in
learning interactivity assistance system
adaptability game component
visual design ensuring success
situations 6. Analyze the presentations for young learners. Determine whether different types of fonts are used, and the headset and size are selected. Determine the distance from which the entire presentation content is clearly visible (Table 5).
During practical classes at University, students learned to identify the criteria and indicators that were essential for analyzing the quality of the author’s e-tools, to analyze the compliance of professional and own developments with the selected criteria. The determination of the weighting factor of each criterion by the method of expert evaluations was organized [10; 16; 36].
For this purpose, in each academic group, students identified a set of criteria for later e-tool evaluation. They minded educational principles; correlation e-tool content with the curriculum; interactivity, multimedia, assistance system; ergonomic requirements.
To determine the weighting factor of each criterion, the students in academic group acted as experts and determined individually the rank of each criterion (from 1 to 4). The experimental group received the data presented in Table 6.
Next, the concordance coefficient was calculated, which indicated the consistency degree of all students’ opinion as “experts”. In the experimental group the value was W = 0.52, indicating the average degree of consistency in expert evaluations. It should be noted that in the control group, after calculating the concordation coefficient, the table of criterion rank needed coordinating and editing.
On the basis of the table of criterion rank, the significance of each criterion was calculated. For that we found the values that were inverse to the rank sum for each criterion, and then determined the required weighting factors. According to the experts, the importance of each criterion was: educational principles 0.36; correlation e-tool content with the curriculum 0.26; interactivity, multimedia, assistance system 0.24; ergonomic requirements 0.14.
The students chose one e-tool for self-evaluations. Every student evaluated the criterion degree in the e-tool and expressed it in points from 0 to 3. For example, 3 points for high level, 2 points for sufficient level, 1 point for medium level, 0 point for low level. After that, every student calculated the e-tool evaluation, taking into account weighting factor of each criterion (by the formula Ф = ∑ V × P Ф = ∑ V × P , where Vk – weighting factor of each criterion on the basis of expert evaluations, Pk – the demonstration degree of each criterion).
Consequently, as a result of the e-tool expert evaluation, every student gave it a general score: 2.51–3.0 for high level, 1.51–2.50 for sufficient level, 0.76 –1.50 for medium level, and 0.0 – 0.75 for low level.
According to the results, students did not always come to the same consensus about the e-tool quality. It indicated different experience levels of using such e-tools, subjectivity in expert evaluation. At the same time, such activities allowed teachers-tobe to pay more attention to suggestions for improving e-tools, before giving their own evaluation about the e-tool quality.
In the final stage of the experiment, we formulated the indicators to determine the level of preliminary expert evaluations of e-tools: ─ importance of preliminary e-tool expert evaluations; ─ requirements to e-tools for young learners; ─ knowledge of expert evaluation content; ─ checking the data reliability; ─ using expert evaluation to indicator ranks; ─ expert evaluation for e-tool requirements for young learners; ─ level of self-readiness for e-tool expert evaluation.
The results of the experiment about the effectiveness of teaching students to e-tool expert evaluation based on the indicators presented in Table 7. In Table 7, the control group is marked with letter C, and the experimental one is marked with letter E.
So, the quantitative data show that there are significant changes in the experimental group as for teaching students for e-tool expert evaluation in comparison with the previous experiment stage: the difference between the control and experimental groups is quite noticeable in almost all indicators.
For example, in the control group the high and sufficient levels as for the second indicator ‘requirements to e-tools for young learners’ is 18.3% and 26.9% accordingly, in the experimental group 42.9% and 36.9% accordingly. A significant difference is between the groups according to the fifth and sixth indicators as ‘using expert evaluation to indicator ranks’ and ‘expert evaluation for e-tool requirements for young learners’. The obtained results are calculated with nonparametric methods of mathematical statistics. In particular, according to Pearson’s criterion χ2: the obtained
values are significantly higher than the critical value, it indicates the effectiveness of teaching students to e-tool expert evaluation.
General criteria and indicators of the ICT quality in teaching and learning, their evaluating and improving were analyzed in some works [4; 5; 9; 8; 26; 28].
No doubt, that a modern teacher should be trained to work in a new digital society, in the face of high expectations regarding teachers’ competences relating to the development of e-tools that promote effective schooling. As for expert evaluations by students, any teacher, in our opinion, should be able to choose and develop their own evaluation methods that are consistent with lesson aims and content, to use evaluation data to improve teaching, and to motivate children’s learning.
The problems of evaluating the teaching and learning quality, e-books, any
curriculum, e-tools are the research subject by many scholars. The most scholars
conclude that educational evaluating is a complex process. The experimental researches
on the problem of educational evaluation are investigated in some works. Ghaida
Alayyar, Petra Fisser, Joke Voogt underline “the Technological Pedagogical Content
Knowledge (TPACK) framework has been used to prepare pre-service science teachers
at the Public Authority of Applied Education and Training in Kuwait for ICT
integration in education. Pre-service teachers worked in teams to design an ICT solution
for an authentic problem they faced during in-school training” [
As for peculiarities of young learners’ teaching the results of Mandina Shadreck’s
pilot studies show that elementary school teachers have a lack in their knowledge and
skills to integrate tools into the learning process with schoolchildren [
To sum up the researchers’ results we confirm our data that the expert evaluation has different activity stages, gradually becoming a common phenomenon. To our mind, the research in the field of e-tool evaluation is connected probably with the standardization and systematization tendency of e-tool content. 6
After the experiment, we came to the conclusion that training students – teachers-to-be for elementary school – for e-tool expert evaluation in Mathematics, Computer Science, English is a complex process. During the experiment, students learned the peculiarities of selecting such e-tools that can be used at the school lessons in different subjects. We have created and developed the methodological support for training students for elementary school to e-tool expert evaluation. The experimental checking passed successfully, as it is confirmed by the methods of mathematical statistics, so we can recommend the offered methodological support for students’ training for evaluation etools for young learners to use.