<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta>
      <journal-title-group>
        <journal-title>September</journal-title>
      </journal-title-group>
    </journal-meta>
    <article-meta>
      <title-group>
        <article-title>Monitoring the efectiveness of the STEAM-oriented environment in general secondary education institutions: approaches to defining criteria</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Oksana V. Ovcharuk</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Nataliia V. Soroko</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Institute for Digitalisation of Education of the NAES of Ukraine</institution>
          ,
          <addr-line>9 M. Berlynskoho Str., Kyiv, 04060</addr-line>
          ,
          <country country="UA">Ukraine</country>
        </aff>
      </contrib-group>
      <pub-date>
        <year>2024</year>
      </pub-date>
      <volume>23</volume>
      <issue>2024</issue>
      <fpage>78</fpage>
      <lpage>87</lpage>
      <abstract>
        <p>The article is devoted to the problems and advantages of monitoring the implementation of STEAM-oriented educational environment in institutions of general secondary education; identifying criteria and indicators for its implementation; analysis of the state of development of STEAM-oriented educational environments in 2024 in Ukraine based on defined criteria. The approaches to monitoring the STEAM-oriented educational environment development for general secondary education are highlighted. The study showed that there is a significant interest of teachers, students, and other participants in the educational process in the implementation and development of a STEAM-oriented educational environment in general education institutions. The study identified gaps in the assessment of the efectiveness of the use of STEM education in Ukrainian schools through a series of surveys and studies, especially in the absence of a single approach to continuous monitoring. Therefore, the authors proposed criteria based on both international approaches and the peculiarities of the Ukrainian education system. The following problems were identified in the Ukrainian educational institutions: funding of STEAM projects; provision of general education institutions with modern laboratories, tools, means of virtual and augmented reality, and special classrooms; maintaining the motivation of teachers and students to participate in STEAMactivities. The article analyses the results of studies by the Ministry of Education and Science of Ukraine as well as own studies carried out in 2022-2023 years. These studies have raised the issue of the efectiveness of the STEAM-oriented educational environment based on the point of view of Ukrainian teachers. The data revealed in the studies enhanced the identification of criteria that can be applied to the assessment of the influence of modern technologies on the educational process and the identification of the key factors contributing to the successful integration of STEM education in general secondary education institutions. The article aims to present the approaches and criteria for monitoring STEAM-oriented educational environments and analyse the challenges faced by Ukrainian secondary educational institutions nowadays.</p>
      </abstract>
      <kwd-group>
        <kwd>eol&gt;STEAM-oriented environment</kwd>
        <kwd>quality monitoring</kwd>
        <kwd>secondary education</kwd>
        <kwd>innovative teaching</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>In the modern educational landscape, STEAM (Science, Technology, Engineering, Arts, and Mathematics)
education has emerged as a crucial framework for preparing students for the requirements of the 21st
century. Integrating these disciplines into school curricula equips students with a broad skill set,
fostering creativity, critical thinking, and problem-solving abilities. STEAM education is important for
the education and training of students at school. This education is important for raising well-rounded
innovative individuals who can contribute to the rapidly developing world. After all, to obtain a future
profession, young people today need quality education, and the ability to adapt to rapid changes in
science, technology, engineering, art and mathematics. Therefore, monitoring the development of
STEAM education is a key process for determining the quality of education.</p>
      <p>
        The development of STEAM education in general secondary education institutions in Ukraine is
conducted by Ukrainian legislation, including the laws “On Education” [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ], “On Full General Secondary
Education” [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ], and “On Scientific and Scientific and Technical Activities” [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]. It also aligns with the
State standards of primary and basic secondary education. The Government approved the Concept
of implementation of state policy in the field of general secondary education reform “New Ukrainian
School” for the period until 2029. Implementation of the Concept will create conditions for reforming
general secondary education, taking into account the experience of the leading countries of the world.
New content standards of general secondary education will be developed, which will be based on a
competency-based and person-oriented approach to education as well as the Concept of the Development
of Science and Mathematics Education (STEM education). Furthermore, the Ministry of Education and
Science of Ukraine has developed and presented a Plan of Measures for the implementation of the
Concept for the Development of Science and Mathematics Education (STEM education) until 2027. All
these resolutions and regulatory documents serve as the foundation for implementing this educational
direction.
      </p>
      <p>
        By implementing the main tasks, the development of STEM education in Ukrainian educational
institutions is ensured at the following levels: elementary – stimulation of curiosity and support of
interest in learning and the search for knowledge, motivation for independent research, creation of
simple devices, constructions, scientific and technical creativity; basic – forming a stable interest
in natural and mathematical subjects, mastering technological literacy and problem-solving skills,
involvement in research, invention, project activities, which will make it possible to increase the
share of those who seek to choose scientific, technical, engineering professions; professional –
indepth mastery of the system of knowledge and skills of STEM education using methods of scientific
research, implementation of innovative projects; higher/professional – formation of specialists in
various scientific and technical, engineering professions on the basis of institutions of higher education,
as well as improving the professional skills of pedagogical workers in the implementation of new
teaching methods, relevant courses and implementation of innovative projects. For the development
of STEM education, it is important to create an appropriate environment and constant support and
evaluation of the efectiveness of functioning [
        <xref ref-type="bibr" rid="ref4 ref5 ref6 ref7">4, 5, 6, 7</xref>
        ]. These activities are important for both students
and educational institutions to understand the problems and gaps that educators face and need to be
addressed. In connection with the large-scale aggression of the Russian Federation against Ukraine in
2022, certain problems have deepened and intensified in general secondary education institutions. In
particular, this applies to the digital environment and support, including the STEM environment.
      </p>
      <p>The research problem of this study is to find out the problems and gaps accompanying STEAM
education in Ukrainian schools based on existing surveys, and also, guided by international approaches,
to identify criteria for evaluating the efectiveness of STEAM education in Ukrainian educational
institutions. Also, the tasks of the study are to analyse the views of Ukrainian teachers and foreign
researchers on the issue of STEAM education, which will help to formulate a vision of monitoring the
efectiveness of the STEM environment in Ukrainian schools in the future.</p>
    </sec>
    <sec id="sec-2">
      <title>2. Literature review</title>
      <p>
        It should be noted that the issue of evaluating the efectiveness of the educational environment is part
of the processes related to monitoring the quality of education in general. According to the Decree of
the Ministry of Education and Science of Ukraine No. 54 adopted on January 16, 2020, education quality
monitoring involves implementing consistent and systematic measures to identify and track trends
in educational quality across the country, specific regions, and educational institutions. It includes
assessing how well actual educational outcomes align with the stated objectives and identifying the
extent, direction, and causes of any deviations from these goals [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ]. That is why it is important to
identify among scientific studies those that carry out the monitoring of the quality of education, the
efectiveness of the STEM education environment and the teachers’ perceptions of the state of such
environment.
      </p>
      <p>
        The problem of monitoring the quality of education was studied by Kondrashova [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ], Hrynevych
et al. [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ], Batechko [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ], Lukina [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ], Denysiuk and Tytarenko [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ], Gong and Wei [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ], Pérez Torres
et al. [
        <xref ref-type="bibr" rid="ref15">15</xref>
        ].
      </p>
      <p>
        Lukina [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ] proposes to define the concept of ‘monitoring of education” according to the model
of the quality of education, which covers the following structural elements. Internal: the quality
of educational processes, the quality of professional training and qualifications of teaching staf, the
quality of students as individuals, the quality of educational technologies, the quality of resource support
(teaching methodical, informational, material-technical), the quality of management technologies and
resources in educational institutions and the quality (efectiveness) of education management, the quality
of objectives and quality norms in the educational system of education, the quality of organisational
structures in the educational system, the professional quality of management personnel in education,
the quality of the education monitoring system (the quality of evaluation mechanisms and procedures),
the quality of education financing mechanisms. The external quality of education in the gymnasium as a
level of training of graduates (future specialists who must implement the tasks set forth by society) [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ].
      </p>
      <p>
        Denysiuk and Tytarenko [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ] suggest focusing on specific aspects when evaluating the educational
environment. These aspects include the formation of IT thinking among the participants of the
educational process, creating an accessible, comfortable, and safe educational environment, and organizing
a system of psychological support. The researchers emphasise that the monitoring study is based on
several conceptual tasks, such as obtaining information about the state of pedagogical activity in general
secondary education institutions within the context of education reform and tracking trends in the
organization of educational activities. They also propose using standardized online questionnaires at
ifve stages of the monitoring study to ensure the process’s repeatability and the structure of the results.
      </p>
      <p>
        Kalogiannakis and Papadakis [
        <xref ref-type="bibr" rid="ref16">16</xref>
        ] examine opportunities for STEM learning early in life, which
is important because developing STEM skills can foster student interest in the future. In particular,
Papadakis [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ] evaluates the game-development approach to teach introductory programming concepts
in secondary education. Papadakis et al. [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ] analysing the STEM education experience in young
children’s education has found that it helps children get the appropriate tools crucial to any successful study.
They concluded that teachers’ perceptions, attitudes, and technological competencies are considered
the primary determinants of technology adoption in curriculum and pedagogy. The important gap that
needs to be addressed is the teachers’ lack of relevant knowledge or the lack of relevant provisions for
their efective integration of innovative technologies into school curricula [
        <xref ref-type="bibr" rid="ref19">19</xref>
        ].
      </p>
      <p>
        Also, the issues of the teachers’ attitudes toward the implementation of digital technologies and
their information literacy are revealed in the works of Ukrainian scholars. Lukianova and Ovcharuk
[
        <xref ref-type="bibr" rid="ref20">20</xref>
        ] pay attention to the analysis of the information literacy of teachers and vulnerable categories
of the population as well as the measures for informational support for elderly people. The authors
present the results of the Ukrainian teachers’ self-assessment of the level of digital competence and state
that this competence needs to be improved. Soroko [
        <xref ref-type="bibr" rid="ref21">21</xref>
        ] focuses on the teachers’ digital competence
development as one of the important factors for the establishment and support of the STEAM-oriented
learning environment in schools.
      </p>
      <p>
        Gong and Wei [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ] emphasise the importance of analysing Innovative Teaching Quality. They focus
on identifying exceptional teaching talents through various evaluations and implementing the role
model concept to highlight the best teaching practices and efective classroom management conducive
to teaching and learning. They introduce three key components for evaluation: multilevel
evaluation, humanity-related investigation and discussion, and developmental random lecture attendance.
Multilevel evaluation involves evaluating various activities of participants in the educational process,
such as classroom teaching, students’ learning experiences, and practical education. Humanity-related
investigation and discussion utilise questionnaire surveys or discussions to address issues and
shortcomings in various aspects, such as curriculum development and updating, teachers’ teaching methods,
and students’ learning styles. Developmental random lecture attendance includes occasional visits by
teachers and management to other teachers’ lectures, providing real-time feedback, exchange of ideas,
and technology skill improvement for curriculum development.
      </p>
      <p>Having reviewed the works of researchers on monitoring the quality of education, and the impact of
STEM education on schooling and the learning environment, this study investigated the state, gaps
and problems that exist in Ukrainian secondary education institutions, and based on this, there were
criteria identified that can be used to determine the efectiveness of the STEM education environment.</p>
    </sec>
    <sec id="sec-3">
      <title>3. Research methods</title>
      <p>To achieve the goal of our research, a review of Ukrainian and foreign research was carried out to
identify common features regarding STEM education in educational institutions. The analysis of surveys
carried out by the Ministry of Education and Science of Ukraine, as well as other Ukrainian research
institutions, made it possible to reveal the attitude of teachers to the use of STEAM laboratories and to
ifnd out the extent of the spread of STEAM education in Ukraine. Such an analysis made it possible to
outline the criteria for evaluating the efectiveness of the STEM environment in secondary education
institutions, which can be used. Own research carried out by the authors in 2022-2023, namely, a survey
of teachers, made it possible to identify several problems and gaps both in the technical support of
schools and allowed to identify gaps in the readiness of teachers to use digital technologies for the
implementation of STEM education. For these surveys, we used quantitative methods of calculating
the obtained data. The generalization of the received problems, which were noted by the teachers,
made it possible to single out problems common to many institutions in the implementation of STEAM
education, which require further solutions at various levels of educational policy.</p>
    </sec>
    <sec id="sec-4">
      <title>4. Research results</title>
      <p>Monitoring a STEAM-oriented educational environment is a complex process that must cover
different procedures and take into account the specifics of the educational and social environment. A
STEAM-oriented educational environment incorporates computer, mobile, and cloud-based
technologies, along with practical, interdisciplinary, and project-based learning methods in the study of natural
and mathematical disciplines. It aims to encourage students’ creative thinking through the integration
of various art forms in the learning process. This approach also enhances the digital and professional
competencies of teachers, helping students to acquire essential 21st-century skills. It motivates students
to explore scientific problems and efectively integrate knowledge across various subjects at national
and international levels.</p>
      <p>It should be noted that despite the many obstacles faced by teachers in Ukraine due to martial law,
teachers actively participate in scientific and practical activities related to the development of STEM
education, which is part of STEAM. This experience is reflected in the articles in the anthologies of
several conferences devoted to this topic. Thus, the collection of materials “STEM school – 2022” contains
11 developments of practising teachers on various issues, in particular, virtual STEM laboratories for
research and experiments in natural and mathematical subjects; modelling and research of physical
phenomena using STEM education; involvement of parents in the implementation of the concept
of development of science and mathematics education (STEM education); methodological cases of
gender-sensitive STEM lessons, etc.</p>
      <p>After the full-scale invasion of Ukraine, the existing STEM centres began to work with IDP children
who do not have the technical ability to get free access to education, as well as to support teachers in
the organization of the educational process. This important process supported local communities and
contributed to the continuity of the educational process. For example, as early as September 2022, the
Educational Initiatives Foundation opened the educational hub “Skills of the Future” based on the STEM
centre in Cherkasy with the support of the EU4Dialogue project of the European Union and the United
Nations Development Program (UNDP). By the beginning of summer 2022, more than 3.5 thousand
teachers from Donetsk, Luhansk, Kherson and other regions were able to improve their qualifications
remotely based on such STEM centres.</p>
      <p>
        Continuing the scientific search, the authors also turn to sources that reveal the importance of
having STEM laboratories in Ukrainian schools. The monitoring study, which was carried out as
part of the determination of the readiness of pedagogical workers for the implementation of the New
Ukrainian School by the Institute of Educational Analytics, confirmed that the directors of general
secondary education institutions in the majority (94.5%) indicate the absence of STEM laboratories in
the educational institution. Only 277 institutions out of 5,440 (5.1%) participating in the study have such
laboratories [
        <xref ref-type="bibr" rid="ref22">22</xref>
        ]. To coordinate the work of STEM laboratories in the Institute of Modernization of the
Education Content of the Ministry of Education and Science of Ukraine, a separate unit is functioning,
the task of which is the implementation of the general strategy for the development and advocacy of
STEM education and the organization and conduct of educational events aimed at the popularization
of STEM education, career guidance work among students. The term of implementation of STEM
programs in Ukrainian educational institutions can be: short-term (from 02 to 24 hours); course (for
summer schools, courses, etc.) (from 24 to 80 hours); medium-term (annual) (from 80 to 120 hours);
long-term, continuous additional education (from 300 to 600 hours). It should be noted that, despite
state support for STEM education, Ukrainian schools face unresolved issues.
      </p>
      <p>
        Many schools lack STEM laboratories due to insuficient funds, as noted by 85.4% of institution
directors. Other barriers mentioned include a shortage of qualified specialists (6.8%), methodical support
(2.7%), and adequate premises (0.5%) for schools. According to the directors, these factors hinder the
establishment of STEM laboratories. Additionally, some respondents indicated that they believe it is
unnecessary for their students (2.8%) or that they have no desire (0.2%) to address this issue [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ].
      </p>
      <p>
        Out of the 277 general secondary education institutions that have STEM laboratories, approximately
one-third (31.8%) have used or are currently using state funding, while 32.9% have received assistance
from their community or city. For the organisation and functioning of the laboratory activities,
methodological support is provided by institutes of postgraduate pedagogical education (15.5%) and methodical
ofices (12.3%). Postgraduate pedagogical education institutions usually receive support from multiple
sources, such as international donors and non-governmental organizations, or both postgraduate
education institutions and students’ parents. In cases where no specific assistance was mentioned, school
principals noted that teachers utilised available equipment, and local entrepreneurs provided support.
Almost 47.2% of surveyed principals in educational institutions with STEM laboratories report a general
lack of necessary equipment, while 31.8% say they have the required equipment [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ].
      </p>
      <p>
        A survey of teachers conducted by the authors of this study indicated that among the main problems
pointed out by respondents in 2020 and 2023 are the following: lack of high-quality Internet – 64.7%,
insuficient material and technical support for students – 56.8%, fan outages electricity – 49.1%; low
level of self-organization and motivation of students – 41.3%; lack of support from parents – 30%; lack
of time due to increased workload on the teacher – 25.9%; insuficient level of material and technical
support of educational institutions – 23.1%; psychological dificulties during distance learning – 14.5%;
dificulties with distance learning of elementary school students – 13.9%; decrease in the quality of
providing educational services – 6.3%; insuficient level of digital competence of the teacher – 5.1%; the
institution was destroyed or damaged during the war – 4.4% [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ].
      </p>
      <p>All these data indicate the existence of a number of problems with the functioning of the STEM
environment in Ukrainian schools among which, for example, non-systematic use of STEM laboratories,
insuficient equipment, lack of access, insuficient training of teachers and low level of digital competence
and information literacy of teachers, etc.</p>
      <p>
        Among the monitoring tools of the STEAM-oriented environment, the results of international
comparative studies of the quality of education can be used, For example, the International Comparative
Study of the Quality of Natural and Mathematical Education (Trends in International Mathematics and
Science Study, TIMSS), the International Study of the Quality of Education (PISA), etc. In particular,
the Programme for International Student Assessment (PISA) was initiated over 23 years ago by the
Organisation for Economic Cooperation and Development (OECD) [
        <xref ref-type="bibr" rid="ref23">23</xref>
        ]. This study is one of the
important sources of information about secondary education in the world. More than 80 countries
participate in the study. Ukraine participated in the last two cycles of PISA – 2018 and 2022. Due to the
full-scale aggression of Russia against Ukraine and related security risks in the 2022 cycle, 15-year-olds
from only 18 regions of Ukraine were able to participate in the study [
        <xref ref-type="bibr" rid="ref23">23</xref>
        ].
      </p>
      <p>
        The results of the study demonstrated a significant decline in academic performance and quality of
education, as can be seen from the chart, especially in the average performance in mathematics, reading
and science in Ukraine and OECD countries between the 2018 and 2022 PISA cycles [
        <xref ref-type="bibr" rid="ref23">23</xref>
        ].
      </p>
      <p>
        It should be noted that the elaborated volume of the presented research allowed us to single out
the criteria that are used to evaluate the efectiveness of the educational environment. In particular,
Laniado et al. [
        <xref ref-type="bibr" rid="ref24">24</xref>
        ] propose a model to evaluate the efectiveness of environmental projects linking it to
the academic success of students. Academic achievement is the amount of educational material that a
student studies in a certain period. This can be any way in which the student has achieved short-term or
long-term academic goals in the learning environment. Yang et al. [
        <xref ref-type="bibr" rid="ref25">25</xref>
        ] propose to assess the student’s
educational achievements, testing and evaluation are conducted. We agree with García-Senín et al.
[
        <xref ref-type="bibr" rid="ref26">26</xref>
        ] who propose to using learning analytics to support STEAM students’ academic achievement and
self-regulated learning. El Mhouti et al. [
        <xref ref-type="bibr" rid="ref27">27</xref>
        ] suggest creating a tool to assess the quality of digital
learning resources. They discuss the criteria for evaluating these resources, specicfi to four dimensions:
academic quality, pedagogical quality, didactic quality, and technical quality. These criteria aim to
guide the design of easy-to-use evaluation instruments. The same position is taken by the Ukrainian
State Center for the Quality of Education, which annually surveys students and teachers regarding the
learning environment and readiness to use ICT.
      </p>
      <p>Based on the mentioned studies, it is worth highlighting such a criterion as the resources of the
educational environment. Indeed, an important criterion is the resources involved and available in the
educational environment. Resources can be divided into software products, facilities and tools, as well
as educational (teaching) and human (teachers, management staf, administrators).</p>
      <p>Another important criterion should be the availability of innovative training and educational methods.
Teacher training is critical because teachers need to be provided with regular learning opportunities to
stay on top of innovation. STEM teacher training workshops should be aimed at creating an opportunity
for secondary STEAM teachers to explore Science, Technology, Art, Engineering and Math (STEAM) as
well as Computer Science training through technology-rich demonstrations, classroom presentations,
hands-on activities and online connections.</p>
      <p>The next criterion is the degree of government attention to the problem of organisation and
functioning of the STEAM environment. This criterion is related to educational policy and the needs of education
and society in STEAM competencies of students.It should be noted that the issue of the implementation
of STEM education is declared in Ukrainian regulatory documents, partly also in schools where STEM
laboratories are established. For example, the Institute for the Modernization of the Content of Education
has a specialized department of STEM education, which has been experimenting in this area for many
years. However, as a result of crisis situations in the state, in connection with the state of war, the wide
distribution of STEM laboratories did not take place, and all this requires further support from the state.</p>
      <p>After considering the aforementioned points, we can outline the following criteria for monitoring a
STEAM-oriented educational environment: academic achievement (success in academics), resources
and educational environment for the implementation of STEAM education, innovative teaching and
learning methods, and government attention to the issue (table 1).</p>
      <p>Analyzing the criterion “academic achievements (academic success)” for teachers, the authors of the
article used information obtained during conferences, seminars, training, webinars, received applications
and web resources of conferences, seminars, and training. The main interest of the participants is in
the study of new digital tools and means for online education of students, in particular, the use of AI,
augmented reality, the creation of virtual laboratories and their efective use in lessons and the STEAM
environment. Today, teachers pay great attention to the safety and security of students and themselves
in the environment and the opportunity to learn safe tools for learning.</p>
    </sec>
    <sec id="sec-5">
      <title>5. Conclusions</title>
      <p>
        The issue of implementation and development of a STEAM-oriented environment is important today
for Ukrainian education. The conducted research showed that today obtaining a quality education
is a challenge and a problem since a full-scale invasion is ongoing in the country. STEAM education
technologies make it possible to raise the educational process to another level and ensure accessibility
and continuity of education, creating opportunities to acquire the necessary skills for citizens of the
21st century. Turning to the issues related to the support of the STEAM environment in secondary
education institutions, it was found that today there are issues that should attract the attention of
educators, in particular, the insuficient level of information literacy and digital skills. This is also
evidenced by research conducted by both Ukrainian and foreign researchers [
        <xref ref-type="bibr" rid="ref19 ref20">19, 20</xref>
        ]. In addition, the
need for continuous professional development of teachers who implement STEAM education and
its elements is evidenced by research and surveys conducted by Papadakis et al. [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ], Kalogiannakis
and Papadakis [
        <xref ref-type="bibr" rid="ref16">16</xref>
        ]. As a result of the analysis and approaches used in other countries, the STEAM
environment is a remarkable achievement, and its efectiveness needs monitoring. Based on the study
conducted and the experience of supporting such an environment, criteria for the efectiveness of the
STEAM environment were proposed, which can serve as a starting point for the evaluation system of
the educational environment and STEAM education.
      </p>
      <p>The main emphasis was placed on the criteria by which the presence and needs of such categories
as teachers and students should be considered, ensuring the functioning of the environment. We
consider it expedient to pay attention to such urgent problems of the lack of STEM laboratories and the
environment, which single out teachers, namely: lack of funds, lack of appropriate specialists, lack of
familiarity with equipment, and lack and insuficiency of methodical support for teachers. These and
other circumstances not only reduce the motivation of teachers to develop STEM and STEAM education
and create the necessary environment at school but also cancel the opportunity for students to develop,
and acquire the necessary skills and competencies for life and future work in a digital, information
society.</p>
      <p>The authors analysed the reports of state institutions and identified four important criteria that can
be used to monitor the efectiveness of a STEAM-oriented environment, namely: academic achievement
(success in academics), resources and educational environment for the implementation of STEAM
education, innovative teaching and learning methods, and state (government) attention to the issue.
These criteria and related indicators of the efectiveness of the STEAM environment take into account
the interests of teachers and students.</p>
      <p>It should be noted that our research is an attempt to systematize teachers’ attitudes toward such
components of the environment as its software, content, and compliance with the interests of students.
Therefore, the best practices of a STEAM-oriented educational environment, the role of such an
environment in overcoming educational losses, and a method of evaluating the efectiveness of a
STEAM-oriented educational environment taking into account the specified criteria can be perspectives
for further scientific research.</p>
      <p>Based on the presented study results several recommendations can be made to evaluate the
STEAMoriented environment in secondary education institutions: state institutions and organisations should
develop plans to support the STEAM environment in schools to support distance learning of students
and education in general, especially in conditions of full-scale aggression by the Russian Federation;
teacher training programs can be guided by developed criteria for teaching teachers to use STEAM tools
in the classroom; educational administrations can conduct surveys of teachers and students, taking into
account the proposed criteria, expanding them with other indicators to determine the state of use and
efectiveness of STEAM environments and STEAM education in their regions.</p>
      <p>The problems common to many institutions in the implementation of STEM education, which require
further solutions at various levels of educational policy, should include insuficient funding and, as
a result, insuficient equipment of STEM laboratories, unpreparedness of personnel and low level of
information literacy of teachers, lack of systematic monitoring of the efectiveness of STEAM -oriented
environment.</p>
      <p>In the future, we see the expansion of tools for monitoring the quality of education due to the addition
of developed criteria to the study of the state of use of new teaching aids. Studying the efectiveness of
the STEAM environment for continuous education, starting from elementary grades, should become an
important area of both research and practical experiments, as well as the exchange of experience from
advanced educational practices of STEAM education.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>Verkhovna</given-names>
            <surname>Rada</surname>
          </string-name>
          of Ukraine, On education,
          <year>2017</year>
          . URL: https://zakon.rada.gov.ua/laws/show/ 2145-19/page?lang=en.
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>Verkhovna</given-names>
            <surname>Rada</surname>
          </string-name>
          of Ukraine, On general secondary education,
          <year>2020</year>
          . URL: https://zakon.rada.gov. ua/laws/show/651-14?lang=en.
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>Verkhovna</given-names>
            <surname>Rada</surname>
          </string-name>
          of Ukraine,
          <source>On scientific and scientific-technical activities</source>
          ,
          <year>2015</year>
          . URL: https: //zakon.rada.gov.ua/laws/show/848-19?lang=en.
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>S. O.</given-names>
            <surname>Semerikov</surname>
          </string-name>
          ,
          <string-name>
            <surname>M. M. Mintii</surname>
            ,
            <given-names>I. S.</given-names>
          </string-name>
          <string-name>
            <surname>Mintii</surname>
          </string-name>
          ,
          <article-title>Review of the course “Development of Virtual and Augmented Reality Software” for STEM teachers: implementation results and improvement potentials</article-title>
          , in: S. H.
          <string-name>
            <surname>Lytvynova</surname>
            ,
            <given-names>S. O.</given-names>
          </string-name>
          <string-name>
            <surname>Semerikov</surname>
          </string-name>
          (Eds.),
          <source>Proceedings of the 4th International Workshop on Augmented Reality in Education (AREdu</source>
          <year>2021</year>
          ), Kryvyi Rih, Ukraine, May
          <volume>11</volume>
          ,
          <year>2021</year>
          , volume
          <volume>2898</volume>
          <source>of CEUR Workshop Proceedings, CEUR-WS.org</source>
          ,
          <year>2021</year>
          , pp.
          <fpage>159</fpage>
          -
          <lpage>177</lpage>
          . URL: https: //ceur-ws.
          <source>org/</source>
          Vol-
          <volume>2898</volume>
          /paper09.pdf.
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>R. P.</given-names>
            <surname>Kukharchuk</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T. A.</given-names>
            <surname>Vakaliuk</surname>
          </string-name>
          ,
          <string-name>
            <given-names>O. V.</given-names>
            <surname>Zaika</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A. V.</given-names>
            <surname>Riabko</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. G.</given-names>
            <surname>Medvediev</surname>
          </string-name>
          ,
          <article-title>Implementation of STEM learning technology in the process of calibrating an NTC thermistor and developing an electronic thermometer based on it</article-title>
          , in: S. Papadakis (Ed.),
          <source>Joint Proceedings of the 10th Illia O. Teplytskyi Workshop on Computer Simulation in Education, and Workshop on Cloudbased Smart Technologies for Open Education (CoSinEi and CSTOE</source>
          <year>2022</year>
          )
          <article-title>co-located with ACNS Conference on Cloud and Immersive Technologies in Education (CITEd</article-title>
          <year>2022</year>
          ), Kyiv, Ukraine, December
          <volume>22</volume>
          ,
          <year>2022</year>
          , volume
          <volume>3358</volume>
          <source>of CEUR Workshop Proceedings, CEUR-WS.org</source>
          ,
          <year>2022</year>
          , pp.
          <fpage>39</fpage>
          -
          <lpage>52</lpage>
          . URL: https://ceur-ws.
          <source>org/</source>
          Vol-
          <volume>3358</volume>
          /paper25.pdf.
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <surname>M. M. Mintii</surname>
            ,
            <given-names>N. M.</given-names>
          </string-name>
          <string-name>
            <surname>Sharmanova</surname>
            ,
            <given-names>A. O.</given-names>
          </string-name>
          <string-name>
            <surname>Mankuta</surname>
            ,
            <given-names>O. S.</given-names>
          </string-name>
          <string-name>
            <surname>Palchevska</surname>
            ,
            <given-names>S. O.</given-names>
          </string-name>
          <string-name>
            <surname>Semerikov</surname>
          </string-name>
          ,
          <article-title>Selection of pedagogical conditions for training STEM teachers to use augmented reality technologies in their work</article-title>
          ,
          <source>Journal of Physics: Conference Series</source>
          <volume>2611</volume>
          (
          <year>2023</year>
          )
          <article-title>012022</article-title>
          . doi:
          <volume>10</volume>
          .1088/
          <fpage>1742</fpage>
          -6596/ 2611/1/012022.
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <surname>M. M. Mintii</surname>
          </string-name>
          ,
          <article-title>STEM education and personnel training: Systematic review</article-title>
          ,
          <source>Journal of Physics: Conference Series</source>
          <volume>2611</volume>
          (
          <year>2023</year>
          )
          <article-title>012025</article-title>
          . doi:
          <volume>10</volume>
          .1088/
          <fpage>1742</fpage>
          -6596/2611/1/012025.
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          <article-title>[8] Ministry of Education and Science of Ukraine, About the order to conduct education quality monitoring</article-title>
          ,
          <year>2020</year>
          . URL: https://zakon.rada.gov.ua/laws/show/z0154-20#Text.
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>L. V.</given-names>
            <surname>Kondrashova</surname>
          </string-name>
          ,
          <article-title>Monitoring of educational processes in higher school</article-title>
          ,
          <source>in: 2005 15th International Crimean Conference Microwave &amp; Telecommunication Technology</source>
          , volume
          <volume>1</volume>
          ,
          <year>2005</year>
          , pp.
          <fpage>129</fpage>
          -
          <lpage>131</lpage>
          Vol.
          <volume>1</volume>
          . doi:
          <volume>10</volume>
          .1109/CRMICO.
          <year>2005</year>
          .
          <volume>1564838</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>L.</given-names>
            <surname>Hrynevych</surname>
          </string-name>
          ,
          <string-name>
            <given-names>O.</given-names>
            <surname>Linnik</surname>
          </string-name>
          ,
          <string-name>
            <surname>J. Herczyński,</surname>
          </string-name>
          <article-title>The new Ukrainian school reform: Achievements, developments and challenges</article-title>
          ,
          <source>European Journal of Education</source>
          <volume>58</volume>
          (
          <year>2023</year>
          )
          <fpage>542</fpage>
          -
          <lpage>560</lpage>
          . doi:
          <volume>10</volume>
          .1111/ejed. 12583.
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>N.</given-names>
            <surname>Batechko</surname>
          </string-name>
          ,
          <article-title>Quality management in higher education sphere: Methodological aspect</article-title>
          ,
          <source>Osvitolohiya</source>
          (
          <year>2017</year>
          )
          <fpage>156</fpage>
          -
          <lpage>162</lpage>
          . doi:
          <volume>10</volume>
          .28925/
          <fpage>2226</fpage>
          -
          <lpage>3012</lpage>
          .
          <year>2017</year>
          .
          <volume>6</volume>
          .156162.
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>T.</given-names>
            <surname>Lukina</surname>
          </string-name>
          ,
          <article-title>Change of the model of state final attestation of gymnasium graduates: Organizational and methodological aspects</article-title>
          ,
          <source>Ukrainian Educational Journal</source>
          (
          <year>2020</year>
          )
          <fpage>42</fpage>
          -
          <lpage>50</lpage>
          . doi:
          <volume>10</volume>
          .32405/
          <fpage>2411</fpage>
          -1317-2020-2-
          <fpage>42</fpage>
          -50.
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>O.</given-names>
            <surname>Denysiuk</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Tytarenko</surname>
          </string-name>
          ,
          <article-title>Indicators of the New Ukrainian School as a tool for measuring education reform implementation by general education teachers</article-title>
          ,
          <source>Educational Analytics of Ukraine</source>
          (
          <year>2023</year>
          )
          <fpage>91</fpage>
          -
          <lpage>103</lpage>
          . URL: https://science.iea.gov.ua/en/2023_4en_
          <fpage>91</fpage>
          -
          <lpage>103</lpage>
          /.
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <given-names>L.</given-names>
            <surname>Gong</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C. Y.</given-names>
            <surname>Wei</surname>
          </string-name>
          ,
          <article-title>Promoting Innovative Teaching Quality Monitoring (TQM) System: A Case Study</article-title>
          at University of Sanya, Hainan, China, in
          <source>: Proceedings of the 10th International Conference on Business, Accounting, Finance and Economics (BAFE</source>
          <year>2022</year>
          ), Atlantis Press,
          <year>2023</year>
          , pp.
          <fpage>401</fpage>
          -
          <lpage>413</lpage>
          . doi:
          <volume>10</volume>
          .2991/978-2-
          <fpage>494069</fpage>
          -99-2_
          <fpage>30</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <given-names>M.</given-names>
            <surname>Pérez Torres</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D. Couso</given-names>
            <surname>Lagarón</surname>
          </string-name>
          ,
          <string-name>
            <surname>C.</surname>
          </string-name>
          <article-title>Marquez Bargalló, Evaluation of STEAM Project-Based Learning (STEAM PBL) Instructional Designs from the STEM Practices Perspective</article-title>
          ,
          <source>Education Sciences</source>
          <volume>14</volume>
          (
          <year>2024</year>
          )
          <article-title>53</article-title>
          . doi:
          <volume>10</volume>
          .3390/educsci14010053.
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <string-name>
            <given-names>M.</given-names>
            <surname>Kalogiannakis</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Papadakis</surname>
          </string-name>
          ,
          <article-title>The Use of Developmentally Mobile Applications for Preparing Pre-Service Teachers to Promote STEM Activities in Preschool Classrooms</article-title>
          , in: S. Papadakis, M. Kalogiannakis (Eds.),
          <source>Mobile Learning Applications in Early Childhood Education</source>
          ,
          <year>2020</year>
          , p.
          <fpage>82</fpage>
          -
          <lpage>100</lpage>
          . doi:
          <volume>10</volume>
          .4018/978-1-
          <fpage>7998</fpage>
          -1486-3.
          <year>ch005</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [17]
          <string-name>
            <given-names>S.</given-names>
            <surname>Papadakis</surname>
          </string-name>
          ,
          <article-title>Evaluating a game-development approach to teach introductory programming concepts in Secondary Education</article-title>
          ,
          <source>International Journal of Technology Enhanced Learning</source>
          <volume>1</volume>
          (
          <year>2020</year>
          )
          <article-title>1</article-title>
          . doi:
          <volume>10</volume>
          .1504/ijtel.
          <year>2020</year>
          .
          <volume>10024287</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [18]
          <string-name>
            <given-names>S.</given-names>
            <surname>Papadakis</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Vaiopoulou</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E.</given-names>
            <surname>Sifaki</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Stamovlasis</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Kalogiannakis</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Vassilakis</surname>
          </string-name>
          ,
          <article-title>Factors That Hinder in-Service Teachers from Incorporating Educational Robotics into Their Daily or Future Teaching Practice</article-title>
          ,
          <source>in: Proceedings of the 13th International Conference on Computer Supported Education - Volume</source>
          <volume>2</volume>
          : CSEDU, SciTePress,
          <year>2021</year>
          , pp.
          <fpage>55</fpage>
          -
          <lpage>63</lpage>
          . doi:
          <volume>10</volume>
          .5220/0010413900550063.
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [19]
          <string-name>
            <given-names>E.</given-names>
            <surname>Tzagkaraki</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Papadakis</surname>
          </string-name>
          ,
          <string-name>
            <surname>M.</surname>
          </string-name>
          <article-title>Kalogiannakis, Exploring the Use of Educational Robotics in Primary School and Its Possible Place in the Curricula</article-title>
          , in: M.
          <string-name>
            <surname>Malvezzi</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          <string-name>
            <surname>Alimisis</surname>
          </string-name>
          , M. Moro (Eds.),
          <article-title>Education in &amp; with Robotics to Foster 21st-Century Skills</article-title>
          , volume
          <volume>982</volume>
          <source>of Studies in Computational Intelligence</source>
          , Springer International Publishing, Cham,
          <year>2021</year>
          , pp.
          <fpage>216</fpage>
          -
          <lpage>229</lpage>
          . doi:
          <volume>10</volume>
          . 1007/978-3-
          <fpage>030</fpage>
          -77022-8_
          <fpage>19</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          [20]
          <string-name>
            <given-names>L.</given-names>
            <surname>Lukianova</surname>
          </string-name>
          ,
          <string-name>
            <given-names>O.</given-names>
            <surname>Ovcharuk</surname>
          </string-name>
          , Information Literacy and
          <article-title>Digital Inclusion: Challenges of the Modern Information Educational Environment in Ukraine</article-title>
          , in: Ł.
          <string-name>
            <surname>Tomczyk</surname>
            ,
            <given-names>F. D.</given-names>
          </string-name>
          <string-name>
            <surname>Guillén-Gámez</surname>
            ,
            <given-names>J. RuizPalmero</given-names>
          </string-name>
          , A. Habibi (Eds.), From Digital Divide to Digital Inclusion: Challenges,
          <source>Perspectives and Trends in the Development of Digital Competences</source>
          , Springer Nature Singapore, Singapore,
          <year>2023</year>
          , pp.
          <fpage>541</fpage>
          -
          <lpage>565</lpage>
          . doi:
          <volume>10</volume>
          .1007/
          <fpage>978</fpage>
          -981-99-7645-4_
          <fpage>25</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          [21]
          <string-name>
            <given-names>N.</given-names>
            <surname>Soroko</surname>
          </string-name>
          ,
          <article-title>Methodology for Teachers' Digital Competence Developing through the Use of the STEAM-oriented Learning Environment</article-title>
          , in: O.
          <string-name>
            <surname>Sokolov</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          <string-name>
            <surname>Zholtkevych</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          <string-name>
            <surname>Yakovyna</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          <string-name>
            <surname>Tarasich</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          <string-name>
            <surname>Kharchenko</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          <string-name>
            <surname>Kobets</surname>
            ,
            <given-names>O.</given-names>
          </string-name>
          <string-name>
            <surname>Burov</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          <string-name>
            <surname>Semerikov</surname>
          </string-name>
          , H. Kravtsov (Eds.),
          <source>Proceedings of the 16th International Conference on ICT in Education, Research and Industrial Applications</source>
          . Integration, Harmonization and
          <string-name>
            <given-names>Knowledge</given-names>
            <surname>Transfer</surname>
          </string-name>
          . Volume II: Workshops, Kharkiv, Ukraine,
          <source>October 06- 10</source>
          ,
          <year>2020</year>
          , volume
          <volume>2732</volume>
          <source>of CEUR Workshop Proceedings, CEUR-WS.org</source>
          ,
          <year>2020</year>
          , pp.
          <fpage>1260</fpage>
          -
          <lpage>1271</lpage>
          . URL: https://ceur-ws.
          <source>org/</source>
          Vol-
          <volume>2732</volume>
          /20201260.pdf.
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          [22]
          <article-title>Information and analytical materials “Study on the implementation of the NUS reform in pilot educational institutions (basic school</article-title>
          )”,
          <year>2023</year>
          . URL: https://iea.gov.ua/wp-content/uploads/2023/ 10/pilot_nus
          <source>_report_2023.pdf.</source>
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          [23]
          <article-title>Ukrainian Center for Evaluation of the Quality of Education, PISA</article-title>
          <year>2022</year>
          ,
          <year>2024</year>
          . URL: https:// testportal.gov.ua/tag/pisa-2022/.
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          [24]
          <string-name>
            <given-names>E.</given-names>
            <surname>Laniado</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Vaghi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E.</given-names>
            <surname>Amodeo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Capriolo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Mascolo</surname>
          </string-name>
          ,
          <article-title>A model to evaluate the efectiveness of environmental projects, in: The Sustainable City VIII, SC13</article-title>
          , WIT Press,
          <year>2013</year>
          , pp.
          <fpage>755</fpage>
          -
          <lpage>765</lpage>
          . doi:
          <volume>10</volume>
          .2495/sc130642.
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          [25]
          <string-name>
            <given-names>J.</given-names>
            <surname>Yang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Qin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <source>Efect of the Duration of Preschool Attendance on Academic Achievements-Evidence from PISA</source>
          <year>2018</year>
          ,
          <source>Child Indicators Research</source>
          <volume>17</volume>
          (
          <year>2023</year>
          )
          <fpage>1</fpage>
          -
          <lpage>22</lpage>
          . doi:
          <volume>10</volume>
          .1007/ s12187-023-10095-1.
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          [26]
          <string-name>
            <given-names>S.</given-names>
            <surname>García-Senín</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Arguedas</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Daradoumis</surname>
          </string-name>
          ,
          <article-title>Using learning analytics to support STEAM students' academic achievement and self-regulated learning</article-title>
          ,
          <source>Research on Education and Media</source>
          <volume>14</volume>
          (
          <year>2022</year>
          )
          <fpage>36</fpage>
          -
          <lpage>45</lpage>
          . doi:
          <volume>10</volume>
          .2478/rem-2022-0005.
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          [27]
          <string-name>
            <given-names>A.</given-names>
            <surname>El Mhouti</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Nasseh</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Erradi</surname>
          </string-name>
          ,
          <article-title>How to evaluate the quality of digital learning resources?</article-title>
          ,
          <source>International Journal of Computer Science Research and Application</source>
          <volume>03</volume>
          (
          <year>2013</year>
          )
          <fpage>27</fpage>
          -
          <lpage>36</lpage>
          . URL: https: //www.researchgate.net/publication/260392089.
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>