This paper presents an evaluation methodology suitable for a designbased learning approach with the Internet of Things technologies for teaching STEM courses. It also aims to evaluate the learning outcomes that such teaching interventions bring to students. The particular educational approach to be evaluated is an ongoing research topic, focusing on IoT sensor data used in school education, for understanding and raising awareness for aquatic ecosystems. This study aims to assess the impact of incorporating this proposal for IoT based teaching interventions in the subjects of Physics, Informatics and Electronics into the regular curriculum of Greek secondary education, as well as to add to the eld literature.
Copyright c by the paper's authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). knowledge of science, technology, engineering and mathematics is applied, at: a) knowledge and understanding of basic physical and chemical parameters of surface water, and b) the ability to correlate, interpret and evaluate changes in the physical and chemical parameters of water in lakes and rivers.
The design of the device used to measure the physicochemical parameters is built using the open-source (hardware and software) Arduino platform. The Arduino platform has been chosen as it is well-established electronics prototyping platform and it o ers multiple variations that provide us with the appropriate core components for building sensing devices with the lowest cost and e ort [PAMC17, CKN05a, CKN05b]. The goal has been to develop an appropriate small, relatively inexpensive, portable device that can easily be deployed, capable of sensing certain aquatic parameters that were assessed as being the most appropriate for reasoning about the ecosystem [TAMC19]. This device has been initially deployed in tests reported in several experiments in sweet water and saltwater, as reported in [TAR+18, TMM+18, TAMC19, TKD+19].
Water quality depends on several di erent variables, the primary ones being water temperature, dissolved oxygen, conductivity, pH, salinity, total dissolved solids, hardness and sedimentation. Taking into account that temperature and dissolved oxygen are the two most important factors a ecting water ecosystems, sensors for Temperature, total dissolved solids, dissolved oxygen and pH were the ones deployed, as the most important for the exercise.
According to Caine [CC91], learning is more e ective when students are involved in complex experiences and are given the opportunity to actively process what they are learning. The inquiry-based learning approach, which focuses on the creation of a construct by the students themselves [GPvEJ13], has been proposed by a large number of researchers [Bro92, BSM+91, PH91, KR97, CS08, CDF+09]. Design-Based Learning is a combination of small research tasks and problem-based learning aimed at devising a technological object. Students are involved in solving realistic design problems that make sense to them, in a real context, by following the engineering design process.
In a set of proposed educational activities, students undertake to design and operate an automated device to monitor the environmental variables of a lake or river. They then utilize the collected data to address potential water pollution issues. ecosystems. Through these activities, students are expected to develop knowledge and skills in mathematics (equation systems, data logging, graphing, statistical analysis), natural sciences (physicochemical correlation, error measurements and estimation, electrical circuits), application engineering (device design, sensor calibration, measurement logging) and technology (programming and code debugging).
The teaching intervention involves a design-based learning approach, which is based on proposing a solution for a speci c problem, for which a technological object is devised. On the other hand, the students use IoT technology to study in real-time a natural ecosystem by observing, analyzing and interpreting real data that is constantly updated [TAMC19, TKD+19, AZA+18, ZAC18]. The purpose of the artefact created is to motivate and engage in learning. It also aims to the students themselves assuming the responsibility of learning, to seek and acquire knowledge, skills and practices through the design challenge as well as practising problem-solving techniques.
The availability of actual measurements of parameters for water quality enables several diverse educationrelated applications and scenarios. For example, the school community can use collected data and analytics during a class to explain phenomena related to the parameters monitored. They can also, organize projects where students monitor environmental parameters to enable them to make informed decisions, acting as responsible citizens and address complex environmental issues. The approach proposed here provides a rich context for learning and guides students to sustained inquiry and revision.
While IoT technologies begin to be introduced in an educational context, there are not yet concrete strategies de ned for introducing IoT in education or methodologies that provide certain steps that the educational stakeholders can follow to yield successful educational results. This research attempts to pave the way for the strategies needed, providing steps that are understandable and usable by the STEM teachers community, towards introducing IoT in education for environmental awareness, and aiming speci cally in understanding the aquatic ecosystems. In this paper, the focus of attention is therefore in formulating the research questions, towards a structured evaluation of the educational results achieved with the approach that is described above. The evaluation structure proposed in this paper can be used by researchers and educators seeking to introduce IoT in education, to evaluate the results of their interventions. It remains to be used and further elaborated in the context of the broader research that was described above. 2
Based on the literature and existing research, the following main research questions can be formulated: 1. Does the involvement of students in the proposed teaching intervention in uence their cognitive improvement in STEM subjects? Can students use new knowledge to interpret phenomena in inland water ecosystems? 2. Is there a change in students' interest in STEM sciences after the teaching intervention? 3. Does the involvement of students in teaching interventions a ect the retention of the knowledge they have acquired for a long period without them engaging again in the subject? 4. Does the involvement of students in engineering activities a ect their ability to solve open problems? [FKD+05] The above research questions can be broken down into the following sub-questions: 1. \Does the involvement of students in the proposed teaching intervention in uence their cognitive improvement in STEM subjects? Can students use new knowledge to interpret phenomena in inland water ecosystems?" (a) Are students' cognitive improvement in STEM subjects in uenced by their involvement in teaching intervention, concerning the Lyceum type (GEL - EPAL) and the area/course they are attending? (b) Does students' cognitive improvement in STEM subjects in uence participation in teaching per sheet? (c) Is students' cognitive improvement in STEM subjects in uenced by their involvement in teaching intervention, by their performance category? The second research question can be broken down into the following sub-questions: (a) Is there a change in the interest of all students in science (Physical Sciences and Applied Engineering) before and after intervention in the Lyceum type (GEL - EPAL)? (b) Is there a change in students' interest in Science (Physical Sciences and Applied Engineering) per sheet before and after intervention? (c) Is there a change in students' interest in science (Natural Sciences and Applied Engineering) by category of school performance to which they belong, before and after intervention? (d) Is there a change in students' interest in employment in the eld of Environmental Sciences / Natural
Sciences / Applied Engineering? (e) If students in the intervention group are interested in the implementation of the intervention, do the students themselves re ect the interest that they have caused?
(a) Is there a di erence in the retention of the cognitive outcomes achieved in all intervention group students after the teaching intervention, concerning the type of high school (GEL - EPAL) and the Sector / Direction they are attending? (b) Is there a di erence in the retention of the cognitive achievements of the intervention group students per sheet after the teaching intervention? (c) Is there a di erence in the retention of the cognitive outcomes achieved by the intervention group students, by their performance category? The question will examine whether students after the intervention can identify and analyze a problem, formulate speci c questions that they can answer, seek relevant knowledge, formulate solutions, evaluate solutions, make decisions, design controls and evaluate the tested solution [KCC+03]. 3
Several experiments were conducted to elaborate on the rst phased of this research. To investigate any pollution problems in water bodies located in their area, students planed their study which follows the scienti c method and tries to answer inquiry questions by collecting and analyzing data, concluding conclusions and making recommendations. In educational activities for monitoring the ecological parameters of aquatic ecosystems, thirty students from two high schools have designed the research by choosing the lake to get samples, setting the inquiry questions and identifying the parameters they would record. Then they constructed and programmed the device to measure the water quality physicochemical parameters. After analyzing the data, they identi ed the correlations between the variables and drew up a nal report. The rst measures were taken with Arduinosensor kit (see Figure 1) in Lake Koumoundourou, (in intensely industrialized and urbanized areas of Greece, see Figure 3). The students had the opportunity to also make real-time measurements and to be informed by a team of researchers of the aquatic environment [TKD+19]. The students were divided into three groups and cyclically involved in exploratory activities: (a) macroscopic recognition with stereoscopes and associated keys, and ecological quality assessment using microscopes; (b) laboratory chemical analyzes for the determination of nutrients using a portable spectrophotometer; and (c) measurements of physicochemical parameters (pH, temperature, electrical conductivity, dissolved oxygen and turbidity) with portable multiparameter instrument.
Following the rst experiments described earlier, more concrete research and evaluation are planned to take place between November 2019 and January 2020. Approximately 63 second grade students of Vocational and General High School from three schools in Attica will participate: 20% from 1st Vocational School (1st EPAL) of Acharnon, 21% from 3rd Vocational School (in Greek initials this is referred as the 3rd EPAL) of Acharnon and 22% from the 3rd Philadelphia General Hight School (Greek General Educational Lyceum, GEL). Students of 1st EPAL Acharnon attend Computer Science courses, 2nd EPAL Acharnes courses in Electrical, Electronics and Automation, while GEL students come from the Positive Studies orientation group. Schools were selected in which teachers' associations will be given the required number of teaching hours to implement the teaching intervention. Nevertheless, it is non-probability sampling.
The research on the Secondary School students will be conducted at the beginning of the school year to have as little experience as possible, and therefore better study the e ectiveness of the teaching intervention.
The research intervention will last eight teaching hours and will consist of ve stages. Initially, a pre-test will be conducted, followed by teaching interventions, then post-test and semi-structured interview with several students.
Finally, students will complete the assessment forms approximately two months after the intervention to check for retain-test. All stages will be conducted by the author with the class teachers present and contributing to the exercise.
Prior to the implementation of the teaching intervention, a pilot application will be preceded by an additional section of the Vocational (EPAL) Department of Informatics (1st EPAL of Korydallos) in order to answer rst the general methodological and didactic issues related to the implementation: the role of the student population, the students' responses worksheets and time limits corresponding to their scope, students 'attitudes about the laboratory and programming environment, and students' views on the concepts being studied. 4
To study the rst research question for triangulation purposes the tools to be used are (a) pre-post tests, (b) observation and (c) semi-structured interviews. In addition to the pre-post tests completed by all participating students, the results of the personal observation of the researcher during the teaching intervention and the students' responses to the semi-structured interviews will be recorded and analysed. To observe the duration of the intervention, a group of students will be selected in each department, in which the researcher will try to focus their interest, the degree of cooperation, the way they work, and other related factors. By the end of the intervention and completing the post-test, approximately 12 students (the ones belonging to the focus groups) are expected to answer a set of prede ned questions during an interview.
The questions for the semi-structured interview with students are: 1. What did your teaching approach look like? 2. Did you nd the teaching approach that was followed interesting? 3. Why was (or was it not, if not the question above) interesting in your opinion? 4. Was this procedure enjoyable?
9. Have you been motivated to learn more compared to the usual form of teaching? 10. Do you think you linked what you did to the corresponding theory?
For the study of the second research issue, the same questionnaire as the rst one will be used again after two months. For the third research question, a problem will be solved and the students will put in the correct order the solution steps and nally, the Guzey or Mahoney questionnaire [Mah10] will be customized for the fourth research topic. 5
Data analysis will use the ANOVA (or MANOVA) tool to test statistically signi cant di erences between students in the original questionnaires, which will be taken into account in the statistical processing of the post-test performance. A t-test and ANOVA analysis of variance will also be used to test for di erences in post-test performance among the students in the three sections to investigate whether the lack of systematic randomization of the students in the sections in some way a ects their performance and research conclusions.
Improvement in student performance will be assessed based on their Hake gain. Variations in students' pretest and post-test performance, seen as a factor of variation among students in each department (within-subject factor) will be analysed, while the use of the di erent classes is also seen as a factor of variation among the students of the three sections (between-subject factor).
Besides, the results of the tests will be statistically analyzed for their internal reliability by calculating Cronbach. 6
The assumption behind this research is that education-focused real-world IoT deployment can help to form a better understanding of our environment and promote sustainable activities, starting at a school level. By using inexpensive and easily available IoT infrastructure to measure, and then visualize and re ect on the data combinations, a more meaningful understanding of our environment is informed, while at the same time citizen science is supported by adding to data sets. Applying to learn in this context is regarded as an e ective way to retain knowledge and enrich students understanding. This assumption remains to be validated by appropriately formed evaluations, that are designed with DBL with IoT artefacts in mind.
In this paper, we have attempted to describe the research methodology that will be applied in evaluating an IoT design-based learning approach to teach STEM courses. The learning outcomes to students with such teaching interventions will be assessed. An issue remains weather the research questions mentioned above are the ones most appropriate to illustrate the relationship between the DBL teaching approach and students' cognitive development and to assess the contribution of IoT technology. It may also be a case that the number of research questions is too broad and therefore the questions need to focus on conceptual and procedural knowledge. [Bro92]
Ann L Brown. Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. The journal of the learning sciences, 2(2):141{178, 1992. [BSM+91] Phyllis C Blumenfeld, Elliot Soloway, Ronald W Marx, Joseph S Krajcik, Mark Guzdial, and Annemarie Palincsar. Motivating project-based learning: Sustaining the doing, supporting the learning.
Educational psychologist, 26(3-4):369{398, 1991.
Renate Nummela Caine and Geo rey Caine. Making connections: Teaching and the human brain.
New York, 1991. [CDF+09] Ioannis Chatzigiannakis, Shlomi Dolev, Sandor P. Fekete, Othon Michail, and Paul G. Spirakis. Not all fair probabilistic schedulers are equivalent. In Tarek Abdelzaher, Michel Raynal, and Nicola Santoro, editors, Principles of Distributed Systems, pages 33{47, Berlin, Heidelberg, 2009. Springer Berlin Heidelberg. [CKN05a] Ioannis Chatzigiannakis, Athanasios Kinalis, and Sotiris Nikoletseas. An adaptive power conservation scheme for heterogeneous wireless sensor networks with node redeployment. In Proceedings of the seventeenth annual ACM symposium on Parallelism in algorithms and architectures, pages 96{105.
ACM, 2005. [CKN05b] Ioannis Chatzigiannakis, Athanasios Kinalis, and Sotiris Nikoletseas. Power conservation schemes for energy e cient data propagation in heterogeneous wireless sensor networks. In 38th Annual Simulation Symposium, pages 60{71. IEEE, 2005. [CS08]
Ioannis Chatzigiannakis and Paul G Spirakis. The dynamics of probabilistic population protocols.
In International Symposium on Distributed Computing, pages 498{499. Springer, 2008. [FKD+05] David Fortus, Joseph Krajcik, Ralph Charles Dershimer, Ronald W Marx, and Rachel MamlokNaaman. Design-based science and real-world problem-solving. International Journal of Science Education, 27(7):855{879, 2005. [GPvEJ13] SM Gomez Puente, M van Eijck, and W Jochems. Facilitating the learning process in design-based learning practices: An investigation of teachers actions in supervising students. Research in Science & Technological Education, 31(3):288{307, 2013. [KCC+03] Janet L Kolodner, Paul J Camp, David Crismond, Barbara Fasse, Jackie Gray, Jennifer Holbrook, Sadhana Puntambekar, and Mike Ryan. Problem-based learning meets case-based reasoning in the middle-school science classroom: Putting learning by design (tm) into practice. The journal of the learning sciences, 12(4):495{547, 2003. [KR97]
Roni Khardon and Dan Roth. Learning to reason. Journal of the ACM (JACM), 44(5):697{725, 1997. [MAC+18] Georgios Mylonas, Dimitrios Amaxilatis, Ioannis Chatzigiannakis, Aris Anagnostopoulos, and Federica Paganelli. Enabling sustainability and energy awareness in schools based on iot and real-world data. IEEE Pervasive Computing, 17(4):53{63, 2018. [Mah10]
Mark Patrick Mahoney. Students' attitudes toward stem: Development of an instrument for high school stem-based programs. Journal of Technology Studies, 36(1):24{34, 2010. [MAL+17] Georgios Mylonas, Dimitrios Amaxilatis, Helena Leligou, Theodore Zahariadis, Emmanouil Zacharioudakis, Joerg Hofstaetter, Andreas Friedl, Federica Paganelli, Giovanni Cu aro, and Jimm Lerch. Addressing behavioral change towards energy e ciency in european educational buildings. In 2017 Global Internet of Things Summit (GIoTS), pages 1{6. IEEE, 2017. [PAMC17] Lidia Pocero, Dimitrios Amaxilatis, Georgios Mylonas, and Ioannis Chatzigiannakis. Open source iot meter devices for smart and energy-e cient school buildings. HardwareX, 1:54 { 67, 2017. [PH91]
Seymour Papert and Idit Harel. Situating constructionism. Constructionism, 36(2):1{11, 1991. [TAMC19] Chrysanthi Tziortzioti, Dimitrios Amaxilatis, Irene Mavrommati, and Ioannis Chatzigiannakis. Iot sensors in sea water environment: Ahoy! experiences from a short summer trial. Electronic Notes in Theoretical Computer Science, 343:117 { 130, 2019. The proceedings of AmI, the 2018 European Conference on Ambient Intelligence. [TAR+18] Chrysanthi Tziortzioti, Giuseppe Andreetti, Lucia Rodino, Irene Mavrommati, Andrea Vitaletti, and Ioannis Chatzigiannakis. Raising awareness for water polution based on game activities using internet of things. In European Conference on Ambient Intelligence, pages 171{187. Springer, 2018.