The study of online inquiry competences (OIC) is an important topic on the information literacy (IL) field. Most of the work has been focused on higher education and high-school students, while less has been done regarding primary education. In this work we investigate the efects of an OIC teaching intervention on search performance of a large group of sixth graders from Finland. Our preliminary results show significant improvements in search performance on the intervened group of students compared to a control group when working on a science research task. This work shed light about the potential benefits of a particular approach to develop OIC on elementary school students.
Pretest Posttest
Session 1 Soc. Sci. Article Science Article Session 2 Science Email Soc. Sci. Email Session 1 Science Article Soc. Sci. Article Session 2 Soc. Sci. Email Science Email 2.3. Task
2.1. Study Design “social science”, and two tasks: “writing an article” and “writing an email response”. The combination of these Our approach to address the above RQ involved a quasi- domains and tasks formed four activities for the stuexperiment following a pretest-posttest design as shown dents, each one with a multifaceted topic, described as in Figure 1. To assess students’ OIC in both pretest and follows: (1) Science article: “Finnish forests”, (2) Sciposttest, we devised a performance-based test focused ence email: “Origins of rain”, (3) Social science article: on four component skills, namely, (1) searching and “Computer games” and (4) Social science email: “Readselecting relevant sources, (2) identifying main ideas ing on digital screens”. One science and social science from sources, (3) evaluating the credibility of sources, task were addressed as part of the pretest and posttest and (4) synthesizing information across multiple sources according to the rotations illustrated in Table 1. [11], which was carried out using NEURONE [12]. For each activity, students were allowed to search in
As for the intervention, this consisted of a training NEURONE within a collection of 20 documents. Three program involving three modules (i.e., Module 1: Ex- of them, marked as relevant sources, were designed by plicit teaching of OIC, which included aspects such as researchers. The remaining 17 pages were authentic query formulation and analysis of search results; Mod- websites. ule 2: Applied skills in a science research task; Module 3: Applied skills in a social science research task) in a span of three to four weeks. The intervention in- 2.4. Session workflow volved conceptual classes, tutorials, and practice without NEURONE. More details of the study design and intervention can be found in [13].
(15 classes) distributed over three cities (i.e., Tampere, Turku, and Jyväskylä). From this group, 344 students were authorized by their parents and 2 of them were absent during the tests. Therefore, our initial sample consisted of 342 students whose mean age was 12.3 (SD=.41) years old. Regarding sex, 165 (48.25%) were girls and 177 (51.75%) were boys.
Classes in which the study was carried out were randomly assigned to control or experimental groups. Note that randomization was school-class-based. As a result, eight classes were assigned to the experimental group (192 students - 46.85% girls, 53.15% boys) and the remaining seven classes to the control group (150 students - 50% girls, 50% boys).
The session workflow in both pretest and posttest involved four stages linked to the above mentioned component skills. First, the search and selection phase was conducted in a maximum of 8 minutes. This phase was completed either when students found all three relevant sources or when time was up. Second, students were given 12 minutes to identify main ideas in the relevant sources. Third, the critical evaluation of the sources was done within 7 minutes. Finally, the synthesis phase was completed in a maximum of 15 minutes. Overall, sessions lasted approximately 50 minutes.
puters and network connectivity to access the NEURONE server. Sessions were supervised by members of the research team. Once the study data collection was completed, we gathered NEURONE database dumps to perform the analyses.
lected during the search and selection stage. In partic- Recall SearchScore ular, we performed both within- and between- subjects Mean Median SD Mean Median SD comparisons based on our study design. We grouped C1 0.7262 0.667 0.2935 2.8956 3 1.4379 all data from the control and experimental groups re- E1 0.7164 0.667 0.3028 2.782 2.5 1.4326 gardless the tasks and domains. C2 0.8602 1 0.2304 3.154 3.1665 1.3865
After pre-processing the data we were able to con- E2 0.8624 1 0.2354 3.4029 3.333 1.3483 solidate a database of 273 students. Records from the remaining 69 students were discarded due to missing Table 3 data, incomplete sessions, or corrupted data due to tech- Search performance results (wr = Wilcoxon Rank Sum, ws = nical issues during sessions (e.g., connection problems, Wilcoxon Signed Rank, * = Significant result at p<0.05). operating system or browser incompatibilities with NEU- Recall SearchScore RONE, which mainly afected the search phase). From Statistic p-value Efect Size Statistic p-value Efect Size this group, 448 sessions belong to the control group C1≈E1 wr=36544 0.7771 0.0121 wr=37411 0.4528 0.0321 and 644 to the experimental group. EC11<<EC22 wwss==13433557 57..25144615ee--81*2* 00..33790012 wwss==61104425.05.5 03..011622*6e-10* 00..13246047 C2<E2 wr=35606 0.3782 0.0133 wr=32319.5 0.0179* 0.0899 ( ) = # 3 ( ) (1)
tion of students in the control and experimental groups ℎ ( ) = 5 ∗ # # ( ) ( ) (2) before the teaching intervention, we first compared
C1 and E1 during the pretest in terms of the perfor
Data include a wide range of variables linked to search mance measures listed on Equations 1 and 2. Our rebehaviors (e.g., dwell time in pages, mouse movements, sults showed no significant diferences between the keystrokes). From this list of variables, in this arti- two groups. cle we only analyzed the efects of the intervention on Next, we conducted within-subject comparisons to search performance, which was expressed in terms of determine the efects of the intervention in the experrecall and what we refer to search score [11]. On the imental group. Results showed a significant increase one hand, given the goal of the search phase in which in recall and search score in both groups (p<0.05) (Tathe student had to bookmark all relevant pages (3 per ble 3). Yet, between-subject comparison showed that task), the recall measure of a student s was appropri- search score achieved by the experimental group (E2) ate to express the end product of the search process. was significantly greater than that of the control group This measure was computed according to Equation 1. (C2) (p<0.05).
On the other hand, the search score of a student s was defined as a way to measure not only the end product but also some aspects of the search process. In par- 4. Discussion ticular, the search score takes into account mistakes as a result of strategies such as trial an error, which Our preliminary findings indicate at large that regardwas observed in some students who took advantage of less of the intervention, both control and experimental system’s feedback after bookmarking and submitting groups exhibited performance improvements in terms random three pages (active bookmarks) to find out if of recall and search score (i.e., pretest-posttest withinthey completed the task. The score is expressed in a 0 subjects comparison). While the gain in the control – 5 scale and it was operationalized as shown in Equa- group was not expected, this could be attributed to a tion 2. A summary of descriptive statistics for these learning efect as a result of the exposure to the NEUtwo measures is listed in Table 2. RONE system during the pretest. Indeed, although the
To perform within- and between-subjects compar- topics and domains of the tasks faced by students in isons, we relied on the t-test and Wilcoxon test (pair- the pretest and posttest were diferent, session strucwise and independent samples respectively) depend- ture and the system were the same, thus students were ing upon data distribution and variance. A confidence likely to grasp system features, session flow, and strateinterval of 95% was used for all tests. Normality was gies to better address the tasks. tested through the Shapiro-Wilk test, whereas homosce- In spite of the performance improvement observed dasticity was tested with the Levene test. in both groups, we found that search score was signif
The work described in this article was partially supported by the TUTELAGE project funded by the National Agency for Research and Development (ANID) (FONDECYT Regular, grant no. 1201610); the iFuCo project funded by the Academy of Finland (grant no. 294186) and ANID (grant AKA/EDU-03); ANID Scholarship Program Doctorado Nacional no. 2019-21192162 and Doctorado Becas Chile no. 7608-2020. icantly better in the experimental group (E2) than in the control group (C2) after the intervention (posttest between-subject comparison). We believe that such diference could be attributed in part to the teaching intervention to which students in the experimental group were exposed to.
We note that no significant diferences were found with respect to recall during the posttest. This can be explained by the product-based nature of the measure itself. Recall indicates success (i.e., finding relevant documents), no matter how students achieved the goal (e.g., trial and error). On the contrary, the fact that search scores were higher in the experimental group (E2) illustrates that students in this group were more cautious when bookmarking pages avoiding making mistakes. This positive efect could be the result of the intervention, yet further analyses are needed to test this hypothesis.
Our future work will target analyses to find out whether the intervention had efects on performance at the level of task and domains. Beyond the scope of search performance, which was the main focus of this article, we will also look at the efects of the teaching intervention on search behaviors (e.g., query formulation, mouse actions) as well as other component skills evaluated in the performance test (i.e., identification of main ideas and synthesis). line inquiry through software-based scafolding, Educ. Psychol. 40 (2005) 235–244. doi:10.1207/ s15326985ep4004_5. [4] American Library Association, Information literacy competency standards for higher education, 2000. URL: https://alair.ala.org/handle/ 11213/7668. [5] M. B. Eisenberg, R. E. Berkowitz, Information
Problem Solving: The Big Six Skills Approach to Library & Information Skills Instruction, Big6 Information Literacy Skills Series, Ablex Publishing, Norwood, NJ, USA, 1990. [6] J. Delors, Learning: The treasure within, UN
ESCO, Paris, France, 1998. [7] B. Lym, H. Grossman, L. Yannotta, M. Talih,
Assessing the assessment: how institutions administered, interpreted, and used sails, Ref.
Serv. Rev. 38 (2010) 168–186. doi:10.1108/ 00907321011020806. [8] S. Majid, Y.-K. Chang, S. Foo, Auditing information literacy skills of secondary school students in singapore, J. of Information Literacy 10 (2016) 44–66. doi:10.11645/10.1.2068. [9] S. Foo, S. Majid, Y. K. Chang, Assessing information literacy skills among young information age students in singapore, Aslib J.
Inf. Manag. 69 (2017) 335–353. doi:10.1108/
AJIM-08-2016-0138. [10] M. Mikkila-Erdmann, E. Sormunen, T. Mikkonen,
N. Erdmann, C. Kiili, M. Quintanilla, R. GonzálezIbáñez, P. H. Leppänen, M. Vauras, A comparative study on learning and teaching online inquiry skills in Finland and Chile, in: European Association for Research on Learning and Instruction, Tampere, Finland, 2017. [11] E. Sormunen, R. González-Ibáñez, C. Kiili, P. H.
Leppänen, M. Mikkilä-Erdmann, N. Erdmann, M. Escobar-Macaya, A performance-based test for assessing students’ online inquiry competences in schools, in: European Conference on Information Literacy, Springer, 2017, pp. 673– 682. doi:10.1007/978-3-319-74334-9_69. [1] P. Stordy, Taxonomy of literacies, J. [12] R. González-Ibáñez, D. Gacitúa, E. Sormunen, Doc. 71 (2015) 456–476. doi:10.1108/ C. Kiili, NEURONE: oNlinE inqUiRy experimenJD-10-2013-0128. tatiON systEm, Proc. Assoc. Inf. Sci. Technol. [2] National Research Council, Inquiry and the na- 54 (2017) 687–689. doi:10.1002/pra2.2017. tional science education standards: A guide for 14505401117. teaching and learning, National Academies Press, [13] E. K. Hämäläinen, C. Kiili, M. Marttunen, Washington, DC, USA, 2000. doi:10.17226/ E. Räikkönen, R. González-Ibáñez, P. H. Leppä9596. nen, Promoting sixth graders’ credibility evalua[3] C. Quintana, M. Zhang, J. Krajcik, A frame- tion of web pages: An intervention study, Comwork for supporting metacognitive aspects of on- put. Hum. Behav. (2020) 106372. doi:10.1016/ j.chb.2020.106372.