=Paper=
{{Paper
|id=Vol-2699/paper21
|storemode=property
|title=Assessing the Impact of an Online Inquiry Teaching Intervention on Sixth Graders’ Search Performance
|pdfUrl=https://ceur-ws.org/Vol-2699/paper21.pdf
|volume=Vol-2699
|authors=Roberto González-Ibáñez,Daniel Gacitúa,Gonzalo Martínez-Ramírez,Jacqueline Köhler,Eero Sormunen,Carita Kiili,Mirjamaija Mikkilä-Erdmann,Norbert Erdmann,Marja Vauras,Paavo H.T. Leppänen
|dblpUrl=https://dblp.org/rec/conf/cikm/Gonzalez-Ibanez20
}}
==Assessing the Impact of an Online Inquiry Teaching Intervention on Sixth Graders’ Search Performance==
<pdf width="1500px">https://ceur-ws.org/Vol-2699/paper21.pdf</pdf>
<pre>
Assessing the impact of an online inquiry teaching
intervention on sixth graders’ search performance
Roberto González-Ibáñeza , Daniel Gacitúaa , Gonzalo Martínez-Ramíreza ,
Jacqueline Köhlera , Eero Sormunenb , Carita Kiilic , Mirjamaija Mikkilä-Erdmannd ,
Norbert Erdmannd , Marja Vaurasd and Paavo H.T. Leppänene
a Departamento de Ingeniería Informática, Universidad de Santiago de Chile
b Faculty of Information Technology and Communication Sciences, Tampere University
c Faculty of Education and Culture, Tampere University
d Department of Teacher Education, University of Turku
e Department of Psychology, University of Jyväskylä



                                          Abstract
                                          The study of online inquiry competences (OIC) is an important topic on the information literacy (IL) field. Most of the work
                                          has been focused on higher education and high-school students, while less has been done regarding primary education. In
                                          this work we investigate the effects of an OIC teaching intervention on search performance of a large group of sixth graders
                                          from Finland. Our preliminary results show significant improvements in search performance on the intervened group of
                                          students compared to a control group when working on a science research task. This work shed light about the potential
                                          benefits of a particular approach to develop OIC on elementary school students.

                                          Keywords
                                          online inquiry competences, information literacy, assessment, tests, elementary schools


1. Introduction and related work                                                                                   veloping OIC in all levels of education. However, to
                                                                                                                   the best of our knowledge, most of the effort on devel-
The rise of the World Wide Web undoubtedly changed                                                                 oping and assessing OIC has been focused on tertiary
the way people look for information and provided an                                                                education students [7] and, in a smaller proportion, on
ever-growing source of information, hence boosting                                                                 high school students [8, 9] and primary education.
research in the field of Information Literacy (IL) [1].                                                               Given the above context, the iFuCo project [10] set
In this context, special attention has been given to in-                                                           an ambitious goal to design and carry out an interven-
quiry skills, which can be defined as the skills to gather,                                                        tion to develop OIC in a group of Finnish and Chilean
interpret, and synthesize different kinds of informa-                                                              elementary school students. While the scope of this
tion and data in order to develop and share answers                                                                project targeted students in both countries, interven-
to questions [2]. From this definition, online inquiry                                                             tions were not identical due to cultural differences. More-
competences (OIC) involve the above skills in addition                                                             over, while the project focused on developing four com-
to the knowledge and the abilities to inquire the Web                                                              ponent skills (i.e., search and locate information, iden-
[3]. OIC are highly valued in the 21st century [4].                                                                tification of main ideas, critical evaluation, and syn-
   [5] established that library and information skills                                                             thesis) [11], in this article we only study the effects of
(which are within the scope of OIC) involve a series of                                                            our intervention on search skills. In particular, we ad-
cognitive activities: task definition, information seek-                                                           dress the following research question (RQ): To what
ing strategies, location, access, use, synthesis, and eval-                                                        extent, if any, can a teaching intervention targeted to
uation of information. [6] emphasized the need of de-                                                              develop OIC on Finnish sixth graders improve their
                                                                                                                   search performance in the context of research tasks in
Proceedings of the CIKM 2020 Workshops, October 19-20, 2020,                                                       multiple domains?
Galway, Ireland                                                                                                       In the next section we introduce the methodological
email: roberto.gonzalez.i@usach.cl (R. González-Ibáñez);                                                           approach as defined in the iFuCo project. Following,
daniel.gacitua@usach.cl (D. Gacitúa); gonzalo.martinez@usach.cl
(G. Martínez-Ramírez); jacqueline.kohler@usach.cl (J. Köhler);
                                                                                                                   we present preliminary results. Finally, we conclude
eero.sormunen@tuni.fi (E. Sormunen); carita.kiili@tuni.fi (C. Kiili);                                              with a brief discussion of our findings.
mirmik@utu.fi (M. Mikkilä-Erdmann); nwmerd@utu.fi (N.
Erdmann); vauras@utu.fi (M. Vauras); paavo.ht.leppanen@jyu.fi
(P.H.T. Leppänen)
                                    © 2020 Copyright for this paper by its authors. Use permitted under Creative
                                    Commons License Attribution 4.0 International (CC BY 4.0).
 CEUR
 Workshop
 Proceedings
               http://ceur-ws.org
               ISSN 1613-0073       CEUR Workshop Proceedings (CEUR-WS.org)
                                                             Table 1
                                                             Task and domain rotations.
                                                                            Pretest                            Posttest
                                                                Session 1          Session 2         Session 1            Session 2
                                                             Soc. Sci. Article   Science Email     Science Article     Soc. Sci. Email
                                                             Science Article     Soc. Sci. Email   Soc. Sci. Article   Science Email
Figure 1: Pretest-Posttest study design.


                                                             2.3. Task
2. Method
                                                         We considered two knowledge domains, “science” and
2.1. Study Design                                        “social science”, and two tasks: “writing an article” and
                                                         “writing an email response”. The combination of these
Our approach to address the above RQ involved a quasi- domains and tasks formed four activities for the stu-
experiment following a pretest-posttest design as shown dents, each one with a multifaceted topic, described as
in Figure 1. To assess students’ OIC in both pretest and follows: (1) Science article: “Finnish forests”, (2) Sci-
posttest, we devised a performance-based test focused ence email: “Origins of rain”, (3) Social science article:
on four component skills, namely, (1) searching and “Computer games” and (4) Social science email: “Read-
selecting relevant sources, (2) identifying main ideas ing on digital screens”. One science and social science
from sources, (3) evaluating the credibility of sources, task were addressed as part of the pretest and posttest
and (4) synthesizing information across multiple sources according to the rotations illustrated in Table 1.
[11], which was carried out using NEURONE [12].             For each activity, students were allowed to search in
   As for the intervention, this consisted of a training NEURONE within a collection of 20 documents. Three
program involving three modules (i.e., Module 1: Ex- of them, marked as relevant sources, were designed by
plicit teaching of OIC, which included aspects such as researchers. The remaining 17 pages were authentic
query formulation and analysis of search results; Mod- websites.
ule 2: Applied skills in a science research task; Mod-
ule 3: Applied skills in a social science research task)
in a span of three to four weeks. The intervention in- 2.4. Session workflow
volved conceptual classes, tutorials, and practice with- The session workflow in both pretest and posttest in-
out NEURONE. More details of the study design and volved four stages linked to the above mentioned com-
intervention can be found in [13].                       ponent skills. First, the search and selection phase was
                                                             conducted in a maximum of 8 minutes. This phase was
2.2. Sample                                                  completed either when students found all three rele-
                                                             vant sources or when time was up. Second, students
We recruited 364 sixth graders from 10 Finnish schools       were given 12 minutes to identify main ideas in the
(15 classes) distributed over three cities (i.e., Tampere,   relevant sources. Third, the critical evaluation of the
Turku, and Jyväskylä). From this group, 344 students         sources was done within 7 minutes. Finally, the syn-
were authorized by their parents and 2 of them were          thesis phase was completed in a maximum of 15 min-
absent during the tests. Therefore, our initial sample       utes. Overall, sessions lasted approximately 50 min-
consisted of 342 students whose mean age was 12.3            utes.
(SD=.41) years old. Regarding sex, 165 (48.25%) were
girls and 177 (51.75%) were boys.
   Classes in which the study was carried out were           2.5. Study setup
randomly assigned to control or experimental groups.         Sessions were conducted in schools, using schools’ com-
Note that randomization was school-class-based. As a         puters and network connectivity to access the NEU-
result, eight classes were assigned to the experimental      RONE server. Sessions were supervised by members of
group (192 students - 46.85% girls, 53.15% boys) and         the research team. Once the study data collection was
the remaining seven classes to the control group (150        completed, we gathered NEURONE database dumps to
students - 50% girls, 50% boys).                             perform the analyses.
3. Results                                                         Table 2
                                                                   Descriptive statistics for recall and search score per group.
As noted above, in this article we focused on data col-
lected during the search and selection stage. In partic-                   Recall                           SearchScore
ular, we performed both within- and between- subjects           Mean Median SD                     Mean Median SD
comparisons based on our study design. We grouped C1 0.7262 0.667                       0.2935 2.8956 3                      1.4379
all data from the control and experimental groups re- E1 0.7164 0.667                   0.3028 2.782            2.5          1.4326
gardless the tasks and domains.                           C2 0.8602 1                   0.2304 3.154            3.1665       1.3865
   After pre-processing the data we were able to con- E2 0.8624 1                       0.2354 3.4029 3.333                  1.3483
solidate a database of 273 students. Records from the
remaining 69 students were discarded due to missing Table 3
data, incomplete sessions, or corrupted data due to tech- Search performance results (wr = Wilcoxon Rank Sum, ws =
nical issues during sessions (e.g., connection problems, Wilcoxon Signed Rank, * = Significant result at p<0.05).
operating system or browser incompatibilities with NEU-                      Recall                           SearchScore
RONE, which mainly affected the search phase). From             Statistic p-value     Effect Size Statistic    p-value    Effect Size
this group, 448 sessions belong to the control group C1≈E1 wr=36544 0.7771            0.0121      wr=37411     0.4528     0.0321
                                                          C1<C2 ws=1435   5.2141e-8*  0.3701      ws=6142.5    0.012*     0.1264
and 644 to the experimental group.                        E1<E2 ws=3357   7.5465e-12* 0.3902      ws=10450.5 3.1626e-10* 0.3407
                                                                   C2<E2   wr=35606   0.3782   0.0133   wr=32319.5   0.0179*   0.0899
                         #𝑅𝑒𝑙𝑒𝑣𝑎𝑛𝑡𝐷𝑜𝑐𝑠𝐵𝑜𝑜𝑘𝑚𝑎𝑟𝑘𝑒𝑑(𝑠)
          𝑅𝑒𝑐𝑎𝑙𝑙(𝑠) =                3                       (1)
                                                              To ensure the homogeneity between the distribu-
                                                           tion of students in the control and experimental groups
   𝑆𝑒𝑎𝑟𝑐ℎ𝑆𝑐𝑜𝑟𝑒(𝑠) = 5 ∗ #𝑅𝑒𝑙𝑒𝑣𝑎𝑛𝑡𝐷𝑜𝑐𝑠𝐵𝑜𝑜𝑘𝑚𝑎𝑟𝑘𝑒𝑑(𝑠)
                              #𝐴𝑐𝑡𝑖𝑣𝑒𝐵𝑜𝑜𝑘𝑚𝑎𝑟𝑘𝑠(𝑠)     (2)  before  the teaching intervention, we first compared
                                                           C1 and E1 during the pretest in terms of the perfor-
   Data include a wide range of variables linked to search mance measures listed on Equations 1 and 2. Our re-
behaviors (e.g., dwell time in pages, mouse movements, sults showed no significant differences between the
keystrokes). From this list of variables, in this arti- two groups.
cle we only analyzed the effects of the intervention on       Next, we conducted within-subject comparisons to
search performance, which was expressed in terms of determine the effects of the intervention in the exper-
recall and what we refer to search score [11]. On the imental group. Results showed a significant increase
one hand, given the goal of the search phase in which in recall and search score in both groups (p<0.05) (Ta-
the student had to bookmark all relevant pages (3 per ble 3). Yet, between-subject comparison showed that
task), the recall measure of a student s was appropri- search score achieved by the experimental group (E2)
ate to express the end product of the search process. was significantly greater than that of the control group
This measure was computed according to Equation 1. (C2) (p<0.05).
On the other hand, the search score of a student s was
defined as a way to measure not only the end product
but also some aspects of the search process. In par- 4. Discussion
ticular, the search score takes into account mistakes
as a result of strategies such as trial an error, which Our preliminary findings indicate at large that regard-
was observed in some students who took advantage of less of the intervention, both control and experimental
system’s feedback after bookmarking and submitting groups exhibited performance improvements in terms
random three pages (active bookmarks) to find out if of recall and search score (i.e., pretest-posttest within-
they completed the task. The score is expressed in a 0 subjects comparison). While the gain in the control
– 5 scale and it was operationalized as shown in Equa- group was not expected, this could be attributed to a
tion 2. A summary of descriptive statistics for these learning effect as a result of the exposure to the NEU-
two measures is listed in Table 2.                         RONE system during the pretest. Indeed, although the
   To perform within- and between-subjects compar- topics and domains of the tasks faced by students in
isons, we relied on the t-test and Wilcoxon test (pair- the pretest and posttest were different, session struc-
wise and independent samples respectively) depend- ture and the system were the same, thus students were
ing upon data distribution and variance. A confidence likely to grasp system features, session flow, and strate-
interval of 95% was used for all tests. Normality was gies to better address the tasks.
tested through the Shapiro-Wilk test, whereas homosce- In spite of the performance improvement observed
dasticity was tested with the Levene test.                 in both groups, we found that search score was signif-
icantly better in the experimental group (E2) than in            line inquiry through software-based scaffolding,
the control group (C2) after the intervention (posttest          Educ. Psychol. 40 (2005) 235–244. doi:10.1207/
between-subject comparison). We believe that such                s15326985ep4004_5.
difference could be attributed in part to the teaching [4] American Library Association, Information lit-
intervention to which students in the experimental group         eracy competency standards for higher edu-
were exposed to.                                                 cation, 2000. URL: https://alair.ala.org/handle/
   We note that no significant differences were found            11213/7668.
with respect to recall during the posttest. This can be [5] M. B. Eisenberg, R. E. Berkowitz, Information
explained by the product-based nature of the measure             Problem Solving: The Big Six Skills Approach to
itself. Recall indicates success (i.e., finding relevant         Library & Information Skills Instruction, Big6 In-
documents), no matter how students achieved the goal             formation Literacy Skills Series, Ablex Publish-
(e.g., trial and error). On the contrary, the fact that          ing, Norwood, NJ, USA, 1990.
search scores were higher in the experimental group [6] J. Delors, Learning: The treasure within, UN-
(E2) illustrates that students in this group were more           ESCO, Paris, France, 1998.
cautious when bookmarking pages avoiding making [7] B. Lym, H. Grossman, L. Yannotta, M. Talih,
mistakes. This positive effect could be the result of the        Assessing the assessment: how institutions ad-
intervention, yet further analyses are needed to test            ministered, interpreted, and used sails, Ref.
this hypothesis.                                                 Serv. Rev. 38 (2010) 168–186. doi:10.1108/
   Our future work will target analyses to find out whether      00907321011020806.
the intervention had effects on performance at the level [8] S. Majid, Y.-K. Chang, S. Foo, Auditing informa-
of task and domains. Beyond the scope of search per-             tion literacy skills of secondary school students
formance, which was the main focus of this article, we           in singapore, J. of Information Literacy 10 (2016)
will also look at the effects of the teaching intervention       44–66. doi:10.11645/10.1.2068.
on search behaviors (e.g., query formulation, mouse [9] S. Foo, S. Majid, Y. K. Chang, Assessing in-
actions) as well as other component skills evaluated in          formation literacy skills among young infor-
the performance test (i.e., identification of main ideas         mation age students in singapore,         Aslib J.
and synthesis).                                                  Inf. Manag. 69 (2017) 335–353. doi:10.1108/
                                                                 AJIM-08-2016-0138.
                                                            [10] M. Mikkila-Erdmann, E. Sormunen, T. Mikkonen,
Acknowledgments                                                  N. Erdmann, C. Kiili, M. Quintanilla, R. González-
                                                                 Ibáñez, P. H. Leppänen, M. Vauras, A compar-
The work described in this article was partially sup-
                                                                 ative study on learning and teaching online in-
ported by the TUTELAGE project funded by the Na-
                                                                 quiry skills in Finland and Chile, in: European
tional Agency for Research and Development (ANID)
                                                                 Association for Research on Learning and In-
(FONDECYT Regular, grant no. 1201610); the iFuCo
                                                                 struction, Tampere, Finland, 2017.
project funded by the Academy of Finland (grant no.
                                                            [11] E. Sormunen, R. González-Ibáñez, C. Kiili, P. H.
294186) and ANID (grant AKA/EDU-03); ANID Schol-
                                                                 Leppänen, M. Mikkilä-Erdmann, N. Erdmann,
arship Program Doctorado Nacional no. 2019-21192162
                                                                 M. Escobar-Macaya, A performance-based test
and Doctorado Becas Chile no. 7608-2020.
                                                                 for assessing students’ online inquiry compe-
                                                                 tences in schools, in: European Conference on
References                                                       Information Literacy, Springer, 2017, pp. 673–
                                                                 682. doi:10.1007/978-3-319-74334-9_69.
  [1] P. Stordy,        Taxonomy of literacies,          J. [12] R. González-Ibáñez, D. Gacitúa, E. Sormunen,
       Doc. 71 (2015) 456–476. doi:10.1108/                      C. Kiili, NEURONE: oNlinE inqUiRy experimen-
       JD-10-2013-0128.                                          tatiON systEm, Proc. Assoc. Inf. Sci. Technol.
  [2] National Research Council, Inquiry and the na-             54 (2017) 687–689. doi:10.1002/pra2.2017.
       tional science education standards: A guide for           14505401117.
       teaching and learning, National Academies Press, [13] E. K. Hämäläinen, C. Kiili, M. Marttunen,
       Washington, DC, USA, 2000. doi:10.17226/                  E. Räikkönen, R. González-Ibáñez, P. H. Leppä-
       9596.                                                     nen, Promoting sixth graders’ credibility evalua-
  [3] C. Quintana, M. Zhang, J. Krajcik, A frame-                tion of web pages: An intervention study, Com-
       work for supporting metacognitive aspects of on-          put. Hum. Behav. (2020) 106372. doi:10.1016/
                                                                 j.chb.2020.106372.

</pre>