On Measuring Learning in Search: A Position Paper

Abstract

This position paper discusses approaches used to evaluate learning that results from searching and interacting with online content. comprehension; evaluation; interactive search and retrieval; learning; measurement; semantic navigation

1. INTRODUCTION

Research on search systems is shifting from an emphasis on information seeking and retrieval to one of information interaction and use. This is an outgrowth of changes in the information landscape where full-text and multimedia information objects in digital format are now readily available in systems that facilitate browsing and direct interaction with these objects. While traditional assessment measures for information seeking and retrieval have focused on e ectiveness and e ciency in retrieving information objects, these are no longer su cient in more immersive and interactive environments.

Our research group has characterized a form of information interaction that takes place in online search environments as semantic navigation [ 6 ], focusing on the multi-level meaning-making and learning that takes place while moving through hyperlinked digital environments. More recently, we have focused explicitly on the inter-connected processes of reading, comprehension, engagement, and learning in the course of digital information interaction [ 2, 3 ]. In this position paper, we discuss some of the approaches that have been used to evaluate learning as a key outcome of information interaction and search.

APPROACHES TO EVALUATION

Past research that evaluates learning in the context of searching is relatively rare [ 4, 11 ], but more recently, increased interest has been shown through a series of \SearchSearch as Learning (SAL), July 21, 2016, Pisa, Italy The copyright for this paper remains with its authors. Copying permitted for private and academic purposes. ing as Learning" workshops1 and associated publications [ 8, 9 ] and publications [ 8, 9 ]. However, there is a wide body of research in related research areas, including text comprehension and hypertext. Taken together, this prior work o ers a range of approaches. 2.1

Models and Theories

Several di erent models of comprehension and learning are commonly referenced in work on searching as learning, with implications for measurement.

The Construction Integration (C-I) model [ 5 ] has been the basis for our own work in this area. It focuses on the cognitive process of comprehension during interaction with content. This is represented as a two-step process. First, the reader creates nodes for all propositions in the text. These nodes form the textbase, within which there is a micro structure that deals with comprehension at the sentence and paragraph level, and a macro structure consisting of the global, overall meaning, or gist of the text [ 5 ]. This distinction is important when evaluating comprehension, as di erent tests of comprehension are sensitive to outcomes at both the micro- and macro-levels. Our research has focused on measurement at the macro-structural level as we are most interested in the reader's understanding of the overall meaning of the text. We have found variation in the ability of standard comprehension tests to measure at both the macro- and micro-levels.

Kuhlthau's Information Search Process model [ 7 ] has been highly in uential in information science. It is informed by a constructivist approach to learning and is insightful in that it portrays learning as a process characterized by distinct phases during the course of interacting with information with associated changes in goals, activities and emotional states. Vakkari's empirical work extended the model in the search domain by demonstrating that searchers' queries and relevance assessments re ect changes in their knowledge state as they search [ 10, 9 ].

Bloom's Taxonomy of Educational Objectives [ 1 ] has served as a framework for a number of recent search studies [ 4, 11 ]. The Taxonomy identi es a set of progressively complex learning objectives that can be used to design or assess learning experiences. It o ers a means of assessing the depth of learning that occurs through search, although it can be challenging to di erentiate between categories and map evidence of learning to them. 1The rst \Searching as Learning" workshop was held at the IIiX 2014 Conference (http://www.diigubc.ca/IIIXSAL).

NEXT STEPS

Drawing upon the range of models and methods outlined here, there is potential to develop and build consensus around a standardized approach to the assessment of learning in search, much as the interactive information retrieval community developed a standard approach to the design of experimental search studies a decade ago. We look forward to engaging with SAL workshop participants to move us closer to this goal. 4. 5.

ACKNOWLEDGEMENTS

Research funding from the University of British Columbia Hampton Fund is gratefully acknowledged, as are the contributions of our colleague Heather O'Brien.

Methods

Methods of assessing searching and learning are interdisciplinary and wide-ranging, and a lengthy review would be required to provide an overview of them (e.g. [ 8, 9 ]). In this position paper, we simply articulate the broader dimensions of methods that have emerged in our own work. 2.2.1

Pre- and Post-task vs. Process

There are two common approaches to assessing learning outcomes of search. The rst approach tends to rely on a post-task test or written summary, and may include a pretask assessment of prior knowledge. We have relied primarily upon this approach in our work to date, comparing learning outcomes resulting from di erent interaction environments. However, results can be di cult to interpret in the absence of interaction data. Process-oriented approaches, on the other hand, capture patterns of behavior and thus can reveal the mechanisms by which learning occurs, such as spending more time in certain sections of documents, or switching more frequently between documents [ 3 ]. 2.2.2

Duration

Typical online search interactions may only take a few seconds or minutes and are not likely to involve signi cant learning on the part of the searcher. In fact, one of the arguments for considering learning as an important search outcome, is to acknowledge the value of \slow search" and search tasks that carry over through multiple search sessions in contrast to the e ciency-based models that predominate in IR research. Therefore, methods for studying learning in search will require search tasks that prompt lengthier searches with high degrees of interaction, multiple sessions, or longitudinal studies. This will allow for learning to be assessed in real-time, as the search process unfolds, and as an immediate and/or sustained outcome of searching. 2.2.3

Customized vs. Generic

A major challenge in assessing learning is the dependency between speci c content, the prior knowledge of the searcher and the learning that occurs. Most of the approaches to assessing learning rely upon tests based on a small number of known content items, such as sets of articles or webpages. The custom development and validation of these instruments is labour intensive and the method does not scale up for use in search studies using large document collections. Alternate, more generic, methods require participants to produce open-ended summaries or reports and assess those reports for evidence of learning. 2.3

Measures

The simplest and most common measures of learning are self-reported knowledge gain and tests of factual knowledge using multiple choice or true and false responses. However, such measures do not account for the complexity of learning as a multi-stage and multi-level process. We have found differences between measures targeting micro and macro levels of comprehension from the C-I Model. Drawing upon insights from Kuhlthau's model and Bloom's Taxonomy, we expect that it will be possible to develop even more sophisticated measures of learning.

References

[1] B. S. Bloom , M. D. Engelhart , E. J. Furst , W. H. Hill , and D. R. Krathwohl . Taxonomy of educational objectives: The classi cation of educational goals . Handbook 1 : Cognitive domain. David McKay , New York, NY, 1956 .

[2] S. Dodson . E ects of eld dependence-independence and passive highlights on comprehension . Master's thesis , University of British Columbia, Vancouver, Canada, 2016 .

[3] L. Freund , R. Kopak , and H. O'Brien. The e ects of textual environment on reading comprehension . Journal of Information Science , 42 ( 1 ): 79 { 93 , 2016 .

[4] B. J. Jansen , D. Booth , and B. Smith. Using the taxonomy of cognitive learning to model online searching . Information Processing & Management , 45 ( 6 ): 643 { 663 , 2009 .

[5] W. Kintsch . Comprehension: A paradigm for cognition . Cambridge University Press, Cambridge, England, 1998 .

[6] R. Kopak , L. Freund , and H. L. O'Brien. Supporting semantic navigation . In Proceedings of the Third Symposium on Information Interaction in Context , pages 359 { 364 , 2010 .

[7] C. Kuhlthau . Seeking meaning: A process approach to library and information services . Libraries Unlimited , Westport, CT , 2004 .

[8] S. Y. Rieh , K. Collins-Thompson , P. Hansen , and H.-J. Lee . Towards searching as a learning process: A review of current perspectives and future directions . Journal of Information Science , 42 ( 1 ): 19 { 34 , 2016 .

[9] P. Vakkari . Searching as learning: A systematization based on literature . Journal of Information Science , 42 ( 1 ):7{ 18 , 2016 .

[10] P. Vakkari , M. Pennanen , and S. Serola . Changes of search terms and tactics while writing a research proposal: A longitudinal case study . Information Processing & Management , 39 ( 3 ): 445 { 463 , 2003 .

[11] M. J. Wilson and M. L. Wilson . A comparison of techniques for measuring sensemaking and learning within participant-generated summaries . Journal of the American Society for Information Science and Technology , 64 ( 2 ): 291 { 306 , 2013 .