From its very origins information science has been concerned with ways and means of storing knowledge, organizing it, as well as helping people to find, use, and learn from it. Theorists of information science conceptually linked information interaction processes with the states of human knowledge (e.g., Belkin's ASK [2] and Dervin's sense-making [3] and information seeking was described as "a process, in which humans purposefully engage in order to change their state of knowledge." [4]. Yet in spite of the long established relationship with learning, only a few empirical studies exist that focus on search as learning (e.g [5][6][7]). However, with the recent special journal issue [8], and with this and two earlier workshops, we observe an increased interest in this topic. One research challenge identified at a previous workshop [9] was how to assess learning in the context of purposeful information seeking. This is where we aim to contribute through this project. This short paper presents our approach, method, and initial data analysis.
Our working definition of learning is any change in person's knowledge structures. We consider that learning can take place at many levels [10] and we are particularly influenced by the cognitive, skill-based, and affective theory of learning outcomes (CSALO) model [1]. This framework contains elements related to searching to learn (e.g., declarative knowledge) as well as learning to search (strategies, tactics, procedural knowledge). According to this model learning outcomes are partially reflected in changes in verbal knowledge, knowledge organization, and cognitive strategies. We are particularly interested in assessing changes in verbal knowledge.
Our prior work [11][12][13] has demonstrated feasibility of using eyetracking to detect relationship between eye movement and knowledge levels. The method takes advantage of a direct relationship between eye movement patterns and cognitive processes. One goal of the project presented in this short paper is to connect eye-tracking measures and traditional IR measures (e.g., number and kind of query reformulations) with measures of learning. Since we are reporting on work in progress, the initial data analysis will serve as only a simple illustration of our approach, while our choices of measures will inspire a discussion.
A lab-based experiment was conducted in the Information eXperience lab at University of Texas at Austin (N=30). Data is reported here for 26 of these subjects (16 females; mean age of all participants 24.5). Participants who volunteered after seeing the recruitment notice posted at the university bulletin were prescreened for their English native level, eye-sight, and topic familiarity. All participants reported daily Internet use longer than an hour and everyday Google usage. Most have been searching online for 7 years or more. The majority also considered themselves as proficient in online information search. To understand how people seek heath information using the Internet and acquire new domain knowledge, we asked each participant to perform three information search tasks (two assigned multifaceted tasks and one self-generated) on health-related topics in counterbalanced order (six rotations), plus one training task. The assigned search tasks followed a simulated work task approach that triggers a realistic information need for participants as they were asked to find useful information for answering the task questions [14] (Table 1).
) Please find at least 3 food items that are considered as good sources of vitamin A. The copyright for this paper remains with its authors. Copying permitted for private and academic purposes. Participants searched publicly available web pages using Google and were asked to save relevant web pages with their typewritten notes and/or information copied/pasted from the source. While there was no time limit, each user session typically lasted from 1.5 to 2 hours. Each participant completed an eHEALS questionnaire, a Pre-and a Post-task Questionnaire, a Post-Search Interview on how they arrived at their solutions for one of the saved web pages per task, and an Exit Questionnaire. During search in the experiment, all of the participants' interactions with the computer system, including eye gaze, brain activity recordings (frontal area), facial expressions (web cam), were recorded. Eye tracking data was collected using a Tobii TX-300 eye-tracker. Participant brain wave levels were recorded using a wireless, consumer-level device headset (MyndWave). At the completion of a session, each participant received $25.
Both the Pre and Post-Task Questionnaires contained two parts: knowledge assessments and interest in a search topic. In knowledge assessments, participants were asked to list as many words or phrases as they can on the topic of a search task with no time limit. As we have just recently finished the study, we focus on participants' responses to the free recall test to identify knowledge gains through information seeking and relate them to basic behavioral measures on Web search, adding eye fixation durations and counts.
Our goals include measuring verbal and concept learning on the search process. We want to measure the difference in participant's knowledge of a search topic before and after each task, hence we need two measurement points. We considered a number of different possibilities of assessing participant's knowledge level on the task topics. We briefly present our deliberations. Factchecking questions before a task were considered inappropriate, because we wanted to avoid exposing participant to the topic's content before they start the search. Since the tasks were conducted on an open web, we could not use a technique such as Sentence Verification Technique (SVT) [15], which requires creation of questions for each document. Our participants were not experts on the topics, hence concept maps and mind-mapping were deemed inappropriate as it is particularly difficult to score for non-experts.
We decided on asking participants to list words and phrases related to each task topic before and after each task. Participants were also asked to annotate relevant web pages and to create from these annotations final notes for each task. Participant entered the annotations while they were on content web pages, whereas the listed words and phrases on pre-and post-task knowledge assessment were from their memory. In addition, we collected a list of keywords and phrases on the assigned task topics from crowd workers on Amazon Mechanical Turk. We plan to use it in assessing participant knowledge by applying automated scoring and calculating semantic similarity using (e.g., using LSA).
Knowledge gain difference in the number of items entered after and before each task (absolute and ratio)
Expertise gain mean frequency of nouns after a task; normalized by the number of nouns used ratio of the mean frequency of nouns after to before a task; normalized by the number of nouns used mean frequency of new nouns used after a task; normalized by the number of nouns used mean rank of nouns listed after a task mean rank of new nouns listed after a task
The methods we used in assessing knowledge included, for example, statement counting [16], word analysis (e.g. word frequency, in particular for nouns), while we plan to use more sophisticated methods in the future (e.g., topic analysis [17] and semantic analysis). The methods aim at assessing knowledge gain and expertise gain. With increasing expertise, people use more sophisticated vocabulary. This sophistication is expressed in the use of less frequent and more specialized vocabulary, hence our use of word usage frequency (and word usage rank) as one of the dependent measures. We used word frequencies and ranks of 1/3 million of most frequent words taken from Google Web Trillion Word Corpus [18] as described by Norvig in chapter 14 in [19].
The mean frequencies and ranks of nouns entered before and after a task differed significantly (Mann-Whitney non-parametric test statistic=229728.5, p=0.0026; Figure 2).
We performed linear regression with the independent variables presented in Table 3 and one dependent variable at a time (Table 2) -thus, we run four regressions. Three of the obtained models (except for ratio of frequencies after and before a task) were significant. However, the values of R 2 were modest and ranged from 0.24 to 0.28. The significant predictors included, 1) number of queries entered and number of SERPs visited in a model with ratio of the number of items entered after and before each task as the dependent variable, and 2) average query length in models with the mean frequency of use of nouns (or new nouns) after a task as the dependent variable.
A plausible interpretation could be that the more queries are issued the more items are entered in the post task knowledge list, and that there is a trade-off with the number of SERPs, namely, with more SERPs visited number of items entered decreases.
For the second and third one, the interpretation is less exciting as it seems to indicate that the longer the average query is the higher the normalized frequency of nouns or new nouns entered in posttask knowledge assessment.
The eye-tracking variables were not found to be significant contributors to the dependent variables of interest. This, perhaps, should not be surprising as they were obtained for all visits to content pages without further differentiation of page content of search task phase. We plan to use more specific eye-tracking measure in our future work.
We reported on our work-in-progress, in which we seek to make a methodological contribution. The results generally indicate a feasibility of the proposed approach, which we may take as an early indication of some success. However, the relative simplicity of employed measures leaves room for improvement and, as indicated throughout the paper, we plan on using more sophisticated assessment techniques.
The broader impact of implicit detection of gains in a person's knowledge and, thus, of learning, lies in its applicability not only to the design of search systems and to improving understanding of human-information interaction but also to a wide variety of information systems, including online learning and intelligent tutoring systems.
This project has been funded in part by IMLS Career award #RE-04-11-0062-11 and in part by a fellowship from School of Information to Jacek Gwizdka.