On an example of a recently conducted user study, we discuss assessment of learning on search as well its correlates in search behaviors and associated eye-tracking measures. Since we are reporting on a work in progress, the study is meant to illustrate our approach and our choices of measures to inspire a discussion.
Our working definition of learning is any change in person’s
knowledge structures. We consider that learning can take place at
many levels [10] and we are particularly influenced by the
cognitive, skill-based, and affective theory of learning outcomes
(CSALO) model [
Our prior work [11–13] has demonstrated feasibility of using eyetracking to detect relationship between eye movement and knowledge levels. The method takes advantage of a direct relationship between eye movement patterns and cognitive processes. One goal of the project presented in this short paper is to connect eye-tracking measures and traditional IR measures (e.g., number and kind of query reformulations) with measures of learning. Since we are reporting on work in progress, the initial The copyright for this paper remains with its authors. Copying permitted for private and academic purposes. data analysis will serve as only a simple illustration of our approach, while our choices of measures will inspire a discussion.
A lab-based experiment was conducted in the Information eXperience lab at University of Texas at Austin (N=30). Data is reported here for 26 of these subjects (16 females; mean age of all participants 24.5). Participants who volunteered after seeing the recruitment notice posted at the university bulletin were prescreened for their English native level, eye-sight, and topic familiarity. All participants reported daily Internet use longer than an hour and everyday Google usage. Most have been searching online for 7 years or more. The majority also considered themselves as proficient in online information search. To understand how people seek heath information using the Internet and acquire new domain knowledge, we asked each participant to perform three information search tasks (two assigned multifaceted tasks and one self-generated) on health-related topics in counterbalanced order (six rotations), plus one training task. The assigned search tasks followed a simulated work task approach that triggers a realistic information need for participants as they were asked to find useful information for answering the task questions [14] (Table 1).
Task 1–Vitamin A: Your teenage cousin has asked your advice in regard to taking vitamin A for health improvement purposes. You have heard conflicting reports about the effects of vitamin A, and you want to explore this topic in order to help your cousin. Specifically, you want to know: 1) What is the recommended dosage of vitamin A for underweight teenagers? 2) What are the health benefits of taking vitamin A? Please find at least 3 benefits and 3 disadvantages of vitamin A. 3) What are the consequences of vitamin A deficiency or excess? Please find 3 consequences of vitamin A deficiency and 3 consequences of its excess. 4) Please find at least 3 food items that are considered as good sources of vitamin A.
Task 2–Hypotension: . Your friend has hypotension. You are curious about this issue and want to investigate more. Specifically, you want to know: 1) What are the causes of hypotension? 2) What are the consequences of hypotension? 3) What are the differences between hypotension and hypertension in terms of symptoms? Please find at least 3 differences in symptoms between them. 4) What are some medical treatments for hypotension? Which solution would you recommend to your friend if he/she also has a heart condition? Why?
Ex.1. Chrohn's disease- I know someone who was recently diagnosed and am curious about the disease.
Ex.2. My friend has lupus. What are the symptoms for lupus? What are the long-term consequences of lupus including the life expectancy? Are there any cures? What treatments are available? Participants searched publicly available web pages using Google and were asked to save relevant web pages with their typewritten notes and/or information copied/pasted from the source. While there was no time limit, each user session typically lasted from 1.5 to 2 hours. Each participant completed an eHEALS questionnaire, a Pre- and a Post-task Questionnaire, a Post-Search Interview on how they arrived at their solutions for one of the saved web pages per task, and an Exit Questionnaire. During search in the experiment, all of the participants’ interactions with the computer system, including eye gaze, brain activity recordings (frontal area), facial expressions (web cam), were recorded. Eye tracking data was collected using a Tobii TX-300 eye-tracker. Participant brain wave levels were recorded using a wireless, consumer-level device headset (MyndWave). At the completion of a session, each participant received $25.
Both the Pre and Post-Task Questionnaires contained two parts: knowledge assessments and interest in a search topic. In knowledge assessments, participants were asked to list as many words or phrases as they can on the topic of a search task with no time limit. As we have just recently finished the study, we focus on participants’ responses to the free recall test to identify knowledge gains through information seeking and relate them to basic behavioral measures on Web search, adding eye fixation durations and counts.
Our goals include measuring verbal and concept learning on the search process. We want to measure the difference in participant's knowledge of a search topic before and after each task, hence we need two measurement points. We considered a number of different possibilities of assessing participant’s knowledge level on the task topics. We briefly present our deliberations. Factchecking questions before a task were considered inappropriate, because we wanted to avoid exposing participant to the topic's content before they start the search. Since the tasks were conducted on an open web, we could not use a technique such as Sentence Verification Technique (SVT) [15], which requires creation of questions for each document. Our participants were not experts on the topics, hence concept maps and mind-mapping were deemed inappropriate as it is particularly difficult to score for non-experts.
We decided on asking participants to list words and phrases related to each task topic before and after each task. Participants were also asked to annotate relevant web pages and to create from these annotations final notes for each task. Participant entered the annotations while they were on content web pages, whereas the listed words and phrases on pre- and post-task knowledge assessment were from their memory. In addition, we collected a list of keywords and phrases on the assigned task topics from crowd workers on Amazon Mechanical Turk. We plan to use it in assessing participant knowledge by applying automated scoring and calculating semantic similarity using (e.g., using LSA).
Knowledge
gain Expertise gain The methods we used in assessing knowledge included, for example, statement counting [16], word analysis (e.g. word frequency, in particular for nouns), while we plan to use more sophisticated methods in the future (e.g., topic analysis [17] and semantic analysis). The methods aim at assessing knowledge gain and expertise gain. With increasing expertise, people use more sophisticated vocabulary. This sophistication is expressed in the use of less frequent and more specialized vocabulary, hence our use of word usage frequency (and word usage rank) as one of the dependent measures. We used word frequencies and ranks of 1/3 million of most frequent words taken from Google Web Trillion Word Corpus [18] as described by Norvig in chapter 14 in [19].
The mean frequencies and ranks of nouns entered before and after a task differed significantly (Mann-Whitney non-parametric test statistic=229728.5, p=0.0026; Figure 2).
We performed linear regression with the independent variables presented in Table 3 and one dependent variable at a time (Table 2) – thus, we run four regressions. Three of the obtained models (except for ratio of frequencies after and before a task) were significant. However, the values of R2 were modest and ranged from 0.24 to 0.28.
Time on task Query count
Query length Number of pages visited
Time on a page Total fixation duration
Count of fixations
Proportion of reading fixations Proportion of durations of reading fixations
Number of SERPs visited The significant predictors included, 1) number of queries entered and number of SERPs visited in a model with ratio of the number of items entered after and before each task as the dependent variable, and 2) average query length in models with the mean frequency of use of nouns (or new nouns) after a task as the dependent variable.
A plausible interpretation could be that the more queries are issued the more items are entered in the post task knowledge list, and that there is a trade-off with the number of SERPs, namely, with more SERPs visited number of items entered decreases. For the second and third one, the interpretation is less exciting as it seems to indicate that the longer the average query is the higher the normalized frequency of nouns or new nouns entered in posttask knowledge assessment.
The eye-tracking variables were not found to be significant contributors to the dependent variables of interest. This, perhaps, should not be surprising as they were obtained for all visits to content pages without further differentiation of page content of search task phase. We plan to use more specific eye-tracking measure in our future work.
We reported on our work-in-progress, in which we seek to make a methodological contribution. The results generally indicate a feasibility of the proposed approach, which we may take as an early indication of some success. However, the relative simplicity of employed measures leaves room for improvement and, as indicated throughout the paper, we plan on using more sophisticated assessment techniques.
The broader impact of implicit detection of gains in a person’s knowledge and, thus, of learning, lies in its applicability not only to the design of search systems and to improving understanding of human-information interaction but also to a wide variety of information systems, including online learning and intelligent tutoring systems.
This project has been funded in part by IMLS Career award #RE04-11-0062-11 and in part by a fellowship from School of Information to Jacek Gwizdka.