=Paper=
{{Paper
|id=Vol-3052/paper5
|storemode=property
|title=A Triangulation Perspective for Search as Learning
|pdfUrl=https://ceur-ws.org/Vol-3052/paper5.pdf
|volume=Vol-3052
|authors=Nilavra Bhattacharya,,Jacek Gwizdka
|dblpUrl=https://dblp.org/rec/conf/cikm/BhattacharyaG21
}}
==A Triangulation Perspective for Search as Learning==
https://ceur-ws.org/Vol-3052/paper5.pdf
A Triangulation Perspective for Search as Learning
Nilavra Bhattacharya1 , Jacek Gwizdka1
1
School of Information, The University of Texas at Austin, USA
Abstract
Search engines and information retrieval (IR) systems are becoming increasingly important as educational platforms to foster
learning. Modern search systems still have room to improve in this regard. We posit that learning-during-search is a good
candidate for a human-centred metric of IR evaluation. This involves measuring two phenomena: learning, and searching.
We discuss ways to measure learning, and propose a conceptual framework for describing searchers’ knowledge-change
during search. We stress the need for developing better measures for the search process, and discuss why we need to rethink
the existing models of information seeking.
1. Introduction at supporting intelligence amplification and knowledge
building [3]. In the last decade, this recognition that IR
As early as in 1980, Bertam Brookes, in his ‘fundamental systems of tomorrow can become “rich learning spaces”
equation’ of information and knowledge: 𝐾[𝑆] + ∆𝐼 = and foster knowledge gain, has led to the emergence of
𝐾[𝑆 + ∆𝑆] had stated that a searcher’s current state the Search as Learning (SAL) research community [4],
of knowledge, 𝐾[𝑆], is changed to the new knowledge and the need to consider learning-during-search as a
structure, 𝐾[𝑆 + ∆𝑆], by exposure to information ∆𝐼, metric for evaluation of Interactive IR (IIR) systems.
with the ∆𝑆 indicating the effect of the change [1, p. 131].
This indicates that searchers acquire new knowledge in
the search process, and the same information ∆𝐼 may 2. Metrics for Learning &
have different effects on different searchers’ knowledge Knowledge
states. Fifteen years later, Marchionini described informa-
tion seeking as “a process, in which humans purposefully 2.1. Experts vs. Novices
engage in order to change their state of knowledge” [2].
Thus, we have known for quite a while that search is If we consider learning-during-search to be a good can-
driven by higher-level human needs, and Information didate for IR evaluation criterion, the next challenge is
Retrieval (IR) is a means to an end, and not the end in how to measure learning, or knowledge acquisition,
itself. possibly in an automated fashion. We can turn to educa-
When we consider information seeking as a process tional psychology literature. A research report by the US
that changes the searcher’s knowledge-state, the question National Research Council [5] identified the following
arises whether the assessment of knowledge-acquisition- key principles about experts’ knowledge, illustrating the
during-search, or learning, should subsume the standard results of successful knowledge acquisition:
IR evaluation metrics and the search interface usability
1. “Experts notice features and meaningful patterns
metrics. It seems that to diagnose a problem or to un-
of information that are not noticed by novices.”
derstand a success of a search system, we would still
need to control the standard aspects of a search system 2. “Experts have acquired a great deal of content
(e.g., results ranking, search user interface design fea- knowledge that is organized in ways that reflect
tures). However, a direct assessment of these “lower- a deep understanding of their subject matter.”
level” aspects would lose on importance. On the other 3. “Experts’ knowledge cannot be reduced to sets of
hand, support for more rapid learning across a number isolated facts or propositions but, instead, reflects
of searchers, and over a range of different search tasks contexts of applicability: that is, the knowledge
can be indicative of an IR system that is more effective is ‘conditionalized’ on a set of circumstances.”
4. “Experts are able to flexibly retrieve important
Proceedings of the CIKM 2021 Workshops, November 1–5, Gold Coast, aspects of their knowledge with little attentional
Queensland, Australia
" nilavra@ieee.org (N. Bhattacharya); iwilds2020@gwizdka.com
effort.”
(J. Gwizdka)
~ https://nilavra.in (N. Bhattacharya); http://gwizdka.com
Some of the above findings have been used by our com-
(J. Gwizdka) munity in the past. E.g, user learning has been measured
0000-0001-7864-7726 (N. Bhattacharya); 0000-0003-2273-3996 by user’s familiarity with concepts and relationships be-
(J. Gwizdka) tween concepts [6], gains in user’s understanding of the
© 2021 Copyright for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0). topic structure [7], and user’s ability to formulate more
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org)
Pre-Search Pre-Post Post-Exp Pre-Exp
Row
Knowledge State No. [Level of Knowledge Change] Intuition
Dist (A) Dist (B) Dist (C)
1 Low Low Low [Low] Expert
2 Low Low HIGH X
3 Low HIGH Low X
(A) 4 Low HIGH HIGH [Low] Slow learner
5 HIGH Low Low X
6 HIGH Low HIGH [High] User gained new knowledge; desirable
[High] Knowledge loss: started near expert-
Post‒Exp Distance
Post-Search Expert 7 HIGH HIGH Low knowledge state before search; went further away
(B)
Knowledge State Knowledge from expert knowledge after search
State
[High] Mis-directed search/learning: started
8 HIGH HIGH HIGH
away from expert-knowledge; went further away
Figure 1: A triangulation perspective for knowledge-change during search. We assume Pre-Search Knowledge, Post-Search
Knowledge, and the reference Expert Knowledge as three vertices of a triangle (left figure). If we can compute the distance
between the triangle’s vertices, and then further dichotomize these distances as HIGH vs. Low, then we can have eight
possible outcomes (right table). ‘X’ denotes outcomes violating the triangle inequality.
effective queries [8, 6]. From the above findings, we can 2.3. A Triangulation Perspective
think about ways to consider Expert’s Knowledge on the
We can conceptualize a triangle-based framework for
search topic as ‘gold-standard’ or ‘ground-truth’ (by al-
searchers’ knowledge-change during search (Fig. 1)
gorithmic parlance), for developing learning based IIR
Searchers initiate a search session with a Pre-Search
evaluation metrics.
Knowledge state. During search, they undergo a change
in knowledge. On conclusion of search, searchers attain
2.2. Measuring Knowledge-Change the Post-Search Knowledge state. We can attempt to mea-
Recent literature on Search-as-Learning adopts three sure this dynamic knowledge-change from a stationary
broad approaches to measure learning, or knowledge- reference point: Expert Knowledge on the search topic
change, with their own strengths and limitations. The (ground-truth). If we imagine these three knowledge-
first approach asks searchers to rate their self-perceived states to be the three vertices of a triangle (Fig. 1, left),
pre-search and post-search knowledge levels [9, 10]. This and if, by some hypothetical metric, we can compute
approach is the easiest to construct, and can be gener- the distance between any two of these knowledge-state
alised over any search topic. However, self-perceptions points, then we have found a way to quantify learning-
may not objectively represent true learning. The second during-search.
approach tests searchers’ knowledge using factual mul- Moving further, if we dichotomize the learning-during-
tiple choice questions (MCQs). The answer options can search as ‘HIGH’ vs ‘Low’, by establishing a threshold
be a mixture of fact-based responses (TRUE, FALSE, or I value for the distances, then we can obtain eight possible
DON’T KNOW ), [11, 12] or recall-based responses (I re- knowledge-change situations (Fig. 1, right table). Three
member / don’t remember seeing this information) [13, 14]. of these eight situations violate the triangle inequality1
Constructing topic-dependant MCQs may take time and (denoted by ‘X’ in the table), and are therefore discarded.
effort, which may be aided by automated question gen- The remaining five valid situations are discussed below.
eration techniques [15]. For evaluation, this approach When Pre-Search Knowledge State and Post-Search
is the easiest, and often automated. However, MCQs al- Knowledge State are both very ‘close’ to Expert Knowl-
low respondents to answer correctly by guesswork. The edge (row 1 in table), we can assume the searcher is an
third approach lets searchers write natural language expert. On the other hand, if Pre-Search Knowledge
summaries or short answers, before and after the search State and Post-Search Knowledge State are close to each
[16, 10]. Depending on experimental design, prompts for other, but are far away from Expert Knowledge (row
writing such responses can be generic (least effort) [17] 4), the searcher is probably a novice, and also a slow
or topic-specific (some effort) [15]. While this approach learner, because on conclusion of search, their knowl-
provides rich information about a searcher’s knowledge edge still remained far away from Expert Level. When
state, evaluating such responses is the most challenging.
1
sum of lengths of any two sides of a triangle is greater than
the third side
the Post-Search Knowledge is closer to Expert than Pre- knowledge-gaps and misconceptions. They have been
Search Knowledge (row 6), it implies that the searcher used for over 50 years to provide a useful and visually
gained ‘good amount’ of new knowledge, and is thus, the appealing way of illustrating and assessing learners’ con-
most desirable situation for Search as Learning. ceptual knowledge [25, 20, 24, 26, 27, 28, 29].
The last two rows of the table in Fig. 1 present two Expert knowledge or “ground truth” can be repre-
interesting, albeit undesirable, possibilities. If the Pre- sented as topical knowledge-graphs of the information
Search Knowledge is closer to Expert, but the Post-Search contained in online encyclopedias and knowledge bases.
Knowledge is further away (row 7), it can signify knowl- Searcher’s pre and post-search knowledge states can
edge loss (which is also a form of knowledge change). On be represented as concept maps or personal knowledge
the other hand, if both the Pre-Search and the Post-Search graphs. The searcher’s graphs will evolve cumulatively
knowledge are far away from Expert, and they are also over time, as the they encounter more information online.
far away from each other (row 8), then it is a case of mis- Construction of the personal knowledge graph can be
directed search, and therefore, misdirected learning. manual (most effort), fully automated (least effort, but
A classic illustration of these two situations is health in- prone to prediction errors), or a human-in-the loop so-
formation seeking. Suppose a user is searching for cause lution (an auto generated map is shown, but the user is
and treatment of a small brownish spot on the wrist. If free to modify it as necessary).
a physician examined the spot, they would immediately Having represented knowledge states as graph-based
identify the spot to be caused by oil-splatter burn during structures, measuring the similarity or distances between
cooking (Expert Knowledge State). The searcher may them becomes equivalent to the graph matching problem.
however, based on search results, come to the incorrect Various algorithms and metrics have been proposed for
conclusion that they have skin cancer [18, 19]. Before exact and inexact graph matching [30]. Many of the solu-
the search, if the searcher correctly guessed that the spot tions take an optimization-problem approach [31]. Some
was due to oil splatter burn, then the situation would be examples include structural similarity matching (compar-
described by row 7 (knowledge loss, or increase in confu- ing diameter, edges, distribution degrees etc.), iterative
sion), whereas if the searcher had no intuition about the matching (comparing the node neighbours), subgraph
cause of the spot before the search, the situation would comparison, and graph isomorphism [32].
be described by row 8. Both situations should be avoided Besides comparing two graphs, other kinds of analyses
by modern IIR systems. can reveal interesting patterns of learning and thinking,
which can be correlated with search process measures.
2.4. Graph-based Operationalization Some of these measures that have been used by Halt-
tunen and Jarvelin [24] are addition, deletion, and dif-
While the framework discussed in Section 2.3 is purely ferences in top-level concept-nodes, depths of hierarchy,
conceptual, we can think of a possible operationalization and number of concepts that were ignored or changed
using graph-based representations, such as concept maps fundamentally. In this regard, Novak and Gowin [25]
[20] or personalized knowledge graphs [21] (the terms have presented well-established scoring scheme to eval-
are used interchangeably in this section). uate concept-maps: 1 point is awarded for each correct
“Learning does not happen all at once . . . it builds relationship (i.e. concept–concept linkage); 5 points for
on and is shaped by what people already know” [3]. each valid level of hierarchy; 10 points for each valid and
The Learning and the Cognitive Sciences have gener- significant cross-link; and 1 point for each example. Such
ally discovered that meaningful “deep learning” (of the analyses methods can further inform the development of
human kind) requires learners to: (i) relate new ideas future operationalizations.
and concepts to previous knowledge and experiences; As our anonymous reviewers mentioned, knowing the
(ii) integrate knowledge into interrelated conceptual sys- goal of the learner is important in this scenario, as that
tems; and (iii) look for patterns and underlying prin- will guide the formation of the learner’s personal map.
ciples [22, 23]. Concept maps are arguably, therefore, Furthermore, a search systems (or internet browsers) may
extremely suited to represent such knowledge struc- provide a special ‘learning mode’ which is dedicated for
tures, connecions, and patterns. A concept-map is a two- measuring learning. This will help to avoid transactional
dimensional, hierarchical node-link diagram (graph) that or navigational search sessions that not necessarily aimed
depicts the structure of knowledge within a discipline, at learning/knowledge acquisition.
as viewed by a student, an instructor, or an expert in a
field or sub-field. The map is composed of concept labels,
each enclosed in a box (graph nodes); a series of labelled 3. Measuring the Search Process
linking lines (labelled edges); and an inclusive, general-
to-specific organization [24]. Concept-maps assess how Learning-during-search involves two intertwined activ-
well students see the “big picture”, and where there are ities: learning, and searching. In Sec. 2, we discussed
approaches to measure learning. The other part of the devices. For instance, Marchionini’s well known infor-
picture involves measuring the search process itself. Past mation seeking process (ISP) [2] models the information
research efforts has largely been devoted to measuring seeking behaviour into eight stages, with connecting
search outcomes: e.g., if a target document was reached, feed-forward and feed-back loops between the stages.
or if relevant results were shown. We argue that a more However, some researchers argue that users never really
human-centred approach for measuring search is trying go “back” to an earlier state; e.g., “when reformulating
to quantify the search process. the query, users do not really go back to the initial situa-
tion, they submit an improved query” [40]. With progress
3.1. Need for Longitudinal Studies of time, there is continuous update of users’ information
need [41] and search context [42]. Thus, the intricate rela-
A major limitation of most IIR research efforts is that tionships between users’ knowledge state, cognitive state,
the user is examined in the short-term, typically over the and other factors influencing search (search context), are
course of a single lab session. The trend is similar in other ever-changing. Perhaps then Spink’s model of the IR
HCI research venues. [33] stressed the need for longitu- interaction process [43], which models interactive search
dinal designs over a decade ago, yet a meta-analysis of as an infinite continuous process of sequential steps, or
1014 user studies reported in the ACM CHI 2020 confer- cycles2 , is better suited to explain information searching
ence revealed that more than 85% of the studies observed behaviour. Like time, there may not be an absolute be-
participants for a day or less. To this day, “longitudi- ginning or end of a user’s information searching process,
nal studies are the exception rather than the norm” [34]. but only search sessions. The user’s cognitive state is
On the other hand, it is quite evident that knowledge always ever-changing and advancing, both during and
acquisition is a longitudinal process, occurring gradu- between these search sessions. So a more realistic model
ally over time [3, 23, 5, 22]. Therefore, most educational will probably mean a fusion of Marchionini’s and Spink’s
curricula in schools and universities are spread across models, where Marchionini’s entire ISP process becomes
several months and years. “An over-reliance on short a cycle inside the Spink’s model, with forward-directed
studies risks inaccurate findings, potentially resulting in arrows only. These types of realistic models, improved
prematurely embracing or disregarding new concepts” and validated by empirical data, will help to explain phe-
[34]. nomena behind next-generation search interactions, such
as searching and multi-tasking, multi-tabbed browsing [3,
3.2. Need for Updated Theoretical p. 36], multi-device searching, and multi-session search-
ing [3, p. 61].
Models
The Information Seeking literature is dominated by a 3.3. Neuro-physiological methods
large number of “multiple arrow-and-box” theoretical
models. These models divide the information seeking Neuro-physiological methods (NP methods) [44] provide
process for complex search-tasks into different stages. an interesting avenue to observe users while they inter-
Some argue that these models are not not “real mod- act with information systems. Two popular NP methods
els” but more of “short-hand common-sense task flows” are eye-tracking [45, 46] and EEG [47]. Eye-tracking
[35, 36]. The mantra of these models have always been can capture eye-movements of users while they exam-
the same: they have “implications for systems design ine information on a screen. EEG captures (changes in)
and practice”. Unfortunately, these models, along with a activation in different brain regions as users consume
significant body of IIR research, has not been able to go information. NP methods provide opportunities to under-
beyond suggestions, to providing concrete design solu- stand and investigate how users gain knowledge during
tions [37]. Moreover, there is great overlap in basic search search. E.g, searchers use words or phrases they read
strategies across many of these models [38], calling into in previous search results, in their future query refor-
question whether so many models are still relevant. Con- mulations [48]. Eye-tracking can detect and model this
sequently, current search systems still predominantly use phenomenon. As a result, a number of recent efforts have
a “one-size-fits-all” approach: one interface is used for all tried to investigate learning (during search) using one or
stages of a search, even for complex search endeavours more NP methods [16, 17, 49, 50, 51, 15]. However, a ma-
[39]. jor limitation of NP methods is that they (still) require lab
Again reiterating [33], we posit that these models, the- environments for data collection. Taking lessons from the
orised decades ago for bulky desktop computers, are in COVID-19 pandemic, as well as for scalability reasons,
need of improvement. Information seeking models have the IIR community needs search process metrics that can
to incorporate the continuous or lifelong nature of online 2
where each cycle consists of one or more interactive feedback
information searching, enabled by the proliferation of occurrences of user’s query input, IR system output, and user’s in-
internet access in various handheld and portable digital terpretation and judgement of the output
measure remote user interaction, preferably over the long to ‘track’ and measure their knowledge progress over
term. Consumer wearable devices (e.g., smartwatches) time, in a manner similar to tracking weight, fitness and
are a promising direction, since they can record physi- physical exercises.
ological data such as heart rate, skin temperature, and
galvanic skin response. White et al. [52] collected such
data at a population scale, and correlated them with the Acknowledgments
population’s search activities, to obtain improvements in
We thank the anonymous reviewers for their very helpful
relevance of result rankings.
and thought provoking suggestions and feedback.
4. Conclusion References
The perspectives and propositions in this paper have
[1] B. C. Brookes, The foundations of information sci-
been shaped by our experience in IIR research. The Infor-
ence. part i. philosophical aspects, Journal of infor-
mation Processing Model from Educational Psychology
mation science 2 (1980) 125–133.
states that information is most likely to be retained by
[2] G. Marchionini, Information Seeking in Electronic
a learner if it makes sense, and has meaning [53, p. 55].
Environments, Cambridge University Press, 1995.
When a piece of information fits into the world-view of
[3] R. W. White, Interactions with Search Systems,
the learner, it is said to make sense; when information
Cambridge University Press, 2016. doi:10.1017/
is relevant to the learner, it has meaning. Our past re-
CBO9781139525305.
search have primarily been in the second aspect of in-
[4] S. Y. Rieh, Research area 1: Searching as learning,
formation retention: relevance judgement. After several
https://rieh.ischool.utexas.edu/research, 2020. [On-
user-studies and analysing multimodal sources of data,
line; accessed 2020-04-19].
we generally conclude that relevant information attracts
[5] N. R. Council, How People Learn: Brain, Mind,
more visual attention, longer eye-dwell time, and more
Experience, and School: Expanded Edition, The
brain activations [45, 54, 17], compared to irrelevant in-
National Academies Press, Washington, DC, 2000.
formation. Metrics which can capture the entire duration
doi:10.17226/9853.
of an experimental trial, or the real-time flow of interac-
[6] P. Pirolli, P. Schank, M. Hearst, C. Diehl, Scat-
tions, usually perform better as predictors, than metrics
ter/gather browsing communicates the topic struc-
which aggregate the entire trial into a set of single num-
ture of a very large text collection, in: Conference
bers [46, 55, 54]. Hence we call for new and improved
on Human Factors in Computing Systems (CHI’96),
measures of the search process.
1996.
In the domain of Search as Learning, we employed
[7] P. Zhang, D. Soergel, Process patterns and concep-
word [56] and sentence [17] embeddings to semanti-
tual changes in knowledge representations during
cally compare searcher’s responses to expert knowledge.
information seeking and sensemaking: A qualita-
Word embeddings provided better visualization of re-
tive user study, Journal of Information Science 42
sults, showing clear separation of Pre-Search Knowledge
(2016).
from Post-Search and Expert Knowledge [56]. We also
[8] Y. Chen, Y. Zhao, Z. Wang, Understanding online
co-related Knowledge Change measures with interaction
health information consumers’ search as a learning
and eye-tracking measures. We saw that people who
process, Library Hi Tech (2020).
learnt ‘less’ spent more reading effort on SERPs [17].
[9] S. Ghosh, M. Rath, C. Shah, Searching as learning:
Conversely, people who learnt ‘more’ were doing less
Exploring search behavior and learning outcomes
reading overall; but most of their reading was on content
in learning-related tasks, in: Conference on Human
pages. These high learners used more specialized terms
Information Interaction & Retrieval (CHIIR), 2018.
in their queries, and reported higher mental workload
[10] H. L. O’Brien, A. Kampen, A. W. Cole, K. Brennan,
(NASA-TLX).
The role of domain knowledge in search as learning,
In conclusion, we reiterate that learning-during-search
in: Conference on Human Information Interaction
is a good candidate for evaluating IR systems. We need
and Retrieval (CHIIR), 2020.
more research to uncover relationships between the
[11] L. Xu, X. Zhou, U. Gadiraju, How does team com-
users’ search process and their learning outcomes. Pro-
position affect knowledge gain of users in collabo-
cess measures can shed light on the various subtle aspects
rative web search?, in: Conference on Hypertext
of human behaviour. If we understand them well, we can
and Social Media (HT’20), 2020.
teach people to be more successful in their information
[12] U. Gadiraju, R. Yu, S. Dietze, P. Holtz, Analyzing
seeking efforts, and maximize their learning outcomes.
knowledge gain of users in informational search
We envision that in the future, searchers will be able
sessions on the web, in: Conference on Human [26] Y. Egusa, H. Saito, M. Takaku, H. Terai, M. Miwa,
Information Interaction & Retrieval (CHIIR), 2018. N. Kando, Using a Concept Map to Evaluate Ex-
[13] S. Kruikemeier, S. Lecheler, M. M. Boyer, Learning ploratory Search, in: Proceedings of the Third Sym-
from news on different media platforms: An eye- posium on Information Interaction in Context, IIiX
tracking experiment, Political Communication 35 ’10, ACM, New York, NY, USA, 2010, pp. 175–184.
(2018) 75–96. doi:10.1145/1840784.1840810.
[14] N. Roy, F. Moraes, C. Hauff, Exploring users’ learn- [27] Y. Egusa, M. Takaku, H. Saito, How Concept
ing gains within search sessions, in: Conference Maps Change if a User Does Search or Not?, in:
on Human Information Interaction and Retrieval Proceedings of the 5th Information Interaction in
(CHIIR), 2020. Context Symposium, IIiX ’14, ACM, New York,
[15] R. Syed, K. Collins-Thompson, P. N. Bennett, NY, USA, 2014, pp. 68–75. doi:10.1145/2637002.
M. Teng, S. Williams, D. W. W. Tay, S. Iqbal, Im- 2637012.
proving learning outcomes with gaze tracking and [28] Y. Egusa, M. Takaku, H. Saito, How to evaluate
automatic question generation, in: The Web Con- searching as learning, in: Searching as Learning
ference (WWW), 2020. Workshop (IIiX 2014 Workshop), Regensburg, Ger-
[16] N. Bhattacharya, J. Gwizdka, Relating eye-tracking many, 2014. URL: http://www.diigubc.ca/IIIXSAL/
measures with changes in knowledge on search program.html.
tasks, in: Symposium on Eye Tracking Research & [29] Y. Egusa, M. Takaku, H. Saito, Evaluating Com-
Applications (ETRA’18), 2018. plex Interactive Searches Using Concept Maps., in:
[17] N. Bhattacharya, J. Gwizdka, Measuring learning SCST@ CHIIR, 2017, pp. 15–17.
during search: Differences in interactions, eye-gaze, [30] Wikipedia, Graph matching, https://en.wikipedia.
and semantic similarity to expert knowledge, in: org/wiki/Graph_matching, 2021. (Accessed on 2021-
CHIIR’19, 2019. 10-14).
[18] D. M. Australia, Why googling symptoms leads to [31] X. Bai, Graph-Based Methods in Computer Vision:
cancer diagnosis and worse for your health, https: Developments and Applications: Developments
//www.dailymail.co.uk/news/article-6927695, 2019. and Applications, IGI global, 2012.
(Accessed on 02/07/2021). [32] Wikipedia, Graph isomorphism, https:
[19] H. Times, Stop it! google is not a real //en.wikipedia.org/wiki/Graph_isomorphism,
doctor and no you don’t have cancer, 2021. (Accessed on 2021-10-14).
https://www.hindustantimes.com/health/ [33] D. Kelly, S. Dumais, J. O. Pedersen, Evaluation
story-EjdPqGmAe2CugNyNdUClSI.html, 2015. challenges and directions for information-seeking
(Accessed on 02/07/2021). support systems, IEEE Computer 42 (2009).
[20] J. D. Novak, Learning, Creating, and Using Knowl- [34] L. Koeman, Hci/ux research: what methods do we
edge: Concept Maps as Facilitative Tools in Schools use? – lisa koeman – blog, https://lisakoeman.nl/
and Corporations, 2nd ed ed., Routledge, New York, blog/hci-ux-research-what-methods-do-we-use/,
NY, 2010. 2020-06-18. (Accessed on 11/08/2020).
[21] K. Balog, P. Mirza, M. G. Skjæveland, Z. Wang, [35] A. Dillon, Personal communication, 2020. [dated
Workshop on personal knowledge graphs, co- 2020-04-29].
located with the 3rd automatic knowledge base [36] A. Dillon, No more information seeking models
construction conference (akbc’21), https://pkgs.ws/, please, https://adillon.ischool.utexas.edu/2006/10/
2021. (Accessed on 2021-10-14). 03/no-more-information-seeking-models-please/,
[22] R. K. Sawyer, The Cambridge handbook of the learn- 2006. [Online; accessed 2020-05-28].
ing sciences, Cambridge University Press, 2005. [37] T. Saracevic, Information science, Jour-
[23] S. A. Ambrose, M. W. Bridges, M. DiPietro, M. C. nal of the American Society for Infor-
Lovett, M. K. Norman, How Learning Works: Seven mation Science 50 (1999) 1051–1063.
Research-Based Principles for Smart Teaching, John doi:10.1002/(SICI)1097-4571(1999)50:
Wiley & Sons, 2010. 12<1051::AID-ASI2>3.0.CO;2-Z.
[24] K. Halttunen, K. Jarvelin, Assessing learning out- [38] O. Hoeber, D. Patel, D. Storie, A study of aca-
comes in two information retrieval learning envi- demic search scenarios and information seeking
ronments, Information Processing & Management behaviour, in: Conference on Human Information
41 (2005) 949–972. doi:10.1016/j.ipm.2004.02. Interaction and Retrieval (CHIIR), 2019, pp. 231–
004. 235.
[25] J. D. Novak, D. B. Gowin, Learning How to Learn, [39] H. C. Huurdeman, J. Kamps, From multistage
Cambridge University Press, 1984. doi:10.1017/ information-seeking models to multistage search
CBO9781139173469. systems, in: Proceedings of the 5th Information
Interaction in Context Symposium, 2014, pp. 145– Psychology 66 (2013) 2289–2294.
154. [52] R. White, R. Ma, Improving search engines via
[40] V. T. Tran, N. Fuhr, Using eye-tracking with dy- large-scale physiological sensing, in: Conference
namic areas of interest for analyzing interactive in- on Research and Development in Information Re-
formation retrieval, in: Conference on Research and trieval (SIGIR’17), 2017.
Development in Information Retrieval (SIGIR’12), [53] D. A. Sousa, How the Brain Learns, Fifth Edition,
2012. Corwin Press, 2017.
[41] X. Huang, D. Soergel, Relevance: An improved [54] J. Gwizdka, R. Hosseini, M. Cole, S. Wang, Temporal
framework for explicating the notion, Journal of the dynamics of eye-tracking and EEG during reading
American Society for Information Science and Tech- and relevance decisions, Journal of the Association
nology 64 (2013) 18–35. doi:10.1002/asi.22811. for Information Science and Technology (2017).
[42] T. Saracevic, The Notion of Relevance in Informa- [55] N. Bhattacharya, S. Rakshit, J. Gwizdka, Towards
tion Science: Everybody knows what relevance is. real-time webpage relevance prediction using con-
But, what is it really?, Synthesis Lectures on Infor- vex hull based eye-tracking features, in: Sympo-
mation Concepts, Retrieval, and Services (2016). sium on Eye Tracking Research & Applications
[43] A. Spink, Study of interactive feedback during medi- (ETRA ’20), 2020.
ated information retrieval., Journal of the American [56] N. Bhattacharya, J. Gwizdka, Visualizing and quan-
Society for Information Science (1997). tifying vocabulary learning during search, in: Pro-
[44] J. Gwizdka, Y. Moshfeghi, M. L. Wilson, Introduc- ceedings of the CIKM 2020 Workshops, October
tion to the special issue on neuro-information sci- 19-20, 2020, Galway, Ireland, 2020.
ence, Journal of the Association for Information
Science and Technology 70 (2019) 911–916.
[45] J. Gwizdka, Characterizing Relevance with Eye-
tracking Measures, in: Proceedings of the 5th In-
formation Interaction in Context Symposium, IIiX
’14, ACM, New York, NY, USA, 2014, pp. 58–67.
doi:10.1145/2637002.2637011, 00028.
[46] N. Bhattacharya, S. Rakshit, J. Gwizdka, P. Kogut,
Relevance prediction from eye-movements using
semi-interpretable convolutional neural networks,
in: Conference on Human Information Interaction
and Retrieval (CHIIR’20), 2020.
[47] Z. Pinkosova, W. J. McGeown, Y. Moshfeghi, The
cortical activity of graded relevance, in: Confer-
ence on Research and Development in Information
Retrieval (SIGIR), 2020, pp. 299–308.
[48] C. Eickhoff, S. Dungs, V. Tran, An eye-tracking
study of query reformulation, in: Proceedings
of the 38th International ACM SIGIR Conference
on Research and Development in Information Re-
trieval, SIGIR ’15, Association for Computing Ma-
chinery, Santiago, Chile, 2015, pp. 13–22. doi:10.
1145/2766462.2767703.
[49] S. Kruikemeier, S. Lecheler, M. M. Boyer, Learning
From News on Different Media Platforms: An Eye-
Tracking Experiment, Political Communication 35
(2018) 75–96.
[50] J. Mao, Y. Liu, N. Kando, M. Zhang, S. Ma, How
does domain expertise affect users’ search inter-
action and outcome in exploratory search?, ACM
Transactions on Information Systems 36 (2018).
[51] M. S. Franklin, J. M. Broadway, M. D. Mrazek,
J. Smallwood, J. W. Schooler, Window to the wan-
dering mind: Pupillometry of spontaneous thought
while reading, Quarterly Journal of Experimental