=Paper=
{{Paper
|id=Vol-1798/paper7
|storemode=property
|title=Contextualizing the Cognition Crisis
|pdfUrl=https://ceur-ws.org/Vol-1798/paper7.pdf
|volume=Vol-1798
|authors=Alamir Novin
|dblpUrl=https://dblp.org/rec/conf/chiir/Novin17a
}}
==Contextualizing the Cognition Crisis==
https://ceur-ws.org/Vol-1798/paper7.pdf
Contextualizing the Cognition Crisis
Alamir Novin
University of British Columbia
1961 East Mall, Vancouver, BC, Canada
alamir.novin@ubc.ca
ABSTRACT controversial term is directed at the finding by the Open Science
Collaboration (OSC) that the vast majority of papers on cognitive
This workshop submission argues for a more socio-technical biases are irreproducible by other researchers [3]. However,
approach to scholarly explorations of task complexity in user
Information Scientists, amongst other scholars, have objected to
search and retrieval. Doing so will lead to results that are not
the methods used by the OSC. Most notably, in their paper
only more applicable outside of the lab and in the "real world,"
"More on 'Estimating the Reproducibility of Psychological
but also to results that are scientifically more 1) consistent, 2)
Science'" the authors Daniel Gilbert, Gary King, Stephen
reproducible, and 3) falsifiable. To demonstrate the importance
Pettigrew, and Timothy Wilson (2016) summarize their back and
of this approach, this submission draws on recent findings that
forth exchange with the OSC thus:
cognitive experiments that fail to contextualize cognitive tasks
lead to irreproducible results both within and outside of the lab.
OSC: “We have provided a credible estimate of the reproducibility
These same problems transfer to complex search tasks that draw
of psychological science.”
from the same cognitive studies (e.g., with confirmation bias) or
limit the contextualization in their own experiments. However, a US: “No, you haven’t, because (1) you violated the basic rules of
distributed-cognition approach acknowledges how cognition is sampling when you selected studies to replicate, (2) you did
distributed amongst contextual factors. The author suggests that unfaithful replications of many of the studies you selected, and (3)
a mixed-methods approach could assist with capturing more you made statistical errors.”
context. Specifically, a population-based survey experiment may OSC (& OTHERS): “We didn’t make statistical errors.”
provide a straight forward method for information search and
retrieval scholars to create an internal valid experimental study In their latest paper, Gilbert et al., (2016) conclude: "while some
with some of the generalizability from survey sampling. With colleagues have challenged our Point 3, none has challenged our
this in mind, the workshop suggests that instead of setting out to Points 1 or 2, probably because it requires no special expertise to
reproduce cognitive effects via mechanistic changes in a work see that these points are inarguable facts" [4]. The author of this
task, the context of the work task should be determined first. workshop will not challenge, but expand on Point 2.
After the context and task are determined, the mechanistic
variables (e.g., interface features) for participants can be This workshop submission builds on these prior studies by
experimented with. Finally, the author draws from a long suggesting that scholars investigating complex search tasks can
scholarly debate between various information scientists and make their designs more context-based by 1) first determining
cognitive science labs to argue that the analysis of observations the context prior to a work task. Next 2) data is collected via a
can determine which cognitive effects provide the best mixed-methods strategy known as a population-based survey
explanatory power. experiment that combines an experimental design testing the
work-task with the field survey sampling methods for recruiting
CCS Concepts a population. 3) Finally, both the collected quantitative and
Information systems~Query representation • Information qualitative data are analyzed in specific sequences to make
systems~Search interfaces inferences on the relationship between a person's context, task,
and their cognition.
Keywords
search engine results page; cognitive bias; biases; document Background: The 'Replicability Crisis' Debate
order; context; cognition; complex search; work tasks The large analysis conducted by the OCR was the
Reproducibility Project, whereby 270 researchers from academic
Introduction institutions across the globe conducted a massive examination of
The argument that research experiments require greater social cognitive-biases on 100 psychology studies from reputable
context is often perceived as an argument for a "soft-science" journals. Only 39% of the studies were reproducible [3].
approach to research. However, what is often forgotten is that Reproducibility problems are not a new phenomenon [5] and
providing social context increases the validity of scientific they affect a variety of fields, but the "soft sciences" (e.g., social
principles fundamental to the hard sciences, specifically the science and psychology) could be at greater risk of them because
reproducibility and falsifiability [1]. To illustrate this point, the they produce the most positive results in comparison [6]. On the
author refers to a current scholarly debate taking place amongst other hand, "hard sciences," such as physics, publish their
scientists, often via Nature journal's magazine, that is negative results more frequently [6]. In response, a few scholars
controversially referred to as the "replicability crisis" [2]. The called for greater protocol in the process of cognitive research.
Daniel Kahneman (2013), the respected psychologist, suggests
CHIIR 2017 Workshop on Supporting Complex Search Tasks, Oslo, Norway. cognitive scientists collaborate by creating a board with a
Copyright for the individual papers remains with the authors. Copying permitted protocol to oversee tests on the replicability of priming effects
for private and academic purposes. This volume is published and copyrighted by [7]. Bavel et al., (2016), has created a set of guidelines for
its editors. Published on CEUR-WS, Volume 1798, http://ceur-ws.org/Vol-1798/
scholars to provide cognitive insights, while still avoiding
overtly deterministic claims [8].
CHIIR 2017 Workshop on Supporting Complex Search Tasks, March 11, 2017, Oslo, Norway Novin
While the Reproducibility Project findings set off a debate on the in one of three ways: 1) They focus on the process and implicitly
value of cognitive studies, there are two contextual points that minimize the impact of context, 2) they minimize the impact of
need to be considered. The first is that the OCS also found that context by explicitly controlling for situational factors, or 3) they
cognitive studies did predict the effect sizes. This point is acknowledge the context, but only in a positivist framework
elaborated on by Bavel et al., (2016) and Gilbert et al., (2016), so whereby all situational factors are required to be "optimal" for a
the author will not be expanding on it in this submission. process to take place. Modelling for both negative and positive
Instead, the author will focus on the more salient second point: outcomes accounts for the dynamics of social cognitive theories
the possibility that robust contextual factors explained the low (i.e., the knowledge transfer between individuals and their
reproducibility. These two points were examined further by a surrounding society) [16]. For example, people do not only work
large-scale analysis conducted by the Many Labs, whereby 13 towards positive outcomes but they also try to avoid negative
studies on cognitive biases were cherry-picked (partly due to outcomes [16]. In the information sciences, scholars have argued
their high-likelihood for replicability) and tested by 36 academic that if a problem is information intensive but requires little
institutions [9]. The study on 6,344 participants found that only external input then it may be successful [17], [18], but this
10 of those 13 studies were replicable. This was presented as requires the task or problem to be designed so that external
troubling news because the 13 studies were picked due to their input is minimized. It also disregards how context can impact
high chance of being reproduced [10]. Furthermore, the Many how we internalize information from our surroundings, how it
Labs scholars found that context did not play an important role affects our motivations, and how we externalize that information
in the observed results [9]. However, in response to this finding, via communication and action. Outside of the lab, context will
one significant irony in the Many Labs theoretical position was impact the rationality behind how we seek information. T.D
raised by Bavel, Mende-Siedlecki, Brady, and Reinero (2016): "if Wilson proposed the drivers of user behaviours are found in the
the effects chosen for replication in these projects were situation [19] and information science scholars should be
predominantly effects which are a priori unlikely to vary by researching "the PERSON performing a ROLE in an
context, then it would come as no surprise that context does not ENVIRONMENT" [20].
predict replication success." To illustrate this point, Bavel et al.,
(2016) conducted an analysis of 100 replication attempts in Distributed Cognition & Search Tasks
psychology, and found "that the extent to which the research Distributed cognition is an expansion on T.D. Wilson's proposal
topic was likely to be contextually sensitive (varying in time, because it argues cognition flows in a cyclic manner between the
culture, or location) was associated with replication success." person's internalization of the environment and the
This finding also supports the claim by the information scientists externalization of their activities [21]. The strong influence of
Gilbert et al., (2016) that "Robustness to changes in condition is contextual variables has been recognized by research in various
important, but it is not relevant to the replicability of psychological fields, such as health professionals calling for a restructuring of
science which is what [OCS] tried to estimate." medical care to consider patient experience [22]–[24], education
experts calling for more focus on distributed cognition [25]–[27]
Replicability of Complex Search Tasks and businesses over the last decade who recognize that corporate
It is important for the sciences to be founded on the concrete culture is a greater determinant of success than strategy [28],
principles of reproducibility and falsifiability. Because [29]. Our behavior, rationale, and sense of self are all dependent
information seeking refers heavily to the cognitive sciences, it is on social components that bring value and a purpose to our
also important for the field to recognize the challenges it faces. everyday situations. Socio-cultural approaches "focus on
Furthermore, it is worth questioning whether the reproducibility processes of interaction of individuals with other people and
of complex search and retrieval tasks may also be too dependent with physical and technological systems" [30]. Within this
on context and cognition to make mechanistic claims that framework, distributed cognition scholars argue that a human's
"control" for contextual factors. With complex searches, the cognition is distributed amongst other members in society and
search process is often dependent on the work task that is their non-human environment (e.g., tools, artefacts, and other
assigned. However, the participant's cognitive understanding of objects) [21]. Therefore, these social components should be
the work task also requires accounting for contextual factors. included in a researcher's analysis of human engagements, such
There are a variety of definitions for both context and situation as information seeking.
and, while scholars often use them interchangeably, they should Our cognition does not reside solely within the confine of a
be distinguished [11]. Although "context" is a difficult concept to brain, but amongst people in our surroundings and artefacts in
define, Brenda Dervin [12] writes that there is general agreement our environment. As cognition becomes further distributed, the
that it accounts for "the here and the now (i.e., time and space)" context of our environment becomes increasingly more
(p.114). Sonnenwald distinguishes context from a situation by important. Understanding how these environmental forces might
arguing that a context may contain a set of several different interact requires theorizing what accounts for abstract concepts
situations [13], p.180). The situation is a moment in "time-space" such as context and cognition. If this is the case with complex
that works as a frame that guides the situational factors within it search tasks, then it should avoid distilling human information
[14]. The situation can consist of several situational factors, seeking into simplistic X causes Y formulas. Even when facile
which Hert (1997, p.21) defines as "variables which were time- formulas are reproducible they can lead to poor inferences if the
space-specific". Finally, the author argues that the context, contextual factors influencing someone's cognition are not
situation, and its factors are separate concepts from the process. accounted for. For example, in the current North-American
The process makes up the framed interactions pieced together, political climate it is often mistakenly thought that people on the
which create several moments in time. far left-wing or far right-wing of the political spectrum do not
Giddens (1991) argued for a dual focus on the structures of the integrate with the rest of a community because they are not
situation and the processes by human agents [15]. However, aware of the facts on the other side [31]. At worst, these
information seeking models that account for context often do so individuals are accused of being "irrational" or having "poor
CHIIR 2017 Workshop on Supporting Complex Search Tasks, March 11, 2017, Oslo, Norway
information literacy skills." However, past and current research explain to the observer the rational behind their choices. This
has demonstrated that those members on either side of the may seem clear, but rarely has the author observed scholars
spectrum are not only fully aware of the information on the explain the reasons behind the chosen sequence for data
other side, but are often more informed than the average person collection and analysis and how the qualitative and quantitative
[32], [33]. Individuals on the fringe of the political spectrum data is "mixed" (i.e., Creswell's explanatory or exploratory).
contextualize new pieces information retrieved to fit into the Third, mixing the methodology of surveys and experiments can
ideological frameworks of their smaller social-contexts [32]. In increase the likelihood of capturing contextual factors. Research
other words, the individual's perception of the relevancy of in information search and retrieval requires accounting for the
information is dependent on the individual's relationship with interactions users have with systems [37], the usefulness of the
smaller social groups (i.e., where a member has strong-ties) more information [38], its influences [36], and the outcome of the
so than their greater community (i.e., where a member has weak- retrieval [39]. Pia Borlund argues researching these factors
ties). Thus, while a scientific observer might be technically requires tailoring the work task to the information environment
correct that the far left or far right seeks information that is and participant: “if the evaluation takes place by involvement of
regarded as "false" by the greater community of which both the university students then the simulated work task situation
observed individual and observer are members, in certain cases it should be to describe a situation they can relate to, and report on
may be more accurate to say that the individual is seeking how the situation was simulated” [39]. To create such a
information that fits within the distributed cognition of smaller situation, the task should be piloted and the final report should
groups within the community first. However, the scientific explain how the situation was simulated [39]. In addition, studies
observer is often only aware of contextual factors of a person's can simulate the context in information search and retrieval
greater community and less aware about their smaller, more further by conducting a population-based survey experiment: "a
private, communities (e.g., marginalized far-right communities). population-based experiment uses survey sampling methods to
Individuals often do not disclose their associations with groups produce a collection of experimental subjects that is
that are perceived as less rational in a community. For example, representative of the target population of interest for a particular
researchers found that on online dating websites "both men and theory, whether that population is a country, a state, an ethnic
women of all political persuasions act as if they prefer same-race group, or some other subgroup. The population represented by
relationships even when they claim not to" [34]. In this scenario, the sample should be representative of the population to which
it is clear that online daters are hiding their less socially- the researcher intends to extend his or her findings" [40]. This
acceptable preferences for people from the same-race. However, method requires participants to be randomly assigned to a
if the scientist observes a person seeking information that is not situation and can take place outside of a lab and within the
relevant to the greater community but unaware of how the population itself, like field studies [40]. Diana C. Mutz argues the
information is relevant to a smaller group, they will categorize main advantage of choosing this method is that "theories can be
this behavior as poor "information literacy." Maintaining strong tested on samples that are representative of the populations to
ties with your social group could be considered a cognitively which they are said to apply" [40]. By doing so, both the internal
"rational" decision – even if it seems irrational to the larger and external validity of an experiment can increase [40]. For
community. example, in Borlund's example on university students, this can
be accomplished by choosing the common area of a university
How to Test for Contextual Influences on library to study and recruiting students within their own familiar
Rational and Cognition environment for an information seeking experiment. The ability
First, the approach researchers use to choose which contextual of population-based survey experiments to be carried outside of
and cognitive variables to control for is a complex search task for the lab and directly in the field means that experiments can
the researchers themselves. My argument for the need to first capture more contextual factors "in the wild."
understand the context of the work task prior to designing the
task still applies. Before choosing the variables to research, Conclusion
researchers should ask what is the context of the research's In summary, to account for both the cognitive and contextual
purpose for the field? For example, is the research to strengthen factors, the author proposes a more distributed-cognition
reproducibility of the field or is it to seek out new theories? As I approach. In light of the "cognitive crisis", instead of setting out
will explain in my second point: how much a science study to reproduce cognitive effects on an interface, the author
should control context to increase reproducibility depends on the suggests addressing this issue from another angle: the context of
context of the science field. the work task should first be determined and then the work task.
Second, John W. Creswell has written a fairly clear explanation After these two realms are considered, the mechanistic variables
about how conducting a qualitative analysis before a quantitative (e.g., such as interface features) for participants can be
analysis can lead to different inferences from a study that experimented with. Cognition is dependent on context – an area
performs mixes the analysis in the opposite sequence [35]. To scholars in distributed cognition explores more fully. Critics in
briefly summarize Creswell, the main question a researcher favor of reproducibility projects may argue that this approach
should ask is whether the researcher intends to conduct an abstracts the concept of "cognition," but that criticism begs the
exploratory or explanatory investigation. to measure the question of whether a concrete framework of cognition-without-
usefulness of information a quantitative and qualitative approach context even exists.
should be applied [36]. A mixed methods design may be best
suited to draw inferences to explain the observed data. In
References
[1] L. Chi Vo, E. Mounoud, and J. Rose, “Dealing with the opposition
addition, mixed methods can increase the responsiveness of of rigor and relevance from Dewey’s pragmatist perspective,”
participants in an otherwise complex research activity that is M@n@gement, vol. 15, no. 4, p. 368, 2012.
asking for a significant amount of effort [35]. More importantly, [2] E. Yong, “Psychologists strike a blow for reproducibility,” Nat.
mixed methods provides an opportunity for participants to News, Nov. 2013.
CHIIR 2017 Workshop on Supporting Complex Search Tasks, March 11, 2017, Oslo, Norway Novin
[3] Open Science Collaboration, “Estimating the reproducibility of [23] G. Gardner, A. Gardner, and J. O’Connell, “Using the Donabedian
psychological science,” Science, vol. 349, no. 6251, pp. aac4716- framework to examine the quality and safety of nursing service
aac4716, Aug. 2015. innovation,” J. Clin. Nurs., vol. 23, no. 1–2, pp. 145–155, Jan. 2014.
[4] D. T. Gilbert, G. King, S. Pettigrew, and T. D. Wilson, “More on [24] S. W. Glickman, K. A. Baggett, C. G. Krubert, E. D. Peterson, and
‘Estimating the Reproducibility of Psychological Science,’”, 2016. K. A. Schulman, “Promoting quality: the health-care organization
[5] T. D. Sterling, “Publication Decisions and Their Possible Effects from a management perspective,” Int. J. Qual. Health Care, vol.
on Inferences Drawn from Tests of Significance--Or Vice Versa,” 19, no. 6, pp. 341–348, Sep. 2007.
J. Am. Stat. Assoc., vol. 54, no. 285, p. 30, Mar. 1959. [25] Brown, L. Ann et al., “Distributed Expertise in the Classroom,” in
[6] D. Fanelli, “‘Positive’ Results Increase Down the Hierarchy of the Distributed cognitions: psychological and educational
Sciences,” PLoS ONE, vol. 5, no. 4, p. e10068, Apr. 2010. considerations, G. Salomon, Ed. Cambridge [England]; New York,
[7] E. Yong, “Nobel laureate challenges psychologists to clean up NY: Cambridge University Press, 1993, pp. 188–228.
their act,” Nature, Oct. 2012. [26] E. M. Meyers, “Mediating group search: Lessons from a middle
[8] J. J. Van Bavel, P. Mende-Siedlecki, W. J. Brady, and D. A. school study,” Teach. Libr., vol. 38, no. 2, p. 24, 2010.
Reinero, “Contextual sensitivity in scientific reproducibility,” [27] E. M. Meyers, K. E. Fisher, and E. Marcoux, “Making Sense of an
Proc. Natl. Acad. Sci., vol. 113, no. 23, pp. 6454–6459, Jun. 2016. Information World: The Everyday‐Life Information Behavior of
[9] R. A. Klein et al., “Investigating variation in replicability,” Soc. Preteens,” The Library, vol. 79, no. 3, 2009.
Psychol., 2014. [28] J. H. Dyer and K. Nobeoka, “Creating and managing a high-
[10] E. Yong, “Psychologists strike a blow for reproducibility,” Nature, performance knowledge-sharing network: the Toyota case,”
Nov. 2013. Strateg. Manag. J., vol. 21, no. 3, pp. 345–367, Mar. 2000.
[11] C. Cool, “The concept of situation in information science,” Annu. [29] T. Kell and G. Carrott, “Culture matters most,” 2005.
Rev. Inf. Sci. Technol., vol. 35, pp. 5–42, 2001. [30] A. Collins, J. Greeno, L. Resnick, B. Berliner, and R. Calfee,
[12] B. Dervin, “Given a context by any other name: Methodological “Cognition and learning,” B Berl. R Calfee Handb. Educ. Psychol.
tools for taming the unruly beast,” Inf. Seek. Context, vol. 13, p. N. Y. Simon Shuster MacMillan, 1992.
38, 1997. [31] H. Landemore, “On Minimal Deliberation, Partisan Activism, and
[13] D. H. Sonnenwald, “Evolving perspectives of human information Teaching People How to Disagree,” Crit. Rev., vol. 25, no. 2, pp.
behavior: Contexts, situations, social networks and information 210–225, Jun. 2013.
horizons,” presented at the Exploring the contexts of information [32] D. M. Kahan, A. R. Landrum, K. Carpenter, L. Helft, and K. H.
behavior: Proceedings of the Second International Conference in Jamieson, “Science Curiosity and Political Information
Information Needs, 1999. Processing,” Adv. Polit. Psychol. Forthcom., 2016.
[14] C. A. Hert, Understanding Information Retrieval Interactions: [33] D. C. Mutz, “Reflections on Hearing the Other Side , in Theory and
Theoretical and Practical Implications. Greenwood Publishing in Practice,” Crit. Rev., vol. 25, no. 2, pp. 260–276, Jun. 2013.
Group, 1997. [34] A. Anderson, S. Goel, G. Huber, N. Malhotra, and D. Watts,
[15] A. Giddens, Modernity and self-identity: Self and society in the late “Political Ideology and Racial Preferences in Online Dating,”
modern age. Stanford University Press, 1991. Sociol. Sci., pp. 28–40, 2014.
[16] C. Abraham and S. Michie, “A taxonomy of behavior change [35] K. B. Rasmussen, “General Approaches to Data Quality and
techniques used in interventions.,” Health Psychol., vol. 27, no. 3, Internet-generated Data,” in The Sage Handbook of online
p. 379, 2008. Research Methods, Fielding, Lee, and Blank, Eds. Sage, 2008.
[17] E. Kuiper, M. Volman, J. Coiro, M. Knobel, C. Lankshear, and D. [36] D. Case, “Sixty years of measuring the use of information and its
Leu, “The Web as a source of information for students in K–12 sources: from consultation to application,” Libr. Digit. Age LIDA
education,” Handb. Res. New Literacies, vol. 5, pp. 241–266, 2008. Proc., vol. 13, 2014.
[18] A. W. Lazonder, “Do two heads search better than one? Effects of [37] D. Kelly, “Methods for evaluating interactive information
student collaboration on web search behaviour and search retrieval systems with users,” Found. Trends Inf. Retr., vol. 3, no.
outcomes,” Br. J. Educ. Technol., vol. 36, no. 3, pp. 465–475, 2005. 1—2, pp. 1–224, 2009.
[19] T. D. Wilson, “Activity theory and information seeking,” Annu. [38] N. J. Belkin, M. Cole, and J. Liu, “A model for evaluation of
Rev. Inf. Sci. Technol., vol. 42, no. 1, pp. 119–161, 2008. interactive information retrieval,” presented at the Proceedings of
[20] T. D. Wilson, “Information behaviour: An interdisciplinary the SIGIR 2009 Workshop on the Future of IR Evaluation, 2009,
perspective,” Inf. Process. Manag., vol. 33, no. 4, pp. 551–572, Jul. pp. 7–8.
1997. [39] P. Borlund, “Interactive Information Retrieval: An Introduction,”
[21] G. Salomon, Ed., Distributed cognitions: psychological and J. Inf. Sci. Theory Pract., vol. 1, no. 3, pp. 12–32, Sep. 2013.
educational considerations, 1st pbk. ed. Cambridge [40] D. C. Mutz, Population-based survey experiments. Princeton
[Cambridgeshire] ; New York: Cambridge University Press, 1997. University Press, 2011.
[22] A. Donabedian, “Evaluating the quality of medical care,” Milbank
Mem. Fund Q., pp. 166–206, 1966.