=Paper=
{{Paper
|id=Vol-2558/short9
|storemode=property
|title=A Metacognitive Triggering Mechanism for Anticipatory Thinking
|pdfUrl=https://ceur-ws.org/Vol-2558/short9.pdf
|volume=Vol-2558
|authors=Alexander Hough,Othalia Larue,Ion Juvina
|dblpUrl=https://dblp.org/rec/conf/aaaifs/HoughLJ19
}}
==A Metacognitive Triggering Mechanism for Anticipatory Thinking==
https://ceur-ws.org/Vol-2558/short9.pdf
A Metacognitive Triggering Mechanism for Anticipatory Thinking
Alexander R. Hough1,2, Othalia Larue1, and Ion Juvina1
1Wright State University, Psychology Department, ASTECCA laboratory, Dayton, OH 45435, USA
2ORISE at Air Force Research Laboratory, Wright-Patterson Air Force Base, OH 45433, USA
hough.15@wright.edu, othalia.larue@wright.edu, and ion.juvina@wright.edu
Abstract trends (trajectory tracking), and being mindful of connec-
Current autonomous systems have the ability to adapt to envi- tions, implications, and interdependencies betw een events
ronmental changes in real-time, but limited ability to engage in (Conditional). Geden et al. (2019) and Klein et al. (2011)
anticipatory thinking (AT) with the flexibility to generalize and have slightly different approaches, however, they both iden-
consider hypothetical future situations. We argue that metacog- tified similar AT processes: recognizing feature and cue re-
nitive processes are important for an and provide supporting lit-
lationships between situations, extrapolation or generaliza-
erature primarily from psychology. As an example, we present
a metacognitive monitoring mechanism implemented in a cog- tion to other states, and construction of mental models based
nitive model and discuss ways to extend the mechanism to allow on available evidence.
for dynamic behavior and anticipatory thinking capabilities. Anticipating future events is crucial. The real world is dy-
namic, often unpredictable, may have ill-defined goals, and
may involve high stakes. The ability to generate, use, and
Anticipatory Thinking1 reason about plans or goals to direct behavior and adapt to
changes is important for intelligent behavior (Newell and
Anticipatory thinking (AT) is the deliberate exploration and Simon 1972; Schank and Abelson 1977) and autonomy
consideration of hypothetical future outcomes in order to (Johnson et al. 2016; Vattam et al. 2013). Goal-driven be-
identify an appropriate action or plan (Amos-Binks and havior leverages discrepancies between expectations and the
Dannenhauer 2019; Geden et al. 2018). AT involves an ar- environment in real-time, and when detected, they are ad-
ray of cognitive processes (Klein et al., 2003; Koziol, Bud- dressed by modifying goals, reasoning about goals, and
ding, and Chidekel 2012), such as mental simulation, recog- learning (Aha 2018; Cox et al. 2016; Mun͂oz, et al. 2019;
nition, preparation, and development of expectancies, which Pozanco, Fernández, and Borrajo 2018; Roberts et al. 2018).
are not completely understood prior to an event occurring Amos-Binks and Dannenhauer (2019) suggest most current
(Klein, Snowden, and Pin 2011; Warwick and Hutton 2007). systems lack AT capabilities, such as the ability to address
It is considered distinct from prediction (Klein et al. 2011) unknown hypothetical future events by identifying and
and is described as gambling with attention in hopes of di- avoiding errors or discrepancies before they occur, and how
recting it towards the most relevant event (Klein et al. 2007). to effectively trade off costs of computation and benefits of
Geden et al. (2019) identified three forms of AT: how past considering a large number of possible futures. AT systems
states led to current states (retrospective branching), antici- need to strike a balance between flexibility and stability in
pating future states and their indicators (prospective branch- order to adapt to dynamic real-world environments before
ing), and focusing on a potential future and working back- conditions change (Bratman, Israel, and Pollack 1988).
wards (backcasting). Klein et al. (2011) takes a more natu- Klein et al. (2011) suggest good AT requires one to be
ralistic approach to AT emphasizing the detection of dis- sensitive to the constraints and affordances based on their
crepancies through recognition and degree of match be- own beliefs, capabilities, and the current situation. Metacog-
tween past, current, and future situations (pattern matching), nitive monitoring could help overcome the computation
using “trajectory” to prepare for the future and extrapolate problem and some of the barriers to AT identified by Klein
Copyright © 2020 for this paper by its authors. Use permitted under Crea-
tive Commons License Attribution 4.0 International (CC BY 4.0).
et al. (2011), such as taking a passive stance, becoming fix- CRT were able to generate heuristic and deliberate answers,
ated on patterns, explaining away evidence or interpreta- more accurately rate performance of others and themselves,
tions, and being overconfident. Similar to AT as a metacog- and were able to better focus on the most relevant features
nitive capability (Amos-Binks and Dannenhauer 2019), we of a problem. Epstein et al. (1996) found that the ability to
emphasize how the calibration between metacognitive mon- shift between heuristic and deliberate thinking was better
itoring and reality could help indicate when AT is needed, than exclusively relying on one. Metacognitive monitoring
how to accomplish it efficiently, and reduce the number of as measured by the CRT, FOR, and related tasks may be
futures to consider. We explore psychological metacogni- more related to actual intelligence than traditional measures,
tive measurements regarding conflict detection and resolv- because it includes motivation and ability (Frederick 2005;
ing processes in simpler tasks and discuss how these capa- Toplak, West, and Stanovich 2011). For instance, Barr et al.
bilities could be extended to AT. (2015) found the CRT positively correlates with cognitive
ability, need for cognition, analogies, the remote associates
test, and negatively with faith in intuition. Furthermore, the
A Critical Role for Metacognition CRT correlates with cognitive ability, performance on heu-
ristic and biases tasks, belief bias, rational thinking, set shift-
Humans use heuristics to make efficient and accurate deci-
ing, and working memory, and predicts rational thinking
sions (Cosmidies and Tooby 1996; Gigerenzer and Gaiss-
performance independent of intelligence, executive func-
maier 2011), but this can lead to systematic error in inappro-
tioning, and thinking dispositions (Toplak et al. 2012).
priate, novel, or misleading environments (Evans and Sta-
Conflict associated with metacognitive experience has
novich 2013; Kahneman 2011; Kahneman and Klein 2009).
been measured using response times (De Neys and Glumicic
A critical ability is recognizing when an approach is inade-
2008; Pennycook et al. 2015a), the FOR (Thompson et al.
quate and suppressing it to come up with an alternative (Sta-
2011), the CRT (Frederick 2005), and activation of specific
novich 2018). This ability is metacognition, which serves to
brain regions including the anterior cingulate cortex (Crox-
detect conflict or mismatch regarding an environment and a
son et al. 2009; Kennerley et al. 2009) and medial prefrontal
strategy, type of processing, or expectation. This detection
cortex (Botvinick et al. 1999; Cohen, Botvinick, and Carter
depends on predictability and cues in the environment, abil-
2000) often associated with cognitive control. This conflict
ity to recognize relevant cues, and whether goals are reach-
is still observed when manipulations are in place to mini-
able (Dannenhauer et al. 2018; Evans and Stanovich 2013;
mize deliberation (Pennycook et al., 2015a; Thompson and
Johnson et al. 2016; Klein 1998; Klein et al. 2007; Penny-
Johnson, 2014), and error signals during comprehension
cook, Fugelsang, and Koehler 2015a; Stanovich 2018;
(Glenberg, Wilkinson, and Epstein 1982; McNamara et al.,
Vattam, et al. 2013). This process is referred to as metacog-
1996) and disfluency appear to prompt similar responses
nitive monitoring or experience, which provides feedback,
(Alter et al. 2007). Metacognitive monitoring appears to in-
leads to control decisions, activates knowledge, and can be
volve both top-down and bottom-up processes, where the
calibrated through experience leading to better regulation
willingness or motivation to engage in analytic thinking
behavior (Efklides 2006; Efklides, Samara, and Petropoulou
(e.g., CRT) appears to be top-down, while the detection of
1999; Flavell 1979). Conflict often triggers the need for a
conflicts that triggers the engagement (e.g., FOR) appears to
different approach towards solving a problem or completing
be bottom-up (Pennycook et al. 2015b; Stanovich 2018).
a task (Butcher and Sumner 2011; Dannenahuer et al. 2018;
Here, we focus primarily on bottom-up processes, but plan
Pennycook 2017; Stanovich 2018). However, it does not al-
on further addressing top-down processes in future work.
ways lead to efficient processing (Pennycook et al. 2015a,
2015b; Swan, Calvillo, and Revlin 2018) or solutions to re-
solve the conflict (Novick and Holyoak 1991). Dynamic Behavior
Metacognitive monitoring is effective but not perfect. It may
Monitoring and Conflict fail to detect conflict (Swan et al. 2018) or may result in
faulty judgments after conflict is detected. Detection may
There are several methods for measuring metacognitive
not direct one to the necessary knowledge to solve the prob-
monitoring (e.g., Gascoine, Higgins, and Wall 2017). Two
lem or implement a strategy (Novick and Holyoak 1991) and
common methods in psychology are the performance-based
the outcome might be influenced by biases, such as overcon-
Cognitive Reflection Test (CRT; Frederick 2005) that re-
fidence with naive individuals (Fischhoff 2012) or confir-
quires overriding a primed heuristic response for a more de-
mation bias with the more experienced (Kahneman 2011).
liberate response, and the subjective-based Feeling of Right-
Klein et al. (2006b) acknowledge that detecting such a con-
ness (FOR; Thompson et al. 2011) that indicates one’s ac-
flict or recognizing insufficient performance is important,
curacy and awareness of their own metacognitive monitor-
but understanding how to modify thinking processes to ad-
ing. Mata, Ferreira, and Sherman (2012) found that those
dress this problem is more valuable. Metacognition could
with better metacognitive awareness as measured by the
help guide conflict resolution by helping to determine sensemaking loop (Russell et al. 1993). During foraging, an
whether the environment calls for more deliberate pro- individual engages in search and filtering of information and
cessing or quick, less elaborate responding. Although hu- then applies effort to give it more structure in an iterative
mans are often good at sizing up the environment (Klein process. During sensemaking, one utilizes schemas to make
1998) and making efficient tradeoffs between speed and ac- hypotheses and conclusions, similar to the construction of a
curacy (Payne, Bettman, and Johnson 1988), exerting men- mental model (e.g., Johnson-Laird 2013). Although not ex-
tal effort is often experienced as aversive (Halpern 2014; plicit in the model, there appears to be a metacognitive pro-
Kahneman 2011) and may be avoided based on an individ- cess. If there is insufficient evidence for a hypothesis, a case
ual’s subjective cost of effort (Westbrook, Kester, and cannot be built, or a discrepancy is detected then the agent
Braver 2013). Similarly, in AT the environment may favor goes back to the foraging loop to fill in the gaps or gather
considering more hypothetical future situations, exploring evidence for a new schema or hypothesis. Similarly, the
some more deeply, or by quickly anticipating and preparing data/frame model (Klein, Moon, and Hoffman 2006a,
for a few. Although typically applied to current events, met- 2006b; Klein et al. 2007) does not explicitly mention meta-
acognitive monitoring could be extended to future events to cognition, but does involve “questioning the frame” that in-
help determine which approach fits better with the environ- cludes anomaly detection or expectancy violations. If there
ment. For instance, if an individual engages in the three is a discrepancy, the existing frame can be discarded, elab-
types of AT (i.e., pattern matching, trajectory tracking, and orated, preserved, reframed, or compared to another.
conditionals) identified by Klein et al. (2011) regarding a Critical thinking is described as the ability to explain, jus-
hypothetical future, the presence of conflict among them tify, extrapolate, relate, and apply in ways that go beyond
could inform whether that hypothetical future is appropriate knowledge and skill, and training in critical thinking and
or if it should be discarded or modified. Metacognitive cali- metacognitive monitoring can enhance understanding and
bration could help determine the appropriate response based generalizability (Halpern 1998, 2014; Willingham 2007).
on one’s understanding of their own abilities and Similar to metacognitive monitoring, after controlling for
knowledge, and how that corresponds to a situation. Re- cognitive ability, critical thinking correlates with the ability
search addressing the understanding process, sensemaking, to avoid cognitive biases by thinking logically even when it
critical thinking, forecasting, and counterfactual thinking conflicts with prior beliefs and thinking dispositions (West,
provide examples of how to identify the source of conflict, Toplak, and Stanovich 2008). To better understand and
how to make sense of and resolve it, and determine which teach these skills, Halpern (2010) developed a comprehen-
potential outcomes are most likely. sive measure, called the Halpern Critical Thinking Assess-
The understanding process was recently defined in a mul- ment (HCTA), which includes decision making, problem
tidisciplinary review as “The acquisition, organization, and solving, hypothesis testing, argument analysis, likelihoods
appropriate use of knowledge to produce a response directed and uncertainties, and verbal reasoning. The HCTA has gen-
towards a goal, when that action is taken with awareness of eralizability across various populations, positively corre-
its perceived purpose” (Hough and Gluck forthcoming, p. lates with years of education, and negatively correlates with
11). The review revealed common features of understanding the frequency of negative life events in a real-world outcome
and discussed how computer science, education, psychol- inventory (Butler 2012). Critical thinking has some similar-
ogy, and philosophy all emphasize the importance of meta- ities to sensemaking, but may be more generalizable and ap-
cognition for understanding capabilities. Metacognition was propriate for AT with little available knowledge.
described as a self-evaluative feedback mechanism for iden- Forecasting involves predicting the probability that spe-
tifying faulty knowledge or gaps that triggers additional pro- cific events will occur. Its associated processes could be
cessing or information search, which helps calibrate mental used during AT to help reduce the unnecessary considera-
representations with the environment (Butcher and Sumner tion of unlikely future outcomes or to determine which are
2011; Forbus and Hinrichs 2006; Kirk and Laird 2014; more likely and should be better prepared for. Accurate fore-
Mayer 1998; Perkins 1998; Perkins and Simmons 1988; casters typically have higher cognitive ability, motivation,
Woodward 2003). Better understanding, like expertise, CRT scores, and open-minded thinking (Juvina et al. under
could increase the quality of AT by directing attention to- review; Mellers et al. 2015; Tetlock and Gardner 2015). In
ward the most relevant features. addition, they often respond faster, have better discrimina-
Sensemaking models also involve components of under- bility and calibration, and learn faster.
standing, such as abstraction of knowledge, development of Counterfactual thinking occurs after an event is experi-
relations, ability to transfer knowledge to distant situations, enced and involves considering forgone outcomes (Byrne
and often involves leveraging domain and context infor- 2016; Kahneman and Miller 1986). It is more typical after
mation to develop frames (Hough and Gluck forthcoming). failures or shortcomings (Hur 2001; Roese and Olsen 1997;
Pirolli and Card’s (2005) sensemaking model includes an Sanna and Turley 1996; Sanna and Turley-Ames 2000) and
information foraging loop (Pirolli and Card 1999) and
often involves ways to correct or improve upon previous be- not Q then not P) in a “partial insight” (Evans 1977; Wason
haviors (Markman et al. 1993; Roese 1997; Roese, Hur, and 1969). Only flipping the correct intuitively compelling card
Pennington 1999). Epstude and Roese (2008) suggest that (A) occurs because modus tollens needs to simulate more
this may depend on the realization that there is a problem or intermediary mental models compared to the modus ponens.
goals are not sufficiently met, which is a form of metacog- Our model was implemented in the ACT-R cognitive ar-
nitive monitoring. Improving future outcomes may be chitecture (Anderson, 2007) with a core affect mechanism
achieved through goal-oriented reasoning (Epstude and (see Juvina, et al. 2018) and a FOR component to drive de-
Roese 2008; Roese and Epstude 2017) or by increases in coupling behavior. Rethinking times, answer changes, and
motivation, persistence, and performance (Dyczewski and fluency are functions of the FOR. The FOR is computed
Markman 2012; Markman, McMullen, and Elizaga 2008). based on the time required to achieve the initial retrieval of
Although after the fact, this type of thinking could provide the answer for the two intuitively compelling cards through
experience to help calibrate metacognitive processes, pro- the initial priming rule (autonomous mind), and serves as a
vide more constructive ways to think about future events, gateway for further processing. We use the temporal module
and help identify relevant alternative possibilities. in ACT-R (Taatgen, van Rijn, and Anderson 2007) to meas-
ure time in ticks, which are noisy and increase in time in a
A Cognitive Model with Metacognitive Monitoring fashion similar to human time estimation. In the Wason task,
We previously developed a model of the Wason card selec- the FOR is computed as a function of the time required to
tion task (Wason 1966, 1968) with initial aspects of meta- achieve the initial retrieval of “A” and “3” through the initial
cognitive monitoring (see Larue, Hough, and Juvina 2018 priming rule. When the FOR is high, the model goes with
for a full description). Our approach was informed by men- the initial answer (i.e., heuristic processing), but when low
tal models (Johnson-Laird, 2013) and dual process theories, cognitive decoupling is launched by the reflective mind and
specifically Stanovich’s (2009) tripartite framework. Sta- carried out by the algorithmic mind. In the model, the time
novich’s (2009) framework provides an explanation of how required to achieve the initial retrievals is assigned to FOR-
reflective and adaptive (characterized by reactivity) human inverse (e.g., higher time means lower FOR). When FOR-
behavior emerges from the interaction of three distinct cog- inverse is below threshold (see Figure 1), the model goes
nitive levels or “minds”. The autonomous mind, responsible with the initial answer (i.e., type 1 processing). When FOR-
for fast behaviors, includes instinctive and over-learned pro- inverse is above threshold, cognitive decoupling is launched
cesses, domain-specific knowledge, and emotional regula- and the model engages in further processing (i.e., type 2 pro-
tion. The algorithmic mind, responsible for cognitive con- cessing) with representations that are copied from its work-
trol, can affect decoupling (i.e., simulation) and serial asso- ing memory based on activation during open retrieval (those
ciative processes. The reflective mind, responsible for delib- with highest activation are retrieved). The representations
erative processing, can trigger or suppress the algorithmic are used in an inner cognitive simulation to indicate which
minds’ decoupling and serial associative processes. In this rules from the reflective mind can be applied. The process
framework, the reflective mind would be the center for met- by which representations are copied and used in a separate
acognitive monitoring. inner simulation is “cognitive decoupling”. The importance
Here we briefly describe our model and in the next para- of further processing is a function of the FOR, which deter-
graph, discuss how this model could be augmented and ap- mines the extent a decoupling result (i.e., wrong, partial, and
plied to AT. In the Wason card selection task, two cards (A complete) is taken into account in the final answer. The
and 7) out of four (A, D, 3, and 7) must be flipped over to model produces an answer when the valuation of a represen-
verify a rule: If “A” is on one side, then there is a “3” on the tation is above a certain threshold. The valuation and arousal
other. “A” and “3” are intuitively compelling to flip over values, which are sub-symbolic quantities added to the cur-
because they are both present in the rule. Flipping over “A” rent sub-symbolic equations of ACT-R, help to define the
is necessary because it can falsify the rule, however, flipping core affect. When a reward is triggered, valuations are up-
over “3” is unnecessary because it may confirm the rule but dated. Rewards are a function of the initial FOR (negative
it cannot falsify it. Two types of logical errors are common: factor in the case of negative reward), which affects answer
the selection of the unnecessary card (3), and the non-selec- selection (“yes” or “no” answers are produced according to
tion of the necessary card (7). We believe selecting the un- how the model “feels” about the answer) (see Figure 1).
necessary card results from a metacognitive failure to detect A training procedure was used to simulate individual dif-
its inadequacy and not selecting the necessary card, involves ferences in heuristic and analytical behavior, where the dif-
incomplete decoupling after the detection and override of ferent degrees of reinforcement allowed the model to learn
selecting the unnecessary card. The incomplete decoupling logical skills and vary in FOR. Metacognitive monitoring
may result from participants applying modus ponens (if P implemented by the FOR determined if and how much ad-
then Q), but failing in the application of modus tollens (if ditional processing occurred. Model simulations produced
to hypothetical future states through mental simulation. Fur-
thermore, when hypothetical futures are considered they
could be weighted based on predicting the probability of
their occurrence through forecasting. This process in com-
bination with the FOR could also help inform when enough
hypothetical futures are considered. These types of pro-
cesses could help highlight the most relevant and likely fu-
ture outcomes, so that less preparation and planning is re-
quired. As the architecture learns more and is placed in con-
text, it reacts to events that it might have previously encoun-
tered. If there is a match (i.e., full or partial) with a previ-
ously developed strategy from cognitive decoupling, this
Figure 1. Answer Selection Processes in Our Cognitive Model strategy is recorded and reinforced in the architecture. The
reinforcement of this strategy will lead to its prioritization
three types of outcomes that are typically observed in hu- in procedural memory over some other strategies and its de-
mans: (1) Complete reliance on the autonomous mind (no clarative components will be faster to retrieve in declarative
decoupling) leading to the observed common error (cor- memory. This means that the next time this strategy is used,
rectly selecting “A” and incorrectly selecting “3” guided by the FOR will better calibrated, resulting in more accurate
confirmation bias), (2) partial decoupling (partial insight) and adaptive behavior with the potential for AT capabilities.
leading to correctly selecting “A” and not falling for the con-
firmation bias (already simulated in a possible world), and
(3) complete decoupling allowing for the activation of the Acknowledgements
counter-information rule which is less often activated.
This research was supported in part by Alex Hough’s ap-
pointment to the Student Research Participation Program at
Discussion the U.S. Air Force Research Laboratory, 711 th Human Per-
formance Wing, Cognitive Science, Models, and Agents
We presented some perspectives in metacognitive monitor- Branch, administered by the Oak Ridge Institute for Science
ing research and some dynamic higher-level cognitive pro- and Education. Alex would like to thank the Chief Scien-
cesses. This research informed the development of our tist’s office of the 711th Human Performance Wing for the
model and we believe the interaction between the FOR (met- opportunity to participate in the Repperager internship pro-
acognitive monitoring) and decoupling (mental simulation) gram. We also thank Kevin Gluck and two anonymous re-
in our model could be applied to AT. viewers for constructive feedback on an earlier draft.
In the model discussed here, the FOR determines whether
there is a need for more deliberation or a different approach
to complete the task. The FOR could be extended to simu- References
late future situations through decoupling and include how Alter, A. L.; Oppenheimer, D. M.; Epley, N.; and Eyre, R. N. 2007.
much the agent “knows” about the environment. For in- Overcoming Intuition: Metacognitive Difficulty Activates Ana-
stance, the FOR could determine how long decoupling lytic Reasoning. Journal of Experimental Psychology: Gen-
should continue (e.g., generating counterfactuals) and when eral 136(4): 569-576.
it should stop. Sensemaking and critical thinking emphasize psycnet.apa.org/doiLanding?doi=10.1037%2F0096-
3445.136.4.569
generating hypotheses and gathering evidence to indicate
Anderson, J. R. 2007. How Can the Human Mind Occur in the
the degree each is supported. The FOR could be informative Physical Universe? New York: Oxford University Press.
when there is a lack of knowledge, failure to find matching
Amos-Binks, A.; and Dannenhauer, D. 2019. Anticipatory Think-
strategies or procedures, or a lack of evidence. A partial ing: A Metacognitive Capability. arxiv.org/pdf/1906.12249.pdf
matching procedure could be used based on the degree of Barr, N.; Pennycook, G.; Stolz, J. A.; & Fugelsang, J. A. 2015.
the FOR, a lower FOR could increase the breadth of future Reasoned Connections: A Dual-process Perspective on Creative
situations to consider, and a strategy or approach could be Thought. Thinking & Reasoning 21(1): 61-75.
chosen out of a hierarchy based on the FOR. Learning can doi.org/10.1080/13546783.2014.895915
occur over time and counterfactuals could be generated and Botvinick, M.; Nystrom, L. E.; Fissell, K.; Carter, C. S.; and Co-
utilized for learning what could have happened based on the hen, J. D. 1999. Conflict Monitoring Versus Selection-For-Action
hypothetical actions corresponding to a given situation or in Anterior Cingulate Cortex. Nature 402(6758): 179.
doi.org/1710/13fb3f03a2bb0d707d2c73597f436e280728
the environment. Although these processes typically apply
to situations in real time, they could be extended and applied
Bratman, M. E.; Israel, D. J.; and Pollack, M. E. 1988. Plans and Evans, J. S. B. 1977. Linguistic Factors in Reasoning. Quarterly
Resource-Bounded Practical Reasoning. Computational Intelli- Journal of Experimental Psychology 29(2): 297-306.
gence 4: 349-355. www.tandfonline.com/doi/abs/10.1080/14640747708400605
doi.org/10.1111/j.1467-8640.1988.tb00284.x Evans, J. S. B.; and Stanovich, K. E. 2013. Dual-Process Theories
Butcher, K.; and Sumner, T. 2011. Self-Directed Learning and The of Higher Cognition: Advancing the Debate. Perspectives On Psy-
Sensemaking Paradox. Human-Computer Interaction 26: 123-159. chological Science 8(3): 223-241.
www.tandfonline.com/doi/abs/10.1080/07370024.2011.556552. doi.org/10.1080/14640747708400605
Butler, H. A. 2012. Halpern Critical Thinking Assessment Predicts Fischoff, B. 2012. Judgment and Decision Making. London and
Real-World Outcomes of Critical Thinking. Applied Cognitive New York: Earthscan
Psychology 26: 721-729. doi.org/10.1002/acp.2851 Flavell, J. H. 1979. Metacognition and Cognitive Monitoring: A
Byrne, R. M. 2016. Counterfactual Thought. Annual Review of New Area of Cognitive–Developmental Inquiry. American Psy-
Psychology 67: 135-157. doi.org/10.1146/annurev-psych-122414- chologist 34(10): 906-911.
033249 dx.doi.org/10.1037/0003-066X.34.10.906
Cohen, J. D.; Botvinick, M.; and Carter, C. S. 2000. Anterior Cin- Forbus, K. D., and Hinrichs, T. R. 2006. Companion Cognitive
gulate and Prefrontal Cortex: Who's in Control?. Nature Neurosci- Systems: A Step Toward Human-level AI. AI Magazine 27: 83-95.
ence 3(5): 421-423. doi.org/10.1038/74783 doi.org/10.1609/aimag.v27i2.1882
Cosmides, L.; and Tooby, J. 1996. Are Humans Good Intuitive Frederick, S. 2005. Cognitive Reflection and Decision Mak-
Statisticians After All? Rethinking Some Conclusions from the Lit- ing. Journal of Economic Perspectives 19(4): 25-42. pubs.aea-
erature on Judgment Under Uncertainty. Cognition 58(1): 1-73. web.org/doi/pdfplus/10.1257/089533005775196732
doi.org/10.1.1.131.8290. Gascoine, L.; Higgins, S.; and Wall, K. 2017. The Assessment of
Croxson, P. L.; Walton, M. E.; O'Reilly, J. X.; Behrens, T. E.; and Metacognition in Children Aged 4–16 Years: A Systematic Re-
Rushworth, M. F. 2009. Effort-Based Cost–Benefit Valuation and view. Review of Education 5(1): 3-57. doi.org/10.1002/rev3.3077
the Human Brain. Journal of Neuroscience 29(14): 4531-4541. Geden, M.; Smith, A.; Campbell, J.; Amos-Binks, A.; Mott, B.;
https://doi.org/10.1523/JNEUROSCI.4515-08.2009 Feng, J.; and Lester, J. 2018. Towards Adaptive Support for Antic-
Cox, M. T.; Alavi, Z.; Dannenhauer, D.; Eyorokon, V.; Munoz- ipatory Thinking. In Proceedings of the Technology, Mind, and So-
Avila, H.; and Perlis, D. 2016. MIDCA: A Metacognitive, Inte- ciety. Washington, DC: ACM. doi.org/10.1145/3183654.3183665
grated Dual-Cycle Architecture for Self-Regulated Autonomy. In Geden, M.; Smith, A.; Campbell, J.; Spain, R.; Amos-Binks, A.;
Proceedings of the Thirteenth AAAI Conference on Artificial In- Mott, B.; …Lester, J. 2019. Construction and Validation of an An-
telligence. Phoenix, AZ: AAAI Press. www.aaai.org/ocs/in- ticipatory Thinking Assessment. PsyAriXiv.
dex.php/AAAI/AAAI16/paper/download/12292/12151 doi.org/10.31234/osf.io/9reby
Dannenhauer, D.; Cox, M. T.; and Munoz-Avila, H. 2018. Declar- Gigerenzer, G.; and Gaissmaier, W. 2011. Heuristic Decision Mak-
ative Metacognitive Expectations for High-Level Cognition. Ad- ing. Annual Review of Psychology 62: 451-482.
vances in Cognitive Systems 6: 231-250. www.cogsys.org/pa- doi.org/10.1146/annurev-psych-120709-145346
pers/ACSvol6/papers/paper-6-15.pdf
Gilovich, T. 1983. Biased Evaluation and Persistence in Gam-
De Neys, W.; and Glumicic, T. 2008. Conflict Monitoring in Dual bling. Journal of Personality and Social Psychology 44(6): 1110.
Process Theories of Thinking. Cognition, 106: 1248–1299. https://pdfs.semanticscholar.org/f644/196083e55f0f824f849625f6
doi.org/10.1016/j.cognition.2007.06.002 e5592aa1d686.pdf
Dyczewski, E. A.; and Markman, K. D. 2012. General Attainabil- Glenberg, A. M.; Wilkinson, A. C.; and Epstein, W. 1982. The il-
ity Beliefs Moderate the Motivational Effects of Counterfactual lusion Of Knowing: Failure in The Self-Assessment of Compre-
Thinking. Journal of Experimental Social Psychology 48(5): hension. Memory and Cognition 10: 597-602. https://link.springer
1217-1220. doi.org/10.1016/j.jesp.2012.04.016 .com/content/pdf/10.3758%2FBF03202442.pdf
Efklides, A. 2006. Metacognition and Affect: What Can Metacog- Halpern, D. F. 1998. Teaching Critical Thinking for Transfer
nitive Experiences Tell Us About the Learning Process? Educa- Across Domains: Disposition, Skills, Structure Training, and Met-
tional Research Review 1(1): 3-14. acognitive Monitoring. American Psychologist 53(4): 449.
doi.org/10.1016/j.edurev.2005.11.001 doi.org/10.1037/0003-066x.53.4.449
Efklides, A.; Samara, A.; and Petropoulou, M. 1999. Feeling of Halpern, D. F. 2010. Halpern Critical Thinking Assessment.
Difficulty: An Aspect of Monitoring That Influences Control. Eu- SCHUHFRIED (Vienna Test System): Moedling, Austria.
ropean Journal of Psychology of Education 14(4): 461-476. www.schuhfried.com/viennatestsystem10/tests-test-sets/all-tests-
http://www.jstor.org/stable/23420265 from-a-z/test/hcta-halpern-critical-thinking-assessment-1/
Epstein, S.; Pacini, R.; Denes-Raj, V.; and Heier, H. 1996. Individ- Halpern, D. F. 2014. Thought and Knowledge: An Introduction to
ual Differences in Intuitive-Experiential and Analytical-Rational Critical Thinking. New York: Psychology Press.
Thinking Styles. Journal of Personality and Social Psychology
71(2): 390-405. Hough, A. R.; and Gluck, K. Forthcoming. The understanding
psycnet.apa.org/doiLanding?doi=10.1037%2F0022- problem in cognitive science. Advances in Cognitive Systems 7.
3514.71.2.390 Hur, T. (2001). The Role of Regulatory Focus in Activation of
Epstude, K.; and Roese, N. J. 2008. The Functional Theory of Counterfactual Thinking. Korean Journal of Social and Personal-
Counterfactual Thinking. Personality and Social Psychology Re- ity Psychology 15: 159-171.
view 12(2): 168-192. Johnson, B.; Roberts, M.; Apker, T.; and Aha, D. W. 2016. Goal
Reasoning with Information Measures. In Proceedings of the
Fourth Conference on Advances in Cognitive Systems 4: 1-14.
Johnson-Laird, P. N. 2013. The Mental Models Perspective. In The Markman, K. D.; McMullen, M. N.; and Elizaga, R. A. 2008.
Oxford Handbook Of Cognitive Psychology, edited by D. Reisberg, Counterfactual thinking, Persistence, and Performance: A Test of
650-667. New York: Oxford University Press. the Reflection and Evaluation Model. Journal of Experimental So-
Juvina, I.; Larue, O.; and Hough, A. 2017. Modeling Valuation and cial Psychology 44(2): 421-428.
Core Affect in A Cognitive Architecture: The Impact of Valence doi.org/10.1016/j.jesp.2007.01.001
and Arousal on Memory and Decision-Making. Cognitive Systems Mata, A.; Ferreira, M. B.; and Sherman, S. J. 2013. The Metacog-
Research 48: 4-24. doi.org/10.1016/j.cogsys.2017.06.002 nitive Advantage of Deliberative Thinkers: A Dual-Process Per-
Juvina, I.; Larue, O.; Widmer, C.; Ganapathy, S.; Nadella, S.; and spective on Overconfidence. Journal of Personality and Social
Minnery, B. under review. Computer-supported Collaborative In- Psychology 105(3): 353. dx.doi.org/10.1037/a0033640
formation Search for Geopolitical Forecasting. Mayer, R. E. 1998. Cognitive, Metacognitive, and Motivational
Kahneman, D. 2011. Thinking, Fast and Slow. New York: Farrar, Aspects of Problem Solving. Instructional Science 26: 49-63.
Straus, and Giroux. link.springer.com/con-
Kahneman, D.; and Klein, G. 2009. Conditions for Intuitive Exper- tent/pdf/10.1023%2FA%3A1003088013286.pdf
tise: A Failure to Disagree. American Psychologist 64(6): 515 526. McNamara, D. S.; Kintsch, E.; Songer, N. B.; and Kintsch, W.
doi.org/10.1037/a0016755 1996. Are Good Texts Always Better? Interactions of Text Coher-
Kahneman, D.; and Miller, D. T. 1986. Norm Theory: Comparing ence, Background Knowledge, and Levels of Understanding in
Reality to Its Alternatives. Psychological Review 93(2): 136. Learning from Text. Cognition and Instruction 14: 1-43.
pdfs.semanticscholar.org/9809/8ee48700173e2f09aeff48c4 doi.org/10.1207/s1532690xci1401_1
06ef943918b5.pdf Mellers, B.; Stone, E., Murray, T.; Minster, A.; Rohrbaugh, N.;
Bishop, M.; ... and Ungar, L. 2015. Identifying and Cultivating Su-
Kennerley, S. W.; Dahmubed, A. F.; Lara, A. H.; and Wallis, J. D. perforecasters as a Method of Improving Probabilistic Predictions.
2009. Neurons in the Frontal Lobe Encode the Value of Multiple Perspectives on Psychological Science 10(3): 267-281.
Decision Variables. Journal of Cognitive Neuroscience 21(6): doi.org/10.1177/1745691615577794
1162-1178.www.mitpressjour-
nals.org/doi/abs/10.1162/jocn.2009.21100 Muñoz-Avila, H.; Dannenhauer, D.; and Reifsnyder, N. 2019. Is
Everything Going According to Plan? Expectations in Goal Rea-
Kirk, J. R.; and Laird, J. 2014. Interactive Task Learning for Sim- soning Agents. In Proceedings of the AAAI Conference on Artifi-
ple Games. Advances in Cognitive Systems 3: 13-30. cial Intelligence 33: 9823-9829.
www.cogsys.org/pdf/paper-9-3-33.pdf doi.org/10.1609/aaai.v33i01.33019823
Klein, G. 1998. Sources of Power: How People Make Decisions. Muñoz -Avila, H.; and Cox, M. T. 2008. Case-based Plan Adapta-
Cambridge, MA: MIT Press. tion: An Analysis and Review. IEEE Intelligent Systems 23: 75–
Klein, G.; Moon., B.; and Hoffman, R. R. 2006a. Making Sense of 81. doi.org/10.1109/MIS.2008.59
Sensemaking 1: Alternative Perspectives. IEEE Intelligent Systems Newell, A.; and Simon, H. 1972. Human Problem Solving. Eng-
21(4): 70-73. doi.org/10.1109/MIS.2006.75 lewood Cliffs, NJ: Prentice-Hall.
Klein, G.; Moon., B.; and Hoffman, R. R. 2006b. Making Sense of Novick, L. R.; and Holyoak, K. J. 1991. Mathematical Problem
Sensemaking 2: A Macro Cognitive Model. IEEE Intelligent Sys- Solving by Analogy. Journal of Experimental Psychology: Learn-
tems 21(5): 88-92. doi.org/10.1109/MIS.2006.100 ing, Memory, and Cognition 17(3): 398-415.
Klein, G.; Phillips, J. K.; Rall, E.; and Peluso, D. A. 2007. A https://pdfs.semanticscholar.org/6bc0/a20b01f0ebfc243d00de3
Data/Frame Theory of Sensemaking. In Expertise out of context: e55f6387f8a77b8.pdf
Proceedings of the 6th international conference on naturalistic de- Payne, J. W.; Bettman, J. R.; and Johnson, E. J. 1988. Adaptive
cision making, edited by R. R. Hoffman, 118-160. New York: Psy- Strategy Selection in Decision Making. Journal of Experimental
chology Press. Psychology: Learning, Memory, and Cognition 14(3): 534-552.
Klein, G.; Ross, K. G.; Moon, B. M.; Klein, D. E.; Hoffman, R. R.; apps.dtic.mil/dtic/tr/fulltext/u2/a170858.pdf
and Hollnagel, E. (2003). Macrocognition. IEEE Intelligent Sys- Pennycook, G. 2017. A Perspective on the Theoretical Foundation
tems, 18(3), 81-85. doi.org/10.1109/MIS.2003.1200735 of Dual Process Models. In Dual Process Theory 2.0, edited by W.
Klein, G.; Snowden, D.; and Pin, C. L. 2007. Anticipatory Think- De Neys, 6-26. New York: Routledge.
ing. In Proceedings of the Eighth International NDM Conference, Pennycook, G.; Fugelsang, J. A.; and Koehler, D. J. 2015a. What
edited by K. Mosier and U. Fischer, 1-8. Pacific Grove, CA, June Makes Us Think? A Three-Stage Dual-Process Model of Analytic
2007. Engagement. Cognitive Psychology 80: 34-72.
Kurtz, C.F.; and Snowden, D.J. 2003. The New Dynamics of Strat- doi.org/10.1016/j.cogpsych.2015.05.001
egy: Sensemaking In A Complex and Complicated World. e-Busi- Pennycook, G.; Fugelsang, J. A.; and Koehler, D. J. 2015b. Every-
ness Management, 42. doi.org/10.1147/sj.423.0462 day Consequences of Analytic Thinking. Current Directions in
Larue, O.; Hough, A.; and Juvina, I. (2018). A Cognitive Model of Psychological Science 24(6): 425-432.
Switching Between Reflective and Reactive Decision Making in doi.org/10.1177/0963721415604610
The Wason Task. In Proceedings of the Sixteenth International Perkins, D. N. 1998. What is Understanding? In Teaching for Un-
Conference on Cognitive Modeling, edited by I. Juvina, J. Houpt, derstanding: Linking Research with Practice, edited by M. S.
and C. Myers, 55-60. Madison, WI: University of Wisconsin. Wiske, 39-57. San Francisco, CA: Jossey-Bass.
Markman, K. D.; Gavanski, I.; Sherman, S. J.; and McMullen, M. Perkins, D. N.; and Simmons, R. 1988. Patterns of Misunderstand-
N. 1993. The Mental Simulation of Better and Worse Possible ing: An Integrative Model for Science, Math, and Programming.
Worlds. Journal of Experimental Social Psychology 29: 87-109. Review of Educational Research 58: 303-326.
doi.org/10.1006/jesp.1993.1005 doi.org/10.3102/00346543058003303
Pirolli, P.; and Russell, D. M. 2011. Introduction to This Special https://psycnet.apa.org/doiLanding?doi=10.1037%2F0033-
Issue on Sensemaking. Human-Computer Interaction 26: 1-8. 295X.114.3.577.
www.tandfonline.com/doi/full/10.1080/07370024.2011.556557 Tetlock, P. E.; and Gardner, D. 2015. Superforcasting: The Art and
Pirolli, P.; and Card, S. K. 1999. Information Foraging. Psycholog- Science of Prediction. New York, NY: Crown.
ical Review 106: 643-675. Thompson, V. A.; and Johnson, S. C. 2014. Conflict, Metacogni-
Pirolli, P.; and Card, S. K. 2005. The Sensemaking Process and tion, and Analytic Thinking. Thinking and Reasoning 20(2): 215-
Leverage Points for Analyst Technology. Paper presented at the 244. doi.org/10.1080/13546783.2013.869763
International Conference on Intelligence Analysis, McLean, VA. Thompson, V. A.; Evans, J. S. B.; and Campbell, J. I. 2013. Match-
pdfs.semanticscholar.org/f6b5/4379043d0ee28bea45555f481af1a ing Bias on the Selection Task: It's Fast and Feels Good. Thinking
693c16c.pdf and Reasoning 19: 431-452.
Pozanco, A.; Fernández, S.; and Borrajo, D. 2018. Learning-driven doi.org/10.1080/13546783.2013.820220
Goal Generation. AI Communications 31: 137–150. Thompson, V. A.; Prowse Turner, J. A.; and Pennycook, G. 2011.
Roberts, M.; Borrajo, D.; Cox, M. T.; and Yorke-Smith, N. 2018. Intuition, Reason, and Metacognition. Cognitive Psychology 63:
Special Issue on Goal Reasoning. AI Communications 31: 115– 107-140. doi.org/10.1016/j.cogpsych.2011.06.001
116. http://doi.org/10.3233/AIC-180754 Toplak, M. E.; West, R. F.; and Stanovich, K. E. 2011. The Cog-
Roese, N. J.; and Epstude, K. 2017. The Functional Theory of nitive Reflection Test as A Predictor of Performance on Heuristics-
Counterfactual Thinking: New Evidence, New Challenges, New And-Biases Tasks. Memory and Cognition 39(7): 1275-1289.
Insights. In Advances in Experimental Social Psychology: Vol. 56, link.springer.com/article/10.3758%2Fs13421-011-0104-1
edited by J. M. Olson, 1-79. Cambridge, MA: Academic Press. Vattam, S.; Klenk, M.; Molineaux, M.; and Aha, D. W. 2013.
doi.org/10.1016/bs.aesp.2017.02.001 Breadth of Approaches to Goal Reasoning: A Research Survey.
Roese, N. J.; and Olson, J. M. 1997. Counterfactual Thinking: The In Goal Reasoning: Papers from the ACS Workshop, edited by D.
Intersection of Affect and Function. In Advances in experimental W. Aha, M. T. Cox, and H. Muñoz-Avila. Technical Report CS-
social psychology: Vol. 29, edited by M. P. Zanna, 1-59. San Di- TR-5029. College Park, MD: University of Maryland, Department
ego, CA: Academic Press. of Computer Science. apps.dtic.mil/dtic/tr/fulltext/u2/a603400.pdf
doi.org/10.1016/S0065-2601(08)60015-5 Warwick, W.; and Hutton, R. J. B. 2007. Computational and The-
Roese, N. J.; Hur, T.; and Pennington, G. L. 1999. Counterfactual oretical Perspectives on Recognition-Primed Decision Making. In
Thinking and Regulatory Focus: Implications for Action Versus Expertise out of context: Proceedings of the sixth international
Inaction and Sufficiency Versus Necessity. Journal of Personality conference on naturalistic decision making, edited by R. R. Hoff-
and Social Psychology 77(6): 1109. man, 429-452. New York: Psychology Press.
doi.org/10.1037//0022-3514.77.6.1109 Wason, P. C. 1966. Reasoning. In New horizons in psychology, ed-
Russell, D. M.; Stefik, M. J.; Pirolli, P.; and Card, S. K. 1993. The ited by B. Foss, 135-151. London, UK: Penguin.
Cost of Structure of Sensemaking. Paper presented at the INTER- Wason, P. C. 1968. Reasoning About a Rule. Quarterly Journal of
CHI 1993 conference on Human Factors in Computing Systems, Experimental Psychology 20: 273-281.
Amsterdam, the Netherlands. doi.org/10.1145/169059.169209 doi.org/10.1080%2F14640746808400161
Sanna, L. J.; and Turley, K. J. 1996. Antecedents to Spontaneous Wason, P. C. 1969. Regression in Reasoning? British Journal of
Counterfactual Thinking: Effects of Expectancy Violation and Psychology 60(4): 471-480.
Outcome Valence. Personality and Social Psychology Bulletin 22: doi.org/10.1111/j.2044-8295.1969.tb01221.x
906-919. doi.org/10.1177%2F0146167296229005 Weick, K.E.; and Sutcliffe, K.M. 2001. Managing the Unexpected:
Sanna, L. J.; and Turley-Ames, K. J. 2000. Counterfactual Inten- Assuring High Performance in An Age of Complexity. San Fran-
sity. European Journal of Social Psychology 30: 273-296. cisco: Jossey-Bass.
doi.org/10.1002/(SICI)1099-0992(200003/04)30:2%3C273::AID- West, R. F.; Toplak, M. E.; and Stanovich, K. E. 2008. Heuristics
EJSP993%3E3.0.CO;2-Y and Biases as Measures of Critical Thinking: Associations with
Schank, R. C.; and Abelson, R. P. 1977. Scripts, Plans, Goals, and Cognitive Ability and Thinking Dispositions. Journal of Educa-
Understanding: An Inquiry into Human Knowledge Structures. tional Psychology 100(4): 930-941.
Hillsdale, NJ: Lawrence Erlbaum Associates. https://psycnet.apa.org/doiLanding?doi=10.1037%2Fa0012842
Stanovich, K. E. 2009. Distinguishing the Reflective, Algorithmic, Westbrook, A.; Kester, D.; and Braver, T. S. 2013. What is the
And Autonomous Minds: Is It Time for A Tri-Process Theory. In Subjective Cost of Cognitive Effort? Load, Trait, and Aging Ef-
Two minds: Dual processes and beyond, edited by J. S. B. T. Evans fects Revealed by Economic Preference. PloS one 8(7): e68210.
and K. Frankish, 55-88. New York: Oxford University Press. doi.org/10.1371/journal.pone.0068210
Stanovich, K. E. 2018. Miserliness in Human Cognition: The In- Willingham, D. T. 2007. Critical Thinking: Why is it So Hard to
teraction of Detection, Override and Mindware. Thinking and Rea- Teach? American Educator 31(3): 8-19.
soning 24(4): 423-444. doi.org/10.1080/13546783.2018.1459314 doi.org/10.3200/AEPR.109.4.21-32
Swan, A. B.; Calvillo, D. P.; and Revlin, R. 2018. To Detect or Not Woodward, J. 2003. Making Things Happen: A Theory of Casual
to Detect: A Replication and Extension of the Three-stage Explanation. New York, NY: Oxford University Press.
Model. Acta Psychologica 187: 54-65.
doi.org/10.1016/j.actpsy.2018.05.003
Taatgen, N. A.; Van Rijn, H.; and Anderson, J. 2007. An Integrated
Theory of Prospective Time Interval Estimation: The Role of Cog-
nition, Attention, and Learning. Psychological Review 114(3):
577-598.