=Paper=
{{Paper
|id=Vol-1647/SAL2016_paper_5
|storemode=property
|title=Metacognitive Knowledge and Metacognitive Regulation in
Time-Constrained in Information Search
|pdfUrl=https://ceur-ws.org/Vol-1647/SAL2016_paper_5.pdf
|volume=Vol-1647
|authors=Anita Crescenzi
|dblpUrl=https://dblp.org/rec/conf/sigir/Crescenzi16
}}
==Metacognitive Knowledge and Metacognitive Regulation in
Time-Constrained in Information Search==
https://ceur-ws.org/Vol-1647/SAL2016_paper_5.pdf
Metacognitive Knowledge and Metacognitive Regulation
in Time-Constrained in Information Search
Anita Crescenzi
School of Information and Library Science, University of North Carolina
Chapel Hill, NC, USA
amcc@email.unc.edu
ABSTRACT metacognitive experiences are influenced by the metacognitive
Learning in search can take the form of changes in conceptual knowledge that they retrieve from memory, and an individual’s
knowledge and procedural knowledge at a cognitive level. metacognitive experiences can, in turn, shape the individual’s
Searchers may also gain insights into their cognitive process metacognitive knowledge.
though their metacognitive experiences and thereby acquire Metacognitive knowledge consists of one’s conscious or
metacognitive knowledge which may also influence their search subconscious beliefs or knowledge about people, tasks and
process and outcomes. This paper describes the concepts of strategies that influence the cognitive process [9]. Metacognitive
metacognition, metacognitive knowledge, metacognitive knowledge includes knowledge about one’s own or others’
experiences and metacognitive self-regulation and findings from cognitive processes, self-efficacy, motivation or interest;
an experimental information search study. In particular, the study knowledge of general strategies that might be used to achieve
provides evidence of participants’ metacognitive experiences and one’s cognitive goals; or knowledge of the task context such as
metacognitive regulation activities relating to orienting to task and the task demands or the information available during this task.
time, monitoring and steering the process, and evaluating the Metacognitive knowledge can change over time, i.e., people can
search process and outcome. learn metacognitive knowledge.
Keywords Flavell [9] describes a learner’s cognitive or affective
Information search; metacognition; metacognitive self-regulation metacognitive experiences during a cognitive task. Efklides [8]
defines metacognitive experiences as “what the person
1. INTRODUCTION experiences during a cognitive endeavor, be it online
Learning during search is often thought of in terms of a searcher’s metacognitive knowledge, ideas and beliefs, or feelings, goals,
gain in cognitive knowledge: searchers learn about the concepts in judgments. Efklides [9] describes metacognitive experiences in
the search or how to do the search. Searchers may also gain terms of one’s judgments or feelings that take place and evolve
insights into their own cognitive processes and use these during the learning process. In particular, metacognitive
metacognitive insights to help determine how to proceed in their experiences include a learner’s feelings of knowing, difficulty,
search. In this paper, I provide an overview of metacognition as confidence, satisfaction, and familiarity as well as learner’s
well as two major constituent parts: knowledge about cognition judgments of learning, solution correctness, and cognitive
and regulation of cognition. I also present evidence from recent demands including estimates of the time and effort required to
research [6, 7] using previously unpublished data to illustrate complete a task.
these concepts.
An individual’s metacognitive knowledge and experiences can
Based on his research on the learning strategies, in 1979, John also influence an individual’s future use of cognitive and
Flavell observed that “young children are quite limited in their metacognitive strategies and processes [9]. Researchers have
knowledge and cognition about cognitive phenomena, or in their analyzed executive control processes involved in the regulation of
metacognition, and do relatively little monitoring of their own cognition. Brown and Palincsar [5] describe several types of
memory, comprehension, and other cognitive enterprises” [9]. He executive functions that one might undertake to regulate their
suggested that promising areas for new research include better cognitive processes:
understanding metacognition and its development as well as
“planning activities prior to undertaking a problem
cognitive monitoring. This differentiation of knowledge about
cognitive phenomena and monitoring of cognition has continued (predicting outcomes, scheduling strategies, and various
in the literature with other researchers also differentiating between forms of vicarious trial and error, etc.), monitoring activities
two components of metacognition: one’s knowledge of cognition during learning (monitoring, testing, revising, and re-
and regulation of cognition [5, 11, 17]. scheduling one's strategies for learning) and checking
outcomes (evaluating the outcome of any strategic actions
Knowledge about cognition was described by Flavell [9] as two against criteria of efficiency and effectiveness).” (p.3,
overlapping concepts: one’s metacognitive knowledge and one’s emphasis in original)
metacognitive experiences during a cognitive task. These concepts A taxonomy of metacognitive activities in text studying and
are closely related. During a cognitive process, an individual’s problems solving developed by Meijer, Veenman and van Hout-
Wolters [14] contains six types of metacognitive activities:
Search as Learning (SAL), July 21, 2016, Pisa, Italy
orientating, planning, executing, monitoring, evaluation, and
The copyright for this paper remains with its authors.
Copying permitted for private and academic purposes. elaboration.
This research seeks to understand if metacognition is observable amount of information remaining). They found greater factual
in interactive information retrieval studies and whether gain for participants using met.a.ware with monitoring or
participants were able to describe their metacognitive experiences evaluation prompts versus pencil and paper measured comparing
and self-regulation activities. Specifically, it asks the following pre-and post-search multiple-choice tests. They also found higher
research question: Do searchers report metacognitive experiences comprehension scores as measured by graded essays with
and metacognitive self-regulation in search? met.a.ware with monitoring prompts than pencil and paper.
2. BACKGROUND We took a preliminary look at metacognition in information
In studies of information-seeking and information search, search in a user study investigating the impact of time pressure
metacognition has been investigated in the information behaviors and system delays on information search [6, 7]. In a lab-based
of adolescents [2], information problem solving of adult experts experiment, we manipulated the time available for searchers to
versus novices [3], and collaborative information seeking [13]. complete a search task (5 minutes vs. no limit, between-subjects)
Bowler [2] created a taxonomy of metacognitive knowledge from as well as the speed with which we presented search results and
her longitudinal study of high school student information seeking documents (immediately vs. adding a 5 second delay the first time
to support the writing of an essay. She uncovered 13 categories of each SERP and document loaded, within-subjects). 43 participants
metacognitive knowledge, including categories relating to completed four search tasks in which they searched for and
knowing about ones’ cognitive processes (e.g., knowing what you bookmarked 8-12 pages providing a good overview of news
don’t know, knowing your strengths and weaknesses), knowledge coverage for that topic.
of strategies to achieve goals (e.g., building a base of knowledge, After participants searched for each task, they completed a post-
communicating with others who serve as information mediators), task questionnaire in which we asked them questions to better
and knowledge of task characteristics which might help guide understand the extent to which they engaged in metacognitive task
selection of strategy (e.g., understanding time and effort, changing monitoring. Overall, participants reported monitoring how well
course after evaluating one’s progress). they were doing on the task (M=4.87, SD=1.53) and how much
Metacognitive regulation is an explicit component of the time they had left on the task (M=4.41, SD=1.8). Participants also
Information Problem Solving Model [3, 4]. The model includes reported high levels of monitoring how much information they
the process and skills needed for information problem solving as had found and still needed (M=5.56, SD=1.23) [7]. Higher levels
well as process regulation activities. These regulation activities of monitoring task progress and time remaining were predicted if
include orientation to the information problem, to the task and to participants were approaching the end of their time limit (see [7]
time; monitoring and steering the problem solving process by for full analysis and statistical models). This suggests that the
monitoring task performance and planning what to do overall and presence of a time limit lead to higher metacognitive regulation
next actions; and testing including evaluating the process and during search. Interestingly, we also found interactions with time
product during the task and at its completion. In an experimental limit and task order: time-limited participants reported more
study using this model as a framework [3], adult experts were progress monitoring for their second task than their first task. This
found to regulate their process more frequently with significantly suggests that participants may have learned how to complete these
more monitoring and steering activities than novices. Experts tasks. In other words, participants’ metacognitive evaluation of
were also found to monitor time more than novices. Another study their performance on the first task may have influenced their
found increased levels of regulation activities among students planning and their performance on their second task.
pursuing a PhD or an education degree compared to freshman
Participants also completed an exit questionnaire with several
psychology or secondary education students [4]. When examining open-ended questions relating to metacognition and a debriefing
individual types of regulation activities, they found that secondary
interview that also served to participants of the experimental
education students conducted fewer orientation activities as well
manipulations. Analysis of the data from the exit questionnaire
as less monitoring and steering of their process than any of the
was not previously presented and is the focus of this paper.
university students.
Search regulation has also been investigated in collaborative 3. METHOD
information-seeking. Lazonder [13] examined search regulation We analyzed responses to open-ended questions from the exit
and performance in an experimental study investigating questionnaire in the study described in the previous section. These
information search in groups versus individuals. He found that questions, shown in Table 1, were designed to provide insight into
pairs of searchers were able to more quickly and correctly answer metacognitive task regulation. They asked about the impact of the
questions and they exhibited more planning and monitoring time limit on 1) the process they used to complete the task, 2) the
behaviors: they used significantly more new search strategies amount of information they found, and 3) the extent to which they
(e.g., switching to search from entering in a URL) and were better read the information they found. They were asked the same
able to identify relevant information within a page. Success was questions about the impact of system speed.
positively correlated with several monitoring and evaluation Table 1. Open-ended questions relating to metacognition
measures: identifying relevant information, re-checking the
answers, and revising answers. Did the (system speed/ time you had to complete the tasks)
impact…
Researchers have also built and evaluated learning systems with
features designed to support metacognition. Stadtler and Bromme …the process you used to complete the tasks (e.g., steps, thought
process)? Please explain.
[18] built the met.a.ware system to provide a structured
notetaking capability with two types of metacognitive prompts: …the amount of information you found? Please explain.
prompts to evaluate the information found (source, author …the extent to which you read the information that you found?
expertise, author bias, and their confidence in information), and Please explain.
prompts to monitor progress (knowledge, comprehension and
Participant’s exit question responses were analyzed using feeling that I was running out of time. So, I began to skim
deductive qualitative content analysis approach. For each sentence faster to complete the task.”
or sentence fragment, we coded for mentions of the participant’s One participant without task time limits described feeling nervous
knowledge of cognition and regulation of cognition. For about the time of the experimental session.
knowledge of cognition, we coded for participant’s mentions of
metacognitive knowledge using Flavell’s definition [9], and for “I felt time pressure during the first task when I had to keep
metacognitive experiences as defined by Flavell [9] and Efklides searching to find any articles. I was nervous this entire process
[8]. For participant’s mentions of regulation of cognition, we used would take much longer than the expect 1.5 hours.”
the regulation activities described by Brand-Gruwel et al. [3, 4]: 4.2 Metacognitive regulation: Orienting to
orientation (to task, time or problem), monitoring and steering,
and testing. These definitions are summarized in Table 2. task or time
Participants also described thinking about and orienting
themselves towards the task they were to perform. One participant
Table 2. Categories used in coding of participant responses described carefully orienting towards the (imposed) search task at
Knowledge of cognition the beginning of the session.
Metacognitive knowledge [9] “At the start as I wanted to make sure I fully understood the
task so I spent more time reading the prompt and going back
Knowledge about one’s own or others’ cognitive processes, again to make sure I knew it well before I started my search”
self-efficacy, motivation or interest
Another participant described setting his/her goal to meet the task
Knowledge of general strategies that might be used to achieve demands even though he/she could have found more information.
one’s cognitive goals
“…since I only needed to find 8-12 articles with a good
Knowledge of types of task and how the contextual factors overview, I didn't feel the need to spend larger amounts of
differ (e.g., task demands or information available) time searching.”
Metacognitive experiences [8, 9] Participants also described differing levels of orienting towards
Feelings of knowing, difficulty, confidence, satisfaction, and the time they had to complete the task. Two participants noted that
familiarity they were not focused on time at all until after the end of their first
task because they ran out of time.
Judgments of learning, solution correctness, and cognitive
demands including estimates of the time and effort required to “I only felt the pressure when I was told that time was up.
complete a task. Other than that I was more focused on the task than the time.”
Attention paid to task-specific knowledge, task characteristics “During the first task, I was not paying attention to the time.
or strategies specific to the task. After that I paid more attention.”
Metacognitive regulation [3, 4] One participant noted that he/she planned how to allocate time for
tasks based using what he/she learned about in the initial tasks.
Orientation to the information problem, to the task and to time
“During the practice test and first test, I felt some pressure to
Monitoring and steering the problem solving process by work quickly. However, once I had a better sense of how the
monitoring task performance and planning what to do overall and system operated, and the speed with which I got through those
next actions tasks, I was less-concerned about the time.”
Testing including evaluating the process and product during the One participant noted needing to adjust after running out of time
task and at its completion on the initial task.
“…after the first search, I realized that five minutes doesn't
4. RESULTS go very far for this task. By the fourth search, I was very
In their open-ended questions, many participants described at least preoccupied with time.”
one aspect of metacognition. Below, we describe participant 4.3 Metacognitive regulation: Monitoring
responses relating to metacognitive experiences and their
engagement in metacognitive self-regulation activities such as progress and steering process
orienting to the imposed task and time, monitoring and steering Participants described monitoring their task progress and adapting
progress, and evaluating outcomes. their search strategies during and between tasks based on their
previous metacognitive experiences. Two participants described
4.1 Metacognitive experience adjusting their task goal based on their (metacognitive) evaluation
Participants described conscious metacognitive experiences of their performance in previous tasks.
relating to the time they were spending on the task and the “One task I did not complete before the task ended. I made the
affective response (time pressure) that accompanied it. Multiple goal of 8 for myself after that and did not try to get 12.”
people described “sensing” or having a feeling that they were
running out of time. “I had plenty of time and thus usually tried to shoot for the
max recommended articles of around 12 as opposed to only
“After the piracy topic [first topic], I felt the pressure 8.”
immediately. It became more acute when I could sense the
time slipping away and I still had only 5 or 6 articles.” Participants mentioned skimming articles rather than reading them
with a time limit (n=22, 100%) and no time limit (n=13, 65%).
“I felt the most pressure about halfway through the second
one, where I had not found enough information yet and had a “Yes. I only skimmed most of the articles, and only carefully
read a few of them.”
One participant described his/her metacognitive knowledge about “Approximately a minute or two into the task, when I had
skimming as a strategy and why skimming was the right strategy found some articles but didn't want to waste time checking to
for this task. see the exact number.”
“I can skim quickly to get a general feeling of what the article 5. CONCLUSIONS
is about, and then go from there to determine if it is relevant Our participants described metacognition in information search;
or not. But, in this case, that was all I could do. Reading, on they self-reported conscious metacognition experiences as well as
the other hand, takes more time but lends more information an impact of metacognitive experiences on their regulation of
directly and can give a better picture of what the article is search. We found examples of multiple metacognitive regulation
specifically about.” activities. Participants described orienting themselves to both the
Multiple participants described how they regulated the search search task and the time available to do the task. They also
process based on their metacognitive experiences. Several described monitoring their search process and steering or adapting
mentioned faster skimming when their time was almost up. their processes as a result. Finally, they described evaluating their
search process and task outcomes.
“I felt the most pressure about halfway through the second
one, where I had not found enough information yet and had a Two particular findings are of particular note. First, higher levels
feeling that I was running out of time. So, I began to skim of metacognitive monitoring of task progress and time available in
faster to complete the task.” the presence of a time limit and the interaction with time spent on
Participants also described abandoning an unfruitful search task suggest that participants may have devoted more attentional
strategy by re-querying more quickly. resources to regulating the task. It is not clear from the results
whether the additional regulation activity had an effect on the
“When I knew there would be a time limit, I scanned articles search outcome; future research will investigate this. Second, the
more quickly and I changed my search strategy as soon as the sequence effect from the questionnaire analysis combined with
article titles began to look less promising.” participant explanations indicate that participants refined their
“I had to think much faster than usual to decide whether the search process after the first task suggests that participants
information was relevant or not. I also had to try different acquired procedural knowledge based on their metacognitive
search queries much more frequently than normal in order to experiences and regulation in their earlier tasks.
find a combination that worked.”
Future research will examine the cognitive and metacognitive
Two participants mentioned increased selectivity in clicking items strategies and metacognitive regulation in information search
from search results page as a result of the delays. using additional methods to gain insight into metacognition and to
“I was more selective in clicking on article because I knew the provide triangulation (e.g., [12]). For example, concurrent think-
system would take a little time to show the article aloud has been used to provide insight into metacognitive
consequently more time used.” experiences and regulation activities that take place during a task
(e.g., [3, 4]). Several questionnaires have been developed and
“the system speed took away a few seconds each time I validated which ask students to self-report their awareness of
opened an article, so I found myself skimming more and more metacognition (e.g., the Metacognitive Awareness Inventory
in order to find enough relevant articles.” [17]), the use of metacognitive strategies during a course (e.g., the
Another described using a more effortful strategy (reading more metacognitive strategies within learning strategies subscale of the
deeply) if they thought it would be relevant. Motivated Strategies for Learning Questionnaire, [16]), or the use
“I skimmed most… when I wasn't able to skim to ascertain of metacognitive strategies used in a particular domain (e.g., the
whether it was relevant I read more closely if I really thought Text-Learning Strategies Inventory, [15]). Eye-tracking (e.g.,
(based on the article title) the article would be relevant.” [19]) and logged computer interactions (e.g., [1, 20]) have also
been used. Future work could also examine the efficacy of
4.4 Metacognitive regulation: Testing or metacognitive prompts or scaffolding in information search (e.g.,
evaluating [1, 18]).
Participants described evaluating their progress or task outcome in 6. ACKNOWLEDGMENTS
terms of the amount or the type of information they found.
Thanks to study participants and my collaborators on this study:
“…I did not think I had enough different examples of articles Diane Kelly and Leif Azzopardi.
(too many articles focusing on China for example).”
“I was not certain that I had the breadth of information needed
7. REFERENCES
for each task.” [1] Bannert, M., Sonnenberg, C., Mengelkamp, C. and Pieger,
E. 2015. Short- and long-term effects of students’ self-
Participants also described learning a strategy for these search directed metacognitive prompts on navigation behavior
tasks. and learning performance. Computers in Human
“At the end I already established some rules as how to Behavior. 52, Nov. 2015, 293-306.
approach each search so it was easier and I was able to do it [2] Bowler, L. 2010. A taxonomy of adolescent metacognitive
faster without feeling as much pressure.” knowledge during the information search process. Library
“…at the end I had acquired a method of searching the topics & Information Science Research. 32, 1, 27–42.
and there were steps I followed to do this.” [3] Brand-Gruwel, S., Wopereis, I. and Vermetten, Y. 2005.
Participants also described the impact on the evaluation of their Information problem solving by experts and novices:
results. Analysis of a complex cognitive skill. Computers in
Human Behavior. 21, 3, 487–508.
[4] Brand-Gruwel, S., Wopereis, I. and Walraven, A. 2009. A [13] Lazonder, A.W. 2005. Do two heads search better than
descriptive model of information problem solving while one? Effects of student collaboration on web search
using internet. Computers & Human Behavior. 53, 4, behaviour and search outcomes. British Journal of
1207-1217. Educational Technology. 36, 3, 465–475.
[5] Brown, A.L. and Palincsar, A.S. 1982. Inducing strategic [14] Meijer, J., Veenman, M.V.J. and van Hout-Wolters,
learning from texts by means of informed, self-control B.H.A.M. 2006. Metacognitive activities in text-studying
training. Center for the Study of Reading, Technical and problem-solving: Development of a taxonomy.
Report No. 262. University of Illinois at Urbana- Educational Research and Evaluation. 12, 3, 209–237.
Champaign, Center for the Study of Reading, Champaign [15] Merchie, E., Van Keer, H., and Vandevelde, S. 2014.
IL. Development of the Text-Learning Strategies Inventory:
[6] Crescenzi, A., Kelly, D. and Azzopardi, L. 2015. Time Assessing and profiling learning from texts in fifth and
Pressure and System Delays in Information Search. sixth grade. Journal of Psychoeducational Assessment. 32,
Proceedings of the 38th International ACM SIGIR 6, 533-547.
Conference on Research and Development in Information [16] Pintrich, P. R., and De Groot, E. V. 1990. Motivational
Retrieval (SIGIR '15). ACM, New York, NY, USA, 767- and self-regulated learning components of classroom
770. academic performance. Journal of Educational
[7] Crescenzi, A., Kelly, D. and Azzopardi, L. 2016. Impacts Psychology. 82, 1, 33-40.
of Time Constraints and System Delays on User [17] Schraw, G. and Dennison, R.S. 1994. Assessing
Experience. In Proceedings of the 2016 ACM on Metacognitive Awareness. Contemporary Educational
Conference on Human Information Interaction and Psychology. 19, 4, 460–475.
Retrieval, CHIIR 2016, ACM. New York. 141-150/
[18] Stadtler, M. and Bromme, R. 2008. Effects of the
[8] Efklides, A. 2001. Metacognitive experiences in problem metacognitive computer-tool met.a.ware on the web search
solving. In Trends and Prospects in Motivation Research. of laypersons. Computers in Human Behavior. 24, 3, 716–
Efklides, A., Kuhl, J. and Sorrentino, R. M. (eds). Springer 737.
Netherlands. Dordrecht. 297-323.
[19] Van Gog, T. and Jarodzka, H. 2013. Eye tracking as a tool
[9] Efklides, A. 2006. Metacognition and affect: What can to study and enhance cognitive and metacognitive
metacognitive experiences tell us about the learning processes in computer-based learning environments. In
process? Educational Research Review. 1, 1, 3–14. International Handbook of Metacognition and Learning
[10] Flavell, J.H. 1979. Metacognition and Cognitive Technologies. R. Azevedo and V. Aleven (eds.), Springer
Monitoring A New Area of Cognitive — Developmental New York. 143-156.
Inquiry. American Psychologist. 34, 10, 906–911. [20] Veenman, M.V.J. 2013. Assessing metacognitive skills in
[11] Jacobs, J.E. and Paris, S.G. 1987. Children’s computerized learning environments. In International
metacognition about reading: Issues in definition, Handbook of Metacognition and Learning Technologies.
measurement, and instruction. Educational Psychologist. R. Azevedo and V. Aleven (eds.), Springer New York.
22, 3-4, 255–278. 157-168.
[12] Jacobse, A.M. and Harskamp, E.G. 2012. Towards
efficient measurement of metacognition in mathematical
problem solving. Metacognition and Learning. 7, 2, p.
133-149.