Learning in search can take the form of changes in conceptual knowledge and procedural knowledge at a cognitive level. Searchers may also gain insights into their cognitive process though their metacognitive experiences and thereby acquire metacognitive knowledge which may also influence their search process and outcomes. This paper describes the concepts of metacognition, metacognitive knowledge, metacognitive experiences and metacognitive self-regulation and findings from an experimental information search study. In particular, the study provides evidence of participants' metacognitive experiences and metacognitive regulation activities relating to orienting to task and time, monitoring and steering the process, and evaluating the search process and outcome.
Learning during search is often thought of in terms of a searcher's gain in cognitive knowledge: searchers learn about the concepts in the search or how to do the search. Searchers may also gain insights into their own cognitive processes and use these metacognitive insights to help determine how to proceed in their search. In this paper, I provide an overview of metacognition as well as two major constituent parts: knowledge about cognition and regulation of cognition. I also present evidence from recent research [6,7] using previously unpublished data to illustrate these concepts.
Based on his research on the learning strategies, in 1979, John Flavell observed that "young children are quite limited in their knowledge and cognition about cognitive phenomena, or in their metacognition, and do relatively little monitoring of their own memory, comprehension, and other cognitive enterprises" [9]. He suggested that promising areas for new research include better understanding metacognition and its development as well as cognitive monitoring. This differentiation of knowledge about cognitive phenomena and monitoring of cognition has continued in the literature with other researchers also differentiating between two components of metacognition: one's knowledge of cognition and regulation of cognition [5,11,17].
Knowledge about cognition was described by Flavell [9] as two overlapping concepts: one's metacognitive knowledge and one's metacognitive experiences during a cognitive task. These concepts are closely related. During a cognitive process, an individual's metacognitive experiences are influenced by the metacognitive knowledge that they retrieve from memory, and an individual's metacognitive experiences can, in turn, shape the individual's metacognitive knowledge.
Metacognitive knowledge consists of one's conscious or subconscious beliefs or knowledge about people, tasks and strategies that influence the cognitive process [9]. Metacognitive knowledge includes knowledge about one's own or others' cognitive processes, self-efficacy, motivation or interest; knowledge of general strategies that might be used to achieve one's cognitive goals; or knowledge of the task context such as the task demands or the information available during this task. Metacognitive knowledge can change over time, i.e., people can learn metacognitive knowledge.
Flavell [9] describes a learner's cognitive or affective metacognitive experiences during a cognitive task. Efklides [8] defines metacognitive experiences as "what the person experiences during a cognitive endeavor, be it online metacognitive knowledge, ideas and beliefs, or feelings, goals, judgments. Efklides [9] describes metacognitive experiences in terms of one's judgments or feelings that take place and evolve during the learning process. In particular, metacognitive experiences include a learner's feelings of knowing, difficulty, confidence, satisfaction, and familiarity as well as learner's judgments of learning, solution correctness, and cognitive demands including estimates of the time and effort required to complete a task. An individual's metacognitive knowledge and experiences can also influence an individual's future use of cognitive and metacognitive strategies and processes [9]. Researchers have analyzed executive control processes involved in the regulation of cognition. Brown and Palincsar [5] describe several types of executive functions that one might undertake to regulate their cognitive processes: "planning activities prior to undertaking a problem (predicting outcomes, scheduling strategies, and various forms of vicarious trial and error, etc.), monitoring activities during learning (monitoring, testing, revising, and rescheduling one's strategies for learning) and checking outcomes (evaluating the outcome of any strategic actions against criteria of efficiency and effectiveness)." (p.3, emphasis in original)
A taxonomy of metacognitive activities in text studying and problems solving developed by Meijer, Veenman and van Hout-Wolters [14] contains six types of metacognitive activities: orientating, planning, executing, monitoring, evaluation, and elaboration.
Search as Learning (SAL), July 21, 2016, Pisa, Italy The copyright for this paper remains with its authors. Copying permitted for private and academic purposes.
This research seeks to understand if metacognition is observable in interactive information retrieval studies and whether participants were able to describe their metacognitive experiences and self-regulation activities. Specifically, it asks the following research question: Do searchers report metacognitive experiences and metacognitive self-regulation in search?
In studies of information-seeking and information search, metacognition has been investigated in the information behaviors of adolescents [2], information problem solving of adult experts versus novices [3], and collaborative information seeking [13]. Bowler [2] created a taxonomy of metacognitive knowledge from her longitudinal study of high school student information seeking to support the writing of an essay. She uncovered 13 categories of metacognitive knowledge, including categories relating to knowing about ones' cognitive processes (e.g., knowing what you don't know, knowing your strengths and weaknesses), knowledge of strategies to achieve goals (e.g., building a base of knowledge, communicating with others who serve as information mediators), and knowledge of task characteristics which might help guide selection of strategy (e.g., understanding time and effort, changing course after evaluating one's progress).
Metacognitive regulation is an explicit component of the Information Problem Solving Model [3,4]. The model includes the process and skills needed for information problem solving as well as process regulation activities. These regulation activities include orientation to the information problem, to the task and to time; monitoring and steering the problem solving process by monitoring task performance and planning what to do overall and next actions; and testing including evaluating the process and product during the task and at its completion. In an experimental study using this model as a framework [3], adult experts were found to regulate their process more frequently with significantly more monitoring and steering activities than novices. Experts were also found to monitor time more than novices. Another study found increased levels of regulation activities among students pursuing a PhD or an education degree compared to freshman psychology or secondary education students [4]. When examining individual types of regulation activities, they found that secondary education students conducted fewer orientation activities as well as less monitoring and steering of their process than any of the university students.
Search regulation has also been investigated in collaborative information-seeking. Lazonder [13] examined search regulation and performance in an experimental study investigating information search in groups versus individuals. He found that pairs of searchers were able to more quickly and correctly answer questions and they exhibited more planning and monitoring behaviors: they used significantly more new search strategies (e.g., switching to search from entering in a URL) and were better able to identify relevant information within a page. Success was positively correlated with several monitoring and evaluation measures: identifying relevant information, re-checking the answers, and revising answers.
Researchers have also built and evaluated learning systems with features designed to support metacognition. Stadtler and Bromme [18] built the met.a.ware system to provide a structured notetaking capability with two types of metacognitive prompts: prompts to evaluate the information found (source, author expertise, author bias, and their confidence in information), and prompts to monitor progress (knowledge, comprehension and amount of information remaining). They found greater factual gain for participants using met.a.ware with monitoring or evaluation prompts versus pencil and paper measured comparing pre-and post-search multiple-choice tests. They also found higher comprehension scores as measured by graded essays with met.a.ware with monitoring prompts than pencil and paper.
We took a preliminary look at metacognition in information search in a user study investigating the impact of time pressure and system delays on information search [6,7]. In a lab-based experiment, we manipulated the time available for searchers to complete a search task (5 minutes vs. no limit, between-subjects) as well as the speed with which we presented search results and documents (immediately vs. adding a 5 second delay the first time each SERP and document loaded, within-subjects). 43 participants completed four search tasks in which they searched for and bookmarked 8-12 pages providing a good overview of news coverage for that topic.
After participants searched for each task, they completed a posttask questionnaire in which we asked them questions to better understand the extent to which they engaged in metacognitive task monitoring. Overall, participants reported monitoring how well they were doing on the task (M=4.87, SD=1.53) and how much time they had left on the task (M=4.41, SD=1.8). Participants also reported high levels of monitoring how much information they had found and still needed (M=5.56, SD=1.23) [7]. Higher levels of monitoring task progress and time remaining were predicted if participants were approaching the end of their time limit (see [7] for full analysis and statistical models). This suggests that the presence of a time limit lead to higher metacognitive regulation during search. Interestingly, we also found interactions with time limit and task order: time-limited participants reported more progress monitoring for their second task than their first task. This suggests that participants may have learned how to complete these tasks. In other words, participants' metacognitive evaluation of their performance on the first task may have influenced their planning and their performance on their second task.
Participants also completed an exit questionnaire with several open-ended questions relating to metacognition and a debriefing interview that also served to participants of the experimental manipulations. Analysis of the data from the exit questionnaire was not previously presented and is the focus of this paper.
We analyzed responses to open-ended questions from the exit questionnaire in the study described in the previous section. These questions, shown in Table 1, were designed to provide insight into metacognitive task regulation. They asked about the impact of the time limit on 1) the process they used to complete the task, 2) the amount of information they found, and 3) the extent to which they read the information they found. They were asked the same questions about the impact of system speed.
Did the (system speed/ time you had to complete the tasks) impact… …the process you used to complete the tasks (e.g., steps, thought process)? Please explain. …the amount of information you found? Please explain. …the extent to which you read the information that you found? Please explain.
Participant's exit question responses were analyzed using deductive qualitative content analysis approach. For each sentence or sentence fragment, we coded for mentions of the participant's knowledge of cognition and regulation of cognition. For knowledge of cognition, we coded for participant's mentions of metacognitive knowledge using Flavell's definition [9], and for metacognitive experiences as defined by Flavell [9] and Efklides [8]. For participant's mentions of regulation of cognition, we used the regulation activities described by Brand-Gruwel et al. [3,4]: orientation (to task, time or problem), monitoring and steering, and testing. These definitions are summarized in Table 2.
Knowledge about one's own or others' cognitive processes, self-efficacy, motivation or interest Knowledge of general strategies that might be used to achieve one's cognitive goals Knowledge of types of task and how the contextual factors differ (e.g., task demands or information available)
Feelings of knowing, difficulty, confidence, satisfaction, and familiarity Judgments of learning, solution correctness, and cognitive demands including estimates of the time and effort required to complete a task.
Attention paid to task-specific knowledge, task characteristics or strategies specific to the task.
Orientation to the information problem, to the task and to time
Testing including evaluating the process and product during the task and at its completion
In their open-ended questions, many participants described at least one aspect of metacognition. Below, we describe participant responses relating to metacognitive experiences and their engagement in metacognitive self-regulation activities such as orienting to the imposed task and time, monitoring and steering progress, and evaluating outcomes.
Participants described conscious metacognitive experiences relating to the time they were spending on the task and the affective response (time pressure) that accompanied it. Multiple people described "sensing" or having a feeling that they were running out of time.
"After the piracy topic [first topic], I felt the pressure immediately. It became more acute when I could sense the time slipping away and I still had only 5 or 6 articles." "I felt the most pressure about halfway through the second one, where I had not found enough information yet and had a feeling that I was running out of time. So, I began to skim faster to complete the task."
One participant without task time limits described feeling nervous about the time of the experimental session. "I felt time pressure during the first task when I had to keep searching to find any articles. I was nervous this entire process would take much longer than the expect 1.5 hours."
Participants also described thinking about and orienting themselves towards the task they were to perform. One participant described carefully orienting towards the (imposed) search task at the beginning of the session.
"At the start as I wanted to make sure I fully understood the task so I spent more time reading the prompt and going back again to make sure I knew it well before I started my search" Another participant described setting his/her goal to meet the task demands even though he/she could have found more information.
"…since I only needed to find 8-12 articles with a good overview, I didn't feel the need to spend larger amounts of time searching."
Participants also described differing levels of orienting towards the time they had to complete the task. Two participants noted that they were not focused on time at all until after the end of their first task because they ran out of time.
"I only felt the pressure when I was told that time was up. Other than that I was more focused on the task than the time." "During the first task, I was not paying attention to the time. After that I paid more attention." One participant noted that he/she planned how to allocate time for tasks based using what he/she learned about in the initial tasks.
"During the practice test and first test, I felt some pressure to work quickly. However, once I had a better sense of how the system operated, and the speed with which I got through those tasks, I was less-concerned about the time."
One participant noted needing to adjust after running out of time on the initial task.
"…after the first search, I realized that five minutes doesn't go very far for this task. By the fourth search, I was very preoccupied with time."
Participants described monitoring their task progress and adapting their search strategies during and between tasks based on their previous metacognitive experiences. Two participants described adjusting their task goal based on their (metacognitive) evaluation of their performance in previous tasks.
"One task I did not complete before the task ended. I made the goal of 8 for myself after that and did not try to get 12." "I had plenty of time and thus usually tried to shoot for the max recommended articles of around 12 as opposed to only 8." Participants mentioned skimming articles rather than reading them with a time limit (n=22, 100%) and no time limit (n=13, 65%). "Yes. I only skimmed most of the articles, and only carefully read a few of them."
One participant described his/her metacognitive knowledge about skimming as a strategy and why skimming was the right strategy for this task.
"I can skim quickly to get a general feeling of what the article is about, and then go from there to determine if it is relevant or not. But, in this case, that was all I could do. Reading, on the other hand, takes more time but lends more information directly and can give a better picture of what the article is specifically about."
Multiple participants described how they regulated the search process based on their metacognitive experiences. Several mentioned faster skimming when their time was almost up.
"I felt the most pressure about halfway through the second one, where I had not found enough information yet and had a feeling that I was running out of time. So, I began to skim faster to complete the task."
Participants also described abandoning an unfruitful search strategy by re-querying more quickly.
"When I knew there would be a time limit, I scanned articles more quickly and I changed my search strategy as soon as the article titles began to look less promising." "I had to think much faster than usual to decide whether the information was relevant or not. I also had to try different search queries much more frequently than normal in order to find a combination that worked."
Two participants mentioned increased selectivity in clicking items from search results page as a result of the delays.
"I was more selective in clicking on article because I knew the system would take a little time to show the article consequently more time used."
"the system speed took away a few seconds each time I opened an article, so I found myself skimming more and more in order to find enough relevant articles."
Another described using a more effortful strategy (reading more deeply) if they thought it would be relevant. "I skimmed most… when I wasn't able to skim to ascertain whether it was relevant I read more closely if I really thought (based on the article title) the article would be relevant."
Participants described evaluating their progress or task outcome in terms of the amount or the type of information they found. "…I did not think I had enough different examples of articles (too many articles focusing on China for example)." "I was not certain that I had the breadth of information needed for each task."
Participants also described learning a strategy for these search tasks.
"At the end I already established some rules as how to approach each search so it was easier and I was able to do it faster without feeling as much pressure." "…at the end I had acquired a method of searching the topics and there were steps I followed to do this."
Participants also described the impact on the evaluation of their results.
"Approximately a minute or two into the task, when I had found some articles but didn't want to waste time checking to see the exact number."
Our participants described metacognition in information search; they self-reported conscious metacognition experiences as well as an impact of metacognitive experiences on their regulation of search. We found examples of multiple metacognitive regulation activities. Participants described orienting themselves to both the search task and the time available to do the task. They also described monitoring their search process and steering or adapting their processes as a result. Finally, they described evaluating their search process and task outcomes.
Two particular findings are of particular note. First, higher levels of metacognitive monitoring of task progress and time available in the presence of a time limit and the interaction with time spent on task suggest that participants may have devoted more attentional resources to regulating the task. It is not clear from the results whether the additional regulation activity had an effect on the search outcome; future research will investigate this. Second, the sequence effect from the questionnaire analysis combined with participant explanations indicate that participants refined their search process after the first task suggests that participants acquired procedural knowledge based on their metacognitive experiences and regulation in their earlier tasks. Future research will examine the cognitive and metacognitive strategies and metacognitive regulation in information search using additional methods to gain insight into metacognition and to provide triangulation (e.g., [12]). For example, concurrent thinkaloud has been used to provide insight into metacognitive experiences and regulation activities that take place during a task (e.g., [3,4]). Several questionnaires have been developed and validated which ask students to self-report their awareness of metacognition (e.g., the Metacognitive Awareness Inventory [17]), the use of metacognitive strategies during a course (e.g., the metacognitive strategies within learning strategies subscale of the Motivated Strategies for Learning Questionnaire, [16]), or the use of metacognitive strategies used in a particular domain (e.g., the Text-Learning Strategies Inventory, [15]). Eye-tracking (e.g., [19]) and logged computer interactions (e.g., [1,20]) have also been used. Future work could also examine the efficacy of metacognitive prompts or scaffolding in information search (e.g., [1,18]).
Thanks to study participants and my collaborators on this study: Diane Kelly and Leif Azzopardi.