Metacognitive Knowledge and Metacognitive Regulation in Time-Constrained in Information Search Anita Crescenzi School of Information and Library Science, University of North Carolina Chapel Hill, NC, USA amcc@email.unc.edu ABSTRACT metacognitive experiences are influenced by the metacognitive Learning in search can take the form of changes in conceptual knowledge that they retrieve from memory, and an individual’s knowledge and procedural knowledge at a cognitive level. metacognitive experiences can, in turn, shape the individual’s Searchers may also gain insights into their cognitive process metacognitive knowledge. though their metacognitive experiences and thereby acquire Metacognitive knowledge consists of one’s conscious or metacognitive knowledge which may also influence their search subconscious beliefs or knowledge about people, tasks and process and outcomes. This paper describes the concepts of strategies that influence the cognitive process [9]. Metacognitive metacognition, metacognitive knowledge, metacognitive knowledge includes knowledge about one’s own or others’ experiences and metacognitive self-regulation and findings from cognitive processes, self-efficacy, motivation or interest; an experimental information search study. In particular, the study knowledge of general strategies that might be used to achieve provides evidence of participants’ metacognitive experiences and one’s cognitive goals; or knowledge of the task context such as metacognitive regulation activities relating to orienting to task and the task demands or the information available during this task. time, monitoring and steering the process, and evaluating the Metacognitive knowledge can change over time, i.e., people can search process and outcome. learn metacognitive knowledge. Keywords Flavell [9] describes a learner’s cognitive or affective Information search; metacognition; metacognitive self-regulation metacognitive experiences during a cognitive task. Efklides [8] defines metacognitive experiences as “what the person 1. INTRODUCTION experiences during a cognitive endeavor, be it online Learning during search is often thought of in terms of a searcher’s metacognitive knowledge, ideas and beliefs, or feelings, goals, gain in cognitive knowledge: searchers learn about the concepts in judgments. Efklides [9] describes metacognitive experiences in the search or how to do the search. Searchers may also gain terms of one’s judgments or feelings that take place and evolve insights into their own cognitive processes and use these during the learning process. In particular, metacognitive metacognitive insights to help determine how to proceed in their experiences include a learner’s feelings of knowing, difficulty, search. In this paper, I provide an overview of metacognition as confidence, satisfaction, and familiarity as well as learner’s well as two major constituent parts: knowledge about cognition judgments of learning, solution correctness, and cognitive and regulation of cognition. I also present evidence from recent demands including estimates of the time and effort required to research [6, 7] using previously unpublished data to illustrate complete a task. these concepts. An individual’s metacognitive knowledge and experiences can Based on his research on the learning strategies, in 1979, John also influence an individual’s future use of cognitive and Flavell observed that “young children are quite limited in their metacognitive strategies and processes [9]. Researchers have knowledge and cognition about cognitive phenomena, or in their analyzed executive control processes involved in the regulation of metacognition, and do relatively little monitoring of their own cognition. Brown and Palincsar [5] describe several types of memory, comprehension, and other cognitive enterprises” [9]. He executive functions that one might undertake to regulate their suggested that promising areas for new research include better cognitive processes: understanding metacognition and its development as well as “planning activities prior to undertaking a problem cognitive monitoring. This differentiation of knowledge about cognitive phenomena and monitoring of cognition has continued (predicting outcomes, scheduling strategies, and various in the literature with other researchers also differentiating between forms of vicarious trial and error, etc.), monitoring activities two components of metacognition: one’s knowledge of cognition during learning (monitoring, testing, revising, and re- and regulation of cognition [5, 11, 17]. scheduling one's strategies for learning) and checking outcomes (evaluating the outcome of any strategic actions Knowledge about cognition was described by Flavell [9] as two against criteria of efficiency and effectiveness).” (p.3, overlapping concepts: one’s metacognitive knowledge and one’s emphasis in original) metacognitive experiences during a cognitive task. These concepts A taxonomy of metacognitive activities in text studying and are closely related. During a cognitive process, an individual’s problems solving developed by Meijer, Veenman and van Hout- Wolters [14] contains six types of metacognitive activities: Search as Learning (SAL), July 21, 2016, Pisa, Italy orientating, planning, executing, monitoring, evaluation, and The copyright for this paper remains with its authors. Copying permitted for private and academic purposes. elaboration. This research seeks to understand if metacognition is observable amount of information remaining). They found greater factual in interactive information retrieval studies and whether gain for participants using met.a.ware with monitoring or participants were able to describe their metacognitive experiences evaluation prompts versus pencil and paper measured comparing and self-regulation activities. Specifically, it asks the following pre-and post-search multiple-choice tests. They also found higher research question: Do searchers report metacognitive experiences comprehension scores as measured by graded essays with and metacognitive self-regulation in search? met.a.ware with monitoring prompts than pencil and paper. 2. BACKGROUND We took a preliminary look at metacognition in information In studies of information-seeking and information search, search in a user study investigating the impact of time pressure metacognition has been investigated in the information behaviors and system delays on information search [6, 7]. In a lab-based of adolescents [2], information problem solving of adult experts experiment, we manipulated the time available for searchers to versus novices [3], and collaborative information seeking [13]. complete a search task (5 minutes vs. no limit, between-subjects) Bowler [2] created a taxonomy of metacognitive knowledge from as well as the speed with which we presented search results and her longitudinal study of high school student information seeking documents (immediately vs. adding a 5 second delay the first time to support the writing of an essay. She uncovered 13 categories of each SERP and document loaded, within-subjects). 43 participants metacognitive knowledge, including categories relating to completed four search tasks in which they searched for and knowing about ones’ cognitive processes (e.g., knowing what you bookmarked 8-12 pages providing a good overview of news don’t know, knowing your strengths and weaknesses), knowledge coverage for that topic. of strategies to achieve goals (e.g., building a base of knowledge, After participants searched for each task, they completed a post- communicating with others who serve as information mediators), task questionnaire in which we asked them questions to better and knowledge of task characteristics which might help guide understand the extent to which they engaged in metacognitive task selection of strategy (e.g., understanding time and effort, changing monitoring. Overall, participants reported monitoring how well course after evaluating one’s progress). they were doing on the task (M=4.87, SD=1.53) and how much Metacognitive regulation is an explicit component of the time they had left on the task (M=4.41, SD=1.8). Participants also Information Problem Solving Model [3, 4]. The model includes reported high levels of monitoring how much information they the process and skills needed for information problem solving as had found and still needed (M=5.56, SD=1.23) [7]. Higher levels well as process regulation activities. These regulation activities of monitoring task progress and time remaining were predicted if include orientation to the information problem, to the task and to participants were approaching the end of their time limit (see [7] time; monitoring and steering the problem solving process by for full analysis and statistical models). This suggests that the monitoring task performance and planning what to do overall and presence of a time limit lead to higher metacognitive regulation next actions; and testing including evaluating the process and during search. Interestingly, we also found interactions with time product during the task and at its completion. In an experimental limit and task order: time-limited participants reported more study using this model as a framework [3], adult experts were progress monitoring for their second task than their first task. This found to regulate their process more frequently with significantly suggests that participants may have learned how to complete these more monitoring and steering activities than novices. Experts tasks. In other words, participants’ metacognitive evaluation of were also found to monitor time more than novices. Another study their performance on the first task may have influenced their found increased levels of regulation activities among students planning and their performance on their second task. pursuing a PhD or an education degree compared to freshman Participants also completed an exit questionnaire with several psychology or secondary education students [4]. When examining open-ended questions relating to metacognition and a debriefing individual types of regulation activities, they found that secondary interview that also served to participants of the experimental education students conducted fewer orientation activities as well manipulations. Analysis of the data from the exit questionnaire as less monitoring and steering of their process than any of the was not previously presented and is the focus of this paper. university students. Search regulation has also been investigated in collaborative 3. METHOD information-seeking. Lazonder [13] examined search regulation We analyzed responses to open-ended questions from the exit and performance in an experimental study investigating questionnaire in the study described in the previous section. These information search in groups versus individuals. He found that questions, shown in Table 1, were designed to provide insight into pairs of searchers were able to more quickly and correctly answer metacognitive task regulation. They asked about the impact of the questions and they exhibited more planning and monitoring time limit on 1) the process they used to complete the task, 2) the behaviors: they used significantly more new search strategies amount of information they found, and 3) the extent to which they (e.g., switching to search from entering in a URL) and were better read the information they found. They were asked the same able to identify relevant information within a page. Success was questions about the impact of system speed. positively correlated with several monitoring and evaluation Table 1. Open-ended questions relating to metacognition measures: identifying relevant information, re-checking the answers, and revising answers. Did the (system speed/ time you had to complete the tasks) impact… Researchers have also built and evaluated learning systems with features designed to support metacognition. Stadtler and Bromme …the process you used to complete the tasks (e.g., steps, thought process)? Please explain. [18] built the met.a.ware system to provide a structured notetaking capability with two types of metacognitive prompts: …the amount of information you found? Please explain. prompts to evaluate the information found (source, author …the extent to which you read the information that you found? expertise, author bias, and their confidence in information), and Please explain. prompts to monitor progress (knowledge, comprehension and Participant’s exit question responses were analyzed using feeling that I was running out of time. So, I began to skim deductive qualitative content analysis approach. For each sentence faster to complete the task.” or sentence fragment, we coded for mentions of the participant’s One participant without task time limits described feeling nervous knowledge of cognition and regulation of cognition. For about the time of the experimental session. knowledge of cognition, we coded for participant’s mentions of metacognitive knowledge using Flavell’s definition [9], and for “I felt time pressure during the first task when I had to keep metacognitive experiences as defined by Flavell [9] and Efklides searching to find any articles. I was nervous this entire process [8]. For participant’s mentions of regulation of cognition, we used would take much longer than the expect 1.5 hours.” the regulation activities described by Brand-Gruwel et al. [3, 4]: 4.2 Metacognitive regulation: Orienting to orientation (to task, time or problem), monitoring and steering, and testing. These definitions are summarized in Table 2. task or time Participants also described thinking about and orienting themselves towards the task they were to perform. One participant Table 2. Categories used in coding of participant responses described carefully orienting towards the (imposed) search task at Knowledge of cognition the beginning of the session. Metacognitive knowledge [9] “At the start as I wanted to make sure I fully understood the task so I spent more time reading the prompt and going back Knowledge about one’s own or others’ cognitive processes, again to make sure I knew it well before I started my search” self-efficacy, motivation or interest Another participant described setting his/her goal to meet the task Knowledge of general strategies that might be used to achieve demands even though he/she could have found more information. one’s cognitive goals “…since I only needed to find 8-12 articles with a good Knowledge of types of task and how the contextual factors overview, I didn't feel the need to spend larger amounts of differ (e.g., task demands or information available) time searching.” Metacognitive experiences [8, 9] Participants also described differing levels of orienting towards Feelings of knowing, difficulty, confidence, satisfaction, and the time they had to complete the task. Two participants noted that familiarity they were not focused on time at all until after the end of their first task because they ran out of time. Judgments of learning, solution correctness, and cognitive demands including estimates of the time and effort required to “I only felt the pressure when I was told that time was up. complete a task. Other than that I was more focused on the task than the time.” Attention paid to task-specific knowledge, task characteristics “During the first task, I was not paying attention to the time. or strategies specific to the task. After that I paid more attention.” Metacognitive regulation [3, 4] One participant noted that he/she planned how to allocate time for tasks based using what he/she learned about in the initial tasks. Orientation to the information problem, to the task and to time “During the practice test and first test, I felt some pressure to Monitoring and steering the problem solving process by work quickly. However, once I had a better sense of how the monitoring task performance and planning what to do overall and system operated, and the speed with which I got through those next actions tasks, I was less-concerned about the time.” Testing including evaluating the process and product during the One participant noted needing to adjust after running out of time task and at its completion on the initial task. “…after the first search, I realized that five minutes doesn't 4. RESULTS go very far for this task. By the fourth search, I was very In their open-ended questions, many participants described at least preoccupied with time.” one aspect of metacognition. Below, we describe participant 4.3 Metacognitive regulation: Monitoring responses relating to metacognitive experiences and their engagement in metacognitive self-regulation activities such as progress and steering process orienting to the imposed task and time, monitoring and steering Participants described monitoring their task progress and adapting progress, and evaluating outcomes. their search strategies during and between tasks based on their previous metacognitive experiences. Two participants described 4.1 Metacognitive experience adjusting their task goal based on their (metacognitive) evaluation Participants described conscious metacognitive experiences of their performance in previous tasks. relating to the time they were spending on the task and the “One task I did not complete before the task ended. I made the affective response (time pressure) that accompanied it. Multiple goal of 8 for myself after that and did not try to get 12.” people described “sensing” or having a feeling that they were running out of time. “I had plenty of time and thus usually tried to shoot for the max recommended articles of around 12 as opposed to only “After the piracy topic [first topic], I felt the pressure 8.” immediately. It became more acute when I could sense the time slipping away and I still had only 5 or 6 articles.” Participants mentioned skimming articles rather than reading them with a time limit (n=22, 100%) and no time limit (n=13, 65%). “I felt the most pressure about halfway through the second one, where I had not found enough information yet and had a “Yes. I only skimmed most of the articles, and only carefully read a few of them.” One participant described his/her metacognitive knowledge about “Approximately a minute or two into the task, when I had skimming as a strategy and why skimming was the right strategy found some articles but didn't want to waste time checking to for this task. see the exact number.” “I can skim quickly to get a general feeling of what the article 5. CONCLUSIONS is about, and then go from there to determine if it is relevant Our participants described metacognition in information search; or not. But, in this case, that was all I could do. Reading, on they self-reported conscious metacognition experiences as well as the other hand, takes more time but lends more information an impact of metacognitive experiences on their regulation of directly and can give a better picture of what the article is search. We found examples of multiple metacognitive regulation specifically about.” activities. Participants described orienting themselves to both the Multiple participants described how they regulated the search search task and the time available to do the task. They also process based on their metacognitive experiences. Several described monitoring their search process and steering or adapting mentioned faster skimming when their time was almost up. their processes as a result. Finally, they described evaluating their search process and task outcomes. “I felt the most pressure about halfway through the second one, where I had not found enough information yet and had a Two particular findings are of particular note. First, higher levels feeling that I was running out of time. So, I began to skim of metacognitive monitoring of task progress and time available in faster to complete the task.” the presence of a time limit and the interaction with time spent on Participants also described abandoning an unfruitful search task suggest that participants may have devoted more attentional strategy by re-querying more quickly. resources to regulating the task. It is not clear from the results whether the additional regulation activity had an effect on the “When I knew there would be a time limit, I scanned articles search outcome; future research will investigate this. Second, the more quickly and I changed my search strategy as soon as the sequence effect from the questionnaire analysis combined with article titles began to look less promising.” participant explanations indicate that participants refined their “I had to think much faster than usual to decide whether the search process after the first task suggests that participants information was relevant or not. I also had to try different acquired procedural knowledge based on their metacognitive search queries much more frequently than normal in order to experiences and regulation in their earlier tasks. find a combination that worked.” Future research will examine the cognitive and metacognitive Two participants mentioned increased selectivity in clicking items strategies and metacognitive regulation in information search from search results page as a result of the delays. using additional methods to gain insight into metacognition and to “I was more selective in clicking on article because I knew the provide triangulation (e.g., [12]). For example, concurrent think- system would take a little time to show the article aloud has been used to provide insight into metacognitive consequently more time used.” experiences and regulation activities that take place during a task (e.g., [3, 4]). Several questionnaires have been developed and “the system speed took away a few seconds each time I validated which ask students to self-report their awareness of opened an article, so I found myself skimming more and more metacognition (e.g., the Metacognitive Awareness Inventory in order to find enough relevant articles.” [17]), the use of metacognitive strategies during a course (e.g., the Another described using a more effortful strategy (reading more metacognitive strategies within learning strategies subscale of the deeply) if they thought it would be relevant. Motivated Strategies for Learning Questionnaire, [16]), or the use “I skimmed most… when I wasn't able to skim to ascertain of metacognitive strategies used in a particular domain (e.g., the whether it was relevant I read more closely if I really thought Text-Learning Strategies Inventory, [15]). Eye-tracking (e.g., (based on the article title) the article would be relevant.” [19]) and logged computer interactions (e.g., [1, 20]) have also been used. Future work could also examine the efficacy of 4.4 Metacognitive regulation: Testing or metacognitive prompts or scaffolding in information search (e.g., evaluating [1, 18]). Participants described evaluating their progress or task outcome in 6. ACKNOWLEDGMENTS terms of the amount or the type of information they found. Thanks to study participants and my collaborators on this study: “…I did not think I had enough different examples of articles Diane Kelly and Leif Azzopardi. (too many articles focusing on China for example).” “I was not certain that I had the breadth of information needed 7. REFERENCES for each task.” [1] Bannert, M., Sonnenberg, C., Mengelkamp, C. and Pieger, E. 2015. Short- and long-term effects of students’ self- Participants also described learning a strategy for these search directed metacognitive prompts on navigation behavior tasks. and learning performance. Computers in Human “At the end I already established some rules as how to Behavior. 52, Nov. 2015, 293-306. approach each search so it was easier and I was able to do it [2] Bowler, L. 2010. A taxonomy of adolescent metacognitive faster without feeling as much pressure.” knowledge during the information search process. Library “…at the end I had acquired a method of searching the topics & Information Science Research. 32, 1, 27–42. and there were steps I followed to do this.” [3] Brand-Gruwel, S., Wopereis, I. and Vermetten, Y. 2005. Participants also described the impact on the evaluation of their Information problem solving by experts and novices: results. Analysis of a complex cognitive skill. Computers in Human Behavior. 21, 3, 487–508. [4] Brand-Gruwel, S., Wopereis, I. and Walraven, A. 2009. A [13] Lazonder, A.W. 2005. Do two heads search better than descriptive model of information problem solving while one? Effects of student collaboration on web search using internet. Computers & Human Behavior. 53, 4, behaviour and search outcomes. British Journal of 1207-1217. Educational Technology. 36, 3, 465–475. [5] Brown, A.L. and Palincsar, A.S. 1982. Inducing strategic [14] Meijer, J., Veenman, M.V.J. and van Hout-Wolters, learning from texts by means of informed, self-control B.H.A.M. 2006. Metacognitive activities in text-studying training. Center for the Study of Reading, Technical and problem-solving: Development of a taxonomy. Report No. 262. University of Illinois at Urbana- Educational Research and Evaluation. 12, 3, 209–237. Champaign, Center for the Study of Reading, Champaign [15] Merchie, E., Van Keer, H., and Vandevelde, S. 2014. IL. Development of the Text-Learning Strategies Inventory: [6] Crescenzi, A., Kelly, D. and Azzopardi, L. 2015. Time Assessing and profiling learning from texts in fifth and Pressure and System Delays in Information Search. sixth grade. Journal of Psychoeducational Assessment. 32, Proceedings of the 38th International ACM SIGIR 6, 533-547. Conference on Research and Development in Information [16] Pintrich, P. R., and De Groot, E. V. 1990. Motivational Retrieval (SIGIR '15). ACM, New York, NY, USA, 767- and self-regulated learning components of classroom 770. academic performance. Journal of Educational [7] Crescenzi, A., Kelly, D. and Azzopardi, L. 2016. Impacts Psychology. 82, 1, 33-40. of Time Constraints and System Delays on User [17] Schraw, G. and Dennison, R.S. 1994. Assessing Experience. In Proceedings of the 2016 ACM on Metacognitive Awareness. Contemporary Educational Conference on Human Information Interaction and Psychology. 19, 4, 460–475. Retrieval, CHIIR 2016, ACM. New York. 141-150/ [18] Stadtler, M. and Bromme, R. 2008. Effects of the [8] Efklides, A. 2001. Metacognitive experiences in problem metacognitive computer-tool met.a.ware on the web search solving. In Trends and Prospects in Motivation Research. of laypersons. Computers in Human Behavior. 24, 3, 716– Efklides, A., Kuhl, J. and Sorrentino, R. M. (eds). Springer 737. Netherlands. Dordrecht. 297-323. [19] Van Gog, T. and Jarodzka, H. 2013. Eye tracking as a tool [9] Efklides, A. 2006. Metacognition and affect: What can to study and enhance cognitive and metacognitive metacognitive experiences tell us about the learning processes in computer-based learning environments. In process? Educational Research Review. 1, 1, 3–14. International Handbook of Metacognition and Learning [10] Flavell, J.H. 1979. Metacognition and Cognitive Technologies. R. Azevedo and V. Aleven (eds.), Springer Monitoring A New Area of Cognitive — Developmental New York. 143-156. Inquiry. American Psychologist. 34, 10, 906–911. [20] Veenman, M.V.J. 2013. Assessing metacognitive skills in [11] Jacobs, J.E. and Paris, S.G. 1987. Children’s computerized learning environments. In International metacognition about reading: Issues in definition, Handbook of Metacognition and Learning Technologies. measurement, and instruction. Educational Psychologist. R. Azevedo and V. Aleven (eds.), Springer New York. 22, 3-4, 255–278. 157-168. [12] Jacobse, A.M. and Harskamp, E.G. 2012. Towards efficient measurement of metacognition in mathematical problem solving. Metacognition and Learning. 7, 2, p. 133-149.