=Paper=
{{Paper
|id=Vol-1736/paper5
|storemode=property
|title=Are you Thinking what I'm Thinking? Representing Metacognition with Question-based Dialogue
|pdfUrl=https://ceur-ws.org/Vol-1736/paper5.pdf
|volume=Vol-1736
|authors=Tracie Farrell Frey,George Gkotsis,Alexander Mikroyannidis
|dblpUrl=https://dblp.org/rec/conf/ectel/FreyGM16
}}
==Are you Thinking what I'm Thinking? Representing Metacognition with Question-based Dialogue==
https://ceur-ws.org/Vol-1736/paper5.pdf
Are you thinking what I’m thinking? Representing
Metacognition with Questionbased Dialogue
Tracie Farrell Frey1 , George Gkotsis2 , and Alexander Mikroyannidis1
1
Knowledge Media Institute, The Open University UK, Walton Hall,
MK7 6AA Milton Keynes, United Kingdom
tracie.farrellfrey@open.ac.uk
alexander.mikroyannidis@open.ac.uk
2
IoPPN, King’s College London, United Kingdom
george.gkotsis@kcl.ac.uk
Abstract. In the following paper, we present Noracle, a tool for creating
representational artefacts of metacognitive thinking in a collaborative, social
environment. The tool uses only questionasking, rather than the typical
question/answer paradigm found in threaded discussions, as a mechanism for
supporting awareness and reflection on metacognitive activity, and for
supporting selfregulated learning. The weblike artefact produced by learner
contributions is intended to support learners in mapping a given domain,
identifying points of convergence and recognizing gaps in the knowledge
representation. In this paper, the authors present the model of the tool, a
usecase scenario and a discussion of the opportunities and limitations related to
this approach.
Keywords: selfregulated learning, reflection, metacognition, learning
analytics, inquiry, knowledge representation, technologyenhanced learning
1 Introduction
The basic metacognitive element of awareness and reflection is selfobservation.
Meaningful selfobservation affords the opportunity for judgement and reaction,
providing evidence of the impact of certain strategies, beliefs and attitudes toward
one's learning [23]. It also requires strong inquiry skills, to ask basic questions like
"what should I observe and how do I best observe it?" toward interpretative questions
such as "why is what I am observing happening and how do I control it?"
Selfobservation seems deceptively easy. If not trained and supported, it can be too
superficial or unstructured to give the individual much insight (ibid). In addition,
though SelfRegulated Learning requires reflection on learning to learn, it is typically
perceived as a more solitary activity occurring outside of the classroom [3].
To support learners in acquiring learning strategy knowledge, we believe it is
necessary to provide tools that allow for 1) social integration of knowledge and
experience about learning, 2) a structured space to explore and represent knowledge,
as well as identify relevant knowledge gaps, and 3) opportunities for reflection and
exchange on how best to address knowledge gaps. In this paper, we present a model
51
Representing Metacognition with Question-based Dialogue - ARTEL16
of a social, structured space for both reflecting on metacognitive assumptions and
representing metacognitive knowledge, using questionbased dialogue. We illustrate
the application of this model at these early stages using a tool called LiteMap [5], and
discuss the possibilities and limitations involved.
Our model, which we refer to as Noracle, is primarily based on the
constructionintegration theory of knowledge acquisition. New knowledge is
integrated into an individual’s conceptual map through reflection, by anchoring it to
existing information [17]. In the context of TechnologyEnhanced Learning, we apply
this model to collecting and integrating strategy knowledge, or metacognition, among
a group of online learners to create a virtual, visual map of inquiries related to their
metacognitive thinking. Through use of questions, rather than answers, we draw on
the traditions of ProblemBased Learning (PBL) and InquiryBased Learning (IBL) to
encourage deeplevel reasoning and support the integration of both cognitive and
metacognitive strategies in learning to learn [8][11]. Noracle is intended to build upon
this tradition, triggering and exploiting human curiosity to support awareness and
reflection. The shared visualization of inquiry that is born through collaboration in
this space is the mechanism by which metacognitive thinking is explicitly
represented, which might not only be “uniquely human”, but also the building block
of contextual knowledge construction [18].
2 Background and Related Work
Inquiry is the cornerstone of all learning. In the next paragraphs, we discuss how
structuring inquiry in a social learning setting can contribute to helping learners
become more aware of how they learn.
Constructivist theory suggests that learners can become more skilled at recognising
certain opportunities and challenges to their learning over time, regulating their
thoughts, emotions, behaviours and learning contexts appropriately [12][24]. These
skills are collectively referred to as SelfRegulated Learning [15][23] and have
become a central goal of contemporary education [19][20]. However, selfregulation
is a process and learners require scaffolding to break through certain challenges. It is
necessary to utilise the social environment of learning to support learners’
selfregulation by exposing them to new perspectives, ideas and methods through
their peers and tutors. In this way, we assert that all selfregulation in learning is
mediated and influenced by what is called SociallyShared Regulated Learning [10].
Social components help to scaffold the process of learning to selfregulate also by
representing and interrogating knowledge within a group. Boud suggested that all
learning originates from the curiosity and motivation of the learner [2].
ProblemBased Learning, InquiryBased Learning, and Collaborative Learning
attempt to trigger this process by providing open, partial pictures of a problem and
relying on students’ collaboration and reasoning to engage students in mapping out
the problem area [7][11][17].
Social Learning approaches necessitate quality learner participation. Research
indicates that learners are generally unskilled in asking deep questions that result in
52
Representing Metacognition with Question-based Dialogue - ARTEL16
highorder thinking processes, such as meaningful reflection [8][9]. Learners also
appear to have difficulty in distilling answers and engaging in cognitive monitoring
[1]. Developing strong skills in questionasking and problemmapping are, therefore,
important precursors to success in reflection on learning. Skills can be strengthened
by association with more highly skilled peers or with a tutor through facilitated
practice [4][8]. Spending a greater portion of time considering learning strategies and
the various implications these strategies have for performance is already a part of both
PBL and IBL [3][11]. However, similar to the acquisition of content knowledge, the
representation of that knowledge is important. Learners need a way of structuring
their strategy knowledge, as well as their selfknowledge, to be able to recognize and
fill in gaps related to how they learn. Noracle is an opportunity to mobilize
technology as both a tool to encourage and represent inquiry.
3 The Noracle Model
In this section we present the main entities of Noracle and discuss their role and
interconnection. Figure 2 illustrates these entities, identified as Classes and
Relationships. Learner is a class that is used to describe the ordinary participants of
Noracle. Apart from a standard set of attributes used to identify them (i.e. username,
email, password), learners are the main agents that interact in the Noracle Space
through various actions, discussed below. A Question is the central Class of Noracle
spaces. Fundamentally, a Question is defined as a freetext field, which is authored by
a Learner. Moreover, a Question can be linked to other Questions so as to form the
web of Questions described below. Once a Question is posed, linking it to other
Questions is optional. A Question linked to another Question joins the space of the
prespecified Noracle Space whereas a Question that is not linked forms a new Space.
Learners can provide feedback on Questions through Annotations and Ratings.
These two entities share the same goal, which is to provide a mechanism for assessing
the usefulness and the quality of a Question. An Annotation is created using a
freetext field and multiple Annotations by an arbitrary number of Learners can be
attached on a Question. For using Noracle in the context of SociallyShared Regulated
Learning, Annotations can be derived from the research literature on SelfRegulated
Learning to indicate whether or not a specific question relates to how the Learner is
thinking, feeling, or behaving, or the context in which learning occurs [15]. An
optional, single Rating is provided by each Learner following a Likert rating scale.
A Moderator is a special type of user who has the permission to make
modifications on the content created in Noracle. The purpose of this user is to be able
to supervise the formation of a Noracle Space and its contents and make sure it
doesn’t deviate from the Noracle objectives and context.
53
Representing Metacognition with Question-based Dialogue - ARTEL16
Figure 2: The Noracle Model
4 Applying Noracle for Metacognitive Representation
To illustrate the concept of Noracle without a functional prototype, we decided to
appropriate a tool for structuring argumentation called LiteMap [5], in which we
bound a small selection of 5 colleagues to deploy only the tools that are representative
of the entities described in the model above to explore challenges in learning to learn.
This included creating a user profile, raising an “Issue” as a Question, providing an
Annotation in the comments, responding with Questions to the Questions of other
Learners, using the “thumbs up/thumbs down” feature as a Rating and exploring the
visualisations of social and issue networks as Space. For the moment, the directional
arrows were ignored, except to illustrate that a connection between two Questions had
been established (see Figure 3). The artefact created is public on LiteMap as “Noracle
Test 1.” While LiteMap is not a perfect representation, we conducted this exercise to
highlight the basic components of the model and the underpinning pedagogical
theories of Noracle.
Noracle intends to train questionasking by demanding that Learners engage only
in questionbased dialogue under supervision and facilitation (of a Moderator, for
example). The starting nodes or Questions that Learners ask are triggered by their
individual curiosity and then expounded upon through the addition of followup
questions (submitted by any user) that help the original asker to expand or narrow
their focus on a particular issue. As the nodes become linked, a web of Questions
emerges that represents the metacognitive reflections of the individuals involved (see
Figure 3). As the web expands, Learners and Moderators can gain insight into what
54
Representing Metacognition with Question-based Dialogue - ARTEL16
the cohort does and does not understand about learning to learn, uncovering gaps in
learner knowledge that can be actioned by an educator (possibly the Moderator).
Through the Rating feature, the Learner can begin to create their own
peerlearning networks by following those users who have proposed the most
highlyrated Questions. The Moderator can also review highlyrated questions with
Learners as part of the classroom content, to improve the quality of their
questionasking by distilling features of useful questions. Additionally, the Moderator
can use this data to improve awareness for the social learning dynamics of the cohort.
Figure 3: Question Web in Noracle
The Annotation feature gives Learners and Moderators additional information
about what type of Question is being asked (whether it relates to thinking, feeling,
behaving or context), to understand where specific challenges might lie. If a particular
Learner consistently asks questions related to a particular area of selfregulation, for
example, this gives Learners and Moderator an indication of the Learner’s interests
and which skills that Learner needs to build, to inform appropriate interventions.
The Annotation feature and visual representation also trigger reflection in other
ways. Suthers discussed this phenomenon in terms of “missing units” triggering
search [21]. Introduction of a gap (i.e. an Annotation field that prompts the user to
think about what kind of question they are asking) encourages learners to consider
how that gap can be filled. In fact, the existence of only Questions in the space has its
own reflexive value in the absence of an Answer entity.
5 Discussion
Noracle as an information system is still at its early development stages and does not
have robust evaluation results at this time. However, we can gain insights about its
effectiveness from the research literature and anecdotal evidence from application of
the model in the physical classroom, as well as the informal LiteMap trial. Noracle
was developed in 2012 by Track2 Facilitation (http://www.track2facilitation.com/) as
a facetoface reflection method (similar to “speeddating” with questions) in the
55
Representing Metacognition with Question-based Dialogue - ARTEL16
context of nonformal learning. Participants have consistently described this method
as being helpful to their process of deliberation and sense of selfesteem in course
evaluations. Experiences with the method tended to confirm prior research findings
that absence of answers leads to selfdiscovery, which is a more satisfying experience
for learners [13]. With facilitation by a moderator, the effects of selfdiscovery on
learning outcomes are even more pronounced [14].
The decision to digitise this tool emerged from the recognition that not all learners
were able to organise and represent what they took away from the experience of
Noracle. They had difficulty remembering who had given them a useful followup
question in the group, for example, and it was difficult to create a joint representation
of complex topics with the limitations of physical space. The "enhancement" that
technology can offer this tool is exactly regarding scale and analytics [8]. The
LiteMap trial indicated that Noracle can be used among an open group of anonymous,
distributed learners, or a closed cohort of students, for example. It can create
representational artefacts that are more considerable and complex than those that
would likely be attempted in a physical classroom, and it can operate in both
synchronous and asynchronous learning environments. Moreover, it can collect data
on users, their contributions and their connections to one another over time.
Representational maps have been shown to resolve some of the issues of
“coherence and convergence” found in typical classroom forums, and they promote
the generation of hypotheses and collaborative activity [22]. This addresses, at least in
part, the issue of motivating learners to ask questions, so that they can become skilled
at other aspects of inquiry [9]. The analytics collected through Noracle can be used in
real time and over time to deliver insights that impact both teaching and learning,
especially in conjunction with a representational artefact. For example, research
indicates that peerlearning in the context of a developmental construct, such as
learning to learn, is more effective than individual study [6]. Being able to estimate
the prior knowledge of a peerlearner has also been shown to produce more positive
impacts learning outcomes [16].
However, Suthers [21] cautioned that representations have their own impacts on
collaborative and individual inquiry. Surely the presence of this artefact limits the
types of discussions that can be had about learning, simply because the tools that are
there to help learners express themselves are limited. Not only do the elements
described in the model limit what can be known from inside of Noracle, but Learners
will additionally produce their own limitations, based on their own perceptions of the
system.
6 Conclusion
Though strategy knowledge is as important as content knowledge in learning, learners
(and teachers) tend to spend much more social, structured time on the perceived
primary task of learning content knowledge and less on the perceived secondary task
of reflection and learning to learn. As a result, many learners are much more aware of
56
Representing Metacognition with Question-based Dialogue - ARTEL16
what they know than why they know it, which frustrates the transfer of learning skills
from one domain to the next. B y scaffolding inquiry in a tool such as Noracle, we
believe that learners can both gain access to new ideas and perspectives on their
learning strategies, and hone their skills in question asking, while contributing to the
representational artefact of metacognitive knowledge created by the group. Over time,
patterns emerge that we believe can provide the learner with insight and give them a
foundation upon which to change or support their current approaches. In the future,
we hope to fully implement this tool, accompanied with preparatory and debriefing
activities that a Moderator can use to facilitate its use. We also intend to conduct a
robust evaluation of the tool and its effects on learner motivation, metacognitive
awareness and general learning outcomes.
References
1. Azevedo, R., & Cromley, J. G. (2004). Does training on selfregulated learning
facilitate students' learning with hypermedia?. Journal of educational psychology,
96(3), 523.
2. Boud, D. (1985). Problembased learning in education for the professions.
Sydney, Australia: Higher Education Research and Development Society of
Australasia.
3. Boud, D., Keogh, R., & Walker, D. (2013). Reflection: Turning experience into
learning. Routledge.
4. Craig, S. D., Sullins, J., Witherspoon, A., & Gholson, B. (2006). The
deeplevelreasoningquestion effect: The role of dialogue and
deeplevelreasoning questions during vicarious learning. Cognition and
Instruction, 24(4), 565591.
5. De Liddo, A., & Buckingham Shum, S. (2016). Collective Intelligence for the
Public Good: New Tools for Crowdsourcing Arguments and Deliberating Online.
Policy, 2014, 2012.
6. De Lisi, R., & Golbeck, S. L. (1999). Implications of Piagetian theory for peer
learning. In A. M. O’Donnell & A. King (Eds.), Cognitive perspectives on peer
learning (pp. 3–37). Mahwah, NJ: Erlbaum.
7. Duch, B. J., Groh, S. E., & Allen, D. E. (2001). The power of problembased
learning: a practical" how to" for teaching undergraduate courses in any
discipline. Stylus Publishing, LLC..
8. Edelson, D. C., Gordin, D. N., & Pea, R. D. (1999). Addressing the challenges of
inquirybased learning through technology and curriculum design. Journal of the
learning sciences, 8(34), 391450.
9. Graesser, A. C., McNamara, D. S., & VanLehn, K. (2005). Scaffolding deep
comprehension strategies through Point&Query, AutoTutor, and iSTART.
Educational psychologist, 40(4), 225234.
10. Hadwin, A., & Oshige, M. (2011). Selfregulation, coregulation, and socially
shared regulation: Exploring perspectives of social in selfregulated learning
theory. Teachers College Record, 113(2), 240264.
57
Representing Metacognition with Question-based Dialogue - ARTEL16
11. HmeloSilver, C. E. (2004). Problembased learning: What and how do students
learn?. Educational psychology review, 16(3), 235266.
12. Jonassen, D. H. (1999). Designing constructivist learning environments.
Instructional design theories and models: A new paradigm of instructional
theory, 2, 215239.
13. Kersh, B. Y. (1962). The motivating effect of learning by directed discovery.
Journal of Educational Psychology, 53(2), 65.
14. Mayer, R. E. (2004). Should there be a threestrikes rule against pure discovery
learning?. American psychologist, 59(1), 14.
15. Pintrich, P. R. (2004). A conceptual framework for assessing motivation and
selfregulated learning in college students. Educational psychology review,16(4),
385407.
16. Sangin, M., Molinari, G., Nüssli, M. A., & Dillenbourg, P. (2011). Facilitating
peer knowledge modeling: Effects of a knowledge awareness tool on
collaborative learning outcomes and processes. Computers in Human Behavior,
27(3), 10591067.
17. Savery, J. R. (2015). Overview of problembased learning: Definitions and
distinctions. Essential readings in problembased learning: Exploring and
extending the legacy of Howard S. Barrows, 515.
18. Shea, N., Boldt, A., Bang, D., Yeung, N., Heyes, C., & Frith, C. D. (2014).
Suprapersonal cognitive control and metacognition. Trends in cognitive
sciences, 18(4), 186193.
19. Siemens, G. (2012). Learning analytics: envisioning a research discipline and a
domain of practice. In: Proceedings of the 2nd International Conference on
Learning Analytics and Knowledge, pp. 48. ACM.
20. Siemens, G., Baker, R. S. (2012). Learning analytics and educational data
mining: towards communication and collaboration. In: Proceedings of the 2nd
international conference on learning analytics and knowledge, pp. 252254.
ACM.
21. Suthers, D. D. (1999, August). The Effects of Representational Bias on
Collaborative Inquiry. In HCI, 1, 362366.
22. Suthers, D. D., & Hundhausen, C. D. (2003). An experimental study of the
effects of representational guidance on collaborative learning processes. The
Journal of the Learning Sciences, 12(2), 183218.
23. Zimmerman, B. J. (1989). A social cognitive view of selfregulated academic
learning. Journal of educational psychology, 81(3), 329.
24. Zimmerman, B. J., Boekarts, M., Pintrich, P. R., & Zeidner, M. (2000). A social
cognitive perspective. Handbook of selfregulation, 13.
25. Zimmerman, B. J., & Schunk, D. H. (Eds.). (2001). Selfregulated learning and
academic achievement: Theoretical perspectives. Routledge.
58