=Paper=
{{Paper
|id=Vol-1103/artel13
|storemode=property
|title=None
|pdfUrl=https://ceur-ws.org/Vol-1103/artel13.pdf
|volume=Vol-1103
}}
==None==
https://ceur-ws.org/Vol-1103/artel13.pdf
Please refer to these proceedings as
Milos Kravcik, Birgit Krogstie, Adam Moore, Viktoria Pammer, Lucia Pan-
nese, Michael Prilla, Wolfgang Reinhardt & Thomas Daniel Ullmann (Eds.):
Proceedings of the 3rd Workshop on Awareness and Reflection in Technol-
ogy Enhanced Learning. In conjunction with the 8th European Conference
on Technology Enhanced Learning: Scaling up learning for sustained im-
pact. Paphos, Cyprus, September 17, 2013. Available online at http:
//ceur-ws.org/.
c 2013 for the individual papers by the papers’ authors. Copying permitted for private
and academic purposes. Re-publication of material from this volume requires permission
by the copyright owners.
The front-cover was created by Harriett Cornish (The Open University, KMi).
Addresses of the editors:
Thomas Daniel Ullmann Wolfgang Reinhardt
Knowledge Media Institute Computer Science Education Group
The Open University Department of Computer Science
Walton Hall University of Paderborn
Milton Keynes Fürstenallee 11
MK7 6AA 33102 Paderborn
United Kingdom Germany
t.ullmann@open.ac.uk wolle@upb.de
Viktoria Pammer Adam Moore
Knowledge Technologies Institute Knowledge and Data Engineering Group
Graz University of Technology School of Computer Science and Statistics
Inffeldgasse 21A Trinity College
8010 Graz Dublin, D2
Austria Ireland
viktoria.pammer@tugraz.at adam.more@cs.tcd.ie
Michael Prilla Milos Kravcik
Institute for Applied Work Science RWTH Aachen University
Ruhr University of Bochum Advanced Community Information Systems (ACIS)
Universitaetsstr. 150 Ahornstr. 55
44780 Bochum 52056 Aachen
Germany Germany
michael.prilla@rub.de kravcik@dbis.rwth-aachen.de
Birgit Krogstie Lucia Pannese
Department of Computer and Information Science imaginary srl
Norwegian University of Science and Technology Innovation Network Politecnico di Milano
Sem Sælands vei 7-9 Via Mauro Macchi, 50
7491 Trondheim 20124 Milano
Norway Italy
birgit@idi.ntnu.no lucia.pannese@i-maginary.it
Awareness and Reflection in Technology Enhanced Learning
Awareness and reflection are viewed differently across the disciplines informing Technology-
Enhanced Learning (CSCW, psychology, educational sciences, computer science...). The
ARTEL workshop series brings together researchers and professionals from different back-
grounds to provide a forum for discussing the multi-faceted area of awareness and reflec-
tion.
The theme of the 2013 workshop was:
How can awareness and reflection support learning in different settings (work,
education, continuing professional development, lifelong learning, etc.). What
are the roles that technology can play to support awareness and reflection in
these contexts?
This theme was covered in several topics of the workshop. The main interests were about
the theoretical discussion of awareness and reflection in TEL and related concepts (e.g.,
collaborative learning, creativity techniques, experiential learning, etc.). The methodolo-
gies to identify, study and analyse awareness and reflection in the context of (technology-
enhanced) learning. Besides theory and methodology the workshop informed about em-
pirical studies about technology support for awareness and reflection. A special aim was
to showcase technology (design, application, evaluation) supporting awareness and reflec-
tion. Here, the central question was how awareness and reflection technologies can help
to enhance the learning experience, by researching learner’s awareness of social context,
knowledge, artefacts and processes, and awareness and reflection in specific contexts, such
as higher education, work-integrated learning, learning networks, etc.
The workshop included a paper session, demo and prototype slam as well as interactive
sessions. The workshop provided a forum for presenting and discussing research on aware-
ness and reflection in TEL, and created an interactive experience that connects partici-
pants’ research, current tools or latest prototypes and models with real end users’ learning
experiences and requirements regarding reflection technology. Researchers and practition-
ers came together to work on the future agenda of ARTEL research and development (see
the topic map below).
Papers and Demos on Awareness and Reflection
The workshop received 11 papers from which eight submissions were selected for the
workshop after the review process. The submissions included in the proceedings are as
follows:
The paper ”Linking Reflective Learning and Knowledge Maturing in Organizations” by
Krogstie, Schmidt, Kunzmann, Krogstie, and Mora links knowledge maturing and reflec-
tive learning in order to better understand support actions for reflection in the workplace.
Guided by three propositions they illustrate the application of their theoretical framework
with two empirical studies in the area of care homes. Based on this, they outline implica-
3
Papers and Demos on Awareness and Reflection
tions for the design of reflection tools.
Ullmann, Wild, and Scott present in their paper ”Reflection - quantifying a rare good”
an approach to determine how rare occurrence of reflections in writings are. Based on
forum posts of online courses, a crowdsourcing approach was chosen to annotate sen-
tences regarding several elements of reflections. These there then analysed to describe
their frequency in texts. With this approach the intuition that reflections are rare in writ-
ings received empirical support.
The paper ”Support for Collaborative Reflection in Healthcare: Comparing two Work-
places” by Prilla and Degeling describes the Talk Reflection App as socio-technical sup-
port for collaborative reflection. The paper makes the case for collaborative reflection and
distinguishes between individual reflection and collaborative reflection (CR). Moreover,
the authors present the evaluation of the Talk Reflection app in two medical cases and
outline strengths and shortcomings of their approach.
Krogstie, Krogstie, and Prilla paper on ”Modeling computer-supported reflective learning:
Combining a high-level timeline view with reflection cycles and tool use” describes the
current state of the Computer Supported Reflective Learning (CSRL) model developed in
the MIRROR FP7 project. They introduce patterns to describe the reflective process. The
evaluation of the model informs the refinements of the model, its notation, as well as usage
instructions.
The paper of Charleer, Klerkx, Santos, and Duval ”Improving awareness and reflection
through collaborative, interactive visualizations of badges” describes the ”Navi Badge-
board”, a tool used on an interactive table to get an overview of the badges people have
received during their learning. The tool aims at generating awareness of the goals and
tasks required for a successful completion of a course.
The paper ”Feeler: feel good and learn better. A tool for promoting reflection about learn-
ing and well-being” by Durall and Toikkanen describes a combination of visualisation
of learning performance in relation to individual well being to achieve a better learning
progress. Their goal is to develop a tool, which spans approaches of the the quantified
self community and of the field of learning analytics aiming at technological support for
individual and collective reflection-after-action processes.
”SpirOnto: Semantically Enhanced Patient Records for Reflective Learning on Spiritual
Care in Palliative Care” by Kunzmann, Roser, Schmidt, and Stiehl describes an approach
in creating, using, and extending an ontology to support (reflective) learning on spiritual
care. Their idea is to enrich documentation of interaction with patients on e.g. palliative
wards with concepts from an ontology including elements of spiritual care for different
cultural and other backgrounds in order to help workers improve their caring skills.
”REFLECT: Community-Driven Scaffolding for Voice-enabled Reflection on the Go”
by Schmidt, Kunzmann, Attwell, Chan, Heinemann-Grüder, Hughes, Lan, Vratny, and
Heberle describes a mobile app called REFLECT, which aims at supporting GPs (General
Practitioners) in their day-to-day reflection via voice-based questions.
4
Future Challenges of Awareness and Reflection
Future Challenges of Awareness and Reflection
A central element of the workshop was to collaborate and to discuss the future challenges
of awareness and reflection research for technology-enhanced learning. The outcome of
this discussion is captured in the following mind map.
Figure 1: Future challenges of awareness and reflection
A version, which can be commented, can be found here: Future Challenge Map of Aware-
ness and Reflection in Technology-Enhanced Learning.
Awareness and Reflection Workshop Series
The official workshop webpage can be found at http://teleurope.eu/artel13
The 3rd Workshop on Awareness and Reflection in Technology-Enhanced Learning (AR-
TEL 2013) is part of a successful series of previous workshops.
• 2nd Workshop on Awareness and Reflection in Technology-Enhanced Learning (AR-
5
Awareness and Reflection Workshop Series
TEL12). Workshop homepage: http://www.teleurope.eu/artel12. Pro-
ceedings: http://ceur-ws.org/Vol-931/.
• 1st European Workshop on Awareness and Reflection in Learning Networks (AR-
Nets11). Workshop homepage: http://teleurope.eu/arnets11. Pro-
ceedings: http://ceur-ws.org/Vol-790/
• Augmenting the Learning Experience with Collaboratice Reflection (ALECR11).
Workshop homepage: http://www.i-maginary.it/ectel2011/index.
html
• 1st Workshop on Awareness and Reflection in Personal Learning Environments
(ARPLE11). Workshop homepage: http://teleurope.eu/arple11. Pro-
ceedings: http://journal.webscience.org/view/events/The_PLE_
Conference_2011/paper.html#group_Proceedings_of_the_1st_
Workshop_on_Awareness_and_Reflection_in_Personal_Learning_
Environments
As with the last year’s workshops, Twitter was used as a back channel before, during
and after the workshop. The vivid Twitter conversation around the hashtag #artel13 is
summarised in the following wordcloud.
Figure 2: Twitter wordcloud of the ARTEL13 workshop
To stay updated about future events, to share your research, or simple to participate with
other researchers, consider joining the Awareness and Reflection in Technology-Enhanced
6
Awareness and Reflection Workshop Series
Learning group.
http://teleurope.eu/artel
We want to use this opportunity to thank the authors for their contributions and the program
committee for their support and reviewing activity.
November 2013 Milos Kravcik, Birgit Krogstie
Adam Moore, Viktoria Pammer
Lucia Pannese, Michael Prilla
Wolfgang Reinhardt, Thomas Daniel Ullmann
7
Organization Committee
Organization Committee
Milos Kravcik, RWTH Aachen University, Germany
Birgit Krogstie, Norwegian University of Science and Technology, Norway
Adam Moore, Trinity College Dublin, Ireland
Viktoria Pammer, Knowledge Technologies Institute, TU Graz, Austria
Lucia Pannese, imaginary, Italy
Michael Prilla, University of Bochum, Germany
Wolfgang Reinhardt, WALLMEDIEN AG / European Association of Technology En-
hanced Learning, Germany
Thomas Ullmann, The Open University, United Kingdom
8
Program Committee
Program Committee
John Cook, University of the West of England, United Kingdom.
Jon Dron, Athabasca University, Canada.
Angela Fessl, Know-Center, Austria.
Peter Kraker, Know-Center, Austria.
Mart Laanpere, Tallinn University, Estonia.
Alexander Mikroyannidis, The Open University, United Kingdom.
Alexander Nussbaumer, University of Graz, Austria.
Eileen O’Donnell, Trinity College Dublin, Ireland.
Andreas Schmidt, Karlsruhe University of Applied Sciences, Germany.
Bettina Renner, Knowledge Media Research Center, Germany.
Fridolin Wild, The Open University, United Kingdom.
9
Supporting FP7 Projects
Supporting FP7 Projects
http://www.mirror-project.eu
http://www.imreal-project.eu
http://learning-layers.eu
http://www.tellme-ip.eu/
http://wespot-project.eu
10
Contents
Awareness and Reflection in Technology Enhanced Learning 3
Papers and Demos on Awareness and Reflection . . . . . . . . . . . . . . . . . 3
Future Challenges of Awareness and Reflection . . . . . . . . . . . . . . . . . 5
Awareness and Reflection Workshop Series . . . . . . . . . . . . . . . . . . . 5
Organization Committee . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Program Committee . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Supporting FP7 Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Linking reflective learning and knowledge maturing in organizations
Birgit R. Krogstie, Andreas P. Schmidt, Christine Kunzmann, John Krogstie, Si-
mone Mora 13
Reflection - quantifying a rare good
Thomas Daniel Ullmann, Fridolin Wild, Peter Scott 29
Support for collaborative reflection in healthcare: comparing two workplaces
Michael Prilla, Martin Degeling 41
Modeling computer-supported reflective learning: Combining a high-level time-
line view with reflection cycles and tool use
Birgit R. Krogstie, John Krogstie, Michael Prilla 53
Improving awareness and reflection through collaborative, interactive visualiza-
tions of badges
Sven Charleer, Joris Klerkx, Jose Luis Santos, Erik Duval 69
Feeler: feel good and learn better. A tool for promoting reflection about learning
and well-being
Eva Durall, Tarmo Toikkanen 83
SpirOnto: Semantically enhanced patient records for reflective learning on spiri-
tual care in palliative care
Christine Kunzmann, Traugott Roser, Andreas P. Schmidt, Tanja Stiehl 91
REFLECT: Community-driven scaffolding for voice-enabled reflection on the go
Andreas P. Schmidt, Christine Kunzmann, Graham Attwell, Elizabeth Chan, Mar-
ius Heinemann-Grüder, Jenny Hughes, Wenlin Lan, Andreas Vratny, Andreas Heberle 97
11
12
Linking Reflective Learning and Knowledge Maturing
in Organizations
Birgit R. Krogstie1, Andreas P. Schmidt2, Christine Kunzmann3, John Krogstie1,
Simone Mora1
1
Norwegian University of Science and Technology (NTNU), Norway
{birgitkr|john.krogstie|simonem@idi.ntnu.no}
2
Karlsruhe University of Applied Sciences, Germany
{andreas_peter.schmidt@hs-karlsruhe.de}
3
Pontydysgu, UK
{kontakt@christine-kunzmann.de}
Abstract: Reflection is a key activity for learning in organizations. While
technology support for reflection on the individual and collaborative level
is promising, it remains challenging to embed these learning activities into
the organization. To better understand and support reflection in the work-
place, it is important to see the mutual dependencies between reflective
learning activities and knowledge maturing. In this paper, we seek to bridge
the gap by presenting a conceptual model linking reflection and knowledge
maturing. Based on the model we put forward three propositions: In reflec-
tive learning, expertise moderates knowledge maturing, discrepancies be-
tween knowledge elements trigger reflection, and the maturity of
knowledge used in reflection influences the reflection process. We use
findings from empirical studies in two care homes to support the proposi-
tions. We address implications for the design of technology enhanced re-
flection support by discussing a prototype reflection tool for care homes.
1 Introduction
Reflection is a key learning activity for organizations, but generally not very well
performed or supported [1]. Reflection allows organizations to implement continuous
improvement with double loop or deutero learning [2]. Critical and collaborative re-
flection is necessary because organizations have to operate in complex situations of
change, with multiple stakeholders and interests [3]. Reflection is currently mostly
approached from an individual or collaborative perspective, focusing on individual
participants and the micro-level of learning. How reflection is embedded into the
organization and contributes to the organization’s goals remains rather challenging.
Knowledge maturing [4] describes knowledge development within and across or-
ganizations from a macro perspective. It concentrates on the evolution of knowledge
from early ideas to standardization. The framework bridges bottom-up, individual and
group driven knowledge processes with top-down organizational perspectives and is a
good candidate for exploring the connections between reflection and organizational
knowledge development.
The goal of this paper is linking the micro-perspective of reflective learning with a
macro-perspective that embeds reflective learning processes into the organization. We
13
Linking reflective learning and knowledge maturing in organizations - ARTEL13
present an integrated model of reflective learning and knowledge maturing. We out-
line theoretical perspectives (section 2) and present a conceptual model and three
propositions (section 3). We next present two cases (section 4) and use them to illus-
trate how our key propositions give insight about the cases (section 5) with implica-
tions for technology support (section 6). We conclude in section 7.
2 Background
In this section we provide a background for our research contribution, addressing
existing work on reflective learning and knowledge development in organizations.
We identify a gap in current research with regard to how these processes are connect-
ed and argue that research on knowledge maturing can be used to fill the gap.
Reflective learning can be considered as the conscious reevaluation of experience
for the purpose of guiding future behavior, acknowledging the need to attend to feel-
ings, ideas and behavior [5]. The essential role of experience and reflection in learn-
ing has long been recognized [6, 7]. In the workplace, work and reflection on work
feed into each other in reflective learning [8, 9].
Reflection can be individual, but can also be a collective activity [10] involving the
articulation and sharing of experiences, and collaborative knowledge construction
(e.g. [11]), and also involves transitions between levels in the organization [12]. Re-
flection on work can be considered as interconnected learning cycles in which work
experience is reconstructed and re-evaluated in reflection sessions (individual or col-
laborative) and the outcomes are fed back into work. Reflection cycles differ in their
characteristics, with implications for what types of tool support may be adequate [13].
Whereas individuals and groups may reflect to reconstruct and elaborate experi-
ences and thereby contribute to work-related problem-solving, an organizational per-
spective on reflection sees reflection as a way of addressing organizational matters
and the implementation of structures and collective action: reflection is a collective
capacity to question assumptions [10]. Key differences between the organizational,
collective and individual perspectives on reflection include the type of reflection con-
tents, access to the contents, and the language used [10]. These differences are related
to the degree to which the knowledge involved is explicit, shared and formalized.
To understand reflection at work, it is necessary to see the bottom-up and the top-
down perspectives in combination. The structures and collective actions implemented
from an organizational perspective have to support reflective processes as seen from
an individual and collaborative perspective [10]. Supporting reflection from the or-
ganizational perspective means creating the opportunities for employees to question
current assumptions and knowledge. There is a trend for increasing decentralization
of problem-solving in enterprises, which implies a need for active reflection processes
challenging and confronting existing knowledge [3]. Whereas a standard process
might be automated, more knowledge-intensive processes are hard to pre-define, and
they emerge and change in unforeseen ways due to the knowledge developed
throughout the process. Certain work-processes are in addition emotionally intensive,
thus parts of the process needs to be adapted in the interaction between people there
and then. Processes can ’move’ through increased process maturity (as a process is
better understood, and thus possible to formalize to a larger degree) or through break-
14
Linking reflective learning and knowledge maturing in organizations - ARTEL13
down in the underlying assumptions behind a formalized process, leading to a worka-
round. Workarounds can have positive effects (e.g. triggering reflection to instill im-
provement in the processes) or negative effects (on compliance, security and safety).
To understand and support reflection, then, taking into account the organizational
perspective as well as that of individuals and groups, it would be helpful to explore in
more detail the connection between reflection and knowledge in an organization. The
reflection cycle model in [13] does not explicitly address the role of knowledge in the
process. Research addressing the use and development of knowledge through individ-
ual and collaborative learning [11, 14] is not directed at reflection in the workplace.
An adaptation of the model of Stahl has been developed [15] to show how reflection
acts as a catalyst for organizational learning on a general level. What remain open are
the more detailed connections between reflection and knowledge development, as
well as implications for the design of technology support.
Nonaka and Takeuchi’s theory on organisational knowledge creation [16] link
knowledge to human activity. Central to their theory is that organisational knowledge
is created through a continuous dialog between tacit and explicit knowledge per-
formed by organisational “communities of interaction” that contribute to the amplifi-
cation and development of new knowledge. They also identify four patterns of inter-
action between tacit and explicit knowledge commonly called modes of knowledge
conversion: Socialisation (creating tacit knowledge from existing tacit knowledge
through shared experience), Externalisation (conversion from tacit (or unstated ex-
plicit) to explicit knowledge), Combination (creation of new explicit knowledge from
existing explicit knowledge), and Internalisation (conversion of explicit knowledge to
tacit knowledge)
The internalisation mode of knowledge creation is closely related to “learning by
doing”; hence the internalisation process is deeply related to action. When tacit and
explicit knowledge interacts, innovation emerges. Nonaka proposes that the interac-
tion is shaped by shifts between modes of knowledge conversion. Adding Nonaka and
Takeuchi’s ontological dimension of knowledge creation, we end up with the ideal-
ized spiral of organisational knowledge creation, which shows how the organisation
can mobilise tacit knowledge created and accumulated at the individual level, organi-
sationally amplified through the four modes of knowledge conversion and crystallised
at higher ontological levels. Thus the authors propose that the interaction between
tacit and explicit knowledge becomes larger in scale as the knowledge creation pro-
cess proceeds up their ontological levels. The spiral process of knowledge creation
starts at the individual level and potentially moves upwards through expanding inter-
action communities crossing sectional, departmental, divisional and possibly organi-
sational boundaries. Note that it is not given that we want all knowledge to move to
the organizational level
Knowledge maturing is a perspective on knowledge development that aims at
bringing together the manifold forms of knowledge inside organizations. Following
Nonaka’s knowledge spiral [16] the knowledge maturing perspective describes
knowledge development as a process that can be structured into discrete phases, each
of which have different characteristics. Knowledge development starts with explora-
tion (Ia) and appropriation (Ib) on an individual level, referring to the emergence of
new ideas. After the distribution in communities (II), knowledge gets transformed
15
Linking reflective learning and knowledge maturing in organizations - ARTEL13
(III) for further outreach, and in phase IV, it enters the organization’s scope with ad-
hoc training (for a more instructional path) or piloting (for a more experimental path,
e.g., for process knowledge). In phase V, knowledge first gets institutionalized within
the company, and finally it moves to external standardization.
Fig. 1. Knowledge maturing phases and its characteristics (from [17])
Knowledge maturing connects the characteristics of each of the phases to forms of
learning and to characteristics of design tools, and shows what has to be accomplished
for a transition. This leads to the insight that learning in early phases is more appro-
priate for those who have high level of expertise in the relevant area, while mature
knowledge allows interaction with those considered novices.
Knowledge maturing not only considers the knowledge level, but also the artifacts
that represent knowledge, such as notes, and documents, but also process models, or
tags or taxonomy terms. While similar development phases can be identified in arti-
facts, e.g., associated with their formality, the relationship between knowledge and
artifacts is more complex: artifacts can use a level of formality that is not appropriate
for the knowledge maturity it represents, which has been found to be a common prob-
lem in enterprise information and knowledge management.
Furthermore, knowledge maturing also identifies activities that contribute [1]. One
of those activities is “reflect on and refine work practices”. Looking at work process
knowledge, one first attempt to analyze the connection between knowledge maturing
and reflection has been made in [18] based on an empirical study in a hospital, and it
was found that the maturity of process knowledge influence the quality of reflection.
Knowledge maturing is a promising perspective on knowledge development to bet-
ter link reflection to organizational knowledge development. Particularly the identifi-
cation of different characteristics of knowledge at different stages of maturity, based
on [15, 18], are important starting points for a further integration. However, to pro-
vide guidance for design of reflection tools we need to look further at the interplay
between the individual and the collective level, which we will address in the follow-
ing section.
3 A Model Connecting Knowledge Maturing and Reflection
In the previous section, we have identified that knowledge maturing appears to be a
promising perspective to linking reflection with organizational knowledge develop-
ment. For informing technology design for supporting reflection, we need to have a
closer look at how elements of a reflection process are influencing and are influenced
16
Linking reflective learning and knowledge maturing in organizations - ARTEL13
by knowledge maturing. We concentrate here on the interplay between the individual
and the collective level. Towards that end, we have created a model by digesting the
theoretical findings as outlined in the previous section and iteratively refining them
with empirical findings that will be used in the following section.
As depicted in Figure 2, we have introduced the two main levels: the individual
level and the level of collective knowledge. On the individual level, it is useful to
distinguish between experience, expertise, and background motivation. An important
basis for reflection is the set of individual experiences, an experience being “the total
response of a person to a situation, including behavior, ideas and feelings” [5]; reflec-
tion addressing single experiences as well as conglomerates of experiences [19].
Individual knowledge develops from an aggregation of these experiences, but
experiences clearly go beyond knowledge as they capture the context in which they
have been made. Individual knowledge and experiences in turn, form part of exper-
tise. This is in line with [20] who identifies knowledge, experience (as in how long /
how many times one has been doing something), and problem solving as fundamental
components [20]. Background motivation finally captures that reflection on the indi-
vidual level is not only connected to expertise, but has an underlying complexity of
goals/motives of the individual. This is particularly important as early phases of
knowledge maturing are driven by individual motives [17].
On the collective side, knowledge maturing adopted a pragmatic view that collec-
tive knowledge is an abstraction of individual knowledge of the members of the col-
lective [21].The relationship is not a simple sum. Individual knowledge can exist
without turning into collective knowledge, if it does not become effective on the col-
lective level (e.g., knowledge related to private activities). On the other hand, collec-
tive knowledge always depends on the learning of individuals.
Artifacts are manifestations, touchable or visible items, either in physical or elec-
tronic form [17]. They are important to communicate knowledge and to construct
new knowledge. Their relationship to knowledge is not an easy one: they can repre-
sent knowledge (both on the individual and the collective level). But while the notion
of knowledge maturing would suggest that artifacts that represent more mature
knowledge is also more formalized, less mature knowledge can be over-formalized
(e.g., formal process models of not well understood processes), while mature
knowledge might lack an adequate representation.
To characterize reflection, the reflection session as a time-limited activity, planned
or spontaneous, individual or collaborative has been introduced [13]. Reflection ses-
sions are often connected to each other, which is in line with the basic assumption of
interconnected learning activities in the knowledge maturing perspective. An im-
portant characteristic of a reflection session is the object of reflection, i.e. ”what is the
reflection about”. This object can be on different levels of abstraction [22] and is
usually connected to some knowledge element in focus. In the reflection session,
individual work experiences, other relevant knowledge and artifacts, to which we
refer as the background of reflection, are used to (re)construct and re-evaluate the
object. The Outcome of reflection involves a change in individual and/or collective
knowledge and artifacts. Not each advancement on the individual level leads to ad-
vancement on the collective level, and collaborative reflection may lead to differing
outcomes for individual participants. Triggering of reflection happens when people
17
Linking reflective learning and knowledge maturing in organizations - ARTEL13
perceive a discrepancy [15]. In the workplace, these triggers may have various rea-
sons, such as needs for sense making and problem solving related to work tasks [23].
Based on these basic conceptualizations, three main propositions have been de-
rived based on a theory-driven analysis of real-world examples, which characterize
the relationship between reflection and knowledge maturing:
Proposition 1) Expertise moderates knowledge maturing through reflection.
Following the observation from knowledge maturing that expertise has a major influ-
ence on individuals’ capabilities to interact with knowledge of different characteris-
tics, we can also find different approaches to reflection between novices and experts
in a field. Novices tend to take collective knowledge for granted, and reflect on their
understanding and internalisation of the collective knowledge, while more experi-
enced individuals also challenge existing collective knowledge. Particularly in collab-
orative reflection sessions, it is important to consider that novices will bring in unfil-
tered and less interpreted experiences, but will need more experienced individuals to
actually bring about the development of collective knowledge.
Proposition 2) The maturity of knowledge used in reflection moderates the re-
flection process. Generalizing the findings from [18], the maturity of knowledge that
is the object of reflection (and to a lesser degree also the maturity of background
knowledge), influences the reflection process. On a general level, more mature
knowledge appears to have more authority and legitimacy [4]; on a more specific
level characteristics of maturity phases such as changeability vs. stability, opening up
vs. filtering, or interest-driven vs. goal-driven influence how individual experiences
are related to the knowledge, and how easy knowledge is further developed. The same
experiences relating to immature knowledge may lead to advancing the knowledge,
while they might get rejected if interpreted with respect to more mature knowledge.
Proposition 3) Discrepancies between knowledge elements trigger reflection and
thereby affect knowledge maturing. As identified, discrepancies between the actual
and the expected are a major trigger for reflection. Our integrated model provides
more specific insights in how discrepancies between different knowledge elements
(i.e., collective knowledge, artifacts, individual experiences, knowledge, or expertise)
might be exploited to trigger reflection:
Discrepancy between collective and individual knowledge and/or experiences can
lead to challenging the collective knowledge and thus developing it towards a higher
degree of maturity. It can also lead to individual learning from collective knowledge,
and to increased knowledge in appropriating collective knowledge.
Discrepancy between collective knowledge and artifacts include over-
formalization, when artifacts are presented to be more mature than the knowledge
they represent, particularly if associated with requirements for compliance. This can
lead to challenging the knowledge/artifact, workarounds, and to re-developing the
artifact. Discrepancy between collective knowledge and artifacts can also result from
under-formalization, e.g. when information is not appropriately recorded. This can
lead to re-development of the artifact. Also the characteristics of interaction can differ
from the characteristics of knowledge, e.g., when knowledge is in a phase where
18
Linking reflective learning and knowledge maturing in organizations - ARTEL13
changeability and openness prevail, but the interaction possibilities restrict this, e.g.,
through access rights and lack of possibilities for contribution.
Fig. 2. A model connecting knowledge and reflection
In the next section we present two cases from care homes. This is a type of work-
place in which reflective practice is relatively established and recognized as im-
portant, giving good opportunities to collect data on the interplay of reflection and
knowledge maturing. In Section 5 we use the cases to illustrate the relevance of our
three propositions.
4 Cases: Work and reflective learning in two care homes
In this section we present two cases from care homes: Case 1 (The Rose Garden) and
Case 2 (The Community Care Home). The homes are located in different European
countries. The residents of the homes are generally elderly people, many suffering
from dementia. Our focus in both of the cases is the work of the carers. This section
presents the research approach and gives some general context about the two cases.
4.1 The Rose Garden Care Home (Case 1)
We conducted an exploratory case study addressing work and reflection among carers
at the Rose Garden, a small-sized, private residential care home. We observed over
19
Linking reflective learning and knowledge maturing in organizations - ARTEL13
two days, mainly in the lounges in which the residents spend most of their day receiv-
ing care from a team of carers. We conducted brief interviews and talked informally
with different categories of staff (including owner, manager, and nurses) when possi-
ble. We collected data on the (at the time largely paper based) information infrastruc-
ture. Photography and note taking were restricted due to residents’ privacy.
Three in-depth interviews (each 60-90 minutes) with carers (C1, C2 and C3) about
their work and workplace learning are the main source of data from the case. C1, who
had been working in the home for three months, is in her early 20s. C2, having
worked in the Rose Garden for about a year, is in her mid 20s. She is a team leader in
the home and an educated nurse from her home country, waiting for accreditation to
be a nurse also in her current country. C3 is a senior carer in her mid 40s, with many
years of experience from care work. The three carers were selected to cover different
degrees of experience, and was the largest group that could be taken out of their daily
work for interviews. The interviews took place in the lunchroom, largely undisturbed,
and were audio recorded and fully transcribed.
A prior study exploring the data from the care home from a different research per-
spective indicated that the relationship between carers’ reflection and the develop-
ment of knowledge in the organization were important to the reflective learning. This
along with the theoretical work presented in Section 3 guided a detailed data analysis
of the carer interviews with a focus on key aspects from the model in Fig. 2. The
analysis confirmed the relevance of the propositions presented in section 3; we select-
ed an illustrative set of examples in section 5. In what follows we provide a brief
description of relevant aspects of the work and learning practices at the Rose Garden,
to give a context for the examples:
In the country of this study, there are no requirements for formal qualifications to
start working as a carer. The majority of the care staff at the Rose Garden is young
and inexperienced (e.g. C1), and the turnover is high.
Carers and residents spend much of their time in daily lounges. Medical supervi-
sion and administration of drugs are done by a nurse. Medical status information
about residents is documented by the carers in various charts/reports. One of the car-
ers on the team is team leader (e.g. C2) responsible for coordination and reporting.
There is one senior carer (C3) in the home, acknowledged for her expertise. She is
consulted by less experienced carers on care work, and by management to help e.g. in
changing a care plan or take care of a difficult conversation with relatives of a resi-
dent.
The carers often have to handle challenging situations with residents. This re-
quires knowledge about the resident (e.g. life history, prior interests) and general
knowledge about care work. The Rose Garden follows a philosophy of person-
centered care: Focus is the person with her life history, interests and integrity [24] To
convey the care principles internally to staff and externally in marketing, a model of
person-centered care is used. A diagram of the model is found on the wall in areas
frequently visited by staff and visitors. The model includes a representation of the
main psychological needs of people with dementia (see description in Section 6),
connected to the process of care work, with personhood as a core element. The model
is generally focused on care principles, not detailed procedures.
Doing care work in the lounge, the carers talk aloud. They explain to the residents
what is happening and frequently ask their opinion or consent, in line with person-
20
Linking reflective learning and knowledge maturing in organizations - ARTEL13
centered care. The carers also update each other, explaining about episodes that they
have experienced or been told about. The work in the lounge facilitates coordination
that enables the carers to help each other out. The information sharing also helps the
carers develop individual and shared knowledge of the residents.
Care work is mainly learnt by doing and from observing peers. The first week as a
carer is spent following a more senior carer. After that the newbie starts doing care
work. On-the-job training and reflective learning are strongly encouraged, the owner
being actively involved in staff training. In the daily morning meeting with the care
staff, the manager goes through the status of all residents, and then typically facili-
tates a general session on care work, linking recent events and current issues to the
more general care principles pushing phase V knowledge (see fig. 1) to the attention
and repeating key points from the bi-weekly lecture. The carers participate with an-
swers, questions, and comments, e.g. on their individual experiences. The information
shared in the morning meeting provides a rationale for action, e.g. for carers to handle
immediate issues or for the manager to change care plans.
4.2 The community care home (Case 2)
Following the study in the Rose Garden care home, we have conducted an exploratory
interview with a senior elderly care nurse in a community care home operated by a
non-profit association. It houses about 75 residents. It is not specialized on any type
of residents.
The researcher had prior contact to the care home so that the interview, which last-
ed for about one hour and was conducted outside the care home, could be focused on
aspects of reflection and knowledge maturing. A second researcher took notes.
The care home is characterized by stability, both in terms of residents and staff.
Some of the residents have stayed for about 20 years, and the last new full-time staff
member started three years ago. Staff is composed of a mixture of certified elderly
care nurses and care worker with little training in addition to on-the-job training.
However, the care home is currently undergoing a change of its processes, facili-
tated by an external consultant, to better accommodate to legal requirements, which
favor a documentation-centric quality management for control. These changes partic-
ularly address an insufficient level of documentation of care activities and improve-
ment processes, while care quality as such is not perceived as an issue.
5 Illustrating our Propositions with Findings from the Cases
In this section we illustrate the relevance of the propositions presented in Section 3 by
referring to examples from Case 1 and Case 2. The examples originate in the inter-
views and observations. For the sake of illustrating our points we have occasionally
combined elements into a more coherent narrative than what is found in the raw data.
21
Linking reflective learning and knowledge maturing in organizations - ARTEL13
5.1 Prop. 1: Expertise Moderates Knowledge Maturing Through Reflection
Case 1 illustrates how carers at different levels of expertise use organizational
knowledge at a high level of maturity, e.g. the care principles outlined in the model of
personalized care, in different ways when reflecting on experiences from care work.
The new carer C1 says: “If we work every day and do not use [care model], we
would be like a robot, not being happy about the job, not working from the
heart.”One day when C1 interacts with a resident in the lounge she finds that he re-
acts unexpectedly. She tries to do the things that normally make him at ease. Howev-
er, nothing works; the resident is obviously still in a distress. The carer is confused.
(As she later explains in the interview: “I know the resident, I know when something
is not right.”). She asks the other carers in the room, who explain about a change in
the medical condition of the resident which might have had some effects on his be-
havior. One of the other carers explains about how she approached a similar situation
with another resident the month before, and that it worked well. The explanation from
her colleagues makes it easier for C1 to understand the behavior of the resident, and
she starts approaching him in a different way. This works better. In the following
morning meeting, the status of the resident is discussed. The manager explains about
his condition. A carer illustrates the challenges with an episode from the day before.
The manager suggests a couple of measures that should be taken in the interaction
with the resident. Doing so she reminds about the importance of respecting his per-
sonhood and asking his consent, as described in the care model. The young carer C1
now feels she understands even better the residents’ reactions the day before.
C1 is learning the care principles by heart and keeps them in a note in her pocket,
to be able to prepare for possible questions from the manager or a care home inspec-
tor and demonstrate legitimacy as a carer. Key questions for her in the use of the prin-
ciples include what principles to apply in a specific situation, and how. In specific
situations, the general principles are clearly not sufficient, as seen in the example with
the resident with the inexplicable behavior. When C1 acquired the resident-specific
knowledge about the medical condition, she was able to re-apply the general principle
of ‘working from the heart’ in her actions towards the resident, perhaps having slight-
ly extended her individual knowledge of what it may mean.
C3, the senior carer, says about using the care model in practice: “[it is] about the
residents, how you treat them, their dignity, how you can give some activities for
them, give them some choice. [] Respect, privacy, dignity for everything.”
C3 explains that she makes tradeoffs in her work, to “do the right thing” under re-
source constraints. C3 says that the management is aware that tradeoffs and minor
workarounds have to be made in daily work. She thinks carers should learn from each
other by “[picking] the good things!”. Whether a particular action is good or bad is a
result of considering several, possibly conflicting concerns. When the manager in-
volves C3 in work to update care plans, C3 sometimes disagrees with the manager’s
suggestions, arguing about the resident’s personhood: “The person comes first”.
22
Linking reflective learning and knowledge maturing in organizations - ARTEL13
5.2 Prop. 2: The Maturity of Knowledge Used in Reflection Moderates the
Reflection Process
In Case 2, it was recurring in the interview that the relevance of documentation to
deliver good care and its role in everyday work is a frequent object of reflection. The
documentation is largely prescribed by legal requirements. While employees are de-
termined to deliver the best care possible within time and resource constraints, the
artifacts they need to comply with are not perceived as best practice, even though they
are presented as level V knowledge. This triggers reflection both on the individual
and team levels. The reason why the knowledge is not recognized as best practice
could be (i) that the artifact is unclear about the knowledge it contains, (ii) a mismatch
between artifact and knowledge (i.e. exemplifying Proposition 3), or (iii) that the
maturity of the knowledge is disputed, meaning that making it prescriptive and un-
changeable creates conflict. The interview indicates a combination of (i) and (ii): the
documentation serves two purposes - one is compliance, which requires much docu-
mentation, the other is improving care, for which meetings are more useful. The ex-
ample shows that a high level of formality of artifacts can lead to recurring reflection
sessions which center around the artifacts and their purpose without advancing the
knowledge the artifacts should represent. Mixing maturity levels can turn reflection
unproductive. It may be that the actual knowledge maturity phase indicates changea-
bility (see Fig. 1) which is not allowed by the artifacts and their associated prescrip-
tion. This can be linked to evidence from MATURE [1] that a high level of formality
of artifacts (e.g. processes) constitutes a barrier to further development of knowledge.
A contrasting example, in which mature knowledge serves its purpose, can be
found in Case 1, in which the care principles play a central role in work and reflec-
tion. Institutionalized throughout the Rose Garden, they serve multiple purposes:
Providing practical guidance when a carer reflects on her daily work, principles (not
procedures) giving some flexibility and encouraging the consideration of a situation
from multiple angles. The principles can be connected to concrete examples and indi-
vidual experiences. The examples, some of which are formalized, help carers under-
stand the principles, and the principles are used to make sense of examples, especially
the carers’ own, individual experiences. Second, the care principles - used externally
vs. authorities and being a clear example of phase V knowledge - legitimize decisions
(e.g. to make a change to a care plan). Finally, the principles are used to strengthen
the feeling of purpose, motivation and togetherness, e.g. giving meaning to work.
5.3 Prop. 3: Discrepancies Between Knowledge Elements Trigger Reflection
and thereby Affect Knowledge Maturing
C3, being an expert, has a lot of individual knowledge enabling her to “read” chal-
lenging situations with residents. She is typically able to immediately see from a situ-
ation what are the essential elements that will affect how the situation evolves, and
what tradeoffs will be involved. The discrepancies of knowledge elements may or
may not lead her to reflect and develop her knowledge. It is however clear from the
interview that to her, an additional concern is not only the residents, but the learning
and well-being of her younger colleagues. The discrepancy between displayed indi-
23
Linking reflective learning and knowledge maturing in organizations - ARTEL13
vidual knowledge to handle certain situations and the collective knowledge needed to
do so, is a trigger for her reflection on how to bring younger carers up to the desired
level of competence. The learning resulting from C3’s guidance of younger carers
adds to the organization’s ability to train its employees.
C3 furthermore contributes to the development of formalized knowledge in the or-
ganization through involvement in the updating of care plans (artifacts). The need for
change in a plan is rooted in a discrepancy between the current state of a resident and
the effects of continuing to implement the existing care plan. The representation of
the state of the resident and the current care plan largely consist of collective
knowledge, and often medical expertise plays a key role in the decisions. Still, exper-
tise with care work helps combine and weigh the parts of the often complex picture of
‘the whole person’ in order to create a good plan.
C2 sees that there is often conflict between the ideals of the care model and re-
sources, e.g. time. ”Yes, it is, it is there. The model is fine [] but if for example I have
to ask people if they want a shower or not, some will not give consent even if they are
really dirty or soaking wet”. Goal conflicts (e.g. what priorities have to be made) are
often discussed in morning meetings.”We discuss solutions, alternatives to organize
the work. Even says to keep up quality of work, if a person is not washed
in the morning before breakfast we can do it after lunch”. C2 has individually over
time been reflecting on the goal conflicts:”We can ask [the residents] many times,
offer a cup of tea and ask again later, but at the end of the day we have to force it.
Being soaking wet on a chair, they will get a sore bottom []. But still, if we are forc-
ing them, we are just considering that we are doing good for the residents. [] But
always, when there are inspections [from the authorities] they will look if we are
involving the residents in the care. ”. C2 reflects on this in our interview:”If we just
have to stick on to theory, the practice is definitely going to be poor. If we are just
sticking on to the practical point of view, some or other day the theory is going to
be…” C2 has been talking to the owner about these issues on several occasions and
thinks that management understands: ”They do understand []. I just told him, see: []
If somebody will not drink we have to send them to the hospital.”
The difference between C2 and C3 is not just one of expertise in care work. C2,
with her nursing background, has more vocabulary to discuss certain aspects of care
with the owner, e.g. referring to medical issues and not just the care model. C2 both
has the needs of a carer to handle discrepancies between knowledge elements (e.g.
need to be on schedule; need to have the residents’ consent) and the ability as a carer
and nurse to confidently verbalize these concerns, both with management and for
collaborative reflection e.g. in the morning meeting. Again we see a link between
expertise and the way knowledge is questioned, shared and potentially developed.
The case of C2 may illustrate the advantage for knowledge development as an em-
ployee can take different perspectives, e. g. by being assigned to two or more roles.
6 Implications for Design
In this section we use an existing tool prototype to illustrate how the insights from
this paper can guide the direction of tool design. For that purpose, we use the Flower
24
Linking reflective learning and knowledge maturing in organizations - ARTEL13
Power (FP) app, which is a simple tool for note taking on work experiences in a care
home applying the care principles presented in Case 1. The tool links note taking to
the psychological needs of the resident (represented as a flower in the model) that
must be met in the person-centered care. The app runs offline on an iPad intended to
be available in the lounge in which the daily care work takes place. The carers can
take reflection notes and examine existing notes whenever they have the time, and
one device is to be shared among the carers in the lounge. Every user can read other
users’ notes. The manager can also read the notes, e.g. to prepare discussions.
The main screen in the FP app shows a key part of the care model – a flower with a
petal for each key psychological need of a person [24]. Clicking the question mark on
a petal opens a window with text explaining this aspect of care in the same way as in
the book and consistently with how it is explained in lectures and morning meetings.
If a user clicks anywhere else on a petal, the list of notes associated with that petal is
shown. Existing notes may be opened, read, and commented. From the same screen,
the user may also create a new note. The header of a note says what care principle is
involved. When the user has written the note and saves it, the note will be added to
the list of notes for that petal. On the main screen, the number of notes indicated in
the respective petal will be incremented. A note can be associated with several petals,
if the user thinks the respective care aspects are all relevant to the note.
Considering the three ways in which knowledge maturing impacts on reflection, as
identified in this paper (see Section 3), we will briefly outline the directions that could
be taken in further developing the FP app to take this into account.
We have seen that expertise moderates knowledge maturing through reflection
(Proposition 1) in the care home. An improvement of the FP app based on this insight
could go in the direction of allowing users to choose between two levels designed to
differentiate the needs of carers at different levels of expertise in care work. One lev-
el, more intended for new carers, may be directed at supporting the learning of care
principles and the linking of work experience to specific principles. The other level
may support the linking of experiences to discrepancies between principles (i.e. ma-
ture knowledge in the organization) and experience, and/or between principles.
We have learnt that the maturity of knowledge used in reflection moderates the re-
flection process (Proposition 2) among carers. Accordingly the tool could support the
promotion of notes to a higher level of maturity, e.g. by categorizing notes into two or
more levels. A second level might reflect that the contents represent knowledge rec-
ognized on a community level. A possible third level could be notes taken into ac-
count in organizational development. Importantly, promotion to a higher level of
maturity would not imply just sharing the note, but also transforming it through pro-
cesses involving reflection, e.g. individually by the manager or collaboratively in a
team meeting. In doing so, the origin could be preserved by keeping links to the orig-
inal notes at a lower level of maturity to facilitate understanding of the rationale.
Linking to the original experiences and ideas is simultaneously a way of acknowledg-
ing the role of contributors in the process, making the impact explicit. A promotion of
knowledge to a higher level of maturity is likely to change the object of reflection
towards the more abstract/general. However, the form of a note could still be a narra-
tive of a single experience, if it is recognized as fit for conveying the message in an
effective/interesting way, i.e. serving as a representative example.
25
Linking reflective learning and knowledge maturing in organizations - ARTEL13
Finally, in the care home, we found discrepancies between knowledge elements
triggering reflection and thereby affecting knowledge maturing (Proposition 3). To
support this in the tool, knowledge-related reflection triggers could be made explicit
by supporting the linking of notes to artifacts, knowledge and experience perceived to
be in conflict. The linking could help carers, as they write reflection notes, make
sense of the discrepancies. Describing discrepancies in the shared, organization-wide
language provided by the care principles may create greater awareness of discrepan-
cies as the notes are read by colleagues and possibly trigger their reflection. Referring
explicitly to discrepancies also supports the transformation to higher levels of maturi-
ty by linking work experiences to more general patterns (e.g. “a X vs. Y conflict”)
which might be used to structure the higher-level notes.
The discrepancies seen in Case 1 involve concerns that are not explicit in the flow-
er model. This indicates a need for the app to support explicit referencing to other
care principles applied in the carers’ reflection on work, for instance personhood,
which is also a key part of the model of personalized care applied in the home. The
set of concepts to which examples and discrepancies can be linked must be large
enough to capture conflicts between the ideals and the reality of care work, suggesting
that concepts such as time and consent may be included.
7 Conclusion
In this paper, we have shown that knowledge maturing provides a useful framework
to embed reflection models into the overall organizational knowledge development.
The integrated model that has been developed in section 3 reveals important insights
about how the organizational perspective and the individual perspective interact: (1)
individual expertise moderating reflection and its contribution to knowledge matur-
ing, (2) the maturity of collective knowledge involved in reflection sessions moderat-
ing the reflection process, and (3) discrepancies between individual knowledge, expe-
riences, artifacts and collective knowledge being triggers for initiating reflection ses-
sions. Qualitative empirical research in two care homes underpinned the relevance of
the propositions in real work settings.
These insights are important foundations for designing reflection support in organ-
izations, both from a technical design point of view, but also from an organizational
point of view. Such reflection support has to be aware of (i) the expertise of the indi-
viduals involved, and of (ii) the maturity of knowledge that is reflected upon and
developed further as part of reflection. Ignoring the different characteristics of reflec-
tion resulting from varying levels of expertise and knowledge maturity may result in
barriers to productive reflection. Furthermore, the integrated model also provides
hints how conflicts between collective and individual perspectives trigger reflection.
Future work will refine the model by applying it to other types of cases, with a fo-
cus on its descriptive power in analyzing reflective learning and knowledge develop-
ment, and its utility in informing the development of technology for reflective learn-
ing. The findings in this paper may be used to enrich current models on reflective
learning as well as models of knowledge development in organizations.
26
Linking reflective learning and knowledge maturing in organizations - ARTEL13
Acknowledgements. This work has been supported by the European Commission
within the 7th Framework Programme as part of the projects MIRROR (no. 257617),
MATURE (no. 216356), and LAYERS (no. 318209).
References
1. Kaschig, A., et al., Organisational Learning from the Perspective of Knowledge Maturing
Activities. IEEE Transactions on Learning Technologies, 2013. 3.
2. Argyris, C. and D. Schön, Organizational learning: A theory of action approach 1978,
Reading, MA: Addison Wesley.
3. Cressey, P., Collective reflection and learning, in Productive Reflection at Work, D. Boud,
P. Cressey, and P. Docherty, Editors. 2006, Routledge.
4. Maier, R. and A. Schmidt, Characterizing Knowledge Maturing in 4th Conference
Professional Knowledge Management - Experiences and Visions (WM '07), N. Gronau,
Editor 2007, GITO: Potsdam. p. 10.
5. Boud, D., R. et al., Reflection: Turning Experience into Learning1985: RoutledgeFalmer.
6. Dewey, J., How we think. A restatement of the relation of reflective thinking to the
educative process (Revised edtn.) ed1933, Boston: D. C. Heath.
7. Kolb, D.A. and R. Fry, Towards an applied theory of experiential learning, in Theories of
Group Processes, C.L. Cooper, Editor 1975, John Wiley: London. p. 33-58.
8. Lave, J., The practice of learning, in Understanding Practice: Perspectives on Activity and
Context, S. Chaiklin and J. Lave, Editors. 1993, Cambridge University Press: p. 20.
9. Schön, D., The Reflective Practitioner1983: Basic Books, Inc.
10. Høyrup, S. and B. Elkjær, Reflection - beyond the individual, in Productive Reflection at
Work, D. Boud, P. Cressey, and P. Docherty, Editors. 2006, Routledge.
11. Stahl, G., Building collaborative knowing, in What We Know About CSCL And
Implementing It In Higher Education, J.-W. Strijbos, P.A. Kirschner, and R.L. Martens,
Editors. 2002, Kluwer Academic Publishers: Boston. p. 53-85.
12. Prilla, M., V. Pammer, and S. Balzert, The Push and Pull of Reflection in Workplace
Learning in EC-TEL2012, Springer: Saarbruecken, Germany.
13. Krogstie, B., et al., Computer support for reflective learning in the workplace: A model. ,
in Int. Conference on Advanced Learning Technologies (ICALT) 2012, ACM: Rome.
14. Kimmerle, J., U. Cress, and C. Held, The interplay between individual and collective
knowledge Knowledge Management Research & Practice, 2010. 8(1): p. 12.
15. Knipfer, K., et al., Reflection as catalyst for organisational learning. Studies in Continuing
Education, 2012. 35: p. 19.
16. Nonaka, I., A Dynamic Theory of Organizational Knowledge Creation. Organization
Science, 1994. 5(1): p. 24.
17. Maier, R. and A. Schmidt, Knowledge Maturing Model. (in submission), 2013.
18. Kump, B., et al., The Role of Reflection in Maturing Organizational Know-how, in 1st
European Workshop ARNets11, in conjunction with EC-TEL 20112011: Palermo, Italy
19. Pammer, V., et al. Reflective Learning at Work - A Position and Discussion Paper. in
ARNets11- Awareness and Reflection in Learning Networks. 2011. Palermo, Italy.
20. Herling, R.W., Operational Definitions of Expertise and Competence. Advances in
Developing Human Resources, 2000. 2(1): p. 14.
21. Schmidt, A., et al., Conceptual Foundations for a Service-oriented Knowledge and
Learning Architecture, in Networked Knowledge - Networked Media, S. Schaffert, K.
Tochtermann, and T. Pellegrini, Editors. 2009, Springer. p. 79-94.
27
Linking reflective learning and knowledge maturing in organizations - ARTEL13
22. Prilla, M., M. Degeling, and T. Herrmann, Collaborative Reflection at Work in GROUP
20122012, ACM: Sanibel Island, Florida, USA.
23. Prilla, M., V. Pammer, and B. Krogstie, Fostering Collaborative Redesign of Work
Practice: Challenges for Tools Supporting Reflection at Work, Submitted 2013.
24. Kitwood, T., Dementia reconsidered, in The person comes first 1997, Open University
Press: Berkshire, UK.
28
Reflection - quantifying a rare good
Thomas Daniel Ullmann? , Fridolin Wild, and Peter Scott
Knowledge Media Institute, The Open University
Walton Hall, MK7 6AA Milton Keynes, United Kingdom
{t.ullmann,fridolin.wild,peter.scott}@open.ac.uk
http://kmi.open.ac.uk
Abstract. Based on a literature review, reflections in written text are
rare. The reported proportions of reflection are based on different base-
lines, making comparisons difficult. In contrast, this research reports on
the proportion of occurrences of elements of reflection based on sentence
level. This metric allows to compare proportions of elements of reflec-
tion. Previous studies are based on courses tailored to foster reflection.
The reported proportions represent more the success of a specific instruc-
tion than informing about proportions of reflections occurring in student
writings in general. This study is based on a large sample of course fo-
rum posts of a virtual learning environment. In total 1000 sentences were
randomly selected and manually classified according to six elements of
reflection. Five raters rated each sentence. Agreement was calculated
based on a majority vote. The proportions of elements of reflection are
reported and its potential application for course analytics demonstrated.
The results indicate that reflections in text are indeed rare, and that
there are differences within elements of reflection.
Keywords: quantification, reflection, reflective thinking, reflective writ-
ing, reflection detection, reflection analytics
1 Introduction
The phenomenon ”reflection”, a pivotal thinking skill, has a rich theoretical
tradition. Several methods have been developed to measure reflection, especially
in the area of reflective learning. Analytical models of reflection in writings can
be distinguished by three types, covering the depth, the breadth, or the process
of reflection (e.g. [2–4, 6, 7, 9, 10, 13–15]). These models decompose reflection
into several elements characterising reflection.
Little is known about the quantities of these reflective elements in texts.
Quantification is the mapping of phenomena into a set of numbers. It is core to
scientific research, as it allows to investigate the properties of the phenomenon
in their context, its relations, probabilities, and patterns, to test theoretical
assumptions, in order to develop rules, laws, towards a general theory.
Regarding reflection, it is still a largely unmapped territory, when it comes
to the quantification of properties of reflection in texts. This research tries to
?
Corresponding author
29
Reflection - quantifying a rare good - ARTEL13
find answers to the question of what can be expected regarding the frequency of
occurrences of reflection in texts. Intuitively reflection occurs rarely in writings.
But, how rarely does reflection actually occur?
While there are attempts to quantify the proportions of elements of reflection
in reflective text the reported results are hard to generalise. Guidelines to com-
pare reflection studies do not exist yet. This research proposes a method based
on a comparable unit of analysis, which allows to estimate frequencies of reflec-
tive elements in texts. It describes the method used to quantify reflection, and
exemplifies the process on six elements of reflection. The study was conducted
in the context of course forum posts of a virtual learning environment.
2 Proportions of reflection in texts
The following literature review outlines findings on three levels of reflection re-
search, starting with a meta-analytical view, followed by research on the depth
of reflection, and proportions of elements of reflection.
Dyment and O’Connell [1, p. 90-91] undertook a meta-review of the quality of
reflection in student writings. They included 11 studies in their review. Amongst
other criteria, they looked at the distribution regarding the depth of reflection.
They classified the outcome of each study either as low, moderate, or high. A
study classified as low for example had a high percentage of texts, which were
mostly descriptive and less reflective. Five studies were categorised as low, four
studies had a moderate level of reflection, and two had a high level of reflection.
According to their review a relatively high proportion of studies report low
to medium levels of reflection, while only two studies achieved high levels of
reflection.
Although the categorisation into levels of reflection is informative, they warn
that the categorisation is to a degree subjective as the used methods in each
study vary making the comparison difficult.
The following section outlines results of individual studies describing propor-
tions of elements of reflection, and their context. Some of the studies are already
covered by Dyment and O’Connell [1]. Additional information was added to
situate these studies in their context. In one case further reported results are
presented. Studies not included in their review are marked with an asterisk.
*Wald et al. [13, p. 43] report on the distribution of reflective levels (depth
of reflection) within a corpus of 93 reflective writings. The reflective writings
stem from second-year students self-selected best reflective writing field notes.
The reflective writings were selected from archived material and were not in
connection with a instructional setup of the researchers. The levels started at
level 1 with ”nonreflective: habitual action” (0% of students), ”nonreflective:
thoughtful action” (18%), ”reflective” (41%), ”critically reflective” (30%), to
level 5 ”transformative learning” (11%).
Wong et al. [15, p. 53-54] conducted a content analysis (and interviews) of
reflective writings in the context of an instructional design specifically targeting
reflective writing. The writings were however not graded. The percentages of 45
30
Reflection - quantifying a rare good - ARTEL13
students regarding the level of reflection were as follows: 13% non-reflectors, 76%
reflectors, and 11% were critical reflectors.
Plack et al. [7, p. 204] analysed reflective writings of 43 journals of students
participating in a course for clinical practice, which had an emphasis on reflec-
tive practice (instructions were given about reflection, journals were not used for
grading). They report frequencies of elements and depth of reflection. Regarding
the depth, 15% of the journals had no evidence of reflection, 43% showed evi-
dence of reflection, and 42% evidence of critical reflection. The percentages for
the elements of reflection were (percentage of journals containing an element):
Reflection in action (23%), reflection on action (38%), reflection for action (28%),
content (35%), process (38%), premise (18%), returns to experience (39%), at-
tends to feelings (38%), and re-evaluation (32%). The most frequent element was
return to experience, which is the category label for a description of an impor-
tant experience. The premise element - critique of own assumptions - is the less
frequent one (see Plack et al. [7, p. 206-7] for descriptions of the elements).
Hatton and Smith [3, p. 41] assessed the written work of 60 students in the
context of a professional program, which used reflection-fostering instructions.
They reported overall percentages of coded units. 60-70% were descriptive reflec-
tive, and more than 30% of dialogic reflection was found in essays after a special
instruction. On average 19 reflective units were found in a writing of 8-12 pages.
*The analysis of Ross [8, p. 24-25 ] took into account 134 papers from 25 stu-
dents (average 5.4 papers per student). The students took a course with a special
focus on reflection. The article reported the following percentages of papers: 22%
were highly reflective, 34% moderately reflective, and 44% low reflective.
Williams et al. [14, p. 7 and 12] report the highest achieved level of reflection
of 56 students, who had to write a reflective journal during a course (the journal
made up 10% of the grade, at least one journal had to be written per week).
The percentages from the lowest level to the highest are: 0% describe learning,
2% analyse learning, 23% verify learning, 36% gain new understanding, and 39%
indicate future behaviour.
The proportions of students or texts regarding level or elements of reflection
have to be used with care. The mapping from the evidences of reflection to el-
ements or levels depends largely on the interpretation of the researchers, and
thus the percentages might be different if the mapping process would have been
done differently. In addition, the reported percentages are on either a level of a
person or journal. Thus, not much can be said about the distribution of reflective
elements or levels in texts. Tentatively, the presented research results might in-
dicate that most texts/students are written/write with a low or medium level of
reflection. Highly reflective texts/students are rather rare. All presented studies
describe that the course was especially designed to foster reflection, which com-
pared to normal courses might result in higher proportions of reflection. The
articles reporting on elements of reflection indicate that some elements occur
more frequently than others. The work of Hatton and Smith [3] is insofar from
special interest for this research as they provide percentages of units of reflec-
tion, which might give indications about the frequencies of reflective utterances
31
Reflection - quantifying a rare good - ARTEL13
in text. However, they do not specify exactly their understanding of a unit,
which makes estimates speculative. For example, if a unit is a sentence, then 19
reflective units in 8-12 pages would indicate that reflections are rare instances
(about 1% of the text assuming 200 sentences for 10 pages). If a unit equals a
paragraph and if we assume that a page consists of five paragraphs, then 38%
of the units of a text would be reflective considering 10 pages.
3 Method
The chosen approach to quantify reflections follows seven points:
– Choose a text corpus and describe its domain and characteristics.
– Unitise the text corpus (in here the unit of analysis are sentences).
– Draw a random sample of units.
– Based on theory of reflection derive elements of reflections, which charac-
terise it.
– Operationalise each element of reflection.
– Device a strategy to gain annotations for the units. In this paper the units
are manually annotated.
– Calculate and report the proportions of the annotated elements of reflection
(quantification).
It is worth to consider controlling the text length (amount of units of each
text), especially for a smaller corpus. For example, if a corpus with a very long
text and several very short texts is used, the randomly selected units will come
mostly from the long text. If the long text is about describing a problem and
the short texts are mostly acknowledging or short introductions of members of
a forum, the uncontrolled corpus will be biased.
The theories on reflection vary regarding the elements which together model
reflection. Furthermore, the elements of reflection are often high level descrip-
tions, which may be too abstract to measure. For each element several subele-
ments can be designed, with the aim to arrive at measurable elements. This list
of subelements may be too large to be administered in a single experiment. A
pre-test can help to find the optimal amount of items. A pre-test is also advisable
to check the measurability of each item. Inhere six items were selected. For each
element of reflection one or two items were selected. The rational is that the
items are better distinguishable by human raters, instead of using several items
from one element, which may be too similar.
4 Text corpus
The text corpus used for this research is based on forum data of the virtual
learning environment1 of the Open University, UK. It consists of two courses on
eLearning, two courses about social work, and one course on science. The data
was de-identified by the researcher.
1
https://github.com/moodleou/
32
Reflection - quantifying a rare good - ARTEL13
course description posts unique unique
sentences sentences
with without
personal personal
pronoun pronoun
eLearning 1 postgraduate course (30 410 3454 4639
credits)
eLearning 2 postgraduate course (30 274 2115 2480
credits); different semester than
eLearning 1 course
social work level 2 course (60 credits) 475 3787 3689
social work level 3 course (60 credits) 103 903 872
science postgraduate course (30 355 2170 3820
credits)
Table 1: Description of text corpus
The forums serve several purposes, for example to support the students, to
have a platform for discussion, exchanging ideas, and to socialise. The forum
posts therefore contain a wide spectrum of writings. The eLearning and social
work courses explicitly stated that reflection is one of the learning goals. How-
ever, special assignments regarding reflection were in general2 not conducted
within the forums, but with forum external means, and thus may be only indi-
rectly present in the forum corpus. The science course did not explicitly aim at
reflection.
From all posts, posts of a character length between 1500 and 3500 charac-
ters were selected. On average the 1677 forum posts were 2121 characters long
(sd=512).
Only course forums, which were core to the course and explicitly embedded
in the activity of the course were kept. Forums for technical support, student
talk, and general course-wide forums were excluded.
From all posts the ones made by the role ”student” are kept, while the other
roles like ”tutor”, ”moderator”, or ”production staff” were filtered.
All texts were split into single sentences. These were then divided into sen-
tences that contain a personal pronoun and sentences that did not contain a
personal pronoun (using the reflection detection architecture of Ullmann [11]
and its extension [12, p. 106f.]). 500 sentences from each set were randomly se-
lected. They form the input of the corpus of sentences used for the annotation
process.
2
some courses had forums to discuss reflection assignments
33
Reflection - quantifying a rare good - ARTEL13
5 Questionnaire
The questionnaire contained four sentences on each page to rate. A maximum
of 30 sentences could be coded by a coder per batch. The instruction of the
questionnaire explained the task. Each category was described with an example
sentence and explanation. The workflow for the raters was: read a sentence,
categorise it, and write a short explanation justifying your choice.
The raters could choose from one of the following seven categories. These
categories follow the reflection model outlined in Ullmann et al. [12, p. 103f.].
Something could/should have been done differently, drawing a conclusion based
on a premise (reasoning), taking another perspective (point of view), intention
to do something, something was successfully learned (achieved), something is
interpreted in a new way (new understanding), and none of these. The raters
were prompted to choose the best category for each sentence. In addition, the
instruction stated that the sentence had to speak for itself. The categories were
presented for each sentence in random order. Pre-experiments showed a preferred
answer bias for the first category. Shuffling the categories aimed at minimising
this bias based on the sequence of the categories. After the rater selected their
answer, they had to justify their choice with a short free text answer.
The categories can be seen as one of several variations to capture an element.
This also means, that the questions do not exhaustively represent an element,
but they cover to a certain extent the essence of it. The following table shows
the mapping between the elements of reflection [12, p. 103f.] and the categories.
Elements of reflection Categories
Description of an experience Something could/should have been done
differently
Critical analysis Drawing a conclusion based on a premise
(reasoning)
Taking other perspectives Taking another perspective (point of view).
into account Something is interpreted in a new way (new
understanding)
Outcome Intention to do something. Something was
successfully learned (achieved). Something is
interpreted in a new way (new understanding)
Table 2: Mapping of elements of reflection to the categories of the
questionnaire.
The question ”something is interpreted in a new way (new understanding)”
can be seen as an outcome dimension of reflection, but also as another take/per-
spective on something.
34
Reflection - quantifying a rare good - ARTEL13
6 Survey
The questionnaire was distributed via a crowdsourcing platform3 . The partici-
pants had to fill out a minimum of four gold questions before they saw the first
item of the survey. Gold questions are items with known answers used to stop
those participants that fail to correctly answer a certain amount of gold ques-
tions from filling out the survey. Additionally, a set of own text validators was
used to discourage participants filling out the questionnaire randomly.
10 batches of 100 sentences were administered. Participants from previous
batches were allowed to rate a new batch. Each sentence was rated by at least
five raters.
The 411 raters came from 17 countries. Most of them were from the USA (n
= 202), GBR (n = 94), and IND (n = 45). The remaining 70 raters came from
14 other nations.
7 Results
The inter-rater agreement between the annotators was measured using Krippen-
dorff’s α [5] for nominal data. The α for the gold data is 0.43, for annotation
and gold data combined it is 0.32, and for the annotation data only is 0.22.
The annotated sentences were then filtered. Only sentences where three or
more raters agreed on (majority vote) remained in the data set. From the original
1000 sentences 623 remain. Krippendorff’s α for this set is 0.36. The α values
of the participants are relatively low, which means that by running the same
experiment on the crowdsourcing platform again, some of the sentences will be
classified differently. It has however the benefit that it does not rely on expert
ratings, which also might be difficult to replicate by other researchers.
The following table shows sentences from the experiment and their classifi-
cation.
Category Example
Something could/should Victor and Morgan you are right that I should
have been done differently have applied better my own learning instead of
using the Uni ones. [names were de-identified
before the survey]
Drawing a conclusion based I imagine this is probably in order to have a
on a premise (reasoning) focus and provide enough detail rather than
skim over the whole area.
Taking another perspective When I am doing FRT work, I often think
(point of view) about how the parents view me when they
know I haven’t got children!
3
http://crowdflower.com/
35
Reflection - quantifying a rare good - ARTEL13
Something is interpreted in After I saw how this lifted her mood and eased
a new way (new understand- her anxiety, I will remember that what we can
ing) view sometimes to be small can actually make
a significant difference.
Intention to do something I would like to be involved in helping with the
site too -although I’m a novice!
Something was successfully This has helped me reflect on my own life and
learned (achieved) experiences whilst allowing me to empathise
with others in their own circumstances, I feel
proud of what I have achieved so far as the
work/life/study balance is always difficult to
navigate but I’m lucky that I have a supportive
family to help.
None of these Bye the way, Audacity is also run under the
CC Attribution licence.
Table 3: Example sentences for each category.
As we are interested in the proportions of reflective elements and not so much
in the replication of every sentence’s judgement, an analysis of the stability of
the proportion over different points in time was conducted. The assumption is
that similar proportions will arise, if the number of sentences is big enough and
the sentences are randomly selected.
To test this assumption the 10 runs were split into 3 equally sized batches of
208 items. The following table shows the results.
batch 1 batch 2 batch 3
none of them 109 102 121
intention to do something 32 30 18
reasoning 26 32 22
successfully learned 21 23 23
could have been done differently 12 7 10
interpreted in a new way 4 5 7
taking another perspective 4 9 6
Table 4: Stability over batches
The proportions of the categories do not vary much in each batch, which
may indicate that although the inter-rater reliability is small since each time
the experiment is replicated the sentences will receive different annotations, the
proportion of elements will stay relatively steady. One exception (in batch 3
of the element intention) may indicate that there are some fluctuations, which
should be considered in an experiment with a bigger sample size.
The next table reports on the proportions of reflective elements in the dataset.
Based on the sentences, which received the majority vote, it shows for all ele-
36
Reflection - quantifying a rare good - ARTEL13
ments its frequencies and its percentages. In addition, it shows the frequencies
for personal and non personal sentences.
all % personal non personal
could have been done differently 29 4.65 18 11
reasoning 80 12.84 42 38
taking another perspective 19 3.05 13 6
interpreted in a new way 16 2.57 13 3
intention to do something 80 12.84 53 27
successfully learned 67 10.75 43 24
none of them 332 53.29 113 219
Table 5: Number of sentences in each category
Focusing on the six elements of reflection, the categories ”reasoning”, ”in-
tention to do something”, and ”successfully learned” are present in more than
10% of the sentences. Besides these relatively frequent, but still rare elements,
”interpreting something in a new way”, ”taking another perspective”, and the
recognition that ”something could have been done differently” occur in two to
five percentage of the sentences. The category ”none of them” has the highest
percentage. ”None” does not necessarily mean that this category represents a
”not reflective” category. It might include sentences, which are reflective, but
not captured in the six items representing reflective elements. The chosen six
elements of reflection are not an exhaustive model of reflection capturing all
possible ways of expressing reflection in texts.
In order to test the influence of personal pronouns in reflective sentences, the
original sample contained the same amount of sentences with personal pronoun
as those without. The table shows that personal sentences have higher category
counts for all reflective elements than non personal sentences. This might indicate
that sentences containing a personal pronoun are more likely to be annotated as
one of the reflective elements.
Assuming that there are relatively constant proportions of reflective elements,
one area of application might be the analysis of courses regarding the baseline
proportions of reflective elements. Compared to this experiment however, more
data from several domains should be taken into account in addition to a more
fine grained model of elements of reflection.
The next table serves as an example for this possibility. It inspects the pro-
portions of reflective elements over the five courses of the data set. The amounts
of sampled sentences from each course vary. The number of sentences for the
first eLearning course was 196, from the second eLearning course was 111, from
the social work level 2 course was 146, level 3 course was 39, and from the sci-
ence course 131. As the social work level 3 course has a much lower sentence
count than the other courses, it was excluded from the results. The courses were
balanced according to their number of sentences. The following table shows the
balanced percentages of elements of reflection for each course.
37
Reflection - quantifying a rare good - ARTEL13
%el1 %el2 %swl2 %sci %all
could have been done differently 3.1 5.4 4.8 6.9 4.7
reasoning 13.3 9.9 17.8 9.9 12.8
taking another perspective 2.6 2.7 4.8 2.3 3.0
interpreted in a new way 3.6 1.8 2.1 2.3 2.6
intention to do something 12.2 10.8 14.4 10.7 12.8
successfully learned 11.2 12.6 8.2 10.7 10.8
none of them 54.1 56.8 47.9 57.3 53.3
Table 6: Percentage of balanced sentences for elements of reflection per
course. el1 and el2: elarning courses; swl2 and swl3 is the social work
course level 2 and level 3; sci: science course
The last column contains the previously reported percentages of the whole
corpus. The courses with more sentences will have more influence on the overall
percentages. Notable is the 9.4% difference between the social work level 2 course
and the science course on the element ”none”. This means that the former course
has nearly for each 10 sentences an additional reflective sentence.
8 Discussion
Based on the intuition that reflection is a rare good (valuable but does not occur
frequently), this research provides evidence about the proportions of reflective
elements in texts. Based on the literature review, research indicates that the
proportions vary on the study level, level of reflection, and elements of reflec-
tion. This research presents the proportions of reflection, based on the unit of
sentences, from six elements of reflection. It concludes that these elements of
reflection are indeed rare, and that some of the elements, for example the ele-
ment ”change of perspective”, or ”something was interpreted in a new way”, are
especially rare. In addition, sentences that contain a personal pronoun, are rated
as having higher frequencies of reflection. This may indicate that sentences, in
which the writer expresses a personal view, are more likely to be rated as reflec-
tive.
In addition, the number of sentences of reflective elements varies between
courses. While these results are interesting for the comparison of courses, the
results have to be taken cautiously, as a bigger sample size would be necessary
to carry out a thorough analysis of the elements of reflection. This would be
needed to balance the cells containing a small number of sentences.
Compared to the research outlined in the theory part, this research shows
proportion of reflective elements from course forum posts, that did not focus on
developing reflective writing skills. This may help as a reference to other courses
that are especially designed to enhance reflective writing.
This research used sentences as the unit of analysis. While this decision helps
to calculate percentages of elements of reflection based on the total sentence
count of a text, it bears its own problem. Certain elements may not be able to
38
Reflection - quantifying a rare good - ARTEL13
be captured in one sentence, and thus the raters might have annotated it as none,
although from the wider context it would belong to an element of reflection.
As stated, the studied elements of reflection were not exhaustive to describe
all forms of reflection. Further research with other elements, and other opera-
tionalisations of the elements would help to extend this research. Furthermore,
it may be fruitful to study text corpora with different contexts. This could help
to determine, which context factors stimulate reflection or hinder it.
The used approach however allows to quantify the proportions of reflective
elements, and it indicates that reflection is a rare good.
References
[1] Dyment, J.E., O’Connell, T.S.: Assessing the quality of reflection in student
journals: a review of the research. Teaching in Higher Education 16, 81–97
(Feb 2011), http://www.tandfonline.com/doi/abs/10.1080/13562517.
2010.507308
[2] Gulwadi, G.B.: Using reflective journals in a sustainable design stu-
dio. International Journal of Sustainability in Higher Education 10(2),
96–106 (Oct 2009), http://www.emeraldinsight.com/journals.htm?
articleid=1776291&show=abstract
[3] Hatton, N., Smith, D.: Reflection in teacher education: Towards defini-
tion and implementation. Teaching and Teacher Education 11(1), 33–
49 (Jan 1995), http://www.sciencedirect.com/science/article/pii/
0742051X9400012U
[4] Korthagen, F., Vasalos, A.: Levels in reflection: core reflection as a means to
enhance professional growth. Teachers and Teaching: Theory and Practice
11, 47–71 (Feb 2005), http://www.tandfonline.com/doi/abs/10.1080/
1354060042000337093
[5] Krippendorff, K.H.: Content Analysis: An Introduction to Its Methodology.
Sage Publications, Inc, third edition edn. (Apr 2012)
[6] Moon, J.A.: A handbook of reflective and experiential learning. Routledge
(Jun 2004)
[7] Plack, M., Driscoll, M., Blissett, S., McKenna, R., Plack, T.: A method for
assessing reflective journal writing. Journal of allied health 34(4), 199–208
(2005)
[8] Ross, D.D.: First steps in developing a reflective approach. Journal of
Teacher Education 40(2), 22–30 (Mar 1989), http://jte.sagepub.com/
content/40/2/22
[9] Scanlan, J.M., Chernomas, W.M.: Developing the reflective teacher.
Journal of Advanced Nursing 25(6), 1138–1143 (Jun 1997),
http://onlinelibrary.wiley.com/doi/10.1046/j.1365-2648.1997.
19970251138.x/abstract
[10] Thorpe, K.: Reflective learning journals: From concept to practice. Re-
flective Practice 5(3), 327–343 (2004), http://www.tandfonline.com/doi/
abs/10.1080/1462394042000270655
39
Reflection - quantifying a rare good - ARTEL13
[11] Ullmann, T.D.: An architecture for the automated detection of textual in-
dicators of reflection. In: Reinhardt, W., Ullmann, T.D., Scott, P., Pam-
mer, V., Conlan, O., Berlanga, A. (eds.) Proceedings of the 1st European
Workshop on Awareness and Reflection in Learning Networks. pp. 138–151.
Palermo, Italy (2011), http://ceur-ws.org/Vol-790/
[12] Ullmann, T.D., Wild, F., Scott, P.: Comparing automatically detected re-
flective texts with human judgements. In: 2nd Workshop on Awareness and
Reflection in Technology-Enhanced Learning. Saarbruecken, Germany (Sep
2012), http://ceur-ws.org/Vol-931/paper8.pdf
[13] Wald, H.S., Borkan, J.M., Taylor, J.S., Anthony, D., Reis, S.P.: Fostering
and evaluating reflective capacity in medical education: Developing the RE-
FLECT rubric for assessing reflective writing. Academic Medicine 87(1), 41–
50 (Jan 2012), http://journals.lww.com/academicmedicine/Abstract/
2012/01000/Fostering_and_Evaluating_Reflective_Capacity_in.15.
aspx
[14] Williams, R.M., Wessel, J., Gemus, M., Foster-Seargeant, E.: Journal writ-
ing to promote reflection by physical therapy students during clinical place-
ments. Physiotherapy Theory & Practice 18(1), 5–15 (Mar 2002)
[15] Wong, F.K., Kember, D., Chung, L.Y.F., Yan, L.: Assessing the level of
student reflection from reflective journals. Journal of Advanced Nurs-
ing 22(1), 48–57 (Jul 1995), http://onlinelibrary.wiley.com/doi/10.
1046/j.1365-2648.1995.22010048.x/abstract
40
Support for Collaborative Reflection in Healthcare:
Comparing two Workplaces
Michael Prilla, Martin Degeling
1
Ruhr-University of Bochum, Institute for Applied Work Science
{michael.prilla, martin.degeling}@ruhr-uni-bochum.de
Abstract. Reflection is a frequent and integral part of daily work, and often it is done
by multiple actors in meetings or during joint work. Its support, however, has mainly
been investigated with respect to educational settings or individual reflection processes.
This paper describes a framework for the support of collaborative reflection at work
and a study on its implementation at two workplaces in healthcare. Results of the study
show which means of supporting collaborative reflection can be beneficial in practice
and that we need to understand this support as a socio-technical task.
1 Introduction: Collaborative Reflection at Work
Reflection is a frequent and integral part of daily work (Boud 1985; Kolb and Fry 1975).
Typical examples are thinking about whether certain decisions were right and groups consid-
ering whether their cooperation is effective. Reflection processes contain three steps: Going
back to experiences, re-evaluating them and drawing conclusions for future work from this
(Prilla et al. 2012a). While this has been investigated for individuals reflecting, collaborative
reflection is done by a group of people sharing their experiences and jointly developing
changes for future work from that (Daudelin 1996; Dyke 2006). The result of such collabora-
tive processes can be relevant for individual work, but also on group and organisational lev-
els. By this, collaborative reflection complements education and vocational training at work
with a process of bottom-up understanding and evaluating work and helps people to learn
from experiences in a self-directed, collective manner (Daudelin 1996; Hoyrup 2004). How-
ever, there are hardly any insights on the design of tools for collaborative reflection.
This papers aims to bridge the resulting gap and describes a study in which support for
collaborative reflection was trialled in two healthcare workplaces. From this, the paper
shows how collaborative reflection can be supported, that we have to understand this support
as a socio-technical task and that there is a need for further work on exploring such support.
In what follows, we describe existing work and insights into collaborative reflection as
well as the “Talk Reflection App” as a means to support such reflection (section 2). After
that, we present the study conducted in the healthcare workplaces (section 3), its results (sec-
tion 4) and a discussion on conclusions to take from our work, including impacts on the de-
sign of collaborative reflection tools (section 5).
41
Support for collaborative reflection in healthcare: comparing two workplaces
2 Computer Supported Collaborative Reflection
2.1 A Framework for Supporting Collaborative Reflection
Tools for the support of reflection have been discussed mainly in educational settings or with
respect to individual reflection. Authors have proposed journals or portfolios or series of
pictures to help users to reconstruct and reflect experiences (Fleck and Fitzpatrick 2009;
Scott 2010). With the exception of support for specific situations such as post-mortem pro-
ject reflection meetings (Kerth 2001) and generic tools such as shared whiteboards (Lin et al.
1999) there is hardly any support for collaborative reflection available.
Differences between individual and collaborative reflection can foremost be seen in sup-
port for communication among participants and sustaining results – to reflect together, par-
ticipants need to exchange experiences, different individual perspectives and understandings
need to be discussed and solutions need to be agreed on. To operationalize the respective
steps of collaborative reflection and the corresponding needs, we created a cyclic blueprint
for collaborative reflection tools based on insights from earlier user studies on collaborative
reflection (Prilla et al., 2012a; Fig. 1).
Fig. 1. Cyclic blueprint of collaborative reflection, based on (Prilla et al. 2012a).
The blueprint shows how collaborative reflection is tightly coupled to phases of individual
reflection and that these phases constantly switch, e.g. if a topic is discussed in a group, then
reflected by individuals with respect to own experiences and later on again reflected within a
group. Such reflection needs a sustainable documentation of experiences to share them with
others and to sustain all relevant aspects – usually, after a certain time details may be forgot-
ten and emotions may fade, which affects the reflection processes afterwards. Moreover,
reflection should end up in sustainable and shareable results if it is to lead to changes in work
– otherwise, only the group reflecting knows about proposed changes ad their origin (Prilla et
al. 2012b). A central requirement for al steps is support for articulation of experiences, ideas,
42
Support for collaborative reflection in healthcare: comparing two workplaces
solution proposal and results and to be able to share these articulations. Table 1 summarizes
the resulting requirements (cf. Prilla et al. 2012a).
Table 1. Requirements for the support of articulation in collaborative reflection.
Articulation Requirement
Experiences Documenting experiences for later use
Individual Reflection Documenting and sharing insights (e.g., in comments)
Collaborative Reflection Sharing similar experiences and ideas for solutions (e.g., in
comments)
Results Documentation and sharing of results
2.2 Supporting Collaborative Reflection of Conversations: The TalkReflection App
The „Talk Reflection App“ (Fig. 2) was developed to support the phases of the collaborative
reflection blueprint shown in Fig. 1 and to implement the requirements documented in Table
1. The app supports reflection on conversations with patients in hospitals, residents in care
homes, relatives of patients and third parties such as social workers, as our studies in
healthcare workplaces revealed that this is a relevant and reflection intensive topic (see Prilla
et al. 2012a). Therefore the app supports articulation to explicate experiences from conversa-
tions as described in Table 1. It also supports sharing these documentations and the articula-
tion of outcomes from collaborative reflection. The usage of the app is explained in the fol-
lowing scenario.
Fig. 2. The Talk Reflection App. Left: Overview of accessible, articulated experiences (documenta-
tions); Right: Form for documenting a conversation.
The caregiver Anna has had a difficult conversation with relatives of a resident in the care
home she is working hat. Afterwards she documents (articulates) the content and topic of the
conversation together with her feelings during the conversation in the app. She describes her
insecurity when telling the son of an older lady that the lady’s medical conditions are getting
worse dramatically, and she might die soon. She uses the content fields to describe this and
43
Support for collaborative reflection in healthcare: comparing two workplaces
the self-assessment sliders below it to document her feelings about the situations (see Fig. 2,
right).
Her colleague Bernd notices that Anna has shared her note with him (Fig. 2, left). When
he reads it, he remembers a similar situation he has been in and comments on Anna’s docu-
mentation, suggesting to refocus the conversation on the relative and to offer him support –
in other words, he reflects on his own experience and makes a suggestions for coping with a
comparable situation in the future. Since the topic is of interest for the rest of the ward, too,
Anna’s case is also discussed in a meeting with all caregivers of the ward and together they
discuss strategies how to support and teach new caregivers in similar situations. They agree
on three things to include in this process and document this outcome of their collaborative
reflection in the Talk Reflection App, connecting it to the cases it is related to. They shared it
with all colleagues on the ward in order to enable them to reconstruct the suggested solution
based on the cases linked to it.
3 Supporting Collaborative Reflection with the Talk Reflection App: A Study at
two Workplaces
The Talk Reflection App was trialled for four weeks in a German hospital specialized in care
for neurological diseases (case 1) and for five weeks in a British care home for people suffer-
ing from dementia (case 2).
In case 1, reflection was focused on conversations between physicians and relatives of pa-
tients, as physicians felt they needed to systematically use experiences from such conversa-
tions to learn how to act professionally if the conversations get emotionally stressful. They
stated that their prior education had not covered this topic and that they would be willing to
form a group helping each other in it by using the Talk Reflection App. The study was con-
ducted with five participants, including inexperienced assistant physicians, who had just
started work at the hospital and more experienced senior physicians (see Table 2).
Table 2. Overview of study participants and tools used to gather data
Participants Data from studies
Case 1 5 physicians (2 experienced, 3 less expe- Questionnaires
(Hospital) rienced) (Pre/Post), interviews,
Case 2 5 caregivers (3 to 20 years of experi- observations (meet-
(Care home) ence), (1 manager) ings), log files
In case 2, the caregivers wanted to reflect on conversations and interactions with others relat-
ed to such conversations, including encounters with residents, their relatives and third parties
such as social workers. Care for people suffering from dementia is especially demanding for
caregivers, as these people might (re)act strangely or even become aggressive. Being able to
talk to them, their relatives and third parties in a professional way not only diminishes the
personal stress level resulting from that, but also improves the reputation of a care home. The
study was conducted with five caregivers, who used the Talk Reflection App to learn about
44
Support for collaborative reflection in healthcare: comparing two workplaces
these situations, and the manager, who wanted to be informed about it, but did not actively
use the app.
In both cases, the app was introduced to the five participants in a common workshop in
order to enable them to work with it in a self-directed manner. Before the workshop, they
were asked to fill out a short (pre-) questionnaire (10 items) to create a baseline on their cur-
rent reflection practice and need for learning about the respective topics. In the middle of the
trial period and at its end, we conducted reflection workshops with the participants, in which
they were asked to skim through experiences documented in the app and choose some for
reflection in the workshop. During the meetings a researcher was present to observe and
document reflection among the participants and to get feedback on the app.
Table 3. Meetings during the trial periods in both cases.
Time Meeting and Tools for data gathering
Observation
Begin of trial period Introduction and training, pre-questionnaire
Middle of trial period Initial reflection meeting on cases in the app
End of trial period Final reflection meeting, post-questionnaire, short
interviews
At the end of the trial period, in addition to the final reflection meeting the participants were
asked to fill out a post-questionnaire (30 items), which included the same items used to cre-
ate the baseline and an additional set of items on aspects such as acceptance, value and im-
pact of using the app, which were aligned to the levels of evaluation on the model of Kirk-
patrick (1975) – typical questions can be found in section 4. Moreover, the participants were
interviewed briefly on their experiences in using the app. Table 3 summarizes the course of
the studies and the tools used for gathering data. This data was complemented by the log data
captured throughout the usage of the app.
4 Results
In both cases, the participants used the Talk Reflection App for the documentation of critical
conversations and (in case 2) interactions in order to reflection on them later. In case 1, for
example, an assistant physician documented a conversation with relatives, which she per-
ceived to be very demanding (see Table 4). She created a comment to document this percep-
tion. In a later meeting with the other participants, she used this documentation as a memory
aid to present this case to her colleagues in a very detailed manner. This caused immediate
reactions by her colleagues and in the resulting reflection session of this case, colleagues
reported similar cases and proposed different ideas how to better deal with such situations. In
the end, they agreed that in these situations, inexperienced physicians should always ask a
senior physicians to join until they were confident to deal with such situations alone.
In case 2, caregivers reflected about the death of a resident, who has been admitted to
hospital against her advice to the relatives and social workers. One of the caregivers had
documented the interaction with the relatives that led to the admittance (see Table 4), and
45
Support for collaborative reflection in healthcare: comparing two workplaces
afterwards used this documentation to reflect on this it with her colleagues and the home
manager. Some senior staff members reported on similar situations in the past and that a
conversation how to deal with the emotional affection directly after the situation had helped
them to overcome their grief. Although some had stated initially that such situations can
happen in a care home and that the group should not take any particular measures, after the
reflection session they agreed that there would be a possibility to have such group conversa-
tions after very demanding situations in the future, which would be led by the manager.
Table 4. Sample Documentation, Comments and Outcomes of using the Talk Reflection App in both
cases.
Articulation Hospital (Case 1) Care home (Case 2)
Talk / “[Patient’s] therapy finished. „The resident passed away suddenly,
Interaction Again relapse, palliative ther- had been here long, was liked by all
apy. Prepared [relatives] for staff. Was ill in the morning and her
begin of home care, asked to guardian admitted her to hospital, Un-
seek professional support for fortunately she passed away [there].
care. Talk was very difficult, This was very distressing to the staff as
parts were not received or they felt it would have been more dig-
blocked out.” nified for the client to be in familiar
surrounding.“
Comment “[Relative] conveys the feel- - (verbal statements)
ing it is our fault. (…) Hears
for the first time that [patient]
is going to die”
Outcome “Problem: Conversation held „After discussing with the homes man-
alone. It should be known that ager about the staff being upset, it was
a senior physician can be decided that staff who were most af-
asked for support” fected get together and discuss
thoughts and feelings.“
The examples shown in Table 4 illustrate that besides documenting experiences and making
them accessible later, using the Talk Reflection App also had an impact on the reflection of
each individual: In the articulation work (cf. Suchman 1996) of documenting their experi-
ences, the participants had also documented insights from reflection. In case 1, the physician
mentioned that she perceived that the relative had blocked out certain information given to
her, and in case 2, the caregiver (verbally, not in the app) stated that the grief of staff was
mainly caused by the manager, relatives and social workers not listening to their advice. This
shows how documenting the cases already triggered reflection.
Looking at the usage of the app in both cases (see Table 5), we can see that is was pre-
dominately used for the documentation of cases. Given that a critical situation does not ap-
pear every day, we consider the creation of 7 documentations in 121 days (case 1) and 18
1
Due to technical problems, there is only solid data for the last 12 days of the trial period, although the
participants stated to have used the app before as well. In addition, some physicians created docu-
46
Support for collaborative reflection in healthcare: comparing two workplaces
documentations in 33 days (case 2) to be sufficient for an initial test, in which in both cases
the users needed some time to adopt the application and integrate it into their work tasks. All
documented experiences were shared with all other participants in both cases.
Despite the sufficient amount of documentations, other features such as commenting and
creating outcomes were used below our expectations (see Table 5): Compared to the amount
of documentations, the number of outcomes (three in case 1, two in case 2) and the number
of comments (nine in case 1 and 14 in case 2) are not sufficient and, especially in the case of
comments, cannot lead to fruitful exchange of experience as it is necessary collaborative
reflection. In addition, some of the created outcomes in the app are a result of reminding the
participants during reflection meetings to also write down their outcomes.
Table 5. Number of Documents, Outcomes and Comments created in the Talk Reflection App in both
cases.
Type of articulation Case 1 Case 2
Documentation of critical conversations / interactions 7 18
Documentation of outcomes from reflection 3 2
Comments on (own / others’) documentations 9 (4/5) 14 (11/3)
Despite the numbers shown in Table 5, an analysis of statements and feedback given by the
participants of both studies during the meetings and in the final interviews shows that they
perceived using the app to be valuable for exchanging experiences, reflection on them in the
group and deriving outcomes for future work. In meetings, we could observe participants
vividly discussing during reflection how certain situations needed to be understood or could
be tackled better in the future. In interviews, participants could easily describe comments
they had made verbally on others’ documents and also some agreements the participants had
come up with to solve certain issues expressed in the documentation. The participants report-
ed that all of these articulations and communications had happened outside the app in face to
face interactions as they happen daily in hospitals and care homes when people meet each
other during work, in meetings or between shifts. These casual interactions were perceived to
cause less effort and led to features such as commenting being used less. This effect could
also be seen in meetings, in which participants used documented experiences from the app to
describe a certain situation to their colleagues and to reflect on it verbally afterwards.
Data from the questionnaires used in the study underpins that there was value from using
the Talk Reflection App in both cases, as it indicates that participants perceived it to have a
positive impact on constructively thinking about conversations (that is, individual reflecting
on them) and discussing conversations with colleagues (part of collaboratively reflecting
them, see Fig. 3). The data also indicates a light effect on the perception of how situations
reflected on could be improved during the trials, which was slightly stronger for case 2. Giv-
en that the trials only lasted four and five weeks, this goes beyond our expectations as we
had expected changes in behaviour to take longer than this period to be implemented. How-
ever, long-term evaluations need to approve this finding.
mentations offline and wanted to save them to the app, although they had no connection to the Inter-
net. This resulted in lost cases.
47
Support for collaborative reflection in healthcare: comparing two workplaces
5
4,0
4,0
4,0
4,0
3,8
4
3,5
3
2
Case
1
1
Case
2
The
app
has
helped
to
The
app
has
helped
to
The
app
has
helped
to
constructively
think
about
discuss
conversations
with
improve
conversations
conversations
afterwards.
colleagues.
with
residents,
relatives
or
others.
Fig. 3. Perceived value of using the Talk Reflection App in both cases, n=5 for both cases.
The participants were also asked to clarify which factors had influenced their reflection prac-
tice. Answers to this show that being aware of reflection as an important topic and organisa-
tionally anchoring it in regular meetings was perceived at least to be as important as using
the app (see Fig. 4). For case 1, the answers even indicate that the participants perceived the
app to be less valuable for reflection that the other factors. These surprising results can be
attributed to the fact that the studies lasted for only four and five weeks and that in this short
period time, the effect of the intervention to more systematically reflect superseded the posi-
tive effect caused by the app. This might have been amplified by the time to adopt the app in
each of the cases as reported above. On the other hand it shows that the sheer process of
introducing the app acted as a reification of reflection, increasing awareness for it – this pro-
vides a good basis for long-term success of the app.
In
the
past
weeks
it
helped
me
in
thinking
about
conversations
with
relatives
and
residents...
5
4,2
3,6
3,8
3,6
3,6
4
3,4
3
Case
1
2
Case
2
1
The
app
has
helped
to
The
app
has
helped
to
The
app
has
helped
to
constructively
think
about
discuss
conversations
with
improve
conversations
with
conversations
afterwards.
colleagues.
residents,
relatives
or
others.
Fig. 4. Factors impacting the reflection practice of participants in both trials.
Although our observations and insights cannot be generalized, as the data from the trials
was collected with low participants numbers, they point to the value potentially created by
tools such as the Talk Reflection App that support the collaborative reflection blueprint
48
Support for collaborative reflection in healthcare: comparing two workplaces
shown in Fig. 1. Further work will have to show whether these positive pointers also show in
other cases with more users and a longer period of usage.
5 Discussion: Reflection as a Socio-Technical Task
Results of the study show different perceived value and adoption of articulation support for
conversation and interaction experiences, comments and outcomes as described in the
framework in section 2.1: While the documentation of experiences and the possibilities to
share them with others was used quite frequently, the discussion of experiences and creation
of outcomes was observed to take place outside the app (but, sometimes, by using it to refer
to its content). This is also indicated by the questionnaire about reflection support (Fig. 4).
The results thus suggest that, with regard to support collaborative reflection, the Talk Reflec-
tion app was mostly used as a tool to prepare (by documenting and sharing cases), to medi-
ate (initial reflection within the documentations) and to trigger (discussion within the group)
reflection. To support communication and articulation about experiences and the documenta-
tion of outcomes from reflection, further work needs to be done. From these findings, we
derived a classification of reflection support as primary and secondary effects happening in
technical and social processes (Table 6).
Table 6. Primary and secondary support of collaborative reflection steps. Insights derived from the
studies and separated by technical (T) and social (S) parts of socio-technical support.
Purpose of Primary Support Secondary Support
Articulation
Documentation of expe- Documentation in app (T) Verbal explanation (S)
riences
Individual reflection Reflective parts in documentation (S and T)
Collaborative Direct communication (S) Comments in app (T)
reflection
Sustaining outcomes Direct Communication (S) Documented results (T)
Our observations indicate that supporting collaborative reflection is a socio-technical design
task: In both cases positive effects on the reflection process where the result of a combina-
tion of social processes with technological components. This explicitly includes that tech-
nical support for reflection needs to be complemented by establishing organizational pro-
cesses.
Although collaborative reflection was conducted successfully in direct communication be-
tween participants, there are also trade-offs to be considered: It is necessary to leave and
share traces of experience exchange and reflection outcomes for those that did not participate
in the reflection process personally. However, during the study participants needed to be
reminded to create comments in the app or to document an outcome. The sufficient number
of documentations also shows that this problem is not caused by the effort it takes to docu-
ment a difficult conversation. On the contrary, we conclude that users have to be triggered to
also use the app for other kinds of communication. “Prompting“ (e.g., Lin & Lehman 1999),
that remembers users of other content and features of a tool, may be helpful for that (e.g. by
49
Support for collaborative reflection in healthcare: comparing two workplaces
showing questions like “Have you been in a similar situation?”). This has to be analyzed in
future studies with the Talk Reflection app.
One limiting factor of the study is that the tests were conducted in participant groups
working closely together. Physicians in case 1 and caregivers in case 2 communicate regular-
ly as they mostly work on the same wards and floors. Therefore using the comments within
the app to exchange experiences may be less beneficial and more time-consuming for them
than just talking about it on the hallway during daily work. Future studies have to shed light
on the question whether groups that do not work together so closely (e.g. different wards of
the same care home or hospital) would use those functions more often.
6 Conclusion and Outlook
This paper describes a study about support for collaborative reflection at two healthcare
workplaces. For the study the “Talk Reflection App” was used that was developed based on
empirical results and a model of collaborative reflection. Analyzing the results of the study,
we suggest that this support has to be understood and implemented as a socio-technical sys-
tem rather than a development challenge. We found that in the studies the documentation and
sharing of situations to reflect about turned out to be a crucial preparation task and trigger for
reflection, while comments as part of collaborative reflection and development of outcomes
where observed to take place mostly in direct communication between participants.
Future studies have to show whether and how these later phases of collaborative reflection
can also be supported. Prompting mechanisms as described above and other concepts are
currently tested and evaluated.
References
Boud D (1985) Reflection: Turning experience into learning. Kogan Page, London
Daudelin MW (1996) Learning from experience through reflection. Organizational Dynamics 24:36–
48.
Dyke M (2006) The role of the “Other” in reflection, knowledge formation and action in a late moder-
nity. International Journal of Lifelong Education 25:105–123.
Fleck R, Fitzpatrick G (2009) Teachers’ and tutors’ social reflection around SenseCam images. Interna-
tional Journal of Human-Computer Studies 67:1024–1036.
Hoyrup S (2004) Reflection as a core process in organisational learning. Journal of Workplace Learn-
ing 16:442–454.
Kerth NL (2001) Project Retrospectives. Dorset House
Kirkpatrick DL (1975) Evaluating training programs. McGraw-Hill Education
Kolb DA, Fry R (1975) Towards an applied theory of experiential learning. In: Cooper C (ed) Theories
of Group Processes. John Wiley, London, pp 33–58
Lin X, Hmelo C, Kinzer CK, Secules TJ (1999) Designing technology to support reflection. Education-
al Technology Research and Development 47:43–62.
Lin X, Lehman JD (1999) Supporting learning of variable control in a computer-based biology envi-
50
Support for collaborative reflection in healthcare: comparing two workplaces
ronment: Effects of prompting college students to reflect on their own thinking. Journal of Research in
Science Teaching 36:837–858.
Prilla M, Degeling M, Herrmann T (2012a) Collaborative Reflection at Work: Supporting Informal
Learning at a Healthcare Workplace. Proceedings of the ACM International Conference on Supporting
Group Work (GROUP 2012). pp 55–64
Prilla M, Pammer V, Balzert S (2012b) The Push and Pull of Reflection in Workplace Learning: De-
signing to Support Transitions Between Individual, Collaborative and Organisational Learning. Pro-
ceedings of the Seventh European Conference on Technology Enhanced Learning. pp 278–291
Scott SG (2010) Enhancing Reflection Skills Through Learning Portfolios: An Empirical Test. Journal
of Management Education 34:430–457.
Suchman L (1996) Supporting articulation work. In: Kling R (ed) Computerization and Controversy:
Value Conflicts and Social Choices,. pp 407–423
51
52
Modeling computer-supported reflective learning:
Combining a high-level timeline view with reflection
cycles and tool use
Birgit R. Krogstie1, John Krogstie1, Michael Prilla2,
1
Norwegian University of Science and Technology, Sem Sælands vei 7-9, 7491 Trondheim,
Norway
{birgitkr; john.krogstie}@idi.ntnu.no
2
University of Bochum, Institute for Applied Work Science, Information and Technology
Management, Bochum, Germany
michael.prilla@rub.de
Abstract. To help conceptualize technology enhanced reflective learning in the
workplace and guide the development of reflection tools, the Computer Sup-
ported Reflective Learning (CSRL) model has been developed in the MIRROR
FP7 EU project. This paper presents the CSRL model in its version 1.2.1, which
compared to previous versions has a stronger focus on the triggering of reflec-
tion, on changes to work resulting from reflection, and on how reflective learn-
ing processes involves transitions between levels in the organization. As part of
addressing these concerns, the model now includes a separate timeline view ex-
plicitly showing levels in the organization. The paper describes how the CSRL
model was evaluated in April-May 2013. Main findings from the evaluation are
outlined, and implications for further refinement of the model and associated
guidelines for its use are discussed.
1 Introduction
In developing technology to support reflective learning in the workplace, there are
multiple challenges. One challenge has been the lack of a conceptual model to repre-
sent the essential aspects of reflective learning in the workplace in a way that informs
the design of reflection tools. This challenge is being addressed in MIRROR, an inte-
grated research project funded under the 7th FP.
One of the tasks in the MIRROR project is to develop a Model of Computer Sup-
ported Reflective Learning (CSRL) [1]. In its final version (2.0) the CSRL model is
intended as a tool for the design of technology in the domain of computer supported
reflective learning in the workplace beyond the MIRROR project, offering a set of
model views and guidelines for their use. The rationale and use of the CSRL model
can be briefly explained as follows:
A group of stakeholders in an organization (managers, employees, and developers)
want to better understand and support reflective learning in the organization. They sit
down together and create a high-level model of the relevant reflection process(es)
with the CSRL timeline view. They also identify cycles of reflective learning connect-
ing work and reflection on work along the timeline. For each cycle they use the CSRL
53
Modeling computer-supported reflective learning - ARTEL13
cycle view to add more detail on activities and the use of tools. To consider the poten-
tial of specific tools to support the reflection, they use the CSRL tool use view to see
what purposes the tools might serve. Thus the group agrees on a shared model of the
reflective learning process(es) in the organization, informing later design decisions.
This paper addresses the MIRROR CSRL model in its version 1.2.1. Our focus is
the combined use of model views at different levels of abstraction. We present the
views and their rationale and report on an evaluation of the views in spring 2013.
The paper is structured as follows: In Section 2 we provide a brief theoretical
background. Section 3 presents the current version of the CSRL model with three
model views. In section 4 we describe the evaluation of the model. In section 5 we
discuss the findings, considering strengths and weaknesses of the proposed approach.
Section 6 concludes the paper, proposing further work to be done.
2 Background
Reflection is critical to workplace learning, enabling employees to make sense of
complex and dynamic situations [2, 3]. Boud et al. [4] (p. 19) define learning through
reflection as “those intellectual and affective activities in which individuals engage to
explore their experiences in order to lead to new understandings and appreciations”.
In the MIRROR project we consider reflective learning to be the conscious re-
evaluation of experience for the purpose of guiding future behavior, acknowledging
the need to attend to feelings, ideas as well as behavior associated with work experi-
ence. In the workplace, work and reflection are closely connected, feeding into each
other [2, 3]. Reflection on work experiences can happen close to work or with more
distance, and it can be based on human memory alone or also on the use of data col-
lected from the work process. Reflection is done individually or collaboratively [5, 6].
Through reflection, an improved understanding of the experience is created, and im-
plications, conclusions, or lessons learned can be derived and applied to work. Cycles
of reflection and learning (e.g. [7-10]) transform work experience into knowledge
applicable to the challenges of daily work. Informal learning through reflection in the
workplace can benefit from the use of technology [11-16].
Models like the CSRL model are developed over time, in collaborative sessions but
also individually and asynchronously. The CSRL model, now in its version 1.2.1, is a
result of emerging, bottom-up model development [17]. The model captures shared
insights about the domain (computer-supported reflective learning in the workplace)
among project stakeholders (scientists and representatives of user organizations) with
varying perspectives and interests. The insights come from conceptual work, user
studies, requirements elicitation, prototype development and evaluations in the project
test bed organizations. The representation of the insights in the CSRL model is based
on an iterative process of discussion, negotiation, systematic collection of input and
hands-on evaluations of the model within the project, and feedback from external
stakeholders, e.g. project reviewers. Through this process, the model aids conceptual
alignment in the project. As there are different perspectives of reflection being re-
searched in MIRROR there are also several conceptual models that are aligned with
the CSRL model in the sense that they extend it by refining aspects of reflective learn-
ing.
54
Modeling computer-supported reflective learning - ARTEL13
The CSRL model is intended as a tool to aid the design of technology for reflective
learning in the workplace. As such it provides a semi-formal description of the con-
ceptual architecture within which reflection tools should be produced and includes a
number of high-level constraints (i.e. business requirements) for more detailed tech-
nical requirements and design choices for apps. The guidelines for use of the CSRL
model are at this stage rudimentary and mainly consist of the course of actions de-
scribed in section 4. What is established is that the model is intended for collaborative
modeling of scenarios, involving developers as well as (representatives of) users.
Particularly in enterprise and organizational modeling, collaborative techniques have
been used for developing a joint high-level model of an area through a facilitated
process [18-20]. There are similarities between this type of modeling and the one used
for parts of the modeling tasks in the evaluation of the CSRL model (section 3).
The CSRL model has a role in identifying/specifying user requirements for reflec-
tion tools and guide the process towards the design of the tools. The model can also
be used to consider the roles of existing tools in a reflective learning scenario, and
help identify roles for which there is currently little support. Before describing how
we evaluated the use of the CSRL model, we outline the model with three views.
3 The MIRROR model CSRL
The version 1.2.1 of the CSRL model was developed to address certain aspects of
reflective learning in the workplace that were not sufficiently addressed in the previ-
ous version of the model. Compared to previous versions the current reflection cycle
has a stronger focus on the triggering of individual and collaborative reflection, on
changes to the work process resulting from the application of reflection outcomes,
and on transitions between reflection cycles across levels in the organization [21].
The model has also been complemented by additional views, that is, model elements
and syntax for certain purposes and phases of using the model. For example, a time-
line view (section 3.3) has been introduced to allow explicit representation of the
levels of the organization involved in the reflective learning processes.
The combination of views in the CSRL is intended to help users of the model meet
simultaneous requirements for model quality: Pragmatic quality in the sense of sup-
porting communication about scenarios by keeping things simple, and semantic quali-
ty in the sense of including enough detail to support the design of tools [22]. The main
views of the model are summarized in Table 1 and described in what follows.
Table 1: CSRL model views and their purposes
Model view Purpose
Reflection cycle Modeling the reflective learning process, focusing activities involved,
view transitions between stages, resulting changes to work, and the triggering
of new reflection cycles
High-level model of reflective learning stories, e.g. representing user
Timeline view
requirements. Providing a structure for modeling reflective learning
cycles. These can be seen as an instantiation of the more abstract reflec-
tion cycle, making it more understandable and acting as a bridge to the
more detailed design information provided in the tool use view
55
Modeling computer-supported reflective learning - ARTEL13
Model view Purpose
Showing how reflection tools support activity in the reflection cycle,
Tool use view
linking tool use to each of the stages
3.1 The reflection cycle view
The reflection cycle view of the CSRL model generically describes the learning cycle
in which reflection on work experience leads to outcomes that feed into changes to
work and/or into further reflection [21]. The main elements in the reflection cycle
view – see Figure 2 - are stages (rounded boxes), activities within the stages (text
items below the name of the stage), transitions between stages in the cycle (closed
arrows), and transitions to new reflection cycles because of a trigger (dashed arrows).
Figure 1: CSRL model v1.2.1 Reflection cycle diagram
In the current version of the CSRL model, increased attention has been given to the
transitions between stages. There are different criteria for when it is appropriate to
make a transition, e.g. when the Data (from work), Frame (for reflection), Outcome
(of reflection) or Change (to work) resulting from the activities in the stage meets
certain requirements [23]. The initiation of a transition to a new reflection cycle hap-
pens when there is a reflection trigger: a discrepancy leading to a state of discomfort
that the individual wants to overcome [24] or to other motivation to change work in
some way. Also, the individual may, for various reasons, involve others in collabora-
tive reflection [25]. Triggering can be external, when some event outside the individ-
ual (e.g. a prompt from a tool) creates awareness of the discrepancy. Note that initia-
56
Modeling computer-supported reflective learning - ARTEL13
tion of a new reflection cycle can start within an existing cycle, e.g. originating in the
Conduct reflection session or Apply outcome stages.
3.2 The tool use view
Tools can support the activities in each stage of the reflection cycle in various ways.
They can help the user by capturing data, providing data, scaffolding the process, and
simulating work [1]. Also there can be tool support for involving others (for work or
reflection) and for determining whether to proceed to the next stage in the cycle.
More information on this view is found in [21].
3.3 The timeline view
A CSRL timeline model (see Figure 2) consists of an upper part representing activity
on the work arena (e.g. doing work and planning work), and a lower part (below the
thick line) for reflection on work experience. Time runs from left to right in the dia-
gram. A model consists of a sequence of steps along the timeline. There are four types
of steps: work (yellow box, vertical lines), reflection triggering (pink arrow, check-
ered), reflection (blue box, horizontal lines), and change to work (green arrow, diago-
nal lines). The horizontal/vertical/diagonal lines and checkering have been added for
the purpose of this paper, to make the elements distinguishable in black-and-white.
Work activities may be categorized as individual, team or organization wide, as
shown in the layers above the thick line. Reflection may be individual or collabora-
tive, as indicated in the layers below the line.
Figure 2: A model in the CSRL timeline view
57
Modeling computer-supported reflective learning - ARTEL13
For each of the boxes and arrows, some information is to be specified, see Figure 3.
Model Symbol Information specified for the element
element
Work Who is doing the work? What is being done?
Trigger Why is reflection triggered? What is the objective of the
initiated reflection?
Reflection Who is reflecting? How does it happen? i.e., if desired,
name/describe the approach.
Change What change is made? Who is affected? (i.e. individual,
team, the entire organization)
Figure 3: Information specified for the elements in the timeline view
4 Evaluating the CSRL model v1.2.1
The CSRL model v1.2.1 was evaluated in April-May 2013 within the MIRROR pro-
ject. There were two main parts of this evaluation:
• First, the CSRL reflection cycle (in an interactive, “clickable” html version) and
the types of tool use (formulated as checklist questions: “How does your app help
the user…”) were applied to the MIRROR apps. The purpose here was to check
whether and how the apps supported the various steps of reflection.
• Next, the CSRL timeline view in combination with the reflection cycle view was
evaluated in a collaborative modeling exercise.
Below we report on how the evaluation was conducted and present key findings.
4.1 Evaluation of the (clickable) CSRL reflection cycle and types of tool use
The CSRL reflection cycle and the tool checklists were evaluated in May 2013 by 8
MIRROR developers and four representatives of user organizations. They individual-
ly applied the checklists to the MIRROR apps they were involved with. All apps were
in this way addressed. The participants were already familiar with the previous ver-
sion of the CSRL reflection cycle with associated types of tool use. The checklists
were used to describe existing apps that have not yet been extensively evaluated, the
answers reflecting how the developers and test beds intend or expect the apps to be
used. After the exercise, the participants filled in a questionnaire, focusing on the
usability and relevance of the clickable model and the tool checklist items.
Findings from the evaluation show that the list of tool use types (cf. section 3.2)
can be used to describe the role of apps for different reflective learning settings and
types of reflection tools. By systematically addressing the questions in the checklists
most of the developers were able to comprehensively describe the intended types of
use of their apps. Answers to the follow-up questionnaire showed that some of the
types of tool use were perceived as less clear, for instance monitoring work or provid-
ing scaffolding. The clarity of some items in the tool checklists should be refined in
58
Modeling computer-supported reflective learning - ARTEL13
the next version. Also there are some aspects of reflection and tool support that are
not explicitly addressed in any of the current items that maybe should be captured in
new items in the list. Especially, “providing scaffolding” seems to be too wide as a
category in light of current focus in the project on guidance and prompting. 6 out of
the 8 developers reported that the exercise of applying the tool checklists to their apps
lead to new ideas about app functionality or use. This means the model served its
intended role as a design tool by helping developers iteratively improve existing apps.
4.2 Evaluation of the CSRL timeline view
An evaluation of the CSRL timeline view was conducted in May 2013 by participants
from the MIRROR project. In the evaluation, 26 participants distributed over five
groups, each including at least one developer and at least one representative from an
organization using or aiming to use MIRROR tools. In the groups they created models
in the CSRL timeline view (see examples in Figure 4 and Figure 5) based on textual
scenarios/stories of reflective learning, which were taken from empirical work in the
test beds, for example a scenario of behavior in a medical emergency. Each group
worked on a scenario that had been created in advance by 1-2 group members familiar
with the app(s). The scenarios were to be of relevance to the partners and include one
or more MIRROR apps. The groups had no permanent facilitation support, but a re-
searcher circulated through the groups to answer questions on the modeling work.
Three of the scenarios prepared for the evaluation included the combined use of
two or more apps. Below is shown the result of two groups modeling their scenarios.
Figure 4: Example 1: CSRL timeline model (scenario including use of two reflection apps)
Figure 5: Example 2: Scenario including use of three reflection apps
59
Modeling computer-supported reflective learning - ARTEL13
It should be noted that all the stories modeled in the evaluation were designed to
have a “happy ending” in the form of change to work at the organizational level. The
modeling ensured a clear focus on this level (and how to get there) through the visual-
ization of the different levels.
After the timeline diagrams had been created, the CSRL reflection cycle model
was used as a template for specifying more detail about each cycle (one diagram per
cycle), in particular the use of reflection tools. Figure 6 shows one of the cycles in the
timeline in Figure 4. It can be seen that the tool “TalkReflect” is being used in the
Plan and do work stage of the cycle, whereas the tool “DocTrain” is used in the Con-
duct reflection session stage. It can also be seen that the reflection session can end in a
new reflection cycle being triggered, or in a change to work. Figure 6 also shows that
the participants understood that an outcome of reflection might also be how to im-
prove learning (in this case by “collection more data”) instead of direct changes to
work procedures.
Figure 6: Reflection cycle detailing a single cycle from the timeline in Figure 4
After the collaborative modeling exercises the participants individually filled in a
questionnaire to provide feedback on the modeling. We will briefly describe key re-
sults from analyzing the data (diagrams and answers to the questionnaires).
Modeling with the CSRL timeline view was perceived as useful by the partici-
pants. To the question “What do you think about collaboratively working on a
scenario model in this way?”, 25 out of 26 participants used positive terms, the last
pointing to a challenge without being negative.
Two questions addressed the usefulness of the timeline view and the cycle view,
respectively, for design. The question about the timeline view was: “Imagine you
were involved in a process of designing another reflective learning tool, either
60
Modeling computer-supported reflective learning - ARTEL13
from scratch or to iteratively improve an existing tool. Would you consider cre-
ating high-level models of scenarios in a way similar to what you did in this exer-
cise? Please briefly explain.”
Answers were generally positive: 23 of 26 answered
yes (many with additional remarks about the purposes for which they would use it
and/or conditions that need to be in place, especially real data from the workplace and
participation of users), two answered no (saying respectively that they would prefer to
use only personas and not scenarios, and that they thought the modeling might not be
beneficial for technical development because the cycles would change due to the
tools), and one can be considered neutral (arguing that s/he would not know because
s/he is not a developer). It is interesting to note that the two respondents (both experts
in software development) who explicitly say they would not use the model for design
were both positive about using the timeline in an early phase of a project to support
communication with users. Follow-up interviews would be interesting in these cases.
Another question addressed the use of the cycle view to aid design: “Imagine you
were involved in a process of designing another reflective learning tool, either
from scratch or to iteratively improve an existing tool. If you were to use the
CSRL cycle model to aid the design process, how would you use it?” The answers
were generally positive, but referred to many different reasons for conducting this
step, only some related to reflection tools. For instance, it was mentioned to use the
cycle model to create a scenario, to understand the problem better and identify people
involved, to ensure completeness of the scenario, to detect flaws in the combination of
tools, to check if all aspects are supported, to better understand how an app can be
useful. A couple of participants did not see the benefit of the detailing after working
on the timeline diagram. Answers to this question and other questions showed that
many participants would like to have more guidelines e.g. on how to proceed from the
timeline to the detailing of the reflection cycles.
One group, which dealt with applying certain tools in a healthcare scenario, was
observed permanently during the exercise (Figure 4 and Figure 6 show models creat-
ed by this group). The group was composed of three representatives from a German
hospital that had tested some tools mentioned in the scenario and wanted to integrate
them into their practice, and three tool designers. Besides one designer who had used
the timeline view earlier, none of the participants knew more about it than what they
had been told in the briefing before the exercise. In the beginning of the group work,
the participants focused their discussion on the scenario and had difficulties translat-
ing their utterances into elements of the model. The more experienced designer took
the lead in this and proposed how it could be done. Over time, other participants took
over this task, which shows that they got accommodated to the way of modeling.
Participants also made free use of the model elements and sometimes extended them
by writing additional text on them such as which tool could be used in certain situa-
tions. From working on the model, they sometimes switched back to talking about the
scenario and made slight changes, e.g. when they realized that it could be made more
efficient in some aspects. Sometimes they forgot to document decisions made during
these discussions and had to be reminded by other group members (predominantly the
designer who was more familiar with the approach). When they switched to describ-
ing the individual cycles taken from the timeline view, negotiation processes between
the tool designers and the hospital representatives started, e.g. when they were dis-
cussing to which extent the tool “DocTrain” can support discussions as indicated at
61
Modeling computer-supported reflective learning - ARTEL13
the bottom of Figure 6. Finally, as can be seen from Figure 4, the group created a
fairly detailed timeline model, which also contained parallel actions. The resulting
model was not as detailed as one would have expected from a professionally run
modeling workshop (Figure 6 exemplifies this well, as it outlines the process of re-
flection but does not provide enough details for third parties such as designers to un-
derstand the process fully) but was sufficiently complete for the group to agree in the
end that they had made significant steps ahead in both creating common understand-
ing of how the tools could be used to produce value in the hospital and improve the
envisioned scenario. They reported that they had had a productive workshop that ad-
vanced their idea of implementing reflection support.
From temporary observations (stepping from group to group during the exercise)
and feedback of other groups and their members, we also learned that the transition
into modeling reflection cycles with the CSRL reflection cycle diagram generally
worked, but identifying the cycles was sometimes challenging. Also the rationale for
this step was a bit unclear in the exercise, which may be due to the fact that discussing
tool features was not a main focus of the exercise (and also there was little time for
it). However, the modeling approach did enforce consideration of, and being explicit
about, what are actual triggers of reflection and the resulting changes to work (or
resulting initiation of more reflection) in the scenarios.
From the resulting timeline diagrams it can be observed that three of the groups
decided to model branching processes, which is an adaptation of the approach. The
need to model branching was commented also in the questionnaire. The diagrams also
show some other deviations from the original modeling language, e.g. positioning of
elements between two levels to express uncertainty about where an element belonged.
4.3 Strengths and limitations to the evaluation
In interpreting the findings, it is important to take several factors into account. The
participants in the evaluation were generally familiar with the model, most of them
having participated in earlier iterations of evaluating and refining the cycle model
(with associated types of tool use). All participants were aware that they were partici-
pating in a process of evaluating the CSRL model in order to improve it, with the
added benefit of getting more insight about the scenarios being modeled. This provid-
ed the participants with background knowledge on reflection cycles and made it pos-
sible to present the timeline view as a high-level view of a reflective learning process
consisting of several reflection cycles. Also the participants were familiar with the
need for the CSRL model to offer ways of representing levels in the organization, and
to be able to model scenarios involving more than one reflection tool. Prior to the
evaluation of the timeline view, the developers and some of the test bed representa-
tives had been participating in the evaluation of the clickable version of the reflection
cycle, applying the associated checklists to their specific apps. All this prior
knowledge of the CSRL model means that we cannot deduce from the evaluation
results how the model would be perceived and used by someone who does not already
know it. It is however likely that whatever is unclear to those familiar with the model,
is likely to be more unclear to somebody who is new to it. Furthermore, the findings
should be understood in light of the stage of development of the reflection tools in the
modeled scenarios, and the status of the scenarios. The tools (MIRROR apps) in ques-
62
Modeling computer-supported reflective learning - ARTEL13
tion are existing tool prototypes, most of which are currently undergoing formative
evaluation. The scenarios were developed before the exercise and did not represent
new learning needs or tool usage. The most novel element, relevant for three of the
groups, was the combined use of tools. It was in the interest of the participants (in
light of their various objectives for their work in the MIRROR project) to develop a
shared understanding of how tools can be combined to support reflective learning in
the organizations in question. The evaluation showed that all groups considered it
useful to improve their shared understanding of their scenarios
The fact that the evaluation consisted in two separate exercises means that we lack
results on the coherent, stepwise application of the cycle, timeline and tool views to a
scenario. The evaluation was done in a project that the researcher responsible for the
model was part of. This might have lead to socially wanted answers, i.e. regarding
what the participants thought about collaboratively modeling the scenarios. Also, it is
a limitation that we do not have observational data from all the collaborative model-
ing groups, which could have been used to get insight about the stepwise development
of the models, including the rationale for each step
It can be considered a strong aspect of the evaluation that the constellation of
stakeholders in each group included app designers and representatives of the user
organization or the project partner representing the type of user organization in ques-
tion. Also in most groups there were other participants that could offer additional
viewpoints. This can be considered a strong aspect of our evaluation, allowing us to
derive insights on the usefulness of the model for creating shared understanding
among stakeholders with different perspectives and interests.
5 Implications for further development of the CSRL model
Based on the evaluation of the CSRL model and the objective for the model to sup-
port the design of technology for reflective learning in the workplace we have identi-
fied main considerations for further development of the model and its guidelines: Use
of the CSRL model for communication among stakeholders, switching between levels
of abstraction, flexibility (vs. rigidity), and the need for guidelines/facilitation.
5.1 Use of the CSRL model for communication among stakeholders
The use of the timeline can be considered as an instantiation of the generic CSRL
cycle view, showing how different cycles can be connected in a reflective learning
process in a specific case. The evaluation showed that modeling reflection as in our
approach can help stakeholders improve their shared understanding of a scenario of
computer-supported reflective learning, as it was obvious in the group observed more
intensively. This came at the cost of losing accuracy and details in the model com-
pared to other ways of modeling (e.g., together with a modeling expert). The draw-
back was however outweighed by the quality of the resulting models and the effect on
collaboratively understanding and designing reflection processes and tools usage in
them. Of particular importance here was the interplay of talking about the model and
the process it represented. Instantiation of an overall model is a traditional technique
63
Modeling computer-supported reflective learning - ARTEL13
for improving understanding among stakeholders of the overall model (improving
pragmatic quality). Our evaluation results indicate that the use of the CSRL timeline
helped participants improve their understanding of how the CSRL cycle model can
used to model different specific cycles. This also made the participants recognize
aspects of this model that were difficult or unclear. The timeline view furthermore
made it possible to add detail to the CSRL cycle view by bringing in the individu-
al/organizational interplay, something that was commented as a lack of semantic qual-
ity (completeness) of an earlier version of the model [17]. Finally, the notation in the
timeline view with the bright colored elements worked well for hands-on collabora-
tive modeling. The notation is adapted to collaborative modeling sessions without
tools, for which a clear differentiation of elements (e.g. through color) is important.
5.2 Switching between different levels of abstraction
It can be considered essential to the effectiveness of modeling that the model supports
shifts between different levels of abstraction. Having different models that represent
aspects on different abstraction levels is good for comprehension of the individual
models, but it is challenging to keep the different models consistent. In this regard
there are some challenges to the current version of the CSRL model.
There is a mismatch in notation that results in some confusion. The notation in the
timeline view (Figure 2) was created with the purpose of highlighting specific aspects
and thereby shifting focus a bit as compared to the cycle view (Figure 1). The result
can however be perceived as inconsistencies between the views, which was com-
mented by some in the evaluation. The main inconsistencies relate to the representa-
tion of triggering and change to work: triggering (initiation of a new reflection cycle)
is represented with a dashed arrow in the cycle view and with a closed arrow in the
timeline view. Change to work is a transition (not a stage) in the cycle view, but one
of the four main elements in the timeline view. This means that the color-coding of
the four elements in the timeline view cannot be transferred to the four stages in the
cycle view. In the guidelines for using the model, this will have to be well explained.
Better matching between the visual format of the model elements, e.g. in the use of
colors, should be discussed as a possible improvement of the notation in order to bet-
ter support switching between different levels of abstraction. Another way to deal
with this is simplifying the model elements even more (see also section 5.3), no more
differentiating e.g. between triggers and transitions on an element level, but using
textboxes to describe both arrows. However, the differentiation of triggers and transi-
tions might not be decisive during the collaborative modeling of stakeholders, but be
best done by experts from the group post-processing the models (see section 5.4).
The timeline view turned out to be helpful to users who had not been trained to ab-
stract from concrete situations in order to model reflection formally. Compared to the
reflection cycle view, the timeline view is much closer to the understanding these
participants have of how reflection takes place in practice. The quality of the resulting
models shows that this helped people create an initial set of specifications of how a
certain set of tools can be used to support reflection in their organizations. The models
provide a starting point for further work and might not have been created in such
quality if the participants had used the more abstract main view of the CSRL model.
64
Modeling computer-supported reflective learning - ARTEL13
Considerations about consistency between levels can also be made with respect to
the connection between the activities in the steps of the reflection cycle and the types
of tool support outlined in the tool use view. This is not a question of notation but of
clarifying the actual mapping between activities and use of tools, which is not 1-1.
As part of the work on the next version of the model one should look more at the
cognitive integration [26, 27] between the different sub-languages, aligning notation
and color usage.
5.3 Flexibility (vs. rigidity) in the modeling
The need to model branching processes was seen in participants’ questionnaire an-
swers, in three of the five timeline diagrams (e.g. Figure 4 and Figure 5) and in the
group we intensively observed. The latter used branching processes to depict that the
same trigger might cause reflection on different levels or with different participants,
and to show that there can be outcomes from reflection that affect different levels of
work and thus continue the reflection process on different levels.
In collaborative modeling it is generally important to keep the modeling notation
used simple [18, 20]. The process-oriented notation used in the timeline view is no
exception to this. On the other hand the need for simple control structures (e.g. paral-
lel tasks, alternative tasks) is found in all standard process modeling notations
(BPMN, UML Activity diagrams, EPC etc.), and also in all but the simplest process
patterns [28]. Thus, the need for representing branching in processes of reflective
learning in the workplace should not come as a surprise. Simple branching should be
considered for the next version of the CSRL model. We should however avoid trying
to formalize the control-flow modeling mechanisms too much, as it will make the
language harder to use. Thus considerations for achieving semantic and pragmatic
quality must be balanced.
5.4 The need for guidelines, facilitation and experts present
In collaborative enterprise modeling [18, 19, 29] facilitation of the modeling process
is regarded as important. However, there are also approaches showing that using the
right metaphors and interaction designs, users might create certain models on their
own, without the help of an expert [30, 31]. In the evaluation we saw participants act
on their own with the modeling elements and others who needed support by more
knowledgeable members of the group. In addition, the challenges described in section
5.2 might have been diminished by some expert support. In the case of modeling re-
flection, facilitation and guidance then need to be available in different ways. For
facilitation, knowledgeable participants might be able to take the role as champions in
the model processes, as it was the case for the designer in the group we intensively
observed. Such people are much more likely to be present when modeling takes place,
e.g. in staff meetings and the like. Guidance then may be given by these users, but can
also be available from written documentation – the need to have an expert onsite
would severely slow down the process. Therefore, as argued above, keeping the com-
plexity of modeling low and providing such documentation might be most suitable in
order to keep the flexibility of modeling high enough to engage end users into it.
65
Modeling computer-supported reflective learning - ARTEL13
Expert support, however, is necessary in the post-processing and follow-up phases
of modeling reflection: If we use the user-driven model phase to gather initial models
and build common understanding of reflection processes and tool usage in them, there
is a need to clean up the models afterwards in order to make them readable and to
refine them to a level of detail and quality sufficient for implementing processes.
These steps might not be feasible without an expert, who knows about the levels of
detail and may also detect flaws and contradictions. Further research will thus also
include finding a away to let stakeholders model as much as possible on their own,
while using expert support for tasks stakeholders cannot do well on their own.
6 Conclusion
Based on results from the evaluation of the MIRROR CSRL model we have argued
that the CSRL model has pragmatic quality supporting communication about reflec-
tive learning scenarios among different stakeholders. The model also has a semantic
quality that allows the representation of more detail to aid the design of computer
support for the scenarios.
To aid the development of the next version of the CSRL model, evaluation of the
stepwise application of all views of the model should be conducted. It would be par-
ticularly useful to do this with subjects who are not already familiar with the model.
This will also address the question whether the model and its views are equally usable
for different stakeholders such as experts, users or developers.
One question that needs to be addressed in the work on the next version of the CSRL
model with associated guidelines is how far into the process of developing CSRL
tools the model should be used. Currently we do not see CSRL model as a tool for
supporting design in the steps immediately preceding coding, but as a conceptual tool
to help identify and agree on user requirements and build an understanding of how
reflection tools could meet these requirements. The guidelines for use of the CSRL
model needs to be clear about the role of the different model views with respect to
different phases in a process of developing CSRL solutions.
7 References
1. Krogstie, B., et al., Computer support for reflective learning in the workplace: A model, in
Proceedings International Conference on Advanced Learning Technologies (ICALT 2012)
2012, ACM: Rome.
2. Lave, J., The practice of learning, in Understanding Practice: Perspectives on Activity and
Context, S. Chaiklin and J. Lave, Editors. 1993, Cambridge University Press: Cambridge.
p. 20.
3. Schön, D., The Reflective Practitioner1983: Basic Books, Inc.
4. Boud, D., R. Keogh, and D. Walker, Reflection: Turning Experience into Learning1985:
RoutledgeFalmer.
5. Høyrup, S., Reflection as a core process in organisational learning. Journal of Workplace
Learning, 2004. 16(8): p. 13.
66
Modeling computer-supported reflective learning - ARTEL13
6. vanWoerkom, M. and M. Croon, Operationalising critically reflective behavior. Personnel
Review, 2008. 37(3): p. 15.
7. Cress, U. and J. Kimmerle, A systemic and cognitive view on collaborative knowledge
building in wikis. Computer-Supported Collaborative Learning, 2008. 3: p. 105-122.
8. Kolb, D.A. and R. Fry, Towards an applied theory of experiential learning, in Theories of
Group Processes, C.L. Cooper, Editor 1975, John Wiley: London. p. 33-58.
9. Korthagen, F. and A. Vasalos, Levels in reflection: Core reflection as a means to enhance
professional growth. Teachers and Teaching: Theory and Practice, 2005. 11(1): p. 25.
10. Stahl, G., Building collaborative knowing, in What We Know About CSCL And Imple-
menting It In Higher Education, J.-W. Strijbos, P.A. Kirschner, and R.L. Martens, Editors.
2002, Kluwer Academic Publishers: Boston. p. 53-85.
11. Kim, D. and S. Lee, Designing Collaborative Reflection Support Tools in e-project Based
Learning Environment. Journal of Interactive Learning Research, 2002. 13(4): p. 375-392.
12. Krogstie, B.R. and M. Divitini. Supporting Reflection in Software Development with Eve-
ryday Working Tools. in COOP. 2010. Aix-en-Provence, France: Springer.
13. Li, I., A. Dey, K., and J. Forlizzi, Understandinng My Data, Myself: Supporting Self-
Reflection with Ubicomp Technologies, in Ubicomp'112011: Bejing, China.
14. Lin, X., et al., Designing Technology to Support Reflection. Educational Technology,
Research and Development, 1999. 47(3): p. 43-62.
15. Siewiorek, N., et al., Reflection Tools in Modeling Activities, in Proceedings ICLS2010,
ISLS: Chicago.
16. Xiao, L., et al. Promoting Reflective Thinking in Collaborative Learning Activities. in
Eighth IEEE International Conference on Advanced Learning Technologies (ICALT).
2008. Santander, Cantrabria, Spain: IEEE.
17. Krogstie, B.R., et al., Collaborative Modelling of Reflection to Inform the Development
and Evaluation of Work-Based Learning Technologies in i-KNOW2012, ACM ICPS: Graz,
Austria.
18. Gjersvik, R., J. Krogstie, and A. Følstad, Participatory Development of Enterprise Process
Models, in Information Modelling Methods and Methodologies, J. Krogstie, K. Siau, and
T. Halpin, Editors. 2004, Idea Group Publishers.
19. Prilla, M. and I. Jahnke, The Socio-Technical Walkthrough for participatory process de-
sign, in Implementing International Services. A tailorable method for market assessment,
modularization and process transfer 2011, Gabler.
20. Stirna, J., A. Persson, and K. Sandkuhl, Participative Enterprise Modelling: Experiences
and Recommendations, in Proceedings Advanced Information Systems Engineering, 19th
International Conference, CAiSE J. Krogstie, A.L. Opdahl, and G. Sindre, Editors. 2007,
Lecture Notes in Computer Science 4495 Springer: Trondheim, Norway.
21. Krogstie, B., M. Prilla, and V. Pammer, Understanding and Supporting Reflective Learning
Processes in the Workplace: The RL@Work Model, in ECTEL 2013, Springer: Paphos,
Cyprus.
22. Krogstie, J., V. Dalberg, and S.M. Jensen, Harmonising Business Processes of Collabora-
tive Networked Organization Using Process Modelling, in Proceedings PROVE 2004: Tou-
louse, France.
67
Modeling computer-supported reflective learning - ARTEL13
23. Krogstie, B., Prilla, M., Pammer, V., Understanding and Supporting Reflective Learning
Processes in the Workplace: The RL@Work Model, in Proceedings of the Eigth European
Conference on Technology Enhanced Learning (EC-TEL 2013).
24. Knipfer, K., et al., Reflection as catalyst for organisational learning. Studies in Continuing
Education, 2012. 35: p. 19.
25. Prilla, M., V. Pammer, and B. Krogstie, Fostering Collaborative Redesign of Work Prac-
tice: Challenges for Tools Supporting Reflection at Work, in In press2013.
26. Krogstie, J., Model-based Development and Evolution of Information Systems: A Quality
Approach 2012, London: Springer.
27. Moody, D.L., The “Physics” of Notations: Toward a Scientific Basis for Constructing Vis-
ual Notations in Software Engineering. IEEE Transactions on Software Engineering, 2009.
35(6).
28. van der Aalst, W.M.P., et al., Workflow Patterns. Distributed and Parallel Databases, 2003.
14(1): p. 5-51.
29. Persson, AA., P. and Stirna. J., Towards Defining a Competence Profile for the Enterprise
Modelling Practitioner, in proceedings PoEM The Practice of Enterprise Modelling 2010,
Springer: Delft, the Netherlands.
30. Prilla, M., Nolte, A., Integrating ordinary users into process management: Towards imple-
menting bottom-up, people-centric BPM. Enterprise, Business-Process, and Information
Systems Modelling, in Proceedings of the International Conference on Business Process
Modeling, Development and Support (13th edition) in conjunction with CAISE’12
(BPMDS 2012), Springer.
31. Nolte, A., Prilla, M., Anyone can use models: Potentials, requirements and support for non-
expert model interaction. International Journal for e-Collaboration (2013).
68
Improving awareness and reflection through
collaborative, interactive visualizations of badges
Sven Charleer, Joris Klerkx, Jose Luis Santos and Erik Duval
Dept. of Computer Science, KU Leuven, Celestijnenlaan 200A
3001 Leuven, Belgium
{Sven.Charleer,Joris.Klerkx,JoseLuis.Santos,Erik.Duval}@cs.kuleuven.be
Abstract. This paper introduces novel ways of improving awareness
and reflection through visualizations of badges as an abstraction of learn-
ing analytics data. We report initial findings with both a personal dash-
board approach, Navi Badgeboard, that provides details on student and
class progress, and a collaborative, interactive tabletop visualization,
Navi Surface, to promote group reflection. We evaluate both approaches
to find improvements among students regarding awareness and reflec-
tion on course activities. Our results indicate that Navi Badgeboard
helps with awareness of personal activity while Navi Surface improves
collaboration resulting in better reflection.
Keywords: learning analytics, learning dashboards, collaboration, re-
flection, awareness, visualization, badges
1 Introduction
Feedback and collaborative discourse, between student and teacher, among stu-
dents and even with external parties, leads to significant gains in learning [2].
Traditional tools for this are exams, class discussions, self-assessment and peer
evaluations, but also (micro-)blogging (Twitter, Wordpress, Facebook) can help
students share and reflect on their work, collaborate, discuss and learn from
class mates [10]. These processes leave behind a multitude of learner traces that
reflect activity and progress of students [17]. We strongly believe that visual-
izations of these traces in so-called learning dashboards can assist in creating a
feedback loop of awareness, reflection, sense-making and impact [17] and improve
motivation [13].
In previous work, we have developed visualizations of learning traces through
tools such as StepUp![14], SAM[6] and TinyARM[13]. While these traces pro-
vide a broad insight on student activities, the abundance of information can be
overwhelming for both student and teacher, even when presented through bar
charts, line charts and parallel coordinates.
This paper focusses on our attempts to visualize an abstraction of trace data.
We limit the data to the essential course goal settings. The goal is to assist in
informing the students individually about their progress and to enable discus-
sion in class. This abstraction can be achieved by defining badges for activities.
69
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
We visualize the data through two applications that we have developed in an
attempt to improve awareness and reflection: a personal learning dashboard to
support individual awareness and reflection and an interactive visualization on
a multitouch tabletop to support collaborative awareness and reflection.
In section 2, we will discuss more examples of learning dashboards, collab-
orative visualizations and the use of badges in learning. Section 3 explains the
setting of the course we use to evaluate our tools and explains the badge sys-
tem in more detail. Section 4 elaborates on 2 approaches: the personal dashboard
and the collaborative visualizations. Section 5 details the evaluation of the tools,
followed by some ideas for future development and conclusions in section 6.
2 Related Work
The Quantified Self (QS)1 movement, which focusses on collecting user traces
and using data for self-improvement, is spreading across multiple domains but is
probably best known for its application in personal health [16]. Through mobile
phone apps and an increase in cheap tracking devices (e.g. Fitbit2 , Nike+3 ),
people can become more aware of their health and modify their behavior by
tracking activities such as walking, sleeping, running, etc. A similar change in
behavior can be achieved with students by applying QS to learning, tracking the
traces students leave behind through e.g. blogging, time tracking etc. [4]
These learning traces can help students become more aware of their activities.
By visualizing these traces through interactive dashboards, students and teachers
are provided with better ways of exploring and understanding this abundance of
data [14][6]. Personal dashboards can also be populated with grade and badge
data [8].
Badges, which are essentially another form of abstraction of the tracked data,
bring with them many benefits and uses: The creation process of the badges can
influence the design of the course [7] and hence create clearer goals for both
student and teacher. Badges can be used as feedback and are proven to directly
impact behavior and motivate students in off- and online courses [11][13][7]. Skill
recognition can be brought outside the classroom to support life long learning
by using badges as certifications in e.g. Massive Open Online Courses [7][5].
As collaboration can have a serious impact on learning [2] and the possibilities
of collaborative visualizations are yet to be explored more fully [9], this paper
does not only look at personal learning dashboards but also at ways of com-
bining interactive discourse with learning traces and more specifically badges.
Our research focusses on awareness and reflection through tabletop displays: as
a collaborative tool in a formal setting, they cause students working in small
groups to articulate and reflect on their insights more than while using more
conventional displays [15] or paper [12]. They can also be used in more infor-
mal settings, a public place like a university hall where collaboration between
1
http://quantifiedself.com
2
http://www.fitbit.com
3
http://nikeplus.nike.com
70
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
strangers helps increase awareness and reflection regarding a specific topic [1].
We focus on how to improve awareness and create a better insight of the learning
process.
3 Learning Analytics Data
3.1 Course Setting
We evaluated our tools in a class of 26 engineering students between the ages
of 20 and 25 following a course on Human-Computer Interaction (HCI). This
year, the course focusses on the design, development and evaluation (usability,
usefulness) of a recommendation application. Students work in groups of 3 and
improve their application through iterative development. The course, while it in-
cludes face-to-face studio sessions on the Science Campus of KU Leuven, is open
to everyone. All data including presentations, course material and reports is pub-
licly accessible online, through Slideshare4 , the course wiki5 and the group blogs
which students are required to update regularly. They use Twitter with a course
specific hashtag (#chikul13 6 ). Discourse happens through class discussions and
comments on each other’s blogs and tweets.
3.2 Learning Traces
The student generates data by blogging, commenting and tweeting. These ac-
tivities leave behind traces that can be used by learning dashboards to visualize
activity and progress. Both individual and group activity can be visualized, but
also data on interaction between students, groups and even interaction with ex-
ternal people is available. Visualizations can help students become aware of their
activities and compare their performance to evaluate their progress in class.
Automated trackers hook up to the RSS feeds of the blogs and connect to
the Twitter API. They gather student generated data, store it in a database and
make it accessible to other applications through a REST service. On these ser-
vices, learning dashboards can be developed visualizing the data. This framework
is explained more in detail in [13].
3.3 Badges
In previous attempts, we have focussed on showing all the collected data. While
these dashboards provide an abundance of information, it is interesting to limit
the feedback to the essentials (e.g. in the HCI course: regular blog activity, com-
menting activity). Badges help abstract the data and create a more generalized
overview of the traces. This can provide a better understanding of the goals
settings and the required activity.
A badge is set up as follows:
4
http://slideshare.net
5
http://ariadne.cs.kuleuven.be/wiki/index.php/Chi_2013
6
https://twitter.com/search/realtime?q=chikul13&src=typd
71
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
– a badge icon with an easily identifiable image related to the semantics of the
badge (see Fig. 1).
– a color coding for categorizing the badge by type (positive, negative, neutral).
– a bronze/silver/gold medal for badges indicates different levels of achieve-
ments.
– a textual description on how the badge can be achieved.
To define what badges we use in our HCI course, we looked at the activities
that are important. Blogging and Twitter play a big role in the course as they
are indicators of commitment and collaboration, so we want to award this be-
havior. For example, badges are awarded for a specific number of tweets, posts,
comments. High comment activity on a post is usually triggered by the content
of that post. Therefore badges can also indicate quality of the content of a blog
post which can be derived from the number of comments it receives, by internals
(students, teachers) and externals (visitors). Inactivity is a behavior students
should attempt to avoid and can be detected through lack of digital traces, that
we translate into a negative badge. In total we defined 51 badges of which 42
were awarded repeatedly bi-weekly. The full list of 51 badges can be found at
http://navi-hci.appspot.com.
Badges are automatically assigned. Certain activity in the tracked data will
trigger an event when requirements are met and a mail is sent to the student
with information on the awarded badge. This badge data is also stored and can
be accessed through a REST service, creating an open data framework on which
other visualizations can easily be developed.
We follow the Mozilla OpenBadge Standard7 , so that students can choose to
publish their awards on social networks.
In the next section, we will discuss how these badges are shared among
students, in an individual and collaborative way.
4 Personal and Collaborative Interactive Visualizations
4.1 Two Approaches
Badges can assist in informing students individually about their progress but
can also play the role of a catalyst for discussion. We present two methods that
attempt to increase awareness and reflection in quite different ways.
The first approach relies on personal dashboards. Navi Badgeboard provides
an overview of achievements and progress. Students can also compare progress
with that of the class. This tool is used in a personal way, usually outside of
class on a desktop computer or mobile device. The data is open and public and
therefore not only students can access each others’ dashboard, but also teachers
and externals.
While the first approach gives students the opportunity to check their progress
on their own time, the second approach is more controlled and direct by facing
7
http://openbadges.org
72
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
students with their achievements (or lack thereof) in a public setting. One way
to achieve this is by projecting Navi Badgeboard in class while the teacher mod-
erates a discussion around specific badges. As only the teacher is in control of
this projection, students play a more passive role in this discussion. To create a
more active discussion where all parties have equal power in steering the con-
versation, a more interactive visualization is necessary with support for multiple
users. Therefore we take our second approach to tabletop displays, creating Navi
Surface which allows students and teacher, or groups of students, to create and
moderate a more open and deeper reflection discourse.
4.2 Navi Badgeboard
Through Navi Badgeboard, the user can discover the badges and therefore the
intended course learning outcomes of the HCI course and find more detailed
information on how these badges can be achieved. It also presents the user with
a list of all students participating in the course. From this list, each student’s
Personal Badge Dashboard can be accessed.
Personal Badge Dashboards contain a list of badges per bi-weekly period (see
Fig. 1): greyed-out badges have not yet been acquired by the student, colored
badges have. The user can cycle through the bi-weekly periods.
Fig. 1. Navi Badgeboard - Personal Badge Overview: A student’s badge overview for
a given period. Colored badges have been achieved, greyed-out ones have not. The
number next to the badge (e.g. the number highlighted by the red circle) indicates how
many students in class have been awarded this badge.
The number next to the badge indicates the total times the badge has been
awarded to the class in the selected period. A high number next to a grey icon
indicates the student is one of the few students without the badge. A low number
next to a colored icon indicates the student is one of the few to have earned this
badge. Depending on the badge, either situation can be a good or a bad thing.
From the Personal Badge Overview, the user can drill down to a Class Pro-
gression View, a visualization of the class’s badge progression over time. An
example is shown in figure 2 where the X-axis represents time and the Y-axis
73
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
Fig. 2. Navi Badgeboard - Class Progression: Every line represents a badge. Circles
represent the moment in time the student has been awarded the respective badge. The
user can choose which badges to visualize through the filter options.
the number of students that have been awarded a specific badge. Every colored
line represents the progression of a badge for the class. The circle indicates when
a particular student was awarded this badge. The Class Progression View gives
the user an immediate idea of how early or late the student is at achieving a
specific badge compared to the rest of the class.
Students can also share their badges as a way of skill recognition to exter-
nal parties. Through a ’send to backpack’ button, students with Mozilla Open
Badges8 accounts can send these badges to their Mozilla Open Badges Backpack
and publish them on social network sites.
Navi Badgeboard is developed using HTML5, Javascript, D3.js9 and Java
deployed on the Google App Engine10 . This application can be accessed from
mobile devices and desktop browsers and is publicly available 11 .
4.3 Navi Surface
An interactive visualization can help students get a better understanding of the
course activity data. Adding collaboration to the process creates opportunities
for a more active discourse around the data. To enable such collaboration, we
developed Navi Surface, a first prototype of a multi-user multi-touch tabletop
display application developed using HTML5, Javascript and Paper.js12 .
8
http://openbadges.org
9
http://d3js.org
10
https://appengine.google.com
11
http://navi-hci.appspot.com
12
http://paperjs.org
74
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
Navi Surface presents the users with a list of students and badges available in
the course. The badges are displayed per bi-weekly period and the user can cycle
through these periods. This information is located at the bottom of the display
and is interactive: each student name and badge can be touched and dragged.
The remainder of the screen is called the Playfield. All interactive items can
be dragged onto the Playfield. The badges in the Playfield light up the names
of students that have been awarded these badges. Student names light up the
badges that have been awarded to the respective students. Dropping badges onto
the Playfield also displays their detailed information.
Touching and holding an item will activate the relationship visualization:
lines will connect the item to all its related items on the Playfield e.g. a student
name will be connected to all its awarded badges. As the application supports
multi-touch, multiple items can be moved and touched simultaneously, creat-
ing more interesting visual relationships (see Fig. 4) and enabling collaborative
interaction with the data.
Fig. 3. Navi Surface: The bottom left shows the list of badges of a specific period.
The bottom right contains the students’ names. The items in the Playfield (top) are
touched and held to display the relationships between them.
The tabletop display can be placed in the classroom. As the CHI course
format is a studio session, students can freely get up and walk to the tabletop
to access their own and class mates’ information. The teacher can also invite
students to the tabletop display. In our CHI course, as students work in groups
of 3, the teacher can invite a group to discuss their progress. While the teacher
can guide the process by dragging items onto the Playfield, students can also
interact and steer the conversation. This engagement causes a personal but also
collaborative interaction and evaluation.
75
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
Fig. 4. Navi Surface: Students actively using the tabletop display application during
our evaluation session.
5 Evaluation
5.1 Navi Badgeboard
22 of the 26 students between the ages of 20 and 25 participated in the evaluation
of Navi Badgeboard. First we evaluated the usability of Navi Badgeboard using
the System Usability Scale (SUS)[3]. It scored 65 which means it scored below
average. We also asked students to fill in an online survey that goes more into
detail on the different functionalities and visualizations which revealed additional
interesting information.
Fig. 5. Navi Badgeboard Functionality Importance Questionnaire: A) Overview of
other students’ badges B) Overview of number of badges achieved by the entire class
C) Overview of all badges and descriptions D) Filter options for the Class Progression
graph E) Class Statistic graph per badge F) Add to OpenBadge backpack
Figure 5 shows how important students consider the different functionalities
using a 5-level Likert item (2 - Not at all important, 6 - Extremely important)
with one extra option (1 - Did not know the feature existed). Students mostly
agreed the dashboard gives an accurate representation of the activity of the
76
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
class. The overview of the badges which shows the students what badges are
achievable in class was rated important. The global overview of the class’s pro-
gression however was deemed less important. As students also seemed to have
less preference for class mates’ personal dashboards, we can assume that they
were more interested in gaining awareness of their own goals and tasks than
awareness of others. This is also confirmed by the lack of interest for the Class
Progression View. These views were very valuable to the teacher and teacher
assistant however.
Fig. 6. Navi Badgeboard questionnaire: A) The system increases my motivation B)
The system promotes commenting on blogs of other students. C) The system promotes
reading blogs of other students. D) The system promotes Twitter participation.
Figure 6 shows the results of 4 5-scale (1- Strongly disagree, 5 - Strongly
agree) Likert questions. Students believed motivation was improved through
the Navi Badgeboard. They were under the impression it promotes commenting,
reading blogs and Twitter, activities these badges were designed to impact. While
this was the case for Navi Badgeboard, our dashboards showing raw activity data
through lists, tables and charts had a lesser impact on motivation [13].
The Google Analytics data showed that most activity happened around the
CHI course’s studio sessions. It can be assumed that students check up on their
badges before a session starts. Students are also notified by email when awarded
a badge. Two students mentioned that they would only visit Navi Badgeboard
at such occasions.
While we have included Open Badges support, almost no students were in-
terested in or even aware of this functionality. While we could improve the usage
of this feature by making it more prominent, most badges might have been too
specific to the course process, giving the students no incentive to share them
externally.
We can conclude that Navi Badgeboard does improve awareness of the goals
and tasks required to successfully complete the course through the Badge Overview
page. Students also regularly check up on their own progress, usually before a
77
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
class session begins. We can assume students wish to reflect on their progress
through these regular visits. However, they seem less interested in other students’
achievements.
5.2 Navi Surface
While we will evaluate further prototypes of Navi Surface during course sessions,
this initial evaluation of the tabletop display application took place during a
poster session just before the end of the CHI course. This gave us the opportunity
to not only evaluate the tool with CHI students but also outsiders. 14 students
walked up to the tabletop to test the application using the think-aloud protocol.
Students were left to experiment alone or in group and hints were only given
when the participant(s) got stuck. Student actions were recorded on video and
they were given a questionnaire afterwards. 10 students approached and tested
the tabletop in groups: 2 groups of 2 students and 2 groups of 3. Not all students
were part of the CHI course.
The application received a SUS score of 71 which is just above average.
However, only taking into account the CHI course students, the application
received a score of 77. We assume that due to the abstract nature of the data, in
its current form, Navi Surface does not give enough insight on the course content
which makes it harder to use for outsiders. We will discuss how we can improve
this further in section 6.
The goal of Navi Surface is to provide better understanding of the data and
thus also increase awareness through collaborative interaction. This collaborative
interaction should also ignite further discussions to create a deeper reflection.
Fig. 7. Navi Surface questionnaire: A) The visualization improves my awareness of
the class’s general progress B) The visualization helps me understand the meaning of
the badges C) The visualization helps me see what badges someone earned in a given
period D) The visualization helps me compare badges of students E) I would like to
use this tool together with the teacher to evaluate my progress F) I would like to use
this tool together with other students to compare our progress
78
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
Figure 7 contains box plots of the results of the 5-scale Likert questionnaire (1
- Strongly disagree, 2 - Strongly agree). Students do not believe their awareness of
class progression was in any way improved. This result was expected as the first
prototype only shows badges for the students dragged into the Playfield area.
Only after dragging all names onto the tabletop would the user get a better idea
of class status regarding the awards. As badges are only shown per period, an
overview of the entire course length is also not available.
Navi Surface was built with multi-user collaboration in mind and while a
single user experience is possible, it was not the goal of the application. This
matches our finding in the questionnaire: there was a preference of using the tool
in group. There was also an interest in using the tool together with a teacher.
Observation of the students while using the tool also confirmed that collab-
oration improved the reflection process as students understood the tool and the
data much quicker. After getting a good grasp of what Navi Surface provided,
they spontaneously started discussing their progress based on the badges. They
reflected on why and how certain badges were achieved and others were not.
They also experienced this collaboration to be more fun.
On the other hand, students faced with Navi Surface by themselves were
more hesitant and needed input from the observer to continue using the tool.
While the questionnaires confirm that they prefer to use it in group, it is clear
that without the collaboration and social discourse this interaction enables, the
actions are less spontaneous and much less deeper reflection occurs.
The tool was also tested on students who were not part of the CHI course.
This however proved less successful especially with students faced with the tool
by themselves. As Navi Surface does not provide any details on the actual data
behind the badges (blog posts, comments and tweets), the data is very unclear
to outsiders. This also affected the SUS score (see above). In section 6, we will
discuss how we can provide more detailed information and hereby also make
Navi Surface more interesting to outsiders.
6 Conclusion and Future Work
Learning dashboards provide a means of visualizing the abundance of traces
which learning analytics allows us to collect from students. We look at simplifying
the data by emphasizing the more important student activities and course goals
and visualizing these through badges.
While the personal dashboard has improved perceived awareness with stu-
dents and the overview of class progress was deemed valuable by teachers and
teacher assistants, we believe Navi Surface has more potential in helping aware-
ness and reflection with students through its collaborative nature. In its current
state, Navi Surface already enables students to understand the data quicker and
plays a catalyst in discussions. Navi Surface makes the process fun and students
show interest in using this tool with class mates but also with teachers.
More evaluations will give us deeper insights and we believe that there are
still many unexplored possibilities which makes further development of this tool
79
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
very interesting. By simply adding more course data to the visualizations and
allowing students to drill down on badges to reveal more data, students could
discover why and how certain badges have been awarded (e.g. the specific blog
post or comment that triggered a badge), creating a better insight of the progress
and a deeper reflection on the learning process.
Adding more detailed course data to the visualization does not only benefit
student and teacher, but could help outsiders comprehend the inner workings of
a course better. Open school days can help students choose their future classes
based on the real data provided through Navi Surface. Parents’ evenings can
become more interactive as parents and teacher can utilize the tabletop to dig
deeper into the details of the learning process of son or daughter.
While we mainly work with blog and Twitter data, Navi Surface can easily be
extended to support even richer learning analytics data. Students in a more in-
quiry based learning environment leave behind richer artifacts (e.g. photographs,
geographical coordinates) which could provide even more interesting visualiza-
tions and therefore an even better insight through Navi Surface. Even MOOCs
(Massive Open Online Courses) can benefit from the abstraction to badges and
the visualization through Navi Surface.
With few developments, many new possibilities open up. We will therefore
continue our research into badges and tabletop displays as a way of improving
awareness and reflection in the class room and beyond.
7 Acknowledgement
The research leading to these results has received funding from the European
Community’s Seventh Framework Programme (FP7/2007-2013) under grant agree-
ment No 318499 - weSPOT project.
References
1. Antle, A.N., Bevans, A., Tanenbaum, J., Seaborn, K., Wang, S.: Futura: design for
collaborative learning and game play on a multi-touch digital tabletop. In: Pro-
ceedings of the fifth international conference on Tangible, embedded, and embodied
interaction, ACM (2011) 93–100
2. Black, P., Wiliam, D.: Assessment and classroom learning. Assessment in education
5(1) (1998) 7–74
3. Brooke, J.: Sus: A quick and dirty usability scale, pages 189–194. usability evalu-
ation in industry (1996)
4. Duval, E.: Attention please!: learning analytics for visualization and recommen-
dation. In: Proceedings of the 1st International Conference on Learning Analytics
and Knowledge, ACM (2011) 9–17
5. Goligoski, E.: Motivating the learner: Mozilla?s open badges program. Access to
Knowledge: A Course Journal 4(1) (2012)
6. Govaerts, S., Verbert, K., Duval, E., Pardo, A.: The student activity meter for
awareness and self-reflection. In: Proceedings of the 2012 ACM annual conference
extended abstracts on Human Factors in Computing Systems Extended Abstracts,
ACM (2012) 869–884
80
Improving awareness and reflection through collaborative, interactive visualizations of
badges - ARTEL13
7. Higashi, R., Abramovich, S., Shoop, R., Schunn, C.: The roles of badges in the
computer science student network. In: 2012 Games+Learning+Society Conference.
(June 2012)
8. Holman, C., Aguilar, S., Fishman, B.: Gradecraft: what can we learn from a game-
inspired learning management system? In: Proceedings of the Third International
Conference on Learning Analytics and Knowledge, ACM (2013) 260–264
9. Isenberg, P., Elmqvist, N., Scholtz, J., Cernea, D., Ma, K.L., Hagen, H.: Col-
laborative visualization: Definition, challenges, and research agenda. Information
Visualization 10(4) (2011) 310–326
10. Lin, W.J., Liu, Y.L., Kakusho, K., Yueh, H.P., Murakami, M., Minoh, M.: Blog
as a tool to develop e-learning experience in an international distance course. In:
Advanced Learning Technologies, 2006. Sixth International Conference on, IEEE
(2006) 290–292
11. McDaniel, R., Lindgren, R., Friskics, J.: Using badges for shaping interactions in
online learning environments. In: Professional Communication Conference (IPCC),
2012 IEEE International, IEEE (2012) 1–4
12. Piper, A.M., Hollan, J.D.: Tabletop displays for small group study: affordances of
paper and digital materials. In: Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems, ACM (2009) 1227–1236
13. Santos, J.L., Charleer, S., Parra, G., Klerkx, J., Duval, E., Verbert, K.: Evaluating
the use of open badges in an open learning environment. In: Proceedings of the
European Conference on Technology Enhanced Learning. (accepted)
14. Santos, J.L., Verbert, K., Govaerts, S., Duval, E.: Addressing learner issues with
stepup!: an evaluation. In: Proceedings of the Third International Conference on
Learning Analytics and Knowledge, ACM (2013) 14–22
15. Shaer, O., Strait, M., Valdes, C., Feng, T., Lintz, M., Wang, H.: Enhancing ge-
nomic learning through tabletop interaction. In: Proceedings of the 2011 annual
conference on Human factors in computing systems, ACM (2011) 2817–2826
16. Smarr, L.: Quantifying your body: A how-to guide from a systems biology per-
spective. Biotechnology Journal 7(8) (2012) 980–991
17. Verbert, K., Duval, E., Klerkx, J., Govaerts, S., Santos, J.L.: Learning analytics
dashboard applications. American Behavioral Scientist (0002764213479363) (2013)
81
82
Feeler: feel good and learn better
A tool for promoting reflection about learning and Well-being
Eva Durall1, Tarmo Toikkanen1
1
Learning Environments research group. School of Arts, Design and Architecture. Aalto
University. Helsinki, Finland
{eva.durall, tarmo.toikkanen}@aalto.fi
Abstract. In this paper we present Feeler, a design-in-progress tool for visuali-
zation of learning performance and well-being with the aim of fostering reflec-
tion and awareness. The project combines two currently promising areas such
as Personal Informatics and Learning Analytics in order to encourage learners
to reflect about their lifestyle and its impact on their learning capabilities. It is
expected that allowing learners to capture and visualize quantitative data about
their states and habits will offer them rich materials that support individual and
collective reflection-after-action processes. This project builds on participatory
design and a research-based design process. Currently, the project is in a prod-
uct design stage. The aim of the project is to develop a working prototype that
follows a slow technology approach that can be tested in learning contexts.
Keywords. research-based design, information visualization, reflection, aware-
ness, learning analytics, personal informatics
1 Introductory Scenario
Saga has difficulties to stay focused on her studies and she feels stressed because she
can hardly complete the tasks. In a tutoring session, Saga’s tutor suggests her that
doing some regular exercise could actually help her to stay focused. Although Saga is
skeptical, her tutor convinces her to use Feeler, a system that monitors her concentra-
tion levels and the amount of physical activity she has during a certain amount of
time.
Feeler combines a head band that tracks brain activity and smart textiles to visual-
ize the data. Small led lights are integrated in two wool wrist bands and they blink
when the person loses attention for a certain time. Thanks to this gentle reminder Saga
is more aware of her current capabilities and acts according to what her body needs.
The light signal helps her to decide when to change the type of task or take a break.
Data about exercise habits is registered through a mobile app. Information about con-
centration levels and physical activity is displayed in a screen. This allows Saga to
identify patterns between the amount of physical activity and how long she is able to
keep her attention. After a while of using Feeler, Saga realizes that after moderate
exercise, she is able to keep concentrated for longer periods of time. She discusses
83
Feeler: feel good and learn better - ARTEL13
this with her peers and with her tutor and gets some suggestions about how to better
plan her schedules.
2 Quantified-Self: a Tool for Self-Understanding
In many societies, computers have become an everyday tool that has adopted diverse
forms: laptop, smartphones, tablets… The combination of these devices with Internet
access and sensors has allowed people to collect data about a myriad of personal as-
pects dealing with physiology, behavior, habits and thoughts. In this context, the
Quantified-Self movement has appeared as a way to develop self-knowledge through
data. The availability of measurable personal data can be used, as [13] highlight, “for
self-reflection to help people become more aware of their own behavior, make better
decisions, and change behavior”. (p.405)
Personal informatics, also known as Quantified-Self, has become quite popular in
fields dealing with sports and health. In sports, some of the currently well-known
body tracking products include Nike+ and its fuelband1, Fitbit2, RunKeeper3 and
Moves4. Concerning wellbeing, applications such as Withings5, HeartMath6, mind-
bloom7 and Ubifit Garden8 offer opportunities to users to learn about their progression
and undertake new challenges concerning healthy habits.
In the field of e-learning, learning analytics takes advantage of the possibilities of
data monitoring in order to understand and improve teaching and learning. Despite the
intention is to empower teachers and learners, some critical voices [2] have warned
that analytics could disempower learners by making them reliant on the institution
feedback.
Considering the key role of self-knowledge for self-regulation and metacognition,
self-understanding should be at the center of systems that monitor student data. In this
sense, some authors [5, 3, 10] have noted that learning analytics should be considered
as a tool for the student. Similarly, [17] highlight the need for a Self-Directed Learn-
ing approach in which students feel ownership, as well as they are able to self-manage
and self-monitor their own learning. From this perspective, everything should be ori-
ented to help learners to take control of their own learning processes and experiences.
In order to encourage self-understanding of learning processes, it is crucial to stop
considering learning as an isolated activity that does not interrelate with other aspects
of peoples’ lives. In general, educational institutions should understand that they are
only one venue where learning happens, and to utilize holistically the other areas of
life where their students are active. Qualitative aspects, such as the student’s well-
1
Nike+ fuelband. http://www.nike.com/us/en_us/c/nikeplus-fuelband
2
Fitbit. http://www.fitbit.com/
3
Runkeeper. http://runkeeper.com/
4
Moves. http://www.moves-app.com
5
Withings. http://www.withings.com
6
HeartMath. http://www.heartmath.com/
7
Mindbloom. https://www.mindbloom.com/lifegame
8
Ubifit Garden. http://dub.washington.edu/projects/ubifit
84
Feeler: feel good and learn better - ARTEL13
being, might be worth to be taken into consideration since they can impact learning
performance. In this paper, we propose an innovative approach to learning analytics
since we combine data about well-being with learning performance. The research
question that drives this project focuses on how to foster reflection about learning
capabilities in relation to a person's well-being.
3 Visualizing the Data for Reflecting
Making sense of large datasets composed by numerical and textual information can be
handled much easier if the information is visualized. Due to the power of images for
synthesizing complex information, information visualization has been recognized as a
powerful tool for reducing cognitive load, offloading short-term memory, allowing
for easier comparisons, and generally facilitating inferences [16, 18].
According to [14], visualizations should be conceived as transformation processes
within the Data-Information-Knowledge continuum. From this perspective, Masud et
al. claim that visualizations are not merely the final outcome of representing data,
information and knowledge, but that they should be understood as a process since
they provide awareness, as well as social and reflective insights.
[7] have also highlighted the strength of visualizations as tools for sense making in
which information is collected, organized, and analyzed to generate knowledge and
inform action. According to these authors, sense making is often a social process in-
volving parallelization of effort, discussion, and consensus building. Some web-based
collaborative visualization systems that go in this direction are Sense.us, Spotfire9,
Wikimapia10, Many-Eyes11, among others.
Visualizing the data can be a powerful resource for supporting reflection, individu-
al or in groups, and therefore gaining awareness. Considering the strong link between
reflection and learning [15], we can anticipate that the reflections that take place
through the analysis of visualizations would lead to learning. In this sense, visualiza-
tions can trigger reflection-after-action processes helping the learner to develop new
understandings and appreciations [1].
Some of the research questions that emerge in this context, is how to make large
volumes of data meaningful for users. How should this data be displayed in order to
improve self-understanding, reflection and awareness? One answer to this question
can be found in the design philosophy underlying slow technology. According to [6],
slow technology responds to the need of actively promoting moments of reflection.
Reproducing their words “A key issue in slow technology, as a design philosophy, is
that we should use slowness in learning, understanding and presence to give people
time to think and reflect” (p.203).
The visualization of information dealing with learning and well-being through
smart textiles could be perceived as an object for reflection in the sense that it encou-
9
Spotfire. TIBCO Software. http://spotfire.tibco.com/discover-spotfire
10
Wikimapia. http://wikimapia.org
11
Many-Eyes. http://many-eyes.com
85
Feeler: feel good and learn better - ARTEL13
rages the person to take some time to think about his/her habits. Smart textiles, also
known as electronic textiles or e-textiles, refer to the use of electronic components
and advanced fibers in garments [8]. Research on these kinds of smart textiles has
advanced during the last years and some applications can be found in the military and
medical sector (Georgia Tech Wearable MotherboardTM 12), work (PROeTex13) and in
sportswear (Nike Hyperdunk+14). Apart from that, smart textile applications can be
also observed in the entertainment industry (midi controller jacquet15), as well as in
fashion design 16 and arts communities (e-motion project17). Smart textiles offer great
opportunities, not only for capturing data but also for displaying it to the person in a
discrete, subtle and personal way.
4 Methods
To design tools that effectively assist self-reflection, it is crucial to understand how
people think about well-being and learning in relation to their everyday practices. For
this reason, the project builds on a research-based design process [11, 12]. It is an
iterative process characterized by the following phases: contextual inquiry, participa-
tory design, product design and prototype as hypothesis. The aim is to involve users
from early phases of the project in order to incorporate their expectations and needs.
In the contextual inquiry, designers focus on achieving a deep understanding of the
socio-cultural context of the design. The information gathered during this phase is
used to develop use scenarios that are discussed in participatory design sessions with
the people who later will use the designed products. Participatory design sessions
provide designers feedback and inspiring ideas that may inform the product design. It
is important to note that despite users contributions are key elements of the design
process, final decisions are taken by the designers. The transparency of the process
and the continuous tests and redesigns guarantee that participants’ views are consid-
ered throughout the process. However, designers are the experts that will make deci-
sions on the prototypes.
At the moment, 6 exploratory interviews have been realized to people aged be-
tween 24-60 years old that combine work and studies and that are concerned about
their well-being. The interviewees were asked to take some pictures and write a short
text about how they would represent well-being, health and mindfulness in their eve-
ryday life. Images and texts were adapted to a card layout and used during the inter-
views as a starting point of the conversation. The information gathered during the
interviews informed the participatory design session that took place during the 2nd
Multidisciplinary Summer School on Design as Inquiry18. The workshop helped to
12
Georgia Tech Wearable Mother BoardTM http://www.gtwm.gatech.edu/
13
http://www.ugent.be/ea/textiles/en/projects/afgelopenprojecten/Proetex.htm
14
Nike Hyperdunk+. http://swoo.sh/17nJBtl
15
Midi controller jacquet. http://kck.st/ZX78u2
16
Fashioning technology. http://www.fashioningtech.com
17
E-motion project. http://www.design.udk-berlin.de/Modedesign/Emotion
18
2nd Multidisciplinary Summer School on Design as Inquiry. http://bit.ly/1cmOunJ
86
Feeler: feel good and learn better - ARTEL13
gain insights of people’s understanding of learning and well-being, as well as to
brainstorm some ideas about what aspects could be worth to quantify and how to
visualize the data. In the short-term, next steps include the development of the con-
cept design, building of low-fi prototypes and the organization of more participatory
design sessions. The aim of the project is to develop a working prototype that can be
tested in learning contexts.
5 Feeler prototype
Feeler is a tool, currently still under development, that allows learners to monitor
some aspects of their well-being, such as the amount of physical activity and concen-
tration levels, in order to improve their learning. Feeler will combine data about per-
sonal well-being with metadata of learning materials such as the amount of time a
student has logged into the system and the times when she connected. The reason for
using learning analytics is for increasing understanding about the conditions in which
a person is more willing to learn.
It is expected that this tool will support learners’ reflective thinking about their
lifestyle and the impact it has in their learning capabilities. By focusing in a personal
matter such as well-being, the tool connects with some of the elements outlined by [4]
about reflective thinking: a state of perplexity, hesitation and doubt; (in case that the
data collected doesn’t correlate to the learners assumptions) and an act of search di-
rected to corroborate or to invalidate the suggested belief (people may feel motivated
to understand why the data collected by the system contradicts their initial thoughts).
The outcomes of engaging in such a reflection process about one’s well-being and
learning performance include (1) new perspectives on experience, (2) changes in be-
havior, (3) readiness for application, and (4) commitment to action [1].
Early prototypes of the suggested tool (fig.1) are based on the use of a headband
that monitors the brain activity, for instance the Melon band19 and a mobile app that
tracks physical activity (Moves5). The head band can register different states of men-
tal activity in order to determine a person’s level of focus. Information about how
much concentrated is the person for a specific amount of time would be displayed
through a smart wool bands placed in the person’s wrists. Depending of the concen-
tration level, some led lights would activate. The more concentrated you are, the more
intense would be the lights sparkling in the wool bands. Less levels of concentration
would be associated to less intensity of the lights. Information about physical activity
is monitored through the mobile app Moves. In this case, no specific action nor extra
device are required. Once downloaded the app, the person just has to carry her phone
wherever she goes and it will detect the type of activity performed (walk, run or cy-
cle), the duration and the distance travelled.
19
Melon. http://kck.st/13uYmbQ
87
Feeler: feel good and learn better - ARTEL13
Fig. 1. Sketches of Feeler prototype.
Data about the level of focus and physical activity will be displayed together
through a screen (fig. 1). The intention is to allow the person to observe trends, get
into details and establish correlations. By offering the users different levels of read-
ing, we expect they would engage in reflection processes that can lead to meaningful
group discussions.
6 Conclusions
The underlying assumption of the research is that information visualization can be
a powerful tool for encouraging reflection and awareness. By drawing the attention to
learning and well-being, the project combines two currently promising areas such as
Personal Informatics and Learning Analytics. It is expected that allowing learners to
capture quantitative data about their states and habits will offer them rich materials
that support reflection processes.
Even if Feeler can be used in very different settings, we consider that the tool has
great potential in higher education since reflective practices help facing life’s chal-
lenges and encourages attention and analysis habits key for addressing the problems
of society [15].
Regarding the design of the prototype, slow technologies bring inspiring since, ra-
ther than designing for effective work, the aim is to foster reflection. In this sense,
some initial sketches focus on smart textiles for displaying the information following
a slow approach. In this sense, we consider that not only the tool, but the design as
well should support reflective practices.
88
Feeler: feel good and learn better - ARTEL13
7 References
1. Boud, D., Keogh, R., Walker, D.: Promoting reflection in learning: A model. In: Boud, D.,
Keogh, R., Walker, D. (eds.) Reflection: Turning experience into learning, pp. 18–40.
Kogan Page Ltd, London (1985)
2. Buckingham, S. & Ferguson, R.: Social Learning Analytics. Available as: Technical Re-
port KMI-11-01, Knowledge Media Institute, The Open University, UK (2011)
3. Clow, D.: The learning analytics cycle: closing the loop effectively. In: LAK12: 2nd Inter-
national Conference on Learning Analytics & Knowledge, pp. 134-138. ACM, NY (2012)
4. Dewey, J.: How we think: A restatement of the relation of reflective thinking to the educa-
tive process. DC Heath, Boston (1933)
5. Duval, E., Verbert, K.: Learning Analytics. eleed 8(1), (2012). Retrieved June 27, 2013
from http://eleed.campussource.de/archive/8/3336
6. Hallnäs, L., Redström, J.: Slow Technology - Designing for Reflection. Personal Ubiqui-
tous Computing 5, 201-212 (2001)
7. Heer, J., Agrawala, M.: Design considerations for collaborative visual analytics. Infor-
mation Visualization 7, 49-62 (2008)
8. Katterfeldt, E.S., Dittert, N. & Schelhowe, H.: Eduwear: Smart Textiles as Ways of Relat-
ing Computing Technology to Everyday Life. In: 8th International Conference on Interac-
tive Design and Children, pp. 9-17. ACM, NY (2009)
9. Kirschner, P, Buckingham, S. and Carr, C.: Visualizing Argumentation: Software Tools
for Collaborative and Educational Sense-Making. Springer, London (2003)
10. Kruse, A., & Pongsajapan, R.: Student-centered learning analytics. CNDLS Thought Pa-
pers, 1-9 (2012). Retrieved June 27, 2013 from
https://cndls.georgetown.edu/m/documents/thoughtpaper-krusepongsajapan.pdf
11. Leinonen, T., Toikkanen, T., & Silfvast, K.: Software as hypothesis: research-based design
methodology. In: 10th Anniversary Conference on Participatory Design 2008, PDC '08,
pp. 61-70. ACM, Bloomington (2008)
12. Leinonen, T.: Designing learning tools - methodological insights. Ph.D. Aalto University
School of Art and Design. Bookwell, Jyvaeskylae (2010)
13. Li, I., Dey, A.K., Forlizzi, J.: Understanding My Data, Myself: Supporting Self-Reflection
with Ubicomp Technologies. In: Proceedings of the 13th international conference on
Ubiquitous computing, UbiComp ’11, pp. 405–414. ACM, NY (2011)
14. Masud, L., Valsecchi, F., Ciuccarelli, P., Ricci, D. & Caviglia, G.: From data to knowledge
- visualizations as transformation processes within the data - information - knowledge con-
tinuum. In: 14th International Conference Information Visualization (IV), pp. 445–499.
IEEE (2010)
15. Rogers, R.: Reflection in Higher Education: A concept Analysis. Innovative Higher Edu-
cation 26(1), 37-54 (2001)
16. Shneiderman, B.: The Eyes Have It: A Task by Data Type Taxonomy for Information Vis-
ualizations. In: IEEE Visual Languages’96, pp.336-343. Boulder, CO, USA (1996)
17. Tan, S.C., Divaharan, S., Tan, L., &Cheah, H.M.: Self-directed learning with ICT: Theory,
Practice and Assessment. Ministry of Education, Singapore (2011). Retrieved June 27,
2013 from http://bit.ly/Z0fgIR
18. Tufte, E.R.: Visual Explanations. Images and Quantities, Evidence and Narrative. Graphic
Press, Connecticut (1997)
89
90
SpirOnto: Semantically Enhanced Patient Records
for Reflective Learning on Spiritual Care
in Palliative Care
Christine Kunzmann1, Traugott Roser2, Andreas P. Schmidt3, Tanja Stiehl4
1
Pontydysgu, UK
{kontakt@christine-kunzmann.de}
4
University of Münster, Germany
{traugott.roser@uni-muenster.de}
3
Karlsruhe University of Applied Sciences, Germany
{andreas_peter.schmidt@hs-karlsruhe.de}
4
Center of Pediatric Palliative Care, LMU Munich, Germany
{tanja.stiehl@med.uni-muenchen.de}
Abstract. Ontologies as shared understanding of a domain of interest can sup-
port reflective processes in spiritual care. Such an ontology has been extracted
from an empirical analysis of historic patient records, which has identified a key
structure. This ontology is supposed to support the reflective learning process of
the palliative care team, which is interdisciplinary. A first prototype for a se-
mantically enhanced patient care documentation system has been developed
which embeds links to spiritual care into practice and helps to create awareness
among other disciplines about the systematic nature of spiritual care.
Keywords: ontologies, spiritual care, patient documentation, reflection
1 Introduction
Palliative care is a challenging multidisciplinary field where different perspectives
need to complement each other, including nurses, doctors, social workers but also the
frequently neglected aspect of spiritual care. Particularly this aspect has become more
complex as a consequence for an increasingly multi-cultural society with a myriad of
religious and spiritual ideas and beliefs.
Currently there is little awareness about spiritual aspects in palliative care in adja-
cent professions (such as physicians or nurses), and the perceived significance of this
part of palliative lags behind other professions. This is due to spiritual care not being
explicitly represented in boundary objects between the professions, most notably in
patient records, but also due to lack of evidence about the effectiveness of spiritual
interventions beyond anecdotal evidence.
91
SpirOnto: Semantically enhanced patient records for reflective learning on spiritual care
in palliative care - ARTEL13
An analysis of their work and learning practices has revealed that due to the de-
manding nature of palliative care, reflective practice can already be identified on a
regular basis, particularly as regular, but informal group meetings, and as institution-
alized “supervision” in larger time intervals. This is an important element of coping
strategies. In these reflection sessions, narratives about patients (from varying time-
frames) are used to deepen the understanding about individual cases, but also to dis-
cover patterns across cases, to rationalise encounters of everyday practice. Team
members have developed a remarkably rich understanding of their work through these
practices.
To promote the understanding of spiritual care, building upon those reflective prac-
tices seems to be a very promising approach. Therefore the work presented in this
paper has concentrated on identifying and designing artefacts that can act as boundary
objects and support the reflective learning process and that can promote the maturing
of knowledge, especially through two activities: getting an overview about individual
cases and discovering patterns across cases.
The key idea of the approach is a spiritual care ontology, which represents a shared
understanding of the domain accessible to all involved professions. This ontology is
used to enhance patient records, represents a scaffold for reflection sessions, and cap-
tures evidence about relationship between patient situations and effective interven-
tions.
In the following sections, we present the ontology and how it was developed (sec-
tion 2) and the concept of how it is designed to support the (collective) learning pro-
cess (section 3) before we present a first prototype in section 4.
2 Spiritual Care Ontology
In order to come up with a meaningful and relevant ontology, an empirical ap-
proach has been chosen to develop the ontology (more details on the process are de-
scribed in Stiehl et al. (2011)). As a first step, a qualitative empirical analysis of 143
records of patients between 2004 and 2010 has been conducted. The concepts found
were iteratively integrated into an (informal) ontology using concept maps. This on-
tology was discussed with practitioners in workshops for relevance and comprehensi-
bility to align the empirical results with the needs in everyday practice.
The structure of the resulting ontology can be decomposed into the following key
elements:
• Facts about a patient or its social environment including relatives and
friends (expressed as direct properties). This includes demographic data, in-
formation about course of disease and care status, but also the cultural back-
ground (e.g., religion, migration background, or whether an individual has
been raised in rural or more urban areas).
• Observations that led to the identification of the facts (having a timestamp
and a possibly rich description of the observation.
• Spiritual concepts that interpret facts (and thus also observations). This in-
cludes a large set of concepts, such as eternity and finiteness, eternal love,
92
SpirOnto: Semantically enhanced patient records for reflective learning on spiritual care
in palliative care - ARTEL13
love, guilt, purity, powerlessness vs. almightiness, or autonomy. The links to
observable behaviour or facts are context-dependent interpretations – different
individuals link the same “facts” to different concepts, depending on their
background and personal expectations. The spiritual concepts originate from
spiritual traditions, but are not limited to a single religion, which is particularly
important with the increasing diversity in religion and culture within society.
• Spiritual interventions are possible spiritual care activities. These can include
active spiritual support, meaningful silence, pastoral interviews, practical con-
sultancy, or rituals.
Figure 1: Small subset of the ontology for spiritual care
It is important that there is a connection between these key concepts. Observations
lead to facts about a patient (or their relatives). These can be linked to spiritual con-
cepts, such as concepts of death, purity, or forgiveness. These links show needs and
unresolved spiritual conflicts that are particularly important in palliative care process-
es. It should be noted that spiritual care is necessary for the patient, but also for the
relatives, and even for the palliative care team. Finally, the spiritual care interventions
link to spiritual concepts so that the identification of concepts can help to identify
appropriate interventions (and their contextualization). Such interventions could be
anointing in a catholic tradition, but also ritual cleansing in other cultures. But it could
93
SpirOnto: Semantically enhanced patient records for reflective learning on spiritual care
in palliative care - ARTEL13
also identify topics for pastoral interviews. A small example of the ontology is shown
in figure 2 (the whole ontology consists of more than a hundred elements).
This ontology does not only allow for representing the knowledge about a patient
and their social environment in a systematic way; it also represents to a certain degree
the knowledge about appropriate spiritual care by providing the relevant concepts and
identifying the possible interventions, which is an analogy to diagnosis and treatment
in medical care.
3 Ontology & the Loops of Learning
The ontology is not only intended to act as a boundary object (together with the pa-
tient record) between the disciplines, but also as a bridge between operational and
reflective processes, which is common in reflective learning:
Figure 2: Levels of reflective learning and the role of the ontology
• On the operational level, carers document their activities as well as any ob-
servations on the state or other aspects deemed to be potentially relevant. This
usually takes place in a chronological way. Palliative care processes, however,
often extend over long periods of time (it also has to be taken into account that
not only the patient, but also relatives are to be cared for). The key idea is that
by annotating these incrementally collected notes with concepts from the on-
tology, a system can tie together distant observations about (possibly) the same
aspect. Also the ontology can guide towards possibly neglected aspects.
94
SpirOnto: Semantically enhanced patient records for reflective learning on spiritual care
in palliative care - ARTEL13
• This forms then the basis for reflection about individual cases. Reflection is
already institutionalized as regular meetings. The ontology can help as a struc-
ture to have a systematic look at aspects that might be relevant for spiritual
care. It is important to note that these observations are typically made by the
various disciplines and need to be put together to have a reasonably complete
overview. The structure of the ontology can show gaps of information and its
use promotes awareness and understanding of relevant spiritual care aspects.
• On a longer timescale, multiple cases can be analysed to enhance the body of
evidence about effectiveness of spiritual care interventions in certain contexts.
Patterns can be discovered, such as differences in age with respect to dealing
with the prospect of dying (such as asking why). Such patterns can be then
used to further enhance the ontology and can feed into targeted research activi-
ties.
4 The System
A first prototype for a novel patient documentation system that is guided by the on-
tology has been developed in a participatory design approach. The resulting system is
based on a flexible backend implemented in Java ontop of Sesame RDF store and
Lucene for fulltext search. The front-end is has been implemented for Windows Tab-
lets and laptops in C# using the Windows Presentation Framework. The front-end is
designed to support offline operation so that the application does not depend on net-
work coverage at all times.
Figure 3: Screenshot showing an observation annotated with concepts from the ontology
(tabs at the bottom)
The prototype allows for entering notes about patients in a chronological way. In a
simple interface, the user can assign concepts from the ontology to the note. Only a
small part of the ontology is static (in the sense that it can only be changed by an ad-
ministrator); apart from that, users can extend the concepts in the ontology if they find
95
SpirOnto: Semantically enhanced patient records for reflective learning on spiritual care
in palliative care - ARTEL13
that something relevant is missing (or rename concepts introduced by others, e.g.,
after a discussion in a meeting). This supports the gradual maturing of the ontology,
not only as part of the reflective sessions, but also as part of everyday practice by
capturing aspects that are not yet covered by the ontology.
5 Outlook
While spiritual care is often belittled as lacking evidence of its effectiveness com-
pared to other disciplines in palliative care, the development of the ontology has al-
ready shown that spiritual care follows a systematic approach. This systematic ap-
proach is made visible through the general structure of the ontology that has been
derived from historic patient records: observations/facts, spiritual concepts as inter-
pretations, and spiritual care interventions. Workshops with physicians, social work-
ers, and carers have shown that the ontology can act as a boundary object between the
disciplines and can create awareness about spiritual care and its relevance for holistic
care.
A first prototype has been built that demonstrates a novel approach to care docu-
mentations where observations can be associated with spiritual care concepts in a
lightweight way. This opens the possibility for enhancing the reflection on the indi-
vidual patient (that already takes place) with a structured representation about indi-
vidual cases (e.g., to more easily discover gaps), and for developing spiritual care
knowledge further (by analysing across cases and collecting the experiences).
We are aiming at trialling the prototype and gaining additional evidence about how
such an ontology can enhance both practice and associated learning processes.
Acknowledgements. This work is supported by the European Commission under the FP7
project LAYERS (no. 318209), http://www.learning-layers.eu. We wish to thank the student
teams at Karlsruhe University of Applied Sciences for their effort to create the first prototype.
They were part of the International Business Solutions Project in the Business Information
Systems program, which has received a teaching award from the Baden-Württemberg Stiftung
supporting its further internationalization.
References
Stiehl, Tanja, Führer, Monika, Roser, Traugott, Kunzmann, Christine, Schmidt, Andreas
(2011). Describing spiritual care within pediatric palliative care. An ontology-based method
for qualitative research In: 12th Congress of the European Association for Palliative Care
2011, Portugal, 2011
96
REFLECT: Community-Driven Scaffolding for
Voice-enabled Reflection on the Go
Andreas P. Schmidt2, Christine Kunzmann1, Graham Attwell1, Elizabeth Chan2, Mar-
ius Heinemann-Grüder2, Jenny Hughes1, Wenlin Lan2, Andreas Vratny2,
Andreas Heberle2
1
Pontydysgu, UK
{kontakt@christine-kunzmann.de|
graham10@mac.com|jenhughes@mac.com}
2
Karlsruhe University of Applied Sciences, Germany
{andreas_peter.schmidt@hs-karlsruhe.de}
Abstract. REFLECT is a mobile app that promotes a regular reflective routine.
It is voice-based so that it can be used, e.g., while driving a car or in similar sit-
uations. The reflection session is scaffolded through decks of questions that can
be configured by the user and shared with others, who in turn can reuse the
questions.
Keywords: Reflection, prompts, scaffolding
1 Introduction
Reflective learning is seen as one of the key activity for workplace learning that is
most neglected because of time pressure in everyday business. This particularly ap-
plies to General Practitioners (GPs) who are on a tight schedule between slots for
consultation and home visits. From the need to make learning activities traceable for
re-certification, there is, however, an interest from doctors to reflect on learning expe-
riences and to follow-up on learning opportunities arising from everyday practice.
Key approach is to create reflection opportunities by utilizing time slots like when
driving in the car from/to a home visit, or commuting.
2 Concept
The key idea behind REFLECT is that reflection support is based on voice interac-
tions, which allows for hands-free operation. Users can record their reflection ses-
sions, and the system transcribes it and sends it to them via e-mail for further pro-
cessing, e.g., for including in a personal note-taking or task management tool, or a
personal portfolio for future reference.
97
REFLECT: Community-driven scaffolding for voice-enabled reflection on the go -
ARTEL13
But reflection also needs scaffolding, particularly
if it is supposed to take place embedded into work-
ing processes like in-between home visits. This is
achieved through recording the reflection session in
the form of a structured interview along a deck of
questions. The app reads the question (via text-to-
speech) and then records the user’s responses. Via
special voice commands (e.g., “next question”), the
user can skip questions.
Useful questions for reflection cannot be pre-
defined by the app designer, as they are situation-
dependent (reflecting on the day/last patient, reflec-
tion on a longer period of time, reflecting after a
training session) and there is no general knowledge
about (i) which situations are relevant, and (ii) which
questions are useful for which type of user. There-
fore the app is complemented by a web-based inter-
face that allows for choosing decks of questions that
have been shared by others, for rating their useful-
ness, and – as soon as the learning becomes more
Figure 1: Android app
confident and experienced in reflective practice to
define own new questions and to share them with others.
This results in a lightweight and community-driven approach to scaffolding reflec-
tion, which also provides the opportunity for maturing the collective knowledge how
to best structure such reflection sessions.
3 System
The system consists of an app to be installed on a smartphone or tablet (the current
version requires Android 4.1 or higher – other systems are planned), and a web-based
backend. The app allows for choosing a deck of questions, reads the questions to the
user and transcribes the responses of the user and reacts voice commands. Towards
that end, the Google Text-to-Speech and Speech-to-Text APIs are used. While this
voice recognition does not deliver 100% accuracy, first tests have shown that under
realistic conditions (e.g., in a car) the system produces a sufficient of quality of the
resulting transcript to be useful for the user.
The backend is based on PHP, and users the Bootstrap framework. It gives the user
the possibility to configure decks of questions, share them with others, use shared
questions from others and rate them.
The app is available from the Google Play Store under http://goo.gl/m0vBc
98
REFLECT: Community-driven scaffolding for voice-enabled reflection on the go -
ARTEL13
Figure 2: REFLECT back-end for defining questions and sharing them
4 Outlook
As part of the Learning Layers project, this app is planned to be evaluated with a larg-
er number of users as part of General Practitioners’ everyday work practice. Further-
more, it is planned to complement the Android app with an iOS solution to cover the
different types of smartphones used by the target group.
Acknowledgements. This work is supported by the European Commission under the FP7
project LAYERS (no. 318209), http://www.learning-layers.eu. The development was undertak-
en by a student team in the International Business Solution Project (IBSP) course within the
Business Information Systems program at Karlsruhe University of Applied Sciences. The
course concept has received a “Fellowship for Innovations in University Teaching” from the
Baden-Württemberg Stiftung, which supports its further internationalization.
99