=Paper=
{{Paper
|id=Vol-1103/paper4
|storemode=property
|title=Modeling computer-supported reflective learning: combining a high-level timeline view with reflection cycles and tool use
|pdfUrl=https://ceur-ws.org/Vol-1103/paper4.pdf
|volume=Vol-1103
|dblpUrl=https://dblp.org/rec/conf/ectel/KrogstieKP13
}}
==Modeling computer-supported reflective learning: combining a high-level timeline view with reflection cycles and tool use==
https://ceur-ws.org/Vol-1103/paper4.pdf
Modeling computer-supported reflective learning:
Combining a high-level timeline view with reflection
cycles and tool use
Birgit R. Krogstie1, John Krogstie1, Michael Prilla2,
1
Norwegian University of Science and Technology, Sem Sælands vei 7-9, 7491 Trondheim,
Norway
{birgitkr; john.krogstie}@idi.ntnu.no
2
University of Bochum, Institute for Applied Work Science, Information and Technology
Management, Bochum, Germany
michael.prilla@rub.de
Abstract. To help conceptualize technology enhanced reflective learning in the
workplace and guide the development of reflection tools, the Computer Sup-
ported Reflective Learning (CSRL) model has been developed in the MIRROR
FP7 EU project. This paper presents the CSRL model in its version 1.2.1, which
compared to previous versions has a stronger focus on the triggering of reflec-
tion, on changes to work resulting from reflection, and on how reflective learn-
ing processes involves transitions between levels in the organization. As part of
addressing these concerns, the model now includes a separate timeline view ex-
plicitly showing levels in the organization. The paper describes how the CSRL
model was evaluated in April-May 2013. Main findings from the evaluation are
outlined, and implications for further refinement of the model and associated
guidelines for its use are discussed.
1 Introduction
In developing technology to support reflective learning in the workplace, there are
multiple challenges. One challenge has been the lack of a conceptual model to repre-
sent the essential aspects of reflective learning in the workplace in a way that informs
the design of reflection tools. This challenge is being addressed in MIRROR, an inte-
grated research project funded under the 7th FP.
One of the tasks in the MIRROR project is to develop a Model of Computer Sup-
ported Reflective Learning (CSRL) [1]. In its final version (2.0) the CSRL model is
intended as a tool for the design of technology in the domain of computer supported
reflective learning in the workplace beyond the MIRROR project, offering a set of
model views and guidelines for their use. The rationale and use of the CSRL model
can be briefly explained as follows:
A group of stakeholders in an organization (managers, employees, and developers)
want to better understand and support reflective learning in the organization. They sit
down together and create a high-level model of the relevant reflection process(es)
with the CSRL timeline view. They also identify cycles of reflective learning connect-
ing work and reflection on work along the timeline. For each cycle they use the CSRL
53
Modeling computer-supported reflective learning - ARTEL13
cycle view to add more detail on activities and the use of tools. To consider the poten-
tial of specific tools to support the reflection, they use the CSRL tool use view to see
what purposes the tools might serve. Thus the group agrees on a shared model of the
reflective learning process(es) in the organization, informing later design decisions.
This paper addresses the MIRROR CSRL model in its version 1.2.1. Our focus is
the combined use of model views at different levels of abstraction. We present the
views and their rationale and report on an evaluation of the views in spring 2013.
The paper is structured as follows: In Section 2 we provide a brief theoretical
background. Section 3 presents the current version of the CSRL model with three
model views. In section 4 we describe the evaluation of the model. In section 5 we
discuss the findings, considering strengths and weaknesses of the proposed approach.
Section 6 concludes the paper, proposing further work to be done.
2 Background
Reflection is critical to workplace learning, enabling employees to make sense of
complex and dynamic situations [2, 3]. Boud et al. [4] (p. 19) define learning through
reflection as “those intellectual and affective activities in which individuals engage to
explore their experiences in order to lead to new understandings and appreciations”.
In the MIRROR project we consider reflective learning to be the conscious re-
evaluation of experience for the purpose of guiding future behavior, acknowledging
the need to attend to feelings, ideas as well as behavior associated with work experi-
ence. In the workplace, work and reflection are closely connected, feeding into each
other [2, 3]. Reflection on work experiences can happen close to work or with more
distance, and it can be based on human memory alone or also on the use of data col-
lected from the work process. Reflection is done individually or collaboratively [5, 6].
Through reflection, an improved understanding of the experience is created, and im-
plications, conclusions, or lessons learned can be derived and applied to work. Cycles
of reflection and learning (e.g. [7-10]) transform work experience into knowledge
applicable to the challenges of daily work. Informal learning through reflection in the
workplace can benefit from the use of technology [11-16].
Models like the CSRL model are developed over time, in collaborative sessions but
also individually and asynchronously. The CSRL model, now in its version 1.2.1, is a
result of emerging, bottom-up model development [17]. The model captures shared
insights about the domain (computer-supported reflective learning in the workplace)
among project stakeholders (scientists and representatives of user organizations) with
varying perspectives and interests. The insights come from conceptual work, user
studies, requirements elicitation, prototype development and evaluations in the project
test bed organizations. The representation of the insights in the CSRL model is based
on an iterative process of discussion, negotiation, systematic collection of input and
hands-on evaluations of the model within the project, and feedback from external
stakeholders, e.g. project reviewers. Through this process, the model aids conceptual
alignment in the project. As there are different perspectives of reflection being re-
searched in MIRROR there are also several conceptual models that are aligned with
the CSRL model in the sense that they extend it by refining aspects of reflective learn-
ing.
54
Modeling computer-supported reflective learning - ARTEL13
The CSRL model is intended as a tool to aid the design of technology for reflective
learning in the workplace. As such it provides a semi-formal description of the con-
ceptual architecture within which reflection tools should be produced and includes a
number of high-level constraints (i.e. business requirements) for more detailed tech-
nical requirements and design choices for apps. The guidelines for use of the CSRL
model are at this stage rudimentary and mainly consist of the course of actions de-
scribed in section 4. What is established is that the model is intended for collaborative
modeling of scenarios, involving developers as well as (representatives of) users.
Particularly in enterprise and organizational modeling, collaborative techniques have
been used for developing a joint high-level model of an area through a facilitated
process [18-20]. There are similarities between this type of modeling and the one used
for parts of the modeling tasks in the evaluation of the CSRL model (section 3).
The CSRL model has a role in identifying/specifying user requirements for reflec-
tion tools and guide the process towards the design of the tools. The model can also
be used to consider the roles of existing tools in a reflective learning scenario, and
help identify roles for which there is currently little support. Before describing how
we evaluated the use of the CSRL model, we outline the model with three views.
3 The MIRROR model CSRL
The version 1.2.1 of the CSRL model was developed to address certain aspects of
reflective learning in the workplace that were not sufficiently addressed in the previ-
ous version of the model. Compared to previous versions the current reflection cycle
has a stronger focus on the triggering of individual and collaborative reflection, on
changes to the work process resulting from the application of reflection outcomes,
and on transitions between reflection cycles across levels in the organization [21].
The model has also been complemented by additional views, that is, model elements
and syntax for certain purposes and phases of using the model. For example, a time-
line view (section 3.3) has been introduced to allow explicit representation of the
levels of the organization involved in the reflective learning processes.
The combination of views in the CSRL is intended to help users of the model meet
simultaneous requirements for model quality: Pragmatic quality in the sense of sup-
porting communication about scenarios by keeping things simple, and semantic quali-
ty in the sense of including enough detail to support the design of tools [22]. The main
views of the model are summarized in Table 1 and described in what follows.
Table 1: CSRL model views and their purposes
Model view Purpose
Reflection cycle Modeling the reflective learning process, focusing activities involved,
view transitions between stages, resulting changes to work, and the triggering
of new reflection cycles
High-level model of reflective learning stories, e.g. representing user
Timeline view
requirements. Providing a structure for modeling reflective learning
cycles. These can be seen as an instantiation of the more abstract reflec-
tion cycle, making it more understandable and acting as a bridge to the
more detailed design information provided in the tool use view
55
Modeling computer-supported reflective learning - ARTEL13
Model view Purpose
Showing how reflection tools support activity in the reflection cycle,
Tool use view
linking tool use to each of the stages
3.1 The reflection cycle view
The reflection cycle view of the CSRL model generically describes the learning cycle
in which reflection on work experience leads to outcomes that feed into changes to
work and/or into further reflection [21]. The main elements in the reflection cycle
view – see Figure 2 - are stages (rounded boxes), activities within the stages (text
items below the name of the stage), transitions between stages in the cycle (closed
arrows), and transitions to new reflection cycles because of a trigger (dashed arrows).
Figure 1: CSRL model v1.2.1 Reflection cycle diagram
In the current version of the CSRL model, increased attention has been given to the
transitions between stages. There are different criteria for when it is appropriate to
make a transition, e.g. when the Data (from work), Frame (for reflection), Outcome
(of reflection) or Change (to work) resulting from the activities in the stage meets
certain requirements [23]. The initiation of a transition to a new reflection cycle hap-
pens when there is a reflection trigger: a discrepancy leading to a state of discomfort
that the individual wants to overcome [24] or to other motivation to change work in
some way. Also, the individual may, for various reasons, involve others in collabora-
tive reflection [25]. Triggering can be external, when some event outside the individ-
ual (e.g. a prompt from a tool) creates awareness of the discrepancy. Note that initia-
56
Modeling computer-supported reflective learning - ARTEL13
tion of a new reflection cycle can start within an existing cycle, e.g. originating in the
Conduct reflection session or Apply outcome stages.
3.2 The tool use view
Tools can support the activities in each stage of the reflection cycle in various ways.
They can help the user by capturing data, providing data, scaffolding the process, and
simulating work [1]. Also there can be tool support for involving others (for work or
reflection) and for determining whether to proceed to the next stage in the cycle.
More information on this view is found in [21].
3.3 The timeline view
A CSRL timeline model (see Figure 2) consists of an upper part representing activity
on the work arena (e.g. doing work and planning work), and a lower part (below the
thick line) for reflection on work experience. Time runs from left to right in the dia-
gram. A model consists of a sequence of steps along the timeline. There are four types
of steps: work (yellow box, vertical lines), reflection triggering (pink arrow, check-
ered), reflection (blue box, horizontal lines), and change to work (green arrow, diago-
nal lines). The horizontal/vertical/diagonal lines and checkering have been added for
the purpose of this paper, to make the elements distinguishable in black-and-white.
Work activities may be categorized as individual, team or organization wide, as
shown in the layers above the thick line. Reflection may be individual or collabora-
tive, as indicated in the layers below the line.
Figure 2: A model in the CSRL timeline view
57
Modeling computer-supported reflective learning - ARTEL13
For each of the boxes and arrows, some information is to be specified, see Figure 3.
Model Symbol Information specified for the element
element
Work Who is doing the work? What is being done?
Trigger Why is reflection triggered? What is the objective of the
initiated reflection?
Reflection Who is reflecting? How does it happen? i.e., if desired,
name/describe the approach.
Change What change is made? Who is affected? (i.e. individual,
team, the entire organization)
Figure 3: Information specified for the elements in the timeline view
4 Evaluating the CSRL model v1.2.1
The CSRL model v1.2.1 was evaluated in April-May 2013 within the MIRROR pro-
ject. There were two main parts of this evaluation:
• First, the CSRL reflection cycle (in an interactive, “clickable” html version) and
the types of tool use (formulated as checklist questions: “How does your app help
the user…”) were applied to the MIRROR apps. The purpose here was to check
whether and how the apps supported the various steps of reflection.
• Next, the CSRL timeline view in combination with the reflection cycle view was
evaluated in a collaborative modeling exercise.
Below we report on how the evaluation was conducted and present key findings.
4.1 Evaluation of the (clickable) CSRL reflection cycle and types of tool use
The CSRL reflection cycle and the tool checklists were evaluated in May 2013 by 8
MIRROR developers and four representatives of user organizations. They individual-
ly applied the checklists to the MIRROR apps they were involved with. All apps were
in this way addressed. The participants were already familiar with the previous ver-
sion of the CSRL reflection cycle with associated types of tool use. The checklists
were used to describe existing apps that have not yet been extensively evaluated, the
answers reflecting how the developers and test beds intend or expect the apps to be
used. After the exercise, the participants filled in a questionnaire, focusing on the
usability and relevance of the clickable model and the tool checklist items.
Findings from the evaluation show that the list of tool use types (cf. section 3.2)
can be used to describe the role of apps for different reflective learning settings and
types of reflection tools. By systematically addressing the questions in the checklists
most of the developers were able to comprehensively describe the intended types of
use of their apps. Answers to the follow-up questionnaire showed that some of the
types of tool use were perceived as less clear, for instance monitoring work or provid-
ing scaffolding. The clarity of some items in the tool checklists should be refined in
58
Modeling computer-supported reflective learning - ARTEL13
the next version. Also there are some aspects of reflection and tool support that are
not explicitly addressed in any of the current items that maybe should be captured in
new items in the list. Especially, “providing scaffolding” seems to be too wide as a
category in light of current focus in the project on guidance and prompting. 6 out of
the 8 developers reported that the exercise of applying the tool checklists to their apps
lead to new ideas about app functionality or use. This means the model served its
intended role as a design tool by helping developers iteratively improve existing apps.
4.2 Evaluation of the CSRL timeline view
An evaluation of the CSRL timeline view was conducted in May 2013 by participants
from the MIRROR project. In the evaluation, 26 participants distributed over five
groups, each including at least one developer and at least one representative from an
organization using or aiming to use MIRROR tools. In the groups they created models
in the CSRL timeline view (see examples in Figure 4 and Figure 5) based on textual
scenarios/stories of reflective learning, which were taken from empirical work in the
test beds, for example a scenario of behavior in a medical emergency. Each group
worked on a scenario that had been created in advance by 1-2 group members familiar
with the app(s). The scenarios were to be of relevance to the partners and include one
or more MIRROR apps. The groups had no permanent facilitation support, but a re-
searcher circulated through the groups to answer questions on the modeling work.
Three of the scenarios prepared for the evaluation included the combined use of
two or more apps. Below is shown the result of two groups modeling their scenarios.
Figure 4: Example 1: CSRL timeline model (scenario including use of two reflection apps)
Figure 5: Example 2: Scenario including use of three reflection apps
59
Modeling computer-supported reflective learning - ARTEL13
It should be noted that all the stories modeled in the evaluation were designed to
have a “happy ending” in the form of change to work at the organizational level. The
modeling ensured a clear focus on this level (and how to get there) through the visual-
ization of the different levels.
After the timeline diagrams had been created, the CSRL reflection cycle model
was used as a template for specifying more detail about each cycle (one diagram per
cycle), in particular the use of reflection tools. Figure 6 shows one of the cycles in the
timeline in Figure 4. It can be seen that the tool “TalkReflect” is being used in the
Plan and do work stage of the cycle, whereas the tool “DocTrain” is used in the Con-
duct reflection session stage. It can also be seen that the reflection session can end in a
new reflection cycle being triggered, or in a change to work. Figure 6 also shows that
the participants understood that an outcome of reflection might also be how to im-
prove learning (in this case by “collection more data”) instead of direct changes to
work procedures.
Figure 6: Reflection cycle detailing a single cycle from the timeline in Figure 4
After the collaborative modeling exercises the participants individually filled in a
questionnaire to provide feedback on the modeling. We will briefly describe key re-
sults from analyzing the data (diagrams and answers to the questionnaires).
Modeling with the CSRL timeline view was perceived as useful by the partici-
pants. To the question “What do you think about collaboratively working on a
scenario model in this way?”, 25 out of 26 participants used positive terms, the last
pointing to a challenge without being negative.
Two questions addressed the usefulness of the timeline view and the cycle view,
respectively, for design. The question about the timeline view was: “Imagine you
were involved in a process of designing another reflective learning tool, either
60
Modeling computer-supported reflective learning - ARTEL13
from scratch or to iteratively improve an existing tool. Would you consider cre-
ating high-level models of scenarios in a way similar to what you did in this exer-
cise? Please briefly explain.” Answers were generally positive: 23 of 26 answered
yes (many with additional remarks about the purposes for which they would use it
and/or conditions that need to be in place, especially real data from the workplace and
participation of users), two answered no (saying respectively that they would prefer to
use only personas and not scenarios, and that they thought the modeling might not be
beneficial for technical development because the cycles would change due to the
tools), and one can be considered neutral (arguing that s/he would not know because
s/he is not a developer). It is interesting to note that the two respondents (both experts
in software development) who explicitly say they would not use the model for design
were both positive about using the timeline in an early phase of a project to support
communication with users. Follow-up interviews would be interesting in these cases.
Another question addressed the use of the cycle view to aid design: “Imagine you
were involved in a process of designing another reflective learning tool, either
from scratch or to iteratively improve an existing tool. If you were to use the
CSRL cycle model to aid the design process, how would you use it?” The answers
were generally positive, but referred to many different reasons for conducting this
step, only some related to reflection tools. For instance, it was mentioned to use the
cycle model to create a scenario, to understand the problem better and identify people
involved, to ensure completeness of the scenario, to detect flaws in the combination of
tools, to check if all aspects are supported, to better understand how an app can be
useful. A couple of participants did not see the benefit of the detailing after working
on the timeline diagram. Answers to this question and other questions showed that
many participants would like to have more guidelines e.g. on how to proceed from the
timeline to the detailing of the reflection cycles.
One group, which dealt with applying certain tools in a healthcare scenario, was
observed permanently during the exercise (Figure 4 and Figure 6 show models creat-
ed by this group). The group was composed of three representatives from a German
hospital that had tested some tools mentioned in the scenario and wanted to integrate
them into their practice, and three tool designers. Besides one designer who had used
the timeline view earlier, none of the participants knew more about it than what they
had been told in the briefing before the exercise. In the beginning of the group work,
the participants focused their discussion on the scenario and had difficulties translat-
ing their utterances into elements of the model. The more experienced designer took
the lead in this and proposed how it could be done. Over time, other participants took
over this task, which shows that they got accommodated to the way of modeling.
Participants also made free use of the model elements and sometimes extended them
by writing additional text on them such as which tool could be used in certain situa-
tions. From working on the model, they sometimes switched back to talking about the
scenario and made slight changes, e.g. when they realized that it could be made more
efficient in some aspects. Sometimes they forgot to document decisions made during
these discussions and had to be reminded by other group members (predominantly the
designer who was more familiar with the approach). When they switched to describ-
ing the individual cycles taken from the timeline view, negotiation processes between
the tool designers and the hospital representatives started, e.g. when they were dis-
cussing to which extent the tool “DocTrain” can support discussions as indicated at
61
Modeling computer-supported reflective learning - ARTEL13
the bottom of Figure 6. Finally, as can be seen from Figure 4, the group created a
fairly detailed timeline model, which also contained parallel actions. The resulting
model was not as detailed as one would have expected from a professionally run
modeling workshop (Figure 6 exemplifies this well, as it outlines the process of re-
flection but does not provide enough details for third parties such as designers to un-
derstand the process fully) but was sufficiently complete for the group to agree in the
end that they had made significant steps ahead in both creating common understand-
ing of how the tools could be used to produce value in the hospital and improve the
envisioned scenario. They reported that they had had a productive workshop that ad-
vanced their idea of implementing reflection support.
From temporary observations (stepping from group to group during the exercise)
and feedback of other groups and their members, we also learned that the transition
into modeling reflection cycles with the CSRL reflection cycle diagram generally
worked, but identifying the cycles was sometimes challenging. Also the rationale for
this step was a bit unclear in the exercise, which may be due to the fact that discussing
tool features was not a main focus of the exercise (and also there was little time for
it). However, the modeling approach did enforce consideration of, and being explicit
about, what are actual triggers of reflection and the resulting changes to work (or
resulting initiation of more reflection) in the scenarios.
From the resulting timeline diagrams it can be observed that three of the groups
decided to model branching processes, which is an adaptation of the approach. The
need to model branching was commented also in the questionnaire. The diagrams also
show some other deviations from the original modeling language, e.g. positioning of
elements between two levels to express uncertainty about where an element belonged.
4.3 Strengths and limitations to the evaluation
In interpreting the findings, it is important to take several factors into account. The
participants in the evaluation were generally familiar with the model, most of them
having participated in earlier iterations of evaluating and refining the cycle model
(with associated types of tool use). All participants were aware that they were partici-
pating in a process of evaluating the CSRL model in order to improve it, with the
added benefit of getting more insight about the scenarios being modeled. This provid-
ed the participants with background knowledge on reflection cycles and made it pos-
sible to present the timeline view as a high-level view of a reflective learning process
consisting of several reflection cycles. Also the participants were familiar with the
need for the CSRL model to offer ways of representing levels in the organization, and
to be able to model scenarios involving more than one reflection tool. Prior to the
evaluation of the timeline view, the developers and some of the test bed representa-
tives had been participating in the evaluation of the clickable version of the reflection
cycle, applying the associated checklists to their specific apps. All this prior
knowledge of the CSRL model means that we cannot deduce from the evaluation
results how the model would be perceived and used by someone who does not already
know it. It is however likely that whatever is unclear to those familiar with the model,
is likely to be more unclear to somebody who is new to it. Furthermore, the findings
should be understood in light of the stage of development of the reflection tools in the
modeled scenarios, and the status of the scenarios. The tools (MIRROR apps) in ques-
62
Modeling computer-supported reflective learning - ARTEL13
tion are existing tool prototypes, most of which are currently undergoing formative
evaluation. The scenarios were developed before the exercise and did not represent
new learning needs or tool usage. The most novel element, relevant for three of the
groups, was the combined use of tools. It was in the interest of the participants (in
light of their various objectives for their work in the MIRROR project) to develop a
shared understanding of how tools can be combined to support reflective learning in
the organizations in question. The evaluation showed that all groups considered it
useful to improve their shared understanding of their scenarios
The fact that the evaluation consisted in two separate exercises means that we lack
results on the coherent, stepwise application of the cycle, timeline and tool views to a
scenario. The evaluation was done in a project that the researcher responsible for the
model was part of. This might have lead to socially wanted answers, i.e. regarding
what the participants thought about collaboratively modeling the scenarios. Also, it is
a limitation that we do not have observational data from all the collaborative model-
ing groups, which could have been used to get insight about the stepwise development
of the models, including the rationale for each step
It can be considered a strong aspect of the evaluation that the constellation of
stakeholders in each group included app designers and representatives of the user
organization or the project partner representing the type of user organization in ques-
tion. Also in most groups there were other participants that could offer additional
viewpoints. This can be considered a strong aspect of our evaluation, allowing us to
derive insights on the usefulness of the model for creating shared understanding
among stakeholders with different perspectives and interests.
5 Implications for further development of the CSRL model
Based on the evaluation of the CSRL model and the objective for the model to sup-
port the design of technology for reflective learning in the workplace we have identi-
fied main considerations for further development of the model and its guidelines: Use
of the CSRL model for communication among stakeholders, switching between levels
of abstraction, flexibility (vs. rigidity), and the need for guidelines/facilitation.
5.1 Use of the CSRL model for communication among stakeholders
The use of the timeline can be considered as an instantiation of the generic CSRL
cycle view, showing how different cycles can be connected in a reflective learning
process in a specific case. The evaluation showed that modeling reflection as in our
approach can help stakeholders improve their shared understanding of a scenario of
computer-supported reflective learning, as it was obvious in the group observed more
intensively. This came at the cost of losing accuracy and details in the model com-
pared to other ways of modeling (e.g., together with a modeling expert). The draw-
back was however outweighed by the quality of the resulting models and the effect on
collaboratively understanding and designing reflection processes and tools usage in
them. Of particular importance here was the interplay of talking about the model and
the process it represented. Instantiation of an overall model is a traditional technique
63
Modeling computer-supported reflective learning - ARTEL13
for improving understanding among stakeholders of the overall model (improving
pragmatic quality). Our evaluation results indicate that the use of the CSRL timeline
helped participants improve their understanding of how the CSRL cycle model can
used to model different specific cycles. This also made the participants recognize
aspects of this model that were difficult or unclear. The timeline view furthermore
made it possible to add detail to the CSRL cycle view by bringing in the individu-
al/organizational interplay, something that was commented as a lack of semantic qual-
ity (completeness) of an earlier version of the model [17]. Finally, the notation in the
timeline view with the bright colored elements worked well for hands-on collabora-
tive modeling. The notation is adapted to collaborative modeling sessions without
tools, for which a clear differentiation of elements (e.g. through color) is important.
5.2 Switching between different levels of abstraction
It can be considered essential to the effectiveness of modeling that the model supports
shifts between different levels of abstraction. Having different models that represent
aspects on different abstraction levels is good for comprehension of the individual
models, but it is challenging to keep the different models consistent. In this regard
there are some challenges to the current version of the CSRL model.
There is a mismatch in notation that results in some confusion. The notation in the
timeline view (Figure 2) was created with the purpose of highlighting specific aspects
and thereby shifting focus a bit as compared to the cycle view (Figure 1). The result
can however be perceived as inconsistencies between the views, which was com-
mented by some in the evaluation. The main inconsistencies relate to the representa-
tion of triggering and change to work: triggering (initiation of a new reflection cycle)
is represented with a dashed arrow in the cycle view and with a closed arrow in the
timeline view. Change to work is a transition (not a stage) in the cycle view, but one
of the four main elements in the timeline view. This means that the color-coding of
the four elements in the timeline view cannot be transferred to the four stages in the
cycle view. In the guidelines for using the model, this will have to be well explained.
Better matching between the visual format of the model elements, e.g. in the use of
colors, should be discussed as a possible improvement of the notation in order to bet-
ter support switching between different levels of abstraction. Another way to deal
with this is simplifying the model elements even more (see also section 5.3), no more
differentiating e.g. between triggers and transitions on an element level, but using
textboxes to describe both arrows. However, the differentiation of triggers and transi-
tions might not be decisive during the collaborative modeling of stakeholders, but be
best done by experts from the group post-processing the models (see section 5.4).
The timeline view turned out to be helpful to users who had not been trained to ab-
stract from concrete situations in order to model reflection formally. Compared to the
reflection cycle view, the timeline view is much closer to the understanding these
participants have of how reflection takes place in practice. The quality of the resulting
models shows that this helped people create an initial set of specifications of how a
certain set of tools can be used to support reflection in their organizations. The models
provide a starting point for further work and might not have been created in such
quality if the participants had used the more abstract main view of the CSRL model.
64
Modeling computer-supported reflective learning - ARTEL13
Considerations about consistency between levels can also be made with respect to
the connection between the activities in the steps of the reflection cycle and the types
of tool support outlined in the tool use view. This is not a question of notation but of
clarifying the actual mapping between activities and use of tools, which is not 1-1.
As part of the work on the next version of the model one should look more at the
cognitive integration [26, 27] between the different sub-languages, aligning notation
and color usage.
5.3 Flexibility (vs. rigidity) in the modeling
The need to model branching processes was seen in participants’ questionnaire an-
swers, in three of the five timeline diagrams (e.g. Figure 4 and Figure 5) and in the
group we intensively observed. The latter used branching processes to depict that the
same trigger might cause reflection on different levels or with different participants,
and to show that there can be outcomes from reflection that affect different levels of
work and thus continue the reflection process on different levels.
In collaborative modeling it is generally important to keep the modeling notation
used simple [18, 20]. The process-oriented notation used in the timeline view is no
exception to this. On the other hand the need for simple control structures (e.g. paral-
lel tasks, alternative tasks) is found in all standard process modeling notations
(BPMN, UML Activity diagrams, EPC etc.), and also in all but the simplest process
patterns [28]. Thus, the need for representing branching in processes of reflective
learning in the workplace should not come as a surprise. Simple branching should be
considered for the next version of the CSRL model. We should however avoid trying
to formalize the control-flow modeling mechanisms too much, as it will make the
language harder to use. Thus considerations for achieving semantic and pragmatic
quality must be balanced.
5.4 The need for guidelines, facilitation and experts present
In collaborative enterprise modeling [18, 19, 29] facilitation of the modeling process
is regarded as important. However, there are also approaches showing that using the
right metaphors and interaction designs, users might create certain models on their
own, without the help of an expert [30, 31]. In the evaluation we saw participants act
on their own with the modeling elements and others who needed support by more
knowledgeable members of the group. In addition, the challenges described in section
5.2 might have been diminished by some expert support. In the case of modeling re-
flection, facilitation and guidance then need to be available in different ways. For
facilitation, knowledgeable participants might be able to take the role as champions in
the model processes, as it was the case for the designer in the group we intensively
observed. Such people are much more likely to be present when modeling takes place,
e.g. in staff meetings and the like. Guidance then may be given by these users, but can
also be available from written documentation – the need to have an expert onsite
would severely slow down the process. Therefore, as argued above, keeping the com-
plexity of modeling low and providing such documentation might be most suitable in
order to keep the flexibility of modeling high enough to engage end users into it.
65
Modeling computer-supported reflective learning - ARTEL13
Expert support, however, is necessary in the post-processing and follow-up phases
of modeling reflection: If we use the user-driven model phase to gather initial models
and build common understanding of reflection processes and tool usage in them, there
is a need to clean up the models afterwards in order to make them readable and to
refine them to a level of detail and quality sufficient for implementing processes.
These steps might not be feasible without an expert, who knows about the levels of
detail and may also detect flaws and contradictions. Further research will thus also
include finding a away to let stakeholders model as much as possible on their own,
while using expert support for tasks stakeholders cannot do well on their own.
6 Conclusion
Based on results from the evaluation of the MIRROR CSRL model we have argued
that the CSRL model has pragmatic quality supporting communication about reflec-
tive learning scenarios among different stakeholders. The model also has a semantic
quality that allows the representation of more detail to aid the design of computer
support for the scenarios.
To aid the development of the next version of the CSRL model, evaluation of the
stepwise application of all views of the model should be conducted. It would be par-
ticularly useful to do this with subjects who are not already familiar with the model.
This will also address the question whether the model and its views are equally usable
for different stakeholders such as experts, users or developers.
One question that needs to be addressed in the work on the next version of the CSRL
model with associated guidelines is how far into the process of developing CSRL
tools the model should be used. Currently we do not see CSRL model as a tool for
supporting design in the steps immediately preceding coding, but as a conceptual tool
to help identify and agree on user requirements and build an understanding of how
reflection tools could meet these requirements. The guidelines for use of the CSRL
model needs to be clear about the role of the different model views with respect to
different phases in a process of developing CSRL solutions.
7 References
1. Krogstie, B., et al., Computer support for reflective learning in the workplace: A model, in
Proceedings International Conference on Advanced Learning Technologies (ICALT 2012)
2012, ACM: Rome.
2. Lave, J., The practice of learning, in Understanding Practice: Perspectives on Activity and
Context, S. Chaiklin and J. Lave, Editors. 1993, Cambridge University Press: Cambridge.
p. 20.
3. Schön, D., The Reflective Practitioner1983: Basic Books, Inc.
4. Boud, D., R. Keogh, and D. Walker, Reflection: Turning Experience into Learning1985:
RoutledgeFalmer.
5. Høyrup, S., Reflection as a core process in organisational learning. Journal of Workplace
Learning, 2004. 16(8): p. 13.
66
Modeling computer-supported reflective learning - ARTEL13
6. vanWoerkom, M. and M. Croon, Operationalising critically reflective behavior. Personnel
Review, 2008. 37(3): p. 15.
7. Cress, U. and J. Kimmerle, A systemic and cognitive view on collaborative knowledge
building in wikis. Computer-Supported Collaborative Learning, 2008. 3: p. 105-122.
8. Kolb, D.A. and R. Fry, Towards an applied theory of experiential learning, in Theories of
Group Processes, C.L. Cooper, Editor 1975, John Wiley: London. p. 33-58.
9. Korthagen, F. and A. Vasalos, Levels in reflection: Core reflection as a means to enhance
professional growth. Teachers and Teaching: Theory and Practice, 2005. 11(1): p. 25.
10. Stahl, G., Building collaborative knowing, in What We Know About CSCL And Imple-
menting It In Higher Education, J.-W. Strijbos, P.A. Kirschner, and R.L. Martens, Editors.
2002, Kluwer Academic Publishers: Boston. p. 53-85.
11. Kim, D. and S. Lee, Designing Collaborative Reflection Support Tools in e-project Based
Learning Environment. Journal of Interactive Learning Research, 2002. 13(4): p. 375-392.
12. Krogstie, B.R. and M. Divitini. Supporting Reflection in Software Development with Eve-
ryday Working Tools. in COOP. 2010. Aix-en-Provence, France: Springer.
13. Li, I., A. Dey, K., and J. Forlizzi, Understandinng My Data, Myself: Supporting Self-
Reflection with Ubicomp Technologies, in Ubicomp'112011: Bejing, China.
14. Lin, X., et al., Designing Technology to Support Reflection. Educational Technology,
Research and Development, 1999. 47(3): p. 43-62.
15. Siewiorek, N., et al., Reflection Tools in Modeling Activities, in Proceedings ICLS2010,
ISLS: Chicago.
16. Xiao, L., et al. Promoting Reflective Thinking in Collaborative Learning Activities. in
Eighth IEEE International Conference on Advanced Learning Technologies (ICALT).
2008. Santander, Cantrabria, Spain: IEEE.
17. Krogstie, B.R., et al., Collaborative Modelling of Reflection to Inform the Development
and Evaluation of Work-Based Learning Technologies in i-KNOW2012, ACM ICPS: Graz,
Austria.
18. Gjersvik, R., J. Krogstie, and A. Følstad, Participatory Development of Enterprise Process
Models, in Information Modelling Methods and Methodologies, J. Krogstie, K. Siau, and
T. Halpin, Editors. 2004, Idea Group Publishers.
19. Prilla, M. and I. Jahnke, The Socio-Technical Walkthrough for participatory process de-
sign, in Implementing International Services. A tailorable method for market assessment,
modularization and process transfer 2011, Gabler.
20. Stirna, J., A. Persson, and K. Sandkuhl, Participative Enterprise Modelling: Experiences
and Recommendations, in Proceedings Advanced Information Systems Engineering, 19th
International Conference, CAiSE J. Krogstie, A.L. Opdahl, and G. Sindre, Editors. 2007,
Lecture Notes in Computer Science 4495 Springer: Trondheim, Norway.
21. Krogstie, B., M. Prilla, and V. Pammer, Understanding and Supporting Reflective Learning
Processes in the Workplace: The RL@Work Model, in ECTEL 2013, Springer: Paphos,
Cyprus.
22. Krogstie, J., V. Dalberg, and S.M. Jensen, Harmonising Business Processes of Collabora-
tive Networked Organization Using Process Modelling, in Proceedings PROVE 2004: Tou-
louse, France.
67
Modeling computer-supported reflective learning - ARTEL13
23. Krogstie, B., Prilla, M., Pammer, V., Understanding and Supporting Reflective Learning
Processes in the Workplace: The RL@Work Model, in Proceedings of the Eigth European
Conference on Technology Enhanced Learning (EC-TEL 2013).
24. Knipfer, K., et al., Reflection as catalyst for organisational learning. Studies in Continuing
Education, 2012. 35: p. 19.
25. Prilla, M., V. Pammer, and B. Krogstie, Fostering Collaborative Redesign of Work Prac-
tice: Challenges for Tools Supporting Reflection at Work, in In press2013.
26. Krogstie, J., Model-based Development and Evolution of Information Systems: A Quality
Approach 2012, London: Springer.
27. Moody, D.L., The “Physics” of Notations: Toward a Scientific Basis for Constructing Vis-
ual Notations in Software Engineering. IEEE Transactions on Software Engineering, 2009.
35(6).
28. van der Aalst, W.M.P., et al., Workflow Patterns. Distributed and Parallel Databases, 2003.
14(1): p. 5-51.
29. Persson, AA., P. and Stirna. J., Towards Defining a Competence Profile for the Enterprise
Modelling Practitioner, in proceedings PoEM The Practice of Enterprise Modelling 2010,
Springer: Delft, the Netherlands.
30. Prilla, M., Nolte, A., Integrating ordinary users into process management: Towards imple-
menting bottom-up, people-centric BPM. Enterprise, Business-Process, and Information
Systems Modelling, in Proceedings of the International Conference on Business Process
Modeling, Development and Support (13th edition) in conjunction with CAISE’12
(BPMDS 2012), Springer.
31. Nolte, A., Prilla, M., Anyone can use models: Potentials, requirements and support for non-
expert model interaction. International Journal for e-Collaboration (2013).
68