This paper describes a model for Computer Support Reflective Learning (CSRL) as a conceptual framework to support the design, application and evaluation for tools supporting reflection as a learning mechanism at work. The CSRL model has been derived from theory and inspired by empirical work done in the MIRROR project. It contains necessary steps of reflection, which form a reflection cycle and are linked to corresponding tools and additional support mechanisms such as scaffolds to enable computer supported reflective learning. It is accompanied by a procedure to use it for the design and analysis of reflection tools in real cases. The model and the procedure to apply it have been evaluated in the MIRROR project. This paper reports on results of this evaluation.
Developing solutions to improve reflective learning in the workplace is a main objective in the MIRROR research project, which is an integrated research project funded under the FP7 of the European Commission. MIRROR seeks to provide tools to empower and motivate employees to learn from reflection on tacit work practices and personal experiences. MIRROR applications offer computer-supported reflective learning (CSRL) tools for individual, social, creative, game-based as well as organizational reflection and real-time learning. The project consortium includes five test bed organizations representing a variety of organizational characteristics and user needs, and the tools under development in the project cover a wide specter of technologies.
Apart from the MIRROR apps, the project produces conceptual tools to support the development of CSRL solutions. One of these is a reference framework for the development of MIRROR apps. The framework includes a model accounting for the role of technology in reflective learning processes – the MIRROR CSRL model - and a set of conceptual tools supporting app development and their use in the test beds.
This paper is addressing the MIRROR CSRL model in a first version and the accompanying stepwise procedure for applying the model to a case of reflective learning in a workplace to aid analysis and design. The procedure was developed, evaluated and delivered as an integral part of version 1 of the model.
In the paper, we use a detailed example to demonstrate the potential of the approach, seeking to invite discussion in the TEL research community about the CSRL model and its use. While theoretically grounded, the focus of the paper is deliberately practical. To underpin our arguments about the qualities of the model we present results from an evaluation and discuss further work in light of these results.
In what follows, Section 2 gives a theoretical background and Section 3 presents the CSRL model. Section 4 outlines the procedure for applying the model to a case. Section 5 presents an example of use of the procedure. Section 6 addresses the evaluation of the model and Section 7 concludes the paper, addressing further work. 2
Reflection is critical to workplace learning, enabling employees to make sense of complex and dynamic situations [1, 2]. Boud et al. [3] (p. 19) defined learning through reflection as “those intellectual and affective activities in which individuals engage to explore their experiences in order to lead to new understandings and appreciations.” In line with this definition, in the MIRROR project we consider reflective learning to be the conscious re-evaluation of experience for the purpose of guiding future behavior, acknowledging the need to attend to feelings, ideas as well as behavior associated with work experience.
In the workplace, work and reflection on work are intertwined [1, 2], keeping each other going and taking inputs from each other. Work creates experiences, and some experiences are reflected upon. Sometimes reflection takes place close to work and at other times with some distance. Sometimes, reflective learning is based “just” on memory, sometimes on data as well.
Reflection on work experience leads to an improved understanding of the experience and allows for deriving implications, conclusions, or lessons learned. In this way reflection transforms experience into knowledge applicable to the challenges of daily work. Reflection and learning thus form a cycle (e.g. [4-7]). The outcome of reflection on work is applied in the work practice.
Apart from being an individual, cognitive process, reflection has a strong social dimension [8, 9]: It is often accomplished collaboratively by a team or working unit, which has a joint task to perform and therefore shares work-related experience.
It is possible to encourage reflection by providing appropriate support. In industrial settings there are reflection “tools” like project debriefings [10-12] demonstrating the value of reflection in work life. Most reflective learning at work, however, occurs without support of technology [13].
Technology has a large potential to increase the efficiency and impact of reflective learning at work [14-19] and can be applied to informal, everyday learning in the workplace]. The design space of possible solutions is vast and growing with the emergence of new technologies potentially applicable to work settings.
There are many examples of successfully modeling experience-based learning as a cycle [4-7]. On the basis of work in MIRROR [20], the reflective learning cycle on work includes a reflection session (the time-limited activity of reflecting – short or long, informal or formal, planned or spontaneous, individual or collaborative, etc.). Furthermore, achieving transitions from work to reflection and back are essential, triggers for reflection and useful outcomes of reflection being key issues.
A model outlining tool support for reflective learning in the workplace should outline how work and reflection are connected, support the description of reflective learning processes and scenarios in different real-life settings, e.g. workplaces, and thereby aid the recognition of differences and commonalities. Also, it should clarify the different roles technology can play in supporting reflection [20, 21]. 3
To support analysis and development of computer supported reflective learning, the use of technology can be linked to steps in a reflective learning cycle. In the MIRROR CSRL model [21], steps of reflective learning form a cycle and are linked to categories of tool use. The learning cycle contains four main steps (see Fig. 1).
The diagram in Fig. 1 can be instantiated with a case of reflective in the workplace comprising several cycles of reflective learning (e.g. an ‘expansion outwards’ of the model). Each learning cycle can be ‘expanded inwards’ to show more detailed steps in the specific reflection cycle, as well as associated use of tools to support the process. This is shown in Fig. 2, in which the rounded rectangles in the middle of the diagram show a detailing of the process steps in Fig. 1. The columns of square boxes to the left and to the right are categories of use of reflection tools supporting the steps: White boxes indicate tool support for capturing data, dotted boxes for providing data, light gray boxes for scaffolding the process, and the dark gray ones show use of tools for simulating the work process. The model in Fig. 2 is the Version 1 of the CSRL model. For a more detailed explanation of the diagram, see [21]. (Please note that the tool categories in Fig. 2, based on the ongoing conceptual developments in MIRROR, have been slightly refined as compared to the ones in [21]. The differences are not essential with respect to the issues addressed in the present paper.) 4
The CSRL model can be used to describe existing processes or practices of reflective learning in an organization (e.g. before the introduction of new solutions), to describe intended use of new solutions (e.g. outlining user requirements), and to describe the actual practices after these solutions have been introduced. The procedure to use the model for these purposes contains three main steps: outlining a story of reflective learning, modeling the reflective learning cycles of the story, and detailing each cycle with steps and associated tool use.
First, the case of reflective learning needs o be explained in context of work processes in the organization, using the perspective of its actors. Collaborative work with scenarios helps elicit rich information from users and the organization, helping users and developers to reach a common understanding of the case. For example, developing stories in which people successfully learn by the aid of reflection tools helps focusing on tasks and goals as well as on learning outcomes and their application. To cover the full potential of the tool in the case organization, it important that the story includes the relevant situations of reflection and tool usage as well as the connections between them (e.g. results of individual reflection feeding into processes of reflective learning on the level of the team or organization). Supporting artifacts are textual descriptions or other representations (e.g. storyboards) outlining the scenario of reflective learning.
A key to understanding and supporting reflective learning is to consider transitions between work and reflection. This includes triggers and circumstances that lead to reflection and the step of bringing insights from reflection back into work, e.g. ensuring that the outcomes of reflection are brought into use in the work process. Artifacts supporting this are diagrams instantiating the learning cycle view of the CSRL model as in Fig. 2 and Fig. 5.
By considering tool use for separate cycles, different usage in different situations is described. This might also create new ideas about tool usage or design, if a tool supporting a reflection session currently does not offer scaffolding of a particular step of reflection, or it does not capture data that are available and could be of potential use. Also, it could lead to considerations about similarities and differences in tool use between different reflection sessions from different cycles in the story, with implications e.g. for the tailoring of user interfaces for the different sessions. This step can be supported by artifacts such as diagrams instantiating the process steps view of the CSRL model (e.g. Fig. 6), with tool categories.
The proposed procedure for applying the CSRL model ends at a point where several artifacts (e.g. an outlined story of reflection and several diagrams) have been developed. Depending on the step of the development process, we propose that the artifacts be useful in different ways: As a resource for design on a more detailed and formally specified level, as a benchmark for evaluation of the modeled solutions, and/or as a basis for communication among developers and users, e.g. in the next round of development (if an iterative approach is used) 5
In this section we show how the CSRL model can be used to describe a real case of reflective learning in the workplace, following the steps outlined above. The workplace is one of the MIRROR test beds: a hospital. In the story, reflection is supported by the ‘Talk Reflection’ app, which helps physicians to reflect on difficult conversations (talks) with patients and relatives.
In this story of reflective learning, the perspective is that of the physician (Fred), who is a participant in all the reflection sessions mentioned in the story. Additionally, it his colleagues are also reflecting, which provides important input to Fred’s reflection. To facilitate the later modeling steps, the story has been divided in three parts. Part 1) An assistant physician (Fred) is working in the stroke unit. Every time a patient is hospitalized with a stroke the relatives are very concerned about what happened and might happen. One day Fred has to explain an older man that his wife has suffered a bad stroke and that she might not recover because it took them to long to get to the hospital. He explains what has happened in her brain and that because of the stroke she might die. He also explains that they will need a decision from the husband whether they should take life-extending measures or not. Suddenly the man gets very angry and shouts at Fred that it is his fault, that he had paid his health insurance for years, that he demands the best treatment for his wife, and that he thinks the hospital staff are not willing to do everything they can. Fred is stunned and does not know how to react. Fortunately a nurse coming into the room is able to calm the old man down and explain to him that they are doing everything possible to save the life of his wife.
During the day Fred keeps thinking about this episode and finally finds a moment to document it in the Talk Reflection app. He first documents the case objectively the way it is required for the patient’s case file, filling in the description (e.g. explaining that he was stunned and did not know how to react to the aggression) and using the self assessments e.g. to quantify his feelings in the situation. He proceeds to add a personal note, reflecting about his experience and formulating the conclusion that he should perhaps have asked a nurse to participate in the conversation in the first place. He shares the documentation and his notes with other assistant physicians that he trusts, to allow them to comment on it in the Talk Reflection app.
Part 2) Next time he logs into the app, several of his colleagues have commented on his documentation. Most have written that they have had similar experiences and that they know how difficult such situations can be. Others describe similar cases with aggressive relatives. For instance, one colleague had once been hit by the wife of a patient. Based on these comments, Fred recognizes his case as an example of a more general issue and decides to bring it up again in the bi-weekly reflection meeting for assistant physicians in the stroke unit.
Part 3) In the bi-weekly meeting the five physicians discuss Fred’s case. Fred starts by briefly explaining about his experience, suggesting that it points to a general issue. His colleagues then explain about their experiences with similar cases. The discussion proceeds with constructive critique of the various approaches. During the discussion the physicians have the Talk Reflection app in front of them on their individual iPads and can all look at the information that has been shared there. Some of the physicians use the app to make quick notes about cases not yet documented, and comments to cases already documented. Closing the discussion in the meeting, the physicians have reached the resolution that it would be best to have relative talks only when there is at least one other person from staff nearby, as with the nurse in Fred’s case. They decide to make this a change to their work routines. Using the evaluation function in the Talk Reflection app, one physician writes down this reflection result. To document its rationale he makes a link to the relevant cases discussed in the meeting. He then shares the documented resolution with all participants of the meeting and the other physicians of the ward.
The diagram in Fig. 5 shows the learning cycles described in the story. We note that there are three cycles in which Fred the physician is involved (drawn with solid arrows in the figures below). The cycles correspond to the three parts of the story in Fig. 4. In the innermost cycle (Part 1), Fred reflects while documenting his experience. (see the cycle shown highlighted in Fig. 5). In the next cycle (Part 2), Fred reflects on the comments provided by his colleagues. Finally, in the outer cycle (Part 3), he reflects with his colleagues in the physician’s meeting. To complement the picture of reflective learning in the story, a cycle capturing the reflection session of Fred’s commenting colleagues has been included and drawn with dashed arrows and boxes.
External factors influencing the process are not shown in the diagram. For instance, in the diagram in Fig. 5 the step of initiating the inner reflection cycle is called ‘Time to document’ – which refers not only that Fred actually has the time, but also to the fact that the organization has routines for documenting conversations with relatives, requiring that such documenting be done.
In what follows, we show how the CSRL model can be used to outline process steps and tool use in two of the reflective learning cycles in Fig. 5. A more complete analysis of the case would have included a detailing of the mid cycles, but we left this out in the paper for reasons of space.
We start with the cycle marked with boldface in Fig. 5, e.g. the inner cycle. This cycle describes individual use of the app for reflecting on single experiences, i.e. relative talks just documented (in the same app) as part of the work process.
Fig. 6 shows the process steps diagram instantiated with the inner learning cycle. The steps from the reference model (see Fig. 6) have been reformulated to more accurately describe what happens in the cycle. Furthermore, steps in the process view of the CSRL model (Fig. 2) that did not seem relevant to the story have been omitted.
The relevant tool categories have been instantiated with brief explanations of how the Talk Reflection app supports the process. In the perspective of the reflective learning cycle, the objective documentation of the relative talk, including behavioral and emotional aspects, can be seen as capturing of data on work experiences. The app provides scaffolding for this data gathering. Reflection is triggered and framed, as the physician is encouraged to write a personal note (implicit in the provision of a personal note field in the documentation template), reflecting on the objectively documented experience. The documentation further helps the physician in reconstructing and understanding the meaning of the experience. Reconstruction, articulation of meaning, and re-evaluation are closely intertwined in this case. The Talk Reflection app does not provide scaffolding for re-evaluation of the experience, but supports the capturing and sharing of the reflection outcome.
For the purpose of illustrating the potential of the model to shed light on different use of tools to support reflective learning in different reflective learning cycles, we proceed to instantiate the process steps and tool categories with another cycle in the Talk Reflection story: the outer cycle, e.g. Part 3 of the story. Here, physicians reflect in their bi-weekly meeting, the outcome being a decision to implement a change in the work routines. The outer cycle is shown with bold lines in Fig. 7, the process model instantiating the cycle is shown in Fig. 8.
The CSRL model and the approach of applying it to a particular case were evaluated in a workshop, which had the additional purpose of informing tool development. The evaluation took place with seven work groups of 24 MIRROR participants. Each group focused on a different story of reflective learning with a particular app in a test bed organization. Every group included at least a developer and a representative of the test bed, group size ranged from 2 to 5. The development of the tools in question had already started before the evaluation workshop, and thus modeling was mostly about refining understanding of the cases and re-designing solutions.
The groups were asked to apply the procedure outlined in Section 3, after an introduction in which an example was briefly presented. Step 1 was slightly shortcut to give more time for steps 2 and 3: A story about reflective learning with the app in the test bed had been written prior to the exercise, based on knowledge of the case obtained through previous collaboration with the test bed1. The groups spent approximately 1,5 hours on developing diagrams with learning cycles (step 2, as in Fig. 2 and Fig. 5) and their detailed steps (step 3, Fig. 6) – to make this easier, participants were not restricted to a certain formalism for the cycle diagrams, but could draw them freely. Then, 30 minutes were used to individually fill in an evaluation form. 22 forms were handed in. The questions in the form were on opinions about the exercise as well as strengths and weaknesses of the model and the procedure for applying it to a case. Besides others, the evaluation form focused on three key questions. Fig. 9 summarizes the answers to these questions (Q7, Q6 and Q13 in the evaluation form): • Did the participants perceive the procedure of applying the model to be useful? • Did the exercise help refine the understanding of the case (descriptive power and usefulness for analysis)? • Did the exercise lead to new design ideas (usefulness for design)?
It can be noted from Fig. 9 that most respondents were positive or at least neutral about the usefulness of the procedure to apply the model (Q7). In another question, most participants regarded the structuring of the procedure for analysis and design to be positive, appreciating the detailed steps and categories of tool use. Regarding the use of a story of reflective learning as a starting point for the instantiation (step 1 in the procedure), the following comment captures the essence of several answers: “Using the story is somehow good AND problematic. [on the positive side] it helps to focus on usage scenarios [and] to link abstract categories and the story [and] to involve the external people, [on the negative side] it restricts the instantiation to what you can have in the story”. One group reported having combined two stories to get a more complete picture of use of their reflection tool.
Concerning the descriptive power of the model with respect to the particular case 17 respondents of Q6 (Fig. 9) answered that the exercise had added detail to the story of reflective learning. Regarding the usefulness of the model for design, Q13 (Fig. 9) was only answered negatively by one participant, two answers were left blank. Thus, 19 of the 22 respondents confirmed that the exercise had given them insights or ideas about the design of the app in question.
Besides these answers, the participants identified strengths and weaknesses of the model as a tool for describing CSRL cases and solutions, including what could or could not be described about the particular app by modeling. As a result, a long wish list for additional capabilities of the model was derived, providing useful input for the further development of the CSRL model.
The diagrams produced by the groups showed great diversity, and the groups generally followed the steps of the procedure, but (as explicitly allowed) adapted the way of drawing the diagrams to their needs whenever there were aspects that they wished to include but that were difficult to represent with the model. These adaptations provided ideas for further development of the model. For some of groups, focus of the exercise was solely on the cycle diagrams, and discussions about the cases seemed to evolve around these diagrams. These happened mostly for cases in which the com1 In two groups the story had not been written in advance and, but could be outlined during the exercise quickly, as the participants already knew relevant scenarios for usage of the app. plexity was high, the processes of reflective learning included several roles and organizational levels, and, in one of the cases, several apps needed to be coupled. This indicates that the cycle diagrams provided a good basis to understand a case of reflection (see above) and the use of technology within this case.
The evaluation must be considered in light of some validity threats. First, it was conducted within the MIRROR project, with participants that (to a varying degree) had prior knowledge of the model. It is thus difficult to conclude from the evaluation about the general use of the model. Also, as mentioned, the stories and the tools modeled were not new, but rather in a process of continued development. These conditions on the other hand allowed the creation of cycle and process step diagrams within a short timeframe (more limited than the one presumably needed and preferred in a typical development process), and as apps were mostly used only within one specific test bed, the evaluation could be conducted with many groups. However, this also means that comparison of results across participants and groups is difficult. At the same time, the differences between the cases ensured a wide range of characteristics and use cases of reflection to be described with the model, enabling and evaluation on a broad basis of real cases. Ownership and commitment of the participants with regard to the specific tools made the work more ‘real’ and is likely to have lead to increased motivation to actively participate, but having user organization and developer working together in a group is representative of a the intended development process. In addition, the time available for the modeling was less than it (probably) would have been in a real case. This was taken into account when considering the resulting diagrams (e.g. their level of detail or coherence). The outcomes of the evaluation, in terms of quality and quantity, indicate that the evaluation reaped the benefits resulting from advantages.
The evaluation of the CSRL model and the associated procedure for its application to a case provided valuable insights about the usefulness of the model and the procedure, confirming the potential of the model to aid analysis and design. We will end the paper by discussing some challenges and future steps.
The focus on a story of reflective learning in which there is a user (persona) seemed to help focus on user needs. Systematic application of the cycle model helped to make the transitions between work and reflection explicit, including how reflection is triggered and how reflection outcomes are made applicable and applied. The modeling of tool usage with the process steps diagram supports a systematic walkthrough of what is supported by the tool and what might be supported by the tool.
Concerning the capturing of all relevant situations and aspects of tool use, it is critical that the story of reflective learning covers the relevant scenarios. The fact that one group during our model evaluation decided to combine two stories suggests that it may be necessary to have several stories covering the relevant app usage and the perspectives of different users. For instance, different stories could focus on the needs and practices of different personas in the organization.
Results on the descriptive power of the model were promising. However, for the communication between developers (designers) and users, diagrams cannot substitute application prototypes (even paper prototypes), which let users try user interfaces and features. To use the unique advantages, MIRROR uses rapid prototyping as a development approach. Using the CSRL model for analysis, in turn, has the advantage of placing the use of the apps into the context of work processes of an organization, watching how use of an app in different settings form parts of the larger picture of reflective learning. Using cycle and other diagrams to provide a visually compact representation grounded in theory of reflective learning and makes it possible to present a rather succinct picture of a CSRL process, which expressive enough to support discussion among developers and potentially useful for communication with users. To use the advantages of both approaches, future work will also be concerned with combining the approaches of using the model and using prototypes.
There are a few shortcomings to the approach presented here. First, it would be useful to have a systematic way of representing external factors impacting on the reflection processes. Second, reflective learning is closely linked to knowledge development in an organization (e.g. individual cases developing to general insights; individual experience developing to team and organizational knowledge, and so on; see [22]), and the model so far lacks the means to represent the levels of this process systematically. The answers to these challenges are likely to lie in a combination of refinement and extension of the CSRL model and refinement of the conceptual tools for its application, e.g. the procedure for model instantiation discussed in this paper. In the development of the second version of the model, refinement of the model and the procedure for its application will go hand in hand.
We plan to apply the model to the same cases in a similar evaluation than described above after one year of using the apps. While the initial evaluation largely focused on intended tool use in the test bed organizations, this next evaluation may focus on the modeling of actual tool use, as the MIRROR apps in question will have been used in the test beds at that time. A comparison of the models of intended and actual tool use may lead to insights about how the tools fill the intended roles. In this evaluation, the application of the CSRL model will be used both for evaluation purposes and for feeding back into the (re) design of tools.
While use of the CSRL model is important within the MIRROR project to support shared conceptual understanding [23] and tool development, we also want it to be used beyond the scope and time of the project. In this respect it is necessary to expose the model to development of CSRL solutions outside MIRROR: While we continue to evaluate it within MIRROR, we would like to encourage other researchers and practitioners to consider applying the first version of the CSRL model for purposes of analysis and design. 8
This work is partially funded by the project ‘MIRROR - Reflective learning at work', funded under the FP7 of the European Commission (project number 257617). The authors thank Martin Degeling for input on the Talk Reflection case and Viktoria Pammer for ideas used in this paper. 9 23.