=Paper=
{{Paper
|id=Vol-1465/artel15
|storemode=property
|title=None
|pdfUrl=https://ceur-ws.org/Vol-1465/artel15.pdf
|volume=Vol-1465
}}
==None==
Proceedings of the 5th Workshop on Awareness and Reflection in Technology-Enhanced Learning In conjunction with the 10th European Conference on Technology - Enhanced Learning: Design for Teaching and Learning in a Networked World Toledo, Spain, September 15, 2015 Milos Kravcik Michael Prilla Alexander Mikroyannidis Thomas Ullmann Viktoria Pammer (Eds.) Please refer to these proceedings as Milos Kravcik, Alexander Mikroyannidis, Viktoria Pammer, Michael Prilla, Thomas Ullmann (Eds.): Proceedings of the 5th Workshop on Awareness and Reflection in Technology Enhanced Learning. In conjunction with the 10th European Conference on Technology Enhanced Learning: Design for Teach- ing and Learning in a Networked World, Toledo, Spain, September 15, 2015. Available online at http://ceur-ws.org/. c 2015 for the individual papers by the papers’ authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. The front-cover was created by Harriett Cornish (The Open University, KMi). Addresses of the editors: Thomas Daniel Ullmann Alexander Mikroyannidis Institute of Educational Technology Knowledge Media Institute The Open University The Open University Walton Hall Walton Hall Milton Keynes Milton Keynes MK7 6AA MK7 6AA United Kingdom United Kingdom t.ullmann@open.ac.uk alexander.mikroyannidis@open.ac.uk Viktoria Pammer Michael Prilla Knowledge Technologies Institute Institute for Applied Work Science Graz University of Technology Ruhr University of Bochum Inffeldgasse 21A Universitaetsstr. 150 8010 Graz 44780 Bochum Austria Germany viktoria.pammer@tugraz.at michael.prilla@rub.de Milos Kravcik RWTH Aachen University Advanced Community Information Systems (ACIS) Ahornstr. 55 52056 Aachen Germany kravcik@dbis.rwth-aachen.de Awareness and reflection in technology enhanced learning Awareness and reflection are viewed differently across the disciplines informing Technol- ogy Enhanced Learning (CSCW, psychology, educational sciences, computer science and others). The ARTEL workshop series brings together researchers and professionals from different backgrounds to provide a forum for discussing the multi-faceted area of aware- ness and reflection. Through the last ARTEL workshops at EC-TEL the addressed topics are converging to- wards the usage of awareness and reflection in practice, its implementation in modern organisations, its impact on learners and questions of feasibility and sustainability for awareness and reflection in education and work. To reflect the growing maturity of re- search in ARTEL over the years the workshop particularly invited contributions that dealt with the application of awareness and reflection in practice. This is encapsulated in the workshop motto: Awareness and Reflection in Practice: How can awareness and reflection tech- nology become common in work practice and how does it change work prac- tices? Summary of the contributions The #ARTEL15 workshop accepted 3 full papers, and 7 short paper. The accepted pa- pers discuss awareness and reflection in formal education, outside formal education, and methods and analytics of awareness and reflection research. Three papers focus on technology enhanced awareness and reflection in formal eduction. The full paper ’Annotations as reflection amplifiers in online learning - an exploratory study’ of Dominique Verpoorten, Wim Westera, and Marcus Specht discusses an annota- tion tool, and experiment to investigate the effects of three conditions: No annotations, free-note annotations, and structured question-based annotations. The paper provides in- sight into annotations as reflection amplifiers. The short paper ’Formal concept analysis for modelling students in a technology-enhanced learning setting’ of Michael A. Bedek, Michael Kickmeier-Rust, and Dietrich Albert presents the Formal Concept Analysis (FCA) framework for visualising a domain with concept lattices. FCA aims at facilitating student reflection upon their learned and still-to-learn concepts via an open learning modelling approach. The short paper ’Learning to look - purpose and design of an awareness-raising online course in veterinary sciences; of Sophie Tasnier, Valeria Busoni, Christian Hanzen, Jeff Van de Poel, Geraldine Bolen, Catherine Delguste, Nadine Antoine, Veronique Delvaux, Tania Art, and Dominique Verpoorten discusses an approach to improve veterinary student visual awareness in use of clinical images. Four papers discuss technology enhanced reflection and awareness outside formal educa- 3 Awareness and reflection workshop series tion. The full paper ’Prompting users to facilitate support needs in collaborative reflection of Oliver Blunk, and Michael Prilla outlines a concept of how to support collaborative reflec- tion with the help of prompts. They show an initial implementation of a prototype and an approach how to evaluate the concept. The short paper ’A course concept for enhancing reflective learning - bringing research project results into the field’ of Nils Faltin, Margret Jung describes the development of a course for learning time management for professionals. The course is based on reflective learning and practice, in a mix of coaching and computer-support. The short paper ’Feeler: supporting awareness and reflection about learning through EEG data’ of Eva Durall, and Teemu Leinonen presents a prototyp which aims at supporting awareness and reflection about learning experiences through student’s EEG data. The short paper ’Mood in the city - data-driven reflection on mood in relation to public spaces’ of Viktoria Pammer discusses potential benefits for stakeholders, as well as system design issues, for reflection on urban development issues with mood as entry point to reflection. Three papers elaborate on methods and analytics of awareness and reflection research in technology enhanced learning. The full paper ’Keywords of written reflection - a comparison between reflective and de- scriptive datasets’ of Thomas Daniel Ullmann investigates a method to derive reflection keywords by contrasting two datasets, one of reflective sentences and another of descrip- tive sentences. The reflection keywords are discussed in the context of a model for reflec- tive writing. The short paper ’The LearnWeb formative assessment extension: Supporting awareness and reflection in blended courses’ of Alana Morais, Ivana Marenzi, and Deirdre Kantz presents work about the development of visualisations summarising activity of users, groups, and classes of a collaborative sharing and working platform. The short paper ’Deploying learning analytics for awareness and reflection in online scien- tific experimentation’ of Alexander Mikroyannidis, Aitor Gomez-Goiri, John Domingue, Christos Tranoris, Daan Pareit, Jono Vanhie-Van Gerwen, and Johann M. Marquez-Barja introduces a European initiative for online learning and experimentation (FORGE), espe- cially its aim to use of learning analytics to support awareness and reflection for learners and educators. Awareness and reflection workshop series The official workshop webpage can be found at http://teleurope.eu/artel15 The 5th Workshop on Awareness and Reflection in Technology Enhanced Learning (AR- TEL 2015) is part of a successful series of workshops. • 4th Workshop on Awareness and Reflection in Technology Enhanced Learning (AR- 4 Awareness and reflection workshop series TEL14). Workshop homepage: http://teleurope.eu/artel14. Proceed- ings: http://ceur-ws.org/Vol-1238/. • 3rd Workshop on Awareness and Reflection in Technology Enhanced Learning (AR- TEL13). Workshop homepage: http://teleurope.eu/artel13. Proceed- ings: http://ceur-ws.org/Vol-1103/. • 2nd Workshop on Awareness and Reflection in Technology Enhanced Learning (ARTEL12). Workshop homepage: http://www.teleurope.eu/artel12. Proceedings: http://ceur-ws.org/Vol-931/. • 1st European Workshop on Awareness and Reflection in Learning Networks (AR- Nets11). Workshop homepage: http://teleurope.eu/arnets11. Pro- ceedings: http://ceur-ws.org/Vol-790/ • Augmenting the Learning Experience with Collaboratice Reflection (ALECR11). Workshop homepage: http://www.i-maginary.it/ectel2011/index. html • 1st Workshop on Awareness and Reflection in Personal Learning Environments (ARPLE11). Workshop homepage: http://teleurope.eu/arple11. Pro- ceedings: http://journal.webscience.org/view/events/The_PLE_ Conference_2011/paper.html#group_Proceedings_of_the_1st_ Workshop_on_Awareness_and_Reflection_in_Personal_Learning_ Environments To stay updated about future events, to share your research, or simple to participate with other researchers, consider joining the group about Awareness and Reflection in Technol- ogy Enhanced Learning: http://teleurope.eu/artel We especially would like to thank the members of the programme committee for their invaluable work in scoping and promoting the workshop and quality assuring the contri- butions with their peer reviews. September 2015 Milos Kravcik, Alexander Mikroyannidis, Viktoria Pammer, Michael Prilla, Thomas Ullmann 5 Organisation committee Organisation committee Milos Kravcik, RWTH Aachen University, Germany Alexander Mikroyannidis, The Open University, United Kingdom Viktoria Pammer, Graz University of Technology, Austria Michael Prilla, University of Bochum, Germany Thomas Ullmann, The Open University, United Kingdom 6 Program committee Program committee Philippe Dessus, Universite Pierre-Mendes-France, France Ines Di Loreto, UTT - Universite de Technologie de Troyes, France Eva Durall, Aalto University, Finland Nils Faltin, IMC AG, Germany Rebecca Ferguson, The Open University, UK Angela Fessl, Know-Center, Austria Michael, Kickmeier-Rust, Graz University of Technology, Austria Kinshuk, Athabasca University, Canada Simon Knight, The Open University, UK Effie Law, University of Leicester, UK Paul Lefrere, The Open University, UK Elvira Popescu, University of Craiova, Romania Carsten Ullrich, DFKI, Germany Dominique Verpoorten, University of Liege, Belgium Riina Vuorikari, European Commission JRC IPTS, Spain 7 Supporting projects Supporting projects http://www.mirror-project.eu http://learning-layers.eu http://www.laceproject.eu/ http://wespot-project.eu http://employid.eu http://www.boost-project.eu http://ict-forge.eu/ 8 Contents Awareness and reflection in technology enhanced learning 3 Summary of the contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Awareness and reflection workshop series . . . . . . . . . . . . . . . . . . . . 4 Organisation committee . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Program committee . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Supporting projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Annotations as reflection amplifiers in online learning - an exploratory study Dominique Verpoorten, Wim Westera, Marcus Specht 11 Formal concept analysis for modelling students in a technology-enhanced learn- ing setting Michael A. Bedek, Michael Kickmeier-Rust, Dietrich Albert 27 Learning to look - purpose and design of an awareness-raising online course in veterinary sciences Sophie Tasnier, Valeria Busoni, Christian Hanzen, Jeff Van de Poel, Geraldine Bolen, Catherine Delguste, Nadine Antoine, Veronique Delvaux, Tania Art, Do- minique Verpoorten 35 Prompting users to facilitate support needs in collaborative reflection Oliver Blunk, Michael Prilla 43 A course concept for enhancing reflective learning - bringing research project results into the field Nils Faltin, Margret Jung 59 Feeler: supporting awareness and reflection about learning through EEG data Eva Durall, Teemu Leinonen 67 Mood in the city - data-driven reflection on mood in relation to public spaces Viktoria Pammer 75 Keywords of written reflection - a comparison between reflective and descriptive datasets Thomas Daniel Ullmann 83 9 CONTENTS The LearnWeb formative assessment extension: Supporting awareness and re- flection in blended courses Alana Morais, Ivana Marenzi, Deirdre Kantz 97 Deploying learning analytics for awareness and reflection in online scientific ex- perimentation Alexander Mikroyannidis, Aitor Gomez-Goiri, John Domingue, Christos Tranoris, Daan Pareit, Jono Vanhie-Van Gerwen, Johann M. Marquez-Barja 105 10 Annotations as reflection amplifiers in online learning – An exploratory study Dominique Verpoorten1, Wim Westera2, Marcus Specht2 1 University of Liège, Belgium {dvepoorten}@ulg.ac.be 1 Welten Institute, The Netherlands {wim, spe}@ou.nl Abstract. In a controlled experiment on the effects of frequent and local digital annotations, 137 volunteers covered an online course at 3 conditions: no/free/question-based electronic annotations. Results show no difference in performance between groups. However, analyses conducted within treatments suggest positive impacts on performance when annotation rates are taken into consideration, and coupled with other reflective enactments. Keywords: annotations, reflection amplifiers, students set the test, widgets, split screen learning 1 Introduction Note-taking, either when listening to lectures or reading texts, is a “totem” of teaching and learning. It seems that for centuries tutors have been expecting that students do take notes and that tutees consider note-taking as a natural activity in a scholarly life [1]. An annotation is conceived as a personal trace left by students on a pre-existing text or speech. Annotations record readers’ efforts to shape their interaction with this content. Research on note-taking has generated debates since Crawford’s early studies in this topic [2]. Promoting annotation behaviours has been a long-lasting concern in distance education. From its beginning, and long before the possibility to think about students in terms of “reflective practitioners” [3], it has been constantly recommended to design paper-based course material with large margins. This liberal use of white space [4, 5] is meant to encourage students to make analytical summary notes of what they would identify as worthy of their attention when they revise. In the 90’s, a vast body of research [6] discussed the many issues raised when moving annotation from paper-based to screen display reading. In the past few years, a renewed interest emerged for the processes of “writing on the reading” in digital activity systems, due to the novel burgeoning opportunities for searching, sharing, indexing, ordering, tag- ging, rating annotations in an “information enrichment” perspective [7]. While the effects of note-taking are well documented for paper-based practice [8, 9, 10], the new wave of research on digital annotations develops concerns in several directions: non-linear or linear annotation techniques [11], annotations as checklists [12], annotation sharing mechanisms [13], collaborative annotation [14, 15], tagging 11 Annotations as reflection amplifiers in online learning - ARTEL15 as annotations [16, 17], multiple displays for annotations [18]. Results reveal various conditions under which Web-based annotation mechanisms are beneficial [19]. Beyond their variety, the new alleys of research (for an extended view on recent work, see [20]) endorse to a large extent [21] the two faces of note-taking already identified by Hartley and Davies [9]: • as a process, annotations help to maintain attention, apprehend the material in a mentally active way and intensify the attendance to the task. By assisting in keep- ing learning going, they can be tokens of reflective engagement during the study task; • as a product, annotations are stored for the future, with possibilities to be reviewed, re-structured, and enriched. Boch and Piolat [8] use a similar distinction but labelled differently: “notes to aid reflection” (process) versus “notes to record information” (product). 1.1 Reflection Amplifiers In this study, the annotations are conceived as “reflection amplifiers”. Following the definition by Verpoorten, Westera, and Specht [22], reflection amplifiers (RAs) refer to deliberate prompting approaches that offer learners structured opportunities to ex- amine and evaluate their learning as it develops. Whereas the promotion of reflection is often associated with post-practice methods of experience recapture [23] through portfolios or learning diaries, RAs are nested in the study material and offered to indi- viduals during learning activities. They induce regular mental tingling for evaluating own learning and nurturing internal feedback [24]. The concise reflection they call for further characterises RAs. As support to con- densed reflective processes, RAs operate though miniature Web applications (some- times called “widgets”) performing a single task, displaying a very clear and appro- priate graphical style, and providing a single interaction point for direct provision of a given kind of data [25], here the personal annotations. In the way they are practised by learners in this study, the annotations meet the common internal characteristics of RAs: brevity, frequency and crisscrossing with the first-order learning activity, i.e. studying the course content. 1.2 Hypotheses In a comparative study an online course was delivered at 3 conditions: without anno- tation tool, with annotation tool and free-style notes, with annotation tool and ques- tion-based notes (i.e. expressed in the format of a question on the read material). The study investigated the effects of the digital annotations – conceived as multiple short episodes of analytical reflection – upon the enhancement of the quality of learning. Two hypotheses guided the experiment. Hypothesis 1. “The availability of an annotation tool and the assignment to use it for frequent and local notes induce higher marks at the test and an increased study time”. Short but repeated efforts of reflection are predicted beneficial to the content internal- 12 Annotations as reflection amplifiers in online learning - ARTEL15 ization because they are seen as a way to stay analytically engaged with the supplied learning material. It is also speculated that such a reflective approach to learning has a price with regard to time spent on the material. Hypothesis 2. “The question-based annotation strategy induces higher marks at the test than the spontaneous way of annotating”. The study includes a concern for anno- tation methods by challenging conventional practice of note-taking “as a student” with a different mode wherein the learner is invited to reflect “as an instructor” (de- tails in section “The annotation methods”). 2 Method 2.1 Independent Variables The intervention variables were the provision of an embedded annotation tool and the exposure to a strategy for frequent and local annotations. 2.2 Dependant Variables The dependent variable was the subjects’ reflective engagement with the content, broken down into seven tangible indices: • Index 1: mark at the final test (FinalTest). This index designated the score obtained at the final test taken after the study session. It measured learners’ achievement through 16 multiple-choice questions assessing knowledge and comprehension; • Index 2: time spent in the course (TimeSpent). This index, measured as the num- ber of “active ten-minute periods” in the course, was an estimate. One active peri- od was counted each time that at least one click occurred in a time span of ten min. Longer periods were left out in an attempt to correct for the time students would spend in activities foreign to the study while still being logged into the course; • Index 3: learning efficiency (LearnEff). It is fair to say that the speed of learning is an important achievement (many performance tests, e.g. IQ tests, use time as one of the main indicators). In order to incorporate this temporal dimension in the measures, the marks at the final test were related to the time spent in the course: slow learners got a lower score per unit of time than fast learners. Low-efficiency students did not necessarily receive lower marks, but they needed more time to reach their mark; • Index 4: number of page views (NumberPages). The browsing behaviour, and in this case the action of re-visiting pages, was considered as an index of reflective engagement because it assumed a meta-learning decision about the need of re- reading the material; • Index 5: quantity of annotations (NumberAnnot). • Index 6: total number of characters for the annotations (CharactInAnnot); • Index 7: number of visits (VisitDash) to the Learning Dashboard (see section “Ap- paratus”). 13 Annotations as reflection amplifiers in online learning - ARTEL15 The indices FinalTest, TimeSpent, LearnEff, and NumberPages were common to the 3 conditions. NumberAnnot, CharactInAnnot, and VisitDash were premised upon the annotation tool and therefore only offered in Treatments 2 and 3. A post-test questionnaire allowed measuring the effects of the intervention on the following additional variables: sense of control and opinion over the annotation expe- rience. 3 Apparatus 3.1 The Online Course The learning material of the experiment was the four-hour online course “Seks en de evolutie” (Sex and the evolution), a course signed and offered in Dutch by the Open Universiteit on the eLearning platform Moodle [26]. It was made of 30 finely illus- trated pages (Fig. 1) of about 800 words each, and 4 interactive animations. It covered quite complex and interrelated notions as defined by Darwin and his followers: muta- tion, natural selection, genetic drift, gene flow, survival of the fittest, etc. On the whole, the course gave an in-depth account about the evolutionary theory and invited learners to use it as an interpretation grid of gender-related behaviours observable in everyday life. In the 3 conditions, the course was introduced by a welcome video and closed with the same multiple-question test. 3.2 The tool The digital annotation tool was a comment box displayed on each page (Fig.1). It kept record of all annotations produced by the learner on this very page. The annotation tool unfolded through a click by the learner. Consistently with the length of the read- ing material and the action requested from learners (frequent but short notes), the surface of the tool was intentionally not large and its function deliberately restrained to the basic typing. As for pedagogy, the annotation tool was offered to promote ana- lytical scrutiny and internalization of the learning material’s meaning by making it possible for learners to capture, within the study task, the gist of what has been read. Figure 1. An annotation in its local context of a standard Web page of the course 14 Annotations as reflection amplifiers in online learning - ARTEL15 In order to prevent effects of fragmentation and to also support the function of “annotations as products” [9], all local annotations were automatically recorded on a single page called “Learning Dashboard” [27], accessible at any time by the student (Fig. 2). On this dashboard, the annotations were organised by section of the course content. Figure 2. All annotations were displayed within a learning dashboard. 3.3 The Annotation Methods Subjects in the treatment groups were asked to make an annotation each time they (re- )visited a page. However, participants in one treatment could encode their annotations in the way they preferred (“free annotations”) while those in the other treatment were requested to produce annotations as questions (“question-based annotations”). Pre- cisely, these participants were asked to put themselves in the shoes of the teacher and to craft questions likely to be used in a final test about the content of the page at hand. In their inventory of reflective techniques, Verpoorten et al. [22] labelled this reflec- tive strategy: “Students set the test”, and described it as: “Learners are asked to make up the questions they might get for their exam”. 3.4 Sample and Schedule Invitations to participate to the experiment were displayed on electronic and paper communication channels of the Open University in the Netherlands, including the homepage of the used course. Dutch dailies and magazines, as well as a psychology popular publication, also received announcements of the study. The registered persons were randomly distributed over the 3 conditions and received credentials for one ver- sion of the online course. They had one month to fill in a background questionnaire (15 min), cover the course (4 hr), take the final test (15 min) and answer the evalua- tion questionnaire (20 min). Out of the 361 initial respondents, 282 entered the course once at the very least but only 137 completed all steps of the study. They composed the final sample: 34 participants in Condition 1 (control group), 54 in Condition 2 (free annotations) and 49 in Condition 3 (annotations as questions). As a reward for their cooperation, they received either an iTunes voucher of 10 euros, or a three- month premium access to a mind-mapping tool (http://www.mindmeister.com), or a USB stick containing applications dedicated to eLearning (http://eduapps.org), or a free entrance to a workshop organised by the Open Universiteit. 15 Annotations as reflection amplifiers in online learning - ARTEL15 4 Results An alpha level of .05 was used for all statistical tests. 4.1 Measures Between Groups Background Questionnaire. To ensure equivalence between treatments, statistical tests were performed on the data collected in the background questionnaire. The pro- cedure exhibited an even distribution in the 3 conditions regarding: • meta-cognitive capacities, measured with a shortened version [28] of the Meta- cognitive Awareness Inventory [29], F(2, 134) = .27, p = .76, ηp2 = .004; • self-reported familiarity with the topic, measured with a 3-point Likert scale, χ2(2, N = 137) = .36, p = .83; • self-reported familiarity with eLearning, measured with a 3-point Likert scale, χ2(2, N = 137) = 3.94, p = .13; • demographics: age F(2, 134) = .4, p = .92, ηp2 = .07 (X = 39, SD = 11), sex χ2(2) = .73, p = .69 (56% female, 44% male), and education level χ2(2, N = 137) = 4.8, p = .09 (75% of the sample ticked the category “Higher education”). Indices. An ANOVA procedure (Table 1) exhibited no significant difference between conditions regarding mean marks obtained at the final test, F(2, 134) = .44, p = .64, ηp2 = .007. Significant differences emerged between conditions with regard to the: • total time spent on the course, F(2, 134) = 3.49, p = .03, ηp2= .05; • number of page views, F(2, 134) = 5.29, p = .006, ηp2 = .07; • learning efficiency (mark at the test/time spent in the course), F(2, 134) = 4.76, p = .01, ηp2 = .01. Table 1. Mean and standard deviation for the indices common to the three conditions Mark at the test Total time spent on Page views Learning efficiency course (in minutes) 1 2 3 1 2 3 1 2 3 1 2 3 (N=34) (N=54) (N=49) M 6.4 6 6.4 250 320 330 57 73 84 0.032 0.024 0.022 SD 2.3 1.7 1.8 120 110 110 23 36 44 0.018 0.014 0.012 Additional Fisher contrast tests disclosed that the differences were significant only against the control group and not between the treatments. This lack of observable divergence made it reasonable and beneficial to statistical power and clarity to rede- fine the treatment conditions as one single group (N = 103) for the following analyses. 16 Annotations as reflection amplifiers in online learning - ARTEL15 4.2 Measures Within Treatment Group Amount of Reflective Enactments. No correlation was found between the mark at the test and the absolute number of annotations (Index 5), characters (Index 6), page views (Index 4) and dashboard views (Index 7). Rates of Reflective Enactments. Beyond the mere amount of reflective actions (NumberAnnot, CharactInAnnot, NumberPages, Dashvisits), the rates at which these enactments occur while studying might be an important aspect of the reflective activi- ty. For this reason, “reflection rates” were calculated to express the displayed reflec- tion per unit of time (minute) for different indices. These rates were obtained for each individual by dividing the quantity of reflective enactments (the different indices) by the individual time spent in the course (Index: TimeSpent). Based on these ratios, post-hoc splits were applied: subjects were categorized against the mean of the group as either high/low annotators (HA/LA via Index 5), high/low producers of annotation characters (HC/LC via Index 6), high/low browsers (HB/LB via Index 4) and high/low visitors of the learning dashboard (HD/LD via Index 7). For instance, participant 45 took 87 annotations (against an average of 43 for the whole group), produced 13958 characters (against an average of 4792), visited a con- tent page 56 times (against an average of 78), and paid 2 visits to the dashboard (against an average of 3). According to the ratios obtained by dividing these indices by the study time (410 minutes for participant 45 against an average 328 minutes), participant 45 was labelled: HAHCLBLD (High Annotator – High producer of Char- acters – Low Browser – Low Dashboarder). It was assumed that this fourfold “learn- ing DNA” captured different facets of the participant’s reflective engagement with the learning material. Assigning such a multivariate reflective engagement profile to the 103 participants revealed some new insights. One-index Learning DNA and Mark at the Test. Table 2 displays the performance of high and low groups for each reflective enactments. When treated independently from each others, the indices deliver significant differences for index 4 (annotations), t(101) = 2.146, p = 0.034, d = 0.35 and for index 5 (characters in annotations), t(101) = 2.76, p = 0.007, d = 0.35. Table 2. High annotators and high producers of characters outperform their low counterparts. This is not the case for high browsers and high dashboarders (index 6). Annotations Characters Dashboard Page views Low High Low High Low High Low High N 59 44 61 42 71 32 53 50 Score 5,8 6,5 5,7 6,7 6 6,3 6,1 6,1 SD 1,81 1,7 1,8 1,4 1,7 1,9 1,5 2 17 Annotations as reflection amplifiers in online learning - ARTEL15 Two-index Learning DNA and Mark at the Test. Twofold combinations of reflec- tive rates, for instance HA+HB (high annotation rate + high browsing rate) versus HA+LB (high annotation rate + low browsing rate) exhibited significant differences at the omnibus ANOVA, F(3, 99) = 3.19, p = .027, ηp2 = .088. Table 3 shows the data for significant cases. Post hoc comparisons using the Fisher LSD test located signifi- cant mean differences between HA+HB and LA+HB (p = 0,002), and HA+HB and LA+LB (p = 0,022), but not between HA+HB and HA+LB (p = 0,082). Table 3. The effect of a high rate of annotation outweighs the effect of browsing rates in two- fold learning DNAs. Twofold learning Mean mark at SD N DNA the test HA+HB 7.1 1.6 23 HA+LB 6 1.7 21 LA+HB 5.6 1.9 29 LA+LB 5.9 1.6 30 Three-index Learning DNA and Mark at the Test.The attempts made with a pro- file combining 3 reflection rates gave a significant mark advantage to the most reflec- tive profile (HA+HB+HD) onto all other combinations. However, the creation of such additional combinations induced more numerous groups (Table 4) and quickly created a problem of statistical power that hampered significance tests. Table 4. Descriptive statistics for more complex learning DNAs Threefold learning DNA Mean score at the test SD N HA+HB+HD 7,8 0,9 5 HA+HB+LD 7 1,7 15 HA+LB+HD 5,6 2,1 8 HA+LB+LD 6,4 1,3 15 LA+HB+HD 6,2 1,7 11 LA+HB+LD 5,2 2,1 18 LA+LB+HD 6,3 2,1 8 LA+LB+LD 5,8 1,4 23 18 Annotations as reflection amplifiers in online learning - ARTEL15 4.3 Qualitative Results The explored qualitative aspects - overall satisfaction, sense of control, perceived intensity of reflection, - were self-reported on 5-point Likert scales in the evaluation questionnaire. Sense of Control. Mann-Whitney test on the sense of control of the high versus low annotation rates (HA/LA) did not disclose significant differences, U = 1225, p = .61, r = .4. But when the browsing rate was added in the profile, the highly engaged people (HA+HB) reported a significantly higher level of control (Mdn = 4) compared to HA+LB (Mdn = 3), LA+HB (Mdn = 3), LA+LB (Mdn = 3), χ²(3, N = 103) = 7.69, p = .04. Stimulation of Reflection by the Annotation Process. When asked about the effect of taking frequent annotations, 71.2% of the sample answered that reflection in- creased, 24.6% that it was not influenced and 4.2% that it diminished. 5 Discussion “Put simply, reflection is about maximising deep and minimising surface approaches to learning.” [30, p. 3]. As a strategy to promote deep learning, this study asked learn- ers to use annotations as “reflection amplifiers”, i.e. brief and repeated reflection af- fordances, interspersed in the learning material and activated, through the support of a dedicated widget, in support to the first-order learning task at hand. These stop-and- think episodes could be seen as a tentative instantiation of “split screen learning” [31] that consists in maintaining a dual focus on the content of the lesson and the acquisi- tion processes that are in play. Overall, the results obtained are disappointing. RAs do expand time on task without delivering benefit for learning achievement: the control group gets the same mark while using less time (see Table 1). Yet, the first hypothesis is not confirmed. From a strict performance-oriented viewpoint, frequent and local annotations are counter-productive. No noticeable performance appeared between distinct uses of the annotation tool: free annotation versus question-based annotation (see section Indices). Yet, the sec- ond hypothesis is not confirmed. This study delivers however one unexpected and intriguing pattern when the anal- ysis operates 2 shifts of focus (see section 4.2): a) from one single reflective enact- ment taken in isolation (writing annotations) to multiple reflective enactments (pro- ducing a longer annotation, navigating amongst course pages, visiting one’s learning dashboard), and b) from the mere amount of these reflective enactments to their rates of use. In this case, results provide insights about ways students balance and combine the primary activity (studying the course) and the secondary reflective activities (an- notations, page re-visits, dashboard views). Here, a different pattern emerges: students who write more personal annotations per unit of time than the average get a higher mark. Combinations of this reflection rate with other reflective enactments (page views, dashboard views) bring extra benefits to performance. The qualitative data also 19 Annotations as reflection amplifiers in online learning - ARTEL15 seems to be affected by combined reflective rates: a significant effect on student’s sense of control is obtained only from high aggregated rates of reflective enactments (see section “Sense of Control”). On this basis, it can be advanced that the dynamics of reflective commitment to a study task encompasses and interweaves several reflective enactments performed at a certain rhythm. It is possible that the reflective passivity of some students might be counteracted by inviting them not only to deploy more reflective actions on the mate- rial but also to accelerate their frequency. 6 Further Work Four main issues raised in this study call for further research. RAs and Performance. Although performance tests are not the only way to measure learning, it remains a legitimate and largely-practised way to assess mastery of con- tent. In this perspective, final scores should reasonably be expected to reflect benefits resulting from using RAs. It has not happened here, at least in comparison with the control group. This lack of benefits from annotations contrasts with other studies in the field [32, 33, 34]. Further empirical studies can help to sort out what the effects of annotations “ought to be” from what they actually accomplish, and most importantly, in what instructional context and for what kind of learners. RAs and Intellectual Dynamics. On a more fundamental level, the study findings, and especially those related to the effects of combined reflective enactment rates (an- notations, revision of annotations, page re-visits), highlight the intellectual dynamics at work in a deep approach of study material. Further investigation is required to es- tablish whether it could be a characteristic of high achievers and a hallmark of intel- lectual life in general to operate an “active study”, defined as an ongoing crisscrossing between a primary learning activity and secondary reflective or meta-cognitive en- actments. This periodic and persistent to-and-fro mental move offers a very different aspect of reflection than the one conveyed by portfolios or learning diaries wherein the experience and the reflection thereon are temporally distinct. If metaphors can be invoked, their complementarity presents as Lego blocks while the real-time dynamics of learning would look more after pinballs. Practical ways to evidence and sustain this interplay between cognitive and reflective landscapes [31] must be explored. Investi- gation of these constant shifts from a primary activity to secondary reflective activi- ties might also benefit from the literature on interrupted tasks [35] since a stop-and- think reflective break can in certain circumstances break the flow or productively coalesce. In this context, interruption rates and optimums [36] become a pivotal no- tion. RAs and Personalized Learning. An important issue for future research on annota- tions is also tied to the selection of relevant frames to ascertain their effects. More research needs to be undertaken to see if prompts used to amplify reflective appraisal of the study material can be related to ownership of learning and sharper feelings that learning is or becomes a “personal matter”. In such an approach, personalised learn- 20 Annotations as reflection amplifiers in online learning - ARTEL15 ing might be seen more as a consequence of seizing action and reflection affordances (37, 38, p. 151) and less as the result of a decision taken by an external agent like a teacher or an adaptive system. RAs and vocabulary. Research in the field will benefit from a closer inspection of the use of the word “reflection” to describe cognitive operations in which the students engage. An observation of the literature devoted to reflection-in-action prompts is up to show how varied the mental processes called reflection can be. For instance, on the one hand can the kind of process invoked in the annotations be called “reflection” or does it resort to “analytical scrutiny”. On the other hand, it does not seem unaccepta- ble to consider the step backward implied by an annotation as a form of reflection. Beyond “processes” – since one always reflects “on” something – improved specifica- tions of “objects” of reflection critical for the growth of professional learners are needed to guide instructors in concrete instructional designs. 7 Limitations of the Study While the results related to hypotheses 1 and 2 have been obtained through acknowl- edged statistical procedures and are, on this basis, totally sound, those related to the compound score of reflective rates must be taken with caution as they emerge from the pooling of the treatment groups (free annotations/question-based annotations) and the use of dichotomized data (e.g. High versus low users of the dashboard), two methodological decisions that can be discussed. With regard to pooling, Verma, Gagliardin, and Ferretti [39] highlight that a “sufficient” degree of comparability is a precondition for such pooling to be meaningful”. In this case, the comparability of the treatment groups at baseline, the identical assignment of taking a note at each page visit made to both groups, and the similar results obtained against the control group were considered sufficient to justify the pooling in an exploratory perspective. With regard to dichotomization, DeCoster, Iselin, and Gallucci [40] conclude their compre- hensive review on the issue by pointing specific circumstances wherein cut-off points are acceptable, among which “the purpose of the research is to investigate how a di- chotomized measure will perform in the field” (p. 364). This is the case with the work on reflective rates which has no other purpose than exploring whether frequent anno- tations, with no effect taken separately (hypotheses 1 and 2 are not confirmed), may, combined to other reflective enactments, reach a certain threshold, at which point they have tangible effect. The use of cut-off points, as obtained through dichotomized indicators, was also influenced by readability concerns: “big categories” (high versus low) help to prevent a “drowning by numbers” effect which can go along with unusu- al and complex variables such as browsing or annotation rates. Based on these mo- tives, it appeared reasonable to proceed as described even though the use of regres- sion analysis might have been a more straightforward method. The decision of not to analyse the content of the annotations to compare it to the course content can also be considered as a second caveat, especially when Natural Language Processing tools, like Latent Semantic Analysis (LSA) or Latent Dirichlet Allocation (LDA), provide sufficiently good measures of it. This approach has been left to further studies for 2 reasons. On the one hand, measuring the quality of an an- 21 Annotations as reflection amplifiers in online learning - ARTEL15 notation is difficult in itself because the learner’s cognitive context around it does exist but cannot always be grasped by the researcher. Furthermore, the neglect of the fine-grained qualitative aspects was a decision flowing from the initial scope of study. Its chief postulate is that quality learning is encouraged by a permanent criss-crossing between an ongoing learning processes and explicit/structured episodes of reflection. The experimental design attempted to reproduce this intertwine somewhat artificially (that experimental design simply failed at supporting this dynamic cannot be exclud- ed) with the annotation RA. The work was therefore more acquainted with quantita- tive measures than with qualitative ones. To address the latter, other instruments and methodologies – out of the initial scope of the study – should enrich the research ef- fort. They would give insights into the actual engagement with the RA and into the quality level of the prompted reflection. Some quality check of the content of the annotations, by students and/or researchers, could also enrich the set-up, for instance by have a group where students try to answer the annotation questions just before passing the test or through annotation sharing mechanisms. The static nature of the RA used in this study is also a limitation. The annotation tool presents as a neutral artefact that becomes available in a pre-defined way. The RA remains ignorant of the profile or the learning activities carried out by the student. In this context of self-instruction, the reflective activity is also deprived of feedback. All these limitations may qualify this RA in this context as simply inappropriate or insufficient. Lastly, the present study takes place in a real-world context with the highs and lows of this approach. Yet, the direction of the effect in each hypothesis may be put in question as some hidden variables (e.g. motivation, self-efficacy, availability for a course taken on a voluntary basis, in the changing situations of life, etc.), beyond those controlled (see section Background questionnaire), may work as moderators. 8 Concluding remarks A growing literature extols the importance to instil reflection and deep approaches to learning in tuition. However, practical and systematic ways to operate are not con- spicuous, at least when it comes to reflection in methods of learning considered as traditional or transmissive [41], in contrast to constructivist methods (problem-based learning, collaborative learning) wherein reflection is claimed to be “built-in” [42, 43, 44]. This study inquired the question: how to induce a more thoughtful autonomous study of learning material? To answer, the experimental setting artificially increased the number of annotations, conceptualised as frequent tinglings for reflection while reading and purposed to support a persistent dynamic mental engagement with the reading material. An assumption guided this work: that such a kind of active and re- flective posture to learning, which constantly articulates the cognitive and the meta- cognitive landscapes, is a key feature of intellectual life. The experimental setting presented here was a simplified attempt to mimic and externalize such fundamental inner dynamic processes via an annotation tool. Eventually, annotations taken alone did not really measure up. However, some elements of the study suggest that the fre- quency and the aggregation of different reflective behaviours can be worth exploring further in connection to quality learning. 22 Annotations as reflection amplifiers in online learning - ARTEL15 References 1. Jackson, H. J.: Marginalia: Readers writing in Books. Yale University Press, New Haven, (2001) 2. Crawford, C. C.: Some experimental studies of the results of college note-taking. Journal of Educational Research,12, 379-386 (1925) 3. Schön, D.: The Reflective Practitioner: How professionals think in action. Temple Smith, London (1983) 4. Commonwealth: Creating learning materials for open and distance learning - A handbook for authors and instructional designers. Commonwealth of Learning, Vancouver (2005) 5. Open University: Guide for the Appointment of Tutorial and Counselling Staff. Open Teaching Toolkits, Open University (1991-1994) 6. Dillon, A.: Reading from paper versus screens: a critical review of the empirical literature. Ergonomics 35(10), 1297-1326 (1992). doi: 10.1080/00140139208967394 7. Priolli, P. (2007): Information Foraging Theory: Adaptive Interaction with Information. Oxford University Press, Oxford-New York (2007) 8. Boch, F., Piolat, A.: Note taking and learning: A summary of research. The WAC Journal 16, 101-113 (2005) 9. Hartley, J., Davies, I. K.: Note-taking: A critical review. Innovations in Education & Training International 15(3), 207-224 (1978) 10. Slotte, V., Lonka, K.: Note-taking review – Practical value for learners. Arob@se 1-2, 79- 86 (2003) 11. Makany, T., Kemp, J., Dror, I. E.: Optimising the use of note-taking as an external cogni- tive aid for increasing learning. British Journal of Educational Technology 40(4), 619-635 (2009). doi: 10.1111/j.1467-8535.2008.00906.x 12. Degani, A., Wiener, E. L.: Human factor of flight-deck checklists: The normal checklist. NASA, Moffett Field (1990). 13. Van der Baaren, J., Schuwer, R., Kirschner, P., Hendriks, M.: Finding your way into an open online learning community. Journal of Interactive Media in Education, 4 (2008). 14. Hwang, W.-Y., Chen, N.-S., Shadiev, R., Li, J.-S.: Effects of reviewing annotations and homework solutions on math learning achievement. British Journal of Educational Tech- nology 42(6), 1016-1028 (2011). 15. Kam, M., Wang, J., Iles, A., Tse, E., Chiu, J., Glaser, D., Canny, J.: Livenotes: a system for cooperative and augmented note-taking in lectures. Paper presented at the CHI '05: Proceedings of the SIGCHI conference on Human factors in computing systems, Portland, Oregon, USA (2005) 16. Glahn, C.: Contextual support of social engagement and reflection on the Web. PhD series, Open Universiteit, Heerlen, The Netherlands (2009). https://www.ou.nl/Docs/Onderzoek/Promoties/2009/Christian_Glahn_thesis.pdf 17. Verpoorten, D., Glahn, C., Chatti, A., Westera, W., Specht, M.: Self-Reported Learning Effects of a Tagging Activity Carried out in a Personal Learning Environment (PLE) by Secondary-School Pupils. International Journal for Cross-Disciplinary Subjects in Educa- tion 2(1), 276-284 (2011). 18. Schilit, B., Golovchinsky, G., Price, M.: Beyond paper: supporting active reading with free form digital ink annotations. Paper presented at the Conference on Human factors in com- puting systems, Los Angeles, CA, USA (1998). http://orbi.ulg.ac.be/handle/2268/151959 19. Kawase, R., Papadakis, G., Herder, E., Nejdl, W.: Creation and Use of Annotations in Dif- ferent Usage Contexts. L3S Research Center Hannover (NA). http://stellarnet.eu/d/1/2/images/a/ae/Sss1.pdf 23 Annotations as reflection amplifiers in online learning - ARTEL15 20. Hwang, W.-Y., Hsu, G.-L.: The Effects of Pre-Reading and Sharing Mechanisms on Learning with the Use of Annotations. Turkish Online Journal of Educational Technology – TOJET 10(2), 234-249 (2011). http://files.eric.ed.gov/fulltext/EJ932242.pdf 21. Glover, I., Xu, Z., Hardaker, G. Online annotation – Research and practices. Computers & Education 49(4), 1308-1320 (2007). doi: 10.1016/j.compedu.2006.02.006 22. Verpoorten, D., Westera, W., Specht, M.: Reflection amplifiers in online courses: a classi- fication framework. Journal of Interactive Learning Research 21(4), 654-666 (2010) http://orbi.ulg.ac.be/handle/2268/151374 23. Boud, D., Keogh, R., & Walker, D.: Reflection, Turning Experience into Learning. Kogan Page, London (1985) 24. Butler, D. L., Winne, P. H.: Feedback and Self-Regulated Learning: A Theoretical Synthe- sis. Review of Educational Research 65(3), 245-281 (1995) 25. Verpoorten, D., Westera, W., Specht, M.: Infusing reflective practice in eLearning courses - can widgets help? International Journal of Technology Enhanced Learning 3(1), 93- 109109 (2011). http://orbi.ulg.ac.be/handle/2268/151348 26. Eshuis, J.H., Goltstein, G.P.: Seks en de evolutie. Open Universiteit Nederland, Heerlen (2007). http://www.ou.nl/eCache/DEF/2/22/850.html 27. Verpoorten, D., Westera, W., Specht, M.: A first approach to “Learning Dashboards” in formal learning contexts. Paper presented at the ADVTEL workshop (1st International Workshop on Enhancing Learning with Ambient Displays and Visualization Techniques) at EC-TEL 2011, Palermo, Italy (2011). http://orbi.ulg.ac.be/handle/2268/151988 28. Bijker, M., Van der Klink, M., Boshuizen, H.: Modeling self-efficacy, self-regulation, self- directed learning and career processes of adult professionals and relations with learning outcomes and labour market success. Paper presented at the 5th EARLI-SIG14 Learning and Professional Development, Munich, Germany (2010) 29. Schraw, G., Dennison, R.: Assessing metacognitive awareness. Contemporary Educational Psychology 19, 460-475 (1994). 30. Hinett, K.: Improving Learning through Reflection.York: Higher Education Academy, York (2002) 31. Claxton, G.: Expanding the capacity to learn: a new end for education? Opening keynote address presented at the British Educational Research Association Annual Conference, University of Warwick, UK. (2006). http://www.buildinglearningpower.co.uk/includes/pdf_library/GC_BERA_Keynote.pdf 32. Hwang, W.-Y., Wang, C.-Y., Sharples, M.: A Study of Multimedia Annotation of Web- Based Materials. Computers and Education, 48(4), 680-699 (2007) 33. Nokelainen, P., Miettinen, M., Kurhila, J., Floréen, P., Tirri, H. : A shared document-based annotation tool to support learner-centred collaborative learning. British Journal of Educa- tional Technology 36(5), 757-770 (2005) 34. Slotte, V., Lonka, K.: Review and process effects of spontaneous note-taking on text com- prehension. Contemporary Educational Psychology, 24, 1-20 (1999) 35. Mark, G., Gudith, D., Klocke, U. The cost of interrupted work: More speed and stress. Pa- per presented at the 26th annual SIGCHI Conference on Human factors in computing sys- tems, Florence, Italy (2008). doi: 10.1145/1357054.1357072 36. Palanque, P., Winckler, M., Martinie, C.: A Formal Model-Based Approach for Designing Interruptions-Tolerant Advanced User Interfaces. Computational Intelligence 340, 143-169 (2011). doi 10.1007/978-3-642-14562-9_8 37. Verpoorten, D., Glahn, C., Kravcik, M., Ternier, S., Specht, M.: Personalisation of learn- ing in virtual learning environments. Learning in the Synergy of Multiple Disciplines. Pro- 24 Annotations as reflection amplifiers in online learning - ARTEL15 ceedings 4th European Conference on Technology Enhanced Learning, EC-TEL 2009, 52- 66 (2009). http://orbi.ulg.ac.be/handle/2268/151458 38. Crawford, M.: The world beyond your head. How to flourish in an age of distraction. Vi- king, Great Britain (2015) 39. Verma, V., Gagliardi, F., Ferretti, C.: On pooling of data and measures. DMQ 84 (2009) 40. DeCoster, J., Iselin, A.-M. R., Gallucci, M.: A conceptual and empirical examination of justifications for dichotomization. Psychological Methods 14(4), 349-366 (2009). doi: 10.1037/a0016956 41. Terhart, E.: Constructivism and teaching: a new paradigm in general didactics? Journal of Curriculum Studies 35(1), 25-44 (2003). 42. Kirschner, P. A., Sweller, J., Clark, R. E.: Why Minimal Guidance During Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and Inquiry-Based Teaching. Educational Psychologist 41(2), 75-86 (2006) 43. Hmelo-Silver, C. E., Duncan, R. G., Chinn, C. A.: Scaffolding and Achievement in Prob- lem-Based and Inquiry Learning: A Response to Kirschner, Sweller, and Clark (2006). Educational Psychologist 42(2), 99-107 (2007). doi: 10.1080/00461520701263368 44. Sweller, J., Kirschner, P. A., Clark, R. E.: Why Minimally Guided Teaching Techniques Do Not Work: A Reply to Commentaries. Educational Psychologist 42(2), 115-121 (2007). doi: 10.1080/00461520701263426 25 26 Formal Concept Analysis for Modelling Students in a Technology-enhanced Learning Setting Michael A. Bedek, Michael Kickmeier-Rust, and Dietrich Albert Knowledge Technologies Institute, Graz University of Technology, Austria {michael.bedek, michael.kickmeier-rust, dietrich.albert}@ tugraz.at Abstract. We suggest the Formal Concept Analysis (FCA) as theoretical back- bone in technology-enhanced learning settings to support a students´ learning process in two ways: i) by engaging with concept lattices, the structure of the knowledge domain and the interrelations of its concepts become explicit, and ii) by providing visual feedback in form of open learner modelling, the student´s reflection on the own strengths and weaknesses is facilitated. For teachers, the FCA provides intuitive visualizations for a set of pedagogically relevant ques- tions, concerning the performance of students on the individual- as well as on the class-level. Keywords: Formal Concept Analysis, Learning Analytics, Visualizations, Learner Modelling 1 Introduction The increasing availability of comprehensive technology-enhanced learning (TEL) environments or single educational tools and apps enables students to easily advance their knowledge without direct support from a teacher. However, teachers are chal- lenged by the need to provide appropriate learning resources and to keep up with students’ learning progress without reverting to exams or tests [1]. Learning analytics and educational data mining are two highly interrelated research fields that aim to help teaches and educators to make previously hidden insights explicit (e.g. [2]). When applying learning analytics and educational data mining in schools, it is of high importance to meet the requirements of teachers and students. Teachers usually want to have user-friendly tools that help them to reduce the time required for personalized assessment and tailored competence development of their students. We suggest the Formal Concept Analysis (FCA) as a framework for addressing these requirements. A so-called FCA-tool has been developed in the course of the EU-funded project weSPOT (http://wespot.net/home), which provides a Working Environment with Social and Personal Open Tools to support students in developing their inquiry based learning skills. In the context of weSPOT, the FCA-tool is mainly used by students by guiding them through a knowledge domain, predefined and en- 27 Formal concept analysis for modelling students - ARTEL15 riched with learning resources by their teacher. For a more technical description of the FCA-tool´s features see [3]. In weSPOT, the FCA-tool supports learners by enabling domain and open learner modelling. The fields of application of the FCA in general, and the FCA-tool in particular, have been extended in the course of the LEA´s BOX project (http://leas-box.eu/) which stands for Learning Analytics Toolbox. In the con- text of LEA´s BOX, the FCA-tool is mainly used by teachers for student modelling and visualization of educational data. By applying the formal concept analysis with students´ performance data, a set of pedagogically relevant questions for teachers can be addressed and visualized. 2 Formal Concept Analysis The FCA describes concepts and concept hierarchies in mathematical terms, based on the application of order and lattice theory [4]. The starting point is the definition of the formal context K which can be described as a triple (G, M, I) consisting of a set of objects G, a set of attributes M and a binary relation I between the objects and the attributes (i.e. “g I m” means “the object g has attribute m”). A formal context can be represented as a cross table, with objects in the rows, attributes in the columns and assigned relations as selected cells. An example of a formal context is shown in Fig. 1. This formal context has been created by the FCA-tools Editor View. Teachers use the Editor View to define the formal context and to add learning resources (URLs or files) which can be assigned to both objects and attributes, respectively. Fig. 1. FCA-tool´s Editor View for creating a domain with objects, attributes, and relations. In order to create a concept lattice, for each subset A ∈ G and B ∈ M, the following derivation operators need to be defined: A ↦ A´ := {m ∈ M I g I m for all g ∈ A}, which is the set of common attributes of the objects in A, and B ↦ B´ := {g ϵ G I g I m for all m ∈ B}, which is the set of objects which have all attributes of B in common. 28 Formal concept analysis for modelling students - ARTEL15 A formal concept is a pair (A, B) which fulfils A’ = B and B´ = A. The set of objects A is called the extension of the formal concept; it is the set of objects that encompass the formal concept. The set B is called the concept’s intension, i.e. the set of attrib- utes, which apply to all objects of the extension. The ordered set of all formal con- cepts is called the concept lattice B(K) (see [5] for details), which can be represented as a labelled line diagram (see Fig. 2). Fig. 2. Concept lattice The concept lattice shown in Fig. 2 has been created by the FCA-tool´s Lattice View. Every node represents a formal concept. The extension A of a particular formal concept is constituted by the objects that can be reached by descending paths from that node. As an example, the node with the label “Goldfish” has the extension {Gold- fish, Tree frog}. The intension B is represented by all attributes that can be reached by an ascending path from that node. In the example above, the intension consists of {is able to swim, lives in / on the water}. 3 Domain Learning and Open Learner Modelling Once the teacher has created the formal context, students can explore the resulting concept lattice by engaging in interactive graph visualizations (see Fig. 2). By select- ing a node, the corresponding concept´s extension and intension are illustrated in a highlighted manner. The concept lattice makes the structure of the knowledge domain and the interrelations of its concepts explicit. Similar as for concept maps, this kind of graphic organizer aims to facilitate meaningful learning by activating prior knowledge and illustrating its relationship with new concepts [6]. In case the teacher also assigned learning resources to the objects and attributes in the FCA-tools Editor View open learner modelling can be supported (see Fig. 3). Vis- ualizations of open learner models (for an overview see [7]) are aiming to facilitate reflection on the side of the students and to support teachers to better understand strengths and weaknesses of their students. 29 Formal concept analysis for modelling students - ARTEL15 Fig. 3. FCA-tools Lattice View for visualizing domain- and learner models. The FCA-tool's Lattice View applies the often-used traffic-light analogy (see e.g. [8]) to show the student the extent to which he or she already consumed learning re- sources. 4 Applying the FCA(-tool) as a teacher Similar as [9] who were the first who applied the FCA with students and their per- formance data we suggest formal contexts with student as “attributes” and problems or test-items as “objects”. The relation between these two sets means “student m has solved test item g”. Fig. 4. Concept lattice with students as attributes (numbers from 01 to 23) and test items (letters a, b, c, d, e, and f) as objects (data reported by [10]). 30 Formal concept analysis for modelling students - ARTEL15 An example of a concept lattice which results from such a formal context is shown in Fig. 4 (the data has been reported by [10]). As briefly outlined in the following sections, such a concept lattice visualizes answers to a set of pedagogical questions which are of high interest for teachers. 4.1 Depicting information from the formal concepts extensions and intensions As mentioned above, the set of test items which have been solved by a particular stu- dent can be directly depicted from the extension of the formal concept with the stu- dents´ label assigned to it. As an example in Fig. 4, student 10 is the only one who solved only a single test item, c, and students 03 and 17 (assigned to the top element of the concept lattice) mastered all problems. When clicking on a particular node the formal concept´s extension and intension is highlighted. As an example shown in Fig. 5 (left side), the student 04 has successfully mastered the test items a and b. Fig. 5. The extension represents the set of test items solved by a student (see student 04) and the intension represents the set of students who solved the particular test item (see test item d) The intension of a formal concept which has an object-label assigned to it indi- cates the set of students which have successfully mastered the according test item. As an example, the problem d in Fig. 5 (right side) has been solved by the students 01, 03, 05, 07 and 17. As it can be also seen, this formal concept located above the formal concept with the object-label e assigned to it. This means, that all students who solved item d were also able to solve item e. 4.2 Highlighting overlaps and differences of students performances The performances of two or more students can be compared when examining the intensions of the formal concepts with the according attribute-labels. As an example, the students 07 and 15 mastered different subsets of problems (see Fig. 4): Student 07 mastered the items b, d, e and f while student 15 mastered the items a, b, c, and f. Both students mastered items b and f (which is the set closure of their intensions) and together they mastered all problems (which is the set union of their intensions). 31 Formal concept analysis for modelling students - ARTEL15 As a teacher, such kind of information might be of great interest since it helps to effectively arrange groups of students when aiming for collaborative, peer-learning (where students learn together in groups). In the example above, the students 07 and 15 together could be tutors for other students. 4.3 Visualizing a classrooms´ learning progress over time The concept lattices shown in Fig. 4 and Fig. 5 are the result of a formal context which is an evaluation of the students´ performances at a certain point in time. How- ever, in some cases it might be of great interest for a teacher to observe the learning progress over a longer period of time. Ideally, all students might be able to master all items at the end of course or the semester. In such a case, all cells in the formal con- text would be filled with crosses. This would result in a concept lattice with only a single formal concept. Fig. 6 exemplifies such a learning progress over time. The concept lattice in the middle results from adding one solved item to the students´ per- formance states (except for the students 03 and 17). The concept lattice on the right results from adding another item to the student´s performance states. Fig. 6. Concept lattices changing over time reflect the learning progress of the class of students In general, the visual appearance of the concept lattice gives a first impression of the student’s coherence with respects to their performance: A concept lattice which looks “complex” due to a large amount of formal concepts is an indication for a high diversity among the students´ performances. 5 Discussion and Outlook In the previous sections, we suggested to apply the FCA to support students and teachers. Students apply the FCA, respectively the FCA-tool, to learn a knowledge domain by interacting with the concept lattice which makes previously hidden interre- lationships between the domain´s concepts explicit. In addition to that, a student´s reflection upon his or her learned and still-to-learn concepts is supported by an open learning modelling approach. Summative evaluation studies on the effect of these pedagogical principles are still ongoing in the course of the weSPOT project. In the context of LEA´s BOX, also teachers apply the FCA(-tool) to visualize the answers to a set of pedagogical questions which are of high interest for them. These 32 Formal concept analysis for modelling students - ARTEL15 pedagogical questions described in this paper are the result of small focus groups and interviews with teachers in the early phase of the LEA´s BOX project. The resulting visualizations as shown above are currently in the spotlight of formative, qualitative evaluation studies with small focused groups of teachers. Current work on the tech- nical side of the project focuses on the development of interactive visualizations which can be easily used by teachers in the classroom. Early feedback of teachers concerns the complexity of the concept lattices, in particular when dealing with a great amount of problems (respectively competences and skills). Conceptual research and the elaboration of ideas on how to reduce this complexity without reducing the amount of information which can be extracted and deduced from the visualizations will be the main focus of our work in the near future. Acknowledgement. The research leading to these results has received funding from the European Community's Seventh Framework Program (FP7) under grant agree- ments no 318499 (weSPOT project) and no. 619762 (LEA´s BOX). This document does not represent the opinion of the EC and the EC is not responsible for any use that might be made of its content. References 1. Shute, V.J.: Simply Assessment. International Journal of Learning, and Media. 1, 1-11 (2009) 2. Steiner, C., Kickmeier-Rust, M., Albert, D.: Learning Analytics and Educational Data Mining: An Overview of Recent Techniques. In: Proceedings of the EC-TEL 2014 work- shop ´Learning analytics for and in serious games´, pp. 6 – 15, (2014) 3. Bedek, M., Kopeinik, S., Pruenster, B., Albert, D.: Applying the Formal Concept Analysis to introduce guidance in an inquiry-based learning environment. In: 15th IEEE Interna- tional Conference on Advanced Learning. IEEE Press, New York (2015, in press) 4. Wille, R.: Restructuring lattice theory: an approach based on hierarchies of concepts. In: Rival, I. (ed.) Ordered sets, pp. 445-470. Reidel, Dordrecht-Boston (1982) 5. Wille, R.: Formal Concept Analysis as Mathematical Theory of Concepts and Concept Hi- erarchies. In: Ganter, B., Stumme, G., Wille, R. (eds) Formal Concept Analysis, pp 1-34. Springer, Berlin (2005) 6. Nesbit, J.C., Adesope, O.O.: Learning With Concept and Knowledge Maps: A Meta- Analysis. Review of Educational Research. 76, 413–448 (2006) 7. Bull, S., Kay, J.: Open Learner Models. Nkambou, R., Bordeau, J., Miziguchi, R. (eds.) Advances in Intelligent Tutoring Systems, pp. 318-388. Springer, Heidelberg: Springer (2010) 8. Arnold, K.E., Pistilly, M.D.: Course Signals at Purdue: Using learning analytics to increase student success, In: Proceedings of the 2nd International Conference on Learning Analyt- ics and Knowledge, pp. 267-270. ACM, New York (2012) 9. Rusch, A., Wille, R.: Knowledge spaces and formal concept analysis. In: Bock, H.H., Po- lasek, W. (eds.) Data analysis and information systems: Statistical and conceptual ap- proaches, pp. 427-436. Springer, Berlin (1996) 10. Korossy, K.: Modeling Knowledge as Competence and Performance. In: Albert, D., Lu- kas, J. (eds.) Knowledge Spaces: Theories, Empirical Research, and Applications, pp. 103- 132. Springer, Mahwah, NJ (1999) 33 34 Learning to Look – Purpose and Design of an Awareness- Raising Online Course in Veterinary Sciences Sophie Tasnier, Valeria Busoni, Christian Hanzen, Jeff Van de Poël, Géraldine Bolen, Catherine Delguste, Nadine Antoine, Véronique Delvaux, Tania Art, and Dominique Verpoorten University of Liège, Belgium {sophie.tasnier, vbusoni, christian.hanzen, jvandepoel, gbolen, cdelguste, nadine.antoine, vdelvaux, tatania.art, dvepoorten}@ulg.ac.be Abstract. This paper reports on a work in progress: an online self-instruction course created to stimulate students’ awareness processes when dealing with pictures. Using non-clinical material, the “Learning to Look” course was de- signed as a preliminary training to the observation of histological sections, radi- ological graphs, and other specialized visual material. Following a presentation of the project, salient results of a feedback questionnaire completed by 382 stu- dents about their experience of the course are provided. Keywords: awareness, veterinary sciences, observation, visual skills, visual acuity, attentional training, clinical images, multimodal literacy, learning to look 1 Introduction Observation is a core-competency for veterinarians and more generally for health professionals [1, 2, 3]. Correct prognosis and appropriate treatment always depend on correct identification of animal behaviors, signals, and symptoms. This “ability to look” is nowadays also directed at a growing range of images (in radiology, cardiolo- gy, endoscopy, microscopy, etc.) more and more invoked in daily veterinary practice. This context spurs the need of a renewed pedagogical reflection on appropriate ways to improve veterinary student visual awareness in use of clinical images. 2 Observed Problems The “Learning to Look” project stemmed from observations made by a group of teachers from the University of Liege: 35 Learning to look - purpose and design of an awareness-raising online course - ARTEL15 • students have problems when confronted with visual material like histological sections, radiological graphs or dynamic recordings of clinical situations; • despite its importance and its indirect assessment in some courses, noticing what is in an image is not a skill taught in a targeted and coordinated way; • several courses train somehow abilities to see. However, the specific visual materi- al and activities they use hamper a generic and methodical approach of picture ex- ploitation (to look, to spot, to describe, to analyze, to interpret). 3 An Online Course To tackle these problems, an online course called “Learning to look” (“Savoir voir”) was designed and offered, on a voluntary basis, to 630 bachelor and master students, as a preparation for specific courses and practical sessions dealing with clinical imag- es. The course was divided in 3 modules: Module A - Learning to look at an image, Module B - Learning to describe an image, and Module C - Learning to interpret an image. This paper focuses on Module A which matches the general topic of ARTEL workshop: awareness. The main purpose of Module A was to train students’ ability to apply sustained at- tendance to an image (see examples in Fig. 1, 2, 3) in order to identify things of inter- est within (or possibly absent from) it. (“Recall that the basic concept at the root of attention is selection: we pick something out from the flux of the available”, [4, p. 86]). Beyond this training of “visual acuity” the module also intended to foster aware- ness to own attentional processes in learning activities prescribing to exercise control over how and what to look. In the Module A of the “Learning to Look” course, these attentional processes were applied by students onto unspecific pictures (i.e. non-clinical images and with- out relation to existing courses). This choice was deliberate and made for two reasons. On the one hand, transversal benefits (not related to a precise type of picture) were targeted. On the other hand, using arbitrary imagery at baseline provided a soft entry in the attentional training process. Medical visuals were steadily incorporated in sub- sequent modules of the eLearning course. The course was released on the institutional eLearning platform Blackboard. Stu- dents got 45 days (from March 1st to mid-April 2015) to cover it according to a self- study modality. 36 Learning to look - purpose and design of an awareness-raising online course - ARTEL15 4 Instructional Design The instructional design of Module A presented as a series of pictures that students had to observe. A question (see examples in captions of Fig. 1, 2, 3) challenged them to find, match, or discriminate visual elements, with a time limit in some cases (a mean time was calculated from the performance of a group of students who acted as beta-testers of the course). For each picture, students answered the question either by clicking on sensible spot(s) on the image or by selecting one option in a list. They received an immediate feedback on their answer’s correctness and on the time they had spent on the image (compared to the yardstick). Feedback was also enriched with pieces of advice (e.g. “You did not find all relevant elements. Try to scan the image in a systematic way” or “You did not spot the elements fast enough. Keep in mind that speed is also a parameter of visual performance”). Following the completion of all exercises, students received a compound awareness score. Pictures were selected in existing material [e.g. 5] and displayed in an Adobe Cap- tivate format in order to benefit from responsive design (mouse-over and embedded countdown features) and easy quizzing. Module A was structured in 3 gradual pools of 10 to15 exercises prompting various aspects of awareness according to a semiolog- ical approach which assumes that because the meaning is not “lying” there on the picture, one has to make an effort to grasp it [6, p. 343]. The Pool “To observe and to spot” had for purpose to help students to realize the importance of sustained awareness, to introduce to a technique of visual scanning, and to understand the notion of “awareness efficiency” (ratio “time spent observ- ing/amount of elements discovered”). Fig. 1. Exercise from the Pool “To observe and to spot”. The picture (credit: ULg) came with the question: “How many animals do you see?” 37 Learning to look - purpose and design of an awareness-raising online course - ARTEL15 The Pool “To compare and to measure” was shaped around selective awareness, systematic capture of differences, and relative dimensions of objects. Fig. 2. Exercise from the Pool “To compare and to measure”. The picture (credit: Freepick) came with the assignment: “In the left frame, click on the horse that has no inverted twin in the right frame”. The Pool “To observe in 3-dimension space” revolved around shape matching and mental construction of 3D objects from either sections of these objects or from 2D representations. Fig. 3. Exercise from the Pool “To observe in 3-dimension space”. The picture (credit: ULg) came with the assignment: “Associate each view to its corresponding radiography”. 38 Learning to look - purpose and design of an awareness-raising online course - ARTEL15 Two reflection amplifiers [7] were added to Pool A in view of prompting students’ metacognitive introspection on their usual way to look and, in contrast, on the value of the visual scanning approach proposed (Fig. 1). Fig. 4. Structured reflective episodes questioned students on their spontaneous way of observ- ing and on their possible improvement through visual scanning. 5 Feedback Questionnaire A feedback questionnaire about the course was sent to the students. It comprised mul- tiple choice questions coupled with open fields for written comments. It was filled in by 382 students, with the following results: • 84% agreed or totally agreed that the exercises contributed to a reinforcement of their ability to analyze an image; • respondents elaborated on the benefits: “I have learnt to go beyond what is the most visible in a picture” (50%), “I have learnt the technique of systematic ap- proach (48%), “I have learnt that you can miss an image if you do not spot all im- portant elements (44%); • 80% were in favor of a module dedicated to digital imagery and based on the same principles; • despite their appreciation of the “Learning to look” course, some students men- tioned that it added workload to an already heavy curriculum; • some students complained about the exercises for which missing only one image element out of many deprived of all the points. 39 Learning to look - purpose and design of an awareness-raising online course - ARTEL15 6 Discussion and Future Work Concern for the training of awareness is a long-standing issue. In 1942, the French philosopher S. Weil assimilated the major outcome of formal education to the devel- opment of attentional skills: “Although today this seems unknown, the training of the faculty of attention is the true goal and almost only value of all study. Most school exercises have a certain intrinsic value, but this is purely of secondary interest. All exercises which help to develop the power of attention are of interest, almost equally so. (…) Those who spend their formative years without developing this faculty of attending and directing mind to an object have missed a chief treasure” [8, p. 85]. Since then, other works [9, 10, 4] have stressed the importance of awareness (and germane notions). Amongst possible objects of attention, pictures form a distinct category. Becoming visually literate is considered as an important endeavor for students, especially nowa- days in veterinarian and medical education wherein static and dynamic digital image- ry has gained momentum [11]. Indeed, at the same time, efforts are made to partly automatize recognition of terabytes of imaging data produced in many domains. But even the best algorithm-oriented processes does not discard human intervention and hybrid human-computer approaches of visual interpretation still appear as relevant [12]. Developing visual awareness remains therefore critical, not to mention its im- portance in new technological areas (augmented reality, quantified self, learning ana- lytics, game-based learning, remote sensing imagery, etc. [13]) generating images that both burden and relieve attentional resources. If eye can learn and must learn, the question of how to teach it is open [14, 15, 16]. The “Learning to Look” project offers a concrete, grounded in practice, and large-scale attempt to exert a competency which is seldom trained for itself, despite its paramount importance for future veterinary practitioners. This instructional setting, promoting a straightforward attention drill, must be further analyzed with regard to its relevance, efficacy conditions, and contribution to multimodal literacy development [17, 18, 19]. In this respect, a detailed assessment of gains in image handling, con- veyed both by Module A as such and by the whole course (Module B - Describe an image and Module C - Interpret an image have been made available to students) is planned for the future. Since the “Learning to Look” course is based on the assumption that awareness development can be stimulated through training from the general to the specific (from arbitrary to clinical images), instructors also plan to explore the possible use of the online course beyond veterinary sciences. 40 Learning to look - purpose and design of an awareness-raising online course - ARTEL15 Based on students’ positive reaction on this first run of the module, the instructors will ascertain the best schedule regarding the course release. Offering its content ac- cording to a distributed practice scheme [20, 21, p. 114] along the year – instead of a massed practice during a short period – could expand the benefits of the module and foster more sustainable learning. Technical options for an improved tracking of stu- dents’ actions and scores will also be inspected. References 1. Nakashima, R., Kobayashi, K., Maeda, E., Yoshikawa, T., Yokosawa, K..: Visual Search of Experts in Medical Image Reading: The Effect of Training, Target Prevalence, and Ex- pert Knowledge. Frontiers in Psychology 4:166 (2013). doi: 10.3389/fpsyg.2013.00166 2. Snowden, P. T., Davies, I. R. L., Roling, P.: Perceptual learning of the detection of fea- tures in X-ray images: A functional role for improvements in adults' visual sensitivity? Journal of Experimental Psychology: Human Perception and Performance 26(1), 379-390 (2000). doi: 10.1037/0096-1523.26.1.379 3. Van der Gijp, A., Van der Schaaf, M. F., Van der Schaaf, I. C., Huige, J. C., Ravesloot, C. J., Van Schaik, J. P., Ten Cate, T. J.: Interpretation of radiological images: towards a framework of knowledge and skills. Adv Health Sci Educ Theory Pract 19(4), 565-580 (2014). doi: 10.1007/s10459-013-9488-y 4. Crawford, M.: The world beyond your head. How to flourish in an age of distraction. Vi- king, Great Britain (2015) 5. Steiner, J.: Look-Alikes: The More You Look, the More You See!. Little, Brown Books for Young Readers (2003) 6. Seale, C.: Social research methods: a reader. Routledge, New York (2004) 7. Verpoorten, D., Westera, W., Specht, M.: Reflection amplifiers in online courses: a classi- fication framework. Journal of Interactive Learning Research 22(2), 167-190 (2011) http://hdl.handle.net/2268/151374 8. Weil, S.: Réflexions sur le bon usage des études scolaires en vue de l'Amour de Dieu [Re- flections on the good use of sholastic studies in view of the Love of God]. In: Perrin, J.-M. (Ed.) Attente de Dieu [Waiting for God], pp. 85-96. Fayard, Paris (1966) 9. Endsley, M. R.: Theoretical Underpinnings of Situation Awareness: A Critical Review. In Endsley, M. R., Garland, D. J. (eds.) Situation Awareness Analysis and Measurement, pp. 3-29. Lawrence Erlbaum Associates, Mahwah, NJ & London (2000) 10. Salomon, G., Globerson, T.: Skill may not be enough: The role of mindfulness in learning and transfer. Internationul Journal of Educutional Research 2, 623-438 (1987) 11. Linsen, L., Hagen, H., Hamann, B., Hege, H.-C.: Visualization in Medicine and Life Sci- ences II - Progress and New Challenges. Springer-Verlag, Berlin, Heidelberg (2012) 12. Marée, R., Rollus, L., Louppe, G., Caubo, O., Rocks, N., Bekaert, S., Cataldo, D., Wehen- kel, L.: A hybrid human-computer approach for large-scale image-based measurements us- ing web services and machine learning. In: Proceedings IEEE International Symposium on Biomedical Imaging. IEEE (2014). 13. Johnson, L., Adams Becker, S., Estrada, V., Freeman, A.: NMC Horizon Report: 2015 Higher Education Edition. The New Media Consortium, Austin, Texas (2015) 14. Dev, P.: Imaging and visualization in medical education. Computer Graphics and Applica- tions 19(3), 21-31 (1999). doi: 10.1109/38.761545 41 Learning to look - purpose and design of an awareness-raising online course - ARTEL15 15. Petersson, H., Sinkvist, D., Wang, C., Smedby, Ö.: Web-based interactive 3D visualization as a tool for improved anatomy learning. Anat Sci Ed 2, 61-68 (2009). doi: 10.1002/ase.76 16. Silén, C., Wirell, S, Kvist, J, Nylander, E, Smedby, O.: Advanced 3D visualization in stu- dent-centred medical education. Med Teach 30(5), 115-24 (2008) 17. Jewitt, C., Kress, G. R.: Multimodal literacy. P. Lang, New York (2003) 18. Unsworth, L.: Teaching multiliteracies across the curriculum. Open University Press, Buckingham-Philadelphia (2001) 19. Schnotz, W., Baadte, C., Johnson, A., Mengelkamp, C.: Knowledge Acquisition from Verbal and Pictorial Information. In: M. J. Lawson, M.J., Kirby, J.R.(eds.) The Quality of Learning. Cambridge University Press, Cambridge (2012) 20. Rohrer, D., Pashler, H.: Recent research on human learning challenges conventional in- structional strategies. Educ. Res. 38, 406-12 (2010) 21. Hattie, J., Yates., G.: Visible learning and the science of how we learn. Routledge, New York (2014) 42 Prompting users to facilitate support needs in collaborative reflection Oliver Blunk, Michael Prilla Ruhr University of Bochum, Germany {oliver.blunk, michael.prilla}@rub.de Abstract: Reflection and collaborative reflection are common ac- tivities at workplace, in which people alone or together with col- leagues revisit experiences to learn for future situations. Often sup- port needs arise during reflection and for example people get stuck and don’t know how to continue. We propose to fill this gap with prompting as tool-supported scaffolding mechanism. Based on liter- ature and our own previous work on reflections support this paper presents a concept using prompting to support different goals based on known support needs in tool-supported collaborative reflection. We show an initial implementation of a prototype and an approach how to evaluate it. This work contributes to the (AR)TEL commu- nity in that it provides a concrete approach for scaffolding in reflec- tion support to be discussed at the workshop and in that it shows how prompting can be used and implemented in tools to evoke re- flection. Keywords: collaborative reflection, prompting, reflection support 1 Introduction Reflection is a common and important activity at workplaces [1]. It includes three stages: returning to past experiences, reassessing them in the context of today’s expe- rience, and then deriving conclusions about how to behave in future [2]. Schön differ- entiated reflection-in-action from reflection-on-action [1], with the former describing reflection about things currently happening and the latter describing reflection about things which happened in the past [1]. However in daily work often problems arise which make collaborative reflection difficult for people and for example sometimes they don’t know how to proceed and thus get stuck while reflecting [3]. Therefore facilitation of reflection is needed. This paper presents a concept using prompting to facilitate different aspects of collabora- tive reflection. The goal is to help users reflect in work place settings and to overcome barriers. After a related work section the concept will be laid out, followed by the implementation of a prototype alongside an approach on how to evaluate the concept. 43 Prompting users to facilitate support needs in collaborative reflection - ARTEL15 2 Related work This section presents work available on reflection (support) at work, including under- lying theories, a reflection model and prompting as the concept used to support reflec- tion in our work. Individual and Collaborative Reflection Reflection is mostly conceptualized as an individual, cognitive activity [1, 2, 4], which creates value by understanding own practice and improving it. There have al- ways been voices on the social side of reflection [5] advocating the common practice of people to reflect together, but only recently research on reflection in groups has gained momentum [6, 7]. In this work the term collaborative reflection describes re- flection in which multiple persons or groups of people are involved. It has been found that such reflection is common practice at many workplaces, and that it has the benefit of producing results which surpass the reflection results of individuals [8]. It should be noted, however, that collaborative reflection always includes individual reflection phases such as applying group results to one’s own situation [9]. Also collaborative reflection needs extra support for communication processes [10] and reflecting to- gether adds complexity to the process of reflection, as multiple perspectives and con- tributions need to be coordinated and aligned [5]. Conceptualizing Collaborative Reflection: Models Supporting (collaborative) reflection needs an understanding and operationalization of reflection processes that enable the development and implementation of support. There are many models available, including the three-stage model by Boud [2] and the cyclic reflection model by Kolb [4], which are used by most work available on reflection. However, these models focus on individual reflection and do not include a perspective on (technology) support for reflection, which make them hard to apply for designing support for reflection in practice. Krogstie, Prilla and Pammer [9] developed a cyclic model that includes both a per- spective on tool support and individual and collaborative reflection aspects (the “Computer Supported Reflective Learning” (CSRL) model shown in Fig. 1). The model contains four different stages, each having a defined input and output and thus describing how the stages feed into each other. In addition it contains triggers for reflection, which represent situations in which the reflection cycle is started. Tools can use this model by connecting to the stages, phases and triggers in order to facili- tate the process and to ensure that required information for each phase is available. Besides the four stages the CSRL model contains several phases, which describe activities that can be helpful in each stage. For example the Initiate Reflection stage contains phases like Set objective and Involve others (no. 2 in Fig. 1), which suggest that it is helpful to think about the reflection session in advance and that tools need to support bringing people together. Furthermore, the model shows that each stage has specific inputs and outputs, which support transitions between the stages. For example 44 Prompting users to facilitate support needs in collaborative reflection - ARTEL15 without data about the work no reflection session can be initiated (“a” in Fig. 1) and without explicit outcomes change is not possible (“c”). Emphasizing the iterative nature of reflection, each stage can spark a new Initiate reflection phase, for example if a session did not result in an outcome and the group needs another reflection ses- sion. Using the model thus can provide support for tool designers to understand the needs of support for reflection processes [9]. In the research presented here it was used for this purpose and serves as a basis for describing goals for reflection support. Fig. 1. The CSRL model by Krogstie, Prilla and Pammer [9] Reflection Support: The need for scaffolding reflection at work The application of reflection support at work faces many difficulties such as time and space available to reflect or continuity in reflection [9, 10]. Available work on dimin- ishing these difficulties and supporting reflection at work can be differentiated into conceptual work describing means and strategies to support reflection, and work re- lated to tools, which describes approaches to use tools to enable reflection. On a conceptual level research often emphasizes the need for facilitation in reflec- tion. Daudelin [5] emphasizes the need for moderation in reflection groups, Zhu [11] describes questions (for further information or to provoke further discussion) as im- portant means to facilitate reflection, and van Woerkom and Croon [12] describe group cultures that include actions such as asking for feedback and questioning groupthink as supportive for reflection. Schön [1] points to the need for more aware- ness for reflection in daily work by suggesting the “reflective practitioner”, and Vince [13] emphasizes the need to establish reflective practice individually and in groups. Concerning tools we found that there are several approaches, which describe dif- ferent levels of tool support for various degrees of reflectivity of users. Tools often allow users to document their problems and challenges by either manually writing 45 Prompting users to facilitate support needs in collaborative reflection - ARTEL15 down what happened [14] or capturing data about daily work [15, 16], which already helps structuring one’s thoughts. More advances tools such as ECHOES by Isaacs et al. [17] remind users to regularly return to such data, reflect on it and implement out- comes of their reflection in practice. Sharing documented issues, which allows people to engage with colleagues in collaborative reflection, is mostly done by sharing the tool used to document experiences with others in face-to-face settings. Online discus- sion forums widely used in other learning contexts [11] and other tools supporting groups [19] have been studied in the context of reflection only on a generic level.. In our work, in which we analyzed the behavior of four different groups using a tool facilitating threaded discussions for experience exchange at workplaces to sup- port reflection [20, 21], we arrived at insights that support and extend the existing work discussed above. In a comparison between more and less effective reflection groups we found facilitation by group members (e.g., asking each other to contribute) and the provision of guidance in the reflection process to be helpful [21]. From an analysis of the content created in the tool we found that providing experiences rather than advice and describing emotions during experiences positively influences the creation of outcomes from reflection [20]. What can be taken away from the existing body of work on reflection support is that there is a need to scaffold (that is, guide, facilitate and structure) the process of reflection in order to enable individuals and groups to learn from and for their work. Prompting as a means to scaffold Reflection As described above there is a need for scaffolding in reflection processes to guide people in individual and collaborative reflection situations. Especially in real work- place situations additional problems arise while reflecting for example due to time restrictions or because workers are not trained in reflection. Among these problems, workers can get stuck or are not sure how to proceed, needing new impulses to con- tinue [3]. Scaffolding can also help to overcome these barriers. There are different means of scaffolding, including scripts [22] that guide users of tools through a step-by-step procedure of learning, ensuring that necessary steps are taken, and open learning environments providing awareness and notification to learn- ers in order to show them possible helpful learning activities. It has been argued in learning research that neither of these ends provides a good solution for all learning processes, and that there is a need to carefully balance guidance and freedom in learn- ing processes [23]. One means of scaffolding that creates this balance is prompting, which has been used in learning environments to facilitate tasks and to stimulate re- flecting about those tasks [24, 25]. Prompts needs to be differentiated from awareness mechanisms and notifications, which often don’t have an instructional character [26]. Prompts are cues designed to stimulate a certain behavior of the recipient [27, 28], but they don’t enforce that behavior [29], leaving the choice whether to react or not to the user. They can be created by oneself (self-prompting), by another person or a system (external-prompting) either randomly, periodically or trigger-based. Usually the person or system creating the prompt (prompter) sends a prompt to the recipient (promptee). Prompts may occur in the form of instructions given to the promptee [25] 46 Prompting users to facilitate support needs in collaborative reflection - ARTEL15 or questions to be answered by the promptee [25], which fits the findings on using questions to stimulate reflection discussed above. From a technical point of view prompts are delivered by a prompting mechanism that may provide different prompts for different situations and purposes. Using a simple example, when setting an alarm clock (prompting mechanism) in the evening, one sets a prompt (the alarm itself) for oneself triggered by a matching time stamp, which most likely has the intention to instruct the user to get up in the morning. To help users with specific tasks it is help- ful to have context-specific rather than abstract prompts [24]. As any scaffolding mechanism may also cause harm if not used in a proper way [23], prompts may also create negative effects. As an example for reminiscence long- er prompts lead to lower response rates (but also elicit longer answers) [30]. They thus have to be designed carefully and the goal designing prompts is to achieve a tradeoff between the intensity of their usage and the effect on the user. Related work has found that prompts are not always answered, which was dubbed negative in an education environment [31], in which prompts were part of the learning assignment. However, given the challenges reflection support faces at work, which often hinder people to reflect at all, we think that even if some prompts remain unan- swered in certain cases, they are still helpful in creating opportunities for and scaf- folding collaborative reflection at work, since we plan to support workers rather than enforcing the way they are using discussions at their workplace. Although prompting has been used in the context of reflection as described above there is no concept for using prompts as scaffolding for various aspects of reflection in general. This paper presents such a concept, in which prompts help to facilitate reflection and assist if users are stuck while reflecting. 3 Concept to facilitate reflection through prompts The following section shows our concept of how prompts can help facilitating (col- laborative) reflection and how to overcome various barriers. For this we derived dif- ferent goals from literature and our own work and connected each of the goals to the CSRL model presented above. We also present examples of prompts supporting those goals. Goals for Reflection Support Stage I: Plan and Do Work In the initial stage of the CSRL cycle the main goal is to provide data to reflect upon. G1 Document surprising experiences in daily work: Reflection is often caused by the discrepancy of reality in contrast how one expected something to happen [1]. Baumer calls this breakdowns, which are reasons to start reflecting [32]. Therefore, complementing means to capture data automatically, there is a need to document ex- periences in order to reflect about them later [10, 14]. Prompts for this goal are in- tended to start reflection and to help people develop a reflective view on their work. 47 Prompting users to facilitate support needs in collaborative reflection - ARTEL15 Stage II: Initiate reflection Reflection may end before it has started, meaning that discrepancies are not followed up or no insights are created. Initiating it includes setting objectives, involving others and supporting descriptions and initial reflections on the experiences shared. G2 Set the objective: Boud et al. argue that reflection is not “and end in itself” and that it is focused towards a goal in order to change future behavior [2]. Therefore setting an objective allows for a more clearly structured reflection session. Prompts may help users to set an individual or group related objective that helps to guide the discussion. This is also part of the input for the next stage in the reflection cycle [9]. G3 Involve others: Involving others into problem-solving offers of getting feedback from others and finding solutions together [8, 10, 12]. Thus in line with recommenda- tion research [33] there is an need to support reflection participants in inviting others such as people someone is frequently getting input from or people the user never worked with before to benefit from new ideas and insights [33]. Prompts can explicit- ly ask for such invitations and may include recommendations. G4 Support individual reflection when describing experiences: Reflection partici- pants should not only describe a problem but also include first explanations and justi- fications. Fleck and Fitzpatrick [15] as well as Hatton and Smith [34] describe this as two different levels with the first one as just descriptive but not being reflective. In order to move away from that level, prompts may guide people into reflection while starting a new discussion thread by asking to think of explanations and interpretations. G5 Elicit problem description: Prompts can also help eliciting the problem descrip- tion by asking for a more detailed description. Alternative explanations and different viewpoints are necessary to reach a form of dialogic reflection [15]. Additionally there might be also the need for prompts to limit the length of posts, since very long posts are often not read fully by others in contrast to very short posts which can’t transport all required information [35]. This goal aims to help setting up the subse- quent stage in the reflection cycle through trying to make sure that enough infor- mation is present. Stage III: Conduct reflection session A major aim of the reflection session is to create outcomes that people may use to improve their practices. This can be done in different ways: G6 Get people to explicitly link to their experiences rather than giving plain ad- vice: As described above we found that sharing experiences leads to an increased likelihood to get reflective outcomes in tool supported collaborative reflection [20]. Prompts may support this goal by asking people to argue from experience instead of giving just plain non-reflected advice. This also connects to the phase of making re- lated experiences available in this stage of the CSRL model [9]. G7 Single Loop Learning & G8 Double Loop Learning [36]: Both goals support learning while reflecting [1, 2]. This may occur either on the level of learning for particular tasks or problems (single loop learning, see [36]) or by deriving more gen- eral insights in the nature and prerequisites of work (double loop learning). Prompts may guide reflection participants to either of these outcomes by asking for solutions to the particular problem or by focusing on supporting people to state what they can 48 Prompting users to facilitate support needs in collaborative reflection - ARTEL15 learn in general respectively how the knowledge can be applied to different problems. This is also helpful for other people who are participating in the discussion thread. Stage IV: Apply outcome Most work on reflection does not include the implementation of insights. However, in order to provide effective support this stage is equally important as the other stages. G9 Check whether stage has been reached: In order to proceed to the application of outcomes at least one outcome needs to be mentioned in the reflection session. Prompts may aim towards helping the author/starter of a discussion to identify out- comes or to start a new reflection cycle to e.g. pursue another direction in the discus- sion or discuss some point in detail. This and G10 are similarly supported by literature as G2 with these goals being more focusing on realizing the objective of the session. G10 Plan application of reflection outcome: This goal supports user in transforming ideas from the reflection session into a detailed plan of how to implement the change. This way the outcome might get more concrete. This can help people engaging into multiple reflection cycles with describing a problem first, then reporting back insights and they have learned after that (see G11). G11 Check application of outcomes of a previous reflection cycle: To support the implementation of change users can be reminded to regularly check their progress in applying outcomes from reflection [17]. Prompts may regularly ask users to self- assess their progress or ask them to revisit their goals. Stage-independent (cross-cutting) goals There are some goals that can be applied to multiple or even all stages of the CSRL model, including training on good reflection practice, support for self-efficacy and the provision of an environment in which reflection works. G12 Train people how reflection works best: Reflection, though seemingly intui- tive, is not easy but has to be learned. Explain to reflection participants how reflection processes work in order to enable them to proceed with reflection is common in some disciplines such as nursing, in which reflective practice is well established and studies explain that being reflective has to be trained [37]. Prompts may enable users to un- derstand the basic concepts of reflection (based on the work of Schön [1] and Boud [2]) and they may provide concrete instructions how to proceed in order to show how collaborative reflection works (e.g., based on the CSRL model [9] discussed above). Though one challenge might be providing a way to deliver adequate personalized feedback regarding learner’s success. Still the means of prompting itself has been used successfully in teaching and education to stimulate users elaborate their answers or to stimulate self-reflection [24, 25]. G13 Support people to feel safe in the environment: In discussion settings when people report of their challenges the fear of being judged or criticized can hinder peo- ple from posting [38]. Prompts for this goal include hints that one could post some- thing anonymously thus avoiding direct criticism which seem to work in education settings [39], or try to remind people to treat each other fairly [40]. 49 Prompting users to facilitate support needs in collaborative reflection - ARTEL15 A prompting concept to implement the goals for reflection support Table 1 shows on overview how the goals above can be implemented by concrete prompts. Each goal is connected to one or more example prompts. Goal Example prompts G1: Document surpris- What did surprise you in your daily work lately? Why? ing experiences in Have a look at your calendar of the last week. What was daily work difficult for you? How? G2: Set the objective What is it you want to know from your colleagues? Ask them a specific question! G3: Involve others Who of your colleagues might help you in this? Mention him in your post to notify him or her (@username). G4: Support individual What has happened? Is there already something you reflection already in learned from it? first post G5: Elicit problem Help your colleagues understand your problem: Try to description describe what things you already tried to solve your issue! G6: Explicitly link to Suggestions are most helpful if they are based on your experiences rather than experience! giving plain advice What would you personally suggest as a solution? Why? G7: Single Loop What have you learned regarding this topic so far from this Learning discussion? What is your personal outcome of this discussion so far? G8: Double Loop What have you learned so far on an abstract level? learning G9: Check whether Do you have an idea from this discussion how to change stage has been reached your approach to the topic? If not, phrase a new question. G10: Plan application How do you want to implement the suggestions of your of reflection outcome colleagues? G11: Check application Did your plan work? What are your experiences with the of outcomes of a previ- change? Tell your colleagues about it. ous reflection cycle G12: Train people how What happened in that situation? How do you think about reflection works best it now? How do you plan to handle similar situation in future? G13: Support people to You can also use the post-anonymously button if you think feel save in the envi- that you are judged. ronment Don’t only directly criticize your colleagues. Show them also what he/she did correctly Table 1. Example prompts for each goal supporting collaborative reflection State of this work and possible additions As can be seen from the goals and corresponding prompts, currently the concept is focused on common problems concerning reflection, and it covers the whole cycle of 50 Prompting users to facilitate support needs in collaborative reflection - ARTEL15 reflection as described in [9]. While the goals as well as the prompts may be extend- ed, to the knowledge of the authors this is the only concept that shows how prompting may influence reflection in such as broad way. However, there are several possibili- ties in extending the concept to pursue facilitate other reflection related aspects: Linking to creativity support: The concept may be complemented by creativity techniques in order to help users getting new ideas about how to approach their work related challenges and how to merge ideas into solutions. One creativity technique to support this might be directed brainstorming [41], in which a topic is split up into individual parts having a brainstorming each. For the reflection participants this adds the benefit of choosing from a number of ideas, which is easier than creating new ones [42]. Santanen et al. also used the means of prompting in their study of directed brainstorming [41], thus linking this to reflection might be promising, but further research is required to study how the concept of directed brainstorming through prompts is also helpful in reflection settings. Setting goals: Setting personal goals is beneficial for personal reflection since people can use this to plan reflection topics or sessions [43]. This is also an opportuni- ty to facilitate reflection through the means of prompting. Prompts could help users in reminding them of looking at their goals to reassess whether the goal has already been met, whether the goal is still worth pursuing or whether the goal is now obsolete. Also prompts can assist users in checking whether discussion contributions are helpful to progress towards their goal. Sustaining reflectivity: Often people who are more experienced in their rob (role) are less likely to have a reflective approach to their work [1, 44]. Prompts may evoke and sustain their reflexivity by helping these people to take a role similar to a mentor in which they can contribute with their experience on topics initiated by colleagues. Mentorships often have various benefits also for the mentor like learning from the protégé, getting new work related information, and extending the network [45]. The approach is to deliver prompts which are not asking the experienced worker directly to think about their work, but to try get their feedback for the work of their mentees, which in turn might possible cause them to think about their work. 4 Implementation and Evaluation: Work in Progress This section reports on the implementation of a prototype incorporating the aforemen- tioned prompting concept, including an approach for its evaluation. Implementation To evaluate the concept we are currently implementing a system supporting reflection in online discussion threads as used in typical tools such as community platforms and learning environments. This enables a group of people working together to talk to- gether about the problems they face in an asynchronous way, not requiring them to find a common time and date [10] and to exchange experiences with colleagues work- ing in the same organization but in different offices. Using prompts might then facili- 51 Prompting users to facilitate support needs in collaborative reflection - ARTEL15 tate collaborative reflection and discussion among the participants in the group. This is especially helpful if used in existing platforms in an organization, as it adds reflec- tion to the platform while preserving a well-known and accepted tool for users1. Fig. 2. Prototype to evaluate the prompting concept The prototype currently features threaded discussions as well as social networking features like profiles, personal messages, friendships, mentioning. It also contains the option to share files and a common news section. Features like mentioning facilitate involving others (G3), making adhering to that type of prompts more simple. Having friendships allows people to expand their network to get automatically get notified of discussions their colleagues participated in. Fig. 2 depicts a discussion thread in the prototype, showing that a thread was al- ready started and that the user viewing the thread has now the possibility to reply. Just below the thread and above the text input area the prompt is shown asking the recipi- ents to think back whether they also experienced something like this and to tell their experiences (G6). This also shows that the prompts are implemented in a way not forcing the recipients to react in any specific way. The prompts will be later on ac- companied by a button containing information that the prompt is trying to help them structuring their posts and thus showing a trajectory of how they can make use of it. Ideas for the Evaluation of the Prompting Approach The concept described in this paper will be evaluated with groups of 10-100 users as part of the European project EmployID working as counsellors in public employment services. The platform is intended to support the exchange of experiences among practitioners and offering an environment for reflection. While the evaluation will 1 The mechanism for adding the prompts to existing platforms will be described in the ECTEL 2015 poster session, see [46], 52 Prompting users to facilitate support needs in collaborative reflection - ARTEL15 begin late in 2015 we already compiled a list of criteria we plan to use to evaluate whether and to what extend the concept is facilitating collaborative reflection: Do the prompts change the behavior of the participants? If prompts are suc- cessful we should be able to see more activity related to reflection (e.g., more ex- periences shared or questions asked). Does more reflection occur in the discussion threads? If the prompts facilitate reflection we should be able to see qualitative and quantitative changes in the re- flection outcomes (e.g., more outcomes documented and more satisfaction with the outcomes among participants). Which prompts help facilitating reflection the most? While our concept is based on thorough literature analysis and previous work it is likely that some prompts are more or less helpful than others. Comparing the outcomes of the two questions mentioned above with the prompt being used may therefore create val- uable insights on how to prompt for reflection. Are there any long-term learning effects following the display of prompts? Prompts should not only guide activity but also help people to become more re- flective. If this is successful we should be able to see the changes measures in the first two questions even if we reduce the amount of prompts provided to users. This is also related to using prompts as a scaffold which is reduced bit by bit over time when the learner doesn’t need the scaffold anymore. In order to answer the questions an evaluation setting with a within-groups design is planned, which includes having two groups in a counter-balanced design. This way one group gets prompts for a certain time and no prompts after that, and the other group starts without getting prompts and then switches to getting prompts. This de- sign has the advantage that the size of the group of participants can be smaller since everybody gets prompts at one point of time. A disadvantage might be that having prompts at first influences the behavior in a long-term way, affecting the time period when said group does not receive prompts anymore. However, this disadvantage could be also used to answer the last question regarding long-term effects of prompts shown to users. To answer the questions we plan to evaluate the concept on two different levels. The first level is whether the prompts are influencing the content of the discussion, that is, whether the users make use of the suggestions, for example whether displaying a prompt asking to evaluate colleagues did leads to more users mentioned in the posts. The second level involves checking whether those interventions also lead to more reflection outcomes in terms of whether the post indicated that the user either learned something or indicated that the user intends to do something differently in future. Prompts and reactions to it could be evaluated through a content analysis as demonstrated in [20]. This may give insights whether the content of written discus- sions may be influenced by the prompt or not. Prilla et al. recently published a coding scheme for analyzing reflection content [20], which allows for checking whether the content shows indication that the users learned something or intends to do something differently in future. Content analysis is also an evaluation strategy previously em- ployed to analyze prompts [24, 47]. Methods like pre- and post-tests on the content like in education settings [25] may not be feasible in workplace settings. Answering 53 Prompting users to facilitate support needs in collaborative reflection - ARTEL15 the question whether a user actually saw and respectively read the prompt could be either answered statistically [25] or in pre-tests technology-supported e.g. with eye- tracking. However eye-tracking doesn’t seem feasible in a larger workplace setting. Further work has to be done to make sure that this case is considered in the evalua- tion. Prompts trying to evoke a more elaborated answer could also be evaluated on a quantitative level. Measuring the amount of words in the corresponding discussion contributions could give a basic insight, whether people adhered to the prompt. Addi- tionally content coding can assess whether contributions contain more descriptive content about problems or approaches the author already tried to solve the problem. Prompts aiming to guide users can be also evaluated through content analysis by checking whether the text contains elements the prompt asked for. In order to evaluate whether prompts targeting the involvement of colleagues are successful, a social net- work analysis may be useful to track changes in the personal network of people re- flecting together in online threaded discussions. Prompts suggesting that content could be posted anonymously can also be evaluated without content analysis but only with the information how often the prompt was displayed and how often the feature was used respectively how often the feature was used after the user has seen the prompt once. 5 Conclusion and Outlook We presented a concept based on using prompts to facilitate tool supported collabora- tive reflection and to support certain reflective behavior. We based the approach on literature analysis and our own previous studies, and we chose this approach because we think it is most suitable to cover a broad bandwidth of support for reflection. The main contribution of this paper is the presented concept of goals supported by prompting to facilitate collaborative reflection – to the knowledge of the authors there is no other concept linking prompts to reflection support in such an extent. Addition- ally the paper already shows a first implementation of a prototype outlining how the concept can be used in a setting with online thread discussion forums. Our work is still in progress, and we think it provides fertile grounds for discussion in the ARTEL workshop. In future work we plan to evaluate the concept with the presented prototype in a workplace setting. We already presented a rough evaluation concept in the paper which shows how we plan to evaluate whether and how different prompts are feasible for collaborative reflection support in a workplace setting. Acknowledgements This work is part of the EmployID (http://employid.eu) project on “Scalable & cost- effective facilitation of professional identity transformation in public employment services” supported by the EC in FP 7 (project no. 619619). We thank all colleagues and associated partners for their cooperation and our fruitful discussions. 54 Prompting users to facilitate support needs in collaborative reflection - ARTEL15 References 1. Schön, D.A.: The Reflective Practitioner: How Professionals Think in Action. Basic Books (1983). 2. Boud, D., Keogh, R., Walker, D.: Reflection: Turning Experience Into Learning. Routledge (1985). 3. Prilla, M.: Collaborative Reflection Support at Work: A Socio-Technical Design Task. ECIS 2014 Proceedings. (2014). 4. Kolb, D.A.: Experiential learning: Experience as the source of learning and de- velopment. Prentice-Hall Englewood Cliffs, New Jersey (1984). 5. Daudelin, M.W.: Learning from experience through reflection. Organizational Dynamics. 24, 36–48 (1996). 6. Baumer, E.P., Khovanskaya, V., Matthews, M., Reynolds, L., Sosik, V.S., Gay, G.K.: Reviewing Reflection: On the Use of Reflection in Interactive System De- sign. In: Proceedings of DIS 2014 (2014). 7. Marcu, G., Dey, A.K., Kiesler, S.: Designing for Collaborative Reflection. In: Proceedings of PervasiveHealth’14 (2014). 8. Hoyrup, S.: Reflection as a core process in organisational learning. Journal of Workplace Learning. 16, 442–454 (2004). 9. Krogstie, B.R., Prilla, M., Pammer, V.: Understanding and Supporting Reflective Learning Processes in the Workplace: The CSRL Model. In: Hernández-Leo, D., Ley, T., Klamma, R., and Harrer, A. (eds.) Scaling up Learning for Sustained Impact. pp. 151–164. Springer Berlin Heidelberg (2013). 10. Prilla, M., Pammer, V., Krogstie, B.: Fostering Collaborative Redesign of Work Practice: Challenges for Tools Supporting Reflection at Work. In: Proceedings of the European Conference on Computer Supported Cooperative Work (ECSCW 2013) (2013). 11. Zhu, E.: Meaning Negotiation, Knowledge Construction, and Mentoring in a Distance Learning Course. (1996). 12. Van Woerkom, M., Croon, M.: Operationalising critically reflective work behav- iour. Personnel Review. 37, 317–331 (2008). 13. Vince, R.: Organizing reflection. Management Learning. 33, 63–78 (2002). 14. Scott, S.G.: Enhancing Reflection Skills Through Learning Portfolios: An Empir- ical Test. Journal of Management Education. 34, 430–457 (2010). 15. Fleck, R., Fitzpatrick, G.: Reflecting on Reflection: Framing a Design Land- scape. In: Proceedings of the 22Nd Conference of the Computer-Human Interac- tion Special Interest Group of Australia on Computer-Human Interaction. pp. 216–223. ACM, New York, NY, USA (2010). 16. Pammer, V., Knipfer, K., Krogstie, B., Wessel, D., Prilla, M., Lindstaedt, S.: Reflective Learning at Work - A Position and Discussion Paper. In: Proceedings of 1st European Workshop on Awareness and Reflection in Learning Networks (ARNets11) at EC-℡ 2011 (2011). 17. Isaacs, E., Konrad, A., Walendowski, A., Lennig, T., Hollis, V., Whittaker, S.: Echoes from the past: how technology mediated reflection improves well-being. 55 Prompting users to facilitate support needs in collaborative reflection - ARTEL15 In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 1071–1080. ACM, Paris, France (2013). 18. Prilla, M.: User and Group Behavior in Computer Support for Collaborative Reflection in Practice: An Explorative Data Analysis. In: Rossitto, C., Ciolfi, L., Martin, D., and Conein, B. (eds.) COOP 2014 - Proceedings of the 11th Interna- tional Conference on the Design of Cooperative Systems. Springer (2014). 19. Lee, S.: Design and analysis of reflection-supporting tools in computer-supported collaborative learning. International Journal of Instructional Technology and Dis- tance Learning. 2, 49–56 (2005). 20. Prilla, M., Nolte, A., Blunk, O., Liedtke, D., Renner, B.: Analyzing Collaborative Reflection Support: A Content Analysis Approach. In: Proceedings of the Euro- pean Conference on Computer Supported Cooperative Work (ECSCW 2015) (2015). 21. Prilla, M.: Supporting Collaborative Reflection at Work: A socio-technical Anal- ysis. AIS Transactions of Human-Computer Interaction. (2015). 22. Kobbe, L., Weinberger, A., Dillenbourg, P., Harrer, A., Hämäläinen, R., Häk- kinen, P., Fischer, F.: Specifying computer-supported collaboration scripts. In- ternational Journal of Computer-Supported Collaborative Learning. 2, 211–224 (2007). 23. Dillenbourg, P., Järvelä, S., Fischer, F.: The evolution of research on computer- supported collaborative learning. In: Technology-enhanced learning. pp. 3–19. Springer (2009). 24. Davis, E.A., Linn, M.C.: Scaffolding students’ knowledge integration: prompts for reflection in KIE. International Journal of Science Education. 22, 819–837 (2000). 25. Thillmann, H., Künsting, J., Wirth, J., Leutner, D.: Is it Merely a Question of “What” to Prompt or Also “When” to Prompt?: The Role of Point of Presentation Time of Prompts in Self-Regulated Learning. Zeitschrift für Pädagogische Psy- chologie. 23, 105–115 (2009). 26. Palanque, P., Winckler, M., Martinie, C.: A Formal Model-Based Approach for Designing Interruptions-Tolerant Advanced User Interfaces. In: Hussmann, H., Meixner, G., and Zuehlke, D. (eds.) Model-Driven Development of Advanced User Interfaces. pp. 143–169. Springer Berlin Heidelberg (2011). 27. Stichter, J.P., Lewis, T.J., Whittaker, T.A., Richter, M., Johnson, N.W., Trussell, R.P.: Assessing Teacher Use of Opportunities to Respond and Effective Class- room Management Strategies Comparisons Among High- and Low-Risk Ele- mentary Schools. Journal of Positive Behavior Interventions. 11, 68–81 (2009). 28. McKenzie-Mohr, D.: Fostering sustainable behavior through community-based social marketing. American Psychologist. 55, 531–537 (2000). 29. Fivush, R., Gray, J.T., Fromhoff, F.A.: Two-year-old talk about the past. Cogni- tive Development. 2, 393–409 (1987). 30. Peesapati, S.T., Schwanda, V., Schultz, J., Lepage, M., Jeong, S., Cosley, D.: Pensieve: Supporting Everyday Reminiscence. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 2027–2036. ACM, New York, NY, USA (2010). 56 Prompting users to facilitate support needs in collaborative reflection - ARTEL15 31. Greene, B.A., Land, S.M.: A Qualitative Analysis of Scaffolding Use in a Re- source-Based Learning Environment Involving the World Wide Web. Journal of Educational Computing Research. 23, 151–179 (2000). 32. Baumer, E.P.: Reflective Informatics: Conceptual Dimensions for Designing Technologies of Reflection. In: Proceedings of CHI 2015 (2015). 33. Manouselis, N., Drachsler, H., Verbert, K., Duval, E.: Recommender Systems for Learning. Springer Science & Business Media (2012). 34. Hatton, N., Smith, D.: Facilitating Reflection: Issues and Research. (1994). 35. Wise, A.F., Zhao, Y., Hausknecht, S.N.: Learning Analytics for Online Discus- sions: A Pedagogical Model for Intervention with Embedded and Extracted Ana- lytics. In: Proceedings of the Third International Conference on Learning Analyt- ics and Knowledge. pp. 48–56. ACM, New York, NY, USA (2013). 36. Argyris, C., Schön, D.A.: Organizational learning II: Theory, method, and prac- tice. Addison-Wesley (1996). 37. Ruth-Sahd, L.A.: Reflective practice: a critical analysis of data-based studies and implications for nursing education. J Nurs Educ. 42, 488–497 (2003). 38. Platzer, H., Blake, D., Ashford, D.: Barriers to learning from reflection: a study of the use of groupwork with post-registration nurses. Journal of Advanced Nurs- ing. 31, 1001–1008 (2000). 39. Tan, N.Y.L., Chen, W., Looi, C.-K.: GroupScribbles as a rapid CSCL tool: Learning experiences of pre-service teachers. Presented at the (2009). 40. Kottkamp, R.B.: Means for Facilitating Reflection. Education and Urban Society. 22, 182–203 (1990). 41. Santanen, E.L., Briggs, R.O., de Vreede, G.: The cognitive network model of creativity: a new causal model of creativity and a new brainstorming technique. In: Proceedings of the 33rd Annual Hawaii International Conference on System Sciences, 2000. p. 10 pp. vol.1– (2000). 42. MacCrimmon, K.R., Wagner, C.: Stimulating Ideas through Creative Software. Management Science. 40, 1514–1532 (1994). 43. McAlpine, L., Weston, C., Beauchamp, C., Wiseman, C., Beauchamp, J.: Build- ing a metacognitive model of reflection. Higher Education. 37, 105–131 (1999). 44. Mamede, S., Schmidt, H.G.: Correlates of Reflective Practice in Medicine. Adv Health Sci Educ Theory Pract. 10, 327–337 (2005). 45. Ragins, B.R., Scandura, T.A.: Burden or blessing? Expected costs and benefits of being a mentor. Journal of Organizational Behavior. 20, 493–509 (1999). 46. Prilla, M., Blunk, O.: Reflective ℡: Augmenting Learning Tools with Reflection Support. In: Proceedings of the Tenth European Conference on Technology En- hanced Learning. Springer (2015). 47. Rosenshine, B., Meister, C., Chapman, S.: Teaching Students to Generate Ques- tions: A Review of the Intervention Studies. REVIEW OF EDUCATIONAL RESEARCH. 66, 181–221 (1996). 57 58 A course concept for enhancing reflective learning - bringing research project results into the field Nils Faltin, Margret Jung IMC information multimedia communication AG, Saarbrücken, Germany {nils.faltin, margret.jung}@im-c.de Abstract. After a successful evaluation at the IMC in the project MIRROR to enhance reflective learning by coaching and usage of two apps developed in the project, the instructional design was further developed. Below three phases are described that show the development from a research prototype to a commercial blended learning course. The first phase was a summative evaluation. Ten staff members took part in a time management coaching at our company IMC. In the second phase a time management open online-course was created. There the par- ticipants were offered to use two apps from the MIRROR project and tele-coach- ing for a fee. In the third phase a blended learning course was created and evalu- ated with three human resource managers. It included the two mentioned apps and individual coaching sessions and group sessions. The human resource man- agers were satisfied with the concept. The course was then included in the inter- nal training catalogue and is currently running. Keywords: survey· time management mooc· time management course· blended learning· Activity Recommendation App – ARA· computer activity tracking tool· soft skills improvement· experiences· personal coaching· group sessions· new concept· reflective learning· survey· project MIRROR· trigger· implemen- tation of reflection outcomes· Manic Time· KnowSelf 1 Introduction Reflection on work experiences can lead to new insights and ideas how to handle work situations better in the future. But the capturing of experiences during work and the reflection on this data is only a part of a more comprehensive offering needed for a successful improvement. The other part is the creation of a viable reflection outcome and the validation of this outcome when it is applied in practice (see [1]). Based on this validation, a change can be approved, reverted, or improved and validated again. Whilst several applications support users to capture data during work in order to pro- vide it in a subsequent reflection session, the later parts of the reflection cycle is often left unsupported. To also cover this part, we created a concept to support people to learn how to improve by reflective learning. One part of this concept is the Activity Recom- mendation App (ARA) that was created in the MIRROR project (see [2]). The app al- lows capturing personal experiences relating to recommendations defined in individual 59 A course concept for enhancing reflective learning - ARTEL15 or collaborative reflection sessions and viewing other members’ experiences if a rec- ommendation targets a team. ARA supports the evaluation of a recommendations’ use- fulness when applied in practice, in order to enable its improvement and support the decision to declare it as learned or to suspense its use. Another part of the concept is a computer activity tracking app that records the com- puter usage in a time management scenario. While analysing the recorded data, reflec- tion is triggered and the user is supported in reviewing the personal learning progress. These two apps had the aim to support reflective learning in our different concepts of time management scenarios. But another part was also very important. We wanted to support the participants in using the apps, in reflecting on recorded data and in imple- menting time management rules. 2 First phase A summative evaluation took place at our company IMC over a period of six weeks (see [3]). Ten staff members took part in a time management coaching. The approach combined the usage of a computer activity tracking tool and the Activity Recommen- dation App with a weekly coaching session. Five participants used the time tracking tool KnowSelf that was created in the MIRROR project and the other five staff mem- bers used the commercial tool ManicTime (see [4]). In the weekly coaching sessions the coach and the coachee reviewed the individual progress, adjusted the time management rules if not appropriate anymore, or, when the particular goal has been achieved and the new behavior has been adopted, decided that no further practice regarding that goal is needed. ARA served well as a support for learning time management by providing a data basis for the coaching sessions. It was used by coach and coachee to set time management goals and to document and monitor the progress in learning new time management techniques. Both benefited from the better preparation for the coaching sessions available with the notes in ARA. Further- more, the app helped the coachees to focus their goals. After the evaluation, the coach and several coachees suggested to improve the instruc- tional design of future trainings by forming peer groups to train time management tech- niques. This would allow participants to benefit from sharing experience data to com- pare own progress with that of others and learn from each other’s experiences. The study showed that coaching is a good instructional setting to support reflective learning. The coaching approach has increased the reflection in the workplace and coaching serves as a trigger for reflection. People need support in reflective learning. Technical applications as tracking apps or ARA are important, but even here the people often need a guidance of a coach. 3 Second phase – time management MOOC Based on the insights from the MIRROR project study, IMC has created a free online course for time management that includes usage of the ARA and KnowSelf (see [5]). 60 A course concept for enhancing reflective learning - ARTEL15 The MOOC includes videos, scripts, self-tests, screencasts and examples of time man- agement rules that learners can create in ARA. In addition to these free elements, learn- ers can opt for a human tele-coach who can be booked for a fee. The purpose and use of ARA and KnowSelf is explained in the course with an example scenario. It shows how an office worker uses both applications to improve his time management. In addition, the usage of both applications is explained in several short screencasts. This MOOC attracted 1000 registered participants in just three months and has now, after one year reached 1600 registered participants. It turned out there was very little usage of the ARA and KnowSelf (measured by down- loads and app registrations). A reason could be that the added value is not visible for learners and their personal use. Another reason could be the complexity of the apps and the related effort at first sight. In addition, there were only very few requests for tele-coaching. A reason could be that participants of a free MOOC course do not expect to pay for additional services and the cost of coaching is relatively high for a personal budget. As it is difficult to study MOOC usage in detail and to watch interview individual MOOC participants a new offer was created that is easier to evaluate and optimize. This resulted in a third phase. 4 Third phase – blended learning format 4.1 The concept Participants of the first study had suggested to form peer groups that work on a common issue. To provide learners with an opportunity to exchange experiences on similar prob- lems we formed a group and a regular meeting. We carried out a time management course with three human resources managers of two companies who had a dual role: participants of the course and at the same time evaluators of our new concept. They should find out if this type of approach is a possible and good program for employees of their company. That should help us to gain further insights into improving our ap- proach. The concept was a blended learning approach with coaching to enhance reflective learn- ing and to promote a successful transfer of theory into practice. The course consisted of three parts: online course, individual coaching sessions and group sessions. Part of the concept was also ARA and Manic Time. We chose ManicTime because it received better evaluations in comparison to KnowSelf in the first phase (see [3]). All these parts are intended as a reflection trigger, they shall support the implementation of solutions and finally support the participants in various ways during reflective learn- ing. 4.2 Reflection triggers and reflective learning supports Online-Course. The time management learning content is only opened for the current participants. There are topic sections that are released week after week. The course 61 A course concept for enhancing reflective learning - ARTEL15 includes videos, scripts, self-tests, screencasts, apps, exercises and summaries of the group sessions. The participants are able to discuss directly in the portal. Working with the learning content, the participants begin to reflect on their own situa- tion and their personal time management. The idea is that they have already thought about the content before they meet the coach or other participants. Individual and group sessions. In the weekly one-hour individual coaching sessions the current learning content is discussed, reflected and aligned with the individual situ- ation of the participant together with the coach. They analyze data from the tracking app and discuss the notes in ARA. The coach asks why the recommendation has a cer- tain rating and how it can be improved. In the weekly one-hour group sessions the participants have the possibility to exchange experiences and to discuss the learning content. They can further discuss exercises in the learning portal as well as the ARA recommendations. Reflection is triggered here by discussions. The participants can propose solutions to each other and motivate each other to follow their goals. ManicTime and Activity Recommendation App. ManicTime can trigger reflection when the user starts to analyze the recorded data. Problems – e.g. spending too much time for unnecessary tasks at work – can be defined. It also supports the user in ob- serving and controlling solution implementation and behavior changes. ARA supports reflective learning by enabling the participants to define problems and solutions. Such solutions are recommendations that the participants have to evaluate for a certain period of time. Step by step they change their behavior by following the recommendations. 4.3 Results In the last group session the coach wanted the participants to discuss their opinions and experiences. The Coach asked them some questions about how they like the course concept, about the named and reported risks and opportunities at the beginning of the course and about their priorities of the course parts. The statements were reported by the coach. All in all the human resource managers were very satisfied with the course format. They liked the blended learning approach. The advantage is that a participant learns by watching videos, reading the scripts and doing homework. In this way, he has already collected experiences before having an individual coaching session. So he can discuss problems, ideas and goals concerning the new topic section. The concept was created with several parts and the intention that all these parts trigger reflection and support reflective learning. We expected that the participants could learn independently while being supported by every part of the concept. After the 12 weeks with the three human resource managers we discussed the results in a group session. They evaluated the concept in their dual role: as participants and then as human re- source managers. They provided improvement proposals, said they were convinced of 62 A course concept for enhancing reflective learning - ARTEL15 the potential of this instructional approach and asked to provide this as a regular course for the employees. Risks and opportunities. One of the named risks was work overload (by course tasks adding to the work load of regular work tasks). After the course the group agreed that this became a problem in the Christmas season. This is in line with our findings from the first phase: When participants are very busy, they want to see benefits as soon as possible. But changes in time management practice based on reflective learning need time to unfold and pay off. That is why participants often become unsatisfied during busy periods although this is exactly the time in which the most participants would like to see improvement. Other risks like breach of trust (confidential information leaking out of the group) or frustration (because of reflection leading to realizing problems that cannot be solved) did luckily not materialize. The hoped-for transfer support of theoretical content into practice took place. The par- ticipants confirmed that compared with other conventional trainings this concept really helped them to learn better. The new aspect was that they could combine theory and personal situations and practical experiences. They had time to practice exercises, make experiences and discuss about individual problems and solutions in this course. Behav- ior changes took place too. This can be seen as the major advantage in effectiveness of supported reflective learning in comparison to conventional trainings. Furthermore, the awareness of personal time management was increased and the par- ticipants optimized their work behavior. They even confirmed a reduction of stress. Priorities. Finally, in the group discussion the participants were asked to prioritize the concept parts regarding their personal improvement (see Fig. 1). Here, 1 is the highest and 5 is the lowest priority. 6 5 4 5 5 5 3 4 4 4 2 3 3 3 1 1 1 1 2 2 2 0 Individual Group Online- Time ARA coaching sessions course tracking tool Participant 1 1 2 3 4 5 Participant 2 1 2 3 4 5 Participant 3 1 2 3 5 4 Participant 1 Participant 2 Participant 3 Fig. 1. Concept parts sorted by priority (1 highest, 5 lowest priority) It is clearly evident that the individual coaching was the most important for the partic- ipants. The reason is that they got suggestions for the practical implementation of the tasks in the individual coaching. They also had a healthy pressure to invest time and 63 A course concept for enhancing reflective learning - ARTEL15 energy into their learning as they knew that the coach would observe and ask about personal learning processes. In addition, their personal situation has always been sub- ject and there was space to talk about frustration, but also to find solutions. On the second place are the group sessions. The experience exchange was important for the participants. One participant emphasized, for example, that it helped him to con- tinue with the course content when he has seen how far the other participants are and that they share experiences. A participant complained that it was spoken too much about the course concept in the group sessions. That was because of their second role as con- cept evaluators. In addition, it was noted that the group sessions should take place less frequently, less than one time per week because there was not enough time between individual and group sessions. In this time the participant cannot make many experi- ences. Then they often discussed issues with other participants that were already dis- cussed with the coach a short while ago. The general content of the online course was ranked third. The theory and the exercises contained in the online course provided some basic knowledge. With new methods and time management techniques, the participants were able to improve their time manage- ment. Overall, ManicTime and ARA were regarded as the least important parts. The human resource managers did not see them as essential for their personal progress. Neverthe- less, they saw benefits in using the apps. ManicTime can be used to measure work time spent on certain tasks which can for example be used as basis for discussions with the supervisor about too high work load. This is an example of an organizational level problem rather than a personal time management problem that needs to be solved on the organizational level. ARA was good for documenting experiences and to observe the own learning progress. But as the experiences were discussed in the individual coaching sessions, only the documentation part of reflective learning was supported in this scenario, and ARA had the least benefit in comparison to the other parts of the concept. We created an ARA widget in the course portal to facilitate the use. There the participants were able to di- rectly and quickly document and share experiences, without having to log in separately in ARA. Still the widget was not used by participants as their need for sharing was already covered by the regular group presence sessions. Proposals for future course runs. The participants were satisfied with the course and saw a high potential in it. As human resource managers, they encouraged us to offer the course with some changes in the internal training catalog. With the support of one of the human resource managers we created an offering for employees. The participants also suggested a stronger guidance by the coach in the group sessions. In addition, the group sessions should not only function for experience exchanges, but also as training sessions with short exercises. After these exercises the participants could discuss the results. 64 A course concept for enhancing reflective learning - ARTEL15 5 Current phase Since May the instance of the course as an official training offering of the companies group is ongoing. The concept is similar to the course with the human resource manag- ers. There is an online-course with ARA and ManicTime and individual as well as group sessions. While number of group sessions and coaching sessions remain roughly the same, the course duration was extended from six to twelve weeks to reduce the weekly work load of the participants. At the moment, four employees participate in the course. The participants come from two different companies. The presence sessions take place in one building in Saar- brücken. One participant works in another city far away and therefore he gets tele- coaching (and some extra coaching sessions as he cannot participate in the group ses- sions). He exchanges experiences with the other participants in the course portal, thanks to the ARA widget. This is a 12-weeks-long course again. But now the individual and group sessions take place alternatively every two weeks. Participants can opt for individual coaching ses- sions to take place more frequently and for a shorter duration in consultation with the coach. 6 Outlook We have learned that the concept works. It supports people in the process of reflective learning. All parts of the concept function as triggers for reflection and together they help participants to implement the reflection outcomes into practice. We are interested in a continuing development and monitoring of reflective learning based on this concept. We have experienced that awareness and reflection technology together with personal coaching help employees to change work practice. Therefore, we are currently planning more courses and studies based on this concept. We experienced that apps can serve as reflection triggers and can support people in reflective learning. But most often nontechnical parts, e.g. human support in a time management course has the best evaluation results. Our aim is to increase the focus on the applications in future studies and to find out the reason why this is so. References 1. Krogstie, Birgit, Michael Prilla, and Viktoria Pammer: “Understanding and Supporting Re- flective Learning Processes in the Workplace: The RL@Work Model.” In Proceedings of the Eigth European Conference on Technology Enhanced Learning (EC-℡ 2013), 2013. 2. MIRROR project for reflective learning at work, http://mirror-project.eu/ 3. Bratić, Marina (KNOW): “The IMC Evaluation of the KnowSelf and ARA App (part of the Time Management Coaching Approach)”, Graz (2014) 4. ManicTime: http://www.manictime.com/ 5. MOOC “Time Management“ (Gernan) on OpenCourseWorld, http://bit.ly/1ufnS0v 65 66 �������������������������������������������������� ��������������������������� ����������������������������� � ������������������������������������������������������������������������������������� ������������������������������ �������������������������������������� ������������������������������������������������������������������������������� �������� ������� ���� ��� ������������� ����� ���� ���������� ����� ���� ���� ��� ���� ��������������������������������������������������������������������������� ���������������������������������������������������������������������������� ��� ����������� ��� ��� �������� ����� ���������� ����� ������� ��� ����� ��� �������� ����� ���������� ���� ����������� ���� �������� �� ������������������� ����� ����� �������� ��� ��������� ������ ��������� ������������ �������� ���� ����� ��� ����������� ������� ��� ����������������������������������������������������������������������������� ������������������������������������������������������������������������������ ������������������������������������������������������������������������������ �������� ������� ������������ ��������������������������������������������������� ���������������������������������������������������������������������������������� ��������� ���� ����������� ���� ����� ������� ��� ������� ��������� �������� �������� ���� ���������������������������������� ����������������������������������������������������������� �� �������������� ����������������������������������������������������������������������������������� ������� ��� ��������� ���� �������� ��� ���� ��������������� ��� ����������� �������� ��� ������ ������������������������������������������������������������������������������������ ���������� ������������ ��������� ������� ������� �������� ������� ��������� ����� ����� ���������������� ���������� �������������������������������������������������������������������������� ������������������������������������������������������������������������������������� ���� �������� ���������� ���������� ���� ����������� ������ ���� ������ ������� ������� ������� ������ ������ ������������� ���� ������ ��� ���� ������� ��������� ��� ��� ������� ������������� ��������� ���������� ������� ��������� ��� ����� ��� ���� ���������� ���� ���������� �������� ������������������������������������������������������������ �������������������������������������������������������������������������������� ����� �������� ������������� ����������� ���� ����������� ���������� ��� ����� ��� �� �������� ������������������������������������������������������������������������������������� ������������������������������������������������������������������������������������ ������������������������������������������������������������������������������� 67 Feeler: supporting awareness and reflection about learning through EEG data - ARTEL15 �� ����������������������������������� ����������������������������������������������������������������������������������� ���������� ��� ���� ��������������� ��� ������� ������� �������� ����� ���������� ������ ����������� ������������ ������� ��������� ����������� ���������� ���� �������������� �� ��������� �� ����� ��� ����� ����������� ����� ���������� ����������� ������ ������������� ����������� ���� ����� ����������������������������������������������������������������������������������������� ������������������������������������������������������� � ������ ��� �������� ��� ����� ����������� ����� ���������� ����������� ������ ������������� ����������� � ���� ������ ������ ������ ����� ����������� ������������ ����� ��������� �������� ������������ �� ������ ������� ����� ���������� �� �� ��� ��� ��� � ����� ���������� �� � � �� ��� � ����������� �� � �� � � � ������� �� �� � �� �� �� �������� ���������� �� � �� �� �� �� ��������� �� �� �� �� �� �� ������� �� �� �� � � �� � ����������������������������������������������������������������������������������� ����������������������������������������������������������������������������������������� ���������� ����� ������ ����� �������������� ���� ������� ��������� �������� ����� ���������� �������������������������������������������������������������������������������� ������������������������������������������������������������������������������� ��������������������������������������������������������������������������������� ������������������������������������������������������������������������������������� ����������������������������������������������������������������������������������������� �������������������������������������������������������������������������������������� ��������������������������������������� �������������������������������������������������������������������������������� ������ ���� ��������� �������� ��� ���������� ��������� ����������� ����� ���������� ����� ����� ����� ���� ��� ���� ����� ������������� ���� ���� ����� ������� �������� ��� �� ���������� ���� ���������������������������������������������������������������������������������������� �������� ������������ �������� �������� ���������� ������� ������� ���� ���������� �������� �������������������������������������������������������������������������������������� ���� ����������� ���� ������ ��� ������� ��� ������ ���� ������� ��� ����������� ���� ��������� ������������������������������������������������������������������������������� ��� ����������� ������������ ���������� ������� ���� ���� ����� ����� ���� ����� ��� �������� 68 Feeler: supporting awareness and reflection about learning through EEG data - ARTEL15 ���� ����� ��� �������� ���������� ���� ���������� ��� �������������� ���� ���� ����� ����� ��� ��������������������������������������������������������������������������� ��������������������������������������������������������������������������������� �������������������������������������������������������������������������������������� �������� ���������� ��������� ��� ������ ��� ����� ��������� ��� ����������� ������������ ���� ������� ��������� ������������ ���� ������� ���������� ����� ���� ���� ��� ���������� ����� ������������������������������������������������������������������������������������� ���������������������������������������������� ����������������������������������������������������������������������������������� �������� ��� ������ ��������� ���� ������� ������ ����������� ����������� ����� ��������� ������������������������������������������������������������������������������������� ����������������������������������������������������������������������������� ������������ ��������������������������������������������������������������������������������� ���������������������������������������������������������������������������������� ������� ���������� ��� ��� ������������������ ��������� ������ ����� ������ ��� �������� ������� ���������� ������� ����� ���� ���� ��� ��������� ������������� ����� ��������� ���� ������������ ����� ���������� ��� ���� ������� ���������� ���������� �������� ���������� ���������� ����� ��������� ���� ��� �� ��������� ����� ���� ����������� ���������� ����� ���� ���������� ����� ������������������������������������������������������������������������������������ ������ ����� ������������ ���� ���������� ��� ����� ���������� ����������� ���� ���� ��������� ����������������������������������������������������������������������������������������� ������������������������������������������������������������������� ������������������� ��������������������������������������������������������������������������������������� ���������������������������������������������������������������������������������������� ��������������� ������� ����� �������������� ��������� ������������� ����� ����� ���� ��� ��� ��������� ������� ��� ��������� ��� ������� ���������� ���� ������������ ������� ������� ��� �������������������������������������������������������������������������������������� ��������������� �� ������������������������������������������������������� �������������������������������������������������������������������������������������� ����� ������� �������� ���������� ��� ���� ���������� ������� ���� ������� ������� ����� ��� �������������������������������������������������������������������������������������� ����������������������������������������������������������������������������������������� ���������������������� ������� ��� ������ ������� �� ������� ����� ����������� �� ������ ��������� ��� ������ �������� ���� ����������������������������������������������������������������������������������� ���������������������������������������������������������������������������������� ������������������������ ��������������������������������������������������������������������������������� �������������������������������������������������������������� ����������������������������������������������������������������������������������� ������������������������������������������� 69 Feeler: supporting awareness and reflection about learning through EEG data - ARTEL15 � � ������������������������������������������������������������������������������������������ ����������������������������������������������������������������������������������� ������������������������������������������������������������������������������������� ��������������������������������������������������������������������������������������� ������������������������������������������������������������������������������������� ���������������������������������������������������������������������������������������� �������������������� ������� ���� ���������� ������� ���� �������� ����� ��������� ���� ���������� ��������� ���� �������������������������������������������������������������������������������������� ���������������������������������������������������������������������������������������� ������ ���� ����� �������� ������� ���� ������� ���������� ��������� ������ ������ ����� ��� ���������� ����������� ���� ���� ������ ���������� ����� ������������ ��� ���� ���������� ��� ���� ��������� �� ��������������������������������� ������� ������� ��� �������� ���� ����� ��������������� ������� ������������ ������ ������ ����������� �� ������� ��� ���� ������� ��������� ����������� ��������� �������������� �������� ������������������������������������������������������������������������������������ �������� ���� ������ ������� ���������� ���� ������� ���� ��� ����������� ��� �� ������� �������� ������ ������������������������������������������������������������������������������������ ���������������������������������������������������������������������������������� �������������������������������������������������������������������������������������� ������������������������������������������������������������������������������������ ��������������������������������������������������������������������������� ���������������������������������������������������������������������������������� ��������������������������������������������������������������������������������������� ��������������������������������������������������������������������������������������� ���������������������������������������������������������������������������������� ������� ������������� ����� �������� ��� �������� ���� ����������� ������ ��� ���������� ��� ��������� �������� ����������������� ����� ����� ����� �� ������� ����� ������ ����������� ��� ��������������� ����� ���� ��������� ����� ������������ ���������� ����� ���� ���������� 70 Feeler: supporting awareness and reflection about learning through EEG data - ARTEL15 ������������������������������������������������������������������������������������������� ��������� ������������ ���������� ���� ����� ��������������� ������������� ������� ��������� ��� ��������������������������������������������������������������������������������������� �������������������������������������������������������������������������������������� �������� ���������� ��� ������������ �������� ��������� ���� ��������� ��������� ������� ����� ������������������������������������������������������������������������������������������ �������������������������������� ���� �� ���������� ���������� ���������� ��� ����� ��� ���� �������������� ������� ������ ������������������������������������������������������������������������������������� �������������� ��� ���� ��� �������� ��������� ���� ���������� ������������� ������ ����� ������ ����� ���������� ����� ��������� ��� ������� ����������� ������� �������� ��������� ���������� ��������� ��� �������� ��������������� ���� ������ ����� ��� ������� ���������� ������ ���������������������������������������������������������������������������������������� ����������������������������������������������������������������������� ���������������������������������������������������������������������������������� ������������������������������������������������������������������������������������ ������������������������������������������������������������������������������ ����������������������������������������������������������������������������������� ��������� ��� �� ������� �������� �������������� ��� ������������������ ���� ������������������ ������������ ����� ����� ������������� ��������� ��� �������� ���������� ��� ����������� ���� ��� �������������������������������������������������������������������������������������� ����������������������������������������������������������������������������������� �������� ����������� ������������� ������ ������ ������� ����� ��������� ������������������� ������ ����������� ������ ��� ������� �������� ������� ���������������� ����� ������ ����� ��� ���� ����������� ��� ��������� ������������� ������� ��� ��������� ��� �� ����� ���� ����� ��������������������������������������������������������������� �������� ��������� ������������� ���� ���� �������� ������ ��������� ��� ����������� ������ ������������������������������������������������������������������������������������� ������� ���� ���������� ��� ������ ����������� ������� ��� ������������ ����������� ���� ���� ��� ������ ��� ��������� ������������ ���������� ��� ����� ���� ������������� ��� �������� ������ ��������� ����� ��� ���� ��������� �������� ���������������� ���� �������� ���������� ���� �������������������������������������������������������������������������������������� ���������� ���� �������� ��� ����� ����� ��� ����� ���� ����� ������� ��������� �������������� ��� ������������������������������������������������������������������������������������ ������� ���� ������� ����������� ���� ����������� ����������� ����� ���� ������ ����� �������������� ������������������������������������������������������������������������������������ �������������������������������������������������������������������������������������� ���������������������������������������������������������������������������������������� ��������� ��� �������� ��� ������ ���� ���������� ��� ������� ����� ��������� ��� ���� ������ ��� ���� ���������������������������������������������������������������������������������������� ��������� ����� ���� ���� ��� ������������ ��������� ��� ����� ���� ����� ��� ������ ������ ������ ������������������������������������������������� ��������������������������������������������������������������������������������� ����� ���������� ���� ������������ ������������ ������� ���� ����������� ����������� ���� ���� 71 Feeler: supporting awareness and reflection about learning through EEG data - ARTEL15 ����������������������������������������������������������������������������������� ������ ���� �������������� ��� ��������� ����� ��� ������� ���������� ��� ��������� ��� ������� �� ������� ����� ����� ������������� ������ ����� ����� �� ������ ����� ����������� ������ ����������� ������������������������������������������������������������������������������������� ���������������������������������������������������������������������������������������� ������������������������������������������������������������������������������������������ ���� ������ ��� ��� �������� �������� ��� ������ ��������� ��������� ���� �������� ���� ���������� ���������������������������������������������� ����� ���� ����� ����������� ��� ������������ ����� ������ ���� ���� ����� ��� �������� ������� ������� ���� ���� ����� ������ ����� ����� �������������� ����������� ���� ���� ������� ����� ������ ��� ���� ��������� ���� ��������������� ���� ���������� ��� ����� ��� ���� ����������� ����� ���������� ���� ���� ��� �������� ��������� ���������� �������� ���������� �������������� ���� ��������������������������������������������������������������������������������������� ���� ����� ������� ���� ��������� ��� ����� ��� ���� ������� ����� ���� ����� ���� ��� ����� ����������������������������������������������������������������������������������� ���������������������������������������������������������������������������������������� ������������������������� �������������������������������������������������������������������������������� �������� ���� ������ ������� ����������� ������ ��� ����� ��� ���� �������� ������� ������� ����� ������� ����� ��� ������� ����� ��� ��������� ��� ������ ��������� ���� ����� ����������� ���� ���������������������������� �� ������������ ����������������������������������������������������������������������������������� ������������������������������������������������������������������������������������� ���� ����������� ����� ��������� ����������� ������ ��� ����������� �������������� ����� ������������������������������������������������������������������������������������� ����� ��� ������� ��� ������ ��������� ������������� ����������� ������� ��������� ��������� ������������������������������������������������������������������������������������������ ��� ������ ������� ����������� ���������� ���� ������ ����� �������� ����������� ��� ������ ��� ���������������������������������������� ���������������������������������������������������������������������������������� ������� ������� ���� ��������������� �������� �������� ����� ���� ������� ��������� ��� ������ ������������������������������������������������������������������������������������� ������ ��� ����� ���� ����������� ��� ���� ����� ��� �������������� ����� ���� ����������� ������ ���������������������������������� �� ������������ ��� ������������������������������������������������������������������������������������������ �������������������������������������������������������������������������������������� ������������������������������������������ 72 Feeler: supporting awareness and reflection about learning through EEG data - ARTEL15 ��� �������� ���� ������������ ���������� ����������� ����������� ������ ���� ���� ���������� ��� ���� ����������������������������������������������������������������� ��� ���������������������������������������������������������������������������������������� ���������������������������������������������������������������������������������������� ��� �������������������������������������������������������������������������������� ��� �������� ���� ���������� ����� �������� ������������ ���������� ���� �������������� �������� ����� ����� ���� ��������� �������������� ��������� ��� �������������� �������� ��� ���� ��������� ���������������������������� ��� ������� ���� ������������� ���� ����������� ��� ������������ �������� �� ������� ����������� ���� ���� ��������� ��� ���� ����� ����������� ��� ���� ��������������� ������������ �������� ��������� �������������������������������������������������������������������������� ��� ������������������������������������������������������������������� ��� ��������������������������������������������������������������������������������������� ������������������������������������������� ��� �������������������������������������������������������������������������������������������� ������������������������������������������������������������������������������������������ ������� ���� ��������� ��� ���� �������� ��� ���� ���������� ���������� ����� ����� �������������� ��������� ���� ������� ���������� ������ ���������� ������������ ��� ���� ����� ������� ������� �������������� ������������������������������������������ ���� ���������������������������������������������������������������������������������������������� ������������������������������������������������������������������ ���� �������������������������������������������������������������������������������������� �������������� �������� ������� �� ����� ����� ��� ��������� ���� ������� ������������ �������������� �������������������������������������������������������������� ���� ������ ��� ���� ������������� ���������� ����������� ��� ���� ������� ��� ��������� ���� �������� ��������������������������������������������������� ���� ������������������������������������������������������������������������������������������� ���������������������������������������������������������������������������������� ���� ������������������������������������������������������������������������������������ ���� �������������������������������������������������������������������������������������������� �������������������������������������������������������������������������������������������� ����������������������������������������������������������������� ���� �������� ������ ��� ��������� ��� ������� ��������� ��� ������� ������� ������� ������ ��������� ���� ����������������������������������������������������������������������������������������� ������� ���� ���������������������������������������������������������������������������������������� ����������������������������������������������������������������������������������� ���� ������������������������������������������������������������������������������������������ �������������������������������������������������� ���� ����������������������������������������������������������������������������������������� ����� ����� ������� ������� ������� ��������� ��� ����������� ���������� ����� ������������ ���� �������������������������������������������������������������������� ���� �������������������������������������������������������������������������������������������� �������������������������������������������������������������������������������������������� ����������������������������������������������������������������� ���� �������� ������ ��� ��������� ��� ������� ��������� ��� ������� ������� ������� ������ ��������� ���� ����������������������������������������������������������������������������������������� ������� 73 74 Mood in the City - Data-Driven Reflection on Mood in Relation to Public Spaces Viktoria Pammer1 Knowledge Technologies Institute, Graz Univ. of Technology Know-Center Graz, Austria Abstract. This paper maps out the design space of urban location- related mood self-tracking as starting point for individual and urban re- flection: What are the benefits for key stakeholders, and what are system design options? Data-driven reflection here means that reflection is based on data; in this instance taking mood related to public spaces as start- ing point for reflection, but “attaching” to mood additional data that contains information about context such as comments, tags or pictures. We argue that individual citizens could become aware of own mood and act on this knowledge, e.g., intentionally seeking relaxing or stimulating places. Urban reflection means both discourse on the liveability of public spaces amongst citizens and by stakeholders who manage or decide on the design of public spaces in cities, the most obvious of whom are city government or building project organizers. 1 Mood in the City Reality impacts humans’ affective states; in public spaces, their design as well as the actions and interactions of other people impact the affective states of all who pass through. The relationship of mood and places has been of interest to all sorts of people for a variety of reasons: In “the pursuit of urban happiness”1 , researchers and (city) designers investigate what sort of city design makes peo- ple feel happy and relaxed. To this purpose, plans for building highways were cancelled in Bogota in 1998 and cycle lanes were planned instead2 . Since then, computer scientists have taken an interest and become involved. An initiative called urbangems 3 analysed Google street view images of London with state-of- the art image analysis methods, and used crowdsourcing to rate images in the dimensions beauty, happiness, quietness, deprivation. The authors found that the amount of greenery is the most positively associated visual cue with beauty, happiness and quietness [15]. Similar results are found in a study based on geo- tagged tweets [1]. In [16], the authors used these findings as basis for providing 1 www.researchswinger.net 2 http://www.bbc.com/travel/feature/20130828-reclaiming-the-streets-in-bogata 3 www.urbangems.org 75 Mood in the city - data-driven reflection on mood in relation to public spaces - ARTEL15 directions within a city that are not based on length of route (shortest path rec- ommended) but on emotional pleasantness. As self-tracking technology has be- come available and acceptable to the masses, galvanic skin response trackers are used in the bio mapping 4 initiative to automatically track emotional arousal of study participants in conjunction with their geographic location. The underlying rationale is to “become aware of our own and each others’ unique body reactions to the environment [to] create a better world” (ibid). In addition to these soci- etally motivated works and initiatives on a socially larger scale, mood tracking is also used for more individualistic purposes. A plethora of mood self-tracking apps exist on the web5 in the Quantified Self spirit. Other works have inves- tigated mood tracking more scientifically: The affective diary [17] investigated mood representations and reflections on it throughout the day, emphasising in re- search questions however the automatic capture of mood and its representations. AffectAura [11] explored user reactions to long-term representation of automatic emotion detection. The authors found that a historic representation of affective data does support memory, but is without “cues” (contextual data) insufficient to reconstruct memory; thus mood data cannot be the only data but needs to be connected to contextual information. The authors did not explore reflective learning however. In [12], a system for location-based emotion tagging has been developed, but not evaluated or used by a significant number of users (WiMo). Within WiMo, users can decide to share their mood tags with others via places. All mood entries are sent to a WiMo server. Note that in all the above-described related work, mood tracking is sometimes manual, and sometimes “automatic” via sensors that approximate mood via physiological reactions. Sharing in these apps fulfils as main purpose that of communicating own mood, but not that of reflecting together, or reflecting on mood in relationship to others’ mood. Finally, I myself have been part of a research team that has explored shared mood tracking in the workplace, finding indications for shared mood tracking to improve collaboration in virtual meetings [4] and work performance in call centers [5]. 2 Reflective Learning Reflective learning (which I will use as synonymous with “reflection”) is the pro- cess of critically exploring the past in order to learn for the future (see e.g. [2]). As such, reflective learning means reviewing the past in order to learn for the future. Learning is to be taken broadly: Learning means changing one’s per- spective, one’s perception, one’s knowledge, planning to act differently in the future, or actually doing so (ibid). It is this direction towards the future, which distinguishes reflective learning from rumination or “mere” awareness; although awareness is a precondition for reflective learning. Reflective learning can be understood as a cognitive process as well as a social process [14]. In the first case, it is the individual actor who learns (individual 4 www.biomapping.net 5 For instance: http://www.moodjam.com, http://www.moodscope.com 76 Mood in the city - data-driven reflection on mood in relation to public spaces - ARTEL15 learning), while in the second case it is a social entity that learns via its members negotiating understanding, best practices, or pre-scribed processes (collaborative learning). In organisational contexts, reflection is seen as key driver for learning (see e.g., [6,7]). In such contexts, individual and collaborative learning naturally intertwine (for examples see e.g., [8]); for instance when an individual actor re- alises that something can only be changed at a collaborative level; or when an individual within a discussion reflects on what the discussed change in strategy will mean for own work practice. By data-driven reflection, I mean the concept that reflection can be based to a significant extent on data; going so far as deriving “triggers for reflection” (the direct reason that makes a person or group reflect, see [8]) from data. This concept is taken by different communities, such as learning analytics, quantified self [3] or personal informatics [10]. 3 Contribution: Data-Driven Reflection on Mood in Relation to Public Spaces In this paper, I investigate data-driven reflection on mood in relation to public spaces. This is a novel concept, both for the field of computers and learning, and for the fields of social innovation and urban development. While the concept of data-driven reflection is not new for the first; it is new for the latter. Vice versa, while using something like “bottom-up dialogue” in an urban context is new in the scientific discourse in computer-supported learning, it is a big part of “business as usual” in social innovation and urban development. I frame the task of driving social innovation and urban development bottom-up as a similar one to reflective learning in organisations: There is a mixture and inter-relationship of individual and collaborative reflection processes, and it is different stakeholders and stakeholder groups who can or should learn. I concep- tualise district communities or cities as social entities that involve people with a variety of roles; from the role of “mere” citizen, to that of building project manager, to that of city official. In this paper, I try to map out the related design space: First, I discuss the benefits for different stakeholders, as perceived benefit of use is one of the key predictive factors of technology acceptance. Second, I discuss system and inter- action design directions - there is a wide variety of possibilities, and decisions will need to be taken on the path towards concretising and implementing a use case of such urban reflection as envisioned. 6 6 An early version of this paper has been presented and discussed at the Smart City Learning Workshop of ECTEL 2014 and is online available at http://www.mifav.uniroma2.it/inevent/events/sclo_ectel2014/index.php? s=201&a=362. The workshop did not publish proceedings however; in addition, the paper has been updated to reflect discussions at said workshop, changes in my emerg- ing understanding of the relationship between urban location-based mood tracking and reflective learning, and comments of ARTEL 2015 reviewers. 77 Mood in the city - data-driven reflection on mood in relation to public spaces - ARTEL15 4 Benefits of Use Technology acceptance has been linked, in organisational settings, to perceived ease of use and benefit [18]. In this section we discuss the potential benefits of location-related mood tracking for two key stakeholder groups in urban settings: Citizens who track their own mood in relation to places and share it in relation to public spaces, and decision makers in the public sector such as city governments or in the private sector such as building project managers. 4.1 Individual Reflection For individuals, location-related mood tracking can serve - at a purely individual level, no sharing is necessary - to become aware of own mood in relation to places. This in turn can be useful to consciously reflect on the interaction between mood and places, and to act on this knowledge: For instance, people could use places as resources for wellbeing, and to avoid where emotionally draining places. They could also consciously aim to change their mood in relation to places, e.g., try to consciously relax in typically stressful places such as crowded public transports. Relevant individuals are not only a city’s citizens, but also tourists, or people who come to a city for work. These processes are cognitive learning processes, and the goal of reflection is for individuals to improve the quality of their personal or work lives. 4.2 Urban Reflection In prior work [13] colleagues and I analysed the functions of sharing information in relationship to reflective learning in an organisational context. We identified four major roles of sharing data for reflection: Data as basis for re-evaluation, as guideline for future behaviour, as starting point for collaborative reflection, and to integrate multiple perspectives. In urban reflection, individuals’ mood in relation to public places would mainly serve as starting point for collaborative reflection. The motivation of the individual person to actually share own mood and additional information can only lie in contributing to making a city “better” - making it more liveable and enjoyable. Thus, sharing will need to be additionally facilitated by smooth and enjoyable user experience in terms of interaction with technology, together with displays of respectful and actual treatment of received input on part of responsible stakeholders, in order to achieve a suitable balance of “ease of use” with “perceived benefit” for people to actually do share their mood and comments. Sharing own mood in relation to public spaces is, from the individual’s point of view, an expression towards an audience that needs to be defined: The audience could be other citizens; thus, sharing own mood in relation to public spaces could be the starting point of an asynchronous public discourse on the “liveability” in public spaces. Shared mood could also address stakeholders that decide on and shape public spaces such as city government or building project organizers. The role of sharing own mood data (and optionally related data such as comments, 78 Mood in the city - data-driven reflection on mood in relation to public spaces - ARTEL15 tags, pictures, etc.) corresponds to what has been called “creating awareness” in [9] as one rationale for triggering a new reflection cycle. Additionally, decision makers could also explicitly ask for focused input from citizens, tourists, people working in the city, etc. This would correspond to “seeking clarification” as rationale for starting a cycle of reflection activities [9]. The multitude of individual moods would be the starting point for re-designing cities. Such processes constitute social processes, and the goal of reflection is for the social entities of district communities (more informal) or cities (including also the formal structures) to improve the “key performance indicators” of a city: Target indicators can be defined and prioritised depending on a community’s current status, but typical indicators would be the quality of urban experience for people living in, working in, or visiting a district or city; or the financial standing of a district or city. 5 System and Interaction Design In this section we discuss system and interaction design options, and emphasize those that we currently think preferable. Concrete design decisions will need to be explored and verified (or rejected) in future empirical studies: We will consider “valid” or “good” design decisions those that lead to appreciable benefits for key stakeholders as discussed in the above section. We assume that mood tracking is done via mobile internet-enabled devices such as smartphones or tablets. But are users prompted to enter their mood, do they enter their mood proactively, or is a hybrid method implemented (e.g., via reminders)? Additionally, it is a priori unclear whether users will express only their mood or add additional context information, e.g., in the form of text, a photo, etc. as users are increasingly used to from other social apps and platforms. We assume, that location does not need to be manually entered into the system but can automatically be obtained via GPS, WiFi positioning, QR-tagged public spaces, etc. Positioning only allows for mood tracking related to the place where one currently is. It is unclear, whether in a system as proposed, mood tracking “after the fact” is desirable, e.g., stating in the evening that in the afternoon in the park one was really relaxed. At the intersection of interaction design and software architecture we place the question of where tracked mood data are stored. In [12], all data are stored on a server, but only shared under specific circumstances. An alternative would be to share every mood entry, i.e., to view the system essentially as a public mood tracking system. At the other end of the privacy spectrum, mood tracking would be individual, and data stored on personal mobile devices. Mood data would only be shared on specific user input. On sharing, mood entries could be shared with or without usernames. The latter is most usual in social apps and platforms. So far, we have discussed the capturing of mood. But how about interacting with location-related mood entries? We argue that users should be able to visualise their own mood in relation to places. But should all users of the system get an overview of mood in the city, or should this be reserved for city government? 79 Mood in the city - data-driven reflection on mood in relation to public spaces - ARTEL15 Should also non-users of the system, as “users of the city”, of the public spaces, be informed about collectively tracked mood? Should shared mood be visualised only in the respective space, or should it be accessible also remotely? In all these cases, visualisation of collectively tracked mood and interactive exploration of captured mood data is an issue. In the case where every visitor of a public space should have the possibility to explore such data, interaction could be via a public website, or be mediated by an in situ ambient device. 6 Outlook As next steps, we will concretise the above discussed benefits for multiple stake- holders as well as system and interaction design options in use cases and pro- totypes around participatory district development activities in Graz. Empirical studies will need to verify whether the above outlined benefits can be reached with urban location-based mood tracking. Acknowledgements The Know-Center is funded within the Austrian COMET Program - Competence Centers for Excellent Technologies - under the auspices of the Austrian Federal Ministry of Transport, Innovation and Technology, the Austrian Federal Ministry of Economy, Family and Youth and by the State of Styria. COMET is managed by the Austrian Research Promotion Agency FFG. References 1. Bertrand, K.Z., Bialik, M., Virdee, K., Gros, A., Bar-Yam, Y.: Sentiment in new york city: A high resolution spatial and temporal view. Tech. rep., New England Complex Systems Institute (2013) 2. Boud, D., Keogh, R., Walker, D.: Promoting reflection in learning: a model. In: Reflection: Turning Experience into Learning, pp. 18–40. Routledge Falmer, New York (1985) 3. Choe, E.K., Lee, N.B., Lee, B., Pratt, W., Kientz, J.A.: Understanding quantified- selfers’ practices in collecting and exploring personal data. In: Proc. of the SIGCHI Conference on Human Factors in Computing Systems. pp. 1143–1152. CHI ’14 (2014), http://doi.acm.org/10.1145/2556288.2557372 4. Fessl, A., Rivera-Pelayo, V., Pammer, V., Braun, S.: Mood tracking in virtual meetings. In: Proceedings of the 7th European Conference on Technology Enhanced Learning. pp. 377–382. EC-TEL’12, Springer-Verlag, Berlin, Heidelberg (2012) 5. Fessl, A., Wesiak, G., Rivera-Pelayo, V., Pammer, V.: In-app reflection guidance for workplace learning. In: accepted at: 10th European Conference on Technoloy Enhanced Learning (ECTEL 2015) (2015) 6. Hoyrup, S.: Reflection as a core process in organisational learning. Journal of Work- place Learning 16(8), 442–454 (2004) 7. Knipfer, K., Kump, B., Wessel, D., Cress, U.: Reflection as catalyst for organisa- tional learning. Studies in Continuing Education 35(1), 30–48 (2013) 80 Mood in the city - data-driven reflection on mood in relation to public spaces - ARTEL15 8. Krogstie, B., Prilla, M., Pammer, V.: Understanding and Supporting Reflective Learning Processes in the Workplace: The RL@Work Model. In: Proceedings of the 8th European Conference on Technology Enhanced Learning (ECTEL 2013) (2013) 9. Krogstie, B.R., Prilla, M., Pammer, V.: Understanding and supporting reflective learning processes in the workplace: The CSRL model. In: Scaling up Learning for Sustained Impact - 8th European Conference, on Technology Enhanced Learning, EC-TEL 2013, Paphos, Cyprus, September 17-21, 2013. pp. 151–164 (2013) 10. Li, I., Dey, A.K., Forlizzi, J.: Understanding my data, myself: Supporting self- reflection with ubicomp technologies. In: Proceedings of the 13th International Conference on Ubiquitous Computing. pp. 405–414. UbiComp ’11, ACM, New York, NY, USA (2011), http://doi.acm.org/10.1145/2030112.2030166 11. McDuff, D., Karlson, A., Kapoor, A., Roseway, A., Czerwinski, M.: Affectaura: An intelligent system for emotional memory. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 849–858. CHI ’12, ACM (2012) 12. Mody, R.N., Willis, K.S., Kerstein, R.: Wimo: Location-based emotion tagging. In: Proceedings of the 8th International Conference on Mobile and Ubiquitous Multimedia. pp. 14:1–14:4. MUM ’09, ACM (2009) 13. Pammer, V., Prilla, M., Divitini, M.: The functions of sharing experiences, observa- tions and insights for reflective learning at work. In: Proceedings of the 2nd Work- shop on Awareness and Reflection in Technology-Enhanced Learning (ARETL 2012) at ECTEL 2012. pp. 117–124. CEUR Workshop Proceedings (2012) 14. Prilla, M., Pammer, V., Balzert, S.: The push and pull of reflection in workplace learning: Designing to support transitions between individual, collaborative and organisational learning. In: 7th European Conference on Technology-Enhanced Learning (ECTEL 2012) (2012) 15. Quercia, D.: Urban: Crowdsourcing for the good of london. In: Proceedings of the 22Nd International Conference on World Wide Web Companion. pp. 591– 592. WWW ’13 Companion, International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland (2013) 16. Quercia, D., Schifanella, R., Aiello, L.M.: The shortest path to happiness: Recom- mending beautiful, quiet, and happy routes in the city. In: Proceedings of Hyper- Text 2014 (2014) 17. Ståhl, A., Höök, K., Svensson, M., Taylor, A.S., Combetto, M.: Experiencing the affective diary. Personal Ubiquitous Comput. 13(5), 365–378 (Jun 2009) 18. Venkatesh, V., Morris, M.G., Davis, G.B., Davis, F.D.: User acceptance of infor- mation technology: Toward a unified view. MIS Q. 27(3), 425–478 (Sep 2003) 81 82 Keywords of written reflection - a comparison between reflective and descriptive datasets Thomas Daniel Ullmann Institute of Educational Technology, The Open University, Walton Hall, MK7 6AA Milton Keynes, United Kingdom t.ullmann@open.ac.uk http://www.open.ac.uk/iet/ Abstract: This study investigates reflection keywords by contrasting two da- tasets, one of reflective sentences and another of descriptive sentences. The log- likelihood statistic reveals several reflection keywords that are discussed in the context of a model for reflective writing. These keywords are seen as a useful building block for tools that can automatically analyse reflection in texts. Keywords: reflection detection, thinking skills analytics, log-likelihood, key- word, key word 1 Introduction Supporting learners with opportunities for reflective practice and fostering their reflec- tive thinking are important educational goals. The UK Quality Assurance Agency for Higher Education (QAA), for example, recommends that all teaching and learning prac- tices be informed by reflection [1]. The Organisation for Economic Co-operation and Development (OECD) places reflection at the ‘heart of key competencies’ [2], and fur- thermore, the Assessment and Analytical Framework for PISA sees reflection and eval- uation as part of their assessment framework for reading literacy [3]. There are many ways for expressing reflective thoughts. A common representation is reflective writing (for example, see [4,5]). Reflective writing is a piece of text that contains the reflections of the writer. For example, reflective writing can be a journal, diary, blog post, or structured worksheet. Researchers frequently analyse reflective writings to determine reflective writing quality and evaluate the success of academic writing programmes. This analysis usually follows a content analysis approach (for example, see [6,7]). Researchers use content analysis to systematically detect all textual evidence that belongs to model categories of reflective writing. However, the content analysis of reflective writing is a time-con- suming process. Automated reflection analytics techniques have the potential of reduc- ing the amount of time necessary to analyse reflective writings. This paper contributes to the research of automated detection of reflection in texts (for example, see [8,9,10]). The automated detection of reflection is linked to one of 83 Keywords of written reflection - ARTEL15 the grand challenges of technology enhanced learning, which is 'e-assessment and au- tomated feedback' [11,12]. In order to create automated systems to assist reflective writ- ing assessment, techniques have to be developed first that can automatically detect re- flection in texts. Once a system can detect reflection, this information can be used to automatically assist the assessment of reflection. Therefore, reflection detection is a base technology with the potential for several applications, for example, e-assessment and automated feedback. For the automated detection of reflection, it is important to identify patterns or reg- ularities found in reflective writings. These regularities bear the potential of being for- malised in computer programs, which then can automatically detect these patterns of reflection in novel texts. This paper shows a method that allows the identification of reflection keywords based on the comparison of datasets with the log-likelihood statis- tic. It then discusses the keywords derived in the context of a model of written reflec- tion. 2 Automated detection of reflection in texts Research in the area of automated detection of reflection aims at the development of those techniques and technologies that can automatically identify the characteristics of reflection in texts. Three techniques have been identified that have been used to auto- matically analyse texts in respect to reflection [13]. They are dictionary-based, rule- based, and machine learning-based approaches. The dictionary-based approach makes use of lists/dictionaries of words. The words contained in a dictionary represent aspects of reflection. These dictionaries can be used to analyse texts with regard to the fre- quency of word occurrences in texts, or to visually highlight detected text passages (for example, see [8]). The rule-based approach makes use of a set of rules. Each rule cap- tures an aspect of reflection. These rules, along with a rule-based system, allow drawing inferences from texts, and can be used to analyse reflective writings (for example, see [9]). The first two approaches make use of expert knowledge in order to construct the dictionaries or rules. The third approach makes use of machine learning. Machine learn- ing algorithms learn regularities or patterns from many examples that represent facets of reflection [13]. The generated machine learning models classify unseen text into cat- egories of reflection. Although machine learning-based approaches can automatically build models to de- tect reflection, the first two approaches rely on explicit knowledge about either words or rules that represent aspects of reflection. The literature of research that applied content analysis to investigate reflective writ- ings indicated that reflective writings exhibit such textual patterns. Hatton and Smith [14] touched on language patterns that aided the coding of dialogic reflection. Fund et al. [15] noted the coordination between idea units as important for identifying reflection types. Poom-Valickis and Mathews [16] mentioned lists of keywords in order to code text units. Hawkes and Romiszowski [17] and Hawkes [18, 19] suggested an associa- tion between discourse markers and reflection. In that research, the guiding framework of the analysis was the model of reflection selected by these authors. 84 Keywords of written reflection - ARTEL15 Another area of the research that investigates patterns of reflection makes use of systemic-functional linguistics [20–28]. This research has in common that it investi- gates text based on a linguistic framework in order to derive a link between the linguis- tic framework and reflection expressed in texts. The guiding framework for this type of research is the linguistic framework. Categories of the linguistic framework are then mapped to categories of reflective writing models in order to explore the relationship between linguistic resources and reflective writing. The approach taken here makes use of the corpus linguistic keyword method using log-likelihood statistic as described by Rayson [29] to find reflection keywords. This method is based on the frequency analysis of two corpora/datasets in order to investi- gate words that occur significantly more frequently in one or another dataset. Here, the datasets consist of reflective and descriptive sentences. The keyword method is used to reveal words that are significantly more or less often used in the dataset of reflective sentences compared with the dataset of descriptive sentences. This approach is different from the above outlined content analysis and systemic- functional approaches because it places data first, and not theory. The aforementioned approaches use theory to interpret data, whereas the approach taken here derives a set of keywords using a statistical method. These empirically derived keywords can then be interpreted in the context of theory. This is at first a data-driven approach that may inform theory [29]. The term 'keyword' has several notions, and within this paper it describes those words that occur significantly more frequently within one dataset than another [30]. Several statistical tests can be used to calculate the 'keyness' of words [31]. Here, the chosen test is the log-likelihood ratio test as described by Rayson [29] (see also [32] for a similar implementation of the log-likelihood test). 3 Models to analyse reflective writings The datasets used to derive keywords were created according to a frequently found distinction made in research that analyses reflective writings: A text can be either de- scriptive/non-reflective or reflective. The lowest level is often described as descriptive, and it contains no reflection; on the other side of the scale are reflective texts, which can be further distinguished according to several levels of reflection (for example, see [33,34–38]). However, the common denominator of these models is the basic distinc- tion between descriptive and reflective texts. In addition to levels of reflection, research into the analysis of reflective writings proposed several models with various model categories that describe constituents of reflective writing. These model constituents describe the breadth, and not the depth, of reflection as the level models. Manual content analysis of reflective writings uses the categories that describe breadth facets of reflective writing, as well as levels of reflec- tion as their coding category schema. Although the model categories vary from research to research, they do share some commonalities. The model used to aid in the interpre- tation of the keywords is based on the model for reflection detection described by 85 Keywords of written reflection - ARTEL15 Ullmann [13]. An older version of this model can be found in the study by Ullmann et al. [9]. The breadth model categories are: Experience: A reflective writing is often about experience or a personal matter. The description of what occurred and the capturing of the important characteristic of the situation provide the background and focal point for reflective writing. The description of the experience captures important parts of the experience, and provides the context and/or the reason for the writing. This category can be frequently found in models that analyse reflection (for example, see [6,7], [39]). Personal: A reflective writing is often of a personal nature. This means that it is often about beliefs, personal assumptions, or knowledge about oneself. The text is writ- ten with a personal voice and shows the development of a perspective on the experience at hand. Several models describe this category (for example, see [20], [39,40]). Feelings: Feelings can be the starting point of a reflection. Feelings often associated with reflection are the feeling of being concerned, having doubts, a feeling of uncer- tainty, frustration, but also feelings such as surprise or excitement. Whereas feelings can be the starting point of a reflection, they can also be the subject matter of the re- flection, for example, reflections on the influence of feelings on thinking and action. Several models that analyse reflective writings contain references to this category (for example, see [7], [39,40]). Critical stance: Expressing an alert or critical mindset is an important part of re- flective writing. Having a critical stance involves being aware of problems and being able to identify or diagnose such problems. Being critical is about questioning assump- tions and opinions, analysing and evaluating problems, judging situations, testing the validity of assumptions, drawing conclusions, and making decisions. This category is mentioned in many models (for example, see [6], [20], [39]). Perspective: Although reflective writings are often written from the first person per- spective, considering other perspectives is an important facet of reflective writing. Ex- amples are the perspective of someone else, a theory; the social, historical, ethical, moral, or political context. Several content analysis models contain this category (for example, see [20], [39], [41]). Outcome: There can be several outcomes from reflective writing. An outcome from reflective writing can be a description of lessons learned, better understanding of the situation or context, new insights, change of perspective or behaviour, and the aware- ness of one's way of thinking. An outcome can be also an intention to do something or any planning for the future. The category outcome is also frequently mentioned in con- tent analysis models used to analyse reflective writings (for example, see [39], [41,42]). These six categories, which stem directly from the research on manual content anal- ysis of reflective writing, form the guiding framework for the interpretation of the re- sults of keyword analysis. 4 Dataset generation process and datasets The datasets of reflective and descriptive sentences were obtained from research that investigated the automated detection of reflection using machine learning (details are 86 Keywords of written reflection - ARTEL15 found in [13]). These two datasets are mostly based on a sample of the British Academic Writing English Corpus (BAWE) [43,44]. The sampled texts are mostly from the dis- ciplines of health, business, and engineering. A sentence splitter divided each sample text from the BAWE text collection into sentences. Seven to ten raters ranked each sentence on a six-point Likert scale as to whether the sentence is descriptive or reflective. A crowdsourcing solution1 was used to distribute the sentences to the raters. An even-numbered scale was used so that the raters had to decide whether the sentence is reflective or descriptive, and to avoid mis- using the neutral point of an odd-numbered scale as the 'don't know' category. The rat- ings on the six-point Likert scale were then dichotomised into the class reflective and descriptive. The aim was then to generate two datasets of approximate equal size to aid the com- parison of both datasets. A sentence was only included into the dataset if it received a 4/5 majority of ratings for either belonging to the reflective or descriptive classes. The decision on which aggregation strategy to choose was based on the 4/5 majority because it represents a more strict quality standard compared with the more lenient simple ma- jority vote. This ensures that only those sentences that received substantial support as belonging to one of the two categories were included in this study. For example, a sen- tence that received ten ratings was included if eight or more of the ten ratings ranked the sentence as reflective (or descriptive). Reliability estimates of the ratings aggregated with majority and 4/5 majority vote were reported by Ullmann [13], who found as sub- stantial for the majority vote a Cohen's kappa of 0.62, and almost perfect for the 4/5 majority vote a Cohen's kappa of 0.92, according to the benchmark of Landis and Koch [45]. From this annotated dataset of highly agreed sentences, a random sample of 500 reflective sentences and 500 descriptive sentences was drawn. All sentences from the two datasets were pre-processed with the same data genera- tion process. This involved the removal of any punctuations, numbers, and superfluous whitespaces, sentence tokenisation to words, and word conversion to lower case. The R environment for statistical computing and graphics [46,47] was used to develop the scripts for data processing and calculation of the log-likelihood ratio. The dataset of reflective sentences contains a total of 12,697 words (2,200 unique words). The average sentence length is 25.39 words. The dataset of descriptive/non- reflective sentences contains a total of 10,284 words (2,800 unique words). The average sentence length is 20.57 words. 5 Results The frequency of each word of each dataset was counted and compared. Word compar- ison is based on the log-likelihood of the two terms [29]. The log-likelihood considers the frequency of the two terms compared with the size of the entire datasets. Table 1 lists the log-likelihood of all words with a log-likelihood higher than 10.83, which rep- resents a p-value < 0.0012, and an effect size calculated with the Bayes Factor2 of > 2 1 CrowdFlower (http://www.crowdflower.com/) 2 http://ucrel.lancs.ac.uk/llwizard.html 87 Keywords of written reflection - ARTEL15 [48]. Word pairs below these thresholds are not listed. Table 1 is sorted by the log- likelihood with the highest log-likelihood at the top, and the lowest at the bottom. Fur- thermore, the table indicates for each term, the frequency of occurrence in the datasets of reflective and descriptive sentences. The column 'Use' indicates with a + and - sign whether the term is overused ('+') in the reflective dataset, which means that it has a higher relative frequency in the reflective dataset, or underused ('-'), which means that it is more frequently used in the dataset of descriptive sentences. For example, the word 'i' is frequently present in the dataset of reflective sentences. It occurs 700 times in the reflective dataset and 105 times in the descriptive dataset. It has the highest log-likelihood ratio of 376.07, which means that the word 'i' occurs unusually often (the column use has a '+'-sign) in the dataset of reflective sentences compared to the dataset of descriptive sentences. The word 'he' is underused in the re- flective dataset (see the '-'-sign), which means it appears unusually often in the descrip- tive dataset according to the used log-likelihood test. Table 1. Log-likelihood of the datasets words. Reflective Descriptive Log- Word Use dataset dataset likelihood i 700 105 376.07 + have 191 35 88.09 + me 107 8 81.75 + my 201 56 59.11 + feel 68 4 56.23 + felt 61 4 48.76 + not 117 29 39.91 + that 285 130 31.25 + more 78 18 28.85 + better 30 1 28.37 + is 72 123 26.41 - this 157 63 24.11 + believe 26 1 23.91 + now 29 2 22.80 + he 5 26 20.35 - by 36 71 20.23 - future 17 0 20.17 + of 285 329 19.24 - was 181 84 18.84 + situation 31 4 18.34 + think 31 4 18.34 + are 30 61 18.32 - if 47 11 17.12 + would 83 29 17.01 + and 369 402 16.92 - but 51 13 16.82 + 88 Keywords of written reflection - ARTEL15 Reflective Descriptive Log- Word Use dataset dataset likelihood never 14 0 16.61 + bit 13 0 15.43 + system 1 13 14.89 - their 11 31 14.60 - could 56 17 14.55 + ae3 0 9 14.47 - knowledge 5 21 14.25 - aware 12 0 14.24 + hindsight 12 0 14.24 + learnt 17 1 14.06 + although 20 2 13.54 + probably 11 0 13.05 + it 160 80 12.98 + place 2 14 12.83 - myself 22 3 12.58 + 6 Discussion Table 1 lists the dataset words with the highest log-likelihood. Their relative frequency differs between datasets, which makes them distinctive. They are words for which it is unlikely that the null hypothesis, where their relative frequencies are the same, is true. These are the keywords defined by the statistical procedure. Their p-value and effect size act as inclusion criteria. An additional criterion could have been used to exclude words that occur relatively infrequently [30]. For example, the word 'ae' for 'A&E' (see footnote 3) occurs nine times in both datasets, which makes it the word with the least occurrences in Table 1. In the following subsections, some of the keywords are discussed, and a link between the keywords and their belongings to one of the categories of the model of reflective writing is established. Several of the keywords are illustrated with sentences obtained from the datasets. The keywords within each sample sentence are highlighted in bold. 6.1 Experience The description of an 'Experience' often entails the description of a situation that oc- curred in the past. One of the keywords directly addresses a 'situation'. An example of a sentence with this key word is, 'On the whole I felt I and the other members of staff did all they could to manage a difficult situation and gave Joseph more than ample 3 The word 'ae' represents 'A&E', which refers to the Accident and Emergency service. As all punctuations have been removed during the data generation process, also the '&' of A&E was removed. 89 Keywords of written reflection - ARTEL15 opportunity to cooperate however, in hindsight I feel some aspects could have been handled differently'. Another example is the sentence, 'My reaction to her and the situ- ation surprised me as I became quiet agitated and in hindsight was probably just as unaccommodating as she was'. The two sentences also contain the keyword 'hindsight', which is used to express a retrospective understanding of a situation. 6.2 Personal It is notable that first person singular pronouns, such as 'I', 'me', 'my', and 'myself', are keywords overused in the reflective dataset. This may indicate that reflective writings are frequently written from the first person perspective, which is also in line with the category 'Personal' of the reflection model as outlined above. The third person singular pronoun 'he' is more associated with a descriptive text. The category 'Personal' is also about own beliefs. The keyword 'believes' may be indicative for expressing beliefs. The following sentence is an example of this: 'Reflecting about conflicts I had in the past, I believe that I could have handled some of them better'. Another example of this is the sentence, 'When the time came to allocate work for the plan I believe the team dynamics were developed enough to assess accurately everybody's strengths and apportioned work accordingly'. The modal verb 'would' can also refer to beliefs. An example is the sentence, 'I felt that as team leader I would have control in the group and I would have more say in the way our team was run, little did I know then'. Another example is the sentence, 'Knowing about the tradition, I would definitely have acted differently, hope- fully being in the position to speak at least a bit of the language'. 6.3 Feelings The words 'feel' and 'felt' are at top of the list, and they occur relatively more often in the dataset of reflective sentences. This is in line the category 'Feelings' of the reflection model. Expressing feelings is often mentioned as part of reflective writing. 6.4 Critical stance Several words can be associated with the category 'Critical stance', for example, the keywords: 'more', 'better', 'if', 'but', 'never', 'could', and 'although'. The word 'more' could relate to the critical thought of a writer that something is lacking and that more of something would be better, or it could relate to the realisation that there is now more of something that was previously not there. For example, the sentence, 'I should be more aware about the power issues and how they silence patients', expresses the first sense, which is the realisation that something is still lacking. The sentence, 'I noticed more discussions taking place after the first couple of sessions, and I felt our group was more established as we began to get to grips with what the vignette would entail and felt comfortable with each other', refers to the second meaning—the realisation of a change. The word 'better' could refer to the critical awareness of the writer that something is now better, as expressed in the sentence, 'I hadn't really thought of it like this before 90 Keywords of written reflection - ARTEL15 but by empathising with Mary's situation I better appreciate the importance of the pa- tient's perspective'. It could also express that something should have been better, as in the sentence, 'I might have better explored Jim's internal thoughts and his wondering about what he finds in the cave'. The conjunction 'if' could express a premise followed by a conclusion as part of reasoning about something. For example, the sentence, 'It would have been helpful if I had shared my concerns about the group with the LSA to start with'. If he had shared his concerns (premise), then it would have been helpful (conclusion). The conjunction 'but' could express a contrasting thought. An example is the sen- tence, 'Looking at it from Marissa's point of view, she may have known that I was on the wrong track, but she probably would not have been able to do anything about it because I am a doctor'. Another example is the sentence, 'I did not do all that well in the exam so maybe I need to prepare differently - but I really don't know how to do it'. A writer could flag with the adverb 'never' a realisation that something was never experienced or that something never happened before. The following sentence is indic- ative for this: 'In the topic I found it most interesting about the lack of invariance prob- lem as I have never realised the fact before although the point is reasonably under- standable'. Another example is the following sentence: 'I had never experienced those same feelings of disconnect in real time though, never felt as though the person talking was somehow not me, until last Tuesday'. The verb 'could' might indicate the awareness of a possibility or alternative. The sentence, 'On the whole our group worked well but could have been improved by more openness and discussion about issues affecting the group, such as social loafing', shows that the writer describes a realisation that there is a real possibility for improvement. Another example is the sentence, 'Reflecting about conflicts I had in the past, I believe that I could have handled some of them better'. The word 'although' could be used in a contrasting way. An example is the following sentence: 'Although throughout my training to date, I have dutifully reflected on vari- ous clinical situations and considered learning objectives within the practice portfolio; I can not say that I had actually fully taken on the implications of what it is to be a truly reflective practitioner'. The writer describes the contrast between the perception of re- flection in previous trainings and the current perception. Another example is the sen- tence, 'Applying the learning cycle proved to be a useful tool, although I was very sceptic at the beginning'. This sentence describes a contrast in perception. The sceptical few dissolved over time. 6.5 Perspective The keyword list from Table 1 does not contain a keyword that allows us to establish a strong link between a keyword and the category 'Perspective'. 91 Keywords of written reflection - ARTEL15 6.6 Outcome The 'Outcome' category of the model contains a retrospective dimension that entails, for example, the description of lessons learned, but also a prospective dimension di- rected to statements about what to do in future. The keyword list contains the word 'learnt'. This word could express that something was learned. An example of this out- come facet is the sentence, 'I have learnt that when I am requesting something different, I must explain my needs fully, and communicate the message more effectively'. An- other example is the sentence, 'I have also learnt from participating in this group that I was, in this case, one of the more dominant group members, and felt confident in ex- pressing my views and ideas'. Another keyword is the noun 'future'. This keyword could be used to express future intentions. An example is the sentence, 'Although this situa- tion didn't have a satisfactory outcome I hope to have learned from the experience and aim to use my new insight to develop my future practice'. Another example is the sen- tence, 'This makes you realise that you could come across this within the profession and luckily from this activity I am now aware of this and I can now make the most of any opportunities that a rise to enable me to take this into account and maybe build up my confidence so in the future I can maybe go onto stand up for what I believe in and also get my opinions noticed if I feel this necessary'. 6.7 Summary Overall, this discussion showed that for several keywords, the log-likelihood statistic can derive words that are in line with the categories of the chosen model of reflection. The keywords listed in Table 1 can be seen as good candidate words for the con- struction of dictionaries. With the shown approach, we can form a set of words, such as a dictionary, that can be used to automatically summarise texts with regard to the frequency of occurrence for each category. However, Table 1 also shows that words, although they are found frequently in one dataset, cannot be used to distinguish com- pletely between reflective and descriptive use. For example, the keyword 'i' is found 700 times in the reflective dataset, but 'i' is also used 105 times in the dataset of de- scriptive sentences. A definite classification of text passages based on single words is also not the aim of dictionary-based approaches. One of the use cases there is that the dictionaries are used to predict important outcomes. This is a quantitative indicator that can be used to corroborate the findings of a study (for example, see [49]). 7 Conclusion and outlook This paper demonstrated the application of the keyword method on a dataset of reflec- tive and descriptive sentences. The log-likelihood statistics was used to determine words with high 'keyness' in either the dataset of reflective sentences or that of descrip- tive sentences. The words derived with the described method represent words that occur with unusual relative frequency in the datasets. In the discussion of the results, several of these keywords were assigned to categories of a model of reflective writing. This step illustrated that the investigated keywords can be associated with the reflection 92 Keywords of written reflection - ARTEL15 model categories. This supports the applicability of the keyword method to derive words important for reflection. An extension of the shown approach is to investigate the 'keyness' of the categories of the reflection model. For example, a comparison of a dataset that describes the out- comes of reflective writings with a reference dataset would allow us to derive keywords of reflection outcomes by adding an in-depth study of keywords for this category. Furthermore, reflection dictionaries can be combined with rule-based systems [9]. Rule-based systems provide more control in modelling relationships between diction- ary words, which could add to the precision of the automated method. As outlined, the automated detection of reflection in writings relies on patterns of reflection, because these patterns can be codified and used for the automated analysis of writings. The study showed that reflective writings contain such patterns at the word level, because there are words that occur significantly more often in the dataset of re- flective sentences than in the dataset of descriptive sentences. 8 References 1. QAA: UK quality code for higher education. Part B: Assuring and enhancing aca- demic quality. Chapter B3: Learning and teaching. (2012). 2. Rychen, D.S., Salganik, L.H.: The Definition and Selection of Key Competencies: Executive Summary. OECD (2005). 3. OECD: PISA 2012 Assessment and Analytical Framework. OECD Publishing (2013). 4. Dyment, J.E., O’Connell, T.S.: The Quality of Reflection in Student Journals: A Review of Limiting and Enabling Factors. Innovative Higher Education. 35, 233– 244 (2010). 5. Moon, J.A.: Learning Journals: A Handbook for Reflective Practice and Profes- sional Development. Routledge (2006). 6. Poldner, E., Van der Schaaf, M., Simons, P.R.-J., Van Tartwijk, J., Wijngaards, G.: Assessing student teachers’ reflective writing through quantitative content analysis. European Journal of Teacher Education. 37, 348–373 (2014). 7. Prilla, M., Renner, B.: Supporting Collaborative Reflection at Work: A Compara- tive Case Analysis. In: Proceedings of the 18th International Conference on Sup- porting Group Work. pp. 182–193. ACM Press, New York, NY, USA (2014). 8. Ullmann, T.D.: An Architecture for the Automated Detection of Textual Indicators of Reflection. In: Reinhardt, W., Ullmann, T.D., Scott, P., Pammer, V., Conlan, O., and Berlanga, A. (eds.) Proceedings of the 1st European Workshop on Aware- ness and Reflection in Learning Networks. pp. 138–151. CEUR-WS.org, Palermo, Italy (2011). 9. Ullmann, T.D., Wild, F., Scott, P.: Comparing Automatically Detected Reflective Texts with Human Judgements. In: Moore, A., Pammer, V., Pannese, L., Prilla, M., Rajagopal, K., Reinhardt, W., Ullmann, T.D., and Voigt, C. (eds.) 2nd Work- shop on Awareness and Reflection in Technology-Enhanced Learning. CEUR- WS.org, Saarbruecken, Germany (2012). 93 Keywords of written reflection - ARTEL15 10. Ullmann, T.D., Wild, F., Scott, P.: Reflection - quantifying a rare good. In: Kravcik, M., Krogstie, B.R., Moore, A., Pammer, V., Pannese, L., Prilla, M., Rein- hardt, W., and Ullmann, T.D. (eds.) Proceedings of the 3rd Workshop on Aware- ness and Reflection in Technology-Enhanced Learning. pp. 29–40. CEUR- WS.org, Paphos, Cyprus (2013). 11. Sutherland, R., Eagle, S., Joubert, M.: A vision and strategy for Technology En- hanced Learning: Report from the STELLAR Network of Excellence. (2012). 12. Fischer, F., Wild, F., Sutherland, R., Zirn, L. eds: Grand Challenges in Technology Enhanced Learning - Outcomes of the 3rd Alpine Rendez-Vous. Springer Interna- tional Publishing (2014). 13. Ullmann, T.D.: Automated detection of reflection in texts - A machine learning based approach. PhD thesis The Open University (submitted). 14. Hatton, N., Smith, D.: Reflection in teacher education: Towards definition and im- plementation. Teaching and Teacher Education. 11, 33–49 (1995). 15. Fund, Z., Court, D., Kramarski, B.: Construction and Application of an Evaluative Tool to Assess Reflection in Teacher-Training Courses. Assessment & Evaluation in Higher Education. 27, 485–499 (2002). 16. Poom-Valickis, K., Mathews, S.: Reflecting others and own practice: an analysis of novice teachers’ reflection skills. Reflective Practice. 14, 420–434 (2013). 17. Hawkes, M., Romiszowski, A.: Examining the reflective outcomes of asynchro- nous computer-mediated communication on inservice teacher development. Jour- nal of Technology and Teacher Education. 9, 285–308 (2001). 18. Hawkes, M.: An analysis of critically reflective teacher dialogue in asynchronous computer-mediated communication. In: IEEE International Conference on Ad- vanced Learning Technologies, 2001. Proceedings. pp. 247 –250 (2001). 19. Hawkes, M.: Linguistic Discourse Variables as Indicators of Reflective Online In- teraction. American Journal of Distance Education. 20, 231–244 (2006). 20. Birney, R.: Reflective Writing: Quantitative Assessment and Identification of Lin- guistic Features, http://repository.wit.ie/2658/, (2012). 21. Forbes, A.: Evidence of learning in reflective practice: a case study of computer- assisted analysis of students’ reflective blogs. New Zealand Association for Coop- erative Education 2011 Conference Proceedings. 11–14 (2011). 22. Luk, J.: Assessing teaching practicum reflections: Distinguishing discourse fea- tures of the “high” and “low” grade reports. System. 36, 624–641 (2008). 23. Reidsema, C., Mort, P.: Assessing reflective writing: Analysis of reflective writing in an engineering design course. Journal of Academic Language and Learning. 3, A117–A129 (2009). 24. Ryan, M.: Improving reflective writing in higher education: a social semiotic per- spective. Teaching in Higher Education. 16, 99–111 (2011). 25. Ryan, M.: Conceptualising and teaching discursive and performative reflection in higher education. Studies in Continuing Education. 34, 207–223 (2012). 26. Ryan, M.: Reflexive writers: Re-thinking writing development and assessment in schools. Assessing Writing. 22, 60–74 (2014). 94 Keywords of written reflection - ARTEL15 27. Shaheed, N., Dong, A.: Reflection and analysis in design student blogs. DS 36: Proceedings DESIGN 2006, the 9th International Design Conference, Dubrovnik, Croatia. (2006). 28. Wharton, S.: Presenting a united front: assessed reflective writing on a group ex- perience. Reflective Practice. 13, 489–501 (2012). 29. Rayson, P.: From key words to key semantic domains. International Journal of Corpus Linguistics. 13, 519–549 (2008). 30. Scott, M.: PC analysis of key words — And key key words. System. 25, 233–245 (1997). 31. Lijffijt, J., Nevalainen, T., Säily, T., Papapetrou, P., Puolamäki, K., Mannila, H.: Significance testing of word frequencies in corpora. Digital Scholarship in the Hu- manities. fqu064 (2014). 32. Dunning, T.: Accurate methods for the statistics of surprise and coincidence. Com- putational linguistics. 19, 61–74 (1993). 33. Ballard, K.K.: Using Van Manen’s model to assess levels of reflectivity among preservice physical education teachers, https://repository.tamu.edu/han- dle/1969.1/4373, (2006). 34. Ip, W.Y., Lui, M.H., Chien, W.T., Lee, I.F., Lam, L.W., Lee, D.: Promoting self- reflection in clinical practice among Chinese nursing undergraduates in Hong Kong. Contemporary Nurse. 41, 253–262 (2012). 35. Kember, D., Jones, A., Loke, A., McKay, J., Sinclair, K., Tse, H., Webb, C., Wong, F., Wong, M., Yeung, E.: Determining the level of reflective thinking from students’ written journals using a coding scheme based on the work of Mezirow. Int. J. of Lifelong Educ. 18, 18–30 (1999). 36. Lai, G., Calandra, B.: Examining the effects of computer-based scaffolds on nov- ice teachers’ reflective journal writing. ETR&D-Educ. Tech. Res. Dev. 58, 421– 437 (2010). 37. McCollum, S.: Insights Into the Process of Guiding reflection During an Early Field experience of Preservice Teachers. (1997). 38. Sparks-Langer, G.M., Simmons, J.M., Pasch, M., Colton, A., Starko, A.: Reflec- tive Pedagogical Thinking: How Can We Promote It and Measure It? Journal of Teacher Education. 41, 23 –32 (1990). 39. Wald, H.S., Borkan, J.M., Taylor, J.S., Anthony, D., Reis, S.P.: Fostering and Evaluating Reflective Capacity in Medical Education: Developing the REFLECT Rubric for Assessing Reflective Writing. Academic Medicine. 87, 41–50 (2012). 40. Wallman, A., Lindblad, A.K., Hall, S., Lundmark, A., Ring, L.: A Categorization Scheme for Assessing Pharmacy Students’ Levels of Reflection During Intern- ships. Am J Pharm Educ. 72, (2008). 41. Plack, M.M., Driscoll, M., Marquez, M., Cuppernull, L., Maring, J., Greenberg, L.: Assessing Reflective Writing on a Pediatric Clerkship by Using a Modified Bloom’s Taxonomy. Ambulatory Pediatrics. 7, 285–291 (2007). 42. Plack, M.M., Driscoll, M., Blissett, S., McKenna, R., Plack, T.P.: A method for assessing reflective journal writing. Journal of allied health. 34, 199–208 (2005). 43. Gardner, S., Nesi, H.: A Classification of Genre Families in University Student Writing. Applied Linguistics. 34, 25–52 (2013). 95 Keywords of written reflection - ARTEL15 44. Nesi, H., Gardner, S.: Genres across the disciplines: Student writing in higher ed- ucation. Cambridge University Press (2012). 45. Landis, J.R., Koch, G.G.: The Measurement of Observer Agreement for Categor- ical Data. Biometrics. 33, 159–174 (1977). 46. R Core Team: R: A Language and Environment for Statistical Computing. R Foun- dation for Statistical Computing, Vienna, Austria (2014). 47. Feinerer, I., Hornik, K.: tm: Text Mining Package. (2014). 48. Wilson, A.: Embracing Bayes factors for key item analysis in corpus linguistics. In: Bieswanger, M. and Koll-Stobbe, A. (eds.) New approaches to the study of linguistic variability. pp. 3–11. Peter Lang, Frankfurt (2013). 49. Chung, C.K., Pennebaker, J.W.: Linguistic Inquiry and Word Count (LIWC): Pro- nounced “Luke” and Other Useful Facts. In: McCarthy, P.M. and Boonthum, C. (eds.) Applied Natural Language Processing and Content Analysis: Advances in Identification, Investigation and Resolution. pp. 206–229. Information Science Reference (an imprint of IGI Global), Hershey, United States of America (2012). 96 The LearnWeb Formative Assessment Extension: Supporting Awareness and Reflection in Blended Courses Alana Morais1, Ivana Marenzi2, Deirdre Kantz3 1 Universidade Federal de Campina Grande, 58429-900 Campina Grande-PB, Brazil alanamorais@copin.ufcg.edu.br 2 Leibniz Universität Hannover, L3S Research Centre, 30167, Hannover, Germany marenzi@l3s.de 3 University of Pavia, Centro Linguistico (Faculty of Medicine), 27100 Pavia, Italy dkantz@unipv.it Abstract. Blended solutions provide a means to orchestrate various types of ac- tivities and to schedule interactions at different times, nonetheless it is difficult to maintain a general overview of the class. In this paper, we build on the LearnWeb Design Framework to design and implement a Formative Assess- ment extension that supports the monitoring of the learning process in order to increase awareness and support reflection in a specific learning scenario. The extension offers a common basis for the various stakeholders (researchers, teachers and students) to collaboratively reflect on and design effective learning activities. Keywords: Awareness, formative assessment, co-design, reflection, TEL. 1 Introduction Very often University courses entail large numbers of students which makes it diffi- cult to design and carry out learning tasks, as well as to assess learning outcomes [13]. The use of technology makes it possible to record the tracks of student activity and to provide the teacher with dedicated analytics to improve awareness [2-3,12]. Learn- ing analytics techniques are a valuable tool to support formative assessment practices that are based on two main pillars: (1) the collection of evidence concerning students’ progress towards learning outcomes; (2) the teachers’ and students’ reflections on the feedback of this information in order to enhance teaching and learning [11]. In this paper we propose a formative assessment strategy based on visualisation techniques to support teachers’ awareness and reflection in University learning con- texts that integrate technology enhanced learning activities in the curriculum. 97 The LearnWeb formative assessment extension - ARTEL15 2 The LearnWeb Formative Assessment Extension as a Means to Support Awareness and Reflection LearnWeb1 is an online learning environment, which allows users to share and col- laboratively work on user-generated resources either uploaded from the desktop, or collected from the web [5-6, 9]. In order to make the knowledge processes explicit both for teachers and for students granting them more agency in learning activities, the LearnWeb system has been designed in keeping with the Learning by Design approach [8,10,14]. The LearnWeb Design Framework (Fig. 1) has been demonstrat- ed to be effective in supporting reflection and collaboration in the co-design of cours- es in the past [7-10]. Now we want to enhance the framework by providing tools that allow teachers to evaluate students’ work throughout their learning pathway. Fig. 1. Web2.0 features to the LearnWeb Design framework (multi-tier model) 2.1 Design of the LearnWeb Formative Assessment Extension In order to provide the teacher with evidence of each student’s (or group’s) involve- ment in the various knowledge processes when carrying out the learning tasks, we need to explicitly associate and display the logged data that corresponds to each activ- ity (see Table 1). In this way, the teacher can refer back to the original framework and course design, and monitor the students’ performance in line with the expected learn- ing goals. In order to address the needs of different scenarios the LearnWeb Formative As- sessment extension has been designed from three main perspectives: (i) a course per- spective, where the teacher has an overview of a specific course and can make com- parisons between/among classes, (ii) a class perspective, where the teacher can moni- 1 http://learnweb.l3s.uni-hannover.de 98 The LearnWeb formative assessment extension - ARTEL15 tor and compare the activities of small groups within the same class, and (iii) a per- sonal perspective where the teacher can visualise information about a specific user. Table 1. Phases of the LearnWeb Design Framework and Logged activities LearnWeb Phases Logged activities Search and exploration Searching, Download, Open resource, Add resource, Delete (Experiencing) resource, Create group, Group joining, Group leaving Annotation and description Tagging resource, Ratings resource, Edit resource, Com- (Conceptualising) ments, Deleting comments Negotiation and discussion Text from comments visualised with WordWanderer 2 (Analysing) Aggregation and presentation Grouping resources and presentation functionality (Applying) Each phase on the user interface is located in a different tab so as to visualise the data relating to each phase in a specific context (see Fig. 2 (1)). The visualisation of each of these perspectives takes into account the activities carried out by students aggre- gated with four groups related to the four LearnWeb Design Framework features as described in Table 1 (see Fig. 2 (2)), thus enabling teachers to analyse the factors involved in the various learning tasks. A specific learning scenario is described in the following section so as to provide a preliminary evaluation of the teacher’s feedback concerning the usefulness of the visualisation of data to support teaching strategies and practice. Fig. 2. LearnWeb Formative Assessment extension interface For the implementation of the Formative Assessment extension, we carried out fre- quency analysis and built the charts using the PrimeFaces3 library. In agreement with the teacher, we started with the visualisation of the data collected in previous years so as to provide a diachronic overview for the teacher who will be able to compare the performances of different courses and reflect on the course design 2 http://wordwanderer.org/ 3 http://primefaces.org/ 99 The LearnWeb formative assessment extension - ARTEL15 of future editions [4]. The objective is to evaluate the work carried out in the past and improve future teaching/learning experiences by personalising and adapting [11]. 2.2 The LearnWeb Formative Assessment Extension in Practice During the Academic Year 2011-2012 we carried out a study at the University of Pavia, in Italy that involved 284 first-year medical students divided into five classes: GolgiA (85 students), GolgiB (71), GolgiC (44), HarveyD (50) and HarveyE (34). The syllabus was based on English linguistics and focused on text-based studies of (bio)-medical English. Students were expected to learn about multimodal theory and how to carry out multimodal text analysis, that is, the study of printed, website, digital and film texts in English and the ways in which these texts are used in different medi- cal and biomedical contexts [1]. For their project work, students were required to carry out research in groups of 8-12 members on the topic “health/bio- medical/scientific education through entertainment of young children and teenagers” and create a corpus of at least 50 websites. The work entailed the annotation of their search trajectories (failures and successes) and the use of the LearnWeb options to communicate with their group members and exchange information and comments. At the time of the described scenario, the Formative Assessment Extension was not yet available, and the teacher had to explore the work done by students by browsing through the various groups. Using the current version of the system, it is now possible to obtain a rapid overview of student contributions with various levels of detail. Fig. 3. Course perspective (Comparison between classes) – Search and Exploration (1) Course Perspective – the interface provides two search fields where the teacher can select two classes to be compared and choose different types of graphs to visual- ise the data. Fig. 3 presents a comparison between the HarveyE and the GolgiA class as regards the first pedagogical phase (Search and Exploration). Students in HarveyE searched and added more resources than the students in GolgiA. Since the project work task was the same for all groups (i.e. build a corpus of at least 50 websites), the teacher might want to intervene and invite the GolgiA groups to speed up the work. 100 The LearnWeb formative assessment extension - ARTEL15 The comparison is about the type of activities carried out by students in the two classes and can be between courses of the past, or courses functioning at the same time. In the first case (past courses), the teacher can see whether the current class is performing better or worse than the previous class. Consequently, the teacher can reflect on how to improve the course design or introduce better explanatory strategies for students by using notifications to communicate with them and give support. In the second case (current courses), the system visualises the actual value at a specific time so that the teacher is constantly up to date on how the students are working in the platform. For example, if one class is performing at a slower rate than another in the same course, the teacher can send a notification and a request to the group leader to speed up the work. This strategy can be useful when the teacher wants to stimulate competition. (2) Class Perspective – the teacher can choose a specific class in the course and se- lect two sub-groups to be compared. In the GolgiA class for example, the results of the comparison between the activities of the Euronics group and the Children`s genes group in the Search and Exploration phase are very similar. Whilst in the Annotation and Description phase (Fig. 4) we notice that a larger number of tags were added by the members of the Euronics group, the students in the Children`s genes group were more active in commenting and editing resources. The results show how the two groups use a different strategy to conceptualise and categorise contents, thus helping the teacher to understand the learning behaviour and evaluate the group work accordingly. The teacher might decide to discuss the findings with students and reflect on their behaviour during the course or the final exam. Fig. 4. Class perspective (group overview) – Annotation and Description (3) Personal Perspective – The window provides two search fields where the teacher can select a specific class and a specific student in that class. The resulting graph shows the number of activities carried out by the student throughout the course (see Fig. 5). This information can help the teacher to better understand the performance of each student and provide personalised feedback, for example according to: (i) the specific role (e.g. group leaders can/should carry out additional activities compared to the other group members), (ii) the given task (e.g. the teacher can encourage the slow- er students to carry their weight). 101 The LearnWeb formative assessment extension - ARTEL15 Fig. 5(a) shows an example of this analysis, where the teacher investigates the pro- file of a student (anonymized in the picture for privacy reasons). The student mainly searched and added resources; strangely no traces for Open resources are logged. This could mean that the student added materials to the group only relying on the title, without checking the content, and the teacher might want to investigate this student’s behaviour further. Fig. 5(b) shows the traces of the same user as regards the pedagog- ical phase of Annotation and Description. He mainly commented resources, but he also used ratings and tags to annotate resources. According to the tasks assigned in the course, the teacher can judge whether the student is behaving as expected, and decide to send him specific feedback or additional directions. Fig. 5. Personal perspective (User Analysis) - (a) Search and Exploration, (b) Annotation and Description Such visualisations are useful to detect an increase or decrease in student participation in learning activities and can encourage the teacher to restructure some tasks or to adopt different pedagogical strategies if deemed necessary. Prompt intervention can be a crucial factor in determining the success or failure of a course. Using the Forma- tive Assessment extension, the teacher is made aware of the dynamics that are taking place in the course and can speedily intervene in order to raise interest when it ap- pears to be waning. As matter of fact, the teacher in Pavia evaluated the prototype of the system and confirmed its potential: “it has practical applications which can save time and allow for the constant realignment of the teaching strategies with the learn- ing goals”. 3 Conclusions and Future Work The LearnWeb Formative Assessment extension is designed to offer a common basis for various stakeholders: for teachers to reflect on the teaching practices and refine their pedagogical strategies; for students to keep track of their personal progress and measure their performance in comparison with their peers; for researchers to realise what functionalities work better to support specific learning tasks and improve the system. While some components of the technical approach are already available, others are under development. For the moment we focused on developing tools to support teach- 102 The LearnWeb formative assessment extension - ARTEL15 er awareness and to facilitate monitoring and mentoring activities. In the future, we will develop the assignment and recommendation component that includes the user interface for the learner. Another step will be the addition of a temporal dimension in order to give a diachronic visualisation of group interactions. The impact of the feed- back provided through the proposed extension on student learning pathways will be investigated in future projects both in Italy and in Brazil. An extended study will be carried out with the aim of analysing how feedback is perceived by students and the impact it has on moulding the next learning stages. 4 References 1. Baldry, A., Thibault, Paul J. Multimodal transcription and text analysis. A multimodal toolkit and coursebook. London and New York: Equinox (2006). 2. Charleer, S., Santos, J. L., Klerkx, J., Duval, E.: LARAe: Learning Analytics Reflection & Awareness environment. In Milos Kravcik et al. (Eds.). In: Proceedings of the ARTEL 2014 Workshop, vol. 1238, pp. 85–87. 3. Govaerts Sten, Verbert Katrien, Klerkx Joris, Erik Duval (2010). Visualizing Activities for Self-reflection and Awareness. Advances in Web-Based Learning (ICWL 2010). 4. Greller, W., Drachsler, H.: Translating Learning into Numbers: A Generic Framework for Learning Analytics. Educational Technology & Society, 15 (3), 42–57 (2012). 5. Marenzi, I., Nejdl, W.: I search therefore I learn. Supporting active and collaborative learn- ing in language teaching. In: A. Okada, T. Connolly, P. Scott (eds.): Collaborative Learn- ing 2.0: Open Educational Resources. IGI Global, pp.103–125 (2012). 6. Marenzi, I., Zerr, S.: Multiliteracies and active learning in CLIL: the development of LearnWeb2.0. IEEE Transactions on Learning Technologies (TLT). Los Alamitos, CA, USA, 5(4), 336–348 (2012). 7. Marenzi, I.: A multiliteracies approach for supporting language learning courses at univer- sity level. In: J. Jovanovic & R. Chiong (eds.): Technological and Social Environments for Interactive Learning. Santa Rosa, CA: Informing Science Press, 249–276 (2013). 8. Marenzi, I., Kantz, D.: ESP course design – a multiliteracies approach. In: Reshaping Learning: Transforming Education through the Fusion of Learning and Technology, Pro- ceedings of the 13th ICALT Conference (2013). 9. Marenzi, I.: Interactive and collaborative supports for CLIL: towards a formal model based on digital literacy. In: R. Kupetz & C. Becker (eds.): Content and Language Integrated Learning (CLIL) by Interaction. Frankfurt am Main: Peter Lang, 75–99 (2014). 10. Marenzi, I.: Multiliteracies and e-learning2.0. Foreign language pedagogy content- and learner-oriented, Vol.28. Frankfurt am Main: Peter Lang (2014). 11. Taibi, D., Kantz, D. and Fulantelli, G.: Supporting formative assessment in content and lan- guage integrated learning: the MWSWeb platform. In: IJTEL, 6,(4), pp.361–379 (2014). 12. Vozniuk, A., Triana, M. J. R., Holzer, A., Govaerts, S., Sandoz, D., Gillet, D.: Contextual Learning Analytics Apps to Create Awareness in Blended Inquiry Learning. In: Proceed- ings of the 14th ITHET (2015). 13. Wang, T. H.: What strategies are effective for formative assessment in an e-learning envi- ronment? In: Journal of Computer Assisted Learning, 23(3), 171–186 (2007). 14. Yelland N., Cope B., Kalantzis M.: Learning by Design: creating pedagogical frameworks for knowledge building in the twenty-first century. Asia-Pacific Journal of Teacher Edu- cation, 36(3), 197–213 (2008). 103 104 ����������������������������������������������� ������������������������������������������������� ���������������������������������������������������������������������������������� ����������������������������������������������������������������� � � �������������������������������������������������������������� ����������������������������������������������������������������������� � ����������������������������� ������������������������� � ���������������� ������������������������������������������ � ������� ������������������������ ��������������� ����������������������������������������������������������������������������� �������� �������� ���� ��������� ����������� ���� ��������� ���� ���������� ��� ���� ���������� ������� ��� ��������� ��������� ����� ������� ��� ����� ��������� ���������� ��� ��������������������������������������������������������������������������������� ������ ��������� ������������� ����� ������ ��������� ������� �� ��������� ����������� ���� ������� ��������� ���� ���������������� ���� ������������ ��������� ����������� ������ ��������� ��������� ���� ���������� ����� ������� ��� ������������ ���������������������������������������������������������������������������������� ����������� ��� ��������� ���������� ��� ���� ������ ��������� ���������� ����� ��� ��������������������������������������������������������������������� ���������������������������������������������������������������������������� ���������������������������������������������������� ����������������� ������� ���������� ��� ���������� ����������� ������ ��������� �������� ������� ��� ���� ������������� ��� ����� �������� ������� ����������� ����� ������ ��� ����� ������������ ���������� ������� ����� ���� ���������� ��� ����� ���� �������� ������������ ������� ������������������������������������������������������������������������������������� ����� ���� ���� ����������� ���� ��������� ����������� ��� ������ ��������� ����� ������� ������������������������������������������������ ������������������������������������� ������ ����� ����� �������� ���������� ������ �������� ��� ���������� ���� �������� �������������������������������������������������������������������������������������� ������� ��� ������ ����� ����� ���� �������� ������� ���������� ������������ ���� ��������� ������������������������������������������������������������� � �������������������������� � ���������������������������� � ����������������������� � ������������������������������ 105 Deploying learning analytics for awareness and reflection - ARTEL15 ��������� ���� �������� ����� ������ ����� ����� �������� ����������� ��� ��������� ���� �� �������������������������������������������������������������� �� �������������������������������������������������������������������������������� ������������ ��������� ���� ���������� ��� ����� ������ ��������� ���� ������ ���������� ���� ����������������������������������������������������������������������������������� ������������������������������������������������������������������������������������ ������������������������������������������������������������������������������������� ��������������������������������������������������������������������������������������� ������ ������ ��� ������� �������� ������� �������� ��������� ������� �������� ���������� �������������������������������������������������������������������� ���������� ������ �������� ���� ��������� �������� ������ ��������� ������� ���������� �������� ������� ��� ����������� ����������� ������������ ����� ����� ������� ������� ���������������� ������ ����� ������������ ����� �������� ��������� ��������� ���� ����������������������������������������������������������������������������������� ���������� ��������� ������������� ���������� ����� ��� ������� ������� ���������� ��� ����� ���������������� ���� ����� ������������ ����������� ���� ������ ������ ��� ����������� ������������ �������������� ���� ����� ���������� ���� ��������� ����� ����������� ���� ��������������������������������������������������������������������������������������� ���� ��������������������������������������������������������������������������������� ���������������������������������������������������������������������������������������� ��������������������������������������������������������������������������������������� ����������������������������������������������������������������������������������������� ������ ��� �� ����� ������ ��� �������� ����� ����� ������ ����� ����� ���� ������ ������������ ���������� ��������� �������� �������� ����� ��������������� ��� ������������������ ����������� ���� ��� ���� ���� ������������������ ��������� ��� ������ ������������ ��� �������������������������������������������������������������������������������������������� ������������������������������������������������������ �������������������������������������������������������������� ����������������� �������� ��������� �������� ��������� ��� �������� ������� ������������ ��������� ����������� �������� ���� ��������� ��������� ����������� ����� ���� ��� ������� ������� ���������� ���������������������������������������������������������������������������������������� ������������������������������������������������������������������������������� ����� ����������������������������������������������������������������������������������� ������� ��������� ��������� �������� ���� ������� ��������� �������������� ������� ������� �������� ������� ���������� ���� ��������� ��� ������� ���� ��������� ���� ������� ��������� ����������� ��� ������� ����� ��� ������ ��� �������� ����������� ������������� ��������� ���� ���������� ���� ������ ���� ���� ������� ��������� ������������ ����� ������� ����� �������� ��������� �������� ��������� ��� �������� ������� ������������ ������� ���� ��������� ��������������������������� ������������������������������������������������������������� � � ������������������������������������������������������������������������������������������� ��������������������������������������������������������������� � ����������������������������������������������������������������������������� ���������������������������������������� � �������������������������� � ������������������������� 106 Deploying learning analytics for awareness and reflection - ARTEL15 �������������������������������������������������������������������������������������� ��� ���� ������ ��������� ����������� �������� �� ��������� ���� ����������� ��� ��������� ������������������������������������������������������������������������������������� �������������������������������������������������� ���������������������� ������ ���� ���������� ���������� �������������� ���� ���������� ������� ������� �������� ������������������������������������������������������������������������������������������� ���� ��������� �� ������������ ���� ��������� ��� �� �������� ����� ����� ��������� �������� ���� ����������������������������������������������������������������������������������������� ������������������������������������������������������������������������������������������ ��������� ���� ��������� ��� ������������ ��������� ������� ��������� ����� ���������� ���������� ������ ��� �������� ����� ���������������������������� ������� ����� ������� ���������������� ����� ������������������������ ����� ��� ������ ��� ������� ������������������������������������������������������������������������������������� ���������������������������������������� � ● ����������� ������� �������������� ��� ����� ������ ���� ��������� ���������� ���� �������� ����������������������������������������������������������������������������������� ��������������������������������������������������������������������������������������� ������� ����������� ���� ������� ��� ��������� ���� ���� ������� ��� ��������� ��������� ���������������������������� ● ��������������������������������������������������������������������������������������� ����� ����� ����� ���� ������� �������������� ������ ����� ����������� ����� ��� ��������� ������ ��� ������ ������������ ���� ���� ��������� ����������� ��� ���� ������� ���� ���� ����������� �������� ���� ������� ��� ��������� ���� ���������� ����� ����� ����� �� ����� ��� ����������������������������������������������������� ● ���������� ������������ ��������� ���� ������������ �������� ����� ����� ����� ���� ���������������������������������������������������������������������������������� ���������� ����� ���� ��������� ���� ���� ������� ��� ������ ����������� ��� ������ ���� ��� �������� �������� ���� ����������� ��� ���� ���� ������� ��������� ����������� ���� ������ ��������������� ● ��������������������������������������������������������������������������������������� ���� ���� ������������ �������� ��� ���� ������� ���� ����������� ��� ������ ��� ����� ���� ������������������������������������������������������������������������������� ������ ���� ��� ������� ������� ���������� ��������� ����������� ���� ���������� ��������� ���������� ● ������������������������������������������������������������������������������� ��� ����� ������ ���������� ��� ���� ������� ������������ ���� ��������� ��������� �������� ������ ��� ���������� ���� ��������� ���� ������� ���� ������� ������� �� ���� ��� ��� ��� ������������������� ● ������������ ��� ����� ������ ���� ��������� ���������� ���� �������� ��� ���� �������� ������ ���������������������������������������������������������������������������������� ���� ������������� ����� ���������� ��� ��������� ���������� ������ ����� ������ ���� ������������������������������������������������������������������������ 107 Deploying learning analytics for awareness and reflection - ARTEL15 ● ����������� ���� ������������ ��� ���������� ���� ������������ ���� ������������� ����� ���������� ����� ���� ����������� ��� ���� �������� ���������� ����� ���� ������������ ��� ������������������������������������������������������������������������������� ��������������� ������������������������������������������������������������������ ����� ��������� ���������� ��� ��� ��������� ��� ������� ��������� ������������ ������ ���� ��������� ��������� ����� ���� ������ ������������ ��� ��������� ��� ������ ���� ���������� ��������� ���� ���������������� ��� ������������ ��� ���� ����������� ����� ���������� ����� ���������� ���� ������������� ��� ��������� ����� ���� ������ �������� ������� ������� ���� ������� ��������� ��� ���� ��������� �������� ������������ ������ �������� ��� ������������� ���������������������������������������������������������������������������������� ���������������������������������������������������������������������������������� �������������������������������������������������������������������������������������� ������������������������������������������������������������������������������������ �������� ����������� ���������� ��� ���� ������ �������� ���������� �������� ���� ���������������������������������������������������������������������������������������� ����������������������������������������������������������������������������������� ����������������������������������������������� ● �������������������������������������������������������������������������������������� �������� ���������� �� ���� ����� ��� �������� ��� ��������� ���� ������������� �������� ��� ���������������������������������������������� ● �������������� ���������� ����� ��� ����������� ��� �������� ��� ���� ��������� ����������� ���������������� ������������� �������������������������������������� ��������������� ���������� ����������������������������������������������������������������������� ������������� ● ���������������������������������������������������������������������������������� ��� ���� ����� �������� ���� ��������� ����� �� �������� ����� ��� �������� ���� ������������ �������������������������������������������������������������������������������� ����� ������������ �������� ����������� ���� ����� ��������� ��� ���� ������ �������� ���������� ��� ���� ���������������� �������� ��� ����� �������� ������ ����������� ���� ���������� ��� ���� ��������� �������� ������ ��������� �� ������� ���� ���������� ����������� ������� �� ������ ��������� ��� ���� ����������� ����� ������ ��� ����� ��� ���� ����� ������� ������ ��������������������������������������� ������������������������������������������������������������� � ������������������������ �� ����������������������������� �� ��������������������������������������������� �� �������������������������������������������� �� ������������������������������������������� 108 Deploying learning analytics for awareness and reflection - ARTEL15 � ����������������������������������������������������������������������������� �������������������� � ���� ����������� ��� ��������� ���������� ������� ������ ��� ������� ��� ������������� ��������������������������������������������������������������������������������������� ������������������������������������������������������������������������������������������ �������������������������������������������������������������������������������������������� ��������� ���� ���������� ���� ������������������ ������������� ��� ������ ��� ���������� ��� ���������� ��������� ����������� ���� ����������� ���������� ��� ���� �������� ��� ������� ��������� ����������� ���� ���������������� ����� ���� ����� ��� ������� ��������� ���������� ���� ���������������������������������������������������������������������������������� ���������� ������ ���� ��������������� �������� ��������� ��������� ��� ���� ����� ��� ���������� ��������������������������������������������������������������� ����������������������������������������������������������������������������������� ���������� ��� ������ ��������� ����������� ��� ���������� ��� ��������� ��� ������ ��������� ��� ������� ����������� ��� ���� ������� ��������� ��������� ���� �������� ����� ��������� �������� ��� ����������������������������������������������������������������������������������������� ��������������������������������������������������������������������������������������� ��������� ��� ���������� �������� ��� ��������� ��������� ������ ���� ����������� ������� ������ ������� ������ ��������� ����� ������� ���� ���������� ������ ��� ���� ������ �������� ���� ������������������������������������������������������������� ������������������������������������������������������������������������������������� ����������� ��� ���� ����������� �������� ����� ��� ���������� ���� ������������ ���� ��������� ��������� ���������� ������ ���� ����� ��� ��� ����� ���������� ������� ����������� ���� ���� ��� ���������������� ����������� ��� ������ ���������� ��� ����� ��� ��� ������ ��������� ��� �������� ������ ���� ��� ���� ���������������� ����������� ����� ����� ��� ������ ���������� �������� ����� 109 Deploying learning analytics for awareness and reflection - ARTEL15 �������������������������������������������������������������������������������������� ����������������������������������������������������������������������������������� ������� �� ������ ���� ������ ����� ������ ��� ���� ���������� ��������� ��������� ��� ���� ��� ����������� �������� ����������� �������� ����� ������ ��� ����������� ��� ���� �������� ��� ������������������������������������������������������������������������������������������� �������� ��� ��� ����������� ��� ����� ����������� ������� ���� ���������� ������� ���� ����� ������ ������ ��� �� �������� �������� ������ ��� ����� ��� �� �������� ����������� ������������ ���� �������������������������������������������������������������������������������� ���������������������������������������������������������������������������������� ���������������������������������������������������������������������� � � ���������������������������������������������������������������������������� �������� � ������������������������������������������������������������������������ �������� ���������������������������������������������������������������������������������� ������������������������������������������������������������������������������������� ��� ������������� ��� ������ ���������� ��� ���� ��� ���������� ����������� ������ ������� ����� ������� ��������� ��� �������� ��� ������ ��������� ��������� ���� �������� ��� ������ ����� ��� �������� ���� ���������� ��� ������������� ��� ������ ��������� ����� �������� ��� ����� ��� ��� ���������� ������ ���������������� �������� ����� ������ ��� ������ ������� �������������� ����������������������������������������������������������������������������������������� ��������������������������������������������������������������������������������������� ��������� ����� ��� ����������� ������������� ���������� ���� ����� �������� ���������� ���������� ��� ������������� ��� ������ ��������� ��� �� ������ ��� ������������ ������ ����������������������� ������������������������������������������������������������� �� ������������������������������������ 110 Deploying learning analytics for awareness and reflection - ARTEL15 ���������������� ����� ������ ���������� ���� ������ ��������� ���� ������� ����������� ����������������� ������ ��� ������ ��� ���� ���� ��� �������� ���� ������������ ��������� ����������� ��������� �������������������������������������������������������������������������������������� �������� ���� ��������� ��� ������ ��� ����������� ���� ��������� �������� ����������� ���������������������������������������������������������������������������������� �������� ���� ��� ���������� ����� ��������� ���� ����������� ��� ������������ ��� ���� ������ ��������� ���������� ������ ���� ��������������� ��� ��������� ��������� ��� ���� ����� ��� ���������� ���� ������� ��������� ���� ���������� �������������� ��� ���� ��� ������������� ����������������������������������������������������������������������������������������� ������� ������������������������������������ � ����������������� ����� ����� ���� ��������� �������� ����� ���� ��������� �������� �������� ���������� ���������� ���� ���������� �������������� ������������ ���� ��������������������������������������������������������� ����������� ���� �������� ��� ���� ������� ��� ���� ���� ��������� ��� ��� �� ������� ��� ���� ����� ������������ ���������� ������ ���������� �������������� ������������ ���� ���� ��������������� ���� ������������������������������������������� ���� ������ ��� �������� ����� ��������� �� ��������� ������ ��� ��������������� ������� ��������������������������������������������������� ���� ����������������������������������������������������������������������������������������� �������� ����� ����������� ��� ��������� �������������� ������ ��� ���� ������� ��� ���� ������ ������������������������������������������������������ ���� ���������� ��� ��������� ����������� ��������� ������������� ���� ������������ �������������� ������������������������������������������������������������������ ���� ��������� ���� ������� ���� ����������� ���� ���� �������� ��� �������� ����� ���� ������� ������ ���������� ����������� ���� ���������� ��� ������������ ��� ���� ����� ������� ������������ ������������������������������������������������������������������ ���� ������������� ��� ���������� ��� �������������������� �������� ��� ������������������� ���������������������������������������������������������������������� ���� ������������������������������������������������������������������������������������������ ������������������������� ������������������������ ���� ��������������� ��� ���� ���������� ���� ��������� ���� ���������� ���� ���� ��������� ��� ��� ������������� �������� ���������� ��� ������� ������� ���������������� ��� ������� ������������������� ����������� ��� ������������ ��� ���� ����� ������� ������������ ������������������������������������������������������������������ ���� ��������� ��� ���� �������� ��� ��������� ����������� ����������� ���� ���������� ��������� ���������������������������������������������������������������������� ����� ������������������������������������������������������������������������������������������ ������������ ��� ���� ����� ���������� �������� ���� ��������� ���� ���������� ������������ ������������������������������� � � 111