Please refer to these proceedings as Milos Kravcik, Alexander Mikroyannidis, Viktoria Pammer, Michael Prilla, Thomas Ullmann, Fridolin Wild (Eds.): Proceedings of the 4th Workshop on Awareness and Reflection in Technology Enhanced Learning. In conjunction with the 9th European Conference on Technology Enhanced Learning: Open Learning and Teaching in Educational Communities. Graz, Austria, Septem- ber 16, 2014. Available online at http://ceur-ws.org/. c 2014 for the individual papers by the papers’ authors. Copying permitted for private and academic purposes. Re-publication of material from this volume requires permission by the copyright owners. The front-cover was created by Harriett Cornish (The Open University, KMi). Addresses of the editors: Thomas Daniel Ullmann Alexander Mikroyannidis Knowledge Media Institute Knowledge Media Institute The Open University The Open University Walton Hall Walton Hall Milton Keynes Milton Keynes MK7 6AA MK7 6AA United Kingdom United Kingdom t.ullmann@open.ac.uk alexander.mikroyannidis@open.ac.uk Viktoria Pammer Fridolin Wild Knowledge Technologies Institute Knowledge Media Institute Graz University of Technology The Open University Inffeldgasse 21A Milton Keynes 8010 Graz MK7 6AA Austria United Kingdom viktoria.pammer@tugraz.at fridolin.wild@open.ac.uk Michael Prilla Milos Kravcik Institute for Applied Work Science RWTH Aachen University Ruhr University of Bochum Advanced Community Information Systems (ACIS) Universitaetsstr. 150 Ahornstr. 55 44780 Bochum 52056 Aachen Germany Germany michael.prilla@rub.de kravcik@dbis.rwth-aachen.de Awareness and reflection in technology enhanced learning Awareness and reflection can be viewed from the differing perspectives of the disciplines informing Technology-Enhanced Learning, such as Psychology, Educational and Learning Sciences, or Computer Science. A common denominator can be identified, though, and enhancing ’awareness’ of learners and other participants involved in learning processes by technology means augmenting for- mal or informal learning experiences, typically in real-time, with information on progress, presence, outcomes, workspace, and the like. Supporting ’reflection’ then means enabling learners to capture, adapt, re-evaluate, and share experience in anticipation of future situa- tions it will prove relevant to. Reflection supported digitally is a creative act, adding sense and meaning to experiences made. Combining support for ”awareness” and ”reflection” bears huge potential for improving the learning and training with respect to utility, self-regulation, usability, and user experi- ence. The ARTEL workshop series brings - for the 4th time in 2014 - together researchers and professionals from different backgrounds to provide a forum for discussing the multi- faceted area of awareness and reflection. For this year 2014, the workshop organizes discussion and meta-reflection amongst re- searchers around the application of awareness and reflection in practice, its impact on learners and questions of feasibility, and sustainability for awareness and reflection in ed- ucation and work. This year’s workshop theme is: How does computer-support for awareness and reflection need to be embed- ded into practical (working or learning) contexts in order for learners to ben- efit from such computer support? Summary of the contributions The #ARTEL14 workshop accepted 4 full papers, 1 short paper, and 4 demo papers. The accepted papers discuss awareness and reflection in diverse settings, such as blue-collar jobs and white-collar jobs, working in small enterprises, or learning at university level. As for the full papers, Maurizio Megliola, Gianluigi Di Vito, Roberto Sanguini, Fridolin Wild, and Paul Lefrere discuss in ”Creating awareness of kinaesthetic learning using the Experience API: current practices, emerging challenges, possible solutions” an interface specification for capturing in particular kinaesthetic learning experiences. The authors also discuss a taxonomy of verbs describing handling and motion. Kinaesthetic skills are in demand for instance in the manufacturing or maintenance sectors. The captured learning experiences can be utilised to generate feedback to the learner. Fridolin Wild, Peter Scott, Paul Lefrere, Jaakko Karjalainen, Kaj Helin, Ambjorn Naeve, and Erik Isaksson situate their work ”Towards data exchange formats for learning expe- 3 Awareness and reflection workshop series riences in manufacturing workplaces” in the manufacturing sector. This work models the experiential and reflective learning process with a view on creating data exchange for- mats, also discussing the necessity to - formally - talk about actions in the real world as a complement to actions performed in virtual environments. Andreas Janson, Sissy-Josefina Ernst, Katja Lehmann, and Jan Marco Leimeister have as background for their work in ”Creating awareness and reflection in a large-scale IS lecture - the application of a peer assessment in a flipped classroom scenario” univer- sity learning in large-scale lectures (much more learners than instructors). They discuss computer-supported peer assessment as possibility to induce reflection on learning con- tent. In contrast to the previous two works, this contributions focusses mostly on cognitive instead of motoric competences. Angela Fessl, Gudrun Wesiak, and Granit Luzhnica showcase in ”Application overlapping user profiles to foster reflective learning at work” the potential benefits of collecting activ- ity logging data from multiple applications in a single user profile application. This work is set on the background of white-collar knowledge workers, with desktop- and web-based activity logging. In their short paper, Michael Prilla, Oliver Blunk, Jenny Bimrose, and Alan Brown discuss in ”Reflection as support for career adaptability: A concept for reflective learning in public administration” reflection as learning mechanism to support professional identity change as a means for organisational change in the context of public employment services. The four demo papers are started off by the contribution by Milos Kravcik, Kateryna Neulinger, and Ralf Klamma, in which the authors showcase widget-based personal learn- ing environments for ”Boosting informal workplace learning in small enterprises”. Nils Faltin, Simon Schwantzer, and Margret Jung present the ”Activity recommendation app - software to evaluate the usefulness of improvement recommendations created in a team”. Min Ji, Christine Michel, Elise Lavoue, and Sebastien George demonstrate the ”DDART: an awareness system to favor reflection during project-based learning”. Last, but not least, Sven Charleer, Jose Luis Santos, Joris Klerkx, and Erik Duval present the ”LARAe: Learning Analytics Reflection & Awareness environment”. Awareness and reflection workshop series The official workshop webpage can be found at http://teleurope.eu/artel14 The 4th Workshop on Awareness and Reflection in Technology-Enhanced Learning (AR- TEL 2014) is part of a successful series of previous workshops. • 3rd Workshop on Awareness and Reflection in Technology-Enhanced Learning (AR- TEL13). Workshop homepage: http://teleurope.eu/artel13. Proceed- ings: http://ceur-ws.org/Vol-1103/. 4 Awareness and reflection workshop series • 2nd Workshop on Awareness and Reflection in Technology-Enhanced Learning (AR- TEL12). Workshop homepage: http://www.teleurope.eu/artel12. Pro- ceedings: http://ceur-ws.org/Vol-931/. • 1st European Workshop on Awareness and Reflection in Learning Networks (AR- Nets11). Workshop homepage: http://teleurope.eu/arnets11. Pro- ceedings: http://ceur-ws.org/Vol-790/ • Augmenting the Learning Experience with Collaboratice Reflection (ALECR11). Workshop homepage: http://www.i-maginary.it/ectel2011/index. html • 1st Workshop on Awareness and Reflection in Personal Learning Environments (ARPLE11). Workshop homepage: http://teleurope.eu/arple11. Pro- ceedings: http://journal.webscience.org/view/events/The_PLE_ Conference_2011/paper.html#group_Proceedings_of_the_1st_ Workshop_on_Awareness_and_Reflection_in_Personal_Learning_ Environments To stay updated about future events, to share your research, or simple to participate with other researchers, consider joining the group about Awareness and Reflection in Technology- Enhanced Learning: http://teleurope.eu/artel We especially would like to thank the members of the programme committee for their invaluable work in scoping and promoting the workshop and quality assuring the contri- butions with their peer reviews. September 2014 Milos Kravcik, Alexander Mikroyannidis, Viktoria Pammer, Michael Prilla, Thomas Ullmann, Fridolin Wild 5 Organisation committee Organisation committee Milos Kravcik, RWTH Aachen University, Germany Alexander Mikroyannidis, The Open University, United Kingdom Viktoria Pammer, Graz University of Technology, Austria Michael Prilla, University of Bochum, Germany Thomas Ullmann, The Open University, United Kingdom Fridolin Wild, The Open University, United Kingdom 6 Program committee Program committee Alexander Nussbaumer, Graz University of Technology, Austria Carsten Ullrich, DFKI, Germany Denis Gillet, EPFL, Switzerland Ines Di Loreto, Université de Technologie de Troyes, France Jaakko Karjalainen, VTT, Finland John Cook, University of West of England, United Kingdom Joris Klerkx, Katholieke Universiteit Leuven, Belgium Kaj Helin, VTT, Finland Kinshuk, Athabasca University, Canada Marcus Specht, Open University, the Netherlands Monica Divitini, IDI-NTNU, Norway Paul Lefrere, CCA, United Kingdom Peter Kraker, Know-Center, Austria Peter Sloep, Open University, the Netherlands Philippe Dessus, Universite Pierre-Mendes-France, France Riina Vuorikari, European Commission JRC IPTS, Spain Simon Knight, The Open University, United Kingdom Stefan Trausan-Matu, University Politehnica of Bucharest, Romania Stefano Bianchi, SOFTECO, Italy Xavier Ochoa, Escuela Superior Politecnica del Litoral, Ecuador 7 Supporting FP7 projects Supporting FP7 projects http://www.mirror-project.eu http://learning-layers.eu http://www.tellme-ip.eu http://wespot-project.eu http://employid.eu http://www.boost-project.eu 8 Contents Awareness and reflection in technology enhanced learning 3 Summary of the contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Awareness and reflection workshop series . . . . . . . . . . . . . . . . . . . . 4 Organisation committee . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Program committee . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Supporting FP7 projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Creating awareness of kinaesthetic learning using the Experience API: current practices, emerging challenges, possible solutions Maurizio Megliola, Gianluigi Di Vito, Roberto Sanguini, Fridolin Wild, Paul Lefrere 11 Towards data exchange formats for learning experiences in manufacturing work- places Fridolin Wild, Peter Scott, Paul Lefrere, Jaakko Karjalainen, Kaj Helin, Ambjorn Naeve, Erik Isaksson 23 Creating awareness and reflection in a large-scale IS lecture - the application of a peer assessment in a flipped classroom scenario Andreas Janson, Sissy-Josefina Ernst, Katja Lehmann, Jan Marco Leimeister 35 Application overlapping user profiles to foster reflective learning at work Angela Fessl, Gudrun Wesiak, Granit Luzhnica 51 Reflection as support for career adaptability: A concept for reflective learning in public administration Michael Prilla, Oliver Blunk, Jenny Bimrose, Alan Brown 65 Boosting informal workplace learning in small enterprises Milos Kravcik, Kateryna Neulinger, Ralf Klamma 73 Activity recommendation app - software to evaluate the usefulness of improve- ment recommendations created in a team Nils Faltin, Simon Schwantzer, Margret Jung 77 DDART: An awareness system to favor reflection during project-based learning Min Ji, Christine Michel, Elise Lavoue and Sebastien George 81 LARAe: Learning analytics reflection & awareness environment Sven Charleer, Jose Luis Santos, Joris Klerkx, Erik Duval 85 9 10 Creating awareness of kinaesthetic learning using the Experience API: current practices, emerging challenges, possible solutions Maurizio Megliola1, Gianluigi Di Vito1, Roberto Sanguini2, Fridolin Wild3, Paul Lefrere3 1 Piksel Ltd., Italy {gianluigi.divito,maurizio.megliola}@piksel.com 2 AgustaWestland, Italy Roberto.Sanguini@agustawestland.com 3 The Open University, Knowledge Media Institute, Milton Keynes, UK {fridolin.wild,paul.lefrere}@open.ac.uk Abstract. We describe our use of the Experience API in preparing blue-collar workers for three frequently arising work contexts, including, for example, the requirement to perform maintenance tasks exactly as specified, consistently, quickly, and without error. We provide some theoretical underpinning for modi- fying and updating the API to remain useful in near-future training scenarios, such as having a shorter time allowed for kinaesthetic learning experiences than in traditional apprenticeships or training. We propose ways to involve a wide range of stakeholders in appraising the API and ensuring that any enhancements to it, or add-ons, are useful, feasible and compatible with current TEL practices and tools, such as learning-design modelling languages. Keywords: Experience sharing, xAPI, verbs. 1 Introduction Apprenticeship today often includes the development of ‘kinaesthetic intelligence’, i.e. the tactile (physical) abilities associated with using the body to create (or ‘do’) something involving highly coordinated and efficient body movements. Prototypical examples of this can be found in the fluid and precise motions of skilled dancers, surgeons, and skilled blue-collar workers. Gardner (2011) links high bodily- kinaesthetic intelligence to “control of one's bodily motions, the capacity to handle objects skillfully, a sense of timing, a clear sense of the goal of a physical action, along with the ability to train responses”. Becoming a skilled kinaesthetic performer through traditional apprenticeship is to- day largely conceptualised to take years, but this golden age of long-cycle training is quickly disappearing. Professional training environments such as in manufacturing face the challenge that the world of repetition that enabled long cycles of training to be cost-justified is increasingly taken over by short-run or personalised production, 11 Creating awareness of kinaesthetic learning using the Experience API - ARTEL14 using advanced factory machinery that does not require physical dexterity (including teachable robots that can emulate dextrous production-line workers), thereby not only causing shifts in the demand for skills, but at the same time downsizing the economi- cally acceptable time-to-competence. Achievements in interoperability for technology-enhanced learning (TEL) over the last decade are making it viable to develop TEL-based alternatives to a traditional apprenticeship. For example, TEL is making it possible to author, exchange, and then orchestrate the re-enactment of learning activities across distributed tools, using learn- ing process and learning design modelling languages (Laurillard & Ljubojevic, 2011; Fuente Valentín, Pardo, & Degado Kloos, 2011; Mueller, Zimmermann, & Peters, 2010; Koper & Tattersall, 2005; Wild, Moedritscher, & Sigurdarson, 2008). What these modelling languages fall short of, however, is the ability to handle hy- brid (human-machine) experiences, to teach machines, or to train people at a distance – as required, for example, in workplaces that include robots or software agents, or in workplaces that include distributed participation across companies in a supply chain. In particular, when it comes to bridging between the virtual and the real world, be- tween digital and physical experiences, present means are ill equipped for supporting the capturing, codification, and sharing of hybrid learning experiences. Moreover, TEL-focused ways of capturing performance and collecting good prac- tice are as of today still resource-intensive, placing a barrier to the spread of innova- tion and hindering resilience of business. That barrier can be somewhat reduced through the use of the present version of the Experience API (ADL, 2013), and we claim could be further reduced through possi- ble extensions and complements to the API, the need for which has become apparent in the TELLME project. The Experience API is is a novel interface specification designed to link sensor networks together to enable the real-time collection and analysis of learning experi- ences conducted in different contexts, with the aim of providing better interoperability between the different types of participating educational systems and devices. Being precondition to mining and modelling, it forms a centrepiece in the canon of next generation techniques and technologies for capturing, codification, and sharing of hybrid learning experiences. This paper presents an overview of how the components and concepts of the Expe- rience API are integrated within the TELL-ME project to foster tracking and analysis of learning and training in three different manufacturing environments, namely Aero- nautics, Furniture, and Textiles. We describe within this contribution, how we deploy an open source Learning Re- cording Store to allow for collecting and reporting learning across the various compo- nents of the TELL-ME system. Moreover, starting with the Aeronautics industry, we define a taxonomy of verbs of handling and motion for capturing in particular kinaes- thetic learning experiences as required in the helicopter industry, when targeting learning about a defined helicopter model and the connected standard maintenance procedures. 12 Creating awareness of kinaesthetic learning using the Experience API - ARTEL14 2 TELLME Learning Locker The ADL Experience API (short ‘xAPI’)1, formerly known as TinCan API, is an extension of the Activity Streams2 specification, a format for capturing activity on social networks, created by companies like Google, Facebook, Microsoft, IBM etc., that allows for statements of experience to be delivered to and stored securely in a Learning Record Store (LRS). These statements of experience are typically learning experiences, but the API can address statements of any kind of experiences a person immerses in, both on- and offline. While the core objects of an xAPI statement (Actor, Verb and Object) derive from the core Activity Streams specification, the Experience API has many more defined constructs for tracking information pertinent for the learner (with captured results such as “score”, “success”, “completion”, “attempt”, and “response”), unlike the Ac- tivity Streams spec which focuses on the publisher. In its most basic application, the Experience API allows one system (the activity ‘provider’) to send a message (also known as the ‘statement’) to another system (aka the ‘Learning Record Store’) about something a user has done. Up until now, this process has mostly taken place inside the organisation’s Learning Management Sys- tem. Anything we wanted to track had to be built as part of the LMS functionality, or needed tailor-made integration. The Experience API now allows sending and receiv- ing data between systems about what someone has done in a more openly defined way (see Figure 1). Figure 1. The experience tracking before and after the advent of the xAPI. This is important, because via the Experience API, any system can send xAPI statements using a standard connection method. This process of sending xAPI state- ments can happen in systems behind the firewall or openly across the Internet using secure connections. 1 http://www.adlnet.gov/tla/experience-api/ 2 http://activitystrea.ms/ 13 Creating awareness of kinaesthetic learning using the Experience API - ARTEL14 Within TELL-ME, we have implemented an instance of the open-source Learning Locker3 Learning Record Store, a LAMP-based project using MongoDB, PHP, and AngularJS, to begin collecting learning activity statements generated by xAPI com- pliant learning activities and reporting on such data. The REST WS interface to the LRS was made available to be used by any compo- nent of the TELL-ME architecture to submit and retrieve xAPI statements. An exam- ple of such xAPI triple submission (with POST), using curl4, is shown below: curl -X POST --data @example.json -H "Content-Type: application/json" -- user 465ea716cebb2476fa0d8eca90c3d4f594e64b51:ccbdb91f75d61b726800313b2aa9f50 f562bad66 -H "x-experience-api-version: 1.0.0" http://demos.polymedia.it/tellme/learninglocker/data/xAPI/statements The example above is parameterized by a reference to a JSON file named exam- ple.json: it contains the actual xAPI statement in form of an actor-verb-object triple: { "actor":{ "objectType": "Agent", "name": "Gianluigi Di Vito", "mbox":"mailto:gianluigi.divito@piksel.com" }, "verb":{ "id":"http://activitystrea.ms/schema/1.0/watch", "display":{ "en-US":"Watched" } }, "object":{ "id":"http://tellme-ip.eu/media/video/152" } } Once statements are logged, they can be queried again, for example, as needed for constraint validation to check, whether a user actually performed a certain required action step – then requiring additional constraint checker components. 3 http://learninglocker.net/ 4 curl - command line tool for transferring data with URL syntax: http://curl.haxx.se/ 14 Creating awareness of kinaesthetic learning using the Experience API - ARTEL14 Figure 1. Learner interaction during maintenance of a helicopter tail rotor: ‘user cleaned corrosion inhibitors’. Same as any LRS implementation, the LearningLocker integrates data from multi- ple systems. The ‘Activity Provider’ can be any sort of system, where the user per- forms learning activity. Examples of such systems include search engines, social bookmark engines, proxy servers, social networking platforms, webinars, blogs, CRMs, as well as specific additional TELL-ME components. An example of the latter is the TELL-ME smart player application under development, which can drop state- ments about a user consuming a video (to the end or just the access of it). Another example is ARgh!, the platform for the delivery of context-based Augmented Reality content on mobile devices at the workplaces, which can drop statements about certain action steps being executed by the learner: Figure 1 shows an example of such user interaction with physical objects during training. Integrating the Experience API requires not only providing an LRS, but also defin- ing the set of ‘predicates’ for logging the experience statements in a meaningful way. The ADL features a standard list of commonly occurring verbs5. This list, however, is not bespoke to manufacturing and is restricted too much to generic, web-based learn- ing activity, excluding interaction with objects of the real world – as required for learning by experience in manufacturing. The same is the case for ADL’s extended list of “Proposed Verbs for the Experience API”6. Once data are mapped to the ele- ments of the Experience API statement (see Section 3), they can be captured in the learning record store. Figure 3 shows examples of such statements stored in the Learning Locker coming from the ARgh! trials. 5 http://adlnet.gov/expapi/verbs/ 6 http://bit.ly/1zGmAzn 15 Creating awareness of kinaesthetic learning using the Experience API - ARTEL14 Figure 2. Argh! statements stored in the Learning Locker The Learning Locker also allows creating personalized queries by means of its re- porting functions, combining data from both learning activities and workplace per- formance, allowing linking of learning activity to performance. One important aspect to be considered is the handling of security and the protec- tion of the privacy of learners. In the Experience API, authentication is tied to the user, not the content. The user can be any person or thing that is asserting the state- ment. The user can be a learner, an instructor, or even a software agent and it can authenticate with OAuth7, a commonly used access delegation mechanism employed by many big names such as Google, Facebook, Salesforce, etc. that eliminates the needs of sharing passwords between applications to exchange data. In this aim, the Learning Locker supports authentication and it is integrated with OAuth 2.0, exposing an API which allows 3rd parties to connect to the API via OAuth 2.0. 7 http://oauth.net/ 16 Creating awareness of kinaesthetic learning using the Experience API - ARTEL14 Figure 3. Learning Locker Reporting page 3 The statement vocabulary (Aeronautics industry) Each xAPI statement follows the syntax of providing a subject, predicate, and ob- ject. While subjects and objects can vary, predicates (the ‘verbs’) are ideally rather slowly changing and can be defined in advance. From a cognitive linguistic perspective, there is prior work on defining verbs of motion and handling – and clarifying their relation to the way humans’ cognitively process them. Roy (2005) explains how humans learn words by grounding them in perception and action. The presented theories are tested computationally by imple- menting them into a series of conversational robots, the latest of which - Ripley - can explain "aspects of context-dependent shifts of word meaning" that other theories fall short of. To construct a vocabulary of verbs that is widely understood, easy to learn, and natural in mapping, the work of Roy provides valuable insights: Roy postulates 17 Creating awareness of kinaesthetic learning using the Experience API - ARTEL14 that "verbs that refer to physical actions are naturally grounded in representations that encode the temporal flow of events" (p. 391). She further details that the grounding of action verbs follows the schema of specifying which force dynamics (out of a limited set) apply and which temporal Allen relations operate (p. 391). Any higher-level composition of action verbs, so Roy (p.392), can be traced back to and expressed in terms of these fundamental temporal and force relations. This provides an angle for defining and structuring the TELL-ME taxonomy of verbs of handling and motion: it provides a basis for ordering from fundamental to composite actions, also defining their similarities. Moreover, it offers insights on how to map these verbs back to perception: it provides a rationale for how many visual overlay elements are required for an augmented reality instruction to express a certain motion verb primitive. For example, a verb 'pick up' requires the overlay visualisation of a grabbing hand as well as a highlight of the object to be picked up, whereas the motion verb 'move' consists of both 'pick up' and 'put down' actions, thus requiring more visual elements to be specified. Palmer et al. (2005) further discuss the "criteria used to define the sets of semantic roles" for building verb classes. It provides insights into the argument structure of framesets (and so-called role-sets) of verb classes (Palmer et al., 2005, section 3). Chatterjee (2001) reviews the cognitive relationship between language and space. He refers to Jackendoff in postulating that the "conceptual structure of verbs decom- poses into primitives such as ‘movement’, ‘path’ and ‘location’ (p.57). From an augmented reality perspective, there is additional prior work of relevance. Robertson and MacIntyre (2009) provide a review of the state of the art in displaying communicative intent in an AR-based system. The proposed taxonomy, however, stays on a level of general applicability across all sorts of augmented reality applica- tions and is lacking the level of handling and motion required for a particular work- place, such as required in learning the maintenance of helicopters. The proposed cate- gories (called ‘style’ strategies) are ‘include’, ‘visible’, ‘find’, ‘label’, ‘recognizable’, ‘focus’, ‘subdue’, ‘visual property’, ‘ghost’, and ‘highlight’ (p.149f). The communi- cative goals signified by these styling operations for visual overlays are listed as ‘show’, ‘property’, ‘state’, ‘location’, ‘reference’, ‘change’, ‘relative-location’, ‘iden- tify’, ‘action’, ‘move’, and ‘enhancement’ (p.148). Other than the work in cognitive linguistics, here, the focus is clearly defined from a technical and not user angle: help- ing the user to identify or move an object is on the same level as setting labels. In the aeronautical field, maintenance operations must be carried out according to official documents issued by the design authority of the aircraft. Such document is called the Maintenance Publication. Usually, it is organised inside the Interactive Electronic Technical Publication (IETP). The AECMA S1000D (European Associa- tion of Aerospace Constructors S1000D) is an international standard for development of IETP, utilizing a Common Source Data Base (CSDB). The standard prescribes rules to name, define, and code everything that is necessary to carry out maintenance activities in terms of: • aircraft model; • systems and subsystems of the aircraft; • maintenance tasks; 18 Creating awareness of kinaesthetic learning using the Experience API - ARTEL14 • location of components; • tools; • additional documentations; • miscellaneous. Every maintenance task is identified by a unique code referring to the Task Cate- gory (e.g. ‘servicing’, ‘repairs’, ‘package’), the Maintenance Activity within the cate- gory (‘drain’, ‘fill’, ‘remove’, ‘clean’, etc.), and - finally - to the definition which gives the procedure and data necessary to carry out maintenance tasks on a specific system or component. The code allows retrieving both procedures and data necessary to e.g. fill containers with fuel, oil, oxygen, nitrogen, air, water, or other fluids. For the TELL-ME taxonomy of verbs of handling and motion the following pre- liminary list was compiled by pilot partners, see Table 1. The initial taxonomy pre- sents the verbs of handling and motion as required in the helicopter industry for a defined helicopter model and the connected standard maintenance procedures. All the actions are handled by the actor ‘certified staff’, which therefore has not been includ- ed in the table. Activity Verb Objects Operation Loaded Cargo Unloaded Cargo Servicing Filled Liquid or gas (fuel, oil, oxygen, nitrogen, air, water). Drained Liquid (fuel, oil, water). Released (pressure) Gas (oxygen, nitrogen). Lubricated System, equipment, component, or item. Cleaned Surface Applied (protection) Surface Removed (ice) Surface Adjusted System, equipment or component. Aligned System, equipment or component. Calibrated System, equipment or component. Inspected (keep Product, system, equipment or compo- serviceable) nent. Changed Liquid (fuel, oil, water). Gas (oxygen, nitrogen). Examination, tests Examined (visual) Product, system, equipment, component and checks equipment, component, or item. Tested (operation / System, equipment or component. function) Tested (function) System, equipment or component. Tested (structure) Structure Designed (data / System, equipment, or component. 19 Creating awareness of kinaesthetic learning using the Experience API - ARTEL14 tolerance check) Monitored (condition) Product, system, equipment or compo- nent. Disconnect, re- Disconnected Equipment, components, or items. move and disas- Removed Equipment, components, or items. semble proce- dures Disassembled Equipment, components, or items. Opened for access Panels or doors (engine bay doors, land- ing gear doors, etc.). Unloaded (download Items. software) Repairs and locally Added material Product, equipment, component or item. make procedures Attached material Product, equipment, component or item. and data Changed (mechanical Structure, surface. strength / structure of material / surface finish of material) Removed Material Repaired Damaged product, system, equipment or component. Assemble, install Assembled Equipment, components and items. and connect pro- Installed Equipment, components and items. cedures Connected Equipment, components and items. Closed after access Panels or doors (engine bay doors, landing gear doors, etc.). Loaded (upload Items. software) Package, handling, Removed from Products, systems equipment or compo- storage and trans- (preservation materi- nents, portation al) Put (in container) Item. Removed (from Item. container) Kept serviceable Products, systems, equipment, or com- (when in storage) ponents. Moved (when in stor- Products, systems, equipment, or com- age) ponents. Prepared for use Item (after storage) Table 1. Activities and their predicates (plus objects). 20 Creating awareness of kinaesthetic learning using the Experience API - ARTEL14 4 Conclusion and outlook In this contribution, we have presented an overview of how the concepts of Experi- ence API can be applied in a manufacturing environment, precisely in the Aeronautics industry. In the context of the TELL-ME project, three different taxonomies of verbs of handling and motion were prepared or are currently being prepared by the pilot partners, one of which has been described in this paper (from the helicopter industry). An instance on the open-source Learning Locker Learning Record Store was made available to any TELL-ME component, allowing them storing and retrieving xAPI statements about the workers activities. For the future development, we intend to add preliminary taxonomies for the other two industry sectors, further mature them, and address issues related to the analytics and statements interpretation, going further ahead in understanding data and how to find meaning from it (with the help of statistical analysis). In addition, we plan to explore how to extend (keyword: ‘constraint checker’) or complement the Experience API to handle emerging challenges such as the need to consider the ergonomics and learning needs of human-robot workspaces. This will involve conversations and col- laborations with multiple stakeholders, such as TEL vendors and users, standards bodies, and participants in robotics projects and projects involving or affected by automation – as envisioned for the Factory of the Future projects. Acknowledgements The research leading to the results presented in this contribution has received fund- ing from the European Community’s Seventh Framework Programme (FP7/2007- 2013) under grant agreement no 318329 (the TELL-ME project). The authors would like to express their gratitude to the partners who have been involved in the related research work in TELL-ME. References 1. Gardner, H. (2011): Frames Of Mind: The Theory of Multiple Intelligences, Basic Books, first published in 1983. 2. Laurillard, D.; Ljubojevic, D. (2011): Evaluating learning designs through the formal rep- resentation of pedagogical patterns, In: Kohls & Wedekind (Eds.): Investigations of E- learning Patterns: Context Factors, Problems, and Solutions, IGI global. 3. Fuente Valentín, L. de la; Pardo, A.; Delgado Kloos, C. (2011): Generic service integration in adaptive learning experiences using IMS learning design, In: Computers & Education, 57:1160-1170. 4. Mueller, D.; Zimmermann, V.; Peters, J. (2010): ISURE: Guideline for the integration of instructional models with open educational content in IMS Learning Design, deliverable d5.2 of the iCoper consortium. 5. Koper, R.; Tattersall, C. (2005): Preface, In: Koper, Tattersall (Eds.): Learning Design: A handbook on modelling and delivering networked education and training, Springer: Berlin. 21 Creating awareness of kinaesthetic learning using the Experience API - ARTEL14 6. Wild, F.; Mödritscher, F.; Sigurdarson, S. (2008): Designing for Change: Mash-Up Per- sonal Learning Environments, In: eLearning Papers, 9 (2008). 7. Dillenbourg, P. (2011): Design for Orchestration, In: Dillenbourg (Ed.): Trends in Orches- tration: Second Research & Technology Scouting Report, deliverable d1.5 of the STELLAR consortium. 8. ADL (2013): Experience API, version 1.0.1, Advanced Distributed Learning (ADL) Initia- tive, U.S. Department of Defense. 9. Roy, D. (2005): Grounding words in perception and action: computational insights, In: TRENDS in Cognitive Sciences, 9 (8):389-396. 10. Palmer, M.; Gildea, D.; Kingsbury, P. (2004): The Proposition Bank: An Annotated Cor- pus of Semantic Roles, In: Computational Linguistics, 31(1):71-106. 11. Chatterjee, A. (2001): Language and space: some interactions, In: TRENDS in Cognitive Sciences , 5(2):55-61. 12. Robertson, C.; MacIntyre, B. (2004): Adapting to Registration Error in an Intent-Based Augmentation System, In: Ong, Nee (Eds): Virtual and Augmented Reality Applications in Manufacturing, Springer: London. 22 Towards data exchange formats for learning experiences in manufacturing workplaces Fridolin Wild1, Peter Scott1, Paul Lefrere1, Jaakko Karjalainen2, Kaj Helin2, Ambjoern Naeve3, Erik Isaksson3 1 The Open University, Knowledge Media Institute, Milton Keynes, UK {fridolin.wild, peter.scott}@open.ac.uk 2 VTT Technical Research Centre of Finland, Human factors in complex systems, Tampere, Finland {jaakko.karjalainen, kaj.helin}@vtt.fi 3 Royal Institute of Technology (KTH), Stockholm, Sweden amb@csc.kth.se Abstract. Manufacturing industries are currently transforming, most notably through the introduction of advanced machinery and increasing degrees of au- tomation. This has caused a shift in skills required, calling for a skills gap to be filled. Learning technology needs to embrace this change and with this contri- bution, we propose a process model for learning by experience to understand and explain learning under these changed conditions. To put this process into practice, we propose two interchange formats for capturing, sharing, and re- enacting pervasive learning activities and for describing workplaces with in- volved things, persons, places, devices, apps, and their set-up. Keywords: Experience sharing, activity model, workplace model, awareness, augmented reality. 1 Introduction The European (and global) manufacturing industry is currently undergoing signifi- cant transformation and will continue to change over the coming years. Intrinsically, the increasing presence and ability of robots and advanced machinery in production lines with their enhanced senses and increased dexterity (Frey & Osborne, 2013, p.38) are what triggers this shift, bringing along rising degrees of computerisation of jobs (ditto) and allowing for delocalisation of production. Extrinsically, this transformation has started to cause a significant skills gap in the EU (and globally) with – on the one side – the highest overall unemployment rates observed in more than a decade (Eurostats, 2014; EC, 2013c, p. 2), especially amongst young people (EC, 2013b; EC, 2013d), and an ever increasing risk of redun- dancy for low and medium skilled workers in production. On the other hand, several hundred thousand jobs in the EU remain unfilled, as there is a shortage in highly skilled personnel in manufacturing (EC, 2013a, p.10). Forecasts predict that this skills gap is likely to widen in coming years up to 2020 23 Towards data exchange formats for learning experiences - ARTEL14 (McKinsey, 2012, p. 45). In fact, manufacturing is currently one of the three sectors hit most hard by this skills shortage in the EU (EC, 2013c, p. 5). Formal secondary and tertiary education haven’t managed to create and won’t succeed in producing the supply required, neither in numbers, nor with respect to matching skills profiles. Moreover, high attrition rates in education have further eroded the foundation. Technology enhanced learning has the potential to play an important role in over- coming this existing skills gap in manufacturing – when applied effectively and when motivating the development of competences in key areas required through the captur- ing and re-enactment of learning activities. Within this contribution, we first define a learning process model that is capable of integrating classical (learning content oriented) and novel pervasive (Augmented Reality and Internet of Things oriented) elements in learning at manufacturing work- places. From there, we introduce a proposal for an activity modelling language (activ- ityML) for representing activity descriptions required in augmented reality enabled learning experiences. Moreover and in section 4, we introduce the needed workplace modelling language (workplaceML), which can be used to describe the tangibles (things, places, persons), configurables (apps, devices), and triggers (detectables, overlays) of a particular workplace. We relate our work to precursors in Section 5 to then wrap up the paper with an outlook and open research challenges. 2 Process model of learning by experience New skills for new jobs not only demand an enhancement of the deep professional skills to achieve a ‘master level of performance’, but also necessitate development and upgrading of competence to innovate, for lifelong learning, and for learning through social interaction (Wild et al., 2013, p. 12f). Achieving a master level of performance and developing competence to innovate in the sense of building up “the ability to generate ideas and artefacts that are new, influential, and valuable” (FET, 2011) are – at least in manufacturing and at least for small and medium enterprises – very closely intertwined. Tacit Explicit Tacit Socialisation Externalisation Explicit Internalisation Combination Table 1. Knowledge conversion modalities. Similarly, the other two, namely lifelong learning and social learning competence, both pay tribute to the observation that “people carry and create knowledge” and that “any company knowledge management strategy must rely primarily on people, and support [of] the knowledge creation chain” (Krassi and Kiviranta, 2013, p.29). Both of them aim to facilitate “bi-directional tacit-explicit knowledge conversion” (Nona- ka, 1994, p.19) along the four modalities listed in Table 1: externalisation (tacit-to- 24 Towards data exchange formats for learning experiences - ARTEL14 explicit), internalisation (explicit-to-tacit), socialisation (tacit-to-tacit), and combina- tion (explicit-to-explicit). While ‘competences’ are typically defined to subsume knowledge, skills and other abilities, in the context of manufacturing – as the word already suggests – motoric and artistic skills require special attention. Kinaesthetic learning elements relate in manu- facturing environments to controlling own body movement and handling objects skil- fully and timely (cf. Gardner, 1984: bodily-kinaesthetic intelligence). With the rise of Wearable Computing, the Internet of Things, and Augmented Re- ality, capturing and observing kinaesthetic performance becomes possible in a funda- mentally different way, as, for example, pioneered in the fitness and health sector. Reflective learning processes that cater for kinaesthetic and non-kinaesthetic ele- ments can be broken down into five distinct process steps: enquire, mix, experience, match, optimise (Wild et al., 2013, p.29ff). The steps do not necessarily prescribe a single route and order, in which they should be taken, but are interconnected as indi- cated in Fig. 1: it is a cyclical model with built-in support for experience tracking, analytics, and guidance, supporting flexible mixes and dynamic optimization for on- and off-the-job workplace learning. Fig. 1 depicts the individual steps of this process model: at its core, blue-collar workers experience learning in an episodic way (on and off the job). Experiencing thereby relates to both re-enacting explicit learning activities as well as engaging in open innovation activities. Experiencing learning tightly interacts with enquiry: whenever novel needs arise or (wicked) problems are encountered on the job, the enquiry step supports the user in identifying relevant learning opportunities (such as gaps in knowledge, new learning opportunities arising, etc.). In parts, this relates to navigational positioning support in the workplace reference space’s skills taxonomy to clearly determine the competence sought after. This, however, also relates to supporting discovery beyond existing or – particularly relevant for SMEs – so-far tacit knowledge. Through tracking of experiences made, potential competence gaps (ignorance) can be uncovered, uncovering thereby either supported by the system in the matching step (see below; aiming to help unveil shortcomings the user is unaware of) or – proactive- ly, where awareness is given – through user enquiry. Once needs or problems are identified, the mixing step comes into play: here, the learner is supported in selecting relevant existing mixes or creating new and adapting existing mixes. While standard problems have standard solutions, smart factories enable their workers to rapidly compile mixes that satisfy needs, but at the same time ensure documentation of knowledge, where it is created. Such mix is essentially a serialized, activity-focused representation of the specific workplace and the jobs to be enacted within it, instantiating an abstract workspace to a level of concreteness where actions are named, locations resolved, and objects as well as tools uniquely identified. Moreover, the activity mix models validation constraints, by which the matching step can determine whether there is evidence that the user actually performed the action steps as required. Constraints model learning flows including exception han- dling. The constraint-matching step picks up on strategic performance indicators and their defined tolerance boundaries set at design time, and connects them to the ob- 25 Towards data exchange formats for learning experiences - ARTEL14 served operational performance as tracked by the experience step. Reports for per- formance analytics can be generated live, condensing performance records (from an xAPI endpoint; ADL, 2013) into comprehensive reports, potentially contrasting per- formance of individuals with (de-identified) benchmarks. Optimisations then take such analytics data and performance benchmarks to rec- ommend repetition, alternative resources, or even a change of path. BLUE%COLLAR%WORKER%% Traces% EXPERIENCE( %%Need,% ?% Enquire( %%Problem% Ac9vity% Mix( XML%incl.% Constraints% %Report,% Match( %Analy9cs% Sugges9on,% Op9mise( Recommenda9on% Fig. 1. Process model for learning by experience. The bi-directional conversion between tacit and explicit knowledge is modelled in the process loops between the big process step ‘experiencing’ and the smaller ones ‘enquiry’, ‘mix’, ‘match’, and ‘optimise’: explication converts tacit knowledge to explicit as indicated by the outputs ‘traces’, ‘needs’, ‘activity mixes’, ‘analytics re- ports’, and ‘recommendations’. When such outputs are used to scaffold a learning experience, they are internalised. Remixing combines existing knowledge, and track- ing and evidence recording helps with converting tacit to explicit knowledge. Moreo- ver, activity mixes can involve socialisation and social sharing. 3 Modelling activities A common representation format is key requirement for an efficient exchange of activity mixes. What makes it particularly challenging to define activity mixes in a process for learning by experience is that it requires not only orchestrating user inter- action across multiple devices within a single activity, but also integrating the track- ing of and reacting to user interaction across these devices and – even more so – their different sensors. Validating that the user actually did something (like moving to a particular location or like picking up a particular object in the real world), requires specifying validation constraints that can be checked and that express which soft- and 26 Towards data exchange formats for learning experiences - ARTEL14 hardware sensors have to pick up on what user (or app) behaviour. For this we pro- pose activityML (activity modelling language), an XML dialect. Fig. 2 provides an overview on the conceptual model of activityML. Fig. 3 adds an example activityML file. The root node is ‘activity’. Each activity needs to specify the URL of the workplace description file, a name, and the language (in addition to the unique activity id). The activity is then broken down into ‘action’ steps, each of them being a self-contained unit, describing ‘summons’ for action chaining, ‘constraints’ for action validation, and ‘messages’ for communication, as well as ‘instruction’ to be shown or the ‘app’ (widget or app) to be launched. Fig. 2. Conceptual model of the activityML. Moreover, styling information is linked using cascading style sheets over the action ‘type’ and the ‘device’ and its ‘viewports’ defined. Currently, there are three view- ports defined: ‘objects’, ‘actions’, and ‘reactions’. They refer to particular areas of the screen reserved for inserting actions and the related display data. Each ‘action’ has a ‘predicate’, which is the verb required for inserting trace state- ments to the xAPI (ADL, 2013) tracking endpoint. Each action can optionally specify a ‘location’, i.e. a defined ‘place’ of the workplace model, in which it shall happen. Chaining of actions is modeled through specifying for each action, which other ac- tions it ‘summons’ – either when launching the action (‘onEnter’) or when events are triggered (‘onTrigger’). The Boolean ‘removeSelf’ decides on whether the action is removed from the viewport or sustained when the summons are executed. A timer can be set to automatically trigger summons after a given interval (in milliseconds). Summoning an action twice will first show, then remove it from the viewport (hence ‘toggle’). Each summoned toggle specifies the ‘viewport’ in which it is toggling an action. The summons are also used to activate in a context-dependent way the ids of actions relevant for the next step. By using the id of a tangible (e.g. thing or person) as the id of an action step, for example, an object detection engine can trigger the launching (or termination) of the according action. 27 Towards data exchange formats for learning experiences - ARTEL14 The ‘constraints’ define, how the system can validate that certain user interaction and other observable conditions are given the way they were modeled. For example, as depicted in Fig. 3, a constraint of type ‘onEnter’ can be defined that checks wheth- er the learner has certain basic ICT skills. Constraints are specified in a given query language (in the example: SQL) and they define their own action branching for ‘on- Satisfied’ and ‘onViolated’ conditions. To enable communication between devices and to allow for communicating with overlays (as specified in the workplace model), ‘messages’ can be used: each ‘mes- sage’ specifies, which ‘target’ (device, thing, person, …) and ‘id’ they want to com- municate with. In case that the message is to a thing, the ‘overlay’ needs to be speci- fied. Messages can also declare explicitly, which communication ‘channel’ they want to use (e.g. a real-time presence channel ‘rpc’ or an ‘xapi’ endpoint). Fig. 3. Example activityML code. Fig. 3 provides an example mock activity model. In this code example, the activity is broken down into five action steps, four of which are to be executed on a tablet PC, while one will be launched on a pair of augmented-reality (see-through) glasses. The user interaction starts with a welcome instruction to check the manufacturing order 28 Towards data exchange formats for learning experiences - ARTEL14 (auto-removed after 500ms) in which a constraint validation is performed checking whether the user has the required skills (from a user profile). If this constraint is validated, the user proceeds to finding the order sheet; other- wise an error message is displayed. To support finding the order sheet object, an ac- tion is launched on the glasses, which is toggled with sending the according id (‘ac- tion15’). Once the object has been found in the viewfinder of the glasses, the next action – playing multimedia instructions in the smart player app – follows, this now again on the tablet PC. 4 Modelling workplaces To create interoperability of applications interpreting activityML, a description of the workplace is required in a defined interchange format. We propose for this work- placeML, an XML dialect to describe the tangibles, configurables, and basic triggers of a workplace. The ‘tangibles’ thereby refer to ‘things’, ‘places’, and ‘persons’, see Fig. 4. The ‘configurables’ fall into two classes, namely ‘devices’ and ‘apps’. Fig. 4. Conceptual model of workplaceML. Finally, the ‘triggers’ group together both ‘detectables’ (such as markers) and the primitives of ‘overlays’. The relationship between tangibles and triggers is crucial: each tangible can specify the corresponding ‘detectable’ to determine how it can be detected: it can name the id of a marker, the id of a feature cloud, or even the id send 29 Towards data exchange formats for learning experiences - ARTEL14 by an Internet of Things component such as an intelligent toolbox that monitors through sensors which tools are taken out or put back in. Moreover, each tangible can list the overlay primitives supported and configure them if required. For example, a ‘YesNo’ visual overlay does not require additional configuration: an app identifying the tangible will automatically overlay a green circle when it is relevant to the current action step (and a red cross, when not). This is different, for example, for an image overlay primitive: in that case, the tangible needs to specify via ‘src’ the path to the image to be displayed (and whether it shall be anchored to the detectable or to the horizon). Fig. 5. Example of a workplaceML file. 30 Towards data exchange formats for learning experiences - ARTEL14 Certain verbs of handling and movement can be predicates of an action step (e.g. ‘lift’ or ‘rotate’) and they overlay primitives can be enabled accordingly in the con- figuration of the tangible. The ‘configurables’ specify for each device the ‘owner’, a human-readable ‘name’, the ‘type’ (e.g. ‘iPad mini’ versus ‘google glass’) and its unique id. This ensures that messages can be delivered and actions can be launched and styled correctly. The ‘apps’ define the URL of the manifest file of e.g. a ‘widget 1.0‘ compliant or ‘Open- Social’ compliant widget. The code example presented in Fig. 5 now provides the required workplace infor- mation on the tangibles (places, things, persons), configurables (apps, devices), and triggers (detectables, overlays). From the bottom to the top, first the overlay primitives are described, i.e. a generic definition of which types of overlays exist and which modality they are overlaid in. For example, there is a person sound and there is an image overlay. Next, the detectables are defined: this enables a pre-trained marker (‘010’), a fea- tureless object model (‘015’), and an event from an Internet of Things sensor (‘020’). There are two types of configurables defined in this example: the devices (e.g. of type ‘ipad’ or ‘moveriobt200’) as well as the apps that can be launched, some of which through calls to the device app to be launched, others as html5 widgets. Finally, definitions of persons, places, and things follow. Here, each tangible can further configure the overlay primitives described at the very end of the script. For example, the thing ‘thehammer1’ is bound to the marker ‘010’ and configured to sup- port image overlays using a picture of the hammer and setting the xyz-offsets as re- quired. 5 Related work In Naeve et al. (2014), we have presented generic, complementary deliberations about workplace models as well as an earlier, less elaborate version of the interchange format for activities proposed (p.48ff). ACTIVITY-DL (Lanquepin, 2013; Barot et al., 2013) builds on the former HAWAI-DL proposal of the same group and provides a hierarchical way to describe tasks for virtual reality environments. While the task description is very advanced, the language lacks capabilities for device and multi-sensor integration. ACTIVITY-DL refers to its precursors MAD (Methode Analytique de Description; Sebillotte & Scapin, 1994) and GTA (Groupware Task Analysis; Veer et al., 1996), both focusing on analysing work tasks in interaction with user interfaces. While both provide con- ceptual insights (e.g. on timing and on condition modelling), they do not provide bindings against an interchange format. 6 Conclusion and outlook In this contribution, we have rooted our motivation for creating the required ex- change formats for capturing and sharing (kinaesthetic) learning experiences in manu- 31 Towards data exchange formats for learning experiences - ARTEL14 facturing workplaces. The transformation the industry is currently undertaking has left a skills gap, which can be closed using learning technology apt to capture, share, and guide in re-enacting innovative production activity. For this, we have described the learning process and proposed two novel interchange formats for exchanging execut- able descriptions of learning by doing activity and workplaces. The exchange formats are implemented in the ARgh! prototype, a first glance of which is published in the proceedings of the main conference (Wild et al., 2014). In a world, where the time required for updating must be significantly smaller than the half-life of knowledge documented, this becomes a key enabler for experience sharing and a corner stone for success. The specifications have already been tested against a range of storyboards of the TELL-ME project and with participants of the joint European doctoral summer school in TEL (JTEL’14). The upcoming user pilots in the TELL-ME are expected to lead to further refinements. In particular, work is undergoing at the moment to further refine the predicate vocabulary and fine-tune it to the three pilot workplaces tested (aviation, furniture production, textile inspection and production). Moreover, the xAPI integra- tion already feeds back to the constraint validation and further updates on query lan- guage and reasoning are to be expected. Acknowledgements The research leading to the results presented in this contribution has received fund- ing from the European Community’s Seventh Framework Programme (FP7/2007- 2013) under grant agreement no. 318329 (the TELL-ME project, http://tellme-ip.eu). The authors would like to express their gratitude to the partners who have been in- volved in the related research work in TELL-ME. References 1. Frey, C.; Osborne, M. (2013): The future of employment: how susceptible are jobs to computerisation?, unpublished manuscript, University of Oxford: Oxford Martin School. 2. European Commission (2013a): Annual Growth Survey 2012, COM(2011) 815 final. 3. European Commission (2013b): Europe 2020 Factsheet on Youth unemployment, online at: http://ec.europa.eu/europe2020/pdf/themes/21_youth_unemployment.pdf 4. European Commission (2013c): Draft joint employment report 2014 (JER), COM(2013) 801 final, online at: http://ec.europa.eu/europe2020/pdf/2014/jer2014_en.pdf 5. European Commission (2013d): Working together for Europe’s young people: A call to action on youth unemployment. COM(2013) 447 final. 6. Eurostats (2014): Unemployment statistics, online at: http://epp.eurostat.ec.europa.eu/statistics_explained/index.php/Unemployment_statistics 7. McKinsey Global Institute (2012): The world at work: Jobs, pay, and skills for 3.5 billion people, June 2012, McKinsey. 8. Wild, Fridolin; Scott, Peter; Da Bormida, Giorgio; Lefrere, Paul; Naeve, Ambjoern; Isaks- son, Erik; Valdina, Alessandro; Nina, Manuel; Marsh, Jesse (2013): Learning By Experi- ence: The TELL-ME Methodology (phase one), deliverable d1.2, TELL-ME consortium. 32 Towards data exchange formats for learning experiences - ARTEL14 9. FET (2011): Creativity and ICT, FET Consultation Workshop, Brussels, 28 November 2011. Report dated December 2011. 10. Krassi, Boris; Kiviranta, Sauli (2013): The unity of human and machine, In: VTT Impulse: Science, 2013(1): 24-29. 11. Nonaka, I. (1994) A Dynamic Theory of Organizational Knowledge Creation, Organiza- tion Science, Vol. 5, No. 1 (Feb., 1994), pp. 14-37. 12. Gardner, H. (1984): Frames of mind: the theory of multiple intelligences, Heinemann. 13. ADL (2013): Experience API, version 1.0.1, Advanced Distributed Learning (ADL) Initia- tive, U.S. Department of Defense. 14. Naeve, A.; Isaksson, E.; Lefrere, P.; Wild, F.; Tobiasson, H.; Walldius, Å; Lantz, A.; Vii- taniemi, J.; Karjalainen, J.; Helin, K.; Nuñes, M.J., Martín, J. (2014): Integrated industrial workplace model reference implementation, deliverable d4.3, TELL-ME consortium. 15. Lanquepin, V.; Carpentier, K.; Lourdeaux, D.; Lhommet, M.; Barot, C.; Amokrane, K. (2013): HUMANS: a HUman Models based Artificial eNvironments Software platform, In: Proceedings of Laval Virtual VRIC’13, March 20-22, 2013 Laval, France, ACM. 16. Barot, C.; Lourdeaux, D.; Burkhardt, J.-M.; Amokrane, K.; Lenne, D. (2013): V3S: A Vir- tual Environment for Risk-Management Training Based on Human-Activity Models, In: Presence, Vol. 22, No. 1, Winter 2013, 1–19 17. Sebillotte, S.; Scapin, D. (1994): From users’ task knowledge to high‐level interface speci- fication, International Journal of Human-Computer Interaction, 6:1, 1-15. 18. Veer, G.; Lenting, B.; Bergevoet, B. (1996): GTA: Groupware Task Analysis - Modeling Complexity, In: Acta Psychologica 91(1996):297-322. 19. Wild, F.; Scott, P.; Karjalainen, J.; Helin, K.; Lind-Kohvakka, S.; Naeve, A. (2014): An augmented reality job performance aid for kinaesthetic learning in manufacturing work places, In: Open Learning and Teaching in Educational Communities, Proceedings of EC- TEL 2014, Springer: Berlin. 33 34 Creating Awareness and Reflection in a Large-Scale IS Lecture – The Application of a Peer Assessment in a Flipped Classroom Scenario Andreas Janson1, Sissy-Josefina Ernst1, Katja Lehmann1, Jan Marco Leimeister1,2 1 Kassel University, Germany {andreas.janson, sissy.ernst, katja.lehmann, leimeister}@uni-kassel.de 2 University of St.Gallen, Switzerland JanMarco.Leimeister@unisg.ch Abstract. Large-scale lectures are a typical way of teaching university students. However, these lectures often lack interaction elements and do not foster awareness and reflection in the learning process. This results in insufficient learning outcomes such as learning satisfaction and success. Therefore, a new approach to engage interaction in such large-scale lectures is the flipped class- room concept which seeks to overcome these challenges by stimulating self- regulated learning phases and improving interaction as well as awareness and reflection in the presence phases of a lecture. However, it is still unclear how to actually increase reflection and awareness through interaction in such learning scenarios. For this purpose, we propose an application of a technology- enhanced peer assessment that is carried out in large-scale information systems lectures. Preliminary evaluation results suggest the potentials of this approach. Thus, we are able to provide first theoretical and practical implications for the application of a technology-enhanced peer assessment in large-scale lectures. Keywords: Awareness, Reflection, Peer Assessment, Large-Scale Lectures, Learning Success, Interaction, Feedback, Educational Objectives 1 Introduction Large-scale lectures with an uneven lecturer-learner proportion (sometimes more than 100 learners per lecturer) are common in learning scenarios of universities [1]. These lectures are characterized by high anonymity and suffer from a lack of interaction in the learning process - not only among learners themselves but also among learners and lecturers [2]. Often, this results in insufficient learning outcomes and brings about unsatisfied learners [3, 4]. This development is alarming since fundamental elements of learning success include the opportunity to ask comprehension questions in order to get feedback, the possibility of sharing one's opinions concerning the learning con- tent and of intensively reflecting on the learning content with colleagues [5, 6]. 35 Creating awareness and reflection in a large-scale IS lecture - ARTEL14 Moreover, dealing and interacting with the learning content during the learning pro- cess creates awareness and reflection regarding the learning process [7]. Additionally, interaction and collaborative learning with peers are regarded as significant predictors in terms of learning success [8] and positively influence the long-term satisfaction of learners [9, 10]. Individual learning success verification, namely in the teaching- learning process, provides individual feedback to learners [11]. This allows learners and lecturers to identify missing knowledge and misunderstandings not during the final exam, but rather early in the course of a continuous learning-progress monitoring system [11] and moreover, create awareness for the relevant specific learning content. Integrating assignments in class which create awareness and reflection to the specific learning content are very complex and addresses the high cognitive level of educa- tional objectives [12] supposed by Bloom [13] and Anderson et al. [14], which are analyzing, evaluating and creating. However, the verification of those assignments is time- and resource-consuming hence impossible to use in a large-scale lecture. Never- theless, introducing interaction and feedback to create awareness and reflection and moreover addressing educational objectives on a high cognitive level for individual learning success measurement in a large-scale lecture is a widespread problem. Didactic mechanisms are needed in order to overcome the above mentioned factors characterizing traditional large-scale lectures. One promising possibility to enhance interaction and feedback and moreover to address high cognitive levels of educational objectives without massively increasing the workload of lecturers is the use of peer assessment as didactic method [15]. By using peer assessment, learners give each other feedback or credit points in terms of a performance during the learning process according to specifically defined criteria. The goal of this paper is to describe the use of peer assessment as interaction supporting component for addressing awareness and reflection in a university large-scale lecture and ultimately for increasing learning success. This paper therefore aims to answer the research question: How is a peer assessment in a large-scale lecture designed to address interaction and to improve the learning scenario? The contribution of this study is according to Gregor [16] a theory of design and action that enables on the one hand practitioners to design learning scenarios with a technology-enhanced peer assessment, and one the other hand de- rives theoretical implications for future research in engineering IT-enabled learning scenarios. In order to answer the research question, the remainder of this paper is structured as follows. First, we provide a brief overview of related work that is concerned with our peer assessment. We then subsequently propose our application of the technolo- gy-enhanced peer assessment in our learning scenario. Afterwards, we present our first evaluation results and provide implications of our results in the discussion. In section 6, we highlight limitations of our study and provide on this basis guidance for future research, before the paper closes with a brief conclusion. 36 Creating awareness and reflection in a large-scale IS lecture - ARTEL14 2 Related Work For a few years, awareness and reflection have been, in the context of technology- enhanced learning, increasingly important and capture more attention since it has been recognized that both are key factors in helping to provide personal support in user-centric learning environments [17, 18]. The pedagogical approach aims at making learners aware of their learning behavior and at the same time intends to empower learners in creating own personal learning environments with individual learning resources while discovering their own learning patterns. 2.1 Awareness In this context, awareness plays a central role focusing on cognitive learning activities and especially on non-observable behavior. Learners should familiarize themselves with their own and individual cognitive processes such as goal-setting, self-evaluation or help-seeking, in order to integrate them into self-regulated learning [19]. Students are confronted and made aware of key actions of their own learning behavior with the intention to make them become aware of their cognitive actions. Thereby, e-learning tools with the possibility to personalize the learning process are able to support awareness. Evaluations indicate that learners feel aware especially of their own efforts and less about the effort of their group members and the members of other groups [20]. 2.2 Reflection Reflection is an important key element in the learning activity as it allows implement- ing continuous improvement in order to cope with complex and permanent changing situations [21]. It is a meta-cognitive process which can be individual and also collec- tive [22], and described as the conscious reevaluation of experience for the purpose of guiding future behavior taking into account feeling, ideas and behavior as well [23]. In the context of technology-based learning, active reflection supports the examina- tion of own achievements as well as the work of peers and pushes for a decentraliza- tion process of problem-solving where learners are challenged and confronted with existing knowledge [21] and finally able to create knowledge [24]. Finding out about a learner’s reflection can be supported by several platforms where learners communicate by sharing reports, problems and solutions concerning their work with peers [25]. Additionally, this exchange enables peers to learn from their peers and at the same time to contribute own work. In this way, learners should take more responsibility of their learning activities and efforts. 37 Creating awareness and reflection in a large-scale IS lecture - ARTEL14 2.3 Peer Assessment In the context of awareness and reflection in the learning process, prior research has shown that learners who interact with their lecturers and colleagues are more actively involved in the learning process [26, 27] and achieved better learning outcomes [8]. The lecturer can assess the learning progress by means of the answers and provide direct feedback. The learners have the opportunity to contribute their ideas and thoughts, thus, also initiating new thought processes [28, 29]. The use of peer assessment in class is an essential possibility to introduce interac- tion in a large-scale lectures and to provide formatively individual feedback in the learning progress as well as corresponding interventions by means of technical-based observation processes even in groups with a high number of learners [30, 31]. Moreo- ver, the use of peer assessment is a favorable method to give learners extensive open- ended free text assignments hence to address awareness and reflection, even in large- scale lectures with more than 100 students, without massively increasing the lectur- er’s workload. In the case of peer-assessment, learners give each other feedback or credit points in terms of a performance or results during the learning process accord- ing to specifically defined criteria [32]. Peer-assessment turns learners into experts themselves and gives them a deeper understanding of the learning content [33]. The application of peer assessment in university teaching brings about, above all, the following advantages opposed to an evaluation solely done by the lecturer: 1. Logistically: Lecturers can save precious time if learners give each other feedback and evaluate each other’s academic performance [33]. 2. Pedagogically: The learners get a deeper understanding of the learning contents by checking and assessing their colleagues’ responses. By reading works of others, one can deepen one’s own knowledge and develop new ideas by evaluating other points of view [13, 33]. 3. Metacognitive: Learners will develop awareness for their own strengths and weak- nesses and will be able to compare and evaluate their own performances, at least to a certain extent [34]. In addition, learners train their abilities to think critically [35, 36] as well as how to evaluate and reflect [37]. 4. Affectively: Learners perceive qualitative feedback from their peer group as more valuable than a lecturer’s grade [33]. Therefore, the application of peer assessment does not only relieve the lecturer but turns learners into experts themselves. First observations show that evaluations done by the peer group agree with the lecturers’ evaluations of the learners’ academic per- formances [38]. Furthermore, studies show that regular feedback given by the peer group has a positive effect on the learner’s learning process [39]. In their literature overview, van Zundert et al. [40] point out that there are only a few existing case studies concerning an experimental setting of peer assessment and that this circum- stance prevents specific insights on how peer assessment has to be designed. Scien- tific literature brings up terms such as peer assessment, peer grading, peer review, and peer feedback, among others. For this paper, we use the term of peer assessment 38 Creating awareness and reflection in a large-scale IS lecture - ARTEL14 meaning that learners of a peer group assess each other’s performances as well as evaluate it according to relevant criteria without giving each other credit points. 3 Theory-motivated Design of a Peer Assessment For the improvement of our learning scenario, we draw on a theory-motivated design approach [41, 42] for engineering learning services [43]. Therefore, we base our sub- sequent design decisions on the constructs linked to our phenomena of interest. In particular, we focus on awareness and reflection as ancillary phenomena as well as on learning outcomes as the main phenomena. Awareness and reflection are closely associated with interaction in a learning scenario. Hence, we implemented a peer assessment in our lecture which supports interaction in the learning scenario and in consequence, awareness, reflection, and ultimately learning outcomes. Figure 1 de- picts our theory-motivated design approach. + Reflection + Learning Interaction + Outcome + Awareness Interaction supporting component Fig. 1. Theory-motivated Design Approach 3.1 Concept of a Large-Scale Flipped Classroom The concept of the presented peer assessment is part of a didactical concept for the flipped classroom, also known as inverted classroom [44] or inverted lecture [45]. This concept is implemented for the first time within an IS lecture at a German uni- versity. By choosing a learner-centered approach, the objectives are to increase the lecture’s quality as well as to convey learner success and satisfaction. The following figure illustrates the flipped classroom concept. We therefore applied the learner- centered concept, which addresses three types of interaction throughout all phases. Referring to the work of Moore [46], the figure below differentiates between learner- content-interaction, learner-lecturer-interaction, and learner-learner-interaction. 39 Creating awareness and reflection in a large-scale IS lecture - ARTEL14 Self-Consistent Preparation Collaborative Preparation LEARNER-CONTENT LEARNER-LECTURER LEARNER-LEARNER Interaction Interaction Interaction Collaborative Clarification Collaborative Application Scenario for the application of a peer assessment Fig. 2. Concept of a Flipped Classroom Learning Cycle Integrating Three Types of Interaction The shown learning cycle has duration of two weeks; it will be repeated 5 times dur- ing one semester. Each cycle compromises four individual phases which are differen- tiated hereinafter: (1) the first phase can be substituted as self- or private study. The learners study small either video- or script-based learning units provided by the lec- turer in a Learning Management System (LMS). (2) During the next phase, every learner prepares a solution for a part of an extensive open-ended free text assignment within an allocated group. Every group needs to bring their solutions on power point slides; these are used as input for the third phase, namely “collaborative clarification”. (3) This phase is held in presence. The intention of this phase is to discuss the previ- ously submitted solutions, to consider further aspects of the findings and to emphasize its strengths. It constitutes the operational scenario for the application of a peer as- sessment, which is presented after a short explanation of the fourth phase. (4) The learning cycle ends with the phase of “collaborative application” which is dedicated to the tutorials. During the tutorials, all learners elaborate a common solution. In spe- cific, they work on assignments concerning business process management and con- ceptual data modeling. 3.2 The Application of a Peer Assessment in a Flipped Classroom Scenario The peer assessment imbeds itself in the third phase of the flipped classroom learning cycle. Its main goals are the collaborative clarification and consolidation of the pre- pared solutions submitted to the lecturer as well as gaining a deeper understanding of the learning content. Usually, an interactive learner-lecturer discussion is the method of choice. Similar to a traditional lecture, the third phase addresses the interaction type of learner-lecturer-interaction. In order to improve interaction, awareness, and reflection in the presence phases of the lecture, we developed a technology-enhanced peer assessment process addressing additional learner-learner- and learner-content- interaction. Instead of the lecturer presenting several group solutions, the learners 40 Creating awareness and reflection in a large-scale IS lecture - ARTEL14 themselves consider strengths and weak points and revise the solutions taking into account the comments made above. By reading and assessing colleagues’ group solu- tions the learners create awareness and reflection regarding their own group solutions. They get aware concerning their own strengths and weak points and they receive new ideas concerning the learning content. Hence, the peer assessment took place in a synchronous and written form via an online chat using a web based application. After finishing the peer assessment, the lecturer adopts the role of a moderator and supports the learners in the organization of a feedback loop. Figure 3 illustrates the structure of the presented peer assessment. LEARNER-CONTENT LEARNER-LEARNER LEARNER-LECTURER Interaction Interaction Interaction Task a) Etherpad 1 Task b) Etherpad 2 Peer Collaborative Assessment Clarification Etherpad 3 Task c) Etherpad 4 Task d) Fig. 3. The Process of a Peer Assessment Aware of the restrictions of our lecture room, we applied a web based application, to enable the learner participating to the process of a peer assessment. We therefore used an etherpad as a collaborative online notepad. Etherpad documents are accessi- ble via web browser and support multiuser usage without having to create multiple user accounts [47]. Being real time, capable etherpads enable people to collaborate on ideas, concepts and brainstorming. The selected etherpad has a chat bar on the right sidebar as well as a basic formatting functionality, and allows anonymous or public access. To help the learners during the peer assessment process, all learners are pro- vided with an etherpad compromising the intended assessment structure. Specifying the date and the learning unit, the created text gets manually stored in text files and uploaded to the LMS. Figure 4 shows one out of four generated etherpad documents during our lecture. 41 Creating awareness and reflection in a large-scale IS lecture - ARTEL14 Fig. 4. Technology-enhanced Peer Assessment 4 Evaluation In order to evaluate our technology-enhanced peer assessment, we surveyed the par- ticipants of our lecture. We therefore provided a paper-based pre- and post-test during the lecture. As stated before, we embedded our peer assessment in a flipped classroom IS lecture. Before the presence phase in the lecture hall and the peer assessment, four groups of students worked collaboratively in an online forum on four different as- signments. Each group prepared group individual presentations to a different assign- ment and posted the assignments to the online forum. Before conducting the lecture, we administered the prepared group assignments to the other groups. In the actual lecture, the other groups assessed the elaborated assignments of the other groups. Afterwards, the lecturer moderated a discussion of the collaborative peer assessment. To evaluate our procedure, a pre-test was administered before conducting the peer assessment. Afterwards, the peer assessment was conducted as described above and at the end of the lecture, the post-test was administered. In the survey, 35 learners who participated in the peer assessment process answered voluntarily both parts of the survey, which contained questions regarding the experience with the peer assessment. All items of the survey were adopted from literature and adapted, if necessary, to our research context. The items for measuring the perception of the peer assessment were adopted from Pearce et al. [48] and perceived learning outcomes were adapted from Eom et al. [49]. The pre-test asked questions about the learners’ experiences and expectations re- garding the peer assessment. The results show that only 26 percent of our sample had previously participated in some sort of peer assessment, i.e. paper-based or technolo- gy-enhanced peer assessment. Considering the phases of the peer assessment, the 42 Creating awareness and reflection in a large-scale IS lecture - ARTEL14 majority (60 percent) of the learners expected that both writing and receiving feed- back would contribute the most to their learning outcomes. 25.7 percent expected that receiving feedback and 5.7 percent expected writing feedback would contribute the most to their learning outcomes (including 8.6 percent nonresponse). Figure 5 pro- vides details about the further results of the pre-test. Both items were measured on a five point Likert response format (1 = strongly agree; 5 = strongly disagree). Overall, the results show that the learners expected up front that the peer assessment would be useful as a scaffold in the learning process (PA1). Also, the learners expected that their peers were qualified enough to provide valuable feedback (PA2). Strongly agree Strongly Disagree 1 2 3 4 5 PA1: As a learning tool, I expect the peer assessment will be useful. Mean 2.38; SD= 0.85 PA2: I think that my peers are well qualified to provide me with critical feedback on my work: Mean 2.51; SD= 0.92 N= 35 Fig. 5. Results of the Pre-Test After conducting the peer assessment, we administered the post-test. First, we meas- ured which part of the peer assessment influenced learning the most. The results show that the learners’ expectations were confirmed, since 45.9 percent reported that learn- ing results most from both writing and receiving feedback. Figure 6 shows further results of the post-test. All items were rated on a five point Likert response format (1 = strongly agree; 5 = strongly disagree) and provide the mean value and standard deviation (SD) of the responses. In addition, a one sample t-test was conducted in order to evaluate whether the mean values for all of the questions are lower than the neutral value. In consequence, the usefulness of the peer assessment is shown imply- ing the rejection of the null hypothesis 𝐻0 : 𝜇 ≥ 3. The results show that several items were rated under the neutral value of 3, indicating, in general, a good fit of the peer assessment. Additionally, the t-test provides evidence that H0 is not supported by several items (at least p<0.05), and can thus be rejected in these cases. The indicator PA3 was rated as good, on average (2.74) showing the overall usefulness of our tech- nology-enhanced peer assessment. PA4 however did not provide significant results. Therefore, we are not able to provide evidence that the peers involved in the peer assessment were actually suitable to conduct the assessment. In contrast, with a highly significant result, PA5 shows that our participants were able to improve their solu- tions after the assessment. Considering the learning outcomes in terms of learning success, LO1 was found to be significant and LO2 was found to be insignificant. These results are not really contradicting, since they demonstrate that our participants felt that the peer assessment itself did not actually affect the learning outcomes and 43 Creating awareness and reflection in a large-scale IS lecture - ARTEL14 that they have learned as much without participating in the assessment process. Fur- ther analysis of item LO3 showed that on average (2.71) the learners noticed an im- provement of the learning experience and quality. Strongly agree Strongly Disagree 1 2 3 4 5 PA3: As a learning tool, peer assessment was very useful. Mean 2.74; SD= 0.74; t= 2.05* PA4: I thought that my peers did a good job in providing me with critical feedback on my work. Mean 2.80; SD= 0.86; t= 1.36 PA5: I think that I improved my written work as a result of the assessment that I received or wrote. Mean 2.40; SD= 0.65; t= 5.45*** LO1: I feel that I learned as much with the peer assessment as I might have without. Mean 2.57; SD= 1.17; t= 2.16* LO2: I feel that I learned more with the peer assessment than without it. Mean 3.11; SD= 1.02; t= 0.61 LO3: The quality of the learning experience with the peer assessment is better than with- out it. Mean 2.71; SD= 0.75; t= 2.25* N= 35 Fig. 6. Results of the Post-Test 5 Discussion The present study was designed to determine whether an application of a technology- enhanced peer assessment is a suitable instrument to engage awareness and reflection of the learners by improving interaction in our lecture. Our results indicate that the expectations of the learners concerning the usefulness of the peer assessment were confirmed. This is in line with previous research results, which also indicated the usefulness of peer assessments in higher education scenarios [50]. However, we con- firmed the suitability of the peer assessment for a new learning scenario driven by a rich interaction in large-scale lectures. Surprisingly, our results show that the percep- tion of the learners did not reveal any significance for the suitability of their peers for the assessment process. The present findings seem to be consistent with other research 44 Creating awareness and reflection in a large-scale IS lecture - ARTEL14 which found that trust in the peer as an assessor is not a significant predictor for learn- ing outcomes [51]. This is also shown by the qualitative insights we gained during our evaluation. One learner states in the evaluation: “The idea is good, but the other students need to col- laborate better.” This opinion also relates to the results of our post-test, which re- vealed that most of the learners were not satisfied with their peers. Therefore, we acknowledge these issues by suggesting the implementation of further components that actually enhance a rich learner-learner-interaction to provide a useful process of awareness and reflection in the learning process. This is also highlighted by the fol- lowing statement of another student: “The procedure is very good and modern, but it depends very much on the fellow students”. Therefore, we also want to highlight the importance of the faithful appropriation of such a learning method. If the learning methods and structures, such as our technology-enhanced peer assessment, are ironi- cally appropriated, learning outcomes may suffer [52–54]. For instance, we noticed that students ironically appropriated the chat function and did not use it for a purpose- ful discussion. As an implication, we would suggest to guide the learning process and provide best-practices how to use the tool faithfully. However, our results also show that the received peer assessment improved the assignments of the learners. We there- fore highlight the importance of the feedback provided by the peers in order to im- prove learning outcomes. Considering the learning outcomes, we found evidence that the peer assessment has no significant impact on the subjective learning outcomes in our study. However, subjective perceptions have to be judged carefully, especially in the context of learn- ing success [55, 56]. Therefore, we cannot make any definite prediction on how peer assessments actually improve learning outcomes in a flipped classroom scenario. Interestingly, further analysis of the learning outcomes showed that the learners no- ticed an improvement of the learning experience and quality. This also relates to our results that the peer assessment as a meta-cognitive scaffold is a useful method to improve the learning process and in consequence increase learning outcomes [54, 57]. To sum up, we sought to address interactivity in the learning process as means to improve awareness, reflection, and learning outcomes in our learning scenario. Con- sidering our evaluation results, we can state that the peer assessment is a useful meth- od for structuring presence phases in a flipped classroom scenario. Hence, we high- light as a practical implication the importance of the learning process and the reflec- tion of the learning outcomes by interacting with it. This procedure also creates awareness of the learning progress, enabling learners to actually improve their self- regulated learning activities which are especially important in flipped classroom sce- narios. 6 Limitations and Future Research This study of a peer assessment in a flipped classroom is still on-going and therefore comes with limitations regarding the evaluation and application. Concerning the eval- uation, we consider the poor response rate in our evaluation. Typically, our lecture is 45 Creating awareness and reflection in a large-scale IS lecture - ARTEL14 attended by 150-200 students. Since we did not provide any compensation for partici- pating in our survey, the external value might be affected, because maybe only learn- ers well-disposed to such innovative learning concepts participated. In consequence, we consider this limitation and plan to evaluate the peer assessment in an additional and compulsory longitudinal online survey to account for the evolving nature of inter- action with e-learning components in the learning process. In addition, we plan to evaluate the actual effects of the peer assessment by conducting an experiment with a peer assessment treatment group and a control group. Furthermore future work should investigate peer assessment as instrument for indi- vidual learning success verification during the learning process. Following Bloom’s [13] suggestion, transfer and verification of learning content should be adjusting to various cognitive levels of educational objectives. In large-scale lectures the verifica- tion of high cognitive levels of educational objectives is very time- and resource- consuming and hence impractical in use. Peer assessment should be investigated as time- and resource-saving manner to measure learning success during the learning process. The other part of our limitations deals with the on-going application of our peer as- sessment. We applied the peer assessment in our lecture for the first time. Hence, we are still adjusting and modifying the process for the deployment of the peer assess- ment. In consequence, our evaluation could be biased by effects that are induced through the first time application, e.g., glitches that are mainly concerned with usabil- ity issues. However, we seek to overcome these limitations with a broad application during the next terms and further insights by this application. This would also include the application within other learning scenarios, especially those that are influenced by cultural differences [58, 59]. 7 Conclusion This paper has examined the application of a peer assessment in a large-scale IS lec- ture arranged in a flipped classroom setting. We therefore provided first evidence of the utility of peer assessments as suitable instruments to increase awareness and re- flection as well as to strengthen learning outcomes in an IS lecture. The results of this investigation show that the peer assessment itself does not affect the learning out- comes, but it does have a positive impact on learning experience and quality. Alt- hough the current study is based on a small sample of participants, the findings sug- gest that the application of a peer assessment might be a useful instrument to effect awareness and reflection. Considerably more work will need to be done to determine the effects of a technology-enhanced peer assessment on awareness and reflection as well as on learning outcomes. 46 Creating awareness and reflection in a large-scale IS lecture - ARTEL14 Acknowledgements The research presented in this paper was funded by the German Federal Ministry of Education and Research in the project kuLtig (www.projekt-kultig.de), FKZ 01BEX05A13. References 1. Leidenfrost, B., Strassnig, B., Schabmann, A., Carbon, C.C.: Verbesserung der Studiensi- tuation für StudienanfängerInnen durch Cascaded Blended Mentoring. Psychologische Rundschau 60, 99–106 (2009) 2. Grießhaber, W.: Neue Medien in der Lehre, http://spzwww.uni- muenster.de/griesha/pub/tnmlehre.pdf 3. Lehmann, K., Söllner, M.: Theory-driven design of a mobile-learning application to sup- port different interaction types in large-scale lectures. ECIS 2014 Proceedings (2014) 4. Lehmann, K., Thillainathan, N., Bitzer, P., Leimeister, J.M.: Performance Dashboard für Dozenten in der universitären Lehre. In: Multikonferenz der Wirtschaftsinformatik (MKWI). Braunschweig, Deutschland (2012) 5. Picciano, A.G.: Beyond student perceptions. Issues of interaction, presence, and perfor- mance in an online course. Journal of Asynchronous learning networks 6, 21–40 (2002) 6. van den Boom, Gerard, Paas, F., van Merriënboer, Jeroen J.G.: Effects of elicited reflec- tions combined with tutor or peer feedback on self-regulated learning and learning out- comes. Learning and Instruction 17, 532–548 (2007) 7. Boud, D., Walker, D.: Promoting reflection in professional courses: The challenge of con- text. Studies in Higher Education 23, 191–206 (1998) 8. Moore, A., Masterson, J.T., Christophel, D.M., Shea, K.A.: College Teacher Immediacy and Student Ratings of Instruction. Communication Education 45, 29–39 (1996) 9. Hardless, C., Nilsson, M., Nuldén, U.: ‘Copernicus’ experiencing a failing project for reflection and learning. Management Learning 36, 181–217 (2005) 10. Alonso, F., Manrique, D., Viñes, J.M.: A moderate constructivist e-learning instructional model evaluated on computer specialists. Computers & Education 53, 57–65 (2009) 11. Mayer, H.O., Hertnagel, J., Weber, H.: Lernzielüberprüfung im eLearning. Oldenbourg, R (2009) 12. Atkins, S., Murphy, K.: Reflection: a review of the literature. J Adv Nurs 18, 1188–1192 (1993) 13. Bloom, B.S.: Taxonomy of educational objectives. David McKay New York (1956) 14. Anderson, L.W., Krathwohl, D.R., Airasian, P.W., Cruikshank, K.A., Mayer, R.E., Pin- trich, P.R., Raths, J., Wittrock, M.C.: A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives, abridged edition. White Plains, NY: Longman (2001) 15. Strijbos, J.-W., Ochoa, T.A., Sluijsmans, D.M., Segers, M.S., Tillema, H.H.: Fostering Interactivity through Formative Peer Assessment in (Web-Based) Collaborative Learning Environments. In: Mourlas, C., Tsianos, N., Germanakos, P. (eds.) Cognitive and Emo- tional Processes in Web-Based Education, pp. 375–395. IGI Global (2009) 47 Creating awareness and reflection in a large-scale IS lecture - ARTEL14 16. Gregor, S.: The nature of theory in information systems. MIS Quarterly 30, 611–642 (2006) 17. Nussbaumer, A., Scheffel, M., Niemann, K., Kravcik, M., Albert, D.: Detecting and Re- flecting Learning Activities in Personal Learning Environments. Proc. of the 2nd Work- shop on Awareness and Reflection in Technology-Enhanced Learning (#artel12) held in conjunction with EC-TEL 2012, Saarbrucken (Germany) (2012) 18. Bodemer, D., Dehler, J.: Group awareness in CSCL environments. Computers in Human Behavior 27, 1043–1045 (2011) 19. Dabbagh, N., Kitsantas, A.: Supporting Self-Regulation in Student-Centered Web-Based Learning Environments. International Journal on E-Learning 3, 40–47 (2004) 20. Phielix, C., Prins, F.J., Kirschner, P.A.: Awareness of group performance in a CSCL- environment: Effects of peer feedback and reflection. Computers in Human Behavior 26, 151–161 (2010) 21. Cressey, P., Boud, D., Docherty, P.: Book Review: Productive Reflection at Work. Man- agement Learning 37, 410–413 (2006) 22. Høyrup, S., Elkjær, B.: Reflection - beyond the individual. In: Boud, D., Cressey, P., Do- cherty, P. (eds.) Productive reflection at work: Learning for changing organizations, pp. 29–42. Routledge (2006) 23. Boud, D., Keogh, R., Walker, D.: Reflection, turning experience into learning. Kogan Page; Nichols Pub., London, New York (1985) 24. Nonaka, I.: A Dynamic Theory of Organizational Knowledge Creation. Organization Sci- ence 5, 14–37 (1994) 25. Fischer, G.: Understanding, fostering, and supporting cultures of participation. interactions 18, 42 (2011) 26. Liu, T.-C., Liang, J.-K., Wang, H.-Y., Chan, T.-W., Wei, L.-H.: Embedding educlick in classroom to enhance interaction. In:, pp. 117–125 (2003) 27. Wang, M.C., Haertel, G.D., Walberg, H.J.: What influences learning? A content analysis of review literature. The Journal of Educational Research 84, 30–43 (1990) 28. Gagné, E.D., Yekovich, C.W., Yekovich, F.R.: The cognitive psychology of school learn- ing. HarperCollins College, New York (1993) 29. Morgan, N., Saxton, J.: Teaching, questioning, and learning. Routledge, New York (1991) 30. Bischof, L.: Die digitale (R) evolution? Chancen und Risiken der Digitalisierung akademi- scher Lehre (2013) 31. Piech, C., Huang, J., Chen, Z., Do, C., Ng, A., Koller, D.: Tuned models of peer assess- ment in MOOCs. arXiv preprint arXiv:1307.2579 (2013) 32. Boud, D., Falchikov, N.: Rethinking assessment in higher education. Learning for the longer term. Routledge (2007) 33. Sadler, P.M., Good, E.: The impact of self-and peer-grading on student learning. Educa- tional assessment 11, 1–31 (2006) 34. Darling-Hammond, L., Ancess, J., Falk, B.: Authentic assessment in action. Studies of schools and students at work. Teachers College Press (1995) 35. Block, J.H., Airasian, P.W., Bloom, B.S., Carroll, J.B.: Mastery learning. Theory and practice. Holt, Rinehart and Winston New York (1971) 36. Zoller, U.: Are lecture and learning compatible? Maybe for LOCS: Unlikely for HOCS. Journal of Chemical Education 70, 195 (1993) 48 Creating awareness and reflection in a large-scale IS lecture - ARTEL14 37. Sitthiworachart, J., Joy, M.: Web-based peer assessment in learning computer program- ming. In:, p. 180 (2003) 38. Kulkarni, C., Wei, K.P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., Klem- mer, S.R.: Peer and self assessment in massive online classes. ACM Trans. Comput.-Hum. Interact. 20, 1–31 (2013) 39. Dochy, F., Segers, M., Sluijsmans, D.: The use of self-, peer and co-assessment in higher education. A review. Studies in Higher Education 24, 331–350 (1999) 40. van Zundert, M., Sluijsmans, D., van Merriënboer, J.: Effective peer assessment processes: Research findings and future directions. Learning and Instruction 20, 270–279 (2010) 41. Briggs, R.O.: On theory-driven design and deployment of collaboration systems. Interna- tional Journal of Human-Computer Studies 64, 573–582 (2006) 42. Gehlert, A., Schermann, M., Pohl, K., Krcmar, H.: Towards a Research Method for Theo- rydriven Design Research. Wirtschaftinformatik Proceedings 2009 (2009) 43. Leimeister, J.M.: Dienstleistungsengineering und -management. Springer Berlin Heidel- berg, Berlin, Heidelberg (2012) 44. Strayer, J.F.: How learning in an inverted classroom influences cooperation, innovation and task orientation. Learning Environ Res 15, 171–193 (2012) 45. Gehringer, E.F., Peddycord, B.W.: The inverted-lecture model. In: Camp, T., Tymann, P., Dougherty, J.D., Nagel, K. (eds.) Proceeding of the 44th ACM technical symposium, p. 489 (2013) 46. Moore, M.G.: Editorial: Three types of interaction. American Journal of Distance Educa- tion 3, 1–7 (1989) 47. Karlhuber, S., Wageneder, G., Freisleben-Teutscher, C.F.: Einsatz kollaborativer Werk- zeuge-Lernen und Lehren mit webbasierten Anwendungen. Lehrbuch für Lernen und Leh- ren mit Technologien (2013) 48. Pearce, J., Mulder, R., Baik, C.: Involving students in peer review. Case studies and practi- cal strategies for university teaching. (2009) 49. Eom, S.B., Wen, H.J., Ashill, N.: The Determinants of Students' Perceived Learning Out- comes and Satisfaction in University Online Education: An Empirical Investigation*. De- cision Sci J Innovative Educ 4, 215–235 (2006) 50. Falchikov, N., Goldfinch, J.: Student Peer Assessment in Higher Education: A Meta- Analysis Comparing Peer and Teacher Marks. Review of Educational Research 70, 287– 322 (2000) 51. van Gennip, Nanine A.E., Segers, M.S., Tillema, H.H.: Peer assessment as a collaborative learning activity: The role of interpersonal variables and conceptions. Learning and In- struction 20, 280–290 (2010) 52. Gupta, S., Bostrom, R.: Technology-Mediated Learning: A Comprehensive Theoretical Model. Journal of the Association for Information Systems 10, 686–714 (2009) 53. Gupta, S., Bostrom, R.: An Investigation of the Appropriation of Technology-Mediated Training Methods Incorporating Enactive and Collaborative Learning. Information Sys- tems Research 24, 454–469 (2013) 54. Gupta, S., Bostrom, R.P., Huber, M.: End-user Training Methods: What We Know, Need to Know. SIGMIS Database 41, 9–39 (2010) 49 Creating awareness and reflection in a large-scale IS lecture - ARTEL14 55. Sitzmann, T., Ely, K., Brown, K.G., Bauer, K.N.: Self-Assessment of Knowledge: A Cog- nitive Learning or Affective Measure? Academy of Management Learning & Education 9, 169–191 (2010) 56. Benbunan-Fich, R.: Is Self-Reported Learning a Proxy Metric for Learning? Perspectives From the Information Systems Literature. Academy of Management Learning & Education 9, 321–328 (2010) 57. Bitzer, P., Janson, A.: Towards a Holistic Understanding of Technology-Mediated Learn- ing Services - a State-of-the-Art Analysis. ECIS 2014 Proceedings (2014) 58. Janson, A., Peters, C., Leimeister, J.M.: Der Weg zur effizienten Bereitstellung kultursen- sitiver Dienstleistungen - erste Schritte mittels systematischer Modularisierung. In: Thomas, O., Nüttgens, M. (eds.) Dienstleistungsmodellierung 2014, pp. 266–286 (2014) 59. Hammer, N., Janson, A., Leimeister, J.: Does culture matter? A qualitative and compara- tive study on eLearning in Germany and China. BLED 2014 Proceedings (2014) 50 Application overlapping user profiles to foster reflective learning at work Angela Fessl, Gudrun Wesiak, and Granit Luzhnica Know-Center Inffeldgasse 13 A - 8010 Graz (afessl, gwesiak, gluzhnica)@know-center.at Abstract. Reflective learning is an important activity of knowledge- workers in order to improve future working-behaviours. The insights gained by reflective learning are based on re-experiencing and re-evaluating past working situations. One time- and cost-effective way to support re- flective learning is the employment of applications that collect data about working processes, store the data in user profiles, and visualise it in order to provide timely feedback to the employees. However, a single applica- tion can only capture part of the data that might be relevant for reflec- tion and the parallel use of several applications leads to high demands on the user regarding the interpretation of relationships between sev- eral single visualizations. A combined visualisation of data captured by different apps should enhance the support for reflection about the work- ing behaviour and experiences. This paper introduces an overlapping user profile application, which combines and aggregates data captured by various applications. The goal of this overlapping application is to provide higher-level reflection possibilities by combining visualisations of different application data in order to better induce and support reflective learning at work. A first proof-of-concept of such an approach indicates that a combined user profile application and especially it’s visualisations can be beneficial with regard to reflective learning and can enhance the awareness about the multiple aspects of a user’s work life. Keywords: Work-place learning, reflective learning, awareness, user pro- files, reflective data analytics 1 Introduction Today’s work environments are constantly becoming more complex, globally integrated, and knowledge-centric. This simultaneously leads to a stronger need of employees who are motivated and capable to reflect upon their activities and as a consequence adjust their working practices to new demands. Especially for knowledge-workers, reflective learning is an important activity to re-experience past situations during work and to learn from them in order to improve their future working-behaviour [2]. One possibility to motivate knowledge workers to become reflective practitioners is to support them with corresponding tools 51 Application overlapping user profiles to foster reflective learning at work - ARTEL14 or applications, which could be easily integrated into their daily work-life [14]. These applications have the task to gather data from work processes and to provide guidance for reflection in form of raising awareness and offering triggers with regard to unusual or extraordinary work related experiences or situations. In contrast to formal learning settings, reflective learning at the workplace deals with informal and self-regulated learning, where challenges like no additional working effort, easy integration in daily working routines as well as a clear benefit for knowledge workers have to be considered right from the beginning. In order to support reflective learning at work, within the EU-funded project MIRROR (http://www.mirror-project.eu) several applications have been devel- oped, which aim at motivating and activating users to reflect upon their indi- vidual working experiences. After the reflection process itself, knowledge work- ers should have gained some benefits or insights for themselves and as con- sequence derive and apply behavioural changes for future working situations. These changes should permanently improve and facilitate the handling of up- coming similar situations or experiences. The applications developed within the MIRROR project have been applied within a wide range of working environments (e.g. care homes, hospitals, IT com- panies, and emergency situations) and support various sets of professionals (e.g. knowledge-workers, nurses, physicians and carers as well as emergency workers). Each of the developed applications collects and gathers different kinds of data and stores them in their corresponding user profiles. This data encompasses on the one hand information about the user. On the other hand it consists of in- formation on users’ work processes, which is captured automatically or inserted manually during the user’s work. Examples are application switches, application usage and documents used while working on a PC as well as manually inserted data such as the current mood status of the user, individual notes, feedback on different working situations, ratings, scores of serious games or quiz results. The collected data is stored within the applications themselves and for some applications additionally in the so-called MIRROR Spaces Framework [20], an underlying data storage system for exchanging data between applications. In the spaces framework, the user’s data is stored in the user profile and is accessible only by its owner. Each of these single applications visualises the data for the user in a sophisticated way with the goal to trigger reflective learning. However, user studies conducted in different environmental settings (e.g. [9], [16]) showed that single applications can only capture part of the data that might be relevant for reflective learning. Participants of these studies asked on the one hand for a better guidance to interpret the data in order to initiate reflection. On the other hand they wanted to see a clearer benefit for themselves, which would serve as motivational trigger to use the application and to reflect about the captured data. Thus, similar to research outcomes from the field of learning analytics, we found that a combination of data is often more adequate for successfully support- ing users. Whereas learning analytics addresses self-reflective learning mostly as important aspect of self-regulated learning in formal learning environments, we focus on work-related reflective learning in informal learning environments. 52 Application overlapping user profiles to foster reflective learning at work - ARTEL14 With this paper we want to present a first approach on how to meaningfully combine and visualize data captured by different applications. The goal is to provide a greater variety of reflective learning opportunities in order to facilitate deeper insights on one’s working experiences. We are aware that this approach raises privacy and security issues which need to be carefully considered when employing the app in a real working environment. However, for this first approach privacy and security were only of secondary interest, but will of course be treated in upcoming research settings. Therefore, we developed the so-called ”MIRROR Integrated User Profile” application (MUP App) which has the task to integrate, summarise, analyse, and visualise data captured by several different applications in order to induce and support reflective learning at work. For a first proof-of-concept, we used two different applications in parallel, namely KnowSelf and the MoodMap App. KnowSelf automatically captures work activities on a PC, whereas the MoodMap App allows knowledge workers to easily state their moods during a working day. We collected, aggregated, and visualised data from a small sample of knowledge workers to to get a first impression of users’ interest and motivation and the app’s usefulness. From this we derived the following three research questions: – RQ1: Are participants interested and willing to use more than one applica- tion in parallel with regard to reflective learning? – RQ2: Does the MUP App as overlapping application facilitate reflection about users’ working experiences and contribute to raising awareness of mul- tiple aspects of their work life? – RQ3: Do participants perceive any individual insights or benefits for them- selves? 2 Related Work 2.1 User Profiles Since the terms user profile and user model are not always used in exactly the same way, it is essential to clarify our understanding and usage of the term user profile, which we base on existing theories regarding user models and on our un- derstanding in MIRROR described by [15]. User models in general are models that computer systems have about their users. The data in such user models is automatically captured by the system and is mainly used in information retrieval and intelligent tutoring systems or user-adaptive learning systems (see e.g., [10, 1]). User models, which are utilised in learning environment systems for mod- eling the learner and the corresponding learning activities, are called learner models. These types of user models are created by the systems automatically and are not directly accessible by the users via user interfaces. Furthermore they are used to adapt teaching strategies or to inform the learner about the learning progress as basis for reflective learning. Additionally [3] suggested that learner models should keep data like knowledge, interests, goals, background, and individual traits, thus abstract concepts relevant for learning. In order to 53 Application overlapping user profiles to foster reflective learning at work - ARTEL14 apply a user model or learner model as basis for reflection on one’s own learning activities, achievements, or progress towards the individual learning goals, it is necessary to make the models accessible and manageable for the user, which was explicitly suggested by [12] and mentioned in [4, 5, 13]. In MIRROR we prefer the term user profile (UP). Although the MIRROR user profile (MUP) is based on theory and research of user models, the term user profile better reflects its mission in MIRROR. First, the purpose of the MUP is to guide and support reflection by mirroring user data in the form of activities, experiences or artefacts of work, notes and insights, moods, work practices, and other concrete data sources back to the user. Secondly, we intended these user profiles to be created and maintained by a mixture of automated methods and manual management, where the process of editing or updating the data may also explicitly trigger reflection. 2.2 Learning Analytics Although learning analytics is not in focus of our work, several approaches, methodologies and technologies of this research area are closely linked to reflec- tive learning. Learning analytics deals with methods for analysing and detecting patterns within data collected from educational settings or learning environ- ments about the learner, and leverage those methods to support adaptation, personalisation, recommendation, and also reflection. Siemens [21] defined learn- ing analytics as ‘the use of intelligent data, learner-produced data, and analysis models to discover information and social connections, and to predict and advise on learning’. The focus of learning analytics is on the support of the learner in formal learning setting, while in our work the focus is to support the knowl- edge worker in an informal learning setting. Nevertheless, the parallel to our work is evident. Also approaches like learning dashboards for example described in [8, 19] present an overview of the learner’s own learning activities and learning progress, and in relation to colleagues at one glance. Such combined visualisa- tions support self-monitoring for learners and awareness for teachers as well as empowers the learners to reflect on their own activity, and that of their peers. Explicit traces (e.g. the learner’s entries in a chat or a discussion forum) and im- plicit traces (e.g. the learner entering a course or clicking on a document) stored in the corresponding learner profiles serve here as basis for the aggregation and visualisation of the gathered data. The main focus of learning analytics is to support the learner while learning in an educational setting or learning environment. Although learning analytics includes also reflective learning approaches (e.g. [18, 17]), our work can be clearly distinguished from these approaches by focusing on knowledge workers in real working environments and and to support reflection on working experiences or working artifacts in order to learn from them to improve future work. 54 Application overlapping user profiles to foster reflective learning at work - ARTEL14 2.3 Reflective Learning Individual reflection takes normally place in every day’s life and obviously also during work or work- related situations. Reflection may be triggered by different reasons for example by conflicts or problems, by unexpected experiences or by a person acting in a complete different way in comparison with the individual (external trigger). But also if an individual feels uncomfortable, for something bothers her or an inner voice is nagging, without being able to make this feeling external (internal trigger). As reaction, a reflection process may be triggered with or without the awareness of the person. This reflection should lead to an individ- ual insight or outcome which may be used to guide or adapt future behaviour. Within MIRROR we follow the definition of [2], who define reflective learning as ‘those intellectual and affective activities in which individuals engage to explore their experiences in order to lead to new understandings and appreciations’. Bringing this together, reflection is both a crucial part of learning and a re- sponse to past work experiences. These experiences as well as the behaviours of the individual engaged serve as starting point for the reflective process. The de- sired outcomes of reflection may lead to personal synthesis, integration of knowl- edge (internalisation), validation of personal knowledge, a new affective state or the decision to take on actions for future events. To achieve these results the characteristics of the individual (learner) have to be taken into account as well as the intention of the individual self. Individual reflection may occur sponta- neous and unconsciously and in any possible situation especially then when it is not expected [6]. Of course it can also be consciously triggered by peers super- visors or by meeting created specifically for that purpose [7]. Within MIRROR we focus to initiate reflective practices with the support of technologies, which might automatically detect unusual working patterns and working behaviours and by making the worker aware of them in form of reflection triggers or explicit reflection guidance e.g. by means of prompts. 3 Examples of MIRROR Applications In the scope of the MIRROR project a series of applications supporting indi- vidual reflection have been developed and evaluated in different settings [16, 9]. Some of the user studies showed that gathering and visualising data captured by single applications is not always enough to initiate reflective learning. To il- lustrate how the MIRROR applications support reflective learning, we want to shortly introduce two applications, namely KnowSelf and the MoodMap App. The same two applications will later be used as example for a possible combined usage and integration via the MUP App. KnowSelf automatically captures work activities (used applications and resources together with the exact time of use) on a PC, provides simplistic project and task recording and presents an overview as well as different visualisations of the captured data [16]. Providing these visualisations regarding time use at work should lead to reflection on personal time management and potentially motivate to consider improvements in this respect. The user profile of KnowSelf 55 Application overlapping user profiles to foster reflective learning at work - ARTEL14 Fig. 1. KnowSelf (left) and MoodMap App (right) is not a conventional user profile, because it consists only of user activities, but not of information about the users themselves. The application stores all work activities captured on the user’s computer, including window focus and title, if applicable the system location (path) of the resource, focus switches, and idle time. Additionally the user can manually record time spent on projects or tasks and save observations. The collected information is displayed on a timeline and as statistics in the form of pie charts. The MoodMap App is a web-based application, which allows knowledge workers to track their mood during a working day or virtual meeting and recapit- ulate their work experiences afterwards. The MoodMap App provides an easy- to-understand user interface to state individual mood points by simply clicking on a bi-dimensional coloured map. Each mood is composed of two dimensions, namely valence (negative to positive feelings) and arousal (low to high energy) based on the model proposed by [11]. Additionally, it provides several visualisa- tions on an individual as well as collaborative level, to make users reflect on the mood development over time or to provide easy comparison possibilities of one’s own mood with the mood of others for example colleagues or team members of the same team. The application related user profile stores information about the user, sharing settings for security and privacy issues as well as individual email settings. Furthermore, individual moods and inherent notes, correspond- ing meetings, context information of a day or meeting, as well as personal diary entries are stored in the internal user profile of the application. 4 The MIRROR Integrated User Profile (MUP) Insights from evaluations conducted separately for KnowSelf and the MoodMap App led to the conclusion that single applications capture only part of the data that might be relevant for reflection [9, 16]. Although the developed applications proved to have high potential to trigger reflection at work, we wanted to go one step further. As a first step, we made triggers from different sources easily 56 Application overlapping user profiles to foster reflective learning at work - ARTEL14 accessible to the users, in order to further facilitate the reflective learning experi- ence and help users to get more insights at one glance. Thereupon we developed the MIRROR User Profile (MUP) concept, which focuses on the combination of the captured data and corresponding sophisticated visualisations. An early prototype of the MIRROR User Profile Application (MUP App) was realised and tested with a small sample of knowledge workers. 4.1 Prerequisites for the MIRROR User Profile In order to efficiently implement a common MIRROR user profile, it is of crucial relevance to use data captured and gathered by several MIRROR applications and not only by a single one. To achieve this, we employed the MIRROR Spaces Framework, an underlying data storage system developed within the MIRROR project to store and exchange data of the applications. For the development of a common user profile based on the MIRROR spaces the following prerequi- sites have to be considered: assumptions regarding (i) data, (ii) reusability, (iii) sharing, (iv) privacy and security, and (v) accessibility by the user interfaces. Data stored in the MUP can be divided into three different types, namely personal data about the user, private data, and shared data. Personal data about the user consists of general information about the user (e.g. name or email ad- dress) and login information. For the data implicitly captured by the MIRROR applications (e.g. work history in KnowSelf) as well as data explicitly inserted by the user (e.g. mood in the MoodMap) it is essential that the user has full control over her data by deciding for each type of captured data, whether it is private or can be shared. Reusability is one of the major potential benefits of the MUP. By storing the data according to a predefined data format, applications are able to reuse not only their data but the data captured by other applications and other users as well. Account information can be stored once in the user profile and then be used by all MIRROR applications. Sharing data is of major relevance for reflection in order to provide possibil- ities for comparing one’s own data with that of colleagues or a whole team. To account for different levels of sharing, settings (e.g. anonymised, sharing within the same team or department) should be very fine-grained. As mentioned above privacy and security are a major concern when storing data in the MIRROR Spaces Framework. It has to be ensured that the privacy settings defined by the users via different applications are always met by all applications, aggregations, and visualisations. Sharing, privacy and security settings along with other data gathered either explicitly or implicitly by applications, should be accessible and modifiable by user-friendly interfaces and visualisations provided by each MIRROR appli- cation. This has the advantage, that the user has full control about the data and has the potential to decide on a very fine-grained level, which data she wants to share with whom and which data should be kept private only. 57 Application overlapping user profiles to foster reflective learning at work - ARTEL14 4.2 The MIRROR Integrated User Profile Application The MIRROR Integrate User Profile Application (MUP App) serves as a bridge between the MIRROR Spaces Framework and various MIRROR applications. It provides services for data administration as well as for directly supporting reflective learning. The latter is achieved by making users aware of unusual or significant behavioural patterns. The MUP App’s service can be used by other applications to show and promote reflective learning by presenting combined data aggregated by different applications or from different users. The tasks of MUP App are two-fold, providing (a) access to the data stored in the corresponding user profiles per user within the MIRROR Spaces Framework and (b) a data analysis service, which aggregates data from different applica- tions (on an individual level) and/or from different users (on a collaborative or organisational level). The aggregated data can be used to raise awareness on relationships between data captured from different applications, make compar- isons along a timeline or among different users, and finally detect patterns that are relevant for individual or collaborative reflection. Reasons for reflection can encompass the need for problem solving, decision-making, emotion regulation, or detection of significant deviations between the individual user and a team. In this first phase of the development we pursue a more general approach directed towards basic types of data that are comparable across different ap- plications. We mostly focus on statistical analysis to extract information on for example the number of different applications used by an individual, on providing a chronological overview of the applications used, on presenting the number of entries in various diaries, and on general information (e.g. when, how often or which data) was captured by each application. The data is presented on different types of charts, which can be selected by the user in order to ensure that the chart fits to the available data. In addition, the user may visualise her data in direct comparison with the data to other users (e.g. her team-members). For the second phase, we will be concentrating on the different types of data captured by various applications, in order to provide analysis on the com- bined data. For instance, combining the usage of the MoodMap App with data captured by KnowSelf, might show a relationship between moods and specific working tasks. This would lead to new insights that may be the basis for ini- tiating reflective learning. As an example, the left of Fig. 2 shows the hourly application usage history of a single user for both KnowSelf and the MoodMap App. The picture on the right of Fig. 2 visualises combined application specific data, namely the number of hourly switches between tasks or resources captured by KnowSelf and the corresponding mood of a user, depicted as separate lines for arousal (mood.energy) and valence (mood.feel) by the MoodMap App. For this visualization, mood values from the MoodMap App depicted in Fig. 1 are expressed as numbers between 0 and 100. 58 Application overlapping user profiles to foster reflective learning at work - ARTEL14 Fig. 2. MUP application with MoodMap and KnowSelf data for a single user 5 Proof of concept In order to investigate the potential of a common MIRROR user profile as sup- port for reflective learning, we conducted a small combined user study employing the KnowSelf and the MoodMap App in parallel. Although we used only two application for this first evaluation, the MUP App is able to handle all applica- tions that store their data in the common user profile. Based on what we have learned from the separate evaluations, we see this study as first proof-of-concept for the MUP App. The goal was to find out whether a combined analysis of data from both user profiles will i) be accepted by the users, ii) enhance the boost of reflective learning, and iii) provide clearer insights or benefits for the single user. 5.1 Setting The participating team consisted of 6 knowledge workers (3 women, 3 men), on average aged between 30 and 40, all of them mostly doing computer work. They used the KnowSelf and the MoodMap App in parallel for two weeks during work. Each day, either in the morning or in the evening they were asked to re-evaluate and reflect about their captured data and write down their insights and thoughts directly within one of the two applications. User activities automatically logged by KnowSelf could only be analysed for 5 persons due to technical reasons on one of the PC’s. At the end of the trial the participants were asked to fill in a questionnaire and to take part in a semi-structured interview. The questionnaire covered information regarding features and functionalities of the applications, usage, and reflective learning. During the interview, com- bined statistics (see Fig. 2) of the captured data were presented and discussed in order to find out the insights and benefits gained for the individual user. 59 Application overlapping user profiles to foster reflective learning at work - ARTEL14 5.2 Results & Discussion The analysis of the log data of both applications is depicted in Fig. 3. Because of the small sample size only descriptive statistics are presented. As measure of central tendency the median is used for the same reasons. Each data point represents the average mood values (in terms of valence and arousal) of one participant in relation to the application usage and working activities (switching frequency and used resources). Whereas there is no trend to be derived from this small sample for the active use of KnowSelf, the number of moods entered per day seems to increase with higher valence and higher arousal values indicated by the participants (i.e. with a more positive mood). Switching frequency was measured in seconds between switching from one resource to another. Fig. 3 (bottom) shows that higher reported valence seems to be connected to longer times between switches (that is a lower switching frequency) and fewer resources used. Interestingly, the arousal level increases with the number of used resources. Fig. 3. Relationship between mood (as valence and arousal), application usage and working activities Analysing the data collected via questionnaires and interviews, we can give first answers to the research questions: RQ1: Are participants interested and willing to use more than one applica- tion in parallel with regard to reflective learning? Ratings from 6 participants answering the questionnaire (using 5pt. agreement scales) indicate that there is an interest in getting support for time-management (Md (median) = 4) as well as in capturing one’s working activities, own mood, and the team mood (all Md =3.5). Participants found the applications easy to use, liked their visu- alisations, rated the presentation of information as comprehensible (all Md =4), 60 Application overlapping user profiles to foster reflective learning at work - ARTEL14 generally liked using the applications and would recommend them to colleagues (for both items Md =4 for KnowSelf and Md =3.5 for MMA, respectively). RQ2: Does the MUP App as overlapping application facilitate reflection about users’ working experiences and contribute to raising awareness of multiple aspects of their work life? The interview results revealed that the combination of data has high potential to trigger reflective learning although we have ambiguous statements in which way. One participant reflected mainly on the number of used applications and its relation to how the level of arousal developed over the day. Another participant mentioned that combined data helped her to detect a working pattern, which occurred especially in the morning. After reading emails the application switches and the arousal level increases, thus she knows that she started to work. Similarly, one of the participants observed that her arousal level is very low in the morning and increases during the day. This was a trigger to compare her arousal level to the average level of her colleagues and reflect upon eventual differences between them. An important feature mentioned by more than one participant was the overlapping visualisation of captured data on the timeline chart. Here the data was understood at one glance, which can facilitate reflective learning and enhance awareness of the multi aspects of their work life. Despite of the different approaches to reflect, for all participants the combination of data captured by both applications was important to understand the relationship between their working activities and moods. RQ3: Do participants perceive any individual insights or benefits for them- selves? Besides the findings already described in relation to RQ2, participants reported some additional insights they gained by reflecting on the captured data provided by the MUP. One participant stated that her arousal level fluctuates during the day. By becoming aware of the falling arousal level she decided to take smaller breaks to better recover during the day. Further insights concerned participants’ self-estimations of how they spend their working day. Whereas one participant stated that the captured data confirmed how she estimated the rela- tionship between working activities and mood development, another participant was rather surprised in the first place. Although she was six to seven hours in the office she spent only four hours in front of her computer. Only after comparing this awareness with her dates in her calendar, she could reproduce her day and explain why this happened. General discussion. In general, this first proof-of concept of the MIRROR In- tegrated User Profile indicates that such overlapping visualisations can facilitate individual reflective learning and raise overall awareness of users’ work life. All six participants used the combined data to reflect on how their working activities are related to mood changes and could gain some individual insights. Nevertheless there a still some points which need further discussion. While KnowSelf captures automatically the resources and applications used on the PC, the moods need to be inserted manually. Having to repeatedly insert a mood in a web based application can distract from the normal working process. One recommenda- tion to alleviate this distraction was to add five different smileys in the system tray to facilitate the mood capturing. Another point for consideration is the 61 Application overlapping user profiles to foster reflective learning at work - ARTEL14 optimal time for reflection. All of the participants perceived the combination of the data captured by the MoodMap App and KnowSelf as useful, because they could check at all times what they were doing during work and how they felt. However, one participant stated that it was not very useful to reflect on how she felt three days ago, but that it was more interesting to become aware of her mood in relation to her work directly while working. For other participants especially the knowledge of how they felt for example three days ago was very important. Especially when the mood could be directly related to the mood note, used applications or used resources. A rather interesting statement from one of the participants was that her working tasks did not influence her mood at all. With respect to the visualisations, the interviews showed that different types of aggregating the data would be useful, so that users could indicate their individual preferences, e.g. to visualise the data along a timeline, to aggregate on an hourly basis, or to offer a summarising view in form a pie chart. In summary the MUP App provides new visualisations based on data cap- tured by different applications, and therefore offers a multitude of new possibili- ties for individual interpretations. In our proof of concept, we only combined data of two applications, but also within this small setting we received different ap- proaches on how the participants interpreted the captured data for themselves and what they learned from it. We also mentioned some shortcomings which must be taken into consideration when proceeding with the development of the MUP App. Nevertheless, our findings encourage the assumption that combining data of more than two applications, leads to more meaningful possibilities to interpret the data and to gain more diverse insights for oneself. 6 Conclusions and Outlook In this paper we presented the new MIRROR integrated User Profile Applica- tion, which aims at supporting reflective learning at work. Based on the results from previous user studies, which evaluated single applications, we derived es- sential requirements for the development of the MUP App and implemented a first prototype. Results from a first small evaluation regarding the parallel usage of two applications indicate that combining data captured by different appli- cations, analysing and visualising them together can further facilitate reflective learning. Furthermore, it can also enhance awareness of the work life by leading the users to get more diverse insights about themselves. Of course, after this first proof-of-concept, user-studies with larger samples and more applications need to follow. Thus, our future work will focus on the integration of further applica- tions developed within the MIRROR project into the MUP App. The goal is to provide different variants of visualising combined data and more sophisticated ways to provide guidance for reflective learning. For example, Fig. 4 combines KnowSelf data with corresponding geo location data captured by another MIR- ROR App. Thus it makes you aware of your working activities in relation to your working places (e.g. customer visits or travel activities) and can provide more triggers for reflective learning. 62 Application overlapping user profiles to foster reflective learning at work - ARTEL14 Fig. 4. Further sophisticated visualisations for the MUP App 7 Acknowledgement The project “MIRROR - Reflective learning at work” is funded under the FP7 of the European Commission (project number 257617). The Know-Center is funded within the Austrian COMET Program - Competence Centers for Ex- cellent Technologies - under the auspices of the Austrian Federal Ministry of Transport, Innovation and Technology, the Austrian Federal Ministry of Econ- omy, Family and Youth and by the State of Styria. COMET is managed by the Austrian Research Promotion Agency FFG. References 1. Adomavicius, G., Tuzhilin, A.: Toward the next generation of recommender sys- tems: a survey of the state-of-the-art and possible extensions. Knowledge and Data Engineering, IEEE Transactions on 17(6), 734–749 (June 2005) 2. Boud, D., Keogh, R., Walker, D.: Reflection: Turning Experience into Learning, chap. Promoting Reflection in Learning: a Model., pp. 18–40. Routledge Falmer, New York (1985) 3. Brusilovsky, P., Millán, E.: User models for adaptive hypermedia and adap- tive educational systems. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) The Adaptive Web, chap. User Models for Adaptive Hypermedia and Adap- tive Educational Systems, pp. 3–5. Springer-Verlag, Berlin, Heidelberg (2007), http://dl.acm.org/citation.cfm?id=1768197.1768199 63 Application overlapping user profiles to foster reflective learning at work - ARTEL14 4. Bull, S., Dimitrova, V., McCalla, G.: Open learner models: Research questions. International Journal of Artificial Intelligence in Education (IJAIED Special Issue) 17 (2), 83–87 (2007) 5. Bull, S., Kay, J.: Student models that invite the learner: The smili:open learner modelling framework. International Journal of Artificial Intelligence in Education 17, 89–120 (2007) 6. Daudelin, M.W.: Learning from experience through reflection. Organizational Dy- namics 24(3), 36–48 (1996) 7. Dingsøyr, T.: Postmortem reviews: purpose and approaches in software engineer- ing. Information and Software Technology 47, 293–303 (2005) 8. Duval, E.: Attention please!: Learning analytics for visualization and recommen- dation. In: Proceedings of the 1st International Conference on Learning Ana- lytics and Knowledge. pp. 9–17. LAK ’11, ACM, New York, NY, USA (2011), http://doi.acm.org/10.1145/2090116.2090118 9. Fessl, A., Rivera-Pelayo, V., Pammer, V., Braun, S.: Mood tracking in virtual meetings. In: Proceedings of the 7th European conference on Technology Enhanced Learning. pp. 377–382. EC-TEL’12, Springer-Verlag, Berlin, Heidelberg (2012) 10. Fischer, G.: User modeling in human-computer interaction. In: User Modeling and User-Adapted Interaction (UMUAI). vol. 11, pp. 65–86 (2001) 11. Itten, J.: Kunst der Farbe. Otto Maier Verlag, Ravensburg, Germany, 1 edn. (1971) 12. Kay, J.: Learner control. User Modeling and User-Adapted Interaction 11, 111–127 (2001) 13. Kay, J., Kummerfield, B.: Adaptive Technologies for Training and Education, chap. Lifelong learner modelling, pp. 140–165. Cambridge University Press (2011) 14. Lindstaedt, S., Kump, B., Beham, G., Pammer, V., Ley, T., Dotan, A., de Hoog, R.: Providing varying degrees of guidance for work-integrated learning. In: Sustaining TEL: From Innovation to Learning and Practice: 5th European Conf.Technology Enhanced Learning (ECTEL 2010). pp. 213–228. LNCS 6383, Springer (2010) 15. Pammer, V.: D4.1 Update - Results of the user studies and requirements on ‘Indi- vidual Reflection at Work’ (2012) 16. Pammer, V., Bratic, M.: Surprise, surprise: Activity log based time analytics for time management. In: CHI Extended Abstracts. pp. 211–216 (2013) 17. Rivera-Pelayo, V., Munk, J., Zacharias, V., Braun, S.: Live interest meter: Learn- ing from quantified feedback in mass lectures. In: Proceedings of the Third Inter- national Conference on Learning Analytics and Knowledge. pp. 23–27. LAK ’13, ACM, New York, NY, USA (2013), http://doi.acm.org/10.1145/2460296.2460302 18. Rivera-Pelayo, V., Zacharias, V., Müller, L., Braun, S.: Applying quantified self approaches to support reflective learning. In: Proceedings of the 2Nd International Conference on Learning Analytics and Knowledge. pp. 111–114. LAK ’12, ACM, New York, NY, USA (2012), http://doi.acm.org/10.1145/2330601.2330631 19. Santos, J.L., Verbert, K., Govaerts, S., Duval, E.: Addressing learner issues with stepup!: An evaluation. In: Proceedings of the Third International Conference on Learning Analytics and Knowledge. pp. 14–22. LAK ’13, ACM, New York, NY, USA (2013), http://doi.acm.org/10.1145/2460296.2460301 20. Schwantzer, S.: D2.2 Software architecture – version 3 (2012), http://mirror- project.eu/showroom-a-publications/downloads/finish/5/68 21. Siemens, G..: What are learning analytics? (Februrary 2010), http://www.elearnspace.org/blog/2010/08/25/what-are-learning-analytics/ 64 Reflection as Support for Career Adaptability: A Concept for Reflective Learning in Public Administration Michael Prilla1, Oliver Blunk1, Jenny Bimrose2, Alan Brown2 1 Ruhr-Universität Bochum, Germany {michael.prilla,oliver.blunk}@rub.de 2 University of Warwick/Institute for Employment Research, Coventry, UK {Jenny.Bimrose,Alan.Brown}@warwick.ac.uk Abstract. Reflective learning is a mechanism to turn experience into learning. As a mechanism for self-directed learning, it is critical for success at work. This is true for individual employees, but also for teams and whole organizations. Change processes are typical situations in which people question their practices, reflect on how they adopt new practice and try to learn from good or bad prac- tice for future work. Such changes can support the development of expertise. In cases involving substantive changes in work roles and behaviours they may even contribute to a process of professional identity transformation, which leads to a deeper understanding of one’s own practice and of the processes involved in adapting it to internal and external constraints. Public administrations are ex- amples of organisations that undergo constant change due to changes in legisla- tion, financial pressure and demands of the public, and the public intensively observes them. These pressures are requiring staff to adapt, including by chang- ing their professional identities. Integrating reflection into the practice of staff can support them in informal learning and improving, and it can thus lead to enhanced and more efficient services for the public. In this paper, we report on an approach of using reflection in Public Employment Services (PES) in Eu- rope, which are currently being transformed from being principally concerned with administration of benefits and provision of advice and guidance to an in- creasing concern with coaching clients and co-operating with employers. 1 Introduction Reflection is a common and desirable process of learning from experience for future work [1–3], which is carried out by individuals or groups in a self-directed manner and as a mostly informal learning process [4–6]. Reflection helps workers to deal with changing work contexts [4], supports them to create innovative practices [7] and ena- bles them to create change in a bottom-up process [6]. Besides daily problems, up- coming and continuous change processes are typical examples in which people ques- tion the way they work(ed) and whether they have adopted new processes adequately. Public administrations are typical examples of organizations facing constant change, due to: legislative changes, budget and staff cuts, requirement for remaining 65 Reflection as support for career adaptability - ARTEL14 staff to become more efficient, the public demand for higher service quality, role changes for staff, and, in some cases, alternative service provision (e.g., [8]). Our work focuses on Public Employment Services (PES) as an example of chal- lenges faced at public administrations: staff are dealing with more clients in a rapidly changing labour market and are expected to offer a wider range of services. In many European countries staff roles are being transformed from offering advice on access to benefits and available job opportunities towards facilitation and coaching where staff are expected to support clients in becoming more self-directed and staff are also expected to understand the labour market better and engage more with employers. Staff therefore need to be capable of adapting to various and often unforeseeable changes. Career adaptability [9] as a process of continuously adapting to changing requirements on the labour market is a central concept in this context. This process is closely connected to self-reflection and reflection in groups [9], but work investigat- ing reflection support tools for career adaptability is not available. This paper con- nects research on reflection support to career adaptability research by presenting a conceptual approach and a prototype to support this process with reflection tools. 2 Related Work 2.1 Career Adaptability and Professional Identity Transformation Career adaptability is the ability to manage successful transitions in employment, training, education and other contexts. It is key for workers dealing with constantly changing requirements on the labour market [9]. Adapting careers, however, needs a transformation of one’s individual and collective professional identity, including as- pects such as work activities and organisation, relations to other professions and pro- fessional culture [10]. This transformation can be triggered by challenges at work and needs self-directed learning, self-reflection and learning in interaction with others [9]. Therefore support needs to include individual and collective means. Public Employment Services (PES) practitioners deal with career adaptability both in their personal careers and in the careers of clients they are supporting. Therefore supporting them in career adaptability not only supports their personal career but also supports their clients to re-enter the labour market. 2.2 Reflective Learning at Work Following Boud (1985) [1] we understand reflection as a process of conscious re- evaluation of experience for the purpose of guiding future behaviour. This perspective is in line with the conception proposed by Schön (1983) [3], who in addition differen- tiates between reflection-in-action and reflection-on-action, and other authors dealing with reflective learning. In addition, we understand work and learning as intertwined [3, 11], and therefore also work and reflection [12]: reflection transforms experience from work into knowledge applicable to the challenges of daily work and thus needs to be understood as a key process for informal learning at the workplace [5]. It is 66 Reflection as support for career adaptability - ARTEL14 mostly triggered when individuals or groups perceive some discrepancy, e.g. contra- dictory information, incongruent feelings, interpersonal conflicts and other occurrenc- es during work, leading to a state of discomfort that the individual or group wants to overcome [13]. Characteristic activities of reflection can then be found in asking for feedback on your work and opinions, critical opinion sharing (and being open to it in the organisation) or challenging groupthink (instead of going with the majority) [14]. In addition to most models we differentiate between individual reflection as a mostly cognitive activity and collaborative reflection, which is done in communica- tion among peers in a group [15]. The latter has been found to create results that transcend the capabilities of a group’s members [7] and it is a promising process for the creation of innovation and change in modern workplaces [4], but it has received less attention in work on reflection at work. Knipfer et al. (2013) [13] point out that as workplaces provide individuals with a social context, individual and collaborative learning are intertwined and must be considered together. It has been shown that reflective learning can be supported by technology (e.g., [15–17]) by providing data or written content on experiences to reflect upon, support- ing retrospective analysis or by scaffolding the reflective process, the documenting and sharing of a decision rationale. More specifically, writing down positive or nega- tive experiences and being prompted regularly to think about them has been shown to be supportive for individuals to engage in continuous reflection [18]. However, as most existing work either supports early phases of reflection (e.g., gathering and shar- ing data) or stems from educational settings, which are often designed in favour of reflective learning, there is still work to be done in the context of reflection at work. 3 Reflection for Professional Identity Transformation: A Concept The development of career adaptability relies on four key dimensions: learning to adapt through challenging work, through updating a substantive knowledge base, by being self-directed and self-reflexive as well as learning through interactions at work [9]. In this section we show how reflection can support these dimensions and how this can be used as a basis of professional identity transformation. Dealing with challenging work can bring up discrepancies in daily work, which (as described above) trigger reflection [13]. Successfully dealing with these situations can lead to confidence in one’s skills and abilities. Reflecting about work and its chal- lenges comes into play when there are no problem solving patterns available for the challenges met and new solutions are needed [19]. To keep up with knowledge in changing fields of work learning through updating a substantive knowledge base is required. While workers often use formal learning offers at work, informal learning can be seen as a key to continuously understand which knowledge is needed and integrate it into one’s context [9]. Reflection can support these needs [5] and the integration of new knowledge [20], helping workers to think about the state of their own knowledge and to identify learning goals, reviewing existing goals and periodically checking whether they are met or need to be altered. 67 Reflection as support for career adaptability - ARTEL14 Adapting through self-directed learning and self-reflexiveness is closely related to individual reflection. Tools can help to sustain issues to be reflected upon and to cre- ate awareness for them [12, 18]. This combines self-directed and externally triggered reflection, for example by setting up and periodically reviewing career goals in a tool. Career adaptability by learning through interactions at work can benefit from sup- port for collaborative reflection. Tools can help to create opportunities for reflection even if individuals cannot meet in person [15]. Individuals can support informal learn- ing of their peers by providing their experiences and insights or helping them to re- flect about their own learning. Additionally colleagues can reflect to support each other, for example, in coping with emotional work and/or stress and in exchanging best practices in dealing with difficult situations. A team can reflect collaboratively to improve their team performance and organize their learning efforts. 4 Applying the Concept: Reflective Learning Needs in Practice Our work is inspired by field visits, workshops and expert interviews at different Eu- ropean Public Employment Services (PES) agencies, including Germany, Slovenia and the UK. In an early phase of this work we are currently exploring needs and op- portunities for reflective learning as well as constraints and potential of implementing it in such workplaces. From this work we describe examples of challenges faced in many European PES and how reflection can be a key process in tackling them. 4.1 Supporting Change by Reflection on Training In one of the agencies (referred to as agency A in this paper) staff are supposed to change from providing advice and guidance to clients on benefits and job opportuni- ties to coaching them to become more self-directed and to take responsibility for their own future by proactively looking for ways to develop their skills and possible future career paths. To support this change staff receive a two-day training on coaching methods and related topics and an additional half-day session some time after training to support the application of the methods in practice. Despite this support, staff mem- bers reported that they had difficulties in implementing this new way of working, and that they were struggling in reaching good results from coaching their clients. This situation is an example of challenging work, and it shows how workers strug- gle with updating their personal knowledge base. Reflecting on their practice of using methods and tools of coaching can help PES practitioners to conduct more successful- ly the transition to be a coach and thus may make training more sustainable. This may approach mostly benefits from individual reflection of goals stemming from training and involving workers in this reflection continuously (by reminding them to reflect). 4.2 Supporting Interaction with External Stakeholders by Reflection In agency B the government requires PES staff to cooperate closely with employers to enhance the conditions of the labour market, including the creation of new jobs, new 68 Reflection as support for career adaptability - ARTEL14 fields of employment and career opportunities. Staff are motivated to adapt to this strategy, but also told us that this does not come easy and that there is a need for good practices in implementing it. Some reported that talking to colleagues from other sub- sidiaries had given them insights into how they might improve this work. Becoming a co-operator with employers can be seen as an example of challenging work, and from the feedback of practitioners we can see which discrepancies it caus- es. We can also see that there is a desire to engage in exchange with others to reflect on such discrepancies. Collaborative reflection on their work with employers can therefore be seen as a means to make sense of typical challenges in this work, to ex- change work practices and to learn from each other. 5 A Prototype for Reflection Support The scenarios above show that support needs for learning about challenges Public Employment Services (PES) practitioners face are diverse, and that support for sus- taining experiences, reflecting upon them, sharing them and finding similar experi- ences need to be close to work tasks. To explore how such support can create impact in PES agencies we created a mobile proto- type supporting the reflective learning sce- narios describe above. Using mobile devic- es makes support independent from corpo- rate IT infrastructures (which are usually hard to access away from the office in PES) and enables users to use the tool when and where they want, for example after talking to employers or after work, e.g. while using public transportation on the way home. In the prototype users can write personal notes about experiences at work (upper part of Fig. 1) and they can enter reflections multiple times about these notes (bottom part of Fig. 1), including an assessment of how they feel about the experience (see the smiley icons in the bottom half of Fig. 1). The prototype also includes an easy to use sharing feature to enable collaborative reflection. To enhance personal engage- ment in collaborative reflection, when shar- ing content with colleagues the system offers users the opportunity to choose from predefined questions (or create a new ques- Fig. 1. The prototype allows users to write tion) to share together with the content. notes and to reflect on them. All notes and This aims at provoking reflection: For ex- reflections can be shared. 69 Reflection as support for career adaptability - ARTEL14 ample, as user might choose a question such as “Did you ever encounter a similar situation? What did you do?” when sharing the description of an issue. This may per- sonally impact colleagues, who feel personally invited to engage with the user sharing the content and motivated to help her. This may help to establish communities of practice helping each other and it facilitates collaborative reflection by engaging users in conversations about challenging work. The tool periodically prompts users individually or collaboratively to revisit past issues and reflections. This can be useful to capture changes in perspectives on expe- riences over time and the resulting insights leading to this change. For example, if a user from agency A experiences she cannot implement a certain aspect of the new coaching process, she may improve over time, also rating this experience more posi- tive after some time (see Fig. 1). It is also possible just to share one of the newer en- tries of a reflection with another user to enable collaborative reflection on specific aspects of the evolving situation. Using the tool in this way builds up an individual and collective knowledge base on aspects related to career adaptability. Users control when they are prompted for reflection: they can let the system (con- textually) determine when to prompt them or they can set reminders to reflect. This for example can be used to notify a user while she is using the bus on the way home and wants to reflect on situations she had experienced that day. This supports self- reflexiveness as part of career adaptability. The prototype provides novel features such as sharing personal questions with re- flection content and periodically promoting users for individual and collaborative reflection, which are directed towards engaging with challenging work and to support career adaptability. Future work will also aim to integrate its features into existing tools in order to better integrate reflection for career adaptability into daily tasks. 6 Discussion and Outlook We have presented ongoing work in supporting career adaptability in public admin- istration workplaces by reflection support. Our work is in its early stages, and we have created a concept for such support, situated it in needs of learning in PES organ- izations as typical examples of public administration and showed its feasibility by implementing a prototype. Next steps will include using the prototype with groups of PES practitioners in different agencies and improving the support it provides. In the ARTEL workshop we would like to discuss the concept and how it may be improved. 7 Acknowledgements The work described in this paper is part of the EmployID project funded by the Euro- pean commission in FP7 (project number 619619). We would like to thank all col- leagues in the project for their cooperation and our fruitful discussions. 70 Reflection as support for career adaptability - ARTEL14 8 References 1. Boud, D.: Reflection: Turning experience into learning. Kogan Page, London (1985). 2. Kolb, D.A.: Experiential learning: Experience as the source of learning and development. Prentice-Hall, Englewood Cliffs, New Jersey (1984). 3. Schön, D.A.: The reflective practitioner. Basic Books, New York (1983). 4. Cressey, P., Boud, D., Docherty, P.: Productive Reflection at Work. In: Boud, D., Cressey, P., and Docherty, P. (eds.) Productive reflection at work: Learning for changing organiza- tions. pp. 11–26. Routledge, New York (2006). 5. Eraut, M.: Informal learning in the workplace. Studies in continuing education. 26, 247– 273 (2004). 6. Prilla, M., Pammer, V., Krogstie, B.: Fostering Collaborative Redesign of Work Practice: Challenges for Tools Supporting Reflection at Work. Proceedings of the European Confer- ence on Computer Supported Cooperative Work (ECSCW 2013) (2013). 7. Hoyrup, S.: Reflection as a core process in organisational learning. Journal of Workplace Learning. 16, 442–454 (2004). 8. Giesbrecht, T., Schenk, B., Schwabe, G.: Learning with Facilitation Affordances: The Case of Citizens’ Advice Services. Proceedings of the Twenty Second European Confer- ence on Information Systems (ECIS 2014) (2014). 9. Brown, A., Bimrose, J., Barnes, S.-A., Hughes, D.: The role of career adaptabilities for mid-career changers. Journal of Vocational Behavior. 80, 754 – 761 (2012). 10. Brown, A.: A dynamic model of occupational identity formation. In: Brown, A. (ed.) Pro- moting Vocational Education and Training: European Perspectives. pp. 59–67. University of Tampere, Tampere (1997). 11. Lave, J., Wenger, E.: Situated learning: Legitimate peripheral participation. Cambridge University Press, Cambridge (1991). 12. Krogstie, B., Prilla, M., Pammer, V.: Understanding and Supporting Reflective Learning Processes in the Workplace: The RL@Work Model. Proceedings of the Eigth European Conference on Technology Enhanced Learning (EC-℡ 2013) (2013). 13. Knipfer, K., Wessel, D., Cress, U.: Reflection as a catalyst for organizational learning. Studies in continuing education. 35, 30–48 (2013). 14. Van Woerkom, M., Croon, M.: Operationalising critically reflective work behaviour. Per- sonnel Review. 37, 317–331 (2008). 15. Prilla, M., Degeling, M., Herrmann, T.: Collaborative Reflection at Work: Supporting In- formal Learning at a Healthcare Workplace. Proceedings of the ACM International Con- ference on Supporting Group Work (GROUP 2012). pp. 55–64 (2012). 16. Fleck, R., Fitzpatrick, G.: Teachers’ and tutors’ social reflection around SenseCam images. International Journal of Human-Computer Studies. 67, 1024–1036 (2009). 17. Xiao, L., Clark, S., Rosson, M.B., Carroll, J.M.: Promoting Reflective Thinking in Collab- orative Learning Activities. Proceedings of the 2008 Eighth IEEE International Conference on Advanced Learning Technologies. pp. 709–711. IEEE Computer Society, Washington, DC, USA (2008). 18. Isaacs, E., Konrad, A., Walendowski, A., Lennig, T., Hollis, V., Whittaker, S.: Echoes from the past: how technology mediated reflection improves well-being. Proceedings of CHI 2013. pp. 1071–1080. ACM, Paris, France (2013). 19. Raelin, J.A.: I don’t have time to think!” versus the art of reflective practice. Reflections. 4, 66–79 (2002). 20. Davis, E.A.: Scaffolding students’ knowledge integration: Prompts for reflection in KIE. International Journal of Science Education. 22, 819–837 (2000). 71 72 Boosting Informal Workplace Learning in Small Enterprises Miloš Kravčík, Kateryna Neulinger, Ralf Klamma Advanced Community Information Systems (ACIS), Informatik 5, RWTH Aachen University, Germany {kravcik, neulinger, klamma}@dbis.rwth-aachen.de Abstract Where participation of small enterprises in vocational education and training decreases, it risks obsolescence of their knowledge base compared to competitors. Currently we are participating in two projects that aim to address the issue of how to boost take-up of informal learning at the workplace: Learning Layers and BOOST. Previous projects [e.g., ROLE] show the importance of having personalised learning solutions with high relevance, high effectiveness and low barriers to use. Therefore we aim to provide predefined and customizable Personal Learning Environments that support awareness and reflection of users, especially workers in small enterprises. Keywords: Informal Workplace Learning, Personal Learning Environments. 1 Introduction Support of informal learning at the workplace is real issue and we attempt to address it in two projects: Learning Layers [1] and BOOST [2]. While the first one is dealing with the problems of scalability and scaffolding, the second one is focusing on small enterprises (up to 20 employees) and their needs. Both of them build on the outcomes of the former ROLE project [3], especially the technological platform that facilitates design and development of Personal Learning Environments (PLEs) [4]. Moreover, BOOST considers innovative methodologies from the BeCome [5] project that identify the Business Goals of small companies and manage the associated learning processes. The PLEs provide customized learning and training solutions that enable to meet the specified Learning Indicators. The overall aim is to support employees in training activities and to facilitate their personal development. For this purpose we want to integrate learning in their work processes. We develop widgets that should support awareness and reflection of various types of users in practice. In this context it is crucial to consider specific constraints and requirements of small companies, in order to make the developed solutions attractive and useful for all different roles: managers, trainers and employees. Our solutions support personal competence development at the workplace in all phases, i.e. planning, learning, and reflection. They help to identify business goals and existing competence gaps. Moreover, they recommend learning resources from existing repositories and suitable peers in communities of practice. 73 Boosting informal workplace learning in small enterprises - ARTEL14 2 BOOST Technical Prototype Our proposed solution should support awareness by augmenting informal learning with relevant information of the business goals, current and target competences of employees, time plans, learning resources and learning progress overviews on various levels (e.g. company, employee). Reflection is an important part of self-regulated learning that helps the users to evaluate their progress and to plan the next steps. These features had to be considered in the BOOST technical prototype, which is still work in progress. It includes this basic workflow: 1. Identify critical business goals in the company. 2. Select employees to address them. 3. Support their learning. 4. Monitor the learning progress of the company and of the individual employees. Our data model is hierarchical: 1. Business Goals (BGs – e.g. Web development). 2. Learning Indicators (LIs – e.g. Web design, information architecture). 3. Learning Resources (LRs – including learning materials, tools and peers). Figure 1: Employee Environment We distinguish 3 different user roles that have different characteristics and requirements: Manager (e.g. business manager, business advisor or consultant), Trainer (e.g. training manager, learning facilitator) and Employee. Manager specifies BGs for the company, decides which BGs are urgent and which of them are relevant for which employee. Moreover, this role can also assess employees and monitors their learning progress. Trainer describes LIs for selected BGs and the relevancy of LIs for individual employees, recommends LRs for the LIs, and chooses relevant Learning Repositories, where additional LRs can be found. Employee (Fig. 1) gets an overview of BGs and LIs assigned to her, together with the recommended LRs. According to 74 Boosting informal workplace learning in small enterprises - ARTEL14 the descriptions of LIs she can search for additional LRs in the predefined Learning Repositories and add them to her portfolio. She can also access the selected LRs in order to learn. Finally, she can monitor her learning progress. The functional requirements for competence management include: 1. Specification of relevant BGs (high level competences), their priorities and time scales. 2. Assignment of LIs (concrete competences) to each BG, considering also time scales. 3. Assignment of LRs to LIs. 4. Assignment of relevant BGs and LIs to employees. 5. Setting up target LI (proficiency) levels for relevant BGs for each employee, considering time scales. 6. Assessment of the start and current LI (proficiency) levels for the employee. 7. Monitoring the training progress in the company and also of each employee (considering also time scales). The functional requirements for the learning support are still relatively vague, as they will be more domain dependent: 1. Community support – sharing experience, communication, and collaboration. 2. Domain specific support – learning and assessment. 3. Annotation of learning resources assigned to LIs. 4. Considering preferences of individuals. 3 Conclusion and Future Work In the first year the BOOST consortium identified the main requirements and designed a solution. Afterwards we have developed the first version of the technical prototype, which has been evaluated in interviews with 15 stakeholders. Based on their outcomes the technical prototype will be updated and enhanced with additional features, including privacy requirements and personalization. The current version is suitable for companies with open environments, where employees do not mind seeing each other’s competences and learning progress. But in many companies more privacy is demanded, where employee can see just his or her data. Another important feature is assignment of timescales to business and learning goals as well as their monitoring and notifications. The new version will be tested in companies. Acknowledgments. The presented research work is partially funded by the German National Agency BiBB within the Lifelong Learning Programme Leaonardo da Vinci: “Business PerfOrmance imprOvement through individual employee Skills Training” (project no: DE/13/LLP-LdV/TOI/147655) and by the 7th Framework Programme large-scale integrated project “Learning Layers” (grant no: 318209). References [1] Learning Layers project http://learning-layers.eu/ [2] BOOST project http://www.boost-project.eu [3] ROLE project http://www.role-project.eu/ [4] Nussbaumer, A., Kravcik, M., Renzel, D., Klamma, R., Berthold, M., & Albert, D. (2014). A Framework for Facilitating Self-Regulation in Responsive Open Learning Environments. arXiv preprint arXiv:1407.5891. [5] BeCome project http://become.dedi.velay.greta.fr 75 76 Activity Recommendation App – Software to Evaluate the Usefulness of Improvement Recommendations Created in a Team Nils Faltin, Simon Schwantzer, Margret Jung IMC information multimedia communication AG, Saarbrücken, Germany {nils.faltin,simon.schwantzer,margret.jung}@im-c.de Abstract. The Activity Recommendation App supports employees in individual and collaborative reflection by capturing discussions and solutions for problems that need to be solved. The app enables employees to record personal experi- ences with the solutions. Based on these experiences the usefulness of a rec- ommendation can be re-evaluated in order to approve, update, or discard the recommendation. The application was successfully evaluated in coaching em- ployees in learning time management techniques. Keywords: ARA – Activity Recommendation App· soft skills improvement· recommendation evaluation· solution· experiences· MIRROR Spaces Frame- work· time management coaching 1 Introduction Reflection on work experiences can lead to new insights and ideas how to handle work situations better in the future. But the capturing of experiences during work and the reflection on this data is only half the way for a successful improvement. The other half is the creation of a viable reflection outcome and the validation of this out- come when it is applied in practice (see [1]). Based on this validation, a change can be approved, reverted, or improved and validated again. Whilst a lot of applications support users to capture data during work in order to pro- vide it in a subsequent reflection session, the second half of the reflection cycle is often left unsupported. To also cover this part, the Activity Recommendation App (ARA) was created in the MIRROR project [2]. It supports the discussion of im- provement ideas in an individual or collaborative reflection session and frames the outcome as recommendation. The app allows capturing personal experiences relating to active recommendations and viewing other members’ experiences if a recommen- dation targets a team. Finally, the ARA supports the evaluation of a recommenda- tions’ usefulness when applied in practice, in order to enable its improvement or sus- pension. By providing these features, the Activity Recommendation App aims to im- prove the application of insights gained from reflection on work. 77 Activity recommendation app - ARTEL14 2 Overview of the Main Functionalities The recommendation is created in an individual or a collaborative reflection session. Major elements of this session are the identification of the concrete issue and a viable solution for this issue. Texts, files, or data from other MIRROR applications can be attached to be used as evidence to back the comprehensibility of a recommendation. The events during the discussion are listed as a kind of minutes. Measurement criteria can be selected to evaluate the usefulness of the recommendation. Before publishing the recommendation, a target person/group is selected and invited to try the new solu- tion. A concrete scenario could look like this (cf. [3]): A team uses ARA to find a solution for their common problem of overtime spent for pending projects. They agree that frequent interruptions can be one reason for this (issue). In the scenario the teams’ solution is to implement three hours of quiet working time a day and to avoid inter- ruptions during that period (recommended solution). Fig. 1. A recommendation in ARA with one experience being entered To test the recommendation in practice, personal experiences are written down to decide about how well the recommended solution applied (see Figure 1). Users cap- ture their experiences by noting down a comment and by rating how well the solution worked (1 to 5 stars). In addition they can record the effort (e.g., the minutes of work- ing time required) and the benefit (e.g., the number of completed tasks) of applying the solution. These experiences are shared with the other members of the target group to benefit from the application in a group. To evaluate a recommendation, the app allows users to view all experiences with an aggregation of the ratings, efforts and benefits captured. All this can then be used to get an overview how well the solution works in practice to be taken as a basis for the decision if the solution should be kept, updated or discarded. 78 Activity recommendation app - ARTEL14 In the continuation of the exemplary scenario, the team discusses the recommenda- tion’s weak points (due to the captured experiences) during the regular team meeting and agrees on adapting it in respect to the selected period in time. It is then re- evaluated, re-discussed, and finally marked as solved when team agrees about a well- functioning final solution. 3 Evaluation & Outlook A summative evaluation of the ARA took place at our company IMC over a period of six weeks. Ten staff members took part in a time management coaching. The ap- proach combined the usage of a computer activity tracking tool and the ARA with a weekly coaching session. In the weekly coaching sessions the coach and the coachee reviewed the individual progress, adjusted the time management rules if not appropri- ate anymore, or, when the particular goal has been achieved and the new behaviour has been adopted, decided that no further practice regarding that goal is needed. The Activity Recommendation App served well as a support for learning time man- agement by providing a data basis for the coaching sessions. It was used by coach and coachee to set time management goals and to document and monitor the progress in learning new time management techniques. Both benefited from the better preparation for the coaching sessions available with the notes in ARA. Furthermore, the app helped the coachees to focus their goals. Two things were missed concerning ARA: It lacks an interface optimized for smartphones and currently no reminder function is available which motivates the user to capture experiences. These shortcomings can be addressed in future development. The coach and several coachees also suggested forming peer groups to train time management techniques. They could then benefit from sharing experience data to compare own progress with that of others and learn from each other’s experiences. IMC has started a free online course for time management that includes usage of the ARA [4]. In addition to the course, learners can book a human tele-coach for a fee. References 1. Krogstie, Birgit, Michael Prilla, and Viktoria Pammer. “Understanding and Supporting Re- flective Learning Processes in the Workplace: The RL@Work Model.” In Proceedings of the Eigth European Conference on Technology Enhanced Learning (EC-℡ 2013), 2013. 2. MIRROR project for reflective learning at work, http://mirror-project.eu/ 3. Video: Team usage scenario for the Activity Recommendation App, http://vimeo.com/66798165 4. MOOC “Time Management“ (Gernan) on OpenCourseWorld, http://bit.ly/1ufnS0v 79 80 DDART: an Awareness System to Favor Reflection during Project-Based Learning Min Ji1, 2, Christine Michel1, 2, Elise Lavoué1, 2, Sébastien George3 1 Université de Lyon, CNRS 2 INSA-Lyon, LIRIS, UMR5205, F-69621, France {min.ji; christine.michel; elise.lavoue}@liris.cnrs.fr 3 LUNAM Université, Université du Maine, EA 4023, LIUM, 72085 Le Mans, France sebastien.george@univ-lemans.fr 1 Introduction Our research aims to improve learners’ reflection and self-regulation in Project-Based Learning (PBL). Actually, we observe that the implementation of PBL in engineering schools, universities or professional training do not benefit from all its capacities, because it is often action (according to the Kolb’s learning cycle) which is favored to the detriment of reflection and personal experience [1]. Our approach considers Self- Regulated Learning (SRL) as a major component of PBL to bring learners to self- reflect on their experience and to apply metacognitive skills. We focus our work on the design and the development of a dashboard based on both reporting and activity traces [2]. The activity traces are automatically produced by the users’ actions and recorded directly by the LMS during the learning activities. The reporting traces are information reported by the learners themselves. Most dash- boards use only automatic activity traces to produce indicators. We state that the ag- gregation of these two types of traces allow producing more meaningful indicators for the learners [3]. Most existing dashboards are designed for the tutors to monitor the learners but they are rarely designed for the learners to support awareness during their activities. Furthermore, the indicators are mostly predefined and the users can rarely build their own indicators [3]. In this paper, we present the DDART system, which is composed of two specific tools: a reporting tool that aims at enhancing learners’ reflection dur- ing project-based learning and a tool to help learners to produce their own indicators for enhancing awareness during their project. These two tools are integrated into a same system (DDART) to enhance learners’ self-regulation thanks to personalized indicators presented on a dashboard. 2 A reporting tool and a dynamic dashboard We developed a Dynamic Dashboard Based on Activity and Reporting Traces (DDART). We chose to implement this dashboard as a plug-in of the Moodle plat- form. In our context, project members use the Moodle tools (wiki, forum, chat...) to 81 DDART: awareness system to favor reflection during project-based learning - ARTEL14 carry out the project. They are also asked to use a reporting tool to describe and keep traces of the project events. The reports are composed of semi-structured sentences so that this text information can be collected and analyzed automatically [2]. Two types of reports are possible: the goal report and the activity report. The former is written at the beginning of the project to assist learners to plan their project and to set the goals they want to achieve. The later can be filled in during the project. By completing the semi-structured sen- tences, learners can describe the ways they carry out the project, their states of mind, their judgments (who do what, when, where, with whom and how), their level of ac- quisition of knowledge and skills. The semi-structured sentences are more flexible than structured sentences and keep the possibility to collect organized and computable data. By applying this reporting tool, learners can self-reflect on how they carried out activities and learn how to organize their ideas and how to write effective reports. We designed a specific interface to help learners to build their own indicators (see Fig. 1). This interface is composed of three main parts: (1) the “parameters” part (see Fig. 1.a), on the left side, contains the list of all the parameters which are available for creating an indicator, (2) the “calculation” part (see Fig. 1.b and c), in the center, al- lows learners to place the parameters and view the indicator results and (3) the “visu- alization modes” part (see Fig. 1.d), on the right side. This user-friendly interface allows learners to create the indicators by dragging and dropping the parameters and the visualization mode. The calculation function is WYSIWYG: the results can be calculated in real-time so that learners can easily adjust the parameters. At last, the presentation of indicators on a dashboard provides awareness to the learners about the way they carry out the project and also about the building of knowledge and skills. Fig. 1. The interface to assist learners to create personalized indicators 82 DDART: awareness system to favor reflection during project-based learning - ARTEL14 The semi-structured sentences of the reporting tool (reporting traces) and the traces of use of Moodle (activity traces) are respectively recorded in an XML database (BaseX) and in a relational database (MySQL). These two kinds of traces are de- scribed according to the same five common entities: Learner, Tool, Activity, Time and Place. They are merged according to a common time basis and are stored into a transformed traces base. The transformed traces are used to produce indicators stored in a dedicated database [2]. An indicator is defined by 5 parameters: • X entity and Y entity: these parameters can be chosen by the learners among the instances of the five entities extracted from the transformed traces (Learner, Tool, Activity, Time, Place). These entities are used to specify the events the learners want to observe. • Value: this parameter sets the type of aggregation proposed to produce the data presented into the indicator. Four possibilities are proposed: frequen- cy, time interval, time spent, content. • Calculation function: the learners can refine the analysis of values by de- fining other mathematic formula based on sum, difference, comparison and average. • Visualization: DDART offers eleven visualization modes for learners (pie chart, bar chart, line chart, gauge chart, social network, scatter chart, area chart, table, tree map, combo chart and Gantt). 3 Conclusion In this paper, we have presented the basis of the DDART system. This system can help learners to collect, analyze and visualize their reporting and activity traces in the form of meaningful indicators. By allowing learners to create their own indicators, we aim at making them learn how to regulate their learning activities. The traces collect- ed in the reporting tool allow the construction of advanced indicators that can help learners to build metacognitive skills. For example, the indicators can support the analysis of behavior by comparing the learners’ feeling about their activities (subjec- tive) with the realization mode of the activities (objectively recorded by the system). References 1. Thomas, J., Mengel, T.: Preparing project managers to deal with complexity – Advanced project management education. International Journal of Project Management. 26, 304- 315 (2008). 2. Ji M., Michel C., Lavoué E., George S. Combining Activity Traces and Reporting Traces to Support Self-Regulation Processes. 13th IEEE International Conference on Advanced Learning Technologies (ICALT 2013), Beijing, China, 15-18 July 2013, p. 432-436. 3. Ji M., Michel C., Lavoué E., George S. DDART, a Dynamic Dashboard for Collection, Analysis and Visualization of Activity and Reporting Traces. 9th European Conference on Technology Enhanced Learning (EC-TEL 2014), Graz, Austria, 16-19 September 2014, to be published. 83 84 LARAe: Learning Analytics Reflection & Awareness environment Sven Charleer, Jose Luis Santos, Joris Klerkx, and Erik Duval Dept. of Computer Science, KU Leuven Leuven, Belgium {Sven.Charleer,JoseLuis.Santos,Joris.Klerkx,Erik.Duval}@cs.kuleuven.be Abstract. Exploring and managing the abundance of data that Learn- ing Analytics generate is a challenge for both teachers and students. This paper introduces a Learning Dashboard that provides an overview, context and content of learner traces to help students with awareness of feedback and progress, and assist teachers with monitoring student effort and outcomes to intervene where needed. Keywords: learning analytics, learning dashboards, awareness, infor- mation visualization, effort, intervention, inquiry-based learning 1 Introduction The purpose of Learning Analytics is understanding and optimizing learning and the environments in which it occurs [1]. Through dashboards, Learning Analytics can help support both teacher and students [2]. Learning Dashboards can rely on many different ways of visualizing raw analytics data e.g. bar, star and bubble charts, interactive histograms, parallel coordinates etc [2]. These visualization techniques can provide broad insights on student activities [3, 4]. By adding teacher traces, our visualization also attempts to provide awareness of feedback to improve its supportive role for both student and teacher. This abundance of data can be abstracted to the essentials [5, 6], but context and content can help provide deeper insights [7]. Following the visual information- seeking mantra of “Overview first, zoom and filter, then details-on-demand” [8], our dashboard presents users with an abstract overview while still retaining a sense of context and providing access to the details. 2 LARAe: Design & Implementation LARAe visualizes traces gathered from 38 engineering students, teachers and ex- ternal participants in an open User Interfaces course. Students worked in groups of 3 and reported weekly through blog posts, comments and Twitter. The course generated 419 blog posts, 1580 comments and 538 tweets. 85 LARAe: Learning Analytics Reflection & Awareness environment - ARTEL14 Fig. 1. LARAe: A. Overview, B. Activities, C. Thread view Every activity is represented by a circle (Figure 1.B) which provides direct access to the related content (e.g. blog post, comment, tweet, retweet). Activities are sorted chronologically, from top left to bottom right. Gradient color values (see Figure 1.A) help recognize the age of an activity. A table (Figure 1.B) structures the activities by student group and type. Every column represents an activity type, every row a student group. The user can sort the data by any activity type. Both activity age and amount help facilitate awareness of (in)active groups. As teaching staff feedback was deemed important by both student and teacher, a second table visualizes activities of teacher activity in a similar way. Context plays an important role in understanding the activities e.g. a com- ment without its surrounding discussion is difficult to assess. We propose a “fo- cus+context” [9] solution which consists of 2 parts: highlighting related events (Figure 1.B) and displaying the content within a thread view (Figure 1.C). Highlighting related activities helps the user to instantly become aware of the distribution of an activity thread across the class e.g. selecting a blog post will highlight what groups provided most contributions. Simultaneously, the thread view shows the content of each related activity, helping assess the quality of the quantitative data. Visualizing discussion thread size can help students discover interesting threads. Teachers might understand low thread size as an indication for need of intervention. The attribute thread size is indicated by a number in each circle (Figure 1.B). 86 LARAe: Learning Analytics Reflection & Awareness environment - ARTEL14 LARAe is a web application developed using HTML5, JavaScript and D3.js1 running on a Node.js2 web service and MongoDB3 database. It supports both the proprietary API and Tin Can API4 . It can easily be extended to support other APIs. The dashboard is designed to run on large displays, desktop computers and tablets. It is available at http://ariadne.cs.kuleuven.be/LARAe/. The dashboard has also been deployed in an inquiry-based learning setting, visualizing the learner traces gathered from the weSPOT Inquiry system5 [10]. Acknowledgment The research leading to these results has received funding from the European Community’s Seventh Framework Programme (FP7/2007- 2013) under grant agreement No 318499 - weSPOT project. References 1. Siemens, G., Long, P.: Penetrating the fog: Analytics in learning and education. Volume 46., Boulder, CO, USA, EDUCAUSE (2011) 30–32 2. Verbert, K., Duval, E., Klerkx, J., Govaerts, S., Santos, J.L.: Learning analytics dashboard applications. American Behavioral Scientist 57(10) (2013) 1500–1509 3. McAuley, J., O’Connor, A., Lewis, D.: Exploring reflection in online communities. In: Proceedings of the 2Nd International Conference on Learning Analytics and Knowledge. LAK ’12, New York, NY, USA, ACM (2012) 102–110 4. Donath, J.: A semantic approach to visualizing online conversations. Communi- cations of the ACM 45(4) (April 2002) 45–49 5. Xiong, R., Donath, J.: Peoplegarden: creating data portraits for users. In: Proceed- ings of the 12th annual ACM symposium on User interface software and technology, ACM (1999) 37–44 6. Arnold, K.E., Pistilli, M.D.: Course signals at purdue: Using learning analytics to increase student success. In: Proceedings of the 2nd International Conference on Learning Analytics and Knowledge, ACM (2012) 267–270 7. Charleer, S., Klerkx, J., Odriozola, S., Luis, J., Duval, E.: Improving awareness and reflection through collaborative, interactive visualizations of badges. In: Proceed- ings of the 3rd Workshop on Awareness and Reflection in Technology-Enhanced Learning. Volume 1103 of CEUR Workshop Proceedings., CEUR-WS.org (2013) 69–81 8. Shneiderman, B.: The eyes have it: a task by data type taxonomy for information visualizations. In: IEEE Symposium on Visual Languages, IEEE (1996) 336–343 9. Furnas, G.W.: A fisheye follow-up: Further reflections on focus + context. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’06, New York, NY, USA, ACM (2006) 999–1008 10. Mikroyannidis, A., Okada, A., Scott, P., Rusman, E., Specht, M., Stefanov, K., Boytchev, P., Protopsaltis, A., Held, P., Hetzner, S., Kikis-Papadakis, K., Chaimala, F.: wespot: A personal and social approach to inquiry-based learning. Journal of Universal Computer Science 19(14) (2013) 2093–2111 1 http://d3js.org 2 http://nodejs.org/ 3 http://www.mongodb.org 4 http://tincanapi.com/ 5 http://portal.ou.nl/documents/7822028/f475d712-5467-40ea-968c-5aa00d951400 87