=Paper=
{{Paper
|id=Vol-1238/artel14
|storemode=property
|title=None
|pdfUrl=https://ceur-ws.org/Vol-1238/artel14.pdf
|volume=Vol-1238
}}
==None==
https://ceur-ws.org/Vol-1238/artel14.pdf
Please refer to these proceedings as
Milos Kravcik, Alexander Mikroyannidis, Viktoria Pammer, Michael Prilla,
Thomas Ullmann, Fridolin Wild (Eds.): Proceedings of the 4th Workshop on
Awareness and Reflection in Technology Enhanced Learning. In conjunction
with the 9th European Conference on Technology Enhanced Learning: Open
Learning and Teaching in Educational Communities. Graz, Austria, Septem-
ber 16, 2014. Available online at http://ceur-ws.org/.
c 2014 for the individual papers by the papers’ authors. Copying permitted for private
and academic purposes. Re-publication of material from this volume requires permission
by the copyright owners.
The front-cover was created by Harriett Cornish (The Open University, KMi).
Addresses of the editors:
Thomas Daniel Ullmann Alexander Mikroyannidis
Knowledge Media Institute Knowledge Media Institute
The Open University The Open University
Walton Hall Walton Hall
Milton Keynes Milton Keynes
MK7 6AA MK7 6AA
United Kingdom United Kingdom
t.ullmann@open.ac.uk alexander.mikroyannidis@open.ac.uk
Viktoria Pammer Fridolin Wild
Knowledge Technologies Institute Knowledge Media Institute
Graz University of Technology The Open University
Inffeldgasse 21A Milton Keynes
8010 Graz MK7 6AA
Austria United Kingdom
viktoria.pammer@tugraz.at fridolin.wild@open.ac.uk
Michael Prilla Milos Kravcik
Institute for Applied Work Science RWTH Aachen University
Ruhr University of Bochum Advanced Community Information Systems (ACIS)
Universitaetsstr. 150 Ahornstr. 55
44780 Bochum 52056 Aachen
Germany Germany
michael.prilla@rub.de kravcik@dbis.rwth-aachen.de
Awareness and reflection in technology enhanced learning
Awareness and reflection can be viewed from the differing perspectives of the disciplines
informing Technology-Enhanced Learning, such as Psychology, Educational and Learning
Sciences, or Computer Science.
A common denominator can be identified, though, and enhancing ’awareness’ of learners
and other participants involved in learning processes by technology means augmenting for-
mal or informal learning experiences, typically in real-time, with information on progress,
presence, outcomes, workspace, and the like. Supporting ’reflection’ then means enabling
learners to capture, adapt, re-evaluate, and share experience in anticipation of future situa-
tions it will prove relevant to. Reflection supported digitally is a creative act, adding sense
and meaning to experiences made.
Combining support for ”awareness” and ”reflection” bears huge potential for improving
the learning and training with respect to utility, self-regulation, usability, and user experi-
ence.
The ARTEL workshop series brings - for the 4th time in 2014 - together researchers and
professionals from different backgrounds to provide a forum for discussing the multi-
faceted area of awareness and reflection.
For this year 2014, the workshop organizes discussion and meta-reflection amongst re-
searchers around the application of awareness and reflection in practice, its impact on
learners and questions of feasibility, and sustainability for awareness and reflection in ed-
ucation and work. This year’s workshop theme is:
How does computer-support for awareness and reflection need to be embed-
ded into practical (working or learning) contexts in order for learners to ben-
efit from such computer support?
Summary of the contributions
The #ARTEL14 workshop accepted 4 full papers, 1 short paper, and 4 demo papers. The
accepted papers discuss awareness and reflection in diverse settings, such as blue-collar
jobs and white-collar jobs, working in small enterprises, or learning at university level.
As for the full papers, Maurizio Megliola, Gianluigi Di Vito, Roberto Sanguini, Fridolin
Wild, and Paul Lefrere discuss in ”Creating awareness of kinaesthetic learning using the
Experience API: current practices, emerging challenges, possible solutions” an interface
specification for capturing in particular kinaesthetic learning experiences. The authors
also discuss a taxonomy of verbs describing handling and motion. Kinaesthetic skills are
in demand for instance in the manufacturing or maintenance sectors. The captured learning
experiences can be utilised to generate feedback to the learner.
Fridolin Wild, Peter Scott, Paul Lefrere, Jaakko Karjalainen, Kaj Helin, Ambjorn Naeve,
and Erik Isaksson situate their work ”Towards data exchange formats for learning expe-
3
Awareness and reflection workshop series
riences in manufacturing workplaces” in the manufacturing sector. This work models the
experiential and reflective learning process with a view on creating data exchange for-
mats, also discussing the necessity to - formally - talk about actions in the real world as a
complement to actions performed in virtual environments.
Andreas Janson, Sissy-Josefina Ernst, Katja Lehmann, and Jan Marco Leimeister have
as background for their work in ”Creating awareness and reflection in a large-scale IS
lecture - the application of a peer assessment in a flipped classroom scenario” univer-
sity learning in large-scale lectures (much more learners than instructors). They discuss
computer-supported peer assessment as possibility to induce reflection on learning con-
tent. In contrast to the previous two works, this contributions focusses mostly on cognitive
instead of motoric competences.
Angela Fessl, Gudrun Wesiak, and Granit Luzhnica showcase in ”Application overlapping
user profiles to foster reflective learning at work” the potential benefits of collecting activ-
ity logging data from multiple applications in a single user profile application. This work
is set on the background of white-collar knowledge workers, with desktop- and web-based
activity logging.
In their short paper, Michael Prilla, Oliver Blunk, Jenny Bimrose, and Alan Brown discuss
in ”Reflection as support for career adaptability: A concept for reflective learning in public
administration” reflection as learning mechanism to support professional identity change
as a means for organisational change in the context of public employment services.
The four demo papers are started off by the contribution by Milos Kravcik, Kateryna
Neulinger, and Ralf Klamma, in which the authors showcase widget-based personal learn-
ing environments for ”Boosting informal workplace learning in small enterprises”.
Nils Faltin, Simon Schwantzer, and Margret Jung present the ”Activity recommendation
app - software to evaluate the usefulness of improvement recommendations created in a
team”.
Min Ji, Christine Michel, Elise Lavoue, and Sebastien George demonstrate the ”DDART:
an awareness system to favor reflection during project-based learning”.
Last, but not least, Sven Charleer, Jose Luis Santos, Joris Klerkx, and Erik Duval present
the ”LARAe: Learning Analytics Reflection & Awareness environment”.
Awareness and reflection workshop series
The official workshop webpage can be found at http://teleurope.eu/artel14
The 4th Workshop on Awareness and Reflection in Technology-Enhanced Learning (AR-
TEL 2014) is part of a successful series of previous workshops.
• 3rd Workshop on Awareness and Reflection in Technology-Enhanced Learning (AR-
TEL13). Workshop homepage: http://teleurope.eu/artel13. Proceed-
ings: http://ceur-ws.org/Vol-1103/.
4
Awareness and reflection workshop series
• 2nd Workshop on Awareness and Reflection in Technology-Enhanced Learning (AR-
TEL12). Workshop homepage: http://www.teleurope.eu/artel12. Pro-
ceedings: http://ceur-ws.org/Vol-931/.
• 1st European Workshop on Awareness and Reflection in Learning Networks (AR-
Nets11). Workshop homepage: http://teleurope.eu/arnets11. Pro-
ceedings: http://ceur-ws.org/Vol-790/
• Augmenting the Learning Experience with Collaboratice Reflection (ALECR11).
Workshop homepage: http://www.i-maginary.it/ectel2011/index.
html
• 1st Workshop on Awareness and Reflection in Personal Learning Environments
(ARPLE11). Workshop homepage: http://teleurope.eu/arple11. Pro-
ceedings: http://journal.webscience.org/view/events/The_PLE_
Conference_2011/paper.html#group_Proceedings_of_the_1st_
Workshop_on_Awareness_and_Reflection_in_Personal_Learning_
Environments
To stay updated about future events, to share your research, or simple to participate with
other researchers, consider joining the group about Awareness and Reflection in Technology-
Enhanced Learning:
http://teleurope.eu/artel
We especially would like to thank the members of the programme committee for their
invaluable work in scoping and promoting the workshop and quality assuring the contri-
butions with their peer reviews.
September 2014 Milos Kravcik,
Alexander Mikroyannidis,
Viktoria Pammer,
Michael Prilla,
Thomas Ullmann,
Fridolin Wild
5
Organisation committee
Organisation committee
Milos Kravcik, RWTH Aachen University, Germany
Alexander Mikroyannidis, The Open University, United Kingdom
Viktoria Pammer, Graz University of Technology, Austria
Michael Prilla, University of Bochum, Germany
Thomas Ullmann, The Open University, United Kingdom
Fridolin Wild, The Open University, United Kingdom
6
Program committee
Program committee
Alexander Nussbaumer, Graz University of Technology, Austria
Carsten Ullrich, DFKI, Germany
Denis Gillet, EPFL, Switzerland
Ines Di Loreto, Université de Technologie de Troyes, France
Jaakko Karjalainen, VTT, Finland
John Cook, University of West of England, United Kingdom
Joris Klerkx, Katholieke Universiteit Leuven, Belgium
Kaj Helin, VTT, Finland
Kinshuk, Athabasca University, Canada
Marcus Specht, Open University, the Netherlands
Monica Divitini, IDI-NTNU, Norway
Paul Lefrere, CCA, United Kingdom
Peter Kraker, Know-Center, Austria
Peter Sloep, Open University, the Netherlands
Philippe Dessus, Universite Pierre-Mendes-France, France
Riina Vuorikari, European Commission JRC IPTS, Spain
Simon Knight, The Open University, United Kingdom
Stefan Trausan-Matu, University Politehnica of Bucharest, Romania
Stefano Bianchi, SOFTECO, Italy
Xavier Ochoa, Escuela Superior Politecnica del Litoral, Ecuador
7
Supporting FP7 projects
Supporting FP7 projects
http://www.mirror-project.eu
http://learning-layers.eu
http://www.tellme-ip.eu
http://wespot-project.eu
http://employid.eu
http://www.boost-project.eu
8
Contents
Awareness and reflection in technology enhanced learning 3
Summary of the contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Awareness and reflection workshop series . . . . . . . . . . . . . . . . . . . . 4
Organisation committee . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Program committee . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Supporting FP7 projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Creating awareness of kinaesthetic learning using the Experience API: current
practices, emerging challenges, possible solutions
Maurizio Megliola, Gianluigi Di Vito, Roberto Sanguini, Fridolin Wild, Paul Lefrere 11
Towards data exchange formats for learning experiences in manufacturing work-
places
Fridolin Wild, Peter Scott, Paul Lefrere, Jaakko Karjalainen, Kaj Helin, Ambjorn
Naeve, Erik Isaksson 23
Creating awareness and reflection in a large-scale IS lecture - the application of a
peer assessment in a flipped classroom scenario
Andreas Janson, Sissy-Josefina Ernst, Katja Lehmann, Jan Marco Leimeister 35
Application overlapping user profiles to foster reflective learning at work
Angela Fessl, Gudrun Wesiak, Granit Luzhnica 51
Reflection as support for career adaptability: A concept for reflective learning in
public administration
Michael Prilla, Oliver Blunk, Jenny Bimrose, Alan Brown 65
Boosting informal workplace learning in small enterprises
Milos Kravcik, Kateryna Neulinger, Ralf Klamma 73
Activity recommendation app - software to evaluate the usefulness of improve-
ment recommendations created in a team
Nils Faltin, Simon Schwantzer, Margret Jung 77
DDART: An awareness system to favor reflection during project-based learning
Min Ji, Christine Michel, Elise Lavoue and Sebastien George 81
LARAe: Learning analytics reflection & awareness environment
Sven Charleer, Jose Luis Santos, Joris Klerkx, Erik Duval 85
9
10
Creating awareness of kinaesthetic learning using the
Experience API: current practices, emerging challenges,
possible solutions
Maurizio Megliola1, Gianluigi Di Vito1,
Roberto Sanguini2, Fridolin Wild3, Paul Lefrere3
1
Piksel Ltd., Italy
{gianluigi.divito,maurizio.megliola}@piksel.com
2
AgustaWestland, Italy
Roberto.Sanguini@agustawestland.com
3
The Open University, Knowledge Media Institute, Milton Keynes, UK
{fridolin.wild,paul.lefrere}@open.ac.uk
Abstract. We describe our use of the Experience API in preparing blue-collar
workers for three frequently arising work contexts, including, for example, the
requirement to perform maintenance tasks exactly as specified, consistently,
quickly, and without error. We provide some theoretical underpinning for modi-
fying and updating the API to remain useful in near-future training scenarios,
such as having a shorter time allowed for kinaesthetic learning experiences than
in traditional apprenticeships or training. We propose ways to involve a wide
range of stakeholders in appraising the API and ensuring that any enhancements
to it, or add-ons, are useful, feasible and compatible with current TEL practices
and tools, such as learning-design modelling languages.
Keywords: Experience sharing, xAPI, verbs.
1 Introduction
Apprenticeship today often includes the development of ‘kinaesthetic intelligence’,
i.e. the tactile (physical) abilities associated with using the body to create (or ‘do’)
something involving highly coordinated and efficient body movements. Prototypical
examples of this can be found in the fluid and precise motions of skilled dancers,
surgeons, and skilled blue-collar workers. Gardner (2011) links high bodily-
kinaesthetic intelligence to “control of one's bodily motions, the capacity to handle
objects skillfully, a sense of timing, a clear sense of the goal of a physical action,
along with the ability to train responses”.
Becoming a skilled kinaesthetic performer through traditional apprenticeship is to-
day largely conceptualised to take years, but this golden age of long-cycle training is
quickly disappearing. Professional training environments such as in manufacturing
face the challenge that the world of repetition that enabled long cycles of training to
be cost-justified is increasingly taken over by short-run or personalised production,
11
Creating awareness of kinaesthetic learning using the Experience API - ARTEL14
using advanced factory machinery that does not require physical dexterity (including
teachable robots that can emulate dextrous production-line workers), thereby not only
causing shifts in the demand for skills, but at the same time downsizing the economi-
cally acceptable time-to-competence.
Achievements in interoperability for technology-enhanced learning (TEL) over the
last decade are making it viable to develop TEL-based alternatives to a traditional
apprenticeship. For example, TEL is making it possible to author, exchange, and then
orchestrate the re-enactment of learning activities across distributed tools, using learn-
ing process and learning design modelling languages (Laurillard & Ljubojevic, 2011;
Fuente Valentín, Pardo, & Degado Kloos, 2011; Mueller, Zimmermann, & Peters,
2010; Koper & Tattersall, 2005; Wild, Moedritscher, & Sigurdarson, 2008).
What these modelling languages fall short of, however, is the ability to handle hy-
brid (human-machine) experiences, to teach machines, or to train people at a distance
– as required, for example, in workplaces that include robots or software agents, or in
workplaces that include distributed participation across companies in a supply chain.
In particular, when it comes to bridging between the virtual and the real world, be-
tween digital and physical experiences, present means are ill equipped for supporting
the capturing, codification, and sharing of hybrid learning experiences.
Moreover, TEL-focused ways of capturing performance and collecting good prac-
tice are as of today still resource-intensive, placing a barrier to the spread of innova-
tion and hindering resilience of business.
That barrier can be somewhat reduced through the use of the present version of the
Experience API (ADL, 2013), and we claim could be further reduced through possi-
ble extensions and complements to the API, the need for which has become apparent
in the TELLME project.
The Experience API is is a novel interface specification designed to link sensor
networks together to enable the real-time collection and analysis of learning experi-
ences conducted in different contexts, with the aim of providing better interoperability
between the different types of participating educational systems and devices. Being
precondition to mining and modelling, it forms a centrepiece in the canon of next
generation techniques and technologies for capturing, codification, and sharing of
hybrid learning experiences.
This paper presents an overview of how the components and concepts of the Expe-
rience API are integrated within the TELL-ME project to foster tracking and analysis
of learning and training in three different manufacturing environments, namely Aero-
nautics, Furniture, and Textiles.
We describe within this contribution, how we deploy an open source Learning Re-
cording Store to allow for collecting and reporting learning across the various compo-
nents of the TELL-ME system. Moreover, starting with the Aeronautics industry, we
define a taxonomy of verbs of handling and motion for capturing in particular kinaes-
thetic learning experiences as required in the helicopter industry, when targeting
learning about a defined helicopter model and the connected standard maintenance
procedures.
12
Creating awareness of kinaesthetic learning using the Experience API - ARTEL14
2 TELLME Learning Locker
The ADL Experience API (short ‘xAPI’)1, formerly known as TinCan API, is an
extension of the Activity Streams2 specification, a format for capturing activity on
social networks, created by companies like Google, Facebook, Microsoft, IBM etc.,
that allows for statements of experience to be delivered to and stored securely in a
Learning Record Store (LRS). These statements of experience are typically learning
experiences, but the API can address statements of any kind of experiences a person
immerses in, both on- and offline.
While the core objects of an xAPI statement (Actor, Verb and Object) derive from
the core Activity Streams specification, the Experience API has many more defined
constructs for tracking information pertinent for the learner (with captured results
such as “score”, “success”, “completion”, “attempt”, and “response”), unlike the Ac-
tivity Streams spec which focuses on the publisher.
In its most basic application, the Experience API allows one system (the activity
‘provider’) to send a message (also known as the ‘statement’) to another system (aka
the ‘Learning Record Store’) about something a user has done. Up until now, this
process has mostly taken place inside the organisation’s Learning Management Sys-
tem. Anything we wanted to track had to be built as part of the LMS functionality, or
needed tailor-made integration. The Experience API now allows sending and receiv-
ing data between systems about what someone has done in a more openly defined
way (see Figure 1).
Figure 1. The experience tracking before and after the advent of the xAPI.
This is important, because via the Experience API, any system can send xAPI
statements using a standard connection method. This process of sending xAPI state-
ments can happen in systems behind the firewall or openly across the Internet using
secure connections.
1
http://www.adlnet.gov/tla/experience-api/
2
http://activitystrea.ms/
13
Creating awareness of kinaesthetic learning using the Experience API - ARTEL14
Within TELL-ME, we have implemented an instance of the open-source Learning
Locker3 Learning Record Store, a LAMP-based project using MongoDB, PHP, and
AngularJS, to begin collecting learning activity statements generated by xAPI com-
pliant learning activities and reporting on such data.
The REST WS interface to the LRS was made available to be used by any compo-
nent of the TELL-ME architecture to submit and retrieve xAPI statements. An exam-
ple of such xAPI triple submission (with POST), using curl4, is shown below:
curl -X POST --data @example.json -H "Content-Type: application/json" --
user
465ea716cebb2476fa0d8eca90c3d4f594e64b51:ccbdb91f75d61b726800313b2aa9f50
f562bad66 -H "x-experience-api-version: 1.0.0"
http://demos.polymedia.it/tellme/learninglocker/data/xAPI/statements
The example above is parameterized by a reference to a JSON file named exam-
ple.json: it contains the actual xAPI statement in form of an actor-verb-object triple:
{
"actor":{
"objectType": "Agent",
"name": "Gianluigi Di Vito",
"mbox":"mailto:gianluigi.divito@piksel.com"
},
"verb":{
"id":"http://activitystrea.ms/schema/1.0/watch",
"display":{
"en-US":"Watched"
}
},
"object":{
"id":"http://tellme-ip.eu/media/video/152"
}
}
Once statements are logged, they can be queried again, for example, as needed for
constraint validation to check, whether a user actually performed a certain required
action step – then requiring additional constraint checker components.
3
http://learninglocker.net/
4
curl - command line tool for transferring data with URL syntax: http://curl.haxx.se/
14
Creating awareness of kinaesthetic learning using the Experience API - ARTEL14
Figure 1. Learner interaction during maintenance of a helicopter
tail rotor: ‘user cleaned corrosion inhibitors’.
Same as any LRS implementation, the LearningLocker integrates data from multi-
ple systems. The ‘Activity Provider’ can be any sort of system, where the user per-
forms learning activity. Examples of such systems include search engines, social
bookmark engines, proxy servers, social networking platforms, webinars, blogs,
CRMs, as well as specific additional TELL-ME components. An example of the latter
is the TELL-ME smart player application under development, which can drop state-
ments about a user consuming a video (to the end or just the access of it). Another
example is ARgh!, the platform for the delivery of context-based Augmented Reality
content on mobile devices at the workplaces, which can drop statements about certain
action steps being executed by the learner: Figure 1 shows an example of such user
interaction with physical objects during training.
Integrating the Experience API requires not only providing an LRS, but also defin-
ing the set of ‘predicates’ for logging the experience statements in a meaningful way.
The ADL features a standard list of commonly occurring verbs5. This list, however, is
not bespoke to manufacturing and is restricted too much to generic, web-based learn-
ing activity, excluding interaction with objects of the real world – as required for
learning by experience in manufacturing. The same is the case for ADL’s extended
list of “Proposed Verbs for the Experience API”6. Once data are mapped to the ele-
ments of the Experience API statement (see Section 3), they can be captured in the
learning record store.
Figure 3 shows examples of such statements stored in the Learning Locker coming
from the ARgh! trials.
5
http://adlnet.gov/expapi/verbs/
6
http://bit.ly/1zGmAzn
15
Creating awareness of kinaesthetic learning using the Experience API - ARTEL14
Figure 2. Argh! statements stored in the Learning Locker
The Learning Locker also allows creating personalized queries by means of its re-
porting functions, combining data from both learning activities and workplace per-
formance, allowing linking of learning activity to performance.
One important aspect to be considered is the handling of security and the protec-
tion of the privacy of learners. In the Experience API, authentication is tied to the
user, not the content. The user can be any person or thing that is asserting the state-
ment. The user can be a learner, an instructor, or even a software agent and it can
authenticate with OAuth7, a commonly used access delegation mechanism employed
by many big names such as Google, Facebook, Salesforce, etc. that eliminates the
needs of sharing passwords between applications to exchange data. In this aim, the
Learning Locker supports authentication and it is integrated with OAuth 2.0, exposing
an API which allows 3rd parties to connect to the API via OAuth 2.0.
7
http://oauth.net/
16
Creating awareness of kinaesthetic learning using the Experience API - ARTEL14
Figure 3. Learning Locker Reporting page
3 The statement vocabulary (Aeronautics industry)
Each xAPI statement follows the syntax of providing a subject, predicate, and ob-
ject. While subjects and objects can vary, predicates (the ‘verbs’) are ideally rather
slowly changing and can be defined in advance.
From a cognitive linguistic perspective, there is prior work on defining verbs of
motion and handling – and clarifying their relation to the way humans’ cognitively
process them. Roy (2005) explains how humans learn words by grounding them in
perception and action. The presented theories are tested computationally by imple-
menting them into a series of conversational robots, the latest of which - Ripley - can
explain "aspects of context-dependent shifts of word meaning" that other theories fall
short of. To construct a vocabulary of verbs that is widely understood, easy to learn,
and natural in mapping, the work of Roy provides valuable insights: Roy postulates
17
Creating awareness of kinaesthetic learning using the Experience API - ARTEL14
that "verbs that refer to physical actions are naturally grounded in representations that
encode the temporal flow of events" (p. 391). She further details that the grounding of
action verbs follows the schema of specifying which force dynamics (out of a limited
set) apply and which temporal Allen relations operate (p. 391). Any higher-level
composition of action verbs, so Roy (p.392), can be traced back to and expressed in
terms of these fundamental temporal and force relations.
This provides an angle for defining and structuring the TELL-ME taxonomy of
verbs of handling and motion: it provides a basis for ordering from fundamental to
composite actions, also defining their similarities. Moreover, it offers insights on how
to map these verbs back to perception: it provides a rationale for how many visual
overlay elements are required for an augmented reality instruction to express a certain
motion verb primitive. For example, a verb 'pick up' requires the overlay visualisation
of a grabbing hand as well as a highlight of the object to be picked up, whereas the
motion verb 'move' consists of both 'pick up' and 'put down' actions, thus requiring
more visual elements to be specified.
Palmer et al. (2005) further discuss the "criteria used to define the sets of semantic
roles" for building verb classes. It provides insights into the argument structure of
framesets (and so-called role-sets) of verb classes (Palmer et al., 2005, section 3).
Chatterjee (2001) reviews the cognitive relationship between language and space.
He refers to Jackendoff in postulating that the "conceptual structure of verbs decom-
poses into primitives such as ‘movement’, ‘path’ and ‘location’ (p.57).
From an augmented reality perspective, there is additional prior work of relevance.
Robertson and MacIntyre (2009) provide a review of the state of the art in displaying
communicative intent in an AR-based system. The proposed taxonomy, however,
stays on a level of general applicability across all sorts of augmented reality applica-
tions and is lacking the level of handling and motion required for a particular work-
place, such as required in learning the maintenance of helicopters. The proposed cate-
gories (called ‘style’ strategies) are ‘include’, ‘visible’, ‘find’, ‘label’, ‘recognizable’,
‘focus’, ‘subdue’, ‘visual property’, ‘ghost’, and ‘highlight’ (p.149f). The communi-
cative goals signified by these styling operations for visual overlays are listed as
‘show’, ‘property’, ‘state’, ‘location’, ‘reference’, ‘change’, ‘relative-location’, ‘iden-
tify’, ‘action’, ‘move’, and ‘enhancement’ (p.148). Other than the work in cognitive
linguistics, here, the focus is clearly defined from a technical and not user angle: help-
ing the user to identify or move an object is on the same level as setting labels.
In the aeronautical field, maintenance operations must be carried out according to
official documents issued by the design authority of the aircraft. Such document is
called the Maintenance Publication. Usually, it is organised inside the Interactive
Electronic Technical Publication (IETP). The AECMA S1000D (European Associa-
tion of Aerospace Constructors S1000D) is an international standard for development
of IETP, utilizing a Common Source Data Base (CSDB). The standard prescribes
rules to name, define, and code everything that is necessary to carry out maintenance
activities in terms of:
• aircraft model;
• systems and subsystems of the aircraft;
• maintenance tasks;
18
Creating awareness of kinaesthetic learning using the Experience API - ARTEL14
• location of components;
• tools;
• additional documentations;
• miscellaneous.
Every maintenance task is identified by a unique code referring to the Task Cate-
gory (e.g. ‘servicing’, ‘repairs’, ‘package’), the Maintenance Activity within the cate-
gory (‘drain’, ‘fill’, ‘remove’, ‘clean’, etc.), and - finally - to the definition which
gives the procedure and data necessary to carry out maintenance tasks on a specific
system or component. The code allows retrieving both procedures and data necessary
to e.g. fill containers with fuel, oil, oxygen, nitrogen, air, water, or other fluids.
For the TELL-ME taxonomy of verbs of handling and motion the following pre-
liminary list was compiled by pilot partners, see Table 1. The initial taxonomy pre-
sents the verbs of handling and motion as required in the helicopter industry for a
defined helicopter model and the connected standard maintenance procedures. All the
actions are handled by the actor ‘certified staff’, which therefore has not been includ-
ed in the table.
Activity Verb Objects
Operation Loaded Cargo
Unloaded Cargo
Servicing Filled Liquid or gas (fuel, oil,
oxygen, nitrogen, air, water).
Drained Liquid (fuel, oil, water).
Released (pressure) Gas (oxygen, nitrogen).
Lubricated System, equipment, component, or item.
Cleaned Surface
Applied (protection) Surface
Removed (ice) Surface
Adjusted System, equipment or component.
Aligned System, equipment or component.
Calibrated System, equipment or component.
Inspected (keep Product, system, equipment or compo-
serviceable) nent.
Changed Liquid (fuel, oil, water).
Gas (oxygen, nitrogen).
Examination, tests Examined (visual) Product, system, equipment, component
and checks equipment, component, or item.
Tested (operation / System, equipment or component.
function)
Tested (function) System, equipment or component.
Tested (structure) Structure
Designed (data / System, equipment, or component.
19
Creating awareness of kinaesthetic learning using the Experience API - ARTEL14
tolerance check)
Monitored (condition) Product, system, equipment or compo-
nent.
Disconnect, re- Disconnected Equipment, components, or items.
move and disas- Removed Equipment, components, or items.
semble proce-
dures Disassembled Equipment, components, or items.
Opened for access Panels or doors (engine bay doors, land-
ing gear doors, etc.).
Unloaded (download Items.
software)
Repairs and locally Added material Product, equipment, component or item.
make procedures Attached material Product, equipment, component or item.
and data
Changed (mechanical Structure, surface.
strength / structure of
material / surface
finish of material)
Removed Material
Repaired Damaged product, system, equipment
or component.
Assemble, install Assembled Equipment, components and items.
and connect pro- Installed Equipment, components and items.
cedures
Connected Equipment, components and items.
Closed after access Panels or doors (engine bay doors,
landing gear doors, etc.).
Loaded (upload Items.
software)
Package, handling, Removed from Products, systems equipment or compo-
storage and trans- (preservation materi- nents,
portation al)
Put (in container) Item.
Removed (from Item.
container)
Kept serviceable Products, systems, equipment, or com-
(when in storage) ponents.
Moved (when in stor- Products, systems, equipment, or com-
age) ponents.
Prepared for use Item
(after storage)
Table 1. Activities and their predicates (plus objects).
20
Creating awareness of kinaesthetic learning using the Experience API - ARTEL14
4 Conclusion and outlook
In this contribution, we have presented an overview of how the concepts of Experi-
ence API can be applied in a manufacturing environment, precisely in the Aeronautics
industry. In the context of the TELL-ME project, three different taxonomies of verbs
of handling and motion were prepared or are currently being prepared by the pilot
partners, one of which has been described in this paper (from the helicopter industry).
An instance on the open-source Learning Locker Learning Record Store was made
available to any TELL-ME component, allowing them storing and retrieving xAPI
statements about the workers activities.
For the future development, we intend to add preliminary taxonomies for the other
two industry sectors, further mature them, and address issues related to the analytics
and statements interpretation, going further ahead in understanding data and how to
find meaning from it (with the help of statistical analysis). In addition, we plan to
explore how to extend (keyword: ‘constraint checker’) or complement the Experience
API to handle emerging challenges such as the need to consider the ergonomics and
learning needs of human-robot workspaces. This will involve conversations and col-
laborations with multiple stakeholders, such as TEL vendors and users, standards
bodies, and participants in robotics projects and projects involving or affected by
automation – as envisioned for the Factory of the Future projects.
Acknowledgements
The research leading to the results presented in this contribution has received fund-
ing from the European Community’s Seventh Framework Programme (FP7/2007-
2013) under grant agreement no 318329 (the TELL-ME project). The authors would
like to express their gratitude to the partners who have been involved in the related
research work in TELL-ME.
References
1. Gardner, H. (2011): Frames Of Mind: The Theory of Multiple Intelligences, Basic Books,
first published in 1983.
2. Laurillard, D.; Ljubojevic, D. (2011): Evaluating learning designs through the formal rep-
resentation of pedagogical patterns, In: Kohls & Wedekind (Eds.): Investigations of E-
learning Patterns: Context Factors, Problems, and Solutions, IGI global.
3. Fuente Valentín, L. de la; Pardo, A.; Delgado Kloos, C. (2011): Generic service integration
in adaptive learning experiences using IMS learning design, In: Computers & Education,
57:1160-1170.
4. Mueller, D.; Zimmermann, V.; Peters, J. (2010): ISURE: Guideline for the integration of
instructional models with open educational content in IMS Learning Design, deliverable
d5.2 of the iCoper consortium.
5. Koper, R.; Tattersall, C. (2005): Preface, In: Koper, Tattersall (Eds.): Learning Design: A
handbook on modelling and delivering networked education and training, Springer: Berlin.
21
Creating awareness of kinaesthetic learning using the Experience API - ARTEL14
6. Wild, F.; Mödritscher, F.; Sigurdarson, S. (2008): Designing for Change: Mash-Up Per-
sonal Learning Environments, In: eLearning Papers, 9 (2008).
7. Dillenbourg, P. (2011): Design for Orchestration, In: Dillenbourg (Ed.): Trends in Orches-
tration: Second Research & Technology Scouting Report, deliverable d1.5 of the
STELLAR consortium.
8. ADL (2013): Experience API, version 1.0.1, Advanced Distributed Learning (ADL) Initia-
tive, U.S. Department of Defense.
9. Roy, D. (2005): Grounding words in perception and action: computational insights, In:
TRENDS in Cognitive Sciences, 9 (8):389-396.
10. Palmer, M.; Gildea, D.; Kingsbury, P. (2004): The Proposition Bank: An Annotated Cor-
pus of Semantic Roles, In: Computational Linguistics, 31(1):71-106.
11. Chatterjee, A. (2001): Language and space: some interactions, In: TRENDS in Cognitive
Sciences , 5(2):55-61.
12. Robertson, C.; MacIntyre, B. (2004): Adapting to Registration Error in an Intent-Based
Augmentation System, In: Ong, Nee (Eds): Virtual and Augmented Reality Applications in
Manufacturing, Springer: London.
22
Towards data exchange formats for learning
experiences in manufacturing workplaces
Fridolin Wild1, Peter Scott1, Paul Lefrere1, Jaakko Karjalainen2,
Kaj Helin2, Ambjoern Naeve3, Erik Isaksson3
1
The Open University, Knowledge Media Institute, Milton Keynes, UK
{fridolin.wild, peter.scott}@open.ac.uk
2
VTT Technical Research Centre of Finland,
Human factors in complex systems, Tampere, Finland
{jaakko.karjalainen, kaj.helin}@vtt.fi
3
Royal Institute of Technology (KTH), Stockholm, Sweden
amb@csc.kth.se
Abstract. Manufacturing industries are currently transforming, most notably
through the introduction of advanced machinery and increasing degrees of au-
tomation. This has caused a shift in skills required, calling for a skills gap to be
filled. Learning technology needs to embrace this change and with this contri-
bution, we propose a process model for learning by experience to understand
and explain learning under these changed conditions. To put this process into
practice, we propose two interchange formats for capturing, sharing, and re-
enacting pervasive learning activities and for describing workplaces with in-
volved things, persons, places, devices, apps, and their set-up.
Keywords: Experience sharing, activity model, workplace model, awareness,
augmented reality.
1 Introduction
The European (and global) manufacturing industry is currently undergoing signifi-
cant transformation and will continue to change over the coming years. Intrinsically,
the increasing presence and ability of robots and advanced machinery in production
lines with their enhanced senses and increased dexterity (Frey & Osborne, 2013, p.38)
are what triggers this shift, bringing along rising degrees of computerisation of jobs
(ditto) and allowing for delocalisation of production.
Extrinsically, this transformation has started to cause a significant skills gap in the
EU (and globally) with – on the one side – the highest overall unemployment rates
observed in more than a decade (Eurostats, 2014; EC, 2013c, p. 2), especially
amongst young people (EC, 2013b; EC, 2013d), and an ever increasing risk of redun-
dancy for low and medium skilled workers in production.
On the other hand, several hundred thousand jobs in the EU remain unfilled, as
there is a shortage in highly skilled personnel in manufacturing (EC, 2013a, p.10).
Forecasts predict that this skills gap is likely to widen in coming years up to 2020
23
Towards data exchange formats for learning experiences - ARTEL14
(McKinsey, 2012, p. 45). In fact, manufacturing is currently one of the three sectors
hit most hard by this skills shortage in the EU (EC, 2013c, p. 5). Formal secondary
and tertiary education haven’t managed to create and won’t succeed in producing the
supply required, neither in numbers, nor with respect to matching skills profiles.
Moreover, high attrition rates in education have further eroded the foundation.
Technology enhanced learning has the potential to play an important role in over-
coming this existing skills gap in manufacturing – when applied effectively and when
motivating the development of competences in key areas required through the captur-
ing and re-enactment of learning activities.
Within this contribution, we first define a learning process model that is capable of
integrating classical (learning content oriented) and novel pervasive (Augmented
Reality and Internet of Things oriented) elements in learning at manufacturing work-
places. From there, we introduce a proposal for an activity modelling language (activ-
ityML) for representing activity descriptions required in augmented reality enabled
learning experiences. Moreover and in section 4, we introduce the needed workplace
modelling language (workplaceML), which can be used to describe the tangibles
(things, places, persons), configurables (apps, devices), and triggers (detectables,
overlays) of a particular workplace. We relate our work to precursors in Section 5 to
then wrap up the paper with an outlook and open research challenges.
2 Process model of learning by experience
New skills for new jobs not only demand an enhancement of the deep professional
skills to achieve a ‘master level of performance’, but also necessitate development
and upgrading of competence to innovate, for lifelong learning, and for learning
through social interaction (Wild et al., 2013, p. 12f).
Achieving a master level of performance and developing competence to innovate
in the sense of building up “the ability to generate ideas and artefacts that are new,
influential, and valuable” (FET, 2011) are – at least in manufacturing and at least for
small and medium enterprises – very closely intertwined.
Tacit Explicit
Tacit Socialisation Externalisation
Explicit Internalisation Combination
Table 1. Knowledge conversion modalities.
Similarly, the other two, namely lifelong learning and social learning competence,
both pay tribute to the observation that “people carry and create knowledge” and that
“any company knowledge management strategy must rely primarily on people, and
support [of] the knowledge creation chain” (Krassi and Kiviranta, 2013, p.29). Both
of them aim to facilitate “bi-directional tacit-explicit knowledge conversion” (Nona-
ka, 1994, p.19) along the four modalities listed in Table 1: externalisation (tacit-to-
24
Towards data exchange formats for learning experiences - ARTEL14
explicit), internalisation (explicit-to-tacit), socialisation (tacit-to-tacit), and combina-
tion (explicit-to-explicit).
While ‘competences’ are typically defined to subsume knowledge, skills and other
abilities, in the context of manufacturing – as the word already suggests – motoric and
artistic skills require special attention. Kinaesthetic learning elements relate in manu-
facturing environments to controlling own body movement and handling objects skil-
fully and timely (cf. Gardner, 1984: bodily-kinaesthetic intelligence).
With the rise of Wearable Computing, the Internet of Things, and Augmented Re-
ality, capturing and observing kinaesthetic performance becomes possible in a funda-
mentally different way, as, for example, pioneered in the fitness and health sector.
Reflective learning processes that cater for kinaesthetic and non-kinaesthetic ele-
ments can be broken down into five distinct process steps: enquire, mix, experience,
match, optimise (Wild et al., 2013, p.29ff). The steps do not necessarily prescribe a
single route and order, in which they should be taken, but are interconnected as indi-
cated in Fig. 1: it is a cyclical model with built-in support for experience tracking,
analytics, and guidance, supporting flexible mixes and dynamic optimization for on-
and off-the-job workplace learning.
Fig. 1 depicts the individual steps of this process model: at its core, blue-collar
workers experience learning in an episodic way (on and off the job). Experiencing
thereby relates to both re-enacting explicit learning activities as well as engaging in
open innovation activities.
Experiencing learning tightly interacts with enquiry: whenever novel needs arise or
(wicked) problems are encountered on the job, the enquiry step supports the user in
identifying relevant learning opportunities (such as gaps in knowledge, new learning
opportunities arising, etc.). In parts, this relates to navigational positioning support in
the workplace reference space’s skills taxonomy to clearly determine the competence
sought after. This, however, also relates to supporting discovery beyond existing or –
particularly relevant for SMEs – so-far tacit knowledge.
Through tracking of experiences made, potential competence gaps (ignorance) can
be uncovered, uncovering thereby either supported by the system in the matching step
(see below; aiming to help unveil shortcomings the user is unaware of) or – proactive-
ly, where awareness is given – through user enquiry.
Once needs or problems are identified, the mixing step comes into play: here, the
learner is supported in selecting relevant existing mixes or creating new and adapting
existing mixes. While standard problems have standard solutions, smart factories
enable their workers to rapidly compile mixes that satisfy needs, but at the same time
ensure documentation of knowledge, where it is created. Such mix is essentially a
serialized, activity-focused representation of the specific workplace and the jobs to be
enacted within it, instantiating an abstract workspace to a level of concreteness where
actions are named, locations resolved, and objects as well as tools uniquely identified.
Moreover, the activity mix models validation constraints, by which the matching
step can determine whether there is evidence that the user actually performed the
action steps as required. Constraints model learning flows including exception han-
dling. The constraint-matching step picks up on strategic performance indicators and
their defined tolerance boundaries set at design time, and connects them to the ob-
25
Towards data exchange formats for learning experiences - ARTEL14
served operational performance as tracked by the experience step. Reports for per-
formance analytics can be generated live, condensing performance records (from an
xAPI endpoint; ADL, 2013) into comprehensive reports, potentially contrasting per-
formance of individuals with (de-identified) benchmarks.
Optimisations then take such analytics data and performance benchmarks to rec-
ommend repetition, alternative resources, or even a change of path.
BLUE%COLLAR%WORKER%%
Traces%
EXPERIENCE(
%%Need,%
?% Enquire( %%Problem%
Ac9vity%
Mix( XML%incl.%
Constraints%
%Report,%
Match( %Analy9cs%
Sugges9on,%
Op9mise( Recommenda9on%
Fig. 1. Process model for learning by experience.
The bi-directional conversion between tacit and explicit knowledge is modelled in
the process loops between the big process step ‘experiencing’ and the smaller ones
‘enquiry’, ‘mix’, ‘match’, and ‘optimise’: explication converts tacit knowledge to
explicit as indicated by the outputs ‘traces’, ‘needs’, ‘activity mixes’, ‘analytics re-
ports’, and ‘recommendations’. When such outputs are used to scaffold a learning
experience, they are internalised. Remixing combines existing knowledge, and track-
ing and evidence recording helps with converting tacit to explicit knowledge. Moreo-
ver, activity mixes can involve socialisation and social sharing.
3 Modelling activities
A common representation format is key requirement for an efficient exchange of
activity mixes. What makes it particularly challenging to define activity mixes in a
process for learning by experience is that it requires not only orchestrating user inter-
action across multiple devices within a single activity, but also integrating the track-
ing of and reacting to user interaction across these devices and – even more so – their
different sensors. Validating that the user actually did something (like moving to a
particular location or like picking up a particular object in the real world), requires
specifying validation constraints that can be checked and that express which soft- and
26
Towards data exchange formats for learning experiences - ARTEL14
hardware sensors have to pick up on what user (or app) behaviour. For this we pro-
pose activityML (activity modelling language), an XML dialect.
Fig. 2 provides an overview on the conceptual model of activityML. Fig. 3 adds an
example activityML file. The root node is ‘activity’. Each activity needs to specify the
URL of the workplace description file, a name, and the language (in addition to the
unique activity id). The activity is then broken down into ‘action’ steps, each of them
being a self-contained unit, describing ‘summons’ for action chaining, ‘constraints’
for action validation, and ‘messages’ for communication, as well as ‘instruction’ to be
shown or the ‘app’ (widget or app) to be launched.
Fig. 2. Conceptual model of the activityML.
Moreover, styling information is linked using cascading style sheets over the action
‘type’ and the ‘device’ and its ‘viewports’ defined. Currently, there are three view-
ports defined: ‘objects’, ‘actions’, and ‘reactions’. They refer to particular areas of the
screen reserved for inserting actions and the related display data.
Each ‘action’ has a ‘predicate’, which is the verb required for inserting trace state-
ments to the xAPI (ADL, 2013) tracking endpoint. Each action can optionally specify
a ‘location’, i.e. a defined ‘place’ of the workplace model, in which it shall happen.
Chaining of actions is modeled through specifying for each action, which other ac-
tions it ‘summons’ – either when launching the action (‘onEnter’) or when events are
triggered (‘onTrigger’). The Boolean ‘removeSelf’ decides on whether the action is
removed from the viewport or sustained when the summons are executed. A timer can
be set to automatically trigger summons after a given interval (in milliseconds).
Summoning an action twice will first show, then remove it from the viewport (hence
‘toggle’). Each summoned toggle specifies the ‘viewport’ in which it is toggling an
action. The summons are also used to activate in a context-dependent way the ids of
actions relevant for the next step. By using the id of a tangible (e.g. thing or person)
as the id of an action step, for example, an object detection engine can trigger the
launching (or termination) of the according action.
27
Towards data exchange formats for learning experiences - ARTEL14
The ‘constraints’ define, how the system can validate that certain user interaction
and other observable conditions are given the way they were modeled. For example,
as depicted in Fig. 3, a constraint of type ‘onEnter’ can be defined that checks wheth-
er the learner has certain basic ICT skills. Constraints are specified in a given query
language (in the example: SQL) and they define their own action branching for ‘on-
Satisfied’ and ‘onViolated’ conditions.
To enable communication between devices and to allow for communicating with
overlays (as specified in the workplace model), ‘messages’ can be used: each ‘mes-
sage’ specifies, which ‘target’ (device, thing, person, …) and ‘id’ they want to com-
municate with. In case that the message is to a thing, the ‘overlay’ needs to be speci-
fied. Messages can also declare explicitly, which communication ‘channel’ they want
to use (e.g. a real-time presence channel ‘rpc’ or an ‘xapi’ endpoint).
Fig. 3. Example activityML code.
Fig. 3 provides an example mock activity model. In this code example, the activity
is broken down into five action steps, four of which are to be executed on a tablet PC,
while one will be launched on a pair of augmented-reality (see-through) glasses. The
user interaction starts with a welcome instruction to check the manufacturing order
28
Towards data exchange formats for learning experiences - ARTEL14
(auto-removed after 500ms) in which a constraint validation is performed checking
whether the user has the required skills (from a user profile).
If this constraint is validated, the user proceeds to finding the order sheet; other-
wise an error message is displayed. To support finding the order sheet object, an ac-
tion is launched on the glasses, which is toggled with sending the according id (‘ac-
tion15’). Once the object has been found in the viewfinder of the glasses, the next
action – playing multimedia instructions in the smart player app – follows, this now
again on the tablet PC.
4 Modelling workplaces
To create interoperability of applications interpreting activityML, a description of
the workplace is required in a defined interchange format. We propose for this work-
placeML, an XML dialect to describe the tangibles, configurables, and basic triggers
of a workplace. The ‘tangibles’ thereby refer to ‘things’, ‘places’, and ‘persons’, see
Fig. 4. The ‘configurables’ fall into two classes, namely ‘devices’ and ‘apps’.
Fig. 4. Conceptual model of workplaceML.
Finally, the ‘triggers’ group together both ‘detectables’ (such as markers) and the
primitives of ‘overlays’. The relationship between tangibles and triggers is crucial:
each tangible can specify the corresponding ‘detectable’ to determine how it can be
detected: it can name the id of a marker, the id of a feature cloud, or even the id send
29
Towards data exchange formats for learning experiences - ARTEL14
by an Internet of Things component such as an intelligent toolbox that monitors
through sensors which tools are taken out or put back in. Moreover, each tangible can
list the overlay primitives supported and configure them if required. For example, a
‘YesNo’ visual overlay does not require additional configuration: an app identifying
the tangible will automatically overlay a green circle when it is relevant to the current
action step (and a red cross, when not). This is different, for example, for an image
overlay primitive: in that case, the tangible needs to specify via ‘src’ the path to the
image to be displayed (and whether it shall be anchored to the detectable or to the
horizon).
Fig. 5. Example of a workplaceML file.
30
Towards data exchange formats for learning experiences - ARTEL14
Certain verbs of handling and movement can be predicates of an action step (e.g.
‘lift’ or ‘rotate’) and they overlay primitives can be enabled accordingly in the con-
figuration of the tangible.
The ‘configurables’ specify for each device the ‘owner’, a human-readable ‘name’,
the ‘type’ (e.g. ‘iPad mini’ versus ‘google glass’) and its unique id. This ensures that
messages can be delivered and actions can be launched and styled correctly. The
‘apps’ define the URL of the manifest file of e.g. a ‘widget 1.0‘ compliant or ‘Open-
Social’ compliant widget.
The code example presented in Fig. 5 now provides the required workplace infor-
mation on the tangibles (places, things, persons), configurables (apps, devices), and
triggers (detectables, overlays).
From the bottom to the top, first the overlay primitives are described, i.e. a generic
definition of which types of overlays exist and which modality they are overlaid in.
For example, there is a person sound and there is an image overlay.
Next, the detectables are defined: this enables a pre-trained marker (‘010’), a fea-
tureless object model (‘015’), and an event from an Internet of Things sensor (‘020’).
There are two types of configurables defined in this example: the devices (e.g. of type
‘ipad’ or ‘moveriobt200’) as well as the apps that can be launched, some of which
through calls to the device app to be launched, others as html5 widgets.
Finally, definitions of persons, places, and things follow. Here, each tangible can
further configure the overlay primitives described at the very end of the script. For
example, the thing ‘thehammer1’ is bound to the marker ‘010’ and configured to sup-
port image overlays using a picture of the hammer and setting the xyz-offsets as re-
quired.
5 Related work
In Naeve et al. (2014), we have presented generic, complementary deliberations
about workplace models as well as an earlier, less elaborate version of the interchange
format for activities proposed (p.48ff).
ACTIVITY-DL (Lanquepin, 2013; Barot et al., 2013) builds on the former
HAWAI-DL proposal of the same group and provides a hierarchical way to describe
tasks for virtual reality environments. While the task description is very advanced, the
language lacks capabilities for device and multi-sensor integration. ACTIVITY-DL
refers to its precursors MAD (Methode Analytique de Description; Sebillotte &
Scapin, 1994) and GTA (Groupware Task Analysis; Veer et al., 1996), both focusing
on analysing work tasks in interaction with user interfaces. While both provide con-
ceptual insights (e.g. on timing and on condition modelling), they do not provide
bindings against an interchange format.
6 Conclusion and outlook
In this contribution, we have rooted our motivation for creating the required ex-
change formats for capturing and sharing (kinaesthetic) learning experiences in manu-
31
Towards data exchange formats for learning experiences - ARTEL14
facturing workplaces. The transformation the industry is currently undertaking has left
a skills gap, which can be closed using learning technology apt to capture, share, and
guide in re-enacting innovative production activity. For this, we have described the
learning process and proposed two novel interchange formats for exchanging execut-
able descriptions of learning by doing activity and workplaces. The exchange formats
are implemented in the ARgh! prototype, a first glance of which is published in the
proceedings of the main conference (Wild et al., 2014).
In a world, where the time required for updating must be significantly smaller than
the half-life of knowledge documented, this becomes a key enabler for experience
sharing and a corner stone for success.
The specifications have already been tested against a range of storyboards of the
TELL-ME project and with participants of the joint European doctoral summer school
in TEL (JTEL’14). The upcoming user pilots in the TELL-ME are expected to lead to
further refinements. In particular, work is undergoing at the moment to further refine
the predicate vocabulary and fine-tune it to the three pilot workplaces tested (aviation,
furniture production, textile inspection and production). Moreover, the xAPI integra-
tion already feeds back to the constraint validation and further updates on query lan-
guage and reasoning are to be expected.
Acknowledgements
The research leading to the results presented in this contribution has received fund-
ing from the European Community’s Seventh Framework Programme (FP7/2007-
2013) under grant agreement no. 318329 (the TELL-ME project, http://tellme-ip.eu).
The authors would like to express their gratitude to the partners who have been in-
volved in the related research work in TELL-ME.
References
1. Frey, C.; Osborne, M. (2013): The future of employment: how susceptible are jobs to
computerisation?, unpublished manuscript, University of Oxford: Oxford Martin School.
2. European Commission (2013a): Annual Growth Survey 2012, COM(2011) 815 final.
3. European Commission (2013b): Europe 2020 Factsheet on Youth unemployment, online
at: http://ec.europa.eu/europe2020/pdf/themes/21_youth_unemployment.pdf
4. European Commission (2013c): Draft joint employment report 2014 (JER), COM(2013)
801 final, online at: http://ec.europa.eu/europe2020/pdf/2014/jer2014_en.pdf
5. European Commission (2013d): Working together for Europe’s young people: A call to
action on youth unemployment. COM(2013) 447 final.
6. Eurostats (2014): Unemployment statistics, online at:
http://epp.eurostat.ec.europa.eu/statistics_explained/index.php/Unemployment_statistics
7. McKinsey Global Institute (2012): The world at work: Jobs, pay, and skills for 3.5 billion
people, June 2012, McKinsey.
8. Wild, Fridolin; Scott, Peter; Da Bormida, Giorgio; Lefrere, Paul; Naeve, Ambjoern; Isaks-
son, Erik; Valdina, Alessandro; Nina, Manuel; Marsh, Jesse (2013): Learning By Experi-
ence: The TELL-ME Methodology (phase one), deliverable d1.2, TELL-ME consortium.
32
Towards data exchange formats for learning experiences - ARTEL14
9. FET (2011): Creativity and ICT, FET Consultation Workshop, Brussels, 28 November
2011. Report dated December 2011.
10. Krassi, Boris; Kiviranta, Sauli (2013): The unity of human and machine, In: VTT Impulse:
Science, 2013(1): 24-29.
11. Nonaka, I. (1994) A Dynamic Theory of Organizational Knowledge Creation, Organiza-
tion Science, Vol. 5, No. 1 (Feb., 1994), pp. 14-37.
12. Gardner, H. (1984): Frames of mind: the theory of multiple intelligences, Heinemann.
13. ADL (2013): Experience API, version 1.0.1, Advanced Distributed Learning (ADL) Initia-
tive, U.S. Department of Defense.
14. Naeve, A.; Isaksson, E.; Lefrere, P.; Wild, F.; Tobiasson, H.; Walldius, Å; Lantz, A.; Vii-
taniemi, J.; Karjalainen, J.; Helin, K.; Nuñes, M.J., Martín, J. (2014): Integrated industrial
workplace model reference implementation, deliverable d4.3, TELL-ME consortium.
15. Lanquepin, V.; Carpentier, K.; Lourdeaux, D.; Lhommet, M.; Barot, C.; Amokrane, K.
(2013): HUMANS: a HUman Models based Artificial eNvironments Software platform,
In: Proceedings of Laval Virtual VRIC’13, March 20-22, 2013 Laval, France, ACM.
16. Barot, C.; Lourdeaux, D.; Burkhardt, J.-M.; Amokrane, K.; Lenne, D. (2013): V3S: A Vir-
tual Environment for Risk-Management Training Based on Human-Activity Models, In:
Presence, Vol. 22, No. 1, Winter 2013, 1–19
17. Sebillotte, S.; Scapin, D. (1994): From users’ task knowledge to high‐level interface speci-
fication, International Journal of Human-Computer Interaction, 6:1, 1-15.
18. Veer, G.; Lenting, B.; Bergevoet, B. (1996): GTA: Groupware Task Analysis - Modeling
Complexity, In: Acta Psychologica 91(1996):297-322.
19. Wild, F.; Scott, P.; Karjalainen, J.; Helin, K.; Lind-Kohvakka, S.; Naeve, A. (2014): An
augmented reality job performance aid for kinaesthetic learning in manufacturing work
places, In: Open Learning and Teaching in Educational Communities, Proceedings of EC-
TEL 2014, Springer: Berlin.
33
34
Creating Awareness and Reflection in a Large-Scale IS
Lecture – The Application of a Peer Assessment in a
Flipped Classroom Scenario
Andreas Janson1, Sissy-Josefina Ernst1, Katja Lehmann1, Jan Marco Leimeister1,2
1
Kassel University, Germany
{andreas.janson, sissy.ernst, katja.lehmann,
leimeister}@uni-kassel.de
2
University of St.Gallen, Switzerland
JanMarco.Leimeister@unisg.ch
Abstract. Large-scale lectures are a typical way of teaching university students.
However, these lectures often lack interaction elements and do not foster
awareness and reflection in the learning process. This results in insufficient
learning outcomes such as learning satisfaction and success. Therefore, a new
approach to engage interaction in such large-scale lectures is the flipped class-
room concept which seeks to overcome these challenges by stimulating self-
regulated learning phases and improving interaction as well as awareness and
reflection in the presence phases of a lecture. However, it is still unclear how to
actually increase reflection and awareness through interaction in such learning
scenarios. For this purpose, we propose an application of a technology-
enhanced peer assessment that is carried out in large-scale information systems
lectures. Preliminary evaluation results suggest the potentials of this approach.
Thus, we are able to provide first theoretical and practical implications for the
application of a technology-enhanced peer assessment in large-scale lectures.
Keywords: Awareness, Reflection, Peer Assessment, Large-Scale Lectures,
Learning Success, Interaction, Feedback, Educational Objectives
1 Introduction
Large-scale lectures with an uneven lecturer-learner proportion (sometimes more than
100 learners per lecturer) are common in learning scenarios of universities [1]. These
lectures are characterized by high anonymity and suffer from a lack of interaction in
the learning process - not only among learners themselves but also among learners
and lecturers [2]. Often, this results in insufficient learning outcomes and brings about
unsatisfied learners [3, 4]. This development is alarming since fundamental elements
of learning success include the opportunity to ask comprehension questions in order
to get feedback, the possibility of sharing one's opinions concerning the learning con-
tent and of intensively reflecting on the learning content with colleagues [5, 6].
35
Creating awareness and reflection in a large-scale IS lecture - ARTEL14
Moreover, dealing and interacting with the learning content during the learning pro-
cess creates awareness and reflection regarding the learning process [7]. Additionally,
interaction and collaborative learning with peers are regarded as significant predictors
in terms of learning success [8] and positively influence the long-term satisfaction of
learners [9, 10]. Individual learning success verification, namely in the teaching-
learning process, provides individual feedback to learners [11]. This allows learners
and lecturers to identify missing knowledge and misunderstandings not during the
final exam, but rather early in the course of a continuous learning-progress monitoring
system [11] and moreover, create awareness for the relevant specific learning content.
Integrating assignments in class which create awareness and reflection to the specific
learning content are very complex and addresses the high cognitive level of educa-
tional objectives [12] supposed by Bloom [13] and Anderson et al. [14], which are
analyzing, evaluating and creating. However, the verification of those assignments is
time- and resource-consuming hence impossible to use in a large-scale lecture. Never-
theless, introducing interaction and feedback to create awareness and reflection and
moreover addressing educational objectives on a high cognitive level for individual
learning success measurement in a large-scale lecture is a widespread problem.
Didactic mechanisms are needed in order to overcome the above mentioned factors
characterizing traditional large-scale lectures. One promising possibility to enhance
interaction and feedback and moreover to address high cognitive levels of educational
objectives without massively increasing the workload of lecturers is the use of peer
assessment as didactic method [15]. By using peer assessment, learners give each
other feedback or credit points in terms of a performance during the learning process
according to specifically defined criteria. The goal of this paper is to describe the use
of peer assessment as interaction supporting component for addressing awareness and
reflection in a university large-scale lecture and ultimately for increasing learning
success. This paper therefore aims to answer the research question: How is a peer
assessment in a large-scale lecture designed to address interaction and to improve the
learning scenario? The contribution of this study is according to Gregor [16] a theory
of design and action that enables on the one hand practitioners to design learning
scenarios with a technology-enhanced peer assessment, and one the other hand de-
rives theoretical implications for future research in engineering IT-enabled learning
scenarios.
In order to answer the research question, the remainder of this paper is structured
as follows. First, we provide a brief overview of related work that is concerned with
our peer assessment. We then subsequently propose our application of the technolo-
gy-enhanced peer assessment in our learning scenario. Afterwards, we present our
first evaluation results and provide implications of our results in the discussion. In
section 6, we highlight limitations of our study and provide on this basis guidance for
future research, before the paper closes with a brief conclusion.
36
Creating awareness and reflection in a large-scale IS lecture - ARTEL14
2 Related Work
For a few years, awareness and reflection have been, in the context of technology-
enhanced learning, increasingly important and capture more attention since it has
been recognized that both are key factors in helping to provide personal support in
user-centric learning environments [17, 18].
The pedagogical approach aims at making learners aware of their learning behavior
and at the same time intends to empower learners in creating own personal learning
environments with individual learning resources while discovering their own learning
patterns.
2.1 Awareness
In this context, awareness plays a central role focusing on cognitive learning activities
and especially on non-observable behavior. Learners should familiarize themselves
with their own and individual cognitive processes such as goal-setting, self-evaluation
or help-seeking, in order to integrate them into self-regulated learning [19]. Students
are confronted and made aware of key actions of their own learning behavior with the
intention to make them become aware of their cognitive actions. Thereby, e-learning
tools with the possibility to personalize the learning process are able to support
awareness. Evaluations indicate that learners feel aware especially of their own efforts
and less about the effort of their group members and the members of other groups
[20].
2.2 Reflection
Reflection is an important key element in the learning activity as it allows implement-
ing continuous improvement in order to cope with complex and permanent changing
situations [21]. It is a meta-cognitive process which can be individual and also collec-
tive [22], and described as the conscious reevaluation of experience for the purpose of
guiding future behavior taking into account feeling, ideas and behavior as well [23].
In the context of technology-based learning, active reflection supports the examina-
tion of own achievements as well as the work of peers and pushes for a decentraliza-
tion process of problem-solving where learners are challenged and confronted with
existing knowledge [21] and finally able to create knowledge [24].
Finding out about a learner’s reflection can be supported by several platforms
where learners communicate by sharing reports, problems and solutions concerning
their work with peers [25]. Additionally, this exchange enables peers to learn from
their peers and at the same time to contribute own work. In this way, learners should
take more responsibility of their learning activities and efforts.
37
Creating awareness and reflection in a large-scale IS lecture - ARTEL14
2.3 Peer Assessment
In the context of awareness and reflection in the learning process, prior research has
shown that learners who interact with their lecturers and colleagues are more actively
involved in the learning process [26, 27] and achieved better learning outcomes [8].
The lecturer can assess the learning progress by means of the answers and provide
direct feedback. The learners have the opportunity to contribute their ideas and
thoughts, thus, also initiating new thought processes [28, 29].
The use of peer assessment in class is an essential possibility to introduce interac-
tion in a large-scale lectures and to provide formatively individual feedback in the
learning progress as well as corresponding interventions by means of technical-based
observation processes even in groups with a high number of learners [30, 31]. Moreo-
ver, the use of peer assessment is a favorable method to give learners extensive open-
ended free text assignments hence to address awareness and reflection, even in large-
scale lectures with more than 100 students, without massively increasing the lectur-
er’s workload. In the case of peer-assessment, learners give each other feedback or
credit points in terms of a performance or results during the learning process accord-
ing to specifically defined criteria [32]. Peer-assessment turns learners into experts
themselves and gives them a deeper understanding of the learning content [33].
The application of peer assessment in university teaching brings about, above all,
the following advantages opposed to an evaluation solely done by the lecturer:
1. Logistically: Lecturers can save precious time if learners give each other feedback
and evaluate each other’s academic performance [33].
2. Pedagogically: The learners get a deeper understanding of the learning contents by
checking and assessing their colleagues’ responses. By reading works of others,
one can deepen one’s own knowledge and develop new ideas by evaluating other
points of view [13, 33].
3. Metacognitive: Learners will develop awareness for their own strengths and weak-
nesses and will be able to compare and evaluate their own performances, at least to
a certain extent [34]. In addition, learners train their abilities to think critically [35,
36] as well as how to evaluate and reflect [37].
4. Affectively: Learners perceive qualitative feedback from their peer group as more
valuable than a lecturer’s grade [33].
Therefore, the application of peer assessment does not only relieve the lecturer but
turns learners into experts themselves. First observations show that evaluations done
by the peer group agree with the lecturers’ evaluations of the learners’ academic per-
formances [38]. Furthermore, studies show that regular feedback given by the peer
group has a positive effect on the learner’s learning process [39]. In their literature
overview, van Zundert et al. [40] point out that there are only a few existing case
studies concerning an experimental setting of peer assessment and that this circum-
stance prevents specific insights on how peer assessment has to be designed. Scien-
tific literature brings up terms such as peer assessment, peer grading, peer review, and
peer feedback, among others. For this paper, we use the term of peer assessment
38
Creating awareness and reflection in a large-scale IS lecture - ARTEL14
meaning that learners of a peer group assess each other’s performances as well as
evaluate it according to relevant criteria without giving each other credit points.
3 Theory-motivated Design of a Peer Assessment
For the improvement of our learning scenario, we draw on a theory-motivated design
approach [41, 42] for engineering learning services [43]. Therefore, we base our sub-
sequent design decisions on the constructs linked to our phenomena of interest.
In particular, we focus on awareness and reflection as ancillary phenomena as well
as on learning outcomes as the main phenomena. Awareness and reflection are closely
associated with interaction in a learning scenario. Hence, we implemented a peer
assessment in our lecture which supports interaction in the learning scenario and in
consequence, awareness, reflection, and ultimately learning outcomes. Figure 1 de-
picts our theory-motivated design approach.
+ Reflection
+ Learning
Interaction
+
Outcome
+ Awareness
Interaction
supporting
component
Fig. 1. Theory-motivated Design Approach
3.1 Concept of a Large-Scale Flipped Classroom
The concept of the presented peer assessment is part of a didactical concept for the
flipped classroom, also known as inverted classroom [44] or inverted lecture [45].
This concept is implemented for the first time within an IS lecture at a German uni-
versity. By choosing a learner-centered approach, the objectives are to increase the
lecture’s quality as well as to convey learner success and satisfaction. The following
figure illustrates the flipped classroom concept. We therefore applied the learner-
centered concept, which addresses three types of interaction throughout all phases.
Referring to the work of Moore [46], the figure below differentiates between learner-
content-interaction, learner-lecturer-interaction, and learner-learner-interaction.
39
Creating awareness and reflection in a large-scale IS lecture - ARTEL14
Self-Consistent Preparation Collaborative Preparation
LEARNER-CONTENT LEARNER-LECTURER LEARNER-LEARNER
Interaction Interaction Interaction
Collaborative Clarification
Collaborative Application Scenario for the application of a peer
assessment
Fig. 2. Concept of a Flipped Classroom Learning Cycle Integrating Three Types of Interaction
The shown learning cycle has duration of two weeks; it will be repeated 5 times dur-
ing one semester. Each cycle compromises four individual phases which are differen-
tiated hereinafter: (1) the first phase can be substituted as self- or private study. The
learners study small either video- or script-based learning units provided by the lec-
turer in a Learning Management System (LMS). (2) During the next phase, every
learner prepares a solution for a part of an extensive open-ended free text assignment
within an allocated group. Every group needs to bring their solutions on power point
slides; these are used as input for the third phase, namely “collaborative clarification”.
(3) This phase is held in presence. The intention of this phase is to discuss the previ-
ously submitted solutions, to consider further aspects of the findings and to emphasize
its strengths. It constitutes the operational scenario for the application of a peer as-
sessment, which is presented after a short explanation of the fourth phase. (4) The
learning cycle ends with the phase of “collaborative application” which is dedicated
to the tutorials. During the tutorials, all learners elaborate a common solution. In spe-
cific, they work on assignments concerning business process management and con-
ceptual data modeling.
3.2 The Application of a Peer Assessment in a Flipped Classroom Scenario
The peer assessment imbeds itself in the third phase of the flipped classroom learning
cycle. Its main goals are the collaborative clarification and consolidation of the pre-
pared solutions submitted to the lecturer as well as gaining a deeper understanding of
the learning content. Usually, an interactive learner-lecturer discussion is the method
of choice. Similar to a traditional lecture, the third phase addresses the interaction
type of learner-lecturer-interaction. In order to improve interaction, awareness, and
reflection in the presence phases of the lecture, we developed a technology-enhanced
peer assessment process addressing additional learner-learner- and learner-content-
interaction. Instead of the lecturer presenting several group solutions, the learners
40
Creating awareness and reflection in a large-scale IS lecture - ARTEL14
themselves consider strengths and weak points and revise the solutions taking into
account the comments made above. By reading and assessing colleagues’ group solu-
tions the learners create awareness and reflection regarding their own group solutions.
They get aware concerning their own strengths and weak points and they receive new
ideas concerning the learning content. Hence, the peer assessment took place in a
synchronous and written form via an online chat using a web based application. After
finishing the peer assessment, the lecturer adopts the role of a moderator and supports
the learners in the organization of a feedback loop. Figure 3 illustrates the structure of
the presented peer assessment.
LEARNER-CONTENT LEARNER-LEARNER LEARNER-LECTURER
Interaction Interaction Interaction
Task a) Etherpad 1
Task b) Etherpad 2
Peer Collaborative
Assessment Clarification
Etherpad 3
Task c)
Etherpad 4
Task d)
Fig. 3. The Process of a Peer Assessment
Aware of the restrictions of our lecture room, we applied a web based application,
to enable the learner participating to the process of a peer assessment. We therefore
used an etherpad as a collaborative online notepad. Etherpad documents are accessi-
ble via web browser and support multiuser usage without having to create multiple
user accounts [47]. Being real time, capable etherpads enable people to collaborate on
ideas, concepts and brainstorming. The selected etherpad has a chat bar on the right
sidebar as well as a basic formatting functionality, and allows anonymous or public
access. To help the learners during the peer assessment process, all learners are pro-
vided with an etherpad compromising the intended assessment structure. Specifying
the date and the learning unit, the created text gets manually stored in text files and
uploaded to the LMS. Figure 4 shows one out of four generated etherpad documents
during our lecture.
41
Creating awareness and reflection in a large-scale IS lecture - ARTEL14
Fig. 4. Technology-enhanced Peer Assessment
4 Evaluation
In order to evaluate our technology-enhanced peer assessment, we surveyed the par-
ticipants of our lecture. We therefore provided a paper-based pre- and post-test during
the lecture. As stated before, we embedded our peer assessment in a flipped classroom
IS lecture. Before the presence phase in the lecture hall and the peer assessment, four
groups of students worked collaboratively in an online forum on four different as-
signments. Each group prepared group individual presentations to a different assign-
ment and posted the assignments to the online forum. Before conducting the lecture,
we administered the prepared group assignments to the other groups. In the actual
lecture, the other groups assessed the elaborated assignments of the other groups.
Afterwards, the lecturer moderated a discussion of the collaborative peer assessment.
To evaluate our procedure, a pre-test was administered before conducting the peer
assessment. Afterwards, the peer assessment was conducted as described above and at
the end of the lecture, the post-test was administered. In the survey, 35 learners who
participated in the peer assessment process answered voluntarily both parts of the
survey, which contained questions regarding the experience with the peer assessment.
All items of the survey were adopted from literature and adapted, if necessary, to our
research context. The items for measuring the perception of the peer assessment were
adopted from Pearce et al. [48] and perceived learning outcomes were adapted from
Eom et al. [49].
The pre-test asked questions about the learners’ experiences and expectations re-
garding the peer assessment. The results show that only 26 percent of our sample had
previously participated in some sort of peer assessment, i.e. paper-based or technolo-
gy-enhanced peer assessment. Considering the phases of the peer assessment, the
42
Creating awareness and reflection in a large-scale IS lecture - ARTEL14
majority (60 percent) of the learners expected that both writing and receiving feed-
back would contribute the most to their learning outcomes. 25.7 percent expected that
receiving feedback and 5.7 percent expected writing feedback would contribute the
most to their learning outcomes (including 8.6 percent nonresponse). Figure 5 pro-
vides details about the further results of the pre-test. Both items were measured on a
five point Likert response format (1 = strongly agree; 5 = strongly disagree). Overall,
the results show that the learners expected up front that the peer assessment would be
useful as a scaffold in the learning process (PA1). Also, the learners expected that
their peers were qualified enough to provide valuable feedback (PA2).
Strongly agree Strongly Disagree
1 2 3 4 5
PA1: As a learning tool, I expect the peer
assessment will be useful.
Mean 2.38; SD= 0.85
PA2: I think that my peers are well qualified
to provide me with critical feedback on my
work:
Mean 2.51; SD= 0.92
N= 35
Fig. 5. Results of the Pre-Test
After conducting the peer assessment, we administered the post-test. First, we meas-
ured which part of the peer assessment influenced learning the most. The results show
that the learners’ expectations were confirmed, since 45.9 percent reported that learn-
ing results most from both writing and receiving feedback. Figure 6 shows further
results of the post-test. All items were rated on a five point Likert response format (1
= strongly agree; 5 = strongly disagree) and provide the mean value and standard
deviation (SD) of the responses. In addition, a one sample t-test was conducted in
order to evaluate whether the mean values for all of the questions are lower than the
neutral value. In consequence, the usefulness of the peer assessment is shown imply-
ing the rejection of the null hypothesis 𝐻0 : 𝜇 ≥ 3. The results show that several items
were rated under the neutral value of 3, indicating, in general, a good fit of the peer
assessment. Additionally, the t-test provides evidence that H0 is not supported by
several items (at least p<0.05), and can thus be rejected in these cases. The indicator
PA3 was rated as good, on average (2.74) showing the overall usefulness of our tech-
nology-enhanced peer assessment. PA4 however did not provide significant results.
Therefore, we are not able to provide evidence that the peers involved in the peer
assessment were actually suitable to conduct the assessment. In contrast, with a highly
significant result, PA5 shows that our participants were able to improve their solu-
tions after the assessment. Considering the learning outcomes in terms of learning
success, LO1 was found to be significant and LO2 was found to be insignificant.
These results are not really contradicting, since they demonstrate that our participants
felt that the peer assessment itself did not actually affect the learning outcomes and
43
Creating awareness and reflection in a large-scale IS lecture - ARTEL14
that they have learned as much without participating in the assessment process. Fur-
ther analysis of item LO3 showed that on average (2.71) the learners noticed an im-
provement of the learning experience and quality.
Strongly agree Strongly Disagree
1 2 3 4 5
PA3: As a learning tool, peer assessment was
very useful.
Mean 2.74; SD= 0.74; t= 2.05*
PA4: I thought that my peers did a good job
in providing me with critical feedback on my
work.
Mean 2.80; SD= 0.86; t= 1.36
PA5: I think that I improved my written work
as a result of the assessment that I received or
wrote.
Mean 2.40; SD= 0.65; t= 5.45***
LO1: I feel that I learned as much with the
peer assessment as I might have without.
Mean 2.57; SD= 1.17; t= 2.16*
LO2: I feel that I learned more with the peer
assessment than without it.
Mean 3.11; SD= 1.02; t= 0.61
LO3: The quality of the learning experience
with the peer assessment is better than with-
out it.
Mean 2.71; SD= 0.75; t= 2.25*
N= 35
Fig. 6. Results of the Post-Test
5 Discussion
The present study was designed to determine whether an application of a technology-
enhanced peer assessment is a suitable instrument to engage awareness and reflection
of the learners by improving interaction in our lecture. Our results indicate that the
expectations of the learners concerning the usefulness of the peer assessment were
confirmed. This is in line with previous research results, which also indicated the
usefulness of peer assessments in higher education scenarios [50]. However, we con-
firmed the suitability of the peer assessment for a new learning scenario driven by a
rich interaction in large-scale lectures. Surprisingly, our results show that the percep-
tion of the learners did not reveal any significance for the suitability of their peers for
the assessment process. The present findings seem to be consistent with other research
44
Creating awareness and reflection in a large-scale IS lecture - ARTEL14
which found that trust in the peer as an assessor is not a significant predictor for learn-
ing outcomes [51].
This is also shown by the qualitative insights we gained during our evaluation. One
learner states in the evaluation: “The idea is good, but the other students need to col-
laborate better.” This opinion also relates to the results of our post-test, which re-
vealed that most of the learners were not satisfied with their peers. Therefore, we
acknowledge these issues by suggesting the implementation of further components
that actually enhance a rich learner-learner-interaction to provide a useful process of
awareness and reflection in the learning process. This is also highlighted by the fol-
lowing statement of another student: “The procedure is very good and modern, but it
depends very much on the fellow students”. Therefore, we also want to highlight the
importance of the faithful appropriation of such a learning method. If the learning
methods and structures, such as our technology-enhanced peer assessment, are ironi-
cally appropriated, learning outcomes may suffer [52–54]. For instance, we noticed
that students ironically appropriated the chat function and did not use it for a purpose-
ful discussion. As an implication, we would suggest to guide the learning process and
provide best-practices how to use the tool faithfully. However, our results also show
that the received peer assessment improved the assignments of the learners. We there-
fore highlight the importance of the feedback provided by the peers in order to im-
prove learning outcomes.
Considering the learning outcomes, we found evidence that the peer assessment
has no significant impact on the subjective learning outcomes in our study. However,
subjective perceptions have to be judged carefully, especially in the context of learn-
ing success [55, 56]. Therefore, we cannot make any definite prediction on how peer
assessments actually improve learning outcomes in a flipped classroom scenario.
Interestingly, further analysis of the learning outcomes showed that the learners no-
ticed an improvement of the learning experience and quality. This also relates to our
results that the peer assessment as a meta-cognitive scaffold is a useful method to
improve the learning process and in consequence increase learning outcomes [54, 57].
To sum up, we sought to address interactivity in the learning process as means to
improve awareness, reflection, and learning outcomes in our learning scenario. Con-
sidering our evaluation results, we can state that the peer assessment is a useful meth-
od for structuring presence phases in a flipped classroom scenario. Hence, we high-
light as a practical implication the importance of the learning process and the reflec-
tion of the learning outcomes by interacting with it. This procedure also creates
awareness of the learning progress, enabling learners to actually improve their self-
regulated learning activities which are especially important in flipped classroom sce-
narios.
6 Limitations and Future Research
This study of a peer assessment in a flipped classroom is still on-going and therefore
comes with limitations regarding the evaluation and application. Concerning the eval-
uation, we consider the poor response rate in our evaluation. Typically, our lecture is
45
Creating awareness and reflection in a large-scale IS lecture - ARTEL14
attended by 150-200 students. Since we did not provide any compensation for partici-
pating in our survey, the external value might be affected, because maybe only learn-
ers well-disposed to such innovative learning concepts participated. In consequence,
we consider this limitation and plan to evaluate the peer assessment in an additional
and compulsory longitudinal online survey to account for the evolving nature of inter-
action with e-learning components in the learning process. In addition, we plan to
evaluate the actual effects of the peer assessment by conducting an experiment with a
peer assessment treatment group and a control group.
Furthermore future work should investigate peer assessment as instrument for indi-
vidual learning success verification during the learning process. Following Bloom’s
[13] suggestion, transfer and verification of learning content should be adjusting to
various cognitive levels of educational objectives. In large-scale lectures the verifica-
tion of high cognitive levels of educational objectives is very time- and resource-
consuming and hence impractical in use. Peer assessment should be investigated as
time- and resource-saving manner to measure learning success during the learning
process.
The other part of our limitations deals with the on-going application of our peer as-
sessment. We applied the peer assessment in our lecture for the first time. Hence, we
are still adjusting and modifying the process for the deployment of the peer assess-
ment. In consequence, our evaluation could be biased by effects that are induced
through the first time application, e.g., glitches that are mainly concerned with usabil-
ity issues. However, we seek to overcome these limitations with a broad application
during the next terms and further insights by this application. This would also include
the application within other learning scenarios, especially those that are influenced by
cultural differences [58, 59].
7 Conclusion
This paper has examined the application of a peer assessment in a large-scale IS lec-
ture arranged in a flipped classroom setting. We therefore provided first evidence of
the utility of peer assessments as suitable instruments to increase awareness and re-
flection as well as to strengthen learning outcomes in an IS lecture. The results of this
investigation show that the peer assessment itself does not affect the learning out-
comes, but it does have a positive impact on learning experience and quality. Alt-
hough the current study is based on a small sample of participants, the findings sug-
gest that the application of a peer assessment might be a useful instrument to effect
awareness and reflection. Considerably more work will need to be done to determine
the effects of a technology-enhanced peer assessment on awareness and reflection as
well as on learning outcomes.
46
Creating awareness and reflection in a large-scale IS lecture - ARTEL14
Acknowledgements
The research presented in this paper was funded by the German Federal Ministry of
Education and Research in the project kuLtig (www.projekt-kultig.de), FKZ
01BEX05A13.
References
1. Leidenfrost, B., Strassnig, B., Schabmann, A., Carbon, C.C.: Verbesserung der Studiensi-
tuation für StudienanfängerInnen durch Cascaded Blended Mentoring. Psychologische
Rundschau 60, 99–106 (2009)
2. Grießhaber, W.: Neue Medien in der Lehre, http://spzwww.uni-
muenster.de/griesha/pub/tnmlehre.pdf
3. Lehmann, K., Söllner, M.: Theory-driven design of a mobile-learning application to sup-
port different interaction types in large-scale lectures. ECIS 2014 Proceedings (2014)
4. Lehmann, K., Thillainathan, N., Bitzer, P., Leimeister, J.M.: Performance Dashboard für
Dozenten in der universitären Lehre. In: Multikonferenz der Wirtschaftsinformatik
(MKWI). Braunschweig, Deutschland (2012)
5. Picciano, A.G.: Beyond student perceptions. Issues of interaction, presence, and perfor-
mance in an online course. Journal of Asynchronous learning networks 6, 21–40 (2002)
6. van den Boom, Gerard, Paas, F., van Merriënboer, Jeroen J.G.: Effects of elicited reflec-
tions combined with tutor or peer feedback on self-regulated learning and learning out-
comes. Learning and Instruction 17, 532–548 (2007)
7. Boud, D., Walker, D.: Promoting reflection in professional courses: The challenge of con-
text. Studies in Higher Education 23, 191–206 (1998)
8. Moore, A., Masterson, J.T., Christophel, D.M., Shea, K.A.: College Teacher Immediacy
and Student Ratings of Instruction. Communication Education 45, 29–39 (1996)
9. Hardless, C., Nilsson, M., Nuldén, U.: ‘Copernicus’ experiencing a failing project for
reflection and learning. Management Learning 36, 181–217 (2005)
10. Alonso, F., Manrique, D., Viñes, J.M.: A moderate constructivist e-learning instructional
model evaluated on computer specialists. Computers & Education 53, 57–65 (2009)
11. Mayer, H.O., Hertnagel, J., Weber, H.: Lernzielüberprüfung im eLearning. Oldenbourg, R
(2009)
12. Atkins, S., Murphy, K.: Reflection: a review of the literature. J Adv Nurs 18, 1188–1192
(1993)
13. Bloom, B.S.: Taxonomy of educational objectives. David McKay New York (1956)
14. Anderson, L.W., Krathwohl, D.R., Airasian, P.W., Cruikshank, K.A., Mayer, R.E., Pin-
trich, P.R., Raths, J., Wittrock, M.C.: A taxonomy for learning, teaching, and assessing: A
revision of Bloom's taxonomy of educational objectives, abridged edition. White Plains,
NY: Longman (2001)
15. Strijbos, J.-W., Ochoa, T.A., Sluijsmans, D.M., Segers, M.S., Tillema, H.H.: Fostering
Interactivity through Formative Peer Assessment in (Web-Based) Collaborative Learning
Environments. In: Mourlas, C., Tsianos, N., Germanakos, P. (eds.) Cognitive and Emo-
tional Processes in Web-Based Education, pp. 375–395. IGI Global (2009)
47
Creating awareness and reflection in a large-scale IS lecture - ARTEL14
16. Gregor, S.: The nature of theory in information systems. MIS Quarterly 30, 611–642
(2006)
17. Nussbaumer, A., Scheffel, M., Niemann, K., Kravcik, M., Albert, D.: Detecting and Re-
flecting Learning Activities in Personal Learning Environments. Proc. of the 2nd Work-
shop on Awareness and Reflection in Technology-Enhanced Learning (#artel12) held in
conjunction with EC-TEL 2012, Saarbrucken (Germany) (2012)
18. Bodemer, D., Dehler, J.: Group awareness in CSCL environments. Computers in Human
Behavior 27, 1043–1045 (2011)
19. Dabbagh, N., Kitsantas, A.: Supporting Self-Regulation in Student-Centered Web-Based
Learning Environments. International Journal on E-Learning 3, 40–47 (2004)
20. Phielix, C., Prins, F.J., Kirschner, P.A.: Awareness of group performance in a CSCL-
environment: Effects of peer feedback and reflection. Computers in Human Behavior 26,
151–161 (2010)
21. Cressey, P., Boud, D., Docherty, P.: Book Review: Productive Reflection at Work. Man-
agement Learning 37, 410–413 (2006)
22. Høyrup, S., Elkjær, B.: Reflection - beyond the individual. In: Boud, D., Cressey, P., Do-
cherty, P. (eds.) Productive reflection at work: Learning for changing organizations, pp.
29–42. Routledge (2006)
23. Boud, D., Keogh, R., Walker, D.: Reflection, turning experience into learning. Kogan
Page; Nichols Pub., London, New York (1985)
24. Nonaka, I.: A Dynamic Theory of Organizational Knowledge Creation. Organization Sci-
ence 5, 14–37 (1994)
25. Fischer, G.: Understanding, fostering, and supporting cultures of participation. interactions
18, 42 (2011)
26. Liu, T.-C., Liang, J.-K., Wang, H.-Y., Chan, T.-W., Wei, L.-H.: Embedding educlick in
classroom to enhance interaction. In:, pp. 117–125 (2003)
27. Wang, M.C., Haertel, G.D., Walberg, H.J.: What influences learning? A content analysis
of review literature. The Journal of Educational Research 84, 30–43 (1990)
28. Gagné, E.D., Yekovich, C.W., Yekovich, F.R.: The cognitive psychology of school learn-
ing. HarperCollins College, New York (1993)
29. Morgan, N., Saxton, J.: Teaching, questioning, and learning. Routledge, New York (1991)
30. Bischof, L.: Die digitale (R) evolution? Chancen und Risiken der Digitalisierung akademi-
scher Lehre (2013)
31. Piech, C., Huang, J., Chen, Z., Do, C., Ng, A., Koller, D.: Tuned models of peer assess-
ment in MOOCs. arXiv preprint arXiv:1307.2579 (2013)
32. Boud, D., Falchikov, N.: Rethinking assessment in higher education. Learning for the
longer term. Routledge (2007)
33. Sadler, P.M., Good, E.: The impact of self-and peer-grading on student learning. Educa-
tional assessment 11, 1–31 (2006)
34. Darling-Hammond, L., Ancess, J., Falk, B.: Authentic assessment in action. Studies of
schools and students at work. Teachers College Press (1995)
35. Block, J.H., Airasian, P.W., Bloom, B.S., Carroll, J.B.: Mastery learning. Theory and
practice. Holt, Rinehart and Winston New York (1971)
36. Zoller, U.: Are lecture and learning compatible? Maybe for LOCS: Unlikely for HOCS.
Journal of Chemical Education 70, 195 (1993)
48
Creating awareness and reflection in a large-scale IS lecture - ARTEL14
37. Sitthiworachart, J., Joy, M.: Web-based peer assessment in learning computer program-
ming. In:, p. 180 (2003)
38. Kulkarni, C., Wei, K.P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., Klem-
mer, S.R.: Peer and self assessment in massive online classes. ACM Trans. Comput.-Hum.
Interact. 20, 1–31 (2013)
39. Dochy, F., Segers, M., Sluijsmans, D.: The use of self-, peer and co-assessment in higher
education. A review. Studies in Higher Education 24, 331–350 (1999)
40. van Zundert, M., Sluijsmans, D., van Merriënboer, J.: Effective peer assessment processes:
Research findings and future directions. Learning and Instruction 20, 270–279 (2010)
41. Briggs, R.O.: On theory-driven design and deployment of collaboration systems. Interna-
tional Journal of Human-Computer Studies 64, 573–582 (2006)
42. Gehlert, A., Schermann, M., Pohl, K., Krcmar, H.: Towards a Research Method for Theo-
rydriven Design Research. Wirtschaftinformatik Proceedings 2009 (2009)
43. Leimeister, J.M.: Dienstleistungsengineering und -management. Springer Berlin Heidel-
berg, Berlin, Heidelberg (2012)
44. Strayer, J.F.: How learning in an inverted classroom influences cooperation, innovation
and task orientation. Learning Environ Res 15, 171–193 (2012)
45. Gehringer, E.F., Peddycord, B.W.: The inverted-lecture model. In: Camp, T., Tymann, P.,
Dougherty, J.D., Nagel, K. (eds.) Proceeding of the 44th ACM technical symposium, p.
489 (2013)
46. Moore, M.G.: Editorial: Three types of interaction. American Journal of Distance Educa-
tion 3, 1–7 (1989)
47. Karlhuber, S., Wageneder, G., Freisleben-Teutscher, C.F.: Einsatz kollaborativer Werk-
zeuge-Lernen und Lehren mit webbasierten Anwendungen. Lehrbuch für Lernen und Leh-
ren mit Technologien (2013)
48. Pearce, J., Mulder, R., Baik, C.: Involving students in peer review. Case studies and practi-
cal strategies for university teaching. (2009)
49. Eom, S.B., Wen, H.J., Ashill, N.: The Determinants of Students' Perceived Learning Out-
comes and Satisfaction in University Online Education: An Empirical Investigation*. De-
cision Sci J Innovative Educ 4, 215–235 (2006)
50. Falchikov, N., Goldfinch, J.: Student Peer Assessment in Higher Education: A Meta-
Analysis Comparing Peer and Teacher Marks. Review of Educational Research 70, 287–
322 (2000)
51. van Gennip, Nanine A.E., Segers, M.S., Tillema, H.H.: Peer assessment as a collaborative
learning activity: The role of interpersonal variables and conceptions. Learning and In-
struction 20, 280–290 (2010)
52. Gupta, S., Bostrom, R.: Technology-Mediated Learning: A Comprehensive Theoretical
Model. Journal of the Association for Information Systems 10, 686–714 (2009)
53. Gupta, S., Bostrom, R.: An Investigation of the Appropriation of Technology-Mediated
Training Methods Incorporating Enactive and Collaborative Learning. Information Sys-
tems Research 24, 454–469 (2013)
54. Gupta, S., Bostrom, R.P., Huber, M.: End-user Training Methods: What We Know, Need
to Know. SIGMIS Database 41, 9–39 (2010)
49
Creating awareness and reflection in a large-scale IS lecture - ARTEL14
55. Sitzmann, T., Ely, K., Brown, K.G., Bauer, K.N.: Self-Assessment of Knowledge: A Cog-
nitive Learning or Affective Measure? Academy of Management Learning & Education 9,
169–191 (2010)
56. Benbunan-Fich, R.: Is Self-Reported Learning a Proxy Metric for Learning? Perspectives
From the Information Systems Literature. Academy of Management Learning & Education
9, 321–328 (2010)
57. Bitzer, P., Janson, A.: Towards a Holistic Understanding of Technology-Mediated Learn-
ing Services - a State-of-the-Art Analysis. ECIS 2014 Proceedings (2014)
58. Janson, A., Peters, C., Leimeister, J.M.: Der Weg zur effizienten Bereitstellung kultursen-
sitiver Dienstleistungen - erste Schritte mittels systematischer Modularisierung. In:
Thomas, O., Nüttgens, M. (eds.) Dienstleistungsmodellierung 2014, pp. 266–286 (2014)
59. Hammer, N., Janson, A., Leimeister, J.: Does culture matter? A qualitative and compara-
tive study on eLearning in Germany and China. BLED 2014 Proceedings (2014)
50
Application overlapping user profiles to foster
reflective learning at work
Angela Fessl, Gudrun Wesiak, and Granit Luzhnica
Know-Center
Inffeldgasse 13
A - 8010 Graz
(afessl, gwesiak, gluzhnica)@know-center.at
Abstract. Reflective learning is an important activity of knowledge-
workers in order to improve future working-behaviours. The insights
gained by reflective learning are based on re-experiencing and re-evaluating
past working situations. One time- and cost-effective way to support re-
flective learning is the employment of applications that collect data about
working processes, store the data in user profiles, and visualise it in order
to provide timely feedback to the employees. However, a single applica-
tion can only capture part of the data that might be relevant for reflec-
tion and the parallel use of several applications leads to high demands
on the user regarding the interpretation of relationships between sev-
eral single visualizations. A combined visualisation of data captured by
different apps should enhance the support for reflection about the work-
ing behaviour and experiences. This paper introduces an overlapping
user profile application, which combines and aggregates data captured
by various applications. The goal of this overlapping application is to
provide higher-level reflection possibilities by combining visualisations of
different application data in order to better induce and support reflective
learning at work. A first proof-of-concept of such an approach indicates
that a combined user profile application and especially it’s visualisations
can be beneficial with regard to reflective learning and can enhance the
awareness about the multiple aspects of a user’s work life.
Keywords: Work-place learning, reflective learning, awareness, user pro-
files, reflective data analytics
1 Introduction
Today’s work environments are constantly becoming more complex, globally
integrated, and knowledge-centric. This simultaneously leads to a stronger need
of employees who are motivated and capable to reflect upon their activities and
as a consequence adjust their working practices to new demands. Especially for
knowledge-workers, reflective learning is an important activity to re-experience
past situations during work and to learn from them in order to improve their
future working-behaviour [2]. One possibility to motivate knowledge workers
to become reflective practitioners is to support them with corresponding tools
51
Application overlapping user profiles to foster reflective learning at work - ARTEL14
or applications, which could be easily integrated into their daily work-life [14].
These applications have the task to gather data from work processes and to
provide guidance for reflection in form of raising awareness and offering triggers
with regard to unusual or extraordinary work related experiences or situations.
In contrast to formal learning settings, reflective learning at the workplace deals
with informal and self-regulated learning, where challenges like no additional
working effort, easy integration in daily working routines as well as a clear benefit
for knowledge workers have to be considered right from the beginning.
In order to support reflective learning at work, within the EU-funded project
MIRROR (http://www.mirror-project.eu) several applications have been devel-
oped, which aim at motivating and activating users to reflect upon their indi-
vidual working experiences. After the reflection process itself, knowledge work-
ers should have gained some benefits or insights for themselves and as con-
sequence derive and apply behavioural changes for future working situations.
These changes should permanently improve and facilitate the handling of up-
coming similar situations or experiences.
The applications developed within the MIRROR project have been applied
within a wide range of working environments (e.g. care homes, hospitals, IT com-
panies, and emergency situations) and support various sets of professionals (e.g.
knowledge-workers, nurses, physicians and carers as well as emergency workers).
Each of the developed applications collects and gathers different kinds of data
and stores them in their corresponding user profiles. This data encompasses on
the one hand information about the user. On the other hand it consists of in-
formation on users’ work processes, which is captured automatically or inserted
manually during the user’s work. Examples are application switches, application
usage and documents used while working on a PC as well as manually inserted
data such as the current mood status of the user, individual notes, feedback
on different working situations, ratings, scores of serious games or quiz results.
The collected data is stored within the applications themselves and for some
applications additionally in the so-called MIRROR Spaces Framework [20], an
underlying data storage system for exchanging data between applications. In the
spaces framework, the user’s data is stored in the user profile and is accessible
only by its owner. Each of these single applications visualises the data for the
user in a sophisticated way with the goal to trigger reflective learning. However,
user studies conducted in different environmental settings (e.g. [9], [16]) showed
that single applications can only capture part of the data that might be relevant
for reflective learning. Participants of these studies asked on the one hand for a
better guidance to interpret the data in order to initiate reflection. On the other
hand they wanted to see a clearer benefit for themselves, which would serve as
motivational trigger to use the application and to reflect about the captured
data. Thus, similar to research outcomes from the field of learning analytics, we
found that a combination of data is often more adequate for successfully support-
ing users. Whereas learning analytics addresses self-reflective learning mostly as
important aspect of self-regulated learning in formal learning environments, we
focus on work-related reflective learning in informal learning environments.
52
Application overlapping user profiles to foster reflective learning at work - ARTEL14
With this paper we want to present a first approach on how to meaningfully
combine and visualize data captured by different applications. The goal is to
provide a greater variety of reflective learning opportunities in order to facilitate
deeper insights on one’s working experiences. We are aware that this approach
raises privacy and security issues which need to be carefully considered when
employing the app in a real working environment. However, for this first approach
privacy and security were only of secondary interest, but will of course be treated
in upcoming research settings.
Therefore, we developed the so-called ”MIRROR Integrated User Profile”
application (MUP App) which has the task to integrate, summarise, analyse,
and visualise data captured by several different applications in order to induce
and support reflective learning at work. For a first proof-of-concept, we used
two different applications in parallel, namely KnowSelf and the MoodMap App.
KnowSelf automatically captures work activities on a PC, whereas the MoodMap
App allows knowledge workers to easily state their moods during a working day.
We collected, aggregated, and visualised data from a small sample of knowledge
workers to to get a first impression of users’ interest and motivation and the
app’s usefulness. From this we derived the following three research questions:
– RQ1: Are participants interested and willing to use more than one applica-
tion in parallel with regard to reflective learning?
– RQ2: Does the MUP App as overlapping application facilitate reflection
about users’ working experiences and contribute to raising awareness of mul-
tiple aspects of their work life?
– RQ3: Do participants perceive any individual insights or benefits for them-
selves?
2 Related Work
2.1 User Profiles
Since the terms user profile and user model are not always used in exactly the
same way, it is essential to clarify our understanding and usage of the term user
profile, which we base on existing theories regarding user models and on our un-
derstanding in MIRROR described by [15]. User models in general are models
that computer systems have about their users. The data in such user models is
automatically captured by the system and is mainly used in information retrieval
and intelligent tutoring systems or user-adaptive learning systems (see e.g., [10,
1]). User models, which are utilised in learning environment systems for mod-
eling the learner and the corresponding learning activities, are called learner
models. These types of user models are created by the systems automatically
and are not directly accessible by the users via user interfaces. Furthermore
they are used to adapt teaching strategies or to inform the learner about the
learning progress as basis for reflective learning. Additionally [3] suggested that
learner models should keep data like knowledge, interests, goals, background,
and individual traits, thus abstract concepts relevant for learning. In order to
53
Application overlapping user profiles to foster reflective learning at work - ARTEL14
apply a user model or learner model as basis for reflection on one’s own learning
activities, achievements, or progress towards the individual learning goals, it is
necessary to make the models accessible and manageable for the user, which was
explicitly suggested by [12] and mentioned in [4, 5, 13].
In MIRROR we prefer the term user profile (UP). Although the MIRROR
user profile (MUP) is based on theory and research of user models, the term user
profile better reflects its mission in MIRROR. First, the purpose of the MUP is
to guide and support reflection by mirroring user data in the form of activities,
experiences or artefacts of work, notes and insights, moods, work practices, and
other concrete data sources back to the user. Secondly, we intended these user
profiles to be created and maintained by a mixture of automated methods and
manual management, where the process of editing or updating the data may also
explicitly trigger reflection.
2.2 Learning Analytics
Although learning analytics is not in focus of our work, several approaches,
methodologies and technologies of this research area are closely linked to reflec-
tive learning. Learning analytics deals with methods for analysing and detecting
patterns within data collected from educational settings or learning environ-
ments about the learner, and leverage those methods to support adaptation,
personalisation, recommendation, and also reflection. Siemens [21] defined learn-
ing analytics as ‘the use of intelligent data, learner-produced data, and analysis
models to discover information and social connections, and to predict and advise
on learning’. The focus of learning analytics is on the support of the learner in
formal learning setting, while in our work the focus is to support the knowl-
edge worker in an informal learning setting. Nevertheless, the parallel to our
work is evident. Also approaches like learning dashboards for example described
in [8, 19] present an overview of the learner’s own learning activities and learning
progress, and in relation to colleagues at one glance. Such combined visualisa-
tions support self-monitoring for learners and awareness for teachers as well as
empowers the learners to reflect on their own activity, and that of their peers.
Explicit traces (e.g. the learner’s entries in a chat or a discussion forum) and im-
plicit traces (e.g. the learner entering a course or clicking on a document) stored
in the corresponding learner profiles serve here as basis for the aggregation and
visualisation of the gathered data.
The main focus of learning analytics is to support the learner while learning
in an educational setting or learning environment. Although learning analytics
includes also reflective learning approaches (e.g. [18, 17]), our work can be clearly
distinguished from these approaches by focusing on knowledge workers in real
working environments and and to support reflection on working experiences or
working artifacts in order to learn from them to improve future work.
54
Application overlapping user profiles to foster reflective learning at work - ARTEL14
2.3 Reflective Learning
Individual reflection takes normally place in every day’s life and obviously also
during work or work- related situations. Reflection may be triggered by different
reasons for example by conflicts or problems, by unexpected experiences or by
a person acting in a complete different way in comparison with the individual
(external trigger). But also if an individual feels uncomfortable, for something
bothers her or an inner voice is nagging, without being able to make this feeling
external (internal trigger). As reaction, a reflection process may be triggered with
or without the awareness of the person. This reflection should lead to an individ-
ual insight or outcome which may be used to guide or adapt future behaviour.
Within MIRROR we follow the definition of [2], who define reflective learning as
‘those intellectual and affective activities in which individuals engage to explore
their experiences in order to lead to new understandings and appreciations’.
Bringing this together, reflection is both a crucial part of learning and a re-
sponse to past work experiences. These experiences as well as the behaviours of
the individual engaged serve as starting point for the reflective process. The de-
sired outcomes of reflection may lead to personal synthesis, integration of knowl-
edge (internalisation), validation of personal knowledge, a new affective state or
the decision to take on actions for future events. To achieve these results the
characteristics of the individual (learner) have to be taken into account as well
as the intention of the individual self. Individual reflection may occur sponta-
neous and unconsciously and in any possible situation especially then when it is
not expected [6]. Of course it can also be consciously triggered by peers super-
visors or by meeting created specifically for that purpose [7]. Within MIRROR
we focus to initiate reflective practices with the support of technologies, which
might automatically detect unusual working patterns and working behaviours
and by making the worker aware of them in form of reflection triggers or explicit
reflection guidance e.g. by means of prompts.
3 Examples of MIRROR Applications
In the scope of the MIRROR project a series of applications supporting indi-
vidual reflection have been developed and evaluated in different settings [16, 9].
Some of the user studies showed that gathering and visualising data captured
by single applications is not always enough to initiate reflective learning. To il-
lustrate how the MIRROR applications support reflective learning, we want to
shortly introduce two applications, namely KnowSelf and the MoodMap App.
The same two applications will later be used as example for a possible combined
usage and integration via the MUP App.
KnowSelf automatically captures work activities (used applications and
resources together with the exact time of use) on a PC, provides simplistic project
and task recording and presents an overview as well as different visualisations
of the captured data [16]. Providing these visualisations regarding time use at
work should lead to reflection on personal time management and potentially
motivate to consider improvements in this respect. The user profile of KnowSelf
55
Application overlapping user profiles to foster reflective learning at work - ARTEL14
Fig. 1. KnowSelf (left) and MoodMap App (right)
is not a conventional user profile, because it consists only of user activities, but
not of information about the users themselves. The application stores all work
activities captured on the user’s computer, including window focus and title, if
applicable the system location (path) of the resource, focus switches, and idle
time. Additionally the user can manually record time spent on projects or tasks
and save observations. The collected information is displayed on a timeline and
as statistics in the form of pie charts.
The MoodMap App is a web-based application, which allows knowledge
workers to track their mood during a working day or virtual meeting and recapit-
ulate their work experiences afterwards. The MoodMap App provides an easy-
to-understand user interface to state individual mood points by simply clicking
on a bi-dimensional coloured map. Each mood is composed of two dimensions,
namely valence (negative to positive feelings) and arousal (low to high energy)
based on the model proposed by [11]. Additionally, it provides several visualisa-
tions on an individual as well as collaborative level, to make users reflect on the
mood development over time or to provide easy comparison possibilities of one’s
own mood with the mood of others for example colleagues or team members
of the same team. The application related user profile stores information about
the user, sharing settings for security and privacy issues as well as individual
email settings. Furthermore, individual moods and inherent notes, correspond-
ing meetings, context information of a day or meeting, as well as personal diary
entries are stored in the internal user profile of the application.
4 The MIRROR Integrated User Profile (MUP)
Insights from evaluations conducted separately for KnowSelf and the MoodMap
App led to the conclusion that single applications capture only part of the data
that might be relevant for reflection [9, 16]. Although the developed applications
proved to have high potential to trigger reflection at work, we wanted to go
one step further. As a first step, we made triggers from different sources easily
56
Application overlapping user profiles to foster reflective learning at work - ARTEL14
accessible to the users, in order to further facilitate the reflective learning experi-
ence and help users to get more insights at one glance. Thereupon we developed
the MIRROR User Profile (MUP) concept, which focuses on the combination
of the captured data and corresponding sophisticated visualisations. An early
prototype of the MIRROR User Profile Application (MUP App) was realised
and tested with a small sample of knowledge workers.
4.1 Prerequisites for the MIRROR User Profile
In order to efficiently implement a common MIRROR user profile, it is of crucial
relevance to use data captured and gathered by several MIRROR applications
and not only by a single one. To achieve this, we employed the MIRROR Spaces
Framework, an underlying data storage system developed within the MIRROR
project to store and exchange data of the applications. For the development of
a common user profile based on the MIRROR spaces the following prerequi-
sites have to be considered: assumptions regarding (i) data, (ii) reusability, (iii)
sharing, (iv) privacy and security, and (v) accessibility by the user interfaces.
Data stored in the MUP can be divided into three different types, namely
personal data about the user, private data, and shared data. Personal data about
the user consists of general information about the user (e.g. name or email ad-
dress) and login information. For the data implicitly captured by the MIRROR
applications (e.g. work history in KnowSelf) as well as data explicitly inserted
by the user (e.g. mood in the MoodMap) it is essential that the user has full
control over her data by deciding for each type of captured data, whether it is
private or can be shared.
Reusability is one of the major potential benefits of the MUP. By storing
the data according to a predefined data format, applications are able to reuse
not only their data but the data captured by other applications and other users
as well. Account information can be stored once in the user profile and then be
used by all MIRROR applications.
Sharing data is of major relevance for reflection in order to provide possibil-
ities for comparing one’s own data with that of colleagues or a whole team. To
account for different levels of sharing, settings (e.g. anonymised, sharing within
the same team or department) should be very fine-grained.
As mentioned above privacy and security are a major concern when storing
data in the MIRROR Spaces Framework. It has to be ensured that the privacy
settings defined by the users via different applications are always met by all
applications, aggregations, and visualisations.
Sharing, privacy and security settings along with other data gathered either
explicitly or implicitly by applications, should be accessible and modifiable
by user-friendly interfaces and visualisations provided by each MIRROR appli-
cation. This has the advantage, that the user has full control about the data and
has the potential to decide on a very fine-grained level, which data she wants to
share with whom and which data should be kept private only.
57
Application overlapping user profiles to foster reflective learning at work - ARTEL14
4.2 The MIRROR Integrated User Profile Application
The MIRROR Integrate User Profile Application (MUP App) serves as a bridge
between the MIRROR Spaces Framework and various MIRROR applications.
It provides services for data administration as well as for directly supporting
reflective learning. The latter is achieved by making users aware of unusual or
significant behavioural patterns. The MUP App’s service can be used by other
applications to show and promote reflective learning by presenting combined
data aggregated by different applications or from different users.
The tasks of MUP App are two-fold, providing (a) access to the data stored in
the corresponding user profiles per user within the MIRROR Spaces Framework
and (b) a data analysis service, which aggregates data from different applica-
tions (on an individual level) and/or from different users (on a collaborative or
organisational level). The aggregated data can be used to raise awareness on
relationships between data captured from different applications, make compar-
isons along a timeline or among different users, and finally detect patterns that
are relevant for individual or collaborative reflection. Reasons for reflection can
encompass the need for problem solving, decision-making, emotion regulation,
or detection of significant deviations between the individual user and a team.
In this first phase of the development we pursue a more general approach
directed towards basic types of data that are comparable across different ap-
plications. We mostly focus on statistical analysis to extract information on for
example the number of different applications used by an individual, on providing
a chronological overview of the applications used, on presenting the number of
entries in various diaries, and on general information (e.g. when, how often or
which data) was captured by each application. The data is presented on different
types of charts, which can be selected by the user in order to ensure that the
chart fits to the available data. In addition, the user may visualise her data in
direct comparison with the data to other users (e.g. her team-members).
For the second phase, we will be concentrating on the different types of
data captured by various applications, in order to provide analysis on the com-
bined data. For instance, combining the usage of the MoodMap App with data
captured by KnowSelf, might show a relationship between moods and specific
working tasks. This would lead to new insights that may be the basis for ini-
tiating reflective learning. As an example, the left of Fig. 2 shows the hourly
application usage history of a single user for both KnowSelf and the MoodMap
App. The picture on the right of Fig. 2 visualises combined application specific
data, namely the number of hourly switches between tasks or resources captured
by KnowSelf and the corresponding mood of a user, depicted as separate lines
for arousal (mood.energy) and valence (mood.feel) by the MoodMap App. For
this visualization, mood values from the MoodMap App depicted in Fig. 1 are
expressed as numbers between 0 and 100.
58
Application overlapping user profiles to foster reflective learning at work - ARTEL14
Fig. 2. MUP application with MoodMap and KnowSelf data for a single user
5 Proof of concept
In order to investigate the potential of a common MIRROR user profile as sup-
port for reflective learning, we conducted a small combined user study employing
the KnowSelf and the MoodMap App in parallel. Although we used only two
application for this first evaluation, the MUP App is able to handle all applica-
tions that store their data in the common user profile. Based on what we have
learned from the separate evaluations, we see this study as first proof-of-concept
for the MUP App. The goal was to find out whether a combined analysis of data
from both user profiles will i) be accepted by the users, ii) enhance the boost of
reflective learning, and iii) provide clearer insights or benefits for the single user.
5.1 Setting
The participating team consisted of 6 knowledge workers (3 women, 3 men), on
average aged between 30 and 40, all of them mostly doing computer work. They
used the KnowSelf and the MoodMap App in parallel for two weeks during work.
Each day, either in the morning or in the evening they were asked to re-evaluate
and reflect about their captured data and write down their insights and thoughts
directly within one of the two applications. User activities automatically logged
by KnowSelf could only be analysed for 5 persons due to technical reasons on
one of the PC’s. At the end of the trial the participants were asked to fill in a
questionnaire and to take part in a semi-structured interview.
The questionnaire covered information regarding features and functionalities
of the applications, usage, and reflective learning. During the interview, com-
bined statistics (see Fig. 2) of the captured data were presented and discussed
in order to find out the insights and benefits gained for the individual user.
59
Application overlapping user profiles to foster reflective learning at work - ARTEL14
5.2 Results & Discussion
The analysis of the log data of both applications is depicted in Fig. 3. Because
of the small sample size only descriptive statistics are presented. As measure
of central tendency the median is used for the same reasons. Each data point
represents the average mood values (in terms of valence and arousal) of one
participant in relation to the application usage and working activities (switching
frequency and used resources). Whereas there is no trend to be derived from this
small sample for the active use of KnowSelf, the number of moods entered per
day seems to increase with higher valence and higher arousal values indicated
by the participants (i.e. with a more positive mood). Switching frequency was
measured in seconds between switching from one resource to another. Fig. 3
(bottom) shows that higher reported valence seems to be connected to longer
times between switches (that is a lower switching frequency) and fewer resources
used. Interestingly, the arousal level increases with the number of used resources.
Fig. 3. Relationship between mood (as valence and arousal), application usage and
working activities
Analysing the data collected via questionnaires and interviews, we can give
first answers to the research questions:
RQ1: Are participants interested and willing to use more than one applica-
tion in parallel with regard to reflective learning? Ratings from 6 participants
answering the questionnaire (using 5pt. agreement scales) indicate that there
is an interest in getting support for time-management (Md (median) = 4) as
well as in capturing one’s working activities, own mood, and the team mood
(all Md =3.5). Participants found the applications easy to use, liked their visu-
alisations, rated the presentation of information as comprehensible (all Md =4),
60
Application overlapping user profiles to foster reflective learning at work - ARTEL14
generally liked using the applications and would recommend them to colleagues
(for both items Md =4 for KnowSelf and Md =3.5 for MMA, respectively).
RQ2: Does the MUP App as overlapping application facilitate reflection about
users’ working experiences and contribute to raising awareness of multiple aspects
of their work life? The interview results revealed that the combination of data
has high potential to trigger reflective learning although we have ambiguous
statements in which way. One participant reflected mainly on the number of
used applications and its relation to how the level of arousal developed over the
day. Another participant mentioned that combined data helped her to detect a
working pattern, which occurred especially in the morning. After reading emails
the application switches and the arousal level increases, thus she knows that
she started to work. Similarly, one of the participants observed that her arousal
level is very low in the morning and increases during the day. This was a trigger
to compare her arousal level to the average level of her colleagues and reflect
upon eventual differences between them. An important feature mentioned by
more than one participant was the overlapping visualisation of captured data
on the timeline chart. Here the data was understood at one glance, which can
facilitate reflective learning and enhance awareness of the multi aspects of their
work life. Despite of the different approaches to reflect, for all participants the
combination of data captured by both applications was important to understand
the relationship between their working activities and moods.
RQ3: Do participants perceive any individual insights or benefits for them-
selves? Besides the findings already described in relation to RQ2, participants
reported some additional insights they gained by reflecting on the captured data
provided by the MUP. One participant stated that her arousal level fluctuates
during the day. By becoming aware of the falling arousal level she decided to
take smaller breaks to better recover during the day. Further insights concerned
participants’ self-estimations of how they spend their working day. Whereas one
participant stated that the captured data confirmed how she estimated the rela-
tionship between working activities and mood development, another participant
was rather surprised in the first place. Although she was six to seven hours in the
office she spent only four hours in front of her computer. Only after comparing
this awareness with her dates in her calendar, she could reproduce her day and
explain why this happened.
General discussion. In general, this first proof-of concept of the MIRROR In-
tegrated User Profile indicates that such overlapping visualisations can facilitate
individual reflective learning and raise overall awareness of users’ work life. All six
participants used the combined data to reflect on how their working activities are
related to mood changes and could gain some individual insights. Nevertheless
there a still some points which need further discussion. While KnowSelf captures
automatically the resources and applications used on the PC, the moods need
to be inserted manually. Having to repeatedly insert a mood in a web based
application can distract from the normal working process. One recommenda-
tion to alleviate this distraction was to add five different smileys in the system
tray to facilitate the mood capturing. Another point for consideration is the
61
Application overlapping user profiles to foster reflective learning at work - ARTEL14
optimal time for reflection. All of the participants perceived the combination of
the data captured by the MoodMap App and KnowSelf as useful, because they
could check at all times what they were doing during work and how they felt.
However, one participant stated that it was not very useful to reflect on how
she felt three days ago, but that it was more interesting to become aware of
her mood in relation to her work directly while working. For other participants
especially the knowledge of how they felt for example three days ago was very
important. Especially when the mood could be directly related to the mood
note, used applications or used resources. A rather interesting statement from
one of the participants was that her working tasks did not influence her mood
at all. With respect to the visualisations, the interviews showed that different
types of aggregating the data would be useful, so that users could indicate their
individual preferences, e.g. to visualise the data along a timeline, to aggregate
on an hourly basis, or to offer a summarising view in form a pie chart.
In summary the MUP App provides new visualisations based on data cap-
tured by different applications, and therefore offers a multitude of new possibili-
ties for individual interpretations. In our proof of concept, we only combined data
of two applications, but also within this small setting we received different ap-
proaches on how the participants interpreted the captured data for themselves
and what they learned from it. We also mentioned some shortcomings which
must be taken into consideration when proceeding with the development of the
MUP App. Nevertheless, our findings encourage the assumption that combining
data of more than two applications, leads to more meaningful possibilities to
interpret the data and to gain more diverse insights for oneself.
6 Conclusions and Outlook
In this paper we presented the new MIRROR integrated User Profile Applica-
tion, which aims at supporting reflective learning at work. Based on the results
from previous user studies, which evaluated single applications, we derived es-
sential requirements for the development of the MUP App and implemented a
first prototype. Results from a first small evaluation regarding the parallel usage
of two applications indicate that combining data captured by different appli-
cations, analysing and visualising them together can further facilitate reflective
learning. Furthermore, it can also enhance awareness of the work life by leading
the users to get more diverse insights about themselves. Of course, after this first
proof-of-concept, user-studies with larger samples and more applications need to
follow. Thus, our future work will focus on the integration of further applica-
tions developed within the MIRROR project into the MUP App. The goal is to
provide different variants of visualising combined data and more sophisticated
ways to provide guidance for reflective learning. For example, Fig. 4 combines
KnowSelf data with corresponding geo location data captured by another MIR-
ROR App. Thus it makes you aware of your working activities in relation to
your working places (e.g. customer visits or travel activities) and can provide
more triggers for reflective learning.
62
Application overlapping user profiles to foster reflective learning at work - ARTEL14
Fig. 4. Further sophisticated visualisations for the MUP App
7 Acknowledgement
The project “MIRROR - Reflective learning at work” is funded under the FP7
of the European Commission (project number 257617). The Know-Center is
funded within the Austrian COMET Program - Competence Centers for Ex-
cellent Technologies - under the auspices of the Austrian Federal Ministry of
Transport, Innovation and Technology, the Austrian Federal Ministry of Econ-
omy, Family and Youth and by the State of Styria. COMET is managed by the
Austrian Research Promotion Agency FFG.
References
1. Adomavicius, G., Tuzhilin, A.: Toward the next generation of recommender sys-
tems: a survey of the state-of-the-art and possible extensions. Knowledge and Data
Engineering, IEEE Transactions on 17(6), 734–749 (June 2005)
2. Boud, D., Keogh, R., Walker, D.: Reflection: Turning Experience into Learning,
chap. Promoting Reflection in Learning: a Model., pp. 18–40. Routledge Falmer,
New York (1985)
3. Brusilovsky, P., Millán, E.: User models for adaptive hypermedia and adap-
tive educational systems. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.)
The Adaptive Web, chap. User Models for Adaptive Hypermedia and Adap-
tive Educational Systems, pp. 3–5. Springer-Verlag, Berlin, Heidelberg (2007),
http://dl.acm.org/citation.cfm?id=1768197.1768199
63
Application overlapping user profiles to foster reflective learning at work - ARTEL14
4. Bull, S., Dimitrova, V., McCalla, G.: Open learner models: Research questions.
International Journal of Artificial Intelligence in Education (IJAIED Special Issue)
17 (2), 83–87 (2007)
5. Bull, S., Kay, J.: Student models that invite the learner: The smili:open learner
modelling framework. International Journal of Artificial Intelligence in Education
17, 89–120 (2007)
6. Daudelin, M.W.: Learning from experience through reflection. Organizational Dy-
namics 24(3), 36–48 (1996)
7. Dingsøyr, T.: Postmortem reviews: purpose and approaches in software engineer-
ing. Information and Software Technology 47, 293–303 (2005)
8. Duval, E.: Attention please!: Learning analytics for visualization and recommen-
dation. In: Proceedings of the 1st International Conference on Learning Ana-
lytics and Knowledge. pp. 9–17. LAK ’11, ACM, New York, NY, USA (2011),
http://doi.acm.org/10.1145/2090116.2090118
9. Fessl, A., Rivera-Pelayo, V., Pammer, V., Braun, S.: Mood tracking in virtual
meetings. In: Proceedings of the 7th European conference on Technology Enhanced
Learning. pp. 377–382. EC-TEL’12, Springer-Verlag, Berlin, Heidelberg (2012)
10. Fischer, G.: User modeling in human-computer interaction. In: User Modeling and
User-Adapted Interaction (UMUAI). vol. 11, pp. 65–86 (2001)
11. Itten, J.: Kunst der Farbe. Otto Maier Verlag, Ravensburg, Germany, 1 edn. (1971)
12. Kay, J.: Learner control. User Modeling and User-Adapted Interaction 11, 111–127
(2001)
13. Kay, J., Kummerfield, B.: Adaptive Technologies for Training and Education, chap.
Lifelong learner modelling, pp. 140–165. Cambridge University Press (2011)
14. Lindstaedt, S., Kump, B., Beham, G., Pammer, V., Ley, T., Dotan, A., de Hoog, R.:
Providing varying degrees of guidance for work-integrated learning. In: Sustaining
TEL: From Innovation to Learning and Practice: 5th European Conf.Technology
Enhanced Learning (ECTEL 2010). pp. 213–228. LNCS 6383, Springer (2010)
15. Pammer, V.: D4.1 Update - Results of the user studies and requirements on ‘Indi-
vidual Reflection at Work’ (2012)
16. Pammer, V., Bratic, M.: Surprise, surprise: Activity log based time analytics for
time management. In: CHI Extended Abstracts. pp. 211–216 (2013)
17. Rivera-Pelayo, V., Munk, J., Zacharias, V., Braun, S.: Live interest meter: Learn-
ing from quantified feedback in mass lectures. In: Proceedings of the Third Inter-
national Conference on Learning Analytics and Knowledge. pp. 23–27. LAK ’13,
ACM, New York, NY, USA (2013), http://doi.acm.org/10.1145/2460296.2460302
18. Rivera-Pelayo, V., Zacharias, V., Müller, L., Braun, S.: Applying quantified self
approaches to support reflective learning. In: Proceedings of the 2Nd International
Conference on Learning Analytics and Knowledge. pp. 111–114. LAK ’12, ACM,
New York, NY, USA (2012), http://doi.acm.org/10.1145/2330601.2330631
19. Santos, J.L., Verbert, K., Govaerts, S., Duval, E.: Addressing learner issues with
stepup!: An evaluation. In: Proceedings of the Third International Conference on
Learning Analytics and Knowledge. pp. 14–22. LAK ’13, ACM, New York, NY,
USA (2013), http://doi.acm.org/10.1145/2460296.2460301
20. Schwantzer, S.: D2.2 Software architecture – version 3 (2012), http://mirror-
project.eu/showroom-a-publications/downloads/finish/5/68
21. Siemens, G..: What are learning analytics? (Februrary 2010),
http://www.elearnspace.org/blog/2010/08/25/what-are-learning-analytics/
64
Reflection as Support for Career Adaptability: A
Concept for Reflective Learning in Public Administration
Michael Prilla1, Oliver Blunk1, Jenny Bimrose2, Alan Brown2
1
Ruhr-Universität Bochum, Germany
{michael.prilla,oliver.blunk}@rub.de
2
University of Warwick/Institute for Employment Research, Coventry, UK
{Jenny.Bimrose,Alan.Brown}@warwick.ac.uk
Abstract. Reflective learning is a mechanism to turn experience into learning.
As a mechanism for self-directed learning, it is critical for success at work. This
is true for individual employees, but also for teams and whole organizations.
Change processes are typical situations in which people question their practices,
reflect on how they adopt new practice and try to learn from good or bad prac-
tice for future work. Such changes can support the development of expertise. In
cases involving substantive changes in work roles and behaviours they may
even contribute to a process of professional identity transformation, which leads
to a deeper understanding of one’s own practice and of the processes involved
in adapting it to internal and external constraints. Public administrations are ex-
amples of organisations that undergo constant change due to changes in legisla-
tion, financial pressure and demands of the public, and the public intensively
observes them. These pressures are requiring staff to adapt, including by chang-
ing their professional identities. Integrating reflection into the practice of staff
can support them in informal learning and improving, and it can thus lead to
enhanced and more efficient services for the public. In this paper, we report on
an approach of using reflection in Public Employment Services (PES) in Eu-
rope, which are currently being transformed from being principally concerned
with administration of benefits and provision of advice and guidance to an in-
creasing concern with coaching clients and co-operating with employers.
1 Introduction
Reflection is a common and desirable process of learning from experience for future
work [1–3], which is carried out by individuals or groups in a self-directed manner
and as a mostly informal learning process [4–6]. Reflection helps workers to deal with
changing work contexts [4], supports them to create innovative practices [7] and ena-
bles them to create change in a bottom-up process [6]. Besides daily problems, up-
coming and continuous change processes are typical examples in which people ques-
tion the way they work(ed) and whether they have adopted new processes adequately.
Public administrations are typical examples of organizations facing constant
change, due to: legislative changes, budget and staff cuts, requirement for remaining
65
Reflection as support for career adaptability - ARTEL14
staff to become more efficient, the public demand for higher service quality, role
changes for staff, and, in some cases, alternative service provision (e.g., [8]).
Our work focuses on Public Employment Services (PES) as an example of chal-
lenges faced at public administrations: staff are dealing with more clients in a rapidly
changing labour market and are expected to offer a wider range of services. In many
European countries staff roles are being transformed from offering advice on access
to benefits and available job opportunities towards facilitation and coaching where
staff are expected to support clients in becoming more self-directed and staff are also
expected to understand the labour market better and engage more with employers.
Staff therefore need to be capable of adapting to various and often unforeseeable
changes. Career adaptability [9] as a process of continuously adapting to changing
requirements on the labour market is a central concept in this context. This process is
closely connected to self-reflection and reflection in groups [9], but work investigat-
ing reflection support tools for career adaptability is not available. This paper con-
nects research on reflection support to career adaptability research by presenting a
conceptual approach and a prototype to support this process with reflection tools.
2 Related Work
2.1 Career Adaptability and Professional Identity Transformation
Career adaptability is the ability to manage successful transitions in employment,
training, education and other contexts. It is key for workers dealing with constantly
changing requirements on the labour market [9]. Adapting careers, however, needs a
transformation of one’s individual and collective professional identity, including as-
pects such as work activities and organisation, relations to other professions and pro-
fessional culture [10]. This transformation can be triggered by challenges at work and
needs self-directed learning, self-reflection and learning in interaction with others [9].
Therefore support needs to include individual and collective means.
Public Employment Services (PES) practitioners deal with career adaptability both
in their personal careers and in the careers of clients they are supporting. Therefore
supporting them in career adaptability not only supports their personal career but also
supports their clients to re-enter the labour market.
2.2 Reflective Learning at Work
Following Boud (1985) [1] we understand reflection as a process of conscious re-
evaluation of experience for the purpose of guiding future behaviour. This perspective
is in line with the conception proposed by Schön (1983) [3], who in addition differen-
tiates between reflection-in-action and reflection-on-action, and other authors dealing
with reflective learning. In addition, we understand work and learning as intertwined
[3, 11], and therefore also work and reflection [12]: reflection transforms experience
from work into knowledge applicable to the challenges of daily work and thus needs
to be understood as a key process for informal learning at the workplace [5]. It is
66
Reflection as support for career adaptability - ARTEL14
mostly triggered when individuals or groups perceive some discrepancy, e.g. contra-
dictory information, incongruent feelings, interpersonal conflicts and other occurrenc-
es during work, leading to a state of discomfort that the individual or group wants to
overcome [13]. Characteristic activities of reflection can then be found in asking for
feedback on your work and opinions, critical opinion sharing (and being open to it in
the organisation) or challenging groupthink (instead of going with the majority) [14].
In addition to most models we differentiate between individual reflection as a
mostly cognitive activity and collaborative reflection, which is done in communica-
tion among peers in a group [15]. The latter has been found to create results that
transcend the capabilities of a group’s members [7] and it is a promising process for
the creation of innovation and change in modern workplaces [4], but it has received
less attention in work on reflection at work. Knipfer et al. (2013) [13] point out that as
workplaces provide individuals with a social context, individual and collaborative
learning are intertwined and must be considered together.
It has been shown that reflective learning can be supported by technology (e.g.,
[15–17]) by providing data or written content on experiences to reflect upon, support-
ing retrospective analysis or by scaffolding the reflective process, the documenting
and sharing of a decision rationale. More specifically, writing down positive or nega-
tive experiences and being prompted regularly to think about them has been shown to
be supportive for individuals to engage in continuous reflection [18]. However, as
most existing work either supports early phases of reflection (e.g., gathering and shar-
ing data) or stems from educational settings, which are often designed in favour of
reflective learning, there is still work to be done in the context of reflection at work.
3 Reflection for Professional Identity Transformation: A
Concept
The development of career adaptability relies on four key dimensions: learning to
adapt through challenging work, through updating a substantive knowledge base, by
being self-directed and self-reflexive as well as learning through interactions at work
[9]. In this section we show how reflection can support these dimensions and how this
can be used as a basis of professional identity transformation.
Dealing with challenging work can bring up discrepancies in daily work, which (as
described above) trigger reflection [13]. Successfully dealing with these situations can
lead to confidence in one’s skills and abilities. Reflecting about work and its chal-
lenges comes into play when there are no problem solving patterns available for the
challenges met and new solutions are needed [19].
To keep up with knowledge in changing fields of work learning through updating a
substantive knowledge base is required. While workers often use formal learning
offers at work, informal learning can be seen as a key to continuously understand
which knowledge is needed and integrate it into one’s context [9]. Reflection can
support these needs [5] and the integration of new knowledge [20], helping workers to
think about the state of their own knowledge and to identify learning goals, reviewing
existing goals and periodically checking whether they are met or need to be altered.
67
Reflection as support for career adaptability - ARTEL14
Adapting through self-directed learning and self-reflexiveness is closely related to
individual reflection. Tools can help to sustain issues to be reflected upon and to cre-
ate awareness for them [12, 18]. This combines self-directed and externally triggered
reflection, for example by setting up and periodically reviewing career goals in a tool.
Career adaptability by learning through interactions at work can benefit from sup-
port for collaborative reflection. Tools can help to create opportunities for reflection
even if individuals cannot meet in person [15]. Individuals can support informal learn-
ing of their peers by providing their experiences and insights or helping them to re-
flect about their own learning. Additionally colleagues can reflect to support each
other, for example, in coping with emotional work and/or stress and in exchanging
best practices in dealing with difficult situations. A team can reflect collaboratively to
improve their team performance and organize their learning efforts.
4 Applying the Concept: Reflective Learning Needs in Practice
Our work is inspired by field visits, workshops and expert interviews at different Eu-
ropean Public Employment Services (PES) agencies, including Germany, Slovenia
and the UK. In an early phase of this work we are currently exploring needs and op-
portunities for reflective learning as well as constraints and potential of implementing
it in such workplaces. From this work we describe examples of challenges faced in
many European PES and how reflection can be a key process in tackling them.
4.1 Supporting Change by Reflection on Training
In one of the agencies (referred to as agency A in this paper) staff are supposed to
change from providing advice and guidance to clients on benefits and job opportuni-
ties to coaching them to become more self-directed and to take responsibility for their
own future by proactively looking for ways to develop their skills and possible future
career paths. To support this change staff receive a two-day training on coaching
methods and related topics and an additional half-day session some time after training
to support the application of the methods in practice. Despite this support, staff mem-
bers reported that they had difficulties in implementing this new way of working, and
that they were struggling in reaching good results from coaching their clients.
This situation is an example of challenging work, and it shows how workers strug-
gle with updating their personal knowledge base. Reflecting on their practice of using
methods and tools of coaching can help PES practitioners to conduct more successful-
ly the transition to be a coach and thus may make training more sustainable. This may
approach mostly benefits from individual reflection of goals stemming from training
and involving workers in this reflection continuously (by reminding them to reflect).
4.2 Supporting Interaction with External Stakeholders by Reflection
In agency B the government requires PES staff to cooperate closely with employers to
enhance the conditions of the labour market, including the creation of new jobs, new
68
Reflection as support for career adaptability - ARTEL14
fields of employment and career opportunities. Staff are motivated to adapt to this
strategy, but also told us that this does not come easy and that there is a need for good
practices in implementing it. Some reported that talking to colleagues from other sub-
sidiaries had given them insights into how they might improve this work.
Becoming a co-operator with employers can be seen as an example of challenging
work, and from the feedback of practitioners we can see which discrepancies it caus-
es. We can also see that there is a desire to engage in exchange with others to reflect
on such discrepancies. Collaborative reflection on their work with employers can
therefore be seen as a means to make sense of typical challenges in this work, to ex-
change work practices and to learn from each other.
5 A Prototype for Reflection Support
The scenarios above show that support needs for learning about challenges Public
Employment Services (PES) practitioners face are diverse, and that support for sus-
taining experiences, reflecting upon them, sharing them and finding similar experi-
ences need to be close to work tasks. To
explore how such support can create impact
in PES agencies we created a mobile proto-
type supporting the reflective learning sce-
narios describe above. Using mobile devic-
es makes support independent from corpo-
rate IT infrastructures (which are usually
hard to access away from the office in PES)
and enables users to use the tool when and
where they want, for example after talking
to employers or after work, e.g. while using
public transportation on the way home.
In the prototype users can write personal
notes about experiences at work (upper part
of Fig. 1) and they can enter reflections
multiple times about these notes (bottom
part of Fig. 1), including an assessment of
how they feel about the experience (see the
smiley icons in the bottom half of Fig. 1).
The prototype also includes an easy to
use sharing feature to enable collaborative
reflection. To enhance personal engage-
ment in collaborative reflection, when shar-
ing content with colleagues the system
offers users the opportunity to choose from
predefined questions (or create a new ques- Fig. 1. The prototype allows users to write
tion) to share together with the content. notes and to reflect on them. All notes and
This aims at provoking reflection: For ex- reflections can be shared.
69
Reflection as support for career adaptability - ARTEL14
ample, as user might choose a question such as “Did you ever encounter a similar
situation? What did you do?” when sharing the description of an issue. This may per-
sonally impact colleagues, who feel personally invited to engage with the user sharing
the content and motivated to help her. This may help to establish communities of
practice helping each other and it facilitates collaborative reflection by engaging users
in conversations about challenging work.
The tool periodically prompts users individually or collaboratively to revisit past
issues and reflections. This can be useful to capture changes in perspectives on expe-
riences over time and the resulting insights leading to this change. For example, if a
user from agency A experiences she cannot implement a certain aspect of the new
coaching process, she may improve over time, also rating this experience more posi-
tive after some time (see Fig. 1). It is also possible just to share one of the newer en-
tries of a reflection with another user to enable collaborative reflection on specific
aspects of the evolving situation. Using the tool in this way builds up an individual
and collective knowledge base on aspects related to career adaptability.
Users control when they are prompted for reflection: they can let the system (con-
textually) determine when to prompt them or they can set reminders to reflect. This
for example can be used to notify a user while she is using the bus on the way home
and wants to reflect on situations she had experienced that day. This supports self-
reflexiveness as part of career adaptability.
The prototype provides novel features such as sharing personal questions with re-
flection content and periodically promoting users for individual and collaborative
reflection, which are directed towards engaging with challenging work and to support
career adaptability. Future work will also aim to integrate its features into existing
tools in order to better integrate reflection for career adaptability into daily tasks.
6 Discussion and Outlook
We have presented ongoing work in supporting career adaptability in public admin-
istration workplaces by reflection support. Our work is in its early stages, and we
have created a concept for such support, situated it in needs of learning in PES organ-
izations as typical examples of public administration and showed its feasibility by
implementing a prototype. Next steps will include using the prototype with groups of
PES practitioners in different agencies and improving the support it provides. In the
ARTEL workshop we would like to discuss the concept and how it may be improved.
7 Acknowledgements
The work described in this paper is part of the EmployID project funded by the Euro-
pean commission in FP7 (project number 619619). We would like to thank all col-
leagues in the project for their cooperation and our fruitful discussions.
70
Reflection as support for career adaptability - ARTEL14
8 References
1. Boud, D.: Reflection: Turning experience into learning. Kogan Page, London (1985).
2. Kolb, D.A.: Experiential learning: Experience as the source of learning and development.
Prentice-Hall, Englewood Cliffs, New Jersey (1984).
3. Schön, D.A.: The reflective practitioner. Basic Books, New York (1983).
4. Cressey, P., Boud, D., Docherty, P.: Productive Reflection at Work. In: Boud, D., Cressey,
P., and Docherty, P. (eds.) Productive reflection at work: Learning for changing organiza-
tions. pp. 11–26. Routledge, New York (2006).
5. Eraut, M.: Informal learning in the workplace. Studies in continuing education. 26, 247–
273 (2004).
6. Prilla, M., Pammer, V., Krogstie, B.: Fostering Collaborative Redesign of Work Practice:
Challenges for Tools Supporting Reflection at Work. Proceedings of the European Confer-
ence on Computer Supported Cooperative Work (ECSCW 2013) (2013).
7. Hoyrup, S.: Reflection as a core process in organisational learning. Journal of Workplace
Learning. 16, 442–454 (2004).
8. Giesbrecht, T., Schenk, B., Schwabe, G.: Learning with Facilitation Affordances: The
Case of Citizens’ Advice Services. Proceedings of the Twenty Second European Confer-
ence on Information Systems (ECIS 2014) (2014).
9. Brown, A., Bimrose, J., Barnes, S.-A., Hughes, D.: The role of career adaptabilities for
mid-career changers. Journal of Vocational Behavior. 80, 754 – 761 (2012).
10. Brown, A.: A dynamic model of occupational identity formation. In: Brown, A. (ed.) Pro-
moting Vocational Education and Training: European Perspectives. pp. 59–67. University
of Tampere, Tampere (1997).
11. Lave, J., Wenger, E.: Situated learning: Legitimate peripheral participation. Cambridge
University Press, Cambridge (1991).
12. Krogstie, B., Prilla, M., Pammer, V.: Understanding and Supporting Reflective Learning
Processes in the Workplace: The RL@Work Model. Proceedings of the Eigth European
Conference on Technology Enhanced Learning (EC-℡ 2013) (2013).
13. Knipfer, K., Wessel, D., Cress, U.: Reflection as a catalyst for organizational learning.
Studies in continuing education. 35, 30–48 (2013).
14. Van Woerkom, M., Croon, M.: Operationalising critically reflective work behaviour. Per-
sonnel Review. 37, 317–331 (2008).
15. Prilla, M., Degeling, M., Herrmann, T.: Collaborative Reflection at Work: Supporting In-
formal Learning at a Healthcare Workplace. Proceedings of the ACM International Con-
ference on Supporting Group Work (GROUP 2012). pp. 55–64 (2012).
16. Fleck, R., Fitzpatrick, G.: Teachers’ and tutors’ social reflection around SenseCam images.
International Journal of Human-Computer Studies. 67, 1024–1036 (2009).
17. Xiao, L., Clark, S., Rosson, M.B., Carroll, J.M.: Promoting Reflective Thinking in Collab-
orative Learning Activities. Proceedings of the 2008 Eighth IEEE International Conference
on Advanced Learning Technologies. pp. 709–711. IEEE Computer Society, Washington,
DC, USA (2008).
18. Isaacs, E., Konrad, A., Walendowski, A., Lennig, T., Hollis, V., Whittaker, S.: Echoes
from the past: how technology mediated reflection improves well-being. Proceedings of
CHI 2013. pp. 1071–1080. ACM, Paris, France (2013).
19. Raelin, J.A.: I don’t have time to think!” versus the art of reflective practice. Reflections.
4, 66–79 (2002).
20. Davis, E.A.: Scaffolding students’ knowledge integration: Prompts for reflection in KIE.
International Journal of Science Education. 22, 819–837 (2000).
71
72
Boosting Informal Workplace Learning
in Small Enterprises
Miloš Kravčík, Kateryna Neulinger, Ralf Klamma
Advanced Community Information Systems (ACIS), Informatik 5,
RWTH Aachen University, Germany
{kravcik, neulinger, klamma}@dbis.rwth-aachen.de
Abstract Where participation of small enterprises in vocational education and
training decreases, it risks obsolescence of their knowledge base compared to
competitors. Currently we are participating in two projects that aim to address
the issue of how to boost take-up of informal learning at the workplace:
Learning Layers and BOOST. Previous projects [e.g., ROLE] show the
importance of having personalised learning solutions with high relevance, high
effectiveness and low barriers to use. Therefore we aim to provide predefined
and customizable Personal Learning Environments that support awareness and
reflection of users, especially workers in small enterprises.
Keywords: Informal Workplace Learning, Personal Learning Environments.
1 Introduction
Support of informal learning at the workplace is real issue and we attempt to address
it in two projects: Learning Layers [1] and BOOST [2]. While the first one is dealing
with the problems of scalability and scaffolding, the second one is focusing on small
enterprises (up to 20 employees) and their needs. Both of them build on the outcomes
of the former ROLE project [3], especially the technological platform that facilitates
design and development of Personal Learning Environments (PLEs) [4]. Moreover,
BOOST considers innovative methodologies from the BeCome [5] project that
identify the Business Goals of small companies and manage the associated learning
processes. The PLEs provide customized learning and training solutions that enable to
meet the specified Learning Indicators. The overall aim is to support employees in
training activities and to facilitate their personal development. For this purpose we
want to integrate learning in their work processes. We develop widgets that should
support awareness and reflection of various types of users in practice. In this context
it is crucial to consider specific constraints and requirements of small companies, in
order to make the developed solutions attractive and useful for all different roles:
managers, trainers and employees. Our solutions support personal competence
development at the workplace in all phases, i.e. planning, learning, and reflection.
They help to identify business goals and existing competence gaps. Moreover, they
recommend learning resources from existing repositories and suitable peers in
communities of practice.
73
Boosting informal workplace learning in small enterprises - ARTEL14
2 BOOST Technical Prototype
Our proposed solution should support awareness by augmenting informal learning
with relevant information of the business goals, current and target competences of
employees, time plans, learning resources and learning progress overviews on various
levels (e.g. company, employee). Reflection is an important part of self-regulated
learning that helps the users to evaluate their progress and to plan the next steps.
These features had to be considered in the BOOST technical prototype, which is still
work in progress. It includes this basic workflow: 1. Identify critical business goals in
the company. 2. Select employees to address them. 3. Support their learning. 4.
Monitor the learning progress of the company and of the individual employees. Our
data model is hierarchical: 1. Business Goals (BGs – e.g. Web development). 2.
Learning Indicators (LIs – e.g. Web design, information architecture). 3. Learning
Resources (LRs – including learning materials, tools and peers).
Figure 1: Employee Environment
We distinguish 3 different user roles that have different characteristics and
requirements: Manager (e.g. business manager, business advisor or consultant),
Trainer (e.g. training manager, learning facilitator) and Employee. Manager specifies
BGs for the company, decides which BGs are urgent and which of them are relevant
for which employee. Moreover, this role can also assess employees and monitors their
learning progress. Trainer describes LIs for selected BGs and the relevancy of LIs for
individual employees, recommends LRs for the LIs, and chooses relevant Learning
Repositories, where additional LRs can be found. Employee (Fig. 1) gets an overview
of BGs and LIs assigned to her, together with the recommended LRs. According to
74
Boosting informal workplace learning in small enterprises - ARTEL14
the descriptions of LIs she can search for additional LRs in the predefined Learning
Repositories and add them to her portfolio. She can also access the selected LRs in
order to learn. Finally, she can monitor her learning progress.
The functional requirements for competence management include: 1. Specification
of relevant BGs (high level competences), their priorities and time scales. 2.
Assignment of LIs (concrete competences) to each BG, considering also time scales.
3. Assignment of LRs to LIs. 4. Assignment of relevant BGs and LIs to employees. 5.
Setting up target LI (proficiency) levels for relevant BGs for each employee,
considering time scales. 6. Assessment of the start and current LI (proficiency) levels
for the employee. 7. Monitoring the training progress in the company and also of each
employee (considering also time scales). The functional requirements for the learning
support are still relatively vague, as they will be more domain dependent: 1.
Community support – sharing experience, communication, and collaboration. 2.
Domain specific support – learning and assessment. 3. Annotation of learning
resources assigned to LIs. 4. Considering preferences of individuals.
3 Conclusion and Future Work
In the first year the BOOST consortium identified the main requirements and
designed a solution. Afterwards we have developed the first version of the technical
prototype, which has been evaluated in interviews with 15 stakeholders. Based on
their outcomes the technical prototype will be updated and enhanced with additional
features, including privacy requirements and personalization. The current version is
suitable for companies with open environments, where employees do not mind seeing
each other’s competences and learning progress. But in many companies more
privacy is demanded, where employee can see just his or her data. Another important
feature is assignment of timescales to business and learning goals as well as their
monitoring and notifications. The new version will be tested in companies.
Acknowledgments. The presented research work is partially funded by the German
National Agency BiBB within the Lifelong Learning Programme Leaonardo da Vinci:
“Business PerfOrmance imprOvement through individual employee Skills Training”
(project no: DE/13/LLP-LdV/TOI/147655) and by the 7th Framework Programme
large-scale integrated project “Learning Layers” (grant no: 318209).
References
[1] Learning Layers project http://learning-layers.eu/
[2] BOOST project http://www.boost-project.eu
[3] ROLE project http://www.role-project.eu/
[4] Nussbaumer, A., Kravcik, M., Renzel, D., Klamma, R., Berthold, M., & Albert, D. (2014).
A Framework for Facilitating Self-Regulation in Responsive Open Learning
Environments. arXiv preprint arXiv:1407.5891.
[5] BeCome project http://become.dedi.velay.greta.fr
75
76
Activity Recommendation App – Software to Evaluate
the Usefulness of Improvement Recommendations
Created in a Team
Nils Faltin, Simon Schwantzer, Margret Jung
IMC information multimedia communication AG, Saarbrücken, Germany
{nils.faltin,simon.schwantzer,margret.jung}@im-c.de
Abstract. The Activity Recommendation App supports employees in individual
and collaborative reflection by capturing discussions and solutions for problems
that need to be solved. The app enables employees to record personal experi-
ences with the solutions. Based on these experiences the usefulness of a rec-
ommendation can be re-evaluated in order to approve, update, or discard the
recommendation. The application was successfully evaluated in coaching em-
ployees in learning time management techniques.
Keywords: ARA – Activity Recommendation App· soft skills improvement·
recommendation evaluation· solution· experiences· MIRROR Spaces Frame-
work· time management coaching
1 Introduction
Reflection on work experiences can lead to new insights and ideas how to handle
work situations better in the future. But the capturing of experiences during work and
the reflection on this data is only half the way for a successful improvement. The
other half is the creation of a viable reflection outcome and the validation of this out-
come when it is applied in practice (see [1]). Based on this validation, a change can be
approved, reverted, or improved and validated again.
Whilst a lot of applications support users to capture data during work in order to pro-
vide it in a subsequent reflection session, the second half of the reflection cycle is
often left unsupported. To also cover this part, the Activity Recommendation App
(ARA) was created in the MIRROR project [2]. It supports the discussion of im-
provement ideas in an individual or collaborative reflection session and frames the
outcome as recommendation. The app allows capturing personal experiences relating
to active recommendations and viewing other members’ experiences if a recommen-
dation targets a team. Finally, the ARA supports the evaluation of a recommenda-
tions’ usefulness when applied in practice, in order to enable its improvement or sus-
pension. By providing these features, the Activity Recommendation App aims to im-
prove the application of insights gained from reflection on work.
77
Activity recommendation app - ARTEL14
2 Overview of the Main Functionalities
The recommendation is created in an individual or a collaborative reflection session.
Major elements of this session are the identification of the concrete issue and a viable
solution for this issue. Texts, files, or data from other MIRROR applications can be
attached to be used as evidence to back the comprehensibility of a recommendation.
The events during the discussion are listed as a kind of minutes. Measurement criteria
can be selected to evaluate the usefulness of the recommendation. Before publishing
the recommendation, a target person/group is selected and invited to try the new solu-
tion.
A concrete scenario could look like this (cf. [3]): A team uses ARA to find a solution
for their common problem of overtime spent for pending projects. They agree that
frequent interruptions can be one reason for this (issue). In the scenario the teams’
solution is to implement three hours of quiet working time a day and to avoid inter-
ruptions during that period (recommended solution).
Fig. 1. A recommendation in ARA with one experience being entered
To test the recommendation in practice, personal experiences are written down to
decide about how well the recommended solution applied (see Figure 1). Users cap-
ture their experiences by noting down a comment and by rating how well the solution
worked (1 to 5 stars). In addition they can record the effort (e.g., the minutes of work-
ing time required) and the benefit (e.g., the number of completed tasks) of applying
the solution. These experiences are shared with the other members of the target group
to benefit from the application in a group.
To evaluate a recommendation, the app allows users to view all experiences with an
aggregation of the ratings, efforts and benefits captured. All this can then be used to
get an overview how well the solution works in practice to be taken as a basis for the
decision if the solution should be kept, updated or discarded.
78
Activity recommendation app - ARTEL14
In the continuation of the exemplary scenario, the team discusses the recommenda-
tion’s weak points (due to the captured experiences) during the regular team meeting
and agrees on adapting it in respect to the selected period in time. It is then re-
evaluated, re-discussed, and finally marked as solved when team agrees about a well-
functioning final solution.
3 Evaluation & Outlook
A summative evaluation of the ARA took place at our company IMC over a period of
six weeks. Ten staff members took part in a time management coaching. The ap-
proach combined the usage of a computer activity tracking tool and the ARA with a
weekly coaching session. In the weekly coaching sessions the coach and the coachee
reviewed the individual progress, adjusted the time management rules if not appropri-
ate anymore, or, when the particular goal has been achieved and the new behaviour
has been adopted, decided that no further practice regarding that goal is needed.
The Activity Recommendation App served well as a support for learning time man-
agement by providing a data basis for the coaching sessions. It was used by coach and
coachee to set time management goals and to document and monitor the progress in
learning new time management techniques. Both benefited from the better preparation
for the coaching sessions available with the notes in ARA. Furthermore, the app
helped the coachees to focus their goals. Two things were missed concerning ARA: It
lacks an interface optimized for smartphones and currently no reminder function is
available which motivates the user to capture experiences. These shortcomings can be
addressed in future development.
The coach and several coachees also suggested forming peer groups to train time
management techniques. They could then benefit from sharing experience data to
compare own progress with that of others and learn from each other’s experiences.
IMC has started a free online course for time management that includes usage of the
ARA [4]. In addition to the course, learners can book a human tele-coach for a fee.
References
1. Krogstie, Birgit, Michael Prilla, and Viktoria Pammer. “Understanding and Supporting Re-
flective Learning Processes in the Workplace: The RL@Work Model.” In Proceedings of
the Eigth European Conference on Technology Enhanced Learning (EC-℡ 2013), 2013.
2. MIRROR project for reflective learning at work, http://mirror-project.eu/
3. Video: Team usage scenario for the Activity Recommendation App,
http://vimeo.com/66798165
4. MOOC “Time Management“ (Gernan) on OpenCourseWorld, http://bit.ly/1ufnS0v
79
80
DDART: an Awareness System to Favor Reflection
during Project-Based Learning
Min Ji1, 2, Christine Michel1, 2, Elise Lavoué1, 2, Sébastien George3
1
Université de Lyon, CNRS
2
INSA-Lyon, LIRIS, UMR5205, F-69621, France
{min.ji; christine.michel; elise.lavoue}@liris.cnrs.fr
3
LUNAM Université, Université du Maine, EA 4023, LIUM, 72085 Le Mans, France
sebastien.george@univ-lemans.fr
1 Introduction
Our research aims to improve learners’ reflection and self-regulation in Project-Based
Learning (PBL). Actually, we observe that the implementation of PBL in engineering
schools, universities or professional training do not benefit from all its capacities,
because it is often action (according to the Kolb’s learning cycle) which is favored to
the detriment of reflection and personal experience [1]. Our approach considers Self-
Regulated Learning (SRL) as a major component of PBL to bring learners to self-
reflect on their experience and to apply metacognitive skills.
We focus our work on the design and the development of a dashboard based on
both reporting and activity traces [2]. The activity traces are automatically produced
by the users’ actions and recorded directly by the LMS during the learning activities.
The reporting traces are information reported by the learners themselves. Most dash-
boards use only automatic activity traces to produce indicators. We state that the ag-
gregation of these two types of traces allow producing more meaningful indicators for
the learners [3].
Most existing dashboards are designed for the tutors to monitor the learners but
they are rarely designed for the learners to support awareness during their activities.
Furthermore, the indicators are mostly predefined and the users can rarely build their
own indicators [3]. In this paper, we present the DDART system, which is composed
of two specific tools: a reporting tool that aims at enhancing learners’ reflection dur-
ing project-based learning and a tool to help learners to produce their own indicators
for enhancing awareness during their project. These two tools are integrated into a
same system (DDART) to enhance learners’ self-regulation thanks to personalized
indicators presented on a dashboard.
2 A reporting tool and a dynamic dashboard
We developed a Dynamic Dashboard Based on Activity and Reporting Traces
(DDART). We chose to implement this dashboard as a plug-in of the Moodle plat-
form. In our context, project members use the Moodle tools (wiki, forum, chat...) to
81
DDART: awareness system to favor reflection during project-based learning - ARTEL14
carry out the project. They are also asked to use a reporting tool to describe and keep
traces of the project events.
The reports are composed of semi-structured sentences so that this text information
can be collected and analyzed automatically [2]. Two types of reports are possible:
the goal report and the activity report. The former is written at the beginning of the
project to assist learners to plan their project and to set the goals they want to achieve.
The later can be filled in during the project. By completing the semi-structured sen-
tences, learners can describe the ways they carry out the project, their states of mind,
their judgments (who do what, when, where, with whom and how), their level of ac-
quisition of knowledge and skills. The semi-structured sentences are more flexible
than structured sentences and keep the possibility to collect organized and computable
data. By applying this reporting tool, learners can self-reflect on how they carried out
activities and learn how to organize their ideas and how to write effective reports.
We designed a specific interface to help learners to build their own indicators (see
Fig. 1). This interface is composed of three main parts: (1) the “parameters” part (see
Fig. 1.a), on the left side, contains the list of all the parameters which are available for
creating an indicator, (2) the “calculation” part (see Fig. 1.b and c), in the center, al-
lows learners to place the parameters and view the indicator results and (3) the “visu-
alization modes” part (see Fig. 1.d), on the right side. This user-friendly interface
allows learners to create the indicators by dragging and dropping the parameters and
the visualization mode. The calculation function is WYSIWYG: the results can be
calculated in real-time so that learners can easily adjust the parameters. At last, the
presentation of indicators on a dashboard provides awareness to the learners about the
way they carry out the project and also about the building of knowledge and skills.
Fig. 1. The interface to assist learners to create personalized indicators
82
DDART: awareness system to favor reflection during project-based learning - ARTEL14
The semi-structured sentences of the reporting tool (reporting traces) and the traces
of use of Moodle (activity traces) are respectively recorded in an XML database
(BaseX) and in a relational database (MySQL). These two kinds of traces are de-
scribed according to the same five common entities: Learner, Tool, Activity, Time
and Place. They are merged according to a common time basis and are stored into a
transformed traces base. The transformed traces are used to produce indicators stored
in a dedicated database [2]. An indicator is defined by 5 parameters:
• X entity and Y entity: these parameters can be chosen by the learners
among the instances of the five entities extracted from the transformed
traces (Learner, Tool, Activity, Time, Place). These entities are used to
specify the events the learners want to observe.
• Value: this parameter sets the type of aggregation proposed to produce the
data presented into the indicator. Four possibilities are proposed: frequen-
cy, time interval, time spent, content.
• Calculation function: the learners can refine the analysis of values by de-
fining other mathematic formula based on sum, difference, comparison
and average.
• Visualization: DDART offers eleven visualization modes for learners (pie
chart, bar chart, line chart, gauge chart, social network, scatter chart, area
chart, table, tree map, combo chart and Gantt).
3 Conclusion
In this paper, we have presented the basis of the DDART system. This system can
help learners to collect, analyze and visualize their reporting and activity traces in the
form of meaningful indicators. By allowing learners to create their own indicators, we
aim at making them learn how to regulate their learning activities. The traces collect-
ed in the reporting tool allow the construction of advanced indicators that can help
learners to build metacognitive skills. For example, the indicators can support the
analysis of behavior by comparing the learners’ feeling about their activities (subjec-
tive) with the realization mode of the activities (objectively recorded by the system).
References
1. Thomas, J., Mengel, T.: Preparing project managers to deal with complexity – Advanced
project management education. International Journal of Project Management. 26, 304-
315 (2008).
2. Ji M., Michel C., Lavoué E., George S. Combining Activity Traces and Reporting Traces
to Support Self-Regulation Processes. 13th IEEE International Conference on Advanced
Learning Technologies (ICALT 2013), Beijing, China, 15-18 July 2013, p. 432-436.
3. Ji M., Michel C., Lavoué E., George S. DDART, a Dynamic Dashboard for Collection,
Analysis and Visualization of Activity and Reporting Traces. 9th European Conference
on Technology Enhanced Learning (EC-TEL 2014), Graz, Austria, 16-19 September
2014, to be published.
83
84
LARAe: Learning Analytics Reflection &
Awareness environment
Sven Charleer, Jose Luis Santos, Joris Klerkx, and Erik Duval
Dept. of Computer Science, KU Leuven
Leuven, Belgium
{Sven.Charleer,JoseLuis.Santos,Joris.Klerkx,Erik.Duval}@cs.kuleuven.be
Abstract. Exploring and managing the abundance of data that Learn-
ing Analytics generate is a challenge for both teachers and students.
This paper introduces a Learning Dashboard that provides an overview,
context and content of learner traces to help students with awareness of
feedback and progress, and assist teachers with monitoring student effort
and outcomes to intervene where needed.
Keywords: learning analytics, learning dashboards, awareness, infor-
mation visualization, effort, intervention, inquiry-based learning
1 Introduction
The purpose of Learning Analytics is understanding and optimizing learning and
the environments in which it occurs [1]. Through dashboards, Learning Analytics
can help support both teacher and students [2].
Learning Dashboards can rely on many different ways of visualizing raw
analytics data e.g. bar, star and bubble charts, interactive histograms, parallel
coordinates etc [2]. These visualization techniques can provide broad insights on
student activities [3, 4]. By adding teacher traces, our visualization also attempts
to provide awareness of feedback to improve its supportive role for both student
and teacher.
This abundance of data can be abstracted to the essentials [5, 6], but context
and content can help provide deeper insights [7]. Following the visual information-
seeking mantra of “Overview first, zoom and filter, then details-on-demand” [8],
our dashboard presents users with an abstract overview while still retaining a
sense of context and providing access to the details.
2 LARAe: Design & Implementation
LARAe visualizes traces gathered from 38 engineering students, teachers and ex-
ternal participants in an open User Interfaces course. Students worked in groups
of 3 and reported weekly through blog posts, comments and Twitter. The course
generated 419 blog posts, 1580 comments and 538 tweets.
85
LARAe: Learning Analytics Reflection & Awareness environment - ARTEL14
Fig. 1. LARAe: A. Overview, B. Activities, C. Thread view
Every activity is represented by a circle (Figure 1.B) which provides direct
access to the related content (e.g. blog post, comment, tweet, retweet). Activities
are sorted chronologically, from top left to bottom right. Gradient color values
(see Figure 1.A) help recognize the age of an activity. A table (Figure 1.B)
structures the activities by student group and type. Every column represents
an activity type, every row a student group. The user can sort the data by any
activity type. Both activity age and amount help facilitate awareness of (in)active
groups. As teaching staff feedback was deemed important by both student and
teacher, a second table visualizes activities of teacher activity in a similar way.
Context plays an important role in understanding the activities e.g. a com-
ment without its surrounding discussion is difficult to assess. We propose a “fo-
cus+context” [9] solution which consists of 2 parts: highlighting related events
(Figure 1.B) and displaying the content within a thread view (Figure 1.C).
Highlighting related activities helps the user to instantly become aware of the
distribution of an activity thread across the class e.g. selecting a blog post will
highlight what groups provided most contributions. Simultaneously, the thread
view shows the content of each related activity, helping assess the quality of the
quantitative data. Visualizing discussion thread size can help students discover
interesting threads. Teachers might understand low thread size as an indication
for need of intervention. The attribute thread size is indicated by a number in
each circle (Figure 1.B).
86
LARAe: Learning Analytics Reflection & Awareness environment - ARTEL14
LARAe is a web application developed using HTML5, JavaScript and D3.js1
running on a Node.js2 web service and MongoDB3 database. It supports both the
proprietary API and Tin Can API4 . It can easily be extended to support other
APIs. The dashboard is designed to run on large displays, desktop computers
and tablets. It is available at http://ariadne.cs.kuleuven.be/LARAe/.
The dashboard has also been deployed in an inquiry-based learning setting,
visualizing the learner traces gathered from the weSPOT Inquiry system5 [10].
Acknowledgment The research leading to these results has received funding
from the European Community’s Seventh Framework Programme (FP7/2007-
2013) under grant agreement No 318499 - weSPOT project.
References
1. Siemens, G., Long, P.: Penetrating the fog: Analytics in learning and education.
Volume 46., Boulder, CO, USA, EDUCAUSE (2011) 30–32
2. Verbert, K., Duval, E., Klerkx, J., Govaerts, S., Santos, J.L.: Learning analytics
dashboard applications. American Behavioral Scientist 57(10) (2013) 1500–1509
3. McAuley, J., O’Connor, A., Lewis, D.: Exploring reflection in online communities.
In: Proceedings of the 2Nd International Conference on Learning Analytics and
Knowledge. LAK ’12, New York, NY, USA, ACM (2012) 102–110
4. Donath, J.: A semantic approach to visualizing online conversations. Communi-
cations of the ACM 45(4) (April 2002) 45–49
5. Xiong, R., Donath, J.: Peoplegarden: creating data portraits for users. In: Proceed-
ings of the 12th annual ACM symposium on User interface software and technology,
ACM (1999) 37–44
6. Arnold, K.E., Pistilli, M.D.: Course signals at purdue: Using learning analytics to
increase student success. In: Proceedings of the 2nd International Conference on
Learning Analytics and Knowledge, ACM (2012) 267–270
7. Charleer, S., Klerkx, J., Odriozola, S., Luis, J., Duval, E.: Improving awareness and
reflection through collaborative, interactive visualizations of badges. In: Proceed-
ings of the 3rd Workshop on Awareness and Reflection in Technology-Enhanced
Learning. Volume 1103 of CEUR Workshop Proceedings., CEUR-WS.org (2013)
69–81
8. Shneiderman, B.: The eyes have it: a task by data type taxonomy for information
visualizations. In: IEEE Symposium on Visual Languages, IEEE (1996) 336–343
9. Furnas, G.W.: A fisheye follow-up: Further reflections on focus + context. In:
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.
CHI ’06, New York, NY, USA, ACM (2006) 999–1008
10. Mikroyannidis, A., Okada, A., Scott, P., Rusman, E., Specht, M., Stefanov,
K., Boytchev, P., Protopsaltis, A., Held, P., Hetzner, S., Kikis-Papadakis, K.,
Chaimala, F.: wespot: A personal and social approach to inquiry-based learning.
Journal of Universal Computer Science 19(14) (2013) 2093–2111
1
http://d3js.org
2
http://nodejs.org/
3
http://www.mongodb.org
4
http://tincanapi.com/
5
http://portal.ou.nl/documents/7822028/f475d712-5467-40ea-968c-5aa00d951400
87