Creating awareness of kinaesthetic learning using the Experience API: current practices, emerging challenges, possible solutions Maurizio Megliola1, Gianluigi Di Vito1, Roberto Sanguini2, Fridolin Wild3, Paul Lefrere3 1 Piksel Ltd., Italy {gianluigi.divito,maurizio.megliola}@piksel.com 2 AgustaWestland, Italy Roberto.Sanguini@agustawestland.com 3 The Open University, Knowledge Media Institute, Milton Keynes, UK {fridolin.wild,paul.lefrere}@open.ac.uk Abstract. We describe our use of the Experience API in preparing blue-collar workers for three frequently arising work contexts, including, for example, the requirement to perform maintenance tasks exactly as specified, consistently, quickly, and without error. We provide some theoretical underpinning for modi- fying and updating the API to remain useful in near-future training scenarios, such as having a shorter time allowed for kinaesthetic learning experiences than in traditional apprenticeships or training. We propose ways to involve a wide range of stakeholders in appraising the API and ensuring that any enhancements to it, or add-ons, are useful, feasible and compatible with current TEL practices and tools, such as learning-design modelling languages. Keywords: Experience sharing, xAPI, verbs. 1 Introduction Apprenticeship today often includes the development of ‘kinaesthetic intelligence’, i.e. the tactile (physical) abilities associated with using the body to create (or ‘do’) something involving highly coordinated and efficient body movements. Prototypical examples of this can be found in the fluid and precise motions of skilled dancers, surgeons, and skilled blue-collar workers. Gardner (2011) links high bodily- kinaesthetic intelligence to “control of one's bodily motions, the capacity to handle objects skillfully, a sense of timing, a clear sense of the goal of a physical action, along with the ability to train responses”. Becoming a skilled kinaesthetic performer through traditional apprenticeship is to- day largely conceptualised to take years, but this golden age of long-cycle training is quickly disappearing. Professional training environments such as in manufacturing face the challenge that the world of repetition that enabled long cycles of training to be cost-justified is increasingly taken over by short-run or personalised production, 11 Creating awareness of kinaesthetic learning using the Experience API - ARTEL14 using advanced factory machinery that does not require physical dexterity (including teachable robots that can emulate dextrous production-line workers), thereby not only causing shifts in the demand for skills, but at the same time downsizing the economi- cally acceptable time-to-competence. Achievements in interoperability for technology-enhanced learning (TEL) over the last decade are making it viable to develop TEL-based alternatives to a traditional apprenticeship. For example, TEL is making it possible to author, exchange, and then orchestrate the re-enactment of learning activities across distributed tools, using learn- ing process and learning design modelling languages (Laurillard & Ljubojevic, 2011; Fuente Valentín, Pardo, & Degado Kloos, 2011; Mueller, Zimmermann, & Peters, 2010; Koper & Tattersall, 2005; Wild, Moedritscher, & Sigurdarson, 2008). What these modelling languages fall short of, however, is the ability to handle hy- brid (human-machine) experiences, to teach machines, or to train people at a distance – as required, for example, in workplaces that include robots or software agents, or in workplaces that include distributed participation across companies in a supply chain. In particular, when it comes to bridging between the virtual and the real world, be- tween digital and physical experiences, present means are ill equipped for supporting the capturing, codification, and sharing of hybrid learning experiences. Moreover, TEL-focused ways of capturing performance and collecting good prac- tice are as of today still resource-intensive, placing a barrier to the spread of innova- tion and hindering resilience of business. That barrier can be somewhat reduced through the use of the present version of the Experience API (ADL, 2013), and we claim could be further reduced through possi- ble extensions and complements to the API, the need for which has become apparent in the TELLME project. The Experience API is is a novel interface specification designed to link sensor networks together to enable the real-time collection and analysis of learning experi- ences conducted in different contexts, with the aim of providing better interoperability between the different types of participating educational systems and devices. Being precondition to mining and modelling, it forms a centrepiece in the canon of next generation techniques and technologies for capturing, codification, and sharing of hybrid learning experiences. This paper presents an overview of how the components and concepts of the Expe- rience API are integrated within the TELL-ME project to foster tracking and analysis of learning and training in three different manufacturing environments, namely Aero- nautics, Furniture, and Textiles. We describe within this contribution, how we deploy an open source Learning Re- cording Store to allow for collecting and reporting learning across the various compo- nents of the TELL-ME system. Moreover, starting with the Aeronautics industry, we define a taxonomy of verbs of handling and motion for capturing in particular kinaes- thetic learning experiences as required in the helicopter industry, when targeting learning about a defined helicopter model and the connected standard maintenance procedures. 12 Creating awareness of kinaesthetic learning using the Experience API - ARTEL14 2 TELLME Learning Locker The ADL Experience API (short ‘xAPI’)1, formerly known as TinCan API, is an extension of the Activity Streams2 specification, a format for capturing activity on social networks, created by companies like Google, Facebook, Microsoft, IBM etc., that allows for statements of experience to be delivered to and stored securely in a Learning Record Store (LRS). These statements of experience are typically learning experiences, but the API can address statements of any kind of experiences a person immerses in, both on- and offline. While the core objects of an xAPI statement (Actor, Verb and Object) derive from the core Activity Streams specification, the Experience API has many more defined constructs for tracking information pertinent for the learner (with captured results such as “score”, “success”, “completion”, “attempt”, and “response”), unlike the Ac- tivity Streams spec which focuses on the publisher. In its most basic application, the Experience API allows one system (the activity ‘provider’) to send a message (also known as the ‘statement’) to another system (aka the ‘Learning Record Store’) about something a user has done. Up until now, this process has mostly taken place inside the organisation’s Learning Management Sys- tem. Anything we wanted to track had to be built as part of the LMS functionality, or needed tailor-made integration. The Experience API now allows sending and receiv- ing data between systems about what someone has done in a more openly defined way (see Figure 1). Figure 1. The experience tracking before and after the advent of the xAPI. This is important, because via the Experience API, any system can send xAPI statements using a standard connection method. This process of sending xAPI state- ments can happen in systems behind the firewall or openly across the Internet using secure connections. 1 http://www.adlnet.gov/tla/experience-api/ 2 http://activitystrea.ms/ 13 Creating awareness of kinaesthetic learning using the Experience API - ARTEL14 Within TELL-ME, we have implemented an instance of the open-source Learning Locker3 Learning Record Store, a LAMP-based project using MongoDB, PHP, and AngularJS, to begin collecting learning activity statements generated by xAPI com- pliant learning activities and reporting on such data. The REST WS interface to the LRS was made available to be used by any compo- nent of the TELL-ME architecture to submit and retrieve xAPI statements. An exam- ple of such xAPI triple submission (with POST), using curl4, is shown below: curl -X POST --data @example.json -H "Content-Type: application/json" -- user 465ea716cebb2476fa0d8eca90c3d4f594e64b51:ccbdb91f75d61b726800313b2aa9f50 f562bad66 -H "x-experience-api-version: 1.0.0" http://demos.polymedia.it/tellme/learninglocker/data/xAPI/statements The example above is parameterized by a reference to a JSON file named exam- ple.json: it contains the actual xAPI statement in form of an actor-verb-object triple: { "actor":{ "objectType": "Agent", "name": "Gianluigi Di Vito", "mbox":"mailto:gianluigi.divito@piksel.com" }, "verb":{ "id":"http://activitystrea.ms/schema/1.0/watch", "display":{ "en-US":"Watched" } }, "object":{ "id":"http://tellme-ip.eu/media/video/152" } } Once statements are logged, they can be queried again, for example, as needed for constraint validation to check, whether a user actually performed a certain required action step – then requiring additional constraint checker components. 3 http://learninglocker.net/ 4 curl - command line tool for transferring data with URL syntax: http://curl.haxx.se/ 14 Creating awareness of kinaesthetic learning using the Experience API - ARTEL14 Figure 1. Learner interaction during maintenance of a helicopter tail rotor: ‘user cleaned corrosion inhibitors’. Same as any LRS implementation, the LearningLocker integrates data from multi- ple systems. The ‘Activity Provider’ can be any sort of system, where the user per- forms learning activity. Examples of such systems include search engines, social bookmark engines, proxy servers, social networking platforms, webinars, blogs, CRMs, as well as specific additional TELL-ME components. An example of the latter is the TELL-ME smart player application under development, which can drop state- ments about a user consuming a video (to the end or just the access of it). Another example is ARgh!, the platform for the delivery of context-based Augmented Reality content on mobile devices at the workplaces, which can drop statements about certain action steps being executed by the learner: Figure 1 shows an example of such user interaction with physical objects during training. Integrating the Experience API requires not only providing an LRS, but also defin- ing the set of ‘predicates’ for logging the experience statements in a meaningful way. The ADL features a standard list of commonly occurring verbs5. This list, however, is not bespoke to manufacturing and is restricted too much to generic, web-based learn- ing activity, excluding interaction with objects of the real world – as required for learning by experience in manufacturing. The same is the case for ADL’s extended list of “Proposed Verbs for the Experience API”6. Once data are mapped to the ele- ments of the Experience API statement (see Section 3), they can be captured in the learning record store. Figure 3 shows examples of such statements stored in the Learning Locker coming from the ARgh! trials. 5 http://adlnet.gov/expapi/verbs/ 6 http://bit.ly/1zGmAzn 15 Creating awareness of kinaesthetic learning using the Experience API - ARTEL14 Figure 2. Argh! statements stored in the Learning Locker The Learning Locker also allows creating personalized queries by means of its re- porting functions, combining data from both learning activities and workplace per- formance, allowing linking of learning activity to performance. One important aspect to be considered is the handling of security and the protec- tion of the privacy of learners. In the Experience API, authentication is tied to the user, not the content. The user can be any person or thing that is asserting the state- ment. The user can be a learner, an instructor, or even a software agent and it can authenticate with OAuth7, a commonly used access delegation mechanism employed by many big names such as Google, Facebook, Salesforce, etc. that eliminates the needs of sharing passwords between applications to exchange data. In this aim, the Learning Locker supports authentication and it is integrated with OAuth 2.0, exposing an API which allows 3rd parties to connect to the API via OAuth 2.0. 7 http://oauth.net/ 16 Creating awareness of kinaesthetic learning using the Experience API - ARTEL14 Figure 3. Learning Locker Reporting page 3 The statement vocabulary (Aeronautics industry) Each xAPI statement follows the syntax of providing a subject, predicate, and ob- ject. While subjects and objects can vary, predicates (the ‘verbs’) are ideally rather slowly changing and can be defined in advance. From a cognitive linguistic perspective, there is prior work on defining verbs of motion and handling – and clarifying their relation to the way humans’ cognitively process them. Roy (2005) explains how humans learn words by grounding them in perception and action. The presented theories are tested computationally by imple- menting them into a series of conversational robots, the latest of which - Ripley - can explain "aspects of context-dependent shifts of word meaning" that other theories fall short of. To construct a vocabulary of verbs that is widely understood, easy to learn, and natural in mapping, the work of Roy provides valuable insights: Roy postulates 17 Creating awareness of kinaesthetic learning using the Experience API - ARTEL14 that "verbs that refer to physical actions are naturally grounded in representations that encode the temporal flow of events" (p. 391). She further details that the grounding of action verbs follows the schema of specifying which force dynamics (out of a limited set) apply and which temporal Allen relations operate (p. 391). Any higher-level composition of action verbs, so Roy (p.392), can be traced back to and expressed in terms of these fundamental temporal and force relations. This provides an angle for defining and structuring the TELL-ME taxonomy of verbs of handling and motion: it provides a basis for ordering from fundamental to composite actions, also defining their similarities. Moreover, it offers insights on how to map these verbs back to perception: it provides a rationale for how many visual overlay elements are required for an augmented reality instruction to express a certain motion verb primitive. For example, a verb 'pick up' requires the overlay visualisation of a grabbing hand as well as a highlight of the object to be picked up, whereas the motion verb 'move' consists of both 'pick up' and 'put down' actions, thus requiring more visual elements to be specified. Palmer et al. (2005) further discuss the "criteria used to define the sets of semantic roles" for building verb classes. It provides insights into the argument structure of framesets (and so-called role-sets) of verb classes (Palmer et al., 2005, section 3). Chatterjee (2001) reviews the cognitive relationship between language and space. He refers to Jackendoff in postulating that the "conceptual structure of verbs decom- poses into primitives such as ‘movement’, ‘path’ and ‘location’ (p.57). From an augmented reality perspective, there is additional prior work of relevance. Robertson and MacIntyre (2009) provide a review of the state of the art in displaying communicative intent in an AR-based system. The proposed taxonomy, however, stays on a level of general applicability across all sorts of augmented reality applica- tions and is lacking the level of handling and motion required for a particular work- place, such as required in learning the maintenance of helicopters. The proposed cate- gories (called ‘style’ strategies) are ‘include’, ‘visible’, ‘find’, ‘label’, ‘recognizable’, ‘focus’, ‘subdue’, ‘visual property’, ‘ghost’, and ‘highlight’ (p.149f). The communi- cative goals signified by these styling operations for visual overlays are listed as ‘show’, ‘property’, ‘state’, ‘location’, ‘reference’, ‘change’, ‘relative-location’, ‘iden- tify’, ‘action’, ‘move’, and ‘enhancement’ (p.148). Other than the work in cognitive linguistics, here, the focus is clearly defined from a technical and not user angle: help- ing the user to identify or move an object is on the same level as setting labels. In the aeronautical field, maintenance operations must be carried out according to official documents issued by the design authority of the aircraft. Such document is called the Maintenance Publication. Usually, it is organised inside the Interactive Electronic Technical Publication (IETP). The AECMA S1000D (European Associa- tion of Aerospace Constructors S1000D) is an international standard for development of IETP, utilizing a Common Source Data Base (CSDB). The standard prescribes rules to name, define, and code everything that is necessary to carry out maintenance activities in terms of: • aircraft model; • systems and subsystems of the aircraft; • maintenance tasks; 18 Creating awareness of kinaesthetic learning using the Experience API - ARTEL14 • location of components; • tools; • additional documentations; • miscellaneous. Every maintenance task is identified by a unique code referring to the Task Cate- gory (e.g. ‘servicing’, ‘repairs’, ‘package’), the Maintenance Activity within the cate- gory (‘drain’, ‘fill’, ‘remove’, ‘clean’, etc.), and - finally - to the definition which gives the procedure and data necessary to carry out maintenance tasks on a specific system or component. The code allows retrieving both procedures and data necessary to e.g. fill containers with fuel, oil, oxygen, nitrogen, air, water, or other fluids. For the TELL-ME taxonomy of verbs of handling and motion the following pre- liminary list was compiled by pilot partners, see Table 1. The initial taxonomy pre- sents the verbs of handling and motion as required in the helicopter industry for a defined helicopter model and the connected standard maintenance procedures. All the actions are handled by the actor ‘certified staff’, which therefore has not been includ- ed in the table. Activity Verb Objects Operation Loaded Cargo Unloaded Cargo Servicing Filled Liquid or gas (fuel, oil, oxygen, nitrogen, air, water). Drained Liquid (fuel, oil, water). Released (pressure) Gas (oxygen, nitrogen). Lubricated System, equipment, component, or item. Cleaned Surface Applied (protection) Surface Removed (ice) Surface Adjusted System, equipment or component. Aligned System, equipment or component. Calibrated System, equipment or component. Inspected (keep Product, system, equipment or compo- serviceable) nent. Changed Liquid (fuel, oil, water). Gas (oxygen, nitrogen). Examination, tests Examined (visual) Product, system, equipment, component and checks equipment, component, or item. Tested (operation / System, equipment or component. function) Tested (function) System, equipment or component. Tested (structure) Structure Designed (data / System, equipment, or component. 19 Creating awareness of kinaesthetic learning using the Experience API - ARTEL14 tolerance check) Monitored (condition) Product, system, equipment or compo- nent. Disconnect, re- Disconnected Equipment, components, or items. move and disas- Removed Equipment, components, or items. semble proce- dures Disassembled Equipment, components, or items. Opened for access Panels or doors (engine bay doors, land- ing gear doors, etc.). Unloaded (download Items. software) Repairs and locally Added material Product, equipment, component or item. make procedures Attached material Product, equipment, component or item. and data Changed (mechanical Structure, surface. strength / structure of material / surface finish of material) Removed Material Repaired Damaged product, system, equipment or component. Assemble, install Assembled Equipment, components and items. and connect pro- Installed Equipment, components and items. cedures Connected Equipment, components and items. Closed after access Panels or doors (engine bay doors, landing gear doors, etc.). Loaded (upload Items. software) Package, handling, Removed from Products, systems equipment or compo- storage and trans- (preservation materi- nents, portation al) Put (in container) Item. Removed (from Item. container) Kept serviceable Products, systems, equipment, or com- (when in storage) ponents. Moved (when in stor- Products, systems, equipment, or com- age) ponents. Prepared for use Item (after storage) Table 1. Activities and their predicates (plus objects). 20 Creating awareness of kinaesthetic learning using the Experience API - ARTEL14 4 Conclusion and outlook In this contribution, we have presented an overview of how the concepts of Experi- ence API can be applied in a manufacturing environment, precisely in the Aeronautics industry. In the context of the TELL-ME project, three different taxonomies of verbs of handling and motion were prepared or are currently being prepared by the pilot partners, one of which has been described in this paper (from the helicopter industry). An instance on the open-source Learning Locker Learning Record Store was made available to any TELL-ME component, allowing them storing and retrieving xAPI statements about the workers activities. For the future development, we intend to add preliminary taxonomies for the other two industry sectors, further mature them, and address issues related to the analytics and statements interpretation, going further ahead in understanding data and how to find meaning from it (with the help of statistical analysis). In addition, we plan to explore how to extend (keyword: ‘constraint checker’) or complement the Experience API to handle emerging challenges such as the need to consider the ergonomics and learning needs of human-robot workspaces. This will involve conversations and col- laborations with multiple stakeholders, such as TEL vendors and users, standards bodies, and participants in robotics projects and projects involving or affected by automation – as envisioned for the Factory of the Future projects. Acknowledgements The research leading to the results presented in this contribution has received fund- ing from the European Community’s Seventh Framework Programme (FP7/2007- 2013) under grant agreement no 318329 (the TELL-ME project). The authors would like to express their gratitude to the partners who have been involved in the related research work in TELL-ME. References 1. Gardner, H. (2011): Frames Of Mind: The Theory of Multiple Intelligences, Basic Books, first published in 1983. 2. Laurillard, D.; Ljubojevic, D. (2011): Evaluating learning designs through the formal rep- resentation of pedagogical patterns, In: Kohls & Wedekind (Eds.): Investigations of E- learning Patterns: Context Factors, Problems, and Solutions, IGI global. 3. Fuente Valentín, L. de la; Pardo, A.; Delgado Kloos, C. (2011): Generic service integration in adaptive learning experiences using IMS learning design, In: Computers & Education, 57:1160-1170. 4. Mueller, D.; Zimmermann, V.; Peters, J. (2010): ISURE: Guideline for the integration of instructional models with open educational content in IMS Learning Design, deliverable d5.2 of the iCoper consortium. 5. Koper, R.; Tattersall, C. (2005): Preface, In: Koper, Tattersall (Eds.): Learning Design: A handbook on modelling and delivering networked education and training, Springer: Berlin. 21 Creating awareness of kinaesthetic learning using the Experience API - ARTEL14 6. Wild, F.; Mödritscher, F.; Sigurdarson, S. (2008): Designing for Change: Mash-Up Per- sonal Learning Environments, In: eLearning Papers, 9 (2008). 7. Dillenbourg, P. (2011): Design for Orchestration, In: Dillenbourg (Ed.): Trends in Orches- tration: Second Research & Technology Scouting Report, deliverable d1.5 of the STELLAR consortium. 8. ADL (2013): Experience API, version 1.0.1, Advanced Distributed Learning (ADL) Initia- tive, U.S. Department of Defense. 9. Roy, D. (2005): Grounding words in perception and action: computational insights, In: TRENDS in Cognitive Sciences, 9 (8):389-396. 10. Palmer, M.; Gildea, D.; Kingsbury, P. (2004): The Proposition Bank: An Annotated Cor- pus of Semantic Roles, In: Computational Linguistics, 31(1):71-106. 11. Chatterjee, A. (2001): Language and space: some interactions, In: TRENDS in Cognitive Sciences , 5(2):55-61. 12. Robertson, C.; MacIntyre, B. (2004): Adapting to Registration Error in an Intent-Based Augmentation System, In: Ong, Nee (Eds): Virtual and Augmented Reality Applications in Manufacturing, Springer: London. 22