Proceedings of the 3rd Workshop on Learner-Oriented Knowledge Management & KM-Oriented E-Learning Learn@Work: Competency Advancement with Learning Templates Wilco Bonestroo1, Tobias Ley2, Barbara Kump2, Stefanie Lindstaedt2 1 University of Twente, Enschede, The Netherlands w.j.bonestroo@utwente.nl 2 Know-Center, Graz, Austria {tley, bkump, slind}@know-center.at Abstract. The APOSDLE project aims to improve knowledge worker productivity by supporting work-integrated learning. Our Work@Learn approach is based on re-using a wide variety of knowledge artefacts within an organization (such as project reports and meeting notes) for learning. Typically these artefacts have been built without any teaching objectives in mind. Within this contribution we present the way competencies are handled within the first APOSDLE prototype and how competency gaps are automatically identified. We then show how the APOSDLE Learning Tool automatically generates learning events relevant to the competency gap by utilizing organizational knowledge artefacts. Early evaluation results of the prototype are provided and future improvements are discussed. Keywords: Workplace learning, work-integrated learning, competency based learning, electronic leaning environment, Learning Templates. 1 Introduction The challenge of the Learn@Work approach is to compile new learning material, using existing organizational content that was not necessarily created with teaching in mind [17][18]. This approach does not rely on the availability of specifically created (e)Learning content. We aim to tap into all the digital resources of an organizational memory which might encompass project reports, studies, notes, intermediate results, plans, graphics, etc. as well as dedicated learning resources, such as course descriptions, handouts and (e)Learning modules. The challenge we are addressing is: How can we make this confusing mix of information accessible to knowledge workers in a way that they can advance their competencies with it? Within another contribution to this conference we explore the technological aspects which have to be addressed in order to meet this challenge. These include specifically searching for context-relevant resources, automatically splitting up these resources into meaningful pieces and enhancing them with metadata to create rich “learning artefacts”. Within this contribution we will focus on how such learning artefacts can be dynamically assembled into Learning Events (see below for detailed explanation) 9 Proceedings of the 3rd Workshop on Learner-Oriented Knowledge Management & KM-Oriented E-Learning 2 which support a learner in the development, maintenance or advancement of a specific competence. These Learning Events go beyond simply presenting context- relevant resources but in addition provide learning guidance by automatically applying instructional design rules. In the following, we will first present our understanding of competencies and the underlying knowledge space theory which allows for effective competency gap analysis. Based on this understanding we will then show how the idea of Learning Templates [20] has been adapted to bridge the identified competency gaps. The results of an early evaluation of the developed software are provided. We conclude this paper with our ideas on future work on the Learn@Work approach. The work and ideas presented here are the outcome of the APOSDLE project (Advanced Process-Oriented Self-Directed Learning Environment) that offers individual learning support to people working with information and contributing new content to an organisation’s knowledge pool. These “knowledge workers” include engineers, researchers, software developers, consultants, and designers. APOSDLE follows a “Learn@Work” approach, meaning that learning takes place in the user’s immediate work environment and context. It offers integrated support for all three roles a knowledge worker interchangeably fills at the workplace: the role of the worker, the role of the learner, and the role of the expert (for more details please refer to www.aposdle.org). APOSDLE is funded within the European Commission’s 6th Framework Program under the IST work program. It is an Integrated Project jointly coordinated by the Know-Center, Austria’s Competence Centre for Knowledge Management, and Joanneum Research. APOSDLE brings together 12 partners from 7 European Countries. 2 A Function-Based View on Competencies The use of competencies has often been advocated as a way to deal with the challenges in workplace learning [10][19]. Specifically, competencies are being used to more closely relate learning to organizational requirements such as organizational goals or task requirements. Putting personal competencies in the centre of professional education seems necessary as the content of tasks is changing so rapidly that requirements can not be defined in detail. The shift to competencies is therefore not a fashionable hype but a necessity for organizations to cope with uncertainty. Because the concept of competency is of research interest in a huge number of different scientific disciplines (e.g., psychology, educational sciences, economics), the term competency lacks a standardized scientific definition. Nonetheless, in all of these disciplines, competency is interpreted as a roughly specialized system of individual and/or collective abilities, proficiencies, or skills that are necessary or sufficient to reach a specific goal [26]. In the Learn@Work approach, we define competencies as personal characteristics of job holders which they bring to bear in different situations. Competencies are hypothetical constructs which determine performance in a job. The term performance is understood to encompass all behaviours relevant for the accomplishment of a certain task in a specific situation [23]. 10 Proceedings of the 3rd Workshop on Learner-Oriented Knowledge Management & KM-Oriented E-Learning Learn@Work: Competency Advancement with Learning Templates 3 This function-based view on competencies has a number of advantages for work- integrated learning as intended for APOSDLE. First, it allows for deriving a worker’s learning needs by comparing task demands with the competencies the worker has available. Within the APOSDLE prototype, the worker’s competencies are stored in the user profile. That way, competency gap analysis is performed based on individual existing and desired levels of skills and knowledge [15]. A personalization of learning experiences is attained by matching resources that fit individual competency requirements of workers. Hence, the user profile constitutes the rationale for individualised educational interventions and has to be updated according to the learning progress. Ideally, this update happens to a large extent automatically, as the learning environment detects the learner’s use of the system. In the case of work- integrated learning, where learning happens directly in the task context, there exists a potential for updating the user profile according to past task executions (task-based competency assessment) instead of diagnosing competencies in extensive (self-) assessment sessions. Competency Model In order to perform both, task-based competency assessment, and competency gap analysis, a formal model is needed that allows for inferences on what competencies are required for a certain task. Given such a model, conclusions could be drawn from a worker’s task performance on her minimum competency state. Given the competency state of a worker, and the competency requirements of a task at hand, a discrepancy could be identified and educational interventions could be initialised. Ley, Lindstaedt and Albert [16] have suggested Competence based Knowledge Space Theory as a model to formalize competencies and their connection to workplace performance for work-integrated learning. With the Competence based Knowledge Space Theory, Korossy [14] has introduced an extension of Knowledge Space Theory [8]. Knowledge Space Theory has been developed in the 1980s and 90s as an attempt to model a person’s competence as close as possible to observable behaviour. It is predominantly concerned with the diagnosis of knowledge and has been applied in adaptive testing and tutoring scenarios and system [2][11]. The fundamental idea of knowledge space theory is that a person’s knowledge state in a certain domain can be understood as the set of problems this person is able to solve. Since solution dependencies exist among the problems, it is possible to present a person only a subset of all problems of a domain in order to diagnose his/her knowledge state. The collection of all possible knowledge states is called a knowledge space. A knowledge space is a partial order and is stable under union. In an attempt to develop Knowledge Space Theory further, Korossy suggested that in addition to the set of problems, one should look at the set of competencies that is knowledge, skills and abilities needed to solve the problems. This would generate information on the reasons for different levels of performance, and thereby help to suggest learning measures. Similar to the set of questions, competencies are also structured in a competence space which results from a surmise relation on the set of competencies. 11 Proceedings of the 3rd Workshop on Learner-Oriented Knowledge Management & KM-Oriented E-Learning 4 The relationship between the two sets (questions and competencies) is formalized by an interpretation function which maps each problem to a subset of competence states which are elements of the competence space. This subset of competence states contains all those competence states in each of which the problem is solvable. The interpretation function induces a representation function which assigns to each of the competence states all problems which are solvable in that competence state. Which problems are solvable is determined by the interpretation function. The Competence based Knowledge Space Theory has been applied in technology enhanced learning applications. For example, Hockemeyer et al. [12] have assigned “competencies required” and “competencies taught” as metadata to a collection of learning objects. Thereby, prerequisite structures are derived for the eLearning content which allow for adaptive tutoring. New course content could easily be integrated, as metadata was only held locally. The first prototype of the APOSLDE system contains a competency model for the learning domain requirements engineering. 47 tasks in this domain were derived from expert interviews, and 33 competencies were found to be necessary to perform these tasks. The competency model also consisted of a mapping of which competencies are required for which of the tasks. The APOSDLE competency model as well as the method for its construction and validation is given in [15]. Currently, the user profile of an APOSDLE user is filled by selecting each task the worker is able to perform, which defines the performance state. The worker’s competence state is inferred from her performance state. When the worker selects a task from a list, the APOSDLE system performs competency gap analysis by comparing the task requirements (interpretation function) with the worker’s competence state. According to the worker’s competency gap, the APOSDLE system provides her with learning resources that are related with the missing competencies. The selection and initialization of learning resources is handled by the learning tool (see next chapter). 3 Learning Templates to Support Self-Directed Learning Within APOSDLE, the Learning Tool is responsible for managing and supporting the learning process. In this section, we outline the Learning Tool’s conceptual ideas, present an overview of the developed software and we conclude with the results from the early evaluation sessions. Conceptual Ideas The Learning Tool is based on the principles of self-direction in learning, and on the relationship between types of desired learning outcomes and instructional strategies. We provide a short description of these two principles below. According to Knowles, self-directed learning is ‘a process in which individuals take the initiative in designing learning experiences, diagnosing learning needs, locating resources, and evaluate learning’ [13] (p. 18). Accordingly, the self-directed learning process consists of five consecutive steps: the identification of a learning need, the identification of a learning goal, the search for learning material, the 12 Proceedings of the 3rd Workshop on Learner-Oriented Knowledge Management & KM-Oriented E-Learning Learn@Work: Competency Advancement with Learning Templates 5 selection and implementation of a learning strategy, and the evaluation of the learning outcome. This is similar to Stubblefield’s four phases, described by Brockett and Hiemstra: initiating, planning, managing, and evaluating [5]. The Learning Tool aims to support Knowles’ steps in the learning process. The second principle on which the tool is based is the relationship between types of learning goals and instructional strategies. This approach is based on Robert Gagné’s conditions of learning [9]. Classification of learning goals is a commonly used technique in instructional design [1][4][21][24]. Our classification was derived from Anderson and Krathwohl. Following these authors, every competency in APOSDLE was classified as either: remember, understand, apply, evaluate, or create. To search for material that can be used for learning, we classified the available material using an instructional classification with the types: conclusion, definition, example, explanation, guideline, howto, question, and summary. For this classification, we were inspired by the IMAT project [7] in which fragments of learning material were classified and used to support the authoring of training material. In the Learning Tool we borrow from the IMAT approach. The Learning Tool selects a Learning Template based on the desired learning outcome. New learning material is compiled according to the selected Learning Template and the search process is guided by the classification of the material. Learning Templates The essential concept in the APOSDLE Learning Tool is the Learning Template. Learning Templates are typical templates whose empty slots can be filled with material such as text and images. The structure of the templates is based on instructional design principles. Accordingly, an instructional strategy can be implemented in a Learning Template. The templates define both what type of material should be presented and what activities learners should undertake. For example, templates can start by providing an explanation followed by an Header exercise. The Learning Templates only need to be created once. Thirty-four Learning Templates were created to support the Content learning of the five types of competencies. To automatically generate learning material, APOSDLE searches for fragments Engagement that fit the slots of the Learning Activity Templates. We refer to a filled-in Learning Template as a Learning Event. Learning Events are presented to APOSDLE’s users. Figure 1 shows a filled-in Learning Template. Learning Events contain Fig. 1. Engagement Activities that are Learning Event with three sections. 13 Proceedings of the 3rd Workshop on Learner-Oriented Knowledge Management & KM-Oriented E-Learning 6 intended to actively engage the user in the learning material. For example, users are asked to compare examples and to critique provided definitions. Our current design supports the first four steps that were identified in Knowles’ five step model. In APOSDLE, the learning need is identified either by users themselves or (in the future) by the system. After a learning need is identified, a learning goal is selected. In APOSDLE the learning goals are represented as competencies. These competencies describe the desired learning outcomes and are classified according to the classification presented before. The selection of a Learning Template is performed by the APOSDLE Learning Tool. This selection is based on the type of the selected competency. Then, appropriate learning material is identified and the material is used to create Learning Events. The user can select the created Learning Events from a list. The final step, the evaluation of the learning outcome, is not performed in the first prototype. Early Evaluation For the formative evaluation of the APOSDLE prototype we performed several evaluation activities, including expert walkthroughs (with usability and instructional experts), evaluation sessions at the application partners and evaluation sessions with students. In terms of Kirkpatrick’s four-level evaluation model [22], we mainly focused on the first and the second level: the users’ reaction to APOSDLE and, to a lesser degree, their learning results. In the student evaluation sessions twelve students participated in the one-hour sessions. The participants were asked to complete a requirements engineering task, while supported by APOSDLE. The studies showed that participants did learn from the tool and that they were able to complete tasks that they would not be able to complete without the tool. The participants did not use all the sections of the Learning Events alike. For example, only 17 percent of the participants used the Engagement Activities. Those who did use them appreciated them. However, the other participants did not appreciate them. In their review of literature on tool use Clarebout and Elen [6] found that ‘students who receive instructional cues or encouragement to use certain options, use the available tools more compared to students who do not receive these cues or encouragement’ (p. 403). This could explain our findings, because we did not provide instructional cues during our sessions. Currently, APOSDLE presents fragments of documents. The fragments were cut out of the original document and APOSDLE provided no feedback on the location of the original document. The evaluation sessions that were performed at the application partners revealed that the users did not appreciate this approach. Besides, the information provided by the documents was sometimes difficult to link to the competencies to acquire. Obviously, the effectiveness of Learning Templates can only be studied when the provided content is suitable. Future work The next version of APOSDLE will take into account the differences between learners to enhance learning. Smith and Ragan [24] referred to the way people learn 14 Proceedings of the 3rd Workshop on Learner-Oriented Knowledge Management & KM-Oriented E-Learning Learn@Work: Competency Advancement with Learning Templates 7 as cognitive styles, Tennant [25] mentioned learning style and conceptual style. We will develop instructional strategies that consider both the users’ stable characteristics, such as their cognitive styles, and the users’ changing characteristics, such as their level of expertise. Therefore, the Learning Tool uses the information that is stored in the User Competency Profile and the information available in other models, such as the integrated domain and competency model. In the first APOSDLE prototype, the competency model is mapped onto a domain model (ontology) in order to select appropriate learning resources. Both, the mapping and the annotation of learning resources with domain model elements were done manually. In the second prototype, the competency model will be embedded into the domain ontology in order to avoid the mapping between the two. Moreover, a tool will be developed for performing supervised automated document annotation. Additionally, the Learning Tool will provide sequences of learning material. Currently, every Learning Event is self-contained and Learning Events do not include references to other Learning Events. However, some subjects and learning goals are harder to master and cannot be learned in one learning session. In the Learning Tool we want to develop the functionality to construct a sort of plan consisting of a series of Learning Events. For the next versions of the APOSDLE system, the evaluation sessions will gradually shift the focus from Kirkpatrick’s lower evaluation levels, such as reaction, towards the higher levels, such as the learning results and the behavioral changes in the workplace. Acknowledgements APOSDLE is partially funded under the FP6 of the European Commission within the IST work program 2004 (FP6-IST-2004-027023). The Know-Center is funded by the Austrian Competence Center program K plus under the auspices of the Austrian Ministry of Transport, Innovation and Technology (www.ffg.at) and by the State of Styria. References 1. Anderson, L. W., & Krathwohl, D. A. (2001). A taxonomy for learning, teaching, and assessing: A revision of bloom's taxonomy of educational objectives: Allyn & Bacon. 2. ALEKS Corp. (2003). ALEKS - A Better State of Knowledge. Retrieved July 3rd, 2007 under http://www.aleks.com 3. APOSDLE Consortium. (2006). Workplace learning study, deliverable d2.1. 4. Bloom, B. S. (1956). Taxonomy of educational objectives: The classification of educational goals (Vol. Handbook I: Cognitive domain / Benjamin S. Bloom, editor). New York: McKay New York; London: Longman. 5. Brockett, R. G., & Hiemstra, R. (1991). Self-direction in adult learning: Perspectives on theory, research, and practice, Routledge series on theory and practice of adult education in North America. London: Routledge. 15 Proceedings of the 3rd Workshop on Learner-Oriented Knowledge Management & KM-Oriented E-Learning 8 6. Clarebout, G., & Elen, J. (2006). Tool use in computer-based learning environments: Towards a research framework. Computers in Human Behavior, 22, 389. 7. De Hoog, R., Kabel, S., Barnard, Y., Boy, G., DeLuca, P., Desmoulins, C., Riemersma, J., & Verstegen, D. (2002). Re-using technical manuals for instruction: Creating instructional material with the tools of the imat project. In Y. Barnard (Ed.), Its (intelligent tutoring systems) 2002, workshop proceedings integrating technical and training documentation (pp. 28-39). San Sebástian. 8. Doignon, J. & Falmagne, J. (1999). Knowledge Spaces. Heidelberg: Springer. 9. Gagné, R. M. (2004). Principles of instructional design (5th rev. and upd. ed.). Belmont, CA: Thomson/Wadsworth. 10. Green, P. C. (1999). Building Robust Competencies: Linking Human Resource Systems to Organizational Strategies. San Francisco: Jossey-Bass. 11. Hockemeyer, C., Held, T. & Albert, D. (1998). RATH - A Relational Adaptive Tutoring Hypertext WWW-Environment Based on Knowledge Space Theory. Computer Aided Learning and Instruction in Science and Engineering. Proceedings of the Fourth International Conference (CALISCE'98), Göteborg, Sweden. 12. Hockemeyer, C., Conlan, O., Wade, V. & Albert, D. (2003). Applying Competence Prerequisite Structures for eLearning and Skill Management. Journal of Universal Computer Science , 9 (12), 1428-1436. 13. Knowles, M. S. (1975). Self-directed learning: A guide for learners and teachers. Englewood Cliffs, NJ: Prentice Hall. 14. Korossy, K. (1997). Extending the theory of knowledge spaces: A competence- performance approach. Zeitschrift für Psychologie , 205, 53-82. 15. Ley, T.; Kump, B.; Lindstaedt, S. N.; Albert, D.; Maiden, N. A. M.; Jones, S. V. (2006) Competence and Performance in Requirements Engineering: Bringing Learning to the Workplace, in Proceedings of the Joint Workshop on Professional Learning, Competence Development and Knowledge Management, October 2006, 42-52, Crete, Greece. 16. Ley, T., Lindstaedt, S. N. & Albert, D. (2005). Supporting Competency Development in Informal Workplace Learning. In K. Althoff, A. Dengel, R. Bergmann, M. Nick & T. Roth- Berghofer (Hrsg.), Professional Knowledge Management (Vol.LNAI 3782, S. 189-202). Berlin: Springer. 17. Lindstaedt, S. N., Ley, T., Mayer, H. (2007) APOSDLE - New Ways to Work, Learn and Collaborate, in N. Gronau (Ed.) Proccedings of the 4th Conference on Professional Knowledge Management WM2007, 28. - 30. März 2007, Potsdam, Germany, 381-382, GITO-Verlag, Berlin. 18. Lindstaedt, S. N. (2006), APOSDLE: Learning Real-Time and Real-Place, Online Educa 2006, Berlin, 30 November 2006. 19. Lucia, A. D. & Lepsinger, R. (1999). Competency Models: Pinpointing critical success factors in organizations. San Francisco: Jossey-Bass. 20. Merrill, M. D. (1994). Instructional design theory. Englewood Cliffs, NJ: Educational Publications, Inc. 21. Reigeluth, C. C. M. (1983). Instructional-design theories and models: An overview of their current status. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc., Publishers. 22. Reiser, R. A., & Dempsey, J. V. (2002). Trends and issues in instructional design and technology. Upper Saddle River, NJ: Prentice Hall. 23. Schmitt, N. & Chan, D. (1998). Personnel Selection. Thousand Oaks, London, New Delhi: Sage. 24. Smith, P. L., & Ragan, T. J. (1999). Instructional design (2nd ed ed., pp. XV, 399). New York: Wiley. 25. Tennant, M. (2006). Psychology and adult learning. Oxon: Routledge. 26. Weinert, F.E. (1999). Concepts of competence. DeSeCo Expert Report. Neuchatel: Swiss Federal Statistical. 16