=Paper=
{{Paper
|id=Vol-306/paper-2
|storemode=property
|title=Learn@Work: Competency Advancement with Learning Templates
|pdfUrl=https://ceur-ws.org/Vol-306/paper02.pdf
|volume=Vol-306
|authors=Wilco Bonestroo,Tobias Ley,Barbara Kump,Stefanie Lindstaedt
}}
==Learn@Work: Competency Advancement with Learning Templates==
https://ceur-ws.org/Vol-306/paper02.pdf
Proceedings of the 3rd Workshop on Learner-Oriented Knowledge Management & KM-Oriented E-Learning
Learn@Work:
Competency Advancement with Learning Templates
Wilco Bonestroo1, Tobias Ley2, Barbara Kump2, Stefanie Lindstaedt2
1
University of Twente, Enschede, The Netherlands
w.j.bonestroo@utwente.nl
2
Know-Center, Graz, Austria
{tley, bkump, slind}@know-center.at
Abstract. The APOSDLE project aims to improve knowledge worker
productivity by supporting work-integrated learning. Our Work@Learn
approach is based on re-using a wide variety of knowledge artefacts within an
organization (such as project reports and meeting notes) for learning. Typically
these artefacts have been built without any teaching objectives in mind. Within
this contribution we present the way competencies are handled within the first
APOSDLE prototype and how competency gaps are automatically identified.
We then show how the APOSDLE Learning Tool automatically generates
learning events relevant to the competency gap by utilizing organizational
knowledge artefacts. Early evaluation results of the prototype are provided and
future improvements are discussed.
Keywords: Workplace learning, work-integrated learning, competency based
learning, electronic leaning environment, Learning Templates.
1 Introduction
The challenge of the Learn@Work approach is to compile new learning material,
using existing organizational content that was not necessarily created with teaching in
mind [17][18]. This approach does not rely on the availability of specifically created
(e)Learning content. We aim to tap into all the digital resources of an organizational
memory which might encompass project reports, studies, notes, intermediate results,
plans, graphics, etc. as well as dedicated learning resources, such as course
descriptions, handouts and (e)Learning modules. The challenge we are addressing is:
How can we make this confusing mix of information accessible to knowledge
workers in a way that they can advance their competencies with it?
Within another contribution to this conference we explore the technological aspects
which have to be addressed in order to meet this challenge. These include specifically
searching for context-relevant resources, automatically splitting up these resources
into meaningful pieces and enhancing them with metadata to create rich “learning
artefacts”. Within this contribution we will focus on how such learning artefacts can
be dynamically assembled into Learning Events (see below for detailed explanation)
9
Proceedings of the 3rd Workshop on Learner-Oriented Knowledge Management & KM-Oriented E-Learning
2
which support a learner in the development, maintenance or advancement of a
specific competence. These Learning Events go beyond simply presenting context-
relevant resources but in addition provide learning guidance by automatically
applying instructional design rules. In the following, we will first present our
understanding of competencies and the underlying knowledge space theory which
allows for effective competency gap analysis. Based on this understanding we will
then show how the idea of Learning Templates [20] has been adapted to bridge the
identified competency gaps. The results of an early evaluation of the developed
software are provided. We conclude this paper with our ideas on future work on the
Learn@Work approach.
The work and ideas presented here are the outcome of the APOSDLE project
(Advanced Process-Oriented Self-Directed Learning Environment) that offers
individual learning support to people working with information and contributing new
content to an organisation’s knowledge pool. These “knowledge workers” include
engineers, researchers, software developers, consultants, and designers. APOSDLE
follows a “Learn@Work” approach, meaning that learning takes place in the user’s
immediate work environment and context. It offers integrated support for all three
roles a knowledge worker interchangeably fills at the workplace: the role of the
worker, the role of the learner, and the role of the expert (for more details please refer
to www.aposdle.org). APOSDLE is funded within the European Commission’s 6th
Framework Program under the IST work program. It is an Integrated Project jointly
coordinated by the Know-Center, Austria’s Competence Centre for Knowledge
Management, and Joanneum Research. APOSDLE brings together 12 partners from 7
European Countries.
2 A Function-Based View on Competencies
The use of competencies has often been advocated as a way to deal with the
challenges in workplace learning [10][19]. Specifically, competencies are being used
to more closely relate learning to organizational requirements such as organizational
goals or task requirements. Putting personal competencies in the centre of
professional education seems necessary as the content of tasks is changing so rapidly
that requirements can not be defined in detail. The shift to competencies is therefore
not a fashionable hype but a necessity for organizations to cope with uncertainty.
Because the concept of competency is of research interest in a huge number of
different scientific disciplines (e.g., psychology, educational sciences, economics), the
term competency lacks a standardized scientific definition. Nonetheless, in all of these
disciplines, competency is interpreted as a roughly specialized system of individual
and/or collective abilities, proficiencies, or skills that are necessary or sufficient to
reach a specific goal [26].
In the Learn@Work approach, we define competencies as personal characteristics
of job holders which they bring to bear in different situations. Competencies are
hypothetical constructs which determine performance in a job. The term performance
is understood to encompass all behaviours relevant for the accomplishment of a
certain task in a specific situation [23].
10
Proceedings of the 3rd Workshop on Learner-Oriented Knowledge Management & KM-Oriented E-Learning
Learn@Work:
Competency Advancement with Learning Templates 3
This function-based view on competencies has a number of advantages for work-
integrated learning as intended for APOSDLE. First, it allows for deriving a worker’s
learning needs by comparing task demands with the competencies the worker has
available. Within the APOSDLE prototype, the worker’s competencies are stored in
the user profile. That way, competency gap analysis is performed based on individual
existing and desired levels of skills and knowledge [15]. A personalization of learning
experiences is attained by matching resources that fit individual competency
requirements of workers. Hence, the user profile constitutes the rationale for
individualised educational interventions and has to be updated according to the
learning progress. Ideally, this update happens to a large extent automatically, as the
learning environment detects the learner’s use of the system. In the case of work-
integrated learning, where learning happens directly in the task context, there exists a
potential for updating the user profile according to past task executions (task-based
competency assessment) instead of diagnosing competencies in extensive (self-)
assessment sessions.
Competency Model
In order to perform both, task-based competency assessment, and competency gap
analysis, a formal model is needed that allows for inferences on what competencies
are required for a certain task. Given such a model, conclusions could be drawn from
a worker’s task performance on her minimum competency state. Given the
competency state of a worker, and the competency requirements of a task at hand, a
discrepancy could be identified and educational interventions could be initialised.
Ley, Lindstaedt and Albert [16] have suggested Competence based Knowledge
Space Theory as a model to formalize competencies and their connection to
workplace performance for work-integrated learning. With the Competence based
Knowledge Space Theory, Korossy [14] has introduced an extension of Knowledge
Space Theory [8]. Knowledge Space Theory has been developed in the 1980s and 90s
as an attempt to model a person’s competence as close as possible to observable
behaviour. It is predominantly concerned with the diagnosis of knowledge and has
been applied in adaptive testing and tutoring scenarios and system [2][11]. The
fundamental idea of knowledge space theory is that a person’s knowledge state in a
certain domain can be understood as the set of problems this person is able to solve.
Since solution dependencies exist among the problems, it is possible to present a
person only a subset of all problems of a domain in order to diagnose his/her
knowledge state. The collection of all possible knowledge states is called a knowledge
space. A knowledge space is a partial order and is stable under union.
In an attempt to develop Knowledge Space Theory further, Korossy suggested that
in addition to the set of problems, one should look at the set of competencies that is
knowledge, skills and abilities needed to solve the problems. This would generate
information on the reasons for different levels of performance, and thereby help to
suggest learning measures. Similar to the set of questions, competencies are also
structured in a competence space which results from a surmise relation on the set of
competencies.
11
Proceedings of the 3rd Workshop on Learner-Oriented Knowledge Management & KM-Oriented E-Learning
4
The relationship between the two sets (questions and competencies) is formalized by
an interpretation function which maps each problem to a subset of competence states
which are elements of the competence space. This subset of competence states
contains all those competence states in each of which the problem is solvable. The
interpretation function induces a representation function which assigns to each of the
competence states all problems which are solvable in that competence state. Which
problems are solvable is determined by the interpretation function.
The Competence based Knowledge Space Theory has been applied in technology
enhanced learning applications. For example, Hockemeyer et al. [12] have assigned
“competencies required” and “competencies taught” as metadata to a collection of
learning objects. Thereby, prerequisite structures are derived for the eLearning
content which allow for adaptive tutoring. New course content could easily be
integrated, as metadata was only held locally.
The first prototype of the APOSLDE system contains a competency model for the
learning domain requirements engineering. 47 tasks in this domain were derived from
expert interviews, and 33 competencies were found to be necessary to perform these
tasks. The competency model also consisted of a mapping of which competencies are
required for which of the tasks. The APOSDLE competency model as well as the
method for its construction and validation is given in [15].
Currently, the user profile of an APOSDLE user is filled by selecting each task the
worker is able to perform, which defines the performance state. The worker’s
competence state is inferred from her performance state. When the worker selects a
task from a list, the APOSDLE system performs competency gap analysis by
comparing the task requirements (interpretation function) with the worker’s
competence state. According to the worker’s competency gap, the APOSDLE system
provides her with learning resources that are related with the missing competencies.
The selection and initialization of learning resources is handled by the learning tool
(see next chapter).
3 Learning Templates to Support Self-Directed Learning
Within APOSDLE, the Learning Tool is responsible for managing and supporting the
learning process. In this section, we outline the Learning Tool’s conceptual ideas,
present an overview of the developed software and we conclude with the results from
the early evaluation sessions.
Conceptual Ideas
The Learning Tool is based on the principles of self-direction in learning, and on the
relationship between types of desired learning outcomes and instructional strategies.
We provide a short description of these two principles below.
According to Knowles, self-directed learning is ‘a process in which individuals
take the initiative in designing learning experiences, diagnosing learning needs,
locating resources, and evaluate learning’ [13] (p. 18). Accordingly, the self-directed
learning process consists of five consecutive steps: the identification of a learning
need, the identification of a learning goal, the search for learning material, the
12
Proceedings of the 3rd Workshop on Learner-Oriented Knowledge Management & KM-Oriented E-Learning
Learn@Work:
Competency Advancement with Learning Templates 5
selection and implementation of a learning strategy, and the evaluation of the learning
outcome. This is similar to Stubblefield’s four phases, described by Brockett and
Hiemstra: initiating, planning, managing, and evaluating [5]. The Learning Tool aims
to support Knowles’ steps in the learning process.
The second principle on which the tool is based is the relationship between types of
learning goals and instructional strategies. This approach is based on Robert Gagné’s
conditions of learning [9]. Classification of learning goals is a commonly used
technique in instructional design [1][4][21][24]. Our classification was derived from
Anderson and Krathwohl. Following these authors, every competency in APOSDLE
was classified as either: remember, understand, apply, evaluate, or create. To search
for material that can be used for learning, we classified the available material using an
instructional classification with the types: conclusion, definition, example,
explanation, guideline, howto, question, and summary. For this classification, we
were inspired by the IMAT project [7] in which fragments of learning material were
classified and used to support the authoring of training material. In the Learning Tool
we borrow from the IMAT approach. The Learning Tool selects a Learning Template
based on the desired learning outcome. New learning material is compiled according
to the selected Learning Template and the search process is guided by the
classification of the material.
Learning Templates
The essential concept in the APOSDLE Learning Tool is the Learning Template.
Learning Templates are typical templates whose empty slots can be filled with
material such as text and images. The structure of the templates is based on
instructional design principles. Accordingly, an instructional strategy can be
implemented in a Learning Template. The templates define both what type of material
should be presented and what activities learners should undertake. For example,
templates can start by providing an
explanation followed by an
Header
exercise. The Learning Templates
only need to be created once.
Thirty-four Learning Templates
were created to support the Content
learning of the five types of
competencies. To automatically
generate learning material,
APOSDLE searches for fragments Engagement
that fit the slots of the Learning Activity
Templates. We refer to a filled-in
Learning Template as a Learning
Event. Learning Events are
presented to APOSDLE’s users.
Figure 1 shows a filled-in Learning
Template. Learning Events contain Fig. 1.
Engagement Activities that are Learning Event with three sections.
13
Proceedings of the 3rd Workshop on Learner-Oriented Knowledge Management & KM-Oriented E-Learning
6
intended to actively engage the user in the learning material. For example, users are
asked to compare examples and to critique provided definitions.
Our current design supports the first four steps that were identified in Knowles’
five step model. In APOSDLE, the learning need is identified either by users
themselves or (in the future) by the system. After a learning need is identified, a
learning goal is selected. In APOSDLE the learning goals are represented as
competencies. These competencies describe the desired learning outcomes and are
classified according to the classification presented before. The selection of a Learning
Template is performed by the APOSDLE Learning Tool. This selection is based on
the type of the selected competency. Then, appropriate learning material is identified
and the material is used to create Learning Events. The user can select the created
Learning Events from a list. The final step, the evaluation of the learning outcome, is
not performed in the first prototype.
Early Evaluation
For the formative evaluation of the APOSDLE prototype we performed several
evaluation activities, including expert walkthroughs (with usability and instructional
experts), evaluation sessions at the application partners and evaluation sessions with
students. In terms of Kirkpatrick’s four-level evaluation model [22], we mainly
focused on the first and the second level: the users’ reaction to APOSDLE and, to a
lesser degree, their learning results. In the student evaluation sessions twelve students
participated in the one-hour sessions. The participants were asked to complete a
requirements engineering task, while supported by APOSDLE. The studies showed
that participants did learn from the tool and that they were able to complete tasks that
they would not be able to complete without the tool. The participants did not use all
the sections of the Learning Events alike. For example, only 17 percent of the
participants used the Engagement Activities. Those who did use them appreciated
them. However, the other participants did not appreciate them. In their review of
literature on tool use Clarebout and Elen [6] found that ‘students who receive
instructional cues or encouragement to use certain options, use the available tools
more compared to students who do not receive these cues or encouragement’ (p. 403).
This could explain our findings, because we did not provide instructional cues during
our sessions.
Currently, APOSDLE presents fragments of documents. The fragments were cut
out of the original document and APOSDLE provided no feedback on the location of
the original document. The evaluation sessions that were performed at the application
partners revealed that the users did not appreciate this approach. Besides, the
information provided by the documents was sometimes difficult to link to the
competencies to acquire. Obviously, the effectiveness of Learning Templates can only
be studied when the provided content is suitable.
Future work
The next version of APOSDLE will take into account the differences between
learners to enhance learning. Smith and Ragan [24] referred to the way people learn
14
Proceedings of the 3rd Workshop on Learner-Oriented Knowledge Management & KM-Oriented E-Learning
Learn@Work:
Competency Advancement with Learning Templates 7
as cognitive styles, Tennant [25] mentioned learning style and conceptual style. We
will develop instructional strategies that consider both the users’ stable characteristics,
such as their cognitive styles, and the users’ changing characteristics, such as their
level of expertise. Therefore, the Learning Tool uses the information that is stored in
the User Competency Profile and the information available in other models, such as
the integrated domain and competency model.
In the first APOSDLE prototype, the competency model is mapped onto a domain
model (ontology) in order to select appropriate learning resources. Both, the mapping
and the annotation of learning resources with domain model elements were done
manually. In the second prototype, the competency model will be embedded into the
domain ontology in order to avoid the mapping between the two. Moreover, a tool
will be developed for performing supervised automated document annotation.
Additionally, the Learning Tool will provide sequences of learning material.
Currently, every Learning Event is self-contained and Learning Events do not include
references to other Learning Events. However, some subjects and learning goals are
harder to master and cannot be learned in one learning session. In the Learning Tool
we want to develop the functionality to construct a sort of plan consisting of a series
of Learning Events.
For the next versions of the APOSDLE system, the evaluation sessions will
gradually shift the focus from Kirkpatrick’s lower evaluation levels, such as reaction,
towards the higher levels, such as the learning results and the behavioral changes in
the workplace.
Acknowledgements
APOSDLE is partially funded under the FP6 of the European Commission within the
IST work program 2004 (FP6-IST-2004-027023). The Know-Center is funded by the
Austrian Competence Center program K plus under the auspices of the Austrian
Ministry of Transport, Innovation and Technology (www.ffg.at) and by the State of
Styria.
References
1. Anderson, L. W., & Krathwohl, D. A. (2001). A taxonomy for learning, teaching, and
assessing: A revision of bloom's taxonomy of educational objectives: Allyn & Bacon.
2. ALEKS Corp. (2003). ALEKS - A Better State of Knowledge. Retrieved July 3rd, 2007
under http://www.aleks.com
3. APOSDLE Consortium. (2006). Workplace learning study, deliverable d2.1.
4. Bloom, B. S. (1956). Taxonomy of educational objectives: The classification of educational
goals (Vol. Handbook I: Cognitive domain / Benjamin S. Bloom, editor). New York:
McKay New York; London: Longman.
5. Brockett, R. G., & Hiemstra, R. (1991). Self-direction in adult learning: Perspectives on
theory, research, and practice, Routledge series on theory and practice of adult education in
North America. London: Routledge.
15
Proceedings of the 3rd Workshop on Learner-Oriented Knowledge Management & KM-Oriented E-Learning
8
6. Clarebout, G., & Elen, J. (2006). Tool use in computer-based learning environments:
Towards a research framework. Computers in Human Behavior, 22, 389.
7. De Hoog, R., Kabel, S., Barnard, Y., Boy, G., DeLuca, P., Desmoulins, C., Riemersma, J.,
& Verstegen, D. (2002). Re-using technical manuals for instruction: Creating instructional
material with the tools of the imat project. In Y. Barnard (Ed.), Its (intelligent tutoring
systems) 2002, workshop proceedings integrating technical and training documentation (pp.
28-39). San Sebástian.
8. Doignon, J. & Falmagne, J. (1999). Knowledge Spaces. Heidelberg: Springer.
9. Gagné, R. M. (2004). Principles of instructional design (5th rev. and upd. ed.). Belmont,
CA: Thomson/Wadsworth.
10. Green, P. C. (1999). Building Robust Competencies: Linking Human Resource Systems to
Organizational Strategies. San Francisco: Jossey-Bass.
11. Hockemeyer, C., Held, T. & Albert, D. (1998). RATH - A Relational Adaptive Tutoring
Hypertext WWW-Environment Based on Knowledge Space Theory. Computer Aided
Learning and Instruction in Science and Engineering. Proceedings of the Fourth
International Conference (CALISCE'98), Göteborg, Sweden.
12. Hockemeyer, C., Conlan, O., Wade, V. & Albert, D. (2003). Applying Competence
Prerequisite Structures for eLearning and Skill Management. Journal of Universal
Computer Science , 9 (12), 1428-1436.
13. Knowles, M. S. (1975). Self-directed learning: A guide for learners and teachers.
Englewood Cliffs, NJ: Prentice Hall.
14. Korossy, K. (1997). Extending the theory of knowledge spaces: A competence-
performance approach. Zeitschrift für Psychologie , 205, 53-82.
15. Ley, T.; Kump, B.; Lindstaedt, S. N.; Albert, D.; Maiden, N. A. M.; Jones, S. V. (2006)
Competence and Performance in Requirements Engineering: Bringing Learning to the
Workplace, in Proceedings of the Joint Workshop on Professional Learning, Competence
Development and Knowledge Management, October 2006, 42-52, Crete, Greece.
16. Ley, T., Lindstaedt, S. N. & Albert, D. (2005). Supporting Competency Development in
Informal Workplace Learning. In K. Althoff, A. Dengel, R. Bergmann, M. Nick & T. Roth-
Berghofer (Hrsg.), Professional Knowledge Management (Vol.LNAI 3782, S. 189-202).
Berlin: Springer.
17. Lindstaedt, S. N., Ley, T., Mayer, H. (2007) APOSDLE - New Ways to Work, Learn and
Collaborate, in N. Gronau (Ed.) Proccedings of the 4th Conference on Professional
Knowledge Management WM2007, 28. - 30. März 2007, Potsdam, Germany, 381-382,
GITO-Verlag, Berlin.
18. Lindstaedt, S. N. (2006), APOSDLE: Learning Real-Time and Real-Place, Online Educa
2006, Berlin, 30 November 2006.
19. Lucia, A. D. & Lepsinger, R. (1999). Competency Models: Pinpointing critical success
factors in organizations. San Francisco: Jossey-Bass.
20. Merrill, M. D. (1994). Instructional design theory. Englewood Cliffs, NJ: Educational
Publications, Inc.
21. Reigeluth, C. C. M. (1983). Instructional-design theories and models: An overview of their
current status. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc., Publishers.
22. Reiser, R. A., & Dempsey, J. V. (2002). Trends and issues in instructional design and
technology. Upper Saddle River, NJ: Prentice Hall.
23. Schmitt, N. & Chan, D. (1998). Personnel Selection. Thousand Oaks, London, New Delhi:
Sage.
24. Smith, P. L., & Ragan, T. J. (1999). Instructional design (2nd ed ed., pp. XV, 399). New
York: Wiley.
25. Tennant, M. (2006). Psychology and adult learning. Oxon: Routledge.
26. Weinert, F.E. (1999). Concepts of competence. DeSeCo Expert Report. Neuchatel: Swiss
Federal Statistical.
16