=Paper=
{{Paper
|id=Vol-34/paper-19
|storemode=property
|title=A Framework to Create Performance Indicators in Knowledge Management
|pdfUrl=https://ceur-ws.org/Vol-34/roy_et_al.pdf
|volume=Vol-34
|dblpUrl=https://dblp.org/rec/conf/pakm/RoyRWS00
}}
==A Framework to Create Performance Indicators in Knowledge Management==
https://ceur-ws.org/Vol-34/roy_et_al.pdf
A Framework To Create Performance Indicators In Knowledge
Management
Rajkumar Roy Francisco M. del Rey Bert van Wegen Andy Steele
Cranfield University, Chamorro Unilever Research Unilever Research
UK Cranfield University,
r.roy@cranfield.ac.uk UK
fmdelrey@hotmail.com
to improve the performance of the company and reach the
business objectives. However managers are striving to
uncover which specific business contributions are down to
Abstract KM solutions. By knowing the real contribution of those
knowledge processes managers obtain on one hand
Knowledge management (KM) is emerging as one information about the achievement of the business
of the most powerful management tools in today’s objectives and on the other hand to see a clear relationship
manufacturing. It looks at the company resources between the KM solutions and the business improvements.
in order to gain competitive advantage. The
management of these resources can mean the Managers use performance measurements to monitor key
difference between success and failure in a issues in businesses. These measures provide the most
competitive environment. This encourages relevant information of the company, showing managers
companies to look for better ways in the how the business is performing. The solution proposed in
management of these intangible assets, developing this paper provides a mechanism for monitoring KM
KM projects in order to provide KM solutions to solutions in those issues related to Business process. This
solve knowledge bottlenecks through Knowledge new methodology is focused on the business objectives to
processes. However, if KM solutions are create performance indicators (PI’s) KM solutions. The
considered as an important part in today’s goals of those PI’s are to highlight the contribution of the
businesses, they should be under the same KM solutions to business process improvements and to
controls as other management solutions, implying measure that contribution according to the business
that Knowledge management solutions should be objectives.
monitored as traditional solutions in order to
assess the impact on the business objectives.
2 Related Research
1 Introduction This topic is a new domain of which there is very little
previous research. However, this research is based on
Knowledge Management” is becoming a fashion word
which people are becoming accustomed to listening to. If some previous studies that are used as a foundation for the
KM workers were asked for a definition for KM, a large new framework and methodology.
list of them according to the role that the interviewee is
developing in the KM domain would be obtained. In the 2.1 Knowledge Management
concrete domain of performance measurement, KM is The researchers have done a survey in the KM project
leveraging the intellectual assets of the company to meet field in order to get the most relevant issues. Interesting
defined business objectives [Sve00]. documents have been found during this survey. The works
that Davenport et al [Dav00] have done in this area are
Senior managers can reveal, through analysing companies, interesting. They have published some important
that some knowledge bottlenecks must be solved in order documents among which can be found specific success
factors arising from KM projects are.
The copyright of this paper belongs to the paper’s authors. Permission to copy
without fee all or part of this material is granted provided that the copies are not Davenport et al’s paper is based on a survey of 31-KM
made or distributed for direct commercial advantage. projects in 23 companies and points out four common
Proc. of the Third Int. Conf. on Practical Aspects of voids that have been found in those projects. These voids
Knowledge Management (PAKM2000) produced KM project failures and are proposed as
Basel, Switzerland, 30-31 Oct. 2000, (U. Reimer, ed.) effectiveness indicators for KM projects. The same authors
http://sunsite.informatik.rwth-aachen.de/Publications/CEUR-WS/Vol-34/ developed another interesting paper related to KM projects
[DeL97]. A classification of KM projects is proposed in a
R. Roy, F.M. del Rey Chamorro, B. van Wegen, A. Steele 18-1
very practical manner. This taxonomy allows its use in a of business strategy and in the external one it is used for
working environment because it has been directly derived benchmarking purposes.
from a survey of 20-KM projects in 10 companies and
translated into more a concrete classification. 2.3 Performance Measurement In Knowledge
Management
On the other hand, the common principles of the KM have
been pointed out [Dav97]. These principles describe how There is little research related to this topic but they can be
the more effectively manage to more familiar issues, on a categorised in four different groups. Some of the most
daily basis. interesting segments of research of each category are
included in the following.
2.2 Performance Measurement
2.3.1 Knowledge measurement
The performance measurement is a field to which
companies have paid much attention recently. A lot of Here we can include sections of research, which aim to
information can be found related to this topic: A measure the knowledge level within an organisation. In
comprehensive survey dealing with performance this topic, Roger E. Bohn [Boh94] in his article,
measurement systems has been published by Neely et al Measuring and Managing Technological Knowledge
[Nel96]. propose a framework for measuring a particular type of
knowledge: Technological knowledge. This framework
The basis of the PM system creation has been set out can be used to more precisely map, evaluate, and compare
[Har97, Nel97]. We can find a detailed five-step method
levels of knowledge. He defines some basic characteristics
for developing a performance measurement system. The
levels in which a measurement system must be studied are of the technological knowledge and set out his framework.
[Nel96]: It ranges from complete ignorance to complete
understanding passing through eight stages.
• The individual performance measures
• The set of performance measures as a whole 2.3.1 Measuring Knowledge Work
• The relationship between the performance Other kind of measures related to knowledge is correlated
measurement system and the environment to knowledge work. Carl R. Moore’s article illustrates his
within which it operates development a set of metrics for measuring and forecasting
knowledge work [Moo99]. By knowledge work, he refers
The individual measure level can be broken down in two to the work done thinking as software. The set of measures
major categories: Lag indicators and Lead indicators is oriented towards software companies, which are
[Kap96]. The Lag indictors are those generic measures that fundamentally based in knowledge work, for evaluating
tend to be core outcome measures, which reflect the knowledge work in a function of software characteristics.
common goals of many strategies; These measures do not
provide an early indication about whether or not the 2.3.3 Measuring Knowledge Management
strategy is being implemented successfully. The lead
Alternatively, a paper related about the quality of current
indicators are those that tend to be unique for particular
knowledge management has been published [Hen99]. It
business units, reflecting the uniqueness of the business
describes an initiative that is currently developing in
strategic unit’s strategy: Such measures are able to reveal
Netherlands within some principle companies. They have
whether or not a business unit is able to reach the short-
developed, in four workshops, a framework in which
term operational improvements but fail to translate them
companies can measure their current situation with respect
into long term business objectives.
to intellectual capacity and related management structure,
in other words to measure how good their knowledge
Thus overall, there is no general consensus as to what a
management is. The project is called KnowMe.
performance measurement should focus on. Depending on
the sector in which the researcher is working on, the
Knowme requires data acquisition from several levels:
dimensions that a performance measurement system should
macro level, intermediate level and micro level. Macro
address, fluctuate. However, the best-known performance
level refers to the overall organisation, with intermediate
measurement framework is the Balanced Scorecard
level to teams or departments; the micro level to individual
[Kap96] that looks at businesses under four perspectives:
employees.
customer, internal, learning and growth and financial
perspective.
2.3.4 Micro To Macro Knowledge Management
Alignment
At the highest level two perspectives can be identified:
Internal and External perspective. In the Internal A six-step framework to align macro to micro knowledge
perspective the performance measurement system is a part management has been developed [Roy00]. The purpose of
R. Roy, F.M. del Rey Chamorro, B. van Wegen, A. Steele 18-2
this framework is to allow organisations to determine what This gap is filled by the lead indicators, which highlight
factors at the operational level should be measured in order the performance of the particular issues of each company.
to fulfil the strategic objectives of the business. The stages Somehow, the organisation’s high-level strategic
of this framework are objectives and measures need to be translated into actions
that each individual can take to contribute to organisation’s
• Determine which issues to address within the goals. However, many organisations have found it difficult
KM project to decompose highlevel strategic measures, especially
• Locate specific measures of the issues nonfinancial ones. Such lead indicators are different
determined in the first state according to the domain in which they are going to be
• Examine the interactions between the applied and the characteristics of the business. In an
measures and the process, and between the example provided by the BSC, in the Product
different measures Development department one of the core outcome measure
was time-to-market but a lead indicator that was added to
On the other hand, he has addressed the difference this lag indicator was the percentage of products for which
between Macro and Micro Knowledge Management. He the first design of a device fully met the customer’s
stated that Micro KM is “where the work gets done. At the functional specification.
operational level: KM projects being conducted within the
organisation. How the business will achieve the KM Learning and growth
targets is determined at this level”. By Macro KM, he • Measures of team performance
Internal • Measures of Individual and
pointed out that it is “where the work gets done. At the • Quality Organisational Alignment
Customer • Time • …
operational level: KM projects being conducted within the Financial
• Customer satisfaction• Cost
• Customer retention • …
organisation. How the business will achieve the KM • Revenue growth and mix • Market share
• Cost reduction/productivity• …
targets is determined at this level”. improvement
• Asset
utilisation/investment
strategy
3 A Gap In The Business Measurement • …
3.1 From The Strategic Level To The Operational
Level
Figure 1: BSC perspectives and strategic measures
Performance measurement systems are key in today’s
businesses. They allow not only monitoring of the business Examination of the KM domain reveals KM solution
performance according to the business objectives but also contributions can be reflected in those lag indicators.
assess the performance in comparison with similar However KM contribution is more valuable each time and
company performances by benchmarking. requires that the lead indicators monitor knowledge
performance.
Many PM systems have been set out but the most popular
is the Balanced Scorecard (BSC). Those PM systems like Of course, the main interest of such lead indicators is to
the BSC are characterised by the mixture of two kinds of clearly express the KM contribution to the business
performance measures: lag indicators or core outcomes in objectives. This point implies that the lead indicators
the strategic level and lead indicators or performance should be derived from those business goals.
drivers in the operational level. Within the BSC for
example companies are seen under four perspectives 3.2 KM Solutions And KM Processes In The
(financial, customer, internal and learning and growth). Operational Level
The lag indicators represent the core outcome measures of
the company but just tell managers how well the company KM initiatives are aimed to provide KM solutions to solve
has performed as a whole. knowledge bottlenecks previously identified in Business
processes at the Operational level such as Innovation or
Lag indicators are derived from the business objectives in Production. These Knowledge bottlenecks can be
order to represent the company performance according to identified through Knowledge mapping and the
the business goals. They are lagging measures, reporting identification of the possible opportunities of
how well an organisation’s strategy worked within a improvement.
previous time. Coupled with this, they are generic, in that
all companies are trying to improve along these
dimensions. Examples of Lag indicators in the BSC are
gathered in the Figure 1.
R. Roy, F.M. del Rey Chamorro, B. van Wegen, A. Steele 18-3
PI’s
Mission of the business
Internal scrutiny at the Environmental Scan at
Business level the Business level
Business Knowledge KM Knowledge Business
processes bottlenecks solutions processes processes
Formulation of the
Business Strategy
Figure 2: Integration of the measurement issues in the KM
issues
STRATEGIC Balanced scorecard or top
LEVEL level measurement system
KM solutions are composed of knowledge processes such
as Knowledge capturing or knowledge sharing which are
implemented to improve those Business processes.
Currently, businesses have a complete measurement OPERATIONAL
system, including all the Business processes but now with LEVEL
the introduction of the Knowledge processes, a new gap Knowledge
Business process
has emerged. The contribution of the Knowledge processes processes
can be measured by the improvement on the business Lag
indicators but there is a lack of KM solution lead
indicators.
Through the implementation of the new KM solutions, a Figure 3: The gap
gap has been created: The new methodology aims to cover
this gap.
4.2 Framework
The framework (Figure 4) is divided into three stages,
3.3 The Gap (stretching from the strategic to the operational level) in
which the measures are developed starting from the
The gap has been outlined previously. On one hand the
Business objectives that ensures the focus of the measures.
Lag indicators measure the performance of the Business
These stages contain the six-step framework that bridges
with respect to the Business objectives and on the other
the previously described gap between the strategic and the
hand there are new Knowledge processes that solve the
operational level. The stages of the framework are:
Knowledge bottleneck in the Business processes. There is
measures are required in order to help monitor the
1. development of the strategic measures from
efficiency and effectiveness in the Knowledge bottlenecks
the business strategy (steps: 1, 2)
solving in order to achieve the Business objectives. The
2. creation of the lead indicators for the
Figure 3 shows the gap in a whole organisation map.
Knowledge processes (steps: 3, 4, 5)
3. implementation of the measures (step: 6)
4 Measurement Framework The first stage of this framework is given by the current
development of PM systems in companies. The KPI
4.1 The Framework Requirements development starts in the second stage and drives to
measures able to monitor KM solution performances. The
The purpose of this framework is to allow organisations to
proposed framework is intended to be generic, but requires
complete the performance measurement systems by adding
further validation. A case study of this framework is
Lead indicators that measure the performance of
included in the next section.
Knowledge processes implemented in a Business process
as a result of a KM initiative. Examining the origins of this
4.2.1 At The Strategic Level
framework exposes its requirements, which can be
gathered in two main points: This first stage of the framework contains two steps. The
aim of this stage is to develop the top-level measures that
• To develop Performance Indicators to can monitor the performance of the Business according to
measure Knowledge process performance the business strategic objectives. Within the first step the
• To develop Performance Indicators that Strategy of the business is defined and based on it, the
monitor Knowledge processes according to strategic measures are created through a top-level measure
the Business objectives development framework, similar to the BSC during the
R. Roy, F.M. del Rey Chamorro, B. van Wegen, A. Steele 18-4
knowledge outcomes to address the knowledge constraints
Mission of the business
of business processes (Figure 5).
Internal scrutiny at the Environmental Scan at
Business level the Business level
The Knowledge process outcomes solve the Business
process knowledge bottlenecks, so those outcomes have to
1. Formulation of the be measured to monitor the performance of the KM
Business Strategy
solution. Those outcomes are taken as entities in the way
STAGE 1 that they are elements that has influence Business
processes. The practitioner can find the Knowledge
STRATEGIC 2. Balanced scorecard or top
LEVEL level measurement system process outcomes for each Knowledge bottleneck looking
at KM solution provided by the KM project.
Inputs
Knowledge bottlenecks ➙ 3. Identification of the Outputs
Knowledge processes ➙ Knowledge process ➙ Knowledge process In the fourth step, practitioners analyse each outcome, in
outcomes
outcomes order to discover and measure the influence of they have in
STAGE 2 the Business processes and measure them. On the other
Inputs
Knowledge process hand, the new measures have to take into account the
outcomes ➙
Business Lag Indicators ➙ 4. Impact on the Business Outputs company strategy in order to set out the contribution of the
Business processes ➙ Attributes
process KM solution to the Business objectives as was said in the
involved ➙
framework requirements. This is expressed through the
Business process lag performance indicators that have
Inputs
been directly derived from the Business strategy (Figure
➙ 5. Measurable action Outputs
Attributes
Business processes generation ➙ Measurable action 3). The business processes are also important in order to
➙
involved develop the PI’s. By knowing the environment in which
the measurement is taking place practitioners can develop
OPERATIONAL STAGE 3
LEVEL suitable PI’s for each specific environment. The
measurement of the knowledge captured and located in
Inputs
Measurable action ➙ 6. Performance indicator Outputs database for a Product Development department is no
Business process development ➙
environment ➙
Performance indicators
comparable to that of a Marketing department.
Knowledge
Business process
processes
KM Knowledge
solution processes
Figure 4: Six-step framework
Requirements Outcomes
second step. The measures that are obtained at the end of Knowledge
this stage are the strategic measures or lag indicators. The bottlenecks
lag indicators can tell whether or not the strategic
objectives are going to be reached but are deficient in
monitoring the operational level processes based on the
strategic objectives. Figure 5: KM project
4.2.2 Bridging The Strategic And The Operational Looking at the outcomes as entities, the practitioners have
Level to discover the attributes of the entities. The attributes on
The second stage of the framework is where the measures which the practitioners are interested are those that express
are derived, and holds three steps. Throughout this stage how the entity is contributing the Business process to reach
three steps are allocated. During the third step of the the Business objectives. This “how” is the key to get the
framework, the Knowledge process outcomes are brought PI’s and takes place in the following step.
to the PI development. KM projects typically identify the
knowledge bottlenecks with in a process and to solve them By knowing how the outcomes are contributing to the
is the requirement of any KM solution. KM solutions are Business process and which Business process is involved,
composed of Knowledge processes that provide the measurement can be highlighted by a measurement
action. Such actions are pointing out a specific issue in the
R. Roy, F.M. del Rey Chamorro, B. van Wegen, A. Steele 18-5
Business process and the measure can be taken according solution proposes to implement a knowledge sharing
to characteristics of the business process in which the process. One of the outcomes targeted by this knowledge
Knowledge bottleneck was found. Those issues are defined process is the “improvement of the standardisation”. This
during the fifth step of the framework. whole process constitutes the third step in the framework
providing the previous knowledge process outcome.
4.2.3 At The Operational Level
The knowledge process outcomes are considered as
This is the third and final stage of the framework. The
entities. Entities are processes that have an influence on
measures exit but are not implemented. It is important to
business processes as was previously stated. The fourth
understand which effect the measures will have on routine
step of the framework aims to analyse the influence of this
operations of the organisation or if can be mapped into
entity on the business process in order to obtain the
existing operational measures. According to the measures
attributes of the entity that solve this knowledge
obtained through the framework, the implementation is
bottleneck. The researchers refer to attributes by the way
different and requires a detailed study for each case.
of describing entities that are improving the business
process.
The results obtained with this approach can be categorised
into two groups: objective and subjective measures. The
objective measures are related to issues, which provide a KM solution model Opportunities for the
value that can be compared in order to follow the evolution measurement solution
of KM solutions and the improvement on business K. bottleneck Improvement in knowledge
processes. The subjective ones provide information about sharing across shifts
the KM solution performance from the viewpoint of Entities (Outcomes) Improvement of the
business process workers. standardisation
Impact Comp. Quality
The objective measures are preferred due to the simplicity on the dimension
of the measurement and ease of comparison among Business Departme Pre-processing/ Processing/
differing measurements. On the other hand, they are not process nt Packaging
affected by the subjectivity of the people that fulfil the How? The implementation of the
questionnaire or that are interviewed. However, those Attribute best practices will increase
measures are not always feasible. The objective measures the quality of the production
monitor the performance of KM solutions within business by reducing the number of
processes and could be susceptible to influence from other defects on the Production
projects that could be implemented in the business process line
such a process has been referred to as overlapping in this Measurable action Reduction of defects due to
research. (MA) the implementation of the
best practice and similar
performance among shifts
4.3 Case Study Performance indicator Ratio: Number of
The researchers propose the following case study with the defects/average number of
aim of showing how the framework works. This case is defects among shifts
focused on the production department. The whole business
is monitored by a PM system as any other company. The Figure 6: Case study
BSC is taken as the PM system and allows senior
managers to control the business under four perspectives For this step, information about the business lag indicators
(Figure 1). The KM project that has been developed for and the department involved is required. That information
this case attempts to improve the efficiency of this highlights that the entity improves the business process:
production business process. The measures that control the “The implementation of the best practices will increase the
production business process performance are gathered in quality of the production by reducing the number of
the Internal perspective of the BSC (Figure 1). defects in the production line”.
Within the business process, the KM project identified the With this entity attribute and further knowledge about the
knowledge bottlenecks that constrained the production business process involved, the action that should be
line. Consequently, the KM project proposes a KM measured is highlighted. This measurable action achieved
solution in which the knowledge bottlenecks are addressed. in the fifth step of the framework is on which practitioners
This case study focuses on a particular knowledge should focuses in order to come up with the measure. The
bottleneck within the KM project. It is “Improvement in measurable action in this case study is “Ratio: Number of
knowledge sharing across shifts”. As a result, the KM defects/ average number of defects among shifts”.
R. Roy, F.M. del Rey Chamorro, B. van Wegen, A. Steele 18-6
5 Discussion The interference of the developed measures with other
projects implemented in the same Business process at the
same time can drive to an incorrect analysis of the KM
5.1 The Framework Focus solution effect and would invalidate the usage of objective
KM is emerging in today’s business as a new tool to measures. In these cases, subjective measures are
deploy the intellectual capital of the business and improve recommended in order to show the real effect of the KM
the business performance. Those opportunities are solutions in a qualitative way.
becoming the KM domain a key area in order to get
competitive advantage and in consequence the necessity of 6. Conclusions
controlling how well the KM solutions are performing in
businesses is also imperative. Measuring performance of Knowledge Management
solutions to achieve business objectives is becoming
Those KM solutions are implemented to solve knowledge popular in industry. An initial study confirmed that there
bottlenecks in a particular Business process such as was no framework that aims to measure the effectiveness
production or marketing among others. In today’s of KM solutions to achieve business objectives.
companies PM systems monitor the performance of all the
Business processes but a new gap in this PM system has This research has developed a 'step by step' framework and
been introduced, with the implementation of a KM a methodological approach to identify the Key
solution in a Business process. The necessity of new Performance Indicators (KPIs) for KM solutions. The
measures becomes evident, to control the contribution to conceptual framework connects the strategic measurement
Business processes of KM solutions in order to allow tools to KM solutions at the operational level. The
managers to monitor Business processes perfectly. methodological approach provides practitioners with a set
of templates that help them to carry out the conceptual
The main point at this stage is to know how to monitor framework. Both of them together produce a structured
those KM solutions. The measurement should be driven by way to develop KPI’s according to the business objectives.
some objectives or goals in which are reflected the The framework is validated on two real life KM projects
performance of the KM solution is reflected. The most from the sponsoring company.
useful manner to assess the performance of the KM
solution is with respect to the business objectives. The KPI’s developed with this framework measure the
However, the business objectives are not explicit enough effectiveness of the KM solutions in business processes.
to be applicable to that low level of business. The solution That allows companies not to only monitor if the
provided by this framework is to follow the PM system knowledge is being managed right but also if the right
that has been monitoring the business so far and to develop knowledge is being managed within the company.
PI’s based on the Lag indicators of the PM system to
measure the performance of each business process. References
[Boh94] Bohn, R. E. Measuring and Managing
5.2 Usage Of The Framework Technological Knowledge, Sloan Management
Although an example of PI creation has been included Review, 1994, pp. 61-73
comments can be incorporated with respect to the
framework characteristics. The managers of today’s [Dav97a]Davenport, T., De Long, D. and Beers, M.
businesses would like to vividly see all operations Building Successful Knowledge Management
implemented in companies and one of the principal Projects, Working Paper, Ernst & Young Center
requirements that the researchers set out before developing for Business Innovation, January 1997.
the framework was the transparency of it.
[Dav97b]Davenport, T. Ten principles of Knowledge
To avoid this black box effect, the PI’s development is Management, Knowledge and Process
easily drive through a set of little steps that start from the Management, 4(3), 1997
outcome identification to the PI generation passing through
an easily understandable analysis of the impact of the [DeL97] De Long, D. Davenport, T. and Beers, M. What is
outcome on the Business process. a Knowledge Management Project? Working
Another point that should be highlighted is the easy usage Paper, Ernst & Young Center for Business
of the framework because it does not require a big Innovation, February, 1997
knowledge from PM systems, due to its clear PI
development way. This is an advantage for reducing the [Har97] Harbour, J. The basics of Performance
implementation cost of the PI development. Measurement, Quality resources, 1997
R. Roy, F.M. del Rey Chamorro, B. van Wegen, A. Steele 18-7
[Hen99] Hendriks, B., Swaak, J., Lansink, A. Van
Amlsfort, T., Heeren, E. and Kalff, P. The
Knowledge Management Measure of Telematica
Instituut of Netherlands, The Journal of the Cap
Gemini Applied Knowledge Management Natural
Work Team, May 1999, pp. 13-23
[Kap96] Kaplan, R. and Norton, D. The Balanced
Scorecard, Harvard Business School press, 1996
[Moo99]Moore, C. R. Performance Measures for
Knowledge Management, Knowledge
Management Handbook, CRC Press LLC
[Nel96] Nelly, A., Gregory, M. and Platts, K.
Performance measurement system design,
International Journal of Operations & production
Management, 15 (4), 1996, pp. 80-116
[Nel96] Neely, A. Richards, H. Mills, J. Platts, K. and
Bourne, M. Designing performance measures: a
structured approach, International Journal of
Operations and Production Management, vol. 17,
n 11, 1997, pp. 1131-1152
[Roy00] Roy, R., Newman, V. and Chandler, M. C.
Aligning Micro to Macro Knowledge
Management, The Third International Conference
on The Practical Application of Knowledge
Management, PAKeM2000 Proceedings, 2000,
pp. 169-180
[Sve00] Sveiby, E. KM info.
http://www.pharmaknowledge.com/kminfo.htm
(accessed 15th January 2000)
R. Roy, F.M. del Rey Chamorro, B. van Wegen, A. Steele 18-8