A Framework To Create Performance Indicators In Knowledge Management Rajkumar Roy Francisco M. del Rey Bert van Wegen Andy Steele Cranfield University, Chamorro Unilever Research Unilever Research UK Cranfield University, r.roy@cranfield.ac.uk UK fmdelrey@hotmail.com to improve the performance of the company and reach the business objectives. However managers are striving to uncover which specific business contributions are down to Abstract KM solutions. By knowing the real contribution of those knowledge processes managers obtain on one hand Knowledge management (KM) is emerging as one information about the achievement of the business of the most powerful management tools in today’s objectives and on the other hand to see a clear relationship manufacturing. It looks at the company resources between the KM solutions and the business improvements. in order to gain competitive advantage. The management of these resources can mean the Managers use performance measurements to monitor key difference between success and failure in a issues in businesses. These measures provide the most competitive environment. This encourages relevant information of the company, showing managers companies to look for better ways in the how the business is performing. The solution proposed in management of these intangible assets, developing this paper provides a mechanism for monitoring KM KM projects in order to provide KM solutions to solutions in those issues related to Business process. This solve knowledge bottlenecks through Knowledge new methodology is focused on the business objectives to processes. However, if KM solutions are create performance indicators (PI’s) KM solutions. The considered as an important part in today’s goals of those PI’s are to highlight the contribution of the businesses, they should be under the same KM solutions to business process improvements and to controls as other management solutions, implying measure that contribution according to the business that Knowledge management solutions should be objectives. monitored as traditional solutions in order to assess the impact on the business objectives. 2 Related Research 1 Introduction This topic is a new domain of which there is very little previous research. However, this research is based on Knowledge Management” is becoming a fashion word which people are becoming accustomed to listening to. If some previous studies that are used as a foundation for the KM workers were asked for a definition for KM, a large new framework and methodology. list of them according to the role that the interviewee is developing in the KM domain would be obtained. In the 2.1 Knowledge Management concrete domain of performance measurement, KM is The researchers have done a survey in the KM project leveraging the intellectual assets of the company to meet field in order to get the most relevant issues. Interesting defined business objectives [Sve00]. documents have been found during this survey. The works that Davenport et al [Dav00] have done in this area are Senior managers can reveal, through analysing companies, interesting. They have published some important that some knowledge bottlenecks must be solved in order documents among which can be found specific success factors arising from KM projects are. The copyright of this paper belongs to the paper’s authors. Permission to copy without fee all or part of this material is granted provided that the copies are not Davenport et al’s paper is based on a survey of 31-KM made or distributed for direct commercial advantage. projects in 23 companies and points out four common Proc. of the Third Int. Conf. on Practical Aspects of voids that have been found in those projects. These voids Knowledge Management (PAKM2000) produced KM project failures and are proposed as Basel, Switzerland, 30-31 Oct. 2000, (U. Reimer, ed.) effectiveness indicators for KM projects. The same authors http://sunsite.informatik.rwth-aachen.de/Publications/CEUR-WS/Vol-34/ developed another interesting paper related to KM projects [DeL97]. A classification of KM projects is proposed in a R. Roy, F.M. del Rey Chamorro, B. van Wegen, A. Steele 18-1 very practical manner. This taxonomy allows its use in a of business strategy and in the external one it is used for working environment because it has been directly derived benchmarking purposes. from a survey of 20-KM projects in 10 companies and translated into more a concrete classification. 2.3 Performance Measurement In Knowledge Management On the other hand, the common principles of the KM have been pointed out [Dav97]. These principles describe how There is little research related to this topic but they can be the more effectively manage to more familiar issues, on a categorised in four different groups. Some of the most daily basis. interesting segments of research of each category are included in the following. 2.2 Performance Measurement 2.3.1 Knowledge measurement The performance measurement is a field to which companies have paid much attention recently. A lot of Here we can include sections of research, which aim to information can be found related to this topic: A measure the knowledge level within an organisation. In comprehensive survey dealing with performance this topic, Roger E. Bohn [Boh94] in his article, measurement systems has been published by Neely et al Measuring and Managing Technological Knowledge [Nel96]. propose a framework for measuring a particular type of knowledge: Technological knowledge. This framework The basis of the PM system creation has been set out can be used to more precisely map, evaluate, and compare [Har97, Nel97]. We can find a detailed five-step method levels of knowledge. He defines some basic characteristics for developing a performance measurement system. The levels in which a measurement system must be studied are of the technological knowledge and set out his framework. [Nel96]: It ranges from complete ignorance to complete understanding passing through eight stages. • The individual performance measures • The set of performance measures as a whole 2.3.1 Measuring Knowledge Work • The relationship between the performance Other kind of measures related to knowledge is correlated measurement system and the environment to knowledge work. Carl R. Moore’s article illustrates his within which it operates development a set of metrics for measuring and forecasting knowledge work [Moo99]. By knowledge work, he refers The individual measure level can be broken down in two to the work done thinking as software. The set of measures major categories: Lag indicators and Lead indicators is oriented towards software companies, which are [Kap96]. The Lag indictors are those generic measures that fundamentally based in knowledge work, for evaluating tend to be core outcome measures, which reflect the knowledge work in a function of software characteristics. common goals of many strategies; These measures do not provide an early indication about whether or not the 2.3.3 Measuring Knowledge Management strategy is being implemented successfully. The lead Alternatively, a paper related about the quality of current indicators are those that tend to be unique for particular knowledge management has been published [Hen99]. It business units, reflecting the uniqueness of the business describes an initiative that is currently developing in strategic unit’s strategy: Such measures are able to reveal Netherlands within some principle companies. They have whether or not a business unit is able to reach the short- developed, in four workshops, a framework in which term operational improvements but fail to translate them companies can measure their current situation with respect into long term business objectives. to intellectual capacity and related management structure, in other words to measure how good their knowledge Thus overall, there is no general consensus as to what a management is. The project is called KnowMe. performance measurement should focus on. Depending on the sector in which the researcher is working on, the Knowme requires data acquisition from several levels: dimensions that a performance measurement system should macro level, intermediate level and micro level. Macro address, fluctuate. However, the best-known performance level refers to the overall organisation, with intermediate measurement framework is the Balanced Scorecard level to teams or departments; the micro level to individual [Kap96] that looks at businesses under four perspectives: employees. customer, internal, learning and growth and financial perspective. 2.3.4 Micro To Macro Knowledge Management Alignment At the highest level two perspectives can be identified: Internal and External perspective. In the Internal A six-step framework to align macro to micro knowledge perspective the performance measurement system is a part management has been developed [Roy00]. The purpose of R. Roy, F.M. del Rey Chamorro, B. van Wegen, A. Steele 18-2 this framework is to allow organisations to determine what This gap is filled by the lead indicators, which highlight factors at the operational level should be measured in order the performance of the particular issues of each company. to fulfil the strategic objectives of the business. The stages Somehow, the organisation’s high-level strategic of this framework are objectives and measures need to be translated into actions that each individual can take to contribute to organisation’s • Determine which issues to address within the goals. However, many organisations have found it difficult KM project to decompose highlevel strategic measures, especially • Locate specific measures of the issues nonfinancial ones. Such lead indicators are different determined in the first state according to the domain in which they are going to be • Examine the interactions between the applied and the characteristics of the business. In an measures and the process, and between the example provided by the BSC, in the Product different measures Development department one of the core outcome measure was time-to-market but a lead indicator that was added to On the other hand, he has addressed the difference this lag indicator was the percentage of products for which between Macro and Micro Knowledge Management. He the first design of a device fully met the customer’s stated that Micro KM is “where the work gets done. At the functional specification. operational level: KM projects being conducted within the organisation. How the business will achieve the KM Learning and growth targets is determined at this level”. By Macro KM, he • Measures of team performance Internal • Measures of Individual and pointed out that it is “where the work gets done. At the • Quality Organisational Alignment Customer • Time • … operational level: KM projects being conducted within the Financial • Customer satisfaction• Cost • Customer retention • … organisation. How the business will achieve the KM • Revenue growth and mix • Market share • Cost reduction/productivity• … targets is determined at this level”. improvement • Asset utilisation/investment strategy 3 A Gap In The Business Measurement • … 3.1 From The Strategic Level To The Operational Level Figure 1: BSC perspectives and strategic measures Performance measurement systems are key in today’s businesses. They allow not only monitoring of the business Examination of the KM domain reveals KM solution performance according to the business objectives but also contributions can be reflected in those lag indicators. assess the performance in comparison with similar However KM contribution is more valuable each time and company performances by benchmarking. requires that the lead indicators monitor knowledge performance. Many PM systems have been set out but the most popular is the Balanced Scorecard (BSC). Those PM systems like Of course, the main interest of such lead indicators is to the BSC are characterised by the mixture of two kinds of clearly express the KM contribution to the business performance measures: lag indicators or core outcomes in objectives. This point implies that the lead indicators the strategic level and lead indicators or performance should be derived from those business goals. drivers in the operational level. Within the BSC for example companies are seen under four perspectives 3.2 KM Solutions And KM Processes In The (financial, customer, internal and learning and growth). Operational Level The lag indicators represent the core outcome measures of the company but just tell managers how well the company KM initiatives are aimed to provide KM solutions to solve has performed as a whole. knowledge bottlenecks previously identified in Business processes at the Operational level such as Innovation or Lag indicators are derived from the business objectives in Production. These Knowledge bottlenecks can be order to represent the company performance according to identified through Knowledge mapping and the the business goals. They are lagging measures, reporting identification of the possible opportunities of how well an organisation’s strategy worked within a improvement. previous time. Coupled with this, they are generic, in that all companies are trying to improve along these dimensions. Examples of Lag indicators in the BSC are gathered in the Figure 1. R. Roy, F.M. del Rey Chamorro, B. van Wegen, A. Steele 18-3 PI’s Mission of the business Internal scrutiny at the Environmental Scan at Business level the Business level Business Knowledge KM Knowledge Business processes bottlenecks solutions processes processes Formulation of the Business Strategy Figure 2: Integration of the measurement issues in the KM issues STRATEGIC Balanced scorecard or top LEVEL level measurement system KM solutions are composed of knowledge processes such as Knowledge capturing or knowledge sharing which are implemented to improve those Business processes. Currently, businesses have a complete measurement OPERATIONAL system, including all the Business processes but now with LEVEL the introduction of the Knowledge processes, a new gap Knowledge Business process has emerged. The contribution of the Knowledge processes processes can be measured by the improvement on the business Lag indicators but there is a lack of KM solution lead indicators. Through the implementation of the new KM solutions, a Figure 3: The gap gap has been created: The new methodology aims to cover this gap. 4.2 Framework The framework (Figure 4) is divided into three stages, 3.3 The Gap (stretching from the strategic to the operational level) in which the measures are developed starting from the The gap has been outlined previously. On one hand the Business objectives that ensures the focus of the measures. Lag indicators measure the performance of the Business These stages contain the six-step framework that bridges with respect to the Business objectives and on the other the previously described gap between the strategic and the hand there are new Knowledge processes that solve the operational level. The stages of the framework are: Knowledge bottleneck in the Business processes. There is measures are required in order to help monitor the 1. development of the strategic measures from efficiency and effectiveness in the Knowledge bottlenecks the business strategy (steps: 1, 2) solving in order to achieve the Business objectives. The 2. creation of the lead indicators for the Figure 3 shows the gap in a whole organisation map. Knowledge processes (steps: 3, 4, 5) 3. implementation of the measures (step: 6) 4 Measurement Framework The first stage of this framework is given by the current development of PM systems in companies. The KPI 4.1 The Framework Requirements development starts in the second stage and drives to measures able to monitor KM solution performances. The The purpose of this framework is to allow organisations to proposed framework is intended to be generic, but requires complete the performance measurement systems by adding further validation. A case study of this framework is Lead indicators that measure the performance of included in the next section. Knowledge processes implemented in a Business process as a result of a KM initiative. Examining the origins of this 4.2.1 At The Strategic Level framework exposes its requirements, which can be gathered in two main points: This first stage of the framework contains two steps. The aim of this stage is to develop the top-level measures that • To develop Performance Indicators to can monitor the performance of the Business according to measure Knowledge process performance the business strategic objectives. Within the first step the • To develop Performance Indicators that Strategy of the business is defined and based on it, the monitor Knowledge processes according to strategic measures are created through a top-level measure the Business objectives development framework, similar to the BSC during the R. Roy, F.M. del Rey Chamorro, B. van Wegen, A. Steele 18-4 knowledge outcomes to address the knowledge constraints Mission of the business of business processes (Figure 5). Internal scrutiny at the Environmental Scan at Business level the Business level The Knowledge process outcomes solve the Business process knowledge bottlenecks, so those outcomes have to 1. Formulation of the be measured to monitor the performance of the KM Business Strategy solution. Those outcomes are taken as entities in the way STAGE 1 that they are elements that has influence Business processes. The practitioner can find the Knowledge STRATEGIC 2. Balanced scorecard or top LEVEL level measurement system process outcomes for each Knowledge bottleneck looking at KM solution provided by the KM project. Inputs Knowledge bottlenecks ➙ 3. Identification of the Outputs Knowledge processes ➙ Knowledge process ➙ Knowledge process In the fourth step, practitioners analyse each outcome, in outcomes outcomes order to discover and measure the influence of they have in STAGE 2 the Business processes and measure them. On the other Inputs Knowledge process hand, the new measures have to take into account the outcomes ➙ Business Lag Indicators ➙ 4. Impact on the Business Outputs company strategy in order to set out the contribution of the Business processes ➙ Attributes process KM solution to the Business objectives as was said in the involved ➙ framework requirements. This is expressed through the Business process lag performance indicators that have Inputs been directly derived from the Business strategy (Figure ➙ 5. Measurable action Outputs Attributes Business processes generation ➙ Measurable action 3). The business processes are also important in order to ➙ involved develop the PI’s. By knowing the environment in which the measurement is taking place practitioners can develop OPERATIONAL STAGE 3 LEVEL suitable PI’s for each specific environment. The measurement of the knowledge captured and located in Inputs Measurable action ➙ 6. Performance indicator Outputs database for a Product Development department is no Business process development ➙ environment ➙ Performance indicators comparable to that of a Marketing department. Knowledge Business process processes KM Knowledge solution processes Figure 4: Six-step framework Requirements Outcomes second step. The measures that are obtained at the end of Knowledge this stage are the strategic measures or lag indicators. The bottlenecks lag indicators can tell whether or not the strategic objectives are going to be reached but are deficient in monitoring the operational level processes based on the strategic objectives. Figure 5: KM project 4.2.2 Bridging The Strategic And The Operational Looking at the outcomes as entities, the practitioners have Level to discover the attributes of the entities. The attributes on The second stage of the framework is where the measures which the practitioners are interested are those that express are derived, and holds three steps. Throughout this stage how the entity is contributing the Business process to reach three steps are allocated. During the third step of the the Business objectives. This “how” is the key to get the framework, the Knowledge process outcomes are brought PI’s and takes place in the following step. to the PI development. KM projects typically identify the knowledge bottlenecks with in a process and to solve them By knowing how the outcomes are contributing to the is the requirement of any KM solution. KM solutions are Business process and which Business process is involved, composed of Knowledge processes that provide the measurement can be highlighted by a measurement action. Such actions are pointing out a specific issue in the R. Roy, F.M. del Rey Chamorro, B. van Wegen, A. Steele 18-5 Business process and the measure can be taken according solution proposes to implement a knowledge sharing to characteristics of the business process in which the process. One of the outcomes targeted by this knowledge Knowledge bottleneck was found. Those issues are defined process is the “improvement of the standardisation”. This during the fifth step of the framework. whole process constitutes the third step in the framework providing the previous knowledge process outcome. 4.2.3 At The Operational Level The knowledge process outcomes are considered as This is the third and final stage of the framework. The entities. Entities are processes that have an influence on measures exit but are not implemented. It is important to business processes as was previously stated. The fourth understand which effect the measures will have on routine step of the framework aims to analyse the influence of this operations of the organisation or if can be mapped into entity on the business process in order to obtain the existing operational measures. According to the measures attributes of the entity that solve this knowledge obtained through the framework, the implementation is bottleneck. The researchers refer to attributes by the way different and requires a detailed study for each case. of describing entities that are improving the business process. The results obtained with this approach can be categorised into two groups: objective and subjective measures. The objective measures are related to issues, which provide a KM solution model Opportunities for the value that can be compared in order to follow the evolution measurement solution of KM solutions and the improvement on business K. bottleneck Improvement in knowledge processes. The subjective ones provide information about sharing across shifts the KM solution performance from the viewpoint of Entities (Outcomes) Improvement of the business process workers. standardisation Impact Comp. Quality The objective measures are preferred due to the simplicity on the dimension of the measurement and ease of comparison among Business Departme Pre-processing/ Processing/ differing measurements. On the other hand, they are not process nt Packaging affected by the subjectivity of the people that fulfil the How? The implementation of the questionnaire or that are interviewed. However, those Attribute best practices will increase measures are not always feasible. The objective measures the quality of the production monitor the performance of KM solutions within business by reducing the number of processes and could be susceptible to influence from other defects on the Production projects that could be implemented in the business process line such a process has been referred to as overlapping in this Measurable action Reduction of defects due to research. (MA) the implementation of the best practice and similar performance among shifts 4.3 Case Study Performance indicator Ratio: Number of The researchers propose the following case study with the defects/average number of aim of showing how the framework works. This case is defects among shifts focused on the production department. The whole business is monitored by a PM system as any other company. The Figure 6: Case study BSC is taken as the PM system and allows senior managers to control the business under four perspectives For this step, information about the business lag indicators (Figure 1). The KM project that has been developed for and the department involved is required. That information this case attempts to improve the efficiency of this highlights that the entity improves the business process: production business process. The measures that control the “The implementation of the best practices will increase the production business process performance are gathered in quality of the production by reducing the number of the Internal perspective of the BSC (Figure 1). defects in the production line”. Within the business process, the KM project identified the With this entity attribute and further knowledge about the knowledge bottlenecks that constrained the production business process involved, the action that should be line. Consequently, the KM project proposes a KM measured is highlighted. This measurable action achieved solution in which the knowledge bottlenecks are addressed. in the fifth step of the framework is on which practitioners This case study focuses on a particular knowledge should focuses in order to come up with the measure. The bottleneck within the KM project. It is “Improvement in measurable action in this case study is “Ratio: Number of knowledge sharing across shifts”. As a result, the KM defects/ average number of defects among shifts”. R. Roy, F.M. del Rey Chamorro, B. van Wegen, A. Steele 18-6 5 Discussion The interference of the developed measures with other projects implemented in the same Business process at the same time can drive to an incorrect analysis of the KM 5.1 The Framework Focus solution effect and would invalidate the usage of objective KM is emerging in today’s business as a new tool to measures. In these cases, subjective measures are deploy the intellectual capital of the business and improve recommended in order to show the real effect of the KM the business performance. Those opportunities are solutions in a qualitative way. becoming the KM domain a key area in order to get competitive advantage and in consequence the necessity of 6. Conclusions controlling how well the KM solutions are performing in businesses is also imperative. Measuring performance of Knowledge Management solutions to achieve business objectives is becoming Those KM solutions are implemented to solve knowledge popular in industry. An initial study confirmed that there bottlenecks in a particular Business process such as was no framework that aims to measure the effectiveness production or marketing among others. In today’s of KM solutions to achieve business objectives. companies PM systems monitor the performance of all the Business processes but a new gap in this PM system has This research has developed a 'step by step' framework and been introduced, with the implementation of a KM a methodological approach to identify the Key solution in a Business process. The necessity of new Performance Indicators (KPIs) for KM solutions. The measures becomes evident, to control the contribution to conceptual framework connects the strategic measurement Business processes of KM solutions in order to allow tools to KM solutions at the operational level. The managers to monitor Business processes perfectly. methodological approach provides practitioners with a set of templates that help them to carry out the conceptual The main point at this stage is to know how to monitor framework. Both of them together produce a structured those KM solutions. The measurement should be driven by way to develop KPI’s according to the business objectives. some objectives or goals in which are reflected the The framework is validated on two real life KM projects performance of the KM solution is reflected. The most from the sponsoring company. useful manner to assess the performance of the KM solution is with respect to the business objectives. The KPI’s developed with this framework measure the However, the business objectives are not explicit enough effectiveness of the KM solutions in business processes. to be applicable to that low level of business. The solution That allows companies not to only monitor if the provided by this framework is to follow the PM system knowledge is being managed right but also if the right that has been monitoring the business so far and to develop knowledge is being managed within the company. PI’s based on the Lag indicators of the PM system to measure the performance of each business process. References [Boh94] Bohn, R. E. Measuring and Managing 5.2 Usage Of The Framework Technological Knowledge, Sloan Management Although an example of PI creation has been included Review, 1994, pp. 61-73 comments can be incorporated with respect to the framework characteristics. The managers of today’s [Dav97a]Davenport, T., De Long, D. and Beers, M. businesses would like to vividly see all operations Building Successful Knowledge Management implemented in companies and one of the principal Projects, Working Paper, Ernst & Young Center requirements that the researchers set out before developing for Business Innovation, January 1997. the framework was the transparency of it. [Dav97b]Davenport, T. Ten principles of Knowledge To avoid this black box effect, the PI’s development is Management, Knowledge and Process easily drive through a set of little steps that start from the Management, 4(3), 1997 outcome identification to the PI generation passing through an easily understandable analysis of the impact of the [DeL97] De Long, D. Davenport, T. and Beers, M. What is outcome on the Business process. a Knowledge Management Project? Working Another point that should be highlighted is the easy usage Paper, Ernst & Young Center for Business of the framework because it does not require a big Innovation, February, 1997 knowledge from PM systems, due to its clear PI development way. This is an advantage for reducing the [Har97] Harbour, J. The basics of Performance implementation cost of the PI development. Measurement, Quality resources, 1997 R. Roy, F.M. del Rey Chamorro, B. van Wegen, A. Steele 18-7 [Hen99] Hendriks, B., Swaak, J., Lansink, A. Van Amlsfort, T., Heeren, E. and Kalff, P. The Knowledge Management Measure of Telematica Instituut of Netherlands, The Journal of the Cap Gemini Applied Knowledge Management Natural Work Team, May 1999, pp. 13-23 [Kap96] Kaplan, R. and Norton, D. The Balanced Scorecard, Harvard Business School press, 1996 [Moo99]Moore, C. R. Performance Measures for Knowledge Management, Knowledge Management Handbook, CRC Press LLC [Nel96] Nelly, A., Gregory, M. and Platts, K. Performance measurement system design, International Journal of Operations & production Management, 15 (4), 1996, pp. 80-116 [Nel96] Neely, A. Richards, H. Mills, J. Platts, K. and Bourne, M. Designing performance measures: a structured approach, International Journal of Operations and Production Management, vol. 17, n 11, 1997, pp. 1131-1152 [Roy00] Roy, R., Newman, V. and Chandler, M. C. Aligning Micro to Macro Knowledge Management, The Third International Conference on The Practical Application of Knowledge Management, PAKeM2000 Proceedings, 2000, pp. 169-180 [Sve00] Sveiby, E. KM info. http://www.pharmaknowledge.com/kminfo.htm (accessed 15th January 2000) R. Roy, F.M. del Rey Chamorro, B. van Wegen, A. Steele 18-8