Proceedings of STPIS'16 Evaluating ERP as a composition of different functionalities from key stakeholder perspectives Nina Habjan, Tomaž Hovelja, Damjan Vavpotič University of Ljubljana Faculty of computer and information science Vecna pot 113, SI-1000 Ljubljana Abstract. The field of evaluation of information systems (IS) success has a long and established history. However, models in the field usually evaluate success on the level of the whole IS while neglecting the differences in impacts of specific IS functionalities on IS success. Moreover, evaluation models often focus only on technological or only on social perspective neglecting that IS should be considered a socio-technical system and that all key stakeholders’ perspectives should be evaluated. Thus, three main perspectives of IS evaluation have to be recognized, namely user/social, technological and managerial perspectives. In this paper we present a novel model for evaluation of ERP success con- sidering all key stakeholder perspectives. The proposed model provides the evaluation of individual functionalities from these perspectives in contrast to established models that consider system as a whole. The model has been tested in a case study in a company from the nautical industry. The results confirmed the usefulness of the evaluation model, es- pecially its ability to provide specific new information to management on the influence of the individual functionalities on ERP success. 1 Introduction Since the mid-nineties, many organizations have decided to implement Enterprise Re- source Planning (ERP) systems in order to improve business efficiency, lower the costs of production, shorten the throughput times, expand their services, provide more relia- ble delivery dates and improve the coordination of global demand, supply and produc- tion. Due to the complexity of these solutions and the need to adapt business processes to selected software solutions, their introduction is challenging, relatively lengthy and typically exceeds the budget. Despite the high investments, many companies are not successful in implementation of ERP system and 70% of implementation projects do not achieve their goals [1]. Investments in enterprise information systems, of which ERP systems are a key component, are a heavy financial burden, so they are under increasing scrutiny and pressure to justify their value and contribution to the productiv- ity, quality, and competitiveness of organizations [2]. Even though the above-described reasons demonstrate a strong need for a systematic evaluation of the ERP systems, it turns out that in practice, information systems in gen- eral and ERP systems in particular are seldom systematically evaluated after the imple- mentation [3]. Besides the lack of time, the reasons for this situation are: the benefits Edited by S. Kowalski, P. Bednar and I. Bider 1 Proceedings of STPIS'16 of ERP systems are mostly indirect thus making the evaluation of direct financial ben- efits difficult; the system is used by a multitude of stakeholders who have different views on when an information system is successful. These stakeholders typically have multiple and often conflicting interests and rarely agree on a set of common aims [4]. Therefore, the management has to realize that social aspects have to be considered as well as those of the non-human parts of the system [34] which means that the system will deliver the expected benefits to the enterprises only if and when the users accept the new technology are aware of its importance for the organization, using it conscientiously. The aim of our research is to present a new model that enables management to eval- uate ERP systems from the point of view of all key stakeholders in the enterprise, thus gaining a complete evaluation of their ERP system from the user, managerial and tech- nological perspective. Fulfilling this aim poses real challenges since ERP systems are notoriously complex, perform a multitude of tasks and usually have a high number of specific functionalities. For this reason, the basic element of our evaluation model is an individual functionality of ERP. This allows the evaluators to highlight the good and bad parts of the ERP system without being forced to give the average score for the entire system. Critical functionalities can be identified by comparing the scores to each other, which can be easily accomplished by displaying the results on the scatter plot. This enables the organization to take appropriate action by preparing necessary im- provements. 2 Literature review In this section we review the relevant literature that studies the main three dimensions of ERP system implementation success: the user perspective, managerial perspective and technological perspective. 2.1 User perspective The user perspective studies one of the key success factors of the implementation of the new system; the desired level of its use. Use of the system is a reflection of the tech- nology acceptance by users. Acceptance, however, is not equivalent to success, alt- hough acceptance of an information system is a necessary precondition for success [6]. Researchers have derived a number of models [7] to measure technology acceptance. Among them, we should mention Rogers’ Theory of diffusion of innovation [8], which tries to explain why, how and at what speed certain innovations spread among potential users, while others do not. As new technology is seen as a novelty in terms of future users, its adoption can be considered the same as adoption of an innovation. The most renowned of the models explaining technology acceptance is Davis’ TAM model [9], which is based on the Theory of reasoned action (TRA) [10]. TAM states that perceived usefulness and perceived ease of use are key factors of technology ac- ceptance. The core of TAM is behavioural intention to use, but this is not an appropriate ©Copyright held by the author(s) 2 Proceedings of STPIS'16 measure if usage is mandated by the organization [13], [14], [15]. TAM2 [16] was de- veloped in response to this situation. Delone and McLean propose that the variable 'use' should be upgraded in terms of the nature of use, as even when use is required, varia- bility in the quality and intensity of use is likely to have a significant impact on the realization of the system benefits [17]. Urbach and Müller say that when the use of IS is required, measuring user satisfac- tion becomes especially useful [18]. This measure is proposed for evaluation of various IS by many other authors as well [19], [20], [21], [22], [23]. User satisfaction is considered as one of the most important measures of IS success [18], [24]. It is generally assumed that satisfied users are more likely to use the system fully and properly [15], which is a prerequisite for success of IS, while unsatisfied users often change technically successful system into a complete failure [25]. The most used among the models for measuring user satisfaction is the one by Bailey and Pearson [26], from which less extensive version of the model was derived [20] which was tested and approved as suitable for ERP systems. User satisfaction is also one of the dimensions of a widely recognized and frequently used model of Delone and McLean [17] on the basis of which the model, designed to measure the success of ERP systems was developed [2]. This model emphasizes the importance of the quality of the system, the quality of information and the quality of support. These three variables are described as the one having an important impact on user satisfaction. This claim was confirmed in several empirical studies [17]. 2.2 Managerial perspective Management is primarily interested in the net benefits of the implemented system for the organization. Measuring the net benefits by financial performance indicators has proved to be inadequate [2], [3], [18]. Martinsons et al. [27] therefore propose to adapt the Balanced Scorecard (BSC) [28], on the basis of which Chand et al. [11] developed a special system of indicators for evaluating the ERP systems. Shang and Seddon [21] propose a comprehensive framework for assessing the ben- efits of the ERP on five different levels - operational, managerial, strategic, IT infra- structure, organizational. Net benefits are also one of the dimensions of the Delone and McLean model [17] and combine the individual impact and organizational impact [14]. Urbach [18] summarizes the measures of the organizational impact of various stud- ies: change in business processes, competitive advantage, reduced costs, improved communication and cooperation, strengthened coordination, improved output, better decision-making, improved performance, overall productivity, quality, customer satis- faction, management control. One can notice that traditional financial indicators do not play an important role in these models. 2.3 Technological perspective The above-mentioned models relate to either user or managerial perspective of the suc- cess of IS. An important aspect of the system is also its technological sophistication. Edited by S. Kowalski, P. Bednar and I. Bider 3 Proceedings of STPIS'16 Shang and Seddon [21] describe IT infrastructure as one of the dimensions of the ben- efits of implementing an ERP system by improving business flexibility, reducing IT costs and increasing capacity. Seddera, Gable and Chan [4] refer to the technical staff (IT professionals) as one of the stakeholders of ERP systems. Usually, technologically sophisticated solutions are those, which comply with the standards and use latest technologies. Key features of quality software are described by ISO / IEC 25010: 2011 standard [29]. Quality ERP system should therefore have these qualities, but non-compliance with the above standards does not necessarily mean that the system in inadequate for the organization. The models studying the technological perspective of ERP success typically focus only on the evaluation of IS as a whole, which forces users to deliver average ratings and prevents the identification of critical parts of the system. Gefen and Ragowsky highlighted the importance of the evaluation of the ERP system by parts in their study [30]. They proposed evaluation of the individual modules, as these are consistent with the business activities/functions, which are mutually different. The nature of each ac- tivity should also have a significant impact on the perceived value of the module. For a comprehensive evaluation of the IS system we therefore have to consider the system as a multitude of diverse components used by the stakeholders with very differ- ent interests. Such “per partes” approach to the evaluation already exists in the field of software development methodologies (SDM). The SDM model, proposed by Vavpotič and Bajec [31] simultaneously addresses both the aspect of user acceptance as well as the aspect of technological efficiency of methodologies. The model was later additionally im- proved by adding a dimension which measures the impact of SDM on the organization [32]. 3 The Proposed Model The proposed model was designed taking into account the two key features of ERP systems. These two features are: a variety of business functions to be supported by the ERP system and a multitude of stakeholders of the system. The purpose of the model is to enable the organization to easily recognize the critical functionalities of the system that need improvement and to identify the satisfaction of all key stakeholders with key functionalities. In the proposed approach, the ERP system is simultaneously evaluated from three perspectives of success, each representing a view of a key stakeholder: user, managerial and technological perspective. Since each user typically does not use the majority of functionalities offered within a single ERP module, we increase the granularity of the evaluation from the level of individual modules as suggested by the literature to the level of individual functionalities which support individual business processes. We speculate that such approach will significantly increase the accuracy and exactness of the evaluation, since it will enable users to accurately identify the good and bad func- tionalities of each module. Thus the model enables managers to gain detailed and sys- tematic insights into the strengths and weaknesses of all the functionalities of the ERP ©Copyright held by the author(s) 4 Proceedings of STPIS'16 system, without being forced to rely on average assessment for the entire module or system when making decisions concerning the ERP system. End-users evaluate the functionalities from the user perspective, middle manage- ment from managerial perspective and IT employees (IT experts) from technological perspective. After completing the evaluation, functions are divided into eight groups in relation to the ratings in each perspective (U -user, T - technological, M - managerial), which may be lower (low - L) or higher (high - H) than the mean value of a perspective. The eight groups are presented in Table 1. Table 1. Improvement strategies based on evaluation of the three perspectives U T M Proposed actions and strategies L L L With functionalities that have below mean values in all measured per- spectives of ERP success, we must consider whether they are neces- sary to deploy at all and if investing in them is justified or can they be replaced with a more appropriate solution. L L H Because these functionalities have in the opinion of management an above mean impact on net benefits, we need to improve the user satis- faction of these features and perform appropriate technology updates. H L L Such functionalities should be modified in a way that they will be tech- nologically more suitable or replaced with functionalities that provide a similar user satisfaction (e.g. are not demanding to use), but are tech- nologically suitable. Prior to this, we have to check whether the changes are economically justified, given that the functionality does not provide above-average net benefits to the enterprise. L H L We must make these functionalities more user friendly while also con- sidering whether the changes are economically justified, given that the functionality does not provide above-average net benefits. H L H Modifications or replacement of such functionalities should be consid- ered so that they become technologically suitable while providing a similar user experience (e.g. are not demanding to learn and use for user already familiar to older functionalities). L H H With these functionalities it is necessary to find the reasons for the dis- satisfaction of end users. This may be due to the fact that users do not have sufficient knowledge and experience in using the selected func- tionality, so we must provide them assistance and training, and present to them the benefits arising from the use of these functionalities. H H L We should analyse if functionalities that do not deliver important net benefits to the enterprise must remain in use. Such functionalities are often administrative functionalities that must be used regardless of the fact that they do not bring a significant economic benefit. H H H It is necessary to ensure that these functionalities remain in this group. Technological suitability can be provided by updates but the updates should not reduce user satisfaction. Edited by S. Kowalski, P. Bednar and I. Bider 5 Proceedings of STPIS'16 3.1 Characteristics of perspectives Since the use of the ERP system is mandatory, we decided to measure user satisfaction instead of adoption in order to evaluate the system from user perspective. Based on existing studies [2], [6], [17] we defined the characteristics of the user/social perspec- tive as follows. Satisfaction with information - satisfaction with the outputs (on-screen displays and various reports) generated by the ERP system and their value in terms of usability and relevance to the user [4]. We define features of information quality in line with the relevant literature [2], [14], [26]. Satisfaction with the system - satisfaction with the system from a technical and design point of view [4] and also from the per- spective of system quality [14]. Satisfaction with the support - satisfaction with the support provided to the users by the IT department and external collaborators who are responsible for support [14]. Impact on the user - how the system affects the individual user [4] in terms of efficiency, productivity, performance and simplification of work. The managerial perspective focuses on alignment with the business strategy and business processes. Regarding the alignment with the business strategy the evaluation focuses mainly on how the system under evaluation affects the costs in case of cost leadership strategy and the improvement of the quality of products in case of differen- tiation strategy [12], [21]. The evaluation of the alignment with business processes fo- cuses on how well the system under assessment supports specific business processes and if it introduces any advanced solutions from the perspective of business processes. The basis for the assessment of the ERP system from the technological perspective is its technological suitability [29] for: the organization, employees and their way of working. We measure technological suitability of ERP for each of the functionalities and consequently the business process, which the functionality supports. 4 Case study 4.1 Methodology and case description To test the proposed evaluation model we followed the case study research protocol as defined by Yin [33]. We developed three different questionnaires, each covering a dif- ferent perspective and intended for different group of survey participants: ERP users, IT experts and management (see Appendix A). All questions used a seven-point Likert scale between 7 (strongly agree) and 1 (strongly disagree), where 4 meant neither agree- ment or disagreement with the statement. The results of the survey were analysed and discussed with the management. The case study was carried out in a Slovenian company, which is engaged in design- ing concepts, plans and tools for serial manufacturers of sailboats and motor yachts as well as its own production of vessels. In 2013, the company employed around 150 peo- ple and sold 110 hybrid vessels. The company started with implementation of ERP system based on MS technologies in 2009. The system replaced the existing system, from which the data was transferred. The first step was to identify the functionalities provided by the system within dif- ferent modules, and to identify the users of these functionalities who can evaluate them. ©Copyright held by the author(s) 6 Proceedings of STPIS'16 We have identified 42 functionalities within seven modules - Financial management (10), Sales and marketing (5), Supply chain management (5), Warehouse management (4), Manufacturing (5), Human resource management (10), Administration (3). The proposed model was used to evaluate the above 42 functionalities. The 24 em- ployees who use the system have evaluated it from the user perspective. Each user eval- uated only the functionalities that he actually uses. Heads of departments further eval- uated functionalities from the managerial point of view. There were 10 evaluators and they evaluated the functionalities that are part of the module which is dominantly used in the department they run. Head of IT evaluated all of the functionalities from techno- logical perspective. 4.2 Results and discussion Average evaluation scores were the highest for the user dimension as presented in Table 2. Satisfaction with support received the best average score. Managerial and technolog- ical dimension received a slightly lower average score, but still higher than 4, which represents a neutral level. A high standard deviation in the managerial aspect is due to low evaluation scores of functionalities within module Sales and marketing. Only the answer to the question about the alignment with the business processes was taken into account, as the manager was not able to respond adequately to the two other questions. Table 2. Average values and standard deviations Perspective Characteristic Average value Standard devi- Cronbach ation alpha user Satisfaction with 5.122 1.083 information Satisfaction with 5.122 1.136 the system 5.255 0.881 0.843 Satisfaction with 5.564 0.701 the support Impact on the in- 5.212 1.270 dividual managerial Impact on quality 5.090 1.343 of products Impact on lower 5.131 4.655 1.428 1.830 0.967 costs Business process 4.722 1.827 alignment technological Technological 4.191 4.191 1.065 1.065 - suitability Evaluation scores for user and managerial perspective of the functionalities are shown on scatter plot in Figure 1. The figure shows that the evaluation on the level of Edited by S. Kowalski, P. Bednar and I. Bider 7 Proceedings of STPIS'16 the individual functionalities instead of modules proved to be adequate, as the function- alities of a particular module are distributed over the entire plot, which means that their evaluation scores vary considerably. The majority of the evaluated functionalities are in the first quadrant, which means that they are suitable both from a user perspective (users are satisfied with functionali- ties), as well as from the managerial perspective (managers estimate that the function- alities help increase competitive advantage and support business process well). A good score does not mean that they should no longer be monitored, as it is crucial for these functionalities to stay where they are, or ideally to move further towards the top right corner of the plot. Fig. 1. Scatter plot showing the results of the evaluation In the third quadrant are the functionalities which neither users nor managers are satisfied with. The score of the Inventory management from the Financial management module (C) is especially interesting. This functionality deviates significantly from the others in this module, with which users are extremely satisfied and management eval- uates them as economically beneficial. According to the comments of the evaluators, low score is arising from difficulties in the recording of unfinished production, which is not tailored sufficiently enough to the nature of production. Users also highlight that the calculation of the value of unfinished production is very complex and incompre- hensible to the users. The fact that three out of four functionalities of the Sales and marketing module are in the third quadrant (D, E, F, G) is worrying, since the sales ©Copyright held by the author(s) 8 Proceedings of STPIS'16 process is one of the most important for company performance. Users here exposed the inadequacy of the ERP system for the nautical industry and its inability to deal with widely different contracts that often significantly change after they are signed. For these reasons, the use of Sales and marketing module is limited and employees are forced to use other tools to support the sales process. Second and fourth quadrant are particularly interesting for the analysis, as they in- clude functionalities with which users are satisfied, but management evaluates that they do not bring much to the competitive advantage (second quadrant) or vice versa (fourth quadrant). Most functionalities in the second quadrant are related to the supporting pro- cesses (Financial management and Human resource management modules). Therefore, lower evaluation score regarding the economic benefits was anticipated. The fourth quadrant contains functionalities, which significantly contribute to the added value created, however users are dissatisfied with them and are reluctant to use them. These are mostly the functionalities which support some of the core business processes. The functionality which stands out the most is Stock from the Supply chain manage- ment module (A), which was given the highest rating by the management, while having relatively low user rating. User dissatisfaction with Stock (A) as well as with Procure- ment planning (B) is due to the fact that it takes a lot of manual work even though the system allows planning of materials. The effective application of MRP proved to be very challenging because of the specific field and way of work. Since the production is not serial, automatic data processing and creation of orders depending on the state of stock is very difficult. The bill of materials (BOM) cannot be defined in advance as each product is customized. The key for the proper functioning of the MRP is the in- tegrity, accuracy and timeliness of the input data and a close cooperation of production, purchase, storage and sales department. One of the evaluators pointed out that users usually expect more from the system, than they are willing to invest in it. All function- alities from the Manufacturing module were evaluated as technologically inadequate. They are intended namely for the needs of the classical serial production, and are thus functionally inadequate. One of the end users said that it is important to know from the beginning exactly what we expect from the system as any subsequent changes of the settings represent a major undertaking from both the financial and time perspective. The functionality associated with the stocks and their valuation (control) also received low evaluation score for the technological perspective. This is due to the complexity of operations and big amount of data in the database which is causing slow performance of the entire system when in use, so we can conclude that their implementation is inef- fective. Use of this functionality is therefore only possible when other employees do not use the system, otherwise other users are not able to work normally due to the un- responsiveness of the system. Occasional unresponsiveness was exposed as a weakness and the source of dissatisfaction by the end-users as well. Non-compliance with the process is the reason for the low technological scores in the functionalities from the Sales and marketing module. Furthermore, the problem is the lack of portability of the system. The system only works on Microsoft Windows operating systems. Other features of quality software are realized in line with the needs of the company. The system is compatible (allows export to other tools for editing), useful, which was Edited by S. Kowalski, P. Bednar and I. Bider 9 Proceedings of STPIS'16 also exposed by the end-users, reliable and safe. Modular structure of the system allows simple maintenance and testing in the special testing environment. 4.3 Discussion of the results with management We turned to the director of the company for the commentary of the results of the eval- uation. He said that the results confirmed his fears about the areas that represent the greatest difficulty in utilization of the system, while he was surprised by the relatively high user satisfaction scores. This new information led him to rethink his position that the system is being underutilized according to the possibilities that it offers by the users. He will push technical management to increase the number of opportunities for ERP use. He believes that the use of the MRP would not be so difficult if they would take better care of the accuracy of the input data, as we have already mentioned in the anal- ysis of the results. He presumes that the mistake they made was entering the data into the system even during the development of the product instead of after the completion of the development, when the greater part of the BOM is already defined. To allow customization of products, but not cause problems because of changes in BOM, he pro- poses the introduction of a fundamental, base BOM, and optional BOMs for the pre- defined packages of equipment that the customer can select. As essential he also points out the definition of the timeline and all the steps in the production process. He sees the evaluation as highly beneficial providing important managerial infor- mation. The results will be used before the implementation of the upgraded system at the new location. The results will help them avoid making the mistakes they made in the past as well as knowing which functionalities they must be vigilant about when educating users. 5 Conclusion Nowadays, the work in the majority of organizations is supported by information sys- tems (IS) to such an extent that the organizations’ performance depends on the IS. This demonstrates a strong need for a systematic evaluation of the ERP systems. Since each user typically does not use the majority of functionalities offered within a single ERP module, we increased the granularity of the evaluation from the level of individual modules as suggested by the literature to the level of individual functionalities which support individual business processes. We show that such approach significantly im- proves the accuracy and exactness of the evaluation, since it enables users to accurately identify the good and bad functionalities of each module. Thus the model enables man- agers to gain detailed and systematic insights into the strengths and weaknesses of all the functionalities of the ERP system, without being forced to rely on average assess- ment for the entire module or system when making decisions concerning the ERP sys- tem. The results confirmed the usefulness of the evaluation model, especially its ability to provide specific new information to management on the influence of the individual functionalities on ERP success. ©Copyright held by the author(s) 10 Proceedings of STPIS'16 Limitations of the paper are as follows. A single case study was conducted thus cau- tion is needed when generalizing its results. Other organizational settings, such as big- ger enterprises might pose additional limitations for the proposed model as it might become necessary to conduct the evaluation in several steps due to large number of functionalities and large number of users. Additional research on these issues would be valuable. Additionally, the interrelation of the three perspectives should be considered as a possible extension of the proposed model in further research. References 1. Ehie, I. C., Madsen, M.: Identifying critical issues in enterprise resource planning (ERP) implementation. Computers in industry 56(6), 545-557 (2005) 2. Gable, G., Sedera, D., Chan, T.: Enterprise systems success: a measurement model. In : Proceedings Twenty-Fourth International Conference on Information Systems, pp.576- 591 (2003) 3. Gable, G., Sedera, D., Chan, T.: Re-conceptualizing information system success: the IS- impact measurement model. Journal of the Association for Information Systems 9(4), 377- 408 (2008) 4. Sedera, D., Gable, G., Chan, T.: Measuring enterprise systems success: the importance of a multiple stakeholder perspective. ECIS 2004 Proceedings, 100 (2004) 5. Mumford, M.: The story of socio-technical design: reflections on its successes, failures and potential. Information Systems Journal 16(4), 317–342 (2006) 6. Petter, S., DeLone, W., McLean, E.: Measuring information systems success: models, di- mensions, measures, and interrelationships. European Journal of Information Systems 17(3), 236-263 (2008) 7. Mather, D., Caputi, P., Jayasuriya, R.: Is the Technology Acceptance Model a Valid Model of User Satisfaction of Information Technology in Environments where Usage is Mandatory? In : ACIS 2002 Proceedings, pp.1241-1250 (2002) 8. Rogers, E. M.: Diffusion of innovations. Simon and Schuster, (2010) 9. Davis, F. D., Bagozzi, R. P., Warshaw, P. R.: User acceptance of computer technology: a comparison of two theoretical models. Management science 35(8), 982-1003 (1989) 10. Fishbein, M., Ajzen, I.: Belief, attitude, intention and behavior: An introduction to theory and research. Addison-Wesley, Massachusetts (1975) 11. Chand, D., Hachey, G., Hunton, J., Owhoso, V., Vasudevan, S.: A balanced scorecard based framework for assessing the strategic impacts of ERP systems. Computers in indus- try 56(6), 558-572 (2005) 12. Porter, M. E. Competitive strategy: Techniques for analyzing industries and competitors. Simon and Schuster. (2008) 13. Rawstorne, P., Jayasuriya, R., Caputi, P.: Issues in predicting and explaining usage be- haviors with the technology acceptance model and the theory of planned behavior when usage is mandatory. In : Proceedings of the twenty first international conference on Infor- mation systems, pp.35-44 (2000) Edited by S. Kowalski, P. Bednar and I. Bider 11 Proceedings of STPIS'16 14. DeLone, W. H., McLean, E. R.: Information systems success: the quest for the dependent variable. Information systems research 3(1), 60-95 (1992) 15. Koh, C. E., Prybutok, V. R., Ryan, S. D., Wu, Y.: A model for mandatory use of software technologies: An integrative approach by applying multiple levels of abstraction of in- forming science. Informing Science: the International Journal of an Emerging Transdisci- pline 13, 177-203 (2010) 16. Venkatesh, V., Davis, F. D.: A theoretical extension of the technology acceptance model: four longitudinal field studies. Management science 46(2), 186-204 (2000) 17. DeLone, W. H., McLean, E. R.: The DeLone and McLean model of information systems success: a ten-year update. Journal of management information systems 19(4), 9-30 (2003) 18. Urbach, N., Müller, B.: The updated DeLone and McLean model of information systems success. In : Information Systems Theory: Explaining and Predicting Our Digital Society, pp. 1-18. Springer New York (2012) 19. Ives, B., Olson, M. H., Baroudi, J. J.: The measurement of user information satisfaction. Communications of the ACM 26(10), 785-793 (1983) 20. Doll, W. J., Torkzadeh, G.: The measurement of end-user computing satisfaction. MIS quarterly, 259-274 (1988) 21. Shang, S., Seddon, P. B.: Assessing and managing the benefits of enterprise systems: the business manager's perspective. Information Systems Journal 12(4), 271-299 (2002) 22. Calisir, F., Calisir, F.: The relation of interface usability characteristics, perceived useful- ness, and perceived ease of use to end-user satisfaction with enterprise resource planning (ERP) systems. Computers in Human Behavior 20(4), 505-515 (2004) 23. Wu, J.-H., Wang, Y.-M.: Measuring ERP success: the ultimate users' view. International Journal of Operations & Production Management 26(8), 882-903 (2006) 24. Seddon, P., Kiew, M.: A Partial Test and Development of the DeLone and McLean Model of IS Success. In : Proceedings of the Fifteenth International Conference on Information Systems, pp.99-110 (1995) 25. Doll, W. J., Torkzadeh, G.: A discrepancy model of end-user computing involvement. Management Science 35(10), 1151-1171 (1989) 26. Bailey, J. E., Pearson, S. W.: Development of a tool for measuring and analyzing computer user satisfaction. Management science 29(5), 530-545 (1983) 27. Martinsons, M., Davison, R., Tse, D.: The balanced scorecard: a foundation for the stra- tegic management of information systems. Decision support systems 25(1), 71-88 (1999) 28. Kaplan, R. S., Norton, D. P.: Using the balanced scorecard as a strategic management system. (1996) 29. ISO, IEC 25010:2011: Systems and Software Engineering—Systems and Software Qual- ity Requirements and Evaluation (SQuaRE)—System and Software Quality Models. (2011) 30. Gefen, D., Ragowsky, A.: A multi-level approach to measuring the benefits of an ERP system in manufacturing firms. Information Systems Management 22(1), 18-25 (2005) 31. Vavpotic, D., Bajec, M.: An approach for concurrent evaluation of technical and social aspects of software development methodologies. Information and software technology 51(2), 528-545 (2009) ©Copyright held by the author(s) 12 Proceedings of STPIS'16 32. Vavpotic, D., Hovelja, T.: Improving the evaluation of software development methodol- ogy adoption and its impact on enterprise performance. Computer Science and Infor- mation Systems 9(1), 165-187 (2012) 33. Yin, R. K.: Case study research: design and methods. 4th ed. Sage Publications, Thousand Oaks, Calif., 240. (2008) Edited by S. Kowalski, P. Bednar and I. Bider 13 Proceedings of STPIS'16 Appendix A User perspective 1. Satisfaction with information provided: The information that I get when using are relevant (complete and useful), concise, accurate and easy to understand. 2. Satisfaction with the system: is reliable, accessible, flexible and easy to use. 3. Satisfaction with the support: IT department and external consultants are accessible and responsive, their support for is reliable and easy to understand. 4. Impact on the individual: The use of facilitates my work and improves my work performance and productivity. Managerial perspective 1. Impact on the quality of products: significantly contributes to the added value created through in- creasing the quality of products. 2. Impact on lower costs: significantly contributes to the added value created through cost reduction. 3. Business process alignment: supports business processes well and introduces advanced solu- tions. Technological perspective 1. Technological suitability: is technologically suitable for the company ©Copyright held by the author(s) 14