Considering previous theoretical models and empirical studies, this study's goal is to develop a tool for assessing the success of a web-based Information System and to evaluate it experimentally. This is Audits, a system used by a non-profitable public organization, O.P.E.K.E.P.E. Success is evaluated based on system's effects on the employees and the organization, as well as the satisfaction of the end users. At the same time, this study explores the factors that facilitate or undermine the success of an IS according to data gathered from the technology acceptance related literature. The proposed model has been tested using primary data from a sample of 192 regular users of the specific system, who actually represent almost two fifths of its total active users. The empirical results only partly verify the relationships examined and contribute in the design of a valid and reliable conceptual model.
This study attempts to address the issue of developing and theoretically substantiating a model that assesses the success of a particular IS used by a Hellenic organization, the Payment and Control Agency for Guidance and Guarantee Community Aid (O.P.E.K.E.P.E.). The explored model is a creative research composition of recognized theoretical models and as such, there is no precedent of a study having the same structure. Its originality consists of the combination of factors incorporated from models that study the acceptance and use of technology and those that constitute the multi-dimensional concept of the success of IS. Moreover, this study and the proposed theoretical framework allows for the evaluation of the success of the specific IS.
Payment and Control Agency for Guidance and Guarantee Community Aid (O.P.E.K.E.P.E.) is a legal entity governed by private law under the control of the European Commission and the European Court of Auditors. Its scope is to pay in time, properly and in a transparent manner the agricultural aid granted by the European Union for the agricultural sector and amounts approximately to 4 billion Euros annually. O.P.E.K.E.P.E. performs administrative and on-site controls on random or on the entirety of the applications submitted by the potential beneficiaries before depositing the money in their bank accounts. The IS called “Audits” is a system having as its principal objective to administer, coordinate and supervise the audits of the organization that are the core of its functions. “Audits” facilitates the operating automation which results in saving valuable working hours, along with the support of decision making by the members and the administration of O.P.E.K.E.P.E.
Kim et al. (2003) consider IS success as the extent of the improvement of the stakeholders’ position according to the person assessing the IS. Au et al. (2002) state that it would be ideal to assess the effectiveness of an IS based on objective criteria, for example cost-profit analysis. To the contrary, this approach is criticized since it is difficult to determine whether certain profits result entirely from using the IS.
Due to the multi-dimensional nature of the IS success, the first attempts to study it as a one-dimensional success were inadequate. DeLone and McLean (1992) reviewed in depth previous theoretical and empirical studies and developed in an assessment model (D&M) that recognizes six factors that constitute the IS success. Seddon (1997) claimed that the original D&M model was confusing as far as the use of IS factor is concerned and, therefore, suggested splitting it in two parts. One part was about the IS success and the other about the IS use, which he defined as an opinion and behavior rather than a way to assess the IS success.
DeLone and McLean (2003) revised their initial model by adding the quality of services dimension, while removing the two dimensions concerning the effects of IS on the users and the organization and replacing them by a wider dimension (net profits from using the IS). Since this dimension is rather vague, considering the specific system it can comprise several groups of interested parties and, thus, be more flexible. Further, they have also added the concept of intention to use that might replace the actual use, where appropriate, when use of the IS is obligatory. Following a similar philosophy, Gable et al. (2008) attempted to redefine the notion of IS success as a multi-dimensional set of factors. Their model known as IS-Impact, suggests that the future IS impact which is related to the expectations arises from the quality of the system and the information.
Moreover, Wixom and Todd (2005) claim that IS success shall be assessed based on two principal stands of research. The one has to do with users’ satisfaction and the other with users’ acceptance of technology. However, although these two approaches have been studied in parallel, not even a single attempt to correlate them is reported in the literature.
As far as the users’ acceptance of technology is concerned, several theoretical
models have been developed based mainly, in addition to information technology
factors, on the sciences of psychology and sociology
Doll and Torkzadeh (1988) contributed in the field of user’s satisfaction by developing the EUCS (End User Computing Satisfaction) that approaches satisfaction through five factors: content, accuracy, form, ease of use and timeliness. Needless to say, these dimensions overlap with factors that are considered for the assessment of other dimensions of the IS success.
The review of several previous empirical studies concluded that there is not a clear and restrictive framework regarding the conceptual definition of each variable concerning IS success. On the contrary, several of the variables, and the way most scholars tend to approach them, appear to overlap, making it difficult to compare the results of different studies. It is worth mentioning that based on the statistics provided by Gable et al. (2008), the review of sixteen studies that focus on the dimension of the user’s satisfaction has shown that it has been assessed by reference to data that overlap at a rate of 98% with data that have been used in other studies to assess other dimensions of the IS success.
This study suggests a model for assessing the IS success which is based on previous success models, models for the assessment of the user’s satisfaction as well as technology acceptance models. To some extent the proposed model is based on the classification of the DeLone and McLean D&M model (1992), as amended by them (2003) and, at the same time, it evaluates the intermediary factors that facilitate, or undermine the IS success.
In particular, the proposed model adopts all three quality dimensions of the D&M, i.e. system quality, information quality and service quality, which can be broadly viewed as the set of characteristics of the IS and its services. In the authors’ view, these characteristics do not constitute clear standards of the IS success, given that technical appropriateness, informative sufficiency and high-level service quality are not “sine qua non” conditions for the IS success since, according to Doll and Torkzadeh (1988), reduced users’ satisfaction can turn a technically robust system into a failure.
Moreover, from the Thompson’s et al. (1991) MPCU model, the dimension of top management support, as well as the complexity of the IS as seen by the users, have been utilized in the proposed model. From the Moore and Benbasat’s (1996) model the dimension of the IS compatibility to the characteristics of the users and their already existing habits has been adopted. Further, the dimension of the perceived behavioral control on behalf of the user has been adopted from the Taylor and Todd study (1991). All the above mentioned dimensions are integrated in the proposed model while examining their connection to the satisfaction of the users along with the IS impact on the employees.
Finally, the impact on the employees and the organization, as well as users’ satisfaction, as integrated in the initial D&M model, are studied and adopted as the success dimensions of the IS. In comparison to the previously mentioned theoretical models, there has been no consideration of the dimensions regarding the actual use or the intention to use the IS. As for users’ satisfaction, the emphasis in this research is on its psychological aspects that relate significantly to the pre-existing attitudes and emotions of the user towards the IS. By choosing this approach and documenting it adequately, it is ensured that any possible overlapping of the data with the examined dimensions will be avoided.
Focusing on the characteristics of IS (information quality, system quality, service quality), Gable et al. (2008) claim that system quality depends on the IS efficiency, both on a technical and designing level. The most detailed approach of the system quality concept was performed by Sedera and Gable (2004), who acknowledge the following variables in respect of quality: ease of use, ease of learning, users’ requirements, system accuracy, flexibility, intelligence and adaptability. Urbach and Müller (2012) define information quality as the desired information characteristics produced by the IS. Byrd et al. (2006) consider as quality standards the timeliness, accuracy, reliability, relevance and completeness of the information. Furthermore, according to Grüter et al. (2010), the concept of service quality embraces all services provided to the users. Moreover, it embraces the services that are provided indirectly through the provision of customized content in real time for the user.
Top management’s support is set out to be the intervention and participation of the
executive and strategic members of the organization in the functions that relate to the
IS
Finally, within the scope of the Thompson’s et al. (1991) MPCU model, complexity is associated with the extent to which users think that it is difficult to understand or use the system. Lin and Shao (2000) acknowledge that complexity affects greatly users’ participation which, in turn, impacts system’s use positively.
A third group of factors that are examined within the current study, concerns the
IS success factors of the IS. More specifically, end user satisfaction, individual and
organizational impact. Doll and Torkzadeh (1988) define end user satisfaction as the
positive attitude of a person towards a specific technological application when
directly interacting with it. In several cases, scholars tend to integrate in this
dimension factors which constitute a different dimension of the IS success in other
models
The term “impact on employees” is a paraphrase of the original term “impact on
people” in the D&M model
Finally, Gable et al. (2008) argue that the impact of IS on the organization is related to the extent that the IS has improved the performance of the organization, as well as its potentials. They acknowledge three factors which are process improvement, increase of potentials and cost reduction. Sedera and Gable (2004) analysis is similar, although in addition to the above mentioned factors, they study the improvement in productivity as a dimension of the organizational impacts factor.
The study population consists of users of the specific IS who have logged in and used it during the most recent auditing period for the organization (475 persons). For gathering the necessary data, the survey method was chosen and a structured questionnaire (mainly with closed type questions) was used for the collection of the data. The questionnaire consists of ten sections, one for each major factor that is included in the research model. Every section consists of subsections, one for each dimension, while for each separate dimension there is a set of relevant questions (Table 2). Furthermore, several demographic factors were recorded. Apart from the questions concerning the demographic characteristics, the Likert five point rating scale is adopted for answering each question.
In line with previous studies and the assessment of the standards used for the evaluation of the system quality, for the scope of this study questions measuring five dimensions (usability, sophistication, system reliability, accessibility and documentation) have been used. Similarly, information quality is evaluated using questions measuring five dimensions (understandability, completeness, usefulness, timeliness and reliability). Additionally, for the assessment of service quality, the selected variables can be arranged in three dimensions (assurance, responsiveness and empathy); these variables are included in the Parasuranam et al. (1988) SERVQUAL and are similar to the ones also used by Ong et al. (2009) and Gorla et al. (2010). Top management support, perceived control, complexity, compatibility and satisfaction have been measured using one dimension for each one of them.
For the evaluation of individual impact, the variables chosen can be classified in three dimensions (job usefulness, decision effectiveness and personal valuation of IS), while the variables to assess organizational impact can be arranged in four dimensions (organizational performance, business process change, management control and service enhancement). The questionnaire was distributed to three active users of the specific IS, in order to pilot test it and identify any possible ambiguity or problematic issues. The finalized questionnaire was uploaded on Google via Google Forms and was made available for purely anonymous responses. The link has been published at the specific IS home page. Moreover, a personal email was sent to every user of the IS that has a registered email address. It is considered that 359 users were directly informed about the existence of the questionnaire, while 192 of them have completed it (response rate of 54%). In comparison to the population, as previously defined, the response rate is 40%.
From the 192 IS users who participated in this study 122 were women and 70 men. Most of them (80%) belong to the 25 and 44 years old age group and are highly educated (55% has a university degree and 35% has a post graduate degree). Almost half of the participants (45%) are agronomists, while the remaining users are of various specialties. The vast majority (77%) of the users position themselves as highly familiar with computer technology. It can be easily conclude from an overall view of the answers provided to the questions regarding users’ opinion of the IS, the participants have a positive attitude towards it. For the element “I have positive feelings for Audits” the average rate of the responses was 4,88 with 113 of the 192 users chosing scale 5, an indication that shows their positive attitude towards the specific IS and constitutes the higher average rate for a separate question in the study.
For the factors and sub-factors of the model, the Kaiser-Meyer-Olkin (KMO), Total Variance Explained (TVE) and Cronbach a indicators were assessed along with the loading values of each variable for every factor. According to Walker and Maddan (2009), KMO values greater than 0,6 show data suitable for factor analysis. For the Cronbach α indicator, most scholars tend to use the value of 0.7 as a threshold, which is supported by Nunnally’s suggestions (1978, p.278), who assumes that on basic research level, the value of 0.7 is acceptable. Furthermore, Hair et al. (1992) set the value of 0.5 as the minimal acceptable value for the factor loadings of each variable.
The conclusion that emerges from the values of the indicators is that they are within the acceptable limits in all cases and without exceptions. Based on the above mentioned data, it is presumed that the factors examined within the scope of the conceptual model of this study can be assessed with significant reliability based on the data extracted from the participants. 5.2
IS success. Among the rest, the correlations between the E1 factor (management support) and E3 (perceived control) are relatively low, while the highest correlations to the success factors are those of the three quality dimensions and the compatibility factor. The table includes, for illustration purposes only, the variable ISS (Information Systems Success) which has been calculated as the mean of the three success factors (G, H and I).
KMO .793 .720 .699 .856 .719 .702 .764 .746 .739 .784 .731 .693 .658 .724 .665 .662 .925
In order to test and verify the model, the SEM method has been used so as to examine whether the model can interpret the data sufficiently. The assessed relations are those between all the IS quality factors and the regulating conditions and the IS success factors along with the internal relations among the separate dimensions of the major factors, i.e. the system characteristics and the facilitating conditions. The results of this analysis are shown in figure 2.
To assess the model validity, a set of indicators has been calculated: CMIN/DF, GFI, CFI, NFI, RMR and RMSEA (Table 6). The values (of model 1) do not fall within the acceptable limits, although marginally in most cases, therefore it is necessary to further process the model. However, the data are appropriate for testing the individual relations, as they record the correlations with statistical significance.
The covariance analysis shows statistically significant relations among the errors of the independent variables of the model (shown in Figure 2 as well). These relations were expected as these factors constitute hyper-factors. One hyper-factor concerns the characteristics of the system, incorporating the three quality dimensions of the revised D&M model, and the other group the three facilitation conditions that were used. The factors of the model that concern IS success remain separate. The results of the new analysis are shown in figure 3.
The same indicators are calculated for the SEM analysis on the amended model. The values of the indicators are excellent based on what was previously mentioned and, therefore, this amended model can interpret very well the data extracted from the study (user satisfaction 69%, individual impact 65%, organizational impact 74%).
The original hypotheses finally supported from the results of the statistical analysis are the following nine (out of twenty): H1a (System Quality - User satisfaction), H2a (Information Quality - User satisfaction), H2b (Information Quality - Individual Impact), H3c (Service Quality - Organizational Impact), H5a (Compatibility - User satisfaction), H5b (Compatibility - Individual Impact), H5c (Compatibility - Organizational Impact), H8 (User satisfaction - Individual Impact) and H9 (Individual Impact - Organizational Impact). 6
To begin with, given that answers to all questions have a high average rate (Table 3), it can be easily assumed that “Audits” is a successful IS and, in any case, its users have a very positive attitude towards the issues that they were asked to assess.
It has been concluded (Fig. 3) that the two most common IS quality dimensions, i.e. system quality and information quality, with the first being more powerful, have a positive effect on users’ satisfaction (.46 and .22 respectively). It seems that users rate technical capabilities of the IS as more important compared to the quality of information. On the contrary, the third dimension, service quality, has no effect on users’ satisfaction. This conclusion can be assigned to the perceived high quality of the specific IS, as well as to the high level of users’ familiarization to technology.
As for the effects on the employees, it seems that only information quality affects them positively (.17) in a direct manner. Moreover, no evidence supporting the positive relation between the system and information quality and the organizational impact is found. However, a slightly positive (.20) direct relation between service quality and impact on the organization can be noticed. As far as the facilitation conditions are concerned, IS compatibility plays a powerful role and has a positive impact on all three success dimensions; the most powerful is the contribution to users’ satisfaction (.46), followed by the effect / impact to the individuals (.39) and to the organization (.14). On the contrary, there is a very little, but statistically significant, effect of the complexity only to organizational impact (.11). Similarly, the effect of top management support to users’ satisfaction is not supported by the results of this study, which can be explained by the obligatory nature of the IS, which therefore reduce the importance of the role of managers when the specific IS is used. This study however, has verified to the greater extent that satisfaction affects the impact on individuals (.46) and, in turn, it affects organizational impact (.65).
Based on the amended model (Figure 3), it appears that the IS characteristics, in the way these have been defined in the current study, have a strong positive direct impact on users’ satisfaction (.61), while the relation with organizational impact is much weaker (.19). The selected facilitating conditions have similar positive affect on satisfaction (.26) and individual impact (.22). Moreover, a positive sequence of effects is observed within the group of success dimensions. Specifically, user satisfaction influences individual impact in a positive manner (.63) and, in turn, individual has a positive effect on organizational impact (.72).
The approach to combine IS success theories together with Technology Acceptance models is proven rather successful, especially with regard to the factor of complexity based on the value of the relations that were documented empirically.
At actual conditions, this study could be of use to organizations in order to assess internally the IS they use, or the effective selection of a new one focusing to the desired requirements that would result in organizational benefits. As far as the organization of the specific IS, it must emphasize on the development of technically robust systems that will operate without hindrance and any operational difficulties, since it seems that these are the characteristics that affect strongly the success of an IS. Furthermore, it is necessary to ensure that all IS provide to users information useful to their work. Last, but not least, it is evident that the adoption of an IS, in order to process and support the work of the employees and the organization, must be carefully selected and be designed based on the compatibility of the new system to the existing routine and habits of the employees, given that, pursuant to the conclusion of this study, this is the factor that affects all the IS success factors. 6.2
It is possible that some aspects of the IS are assessed by users in such manner that
renders the verification of the proposed model rather difficult or less reliable,
probably because of the “halo effect”
This study does not consider the possible effect of the personal and organizational impact on the factor of the users’ satisfaction. Hence, it would be interesting to study the implication of the possible regenerating nature of the relation between the satisfaction and the impact on the people and the organizations. Moreover, the fact that the use of the IS is compulsory sets another restriction and hinders the generalization of the conclusions. Furthermore, the study of the IS success was focused on users, although, according to Seddon et al. (1999), success concerns other parties as well and, as a result, it would be useful to study this aspect in the future.
Lastly, this study emphasizes on the attitudes and perceptions of the IS users and not on objective assessment standards. To that direction, it would be useful to cross check the results by real data relating to the productivity of the employees along with the performance of the organization regarding the cost of the executed audits.