=Paper=
{{Paper
|id=Vol-1498/HAICTA_2015_paper89
|storemode=property
|title=Assessing the Success of an Information System: the Case of Audits for O.P.E.K.E.P.E.
|pdfUrl=https://ceur-ws.org/Vol-1498/HAICTA_2015_paper89.pdf
|volume=Vol-1498
|dblpUrl=https://dblp.org/rec/conf/haicta/GalanisC15
}}
==Assessing the Success of an Information System: the Case of Audits for O.P.E.K.E.P.E.==
https://ceur-ws.org/Vol-1498/HAICTA_2015_paper89.pdf
Assessing the Success of an Information System: the Case
of Audits for O.P.E.K.E.P.E.
Nikolaos A. Galanis1, Prodromos D. Chatzoglou2
1
Payment and Control Agency for Guidance and Guarantee Community Aid, Greece,
e-mail: nikolaos.galanis@opekepe.gr
2
Production and Management Engineering Department, Democritus University of Thrace,
Xanthi, Greece, e-mail: pchatzog@pme.duth.gr
Abstract. Considering previous theoretical models and empirical studies, this
study’s goal is to develop a tool for assessing the success of a web-based
Information System and to evaluate it experimentally. This is Audits, a system
used by a non-profitable public organization, O.P.E.K.E.P.E. Success is
evaluated based on system’s effects on the employees and the organization, as
well as the satisfaction of the end users. At the same time, this study explores
the factors that facilitate or undermine the success of an IS according to data
gathered from the technology acceptance related literature. The proposed
model has been tested using primary data from a sample of 192 regular users
of the specific system, who actually represent almost two fifths of its total
active users. The empirical results only partly verify the relationships
examined and contribute in the design of a valid and reliable conceptual model.
Keywords: Success of an Information System, O.P.E.K.E.P.E., Satisfaction of
the user, Effects of Information Systems, E-government in Agriculture
1 Introduction
This study attempts to address the issue of developing and theoretically
substantiating a model that assesses the success of a particular IS used by a Hellenic
organization, the Payment and Control Agency for Guidance and Guarantee
Community Aid (O.P.E.K.E.P.E.). The explored model is a creative research
composition of recognized theoretical models and as such, there is no precedent of a
study having the same structure. Its originality consists of the combination of factors
incorporated from models that study the acceptance and use of technology and those
that constitute the multi-dimensional concept of the success of IS. Moreover, this
study and the proposed theoretical framework allows for the evaluation of the
success of the specific IS.
Payment and Control Agency for Guidance and Guarantee Community Aid
(O.P.E.K.E.P.E.) is a legal entity governed by private law under the control of the
European Commission and the European Court of Auditors. Its scope is to pay in
time, properly and in a transparent manner the agricultural aid granted by the
798
European Union for the agricultural sector and amounts approximately to 4 billion
Euros annually. O.P.E.K.E.P.E. performs administrative and on-site controls on
random or on the entirety of the applications submitted by the potential beneficiaries
before depositing the money in their bank accounts. The IS called “Audits” is a
system having as its principal objective to administer, coordinate and supervise the
audits of the organization that are the core of its functions. “Audits” facilitates the
operating automation which results in saving valuable working hours, along with the
support of decision making by the members and the administration of O.P.E.K.E.P.E.
2 Literature Review
Kim et al. (2003) consider IS success as the extent of the improvement of the
stakeholders’ position according to the person assessing the IS. Au et al. (2002) state
that it would be ideal to assess the effectiveness of an IS based on objective criteria,
for example cost-profit analysis. To the contrary, this approach is criticized since it is
difficult to determine whether certain profits result entirely from using the IS.
Due to the multi-dimensional nature of the IS success, the first attempts to study it
as a one-dimensional success were inadequate. DeLone and McLean (1992) reviewed
in depth previous theoretical and empirical studies and developed in an assessment
model (D&M) that recognizes six factors that constitute the IS success. Seddon
(1997) claimed that the original D&M model was confusing as far as the use of IS
factor is concerned and, therefore, suggested splitting it in two parts. One part was
about the IS success and the other about the IS use, which he defined as an opinion
and behavior rather than a way to assess the IS success.
DeLone and McLean (2003) revised their initial model by adding the quality of
services dimension, while removing the two dimensions concerning the effects of IS
on the users and the organization and replacing them by a wider dimension (net
profits from using the IS). Since this dimension is rather vague, considering the
specific system it can comprise several groups of interested parties and, thus, be more
flexible. Further, they have also added the concept of intention to use that might
replace the actual use, where appropriate, when use of the IS is obligatory. Following
a similar philosophy, Gable et al. (2008) attempted to redefine the notion of IS
success as a multi-dimensional set of factors. Their model known as IS-Impact,
suggests that the future IS impact which is related to the expectations arises from the
quality of the system and the information.
Moreover, Wixom and Todd (2005) claim that IS success shall be assessed based
on two principal stands of research. The one has to do with users’ satisfaction and the
other with users’ acceptance of technology. However, although these two approaches
have been studied in parallel, not even a single attempt to correlate them is reported
in the literature.
As far as the users’ acceptance of technology is concerned, several theoretical
models have been developed based mainly, in addition to information technology
factors, on the sciences of psychology and sociology (Venkatesh et al. 2003). A
common example is the Thompson et al. (1991) Model of PC Utilization (MPCU)
799
that calculates the extent of an IS use based on six factors: compatibility, complexity,
long-term effects, emotions, social factors and facilitation conditions.
Doll and Torkzadeh (1988) contributed in the field of user’s satisfaction by
developing the EUCS (End User Computing Satisfaction) that approaches
satisfaction through five factors: content, accuracy, form, ease of use and timeliness.
Needless to say, these dimensions overlap with factors that are considered for the
assessment of other dimensions of the IS success.
The review of several previous empirical studies concluded that there is not a clear
and restrictive framework regarding the conceptual definition of each variable
concerning IS success. On the contrary, several of the variables, and the way most
scholars tend to approach them, appear to overlap, making it difficult to compare the
results of different studies. It is worth mentioning that based on the statistics
provided by Gable et al. (2008), the review of sixteen studies that focus on the
dimension of the user’s satisfaction has shown that it has been assessed by reference
to data that overlap at a rate of 98% with data that have been used in other studies to
assess other dimensions of the IS success.
Table 1. Synopsis of selected empirical studies.
Authors Short description Study effects Support
Cheung & Lee Satisfaction from web- Information Quality !Satisfaction Yes
(2008) based IS System Quality ! Satisfaction Yes
Effect of IS quality on IS Information Quality !Organizational Impact Yes
Gorla et al.
success in terms of System Quality ! Organizational Impact Yes
(2010)
organizational impact Service Quality !Organizational Impact Yes
Information Quality!Usefulness Yes
Floropoulos et Assessment of success of &Satisfaction
Service Quality ! Usefulness & Satisfaction Yes
al. (2010) the greek taxation IS
Usefulness ! Satisfaction Yes
Information Quality !Success Perception Yes
Al-adaileh IS success model on the Usefulness ! Success Perception No
(2009) user’s side Ease of Use ! Success Perception Yes
Management Support ! Success Perception Yes
System Quality ! Use Yes
System Quality ! Satisfaction Yes
Empirical study of the Information Quality ! Use Yes
Halawi et al.
success of a knowledge Information Quality ! Satisfaction Yes
(2007)
management IS Service Quality ! Use No
Service Quality ! Satisfaction Yes
Satisfaction ! Individual Impact Yes
Use !Satisfaction Yes
Empirical evaluation of the
Iivari (2005) Satisfaction !Use Yes
D&M model
Satisfaction ! Individual Impact Yes
Wixom&Watson Empirical research of a System Quality ! Organizational Impact Yes
(2001) storage data IS Information Quality ! Organizational Impact Yes
System Quality ! Satisfaction Yes
Negash et al. Quality and Effectiveness
Information Quality ! Satisfaction Yes
(2003) of web-based IS
Service Quality ! Satisfaction No
System Quality ! Individual Impact No
Byrd et al. IS effect on organizational
Information Quality ! Individual Impact No
(2006) costs
Individual Impact ! Organizational Impact Yes
800
3 Conceptual Framework
3.1 The Research Model
This study suggests a model for assessing the IS success which is based on
previous success models, models for the assessment of the user’s satisfaction as well
as technology acceptance models. To some extent the proposed model is based on the
classification of the DeLone and McLean D&M model (1992), as amended by them
(2003) and, at the same time, it evaluates the intermediary factors that facilitate, or
undermine the IS success.
In particular, the proposed model adopts all three quality dimensions of the D&M,
i.e. system quality, information quality and service quality, which can be broadly
viewed as the set of characteristics of the IS and its services. In the authors’ view,
these characteristics do not constitute clear standards of the IS success, given that
technical appropriateness, informative sufficiency and high-level service quality are
not “sine qua non” conditions for the IS success since, according to Doll and
Torkzadeh (1988), reduced users’ satisfaction can turn a technically robust system
into a failure.
Moreover, from the Thompson’s et al. (1991) MPCU model, the dimension of top
management support, as well as the complexity of the IS as seen by the users, have
been utilized in the proposed model. From the Moore and Benbasat’s (1996) model
the dimension of the IS compatibility to the characteristics of the users and their
already existing habits has been adopted. Further, the dimension of the perceived
behavioral control on behalf of the user has been adopted from the Taylor and Todd
study (1991). All the above mentioned dimensions are integrated in the proposed
model while examining their connection to the satisfaction of the users along with the
IS impact on the employees.
Finally, the impact on the employees and the organization, as well as users’
satisfaction, as integrated in the initial D&M model, are studied and adopted as the
success dimensions of the IS. In comparison to the previously mentioned theoretical
models, there has been no consideration of the dimensions regarding the actual use or
the intention to use the IS. As for users’ satisfaction, the emphasis in this research is
on its psychological aspects that relate significantly to the pre-existing attitudes and
emotions of the user towards the IS. By choosing this approach and documenting it
adequately, it is ensured that any possible overlapping of the data with the examined
dimensions will be avoided.
3.2 Information System Characteristics
Focusing on the characteristics of IS (information quality, system quality, service
quality), Gable et al. (2008) claim that system quality depends on the IS efficiency,
both on a technical and designing level. The most detailed approach of the system
quality concept was performed by Sedera and Gable (2004), who acknowledge the
801
following variables in respect of quality: ease of use, ease of learning, users’
requirements, system accuracy, flexibility, intelligence and adaptability. Urbach and
Müller (2012) define information quality as the desired information characteristics
produced by the IS. Byrd et al. (2006) consider as quality standards the timeliness,
accuracy, reliability, relevance and completeness of the information. Furthermore,
according to Grüter et al. (2010), the concept of service quality embraces all services
provided to the users. Moreover, it embraces the services that are provided indirectly
through the provision of customized content in real time for the user.
3.3 Regulating Factors
Top management’s support is set out to be the intervention and participation of the
executive and strategic members of the organization in the functions that relate to the
IS (Jarvenpaa and Ives, 1991). Further, Moore and Benbasat (1996) understand the
concept of work compatibility as the extent to which the current recipients
understand the system’s innovations as consistent with their existing principles,
values, needs and experiences. Moreover, by implementing the concept of perceived
behavioral control, Taylor and Todd (1991) refer to users’ perceptions regarding the
external and internal obstacles in accepting and using technology that relate to the
available resources and the existing technological background.
Finally, within the scope of the Thompson’s et al. (1991) MPCU model,
complexity is associated with the extent to which users think that it is difficult to
understand or use the system. Lin and Shao (2000) acknowledge that complexity
affects greatly users’ participation which, in turn, impacts system’s use positively.
3.4 Success Factors of the Information System
A third group of factors that are examined within the current study, concerns the
IS success factors of the IS. More specifically, end user satisfaction, individual and
organizational impact. Doll and Torkzadeh (1988) define end user satisfaction as the
positive attitude of a person towards a specific technological application when
directly interacting with it. In several cases, scholars tend to integrate in this
dimension factors which constitute a different dimension of the IS success in other
models (e.g. Ong et al. 2009). As a result, since this study also examines information
quality and system quality as separate dimensions of the IS success, the authors
choose a different approach for measuring end user’s satisfaction. It is defined as the
overall satisfaction of a user as perceived by him based on his psychological and
emotional notions and stands towards the system as a whole. The approach is
consistent not only to the Au’s et al. (2002) proposal, which determines satisfaction
as the extent of the total positive assessment and the degree of pleasure that arises
from the use of the IS but, also, to the Wang’s (2008) study where it is claimed that
users’ satisfaction must be measured in a direct way in order to determine the total
degree of satisfaction, and not indirectly through other factors. Mckinney et al.
(2002) hold the same opinion and state that user’s satisfaction reflects on how
pleased, satisfied, excited and positively disposed he is regarding the system’s use.
802
The term “impact on employees” is a paraphrase of the original term “impact on
people” in the D&M model (DeLone and McLean, 1992). Gable et al. (2008) claim
that the IS impact on people is related to the way it affects their personal capabilities
and their productivity. Hou (2012) includes also the decision making dimension in
the dimension of personal performance.
Finally, Gable et al. (2008) argue that the impact of IS on the organization is
related to the extent that the IS has improved the performance of the organization, as
well as its potentials. They acknowledge three factors which are process
improvement, increase of potentials and cost reduction. Sedera and Gable (2004)
analysis is similar, although in addition to the above mentioned factors, they study
the improvement in productivity as a dimension of the organizational impacts factor.
Fig. 1. The research model.
4 Research Methodology
The study population consists of users of the specific IS who have logged in and
used it during the most recent auditing period for the organization (475 persons). For
gathering the necessary data, the survey method was chosen and a structured
questionnaire (mainly with closed type questions) was used for the collection of the
data. The questionnaire consists of ten sections, one for each major factor that is
included in the research model. Every section consists of subsections, one for each
dimension, while for each separate dimension there is a set of relevant questions
(Table 2). Furthermore, several demographic factors were recorded. Apart from the
questions concerning the demographic characteristics, the Likert five point rating
scale is adopted for answering each question.
In line with previous studies and the assessment of the standards used for the
evaluation of the system quality, for the scope of this study questions measuring five
dimensions (usability, sophistication, system reliability, accessibility and
803
documentation) have been used. Similarly, information quality is evaluated using
questions measuring five dimensions (understandability, completeness, usefulness,
timeliness and reliability). Additionally, for the assessment of service quality, the
selected variables can be arranged in three dimensions (assurance, responsiveness
and empathy); these variables are included in the Parasuranam et al. (1988)
SERVQUAL and are similar to the ones also used by Ong et al. (2009) and Gorla et
al. (2010). Top management support, perceived control, complexity, compatibility
and satisfaction have been measured using one dimension for each one of them.
Table 2. Variables’ sources.
Factor Items Sources
Gable et al.(2008), Cheung & Lee (2012), Grüter et
C1. Usability 7 al. (2010), Gorla et al. (2010), Sedera & Gable
(2004), Zheng et al. (2013), Elling et al. (2012)
C2. Sophistication 3 Gable et al. (2008), Gorla et al. (2010)
C3. Relability 4 Gable et al.(2008), Grüter et al.(2010)
Cheung & Lee (2012), Grüter et al. (2010), Gable et
C4. Accessibility 4
al. (2008), Byrd et al. (2006), Negash et al. (2003)
C5. Documentation 3 Hasan & Abuelrub (2011), Gorla et al. (2010)
D1. Understandability 3 Gable et al. (2008), Cheung και Lee (2012)
D2. Completeness 3 Grüter et al. (2010), Byrd et al. (2006)
Cheung και Lee (2012), Gable et al. (2008), Byrd et
D3. Usefulness 5
al. (2006), Ong et al. (2009)
Byrd et al. (2006), Negash et al. (2003), Hasan &
D4. Timeliness 3
Abuelrub (2011)
Byrd et al. (2006), Cheung & Lee (2012), Negash et
D5. Relability 3
al. (2003), Grüter et al. (2010), Gable et al. (2008)
Β1. Assurance 4 Grüter et al. (2010), Gorla et al. (2010)
Β2. Responsiveness 3 Ong et al. (2009), Gorla et al. (2010)
Β3. Empathy 3 Gorla et al. (2010)
Ε1. Management support 3 Thompson et al. (1991)
Ε2. Compatibilty 3 Moore & Benbasat (1996)
Ε3. Behavioral Control 3 Taylor & Todd (1991)
Ε4. Complexity 3 Thompson et al. (1991)
Xiao & Dasgupta (2002), Ong et al. (2009), Grüter et
G1. User satisfaction 10
al. (2010), Sun & Teng (2012), Wang (2008)
Gable et al. (2008), Hou (2012), Ong et al. (2009),
Η.1 Job usefulness 9
Wu & Wang (2006), Eom (2013), Sun & Teng (2012)
Η2. Decision effectiveness 4 Hou (2012)
Η3. Personal valuation of IS 4 Wang (2008)
I1. Organizational performance 4 Byrd et al. (2006), Gable et al. (2008)
I2. Business Process Change 3 Gable et al. (2008)
I3. Management Control 3 Torkzadeh & Doll (1999), Byrd et al. (2006)
I4. Services Enhancement 3 Gorla et al. (2010)
804
For the evaluation of individual impact, the variables chosen can be classified in
three dimensions (job usefulness, decision effectiveness and personal valuation of
IS), while the variables to assess organizational impact can be arranged in four
dimensions (organizational performance, business process change, management
control and service enhancement). The questionnaire was distributed to three active
users of the specific IS, in order to pilot test it and identify any possible ambiguity or
problematic issues. The finalized questionnaire was uploaded on Google via Google
Forms and was made available for purely anonymous responses. The link has been
published at the specific IS home page. Moreover, a personal email was sent to every
user of the IS that has a registered email address. It is considered that 359 users were
directly informed about the existence of the questionnaire, while 192 of them have
completed it (response rate of 54%). In comparison to the population, as previously
defined, the response rate is 40%.
5 Study Results
From the 192 IS users who participated in this study 122 were women and 70
men. Most of them (80%) belong to the 25 and 44 years old age group and are highly
educated (55% has a university degree and 35% has a post graduate degree). Almost
half of the participants (45%) are agronomists, while the remaining users are of
various specialties. The vast majority (77%) of the users position themselves as
highly familiar with computer technology. It can be easily conclude from an overall
view of the answers provided to the questions regarding users’ opinion of the IS, the
participants have a positive attitude towards it. For the element “I have positive
feelings for Audits” the average rate of the responses was 4,88 with 113 of the 192
users chosing scale 5, an indication that shows their positive attitude towards the
specific IS and constitutes the higher average rate for a separate question in the study.
5.1 Factor Analysis
For the factors and sub-factors of the model, the Kaiser-Meyer-Olkin (KMO),
Total Variance Explained (TVE) and Cronbach a indicators were assessed along with
the loading values of each variable for every factor. According to Walker and
Maddan (2009), KMO values greater than 0,6 show data suitable for factor analysis.
For the Cronbach α indicator, most scholars tend to use the value of 0.7 as a
threshold, which is supported by Nunnally’s suggestions (1978, p.278), who assumes
that on basic research level, the value of 0.7 is acceptable. Furthermore, Hair et al.
(1992) set the value of 0.5 as the minimal acceptable value for the factor loadings of
each variable.
The conclusion that emerges from the values of the indicators is that they are
within the acceptable limits in all cases and without exceptions. Based on the above
mentioned data, it is presumed that the factors examined within the scope of the
conceptual model of this study can be assessed with significant reliability based on
the data extracted from the participants.
805
5.2 Correlation Analysis
Table 5 demonstrates that the factors of the model are greatly correlated. The only
statistically non significant relation is the one between factors E1: Top Management
Support and F: Complexity. By focusing exclusively on the correlations between the
dimensions of quality and regulating conditions and the factors of success, it is
observed that the factor of complexity (F) is less correlated to the three factors of the
IS success. Among the rest, the correlations between the E1 factor (management
support) and E3 (perceived control) are relatively low, while the highest correlations
to the success factors are those of the three quality dimensions and the compatibility
factor. The table includes, for illustration purposes only, the variable ISS
(Information Systems Success) which has been calculated as the mean of the three
success factors (G, H and I).
Table 3. Factor analysis and reliability testing of dimensions.
Dimen Ques- Factor Cron-
Factors Mean St.D KMO TVE
-sions tions Loadings bach a
B: Service Β.1 1-4 4,37 .627 .793 61.903 .775 - .800 .788
Β.2 1-3 4,50 .609 .720 76.310 .858 - .896 .841
Quality
Β.3 1-3 4,39 .660 .699 78.137 .833 - .920 .858
C.1 1-7 4,53 .590 .856 67.658 .710 - .876 .911
C: System C.2 1-3 4,40 .579 .719 73.972 .849 - .871 .824
Quality C.3 1-4 4,34 .682 .702 68.450 .778 - .860 .844
C.4 1-4 4,49 .644 .764 73.481 .833 - .883 .878
D.1 1-3 4,58 .602 .746 85.900 .905 - .943 .917
D: Information D.2 1-3 4,31 .637 .739 82.498 .893 - .928 .894
D.3 1-3, 5 4,49 .591 .784 72.308 .745 - .889 .860
Quality
D.4 1-3 4,27 .683 .731 80.232 .870 - .914 .876
D.5 1-3 4,41 .679 .693 84.032 .861 - .954 .904
E: Facilitating Ε.1 1-3 4,27 .741 .658 72.642 .802 - .909 .806
Ε.2 1-3 4,32 .743 .724 86.005 .886 - .951 .914
Conditions
Ε.3 1-3 4,62 .525 .665 72.500 .776 - .902 .800
F: Complexity F.1 1-3 3,45 1.558 .662 72.798 .793 - .907 .786
G: Satisfaction G.1 2-9 4,40 .624 .925 72.776 .766 - .888 .944
H: Individual Η.1 2-4, 6-9 4,23 .756 .936 74.767 .815 - .906 .950
Η.2 1-4 3,94 .914 .823 85.411 .873 - .939 .943
Impact
Η.3 1-4 4,53 .590 .767 73.278 .827 - .887 .877
I: I.1 1-4 4,45 .703 .806 77.787 .796 - .927 .902
Organizational I.2 1-3 4,30 .701 .703 79.748 .847 - .901 .872
I.3 1-3 4,26 .778 .720 81.430 .864 - .930 .881
Impact
I.4 1-3 4,08 .944 .701 86.788 .884 - .946 .924
Table 4. Factor analysis and reliability testing of factors
Factors KMO TVE Factor Loadings Cronbach alpha
B .755 86.908 .916 - .937 .924
C .785 74.307 .836 - .911 .883
D .861 76.884 .802 - .920 .920
E .607 63.236 .657 - .842 .709
F .662 72.798 .793 - .907 .786
G .925 72.776 .766 - .888 .944
H .698 82.566 .864 - .946 .882
I .813 74.331 .814 - .918 .874
806
Table 5. Results of Spearman analysis
B C D E1 E2 E3 F G H
Β: Service Quality 1
-
C: System Quality ,733 1
,000 -
D: Information Quality ,782 ,845 1
,000 ,000 -
Ε1: Management Support ,421 ,482 ,544 1
,000 ,000 ,000 -
Ε2: Compatibility ,571 ,640 ,698 ,545 1
,000 ,000 ,000 ,000 -
Ε3: Perceived control ,391 ,515 ,474 ,360 ,443 1
,000 ,000 ,000 ,000 ,000 -
F: Complexity ,206 ,270 ,194 ,117 ,229 ,219 1
,004 ,000 ,007 ,108 ,001 ,002 -
G: User satisfaction ,626 ,782 ,768 ,460 ,734 ,457 ,263 1
,000 ,000 ,000 ,000 ,000 ,000 ,000 -
Η: Individual Impact ,560 ,636 ,666 ,461 ,704 ,346 ,187 ,767 1
,000 ,000 ,000 ,000 ,000 ,000 ,009 ,000 -
I: Organizational Impact ,575 ,577 ,636 ,452 ,668 ,340 ,276 ,701 ,816
,000 ,000 ,000 ,000 ,000 ,000 ,000 ,000 ,000
ISS: IS Success ,639 ,716 ,745 ,499 ,766 ,409 ,264 ,882 ,943
,000 ,000 ,000 ,000 ,000 ,000 ,000 ,000 ,000
5.3 Analysis based on the Structural Equation Modeling Method (SEM)
In order to test and verify the model, the SEM method has been used so as to
examine whether the model can interpret the data sufficiently. The assessed relations
are those between all the IS quality factors and the regulating conditions and the IS
success factors along with the internal relations among the separate dimensions of the
major factors, i.e. the system characteristics and the facilitating conditions. The
results of this analysis are shown in figure 2.
To assess the model validity, a set of indicators has been calculated: CMIN/DF,
GFI, CFI, NFI, RMR and RMSEA (Table 6). The values (of model 1) do not fall
within the acceptable limits, although marginally in most cases, therefore it is
necessary to further process the model. However, the data are appropriate for testing
the individual relations, as they record the correlations with statistical significance.
The covariance analysis shows statistically significant relations among the errors
of the independent variables of the model (shown in Figure 2 as well). These
relations were expected as these factors constitute hyper-factors. One hyper-factor
concerns the characteristics of the system, incorporating the three quality dimensions
of the revised D&M model, and the other group the three facilitation conditions that
were used. The factors of the model that concern IS success remain separate. The
results of the new analysis are shown in figure 3.
The same indicators are calculated for the SEM analysis on the amended model.
The values of the indicators are excellent based on what was previously mentioned
and, therefore, this amended model can interpret very well the data extracted from
the study (user satisfaction 69%, individual impact 65%, organizational impact 74%).
807
Fig. 2. SEM analysis on the initial model (model 1).
Fig. 3. The amended model (model 2).
Table 6. SEM analysis indicators – Initial model
CMIN/DF GFI CFI NFI RMR RMSEA
Model 1 8.089 0.864 0.877 0.865 0.136 0.193
Model 2 1.681 0.980 0.994 0,986 0.083 0.06
Accepted Values <2 >0.9 >0.9 >0.95 <0.1 ≤0.07
Baumgartner Hu and Hu and Hu and
Hair et al.
Byrne (1989) and Hombur Bentler Bentler Bentler
(1992)
(1996) (1999) (1999) (1999)
808
The original hypotheses finally supported from the results of the statistical
analysis are the following nine (out of twenty): H1a (System Quality - User
satisfaction), H2a (Information Quality - User satisfaction), H2b (Information
Quality - Individual Impact), H3c (Service Quality - Organizational Impact), H5a
(Compatibility - User satisfaction), H5b (Compatibility - Individual Impact), H5c
(Compatibility - Organizational Impact), H8 (User satisfaction - Individual Impact)
and H9 (Individual Impact - Organizational Impact).
6 Conclusions
To begin with, given that answers to all questions have a high average rate (Table
3), it can be easily assumed that “Audits” is a successful IS and, in any case, its users
have a very positive attitude towards the issues that they were asked to assess.
It has been concluded (Fig. 3) that the two most common IS quality dimensions,
i.e. system quality and information quality, with the first being more powerful, have a
positive effect on users’ satisfaction (.46 and .22 respectively). It seems that users
rate technical capabilities of the IS as more important compared to the quality of
information. On the contrary, the third dimension, service quality, has no effect on
users’ satisfaction. This conclusion can be assigned to the perceived high quality of
the specific IS, as well as to the high level of users’ familiarization to technology.
As for the effects on the employees, it seems that only information quality affects
them positively (.17) in a direct manner. Moreover, no evidence supporting the
positive relation between the system and information quality and the organizational
impact is found. However, a slightly positive (.20) direct relation between service
quality and impact on the organization can be noticed. As far as the facilitation
conditions are concerned, IS compatibility plays a powerful role and has a positive
impact on all three success dimensions; the most powerful is the contribution to
users’ satisfaction (.46), followed by the effect / impact to the individuals (.39) and
to the organization (.14). On the contrary, there is a very little, but statistically
significant, effect of the complexity only to organizational impact (.11). Similarly,
the effect of top management support to users’ satisfaction is not supported by the
results of this study, which can be explained by the obligatory nature of the IS, which
therefore reduce the importance of the role of managers when the specific IS is used.
This study however, has verified to the greater extent that satisfaction affects the
impact on individuals (.46) and, in turn, it affects organizational impact (.65).
Based on the amended model (Figure 3), it appears that the IS characteristics, in
the way these have been defined in the current study, have a strong positive direct
impact on users’ satisfaction (.61), while the relation with organizational impact is
much weaker (.19). The selected facilitating conditions have similar positive affect
on satisfaction (.26) and individual impact (.22). Moreover, a positive sequence of
effects is observed within the group of success dimensions. Specifically, user
satisfaction influences individual impact in a positive manner (.63) and, in turn,
individual has a positive effect on organizational impact (.72).
809
The approach to combine IS success theories together with Technology
Acceptance models is proven rather successful, especially with regard to the factor of
complexity based on the value of the relations that were documented empirically.
6.1 Implications and Practical Impact
At actual conditions, this study could be of use to organizations in order to assess
internally the IS they use, or the effective selection of a new one focusing to the
desired requirements that would result in organizational benefits. As far as the
organization of the specific IS, it must emphasize on the development of technically
robust systems that will operate without hindrance and any operational difficulties,
since it seems that these are the characteristics that affect strongly the success of an
IS. Furthermore, it is necessary to ensure that all IS provide to users information
useful to their work. Last, but not least, it is evident that the adoption of an IS, in
order to process and support the work of the employees and the organization, must be
carefully selected and be designed based on the compatibility of the new system to
the existing routine and habits of the employees, given that, pursuant to the
conclusion of this study, this is the factor that affects all the IS success factors.
6.2 Limitations – Suggestions
It is possible that some aspects of the IS are assessed by users in such manner that
renders the verification of the proposed model rather difficult or less reliable,
probably because of the “halo effect” (Thorndike, 1920). To that aim, it would be
more efficient to assess this model using another IS, in order to extract more useful
and reliable conclusions about its validity.
This study does not consider the possible effect of the personal and organizational
impact on the factor of the users’ satisfaction. Hence, it would be interesting to study
the implication of the possible regenerating nature of the relation between the
satisfaction and the impact on the people and the organizations. Moreover, the fact
that the use of the IS is compulsory sets another restriction and hinders the
generalization of the conclusions. Furthermore, the study of the IS success was
focused on users, although, according to Seddon et al. (1999), success concerns other
parties as well and, as a result, it would be useful to study this aspect in the future.
Lastly, this study emphasizes on the attitudes and perceptions of the IS users and
not on objective assessment standards. To that direction, it would be useful to cross
check the results by real data relating to the productivity of the employees along with
the performance of the organization regarding the cost of the executed audits.
810
References
1. Al-adaileh, R.M. (2009), “An Evaluation of Information Systems Success: A
User Perspective - the Case of Jordan Telecom Group”, European Journal of
Scientific Research, Vol. 37, No. 2, pp. 226-239.
2. Au, Ν., Ngai E. and Cheng, E. (2002), “A critical review of end-user information
system satisfaction research and a new research framework”, Omega, Vol. 30,
No. 6, pp. 451-478.
3. Byrd, T.A., Thrasher, E.H., Lang, T. and Davidson, N.W. (2006), “A process-
oriented perspective of IS success: Examining the impact of IS on operational
cost”, Omega, Vol. 34, No. 2006, pp. 448-460.
4. Byrne, B.M. (1989), A primer of LISREL: Basic applications and programming
for confirmatory factor analytic models. Springer-Verlag, New York.
5. Cheung, C. and Lee, M. (2008), “The Structure of Web-Based Information
Systems Satisfaction: An Application of Confirmatory Factor Analysis”, In
Filipe, J. and Cordeiro, J. (eds), “WEBIST 2007, LNBIP 8”, Springer-Verlag,
Berlin Heidelberg.
6. DeLone, W.H. and McLean, R.E. (2003), “The DeLone and McLean Model of
Information Systems Success: A Ten-Year Update”, Journal of Management
Information Systems, Vol. 19, No. 4, pp. 9-30.
7. DeLone, W.H. and McLean, R.E. (1992), “Information Systems Success: The
Quest for the Dependent Variable”, Information Systems Research, Vol. 3, No. 1,
pp. 60-95.
8. Doll, W., Torkzadeh, G. (1988), “The measurement of end-user computing
satisfaction”, MIS Quarterly, Vol. 12, No. 2, pp. 259-274.
9. Elling, S., Lentz, L., Jong, M. and Bergh, H. (2012), “Measuring the quality of
governmental websites in a controlled versus an online setting with the ‘Website
Evaluation Questionnaire”, Government Information Quarterly, Vol. 29, No. 3,
pp. 383-393.
10. Eom, S. (2013), “Testing the Seddon Model of Information System Success in an
E-Learning Context: Implications for Evaluating DSS”, In J.E Hernández, S. Liu,
B. Delibasic, P. Zarate, F. Dargam and R. Ribeiro (eds), “Decision Support
Systems II – Recent Developments Applied to DSS Network Environments”,
Springer-Verlag, Berlin Heidelberg.
11. Floropoulos, J., Spathis, C., Halvatzis, D. and Tsipouridou, M. (2010),
“Measuring the success of the Greek Taxation Information System”, International
Journal of Information Management, Vol. 30, No. 1, pp. 47-56.
12. Gable, G.G., Sedera, D. and Chan, T. (2008), “Re-conceptualizing Information
System Success: The IS-Impact Measurement Model”, Journal of the Association
for Information Systems, Vol. 9, No. 7, pp. 377-408.
13. Gorla, G., Somers, T.M. and Wong, B. (2010), “Organizational impact of system
quality, information quality, and service quality”, Journal of Strategic
Information Systems, Vol. 19, No. 3, pp. 207-228.
14. Grüter, N., Somers, T.M., Wong, B. and Myrach, T. (2010), “E-Success: An
Instrument for Measuring Website Success”, In Ulrike Gretzel, Rob Law and M.
811
Fuchs (eds), “Information and Communication Technologies in Tourism 2010”,
Springer Vienna.
15. Hair, J.T., Anderson, R.E., Tatham, R.L. and Black, W.C. (1992), Multivariate
Data Analysis with Readings, 3rd ed., Macmillan, New York.
16. Halawi, L.A, McCarthy, R.V. and Aronson, J.E (2007), “An empirical
investigation of knowledge-management systems’ success”, The Journal of
Computer Information Systems, Vol. 48, No. 2, pp. 121-135.
17. Hasan, L. and Abuelrub, E. (2011), “Assessing the quality of web sites”, Applied
Computing and Informatics, Vol 9, No. 1, pp 11-29.
18. Hou, C. (2012), “Examining the effect of user satisfaction on system usage and
individual performance with business intelligence systems: An empirical study of
Taiwan’s electronics industry”, International Journal of Information
Management, Vol. 32, No. 6, pp. 560-573.
19. Hu, L. and Bentler, P.M. (1999), “Cutoff criteria for fit indexes in covariance
structure analysis: Conventional criteria versus new alternatives”, Structural
Equation Modeling, Vol. 6, No.1, pp. 1-55.
20. Iivari, J. (2005), “An Empirical Test of the DeLone-McLean Model of
Information System Success”, The DATA BASE for Advances in Information
Systems, Vol. 36, No. 2, pp. 8-27.
21. Jarvenpaa, S.L. and Ives, B. (1991), “Executive involvement and participation in
the management of information technology”, MIS Quarterly, Vol. 15, No. 2, pp.
205-227.
22. Kim, Y.J, Garrity, E.J. and Sanders, G.L. (2003), “Success Measures of
Information Systems”, In Bidgoli, H., Eom, S.B. and Prestage, A. (eds),
“Encyclopedia of Information Systems, Vol. 4”, Academic Press, San Diego, CA.
23. Lin, W.T. and Shao, B.M. (2000), “The relationship between user participation
and system success: a simultaneous contingency approach”, Information &
Management, Vol. 37, No. 6, pp. 283-295.
24. McKinney, V., Yoon, K. and Zahedi, F. (2002), “The measurement of web-
customer satisfaction: An expectation and disconfirmation approach”,
Information Systems Research, Vol. 13, No. 3, pp. 296-315.
25. Moore, G.C. and Benbasat, I. (1996), “Integrating Diffusion of Innovations and
Theory of Reasoned Action models to predict utilization of information
technology by end-users”, In K. Kautz and J. Pries-Heje (eds), “Diffusion and
Adoption of Information Technology”, Springer-Verlag, US.
26. Negash, E., Ryan, T. and Igbaria, M. (2003), “Quality and effectiveness in Web-
based customer support systems”, Information & Management, Vol. 40, No.
2003, pp. 757-768.
27. Nunnally, J.C. (1978), Psychometric Theory, 2nd ed., McGraw-Hill, New York.
28. Ong, C., Day, M. and Hsu, W. (2009), “The measurement of user satisfaction
with question answering systems”, Information & Management, Vol. 46, No. 7,
pp 397-403.
29. Parasuraman, A., Zeithaml, V.A. and Berry, L.L. (1988), “SERVQUAL: A
multiple item scale for measuring consumer perceptions of service quality”,
Journal of Retailing, Vol. 64, No. 1, pp. 12-40.
812
30. Seddon, P.B., Staples, S., Patnayakuni, R. and Bowtell, M. (1999), “Dimensions
of Information Systems Success”, Communications of the Association for
Information Systems, Vol. 20, No. 2, pp. 1-61.
31. Sedera, D. and Gable, G. (2004), “A factor and structural equation analysis of the
enterprise systems success measurement model”. In: Appelgate, L., Galliers, R.,
DeGross, J.I. (eds), “Proceedings of the (Twenty-Fifth) 25th International
Conference on Information Systems”, Association for Information Systems,
Washington, DC, USA.
32. Sun, J. and Teng, J.T.C. (2012), “Information Systems Use: Construct
conceptualization and scale development”, Computers in Human Behavior, Vol.
28, No. 5, pp. 1564-1574.
33. Taylor, S. and Todd, P.A. (1995), “Understanding Information Technology
Usage: A Test of Competing Models”, Information Systems Research, Vol. 6,
No. 2, pp. 144-176.
34. Thompson, R.L., Higgins, C.A., and Howell, J.M. (1991), “Personal Computing:
Toward a Conceptual Model of Utilization,” MIS Quarterly, Vol. 15, No. 1, pp.
124-143.
35. Thorndike, E.L. (1920), “A constant error in psychological ratings,” Journal of
Applied Psychology, Vol. 4, No. 1, pp. 25-29.
36. Urbach, N. and Müller, B. (2012), “The Updated DeLone and McLean Model of
Information Systems Success”, In Dwivedi, Y.K., Wade, M.R. and Schneberger,
S.L. (eds), “Information Systems Theory: Explaining and Predicting Our Digital
Society”, Springer, New York.
37. Venkatesh, V., Morris, M.G., Davis, G.B. and Davis, F.D. (2003), “User
Acceptance of Information Technology: Toward a Unified View”, Management
Information System Quarterly, Vol, 27, No. 3, pp. 425-478.
38. Wang, Y. (2008), “Assessing e-commerce systems success: a respecification and
validation of the DeLone and McLean model of IS success”, Information System
Journal, Vol 18, No. 5, pp. 529-557.
39. Wixom, B.H. and Todd P.A. (2005), “A theoretical integration of user
satisfaction and technology acceptance”, Information Systems Research, Vol. 16,
No. 1,pp. 85-102.
40. Wixom, B.H. and Watson, H.J. (2001), “An empirical investigation of the factors
affecting data warehousing success”, MIS Quarterly, Vol. 25, No. 1, pp. 17-41.
41. Wu, J.H. and Wang, Y.M. (2006), “Measuring KMS success: A respecification of
the DeLone and McLean’s model”, Information & Management, Vol. 43, No. 6,
pp. 728-739.
42. Zheng, Y. and Zhao, K. and Stylianou, A. (2013), “The impacts of information
quality and system quality on users' continuance intention in information-
exchange virtual communities: An empirical investigation”, Decision Support
Systems, Vol. 56, No. 2013, pp. 513-524.
43. Xiao, L. and Dasgupta, S., “Measurement of User Satisfaction with Web-based
Systems: An Empirical Study”, Americas Conference on Information Systems, 9-
11 August 2002, Dallas, TX.
813