=Paper=
{{Paper
|id=Vol-2762/paper2
|storemode=property
|title=Quality assessment models of information systems and projects: analysis and data based enhancing for GIS projects
|pdfUrl=https://ceur-ws.org/Vol-2762/paper2.pdf
|volume=Vol-2762
|authors=Mark Israel,Vyacheslav Kharchenko
|dblpUrl=https://dblp.org/rec/conf/ictes/IsraelK20
}}
==Quality assessment models of information systems and projects: analysis and data based enhancing for GIS projects==
https://ceur-ws.org/Vol-2762/paper2.pdf
Quality Assessment Models of Information Systems:
Analysis and Data Based Enhancing for GIS Projects
Mark Israela , Vyacheslav Kharchenkob
a
Holon Institute of Technology, Golomb st. 52, Holon, Israel
b
National Aerospace University “Kharkiv Aviation Institute”, Chkalova st. 17, Kharkiv, Ukraine
Abstract
Writing this paper supposed checking the availability of various definitions of quality of the informa-
tion systems or projects on the basis of various factors included into the information project in this
area that are divided into 3 categories: GIS operators, system performance specialists in the GIS field
an GIS developers. Existed standards requirements to quality, software and information systems (IS)
quality models are analyzed. Approaches and techniques to IS quality assessment are discussed con-
sidering specific application domains. In this study we examine whether the approach to the quality
of systems and information projects in the field of GIS is the same as the quality approach of systems
and information projects in other fields. As a result of a study, a model for quality assessment of the IS
for geographic information systems is offered. This model and technique of the assessing is based on
statistical processing of data collected in the course of search described above.
Keywords
Information systems, GIS, Quality factors, Quality metrics, Critical factors success, IS Quality
1. Introduction
1.1. Motivation
The quality of the information systems or projects (IS – Information Systems) is determined
with the use of a number of models that include the quality indicators/factors/characteristics
and are subject to quantification.
The requirement to establish a quality model has been felt by users for the purpose of
quantitative and qualitative evaluating the software quality. The known quality models, which
are most hierarchical models, based on quality criteria and associated metrics. All such models
are categorized into three kinds according to the means by which these models have been
generated.
First one is the theoretical model based on the hypothesis relations among variables. Second
one is the data-driven which are based on statistical analysis. Third is the combined model in
which intuitions are used to determine the basic type of the model and data analysis is used to
determine the constants of the model. Practically in most of the cases the combined model is
adapted [1].
ICT&ES-2020: Information-Communication Technologies & Embedded Systems, November 12, 2020, Mykolaiv, Ukraine
" marki@hit.ac.il (M. Israel); v.kharchenko@csn.khai.edu (V. Kharchenko)
0000-0001-5352-077X (V. Kharchenko)
© 2020 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
Hence, the topic of quality assessment of the information systems or projects is versatile and
has become an object of active discussion in the literature from different points of view.
There is a number of populations for quality assessment of the information systems or
projects. The populations are commonly discussed in the literature as the most detailed models.
These populations differ by accents on various aspects of efficiency of the information systems
or projects: efficiency for user, efficiency of organization and company profitability.
The introduction of GIS in the mapping process has produced a completely new type of user
different from the traditional map user. This situation requires the identification of shared
requirements for a rational and accurate implementation of such systems [2]. Quality of Service
(QoS) based service discovery, popularly studied in traditional Web Services, applies also to
Geospatial Web Services cite3. Geospatial web services have been an active area of research in
the context of geospatial non-interoperability problems. The collaborative effort by the industry
and federal geospatial clearinghouses has focused on the standardization process to mitigate
the non-interoperability problems [3].
1.2. State of the art
The IS literature provides several definitions and measures of IS success. As [4] stated that,
although there are nearly as many measures as there are studies; obviously, there is no ultimate
definition of IS success. The definition of IS success may vary in respect of different types of IS
that yield different benefits for individuals, workgroups and organizations [4].
Software quality model is usually defined as a set of characteristics and relationships between
them, which actually provides the basis for specifying the requirements of quality, and evaluating
quality. A lot of software quality models (SWQM) have been introduced for the last decades [5].
Likewise, the growing impact of the component based software development (CBSD), in
particular, to develop customizable, cost effective, just in time and reusable large scale and
complex software systems. To build complete software solution the main focus is to create high
quality parts and later join them together. One of the most critical processes in CBSD is the
selection of software component as per end user criteria. Quality model plays an important role
in the component selection process [6].
The Total Quality Management (TQM) movement uses the principles of empowerment and
employees’ goals. TQM highlights the need of management commitment in the first place.
Management as well as every employee have to be made aware and adopt a quality policy
and everyone is expected to commit to quality, without resistance. The software development
process is called a life cycle. The production of quality products depends on the quality of the
process. Errors in the early stages of the life cycle usually result in more re-work and are more
difficult to correct and, naturally, even more cost demanding [7].
In recent years, research in the field of quality systems and information projects has tended
towards the insight that models must take into account ancillary parameters such as: Area of
activity of an information system or project, users’ preferences in the same area, which often
stem from the uniqueness of the users in a specific area [8].
A quantitative measure of the degree to which software items possess a given quality attribute
is the quality metric of that software. It is computationally a function whose inputs are software
data (usually the source code) and whose output is a single numerical value that can be inter-
preted as the degree to which the software possesses a given quality attribute. Consequently,
traditional software quality metrics is the equivalent quantifiable measure of software attributes
that are fundamental to structured design without account of object orientation. Hence, software
metrics can measure quality by manually computing and examining the source code or using
their equivalent automated tools [9].
It is evident that in the issue of modelling methods for evaluating the quality of systems or
information projects, additional aspects need to be incorporated. Such as the measurement
time in different development phases, product as well as process-related quality factors, a set
of quality metrics measureable on the different type of artifacts such as document, model and
source code, and finally a specific mechanism to apply dynamic weights to quality factors to
determine their impacts on final quality of a product based on its application domain [10].
While information systems (IS) success models have received much attention among re-
searchers, there is a general scarcity of research conducted to measure the GIS success. This
paper proposes a success model for measuring GIS success by extending and modifying previous
IS success models [11].
The study includes an assessment of the degree of difference in the importance of quality
characteristics in various software projects compared to the importance of quality characteristics
of software projects in the GIS area.
Based on the literature review, it can be determined that the existing models do not give
answer regarding the quality assessment of the IS and, particularly, GIS projects. Such a
statement has many reasons - the topic of quality depends on the area of organization, size of
organization, technological level of employees, purposes of organization, different approaches
to quality, as well as employees’ specialization in the organization.
The quality of information systems can be seen as a linear combination of three aspects:
service quality, software quality and quality of information. Moreover, depending on the life
cycle of the information systems, the importance of specified aspects varies. This article will
not discuss an aspect of information quality.
Thus, the objective of this paper is to verify the conformity of the existing quality models for
general information systems or projects (IS), particularly, for GIS projects [2]. The additional
objective of the study is to offer a model for quality assessment of software projects in the GIS
area which are based on the statistical data processing.
2. Quality Issue of Information Systems or Projects
The information systems quality described using the following settings:
• “Quality is a totality of properties and characteristics of a product or service ensuring the
opportunity to meet the necessary requirements... “ [1];
• Quality is a requirement of customers, not a prescription of engineers, not a commercial
need, not a general or legal prescription. It is based on the customer familiarization
with a product or service measured in accordance with the requirements for products or
customer requirements, in the technical, operational or subjective aspects, and always
represents an appropriate goal in a competitive market.
The quality in the area of the information systems or projects can be considered as a func-
tion consisting of three indicators that influence the success of the information system of a
company/organization:
• - information system/project indicator;
• - user indicator;
• - organization performance indicator.
DeLone and McLean [4], offered the first holistic model in 1992.
A wide range of studies offered the IS success model. These models offer own determination
of the IS success and factors that influence the success of this IS, along with this, the first
generalization of many points of view on the IS success and its promotion was performed [4],
who developed an IS success model.
This model attracted great attention of the IS researchers, it can be considered as one of the
popular models having the greatest influence in the IS studies [12], DeLone and McLean [4], had
made a meaningful progress, as shown in Figure 1. They conducted a comprehensive search in
the literature highlighting the IS success that was published from 1981 till 1987 in seven issues
on the topic of the IS, and offered the IS success model:
Figure 1: Description scheme of the first DeLone and McLean model [4]
After the publication of the DeLone & McLean IS success model, several researches on
this topic were published offering to control or expand the model. This has led to more
comprehensive understanding of the IS model.
The works published in this area studied this model from several points of view. The model
includes control when asserting that the success of one indicator does not directly lead to the
success of the next indicator.
If this were so, it would only be necessary just to ensure the system quality in order to achieve
the success of the entire IS.
According to the results of the published studies, the enhanced IS success model was offered,
as described in Figure 2.
Figure 2: Description scheme of the updated DeLone and McLean model (2003) [4]
Three popular aspects of the models differences can be determined as following:
1. adding the service quality to reflect the importance of service and support of the successful
electronic sales using the systems;
2. adding the user intent when evaluating the user relation;
3. combination of personal influence and corporate influence on the absolute improvement
structure;
The taxonomy of the changed categories included the system, information and service quality,
intentions to use, use, satisfaction of the user’s demands, absolute improvement.
The DeLone & McLean IS success models (1992, 2003) can be used as the basis for choosing
the relevant IS means.
The researchers had to choose several appropriate means of achieving success in accordance
with the goals and the phenomena studied, as well as to consider possible relationships between
success indicators during the study model building [11].
3. Geographic Information systems (GIS)
The Geographic Information Systems (GIS) are the software products (sometimes in combination
with the equipment element) which purpose is the creation, management, processing, analysis
and distribution of various data being important from the geographical point of view.
The GIS development advances as a result of growing awareness of the potential for the GIS
use. The GIS is a relatively new area of public use which develops over time. It allows for data
processing to increase functionality, and as a result, the demand for system data increases.
The GIS advantage is the simplification and visual representation of common phenomena
and appearance of efficient alternatives for decision-makers. The GIS systems allow for data
transmitting and receiving by many methods, including high-speed data transmission [2].
In the light of the foregoing, the quality assurance models approved in the area of information
systems are unlikely to be used for quality assessment in the field of GIS application.
As it was mentioned above, the Geographic Information Systems (GIS) are the most practical
for extensive data collection, storage, processing, query and presentation in various areas [3].
The GIS differ from the other information systems or projects, first of all, by the geographical
indicator of data. Geo-location is a very important indicator designed for simplification of the
data processing, classification of large data volumes, performance of dimension of data and
complex analyses [2].
The GIS area based on five critical factors [11].
People. Specialists working with GIS are characterized by multi-functionality that appears
on the basis of the necessity to combine data of various content with the approved Geographic
Information Systems. The general quality of specialists working with GIS consists in knowledge
of geography and related systems.
Data. The data in the Geographic Information Systems are characterized by:
• a wide variety of sources of information that in most cases, comes from the territory,
from work offices, data pools;
• a wide variety of data forms - formats, graphic data, alphanumeric tabular data.
Software. To normalize the existing data, there is a need in increasing the graphical func-
tionality in combination with classical functionality for data processing. In addition, image
processing and classification of results are sometimes required.
Hardware. At the stage of data collection intended for in the GIS use, special facilities, for
example, RTK antennas, GPS facilities, cameras, sensors, handheld computer, mobile phones,
etc.
Procedures/Methods. The stage when the use of the special GIS methods is recognized is
the stage of data query. Using the extensive queries, for example, “show the objects included
into the inquiry triangle” that are saved only for the GIS use.
4. Case Study
The studies were carried out among specialists who work in various fields of application of
information systems, including GIS, in Israel. The population includes system operators, system
performance specialists and system designers in the following areas:
Figure 3: Model offered for assessing the GIS information projects or software products success [11]
• Mobile Application;
• Document Management System;
• Internet Site & Web Application;
• BI & Big Data;
• Enterprise Resource Planning & Customer Relationship Management (ERP & CRM);
• GIS & Map Library.
4.1. Quality factors and indicators
Quality factors, criteria and indices for systems or information projects that have been reviewed
among the respondents. Respondents were required to rate the level of importance of the factors
on the Likeret scale 1-5.
Table 1: List of quality factors, indicators and criteria.
Quality factor A factor description (validated by the experts)
Efficiency A quality indicator used for assessing the capability of the
software product to ensure proper use with regard to the
amount of resources available for the product.
Flexibility A quality factor used for assessing the effort required to make
modifications to a work product/service.
Integrity A characteristic used for assessing the access to a
program/service or data by unauthorized persons.
Maintainability A quality factor used for assessing the flexibility of a software
product/service to modifications and improvements.
Performance A quality factor used for assessing the impact of the action time
characteristics on the productivity of the corresponding software
product/service (in accordance with the user needs).
Reliability A quality factor used for assessing the capability of a software
product to preserve the product performance at a certain level
when used in certain conditions.
Reusability A quality factor used for assessing the easiness providing for
reusing a program or existing elements of a software
product/service.
Testability A quality factor used for assessing the capability of
troubleshooting a software product/service using a predefined
procedure.
Modifiability A criterion used for assessing the program adaptability to
modifications or processing in accordance with modifications in
the operating environment and individual functional
requirements.
Interoperability A criterion used for assessing the software product/service
capability to interaction with other systems in the user’s
operating environment.
Correctness A criterion used for assessing the easiness to remedy minor
defects appearing from transition from a version of the software
product/service during the use.
Accuracy An indicator used for assessing the extent the program meets the
user requirements.
Availability An indicator used for assessing the extent the product/service is
practicable and available for user.
Changeability An indicator used for assessing the efforts to change the
working product/service.
Changeability An indicator used for assessing the software product capability
of duly functioning to comply with the preinstalled products.
Quality factor A factor description (validated by the experts)
Understandability An indicator used for assessing the software product
understandability for user, the program compliance with
requirements and options to use it for various tasks the user
faces.
Usability An indicator used for assessing the software product availability
for training, functionality and attractiveness for user.
Visibility An indicator used for assessing the software product/service
visibility.
Security level An indicator used for assessing the information security using
the software product/service.
Robustness An indicator used for assessing the capability of the computer
system to withstand errors during the program operation.
Scalability An indicator used for assessing the easiness of making changes
to the system operation or elements to expand the current
features of the software product/service.
Supportability An indicator used for adaptation and acceptability of services in
aspect of calculations and configuration. Easiness of the system
installation and fault localization.
Transferability An indicator used for assessing capability of the software
product to transfer from one version of equipment or operating
system to the other.
Interface Facility An indicator used for assessing the easiness of connecting two
software products/services not interfering the performance.
Portability An indicator used for assessing the software product availability
to transition from one working environment to the other.
4.2. Study Results
The study was conducted using the quantification method. The main reason for choosing this
method was to verify the quantitative conformity between various factors and between qualities
of factors. A sufficient number of respondents completed the survey and a sufficient number of
questionnaires were drilled in to make the first quantitative conclusions based on the data. The
survey was conducted in a guided manner, to avoid misunderstanding the definitions of success
metrics.
Questionnaires were validated by 5 experts in the field of information systems. In the next
step, the questionnaire were modified according to the comments of experts.
The paradigms of this study were not combined, since the literature broadly and comprehen-
sively discusses the IS, that is why there was no need in preliminary qualitative study before
identifying the factors that primarily influence the quality.
To conduct the study as part of this work, 147 participants were selected and distributed by 7
areas of IS use.
Three main techniques of statistical analyses were conducted:
Table 2
Distribution of the study population
Application area Number of respondents Percentage
Mobile Application 14 9.52%
Document Management System 12 8.16%
Internet site and Web Application 15 10.2%
BI & Big Data 12 8.16%
ERP & CRM 22 14.97%
Enterprise portal and knowledge management 10 6.8%
GIS & Map Library 62 42.18%
Total 147 100%
• The PCA analysis was performed to verify the following assumption of the study - is
there a difference in the calculation of quality indicators/criteria/characteristics among
specialists engaged in the six areas of the information systems use and specialists engaged
in the GIS area?
• The MANOVA analysis was performed to verify the following assumption of the study -
is there a difference in the calculation of quality indicators/criteria/characteristics among
specialists engaged in the GIS area in the specialties mentioned above?
• The EFA analysis was performed to build a quality assessment model that is one of the
study objectives.
Figure 4: Pie chart of distribution of the study population
In the GIS area, 81 participants completed the survey, 66 participants completed the survey of
the 6 other areas. This number is not sufficient to verify the study assumptions. The interviewed
study participants were selected by personal communication with the researcher. This study
was conducted among specialists in the state of Israel. In this study, no experience was averaged
for research purposes.10 variables that were highly rated out of 25 variables and were selected
according to the Top 10 principle. The calculation of Multivariate analysis of variance - Manova
results in the GIS area was performed for only 10 variables indicated above due to constrains of
calculations and number of variables compared to the number of observations. The calculation
of the Manova results requires a minimum ratio of 1:3 of the number of variables compared to
the number of observations, respectively.
4.3. Result discussion
It was found out that there were significant differences in parameters between different areas.
A similar conclusion is expected since the issues at hand are about various important areas
that differ in technological level, goals and purpose of the software product/information products
which differ depending on the area.
The graph (Figure 5) shows the PCA results for the variance of 10 variables that are most
suitable for analysis.
Figure 5: Distribution of variances among 10 variables with the most noticeable differences
Based on this graph, the Principle Component Analysis - PCA can be performed using 2
main factors: PC1 and PC2. It should be taken into account that the vertical axis indicates the
variance between the PCA distribution, this allows making sure that there is a large difference
between the PC1 and PC2 values [13], so we can continue testing based on these 2 factors, as
shown at Figure 6.
Figure 6: Clustering of the study results in distribution of GIS in comparison with other areas consid-
ering the “centers of gravity” of two groups
No differences were found out between different functions in the area of the GIS use, since all
three functions belong to the same parameters "as a whole" and determine their importance. In
addition, all three groups are equally important for the GIS development, so that their parameters
do not differ from each other.
This is explained by using the agile method as the accepted method for the GIS software
products/information projects development.
The method is based on the deep interest of end users in the development process, and this
significantly ensures a unified adaptation of significant quality factors.
4.4. Offered model for the information systems or projects quality
assessment in the GIS area
According to the FA results, it was found out that the p-value was lower than 1% for the GIS
data and other sampling data:
• test of the hypothesis that 3 factors are sufficient;
• the chi square statistic is 486.33 on 228 degrees of freedom;
• the p-value is 8.34e-21.
In addition, the hierarchic two-tier model was built to analyze 25 parameters between 3
variables on the basis of the analysis.
The hierarchic two-tier model will be used for the Exploratory Factor Analysis - EFA Results.
Based on the statistical processing of data collected by interviewing operators, program
performance specialists and GIS developers, a model for the information systems or projects
quality assessment in the GIS area can be offered.
In this case, it is necessary to coordinate the time for measuring success factors and establish
a process for implementing the model.
Based on the impact of quality factors/indicators/criteria used as independent variables, three
dependent variables can be correlated with three main stages of the information project/software
product development, as shown at Figure 7:
• pre-Implementation;
• implementation;
• post-Implementation.
Dividing the information project/software product development software here is a generic
division. The third stage contains activities like operation, maintenance, service and support
etc.
Figure 7: Offered model for the information systems or projects quality assessment in the GIS area
The values that appear above the arrows are weights, which represent the magnitude of the
impact of a success factor on the success of an information system or information project, in
the proposed model. For convenience, the model can be presented as a Venn diagram (Figure 8).
Figure 8: Venn diagram of the model offered for the information systems or projects quality assessment
in the GIS area with the distribution at three stages of the information system or project development
life cycle
5. Conclusions and Recommendations for Further Studies
For the purposes of in-depth study of the problem, it is advisable to include additional points
into the further studies that may influence the method of the information projects/software
products quality assessment in the GIS area.
Increase in the number of the study participants. It is necessary to increase the number
of the study participants, for example, to create similar prerequisites in terms of quantification,
taking into account the importance of the work experience of the study participants. In other
words, it is necessary to create research cohorts to be similar in terms of size and work experience.
Expanding the geography of the study participants. The next study should be carried
out in several countries with as different cultures and technologies as possible. This provides a
solid base for the model offered in the study for the GIS information projects quality assessment.
Expanding a number of specialties for working in the GIS area. The study should
combine classification of quality assurance specialists in this area.
Periodization of the model offered in the study. It is necessary to conduct a study in
order for the purposes of periodization of the model offered in this study.
Increased number of observations. It is necessary to substantiate the study results in
details, increase the number of observations before sampling, up to at least 75 observations in
each area. The calculation of the Manova results requires a minimum ratio of 1:3 of the number
of variables compared to the number of observations, respectively.
The future research will be dedicated to development of quality models for information
systems or information projects, based on the field of activity of an organization, the users of the
information systems or information projects and work environment in which the organization
operates.
References
[1] S. K. Dubey, S. Ghosh, A. Rana, Comparison of software quality models: An analytical
approach, Communications of the ACM 2 (2012) 111–119.
[2] M. Caprioli, E. Tarantino, Standards and quality in gis contexts, in: FIG Working Week,
2003.
[3] G. Subbiah, A. Alam, L. Khan, B. Thuraisingham, Geospatial data qualities as web services
performance metrics, in: Proceedings of the 15th annual ACM international symposium
on Advances in geographic information systems - GIS '07, ACM Press, 2007. URL: https:
//doi.org/10.1145/1341012.1341090. doi:10.1145/1341012.1341090.
[4] W. DeLone, E. McLean, Information systems success revisited, in: Proceedings of the
35th Annual Hawaii International Conference on System Sciences, IEEE Comput. Soc, ????
URL: https://doi.org/10.1109/hicss.2002.994345. doi:10.1109/hicss.2002.994345.
[5] O. Gordieiev, V. Kharchenko, N. Fominykh, V. Sklyar, Evolution of software quality
models in context of the standard ISO 25010, in: Proceedings of the Ninth International
Conference on Dependability and Complex Systems DepCoS-RELCOMEX. June 30 –
July 4, 2014, Brunów, Poland, Springer International Publishing, 2014, pp. 223–232. URL:
https://doi.org/10.1007/978-3-319-07013-1_21. doi:10.1007/978-3-319-07013-1_21.
[6] N. Upadhyay, B. M. Despande, V. P. Agrawal, Towards a software component quality
model, in: Advances in Computer Science and Information Technology, Springer Berlin
Heidelberg, 2011, pp. 398–412. URL: https://doi.org/10.1007/978-3-642-17857-3_40. doi:10.
1007/978-3-642-17857-3_40.
[7] E. Georgiadou, Software Quality Journal 11 (2003) 313–323. URL: https://doi.org/10.1023/a:
1025817312035. doi:10.1023/a:1025817312035.
[8] D. Scapin, C. Leulier, J. Vanderdonckt, C. Mariage, C. Bastien, C. Farenc, P. Palanque,
R. Bastide, A framework for organizing web usability guidelines, in: Proceedings of 7th
International Conference on Human Factors and the Web HFWeb, Springer, 2000, pp. 1–23.
[9] Y. U. Mshelia, Mecot: A software quality metrics collection tool, Journal of Systems
Integration (2019) 21–35.
[10] H. Rashidi, M. Sadeghzadeh Hemayati, Software quality models: A comprehensive review
and analysis, Journal of Electrical and Computer Engineering Innovations 6 (2018). URL:
https://doi.org/10.22061/jecei.2019.1076. doi:10.22061/jecei.2019.1076.
[11] K. A. Eldrandaly, S. M. Naguib, M. M. Hassan, A model for measuring geographic informa-
tion systems success, Journal of Geographic Information System 07 (2015) 328–347. URL:
https://doi.org/10.4236/jgis.2015.74026. doi:10.4236/jgis.2015.74026.
[12] D. Nabil, A. Mosad, H. A. Hefny, Web-based applications quality factors: A survey and
a proposed conceptual model, Egyptian Informatics Journal 12 (2011) 211–217. URL:
https://doi.org/10.1016/j.eij.2011.09.003. doi:10.1016/j.eij.2011.09.003.
[13] S. Mishra, U. Sarkar, S. Taraphder, S. Datta, D. Swain, R. Saikhom, S. Panda, M. Laishram,
Principal component analysis, International Journal of Livestock Research (2017) 1. URL:
https://doi.org/10.5455/ijlr.20170415115235. doi:10.5455/ijlr.20170415115235.