1 Applying PQM to a Regional Portal Mª Ángeles Moraga, Coral Calero, Mario Piattini Abstract —Portals have evolved from being simple providers of Web page access and corporate databases into being sites that support intelligent management, integration of applications and collaborative processing. Portals can be considered as an evolution of datawarehouses by extending its application to Intranets and giving access to all information resources and knowledge of a firm. All these facts make it essential that developed portals are of good quality. Our aim is to define a portal quality model and, to achieve it, we have used the GQM method. As our model is in the definition phase now, only its two first phases have been used. That is the reason why our model must be considered as a first approach and not as a close and definitive model. Our model, called PQM, is composed of six dimensions: tangibles, reliability, responsiveness, assurance, empathy and data quality. These dimensions have been obtained by means of the SERVQUAL method. Moreover, we have done a survey of the workers of the portal of Castilla-La Mancha (castillalamancha.es) to obtain their opinion and assesment about different aspects of this portal in relation to PQM. Index Terms — Quality assurance, quality concepts, model development. —————————— u —————————— 1 INTRODUCTION tals integrate all kinds of information, applications and ser- P ortals prov ide a way to access to a local or remote vices into a unique environment, according to the personal experience of each user. With a portal, a firm can give ap- propriate information to the correct subject. Hence, portals network, to a company in the case of a corporate portal or to general-interest topics and services in the case of a public can be considered as the entry door to the firm [8]. portal. According to [11], portals can be divided into genera- tions: In general, portals provide [11] us with: - First generation: Access Portal: They were used to pro- - A custom framework for presenting pages and compo- vide a set of links to other information and resources. nents within each page and organizing information for spe- - Second generation: Aggregation Portals: They bring in- cific communities. formation back to the portal so that the user does not have to - Personalization capabilities for individual users. go anywhere else. - A set of “portlets” (components that integrate data, ap- - Other generations: Workspace Portals (where the portal plications, content, and resources and present information becomes the users’ work environment, including all the to the portal user) appropriate information, tools, and resources) and Adaptive - A single sign -on to the set of applications accessed via Portals (where the portal experience, itself, dynamically de- the portal. pending on the user’s context and the ongoing process). - Other features, such as search and collaboration. However, the research on portals is still beginning. Some Therefore, portals have evolved from being simple pro- key points for its evolution are [17] listed below: viders of Web page access and corporate databases into be- - The percentage of organizations that treat portals as ing sites that support intelligent management, integration of core systems will increase from less than 10 percent in 2003 applications and collaborative processing. Portals can be to 15 percent in 2004 to 35 percent by 2007. considered as an evolution of datawarehouses by extending - The portal market is only 40 percent introduced now; its application to Intranets and giving access to all informa- by the end of 2004, 85 percent of the Global 2000 will have tion resources and knowledge of a firm [4]. One of the ad- implemented an enterprise portal framework. This fact will vantages of portals is its ability to integrate and personalize lead to a 161 percent growth in revenue in 2004. several technologies (groupware, databases, dataware- - The portal market has contracted 26 percent since 2002; houses, e-mail, meta -data, intelligent management systems, by the end of 2004, another 20 percent of portal players will etc.) in a unique business management tool. Moreover, por- exit the market. The fact of achieving that a portal is of good quality is fun- ———————————————— damental because it is the only mechanism that makes users • M. Moraga is with the Department of Lenguajes y Sistemas Informáticos, University of Castilla-La Mancha, Paseo de la Universidad,4. 13001 Ciu- return [13]. In fact, quality has been a research subject in dad Real. Spain. E-mail: mmoraga@proyectos.inf-cr.uclm.es several fields: • C. Calero is with the Department of Lenguajes y Sistemas Informáticos, In [1], a quality model for the selection of ERP systems is University of Castilla-La Mancha, Paseo de la Universidad,4. 13001 Ciu- dad Real. Spain. E-mail: Coral.Calero@uclm.es proposed. They choose the ISO/IEC 9126-1 quality Standard • P. Piattini is with the Department of Lenguajes y Sistemas Informáticos, as a framework and suggest a methodology to adapt it to University of Castilla-La Mancha, Paseo de la Universidad,4. 13001 Ciu- specific domains. In [10], authors show us a systematic way dad Real. Spain. E-mail: Mario.Piattini@uclm.es. to specify the relevant quality attributes involved in the ar- 2 QUATIC’2004 PROCEEDINGS chitectural design process. In [7], authors show us a model 2 GQM METHOD to value and select the e-commerce websites in a B2C envi- The GQM method [18] is a systematic approach for tailoring ronment (Business-to-consumer). In [15], a way to develop and integrating goals to models of the software processes, web design guidelines through a quality function is shown. products and quality perspectives of interest, based upon In [19], a conceptual model and an in strument to measure the specific needs of the project and the organisation. website quality are developed and the websites quality This method starts top-down with the definition of an factors that are important to consumers are defined. In [2], explicit measurement goal. The goal is refined into several authors suggest a Web quality model for the classification of questions that break down the issue into its major metrics and web methodologies. components. Then, each question is refined into metrics that In [14], the SERVQUAL model is described. This model should provide information to answer those questions. consists of five dimensions and 22 items used to measure the The GQM method is composed of four phases [18]: different elements of service quality across a broad spectrum 1.- Planning phase. A GQM team is established, the of services. In [9], authors develop a framework to measure improvement area and the application project are selected, a service quality based on web using the SERVQUAL model project team is established and finally, as a result, we obtain as a starting point. a project plan. However, there is not a specific framework to control the 2.- Definition phase. The measurement plan is defined quality of portals, probably due to the recent developing of and documented (goal, questions, hypotheses and metrics portals. are defined). In order to put an end to this lack, we present a first 3.- Data collection phase. In this phase, the data proposal of a portal quality model. This proposal has been collection takes place, resulting in the collected data. made using, as a basis, the GQM method [18] and the 4.- Interpretation phase. During this phase, the collected model proposed by [14] to obtain the goals. We have data are processed with respect to the defined metrics into selected the SERVQUAL model because it has been measurement results, that provide answers to the defined successfully adapted to the quality achievement of different questions. With these answers, it will be possible to evaluate topics (as tourism, information systems, automobile whether the goal has been achieved or not. industry, etc ) and the GQM method because facilitates the As we have said before, we have followed the GQM definition of the portal quality model with a given goal. method to define a portal quality model, but we have only Also, we have done a survey of the workers of the used its two first phases. The other two phases will be used Castilla-La Mancha portal (castillalamancha.es) to obtain when the model has been validated. In the next section, we their assessment about different aspects of the portal with will show how our PQM model has been obtained. the objective of evaluating the quality of the above mentioned portal in relation to each dimension of quality that we have defined. 3 QUALITY MODEL Moreover, the model proposed by us is valid not only for The PQM model (Portal Quality Model) has been developed corporate portals but also for public ones considering that using the two first phases defined in the GQM method. nowadays public portals include public and private Taking into account the fact that the quality of a portal is components, as corporate portals. The main difference difficult both to define and to measure, the model that we between corporate and public portals is that in public present below must be considered as a first proposal of portals the business is the portal by itself, while in corporate framework. This model can be used to measure the quality portals, the business is to be open to the public to present of a portal, considering that the quality of the portal can be their products, services, image or beliefs. In spite of this defined as the degree which the portal facilitates services difference, all portals are becoming multipurpose. Besides, and relevant information to the customer with. nowadays, public portals include both public and private The activities carried out in the achieved phases are components like corporate portals. detailed below. At last, we can emphasize that we have only used the Goal and Question phases of the GQM method to achieve 3.1 First phase: planning our aim (to define a portal quality model). So, we have put The first activity that we carried out in the planning phase off the definition of metrics until later on (when the model was to establish a GQM team independent of the project will have been validated). team. Then, the area that we wanted to improve was This paper is structured as follows: In section two, the selected. In our case, it was the quality of portals. The project GQM method is shown; in section three, the quality model team was formed by all the developers of that we have developed is explained and the phases that we castillalama ncha.es portal. have achieved are specified. In section four, the castillalamancha.es portal is described and in section five, it 3.2 Second phase: definition is explained the survey that has been done of the workers of The second phase, called definition, is the most important the castillalamancha.es portal and the obtained results are one. analysed. Finally, in the last section, conclusions and future In the first activity of this phase, the goal was defined. In work are shown. our case, the goal was: “To improve the quality of portals”. The next activity was to refine the goal into several ques- tions. We used the SERVQUAL model [14], to carry out this MORAGA ET AL.: APPLYING PQM TO A REGION AL PORTAL. 3 definition. The SERVQUAL model is composed of five di- which information is inputted, processed or presented to mensions: tangibles, reliability, responsiveness, assurance them? and empathy. These dimensions were adapted to portals. - Contextual DQ: To what degree does the informa- Moreover, we added another one that is related to data qual- tion provided meet the needs of the users? ity. Once the portal quality model (PQM) was defined, we However, we thought that the breakdown only to that decided to do a survey of the workers of the portal of Cas- level was not enough because it was too much generic, so tilla-La Mancha (castillalamancha.es) to obtain their opinion we divided some of these dimensions into subdimensions. about different dimensions of the portal Then, we show the six dimensions (questions) that make up our model (of quality of portals) together with their subdimensions (subquestions). 4.- THE PORTAL OF CASTILLA-L A MANCHA - Tangible: This dimension indicates if "the portal con- (CASTILLALAMANCHA. ES). tains all the software and hardware infrastructures needed In this section, we will explain the aim and structure of the according to its functionality”. portal of Castilla-La Mancha, since we have done a survey of - Reliability: It is the "ability of the portal to perform its its workers to know which dimensions of the model should functionality accurately". In addition, this dimension will be be improved and which dimensions are correct. Also, we affected by: have proved the reliability of the survey. - Availability: The portal must be always operative. The Castilla-La Mancha portal (www.castillalamancha.es) - Search Quality: The results that the portal pro- is a corporate portal that is in operation since April, 2002. vides when making a search must be appropriate to the re- Its main goal is to generate Internet projects in the region of quest made by the user. Castilla-La Mancha and help to achieve their success. - Responsiveness: It is the "willingness of the portal to Regarding the portal workers, they can be divided into help and to provide its functionality in an immediate form two profiles according to the tasks they develop: to the users ". In this dimension, we distinguish the follow- ? Contents Responsible. They manage the contents ing subdimensions: that are published in the different channels of the - Scalability: It refers to the ability of the portal to castillalamancha.es portal. smoothly adapt to increasing workloads as a result of addi- ? Technical Support. These workers can be divided tional users, an increase in traffic volume or the execution of into two groups: more complex transactions [6]. - Programmers: they must maintain the technological as- - Speed: It relates to the response times experienced pects of the portal. by portal users [6]. - Designers: they are in charge of the image of the portal. - Assurance: It is “the ability of the portal to convey trust This portal is aimed at providing, above all, information and confidence”. This dimension will be affected by: about Castilla-La Mancha. - Confidentiality: Ability to keep the privacy of the We can distinguish several channels within this portal users. (each of them contains information related to a specific - Empathy: We define this dimension as the "ability of the topic): portal to provide caring and individual attention ". In this - Technology channel: it provides information about sci- dimension, we distinguish the following subdimensions: entific, technological, energy or astronomical issues among - Navigation: The portal must provide a simple and others. intuitive navigation when using it. - Sixth province channel: it establishes contact with castel- - Presentation: The portal must have a clear and lano-manchegos (people from Castilla La Mancha) that, at uniform interface. present, are living outside. - Integration: All the components of the portal must - Enterprise channel: You can find events related to en- be integrated into a coherent form. terprises among other things. - Personalization: The portal must be capable of - Employment channel: People can find a new job. adapting to the user’s priorities. - Agro channel: It contains information related to agricul- Furthermore, due to the big amount of data that are han- ture. dled in a portal and taking into account that it is fundamen- - Environment channel: it deals with human resources, tal that these data are of good quality, we thought it is nec- pollution, waste, animals, vegetation, water, etc. essary to add a new dimension: - Women and elderly channel: It contains interesting is - - Data quality (DQ): This dimension is defined as "quality sues for these groups of people. of the data contained in the portal". According to [3], we can - Children channel: Basic knowledge about the region will distinguish four subdimensions: be acquired by children. - Intrinsic DQ: What degree of care was taken in the These channels are focused on Castilla-La Mancha region. creation and preparation of information? Moreover, there are e-commerce, links to others websites, e- - Representation DQ: What degree of care was taken learning, and others interesting things. in the presentation and organization of information for us- ers? - Accessibility DQ: What degree of freedom do us- ers have to use data, define and/or refine the manner in 4 QUATIC’2004 PROCEEDINGS  σ 2p + σ i2  (1) ρ XX ' = 2 * 1 −    σ 2  5.- SURVEY x Questionnaires are, probably, the most commonly used re- search method according to [16]. By using a questionnaire, where: we try to obtain the opinion of people about different as- s P2 shows the variance of the marks obtained by the sub- pects in order to evaluate each dimension of quality. jects in the even items. The questionnaire consists of forty two questions (see s i 2 shows the variance of the marks obtained by the sub- Appendix). The items that compose each dimension (which jects in the odd items . is equivalent to indicate the dimension which each of the s X2 shows the variance of the global marks. questions asked in the questionnaire corresponds to) are: We have applied the above exposed formula to the results - In the Tangible dimension, we classify questions Q26, obtained by the survey done of the workers of castillalaman- Q39. cha.es and we have obtained that the reliability of the survey is - In the Reliability dimension, we have questions Q11, 0.9. Therefore, we can state that our survey is reliable because Q15, Q29, Q41. the reliability coefficient can vary from 0 to 1, where 0 means - In the Responsiveness dimension, we have questions that there is no reliability and 1 means there is a maximum of Q1, Q3, Q4, Q12, Q13, Q14, Q38. reliability. - In the Assurance dimension, we classify questions Q21, Q22, Q23. 5.2 Results - In the Empathy dimension, we have questions Q2, Q5, Our following step was to analyse the results obtained by Q6, Q10, Q16, Q17, Q18, Q19, Q20, Q24, Q25, Q27, Q28, Q32, the survey. Q33, Q34, Q35, Q36, Q37, Q40, Q42. Due to the different profiles of workers, not all the sub- - In the Data Quality dimension, we classify questions Q7, jects answered all the questions that were raised. Therefore, Q8, Q9, Q30, Q31. the results of each question (table 1) have been obtained Before the workers of the portal answered the question- depending on the subjects that answered it. naire, it was evaluated by a small group of users in order to In the next table, the average value for each of dimension verify its understandability. According to the results, some is shown. aesthetic modifications were made so as to obtain a better understanding. 5.1 Survey Reliability It is necessary to prove the reliability of the survey to state that the results obtained by it are useful. A survey is reliable if it is made several times by the same subjects and the result obtained is always the same. Several methods to calculate the reliability of a survey have been proposed. These methods try to calculate the reli- ability coefficient (defined as the correlation among the Table 1. Average value for each dimension. marks obtained by the subjects in two parallel forms of a test). [12]. We can use different methods to calculate the em- pirical value of the reliability coefficient. One of these meth- The dimension with the lowest value is assurance and ods is the Split Halves Method which is the method that we therefore, it is necessary to improve it by carrying out differ- have used because it only needs to make the test once. ent tasks that have been proposed to achieve this goal. In We tried to guarantee that the two halves that the test is general, almost all the dimensions values are around the divided into would be parallel. So, we considered that a half average. Hence, we have defined several tasks to improve of the test was made up of even questions and the other half the dimensions that have the lowest values. We can carry was made up of odd questions. This is better than consider- out all the tasks and later, make the test again to verify if the ing that the first part of the test is a half and the second part quality of the portal has been improved. The dimension that is the other half because subjects are more tired when an- obtains a higher value on average is data quality. As a result, swering the second part of the test. we can conclude that the information provided by the portal The reliability of the survey does not only depend on the turns out to be correct so the portal fulfils its main goal, characteristics of the survey but subjects that have answered which is to offer classified information through different the questionnaire are important as well channels. We calculate the reliability with the formula proposed by Guttman-Flanagan, the equation is [5]: 6.- CONCLUSIONS AND FUTURE WORKS We know that portals have lately risen. Therefore, it is very important to assure its quality. Customers will choose por- tals that are more appropriate to their needs and objectives MORAGA ET AL.: APPLYING PQM TO A REGION AL PORTAL. 5 and this fact implies that if a portal is not of a good enough quality, it will disappear. Our aim has been to define a portal quality model and to do so, we have used GQM method. As our model is in the definition phase, we have only used the two first phases. Moreover, this means that the model presented here must be considered as a first proposal but not as a close and defini- tive model. Our model, called PQM, is composed of six dimensions: tangibles, reliability, responsiveness, assurance, empathy and data quality. We must carry out several future works due to the initial state of our quality model. The validation of the model is our first and main objective and to achieve it, we are going to carry out several empirical validations from surveys of cus- tomers, experts, etc. to controlled experiment and study cases. We will make all these works with different kinds of portals in order to have the model completely validated. Once our model has been validated, we will study and clas- sify, on the one hand, existing metrics and on the other, met- rics of other environments but applicable to portals. If it is necessary, we will define new metrics, that must be formally and empirically validated, and will be used for the last phases of the GQM method. We are working in the prepara- tion of a questionnaire for users of the portal. APPENDIX ACKNOWLEDGEMENT This research is part of the PLATIA project supported by the Consejeria de Ciencia y Tecnología of Junta de Comuni- dades de Castilla-La Mancha (Spain) and the CALIPO pro- ject (TIC2003-07804-C05-03) supported by the Ministerio de Ciencia y Tecnología. 6 QUATIC’2004 PROCEEDINGS Coral Calero is PhD in Computer Science. Associate Professor at the REFERENCES Escuela Superior de Informática of the University of Castilla-La Mancha in Ciudad Real (Spain). She is a member of the Alarcos Research [1] Botella, P., et al., Towards a Quality Model for the Selection of ERP Group, in the same University. Her research interests are: advanced Systems. Component-Based Software Quality, 2003 pp. 225-245. databases design, database/datawarehouse quality, web/portal quality, software metrics and empirical software engineering. She is author of [2] Calero, C., J. Ruiz, and M. Piattini. A web metrics survey using articles and papers in national and international conferences on these WQM. in Fourth International Conference on Web Engineering. N. koch, subjects. Her e-mail is: Coral.Calero@uclm.es P. Fraternali, M. Wirsing (Eds.): ICWE 2004, LNCS 3140, pp. 147-160, 2004. Springer-Verlag. [3] Dedeke, A. and B. Kahn. Model-Based quality evaluation: a com- Mario Piattini is PhD in Computer Science. Full Professor at the Es- parison of Internet classified operated by newspapers and non-newspaper firms. cuela Superior de Informática of the University of Castilla-La Mancha. in Proceedings of the Seventh International Conference on Information Author of several books and papers on databases, software engineering and information systems. He leads the ALARCOS research group of the Quality. 2002. pp. 142-154. Department of Computer Science at the University of Castilla-La Man- [4] Dias, C., Corporate portals: a literature review of a new concept in In- cha, in Ciudad Real, Spain. His research interests are: advanced data- formation Management. International Journal of Information Management., base design, database quality, software metrics, object oriented metrics, 2001. 21 pp. 269-287. software maintenance. His e-mail is Mario.Piattini@uclm.es [5] Gulliksen, H., Theory of mental tests. 1950, New York. John Wiley & Sons. [6] Gurugé, A., Corporate Portals Empowered with XML and Web Ser- vices. 2003, Amsterdam. Digital Press. [7] Hangjung, Z. and K. Ramamurthy. A choice model for assessing and selecting e-commerce websites in a B2C environment. in Eighth Americas Conference on Information Systems. 2002. pp. 348-355. [8] Kvitka, C., Profiting with portals. Oracle Magazine, 2002. 21. May/June. [9] Li, Y.N., K.C. Tan, and M. Xie, Measuring Web-based service qual- ity. Total Quality Ma nagement, 2002. 13(5) pp. 685-700. [10] Losavio, F., Quality Models to Design Software Architecture. Jour- nal of Object Technology., 2002. 1(4) pp. 165-178. [11] Marshak, D.S. and P.B. Seybold, What customer-centric executives need to know about portals. An Executive's Guide to Portals. 2003. January pp. 1-4. [12] Muñiz, J., Teoría clásica de los tests. 1994, Madrid. Pirámide. [13] Offutt, A.J., Quality attributes of web software applications. IEEE Software., 2002. March-April pp. 25-32. [14] Parasuraman, A., V.A. Zeithami, and L.L. Berry, SERVQUAL: a multi-item scale for measuring consumer perceptions of service quality. Journal of Retailing., 1998. 67(4) pp. 420-450. [15] Park, H. and S.J. Noh, Enhacement of Web design quality through the QFD approach. Total Quality Ma nagement, 2002. 13(3) pp. 393-401. [16] Pfleeger, S.A. and B.A. Kitchenham, Principles of Survey Re- search. Software Engineering, 2001. 26(6) pp. 16-18. [17] Research, O., Content Consolid ation, in Portals Magazine. 2004. [18] Solingen, R.v. and E. Berghout, The Goal/Question/Metric Method. A Practical Guide for Quality Improvement of Software Development. 1999, London. Mc Graw Hill. [19] Webb, H.W. and L.A. Webb. B2C Electronic Commerce Websites: an Analysis of Quality Factors. in Eighth Americas Conference on Informa- tion Systems. 2002. pp. 340-347. Mª Ángeles Moraga received her MSc in Computer Science and her Technical Degree in Computer Science by the University of Castilla-La Mancha (UCLM). Nowadays she is developing her PhD at the UCLM. She holds a FPI grant from the Spanish Ministerio de Educación y Ciencia. She is a member of the Alarcos Research Group, in the same University, specialized in Information Systems, Databases and Sof tware Engineering. Her research interests are: Portals, Quality & Metrics, Software Quality. Her e-mail is mmoraga@proyectos.inf -cr.uclm.es