Adoption of Usability Engineering Methods: A Measurement-Based Strategy Eduard Metzker Ahmed Seffah Vector Informatik, Stuggart Germany EHL-LHR, Lausanne Switzerland eduard.metzker@gmx.net seffah.ahmed@ehl.ch ABSTRACT Furthermore, there is little hard evidence backing up new In the context of a software development organization, two technology to be adopted, and their costs and benefits are strategies are possible for introducing and institutionalizing rarely understood [1]. Without this data, choosing a particular new usability engineering methods. The first one, expert-based technology or methodology for a project at hand essentially is institutionalization, require to resort to third party companies a random act with many consequences [2]. The findings from or experts that can, based its previous expertise, assist the a very large survey made by Standish group, new technology team in selecting, implementing and institutionalizing is one of the top ten reasons for projects failure or success usability engineering methods and tools. The second one, a [27]. measurement-based strategy, is based on empirical evidence for learning and assessing the appropriateness, usefulness of a In order to support and effective adoption, a new metrics- usability engineering method. This paper proposed to combine based approach comprising a process model and a support these two approaches in a single process metrics support environment are presented in this paper. A large case study environment for selecting and institutionalizing usability was developed to assess the acceptance of the approach by engineering methods. The proposed approach has been development teams. The evaluation involved 44 professional validated via in a cross-organizational empirical study software engineers from five medium to large-sized involving several software engineers from five mediums to organizations. The evaluation method is discussed including large sized software development companies. context, method, subjects, procedure and results. Implications of the results on the design of metrics-based strategy are discussed for adopting new technology and assessing their Keywords acceptance by project teams. Based on the results, a set of Metrics, usability, usability engineering methods, guidelines is derived to optimize the acceptance of metrics institutionalization, adoption, software developments exploitation approaches by project personnel. organization 2. THE PROPOSED METRICS SUPPORT 1. INTRODUCTION ENVIRONMENT Within the scope of this research, by adoption we refer to the process and the related tools for selecting the appropriate new The overall goal of the proposed approach, called Adoption- software development technology while assessing their Centric Usability Engineering (ACUE), is to facilitate the suitability to the project needs and size as well as the adoption of UE methods by software engineering practitioners capability of the personnel to use effectively and efficiently and thereby improve their integration into existing software the new established technology. Adoption has been always a development methodologies and practices. ACUE is designed key challenge for software development organizations [28]. It to support project teams in institutionalizing this abstract was reported that the management staff commitment and the knowledge about UE methods and to help them transfer this involvement of the personnel represent the top factors that knowledge into their development processes. UE methods are impact on the success with a new technology when first perceived as integrated into an existing software development introduced [27, 29, and 30]. process when they are adopted by the project team, i.e. when they are accepted and performed by the project team. However, despite management efforts made by organizations to render the transition more “user-friendly”, the associated ACUE exploits empirical data collected in different projects to help and training material, although precise and perfectly yield stronger evidence on how the method works in a certain describing the new method are often delivered in an esoteric context. The data form an empirical base to guide the and unreadable language. Another important factor is that improvement of UE methods and to facilitate the informed organizations and managers are usually overly optimist in selection and deployment of UE methods in future projects. If their employees’ ability to quickly master a new technology. this approach is applied repeatedly in a number of projects The reality is that understanding how to apply the technology over time, it leads to an incremental construction of a body of is a long and arduous process. evidence to guide usability engineering method selection (Figure 1). increase its probability of selection in similar projects in the future. The characteristics of the project are specified using the context models stored in the repository. A context model includes context factors to describe various project constraints such as the resources available in the project or the type of product to be developed. Each context model consists of a set of factors that can have nominal, ordinal or interval scale measures [11]. An example for an ordinal factor that describes a property of a product to be developed is ‘user interface interaction complexity’. This factor may have three characteristics ‘text-based interface’, ‘graphical user interface’ or ’multi-modal interface’. Depending on the project characteristics, appropriate methods are suggested by the system. Candidate methods are ranked according to two different criteria: (1) similarity between the method and the project characteristics, (2) results of assessments from project teams that used the methods in previous projects. Within, the system the usability engineering methods are provided in the format of method packages. Each package contains a textual description of a method that is structured according to the pyramid principle [12]. Auxiliary material such as templates, checklists and tools is linked to each Figure 1: The overall view of ACUE approach package. This material facilitates easy compliance of the method described. The process guidance remains passive and The support environment is called ESPrEE (Evidence-based does not enforce the performance of the methods proposed. Software PRocess Evolution Environment). The components of ESPrEE are integrated via a web portal and they be The constraints of industrial software development projects remotely accessed using any web browser. Its core often enforce the invention of new methods or the adaptation functionalities are: and streamlining of existing methods. For these reasons the environments provides means for capturing methods and • At the beginning of a new project, the metrics-based integrating them into the repository. method selection component of the environment is used to configure the set of usability engineering methods that 3. EVALUATION will be applied in the project • After the method selection has been completed, the 3.1. Specific purpose of the evaluation environment generates a project-specific hyper-media workspace in which the methods selected are graphically A large number of measurement programs are suspended or - visualized according to the project phases in which their in the worst case – failed. This is because the measurement usage is intended program is not accepted by stakeholders for the following • At the completion of major project phases or in post reasons [3, 4, 5, 6 and 7]: mortem sessions [13], the quality of the methods employed is assessed by the project team against a 1. The measurement process is perceived as tedious and quality model. For this purpose quality models contain a time consuming set of quality factors and carefully defined rating scales 2. Effort and benefits of the program are poorly distributed 3. The impact on daily practice is perceived as being too The core of the system is a fuzzy multi-criteria decision- low to justify sustained effort making engine. The characteristics of the usability methods 4. Metrics support tools and/or processes are difficult to use and projects are defined as sets of fuzzy sets. Based on these models the engine is able to compute similarity measures for To examine if the proposed approach addresses these issues, projects and methods to facilitate decisions based on analogy. the evaluation study was designed with the following The engine is coupled with and assessment component. If a questions in mind: method receives a poor assessment in a certain project context, the method’s characteristics are automatically adapted to • Does each member of the project team understand the reduce the probability of the method being selected in similar basic principles and structure of the method without projects. On the other hand, if a method has successfully been extensive training? applied in a certain project, its characteristics are adapted to • How do project managers assess the potential quantitative 3.4 Method effects of the approach on their practices? Would they In this study, Davis’ technology acceptance model (TAM) use the approach in their setting? [14] was used. TAM’s assessment dimensions ‘perceived • Which problems the project personnel may face when utility’ and ‘perceived ease of use’ were extended while applying the metrics support tool underlying the proposed adding ‘understandability’ as a third dimension. TAM framework? postulates that tool acceptance can be predicted by measuring two dimensions: the perceived usefulness (PU) and the 3.2. Context of the evaluation perceived ease-of-use (PEU) of a system. We used a set of five medium- to large-size software The perceived usefulness of the system expresses the engineering companies developing advanced next-generation “subjective probability that using a specific application home entertainment systems, driver assistance technology for system will increase (the user’s) job performance within passenger cars and military support systems. The usability of these systems has been recognized by the organizations as a an organizational context”, i.e. it is a measure for the crucial quality factor. While usability engineering methods perceived utility of the system. [10] are well-know by these companies to ensure the development of software with high usability, no experience The perceived ease-of-use is the main factor that with usability engineering was available in the engineering influences the acceptance of a system. Davis defines teams. ESPrEE was configured for this environment. perceived ease-of-use as “the degree to which the user Appropriate context and quality models were defined and expects the target system to be free of effort”, i.e. it is a usability engineering methods were captured in method measure for the usability of a system. packages. Resources included successful methods invented in previous industrial engineering projects such as reported in [16], methods distilled from literature on usability engineering Together, perceived ease-of-use and perceived [10, 17, 18], and recent research results such as Spencer’s usefulness constitute the person’s attitude toward using ‘streamlined cognitive walkthrough’[19]. This initial a system. The attitude (A) and the perceived ease-of-use population of the support tool was performed by a team of (PEU) influence the behavioral intention (BI) which can professional usability engineering experts and took about 2.5 be used to predict the actual use of the system. The man-months of effort. The participating organizations were technology acceptance model (TAM) part of a government-supported software engineering research consortium. However, no organization committed to adopt the TAM’s dimension of perceived utility was further divided into approach prior to the evaluation. the following sub-dimensions: 3.3 The Subjects • Perceived compatibility with daily practice • Perceived increase in efficiency All 44 subjects participated in the study on a voluntary basis. • Perceived usefulness Of them, 39 are full-time employees working as software • Perceived support of working tasks engineers for the companies described in section 3.2. Five subjects were graduate students working for the companies on The sub-dimension of perceived utility was measured by a part-time basis. The subjects were involved in developing qualitative effects analysis while usability was examined by highly interactive software systems in a variety of domains, studying user behavior during the user’s interaction with the e.g. driver assistance systems, home entertainment, and metrics support environment [10]. Understandability was military defense systems. Based on their experience with the examined via a knowledge test in which the subjects answered development of highly interactive systems, the subjects were questions on the concepts of the overall approach and the classified into three groups: new employees (NE, 10), metrics support environment. The knowledge test was software engineers (SE, 21), and usability engineers (UE, 13) performed before and after the interaction with the tool to [10]. In the following, these groups are referred to as user study if and how the understanding of the concepts by the groups for reasons of simplicity. subjects changes. To implement the study two basic techniques were deployed: questionnaires and scenario-based task solution. The purpose of the two questionnaires deployed (q1 and q2) are summarized in table 1. Table 1: Purpose of the questionnaires deployed Data collected q1 • Subject characteristics (age, qualification, professional experience) • Typical role of subject in the software engineering process • The subjects knowledge on the concepts of the approach and the metrics environment (pre-test, based on the introductory material supplied) q2 • The subjects knowledge on the concepts of the approach and the metrics environment (post-test, based on the scenario-guided interaction with the metrics environment) • The subjects assessment of the utility and usability of the metrics environment The scenario-based tasks that were specific for each user 4.5 Tasks group forms the core of the evaluation. While the subjects To identify potential problems and study the acceptance of the were working on a task, behavioral data and any comments metrics support environment under realistic conditions, made while thinking out loud were captured as a basis for specific usage scenarios were developed for each user group. improving the metrics environment. Section 4.5 describes the Each scenario reflects the role of the subject and details the tasks that were performed by the subjects while the exact tasks to be solved. evaluation procedure and deployment of the methods is described in section 4.6. All scenarios were embedded in a cover story that set a common background for all scenarios. Scenario S0 is equivalent for all user groups. In S0, the subjects were allowed to freely explore all components of the metrics environment. The other tasks to be solved in each scenario are different among the user groups (Table 2). Table 2: Tasks to be solved in scenarios Tasks NE-S1 Find an explanation of the term usability engineering. Mark the section for later exploration. NE-S2 Find an introductory article about evaluation and tests. Review the material supplied. NE-S3 Open the hypermedia workspace for the project DIGital. Find and open the method package on heuristic evaluation. SE-S1 Browse all method packages available for project DIGital. Find and display a package on heuristic evaluation. Assess the quality of the method heuristic evaluation. SE-S2 Comment the method ‚heuristic evaluation‘. Edit the method ‚heuristic evaluation‘. Extend the method package with a checklist to be used in ‚heuristic evaluation‘. SE-S3 Create a new method package. Fill the package with given raw input material. Specify the meta- data of the methods context model. Link the package to related packages. UE-S1 Create a new project PORTAL. Choose a context model and specify the project characteristics via the context model. Choose appropriate method packages based on the project characteristics specified. Trigger generation of hypermedia workspace for the project PORTAL. UE-S2 Manage access to the hypermedia workspace for the project PORTAL. Invite project team members. Manage access levels of project team members. 3.6. Test procedure and scripts the tasks. After the tasks of all scenarios were completed, questionnaire q2 was handed out to the subject. The evaluation Prior to the evaluation sessions, all the subjects received is concluded with a brief free discussion. introductory material. It described the concepts of the metrics approach and the components of the related environment. Two observers were involved in each evaluation session. One Single subjects, who, for some reason, had no access to the observer recorded the behavioral data, while the other was material prior to the evaluation, were given the opportunity to responsible for writing down comments from the subjects study printouts of the material. Each subject had an individual were thinking aloud. During the session, the subjects worked session, no group sessions were performed. with a laptop with each evaluation lasting of roughly two hours. The evaluation session started with a short introduction of the participants, the procedure, and the objectives of the study. 4. RESULTS AND FINDINGS The subjects were explicitly informed that the goal of the evaluation was to assess the utility of the approach and not the The qualitative effects analysis shows that the proposed capabilities of the participants and that all data would be metrics-based strategy is strongly accepted by the main target treated confidentially. First questionnaire q1 was handed out. group. However the support environment receives higher- Next the tasks to be solved were handed out in form of than-average ratings from all subject groups. scenarios. Scenario S0 was performed by each participant to promote a free exploration of the system. The time for S0 was 4.1 Understandability of the approach limited to 20 minutes. Next, the group-specific scenarios were handed out to the subjects. No time limit was set for task The understanding of the metrics collection approach and the completion. The subjects were encouraged to articulate metrics support environment by the subjects was measured impressions and problems and think aloud while performing before and after usage of the support system via the questionnaires q1 and q2. After reading the introductory material, the average percentage of correct answers was 36%. 3,4 NE 4,3 3,4 Subsequent to performing the scenario-based tasks, this value 3,5 3,6 doubled, being up to 63%. The performance of the groups and the overall performance of the subjects are depicted in figure 3,3 SE 4,4 2. It shows that even the relatively short time of usage of the 3,1 3,5 metrics environment led to a significant increase in the 3,4 understanding of the concepts and components of the 3,7 approach. The increased understanding of the overall approach UE 4,5 3,9 4,1 was lowest in the group of new employees (NE). However this 4,3 can be easily explained since their scenarios (NE-S1-S3) did 3,4 not comprise the usage of all components of the metrics ∅ 4,4 3,3 support system. 3,7 3,7 1 2 3 4 5 43% NE 48% compatibility with daily practice 35% increase in knowledge SE 71% usefulness as a tool 35% increase in the efficiency of UE UE 62% support of task ∅ 36% Figure 3: Results of the qualitative effects analysis 63% (assessment scores for utility dimensions with respect to user group, 1: very low, 5: very high) 0% 20% 40% 60% 80% post-test pre-test The results indicate that the approach and its support environment were generally assessed by all groups as higher- than-average useful and reasonable. All subjects seem to Figure 2: Pre- and post-test scores in knowledge tests with highly appreciate the potential of the approach for increasing respect to subject groups (in percentage of correctly their knowledge of usability engineering methods. The answered questions) dimension ‘task support’ receives the highest scores from the UE group. This corresponds with the pre-configuration of the 4.2 Usefulness of the approach support environment with usability engineering methods and the subjects role as usability engineers. It could be concluded For the qualitative effects analysis, the subjects were asked to that the assessment of this dimension by the other groups assess the properties of the metrics support environments could be further enhanced, if also usability engineering along with the utility dimensions defined in section 3.4Each methods for other areas such as ‘requirements engineering’ or subject filled out questionnaire q2 after performing the methods for enhancing programmer productivity were scenario-specific tasks. For this reason questionnaire q2 integrated into the method repository of the metrics includes a number of items to be rated on a five-level Likert environment. This result underpins the necessity to provide scale [20] for each dimension. Figure 3 sets out the results of benefits for all groups of project personnel involved in metrics the qualitative effects analysis. The bars represent ratings of collection and exploitation. the assessment dimensions. The mean ratings were calculated for each dimension and grouped according to the subject 4.3 Recommendations for usability groups. improvements The behavioral data and user comments recorded during task performance suggest that there is potential for improving the usability of the metrics support environment. The distribution of usability issues identified by subjects across the client components of the metrics support environment are presented in figure 4. Most improvement suggestions are related to the components for process guidance and metrics-based decision support. The high number of usability issues identified for the process guidance component can be partially explained by the fact, that the scenarios of all user groups (NE, SE, UE) included interaction with the process guidance component. deploying the methods in the project. Finally the project 7% assesses the quality of the methods deployed. The approach is supported by a tool, an intranet that offers a web-based 11% support system. The approach has been successfully 20% implemented in industry. 32% Instead of evaluating the approach in an isolated longitudinal 30% case study, a study was performed to examine the acceptance of the approach by practitioners from various organizations. 0% 10% 20% 30% 40% The acceptance was measured in scenario-driven evaluation cross component issues sessions, by capturing the understandability, perceived utility web portal and perceived usability of the approach. The results indicate method assessment and improvement component process guidance component that the approach is accepted by the subject groups examined. metrics-based method selection component We recommend using the study described as a template to estimate the initial acceptance when introducing tool Figure 4: Distribution of the usability issues identified over supported measurement programs into organizations. Such client components of metrics support environment studies can be useful for early identification of acceptance problems that hamper the log-term success of metrics One issue is that more assistance in working with the UE programs. The usability of the metrics environment will be methods is appreciated by the subjects. In particular novice further improved using the feedback gathered. users could profit from concepts such as wizards to augment metrics capturing. Moreover the consistency between the 6. ACKNOWLEDGEMENTS applications should be increased. Finally the parts of the terminology used in the graphical user interfaces of the Part of the empirical study presented in this paper was metrics environment should be revised for more originally conducted at Daimler Chrysler. We would like to comprehensible terms. One example given was that subjects thank Elke Wetzstein and Gerd Jan Tschoepe from the suggested changing the term “project context model” to Institute for Industrial Psychology of the Humboldt University “project characteristics”. Berlin for their support in preparing and conducting the evaluation sessions. Part of this research work was funded by the BMBF (German Ministry for Research and Education) under the project EMBASSI (01IL904I). We would like to 5. A CONCLUDING REMARK thank also the National Science and Engineering Research Council of Canada and Daimler Chrysler for their financial In this paper, an approach for adopting UE methods was support to the human-centered software engineering group at proposed. It consists of a three-phased process. First, usability Concordia University. engineering methods are selected for a new project based on the projects constraints. Then the project team is supported in [5] R. T. Hughes, “Expert Judgment as an Estimating REFERENCES Method,” Information and Software Technology, pp. 67-75, 1996. [1] D. E. Perry, A. A. Porter, and L. G. Votta, [6] F. Niessink and H. Van Vliet, “Measurements “Empirical Studies of Software Engineering: A Should Generate Value, Rather than Data.,” in Proc. Roadmap,” in Proc. International Conference on Sixth IEEE International Symposium on Software Software Engineering (ICSE2000), 2000, pp. 345 - Metrics, 1999, pp. 31-39. 355. [7] O. Laitenberger and H. M. Dreyer, “Evaluating the [2] V. R. Basili, F. Shull, and F. Lanubile, “Building Usefulness and the Ease of Use of a Web-based Knowledge Through Families of Experiments,” Inspection Data Collection Tool,” in Proc. Fifth IEEE Transactions on Software Engineering, vol. IEEE International Symposium on Software Metrics, 25, pp. 456-473, 1999. 1998. [3] D. R. Goldenson, A. Gopal, and T. Mukhopadhyay, [8] S. Komi-Sirviö, P. Parviainen, and J. Ronkainen, “Determinants of Success in Software Measurement “Measurement Automation: Methodological Programs: Initial Results,” in Proc. Sixth IEEE Background and Practical Solutions - A Multiple International Symposium on Software Metrics, Case Study,” in Proc. 7th IEEE International 1999, pp. 10-21. Software Metrics Symposium, 2001, pp. 306-316. [4] B. G. Silverman, “Software Cost and Productivity [9] L. Rosenberg and L. Hyatt, “Developing a Improvements: An Analogical View,” IEEE Successful Metrics Program,” in Proc. 8th Annual Computer, vol. May 1985, pp. 86-96, 1985. Software Technology Conference, 1996. [10] J. Nielsen, Usability Engineering: Morgan Kaufman Publishers, 1994. [11] L. Briand, K. El Emam, and S. Morasca, “On the [20] C. M. Judd, E. R. Smith, and L. H. Kidder, Research Application of Measurement Theory in Software Methods in Social Relations, 6 ed: Harcourt Brace Engineering,” Empirical Software Engineering, vol. Jovanovich College Publishers, 1991. 1, pp. 61-88, 1996. [21] G. F. Smith and S. T. March, “Design and Natural [12] B. Minto, The Pyramid Principle - Logic in Writing Science Research on Information Technology,” and Thinking, 3rd ed. London: Minto International Decision Support Systems, vol. 15, pp. 251-266, Inc., 1987. 1995. [13] A. Birk, T. Dingsøyr, and T. Stålhane, “Postmortem: [22] R. D. Galliers and F. F. Land, “Choosing Never Leave a Project without it,” IEEE Software, Appropriate Information Systems Research vol. 19, pp. 43-45, 2002. Methodologies,” Communications of the ACM, vol. [14] F. D. Davis, “A Technology Acceptance Model for 30, pp. 900-902, 1987. Empirically Testing New End-User Information [23] T. DeMarco and T. Lister, Peopleware: Productive Systems: Theory and Results,,” in MIT Sloan Projects and Teams, 2. ed. New York: Dorset House School of Management. Cambridge, MA, USA: MIT Publishing, 1999. Sloan School of Management, 1986. [24] Y. Malhotra and D. F. Galletta, “Extending the [15] B. A. Kitchenham, “Evaluating Software Technology Acceptance Model to Account for Engineering Methods and Tools,” ACM SIGSoft Social Influence: Theoretical Bases and Empirical Software Engineering Notes, pp. 11-15, 1996. Validation,” in Proc. Thirty-Second Annual Hawaii [16] E. Metzker and M. Offergeld, “An Interdisciplinary International Conference on System Sciences, 1999, Approach for Successfully Integrating Human- pp. pp. 1006. Centered Design Methods Into Development [25] A. Cockburn, Agile Software Development: Processes Practiced by Industrial Software Addison Wesley Longman, 2001. Development Organizations,” in Engineering for [26] N. Juristo and A. M. Moreno, Basics of Software Human Computer Interaction: 8th IFIP International Engineering Experimentation. Dordrecht: Kluwer Conference, EHCI 2001(EHCI' 01), Lecture Notes in Academic Publishers, 2001. Computer Science, R. Little and L. Nigay, Eds. [27] Standish Group. “CHAOS Chronicles or CHAOS: A Toronto, Canada: Springer, 2001, pp. 21-36. Recipe For Success”. 1995. [17] L. L. Constantine and L. A. D. Lockwood, Software [28] Bayer, J. and Melone, N. Adoption of Software for Use: A Practical Guide to the Models and Engineering Innovations in Organizations. Technical Methods of Usage-Centered Design: Addison- Report CMU/SEI-89-TR-017, Software Engineering Wesley, 1999. Institute, Carnegie Mellon University. [18] D. J. Mayhew, The Usability Engineering Lifecycle: [29] Desmarais M.C., Leclair R., Fiset J.Y., Talbi H. A Practioner' s Handbook for User Interface Design: “Cost-Justifying Electronic Performance Support Morgan Kaufman Publishers, 1999. Systems.” Communications of the ACM, Vol. 40, [19] R. Spencer, “The Streamlined Cognitive No. 7, July 1997. Walkthrough Method: Working Around Social [30] Howard R. “Software Process Maturity: Measuring Constraints Encountered in a Software Development Its Impact on Productivity and Quality.” IEEE Company,” in Proc. Conference on Human Factors International Software Metrics Symposium, May in Computing Systems (CHI' 00), 2000, pp. 353-359. 1993.