Bringing Usability Evaluation into Practice: Field Studies in Two Software Organizations Jakob Otkjær Bak Kim Nguyen Peter Risgaard Jan Stage TARGIT A/S Logica EUCNORD Aalborg University Aalborgvej 94 Fredrik Bajers Vej 1 Hånbækvej 50 Department of DK-9800 Hjørring DK-9220 Aalborg East DK-9900 Frederikshavn Computer Science Denmark Denmark Denmark DK-9220 Aalborg East jb@targit.com kimmeren@gmail.com pri@eucnord.dk Denmark jans@cs.aau.dk ABSTRACT organizations are still not conducting any form of usability This paper explores how obstacles to usability evaluations in a evaluation in their development process [21]. software organization can be affected. We present two field There have been considerable efforts to affect the obstacles that studies, each conducted in a software organization that had no prevent these software organizations from deploying usability previous experience with usability evaluation. In each study, we evaluation techniques. A major approach has provided techniques first interviewed key stakeholders to identify their opinion about that are supposed to ease the deployment. This approach has only significant obstacles to conducting usability evaluations. Then we had limited success and mostly in software organizations that are demonstrated the benefits of a usability evaluation by evaluating already conducting usability evaluations. The reason may be that the usability of one of their software products, while being most of the proposed techniques are highly technical and designed observed by the developers, and presenting the evaluation results by experts to be used by experts or at least by well-trained to the developers. Finally, the key stakeholders were interviewed professionals [3]. again to establish the effect of the demonstration. The demonstration of benefits had a positive effect on some of the key A basically different approach has been to affect key obstacles, while others were unaffected. One organization stakeholders’ attitudes to usability evaluation. This has mostly expressed future plans for conducting usability evaluations while been done on a general level by documenting how other the other was still reluctant. organizations have benefitted from deploying usability evaluation techniques in their development process. A study found that collection of user data, setting usability goals and conducting Categories and Subject Descriptors usability walkthroughs had a positive effect [13]. Another study H.5.2. [Information Interfaces and Presentation]: User documented that deployment of user-centered design in the Interfaces – Evaluation/methodology. K.6.1 [Management of development life cycle of a software company, specifically by Computing and Information Systems]: Project and People integration of use cases in the development process, supported Management – Staffing, Systems development, Training. decision making [17].Karat provides evidence about the cost and benefit of usability evaluation [11]. The difficulty is, however, General Terms that often the cost is paid by the software organization, while the Experimentation, Human Factors. benefit is gained by the customer. Yet there are exceptions. A study established that evaluation of software for usability can lead to increased sale of products [12]. Another study demonstrated Keywords that the need for user support decreased with better usability [20]. Usability evaluation, software organizations, development Experience with deployment of usability work is usually focused practice, empirical study. on larger organizations. However, a study in a smaller organization also presents activities that were successful [5]. 1. INTRODUCTION Another study focused specifically on usability evaluation and Usability is a fundamental attribute of interactive systems [7], and concludes that quick, cheap and effective evaluations can be it is critical to their success or failure on the market [10]. conducted [19]. Evaluation of usability has been documented to be economically Only few studies have focused on affecting the attitudes to feasible because of increased sales [11], increased user usability evaluation on a specific level; that is in a particular productivity [12], decreased training costs [4] and decreased software organization. This paper reports from two field studies, needs for user support [20]. Despite these facts, many software where we tried to overcome obstacles to usability evaluation by affecting the attitudes of key stakeholders. This was done by Permission to make digital or hard copies of all or part of this work for demonstrating how that particular organization could benefit from personal or classroom use is granted without fee provided that copies are deploying usability evaluation in their development process. In not made or distributed for profit or commercial advantage and that section 2 we present related work on affecting obstacles to copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, usability evaluation. Section 3 presents the method used in of the requires prior specific permission and/or a fee. two field studies. In section 4, we provide the results from the Conference’04, Month 1–2, 2004, City, State, Country. field studies. In section 5, we discuss our results. Finally, section Copyright 2004 ACM 1-58113-000-0/00/0004…$5.00. 6 provides the conclusion. 2. RELATED WORK Participants. Three participants from company A were involved The majority of studies that try to affect obstacles to usability in the collaboration; a section manager, a developer and a user evaluation focuses on usability guidelines and methods for consultant. The section manager was in charge of the incorporating usability in the development process. Gould and development team, the developer was responsible for the user Lewis were among the first to provide guidelines for the interface design and the user consultant was responsible for deployment of usability in the design process [6]. A study contact to users and for their education. questioned the relevance of guidelines to usability and discussed Procedure. The study was conducted in 3 steps. The first step appropriate sources of guidance [3]. Overall guidelines directed at was an initial meeting with the section manager of the care the developers are widely used. A study identified the gap management solutions department, the user consultant and the between designers and users as the major obstacle to deploying developer responsible of the user interface design. The purpose usability and suggested usability engineering methodologies to was to determine obstacles to usability evaluation in the company help overcome this obstacle [23]. Grudin presented suggestions to and select the part of the system to evaluate. After the meeting, overcome this gap based on long term experiences [8]. the three participants were asked to write down weaknesses and Solutions to overcome organizational obstacles to usability obstacles to integration of usability in their development process. evaluation are presented in some papers. They tend to advise what The second step was the evaluation of the system. We used the usability practitioners can do to sell usability to the organization. Instant Data Analysis (IDA) method [14]. After the evaluation, Mayhew suggests three phases and for each phase how, why and the test results were emailed to the three participants and what to do to sell usability [16]. A study concluded that subsequently presented in combination with redesign proposals. communicating the message of usability is not enough; the facts must be solid and documented [24]. Another study complements After a month, the third step was conducted. A meeting was held, this by concluding that experiences with usability have to be where the developer and user consultant were interviewed about presented in a way that appeals to upper management’s mindset their experiences with the usability evaluation and its result. They with emphasis on the monetary benefits [1]. were also asked if any changes had been made to the system or their work process. A telephone interview was conducted with the Resource-related obstacles have also been studied. Based on section manager who was asked the same questions. experiences from several organizations, Nielsen states that there are considerable monetary benefits from conducting usability Setting. The meetings were held in a conference room in the evaluations [18]. A study emphasized that automation is a way to company. The usability tests were conducted with real users and complement existing usability evaluation methods [9]. took place at the users’ workplaces. The user consultant and developer observed the first test session. Only a few researchers have tried to measure the effect of deploying new usability methods in software organizations. One Data collection and analysis. We recorded of the interviews and study concluded that nurturing the developers’ skills in user- collected the forms with opinions about weaknesses and centered design was a major factor in developing more usable obstacles. Each interview was conducted according to an systems [22]. A different study provided a usability engineer to a interview guide [15]. Later, the recorded interviews were software organization. This helped developers shift focus toward condensed using a method called “condensation of meaning” [15], design and assume a role as the users’ advocate [2]. and this outcome was then analyzed. The analysis was conducted by two persons separately. These two persons individually pointed out statements from the condensed data and grouped them 3. METHOD into obstacles. Finally, the they negotiated a joint list of We have conducted field studies in two software organizations, weaknesses and obstacles. where we tried to demonstrate the benefits of usability evaluation in an ongoing development process. 3.2 Company B The company produced wireless technology. At the time of the 3.1 Company A study, it was divided into four units: technology, consumer The company had, at the time of the study, 150 employees with products, network systems and healthcare. There were 180 headquarter in Denmark and branches in Canada, USA and employees, most of them located in the headquarter in Denmark. Romania. Its business was separated in four units: supply chain There were branches in USA, Hong Kong and Romania. Our solutions, postal solutions, airport solutions and care management collaboration with this company was carried out with the solutions. Our collaboration was with the care management healthcare unit that had 10 employees, where 5 of them were solution unit that had 12 employees, of which 7 were software developers. The system evaluated, was a device for home use by developers. The system we evaluated was a planning module for a elderly people to send health data to a monitoring center. This healthcare management system used by nurses and home system was recently developed and had a simple user interface. assistants to plan both care for citizens and staff working hours. The system had been developed some years before and updated The company’s motivation for participating was an initial interest regularly. Initially, it had a non-graphical user interface. Later, it in usability evaluation, based on knowledge about another was supplemented with a graphical user interface. company’s successful experiences. Furthermore, the customer of the product in question required a usability evaluation. The company’s motivation for participating was curiosity about usability evaluation and a desire to see if it could be integrated in Participants. Throughout the collaboration, the main contact the development process without being too costly. It was not person was the user consultant for the product in question. The triggered by customer demands. user consultant was responsible for verification and quality assessment of the product. In addition, a developer observed and Setting. Most meetings were held at company B. The post- provided technical assistance during the usability evaluation. evaluation meeting was held at the university, and the evaluation was conducted in our usability laboratory. Procedure. The study was conducted in three steps. The first step was an introductory meeting with the user consultant. The Data collection and analysis. The interviews with the user purpose was to gain an overview of the product and clarify mutual consultant were video recorded. Each interview was based on an expectations. interview guide [15]. The recordings were processed with “condensation of meaning” [15]. The analysis was done in exactly The second step was the usability evaluation. The results from the the same way as with company A. evaluation were emailed the day after the evaluation. Interviews were made shortly after. The results from the evaluation were presented along with redesign proposals at a meeting. 4. Results This section presents the results of our study in the two software The third step involved two parts. Six months after the evaluation, organizations. The results are summarized in Table 1. the user consultant was interviewed to assess the effect. Two months later, the user consultant was interviewed again about the current obstacles in the company. 4.1 Resource Demands The two software organizations initially had some obstacles in Table 1. Essential statements from company A and B before common. Both were convinced that usability evaluation was very and after the trial evaluation. time consuming and costly, as stated by the section manager in Obstacle Initial statements Final statements company A. The developer and user consultant also agreed that Resource Company A: “It would be Company A: “I can see it time and money were major obstacles. The main obstacle was the demands a high increase in the price being conducted on special expectation about the time it would take to conduct the and maybe delay the products or occasions, evaluations and make software changes. development two weeks or places where we deem it more. The customer should extra important or are Company B was looking for an inexpensive opportunity to then be ready to pay suspicious about a poorly evaluate the usability of their product. The resource demands of 100.000 kr. more than designed user interface. But usability evaluation were underlined by the user consultant from now.” nothing regularly, there is company B in the following way; “The resource demand will typically no time for it in our always be an obstacle” and “... when we don’t know what is Company B: “...when we development process.” don’t know what is needed needed to conduct an evaluation, it will probably take too much of to conduct an evaluation, Company B: “There are no our time”. then it will probably take resources for usability tests, In the final meetings, both organizations still stated resources in too much of our time.” we really want to, but there’s no money for it at the relation to time and money as being a main obstacle. It was most moment.” prominent in company B, where the user consultant made Lack of Company A: “Knowledge Company A: “... the statements such as “We don’t have the resources to conduct a knowledge about the right solution is evaluations gave an insight usability test.” and “... it would take too much time for us ... we an obstacle to integrating into how the system was don’t have the experience”. usability evaluation in the actually used by a development process.” prospective end user.” Company A expressed this obstacle both in the interviews and the forms. In a discussion of gains from usability evaluations, the user Company B: “...we have Company B: “I have gained consultant said “... it would be too expensive to reveal the very little knowledge about some knowledge, but not problems this way”. When asked about the downsides of usability usability evaluations.” enough to conduct an evaluation, the developer stated “I still think a lot of time is spent evaluation on my own.” on it. You really don’t have much time here”. The user consultant User Company A: “The users Company A: “The usability stressed that resources is the most important factor “It all comes involve- don’t think enough about problems occurred ment what they are shown. If unexpectedly, and related down to resources; the bottom line is always the focus point.” they see something smart, more to user errors or lack Resource demand as a main obstacle was also apparent in the they want it. They don’t of users’ understanding.” forms. The section manager did only consider it relevant for think about the problems a new solution can special cases. On the other hand, he was surprised by the prompt generate.” delivery of results, and the user consultant concurred “The results Structure Company A: “Often, the Company A: “...the were delivered very fast. I assumed it would take 3-4 weeks.” of the database layer and other development system and The resource demand of introducing usability evaluation was system function-related layers are environment is not up to limiting the user interface. date.” initially one of the major obstacles for both companies. The use of You lock a lot in the the low-cost method [14], gave the user consultant from company beginning of the project.” A an entirely different view “It changed my idea of how much Manage- Company A: “I actually time usability evaluations take.” The section manager’s attitude ment don’t think the need for also changed. The change in company B was even more interests usability evaluations is prominent as the user consultant expressed “If there is money for apparent to upper usability evaluation, we will certainly deploy it in the management. Usability is development process”. taken for granted...” 4.2 Lack of Knowledge The insights gained from the usability evaluations were also Both companies stated that their knowledge of usability mentioned in the final meeting with the user consultant “You can evaluation was initially at a very low level. Company B had some tell if the system is intuitive to use, if they can push the right knowledge from another software organization that conducted buttons and read the display. These are things we cannot answer usability evaluations, but only on the general level that usability by discussing it in the development department. It is things we evaluation can give useful information to developers. They did don’t think about.” The user consultant also stated that the results not have any knowledge about usability work practices. Company from the evaluation were of great use in her daily work. In certain A had some knowledge from another department, where a design discussions, she was able to use the results as examples of usability evaluation had been conducted once, but no evaluations actual user behaviour. The introduction also had an impact on the had ever been done in the care management unit. The lack of user consultant’s knowledge about usability evaluations. Initially, knowledge also extended to the users’ application of the system she had no knowledge about it, but in the final meeting she as the section manager stated “It would be great to get the mentioned; “If we need a test of a future product, we know what knowledge into the organization; this could be used by the usability evaluation is and what it can be used for, and we know developers to make the product more usable for the end user.” when to test. So we can use this process for a lot of purposes.” The developer agreed; “We lack knowledge about the users’ professional world.” 4.3 User Involvement The two software organizations differed considerably in their Lack of knowledge about usability evaluations was still expressed thinking about end users. Company B wanted the end users to be as an obstacle for both companies after the demonstration of able to use the product with a minimum of training and a very usability evaluation. For company A, this applied to knowledge small and easy to read manual. In company A, the user consultant about evaluations and usability in general. The developer stated expressed “Our system is so complex that training is a necessity; “As a developer, I find it hard to decide when to involve users in in no way would the end user ever be able to use the system the development process.” In relation to the question when users without the training we give them.” should be involved, the section manager said “Usability The users were contributing with proposals for changes to the evaluations can only be conducted in the final phases of a system developed by company A, but this was actually considered development process.” The lack of knowledge about usability more of a complication. For example, the developer mentioned evaluation was also expressed by the user consultant from “The users lack knowledge about the development process and company B “I have gained some knowledge, but not enough to the time it takes”, and the user consultant stated “The users do not conduct an evaluation on my own.” have an overview of the system and its structure, and they might The lack of knowledge regarding the users’ application of the disagree about new functionality.” The section manager also system as well as usability evaluation in general was the obstacle mentioned difficulties related to the involvement of the end users that was affected most in our study. An example of this was given “The users don’t think enough about what they’re shown. If they by the user consultant in company A “...three of us discussed a see something smart, they want it. They don’t think about the design solution, but we were not able to agree, so we called a user problems a feature can generate.” and found the answer ... if you want something tested, you can After the demonstration, obstacles relating to user involvement just grab a user and ask for his or her opinion.” This approach had were only expressed in company A. The user consultant spoke of not been employed prior to our demonstration of usability their users as being too numerous and geographically spread “... evaluation. The demonstration made the employees experience to reach out to 50% or even 10% of our users, that cannot be that users can be involved in a constructive way in the done. We have too many users.” Furthermore, usability evaluation development process. Other statements from the user consultant of a product during development would be hard to conduct, in company A underlined that the usability evaluation gave because they would be forced to use inexperienced users, which insight into the users’ work routines “Your tests show that it has a would make the tests difficult “...it would most likely “drown” in lot to do with work routines, and that has given us motivation for explanations of the new functions.” The section manager following up in the next release.” The importance of the expressed a similar concern about involving users in an evaluation evaluators was also stressed “Your tests show some subconscious “For the users to be involved in a test, they would have to be things, and the users don’t catch them themselves. There has to be pulled away from their work. That costs money for the customer an observer to catch those things.” and will be a burden.” Company A was also reluctant to involve In the post-evaluation meeting and the final meeting with users, because their understanding of the problems found in the company B, several findings pertaining to the lack knowledge usability evaluation was that it was the users’ lack of knowledge were emphasized. The user consultant and developer expressed a about the system that caused the problems, as expressed by the general satisfaction with the evaluation. Observing all sessions as developer “The usability problems occurred unexpectedly, and they happened, gave them “… an insight into the way the system related more to user errors or lack of users’ understanding.”. was actually used by a prospective end user”, as expressed by the The introduction of usability evaluation gave the participants user consultant. The evaluations revealed problems that had not from company A a deeper insight into the users’ way of using the previously been identified by the user consultant or developer. system. Yet this insight also emphasized user involvement as an Both of them agreed upon the usefulness of this insight and obstacle. For example, the user consultant expressed it this way thereby of the evaluations. In the final meeting with the user “Are the problems occurring just because the users have adopted a consultant, these attitudes and viewpoints were still completely wrong work routine ... the users lack an understanding of the use intact. She said “When our new product is almost finished, it will of the system.” be evaluated in the same manner ...” 4.4 Structure of the System Company A’s motivation for participating in the experiment was Company A had an obstacle regarding the structure of the system. curiosity about the nature of usability evaluation and its practical This was expressed by the section manager. The developer also use. Company B had a need to gain knowledge about usability mentioned the difficulties with the system structure “The system evaluation because of customer demands. This difference in is used in different ways. With major changes there is a risk of motivation might have had an impact on the obstacles identified. removing existing functionality and introducing new errors in For example, the number of obstacles identified in company B properly working parts of the system.” Although the structure in was only two, while it in company A was four before the itself is not an obstacle to usability evaluation, correcting the introduction and five after. Moreover, an obstacle identified in problems found could be very difficult as expressed by the user company A related to the users and the difficulties of meeting consultant “Some parts of the system are hardcoded and cannot be with the users. Company B also had difficulties with creating changed, although the users see it as a small change.” contact with users, but it was not expressed as an obstacle. Overall, company A had a tendency to see obstacles rather than The introduction of usability evaluation had no tangible effect on benefits of usability evaluation, which indicate a lack of this obstacle, but reveals a need to prepare developers for motivation that makes it even more difficult to deploy usability potential changes in the system structure. evaluation. 4.5 Management Interests To increase the motivation, a software organization needs to The participants from company A expressed an obstacle in experience that usability evaluation can fulfill relevant needs. relation to management, but only after the demonstration. When Company B was more willing to deploy usability evaluations than asked how apparent the importance of usability was for company A after the demonstration. Another factor relating to the management, the developer said “I actually don’t think it’s greater effect might have been that the employees from company apparent for management. Usability is taken for granted ...” The B observed all the sessions of the usability evaluation, whereas user consultant stated in relation to this obstacle “My attitude and the employees from company A, observed only one session. The position to the matter isn’t opposed to it, but reprioritization has experiences with company A also showed that decisions to to come from the management level.” In company B, the obstacle integrate and prioritize evaluations had to come from top level of management interests was also expressed by the user management. Therefore it could be beneficial to include consultant “Management has decided to postpone usability participants from that level in a demonstration. evaluations until sales have gone up.” This obstacle was not identified in the initial statements, but only 6. Conclusion in the final statements. It emerged because of our direct question The purpose of this study was to observe how the introduction of whether the company would consider deploying usability usability evaluation affects significant obstacles to usability evaluation in the development process in the near future. evaluation in software organizations. To inquire into this, a usability evaluation was demonstrated to two software organizations. This included that we conducted a usability 5. Discussion evaluation and presented the evaluation results to the two The results of this study show that specific obstacles such as the software organizations. resource demands and lack of knowledge about users and usability evaluation methods have been affected. The quick The results show that the introduction of usability evaluation feedback from the evaluation to the software organization was a provided the software organizations with insight into the users’ significant reason why company A would consider usability use of the system. Furthermore, they experienced that usability evaluation in the future. The fact that the participants from the evaluations are not nearly as resource demanding as expected. two software organizations observed one or more test sessions This illustrates that the stakeholders’ attitudes to these obstacles increased their insight into the methodology and the users’ ways were affected. However, none of the obstacles identified in the of using the system considerable. This was clear from the positive two software organizations were completely resolved. Two of the comments that participants from both companies made about initial obstacles, user involvement and structure of the system, observing the tests. were not affected by the demonstration of usability evaluationl. The fact that the software organizations were affected by This study shows that it is possible to motivate software observing the benefits of usability evaluation is a valuable organizations toward usability evaluation. This was achieved contribution of this study, and should be a point of focus in through the approach in which the companies’ products were further research. This is also where this study differs from related evaluated. This underlines the relevance of research in this topic work within this area. As mentioned in section 2, many of the based on other approaches than providing guidelines and previous studies have focused on providing guidelines or principles, which has been covered to a great extent. principles for deploying usability practices. The purpose of these There are some important limitations to our study. The two has been to ease the deployment of usability evaluation in the software organizations were quite similar. Also, we interviewed development process [3, 6]. In contrast, the purpose of our study quite few persons in these organizations. In both organizations, was to let company representatives observe the benefits of we focused in particular on the benefits and time taken; we did usability evaluation. not deal explicitly with the costs for the two organizations. The An important factor when deploying usability evaluation is the main source of data was interviews combined with forms in one motivation of the software organization. In our study we observed of the organizations. Finally, the specific method used in the a different motivation between the two software organizations. evaluations might have affected the results. It would be interesting to extend the study to more organizations and Interaction, Elsevier Science Inc., 2nd edition, 1997, pages stakeholders and use different methods both for data collection 767- 778. ISBN: 0444818626. and for the evaluation. [13] Brenda Kerton. Introducing usability at London Life Acknowledgments. The research behind this paper was partly insurance company: a process perspective. In CHI ’97: CHI financed by the Danish Research Councils (grant numbers 2106- ’97 extended abstracts on Human factors in computing 04-0022 and 2106-08-0011). We are very grateful to the two systems, ACM Press, 1997, pages 77-78. ISBN: 0-89791- software organizations and the stakeholders that participated in 926-2. the study. We would also like to thank the anonymous reviewers [14] Jesper Kjeldskov, Mikael B. Skov, Jan Stage. Instant data for their comments and advice. analysis: conducting usability evaluations in a day. In NordiCHI ’04: Proceedings of the third Nordic conference References on Humancomputer interaction, ACM Press, 2004. ISBN: 1- [1] Sarah Bloomer, Rachel Croft. Pitching usability to your 58113-857-1. organization. In interactions, ACM Press, 1998, (4,6), pages [15] Steinar Kvale. Interview - En introduktion til det kvalitative 18-26. ISSN: 1072-5520. forskningsinterview. Hans Reitzel, 2.edition, 1998. ISBN: [2] Inger Boivie, Jan Gulliksen, Bengt G¨oransson. The lonesome 87-412-2816-2. cowboy: A study of the usability designer role in systems [16] Deborah J. Mayhew. Business: Strategic development of the development. In Interacting with Computers, Elsevier usability engineering function. In interactions, ACM Press, Science B.V., (18,4), 2006, pages 601-634. 1999, (6,5), pages 27-34. ISSN: 1072-5520. [3] Jim Carter. Incorporating standards and guidelines in an [17] Karsten Nebe, Lennart Gr¨otzbach. Aligning user centered approach that balances usability concerns for developers and design activities with established software development end users. In Interacting with Computers, Elsevier Science practices. In NordiCHI ’06: Proceedings of the 4th Nordic B.V., (12,2), 1999, pages 179-206. conference on Human-computer interaction, ACM Press, [4] Susan M. Dray, Clare Marie Karat. Human factors cost 2006, pages 485-486. ISBN: 1-59593-325-5. justification for an internal development project. In [18] Jakob Nielsen. Why GUI panic is good panic. In Costjustifying usability, Academic Press, Inc., 1994, pages interactions, ACM Press, 1994, (2,1), pages 55-58. ISSN: 111-122. ISBN: 0-12-095810-4. 1072-5520. [5] Carola B. Fellenz. Introducing usability into smaller [19] Jerilyn Prescott, Matt Crichton. Usability testing: a quick, organizations. In interactions, ACM Press, 1997, pages 29- cheap, and effective method. In SIGUCCS ’99: Proceedings 33. ISSN: 1072-5520. of the 27th annual ACM SIGUCCS conference on User [6] John D. Gould, Clayton Lewis. Designing for usability: key services, ACM Press, 1999, pages 176-179. ISBN: 1-58113- principles and what designers think. In Communications of 144-5. the ACM, ACM Press, 1985, (28,3), pages 300-311. [20] S. Reed Who defines usability? You do!. In PC//Computing, [7] Toni Granollers. User Centred Design Process Model. (Dec), 1992, pages 220-232. ISBN: 0-12-095810-4. Integration of Usability Engineering and Software [21] Stephanie Rosenbaum, Sarah Bloomer, Dave Rinehart, Engineering. In Proceedings of interact 2003. Found at: Janice Rohn, Ken Dye, Judee Humburg, Jakob Nielsen, http://www.griho.udl.es/publicacions/2003/Doctoral%20 DennisWixon. What makes strategic usability fail?: lessons Consortium%20(Interact%2003).pdf, last seen April 11th learned from the field. In CHI ’99: CHI ’99 extended 2007. abstracts on human factors in computing systems, ACM [8] Jonathan Grudin. Obstacles to user involvement in software Press, 1999, pages 93-94,. ISBN: 1-58113-158-5. product development, with implications for CSCW. In [22] Ahmed Seffah, Alina Andreevskaia. Empowering software International Journal of Man-Machine Studies, Academic engineers in human-centered design. In ICSE ’03: Press Ltd., 1991, (34,3), pages 435-452. ISSN: 0020-7373. Proceedings of the 25th International Conference on [9] Melody Y. Ivory, Marti A Hearst. The state of the art in Software Engineering, IEEE Computer Society, 2003, pages automating usability evaluation of user interfaces. In ACM 653-658. Comput. Surv., ACM Press, 2001, (33,4), pages 470-516. [23] Desir Sy. Bridging the communication gap in the workplace ISSN: 0360-0300. with usability engineering. In SIGDOC ’94: Proceedings of [10] Claire Marie Karat. A business case approach to usability the 12th annual international conference on Systems cost justification. In Cost justifying usability, Academic documentation, ACM Press, 1994, pages 208-212. ISBN: 0- Press, Inc., 1994. ISBN: 0-12-095810-4. 89791-681-6. [11] Clare Marie Karat. A Comparison of User Interface [24] Leslie Tudor. Obstacles to user involvement in software Evaluation Methods. In Usability Inspection Methods, 1994, product development, with implications for CSCW. In pages 203-233. ISBN: 0-471-01877-5. Human factors: does your management hear you?, ACM Press, 1998, (5,1), pages 16-24. ISSN: 1072-5520. [12] Clare Marie Karat. Cost-justifying usability engineering in the software life cycle. In Handbook of Human-Computer