Bringing Usability Evaluation into Practice: Field Studies in Two Software Organizations

Abstract

This paper explores how obstacles to usability evaluations in a software organization can be affected. We present two field studies, each conducted in a software organization that had no previous experience with usability evaluation. In each study, we first interviewed key stakeholders to identify their opinion about significant obstacles to conducting usability evaluations. Then we demonstrated the benefits of a usability evaluation by evaluating the usability of one of their software products, while being observed by the developers, and presenting the evaluation results to the developers. Finally, the key stakeholders were interviewed again to establish the effect of the demonstration. The demonstration of benefits had a positive effect on some of the key obstacles, while others were unaffected. One organization expressed future plans for conducting usability evaluations while the other was still reluctant.

Keywords

• eol>Usability evaluation
• software practice
• empirical study

1. INTRODUCTION

Usability is a fundamental attribute of interactive systems [ 7 ], and it is critical to their success or failure on the market [ 10 ]. Evaluation of usability has been documented to be economically feasible because of increased sales [ 11 ], increased user productivity [ 12 ], decreased training costs [ 4 ] and decreased needs for user support [ 20 ]. Despite these facts, many software Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

Conference’04, Month 1–2, 2004, City, State, Country.

Copyright 2004 ACM 1-58113-000-0/00/0004…$5.00. organizations are still not conducting any form of usability evaluation in their development process [ 21 ].

There have been considerable efforts to affect the obstacles that prevent these software organizations from deploying usability evaluation techniques. A major approach has provided techniques that are supposed to ease the deployment. This approach has only had limited success and mostly in software organizations that are already conducting usability evaluations. The reason may be that most of the proposed techniques are highly technical and designed by experts to be used by experts or at least by well-trained professionals [ 3 ].

A basically different approach has been to affect key stakeholders’ attitudes to usability evaluation. This has mostly been done on a general level by documenting how other organizations have benefitted from deploying usability evaluation techniques in their development process. A study found that collection of user data, setting usability goals and conducting usability walkthroughs had a positive effect [ 13 ]. Another study documented that deployment of user-centered design in the development life cycle of a software company, specifically by integration of use cases in the development process, supported decision making [ 17 ].Karat provides evidence about the cost and benefit of usability evaluation [ 11 ]. The difficulty is, however, that often the cost is paid by the software organization, while the benefit is gained by the customer. Yet there are exceptions. A study established that evaluation of software for usability can lead to increased sale of products [ 12 ]. Another study demonstrated that the need for user support decreased with better usability [ 20 ]. Experience with deployment of usability work is usually focused on larger organizations. However, a study in a smaller organization also presents activities that were successful [ 5 ]. Another study focused specifically on usability evaluation and concludes that quick, cheap and effective evaluations can be conducted [ 19 ].

Only few studies have focused on affecting the attitudes to usability evaluation on a specific level; that is in a particular software organization. This paper reports from two field studies, where we tried to overcome obstacles to usability evaluation by affecting the attitudes of key stakeholders. This was done by demonstrating how that particular organization could benefit from deploying usability evaluation in their development process. In section 2 we present related work on affecting obstacles to usability evaluation. Section 3 presents the method used in of the two field studies. In section 4, we provide the results from the field studies. In section 5, we discuss our results. Finally, section 6 provides the conclusion.

2. RELATED WORK

The majority of studies that try to affect obstacles to usability evaluation focuses on usability guidelines and methods for incorporating usability in the development process. Gould and Lewis were among the first to provide guidelines for the deployment of usability in the design process [ 6 ]. A study questioned the relevance of guidelines to usability and discussed appropriate sources of guidance [ 3 ]. Overall guidelines directed at the developers are widely used. A study identified the gap between designers and users as the major obstacle to deploying usability and suggested usability engineering methodologies to help overcome this obstacle [ 23 ]. Grudin presented suggestions to overcome this gap based on long term experiences [ 8 ]. Solutions to overcome organizational obstacles to usability evaluation are presented in some papers. They tend to advise what usability practitioners can do to sell usability to the organization. Mayhew suggests three phases and for each phase how, why and what to do to sell usability [ 16 ]. A study concluded that communicating the message of usability is not enough; the facts must be solid and documented [ 24 ]. Another study complements this by concluding that experiences with usability have to be presented in a way that appeals to upper management’s mindset with emphasis on the monetary benefits [ 1 ].

Resource-related obstacles have also been studied. Based on experiences from several organizations, Nielsen states that there are considerable monetary benefits from conducting usability evaluations [ 18 ]. A study emphasized that automation is a way to complement existing usability evaluation methods [ 9 ]. Only a few researchers have tried to measure the effect of deploying new usability methods in software organizations. One study concluded that nurturing the developers’ skills in usercentered design was a major factor in developing more usable systems [ 22 ]. A different study provided a usability engineer to a software organization. This helped developers shift focus toward design and assume a role as the users’ advocate [ 2 ].

3. METHOD

We have conducted field studies in two software organizations, where we tried to demonstrate the benefits of usability evaluation in an ongoing development process.

3.1 Company A

The company had, at the time of the study, 150 employees with headquarter in Denmark and branches in Canada, USA and Romania. Its business was separated in four units: supply chain solutions, postal solutions, airport solutions and care management solutions. Our collaboration was with the care management solution unit that had 12 employees, of which 7 were software developers. The system we evaluated was a planning module for a healthcare management system used by nurses and home assistants to plan both care for citizens and staff working hours. The system had been developed some years before and updated regularly. Initially, it had a non-graphical user interface. Later, it was supplemented with a graphical user interface.

The company’s motivation for participating was curiosity about usability evaluation and a desire to see if it could be integrated in the development process without being too costly. It was not triggered by customer demands.

Participants. Three participants from company A were involved in the collaboration; a section manager, a developer and a user consultant. The section manager was in charge of the development team, the developer was responsible for the user interface design and the user consultant was responsible for contact to users and for their education.

Procedure. The study was conducted in 3 steps. The first step was an initial meeting with the section manager of the care management solutions department, the user consultant and the developer responsible of the user interface design. The purpose was to determine obstacles to usability evaluation in the company and select the part of the system to evaluate. After the meeting, the three participants were asked to write down weaknesses and obstacles to integration of usability in their development process. The second step was the evaluation of the system. We used the Instant Data Analysis (IDA) method [ 14 ]. After the evaluation, the test results were emailed to the three participants and subsequently presented in combination with redesign proposals. After a month, the third step was conducted. A meeting was held, where the developer and user consultant were interviewed about their experiences with the usability evaluation and its result. They were also asked if any changes had been made to the system or their work process. A telephone interview was conducted with the section manager who was asked the same questions.

Setting. The meetings were held in a conference room in the company. The usability tests were conducted with real users and took place at the users’ workplaces. The user consultant and developer observed the first test session.

Data collection and analysis. We recorded of the interviews and collected the forms with opinions about weaknesses and obstacles. Each interview was conducted according to an interview guide [ 15 ]. Later, the recorded interviews were condensed using a method called “condensation of meaning” [ 15 ], and this outcome was then analyzed. The analysis was conducted by two persons separately. These two persons individually pointed out statements from the condensed data and grouped them into obstacles. Finally, the they negotiated a joint list of weaknesses and obstacles.

3.2 Company B

The company produced wireless technology. At the time of the study, it was divided into four units: technology, consumer products, network systems and healthcare. There were 180 employees, most of them located in the headquarter in Denmark. There were branches in USA, Hong Kong and Romania. Our collaboration with this company was carried out with the healthcare unit that had 10 employees, where 5 of them were developers. The system evaluated, was a device for home use by elderly people to send health data to a monitoring center. This system was recently developed and had a simple user interface. The company’s motivation for participating was an initial interest in usability evaluation, based on knowledge about another company’s successful experiences. Furthermore, the customer of the product in question required a usability evaluation. Participants. Throughout the collaboration, the main contact person was the user consultant for the product in question. The user consultant was responsible for verification and quality assessment of the product. In addition, a developer observed and provided technical assistance during the usability evaluation. Procedure. The study was conducted in three steps. The first step was an introductory meeting with the user consultant. The purpose was to gain an overview of the product and clarify mutual expectations.

The second step was the usability evaluation. The results from the evaluation were emailed the day after the evaluation. Interviews were made shortly after. The results from the evaluation were presented along with redesign proposals at a meeting. The third step involved two parts. Six months after the evaluation, the user consultant was interviewed to assess the effect. Two months later, the user consultant was interviewed again about the current obstacles in the company.

Setting. Most meetings were held at company B. The postevaluation meeting was held at the university, and the evaluation was conducted in our usability laboratory.

Data collection and analysis. The interviews with the user consultant were video recorded. Each interview was based on an interview guide [ 15 ]. The recordings were processed with “condensation of meaning” [ 15 ]. The analysis was done in exactly the same way as with company A.

4. Results

This section presents the results of our study in the two software organizations. The results are summarized in Table 1.

4.1 Resource Demands

The two software organizations initially had some obstacles in common. Both were convinced that usability evaluation was very time consuming and costly, as stated by the section manager in company A. The developer and user consultant also agreed that time and money were major obstacles. The main obstacle was the expectation about the time it would take to conduct the evaluations and make software changes.

Company B was looking for an inexpensive opportunity to evaluate the usability of their product. The resource demands of usability evaluation were underlined by the user consultant from company B in the following way; “The resource demand will always be an obstacle” and “... when we don’t know what is needed to conduct an evaluation, it will probably take too much of our time”.

In the final meetings, both organizations still stated resources in relation to time and money as being a main obstacle. It was most prominent in company B, where the user consultant made statements such as “We don’t have the resources to conduct a usability test.” and “... it would take too much time for us ... we don’t have the experience”.

Company A expressed this obstacle both in the interviews and the forms. In a discussion of gains from usability evaluations, the user consultant said “... it would be too expensive to reveal the problems this way”. When asked about the downsides of usability evaluation, the developer stated “I still think a lot of time is spent on it. You really don’t have much time here”. The user consultant stressed that resources is the most important factor “It all comes down to resources; the bottom line is always the focus point.” Resource demand as a main obstacle was also apparent in the forms. The section manager did only consider it relevant for special cases. On the other hand, he was surprised by the prompt delivery of results, and the user consultant concurred “The results were delivered very fast. I assumed it would take 3-4 weeks.” The resource demand of introducing usability evaluation was initially one of the major obstacles for both companies. The use of the low-cost method [ 14 ], gave the user consultant from company A an entirely different view “It changed my idea of how much time usability evaluations take.” The section manager’s attitude also changed. The change in company B was even more prominent as the user consultant expressed “If there is money for usability evaluation, we will certainly deploy it in the development process”.

4.2 Lack of Knowledge

Both companies stated that their knowledge of usability evaluation was initially at a very low level. Company B had some knowledge from another software organization that conducted usability evaluations, but only on the general level that usability evaluation can give useful information to developers. They did not have any knowledge about usability work practices. Company A had some knowledge from another department, where a usability evaluation had been conducted once, but no evaluations had ever been done in the care management unit. The lack of knowledge also extended to the users’ application of the system as the section manager stated “It would be great to get the knowledge into the organization; this could be used by the developers to make the product more usable for the end user.” The developer agreed; “We lack knowledge about the users’ professional world.” Lack of knowledge about usability evaluations was still expressed as an obstacle for both companies after the demonstration of usability evaluation. For company A, this applied to knowledge about evaluations and usability in general. The developer stated “As a developer, I find it hard to decide when to involve users in the development process.” In relation to the question when users should be involved, the section manager said “Usability evaluations can only be conducted in the final phases of a development process.” The lack of knowledge about usability evaluation was also expressed by the user consultant from company B “I have gained some knowledge, but not enough to conduct an evaluation on my own.” The lack of knowledge regarding the users’ application of the system as well as usability evaluation in general was the obstacle that was affected most in our study. An example of this was given by the user consultant in company A “...three of us discussed a design solution, but we were not able to agree, so we called a user and found the answer ... if you want something tested, you can just grab a user and ask for his or her opinion.” This approach had not been employed prior to our demonstration of usability evaluation. The demonstration made the employees experience that users can be involved in a constructive way in the development process. Other statements from the user consultant in company A underlined that the usability evaluation gave insight into the users’ work routines “Your tests show that it has a lot to do with work routines, and that has given us motivation for following up in the next release.” The importance of the evaluators was also stressed “Your tests show some subconscious things, and the users don’t catch them themselves. There has to be an observer to catch those things.” In the post-evaluation meeting and the final meeting with company B, several findings pertaining to the lack knowledge were emphasized. The user consultant and developer expressed a general satisfaction with the evaluation. Observing all sessions as they happened, gave them “… an insight into the way the system was actually used by a prospective end user”, as expressed by the user consultant. The evaluations revealed problems that had not previously been identified by the user consultant or developer. Both of them agreed upon the usefulness of this insight and thereby of the evaluations. In the final meeting with the user consultant, these attitudes and viewpoints were still completely intact. She said “When our new product is almost finished, it will be evaluated in the same manner ...” The insights gained from the usability evaluations were also mentioned in the final meeting with the user consultant “You can tell if the system is intuitive to use, if they can push the right buttons and read the display. These are things we cannot answer by discussing it in the development department. It is things we don’t think about.” The user consultant also stated that the results from the evaluation were of great use in her daily work. In certain design discussions, she was able to use the results as examples of actual user behaviour. The introduction also had an impact on the user consultant’s knowledge about usability evaluations. Initially, she had no knowledge about it, but in the final meeting she mentioned; “If we need a test of a future product, we know what usability evaluation is and what it can be used for, and we know when to test. So we can use this process for a lot of purposes.”

4.3 User Involvement

The two software organizations differed considerably in their thinking about end users. Company B wanted the end users to be able to use the product with a minimum of training and a very small and easy to read manual. In company A, the user consultant expressed “Our system is so complex that training is a necessity; in no way would the end user ever be able to use the system without the training we give them.” The users were contributing with proposals for changes to the system developed by company A, but this was actually considered more of a complication. For example, the developer mentioned “The users lack knowledge about the development process and the time it takes”, and the user consultant stated “The users do not have an overview of the system and its structure, and they might disagree about new functionality.” The section manager also mentioned difficulties related to the involvement of the end users “The users don’t think enough about what they’re shown. If they see something smart, they want it. They don’t think about the problems a feature can generate.” After the demonstration, obstacles relating to user involvement were only expressed in company A. The user consultant spoke of their users as being too numerous and geographically spread “... to reach out to 50% or even 10% of our users, that cannot be done. We have too many users.” Furthermore, usability evaluation of a product during development would be hard to conduct, because they would be forced to use inexperienced users, which would make the tests difficult “...it would most likely “drown” in explanations of the new functions.” The section manager expressed a similar concern about involving users in an evaluation “For the users to be involved in a test, they would have to be pulled away from their work. That costs money for the customer and will be a burden.” Company A was also reluctant to involve users, because their understanding of the problems found in the usability evaluation was that it was the users’ lack of knowledge about the system that caused the problems, as expressed by the developer “The usability problems occurred unexpectedly, and related more to user errors or lack of users’ understanding.”. The introduction of usability evaluation gave the participants from company A a deeper insight into the users’ way of using the system. Yet this insight also emphasized user involvement as an obstacle. For example, the user consultant expressed it this way “Are the problems occurring just because the users have adopted a wrong work routine ... the users lack an understanding of the use of the system.”

4.4 Structure of the System

Company A had an obstacle regarding the structure of the system. This was expressed by the section manager. The developer also mentioned the difficulties with the system structure “The system is used in different ways. With major changes there is a risk of removing existing functionality and introducing new errors in properly working parts of the system.” Although the structure in itself is not an obstacle to usability evaluation, correcting the problems found could be very difficult as expressed by the user consultant “Some parts of the system are hardcoded and cannot be changed, although the users see it as a small change.” The introduction of usability evaluation had no tangible effect on this obstacle, but reveals a need to prepare developers for potential changes in the system structure.

4.5 Management Interests

The participants from company A expressed an obstacle in relation to management, but only after the demonstration. When asked how apparent the importance of usability was for management, the developer said “I actually don’t think it’s apparent for management. Usability is taken for granted ...” The user consultant stated in relation to this obstacle “My attitude and position to the matter isn’t opposed to it, but reprioritization has to come from the management level.” In company B, the obstacle of management interests was also expressed by the user consultant “Management has decided to postpone usability evaluations until sales have gone up.” This obstacle was not identified in the initial statements, but only in the final statements. It emerged because of our direct question whether the company would consider deploying usability evaluation in the development process in the near future.

5. Discussion

The results of this study show that specific obstacles such as the resource demands and lack of knowledge about users and usability evaluation methods have been affected. The quick feedback from the evaluation to the software organization was a significant reason why company A would consider usability evaluation in the future. The fact that the participants from the two software organizations observed one or more test sessions increased their insight into the methodology and the users’ ways of using the system considerable. This was clear from the positive comments that participants from both companies made about observing the tests.

The fact that the software organizations were affected by observing the benefits of usability evaluation is a valuable contribution of this study, and should be a point of focus in further research. This is also where this study differs from related work within this area. As mentioned in section 2, many of the previous studies have focused on providing guidelines or principles for deploying usability practices. The purpose of these has been to ease the deployment of usability evaluation in the development process [ 3, 6 ]. In contrast, the purpose of our study was to let company representatives observe the benefits of usability evaluation.

An important factor when deploying usability evaluation is the motivation of the software organization. In our study we observed a different motivation between the two software organizations. Company A’s motivation for participating in the experiment was curiosity about the nature of usability evaluation and its practical use. Company B had a need to gain knowledge about usability evaluation because of customer demands. This difference in motivation might have had an impact on the obstacles identified. For example, the number of obstacles identified in company B was only two, while it in company A was four before the introduction and five after. Moreover, an obstacle identified in company A related to the users and the difficulties of meeting with the users. Company B also had difficulties with creating contact with users, but it was not expressed as an obstacle. Overall, company A had a tendency to see obstacles rather than benefits of usability evaluation, which indicate a lack of motivation that makes it even more difficult to deploy usability evaluation.

To increase the motivation, a software organization needs to experience that usability evaluation can fulfill relevant needs. Company B was more willing to deploy usability evaluations than company A after the demonstration. Another factor relating to the greater effect might have been that the employees from company B observed all the sessions of the usability evaluation, whereas the employees from company A, observed only one session. The experiences with company A also showed that decisions to integrate and prioritize evaluations had to come from top level management. Therefore it could be beneficial to include participants from that level in a demonstration.

6. Conclusion

The purpose of this study was to observe how the introduction of usability evaluation affects significant obstacles to usability evaluation in software organizations. To inquire into this, a usability evaluation was demonstrated to two software organizations. This included that we conducted a usability evaluation and presented the evaluation results to the two software organizations.

The results show that the introduction of usability evaluation provided the software organizations with insight into the users’ use of the system. Furthermore, they experienced that usability evaluations are not nearly as resource demanding as expected. This illustrates that the stakeholders’ attitudes to these obstacles were affected. However, none of the obstacles identified in the two software organizations were completely resolved. Two of the initial obstacles, user involvement and structure of the system, were not affected by the demonstration of usability evaluationl. This study shows that it is possible to motivate software organizations toward usability evaluation. This was achieved through the approach in which the companies’ products were evaluated. This underlines the relevance of research in this topic based on other approaches than providing guidelines and principles, which has been covered to a great extent.

There are some important limitations to our study. The two software organizations were quite similar. Also, we interviewed quite few persons in these organizations. In both organizations, we focused in particular on the benefits and time taken; we did not deal explicitly with the costs for the two organizations. The main source of data was interviews combined with forms in one of the organizations. Finally, the specific method used in the evaluations might have affected the results. It would be interesting to extend the study to more organizations and stakeholders and use different methods both for data collection and for the evaluation.

Acknowledgments. The research behind this paper was partly financed by the Danish Research Councils (grant numbers 210604-0022 and 2106-08-0011). We are very grateful to the two software organizations and the stakeholders that participated in the study. We would also like to thank the anonymous reviewers for their comments and advice.

References

[1] Sarah Bloomer , Rachel Croft . Pitching usability to your organization . In interactions, ACM Press, 1998 , ( 4 ,6), pages 18 - 26 . ISSN: 1072 - 5520 .

[2] Inger Boivie , Jan Gulliksen, Bengt G¨ oransson. The lonesome cowboy: A study of the usability designer role in systems development . In Interacting with Computers , Elsevier Science B.V. , ( 18 ,4), 2006 , pages 601 - 634 .

[3] Jim Carter . Incorporating standards and guidelines in an approach that balances usability concerns for developers and end users . In Interacting with Computers , Elsevier Science B.V. , ( 12 ,2), 1999 , pages 179 - 206 .

[4] Susan M. Dray , Clare Marie Karat. Human factors cost justification for an internal development project . In Costjustifying usability, Academic Press, Inc., 1994 , pages 111 - 122 . ISBN: 0 - 12 -095810-4.

[5] Carola B. Fellenz . Introducing usability into smaller organizations . In interactions, ACM Press, 1997 , pages 29 - 33 . ISSN: 1072 - 5520 .

[6] John D. Gould , Clayton Lewis . Designing for usability: key principles and what designers think . In Communications of the ACM , ACM Press, 1985 , ( 28 ,3), pages 300 - 311 .

[7] Toni Granollers . User Centred Design Process Model. Integration of Usability Engineering and Software Engineering . In Proceedings of interact 2003 . Found at: http://www.griho.udl.es/publicacions/2003/Doctoral%20 Consortium% 20 (Interact% 2003 ).pdf, last seen April 11th 2007 .

[8] Jonathan Grudin . Obstacles to user involvement in software product development, with implications for CSCW . In International Journal of Man-Machine Studies , Academic Press Ltd., 1991 , ( 34 ,3), pages 435 - 452 . ISSN: 0020 - 7373 .

[9] Melody Y. Ivory , Marti A Hearst. The state of the art in automating usability evaluation of user interfaces . In ACM Comput. Surv ., ACM Press, 2001 , ( 33 ,4), pages 470 - 516 . ISSN: 0360 - 0300 .

[10] Claire Marie Karat . A business case approach to usability cost justification . In Cost justifying usability , Academic Press, Inc., 1994 . ISBN: 0 - 12 -095810-4.

[11] Clare Marie Karat . A Comparison of User Interface Evaluation Methods . In Usability Inspection Methods , 1994 , pages 203 - 233 . ISBN: 0 - 471 -01877-5.

[12] Clare Marie Karat . Cost-justifying usability engineering in the software life cycle . In Handbook of Human-Computer Interaction, Elsevier Science Inc., 2nd edition , 1997 , pages 767 - 778 . ISBN: 0444818626 .

[13] Brenda Kerton . Introducing usability at London Life insurance company: a process perspective . In CHI '97: CHI ' 97 extended abstracts on Human factors in computing systems , ACM Press, 1997 , pages 77 - 78 . ISBN: 0 - 89791 - 926-2.

[14] Jesper Kjeldskov , Mikael B. Skov , Jan Stage . Instant data analysis: conducting usability evaluations in a day . In NordiCHI '04: Proceedings of the third Nordic conference on Humancomputer interaction , ACM Press, 2004 . ISBN: 1 - 58113 -857-1.

[15] Steinar Kvale . Interview - En introduktion til det kvalitative forskningsinterview . Hans Reitzel , 2 .edition, 1998 . ISBN: 87 - 412 -2816-2.

[16] Deborah J. Mayhew . Business: Strategic development of the usability engineering function . In interactions, ACM Press, 1999 , ( 6 ,5), pages 27 - 34 . ISSN: 1072 - 5520 .

[17] Karsten Nebe , Lennart Gr¨otzbach. Aligning user centered design activities with established software development practices . In NordiCHI '06: Proceedings of the 4th Nordic conference on Human-computer interaction , ACM Press, 2006 , pages 485 - 486 . ISBN: 1 - 59593 -325-5.

[18] Jakob Nielsen . Why GUI panic is good panic . In interactions, ACM Press, 1994 , ( 2 ,1), pages 55 - 58 . ISSN: 1072 - 5520 .

[19] Jerilyn Prescott , Matt Crichton . Usability testing: a quick, cheap, and effective method . In SIGUCCS '99: Proceedings of the 27th annual ACM SIGUCCS conference on User services , ACM Press, 1999 , pages 176 - 179 . ISBN: 1 - 58113 - 144-5.

[20] S. Reed Who defines usability? You do! . In PC//Computing, (Dec), 1992 , pages 220 - 232 . ISBN: 0 - 12 -095810-4.

[21] Stephanie Rosenbaum , Sarah Bloomer, Dave Rinehart, Janice Rohn, Ken Dye, Judee Humburg, Jakob Nielsen, DennisWixon. What makes strategic usability fail?: lessons learned from the field . In CHI '99: CHI ' 99 extended abstracts on human factors in computing systems , ACM Press, 1999 , pages 93 - 94 ,. ISBN: 1 - 58113 -158-5.

[22] Ahmed Seffah , Alina Andreevskaia . Empowering software engineers in human-centered design . In ICSE '03: Proceedings of the 25th International Conference on Software Engineering , IEEE Computer Society, 2003 , pages 653 - 658 .

[23] Desir Sy . Bridging the communication gap in the workplace with usability engineering . In SIGDOC '94: Proceedings of the 12th annual international conference on Systems documentation , ACM Press, 1994 , pages 208 - 212 . ISBN: 0 - 89791 -681-6.

[24] Leslie Tudor . Obstacles to user involvement in software product development, with implications for CSCW. In Human factors: does your management hear you? , ACM Press, 1998 , ( 5 ,1), pages 16 - 24 . ISSN: 1072 - 5520 .