=Paper=
{{Paper
|id=Vol-490/paper-8
|storemode=property
|title=Bringing Usability Evaluation into Practice: Field Studies in Two Software Organization
|pdfUrl=https://ceur-ws.org/Vol-490/paper_08.pdf
|volume=Vol-490
|dblpUrl=https://dblp.org/rec/conf/iused/BakNRS09
}}
==Bringing Usability Evaluation into Practice: Field Studies in Two Software Organization==
https://ceur-ws.org/Vol-490/paper_08.pdf
Bringing Usability Evaluation into Practice:
Field Studies in Two Software Organizations
Jakob Otkjær Bak Kim Nguyen Peter Risgaard Jan Stage
TARGIT A/S Logica EUCNORD Aalborg University
Aalborgvej 94 Fredrik Bajers Vej 1 Hånbækvej 50 Department of
DK-9800 Hjørring DK-9220 Aalborg East DK-9900 Frederikshavn Computer Science
Denmark Denmark Denmark DK-9220 Aalborg East
jb@targit.com kimmeren@gmail.com pri@eucnord.dk Denmark
jans@cs.aau.dk
ABSTRACT organizations are still not conducting any form of usability
This paper explores how obstacles to usability evaluations in a evaluation in their development process [21].
software organization can be affected. We present two field There have been considerable efforts to affect the obstacles that
studies, each conducted in a software organization that had no prevent these software organizations from deploying usability
previous experience with usability evaluation. In each study, we evaluation techniques. A major approach has provided techniques
first interviewed key stakeholders to identify their opinion about that are supposed to ease the deployment. This approach has only
significant obstacles to conducting usability evaluations. Then we had limited success and mostly in software organizations that are
demonstrated the benefits of a usability evaluation by evaluating already conducting usability evaluations. The reason may be that
the usability of one of their software products, while being most of the proposed techniques are highly technical and designed
observed by the developers, and presenting the evaluation results by experts to be used by experts or at least by well-trained
to the developers. Finally, the key stakeholders were interviewed professionals [3].
again to establish the effect of the demonstration. The
demonstration of benefits had a positive effect on some of the key A basically different approach has been to affect key
obstacles, while others were unaffected. One organization stakeholders’ attitudes to usability evaluation. This has mostly
expressed future plans for conducting usability evaluations while been done on a general level by documenting how other
the other was still reluctant. organizations have benefitted from deploying usability evaluation
techniques in their development process. A study found that
collection of user data, setting usability goals and conducting
Categories and Subject Descriptors usability walkthroughs had a positive effect [13]. Another study
H.5.2. [Information Interfaces and Presentation]: User documented that deployment of user-centered design in the
Interfaces – Evaluation/methodology. K.6.1 [Management of development life cycle of a software company, specifically by
Computing and Information Systems]: Project and People integration of use cases in the development process, supported
Management – Staffing, Systems development, Training. decision making [17].Karat provides evidence about the cost and
benefit of usability evaluation [11]. The difficulty is, however,
General Terms that often the cost is paid by the software organization, while the
Experimentation, Human Factors. benefit is gained by the customer. Yet there are exceptions. A
study established that evaluation of software for usability can lead
to increased sale of products [12]. Another study demonstrated
Keywords that the need for user support decreased with better usability [20].
Usability evaluation, software organizations, development Experience with deployment of usability work is usually focused
practice, empirical study. on larger organizations. However, a study in a smaller
organization also presents activities that were successful [5].
1. INTRODUCTION Another study focused specifically on usability evaluation and
Usability is a fundamental attribute of interactive systems [7], and concludes that quick, cheap and effective evaluations can be
it is critical to their success or failure on the market [10]. conducted [19].
Evaluation of usability has been documented to be economically
Only few studies have focused on affecting the attitudes to
feasible because of increased sales [11], increased user
usability evaluation on a specific level; that is in a particular
productivity [12], decreased training costs [4] and decreased
software organization. This paper reports from two field studies,
needs for user support [20]. Despite these facts, many software
where we tried to overcome obstacles to usability evaluation by
affecting the attitudes of key stakeholders. This was done by
Permission to make digital or hard copies of all or part of this work for demonstrating how that particular organization could benefit from
personal or classroom use is granted without fee provided that copies are deploying usability evaluation in their development process. In
not made or distributed for profit or commercial advantage and that
section 2 we present related work on affecting obstacles to
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
usability evaluation. Section 3 presents the method used in of the
requires prior specific permission and/or a fee. two field studies. In section 4, we provide the results from the
Conference’04, Month 1–2, 2004, City, State, Country. field studies. In section 5, we discuss our results. Finally, section
Copyright 2004 ACM 1-58113-000-0/00/0004…$5.00. 6 provides the conclusion.
2. RELATED WORK Participants. Three participants from company A were involved
The majority of studies that try to affect obstacles to usability in the collaboration; a section manager, a developer and a user
evaluation focuses on usability guidelines and methods for consultant. The section manager was in charge of the
incorporating usability in the development process. Gould and development team, the developer was responsible for the user
Lewis were among the first to provide guidelines for the interface design and the user consultant was responsible for
deployment of usability in the design process [6]. A study contact to users and for their education.
questioned the relevance of guidelines to usability and discussed Procedure. The study was conducted in 3 steps. The first step
appropriate sources of guidance [3]. Overall guidelines directed at was an initial meeting with the section manager of the care
the developers are widely used. A study identified the gap management solutions department, the user consultant and the
between designers and users as the major obstacle to deploying developer responsible of the user interface design. The purpose
usability and suggested usability engineering methodologies to was to determine obstacles to usability evaluation in the company
help overcome this obstacle [23]. Grudin presented suggestions to and select the part of the system to evaluate. After the meeting,
overcome this gap based on long term experiences [8]. the three participants were asked to write down weaknesses and
Solutions to overcome organizational obstacles to usability obstacles to integration of usability in their development process.
evaluation are presented in some papers. They tend to advise what The second step was the evaluation of the system. We used the
usability practitioners can do to sell usability to the organization. Instant Data Analysis (IDA) method [14]. After the evaluation,
Mayhew suggests three phases and for each phase how, why and the test results were emailed to the three participants and
what to do to sell usability [16]. A study concluded that subsequently presented in combination with redesign proposals.
communicating the message of usability is not enough; the facts
must be solid and documented [24]. Another study complements After a month, the third step was conducted. A meeting was held,
this by concluding that experiences with usability have to be where the developer and user consultant were interviewed about
presented in a way that appeals to upper management’s mindset their experiences with the usability evaluation and its result. They
with emphasis on the monetary benefits [1]. were also asked if any changes had been made to the system or
their work process. A telephone interview was conducted with the
Resource-related obstacles have also been studied. Based on section manager who was asked the same questions.
experiences from several organizations, Nielsen states that there
are considerable monetary benefits from conducting usability Setting. The meetings were held in a conference room in the
evaluations [18]. A study emphasized that automation is a way to company. The usability tests were conducted with real users and
complement existing usability evaluation methods [9]. took place at the users’ workplaces. The user consultant and
developer observed the first test session.
Only a few researchers have tried to measure the effect of
deploying new usability methods in software organizations. One Data collection and analysis. We recorded of the interviews and
study concluded that nurturing the developers’ skills in user- collected the forms with opinions about weaknesses and
centered design was a major factor in developing more usable obstacles. Each interview was conducted according to an
systems [22]. A different study provided a usability engineer to a interview guide [15]. Later, the recorded interviews were
software organization. This helped developers shift focus toward condensed using a method called “condensation of meaning” [15],
design and assume a role as the users’ advocate [2]. and this outcome was then analyzed. The analysis was conducted
by two persons separately. These two persons individually
pointed out statements from the condensed data and grouped them
3. METHOD into obstacles. Finally, the they negotiated a joint list of
We have conducted field studies in two software organizations, weaknesses and obstacles.
where we tried to demonstrate the benefits of usability evaluation
in an ongoing development process. 3.2 Company B
The company produced wireless technology. At the time of the
3.1 Company A study, it was divided into four units: technology, consumer
The company had, at the time of the study, 150 employees with products, network systems and healthcare. There were 180
headquarter in Denmark and branches in Canada, USA and employees, most of them located in the headquarter in Denmark.
Romania. Its business was separated in four units: supply chain There were branches in USA, Hong Kong and Romania. Our
solutions, postal solutions, airport solutions and care management collaboration with this company was carried out with the
solutions. Our collaboration was with the care management healthcare unit that had 10 employees, where 5 of them were
solution unit that had 12 employees, of which 7 were software developers. The system evaluated, was a device for home use by
developers. The system we evaluated was a planning module for a elderly people to send health data to a monitoring center. This
healthcare management system used by nurses and home system was recently developed and had a simple user interface.
assistants to plan both care for citizens and staff working hours.
The system had been developed some years before and updated The company’s motivation for participating was an initial interest
regularly. Initially, it had a non-graphical user interface. Later, it in usability evaluation, based on knowledge about another
was supplemented with a graphical user interface. company’s successful experiences. Furthermore, the customer of
the product in question required a usability evaluation.
The company’s motivation for participating was curiosity about
usability evaluation and a desire to see if it could be integrated in Participants. Throughout the collaboration, the main contact
the development process without being too costly. It was not person was the user consultant for the product in question. The
triggered by customer demands. user consultant was responsible for verification and quality
assessment of the product. In addition, a developer observed and Setting. Most meetings were held at company B. The post-
provided technical assistance during the usability evaluation. evaluation meeting was held at the university, and the evaluation
was conducted in our usability laboratory.
Procedure. The study was conducted in three steps. The first step
was an introductory meeting with the user consultant. The Data collection and analysis. The interviews with the user
purpose was to gain an overview of the product and clarify mutual consultant were video recorded. Each interview was based on an
expectations. interview guide [15]. The recordings were processed with
“condensation of meaning” [15]. The analysis was done in exactly
The second step was the usability evaluation. The results from the
the same way as with company A.
evaluation were emailed the day after the evaluation. Interviews
were made shortly after. The results from the evaluation were
presented along with redesign proposals at a meeting. 4. Results
This section presents the results of our study in the two software
The third step involved two parts. Six months after the evaluation, organizations. The results are summarized in Table 1.
the user consultant was interviewed to assess the effect. Two
months later, the user consultant was interviewed again about the
current obstacles in the company. 4.1 Resource Demands
The two software organizations initially had some obstacles in
Table 1. Essential statements from company A and B before common. Both were convinced that usability evaluation was very
and after the trial evaluation. time consuming and costly, as stated by the section manager in
Obstacle Initial statements Final statements company A. The developer and user consultant also agreed that
Resource Company A: “It would be Company A: “I can see it time and money were major obstacles. The main obstacle was the
demands a high increase in the price being conducted on special expectation about the time it would take to conduct the
and maybe delay the products or occasions, evaluations and make software changes.
development two weeks or places where we deem it
more. The customer should extra important or are Company B was looking for an inexpensive opportunity to
then be ready to pay suspicious about a poorly evaluate the usability of their product. The resource demands of
100.000 kr. more than designed user interface. But usability evaluation were underlined by the user consultant from
now.” nothing regularly, there is company B in the following way; “The resource demand will
typically no time for it in our
always be an obstacle” and “... when we don’t know what is
Company B: “...when we development process.”
don’t know what is needed
needed to conduct an evaluation, it will probably take too much of
to conduct an evaluation, Company B: “There are no our time”.
then it will probably take resources for usability tests,
In the final meetings, both organizations still stated resources in
too much of our time.” we really want to, but
there’s no money for it at the
relation to time and money as being a main obstacle. It was most
moment.” prominent in company B, where the user consultant made
Lack of Company A: “Knowledge Company A: “... the statements such as “We don’t have the resources to conduct a
knowledge about the right solution is evaluations gave an insight usability test.” and “... it would take too much time for us ... we
an obstacle to integrating into how the system was don’t have the experience”.
usability evaluation in the actually used by a
development process.” prospective end user.” Company A expressed this obstacle both in the interviews and the
forms. In a discussion of gains from usability evaluations, the user
Company B: “...we have Company B: “I have gained consultant said “... it would be too expensive to reveal the
very little knowledge about some knowledge, but not problems this way”. When asked about the downsides of usability
usability evaluations.” enough to conduct an evaluation, the developer stated “I still think a lot of time is spent
evaluation on my own.” on it. You really don’t have much time here”. The user consultant
User Company A: “The users Company A: “The usability
stressed that resources is the most important factor “It all comes
involve- don’t think enough about problems occurred
ment what they are shown. If unexpectedly, and related
down to resources; the bottom line is always the focus point.”
they see something smart, more to user errors or lack Resource demand as a main obstacle was also apparent in the
they want it. They don’t of users’ understanding.”
forms. The section manager did only consider it relevant for
think about the problems a
new solution can
special cases. On the other hand, he was surprised by the prompt
generate.” delivery of results, and the user consultant concurred “The results
Structure Company A: “Often, the Company A: “...the were delivered very fast. I assumed it would take 3-4 weeks.”
of the database layer and other development system and
The resource demand of introducing usability evaluation was
system function-related layers are environment is not up to
limiting the user interface. date.” initially one of the major obstacles for both companies. The use of
You lock a lot in the the low-cost method [14], gave the user consultant from company
beginning of the project.” A an entirely different view “It changed my idea of how much
Manage- Company A: “I actually time usability evaluations take.” The section manager’s attitude
ment don’t think the need for also changed. The change in company B was even more
interests usability evaluations is prominent as the user consultant expressed “If there is money for
apparent to upper usability evaluation, we will certainly deploy it in the
management. Usability is development process”.
taken for granted...”
4.2 Lack of Knowledge The insights gained from the usability evaluations were also
Both companies stated that their knowledge of usability mentioned in the final meeting with the user consultant “You can
evaluation was initially at a very low level. Company B had some tell if the system is intuitive to use, if they can push the right
knowledge from another software organization that conducted buttons and read the display. These are things we cannot answer
usability evaluations, but only on the general level that usability by discussing it in the development department. It is things we
evaluation can give useful information to developers. They did don’t think about.” The user consultant also stated that the results
not have any knowledge about usability work practices. Company from the evaluation were of great use in her daily work. In certain
A had some knowledge from another department, where a design discussions, she was able to use the results as examples of
usability evaluation had been conducted once, but no evaluations actual user behaviour. The introduction also had an impact on the
had ever been done in the care management unit. The lack of user consultant’s knowledge about usability evaluations. Initially,
knowledge also extended to the users’ application of the system she had no knowledge about it, but in the final meeting she
as the section manager stated “It would be great to get the mentioned; “If we need a test of a future product, we know what
knowledge into the organization; this could be used by the usability evaluation is and what it can be used for, and we know
developers to make the product more usable for the end user.” when to test. So we can use this process for a lot of purposes.”
The developer agreed; “We lack knowledge about the users’
professional world.”
4.3 User Involvement
The two software organizations differed considerably in their
Lack of knowledge about usability evaluations was still expressed thinking about end users. Company B wanted the end users to be
as an obstacle for both companies after the demonstration of able to use the product with a minimum of training and a very
usability evaluation. For company A, this applied to knowledge small and easy to read manual. In company A, the user consultant
about evaluations and usability in general. The developer stated expressed “Our system is so complex that training is a necessity;
“As a developer, I find it hard to decide when to involve users in in no way would the end user ever be able to use the system
the development process.” In relation to the question when users without the training we give them.”
should be involved, the section manager said “Usability
The users were contributing with proposals for changes to the
evaluations can only be conducted in the final phases of a
system developed by company A, but this was actually considered
development process.” The lack of knowledge about usability
more of a complication. For example, the developer mentioned
evaluation was also expressed by the user consultant from
“The users lack knowledge about the development process and
company B “I have gained some knowledge, but not enough to
the time it takes”, and the user consultant stated “The users do not
conduct an evaluation on my own.”
have an overview of the system and its structure, and they might
The lack of knowledge regarding the users’ application of the disagree about new functionality.” The section manager also
system as well as usability evaluation in general was the obstacle mentioned difficulties related to the involvement of the end users
that was affected most in our study. An example of this was given “The users don’t think enough about what they’re shown. If they
by the user consultant in company A “...three of us discussed a see something smart, they want it. They don’t think about the
design solution, but we were not able to agree, so we called a user problems a feature can generate.”
and found the answer ... if you want something tested, you can
After the demonstration, obstacles relating to user involvement
just grab a user and ask for his or her opinion.” This approach had
were only expressed in company A. The user consultant spoke of
not been employed prior to our demonstration of usability
their users as being too numerous and geographically spread “...
evaluation. The demonstration made the employees experience
to reach out to 50% or even 10% of our users, that cannot be
that users can be involved in a constructive way in the
done. We have too many users.” Furthermore, usability evaluation
development process. Other statements from the user consultant
of a product during development would be hard to conduct,
in company A underlined that the usability evaluation gave
because they would be forced to use inexperienced users, which
insight into the users’ work routines “Your tests show that it has a
would make the tests difficult “...it would most likely “drown” in
lot to do with work routines, and that has given us motivation for
explanations of the new functions.” The section manager
following up in the next release.” The importance of the
expressed a similar concern about involving users in an evaluation
evaluators was also stressed “Your tests show some subconscious
“For the users to be involved in a test, they would have to be
things, and the users don’t catch them themselves. There has to be
pulled away from their work. That costs money for the customer
an observer to catch those things.”
and will be a burden.” Company A was also reluctant to involve
In the post-evaluation meeting and the final meeting with users, because their understanding of the problems found in the
company B, several findings pertaining to the lack knowledge usability evaluation was that it was the users’ lack of knowledge
were emphasized. The user consultant and developer expressed a about the system that caused the problems, as expressed by the
general satisfaction with the evaluation. Observing all sessions as developer “The usability problems occurred unexpectedly, and
they happened, gave them “… an insight into the way the system related more to user errors or lack of users’ understanding.”.
was actually used by a prospective end user”, as expressed by the
The introduction of usability evaluation gave the participants
user consultant. The evaluations revealed problems that had not
from company A a deeper insight into the users’ way of using the
previously been identified by the user consultant or developer.
system. Yet this insight also emphasized user involvement as an
Both of them agreed upon the usefulness of this insight and
obstacle. For example, the user consultant expressed it this way
thereby of the evaluations. In the final meeting with the user
“Are the problems occurring just because the users have adopted a
consultant, these attitudes and viewpoints were still completely
wrong work routine ... the users lack an understanding of the use
intact. She said “When our new product is almost finished, it will
of the system.”
be evaluated in the same manner ...”
4.4 Structure of the System Company A’s motivation for participating in the experiment was
Company A had an obstacle regarding the structure of the system. curiosity about the nature of usability evaluation and its practical
This was expressed by the section manager. The developer also use. Company B had a need to gain knowledge about usability
mentioned the difficulties with the system structure “The system evaluation because of customer demands. This difference in
is used in different ways. With major changes there is a risk of motivation might have had an impact on the obstacles identified.
removing existing functionality and introducing new errors in For example, the number of obstacles identified in company B
properly working parts of the system.” Although the structure in was only two, while it in company A was four before the
itself is not an obstacle to usability evaluation, correcting the introduction and five after. Moreover, an obstacle identified in
problems found could be very difficult as expressed by the user company A related to the users and the difficulties of meeting
consultant “Some parts of the system are hardcoded and cannot be with the users. Company B also had difficulties with creating
changed, although the users see it as a small change.” contact with users, but it was not expressed as an obstacle.
Overall, company A had a tendency to see obstacles rather than
The introduction of usability evaluation had no tangible effect on benefits of usability evaluation, which indicate a lack of
this obstacle, but reveals a need to prepare developers for motivation that makes it even more difficult to deploy usability
potential changes in the system structure. evaluation.
4.5 Management Interests To increase the motivation, a software organization needs to
The participants from company A expressed an obstacle in experience that usability evaluation can fulfill relevant needs.
relation to management, but only after the demonstration. When Company B was more willing to deploy usability evaluations than
asked how apparent the importance of usability was for company A after the demonstration. Another factor relating to the
management, the developer said “I actually don’t think it’s greater effect might have been that the employees from company
apparent for management. Usability is taken for granted ...” The B observed all the sessions of the usability evaluation, whereas
user consultant stated in relation to this obstacle “My attitude and the employees from company A, observed only one session. The
position to the matter isn’t opposed to it, but reprioritization has experiences with company A also showed that decisions to
to come from the management level.” In company B, the obstacle integrate and prioritize evaluations had to come from top level
of management interests was also expressed by the user management. Therefore it could be beneficial to include
consultant “Management has decided to postpone usability participants from that level in a demonstration.
evaluations until sales have gone up.”
This obstacle was not identified in the initial statements, but only 6. Conclusion
in the final statements. It emerged because of our direct question The purpose of this study was to observe how the introduction of
whether the company would consider deploying usability usability evaluation affects significant obstacles to usability
evaluation in the development process in the near future. evaluation in software organizations. To inquire into this, a
usability evaluation was demonstrated to two software
organizations. This included that we conducted a usability
5. Discussion evaluation and presented the evaluation results to the two
The results of this study show that specific obstacles such as the software organizations.
resource demands and lack of knowledge about users and
usability evaluation methods have been affected. The quick The results show that the introduction of usability evaluation
feedback from the evaluation to the software organization was a provided the software organizations with insight into the users’
significant reason why company A would consider usability use of the system. Furthermore, they experienced that usability
evaluation in the future. The fact that the participants from the evaluations are not nearly as resource demanding as expected.
two software organizations observed one or more test sessions This illustrates that the stakeholders’ attitudes to these obstacles
increased their insight into the methodology and the users’ ways were affected. However, none of the obstacles identified in the
of using the system considerable. This was clear from the positive two software organizations were completely resolved. Two of the
comments that participants from both companies made about initial obstacles, user involvement and structure of the system,
observing the tests. were not affected by the demonstration of usability evaluationl.
The fact that the software organizations were affected by This study shows that it is possible to motivate software
observing the benefits of usability evaluation is a valuable organizations toward usability evaluation. This was achieved
contribution of this study, and should be a point of focus in through the approach in which the companies’ products were
further research. This is also where this study differs from related evaluated. This underlines the relevance of research in this topic
work within this area. As mentioned in section 2, many of the based on other approaches than providing guidelines and
previous studies have focused on providing guidelines or principles, which has been covered to a great extent.
principles for deploying usability practices. The purpose of these There are some important limitations to our study. The two
has been to ease the deployment of usability evaluation in the software organizations were quite similar. Also, we interviewed
development process [3, 6]. In contrast, the purpose of our study quite few persons in these organizations. In both organizations,
was to let company representatives observe the benefits of we focused in particular on the benefits and time taken; we did
usability evaluation. not deal explicitly with the costs for the two organizations. The
An important factor when deploying usability evaluation is the main source of data was interviews combined with forms in one
motivation of the software organization. In our study we observed of the organizations. Finally, the specific method used in the
a different motivation between the two software organizations. evaluations might have affected the results. It would be
interesting to extend the study to more organizations and Interaction, Elsevier Science Inc., 2nd edition, 1997, pages
stakeholders and use different methods both for data collection 767- 778. ISBN: 0444818626.
and for the evaluation. [13] Brenda Kerton. Introducing usability at London Life
Acknowledgments. The research behind this paper was partly insurance company: a process perspective. In CHI ’97: CHI
financed by the Danish Research Councils (grant numbers 2106- ’97 extended abstracts on Human factors in computing
04-0022 and 2106-08-0011). We are very grateful to the two systems, ACM Press, 1997, pages 77-78. ISBN: 0-89791-
software organizations and the stakeholders that participated in 926-2.
the study. We would also like to thank the anonymous reviewers [14] Jesper Kjeldskov, Mikael B. Skov, Jan Stage. Instant data
for their comments and advice. analysis: conducting usability evaluations in a day. In
NordiCHI ’04: Proceedings of the third Nordic conference
References on Humancomputer interaction, ACM Press, 2004. ISBN: 1-
[1] Sarah Bloomer, Rachel Croft. Pitching usability to your 58113-857-1.
organization. In interactions, ACM Press, 1998, (4,6), pages [15] Steinar Kvale. Interview - En introduktion til det kvalitative
18-26. ISSN: 1072-5520. forskningsinterview. Hans Reitzel, 2.edition, 1998. ISBN:
[2] Inger Boivie, Jan Gulliksen, Bengt G¨oransson. The lonesome 87-412-2816-2.
cowboy: A study of the usability designer role in systems [16] Deborah J. Mayhew. Business: Strategic development of the
development. In Interacting with Computers, Elsevier usability engineering function. In interactions, ACM Press,
Science B.V., (18,4), 2006, pages 601-634. 1999, (6,5), pages 27-34. ISSN: 1072-5520.
[3] Jim Carter. Incorporating standards and guidelines in an [17] Karsten Nebe, Lennart Gr¨otzbach. Aligning user centered
approach that balances usability concerns for developers and design activities with established software development
end users. In Interacting with Computers, Elsevier Science practices. In NordiCHI ’06: Proceedings of the 4th Nordic
B.V., (12,2), 1999, pages 179-206. conference on Human-computer interaction, ACM Press,
[4] Susan M. Dray, Clare Marie Karat. Human factors cost 2006, pages 485-486. ISBN: 1-59593-325-5.
justification for an internal development project. In [18] Jakob Nielsen. Why GUI panic is good panic. In
Costjustifying usability, Academic Press, Inc., 1994, pages interactions, ACM Press, 1994, (2,1), pages 55-58. ISSN:
111-122. ISBN: 0-12-095810-4. 1072-5520.
[5] Carola B. Fellenz. Introducing usability into smaller [19] Jerilyn Prescott, Matt Crichton. Usability testing: a quick,
organizations. In interactions, ACM Press, 1997, pages 29- cheap, and effective method. In SIGUCCS ’99: Proceedings
33. ISSN: 1072-5520. of the 27th annual ACM SIGUCCS conference on User
[6] John D. Gould, Clayton Lewis. Designing for usability: key services, ACM Press, 1999, pages 176-179. ISBN: 1-58113-
principles and what designers think. In Communications of 144-5.
the ACM, ACM Press, 1985, (28,3), pages 300-311. [20] S. Reed Who defines usability? You do!. In PC//Computing,
[7] Toni Granollers. User Centred Design Process Model. (Dec), 1992, pages 220-232. ISBN: 0-12-095810-4.
Integration of Usability Engineering and Software [21] Stephanie Rosenbaum, Sarah Bloomer, Dave Rinehart,
Engineering. In Proceedings of interact 2003. Found at: Janice Rohn, Ken Dye, Judee Humburg, Jakob Nielsen,
http://www.griho.udl.es/publicacions/2003/Doctoral%20 DennisWixon. What makes strategic usability fail?: lessons
Consortium%20(Interact%2003).pdf, last seen April 11th learned from the field. In CHI ’99: CHI ’99 extended
2007. abstracts on human factors in computing systems, ACM
[8] Jonathan Grudin. Obstacles to user involvement in software Press, 1999, pages 93-94,. ISBN: 1-58113-158-5.
product development, with implications for CSCW. In [22] Ahmed Seffah, Alina Andreevskaia. Empowering software
International Journal of Man-Machine Studies, Academic engineers in human-centered design. In ICSE ’03:
Press Ltd., 1991, (34,3), pages 435-452. ISSN: 0020-7373. Proceedings of the 25th International Conference on
[9] Melody Y. Ivory, Marti A Hearst. The state of the art in Software Engineering, IEEE Computer Society, 2003, pages
automating usability evaluation of user interfaces. In ACM 653-658.
Comput. Surv., ACM Press, 2001, (33,4), pages 470-516. [23] Desir Sy. Bridging the communication gap in the workplace
ISSN: 0360-0300. with usability engineering. In SIGDOC ’94: Proceedings of
[10] Claire Marie Karat. A business case approach to usability the 12th annual international conference on Systems
cost justification. In Cost justifying usability, Academic documentation, ACM Press, 1994, pages 208-212. ISBN: 0-
Press, Inc., 1994. ISBN: 0-12-095810-4. 89791-681-6.
[11] Clare Marie Karat. A Comparison of User Interface [24] Leslie Tudor. Obstacles to user involvement in software
Evaluation Methods. In Usability Inspection Methods, 1994, product development, with implications for CSCW. In
pages 203-233. ISBN: 0-471-01877-5. Human factors: does your management hear you?, ACM
Press, 1998, (5,1), pages 16-24. ISSN: 1072-5520.
[12] Clare Marie Karat. Cost-justifying usability engineering in
the software life cycle. In Handbook of Human-Computer