=Paper=
{{Paper
|id=None
|storemode=property
|title=Using and Integrating Discount Usability Engineering in the Life Cycle of a Health Care Web Application
|pdfUrl=https://ceur-ws.org/Vol-729/paper4.pdf
|volume=Vol-729
}}
==Using and Integrating Discount Usability Engineering in the Life Cycle of a Health Care Web Application==
https://ceur-ws.org/Vol-729/paper4.pdf
Using and integrating Discount Usability Engineering in
the life cycle of a health care Web application
Fabrício Ferracioli Maria Angélica de Oliveira
Universidade Estadual de Londrina Camargo-Brunetto
Rodovia Celso Garcia Cid - PR 445 km 380 - Universidade Estadual de Londrina
Campus Universitário Rodovia Celso Garcia Cid - PR 445 km 380 -
Londrina, Brazil Campus Universitário
fabricioferracioli@gmail.com Londrina, Brazil
angelica@uel.br
ABSTRACT In our university, there is a health care unit responsible for
Usability is an important characteristic in any interactive the physical rehabilitation of patients with chronic respira-
system, but sometimes is neglected by some software de- tory diseases. Chronic diseases in general require continu-
velopment teams because of the lack of knowledge of these ous attention to control the occurrence of crisis. Aiming to
about usability techniques. They think that the methods help health care professionals (mainly in cardiac and res-
are hard to learn, execute and integrate on the software de- piratory rehabilitation) to accompany their patients indi-
velopment life cycle. A kind of application that suffer with vidually as well as to study groups of patients, according
usability problems is the health care software. Our team to specific features, it was designed, developed and imple-
work with a health care software developed by people from mented the software SacarWeb (Software for Assessment of
computer science and physiotherapy, without people related Cardiac and Respiratory System for Web) [5]. A team of He-
to usability. After various functions developed, the users alth Informatics of our university works continuously in this
continuously claim for easier ways to use the software, and software. Cardiac and respiratory assessment can be made
we detect some learnability problems. At this point we de- through several types of exams, that have specific material
cided to focus in usability, using the methods of Discount and methods for both, execution techniques and analysis of
Usability Engineering. This work presents the methodology results (interpretation). Despite of a lot of effort in produ-
used during the heuristic evaluation and usability tests. The cing improvements, the software have presented some dif-
results show that is possible to reach a satisfactory amount ficulties of use. Aiming to reduce this difficulties, we have
of problems to correct, great feedback from users, similar begun to investigate about software usability.
results between heuristic evaluation and usability tests with
users and that even people with few knowledge about usa- Intuitively the the development’s team of SacarWeb begun
to apply some fast usability tests with people from the team
bility can learn and conduct tests like ones of the Usability
Engineering. Additionally, these methods can be integrated and to make some design decisions based on what they think
in our software life cycle, what will avoid future re-work. that is good about usability. Later, the team evolved to a
better approach, trying to base their design solutions in the
Categories and Subject Descriptors literature or in rational data about similar systems. That’s
J.3 [Computer Applications]: Life and Medical Scien- the phase that the team is encountered today. Unsatisfied
ces—health; D.2.4 [Software Engineering]: Software/Program with design bad results, generic and poor data, the team
Verification decided to evolve to a better approach doing tests with real
users and with well defined methodology. The evolution of
care about usability from the team is like the one described
General Terms in [22].
Design, Human Factors
Usability is known for its importance and results that can
Keywords be reached. It is important because applications designed
Usability, Discount Usability, Health Care Web Applications with usability in mind can achieve better satisfaction of their
users, improved performance and better effectiveness [17].
1. INTRODUCTION Some studies show that the investment return rate is high
[21, 23], the benefits for the users considering ease of lear-
ning and the reached objectives are also very high. In the
majority of cases it is better to invest in improving usability
that in training people to use software [21, 12].
Despite of its importance, usability is known by its adoption
difficulties in teams consisting only of software developers
or people with deep knowledge in Computer Science [10].
Besides, it is not so common to include this discipline in
the undergraduate Computer Science Curriculum [7]. So- software of the usability evaluation; in section 4 it is pre-
metimes the professionals simply do not give the necessary sented the methodology of applying the Discount usability
importance to usability, thinking only in working software. Engineering. In section 5 the results are shown and discus-
Even considering those that think important to adopt usa- sed. Finally conclusion and future work are described in the
bility techniques during the design, it is supposed that such section 6.
techniques are difficult to learn and use, once they require
high knowledge and experience [20]. 2. DISCOUNT USABILITY ENGINEERING
The Usability Engineering defines quantitatively the usabi-
It seems that this is one of reasons of poor usability in several
lity goals of an application in the beggining of its life cycle.
areas, resulting in low rates of software acceptance and high
The results are taken throughout the product life cycle [10].
rates of software rejection. One of these areas is health care
The full process has eleven stages, but an usability effort
applications [2, 15].
that does not include all the stages can be successful. This
is the aim of the Discount Usability Engineering [20].
Traditional approaches to evaluate usability are well known
by their good results. Normally they give rich information
Discount Usability is an alternative to traditional Usability
about the use of the software and compliance with known
Engineering approaches that use only methods that require
patterns. These approaches have high time consuming, be-
specific knowledge of usability, complex environments for the
cause of the amount of data generated by methods to analy-
analysis and large experience. The purpose of such appro-
sis and require more knowledge to be performed, once the
aches is to identify usability problems even if the tests are
methodology of such methods is highly rigorous. One ap-
made by people without experience or specific knowledge
proach example is the full Usability Engineering life cycle,
about usability. Because of this, there is a better chance to
consisting of eleven general stages, where various options of
be used in practical design situations [20, 19].
methods are available, including traditional ones. Such ap-
proaches can discourage the adoption of usability techniques
The Discount Usability Engineering is based in four methods:
in teams without or few knowledge about them [20].
There are alternative approaches to traditional methods to • User and task observation
evaluate software usability with similar quality of results
indicated for people that know the importance of software • Scenarios
usability, but do not have enough resources, or the neces-
sary knowledge to adopt traditional methods [20]. One of • Simplified think aloud
these approaches is known by Discount Usability [19], that
uses only the most important techniques of the full Usability • Heuristic Evaluation
Engineering.
The four techniques are based in both inspection and test
Similar situation is found in the development team of the
techniques. Inspection techniques do not use final users du-
health care Web application, called SacarWeb [5], at our
ring the tests, test techniques use them. Better results are
university. The design and development team consist of pe-
reached mixing these two kinds of techniques [11].
ople related to computing and physiotherapy, but no one
with deep knowledge of usability. Only common sense was
User evaluation is the basic principle and first step of any
used during the interface development and no one study or
Usability Engineering method, and it is maintained in the
usability inspection was done in all the application life cycle,
Discount Usability. The aim of this task is to know the
because of the lack of knowledge of this area in the members
user, or at least to have a feeling on how the product will be
of the team.
used. The users observation must address their individual
characteristics, the tasks performed, the reasons to make a
After the development of several improvements related to
task and the users’ evolution. This can be achieved in many
software functionality [8, 9], some difficulties of learning how
ways, like visiting the users’ workplace, since the observation
to use the software remained. Learnability is an important
does not interfere in users’ workflow. Because of it, the
metric in the addressed Web application, SacarWeb, consi-
observer should keep quiet and let the users work as they
dering high rotativity of users, once most of them are stu-
normally do without interference.
dents of physiotherapy. At this point, the team decided
to verify the software usability using alternative techniques,
A prototype is a working model built to develop and test
like the ones in Discount Usability, aiming to improve the
design ideas that can be used to examine content, aesthetics
software usability.
and interaction techniques from the perspective of designers,
clients and users. Usability professionals test prototypes as
Based on this, the objective of this work is to indetify a set
they perform typical tasks of the intended use of the product,
of usability problems in SacarWeb in a way to prioritize the
once prototypes are a good way to find usability problems in
future improvements in the application through Discount
early stages of the design process. Prototypes more similar
Usability Engineering.
to the final product are called high fidelity, while those less
similar are low fidelity [27]. Scenarios are a kind of low
The remaining of this paper is organized in this way: in
level prototype that cut down complexity of implementation.
section 2 it is presented the main concepts of Discount Usa-
They reduce the level of functionality and the number of
bility Engineering; in section 3 it is described the target
features. They simulate only some parts of the application,
but for that the user needs to follow a previously planned of different equations of normal values, including Brazilian
path [20]. standard [6], American standard due to Knudson [14] and
European standard by the Official Statement of the Euro-
In the simplified think aloud method one user at a time uses pean Respiratory Society [16]; Exams to test exercise capa-
the system for a given set of tasks while is asked to think out city: Shuttle Walk Test [25], the 6 Minutes Walking Test
loud. By verbalizing their thoughts, users allow an observer [26].
to determine not just what they are doing, but also why
they are doing it. This is hard to obtain in other methods, Various iterations for development of new functionalities oc-
and can be a rich source of information about the inter- curred in the software life cycle, but no one tested the usa-
face [11], pinpointing interface elements that cause misun- bility. Some of these functionalities were to improve the
derstandings. The main differences between traditional and software use, others to improve the performance [1], [8], but
simplified think aloud is the presence of psychologists, user the results were not the expected. Often the users claim for
interface experts and videotapes on the traditional method. better consistency of the application, like the standardiza-
To analyse these sources of information takes much time and tion of some application behaviors and screens, and to make
can intimidate developers, for example. The simplified think the information about patients or other secondary functio-
aloud use only notes taken by the observer about what the nalities more easier to find and use.
user says and does while using the system [20]. A general
protocol to perform this test is given by [3]. 4. METHODOLOGY
Only most commom tasks used in SacarWeb were analyzed,
Heuristic evaluation is a systematic method for the usabi- because these are the most used in the software. The practi-
lity inspection of one interface with aim to find problems in cal results inspecting the common tasks appears faster than
its design and it is considered the most common informal analyzing the full software [20]. These tasks are:
method [11]. The heuristics are collections of few usabi-
lity guidelines that are good to be attended in an applica-
tion. With them, even non experts can find many usability • To insert patient’s identity;
problems. The heuristic evaluation involves a small set of
evaluators using the software and examining the interface, • To insert patient’s exams results (six different exams);
judging its compliance with recognized usability principles
found in the heuristics. Its conducted verifying the inter- • To verify patient’s evaluation through various evalua-
face and trying to formulate an opinion about what’s good tions;
and bad about it. The findings can be completed with the
• To apply a filter to extract group of patients satisfying
results of the simplified think aloud [24, 18, 20].
certain conditions.
As Discount Usability is an approach that aims to optimize
the use of resources, it is easy to apply and can achieve re- In Heuristic Evaluation, these tasks are analyzed by suitable
sults with cost benefit better than traditional methods. This evaluators that interacts and evaluate the interface and in
is just because the emphasis is in early and fast interactions Usability Evaluation, these tasks are analyzed by evaluators
with frequent usability entrance. too, but through observation of the users interaction with
software. The data inserted in the software were collected
3. SACARWEB from real evaluations with patients, and the printed forms
SacarWeb (Software for Cardiac and Respiratory Evaluation were the same used by the clinical evaluators.
on the Web) is a software used by a physiotherapy research
group at Universidade Estadual de Londrina (UEL) respon- The aims of the usability test was:
sible for rehabilitation of patients with chronic respiratory
diseases [5]. These diseases, in general, need to be monito-
red continuously to control the crisis occurrence. With the • to improve the learning rate;
aim to help the professionals to follow their patients indi- • to decrease the occurence of user’s error.
vidually and study groups of patients according to specific
characteristics the SacarWeb was created. Using the soft-
ware, students, researchers and teachers can teach and learn These objectives were defined, considering that the most of
concepts, keeping a database with patients identity, results the user are students of physiotherapy, with different grades
of clinical exams, and using it to observe the evolution of of knowledge of the specific domain, and also there is a high
patients as well as to extract group of patients with features rotating of users. It is important that the evaluation has
of interest. clear objectives in order that the evaluator has the correct
focus on users and interface during the evaluation [20].
By means the module Patient Assessment, which has the
core system functionalities, it is possible to perform data Both categories of users during the evaluations: real users
collecting and analysis of patient exams. Each assessment and heuristic evaluators use a similar computer environ-
consist of a set of exams. The exams available are grouped ment, and it was established the following standard for the
by category: Questionnaires to assess quality of life: Chro- evaluations:
nic Respiratory Disease Questionnaire (CRQ) and St. Ge-
orge’s Respiratory Questionnaire Paul Jones (SGRQ) [13];
Tests of pulmonary function: The system provides the use • Mouse and keyboard compatible device;
• Browser Mozilla Firefox version 3.6.x, with javascript who compiled one document with all the usability problems
and cookies enabled; found. This document was sent again to all evaluators for
a severity measurement, that will guide the development of
• 1024x768 minimum screen resolution; new solutions. The severity measurement was made using
the proportion proposed by Jakob Nielsen [20], where:
In order to avoid external interference during evaluation, it
was recommended the following: to minimize the presence
• 0 - its not a usability problem at all
or moviment of other people in the room, no ringing phones,
do not use other programs or operating system. • 1 - cosmetic problem only, need not be fixed unless
extra time is available in project
By default, Discount Usability recommends the use of sce-
narios to evaluate the software. In our case, the software its • 2 - minor usability problem, fixing this should be given
in advanced development stage. Because its not the aim of low priority
this work to compare an alternative design, but to evaluate
the actual solution, we decided to use the high fidelity pro- • 3 - major usability problem, important to fix, so should
totypes instead of scenarios. The similarity with the final be high priority
interface is the major characteristic of this prototype kind.
By the use of the same methods, interaction techniques and • 4 - usability catastrophe, imperative to fix this before
look of the final product, high fidelity prototypes offer more product can be released
realist interactions. Despite of this, any fidelity level is capa-
ble to identify usability problems, and there is no difference
in the amount of problems found by the different levels [4, If the evaluators need to access the software to remember
27]. The decision in this work was made mainly to speed up some interface problem and then rating the severity, they
the evaluation. can use the same login and password used during the evalu-
ation.
4.1 Heuristic Evaluation
The heuristic evaluation was made in SacarWeb by two eva- 4.2 Usability Evaluation
luators, where only one is part of the actual development The interface evaluation with users was conducted by one
team. This one has the best knowledge about the applica- person of the SacarWeb development team, using the sim-
tion domain among the evaluators, but all the evaluators plified thinking aloud method. In the tests, the evaluator
have few or no experience with heuristic evaluation. They has taken notes of all the user actions that provided some
followed the ten heuristics made by Jakob Nielsen [20], that usability insight and his own considerations based in perso-
can be applied in the majority of systems. In order to do the nal perception. Parallely, the user testing the interface was
evaluation, one application guide was given to evaluators. incentived to share his thoughts and feelings while using the
This guide has information about the evaluation and sys- software. During the evaluation all the dialog was recorded
tem’s knowledge domain. These informations include how to assist in the evaluation. After the user test, the evaluator
to operate in specific tasks, how to read correctly the prin- wrote a document with his perceptions and usability pro-
ted forms, which tasks have to be done, the tasks order blems found, based in the notes taken, audio recorded and
and so on. One software maintainer was responsible for as- his usability knowledge.
sisting evaluators during the heuristic evaluation, providing
information about the knowledge domain or how to operate A pilot evaluation was made before the real evaluations with
the application. This kind of information was given only if two not representative users. This evaluation was made to
needed and after the evaluator have exposed the usability calibrate the real test. One change due to this pilot was
problem in question. the data to be considered for learnability and initial perfor-
mance. Initially, the evaluation analyses the two interacti-
For a better understanding of the tasks to be evaluated in ons with exams sets inserted by the user. But the gain of
the software, heuristic evaluators were asked to perform an efficiency from the first to the second interaction with inter-
interface evaluation with high quantities of data to insert face is much huge, interfering in the point that we want to
into the system. Each evaluator has introduced data of two measure. It was expected that the same behavior occured
patients with three complete sets of exams, in such way that in the evaluation with real users. The pilot evaluation gave
each task was performed at least two times. During and after us a notion of time in what users can perform the request
the evaluation the evaluators were asked to compare the user actions, what is important, to give an expected time to the
interface with the heuristics, and filling their findings in an evaluation for the real users during the tests.
online standardized document with the heuristics and tasks
provided, in order to facilitate the application of the method The real evaluation was made with four potential users of
of heuristic evaluation. For each task, the evaluator should the application, with no previous contact with the software,
only to identify the associated heuristic with the problem and basic knowledge related to application domain. The
detected, to describe it and also to justify why such problem real users were divided by specialty into the system, testing
violates the heuristic. only the exams they know. The number of users for testing
is enough, once it is possible to find the majority of inter-
After all the evaluations have been completed, each do- face problems [20]. The tests were conducted only with the
cument with the findings of the evaluators was revised by evaluator and user, at a room in the laboratory where it is
the same SacarWeb maintainer that helped the evaluators, common to use the SacarWeb.
5. RESULTS AND DISCUSSION daily tasks, speeding up and easing some process manually
The Heuristic evaluations have identified many problems. done.
The two evaluators were capable to evaluate the interface,
based on the provided material. The amount of problems Other important finding was the similarity of problems en-
found varied with the knowledge of the domain, where the countered in the heuristic evaluation and in the interface
greater the knowledge about the system’s knowledge do- evaluation with users. 22 of the interface evaluations pro-
main, better the results. The interval of problems found blems’ (25.29%) were found in the heuristic evaluation too,
were [32-78], with 113 problems found, where 4 (3.67%) of but some serious problems were found only in one of the
them are common among all evaluators, which resulted in evaluations, showing the importance of the two kinds used
109 distinct problems found. This is an expected result, be- in this work. The total of serious problems found in the in-
cause a heuristic evaluation is better when several people terface evaluation were 3 (3.45% of total). These problems
conduct it, once more problems can be found [24]. Another were the ones that seriously impacted the users in the com-
factor that can influence the results are the major field of pletion of tasks, terminology or concepts misunderstood that
evaluator’s knowledge. In this work, there were evaluators impacted in the comprehension of system or items that se-
from Computer Science and Interface Design. The two kinds riously slow down the users or estimulate them to redo a
can found the same problems, but with different analysis and finished task. Therefore, the total amount of distinct pro-
workarounds. Also, the problems related to specific kno- blems found were 196, the 109 problems encountered in the
wledge domain are important, because they have great vali- heuristc evaluation plus the 87 distinct problems of the usa-
dity once they are based in the evaluator’s knowledge about bility evaluation with real users.
some specific topic that he deeply knows. The generated
report was a useful resource to development team, because About the two objectives of the usability tests, it is sugges-
they can suggest solutions for future development and im- ted that learning can be much improved after the interface
provement of the system, based in well designed principles. redesign considering serious problems found. As they are
The severity measurement can guide the development, where related to core concepts for the users, that actualy are dif-
the problems with greater score will have priority in their so- ficult to understand, a better interface approach can solve
lutions against others. Because of these results, it is sugges- this. With this, it is expected that the amount of users’ error
ted that the heuristic evaluation can be easily integrated in decrease, because the software will be easy to use. Another
the SacarWeb’s development cycle, and it is expected that two main problems are consistency and information needed
when applied during the project the results can be better to be remembered by users through some screens. These
and the changes in the final software reduced. two points are related to heuristics [20], and a better care
of these points in the some parts of the system can improve
During the tests with the real users of the application we the user experience through the entire software.
found 109 problems. One important finding was that the
general places of the system have common problems with The development team can use the findings of the tests to
different users. Some of them are related to problems with guide their next development iterations, having some cle-
some concepts and terminologies used during the patients arly points with major problems to attack, and the severity
evaluation and into the system in different ways. Generally, measurement and users feedback can guide the development.
the problems found aren’t major problems, that interfere The maintainer that conducted the usability evaluations can
in the system learning, but some of them can severely slow guide the problems corrections, because of his knowledge
the user’s actions. Despite of these general findings, one about them, users and system. With this, it is suggested
major problem was found with one of the system’s function, that is possible to integrate the Discount Usability in Sa-
where 75% of the users could not complete the task. Another carWeb’s life cycle and we have a set of usability problems
major finding was the lack of consistency in different parts to correct in their next development iteration.
of the system. Users created some bad and nocive habits for
them during the system learning that difficulted the task A related study reported that evaluators, with similar kno-
completion, mostly because of unexpected system answers wledge of Computer Science and usability, as that observed
or behaviors, that were repeated through the entire system. in evaluation of Sacarweb, can differ from a good and a bad
In various places some behaviors were not necessary, but the design after learn some usability concepts [19]. As expected
users felt more comfortable when doing it, even when it slow by [20], people with few knowledge about usability can find
down the process. One interest finding was the similarity of usability problems with heuristics and conduct usability ins-
problems between the pilot and the real evaluation, but in pections. In our case, evaluators with more knowledge, or
some cases the real users have much better performance than about the inspections methods, or about the system tested,
the pilot users. It is suggested that this happened because obtained better results.
of the knowledge of the domain that real users have. The
performance gains were observed with real users, and also 6. CONCLUSION AND FUTURE WORK
in the pilot evaluation, confirming the importance of the This work showed the use of techniques proposed by Dis-
calibration with a pilot study. In the exams, the specific count Usability Engineering by the development’s team of
part of the system, we found that the user’s knowledge of SacarWeb. Despite the lack of knowledge about usability,
domain was important, and helped him to evaluate correctly
the group has learned and used the techniques, applying the
the interface and to input the data, differing from the general
knowledge by means of the heuristic evaluation and usabi-
users of the pilot evaluation. Despite some problems, and lity tests. The results of these two tests converged, and are
one task that is difficult to learn, the users felt comfortable a great source for the future development of SacarWeb. By
using the system and they think that it can be useful in their
means of the simplified thinking aloud the users gave us rich
feedback about the interface and concepts of the software. and B. Manaris. Incorporating human-computer
All the evaluations were done with people from team, which interaction into the undergraduate computer science
suggests that these methods can be integrated in the soft- curriculum. SIGCSE Bull., 34:211–212, feb 2002.
ware life cycle. [8] F. Ferracioli and M. A. de Oliveira
Camargo-Brunetto. A utilização das recomendações do
With the great amount of problems found, for future work w3c no sacarweb. Anais do XVI EAIC, 2007.
we suggest a development of a second design based on the [9] F. Ferracioli and M. A. de Oliveira
findings of this work and a second usability test, comparing Camargo-Brunetto. Internacionalização de aplicações
the results that will be found with the ones of this work. A web: uma ferramenta para acelerar a tarefa de
good way to do this task and to have a great impact in the traduçãoo em um ambiente de aplicaçãoo médica.
system is to follow the priority given to the problems found Journal of Health Informatics, 2(2):29–37, apr 2010.
after the heuristic evaluation. Another suggestion is the in- [10] X. Ferre, N. Jurista, H. Windl, and L. Constantine.
tegration of the Discount Usability techniques in this second Usability basics for software developers. IEEE
design phase, with early testing and more clear usability go- Software, 18(1):22–29, Jan 2001.
als defined. With the current objectives reached, it is expec- [11] A. Holzinger. Usability engineering methods for
ted to increase users’ satisfaction and better memorability software developers. Commun. ACM, 48(1):71–74,
rates in the second design. Another suggestion is the real January 2005.
integration and care with usability through a well accepted
[12] J. A. Jacko and A. Sears. 1 edition, sep.
and documented approach, like the Discount Usability tech-
niques, in the development cycle of the application, moving [13] P. Jones, F. Quirk, and C. Baveystock. The st george’s
our maturity stage on usability from level 3 to 4 [22]. respiratory questionnaire. Respiratory Medicine,
85:25–31, September 1991.
[14] R. J. Knudson, R. C. Slatin, M. D. Lebowitz, and
7. ACKNOWLEDGEMENT B. Burrows. The maximal expiratory flow-volume
We thanks to CNPq for grants, Prof. Dr. Antonio Fernando curve. normal standards, variability, and effects of age.
Brunetto (in memoriam), the expertise in respiratory reha- The American review of respiratory disease,
bilitation at Physiotherapy Department (UEL) , Prof. Dr. 113(5):587–600, may 1976.
Fabio Pitta, Nidia Hernandez and Thaı́s Sant’Anna (from [15] A. W. Kushniruk, V. L. Patel, and J. J. Cimino.
Physiotherapy Department - UEL), by organizing the mate- Usability testing in medical informatics: cognitive
rial of patients, indicating people to participate in the evalu- approaches to evaluation of information systems and
ations and by give in the usability test’s local, the students user interfaces. Proceedings : a conference of the
of Physiotherapy for the colaboration in the user evaluation American Medical Informatics Association / ... AMIA
and Carlos Henrique Ferraz, for the colaboration in heuristic Annual Fall Symposium. AMIA Fall Symposium,
evaluation. pages 218–222, 1997.
[16] M. R. Miller, J. Hankinson, V. Brusasco, F. Burgos,
8. REFERENCES R. Casaburi, A. Coates, R. Crapo, P. Enright,
[1] T. A. Amorim, M. A. de Oliveira Camargo-Brunetto, C. P. M. van der Grinten, P. Gustafsson, R. Jensen,
D. dos Santos Kaster, and F. Ferracioli. Remodelagem D. C. Johnson, N. MacIntyre, R. McKay, D. Navajas,
do software sacar-web usando técnicas de engenharia O. F. Pedersen, R. Pellegrino, G. Viegi, and
reversa e reengeharia de software. Anais do CBIS J. Wanger. Standardisation of spirometry. European
2006, 2006. Respiratory Journal, 25(2):319–338, August 2005.
[2] D. Badenoch and A. Tomlin. How electronic [17] M. Morandini. Ergo-Monitor: Monitoramento da
communication is changing health care: Usability is usabilidade em ambiente Web por meio de análise de
main barrier to effective electronic information arquivos de Log. PhD thesis, Universidade Federal de
systems. BMJ, 328(7455):1564+, June 2004. Santa Catarina, November 2003.
[3] T. Boren and J. Ramey. Thinking aloud: reconciling [18] J. Nielsen. Finding usability problems through
theory and practice. IEEE Transactions on heuristic evaluation. In Proceedings of the SIGCHI
Professional Communication, 43(3):261–278, sep 2000. conference on Human factors in computing systems,
[4] A. Coyette, S. Kieffer, and J. Vanderdonckt. CHI ’92, pages 373–380, New York, NY, USA, 1992.
Multi-fidelity prototyping of user interfaces. In ACM.
C. Baranauskas, P. Palanque, J. Abascal, and [19] J. Nielsen. Guerrilla hci: Using discount usability
S. Barbosa, editors, Human-Computer Interaction – engineering to penetrate the intimidation barrier.
INTERACT 2007, volume 4662 of Lecture Notes in Technical report, Nielsen Norman Group, 1994.
Computer Science, chapter 16, pages 150–164. [20] J. Nielsen. Usability Engineering. AP Professional,
Springer Berlin / Heidelberg, Berlin, Heidelberg, 2007. 1998.
[5] M. A. de Oliveira Camargo-Brunetto and A. R. Rossi. [21] J. Nielsen. Return on investment for usability.
Uma aplicação web para análise de exames Technical report, Nielsen Norman Group, 2003.
cardiorrespiratórios. In Simpósio de Qualidade de [22] J. Nielsen. Corporate usability maturity: Stages 1-4.
Software - Workshop de Informática Médica: Anais do Technical report, Nielsen Norman Group, 2006.
Workshop de Informática Médica, June 2005. [23] J. Nielsen. Usability roi declining, but still strong.
[6] S. B. de Pneumologia e Tisiologia. I Consenso Technical report, Nielsen Norman Group, 2008.
Brasileiro sobre Espirometria, 3 edition, 1996. [24] J. Nielsen and R. Molich. Heuristic evaluation of user
[7] S. Douglas, M. Tremaine, L. Leventhal, C. E. Wills,
interfaces. In CHI ’90: Proceedings of the SIGCHI
conference on Human factors in computing systems,
pages 249–256, New York, NY, USA, 1990. ACM.
[25] G. E. Payne and J. D. Skehan. Shuttle walking test: a
new approach for evaluating patients with
pacemakers. Heart, 75(4):414–418, apr 1996.
[26] T. Troosters, R. Gosselink, and M. Decramer. Six
minute walking distance in healthy elderly subjects.
European Respiratory Journal, 14(2):270–274, 1999.
[27] M. Walker, L. Takayama, and J. A. Landay.
High-fidelity or low-fidelity, paper or computer
choosing attributes when testing web prototypes.
Human Factors and Ergonomics Society Annual
Meeting Proceedings, pages 661–665, 2002.