=Paper=
{{Paper
|id=Vol-490/paper-4
|storemode=property
|title=A Measurement-Based for Adopting Usability Engineering Methods and Tools
|pdfUrl=https://ceur-ws.org/Vol-490/paper_04.pdf
|volume=Vol-490
|dblpUrl=https://dblp.org/rec/conf/iused/MetzkerS09
}}
==A Measurement-Based for Adopting Usability Engineering Methods and Tools==
https://ceur-ws.org/Vol-490/paper_04.pdf
Adoption of Usability Engineering Methods:
A Measurement-Based Strategy
Eduard Metzker Ahmed Seffah
Vector Informatik, Stuggart Germany EHL-LHR, Lausanne Switzerland
eduard.metzker@gmx.net seffah.ahmed@ehl.ch
ABSTRACT Furthermore, there is little hard evidence backing up new
In the context of a software development organization, two technology to be adopted, and their costs and benefits are
strategies are possible for introducing and institutionalizing rarely understood [1]. Without this data, choosing a particular
new usability engineering methods. The first one, expert-based technology or methodology for a project at hand essentially is
institutionalization, require to resort to third party companies a random act with many consequences [2]. The findings from
or experts that can, based its previous expertise, assist the a very large survey made by Standish group, new technology
team in selecting, implementing and institutionalizing is one of the top ten reasons for projects failure or success
usability engineering methods and tools. The second one, a [27].
measurement-based strategy, is based on empirical evidence
for learning and assessing the appropriateness, usefulness of a In order to support and effective adoption, a new metrics-
usability engineering method. This paper proposed to combine based approach comprising a process model and a support
these two approaches in a single process metrics support environment are presented in this paper. A large case study
environment for selecting and institutionalizing usability was developed to assess the acceptance of the approach by
engineering methods. The proposed approach has been development teams. The evaluation involved 44 professional
validated via in a cross-organizational empirical study software engineers from five medium to large-sized
involving several software engineers from five mediums to organizations. The evaluation method is discussed including
large sized software development companies. context, method, subjects, procedure and results. Implications
of the results on the design of metrics-based strategy are
discussed for adopting new technology and assessing their
Keywords acceptance by project teams. Based on the results, a set of
Metrics, usability, usability engineering methods, guidelines is derived to optimize the acceptance of metrics
institutionalization, adoption, software developments exploitation approaches by project personnel.
organization
2. THE PROPOSED METRICS SUPPORT
1. INTRODUCTION
ENVIRONMENT
Within the scope of this research, by adoption we refer to the
process and the related tools for selecting the appropriate new The overall goal of the proposed approach, called Adoption-
software development technology while assessing their Centric Usability Engineering (ACUE), is to facilitate the
suitability to the project needs and size as well as the adoption of UE methods by software engineering practitioners
capability of the personnel to use effectively and efficiently and thereby improve their integration into existing software
the new established technology. Adoption has been always a development methodologies and practices. ACUE is designed
key challenge for software development organizations [28]. It to support project teams in institutionalizing this abstract
was reported that the management staff commitment and the knowledge about UE methods and to help them transfer this
involvement of the personnel represent the top factors that knowledge into their development processes. UE methods are
impact on the success with a new technology when first perceived as integrated into an existing software development
introduced [27, 29, and 30]. process when they are adopted by the project team, i.e. when
they are accepted and performed by the project team.
However, despite management efforts made by organizations
to render the transition more “user-friendly”, the associated ACUE exploits empirical data collected in different projects to
help and training material, although precise and perfectly yield stronger evidence on how the method works in a certain
describing the new method are often delivered in an esoteric context. The data form an empirical base to guide the
and unreadable language. Another important factor is that improvement of UE methods and to facilitate the informed
organizations and managers are usually overly optimist in selection and deployment of UE methods in future projects. If
their employees’ ability to quickly master a new technology. this approach is applied repeatedly in a number of projects
The reality is that understanding how to apply the technology over time, it leads to an incremental construction of a body of
is a long and arduous process. evidence to guide usability engineering method selection
(Figure 1).
increase its probability of selection in similar projects in the
future.
The characteristics of the project are specified using the
context models stored in the repository. A context model
includes context factors to describe various project constraints
such as the resources available in the project or the type of
product to be developed. Each context model consists of a set
of factors that can have nominal, ordinal or interval scale
measures [11]. An example for an ordinal factor that describes
a property of a product to be developed is ‘user interface
interaction complexity’. This factor may have three
characteristics ‘text-based interface’, ‘graphical user interface’
or ’multi-modal interface’. Depending on the project
characteristics, appropriate methods are suggested by the
system. Candidate methods are ranked according to two
different criteria: (1) similarity between the method and the
project characteristics, (2) results of assessments from project
teams that used the methods in previous projects.
Within, the system the usability engineering methods are
provided in the format of method packages. Each package
contains a textual description of a method that is structured
according to the pyramid principle [12]. Auxiliary material
such as templates, checklists and tools is linked to each
Figure 1: The overall view of ACUE approach package. This material facilitates easy compliance of the
method described. The process guidance remains passive and
The support environment is called ESPrEE (Evidence-based does not enforce the performance of the methods proposed.
Software PRocess Evolution Environment). The components
of ESPrEE are integrated via a web portal and they be The constraints of industrial software development projects
remotely accessed using any web browser. Its core often enforce the invention of new methods or the adaptation
functionalities are: and streamlining of existing methods. For these reasons the
environments provides means for capturing methods and
• At the beginning of a new project, the metrics-based integrating them into the repository.
method selection component of the environment is used
to configure the set of usability engineering methods that 3. EVALUATION
will be applied in the project
• After the method selection has been completed, the 3.1. Specific purpose of the evaluation
environment generates a project-specific hyper-media
workspace in which the methods selected are graphically A large number of measurement programs are suspended or -
visualized according to the project phases in which their in the worst case – failed. This is because the measurement
usage is intended program is not accepted by stakeholders for the following
• At the completion of major project phases or in post reasons [3, 4, 5, 6 and 7]:
mortem sessions [13], the quality of the methods
employed is assessed by the project team against a 1. The measurement process is perceived as tedious and
quality model. For this purpose quality models contain a time consuming
set of quality factors and carefully defined rating scales 2. Effort and benefits of the program are poorly distributed
3. The impact on daily practice is perceived as being too
The core of the system is a fuzzy multi-criteria decision- low to justify sustained effort
making engine. The characteristics of the usability methods 4. Metrics support tools and/or processes are difficult to use
and projects are defined as sets of fuzzy sets. Based on these
models the engine is able to compute similarity measures for To examine if the proposed approach addresses these issues,
projects and methods to facilitate decisions based on analogy. the evaluation study was designed with the following
The engine is coupled with and assessment component. If a questions in mind:
method receives a poor assessment in a certain project context,
the method’s characteristics are automatically adapted to • Does each member of the project team understand the
reduce the probability of the method being selected in similar basic principles and structure of the method without
projects. On the other hand, if a method has successfully been extensive training?
applied in a certain project, its characteristics are adapted to
• How do project managers assess the potential quantitative 3.4 Method
effects of the approach on their practices? Would they In this study, Davis’ technology acceptance model (TAM)
use the approach in their setting? [14] was used. TAM’s assessment dimensions ‘perceived
• Which problems the project personnel may face when utility’ and ‘perceived ease of use’ were extended while
applying the metrics support tool underlying the proposed adding ‘understandability’ as a third dimension. TAM
framework? postulates that tool acceptance can be predicted by measuring
two dimensions: the perceived usefulness (PU) and the
3.2. Context of the evaluation perceived ease-of-use (PEU) of a system.
We used a set of five medium- to large-size software The perceived usefulness of the system expresses the
engineering companies developing advanced next-generation “subjective probability that using a specific application
home entertainment systems, driver assistance technology for
system will increase (the user’s) job performance within
passenger cars and military support systems. The usability of
these systems has been recognized by the organizations as a an organizational context”, i.e. it is a measure for the
crucial quality factor. While usability engineering methods perceived utility of the system.
[10] are well-know by these companies to ensure the
development of software with high usability, no experience The perceived ease-of-use is the main factor that
with usability engineering was available in the engineering influences the acceptance of a system. Davis defines
teams. ESPrEE was configured for this environment. perceived ease-of-use as “the degree to which the user
Appropriate context and quality models were defined and expects the target system to be free of effort”, i.e. it is a
usability engineering methods were captured in method measure for the usability of a system.
packages. Resources included successful methods invented in
previous industrial engineering projects such as reported in
[16], methods distilled from literature on usability engineering Together, perceived ease-of-use and perceived
[10, 17, 18], and recent research results such as Spencer’s usefulness constitute the person’s attitude toward using
‘streamlined cognitive walkthrough’[19]. This initial a system. The attitude (A) and the perceived ease-of-use
population of the support tool was performed by a team of (PEU) influence the behavioral intention (BI) which can
professional usability engineering experts and took about 2.5 be used to predict the actual use of the system. The
man-months of effort. The participating organizations were technology acceptance model (TAM)
part of a government-supported software engineering research
consortium. However, no organization committed to adopt the TAM’s dimension of perceived utility was further divided into
approach prior to the evaluation. the following sub-dimensions:
3.3 The Subjects • Perceived compatibility with daily practice
• Perceived increase in efficiency
All 44 subjects participated in the study on a voluntary basis. • Perceived usefulness
Of them, 39 are full-time employees working as software • Perceived support of working tasks
engineers for the companies described in section 3.2. Five
subjects were graduate students working for the companies on The sub-dimension of perceived utility was measured by
a part-time basis. The subjects were involved in developing qualitative effects analysis while usability was examined by
highly interactive software systems in a variety of domains, studying user behavior during the user’s interaction with the
e.g. driver assistance systems, home entertainment, and metrics support environment [10]. Understandability was
military defense systems. Based on their experience with the examined via a knowledge test in which the subjects answered
development of highly interactive systems, the subjects were questions on the concepts of the overall approach and the
classified into three groups: new employees (NE, 10), metrics support environment. The knowledge test was
software engineers (SE, 21), and usability engineers (UE, 13) performed before and after the interaction with the tool to
[10]. In the following, these groups are referred to as user study if and how the understanding of the concepts by the
groups for reasons of simplicity. subjects changes.
To implement the study two basic techniques were deployed:
questionnaires and scenario-based task solution. The purpose
of the two questionnaires deployed (q1 and q2) are
summarized in table 1.
Table 1: Purpose of the questionnaires deployed
Data collected
q1 • Subject characteristics (age, qualification, professional experience)
• Typical role of subject in the software engineering process
• The subjects knowledge on the concepts of the approach and the metrics environment (pre-test, based
on the introductory material supplied)
q2 • The subjects knowledge on the concepts of the approach and the metrics environment (post-test,
based on the scenario-guided interaction with the metrics environment)
• The subjects assessment of the utility and usability of the metrics environment
The scenario-based tasks that were specific for each user 4.5 Tasks
group forms the core of the evaluation. While the subjects To identify potential problems and study the acceptance of the
were working on a task, behavioral data and any comments metrics support environment under realistic conditions,
made while thinking out loud were captured as a basis for specific usage scenarios were developed for each user group.
improving the metrics environment. Section 4.5 describes the Each scenario reflects the role of the subject and details the
tasks that were performed by the subjects while the exact tasks to be solved.
evaluation procedure and deployment of the methods is
described in section 4.6. All scenarios were embedded in a cover story that set a
common background for all scenarios. Scenario S0 is
equivalent for all user groups. In S0, the subjects were allowed
to freely explore all components of the metrics environment.
The other tasks to be solved in each scenario are different
among the user groups (Table 2).
Table 2: Tasks to be solved in scenarios
Tasks
NE-S1 Find an explanation of the term usability engineering. Mark the section for later exploration.
NE-S2 Find an introductory article about evaluation and tests. Review the material supplied.
NE-S3 Open the hypermedia workspace for the project DIGital. Find and open the method package on
heuristic evaluation.
SE-S1 Browse all method packages available for project DIGital. Find and display a package on heuristic
evaluation. Assess the quality of the method heuristic evaluation.
SE-S2 Comment the method ‚heuristic evaluation‘. Edit the method ‚heuristic evaluation‘. Extend the
method package with a checklist to be used in ‚heuristic evaluation‘.
SE-S3 Create a new method package. Fill the package with given raw input material. Specify the meta-
data of the methods context model. Link the package to related packages.
UE-S1 Create a new project PORTAL. Choose a context model and specify the project characteristics via
the context model. Choose appropriate method packages based on the project characteristics
specified. Trigger generation of hypermedia workspace for the project PORTAL.
UE-S2 Manage access to the hypermedia workspace for the project PORTAL. Invite project team
members. Manage access levels of project team members.
3.6. Test procedure and scripts the tasks. After the tasks of all scenarios were completed,
questionnaire q2 was handed out to the subject. The evaluation
Prior to the evaluation sessions, all the subjects received is concluded with a brief free discussion.
introductory material. It described the concepts of the metrics
approach and the components of the related environment. Two observers were involved in each evaluation session. One
Single subjects, who, for some reason, had no access to the observer recorded the behavioral data, while the other was
material prior to the evaluation, were given the opportunity to responsible for writing down comments from the subjects
study printouts of the material. Each subject had an individual were thinking aloud. During the session, the subjects worked
session, no group sessions were performed. with a laptop with each evaluation lasting of roughly two
hours.
The evaluation session started with a short introduction of the
participants, the procedure, and the objectives of the study. 4. RESULTS AND FINDINGS
The subjects were explicitly informed that the goal of the
evaluation was to assess the utility of the approach and not the The qualitative effects analysis shows that the proposed
capabilities of the participants and that all data would be metrics-based strategy is strongly accepted by the main target
treated confidentially. First questionnaire q1 was handed out. group. However the support environment receives higher-
Next the tasks to be solved were handed out in form of than-average ratings from all subject groups.
scenarios. Scenario S0 was performed by each participant to
promote a free exploration of the system. The time for S0 was 4.1 Understandability of the approach
limited to 20 minutes. Next, the group-specific scenarios were
handed out to the subjects. No time limit was set for task The understanding of the metrics collection approach and the
completion. The subjects were encouraged to articulate metrics support environment by the subjects was measured
impressions and problems and think aloud while performing before and after usage of the support system via the
questionnaires q1 and q2. After reading the introductory
material, the average percentage of correct answers was 36%. 3,4
NE 4,3
3,4
Subsequent to performing the scenario-based tasks, this value 3,5
3,6
doubled, being up to 63%. The performance of the groups and
the overall performance of the subjects are depicted in figure 3,3
SE 4,4
2. It shows that even the relatively short time of usage of the 3,1
3,5
metrics environment led to a significant increase in the 3,4
understanding of the concepts and components of the 3,7
approach. The increased understanding of the overall approach UE 4,5
3,9
4,1
was lowest in the group of new employees (NE). However this 4,3
can be easily explained since their scenarios (NE-S1-S3) did
3,4
not comprise the usage of all components of the metrics ∅ 4,4
3,3
support system. 3,7
3,7
1 2 3 4 5
43%
NE 48% compatibility with daily practice
35% increase in knowledge
SE 71% usefulness as a tool
35% increase in the efficiency of UE
UE 62%
support of task
∅
36% Figure 3: Results of the qualitative effects analysis
63% (assessment scores for utility dimensions with respect to
user group, 1: very low, 5: very high)
0% 20% 40% 60% 80%
post-test pre-test The results indicate that the approach and its support
environment were generally assessed by all groups as higher-
than-average useful and reasonable. All subjects seem to
Figure 2: Pre- and post-test scores in knowledge tests with highly appreciate the potential of the approach for increasing
respect to subject groups (in percentage of correctly their knowledge of usability engineering methods. The
answered questions) dimension ‘task support’ receives the highest scores from the
UE group. This corresponds with the pre-configuration of the
4.2 Usefulness of the approach support environment with usability engineering methods and
the subjects role as usability engineers. It could be concluded
For the qualitative effects analysis, the subjects were asked to that the assessment of this dimension by the other groups
assess the properties of the metrics support environments could be further enhanced, if also usability engineering
along with the utility dimensions defined in section 3.4Each methods for other areas such as ‘requirements engineering’ or
subject filled out questionnaire q2 after performing the methods for enhancing programmer productivity were
scenario-specific tasks. For this reason questionnaire q2 integrated into the method repository of the metrics
includes a number of items to be rated on a five-level Likert environment. This result underpins the necessity to provide
scale [20] for each dimension. Figure 3 sets out the results of benefits for all groups of project personnel involved in metrics
the qualitative effects analysis. The bars represent ratings of collection and exploitation.
the assessment dimensions. The mean ratings were calculated
for each dimension and grouped according to the subject 4.3 Recommendations for usability
groups.
improvements
The behavioral data and user comments recorded during task
performance suggest that there is potential for improving the
usability of the metrics support environment. The distribution
of usability issues identified by subjects across the client
components of the metrics support environment are presented
in figure 4.
Most improvement suggestions are related to the components
for process guidance and metrics-based decision support. The
high number of usability issues identified for the process
guidance component can be partially explained by the fact,
that the scenarios of all user groups (NE, SE, UE) included
interaction with the process guidance component.
deploying the methods in the project. Finally the project
7% assesses the quality of the methods deployed. The approach is
supported by a tool, an intranet that offers a web-based
11%
support system. The approach has been successfully
20% implemented in industry.
32%
Instead of evaluating the approach in an isolated longitudinal
30% case study, a study was performed to examine the acceptance
of the approach by practitioners from various organizations.
0% 10% 20% 30% 40% The acceptance was measured in scenario-driven evaluation
cross component issues sessions, by capturing the understandability, perceived utility
web portal and perceived usability of the approach. The results indicate
method assessment and improvement component
process guidance component that the approach is accepted by the subject groups examined.
metrics-based method selection component We recommend using the study described as a template to
estimate the initial acceptance when introducing tool
Figure 4: Distribution of the usability issues identified over supported measurement programs into organizations. Such
client components of metrics support environment studies can be useful for early identification of acceptance
problems that hamper the log-term success of metrics
One issue is that more assistance in working with the UE programs. The usability of the metrics environment will be
methods is appreciated by the subjects. In particular novice further improved using the feedback gathered.
users could profit from concepts such as wizards to augment
metrics capturing. Moreover the consistency between the 6. ACKNOWLEDGEMENTS
applications should be increased. Finally the parts of the
terminology used in the graphical user interfaces of the Part of the empirical study presented in this paper was
metrics environment should be revised for more originally conducted at Daimler Chrysler. We would like to
comprehensible terms. One example given was that subjects thank Elke Wetzstein and Gerd Jan Tschoepe from the
suggested changing the term “project context model” to Institute for Industrial Psychology of the Humboldt University
“project characteristics”. Berlin for their support in preparing and conducting the
evaluation sessions. Part of this research work was funded by
the BMBF (German Ministry for Research and Education)
under the project EMBASSI (01IL904I). We would like to
5. A CONCLUDING REMARK thank also the National Science and Engineering Research
Council of Canada and Daimler Chrysler for their financial
In this paper, an approach for adopting UE methods was support to the human-centered software engineering group at
proposed. It consists of a three-phased process. First, usability Concordia University.
engineering methods are selected for a new project based on
the projects constraints. Then the project team is supported in
[5] R. T. Hughes, “Expert Judgment as an Estimating
REFERENCES Method,” Information and Software Technology, pp.
67-75, 1996.
[1] D. E. Perry, A. A. Porter, and L. G. Votta, [6] F. Niessink and H. Van Vliet, “Measurements
“Empirical Studies of Software Engineering: A Should Generate Value, Rather than Data.,” in Proc.
Roadmap,” in Proc. International Conference on Sixth IEEE International Symposium on Software
Software Engineering (ICSE2000), 2000, pp. 345 - Metrics, 1999, pp. 31-39.
355. [7] O. Laitenberger and H. M. Dreyer, “Evaluating the
[2] V. R. Basili, F. Shull, and F. Lanubile, “Building Usefulness and the Ease of Use of a Web-based
Knowledge Through Families of Experiments,” Inspection Data Collection Tool,” in Proc. Fifth
IEEE Transactions on Software Engineering, vol. IEEE International Symposium on Software Metrics,
25, pp. 456-473, 1999. 1998.
[3] D. R. Goldenson, A. Gopal, and T. Mukhopadhyay, [8] S. Komi-Sirviö, P. Parviainen, and J. Ronkainen,
“Determinants of Success in Software Measurement “Measurement Automation: Methodological
Programs: Initial Results,” in Proc. Sixth IEEE Background and Practical Solutions - A Multiple
International Symposium on Software Metrics, Case Study,” in Proc. 7th IEEE International
1999, pp. 10-21. Software Metrics Symposium, 2001, pp. 306-316.
[4] B. G. Silverman, “Software Cost and Productivity [9] L. Rosenberg and L. Hyatt, “Developing a
Improvements: An Analogical View,” IEEE Successful Metrics Program,” in Proc. 8th Annual
Computer, vol. May 1985, pp. 86-96, 1985. Software Technology Conference, 1996.
[10] J. Nielsen, Usability Engineering: Morgan Kaufman
Publishers, 1994.
[11] L. Briand, K. El Emam, and S. Morasca, “On the [20] C. M. Judd, E. R. Smith, and L. H. Kidder, Research
Application of Measurement Theory in Software Methods in Social Relations, 6 ed: Harcourt Brace
Engineering,” Empirical Software Engineering, vol. Jovanovich College Publishers, 1991.
1, pp. 61-88, 1996. [21] G. F. Smith and S. T. March, “Design and Natural
[12] B. Minto, The Pyramid Principle - Logic in Writing Science Research on Information Technology,”
and Thinking, 3rd ed. London: Minto International Decision Support Systems, vol. 15, pp. 251-266,
Inc., 1987. 1995.
[13] A. Birk, T. Dingsøyr, and T. Stålhane, “Postmortem: [22] R. D. Galliers and F. F. Land, “Choosing
Never Leave a Project without it,” IEEE Software, Appropriate Information Systems Research
vol. 19, pp. 43-45, 2002. Methodologies,” Communications of the ACM, vol.
[14] F. D. Davis, “A Technology Acceptance Model for 30, pp. 900-902, 1987.
Empirically Testing New End-User Information [23] T. DeMarco and T. Lister, Peopleware: Productive
Systems: Theory and Results,,” in MIT Sloan Projects and Teams, 2. ed. New York: Dorset House
School of Management. Cambridge, MA, USA: MIT Publishing, 1999.
Sloan School of Management, 1986. [24] Y. Malhotra and D. F. Galletta, “Extending the
[15] B. A. Kitchenham, “Evaluating Software Technology Acceptance Model to Account for
Engineering Methods and Tools,” ACM SIGSoft Social Influence: Theoretical Bases and Empirical
Software Engineering Notes, pp. 11-15, 1996. Validation,” in Proc. Thirty-Second Annual Hawaii
[16] E. Metzker and M. Offergeld, “An Interdisciplinary International Conference on System Sciences, 1999,
Approach for Successfully Integrating Human- pp. pp. 1006.
Centered Design Methods Into Development [25] A. Cockburn, Agile Software Development:
Processes Practiced by Industrial Software Addison Wesley Longman, 2001.
Development Organizations,” in Engineering for [26] N. Juristo and A. M. Moreno, Basics of Software
Human Computer Interaction: 8th IFIP International Engineering Experimentation. Dordrecht: Kluwer
Conference, EHCI 2001(EHCI' 01), Lecture Notes in Academic Publishers, 2001.
Computer Science, R. Little and L. Nigay, Eds. [27] Standish Group. “CHAOS Chronicles or CHAOS: A
Toronto, Canada: Springer, 2001, pp. 21-36. Recipe For Success”. 1995.
[17] L. L. Constantine and L. A. D. Lockwood, Software [28] Bayer, J. and Melone, N. Adoption of Software
for Use: A Practical Guide to the Models and Engineering Innovations in Organizations. Technical
Methods of Usage-Centered Design: Addison- Report CMU/SEI-89-TR-017, Software Engineering
Wesley, 1999. Institute, Carnegie Mellon University.
[18] D. J. Mayhew, The Usability Engineering Lifecycle: [29] Desmarais M.C., Leclair R., Fiset J.Y., Talbi H.
A Practioner' s Handbook for User Interface Design: “Cost-Justifying Electronic Performance Support
Morgan Kaufman Publishers, 1999. Systems.” Communications of the ACM, Vol. 40,
[19] R. Spencer, “The Streamlined Cognitive No. 7, July 1997.
Walkthrough Method: Working Around Social [30] Howard R. “Software Process Maturity: Measuring
Constraints Encountered in a Software Development Its Impact on Productivity and Quality.” IEEE
Company,” in Proc. Conference on Human Factors International Software Metrics Symposium, May
in Computing Systems (CHI' 00), 2000, pp. 353-359. 1993.