Knowledge of the factors that affect developers' priority of usability evaluation results is important in order to improve the interplay between usability evaluation and software development. In the presented study, the effect of usability inspection severity ratings on the developers' priority of evaluation results was investigated. The usability inspection results with higher severity ratings were associated with higher developer priority. This result contradicts Sawyer et al. [7], but is in line with Law's [5, 6] finding related to the impact of user test results. The findings serve as a reminder for HCI professionals to focus on high severity issues.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
usability inspectors’ severity ratings had no effect on the impact
of the evaluation results; reported impact ratios were 72% (low
severity issues), 71% (medium severity issues), 72% (high
severity issues). In contrast to this finding Law [
The research problem of the present study was formulated as: What is the effect of usability inspectors’ severity ratings on developers’ priority of usability inspection results? The null hypothesis of the study (no effect of severity ratings) followed the findings of Sawyer et al., and the alternative hypothesis (H1) was formulated in line with the findings presented by Law: H1: High severity issues will tend to be prioritized higher by developers than low severity issues.
Usability inspections were conducted as group-based expert
walkthroughs [
Three months after the evaluation reports had been delivered individual interviews were conducted with development team representatives. The representatives were requested to prioritize all usability issues according to the following: High (change has already been done, or will be done no later than six months after receiving the evaluation report), Medium (change is relevant but will not be prioritized the first six months), Low (change will not be prioritized), Wrong (the item is perceived by the developer to be a misjudgment). In order to align the resulting developers’ priorities with the impact ratio definitions of Law and Sawyer et al., the priority High was recoded as ”Change”, and the priorities Medium, Low and Wrong were recoded as “No change”.
The evaluation groups generated totally 167 usability issues. The three objects of evaluation were associated with 44, 61, and 62 usability issues respectively. The total impact ratio (number of issues associated with change/total number of issues [following 7 and 6]) was 27%, which is relatively low. The relationship between the developers’ priorities and the usability inspectors’ severity ratings is presented in Table 1. Visual inspection of Table 1 shows a tendency towards higher priority given to usability issues with severity ratings serious and critical. A Pearson Chi-Square test showed statistically significant differences in priority between severity rating groups; X2=14.446, df=3, p(one-sided)=.001.
The presented results indicate that severity ratings may have significant impact on developers’ priority of results from usability 1 The study reported by Følstad also included separate evaluation groups with work-domain experts. The results of these groups were not included in the current study, in order to make a clearcut comparison with the findings of Law and Sawyer et al. inspections. This finding contributes to our understanding of severity ratings as a characteristic of usability evaluation results that may help to identify which usability evaluation results that are needed in software development.
The finding is particularly interesting since it contradicts the conclusions of Sawyer et al. and therefore may provoke necessary rethinking regarding usability inspectors ability to provide severity assessments that are useful to software engineers. It is also interesting to note that the results are fully in line with Law’s findings related to severity ratings of user test results. The present study may thus serve to strengthen Law’s conclusions. Curiously, the impact ratios of the different severity levels in Law’s study and the present study are close to being identical. Why, then, do the present study indicate that the severity ratings of usability inspection results may have an effect on the developers’ priority, when Sawyer et al. did not find a similar effect? One reason may be the relatively high impact ratios reported by Sawyer et al., something that may well result in a greater proportion of low severity issues being prioritized. Another reason may be that the present study, as the study of Law, favored high severity evaluation results since the usability evaluations were conducted relatively late in the development process [cf. 3]. Sawyer et al. do not state which development phases their usability inspections were associated with, but their relatively high impact ratios suggest that their inspections possibly may have been conducted in earlier project phases. The present study, as the study of Law, indicates that the identification of a low severity usability issue typically is of less value to software developers than the identification of a high severity issue. This should serve as a reminder for HCI professionals to spend evaluation resources on identification and communication of higher severity usability issues.
This paper has been written as part of the RECORD project, supported by the VERDIKT program of the Norwegian Research Council.