Effectiveness and efficiency as conflicting requirements in designing emergency mission reporting Erik G. Nilsson Asbjørn Følstad SINTEF SINTEF Pb124 Blindern, 0314 Oslo, Pb124 Blindern, 0314 Oslo, Norway Norway egn@sintef.no asf@sintef.no particular case; i.e. that in some cases it may be more ABSTRACT Aspects of usability, such as effectiveness and efficiency, fruitful to regard effectiveness and efficiency as are critical for users' overall experience of an interactive requirements or design issues, rather than as variables of a system. In response to the on-going debate on the UX or usability study that may or may not be correlated. relationship between different aspects of usability in BACKGROUND usability studies, we present an example of a User Interface (UI) design case where the relationship between Previous work effectiveness and efficiency should be considered as a The correlation debate requirement or design issue, rather than as variables in Sauro and Kindlund [10] presented the empirically founded usability studies. In the presented case - status reporting single usability metric (SUM). SUM is based on a from an in-vehicle support system for emergency missions - quantitative model where the three standard aspects of these aspects of usability were perceived as conflicting usability (effectiveness, efficiency and satisfaction) are rather than as positively correlated. We present various summarised in one score. design solutions to the task of status reporting and show A basic assumption of SUM is that there are fairly high how the solutions support effectiveness and efficiency in correlations between the three standard aspects of usability. different ways. Finally, we point out some characteristics of This assumption is controversial within the Human- the case that could explain our findings and we suggest how Computer Interaction (HCI) community, as is seen in a future research may obtain more insight into which types of meta-study by Hornbæk and Law [6]. They conclude that applications that may possess similar properties. correlations between effectiveness, efficiency and satisfaction are generally low, and are lower than was found Author Keywords by Sauro and Kindlund. Emergency response, usability, effectiveness, efficiency. In 2009, Sauro and Lewis [11], in response to Hornbæk and ACM Classification Keywords Law, reported what they described as strong correlations H.5.m. Information interfaces and presentation (e.g., HCI): between the standard usability aspects (r between .44 and Miscellaneous. .60 at task level measurements) on the basis of data from 90 usability tests. Sauro and Lewis suggested that the higher INTRODUCTION correlations obtained in their study may in part be explained Usability is seen as a concept that is included in the broader by their study being more representative of the kind of concept of user experience (UX) [9]. Consequently, the usability tests typically conducted by usability professionals usability of an interactive system is critical for the users' whereas Hornbæk and Law’s study is more representative experience, and research on usability is important to extend of the HCI field at large. our knowledge in the field of UX. Conflicting requirements and forces in design The usability of an interactive system is defined as “the Seeing usability aspects as requirements is not new to the extent to which a product can be used by specified users to field of HCI. Cockton [3] discusses the need to align achieve specified goals with effectiveness, efficiency and usability evaluation metrics with stakeholders’ goals and satisfaction in a specified context of use” [7]. Effectiveness requirements for an interactive system. Jokela has described is understood as goal achievement, efficiency involves the how to specify usability requirements in call-for-tenders resources used in reaching the goal, while satisfaction is [8]. related to user perceptions. It is known that requirements to a system under A current debate is related to the degree of correlation development may be in conflict with each other. between these aspects of usability [6, 10, 11], where the Sommerville [12] treats this aspect of requirements aspects are perceived mainly as variables in usability engineering as a negotiation during requirements analysis. studies. In this paper we present an alternative perspective Such negotiation will typically be revisited throughout the to the correlation debate supported by observations in a systems development process as requirements emerge or evolve. Analytical One important aspect of design is to balance conflicting eval. 2 Prototype Analytical Prototype Prototype requirements. Design is about making choices [1]. Using 1 eval. 1 2 3 prototypes iteratively helps us to make these choices when Empirical eval. requirements are not perfect. Within design patterns, conflicting requirements or design constraints may be described as forces to be considered during design [2]. Figure 1. Process steps from Prototype 1 to 3. Non-coloured process steps are covered in the present study. The context: emergency mission reporting By emergency missions we mean emergency responses by The visual prototype was developed through a user-centred professional personnel, coordinated through a central unit. process. An initial set of requirements had been established The particular emergency context in this study is ambulance prior to the current project. The lessons learnt of this paper responses. are the result of process steps where we conducted an expert evaluation of an initial visual prototype (Prototype The particular task targeted in the present study is the status 1), refined the prototype (Prototype 2), and finally reporting conducted by the ambulance personnel throughout conducted empirical evaluations and usability inspections the mission, where the personnel are required to report on the refined visual prototype. The process steps are when they enter one of a set of predefined statuses. The visualised in Figure 1. status values have a natural sequence, but in certain cases one status may be skipped or the rescue task may be Prototype 1 was a non-clickable visual presentation of the cancelled/finished before all statuses are visited. layout and suggested functionality. The first analytical evaluation was an informal expert evaluation with two For the present study, three users or stakeholders are of independent usability experts (the authors of this paper). particular interest: (a) The ambulance personnel as end users of the mobile device, (b) the central unit as receivers On the basis of the informal expert evaluation, the of the status reports, and (c) the legislators providing developer presented a clickable Prototype 2. This prototype regulatory requirements on emergency health care. was subjected to analytical and empirical evaluations with real users. Typically, an ambulance is manned with a driver and a paramedic. The end-users’ environment of reporting is Analytical evaluations were conducted as group-based highly efficiency oriented. On the road, the ambulance may expert walkthroughs [4]. Two sessions were conducted, drive at high speed, the on-board paramedic may be with four or five ambulance personnel as evaluators in each occupied with a patient and at pickup and delivery every group. second potentially counts in order to save lives. Empirical evaluations were conducted as an adapted The requirements regarding the end users’ primary task – version of cooperative usability testing [5], with alternating conducting an efficient emergency mission – may be in phases of interaction and interpretation. Eight ambulance conflict with the requirements of the central unit or from the personnel participated in individual testing sessions. regulatory requirements given by legislators. To the On the basis of the evaluations, the test leaders established ambulance personnel on a mission, status reporting may be overall redesign suggestions and a set of usability considered to be “noise” that should take as little time as predictions. possible. From the perspective of the central unit, in cases of complaints about the response, or in the case of audits on DESIGN FOR STATUS REPORTING compliance with regulatory requirements, the status The main screen of Prototype 2 is presented in Figure 2. reporting should be of high quality: it should never be Through this screen (presented on an 8 inch touch screen), forgotten, and it should always be reported with correct the functionality of the support system - including status time stamps. reporting – is available. OBJECTIVE AND METHOD One may wonder whether there are many design issues In the example case studied, the objective was to design the connected to a task as simple as status reporting. The functionality for status reporting for in-vehicle users in usability evaluations showed that indeed there are. To our ambulances. During our work with the visual prototype, we surprise, the users were very concerned regarding the discovered that this reporting involved an interesting needed number of screen taps, the location of different conflict between effectiveness and efficiency requirements. buttons, the layout of the buttons and the labels on certain In this paper we want to share these as lessons learnt; as an buttons – issues that are normally more present in the mind- example of an application with such a conflict. The set of usability experts rather than end users. experiences were attained through the development process, however without the support of a formal research design. Available No active tasks Map view Solution C: Automatic reporting. This means using some criteria that make it sufficiently likely that a status change has occurred to update the status automatically, thus requiring no user interaction (as there is no interaction, this solution is not illustrated). One example of this is when an ambulance has been notified and has driven for a certain distance above a certain speed, the status should be changed to “started driving”. In the same way, when the status is “started driving”, the ambulance is in a certain vicinity of the emergency site and the ambulance has been standing still for a certain period of time, the status should be changed to “arrived at incident”. Effectiveness perspective Road To support the effectiveness goal of making sure that the reported statuses are correct, a solution that minimises the Figure 2: Main screen of Prototype 2. The status button is at chances for making errors is needed. Because of the small the upper left. screen, and especially when operated while driving, the precision of taps on the screen may be fairly low. For the In the following, we focus on three main design alternatives same reason, unintentionally tapping more than once on the for status reporting: Solutions A, B and C. Solution A was screen may easily happen. used in Prototype 2; Solutions B and C were suggested during the evaluation of Prototype 2, and were thus not Solution A supports correct status reporting best. By evaluated in the case we describe in this paper. presenting the possible status values, the user will both get a degree of consciousness with regards to statuses and Solution A: Button opening menu. A button on the reporting them, and by presenting them in a menu the user periphery of the screen (labelled “Available” in Figure 2) must make a conscious choice for the new value. As the shows the current status. When tapped, a full screen menu choices are explicit and organised in the “natural” is used to change the status. The suggested status button sequence, the risk of making an incorrect choice is reduced. menu is presented in Figure 3. The buttons for passed statuses are passive, showing valuable information like the Although it may seem that Solution B supports correct time stamp for the status change and distance travelled reporting in “normal” cases, it increases the risk of making since the change. Clicking on one of the status buttons errors, either by unintentionally tapping more than once on closes the menu, updates the status information on the the button (and thus doing two status changes) or by current status button and returns to the screen on which the tapping on the status button unintentionally, for example current status button was tapped. while wanting to tap on one of the buttons next to the status button. The former error may be avoided by inducing a Solution B: Toggling button (one-click status update). forced delay between subsequent status changes. The latter When the button that shows current status is tapped, the error is difficult to avoid and will introduce the need for status changes to the next status in the “natural” sequence. functionality for correcting the status – functionality that Thus there will be no submenu, and the status may be will anyway be needed to do “unnatural” status changes. changed with one tap on the top left-hand button on the main screen (Figure 2). To what degree Solution C supports correct reporting depends on the quality of the automatic reasoning, but there is always a risk that a false status change is reported. This Close may, for example, cause the central unit to believe that an emergency mission has been accepted by an ambulance, Turned out Arrived pickup Left pickup when, in fact, it has not. This is an argument for only using such reasoning for reminding users about status changes, not for automatic reporting, alternatively forcing the users to confirm automatic status changes. To support the effectiveness goal of making sure that the Arrived place Available Available status changes are reported at the correct time, none of the at central of delivery solutions are optimal. The importance of assuring that reporting is indeed performed, may point to Solution C or a Figure 3: Suggested status reporting menu in Solution A reminder combined with Solution A or B, but the automatic reasoning about status changes requires that the ambulance has been driving for a while before the status change is detected. Thus, the time that is reported for the status Solution A is the least efficient one - in the evaluations, a change, which is important from a legislative point of view, number of users found the menu unnecessary. It requires a will be incorrect. This could be compensated for by setting number of clicks and, at least for inexperienced users, a bit the time for the status change to the time when the of reading to find the correct button to press. ambulance started driving, but there may also be cases The efficiency of this solution also depends on the layout of where this is not correct. the menu choices. Prototype 2 presented the status choices In summary, the “best” solution from the in their natural sequence, with the buttons in fixed effectiveness/control perspective seems to be Solution A positions. An alternative design proposed during the with reminder functionality. informal usability expert walk through was to organise the buttons as a “carousel”, always showing the next natural Efficiency perspective choice as the topmost choice, and using different sizes of The users in the ambulance focus on the main task of rescuing buttons to illustrate how “natural” it was to choose a given lives at an emergency site. They know what the status is, and status, as illustrated in Figure 4. This solution is potentially the sequence of status changes is identical or very similar in all highly efficient for handling normal status changes, but is a emergency missions, so reporting status changes is of little bit “unstable”, in the sense that the same menu choices value for them. Thus, an important goal of the users inside the appear at different positions in different contexts. Although vehicle is to perform this task as efficiently as possible; i.e., not presented as part of the prototype used in the using as few screen taps as possible and reducing the need for evaluations with the end users, other findings from these reading items on the screen. evaluations showed that the users have a strong urge to Seen from this perspective, Solution C is best suited, as it have consistent locations of screen elements. requires no actions by the users. The variant requiring In summary, the “best” solution from the efficiency confirmation by the users also seems well suited, although perspective is Solution C, and if augmented with a such confirmation may come at very unsuitable moments. confirmation function, it is probably equal to Solution B, As the users may be performing a highly attention-requiring depending on how the unsolved issues with regards to this task, a reasonable design solution is that the users should solution are resolved. choose the appropriate time to perform user interactions. Such confirmations violate this principle, but may still be a DISCUSSION usable compromise. Designing status reporting The end users' needs for efficiency in the reporting task Solution B also supports efficiency to a large degree. For indicate that reporting should preferably be performed “normal” emergency missions performed by experienced automatically. If forced to perform reporting, the user users knowing the sequence of the possible statuses, interface for doing this should require as few taps and as updating the status may be done with one tap on the screen. little reading as possible. During the evaluation activities, A possible solution to correcting errors and handling the end users communicated a desire for being able to “unnatural” status changes is to have the status button as a operate the routine parts of the reporting task almost split button like the back and forward button in most web “blindfolded”. Taking the effectiveness perspective, this browsers, that may be used both for doing direct operations desire is risky, as the chances of performing erroneous and for opening a menu. This is a solution that works well reporting increase when the user is not reading text on the on a desktop computer, but that requires a level of precision screen. when tapping that is neither anticipated nor desired on a touch screen solution used in a vehicle while driving. Although such use is a special risk for Solution B, it should also be mentioned that both layout choices for the status reporting menu in Solution A invite “blindfolded” use for Left pickup experienced users. Confirmation of “unnatural” choices is one way of reducing this risk. Another way of compromising between the two perspectives is to use aural Arrive pickup Arrived feedback to confirm the choices. This may be well suited in delivery all three solutions, but maybe most important in Solutions B Turned out and C. A drawback of using sound is the noisy environment Available in an ambulance. Generalising our findings Available central Although it is often the case that effectiveness and efficiency correlate positively [11], our example shows that this is not always the case. As is foreseen by Sommerville, Figure 4: “Carousel” version of status reporting menu multiple stakeholders typically imply conflicting requirements. It should therefore be no surprise that requirements concerning the usability aspects may also be FUTURE WORK in conflict. Though efficiency in reporting may be more It is risky to make general conclusions based on highly prioritised by the ambulance personnel, effectiveness observations in only one example case and we do not claim may be seen as more important from the perspective of the that the reasons for the conflict between efficiency and central unit. effectiveness in our study stem only from the possible reasons that have been pointed out. Neither may we In our view, it is important to be able to identify cases with conclude that all other cases with similar characteristics will conflicts between the two, as this may have important display the same conflict. But we hope that the observations implications for the usability – and, by extension, the UX of and discussion may serve as inspiration for discussions on the interactive system. In this section we point out some the relationship between effectiveness and efficiency as possible reasons why the conflict occurs in the given case. aspects of usability and UX; in particular with reference to We assume that other cases with similar characteristics may the characteristics of cases where a conflict between these experience the same conflict. two aspects is likely to occur. (1) Conflicts between stakeholders. The effectiveness needs from the central unit and the need for compliance ACKNOWLEDGMENTS with regulatory requirements, conflict with the end users’ This work has been done within the research projects needs of being effective and efficient when performing the FLAMINKO and EMERGENCY, both financed by the emergency mission, making efficiency in the reporting task Norwegian Research Council VERDIKT program and the of prime importance. In other application areas, different participating partners. Thanks to Locus (www.locus.no) for stakeholders often have similar interests, e.g. to make a their participation in the case. purchase process as smooth as possible in an eCommerce system. REFERENCES [1] Beadouin-Lafon, M., and Mackay, W.E. Prototyping (2) Nature of application area. The users in the ambulance tools and techniques. In Sears, A., Jacko, J., A. (Eds.) experience that their primary task of saving lives conflicts The human computer-interaction handbook. 2. ed. with the secondary task of reporting their status. The task Lawrence Erlbaum Associates, New York, NY (2008), conflict is accentuated in the given application area as the 1017-1040. prime task is highly attention-demanding. Other application [2] Borchers, J. A Pattern Approach to Interaction Design. areas, characterized by the primary task being conducted in the application, may not observe such conflicts. In Proc DIS '00. ACM Press (2000), 369-378. [3] Cockton, G. Putting Value into E-valu-ation. In eds. E. (3) Strong legislative requirements. The strong legislative Law, E. L-C., Hvannberg, E., Cockton, G. (Eds.) requirements make correct reporting much more important Maturing Usability: Quality in Software, Interaction than in cases where incorrect information would at worst and Value. Springer (2008), 287-317. lead to a package being delivered to a wrong address, or a small economic loss. It should also be mentioned that [4] Følstad, A. 2007 Group-based Expert Walkthrough”. In conforming to legislation is also in the interests of the users Scapin, D., Law, E. (Eds.) R3UEMs: Review, Report in the ambulance, thus raising a conflict of interest for these and Refine Usability Evaluation Methods. Proceedings users independently of other stakeholders. of the 3. COST294-MAUSE International Workshop (2007), 58-60. http://cost294.org/upload/522.pdf We perceive our findings and the possible reasons for them [5] Frøkjær, E., Hornbæk, K. Cooperative Usability as a relevant input to the correlation debate. When the discussion – as it seems to be at present – is oriented Testing: Complementing Usability Tests with User- towards correlation of usability aspects as a general Supported Interpretation Sessions. In Proc. CHI 2005. phenomenon in usability studies, we may lose sight of the ACM Press (2005), 1383-1386. most important place for considering the relationship [6] Hornbæk, K., Law, E. L-C. Meta-Analysis of between effectiveness and efficiency; namely, in the Correlations among Usability Measures. In Proc. CHI requirements and design phases. 2007. ACM Press (2007), 617-626. Our findings may also serve as basis for discussions about [7] ISO. Ergonomic requirements for office work with the applicability of SUM. In cases where the standard visual display terminals (VDTs) – Part 11: Guidance usability aspects can be seen to contain conflicting on usability, International standard ISO 9241- requirements, some caution may be needed when applying 11:1998(E). SUM. However, it may well be that if conflicting [8] Jokela, T. Determining usability requirements into a requirements are well-managed throughout design and call-for-tenders: a case study on the development of a development, SUM may still provide an adequate single healthcare system. In Proc. NordiCHI 2008. ACM estimate of overall system usability – though valuable Press (2008), 256-265. details about the standard aspects of usability may also be needed. [9] Law, E., Abrahão, S., Stage, J. Proceedings of [11] Sauro, J., Lewis, J.R. Correlations among Prototypical international workshop on the interplay between user Usability Metrics: Evidence for the Construct of experience evaluation and software development (I- Usability. In Proc. CHI 2009. ACM Press (2009), UxSED 2010), editorial introduction. The CEUR 1609-1618. workshop proceedings series, 656 (2010). http://ceur- [12] Sommerville, I., Sawyer, P. (1997) Requirements ws.org/Vol-656/ Engineering. John Wiley & Sons, Chichester, UK. [10] Sauro, J., Kindlund, E. A Method to Standardize Usability Metrics into a Single Score. In Proc. CHI 2005. ACM Press (2005), 401-409.