Aspects of usability, such as effectiveness and efficiency, are critical for users' overall experience of an interactive system. In response to the on-going debate on the relationship between different aspects of usability in usability studies, we present an example of a User Interface (UI) design case where the relationship between effectiveness and efficiency should be considered as a requirement or design issue, rather than as variables in usability studies. In the presented case - status reporting from an in-vehicle support system for emergency missions these aspects of usability were perceived as conflicting rather than as positively correlated. We present various design solutions to the task of status reporting and show how the solutions support effectiveness and efficiency in different ways. Finally, we point out some characteristics of the case that could explain our findings and we suggest how future research may obtain more insight into which types of applications that may possess similar properties.
INTRODUCTION
Usability is seen as a concept that is included in the broader
concept of user experience (UX) [
The usability of an interactive system is defined as “the
extent to which a product can be used by specified users to
achieve specified goals with effectiveness, efficiency and
satisfaction in a specified context of use” [
A current debate is related to the degree of correlation
between these aspects of usability [
A basic assumption of SUM is that there are fairly high
correlations between the three standard aspects of usability.
This assumption is controversial within the
HumanComputer Interaction (HCI) community, as is seen in a
meta-study by Hornbaek and Law [
In 2009, Sauro and Lewis [
Conflicting requirements and forces in design
Seeing usability aspects as requirements is not new to the
field of HCI. Cockton [
It is known that requirements to a system under
development may be in conflict with each other.
Sommerville [
One important aspect of design is to balance conflicting
requirements. Design is about making choices [
The particular task targeted in the present study is the status reporting conducted by the ambulance personnel throughout the mission, where the personnel are required to report when they enter one of a set of predefined statuses. The status values have a natural sequence, but in certain cases one status may be skipped or the rescue task may be cancelled/finished before all statuses are visited. For the present study, three users or stakeholders are of particular interest: (a) The ambulance personnel as end users of the mobile device, (b) the central unit as receivers of the status reports, and (c) the legislators providing regulatory requirements on emergency health care. Typically, an ambulance is manned with a driver and a paramedic. The end-users’ environment of reporting is highly efficiency oriented. On the road, the ambulance may drive at high speed, the on-board paramedic may be occupied with a patient and at pickup and delivery every second potentially counts in order to save lives.
The requirements regarding the end users’ primary task – conducting an efficient emergency mission – may be in conflict with the requirements of the central unit or from the regulatory requirements given by legislators. To the ambulance personnel on a mission, status reporting may be considered to be “noise” that should take as little time as possible. From the perspective of the central unit, in cases of complaints about the response, or in the case of audits on compliance with regulatory requirements, the status reporting should be of high quality: it should never be forgotten, and it should always be reported with correct time stamps.
OBJECTIVE AND METHOD In the example case studied, the objective was to design the functionality for status reporting for in-vehicle users in ambulances. During our work with the visual prototype, we discovered that this reporting involved an interesting conflict between effectiveness and efficiency requirements. In this paper we want to share these as lessons learnt; as an example of an application with such a conflict. The experiences were attained through the development process, however without the support of a formal research design. Prototype 1
Analytical eval. 1
Prototype 2
Prototype
3 Analytical eval. 2 Empirical eval.
The visual prototype was developed through a user-centred process. An initial set of requirements had been established prior to the current project. The lessons learnt of this paper are the result of process steps where we conducted an expert evaluation of an initial visual prototype (Prototype 1), refined the prototype (Prototype 2), and finally conducted empirical evaluations and usability inspections on the refined visual prototype. The process steps are visualised in Figure 1.
Prototype 1 was a non-clickable visual presentation of the layout and suggested functionality. The first analytical evaluation was an informal expert evaluation with two independent usability experts (the authors of this paper). On the basis of the informal expert evaluation, the developer presented a clickable Prototype 2. This prototype was subjected to analytical and empirical evaluations with real users.
Analytical evaluations were conducted as group-based
expert walkthroughs [
Empirical evaluations were conducted as an adapted
version of cooperative usability testing [
DESIGN FOR STATUS REPORTING The main screen of Prototype 2 is presented in Figure 2. Through this screen (presented on an 8 inch touch screen), the functionality of the support system - including status reporting – is available.
One may wonder whether there are many design issues connected to a task as simple as status reporting. The usability evaluations showed that indeed there are. To our surprise, the users were very concerned regarding the needed number of screen taps, the location of different buttons, the layout of the buttons and the labels on certain buttons – issues that are normally more present in the mindset of usability experts rather than end users.
Road
In the following, we focus on three main design alternatives for status reporting: Solutions A, B and C. Solution A was used in Prototype 2; Solutions B and C were suggested during the evaluation of Prototype 2, and were thus not evaluated in the case we describe in this paper.
Solution A: Button opening menu. A button on the periphery of the screen (labelled “Available” in Figure 2) shows the current status. When tapped, a full screen menu is used to change the status. The suggested status button menu is presented in Figure 3. The buttons for passed statuses are passive, showing valuable information like the time stamp for the status change and distance travelled since the change. Clicking on one of the status buttons closes the menu, updates the status information on the current status button and returns to the screen on which the current status button was tapped.
Solution B: Toggling button (one-click status update). When the button that shows current status is tapped, the status changes to the next status in the “natural” sequence. Thus there will be no submenu, and the status may be changed with one tap on the top left-hand button on the main screen (Figure 2).
Close Solution C: Automatic reporting. This means using some criteria that make it sufficiently likely that a status change has occurred to update the status automatically, thus requiring no user interaction (as there is no interaction, this solution is not illustrated). One example of this is when an ambulance has been notified and has driven for a certain distance above a certain speed, the status should be changed to “started driving”. In the same way, when the status is “started driving”, the ambulance is in a certain vicinity of the emergency site and the ambulance has been standing still for a certain period of time, the status should be changed to “arrived at incident”.
Effectiveness perspective To support the effectiveness goal of making sure that the reported statuses are correct, a solution that minimises the chances for making errors is needed. Because of the small screen, and especially when operated while driving, the precision of taps on the screen may be fairly low. For the same reason, unintentionally tapping more than once on the screen may easily happen.
Solution A supports correct status reporting best. By presenting the possible status values, the user will both get a degree of consciousness with regards to statuses and reporting them, and by presenting them in a menu the user must make a conscious choice for the new value. As the choices are explicit and organised in the “natural” sequence, the risk of making an incorrect choice is reduced. Although it may seem that Solution B supports correct reporting in “normal” cases, it increases the risk of making errors, either by unintentionally tapping more than once on the button (and thus doing two status changes) or by tapping on the status button unintentionally, for example while wanting to tap on one of the buttons next to the status button. The former error may be avoided by inducing a forced delay between subsequent status changes. The latter error is difficult to avoid and will introduce the need for functionality for correcting the status – functionality that will anyway be needed to do “unnatural” status changes. To what degree Solution C supports correct reporting depends on the quality of the automatic reasoning, but there is always a risk that a false status change is reported. This may, for example, cause the central unit to believe that an emergency mission has been accepted by an ambulance, when, in fact, it has not. This is an argument for only using such reasoning for reminding users about status changes, not for automatic reporting, alternatively forcing the users to confirm automatic status changes.
To support the effectiveness goal of making sure that the status changes are reported at the correct time, none of the solutions are optimal. The importance of assuring that reporting is indeed performed, may point to Solution C or a reminder combined with Solution A or B, but the automatic reasoning about status changes requires that the ambulance has been driving for a while before the status change is detected. Thus, the time that is reported for the status change, which is important from a legislative point of view, will be incorrect. This could be compensated for by setting the time for the status change to the time when the ambulance started driving, but there may also be cases where this is not correct.
In summary, the “best” solution from the effectiveness/control perspective seems to be Solution A with reminder functionality.
Efficiency perspective The users in the ambulance focus on the main task of rescuing lives at an emergency site. They know what the status is, and the sequence of status changes is identical or very similar in all emergency missions, so reporting status changes is of little value for them. Thus, an important goal of the users inside the vehicle is to perform this task as efficiently as possible; i.e., using as few screen taps as possible and reducing the need for reading items on the screen.
Seen from this perspective, Solution C is best suited, as it requires no actions by the users. The variant requiring confirmation by the users also seems well suited, although such confirmation may come at very unsuitable moments. As the users may be performing a highly attention-requiring task, a reasonable design solution is that the users should choose the appropriate time to perform user interactions. Such confirmations violate this principle, but may still be a usable compromise.
Solution B also supports efficiency to a large degree. For “normal” emergency missions performed by experienced users knowing the sequence of the possible statuses, updating the status may be done with one tap on the screen. A possible solution to correcting errors and handling “unnatural” status changes is to have the status button as a split button like the back and forward button in most web browsers, that may be used both for doing direct operations and for opening a menu. This is a solution that works well on a desktop computer, but that requires a level of precision when tapping that is neither anticipated nor desired on a touch screen solution used in a vehicle while driving.
Left pickup Arrive pickup Turned out
Arrived delivery Available Available central Solution A is the least efficient one - in the evaluations, a number of users found the menu unnecessary. It requires a number of clicks and, at least for inexperienced users, a bit of reading to find the correct button to press.
The efficiency of this solution also depends on the layout of the menu choices. Prototype 2 presented the status choices in their natural sequence, with the buttons in fixed positions. An alternative design proposed during the informal usability expert walk through was to organise the buttons as a “carousel”, always showing the next natural choice as the topmost choice, and using different sizes of buttons to illustrate how “natural” it was to choose a given status, as illustrated in Figure 4. This solution is potentially highly efficient for handling normal status changes, but is a bit “unstable”, in the sense that the same menu choices appear at different positions in different contexts. Although not presented as part of the prototype used in the evaluations with the end users, other findings from these evaluations showed that the users have a strong urge to have consistent locations of screen elements.
In summary, the “best” solution from the efficiency perspective is Solution C, and if augmented with a confirmation function, it is probably equal to Solution B, depending on how the unsolved issues with regards to this solution are resolved.
DISCUSSION Designing status reporting The end users' needs for efficiency in the reporting task indicate that reporting should preferably be performed automatically. If forced to perform reporting, the user interface for doing this should require as few taps and as little reading as possible. During the evaluation activities, the end users communicated a desire for being able to operate the routine parts of the reporting task almost “blindfolded”. Taking the effectiveness perspective, this desire is risky, as the chances of performing erroneous reporting increase when the user is not reading text on the screen.
Although such use is a special risk for Solution B, it should also be mentioned that both layout choices for the status reporting menu in Solution A invite “blindfolded” use for experienced users. Confirmation of “unnatural” choices is one way of reducing this risk. Another way of compromising between the two perspectives is to use aural feedback to confirm the choices. This may be well suited in all three solutions, but maybe most important in Solutions B and C. A drawback of using sound is the noisy environment in an ambulance.
Generalising our findings
Although it is often the case that effectiveness and
efficiency correlate positively [
In our view, it is important to be able to identify cases with conflicts between the two, as this may have important implications for the usability – and, by extension, the UX of the interactive system. In this section we point out some possible reasons why the conflict occurs in the given case. We assume that other cases with similar characteristics may experience the same conflict. (1) Conflicts between stakeholders. The effectiveness needs from the central unit and the need for compliance with regulatory requirements, conflict with the end users’ needs of being effective and efficient when performing the emergency mission, making efficiency in the reporting task of prime importance. In other application areas, different stakeholders often have similar interests, e.g. to make a purchase process as smooth as possible in an eCommerce system. (2) Nature of application area. The users in the ambulance experience that their primary task of saving lives conflicts with the secondary task of reporting their status. The task conflict is accentuated in the given application area as the prime task is highly attention-demanding. Other application areas, characterized by the primary task being conducted in the application, may not observe such conflicts. (3) Strong legislative requirements. The strong legislative requirements make correct reporting much more important than in cases where incorrect information would at worst lead to a package being delivered to a wrong address, or a small economic loss. It should also be mentioned that conforming to legislation is also in the interests of the users in the ambulance, thus raising a conflict of interest for these users independently of other stakeholders.
We perceive our findings and the possible reasons for them as a relevant input to the correlation debate. When the discussion – as it seems to be at present – is oriented towards correlation of usability aspects as a general phenomenon in usability studies, we may lose sight of the most important place for considering the relationship between effectiveness and efficiency; namely, in the requirements and design phases.
Our findings may also serve as basis for discussions about the applicability of SUM. In cases where the standard usability aspects can be seen to contain conflicting requirements, some caution may be needed when applying SUM. However, it may well be that if conflicting requirements are well-managed throughout design and development, SUM may still provide an adequate single estimate of overall system usability – though valuable details about the standard aspects of usability may also be needed.
FUTURE WORK It is risky to make general conclusions based on observations in only one example case and we do not claim that the reasons for the conflict between efficiency and effectiveness in our study stem only from the possible reasons that have been pointed out. Neither may we conclude that all other cases with similar characteristics will display the same conflict. But we hope that the observations and discussion may serve as inspiration for discussions on the relationship between effectiveness and efficiency as aspects of usability and UX; in particular with reference to the characteristics of cases where a conflict between these two aspects is likely to occur.
ACKNOWLEDGMENTS This work has been done within the research projects FLAMINKO and EMERGENCY, both financed by the Norwegian Research Council VERDIKT program and the participating partners. Thanks to Locus (www.locus.no) for their participation in the case.