Evaluation of user interface adaptation strategies in the process of model-driven user interface development Kai Breiner, Volkmar Gauckler Marc Seissler Gerrit Meixner University of Kaiserslautern University of Kaiserslautern German Research Center for Software Engineering Research Center for Human-Machine- Artificial Intelligence (DFKI) Group Interaction Trippstadter Str. 122 Gottlieb-Daimler Str. Gottlieb-Daimler Str. 67663, Kaiserslautern, 67663, Kaiserslautern, Germany 67663, Kaiserslautern, Germany Germany breiner@cs.uni-kl.de, Marc.Seissler@mv.uni-kl.de Gerrit.Meixner@dfki.de volkmar.gauckler@gmx.net ABSTRACT concerning ubiquitous computing – also in production In this paper, we describe the evaluation of our prototype environments – is becoming a reality [10]. Universal Control Device (UCD), which enables the control of various devices in modern dynamic production In today’s production environments, technical devices often environments, while being able to adapt itself to the current stem from multiple vendors using heterogeneous user configuration of the environment. While it is hard to apply interfaces that differ in terms of complexity, look&feel, and traditional user interface design heuristics in recent interaction styles. Such highly complex and networked paradigms – such as Ambient Intelligence – there is a technical devices or systems can provide any information at demand for suitable compensation strategies addressing any time and in any place. This advantage can turn out to be usage errors, which can be met by applying an adequate a disadvantage when information is not presented properly adaptation strategy. In a pilot study, we gained experience according to the users’ needs. This leads to problems, regarding differences in the performance of selected especially concerning the usability of the user interface. adaptation strategies in the case of our demonstrator. The level of acceptance of a user interface largely depends on its ease and convenience of use. A user can work with a Author Keywords technical device more efficiently if the user interface is Universal Control Device, Adaptation Strategies, tailored to the users’ needs, on the one hand, and to their SmartFactory. abilities on the other hand. Therefore, providing ACM Classification Keywords information in a context- and location-sensitive manner H5.2. Information interfaces and presentation (e.g., HCI): (depending on user, situation, machine, environmental Evaluation/methodologie, Prototyping, User-centered conditions, etc.) has to be ensured. design. Hence, in the following we will give a short introduction to INTRODUCTION the SmartFactoryKL, which serves as a demonstration The ongoing technological development of environment for future intelligent production environments, microelectronics and communication technology is leading and a Universal Control Device (UCD), which provides to more pervasive communication between single devices holistic control to various devices in such environments. or entire pervasive networks of intelligent devices (smart Further, we give a brief introduction to user interface phone, PDA, netbook, etc.). Especially industrial devices adaptation, usage errors, as well as to their compensation. and applications can take advantage of modern smart After presenting our idea of how to approach compensation technologies, e.g., based on ad-hoc networks, dynamic by using adaptation strategies, we describe the set-up of the system collaboration, and context-adaptive human-machine corresponding controlled experiment and the preliminary interaction systems. The vision of Mark Weiser [1] results of the pilot study conducted. We conclude with the interpretation of how the results contribute towards our hypothesis. Permission to make digital or hard copies of all or part of this work for KL personal or classroom use is granted without fee provided that copies are SmartFactory Pre-proceedings of the 5th International not made or distributed for profit orWorkshop on Model commercial Drivenand advantage Development that copies Besides these aspects, modern production environments are ofbear Advanced User Interfaces this notice (MDDAUI and the full citation2010): Bridging on the between first page. To User copyExperience otherwise, characterized by a modular layout. Entire modules can be and or UI Engineering, republish, organized to post at the 28th on servers ACM or to Conferencetoonlists, redistribute Human Factorsprior requires in Computing Systems (CHI 2010), Atlanta, Georgia, USA, April 10, 2010. replaced or reorganized. Furthermore, these environments specific permission and/or a fee. CHI 2009, April 4–9, 2009, Boston, Massachusetts, USA. are able to react to errors occurring in the production Copyright © 2010 for the individual papers by the papers' authors. Copying process (e.g., device malfunction) and to dynamically Copyright 2009 ACM 978-1-60558-246-7/09/04...$5.00. permitted for private and academic purposes. Re-publication of material from this volume requires permission by the copyright owners. This volume is published by reorganize parts of the process in order to ensure the its editors. production process. Thus, this also affects the user who 17 interacts with the individual devices – the user’s workflow Basically, there are two kinds of operating errors that may will change, or devices will not be available anymore. lead to a failure of the system. Serving as a demonstration environment, the In the first case, there are slips. Here, the user had the SmartFactoryKL in Kaiserslautern, Germany is able to correct intention but executed the task in the wrong way. simulate such a process. In previous work, we developed a Most often this is caused by poor physical skills, or by the UCD, which is able to provide access to various devices of user interface just being just inadequate for use (e.g., the SmartFactoryKL [3][4]. buttons too small). In the second case – which is more interesting in our example – there are errors. Errors are Universal Control Device (UCD) As a result of a model-driven process, the user interface of characterized by the user having the wrong intention while the UCD is being generated at run-time [2]. Starting with a executing the task. This is caused by a misunderstanding of topological description of the environment, enriched with the user interface (e.g., if the user interface is offering user tasks on the single devices and information about how control over devices that are not available physically). to address the single devices, this information is sufficient for generating a functional user interface [2][4]. In order to Usage Error Cause remain functional to the user, this user interface has to correspond to the current configuration of the environment and is additionally restricted to the functionality as specified in the underlying model. During field studies, faced by the need to adapt the user interface, we Compensation encountered the demand for a systematic strategy that would support the user as much as possible [3]. This was Adapt to Misunder- Error the trigger for a study on adaptation, which we will describe Environment standing in the following. Slips Adapt to User Skills ADAPTATION Figure 1. Relationship between usage errors, their causes, and After giving a brief overview of different types of compensation. adaptation – their properties as well as their impact on the users’ workflow – we will elaborate types of usage errors Compensation resulting from static user interfaces and how adaptive user Depending on the type of usage error, there are different interfaces contribute to the compensation of such errors. ways to compensate in order to minimize the effect. Slips, Static versus Dynamic User Interfaces which are caused by poor skills of the user, can be On the one hand, one important usability quality attribute is prevented by adapting the user interface to the user. In case memorizability. The ease to remember helps users to speed of our application domain – intelligent production up the process of interacting with the user interface [5][6]. environments – we are dealing with predefined user roles Since humans have a very visual memory, the way a certain and user groups and are therefore able to tailor the user user interface is structured is essential for finding items interface to the needs of these user groups. faster. After a while, users form a coherent model of the In dynamic environments, errors can be prevented by user interface and are able to recall how to execute their adhering to design heuristics as mentioned earlier. The workflow. If the system (and therefore the user interface) system has to represent the current configuration of the user changes frequently, the user will not be able to form such a interface, while supporting the development of a mental model [7][8]. On the other hand, there are user interface model. Hence, a method of adaptation needs to be chosen heuristics demanding that the user interface matches the that contributes to these heuristics. real world [6]. In order to provide control over devices in dynamic environments – such as the SmartFactoryKL – it is Types of Adaptation extremely vital to match the user interface to the real world There are different methods of how an adaptation of the in order to remain functional. user interface can be executed. These methods differ in their way of execution as well as in their degree of intrusion These simple rules about how to develop a user interface into the users’ workflow. In case of the UCD, a simple appear to be contradictory for future information systems in adaptation scenario would be the appearance or our application domain. Hence, there is a need for an disappearance of devices in the device selection list. In the adaptation strategy that does not violate usability quality following, we will refer to this example while giving details attributes leading to usage errors. about the individual methods. Usage Errors The first method – which was implemented initially and led In general, for each kind of usage error, there is a basic to the investigation described in this paper – is ad-hoc cause (see Figure 1). Using the right compensation adaptation. Here, devices are added or removed from a technique – such as adaptation of the user interface – the device list according to the physical status of the respective users can be actively supported in preventing usage errors. 18 devices. Unless a regular user permanently observes the hypotheses are tailored to the specific set-up of the device list, he or she will not notice any change. controlled experiment. Furthermore, this will be distracting for the user, because H1. Effectiveness – completion rate the structure of the user interface changes without We assume that, on average, at least 85,6% of the test tasks notification. can be completed without help. Providing more information about the system state leads to Explanation: This hypothesis assumes that test persons can the second adaptation method – notification. Now, the user deal with the user interfaces and have understood their tasks interface provides information about the change and and therefore can complete at least 6 out of the 7 tasks. therefore supports the user in adapting his or her mental model. In case of the device list example, we implemented H2. Effectiveness – assistance this method by applying the so-called instant-messenger 100% of the given tasks can be completed (with help – if metaphor [9]. This means that new or defective devices will needed). be emphasized graphically, which provides information for Explanation: The user interface ensures consistency the user to understand the current state of the system. between environment and visualization independent of the However, the user may just overlook the notifications if he adaptation strategy. Therefore, it should be possible to or she is distracted, and then the system cannot provide complete each task. further support. H3. Efficiency – strategy performance Thus, we implemented a third adaptation strategy, which Confirmation outperforms notification, which outperforms aims at confirmation by the user. Here, the user has to ad-hoc adaptation. actively confirm the change to the user interface. Referring to the device list of the example above, this strategy was Explanation: On the basis of the different attributes implemented in terms of a dialog box asking for elaborated earlier, we assume this ranking according to the confirmation from the user that he or she has actually performance of the different strategies. noticed the adaptation of the system. Thus, the system can EVALUATION – PILOT STUDY be almost sure that the user is aware of the adaptation and is To evaluate these hypotheses, we decided to conduct a supported in the presumably most adequate way. A controlled experiment. We implemented three instances of negative side-effect of this strategy is that the confirmation our model-driven process [2][3][4], including the dialog may distract the user during his/her regular corresponding adaptation strategies. The test persons had to workflow. complete seven tasks on the user interface, while the Compensation simulated production environment always reconfigured Each of the adaptation strategies differs as to how much it between task 2 and task 3. The idea was that the adaptation contributes to the compensation of possible usage errors. should only affect task 4, which had to be executed in a different way (as a result of the adaptation) than indicated at Ad-hoc adaptation ensures consistency between the first. The other tasks served as an indication that the test controllable environment and the corresponding user persons perform in a similar way. We conducted a pilot interface in order to prevent usage errors with respect to study with several computer science students. All six test outdated user interfaces. But this approach provides no persons were between 21 and 34 years old. active support to the user at all. After being asked for personal statistical information, the Besides consistency with the current configuration of the test persons got a thorough introduction to the production environment, notification provides limited feedback to the industry domain. They were provided with a detailed user (e.g., by visually emphasizing new devices). Due to the explanation of the simulated production environment as fact that this communication with the user is unidirectional, well as of our Universal Control Device. After execution of such notifications can be easily overlooked by the user. task 2, we initiated the reconfiguration of the simulation Confirmation provides all the functionality of the first two and, depending on the strategy used (of which the test approaches on the one hand and demands confirmation by persons were not aware) the user interface adapted itself, the user on the other hand. Thus, the user interface is aware notified the user, or demanded confirmation. The strategies of the fact that the user has recognized the new were distributed equally between the tests. Due to the configuration and can therefore proceed with the regular adaptation of the environment, the user should not be able functionality. to complete task 4 in the way the task description called for. One of three redundant pumps was removed from the Hypothesis system. Here, the user should conclude that (as displayed in According to the diverse properties of these strategies, we the documentation of the simulated environment) there are wanted to investigate which is the most adequate one in the various ways to complete this task and ask the moderator if case of our model-driven approach and therefore we this is possible. The test was completed with a survey about formulated hypotheses that we are going to verify or falsify the subjective properties of the user interface. initially in a preliminary study described below. The 19 216 there are special requirements. According to a shown dissonance in user interface design, when being applied in 173 these environments, we have shown there is a special need for compensating usage errors. This can be achieved by 130 systematically integrating adaptation strategies into the time [s] model-driven development process. Since multiple 86 strategies exist, which provide different user experience, the performance with respect to our demonstrator was 43 evaluated in a pilot study of a controlled experiment. First results show that there are differences in the performance 00 1 2 3 Tasks 4 5 6 7 and therefore some of our hypotheses could be verified. Figure 2. The performance of the 7 tasks (median, max/min ACKNOWLEDGMENTS and quantile). Our work as well as the GaBi project is funded in part by the German Research Foundation (DFG). Results Figure 2 shows the results of the performance of the single REFERENCES tasks. As we assumed before conducting the experiment, all 1. Weiser, M. The computer for the 21st century. Scientific tasks (the control tasks) except task 4 were performed American, 265, 3 (1991), 94-104. similarly. Referring to the standard deviation, which is an 2. Breiner, K., Görlich, D., Maschino, O., and Meixner, G. average of 16 seconds in case of the control tasks and 38 Towards automatically interfacing application services seconds in case of task 4, the difference in performance can integrated in an automated model based user interface be easily identified. Thus, we conclude that the results of generation process, Proc. of the 4th International this preliminary experiment are representative. Workshop on Model-Driven Development of Advanced When executing task 4, the test persons recognized the User Interfaces (MDDAUI), CEUR Workshop redundancy of the three pumps and asked if they could use Proceedings Vol-439 (2009). another pump, which was intended. 3. Breiner, K., Görlich, D., Maschino, O., Meixner, G., and Zühlke, D. Run-Time Adaptation of a Universal User Discussion of the Results Interface for Ambient Intelligent Production All test persons were able to complete all tasks (with help Environments. Proc. of the 13th International in case of task 4), which confirms hypothesis H1. Since all Conference on Human-Computer Interaction (HCII-09), of them needed help when executing task 4 and only during LNCS Vol. 5613 (2009), 663-672. task 4, the estimated 85,6% of hypothesis H2 was almost exactly confirmed. 4. Breiner, K., Görlich, D., Maschino, O., and Meixner, G. Automatische Generierung voll funktionsfähiger For ah-hoc adaptation, the execution of task 4 took an mobiler Bediensoftware aus Benutzungs- und average of 117 seconds, for notification 125 seconds, and Funktionsmodellen. Proc. of Informatik, LNI Vol. P-154 for confirmation 89 seconds. Hence, hypothesis H3 cannot (2009), 2210-2215. be entirely confirmed, as ad-hoc adaptation outperformed notification, but still confirmation outperformed both of the 5. Gould, J. D. and Lewis, C. 1985. Designing for other strategies. usability: key principles and what designers think. Communications of the ACM, 28, 3 (1985), 300-311. Threats to validity Because all the test persons were only computer science 6. Nielsen, J. Ten Usability Heuristics. students, this may have had an effect on the result. But on www.useit.com/papers/heuristic/heuristic_list.html. the other hand, those students were already familiar with 7. Norcio, A. F., and Stanley, J. Adaptive Human - our previous work (without knowing the content of the Computer Interfaces. NRL Report 9148, Naval Research experiment), which could also be a good simulation of Laboratory (1988). domain knowledge. Nevertheless, the experiment needs to 8. Mitchell, J. and Shneiderman, B. Dynamic versus static be conducted (and will be) using production industry menus: an exploratory comparison. SIGCHI Bulletin, domain experts. The most important threat to validity to 20, 4 (1989), 33-37. mention is the fact that this was only the pilot study for the described experiment. This means that the sample was too 9. Lee, L. and Johnson T. URCousin: Universal Remote small and therefore has no real statistical evidence, but it Control User Interface. Proc. of the Human Interface serves as a first indication as to whether an investigation Technologies Conference (2006). according our idea would make sense. 10. Zuehlke, D.: SmartFactory – From Vision to Reality in CONCLUSION Factory Technologies. Proc. of the 17th International When dealing with user interfaces in highly dynamic Federation of Automatic Control (IFAC) World environments, such as intelligent production environments, Congress (2008), 82-89. 20